Crawl Frontier 0.2.0 documentation¶
This documentation contains everything you need to know about Crawl Frontier.
- What is a Crawl Frontier?
- Learn what a crawl frontier is and how to use it.
- Architecture overview
- See how Crawl Frontier works and its different components.
- Frontier objects
- Understand the classes used to represent links and pages.
- Frontier API
- Learn how to use the frontier.
- See how to configure Crawl Frontier.
Extending Crawl Frontier¶
Built-in services and tools¶
- Using the Frontier with Scrapy
- Learn how to use Crawl Frontier with Scrapy.
- Using the Frontier with Requests
- Learn how to use Crawl Frontier with Requests.
- Graph Manager
- Define fake crawlings for websites to test your frontier.
- Testing a Frontier
- Test your frontier in an easy way.
- Recording a Scrapy crawl
- Create Scrapy crawl recordings and reproduce them later.
- Scrapy Seed Loaders
- Scrapy middlewares for seed loading