Crawl Frontier 0.2.0 documentation¶
This documentation contains everything you need to know about Crawl Frontier.
First steps¶
- Crawl Frontier at a glance
- Understand what Crawl Frontier is and how it can help you.
- Installation Guide
- Get Crawl Frontier installed on your computer.
Basic concepts¶
- What is a Crawl Frontier?
- Learn what a crawl frontier is and how to use it.
- Architecture overview
- See how Crawl Frontier works and its different components.
- Frontier objects
- Understand the classes used to represent links and pages.
- Frontier API
- Learn how to use the frontier.
- Settings
- See how to configure Crawl Frontier.
Extending Crawl Frontier¶
- Middlewares
- Filter or alter information for links and pages.
- Backends
- Define your own crawling logic.
Built-in services and tools¶
- Using the Frontier with Scrapy
- Learn how to use Crawl Frontier with Scrapy.
- Using the Frontier with Requests
- Learn how to use Crawl Frontier with Requests.
- Graph Manager
- Define fake crawlings for websites to test your frontier.
- Testing a Frontier
- Test your frontier in an easy way.
- Recording a Scrapy crawl
- Create Scrapy crawl recordings and reproduce them later.
- Scrapy Seed Loaders
- Scrapy middlewares for seed loading
All the rest¶
- Examples
- Some example projects and scripts using Crawl Frontier.
- Tests
- How to run and write Crawl Frontier tests.
- Release Notes
- See what has changed in recent Crawl Frontier versions.