Crawl Frontier 0.2.0 documentation

This documentation contains everything you need to know about Crawl Frontier.

First steps

Crawl Frontier at a glance
Understand what Crawl Frontier is and how it can help you.
Installation Guide
Get Crawl Frontier installed on your computer.

Basic concepts

What is a Crawl Frontier?
Learn what a crawl frontier is and how to use it.
Architecture overview
See how Crawl Frontier works and its different components.
Frontier objects
Understand the classes used to represent links and pages.
Frontier API
Learn how to use the frontier.
See how to configure Crawl Frontier.

Extending Crawl Frontier

Filter or alter information for links and pages.
Define your own crawling logic.

Built-in services and tools

Using the Frontier with Scrapy
Learn how to use Crawl Frontier with Scrapy.
Using the Frontier with Requests
Learn how to use Crawl Frontier with Requests.
Graph Manager
Define fake crawlings for websites to test your frontier.
Testing a Frontier
Test your frontier in an easy way.
Recording a Scrapy crawl
Create Scrapy crawl recordings and reproduce them later.
Scrapy Seed Loaders
Scrapy middlewares for seed loading

All the rest

Some example projects and scripts using Crawl Frontier.
How to run and write Crawl Frontier tests.
Release Notes
See what has changed in recent Crawl Frontier versions.