Examples

The project repo includes an examples folder with some scripts and projects using CrawlFrontier:

examples/
    requests/
    scrapy_frontier/
    scrapy_recording/
    scripts/
  • requests: Example script with Requests library.
  • scrapy_frontier: Scrapy Frontier example project.
  • scrapy_recording: Scrapy Recording example project.
  • scripts: Some simple scripts.

Note

This examples may need to install additional libraries in order to work.

You can install them using pip:

pip install -r requirements/examples.txt

requests

A simple script that follow all the links from a site using Requests library.

How to run it:

python links_follower.py

scrapy_frontier

A simple script with a spider that follows all the links for the sites defined in a seeds.txt file.

How to run it:

scrapy crawl example

scrapy_recording

A simple script with a spider that follows all the links for a site, recording crawling results.

How to run it:

scrapy crawl recorder

scripts

Some sample scripts on how to use different frontier components.