A diagram showing architecture of running modes:
|Single process||single process running the crawler|
|Distributed||spiders, strategy worker (s) and db worker(s).|
Frontera is instantiated in the same process as fetcher (for example in Scrapy). Read more on how to use that mode here.
This mode is suitable for developing the crawling strategy locally and applications where its critical to fetch small number of documents fast.
BACKENDin spider processes set to
- In DB and SW workers
BACKENDshould point to
DistributedBackendsubclasses. And selected backend have to be configured.
- Every spider process should have it’s own
SPIDER_PARTITION_ID, starting from 0 to
SPIDER_FEED_PARTITIONS. Last must be accessible also to all DB worker instances.
- Every SW worker process should have it’s own
SCORING_PARTITION_ID, starting from 0 to
SPIDER_LOG_PARTITIONS. Last must be accessible to all SW worker instances.
- Both spiders and workers should have it’s
MESSAGE_BUSsetting set to the message bus class of your choice and selected message bus have to be configured.
Only Kafka message bus can be used in this mode out of the box.
This mode is designed for crawling of web-scale large amount of domains and pages.