This section documents the Frontera core API, and is intended for developers of middlewares and backends.
Frontera API / Manager¶
The main entry point to Frontera API is the
object, passed to middlewares and backend through the from_manager class method. This object provides access to all
Frontera core components, and is the only way for middlewares and backend to access them and hook their
functionality into Frontera.
FrontierManager is responsible for loading the installed
middlewares and backend, as well as for managing the data flow around the whole frontier.
Loading from settings¶
FrontierManager can be initialized using parameters the most
common way of doing this is using Frontera Settings.
This can be done through the
class method, using either a string path:
>>> from frontera import FrontierManager >>> frontier = FrontierManager.from_settings('my_project.frontier.settings')
BaseSettings object instance:
>>> from frontera import FrontierManager, Settings >>> settings = Settings() >>> settings.MAX_PAGES = 0 >>> frontier = FrontierManager.from_settings(settings)
It can also be initialized without parameters, in this case the frontier will use the default settings:
>>> from frontera import FrontierManager, Settings >>> frontier = FrontierManager.from_settings()
Starting/Stopping the frontier¶
Sometimes, frontier components need to perform initialization and finalization operations. The frontier mechanism to
notify the different components of the frontier start and stop is done by the
stop() methods respectively.
auto_start frontier value is activated,
this means that components will be notified once the
FrontierManager object is created.
If you need to have more fine control of when different components are initialized, deactivate
auto_start and manually call frontier API
stop() method is not automatically called
auto_start is active (because frontier is
not aware of the crawling state). If you need to notify components of frontier end you should call the method
Once frontier is running, the usual process is the one described in the data flow section.
Crawler asks the frontier for next pages using the
Each time the frontier returns a non empty list of pages (data available), is what we call a frontier iteration.
Current frontier iteration can be accessed using the
Finishing the frontier¶
Crawl can be finished either by the Crawler or by the Frontera. Frontera will finish when a maximum number
of pages is returned. This limit is controlled by the
max_requests has a value of 0 (default value)
the frontier will continue indefinitely.
Once the frontier is finished, no more pages will be returned by the
get_next_requests method and
finished attribute will be True.
In some cases while testing, frontier components need to act in a different way than they usually do (for instance
domain middleware accepts non valid URLs like
'B1' when parsing
domain urls in test mode).
Components can know if the frontier is in test mode via the boolean
Another ways of using the frontier¶
Communication with the frontier can also be done through other mechanisms such as an HTTP API or a queue system. These functionalities are not available for the time being, but hopefully will be included in future versions.