Learn how to build a web-poet framework.
Page objects should be flexible enough to be used with:
synchronous or asynchronous code, callback-based and
async def / awaitbased,
single-node and distributed systems,
different underlying HTTP implementations - or without HTTP support at all, etc.
A web-poet framework must support building a page object given a page object class.
It must be able to build input objects for a page object based on type hints on the page object class, i.e. dependency injection, and additional input data required by those input objects, such as a target URL or a dictionary of page parameters.
You can implement dependency injection with the andi library, which handles
annotations, as well as indirect dependencies. For practical examples, see the
source code of scrapy-poet and of the
To provide a better experience to your users, consider extending your web-poet framework further to:
Support as many input classes from the
web_poet.page_inputsmodule as possible.
Allow users to request an output item directly, instead of requesting a page object just to call its
If you do, consider supporting both synchronous and asynchronous definitions of the
to_itemmethod, e.g. using
Support additional requests.