Framework specification

Learn how to build a web-poet framework.

Design principles

Page objects should be flexible enough to be used with:

  • synchronous or asynchronous code, callback-based and async def / await based,

  • single-node and distributed systems,

  • different underlying HTTP implementations - or without HTTP support at all, etc.

Minimum requirements

A web-poet framework must support building a page object given a page object class.

It must be able to build input objects for a page object based on type hints on the page object class, i.e. dependency injection, and additional input data required by those input objects, such as a target URL or a dictionary of page parameters.

You can implement dependency injection with the andi library, which handles signature inspection, Optional and Union annotations, as well as indirect dependencies. For practical examples, see the source code of scrapy-poet and of the web_poet.example module.

Additional features

To provide a better experience to your users, consider extending your web-poet framework further to: