Supporting RetriesΒΆ
Web-poet frameworks must catch Retry
exceptions raised from the to_item()
method of a
page object.
When Retry
is caught:
The original request whose response was fed into the page object must be retried.
A new page object must be created, of the same type as the original page object, and with the same input, except for the response data, which must be the new response.
The to_item()
method of the new page object may
raise Retry
again. Web-poet frameworks must
allow multiple retries of page objects, repeating the
Retry
-capturing logic.
However, web-poet frameworks are also encouraged to limit the amount of retries per page object. When retries are exceeded for a given page object, the page object output is ignored. At the moment, web-poet does not enforce any specific maximum number of retries on web-poet frameworks.