From the ground up

Learn why and how web-poet came to be as you transform a simple, rigid starting web scraping code snippet into maintainable, reusable web-poet code.

Writing reusable parsing code

Imagine you are writing code to scrape a book web page from books.toscrape.com, and you implement a scrape function like this:

import requests
from parsel import Selector


def scrape(url: str) -> dict:
    response = requests.get(url)
    selector = Selector(response.text)
    return {
        "url": response.url,
        "title": selector.css("h1").get(),
    }

item = scrape("http://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html")

This scrape function is simple, but it has a big issue: it only supports downloading the specified URL using the requests library. What if you want to use aiohttp, for concurrency support? What if you want to run scrape with a local snapshot of a URL response, to write an automated test for scrape that does not rely on a network connection?

The first step towards addressing this issue is to split your scrape function into 2 separate functions, download and parse:

import requests
from parsel import Selector


def parse(response: requests.Response) -> dict:
    selector = Selector(response.text)
    return {
        "url": response.url,
        "title": selector.css("h1").get(),
    }

def download(url: str) -> requests.Response:
    return requests.get(url)

url = "http://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html"
response = download(url)
item = parse(response)

Now that download and parse are separate functions, you can replace download with an alternative implementation that uses aiohttp, or that reads from local files.

There is still an issue, though: parse expects an instance of requests.Response. Any alternative implementation of download would need to create a response object of the same type, forcing a dependency on requests even if downloads are handled with a different library.

So you need to change the input of the parse function into something that will not tie you to a specific download library. One option is to create your own, download-independent Response class, to store the response data that any download function should be able to provide:

import requests
from dataclasses import dataclass
from parsel import Selector


@dataclass
class Response:
    url: str
    text: str


def parse(response: Response) -> dict:
    selector = Selector(response.text)
    return {
        "url": response.url,
        "title": selector.css("h1").get(),
    }


def download(url: str) -> Response:
    response = requests.get(url)
    return Response(url=response.url, text=response.text)


url = "http://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html"
response = download(url)
item = parse(response)

The parse function is no longer tied to any specific download library, and alternative versions of the download function can be implemented with other libraries.

Parsing with web-poet

web-poet asks you to organize your code in a very similar way. Let’s convert the parse function into a web-poet page object class:

import requests
from web_poet import Injectable, HttpResponse


class BookPage(Injectable):
    def __init__(self, response: HttpResponse):
        self.response = response

    def to_item(self) -> dict:
        return {
            "url": self.response.url,
            "title": self.response.css("h1").get(),
        }


def download(url: str) -> Response:
    response = requests.get(url)
    return HttpResponse(
        url=response.url,
        body=response.content,
        headers=response.headers,
    )


url = "http://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html"
response = download(url)
book_page = BookPage(response=response)
item = book_page.to_item()

Differences from a previous example:

• web-poet provides a standard HttpResponse class, with helper methods like css().

  Note how headers are passed when creating an HttpResponse instance. This is needed to properly decode the body (which is bytes) as text using web browser rules. It involves checking the Content-Encoding header, HTML meta tags, BOM markers in the body, etc.

• Instead of the parse function we’ve got a BookPage class, which inherits from the Injectable base class, receives response data in its __init__ method, and returns the extracted item in the to_item() method. to_item is a standard method name used by web-poet.

Receiving a response argument in __init__ is very common for page objects, so web-poet provides a shortcut for it: inherit from WebPage, which provides this __init__ method implementation. You can then refactor your BookPage class as follows:

from web_poet import WebPage

class BookPage(WebPage):
    def to_item(self) -> dict:
        return {
            "url": self.response.url,
            "title": self.response.css("h1").get(),
        }

WebPage even provides shortcuts for some response attributes and methods:

from web_poet import WebPage

class BookPage(WebPage):
    def to_item(self) -> dict:
        return {
            "url": self.url,
            "title": self.css("h1").get(),
        }

At this point you may be wondering why web-poet requires you to write a class with a to_item method rather than a function. The answer is flexibility.

For example, the use of a class instead of a function makes fields possible, which make parsing code easier to read:

from web_poet import WebPage, field


class BookPage(WebPage):
    @field
    def url(self):
        return self.url

    @field
    def title(self):
        return self.css("h1").get()

Using fields also makes it unnecessary to define to_item() manually, and allows reading individual fields when you don’t need the complete to_item() output.

Note

The BookPage.to_item() method is async in the example above. See Fields for more information.

Using classes also makes it easy, for example, to implement dependency injection, which is how web-poet builds inputs.

Downloading with web-poet

What about the implementation of the download function? How would you implement that in web-poet? Well, ideally, you wouldn’t.

To parse data from a web page using web-poet, you would only need to write the parsing part, e.g. the BookPage page object class above.

Then, you let a web-poet framework handle the download part for you. You pass that framework the URL of a web page to parse, and either a page object class (the BookPage class here) or an item class, and that’s it:

item = some_framework.get(url, BookPage)

web-poet does not provide any framework, beyond an example one featured in the tutorial and not intended for production. The role of web-poet is to define a specification on how to write parsing logic so that it can be reused with different frameworks.

Page object classes should be flexible enough to be used with very different frameworks, including:

• synchronous or asynchronous frameworks

• asynchronous frameworks based on callbacks or based on coroutines (async def / await syntax)

• single-node and distributed systems

• different underlying HTTP implementations, or even implementations with no HTTP support at all