Soupstars 🍲 ⭐ 💥

Soupstars makes it easier than ever to build web parsers in Python.

Install it with pip.

pip install soupstars

Let’s go!

Quickstart

You need two objects to get started.

>>> from soupstars import Parser, serialize

We’ll build a parser to extract data from a github page.

>>> class GithubParser(Parser):
...    "Parse data from a github page"
...
...    @serialize
...    def title(self):
...        return str(self.h1.text.strip())

Now all we need is a github web page to parse.

>>> parser = GithubParser("https://github.com/tjwaterman99/soupstars")

Let’s see what we’ve got!

>>> parser.to_dict()
{'title': 'tjwaterman99/soupstars'}

You’re now ready to start building your own web parsers with soupstars. Nice job. 🍻

Going further

We recommend starting with the examples.

Examples

Soupstars includes various examples that you can reference or use directly.

NYTimes

Extract article links and article metadata from nytimes.com

class soupstars.examples.nytimes.NytimesArticleParser(url)

Parse attributes from a NY times article.

>>> from soupstars.examples.nytimes import NytimesArticleParser

title()

The title of the article.

author()

The author(s) of the article.

class soupstars.examples.nytimes.NytimesLinkParser(url)

Parse the links from a NY times webpage.

Parameters:url (str) – The webpage to parse

>>> from soupstars.examples.nytimes import NytimesLinkParser

article_links()

A list of links that point to NYTimes articles

non_article_links()

A list of links that points to NYTimes pages that are not articles.

Economist

Extract metadata from economist index and article pages

class soupstars.examples.economist.WeeklyIndexPages(**kwargs)

Example model for storing the results of the parser

class soupstars.examples.economist.WeeklyIndexPageParser(url)

Parse metadata from the weekly updated index pages

Model

alias of WeeklyIndexPages

base_url()

The url used

article_date()

The date of the article

status_code()

Status code of the request

num_articles()

The number of articles foudn on the page

For more specific questions about using the library, refer to the api documentation.

API

The primary Parser class and serialize decorator are available from the models and serializers modules.

>>> from soupstars.models import Parser
>>> from soupstars.serializers import serialize

Those objects are also on the top-level api.

>>> from soupstars import Parser, serialize

Models

The primary model provided by soupstars is the Parser class. It should generally be subclassed when building your own parsers.

When you initialize a parser with a url, it automatically downloads the webpage at that url and stores both the request and response as attributes.

>>> from soupstars import Parser, serialize
>>> class MyParser(Parser):
...     @serialize
...     def item(self):
...         return 'An item!'
>>> parser = MyParser('https://jsonplaceholder.typicode.com/todos/1')
>>> print(parser.response)
<Response [200]>
>>> print(parser.request)
<PreparedRequest [GET]>

class soupstars.models.Parser(url)

Primary class for building parsers.

Parameters:url (str) – The url to parse

serializer_names()

Returns a list of the names of the functions to be serialized.

serializer_functions()

Returns a list of the functions to be serialized.

to_tuples()

Returns a list of (name, value) tuples of each function to be serialized.

to_dict()

Convert the parser to a dictionary, with keys the names of each serializer and values the value of each serializer

to_json()

Convert the parser to a JSON object

Serializers

Serializers help convert parsers into storable objects. The functions defined in this module are used to instruct soupstars about how to perform the serialization.

soupstars.serializers.serialize(function)

Decorating a function defined on a parser with serialize instructs soupstars to include that function’s return value when building its own serialization.

>>> from soupstars import Parser, serialize
>>> class MyParser(Parser):
...     @serialize
...     def length(self):
...         return len(self.response.content)
...
>>> parser = MyParser('https://jsonplaceholder.typicode.com/todos/1')
>>> parser.serializer_names()
['length']
>>> 'length' in parser.to_dict()
True

Mixins

Soupstars offers mixins for saving Parser objects to databases.

SQLAlchemy Mixins

Mixins for saving parsers via SQLAlchemy

class soupstars.mixins.sqlalchemy_mixins.SqlalchemyMixin

Use as a mixin on a Parser to save its data to a SQLAlchemy model

load_session()

Loads a SQLAlchemy session object.

save()

Saves the parser to the SQLAlchemy Model defined on the class