Before we get into the practical implementation,
let's get introduced to Beautiful Soup.
What is Beautiful Soup?
Straight from its documentation,
Beautiful Soup is a Python library for pulling data out of a HTML and XML files.
Beautiful Soup helps you pull particular content from a web page,
remove the HTML markup, and save the information.
It is a tool for web scraping that helps you clean up and parse
the documents you have pulled down from the web.
Why use Beautiful Soup at all for this though?
There are at least three good reasons for this.
It is simple and Pythonic.
Beautiful Soup provides a few simple methods and Pythonic idioms for navigating,
searching, and modifying a parse tree,
which is a toolkit for dissecting a document and extracting what you need.
It doesn't take too much code to write an application.
Beautiful Soup automatically converts incoming documents to
Unicode and outgoing documents to UTF-8.
You don't have to think about encodings,
unless the document doesn't specify an encoding and
Beautiful Soup can detect one.
Then, you just have to specify the original encoding.
Beautiful Soup sits on top of popular Python parsers, like LXML and HTML5 Lib.
This allows us to try our different parsing
strategies or trade speed for flexibility.
Parsing HTML means going through the HTML code and
extracting relevant information.
Let's look at the type of parsers usually used with Beautiful Soup.
There is Python's built in HTML parser, then there is LXML's HTML parser,
and also HTML5 Lib.
All of them have their pros and cons.
If I could recommend one, then I'd recommend the LXML.
It is very quick and works rather well.
Beautiful Soup parses the given HTML document into a tree of Python objects.
There are four main Python objects that you need to know about.
Let's start with tag.
The Tag object refers to an actual XML or HTML tag in the document.
You can access the name of a tag using tag.name.
You can set a tag's name to something else.
You can also access different attributes like class and id for tag.
The text within a tag is stored as a NavigableString in Beautiful Soup.
Beautiful Soup also allows you to access the comments in a web page.
These comments are stored as a Comment object,
which is also basically a NavigableString.
The BeautifulSoup object is used to represent the document as a whole.
Since it is not an actual object, it does not have any name or attributes.