You have a page that you want to scrape.
You made a request to that URL, and you've downloaded the HTML contents.
Now what?
Well, that's exactly what we'll see in this demo,
how we can use Beautiful Soup to parse HTML.
Beautiful Soup is an interestingly named Python
library to extract data from XML, as well as HTML files.
This creates a parse tree for your HTML data and gives you
intuitive APIs to navigate this parse tree.
The current version of Beautiful Soup at the time of
this recording is Beautiful Soup 4.
Get it with a simple pip install.
Beautiful Soup allows you to specify different parsers
to parse your XML and HTML data.
I'm going to pip install these parsers as well so I
can use them with Beautiful Soup.
I'll start with lxml.
I'll also install the html5lib parser.
We'll talk about the nuances of the different parsers
in just a bit when we use them.
Import the Beautiful Soup library and let's get started.
We'll start off with some very simple HTML.
Here is a simple HTML file with a top-level HTML tag.
Nested within the HTML tag,
we have a header tag and the title tag nested within this head.
The head tag usually contains the page title and the metadata.
The body tag contains the actual contents that are displayed within the browser.
Observe that this particular HTML page has three
paragraphs of text along with some links.
In order to use Beautiful Soup to parse this HTML data,
you need to pass in this HTML string and create a BeautifulSoup object.
The soup object is the root of the navigable parse tree that is created.
Soup.prettify will pretty print the HTML contents of the soup.
The contents of the specific HTML tags can be
accessed via the properties of the soup.
Soup.title will give you the contents of the title tag.
Similarly,
soup.body will allow you to access the contents of the body tag of your HTML.
If you want the tag name of a particular HTML element,
you need to invoke the name property on that tag.
Soup.body .name will give you body.
When Beautiful Soup creates this parse tree, every element knows its parent.
Soup.body .parent .name is html, the HTML tag is the parent of the body tag.
There's just one title and one body tag in our HTML page.
What if we access a tag of which we have multiple elements?
For example,
soup.p gives us the contents of just the first p tag in our HTML string.
When you have multiple elements with the same tag,
accessing that tag using the soup object returns just the first element.
In exactly the same way, if you access soup.a,
this will return the first a tag in your HTML string.
If you've ever worked with HTML in the real world,
you know that HTML pages are often not well-formatted.
For example, in this HTML here,
we have an h1 tag which has no corresponding closing tag,
the th tag is not well-defined at all,
it doesn't have the corresponding closing brackets,
and so on.
The cool thing when you use Beautiful Soup is that you can have different
parsers that work well even with ill-formatted HTML.
I'll now create a soup object using this ill-formatted HTML.
When I don't specify a parser,
the default parser that is used is the lxml parser.
This parser is very performant, it works extremely fast.
It's lenient, it can work with ill-formatted HTML as well.
However, it has an external dependency on a C library.
Here is what the parsed HTML looks like with this parser.
Observe that it is now well-formatted.
The parser has done that for us.
When you create the soup object,
you can explicitly specify what parser you want to use.
Here I want to explicitly use the lxml parser, and here is my parsed HTML.
You can construct your soup object using Python's default HTML parser as well.
This has good speed, it's a fairly lenient parser,
but it's not as fast as lxml,
and this parser is not as lenient as the html5lib parser.
Let's take a look at the resulting HTML when we use this html.parser.
The resulting HTML is not as clean as the HTML that was
generated when we use the lxml parser.
If you want your soup objects to have a valid HTML5 format,
the html5lib parser is the one that you should use.
It's extremely lenient, it parses pages as a web browser does.
The resulting HTML will be valid HTML5.
However, this is fairly slow and has an external dependency on a Python library.
And here is valid HTML5 generated using our html5lib parser.
Beautiful Soup can be used to parse XML trees as well.
Simply specify XML as your parse format and observe that your
resulting HTML page is now rendered as XML.
If you want to use the XML parser in the lxml library,
this is how you'd specify your soup object.