Now regular expressions as you have seen are
extremely powerful and quite amazing.
But they don't really understand the HTML structure of a web page.
Now if you could have something which understands the HTML structure and
allows you to parse the web page in the form of a parse tree,
and also allows you to use regular expressions,
that would be simply awesome.
Well, that's exactly what Beautiful Soup does.
Scraping web pages is hard and messy.
And Beautiful Soup tries to make things easy for you.
It's a Python library for pulling data out of HTML and XML files.
Makes it easy to navigate your HTML structure in the form of a parse tree and
use powerful constructs such as regular expressions as well.
Beautiful Soup 4 is the latest version at the time of this recording.
You can get it with pip install.
Beautiful Soup uses different parsers to parse and
construct the parse tree from HTML data.
Lxml is one such parser, go ahead and install lxml as well.
Another parser that Beautiful Soup can use is the html5lib parser,
which parses your web page in the HTML5 format as a browser would view it.
We'll work with these parsers in detail later on in the course.
But for now, just get them set up.
The next step is to set up the inputs for this demo.
Beautiful Soup is imported as from bs4 import BeautifulSoup.
This is what will give you the soup object to parse our HTML tree.
As we get introduced to Beautiful Soup,
let's work with some HTML content already present in the form of a string.
This is the content that could make up the structure of a simple web page.
Observe that within the html tag, we have a header tag with a title.
Under the body element of this page,
we have three paragraphs which give us The Dormouse's story.
This happens to be the sample HTML from Beautiful Soup's documentation.
In order to have Beautiful Soup set up a parse tree from this HTML string,
you need to instantiate a soup object.
Instantiate the BeautifulSoup class and pass in your HTML.
Beautiful Soup will construct this parse tree using the default parser,
which, if lxml is installed, happens to be the lxml parser.
And here are the contents of our soup object.
The soup object can be thought of as the root of your
document and is the root of your parse tree.
The type of the soup object is BeautifulSoup.
It's not just a string.
The HTML that you extracted from a real web page may not be clean,
may not be well-indented.
If you want to view it in a prettified format, you can use soup.prettify.
Soup.prettify makes it easy for you to examine the content of your
web page by structuring it and indenting it nicely.
The soup object understands the HTML elements that
are present within the parse tree,
and you can use the dot notation to access the contents of an element directly.
Soup.html will give you the contents of the html tag within your page.
In fact,
an element may not be directly present or accessible from the root of the tree.
You can access other html tags directly as well using the dot notation.
Soup.head will give you the contents of the header tag within your HTML.
If you want to directly access the contents of the title tag,
simply call the title property on your soup.
Soup.title will give you the title directly.
Accessing the tag directly gives you the tag,
as well as the contents within the tag.
Let's say you want to access the string contents of a tag,
you can call soup.title .string.
And here, in an extremely clean and structured manner,
you've extracted the string contents of the title tag.
Now the .string property only works if your html
tag contains exactly one string.
It can't contain other nested elements.
We'll talk more about that in a bit.
If you want to access the name of the HTML element within your parse tree,
you can use the .name property.
The name of the title HTML element is, well, title.
Now all of the HTML content that is displayed within a
browser is present in the body tag.
So if you access soup.body, this will give you an entire nested HTML.
There are obviously HTML elements within your page
which are present more than once.
For example, the paragraph tab.
So if you see soup.body .p, what paragraph do you get?
Well, you'll get the first paragraph element within your body tag.
When you access tags using the dot notation,
that gives you access to the first element.
What if you say soup.p directly?
This is, of course, allowed.
It'll look within the entire soup object to find the first paragraph tag,
and that's what will be returned.
Let's say you have a bold tag within your HTML,
soup.b will access the first bold tag that is present,
and its contents will be returned.
The HTML that we are working with has multiple anchor links.
In fact, it has three.
Soup.a will give you access to the first a tag or the
first anchor link within our page.
Every element within your Beautiful Soup parse tree has a parent property.
Soup.a .parent will give you the parent of the a tag.
Parent accesses the immediate parent within the HTML tree structure,
not other ancestors.
Every navigable element in the parse tree has a parent property,
so if you do soup.a .parent .parent, it will find the parent of the parent,
the parent of the p is the body tag, and that's what you get here.
This is how Beautiful Soup understands the structure of a web
page and sets it up so you can navigate to tags directly and
then navigate up and down the tree.
Beautiful Soup doesn't just work with web pages hosted at URLs.
You can also work with HTML files present on your local machine.
Here is a simple HTML file,
which I've set up with some content about the Tom and Jerry cartoon.
This page has a header, title, body, a number of links.
It also has a number of different images,
and it has a bunch of paragraphs containing text
about the Tom and Jerry cartoon.
This HTML file is present within my files directly,
under my current working directory.
And I open it up on Chrome browser, and this is what it looks like.
A quick glance shows you that there are five images
and four links within this page.
Let's go back to our Jupyter notebook.
I'm going to read in the contents of this HTML file
using the Python open command,
and I'll instantiate a soup object with the files contents,
and I'm going to parse this HTML file using the lxml parser.
I've specified the parser explicitly here.
And, as you remember,
soup.prettify will allow you to view a prettified version of your
HTML with all of the HTML elements neatly indented.
You can access html tags directly using the dot notation on your soup object.
Soup.head will allow you to access the header tag present within your HTML.
If you have multiple HTML elements with the same tag,
when you use the dot notation with your soup or any nested element,
you'll get the first tag.
Soup.image gives us access to the first image element on our HTML page.
We happen to have exactly one div within this HTML.
Soup.div will give us access to the contents of that div.
This is the div which contains all of the images.
If you have an italic element within your page,
soup.i will give you access to that element directly.
If an element does not exist, that particular dot property will return nothing.