When you're scraping web pages in the real world,
you would typically use a library such as requests to
download the HTML contents of the page and then use
Beautiful Soup to parse those HTML contents.
For example, here I make a GET request to https.google .com,
and let's say this is the page that I want to parse.
Here is the response that I receive,
it was a successful response with a status of 200.
The HTML contents of this response will be present in the
text property of the response object.
I can now pass this into the BeautifulSoup object,
parse it using, say, the lxml parser, and I get a soup object.
And once the soup object is available to me I can
programmatically extract elements from this HTML page.
Now that you understand this big picture flow,
let's work with an HTML page that I have on my local machine
here which has a more predictable structure.
I have an HTML page here called GiantPanda.html that is present under
the files folder in my current working directory,
and I've opened it up using the Chrome browser.
You can simply right-click on any HTML file and say open in Chrome,
and it'll open up within a browser window.
Here is what this file looks like.
You can see that we have a few header elements,
some elements in different colors, we have an image,
and also a link.
If you want to see the exact HTML structure of this file,
you can open up GiantPanda.html in any text editor,
such as Sublime Text.
It's a fairly simple web page,
and while we are learning how to parse this web page,
it's often useful to work with a web page that is under your
control so that it doesn't change under you.
I'm going to read in the contents of this local file
using the Python file open command, open up GiantPanda.html in the read mode,
and pass in the file handle into a BeautifulSoup object.
I'll parse this using the lxml parser.
Once we have a soup object instantiated you can view the HTML contents
within your Jupyter Notebook by calling soup.prettify.
Here are the HTML contents of this page,
and we'll see how we can extract tags and elements from this HTML.
Soup.title will give us access to the title tag of this web page,
and the title tag is what is displayed in the tab of your web browser.
Our title simply says Giant Panda.
Every HTML element present within your web page is
available as a tag within Beautiful Soup.
The type of this element is bs4.element .Tag, as you see here on screen.
We have a header, h2, within our page.
The h2 property will give you access to the first header.
If you extract and view the soup.h2 tag,
you can see our header 2 is this giant panda text in blue.
Our page also contains a header 1 element,
and this can be accessed by invoking the soup.h1 property.
This is the header that you see right on top of the screen.
Every tag extracted from your soup has a name,
and the name is the name of the HTML element.
The name of this tag is h1.
Beautiful Soup is extremely powerful,
you can update the name of your tag programmatically,
I've set this to a user_defined tag right now.
This updated name then becomes part of the soup object,
and you can write the soup out to a new HTML or an XML file if you want to.
In an HTML page,
the a tag is what you use to set up reference links to other pages.
Soup.a will access the first link available within your HTML page.
This is the link to the Wikipedia article.
The HTML elements in a web page may have attributes
associated with that element.
For example,
the href element in the a tag gives you the link that the a tag points to.
You can access specific attributes from a tag specifying square brackets,
as you see here on screen.
If you want to view all of the attributes associated
with a particular HTML element,
the .attrs property on a tag will show you all attributes
of that tag in the form of a dictionary.
If you have img elements within your HTML page,
you can access the first of the img elements using soup.img.
This will automatically extract the very first img tag.
This is the image of our panda.
Observe that this img tag has two attributes,
the alt attribute, as well as the src attribute,
and if you look at the attributes property of this tag,
you will see that both of the attributes are available within this dictionary.
The soup object makes it very easy for you to access HTML elements by tag.
Soup.b will give you the first bolded element within your HTML page,
so if you have multiple bolded elements,
only the first one will be returned.
Observe that this bolded element has a class attribute specified,
class=panda, and if you look up the class attribute using square brackets,
you'll get the value associated with the class attribute.
We have multiple paragraphs, or p tags, within our HTML document.
Soup.p will give us access to the first paragraph tag.
Observe that this paragraph tag also has a class attribute,
and we've applied multiple CSS classes to this HTML element,
panda as well as highlight.
If you look up the value for the class attribute for this HTML tag,
you'll get a list of classes associated with this attribute.
You can look up attributes for a tag using square brackets.
You can also use the get_attribute_list function.
Specify the name of the attribute that you want to look up,
and you'll get the result.
A tag in Beautiful Soup is essentially an HTML element within your document.
If you want the contents of that element,
you can access the tag.string property.
Observe that tag.string is of type NavigableString.
This NavigableString refers to the text contents within any HTML element.
Tag.string gives us the text within our paragraph tag.
It's possible for your HTML page to include comments as well.
You can access comments using Beautiful Soup.
Within our HTML page for giant pandas, we have a comment within an i tag,
and this comment is accessible via our soup object as well.