In this demo,
we'll get more practice working with Beautiful Soup to
parse and extract content from web pages.
We'll also understand the basic elements that make up a parse tree,
tags, attributes, strings, and comments.
For this demo we'll parse and extract content from this very simple website.
Let's open up this website using webbrowser.open and see what it looks like.
You can see that the page is kind of interesting.
It has a logo, a header, Totally Normal Gifts,
a paragraph, and a table with the number of gift options.
As is our usual practice, before we scrape content from the web page,
I'm going to right-click and hit Inspect and view
the HTML structure of this page.
I'm going to expand this header tag here.
Within it CSS styles are specified within this style html tag.
You can go down and visualize the contents within
the body of this web page as well.
This is what is displayed within the browser.
This outer div happens to contain a nested inner div,
which contains the paragraphs under Totally Normal Gifts.
As you can see, nested within this div are multiple paragraph elements.
You can select each element in turn, expand the element, and view its contents.
Once you get a basic idea of what the web page looks like,
you can then use Beautiful Soup to parse the content of this web page.
Here is the table, which contains all of the gift items.
If you expand this table element, you'll find within it a tbody element.
This tbody HTML element is what contains the rows of the tables.
Each gift is assigned a specific row represented by the tr element,
and within each row, we have td elements for the cells of the table.
You can select individual cells and view contents within it.
Some of these cells have text, other cells have image tags within them.
And at the very bottom of this page,
you have a footer element which contains the
copyright information for this web page.
This is just a fun, fake web page to help you learn scraping.
Let's switch back to our Jupyter notebook here,
and I'm going to use the request API to make a get-request to this URL
and extract the text information that was downloaded.
In order to use Beautiful Soup,
let's construct a soup object with the contents of this HTML,
and we'll use the lxml parcel.
Lxml is often the parcel of choice because it's lightweight and
very lenient with the HTML that it parses.
You can invoke the soup.prettify function in order to
view a prettified version of your HTML.
Now this particular web page that we downloaded and parsed has a style tag
which contains the CSS styles that apply to this web page.
You can access the contents of a tag directly using the dot notation.
Soup.style will give you contents of the style tag.
If you have multiple HTML elements with the same tag,
when you use the dot notation,
the first HTML element that matches will be returned.
Soup.tr will return the first row in our table,
which happens to be the header row.
Similarly,
soup.image will return an element which happens to be the
first image within your soup object, which is the logo.
The soup object also supports a variety of functions
allowing you to look for specific elements.
The find function will allow you to specify the tag that you're searching for.
Find will return just the first instance of that HTML element,
just like the dot notation.
If you have multiple HTML elements within your web page with the same tag,
you need to use the find_all function.
Find_all('tr') will return all row elements within your page.
These row elements have been returned in the form of a list,
and you can look up individual rows using the row index.
At row(0) we have the header row.
At row(1) we have the row which has the first gift item.
You'll find that when you use Beautiful Soup,
the find_all function is what you'll use the most often.
Let's try this once again.
Let's find all of the elements which make up the table cells, find_all('td').
This will return a Python list of all of the table cells that match.
There are a total of 20 on this web page.
Once you have this information in the form of an iterable object,
you can run a for loop to access each table cell individually.
When you're searching by html tag, you get the tag itself,
as well as the contents of the tag.
If you're not interested in the entire HTML element,
you only want the text content within HTML elements,
you need to use the .text property for each html tag.
This for loop here will print out all of the text within
every table cell on this web page.
Using the find_all function,
you can be very explicit about the elements that you want to extract.
Iterate over all rows within this web page and all cells within each row.
That's what this for loop does.
For each cell we extract the text content,
and we strip out the extraneous white spaces in the
text and print it out to screen.