We've already studied a number of different filtering and searching
operations that you can perform on the HTML parse tree.
In this demo,
we'll take it a little further and see a few advanced search operations.
On this brand-new Jupyter notebook,
we'll work with an HTML file that we are familiar with,
TomJerry_WithImages.html.
We'll create and instantiate a soup object using the lxml parser.
If you want to refresh your memory,
take a look at what the HTML structure of this file looks like.
You can see that the title is The Story of Tom and Jerry.
It contains images and links to other pages.
We've already worked extensively with the find function on your
soup object and all of the element tags as well.
Find finds the first instance of any matching tag.
Soup.find ('img') will find the first image tag that
matches within your HTML content.
In order to find all elements that match within a page,
you'll use the find_all function.
Now all of this is very familiar to us.
You can perform a find_all on your soup object with this syntax as well.
Here I've specified the parentheses and a tag within parentheses.
This will perform a find_all operation for the a tag.
Let's say you have hundreds of a tags within your HTML page,
but you're interested in only the first three.
You can specify limit as an input argument.
Limit 3 will give me the first three a tags that match.
By default Beautiful Soup searches for content within
your HTML page in a recursive manner.
If you want to perform a non-recursive search,
you can simply say recursive False.
A non-recursive search for the a tag returns nothing
because there are no anchor tags which are direct
children to the top-level soup object.
The a tags in our HTML page are nested within the body element,
which is why you need to specify recursive True to
access the a tags on our page.
Our top-level soup object contains no direct children that are p tags.
So a search for p tags with recursive False will return nothing.
However, we do have paragraph tags that are direct children to the body element.
So soup.body .find_all (p) with with recursive False will return those
paragraph elements that are direct children to the body.
The limit input argument as we've seen allows you to
limit the results that are returned.
This find_all will find the first two html tags that
have a value for the HREF property.
These are the two a elements.
The find,
as well as find_all functions are available for all
navigable objects within your HTML tree.
Here I do a soup.body .findAll ('p') elements.
I just want 1 limit 1, but search for this in a non-recursive manner,
which means search just the direct children.
This gives us the first paragraph element that is the direct child of body.
We've seen that you can use find and find_all to search
for elements by attributes as well.
Now this search extends to custom attributes that you
might specify in your HTML as well.
Here one of my html tags has a custom attribute called name.
The name attribute, however,
can't be used directly within your find_all function in this way.
You'll get a TypeError.
This is because the name of the first input argument is also name.
So you can't specify the name attribute in this manner.
If you want to search within your parse tree for the custom name attribute,
you need to specify it using the attributes
dictionary as you see here on screen.
This finds all image tags with the name attribute set to William_Hanna,
and there's exactly one image tag here within our web page.
Let's say you have a custom attribute for an HTML element in your web page,
which has the dash.
Here is a custom attribute extra-info with a dash or a hyphen in between.
And this cannot be specified directly as a keyword within find_all.
So if you have an attribute express like this,
you'll need to use the attributes dictionary once again in order
to perform your search and filter operation.
This finds all image tags with the custom attribute,
extra-info equal to Tom_Cat.
And there's exactly one image tag in our HTML which matches.
Beautiful Soup also allows you to perform searches which look like this,
attrs(character).
It'll search for all values of character within the class attribute.
Within your attributes dictionary if you don't specify a key,
only specify a set of values,
that's assumed to apply to the class attribute on your HTML element.
Let's use this filter operation in a find_all function once again.
Find_all class attributes which have a value which start with ani.
This will automatically apply to the class attributes on your HTML elements.
Here is a paragraph tag which has the class value equal to animated,
which is a match.
You can search and filter your parse tree using the
contents of your html tags as well.
This find_all function finds all navigable strings,
within your parse tree which contain the term Tom.
Exactly one is found.
Let's try this once again.
Find_all navigable strings which contain the term Jerry.
Exactly one is found once again.
If you want to find tags that contain these strings,
you can modify your search term a little bit.
Find_all a tags which contain the string Tom.
There is exactly one tag that matches.
This a tag contains within it the string Tom.
Let's try this once again.
Find_all a tags within your HTML which contain the string contents Jerry.
You can search within your parse tree using the
string contents of your HTML as well.
Let's tweak the search a little more.
Find_all b tags or p tags within your HTML which contain the string cat.
There's exactly one match that is a b tag which contains the word cat.
You can filter by the string content within an html
tag with custom functions as well.
Let's find all string content, which is the only string within the outer tag.
This custom function returns true if the string s is equal to s.parent .string.
It's the only string child of the parent.
Let's apply this custom function to filter by the string property in our HTML.
String is equal to the only string within a tag.
And here are all of the strings,
which are the only child of the outer parent tag.