Beautiful Soup can work with not just HTML documents but also XML
documents provided you use the right kind of parser.
In this demo, we'll see how you can read in an HTML document as XML.
We'll also see how you can use XPath to select XML
elements directly working with lxml.
If you want to read in an XML document or read in your
HTML document in the XML format, simply specify XML as your parser.
This will use the lxml library under the hood and read
in your HTML document as an XML.
So even if you're working with an XML document,
once you have a soup object,
you can access the elements within your XML document
in exactly the same way as before.
In order to understand how you can use XPath with your XML documents,
let's work directly with the XML parse tree that the lxml library provides us.
The etree module within my lxml library allows me to instantiate an HTML parser,
which I'll then use to parse my HTML document as an XML.
Now my HTML contents have been interpreted as XML,
and the tree object contains the root of my XML tree.
If you're parsing this tree element to the type function in Python,
you can see it's of type lxml.etree_ElementTree.
Now that we have an XML tree,
we can now use XPath selection to select elements in this tree.
The forward slash performed the selection from the
root node of your XML document.
This XPath expression finds the HTML node directly under the root node.
This finds a match because the HTML node is directly under the root node.
Let's try this, /head,
find the head element that is directly under the root node,
and this will find no match.
The head element is nested under other nodes.
It's not directly available under the root node.
Here is an XPath expression that navigates down our XML tree.
Start from the root node, go to HTML, then head under it,
and then title, and select that element.
That just selects the XML element.
If you're interested in the text content of this XML element,
simply specify the text followed by parentheses at the end.
This extracts the text contents of our title.
If you don't want to navigate down from the root node,
but you want to select nodes in the document from the
current node that matched the selection,
no matter where they are,
you'll start your XPath expression with the double forward slash.
The double forward slash with the wild card asterisk will select
all of the nodes that exist in our XML tree,
no matter how deep they are nested.
Here is an XPath expression that matches all of the paragraph nodes that
exist in our XML tree somewhere under the current node.
There are two paragraph elements that match, both have been returned.
Here is an XPath expression that matches any i tag
element somewhere in the document tree.
And for any element that matches if you want to extract
the text content of that element,
simply specify the text function at the very end.
Here is the text contained within the i node of our XML tree.
This XPath expression that you see here matches all of the image
nodes that exist within this XML parse tree.
XPath expressions allow you to filter your nodes
based on attribute values as well.
Here I want to find all nodes within my XML tree which have id link1.
There's exactly one node that matches.
That's what is returned.
Let's extract the text contents of this node by using
the text function at the very end.
And the text here is William Hanna.
Let's get some more practice with ID-based selection.
I want all nodes with id link3, extract the text content.
There's just one node with this ID.
It has the content Tom.
Attributes are always prefixed using the ampersand.
If you want to find all nodes with class character and
extract the text content of those nodes,
here is a regular expression for that.
There are two nodes that match this in our XML tree.
They contain the text Tom and Jerry.
Here is an XPath specification that finds and extracts the text of all
of the anchor tags that exist in our parse tree.
Here are four matches.
You can also access a particular node using an index.
This expression accesses the fourth node with a tag and
extracts the text contents of that node.
This is the Jerry node.
You can have XPath extract the value associated with
a particular attribute as well.
Specify the src attribute at the very end,
and this will extract the value of all source attributes in your image tags.
What if I want the value of the source attribute for the first image tag?
Simply specify the index value (1) for the image tag,
and that will extract the source for the first image.