If you've got experience developing for the web,
HTML and CSS selectors may be second nature.
There's another choice though that comes from the XML world, XPath.
HTML is an XML‑like format; therefore,
we can use XML tools like XPath to extract data.
Think of an HTML document like a file structure or a tree.
There's HTML at the top then the HTML has two children, Head and Body. Head
has a child of Title, and inside of Title, there's text that says Car Buying
Web Page. In the same way, Body has two children,
H1 and ul. H1 has text inside that says, Used Tesla Cars in
Your Area, and ul has three li tags inside.
Of course, there are even more child elements inside each li.
Remember when we used li as a CSS selector to get all the li elements?
The XPath selector is similar.
It's //li.
Double slash means look anywhere. Then li means look for li tags.
So this XPath code will look anywhere in the HTML for li tags and return
all of them in a list. Combined selectors work too.
With CSS,
we could say, find a ul with class of listings then look
inside for an li with id of vin3827.
The XPath syntax is a little more verbose, but it does the same thing.
Remember,
double slash means, look anywhere for a ul with class of
Listings. Then the single slash before li means, look only inside
that ul for an li with the id of vin3827.
XPath and CSS selectors are both great,
but which is better? In general, use whichever works best or
whatever you're most comfortable with.
For web developers, CSS selectors are often a good starting point.
Plus, that's what Beautiful Soup supports. XPath, though,
can do anything CSS selectors can do and more, so they're worth learning too.
It's best to learn by doing on your web scraping projects. Try different things
and know that both XPath and CSS selectors are very useful.
Speaking of learning by doing,
let's see how to use Chrome developer tools to extract data from real web pages.