In this video,
we will examine using CSS and XPath selectors to extract links from
web pages. The browser web page is generally constructed using a
series of hierarchical tags combining HTML,
CSS styling, and JavaScript.
The majority of languages and web scraping frameworks have
methods for extracting data from web pages that are based
on what we call selectors.
CSS selectors are one of the primary methods used for extracting data.
CSS selectors are implemented in all of the core
languages either directly or using a library.
When we construct a web page, we do so using tags.
So, for example, we use p to specify a paragraph or h1 for a heading.
Then we decorate our tags using attributes like ID and class.
CSS selectors allow us to query and select elements from our
web page by matching any elements that fit a particular search
pattern that we construct.
There are quite a number of different options you can use for CSS selectors.
On a high level, we can find elements that contain a
certain type of class or a specific ID.
Our selector is not limited to one query.
We can actually chain multiple queries together and
make quite powerful selectors, but you have to be careful of the nuances.
For example,
.classA.classB selects elements that use both class A and class B; whereas,
.classA and classB only selects elements that are of type class B that are
children or descendants of elements contained in class A.
The difference here is, if you look closely,
there is a space between the two queries.
So while there is a comprehensive list of CSS selectors available in general,
the actual syntax available to you depends on the
implementation by your web scraping language of choice.
So it's important that you do some research yourself into
this before you commit to a specific approach to extracting
the media links that you require.
XPath is another tool in our bag of tricks when trying to extract media links.
While CSS is certainly the preferred and most recommended method to use by
default and, in most instances, is faster, there are some cases where you
cannot do what you need to do in CSS and have to use other tools like XPath.
You will recall I mentioned the importance of using the right tool for the
job at the start of this section.
Well, this is the perfect example.
XPath syntax looks very similar to CSS.
One key difference is that it generally needs a different type of
library to execute than CSS and also uses slashes to indicate the
path of the selector that you're defining.
Here is a side‑by‑side comparison of the two. To
select all paragraph elements in CSS, we use p; whereas, in XPath, we use //p.
Finding an element of an ID of foo is defined as #foo
in CSS, and in XPath //*[@id=foo].
You might think it's long‑winded,
but there are absolutely situations where you're going to
need to use XPath instead of CSS selectors.
One example is that in some CSS selector engines,
there's no way to specify a query to find elements that contain
specific text; whereas, this is possible in XPath. Also,
XPath allows you to write a query that can search both forwards and
backwards in a document; whereas, in general,
CSS implementations generally do not. Again, there's
always exceptions to the rule,
so please refer to the documentation of your tool of choice before
planning how you're going to extract your links.