In the previous video, we examined how to identify where links are coming from.
In this video,
we will look at how to extract those links so we can work with them and a
ultimately download and process the web media we are targeting.
As we have seen,
data can be structured in a number of different ways and
come in a number of different file formats.
Depending on the data and how it's presented to us,
we can use a number of different approaches to extract the links
that we will then use to get the data that we want.
In this lesson,
we will look first at how to do this and then how to
determine the content type that, in turn,
helps us decide what tool to use to download the media data from the website.
The methods we'll examine for extracting media file names
and links are simple string manipulation,
regex, CSS, and XPath.
In this video, we'll examine the first two,
and the remaining two we'll cover in the next video.
Each of these methods has their own pros and cons,
and each could take up an entire course in its own right.
So for this lesson, we're going to cover the highlights,
and I will give you pointers on where to go and how to
find out more if you need to dig deeper.
The reason we're looking at all of these selection options is that web scraping
general requires you to chain together a number of different technologies and
solutions to get the end result that you require.
It's very common when trying to extract web data and web media to,
say,
select one part of data with one tool and then chain and process it
through to another technology and feed the combined result of that
into some final processing utility.
We're going to talk about string manipulation first as it's the most basic,
but should also be rarely used.
The reason for this is that there are better methods available.
However, sometimes it makes sense.
Selecting a link using string manipulation means using basic commands,
such as the following, to find part of a string and return it.
The commands are in Python,
but every one of them have equivalents in other programming languages.
We start by declaring a simple piece of test HTML that
contains a link to on audio MP3 file.
We then use the find method to locate the position of both the start
and the end of the tag that wraps the audio link.
In this case, it starts with href and ends with a closing angle bracket.
Finally,
we extract the substring that exists between the starting position and the
ending position and end up with the exact link that we require.
Regex is an engine available in most languages.
It takes in a string of text and defines a pattern.
And with that pattern, you can help to match,
locate,
and extract parts of the input text. Regex is an incredible technology and
very useful to have in your own web scraping toolkit.
Like most things, it shouldn't be used as the first solution to everything,
however,
as depending on the complexity of the regex expression that you
use, it can be a very expensive algorithm to execute in CPU time
and is generally avoided in preference to other methods that use a
more structured approach.
Let's take the same example we had earlier and use a
regex expression to extract the link.
We first set up our variable string that contains the test HTML.
Finally,
we send in that rather cryptic expression as a parameter, and this
tells the regex engine to extract the same string we previously
defined, but using its own particular syntax. Again, we can see the
end result is exactly what we require.