In this video, we're going to extract links for web media.
We'll talk about different kinds of links, how to identify and find them,
and how to use developer tools to assist in the task.
We'll get started by extracting links for some simple media data,
such as images.
The first thing we're going to extract media links from is a
simple test website. For our first link,
we can see from the browser pretty easily what part of the page refers to what
type of media and what kind of links are there. The product title and the rest
of the information about the book is text based.
There are two links to external web pages via the publisher link
and the book landing page link. And finally,
we have a link that displays the main image that we
want to identify and extract.
Knowing where to find the links to the media that you
want to download is obviously critical.
You might think that they are always in the page source,
but this was not always the case.
Let's look at two examples. In the first, we can see a simple site
where everything is laid out in the page source.
So it's easy to follow and track down where the data is coming from.
The origin and the source path of the image that
we're looking for is really obvious.
It's right there in front of us. In this next example, however,
when we look at the source code of the page,
we don't see any reference to the image whatsoever.
This is because the image is injected into the page document by an external
process such as a linked CSS file are maybe JavaScript.
It's important to remember that you can have 1, 2, or 100
web pages that look 100% exactly the same.
But you can also have 1, 2, or 100 or more methods of
constructing the website. So you cannot assume things are
always going to be as you think.
It's all down to how the website and the individual page was
designed and if the developer used a framework to lay out the
page and manage its resources.
For this reason,
there will always be an element of investigation and
analysis before you start finding links on a website and
scrape it for either text or web media.
The first step in approaching any scrape project is to analyze the target data
you want to acquire and figure out where it's coming from.
While there are a number of specialist tools that you can use,
my first port of call is normally the developer tools that
are built into all modern web brothers.
Developer tools were primarily designed to help developers build and
debug the web pages and websites that they're building.
However,
they're also in extremely powerful tool to help us understand how a
website is built and how the data that is being displayed to us is
generated and where it comes from in the first place.
The easiest way to launch developer tools in the browser is to
right‑click on the page and select Inspect.
This is pretty much the same no matter if you're using
Chrome, Edge, Firefox, or Safari, etc.
For our example purposes, we will use the developer tools built into Chrome.
The principal for what developer tools offers is
the same in all of the browsers.
The main area in developer tools are tabs that are separated into information
about the elements that make up the page that you're viewing, the console
where you interact with the page dynamically and see any messages emitted of
use to the developer, the Sources tab that contains a list of all of the
individual component parts that are used to make up the web page that you're
viewing, and the Network tab that monitors everything that is sent and
received while the page is both built and used. The Performance, Memory,
Security, and Lighthouse tabs are mostly of interest when building sites,
so we won't refer to those in this course.
Finally,
the Application tab lets us view everything in one place that refers to the web
page when it is being used as an application. This means any storage the page
uses, background services, and other useful information. From a media scraping
point of view, this can be very helpful,
as you can often get indicators in local and session storage that assist in
identifying information relevant to the media format being used.
Another useful tab in Chrome Developer Tools is the Media tab.
This can be hidden by default, so to view it,
click on More tools and Media to show this tab. When you're
viewing a page that contains video data,
this tab can be very useful in helping to plan help how to download the data
that you need to extract. Both YouTube and Netflix are good examples to check
out this function. You can see from both sites that the Media tab provides
the same useful detailed information.
The information that we view here can be very detailed,
in fact, mostly more than we actually need when scraping web media.
But it is important to know it's there because there will be
situations where you need to dig deep, and this is generally where
you go to find that particular bit of information that will help
you meet your media scraping goals.