In this video,
we'll take everything that we have learned and put it together to
download a page and identify the links that it contains.
Now that we know how to identify the web media we want to
download and some of the ways to extract the links,
let's run some code to extract different links from a sample web page.
First,
we'll have a look at the sample web page and the HTML
code that was used to construct it.
It's extremely basic.
Our aim here is to extract the link that allows us to download the main image.
Now here's something important I want to point out.
You'll recall earlier that we discussed absolute and relative links,
and here you can see an example of this where the image uses
a relative link and not an absolute one.
This is something we'll have to address in our code.
Using Visual Studio Code,
we create a new file and save it with the py extensions
so that VS Code knows it's Python.
We'll be working with two libraries, requests and lxml.
We declare a variable with the URL location of the page we want to
scrape and then declare another variable called page to store the
results of calling the requests.get command where we pass in the page
location variable that we want to scrape.
Calling requests.get calls the page to be retrieved from the server.
The next thing we want to do is extract the page source
or the HTML from the server response.
So we create a new variable of type html from the lxml library
to store the actual HTML itself in a structured format that we
can then subsequently parse.
Now that we have a HTML object containing all of the page code,
we can then use, in this case,
an XPath query to extract all of the links from the page that are of type image.
We specify this as //img/@src as we only want to get the file
name itself and not all of the surrounding markup information
or other element attributes.
When we're looking at the page source,
we noted that the image was a relative link and not an absolute one.
To account for this,
we need to get the base domain address and then add this to
the image file path that we extract to make up a fully
qualified web media resource URL.
In this example,
it's sufficient to check for the existence of the HTTP
protocol in the string that we've extracted.
Here we examine the string extracted.
And if it starts with HTTP, we assume it's an absolute references.
Otherwise, we flag it as relative,
and we update the URL accordingly by adding the base
domain to the start of the string.
Now that we have the link extracted,
we can move on to talking about downloading the media itself.