In this video,
we will look at steps we need to take to investigate the web media we wish to
download before proceeding to the download action itself.
Preparing to download media files involves checking with the
server what the details of the file you're about to download are,
and then taking the appropriate action.
You can, of course,
just go straight and issue a GET command for the files, assuming that this
is the default method for retrieving the media file, and that's fine as part
of your overall production process, however,
for a more robust solution,
especially during the preliminary investigatory stage,
it's good to check the properties of what you're about to download
by using the HEAD command. When investigating a website, one of the
useful tools that I use is Postman.
It's available on Windows, Mac, and Linux, and there's a
free plan for individual developers.
It's great for testing things out quickly, and extremely useful when you
have to start working with different APIs and using POST methods to log
into sites and retrieve different resources,
including media files, from certain websites.
Let's start working on some media now by examining some of the media
we might want to download from a test website.
The first thing to do in Postman is to select the type of
HTTP request that you're going to send. For our first test
we'll use the HEAD command,
and we'll put the URL of the media we want to
investigate into the location input box.
We then click the Send button and examine the results that come back.
We can see some interesting information in the headers
of the response that's returned.
First of all, we see the content‑type; it's image/jpeg.
The next thing of interest is the content‑length.
The content‑length is the size of the resource expressed in bytes.
To visualize this better, we can divide it by 1024, so the image
we're downloading in this case is about 49 kilobytes.
Two other interesting things to note are first, the last‑modified value.
This tells us when this particular resource was last updated, and this is
extremely useful as it can help us only download resources that have
changed since the last time we visited the site.
The second major thing of note is the accept‑range header. In
this case it says bytes, and this tells us that the server is
designed so that we can not only say, give us a particular file,
but we can actually specify a position in the file
that we want to download from.
We're going to use this particular feature later when we look at both
concurrent scraping of large media files and auto resuming files where a
network connection has broken in the middle of data transfer.
It would be considered good practice to carry out a solid
investigation of the site that you're going to scrape for
web media before you start coding.
Both postman and the developer tools built into all of the modern
browsers are excellent starting points for this.
Some of the other tools you could use are Charles Proxy and Fiddler2,
both of which operate on Windows, macOS, and Linux.
You don't need to use one of these tools, of course,
you could just work, if you wish, from the command line, curl or
similar, but regardless of how you approach the task,
the message here is that spending time carrying out a careful
investigation of the media you want to scrape,
taking into account things like content type,
average file size,
frequency of file modification, and the availability of the accept‑range
flag that allows us to scrape media concurrently, and also auto‑resume
from failed network transfers, is a very important step in the overall
process that shouldn't be neglected.