In this video,
we'll discuss methods that you can use to optimize your file download
process. When we're downloading a lot of files, it's important to be
able to optimize the process so that you're maximizing compute
resources that you're utilizing.
Having said that,
it's also very important to remember there is a balance between using your
resources optimally and being very mindful and careful at the same time of the
impact that your scraping has on the target website.
You certainly don't want to be the person who is held
responsible for putting such pressure on a website that the
website sees it as a denial‑of‑service attack.
Please ensure that you always obey the rules of the
website and only scrape responsibly, legally, and ethically.
You will recall in an earlier lecture,
we discussed large files and the Accept‑Ranges header.
The presence of Accept‑Ranges in the header tells us that the
service supports partial downloading so we can exploit this
when dealing with large files.
In this example,
we can see that the content‑length of the file is more
than 250 MB. So we will use multiple request objects to
download the files in different, smaller parts.
Then when we're done stitch all those parts back
together again into the one original file.
We can use this and other concurrent techniques to download large
files in parallel, and this will be quicker than if we were to use one
simple sequential approach. In our example code, you can see that we
examine the overall content size,
and then we create five request instances and let each
instance download 20% of our file.
The main point here to observe is that each individual request
instance uses a specific header called Range that specifies the start
and end byte range that it wants to receive.
So the code is very simple.
We import our requests module,
we declare the URL of the file we want to download,
and then we set up the range of the headers that we want to get.
When we run the code, we can see it downloading into the five
different files, and we can then combine them back together and see
if our code worked. As expected, it did.
We now have a full file again reconstructed from multiple parts
downloaded by different requests instances. The power of being able
to download a specific portion of a file,
media, or otherwise is extremely useful.
One of the things we can use it for is auto‑resuming file downloads where a
network failure occurred causing us to download only part of the file. By
combining the known size of the target file and referring to the header
attribute content length and the size of the data that we were able to
download before somebody pulled the plug on the internet, we can easily work
out where we need to resume and use the requests instance to get the file
that we want,
starting from the range in the remote file where we experienced the failure.
This is a very useful trick to know and to implement and is a
good pattern to use when you're trying to ensure that your web
media scraping processes are robust and can handle general
network failures gracefully.