To wrap up the course,
let's talk about some best practices and tips to
increase efficiency while scraping.
Avoid loading images.
Most of the modern dynamic websites have lots of images.
When a page loads, Selenium loads all the elements in it,
including those images.
If we don't interact with them much,
this becomes useless and slows our script down.
This is particularly problematic and apparent if we
run our scripts hundreds of times.
The workaround to this is to use ChromeOptions to
load pages without loading images.
Employing disk cache.
In modern web browsers, disk caching reduces page loading times by a good margin.
This basically stores all the website assets,
like CSS, JavaScript, and disk storage for faster loading,
which is helpful when you load multiple pages to the same website.
Again,
all you have to do to implement it practically is to use ChromeOptions to
configure the driver to do this while initializing.
Close driver with quit.
It is important to properly close the driver after finishing the automation,
especially if you run your scripts periodically.
When you invoke your Python automation script to do things for you,
it uses additional resources and processes for the Selenium WebDrivers.
After the Python script finishes execution,
it doesn't release those additional resources if you don't tell it to do so.
A method to do that is driver.close.
But it doesn't always stop the WebDriver running the background.
For example,
if you are doing things using multiple tabs of the Selenium web browser,
then it closes only the current one,
which is why it is recommended to use driver.quit,
which closes all the tabs of Selenium web browser.
Now let's look at some of the best practices employed in the field.
Respect robots.txt.
As I have mentioned before,
robots.txt is a text file webmasters create to instruct robots
how to crawl and index pages on the website.
So, this file generally contains instructions for the crawlers.
Robots.txt should be the first thing to check when
you're planning to scrape a website.
Every website would have some set of rules on how bots and spiders
should interact with the site in the robots.txt file.
Some websites block bots altogether in the robots file.
If that is the case, it's best to leave the site and not attempt to crawl them.
Scraping sites that block bots is illegal.
Apart from just blocking,
the robots file also specifies a set of rules that they
consider as good behavior on that site,
such as areas that are allowed to be crawled,
restricted pages, and frequently limits the crawling.
Don't hit web servers too frequently.
Web servers are not failproof.
Any web server will slow down or crash if the load on it exceeds
a certain limit up to which it can handle.
Sending multiple requests too frequently can result in the website
server going down or the site becoming too slow to load.
This creates a bad user experience for human visitors on the website.
While scraping,
you should always hit the website with a reasonable time gap and
keep the number of parallel requests in control.
Don't scrape during peak hours.
To make sure that a website isn't slowed down due to
high traffic accounting for humans, as well as bots,
it is better to schedule your web crawling tasks to run in the off-peak hours.
The off-peak hours of the site can be determined by the
geolocation or where the site's traffic is from.
By scraping during the off-peak hours,
you can avoid any possible load you might put on
the server during the peak hours.
This will, in turn,
benefit you in the sense that it'll help in significantly
improving the speed of the scraping process.
Use a headless browser.
This is, again, something that is mentioned during a demo,
but it does bear repeating.
A headless browser is faster and lightweight.
Avoid using the same crawling pattern.
Only robots follow the same crawling pattern,
because unless specified otherwise,
program bots follow a logic that is usually very specific.
Sites that have intelligent anti-crawling mechanisms can easily detect
agents from finding the pattern in their actions.
Embrace transparency.
Don't misrepresent your purpose or use deceptive methods to gain access.
If you have a login and password that identifies you,
to gain access to a source, use it.
Don't hide who you are.
If possible, share your credentials.