Let's look at some common challenges you'll be up
against while scraping web pages.
First up is the lack of bot access.
Websites are free to choose whether they allow agents or bots,
and you'll come across plenty of them that do not allow automated web crawling.
Reasons could vary from the malicious to the benign where they simply
don't want you replicating the content elsewhere.
As for malicious reasons,
there are times when bots scrape data with the intention of
gaining competitive advantage and drain the server resources of
the website they are scraping from, thus,
adversely affecting site performance.
To see if a website allows scraping,
always check the robots.txt file in which they will have specified what bots are
allowed or disallowed and on which sections of the website.
You can find the file at www., the website name, .com / robots.txt.
Also,
do check the terms of service in order to under what they allow
you to do with the data you've just scraped.
Captcha's are completely automated public Turing tests
to tell computers and humans apart, has been implemented to do just that.
Differentiating between computers and humans and consequently keeping spam away.
In the presence of captcha's, basic scraping scripts tend to fail.
However, advancements have been made in getting around these,
although the same can be said for captcha's as well.
Although the technology to overcome captcha's can be implemented,
they could still slow down the scraping process quite a bit.
Frequent structural changes.
In order to keep up with the advancements in UI and UX,
and to add more features, websites under regular structural changes.
The web scrapers are specifically written with respect to the code
elements of the web page at the point of setup,
so frequent changes complicates the codes,
giving scrapers some sort of a hard time.
Though every structural change will not effect the web scraper setup,
but as any sort of change may result in data loss,
it is recommended to keep a tab on the updates made to the site or to outsource
web scraping projects to a dedicated service provider who will take complete
care of the monitoring and maintenance of crawlers.
Getting blocked or banned.
IP blocking is an issue that's rarely a problem to the good crawling bots.
However,
there can be false positives and sometimes even the
harmless bots could get blocked by the IP blocking
mechanisms implemented by target sites.
IP blocking typically happens when a server detects unnaturally
high number of requests from the same IP address or if the
crawler makes multiple parallel requests.
Some IP blocking mechanisms are a bit too aggressive and can block the
crawler even if it follows the best practices of web scraping.
Websites can also make use of honeypot traps.
Honeypots are meant to detect crawlers or scrapers.
These can be hidden links that are not visible to the users,
but can be extracted by scrapers or spiders.
Once your crawler visits such a link,
your IP address can be flagged for further investigation,
or even be instantly blocked.
There are also bot blocking services that could be integrated into the
website in order to identify and block automated web crawlers.
Dynamic content.
Although websites are increasingly becoming interactive and user friendly,
this has a reverse effect on web crawlers.
In fact,
new website with a lot of dynamic crawling practices
are not at all crawler friendly.
Examples of these are lazy loading images, infinite scrolling, etc.
Selenium has plenty of techniques for getting around these,
some of which we have already seen in this course.
Questionable data quality.
It is important that the quality of the data
collected by the scraper is kept high.
Faulty data can cause serious problems if you're using any ML or
AI technologies on top of the collected data.
Making sure the data meets quality guidelines while crawling is
difficult because it needs to be performed in real time.
One thing you can do for this is to write test cases.
You can make sure whatever your scrapers are extracting is
correct and they are not scraping any bad data.