With that, we come to the end of this module and this course.
This module was compromised mainly of two different parts,
challenges faced during and tips for web scraping.
We explored six different types of challenges,
namely lack of bot access, captcha's, frequent structural changes,
getting blocked or banned, dynamic content,
and questionable data quality.
We went over three tips to increase efficiency of scraping.
Avoid loading images, employing disk cache, and closing driver with quit.
And lastly, we learned six widely accepted best practices for scraping,
which are respecting the robots.txt file,
not hitting servers frequently, not to scrape during peak hours,
using a headless browser, avoiding the same crawling pattern,
and embracing transparency.
That's it.
It's been an absolute pleasure presenting the course to you.
I hope you enjoyed the learning process as much as I enjoyed teaching.
Good luck.