[Autogenerated] We've covered a bit of ground in this module,
so let's quickly review what we have learned so far.
First,
we started off by understanding what crawling and scraping
are and how both compared to each other.
And what are their main differences.
Then we explore in death what problems may arise if we disregard the
terms of service of a site we want to crawl in scrape,
such as actions that the site owner may take and also look at some
possible legal consequences that may come up.
We then talked about the typical contour arguments that people
tend to use to justify scraping third-party sites and how they
usually lead to falsy conclusions.
Based on this, we looked at advice in general and how to crawl in scrape,
such as using a reasonable crawl rate,
using a user agent string and respecting robots dot txt.
Then we quickly looked at various standard HTTP python libraries that
could be used to retrieve content from the web.
After that, we also explore what type of content is possible to parse,
and finally we briefly introduced scraping and
later to understand its real value.
We explore how to extract data without using it,
which we saw in a demo in the next module will explore are two
scaffold and run our first project with scrapie.