[Autogenerated] to set the stage for this course there are two
main concepts we need to fully understand.
These are crawling and scraping.
So what do these words actually mean?
Let's have a look at their definitions.
We can define crawling as the act of programmatically
retrieving or downloading a webpage is data and then extracting
all these hyperlinks and following them.
Then the process is repeated for every other page which is
referenced by the hyperlinks on the main page and so on.
So essentially the process can go many levels deep.
An incredibly sophisticated version of this process is what google does,
which is one of the fundamental parts behind the power of this amazing web
search engine crawling is associated with creating web indexes and getting
web pages to be found on the web scraping on the other hand is the act of
programmatically retrieving or downloading a webpage data and extract the
very specific information from it.
As we'll see shortly scraping has a much less
favorable reputation than crawling,
which is widely used by the world's most prominent search engines.
So let's compare each how does crawling and scraping compared to each other.
Let's start off with crawling, crawling is associated with the act of retrieving,
downloading and indexing websites,
scraping on the other hand is associated with retrieving
data directly from specific websites.
So crawling is done by search engines and that is
associated mostly with legitimate use,
scraping on the other hand is mostly done for data and market analysis to
gain a competitive edge or competitive advantage Crawley has a good
reputation due mostly to the success of search engines,
scraping, on the other hand,
has a less credible reputation as it is usually used to get
specific details from targeted sites,
Crawley has a much general scope as it is used for
fetching pages from specific sites.
However, scraping is more specific.
Its scope is to mostly mined data crawling produces larger scale results,
think how massive web search indexes are,
and also how quickly you can find anything with a search engine scraping,
on the other hand, is much smaller in scale,
producing really tiny datasets to some extent,
compared to those from a crawler.
These datasets are very specific,
know that we know and understand the differences between both.
Let's have a look at some of the controversy regarding both technologies.