[Autogenerated] now that we talked quite a bit about
potential implications of scraping.
If precautions are not taken,
let's talk about scraping and what problem it solves.
So what Scrapy before we can answer this question and fully appreciate the
problems greatly solved from a technical point of view,
let's have a look at the python technical landscape and
what other libraries are available from.
A technical point, scraping has two fundamental technical challenges to it.
On one hand,
there is a need to being able to fetch content from a web page or site
and once that content has been retrieved there is a need to parse that
content in order to extract data from it,
which is ultimately the goal of scraping with python.
There are a number of built in libraries that can be
used to fetch content from the web.
Some of the python libraries that can be used for fetching content from a
site are the URL lib library for python 2.7 and all the versions of python
to the URL lived to library is available,
then we have the request library which should be your default
library of choice as it is easy to use and great for Hanley
requests then we also have the HDP lib library which is older and
no longer actively supported.
Finally,
there's a google version of this library called http lib to which
offers more fine grained control over handling requests.
So that gives us a quick overview of what python libraries
are available for handling requests.
Now, let's quickly talk about parsing content.
Once the content has been retrieved from a webpage,
there is a parsing process that must take place in order to be
able to extract data desired from the site.
Depending on the nature of the data being extracted,
that parsing process might involve using a parcel that is able to
understand html and extract the value from specific html tags.
It is also possible that if the data is deeply nested within the html,
that a dumb partial might be required.
This allows to dive deeply into the pages document
object model to get specific tags.
And finally, if the data itself is embedded within an image,
it might be necessary to use computer vision to parse and perform optical
character recognition on the image to extract the text.
So as you can see,
not all content is created equal and therefore depending on
the nature of the data being extracted, different parcels might be required.
This adds to the scraping process, another layer of technical complexity.
So what is Scrapy and how can it help?
Scrapy is a python application framework that is
specifically designed for crawling websites and
extracting structured data from those sites,
which can be used for a wide range of cases,
such as data mining, information processing or historical archiving.
Scrapy was originally built for scraping but now it's mostly
used for web crawling as we saw earlier,
there are some fundamental differences between both concepts.
One of the things I like about scraping is that even
though it was designed for web scraping,
it can actually be used for extracting data using Apis as well
or simply as a general purpose web crawler.
So what else could Scrapy be used for?
We said this great,
we can be used for working with Apis which is really great as it
provides the whole application framework to do that,
but you can also use Scrapy as a general purpose web crawler that crawls
your own site to make sure that there are no broken links.
Think of it as having your own crawler that ensures that your
website has no links that redirect to a 404 page.
Wouldn't that be awesome?
Alt, delete that same service could be provided to others.
Imagine a web design agency that can provide this sort of service
to each of the websites of their customers.
Wouldn't that be cool.
This is where an application framework like Scrapy comes in to add value.
So when we think about crawling and scraping,
we should also be creative about other applications that do not
necessarily imply harvesting data from other people's sites,
but to better understand the value of scraping,
it's best we have a quick demo on how to implement an internal website.
Quality control crawler and scraper without
scraping simply using python libraries