From a previous discussion, it's clear that web scraping involves two phases.
You first need to fetch content from a target URL.
You then need to parse the content that you fetched.
Fetching content involves programmatically accessing the contents of a website.
Parsing content involves extracting the information
from the content that you fetched.
The way you'd fetch content from a website is by making HTTP requests to
that site programmatically using a client library.
There are many different techniques that involve parsing
content from a website: HTML parsing, DOM parsing,
and the use of computer vision technologies.
Before we move on to parsing content,
let's talk about using HTTP client libraries to make HTTP
requests and download content from a URL.
Now there are many such libraries available in Python.
If you look online, things can get quite confusing.
There is urllib.
There seems to be urllib2.
There is requests, there is httplib, and httplib2.
Which one do you use and when?
Let's look at all of the Python libraries available out there.
The first that you might encounter is requests.
This is a high-level, intuitive API and by far the easiest to use.
Then there is httplib2 and httplib.
Both of these reference the same HTTP client library.
Httplib2 is the more powerful version.
It's basically httplib which was enhanced by Google.
And then you might also come across urllib and urllib2.
Now these two libraries are different packages when you use Python 2.7,
and they're non-overlapping.
Urllib2 can't do everything urllib can.
They're supposed to be used together.
But when you're working in Python 3, as we shall do in this course,
we have just the one urllib.
It has four namespaces,
and it contains everything that was originally included across both libraries.
So here are the choices laid out for you when you're
looking for HTTP libraries in Python.
So which one should you choose and why?
If you want an intuitive and easy-to-use library which
makes working with web requests very, very simple,
requests should be your default choice.
Requests is a very high-level API.
If you want more granular control over the HTTP requests that you make,
you might choose to use httplib2.
This is Google's version of httplib.
Httplib2 will give you fine-grained control over
the HTTP requests that you make.
You can configure them in a very granular manner.
Httplib,
there's little reason to use this directly when you
have the more powerful version.
Now if you don't want to pip install an additional package on your machine,
an easy way to make HTTP requests is to use urllib.
This is a part of Python standard library.
There are four distinct namespaces for different operations.
Urllib.request is for opening and reading URLs,
urllib.error contains the exceptions raised by urllib.request,
urllib.parse is for parsing URLs,
and urllib.robotparser is for parsing the robots.txt file.
The single urllib package and namespace in Python 3 subsumes the old urllib2.
So if you're using Python 3.x, you only need to worry about urllib.
Urllib2 is Python 2 only.
Now that we got a big-picture understanding of all of these libraries,
and then you would choose one over the other,
let's move on.
You can use any of these client libraries in order to fetch the web
content that web servers make available on HTTP endpoints.
These HTTP endpoints are essentially the URLs that you use.
Browsers make HTTP requests under the hood to get web pages.
Web scraping involves making the same requests programmatically.
And if you want to make programmatic requests to web servers,
there are many libraries and utilities available.
You can make requests from the command line or from a
programming language of your choice.
If you want to make web requests from a command line, you'll use cURL.
And if you want to make these requests programmatically,
well, Python itself gives you a huge array of choices.
We won't even need to worry about the other languages.