In this demo,
we'll see how we can make HTTP requests and handle
responses with a different Python client,
the urllib library.
The urllib library is built on top of httplib,
which we've already worked with in an earlier demo,
and it offers a higher-level interface for fetching data from URLs.
Now, previously, Python version 2 had two libraries, urllib and urllib2.
Both of these have been merged together in Python 3.
All of the functionality available across these original two libraries are
now available in different namespaces within urllib.
So if you're working with Python 3, simply import the urllib module.
Once again, in order to explore the functionality that urllib offers,
we'll work with the httpbin.org site.
This is a site that you are now familiar with,
this is what it looks like, and we'll make GET,
PUT and POST requests using urllib this time around.
If you want to download the contents of a particular URL using urllib,
simply invoke the urllib.request .url open function and specify the URL.
The URL open function makes a GET request to the target resource.
The response that you get here is a file-like object that
allows you to read the contents that have been downloaded
from the resource that you requested.
The response object has a geturl function,
which tells you what URL was originally requested,
and the getcode will give you the status code of the response.
You can access the header information of the response by
invoking the info function on the response object.
Here are all of the headers that were returned from httpbin.org.
The response object acts kind of like a file handle.
The read function allows you to read in the contents that have been downloaded.
The contents have been shown in the data variable,
and I'm going to call data.decode UTF-8 in order to get the actual HTML.
Once you have programmatic access to the HTML contents,
you can use scraping tools such as Beautiful Soup to parse this HTML.
The resp.status and the resp.reason properties gives you
additional information about the response.
The response object that the URL open function returns acts like
a file handle in that if you try to read the response once again
after it has already been read,
you'll find that the contents of the response are completely empty.
Imagine that the file pointer has moved to the end of the file and can no
longer read contents from the beginning of the file.
If you want to make other types of HTTP requests to a target resource,
you need to instantiate a urllib request object.
Let's make an OPTIONS request to our httpbin.org URL to see what
kinds of requests are supported by this URL.
Urllib.request .urlopen accepts the request object as an input argument as well.
And once again,
resp.info will give you access to all of the headers in our response.
If you want to make POST requests using the urllib library,
you'll need to encode your post_data.
Unencoded post_data is not accepted by urllib.
I'll first need to URL-encode my POST data so that special
characters are understood by my server.
I've then encoded the resulting data in the ASCII format,
and then I make a POST request to httpbin.org /post.
The request object allows me to specify the kind of HTTP request I want to make.
This happens to be a POST request as specified in my
URL and ASCII-encoded POST data.
I've also specified an additional header.
I'm trying to indicate to the server that this request was made by the
Mozilla Firefox browser rather than using a Python library.
Observe that the OS info is present here as well.
This is what the User-Agent would look like if you were to
actually make this request using Firefox.
Pass this request object into the URL open function,
and let's print out the headers in our response.
Let's now read and decode the response that we received
for the POST request that we just made.
Observe that the User-Agent in the response is the
User-Agent that we had sent up in the request.
When you work with HTTP request and response programmatically,
it's good practice to make your request using a with block.
With urllib.request .urlopen, then I specify the URL that I want to open,
and once I have access to the response within the with body,
I can download the contents of the response decoded
and do whatever else I need to.
Observe here that I made a request to google.com,
but I've also specified a search term using a query parameter,
q=pluralsight.
Now it turns out that Google really doesn't allow you to
programmatically search for terms using the URL in this format.
The response is an error.
An exception has been thrown.
At the bottom, you can see that it's an HTTP Error 403: Forbidden.
This is what Google returned.
Encountering errors when you're scraping URLs is quite common,
which is why you need to be able to handle these errors within your code.
From urllib.error, import the URLError and HTTPError exceptions.
If the server cannot handle your request for any reason,
it typically throws an HTTP error.
For example, authentication errors are also HTTP errors.
It's good practice to make your requests to a server using a try-except block,
handle the error explicitly within the except block.
Here our HTTP error is the Forbidden 403.
A URL error is a more generic error.
Let's say you're trying to access a resource that does not exist,
for example, the site does not exist, the urllib library will use a URLError.
You can see that the reason for this error indicates
that the server name does not exist.
This request was made to a non-existent URL.
The parsed namespace in urllib has specific functions that
allow you to parse the components of a URL.
Invoke the urlparse function and specify the URL as you see here on screen.
The parsed_data that you get in the response will contain
the individual components of your URL.
Every URL is made up of sub-components,
and these sub-components can be parsed using the urllib.urlparse function.
Observe that we have a scheme, a location, and a path within this URL.
The path gives the location of the file on this server.
You can see that the scheme for this URL is http.
That is the protocol.
You can see that the location of the URL is loonycorn.in :80.
The port at which the server runs is also part of the location.
Here is a URL with some more sub-components.
We have the scheme, location, and path, and in addition,
we also have a query string.
After the path that indicates the location of the file that
we're trying to access on the server, we have a question mark,
and everything that follows a question mark becomes part of the query string.
Here is an example of a different URL which has a fragment sub-component.
A fragment in the URL is everything after the hash.
The fragment in our URL is #courses.
The fragment in the URL is often used as a bookmark within a page.