In this demo,
we'll see how you can use the httplib client library in order to
make the URL requests to web servers to retrieve contents and to
perform other operations as well.
Here is our current working directory.
We'll be writing code in Python 3 using Jupyter Notebooks.
Under our current working directory,
I have a files subfolder which contains a bunch of HTML files that
we'll parse during the demos of this course.
The code that we'll write using Jupyter Notebooks will all be
placed in the current working directory,
which is one level up.
Click on the New dropdown and select Python 3 as your kernel.
This will bring up a new tab with a new Jupyter Notebook.
Rename this notebook so that it has a meaningful name.
I'll call it httplib2 because that is the HTTP client
library that we're about to use.
When you're working with httplib,
you might be confused between the httplib version and httplib2.
Both of these are HTTP client libraries which allow very fine-grained
access to the HTTP request that you make to your URLs.
Httplib2 is Google's version.
It's a more powerful implementation of the httplib
library and is often the one that is used.
Go ahead and install this Python package onto your local
machine with a simple pip install.
Set up the import statement for httplib2.
We'll also use the webbrowser module in order to open up and explore
the pages that we want to parse from specific URLs.
When you're learning how HTTP requests work,
it's useful to work with a web page that supports all kinds of
HTTP operations that you can play around with.
One such page is httpbin.org, and that's what we're going to use here.
I'm going to invoke the webbrowser.open function in order to
open up this URL and see what it has to offer.
You can see here that it's a very simple HTTP request and
response service that allows you to get a feel for how the
different HTTP operations work.
Observe all of the operations that this page supports.
You can try out different HTTP methods, authentication,
understand what the different status codes mean,
and so on.
Let's click through to the HTTP Methods and see what this page supports.
It supports DELETE, GET, PATCH, POST, as well as PUT operations.
If you want to understand what parameters can be
passed through to a specific request,
you can click through and view additional details.
If you click on the section header, once again,
it'll close this individual section,
and you can click through and explore the other sections as well.
This is how we'll make GET requests.
GET requests are what we'll use to retrieve information from a URL request.
You can also make POST requests to the site and mimic the
updation of a resource at this URL using this POST.
No actual updates will be performed.
If you want to test out how your program responds to certain status codes,
you can go to the Status codes section and request for certain
status codes to be returned by the site.
Forcing the server to return specific status codes will
allow you to test your local programs to see that it
responds correctly to these status codes.
You can click through to the individual sections and see what
status codes can be returned by httpbin.org.
Here are specific codes and what they mean.
300 level codes are typically redirections.
200 level codes indicate a successful response from the web server.
400 level codes indicate client errors.
500 codes indicate server errors.
There are a bunch of interesting details to explore on this page.
For example,
you have a Redirects section which allows you to force
httpbin.org to redirect your request.
In the code that we are about to write,
we'll see how we can handle redirects from within Python.
Now let's move out of this browser window and head back to our Jupyter Notebook.
In order to be able to make HTTP requests,
you need to instantiate the HTTP object from httplib2.
Invoke the request function, and when you specify no additional parameters,
this is by default a GET request that allows you to
retrieve information from a URL.
This returns two bits of information, the response headers,
as well as the response body.
Resp holds the response headers.
You can repeat pprint resp out to screen,
and you can see that these are headers or metadata information
that we get from the URL that we just accessed.
HTTP headers are name-value pairs that are parts of requests,
as well as responses.
These are the header fields in the response that we received from the
GET request that we made to the httpbin.org URL.
The content-location key gives us what URL resource we actually accessed.
Server tells us what server is running at that URL.
It happens to be an nginx server.
The response that was received when we made our GET
request is of type httplib2.Response.
The status property on the response gives us the status
code that we received from the URL.
A status code of 200 indicates that we received the response
successfully and there were no errors.
Sometimes if a resource at a particular URL has moved,
the server might be configured to redirect you to the new URL.
In that case, resp.previous will contain the original URL that you requested,
whereas we might have received the response from the
resource that we were redirected to.
Previous is empty here,
indicating that there was no redirect for our particular request.
When we made our HTTP GET request,
the actual contents of the response were placed in the data variable,
and you can see that this data contains the HTML of the httpbin.org site.
You can see that the HTML content has been retrieved in the form of bytes.
And once you have this content,
you can parse this information to extract whatever details you need.
You can call data.decode to convert this HTML to a string format as well.
And once you have it as a string,
you can perform Python string operations on this HTML content as well.
When you open up a URL, such as google.com,
on your browser,
what you're essentially doing is making a GET request to that URL.
The http.request function allows us to make that
same GET request programmatically.
Let's see the response headers that were received from google.com.
Every header has a special meaning and gives you
some information about the server.
Observe that google.com sets a cookie on your browser to track what
you're querying for so that it can show you ads.
This is something you might have known,
but observe that the set-cookie header that you see here in
the response is what sets this cookie.
The data that is returned from google.com happens to be,
once again, in the bytes format,
and you can decode this data by invoking the decode function on your byte array.
The format that you specify for your decode operation depends
on what URL you retrieve this resource from.
Different URLs will have their data encoded in different formats.