In this demo,
we'll see how you can make HTTP GET requests to a target URL
using a number of different Python libraries.
All of the demos in this class will be written using the Jupyter
notebook integrated development environment.
Here is my current working directory.
Under Here I have a files subfolder,
which contains a number of different HTML files that we'll use in our demos.
Go back one level up.
In the current working directory,
click on the new dropdown and select the Python 3 kernel.
This will bring up a new notebook on a new tab running Python 3.
It's currently untitled.
Click on the name of the notebook and give it a meaningful name.
In this demo, we'll see how we can make HTTP requests to target URLs,
using a number of different Python libraries starting with httplib2.
Httplib2 is a client library used to make HTTP requests,
not just GET requests, but PUT, POST, and DELETE requests as well.
This library allows very fine-grain control over your requests.
That is its main point.
Httplib2 is Google's more powerful implementation of httplib,
and that's what we'll use here.
We've installed the httplib2 library.
Another library that you can use to make HTTP requests to
target URLs is the requests library.
You can get it with a pip install as you see here on our screen.
This offers a higher level easy and intuitive API as compared with httplib2,
as well as urllib.
If making simple requests is what you're looking for,
you'll prefer to use the requests Python library as
opposed to any other library.
Once you have these libraries installed,
set up the import statements for the libraries that we'll use.
Observe that you're importing the urllib library.
We haven't separately installed this because this comes built
in along with your Python distribution.
Urllib is built on top of httplib and offers a higher level
interface for fetching data from URLs.
Earlier versions of Python, specifically Python 2,
had two libraries, urllib and urllib2.
Both of these have been merged into one single library called urllib.
We've also imported the web browser module in Python,
which allows us to explore the pages.
I'm going to open up http:// www.pluralsight .com using web browser.
Here I am on pluralsight.com.
Observe that little lock icon next to my URL indicating
that this is a secure HTTPS connection.
How did that happen?
Well, we'll see in a bit.
If you're working with web pages and around Chrome,
you have built-in Chrome developer tools available for
you in order to inspect your web page.
Simply right-click and choose the Inspect option.
And this will bring up a little pin here allowing you to
inspect the HTML associated with your web page.
You can do this for any web page you access.
You can select individual elements,
expand these elements and see the structure of your page before you
begin parsing and extracting data from this page.
Let's switch back to our Jupyter notebook here,
and I'm going to instantiate my first request object using httplib2.http.
In order to make a GET request to pluralsight.com,
you simply need to invoke the request function on this HTTP object.
Observe that you're making a request to http:// www.pluralsight .com,
the non-secure version of the site.
The GET request using httplib2 returns two objects,
a response object with the headers of the response and
the actual content of the response.
Let's take a look at the information contained in the response object.
You can see that these were the headers returned by pluralsight.com.
Pluralsight set a cookie on our browser,
and you can see that it caches the Pluralsight page for about 15 minutes.
Another interesting detail here,
observe that the response was received from https:// pluralsight.com,
the secure version of the website.
We made a request to the non-secure version,
but we got a response from the secure version.
This is because the original HTTP URL was
automatically redirected to the HTTPS URL.
The previous property on the response object will allow you to
view the headers of the previous response,
the one that was redirected.
The previous response had a status of 301,
indicating a redirect,
and the URL which performed this redirection was the HTTP
URL to which we made the original request.
In addition to the response headers,
the GET request would've also downloaded the actual content from this website.
This content is available in the byte format as you see here on screen.
We can decode this content using UTF-8 because it has been
UTF-8 encoded and you'll get an HTML string.
You can print this HTML out to screen,
and this is the HTML that makes up pluralsight.com.
Once you have this HTML on your local machine,
you can use Beautiful Soup,
as well as regular expressions in order to parse and
extract content from this HTML.
Let's make another GET request here.
We'll use another client library, urllib.request .urlopen,
and the request we make to google.com,
and we'll search for Pluralsight on Google.
Our request is to google.com /search,
and the query parameter that we specified is q=pluralsight.
This is the search term.
When you hit Shift+Enter and make this request,
you'll get an error in the response,
and you'll need to scroll down to the bottom to see what this error is about.
HTTP Error 403: Forbidden.
This is because Google does not allow Python libraries
to make HTTP requests to its site.
The source of any request from a client,
whether it's a browser or a Python library,
is identified by the header User-Agent.
The User-Agent is set by default when you use
urllib or any other Python library.
You can explicitly specify a different header in
order to make this request to Google.
I've instantiated a urllib request object with this new
header information for User-Agent,
and I'm going to use this request to make a request to google.com.
And this time around,
we get a successful response thanks to our User-Agent header.
If you want to access the headers in your response,
you can use the info function on your response object.
Observe that a response header says Connection: close.
When you use urllib to make requests to a target URL,
the connection is closed automatically after the request is complete.
The URL to which the original request was made and the status code of the
response can be accessed using GET URL and GET code respectively.
If you want to access the contents of the response,
you'll need to use the response.read function.
You can decode this content, it's in the ISO-8859-1 format.
And you can print out the resulting HTML.
The encoded format of the content is specified in the header of the response.
Either it's UTF-8 or ISO.
Once you've invoked a response.read on a particular response object,
if you invoke the read function once again,
you'll get nothing.
This is because the connection has been closed once you've read out the data.
Let's move on to the requests Python library.
This is by far the easiest and most intuitive library
to use for any kind of HTTP request.
This is how you make a GET request to http:// pluralsight.com.
And here is the response.
It's of type requests.models .Response.
Use the status code and URL property to get the status
of your response and the target URL.
Observe that the target URL here is the secure version of Pluralsight,
thanks to the redirect.
The headers property on this response object will give you access
to the header information for the response.
If you want to access the actual content of the response,
you'll need to access the text property on the response object.