All of us click on URLs every day, but what's really inside a URL,
and how can that help us scrape web pages?
That's what we'll explore in this section.
Here's a nice car buying website, iSeeCars.
They've got car listings and a nice search and filter form.
Lebanon, Kansas?
Yeah, it turns out that's the geographical center of America.
Let's take a look at the URL.
Wow, that's ugly.
How about this?
Better?
Let's break it down.
URLs consist of several parts.
First is the scheme.
For the web, http or https, followed by ://.
That tells the server we're using the HTTP protocol.
Then the host.
In this example, it's iseecars.com.
That's the server's name on the web.
Sometimes the server's name is followed by a colon and a port number.
The default is port 80 for HTTP and 443 for HTTPS.
As long as you're just scraping normal websites,
the defaults should work just fine.
Next is the path, /used‑cars/used‑tesla‑for‑sale.
The path is the address of the data on the server,
and it can include a file name that ends in .html.
Or it could just be a path like this example.
Last but by no means least is the query string or URL fragment.
Technically, everything after a question mark is a query string,
or after a hash, it's a URL fragment.
In practice,
query strings and URL fragments deliver virtually the same capabilities.
Either way, I'll just call it a query string.
The query string contains a list of key‑value pairs separated by ampersands.
For example, location=66952.
That's the ZIP Code for Lebanon, Kansas.
Location is the key, and 66952 is the value.
Next after an ampersand, Radius=all.
That's the nationwide value for a nationwide search.
Each site can have its own key names.
You just have to experiment to use the data and manipulate the page.
Some of the keys are cryptic, like the last one.
Who knows what that means.
Now this page has a nice search and filter form.
Guess what?
The form just changes the query string.
In a browser when you change the form, it reloads URL with new query values.
That's a very common pattern, and it can really help you with scraping.
We don't have to use the form anymore, as you know how to hack the URLs.
Want to change the ZIP Code?
Simply change location= to the new value and load with the new URL.
Very handy.
Python has some of the best string manipulation functions.
One of my favorite features of Python 3.6 and higher are f‑strings.
Officially, it's a formatted string literal, but f‑strings are easier to say.
This code shows how you can easily build a start URL.
See how f‑strings work?
They're a regular string with an f in front.
Then you can insert any variable in curly braces.
That way,
it's really easy to manipulate the URL to suit your application or to
loop through several URLs with different URL settings.
The Python request library advertises itself as HTTP for humans,
and it really delivers.
Armed with a start URL, you can try out the request library.
Remember the HTTP GET verb?
Use request.get with a start URL to send to a GET request to a web server.
It will capture the response and save it in the variable downloaded_page.
Only, the value in downloaded_page is a full response object.
To see the actual HTML content, print out downloaded_page.text.
You'll see more on the request library in a later module,
but for many situations, this code snippet is all you'll need.
In this module,
we've explored human browsing versus scraping by an automated browser,
an overview of the HTTP protocol, and URLs and URL hacking.
With that under your belt,
let's move on to extracting data from web pages with CSS and XPath.