You may already know all about HTTP.
Still, a short review is essential to avoid web scraping problems.
Browsers automatically handle the HTTP details for a human.
A web scraper, though, has to manage the details and get it right.
The web and HTTP runs on messages, and there are two types,
request messages and response messages.
Let's say you fire up your browser and enter a URL.
The browser sends a request to the server,
the server then decides what to do and sends back a response.
Even within a single web page, this process may repeat many times.
Each image or JavaScript file requires its own request and response.
HTTP is an abbreviation for Hypertext Transfer Protocol.
The name sounds like HTML, but they're very different.
HTTP is used for the messages that retrieve HTML from a server.
You'll also see HTTPS.
The S is for secure.
It's exactly the same HTTP protocol but encrypted for greater security.
What's in an HTTP request?
There's always a web address or URL and a verb.
Plus, there's usually other information like the user agent.
You already know what URL looks like,
and we're going to explore URLs further in just a bit.
But what's an HTTP verb, also called on HTTP method? The verb
tells the server more about the type of request and what action
to take, what to do with the message.
When you're writing code that works with an HTTP API, you may use several verbs,
but for web scraping, there are only two we normally use.
GET simply retrieves an HTML page.
GET is the default verb and just means get some data from the server.
Most of the time, that's all you need.
POST sends data to the server like a form submission.
Sometimes we need to submit a form to log in or otherwise
persuade the server to return the data we want.
The user agent is a string that identifies the browser or web scraper.
It's optional,
but something you may need to pay attention to. The user agent
string doesn't make for a very pretty slide,
but it's got all kinds of interesting information about the
browser version and operating system.
It might not look like much to a human, but the server loves it.
I promised that we'd look at URLs in more detail.
Let's do that next.