As data analysts,
we know that there is a lot of competitive intelligence available on the web.
Many organizations have competitive intelligence teams who are
responsible for taking a look at what the competition is up to to
make sure that they're not falling behind.
Now there has to be a way to extract content from
the web in a structured manner,
and this typically involves techniques just scraping web pages.
Let's see how we can access content that is hosted on the web.
Now, typically, when you want to access something on the web,
you'll sit down on your machine,
you'll open up your web browser and type in a URL.
A URL is a resource locator which allows you to uniquely
identify where you want to get your content from.
It identifies the target resource.
But when you hit a web page when you type in a URL, what goes on under the hood?
Under the hood, your web page is making an HTTP request to a server.
This request can be made from a browser on your
desktop or from your mobile phone.
This HTTP request is then propagated through the internet,
and it hits the web server where the resources hosted are located,
where the website is hosted.
Now this request might contain a bunch of configurable parameters.
If you're using a search engine, you might send along a search term.
Along with the actual body of the request,
when you send an HTTP request,
you also send a long additional metadata information in the form of headers.
The headers give additional information to the web
server on how to deal with your request.
The website essentially looks at the request and says,
Okay, you want the information that I have,
here it is.
And it returns this information in the form of an HTTP response.
The body of the response will contain the actual
content from the target resource.
In addition,
the HTTP response will also contain metadata information in the form of headers.
The HTTP response travels over the internet.
And, finally, the content reaches your machine or your mobile phone.
And on the client end,
your browser knows how to render this content so that you can view it.
So what is this HTTP?
It stands for hypertext transfer protocol.
It's a simple,
ubiquitous text-based protocol used by browsers and
apps to access and modify web content.
HTTP is a client-server protocol.
The client makes requests and receives responses from the server.
Now a client can be anything.
It can be a browser running on your desktop or your mobile phone.
Mobile apps can also be clients.
They have libraries with which they can make HTTP requests.
Or you can have programs running on your machine making these requests
and receiving responses and parsing these responses.
HTTP clients make requests to HTTP servers.
HTTP servers are where web pages and web content is hosted.
Now this web content may be static or dynamic.
Static content refers to HTML and CSS,
which is not changed by client-side scripts.
All browsers support a scripting language called JavaScript.
If the JavaScript running on your browser updates the HTML
and adds interactivity to your web page,
that is a dynamic web page.
The tools that we'll study to scrape web pages in this
course will work with just static pages,
not with dynamic content.
That requires additional extensions.
Common web servers that are capable of hosting web content are Apache,
nginx, IIS from Microsoft, and GWS from Google.
Now when we talk about clients making HTTP requests,
there are actually different types of HTTP requests.
There is the simple GET request that you make to
fetch resources from the target URL.
A GET request is what your browser makes when it's
retrieving HTML content from a web server.
Let's say you're logged into some kind of social media website,
it's Facebook, and you update your status message.
That would involve a POST request because a POST is what
is used to create or update resources.
There is also another kind of HTTP request which you can
use to create or update resources, and that is the PUT request.
You'll use the PUT request as opposed to the POST if
your update or creation is idempotent.
If the same PUT request is made to a server more than once,
the additional PUT request should have no effect.
That is idempotent.
If you want to make a request to get only the metadata
header information from a target URL, you'll use the HEAD request.
If you want to delete resources on a site, you'll use the DELETE HTTP request.
Now whether these requests are implemented on a
particular website is up to the website owner.
A website which allows only reading of resources will support
only the GET request and not the others.
Now that we've understood HTTP requests,
we can move on and discuss HTTP responses.
Now servers which host websites and web applications
are standing by to field requests.
And once they receive a request, they parse it,
understand what is requested, and send back HTTP responses.
Every HTTP response has a few standard fields.
The response typically includes a status code.
The code can be 404 or 200.
All status codes have meaning.
200 says the response was sent back successfully.
404 says the page was not found.
All HTTP responses include a response header with metadata information,
and they also include the body of the response,
which is typically understood by a web browser.
The body of the response is what the browser displays to screen.
The body can also contain other content such as JSON
that you'd create from the server.