We know that we can access web content by making HTTP requests to
a target URL and getting a response in return.
But what does it mean to scrape web content?
As I've mentioned earlier, websites contain a lot of information,
and you might want to programmatically extract this information.
That this web scraping.
Web scraping is the automated extraction of data from websites.
You programmatically make HTTP requests,
download the content, and extract content.
This has to be done in an automated manner for it to be called web scraping.
Before you can extract the content from a target URL,
the website content first needs to be fetched.
You need to use HTTP requests to fetch the content,
and once this content is available within your program,
you use special tools to parse this content.
This content has a specific structure that is HTML.
HTML has a tree-like structure that you need to be able to navigate.
After you parse the HTML, you'll then be able to extract specific information,
the information that you're interested in.
When you go to a website,
you'll see there are different pages that render different kinds of content.
Any website is a collection of web pages.
And web pages typically have content pertaining to a topic.
The structure and content of all web pages are defined using markup.
This markup is written in HTML, that is hypertext markup language.
This markup is a special hierarchical format that is understood by web browsers.
And web browsers know how to render this markup so you
see the pretty pages on your screen.
The styles associated with website are added using CSS.
CSS contains classes which define the style for your HTML elements.
CSS stands for cascading style sheets.
You can, of course, fetch this HTML information using a browser.
The same HTML markup can be accessed or fetched via HTTP
request made from client libraries.
These are programmatic libraries that you can use to
fetch content from a web server.
The HTML content that is returned possesses an in-built hierarchical structure.
And you need a parser to extract content from this hierarchical structure.
Parsers understand the hierarchy and can then extract information.