Now that we know what web scraping is all about,
let's talk about the two phases in modern web scraping,
fetching web content and parsing the content that you've
downloaded and fetched from a target URL.
If you want to fetch web content,
you'll make HTTP typically GET requests to a URL,
and you'll download the content so that it's available within your program.
Once you have the content available,
you can parse it using many different techniques.
You can perform HTML parsing, DOM parsing,
you can even extract HTML content using computer vision techniques.
If you're working in Python,
you can fetch content using one of the many client libraries available.
There is requests, there is urllib, and httplib2.
Requests is an intuitive and easy to use API that should be a preferred choice.
Httplib2 is for more granular control over your HTTP requests.
It's Google's version of the httplib library.
Urllib is available as a part of the standard Python distribution.
Websites are hosted on web servers.
And web servers make the content of websites available on HTTP endpoints.
And these HTTP endpoints can be accessed using URLs.
When you use a web browser on your machine to
access a web page at a certain URL,
under the hood your browser is making an HTTP GET
request to get the content of web pages.
This same GET request can be made programmatically as well.
And that's what web scraping is all about.
Web scraping makes requests programmatically,
downloads content from web pages, and once this content is available,
you can extract information from this content.
There are many libraries and utilities available to make HTTP
requests to web pages to download content.
If you're working on the command line,
cURL is a utility that you might want to use.
If you are working within a programming language such as Python,
there are a number of Python libraries that give you programmatic access to HTTP
requests and the responses that are received from websites.
Once you have the content available on your machine,
you can perform some kind of parsing operation to extract
the information that you're interested in.
Now this parsing can be HTML parsing or DOM parsing.
When we speak of HTML parsing, we typically refer to static pages.
You parse the HTML and the CSS associated with web content.
With DOM parsing, we talk of dynamic content in addition to static content.
Dynamic content is content that is generated by scripts,
that is JavaScript running with the new web browser.
HTML parsing cannot parse dynamic changes made by JavaScript,
whereas with DOM parsing,
it can parse dynamic changes made by code elements such as JavaScript.
Now when a browser renders an HTML page to screen,
behind the scenes, it sets up a document object model,
which is basically a hierarchical tree structure of your HTML.
When you work with HTML parsing,
you're working with structure text rather than the
object representation of the DOM, or the document object model.
With DOM parsing,
it can work either with text elements or with objects in their DOM.
In this course, we'll focus our attention on HTML parsing using Beautiful Soup.
Now HTML parsing can also be performed using regular expressions.
Regular expressions are all about pattern matching after all.
Let's discuss parsing using regular expressions before we
move on to discussing Beautiful Soup.
Regular expressions are a sequence of characters that
define a search pattern within text.
It's used to find strings that satisfy specific rules.
Regular expressions can be used with any kind of text,
not just with HTML, but when you get HTML content from a URL,
it's text after all, which means you can use regular expressions there as well.
Regular expressions are based on text pattern matching,
and regular expressions can range from the very simple to arbitrarily
complex as you'll see in the demos that follow.
Regular expressions are very similar to grep and other
string utilities that you might have used if you've worked
in the UNIX operating system.
Regular expressions are not specific to Python.
They are a portable syntax and can be used in almost any
programming language of your choice.
All high-level programming languages typically have a specific library or
package which manages and handles regular expressions.
Regular expressions basically define rules,
and then the regular expression pattern matcher will look
for strings that match those rules.
Regular expressions are a classic rule-based approach in
that they are very local in nature.
Regular expressions do not understand the structure of the document as a whole.
They only look for patterns.
Regular expressions are also very, very complex.
The syntax is arcane and error-prone.
A single character can make a huge difference.
I have to be honest with you,
I've never found working with regular expressions easy.
Regular expressions are also fragile,
especially if you're working with HTML content.
HTML markup in the real world tends to be messy and incomplete.
You might have open tags which haven't been closed.
Well, regular expressions won't be able to handle those.
Regular expressions are still useful, though,
because they offer a quick and easy way to parse content
that you've downloaded from the web.
But if you really want to scrape web content well,
you'll want to use Beautiful Soup.
Beautiful Soup is a Python package for parsing HTML,
as well as XML,
and it has the ability to work with malformed markup such as missing tags.
Even if your HTML markup isn't perfect,
Beautiful Soup will add in all of the important tags that are missing and give
you well-formed HTML to work with and extract content from.
Beautiful Soup is an easy-to-use and intuitive Python package,
and it mitigates the weaknesses of regular expressions for parsing.
When you parse your HTML using Beautiful Soup,
it forms a parse tree of your entire HTML content by default.
You can navigate up and down this parse tree,
go sideways, and also perform search and filtering operations.
Beautiful Soup is relatively simple and intuitive to use,
and it's also robust to problems in markup being parsed.
Markup is very unstructured and is unclean normally out in the real world.
Beautiful Soup takes care of that for you.
If you want to work with DOM parsing,
your options are use a framework such as Scrapy or Selenium.
We'll talk more about Scrapy in just a bit,
but you should know that Scrapy is a framework that combines all of these.
Making HTTP requests, performing HTML, as well as DOM parsing.
If you want to build a production-grade system for crawling
the web and parsing content from the web,
Scrapy is what you do since it's a framework for building production-grade,
heavy-duty, web parsing systems.
Now before we go on with Scrapy, let's talk about this last little branch here.
There are computer vision applications out there in the world,
which make it easy to parse web content.
An example is this technology called Sikuli.
Sikuli is a visual technology to automate and test graphical user interfaces.
Sikuli does not parse HTML, it simply extracts UI elements using computer vision.