Before we move on to talking about Scrapy,
let's continue our discussion of parsing web content.
We had left it midway.
We know that web scraping involves fetching content and parsing content,
and in parsing content we've specifically spoken of
HTML parsing with Beautiful Soup.
Here we'll discuss HTML parsing and DOM parsing in a little more detail.
We'll understand and compare the two.
When we speak of HTML parsing, we typically refer to static pages.
You parse the HTML, and the CSS associated with web content.
With DOM parsing, we talk of Dynamic content in addition to Static content.
Dynamic content is content that is generated by scripts,
that is JavaScript running within your web browser.
HTML parsing cannot pass dynamic changes made by JavaScript,
whereas with DOM parsing,
it can pass dynamic changes made by code elements such as JavaScript.
Now when a browser renders an HTML page to screen,
behind the scenes it sets up a document object model,
which is basically a hierarchical structure of your HTML.
When you work with HTML parsing, you're working with structured text,
rather than the object representation of the DOM,
or the document object model.
With DOM parsing, it can work either with text elements,
or with objects in the DOM.
If you want to perform HTML parsing, Beautiful Soup is a great library,
far better than using regular expressions or grep utilities.
If you want to work with DOM parsing,
your options are use a framework such as Scrapy or Selenium.
We'll talk more about Scrapy in just a bit,
but you should know that Scrapy is a framework that
combines all of these making HTTP requests,
performing HTML, as well as DOM parsing.
If you want to build a production-grade system for crawling
the web and parsing content from the web,
Scrapy is what you'd use as a framework for building production-grade,
heavy-duty web parsing systems.
Now before we go on with Scrapy, let's talk about this last little branch here.
There are computer vision applications out there in the world,
which make it easy to parse web content.
An example, this is technology called Sikuli.
Sikuli is a visual technology to automate and test graphical user interfaces.
Sikuli does not parse HTML, it simply extracts UI elements using computer vision.