And this demo brings us to the very end of this module on
working with the parse tree using Beautiful Soup.
We started off by understanding the HTML tree structure and the hierarchical
relationship that exists between the nodes of an HTML tree.
We then installed the Beautiful Soup package on our machine and
installed the different parser libraries that we use.
We saw how we could create soup objects to represent our HTML tree structure.
We then worked with tags, attributes, NavigableStrings, and comments.
We then saw how we could perform filter operations using the find and
findAll functions to search for specific elements.
We used filters with tags, attribute values,
regular expressions, as well as custom functions.
We then moved on to something that was fairly real world.
We saw how we could use Beautiful Soup to extract links from documents,
all of the links leading off a certain page.
And, finally,
we rounded off this module by using Soup Strainer to
parse just parts of a document.
In the next module,
we'll move on from Beautiful Soup onto the Scrapy application framework.
We'll see how we can use the Scrapy shell to
interactively extract content from web pages.