I'm Clarke Bishop, a big data engineer.
Welcome to the final module of Exploring Web Scraping with Python.
It's the Web Scraping Wrap Up.
We'll do a quick review,
and I'll provide some resources to help you continue your web scraping journey.
It seems like there's an unlimited number of scraping applications,
and we looked at several examples.
The web is full of profitable data that can lead to new
business models and new opportunities.
Whether you're focused on marketing, research,
finance, or something else, pay attention,
and you'll start seeing scraping applications everywhere.
For the course, I decided to scrape up a Tesla,
and I showed you several car buying websites.
You saw how to extract data and how to get it into pandas for further analysis.
Now, you see that all web scraping starts with downloading some HTML,
and that means sending a web request.
We explored the web request and response protocol called HTTP
then learned the key parts of an HTTP request,
the URL, a verb, and the user agent string.
You learned how to take apart a URL to simplify scraping the data you want.
In our module on selectors,
you learned all about selecting the specific elements
you want to extract from an HTML page.
It's SQL for web scraping.
You know how to use HTML and CSS selectors,
including ID and class, XPath selectors,
and Chrome developer tools to quickly find the right
selector and extract the data you want.
I gave you some guidelines to help keep you out of legal trouble.
For the best advice, always consult an attorney and be careful,
especially if you're scraping for profit,
scraping nonpublic data, or scraping personal information,
otherwise you're usually safe.
The worst case is that you can receive a cease and
desist letter and have to stop scraping.
In the module on basic web scraping, I showed you how to set up a Python project,
how to use requests to download web pages, and how
Beautiful Soup can extract data.
I even showed you some JupyterLab tricks to speed up
your web scraping development.
Where should you go next?
Well, play around with Chrome DevTools, they're right there in your browser,
and you can't hurt anything, and practice creating selectors.
The more you do, the better you'll get.
In the Advanced Web Scraping module, I showed you requests‑html.
It's similar to requests, but also lets you extract data.
So take it for a spin.
When we looked at the Scrapy framework,
you learned the difference between libraries and frameworks.
We explored the Scrapy CLI, specifically,
how to set up a project, run a crawler,
and how to refine your selectors with the Scrapy shell.
I showed you how to write a spider in 20 lines of code.
Amazing. Where to go next? Create a Scrapy project and write
your own spider and explore scrapy‑splash.
It's Scrapy's solution to the JavaScript rendering problem. In our
module on Advanced Web Scraping, you discovered how JavaScript can
be a problem, and you learned two ways to overcome the problem,
requests‑html and Selenium.
Where to go next? Try out both pyppeteer and Selenium, pick one, and decide
to really drill down and learn more. Personally, for my next scraping
project, I'll either use requests‑html or Scrapy.
I recommend that you pick one and really learn it.
Thank you! Thank you! Thank you! I've had a great time creating this course.
Tell me about the cool scraping projects you're doing, @ClarkeBishop on
Twitter, clarkebishop.com on the web, or clarkebishop on LinkedIn.
I hope you got a lot from this course,
and it gets you started on web scraping. For me, I'm going to
go for a drive in my new Tesla. See ya!