All modern browsers have built‑in developer tools.
I'll show you the tools in Chrome, but other browsers are similar.
Developer tools can be a big help in selecting the
exact content you want to scrape.
In this section,
we'll look at real web pages and put what you've learned into practice.
In the demo, you'll learn how to access Chrome's developer tools,
using the separate panes inside of developer tools,
editing styles to better visualize selectors,
generating CSS selectors directly inside of Chrome,
generating XPath selectors inside of Chrome,
and some real world challenges.
Here's our simplified HTML page one more time.
Let's open up Chrome Developer Tools.
On a Mac, it's Command+Shift+C, on a PC,
Ctrl+Shift+C, or you can just hover over the element you want to select,
like this price, right‑click, and select Inspect.
See how Chrome highlighted the price HTML in the elements pane?
In the top pane, there are all the HTML elements,
and in the bottom pane, the CSS styles.
We can use both to help us find the right selector.
If you missed or clicked on the wrong tag,
you can click another HTML tag, like p with class description.
Whatever you click on,
it's highlighted in the top pane and in the actual browser window.
Here's a trick you'll like, all this is editable.
Let's go pick p with the class price again.
See the .price style in the CSS pane, you can add a style,
say, background‑color, and set it to orange.
Chrome even gives you valid options along the way,
so that makes it even easier to pick. Now you can
easily see all the p .price elements,
and you know what you'll get if you select them.
Don't like all that orange? Just uncheck the checkbox next to background‑color,
and you can turn the style off.
Or if you really mess up,
refresh the page to clear all of your changes, and don't worry,
you're not changing anything on the web, just in your local browser.
I want to show you another great developer tools trick.
Chrome will create the selectors for you.
First, I added this scratch pad to the page, but I hid it before.
Let me turn that back on.
There we go unchecking the checkbox again.
There it is. Now so you want to select the li with id="vin 9381".
Right‑click the tag, then go to Copy, and Copy selector.
It seems like nothing happened, but it did.
Click in the scratch pad, then paste the clipboard.
Whoa!
It copied the ID selector.
Let's try this with p .price. Right‑click, Copy, Copy
selector, click in the scratch pad, and paste.
See how this works? Chrome will do all the heavy selector work.
Well, you may need to tweak it a little bit, but Chrome gets you close.
You can even copy XPath selectors. Right‑click, Copy, and Copy
XPath, paste this into scratch pad, and there you go.
This is really a fabulous tool for being able to see how all this works and
find the selectors you need right inside of Chrome. Our simplified HTML
page has been great for learning the concepts,
but let's get real and put our skills to work on the real web.
You're looking at the Tesla page on Wikipedia.
There's always plenty of good stuff to scrape on Wikipedia,
but let's scroll down to production and sales by quarter.
Tesla's been ramping up production,
but I might not want one of the very first cars
produced in the third quarter of 2017.
This is a pretty good example to get you started with real world scraping.
You know what to do. Right‑click on one of the elements you want,
then click Inspect.
Oh, it's a table. Tables are neatly structured and always great for scraping.
Let's select the whole table.
Scroll up to find the table tag in the HTML elements pane.
There it is, table.wikitable should do the trick.
To be sure, let's drop down to the styles pane and
change the background color to orange.
There we go. Then scroll up and down to make sure we
won't select anything we don't want.
I'm not going to scroll too much or I'll make you seasick.
I checked the whole page, though,
and table.wikitable will do exactly what we want.
Let me refresh the page to get rid of the orange and
click the X to close developer tools.
The other thing that's interesting about this page are the source footnotes.
Let's click on one, number 105 looks interesting.
It ultimately takes us to a Tesla press release.
So that's where the data came from,
right from a Tesla press release, so, we could have
scraped Tesla and built our own table.
Once you get good at scraping, you see opportunities to scrape everywhere.
Let's check out another interesting example from Bloomberg.
I was searching to see if any Tesla models might
have had more problems than others, and I found this interesting graph.
Sure enough, February 2019 was a bad month.
Now many of these problems may have been fixed right away,
but it's still something to look out for.
It's more advanced,
but you can actually scrape graphs, as long as they're built with HTML.
This won't work for image files like jpeg or png though. It takes
a bit of detective work to analyze a graph.
As always, start by right‑clicking and choosing Inspect . Hmm, a wrecked tag.
What's that?
Oh, I see.
It seems to be part of an svg section.
See the breadcrumbs at the bottom of the HTML pane? Hover over svg,
and it highlights the tag.
So this entire graph is an svg.
Then there are some g tags.
Those are used to group svg elements like the wrecked we started
with. The last g in the list looks like a series.
This may be the data we want.
Click the g, then hover over the g tag.
It's looking good. Now collapse the g tag we've been
inspecting, that one is series‑0, the g tag right below is series‑1.
Remember, I told you it takes some detective work? Well the lower part of each
column is in series‑0 and the upper blue part is in series‑1.
I had to poke around some to discover the connections.
Also notice that when you hover over a column,
a pop‑up appears that shows the data for that column, that's the data we
want, only, the pop‑up is generated by JavaScript,
so we can't see the data.
These are the real world problems you'll encounter
in your web scraping projects.
I'll give you one solution on what you can do.
Hover over July of 2018.
The lower part is 76.4,
and the upper blue part of the column is 4.6. Right‑click and Inspect again.
The height of that column is 118, so 118 maps to 76.4. Right‑click and
inspect the blue top part of the column, the height is 7,
so that maps to 4.6.
So, scrape the table and apply some ratios, and now you've got the data.
Try this for homework.
Click one of the g tags, then Copy and Copy selector.
The result will be a very long selector. Real world web scraping can be complex,
but it isn't really any harder than the simple examples we started with.
You just have to stay organized and take it step by step.
In this module,
you learned all about selecting the specific elements
you want to extract from an HTML page, its SQL for web scraping.
Now you know how to use HTML and CSS selectors,
including ID and class, XPath selectors, and Chrome developer tools to
quickly find the right selector and extract the data you want.
We're going to be writing some code soon.
First though,
I want to keep you out of trouble by exploring the
legal issues with web scraping.