[Autogenerated] all right.
We cover a lot of ground so far and talked about non legal,
legal and technical implications of crawling scraping.
So now it's time for a demo.
Let's put these concepts into practice by writing a Python
script that is able to extract data for a quality insurance
purposes without scraping,
as this will later help us to appreciate the value of scrapie
AD DS as a framework without further ado,
let's jump right in.
I have visual studio code open,
and here I have a Python I'll that will host the script for this demo.
Let's open up the built in terminal and before we start writing any code,
let's install the required python packages.
With this command, we can install the request module,
which will be responsible for executing HTP requests.
So let's import the request library to be able to process.
Response is obtained.
We need to be able to parse.
Um, well, first,
be using plain regular expressions to parse out the HTML Markab return.
For that, we need to import the Ari module.
The goal is to extract all the u.
R.
L s from the HTML market return,
we can accomplish our goal by using regular expressions but
to show you different ways of achieving.
This will also do the same with partners,
So let's go ahead and install these partners first.
Let's run this command to install the L XML part sir,
which is quite popular and fast.
It is widely used to parse XML N HTML.
Then let's install the HTML part, sir,
with this command and also the select to lack sparser using this command.
And finally, let's install the beautiful soup module,
which is what we'll be using throughout the next modules with scrapie.
But we'll have a quick look at it right now,
so let's create a python function called extract links
reggae AD CS that will be responsible for extracting the
Urals from the market response.
We passed the TXT parameter,
which represents the HTML mark apps using the compile method from the Ari
Library we can pass it the regular expression we want to look for.
In this case,
we're looking for your oils by parsing HTML a tags indifferent
if the tag is in lowercase or upper case.
If any of the context of txt,
which is the HTML Marca matches that regular expression which
is achieved by calling the final method.
Then those results are returned, which would be a list of URL SSHD.
Now let's call the get method from the request library and let's
fetch the HTML content from my own website.
Next, let's pass the result of the call to the get method,
which is the HTML Marca from my website to the extract Links
Ray AJAX Method and print out the results.
Let's save and run the script in the terminal and see what we get.
Cool.
A list of all the girls from my web page.
This doesn't look very nice, so let's create a function to print it line by line,
which we can do as follows.
So now let's use this function to print out the
results instead and run the script Cool.
This is much better now.
Instead of using regular expressions,
let's use the L XML Par Sur to implement the same logic.
Let's do this by creating a new function.
Call extract Lynx l XML.
So we create an empty list that will contain the girls and then from the
markup we look for ETags using the DOM dot XPath method.
So the a ETags reference with an XPath syntax.
Each attack phone is added to the list, which is then returned to try this out.
Let's invoke this function and run the script again.
As you can see, it's exactly the same results,
but using a different parsing mechanism.
No, let's do the same, but instead using the HTML partner approach.
We can do this by creating and extract links HTML Partner Function,
to which we pass the txt parameter that represents the HTML mark apps.
We also have an empty list and then we create an instance of HTML part sir.
And for each DOM element that is an attack.
We get the value of its href attributes and added to the list,
which is then returned to try it out.
Let's invoke this function as follows As you can see,
The results are also the same, but using another partner method.
Finally, let's try out the parsing method that will be using with scrapie,
which is beautiful soup.
Let's define the extract Bs function and pass it the
extracted HTML markup as the txt parameter.
We also define an empty list and instantiate the beautiful super class,
bypassing the markup, and also indicate that it will use the L XML part,
sir.
Then we loop through all the tags and find a ETags.
The value of the href property of each attack is then added to the list,
which is finally returned.
As you might have already noticed by using beautiful soup,
the syntax is clearer and the logic easier to implement than
using any other approaches you so far.
All right, let's try it out.
As you can see, it also works.
Using regular expressions is by far the fastest in terms of code execution.
But it is the less reliable approach for parsing HTML.
We can check out the execution of beautiful soup by using the
time it method as followed to this method.
We pass in the instructions that we want to benchmark.
Such is the set up and the actual code both as strings.
If we run the script, we can check out the execution time.
Not bad at all.
This is how fast the parsing with beautiful soup took.
So far, we have only crawled one level.
So what if we want to crawl the URL obtained from crawling my own site and
go one level deeper level two or if we want to print out just a subset of
those results say we are just interested in filtering you are else that
are related to Pluralsight courses.
We could do this by checking if the URL we want to print is
contained within the filter criteria,
which we can check with the in filter function.
If the URL string contains one of the filtering keywords, then we returned true.
So let's try it out and check the results.
We can see, only you are else related to Pluralsight courses.
If we want to get the girls contained within the pages
referred by those Pluralsight courses,
then we need to get each of those individual URL is and
parts the content of each of those pages.
To do that, we need to define the follow list function,
which is very similar to the print list with filter function.
What is different is that when we reach level two,
we need to make a request to eat URL found a level one.
Get the girls contained within each, and then print out the results.
We can try it out and run the code cool.
We now see the URL is contained within each of the Pluralsight courses,
pages referenced by my site.
But what happens if we want to go another level down and
extract the Earls of those Pluralsight courses?
Pages?
You can see that at this point,
if we keep on using the same approach we have been using,
we will probably need to nest some code within the follow
this function and add another request and loop within the
second loop for every level deep.
It would be the same.
Ultimately, as you can see, things start to get a bit complicated.
Each level we go down.
There are two solutions to this problem.
Either we create a recursive function,
or we can use a framework like scrapie that is designed for crawling sites,
no matter how many levels deep.