You can configure the same Scrapy project to have multiple spiders.
Here we've created a new Python file called actual_company_details.py,
which will hold our second spider within the same Scrapy SimpleSpider project.
I've created this file actual_company_details.py under the
spiders subfolder under SimpleSpider.
Let's switch back to editing this file where we'll
write the code for our actual spider.
Import the scrapy module.
And specify a filename variable here,
which will hold the scraped contents from our Yahoo!
Finance page.
Once we extract these contents using Scrapy, we'll store it in this text file.
Go ahead and set up a new class representing your spider.
Observe that this class inherits from the scrapy.Spider base class.
Ensure that you give the spider a different unique name.
I've called it actual_company_details.
The allowed_domains and start_urls remain the same as before.
We're scraping the same site, but we'll store the contents within our text file.
Define the parse function, which takes as its input argument the response object,
which contains the URL contents.
We'll use the same XPath as before to extract a list of
company names and the corresponding intraday prices for
the stocks of these companies.
Instantiate the variable count,
which holds the length of this company_names_list,
and open up the filename, actual_company_details.txt,
in the append mode.
Run a for loop through all of the company details
that we've extracted using XPath,
and write out the company name and the company price to this file.
And that's it.
That's all we need to do to set up this simple spider,
which writes out to a file.
Save this file that you've edited,
and let's switch over to our terminal window and run this spider.
Invoke the scrapy crawl command and specify the name of the spider,
actual_company_details.
This spider will go ahead and crawl the Yahoo!
Finance page as before, but this time,
the extracted contents will not be written out to screen.
Instead, if you run an ls command within your current working directory,
you'll find that a new text file has been created here
called actual_company_details.txt.
Let's run a ls command to see the contents of this text file,
and this contains all of the information that we extracted from the Yahoo!
Finance page using our SimpleSpider.