We talked a fair bit about GET forms and their utility.
The other kind of forms are POST forms.
Compared to GET forms, both forms are not screen‑friendly.
Unlike GET forms,
you can't share the URL so that you and the person you're
sending it to view the same information.
You type in the term you're looking for,
click Search, and there's nothing in the URL,
like this for example.
This is a web page on the website of the Californian
Department of Consumer Affairs.
Among other things,
the DCA licenses or certifies practitioners in more than 255 professions.
Let's say you want to search for the licenses of a few professionals.
We'll enter some values in these fields.
For boards and bureaus, let's select the Dental Board of California.
For license type, let's say Dentist License.
We won't fill in the license number and business name,
and we type in John for the first name and Kelly
for the last and click on Search.
There we go.
There seems to be eight people holding a dentist license from the DCA
with the first name John and second name Kelly.
Let's check the URL.
As you can see, it gives you no indication about what you search for.
No parameters are present in the query string.
The parameters, the queue for your search for example, aren't in the URL.
They're hidden in the actual request.
If you try and get the names of the dentist using the same trick as before,
you'll get nothing in return.
You can search the response text to make sure.
What this means is that every time you request something from a POST‑based form,
you have to pretend you filled out the form and clicked the button.
That is, when we clicked on Search, it also sent the server a bunch of data,
all of the options we typed in or the drop‑downs we selected.
We're going to use Chrome's network tools to analyze all the
requests our browser sends to the server,
then imitate them in Python.
To do this, go back to the page you started your search from,
go to the View tab, Developer, and click on Developer Tools.
Here, select the Network tab.
Fill the form out and submit it.
Then, you can see the Network tab bustling with activity.
Now, scroll up to the top of the network ping,
select the segment of the URL _____, which would be this for us.
In the Headers section, scroll down until you see Form Data.
This form data is what we need to send along with our request.
We just need to convert it to a dictionary and send it along.
This is the section of the code that does that.
As you can see, if you go back to the developer tools,
everything that is present there is converted as a
dictionary back in the notebook.
And this obviously is the URL of the page.
The rest of it is standard procedure except for a couple of things.
We use requests.post to make our request.
And instead of providing just the URL here like we did before,
we also passed in the form data.
This is the result.
All the names in the page are printed out here as well.
By learning how to handle these two types of form data,
you're in prime position to scrape a large majority of the websites out there.