- [Instructor] In this example, we're going
to take a closer look at the data frame structure
and how we can inspect its content.
And again, just as a reminder here,
data frames are a fairly large subject
and I'm not going to cover all the features here.
I'm just going to focus on some of the basics
and I would suggest you check out
some of the other courses here in the library
to learn more about Pandas
and data frames if you're interested.
So let's open up our example file.
It's going to be pandas_explore,
and I already have some code
to load the financial sample data
which will give us a data frame.
So let's start off by examining the data frames shape
which will give us the dimensions of the data
and to do that,
I'll print out the data frames shape.
And let's also use the describe function
which will give us
some detailed summary information about the data.
All right, so let's go ahead and save
and let's collapse this down so we have some more room.
All right, and let's run this,
so run this in the terminal.
Okay, and so we can see here in the output
that the data is 700 rows by 16 columns
and the describe function gives us
some summary about the data.
Now by default, Pandas only includes
the columns that have numerical data in the describe output,
but you can include the other columns if you want to
and you can see right here in the middle,
we have these ellipses.
That's because Pandas can't fit all of the columns
in the amount of width that the terminal has here,
and we'll go back and fix that in a moment.
Let's also output the first
and last five rows of the data
so we can see the actual data
and we'll use the head and the tail functions for that.
So I'll use the head function
and I want to see five rows,
and then I'll use the tail function
to get the last five rows.
All right, let's comment previous functions out
and then let's run this again.
All right, so now we can see the first
and last five rows of the data
and let's see, we can also see
that again, we're getting some ellipses
because it can't fit all of the rows.
It's only showing a subset of some of the rows
and we can also see that we've got some extra decimal points
that we don't really need.
Let's go back and tighten this up a little bit
by setting some Pandas display options.
So first on Pandas,
I'm going to call the set_option function
and I'm going to set the display.max.columns option to none.
And then I'll call set_option again
and I'll set the display.precision to two.
All right, so now let's save that.
Now let's run one more time
and here we go.
Okay, and now we can see
that all of the columns are being displayed.
So you can see here are the first set of columns
and then this little backslash
means that the output is continued on the next line,
and that's down here.
So we have the various columns showing the sales and so on
and then another backslash
which means that the data keeps on going, that's here.
And then we have the same output,
only in this case it's the tail.
So these are the last five rows
and again, now we're seeing
a continuation of all the different columns,
and you'll also notice that the display.precision
has been set to two decimal places.
All right, let's try out some functions
for inspecting the actual data itself.
Let's comment these out.
So let's try inspecting some of the data.
The value_counts function can be used to see how many
of each data value are present in a given column of data.
So let's use that function to see
how many times each product type appears in the dataset.
So I'm going to print out the data frame
and what I'm going to do
is I'm going to ask for the Product column,
and on the Product column
I'm going to call the function value_counts.
And let's also use the unique function
to determine all of the unique values
that appear in the column
and once again, we'll use the Product column for this.
So in the Product column,
we're going to call unique, all right.
Okay, let's go ahead.
Let's also use some basic math functions.
Let's use the min, max,
and sum functions to get more information out of our table.
So I want to see what the min
and max values are for the Profit column,
so let's print out on the data frame.
We'll look for the Profit column
and I'll call max,
and then we'll do the same thing with min.
All right, and let's also get the sum
of that column as well.
Oops, there we go.
Okay, so let's save
and let's run this.
Okay, so we have the value_counts
so that's showing us the Product column
and we can see the individual product counts.
So this is the number of times
that this product value appears in the Product column.
So we can see that the most common one is the Paseo,
followed by the Velo, followed by VTT, and so on,
and then it also gives us the data type
which is an int64.
Then we have the output of the unique function.
That's this list right here,
so these are all the unique values
that appear in that column.
Then we have the max of that column which is 262,000ish
and then we have the min
which is minus 40,000ish,
and then we have the sum of that column
which again, looks like 1., what's that?
Looks like 16 million, almost 17 million.
Okay, but suppose I only want to see the sum
of the profits for a specific product,
Pandas gives me a way to query the data frame
using a function called L-O-C.
And this isn't the only way to do this,
but it's a fairly common way.
So let's update the code to only show the total profits
for the Carretera product.
So I'm going to print out the result
and on the data frame,
I'm going to use the loc operator
and inside that I'm going to use a query statement.
So I'm going to query on the product
where the product is equal to the Carretera string.
And for that set of rows
where the product is equal to Carretera,
I want to get the Profit column
and what I'm going to do is call
the sum function on that result.
So this statement will give me the sum
of this column that meets this criteria.
All right, so let's go ahead
and comment out our previous code,
and let's save and let's run,
and we can see that this is the total sum
for all of the profits where the product is Carretera.
In fact, now we know enough about Pandas
to get a breakout of the profits for each of the products,
so let's write some code to create that output.
So what I'm going to do first is use
the Python locale module to set the format of the output
as currency in the user's locale
and you can see that if I scroll back up to the top,
I've imported the Python locale module
from the standard library.
So what I'm going to do first is on the locale,
I'm going to call set locale
and I'm going to use the locale.LC_ALL option
to set all of the locale options
to my local machine locale.
So this will affect how the currency is output.
Then I'm going to loop over each individual product name.
Well, we know how to do that
because we know how to use the unique function, right?
So for each product name in the Product column
and we're going to get the unique values
from that product column,
what I'm going to do is declare
a variable called total,
and I'm going to get from my LOC operator,
I'm going to ask for the Product column
where product is equal to the product name
that I'm currently looking at.
I want the Profit column
and I'm going to get the sum of the Profit column.
So whereas in this case,
we were only interested in a hard-coded string,
now I'm going to loop
over all of the unique values in that column
and then I'll just print out the result.
So let's print out that the Profits for,
and we'll put the product name in the output.
And then I'll use my locale
and I'll call the currency function on the total,
and I want grouping to be true.
So this will use whatever the locale uses
for grouping hundreds and thousands and so on
and now in the US that's commas,
but it might be different in your locale.
It might be a period character or something else.
All right, so let's go ahead and save that
and let's comment out this previous example,
and let's run it.
All right, and we can see in the output
that we have all the products
and we have the totals for each,
and it's in the local machine's currency
which in this case is dollars
grouped by hundreds and thousands
and millions using commas, right?
So you can see that I've just scratched the surface
of using data frames here,
but I'm sure this gives you an idea
of the kind of power that Pandas gives you
when working with datasets from Excel or CSV files.