Product developers’ guide to getting started with AI — Part 1: Introduction to dataframes
Tags: ai, pandasdataframe, python, deved
Published: 2021-09-15 23:35:05


## TLDR
When working with AI, it’s important to know how to import data sets, read through tables, and understand what the structure is.

## Outline
1. Introduction
2. Before we begin
3. Downloading Prerequisites
4. My first Dataframe
5. Reading Metadata
6. Viewing Data
7. Data Analysis
8. Conclusion

## Introduction
Welcome to the “Product developers' guide to getting started with AI”. In this series, we’ll go over key concepts and run through examples using Pandas. First, we will cover setting up your development environment and learning how to inspect your data. Then, you’ll be ready to tackle the more exciting parts of AI throughout this series.

## Before we begin
For the most part, Google Collab has everything already installed except the dataset, skip to My First Dataframe. However, if you want to run it locally then follow the next step. We’ll be using:

* Python
* Pandas
* NumPy

## Downloading Prerequisites (Optional)
When getting started with AI, 2 important libraries you’ll be using every day are Pandas and Numpy. Follow the link here for instructions to install [Python](https://www.python.org/downloads/), [Pandas](https://pandas.pydata.org/getting_started.html), [NumPy](https://numpy.org/install/), and access to [Google Collab](https://colab.research.google.com/).

## My First Dataframe
First, we’ll begin by going through how to upload files and download our first data set, the [Titanic](https://github.com/pandas-dev/pandas/tree/master/doc/data/titanic.csv), hosted by the Pandas community on Github.
Open up [Google Collab](https://colab.research.google.com/) and click on the new notebook button.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hyaq7ziz6edb6isbtmae.png)_Click on New notebook_

Next, we’ll begin by importing titanic.csv to create your first dataframe. Go to the file tab, and click on the file with the arrow to upload from your computer.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vyzw3pynpvgdidgm9s69.png)_Click on the file with the arrow icon to import titanic.csv_

Then import Pandas, Numpy, and use read_csv to extract our CSV data into a dataframe.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3ygya6y6pk8xme899voo.png)_At the beginning, import the libraries and file via code_

Type the name of the dataframe to view it. Here we call it df, so in the next cell we type df. To run the cell use Shift+Enter or click the run icon at the left.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ul7x8cu0kmv18793bgy8.png)_Display entire dataframe_

## Reading Metadata
Unlike a table, a dataframe has some extra data behind the scenes, called metadata. Metadata is used to organize its structure and can be viewed in Pandas by using the describe, info, and columns method. Let’s say we wanted to know how many rows and columns contain non-empty values or how much storage the data takes up.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jo3wj497qlqib36g1mvc.gif)

Info is a great method that product developers who have worked with SQL will find similar to the EXPLAIN command. It tells us valuable information about the storage space used, column information, number of rows, indices, and types. All while organizing it into an easy-to-read table.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1lwhtzz0ivaw8k6qhxui.png)_Show all information about the dataframe_

Describe is a method best used to summarize the numerical data by calculating a quick mathematical summary and displaying the count, mean, min, max, standard deviation, and percentiles.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/y3ogdcvtxnyfobcq4n0u.png)_Default output of describe_

This is by default equivalent to df.describe(include=[np.number])

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/zdhcpzfr49he46rfgjoa.png)_Describe all numbers_

By adding the object keyword, describe looks for the unique, top, and frequency of the data for object data, such as strings and timestamps instead. Here, it selects the columns that have a data type of object from the output.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/h29ecnkrv921dwm7b95p.png)_Describe all objects_

Conversely, you may also use exclude instead of include to get the reverse outputs.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/lw4cdi33wew17uwoeysz.png)_Describe everything that is not an object_

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/17rqp2hsx9hnvsrb2afw.png)_Describe everything that is not a number_

But, columns is an interesting method that is used to read metadata and select data. To get the metadata of a column, call it on a dataframe to get the index names.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0rz3bipwg05o5nbjmizt.png)_Display all index names_

There are two ways to select a column, using either the index position or index name. The index position can be found from the metadata of info on the left.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/eyj3vmme2v8tthdzg52s.png)_Access by index position_

The index name can be found from the output of columns.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hchaocc3lygne2hix2pk.png)_Access by index name_

## Viewing Data
But most of the time, especially when working with AI, you’ll have very large datasets and it may not be feasible or necessary to display everything. Dataframes have other features to view parts of the data, by using the head, tail, loc, and iloc method.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/z4ffhwjnulzzac4vu0ox.png)_Time to use Python to chop down the data_

Let’s take a look using indexing with the head or tail method.

To view the data on the first 5 rows, we use head(5)

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/y314f0ec1du3lqjafxg5.png)_Head refers to the start of the dataframe_

Then, to view the data for the last 5 rows, we use tail(5)

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/a70bz237b4cc1na4ewac.png)_Tail refers to the end of the dataframe_

We can view multiple columns using loc, specifying the row index found on the left of the dataframe, along with the names of the columns to view. Since our row index is unlabeled, we use integers to quickly access them. The ‘:’ command is to set a range of values, to include everything.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/bizqrguv4fj4leqc8srg.png)_View each Name, Ticket, and Fare_

Similarly to loc, you can also use the index position with the iloc command instead.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1iert6efsd3fkeay3xkz.png)_Name, Ticket, and Fare are 3, 8, and 9 respectively_

## Data Analysis
Combining what we’ve learned, let’s answer common data analysis questions about the Titanic dataset that data scientists and marketing ask themselves every day.

How many people were aboard the Titanic when it sank?
* From info, we see that 889 people embarked on the ship
How much did the average passenger pay?
* From describe, the mean fare was $32
What was the standard deviation or “std” between ticket prices?
* From describe, std of the fare is $50
What was the highest cost for a ticket?
* From describe, the max fare is $512
Who was the first person to pay for a ticket?
* Using the head on the name column, Mr. Owen Harris Braund
Who was the last person to pay for a ticket?
* Using the tail on the name column, Mr. Patrick Dooley
Who was the 100th person to purchase a ticket?
* Using iloc for row 99 of column[3], since position starts from 0, Mr. Sinai Kantor

## Conclusion
That covers the info, describe, and columns functions for reading metadata and head, tail, loc, and iloc for viewing dataframes. Check back next week for our next guide, “Surfing through dataframes”, where we’ll be taking a look at how to search through our imported data by grouping, ordering, and rearranging the dataframe’s structure.