PyCon 2018: Using pandas for Better (and Worse) Data Science¶
GitHub repository: https://github.com/justmarkham/pycon-2018-tutorial¶
Instructor: Kevin Markham¶
- GitHub: https://github.com/justmarkham
- Twitter: https://twitter.com/justmarkham
- YouTube: https://www.youtube.com/dataschool
- Website: http://www.dataschool.io
In [1]:
import pandas as pd
pd.__version__
Out[1]:
In [2]:
import matplotlib.pyplot as plt
%matplotlib inline
In [3]:
# ri stands for Rhode Island
ri = pd.read_csv('police.csv')
In [4]:
# what does each row represent?
ri.head()
Out[4]:
In [5]:
# what do these numbers mean?
ri.shape
Out[5]:
In [6]:
# what do these types mean?
ri.dtypes
Out[6]:
- What does NaN mean?
- Why might a value be missing?
- Why mark it as NaN? Why not mark it as a 0 or an empty string or a string saying "Unknown"?
In [7]:
# what are these counts? how does this work?
ri.isnull().sum()
Out[7]:
In [8]:
(True == 1) and (False == 0)
Out[8]:
In [9]:
# axis=1 also works, inplace is False by default, inplace=True avoids assignment statement
ri.drop('county_name', axis='columns', inplace=True)
In [10]:
ri.shape
Out[10]:
In [11]:
ri.columns
Out[11]:
In [12]:
# alternative method
ri.dropna(axis='columns', how='all').shape
Out[12]:
Lessons:
- Pay attention to default arguments
- Check your work
- There is more than one way to do everything in pandas
In [13]:
# when someone is stopped for speeding, how often is it a man or woman?
ri[ri.violation == 'Speeding'].driver_gender.value_counts(normalize=True)
Out[13]:
In [14]:
# alternative
ri.loc[ri.violation == 'Speeding', 'driver_gender'].value_counts(normalize=True)
Out[14]:
In [15]:
# when a man is pulled over, how often is it for speeding?
ri[ri.driver_gender == 'M'].violation.value_counts(normalize=True)
Out[15]:
In [16]:
# repeat for women
ri[ri.driver_gender == 'F'].violation.value_counts(normalize=True)
Out[16]:
In [17]:
# combines the two lines above
ri.groupby('driver_gender').violation.value_counts(normalize=True)
Out[17]:
What are some relevant facts that we don't know?
Lessons:
- There is more than one way to understand a question
In [18]:
# ignore gender for the moment
ri.search_conducted.value_counts(normalize=True)
Out[18]:
In [19]:
# how does this work?
ri.search_conducted.mean()
Out[19]:
In [20]:
# search rate by gender
ri.groupby('driver_gender').search_conducted.mean()
Out[20]:
Does this prove that gender affects who gets searched?
In [21]:
# include a second factor
ri.groupby(['violation', 'driver_gender']).search_conducted.mean()
Out[21]:
Does this prove causation?
Lessons:
- Causation is difficult to conclude, so focus on relationships
- Include all relevant factors when studying a relationship
In [22]:
ri.isnull().sum()
Out[22]:
In [23]:
# maybe search_type is missing any time search_conducted is False?
ri.search_conducted.value_counts()
Out[23]:
In [24]:
# test that theory, why is the Series empty?
ri[ri.search_conducted == False].search_type.value_counts()
Out[24]:
In [25]:
# value_counts ignores missing values by default
ri[ri.search_conducted == False].search_type.value_counts(dropna=False)
Out[25]:
In [26]:
# when search_conducted is True, search_type is never missing
ri[ri.search_conducted == True].search_type.value_counts(dropna=False)
Out[26]:
In [27]:
# alternative
ri[ri.search_conducted == True].search_type.isnull().sum()
Out[27]:
Lessons:
- Verify your assumptions about your data
- pandas functions ignore missing values by default
In [28]:
# multiple types are separated by commas
ri.search_type.value_counts(dropna=False)
Out[28]:
In [29]:
# use bracket notation when creating a column
ri['frisk'] = ri.search_type == 'Protective Frisk'
In [30]:
ri.frisk.dtype
Out[30]:
In [31]:
# includes exact matches only
ri.frisk.sum()
Out[31]:
In [32]:
# is this the answer?
ri.frisk.mean()
Out[32]:
In [33]:
# uses the wrong denominator (includes stops that didn't involve a search)
ri.frisk.value_counts()
Out[33]:
In [34]:
161 / (91580 + 161)
Out[34]:
In [35]:
# includes partial matches
ri['frisk'] = ri.search_type.str.contains('Protective Frisk')
In [36]:
# seems about right
ri.frisk.sum()
Out[36]:
In [37]:
# frisk rate during a search
ri.frisk.mean()
Out[37]:
In [38]:
# str.contains preserved missing values from search_type
ri.frisk.value_counts(dropna=False)
Out[38]:
In [39]:
# excludes stops that didn't involve a search
274 / (2922 + 274)
Out[39]:
Lessons:
- Use string methods to find partial matches
- Use the correct denominator when calculating rates
- pandas calculations ignore missing values
- Apply the "smell test" to your results
In [40]:
# this works, but there's a better way
ri.stop_date.str.slice(0, 4).value_counts()
Out[40]:
In [41]:
# make sure you create this column
combined = ri.stop_date.str.cat(ri.stop_time, sep=' ')
ri['stop_datetime'] = pd.to_datetime(combined)
In [42]:
ri.dtypes
Out[42]:
In [43]:
# why is 2005 so much smaller?
ri.stop_datetime.dt.year.value_counts()
Out[43]:
Lessons:
- Consider removing chunks of data that may be biased
- Use the datetime data type for dates and times
In [44]:
ri.drugs_related_stop.dtype
Out[44]:
In [45]:
# baseline rate
ri.drugs_related_stop.mean()
Out[45]:
In [46]:
# can't groupby 'hour' unless you create it as a column
ri.groupby(ri.stop_datetime.dt.hour).drugs_related_stop.mean()
Out[46]:
In [47]:
# line plot by default (for a Series)
ri.groupby(ri.stop_datetime.dt.hour).drugs_related_stop.mean().plot()
Out[47]:
In [48]:
# alternative: count drug-related stops by hour
ri.groupby(ri.stop_datetime.dt.hour).drugs_related_stop.sum().plot()
Out[48]:
Lessons:
- Use plots to help you understand trends
- Create exploratory plots using pandas one-liners
In [49]:
ri.stop_datetime.dt.hour.value_counts()
Out[49]:
In [50]:
ri.stop_datetime.dt.hour.value_counts().plot()
Out[50]:
In [51]:
ri.stop_datetime.dt.hour.value_counts().sort_index().plot()
Out[51]:
In [52]:
# alternative method
ri.groupby(ri.stop_datetime.dt.hour).stop_date.count().plot()
Out[52]:
Lessons:
- Be conscious of sorting when plotting
In [53]:
# mark bad data as missing
ri.stop_duration.value_counts()
Out[53]:
In [54]:
# what four things are wrong with this code?
# ri[ri.stop_duration == 1 | ri.stop_duration == 2].stop_duration = 'NaN'
In [55]:
# what two things are still wrong with this code?
ri[(ri.stop_duration == '1') | (ri.stop_duration == '2')].stop_duration = 'NaN'
In [56]:
# assignment statement did not work
ri.stop_duration.value_counts()
Out[56]:
In [57]:
# solves SettingWithCopyWarning
ri.loc[(ri.stop_duration == '1') | (ri.stop_duration == '2'), 'stop_duration'] = 'NaN'
In [58]:
# confusing!
ri.stop_duration.value_counts(dropna=False)
Out[58]:
In [59]:
# replace 'NaN' string with actual NaN value
import numpy as np
ri.loc[ri.stop_duration == 'NaN', 'stop_duration'] = np.nan
In [60]:
ri.stop_duration.value_counts(dropna=False)
Out[60]:
In [61]:
# alternative method
ri.stop_duration.replace(['1', '2'], value=np.nan, inplace=True)
Lessons:
- Ambiguous data should be marked as missing
- Don't ignore the SettingWithCopyWarning
- NaN is not a string
10. What is the mean stop_duration for each violation_raw?¶
In [62]:
# make sure you create this column
mapping = {'0-15 Min':8, '16-30 Min':23, '30+ Min':45}
ri['stop_minutes'] = ri.stop_duration.map(mapping)
In [63]:
# matches value_counts for stop_duration
ri.stop_minutes.value_counts()
Out[63]:
In [64]:
ri.groupby('violation_raw').stop_minutes.mean()
Out[64]:
In [65]:
ri.groupby('violation_raw').stop_minutes.agg(['mean', 'count'])
Out[65]:
Lessons:
- Convert strings to numbers for analysis
- Approximate when necessary
- Use count with mean to looking for meaningless means
11. Plot the results of the first groupby from the previous exercise¶
In [66]:
# what's wrong with this?
ri.groupby('violation_raw').stop_minutes.mean().plot()
Out[66]:
In [67]:
# how could this be made better?
ri.groupby('violation_raw').stop_minutes.mean().plot(kind='bar')
Out[67]:
In [68]:
ri.groupby('violation_raw').stop_minutes.mean().sort_values().plot(kind='barh')
Out[68]:
Lessons:
- Don't use a line plot to compare categories
- Be conscious of sorting and orientation when plotting
12. Compare the age distributions for each violation¶
In [69]:
# good first step
ri.groupby('violation').driver_age.describe()
Out[69]:
In [70]:
# histograms are excellent for displaying distributions
ri.driver_age.plot(kind='hist')
Out[70]:
In [71]:
# similar to a histogram
ri.driver_age.value_counts().sort_index().plot()
Out[71]:
In [72]:
# can't use the plot method
ri.hist('driver_age', by='violation')
Out[72]:
In [73]:
# what changed? how is this better or worse?
ri.hist('driver_age', by='violation', sharex=True)
Out[73]:
In [74]:
# what changed? how is this better or worse?
ri.hist('driver_age', by='violation', sharex=True, sharey=True)
Out[74]:
Lessons:
- Use histograms to show distributions
- Be conscious of axes when using grouped plots
13. Pretend you don't have the driver_age column, and create it from driver_age_raw (and call it new_age)¶
In [75]:
ri.head()
Out[75]:
In [76]:
# appears to be year of stop_date minus driver_age_raw
ri.tail()
Out[76]:
In [77]:
ri['new_age'] = ri.stop_datetime.dt.year - ri.driver_age_raw
In [78]:
# compare the distributions
ri[['driver_age', 'new_age']].hist()
Out[78]:
In [79]:
# compare the summary statistics (focus on min and max)
ri[['driver_age', 'new_age']].describe()
Out[79]:
In [80]:
# calculate how many ages are outside that range
ri[(ri.new_age < 15) | (ri.new_age > 99)].shape
Out[80]:
In [81]:
# raw data given to the researchers
ri.driver_age_raw.isnull().sum()
Out[81]:
In [82]:
# age computed by the researchers (has more missing values)
ri.driver_age.isnull().sum()
Out[82]:
In [83]:
# what does this tell us? researchers set driver_age as missing if less than 15 or more than 99
5621-5327
Out[83]:
In [84]:
# driver_age_raw NOT MISSING, driver_age MISSING
ri[(ri.driver_age_raw.notnull()) & (ri.driver_age.isnull())].head()
Out[84]:
In [85]:
# set the ages outside that range as missing
ri.loc[(ri.new_age < 15) | (ri.new_age > 99), 'new_age'] = np.nan
In [86]:
ri.new_age.equals(ri.driver_age)
Out[86]:
Lessons:
- Don't assume that the head and tail are representative of the data
- Columns with missing values may still have bad data (driver_age_raw)
- Data cleaning sometimes involves guessing (driver_age)
- Use histograms for a sanity check