This is a guided project from Dataquest.io and there are many like it, but this one is mine.

The purpose of this project is to explore how using the pandas plotting functionality along with Jupyter notebook allows us to explore data quickly using visualizations.

We'll be working with a dataset on the job outcomes of students who graduated from college between 2010 and 2012. The original data on job outcomes was released by American Community Survey, which conducts surveys and aggregates the data. FiveThirtyEight cleaned the dataset and released it on their Github repo.

Headers for recent-grads.csv are shown below:

Header | Description |
---|---|

Rank | Rank by median earnings |

Major_code | Major code, FO1DP in ACS PUMS |

Major | Major description |

Major_category | Category of major from Carnevale et al |

Total | Total number of people with major |

Sample_size | Sample size (unweighted) of full-time, year-round ONLY (used for earnings) |

Men | Male graduates |

Women | Female graduates |

ShareWomen | Women as share of total |

Employed | Number employed (ESR == 1 or 2) |

Full_time | Employed 35 hours or more |

Part_time | Employed less than 35 hours |

Full_time_year_round | Employed at least 50 weeks (WKW == 1) and at least 35 hours (WKHP >= 35) |

Unemployed | Number unemployed (ESR == 3) |

Unemployment_rate | Unemployed / (Unemployed + Employed) |

Median | Median earnings of full-time, year-round workers |

P25th | 25th percentile of earnings |

P75th | 75th percentile of earnings |

College_jobs | Number with job requiring a college degree |

Non_college_jobs | Number with job not requiring a college degree |

Low_wage_jobs | Number in low-wage service jobs |

After a quick inspection and cleaning of the dataset, we will move on to using various types of plots/visualizations to help answer questions like these:

- Do students in more popular majors make more money?
- Using scatter plots

- How many majors are predominantly male? Predominantly female?
- Using histograms

- Which category of majors have the most students?
- Using bar plots

- Students that majored in more popular subjects make less money.
- 76 majors are predominantly male. 96 majors are predominantly female.
- Men dominate the majors that are associated with higher median salaries
- Women dominate the majors that are associated with lower median salaries
- The majority of salaries are between \$30,000 and \\$40,000
- Unemployment is lower in the top 10 majors (by median salary) than those in the bottom 10 majors
- The most chosen majors by students are:
`Business`

`Humanities & Liberal Arts`

`Education`

`Engineering`

`Social Science`

- The most chosen majors by women are:
`Business`

`Education`

`Humanities & Liberal Arts`

`Health`

`Psychology & Social Work`

- The most chosen majors by men are:
`Business`

`Engineering`

`Humanities & Liberal Arts`

`Social Science`

`Computer Science & Mathematics`

In [1]:

```
#load necessary libraries using conventional aliases
import pandas as pd
from pandas.plotting import scatter_matrix
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
#read the file into a dataframe using pandas
recent_grads = pd.read_csv('recent-grads.csv')
```

Let's take a look at what we have! There are many ways to inspect a dataframe and below are just a few of the classics.

I like these (in this order) since it not only helps us to become familiar with the data itself but also helps in identifying any issues that we may need to address before moving on to any kind of data manipulation, analysis, or constructing plots.

In [2]:

```
#inspect the dataframe by looking at the first entry
recent_grads.iloc[0]
```

Out[2]:

In [3]:

```
#inspect the dataframe by looking at the first five rows
recent_grads.head()
```

Out[3]:

In [4]:

```
#inspect the dataframe by looking at the last five rows
recent_grads.tail()
```

Out[4]:

In [5]:

```
#inspect the dataframe by way of statistical summaries on columns
recent_grads.describe()
```

Out[5]:

In [6]:

```
#inspect the dataframe by summary on dtypes and columns
recent_grads.info()
```

In [7]:

```
#store original number of rows of the dataframe (173)
raw_data_count = recent_grads.shape[0]
```

The `df.info()`

method has revealed something interesting: our dataframe contains null values. Let's see what that's all about.

In [8]:

```
#display all rows with null values as a dataframe
recent_grads[recent_grads.isnull().any(axis=1)]
```

Out[8]:

Thankfully, the null values in our dataframe are confined to just the one row.

Since plotting often requires two variables with an equal number of values/observations for each, it is quite possible we end up throwing errors if we keep this row in our dataframe. Best to just drop it and keep going.

In [9]:

```
#remove rows containing null values
recent_grads.dropna(inplace=True)
cleaned_data_count = recent_grads.shape[0]
print('Difference in number of rows before and after cleaning:', raw_data_count - cleaned_data_count)
```

Scatter plots are a great tool for visually inspecting data for relationships between two variables.

Things to look for in a scatter plot:

- how changes in one variable affect the other
- general trend as we read our data from left to right on the plot
- clustering of data points

In [10]:

```
#scatter plot of 'Median vs. Total'
recent_grads.plot(x='Total', y='Median', kind='scatter', title= 'Median vs. Total')
```

Out[10]:

Although there is a lot of variability in this scatter plot (i.e. correlation coefficient `r`

would be low in magnitude) it is clear that `r`

would also be negative. The trend becomes more clear as the total number of graduates with a particular degree increases; the median salary associated with that degree tends to decrease. Conversely, degrees with very few graduates can range from low median salaries to very high median salaries.

I'm not an economist but I believe the majority of this is following the simple rule of supply and demand.

Also, never forget human nature: some who graduate, do so with a degree that has a tendency to pay very little. Unfortunately, I believe this is where my personal data point is currently living. But I've "called in the movers" and DQ is helping!

In [11]:

```
#scatter plot of 'Median vs. ShareWomen'
recent_grads.plot(x= 'ShareWomen', y= 'Median', kind= 'scatter', title= 'Median vs. ShareWomen')
```

Out[11]:

Similar to the previous graph, the above graph shows that `r`

would be small but also clearly negative. This is a very discouraging observation. The trend suggests that the majors that are dominated by women tend to have a lower median salary.

So what does it show for the men? Since 'Men' + 'Women' = 'Total', the graph should simply be a flipped version (relection) of the one above. Let's generate that plot after adding a new column `ShareMen`

to our dataframe.

We will use the idea that: `recent_grads['ShareWomen] + recent_grads['ShareMen'] = 1.0`

in order to initialize and populate the new column with values.

In [12]:

```
recent_grads['ShareMen'] = 1 - recent_grads['ShareWomen']
recent_grads.plot(x= 'ShareMen', y= 'Median', kind= 'scatter', title= 'Median vs. ShareMen')
```

Out[12]:

As we expected: it is a reflection of the graph above for women. Can someone remind me...what year is it again?! I wish I had some "time data" to plot in order to confirm this is a dying trend.

Since "money makes the world go 'round", we have decided to plot various observations against it to see what could possibly be influencing it.

In [13]:

```
#scatter plot of 'Median vs. Full_time'
recent_grads.plot(x= 'Full_time', y= 'Median', kind= 'scatter', title= 'Median vs. Full_time')
```

Out[13]:

In [14]:

```
#scatter plot of 'Median vs. Employed'
recent_grads.plot(x= 'Employed', y= 'Median', kind= 'scatter', title= 'Median vs. Employed')
```

Out[14]:

In [15]:

```
#scatter plot of 'Median vs. Unemployed'
recent_grads.plot(x= 'Unemployed', y= 'Median', kind= 'scatter', title= 'Median vs. Unemployed')
```

Out[15]:

The three graphs above, with `Median`

on the y-axis, resemble our first graph of 'Median vs. Total'.

It appears that none of these relationships are particularly strong or very good for prediction, save for one scenario: a lot of people are involved (e.g. a lot of employed people --> low associated salary; a lot of full time eployees --> low associated salary). This matches with what we saw for students graduating with a popular major.

Ok, apart from the misogynistic chromosomal finding earlier, nothing is jumping out as being an influence on salary. What about just finding a job? What does our data say about that? Let's take a look!

In [16]:

```
#scatter plot of 'Employed vs. Total'
recent_grads.plot(x='Total', y='Employed', kind='scatter', title= 'Employed vs. Total')
```

Out[16]:

Now it looks like we've got something real! Clearly there is a strong positive correlation happening here; the more graduates we have with a particular degree, the more people are becoming employed. Sounds great! But beware the lurking variables...they like to lurk, so we need to look! Or, written in pythonic language: correlation != causation --> True.

In [17]:

```
#scatter plot of 'Unemployed vs. Total'
recent_grads.plot(x='Total', y='Unemployed', kind='scatter', title= 'Unemployed vs. Total')
```

Out[17]:

Wait...huh? What just happened? Clearly not as strong of a positive correlation as the graph above it, but it is showing that as the total number of graduates increases, so does unemployment. That sounds bad and would seem to contradict what we thought was happening in the previous scatter plot. Let's take a look at one more before moving on to other visualizations.

In [18]:

```
#scatter plot of 'Employed vs. Unemployed'
recent_grads.plot(x='Unemployed', y='Employed', kind='scatter', title= 'Employed vs. Unemployed')
```

Out[18]:

Ok, this just seems counter-intuitive: as the number of unemployed graduates with a particular degree increases, so too does the number of employed with the same degree?!

But if we take a moment to let that sink in and keeping in mind the one thing I know about economics (supply & demand) it begins to make sense because of the connection between `Total`

and `Employed`

|`Unemployed`

.

One of the reasons popular degrees are so popular is because they have good employment prospects. This creates the relationship we see between `Total`

and `Employed`

.

Supply and demand is then responsible for the relationship we see between `Total`

and `Unemployed`

.

Lastly, it is the transitive property of these metrics (i.e. a=b, b=c, therefore a=c) that is responsible for the pattern we see in 'Employed vs. Unemployed'.

Although they may look like simple bar charts (they aren't), histograms are best used when we want a visual representation of the frequency distribution of our data. They are great for comparing data with large value ranges.

Let's generate some plots and see what we can see!

In [19]:

```
ax = recent_grads['Sample_size'].hist(bins=45, range=(0,450))
ax.set_title('Sample Size Distribution')
print('Number of majors that used a Sample_size less than 450:',
recent_grads[recent_grads['Sample_size'] < 450].shape[0],
'out of', cleaned_data_count, '\n')
```

Because sample sizes vary so dramatically across the data, we have decided to shorten the range to approximately the bottom 10% of sample sizes in favour of graph readability. Although this a dramatic shortening of the data, it still accounts for 141/172 of our dataset.

The graph clearly shows a skew to the right which is a sign that the mean is higher than the median.

It also signals that the vast majority of the data were gathered using small sample sizes.

In [20]:

```
ax = recent_grads['Median'].hist(bins=50, range=(20000,110000))
ax.set_title('Median Salary Distribution')
print('Number of majors included in the graph:',
recent_grads[recent_grads['Median'].between(20000, 110000)].shape[0],
'out of', cleaned_data_count, '\n')
print('Number of majors with salary between $30,000 and $40,000:',
recent_grads[recent_grads['Median'].between(30000, 40000)].shape[0],
'out of', cleaned_data_count, '\n')
```

Again we see a skew to the right due to that one salary above \$100,000. We can also see that the majority of salaries fall between \\$30,000 and \$40,000 (101/172).

In [21]:

```
ax = recent_grads['Employed'].hist(bins=40, range=(0,60000))
ax.set_title('Employed Distribution')
print('Number of majors included in the graph:',
recent_grads[recent_grads['Employed'].between(0, 60000)].shape[0],
'out of', cleaned_data_count, '\n')
```

Again, due to skewness, we have decided to shorten the range in favour of graph readability. As we've seen previously, most majors fall into the lower bins.

In [22]:

```
ax = recent_grads['ShareMen'].hist(bins=25, range=(0,1))
ax.set_title('Men Distribution')
print('Number of majors included in the graph:',
recent_grads[recent_grads['ShareMen'].between(0, 1)].shape[0],
'out of', cleaned_data_count, '\n')
```

At first glance, it appears this distribution is quite even. However, upon closer inspection and modifying the number of `bins`

, we can clearly see that it is not completely so.

In [23]:

```
ax = recent_grads['ShareMen'].hist(bins=2, range=(0,1))
ax.set_title('Men Distribution')
print('Number of majors with Men in range:',
recent_grads[recent_grads['ShareMen'].between(0, 1)].shape[0],
'out of', cleaned_data_count, '\n')
print('Number of majors that are predominantly men:',
recent_grads[recent_grads['ShareMen'].between(0.5, 1)].shape[0],
'out of', cleaned_data_count, '\n')
```

Using only 2 `bins`

clearly shows **less than half** of all majors are predominantly comprised of men.

Specifically: 76 out of 172 majors are predominantly male.

In [24]:

```
ax = recent_grads['ShareWomen'].hist(bins=2, range=(0,1))
ax.set_title('Women Distribution')
print('Number of majors with Women in range:',
recent_grads[recent_grads['ShareWomen'].between(0, 1)].shape[0],
'out of', cleaned_data_count, '\n')
print('Number of majors that are predominantly women:',
recent_grads[recent_grads['ShareWomen'].between(0.5, 1)].shape[0],
'out of', cleaned_data_count, '\n')
```

Inversely, **more than half** of all majors are predominantly comprised of women.

Specifically: 96 out of 172 majors are predominantly female.

A scatter matrix plot combines both scatter plots and histograms into one grid of plots and allows us to explore potential relationships and distributions simultaneously.

A scatter matrix plot consists of `n`

by `n`

plots on a grid, where `n`

is the number of columns. The plots on the diagonal are histograms and the non-diagonal plots are scatter plots.

Scatter matrix plots are good for exploratory data analysis. They are great for when we want to quickly visualize distributions or discover relationships amongst multiple variables. Since this method returns a `n`

by `n`

plot, we must be careful not to pass too large of a dataframe or the output quickly becomes unreadable.

In [25]:

```
scatter_matrix(recent_grads[['Sample_size', 'Median']], figsize=(10,10))
```

Out[25]:

In [26]:

```
scatter_matrix(recent_grads[['Sample_size', 'Median', 'Unemployment_rate']],
figsize=(10,10))
```

Out[26]:

Bar graphs are easy to understand, widely used, and can show changes over time. That gives them an advantage over other graphs that are difficult to read or can only show a single data set.

Most bar charts are plotted vertically, meaning the taller the bar the larger the category, but the data can be plotted horizontally instead meaning the longer the bar the larger the category. This is a good way to present data with long labels that would be hard to display below a vertical bar.

The first two bar graphs below clearly demonstrate this phenomenon. For this reason, we switched to horizontally plotted bar graphs for greater readability.

In [27]:

```
ax = recent_grads[:10].plot.bar(x='Major', y='ShareWomen', legend= False, title= 'ShareWomen vs. Major, top 10')
ax.set_ylabel('ShareWomen')
```

Out[27]:

In [28]:

```
ax = recent_grads[-10:].plot.bar(x='Major', y='ShareWomen', legend= False, title= 'ShareWomen vs. Major, bottom 10')
ax.set_ylabel('ShareWomen')
```

Out[28]:

The above two graphs clearly show a gender bias when it comes to the selection of a major. Of the top ten majors (by median salary), only one is more than 50% female. Whereas of the bottom ten majors we see that every major is comprised of more than 50% female students.

In [29]:

```
ax = recent_grads[:10].plot.barh(x='Major', y='Unemployment_rate', legend= False,
title= 'Unemployment_rate vs. Major, top 10')
ax.set_xlabel('Unemployment_rate')
```

Out[29]:

In [30]:

```
ax = recent_grads[-10:].plot.barh(x='Major', y='Unemployment_rate', legend= False,
title= 'Unemployment_rate vs. Major, bottom 10')
ax.set_xlabel('Unemployment_rate')
```

Out[30]:

The above two graphs show that the unemployement rates for the top ten majors are, on average, lower than those of the bottom ten majors.

In [31]:

```
rg_grouped = recent_grads.groupby(["Major_category"])
ax = rg_grouped['Total'].sum().sort_values().plot(kind= 'barh')
ax.set_xlabel('Total Students')
ax.set_title('Total vs. Major_category')
#rg_grouped['Total'].sum().sort_values(ascending= False).head()
```

Out[31]:

The top five most popular categories for majors are:

`Business`

`Humanities & Liberal Arts`

`Education`

`Engineering`

`Social Science`

In [32]:

```
rg_grouped = recent_grads.groupby(["Major_category"])
ax = rg_grouped[['Women', 'Men']].sum().sort_values('Women').plot(kind= 'barh', figsize= (10, 8))
ax.set_xlabel('Total Students')
ax.set_ylabel('Category of Major, sorted by Women')
ax.set_title('Total vs. Major_category')
```

Out[32]:

In [33]:

```
rg_grouped = recent_grads.groupby(["Major_category"])
ax = rg_grouped[['Women', 'Men']].sum().sort_values('Men').plot(kind= 'barh', figsize= (10, 8))
ax.set_xlabel('Total Students')
ax.set_ylabel('Category of Major, sorted by Men')
ax.set_title('Total vs. Major_category')
```

Out[33]:

The top five categories of majors chosen by women are:

`Business`

`Education`

`Humanities & Liberal Arts`

`Health`

`Psychology & Social Work`

The top five categories of majors chosen by men are:

`Business`

`Engineering`

`Humanities & Liberal Arts`

`Social Science`

`Computer Science & Mathematics`

Plotting data is fun! It also (quite litterally) gives us the ability to look at our data in many different ways. This is of tremendous help when we are trying to extract meaningful insights from a dataset. It's also just a great philosophy in life in general: try to see things from as many different angles as you can!

The plots that we have generated have helped us to realize:

- Men dominate the majors that are associated with higher median salaries
- Women dominate the majors that are associated with lower median salaries
- The majority of salaries are between \$30,000 and \\$40,000
- Unemployment is lower in the top 10 majors (by median salary) than those in the bottom 10 majors
- The most chosen majors by students are:
`Business`

,`Humanities & Liberal Arts`

,`Education`

,`Engineering`

, and`Social Science`

- The most chosen majors by women are:
`Business`

,`Education`

,`Humanities & Liberal Arts`

,`Health`

, and`Psychology & Social Work`

- The most chosen majors by men are:
`Business`

,`Engineering`

,`Humanities & Liberal Arts`

,`Social Science`

, and`Computer Science & Mathematics`