In this guided project, we'll explore how using the pandas plotting functionality with some standard visualizations.
We'll be working with a dataset on the job outcomes of students who graduated from college between 2010 and 2012. The original data on job outcomes was released by American Community Survey
, which conducts surveys and aggregates the data. FiveThirtyEight
cleaned the dataset and released it on their Github repo.
The name of the file is recent-grads.csv
. Each row in the dataset represents a different major in college and contains information on gender diversity, employment rates, median salaries, and more. Here are some of the columns in the dataset:
Rank
- Rank by median earnings (the dataset is ordered by this column).
Major_code
- Major code.
Major
- Major description.
Major_category
- Category of major.
Total
- Total number of people with major.
Sample_size
- Sample size (unweighted) of full-time.
Men
- Male graduates.
Women
- Female graduates.
ShareWomen
- Women as share of total.
Employed
- Number employed.
Median
- Median salary of full-time, year-round workers.
Low_wage_jobs
- Number in low-wage service jobs.
Full_time
- Number employed 35 hours or more.
Part_time
- Number employed less than 35 hours.
Using visualizations, we can start to explore questions from the dataset like:
Do students in more popular majors make more money?
How many majors are predominantly male? Predominantly female?
Which category of majors have the most students?
We'll explore how to do these and more while primarily working in pandas. Before we start creating data visualizations, let's import the libraries we need and remove rows containing null values.
import pandas as pd
import matplotlib.pyplot as plt
recent_grads = pd.read_csv('recent-grads.csv')
recent_grads.head()
Rank | Major_code | Major | Total | Men | Women | Major_category | ShareWomen | Sample_size | Employed | Full_time | Part_time | Full_time_year_round | Unemployed | Unemployment_rate | Median | P25th | P75th | College_jobs | Non_college_jobs | Low_wage_jobs | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
0 | 1 | 2419 | PETROLEUM ENGINEERING | 2339.0 | 2057.0 | 282.0 | Engineering | 0.120564 | 36 | 1976 | 1849 | 270 | 1207 | 37 | 0.018381 | 110000 | 95000 | 125000 | 1534 | 364 | 193 |
1 | 2 | 2416 | MINING AND MINERAL ENGINEERING | 756.0 | 679.0 | 77.0 | Engineering | 0.101852 | 7 | 640 | 556 | 170 | 388 | 85 | 0.117241 | 75000 | 55000 | 90000 | 350 | 257 | 50 |
2 | 3 | 2415 | METALLURGICAL ENGINEERING | 856.0 | 725.0 | 131.0 | Engineering | 0.153037 | 3 | 648 | 558 | 133 | 340 | 16 | 0.024096 | 73000 | 50000 | 105000 | 456 | 176 | 0 |
3 | 4 | 2417 | NAVAL ARCHITECTURE AND MARINE ENGINEERING | 1258.0 | 1123.0 | 135.0 | Engineering | 0.107313 | 16 | 758 | 1069 | 150 | 692 | 40 | 0.050125 | 70000 | 43000 | 80000 | 529 | 102 | 0 |
4 | 5 | 2405 | CHEMICAL ENGINEERING | 32260.0 | 21239.0 | 11021.0 | Engineering | 0.341631 | 289 | 25694 | 23170 | 5180 | 16697 | 1672 | 0.061098 | 65000 | 50000 | 75000 | 18314 | 4440 | 972 |
recent_grads.tail()
Rank | Major_code | Major | Total | Men | Women | Major_category | ShareWomen | Sample_size | Employed | Full_time | Part_time | Full_time_year_round | Unemployed | Unemployment_rate | Median | P25th | P75th | College_jobs | Non_college_jobs | Low_wage_jobs | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
168 | 169 | 3609 | ZOOLOGY | 8409.0 | 3050.0 | 5359.0 | Biology & Life Science | 0.637293 | 47 | 6259 | 5043 | 2190 | 3602 | 304 | 0.046320 | 26000 | 20000 | 39000 | 2771 | 2947 | 743 |
169 | 170 | 5201 | EDUCATIONAL PSYCHOLOGY | 2854.0 | 522.0 | 2332.0 | Psychology & Social Work | 0.817099 | 7 | 2125 | 1848 | 572 | 1211 | 148 | 0.065112 | 25000 | 24000 | 34000 | 1488 | 615 | 82 |
170 | 171 | 5202 | CLINICAL PSYCHOLOGY | 2838.0 | 568.0 | 2270.0 | Psychology & Social Work | 0.799859 | 13 | 2101 | 1724 | 648 | 1293 | 368 | 0.149048 | 25000 | 25000 | 40000 | 986 | 870 | 622 |
171 | 172 | 5203 | COUNSELING PSYCHOLOGY | 4626.0 | 931.0 | 3695.0 | Psychology & Social Work | 0.798746 | 21 | 3777 | 3154 | 965 | 2738 | 214 | 0.053621 | 23400 | 19200 | 26000 | 2403 | 1245 | 308 |
172 | 173 | 3501 | LIBRARY SCIENCE | 1098.0 | 134.0 | 964.0 | Education | 0.877960 | 2 | 742 | 593 | 237 | 410 | 87 | 0.104946 | 22000 | 20000 | 22000 | 288 | 338 | 192 |
recent_grads.describe()
Rank | Major_code | Total | Men | Women | ShareWomen | Sample_size | Employed | Full_time | Part_time | Full_time_year_round | Unemployed | Unemployment_rate | Median | P25th | P75th | College_jobs | Non_college_jobs | Low_wage_jobs | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
count | 173.000000 | 173.000000 | 172.000000 | 172.000000 | 172.000000 | 172.000000 | 173.000000 | 173.000000 | 173.000000 | 173.000000 | 173.000000 | 173.000000 | 173.000000 | 173.000000 | 173.000000 | 173.000000 | 173.000000 | 173.000000 | 173.000000 |
mean | 87.000000 | 3879.815029 | 39370.081395 | 16723.406977 | 22646.674419 | 0.522223 | 356.080925 | 31192.763006 | 26029.306358 | 8832.398844 | 19694.427746 | 2416.329480 | 0.068191 | 40151.445087 | 29501.445087 | 51494.219653 | 12322.635838 | 13284.497110 | 3859.017341 |
std | 50.084928 | 1687.753140 | 63483.491009 | 28122.433474 | 41057.330740 | 0.231205 | 618.361022 | 50675.002241 | 42869.655092 | 14648.179473 | 33160.941514 | 4112.803148 | 0.030331 | 11470.181802 | 9166.005235 | 14906.279740 | 21299.868863 | 23789.655363 | 6944.998579 |
min | 1.000000 | 1100.000000 | 124.000000 | 119.000000 | 0.000000 | 0.000000 | 2.000000 | 0.000000 | 111.000000 | 0.000000 | 111.000000 | 0.000000 | 0.000000 | 22000.000000 | 18500.000000 | 22000.000000 | 0.000000 | 0.000000 | 0.000000 |
25% | 44.000000 | 2403.000000 | 4549.750000 | 2177.500000 | 1778.250000 | 0.336026 | 39.000000 | 3608.000000 | 3154.000000 | 1030.000000 | 2453.000000 | 304.000000 | 0.050306 | 33000.000000 | 24000.000000 | 42000.000000 | 1675.000000 | 1591.000000 | 340.000000 |
50% | 87.000000 | 3608.000000 | 15104.000000 | 5434.000000 | 8386.500000 | 0.534024 | 130.000000 | 11797.000000 | 10048.000000 | 3299.000000 | 7413.000000 | 893.000000 | 0.067961 | 36000.000000 | 27000.000000 | 47000.000000 | 4390.000000 | 4595.000000 | 1231.000000 |
75% | 130.000000 | 5503.000000 | 38909.750000 | 14631.000000 | 22553.750000 | 0.703299 | 338.000000 | 31433.000000 | 25147.000000 | 9948.000000 | 16891.000000 | 2393.000000 | 0.087557 | 45000.000000 | 33000.000000 | 60000.000000 | 14444.000000 | 11783.000000 | 3466.000000 |
max | 173.000000 | 6403.000000 | 393735.000000 | 173809.000000 | 307087.000000 | 0.968954 | 4212.000000 | 307933.000000 | 251540.000000 | 115172.000000 | 199897.000000 | 28169.000000 | 0.177226 | 110000.000000 | 95000.000000 | 125000.000000 | 151643.000000 | 148395.000000 | 48207.000000 |
raw_data_count = recent_grads.shape
raw_data_count
(173, 21)
There are currently 173 rows with 21 columns in it.
Looking into the values we will drop rows with missing values. Matplotlib expects that columns of values we pass in have matching lengths and missing values will cause matplotlib to throw errors.
recent_grads = recent_grads.dropna()
cleaned_data_count = recent_grads.shape
cleaned_data_count
(172, 21)
After dropping null values we can see only one row was dropped which means only one row contained missing values.
Now, we will work with scatter plots to visualize this cleaned data.
With scatter plot we will explore the following questions:
For this, we will explore the following relations:
Sample_size
and Median
Sample_size
and Unemployment_rate
Full_time
and Median
ShareWomen
and Unemployment_rate
Men
and Median
Women
and Median
# Scatter plot showing Sample size vs Median
recent_grads.plot(x='Sample_size', y='Median', kind='scatter', title='Sample Size vs. Median', figsize=(5,8))
<matplotlib.axes._subplots.AxesSubplot at 0x7f62bc890518>
# Scatter plot showing Sample size vs Unemployment rate
recent_grads.plot(x='Sample_size', y='Unemployment_rate', kind='scatter', title='Sample Size vs. Unemployment rate', figsize=(5,8))
<matplotlib.axes._subplots.AxesSubplot at 0x7f62bc7d6b38>
# Scatter plot showing Full time vs Median
recent_grads.plot(x='Full_time', y='Median', kind='scatter', title='Full Time vs. Median', figsize=(6,8))
<matplotlib.axes._subplots.AxesSubplot at 0x7f62bc300860>
# Scatter plot showing Share Women vs Unemployment rate
recent_grads.plot(x='ShareWomen', y='Unemployment_rate', kind='scatter', title='Share Women vs. Unemployment_rate', figsize=(5,8))
<matplotlib.axes._subplots.AxesSubplot at 0x7f62bc2d9b00>
# Scatter plot showing Men vs Median
recent_grads.plot(x='Men', y='Median', kind='scatter', title='Men vs. Median', figsize=(6,8))
<matplotlib.axes._subplots.AxesSubplot at 0x7f62bc292b00>
# Scatter plot showing Women vs Median
recent_grads.plot(x='Women', y='Median', kind='scatter', title='Women vs. Median', figsize=(5,8))
<matplotlib.axes._subplots.AxesSubplot at 0x7f62bc1ac908>
Conclusions:
Do students in more popular majors make more money?
Full Time vs Median
scatterplot does not show any strong trends that full-time workers with a popular major have a higher median income than other FT workers with unpopular majors.
Additionally, the Total vs Median
scatterplot also does not indicate any significant bias towards having a higher median income for popular majors.
Do students that majored in subjects that were majority female make more money?
Share of Women vs Median
scatterplot shows a downward trend indicating that a major with a larger percentage of women means less median income. Could this mean women are more prone to making less money in the majors that are popular among them?
Is there any link between the number of full-time employees and median salary?
Full Time vs Median
scatterplot does not show any correlation between the two datasets.
Now, we would hop on to histogram for exploring the distribution of values in a column.
recent_grads['Sample_size'].plot(kind='hist')
<matplotlib.axes._subplots.AxesSubplot at 0x7f62bc16e550>
In the above Sample Size histogram, one can see that the majority of the sample sizes for the different majors are under a 1,000 count.
recent_grads['Sample_size'].hist(bins=25, range=(0,5000))
<matplotlib.axes._subplots.AxesSubplot at 0x7f62bc113be0>
It looks like the ranges between 25K and 50K are most common. Let's zoom in further on this part.
recent_grads['Median'].hist(bins=30, range=(20000,110000))
<matplotlib.axes._subplots.AxesSubplot at 0x7f62bc029e80>
recent_grads['Full_time'].hist(bins=10, range=(0,250000))
<matplotlib.axes._subplots.AxesSubplot at 0x7f62bbf6d978>
It looks like very common (median) salaries are 35-36K and 40-41K.
Let's looking further in the relations between (1)total number of majors (2)the median salary (3) percentage of women by creating a scatter-matrix for these three.
recent_grads['ShareWomen'].hist(bins=15,range=(0.0,1.1))
<matplotlib.axes._subplots.AxesSubplot at 0x7f62bbef2eb8>
recent_grads['Unemployment_rate'].hist(bins=30,range=(0.000,0.180))
<matplotlib.axes._subplots.AxesSubplot at 0x7f62bbf12438>
recent_grads['Men'].hist(bins=20, range=(0,175000))
<matplotlib.axes._subplots.AxesSubplot at 0x7f62bbf34908>
recent_grads['Women'].hist(bins=20, range=(0,300000))
<matplotlib.axes._subplots.AxesSubplot at 0x7f62bbdfdc88>
What percent of majors are predominantly male? Predominantly female?
There are a total of 172 majors. Looking at the histogram above for column 'ShareWomen', it's clear that more than 50% of majors had predominantly more women than men. By applying a Boolean filter for all majors that had greater than 50% of women, we found out there are a total of 96 female-dominated majors. On the other hand, there are 76 majors that have predominantly men.
Scatter Matrix plot
from pandas.plotting import scatter_matrix
scatter_matrix(recent_grads[['Sample_size', 'Median']], figsize=(10,10))
array([[<matplotlib.axes._subplots.AxesSubplot object at 0x7f62bc141208>, <matplotlib.axes._subplots.AxesSubplot object at 0x7f62bbf3bd68>], [<matplotlib.axes._subplots.AxesSubplot object at 0x7f62bbd34f60>, <matplotlib.axes._subplots.AxesSubplot object at 0x7f62bbc52550>]], dtype=object)
For the above Median vs Sample Size
scatter matrix, there is no noticeable correlations between the two columns. Both separate histograms are skewed to the left, meaning most of the sample sizes were lower than 1,000 and most of the average median incomes were below 40,000 USD.
scatter_matrix(recent_grads[['Sample_size', 'Median', 'Unemployment_rate']], figsize=(10,10))
array([[<matplotlib.axes._subplots.AxesSubplot object at 0x7f62bbb7edd8>, <matplotlib.axes._subplots.AxesSubplot object at 0x7f62bbb096d8>, <matplotlib.axes._subplots.AxesSubplot object at 0x7f62bbabd940>], [<matplotlib.axes._subplots.AxesSubplot object at 0x7f62bba70ba8>, <matplotlib.axes._subplots.AxesSubplot object at 0x7f62bba25e10>, <matplotlib.axes._subplots.AxesSubplot object at 0x7f62bb9e60b8>], [<matplotlib.axes._subplots.AxesSubplot object at 0x7f62bba19320>, <matplotlib.axes._subplots.AxesSubplot object at 0x7f62bb9cc550>, <matplotlib.axes._subplots.AxesSubplot object at 0x7f62bb9cc5c0>]], dtype=object)
In the scatter matrix plots above, there was no obvious trends or correlation among the three columns.
Bar plot
Now let's also create bar-plots for the first 10 and for the last 10 majors in the list to see what we can learn from that.
recent_grads[:10].plot.barh(x='Major', y='ShareWomen', legend=False, color=['black', 'red', 'green', 'blue', 'cyan','black', 'red', 'green', 'blue', 'cyan'])
recent_grads[-10:].plot.barh(x='Major', y='ShareWomen', legend=False, color=['black', 'red', 'green', 'blue', 'cyan','black', 'red', 'green', 'blue'])
<matplotlib.axes._subplots.AxesSubplot at 0x7f62b9f81780>
Given that the order of the majors are based on descending order of the median income for each major, it is clear on the barplots that women tend to dominate majors that are on the lower spectrum of income, while males tend to dominate the higher spectrum of income.
Conclusion:
Let's wrap-up with sharing some of the observations that we made above:
The most popular majors are not those that result in the highest (median) salaries. Majors with higher percentages of women (which is actually a majority) tend to have lower (median) salaries. Clearly, we've only be scratching the surface. The dataset contains a wealth of interesting data to explore. Possibly to be continued at another occassion!