#!/usr/bin/env python
# coding: utf-8

# # Nobody Likes Traffic
# ## Signs It's Slowing Down on I-94

# ## Introduction
# This analysis is attempting to identify factors associated with heavy traffic.
# 
# The dataset is comprised of daily and hourly entries for westbound traffic midway between Minneapolis and St. Paul, the largest and capital cities, respectively, in Minnesota. It was collected from a Department of Transportation station on interstate 94 between October 2012 and September 2018.
# 
# The dataset was created by John Hogue from the UCI Machine Learning Repository.
# https://archive.ics.uci.edu/ml/datasets/Metro+Interstate+Traffic+Volume
# 
# The colors used are the University of Minnesota school colors, gold for graphs and maroon for line plots. Go Gophers!

# In[1]:


# import libraries, open file, set display options
import pandas as pd
import matplotlib.pyplot as plt
get_ipython().run_line_magic('matplotlib', 'inline')
import numpy as np
import pprint
pd.options.display.float_format = '{:20,.4f}'.format
traffic = pd.read_csv("Metro_Interstate_Traffic_Volume.csv")
pd.set_option('display.max_rows', None)
pd.set_option('display.max_columns', None)
pd.set_option('display.width', 1000)
pd.set_option('display.colheader_justify', 'center')
pd.set_option('display.precision', 3)


# ## Examine Data
# A quick look at the beginning and ending of the data set shows nine columns with a mix of  categorical text and numeric information. The source of the data set states that an automatic traffic recorder (ATR) was used to tabulate traffic volume. It is expressed in the number of vehicles per hour.
# * There don't appear to be any missing values.
# * The statistical description looks unusual for rain and snow. There aren't any quartile values listed.
# * The units for temperature don't look familiar and the max rain value seems high as well.

# In[2]:


# display head, tail, and basic info
display(traffic.head())
display(traffic.tail())
display(traffic.info())
display(traffic.describe())


# ## Clean Data
# * Looking at the frequency of no rain and no snow verses some of either shows that the data set statistics for both columns are dominated by hours in which there was no precipitation. Both of these columns report millimeters per hour. The number of hours with some snow seems particularly low for this geographical region.
#     * Creating a column with all positive rain values shows one outlier of 9,800 mm of rain that can be removed.
#     * Creating a similar snow column doesn't reveal any outliers.
# * The temperature column is in degrees Kelvin, which is useful in chemistry or physics but not traffic analysis. Converting to degrees Fahrenheit would be standard in the US.
#     * The temperature column has ten rows with outlier temperature values that can be removed.

# In[3]:


# separate rain column by no rain or some rain
no_rain = traffic[traffic["rain_1h"] == 0]
some_rain = traffic[traffic["rain_1h"] > 0]
print("Hours with no rain: ", len(no_rain))
print("Hours with some rain", len(some_rain))
# separate snow column by no snow or some snow
no_snow = traffic[traffic["snow_1h"] == 0]
some_snow = traffic[traffic["snow_1h"] > 0]
print("Hours with no snow: ", len(no_snow))
print("Hours with some snow: ", len(some_snow))


# In[4]:


# create a some_rain column with any rain measurements greater than 0
traffic["some_rain"] = traffic["rain_1h"][traffic["rain_1h"] > 0]
# basic stats for some_rain
display(traffic["some_rain"].describe())


# In[5]:


# remove row with outlier rain values
traffic = traffic[traffic["rain_1h"]<100]
# basic stats for some_rain
display(traffic["some_rain"].describe())


# In[6]:


# create a some_snow column with any snow measurements greater than 0
traffic["some_snow"] = traffic["snow_1h"][traffic["snow_1h"] > 0]
# basic stats for some_snow
display(traffic["some_snow"].describe())


# In[7]:


# convert temperature column from Kelvin to Fahrenheit
def k_to_f(k_temp):
    f_temp = ((k_temp-273.15)*(9/5)) + 32
    return f_temp

traffic["temp"] = traffic["temp"].map(k_to_f)


# In[8]:


# basic stats for temp column
display(traffic["temp"].describe())


# In[9]:


# # look at outlier temp rows
# cold_days = traffic[traffic["temp"]<-100]
# display(cold_days)


# In[10]:


# remove rows with outlier temperatures
traffic = traffic[traffic["temp"]>-100]
# basic stats for temp column
display(traffic["temp"].describe())


# ## Overall traffic volume
# 
# The previous statistics for traffic volume shows that the minimum value is 0, the maximum value is 7280, and that traffic is usually somewhere in the middle. A histogram shows an interesting distribution. The mean is heavily influenced by how often the traffic is really light and how often the traffic is fairly bad.

# In[11]:


# create histogram of traffic volume for entire data set
traffic["traffic_volume"].plot.hist(color="#FFCC33")
plt.title("Vehicles per Hour")
plt.xticks([1000,3000,5000,7000], [1000,3000,5000,7000])
plt.yticks([2000,4000,6000,8000], [2000,4000,6000,8000])
plt.show()


# ## Aggregate by day or night
# Creating a daytime group (7:00 am to 6:00 pm) and a nighttime group (7:00 pm to 6:00 am) reveals unsurprising differences in traffic volume.
# * The histograms clearly show:
#     * a left skewed normal distribution for daytime traffic, with the mean traffic volume of about 4,600 vehicles per hour.
#     * a right skewed distribution for nighttime traffic. The mean traffic volume is a little less than 1,800 vehicles per hour.
# * This would suggest that including the nighttime data could invalidate any conclusion. That being said, there is some occurrence of high traffic in the evenings. It would be worthwhile to take a look and see if this is just overlap from the day and what the conditions where that caused the increase.

# In[12]:


# transform column to datetime
traffic["date_time"] = pd.to_datetime(traffic["date_time"])
# create date_time series
day_night = traffic["date_time"].dt.hour
# create day traffic series from 0700 to 1800
day_traffic = day_night.between(7, 18)
day_traffic = traffic.loc[day_traffic].copy()
# create night traffic series from 1900 to 0600
night_traffic = day_night.between(19, 24) | day_night.between(0, 6)
night_traffic = traffic.loc[night_traffic].copy()
# verify
print("The number of rows and columns for the day and night series.")
display(day_traffic.shape)
display(night_traffic.shape)


# In[13]:


# create two graphs
plt.figure(figsize=(10,7))
# daytime traffic volume histogram
plt.subplot(2,2,1)
day_traffic["traffic_volume"].plot.hist(color="#FFCC33")
plt.title("Daytime Vehicles per Hour")
plt.xticks([1000,3000,5000,7000], [1000,3000,5000,7000])
plt.xlim(0,7500)
plt.ylim(0,8000)
plt.ylabel("")
# nighttime traffic volume histogram
plt.subplot(2,2,2)
night_traffic["traffic_volume"].plot.hist(color="#FFCC33")
plt.title("Nighttime Vehicles per Hour")
plt.xticks([1000,3000,5000,7000], [1000,3000,5000,7000])
plt.xlim(0,7500)
plt.ylim(0,8000)
plt.ylabel("")
plt.show()


# In[14]:


# basic daytime and nighttime stats
display(day_traffic.describe())
display(night_traffic.describe())


# ## Daytime Group
# * Looking at daytime values initially will provide most of the analysis.
# * The mean traffic volume by month is not that far off the mean traffic volume for all day entries, 4762. The standard deviation between the months is only 190. The graph does show two dips, with one corresponding to winter months in the US. The other looks like it occurs during July, which is unusual because that's during the summer when traffic tends to be high.
#     * A closer look at the annual distribution for the month of July shows an atypical decrease in 2016. A quick google search points to a large highway construction project taking place that year. https://www.mprnews.org/story/2016/07/22/i94-stpaul-shutdown-twin-cities-weekend-road-woes
# * The mean daytime traffic volume does change considerably depending on the day of the week. Saturday shows a considerable dip and Sunday a little more. But there are still plenty of cars making the drive!
# * Looking at the traffic volume for weekdays by hour indicates a peak in the morning (by 7:00 am if not earlier) and again in the afternoon at 4:00 pm.
# * A similar look at weekend hours shows a slow build in the morning with fairly steady traffic until evening.

# In[15]:


# create month column and convert info in date_time column to months 1-12
day_traffic["month"] = day_traffic["date_time"].dt.month
# group daytime by month and get the mean column values for each month
day_traffic_month_group = day_traffic.groupby("month").mean()
# basic stats for daytime month group mean traffic volume
display(day_traffic_month_group["traffic_volume"].describe())
# line plot of daytime month group mean traffic volumes
day_traffic_month_group["traffic_volume"].plot.line(c="#7A0019")
plt.title("Daytime Mean Traffic Volume by Month")
plt.yticks([4400,4600,4800], [4400,4600,4800])
plt.xlabel("")
plt.xticks([1,2,3,4,5,6,7,8,9,10,11,12],["Jan","Feb","Mar","Apr","May","Jun",
                                         "Jul","Aug","Sep","Oct","Nov","Dec"])
plt.show()


# In[16]:


# group July by year
july_months = traffic[traffic["date_time"].dt.month == 7]
july_yearly_group = july_months.groupby(traffic["date_time"].dt.year)
# line plot of July yearly group mean traffic volume
july_yearly_group["traffic_volume"].mean().plot.line(c="#7A0019")
plt.title("July Mean Traffic Volume by Year")
plt.xlabel("")
plt.show()


# In[17]:


# create day_of_week column and convert info in date_time column to days 0-6
day_traffic["day_of_week"] = day_traffic["date_time"].dt.dayofweek
# group daytime by day of week and get the mean column values for each day
day_traffic_day_group = day_traffic.groupby("day_of_week").mean()
# line plot of daytime week of day group mean traffic volumes
day_traffic_day_group["traffic_volume"].plot.line(c="#7A0019")
plt.title("Daytime Mean Traffic Volume by Day")
plt.yticks([3500,4000,4500,5000],[3500,4000,4500,5000])
plt.xlabel("")
plt.xticks([0,1,2,3,4,5,6],["Mon","Tue","Wed","Thur","Fri","Sat","Sun"])
plt.show()


# In[18]:


# create an hour column and convert info in date_time column to hours 0 to 23
day_traffic["hour"] = day_traffic["date_time"].dt.hour
# group daytime into weekdays (4=Friday) and weekends (5=Saturday)
weekdays = day_traffic[day_traffic["day_of_week"] <= 4]
weekends = day_traffic[day_traffic["day_of_week"] >= 5]
# group weekday entries by hour and get the mean column values for each hour
weekdays_hours_group = weekdays.groupby("hour").mean()
# group weekend entries by hour and get the mean column values for each hour
weekends_hours_group = weekends.groupby("hour").mean()
# basic stats for daytime weekday hour group mean traffic volume
# display(weekdays_hours_group["traffic_volume"].describe())
# create two graphs
plt.figure(figsize=(10,6))
# line plot of weekdays hours group mean traffic volumes 
plt.subplot(2,2,1)
weekdays_hours_group["traffic_volume"].plot.line(c="#7A0019")
plt.title("Weekday Traffic Volume by Hour")
plt.ylim(0,6500)
# line plot of weekends hours group mean traffic volumes
plt.subplot(2,2,2)
weekends_hours_group["traffic_volume"].plot.line(c="#7A0019")
plt.title("Weekend Traffic Volulme by Hour")
plt.ylim(0,6500)
plt.show()


# ## Nightime Group

# In[19]:


# line plot for nighttime traffic vollume by hour
night_traffic["hour"] = night_traffic["date_time"].dt.hour
nighttime_hours_group = night_traffic.groupby("hour").mean()
nighttime_hours_group["traffic_volume"].plot.bar(color="#FFCC33")
plt.ylim(0,6000)
plt.title("Nightime Mean Traffic Volume by Hour")
plt.xlabel("")
plt.show()


# ## Weather
# ### Part 1
# There are small correlations between daytime traffic volume and measurable amounts of snow and rain. Neither of these associations would be obvious though.
# * The snow correlation is probably just an artifact due to the low sample number.
# * The subtle relationship between rain and traffic volume appears from the scatter chart to be less causation than simple correlation. It is likely that there are simply a fair number of rainy days in this area.

# In[20]:


# Pearson correlation coefficient for all daytime columns
day_traffic.corr()


# In[21]:


# scatter plot for daytime traffic volume and some snow
plt.scatter(day_traffic["traffic_volume"],day_traffic["some_snow"],c="#FFCC33")
plt.title("Traffic Volume and Snow")
plt.xlabel("Vehicles per Hour")
plt.ylabel("Snow in mm per Hour")
plt.show()


# In[22]:


# scatter plot for daytime traffic volume and some rain
plt.scatter(day_traffic["traffic_volume"], day_traffic["some_rain"],c="#FFCC33")
plt.title("Traffic and Rain")
plt.xlabel("Vehicles per Hour")
plt.ylabel("Rain in mm per Hour")
plt.show()


# ## Weather
# ### Part 2
# * Grouping daytime traffic by the categorical options in the ```weather_main``` column provides an interesting graph but no real insight. The traffic volume seems to be fairly similar across the group.
# * Grouping daytime traffic by the categorical options in the ```weather_description``` doesn't reveal any surprises either. The highest traffic volumes were on days with snow, so maybe we can conclude the Minnesotans are pretty resilient to bad weather!

# In[23]:


# group day traffic by weather main and weather description
weather_main = day_traffic.groupby("weather_main").mean()
weather_description = day_traffic.groupby("weather_description").mean()


# In[24]:


# bar chart for weather main
weather_main["traffic_volume"].plot.barh(color="#FFCC33")
plt.xticks([1000,3000,5000], [1000,3000,5000])
plt.ylabel("")
plt.title("Weather and Daytime Traffic Volume")
plt.show()


# In[25]:


# bar chart for weather description
weather_description["traffic_volume"].plot.barh(figsize=(7,10), color="#FFCC33")
plt.xticks([1000,3000,5000], [1000,3000,5000])
plt.ylabel("")
plt.title("Weather and Daytime Traffic Volume")
plt.show()


# ## Conclusion
# 
# * It's a pretty safe bet that if it's a weekday, especially during rush hour that your going to see a lot of traffic on I-94. On the weekends there is less traffic, and in the evenings even less.
# * There is also a decrease in traffic volume the winter months.
# * Looking at weather conditions showed no reliable correlations. Even clear skies did not show a relationship with high traffic volumes.
# * Nighttime traffic volumes are heavily influenced by hour. Traffic increases dramatically at 5:00 am and 6:00 am, then tapers off after 7:00 pm.
# * It is notable that using mean values makes it difficult to be aware of larger values. Because traffic is such a consequential and daily factor for so many entities, it might be worth reducing the sample to a smaller set of high traffic volume rows.
# * Lastly, because this data set has a large amount of granular weather information, comparing similar hours with good weather and bad weather could be informative.