#!/usr/bin/env python
# coding: utf-8

# # **The Euro on Forex**
# ## **Exploring Twenty Years of Volatility**
# ### History
# The currency exchange market has been around since coins were first used. At the end of the 1800s the development of the gold standard fixed currency values to a certain amount of gold, allowing for an internationally stable monetary system (1). Eventually the amount of gold was not enough to keep up with the demand for currency. After World War II the Bretton Woods Accord set the US Dollar as the global fixed currency. This was an adjustable rate, but still determined by central banks and state actors (2). The US dollar was pegged to gold at the time, but during the 1970s the same problem (i.e. amount of gold vs currency in circulation) occurred with the dollar. In August of 1971 President Richard Nixon unilaterally canceled direct international convertibility of the US dollar to gold, inadvertently beginning a regime of free-floating currencies (3).
# 
# ### Today
# The modern foreign exchange market (Forex, FX, or currency market) is a direct broker to dealer global marketplace that determines the market value and exchange rate of currencies all over the world. It is open 24 hours a day, 5 days a week. Currencies are always traded in pairs, and the value expressed is of one currency relative to another. For example, at the end of the day on 11/19/21 the Euro to US dollar rate was 1.13, or $1.13 equals €1. All of the twenty most traded currencies are free-floating, their value is determined by political conditions, economic factors and market psychology.
# 
# Forex is by far the largest financial market in the world, in 2019 having a daily volume of over $6.6 trillion (4). The Euro to US dollar (EURUSD) is the most traded currency pair, taking about 30% of the multi-billion dollar Forex turnover (5).
# 
# ### Methodology
# This study examines the volatility of the value of the Euro to eight other currencies. It uses the 2019 Triennial Central Bank Survey by the Bank for International Settlements to determine four currencies that represent large portions of the market and four currencies that represent small portions of the market. The highly traded currencies are the US dollar, Japanese yen, UK pound, and Australian dollar. The Israeli shekel, Phillipine peso, Malaysian ringgit, and Romanian leu comprise the group of least traded currencies.
# 
# The data set used is based on daily closing values provided by the European Central Bank and compiled by Daria Chemkaeva. It is updated weekly, the version used for this study was downloaded from Kaggle in November 2021.
# 
# ### Summary
# Text with conclusion.
# <br><br>
# 
# ## **Initial Look**
# The first steps are to import the basic libraries, open the file, and take an initial look at the dataset.
# 
# 

# In[1]:


# import libraries and set display options
import pandas as pd
import pprint

pd.options.display.float_format = '{:20,.4f}'.format
pd.set_option('display.max_rows', None)
pd.set_option('display.max_columns', None)
pd.set_option('display.width', 3000)
pd.set_option('display.colheader_justify', 'center')
pd.set_option('display.precision', 3)
# import json
# s = json.load( open("bmh_matplotlibrc.json") )  #edit path to json file
# matplotlib.rcParams.update(s)

# open file
x_rates = pd.read_csv("euro-daily-hist_1999_2020.csv")

# head, tail, and basic info
display(x_rates.head())
display(x_rates.tail())
display(x_rates.info())


# ## **Initial Look Continued**
# The dataset covers the time period from January 1999 to January 2021. It has 5699 rows and 41 columns. Some of these columns have less than half the number of entries as other columns. In the original data set, cells with missing entries have a hyphen (-). This causes problems with a lot of the functions that will be used. For the time being the hyphens will be converted to the value ```NaN```, which preserves the row of information.
# ## **Data Cleaning**
# The first steps will be:
# * Clean and modify column names
# * Convert to `datetime` data type
# * Sort the values with oldest first and reset the index
# * Clean currency columns
# * Convert exchange rates to floats

# In[2]:


# strip brackets and trailing space from country names
# replace remaining space with underscore
# lower case all column names
x_rates.columns = x_rates.columns.str.replace("[","").str.replace("]","")
x_rates.columns = x_rates.columns.str.rstrip()
x_rates.columns = x_rates.columns.str.replace(" ","_")
x_rates.columns = x_rates.columns.str.lower()

# rename columns
x_rates.rename(columns={"period\\unit:":"date",
                        "chinese_yuan_renminbi":"chinese_yuan",
                        "uk_pound_sterling":"uk_pound"}, inplace=True)

# convert datetime
x_rates["date"] = pd.to_datetime(x_rates["date"])

# resort and reindex
x_rates.sort_values("date", inplace=True)
x_rates.reset_index(drop=True, inplace=True)

# convert hyphens in currency columns to NaN
import numpy as np
x_rates = x_rates.replace("-", np.nan)

# convert exchange rate values to float
x_rates.iloc[:,1:] = x_rates.iloc[:,1:].astype(float)

# set style for cells with centered output
from IPython.core.display import HTML
HTML("""
<style>
.output_png {
    display: table-cell; text-align: center; vertical-align: middle}
</style>
""")

display(x_rates.head())


# ## **Euro to Dollar**
# The change over time in Euro exchange rate can be visualized in a line graph. It and the US dollar are the first currency pair to be examined.

# In[3]:


# create a us dollar to euro dataframe and add 30 day rolling mean column
us_dollar = x_rates[["date","us_dollar"]].copy()
us_dollar["rolling_mean_30"] = us_dollar["us_dollar"].rolling(30).mean()

# line graph for dollar to euro exchange rate
import matplotlib.pyplot as plt
import matplotlib.dates as mdates
get_ipython().run_line_magic('matplotlib', 'inline')
fig, ax = plt.subplots(figsize=(10,3.75))
ax.plot(us_dollar["date"], us_dollar["rolling_mean_30"])
ax.set(xlabel="Year",
       ylabel="US dollar",
       title="US Dollar to Euro Exhcange Rates")
ax.set_xticks(["2000-01-04 00:00:00", "2005-01-04 00:00:00", "2010-01-04 00:00:00",
               "2015-01-04 00:00:00", "2020-01-04 00:00:00"])
ax.set_xticklabels([2000,2005,2010,2015,2020])
plt.show()


# ## **Euro to Dollar Continued**
# The line graph shows that the exchange rate between the Euro and US dollar is pretty variable. There are also a lot of gaps in the graph. These represent points in the data frame where information for a particular day is missing. This is the most accurate representation of the data, but it's not very easy to look at. A quick visual check can illustrate this overlap and then compare the original data frame with one where the missing dates are removed.

# In[4]:


# create nan group and get dates
nan_dollar_group = us_dollar[us_dollar["us_dollar"].isnull()]
nan_days = nan_dollar_group["date"]

# line graph showing dollar to euro showing NaN values with and without highlight
fig, (ax1, ax2) = plt.subplots(nrows=2, ncols=1, figsize=(10,7.5))
ax1.plot(us_dollar["date"], us_dollar["rolling_mean_30"])
ax1.set(title="dataset w/ NaN values")
ax2.plot(us_dollar["date"], us_dollar["rolling_mean_30"])
ax2.set(title="dataset w/ corresponding dates in light blue")
for ax in ax1, ax2:
    ax.set_xticks(["2000-01-04 00:00:00", "2005-01-04 00:00:00", "2010-01-04 00:00:00",
               "2015-01-04 00:00:00", "2020-01-04 00:00:00"])
    ax.set_xticklabels([2000,2005,2010,2015,2020])
for xc in nan_days:
    plt.axvline(x=xc, color="lightcyan")
fig.suptitle("US dollar to Euro exchange rates", fontsize=20)
fig.tight_layout(pad=3.0)
plt.show()

# create a dollar to euro dataframe with NaN dates removed
US_dollar_nonull = x_rates[x_rates["us_dollar"].notna()].copy()
US_dollar_nonull["rolling_mean_30"] = US_dollar_nonull["us_dollar"].rolling(30).mean()

# line graph showing 30 day rolling mean with and without NaN rows
fig, (ax1, ax2) = plt.subplots(nrows=2, ncols=1, figsize=(10,7.5))
ax1.plot(us_dollar["date"], us_dollar["rolling_mean_30"])
ax1.set(title="NaN rows included")
ax2.plot(US_dollar_nonull["date"], US_dollar_nonull["rolling_mean_30"])
ax2.set(title="NaN rows removed")
for ax in ax1, ax2:
    ax.set_xticks(["2000-01-04 00:00:00", "2005-01-04 00:00:00", "2010-01-04 00:00:00",
               "2015-01-04 00:00:00", "2020-01-04 00:00:00"])
    ax.set_xticklabels([2000,2005,2010,2015,2020])
fig.suptitle("US dollar to Euro exchange rates", fontsize=20)
fig.tight_layout(pad=3.0)
plt.show()


# ## **Euro to Dollar Conclusion**
# The third graph shows vertical blue stripes for all dates with missing information. These correspond perfectly with the gaps in the original chart. The final chart, with these rows removed, appears to be good match and is easier to look at. This also demonstrates how the rolling mean handles NaN values within a data frame. In either case the shape of the line is identical where the information exists, therefor both are similarly accurate representations of the available data. All further currency data frames will be handled the same.
# 
# ## **Euro & Traded Pairs**
# ### Euro to Highly Traded Currencies
# The line graphs represent the exchange rates for the Euro to the US dollar, Japanese yen, UK pound, and Australian dollar. These four currencies, in addition to the Euro, are the top five currencies in the Forex market. All five together comprise over 75% of daily trades (4).
# ### Euro to Least Traded Currencies
# The line graphs represent the exchange rates for the Euro to the Israeli shekel, Philippine peso, Malaysian ringgit, and Romanian leu. These four currencies have low trade volumes, they comprise less than 0.4% of daily trades (4).

# In[5]:


# create a list of data frame names
# not sure this is neccessary but trying df = df_list.index(currency) in the framer function doesn't seem to work
# create a list of data frames for each currency with log rate of the exchange rate, 30 day rolling mean, and year
col_names = []
df_dict = {}
for currency in x_rates.columns[1:]:
    df_name = currency
    col_names.append(df_name)
    df = x_rates[["date", currency]].copy()
    df = df[df[currency].notna()]
    df["log_rate"] = np.log(df.iloc[:,1]/df.iloc[:,1].shift()) # getting the log of the exchange rate # double check this is the correct way to get log
    df["rolling_mean_30"] = df[currency].rolling(30).mean()
    df["year"] = df["date"].dt.year
    df_dict[currency] = df

# list of countries for comparisson    
c_list = ["us_dollar", "japanese_yen", "uk_pound", "australian_dollar",
          "israeli_shekel", "philippine_peso", "malaysian_ringgit", "romanian_leu"]

# line graph showing Euro to each high frequency trading pairs
fig, ((ax1, ax2), (ax3, ax4)) = plt.subplots(nrows=2, ncols=2, figsize=(12,8))
fig.tight_layout(pad=3)
for ax, currency in zip((ax1, ax2, ax3, ax4), (df_dict["us_dollar"], df_dict["japanese_yen"], df_dict["uk_pound"], df_dict["australian_dollar"])):
    ax.plot(currency["date"], currency["rolling_mean_30"])
for ax in ax1, ax2, ax3, ax4:
    ax.set_xticks(["2000-01-04 00:00:00", "2005-01-04 00:00:00", "2010-01-04 00:00:00",
               "2015-01-04 00:00:00", "2020-01-04 00:00:00"])
    ax.set_xticklabels([2000,2005,2010,2015,2020])
    ax.set(yticks=[],
           xlabel="Year")
#   ax.set_ylabel(str(currency)) # why not, i can get it to change to a string "currency", but not a variable
ax1.set(ylabel="US dollar")
ax2.set(ylabel="Japanese yen")
ax3.set(ylabel="UK pound")
ax4.set(ylabel="Australian dollar")
fig.suptitle("Euro to High Frequency Pairs", fontsize=20)
plt.show()

# line graph showing Euro to each low frequency trading pairs
fig, ((ax1, ax2), (ax3, ax4)) = plt.subplots(nrows=2, ncols=2, figsize=(12,8))
fig.tight_layout(pad=3)
for ax, currency in zip((ax1, ax2, ax3, ax4), (df_dict["israeli_shekel"], df_dict["philippine_peso"], df_dict["malaysian_ringgit"], df_dict["romanian_leu"])):
    ax.plot(currency["date"], currency["rolling_mean_30"])
for ax in ax1, ax2, ax3, ax4:
    ax.set_xticks(["2000-01-04 00:00:00", "2005-01-04 00:00:00", "2010-01-04 00:00:00",
               "2015-01-04 00:00:00", "2020-01-04 00:00:00"])
    ax.set_xticklabels([2000,2005,2010,2015,2020])
    ax.set(yticks=[],
           xlabel="Year")
#   ax.set_ylabel(str(currency)) # why not, i can get it to change to a string "currency", but not a variable
ax1.set(ylabel="Israeli shekel")
ax2.set(ylabel="Philippine peso")
ax3.set(ylabel="Malaysian ringgit")
ax4.set(ylabel="Romanian leu")
fig.suptitle("Euro to Low Frequency Pairs", fontsize=20)
plt.show()


# ## **Euro & Traded Pairs Conclusion**
# 
# ### High Frequency Pairs
# * The US dollar, Japanese yen, and UK pound all show a tight ratio with the Euro in the early 2000s.
# * Both the Japanese yen and Australian dollar have a period in late 2011 where the exchange ratio gets tight and then rebounds. Is it really at the same time? Are there any other countries with the same pattern and what could the cause have been?
# ### Low Frequency Pairs
# * The Israeli shekel appears to have increased in value and the Romanian leu is definitely loosing value.
# * The shekel and leu appear to be fairly stable, or have low volatility.
# * The Israeli shekel and Malaysian ringgit also show the tight ratio of the early 2000s. This is seen to a lesser extent in the Phillipine peso.
# 
# ## **The Early Struggle and Other Alignments**
# An overlay of all currencies focused on the first decade of the Euro shows it's initial struggle and eventual success.

# In[6]:


x_values1 = df_dict["us_dollar"]["date"]
y_values1 = df_dict["us_dollar"]["rolling_mean_30"]

x_values2 = df_dict["japanese_yen"]["date"]
y_values2 = df_dict["japanese_yen"]["rolling_mean_30"]

x_values3 = df_dict["uk_pound"]["date"]
y_values3 = df_dict["uk_pound"]["rolling_mean_30"]

x_values4 = df_dict["australian_dollar"]["date"]
y_values4 = df_dict["australian_dollar"]["rolling_mean_30"]

fig = plt.figure(figsize=(12,8))
ax = fig.add_subplot(111, label = "1")
ax2 = fig.add_subplot(111, label = "2", frame_on = False)
ax3 = fig.add_subplot(111, label = "3", frame_on = False)
ax4 = fig.add_subplot(111, label = "4", frame_on = False)
# for ax in ax, ax2, ax3, ax4:
#     ax.xaxis.set_major_locator(years) # format the x ticks
#     ax.xaxis.set_major_formatter(years_fmt)
#     fmt_three_years = mdates.MonthLocator(interval = 36) # Major ticks every 3 years
#     ax.xaxis.set_major_locator(fmt_three_years)
#     ax.set(yticks=[],
#            xlabel="Year")
ax.plot(x_values1, y_values1, color = "blue")
ax.set_yticks([])
# ax.set_xlabel("x label 1", color = "C0")
# ax.set_ylabel("y label 1", color = "C0")
# ax.tick_params(axis = 'x', colors = "C0")
# ax.tick_params(axis = 'y', colors = "C0")
ax2.plot(x_values2, y_values2, color = "green")
ax2.set_yticks([])
# ax2.xaxis.tick_top()
# ax2.yaxis.tick_right()
# ax2.set_xlabel('x label 2', color = "C1")
# ax2.set_ylabel('y label 2', color = "C1")
# ax2.xaxis.set_label_position('top')
# ax2.yaxis.set_label_position('right')
# ax2.tick_params(axis = 'x', colors = "C1")
# ax2.tick_params(axis = 'y', colors = "C1")
ax3.plot(x_values3, y_values3, color = "yellow")
ax3.set_yticks([])
ax4.plot(x_values4, y_values4, color = "purple")
ax4.set_yticks([])

plt.show()


# In[7]:


x_values1 = df_dict["israeli_shekel"]["date"]
y_values1 = df_dict["israeli_shekel"]["rolling_mean_30"]

x_values2 = df_dict["philippine_peso"]["date"]
y_values2 = df_dict["philippine_peso"]["rolling_mean_30"]

x_values3 = df_dict["malaysian_ringgit"]["date"]
y_values3 = df_dict["malaysian_ringgit"]["rolling_mean_30"]

x_values4 = df_dict["romanian_leu"]["date"]
y_values4 = df_dict["romanian_leu"]["rolling_mean_30"]

fig = plt.figure(figsize=(12,8))
ax = fig.add_subplot(111, label = "1")
ax2 = fig.add_subplot(111, label = "2", frame_on = False)
ax3 = fig.add_subplot(111, label = "3", frame_on = False)
ax4 = fig.add_subplot(111, label = "4", frame_on = False)
# for ax in ax, ax2, ax3, ax4:
#     ax.xaxis.set_major_locator(years) # format the x ticks
#     ax.xaxis.set_major_formatter(years_fmt)
#     fmt_three_years = mdates.MonthLocator(interval = 36) # Major ticks every 3 years
#     ax.xaxis.set_major_locator(fmt_three_years)
#     ax.set(yticks=[],
#            xlabel="Year")
ax.plot(x_values1, y_values1, color = "blue")
ax.set_yticks([])
# ax.set_xlabel("x label 1", color = "C0")
# ax.set_ylabel("y label 1", color = "C0")
# ax.tick_params(axis = 'x', colors = "C0")
# ax.tick_params(axis = 'y', colors = "C0")
ax2.plot(x_values2, y_values2, color = "green")
ax2.set_yticks([])
# ax2.xaxis.tick_top()
# ax2.yaxis.tick_right()
# ax2.set_xlabel('x label 2', color = "C1")
# ax2.set_ylabel('y label 2', color = "C1")
# ax2.xaxis.set_label_position('top')
# ax2.yaxis.set_label_position('right')
# ax2.tick_params(axis = 'x', colors = "C1")
# ax2.tick_params(axis = 'y', colors = "C1")
ax3.plot(x_values3, y_values3, color = "yellow")
ax3.set_yticks([])
ax4.plot(x_values4, y_values4, color = "purple")
ax4.set_yticks([])

plt.show()


# ## **The Early Struggle and Other Alignments Conclusion**
# ### Stuff
# * Stuff
# * Stuff
# 
# ## **Volatility**
# Stuff

# In[8]:


# function to create df with year and annual volatility for every row
volatility_df_list = []
def volatizer(currency):
    annual_df_list = [currency[currency['year'] == y] for y in currency['year'].unique()] # list of annual dfs
    c_name = currency.columns[1]
    row_dict = {} # dictionary with year:annual_volatility as key:value 
    for frame in annual_df_list:
        year_name = frame.iat[0,4] # first cell of the "year" column, becomes the "year" key for row_dict
        annual_volatility = frame["log_rate"].std()*253**.5 # volatility measured by standard deviation * 253 trading days per year raised to the 0.5 power
        row_dict[year_name] = annual_volatility # dictionary with year:annual_volatility as key:value 
    df = pd.DataFrame.from_dict(row_dict, orient="index", columns=[c_name+"_annual_vol"]) # new df from dictionary indexing on year
    volatility_df_list.append(df)

# apply volatizer to each currency df
for key in df_dict:
    df_dict[key] = volatizer(df_dict[key])
    
# create an annual volatility data frame with all 40 currencies
annual_volatility = pd.concat(volatility_df_list, axis=1)


# In[9]:


display(annual_volatility.head())
annual_volatility["year"] = annual_volatility.index
display(annual_volatility.head())


# In[10]:


# horizontal bar chart showing volatility
plt.barh(y=annual_volatility.iloc[:,40], width=annual_volatility.iloc[:,2])
plt.show()


# ## **Volatility Conclusion**
# * Stuff
# * Stuff
# 
# ## **Volatility Map**
# * Stuff

# In[11]:


# build and animate the map


# ## **Volatility Map Conclusion**
# * Stuff
# * Stuff
# 
# ## **Further Research**
# Stuff
# 
# ## **Conclusion**
# Stuff

# In[12]:


# # need this to correctly get year ticks in DQ server
# years = mdates.YearLocator() # every year
# years_fmt = mdates.DateFormatter('%Y')
# for ax in ax1, ax2, ax3, ax4:
#     ax.xaxis.set_major_locator(years) # format the x ticks
#     ax.xaxis.set_major_formatter(years_fmt)
#     fmt_three_years = mdates.MonthLocator(interval = 36) # Major ticks every 3 years
#     ax.xaxis.set_major_locator(fmt_three_years)
#     ax.set(yticks=[],
#            xlabel="Year")


# In[13]:


from IPython.core.display import HTML
# def css_styling():
#     styles = open("custom.css", "r").read() #or edit path to custom.css
#     return HTML(styles)
# css_styling()
display(HTML("<style>.container { width:70% !important; }</style>"))