#!/usr/bin/env python
# coding: utf-8

# # Comparative geometric mean returns
# 
# We examine economic and financial time series where Holt-Winters 
# is used to forecast one-year ahead. Daily data for bonds, equity, 
# and gold is then analyzed.
# 
# Our focus is on geometric mean returns since they 
# optimally express mean-variance under logarithmic utility. 
# We shall cover portfolio optimization in another notebook.
# 
# [ ] TODO: *use sympy to symbolically derive geometric mean 
# return from the moments of an asset's return distribution.* 
# Our function georet() gives a *numerical* approximation. 

# *Dependencies:*
# 
# - Repository: https://github.com/rsvp/fecon235
# - Python: matplotlib, pandas
#      
# *CHANGE LOG*
# 
#     2016-01-05  MAJOR REWRITE: use pattern from monthly and daily series
#                    for new functions groupget, grouppc, groupgeoret.
#                    Forecast print out replaced by preservable groupholtf.
#                    Dictionary comprehension clarifies code.
#     2016-01-03  Fix issue #2 with v4 and p6 upgrades.
#     2015-05-26  Code revision using template v14.12.21.
#     2014-10-11  Code review. Template 2014-09-28.
#     2014-09-01  First version.

# In[1]:


from fecon235.fecon235 import *


# In[2]:


#  PREAMBLE-p6.15.1223 :: Settings and system details
from __future__ import absolute_import, print_function
system.specs()
pwd = system.getpwd()   # present working directory as variable.
print(" ::  $pwd:", pwd)
#  If a module is modified, automatically reload it:
get_ipython().run_line_magic('load_ext', 'autoreload')
get_ipython().run_line_magic('autoreload', '2')
#       Use 0 to disable this feature.

#  Notebook DISPLAY options:
#      Represent pandas DataFrames as text; not HTML representation:
import pandas as pd
pd.set_option( 'display.notebook_repr_html', False )
#  Beware, for MATH display, use %%latex, NOT the following:
#                   from IPython.display import Math
#                   from IPython.display import Latex
from IPython.display import HTML # useful for snippets
#  e.g. HTML('<iframe src=http://en.mobile.wikipedia.org/?useformat=mobile width=700 height=350></iframe>')
from IPython.display import Image 
#  e.g. Image(filename='holt-winters-equations.png', embed=True) # url= also works
from IPython.display import YouTubeVideo
#  e.g. YouTubeVideo('1j_HxD4iLn8', start='43', width=600, height=400)
from IPython.core import page
get_ipython().set_hook('show_in_pager', page.as_hook(page.display_page), 0)
#  Or equivalently in config file: "InteractiveShell.display_page = True", 
#  which will display results in secondary notebook pager frame in a cell.

#  Generate PLOTS inside notebook, "inline" generates static png:
get_ipython().run_line_magic('matplotlib', 'inline')
#          "notebook" argument allows interactive zoom and resize.


# ## Download data and construct a dataframe
# 
# We retrieve the following data of monthly frequency: **(aggregated) inflation, 
# bonds (zero coupon equivalent of 10-y Treasury), equities (S&P 500), and 
# gold (London PM fix)** -- all denominated in US dollars -- **then lastly, the 
# real trade-weighted USD index (Federal Reserve) and US home prices (per Case-Shiller).** 
# The details for each series is given in their respective notebooks. 
# If the available data has daily frequency, we use the pandas method called 
# "resampling" to induce monthly data (enter "monthly??" in an 
# input cell for more details).
# 
# ATTENTION: *The inclusion of home prices, unfortunately, will create a 3-month lag, 
# due to their release cycle. Since this is a comparative study, 
# the rest of the data will appear somewhat stale, but this 
# section is intended for long-term trends.* 
# Second half of this notebook will examine more responsive daily data.

# In[3]:


#  Specify monthly series of interest as a dictionary:
msdic = {'Infl' : m4infl, 'Zero10' : m4zero10, 'SPX' : m4spx, 
         'XAU' : m4xau, 'USD' : m4usdrtb, 'Homes' : m4homepx }

#  Download data into a dataframe:
msdf = groupget( msdic )
#     "groupget??" at input cell gives function details.


# After downloading the level series, we compute the YoY percentage change 
# for each series. *This will be the a trailing 12-month statistic, 
# thus it is overlapping.*

# In[4]:


#  Construct the mega YoY dataframe:
mega = grouppc( msdf, freq=12 )


# ### Define start time and get stats

# In[5]:


#  Define start time as t0
t0 = '1988'

#  We can easily rerun the rest of this notebook 
#  by specifying another start time, then: Cell > Run All Below


# In[6]:


#  Slice the data:
stats( mega[t0:] )


# There is not much correlation among our assets, 
# except a mild negative between gold XAU and USD. 
# (2015-05-26 at -0.51)
# 
# 
# ## Boxplot of overlapping annual changes
# 
# The boxplot gives us an idea of the range of annual returns, 
# and their persistence due to overlap. Thus trends 
# can be easily discerned.
# 
# It is also a visual aid for the geometric mean returns 
# which is most significant as investment metric. 
# 
# As usual, the *red line* plots the median, but 
# the **red dot** represents the latest point.

# In[7]:


#  Overlapping YoY percentage change, recently:
boxplot(mega[t0:], 'Assets YoYm')
#  where the red dot represents the latest point.


# Red dot outside the mid-range box alerts us to unusual conditions. 
# Attention should also be paid to the extreme value "slash" marks 
# (where outliers are also revealed).
# 

# ## Geometric mean returns on non-overlapping periods
# 
# David E. Shaw, famous for his proprietary hedge fund, remarked that 
# one of the most important equations in finance is the penalization 
# of arithmetic mean by one-half of variance:
# 
# $ g = \mu - (\sigma^2 / 2) $
# 
# which turns out to be our geometric mean return. It is an approximation, 
# by the way, but good enough to maximize, instead of considering 
# intricate mean-variance trade-off. We find it useful also as a metric 
# for economic variables.
# 
# The source code shows us that georet() first gives us 
# the *geometric* mean return, followed by 
# the **arithmetic mean return and volatility**, 
# then finally, the yearly frequency used -- in list format.

# In[8]:


#  How are we computing geometric mean returns?
#  Just add "?" or "??" to variables, procedures, etc.
#  to find out the details, e.g.

get_ipython().run_line_magic('pinfo2', 'georet')


# In[9]:


#  Geometric mean returns, non-overlapping, annualized:
groupgeoret( msdf[t0:], yearly=12 )
    
#  Note that we applied groupgeoret to msdf, not mega.
#  Generally georet requires price levels.
#  groupgeoret is just georet for group dataframes.


# #### Note: the geometric returns do not include interest and dividend payouts for bonds and equities. Some observations in chronological order:
# 
# - 2014-09-01, georet since 2010
#     - Inflation at 1.7% which is below Fed target of 2%.
#     - Total return on bonds, approx 2.52 + 2.33 = 4.85%
#     - Total return on equity, approx. 11.9 + 2 = 13.9% -- very heated.
#     - Gold indecisive about breaking 1260 LTS.
#     - USD though at -0.69%, will strengthen given Draghi wanting weak EUR.
#  
#  
#  - 2014-10-11, georet since 2004
#     - Inflation over ten years is running 2% annually.
#     - Gold dominates over ten years.
#     - Gold recently holds at 1180 triple local bottom.
# 
# 
#  - 2014-10-12, georet since 1988
#     - Inflation in the long-run about 3% annually.
#     - Bond price alone increases 2% annually (excludes interest income).
#     - Gold at 1.73% does not keep up with inflation.
#     - USD at break-even over the long-run.
#     - Home prices have georet of 3.6%.
# 
# 
# - 2015-05-27, georet since 1988
#     - Inflation in the long-run drops 70 bp to about 2.3% annually.
#     - Bond price continues its increase at 2% annually.
#     - Gold at 3.4% reacting more to stronger USD (cf. correlation).
#     - Home prices also have georet of 3.4% (but low 2.6% volatility).
# 
# 
# - 2016-01-03, georet since 1988
#     - Long-run inflation is 2.3% annually (Current Fed target: 2.0%).
#     - Bond price increases at 2% annually (but Fed has just hiked rates!).
#     - Equities at robust 7.6% annually (but ZIRP is finished).
#     - Gold moving along at 2.9% (reflecting horrible 2015 year).
#     - Nominal home prices at steady 3.4% per annum. 

# # Forecasts using Holt-Winters method
# 
# We forecast one-year ahead using the monthly data. 
# Note that the most current infl level is rebased to 1, 
# thus 1.02 would signify 2% increase.

# In[10]:


#  These 12-periods ahead forecasts use default alpha and beta values
#  found to be optimal for a fixed Kalman filter.

groupholtf( msdf, h=12 )


# ### Forecast log for monthly data
# 
# Changing Holt-Winters *alpha* from 0.20 to 0.10 varies the forecast only slightly. The important parameter is *beta* to capture trend effects. Currently we shall rely on default Holt-Winters settings for robustness.
# 
# 
# - 2014-09-01, Twelve-month Forecasts given data through 2014-07-01:
#     - Inflation at 1.44%.
#     - 10-y Bonds price -6.7%, thus rate +75 bp given zero10dur. 
#     - SPX +16.6% to 2280.
#     - Gold tanks from 1286 to 1067. 
#     - USD +1.4% broadly.
# 
# 
# - 2014-10-11, Twelve-month Forecasts given ten-year data, robust HW:
#     - Inflation at 1.9%
#     - Zero10 indicates slight downward pressure on interest rates.
#     - SPX to 2239, but market seems skeptical.
#     - Gold tanks to 1184 (region which we have seen just recently).
#     - USD definitely has an upward bias against all FX, even NZD and AUD.
#     - Home prices looking to increase from \$203K to \$220K 
# 
# 
# - 2015-05-28, Twelve-month Forecasts given data through 2015-03-01, robust HW:
#     - Inflation at 0.5% (which seems dramatic).
#     - Zero10 price increases by 8.92%, thus 10-year rate decreases by 100 bp.
#     - SPX to 2322, but no metric says it's fair valued.
#     - Gold to 1130, which would break support.
#     - USD very strong, up 12% globally (QE-EU started, and possible Grexit).
#     - Home prices looking to increase from \$214K to \$223K
# 
# 
# - 2016-01-03, Twelve-month Forecasts given data through 2015-10-01, robust HW:
#     - Inflation at 1.1% (still below Fed target, see https://git.io/fed)
#     - Bonds and equities, forecasting Zero10 and SPX, will be flat.
#     - Gold to continue down trend, expected to fall to \$941 in a year.
#     - USD 10% higher in light of divergence between Fed hike(s) and ECB QE.
#     - Home prices expected to be flat (but watch mortgage spreads).

# # DAILY DATA, including major FX
# 
# We examine bonds (zero coupon equivalent of 10-y Treasury), 
# equities (SPX), gold (XAU), EURUSD, and USDJPY 
# at higher frequency (daily) for the most recent developments. 
# 
# *Inflation, real trade-weighted USD index, and US home price data 
# have a slow monthly release schedule. 
# And for home price data, there is a three month lag.* 

# In[11]:


#  Specify daily series of interest as a dictionary
#  where key is name, and value is its data code:
dsdic = { 'Zero10' : d4zero10, 'SPX' : d4spx, 'XAU' : d4xau, 
          'EURUSD' : d4eurusd, 'USDJPY' : d4usdjpy }


# In[12]:


#  Download data into a dataframe:
dsdf = groupget( dsdic )


# In[13]:


#  Construct the dega YoY percent dataframe:
dega = grouppc( dsdf, freq=256 )
#                         ^for daily data


# In[14]:


#  Set the start date for daily series:
u0 = '2010-01-01'


# In[15]:


#  Plot overlapping percentage changes:
boxplot( dega[u0:], 'Assets YoYd' )

#  Note that the "last" timestamp will be more
#  recent than for the monthly series.


# Although monthly data is more suitable for making long-term forecasts, 
# daily data is much more sensitive to immediate market perturbations. 
# 
# 2016-01-03  Good example of the foregoing remark is the 
# reaction in the overall market due to the first Fed rate hike 
# in almost a decade on 2015-12-16. ZIRP, zero interest rate program, 
# has been terminated, along with US quantitative easing, 
# thus asset prices must adjust to financing constraints. 
# Note how equities and gold are now below their mid-range boxes.

# In[16]:


stats(dega[u0:])


# - 2015-05-29, Suprisingly, very little correlation between EURUSD and USDJPY: -6%. Gold appears more correlated with USDJPY at -87% than EURUSD at +6%
# 
# - 2016-01-05, Given the latest Fed hike, the correlation to watch is between equities and bonds (-0.69 SPX and Zero10).

# In[17]:


#  What are the latest daily prices?
tail( dsdf )


# In[18]:


#  Geometric mean returns, non-overlapping, annualized:
groupgeoret( dsdf[u0:], yearly=256 )


# ### Closing remarks on daily data
# 
# - 2014-10-11, Really near-term picture is too bright for SPX while XAU looks dark. Sell stocks, and start to accumulate gold.
# 
# 
# - 2015-05-28, XAU georet changed from 2.6% to 1.6%. Zero10 monthly forecast is basically unchanged. Real rate is what matters for gold. USD stronger by 4.8% against both the EUR and JPY.
# 
# 
# - 2016-01-03, XAU georet changed from 1.6% to -0.41%, commodities including oil going through a bear market. Bonds have not sold off despite 2015-12-16 Fed rate hike, probably due to world appetite for USD which is stronger by about 4.3% against EUR and JPY. SPX looks vulnerable given the past maxims about rate hikes, but the Fed is actually still very accomodative.
# 
# [ ] TODO: notebook on r\* the so-called natural interest rate.