Module biogeme.results

Examples of use of each function

This webpage is for programmers who need examples of use of the functions of the class. The examples are designed to illustrate the syntax. They do not correspond to any meaningful model. For examples of models, visit biogeme.epfl.ch.

In [1]:
import datetime
print(datetime.datetime.now())
2021-08-04 16:40:25.097857
In [2]:
import biogeme.version as ver
print(ver.getText())
biogeme 3.2.8 [2021-08-04]
Version entirely written in Python
Home page: http://biogeme.epfl.ch
Submit questions to https://groups.google.com/d/forum/biogeme
Michel Bierlaire, Transport and Mobility Laboratory, Ecole Polytechnique Fédérale de Lausanne (EPFL)

In [3]:
import numpy as np
import pandas as pd
In [4]:
import biogeme.biogeme as bio
import biogeme.database as db
import biogeme.results as res
from biogeme.expressions import Beta, Variable, exp

Definition of a database

In [5]:
df = pd.DataFrame({'Person': [1, 1, 1, 2, 2],
                   'Exclude': [0, 0, 1, 0, 1],
                   'Variable1': [1, 2, 3, 4, 5],
                   'Variable2': [10, 20, 30,40, 50],
                   'Choice': [1, 2, 3, 1, 2],
                   'Av1': [0, 1, 1, 1, 1],
                   'Av2': [1, 1, 1, 1, 1],
                   'Av3': [0, 1, 1, 1, 1]})
myData = db.Database('test', df)

Definition of various expressions

In [6]:
Variable1 = Variable('Variable1')
Variable2 = Variable('Variable2')
beta1 = Beta('beta1', -1.0, -3, 3, 0)
beta2 = Beta('beta2', 2.0, -3, 10, 0)
likelihood = -beta1**2 * Variable1 - exp(beta2 * beta1) * \
    Variable2 - beta2**4
simul = beta1 / Variable1 + beta2 / Variable2
dictOfExpressions = {'loglike':likelihood,
                     'beta1':beta1,
                     'simul':simul}

Creation of the BIOGEME object

In [7]:
myBiogeme = bio.BIOGEME(myData, dictOfExpressions)
myBiogeme.modelName = 'simpleExample'
results = myBiogeme.estimate(bootstrap=10)
print(results)
Results for model simpleExample
Output file (HTML):			simpleExample~05.html
Nbr of parameters:		2
Sample size:			5
Excluded data:			0
Init log likelihood:		-67.08858
Final log likelihood:		-67.06549
Likelihood ratio test (init):		0.04618125
Rho square (init):			0.000344
Rho bar square (init):			-0.0295
Akaike Information Criterion:	138.131
Bayesian Information Criterion:	137.3499
Final gradient norm:		3.900312e-07
beta1          : -1.27[0.115 -11.1 0][0.0137 -92.8 0][0.0103 -123 0]
beta2          : 1.25[0.0848 14.7 0][0.0591 21.1 0][0.0447 27.9 0]
('beta2', 'beta1'):	0.00167	0.171	19.3	0	0.000811	1	55.6	0

Dump results on a file

In [8]:
f = results.writePickle()
print(f)
simpleExample~06.pickle

Results can be imported from a file previously generated

In [9]:
readResults = res.bioResults(pickleFile=f)
print(readResults)
Results for model simpleExample
Output file (HTML):			simpleExample~05.html
Nbr of parameters:		2
Sample size:			5
Excluded data:			0
Init log likelihood:		-67.08858
Final log likelihood:		-67.06549
Likelihood ratio test (init):		0.04618125
Rho square (init):			0.000344
Rho bar square (init):			-0.0295
Akaike Information Criterion:	138.131
Bayesian Information Criterion:	137.3499
Final gradient norm:		3.900312e-07
beta1          : -1.27[0.115 -11.1 0][0.0137 -92.8 0][0.0103 -123 0]
beta2          : 1.25[0.0848 14.7 0][0.0591 21.1 0][0.0447 27.9 0]
('beta2', 'beta1'):	0.00167	0.171	19.3	0	0.000811	1	55.6	0

Results can be formatted in LaTeX

In [10]:
print(readResults.getLaTeX())
%% This file is designed to be included into a LaTeX document
%% See http://www.latex-project.org for information about LaTeX
%% simpleExample - Report from biogeme 3.2.8 [2021-08-04]
%% biogeme 3.2.8 [2021-08-04]
%% Version entirely written in Python
%% Home page: http://biogeme.epfl.ch
%% Submit questions to https://groups.google.com/d/forum/biogeme
%% Michel Bierlaire, Transport and Mobility Laboratory, Ecole Polytechnique Fédérale de Lausanne (EPFL)

%% This file has automatically been generated on 2021-08-04 16:40:25.980281</p>

%%Database name: test

%% General statistics
\section{General statistics}
\begin{tabular}{ll}
Number of estimated parameters & 2 \\
Sample size & 5 \\
Excluded observations & 0 \\
Init log likelihood & -67.08858 \\
Final log likelihood & -67.06549 \\
Likelihood ratio test for the init. model & 0.04618125 \\
Rho-square for the init. model & 0.000344 \\
Rho-square-bar for the init. model & -0.0295 \\
Akaike Information Criterion & 138.131 \\
Bayesian Information Criterion & 137.3499 \\
Final gradient norm & 3.9003E-07 \\
Bootstrapping time & 0:00:00.032105 \\
Nbr of threads & 16 \\
Algorithm & \verb$Newton with trust region for simple bound constraints$ \\
Proportion analytical hessian & \verb$100.0%$ \\
Relative projected gradient & \verb$7.195744e-09$ \\
Relative change & \verb$7.730185e-05$ \\
Number of iterations & \verb$2$ \\
Number of function evaluations & \verb$7$ \\
Number of gradient evaluations & \verb$3$ \\
Number of hessian evaluations & \verb$3$ \\
Cause of termination & \verb$Relative gradient = 7.2e-09 <= 6.1e-06$ \\
Optimization time & \verb$0:00:00.002660$ \\
\end{tabular}

%%Parameter estimates
\section{Parameter estimates}
\begin{tabular}{lrrrrrrrrrr}
\toprule
{} &  Value &  Std err &  t-test &  p-value &  Rob. Std err &  Rob. t-test &  Rob. p-value &  Bootstrap[10] Std err &  Bootstrap t-test &  Bootstrap p-value \\
\midrule
beta1 &  -1.27 &    0.115 &   -11.1 &      0.0 &        0.0137 &        -92.8 &           0.0 &                 0.0103 &            -123.0 &                0.0 \\
beta2 &   1.25 &   0.0848 &    14.7 &      0.0 &        0.0591 &         21.1 &           0.0 &                 0.0447 &              27.9 &                0.0 \\
\bottomrule
\end{tabular}

%%Correlation
\section{Correlation}
\begin{tabular}{lrrrrrrrrrrrr}
\toprule
{} &  Covariance &  Correlation &  t-test &  p-value &  Rob. cov. &  Rob. corr. &  Rob. t-test &  Rob. p-value &  Boot. cov. &  Boot. corr. &  Boot. t-test &  Boot. p-value \\
\midrule
beta2-beta1 &     0.00167 &        0.171 &    19.3 &      0.0 &   0.000811 &         1.0 &         55.6 &           0.0 &    0.000462 &          1.0 &          73.3 &            0.0 \\
\bottomrule
\end{tabular}

Results can be formatted in HTML

In [11]:
print(readResults.getHtml())
<html>
<head>
<script src="http://transp-or.epfl.ch/biogeme/sorttable.js"></script>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>simpleExample - Report from biogeme 3.2.8 [2021-08-04]</title>
<meta name="keywords" content="biogeme, discrete choice, random utility">
<meta name="description" content="Report from biogeme 3.2.8 [2021-08-04]">
<meta name="author" content="{bv.author}">
<style type=text/css>
.biostyle
	{font-size:10.0pt;
	font-weight:400;
	font-style:normal;
	font-family:Courier;}
.boundstyle
	{font-size:10.0pt;
	font-weight:400;
	font-style:normal;
	font-family:Courier;
        color:red}
</style>
</head>
<body bgcolor="#ffffff">
<p>biogeme 3.2.8 [2021-08-04]</p>
<p><a href="https://www.python.org/" target="_blank">Python</a> package</p>
<p>Home page: <a href="http://biogeme.epfl.ch" target="_blank">http://biogeme.epfl.ch</a></p>
<p>Submit questions to <a href="https://groups.google.com/d/forum/biogeme" target="_blank">https://groups.google.com/d/forum/biogeme</a></p>
<p><a href="http://people.epfl.ch/michel.bierlaire">Michel Bierlaire</a>, <a href="http://transp-or.epfl.ch">Transport and Mobility Laboratory</a>, <a href="http://www.epfl.ch">Ecole Polytechnique F&#233;d&#233;rale de Lausanne (EPFL)</a></p>
<p>This file has automatically been generated on 2021-08-04 16:40:26.010062</p>
<table>
<tr class=biostyle><td align=right><strong>Report file</strong>:	</td><td>simpleExample~05.html</td></tr>
<tr class=biostyle><td align=right><strong>Database name</strong>:	</td><td>test</td></tr>
</table>
<h1>Estimation report</h1>
<table border="0">
<tr class=biostyle><td align=right ><strong>Number of estimated parameters</strong>: </td> <td>2</td></tr>
<tr class=biostyle><td align=right ><strong>Sample size</strong>: </td> <td>5</td></tr>
<tr class=biostyle><td align=right ><strong>Excluded observations</strong>: </td> <td>0</td></tr>
<tr class=biostyle><td align=right ><strong>Init log likelihood</strong>: </td> <td>-67.08858</td></tr>
<tr class=biostyle><td align=right ><strong>Final log likelihood</strong>: </td> <td>-67.06549</td></tr>
<tr class=biostyle><td align=right ><strong>Likelihood ratio test for the init. model</strong>: </td> <td>0.04618125</td></tr>
<tr class=biostyle><td align=right ><strong>Rho-square for the init. model</strong>: </td> <td>0.000344</td></tr>
<tr class=biostyle><td align=right ><strong>Rho-square-bar for the init. model</strong>: </td> <td>-0.0295</td></tr>
<tr class=biostyle><td align=right ><strong>Akaike Information Criterion</strong>: </td> <td>138.131</td></tr>
<tr class=biostyle><td align=right ><strong>Bayesian Information Criterion</strong>: </td> <td>137.3499</td></tr>
<tr class=biostyle><td align=right ><strong>Final gradient norm</strong>: </td> <td>3.9003E-07</td></tr>
<tr class=biostyle><td align=right ><strong>Bootstrapping time</strong>: </td> <td>0:00:00.032105</td></tr>
<tr class=biostyle><td align=right ><strong>Nbr of threads</strong>: </td> <td>16</td></tr>
<tr class=biostyle><td align=right ><strong>Algorithm</strong>: </td> <td>Newton with trust region for simple bound constraints</td></tr>
<tr class=biostyle><td align=right ><strong>Proportion analytical hessian</strong>: </td> <td>100.0%</td></tr>
<tr class=biostyle><td align=right ><strong>Relative projected gradient</strong>: </td> <td>7.195744e-09</td></tr>
<tr class=biostyle><td align=right ><strong>Relative change</strong>: </td> <td>7.730184857509093e-05</td></tr>
<tr class=biostyle><td align=right ><strong>Number of iterations</strong>: </td> <td>2</td></tr>
<tr class=biostyle><td align=right ><strong>Number of function evaluations</strong>: </td> <td>7</td></tr>
<tr class=biostyle><td align=right ><strong>Number of gradient evaluations</strong>: </td> <td>3</td></tr>
<tr class=biostyle><td align=right ><strong>Number of hessian evaluations</strong>: </td> <td>3</td></tr>
<tr class=biostyle><td align=right ><strong>Cause of termination</strong>: </td> <td>Relative gradient = 7.2e-09 <= 6.1e-06</td></tr>
<tr class=biostyle><td align=right ><strong>Optimization time</strong>: </td> <td>0:00:00.002660</td></tr>
</table>
<h1>Estimated parameters</h1>
<table border="1">
<tr class=biostyle><th>Name</th><th>Value</th><th>Std err</th><th>t-test</th><th>p-value</th><th>Rob. Std err</th><th>Rob. t-test</th><th>Rob. p-value</th><th>Bootstrap[10] Std err</th><th>Bootstrap t-test</th><th>Bootstrap p-value</th></tr>
<tr class=biostyle><td>beta1</td><td>-1.27</td><td>0.115</td><td>-11.1</td><td>0</td><td>0.0137</td><td>-92.8</td><td>0</td><td>0.0103</td><td>-123</td><td>0</td></tr>
<tr class=biostyle><td>beta2</td><td>1.25</td><td>0.0848</td><td>14.7</td><td>0</td><td>0.0591</td><td>21.1</td><td>0</td><td>0.0447</td><td>27.9</td><td>0</td></tr>
</table>
<h2>Correlation of coefficients</h2>
<table border="1">
<tr class=biostyle><th>Coefficient1</th><th>Coefficient2</th><th>Covariance</th><th>Correlation</th><th>t-test</th><th>p-value</th><th>Rob. cov.</th><th>Rob. corr.</th><th>Rob. t-test</th><th>Rob. p-value</th><th>Boot. cov.</th><th>Boot. corr.</th><th>Boot. t-test</th><th>Boot. p-value</th></tr>
<tr class=biostyle><td>beta2</td><td>beta1</td><td>0.00167</td><td>0.171</td><td>19.3</td><td>0</td><td>0.000811</td><td>1</td><td>55.6</td><td>0</td><td>0.000462</td><td>1</td><td>73.3</td><td>0</td></tr>
</table>
<p>Smallest eigenvalue: 73.054</p>
<p>Largest eigenvalue: 147.802</p>
<p>Condition number: 2.02318</p>
</html>

General statistics, including a suggested format.

In [12]:
statistics = readResults.getGeneralStatistics()
statistics
Out[12]:
{'Number of estimated parameters': (2, ''),
 'Sample size': (5, ''),
 'Excluded observations': (0, ''),
 'Init log likelihood': (-67.08858110354943, '.7g'),
 'Final log likelihood': (-67.06549047946355, '.7g'),
 'Likelihood ratio test for the init. model': (0.046181248171762945, '.7g'),
 'Rho-square for the init. model': (0.00034418113643275294, '.3g'),
 'Rho-square-bar for the init. model': (-0.029467151389933388, '.3g'),
 'Akaike Information Criterion': (138.1309809589271, '.7g'),
 'Bayesian Information Criterion': (137.3498567837953, '.7g'),
 'Final gradient norm': (3.900312424450142e-07, '.4E'),
 'Bootstrapping time': (datetime.timedelta(microseconds=32105), ''),
 'Nbr of threads': (16, '')}

The suggested format can be used as follows

for k, (v, p) in statistics.items(): print(f'{k}:\t{v:{p}}')

This result can be generated directly with the following function

In [13]:
print(results.printGeneralStatistics())
Number of estimated parameters:	2
Sample size:	5
Excluded observations:	0
Init log likelihood:	-67.08858
Final log likelihood:	-67.06549
Likelihood ratio test for the init. model:	0.04618125
Rho-square for the init. model:	0.000344
Rho-square-bar for the init. model:	-0.0295
Akaike Information Criterion:	138.131
Bayesian Information Criterion:	137.3499
Final gradient norm:	3.9003E-07
Bootstrapping time:	0:00:00.032105
Nbr of threads:	16

Estimated parameters as pandas dataframe

In [14]:
readResults.getEstimatedParameters()
Out[14]:
Value Std err t-test p-value Rob. Std err Rob. t-test Rob. p-value Bootstrap[10] Std err Bootstrap t-test Bootstrap p-value
beta1 -1.273264 0.115144 -11.057997 0.0 0.013724 -92.776663 0.0 0.010336 -123.181694 0.0
beta2 1.248769 0.084830 14.720836 0.0 0.059086 21.134794 0.0 0.044724 27.921603 0.0

Correlation results

In [15]:
readResults.getCorrelationResults()
Out[15]:
Covariance Correlation t-test p-value Rob. cov. Rob. corr. Rob. t-test Rob. p-value Boot. cov. Boot. corr. Boot. t-test Boot. p-value
beta2-beta1 0.001671 0.171121 19.280039 0.0 0.000811 1.0 55.597975 0.0 0.000462 0.999972 73.340452 0.0

Obtain the values of the parameters

In [16]:
readResults.getBetaValues()
Out[16]:
{'beta1': -1.2732639874991438, 'beta2': 1.2487688117902658}
In [17]:
readResults.getBetaValues(myBetas=['beta2'])
Out[17]:
{'beta2': 1.2487688117902658}

Variance-covariance matrix (Rao-Cramer)

In [18]:
readResults.getVarCovar()
Out[18]:
beta1 beta2
beta1 0.013258 0.001671
beta2 0.001671 0.007196

Variance-covariance matrix (robust)

In [19]:
readResults.getRobustVarCovar()
Out[19]:
beta1 beta2
beta1 0.000188 0.000811
beta2 0.000811 0.003491

Variance-covaraince matrix (bootstrap)

In [20]:
readResults.getBootstrapVarCovar()
Out[20]:
beta1 beta2
beta1 0.000107 0.000462
beta2 0.000462 0.002

Draws for sensitivity analysis are generated using bootstrapping. Any indicator can be generated by the model for each draw, and its empirical distribution can be investigate .

In [21]:
readResults.getBetasForSensitivityAnalysis(['beta1', 'beta2'],
                                           size=10)
Out[21]:
[{'beta1': -1.269026040524464, 'beta2': 1.2669668838392414},
 {'beta1': -1.261108581300999, 'beta2': 1.300751700209829},
 {'beta1': -1.264979774201305, 'beta2': 1.2842631765105208},
 {'beta1': -1.264979774201305, 'beta2': 1.2842631765105208},
 {'beta1': -1.2925578214686644, 'beta2': 1.1643222175102774},
 {'beta1': -1.257397879951302, 'beta2': 1.31651208100832},
 {'beta1': -1.2649797742013051, 'beta2': 1.2842631765105208},
 {'beta1': -1.2732639874991438, 'beta2': 1.2487688117902658},
 {'beta1': -1.257397879951302, 'beta2': 1.31651208100832},
 {'beta1': -1.261108581300999, 'beta2': 1.300751700209829}]

Results can be produced in the ALOGIT F12 format

In [22]:
readResults.getF12()
Out[22]:
'                                                                  simpleExample\nFrom biogeme 3.2.8                                      2021-08-04 16:40:26  \nEND\n   0      beta1 F  -1.273263987499e+00 +1.372396830767e-02\n   0      beta2 F  +1.248768811790e+00 +5.908592227624e-02\n  -1\n       5                  0                   0 -6.706549047946e+01\n   0   0  2021-08-04 16:40:26\n 100000\n'

Miscellaneous functions

Calculation of the $p$-value

In [23]:
res.calcPValue(1.96)
Out[23]:
0.04999579029644097