GAMS Jupyter Notebooks

GAMS Jupyter Notebooks allow to use notebook technology in combination with GAMS. If you just want to learn GAMS there are probably better ways doing this. Notebooks allow you to combine GAMS and Python. The former works great with well structured data and optimization models, while the latter is very rich in features to retrieve, manipulate, and visualize data that comes in all sort of ways. Combining GAMS and Python in a notebook it is relatively easy to tell an optimization story with text, data, graphs, math, and models.

The GAMS Jupyter Notebook builds on top of the Python 3 kernel. So by default the notebook cells are Python cells. Cells can be turned into GAMS cells, i.e. cells with GAMS syntax, using the Jupyter magic facility (first line in a cell is %%gams). GAMS magic commands enable GAMS support in Python Jupyter notebooks. Beside running GAMS code, it is possible to transfer data between GAMS and Python. In order to enable the GAMS magic commands, it is required to load the extension gams_magic:

In [1]:
%load_ext gams_magic

There are a few other useful command in connection with running GAMS in a Jupyter notebook. Some transformation functions for pandas dataframes useful for exchange with GAMS have been collected in the DataTransform.ipynb. The next cell will execute that notebook and make such data transformation function, e.g. gt_from2dim (see below) available in this notebook. %%capture captures the output from the execution of the notebook and does not clutter your output.

In [2]:
%run ~/share/DataTransform.ipynb

Somehow one output from a cell is sometimes not enough, e.g. if you want to display a couple of tables. The display function allows you to do this but needs to imported. As an example, we display a Python list:

In [3]:
from IPython.display import display
[1, 2, 3]

Running GAMS code

Running GAMS code can be done by using either %gams (line magic) or %%gams (cell magic). While %gams can be used for running a single line of GAMS code, %%gams makes the whole cell a GAMS cell.

In [4]:
%gams set i;
In [5]:
set j;
parameter p(i,j);
parameter p2(i,j);

The GAMS compiler and execution system has been adjusted so one can run a GAMS cell multiple time, even if it contains a declaration or an equation definition, which is normally not possible in the GAMS system. The execution of the next two cells does not create a problem, which mimics the execution, modification, and reexecution of a cell.

In [6]:
set i / peter,paul,mary /, j / A,B,C /;
parameter p2(i,j) / set.i.set.j 1 /;
In [7]:
set i / i1*i5 /, j /j1*j5 /;
parameter p2(i,j) / set.i.set.j 1 /;

You won't see any output from a GAMS cell (unless there is a solve executed in the cell, see below). All output goes to the log and lst file. If you really need to see this you can use magic command %gams_log and %gams_lst to display the content of the log and listing file of the most recent GAMS execution. The next cell displays the content of listing file of the last run GAMS cell or line magic. The -e only display the section of the listing file associated with the execution:

In [8]:
%gams display p2;
%gams_lst -e
E x e c u t i o n

----     11 PARAMETER p2  

            j1          j2          j3          j4          j5

i1       1.000       1.000       1.000       1.000       1.000
i2       1.000       1.000       1.000       1.000       1.000
i3       1.000       1.000       1.000       1.000       1.000
i4       1.000       1.000       1.000       1.000       1.000
i5       1.000       1.000       1.000       1.000       1.000

There is a limit to the execution, modification, and reexecution of GAMS cells. If the type or the dimensionality of a symbol changes, you will need to execute the notebook from scratch and do a controlled reset of the entire GAMS database via %gams_reset. For example, since we declared parameter p2 already over (i,j) we cannot change our mind and redeclare p2 as parameter p2(i,i,j):

%gams parameter p2(i,i,j);

This will give you a compilation error and an exception in the cell execution (uncomment the line in the next cell to do so):

In [9]:
#%gams parameter p2(i,i,j);

With a %gams_reset we can reset the GAMS database and can declare symbols with a different type and domain/dimension. All other things in the GAMS database are gone, too. So we need to redeclare the sets i and j, too. The state of the GAMS database is kept in various files that can easily clutter your directory. The %gams_cleanup call helps you to clean the directory of temporary files. The option -k keeps the most recent GAMS database, hence the %gams_cleanup -k is a save call anywhere in your notebook.

In [10]:
%gams set i,j; parameter p(i,j), p2(i,i,j);
%gams_cleanup -k

Pushing Data from Python to GAMS

%gams_push transfers data from Python to GAMS. Supported data types for pushing data are lists, pandas.DataFrame and numpy arrays:

In [11]:
# Define Python lists with data
i = ['i1', 'i2', 'i3']
j = ['j1', 'j2']
p = [('i1', 'j1', 1.1), ('i1', 'j2', 2.2), ('i2', 'j1', 3.3), ('i2', 'j2', 4.4), ('i3', 'j1', 5.5), ('i3', 'j2', 6.6)]
%gams_push i j p

As mentioned above the execution of a %%gams cell or %gams and %gams_push line magic does not produce output. If one wants to verify that the data ended up in GAMS we can display the symbols in GAMS and output the corresponding part of the listing file:

In [12]:
%gams display i,j,p;
%gams_lst -e
E x e c u t i o n

----     12 SET i  

i1,    i2,    i3

----     12 SET j  

j1,    j2

----     12 PARAMETER p  

            j1          j2

i1       1.100       2.200
i2       3.300       4.400
i3       5.500       6.600

The next cell turns a Python list into a pandas.DataFrame, multiplies the value by 2 and displays the dataframe with IPythons's display. We actually display the transformed p2 (via function gt_pivot2d found in the DataTransformation notebook run at the top of the notebook), so the table looks nicer. Next, we sends the pandas.DataFrame down to GAMS via the %gams_push command. Via the GAMS display and the output of the relevant part of the listing file we see that the %gams_push succeeded:

In [13]:
import pandas as pd
# turn the Python list p into a pandas.Dataframe p2 and send this down to GAMS
pp = pd.DataFrame(p)
# multiply the value by 2:
pp[2] = 2*pp[2]
# display a nicer version of the dataframe:
%gams parameter pp(i,j)
%gams_push pp
%gams display pp;
%gams_lst -e
j1 j2
i1 2.2 4.4
i2 6.6 8.8
i3 11.0 13.2
E x e c u t i o n

----     18 PARAMETER pp  

            j1          j2

i1       2.200       4.400
i2       6.600       8.800
i3      11.000      13.200

When using numpy arrays in order to push data into GAMS, the data is assumed to be dense. The correspondng sets are defined automatically from 1..n, 1..m, etc depending on the data that is pushed.

In [14]:
import numpy as np
data = [[[1.1,-1.1], [2.2,-2.2]], [[3.3,-3.3], [4.4,-4.4]], [[5.5,-5.5], [6.6,-6.6]]]
p3 = np.array(data)
In [15]:
%gams set i, j, k; parameter p3(i,j,k);
%gams_push p3
%gams display i,j,k,p3;
%gams_lst -e
E x e c u t i o n

----     23 SET i  

1,    2,    3

----     23 SET j  

1,    2

----     23 SET k  

1,    2

----     23 PARAMETER p3  3-dim Matrix

              1           2

1.1       1.100      -1.100
1.2       2.200      -2.200
2.1       3.300      -3.300
2.2       4.400      -4.400
3.1       5.500      -5.500
3.2       6.600      -6.600

Pulling Data from GAMS to Python

The line magic %gams_pull transfers data from GAMS to Python in different formats. Supported formats are lists (default), pandas.DataFrame and numpy arrays. The following example pulls the sets i, j, and parameter p3 from GAMS into lists. For multi-dimensional symbols the records become Python tuples. Currently, the renaming functionality %gams_pull gamsSym=pySymbol is not yet supported.

In [16]:
%gams_pull p3 i j
['1', '2', '3']
['1', '2']
[('1', '1', '1', 1.1),
 ('1', '1', '2', -1.1),
 ('1', '2', '1', 2.2),
 ('1', '2', '2', -2.2),
 ('2', '1', '1', 3.3),
 ('2', '1', '2', -3.3),
 ('2', '2', '1', 4.4),
 ('2', '2', '2', -4.4),
 ('3', '1', '1', 5.5),
 ('3', '1', '2', -5.5),
 ('3', '2', '1', 6.6),
 ('3', '2', '2', -6.6)]

The switch -d will populate pandas.DataFrames instead of lists with the GAMS data. The dataframes that are pushed into or pulled from GAMS have a very specific layout. There is a record index and the GAMS domains show up as columns in the dataframe. For parameters, there is an extra value column. For variables and equations we find extra columns level, marginal, lower, upper, and scale. The method head() used in the IPython display provides only the first 5 records of a pandas.DataFrame:

In [17]:
%gams variable x(i) /1.l 1, 2.m 3/;
%gams_pull -d i j p3 x
display(i, j, p3.head(), x)
0 1
1 2
2 3
0 1
1 2
i j k value
0 1 1 1 1.1
1 1 1 2 -1.1
2 1 2 1 2.2
3 1 2 2 -2.2
4 2 1 1 3.3
i level marginal lower upper scale
0 1 1.0 0.0 -inf inf 1.0
1 2 0.0 3.0 -inf inf 1.0

The data transformation functions available from DataTransformations.ipynb help to convert between this format and formats more suitable for display of other transformations in Python. The following lines give a quick overview of the transformation functionality:

In [18]:
%gams parameter r(i,j); r(i,j) = uniformInt(1,10);
%gams_pull -d r
i j value
0 1 1 2.0
1 1 2 9.0
2 2 1 6.0
3 2 2 4.0
4 3 1 3.0
5 3 2 3.0
1 2
1 2.0 9.0
2 6.0 4.0
3 3.0 3.0
i j value
0 1 1 2.0
1 1 2 9.0
2 2 1 6.0
3 2 2 4.0
4 3 1 3.0
5 3 2 3.0

The switch -n will populate numpy arrays instead lists with the GAMS parameters. This format works with parameters only! The GAMS data will be dropped into a dense numpy array:

In [19]:
%gams parameter p4(i,j) / 1.1 1, 2.2 2 /;
%gams_pull -n p4
array([[1., 0.],
       [0., 2.],
       [0., 0.]])

Troubleshooting and Hints

  • Paths to notebooks must not contain whitespaces. A notebook file itself (*.ipynb) can.
  • The directory containing the notebook will clutter up with temporary files. A %gams_cleanup or %gams_cleanup -k if you want to continue working in this session
  • The temporary file are useful for debugging, see below. The naming of the temporary files is not very sophisticated, so it can come to file nameing conflicts if you run two notebooks in the same directory at the same time (in different browser tabs). Create subdirectories and move the notebook into the subdirectories if you run into this problem.
  • As soon as en error occurs while running GAMS code (the notebook exception is a GamsExecption), it can be useful to examine the listing file (*.lst) using %cat path/to/listing.lst or %gams_lst. The path of the listing file can be found in the last line of the output of a failing cell.
In [20]:
%gams Parameter p5(i,j,l);
GamsExceptionExecution                    Traceback (most recent call last)
<ipython-input-20-303883e6f8f6> in <module>()
----> 1 get_ipython().run_line_magic('gams', 'Parameter p5(i,j,l);')

/opt/miniconda3/envs/JupyterHub/lib/python3.6/site-packages/IPython/core/ in run_line_magic(self, magic_name, line, _stack_depth)
   2129                 kwargs['local_ns'] = sys._getframe(stack_depth).f_locals
   2130             with self.builtin_trap:
-> 2131                 result = fn(*args,**kwargs)
   2132             return result

<decorator-gen-131> in gams(self, line, cell)

/opt/miniconda3/envs/JupyterHub/lib/python3.6/site-packages/IPython/core/ in <lambda>(f, *a, **k)
    185     # but it's overkill for just that one bit of state.
    186     def magic_deco(arg):
--> 187         call = lambda f, *a, **k: f(*a, **k)
    189         if callable(arg):

/opt/miniconda3/envs/JupyterHub/lib/python3.6/site-packages/gams_magic/ in gams(self, line, cell)
    391             opt.traceopt = 3
    392             with open(jobName + ".log", "w") as logFile:
--> 393       , checkpoint=self.cp, output=logFile)
    394             solveSummary = self.parseTraceFile(trcFilePath)

/opt/miniconda3/envs/JupyterHub/lib/python3.6/site-packages/gams/ in (execution.c:22303)()

GamsExceptionExecution: GAMS return code not 0 (2), check /home/share/jupyter_examples/gamsJupyter16.lst for more details
In [22]:
#%cat /home/share/jupyter_examples/gamsJupyter16.lst
GAMS 25.2.0  rb031c848 ALFA Released  2Aug18 LEX-LEG x86 64bit/Linux                                                                                                                                                                  09/07/18 17:46:48 Page 26
G e n e r a l   A l g e b r a i c   M o d e l i n g   S y s t e m
C o m p i l a t i o n

  27  Parameter p5(i,j,l);
****                   $120
**** LINE      1 INPUT       /home/share/jupyter_examples/gamsJupyter16.gms
GAMS 25.2.0  rb031c848 ALFA Released  2Aug18 LEX-LEG x86 64bit/Linux                                                                                                                                                                  09/07/18 17:46:48 Page 27
G e n e r a l   A l g e b r a i c   M o d e l i n g   S y s t e m
Error Messages

120  Unknown identifier entered as set

**** 1 ERROR(S)   0 WARNING(S)

COMPILATION TIME     =        0.000 SECONDS      3 MB  25.2.0 rb031c848 LEX-LEG

USER: Eval License GAMS Software GmbH                S180807/0001CO-GEN
      OR2018                                                    DC13509

**** ************* ALFA release
**** GAMS Base Module 25.2.0 rb031c848 ALFA Released 01Sep18 LEG x86 64bit/Linux   
**** ************* ALFA release


Restart    /home/share/jupyter_examples/_gams_py_gcp0.g00
Input      /home/share/jupyter_examples/gamsJupyter16.gms
Output     /home/share/jupyter_examples/gamsJupyter16.lst
Save       /home/share/jupyter_examples/_gams_py_gcp13.g0?