Thierry Parmentelat,
Mohamed Naoufal Mahfoudi,
Thierry Turletti,
and Walid Dabbous,
Inria
Licence CC BY-NC-ND

## Foreword on Jupyter notebooks¶

### How to run a notebook¶

For people not familiar with notebooks, let us simply stress that:

• in order to evaluate a code cell, you first select it (click in it), and then press Shift-Enter - or, equivalent, click the right arrow button in the menubar. The next cell gets selected, so you can essentially run you way through the document by selecting the first cell, and then pressing Shift-Enter until you're done.

• you can also run the whole notebook in a single click - although this is not the recommended technique - from the menubar with Cell → Run All

### Where to run notebooks¶

There are several ways to run a notebook:

1. some public sites offer the ability to host notebooks; the present notebook for example can be run on mybinder.org

2. you can also simply install jupyter on your own machine, and run the notebook from there; please refer to Jupyter's installation instructions for details.

The method that you chose is important, because :

• on the one hand, using a public infrastructure removes the burden of having to install anything, so you can start playing around right away;

• however, your ssh private key is of course unreachable from mybinder or any other public infrastructure, so as far as actually triggering experiments on R2lab is concerned, you will not have the necessary credentials until you go for the second option. You can still use the visualisation tools though, as some pre-gathered data are part of the git repository.

## Purpose¶

R2lab's radiomap is a set of measurements that act as a calibration of the testbed. The goal is to measure and visualize received power at all node locations when a radio signal is sent from any given sender node.

Additionally, that same experiment can be carried out with various settings for the emitted signal, like emission power, Tx rate, channel frequency, and with various antenna setups (single antenna, multiple antennas).

## Workflow¶

End to end experiment involves 2 successive stages:

1. data acquisition : per se, including post-processing (aggregation); this can be carried out with the acquiremap.py python script, or interactively through the first part of the present notebook; this of course requires at the very least a reservation in the testbed, and as pointed out above is unfortunately not possible from a publicly hosted notebook;
2. visualization : interactively, through the second part of this notebook.

For convenience, this git repository also contains a directory datasample that contains one (partial) dataset obtained by running the first-stage acquisition script, so that visualization can be performed right away, as a way to give a quick sense of the results.

In [ ]:
# by default we want to use the pre-shipped data

# Data acquisition with acquiremap.py¶

The àcquiremap.pypython script is designed to expose to the outside 3 levels of scenarios:

• a one_run python function, that runs a complete set of measurements on all nodes, with a specific combination of environment settings (like transmission power, number of antennas, and similar)

• a all_runs python function, that calls one_run with all possible values for the environment settings

• the script itself, when invoked from the command-line, calls all_runs with the environment settings specified as command-line options.

Additionally, all these functions can be instructed to perform node initializations (load a fresh image on all nodes, and turn off unused nodes). When this feature is turned on with in multiple-runs mode (be it from python or from the shell), nodes initialization is performed only once before the the first invokation of one_run.

##### Digression¶

As a side note, we recommend using the following trick:

In [1]:
# for convenience, we use this notebook extension
# that will reload any imported python module
# this is handy if you want to use a tex editor to
# change the code in separate python files while you run the notebook
import nest_asyncio
nest_asyncio.apply()

## one_run¶

Back to data acquisition, the one_run python function performs data collection on a set of nodes (default is all nodes), for one given setting of the environment variables.

In [ ]:
from acquiremap import one_run
In [ ]:

The one_run function implements a scenario in which each node sends a number of ping packets to every other node.

In addition, all nodes run a tcpdump process, and at the end of the run, every pcap file (called fit<N>.pcap at node N) is analyzed locally at node N to retrieve the RSSI values received from each other node on its antenna(s), and the result is stored in file result-N.txt (still on node N).

When all nodes are done, the results are fetched from all nodes and centralized on this laptop in a directory called t{}-r{}-a{}-ch{} containing all the files retrieved from all nodes, i.e., fitN.pcap, result-N.txt for all N nodes.

With:

• t{} identifies Tx power for sender nodes in dBm (e.g. 5 to 14):
• r{} identifies PHY Tx rate used on all nodes (1 to 54 Mbps)
• a{} identifies the antenna mask on all nodes
• ch{} denotes the WiFi channel used for transmission

At that point, the post-processing function processmap.py is invoked to generate intermediate files rssi-<N>.txt (one per node), and eventually one consolidated file RSSI.txt, that will be used to plot the radiomap.

IMPORTANT NOTE: Both Atheros 93xx (with ath9k driver) and Intel 5300 (with iwlwifi driver) a/b/g/n NICs are now supported in these scripts. However, both cards do no have the same features and in particular, the iwlwifi driver limits the number of wireless stations in the same IBSS (parameter IWLAGN_STATION_COUNT) to a dozen! So, if you run the script with the 37 nodes, you will observe strange behaviors... Also the Intel 5300 cards are not allowed to use the 5GHz band in Ad Hoc mode. One good point is that it is possible with these cards to decrease the TX power to 0dBm (the lower bound for Atheros cards is 5dBm).

## all_runs¶

This function simply calls one_run with several combinations (a cartesian product) of environment settings.

In [2]:
from acquiremap import all_runs
In [3]:
help(all_runs)
Help on function all_runs in module acquiremap:

all_runs(wireless_driver, tx_powers, phy_rates, antenna_masks, channels, *args, **kwds)
calls one_run with the cartesian product of
tx_powers, phy_rates, antenna_masks and channels, that are expected to
be lists of strings

All other arguments to one_run may/must be specified as well

Example:
all_runs([5, 14], [1], [1], [1, 40], ...)
will call one_run exactly 4 times

For example, instead of expecting a paramater tx_power that is a simple string, it expects parameter tx_powers that is a list of tx_power strings to consider. So for example

would result in 4 runs of one_run with the 2 possible values for phy_rates multiplied by the 2 possible values for antenna_masks

## Shell interface¶

The command-interface lets you essentially call all_runs directly from your shell

In [4]:
#!./acquiremap.py --help

So by default this simple shell script will run the scenario with all values for Tx power, PHY Tx rate and Antenna configurations. Book R2lab for at least 2 hours to run it.

Any of the one_run or all_runs functions accept a keyword-only parameter named run_name, that specifies the name of the subdirectory where results will be stored. It defaults to myradiomap but you are encouraged to provide your own in order to isolate your results.

Tweak the following cell to run your own data collection campaign:

In [5]:
# set this to True if you want to run your own data collection experiment

use_my_data = True
In [6]:
# if that is the case you need to set these as well

if use_my_data:
# this plays the same role as 'datasample' -- see below
# but to store your own data
# enter here the name of the slice for which you have a valid reservation
In [7]:
# run the data collection
if use_my_data:
13:47:24.056 SCHEDULER 1D + 0R + 152I = 153: Emergency exit upon exception in critical job
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
/var/folders/6n/2x3z_fxs0pxcyh_g3bkjzfyr0000gn/T/ipykernel_26803/2447732765.py in <module>

355                     # record any failure
356                     if not one_run(wireless_driver, tx_power, phy_rate, antenna_mask,
--> 357                                    channel, *args, **kwds):
358                         overall = False
359                     # make sure images will get loaded only once

312     # if not in dry-run mode, let's proceed to the actual experiment
313 #    ok = scheduler.orchestrate(jobs_window=jobs_window)
--> 314     ok = scheduler.orchestrate()
315     # give details if it failed
316     if not ok:

~/miniconda3/envs/r2lab-demos/lib/python3.7/site-packages/asynciojobs/purescheduler.py in run(self, *args, **kwds)
750         """
751         loop = asyncio.get_event_loop()
--> 752         return loop.run_until_complete(self.co_run(*args, **kwds))
753
754     # define the alias for legacy

~/miniconda3/envs/r2lab-demos/lib/python3.7/site-packages/nest_asyncio.py in run_until_complete(self, future)
87                 raise RuntimeError(
88                     'Event loop stopped before Future completed.')
---> 89             return f.result()
90
91     def _run_once(self):

~/miniconda3/envs/r2lab-demos/lib/python3.7/asyncio/futures.py in result(self)
176         self.__log_traceback = False
177         if self._exception is not None:
--> 178             raise self._exception
179         return self._result
180

247                 # We use the send method directly, because coroutines
248                 # don't have __iter__ and __next__ methods.
--> 249                 result = coro.send(None)
250             else:
251                 result = coro.throw(exc)

~/miniconda3/envs/r2lab-demos/lib/python3.7/site-packages/asynciojobs/scheduler.py in co_run(self)
132                 exc = job.raised_exception()
133                 if exc:
--> 134                     raise exc
135         # we should not reach this point
136         raise ValueError("Internal error in Scheduler.co_run()")

247                 # We use the send method directly, because coroutines
248                 # don't have __iter__ and __next__ methods.
--> 249                 result = coro.send(None)
250             else:
251                 result = coro.throw(exc)

~/miniconda3/envs/r2lab-demos/lib/python3.7/site-packages/asynciojobs/window.py in wrapped()
36             await self.queue.put(1)
37             job._running = True                         # pylint: disable=w0212
---> 38             value = await job.co_run()
39             # release slot in the queue
40             await self.queue.get()

~/miniconda3/envs/r2lab-demos/lib/python3.7/site-packages/apssh/sshjob.py in co_run(self)
218                 result = await command.co_run_local(self.node)
219             else:
--> 220                 result = await command.co_run_remote(self.node)
221
222             # has the command failed ?

~/miniconda3/envs/r2lab-demos/lib/python3.7/site-packages/apssh/commands.py in co_run_remote(self, node)
234         self._verbose_message(node, "Run: -> {}".format(command))
235         # need an ssh connection
--> 236         connected = await node.connect_lazy()
237         if not connected:
238             return

~/miniconda3/envs/r2lab-demos/lib/python3.7/site-packages/apssh/sshproxy.py in connect_lazy(self)
252         async with self._connect_lock:
253             if self.conn is None:
--> 254                 await self._connect()
255         return self.conn
256

~/miniconda3/envs/r2lab-demos/lib/python3.7/site-packages/apssh/sshproxy.py in _connect(self)
261         if self.gateway:
262             return await self._connect_tunnel()
--> 263         return await self._connect_direct()
264
265     async def _connect_direct(self):

~/miniconda3/envs/r2lab-demos/lib/python3.7/site-packages/apssh/sshproxy.py in _connect_direct(self)
289                     config=None,
290                 ),
--> 291                 timeout=self.timeout)
292
293     async def _connect_tunnel(self):

440
441         if fut.done():
--> 442             return fut.result()
443         else:
444             fut.remove_done_callback(cb)

~/miniconda3/envs/r2lab-demos/lib/python3.7/asyncio/futures.py in result(self)
176         self.__log_traceback = False
177         if self._exception is not None:
--> 178             raise self._exception
179         return self._result
180

247                 # We use the send method directly, because coroutines
248                 # don't have __iter__ and __next__` methods.
--> 249                 result = coro.send(None)
250             else:
251                 result = coro.throw(exc)

~/miniconda3/envs/r2lab-demos/lib/python3.7/site-packages/asyncssh/connection.py in create_connection(client_factory, host, port, **kwargs)
5893     """
5894
-> 5895     conn = await connect(host, port, client_factory=client_factory, **kwargs)
5896
5897     return conn, conn.get_owner()

~/miniconda3/envs/r2lab-demos/lib/python3.7/site-packages/asyncssh/connection.py in connect(host, port, tunnel, family, flags, local_addr, options, **kwargs)
5632
5633     if not options or kwargs:
-> 5634         options = SSHClientConnectionOptions(options, **kwargs)
5635
5636     return await _connect(host, port, loop, tunnel, family, flags, local_addr,

~/miniconda3/envs/r2lab-demos/lib/python3.7/site-packages/asyncssh/misc.py in __init__(self, options, **kwargs)
210
211         self.kwargs.update(kwargs)
--> 212         self.prepare(**self.kwargs)
213
214     def prepare(self):

TypeError: prepare() got an unexpected keyword argument 'config'

## Data naming scheme¶

The git repository comes with a pre-populated dataset collected by us, in the datasample directory. This contains all the RSSI information to run this visualization.

If you have collected RSSI data, you will be able to

The datasample directory contains a collection of RSSI.txt files in the following subdirectories:

Where

• t5 means an emission power of 5dBm

• r1 means PHY rate=1Mbps

• a1 means 1 single valid antenna - and so 2 values in RSSI.txt

• a3 means 2 valid antennas - and so 3 values in RSSI.txt
• a7 means 3 valid antennas - and so 4 value in RSSI.txt

• ch1 means channel 1, i.e. 2412 MHz frequency

This naming scheme is implemented in a helper function in acquiremap:

In [ ]:
from acquiremap import naming_scheme
In [ ]:
# here's what the naming scheme looks like

## Preparation¶

#### Interactive notebook¶

In [ ]:
# interactive_output is used to refresh
# a visualization based on user input
from ipywidgets import interactive_output, fixed

#### Importing plotly¶

In [ ]:
# plotly is one visualization library
import plotly
import plotly.plotly as py
import plotly.graph_objs as go
In [ ]:
# using plotly in offline mode is a requirement
# in interactive mode - too slow otherwise
import plotly.offline as pyoff
pyoff.init_notebook_mode()

#### Importing a few utilities to retrieve RSSI data from files and convert it to arrays¶

In order to leave low-details out of the notebook, we have chosen to ship such small matters in separate files as python modules, right in this directory.

In [ ]:
# to extract data from a RSSI file
In [ ]:
### R2lab common utilities
# a mapping node -> gridx, gridy
from r2lab import R2labMap
# how to initialize a dataframe
from r2lab import MapDataFrame

#### Importing the dashboard¶

In [ ]:
# this module deals with gory details of ipywidgets
# its only purpose is to to come up with some
# reasonably compact layout for our control buttons
from dashboard import dashboard

## 2D heatmaps using bokeh heatmaps¶

With this in place, we can define a first visualization angle that relies on a 2D representation using bokeh. This approach gives IMHO better results than with plotly - that we will see next - as we are able to alter the figure when a change is made via the dashboard, instead of having to redraw it, which leads to a flickering effect.

In [ ]:
# just in case we want to start running from here
try:
# is this variable defined ?
except:
# if not let's use the data that ships with the git repo

#### Importing bokeh¶

In [ ]:
# again just in case
from ipywidgets import interactive_output, fixed
from dashboard import dashboard
from acquiremap import naming_scheme
In [ ]:
from bokeh.plotting import figure, show
from bokeh.io import output_notebook, push_notebook

from bokeh.models import ColorBar, ColumnDataSource, LinearColorMapper

output_notebook()
In [ ]:
# convert data into x, y and z
In [ ]:
# from https://bokeh.pydata.org/en/latest/docs/gallery/unemployment.html
# this is the colormap from the original NYTimes plot
#
# xxx this colormap does not reflect our use case very well and needs more work
# 0 means very strong power, while -100 means we can hardly notice the signal
#
colors = ["#75968f", "#a5bab7", "#c9d9d3", "#e2e2e2", "#dfccce", "#ddb7b1", "#cc7878", "#933b41", "#550b1d"]
colormapper = LinearColorMapper(palette=colors, low=-100, high=0.)
In [ ]:
# one global static object is good enough
r2labmap = R2labMap()

def init_bokeh():

cds = ColumnDataSource(df)

tools = "hover,save,pan,box_zoom,reset,wheel_zoom"
fig = figure(
plot_width = 900, plot_height=500,
tools = tools, toolbar_location = 'right'
)
# create the rectangles that make the heatmap
fig.rect(x='x', y='y', width=1, height=1,
'transform':colormapper,
},
source = cds)
# show the figure, return handle for updates
handle = show(fig, notebook_handle=True)

# return as a tuple:
# * dataframe is where animation will publish updates
# * handle is an internal bokeh object needed to actually
#   redisplay the changes
return df, cds, handle
In [ ]:
global dataframe, cds, handle
# locate corresponding data file
return

cds.data = cds.from_df(dataframe)
push_notebook(handle)
In [ ]:
# create data and figure
dataframe, cds, handle = init_bokeh()

# interactively update it with an UI
In [ ]:
# debug
# dataframe

## 2D radiomaps using plotly heatmaps¶

Using plotly tends to a simpler code, at the cost of a less pleasant visual effect when the dashboard is used to tweak an experimental setting.

Here we leverage plotly's Heatmap tool:

In [ ]:
# locate corresponding data file
return

heatmap = go.Heatmap( x=X, y=Y, z=Z, text=T,
zmin=-100, zmax=0, zauto=False, opacity=1)
axis = [heatmap]
layout = go.Layout(
title="R2lab Radio-Map: Rx power (in dBm) when fit{:02d} is transmitting<br>from {}"
.format(sender, filename))
figure = go.Figure(data = axis, layout=layout)
pyoff.iplot(figure)

NOTE: you need to play with at least one control for the figure to actually show up

In [ ]:

### 3D surface using ipyvolume¶

In [ ]:
import ipyvolume as ipv
In [ ]:
In [ ]:
from matplotlib import cm

def colormap(Z):
colormap = cm.coolwarm
znorm = Z - Z.min()
znorm /= znorm.ptp()
znorm.min(), znorm.max()
color = colormap(znorm)
# just doing this magic as-is from the above link
return color[...,:3]
In [ ]:
# locate corresponding data file
return

ipv.figure()
ipv.plot_surface(X, Y, Z, color=colormap(Z))
ipv.show()
In [ ]:

### 3D with plotly (broken)¶

Alternatively, we could use a 3D view based on plotly to explore the same results; here is what you would see if you used plotly's Surface object instead.

In [ ]:
# locate corresponding data file
return

surface = go.Surface(x=X, y=Y, z=Z, text=T, zmin=-100, zmax=0,
#connectgaps=False,
)
data = go.Data([surface])
axis = dict(
showbackground=True, # show axis background
backgroundcolor="rgb(204, 204, 204)", # set background color to grey
gridcolor="rgb(255, 255, 255)",       # set grid line color
zerolinecolor="rgb(255, 255, 255)",   # set zero grid line color
)
title = "R2lab Radio-Map: Rx power (in dBm)"\
"when fit{:02d} is transmitting<br>from {}"\
.format(sender, filename)
layout = go.Layout(
autosize=True,
title = title,
scene=go.Scene(
xaxis=go.XAxis(axis), # set x-axis style
yaxis=go.YAxis(axis), # set y-axis style
zaxis=go.ZAxis(axis, title="RSSI (dBm)  ")  # set z-axis style
)
)

figure = go.Figure(data=data, layout=layout)
pyoff.iplot(figure)

NOTE: again, you need to play with at least one control for the figure to actually show.

NOTE2: somehow this seems to have broken recently, it looks like Surface won't accept the zmin and zmax parameters any more..

In [ ]: