Notebook

TriScale¶

Case Study - Congestion-Control Schemes¶

This notebook presents a use case of the TriScale framework, which details a comparison between 17 congestion-control schemes (listed below). Elements of this use case are described in the TriScale paper (submitted to NSDI'2020).

The evaluation is based on the MahiMahi emulator.
The evaluation data have been collected through the Pantheon framework.
The evaluation uses the calibrated to the real path from AWS California to Mexico (see details in the Data Collection section).
The evaluation considers the single flow scenario only.
The evaluation focuses on the long-term behavior of full-throttle flows - that is, flows whose own throttling/limiting factor is the congestion control.
The evaluation considers two performance dimensions:
- The one-way delay
- The (egress) throughput

List of Schemes
Evaluation Objectives
List of Imports
Download Source Files and Data
Experiment Design
Data Collection
Analysis
Conclusions

List of Schemes¶

The congestion-control schemes included in the evaluation are:

TCP BBR
Copa
TCP Cubic
FillP
FillP-Sheep
Indigo
LEDBAT
PCC-Allegro
PCC-Expr
QUIC Cubic
SCReAM
TaoVA-100x
TCP Vegas
Verus
PCC-Vivace
WebRTC media

Evaluation Objectives¶

[Back to top]

This evaluation aims to compare congestion-control schemes. For a fair comparison, all schemes are tested using the MahiMahi network emulator.

The purpose of this evaluation is not to "crown" one scheme, but rather to illustrate with a concrete example how TriScale can be used and how the framework avoids certain shortcomings in the experiment design and data analysis.

List of Imports¶

[Back to top]

In [1]:

from pathlib import Path
import json
import yaml
import numpy as np
import os
import pandas as pd
import zipfile

from plotly.subplots import make_subplots
import plotly.io as pio
pio.renderers.default = "notebook"

Download Source Files and Data¶

[Back to top]

The entire dataset and source code of this case study is available on Zenodo:

The wget commands below download the required files to reproduce this case study. Beware! the .zip file is 2.7G. Downloading and unzipping might take a while...

In [2]:

# Set `download = True` to download (and extract) the data from this case study
# Eventually, adjust the record_id for the file version you are interested in

# For reproducing the original TriScale paper, set `record_id = 3451418`

download = False
record_id = 3458116 # version 2 (https://doi.org/10.5281/zenodo.3458116)

files= ['triscale.py',
        'helpers.py',
        'triplots.py',
        'UseCase_Pantheon.zip']
if download:
    for file in files:
        print(file)
        url = 'https://zenodo.org/record/'+str(record_id)+'/files/'+file 
        os.system('wget %s' %url)
        if file[-4:] == '.zip':    
            with zipfile.ZipFile(file,"r") as zip_file:
                zip_file.extractall()
        print('Done.')

We now important the custom modules that we just downloaded.

triscale is the main module of TriScale. It contains all the functions from TriScale's API; that is, functions meant to be called by the user.
triplots is a module from TriScale that contains all its plotting functions.
pantheon is a module specific to this case study. It contains helper functions that loop throught the different congestion-control schemes, call TriScale API functions, and create visualizations

In [3]:

import triscale
import triplots
import UseCase_Pantheon.pantheon as pantheon

Experiment Design¶

[Back to top]

One needs to answer a set of questions to fully design an experiment. They relate to three different time scales:

[Run] How long is a run?
[Series] How many runs in a series? When should they run?
[Sequel] How many series?

TriScale assists the user in answering these questions, as described below.

1. How long is a run?¶

[Back to top]

The scenario evaluates the long-term behavior of full-throttle flows - that is, flows whose own throttling/limiting factor is the congestion control. Thus, one must decide on a running time that is long enough to actually capture the long-term behavior.

TriScale implements a convergence test that helps assessing whether a run is indeed long enough to estimate the long-term behavior. Passing this test suggests that the protocol behavior has converged (with a certain level of confidence): the run is then considered long-enough to estimate the long-term behavior.

All networking protocols are differents, and there is no reason to assume that the same run time is necessary for all (certain protocols may be very stable, others require more time to converge). The only way to estimate the required length is to actually test the different protocols, and observe when the runs become long enough to pass the convergence test.

Hence, we perform a preliminary series of tests with varying run time, and test the convergence of the different congestion-control scemes. The first thing to do is to specify the parameters for the convergence test and the performance metrics.

In [4]:

# Parameters for the convergence test
# -> TriScale defaults
convergence = {'expected'  : True,
               'confidence': 95,  # in %
               'tolerance' : 1,   # in %
              }
# Throughput metric -> Median
metric_tput = {'name':'Average Throughput',
               'unit': 'Mbit/s',
               'measure':50,
               'bounds':[0,120],  # expected value range
               'tag':'throughput' # do not change the tag
              }
# Delay metric -> 95th percentile
metric_delay = {'name':'95th perc. of One-way delay (ms)',
                'unit': 'ms',
                'measure':95,
                'bounds':[0,100], # expected value range
                'tag':'delay'     # do not change the tag
               }

These are all the inputs TriScales requires to perform the convergence test and compute the metrics.

The convergence test is implemented in a dedicated convergence_test() function in the helpers.py module.
The analysis_metric() function is the user API, implemented in the triscale.py module. The analysis_metric() calls the convergence_test(), handles the plotting, and produces the textual outputs.
The compute_metric() function is a simple wrapper that loops through all congestion-control scemes and runs and calls the analysis_metric() function. It returns a Pandas dataframe with all convergence test results and metric values.

In [5]:

# Construct the path to the different test results
# (you may need to adjust `result_dir_path` if you did change the default location of file download)
result_dir_path = Path('UseCase_Pantheon/PantheonData/10_20_30_40_50_60s')
result_dir_list = [x for x in result_dir_path.iterdir() if x.is_dir()]

# Meta data file name
meta_data_file  = 'pantheon_metadata.json'

# Config file name and path
config_file     = Path('UseCase_Pantheon/PantheonData/config.yml')

# Output file
out_name        = Path('UseCase_Pantheon/PantheonData/prelim_metrics.csv')

# Metric list
metric_list = [metric_tput, metric_delay]

# Execute convergence tests and compute the metrics
metrics_design = pantheon.compute_metric( result_dir_list, 
                                          meta_data_file, 
                                          convergence,
                                          metric_list,
                                          plot=False,
                                          out_name=out_name,
                                          verbose=False)

Output retrieved from file. Skipping computation.

In [6]:

# Visualize the results: plot the number of converged runs (out of 10)
convergence_results, figure = pantheon.compute_plot_convergence(metrics_design, 
                                                                config_file, 
                                                                'runtime', 
                                                                show=False)
figure.show()

TriScale¶

Case Study - Congestion-Control Schemes¶

Menu¶

List of Schemes¶

Evaluation Objectives¶

List of Imports¶

Download Source Files and Data¶

Experiment Design¶

1. How long is a run?¶