Evaluating a Multi-Object Tracking Model on MOT Challenge¶

This notebook provides a framework for evaluating FairMOT on the MOT Challenge dataset.

The MOT Challenge datasets are some of the most common benchmarking datasets for measuring multi-object tracking performance on pedestrian data. They provide distinct datasets every few years; their current offerings include MOT15, MOT16/17, and MOT 19/20. These datasets contain various annotated video sequences, each with different tracking difficulties. Additionally, MOT Challenge provides detections for tracking algorithms without detection components.

The goal of this notebook is to re-produce published results on the MOT challenge using the state-of-the-art FairMOT approach.

Initialization¶

In [1]:

# Ensure edits to libraries are loaded and plotting is shown in the notebook.
%reload_ext autoreload
%autoreload 2
%matplotlib inline

In [2]:

import os
import os.path as osp
import sys
import time

from urllib.parse import urljoin
import torch
import torchvision

sys.path.append("../../")
from utils_cv.common.data import data_path, download, unzip_url
from utils_cv.common.gpu import which_processor, is_windows
from utils_cv.tracking.data import Urls
from utils_cv.tracking.dataset import TrackingDataset
from utils_cv.tracking.model import TrackingLearner

# Change matplotlib backend so that plots are shown for windows
if is_windows():
    plt.switch_backend("TkAgg")

print(f"TorchVision: {torchvision.__version__}")
which_processor()

TorchVision: 0.4.0a0+6b959ee
Torch is using GPU: Tesla K80

The above torchvision command displays your machine's GPUs (if it has any) and the compute that torch/torchvision is using.

Next, we will set some model runtime parameters. Here we will specify the default FairMOT model (dla34) and will evaluate against the MOT17 dataset.

In [9]:

CONF_THRES = 0.4
TRACK_BUFFER = 30

# Downloaded MOT Challendage data path
MOT_ROOT_PATH = "../../data/"
RESULT_ROOT = "./results"
EXP_NAME = "MOT_val_all_dla34"

BASELINE_MODEL = "./models/all_dla34.pth"
MOTCHALLENGE_BASE_URL = "https://motchallenge.net/data/"

# train on the GPU or on the CPU, if a GPU is not available
device = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu")
print(f"Using torch device: {device}")

Using torch device: cuda

Model and Dataset Initialization¶

Now, we will download the MOT17 dataset and save it to MOT_SAVED_PATH. Note: MOT17 is around 5GB and it may take some time to download.

In [4]:

mot_path = urljoin(MOTCHALLENGE_BASE_URL, "MOT17.zip")
mot_train_path = osp.join(MOT_ROOT_PATH, "MOT17", "train")
mot_test_path = osp.join(MOT_ROOT_PATH, "MOT17", "test")
# seqs_str:  various video sequences subfolder names under MOT challenge data
train_seqs = [
    "MOT17-02-SDP",
    "MOT17-04-SDP",
    "MOT17-05-SDP",
    "MOT17-09-SDP",
    "MOT17-10-SDP",
    "MOT17-11-SDP",
    "MOT17-13-SDP",
]
test_seqs = [
    "MOT17-01-SDP",
    "MOT17-03-SDP",
    "MOT17-06-SDP",
    "MOT17-07-SDP",
    "MOT17-08-SDP",
    "MOT17-12-SDP",
    "MOT17-14-SDP",
]

unzip_url(mot_path, dest=MOT_ROOT_PATH, exist_ok=True)
print(f"Training data saved to {mot_train_path}")
print(f"Test data saved to {mot_test_path}")

Training data saved to ../../data/MOT17/train
Test data saved to ../../data/MOT17/test

The pre-trained, baseline FairMOT model - all_dla34.pth- can be downloaded here.

Please upload and save all_dla34.pth to the BASELINE_MODEL path.

The code below initializes and loads the model using the TrackingLearner class.

In [5]:

tracker = TrackingLearner(None, BASELINE_MODEL)

Evaluate on Training Set¶

MOT17 provides ground truth annotations for only the training set, so we will be using the training set for evaluation.

To evaluate FairMOT on this dataset, we take advantage of the py-motmetrics repository.

In [6]:

strsummary = tracker.eval_mot(
    conf_thres=CONF_THRES,
    track_buffer=TRACK_BUFFER,    
    data_root=mot_train_path,
    seqs=train_seqs,
    result_root=RESULT_ROOT,
    exp_name=EXP_NAME,
)
print(strsummary)

loaded ./models/all_dla34.pth, epoch 10
Saved tracking results to ./results/MOT_val_all_dla34/MOT17-02-SDP.txt
Evaluate seq: MOT17-02-SDP
loaded ./models/all_dla34.pth, epoch 10
Saved tracking results to ./results/MOT_val_all_dla34/MOT17-04-SDP.txt
Evaluate seq: MOT17-04-SDP
loaded ./models/all_dla34.pth, epoch 10
Saved tracking results to ./results/MOT_val_all_dla34/MOT17-05-SDP.txt
Evaluate seq: MOT17-05-SDP
loaded ./models/all_dla34.pth, epoch 10
Saved tracking results to ./results/MOT_val_all_dla34/MOT17-09-SDP.txt
Evaluate seq: MOT17-09-SDP
loaded ./models/all_dla34.pth, epoch 10
Saved tracking results to ./results/MOT_val_all_dla34/MOT17-10-SDP.txt
Evaluate seq: MOT17-10-SDP
loaded ./models/all_dla34.pth, epoch 10
Saved tracking results to ./results/MOT_val_all_dla34/MOT17-11-SDP.txt
Evaluate seq: MOT17-11-SDP
loaded ./models/all_dla34.pth, epoch 10
Saved tracking results to ./results/MOT_val_all_dla34/MOT17-13-SDP.txt
Evaluate seq: MOT17-13-SDP
              IDF1   IDP   IDR  Rcll  Prcn  GT  MT  PT ML   FP    FN IDs    FM  MOTA  MOTP IDt IDa IDm
MOT17-02-SDP 63.9% 77.4% 54.4% 68.5% 97.4%  62  22  31  9  344  5855 183   656 65.7% 0.193  98  33  13
MOT17-04-SDP 83.7% 86.1% 81.3% 86.3% 91.4%  83  51  20 12 3868  6531  32   201 78.1% 0.171   5  20   2
MOT17-05-SDP 75.9% 82.9% 70.1% 81.2% 96.0% 133  63  59 11  236  1303  79   207 76.6% 0.199  83  26  40
MOT17-09-SDP 65.4% 71.6% 60.2% 81.1% 96.5%  26  19   7  0  158  1006  52   105 77.2% 0.165  37  12   7
MOT17-10-SDP 65.0% 72.1% 59.2% 78.8% 96.0%  57  32  25  0  418  2721 149   404 74.4% 0.213  89  43  14
MOT17-11-SDP 85.6% 87.9% 83.4% 90.4% 95.2%  75  52  19  4  426   910  38   134 85.4% 0.157  24  19  13
MOT17-13-SDP 77.0% 82.0% 72.5% 83.9% 94.8% 110  74  29  7  534  1878  86   373 78.5% 0.205  72  26  35
OVERALL      76.8% 82.3% 71.9% 82.0% 93.9% 546 313 190 43 5984 20204 619  2080 76.1% 0.182 408 179 124

Evaluate on Test Set¶

For evaluating a model on the testing dataset, the MOT Challenge provides the MOT evaluation server. Here, a user can upload and submit a txt file of prediction results; the service will return metrics. After uploading our results to the MOT17 evaluation server, we can see a MOTA of 68.5 using the 'all_dla34.pth' baseline model.

The reported evaluation results from FairMOT paper with test set are as follows:

Dataset	MOTA	IDF1	IDS	MT	ML	FPS
MOT16	68.7	70.4	953	39.5%	19.0%	25.9
MOT17	67.5	69.8	2868	37.7%	20.8%	25.9

In [7]:

tracker.eval_mot(
    conf_thres=CONF_THRES,
    track_buffer=TRACK_BUFFER,    
    data_root=mot_test_path,
    seqs=test_seqs,
    result_root=RESULT_ROOT,
    exp_name=EXP_NAME,
    run_eval=False,
)

Saved tracking results to ./results/MOT_val_all_dla34/MOT17-01-SDP.txt
Saved tracking results to ./results/MOT_val_all_dla34/MOT17-03-SDP.txt
Saved tracking results to ./results/MOT_val_all_dla34/MOT17-06-SDP.txt
Saved tracking results to ./results/MOT_val_all_dla34/MOT17-07-SDP.txt
Saved tracking results to ./results/MOT_val_all_dla34/MOT17-08-SDP.txt
Saved tracking results to ./results/MOT_val_all_dla34/MOT17-12-SDP.txt
Saved tracking results to ./results/MOT_val_all_dla34/MOT17-14-SDP.txt