Notebook

Licensed under the MIT License.

Fine-tuning an Action Recognition Model on HMDB51¶

In this notebook, we demonstrate how to get the SOTA results for the HMDB51 human action dataset using the R(2+1)D model.

Initialization¶

Import all the functions we need

In [1]:

import sys
sys.path.append("../../")
import time
import os
import numpy as np
import torch
import torch.cuda as cuda
import torch.nn as nn
import torchvision
import requests

from utils_cv.action_recognition.model import VideoLearner 
from utils_cv.action_recognition.dataset import VideoRecord, VideoDataset
from utils_cv.action_recognition.data import Urls
from utils_cv.common.gpu import system_info
from utils_cv.common.data import data_path, download

system_info()

3.7.6 (default, Jan  8 2020, 19:59:22) 
[GCC 7.3.0] 

PyTorch 1.2.0 

Torch-vision 0.4.0a0 

Available devices:
0: Tesla K80

This shows your machine's GPUs (if it has any) and the computing device torch/torchvision is using.

In [2]:

# Ensure edits to libraries are loaded and plotting is shown in the notebook.
%reload_ext autoreload
%autoreload 2
%matplotlib inline

Next, set some model runtime parameters.

Using the paramters below, one can reproduce published results on the HMDB-51 dataset. Video-clip accuracy of close to 80% can be obtained using as input to the DNN 32 consecutive frames (MODEL_INPUT_SIZE) and running 48 training epochs. Even using only 8 epochs the accuracy already reaches 75%. Inference and training speed can be reduced by a factor of 3-4 by setting MODEL_INPUT_SIZE=8 and using higher batch size, however at a significant loss in accuracy.

In [3]:

# Number of consecutive frames used as input to the DNN. Recommended: 32 for high accuracy, 8 for inference speed.
MODEL_INPUT_SIZE = 32

# Batch size. Reduce if running out of memory.
BATCH_SIZE = 4

# Number of training epochs
EPOCHS = 8

# Learning rate
LR = 0.0001

Downloading HMDB51¶

Since we're trying to train our model ont he hmdb51 dataset, we'll need to download it. You find the link to download it here. Once downloaded, set VIDEO_DIR to the location of the videos.

In [4]:

DATA_ROOT = os.path.join(str(data_path()), "misc/hmdb51")
VIDEO_DIR = os.path.join(DATA_ROOT, "videos")

For this notebook, we'll try to reproduce the SOTA results on HMDB51 using a specific train/test split known simply as "split1". Download the split and specify where the download location.

In [5]:

TRAIN_SPLIT = download(Urls.hmdb_train_split_1, os.path.join(DATA_ROOT, "hmdb51_train_split_1.txt"))
TEST_SPLIT = download(Urls.hmdb_test_split_1, os.path.join(DATA_ROOT, "hmdb51_test_split_1.txt"))

Prepare HMDB51 using the Video Dataset¶

We can load the downloaded HMBD51 dataset into our VideoDataset class by simply passing it the location of the folder. We demonstrated how this is done in the previous 01_training_introduction.ipynb notebook.

However, because we're trying to reproduce the results in the paper, we'll have to use a specific split of the training and testing data as defined by the TRAIN_SPLIT and TEST_SPLIT files. This is refered to in the paper as "split1". Conviniently, the VideoDataset class can take in a train_split_file and a test_split_file as parameters, so we can pass it directly into the VideoDataset.

For these parameters to work, the file must come in the following format:

<action_type>/<video_name> <label_number>

Here's what the top part hmdb51_vid_train_split_1.txt looks like.

fencing/AHF_Rapier_fencing_f_cm_np2_le_bad_2 13
situp/Timed_situps_1_minute_situp_f_nm_np1_ri_med_1 38
brush_hair/Aussie_Brunette_Brushing_Hair_II_brush_hair_u_nm_np2_le_goo_0 0
swing_baseball/BaseballHitinSlowMotion_swing_baseball_f_nm_np1_le_bad_1 43
clap/#20_Rhythm_clap_u_nm_np1_le_goo_3 4
laugh/Skype_Laughter_by_Hungary_laugh_h_nm_np2_fr_goo_1 23
ride_horse/ChampagneRunattheParkHT_ride_horse_f_cm_np1_ri_med_1 31
swing_baseball/practicingmybaseballswing2009_swing_baseball_f_cm_np1_fr_med_5 43
...

Notice that the extension (".mp3", ".avi") is passed in separately.

In [6]:

data = VideoDataset(
    VIDEO_DIR,
    train_split_file=TRAIN_SPLIT,
    test_split_file=TEST_SPLIT,
    batch_size=BATCH_SIZE,
    sample_length=MODEL_INPUT_SIZE,
    video_ext="avi",
)

Here's what the train/test split ends up looking like.

In [7]:

print(
    f"Training dataset: {len(data.train_ds)} | Training DataLoader: {data.train_dl} \
    \nTesting dataset: {len(data.test_ds)} | Testing DataLoader: {data.test_dl}"
)

Training dataset: 3570 | Training DataLoader: <torch.utils.data.dataloader.DataLoader object at 0x7f569f13f990>     
Testing dataset: 1530 | Testing DataLoader: <torch.utils.data.dataloader.DataLoader object at 0x7f569957a510>

Now that the data is loaded, we can take a quick look at some samples.

In [8]:

data.show_batch(rows=2)

/home/jiata/anaconda3/envs/cv/lib/python3.7/site-packages/torch/nn/functional.py:2479: UserWarning: Default upsampling behavior when mode=bilinear is changed to align_corners=False since 0.4.0. Please specify align_corners=True if the old behavior is desired. See the documentation of nn.Upsample for details.
  "See the documentation of nn.Upsample for details.".format(mode))

<Figure size 432x288 with 0 Axes>

Lets also take a look at the (default) parameters that we're using:

In [9]:

print(f"""\
sample_length: {data.sample_length}
sample_step: {data.sample_step}
temporal_jitter: {data.temporal_jitter}
temporal_jitter_step: {data.temporal_jitter_step}
random_shift: {data.random_shift}
""")

sample_length: 32
sample_step: 1
temporal_jitter: True
temporal_jitter_step: 2
random_shift: True

Setup the Video Learner¶

In [10]:

learner = VideoLearner(data, num_classes=51)

Loading r2plus1d_34_32_ig65m model

Using cache found in /home/jiata/.cache/torch/hub/moabitcoin_ig65m-pytorch_master

In [11]:

learner.model.fc

Out[11]:

Linear(in_features=512, out_features=51, bias=True)

In [12]:

learner.fit(lr=LR, epochs=EPOCHS)

Params to learn:
	full network
Epoch 1 =========================================================
lr=[0.0001]
 Phase train: batch 10 of 893
 Phase train: batch 20 of 893
 Phase train: batch 30 of 893
 Phase train: batch 40 of 893
 Phase train: batch 50 of 893
 Phase train: batch 60 of 893
 Phase train: batch 70 of 893
 Phase train: batch 80 of 893
 Phase train: batch 90 of 893
 Phase train: batch 100 of 893
 Phase train: batch 110 of 893
 Phase train: batch 120 of 893
 Phase train: batch 130 of 893
 Phase train: batch 140 of 893
 Phase train: batch 150 of 893
 Phase train: batch 160 of 893
 Phase train: batch 170 of 893
 Phase train: batch 180 of 893
 Phase train: batch 190 of 893
 Phase train: batch 200 of 893
 Phase train: batch 210 of 893
 Phase train: batch 220 of 893
 Phase train: batch 230 of 893
 Phase train: batch 240 of 893
 Phase train: batch 250 of 893
 Phase train: batch 260 of 893
 Phase train: batch 270 of 893
 Phase train: batch 280 of 893
 Phase train: batch 290 of 893
 Phase train: batch 300 of 893
 Phase train: batch 310 of 893
 Phase train: batch 320 of 893
 Phase train: batch 330 of 893
 Phase train: batch 340 of 893
 Phase train: batch 350 of 893
 Phase train: batch 360 of 893
 Phase train: batch 370 of 893
 Phase train: batch 380 of 893
 Phase train: batch 390 of 893
 Phase train: batch 400 of 893
 Phase train: batch 410 of 893
 Phase train: batch 420 of 893
 Phase train: batch 430 of 893
 Phase train: batch 440 of 893
 Phase train: batch 450 of 893
 Phase train: batch 460 of 893
 Phase train: batch 470 of 893
 Phase train: batch 480 of 893
 Phase train: batch 490 of 893
 Phase train: batch 500 of 893
 Phase train: batch 510 of 893
 Phase train: batch 520 of 893
 Phase train: batch 530 of 893
 Phase train: batch 540 of 893
 Phase train: batch 550 of 893
 Phase train: batch 560 of 893
 Phase train: batch 570 of 893
 Phase train: batch 580 of 893
 Phase train: batch 590 of 893
 Phase train: batch 600 of 893
 Phase train: batch 610 of 893
 Phase train: batch 620 of 893
 Phase train: batch 630 of 893
 Phase train: batch 640 of 893
 Phase train: batch 650 of 893
 Phase train: batch 660 of 893
 Phase train: batch 670 of 893
 Phase train: batch 680 of 893
 Phase train: batch 690 of 893
 Phase train: batch 700 of 893
 Phase train: batch 710 of 893
 Phase train: batch 720 of 893
 Phase train: batch 730 of 893
 Phase train: batch 740 of 893
 Phase train: batch 750 of 893
 Phase train: batch 760 of 893
 Phase train: batch 770 of 893
 Phase train: batch 780 of 893
 Phase train: batch 790 of 893
 Phase train: batch 800 of 893
 Phase train: batch 810 of 893
 Phase train: batch 820 of 893
 Phase train: batch 830 of 893
 Phase train: batch 840 of 893
 Phase train: batch 850 of 893
 Phase train: batch 860 of 893
 Phase train: batch 870 of 893
 Phase train: batch 880 of 893
 Phase train: batch 890 of 893
train took 3670.92 sec | loss = 3.3312 | top1_acc = 30.7283  | top5_acc = 57.1989
 Phase valid: batch 10 of 383
 Phase valid: batch 20 of 383
 Phase valid: batch 30 of 383
 Phase valid: batch 40 of 383
 Phase valid: batch 50 of 383
 Phase valid: batch 60 of 383
 Phase valid: batch 70 of 383
 Phase valid: batch 80 of 383
 Phase valid: batch 90 of 383
 Phase valid: batch 100 of 383
 Phase valid: batch 110 of 383
 Phase valid: batch 120 of 383
 Phase valid: batch 130 of 383
 Phase valid: batch 140 of 383
 Phase valid: batch 150 of 383
 Phase valid: batch 160 of 383
 Phase valid: batch 170 of 383
 Phase valid: batch 180 of 383
 Phase valid: batch 190 of 383
 Phase valid: batch 200 of 383
 Phase valid: batch 210 of 383
 Phase valid: batch 220 of 383
 Phase valid: batch 230 of 383
 Phase valid: batch 240 of 383
 Phase valid: batch 250 of 383
 Phase valid: batch 260 of 383
 Phase valid: batch 270 of 383
 Phase valid: batch 280 of 383
 Phase valid: batch 290 of 383
 Phase valid: batch 300 of 383
 Phase valid: batch 310 of 383
 Phase valid: batch 320 of 383
 Phase valid: batch 330 of 383
 Phase valid: batch 340 of 383
 Phase valid: batch 350 of 383
 Phase valid: batch 360 of 383
 Phase valid: batch 370 of 383
 Phase valid: batch 380 of 383
valid took 493.40 sec | loss = 2.1920 | top1_acc = 54.3137  | top5_acc = 78.5621
Epoch 2 =========================================================
lr=[0.0001]
 Phase train: batch 10 of 893
 Phase train: batch 20 of 893
 Phase train: batch 30 of 893
 Phase train: batch 40 of 893
 Phase train: batch 50 of 893
 Phase train: batch 60 of 893
 Phase train: batch 70 of 893
 Phase train: batch 80 of 893
 Phase train: batch 90 of 893
 Phase train: batch 100 of 893
 Phase train: batch 110 of 893
 Phase train: batch 120 of 893
 Phase train: batch 130 of 893
 Phase train: batch 140 of 893
 Phase train: batch 150 of 893
 Phase train: batch 160 of 893
 Phase train: batch 170 of 893
 Phase train: batch 180 of 893
 Phase train: batch 190 of 893
 Phase train: batch 200 of 893
 Phase train: batch 210 of 893
 Phase train: batch 220 of 893
 Phase train: batch 230 of 893
 Phase train: batch 240 of 893
 Phase train: batch 250 of 893
 Phase train: batch 260 of 893
 Phase train: batch 270 of 893
 Phase train: batch 280 of 893
 Phase train: batch 290 of 893
 Phase train: batch 300 of 893
 Phase train: batch 310 of 893
 Phase train: batch 320 of 893
 Phase train: batch 330 of 893
 Phase train: batch 340 of 893
 Phase train: batch 350 of 893
 Phase train: batch 360 of 893
 Phase train: batch 370 of 893
 Phase train: batch 380 of 893
 Phase train: batch 390 of 893
 Phase train: batch 400 of 893
 Phase train: batch 410 of 893
 Phase train: batch 420 of 893
 Phase train: batch 430 of 893
 Phase train: batch 440 of 893
 Phase train: batch 450 of 893
 Phase train: batch 460 of 893
 Phase train: batch 470 of 893
 Phase train: batch 480 of 893
 Phase train: batch 490 of 893
 Phase train: batch 500 of 893
 Phase train: batch 510 of 893
 Phase train: batch 520 of 893
 Phase train: batch 530 of 893
 Phase train: batch 540 of 893
 Phase train: batch 550 of 893
 Phase train: batch 560 of 893
 Phase train: batch 570 of 893
 Phase train: batch 580 of 893
 Phase train: batch 590 of 893
 Phase train: batch 600 of 893
 Phase train: batch 610 of 893
 Phase train: batch 620 of 893
 Phase train: batch 630 of 893
 Phase train: batch 640 of 893
 Phase train: batch 650 of 893
 Phase train: batch 660 of 893
 Phase train: batch 670 of 893
 Phase train: batch 680 of 893
 Phase train: batch 690 of 893
 Phase train: batch 700 of 893
 Phase train: batch 710 of 893
 Phase train: batch 720 of 893
 Phase train: batch 730 of 893
 Phase train: batch 740 of 893
 Phase train: batch 750 of 893
 Phase train: batch 760 of 893
 Phase train: batch 770 of 893
 Phase train: batch 780 of 893
 Phase train: batch 790 of 893
 Phase train: batch 800 of 893
 Phase train: batch 810 of 893
 Phase train: batch 820 of 893
 Phase train: batch 830 of 893
 Phase train: batch 840 of 893
 Phase train: batch 850 of 893
 Phase train: batch 860 of 893
 Phase train: batch 870 of 893
 Phase train: batch 880 of 893
 Phase train: batch 890 of 893
train took 3653.16 sec | loss = 2.1234 | top1_acc = 64.0896  | top5_acc = 88.7955
 Phase valid: batch 10 of 383
 Phase valid: batch 20 of 383
 Phase valid: batch 30 of 383
 Phase valid: batch 40 of 383
 Phase valid: batch 50 of 383
 Phase valid: batch 60 of 383
 Phase valid: batch 70 of 383
 Phase valid: batch 80 of 383
 Phase valid: batch 90 of 383
 Phase valid: batch 100 of 383
 Phase valid: batch 110 of 383
 Phase valid: batch 120 of 383
 Phase valid: batch 130 of 383
 Phase valid: batch 140 of 383
 Phase valid: batch 150 of 383
 Phase valid: batch 160 of 383
 Phase valid: batch 170 of 383
 Phase valid: batch 180 of 383
 Phase valid: batch 190 of 383
 Phase valid: batch 200 of 383
 Phase valid: batch 210 of 383
 Phase valid: batch 220 of 383
 Phase valid: batch 230 of 383
 Phase valid: batch 240 of 383
 Phase valid: batch 250 of 383
 Phase valid: batch 260 of 383
 Phase valid: batch 270 of 383
 Phase valid: batch 280 of 383
 Phase valid: batch 290 of 383
 Phase valid: batch 300 of 383
 Phase valid: batch 310 of 383
 Phase valid: batch 320 of 383
 Phase valid: batch 330 of 383
 Phase valid: batch 340 of 383
 Phase valid: batch 350 of 383
 Phase valid: batch 360 of 383
 Phase valid: batch 370 of 383
 Phase valid: batch 380 of 383
valid took 492.99 sec | loss = 1.3061 | top1_acc = 65.8824  | top5_acc = 87.9739
Epoch 3 =========================================================
lr=[0.0001]
 Phase train: batch 10 of 893
 Phase train: batch 20 of 893
 Phase train: batch 30 of 893
 Phase train: batch 40 of 893
 Phase train: batch 50 of 893
 Phase train: batch 60 of 893
 Phase train: batch 70 of 893
 Phase train: batch 80 of 893
 Phase train: batch 90 of 893
 Phase train: batch 100 of 893
 Phase train: batch 110 of 893
 Phase train: batch 120 of 893
 Phase train: batch 130 of 893
 Phase train: batch 140 of 893
 Phase train: batch 150 of 893
 Phase train: batch 160 of 893
 Phase train: batch 170 of 893
 Phase train: batch 180 of 893
 Phase train: batch 190 of 893
 Phase train: batch 200 of 893
 Phase train: batch 210 of 893
 Phase train: batch 220 of 893
 Phase train: batch 230 of 893
 Phase train: batch 240 of 893
 Phase train: batch 250 of 893
 Phase train: batch 260 of 893
 Phase train: batch 270 of 893
 Phase train: batch 280 of 893
 Phase train: batch 290 of 893
 Phase train: batch 300 of 893
 Phase train: batch 310 of 893
 Phase train: batch 320 of 893
 Phase train: batch 330 of 893
 Phase train: batch 340 of 893
 Phase train: batch 350 of 893
 Phase train: batch 360 of 893
 Phase train: batch 370 of 893
 Phase train: batch 380 of 893
 Phase train: batch 390 of 893
 Phase train: batch 400 of 893
 Phase train: batch 410 of 893
 Phase train: batch 420 of 893
 Phase train: batch 430 of 893
 Phase train: batch 440 of 893
 Phase train: batch 450 of 893
 Phase train: batch 460 of 893
 Phase train: batch 470 of 893
 Phase train: batch 480 of 893
 Phase train: batch 490 of 893
 Phase train: batch 500 of 893
 Phase train: batch 510 of 893
 Phase train: batch 520 of 893
 Phase train: batch 530 of 893
 Phase train: batch 540 of 893
 Phase train: batch 550 of 893
 Phase train: batch 560 of 893
 Phase train: batch 570 of 893
 Phase train: batch 580 of 893
 Phase train: batch 590 of 893
 Phase train: batch 600 of 893
 Phase train: batch 610 of 893
 Phase train: batch 620 of 893
 Phase train: batch 630 of 893
 Phase train: batch 640 of 893
 Phase train: batch 650 of 893
 Phase train: batch 660 of 893
 Phase train: batch 670 of 893
 Phase train: batch 680 of 893
 Phase train: batch 690 of 893
 Phase train: batch 700 of 893
 Phase train: batch 710 of 893
 Phase train: batch 720 of 893
 Phase train: batch 730 of 893
 Phase train: batch 740 of 893
 Phase train: batch 750 of 893
 Phase train: batch 760 of 893
 Phase train: batch 770 of 893
 Phase train: batch 780 of 893
 Phase train: batch 790 of 893
 Phase train: batch 800 of 893
 Phase train: batch 810 of 893
 Phase train: batch 820 of 893
 Phase train: batch 830 of 893
 Phase train: batch 840 of 893
 Phase train: batch 850 of 893
 Phase train: batch 860 of 893
 Phase train: batch 870 of 893
 Phase train: batch 880 of 893
 Phase train: batch 890 of 893
train took 3652.64 sec | loss = 1.4454 | top1_acc = 74.0056  | top5_acc = 93.5574
 Phase valid: batch 10 of 383
 Phase valid: batch 20 of 383
 Phase valid: batch 30 of 383
 Phase valid: batch 40 of 383
 Phase valid: batch 50 of 383
 Phase valid: batch 60 of 383
 Phase valid: batch 70 of 383
 Phase valid: batch 80 of 383
 Phase valid: batch 90 of 383
 Phase valid: batch 100 of 383
 Phase valid: batch 110 of 383
 Phase valid: batch 120 of 383
 Phase valid: batch 130 of 383
 Phase valid: batch 140 of 383
 Phase valid: batch 150 of 383
 Phase valid: batch 160 of 383
 Phase valid: batch 170 of 383
 Phase valid: batch 180 of 383
 Phase valid: batch 190 of 383
 Phase valid: batch 200 of 383
 Phase valid: batch 210 of 383
 Phase valid: batch 220 of 383
 Phase valid: batch 230 of 383
 Phase valid: batch 240 of 383
 Phase valid: batch 250 of 383
 Phase valid: batch 260 of 383
 Phase valid: batch 270 of 383
 Phase valid: batch 280 of 383
 Phase valid: batch 290 of 383
 Phase valid: batch 300 of 383
 Phase valid: batch 310 of 383
 Phase valid: batch 320 of 383
 Phase valid: batch 330 of 383
 Phase valid: batch 340 of 383
 Phase valid: batch 350 of 383
 Phase valid: batch 360 of 383
 Phase valid: batch 370 of 383
 Phase valid: batch 380 of 383
valid took 492.71 sec | loss = 1.1450 | top1_acc = 71.4379  | top5_acc = 89.2811
Epoch 4 =========================================================
lr=[0.0001]
 Phase train: batch 10 of 893
 Phase train: batch 20 of 893
 Phase train: batch 30 of 893
 Phase train: batch 40 of 893
 Phase train: batch 50 of 893
 Phase train: batch 60 of 893
 Phase train: batch 70 of 893
 Phase train: batch 80 of 893
 Phase train: batch 90 of 893
 Phase train: batch 100 of 893
 Phase train: batch 110 of 893
 Phase train: batch 120 of 893
 Phase train: batch 130 of 893
 Phase train: batch 140 of 893
 Phase train: batch 150 of 893
 Phase train: batch 160 of 893
 Phase train: batch 170 of 893
 Phase train: batch 180 of 893
 Phase train: batch 190 of 893
 Phase train: batch 200 of 893
 Phase train: batch 210 of 893
 Phase train: batch 220 of 893
 Phase train: batch 230 of 893
 Phase train: batch 240 of 893
 Phase train: batch 250 of 893
 Phase train: batch 260 of 893
 Phase train: batch 270 of 893
 Phase train: batch 280 of 893
 Phase train: batch 290 of 893
 Phase train: batch 300 of 893
 Phase train: batch 310 of 893
 Phase train: batch 320 of 893
 Phase train: batch 330 of 893
 Phase train: batch 340 of 893
 Phase train: batch 350 of 893
 Phase train: batch 360 of 893
 Phase train: batch 370 of 893
 Phase train: batch 380 of 893
 Phase train: batch 390 of 893
 Phase train: batch 400 of 893
 Phase train: batch 410 of 893
 Phase train: batch 420 of 893
 Phase train: batch 430 of 893
 Phase train: batch 440 of 893
 Phase train: batch 450 of 893
 Phase train: batch 460 of 893
 Phase train: batch 470 of 893
 Phase train: batch 480 of 893
 Phase train: batch 490 of 893
 Phase train: batch 500 of 893
 Phase train: batch 510 of 893
 Phase train: batch 520 of 893
 Phase train: batch 530 of 893
 Phase train: batch 540 of 893
 Phase train: batch 550 of 893
 Phase train: batch 560 of 893
 Phase train: batch 570 of 893
 Phase train: batch 580 of 893
 Phase train: batch 590 of 893
 Phase train: batch 600 of 893
 Phase train: batch 610 of 893
 Phase train: batch 620 of 893
 Phase train: batch 630 of 893
 Phase train: batch 640 of 893
 Phase train: batch 650 of 893
 Phase train: batch 660 of 893
 Phase train: batch 670 of 893
 Phase train: batch 680 of 893
 Phase train: batch 690 of 893
 Phase train: batch 700 of 893
 Phase train: batch 710 of 893
 Phase train: batch 720 of 893
 Phase train: batch 730 of 893
 Phase train: batch 740 of 893
 Phase train: batch 750 of 893
 Phase train: batch 760 of 893
 Phase train: batch 770 of 893
 Phase train: batch 780 of 893
 Phase train: batch 790 of 893
 Phase train: batch 800 of 893
 Phase train: batch 810 of 893
 Phase train: batch 820 of 893
 Phase train: batch 830 of 893
 Phase train: batch 840 of 893
 Phase train: batch 850 of 893
 Phase train: batch 860 of 893
 Phase train: batch 870 of 893
 Phase train: batch 880 of 893
 Phase train: batch 890 of 893
train took 3651.95 sec | loss = 1.0596 | top1_acc = 79.6639  | top5_acc = 95.6022
 Phase valid: batch 10 of 383
 Phase valid: batch 20 of 383
 Phase valid: batch 30 of 383
 Phase valid: batch 40 of 383
 Phase valid: batch 50 of 383
 Phase valid: batch 60 of 383
 Phase valid: batch 70 of 383
 Phase valid: batch 80 of 383
 Phase valid: batch 90 of 383
 Phase valid: batch 100 of 383
 Phase valid: batch 110 of 383
 Phase valid: batch 120 of 383
 Phase valid: batch 130 of 383
 Phase valid: batch 140 of 383
 Phase valid: batch 150 of 383
 Phase valid: batch 160 of 383
 Phase valid: batch 170 of 383
 Phase valid: batch 180 of 383
 Phase valid: batch 190 of 383
 Phase valid: batch 200 of 383
 Phase valid: batch 210 of 383
 Phase valid: batch 220 of 383
 Phase valid: batch 230 of 383
 Phase valid: batch 240 of 383
 Phase valid: batch 250 of 383
 Phase valid: batch 260 of 383
 Phase valid: batch 270 of 383
 Phase valid: batch 280 of 383
 Phase valid: batch 290 of 383
 Phase valid: batch 300 of 383
 Phase valid: batch 310 of 383
 Phase valid: batch 320 of 383
 Phase valid: batch 330 of 383
 Phase valid: batch 340 of 383
 Phase valid: batch 350 of 383
 Phase valid: batch 360 of 383
 Phase valid: batch 370 of 383
 Phase valid: batch 380 of 383
valid took 492.73 sec | loss = 1.0965 | top1_acc = 72.2222  | top5_acc = 91.5033
Epoch 5 =========================================================
lr=[0.0001]
 Phase train: batch 10 of 893
 Phase train: batch 20 of 893
 Phase train: batch 30 of 893
 Phase train: batch 40 of 893
 Phase train: batch 50 of 893
 Phase train: batch 60 of 893
 Phase train: batch 70 of 893
 Phase train: batch 80 of 893
 Phase train: batch 90 of 893
 Phase train: batch 100 of 893
 Phase train: batch 110 of 893
 Phase train: batch 120 of 893
 Phase train: batch 130 of 893
 Phase train: batch 140 of 893
 Phase train: batch 150 of 893
 Phase train: batch 160 of 893
 Phase train: batch 170 of 893
 Phase train: batch 180 of 893
 Phase train: batch 190 of 893
 Phase train: batch 200 of 893
 Phase train: batch 210 of 893
 Phase train: batch 220 of 893
 Phase train: batch 230 of 893
 Phase train: batch 240 of 893
 Phase train: batch 250 of 893
 Phase train: batch 260 of 893
 Phase train: batch 270 of 893
 Phase train: batch 280 of 893
 Phase train: batch 290 of 893
 Phase train: batch 300 of 893
 Phase train: batch 310 of 893
 Phase train: batch 320 of 893
 Phase train: batch 330 of 893
 Phase train: batch 340 of 893
 Phase train: batch 350 of 893
 Phase train: batch 360 of 893
 Phase train: batch 370 of 893
 Phase train: batch 380 of 893
 Phase train: batch 390 of 893
 Phase train: batch 400 of 893
 Phase train: batch 410 of 893
 Phase train: batch 420 of 893
 Phase train: batch 430 of 893
 Phase train: batch 440 of 893
 Phase train: batch 450 of 893
 Phase train: batch 460 of 893
 Phase train: batch 470 of 893
 Phase train: batch 480 of 893
 Phase train: batch 490 of 893
 Phase train: batch 500 of 893
 Phase train: batch 510 of 893
 Phase train: batch 520 of 893
 Phase train: batch 530 of 893
 Phase train: batch 540 of 893
 Phase train: batch 550 of 893
 Phase train: batch 560 of 893
 Phase train: batch 570 of 893
 Phase train: batch 580 of 893
 Phase train: batch 590 of 893
 Phase train: batch 600 of 893
 Phase train: batch 610 of 893
 Phase train: batch 620 of 893
 Phase train: batch 630 of 893
 Phase train: batch 640 of 893
 Phase train: batch 650 of 893
 Phase train: batch 660 of 893
 Phase train: batch 670 of 893
 Phase train: batch 680 of 893
 Phase train: batch 690 of 893
 Phase train: batch 700 of 893
 Phase train: batch 710 of 893
 Phase train: batch 720 of 893
 Phase train: batch 730 of 893
 Phase train: batch 740 of 893
 Phase train: batch 750 of 893
 Phase train: batch 760 of 893
 Phase train: batch 770 of 893
 Phase train: batch 780 of 893
 Phase train: batch 790 of 893
 Phase train: batch 800 of 893
 Phase train: batch 810 of 893
 Phase train: batch 820 of 893
 Phase train: batch 830 of 893
 Phase train: batch 840 of 893
 Phase train: batch 850 of 893
 Phase train: batch 860 of 893
 Phase train: batch 870 of 893
 Phase train: batch 880 of 893
 Phase train: batch 890 of 893
train took 3651.09 sec | loss = 0.8171 | top1_acc = 83.8655  | top5_acc = 96.9188
 Phase valid: batch 10 of 383
 Phase valid: batch 20 of 383
 Phase valid: batch 30 of 383
 Phase valid: batch 40 of 383
 Phase valid: batch 50 of 383
 Phase valid: batch 60 of 383
 Phase valid: batch 70 of 383
 Phase valid: batch 80 of 383
 Phase valid: batch 90 of 383
 Phase valid: batch 100 of 383
 Phase valid: batch 110 of 383
 Phase valid: batch 120 of 383
 Phase valid: batch 130 of 383
 Phase valid: batch 140 of 383
 Phase valid: batch 150 of 383
 Phase valid: batch 160 of 383
 Phase valid: batch 170 of 383
 Phase valid: batch 180 of 383
 Phase valid: batch 190 of 383
 Phase valid: batch 200 of 383
 Phase valid: batch 210 of 383
 Phase valid: batch 220 of 383
 Phase valid: batch 230 of 383
 Phase valid: batch 240 of 383
 Phase valid: batch 250 of 383
 Phase valid: batch 260 of 383
 Phase valid: batch 270 of 383
 Phase valid: batch 280 of 383
 Phase valid: batch 290 of 383
 Phase valid: batch 300 of 383
 Phase valid: batch 310 of 383
 Phase valid: batch 320 of 383
 Phase valid: batch 330 of 383
 Phase valid: batch 340 of 383
 Phase valid: batch 350 of 383
 Phase valid: batch 360 of 383
 Phase valid: batch 370 of 383
 Phase valid: batch 380 of 383
valid took 493.05 sec | loss = 1.1150 | top1_acc = 72.0915  | top5_acc = 90.7843
Epoch 6 =========================================================
lr=[0.0001]
 Phase train: batch 10 of 893
 Phase train: batch 20 of 893
 Phase train: batch 30 of 893
 Phase train: batch 40 of 893
 Phase train: batch 50 of 893
 Phase train: batch 60 of 893
 Phase train: batch 70 of 893
 Phase train: batch 80 of 893
 Phase train: batch 90 of 893
 Phase train: batch 100 of 893
 Phase train: batch 110 of 893
 Phase train: batch 120 of 893
 Phase train: batch 130 of 893
 Phase train: batch 140 of 893
 Phase train: batch 150 of 893
 Phase train: batch 160 of 893
 Phase train: batch 170 of 893
 Phase train: batch 180 of 893
 Phase train: batch 190 of 893
 Phase train: batch 200 of 893
 Phase train: batch 210 of 893
 Phase train: batch 220 of 893
 Phase train: batch 230 of 893
 Phase train: batch 240 of 893
 Phase train: batch 250 of 893
 Phase train: batch 260 of 893
 Phase train: batch 270 of 893
 Phase train: batch 280 of 893
 Phase train: batch 290 of 893
 Phase train: batch 300 of 893
 Phase train: batch 310 of 893
 Phase train: batch 320 of 893
 Phase train: batch 330 of 893
 Phase train: batch 340 of 893
 Phase train: batch 350 of 893
 Phase train: batch 360 of 893
 Phase train: batch 370 of 893
 Phase train: batch 380 of 893
 Phase train: batch 390 of 893
 Phase train: batch 400 of 893
 Phase train: batch 410 of 893
 Phase train: batch 420 of 893
 Phase train: batch 430 of 893
 Phase train: batch 440 of 893
 Phase train: batch 450 of 893
 Phase train: batch 460 of 893
 Phase train: batch 470 of 893
 Phase train: batch 480 of 893
 Phase train: batch 490 of 893
 Phase train: batch 500 of 893
 Phase train: batch 510 of 893
 Phase train: batch 520 of 893
 Phase train: batch 530 of 893
 Phase train: batch 540 of 893
 Phase train: batch 550 of 893
 Phase train: batch 560 of 893
 Phase train: batch 570 of 893
 Phase train: batch 580 of 893
 Phase train: batch 590 of 893
 Phase train: batch 600 of 893
 Phase train: batch 610 of 893
 Phase train: batch 620 of 893
 Phase train: batch 630 of 893
 Phase train: batch 640 of 893
 Phase train: batch 650 of 893
 Phase train: batch 660 of 893
 Phase train: batch 670 of 893
 Phase train: batch 680 of 893
 Phase train: batch 690 of 893
 Phase train: batch 700 of 893
 Phase train: batch 710 of 893
 Phase train: batch 720 of 893
 Phase train: batch 730 of 893
 Phase train: batch 740 of 893
 Phase train: batch 750 of 893
 Phase train: batch 760 of 893
 Phase train: batch 770 of 893
 Phase train: batch 780 of 893
 Phase train: batch 790 of 893
 Phase train: batch 800 of 893
 Phase train: batch 810 of 893
 Phase train: batch 820 of 893
 Phase train: batch 830 of 893
 Phase train: batch 840 of 893
 Phase train: batch 850 of 893
 Phase train: batch 860 of 893
 Phase train: batch 870 of 893
 Phase train: batch 880 of 893
 Phase train: batch 890 of 893
train took 3652.19 sec | loss = 0.7240 | top1_acc = 84.9860  | top5_acc = 96.9468
 Phase valid: batch 10 of 383
 Phase valid: batch 20 of 383
 Phase valid: batch 30 of 383
 Phase valid: batch 40 of 383
 Phase valid: batch 50 of 383
 Phase valid: batch 60 of 383
 Phase valid: batch 70 of 383
 Phase valid: batch 80 of 383
 Phase valid: batch 90 of 383
 Phase valid: batch 100 of 383
 Phase valid: batch 110 of 383
 Phase valid: batch 120 of 383
 Phase valid: batch 130 of 383
 Phase valid: batch 140 of 383
 Phase valid: batch 150 of 383
 Phase valid: batch 160 of 383
 Phase valid: batch 170 of 383
 Phase valid: batch 180 of 383
 Phase valid: batch 190 of 383
 Phase valid: batch 200 of 383
 Phase valid: batch 210 of 383
 Phase valid: batch 220 of 383
 Phase valid: batch 230 of 383
 Phase valid: batch 240 of 383
 Phase valid: batch 250 of 383
 Phase valid: batch 260 of 383
 Phase valid: batch 270 of 383
 Phase valid: batch 280 of 383
 Phase valid: batch 290 of 383
 Phase valid: batch 300 of 383
 Phase valid: batch 310 of 383
 Phase valid: batch 320 of 383
 Phase valid: batch 330 of 383
 Phase valid: batch 340 of 383
 Phase valid: batch 350 of 383
 Phase valid: batch 360 of 383
 Phase valid: batch 370 of 383
 Phase valid: batch 380 of 383
valid took 492.28 sec | loss = 0.9616 | top1_acc = 76.4706  | top5_acc = 94.7059
Epoch 7 =========================================================
lr=[1e-05]
 Phase train: batch 10 of 893
 Phase train: batch 20 of 893
 Phase train: batch 30 of 893
 Phase train: batch 40 of 893
 Phase train: batch 50 of 893
 Phase train: batch 60 of 893
 Phase train: batch 70 of 893
 Phase train: batch 80 of 893
 Phase train: batch 90 of 893
 Phase train: batch 100 of 893
 Phase train: batch 110 of 893
 Phase train: batch 120 of 893
 Phase train: batch 130 of 893
 Phase train: batch 140 of 893
 Phase train: batch 150 of 893
 Phase train: batch 160 of 893
 Phase train: batch 170 of 893
 Phase train: batch 180 of 893
 Phase train: batch 190 of 893
 Phase train: batch 200 of 893
 Phase train: batch 210 of 893
 Phase train: batch 220 of 893
 Phase train: batch 230 of 893
 Phase train: batch 240 of 893
 Phase train: batch 250 of 893
 Phase train: batch 260 of 893
 Phase train: batch 270 of 893
 Phase train: batch 280 of 893
 Phase train: batch 290 of 893
 Phase train: batch 300 of 893
 Phase train: batch 310 of 893
 Phase train: batch 320 of 893
 Phase train: batch 330 of 893
 Phase train: batch 340 of 893
 Phase train: batch 350 of 893
 Phase train: batch 360 of 893
 Phase train: batch 370 of 893
 Phase train: batch 380 of 893
 Phase train: batch 390 of 893
 Phase train: batch 400 of 893
 Phase train: batch 410 of 893
 Phase train: batch 420 of 893
 Phase train: batch 430 of 893
 Phase train: batch 440 of 893
 Phase train: batch 450 of 893
 Phase train: batch 460 of 893
 Phase train: batch 470 of 893
 Phase train: batch 480 of 893
 Phase train: batch 490 of 893
 Phase train: batch 500 of 893
 Phase train: batch 510 of 893
 Phase train: batch 520 of 893
 Phase train: batch 530 of 893
 Phase train: batch 540 of 893
 Phase train: batch 550 of 893
 Phase train: batch 560 of 893
 Phase train: batch 570 of 893
 Phase train: batch 580 of 893
 Phase train: batch 590 of 893
 Phase train: batch 600 of 893
 Phase train: batch 610 of 893
 Phase train: batch 620 of 893
 Phase train: batch 630 of 893
 Phase train: batch 640 of 893
 Phase train: batch 650 of 893
 Phase train: batch 660 of 893
 Phase train: batch 670 of 893
 Phase train: batch 680 of 893
 Phase train: batch 690 of 893
 Phase train: batch 700 of 893
 Phase train: batch 710 of 893
 Phase train: batch 720 of 893
 Phase train: batch 730 of 893
 Phase train: batch 740 of 893
 Phase train: batch 750 of 893
 Phase train: batch 760 of 893
 Phase train: batch 770 of 893
 Phase train: batch 780 of 893
 Phase train: batch 790 of 893
 Phase train: batch 800 of 893
 Phase train: batch 810 of 893
 Phase train: batch 820 of 893
 Phase train: batch 830 of 893
 Phase train: batch 840 of 893
 Phase train: batch 850 of 893
 Phase train: batch 860 of 893
 Phase train: batch 870 of 893
 Phase train: batch 880 of 893
 Phase train: batch 890 of 893
train took 3651.35 sec | loss = 0.5524 | top1_acc = 89.7479  | top5_acc = 98.3473
 Phase valid: batch 10 of 383
 Phase valid: batch 20 of 383
 Phase valid: batch 30 of 383
 Phase valid: batch 40 of 383
 Phase valid: batch 50 of 383
 Phase valid: batch 60 of 383
 Phase valid: batch 70 of 383
 Phase valid: batch 80 of 383
 Phase valid: batch 90 of 383
 Phase valid: batch 100 of 383
 Phase valid: batch 110 of 383
 Phase valid: batch 120 of 383
 Phase valid: batch 130 of 383
 Phase valid: batch 140 of 383
 Phase valid: batch 150 of 383
 Phase valid: batch 160 of 383
 Phase valid: batch 170 of 383
 Phase valid: batch 180 of 383
 Phase valid: batch 190 of 383
 Phase valid: batch 200 of 383
 Phase valid: batch 210 of 383
 Phase valid: batch 220 of 383
 Phase valid: batch 230 of 383
 Phase valid: batch 240 of 383
 Phase valid: batch 250 of 383
 Phase valid: batch 260 of 383
 Phase valid: batch 270 of 383
 Phase valid: batch 280 of 383
 Phase valid: batch 290 of 383
 Phase valid: batch 300 of 383
 Phase valid: batch 310 of 383
 Phase valid: batch 320 of 383
 Phase valid: batch 330 of 383
 Phase valid: batch 340 of 383
 Phase valid: batch 350 of 383
 Phase valid: batch 360 of 383
 Phase valid: batch 370 of 383
 Phase valid: batch 380 of 383
valid took 493.12 sec | loss = 0.8916 | top1_acc = 77.9739  | top5_acc = 94.9673
Epoch 8 =========================================================
lr=[1e-05]
 Phase train: batch 10 of 893
 Phase train: batch 20 of 893
 Phase train: batch 30 of 893
 Phase train: batch 40 of 893
 Phase train: batch 50 of 893
 Phase train: batch 60 of 893
 Phase train: batch 70 of 893
 Phase train: batch 80 of 893
 Phase train: batch 90 of 893
 Phase train: batch 100 of 893
 Phase train: batch 110 of 893
 Phase train: batch 120 of 893
 Phase train: batch 130 of 893
 Phase train: batch 140 of 893
 Phase train: batch 150 of 893
 Phase train: batch 160 of 893
 Phase train: batch 170 of 893
 Phase train: batch 180 of 893
 Phase train: batch 190 of 893
 Phase train: batch 200 of 893
 Phase train: batch 210 of 893
 Phase train: batch 220 of 893
 Phase train: batch 230 of 893
 Phase train: batch 240 of 893
 Phase train: batch 250 of 893
 Phase train: batch 260 of 893
 Phase train: batch 270 of 893
 Phase train: batch 280 of 893
 Phase train: batch 290 of 893
 Phase train: batch 300 of 893
 Phase train: batch 310 of 893
 Phase train: batch 320 of 893
 Phase train: batch 330 of 893
 Phase train: batch 340 of 893
 Phase train: batch 350 of 893
 Phase train: batch 360 of 893
 Phase train: batch 370 of 893
 Phase train: batch 380 of 893
 Phase train: batch 390 of 893
 Phase train: batch 400 of 893
 Phase train: batch 410 of 893
 Phase train: batch 420 of 893
 Phase train: batch 430 of 893
 Phase train: batch 440 of 893
 Phase train: batch 450 of 893
 Phase train: batch 460 of 893
 Phase train: batch 470 of 893
 Phase train: batch 480 of 893
 Phase train: batch 490 of 893
 Phase train: batch 500 of 893
 Phase train: batch 510 of 893
 Phase train: batch 520 of 893
 Phase train: batch 530 of 893
 Phase train: batch 540 of 893
 Phase train: batch 550 of 893
 Phase train: batch 560 of 893
 Phase train: batch 570 of 893
 Phase train: batch 580 of 893
 Phase train: batch 590 of 893
 Phase train: batch 600 of 893
 Phase train: batch 610 of 893
 Phase train: batch 620 of 893
 Phase train: batch 630 of 893
 Phase train: batch 640 of 893
 Phase train: batch 650 of 893
 Phase train: batch 660 of 893
 Phase train: batch 670 of 893
 Phase train: batch 680 of 893
 Phase train: batch 690 of 893
 Phase train: batch 700 of 893
 Phase train: batch 710 of 893
 Phase train: batch 720 of 893
 Phase train: batch 730 of 893
 Phase train: batch 740 of 893
 Phase train: batch 750 of 893
 Phase train: batch 760 of 893
 Phase train: batch 770 of 893
 Phase train: batch 780 of 893
 Phase train: batch 790 of 893
 Phase train: batch 800 of 893
 Phase train: batch 810 of 893
 Phase train: batch 820 of 893
 Phase train: batch 830 of 893
 Phase train: batch 840 of 893
 Phase train: batch 850 of 893
 Phase train: batch 860 of 893
 Phase train: batch 870 of 893
 Phase train: batch 880 of 893
 Phase train: batch 890 of 893
train took 3651.87 sec | loss = 0.5042 | top1_acc = 90.3642  | top5_acc = 98.4034
 Phase valid: batch 10 of 383
 Phase valid: batch 20 of 383
 Phase valid: batch 30 of 383
 Phase valid: batch 40 of 383
 Phase valid: batch 50 of 383
 Phase valid: batch 60 of 383
 Phase valid: batch 70 of 383
 Phase valid: batch 80 of 383
 Phase valid: batch 90 of 383
 Phase valid: batch 100 of 383
 Phase valid: batch 110 of 383
 Phase valid: batch 120 of 383
 Phase valid: batch 130 of 383
 Phase valid: batch 140 of 383
 Phase valid: batch 150 of 383
 Phase valid: batch 160 of 383
 Phase valid: batch 170 of 383
 Phase valid: batch 180 of 383
 Phase valid: batch 190 of 383
 Phase valid: batch 200 of 383
 Phase valid: batch 210 of 383
 Phase valid: batch 220 of 383
 Phase valid: batch 230 of 383
 Phase valid: batch 240 of 383
 Phase valid: batch 250 of 383
 Phase valid: batch 260 of 383
 Phase valid: batch 270 of 383
 Phase valid: batch 280 of 383
 Phase valid: batch 290 of 383
 Phase valid: batch 300 of 383
 Phase valid: batch 310 of 383
 Phase valid: batch 320 of 383
 Phase valid: batch 330 of 383
 Phase valid: batch 340 of 383
 Phase valid: batch 350 of 383
 Phase valid: batch 360 of 383
 Phase valid: batch 370 of 383
 Phase valid: batch 380 of 383
valid took 493.06 sec | loss = 0.8805 | top1_acc = 78.6928  | top5_acc = 95.1634

Model Test¶

Finally, we want to evaluate our model on our test set. To do so, we'll uniformly sample 10 clips from each test video, to produce a tensor of [10 x 3 x (8 or 32) x 112 x 112] that we call our prediction on. This will give us the clip-level accuracy for batches of 10 at time. We also want to calculate the video-level accuracy which we can get by simply averaging the clip-level accuracy.

In [13]:

ret = learner.evaluate()

1530 samples of torch.Size([3, 32, 112, 112])
Processsing 100 of 1530 samples..
Processsing 200 of 1530 samples..
Processsing 300 of 1530 samples..
Processsing 400 of 1530 samples..
Processsing 500 of 1530 samples..
Processsing 600 of 1530 samples..
Processsing 700 of 1530 samples..
Processsing 800 of 1530 samples..
Processsing 900 of 1530 samples..
Processsing 1000 of 1530 samples..
Processsing 1100 of 1530 samples..
Processsing 1200 of 1530 samples..
Processsing 1300 of 1530 samples..
Processsing 1400 of 1530 samples..
Processsing 1500 of 1530 samples..
Avg. inference time per video (1530 clips) = 3174.63 ms
Video prediction accuracy = 0.8
Clip prediction accuracy = 0.77

The reported accuracy from "Closer look" paper is 74.5% (clip accuracy of 66.1% on split1 based on VMZ repo). You can see that by using the following parameters, we are able to reproduce the state of the art result at ~77% accuracy.

MODEL_INPUT_SIZE = 32
BATCH_SIZE = 4
EPOCHS = 8
LR = 0.0001

In [ ]: