Notebook

Deep Learning Models -- A collection of various deep learning architectures, models, and tips for TensorFlow and PyTorch in Jupyter Notebooks.

Author: Sebastian Raschka
GitHub Repository: https://github.com/rasbt/deeplearning-models

In [1]:

%load_ext watermark
%watermark -a 'Sebastian Raschka' -v -p torch

Sebastian Raschka 

CPython 3.7.3
IPython 7.9.0

torch 1.7.0

Runs on CPU or GPU (if available)

Model Zoo -- Reproducible Results with Deterministic Behavior and Runtime Benchmark¶

In this notebook, we are benchmarking the performance impact of setting PyTorch to deterministic behavior. In general, there are two aspects for reproducible resuls in PyTorch,

Setting a random seed
Setting cuDNN and PyTorch algorithmic behavior to deterministic

For more details, please see https://pytorch.org/docs/stable/notes/randomness.html

1. Setting a random seed¶

I recommend using a function like the following one prior to using dataset loaders and initializing a model if you want to ensure the data is shuffled in the same manner if you rerun this notebook and the model gets the same initial random weights:

In [2]:

def set_all_seeds(seed):
    os.environ["PL_GLOBAL_SEED"] = str(seed)
    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)

2. Setting cuDNN and PyTorch algorithmic behavior to deterministic¶

Similar to the set_all_seeds function above, I recommend setting the behavior of PyTorch and cuDNN to deterministic (this is particulary relevant when using GPUs). We can also define a function for that:

In [3]:

def set_deterministic():
    if torch.cuda.is_available():
        torch.backends.cudnn.benchmark = False
        torch.backends.cudnn.deterministic = True
    torch.set_deterministic(True)

1) Setup¶

After setting up the general configuration in this section, the following two sections will train a ResNet-101 model without and with deterministic behavior to get a sense how using deterministic options affect the runtime speed.

In [4]:

import os
import numpy as np
import torch
import random

In [5]:

##########################
### SETTINGS
##########################

# Device
CUDA_DEVICE_NUM = 1 # change as appropriate
DEVICE = torch.device('cuda:%d' % CUDA_DEVICE_NUM if torch.cuda.is_available() else 'cpu')
print('Device:', DEVICE)

# Data settings
num_classes = 10

# Hyperparameters
random_seed = 1
learning_rate = 0.01
batch_size = 128
num_epochs = 50

Device: cuda:1

In [6]:

import sys

sys.path.insert(0, "..") # to include ../helper_evaluate.py etc.

from helper_evaluate import compute_accuracy
from helper_data import get_dataloaders_cifar10
from helper_train import train_classifier_simple_v1

2) Run without Deterministic Behavior¶

Before we enable deterministic behavior, we will run a ResNet-101 with otherwise the exact same settings for comparison. Note that setting random seeds doesn't affect the timing results.

In [7]:

### Set random seed ###
set_all_seeds(random_seed)

In [8]:

##########################
### Dataset
##########################

train_loader, valid_loader, test_loader = get_dataloaders_cifar10(
    batch_size, 
    num_workers=0, 
    validation_fraction=0.1)

Files already downloaded and verified

In [9]:

##########################
### Model
##########################

from deterministic_benchmark_utils import resnet101

model = resnet101(num_classes, grayscale=False)

model = model.to(DEVICE)
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

In [10]:

_ = train_classifier_simple_v1(num_epochs=num_epochs, model=model, 
                               optimizer=optimizer, device=DEVICE, 
                               train_loader=train_loader, valid_loader=valid_loader, 
                               logging_interval=200)

Epoch: 001/050 | Batch 0000/0352 | Loss: 2.6271
Epoch: 001/050 | Batch 0200/0352 | Loss: 2.0193
***Epoch: 001/050 | Train. Acc.: 26.831% | Loss: 2.772
***Epoch: 001/050 | Valid. Acc.: 26.300% | Loss: 2.642
Time elapsed: 1.17 min
Epoch: 002/050 | Batch 0000/0352 | Loss: 1.9047
Epoch: 002/050 | Batch 0200/0352 | Loss: 1.6645
***Epoch: 002/050 | Train. Acc.: 41.080% | Loss: 1.581
***Epoch: 002/050 | Valid. Acc.: 40.780% | Loss: 1.588
Time elapsed: 2.39 min
Epoch: 003/050 | Batch 0000/0352 | Loss: 1.4886
Epoch: 003/050 | Batch 0200/0352 | Loss: 1.4483
***Epoch: 003/050 | Train. Acc.: 47.693% | Loss: 1.428
***Epoch: 003/050 | Valid. Acc.: 47.480% | Loss: 1.439
Time elapsed: 3.63 min
Epoch: 004/050 | Batch 0000/0352 | Loss: 1.3687
Epoch: 004/050 | Batch 0200/0352 | Loss: 1.3750
***Epoch: 004/050 | Train. Acc.: 57.487% | Loss: 1.189
***Epoch: 004/050 | Valid. Acc.: 56.020% | Loss: 1.231
Time elapsed: 4.85 min
Epoch: 005/050 | Batch 0000/0352 | Loss: 1.2162
Epoch: 005/050 | Batch 0200/0352 | Loss: 1.3256
***Epoch: 005/050 | Train. Acc.: 58.827% | Loss: 1.151
***Epoch: 005/050 | Valid. Acc.: 57.240% | Loss: 1.192
Time elapsed: 6.08 min
Epoch: 006/050 | Batch 0000/0352 | Loss: 1.2045
Epoch: 006/050 | Batch 0200/0352 | Loss: 1.2144
***Epoch: 006/050 | Train. Acc.: 62.062% | Loss: 1.085
***Epoch: 006/050 | Valid. Acc.: 59.180% | Loss: 1.176
Time elapsed: 7.31 min
Epoch: 007/050 | Batch 0000/0352 | Loss: 1.0280
Epoch: 007/050 | Batch 0200/0352 | Loss: 1.0381
***Epoch: 007/050 | Train. Acc.: 67.880% | Loss: 0.917
***Epoch: 007/050 | Valid. Acc.: 64.660% | Loss: 1.040
Time elapsed: 8.53 min
Epoch: 008/050 | Batch 0000/0352 | Loss: 0.9092
Epoch: 008/050 | Batch 0200/0352 | Loss: 0.9647
***Epoch: 008/050 | Train. Acc.: 69.656% | Loss: 0.873
***Epoch: 008/050 | Valid. Acc.: 64.820% | Loss: 1.034
Time elapsed: 9.75 min
Epoch: 009/050 | Batch 0000/0352 | Loss: 0.7789
Epoch: 009/050 | Batch 0200/0352 | Loss: 0.8018
***Epoch: 009/050 | Train. Acc.: 67.900% | Loss: 0.935
***Epoch: 009/050 | Valid. Acc.: 62.060% | Loss: 1.131
Time elapsed: 10.98 min
Epoch: 010/050 | Batch 0000/0352 | Loss: 0.6950
Epoch: 010/050 | Batch 0200/0352 | Loss: 0.7482
***Epoch: 010/050 | Train. Acc.: 69.613% | Loss: 0.912
***Epoch: 010/050 | Valid. Acc.: 62.900% | Loss: 1.182
Time elapsed: 12.20 min
Epoch: 011/050 | Batch 0000/0352 | Loss: 0.5364
Epoch: 011/050 | Batch 0200/0352 | Loss: 0.7148
***Epoch: 011/050 | Train. Acc.: 70.978% | Loss: 0.890
***Epoch: 011/050 | Valid. Acc.: 62.500% | Loss: 1.269
Time elapsed: 13.42 min
Epoch: 012/050 | Batch 0000/0352 | Loss: 0.5508
Epoch: 012/050 | Batch 0200/0352 | Loss: 0.5948
***Epoch: 012/050 | Train. Acc.: 67.689% | Loss: 1.032
***Epoch: 012/050 | Valid. Acc.: 60.420% | Loss: 1.399
Time elapsed: 14.65 min
Epoch: 013/050 | Batch 0000/0352 | Loss: 0.4297
Epoch: 013/050 | Batch 0200/0352 | Loss: 0.6009
***Epoch: 013/050 | Train. Acc.: 64.773% | Loss: 1.240
***Epoch: 013/050 | Valid. Acc.: 57.120% | Loss: 1.726
Time elapsed: 15.87 min
Epoch: 014/050 | Batch 0000/0352 | Loss: 0.4545
Epoch: 014/050 | Batch 0200/0352 | Loss: 0.4772
***Epoch: 014/050 | Train. Acc.: 65.091% | Loss: 1.284
***Epoch: 014/050 | Valid. Acc.: 56.340% | Loss: 1.817
Time elapsed: 17.10 min
Epoch: 015/050 | Batch 0000/0352 | Loss: 0.2806
Epoch: 015/050 | Batch 0200/0352 | Loss: 0.3789
***Epoch: 015/050 | Train. Acc.: 63.369% | Loss: 1.385
***Epoch: 015/050 | Valid. Acc.: 54.080% | Loss: 2.058
Time elapsed: 18.32 min
Epoch: 016/050 | Batch 0000/0352 | Loss: 0.2412
Epoch: 016/050 | Batch 0200/0352 | Loss: 0.3509
***Epoch: 016/050 | Train. Acc.: 80.284% | Loss: 0.615
***Epoch: 016/050 | Valid. Acc.: 67.080% | Loss: 1.290
Time elapsed: 19.55 min
Epoch: 017/050 | Batch 0000/0352 | Loss: 0.2912
Epoch: 017/050 | Batch 0200/0352 | Loss: 0.3700
***Epoch: 017/050 | Train. Acc.: 75.598% | Loss: 0.832
***Epoch: 017/050 | Valid. Acc.: 64.440% | Loss: 1.435
Time elapsed: 20.77 min
Epoch: 018/050 | Batch 0000/0352 | Loss: 0.2656
Epoch: 018/050 | Batch 0200/0352 | Loss: 0.3249
***Epoch: 018/050 | Train. Acc.: 77.631% | Loss: 0.675
***Epoch: 018/050 | Valid. Acc.: 64.800% | Loss: 1.230
Time elapsed: 21.99 min
Epoch: 019/050 | Batch 0000/0352 | Loss: 0.4638
Epoch: 019/050 | Batch 0200/0352 | Loss: 0.4722
***Epoch: 019/050 | Train. Acc.: 83.680% | Loss: 0.510
***Epoch: 019/050 | Valid. Acc.: 68.220% | Loss: 1.236
Time elapsed: 23.21 min
Epoch: 020/050 | Batch 0000/0352 | Loss: 0.1987
Epoch: 020/050 | Batch 0200/0352 | Loss: 0.1785
***Epoch: 020/050 | Train. Acc.: 85.493% | Loss: 0.462
***Epoch: 020/050 | Valid. Acc.: 69.540% | Loss: 1.301
Time elapsed: 24.44 min
Epoch: 021/050 | Batch 0000/0352 | Loss: 0.1715
Epoch: 021/050 | Batch 0200/0352 | Loss: 0.2591
***Epoch: 021/050 | Train. Acc.: 87.880% | Loss: 0.379
***Epoch: 021/050 | Valid. Acc.: 69.440% | Loss: 1.297
Time elapsed: 25.66 min
Epoch: 022/050 | Batch 0000/0352 | Loss: 0.0751
Epoch: 022/050 | Batch 0200/0352 | Loss: 0.1399
***Epoch: 022/050 | Train. Acc.: 64.718% | Loss: 1.335
***Epoch: 022/050 | Valid. Acc.: 54.700% | Loss: 1.915
Time elapsed: 26.89 min
Epoch: 023/050 | Batch 0000/0352 | Loss: 0.4129
Epoch: 023/050 | Batch 0200/0352 | Loss: 0.1313
***Epoch: 023/050 | Train. Acc.: 83.102% | Loss: 0.593
***Epoch: 023/050 | Valid. Acc.: 66.320% | Loss: 1.510
Time elapsed: 28.11 min
Epoch: 024/050 | Batch 0000/0352 | Loss: 0.1770
Epoch: 024/050 | Batch 0200/0352 | Loss: 0.1969
***Epoch: 024/050 | Train. Acc.: 85.444% | Loss: 0.483
***Epoch: 024/050 | Valid. Acc.: 67.820% | Loss: 1.462
Time elapsed: 29.34 min
Epoch: 025/050 | Batch 0000/0352 | Loss: 0.0875
Epoch: 025/050 | Batch 0200/0352 | Loss: 0.0878
***Epoch: 025/050 | Train. Acc.: 83.856% | Loss: 0.572
***Epoch: 025/050 | Valid. Acc.: 66.040% | Loss: 1.633
Time elapsed: 30.56 min
Epoch: 026/050 | Batch 0000/0352 | Loss: 0.1272
Epoch: 026/050 | Batch 0200/0352 | Loss: 0.1095
***Epoch: 026/050 | Train. Acc.: 89.209% | Loss: 0.346
***Epoch: 026/050 | Valid. Acc.: 69.160% | Loss: 1.430
Time elapsed: 31.79 min
Epoch: 027/050 | Batch 0000/0352 | Loss: 0.0530
Epoch: 027/050 | Batch 0200/0352 | Loss: 0.1496
***Epoch: 027/050 | Train. Acc.: 85.542% | Loss: 0.496
***Epoch: 027/050 | Valid. Acc.: 66.340% | Loss: 1.574
Time elapsed: 33.01 min
Epoch: 028/050 | Batch 0000/0352 | Loss: 0.1304
Epoch: 028/050 | Batch 0200/0352 | Loss: 0.1132
***Epoch: 028/050 | Train. Acc.: 81.762% | Loss: 0.664
***Epoch: 028/050 | Valid. Acc.: 65.660% | Loss: 1.656
Time elapsed: 34.24 min
Epoch: 029/050 | Batch 0000/0352 | Loss: 0.1230
Epoch: 029/050 | Batch 0200/0352 | Loss: 0.1004
***Epoch: 029/050 | Train. Acc.: 90.560% | Loss: 0.294
***Epoch: 029/050 | Valid. Acc.: 70.400% | Loss: 1.322
Time elapsed: 35.47 min
Epoch: 030/050 | Batch 0000/0352 | Loss: 0.0776
Epoch: 030/050 | Batch 0200/0352 | Loss: 0.1733
***Epoch: 030/050 | Train. Acc.: 89.807% | Loss: 0.349
***Epoch: 030/050 | Valid. Acc.: 68.740% | Loss: 1.638
Time elapsed: 36.69 min
Epoch: 031/050 | Batch 0000/0352 | Loss: 0.0538
Epoch: 031/050 | Batch 0200/0352 | Loss: 0.1678
***Epoch: 031/050 | Train. Acc.: 92.460% | Loss: 0.249
***Epoch: 031/050 | Valid. Acc.: 71.440% | Loss: 1.503
Time elapsed: 37.91 min
Epoch: 032/050 | Batch 0000/0352 | Loss: 0.1050
Epoch: 032/050 | Batch 0200/0352 | Loss: 0.1882
***Epoch: 032/050 | Train. Acc.: 93.773% | Loss: 0.198
***Epoch: 032/050 | Valid. Acc.: 72.300% | Loss: 1.377
Time elapsed: 39.14 min
Epoch: 033/050 | Batch 0000/0352 | Loss: 0.0545
Epoch: 033/050 | Batch 0200/0352 | Loss: 0.1488
***Epoch: 033/050 | Train. Acc.: 90.589% | Loss: 0.328
***Epoch: 033/050 | Valid. Acc.: 69.160% | Loss: 1.553
Time elapsed: 40.36 min
Epoch: 034/050 | Batch 0000/0352 | Loss: 0.0395
Epoch: 034/050 | Batch 0200/0352 | Loss: 0.0678
***Epoch: 034/050 | Train. Acc.: 92.429% | Loss: 0.245
***Epoch: 034/050 | Valid. Acc.: 70.360% | Loss: 1.454
Time elapsed: 41.59 min
Epoch: 035/050 | Batch 0000/0352 | Loss: 0.0534
Epoch: 035/050 | Batch 0200/0352 | Loss: 0.1108
***Epoch: 035/050 | Train. Acc.: 88.993% | Loss: 0.384
***Epoch: 035/050 | Valid. Acc.: 69.060% | Loss: 1.518
Time elapsed: 42.82 min
Epoch: 036/050 | Batch 0000/0352 | Loss: 0.1118
Epoch: 036/050 | Batch 0200/0352 | Loss: 0.0408
***Epoch: 036/050 | Train. Acc.: 93.811% | Loss: 0.195
***Epoch: 036/050 | Valid. Acc.: 72.640% | Loss: 1.404
Time elapsed: 44.04 min
Epoch: 037/050 | Batch 0000/0352 | Loss: 0.0635
Epoch: 037/050 | Batch 0200/0352 | Loss: 0.0642
***Epoch: 037/050 | Train. Acc.: 92.851% | Loss: 0.236
***Epoch: 037/050 | Valid. Acc.: 71.040% | Loss: 1.596
Time elapsed: 45.27 min
Epoch: 038/050 | Batch 0000/0352 | Loss: 0.0662
Epoch: 038/050 | Batch 0200/0352 | Loss: 0.0905
***Epoch: 038/050 | Train. Acc.: 91.587% | Loss: 0.289
***Epoch: 038/050 | Valid. Acc.: 69.600% | Loss: 1.638
Time elapsed: 46.49 min
Epoch: 039/050 | Batch 0000/0352 | Loss: 0.0744
Epoch: 039/050 | Batch 0200/0352 | Loss: 0.0106
***Epoch: 039/050 | Train. Acc.: 94.356% | Loss: 0.182
***Epoch: 039/050 | Valid. Acc.: 71.500% | Loss: 1.577
Time elapsed: 47.71 min
Epoch: 040/050 | Batch 0000/0352 | Loss: 0.0087
Epoch: 040/050 | Batch 0200/0352 | Loss: 0.0522
***Epoch: 040/050 | Train. Acc.: 85.756% | Loss: 0.536
***Epoch: 040/050 | Valid. Acc.: 66.460% | Loss: 1.836
Time elapsed: 48.94 min
Epoch: 041/050 | Batch 0000/0352 | Loss: 0.0759
Epoch: 041/050 | Batch 0200/0352 | Loss: 0.0693
***Epoch: 041/050 | Train. Acc.: 91.647% | Loss: 0.283
***Epoch: 041/050 | Valid. Acc.: 69.760% | Loss: 1.621
Time elapsed: 50.16 min
Epoch: 042/050 | Batch 0000/0352 | Loss: 0.1359
Epoch: 042/050 | Batch 0200/0352 | Loss: 0.0824
***Epoch: 042/050 | Train. Acc.: 72.082% | Loss: 1.133
***Epoch: 042/050 | Valid. Acc.: 59.880% | Loss: 2.117
Time elapsed: 51.37 min
Epoch: 043/050 | Batch 0000/0352 | Loss: 0.3177
Epoch: 043/050 | Batch 0200/0352 | Loss: 0.0631
***Epoch: 043/050 | Train. Acc.: 95.522% | Loss: 0.138
***Epoch: 043/050 | Valid. Acc.: 72.840% | Loss: 1.308
Time elapsed: 52.60 min
Epoch: 044/050 | Batch 0000/0352 | Loss: 0.0383
Epoch: 044/050 | Batch 0200/0352 | Loss: 0.0217
***Epoch: 044/050 | Train. Acc.: 98.031% | Loss: 0.060
***Epoch: 044/050 | Valid. Acc.: 74.800% | Loss: 1.464
Time elapsed: 53.82 min
Epoch: 045/050 | Batch 0000/0352 | Loss: 0.0097
Epoch: 045/050 | Batch 0200/0352 | Loss: 1.6721
***Epoch: 045/050 | Train. Acc.: 63.620% | Loss: 1.092
***Epoch: 045/050 | Valid. Acc.: 58.180% | Loss: 1.276
Time elapsed: 55.05 min
Epoch: 046/050 | Batch 0000/0352 | Loss: 1.0475
Epoch: 046/050 | Batch 0200/0352 | Loss: 0.5436
***Epoch: 046/050 | Train. Acc.: 92.420% | Loss: 0.235
***Epoch: 046/050 | Valid. Acc.: 72.740% | Loss: 0.935
Time elapsed: 56.27 min
Epoch: 047/050 | Batch 0000/0352 | Loss: 0.3582
Epoch: 047/050 | Batch 0200/0352 | Loss: 0.1439
***Epoch: 047/050 | Train. Acc.: 97.762% | Loss: 0.068
***Epoch: 047/050 | Valid. Acc.: 73.740% | Loss: 1.242
Time elapsed: 57.49 min
Epoch: 048/050 | Batch 0000/0352 | Loss: 0.0733
Epoch: 048/050 | Batch 0200/0352 | Loss: 0.0193
***Epoch: 048/050 | Train. Acc.: 98.331% | Loss: 0.050
***Epoch: 048/050 | Valid. Acc.: 74.080% | Loss: 1.582
Time elapsed: 58.72 min
Epoch: 049/050 | Batch 0000/0352 | Loss: 0.0125
Epoch: 049/050 | Batch 0200/0352 | Loss: 0.0322
***Epoch: 049/050 | Train. Acc.: 96.682% | Loss: 0.105
***Epoch: 049/050 | Valid. Acc.: 73.420% | Loss: 1.699
Time elapsed: 59.94 min
Epoch: 050/050 | Batch 0000/0352 | Loss: 0.0185
Epoch: 050/050 | Batch 0200/0352 | Loss: 0.1059
***Epoch: 050/050 | Train. Acc.: 97.193% | Loss: 0.085
***Epoch: 050/050 | Valid. Acc.: 73.120% | Loss: 1.708
Time elapsed: 61.16 min
Total Training Time: 61.16 min

3) Run with Deterministic Behavior¶

In this section, we set the deterministic behavior via the set_deterministic() function defined at the top of this notebook and compare how it affects the runtime speed of the ResNet-101 model. (Note that setting random seeds doesn't affect the timing results.)

In [11]:

set_deterministic()

In [12]:

### Set random seed ###
set_all_seeds(random_seed)

In [13]:

##########################
### Dataset
##########################

train_loader, valid_loader, test_loader = get_dataloaders_cifar10(
    batch_size, 
    num_workers=0, 
    validation_fraction=0.1)

Files already downloaded and verified

In [14]:

##########################
### Model
##########################

from deterministic_benchmark_utils import resnet101

model = resnet101(num_classes, grayscale=False)

model = model.to(DEVICE)
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

In [15]:

_ = train_classifier_simple_v1(num_epochs=num_epochs, model=model, 
                               optimizer=optimizer, device=DEVICE, 
                               train_loader=train_loader, valid_loader=valid_loader, 
                               logging_interval=200)

Epoch: 001/050 | Batch 0000/0352 | Loss: 2.6271
Epoch: 001/050 | Batch 0200/0352 | Loss: 2.2247
***Epoch: 001/050 | Train. Acc.: 20.160% | Loss: 3.197
***Epoch: 001/050 | Valid. Acc.: 19.660% | Loss: 2.888
Time elapsed: 1.21 min
Epoch: 002/050 | Batch 0000/0352 | Loss: 2.1651
Epoch: 002/050 | Batch 0200/0352 | Loss: 1.8593
***Epoch: 002/050 | Train. Acc.: 34.807% | Loss: 1.741
***Epoch: 002/050 | Valid. Acc.: 34.780% | Loss: 1.740
Time elapsed: 2.44 min
Epoch: 003/050 | Batch 0000/0352 | Loss: 1.6749
Epoch: 003/050 | Batch 0200/0352 | Loss: 1.5481
***Epoch: 003/050 | Train. Acc.: 43.656% | Loss: 1.581
***Epoch: 003/050 | Valid. Acc.: 43.020% | Loss: 1.567
Time elapsed: 3.66 min
Epoch: 004/050 | Batch 0000/0352 | Loss: 1.4751
Epoch: 004/050 | Batch 0200/0352 | Loss: 1.3720
***Epoch: 004/050 | Train. Acc.: 40.811% | Loss: 2.705
***Epoch: 004/050 | Valid. Acc.: 40.340% | Loss: 2.576
Time elapsed: 4.88 min
Epoch: 005/050 | Batch 0000/0352 | Loss: 1.3779
Epoch: 005/050 | Batch 0200/0352 | Loss: 1.3833
***Epoch: 005/050 | Train. Acc.: 49.793% | Loss: 1.380
***Epoch: 005/050 | Valid. Acc.: 50.000% | Loss: 1.402
Time elapsed: 6.11 min
Epoch: 006/050 | Batch 0000/0352 | Loss: 1.2832
Epoch: 006/050 | Batch 0200/0352 | Loss: 1.2519
***Epoch: 006/050 | Train. Acc.: 41.536% | Loss: 1.599
***Epoch: 006/050 | Valid. Acc.: 41.540% | Loss: 1.616
Time elapsed: 7.34 min
Epoch: 007/050 | Batch 0000/0352 | Loss: 1.4472
Epoch: 007/050 | Batch 0200/0352 | Loss: 1.2238
***Epoch: 007/050 | Train. Acc.: 57.633% | Loss: 1.171
***Epoch: 007/050 | Valid. Acc.: 57.100% | Loss: 1.219
Time elapsed: 8.56 min
Epoch: 008/050 | Batch 0000/0352 | Loss: 1.0530
Epoch: 008/050 | Batch 0200/0352 | Loss: 1.0490
***Epoch: 008/050 | Train. Acc.: 62.620% | Loss: 1.032
***Epoch: 008/050 | Valid. Acc.: 60.460% | Loss: 1.109
Time elapsed: 9.79 min
Epoch: 009/050 | Batch 0000/0352 | Loss: 1.0152
Epoch: 009/050 | Batch 0200/0352 | Loss: 0.9678
***Epoch: 009/050 | Train. Acc.: 56.600% | Loss: 1.252
***Epoch: 009/050 | Valid. Acc.: 55.620% | Loss: 1.311
Time elapsed: 11.01 min
Epoch: 010/050 | Batch 0000/0352 | Loss: 0.9740
Epoch: 010/050 | Batch 0200/0352 | Loss: 0.9075
***Epoch: 010/050 | Train. Acc.: 60.382% | Loss: 1.130
***Epoch: 010/050 | Valid. Acc.: 57.740% | Loss: 1.254
Time elapsed: 12.24 min
Epoch: 011/050 | Batch 0000/0352 | Loss: 0.7509
Epoch: 011/050 | Batch 0200/0352 | Loss: 0.9028
***Epoch: 011/050 | Train. Acc.: 70.091% | Loss: 0.851
***Epoch: 011/050 | Valid. Acc.: 64.060% | Loss: 1.045
Time elapsed: 13.46 min
Epoch: 012/050 | Batch 0000/0352 | Loss: 0.6871
Epoch: 012/050 | Batch 0200/0352 | Loss: 0.8642
***Epoch: 012/050 | Train. Acc.: 71.362% | Loss: 0.835
***Epoch: 012/050 | Valid. Acc.: 64.200% | Loss: 1.079
Time elapsed: 14.68 min
Epoch: 013/050 | Batch 0000/0352 | Loss: 0.5075
Epoch: 013/050 | Batch 0200/0352 | Loss: 0.7813
***Epoch: 013/050 | Train. Acc.: 68.644% | Loss: 0.916
***Epoch: 013/050 | Valid. Acc.: 62.620% | Loss: 1.169
Time elapsed: 15.92 min
Epoch: 014/050 | Batch 0000/0352 | Loss: 0.5169
Epoch: 014/050 | Batch 0200/0352 | Loss: 0.7422
***Epoch: 014/050 | Train. Acc.: 73.640% | Loss: 0.769
***Epoch: 014/050 | Valid. Acc.: 65.380% | Loss: 1.090
Time elapsed: 17.14 min
Epoch: 015/050 | Batch 0000/0352 | Loss: 0.4203
Epoch: 015/050 | Batch 0200/0352 | Loss: 0.6845
***Epoch: 015/050 | Train. Acc.: 67.880% | Loss: 0.965
***Epoch: 015/050 | Valid. Acc.: 62.080% | Loss: 1.229
Time elapsed: 18.37 min
Epoch: 016/050 | Batch 0000/0352 | Loss: 0.4785
Epoch: 016/050 | Batch 0200/0352 | Loss: 0.6260
***Epoch: 016/050 | Train. Acc.: 68.316% | Loss: 1.007
***Epoch: 016/050 | Valid. Acc.: 60.440% | Loss: 1.380
Time elapsed: 19.59 min
Epoch: 017/050 | Batch 0000/0352 | Loss: 0.4362
Epoch: 017/050 | Batch 0200/0352 | Loss: 0.5386
***Epoch: 017/050 | Train. Acc.: 75.922% | Loss: 0.755
***Epoch: 017/050 | Valid. Acc.: 65.360% | Loss: 1.277
Time elapsed: 20.82 min
Epoch: 018/050 | Batch 0000/0352 | Loss: 0.3375
Epoch: 018/050 | Batch 0200/0352 | Loss: 0.5347
***Epoch: 018/050 | Train. Acc.: 79.782% | Loss: 0.623
***Epoch: 018/050 | Valid. Acc.: 68.000% | Loss: 1.168
Time elapsed: 22.05 min
Epoch: 019/050 | Batch 0000/0352 | Loss: 0.3641
Epoch: 019/050 | Batch 0200/0352 | Loss: 0.5012
***Epoch: 019/050 | Train. Acc.: 60.073% | Loss: 1.186
***Epoch: 019/050 | Valid. Acc.: 57.720% | Loss: 1.249
Time elapsed: 23.27 min
Epoch: 020/050 | Batch 0000/0352 | Loss: 1.0688
Epoch: 020/050 | Batch 0200/0352 | Loss: 0.8022
***Epoch: 020/050 | Train. Acc.: 74.598% | Loss: 0.724
***Epoch: 020/050 | Valid. Acc.: 67.060% | Loss: 0.991
Time elapsed: 24.50 min
Epoch: 021/050 | Batch 0000/0352 | Loss: 0.6721
Epoch: 021/050 | Batch 0200/0352 | Loss: 0.5717
***Epoch: 021/050 | Train. Acc.: 80.036% | Loss: 0.582
***Epoch: 021/050 | Valid. Acc.: 69.000% | Loss: 1.022
Time elapsed: 25.72 min
Epoch: 022/050 | Batch 0000/0352 | Loss: 0.3786
Epoch: 022/050 | Batch 0200/0352 | Loss: 0.4261
***Epoch: 022/050 | Train. Acc.: 71.816% | Loss: 0.984
***Epoch: 022/050 | Valid. Acc.: 61.340% | Loss: 1.499
Time elapsed: 26.95 min
Epoch: 023/050 | Batch 0000/0352 | Loss: 0.3137
Epoch: 023/050 | Batch 0200/0352 | Loss: 0.3047
***Epoch: 023/050 | Train. Acc.: 80.947% | Loss: 0.599
***Epoch: 023/050 | Valid. Acc.: 67.320% | Loss: 1.299
Time elapsed: 28.17 min
Epoch: 024/050 | Batch 0000/0352 | Loss: 0.2161
Epoch: 024/050 | Batch 0200/0352 | Loss: 0.2751
***Epoch: 024/050 | Train. Acc.: 85.284% | Loss: 0.453
***Epoch: 024/050 | Valid. Acc.: 69.680% | Loss: 1.193
Time elapsed: 29.40 min
Epoch: 025/050 | Batch 0000/0352 | Loss: 0.1900
Epoch: 025/050 | Batch 0200/0352 | Loss: 0.2022
***Epoch: 025/050 | Train. Acc.: 88.496% | Loss: 0.340
***Epoch: 025/050 | Valid. Acc.: 71.800% | Loss: 1.134
Time elapsed: 30.62 min
Epoch: 026/050 | Batch 0000/0352 | Loss: 0.1252
Epoch: 026/050 | Batch 0200/0352 | Loss: 0.3572
***Epoch: 026/050 | Train. Acc.: 87.156% | Loss: 0.403
***Epoch: 026/050 | Valid. Acc.: 71.300% | Loss: 1.200
Time elapsed: 31.85 min
Epoch: 027/050 | Batch 0000/0352 | Loss: 0.1662
Epoch: 027/050 | Batch 0200/0352 | Loss: 0.3703
***Epoch: 027/050 | Train. Acc.: 89.871% | Loss: 0.308
***Epoch: 027/050 | Valid. Acc.: 71.880% | Loss: 1.199
Time elapsed: 33.07 min
Epoch: 028/050 | Batch 0000/0352 | Loss: 0.2198
Epoch: 028/050 | Batch 0200/0352 | Loss: 0.2505
***Epoch: 028/050 | Train. Acc.: 86.404% | Loss: 0.438
***Epoch: 028/050 | Valid. Acc.: 70.160% | Loss: 1.262
Time elapsed: 34.30 min
Epoch: 029/050 | Batch 0000/0352 | Loss: 0.1404
Epoch: 029/050 | Batch 0200/0352 | Loss: 0.0904
***Epoch: 029/050 | Train. Acc.: 83.700% | Loss: 0.564
***Epoch: 029/050 | Valid. Acc.: 66.820% | Loss: 1.514
Time elapsed: 35.53 min
Epoch: 030/050 | Batch 0000/0352 | Loss: 0.2442
Epoch: 030/050 | Batch 0200/0352 | Loss: 0.2605
***Epoch: 030/050 | Train. Acc.: 76.907% | Loss: 0.882
***Epoch: 030/050 | Valid. Acc.: 64.300% | Loss: 1.780
Time elapsed: 36.76 min
Epoch: 031/050 | Batch 0000/0352 | Loss: 0.2206
Epoch: 031/050 | Batch 0200/0352 | Loss: 0.1347
***Epoch: 031/050 | Train. Acc.: 79.053% | Loss: 0.773
***Epoch: 031/050 | Valid. Acc.: 64.720% | Loss: 1.691
Time elapsed: 37.98 min
Epoch: 032/050 | Batch 0000/0352 | Loss: 0.1039
Epoch: 032/050 | Batch 0200/0352 | Loss: 0.1669
***Epoch: 032/050 | Train. Acc.: 84.833% | Loss: 0.505
***Epoch: 032/050 | Valid. Acc.: 67.180% | Loss: 1.558
Time elapsed: 39.21 min
Epoch: 033/050 | Batch 0000/0352 | Loss: 0.0840
Epoch: 033/050 | Batch 0200/0352 | Loss: 0.1019
***Epoch: 033/050 | Train. Acc.: 89.533% | Loss: 0.328
***Epoch: 033/050 | Valid. Acc.: 70.920% | Loss: 1.309
Time elapsed: 40.44 min
Epoch: 034/050 | Batch 0000/0352 | Loss: 0.1585
Epoch: 034/050 | Batch 0200/0352 | Loss: 0.1356
***Epoch: 034/050 | Train. Acc.: 87.969% | Loss: 0.408
***Epoch: 034/050 | Valid. Acc.: 69.360% | Loss: 1.486
Time elapsed: 41.67 min
Epoch: 035/050 | Batch 0000/0352 | Loss: 0.2327
Epoch: 035/050 | Batch 0200/0352 | Loss: 0.0837
***Epoch: 035/050 | Train. Acc.: 87.382% | Loss: 0.446
***Epoch: 035/050 | Valid. Acc.: 69.560% | Loss: 1.468
Time elapsed: 42.90 min
Epoch: 036/050 | Batch 0000/0352 | Loss: 0.0715
Epoch: 036/050 | Batch 0200/0352 | Loss: 0.0714
***Epoch: 036/050 | Train. Acc.: 92.993% | Loss: 0.221
***Epoch: 036/050 | Valid. Acc.: 73.320% | Loss: 1.318
Time elapsed: 44.12 min
Epoch: 037/050 | Batch 0000/0352 | Loss: 0.0546
Epoch: 037/050 | Batch 0200/0352 | Loss: 0.1362
***Epoch: 037/050 | Train. Acc.: 93.987% | Loss: 0.190
***Epoch: 037/050 | Valid. Acc.: 73.420% | Loss: 1.317
Time elapsed: 45.37 min
Epoch: 038/050 | Batch 0000/0352 | Loss: 0.0735
Epoch: 038/050 | Batch 0200/0352 | Loss: 0.0330
***Epoch: 038/050 | Train. Acc.: 94.029% | Loss: 0.190
***Epoch: 038/050 | Valid. Acc.: 72.420% | Loss: 1.421
Time elapsed: 46.59 min
Epoch: 039/050 | Batch 0000/0352 | Loss: 0.1424
Epoch: 039/050 | Batch 0200/0352 | Loss: 0.1515
***Epoch: 039/050 | Train. Acc.: 94.256% | Loss: 0.180
***Epoch: 039/050 | Valid. Acc.: 73.500% | Loss: 1.354
Time elapsed: 47.82 min
Epoch: 040/050 | Batch 0000/0352 | Loss: 0.0455
Epoch: 040/050 | Batch 0200/0352 | Loss: 0.0889
***Epoch: 040/050 | Train. Acc.: 91.420% | Loss: 0.309
***Epoch: 040/050 | Valid. Acc.: 72.240% | Loss: 1.485
Time elapsed: 49.05 min
Epoch: 041/050 | Batch 0000/0352 | Loss: 0.0589
Epoch: 041/050 | Batch 0200/0352 | Loss: 0.0381
***Epoch: 041/050 | Train. Acc.: 87.698% | Loss: 0.499
***Epoch: 041/050 | Valid. Acc.: 68.860% | Loss: 1.768
Time elapsed: 50.28 min
Epoch: 042/050 | Batch 0000/0352 | Loss: 0.1281
Epoch: 042/050 | Batch 0200/0352 | Loss: 0.0269
***Epoch: 042/050 | Train. Acc.: 95.911% | Loss: 0.128
***Epoch: 042/050 | Valid. Acc.: 75.200% | Loss: 1.303
Time elapsed: 51.51 min
Epoch: 043/050 | Batch 0000/0352 | Loss: 0.0151
Epoch: 043/050 | Batch 0200/0352 | Loss: 0.3936
***Epoch: 043/050 | Train. Acc.: 95.720% | Loss: 0.128
***Epoch: 043/050 | Valid. Acc.: 75.320% | Loss: 1.133
Time elapsed: 52.74 min
Epoch: 044/050 | Batch 0000/0352 | Loss: 0.1388
Epoch: 044/050 | Batch 0200/0352 | Loss: 0.0791
***Epoch: 044/050 | Train. Acc.: 93.296% | Loss: 0.235
***Epoch: 044/050 | Valid. Acc.: 73.260% | Loss: 1.451
Time elapsed: 53.97 min
Epoch: 045/050 | Batch 0000/0352 | Loss: 0.0067
Epoch: 045/050 | Batch 0200/0352 | Loss: 0.0268
***Epoch: 045/050 | Train. Acc.: 96.744% | Loss: 0.106
***Epoch: 045/050 | Valid. Acc.: 75.040% | Loss: 1.363
Time elapsed: 55.19 min
Epoch: 046/050 | Batch 0000/0352 | Loss: 0.0690
Epoch: 046/050 | Batch 0200/0352 | Loss: 0.0792
***Epoch: 046/050 | Train. Acc.: 93.733% | Loss: 0.211
***Epoch: 046/050 | Valid. Acc.: 73.500% | Loss: 1.439
Time elapsed: 56.42 min
Epoch: 047/050 | Batch 0000/0352 | Loss: 0.1034
Epoch: 047/050 | Batch 0200/0352 | Loss: 0.0920
***Epoch: 047/050 | Train. Acc.: 92.091% | Loss: 0.276
***Epoch: 047/050 | Valid. Acc.: 71.780% | Loss: 1.538
Time elapsed: 57.65 min
Epoch: 048/050 | Batch 0000/0352 | Loss: 0.0308
Epoch: 048/050 | Batch 0200/0352 | Loss: 0.0766
***Epoch: 048/050 | Train. Acc.: 95.351% | Loss: 0.153
***Epoch: 048/050 | Valid. Acc.: 75.200% | Loss: 1.408
Time elapsed: 58.88 min
Epoch: 049/050 | Batch 0000/0352 | Loss: 0.0672
Epoch: 049/050 | Batch 0200/0352 | Loss: 0.0637
***Epoch: 049/050 | Train. Acc.: 91.213% | Loss: 0.321
***Epoch: 049/050 | Valid. Acc.: 71.400% | Loss: 1.533
Time elapsed: 60.11 min
Epoch: 050/050 | Batch 0000/0352 | Loss: 0.0195
Epoch: 050/050 | Batch 0200/0352 | Loss: 0.0353
***Epoch: 050/050 | Train. Acc.: 89.211% | Loss: 0.313
***Epoch: 050/050 | Valid. Acc.: 72.620% | Loss: 0.953
Time elapsed: 61.35 min
Total Training Time: 61.35 min

4) Result¶

In this particular case, the deterministic behavior does not seem to influence performance noticeably.