Learn data science and machine learning by building real-world projects on Jovian

Assignment 4 - Speech Command Recognition with M3, M5, M11, M18 CNN networks - 30 Epoch

Assignment 4 - Final Course Project for jovian.ml ZeroToGANS course on the implementation of neural networks using PyTorch.

This notebook implements speech command recognition using convolutional neural networks trained on the Google SpeechCommand dataset.

The networks are based on a 3, 5, 11 or 18 layer architecture convolutional neural networks (M3, M5, M11, M18) as described in this Very Deep Convolutional Neural Networks For Raw Waveforms paper. The networks are trained on the time domain waveform inputs of the SpeechCommand dataset.

The dataset is part of the Pytorch common datasets [https://pytorch.org/audio/stable/datasets.html]. There is more information on the dataset in this Speech Commands paper. The dataset consists of more than 105,000 WAVE audio files of various speakers saying thirtyfive different words such as "yes", "no", "up", "down", "left", "right", "on", "off", "stop", "go" and numerical digits 0-9. Similarly to MNIST dataset for images, using the SpeechCommand dataset enables us to understand and work with techniques involved in audio processing and recognition.

In [2]:
project_name='Assignment 4 - Speech Command Recognition with M3, M5, M11, M18 CNN networks'
In [4]:
# Uncomment the following line to run in Google Colab

# CPU:
#!pip install torch==1.7.0+cpu torchvision==0.8.1+cpu torchaudio==0.7.0 -f https://download.pytorch.org/whl/torch_stable.html

# GPU:
!pip install torch==1.7.0+cu101 torchvision==0.8.1+cu101 torchaudio==0.7.0 -f https://download.pytorch.org/whl/torch_stable.html

import os

import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
import torchaudio

import matplotlib.pyplot as plt
import IPython.display as ipd
from tqdm.notebook import tqdm
Looking in links: https://download.pytorch.org/whl/torch_stable.html Requirement already satisfied: torch==1.7.0+cu101 in /usr/local/lib/python3.6/dist-packages (1.7.0+cu101) Requirement already satisfied: torchvision==0.8.1+cu101 in /usr/local/lib/python3.6/dist-packages (0.8.1+cu101) Collecting torchaudio==0.7.0 Downloading https://files.pythonhosted.org/packages/3f/23/6b54106b3de029d3f10cf8debc302491c17630357449c900d6209665b302/torchaudio-0.7.0-cp36-cp36m-manylinux1_x86_64.whl (7.6MB) |████████████████████████████████| 7.6MB 11.9MB/s Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from torch==1.7.0+cu101) (1.19.5) Requirement already satisfied: future in /usr/local/lib/python3.6/dist-packages (from torch==1.7.0+cu101) (0.16.0) Requirement already satisfied: typing-extensions in /usr/local/lib/python3.6/dist-packages (from torch==1.7.0+cu101) (3.7.4.3) Requirement already satisfied: dataclasses in /usr/local/lib/python3.6/dist-packages (from torch==1.7.0+cu101) (0.8) Requirement already satisfied: pillow>=4.1.1 in /usr/local/lib/python3.6/dist-packages (from torchvision==0.8.1+cu101) (7.0.0) Installing collected packages: torchaudio Successfully installed torchaudio-0.7.0
/usr/local/lib/python3.6/dist-packages/torchaudio/backend/utils.py:54: UserWarning: "sox" backend is being deprecated. The default backend will be changed to "sox_io" backend in 0.8.0 and "sox" backend will be removed in 0.9.0. Please migrate to "sox_io" backend. Please refer to https://github.com/pytorch/audio/issues/903 for the detail. '"sox" backend is being deprecated. '
In [5]:
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print(device)
cuda

Import the data

In [6]:
import jovian
In [9]:
path = 'Torchaudio SpeechCommands dataset'
description = '105 000 audio speech command .wav files'

jovian.log_dataset(path=path, description=description)
[jovian] Please enter your API key ( from https://jovian.ai/ ): API KEY: ·········· [jovian] Dataset logged.
In [10]:
!rm -rf ./*
!mkdir -p data
In [11]:
from torchaudio.datasets import SPEECHCOMMANDS

class SubsetSpeechCommand(SPEECHCOMMANDS):
    def __init__(self, subset: str = None):
        super().__init__("./data", download=True)

        def load_list(filename):
            filepath = os.path.join(self._path, filename)
            with open(filepath) as fileobj:
                return [os.path.join(self._path, line.strip()) for line in fileobj]

        if subset == "validation":
            print('Load validation list')
            self._walker = load_list("validation_list.txt")
        elif subset == "testing":
            print('Load testing list')
            self._walker = load_list("testing_list.txt")
        elif subset == "training":
            print('Load training list')
            excludes = load_list("validation_list.txt") + load_list("testing_list.txt")
            excludes = set(excludes)
            self._walker = [w for w in self._walker if w not in excludes]

# Create training, validation and testing split of the data.
train_set = SubsetSpeechCommand("training")
validation_set = SubsetSpeechCommand("validation")
test_set = SubsetSpeechCommand("testing")
HBox(children=(FloatProgress(value=0.0, max=2428923189.0), HTML(value='')))
Load training list Load validation list Load testing list

Review samples of the datasets

In [12]:
#Get first entry of training, validation and test set for review
#tuple structure: (waveform, sample_rate, wordclass, speaker_id, utterance_nr)
waveform0, sample_rate0, wordclass0, speaker_id0, utterance_nr0 = train_set[0]
waveform1, sample_rate1, wordclass1, speaker_id1, utterance_nr1 = validation_set[0]
waveform2, sample_rate2, wordclass2, speaker_id2, utterance_nr2 = test_set[0]
In [13]:
print('Sample of training set')
waveform0, sample_rate0, wordclass0, speaker_id0, utterance_nr0
Sample of training set
Out[13]:
(tensor([[-0.0658, -0.0709, -0.0753,  ..., -0.0700, -0.0731, -0.0704]]),
 16000,
 'backward',
 '0165e0e8',
 0)
In [14]:
print('Sample first entry of validation set')
waveform1, sample_rate1, wordclass1, speaker_id1, utterance_nr1
Sample first entry of validation set
Out[14]:
(tensor([[-0.0004, -0.0007, -0.0009,  ...,  0.0062,  0.0058,  0.0057]]),
 16000,
 'right',
 'a69b9b3e',
 0)
In [15]:
print('Sample first entry of test set')
waveform2, sample_rate2, wordclass2, speaker_id2, utterance_nr2
Sample first entry of test set
Out[15]:
(tensor([[ 3.0518e-05,  3.0518e-05,  1.5259e-04,  ..., -7.3242e-04,
          -4.4861e-03,  4.6387e-03]]), 16000, 'right', 'bb05582b', 3)
In [16]:
print('Sample 500th entry of all sets\n')
train_set[500], validation_set[500], test_set[500]
Sample 500th entry of all sets
Out[16]:
((tensor([[ 0.0005,  0.0003, -0.0003,  ...,  0.0003,  0.0005,  0.0014]]),
  16000,
  'backward',
  '616420be',
  4),
 (tensor([[-0.0001, -0.0002, -0.0002,  ..., -0.0003, -0.0003, -0.0002]]),
  16000,
  'eight',
  '856eb138',
  0),
 (tensor([[-0.0033, -0.0055, -0.0024,  ...,  0.0049,  0.0037,  0.0030]]),
  16000,
  'eight',
  '80c45ed6',
  1))
In [17]:
#View some of the training set audio waveforms
rows = 3
cols = 3
n = rows*cols
fig, axes = plt.subplots(rows, cols, figsize=(14, 16))
for i in range (9):
  r = i // cols
  c = i % cols
  ax = axes[r][c]
  ax.plot(train_set[i][0].t().numpy())
  ax.set_title("Word: " + test_set[i][2] +"    S'rate: {}".format(test_set[i][1]) + '\n Wform shape: {}'.format(waveform2.size()))

plt.show()
Notebook Image
In [18]:
wordclasses = sorted(list(set(element[2] for element in train_set)))
wordclasses
Out[18]:
['backward',
 'bed',
 'bird',
 'cat',
 'dog',
 'down',
 'eight',
 'five',
 'follow',
 'forward',
 'four',
 'go',
 'happy',
 'house',
 'learn',
 'left',
 'marvin',
 'nine',
 'no',
 'off',
 'on',
 'one',
 'right',
 'seven',
 'sheila',
 'six',
 'stop',
 'three',
 'tree',
 'two',
 'up',
 'visual',
 'wow',
 'yes',
 'zero']
In [19]:
#Nr of entries per Training/Validation/Testing dataset
print('Total number of entries in training set :', (len(train_set)))
print('Total number of entries in validation set :', (len(validation_set)))
print('Total number of entries in test set :', (len(test_set)))

Total number of entries in training set : 84843 Total number of entries in validation set : 9981 Total number of entries in test set : 11005
In [23]:
#Count the nr of entries per wordclass in training set
print('Total number of entries in training set :', (len(train_set)))
print('Number of word class entries :', (len(wordclasses)))
print('\nWord class : Count')
for j in range (len(wordclasses)):
  count = 0
  for i in range (len(train_set)):
    if (train_set[i][2]==wordclasses[j]):
      count=count+1
  print(wordclasses[j], ' :', count)
Total number of entries in training set : 84843 Number of word class entries : 35 Word class : Count backward : 1346 bed : 1594 bird : 1697 cat : 1657 dog : 1711 down : 3134 eight : 3033 five : 3240 follow : 1275 forward : 1256 four : 2955 go : 3106 happy : 1632 house : 1727 learn : 1286 left : 3037 marvin : 1710 nine : 3170 no : 3130 off : 2970 on : 3086 one : 3140 right : 3019 seven : 3205 sheila : 1606 six : 3088 stop : 3111 three : 2966 tree : 1407 two : 3111 up : 2948 visual : 1288 wow : 1724 yes : 3228 zero : 3250
In [24]:
#Count the nr of entries per wordclass in validation set
print('Total number of entries in validation set :', (len(validation_set)))
print('Number of word class entries :', (len(wordclasses)))
print('\nWord class : Count')
for j in range (len(wordclasses)):
  count = 0
  for i in range (len(validation_set)):
    if (validation_set[i][2]==wordclasses[j]):
      count=count+1
  print(wordclasses[j], ' :', count)
Total number of entries in validation set : 9981 Number of word class entries : 35 Word class : Count backward : 153 bed : 213 bird : 182 cat : 180 dog : 197 down : 377 eight : 346 five : 367 follow : 132 forward : 146 four : 373 go : 372 happy : 219 house : 195 learn : 128 left : 352 marvin : 195 nine : 356 no : 406 off : 373 on : 363 one : 351 right : 363 seven : 387 sheila : 204 six : 378 stop : 350 three : 356 tree : 159 two : 345 up : 350 visual : 139 wow : 193 yes : 397 zero : 384
In [25]:
#Count the nr of entries per wordclass in test set
print('Total number of entries in test set :', (len(test_set)))
print('Number of word class entries :', (len(wordclasses)))
print('\nWord class : Count')
for j in range (len(wordclasses)):
  count = 0
  for i in range (len(test_set)):
    if (test_set[i][2]==wordclasses[j]):
      count=count+1
  print(wordclasses[j], ' :', count)
Total number of entries in test set : 11005 Number of word class entries : 35 Word class : Count backward : 165 bed : 207 bird : 185 cat : 194 dog : 220 down : 406 eight : 408 five : 445 follow : 172 forward : 155 four : 400 go : 402 happy : 203 house : 191 learn : 161 left : 412 marvin : 195 nine : 408 no : 405 off : 402 on : 396 one : 399 right : 396 seven : 406 sheila : 212 six : 394 stop : 411 three : 405 tree : 193 two : 424 up : 425 visual : 165 wow : 206 yes : 419 zero : 418

Play back an audio data file

In [26]:
waveform_test, *_ = test_set[0]
print(test_set[0])
ipd.Audio(waveform_test.numpy(), rate=sample_rate2)
(tensor([[ 3.0518e-05, 3.0518e-05, 1.5259e-04, ..., -7.3242e-04, -4.4861e-03, 4.6387e-03]]), 16000, 'right', 'bb05582b', 3)
Out[26]:

Transform an audio data file

In [27]:
new_sr = 8000
transform = torchaudio.transforms.Resample(orig_freq=sample_rate2, new_freq=new_sr)
transformed = transform(waveform_test)

ipd.Audio(transformed.numpy(), rate=new_sr)
Out[27]:

Data utility functions

In [28]:
def index_to_wordclass(index):
    # Return word based on index in wordclasses
    return wordclasses[index]

def wordclass_to_index(word):
    # Return the index of the word in wordclasses
    return torch.tensor(wordclasses.index(word))
In [29]:
def pad_sequence(batch):
    # Make all tensor in a batch the same length by padding with zeros
    batch = [item.t() for item in batch]
    batch = torch.nn.utils.rnn.pad_sequence(batch, batch_first=True, padding_value=0.)
    return batch.permute(0, 2, 1)
In [30]:
def collate_fn(batch):
    # A data tuple has the format:
    # waveform, sample_rate, wordclass, speaker_id, utterance_nr

    tensors, targets = [], []

    # Gather in lists, and encode wordclasses as indices
    for waveform, _, wordclass, *_ in batch:
        tensors += [waveform]
        targets += [wordclass_to_index(wordclass)]

    # Group the list of tensors into a batched tensor
    tensors = pad_sequence(tensors)
    # stack - Concatenates a sequence of tensors along a new dimension
    targets = torch.stack(targets)

    return tensors, targets
In [31]:
batch_size = 400
In [32]:
if device == "cuda":
    num_workers = 1
    pin_memory = True
else:
    num_workers = 0
    pin_memory = False
In [33]:
#Dataloaders
train_loader = torch.utils.data.DataLoader(
    train_set,
    batch_size=batch_size,
    shuffle=True,
    collate_fn=collate_fn,
    num_workers=num_workers,
    pin_memory=pin_memory,
)
test_loader = torch.utils.data.DataLoader(
    test_set,
    batch_size=batch_size,
    shuffle=False,
    drop_last=False,
    collate_fn=collate_fn,
    num_workers=num_workers,
    pin_memory=pin_memory,
)
validation_loader = torch.utils.data.DataLoader(
    validation_set,
    batch_size=batch_size,
    shuffle=False,
    drop_last=False,
    collate_fn=collate_fn,
    num_workers=num_workers,
    pin_memory=pin_memory,
)
In [34]:
def count_parameters(model):
    return sum(p.numel() for p in model.parameters() if p.requires_grad)

M3 network definition

In [35]:
#M3
class M3(nn.Module):
    def __init__(self, n_input=1, n_output=35, stride=4, n_channel=128):
        super().__init__()
        self.conv1 = nn.Conv1d(n_input, 2*n_channel, kernel_size=80, stride=stride)
        self.bn1 = nn.BatchNorm1d(2*n_channel)
        self.pool1 = nn.MaxPool1d(4)
        self.conv2 = nn.Conv1d(2*n_channel, 2*n_channel, kernel_size=3, stride=1, padding=1)
        self.bn2 = nn.BatchNorm1d(2*n_channel)
        self.pool2 = nn.MaxPool1d(4)
        self.fc1 = nn.Linear(2 * n_channel, n_output)

    def forward(self, x):
        x = self.conv1(x)
        x = F.relu(self.bn1(x))
        x = self.pool1(x)
        x = self.conv2(x)
        x = F.relu(self.bn2(x))
        x = self.pool2(x)
        x = F.avg_pool1d(x, x.shape[-1])
        x = x.permute(0, 2, 1)
        x = self.fc1(x)
        return F.log_softmax(x, dim=2)

print('M3 model')
print('Model n_input', transformed.shape[0])
print('Model n_output', len(wordclasses))
print('transformed :', transformed[0])
model_M3 = M3(n_input=transformed.shape[0], n_output=len(wordclasses))
model_M3.to(device)
print(model_M3)

n = count_parameters(model_M3)
print("Number of parameters: %s" % n)
M3 model Model n_input 1 Model n_output 35 transformed : tensor([ 3.8866e-05, 3.7267e-05, -2.9491e-05, ..., 2.1669e-05, -1.0598e-04, -9.1377e-04]) M3( (conv1): Conv1d(1, 256, kernel_size=(80,), stride=(4,)) (bn1): BatchNorm1d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (pool1): MaxPool1d(kernel_size=4, stride=4, padding=0, dilation=1, ceil_mode=False) (conv2): Conv1d(256, 256, kernel_size=(3,), stride=(1,), padding=(1,)) (bn2): BatchNorm1d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (pool2): MaxPool1d(kernel_size=4, stride=4, padding=0, dilation=1, ceil_mode=False) (fc1): Linear(in_features=256, out_features=35, bias=True) ) Number of parameters: 227619

M5 network definition

In [36]:
class M5(nn.Module):
    def __init__(self, n_input=1, n_output=35, stride=4, n_channel=128):
        super().__init__()
        self.conv1 = nn.Conv1d(n_input, n_channel, kernel_size=80, stride=stride)
        self.bn1 = nn.BatchNorm1d(n_channel)
        self.pool1 = nn.MaxPool1d(4)
        self.conv2 = nn.Conv1d(n_channel, n_channel, kernel_size=3,stride=1,padding=1)
        self.bn2 = nn.BatchNorm1d(n_channel)
        self.pool2 = nn.MaxPool1d(4)
        self.conv3 = nn.Conv1d(n_channel, 2 * n_channel, kernel_size=3,stride=1,padding=1)
        self.bn3 = nn.BatchNorm1d(2 * n_channel)
        self.pool3 = nn.MaxPool1d(4)
        self.conv4 = nn.Conv1d(2 * n_channel, 4 * n_channel, kernel_size=3,stride=1,padding=1)
        self.bn4 = nn.BatchNorm1d(4 * n_channel)
        self.pool4 = nn.MaxPool1d(4)
        self.fc1 = nn.Linear(4 * n_channel, n_output)

    def forward(self, x):
        x = self.conv1(x)
        x = F.relu(self.bn1(x))
        x = self.pool1(x)
        x = self.conv2(x)
        x = F.relu(self.bn2(x))
        x = self.pool2(x)
        x = self.conv3(x)
        x = F.relu(self.bn3(x))
        x = self.pool3(x)
        x = self.conv4(x)
        x = F.relu(self.bn4(x))
        x = self.pool4(x)
        x = F.avg_pool1d(x, x.shape[-1])
        x = x.permute(0, 2, 1)
        x = self.fc1(x)
        return F.log_softmax(x, dim=2)

print('M5 model')
print('Model n_input', transformed.shape[0])
print('Model n_output', len(wordclasses))
model_M5 = M5(n_input=transformed.shape[0], n_output=len(wordclasses))
model_M5.to(device)
print(model_M5)

n = count_parameters(model_M5)
print("Number of parameters: %s" % n)
M5 model Model n_input 1 Model n_output 35 M5( (conv1): Conv1d(1, 128, kernel_size=(80,), stride=(4,)) (bn1): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (pool1): MaxPool1d(kernel_size=4, stride=4, padding=0, dilation=1, ceil_mode=False) (conv2): Conv1d(128, 128, kernel_size=(3,), stride=(1,), padding=(1,)) (bn2): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (pool2): MaxPool1d(kernel_size=4, stride=4, padding=0, dilation=1, ceil_mode=False) (conv3): Conv1d(128, 256, kernel_size=(3,), stride=(1,), padding=(1,)) (bn3): BatchNorm1d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (pool3): MaxPool1d(kernel_size=4, stride=4, padding=0, dilation=1, ceil_mode=False) (conv4): Conv1d(256, 512, kernel_size=(3,), stride=(1,), padding=(1,)) (bn4): BatchNorm1d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (pool4): MaxPool1d(kernel_size=4, stride=4, padding=0, dilation=1, ceil_mode=False) (fc1): Linear(in_features=512, out_features=35, bias=True) ) Number of parameters: 571939

M11 network definition

In [37]:
class M11(nn.Module):
    def __init__(self, n_input=1, n_output=35, stride=4, n_channel=64):
        super().__init__()
        self.conv1 = nn.Conv1d(n_input, n_channel, kernel_size=80, stride=stride)
        self.bn1 = nn.BatchNorm1d(n_channel)
        self.pool1 = nn.MaxPool1d(4)

        self.conv2 = nn.Conv1d(n_channel, n_channel, kernel_size=3,padding=1)
        self.bn2 = nn.BatchNorm1d(n_channel)
        self.conv3 = nn.Conv1d(n_channel, n_channel, kernel_size=3,padding=1)
        self.bn3 = nn.BatchNorm1d(n_channel)
        self.pool2 = nn.MaxPool1d(4)

        self.conv4 = nn.Conv1d(n_channel, 2 * n_channel, kernel_size=3,padding=1)
        self.bn4 = nn.BatchNorm1d(2 * n_channel)
        self.conv5 = nn.Conv1d(2 * n_channel, 2 * n_channel, kernel_size=3,padding=1)
        self.bn5 = nn.BatchNorm1d(2 * n_channel)
        self.pool3 = nn.MaxPool1d(4)
 
        self.conv6 = nn.Conv1d(2 * n_channel, 4 * n_channel, kernel_size=3,padding=1)
        self.bn6 = nn.BatchNorm1d(4 * n_channel)
        self.conv7 = nn.Conv1d(4 * n_channel, 4 * n_channel, kernel_size=3,padding=1)
        self.bn7 = nn.BatchNorm1d(4 * n_channel)
        self.conv8 = nn.Conv1d(4 * n_channel, 4 * n_channel, kernel_size=3,padding=1)
        self.bn8 = nn.BatchNorm1d(4 * n_channel)
        self.pool4 = nn.MaxPool1d(4)

        self.conv9 = nn.Conv1d(4 * n_channel, 8 * n_channel, kernel_size=3,padding=1)
        self.bn9 = nn.BatchNorm1d(8 * n_channel)
        self.conv10 = nn.Conv1d(8 * n_channel, 8 * n_channel, kernel_size=3,padding=1)
        self.bn10 = nn.BatchNorm1d(8 * n_channel)

        self.fc1 = nn.Linear(8 * n_channel, n_output)

    def forward(self, x):
        x = self.conv1(x)
        x = F.relu(self.bn1(x))
        x = self.pool1(x)

        x = self.conv2(x)
        x = F.relu(self.bn2(x))
        x = self.conv3(x)
        x = F.relu(self.bn3(x))
        x = self.pool2(x)

        x = self.conv4(x)
        x = F.relu(self.bn4(x))
        x = self.conv5(x)
        x = F.relu(self.bn5(x))
        x = self.pool3(x)

        x = self.conv6(x)
        x = F.relu(self.bn6(x))
        x = self.conv7(x)
        x = F.relu(self.bn7(x))
        x = self.conv8(x)
        x = F.relu(self.bn8(x))
        x = self.pool4(x)

        x = self.conv9(x)
        x = F.relu(self.bn9(x))
        x = self.conv10(x)
        x = F.relu(self.bn10(x))

        x = F.avg_pool1d(x, x.shape[-1])
        x = x.permute(0, 2, 1)
        x = self.fc1(x)
        return F.log_softmax(x, dim=2)

print('M11 model')
print('Model n_input', transformed.shape[0])
print('Model n_output', len(wordclasses))
model_M11 = M11(n_input=transformed.shape[0], n_output=len(wordclasses))
model_M11.to(device)
print(model_M11)

n = count_parameters(model_M11)
print("Number of parameters: %s" % n)
M11 model Model n_input 1 Model n_output 35 M11( (conv1): Conv1d(1, 64, kernel_size=(80,), stride=(4,)) (bn1): BatchNorm1d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (pool1): MaxPool1d(kernel_size=4, stride=4, padding=0, dilation=1, ceil_mode=False) (conv2): Conv1d(64, 64, kernel_size=(3,), stride=(1,), padding=(1,)) (bn2): BatchNorm1d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv3): Conv1d(64, 64, kernel_size=(3,), stride=(1,), padding=(1,)) (bn3): BatchNorm1d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (pool2): MaxPool1d(kernel_size=4, stride=4, padding=0, dilation=1, ceil_mode=False) (conv4): Conv1d(64, 128, kernel_size=(3,), stride=(1,), padding=(1,)) (bn4): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv5): Conv1d(128, 128, kernel_size=(3,), stride=(1,), padding=(1,)) (bn5): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (pool3): MaxPool1d(kernel_size=4, stride=4, padding=0, dilation=1, ceil_mode=False) (conv6): Conv1d(128, 256, kernel_size=(3,), stride=(1,), padding=(1,)) (bn6): BatchNorm1d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv7): Conv1d(256, 256, kernel_size=(3,), stride=(1,), padding=(1,)) (bn7): BatchNorm1d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv8): Conv1d(256, 256, kernel_size=(3,), stride=(1,), padding=(1,)) (bn8): BatchNorm1d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (pool4): MaxPool1d(kernel_size=4, stride=4, padding=0, dilation=1, ceil_mode=False) (conv9): Conv1d(256, 512, kernel_size=(3,), stride=(1,), padding=(1,)) (bn9): BatchNorm1d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv10): Conv1d(512, 512, kernel_size=(3,), stride=(1,), padding=(1,)) (bn10): BatchNorm1d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (fc1): Linear(in_features=512, out_features=35, bias=True) ) Number of parameters: 1799267

M18 network definition

In [38]:
class M18(nn.Module):
    def __init__(self, n_input=1, n_output=35, stride=4, n_channel=64):
        super().__init__()
        self.conv1 = nn.Conv1d(n_input, n_channel, kernel_size=80, stride=stride)
        self.bn1 = nn.BatchNorm1d(n_channel)
        self.pool1 = nn.MaxPool1d(4, stride=None)

        self.conv2 = nn.Conv1d(n_channel, n_channel, kernel_size=3,padding=1)
        self.bn2 = nn.BatchNorm1d(n_channel)
        self.conv3 = nn.Conv1d(n_channel, n_channel, kernel_size=3,padding=1)
        self.bn3 = nn.BatchNorm1d(n_channel)
        self.conv4 = nn.Conv1d(n_channel, n_channel, kernel_size=3,padding=1)
        self.bn4 = nn.BatchNorm1d(n_channel)
        self.conv5 = nn.Conv1d(n_channel, n_channel, kernel_size=3,padding=1)
        self.bn5 = nn.BatchNorm1d(n_channel)
        self.pool2 = nn.MaxPool1d(4, stride=None)

        self.conv6 = nn.Conv1d(n_channel, 2 * n_channel, kernel_size=3,padding=1)
        self.bn6 = nn.BatchNorm1d(2 * n_channel)
        self.conv7 = nn.Conv1d(2 * n_channel, 2 * n_channel, kernel_size=3,padding=1)
        self.bn7 = nn.BatchNorm1d(2 * n_channel)
        self.conv8 = nn.Conv1d(2 * n_channel, 2 * n_channel, kernel_size=3,padding=1)
        self.bn8 = nn.BatchNorm1d(2 * n_channel)
        self.conv9 = nn.Conv1d(2 * n_channel, 2 * n_channel, kernel_size=3,padding=1)
        self.bn9 = nn.BatchNorm1d(2 * n_channel)
        self.pool3 = nn.MaxPool1d(4, stride=None)
 
        self.conv10 = nn.Conv1d(2 * n_channel, 4 * n_channel, kernel_size=3,padding=1)
        self.bn10 = nn.BatchNorm1d(4 * n_channel)
        self.conv11 = nn.Conv1d(4 * n_channel, 4 * n_channel, kernel_size=3,padding=1)
        self.bn11 = nn.BatchNorm1d(4 * n_channel)
        self.conv12 = nn.Conv1d(4 * n_channel, 4 * n_channel, kernel_size=3,padding=1)
        self.bn12 = nn.BatchNorm1d(4 * n_channel)
        self.conv13 = nn.Conv1d(4 * n_channel, 4 * n_channel, kernel_size=3,padding=1)
        self.bn13 = nn.BatchNorm1d(4 * n_channel)
        self.pool4 = nn.MaxPool1d(4, stride=None)

        self.conv14 = nn.Conv1d(4 * n_channel, 8 * n_channel, kernel_size=3,padding=1)
        self.bn14 = nn.BatchNorm1d(8 * n_channel)
        self.conv15 = nn.Conv1d(8 * n_channel, 8 * n_channel, kernel_size=3,padding=1)
        self.bn15 = nn.BatchNorm1d(8 * n_channel)
        self.conv16 = nn.Conv1d(8 * n_channel, 8 * n_channel, kernel_size=3,padding=1)
        self.bn16 = nn.BatchNorm1d(8 * n_channel)
        self.conv17 = nn.Conv1d(8 * n_channel, 8 * n_channel, kernel_size=3,padding=1)
        self.bn17 = nn.BatchNorm1d(8 * n_channel)

        self.fc1 = nn.Linear(8 * n_channel, n_output)

    def forward(self, x):
        x = self.conv1(x)
        x = F.relu(self.bn1(x))
        x = self.pool1(x)

        x = self.conv2(x)
        x = F.relu(self.bn2(x))
        x = self.conv3(x)
        x = F.relu(self.bn3(x))
        x = self.conv4(x)
        x = F.relu(self.bn4(x))
        x = self.conv5(x)
        x = F.relu(self.bn5(x))
        x = self.pool2(x)

        x = self.conv6(x)
        x = F.relu(self.bn6(x))
        x = self.conv7(x)
        x = F.relu(self.bn7(x))
        x = self.conv8(x)
        x = F.relu(self.bn8(x))
        x = self.conv9(x)
        x = F.relu(self.bn9(x))
        x = self.pool3(x)

        x = self.conv10(x)
        x = F.relu(self.bn10(x))
        x = self.conv11(x)
        x = F.relu(self.bn11(x))
        x = self.conv12(x)
        x = F.relu(self.bn12(x))
        x = self.conv13(x)
        x = F.relu(self.bn13(x))
        x = self.pool4(x)

        x = self.conv14(x)
        x = F.relu(self.bn14(x))
        x = self.conv15(x)
        x = F.relu(self.bn15(x))
        x = self.conv16(x)
        x = F.relu(self.bn16(x))
        x = self.conv17(x)
        x = F.relu(self.bn17(x))

        x = F.avg_pool1d(x, x.shape[-1])
        x = x.permute(0, 2, 1)
        x = self.fc1(x)
        return F.log_softmax(x, dim=2)

print('M18 model')
print('Model n_input', transformed.shape[0])
print('Model n_output', len(wordclasses))
model_M18 = M18(n_input=transformed.shape[0], n_output=len(wordclasses))
model_M18.to(device)
print(model_M18)

n = count_parameters(model_M18)
print("Number of parameters: %s" % n)
M18 model Model n_input 1 Model n_output 35 M18( (conv1): Conv1d(1, 64, kernel_size=(80,), stride=(4,)) (bn1): BatchNorm1d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (pool1): MaxPool1d(kernel_size=4, stride=4, padding=0, dilation=1, ceil_mode=False) (conv2): Conv1d(64, 64, kernel_size=(3,), stride=(1,), padding=(1,)) (bn2): BatchNorm1d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv3): Conv1d(64, 64, kernel_size=(3,), stride=(1,), padding=(1,)) (bn3): BatchNorm1d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv4): Conv1d(64, 64, kernel_size=(3,), stride=(1,), padding=(1,)) (bn4): BatchNorm1d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv5): Conv1d(64, 64, kernel_size=(3,), stride=(1,), padding=(1,)) (bn5): BatchNorm1d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (pool2): MaxPool1d(kernel_size=4, stride=4, padding=0, dilation=1, ceil_mode=False) (conv6): Conv1d(64, 128, kernel_size=(3,), stride=(1,), padding=(1,)) (bn6): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv7): Conv1d(128, 128, kernel_size=(3,), stride=(1,), padding=(1,)) (bn7): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv8): Conv1d(128, 128, kernel_size=(3,), stride=(1,), padding=(1,)) (bn8): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv9): Conv1d(128, 128, kernel_size=(3,), stride=(1,), padding=(1,)) (bn9): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (pool3): MaxPool1d(kernel_size=4, stride=4, padding=0, dilation=1, ceil_mode=False) (conv10): Conv1d(128, 256, kernel_size=(3,), stride=(1,), padding=(1,)) (bn10): BatchNorm1d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv11): Conv1d(256, 256, kernel_size=(3,), stride=(1,), padding=(1,)) (bn11): BatchNorm1d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv12): Conv1d(256, 256, kernel_size=(3,), stride=(1,), padding=(1,)) (bn12): BatchNorm1d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv13): Conv1d(256, 256, kernel_size=(3,), stride=(1,), padding=(1,)) (bn13): BatchNorm1d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (pool4): MaxPool1d(kernel_size=4, stride=4, padding=0, dilation=1, ceil_mode=False) (conv14): Conv1d(256, 512, kernel_size=(3,), stride=(1,), padding=(1,)) (bn14): BatchNorm1d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv15): Conv1d(512, 512, kernel_size=(3,), stride=(1,), padding=(1,)) (bn15): BatchNorm1d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv16): Conv1d(512, 512, kernel_size=(3,), stride=(1,), padding=(1,)) (bn16): BatchNorm1d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv17): Conv1d(512, 512, kernel_size=(3,), stride=(1,), padding=(1,)) (bn17): BatchNorm1d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (fc1): Linear(in_features=512, out_features=35, bias=True) ) Number of parameters: 3696611
In [39]:
def train(model, epoch, log_interval):
    model.train()
    right = 0
    for batch_index, (data, target) in enumerate(train_loader):

        data = data.to(device)
        target = target.to(device)

        # apply transform and model on whole batch directly on device
        data = transform(data)
        output = model(data)

        pred = get_probable_idx(output)
        right += nr_of_right(pred, target)

        # negative log-likelihood for a tensor of size (batch x 1 x n_output)
        loss = F.nll_loss(output.squeeze(), target)

        optimiser.zero_grad()
        loss.backward()
        optimiser.step()

        # print training stats
        if batch_index % log_interval == 0:
            print(f"Train Epoch: {epoch} [{batch_index * len(data)}/{len(train_loader.dataset)} ({100. * batch_index / len(train_loader):.0f}%)]\tLoss: {loss.item():.6f}\tAccuracy: {right}/{len(train_loader.dataset)} ({100. * right / len(train_loader.dataset):.0f}%)")
  
        # update progress bar
        pbar.update(pbar_update)
        # record loss
        losses_train.append(loss.item())

    acc = 100. * (right/len(train_loader.dataset))
    accuracy_train.append(acc)
In [40]:
def nr_of_right(pred, target):
    # count nr of right predictions
    return pred.squeeze().eq(target).sum().item()
In [41]:
def get_probable_idx(tensor):
    # find most probable wordclass index for each element in the batch
    return tensor.argmax(dim=-1)
In [42]:
def validate(model, epoch):
    #Stop training
    model.eval()
    
    right = 0
    for data, target in validation_loader:

        data = data.to(device)
        target = target.to(device)

        # apply transform and model on whole batch directly on device
        data = transform(data)
        output = model(data)

        pred = get_probable_idx(output)
        right += nr_of_right(pred, target)

        # negative log-likelihood for a tensor of size (batch x 1 x n_output)
        loss = F.nll_loss(output.squeeze(), target)

        # update progress bar
        pbar.update(pbar_update)

    print(f"\nValidation Epoch: {epoch} \tLoss: {loss.item():.6f}\tAccuracy: {right}/{len(validation_loader.dataset)} ({100. * right / len(validation_loader.dataset):.0f}%)\n")
     
    acc = 100. * right / len(validation_loader.dataset)
    accuracy_validation.append(acc)
    losses_validation.append(loss.item())
In [43]:
def test(model):
    #Stop training
    model.eval()
    
    right = 0
    for data, target in test_loader:

        data = data.to(device)
        target = target.to(device)

        # apply transform and model on whole batch directly on device
        data = transform(data)
        output = model(data)

        pred = get_probable_idx(output)
        right += nr_of_right(pred, target)

    print(f"\nTest set accuracy: {right}/{len(test_loader.dataset)} ({100. * right / len(test_loader.dataset):.0f}%)\n")

    return (100. * right / len(test_loader.dataset))

M3 Train

In [44]:
#Set the epochs
epoch_count = 30
In [45]:
!pip install jovian --upgrade --quiet
In [46]:
import jovian
In [47]:
jovian.log_hyperparams(arch='M3', 
                       epochs=epoch_count, 
                       lr=0.01, 
                       scheduler='optim.lr_scheduler.StepLR', 
                       weight_decay=0.0001, 
                       opt='Adam')
[jovian] Hyperparams logged.
In [48]:
#M3 train & validate
log_interval = 20

pbar_update = 1 / (len(train_loader) + len(test_loader))
losses_train = []
losses_validation = []
accuracy_train = []
accuracy_validation = []

#Use Adam optimiser as used in paper. Weight decay set to 0.0001 
#Initially train with learning rate of 0.01. Use a scheduler to decrease it to 0.001 during training after 20 epochs
optimiser = optim.Adam(model_M3.parameters(), lr=0.01, weight_decay=0.0001)
scheduler = optim.lr_scheduler.StepLR(optimiser, step_size=20, gamma=0.1)  # reduce the learning after 20 epochs by a factor of 10

# Transform must be on same device as model and data
transform = transform.to(device)
with tqdm(total=epoch_count) as pbar:
    for epoch in range(1, epoch_count + 1):
        train(model_M3, epoch, log_interval)
        validate(model_M3, epoch)
        scheduler.step()
HBox(children=(FloatProgress(value=0.0, max=30.0), HTML(value='')))
Train Epoch: 1 [0/84843 (0%)] Loss: 3.611686 Accuracy: 11/84843 (0%) Train Epoch: 1 [8000/84843 (9%)] Loss: 3.209989 Accuracy: 641/84843 (1%) Train Epoch: 1 [16000/84843 (19%)] Loss: 3.118127 Accuracy: 1633/84843 (2%) Train Epoch: 1 [24000/84843 (28%)] Loss: 2.938745 Accuracy: 2795/84843 (3%) Train Epoch: 1 [32000/84843 (38%)] Loss: 2.923880 Accuracy: 4078/84843 (5%) Train Epoch: 1 [40000/84843 (47%)] Loss: 2.766141 Accuracy: 5458/84843 (6%) Train Epoch: 1 [48000/84843 (56%)] Loss: 2.772159 Accuracy: 7080/84843 (8%) Train Epoch: 1 [56000/84843 (66%)] Loss: 2.741093 Accuracy: 8944/84843 (11%) Train Epoch: 1 [64000/84843 (75%)] Loss: 2.582795 Accuracy: 10838/84843 (13%) Train Epoch: 1 [72000/84843 (85%)] Loss: 2.589050 Accuracy: 12897/84843 (15%) Train Epoch: 1 [80000/84843 (94%)] Loss: 2.605732 Accuracy: 15109/84843 (18%) Validation Epoch: 1 Loss: 1.420874 Accuracy: 2879/9981 (29%) Train Epoch: 2 [0/84843 (0%)] Loss: 2.606442 Accuracy: 96/84843 (0%) Train Epoch: 2 [8000/84843 (9%)] Loss: 2.487908 Accuracy: 2272/84843 (3%) Train Epoch: 2 [16000/84843 (19%)] Loss: 2.364818 Accuracy: 4579/84843 (5%) Train Epoch: 2 [24000/84843 (28%)] Loss: 2.579946 Accuracy: 7020/84843 (8%) Train Epoch: 2 [32000/84843 (38%)] Loss: 2.379199 Accuracy: 9534/84843 (11%) Train Epoch: 2 [40000/84843 (47%)] Loss: 2.490372 Accuracy: 12114/84843 (14%) Train Epoch: 2 [48000/84843 (56%)] Loss: 2.440732 Accuracy: 14692/84843 (17%) Train Epoch: 2 [56000/84843 (66%)] Loss: 2.265979 Accuracy: 17301/84843 (20%) Train Epoch: 2 [64000/84843 (75%)] Loss: 2.454190 Accuracy: 19879/84843 (23%) Train Epoch: 2 [72000/84843 (85%)] Loss: 2.364256 Accuracy: 22642/84843 (27%) Train Epoch: 2 [80000/84843 (94%)] Loss: 2.271595 Accuracy: 25433/84843 (30%) Validation Epoch: 2 Loss: 2.081413 Accuracy: 3259/9981 (33%) Train Epoch: 3 [0/84843 (0%)] Loss: 2.305207 Accuracy: 132/84843 (0%) Train Epoch: 3 [8000/84843 (9%)] Loss: 2.246730 Accuracy: 2935/84843 (3%) Train Epoch: 3 [16000/84843 (19%)] Loss: 2.238365 Accuracy: 5695/84843 (7%) Train Epoch: 3 [24000/84843 (28%)] Loss: 2.122352 Accuracy: 8613/84843 (10%) Train Epoch: 3 [32000/84843 (38%)] Loss: 2.154294 Accuracy: 11516/84843 (14%) Train Epoch: 3 [40000/84843 (47%)] Loss: 2.213515 Accuracy: 14342/84843 (17%) Train Epoch: 3 [48000/84843 (56%)] Loss: 2.078036 Accuracy: 17431/84843 (21%) Train Epoch: 3 [56000/84843 (66%)] Loss: 2.223289 Accuracy: 20414/84843 (24%) Train Epoch: 3 [64000/84843 (75%)] Loss: 2.220011 Accuracy: 23465/84843 (28%) Train Epoch: 3 [72000/84843 (85%)] Loss: 2.220502 Accuracy: 26453/84843 (31%) Train Epoch: 3 [80000/84843 (94%)] Loss: 2.294836 Accuracy: 29456/84843 (35%) Validation Epoch: 3 Loss: 1.389359 Accuracy: 3345/9981 (34%) Train Epoch: 4 [0/84843 (0%)] Loss: 2.312782 Accuracy: 146/84843 (0%) Train Epoch: 4 [8000/84843 (9%)] Loss: 2.257536 Accuracy: 3195/84843 (4%) Train Epoch: 4 [16000/84843 (19%)] Loss: 2.191698 Accuracy: 6346/84843 (7%) Train Epoch: 4 [24000/84843 (28%)] Loss: 2.203774 Accuracy: 9551/84843 (11%) Train Epoch: 4 [32000/84843 (38%)] Loss: 2.123116 Accuracy: 12811/84843 (15%) Train Epoch: 4 [40000/84843 (47%)] Loss: 2.154865 Accuracy: 15928/84843 (19%) Train Epoch: 4 [48000/84843 (56%)] Loss: 2.178312 Accuracy: 19262/84843 (23%) Train Epoch: 4 [56000/84843 (66%)] Loss: 2.159049 Accuracy: 22494/84843 (27%) Train Epoch: 4 [64000/84843 (75%)] Loss: 2.244519 Accuracy: 25780/84843 (30%) Train Epoch: 4 [72000/84843 (85%)] Loss: 1.997047 Accuracy: 29084/84843 (34%) Train Epoch: 4 [80000/84843 (94%)] Loss: 2.037792 Accuracy: 32396/84843 (38%) Validation Epoch: 4 Loss: 2.624174 Accuracy: 3499/9981 (35%) Train Epoch: 5 [0/84843 (0%)] Loss: 2.257257 Accuracy: 147/84843 (0%) Train Epoch: 5 [8000/84843 (9%)] Loss: 2.059512 Accuracy: 3386/84843 (4%) Train Epoch: 5 [16000/84843 (19%)] Loss: 2.043615 Accuracy: 6718/84843 (8%) Train Epoch: 5 [24000/84843 (28%)] Loss: 1.986933 Accuracy: 10071/84843 (12%) Train Epoch: 5 [32000/84843 (38%)] Loss: 1.972727 Accuracy: 13493/84843 (16%) Train Epoch: 5 [40000/84843 (47%)] Loss: 1.903362 Accuracy: 16960/84843 (20%) Train Epoch: 5 [48000/84843 (56%)] Loss: 2.164977 Accuracy: 20358/84843 (24%) Train Epoch: 5 [56000/84843 (66%)] Loss: 2.003762 Accuracy: 23866/84843 (28%) Train Epoch: 5 [64000/84843 (75%)] Loss: 1.975872 Accuracy: 27360/84843 (32%) Train Epoch: 5 [72000/84843 (85%)] Loss: 2.107402 Accuracy: 30859/84843 (36%) Train Epoch: 5 [80000/84843 (94%)] Loss: 1.952282 Accuracy: 34387/84843 (41%) Validation Epoch: 5 Loss: 1.675685 Accuracy: 3787/9981 (38%) Train Epoch: 6 [0/84843 (0%)] Loss: 2.120373 Accuracy: 172/84843 (0%) Train Epoch: 6 [8000/84843 (9%)] Loss: 2.107994 Accuracy: 3590/84843 (4%) Train Epoch: 6 [16000/84843 (19%)] Loss: 1.882550 Accuracy: 7217/84843 (9%) Train Epoch: 6 [24000/84843 (28%)] Loss: 1.987374 Accuracy: 10748/84843 (13%) Train Epoch: 6 [32000/84843 (38%)] Loss: 2.111754 Accuracy: 14314/84843 (17%) Train Epoch: 6 [40000/84843 (47%)] Loss: 2.015441 Accuracy: 17831/84843 (21%) Train Epoch: 6 [48000/84843 (56%)] Loss: 1.901028 Accuracy: 21434/84843 (25%) Train Epoch: 6 [56000/84843 (66%)] Loss: 1.859342 Accuracy: 25214/84843 (30%) Train Epoch: 6 [64000/84843 (75%)] Loss: 2.015305 Accuracy: 28911/84843 (34%) Train Epoch: 6 [72000/84843 (85%)] Loss: 2.066906 Accuracy: 32472/84843 (38%) Train Epoch: 6 [80000/84843 (94%)] Loss: 2.063489 Accuracy: 36153/84843 (43%) Validation Epoch: 6 Loss: 1.539431 Accuracy: 4326/9981 (43%) Train Epoch: 7 [0/84843 (0%)] Loss: 2.024615 Accuracy: 188/84843 (0%) Train Epoch: 7 [8000/84843 (9%)] Loss: 2.009978 Accuracy: 3780/84843 (4%) Train Epoch: 7 [16000/84843 (19%)] Loss: 2.010434 Accuracy: 7478/84843 (9%) Train Epoch: 7 [24000/84843 (28%)] Loss: 1.815061 Accuracy: 11169/84843 (13%) Train Epoch: 7 [32000/84843 (38%)] Loss: 2.164404 Accuracy: 14792/84843 (17%) Train Epoch: 7 [40000/84843 (47%)] Loss: 1.930096 Accuracy: 18487/84843 (22%) Train Epoch: 7 [48000/84843 (56%)] Loss: 1.816808 Accuracy: 22345/84843 (26%) Train Epoch: 7 [56000/84843 (66%)] Loss: 1.901334 Accuracy: 26133/84843 (31%) Train Epoch: 7 [64000/84843 (75%)] Loss: 1.939349 Accuracy: 29883/84843 (35%) Train Epoch: 7 [72000/84843 (85%)] Loss: 2.012348 Accuracy: 33683/84843 (40%) Train Epoch: 7 [80000/84843 (94%)] Loss: 1.711180 Accuracy: 37528/84843 (44%) Validation Epoch: 7 Loss: 1.452380 Accuracy: 4606/9981 (46%) Train Epoch: 8 [0/84843 (0%)] Loss: 1.872939 Accuracy: 201/84843 (0%) Train Epoch: 8 [8000/84843 (9%)] Loss: 1.758574 Accuracy: 3895/84843 (5%) Train Epoch: 8 [16000/84843 (19%)] Loss: 1.894235 Accuracy: 7757/84843 (9%) Train Epoch: 8 [24000/84843 (28%)] Loss: 1.879978 Accuracy: 11626/84843 (14%) Train Epoch: 8 [32000/84843 (38%)] Loss: 1.906717 Accuracy: 15456/84843 (18%) Train Epoch: 8 [40000/84843 (47%)] Loss: 1.895869 Accuracy: 19319/84843 (23%) Train Epoch: 8 [48000/84843 (56%)] Loss: 2.044375 Accuracy: 23222/84843 (27%) Train Epoch: 8 [56000/84843 (66%)] Loss: 1.855077 Accuracy: 27066/84843 (32%) Train Epoch: 8 [64000/84843 (75%)] Loss: 1.879273 Accuracy: 30956/84843 (36%) Train Epoch: 8 [72000/84843 (85%)] Loss: 1.805776 Accuracy: 34874/84843 (41%) Train Epoch: 8 [80000/84843 (94%)] Loss: 1.806621 Accuracy: 38823/84843 (46%) Validation Epoch: 8 Loss: 1.379785 Accuracy: 4608/9981 (46%) Train Epoch: 9 [0/84843 (0%)] Loss: 2.058289 Accuracy: 186/84843 (0%) Train Epoch: 9 [8000/84843 (9%)] Loss: 1.856989 Accuracy: 4036/84843 (5%) Train Epoch: 9 [16000/84843 (19%)] Loss: 1.789432 Accuracy: 7888/84843 (9%) Train Epoch: 9 [24000/84843 (28%)] Loss: 2.091699 Accuracy: 11742/84843 (14%) Train Epoch: 9 [32000/84843 (38%)] Loss: 1.858688 Accuracy: 15632/84843 (18%) Train Epoch: 9 [40000/84843 (47%)] Loss: 1.812303 Accuracy: 19602/84843 (23%) Train Epoch: 9 [48000/84843 (56%)] Loss: 1.817957 Accuracy: 23607/84843 (28%) Train Epoch: 9 [56000/84843 (66%)] Loss: 1.971747 Accuracy: 27618/84843 (33%) Train Epoch: 9 [64000/84843 (75%)] Loss: 1.921376 Accuracy: 31606/84843 (37%) Train Epoch: 9 [72000/84843 (85%)] Loss: 1.820020 Accuracy: 35665/84843 (42%) Train Epoch: 9 [80000/84843 (94%)] Loss: 1.805140 Accuracy: 39688/84843 (47%) Validation Epoch: 9 Loss: 1.755788 Accuracy: 4245/9981 (43%) Train Epoch: 10 [0/84843 (0%)] Loss: 1.677053 Accuracy: 219/84843 (0%) Train Epoch: 10 [8000/84843 (9%)] Loss: 1.770779 Accuracy: 4202/84843 (5%) Train Epoch: 10 [16000/84843 (19%)] Loss: 1.868864 Accuracy: 8121/84843 (10%) Train Epoch: 10 [24000/84843 (28%)] Loss: 1.858415 Accuracy: 12144/84843 (14%) Train Epoch: 10 [32000/84843 (38%)] Loss: 1.705866 Accuracy: 16225/84843 (19%) Train Epoch: 10 [40000/84843 (47%)] Loss: 1.676852 Accuracy: 20323/84843 (24%) Train Epoch: 10 [48000/84843 (56%)] Loss: 1.789555 Accuracy: 24424/84843 (29%) Train Epoch: 10 [56000/84843 (66%)] Loss: 1.657589 Accuracy: 28525/84843 (34%) Train Epoch: 10 [64000/84843 (75%)] Loss: 1.708552 Accuracy: 32665/84843 (39%) Train Epoch: 10 [72000/84843 (85%)] Loss: 1.682401 Accuracy: 36804/84843 (43%) Train Epoch: 10 [80000/84843 (94%)] Loss: 1.729356 Accuracy: 40887/84843 (48%) Validation Epoch: 10 Loss: 2.256757 Accuracy: 3935/9981 (39%) Train Epoch: 11 [0/84843 (0%)] Loss: 1.697600 Accuracy: 209/84843 (0%) Train Epoch: 11 [8000/84843 (9%)] Loss: 1.802842 Accuracy: 4250/84843 (5%) Train Epoch: 11 [16000/84843 (19%)] Loss: 1.708888 Accuracy: 8328/84843 (10%) Train Epoch: 11 [24000/84843 (28%)] Loss: 1.856585 Accuracy: 12552/84843 (15%) Train Epoch: 11 [32000/84843 (38%)] Loss: 1.886523 Accuracy: 16711/84843 (20%) Train Epoch: 11 [40000/84843 (47%)] Loss: 1.624810 Accuracy: 20867/84843 (25%) Train Epoch: 11 [48000/84843 (56%)] Loss: 1.962589 Accuracy: 25059/84843 (30%) Train Epoch: 11 [56000/84843 (66%)] Loss: 1.721665 Accuracy: 29253/84843 (34%) Train Epoch: 11 [64000/84843 (75%)] Loss: 1.734469 Accuracy: 33394/84843 (39%) Train Epoch: 11 [72000/84843 (85%)] Loss: 1.767979 Accuracy: 37608/84843 (44%) Train Epoch: 11 [80000/84843 (94%)] Loss: 1.602943 Accuracy: 41885/84843 (49%) Validation Epoch: 11 Loss: 1.485942 Accuracy: 4833/9981 (48%) Train Epoch: 12 [0/84843 (0%)] Loss: 1.762754 Accuracy: 210/84843 (0%) Train Epoch: 12 [8000/84843 (9%)] Loss: 1.751369 Accuracy: 4308/84843 (5%) Train Epoch: 12 [16000/84843 (19%)] Loss: 1.738923 Accuracy: 8573/84843 (10%) Train Epoch: 12 [24000/84843 (28%)] Loss: 1.638562 Accuracy: 12809/84843 (15%) Train Epoch: 12 [32000/84843 (38%)] Loss: 1.666570 Accuracy: 16934/84843 (20%) Train Epoch: 12 [40000/84843 (47%)] Loss: 1.753682 Accuracy: 21168/84843 (25%) Train Epoch: 12 [48000/84843 (56%)] Loss: 1.752423 Accuracy: 25387/84843 (30%) Train Epoch: 12 [56000/84843 (66%)] Loss: 1.794779 Accuracy: 29685/84843 (35%) Train Epoch: 12 [64000/84843 (75%)] Loss: 1.645765 Accuracy: 34004/84843 (40%) Train Epoch: 12 [72000/84843 (85%)] Loss: 1.733935 Accuracy: 38294/84843 (45%) Train Epoch: 12 [80000/84843 (94%)] Loss: 1.745291 Accuracy: 42619/84843 (50%) Validation Epoch: 12 Loss: 1.247117 Accuracy: 5127/9981 (51%) Train Epoch: 13 [0/84843 (0%)] Loss: 1.759268 Accuracy: 215/84843 (0%) Train Epoch: 13 [8000/84843 (9%)] Loss: 1.746645 Accuracy: 4493/84843 (5%) Train Epoch: 13 [16000/84843 (19%)] Loss: 1.661200 Accuracy: 8950/84843 (11%) Train Epoch: 13 [24000/84843 (28%)] Loss: 1.708070 Accuracy: 13346/84843 (16%) Train Epoch: 13 [32000/84843 (38%)] Loss: 1.643022 Accuracy: 17758/84843 (21%) Train Epoch: 13 [40000/84843 (47%)] Loss: 1.710071 Accuracy: 22066/84843 (26%) Train Epoch: 13 [48000/84843 (56%)] Loss: 1.646698 Accuracy: 26399/84843 (31%) Train Epoch: 13 [56000/84843 (66%)] Loss: 1.588037 Accuracy: 30829/84843 (36%) Train Epoch: 13 [64000/84843 (75%)] Loss: 1.610463 Accuracy: 35170/84843 (41%) Train Epoch: 13 [72000/84843 (85%)] Loss: 1.566988 Accuracy: 39620/84843 (47%) Train Epoch: 13 [80000/84843 (94%)] Loss: 1.761059 Accuracy: 44010/84843 (52%) Validation Epoch: 13 Loss: 1.516118 Accuracy: 4499/9981 (45%) Train Epoch: 14 [0/84843 (0%)] Loss: 1.814068 Accuracy: 218/84843 (0%) Train Epoch: 14 [8000/84843 (9%)] Loss: 1.615931 Accuracy: 4554/84843 (5%) Train Epoch: 14 [16000/84843 (19%)] Loss: 1.575307 Accuracy: 8942/84843 (11%) Train Epoch: 14 [24000/84843 (28%)] Loss: 1.690426 Accuracy: 13403/84843 (16%) Train Epoch: 14 [32000/84843 (38%)] Loss: 1.526441 Accuracy: 17892/84843 (21%) Train Epoch: 14 [40000/84843 (47%)] Loss: 1.653955 Accuracy: 22304/84843 (26%) Train Epoch: 14 [48000/84843 (56%)] Loss: 1.586740 Accuracy: 26758/84843 (32%) Train Epoch: 14 [56000/84843 (66%)] Loss: 1.621158 Accuracy: 31308/84843 (37%) Train Epoch: 14 [64000/84843 (75%)] Loss: 1.637319 Accuracy: 35811/84843 (42%) Train Epoch: 14 [72000/84843 (85%)] Loss: 1.549008 Accuracy: 40168/84843 (47%) Train Epoch: 14 [80000/84843 (94%)] Loss: 1.599516 Accuracy: 44596/84843 (53%) Validation Epoch: 14 Loss: 1.630709 Accuracy: 5398/9981 (54%) Train Epoch: 15 [0/84843 (0%)] Loss: 1.790581 Accuracy: 220/84843 (0%) Train Epoch: 15 [8000/84843 (9%)] Loss: 1.674358 Accuracy: 4665/84843 (5%) Train Epoch: 15 [16000/84843 (19%)] Loss: 1.543892 Accuracy: 9158/84843 (11%) Train Epoch: 15 [24000/84843 (28%)] Loss: 1.549493 Accuracy: 13742/84843 (16%) Train Epoch: 15 [32000/84843 (38%)] Loss: 1.586549 Accuracy: 18141/84843 (21%) Train Epoch: 15 [40000/84843 (47%)] Loss: 1.620462 Accuracy: 22588/84843 (27%) Train Epoch: 15 [48000/84843 (56%)] Loss: 1.698940 Accuracy: 27116/84843 (32%) Train Epoch: 15 [56000/84843 (66%)] Loss: 1.648494 Accuracy: 31593/84843 (37%) Train Epoch: 15 [64000/84843 (75%)] Loss: 1.614806 Accuracy: 36151/84843 (43%) Train Epoch: 15 [72000/84843 (85%)] Loss: 1.431829 Accuracy: 40726/84843 (48%) Train Epoch: 15 [80000/84843 (94%)] Loss: 1.720577 Accuracy: 45293/84843 (53%) Validation Epoch: 15 Loss: 1.450931 Accuracy: 5770/9981 (58%) Train Epoch: 16 [0/84843 (0%)] Loss: 1.606430 Accuracy: 227/84843 (0%) Train Epoch: 16 [8000/84843 (9%)] Loss: 1.512588 Accuracy: 4890/84843 (6%) Train Epoch: 16 [16000/84843 (19%)] Loss: 1.588787 Accuracy: 9536/84843 (11%) Train Epoch: 16 [24000/84843 (28%)] Loss: 1.512528 Accuracy: 14212/84843 (17%) Train Epoch: 16 [32000/84843 (38%)] Loss: 1.615800 Accuracy: 18928/84843 (22%) Train Epoch: 16 [40000/84843 (47%)] Loss: 1.695825 Accuracy: 23549/84843 (28%) Train Epoch: 16 [48000/84843 (56%)] Loss: 1.462260 Accuracy: 28124/84843 (33%) Train Epoch: 16 [56000/84843 (66%)] Loss: 1.577972 Accuracy: 32681/84843 (39%) Train Epoch: 16 [64000/84843 (75%)] Loss: 1.614445 Accuracy: 37256/84843 (44%) Train Epoch: 16 [72000/84843 (85%)] Loss: 1.641152 Accuracy: 41806/84843 (49%) Train Epoch: 16 [80000/84843 (94%)] Loss: 1.621347 Accuracy: 46302/84843 (55%) Validation Epoch: 16 Loss: 1.282443 Accuracy: 4724/9981 (47%) Train Epoch: 17 [0/84843 (0%)] Loss: 1.500155 Accuracy: 228/84843 (0%) Train Epoch: 17 [8000/84843 (9%)] Loss: 1.587316 Accuracy: 4891/84843 (6%) Train Epoch: 17 [16000/84843 (19%)] Loss: 1.607518 Accuracy: 9576/84843 (11%) Train Epoch: 17 [24000/84843 (28%)] Loss: 1.455303 Accuracy: 14266/84843 (17%) Train Epoch: 17 [32000/84843 (38%)] Loss: 1.645941 Accuracy: 18918/84843 (22%) Train Epoch: 17 [40000/84843 (47%)] Loss: 1.628074 Accuracy: 23571/84843 (28%) Train Epoch: 17 [48000/84843 (56%)] Loss: 1.567611 Accuracy: 28236/84843 (33%) Train Epoch: 17 [56000/84843 (66%)] Loss: 1.727864 Accuracy: 32971/84843 (39%) Train Epoch: 17 [64000/84843 (75%)] Loss: 1.509888 Accuracy: 37644/84843 (44%) Train Epoch: 17 [72000/84843 (85%)] Loss: 1.427695 Accuracy: 42284/84843 (50%) Train Epoch: 17 [80000/84843 (94%)] Loss: 1.607625 Accuracy: 46850/84843 (55%) Validation Epoch: 17 Loss: 1.827033 Accuracy: 4182/9981 (42%) Train Epoch: 18 [0/84843 (0%)] Loss: 1.467207 Accuracy: 249/84843 (0%) Train Epoch: 18 [8000/84843 (9%)] Loss: 1.672670 Accuracy: 4943/84843 (6%) Train Epoch: 18 [16000/84843 (19%)] Loss: 1.510329 Accuracy: 9740/84843 (11%) Train Epoch: 18 [24000/84843 (28%)] Loss: 1.498154 Accuracy: 14501/84843 (17%) Train Epoch: 18 [32000/84843 (38%)] Loss: 1.584302 Accuracy: 19162/84843 (23%) Train Epoch: 18 [40000/84843 (47%)] Loss: 1.580498 Accuracy: 23808/84843 (28%) Train Epoch: 18 [48000/84843 (56%)] Loss: 1.594174 Accuracy: 28491/84843 (34%) Train Epoch: 18 [56000/84843 (66%)] Loss: 1.500661 Accuracy: 33254/84843 (39%) Train Epoch: 18 [64000/84843 (75%)] Loss: 1.428344 Accuracy: 38097/84843 (45%) Train Epoch: 18 [72000/84843 (85%)] Loss: 1.565974 Accuracy: 42803/84843 (50%) Train Epoch: 18 [80000/84843 (94%)] Loss: 1.454196 Accuracy: 47479/84843 (56%) Validation Epoch: 18 Loss: 1.042869 Accuracy: 5641/9981 (57%) Train Epoch: 19 [0/84843 (0%)] Loss: 1.460743 Accuracy: 240/84843 (0%) Train Epoch: 19 [8000/84843 (9%)] Loss: 1.583013 Accuracy: 4797/84843 (6%) Train Epoch: 19 [16000/84843 (19%)] Loss: 1.360214 Accuracy: 9502/84843 (11%) Train Epoch: 19 [24000/84843 (28%)] Loss: 1.545313 Accuracy: 14305/84843 (17%) Train Epoch: 19 [32000/84843 (38%)] Loss: 1.394801 Accuracy: 19097/84843 (23%) Train Epoch: 19 [40000/84843 (47%)] Loss: 1.617300 Accuracy: 23865/84843 (28%) Train Epoch: 19 [48000/84843 (56%)] Loss: 1.547934 Accuracy: 28756/84843 (34%) Train Epoch: 19 [56000/84843 (66%)] Loss: 1.651792 Accuracy: 33536/84843 (40%) Train Epoch: 19 [64000/84843 (75%)] Loss: 1.550510 Accuracy: 38287/84843 (45%) Train Epoch: 19 [72000/84843 (85%)] Loss: 1.505254 Accuracy: 43099/84843 (51%) Train Epoch: 19 [80000/84843 (94%)] Loss: 1.439458 Accuracy: 47979/84843 (57%) Validation Epoch: 19 Loss: 0.945859 Accuracy: 5445/9981 (55%) Train Epoch: 20 [0/84843 (0%)] Loss: 1.675861 Accuracy: 229/84843 (0%) Train Epoch: 20 [8000/84843 (9%)] Loss: 1.410609 Accuracy: 5072/84843 (6%) Train Epoch: 20 [16000/84843 (19%)] Loss: 1.269190 Accuracy: 9950/84843 (12%) Train Epoch: 20 [24000/84843 (28%)] Loss: 1.381891 Accuracy: 14925/84843 (18%) Train Epoch: 20 [32000/84843 (38%)] Loss: 1.413683 Accuracy: 19773/84843 (23%) Train Epoch: 20 [40000/84843 (47%)] Loss: 1.350582 Accuracy: 24716/84843 (29%) Train Epoch: 20 [48000/84843 (56%)] Loss: 1.544868 Accuracy: 29528/84843 (35%) Train Epoch: 20 [56000/84843 (66%)] Loss: 1.508256 Accuracy: 34381/84843 (41%) Train Epoch: 20 [64000/84843 (75%)] Loss: 1.476793 Accuracy: 39218/84843 (46%) Train Epoch: 20 [72000/84843 (85%)] Loss: 1.563268 Accuracy: 44081/84843 (52%) Train Epoch: 20 [80000/84843 (94%)] Loss: 1.565778 Accuracy: 48925/84843 (58%) Validation Epoch: 20 Loss: 1.371357 Accuracy: 5446/9981 (55%) Train Epoch: 21 [0/84843 (0%)] Loss: 1.479170 Accuracy: 233/84843 (0%) Train Epoch: 21 [8000/84843 (9%)] Loss: 1.361151 Accuracy: 5217/84843 (6%) Train Epoch: 21 [16000/84843 (19%)] Loss: 1.262655 Accuracy: 10384/84843 (12%) Train Epoch: 21 [24000/84843 (28%)] Loss: 1.204204 Accuracy: 15647/84843 (18%) Train Epoch: 21 [32000/84843 (38%)] Loss: 1.386479 Accuracy: 20909/84843 (25%) Train Epoch: 21 [40000/84843 (47%)] Loss: 1.370883 Accuracy: 26141/84843 (31%) Train Epoch: 21 [48000/84843 (56%)] Loss: 1.235948 Accuracy: 31393/84843 (37%) Train Epoch: 21 [56000/84843 (66%)] Loss: 1.327088 Accuracy: 36656/84843 (43%) Train Epoch: 21 [64000/84843 (75%)] Loss: 1.206203 Accuracy: 41936/84843 (49%) Train Epoch: 21 [72000/84843 (85%)] Loss: 1.295178 Accuracy: 47216/84843 (56%) Train Epoch: 21 [80000/84843 (94%)] Loss: 1.490041 Accuracy: 52495/84843 (62%) Validation Epoch: 21 Loss: 1.080816 Accuracy: 6563/9981 (66%) Train Epoch: 22 [0/84843 (0%)] Loss: 1.298124 Accuracy: 260/84843 (0%) Train Epoch: 22 [8000/84843 (9%)] Loss: 1.378322 Accuracy: 5562/84843 (7%) Train Epoch: 22 [16000/84843 (19%)] Loss: 1.131456 Accuracy: 10856/84843 (13%) Train Epoch: 22 [24000/84843 (28%)] Loss: 1.167584 Accuracy: 16198/84843 (19%) Train Epoch: 22 [32000/84843 (38%)] Loss: 1.343730 Accuracy: 21530/84843 (25%) Train Epoch: 22 [40000/84843 (47%)] Loss: 1.277439 Accuracy: 26802/84843 (32%) Train Epoch: 22 [48000/84843 (56%)] Loss: 1.323701 Accuracy: 32093/84843 (38%) Train Epoch: 22 [56000/84843 (66%)] Loss: 1.427235 Accuracy: 37372/84843 (44%) Train Epoch: 22 [64000/84843 (75%)] Loss: 1.287931 Accuracy: 42741/84843 (50%) Train Epoch: 22 [72000/84843 (85%)] Loss: 1.232031 Accuracy: 48019/84843 (57%) Train Epoch: 22 [80000/84843 (94%)] Loss: 1.264105 Accuracy: 53376/84843 (63%) Validation Epoch: 22 Loss: 1.116131 Accuracy: 6600/9981 (66%) Train Epoch: 23 [0/84843 (0%)] Loss: 1.283160 Accuracy: 269/84843 (0%) Train Epoch: 23 [8000/84843 (9%)] Loss: 1.248599 Accuracy: 5642/84843 (7%) Train Epoch: 23 [16000/84843 (19%)] Loss: 1.280800 Accuracy: 11042/84843 (13%) Train Epoch: 23 [24000/84843 (28%)] Loss: 1.279341 Accuracy: 16403/84843 (19%) Train Epoch: 23 [32000/84843 (38%)] Loss: 1.251322 Accuracy: 21697/84843 (26%) Train Epoch: 23 [40000/84843 (47%)] Loss: 1.204175 Accuracy: 27080/84843 (32%) Train Epoch: 23 [48000/84843 (56%)] Loss: 1.239731 Accuracy: 32459/84843 (38%) Train Epoch: 23 [56000/84843 (66%)] Loss: 1.323401 Accuracy: 37667/84843 (44%) Train Epoch: 23 [64000/84843 (75%)] Loss: 1.354200 Accuracy: 42980/84843 (51%) Train Epoch: 23 [72000/84843 (85%)] Loss: 1.260509 Accuracy: 48358/84843 (57%) Train Epoch: 23 [80000/84843 (94%)] Loss: 1.395313 Accuracy: 53699/84843 (63%) Validation Epoch: 23 Loss: 1.009252 Accuracy: 6642/9981 (67%) Train Epoch: 24 [0/84843 (0%)] Loss: 1.183875 Accuracy: 283/84843 (0%) Train Epoch: 24 [8000/84843 (9%)] Loss: 1.305216 Accuracy: 5619/84843 (7%) Train Epoch: 24 [16000/84843 (19%)] Loss: 1.129976 Accuracy: 11037/84843 (13%) Train Epoch: 24 [24000/84843 (28%)] Loss: 1.145497 Accuracy: 16398/84843 (19%) Train Epoch: 24 [32000/84843 (38%)] Loss: 1.264149 Accuracy: 21788/84843 (26%) Train Epoch: 24 [40000/84843 (47%)] Loss: 1.273404 Accuracy: 27191/84843 (32%) Train Epoch: 24 [48000/84843 (56%)] Loss: 1.334382 Accuracy: 32534/84843 (38%) Train Epoch: 24 [56000/84843 (66%)] Loss: 1.277816 Accuracy: 37842/84843 (45%) Train Epoch: 24 [64000/84843 (75%)] Loss: 1.222430 Accuracy: 43218/84843 (51%) Train Epoch: 24 [72000/84843 (85%)] Loss: 1.192959 Accuracy: 48604/84843 (57%) Train Epoch: 24 [80000/84843 (94%)] Loss: 1.272612 Accuracy: 54003/84843 (64%) Validation Epoch: 24 Loss: 0.948163 Accuracy: 6670/9981 (67%) Train Epoch: 25 [0/84843 (0%)] Loss: 1.274445 Accuracy: 261/84843 (0%) Train Epoch: 25 [8000/84843 (9%)] Loss: 1.164289 Accuracy: 5589/84843 (7%) Train Epoch: 25 [16000/84843 (19%)] Loss: 1.241250 Accuracy: 10956/84843 (13%) Train Epoch: 25 [24000/84843 (28%)] Loss: 1.249167 Accuracy: 16355/84843 (19%) Train Epoch: 25 [32000/84843 (38%)] Loss: 1.296552 Accuracy: 21699/84843 (26%) Train Epoch: 25 [40000/84843 (47%)] Loss: 1.211671 Accuracy: 27126/84843 (32%) Train Epoch: 25 [48000/84843 (56%)] Loss: 1.103432 Accuracy: 32586/84843 (38%) Train Epoch: 25 [56000/84843 (66%)] Loss: 1.128119 Accuracy: 37950/84843 (45%) Train Epoch: 25 [64000/84843 (75%)] Loss: 1.325318 Accuracy: 43329/84843 (51%) Train Epoch: 25 [72000/84843 (85%)] Loss: 1.283307 Accuracy: 48768/84843 (57%) Train Epoch: 25 [80000/84843 (94%)] Loss: 1.239789 Accuracy: 54182/84843 (64%) Validation Epoch: 25 Loss: 1.012084 Accuracy: 6644/9981 (67%) Train Epoch: 26 [0/84843 (0%)] Loss: 1.263106 Accuracy: 262/84843 (0%) Train Epoch: 26 [8000/84843 (9%)] Loss: 1.098730 Accuracy: 5697/84843 (7%) Train Epoch: 26 [16000/84843 (19%)] Loss: 1.256716 Accuracy: 11160/84843 (13%) Train Epoch: 26 [24000/84843 (28%)] Loss: 1.159907 Accuracy: 16595/84843 (20%) Train Epoch: 26 [32000/84843 (38%)] Loss: 1.289369 Accuracy: 22031/84843 (26%) Train Epoch: 26 [40000/84843 (47%)] Loss: 1.265374 Accuracy: 27408/84843 (32%) Train Epoch: 26 [48000/84843 (56%)] Loss: 1.222897 Accuracy: 32823/84843 (39%) Train Epoch: 26 [56000/84843 (66%)] Loss: 1.182980 Accuracy: 38234/84843 (45%) Train Epoch: 26 [64000/84843 (75%)] Loss: 1.355415 Accuracy: 43663/84843 (51%) Train Epoch: 26 [72000/84843 (85%)] Loss: 1.223963 Accuracy: 49023/84843 (58%) Train Epoch: 26 [80000/84843 (94%)] Loss: 1.207160 Accuracy: 54433/84843 (64%) Validation Epoch: 26 Loss: 1.181437 Accuracy: 6646/9981 (67%) Train Epoch: 27 [0/84843 (0%)] Loss: 1.181836 Accuracy: 281/84843 (0%) Train Epoch: 27 [8000/84843 (9%)] Loss: 1.044987 Accuracy: 5687/84843 (7%) Train Epoch: 27 [16000/84843 (19%)] Loss: 1.160532 Accuracy: 11092/84843 (13%) Train Epoch: 27 [24000/84843 (28%)] Loss: 1.147508 Accuracy: 16557/84843 (20%) Train Epoch: 27 [32000/84843 (38%)] Loss: 1.230024 Accuracy: 22010/84843 (26%) Train Epoch: 27 [40000/84843 (47%)] Loss: 1.203670 Accuracy: 27452/84843 (32%) Train Epoch: 27 [48000/84843 (56%)] Loss: 1.310799 Accuracy: 32899/84843 (39%) Train Epoch: 27 [56000/84843 (66%)] Loss: 1.233989 Accuracy: 38280/84843 (45%) Train Epoch: 27 [64000/84843 (75%)] Loss: 1.178283 Accuracy: 43686/84843 (51%) Train Epoch: 27 [72000/84843 (85%)] Loss: 1.357658 Accuracy: 49108/84843 (58%) Train Epoch: 27 [80000/84843 (94%)] Loss: 1.185840 Accuracy: 54625/84843 (64%) Validation Epoch: 27 Loss: 1.035126 Accuracy: 6635/9981 (66%) Train Epoch: 28 [0/84843 (0%)] Loss: 1.296985 Accuracy: 269/84843 (0%) Train Epoch: 28 [8000/84843 (9%)] Loss: 1.178500 Accuracy: 5770/84843 (7%) Train Epoch: 28 [16000/84843 (19%)] Loss: 1.102986 Accuracy: 11300/84843 (13%) Train Epoch: 28 [24000/84843 (28%)] Loss: 1.144303 Accuracy: 16699/84843 (20%) Train Epoch: 28 [32000/84843 (38%)] Loss: 1.304479 Accuracy: 22181/84843 (26%) Train Epoch: 28 [40000/84843 (47%)] Loss: 1.284057 Accuracy: 27576/84843 (33%) Train Epoch: 28 [48000/84843 (56%)] Loss: 1.194289 Accuracy: 33089/84843 (39%) Train Epoch: 28 [56000/84843 (66%)] Loss: 1.269337 Accuracy: 38563/84843 (45%) Train Epoch: 28 [64000/84843 (75%)] Loss: 1.203196 Accuracy: 43980/84843 (52%) Train Epoch: 28 [72000/84843 (85%)] Loss: 1.359989 Accuracy: 49429/84843 (58%) Train Epoch: 28 [80000/84843 (94%)] Loss: 1.273243 Accuracy: 54851/84843 (65%) Validation Epoch: 28 Loss: 1.075013 Accuracy: 6562/9981 (66%) Train Epoch: 29 [0/84843 (0%)] Loss: 1.260172 Accuracy: 265/84843 (0%) Train Epoch: 29 [8000/84843 (9%)] Loss: 1.184878 Accuracy: 5683/84843 (7%) Train Epoch: 29 [16000/84843 (19%)] Loss: 1.159357 Accuracy: 11184/84843 (13%) Train Epoch: 29 [24000/84843 (28%)] Loss: 1.211139 Accuracy: 16677/84843 (20%) Train Epoch: 29 [32000/84843 (38%)] Loss: 1.264163 Accuracy: 22135/84843 (26%) Train Epoch: 29 [40000/84843 (47%)] Loss: 1.161580 Accuracy: 27580/84843 (33%) Train Epoch: 29 [48000/84843 (56%)] Loss: 1.136408 Accuracy: 33021/84843 (39%) Train Epoch: 29 [56000/84843 (66%)] Loss: 1.242439 Accuracy: 38459/84843 (45%) Train Epoch: 29 [64000/84843 (75%)] Loss: 1.278853 Accuracy: 43912/84843 (52%) Train Epoch: 29 [72000/84843 (85%)] Loss: 1.248096 Accuracy: 49480/84843 (58%) Train Epoch: 29 [80000/84843 (94%)] Loss: 1.123959 Accuracy: 54955/84843 (65%) Validation Epoch: 29 Loss: 0.982625 Accuracy: 6775/9981 (68%) Train Epoch: 30 [0/84843 (0%)] Loss: 1.160156 Accuracy: 274/84843 (0%) Train Epoch: 30 [8000/84843 (9%)] Loss: 1.290961 Accuracy: 5775/84843 (7%) Train Epoch: 30 [16000/84843 (19%)] Loss: 1.227991 Accuracy: 11250/84843 (13%) Train Epoch: 30 [24000/84843 (28%)] Loss: 1.080591 Accuracy: 16774/84843 (20%) Train Epoch: 30 [32000/84843 (38%)] Loss: 1.181840 Accuracy: 22222/84843 (26%) Train Epoch: 30 [40000/84843 (47%)] Loss: 1.192722 Accuracy: 27731/84843 (33%) Train Epoch: 30 [48000/84843 (56%)] Loss: 1.325138 Accuracy: 33083/84843 (39%) Train Epoch: 30 [56000/84843 (66%)] Loss: 1.295396 Accuracy: 38552/84843 (45%) Train Epoch: 30 [64000/84843 (75%)] Loss: 1.253654 Accuracy: 44004/84843 (52%) Train Epoch: 30 [72000/84843 (85%)] Loss: 1.095768 Accuracy: 49486/84843 (58%) Train Epoch: 30 [80000/84843 (94%)] Loss: 1.270935 Accuracy: 54997/84843 (65%) Validation Epoch: 30 Loss: 1.239052 Accuracy: 6716/9981 (67%)
In [49]:
# Plot training loss
plt.plot(losses_train, 'b', label='Train loss')
plt.legend(loc="upper left")
plt.title("M3 model training losses over all iterations")
plt.show()

# Plot validation loss
plt.plot(losses_validation, 'r', label='Valid loss')
plt.legend(loc="upper left")
plt.title("M3 model validation losses over validation epochs")
plt.show()

# Plot accuracy
plt.plot(accuracy_train, 'b', label='Train acc')
plt.plot(accuracy_validation,'r', label ='Valid acc')
plt.legend(loc="upper left")
plt.title("M3 model training & validation accuracy over epochs")
plt.show()
Notebook Image
Notebook Image
Notebook Image
In [50]:
M3_test_acc_result = test(model_M3)
print(M3_test_acc_result)
Test set accuracy: 7255/11005 (66%) 65.92457973648342
In [51]:
jovian.log_metrics({ 'M3_test_acc_result': M3_test_acc_result })
[jovian] Metrics logged.
In [52]:
jovian.commit(project=project_name)
[jovian] Detected Colab notebook... [jovian] Uploading colab notebook to Jovian... [jovian] Capturing environment.. [jovian] Attaching records (metrics, hyperparameters, dataset etc.) [jovian] Committed successfully! https://jovian.ai/gtaljaard/assignment-4-speech-command-recognition-with-m3-m5-m11-m18-cnn-networks

M5 Train

In [53]:
!pip install jovian --upgrade --quiet
In [54]:
import jovian
In [55]:
jovian.log_hyperparams(arch='M5', 
                       epochs=epoch_count, 
                       lr=0.01, 
                       scheduler='optim.lr_scheduler.StepLR', 
                       weight_decay=0.0001, 
                       opt='Adam')
[jovian] Hyperparams logged.
In [56]:
#M5 train & validate
log_interval = 20
#epoch_count = 1

pbar_update = 1 / (len(train_loader) + len(test_loader))
losses_train = []
losses_validation = []
accuracy_train = []
accuracy_validation = []

#Use Adam optimiser as used in paper. Weight decay set to 0.0001 
#Initially train with learning rate of 0.01. Use a scheduler to decrease it to 0.001 during training after 20 epochs
optimiser = optim.Adam(model_M5.parameters(), lr=0.01, weight_decay=0.0001)
scheduler = optim.lr_scheduler.StepLR(optimiser, step_size=20, gamma=0.1)  # reduce the learning after 20 epochs by a factor of 10

# Transform must be on same device as model and data
transform = transform.to(device)
with tqdm(total=epoch_count) as pbar:
    for epoch in range(1, epoch_count + 1):
        train(model_M5, epoch, log_interval)
        validate(model_M5, epoch)
        scheduler.step()
HBox(children=(FloatProgress(value=0.0, max=30.0), HTML(value='')))
Train Epoch: 1 [0/84843 (0%)] Loss: 3.679375 Accuracy: 8/84843 (0%) Train Epoch: 1 [8000/84843 (9%)] Loss: 3.200036 Accuracy: 608/84843 (1%) Train Epoch: 1 [16000/84843 (19%)] Loss: 2.821932 Accuracy: 2059/84843 (2%) Train Epoch: 1 [24000/84843 (28%)] Loss: 2.338980 Accuracy: 4311/84843 (5%) Train Epoch: 1 [32000/84843 (38%)] Loss: 2.179656 Accuracy: 7330/84843 (9%) Train Epoch: 1 [40000/84843 (47%)] Loss: 1.892660 Accuracy: 10836/84843 (13%) Train Epoch: 1 [48000/84843 (56%)] Loss: 1.658333 Accuracy: 14654/84843 (17%) Train Epoch: 1 [56000/84843 (66%)] Loss: 1.608859 Accuracy: 18922/84843 (22%) Train Epoch: 1 [64000/84843 (75%)] Loss: 1.516262 Accuracy: 23596/84843 (28%) Train Epoch: 1 [72000/84843 (85%)] Loss: 1.386236 Accuracy: 28420/84843 (33%) Train Epoch: 1 [80000/84843 (94%)] Loss: 1.236619 Accuracy: 33489/84843 (39%) Validation Epoch: 1 Loss: 0.896934 Accuracy: 5177/9981 (52%) Train Epoch: 2 [0/84843 (0%)] Loss: 1.223309 Accuracy: 262/84843 (0%) Train Epoch: 2 [8000/84843 (9%)] Loss: 1.147168 Accuracy: 5555/84843 (7%) Train Epoch: 2 [16000/84843 (19%)] Loss: 1.195996 Accuracy: 11110/84843 (13%) Train Epoch: 2 [24000/84843 (28%)] Loss: 0.926723 Accuracy: 16756/84843 (20%) Train Epoch: 2 [32000/84843 (38%)] Loss: 1.040996 Accuracy: 22601/84843 (27%) Train Epoch: 2 [40000/84843 (47%)] Loss: 0.831601 Accuracy: 28599/84843 (34%) Train Epoch: 2 [48000/84843 (56%)] Loss: 0.966543 Accuracy: 34617/84843 (41%) Train Epoch: 2 [56000/84843 (66%)] Loss: 0.833464 Accuracy: 40605/84843 (48%) Train Epoch: 2 [64000/84843 (75%)] Loss: 0.866042 Accuracy: 46753/84843 (55%) Train Epoch: 2 [72000/84843 (85%)] Loss: 0.785763 Accuracy: 52977/84843 (62%) Train Epoch: 2 [80000/84843 (94%)] Loss: 0.738287 Accuracy: 59242/84843 (70%) Validation Epoch: 2 Loss: 0.744588 Accuracy: 5202/9981 (52%) Train Epoch: 3 [0/84843 (0%)] Loss: 0.874572 Accuracy: 301/84843 (0%) Train Epoch: 3 [8000/84843 (9%)] Loss: 0.797574 Accuracy: 6508/84843 (8%) Train Epoch: 3 [16000/84843 (19%)] Loss: 0.646058 Accuracy: 12925/84843 (15%) Train Epoch: 3 [24000/84843 (28%)] Loss: 0.693227 Accuracy: 19458/84843 (23%) Train Epoch: 3 [32000/84843 (38%)] Loss: 0.674762 Accuracy: 25968/84843 (31%) Train Epoch: 3 [40000/84843 (47%)] Loss: 0.612714 Accuracy: 32509/84843 (38%) Train Epoch: 3 [48000/84843 (56%)] Loss: 0.587187 Accuracy: 39081/84843 (46%) Train Epoch: 3 [56000/84843 (66%)] Loss: 0.682911 Accuracy: 45645/84843 (54%) Train Epoch: 3 [64000/84843 (75%)] Loss: 0.670462 Accuracy: 52164/84843 (61%) Train Epoch: 3 [72000/84843 (85%)] Loss: 0.563404 Accuracy: 58711/84843 (69%) Train Epoch: 3 [80000/84843 (94%)] Loss: 0.586875 Accuracy: 65288/84843 (77%) Validation Epoch: 3 Loss: 0.544732 Accuracy: 7123/9981 (71%) Train Epoch: 4 [0/84843 (0%)] Loss: 0.444439 Accuracy: 351/84843 (0%) Train Epoch: 4 [8000/84843 (9%)] Loss: 0.629947 Accuracy: 6878/84843 (8%) Train Epoch: 4 [16000/84843 (19%)] Loss: 0.602237 Accuracy: 13509/84843 (16%) Train Epoch: 4 [24000/84843 (28%)] Loss: 0.570960 Accuracy: 20124/84843 (24%) Train Epoch: 4 [32000/84843 (38%)] Loss: 0.573054 Accuracy: 26843/84843 (32%) Train Epoch: 4 [40000/84843 (47%)] Loss: 0.416008 Accuracy: 33647/84843 (40%) Train Epoch: 4 [48000/84843 (56%)] Loss: 0.641524 Accuracy: 40333/84843 (48%) Train Epoch: 4 [56000/84843 (66%)] Loss: 0.526742 Accuracy: 46990/84843 (55%) Train Epoch: 4 [64000/84843 (75%)] Loss: 0.524154 Accuracy: 53788/84843 (63%) Train Epoch: 4 [72000/84843 (85%)] Loss: 0.615018 Accuracy: 60445/84843 (71%) Train Epoch: 4 [80000/84843 (94%)] Loss: 0.515469 Accuracy: 67151/84843 (79%) Validation Epoch: 4 Loss: 0.609637 Accuracy: 6129/9981 (61%) Train Epoch: 5 [0/84843 (0%)] Loss: 0.470834 Accuracy: 347/84843 (0%) Train Epoch: 5 [8000/84843 (9%)] Loss: 0.519897 Accuracy: 7078/84843 (8%) Train Epoch: 5 [16000/84843 (19%)] Loss: 0.456270 Accuracy: 13897/84843 (16%) Train Epoch: 5 [24000/84843 (28%)] Loss: 0.480340 Accuracy: 20760/84843 (24%) Train Epoch: 5 [32000/84843 (38%)] Loss: 0.537243 Accuracy: 27601/84843 (33%) Train Epoch: 5 [40000/84843 (47%)] Loss: 0.473347 Accuracy: 34437/84843 (41%) Train Epoch: 5 [48000/84843 (56%)] Loss: 0.405917 Accuracy: 41297/84843 (49%) Train Epoch: 5 [56000/84843 (66%)] Loss: 0.489415 Accuracy: 48174/84843 (57%) Train Epoch: 5 [64000/84843 (75%)] Loss: 0.506211 Accuracy: 55032/84843 (65%) Train Epoch: 5 [72000/84843 (85%)] Loss: 0.555165 Accuracy: 61850/84843 (73%) Train Epoch: 5 [80000/84843 (94%)] Loss: 0.538229 Accuracy: 68699/84843 (81%) Validation Epoch: 5 Loss: 1.104323 Accuracy: 7234/9981 (72%) Train Epoch: 6 [0/84843 (0%)] Loss: 0.453177 Accuracy: 354/84843 (0%) Train Epoch: 6 [8000/84843 (9%)] Loss: 0.433283 Accuracy: 7208/84843 (8%) Train Epoch: 6 [16000/84843 (19%)] Loss: 0.538747 Accuracy: 14045/84843 (17%) Train Epoch: 6 [24000/84843 (28%)] Loss: 0.464524 Accuracy: 21021/84843 (25%) Train Epoch: 6 [32000/84843 (38%)] Loss: 0.496080 Accuracy: 27960/84843 (33%) Train Epoch: 6 [40000/84843 (47%)] Loss: 0.465593 Accuracy: 34866/84843 (41%) Train Epoch: 6 [48000/84843 (56%)] Loss: 0.455706 Accuracy: 41849/84843 (49%) Train Epoch: 6 [56000/84843 (66%)] Loss: 0.494361 Accuracy: 48764/84843 (57%) Train Epoch: 6 [64000/84843 (75%)] Loss: 0.558386 Accuracy: 55636/84843 (66%) Train Epoch: 6 [72000/84843 (85%)] Loss: 0.468302 Accuracy: 62486/84843 (74%) Train Epoch: 6 [80000/84843 (94%)] Loss: 0.583563 Accuracy: 69291/84843 (82%) Validation Epoch: 6 Loss: 0.448210 Accuracy: 8161/9981 (82%) Train Epoch: 7 [0/84843 (0%)] Loss: 0.541128 Accuracy: 337/84843 (0%) Train Epoch: 7 [8000/84843 (9%)] Loss: 0.411102 Accuracy: 7196/84843 (8%) Train Epoch: 7 [16000/84843 (19%)] Loss: 0.382305 Accuracy: 14163/84843 (17%) Train Epoch: 7 [24000/84843 (28%)] Loss: 0.399029 Accuracy: 21113/84843 (25%) Train Epoch: 7 [32000/84843 (38%)] Loss: 0.565331 Accuracy: 28057/84843 (33%) Train Epoch: 7 [40000/84843 (47%)] Loss: 0.528239 Accuracy: 35028/84843 (41%) Train Epoch: 7 [48000/84843 (56%)] Loss: 0.410433 Accuracy: 42020/84843 (50%) Train Epoch: 7 [56000/84843 (66%)] Loss: 0.503528 Accuracy: 48990/84843 (58%) Train Epoch: 7 [64000/84843 (75%)] Loss: 0.554294 Accuracy: 55928/84843 (66%) Train Epoch: 7 [72000/84843 (85%)] Loss: 0.420849 Accuracy: 62889/84843 (74%) Train Epoch: 7 [80000/84843 (94%)] Loss: 0.555479 Accuracy: 69842/84843 (82%) Validation Epoch: 7 Loss: 0.476227 Accuracy: 7965/9981 (80%) Train Epoch: 8 [0/84843 (0%)] Loss: 0.474673 Accuracy: 347/84843 (0%) Train Epoch: 8 [8000/84843 (9%)] Loss: 0.476040 Accuracy: 7297/84843 (9%) Train Epoch: 8 [16000/84843 (19%)] Loss: 0.364525 Accuracy: 14302/84843 (17%) Train Epoch: 8 [24000/84843 (28%)] Loss: 0.509280 Accuracy: 21321/84843 (25%) Train Epoch: 8 [32000/84843 (38%)] Loss: 0.416610 Accuracy: 28341/84843 (33%) Train Epoch: 8 [40000/84843 (47%)] Loss: 0.409597 Accuracy: 35312/84843 (42%) Train Epoch: 8 [48000/84843 (56%)] Loss: 0.421205 Accuracy: 42330/84843 (50%) Train Epoch: 8 [56000/84843 (66%)] Loss: 0.433508 Accuracy: 49295/84843 (58%) Train Epoch: 8 [64000/84843 (75%)] Loss: 0.485407 Accuracy: 56278/84843 (66%) Train Epoch: 8 [72000/84843 (85%)] Loss: 0.494349 Accuracy: 63265/84843 (75%) Train Epoch: 8 [80000/84843 (94%)] Loss: 0.376893 Accuracy: 70228/84843 (83%) Validation Epoch: 8 Loss: 1.870698 Accuracy: 7844/9981 (79%) Train Epoch: 9 [0/84843 (0%)] Loss: 0.506868 Accuracy: 350/84843 (0%) Train Epoch: 9 [8000/84843 (9%)] Loss: 0.497237 Accuracy: 7267/84843 (9%) Train Epoch: 9 [16000/84843 (19%)] Loss: 0.345686 Accuracy: 14320/84843 (17%) Train Epoch: 9 [24000/84843 (28%)] Loss: 0.555300 Accuracy: 21382/84843 (25%) Train Epoch: 9 [32000/84843 (38%)] Loss: 0.445774 Accuracy: 28393/84843 (33%) Train Epoch: 9 [40000/84843 (47%)] Loss: 0.446099 Accuracy: 35407/84843 (42%) Train Epoch: 9 [48000/84843 (56%)] Loss: 0.383177 Accuracy: 42428/84843 (50%) Train Epoch: 9 [56000/84843 (66%)] Loss: 0.361311 Accuracy: 49421/84843 (58%) Train Epoch: 9 [64000/84843 (75%)] Loss: 0.379214 Accuracy: 56429/84843 (67%) Train Epoch: 9 [72000/84843 (85%)] Loss: 0.373450 Accuracy: 63437/84843 (75%) Train Epoch: 9 [80000/84843 (94%)] Loss: 0.476715 Accuracy: 70446/84843 (83%) Validation Epoch: 9 Loss: 0.257025 Accuracy: 8699/9981 (87%) Train Epoch: 10 [0/84843 (0%)] Loss: 0.404351 Accuracy: 354/84843 (0%) Train Epoch: 10 [8000/84843 (9%)] Loss: 0.372246 Accuracy: 7385/84843 (9%) Train Epoch: 10 [16000/84843 (19%)] Loss: 0.384944 Accuracy: 14376/84843 (17%) Train Epoch: 10 [24000/84843 (28%)] Loss: 0.447159 Accuracy: 21425/84843 (25%) Train Epoch: 10 [32000/84843 (38%)] Loss: 0.402930 Accuracy: 28505/84843 (34%) Train Epoch: 10 [40000/84843 (47%)] Loss: 0.396107 Accuracy: 35564/84843 (42%) Train Epoch: 10 [48000/84843 (56%)] Loss: 0.369256 Accuracy: 42596/84843 (50%) Train Epoch: 10 [56000/84843 (66%)] Loss: 0.488550 Accuracy: 49574/84843 (58%) Train Epoch: 10 [64000/84843 (75%)] Loss: 0.570274 Accuracy: 56615/84843 (67%) Train Epoch: 10 [72000/84843 (85%)] Loss: 0.463297 Accuracy: 63645/84843 (75%) Train Epoch: 10 [80000/84843 (94%)] Loss: 0.427048 Accuracy: 70711/84843 (83%) Validation Epoch: 10 Loss: 1.564104 Accuracy: 7360/9981 (74%) Train Epoch: 11 [0/84843 (0%)] Loss: 0.412347 Accuracy: 348/84843 (0%) Train Epoch: 11 [8000/84843 (9%)] Loss: 0.450841 Accuracy: 7317/84843 (9%) Train Epoch: 11 [16000/84843 (19%)] Loss: 0.589727 Accuracy: 14361/84843 (17%) Train Epoch: 11 [24000/84843 (28%)] Loss: 0.419228 Accuracy: 21459/84843 (25%) Train Epoch: 11 [32000/84843 (38%)] Loss: 0.439322 Accuracy: 28556/84843 (34%) Train Epoch: 11 [40000/84843 (47%)] Loss: 0.463837 Accuracy: 35610/84843 (42%) Train Epoch: 11 [48000/84843 (56%)] Loss: 0.504982 Accuracy: 42642/84843 (50%) Train Epoch: 11 [56000/84843 (66%)] Loss: 0.458225 Accuracy: 49732/84843 (59%) Train Epoch: 11 [64000/84843 (75%)] Loss: 0.322770 Accuracy: 56797/84843 (67%) Train Epoch: 11 [72000/84843 (85%)] Loss: 0.355502 Accuracy: 63895/84843 (75%) Train Epoch: 11 [80000/84843 (94%)] Loss: 0.387963 Accuracy: 70925/84843 (84%) Validation Epoch: 11 Loss: 0.889939 Accuracy: 7102/9981 (71%) Train Epoch: 12 [0/84843 (0%)] Loss: 0.347913 Accuracy: 356/84843 (0%) Train Epoch: 12 [8000/84843 (9%)] Loss: 0.337602 Accuracy: 7501/84843 (9%) Train Epoch: 12 [16000/84843 (19%)] Loss: 0.389245 Accuracy: 14609/84843 (17%) Train Epoch: 12 [24000/84843 (28%)] Loss: 0.388557 Accuracy: 21666/84843 (26%) Train Epoch: 12 [32000/84843 (38%)] Loss: 0.377709 Accuracy: 28730/84843 (34%) Train Epoch: 12 [40000/84843 (47%)] Loss: 0.362607 Accuracy: 35781/84843 (42%) Train Epoch: 12 [48000/84843 (56%)] Loss: 0.394197 Accuracy: 42825/84843 (50%) Train Epoch: 12 [56000/84843 (66%)] Loss: 0.508421 Accuracy: 49918/84843 (59%) Train Epoch: 12 [64000/84843 (75%)] Loss: 0.422594 Accuracy: 56963/84843 (67%) Train Epoch: 12 [72000/84843 (85%)] Loss: 0.402469 Accuracy: 63960/84843 (75%) Train Epoch: 12 [80000/84843 (94%)] Loss: 0.335717 Accuracy: 70951/84843 (84%) Validation Epoch: 12 Loss: 0.294111 Accuracy: 8100/9981 (81%) Train Epoch: 13 [0/84843 (0%)] Loss: 0.332361 Accuracy: 359/84843 (0%) Train Epoch: 13 [8000/84843 (9%)] Loss: 0.471854 Accuracy: 7460/84843 (9%) Train Epoch: 13 [16000/84843 (19%)] Loss: 0.349878 Accuracy: 14626/84843 (17%) Train Epoch: 13 [24000/84843 (28%)] Loss: 0.435345 Accuracy: 21819/84843 (26%) Train Epoch: 13 [32000/84843 (38%)] Loss: 0.385956 Accuracy: 28932/84843 (34%) Train Epoch: 13 [40000/84843 (47%)] Loss: 0.532963 Accuracy: 35871/84843 (42%) Train Epoch: 13 [48000/84843 (56%)] Loss: 0.400751 Accuracy: 42927/84843 (51%) Train Epoch: 13 [56000/84843 (66%)] Loss: 0.505520 Accuracy: 49974/84843 (59%) Train Epoch: 13 [64000/84843 (75%)] Loss: 0.300064 Accuracy: 57066/84843 (67%) Train Epoch: 13 [72000/84843 (85%)] Loss: 0.465522 Accuracy: 64117/84843 (76%) Train Epoch: 13 [80000/84843 (94%)] Loss: 0.423109 Accuracy: 71185/84843 (84%) Validation Epoch: 13 Loss: 0.339391 Accuracy: 8104/9981 (81%) Train Epoch: 14 [0/84843 (0%)] Loss: 0.360610 Accuracy: 361/84843 (0%) Train Epoch: 14 [8000/84843 (9%)] Loss: 0.434245 Accuracy: 7374/84843 (9%) Train Epoch: 14 [16000/84843 (19%)] Loss: 0.378827 Accuracy: 14495/84843 (17%) Train Epoch: 14 [24000/84843 (28%)] Loss: 0.330523 Accuracy: 21576/84843 (25%) Train Epoch: 14 [32000/84843 (38%)] Loss: 0.334730 Accuracy: 28675/84843 (34%) Train Epoch: 14 [40000/84843 (47%)] Loss: 0.394331 Accuracy: 35815/84843 (42%) Train Epoch: 14 [48000/84843 (56%)] Loss: 0.407279 Accuracy: 42899/84843 (51%) Train Epoch: 14 [56000/84843 (66%)] Loss: 0.403670 Accuracy: 49941/84843 (59%) Train Epoch: 14 [64000/84843 (75%)] Loss: 0.310442 Accuracy: 57046/84843 (67%) Train Epoch: 14 [72000/84843 (85%)] Loss: 0.339602 Accuracy: 64185/84843 (76%) Train Epoch: 14 [80000/84843 (94%)] Loss: 0.348903 Accuracy: 71314/84843 (84%) Validation Epoch: 14 Loss: 0.794774 Accuracy: 8614/9981 (86%) Train Epoch: 15 [0/84843 (0%)] Loss: 0.364461 Accuracy: 354/84843 (0%) Train Epoch: 15 [8000/84843 (9%)] Loss: 0.364585 Accuracy: 7417/84843 (9%) Train Epoch: 15 [16000/84843 (19%)] Loss: 0.432593 Accuracy: 14586/84843 (17%) Train Epoch: 15 [24000/84843 (28%)] Loss: 0.316601 Accuracy: 21735/84843 (26%) Train Epoch: 15 [32000/84843 (38%)] Loss: 0.389377 Accuracy: 28910/84843 (34%) Train Epoch: 15 [40000/84843 (47%)] Loss: 0.452709 Accuracy: 35992/84843 (42%) Train Epoch: 15 [48000/84843 (56%)] Loss: 0.434766 Accuracy: 42991/84843 (51%) Train Epoch: 15 [56000/84843 (66%)] Loss: 0.389990 Accuracy: 50060/84843 (59%) Train Epoch: 15 [64000/84843 (75%)] Loss: 0.312573 Accuracy: 57185/84843 (67%) Train Epoch: 15 [72000/84843 (85%)] Loss: 0.380443 Accuracy: 64255/84843 (76%) Train Epoch: 15 [80000/84843 (94%)] Loss: 0.300701 Accuracy: 71387/84843 (84%) Validation Epoch: 15 Loss: 0.329043 Accuracy: 8436/9981 (85%) Train Epoch: 16 [0/84843 (0%)] Loss: 0.411329 Accuracy: 357/84843 (0%) Train Epoch: 16 [8000/84843 (9%)] Loss: 0.327841 Accuracy: 7505/84843 (9%) Train Epoch: 16 [16000/84843 (19%)] Loss: 0.387103 Accuracy: 14643/84843 (17%) Train Epoch: 16 [24000/84843 (28%)] Loss: 0.468313 Accuracy: 21794/84843 (26%) Train Epoch: 16 [32000/84843 (38%)] Loss: 0.380608 Accuracy: 28905/84843 (34%) Train Epoch: 16 [40000/84843 (47%)] Loss: 0.404311 Accuracy: 35970/84843 (42%) Train Epoch: 16 [48000/84843 (56%)] Loss: 0.401409 Accuracy: 43077/84843 (51%) Train Epoch: 16 [56000/84843 (66%)] Loss: 0.356435 Accuracy: 50175/84843 (59%) Train Epoch: 16 [64000/84843 (75%)] Loss: 0.384459 Accuracy: 57329/84843 (68%) Train Epoch: 16 [72000/84843 (85%)] Loss: 0.411760 Accuracy: 64504/84843 (76%) Train Epoch: 16 [80000/84843 (94%)] Loss: 0.350388 Accuracy: 71675/84843 (84%) Validation Epoch: 16 Loss: 0.233272 Accuracy: 8477/9981 (85%) Train Epoch: 17 [0/84843 (0%)] Loss: 0.362397 Accuracy: 357/84843 (0%) Train Epoch: 17 [8000/84843 (9%)] Loss: 0.442485 Accuracy: 7433/84843 (9%) Train Epoch: 17 [16000/84843 (19%)] Loss: 0.440732 Accuracy: 14564/84843 (17%) Train Epoch: 17 [24000/84843 (28%)] Loss: 0.360361 Accuracy: 21717/84843 (26%) Train Epoch: 17 [32000/84843 (38%)] Loss: 0.370121 Accuracy: 28844/84843 (34%) Train Epoch: 17 [40000/84843 (47%)] Loss: 0.351120 Accuracy: 36002/84843 (42%) Train Epoch: 17 [48000/84843 (56%)] Loss: 0.449296 Accuracy: 43120/84843 (51%) Train Epoch: 17 [56000/84843 (66%)] Loss: 0.366723 Accuracy: 50232/84843 (59%) Train Epoch: 17 [64000/84843 (75%)] Loss: 0.361119 Accuracy: 57307/84843 (68%) Train Epoch: 17 [72000/84843 (85%)] Loss: 0.431330 Accuracy: 64360/84843 (76%) Train Epoch: 17 [80000/84843 (94%)] Loss: 0.408258 Accuracy: 71514/84843 (84%) Validation Epoch: 17 Loss: 1.110281 Accuracy: 8301/9981 (83%) Train Epoch: 18 [0/84843 (0%)] Loss: 0.375765 Accuracy: 352/84843 (0%) Train Epoch: 18 [8000/84843 (9%)] Loss: 0.439161 Accuracy: 7452/84843 (9%) Train Epoch: 18 [16000/84843 (19%)] Loss: 0.395859 Accuracy: 14560/84843 (17%) Train Epoch: 18 [24000/84843 (28%)] Loss: 0.350249 Accuracy: 21670/84843 (26%) Train Epoch: 18 [32000/84843 (38%)] Loss: 0.447112 Accuracy: 28825/84843 (34%) Train Epoch: 18 [40000/84843 (47%)] Loss: 0.318871 Accuracy: 35936/84843 (42%) Train Epoch: 18 [48000/84843 (56%)] Loss: 0.346760 Accuracy: 43107/84843 (51%) Train Epoch: 18 [56000/84843 (66%)] Loss: 0.377471 Accuracy: 50192/84843 (59%) Train Epoch: 18 [64000/84843 (75%)] Loss: 0.403838 Accuracy: 57322/84843 (68%) Train Epoch: 18 [72000/84843 (85%)] Loss: 0.441702 Accuracy: 64422/84843 (76%) Train Epoch: 18 [80000/84843 (94%)] Loss: 0.418436 Accuracy: 71534/84843 (84%) Validation Epoch: 18 Loss: 0.179589 Accuracy: 8372/9981 (84%) Train Epoch: 19 [0/84843 (0%)] Loss: 0.412119 Accuracy: 350/84843 (0%) Train Epoch: 19 [8000/84843 (9%)] Loss: 0.432262 Accuracy: 7234/84843 (9%) Train Epoch: 19 [16000/84843 (19%)] Loss: 0.328371 Accuracy: 14268/84843 (17%) Train Epoch: 19 [24000/84843 (28%)] Loss: 0.317724 Accuracy: 21417/84843 (25%) Train Epoch: 19 [32000/84843 (38%)] Loss: 0.335858 Accuracy: 28546/84843 (34%) Train Epoch: 19 [40000/84843 (47%)] Loss: 0.322377 Accuracy: 35672/84843 (42%) Train Epoch: 19 [48000/84843 (56%)] Loss: 0.476986 Accuracy: 42742/84843 (50%) Train Epoch: 19 [56000/84843 (66%)] Loss: 0.338926 Accuracy: 49903/84843 (59%) Train Epoch: 19 [64000/84843 (75%)] Loss: 0.388219 Accuracy: 56958/84843 (67%) Train Epoch: 19 [72000/84843 (85%)] Loss: 0.339961 Accuracy: 64083/84843 (76%) Train Epoch: 19 [80000/84843 (94%)] Loss: 0.354864 Accuracy: 71240/84843 (84%) Validation Epoch: 19 Loss: 0.161501 Accuracy: 8512/9981 (85%) Train Epoch: 20 [0/84843 (0%)] Loss: 0.435381 Accuracy: 348/84843 (0%) Train Epoch: 20 [8000/84843 (9%)] Loss: 0.439417 Accuracy: 7459/84843 (9%) Train Epoch: 20 [16000/84843 (19%)] Loss: 0.398828 Accuracy: 14633/84843 (17%) Train Epoch: 20 [24000/84843 (28%)] Loss: 0.335384 Accuracy: 21796/84843 (26%) Train Epoch: 20 [32000/84843 (38%)] Loss: 0.342249 Accuracy: 28955/84843 (34%) Train Epoch: 20 [40000/84843 (47%)] Loss: 0.354554 Accuracy: 36120/84843 (43%) Train Epoch: 20 [48000/84843 (56%)] Loss: 0.434681 Accuracy: 43207/84843 (51%) Train Epoch: 20 [56000/84843 (66%)] Loss: 0.336001 Accuracy: 50401/84843 (59%) Train Epoch: 20 [64000/84843 (75%)] Loss: 0.339834 Accuracy: 57541/84843 (68%) Train Epoch: 20 [72000/84843 (85%)] Loss: 0.427533 Accuracy: 64697/84843 (76%) Train Epoch: 20 [80000/84843 (94%)] Loss: 0.485117 Accuracy: 71777/84843 (85%) Validation Epoch: 20 Loss: 0.648743 Accuracy: 8567/9981 (86%) Train Epoch: 21 [0/84843 (0%)] Loss: 0.395031 Accuracy: 349/84843 (0%) Train Epoch: 21 [8000/84843 (9%)] Loss: 0.277579 Accuracy: 7575/84843 (9%) Train Epoch: 21 [16000/84843 (19%)] Loss: 0.305243 Accuracy: 14942/84843 (18%) Train Epoch: 21 [24000/84843 (28%)] Loss: 0.230128 Accuracy: 22341/84843 (26%) Train Epoch: 21 [32000/84843 (38%)] Loss: 0.283402 Accuracy: 29750/84843 (35%) Train Epoch: 21 [40000/84843 (47%)] Loss: 0.227811 Accuracy: 37163/84843 (44%) Train Epoch: 21 [48000/84843 (56%)] Loss: 0.228918 Accuracy: 44590/84843 (53%) Train Epoch: 21 [56000/84843 (66%)] Loss: 0.269471 Accuracy: 52046/84843 (61%) Train Epoch: 21 [64000/84843 (75%)] Loss: 0.256335 Accuracy: 59510/84843 (70%) Train Epoch: 21 [72000/84843 (85%)] Loss: 0.273809 Accuracy: 66954/84843 (79%) Train Epoch: 21 [80000/84843 (94%)] Loss: 0.313194 Accuracy: 74381/84843 (88%) Validation Epoch: 21 Loss: 0.254740 Accuracy: 9185/9981 (92%) Train Epoch: 22 [0/84843 (0%)] Loss: 0.197295 Accuracy: 382/84843 (0%) Train Epoch: 22 [8000/84843 (9%)] Loss: 0.240721 Accuracy: 7864/84843 (9%) Train Epoch: 22 [16000/84843 (19%)] Loss: 0.302697 Accuracy: 15383/84843 (18%) Train Epoch: 22 [24000/84843 (28%)] Loss: 0.283399 Accuracy: 22814/84843 (27%) Train Epoch: 22 [32000/84843 (38%)] Loss: 0.217230 Accuracy: 30308/84843 (36%) Train Epoch: 22 [40000/84843 (47%)] Loss: 0.146620 Accuracy: 37795/84843 (45%) Train Epoch: 22 [48000/84843 (56%)] Loss: 0.232884 Accuracy: 45273/84843 (53%) Train Epoch: 22 [56000/84843 (66%)] Loss: 0.226973 Accuracy: 52762/84843 (62%) Train Epoch: 22 [64000/84843 (75%)] Loss: 0.192257 Accuracy: 60274/84843 (71%) Train Epoch: 22 [72000/84843 (85%)] Loss: 0.174776 Accuracy: 67798/84843 (80%) Train Epoch: 22 [80000/84843 (94%)] Loss: 0.234974 Accuracy: 75305/84843 (89%) Validation Epoch: 22 Loss: 0.236908 Accuracy: 9186/9981 (92%) Train Epoch: 23 [0/84843 (0%)] Loss: 0.201676 Accuracy: 377/84843 (0%) Train Epoch: 23 [8000/84843 (9%)] Loss: 0.213076 Accuracy: 7946/84843 (9%) Train Epoch: 23 [16000/84843 (19%)] Loss: 0.179018 Accuracy: 15505/84843 (18%) Train Epoch: 23 [24000/84843 (28%)] Loss: 0.257552 Accuracy: 23021/84843 (27%) Train Epoch: 23 [32000/84843 (38%)] Loss: 0.180480 Accuracy: 30530/84843 (36%) Train Epoch: 23 [40000/84843 (47%)] Loss: 0.249078 Accuracy: 38031/84843 (45%) Train Epoch: 23 [48000/84843 (56%)] Loss: 0.225377 Accuracy: 45574/84843 (54%) Train Epoch: 23 [56000/84843 (66%)] Loss: 0.256208 Accuracy: 53060/84843 (63%) Train Epoch: 23 [64000/84843 (75%)] Loss: 0.222297 Accuracy: 60581/84843 (71%) Train Epoch: 23 [72000/84843 (85%)] Loss: 0.174348 Accuracy: 68096/84843 (80%) Train Epoch: 23 [80000/84843 (94%)] Loss: 0.245362 Accuracy: 75605/84843 (89%) Validation Epoch: 23 Loss: 0.203503 Accuracy: 9196/9981 (92%) Train Epoch: 24 [0/84843 (0%)] Loss: 0.143904 Accuracy: 383/84843 (0%) Train Epoch: 24 [8000/84843 (9%)] Loss: 0.224440 Accuracy: 7942/84843 (9%) Train Epoch: 24 [16000/84843 (19%)] Loss: 0.180008 Accuracy: 15516/84843 (18%) Train Epoch: 24 [24000/84843 (28%)] Loss: 0.224399 Accuracy: 23051/84843 (27%) Train Epoch: 24 [32000/84843 (38%)] Loss: 0.153062 Accuracy: 30609/84843 (36%) Train Epoch: 24 [40000/84843 (47%)] Loss: 0.242340 Accuracy: 38185/84843 (45%) Train Epoch: 24 [48000/84843 (56%)] Loss: 0.197323 Accuracy: 45735/84843 (54%) Train Epoch: 24 [56000/84843 (66%)] Loss: 0.217974 Accuracy: 53235/84843 (63%) Train Epoch: 24 [64000/84843 (75%)] Loss: 0.203447 Accuracy: 60788/84843 (72%) Train Epoch: 24 [72000/84843 (85%)] Loss: 0.203183 Accuracy: 68302/84843 (81%) Train Epoch: 24 [80000/84843 (94%)] Loss: 0.224392 Accuracy: 75825/84843 (89%) Validation Epoch: 24 Loss: 0.296059 Accuracy: 9221/9981 (92%) Train Epoch: 25 [0/84843 (0%)] Loss: 0.189224 Accuracy: 380/84843 (0%) Train Epoch: 25 [8000/84843 (9%)] Loss: 0.197070 Accuracy: 7933/84843 (9%) Train Epoch: 25 [16000/84843 (19%)] Loss: 0.185826 Accuracy: 15490/84843 (18%) Train Epoch: 25 [24000/84843 (28%)] Loss: 0.238932 Accuracy: 23054/84843 (27%) Train Epoch: 25 [32000/84843 (38%)] Loss: 0.145945 Accuracy: 30652/84843 (36%) Train Epoch: 25 [40000/84843 (47%)] Loss: 0.208481 Accuracy: 38219/84843 (45%) Train Epoch: 25 [48000/84843 (56%)] Loss: 0.222141 Accuracy: 45791/84843 (54%) Train Epoch: 25 [56000/84843 (66%)] Loss: 0.193078 Accuracy: 53333/84843 (63%) Train Epoch: 25 [64000/84843 (75%)] Loss: 0.268021 Accuracy: 60871/84843 (72%) Train Epoch: 25 [72000/84843 (85%)] Loss: 0.181645 Accuracy: 68412/84843 (81%) Train Epoch: 25 [80000/84843 (94%)] Loss: 0.161807 Accuracy: 75999/84843 (90%) Validation Epoch: 25 Loss: 0.160351 Accuracy: 9202/9981 (92%) Train Epoch: 26 [0/84843 (0%)] Loss: 0.221939 Accuracy: 376/84843 (0%) Train Epoch: 26 [8000/84843 (9%)] Loss: 0.206715 Accuracy: 7953/84843 (9%) Train Epoch: 26 [16000/84843 (19%)] Loss: 0.255378 Accuracy: 15523/84843 (18%) Train Epoch: 26 [24000/84843 (28%)] Loss: 0.247545 Accuracy: 23057/84843 (27%) Train Epoch: 26 [32000/84843 (38%)] Loss: 0.184941 Accuracy: 30631/84843 (36%) Train Epoch: 26 [40000/84843 (47%)] Loss: 0.180947 Accuracy: 38184/84843 (45%) Train Epoch: 26 [48000/84843 (56%)] Loss: 0.175474 Accuracy: 45794/84843 (54%) Train Epoch: 26 [56000/84843 (66%)] Loss: 0.238699 Accuracy: 53345/84843 (63%) Train Epoch: 26 [64000/84843 (75%)] Loss: 0.190480 Accuracy: 60924/84843 (72%) Train Epoch: 26 [72000/84843 (85%)] Loss: 0.179081 Accuracy: 68468/84843 (81%) Train Epoch: 26 [80000/84843 (94%)] Loss: 0.242031 Accuracy: 76051/84843 (90%) Validation Epoch: 26 Loss: 0.196848 Accuracy: 9207/9981 (92%) Train Epoch: 27 [0/84843 (0%)] Loss: 0.176752 Accuracy: 385/84843 (0%) Train Epoch: 27 [8000/84843 (9%)] Loss: 0.134787 Accuracy: 8003/84843 (9%) Train Epoch: 27 [16000/84843 (19%)] Loss: 0.166365 Accuracy: 15588/84843 (18%) Train Epoch: 27 [24000/84843 (28%)] Loss: 0.217781 Accuracy: 23190/84843 (27%) Train Epoch: 27 [32000/84843 (38%)] Loss: 0.222927 Accuracy: 30762/84843 (36%) Train Epoch: 27 [40000/84843 (47%)] Loss: 0.202337 Accuracy: 38356/84843 (45%) Train Epoch: 27 [48000/84843 (56%)] Loss: 0.175472 Accuracy: 45949/84843 (54%) Train Epoch: 27 [56000/84843 (66%)] Loss: 0.159321 Accuracy: 53529/84843 (63%) Train Epoch: 27 [64000/84843 (75%)] Loss: 0.181302 Accuracy: 61123/84843 (72%) Train Epoch: 27 [72000/84843 (85%)] Loss: 0.183700 Accuracy: 68672/84843 (81%) Train Epoch: 27 [80000/84843 (94%)] Loss: 0.213071 Accuracy: 76273/84843 (90%) Validation Epoch: 27 Loss: 0.166645 Accuracy: 9188/9981 (92%) Train Epoch: 28 [0/84843 (0%)] Loss: 0.212189 Accuracy: 373/84843 (0%) Train Epoch: 28 [8000/84843 (9%)] Loss: 0.172711 Accuracy: 7982/84843 (9%) Train Epoch: 28 [16000/84843 (19%)] Loss: 0.174262 Accuracy: 15601/84843 (18%) Train Epoch: 28 [24000/84843 (28%)] Loss: 0.172311 Accuracy: 23192/84843 (27%) Train Epoch: 28 [32000/84843 (38%)] Loss: 0.238622 Accuracy: 30816/84843 (36%) Train Epoch: 28 [40000/84843 (47%)] Loss: 0.149252 Accuracy: 38399/84843 (45%) Train Epoch: 28 [48000/84843 (56%)] Loss: 0.233139 Accuracy: 45986/84843 (54%) Train Epoch: 28 [56000/84843 (66%)] Loss: 0.166495 Accuracy: 53582/84843 (63%) Train Epoch: 28 [64000/84843 (75%)] Loss: 0.189434 Accuracy: 61171/84843 (72%) Train Epoch: 28 [72000/84843 (85%)] Loss: 0.142340 Accuracy: 68714/84843 (81%) Train Epoch: 28 [80000/84843 (94%)] Loss: 0.158354 Accuracy: 76283/84843 (90%) Validation Epoch: 28 Loss: 0.157085 Accuracy: 9216/9981 (92%) Train Epoch: 29 [0/84843 (0%)] Loss: 0.122715 Accuracy: 389/84843 (0%) Train Epoch: 29 [8000/84843 (9%)] Loss: 0.176554 Accuracy: 8031/84843 (9%) Train Epoch: 29 [16000/84843 (19%)] Loss: 0.153073 Accuracy: 15672/84843 (18%) Train Epoch: 29 [24000/84843 (28%)] Loss: 0.171134 Accuracy: 23284/84843 (27%) Train Epoch: 29 [32000/84843 (38%)] Loss: 0.190553 Accuracy: 30906/84843 (36%) Train Epoch: 29 [40000/84843 (47%)] Loss: 0.153314 Accuracy: 38518/84843 (45%) Train Epoch: 29 [48000/84843 (56%)] Loss: 0.186997 Accuracy: 46127/84843 (54%) Train Epoch: 29 [56000/84843 (66%)] Loss: 0.148230 Accuracy: 53743/84843 (63%) Train Epoch: 29 [64000/84843 (75%)] Loss: 0.205280 Accuracy: 61300/84843 (72%) Train Epoch: 29 [72000/84843 (85%)] Loss: 0.205373 Accuracy: 68899/84843 (81%) Train Epoch: 29 [80000/84843 (94%)] Loss: 0.205607 Accuracy: 76503/84843 (90%) Validation Epoch: 29 Loss: 0.168752 Accuracy: 9136/9981 (92%) Train Epoch: 30 [0/84843 (0%)] Loss: 0.174106 Accuracy: 381/84843 (0%) Train Epoch: 30 [8000/84843 (9%)] Loss: 0.172931 Accuracy: 8008/84843 (9%) Train Epoch: 30 [16000/84843 (19%)] Loss: 0.175244 Accuracy: 15667/84843 (18%) Train Epoch: 30 [24000/84843 (28%)] Loss: 0.180944 Accuracy: 23288/84843 (27%) Train Epoch: 30 [32000/84843 (38%)] Loss: 0.148970 Accuracy: 30939/84843 (36%) Train Epoch: 30 [40000/84843 (47%)] Loss: 0.169565 Accuracy: 38561/84843 (45%) Train Epoch: 30 [48000/84843 (56%)] Loss: 0.206594 Accuracy: 46160/84843 (54%) Train Epoch: 30 [56000/84843 (66%)] Loss: 0.140612 Accuracy: 53783/84843 (63%) Train Epoch: 30 [64000/84843 (75%)] Loss: 0.199289 Accuracy: 61390/84843 (72%) Train Epoch: 30 [72000/84843 (85%)] Loss: 0.185448 Accuracy: 69004/84843 (81%) Train Epoch: 30 [80000/84843 (94%)] Loss: 0.200791 Accuracy: 76619/84843 (90%) Validation Epoch: 30 Loss: 0.141483 Accuracy: 9174/9981 (92%)
In [57]:
# Plot training loss
plt.plot(losses_train, 'b', label='Train loss')
plt.legend(loc="upper left")
plt.title("M5 model training losses over all iterations")
plt.show()

# Plot validation loss
plt.plot(losses_validation, 'r', label='Valid loss')
plt.legend(loc="upper left")
plt.title("M5 model validation losses over validation epochs")
plt.show()

# Plot accuracy
plt.plot(accuracy_train, 'b', label='Train acc')
plt.plot(accuracy_validation,'r', label ='Valid acc')
plt.legend(loc="upper left")
plt.title("M5 model training & validation accuracy over epochs")
plt.show()
Notebook Image
Notebook Image
Notebook Image
In [58]:
M5_test_acc_result = test(model_M5)
print(M5_test_acc_result)
Test set accuracy: 10048/11005 (91%) 91.30395274875056
In [59]:
jovian.log_metrics({ 'M5_test_acc_result': M5_test_acc_result })
[jovian] Metrics logged.
In [60]:
jovian.commit(project=project_name)
[jovian] Detected Colab notebook... [jovian] Uploading colab notebook to Jovian... [jovian] Capturing environment.. [jovian] Attaching records (metrics, hyperparameters, dataset etc.) [jovian] Committed successfully! https://jovian.ai/gtaljaard/assignment-4-speech-command-recognition-with-m3-m5-m11-m18-cnn-networks

M11 Train

In [61]:
!pip install jovian --upgrade --quiet
In [62]:
import jovian
In [63]:
jovian.log_hyperparams(arch='M11', 
                       epochs=epoch_count, 
                       lr=0.01, 
                       scheduler='optim.lr_scheduler.StepLR', 
                       weight_decay=0.0001, 
                       opt='Adam')
[jovian] Hyperparams logged.
In [64]:
#M11 train & validate
log_interval = 20
#epoch_count = 1

pbar_update = 1 / (len(train_loader) + len(test_loader))
losses_train = []
losses_validation = []
accuracy_train = []
accuracy_validation = []

#Use Adam optimiser as used in paper. Weight decay set to 0.0001 
#Initially train with learning rate of 0.01. Use a scheduler to decrease it to 0.001 during training after 20 epochs
optimiser = optim.Adam(model_M11.parameters(), lr=0.01, weight_decay=0.0001)
scheduler = optim.lr_scheduler.StepLR(optimiser, step_size=20, gamma=0.1)  # reduce the learning after 20 epochs by a factor of 10

# Transform must be on same device as model and data
transform = transform.to(device)
with tqdm(total=epoch_count) as pbar:
    for epoch in range(1, epoch_count + 1):
        train(model_M11, epoch, log_interval)
        validate(model_M11, epoch)
        scheduler.step()
HBox(children=(FloatProgress(value=0.0, max=30.0), HTML(value='')))
Train Epoch: 1 [0/84843 (0%)] Loss: 3.585569 Accuracy: 10/84843 (0%) Train Epoch: 1 [8000/84843 (9%)] Loss: 3.572021 Accuracy: 362/84843 (0%) Train Epoch: 1 [16000/84843 (19%)] Loss: 3.366140 Accuracy: 942/84843 (1%) Train Epoch: 1 [24000/84843 (28%)] Loss: 2.997586 Accuracy: 1837/84843 (2%) Train Epoch: 1 [32000/84843 (38%)] Loss: 2.657705 Accuracy: 3367/84843 (4%) Train Epoch: 1 [40000/84843 (47%)] Loss: 2.186844 Accuracy: 5659/84843 (7%) Train Epoch: 1 [48000/84843 (56%)] Loss: 1.931446 Accuracy: 8654/84843 (10%) Train Epoch: 1 [56000/84843 (66%)] Loss: 1.642858 Accuracy: 12430/84843 (15%) Train Epoch: 1 [64000/84843 (75%)] Loss: 1.414458 Accuracy: 16683/84843 (20%) Train Epoch: 1 [72000/84843 (85%)] Loss: 1.356861 Accuracy: 21267/84843 (25%) Train Epoch: 1 [80000/84843 (94%)] Loss: 1.169699 Accuracy: 26279/84843 (31%) Validation Epoch: 1 Loss: 1.179789 Accuracy: 5383/9981 (54%) Train Epoch: 2 [0/84843 (0%)] Loss: 1.098459 Accuracy: 274/84843 (0%) Train Epoch: 2 [8000/84843 (9%)] Loss: 1.152526 Accuracy: 5588/84843 (7%) Train Epoch: 2 [16000/84843 (19%)] Loss: 1.007552 Accuracy: 11260/84843 (13%) Train Epoch: 2 [24000/84843 (28%)] Loss: 0.789378 Accuracy: 17112/84843 (20%) Train Epoch: 2 [32000/84843 (38%)] Loss: 0.897947 Accuracy: 23057/84843 (27%) Train Epoch: 2 [40000/84843 (47%)] Loss: 0.852451 Accuracy: 29137/84843 (34%) Train Epoch: 2 [48000/84843 (56%)] Loss: 0.838927 Accuracy: 35217/84843 (42%) Train Epoch: 2 [56000/84843 (66%)] Loss: 0.595300 Accuracy: 41412/84843 (49%) Train Epoch: 2 [64000/84843 (75%)] Loss: 0.762491 Accuracy: 47669/84843 (56%) Train Epoch: 2 [72000/84843 (85%)] Loss: 0.594743 Accuracy: 54040/84843 (64%) Train Epoch: 2 [80000/84843 (94%)] Loss: 0.665707 Accuracy: 60421/84843 (71%) Validation Epoch: 2 Loss: 0.151432 Accuracy: 6997/9981 (70%) Train Epoch: 3 [0/84843 (0%)] Loss: 0.551428 Accuracy: 327/84843 (0%) Train Epoch: 3 [8000/84843 (9%)] Loss: 0.606383 Accuracy: 6854/84843 (8%) Train Epoch: 3 [16000/84843 (19%)] Loss: 0.525340 Accuracy: 13476/84843 (16%) Train Epoch: 3 [24000/84843 (28%)] Loss: 0.583274 Accuracy: 20129/84843 (24%) Train Epoch: 3 [32000/84843 (38%)] Loss: 0.404753 Accuracy: 26698/84843 (31%) Train Epoch: 3 [40000/84843 (47%)] Loss: 0.583123 Accuracy: 33375/84843 (39%) Train Epoch: 3 [48000/84843 (56%)] Loss: 0.571666 Accuracy: 39994/84843 (47%) Train Epoch: 3 [56000/84843 (66%)] Loss: 0.532828 Accuracy: 46577/84843 (55%) Train Epoch: 3 [64000/84843 (75%)] Loss: 0.545756 Accuracy: 53293/84843 (63%) Train Epoch: 3 [72000/84843 (85%)] Loss: 0.611114 Accuracy: 59954/84843 (71%) Train Epoch: 3 [80000/84843 (94%)] Loss: 0.569022 Accuracy: 66640/84843 (79%) Validation Epoch: 3 Loss: 0.594404 Accuracy: 6899/9981 (69%) Train Epoch: 4 [0/84843 (0%)] Loss: 0.673416 Accuracy: 326/84843 (0%) Train Epoch: 4 [8000/84843 (9%)] Loss: 0.458806 Accuracy: 7003/84843 (8%) Train Epoch: 4 [16000/84843 (19%)] Loss: 0.459296 Accuracy: 13844/84843 (16%) Train Epoch: 4 [24000/84843 (28%)] Loss: 0.443368 Accuracy: 20674/84843 (24%) Train Epoch: 4 [32000/84843 (38%)] Loss: 0.428769 Accuracy: 27482/84843 (32%) Train Epoch: 4 [40000/84843 (47%)] Loss: 0.490135 Accuracy: 34313/84843 (40%) Train Epoch: 4 [48000/84843 (56%)] Loss: 0.488127 Accuracy: 41169/84843 (49%) Train Epoch: 4 [56000/84843 (66%)] Loss: 0.484023 Accuracy: 48017/84843 (57%) Train Epoch: 4 [64000/84843 (75%)] Loss: 0.588856 Accuracy: 54844/84843 (65%) Train Epoch: 4 [72000/84843 (85%)] Loss: 0.490496 Accuracy: 61660/84843 (73%) Train Epoch: 4 [80000/84843 (94%)] Loss: 0.485045 Accuracy: 68506/84843 (81%) Validation Epoch: 4 Loss: 0.694646 Accuracy: 8194/9981 (82%) Train Epoch: 5 [0/84843 (0%)] Loss: 0.403648 Accuracy: 348/84843 (0%) Train Epoch: 5 [8000/84843 (9%)] Loss: 0.461108 Accuracy: 7078/84843 (8%) Train Epoch: 5 [16000/84843 (19%)] Loss: 0.395683 Accuracy: 13956/84843 (16%) Train Epoch: 5 [24000/84843 (28%)] Loss: 0.400695 Accuracy: 20964/84843 (25%) Train Epoch: 5 [32000/84843 (38%)] Loss: 0.363401 Accuracy: 27903/84843 (33%) Train Epoch: 5 [40000/84843 (47%)] Loss: 0.402273 Accuracy: 34788/84843 (41%) Train Epoch: 5 [48000/84843 (56%)] Loss: 0.348425 Accuracy: 41791/84843 (49%) Train Epoch: 5 [56000/84843 (66%)] Loss: 0.406409 Accuracy: 48799/84843 (58%) Train Epoch: 5 [64000/84843 (75%)] Loss: 0.383730 Accuracy: 55719/84843 (66%) Train Epoch: 5 [72000/84843 (85%)] Loss: 0.386004 Accuracy: 62726/84843 (74%) Train Epoch: 5 [80000/84843 (94%)] Loss: 0.418721 Accuracy: 69672/84843 (82%) Validation Epoch: 5 Loss: 0.523503 Accuracy: 8224/9981 (82%) Train Epoch: 6 [0/84843 (0%)] Loss: 0.373766 Accuracy: 348/84843 (0%) Train Epoch: 6 [8000/84843 (9%)] Loss: 0.377695 Accuracy: 7241/84843 (9%) Train Epoch: 6 [16000/84843 (19%)] Loss: 0.402355 Accuracy: 14223/84843 (17%) Train Epoch: 6 [24000/84843 (28%)] Loss: 0.434439 Accuracy: 21228/84843 (25%) Train Epoch: 6 [32000/84843 (38%)] Loss: 0.412714 Accuracy: 28308/84843 (33%) Train Epoch: 6 [40000/84843 (47%)] Loss: 0.408991 Accuracy: 35371/84843 (42%) Train Epoch: 6 [48000/84843 (56%)] Loss: 0.387806 Accuracy: 42464/84843 (50%) Train Epoch: 6 [56000/84843 (66%)] Loss: 0.372128 Accuracy: 49449/84843 (58%) Train Epoch: 6 [64000/84843 (75%)] Loss: 0.414450 Accuracy: 56464/84843 (67%) Train Epoch: 6 [72000/84843 (85%)] Loss: 0.446242 Accuracy: 63476/84843 (75%) Train Epoch: 6 [80000/84843 (94%)] Loss: 0.403002 Accuracy: 70510/84843 (83%) Validation Epoch: 6 Loss: 0.365577 Accuracy: 8391/9981 (84%) Train Epoch: 7 [0/84843 (0%)] Loss: 0.421883 Accuracy: 343/84843 (0%) Train Epoch: 7 [8000/84843 (9%)] Loss: 0.455928 Accuracy: 7280/84843 (9%) Train Epoch: 7 [16000/84843 (19%)] Loss: 0.342951 Accuracy: 14393/84843 (17%) Train Epoch: 7 [24000/84843 (28%)] Loss: 0.364803 Accuracy: 21447/84843 (25%) Train Epoch: 7 [32000/84843 (38%)] Loss: 0.319706 Accuracy: 28579/84843 (34%) Train Epoch: 7 [40000/84843 (47%)] Loss: 0.411836 Accuracy: 35668/84843 (42%) Train Epoch: 7 [48000/84843 (56%)] Loss: 0.399525 Accuracy: 42708/84843 (50%) Train Epoch: 7 [56000/84843 (66%)] Loss: 0.366218 Accuracy: 49792/84843 (59%) Train Epoch: 7 [64000/84843 (75%)] Loss: 0.341258 Accuracy: 56897/84843 (67%) Train Epoch: 7 [72000/84843 (85%)] Loss: 0.419228 Accuracy: 64025/84843 (75%) Train Epoch: 7 [80000/84843 (94%)] Loss: 0.447225 Accuracy: 71045/84843 (84%) Validation Epoch: 7 Loss: 1.498102 Accuracy: 7964/9981 (80%) Train Epoch: 8 [0/84843 (0%)] Loss: 0.269891 Accuracy: 370/84843 (0%) Train Epoch: 8 [8000/84843 (9%)] Loss: 0.402471 Accuracy: 7373/84843 (9%) Train Epoch: 8 [16000/84843 (19%)] Loss: 0.369642 Accuracy: 14475/84843 (17%) Train Epoch: 8 [24000/84843 (28%)] Loss: 0.422687 Accuracy: 21563/84843 (25%) Train Epoch: 8 [32000/84843 (38%)] Loss: 0.472382 Accuracy: 28642/84843 (34%) Train Epoch: 8 [40000/84843 (47%)] Loss: 0.341829 Accuracy: 35835/84843 (42%) Train Epoch: 8 [48000/84843 (56%)] Loss: 0.458805 Accuracy: 42993/84843 (51%) Train Epoch: 8 [56000/84843 (66%)] Loss: 0.431611 Accuracy: 50136/84843 (59%) Train Epoch: 8 [64000/84843 (75%)] Loss: 0.409545 Accuracy: 57194/84843 (67%) Train Epoch: 8 [72000/84843 (85%)] Loss: 0.344727 Accuracy: 64283/84843 (76%) Train Epoch: 8 [80000/84843 (94%)] Loss: 0.439187 Accuracy: 71413/84843 (84%) Validation Epoch: 8 Loss: 0.319593 Accuracy: 8604/9981 (86%) Train Epoch: 9 [0/84843 (0%)] Loss: 0.385267 Accuracy: 350/84843 (0%) Train Epoch: 9 [8000/84843 (9%)] Loss: 0.377325 Accuracy: 7385/84843 (9%) Train Epoch: 9 [16000/84843 (19%)] Loss: 0.391999 Accuracy: 14497/84843 (17%) Train Epoch: 9 [24000/84843 (28%)] Loss: 0.403624 Accuracy: 21677/84843 (26%) Train Epoch: 9 [32000/84843 (38%)] Loss: 0.370835 Accuracy: 28854/84843 (34%) Train Epoch: 9 [40000/84843 (47%)] Loss: 0.309865 Accuracy: 36014/84843 (42%) Train Epoch: 9 [48000/84843 (56%)] Loss: 0.338582 Accuracy: 43208/84843 (51%) Train Epoch: 9 [56000/84843 (66%)] Loss: 0.367929 Accuracy: 50388/84843 (59%) Train Epoch: 9 [64000/84843 (75%)] Loss: 0.445400 Accuracy: 57526/84843 (68%) Train Epoch: 9 [72000/84843 (85%)] Loss: 0.407664 Accuracy: 64694/84843 (76%) Train Epoch: 9 [80000/84843 (94%)] Loss: 0.366267 Accuracy: 71803/84843 (85%) Validation Epoch: 9 Loss: 0.384056 Accuracy: 8636/9981 (87%) Train Epoch: 10 [0/84843 (0%)] Loss: 0.358513 Accuracy: 358/84843 (0%) Train Epoch: 10 [8000/84843 (9%)] Loss: 0.292140 Accuracy: 7485/84843 (9%) Train Epoch: 10 [16000/84843 (19%)] Loss: 0.266518 Accuracy: 14727/84843 (17%) Train Epoch: 10 [24000/84843 (28%)] Loss: 0.273654 Accuracy: 21947/84843 (26%) Train Epoch: 10 [32000/84843 (38%)] Loss: 0.305667 Accuracy: 29147/84843 (34%) Train Epoch: 10 [40000/84843 (47%)] Loss: 0.352562 Accuracy: 36382/84843 (43%) Train Epoch: 10 [48000/84843 (56%)] Loss: 0.318084 Accuracy: 43525/84843 (51%) Train Epoch: 10 [56000/84843 (66%)] Loss: 0.354541 Accuracy: 50698/84843 (60%) Train Epoch: 10 [64000/84843 (75%)] Loss: 0.395522 Accuracy: 57885/84843 (68%) Train Epoch: 10 [72000/84843 (85%)] Loss: 0.362150 Accuracy: 64998/84843 (77%) Train Epoch: 10 [80000/84843 (94%)] Loss: 0.318516 Accuracy: 72134/84843 (85%) Validation Epoch: 10 Loss: 0.326146 Accuracy: 8669/9981 (87%) Train Epoch: 11 [0/84843 (0%)] Loss: 0.239036 Accuracy: 371/84843 (0%) Train Epoch: 11 [8000/84843 (9%)] Loss: 0.297447 Accuracy: 7504/84843 (9%) Train Epoch: 11 [16000/84843 (19%)] Loss: 0.293640 Accuracy: 14730/84843 (17%) Train Epoch: 11 [24000/84843 (28%)] Loss: 0.329325 Accuracy: 21984/84843 (26%) Train Epoch: 11 [32000/84843 (38%)] Loss: 0.313625 Accuracy: 29171/84843 (34%) Train Epoch: 11 [40000/84843 (47%)] Loss: 0.314074 Accuracy: 36410/84843 (43%) Train Epoch: 11 [48000/84843 (56%)] Loss: 0.346095 Accuracy: 43574/84843 (51%) Train Epoch: 11 [56000/84843 (66%)] Loss: 0.381079 Accuracy: 50794/84843 (60%) Train Epoch: 11 [64000/84843 (75%)] Loss: 0.359147 Accuracy: 58002/84843 (68%) Train Epoch: 11 [72000/84843 (85%)] Loss: 0.281165 Accuracy: 65254/84843 (77%) Train Epoch: 11 [80000/84843 (94%)] Loss: 0.421590 Accuracy: 72437/84843 (85%) Validation Epoch: 11 Loss: 0.923039 Accuracy: 8541/9981 (86%) Train Epoch: 12 [0/84843 (0%)] Loss: 0.311262 Accuracy: 367/84843 (0%) Train Epoch: 12 [8000/84843 (9%)] Loss: 0.303268 Accuracy: 7565/84843 (9%) Train Epoch: 12 [16000/84843 (19%)] Loss: 0.218480 Accuracy: 14836/84843 (17%) Train Epoch: 12 [24000/84843 (28%)] Loss: 0.333585 Accuracy: 22093/84843 (26%) Train Epoch: 12 [32000/84843 (38%)] Loss: 0.361903 Accuracy: 29290/84843 (35%) Train Epoch: 12 [40000/84843 (47%)] Loss: 0.366659 Accuracy: 36457/84843 (43%) Train Epoch: 12 [48000/84843 (56%)] Loss: 0.311656 Accuracy: 43650/84843 (51%) Train Epoch: 12 [56000/84843 (66%)] Loss: 0.273842 Accuracy: 50904/84843 (60%) Train Epoch: 12 [64000/84843 (75%)] Loss: 0.295889 Accuracy: 58113/84843 (68%) Train Epoch: 12 [72000/84843 (85%)] Loss: 0.330392 Accuracy: 65332/84843 (77%) Train Epoch: 12 [80000/84843 (94%)] Loss: 0.404742 Accuracy: 72543/84843 (86%) Validation Epoch: 12 Loss: 0.450162 Accuracy: 8325/9981 (83%) Train Epoch: 13 [0/84843 (0%)] Loss: 0.308305 Accuracy: 358/84843 (0%) Train Epoch: 13 [8000/84843 (9%)] Loss: 0.428149 Accuracy: 7532/84843 (9%) Train Epoch: 13 [16000/84843 (19%)] Loss: 0.287258 Accuracy: 14754/84843 (17%) Train Epoch: 13 [24000/84843 (28%)] Loss: 0.247176 Accuracy: 22022/84843 (26%) Train Epoch: 13 [32000/84843 (38%)] Loss: 0.298917 Accuracy: 29278/84843 (35%) Train Epoch: 13 [40000/84843 (47%)] Loss: 0.334775 Accuracy: 36499/84843 (43%) Train Epoch: 13 [48000/84843 (56%)] Loss: 0.305919 Accuracy: 43785/84843 (52%) Train Epoch: 13 [56000/84843 (66%)] Loss: 0.314405 Accuracy: 51026/84843 (60%) Train Epoch: 13 [64000/84843 (75%)] Loss: 0.345493 Accuracy: 58236/84843 (69%) Train Epoch: 13 [72000/84843 (85%)] Loss: 0.326819 Accuracy: 65452/84843 (77%) Train Epoch: 13 [80000/84843 (94%)] Loss: 0.343510 Accuracy: 72643/84843 (86%) Validation Epoch: 13 Loss: 0.238930 Accuracy: 8748/9981 (88%) Train Epoch: 14 [0/84843 (0%)] Loss: 0.282969 Accuracy: 364/84843 (0%) Train Epoch: 14 [8000/84843 (9%)] Loss: 0.341804 Accuracy: 7615/84843 (9%) Train Epoch: 14 [16000/84843 (19%)] Loss: 0.286332 Accuracy: 14905/84843 (18%) Train Epoch: 14 [24000/84843 (28%)] Loss: 0.320405 Accuracy: 22200/84843 (26%) Train Epoch: 14 [32000/84843 (38%)] Loss: 0.376069 Accuracy: 29494/84843 (35%) Train Epoch: 14 [40000/84843 (47%)] Loss: 0.332485 Accuracy: 36724/84843 (43%) Train Epoch: 14 [48000/84843 (56%)] Loss: 0.280653 Accuracy: 43983/84843 (52%) Train Epoch: 14 [56000/84843 (66%)] Loss: 0.228556 Accuracy: 51211/84843 (60%) Train Epoch: 14 [64000/84843 (75%)] Loss: 0.347143 Accuracy: 58468/84843 (69%) Train Epoch: 14 [72000/84843 (85%)] Loss: 0.311370 Accuracy: 65715/84843 (77%) Train Epoch: 14 [80000/84843 (94%)] Loss: 0.370496 Accuracy: 72934/84843 (86%) Validation Epoch: 14 Loss: 0.498596 Accuracy: 8777/9981 (88%) Train Epoch: 15 [0/84843 (0%)] Loss: 0.357493 Accuracy: 356/84843 (0%) Train Epoch: 15 [8000/84843 (9%)] Loss: 0.361007 Accuracy: 7545/84843 (9%) Train Epoch: 15 [16000/84843 (19%)] Loss: 0.296049 Accuracy: 14739/84843 (17%) Train Epoch: 15 [24000/84843 (28%)] Loss: 0.341102 Accuracy: 22033/84843 (26%) Train Epoch: 15 [32000/84843 (38%)] Loss: 0.253771 Accuracy: 29294/84843 (35%) Train Epoch: 15 [40000/84843 (47%)] Loss: 0.295705 Accuracy: 36561/84843 (43%) Train Epoch: 15 [48000/84843 (56%)] Loss: 0.335840 Accuracy: 43862/84843 (52%) Train Epoch: 15 [56000/84843 (66%)] Loss: 0.287391 Accuracy: 51120/84843 (60%) Train Epoch: 15 [64000/84843 (75%)] Loss: 0.208548 Accuracy: 58384/84843 (69%) Train Epoch: 15 [72000/84843 (85%)] Loss: 0.386529 Accuracy: 65627/84843 (77%) Train Epoch: 15 [80000/84843 (94%)] Loss: 0.315614 Accuracy: 72885/84843 (86%) Validation Epoch: 15 Loss: 0.284002 Accuracy: 8598/9981 (86%) Train Epoch: 16 [0/84843 (0%)] Loss: 0.280225 Accuracy: 362/84843 (0%) Train Epoch: 16 [8000/84843 (9%)] Loss: 0.280853 Accuracy: 7574/84843 (9%) Train Epoch: 16 [16000/84843 (19%)] Loss: 0.284227 Accuracy: 14854/84843 (18%) Train Epoch: 16 [24000/84843 (28%)] Loss: 0.287861 Accuracy: 22143/84843 (26%) Train Epoch: 16 [32000/84843 (38%)] Loss: 0.337570 Accuracy: 29402/84843 (35%) Train Epoch: 16 [40000/84843 (47%)] Loss: 0.296959 Accuracy: 36668/84843 (43%) Train Epoch: 16 [48000/84843 (56%)] Loss: 0.308201 Accuracy: 43929/84843 (52%) Train Epoch: 16 [56000/84843 (66%)] Loss: 0.268330 Accuracy: 51190/84843 (60%) Train Epoch: 16 [64000/84843 (75%)] Loss: 0.310984 Accuracy: 58491/84843 (69%) Train Epoch: 16 [72000/84843 (85%)] Loss: 0.270038 Accuracy: 65740/84843 (77%) Train Epoch: 16 [80000/84843 (94%)] Loss: 0.318828 Accuracy: 72984/84843 (86%) Validation Epoch: 16 Loss: 0.389761 Accuracy: 8818/9981 (88%) Train Epoch: 17 [0/84843 (0%)] Loss: 0.243253 Accuracy: 370/84843 (0%) Train Epoch: 17 [8000/84843 (9%)] Loss: 0.318404 Accuracy: 7561/84843 (9%) Train Epoch: 17 [16000/84843 (19%)] Loss: 0.277490 Accuracy: 14837/84843 (17%) Train Epoch: 17 [24000/84843 (28%)] Loss: 0.260782 Accuracy: 22110/84843 (26%) Train Epoch: 17 [32000/84843 (38%)] Loss: 0.353511 Accuracy: 29393/84843 (35%) Train Epoch: 17 [40000/84843 (47%)] Loss: 0.248683 Accuracy: 36666/84843 (43%) Train Epoch: 17 [48000/84843 (56%)] Loss: 0.331552 Accuracy: 43954/84843 (52%) Train Epoch: 17 [56000/84843 (66%)] Loss: 0.301807 Accuracy: 51233/84843 (60%) Train Epoch: 17 [64000/84843 (75%)] Loss: 0.318565 Accuracy: 58542/84843 (69%) Train Epoch: 17 [72000/84843 (85%)] Loss: 0.275935 Accuracy: 65817/84843 (78%) Train Epoch: 17 [80000/84843 (94%)] Loss: 0.309896 Accuracy: 73098/84843 (86%) Validation Epoch: 17 Loss: 0.947130 Accuracy: 8833/9981 (88%) Train Epoch: 18 [0/84843 (0%)] Loss: 0.224707 Accuracy: 369/84843 (0%) Train Epoch: 18 [8000/84843 (9%)] Loss: 0.335946 Accuracy: 7550/84843 (9%) Train Epoch: 18 [16000/84843 (19%)] Loss: 0.288693 Accuracy: 14865/84843 (18%) Train Epoch: 18 [24000/84843 (28%)] Loss: 0.234314 Accuracy: 22158/84843 (26%) Train Epoch: 18 [32000/84843 (38%)] Loss: 0.292084 Accuracy: 29471/84843 (35%) Train Epoch: 18 [40000/84843 (47%)] Loss: 0.292446 Accuracy: 36768/84843 (43%) Train Epoch: 18 [48000/84843 (56%)] Loss: 0.215732 Accuracy: 44119/84843 (52%) Train Epoch: 18 [56000/84843 (66%)] Loss: 0.214704 Accuracy: 51448/84843 (61%) Train Epoch: 18 [64000/84843 (75%)] Loss: 0.239321 Accuracy: 58788/84843 (69%) Train Epoch: 18 [72000/84843 (85%)] Loss: 0.293973 Accuracy: 66081/84843 (78%) Train Epoch: 18 [80000/84843 (94%)] Loss: 0.312830 Accuracy: 73329/84843 (86%) Validation Epoch: 18 Loss: 0.305132 Accuracy: 9001/9981 (90%) Train Epoch: 19 [0/84843 (0%)] Loss: 0.264762 Accuracy: 363/84843 (0%) Train Epoch: 19 [8000/84843 (9%)] Loss: 0.338734 Accuracy: 7551/84843 (9%) Train Epoch: 19 [16000/84843 (19%)] Loss: 0.253438 Accuracy: 14818/84843 (17%) Train Epoch: 19 [24000/84843 (28%)] Loss: 0.232150 Accuracy: 22167/84843 (26%) Train Epoch: 19 [32000/84843 (38%)] Loss: 0.234096 Accuracy: 29489/84843 (35%) Train Epoch: 19 [40000/84843 (47%)] Loss: 0.297634 Accuracy: 36769/84843 (43%) Train Epoch: 19 [48000/84843 (56%)] Loss: 0.289542 Accuracy: 44054/84843 (52%) Train Epoch: 19 [56000/84843 (66%)] Loss: 0.286484 Accuracy: 51359/84843 (61%) Train Epoch: 19 [64000/84843 (75%)] Loss: 0.295201 Accuracy: 58642/84843 (69%) Train Epoch: 19 [72000/84843 (85%)] Loss: 0.315682 Accuracy: 65854/84843 (78%) Train Epoch: 19 [80000/84843 (94%)] Loss: 0.343588 Accuracy: 73141/84843 (86%) Validation Epoch: 19 Loss: 0.680973 Accuracy: 8606/9981 (86%) Train Epoch: 20 [0/84843 (0%)] Loss: 0.283653 Accuracy: 363/84843 (0%) Train Epoch: 20 [8000/84843 (9%)] Loss: 0.304983 Accuracy: 7523/84843 (9%) Train Epoch: 20 [16000/84843 (19%)] Loss: 0.308048 Accuracy: 14816/84843 (17%) Train Epoch: 20 [24000/84843 (28%)] Loss: 0.285467 Accuracy: 22156/84843 (26%) Train Epoch: 20 [32000/84843 (38%)] Loss: 0.281023 Accuracy: 29484/84843 (35%) Train Epoch: 20 [40000/84843 (47%)] Loss: 0.254127 Accuracy: 36812/84843 (43%) Train Epoch: 20 [48000/84843 (56%)] Loss: 0.273000 Accuracy: 44116/84843 (52%) Train Epoch: 20 [56000/84843 (66%)] Loss: 0.242770 Accuracy: 51410/84843 (61%) Train Epoch: 20 [64000/84843 (75%)] Loss: 0.397047 Accuracy: 58661/84843 (69%) Train Epoch: 20 [72000/84843 (85%)] Loss: 0.296824 Accuracy: 65922/84843 (78%) Train Epoch: 20 [80000/84843 (94%)] Loss: 0.372225 Accuracy: 73245/84843 (86%) Validation Epoch: 20 Loss: 0.215204 Accuracy: 8737/9981 (88%) Train Epoch: 21 [0/84843 (0%)] Loss: 0.267304 Accuracy: 372/84843 (0%) Train Epoch: 21 [8000/84843 (9%)] Loss: 0.173948 Accuracy: 7810/84843 (9%) Train Epoch: 21 [16000/84843 (19%)] Loss: 0.231238 Accuracy: 15290/84843 (18%) Train Epoch: 21 [24000/84843 (28%)] Loss: 0.227510 Accuracy: 22794/84843 (27%) Train Epoch: 21 [32000/84843 (38%)] Loss: 0.195003 Accuracy: 30344/84843 (36%) Train Epoch: 21 [40000/84843 (47%)] Loss: 0.170166 Accuracy: 37916/84843 (45%) Train Epoch: 21 [48000/84843 (56%)] Loss: 0.141349 Accuracy: 45482/84843 (54%) Train Epoch: 21 [56000/84843 (66%)] Loss: 0.125504 Accuracy: 53067/84843 (63%) Train Epoch: 21 [64000/84843 (75%)] Loss: 0.219653 Accuracy: 60637/84843 (71%) Train Epoch: 21 [72000/84843 (85%)] Loss: 0.167250 Accuracy: 68228/84843 (80%) Train Epoch: 21 [80000/84843 (94%)] Loss: 0.164657 Accuracy: 75813/84843 (89%) Validation Epoch: 21 Loss: 0.176750 Accuracy: 9345/9981 (94%) Train Epoch: 22 [0/84843 (0%)] Loss: 0.147646 Accuracy: 382/84843 (0%) Train Epoch: 22 [8000/84843 (9%)] Loss: 0.208529 Accuracy: 8013/84843 (9%) Train Epoch: 22 [16000/84843 (19%)] Loss: 0.163333 Accuracy: 15623/84843 (18%) Train Epoch: 22 [24000/84843 (28%)] Loss: 0.140554 Accuracy: 23234/84843 (27%) Train Epoch: 22 [32000/84843 (38%)] Loss: 0.238987 Accuracy: 30865/84843 (36%) Train Epoch: 22 [40000/84843 (47%)] Loss: 0.153710 Accuracy: 38484/84843 (45%) Train Epoch: 22 [48000/84843 (56%)] Loss: 0.107350 Accuracy: 46103/84843 (54%) Train Epoch: 22 [56000/84843 (66%)] Loss: 0.141491 Accuracy: 53714/84843 (63%) Train Epoch: 22 [64000/84843 (75%)] Loss: 0.157639 Accuracy: 61366/84843 (72%) Train Epoch: 22 [72000/84843 (85%)] Loss: 0.194207 Accuracy: 68974/84843 (81%) Train Epoch: 22 [80000/84843 (94%)] Loss: 0.155135 Accuracy: 76612/84843 (90%) Validation Epoch: 22 Loss: 0.189430 Accuracy: 9384/9981 (94%) Train Epoch: 23 [0/84843 (0%)] Loss: 0.134887 Accuracy: 381/84843 (0%) Train Epoch: 23 [8000/84843 (9%)] Loss: 0.128560 Accuracy: 8068/84843 (10%) Train Epoch: 23 [16000/84843 (19%)] Loss: 0.177912 Accuracy: 15728/84843 (19%) Train Epoch: 23 [24000/84843 (28%)] Loss: 0.126023 Accuracy: 23385/84843 (28%) Train Epoch: 23 [32000/84843 (38%)] Loss: 0.136928 Accuracy: 31049/84843 (37%) Train Epoch: 23 [40000/84843 (47%)] Loss: 0.108540 Accuracy: 38722/84843 (46%) Train Epoch: 23 [48000/84843 (56%)] Loss: 0.118566 Accuracy: 46401/84843 (55%) Train Epoch: 23 [56000/84843 (66%)] Loss: 0.144308 Accuracy: 54052/84843 (64%) Train Epoch: 23 [64000/84843 (75%)] Loss: 0.137485 Accuracy: 61710/84843 (73%) Train Epoch: 23 [72000/84843 (85%)] Loss: 0.142482 Accuracy: 69385/84843 (82%) Train Epoch: 23 [80000/84843 (94%)] Loss: 0.162241 Accuracy: 77051/84843 (91%) Validation Epoch: 23 Loss: 0.178361 Accuracy: 9383/9981 (94%) Train Epoch: 24 [0/84843 (0%)] Loss: 0.104372 Accuracy: 388/84843 (0%) Train Epoch: 24 [8000/84843 (9%)] Loss: 0.110040 Accuracy: 8084/84843 (10%) Train Epoch: 24 [16000/84843 (19%)] Loss: 0.131587 Accuracy: 15755/84843 (19%) Train Epoch: 24 [24000/84843 (28%)] Loss: 0.115006 Accuracy: 23427/84843 (28%) Train Epoch: 24 [32000/84843 (38%)] Loss: 0.104690 Accuracy: 31114/84843 (37%) Train Epoch: 24 [40000/84843 (47%)] Loss: 0.136706 Accuracy: 38786/84843 (46%) Train Epoch: 24 [48000/84843 (56%)] Loss: 0.155501 Accuracy: 46472/84843 (55%) Train Epoch: 24 [56000/84843 (66%)] Loss: 0.140725 Accuracy: 54170/84843 (64%) Train Epoch: 24 [64000/84843 (75%)] Loss: 0.140010 Accuracy: 61858/84843 (73%) Train Epoch: 24 [72000/84843 (85%)] Loss: 0.137161 Accuracy: 69523/84843 (82%) Train Epoch: 24 [80000/84843 (94%)] Loss: 0.162065 Accuracy: 77180/84843 (91%) Validation Epoch: 24 Loss: 0.176329 Accuracy: 9382/9981 (94%) Train Epoch: 25 [0/84843 (0%)] Loss: 0.135775 Accuracy: 388/84843 (0%) Train Epoch: 25 [8000/84843 (9%)] Loss: 0.095977 Accuracy: 8093/84843 (10%) Train Epoch: 25 [16000/84843 (19%)] Loss: 0.176542 Accuracy: 15777/84843 (19%) Train Epoch: 25 [24000/84843 (28%)] Loss: 0.105998 Accuracy: 23527/84843 (28%) Train Epoch: 25 [32000/84843 (38%)] Loss: 0.124498 Accuracy: 31254/84843 (37%) Train Epoch: 25 [40000/84843 (47%)] Loss: 0.110246 Accuracy: 38944/84843 (46%) Train Epoch: 25 [48000/84843 (56%)] Loss: 0.110578 Accuracy: 46665/84843 (55%) Train Epoch: 25 [56000/84843 (66%)] Loss: 0.108418 Accuracy: 54366/84843 (64%) Train Epoch: 25 [64000/84843 (75%)] Loss: 0.127966 Accuracy: 62081/84843 (73%) Train Epoch: 25 [72000/84843 (85%)] Loss: 0.129688 Accuracy: 69766/84843 (82%) Train Epoch: 25 [80000/84843 (94%)] Loss: 0.116668 Accuracy: 77447/84843 (91%) Validation Epoch: 25 Loss: 0.182534 Accuracy: 9369/9981 (94%) Train Epoch: 26 [0/84843 (0%)] Loss: 0.132125 Accuracy: 386/84843 (0%) Train Epoch: 26 [8000/84843 (9%)] Loss: 0.109317 Accuracy: 8133/84843 (10%) Train Epoch: 26 [16000/84843 (19%)] Loss: 0.101374 Accuracy: 15870/84843 (19%) Train Epoch: 26 [24000/84843 (28%)] Loss: 0.114454 Accuracy: 23603/84843 (28%) Train Epoch: 26 [32000/84843 (38%)] Loss: 0.108546 Accuracy: 31321/84843 (37%) Train Epoch: 26 [40000/84843 (47%)] Loss: 0.105950 Accuracy: 39057/84843 (46%) Train Epoch: 26 [48000/84843 (56%)] Loss: 0.113966 Accuracy: 46762/84843 (55%) Train Epoch: 26 [56000/84843 (66%)] Loss: 0.183211 Accuracy: 54477/84843 (64%) Train Epoch: 26 [64000/84843 (75%)] Loss: 0.140960 Accuracy: 62191/84843 (73%) Train Epoch: 26 [72000/84843 (85%)] Loss: 0.100069 Accuracy: 69868/84843 (82%) Train Epoch: 26 [80000/84843 (94%)] Loss: 0.086330 Accuracy: 77583/84843 (91%) Validation Epoch: 26 Loss: 0.214510 Accuracy: 9376/9981 (94%) Train Epoch: 27 [0/84843 (0%)] Loss: 0.128982 Accuracy: 387/84843 (0%) Train Epoch: 27 [8000/84843 (9%)] Loss: 0.157619 Accuracy: 8116/84843 (10%) Train Epoch: 27 [16000/84843 (19%)] Loss: 0.100427 Accuracy: 15861/84843 (19%) Train Epoch: 27 [24000/84843 (28%)] Loss: 0.095307 Accuracy: 23634/84843 (28%) Train Epoch: 27 [32000/84843 (38%)] Loss: 0.081402 Accuracy: 31378/84843 (37%) Train Epoch: 27 [40000/84843 (47%)] Loss: 0.129373 Accuracy: 39115/84843 (46%) Train Epoch: 27 [48000/84843 (56%)] Loss: 0.119725 Accuracy: 46861/84843 (55%) Train Epoch: 27 [56000/84843 (66%)] Loss: 0.099637 Accuracy: 54600/84843 (64%) Train Epoch: 27 [64000/84843 (75%)] Loss: 0.105555 Accuracy: 62293/84843 (73%) Train Epoch: 27 [72000/84843 (85%)] Loss: 0.085223 Accuracy: 70023/84843 (83%) Train Epoch: 27 [80000/84843 (94%)] Loss: 0.139224 Accuracy: 77692/84843 (92%) Validation Epoch: 27 Loss: 0.188030 Accuracy: 9366/9981 (94%) Train Epoch: 28 [0/84843 (0%)] Loss: 0.093740 Accuracy: 390/84843 (0%) Train Epoch: 28 [8000/84843 (9%)] Loss: 0.090493 Accuracy: 8152/84843 (10%) Train Epoch: 28 [16000/84843 (19%)] Loss: 0.097867 Accuracy: 15924/84843 (19%) Train Epoch: 28 [24000/84843 (28%)] Loss: 0.073747 Accuracy: 23688/84843 (28%) Train Epoch: 28 [32000/84843 (38%)] Loss: 0.112657 Accuracy: 31434/84843 (37%) Train Epoch: 28 [40000/84843 (47%)] Loss: 0.166300 Accuracy: 39162/84843 (46%) Train Epoch: 28 [48000/84843 (56%)] Loss: 0.069574 Accuracy: 46930/84843 (55%) Train Epoch: 28 [56000/84843 (66%)] Loss: 0.107159 Accuracy: 54657/84843 (64%) Train Epoch: 28 [64000/84843 (75%)] Loss: 0.109884 Accuracy: 62414/84843 (74%) Train Epoch: 28 [72000/84843 (85%)] Loss: 0.136314 Accuracy: 70161/84843 (83%) Train Epoch: 28 [80000/84843 (94%)] Loss: 0.076849 Accuracy: 77902/84843 (92%) Validation Epoch: 28 Loss: 0.237624 Accuracy: 9354/9981 (94%) Train Epoch: 29 [0/84843 (0%)] Loss: 0.107723 Accuracy: 390/84843 (0%) Train Epoch: 29 [8000/84843 (9%)] Loss: 0.131515 Accuracy: 8157/84843 (10%) Train Epoch: 29 [16000/84843 (19%)] Loss: 0.051344 Accuracy: 15955/84843 (19%) Train Epoch: 29 [24000/84843 (28%)] Loss: 0.080669 Accuracy: 23718/84843 (28%) Train Epoch: 29 [32000/84843 (38%)] Loss: 0.076624 Accuracy: 31524/84843 (37%) Train Epoch: 29 [40000/84843 (47%)] Loss: 0.118137 Accuracy: 39282/84843 (46%) Train Epoch: 29 [48000/84843 (56%)] Loss: 0.087313 Accuracy: 47064/84843 (55%) Train Epoch: 29 [56000/84843 (66%)] Loss: 0.107410 Accuracy: 54816/84843 (65%) Train Epoch: 29 [64000/84843 (75%)] Loss: 0.094526 Accuracy: 62568/84843 (74%) Train Epoch: 29 [72000/84843 (85%)] Loss: 0.097772 Accuracy: 70297/84843 (83%) Train Epoch: 29 [80000/84843 (94%)] Loss: 0.094308 Accuracy: 78042/84843 (92%) Validation Epoch: 29 Loss: 0.204033 Accuracy: 9340/9981 (94%) Train Epoch: 30 [0/84843 (0%)] Loss: 0.101887 Accuracy: 385/84843 (0%) Train Epoch: 30 [8000/84843 (9%)] Loss: 0.096784 Accuracy: 8172/84843 (10%) Train Epoch: 30 [16000/84843 (19%)] Loss: 0.129782 Accuracy: 15963/84843 (19%) Train Epoch: 30 [24000/84843 (28%)] Loss: 0.064208 Accuracy: 23757/84843 (28%) Train Epoch: 30 [32000/84843 (38%)] Loss: 0.080479 Accuracy: 31553/84843 (37%) Train Epoch: 30 [40000/84843 (47%)] Loss: 0.100267 Accuracy: 39333/84843 (46%) Train Epoch: 30 [48000/84843 (56%)] Loss: 0.133616 Accuracy: 47123/84843 (56%) Train Epoch: 30 [56000/84843 (66%)] Loss: 0.115959 Accuracy: 54880/84843 (65%) Train Epoch: 30 [64000/84843 (75%)] Loss: 0.108037 Accuracy: 62655/84843 (74%) Train Epoch: 30 [72000/84843 (85%)] Loss: 0.103674 Accuracy: 70411/84843 (83%) Train Epoch: 30 [80000/84843 (94%)] Loss: 0.078238 Accuracy: 78180/84843 (92%) Validation Epoch: 30 Loss: 0.221915 Accuracy: 9321/9981 (93%)
In [65]:
# Plot training loss
plt.plot(losses_train, 'b', label='Train loss')
plt.legend(loc="upper left")
plt.title("M11 model training losses over all iterations")
plt.show()

# Plot validation loss
plt.plot(losses_validation, 'r', label='Valid loss')
plt.legend(loc="upper left")
plt.title("M11 model validation losses over validation epochs")
plt.show()

# Plot accuracy
plt.plot(accuracy_train, 'b', label='Train acc')
plt.plot(accuracy_validation,'r', label ='Valid acc')
plt.legend(loc="upper left")
plt.title("M11 model training & validation accuracy over epochs")
plt.show()
Notebook Image
Notebook Image
Notebook Image
In [66]:
M11_test_acc_result = test(model_M11)
print(M11_test_acc_result)
Test set accuracy: 10220/11005 (93%) 92.86687869150386
In [67]:
jovian.log_metrics({ 'M11_test_acc_result': M11_test_acc_result })
[jovian] Metrics logged.
In [68]:
jovian.commit(project=project_name)
[jovian] Detected Colab notebook... [jovian] Uploading colab notebook to Jovian... [jovian] Capturing environment.. [jovian] Attaching records (metrics, hyperparameters, dataset etc.) [jovian] Committed successfully! https://jovian.ai/gtaljaard/assignment-4-speech-command-recognition-with-m3-m5-m11-m18-cnn-networks

M18 Train

In [69]:
!pip install jovian --upgrade --quiet
In [70]:
import jovian
In [71]:
jovian.log_hyperparams(arch='M18', 
                       epochs=epoch_count, 
                       lr=0.01, 
                       scheduler='optim.lr_scheduler.StepLR', 
                       weight_decay=0.0001, 
                       opt='Adam')
[jovian] Hyperparams logged.
In [72]:
#M18 train & validate
log_interval = 20
#epoch_count = 1

pbar_update = 1 / (len(train_loader) + len(test_loader))
losses_train = []
losses_validation = []
accuracy_train = []
accuracy_validation = []

#Use Adam optimiser as used in paper. Weight decay set to 0.0001 
#Initially train with learning rate of 0.01. Use a scheduler to decrease it to 0.001 during training after 20 epochs
optimiser = optim.Adam(model_M18.parameters(), lr=0.01, weight_decay=0.0001)
scheduler = optim.lr_scheduler.StepLR(optimiser, step_size=20, gamma=0.1)  # reduce the learning after 20 epochs by a factor of 10

# Transform must be on same device as model and data
transform = transform.to(device)
with tqdm(total=epoch_count) as pbar:
    for epoch in range(1, epoch_count + 1):
        train(model_M18, epoch, log_interval)
        validate(model_M18, epoch)
        scheduler.step()
HBox(children=(FloatProgress(value=0.0, max=30.0), HTML(value='')))
Train Epoch: 1 [0/84843 (0%)] Loss: 3.587171 Accuracy: 17/84843 (0%) Train Epoch: 1 [8000/84843 (9%)] Loss: 3.525383 Accuracy: 301/84843 (0%) Train Epoch: 1 [16000/84843 (19%)] Loss: 3.425296 Accuracy: 671/84843 (1%) Train Epoch: 1 [24000/84843 (28%)] Loss: 3.337733 Accuracy: 1248/84843 (1%) Train Epoch: 1 [32000/84843 (38%)] Loss: 3.189684 Accuracy: 1893/84843 (2%) Train Epoch: 1 [40000/84843 (47%)] Loss: 3.102335 Accuracy: 2694/84843 (3%) Train Epoch: 1 [48000/84843 (56%)] Loss: 2.866321 Accuracy: 3877/84843 (5%) Train Epoch: 1 [56000/84843 (66%)] Loss: 2.311564 Accuracy: 5835/84843 (7%) Train Epoch: 1 [64000/84843 (75%)] Loss: 2.146545 Accuracy: 8445/84843 (10%) Train Epoch: 1 [72000/84843 (85%)] Loss: 1.920300 Accuracy: 11544/84843 (14%) Train Epoch: 1 [80000/84843 (94%)] Loss: 1.602972 Accuracy: 15001/84843 (18%) Validation Epoch: 1 Loss: 2.559460 Accuracy: 1518/9981 (15%) Train Epoch: 2 [0/84843 (0%)] Loss: 1.932589 Accuracy: 164/84843 (0%) Train Epoch: 2 [8000/84843 (9%)] Loss: 1.562580 Accuracy: 3906/84843 (5%) Train Epoch: 2 [16000/84843 (19%)] Loss: 1.636047 Accuracy: 8110/84843 (10%) Train Epoch: 2 [24000/84843 (28%)] Loss: 1.459603 Accuracy: 12458/84843 (15%) Train Epoch: 2 [32000/84843 (38%)] Loss: 1.220435 Accuracy: 17077/84843 (20%) Train Epoch: 2 [40000/84843 (47%)] Loss: 1.435502 Accuracy: 21901/84843 (26%) Train Epoch: 2 [48000/84843 (56%)] Loss: 1.340975 Accuracy: 26812/84843 (32%) Train Epoch: 2 [56000/84843 (66%)] Loss: 1.224982 Accuracy: 31731/84843 (37%) Train Epoch: 2 [64000/84843 (75%)] Loss: 1.097089 Accuracy: 36898/84843 (43%) Train Epoch: 2 [72000/84843 (85%)] Loss: 1.128029 Accuracy: 42182/84843 (50%) Train Epoch: 2 [80000/84843 (94%)] Loss: 1.244310 Accuracy: 47595/84843 (56%) Validation Epoch: 2 Loss: 1.385597 Accuracy: 4490/9981 (45%) Train Epoch: 3 [0/84843 (0%)] Loss: 1.226270 Accuracy: 251/84843 (0%) Train Epoch: 3 [8000/84843 (9%)] Loss: 1.099689 Accuracy: 5481/84843 (6%) Train Epoch: 3 [16000/84843 (19%)] Loss: 1.027139 Accuracy: 11086/84843 (13%) Train Epoch: 3 [24000/84843 (28%)] Loss: 1.084924 Accuracy: 16722/84843 (20%) Train Epoch: 3 [32000/84843 (38%)] Loss: 0.955434 Accuracy: 22483/84843 (26%) Train Epoch: 3 [40000/84843 (47%)] Loss: 1.001360 Accuracy: 28245/84843 (33%) Train Epoch: 3 [48000/84843 (56%)] Loss: 0.892849 Accuracy: 34084/84843 (40%) Train Epoch: 3 [56000/84843 (66%)] Loss: 0.951414 Accuracy: 39989/84843 (47%) Train Epoch: 3 [64000/84843 (75%)] Loss: 0.827529 Accuracy: 45984/84843 (54%) Train Epoch: 3 [72000/84843 (85%)] Loss: 0.837209 Accuracy: 51928/84843 (61%) Train Epoch: 3 [80000/84843 (94%)] Loss: 0.896117 Accuracy: 57932/84843 (68%) Validation Epoch: 3 Loss: 1.863572 Accuracy: 5439/9981 (54%) Train Epoch: 4 [0/84843 (0%)] Loss: 0.754811 Accuracy: 310/84843 (0%) Train Epoch: 4 [8000/84843 (9%)] Loss: 0.789911 Accuracy: 6258/84843 (7%) Train Epoch: 4 [16000/84843 (19%)] Loss: 0.797237 Accuracy: 12363/84843 (15%) Train Epoch: 4 [24000/84843 (28%)] Loss: 0.676867 Accuracy: 18618/84843 (22%) Train Epoch: 4 [32000/84843 (38%)] Loss: 0.709155 Accuracy: 24909/84843 (29%) Train Epoch: 4 [40000/84843 (47%)] Loss: 0.598464 Accuracy: 31189/84843 (37%) Train Epoch: 4 [48000/84843 (56%)] Loss: 0.759475 Accuracy: 37524/84843 (44%) Train Epoch: 4 [56000/84843 (66%)] Loss: 0.627822 Accuracy: 43903/84843 (52%) Train Epoch: 4 [64000/84843 (75%)] Loss: 0.645737 Accuracy: 50217/84843 (59%) Train Epoch: 4 [72000/84843 (85%)] Loss: 0.614083 Accuracy: 56614/84843 (67%) Train Epoch: 4 [80000/84843 (94%)] Loss: 0.584406 Accuracy: 62983/84843 (74%) Validation Epoch: 4 Loss: 1.092978 Accuracy: 6667/9981 (67%) Train Epoch: 5 [0/84843 (0%)] Loss: 0.704897 Accuracy: 315/84843 (0%) Train Epoch: 5 [8000/84843 (9%)] Loss: 0.738553 Accuracy: 6522/84843 (8%) Train Epoch: 5 [16000/84843 (19%)] Loss: 0.573499 Accuracy: 13081/84843 (15%) Train Epoch: 5 [24000/84843 (28%)] Loss: 0.619337 Accuracy: 19610/84843 (23%) Train Epoch: 5 [32000/84843 (38%)] Loss: 0.590924 Accuracy: 26160/84843 (31%) Train Epoch: 5 [40000/84843 (47%)] Loss: 0.593159 Accuracy: 32749/84843 (39%) Train Epoch: 5 [48000/84843 (56%)] Loss: 0.714651 Accuracy: 39342/84843 (46%) Train Epoch: 5 [56000/84843 (66%)] Loss: 0.639842 Accuracy: 45823/84843 (54%) Train Epoch: 5 [64000/84843 (75%)] Loss: 0.599553 Accuracy: 52423/84843 (62%) Train Epoch: 5 [72000/84843 (85%)] Loss: 0.529884 Accuracy: 59100/84843 (70%) Train Epoch: 5 [80000/84843 (94%)] Loss: 0.688704 Accuracy: 65739/84843 (77%) Validation Epoch: 5 Loss: 1.391300 Accuracy: 6824/9981 (68%) Train Epoch: 6 [0/84843 (0%)] Loss: 0.554447 Accuracy: 340/84843 (0%) Train Epoch: 6 [8000/84843 (9%)] Loss: 0.640327 Accuracy: 6866/84843 (8%) Train Epoch: 6 [16000/84843 (19%)] Loss: 0.582014 Accuracy: 13509/84843 (16%) Train Epoch: 6 [24000/84843 (28%)] Loss: 0.518994 Accuracy: 20204/84843 (24%) Train Epoch: 6 [32000/84843 (38%)] Loss: 0.639192 Accuracy: 26883/84843 (32%) Train Epoch: 6 [40000/84843 (47%)] Loss: 0.407475 Accuracy: 33715/84843 (40%) Train Epoch: 6 [48000/84843 (56%)] Loss: 0.531720 Accuracy: 40428/84843 (48%) Train Epoch: 6 [56000/84843 (66%)] Loss: 0.485748 Accuracy: 47181/84843 (56%) Train Epoch: 6 [64000/84843 (75%)] Loss: 0.674965 Accuracy: 53907/84843 (64%) Train Epoch: 6 [72000/84843 (85%)] Loss: 0.465032 Accuracy: 60620/84843 (71%) Train Epoch: 6 [80000/84843 (94%)] Loss: 0.436731 Accuracy: 67298/84843 (79%) Validation Epoch: 6 Loss: 1.136094 Accuracy: 6398/9981 (64%) Train Epoch: 7 [0/84843 (0%)] Loss: 0.609077 Accuracy: 322/84843 (0%) Train Epoch: 7 [8000/84843 (9%)] Loss: 0.482748 Accuracy: 7009/84843 (8%) Train Epoch: 7 [16000/84843 (19%)] Loss: 0.468035 Accuracy: 13816/84843 (16%) Train Epoch: 7 [24000/84843 (28%)] Loss: 0.461517 Accuracy: 20587/84843 (24%) Train Epoch: 7 [32000/84843 (38%)] Loss: 0.458713 Accuracy: 27386/84843 (32%) Train Epoch: 7 [40000/84843 (47%)] Loss: 0.377822 Accuracy: 34168/84843 (40%) Train Epoch: 7 [48000/84843 (56%)] Loss: 0.423099 Accuracy: 40951/84843 (48%) Train Epoch: 7 [56000/84843 (66%)] Loss: 0.504748 Accuracy: 47744/84843 (56%) Train Epoch: 7 [64000/84843 (75%)] Loss: 0.450286 Accuracy: 54484/84843 (64%) Train Epoch: 7 [72000/84843 (85%)] Loss: 0.499776 Accuracy: 61246/84843 (72%) Train Epoch: 7 [80000/84843 (94%)] Loss: 0.387165 Accuracy: 68027/84843 (80%) Validation Epoch: 7 Loss: 0.971592 Accuracy: 7769/9981 (78%) Train Epoch: 8 [0/84843 (0%)] Loss: 0.541201 Accuracy: 338/84843 (0%) Train Epoch: 8 [8000/84843 (9%)] Loss: 0.508728 Accuracy: 7047/84843 (8%) Train Epoch: 8 [16000/84843 (19%)] Loss: 0.455650 Accuracy: 13858/84843 (16%) Train Epoch: 8 [24000/84843 (28%)] Loss: 0.543722 Accuracy: 20616/84843 (24%) Train Epoch: 8 [32000/84843 (38%)] Loss: 0.517971 Accuracy: 27523/84843 (32%) Train Epoch: 8 [40000/84843 (47%)] Loss: 0.377575 Accuracy: 34327/84843 (40%) Train Epoch: 8 [48000/84843 (56%)] Loss: 0.507916 Accuracy: 41141/84843 (48%) Train Epoch: 8 [56000/84843 (66%)] Loss: 0.508274 Accuracy: 47976/84843 (57%) Train Epoch: 8 [64000/84843 (75%)] Loss: 0.467154 Accuracy: 54756/84843 (65%) Train Epoch: 8 [72000/84843 (85%)] Loss: 0.524475 Accuracy: 61544/84843 (73%) Train Epoch: 8 [80000/84843 (94%)] Loss: 0.378832 Accuracy: 68341/84843 (81%) Validation Epoch: 8 Loss: 0.773573 Accuracy: 7582/9981 (76%) Train Epoch: 9 [0/84843 (0%)] Loss: 0.461437 Accuracy: 339/84843 (0%) Train Epoch: 9 [8000/84843 (9%)] Loss: 0.539451 Accuracy: 7133/84843 (8%) Train Epoch: 9 [16000/84843 (19%)] Loss: 0.464330 Accuracy: 13927/84843 (16%) Train Epoch: 9 [24000/84843 (28%)] Loss: 0.385333 Accuracy: 20835/84843 (25%) Train Epoch: 9 [32000/84843 (38%)] Loss: 0.515938 Accuracy: 27712/84843 (33%) Train Epoch: 9 [40000/84843 (47%)] Loss: 0.486135 Accuracy: 34533/84843 (41%) Train Epoch: 9 [48000/84843 (56%)] Loss: 0.425401 Accuracy: 41387/84843 (49%) Train Epoch: 9 [56000/84843 (66%)] Loss: 0.570434 Accuracy: 48331/84843 (57%) Train Epoch: 9 [64000/84843 (75%)] Loss: 0.404307 Accuracy: 55215/84843 (65%) Train Epoch: 9 [72000/84843 (85%)] Loss: 0.530527 Accuracy: 62178/84843 (73%) Train Epoch: 9 [80000/84843 (94%)] Loss: 0.540011 Accuracy: 69025/84843 (81%) Validation Epoch: 9 Loss: 0.994207 Accuracy: 7221/9981 (72%) Train Epoch: 10 [0/84843 (0%)] Loss: 0.407728 Accuracy: 349/84843 (0%) Train Epoch: 10 [8000/84843 (9%)] Loss: 0.441685 Accuracy: 7098/84843 (8%) Train Epoch: 10 [16000/84843 (19%)] Loss: 0.405008 Accuracy: 14005/84843 (17%) Train Epoch: 10 [24000/84843 (28%)] Loss: 0.461201 Accuracy: 20809/84843 (25%) Train Epoch: 10 [32000/84843 (38%)] Loss: 0.416093 Accuracy: 27652/84843 (33%) Train Epoch: 10 [40000/84843 (47%)] Loss: 0.493316 Accuracy: 34511/84843 (41%) Train Epoch: 10 [48000/84843 (56%)] Loss: 0.443412 Accuracy: 41453/84843 (49%) Train Epoch: 10 [56000/84843 (66%)] Loss: 0.387583 Accuracy: 48402/84843 (57%) Train Epoch: 10 [64000/84843 (75%)] Loss: 0.422860 Accuracy: 55311/84843 (65%) Train Epoch: 10 [72000/84843 (85%)] Loss: 0.410142 Accuracy: 62226/84843 (73%) Train Epoch: 10 [80000/84843 (94%)] Loss: 0.489756 Accuracy: 69169/84843 (82%) Validation Epoch: 10 Loss: 0.319569 Accuracy: 7873/9981 (79%) Train Epoch: 11 [0/84843 (0%)] Loss: 0.441598 Accuracy: 346/84843 (0%) Train Epoch: 11 [8000/84843 (9%)] Loss: 0.327542 Accuracy: 7175/84843 (8%) Train Epoch: 11 [16000/84843 (19%)] Loss: 0.549361 Accuracy: 14114/84843 (17%) Train Epoch: 11 [24000/84843 (28%)] Loss: 0.366546 Accuracy: 21075/84843 (25%) Train Epoch: 11 [32000/84843 (38%)] Loss: 0.493192 Accuracy: 28011/84843 (33%) Train Epoch: 11 [40000/84843 (47%)] Loss: 0.459920 Accuracy: 34908/84843 (41%) Train Epoch: 11 [48000/84843 (56%)] Loss: 0.410225 Accuracy: 41867/84843 (49%) Train Epoch: 11 [56000/84843 (66%)] Loss: 0.410714 Accuracy: 48887/84843 (58%) Train Epoch: 11 [64000/84843 (75%)] Loss: 0.552175 Accuracy: 55917/84843 (66%) Train Epoch: 11 [72000/84843 (85%)] Loss: 0.484379 Accuracy: 62819/84843 (74%) Train Epoch: 11 [80000/84843 (94%)] Loss: 0.435677 Accuracy: 69781/84843 (82%) Validation Epoch: 11 Loss: 0.298372 Accuracy: 8258/9981 (83%) Train Epoch: 12 [0/84843 (0%)] Loss: 0.529828 Accuracy: 339/84843 (0%) Train Epoch: 12 [8000/84843 (9%)] Loss: 0.446414 Accuracy: 7275/84843 (9%) Train Epoch: 12 [16000/84843 (19%)] Loss: 0.368270 Accuracy: 14302/84843 (17%) Train Epoch: 12 [24000/84843 (28%)] Loss: 0.484432 Accuracy: 21344/84843 (25%) Train Epoch: 12 [32000/84843 (38%)] Loss: 0.451176 Accuracy: 28268/84843 (33%) Train Epoch: 12 [40000/84843 (47%)] Loss: 0.421888 Accuracy: 35231/84843 (42%) Train Epoch: 12 [48000/84843 (56%)] Loss: 0.380275 Accuracy: 42170/84843 (50%) Train Epoch: 12 [56000/84843 (66%)] Loss: 0.333894 Accuracy: 49204/84843 (58%) Train Epoch: 12 [64000/84843 (75%)] Loss: 0.494247 Accuracy: 56133/84843 (66%) Train Epoch: 12 [72000/84843 (85%)] Loss: 0.325777 Accuracy: 63101/84843 (74%) Train Epoch: 12 [80000/84843 (94%)] Loss: 0.358946 Accuracy: 70107/84843 (83%) Validation Epoch: 12 Loss: 0.918404 Accuracy: 8041/9981 (81%) Train Epoch: 13 [0/84843 (0%)] Loss: 0.381628 Accuracy: 349/84843 (0%) Train Epoch: 13 [8000/84843 (9%)] Loss: 0.433693 Accuracy: 7180/84843 (8%) Train Epoch: 13 [16000/84843 (19%)] Loss: 0.442880 Accuracy: 14055/84843 (17%) Train Epoch: 13 [24000/84843 (28%)] Loss: 0.322268 Accuracy: 21060/84843 (25%) Train Epoch: 13 [32000/84843 (38%)] Loss: 0.462307 Accuracy: 28075/84843 (33%) Train Epoch: 13 [40000/84843 (47%)] Loss: 0.336847 Accuracy: 35060/84843 (41%) Train Epoch: 13 [48000/84843 (56%)] Loss: 0.424574 Accuracy: 42105/84843 (50%) Train Epoch: 13 [56000/84843 (66%)] Loss: 0.320075 Accuracy: 49135/84843 (58%) Train Epoch: 13 [64000/84843 (75%)] Loss: 0.408692 Accuracy: 56110/84843 (66%) Train Epoch: 13 [72000/84843 (85%)] Loss: 0.444735 Accuracy: 63134/84843 (74%) Train Epoch: 13 [80000/84843 (94%)] Loss: 0.351144 Accuracy: 70196/84843 (83%) Validation Epoch: 13 Loss: 0.980500 Accuracy: 7921/9981 (79%) Train Epoch: 14 [0/84843 (0%)] Loss: 0.391851 Accuracy: 346/84843 (0%) Train Epoch: 14 [8000/84843 (9%)] Loss: 0.502600 Accuracy: 7224/84843 (9%) Train Epoch: 14 [16000/84843 (19%)] Loss: 0.414249 Accuracy: 14180/84843 (17%) Train Epoch: 14 [24000/84843 (28%)] Loss: 0.336417 Accuracy: 21195/84843 (25%) Train Epoch: 14 [32000/84843 (38%)] Loss: 0.423969 Accuracy: 28239/84843 (33%) Train Epoch: 14 [40000/84843 (47%)] Loss: 0.399629 Accuracy: 35292/84843 (42%) Train Epoch: 14 [48000/84843 (56%)] Loss: 0.461126 Accuracy: 42245/84843 (50%) Train Epoch: 14 [56000/84843 (66%)] Loss: 0.343521 Accuracy: 49260/84843 (58%) Train Epoch: 14 [64000/84843 (75%)] Loss: 0.482374 Accuracy: 56234/84843 (66%) Train Epoch: 14 [72000/84843 (85%)] Loss: 0.402608 Accuracy: 63236/84843 (75%) Train Epoch: 14 [80000/84843 (94%)] Loss: 0.399460 Accuracy: 70201/84843 (83%) Validation Epoch: 14 Loss: 0.479671 Accuracy: 8293/9981 (83%) Train Epoch: 15 [0/84843 (0%)] Loss: 0.479733 Accuracy: 342/84843 (0%) Train Epoch: 15 [8000/84843 (9%)] Loss: 0.386752 Accuracy: 7189/84843 (8%) Train Epoch: 15 [16000/84843 (19%)] Loss: 0.313140 Accuracy: 14195/84843 (17%) Train Epoch: 15 [24000/84843 (28%)] Loss: 0.413696 Accuracy: 21227/84843 (25%) Train Epoch: 15 [32000/84843 (38%)] Loss: 0.420278 Accuracy: 28266/84843 (33%) Train Epoch: 15 [40000/84843 (47%)] Loss: 0.447525 Accuracy: 35242/84843 (42%) Train Epoch: 15 [48000/84843 (56%)] Loss: 0.434598 Accuracy: 42236/84843 (50%) Train Epoch: 15 [56000/84843 (66%)] Loss: 0.437864 Accuracy: 49315/84843 (58%) Train Epoch: 15 [64000/84843 (75%)] Loss: 0.395675 Accuracy: 56332/84843 (66%) Train Epoch: 15 [72000/84843 (85%)] Loss: 0.408649 Accuracy: 63349/84843 (75%) Train Epoch: 15 [80000/84843 (94%)] Loss: 0.295643 Accuracy: 70368/84843 (83%) Validation Epoch: 15 Loss: 0.651688 Accuracy: 7848/9981 (79%) Train Epoch: 16 [0/84843 (0%)] Loss: 0.375843 Accuracy: 357/84843 (0%) Train Epoch: 16 [8000/84843 (9%)] Loss: 0.387292 Accuracy: 7306/84843 (9%) Train Epoch: 16 [16000/84843 (19%)] Loss: 0.307929 Accuracy: 14342/84843 (17%) Train Epoch: 16 [24000/84843 (28%)] Loss: 0.409047 Accuracy: 21381/84843 (25%) Train Epoch: 16 [32000/84843 (38%)] Loss: 0.344103 Accuracy: 28435/84843 (34%) Train Epoch: 16 [40000/84843 (47%)] Loss: 0.344733 Accuracy: 35452/84843 (42%) Train Epoch: 16 [48000/84843 (56%)] Loss: 0.350362 Accuracy: 42465/84843 (50%) Train Epoch: 16 [56000/84843 (66%)] Loss: 0.309894 Accuracy: 49567/84843 (58%) Train Epoch: 16 [64000/84843 (75%)] Loss: 0.459378 Accuracy: 56595/84843 (67%) Train Epoch: 16 [72000/84843 (85%)] Loss: 0.521180 Accuracy: 63660/84843 (75%) Train Epoch: 16 [80000/84843 (94%)] Loss: 0.355333 Accuracy: 70708/84843 (83%) Validation Epoch: 16 Loss: 0.444694 Accuracy: 8107/9981 (81%) Train Epoch: 17 [0/84843 (0%)] Loss: 0.325685 Accuracy: 356/84843 (0%) Train Epoch: 17 [8000/84843 (9%)] Loss: 0.436689 Accuracy: 7215/84843 (9%) Train Epoch: 17 [16000/84843 (19%)] Loss: 0.371239 Accuracy: 14279/84843 (17%) Train Epoch: 17 [24000/84843 (28%)] Loss: 0.357393 Accuracy: 21340/84843 (25%) Train Epoch: 17 [32000/84843 (38%)] Loss: 0.291569 Accuracy: 28325/84843 (33%) Train Epoch: 17 [40000/84843 (47%)] Loss: 0.430240 Accuracy: 35398/84843 (42%) Train Epoch: 17 [48000/84843 (56%)] Loss: 0.341814 Accuracy: 42510/84843 (50%) Train Epoch: 17 [56000/84843 (66%)] Loss: 0.482547 Accuracy: 49575/84843 (58%) Train Epoch: 17 [64000/84843 (75%)] Loss: 0.289386 Accuracy: 56613/84843 (67%) Train Epoch: 17 [72000/84843 (85%)] Loss: 0.546193 Accuracy: 63654/84843 (75%) Train Epoch: 17 [80000/84843 (94%)] Loss: 0.315254 Accuracy: 70662/84843 (83%) Validation Epoch: 17 Loss: 1.305806 Accuracy: 7335/9981 (73%) Train Epoch: 18 [0/84843 (0%)] Loss: 0.421811 Accuracy: 345/84843 (0%) Train Epoch: 18 [8000/84843 (9%)] Loss: 0.426203 Accuracy: 7322/84843 (9%) Train Epoch: 18 [16000/84843 (19%)] Loss: 0.345160 Accuracy: 14411/84843 (17%) Train Epoch: 18 [24000/84843 (28%)] Loss: 0.386622 Accuracy: 21491/84843 (25%) Train Epoch: 18 [32000/84843 (38%)] Loss: 0.342472 Accuracy: 28586/84843 (34%) Train Epoch: 18 [40000/84843 (47%)] Loss: 0.377946 Accuracy: 35681/84843 (42%) Train Epoch: 18 [48000/84843 (56%)] Loss: 0.428618 Accuracy: 42832/84843 (50%) Train Epoch: 18 [56000/84843 (66%)] Loss: 0.299141 Accuracy: 49871/84843 (59%) Train Epoch: 18 [64000/84843 (75%)] Loss: 0.385187 Accuracy: 56922/84843 (67%) Train Epoch: 18 [72000/84843 (85%)] Loss: 0.474375 Accuracy: 63958/84843 (75%) Train Epoch: 18 [80000/84843 (94%)] Loss: 0.448246 Accuracy: 70983/84843 (84%) Validation Epoch: 18 Loss: 0.226503 Accuracy: 8058/9981 (81%) Train Epoch: 19 [0/84843 (0%)] Loss: 0.324405 Accuracy: 361/84843 (0%) Train Epoch: 19 [8000/84843 (9%)] Loss: 0.360499 Accuracy: 7278/84843 (9%) Train Epoch: 19 [16000/84843 (19%)] Loss: 0.422959 Accuracy: 14381/84843 (17%) Train Epoch: 19 [24000/84843 (28%)] Loss: 0.383157 Accuracy: 21520/84843 (25%) Train Epoch: 19 [32000/84843 (38%)] Loss: 0.401100 Accuracy: 28627/84843 (34%) Train Epoch: 19 [40000/84843 (47%)] Loss: 0.327158 Accuracy: 35717/84843 (42%) Train Epoch: 19 [48000/84843 (56%)] Loss: 0.475709 Accuracy: 42814/84843 (50%) Train Epoch: 19 [56000/84843 (66%)] Loss: 0.447375 Accuracy: 49851/84843 (59%) Train Epoch: 19 [64000/84843 (75%)] Loss: 0.363308 Accuracy: 56887/84843 (67%) Train Epoch: 19 [72000/84843 (85%)] Loss: 0.364523 Accuracy: 63952/84843 (75%) Train Epoch: 19 [80000/84843 (94%)] Loss: 0.364479 Accuracy: 71015/84843 (84%) Validation Epoch: 19 Loss: 1.340748 Accuracy: 7459/9981 (75%) Train Epoch: 20 [0/84843 (0%)] Loss: 0.394114 Accuracy: 354/84843 (0%) Train Epoch: 20 [8000/84843 (9%)] Loss: 0.403494 Accuracy: 7254/84843 (9%) Train Epoch: 20 [16000/84843 (19%)] Loss: 0.414734 Accuracy: 14258/84843 (17%) Train Epoch: 20 [24000/84843 (28%)] Loss: 0.331005 Accuracy: 21352/84843 (25%) Train Epoch: 20 [32000/84843 (38%)] Loss: 0.368372 Accuracy: 28438/84843 (34%) Train Epoch: 20 [40000/84843 (47%)] Loss: 0.322633 Accuracy: 35503/84843 (42%) Train Epoch: 20 [48000/84843 (56%)] Loss: 0.333580 Accuracy: 42586/84843 (50%) Train Epoch: 20 [56000/84843 (66%)] Loss: 0.321315 Accuracy: 49637/84843 (59%) Train Epoch: 20 [64000/84843 (75%)] Loss: 0.336883 Accuracy: 56723/84843 (67%) Train Epoch: 20 [72000/84843 (85%)] Loss: 0.371565 Accuracy: 63750/84843 (75%) Train Epoch: 20 [80000/84843 (94%)] Loss: 0.365199 Accuracy: 70807/84843 (83%) Validation Epoch: 20 Loss: 1.113444 Accuracy: 8112/9981 (81%) Train Epoch: 21 [0/84843 (0%)] Loss: 0.296259 Accuracy: 362/84843 (0%) Train Epoch: 21 [8000/84843 (9%)] Loss: 0.293128 Accuracy: 7563/84843 (9%) Train Epoch: 21 [16000/84843 (19%)] Loss: 0.318249 Accuracy: 14882/84843 (18%) Train Epoch: 21 [24000/84843 (28%)] Loss: 0.296667 Accuracy: 22241/84843 (26%) Train Epoch: 21 [32000/84843 (38%)] Loss: 0.252305 Accuracy: 29617/84843 (35%) Train Epoch: 21 [40000/84843 (47%)] Loss: 0.221819 Accuracy: 37018/84843 (44%) Train Epoch: 21 [48000/84843 (56%)] Loss: 0.151378 Accuracy: 44443/84843 (52%) Train Epoch: 21 [56000/84843 (66%)] Loss: 0.209818 Accuracy: 51880/84843 (61%) Train Epoch: 21 [64000/84843 (75%)] Loss: 0.191403 Accuracy: 59311/84843 (70%) Train Epoch: 21 [72000/84843 (85%)] Loss: 0.217283 Accuracy: 66697/84843 (79%) Train Epoch: 21 [80000/84843 (94%)] Loss: 0.215745 Accuracy: 74127/84843 (87%) Validation Epoch: 21 Loss: 0.162499 Accuracy: 9180/9981 (92%) Train Epoch: 22 [0/84843 (0%)] Loss: 0.219118 Accuracy: 371/84843 (0%) Train Epoch: 22 [8000/84843 (9%)] Loss: 0.200925 Accuracy: 7868/84843 (9%) Train Epoch: 22 [16000/84843 (19%)] Loss: 0.215222 Accuracy: 15330/84843 (18%) Train Epoch: 22 [24000/84843 (28%)] Loss: 0.230711 Accuracy: 22787/84843 (27%) Train Epoch: 22 [32000/84843 (38%)] Loss: 0.170707 Accuracy: 30250/84843 (36%) Train Epoch: 22 [40000/84843 (47%)] Loss: 0.216858 Accuracy: 37686/84843 (44%) Train Epoch: 22 [48000/84843 (56%)] Loss: 0.296102 Accuracy: 45113/84843 (53%) Train Epoch: 22 [56000/84843 (66%)] Loss: 0.224326 Accuracy: 52580/84843 (62%) Train Epoch: 22 [64000/84843 (75%)] Loss: 0.190691 Accuracy: 60055/84843 (71%) Train Epoch: 22 [72000/84843 (85%)] Loss: 0.212582 Accuracy: 67543/84843 (80%) Train Epoch: 22 [80000/84843 (94%)] Loss: 0.214685 Accuracy: 75008/84843 (88%) Validation Epoch: 22 Loss: 0.177364 Accuracy: 9167/9981 (92%) Train Epoch: 23 [0/84843 (0%)] Loss: 0.228292 Accuracy: 370/84843 (0%) Train Epoch: 23 [8000/84843 (9%)] Loss: 0.203103 Accuracy: 7867/84843 (9%) Train Epoch: 23 [16000/84843 (19%)] Loss: 0.210349 Accuracy: 15402/84843 (18%) Train Epoch: 23 [24000/84843 (28%)] Loss: 0.128483 Accuracy: 22938/84843 (27%) Train Epoch: 23 [32000/84843 (38%)] Loss: 0.226514 Accuracy: 30366/84843 (36%) Train Epoch: 23 [40000/84843 (47%)] Loss: 0.180384 Accuracy: 37874/84843 (45%) Train Epoch: 23 [48000/84843 (56%)] Loss: 0.153209 Accuracy: 45388/84843 (53%) Train Epoch: 23 [56000/84843 (66%)] Loss: 0.228650 Accuracy: 52906/84843 (62%) Train Epoch: 23 [64000/84843 (75%)] Loss: 0.215307 Accuracy: 60409/84843 (71%) Train Epoch: 23 [72000/84843 (85%)] Loss: 0.207441 Accuracy: 67920/84843 (80%) Train Epoch: 23 [80000/84843 (94%)] Loss: 0.204934 Accuracy: 75460/84843 (89%) Validation Epoch: 23 Loss: 0.196643 Accuracy: 9154/9981 (92%) Train Epoch: 24 [0/84843 (0%)] Loss: 0.116506 Accuracy: 388/84843 (0%) Train Epoch: 24 [8000/84843 (9%)] Loss: 0.205299 Accuracy: 7908/84843 (9%) Train Epoch: 24 [16000/84843 (19%)] Loss: 0.255465 Accuracy: 15448/84843 (18%) Train Epoch: 24 [24000/84843 (28%)] Loss: 0.200980 Accuracy: 22965/84843 (27%) Train Epoch: 24 [32000/84843 (38%)] Loss: 0.198600 Accuracy: 30503/84843 (36%) Train Epoch: 24 [40000/84843 (47%)] Loss: 0.168644 Accuracy: 38021/84843 (45%) Train Epoch: 24 [48000/84843 (56%)] Loss: 0.230562 Accuracy: 45511/84843 (54%) Train Epoch: 24 [56000/84843 (66%)] Loss: 0.121427 Accuracy: 53052/84843 (63%) Train Epoch: 24 [64000/84843 (75%)] Loss: 0.201823 Accuracy: 60595/84843 (71%) Train Epoch: 24 [72000/84843 (85%)] Loss: 0.189215 Accuracy: 68114/84843 (80%) Train Epoch: 24 [80000/84843 (94%)] Loss: 0.201265 Accuracy: 75639/84843 (89%) Validation Epoch: 24 Loss: 0.182494 Accuracy: 9215/9981 (92%) Train Epoch: 25 [0/84843 (0%)] Loss: 0.141202 Accuracy: 382/84843 (0%) Train Epoch: 25 [8000/84843 (9%)] Loss: 0.221016 Accuracy: 7907/84843 (9%) Train Epoch: 25 [16000/84843 (19%)] Loss: 0.232705 Accuracy: 15457/84843 (18%) Train Epoch: 25 [24000/84843 (28%)] Loss: 0.208431 Accuracy: 23043/84843 (27%) Train Epoch: 25 [32000/84843 (38%)] Loss: 0.143560 Accuracy: 30598/84843 (36%) Train Epoch: 25 [40000/84843 (47%)] Loss: 0.167081 Accuracy: 38156/84843 (45%) Train Epoch: 25 [48000/84843 (56%)] Loss: 0.137688 Accuracy: 45702/84843 (54%) Train Epoch: 25 [56000/84843 (66%)] Loss: 0.167855 Accuracy: 53258/84843 (63%) Train Epoch: 25 [64000/84843 (75%)] Loss: 0.193263 Accuracy: 60805/84843 (72%) Train Epoch: 25 [72000/84843 (85%)] Loss: 0.165139 Accuracy: 68377/84843 (81%) Train Epoch: 25 [80000/84843 (94%)] Loss: 0.238300 Accuracy: 75908/84843 (89%) Validation Epoch: 25 Loss: 0.182906 Accuracy: 9186/9981 (92%) Train Epoch: 26 [0/84843 (0%)] Loss: 0.154624 Accuracy: 382/84843 (0%) Train Epoch: 26 [8000/84843 (9%)] Loss: 0.201173 Accuracy: 7951/84843 (9%) Train Epoch: 26 [16000/84843 (19%)] Loss: 0.153003 Accuracy: 15499/84843 (18%) Train Epoch: 26 [24000/84843 (28%)] Loss: 0.163543 Accuracy: 23063/84843 (27%) Train Epoch: 26 [32000/84843 (38%)] Loss: 0.180528 Accuracy: 30660/84843 (36%) Train Epoch: 26 [40000/84843 (47%)] Loss: 0.171404 Accuracy: 38227/84843 (45%) Train Epoch: 26 [48000/84843 (56%)] Loss: 0.182881 Accuracy: 45809/84843 (54%) Train Epoch: 26 [56000/84843 (66%)] Loss: 0.146421 Accuracy: 53373/84843 (63%) Train Epoch: 26 [64000/84843 (75%)] Loss: 0.126890 Accuracy: 60960/84843 (72%) Train Epoch: 26 [72000/84843 (85%)] Loss: 0.244570 Accuracy: 68515/84843 (81%) Train Epoch: 26 [80000/84843 (94%)] Loss: 0.225684 Accuracy: 76054/84843 (90%) Validation Epoch: 26 Loss: 0.232998 Accuracy: 9173/9981 (92%) Train Epoch: 27 [0/84843 (0%)] Loss: 0.151793 Accuracy: 386/84843 (0%) Train Epoch: 27 [8000/84843 (9%)] Loss: 0.157708 Accuracy: 7991/84843 (9%) Train Epoch: 27 [16000/84843 (19%)] Loss: 0.192628 Accuracy: 15580/84843 (18%) Train Epoch: 27 [24000/84843 (28%)] Loss: 0.234515 Accuracy: 23189/84843 (27%) Train Epoch: 27 [32000/84843 (38%)] Loss: 0.204080 Accuracy: 30772/84843 (36%) Train Epoch: 27 [40000/84843 (47%)] Loss: 0.197971 Accuracy: 38363/84843 (45%) Train Epoch: 27 [48000/84843 (56%)] Loss: 0.186107 Accuracy: 45937/84843 (54%) Train Epoch: 27 [56000/84843 (66%)] Loss: 0.216745 Accuracy: 53518/84843 (63%) Train Epoch: 27 [64000/84843 (75%)] Loss: 0.182022 Accuracy: 61092/84843 (72%) Train Epoch: 27 [72000/84843 (85%)] Loss: 0.167147 Accuracy: 68676/84843 (81%) Train Epoch: 27 [80000/84843 (94%)] Loss: 0.147802 Accuracy: 76256/84843 (90%) Validation Epoch: 27 Loss: 0.239390 Accuracy: 9109/9981 (91%) Train Epoch: 28 [0/84843 (0%)] Loss: 0.152747 Accuracy: 382/84843 (0%) Train Epoch: 28 [8000/84843 (9%)] Loss: 0.161108 Accuracy: 7941/84843 (9%) Train Epoch: 28 [16000/84843 (19%)] Loss: 0.181950 Accuracy: 15566/84843 (18%) Train Epoch: 28 [24000/84843 (28%)] Loss: 0.161549 Accuracy: 23168/84843 (27%) Train Epoch: 28 [32000/84843 (38%)] Loss: 0.186981 Accuracy: 30751/84843 (36%) Train Epoch: 28 [40000/84843 (47%)] Loss: 0.185502 Accuracy: 38317/84843 (45%) Train Epoch: 28 [48000/84843 (56%)] Loss: 0.138564 Accuracy: 45922/84843 (54%) Train Epoch: 28 [56000/84843 (66%)] Loss: 0.165149 Accuracy: 53527/84843 (63%) Train Epoch: 28 [64000/84843 (75%)] Loss: 0.123159 Accuracy: 61117/84843 (72%) Train Epoch: 28 [72000/84843 (85%)] Loss: 0.168681 Accuracy: 68730/84843 (81%) Train Epoch: 28 [80000/84843 (94%)] Loss: 0.217952 Accuracy: 76304/84843 (90%) Validation Epoch: 28 Loss: 0.179299 Accuracy: 9163/9981 (92%) Train Epoch: 29 [0/84843 (0%)] Loss: 0.151821 Accuracy: 382/84843 (0%) Train Epoch: 29 [8000/84843 (9%)] Loss: 0.134071 Accuracy: 8028/84843 (9%) Train Epoch: 29 [16000/84843 (19%)] Loss: 0.125948 Accuracy: 15654/84843 (18%) Train Epoch: 29 [24000/84843 (28%)] Loss: 0.141070 Accuracy: 23303/84843 (27%) Train Epoch: 29 [32000/84843 (38%)] Loss: 0.157296 Accuracy: 30946/84843 (36%) Train Epoch: 29 [40000/84843 (47%)] Loss: 0.120730 Accuracy: 38549/84843 (45%) Train Epoch: 29 [48000/84843 (56%)] Loss: 0.101001 Accuracy: 46178/84843 (54%) Train Epoch: 29 [56000/84843 (66%)] Loss: 0.133927 Accuracy: 53807/84843 (63%) Train Epoch: 29 [64000/84843 (75%)] Loss: 0.142309 Accuracy: 61419/84843 (72%) Train Epoch: 29 [72000/84843 (85%)] Loss: 0.147148 Accuracy: 69023/84843 (81%) Train Epoch: 29 [80000/84843 (94%)] Loss: 0.172224 Accuracy: 76634/84843 (90%) Validation Epoch: 29 Loss: 0.197929 Accuracy: 9160/9981 (92%) Train Epoch: 30 [0/84843 (0%)] Loss: 0.160155 Accuracy: 382/84843 (0%) Train Epoch: 30 [8000/84843 (9%)] Loss: 0.146723 Accuracy: 8002/84843 (9%) Train Epoch: 30 [16000/84843 (19%)] Loss: 0.089232 Accuracy: 15655/84843 (18%) Train Epoch: 30 [24000/84843 (28%)] Loss: 0.139135 Accuracy: 23297/84843 (27%) Train Epoch: 30 [32000/84843 (38%)] Loss: 0.130675 Accuracy: 30920/84843 (36%) Train Epoch: 30 [40000/84843 (47%)] Loss: 0.131904 Accuracy: 38560/84843 (45%) Train Epoch: 30 [48000/84843 (56%)] Loss: 0.153063 Accuracy: 46205/84843 (54%) Train Epoch: 30 [56000/84843 (66%)] Loss: 0.134736 Accuracy: 53844/84843 (63%) Train Epoch: 30 [64000/84843 (75%)] Loss: 0.126240 Accuracy: 61453/84843 (72%) Train Epoch: 30 [72000/84843 (85%)] Loss: 0.177946 Accuracy: 69090/84843 (81%) Train Epoch: 30 [80000/84843 (94%)] Loss: 0.135125 Accuracy: 76698/84843 (90%) Validation Epoch: 30 Loss: 0.132336 Accuracy: 9158/9981 (92%)
In [80]:
# Plot training loss
plt.plot(losses_train, 'b', label='Train loss')
plt.legend(loc="upper left")
plt.title("M18 model training losses over all iterations")
plt.show()

# Plot validation loss
plt.plot(losses_validation, 'r', label='Valid loss')
plt.legend(loc="upper left")
plt.title("M18 model validation losses over validation epochs")
plt.show()

# Plot accuracy
plt.plot(accuracy_train, 'b', label='Train acc')
plt.plot(accuracy_validation,'r', label ='Valid acc')
plt.legend(loc="upper left")
plt.title("M18 model training & validation accuracy over epochs")
plt.show()
Notebook Image
Notebook Image
Notebook Image
In [74]:
M18_test_acc_result = test(model_M18)
print(M18_test_acc_result)
Test set accuracy: 9937/11005 (90%) 90.29532030895048
In [75]:
jovian.log_metrics({ 'M18_test_acc_result': M18_test_acc_result })
[jovian] Metrics logged.

Conclusions

This notebook implemented speech command recognition using convolutional neural networks trained on the Google SpeechCommand dataset.

M3, M5, M11 and M18 layer architecture convolutional neural networks were implemented and the networks were trained on the time domain waveform inputs of the SpeechCommand dataset.

The dataset consists of 105 000 .wav files that was split into a training (84843 files), validation (9981 files) and testing (11005 files) datasets.

The files were pre-processed by resampling each to 8kHz which results in smaller files to enable faster processing.

An Adam optimiser was used, with weight decay set to 0.0001. Initial learning rate was set at 0.01, and a scheduler was used to decrease learning reate to 0.001 during training after 20 epochs

With the network being trained for 30 epochs in this notebook, the M3-M11 networks became increasingly accurate, and the M18 network showed the same accuracy as the M5 network.

Networks were trained with increased 50 and 100 epochs in other notebooks with incremental improvements. The results and graphs are shown below and in Appendixes.

  • 30 Epoch network accuracy: M3 - 65%, M5 - 91%, M11 - 93%, M18 - 91%
  • 50 Epoch network accuracy: M3 - 67%, M5 - 91%, M11 - 93%, M18 - 91%
  • 100 Epoch network accuracy: M3 - 68%, M5 - 91%, M11 - 92%, M18 - 92%
In [76]:
print('Number of epochs:', epoch_count)
print(f"\nM3 model accuracy: {(M3_test_acc_result):.0f}%")
print(f"M5 model accuracy: {(M5_test_acc_result):.0f}%")
print(f"M11 model accuracy: {(M11_test_acc_result):.0f}%")
print(f"M18 model accuracy: {(M18_test_acc_result):.0f}%")
Number of epochs: 30 M3 model accuracy: 66% M5 model accuracy: 91% M11 model accuracy: 93% M18 model accuracy: 90%

Future work

The challenges of different datasets recorded with different microphones/sample rates/files formats/speech lengths can be explored.

Preprocessing and transforming the audio into MFCC or mel-spectrogram arrays can be performed to investigate speed of training differences, alternative architectures and recognition accuracy.

Audio data augmentation techniques (such as adding noise) to improve recognition robustness can be investigated.

Alternative architectures such as LSTM or RNN networks that encode memory can be investigated as the likehood of a current phoneme being pronounced is affected by previous phonemes that were present in the word structure.

References

[1] Warden, Pete. (2018). Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition.

[2] Dai, Wei & Dai, Chia & Qu, Shuhui & Li, Juncheng & Das, Samarjit. (2016). Very Deep Convolutional Neural Networks for Raw Waveforms. https://arxiv.org/pdf/1610.00087.pdf

[3] https://pytorch.org/tutorials/intermediate/speech_command_recognition_with_torchaudio.html

[4] https://pytorch.org/docs/stable/index.html

[5] https://pytorch.org/audio/stable/datasets.html

Appendix 1 - 50 Epoch training result graphs

M3 model training losses over all iterations - 50 epochs

M3 model training losses over all iterations-50epochs.png

M3 model validation losses over validation epochs - 50 epochs

M3 model validation losses over validation epochs-50epochs.png

M3 model training & validation accuracy over epochs - 50 epochs

M3 model training _ validation accuracy over epochs-50epochs.png

M5 model training losses over all iterations - 50 epochs

M5 model training losses over all iterations-50epeochs.png

M5 model validation losses over validation epochs - 50epochs

M5 model validation losses over validation epochs-50epochs.png

M5 model training & validation accuracy over epochs - 50 epochs

M5 model training _ validation accuracy over epochs-50epochs.png

M11 model training losses over all iterations - 50epochs

M11 model training losses over all iterations-50epochs.png

M11 model validation losses over validation epochs - 50 epochs

M11 model validation losses over validation epochs-50epochs.png

M11 model training & validation accuracy over epochs - 50 epochs

M11 model training _ validation accuracy over epochs-50epochs.png

M18 model training losses over all iterations - 50 epochs

M18 model training losses over all iterations-50epochs.png

M18 model validation losses over validation epochs - 50 epochs

M18 model validation losses over validation epochs-50epochs.png

M18 model training & validation accuracy over epochs - 50 epochs

M18 model training _ validation accuracy over epochs-50epochs.png

Appendix B - 100 Epoch training result graphs

M3 model training losses over all iterations - 100 epochs

M3 model training losses over all iterations - 100 epochs.png

M3 model validation losses over validation epochs - 100 epochs

M3 model validation losses over validation epochs - 100 epochs.png

M3 model training & validation accuracy over epochs - 100 epochs