Jovian
⭐️
Sign In
In [1]:
import jovian
In [ ]:
!pip install jovian --user

Assigment Definition:

In this assignement, we will compare different Deep Convolutional Neural Networks in terms of:

  • "number of parameters"
  • "inference time"
  • "performance".

You will be required to construct the following networks:

  1. AlexNet
  2. VGG-16
  3. Your custom Deep CNN (CNN_5x5_Net): use only 5x5 convolutional kernels (hint: in VGG-16, all convolutional kernels were 3x3)
  • Check the number of parameters vs inference time (by generating random noise image and feed-forward through the networks)

  • Explain the results: why one was faster than the another, what was the key point?

  • Add "Batch Normalization" and/or "Dropout" layers to the networks (AlexNet, VGG-16, Your custom CNN_5x5_Net)

  • Check how does inference time is changing after adding those layers Note: Framework: we will be using Keras in this assignment.

In [2]:
from datetime import datetime
import tensorflow.keras as keras
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Activation, Dropout, Flatten, Conv2D, MaxPooling2D
import numpy as np


def alexnet(input_data_shape=(224, 224, 3, ), number_of_classes=10):
    model = Sequential()

    # 1st Convolutional Layer
    model.add(Conv2D(filters=96, input_shape=input_data_shape, kernel_size=(11, 11), strides=(4, 4), padding='valid', activation='relu'))
    # Max Pooling
    model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2), padding='valid'))

    # 2nd Convolutional Layer
    model.add(Conv2D(filters=256, kernel_size=(11, 11), strides=(1, 1), padding='valid', activation='relu'))
    # Max Pooling
    model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2), padding='valid'))

    # 3rd Convolutional Layer
    model.add(Conv2D(filters=384, kernel_size=(3, 3), strides=(1, 1), padding='valid', activation='relu'))

    # 4th Convolutional Layer
    model.add(Conv2D(filters=384, kernel_size=(3, 3), strides=(1, 1), padding='valid', activation='relu'))

    # 5th Convolutional Layer
    model.add(Conv2D(filters=256, kernel_size=(3, 3), strides=(1, 1), padding='valid', activation='relu'))
    # Max Pooling
    model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2), padding='valid'))

    # Flatten the feature maps to pass them to Fully Connected Layers
    model.add(Flatten())

    # Fully Connected Layers
    model.add(Dense(4096, activation='relu'))
    model.add(Dense(4096, activation='relu'))
    model.add(Dense(number_of_classes, activation='softmax'))

    model.summary()
    return model
In [3]:
def vgg_16(input_data_shape=(224, 224, 3,), number_of_classes=10):
    model = Sequential()
    # Block 1
    model.add(Conv2D(filters=64, input_shape=input_data_shape, kernel_size=(3, 3), padding='same', activation='relu'))
    model.add(Conv2D(filters=64, input_shape=input_data_shape, kernel_size=(3, 3), padding='same', activation='relu'))
    model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))

    # Block 2
    model.add(Conv2D(filters=128, input_shape=input_data_shape, kernel_size=(3, 3), padding='same', activation='relu'))
    model.add(Conv2D(filters=128, input_shape=input_data_shape, kernel_size=(3, 3), padding='same', activation='relu'))
    model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))

    # Block 3
    model.add(Conv2D(filters=256, input_shape=input_data_shape, kernel_size=(3, 3), padding='same', activation='relu'))
    model.add(Conv2D(filters=256, input_shape=input_data_shape, kernel_size=(3, 3), padding='same', activation='relu'))
    model.add(Conv2D(filters=256, input_shape=input_data_shape, kernel_size=(3, 3), padding='same', activation='relu'))
    model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))

    # Block 4
    model.add(Conv2D(filters=512, input_shape=input_data_shape, kernel_size=(3, 3), padding='same', activation='relu'))
    model.add(Conv2D(filters=512, input_shape=input_data_shape, kernel_size=(3, 3), padding='same', activation='relu'))
    model.add(Conv2D(filters=512, input_shape=input_data_shape, kernel_size=(3, 3), padding='same', activation='relu'))
    model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))

    # Block 5
    model.add(Conv2D(filters=512, input_shape=input_data_shape, kernel_size=(3, 3), padding='same', activation='relu'))
    model.add(Conv2D(filters=512, input_shape=input_data_shape, kernel_size=(3, 3), padding='same', activation='relu'))
    model.add(Conv2D(filters=512, input_shape=input_data_shape, kernel_size=(3, 3), padding='same', activation='relu'))
    model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))

    # Flatten the feature maps to pass them to Fully Connected Layers
    model.add(Flatten())

    # fully connected layers
    model.add(Dense(4096, activation='relu'))
    model.add(Dense(4096, activation='relu'))
    model.add(Dense(number_of_classes, activation='softmax'))

    # Create model.
    model.summary()

    return model
In [4]:
def cnn_5x5_net(input_data_shape=(224, 224, 3), number_of_classes=10):
    model = Sequential()
    # Block 1
    model.add(Conv2D(filters=64, input_shape=input_data_shape, kernel_size=(5, 5), padding='same', activation='relu'))
    model.add(Conv2D(filters=64, input_shape=input_data_shape, kernel_size=(5, 5), padding='same', activation='relu'))
    model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))

    # Block 2
    model.add(Conv2D(filters=128, input_shape=input_data_shape, kernel_size=(5, 5), padding='same', activation='relu'))
    model.add(Conv2D(filters=128, input_shape=input_data_shape, kernel_size=(5, 5), padding='same', activation='relu'))
    model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))

    # Block 3
    model.add(Conv2D(filters=256, input_shape=input_data_shape, kernel_size=(5, 5), padding='same', activation='relu'))
    model.add(Conv2D(filters=256, input_shape=input_data_shape, kernel_size=(5, 5), padding='same', activation='relu'))
    model.add(Conv2D(filters=256, input_shape=input_data_shape, kernel_size=(5, 5), padding='same', activation='relu'))
    model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))

    # Block 4
    model.add(Conv2D(filters=512, input_shape=input_data_shape, kernel_size=(5, 5), padding='same', activation='relu'))
    model.add(Conv2D(filters=512, input_shape=input_data_shape, kernel_size=(5, 5), padding='same', activation='relu'))
    model.add(Conv2D(filters=512, input_shape=input_data_shape, kernel_size=(5, 5), padding='same', activation='relu'))
    model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))

    # Block 5
    model.add(Conv2D(filters=512, input_shape=input_data_shape, kernel_size=(5, 5), padding='same', activation='relu'))
    model.add(Conv2D(filters=512, input_shape=input_data_shape, kernel_size=(5, 5), padding='same', activation='relu'))
    model.add(Conv2D(filters=512, input_shape=input_data_shape, kernel_size=(5, 5), padding='same', activation='relu'))
    model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))

    # Flatten the feature maps to pass them to Fully Connected Layers
    model.add(Flatten())

    # fully connected layers
    model.add(Dense(4096, activation='relu'))
    model.add(Dense(4096, activation='relu'))
    model.add(Dense(number_of_classes, activation='softmax'))

    # Create model.
    model.summary()

    return model
    
    

Set the inference hyper-parameters:

In [5]:
batch_size = 128
num_classes = 10
num_of_training_iteration = 100

Set the input image shape/size

In [6]:
input_data_shape = (224, 224, 3)

Build AlexNet & VGG-16 & CNN_5x5_Net networks (default input image shape is 224x224x3)

In [7]:
alexnet_model = alexnet(input_data_shape=input_data_shape)
vgg16_model = vgg_16(input_data_shape=input_data_shape)
cnn_5x5_net_model = cnn_5x5_net(input_data_shape=input_data_shape)
WARNING:tensorflow:From /home/eldar/.conda/envs/dltokio/lib/python3.7/site-packages/tensorflow/python/ops/resource_variable_ops.py:435: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version. Instructions for updating: Colocations handled automatically by placer. _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= conv2d (Conv2D) (None, 54, 54, 96) 34944 _________________________________________________________________ max_pooling2d (MaxPooling2D) (None, 27, 27, 96) 0 _________________________________________________________________ conv2d_1 (Conv2D) (None, 17, 17, 256) 2973952 _________________________________________________________________ max_pooling2d_1 (MaxPooling2 (None, 8, 8, 256) 0 _________________________________________________________________ conv2d_2 (Conv2D) (None, 6, 6, 384) 885120 _________________________________________________________________ conv2d_3 (Conv2D) (None, 4, 4, 384) 1327488 _________________________________________________________________ conv2d_4 (Conv2D) (None, 2, 2, 256) 884992 _________________________________________________________________ max_pooling2d_2 (MaxPooling2 (None, 1, 1, 256) 0 _________________________________________________________________ flatten (Flatten) (None, 256) 0 _________________________________________________________________ dense (Dense) (None, 4096) 1052672 _________________________________________________________________ dense_1 (Dense) (None, 4096) 16781312 _________________________________________________________________ dense_2 (Dense) (None, 10) 40970 ================================================================= Total params: 23,981,450 Trainable params: 23,981,450 Non-trainable params: 0 _________________________________________________________________ _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= conv2d_5 (Conv2D) (None, 224, 224, 64) 1792 _________________________________________________________________ conv2d_6 (Conv2D) (None, 224, 224, 64) 36928 _________________________________________________________________ max_pooling2d_3 (MaxPooling2 (None, 112, 112, 64) 0 _________________________________________________________________ conv2d_7 (Conv2D) (None, 112, 112, 128) 73856 _________________________________________________________________ conv2d_8 (Conv2D) (None, 112, 112, 128) 147584 _________________________________________________________________ max_pooling2d_4 (MaxPooling2 (None, 56, 56, 128) 0 _________________________________________________________________ conv2d_9 (Conv2D) (None, 56, 56, 256) 295168 _________________________________________________________________ conv2d_10 (Conv2D) (None, 56, 56, 256) 590080 _________________________________________________________________ conv2d_11 (Conv2D) (None, 56, 56, 256) 590080 _________________________________________________________________ max_pooling2d_5 (MaxPooling2 (None, 28, 28, 256) 0 _________________________________________________________________ conv2d_12 (Conv2D) (None, 28, 28, 512) 1180160 _________________________________________________________________ conv2d_13 (Conv2D) (None, 28, 28, 512) 2359808 _________________________________________________________________ conv2d_14 (Conv2D) (None, 28, 28, 512) 2359808 _________________________________________________________________ max_pooling2d_6 (MaxPooling2 (None, 14, 14, 512) 0 _________________________________________________________________ conv2d_15 (Conv2D) (None, 14, 14, 512) 2359808 _________________________________________________________________ conv2d_16 (Conv2D) (None, 14, 14, 512) 2359808 _________________________________________________________________ conv2d_17 (Conv2D) (None, 14, 14, 512) 2359808 _________________________________________________________________ max_pooling2d_7 (MaxPooling2 (None, 7, 7, 512) 0 _________________________________________________________________ flatten_1 (Flatten) (None, 25088) 0 _________________________________________________________________ dense_3 (Dense) (None, 4096) 102764544 _________________________________________________________________ dense_4 (Dense) (None, 4096) 16781312 _________________________________________________________________ dense_5 (Dense) (None, 10) 40970 ================================================================= Total params: 134,301,514 Trainable params: 134,301,514 Non-trainable params: 0 _________________________________________________________________ _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= conv2d_18 (Conv2D) (None, 224, 224, 64) 4864 _________________________________________________________________ conv2d_19 (Conv2D) (None, 224, 224, 64) 102464 _________________________________________________________________ max_pooling2d_8 (MaxPooling2 (None, 112, 112, 64) 0 _________________________________________________________________ conv2d_20 (Conv2D) (None, 112, 112, 128) 204928 _________________________________________________________________ conv2d_21 (Conv2D) (None, 112, 112, 128) 409728 _________________________________________________________________ max_pooling2d_9 (MaxPooling2 (None, 56, 56, 128) 0 _________________________________________________________________ conv2d_22 (Conv2D) (None, 56, 56, 256) 819456 _________________________________________________________________ conv2d_23 (Conv2D) (None, 56, 56, 256) 1638656 _________________________________________________________________ conv2d_24 (Conv2D) (None, 56, 56, 256) 1638656 _________________________________________________________________ max_pooling2d_10 (MaxPooling (None, 28, 28, 256) 0 _________________________________________________________________ conv2d_25 (Conv2D) (None, 28, 28, 512) 3277312 _________________________________________________________________ conv2d_26 (Conv2D) (None, 28, 28, 512) 6554112 _________________________________________________________________ conv2d_27 (Conv2D) (None, 28, 28, 512) 6554112 _________________________________________________________________ max_pooling2d_11 (MaxPooling (None, 14, 14, 512) 0 _________________________________________________________________ conv2d_28 (Conv2D) (None, 14, 14, 512) 6554112 _________________________________________________________________ conv2d_29 (Conv2D) (None, 14, 14, 512) 6554112 _________________________________________________________________ conv2d_30 (Conv2D) (None, 14, 14, 512) 6554112 _________________________________________________________________ max_pooling2d_12 (MaxPooling (None, 7, 7, 512) 0 _________________________________________________________________ flatten_2 (Flatten) (None, 25088) 0 _________________________________________________________________ dense_6 (Dense) (None, 4096) 102764544 _________________________________________________________________ dense_7 (Dense) (None, 4096) 16781312 _________________________________________________________________ dense_8 (Dense) (None, 10) 40970 ================================================================= Total params: 160,453,450 Trainable params: 160,453,450 Non-trainable params: 0 _________________________________________________________________

Train the networks on MNIST dataset:

In [8]:
alexnet_model.compile(loss=keras.losses.categorical_crossentropy, optimizer='adam', metrics=["accuracy"])
vgg16_model.compile(loss=keras.losses.categorical_crossentropy, optimizer='adam', metrics=["accuracy"])
cnn_5x5_net_model.compile(loss=keras.losses.categorical_crossentropy, optimizer='adam', metrics=["accuracy"])

alexnet_inference_time = []
vgg16_inference_time = []
cnn_5x5_net_inference_time = []

# dummy tensor to check the inference time of each network
x_test = np.random.rand(batch_size, input_data_shape[0], input_data_shape[1], input_data_shape[2])

for _ in range(num_of_training_iteration):
    alexnet_inference_start = datetime.now()
    alexnet_inference = alexnet_model.predict_classes(x_test)
    alexnet_inference_finish = datetime.now()
    alexnet_inference_time.append(alexnet_inference_finish - alexnet_inference_start)

for _ in range(num_of_training_iteration):
    vgg16_inference_start = datetime.now()
    vgg_inference = vgg16_model.predict_classes(x_test)
    vgg16_inference_finish = datetime.now()
    vgg16_inference_time.append(vgg16_inference_finish - vgg16_inference_start)
    
for _ in range(num_of_training_iteration):
    cnn_5x5_net_inference_start = datetime.now()
    cnn_5x5_net_inference = cnn_5x5_net_model.predict_classes(x_test)
    cnn_5x5_net_inference_finish = datetime.now()
    cnn_5x5_net_inference_time.append(cnn_5x5_net_inference_finish - cnn_5x5_net_inference_start)


print("Average Inference time for AlexNet: {}".format(np.mean(alexnet_inference_time)))
print("Average Inference time for VGG-16: {}".format(np.mean(vgg16_inference_time)))
print("Average Inference time for CNN_5x5_NET {}".format(np.mean(cnn_5x5_net_inference_time)))
Average Inference time for AlexNet: 0:00:00.157790 Average Inference time for VGG-16: 0:00:00.932858 Average Inference time for CNN_5x5_NET 0:00:01.201763

Clear Session

In [1]:
import tensorflow as tf
tf.keras.backend.clear_session()

Import libraries

In [2]:
from datetime import datetime
import tensorflow.keras as keras
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Activation, Dropout, Flatten, Conv2D, MaxPooling2D
import numpy as np

Import BatchNormalization and Dropout:

In [3]:
from tensorflow.keras.layers import BatchNormalization, Dropout, Activation
dropout_rate = 0.25

AlexNet with BatchNormalization and Dropout:

In [4]:
def alexnet_bd(input_data_shape=(224, 224, 3, ), number_of_classes=10):
    model = Sequential()

    # 1st Convolutional Layer
    model.add(Conv2D(filters=96, input_shape=input_data_shape, kernel_size=(11, 11), strides=(4, 4), padding='valid'))
    # Batch Normalization
    model.add(BatchNormalization())
    # Adding non-linearity
    model.add(Activation('relu'))
    # Max Pooling
    model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2), padding='valid'))
    # Dropout
    model.add(Dropout(dropout_rate))

    # 2nd Convolutional Layer
    model.add(Conv2D(filters=256, kernel_size=(11, 11), strides=(1, 1), padding='valid'))
    # Batch Normalization
    model.add(BatchNormalization())
    # Adding non-linearity
    model.add(Activation('relu'))
    # Max Pooling
    model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2), padding='valid'))
    # Dropout
    model.add(Dropout(dropout_rate))

    # 3rd Convolutional Layer
    model.add(Conv2D(filters=384, kernel_size=(3, 3), strides=(1, 1), padding='valid'))
    # Batch Normalization
    model.add(BatchNormalization())
    # Adding non-linearity
    model.add(Activation('relu'))

    # 4th Convolutional Layer
    model.add(Conv2D(filters=384, kernel_size=(3, 3), strides=(1, 1), padding='valid'))
    # Batch Normalization
    model.add(BatchNormalization())
    # Adding non-linearity
    model.add(Activation('relu'))

    # 5th Convolutional Layer
    model.add(Conv2D(filters=256, kernel_size=(3, 3), strides=(1, 1), padding='valid'))
    # Batch Normalization
    model.add(BatchNormalization())
    # Adding non-linearity
    model.add(Activation('relu'))
    # Max Pooling
    model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2), padding='valid'))
    # Dropout
    model.add(Dropout(dropout_rate))

    # Flatten the feature maps to pass them to Fully Connected Layers
    model.add(Flatten())

    # Fully Connected Layers
    model.add(Dense(4096))
    model.add(BatchNormalization())
    model.add(Activation('relu'))
    model.add(Dense(4096))
    model.add(BatchNormalization())
    model.add(Activation('relu'))
    model.add(Dense(number_of_classes, activation='softmax'))

    model.summary()
    return model

VGG-16 with BatchNormalization and Dropout:

In [5]:
def vgg_16_bd(input_data_shape=(224, 224, 3,), number_of_classes=10):
    model = Sequential()
    # Block 1
    model.add(Conv2D(filters=64, input_shape=input_data_shape, kernel_size=(3, 3), padding='same'))
    model.add(BatchNormalization())
    model.add(Activation('relu'))
    model.add(Conv2D(filters=64, input_shape=input_data_shape, kernel_size=(3, 3), padding='same'))
    model.add(BatchNormalization())
    model.add(Activation('relu'))
    model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))
    model.add(Dropout(dropout_rate))

    # Block 2
    model.add(Conv2D(filters=128, input_shape=input_data_shape, kernel_size=(3, 3), padding='same'))
    model.add(BatchNormalization())
    model.add(Activation('relu'))
    model.add(Conv2D(filters=128, input_shape=input_data_shape, kernel_size=(3, 3), padding='same'))
    model.add(BatchNormalization())
    model.add(Activation('relu'))
    model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))
    model.add(Dropout(dropout_rate))

    # Block 3
    model.add(Conv2D(filters=256, input_shape=input_data_shape, kernel_size=(3, 3), padding='same'))
    model.add(BatchNormalization())
    model.add(Activation('relu'))
    model.add(Conv2D(filters=256, input_shape=input_data_shape, kernel_size=(3, 3), padding='same'))
    model.add(BatchNormalization())
    model.add(Activation('relu'))
    model.add(Conv2D(filters=256, input_shape=input_data_shape, kernel_size=(3, 3), padding='same'))
    model.add(BatchNormalization())
    model.add(Activation('relu'))
    model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))
    model.add(Dropout(dropout_rate))

    # Block 4
    model.add(Conv2D(filters=512, input_shape=input_data_shape, kernel_size=(3, 3), padding='same'))
    model.add(BatchNormalization())
    model.add(Activation('relu'))
    model.add(Conv2D(filters=512, input_shape=input_data_shape, kernel_size=(3, 3), padding='same'))
    model.add(BatchNormalization())
    model.add(Activation('relu'))
    model.add(Conv2D(filters=512, input_shape=input_data_shape, kernel_size=(3, 3), padding='same'))
    model.add(BatchNormalization())
    model.add(Activation('relu'))
    model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))
    model.add(Dropout(dropout_rate))

    # Block 5
    model.add(Conv2D(filters=512, input_shape=input_data_shape, kernel_size=(3, 3), padding='same'))
    model.add(BatchNormalization())
    model.add(Activation('relu'))
    model.add(Conv2D(filters=512, input_shape=input_data_shape, kernel_size=(3, 3), padding='same'))
    model.add(BatchNormalization())
    model.add(Activation('relu'))
    model.add(Conv2D(filters=512, input_shape=input_data_shape, kernel_size=(3, 3), padding='same'))
    model.add(BatchNormalization())
    model.add(Activation('relu'))
    model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))
    model.add(Dropout(dropout_rate))

    # Flatten the feature maps to pass them to Fully Connected Layers
    model.add(Flatten())

    # fully connected layers
    model.add(Dense(4096))
    model.add(BatchNormalization())
    model.add(Activation('relu'))
    model.add(Dense(4096))
    model.add(BatchNormalization())
    model.add(Activation('relu'))
    model.add(Dense(number_of_classes, activation='softmax'))

    # Create model.
    model.summary()

    return model

CNN_5x5_Net with BatchNormalization and Dropout:

In [6]:
def cnn_5x5_net_bd(input_data_shape=(224, 224, 3), number_of_classes=10):
    model = Sequential()
    # Block 1
    model.add(Conv2D(filters=64, input_shape=input_data_shape, kernel_size=(5, 5), padding='same'))
    model.add(BatchNormalization())
    model.add(Activation('relu'))
    model.add(Conv2D(filters=64, input_shape=input_data_shape, kernel_size=(5, 5), padding='same'))
    model.add(BatchNormalization())
    model.add(Activation('relu'))
    model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))
    model.add(Dropout(dropout_rate))

    # Block 2
    model.add(Conv2D(filters=128, input_shape=input_data_shape, kernel_size=(5, 5), padding='same'))
    model.add(BatchNormalization())
    model.add(Activation('relu'))
    model.add(Conv2D(filters=128, input_shape=input_data_shape, kernel_size=(5, 5), padding='same'))
    model.add(BatchNormalization())
    model.add(Activation('relu'))
    model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))
    model.add(Dropout(dropout_rate))

    # Block 3
    model.add(Conv2D(filters=256, input_shape=input_data_shape, kernel_size=(5, 5), padding='same'))
    model.add(BatchNormalization())
    model.add(Activation('relu'))
    model.add(Conv2D(filters=256, input_shape=input_data_shape, kernel_size=(5, 5), padding='same'))
    model.add(BatchNormalization())
    model.add(Activation('relu'))
    model.add(Conv2D(filters=256, input_shape=input_data_shape, kernel_size=(5, 5), padding='same'))
    model.add(BatchNormalization())
    model.add(Activation('relu'))
    model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))
    model.add(Dropout(dropout_rate))

    # Block 4
    model.add(Conv2D(filters=512, input_shape=input_data_shape, kernel_size=(5, 5), padding='same'))
    model.add(BatchNormalization())
    model.add(Activation('relu'))
    model.add(Conv2D(filters=512, input_shape=input_data_shape, kernel_size=(5, 5), padding='same'))
    model.add(BatchNormalization())
    model.add(Activation('relu'))
    model.add(Conv2D(filters=512, input_shape=input_data_shape, kernel_size=(5, 5), padding='same'))
    model.add(BatchNormalization())
    model.add(Activation('relu'))
    model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))
    model.add(Dropout(dropout_rate))

    # Block 5
    model.add(Conv2D(filters=512, input_shape=input_data_shape, kernel_size=(5, 5), padding='same'))
    model.add(BatchNormalization())
    model.add(Activation('relu'))
    model.add(Conv2D(filters=512, input_shape=input_data_shape, kernel_size=(5, 5), padding='same'))
    model.add(BatchNormalization())
    model.add(Activation('relu'))
    model.add(Conv2D(filters=512, input_shape=input_data_shape, kernel_size=(5, 5), padding='same'))
    model.add(BatchNormalization())
    model.add(Activation('relu'))
    model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))
    model.add(Dropout(dropout_rate))

    # Flatten the feature maps to pass them to Fully Connected Layers
    model.add(Flatten())

    # fully connected layers
    model.add(Dense(4096))
    model.add(BatchNormalization())
    model.add(Activation('relu'))
    model.add(Dense(4096))
    model.add(BatchNormalization())
    model.add(Activation('relu'))
    model.add(Dense(number_of_classes, activation='softmax'))

    # Create model.
    model.summary()

    return model

Set the inference hyper-parameters:

In [7]:
batch_size = 128
num_classes = 10
num_of_training_iteration = 100

Set the input image shape/size

In [8]:
input_data_shape = (224, 224, 3)

Build AlexNet & VGG-16 & CNN_5x5_Net networks with BatchNormalization and Dropout (default input image shape is 224x224x3)

In [9]:
alexnet_bd_model = alexnet_bd(input_data_shape=input_data_shape)
vgg16_bd_model = vgg_16_bd(input_data_shape=input_data_shape)
#cnn_5x5_net_bd_model = cnn_5x5_net_bd(input_data_shape=input_data_shape)
WARNING:tensorflow:From /home/eldar/.conda/envs/dltokio/lib/python3.7/site-packages/tensorflow/python/ops/resource_variable_ops.py:435: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version. Instructions for updating: Colocations handled automatically by placer. WARNING:tensorflow:From /home/eldar/.conda/envs/dltokio/lib/python3.7/site-packages/tensorflow/python/keras/layers/core.py:143: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version. Instructions for updating: Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`. _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= conv2d (Conv2D) (None, 54, 54, 96) 34944 _________________________________________________________________ batch_normalization_v1 (Batc (None, 54, 54, 96) 384 _________________________________________________________________ activation (Activation) (None, 54, 54, 96) 0 _________________________________________________________________ max_pooling2d (MaxPooling2D) (None, 27, 27, 96) 0 _________________________________________________________________ dropout (Dropout) (None, 27, 27, 96) 0 _________________________________________________________________ conv2d_1 (Conv2D) (None, 17, 17, 256) 2973952 _________________________________________________________________ batch_normalization_v1_1 (Ba (None, 17, 17, 256) 1024 _________________________________________________________________ activation_1 (Activation) (None, 17, 17, 256) 0 _________________________________________________________________ max_pooling2d_1 (MaxPooling2 (None, 8, 8, 256) 0 _________________________________________________________________ dropout_1 (Dropout) (None, 8, 8, 256) 0 _________________________________________________________________ conv2d_2 (Conv2D) (None, 6, 6, 384) 885120 _________________________________________________________________ batch_normalization_v1_2 (Ba (None, 6, 6, 384) 1536 _________________________________________________________________ activation_2 (Activation) (None, 6, 6, 384) 0 _________________________________________________________________ conv2d_3 (Conv2D) (None, 4, 4, 384) 1327488 _________________________________________________________________ batch_normalization_v1_3 (Ba (None, 4, 4, 384) 1536 _________________________________________________________________ activation_3 (Activation) (None, 4, 4, 384) 0 _________________________________________________________________ conv2d_4 (Conv2D) (None, 2, 2, 256) 884992 _________________________________________________________________ batch_normalization_v1_4 (Ba (None, 2, 2, 256) 1024 _________________________________________________________________ activation_4 (Activation) (None, 2, 2, 256) 0 _________________________________________________________________ max_pooling2d_2 (MaxPooling2 (None, 1, 1, 256) 0 _________________________________________________________________ dropout_2 (Dropout) (None, 1, 1, 256) 0 _________________________________________________________________ flatten (Flatten) (None, 256) 0 _________________________________________________________________ dense (Dense) (None, 4096) 1052672 _________________________________________________________________ batch_normalization_v1_5 (Ba (None, 4096) 16384 _________________________________________________________________ activation_5 (Activation) (None, 4096) 0 _________________________________________________________________ dense_1 (Dense) (None, 4096) 16781312 _________________________________________________________________ batch_normalization_v1_6 (Ba (None, 4096) 16384 _________________________________________________________________ activation_6 (Activation) (None, 4096) 0 _________________________________________________________________ dense_2 (Dense) (None, 10) 40970 ================================================================= Total params: 24,019,722 Trainable params: 24,000,586 Non-trainable params: 19,136 _________________________________________________________________ _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= conv2d_5 (Conv2D) (None, 224, 224, 64) 1792 _________________________________________________________________ batch_normalization_v1_7 (Ba (None, 224, 224, 64) 256 _________________________________________________________________ activation_7 (Activation) (None, 224, 224, 64) 0 _________________________________________________________________ conv2d_6 (Conv2D) (None, 224, 224, 64) 36928 _________________________________________________________________ batch_normalization_v1_8 (Ba (None, 224, 224, 64) 256 _________________________________________________________________ activation_8 (Activation) (None, 224, 224, 64) 0 _________________________________________________________________ max_pooling2d_3 (MaxPooling2 (None, 112, 112, 64) 0 _________________________________________________________________ dropout_3 (Dropout) (None, 112, 112, 64) 0 _________________________________________________________________ conv2d_7 (Conv2D) (None, 112, 112, 128) 73856 _________________________________________________________________ batch_normalization_v1_9 (Ba (None, 112, 112, 128) 512 _________________________________________________________________ activation_9 (Activation) (None, 112, 112, 128) 0 _________________________________________________________________ conv2d_8 (Conv2D) (None, 112, 112, 128) 147584 _________________________________________________________________ batch_normalization_v1_10 (B (None, 112, 112, 128) 512 _________________________________________________________________ activation_10 (Activation) (None, 112, 112, 128) 0 _________________________________________________________________ max_pooling2d_4 (MaxPooling2 (None, 56, 56, 128) 0 _________________________________________________________________ dropout_4 (Dropout) (None, 56, 56, 128) 0 _________________________________________________________________ conv2d_9 (Conv2D) (None, 56, 56, 256) 295168 _________________________________________________________________ batch_normalization_v1_11 (B (None, 56, 56, 256) 1024 _________________________________________________________________ activation_11 (Activation) (None, 56, 56, 256) 0 _________________________________________________________________ conv2d_10 (Conv2D) (None, 56, 56, 256) 590080 _________________________________________________________________ batch_normalization_v1_12 (B (None, 56, 56, 256) 1024 _________________________________________________________________ activation_12 (Activation) (None, 56, 56, 256) 0 _________________________________________________________________ conv2d_11 (Conv2D) (None, 56, 56, 256) 590080 _________________________________________________________________ batch_normalization_v1_13 (B (None, 56, 56, 256) 1024 _________________________________________________________________ activation_13 (Activation) (None, 56, 56, 256) 0 _________________________________________________________________ max_pooling2d_5 (MaxPooling2 (None, 28, 28, 256) 0 _________________________________________________________________ dropout_5 (Dropout) (None, 28, 28, 256) 0 _________________________________________________________________ conv2d_12 (Conv2D) (None, 28, 28, 512) 1180160 _________________________________________________________________ batch_normalization_v1_14 (B (None, 28, 28, 512) 2048 _________________________________________________________________ activation_14 (Activation) (None, 28, 28, 512) 0 _________________________________________________________________ conv2d_13 (Conv2D) (None, 28, 28, 512) 2359808 _________________________________________________________________ batch_normalization_v1_15 (B (None, 28, 28, 512) 2048 _________________________________________________________________ activation_15 (Activation) (None, 28, 28, 512) 0 _________________________________________________________________ conv2d_14 (Conv2D) (None, 28, 28, 512) 2359808 _________________________________________________________________ batch_normalization_v1_16 (B (None, 28, 28, 512) 2048 _________________________________________________________________ activation_16 (Activation) (None, 28, 28, 512) 0 _________________________________________________________________ max_pooling2d_6 (MaxPooling2 (None, 14, 14, 512) 0 _________________________________________________________________ dropout_6 (Dropout) (None, 14, 14, 512) 0 _________________________________________________________________ conv2d_15 (Conv2D) (None, 14, 14, 512) 2359808 _________________________________________________________________ batch_normalization_v1_17 (B (None, 14, 14, 512) 2048 _________________________________________________________________ activation_17 (Activation) (None, 14, 14, 512) 0 _________________________________________________________________ conv2d_16 (Conv2D) (None, 14, 14, 512) 2359808 _________________________________________________________________ batch_normalization_v1_18 (B (None, 14, 14, 512) 2048 _________________________________________________________________ activation_18 (Activation) (None, 14, 14, 512) 0 _________________________________________________________________ conv2d_17 (Conv2D) (None, 14, 14, 512) 2359808 _________________________________________________________________ batch_normalization_v1_19 (B (None, 14, 14, 512) 2048 _________________________________________________________________ activation_19 (Activation) (None, 14, 14, 512) 0 _________________________________________________________________ max_pooling2d_7 (MaxPooling2 (None, 7, 7, 512) 0 _________________________________________________________________ dropout_7 (Dropout) (None, 7, 7, 512) 0 _________________________________________________________________ flatten_1 (Flatten) (None, 25088) 0 _________________________________________________________________ dense_3 (Dense) (None, 4096) 102764544 _________________________________________________________________ batch_normalization_v1_20 (B (None, 4096) 16384 _________________________________________________________________ activation_20 (Activation) (None, 4096) 0 _________________________________________________________________ dense_4 (Dense) (None, 4096) 16781312 _________________________________________________________________ batch_normalization_v1_21 (B (None, 4096) 16384 _________________________________________________________________ activation_21 (Activation) (None, 4096) 0 _________________________________________________________________ dense_5 (Dense) (None, 10) 40970 ================================================================= Total params: 134,351,178 Trainable params: 134,326,346 Non-trainable params: 24,832 _________________________________________________________________

Train AlexNet and VGG-16 with BatchNormalization and Dropout on MNIST dataset:

In [10]:
alexnet_bd_model.compile(loss=keras.losses.categorical_crossentropy, optimizer='adam', metrics=["accuracy"])
vgg16_bd_model.compile(loss=keras.losses.categorical_crossentropy, optimizer='adam', metrics=["accuracy"])
#cnn_5x5_net_bd_model.compile(loss=keras.losses.categorical_crossentropy, optimizer='adam', metrics=["accuracy"])

alexnet_bd_inference_time = []
vgg16_bd_inference_time = []
#cnn_5x5_net_bd_inference_time = []

# dummy tensor to check the inference time of each network
x_test = np.random.rand(batch_size, input_data_shape[0], input_data_shape[1], input_data_shape[2])

for _ in range(num_of_training_iteration):
    alexnet_bd_inference_start = datetime.now()
    alexnet_bd_inference = alexnet_bd_model.predict_classes(x_test)
    alexnet_bd_inference_finish = datetime.now()
    alexnet_bd_inference_time.append(alexnet_bd_inference_finish - alexnet_bd_inference_start)

for _ in range(num_of_training_iteration):
    vgg16_bd_inference_start = datetime.now()
    vgg_bd_inference = vgg16_bd_model.predict_classes(x_test)
    vgg16_bd_inference_finish = datetime.now()
    vgg16_bd_inference_time.append(vgg16_bd_inference_finish - vgg16_bd_inference_start)
"""
for _ in range(num_of_training_iteration):
    cnn_5x5_net_bd_inference_start = datetime.now()
    cnn_5x5_net_bd_inference = cnn_5x5_net_bd_model.predict_classes(x_test)
    cnn_5x5_net_bd_inference_finish = datetime.now()
    cnn_5x5_net_bd_inference_time.append(cnn_5x5_net_bd_inference_finish - cnn_5x5_net_bd_inference_start)
"""

print("Average Inference time for AlexNet: {}".format(np.mean(alexnet_bd_inference_time)))
print("Average Inference time for VGG-16: {}".format(np.mean(vgg16_bd_inference_time)))
#print("Average Inference time for CNN_5x5_NET {}".format(np.mean(cnn_5x5_net_bd_inference_time)))
Average Inference time for AlexNet: 0:00:00.167202 Average Inference time for VGG-16: 0:00:01.012128

Build CNN_5x5_Net networks with BatchNormalization and Dropout (default input image shape is 224x224x3)

In [11]:
cnn_5x5_net_bd_model = cnn_5x5_net_bd(input_data_shape=input_data_shape)
_________________________________________________________________ Layer (type) Output Shape Param # ================================================================= conv2d_18 (Conv2D) (None, 224, 224, 64) 4864 _________________________________________________________________ batch_normalization_v1_22 (B (None, 224, 224, 64) 256 _________________________________________________________________ activation_22 (Activation) (None, 224, 224, 64) 0 _________________________________________________________________ conv2d_19 (Conv2D) (None, 224, 224, 64) 102464 _________________________________________________________________ batch_normalization_v1_23 (B (None, 224, 224, 64) 256 _________________________________________________________________ activation_23 (Activation) (None, 224, 224, 64) 0 _________________________________________________________________ max_pooling2d_8 (MaxPooling2 (None, 112, 112, 64) 0 _________________________________________________________________ dropout_8 (Dropout) (None, 112, 112, 64) 0 _________________________________________________________________ conv2d_20 (Conv2D) (None, 112, 112, 128) 204928 _________________________________________________________________ batch_normalization_v1_24 (B (None, 112, 112, 128) 512 _________________________________________________________________ activation_24 (Activation) (None, 112, 112, 128) 0 _________________________________________________________________ conv2d_21 (Conv2D) (None, 112, 112, 128) 409728 _________________________________________________________________ batch_normalization_v1_25 (B (None, 112, 112, 128) 512 _________________________________________________________________ activation_25 (Activation) (None, 112, 112, 128) 0 _________________________________________________________________ max_pooling2d_9 (MaxPooling2 (None, 56, 56, 128) 0 _________________________________________________________________ dropout_9 (Dropout) (None, 56, 56, 128) 0 _________________________________________________________________ conv2d_22 (Conv2D) (None, 56, 56, 256) 819456 _________________________________________________________________ batch_normalization_v1_26 (B (None, 56, 56, 256) 1024 _________________________________________________________________ activation_26 (Activation) (None, 56, 56, 256) 0 _________________________________________________________________ conv2d_23 (Conv2D) (None, 56, 56, 256) 1638656 _________________________________________________________________ batch_normalization_v1_27 (B (None, 56, 56, 256) 1024 _________________________________________________________________ activation_27 (Activation) (None, 56, 56, 256) 0 _________________________________________________________________ conv2d_24 (Conv2D) (None, 56, 56, 256) 1638656 _________________________________________________________________ batch_normalization_v1_28 (B (None, 56, 56, 256) 1024 _________________________________________________________________ activation_28 (Activation) (None, 56, 56, 256) 0 _________________________________________________________________ max_pooling2d_10 (MaxPooling (None, 28, 28, 256) 0 _________________________________________________________________ dropout_10 (Dropout) (None, 28, 28, 256) 0 _________________________________________________________________ conv2d_25 (Conv2D) (None, 28, 28, 512) 3277312 _________________________________________________________________ batch_normalization_v1_29 (B (None, 28, 28, 512) 2048 _________________________________________________________________ activation_29 (Activation) (None, 28, 28, 512) 0 _________________________________________________________________ conv2d_26 (Conv2D) (None, 28, 28, 512) 6554112 _________________________________________________________________ batch_normalization_v1_30 (B (None, 28, 28, 512) 2048 _________________________________________________________________ activation_30 (Activation) (None, 28, 28, 512) 0 _________________________________________________________________ conv2d_27 (Conv2D) (None, 28, 28, 512) 6554112 _________________________________________________________________ batch_normalization_v1_31 (B (None, 28, 28, 512) 2048 _________________________________________________________________ activation_31 (Activation) (None, 28, 28, 512) 0 _________________________________________________________________ max_pooling2d_11 (MaxPooling (None, 14, 14, 512) 0 _________________________________________________________________ dropout_11 (Dropout) (None, 14, 14, 512) 0 _________________________________________________________________ conv2d_28 (Conv2D) (None, 14, 14, 512) 6554112 _________________________________________________________________ batch_normalization_v1_32 (B (None, 14, 14, 512) 2048 _________________________________________________________________ activation_32 (Activation) (None, 14, 14, 512) 0 _________________________________________________________________ conv2d_29 (Conv2D) (None, 14, 14, 512) 6554112 _________________________________________________________________ batch_normalization_v1_33 (B (None, 14, 14, 512) 2048 _________________________________________________________________ activation_33 (Activation) (None, 14, 14, 512) 0 _________________________________________________________________ conv2d_30 (Conv2D) (None, 14, 14, 512) 6554112 _________________________________________________________________ batch_normalization_v1_34 (B (None, 14, 14, 512) 2048 _________________________________________________________________ activation_34 (Activation) (None, 14, 14, 512) 0 _________________________________________________________________ max_pooling2d_12 (MaxPooling (None, 7, 7, 512) 0 _________________________________________________________________ dropout_12 (Dropout) (None, 7, 7, 512) 0 _________________________________________________________________ flatten_2 (Flatten) (None, 25088) 0 _________________________________________________________________ dense_6 (Dense) (None, 4096) 102764544 _________________________________________________________________ batch_normalization_v1_35 (B (None, 4096) 16384 _________________________________________________________________ activation_35 (Activation) (None, 4096) 0 _________________________________________________________________ dense_7 (Dense) (None, 4096) 16781312 _________________________________________________________________ batch_normalization_v1_36 (B (None, 4096) 16384 _________________________________________________________________ activation_36 (Activation) (None, 4096) 0 _________________________________________________________________ dense_8 (Dense) (None, 10) 40970 ================================================================= Total params: 160,503,114 Trainable params: 160,478,282 Non-trainable params: 24,832 _________________________________________________________________

Train CNN_5x5_Net with BatchNormalization and Dropout on MNIST dataset:

In [12]:
cnn_5x5_net_bd_model.compile(loss=keras.losses.categorical_crossentropy, optimizer='adam', metrics=["accuracy"])
cnn_5x5_net_bd_inference_time = []

for _ in range(num_of_training_iteration):
    cnn_5x5_net_bd_inference_start = datetime.now()
    cnn_5x5_net_bd_inference = cnn_5x5_net_bd_model.predict_classes(x_test)
    cnn_5x5_net_bd_inference_finish = datetime.now()
    cnn_5x5_net_bd_inference_time.append(cnn_5x5_net_bd_inference_finish - cnn_5x5_net_bd_inference_start)
    
print("Average Inference time for CNN_5x5_NET {}".format(np.mean(cnn_5x5_net_bd_inference_time)))
Average Inference time for CNN_5x5_NET 0:00:01.301401
In [14]:
import jovian
jovian.commit()
[jovian] Saving notebook..
[jovian] Updating notebook "dac35e1a7479468e90af9e5d114ca732" on https://jvn.io [jovian] Uploading notebook.. [jovian] Capturing environment.. [jovian] Committed successfully! https://jvn.io/rubensilver/dac35e1a7479468e90af9e5d114ca732
In [ ]: