Jovian
⭐️
Sign In

Ref:

Self made hand digits and its training+prediction using Keras Conv2D model

In [1]:
from keras.models import Sequential
from keras.layers import Dense,Conv2D,MaxPooling2D,Flatten,Dropout
from keras.optimizers import Adam
from keras.losses import categorical_crossentropy
from keras.callbacks import TensorBoard
from keras.utils import plot_model
from keras.utils import to_categorical
import cv2 as cv
import os
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.model_selection import train_test_split
from keras.utils import to_categorical
from sklearn.metrics import confusion_matrix,accuracy_score

from keras import backend as K # required to clear the graph session, if any execution was done before
Using TensorFlow backend.

Read the handwritten digits and visualize them

In [20]:
digit_dir = "./Hand_Digits/"
cnt = 1
for dir in os.walk(digit_dir):
    for folder in dir[1]:
        for file in os.walk(digit_dir+folder):
            full_path = digit_dir+folder+"/"+file[-1][0]
#             print(full_path)
            im = cv.imread(full_path)
            plt.subplot(2,5,cnt)
            plt.imshow(im)
            plt.title(file[-1][0])
            plt.axis("off")
            cnt +=1
plt.show()            
            
Notebook Image

Image augmentation

using augmentation (ImageDataGenerator() in Keras), we will increase the image dataset by chnaging some standrad parameters

In [32]:
from keras.preprocessing.image import ImageDataGenerator
In [39]:
datagen = ImageDataGenerator(rotation_range=25,width_shift_range=.1,height_shift_range=.1,zoom_range=.5,fill_mode="nearest",
                  horizontal_flip=False,vertical_flip=False)
In [40]:
datagen
Out[40]:
<keras.preprocessing.image.ImageDataGenerator at 0x1f137595b08>
In [43]:
digit_dir = "./Hand_Digits/"
cnt = 1
for dir in os.walk(digit_dir):
    for folder in dir[1]:
        for file in os.walk(digit_dir+folder):
            full_path = digit_dir+folder+"/"+file[-1][0]
            im = cv.imread(full_path)
            im_batch = np.expand_dims(im,axis=0)
            num_aug = 0
            NUM_TO_AUGMENT = 200 # limiting the number ig augmented image for each digits, can be chnaged later
            for im_aug in datagen.flow(im_batch,batch_size=1,save_to_dir="digits_augmented_image",save_prefix=file[-1][0],save_format="png"):
                if num_aug > NUM_TO_AUGMENT:
                    break
                num_aug += 1            

read each image and label them and create dataFrame using pandas

In [2]:
aug_img_dir = "./digits_augmented_image/"
image_database = pd.DataFrame(columns=["digits","labels"])

cnt = 0
for data in os.walk(aug_img_dir):
    lof = data[-1]
    for file in lof:
        pof = aug_img_dir+file
        im = cv.imread(pof)
#         im = im.reshape(-1,20,20,3)
        label = file.split("_")[0] 
        image_database.loc[cnt,"digits"] = im
        image_database.loc[cnt,"labels"] = label
        cnt += 1
image_database
Out[2]:

Binarize the label using to_categorical() of keras

In [3]:
bin_labels = to_categorical(image_database.labels)
bin_labels
Out[3]:
array([[1., 0., 0., ..., 0., 0., 0.],
       [1., 0., 0., ..., 0., 0., 0.],
       [1., 0., 0., ..., 0., 0., 0.],
       ...,
       [0., 0., 0., ..., 0., 0., 1.],
       [0., 0., 0., ..., 0., 0., 1.],
       [0., 0., 0., ..., 0., 0., 1.]], dtype=float32)

Check the shape

In [4]:
image_database["digits"][0].shape
Out[4]:
(20, 20, 3)

Convert the shape to 4D shape that keras understands

In [5]:
image_batch = np.array([im for im in image_database["digits"]]).reshape(-1,20,20,3)
In [6]:
image_batch.shape
Out[6]:
(2001, 20, 20, 3)
In [7]:
image_batch.ndim # number of dimensions
Out[7]:
4

Create Keras model

set seed value to unique for reproducibility
In [8]:
def reproduceResult():    
    # Seed value (can actually be different for each attribution step)
    seed_value= 0

    # 1. Set `PYTHONHASHSEED` environment variable at a fixed value
    import os
    os.environ['PYTHONHASHSEED']=str(seed_value)

    # 2. Set `python` built-in pseudo-random generator at a fixed value
    import random
    random.seed(seed_value)

    # 3. Set `numpy` pseudo-random generator at a fixed value
    import numpy as np
    np.random.seed(seed_value)

    # 4. Set `tensorflow` pseudo-random generator at a fixed value
    import tensorflow as tf
    tf.compat.v1.set_random_seed(seed_value)
In [10]:
reproduceResult()

# if model:
#     del model
model = Sequential()

model.add(Conv2D(filters=32,kernel_size=(2,2),activation="relu",input_shape=image_database["digits"][0].shape,name="l1"))
model.add(MaxPooling2D(name="l2"))

model.add(Conv2D(filters=64,kernel_size=(2,2),activation="relu",name="l3"))
model.add(MaxPooling2D(name="l4"))
model.add(Dropout(rate=.2,name="l5"))
          
model.add(Conv2D(filters=64,kernel_size=(2,2),activation="relu",name="l6"))
model.add(MaxPooling2D(name="l7"))
model.add(Dropout(rate=.2,name="l8"))
model.add(Flatten(name="l9"))
model.add(Dense(10,activation="softmax",name="l10"))

plot the model

In [11]:
plot_model(model,rankdir="LR")
Out[11]:
Notebook Image

configure the model for training by compiling

In [12]:
model.compile(optimizer="adam",loss="categorical_crossentropy",metrics=["accuracy"])

Create train and test data

In [13]:
X_train,X_test,y_train,y_test = train_test_split(image_batch,bin_labels,test_size=.2,random_state=3,shuffle=True)
In [14]:
print(X_train.shape)
print(X_test.shape)
print(y_train.shape)
print(y_test.shape)
(1600, 20, 20, 3) (401, 20, 20, 3) (1600, 10) (401, 10)

Train the model using fit()

In [15]:
model.fit(x=X_train,y=y_train,batch_size=10,epochs=30,validation_split=.2,verbose=1)
Train on 1280 samples, validate on 320 samples Epoch 1/30 1280/1280 [==============================] - 1s 1ms/step - loss: 10.3988 - accuracy: 0.1234 - val_loss: 2.1801 - val_accuracy: 0.1781 Epoch 2/30 1280/1280 [==============================] - 1s 745us/step - loss: 2.2612 - accuracy: 0.1734 - val_loss: 2.1193 - val_accuracy: 0.2594 Epoch 3/30 1280/1280 [==============================] - 1s 891us/step - loss: 2.1669 - accuracy: 0.2203 - val_loss: 2.0794 - val_accuracy: 0.2969 Epoch 4/30 1280/1280 [==============================] - 1s 879us/step - loss: 2.1208 - accuracy: 0.2516 - val_loss: 2.0161 - val_accuracy: 0.2688 Epoch 5/30 1280/1280 [==============================] - 1s 782us/step - loss: 2.0630 - accuracy: 0.2844 - val_loss: 1.7240 - val_accuracy: 0.3906 Epoch 6/30 1280/1280 [==============================] - 1s 830us/step - loss: 1.9751 - accuracy: 0.3141 - val_loss: 1.6840 - val_accuracy: 0.4375 Epoch 7/30 1280/1280 [==============================] - 1s 1ms/step - loss: 1.8952 - accuracy: 0.3438 - val_loss: 1.6234 - val_accuracy: 0.4344 Epoch 8/30 1280/1280 [==============================] - 1s 845us/step - loss: 1.8216 - accuracy: 0.3688 - val_loss: 1.5174 - val_accuracy: 0.5031 Epoch 9/30 1280/1280 [==============================] - 1s 770us/step - loss: 1.7384 - accuracy: 0.4000 - val_loss: 1.3920 - val_accuracy: 0.5437 Epoch 10/30 1280/1280 [==============================] - 1s 779us/step - loss: 1.6250 - accuracy: 0.4508 - val_loss: 1.4115 - val_accuracy: 0.5688 Epoch 11/30 1280/1280 [==============================] - 1s 799us/step - loss: 1.5779 - accuracy: 0.4656 - val_loss: 1.2290 - val_accuracy: 0.6031 Epoch 12/30 1280/1280 [==============================] - 1s 757us/step - loss: 1.4598 - accuracy: 0.5000 - val_loss: 1.2004 - val_accuracy: 0.5781 Epoch 13/30 1280/1280 [==============================] - 1s 767us/step - loss: 1.4285 - accuracy: 0.5063 - val_loss: 1.3032 - val_accuracy: 0.5688 Epoch 14/30 1280/1280 [==============================] - 1s 755us/step - loss: 1.3484 - accuracy: 0.5320 - val_loss: 1.0210 - val_accuracy: 0.6375 Epoch 15/30 1280/1280 [==============================] - 1s 794us/step - loss: 1.2010 - accuracy: 0.5844 - val_loss: 1.0336 - val_accuracy: 0.6625 Epoch 16/30 1280/1280 [==============================] - 1s 827us/step - loss: 1.2189 - accuracy: 0.5727 - val_loss: 0.7510 - val_accuracy: 0.7375 Epoch 17/30 1280/1280 [==============================] - 1s 757us/step - loss: 1.0240 - accuracy: 0.6445 - val_loss: 0.6464 - val_accuracy: 0.7875 Epoch 18/30 1280/1280 [==============================] - 1s 769us/step - loss: 0.9608 - accuracy: 0.6773 - val_loss: 0.7046 - val_accuracy: 0.7906 Epoch 19/30 1280/1280 [==============================] - 1s 800us/step - loss: 0.8774 - accuracy: 0.6898 - val_loss: 0.5687 - val_accuracy: 0.7969 Epoch 20/30 1280/1280 [==============================] - 1s 771us/step - loss: 0.9053 - accuracy: 0.6898 - val_loss: 0.5316 - val_accuracy: 0.8125 Epoch 21/30 1280/1280 [==============================] - 1s 777us/step - loss: 0.7928 - accuracy: 0.7266 - val_loss: 0.4623 - val_accuracy: 0.8656 Epoch 22/30 1280/1280 [==============================] - 1s 732us/step - loss: 0.7901 - accuracy: 0.7437 - val_loss: 0.4829 - val_accuracy: 0.8406 Epoch 23/30 1280/1280 [==============================] - 1s 791us/step - loss: 0.7165 - accuracy: 0.7664 - val_loss: 0.4712 - val_accuracy: 0.8438 Epoch 24/30 1280/1280 [==============================] - 1s 794us/step - loss: 0.6695 - accuracy: 0.7758 - val_loss: 0.4975 - val_accuracy: 0.8313 Epoch 25/30 1280/1280 [==============================] - 1s 751us/step - loss: 0.5996 - accuracy: 0.7992 - val_loss: 0.4072 - val_accuracy: 0.8719 Epoch 26/30 1280/1280 [==============================] - 1s 794us/step - loss: 0.6289 - accuracy: 0.7953 - val_loss: 0.4112 - val_accuracy: 0.8625 Epoch 27/30 1280/1280 [==============================] - 1s 782us/step - loss: 0.5566 - accuracy: 0.8133 - val_loss: 0.4004 - val_accuracy: 0.8687 Epoch 28/30 1280/1280 [==============================] - 1s 757us/step - loss: 0.5777 - accuracy: 0.8086 - val_loss: 0.3996 - val_accuracy: 0.8719 Epoch 29/30 1280/1280 [==============================] - 1s 735us/step - loss: 0.5540 - accuracy: 0.8211 - val_loss: 0.3933 - val_accuracy: 0.8625 Epoch 30/30 1280/1280 [==============================] - 1s 761us/step - loss: 0.5035 - accuracy: 0.8375 - val_loss: 0.3391 - val_accuracy: 0.8875
Out[15]:
<keras.callbacks.callbacks.History at 0x210edc47d08>

Evaluate the model

In [16]:
model.evaluate(X_test,y_test,batch_size=2)
401/401 [==============================] - 0s 623us/step
Out[16]:
[0.376866121085372, 0.8728179335594177]

Make predictions on test data

In [17]:
y_pred = model.predict_classes(X_test)
# y_pred
In [476]:
# y_test[2]
In [18]:
print(model.metrics_names)
model.evaluate(X_test,y_test)
['loss', 'accuracy'] 401/401 [==============================] - 0s 155us/step
Out[18]:
[0.3768661238457497, 0.8728179335594177]

Get the verbose history

In [19]:
model.history.history.keys()
Out[19]:
dict_keys(['val_loss', 'val_accuracy', 'loss', 'accuracy'])

Plot the loss and accuracy parameters using seaborn

In [20]:
plt.figure(figsize=(16,4))
plt.subplot(1,2,1)
sns.lineplot(model.history.epoch,model.history.history["loss"],label="loss")
sns.lineplot(model.history.epoch,model.history.history["accuracy"],label="accuracy")
plt.subplot(1,2,2)
sns.lineplot(model.history.epoch,model.history.history["val_loss"],label="val_loss")
sns.lineplot(model.history.epoch,model.history.history["val_accuracy"],label="val_accuracy")
plt.legend()
plt.show()
Notebook Image

Get the index where "1" is available, that will same as digit number

In [28]:
y_test_label = np.argmax(y_test,axis=1)
y_test_label
Out[28]:
array([4, 6, 7, 9, 1, 0, 1, 5, 0, 3, 5, 2, 3, 7, 9, 8, 1, 6, 3, 0, 0, 8,
       6, 1, 4, 3, 8, 9, 1, 4, 8, 1, 6, 6, 4, 0, 5, 3, 4, 1, 9, 3, 2, 5,
       2, 4, 2, 8, 2, 5, 7, 8, 0, 0, 1, 9, 2, 2, 6, 9, 1, 3, 2, 6, 0, 9,
       7, 6, 7, 4, 9, 0, 3, 4, 7, 0, 3, 4, 8, 2, 7, 7, 8, 5, 0, 1, 3, 4,
       9, 1, 7, 0, 4, 7, 6, 6, 9, 7, 4, 4, 7, 9, 4, 0, 6, 1, 0, 6, 6, 0,
       4, 9, 6, 4, 8, 2, 5, 5, 5, 8, 6, 4, 0, 9, 3, 6, 5, 7, 0, 9, 9, 0,
       3, 5, 1, 8, 0, 2, 2, 1, 6, 3, 7, 7, 3, 4, 5, 2, 2, 9, 3, 6, 8, 8,
       8, 7, 1, 2, 5, 7, 0, 0, 1, 2, 5, 8, 3, 2, 1, 0, 7, 5, 7, 6, 7, 6,
       8, 8, 0, 8, 6, 1, 8, 5, 3, 6, 3, 6, 4, 6, 5, 4, 5, 5, 0, 1, 8, 6,
       4, 7, 8, 1, 5, 6, 4, 8, 0, 9, 9, 5, 6, 2, 0, 0, 3, 2, 8, 6, 8, 6,
       1, 4, 1, 9, 9, 3, 5, 0, 6, 4, 3, 6, 0, 0, 8, 0, 8, 1, 9, 0, 9, 9,
       1, 3, 8, 4, 8, 2, 1, 2, 2, 0, 1, 7, 4, 4, 3, 3, 7, 6, 0, 1, 7, 2,
       5, 1, 9, 1, 2, 5, 5, 7, 8, 3, 4, 9, 4, 9, 8, 5, 7, 4, 0, 8, 1, 5,
       3, 1, 3, 8, 9, 4, 5, 7, 9, 2, 3, 5, 6, 5, 8, 0, 5, 3, 8, 6, 1, 9,
       7, 2, 5, 2, 0, 6, 1, 6, 0, 4, 7, 5, 6, 4, 7, 8, 2, 2, 4, 9, 2, 9,
       9, 1, 1, 7, 9, 1, 6, 2, 4, 8, 0, 9, 7, 9, 9, 6, 1, 8, 6, 4, 1, 8,
       0, 0, 1, 9, 9, 0, 9, 2, 6, 3, 4, 1, 2, 1, 4, 2, 8, 7, 7, 9, 0, 1,
       0, 2, 1, 4, 0, 8, 4, 7, 7, 4, 6, 1, 7, 5, 4, 9, 3, 7, 5, 5, 7, 8,
       4, 3, 4, 4, 8], dtype=int64)

use above line for confusion matrix because Y-label is having multiclass labels for each observations

In [22]:
confusion_matrix(np.argmax(y_test,axis=1),y_pred)
Out[22]:
array([[35,  0,  3,  0,  0,  4,  2,  0,  0,  1],
       [ 1, 40,  3,  0,  0,  0,  0,  0,  0,  0],
       [ 1,  0, 31,  1,  1,  1,  0,  0,  0,  0],
       [ 0,  0,  1, 30,  1,  0,  0,  0,  0,  0],
       [ 0,  1,  0,  0, 40,  0,  2,  2,  0,  0],
       [ 0,  2,  0,  0,  0, 33,  1,  0,  0,  0],
       [ 3,  1,  0,  0,  6,  0, 31,  1,  0,  0],
       [ 0,  1,  1,  0,  1,  1,  0, 33,  2,  0],
       [ 0,  0,  0,  0,  0,  0,  0,  0, 42,  0],
       [ 1,  1,  2,  1,  0,  0,  0,  0,  1, 35]], dtype=int64)

Plot confusion matrix using heatmap

In [23]:
sns.heatmap(confusion_matrix(np.argmax(y_test,axis=1),y_pred),annot=True,fmt="d",linewidths=1,linecolor="r",square=True)
Out[23]:
<matplotlib.axes._subplots.AxesSubplot at 0x210ee417b48>
Notebook Image

Lets take a look at some of the misclassified images. we will inspect them to understand if it was a tough one to predict or not.

In [35]:
y_test.size
Out[35]:
4010
In [49]:
x=(y_pred-y_test_label!=0).tolist()
x
x=[i for i,l in enumerate(x) if l!=False]
x[0]

Out[49]:
1
In [58]:
plt.figure(figsize=(15,18))
for i in range(25):
    plt.subplot(5,5,i+1)
    plt.imshow(X_test[x[i]][:,:,0],cmap="gray")
    plt.xlabel('Real {}, Predicted {}'.format(y_test_label[x[i]],y_pred[x[i]]))
plt.show()


Notebook Image

Conclusion!!!

If you see, all the above misclassified images are the only one which has bad prediction. We can make it better by adjusting the zoom level during Image generator function call
In [ ]: