Chapter 77

8. Neural networks and deep learning

NotebookPython 3 (ipykernel)75 cells

In [1]python · cell 1

python

%autosave 0

Output

Autosave disabled

8. Neural networks and deep learning

This week, we'll learn about neural nets and build a model for classifying images of clothes

8.1 Fashion classification

Dataset:

Links:

https://cs231n.github.io/

In [ ]python · cell 3

python

!git clone git@github.com:alexeygrigorev/clothing-dataset-small.git

8.2 TensorFlow and Keras

Installing TensorFlow
Loading images

In [2]python · cell 5

python

import numpy as np
import matplotlib.pyplot as plt

%matplotlib inline

In [3]python · cell 6

python

import tensorflow as tf
from tensorflow import keras

In [4]python · cell 7

python

from tensorflow.keras.preprocessing.image import load_img

In [10]python · cell 8

python

path = './clothing-dataset-small/train/t-shirt'
name = '5f0a3fa0-6a3d-4b68-b213-72766a643de7.jpg'
fullname = f'{path}/{name}'
load_img(fullname)

Output

<PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=400x533 at 0x7F6599CE5208>

[省略较大 image/png 输出]

In [11]python · cell 9

python

img = load_img(fullname, target_size=(299, 299))

In [12]python · cell 10

python

x = np.array(img)
x.shape

Output

(299, 299, 3)

8.3 Pre-trained convolutional neural networks

Imagenet dataset: https://www.image-net.org/
Pre-trained models: https://keras.io/api/applications/

In [5]python · cell 12

python

from tensorflow.keras.applications.xception import Xception
from tensorflow.keras.applications.xception import preprocess_input
from tensorflow.keras.applications.xception import decode_predictions

In [6]python · cell 13

python

model = Xception(weights='imagenet', input_shape=(299, 299, 3))

Output

Downloading data from https://github.com/fchollet/deep-learning-models/releases/download/v0.4/xception_weights_tf_dim_ordering_tf_kernels.h5
91889664/91884032 [==============================] - 2s 0us/step

In [18]python · cell 14

python

X = np.array([x])

In [19]python · cell 15

python

X.shape

Output

(1, 299, 299, 3)

In [24]python · cell 16

python

X = preprocess_input(X)

In [27]python · cell 17

python

pred = model.predict(X)

In [31]python · cell 18

python

decode_predictions(pred)

Output

Downloading data from https://storage.googleapis.com/download.tensorflow.org/data/imagenet_class_index.json
40960/35363 [==================================] - 0s 0us/step

[[('n03595614', 'jersey', 0.6792451),
  ('n02916936', 'bulletproof_vest', 0.039600316),
  ('n04370456', 'sweatshirt', 0.035299566),
  ('n03710637', 'maillot', 0.010884117),
  ('n04525038', 'velvet', 0.0018057569)]]

8.4 Convolutional neural networks

Types of layers: convolutional and dense
Convolutional layers and filters
Dense layers

There are more layers. Read here: https://cs231n.github.io/

8.5 Transfer learning

Reading data with ImageDataGenerator
Train Xception on smaller images (150x150)

(Better to run it with a GPU)

In [6]python · cell 21

python

from tensorflow.keras.preprocessing.image import ImageDataGenerator

In [8]python · cell 22

python

train_gen = ImageDataGenerator(preprocessing_function=preprocess_input)

train_ds = train_gen.flow_from_directory(
    './clothing-dataset-small/train',
    target_size=(150, 150),
    batch_size=32
)

Output

Found 3068 images belonging to 10 classes.

In [10]python · cell 23

python

train_ds.class_indices

Output

{'dress': 0,
 'hat': 1,
 'longsleeve': 2,
 'outwear': 3,
 'pants': 4,
 'shirt': 5,
 'shoes': 6,
 'shorts': 7,
 'skirt': 8,
 't-shirt': 9}

In [14]python · cell 24

python

X, y = next(train_ds)

In [18]python · cell 25

python

y[:5]

Output

array([[0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],
       [0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],
       [0., 0., 0., 0., 0., 1., 0., 0., 0., 0.],
       [0., 0., 0., 0., 1., 0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0., 0., 0., 0., 0., 1.]], dtype=float32)

In [9]python · cell 26

python

val_gen = ImageDataGenerator(preprocessing_function=preprocess_input)

val_ds = val_gen.flow_from_directory(
    './clothing-dataset-small/validation',
    target_size=(150, 150),
    batch_size=32,
    shuffle=False
)

Output

Found 341 images belonging to 10 classes.

In [10]python · cell 27

python

base_model = Xception(
    weights='imagenet',
    include_top=False,
    input_shape=(150, 150, 3)
)

base_model.trainable = False



inputs = keras.Input(shape=(150, 150, 3))

base = base_model(inputs, training=False)

vectors = keras.layers.GlobalAveragePooling2D()(base)

outputs = keras.layers.Dense(10)(vectors)

model = keras.Model(inputs, outputs)

Output

Downloading data from https://github.com/fchollet/deep-learning-models/releases/download/v0.4/xception_weights_tf_dim_ordering_tf_kernels_notop.h5
83689472/83683744 [==============================] - 1s 0us/step

In [50]python · cell 28

python

learning_rate = 0.01
optimizer = keras.optimizers.Adam(learning_rate=learning_rate)

loss = keras.losses.CategoricalCrossentropy(from_logits=True)

model.compile(optimizer=optimizer, loss=loss, metrics=['accuracy'])

In [51]python · cell 29

python

history = model.fit(train_ds, epochs=10, validation_data=val_ds)

Output

WARNING:tensorflow:sample_weight modes were coerced from
  ...
    to  
  ['...']
WARNING:tensorflow:sample_weight modes were coerced from
  ...
    to  
  ['...']
Train for 96 steps, validate for 11 steps
Epoch 1/10
96/96 [==============================] - 20s 211ms/step - loss: 1.2568 - accuracy: 0.6698 - val_loss: 0.8537 - val_accuracy: 0.7449
Epoch 2/10
96/96 [==============================] - 16s 166ms/step - loss: 0.5745 - accuracy: 0.8188 - val_loss: 0.6850 - val_accuracy: 0.8065
Epoch 3/10
96/96 [==============================] - 16s 166ms/step - loss: 0.3261 - accuracy: 0.8885 - val_loss: 0.7734 - val_accuracy: 0.8152
Epoch 4/10
96/96 [==============================] - 16s 166ms/step - loss: 0.2215 - accuracy: 0.9244 - val_loss: 0.9123 - val_accuracy: 0.7918
Epoch 5/10
96/96 [==============================] - 16s 166ms/step - loss: 0.1433 - accuracy: 0.9475 - val_loss: 0.8244 - val_accuracy: 0.8094
Epoch 6/10
96/96 [==============================] - 16s 167ms/step - loss: 0.0897 - accuracy: 0.9707 - val_loss: 0.8517 - val_accuracy: 0.8035
Epoch 7/10
96/96 [==============================] - 16s 166ms/step - loss: 0.0855 - accuracy: 0.9703 - val_loss: 0.7886 - val_accuracy: 0.8299
Epoch 8/10
96/96 [==============================] - 16s 167ms/step - loss: 0.0907 - accuracy: 0.9661 - val_loss: 0.8840 - val_accuracy: 0.8123
Epoch 9/10
96/96 [==============================] - 16s 167ms/step - loss: 0.0458 - accuracy: 0.9876 - val_loss: 0.9531 - val_accuracy: 0.8035
Epoch 10/10
96/96 [==============================] - 16s 166ms/step - loss: 0.0465 - accuracy: 0.9876 - val_loss: 0.9695 - val_accuracy: 0.7889

In [59]python · cell 30

python

#plt.plot(history.history['accuracy'], label='train')
plt.plot(history.history['val_accuracy'], label='val')
plt.xticks(np.arange(10))
plt.legend()

Output

<matplotlib.legend.Legend at 0x7fe20fc3c860>

<Figure size 432x288 with 1 Axes>

8.6 Adjusting the learning rate

What's the learning rate
Trying different values

In [60]python · cell 32

python

def make_model(learning_rate=0.01):
    base_model = Xception(
        weights='imagenet',
        include_top=False,
        input_shape=(150, 150, 3)
    )

    base_model.trainable = False

    #########################################

    inputs = keras.Input(shape=(150, 150, 3))
    base = base_model(inputs, training=False)
    vectors = keras.layers.GlobalAveragePooling2D()(base)
    outputs = keras.layers.Dense(10)(vectors)
    model = keras.Model(inputs, outputs)
    
    #########################################

    optimizer = keras.optimizers.Adam(learning_rate=learning_rate)
    loss = keras.losses.CategoricalCrossentropy(from_logits=True)

    model.compile(
        optimizer=optimizer,
        loss=loss,
        metrics=['accuracy']
    )
    
    return model

In [61]python · cell 33

python

scores = {}

for lr in [0.0001, 0.001, 0.01, 0.1]:
    print(lr)

    model = make_model(learning_rate=lr)
    history = model.fit(train_ds, epochs=10, validation_data=val_ds)
    scores[lr] = history.history

    print()
    print()

Output

0.0001
WARNING:tensorflow:sample_weight modes were coerced from
  ...
    to  
  ['...']
WARNING:tensorflow:sample_weight modes were coerced from
  ...
    to  
  ['...']
Train for 96 steps, validate for 11 steps
Epoch 1/10
96/96 [==============================] - 20s 208ms/step - loss: 1.9262 - accuracy: 0.3403 - val_loss: 1.5806 - val_accuracy: 0.4897
Epoch 2/10
96/96 [==============================] - 16s 166ms/step - loss: 1.3832 - accuracy: 0.5613 - val_loss: 1.2368 - val_accuracy: 0.6012
Epoch 3/10
96/96 [==============================] - 16s 166ms/step - loss: 1.1397 - accuracy: 0.6463 - val_loss: 1.0575 - val_accuracy: 0.6716
Epoch 4/10
96/96 [==============================] - 16s 166ms/step - loss: 1.0022 - accuracy: 0.6858 - val_loss: 0.9497 - val_accuracy: 0.7214
Epoch 5/10
96/96 [==============================] - 16s 166ms/step - loss: 0.9087 - accuracy: 0.7141 - val_loss: 0.8773 - val_accuracy: 0.7566
Epoch 6/10
96/96 [==============================] - 16s 166ms/step - loss: 0.8401 - accuracy: 0.7376 - val_loss: 0.8281 - val_accuracy: 0.7713
Epoch 7/10
96/96 [==============================] - 16s 166ms/step - loss: 0.7886 - accuracy: 0.7565 - val_loss: 0.7873 - val_accuracy: 0.7801
Epoch 8/10
96/96 [==============================] - 16s 167ms/step - loss: 0.7453 - accuracy: 0.7637 - val_loss: 0.7537 - val_accuracy: 0.7830
Epoch 9/10
96/96 [==============================] - 16s 167ms/step - loss: 0.7094 - accuracy: 0.7757 - val_loss: 0.7299 - val_accuracy: 0.7859
Epoch 10/10
96/96 [==============================] - 16s 167ms/step - loss: 0.6791 - accuracy: 0.7806 - val_loss: 0.7060 - val_accuracy: 0.7889


0.001
WARNING:tensorflow:sample_weight modes were coerced from
  ...
    to  
  ['...']
WARNING:tensorflow:sample_weight modes were coerced from
  ...
    to  
  ['...']
Train for 96 steps, validate for 11 steps
Epoch 1/10
96/96 [==============================] - 20s 204ms/step - loss: 1.1293 - accuracy: 0.6199 - val_loss: 0.7179 - val_accuracy: 0.7625
Epoch 2/10
96/96 [==============================] - 16s 166ms/step - loss: 0.6555 - accuracy: 0.7761 - val_loss: 0.6018 - val_accuracy: 0.8035
Epoch 3/10
96/96 [==============================] - 16s 166ms/step - loss: 0.5239 - accuracy: 0.8230 - val_loss: 0.5708 - val_accuracy: 0.8094
Epoch 4/10
96/96 [==============================] - 16s 166ms/step - loss: 0.4409 - accuracy: 0.8553 - val_loss: 0.5574 - val_accuracy: 0.8035
Epoch 5/10
96/96 [==============================] - 16s 167ms/step - loss: 0.3796 - accuracy: 0.8820 - val_loss: 0.5373 - val_accuracy: 0.8152
Epoch 6/10
96/96 [==============================] - 16s 167ms/step - loss: 0.3307 - accuracy: 0.9029 - val_loss: 0.5273 - val_accuracy: 0.8270
Epoch 7/10
96/96 [==============================] - 16s 167ms/step - loss: 0.2973 - accuracy: 0.9153 - val_loss: 0.5111 - val_accuracy: 0.8152
Epoch 8/10
96/96 [==============================] - 16s 167ms/step - loss: 0.2657 - accuracy: 0.9316 - val_loss: 0.5245 - val_accuracy: 0.8006
Epoch 9/10
96/96 [==============================] - 16s 168ms/step - loss: 0.2394 - accuracy: 0.9426 - val_loss: 0.5233 - val_accuracy: 0.8299
Epoch 10/10
96/96 [==============================] - 16s 168ms/step - loss: 0.2137 - accuracy: 0.9505 - val_loss: 0.5017 - val_accuracy: 0.8240


0.01
WARNING:tensorflow:sample_weight modes were coerced from
  ...
    to  
  ['...']
WARNING:tensorflow:sample_weight modes were coerced from
  ...
    to  
  ['...']
Train for 96 steps, validate for 11 steps
Epoch 1/10
96/96 [==============================] - 20s 209ms/step - loss: 1.1734 - accuracy: 0.6701 - val_loss: 1.0102 - val_accuracy: 0.7390
Epoch 2/10
96/96 [==============================] - 16s 166ms/step - loss: 0.5969 - accuracy: 0.8129 - val_loss: 0.8572 - val_accuracy: 0.7771
Epoch 3/10
96/96 [==============================] - 16s 167ms/step - loss: 0.3137 - accuracy: 0.8902 - val_loss: 0.7792 - val_accuracy: 0.7947
Epoch 4/10
96/96 [==============================] - 16s 166ms/step - loss: 0.2034 - accuracy: 0.9273 - val_loss: 0.9025 - val_accuracy: 0.7859
Epoch 5/10
96/96 [==============================] - 16s 166ms/step - loss: 0.1677 - accuracy: 0.9423 - val_loss: 0.9980 - val_accuracy: 0.7713
Epoch 6/10
96/96 [==============================] - 16s 166ms/step - loss: 0.2104 - accuracy: 0.9205 - val_loss: 1.0246 - val_accuracy: 0.8123
Epoch 7/10
96/96 [==============================] - 16s 166ms/step - loss: 0.1228 - accuracy: 0.9615 - val_loss: 0.9585 - val_accuracy: 0.8094
Epoch 8/10
96/96 [==============================] - 16s 167ms/step - loss: 0.0779 - accuracy: 0.9743 - val_loss: 0.8759 - val_accuracy: 0.8123
Epoch 9/10
96/96 [==============================] - 16s 166ms/step - loss: 0.0651 - accuracy: 0.9791 - val_loss: 0.8844 - val_accuracy: 0.8211
Epoch 10/10
96/96 [==============================] - 16s 167ms/step - loss: 0.0619 - accuracy: 0.9817 - val_loss: 1.0551 - val_accuracy: 0.7889


0.1
WARNING:tensorflow:sample_weight modes were coerced from
  ...
    to  
  ['...']
WARNING:tensorflow:sample_weight modes were coerced from
  ...
    to  
  ['...']
Train for 96 steps, validate for 11 steps
Epoch 1/10
96/96 [==============================] - 20s 203ms/step - loss: 10.4920 - accuracy: 0.6327 - val_loss: 6.0144 - val_accuracy: 0.7537
Epoch 2/10
96/96 [==============================] - 16s 166ms/step - loss: 5.2193 - accuracy: 0.7744 - val_loss: 7.3142 - val_accuracy: 0.7566
Epoch 3/10
96/96 [==============================] - 16s 166ms/step - loss: 3.4999 - accuracy: 0.8387 - val_loss: 8.0551 - val_accuracy: 0.7683
Epoch 4/10
96/96 [==============================] - 16s 166ms/step - loss: 2.9493 - accuracy: 0.8625 - val_loss: 9.5917 - val_accuracy: 0.7507
Epoch 5/10
96/96 [==============================] - 16s 166ms/step - loss: 2.4901 - accuracy: 0.8814 - val_loss: 9.1683 - val_accuracy: 0.7507
Epoch 6/10
96/96 [==============================] - 16s 166ms/step - loss: 2.1291 - accuracy: 0.9016 - val_loss: 9.2850 - val_accuracy: 0.7654
Epoch 7/10
96/96 [==============================] - 16s 166ms/step - loss: 1.4794 - accuracy: 0.9185 - val_loss: 10.4883 - val_accuracy: 0.7566
Epoch 8/10
96/96 [==============================] - 16s 166ms/step - loss: 1.4156 - accuracy: 0.9211 - val_loss: 11.8833 - val_accuracy: 0.7742
Epoch 9/10
96/96 [==============================] - 16s 167ms/step - loss: 1.1350 - accuracy: 0.9400 - val_loss: 10.6944 - val_accuracy: 0.7625
Epoch 10/10
96/96 [==============================] - 16s 167ms/step - loss: 1.1251 - accuracy: 0.9475 - val_loss: 8.6116 - val_accuracy: 0.7889

In [65]python · cell 34

python

del scores[0.1]
del scores[0.0001]

In [68]python · cell 35

python

for lr, hist in scores.items():
    #plt.plot(hist['accuracy'], label=('train=%s' % lr))
    plt.plot(hist['val_accuracy'], label=('val=%s' % lr))

plt.xticks(np.arange(10))
plt.legend()

Output

<matplotlib.legend.Legend at 0x7fe0acceac88>

<Figure size 432x288 with 1 Axes>

In [ ]python · cell 36

python

learning_rate = 0.001

8.7 Checkpointing

Saving the best model only
Training a model with callbacks

In [70]python · cell 38

python

model.save_weights('model_v1.h5', save_format='h5')

In [21]python · cell 39

python

chechpoint = keras.callbacks.ModelCheckpoint(
    'xception_v1_{epoch:02d}_{val_accuracy:.3f}.h5',
    save_best_only=True,
    monitor='val_accuracy',
    mode='max'
)

In [79]python · cell 40

python

learning_rate = 0.001

model = make_model(learning_rate=learning_rate)

history = model.fit(
    train_ds,
    epochs=10,
    validation_data=val_ds,
    callbacks=[chechpoint]
)

Output

WARNING:tensorflow:sample_weight modes were coerced from
  ...
    to  
  ['...']
WARNING:tensorflow:sample_weight modes were coerced from
  ...
    to  
  ['...']
Train for 96 steps, validate for 11 steps
Epoch 1/10
96/96 [==============================] - 21s 216ms/step - loss: 1.1353 - accuracy: 0.6170 - val_loss: 0.7258 - val_accuracy: 0.7801
Epoch 2/10
96/96 [==============================] - 16s 168ms/step - loss: 0.6469 - accuracy: 0.7735 - val_loss: 0.6332 - val_accuracy: 0.7859
Epoch 3/10
96/96 [==============================] - 16s 169ms/step - loss: 0.5182 - accuracy: 0.8243 - val_loss: 0.5905 - val_accuracy: 0.8094
Epoch 4/10
96/96 [==============================] - 16s 170ms/step - loss: 0.4390 - accuracy: 0.8553 - val_loss: 0.5550 - val_accuracy: 0.8152
Epoch 5/10
96/96 [==============================] - 16s 170ms/step - loss: 0.3827 - accuracy: 0.8827 - val_loss: 0.5437 - val_accuracy: 0.8211
Epoch 6/10
96/96 [==============================] - 16s 170ms/step - loss: 0.3342 - accuracy: 0.8990 - val_loss: 0.5319 - val_accuracy: 0.8358
Epoch 7/10
96/96 [==============================] - 16s 166ms/step - loss: 0.2978 - accuracy: 0.9149 - val_loss: 0.5169 - val_accuracy: 0.8299
Epoch 8/10
96/96 [==============================] - 16s 166ms/step - loss: 0.2652 - accuracy: 0.9283 - val_loss: 0.5422 - val_accuracy: 0.8299
Epoch 9/10
96/96 [==============================] - 16s 166ms/step - loss: 0.2419 - accuracy: 0.9387 - val_loss: 0.5234 - val_accuracy: 0.8211
Epoch 10/10
96/96 [==============================] - 16s 166ms/step - loss: 0.2157 - accuracy: 0.9498 - val_loss: 0.5320 - val_accuracy: 0.8328

8.8 Adding more layers

Adding one inner dense layer
Experimenting with different sizes of inner layer

In [3]python · cell 42

python

def make_model(learning_rate=0.01, size_inner=100):
    base_model = Xception(
        weights='imagenet',
        include_top=False,
        input_shape=(150, 150, 3)
    )

    base_model.trainable = False

    #########################################

    inputs = keras.Input(shape=(150, 150, 3))
    base = base_model(inputs, training=False)
    vectors = keras.layers.GlobalAveragePooling2D()(base)
    
    inner = keras.layers.Dense(size_inner, activation='relu')(vectors)
    
    outputs = keras.layers.Dense(10)(inner)
    
    model = keras.Model(inputs, outputs)
    
    #########################################

    optimizer = keras.optimizers.Adam(learning_rate=learning_rate)
    loss = keras.losses.CategoricalCrossentropy(from_logits=True)

    model.compile(
        optimizer=optimizer,
        loss=loss,
        metrics=['accuracy']
    )
    
    return model

In [13]python · cell 43

python

learning_rate = 0.001

scores = {}

for size in [10, 100, 1000]:
    print(size)

    model = make_model(learning_rate=learning_rate, size_inner=size)
    history = model.fit(train_ds, epochs=10, validation_data=val_ds)
    scores[size] = history.history

    print()
    print()

Output

10
Downloading data from https://github.com/fchollet/deep-learning-models/releases/download/v0.4/xception_weights_tf_dim_ordering_tf_kernels_notop.h5
83689472/83683744 [==============================] - 1s 0us/step
[2021-11-03 13:35:43.005 ip-172-16-64-149:21634 INFO utils.py:27] RULE_JOB_STOP_SIGNAL_FILENAME: None
[2021-11-03 13:35:43.104 ip-172-16-64-149:21634 INFO profiler_config_parser.py:111] Unable to find config at /opt/ml/input/config/profilerconfig.json. Profiler is disabled.
WARNING:tensorflow:sample_weight modes were coerced from
  ...
    to  
  ['...']
WARNING:tensorflow:sample_weight modes were coerced from
  ...
    to  
  ['...']
Train for 96 steps, validate for 11 steps
Epoch 1/10
96/96 [==============================] - 42s 438ms/step - loss: 1.1849 - accuracy: 0.6033 - val_loss: 0.8076 - val_accuracy: 0.7243
Epoch 2/10
96/96 [==============================] - 16s 169ms/step - loss: 0.7314 - accuracy: 0.7474 - val_loss: 0.7305 - val_accuracy: 0.7507
Epoch 3/10
96/96 [==============================] - 16s 169ms/step - loss: 0.5834 - accuracy: 0.8018 - val_loss: 0.6308 - val_accuracy: 0.7889
Epoch 4/10
96/96 [==============================] - 16s 170ms/step - loss: 0.4878 - accuracy: 0.8370 - val_loss: 0.6715 - val_accuracy: 0.7859
Epoch 5/10
96/96 [==============================] - 16s 170ms/step - loss: 0.4128 - accuracy: 0.8647 - val_loss: 0.5720 - val_accuracy: 0.8094
Epoch 6/10
96/96 [==============================] - 16s 170ms/step - loss: 0.3512 - accuracy: 0.8980 - val_loss: 0.5757 - val_accuracy: 0.8152
Epoch 7/10
96/96 [==============================] - 16s 171ms/step - loss: 0.3043 - accuracy: 0.9091 - val_loss: 0.5484 - val_accuracy: 0.8152
Epoch 8/10
96/96 [==============================] - 16s 172ms/step - loss: 0.2564 - accuracy: 0.9306 - val_loss: 0.5492 - val_accuracy: 0.7918
Epoch 9/10
96/96 [==============================] - 16s 171ms/step - loss: 0.2220 - accuracy: 0.9459 - val_loss: 0.5523 - val_accuracy: 0.7977
Epoch 10/10
96/96 [==============================] - 16s 172ms/step - loss: 0.2006 - accuracy: 0.9462 - val_loss: 0.5594 - val_accuracy: 0.8152


100
WARNING:tensorflow:sample_weight modes were coerced from
  ...
    to  
  ['...']
WARNING:tensorflow:sample_weight modes were coerced from
  ...
    to  
  ['...']
Train for 96 steps, validate for 11 steps
Epoch 1/10
96/96 [==============================] - 20s 208ms/step - loss: 0.9623 - accuracy: 0.6819 - val_loss: 0.6446 - val_accuracy: 0.7801
Epoch 2/10
96/96 [==============================] - 17s 173ms/step - loss: 0.4784 - accuracy: 0.8341 - val_loss: 0.6494 - val_accuracy: 0.7918
Epoch 3/10
96/96 [==============================] - 17s 172ms/step - loss: 0.3548 - accuracy: 0.8752 - val_loss: 0.6583 - val_accuracy: 0.7830
Epoch 4/10
96/96 [==============================] - 17s 172ms/step - loss: 0.2430 - accuracy: 0.9166 - val_loss: 0.6221 - val_accuracy: 0.8006
Epoch 5/10
96/96 [==============================] - 17s 172ms/step - loss: 0.1873 - accuracy: 0.9413 - val_loss: 0.6345 - val_accuracy: 0.8035
Epoch 6/10
96/96 [==============================] - 17s 172ms/step - loss: 0.1431 - accuracy: 0.9566 - val_loss: 0.6523 - val_accuracy: 0.8065
Epoch 7/10
96/96 [==============================] - 16s 172ms/step - loss: 0.1153 - accuracy: 0.9700 - val_loss: 0.8347 - val_accuracy: 0.7713
Epoch 8/10
96/96 [==============================] - 17s 172ms/step - loss: 0.0802 - accuracy: 0.9795 - val_loss: 0.7221 - val_accuracy: 0.7859
Epoch 9/10
96/96 [==============================] - 17s 172ms/step - loss: 0.0647 - accuracy: 0.9870 - val_loss: 0.7199 - val_accuracy: 0.8152
Epoch 10/10
96/96 [==============================] - 17s 172ms/step - loss: 0.0518 - accuracy: 0.9922 - val_loss: 0.7012 - val_accuracy: 0.8123


1000
WARNING:tensorflow:sample_weight modes were coerced from
  ...
    to  
  ['...']
WARNING:tensorflow:sample_weight modes were coerced from
  ...
    to  
  ['...']
Train for 96 steps, validate for 11 steps
Epoch 1/10
96/96 [==============================] - 20s 208ms/step - loss: 1.5152 - accuracy: 0.6633 - val_loss: 0.9100 - val_accuracy: 0.7771
Epoch 2/10
96/96 [==============================] - 16s 172ms/step - loss: 0.5736 - accuracy: 0.8237 - val_loss: 0.7562 - val_accuracy: 0.7830
Epoch 3/10
96/96 [==============================] - 16s 171ms/step - loss: 0.4004 - accuracy: 0.8703 - val_loss: 0.9595 - val_accuracy: 0.7507
Epoch 4/10
96/96 [==============================] - 16s 171ms/step - loss: 0.2576 - accuracy: 0.9065 - val_loss: 0.7193 - val_accuracy: 0.8065
Epoch 5/10
96/96 [==============================] - 16s 171ms/step - loss: 0.1669 - accuracy: 0.9410 - val_loss: 0.8278 - val_accuracy: 0.7977
Epoch 6/10
96/96 [==============================] - 17s 174ms/step - loss: 0.1255 - accuracy: 0.9563 - val_loss: 0.8112 - val_accuracy: 0.8035
Epoch 7/10
96/96 [==============================] - 16s 171ms/step - loss: 0.1177 - accuracy: 0.9596 - val_loss: 0.8148 - val_accuracy: 0.8123
Epoch 8/10
96/96 [==============================] - 16s 171ms/step - loss: 0.1825 - accuracy: 0.9342 - val_loss: 0.9044 - val_accuracy: 0.7889
Epoch 9/10
96/96 [==============================] - 16s 171ms/step - loss: 0.1303 - accuracy: 0.9537 - val_loss: 1.0076 - val_accuracy: 0.7859
Epoch 10/10
96/96 [==============================] - 16s 172ms/step - loss: 0.0891 - accuracy: 0.9694 - val_loss: 1.1218 - val_accuracy: 0.8006

In [16]python · cell 44

python

for size, hist in scores.items():
    plt.plot(hist['val_accuracy'], label=('val=%s' % size))

plt.xticks(np.arange(10))
plt.yticks([0.78, 0.80, 0.82, 0.825, 0.83])
plt.legend()

Output

<matplotlib.legend.Legend at 0x7f03ada03b00>

<Figure size 432x288 with 1 Axes>

8.9 Regularization and dropout

Regularizing by freezing a part of the network
Adding dropout to our model
Experimenting with different values

In [8]python · cell 46

python

def make_model(learning_rate=0.01, size_inner=100, droprate=0.5):
    base_model = Xception(
        weights='imagenet',
        include_top=False,
        input_shape=(150, 150, 3)
    )

    base_model.trainable = False

    #########################################

    inputs = keras.Input(shape=(150, 150, 3))
    base = base_model(inputs, training=False)
    vectors = keras.layers.GlobalAveragePooling2D()(base)
    
    inner = keras.layers.Dense(size_inner, activation='relu')(vectors)
    drop = keras.layers.Dropout(droprate)(inner)
    
    outputs = keras.layers.Dense(10)(drop)
    
    model = keras.Model(inputs, outputs)
    
    #########################################

    optimizer = keras.optimizers.Adam(learning_rate=learning_rate)
    loss = keras.losses.CategoricalCrossentropy(from_logits=True)

    model.compile(
        optimizer=optimizer,
        loss=loss,
        metrics=['accuracy']
    )
    
    return model

In [13]python · cell 47

python

learning_rate = 0.001
size = 100

scores = {}

for droprate in [0.0, 0.2, 0.5, 0.8]:
    print(droprate)

    model = make_model(
        learning_rate=learning_rate,
        size_inner=size,
        droprate=droprate
    )

    history = model.fit(train_ds, epochs=30, validation_data=val_ds)
    scores[droprate] = history.history

    print()
    print()

Output

0.0
[2021-11-03 21:19:20.707 ip-172-16-13-140:18999 INFO utils.py:27] RULE_JOB_STOP_SIGNAL_FILENAME: None
[2021-11-03 21:19:20.814 ip-172-16-13-140:18999 INFO profiler_config_parser.py:111] Unable to find config at /opt/ml/input/config/profilerconfig.json. Profiler is disabled.
WARNING:tensorflow:sample_weight modes were coerced from
  ...
    to  
  ['...']
WARNING:tensorflow:sample_weight modes were coerced from
  ...
    to  
  ['...']
Train for 96 steps, validate for 11 steps
Epoch 1/30
96/96 [==============================] - 56s 583ms/step - loss: 0.9642 - accuracy: 0.6747 - val_loss: 0.6832 - val_accuracy: 0.7566
Epoch 2/30
96/96 [==============================] - 15s 158ms/step - loss: 0.5163 - accuracy: 0.8230 - val_loss: 0.6058 - val_accuracy: 0.7918
Epoch 3/30
96/96 [==============================] - 15s 160ms/step - loss: 0.3458 - accuracy: 0.8885 - val_loss: 0.5577 - val_accuracy: 0.8270
Epoch 4/30
96/96 [==============================] - 15s 161ms/step - loss: 0.2575 - accuracy: 0.9218 - val_loss: 0.5166 - val_accuracy: 0.8416
Epoch 5/30
96/96 [==============================] - 15s 157ms/step - loss: 0.2002 - accuracy: 0.9394 - val_loss: 0.5974 - val_accuracy: 0.8152
Epoch 6/30
96/96 [==============================] - 15s 159ms/step - loss: 0.1322 - accuracy: 0.9684 - val_loss: 0.5695 - val_accuracy: 0.8152
Epoch 7/30
96/96 [==============================] - 15s 158ms/step - loss: 0.0874 - accuracy: 0.9863 - val_loss: 0.6004 - val_accuracy: 0.8152
Epoch 8/30
96/96 [==============================] - 15s 160ms/step - loss: 0.0674 - accuracy: 0.9902 - val_loss: 0.5810 - val_accuracy: 0.8211
Epoch 9/30
96/96 [==============================] - 15s 157ms/step - loss: 0.0485 - accuracy: 0.9971 - val_loss: 0.6084 - val_accuracy: 0.8270
Epoch 10/30
96/96 [==============================] - 15s 158ms/step - loss: 0.0326 - accuracy: 0.9987 - val_loss: 0.6066 - val_accuracy: 0.8416
Epoch 11/30
96/96 [==============================] - 15s 158ms/step - loss: 0.0235 - accuracy: 0.9990 - val_loss: 0.6404 - val_accuracy: 0.8152
Epoch 12/30
96/96 [==============================] - 15s 159ms/step - loss: 0.0229 - accuracy: 0.9990 - val_loss: 0.6242 - val_accuracy: 0.8270
Epoch 13/30
96/96 [==============================] - 15s 158ms/step - loss: 0.0171 - accuracy: 0.9997 - val_loss: 0.6530 - val_accuracy: 0.8328
Epoch 14/30
96/96 [==============================] - 15s 157ms/step - loss: 0.0130 - accuracy: 0.9997 - val_loss: 0.6650 - val_accuracy: 0.8358
Epoch 15/30
96/96 [==============================] - 15s 160ms/step - loss: 0.0138 - accuracy: 0.9993 - val_loss: 0.6573 - val_accuracy: 0.8328
Epoch 16/30
96/96 [==============================] - 15s 158ms/step - loss: 0.0117 - accuracy: 0.9993 - val_loss: 0.7331 - val_accuracy: 0.8152
Epoch 17/30
96/96 [==============================] - 15s 161ms/step - loss: 0.0121 - accuracy: 0.9997 - val_loss: 0.6736 - val_accuracy: 0.8270
Epoch 18/30
96/96 [==============================] - 15s 157ms/step - loss: 0.0105 - accuracy: 0.9993 - val_loss: 0.6732 - val_accuracy: 0.8299
Epoch 19/30
96/96 [==============================] - 15s 157ms/step - loss: 0.0065 - accuracy: 0.9997 - val_loss: 0.7275 - val_accuracy: 0.8240
Epoch 20/30
96/96 [==============================] - 15s 158ms/step - loss: 0.0087 - accuracy: 0.9997 - val_loss: 0.6873 - val_accuracy: 0.8328
Epoch 21/30
96/96 [==============================] - 15s 158ms/step - loss: 0.0074 - accuracy: 0.9993 - val_loss: 0.7036 - val_accuracy: 0.8387
Epoch 22/30
96/96 [==============================] - 16s 162ms/step - loss: 0.0048 - accuracy: 0.9997 - val_loss: 0.7190 - val_accuracy: 0.8358
Epoch 23/30
96/96 [==============================] - 15s 159ms/step - loss: 0.0062 - accuracy: 0.9997 - val_loss: 0.7420 - val_accuracy: 0.8240
Epoch 24/30
96/96 [==============================] - 15s 158ms/step - loss: 0.0089 - accuracy: 0.9990 - val_loss: 0.7259 - val_accuracy: 0.8358
Epoch 25/30
96/96 [==============================] - 15s 159ms/step - loss: 0.0092 - accuracy: 0.9987 - val_loss: 0.7479 - val_accuracy: 0.8299
Epoch 26/30
96/96 [==============================] - 16s 163ms/step - loss: 0.0033 - accuracy: 0.9997 - val_loss: 0.7662 - val_accuracy: 0.8358
Epoch 27/30
96/96 [==============================] - 16s 165ms/step - loss: 0.0097 - accuracy: 0.9987 - val_loss: 0.8429 - val_accuracy: 0.8211
Epoch 28/30
96/96 [==============================] - 15s 159ms/step - loss: 0.0072 - accuracy: 0.9990 - val_loss: 0.8050 - val_accuracy: 0.8358
Epoch 29/30
96/96 [==============================] - 15s 159ms/step - loss: 0.0125 - accuracy: 0.9964 - val_loss: 0.8644 - val_accuracy: 0.8152
Epoch 30/30
96/96 [==============================] - 15s 157ms/step - loss: 0.0091 - accuracy: 0.9987 - val_loss: 0.8397 - val_accuracy: 0.8358


0.2
WARNING:tensorflow:sample_weight modes were coerced from
  ...
    to  
  ['...']
WARNING:tensorflow:sample_weight modes were coerced from
  ...
    to  
  ['...']
Train for 96 steps, validate for 11 steps
Epoch 1/30
96/96 [==============================] - 19s 201ms/step - loss: 1.0421 - accuracy: 0.6421 - val_loss: 0.6269 - val_accuracy: 0.8152
Epoch 2/30
96/96 [==============================] - 15s 157ms/step - loss: 0.5996 - accuracy: 0.7862 - val_loss: 0.5930 - val_accuracy: 0.7830
Epoch 3/30
96/96 [==============================] - 15s 159ms/step - loss: 0.4595 - accuracy: 0.8390 - val_loss: 0.5409 - val_accuracy: 0.8152
Epoch 4/30
96/96 [==============================] - 15s 159ms/step - loss: 0.3463 - accuracy: 0.8866 - val_loss: 0.4990 - val_accuracy: 0.8182
Epoch 5/30
96/96 [==============================] - 16s 162ms/step - loss: 0.2764 - accuracy: 0.9113 - val_loss: 0.4907 - val_accuracy: 0.8446
Epoch 6/30
96/96 [==============================] - 15s 160ms/step - loss: 0.2235 - accuracy: 0.9299 - val_loss: 0.5487 - val_accuracy: 0.8270
Epoch 7/30
96/96 [==============================] - 15s 159ms/step - loss: 0.1744 - accuracy: 0.9505 - val_loss: 0.5264 - val_accuracy: 0.8152
Epoch 8/30
96/96 [==============================] - 16s 162ms/step - loss: 0.1422 - accuracy: 0.9645 - val_loss: 0.5496 - val_accuracy: 0.8299
Epoch 9/30
96/96 [==============================] - 16s 162ms/step - loss: 0.1153 - accuracy: 0.9690 - val_loss: 0.5453 - val_accuracy: 0.8152
Epoch 10/30
96/96 [==============================] - 15s 159ms/step - loss: 0.0955 - accuracy: 0.9769 - val_loss: 0.5783 - val_accuracy: 0.8152
Epoch 11/30
96/96 [==============================] - 15s 160ms/step - loss: 0.0775 - accuracy: 0.9834 - val_loss: 0.5469 - val_accuracy: 0.8270
Epoch 12/30
96/96 [==============================] - 15s 158ms/step - loss: 0.0595 - accuracy: 0.9886 - val_loss: 0.5578 - val_accuracy: 0.8446
Epoch 13/30
96/96 [==============================] - 15s 159ms/step - loss: 0.0514 - accuracy: 0.9922 - val_loss: 0.6008 - val_accuracy: 0.8416
Epoch 14/30
96/96 [==============================] - 15s 161ms/step - loss: 0.0491 - accuracy: 0.9909 - val_loss: 0.5807 - val_accuracy: 0.8358
Epoch 15/30
96/96 [==============================] - 15s 160ms/step - loss: 0.0407 - accuracy: 0.9948 - val_loss: 0.6196 - val_accuracy: 0.8270
Epoch 16/30
96/96 [==============================] - 15s 159ms/step - loss: 0.0384 - accuracy: 0.9928 - val_loss: 0.6152 - val_accuracy: 0.8387
Epoch 17/30
96/96 [==============================] - 15s 158ms/step - loss: 0.0330 - accuracy: 0.9935 - val_loss: 0.5942 - val_accuracy: 0.8416
Epoch 18/30
96/96 [==============================] - 15s 158ms/step - loss: 0.0286 - accuracy: 0.9932 - val_loss: 0.7247 - val_accuracy: 0.8152
Epoch 19/30
96/96 [==============================] - 15s 160ms/step - loss: 0.0319 - accuracy: 0.9941 - val_loss: 0.6897 - val_accuracy: 0.8387
Epoch 20/30
96/96 [==============================] - 15s 160ms/step - loss: 0.0289 - accuracy: 0.9941 - val_loss: 0.6490 - val_accuracy: 0.8270
Epoch 21/30
96/96 [==============================] - 15s 161ms/step - loss: 0.0329 - accuracy: 0.9928 - val_loss: 0.7535 - val_accuracy: 0.8152
Epoch 22/30
96/96 [==============================] - 15s 160ms/step - loss: 0.0310 - accuracy: 0.9938 - val_loss: 0.6456 - val_accuracy: 0.8270
Epoch 23/30
96/96 [==============================] - 15s 161ms/step - loss: 0.0259 - accuracy: 0.9954 - val_loss: 0.7003 - val_accuracy: 0.8328
Epoch 24/30
96/96 [==============================] - 15s 159ms/step - loss: 0.0195 - accuracy: 0.9961 - val_loss: 0.7779 - val_accuracy: 0.8094
Epoch 25/30
96/96 [==============================] - 15s 160ms/step - loss: 0.0291 - accuracy: 0.9932 - val_loss: 0.8031 - val_accuracy: 0.8065
Epoch 26/30
96/96 [==============================] - 15s 159ms/step - loss: 0.0256 - accuracy: 0.9932 - val_loss: 0.7989 - val_accuracy: 0.8182
Epoch 27/30
96/96 [==============================] - 15s 161ms/step - loss: 0.0254 - accuracy: 0.9935 - val_loss: 0.7228 - val_accuracy: 0.8416
Epoch 28/30
96/96 [==============================] - 15s 161ms/step - loss: 0.0217 - accuracy: 0.9948 - val_loss: 0.7898 - val_accuracy: 0.8094
Epoch 29/30
96/96 [==============================] - 15s 161ms/step - loss: 0.0190 - accuracy: 0.9964 - val_loss: 0.8253 - val_accuracy: 0.8152
Epoch 30/30
96/96 [==============================] - 15s 158ms/step - loss: 0.0267 - accuracy: 0.9928 - val_loss: 0.8274 - val_accuracy: 0.8065


0.5
WARNING:tensorflow:sample_weight modes were coerced from
  ...
    to  
  ['...']
WARNING:tensorflow:sample_weight modes were coerced from
  ...
    to  
  ['...']
Train for 96 steps, validate for 11 steps
Epoch 1/30
96/96 [==============================] - 19s 201ms/step - loss: 1.3015 - accuracy: 0.5668 - val_loss: 0.7597 - val_accuracy: 0.7302
Epoch 2/30
96/96 [==============================] - 15s 159ms/step - loss: 0.8380 - accuracy: 0.7190 - val_loss: 0.6347 - val_accuracy: 0.7977
Epoch 3/30
96/96 [==============================] - 15s 160ms/step - loss: 0.6791 - accuracy: 0.7598 - val_loss: 0.5996 - val_accuracy: 0.8035
Epoch 4/30
96/96 [==============================] - 16s 163ms/step - loss: 0.5698 - accuracy: 0.8044 - val_loss: 0.5472 - val_accuracy: 0.8152
Epoch 5/30
96/96 [==============================] - 15s 159ms/step - loss: 0.4974 - accuracy: 0.8331 - val_loss: 0.5318 - val_accuracy: 0.8240
Epoch 6/30
96/96 [==============================] - 16s 162ms/step - loss: 0.4530 - accuracy: 0.8429 - val_loss: 0.5004 - val_accuracy: 0.8387
Epoch 7/30
96/96 [==============================] - 15s 161ms/step - loss: 0.3973 - accuracy: 0.8664 - val_loss: 0.5179 - val_accuracy: 0.8211
Epoch 8/30
96/96 [==============================] - 16s 162ms/step - loss: 0.3331 - accuracy: 0.8866 - val_loss: 0.5019 - val_accuracy: 0.8328
Epoch 9/30
96/96 [==============================] - 15s 160ms/step - loss: 0.3214 - accuracy: 0.8869 - val_loss: 0.5001 - val_accuracy: 0.8270
Epoch 10/30
96/96 [==============================] - 16s 162ms/step - loss: 0.2866 - accuracy: 0.9035 - val_loss: 0.5753 - val_accuracy: 0.8123
Epoch 11/30
96/96 [==============================] - 15s 161ms/step - loss: 0.2766 - accuracy: 0.9045 - val_loss: 0.5196 - val_accuracy: 0.8328
Epoch 12/30
96/96 [==============================] - 16s 163ms/step - loss: 0.2353 - accuracy: 0.9188 - val_loss: 0.5102 - val_accuracy: 0.8416
Epoch 13/30
96/96 [==============================] - 15s 161ms/step - loss: 0.2174 - accuracy: 0.9241 - val_loss: 0.5461 - val_accuracy: 0.8270
Epoch 14/30
96/96 [==============================] - 15s 159ms/step - loss: 0.1872 - accuracy: 0.9345 - val_loss: 0.5343 - val_accuracy: 0.8416
Epoch 15/30
96/96 [==============================] - 15s 159ms/step - loss: 0.1835 - accuracy: 0.9384 - val_loss: 0.5541 - val_accuracy: 0.8299
Epoch 16/30
96/96 [==============================] - 15s 158ms/step - loss: 0.1647 - accuracy: 0.9505 - val_loss: 0.5677 - val_accuracy: 0.8240
Epoch 17/30
96/96 [==============================] - 15s 160ms/step - loss: 0.1602 - accuracy: 0.9475 - val_loss: 0.5600 - val_accuracy: 0.8387
Epoch 18/30
96/96 [==============================] - 15s 158ms/step - loss: 0.1437 - accuracy: 0.9550 - val_loss: 0.5300 - val_accuracy: 0.8270
Epoch 19/30
96/96 [==============================] - 15s 160ms/step - loss: 0.1383 - accuracy: 0.9550 - val_loss: 0.5526 - val_accuracy: 0.8358
Epoch 20/30
96/96 [==============================] - 15s 159ms/step - loss: 0.1418 - accuracy: 0.9495 - val_loss: 0.5425 - val_accuracy: 0.8446
Epoch 21/30
96/96 [==============================] - 15s 159ms/step - loss: 0.1214 - accuracy: 0.9615 - val_loss: 0.5780 - val_accuracy: 0.8534
Epoch 22/30
96/96 [==============================] - 15s 159ms/step - loss: 0.0991 - accuracy: 0.9684 - val_loss: 0.6000 - val_accuracy: 0.8358
Epoch 23/30
96/96 [==============================] - 15s 158ms/step - loss: 0.1028 - accuracy: 0.9664 - val_loss: 0.5995 - val_accuracy: 0.8065
Epoch 24/30
96/96 [==============================] - 15s 160ms/step - loss: 0.0974 - accuracy: 0.9674 - val_loss: 0.6249 - val_accuracy: 0.8094
Epoch 25/30
96/96 [==============================] - 15s 160ms/step - loss: 0.1086 - accuracy: 0.9602 - val_loss: 0.5936 - val_accuracy: 0.8299
Epoch 26/30
96/96 [==============================] - 15s 158ms/step - loss: 0.0951 - accuracy: 0.9697 - val_loss: 0.6485 - val_accuracy: 0.8240
Epoch 27/30
96/96 [==============================] - 16s 162ms/step - loss: 0.0916 - accuracy: 0.9694 - val_loss: 0.5958 - val_accuracy: 0.8328
Epoch 28/30
96/96 [==============================] - 16s 163ms/step - loss: 0.0821 - accuracy: 0.9729 - val_loss: 0.6749 - val_accuracy: 0.8328
Epoch 29/30
96/96 [==============================] - 15s 159ms/step - loss: 0.1033 - accuracy: 0.9602 - val_loss: 0.6781 - val_accuracy: 0.8270
Epoch 30/30
96/96 [==============================] - 15s 161ms/step - loss: 0.0970 - accuracy: 0.9645 - val_loss: 0.6907 - val_accuracy: 0.8299


0.8
WARNING:tensorflow:Large dropout rate: 0.8 (>0.5). In TensorFlow 2.x, dropout() uses dropout rate instead of keep_prob. Please ensure that this is intended.
WARNING:tensorflow:sample_weight modes were coerced from
  ...
    to  
  ['...']
WARNING:tensorflow:sample_weight modes were coerced from
  ...
    to  
  ['...']
Train for 96 steps, validate for 11 steps
Epoch 1/30
WARNING:tensorflow:Large dropout rate: 0.8 (>0.5). In TensorFlow 2.x, dropout() uses dropout rate instead of keep_prob. Please ensure that this is intended.
WARNING:tensorflow:Large dropout rate: 0.8 (>0.5). In TensorFlow 2.x, dropout() uses dropout rate instead of keep_prob. Please ensure that this is intended.
96/96 [==============================] - 19s 199ms/step - loss: 1.8235 - accuracy: 0.3879 - val_loss: 1.1518 - val_accuracy: 0.6158
Epoch 2/30
96/96 [==============================] - 15s 157ms/step - loss: 1.4182 - accuracy: 0.5010 - val_loss: 0.9966 - val_accuracy: 0.7155
Epoch 3/30
96/96 [==============================] - 15s 159ms/step - loss: 1.2934 - accuracy: 0.5310 - val_loss: 0.8925 - val_accuracy: 0.7331
Epoch 4/30
96/96 [==============================] - 15s 159ms/step - loss: 1.2507 - accuracy: 0.5417 - val_loss: 0.8229 - val_accuracy: 0.7683
Epoch 5/30
96/96 [==============================] - 15s 158ms/step - loss: 1.1791 - accuracy: 0.5818 - val_loss: 0.8120 - val_accuracy: 0.7537
Epoch 6/30
96/96 [==============================] - 15s 159ms/step - loss: 1.1207 - accuracy: 0.5896 - val_loss: 0.7615 - val_accuracy: 0.7625
Epoch 7/30
96/96 [==============================] - 15s 159ms/step - loss: 1.0704 - accuracy: 0.6043 - val_loss: 0.7264 - val_accuracy: 0.7918
Epoch 8/30
96/96 [==============================] - 15s 158ms/step - loss: 1.0273 - accuracy: 0.6261 - val_loss: 0.6978 - val_accuracy: 0.7742
Epoch 9/30
96/96 [==============================] - 15s 158ms/step - loss: 1.0470 - accuracy: 0.6102 - val_loss: 0.7046 - val_accuracy: 0.7918
Epoch 10/30
96/96 [==============================] - 15s 158ms/step - loss: 0.9995 - accuracy: 0.6157 - val_loss: 0.6775 - val_accuracy: 0.7947
Epoch 11/30
96/96 [==============================] - 15s 161ms/step - loss: 0.9725 - accuracy: 0.6375 - val_loss: 0.6745 - val_accuracy: 0.7977
Epoch 12/30
96/96 [==============================] - 15s 158ms/step - loss: 0.9577 - accuracy: 0.6323 - val_loss: 0.6670 - val_accuracy: 0.7918
Epoch 13/30
96/96 [==============================] - 16s 163ms/step - loss: 0.9401 - accuracy: 0.6392 - val_loss: 0.6382 - val_accuracy: 0.7918
Epoch 14/30
96/96 [==============================] - 15s 161ms/step - loss: 0.9195 - accuracy: 0.6477 - val_loss: 0.6157 - val_accuracy: 0.8065
Epoch 15/30
96/96 [==============================] - 16s 163ms/step - loss: 0.8835 - accuracy: 0.6662 - val_loss: 0.6243 - val_accuracy: 0.8006
Epoch 16/30
96/96 [==============================] - 15s 158ms/step - loss: 0.8784 - accuracy: 0.6591 - val_loss: 0.6263 - val_accuracy: 0.7889
Epoch 17/30
96/96 [==============================] - 15s 158ms/step - loss: 0.8439 - accuracy: 0.6780 - val_loss: 0.6110 - val_accuracy: 0.8035
Epoch 18/30
96/96 [==============================] - 15s 159ms/step - loss: 0.8262 - accuracy: 0.6757 - val_loss: 0.6072 - val_accuracy: 0.8065
Epoch 19/30
96/96 [==============================] - 15s 159ms/step - loss: 0.8425 - accuracy: 0.6698 - val_loss: 0.6029 - val_accuracy: 0.8094
Epoch 20/30
96/96 [==============================] - 15s 159ms/step - loss: 0.8072 - accuracy: 0.6913 - val_loss: 0.6007 - val_accuracy: 0.7889
Epoch 21/30
96/96 [==============================] - 16s 164ms/step - loss: 0.8056 - accuracy: 0.6920 - val_loss: 0.5905 - val_accuracy: 0.8035
Epoch 22/30
96/96 [==============================] - 16s 163ms/step - loss: 0.7889 - accuracy: 0.6842 - val_loss: 0.5832 - val_accuracy: 0.8035
Epoch 23/30
96/96 [==============================] - 16s 164ms/step - loss: 0.7635 - accuracy: 0.6965 - val_loss: 0.6038 - val_accuracy: 0.8094
Epoch 24/30
96/96 [==============================] - 15s 159ms/step - loss: 0.7756 - accuracy: 0.6962 - val_loss: 0.5772 - val_accuracy: 0.8065
Epoch 25/30
96/96 [==============================] - 15s 159ms/step - loss: 0.7670 - accuracy: 0.6995 - val_loss: 0.6003 - val_accuracy: 0.8094
Epoch 26/30
96/96 [==============================] - 15s 158ms/step - loss: 0.7333 - accuracy: 0.7066 - val_loss: 0.5757 - val_accuracy: 0.8211
Epoch 27/30
96/96 [==============================] - 15s 161ms/step - loss: 0.7398 - accuracy: 0.7080 - val_loss: 0.5493 - val_accuracy: 0.8240
Epoch 28/30
96/96 [==============================] - 16s 162ms/step - loss: 0.7143 - accuracy: 0.7171 - val_loss: 0.5323 - val_accuracy: 0.8328
Epoch 29/30
96/96 [==============================] - 16s 162ms/step - loss: 0.7117 - accuracy: 0.7122 - val_loss: 0.5473 - val_accuracy: 0.8094
Epoch 30/30
96/96 [==============================] - 16s 165ms/step - loss: 0.6907 - accuracy: 0.7256 - val_loss: 0.5628 - val_accuracy: 0.8299

In [16]python · cell 48

python

for droprate, hist in scores.items():
    plt.plot(hist['val_accuracy'], label=('val=%s' % droprate))

plt.ylim(0.78, 0.86)
plt.legend()

Output

<matplotlib.legend.Legend at 0x7feda76a0b38>

<Figure size 432x288 with 1 Axes>

In [23]python · cell 49

python

hist = scores[0.0]
plt.plot(hist['val_accuracy'], label=0.0)

hist = scores[0.2]
plt.plot(hist['val_accuracy'], label=0.2)

plt.legend()
#plt.plot(hist['accuracy'], label=('val=%s' % droprate))

Output

<matplotlib.legend.Legend at 0x7feda74902e8>

<Figure size 432x288 with 1 Axes>

8.10 Data augmentation

Different data augmentations
Training a model with augmentations
How to select data augmentations?

In [17]python · cell 51

python

train_gen = ImageDataGenerator(
    preprocessing_function=preprocess_input,
#     vertical_flip=True,
)

train_ds = train_gen.flow_from_directory(
    './clothing-dataset-small/train',
    target_size=(150, 150),
    batch_size=32
)

val_gen = ImageDataGenerator(preprocessing_function=preprocess_input)

val_ds = val_gen.flow_from_directory(
    './clothing-dataset-small/validation',
    target_size=(150, 150),
    batch_size=32,
    shuffle=False
)

Output

Found 3068 images belonging to 10 classes.
Found 341 images belonging to 10 classes.

In [18]python · cell 52

python

learning_rate = 0.001
size = 100
droprate = 0.2

model = make_model(
    learning_rate=learning_rate,
    size_inner=size,
    droprate=droprate
)

history = model.fit(train_ds, epochs=50, validation_data=val_ds)

Output

WARNING:tensorflow:sample_weight modes were coerced from
  ...
    to  
  ['...']
WARNING:tensorflow:sample_weight modes were coerced from
  ...
    to  
  ['...']
Train for 96 steps, validate for 11 steps
Epoch 1/50
96/96 [==============================] - 19s 200ms/step - loss: 1.0738 - accuracy: 0.6346 - val_loss: 0.6670 - val_accuracy: 0.7859
Epoch 2/50
96/96 [==============================] - 16s 162ms/step - loss: 0.6210 - accuracy: 0.7846 - val_loss: 0.5732 - val_accuracy: 0.8182
Epoch 3/50
96/96 [==============================] - 16s 162ms/step - loss: 0.4573 - accuracy: 0.8387 - val_loss: 0.5620 - val_accuracy: 0.8182
Epoch 4/50
96/96 [==============================] - 15s 161ms/step - loss: 0.3637 - accuracy: 0.8797 - val_loss: 0.5376 - val_accuracy: 0.8094
Epoch 5/50
96/96 [==============================] - 16s 162ms/step - loss: 0.2951 - accuracy: 0.9042 - val_loss: 0.5465 - val_accuracy: 0.8065
Epoch 6/50
96/96 [==============================] - 16s 162ms/step - loss: 0.2348 - accuracy: 0.9283 - val_loss: 0.6016 - val_accuracy: 0.7918
Epoch 7/50
96/96 [==============================] - 16s 163ms/step - loss: 0.1910 - accuracy: 0.9446 - val_loss: 0.5404 - val_accuracy: 0.8299
Epoch 8/50
96/96 [==============================] - 16s 162ms/step - loss: 0.1439 - accuracy: 0.9589 - val_loss: 0.5493 - val_accuracy: 0.8182
Epoch 9/50
96/96 [==============================] - 16s 163ms/step - loss: 0.1177 - accuracy: 0.9690 - val_loss: 0.5867 - val_accuracy: 0.8328
Epoch 10/50
96/96 [==============================] - 16s 163ms/step - loss: 0.1066 - accuracy: 0.9736 - val_loss: 0.5887 - val_accuracy: 0.8240
Epoch 11/50
96/96 [==============================] - 16s 164ms/step - loss: 0.0842 - accuracy: 0.9821 - val_loss: 0.6101 - val_accuracy: 0.8240
Epoch 12/50
83/96 [========================>.....] - ETA: 1s - loss: 0.0721 - accuracy: 0.9832

[0;31m---------------------------------------------------------------------------[0m
[0;31mKeyboardInterrupt[0m                         Traceback (most recent call last)
[0;32m<ipython-input-18-e333a961bbf9>[0m in [0;36m<module>[0;34m[0m
[1;32m      9[0m )
[1;32m     10[0m [0;34m[0m[0m
[0;32m---> 11[0;31m [0mhistory[0m [0;34m=[0m [0mmodel[0m[0;34m.[0m[0mfit[0m[0;34m([0m[0mtrain_ds[0m[0;34m,[0m [0mepochs[0m[0;34m=[0m[0;36m50[0m[0;34m,[0m [0mvalidation_data[0m[0;34m=[0m[0mval_ds[0m[0;34m)[0m[0;34m[0m[0;34m[0m[0m
[0m
[0;32m~/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training.py[0m in [0;36mfit[0;34m(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_freq, max_queue_size, workers, use_multiprocessing, **kwargs)[0m
[1;32m    823[0m         [0mmax_queue_size[0m[0;34m=[0m[0mmax_queue_size[0m[0;34m,[0m[0;34m[0m[0;34m[0m[0m
[1;32m    824[0m         [0mworkers[0m[0;34m=[0m[0mworkers[0m[0;34m,[0m[0;34m[0m[0;34m[0m[0m
[0;32m--> 825[0;31m         use_multiprocessing=use_multiprocessing)
[0m[1;32m    826[0m [0;34m[0m[0m
[1;32m    827[0m   def evaluate(self,

[0;32m~/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_v2.py[0m in [0;36mfit[0;34m(self, model, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_freq, max_queue_size, workers, use_multiprocessing, **kwargs)[0m
[1;32m    340[0m                 [0mmode[0m[0;34m=[0m[0mModeKeys[0m[0;34m.[0m[0mTRAIN[0m[0;34m,[0m[0;34m[0m[0;34m[0m[0m
[1;32m    341[0m                 [0mtraining_context[0m[0;34m=[0m[0mtraining_context[0m[0;34m,[0m[0;34m[0m[0;34m[0m[0m
[0;32m--> 342[0;31m                 total_epochs=epochs)
[0m[1;32m    343[0m             [0mcbks[0m[0;34m.[0m[0mmake_logs[0m[0;34m([0m[0mmodel[0m[0;34m,[0m [0mepoch_logs[0m[0;34m,[0m [0mtraining_result[0m[0;34m,[0m [0mModeKeys[0m[0;34m.[0m[0mTRAIN[0m[0;34m)[0m[0;34m[0m[0;34m[0m[0m
[1;32m    344[0m [0;34m[0m[0m

[0;32m~/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_v2.py[0m in [0;36mrun_one_epoch[0;34m(model, iterator, execution_function, dataset_size, batch_size, strategy, steps_per_epoch, num_samples, mode, training_context, total_epochs)[0m
[1;32m    126[0m         step=step, mode=mode, size=current_batch_size) as batch_logs:
[1;32m    127[0m       [0;32mtry[0m[0;34m:[0m[0;34m[0m[0;34m[0m[0m
[0;32m--> 128[0;31m         [0mbatch_outs[0m [0;34m=[0m [0mexecution_function[0m[0;34m([0m[0miterator[0m[0;34m)[0m[0;34m[0m[0;34m[0m[0m
[0m[1;32m    129[0m       [0;32mexcept[0m [0;34m([0m[0mStopIteration[0m[0;34m,[0m [0merrors[0m[0;34m.[0m[0mOutOfRangeError[0m[0;34m)[0m[0;34m:[0m[0;34m[0m[0;34m[0m[0m
[1;32m    130[0m         [0;31m# TODO(kaftan): File bug about tf function and errors.OutOfRangeError?[0m[0;34m[0m[0;34m[0m[0;34m[0m[0m

[0;32m~/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_v2_utils.py[0m in [0;36mexecution_function[0;34m(input_fn)[0m
[1;32m     96[0m     [0;31m# `numpy` translates Tensors to values in Eager mode.[0m[0;34m[0m[0;34m[0m[0;34m[0m[0m
[1;32m     97[0m     return nest.map_structure(_non_none_constant_value,
[0;32m---> 98[0;31m                               distributed_function(input_fn))
[0m[1;32m     99[0m [0;34m[0m[0m
[1;32m    100[0m   [0;32mreturn[0m [0mexecution_function[0m[0;34m[0m[0;34m[0m[0m

[0;32m~/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/tensorflow_core/python/eager/def_function.py[0m in [0;36m__call__[0;34m(self, *args, **kwds)[0m
[1;32m    566[0m         [0mxla_context[0m[0;34m.[0m[0mExit[0m[0;34m([0m[0;34m)[0m[0;34m[0m[0;34m[0m[0m
[1;32m    567[0m     [0;32melse[0m[0;34m:[0m[0;34m[0m[0;34m[0m[0m
[0;32m--> 568[0;31m       [0mresult[0m [0;34m=[0m [0mself[0m[0;34m.[0m[0m_call[0m[0;34m([0m[0;34m*[0m[0margs[0m[0;34m,[0m [0;34m**[0m[0mkwds[0m[0;34m)[0m[0;34m[0m[0;34m[0m[0m
[0m[1;32m    569[0m [0;34m[0m[0m
[1;32m    570[0m     [0;32mif[0m [0mtracing_count[0m [0;34m==[0m [0mself[0m[0;34m.[0m[0m_get_tracing_count[0m[0;34m([0m[0;34m)[0m[0;34m:[0m[0;34m[0m[0;34m[0m[0m

[0;32m~/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/tensorflow_core/python/eager/def_function.py[0m in [0;36m_call[0;34m(self, *args, **kwds)[0m
[1;32m    597[0m       [0;31m# In this case we have created variables on the first call, so we run the[0m[0;34m[0m[0;34m[0m[0;34m[0m[0m
[1;32m    598[0m       [0;31m# defunned version which is guaranteed to never create variables.[0m[0;34m[0m[0;34m[0m[0;34m[0m[0m
[0;32m--> 599[0;31m       [0;32mreturn[0m [0mself[0m[0;34m.[0m[0m_stateless_fn[0m[0;34m([0m[0;34m*[0m[0margs[0m[0;34m,[0m [0;34m**[0m[0mkwds[0m[0;34m)[0m  [0;31m# pylint: disable=not-callable[0m[0;34m[0m[0;34m[0m[0m
[0m[1;32m    600[0m     [0;32melif[0m [0mself[0m[0;34m.[0m[0m_stateful_fn[0m [0;32mis[0m [0;32mnot[0m [0;32mNone[0m[0;34m:[0m[0;34m[0m[0;34m[0m[0m
[1;32m    601[0m       [0;31m# Release the lock early so that multiple threads can perform the call[0m[0;34m[0m[0;34m[0m[0;34m[0m[0m

[0;32m~/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py[0m in [0;36m__call__[0;34m(self, *args, **kwargs)[0m
[1;32m   2361[0m     [0;32mwith[0m [0mself[0m[0;34m.[0m[0m_lock[0m[0;34m:[0m[0;34m[0m[0;34m[0m[0m
[1;32m   2362[0m       [0mgraph_function[0m[0;34m,[0m [0margs[0m[0;34m,[0m [0mkwargs[0m [0;34m=[0m [0mself[0m[0;34m.[0m[0m_maybe_define_function[0m[0;34m([0m[0margs[0m[0;34m,[0m [0mkwargs[0m[0;34m)[0m[0;34m[0m[0;34m[0m[0m
[0;32m-> 2363[0;31m     [0;32mreturn[0m [0mgraph_function[0m[0;34m.[0m[0m_filtered_call[0m[0;34m([0m[0margs[0m[0;34m,[0m [0mkwargs[0m[0;34m)[0m  [0;31m# pylint: disable=protected-access[0m[0;34m[0m[0;34m[0m[0m
[0m[1;32m   2364[0m [0;34m[0m[0m
[1;32m   2365[0m   [0;34m@[0m[0mproperty[0m[0;34m[0m[0;34m[0m[0m

[0;32m~/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py[0m in [0;36m_filtered_call[0;34m(self, args, kwargs)[0m
[1;32m   1609[0m          if isinstance(t, (ops.Tensor,
[1;32m   1610[0m                            resource_variable_ops.BaseResourceVariable))),
[0;32m-> 1611[0;31m         self.captured_inputs)
[0m[1;32m   1612[0m [0;34m[0m[0m
[1;32m   1613[0m   [0;32mdef[0m [0m_call_flat[0m[0;34m([0m[0mself[0m[0;34m,[0m [0margs[0m[0;34m,[0m [0mcaptured_inputs[0m[0;34m,[0m [0mcancellation_manager[0m[0;34m=[0m[0;32mNone[0m[0;34m)[0m[0;34m:[0m[0;34m[0m[0;34m[0m[0m

[0;32m~/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py[0m in [0;36m_call_flat[0;34m(self, args, captured_inputs, cancellation_manager)[0m
[1;32m   1690[0m       [0;31m# No tape is watching; skip to running the function.[0m[0;34m[0m[0;34m[0m[0;34m[0m[0m
[1;32m   1691[0m       return self._build_call_outputs(self._inference_function.call(
[0;32m-> 1692[0;31m           ctx, args, cancellation_manager=cancellation_manager))
[0m[1;32m   1693[0m     forward_backward = self._select_forward_and_backward_functions(
[1;32m   1694[0m         [0margs[0m[0;34m,[0m[0;34m[0m[0;34m[0m[0m

[0;32m~/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py[0m in [0;36mcall[0;34m(self, ctx, args, cancellation_manager)[0m
[1;32m    543[0m               [0minputs[0m[0;34m=[0m[0margs[0m[0;34m,[0m[0;34m[0m[0;34m[0m[0m
[1;32m    544[0m               [0mattrs[0m[0;34m=[0m[0;34m([0m[0;34m"executor_type"[0m[0;34m,[0m [0mexecutor_type[0m[0;34m,[0m [0;34m"config_proto"[0m[0;34m,[0m [0mconfig[0m[0;34m)[0m[0;34m,[0m[0;34m[0m[0;34m[0m[0m
[0;32m--> 545[0;31m               ctx=ctx)
[0m[1;32m    546[0m         [0;32melse[0m[0;34m:[0m[0;34m[0m[0;34m[0m[0m
[1;32m    547[0m           outputs = execute.execute_with_cancellation(

[0;32m~/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/tensorflow_core/python/eager/execute.py[0m in [0;36mquick_execute[0;34m(op_name, num_outputs, inputs, attrs, ctx, name)[0m
[1;32m     59[0m     tensors = pywrap_tensorflow.TFE_Py_Execute(ctx._handle, device_name,
[1;32m     60[0m                                                [0mop_name[0m[0;34m,[0m [0minputs[0m[0;34m,[0m [0mattrs[0m[0;34m,[0m[0;34m[0m[0;34m[0m[0m
[0;32m---> 61[0;31m                                                num_outputs)
[0m[1;32m     62[0m   [0;32mexcept[0m [0mcore[0m[0;34m.[0m[0m_NotOkStatusException[0m [0;32mas[0m [0me[0m[0;34m:[0m[0;34m[0m[0;34m[0m[0m
[1;32m     63[0m     [0;32mif[0m [0mname[0m [0;32mis[0m [0;32mnot[0m [0;32mNone[0m[0;34m:[0m[0;34m[0m[0;34m[0m[0m

[0;31mKeyboardInterrupt[0m:

In [12]python · cell 53

python

hist = history.history
plt.plot(hist['val_accuracy'], label='val')
plt.plot(hist['accuracy'], label='train')

plt.legend()

Output

<matplotlib.legend.Legend at 0x7fc75018f438>

<Figure size 432x288 with 1 Axes>

8.11 Training a larger model

Train a 299x299 model

In [19]python · cell 55

python

def make_model(input_size=150, learning_rate=0.01, size_inner=100,
               droprate=0.5):

    base_model = Xception(
        weights='imagenet',
        include_top=False,
        input_shape=(input_size, input_size, 3)
    )

    base_model.trainable = False

    #########################################

    inputs = keras.Input(shape=(input_size, input_size, 3))
    base = base_model(inputs, training=False)
    vectors = keras.layers.GlobalAveragePooling2D()(base)
    
    inner = keras.layers.Dense(size_inner, activation='relu')(vectors)
    drop = keras.layers.Dropout(droprate)(inner)
    
    outputs = keras.layers.Dense(10)(drop)
    
    model = keras.Model(inputs, outputs)
    
    #########################################

    optimizer = keras.optimizers.Adam(learning_rate=learning_rate)
    loss = keras.losses.CategoricalCrossentropy(from_logits=True)

    model.compile(
        optimizer=optimizer,
        loss=loss,
        metrics=['accuracy']
    )
    
    return model

In [25]python · cell 56

python

input_size = 299

In [38]python · cell 57

python

train_gen = ImageDataGenerator(
    preprocessing_function=preprocess_input,
    shear_range=10,
    zoom_range=0.1,
    horizontal_flip=True
)

train_ds = train_gen.flow_from_directory(
    './clothing-dataset-small/train',
    target_size=(input_size, input_size),
    batch_size=32
)


val_gen = ImageDataGenerator(preprocessing_function=preprocess_input)

val_ds = train_gen.flow_from_directory(
    './clothing-dataset-small/validation',
    target_size=(input_size, input_size),
    batch_size=32,
    shuffle=False
)

Output

Found 3068 images belonging to 10 classes.
Found 341 images belonging to 10 classes.

In [39]python · cell 58

python

checkpoint = keras.callbacks.ModelCheckpoint(
    'xception_v4_1_{epoch:02d}_{val_accuracy:.3f}.h5',
    save_best_only=True,
    monitor='val_accuracy',
    mode='max'
)

In [40]python · cell 59

python

learning_rate = 0.0005
size = 100
droprate = 0.2

model = make_model(
    input_size=input_size,
    learning_rate=learning_rate,
    size_inner=size,
    droprate=droprate
)

history = model.fit(train_ds, epochs=50, validation_data=val_ds,
                   callbacks=[checkpoint])

Output

WARNING:tensorflow:sample_weight modes were coerced from
  ...
    to  
  ['...']
WARNING:tensorflow:sample_weight modes were coerced from
  ...
    to  
  ['...']
Train for 96 steps, validate for 11 steps
Epoch 1/50
96/96 [==============================] - 78s 816ms/step - loss: 1.0387 - accuracy: 0.6793 - val_loss: 0.5567 - val_accuracy: 0.8211
Epoch 2/50
96/96 [==============================] - 77s 805ms/step - loss: 0.5472 - accuracy: 0.8214 - val_loss: 0.4331 - val_accuracy: 0.8622
Epoch 3/50
96/96 [==============================] - 74s 772ms/step - loss: 0.4525 - accuracy: 0.8491 - val_loss: 0.4360 - val_accuracy: 0.8504
Epoch 4/50
96/96 [==============================] - 75s 778ms/step - loss: 0.4000 - accuracy: 0.8605 - val_loss: 0.3728 - val_accuracy: 0.8856
Epoch 5/50
96/96 [==============================] - 75s 785ms/step - loss: 0.3586 - accuracy: 0.8853 - val_loss: 0.3832 - val_accuracy: 0.8768
Epoch 6/50
96/96 [==============================] - 75s 777ms/step - loss: 0.3261 - accuracy: 0.8934 - val_loss: 0.3828 - val_accuracy: 0.8827
Epoch 7/50
96/96 [==============================] - 75s 778ms/step - loss: 0.3105 - accuracy: 0.8957 - val_loss: 0.3707 - val_accuracy: 0.8768
Epoch 8/50
96/96 [==============================] - 74s 772ms/step - loss: 0.2857 - accuracy: 0.9055 - val_loss: 0.3801 - val_accuracy: 0.8592
Epoch 9/50
96/96 [==============================] - 74s 773ms/step - loss: 0.2686 - accuracy: 0.9058 - val_loss: 0.3884 - val_accuracy: 0.8563
Epoch 10/50
96/96 [==============================] - 74s 776ms/step - loss: 0.2575 - accuracy: 0.9143 - val_loss: 0.3733 - val_accuracy: 0.8710
Epoch 11/50
96/96 [==============================] - 75s 778ms/step - loss: 0.2372 - accuracy: 0.9208 - val_loss: 0.3582 - val_accuracy: 0.8680
Epoch 12/50
96/96 [==============================] - 74s 776ms/step - loss: 0.2231 - accuracy: 0.9250 - val_loss: 0.3572 - val_accuracy: 0.8798
Epoch 13/50
96/96 [==============================] - 75s 783ms/step - loss: 0.2075 - accuracy: 0.9289 - val_loss: 0.3355 - val_accuracy: 0.9032
Epoch 14/50
96/96 [==============================] - 74s 772ms/step - loss: 0.2042 - accuracy: 0.9325 - val_loss: 0.3347 - val_accuracy: 0.8915
Epoch 15/50
96/96 [==============================] - 76s 796ms/step - loss: 0.1941 - accuracy: 0.9312 - val_loss: 0.3390 - val_accuracy: 0.8886
Epoch 16/50
96/96 [==============================] - 74s 773ms/step - loss: 0.1773 - accuracy: 0.9439 - val_loss: 0.3915 - val_accuracy: 0.8768
Epoch 17/50
96/96 [==============================] - 76s 789ms/step - loss: 0.1729 - accuracy: 0.9420 - val_loss: 0.3564 - val_accuracy: 0.8915
Epoch 18/50
96/96 [==============================] - 75s 777ms/step - loss: 0.1632 - accuracy: 0.9439 - val_loss: 0.3387 - val_accuracy: 0.8944
Epoch 19/50
96/96 [==============================] - 75s 783ms/step - loss: 0.1548 - accuracy: 0.9462 - val_loss: 0.3415 - val_accuracy: 0.8768
Epoch 20/50
96/96 [==============================] - 74s 775ms/step - loss: 0.1484 - accuracy: 0.9527 - val_loss: 0.3625 - val_accuracy: 0.8651
Epoch 21/50
96/96 [==============================] - 75s 781ms/step - loss: 0.1395 - accuracy: 0.9557 - val_loss: 0.3616 - val_accuracy: 0.8739
Epoch 22/50
25/96 [======>.......................] - ETA: 51s - loss: 0.1452 - accuracy: 0.9583

[0;31m---------------------------------------------------------------------------[0m
[0;31mKeyboardInterrupt[0m                         Traceback (most recent call last)
[0;32m~/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_v2.py[0m in [0;36mon_epoch[0;34m(self, epoch, mode)[0m
[1;32m    766[0m     [0;32mtry[0m[0;34m:[0m[0;34m[0m[0;34m[0m[0m
[0;32m--> 767[0;31m       [0;32myield[0m [0mepoch_logs[0m[0;34m[0m[0;34m[0m[0m
[0m[1;32m    768[0m     [0;32mfinally[0m[0;34m:[0m[0;34m[0m[0;34m[0m[0m

[0;32m~/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_v2.py[0m in [0;36mfit[0;34m(self, model, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_freq, max_queue_size, workers, use_multiprocessing, **kwargs)[0m
[1;32m    341[0m                 [0mtraining_context[0m[0;34m=[0m[0mtraining_context[0m[0;34m,[0m[0;34m[0m[0;34m[0m[0m
[0;32m--> 342[0;31m                 total_epochs=epochs)
[0m[1;32m    343[0m             [0mcbks[0m[0;34m.[0m[0mmake_logs[0m[0;34m([0m[0mmodel[0m[0;34m,[0m [0mepoch_logs[0m[0;34m,[0m [0mtraining_result[0m[0;34m,[0m [0mModeKeys[0m[0;34m.[0m[0mTRAIN[0m[0;34m)[0m[0;34m[0m[0;34m[0m[0m

[0;32m~/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_v2.py[0m in [0;36mrun_one_epoch[0;34m(model, iterator, execution_function, dataset_size, batch_size, strategy, steps_per_epoch, num_samples, mode, training_context, total_epochs)[0m
[1;32m    127[0m       [0;32mtry[0m[0;34m:[0m[0;34m[0m[0;34m[0m[0m
[0;32m--> 128[0;31m         [0mbatch_outs[0m [0;34m=[0m [0mexecution_function[0m[0;34m([0m[0miterator[0m[0;34m)[0m[0;34m[0m[0;34m[0m[0m
[0m[1;32m    129[0m       [0;32mexcept[0m [0;34m([0m[0mStopIteration[0m[0;34m,[0m [0merrors[0m[0;34m.[0m[0mOutOfRangeError[0m[0;34m)[0m[0;34m:[0m[0;34m[0m[0;34m[0m[0m

[0;32m~/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_v2_utils.py[0m in [0;36mexecution_function[0;34m(input_fn)[0m
[1;32m     97[0m     return nest.map_structure(_non_none_constant_value,
[0;32m---> 98[0;31m                               distributed_function(input_fn))
[0m[1;32m     99[0m [0;34m[0m[0m

[0;32m~/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/tensorflow_core/python/eager/def_function.py[0m in [0;36m__call__[0;34m(self, *args, **kwds)[0m
[1;32m    567[0m     [0;32melse[0m[0;34m:[0m[0;34m[0m[0;34m[0m[0m
[0;32m--> 568[0;31m       [0mresult[0m [0;34m=[0m [0mself[0m[0;34m.[0m[0m_call[0m[0;34m([0m[0;34m*[0m[0margs[0m[0;34m,[0m [0;34m**[0m[0mkwds[0m[0;34m)[0m[0;34m[0m[0;34m[0m[0m
[0m[1;32m    569[0m [0;34m[0m[0m

[0;32m~/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/tensorflow_core/python/eager/def_function.py[0m in [0;36m_call[0;34m(self, *args, **kwds)[0m
[1;32m    598[0m       [0;31m# defunned version which is guaranteed to never create variables.[0m[0;34m[0m[0;34m[0m[0;34m[0m[0m
[0;32m--> 599[0;31m       [0;32mreturn[0m [0mself[0m[0;34m.[0m[0m_stateless_fn[0m[0;34m([0m[0;34m*[0m[0margs[0m[0;34m,[0m [0;34m**[0m[0mkwds[0m[0;34m)[0m  [0;31m# pylint: disable=not-callable[0m[0;34m[0m[0;34m[0m[0m
[0m[1;32m    600[0m     [0;32melif[0m [0mself[0m[0;34m.[0m[0m_stateful_fn[0m [0;32mis[0m [0;32mnot[0m [0;32mNone[0m[0;34m:[0m[0;34m[0m[0;34m[0m[0m

[0;32m~/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py[0m in [0;36m__call__[0;34m(self, *args, **kwargs)[0m
[1;32m   2362[0m       [0mgraph_function[0m[0;34m,[0m [0margs[0m[0;34m,[0m [0mkwargs[0m [0;34m=[0m [0mself[0m[0;34m.[0m[0m_maybe_define_function[0m[0;34m([0m[0margs[0m[0;34m,[0m [0mkwargs[0m[0;34m)[0m[0;34m[0m[0;34m[0m[0m
[0;32m-> 2363[0;31m     [0;32mreturn[0m [0mgraph_function[0m[0;34m.[0m[0m_filtered_call[0m[0;34m([0m[0margs[0m[0;34m,[0m [0mkwargs[0m[0;34m)[0m  [0;31m# pylint: disable=protected-access[0m[0;34m[0m[0;34m[0m[0m
[0m[1;32m   2364[0m [0;34m[0m[0m

[0;32m~/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py[0m in [0;36m_filtered_call[0;34m(self, args, kwargs)[0m
[1;32m   1610[0m                            resource_variable_ops.BaseResourceVariable))),
[0;32m-> 1611[0;31m         self.captured_inputs)
[0m[1;32m   1612[0m [0;34m[0m[0m

[0;32m~/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py[0m in [0;36m_call_flat[0;34m(self, args, captured_inputs, cancellation_manager)[0m
[1;32m   1691[0m       return self._build_call_outputs(self._inference_function.call(
[0;32m-> 1692[0;31m           ctx, args, cancellation_manager=cancellation_manager))
[0m[1;32m   1693[0m     forward_backward = self._select_forward_and_backward_functions(

[0;32m~/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py[0m in [0;36mcall[0;34m(self, ctx, args, cancellation_manager)[0m
[1;32m    544[0m               [0mattrs[0m[0;34m=[0m[0;34m([0m[0;34m"executor_type"[0m[0;34m,[0m [0mexecutor_type[0m[0;34m,[0m [0;34m"config_proto"[0m[0;34m,[0m [0mconfig[0m[0;34m)[0m[0;34m,[0m[0;34m[0m[0;34m[0m[0m
[0;32m--> 545[0;31m               ctx=ctx)
[0m[1;32m    546[0m         [0;32melse[0m[0;34m:[0m[0;34m[0m[0;34m[0m[0m

[0;32m~/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/tensorflow_core/python/eager/execute.py[0m in [0;36mquick_execute[0;34m(op_name, num_outputs, inputs, attrs, ctx, name)[0m
[1;32m     60[0m                                                [0mop_name[0m[0;34m,[0m [0minputs[0m[0;34m,[0m [0mattrs[0m[0;34m,[0m[0;34m[0m[0;34m[0m[0m
[0;32m---> 61[0;31m                                                num_outputs)
[0m[1;32m     62[0m   [0;32mexcept[0m [0mcore[0m[0;34m.[0m[0m_NotOkStatusException[0m [0;32mas[0m [0me[0m[0;34m:[0m[0;34m[0m[0;34m[0m[0m

[0;31mKeyboardInterrupt[0m: 

During handling of the above exception, another exception occurred:

[0;31mKeyError[0m                                  Traceback (most recent call last)
[0;32m<ipython-input-40-89b8717352a9>[0m in [0;36m<module>[0;34m[0m
[1;32m     11[0m [0;34m[0m[0m
[1;32m     12[0m history = model.fit(train_ds, epochs=50, validation_data=val_ds,
[0;32m---> 13[0;31m                    callbacks=[checkpoint])
[0m
[0;32m~/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training.py[0m in [0;36mfit[0;34m(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_freq, max_queue_size, workers, use_multiprocessing, **kwargs)[0m
[1;32m    823[0m         [0mmax_queue_size[0m[0;34m=[0m[0mmax_queue_size[0m[0;34m,[0m[0;34m[0m[0;34m[0m[0m
[1;32m    824[0m         [0mworkers[0m[0;34m=[0m[0mworkers[0m[0;34m,[0m[0;34m[0m[0;34m[0m[0m
[0;32m--> 825[0;31m         use_multiprocessing=use_multiprocessing)
[0m[1;32m    826[0m [0;34m[0m[0m
[1;32m    827[0m   def evaluate(self,

[0;32m~/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_v2.py[0m in [0;36mfit[0;34m(self, model, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_freq, max_queue_size, workers, use_multiprocessing, **kwargs)[0m
[1;32m    395[0m                       total_epochs=1)
[1;32m    396[0m                   cbks.make_logs(model, epoch_logs, eval_result, ModeKeys.TEST,
[0;32m--> 397[0;31m                                  prefix='val_')
[0m[1;32m    398[0m [0;34m[0m[0m
[1;32m    399[0m     [0;32mreturn[0m [0mmodel[0m[0;34m.[0m[0mhistory[0m[0;34m[0m[0;34m[0m[0m

[0;32m~/anaconda3/envs/tensorflow2_p36/lib/python3.6/contextlib.py[0m in [0;36m__exit__[0;34m(self, type, value, traceback)[0m
[1;32m     97[0m                 [0mvalue[0m [0;34m=[0m [0mtype[0m[0;34m([0m[0;34m)[0m[0;34m[0m[0;34m[0m[0m
[1;32m     98[0m             [0;32mtry[0m[0;34m:[0m[0;34m[0m[0;34m[0m[0m
[0;32m---> 99[0;31m                 [0mself[0m[0;34m.[0m[0mgen[0m[0;34m.[0m[0mthrow[0m[0;34m([0m[0mtype[0m[0;34m,[0m [0mvalue[0m[0;34m,[0m [0mtraceback[0m[0;34m)[0m[0;34m[0m[0;34m[0m[0m
[0m[1;32m    100[0m             [0;32mexcept[0m [0mStopIteration[0m [0;32mas[0m [0mexc[0m[0;34m:[0m[0;34m[0m[0;34m[0m[0m
[1;32m    101[0m                 [0;31m# Suppress StopIteration *unless* it's the same exception that[0m[0;34m[0m[0;34m[0m[0;34m[0m[0m

[0;32m~/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_v2.py[0m in [0;36mon_epoch[0;34m(self, epoch, mode)[0m
[1;32m    769[0m       [0;32mif[0m [0mmode[0m [0;34m==[0m [0mModeKeys[0m[0;34m.[0m[0mTRAIN[0m[0;34m:[0m[0;34m[0m[0;34m[0m[0m
[1;32m    770[0m         [0;31m# Epochs only apply to `fit`.[0m[0;34m[0m[0;34m[0m[0;34m[0m[0m
[0;32m--> 771[0;31m         [0mself[0m[0;34m.[0m[0mcallbacks[0m[0;34m.[0m[0mon_epoch_end[0m[0;34m([0m[0mepoch[0m[0;34m,[0m [0mepoch_logs[0m[0;34m)[0m[0;34m[0m[0;34m[0m[0m
[0m[1;32m    772[0m       [0mself[0m[0;34m.[0m[0mprogbar[0m[0;34m.[0m[0mon_epoch_end[0m[0;34m([0m[0mepoch[0m[0;34m,[0m [0mepoch_logs[0m[0;34m)[0m[0;34m[0m[0;34m[0m[0m
[1;32m    773[0m [0;34m[0m[0m

[0;32m~/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/tensorflow_core/python/keras/callbacks.py[0m in [0;36mon_epoch_end[0;34m(self, epoch, logs)[0m
[1;32m    300[0m     [0mlogs[0m [0;34m=[0m [0mlogs[0m [0;32mor[0m [0;34m{[0m[0;34m}[0m[0;34m[0m[0;34m[0m[0m
[1;32m    301[0m     [0;32mfor[0m [0mcallback[0m [0;32min[0m [0mself[0m[0;34m.[0m[0mcallbacks[0m[0;34m:[0m[0;34m[0m[0;34m[0m[0m
[0;32m--> 302[0;31m       [0mcallback[0m[0;34m.[0m[0mon_epoch_end[0m[0;34m([0m[0mepoch[0m[0;34m,[0m [0mlogs[0m[0;34m)[0m[0;34m[0m[0;34m[0m[0m
[0m[1;32m    303[0m [0;34m[0m[0m
[1;32m    304[0m   [0;32mdef[0m [0mon_train_batch_begin[0m[0;34m([0m[0mself[0m[0;34m,[0m [0mbatch[0m[0;34m,[0m [0mlogs[0m[0;34m=[0m[0;32mNone[0m[0;34m)[0m[0;34m:[0m[0;34m[0m[0;34m[0m[0m

[0;32m~/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/tensorflow_core/python/keras/callbacks.py[0m in [0;36mon_epoch_end[0;34m(self, epoch, logs)[0m
[1;32m    990[0m           [0mself[0m[0;34m.[0m[0m_save_model[0m[0;34m([0m[0mepoch[0m[0;34m=[0m[0mepoch[0m[0;34m,[0m [0mlogs[0m[0;34m=[0m[0mlogs[0m[0;34m)[0m[0;34m[0m[0;34m[0m[0m
[1;32m    991[0m       [0;32melse[0m[0;34m:[0m[0;34m[0m[0;34m[0m[0m
[0;32m--> 992[0;31m         [0mself[0m[0;34m.[0m[0m_save_model[0m[0;34m([0m[0mepoch[0m[0;34m=[0m[0mepoch[0m[0;34m,[0m [0mlogs[0m[0;34m=[0m[0mlogs[0m[0;34m)[0m[0;34m[0m[0;34m[0m[0m
[0m[1;32m    993[0m     [0;32mif[0m [0mself[0m[0;34m.[0m[0mmodel[0m[0;34m.[0m[0m_in_multi_worker_mode[0m[0;34m([0m[0;34m)[0m[0;34m:[0m[0;34m[0m[0;34m[0m[0m
[1;32m    994[0m       [0;31m# For multi-worker training, back up the weights and current training[0m[0;34m[0m[0;34m[0m[0;34m[0m[0m

[0;32m~/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/tensorflow_core/python/keras/callbacks.py[0m in [0;36m_save_model[0;34m(self, epoch, logs)[0m
[1;32m   1009[0m                   int) or self.epochs_since_last_save >= self.period:
[1;32m   1010[0m       [0mself[0m[0;34m.[0m[0mepochs_since_last_save[0m [0;34m=[0m [0;36m0[0m[0;34m[0m[0;34m[0m[0m
[0;32m-> 1011[0;31m       [0mfilepath[0m [0;34m=[0m [0mself[0m[0;34m.[0m[0m_get_file_path[0m[0;34m([0m[0mepoch[0m[0;34m,[0m [0mlogs[0m[0;34m)[0m[0;34m[0m[0;34m[0m[0m
[0m[1;32m   1012[0m [0;34m[0m[0m
[1;32m   1013[0m       [0;32mtry[0m[0;34m:[0m[0;34m[0m[0;34m[0m[0m

[0;32m~/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/tensorflow_core/python/keras/callbacks.py[0m in [0;36m_get_file_path[0;34m(self, epoch, logs)[0m
[1;32m   1053[0m     if not self.model._in_multi_worker_mode(
[1;32m   1054[0m     ) or multi_worker_util.should_save_checkpoint():
[0;32m-> 1055[0;31m       [0;32mreturn[0m [0mself[0m[0;34m.[0m[0mfilepath[0m[0;34m.[0m[0mformat[0m[0;34m([0m[0mepoch[0m[0;34m=[0m[0mepoch[0m [0;34m+[0m [0;36m1[0m[0;34m,[0m [0;34m**[0m[0mlogs[0m[0;34m)[0m[0;34m[0m[0;34m[0m[0m
[0m[1;32m   1056[0m     [0;32melse[0m[0;34m:[0m[0;34m[0m[0;34m[0m[0m
[1;32m   1057[0m       [0;31m# If this is multi-worker training, and this worker should not[0m[0;34m[0m[0;34m[0m[0;34m[0m[0m

[0;31mKeyError[0m: 'val_accuracy'

8.12 Using the model

Loading the model
Evaluating the model
Getting predictions

In [3]python · cell 61

python

import tensorflow as tf
from tensorflow import keras

In [13]python · cell 62

python

from tensorflow.keras.preprocessing.image import ImageDataGenerator
from tensorflow.keras.preprocessing.image import load_img

from tensorflow.keras.applications.xception import preprocess_input

In [7]python · cell 63

python

test_gen = ImageDataGenerator(preprocessing_function=preprocess_input)

test_ds = test_gen.flow_from_directory(
    './clothing-dataset-small/test',
    target_size=(299, 299),
    batch_size=32,
    shuffle=False
)

Output

Found 372 images belonging to 10 classes.

In [11]python · cell 64

python

model = keras.models.load_model('xception_v4_1_13_0.903.h5')

In [10]python · cell 65

python

model.evaluate(test_ds)

Output

WARNING:tensorflow:sample_weight modes were coerced from
  ...
    to  
  ['...']
12/12 [==============================] - 8s 645ms/step - loss: 0.2939 - accuracy: 0.8978

[0.29389633300403756, 0.89784944]

In [12]python · cell 66

python

path = 'clothing-dataset-small/test/pants/c8d21106-bbdb-4e8d-83e4-bf3d14e54c16.jpg'

In [15]python · cell 67

python

img = load_img(path, target_size=(299, 299))

In [16]python · cell 68

python

import numpy as np

In [18]python · cell 69

python

x = np.array(img)
X = np.array([x])
X.shape

Output

(1, 299, 299, 3)

In [21]python · cell 70

python

X = preprocess_input(X)

In [23]python · cell 71

python

pred = model.predict(X)

In [25]python · cell 72

python

classes = [
    'dress',
    'hat',
    'longsleeve',
    'outwear',
    'pants',
    'shirt',
    'shoes',
    'shorts',
    'skirt',
    't-shirt'
]

In [26]python · cell 73

python

dict(zip(classes, pred[0]))

Output

{'dress': -1.4282539,
 'hat': -5.522186,
 'longsleeve': -3.1655293,
 'outwear': -2.201648,
 'pants': 9.294684,
 'shirt': -3.4289198,
 'shoes': -4.2395606,
 'shorts': 3.4339347,
 'skirt': -4.194675,
 't-shirt': -2.9939806}

8.13 Summary

We can use pre-trained models for general image classification
Convolutional layers let us turn an image into a vector
Dense layers use the vector to make the predictions
Instead of training a model from scratch, we can use transfer learning and re-use already trained convolutional layers
First, train a small model (150x150) before training a big one (299x299)
Learning rate - how fast the model trians. Fast learners aren't always best ones
We can save the best model using callbacks and checkpointing
To avoid overfitting, use dropout and augmentation

8.14 Explore more

Add more data, e.g. Zalando, etc (ADD LINKS)
Albumentations - another way of generating augmentations
Use PyTorch or MXNet instead of TensorFlow/Keras
In addition to Xception, there are others architectures - try them

Other projects:

cats vs dogs
Hotdog vs not hotdog
Category of images

Chapter 77

8. Neural networks and deep learning

NotebookPython 3 (ipykernel)75 cells

In [1]python · cell 1

python

%autosave 0

Output

Autosave disabled

8. Neural networks and deep learning

This week, we'll learn about neural nets and build a model for classifying images of clothes

8.1 Fashion classification

Dataset:

Links:

https://cs231n.github.io/

In [ ]python · cell 3

python

!git clone git@github.com:alexeygrigorev/clothing-dataset-small.git

8.2 TensorFlow and Keras

Installing TensorFlow
Loading images

In [2]python · cell 5

python

import numpy as np
import matplotlib.pyplot as plt

%matplotlib inline

In [3]python · cell 6

python

import tensorflow as tf
from tensorflow import keras

In [4]python · cell 7

python

from tensorflow.keras.preprocessing.image import load_img

In [10]python · cell 8

python

path = './clothing-dataset-small/train/t-shirt'
name = '5f0a3fa0-6a3d-4b68-b213-72766a643de7.jpg'
fullname = f'{path}/{name}'
load_img(fullname)

Output

<PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=400x533 at 0x7F6599CE5208>

[省略较大 image/png 输出]

In [11]python · cell 9

python

img = load_img(fullname, target_size=(299, 299))

In [12]python · cell 10

python

x = np.array(img)
x.shape

Output

(299, 299, 3)

8.3 Pre-trained convolutional neural networks

Imagenet dataset: https://www.image-net.org/
Pre-trained models: https://keras.io/api/applications/

In [5]python · cell 12

python

from tensorflow.keras.applications.xception import Xception
from tensorflow.keras.applications.xception import preprocess_input
from tensorflow.keras.applications.xception import decode_predictions

In [6]python · cell 13

python

model = Xception(weights='imagenet', input_shape=(299, 299, 3))

Output

Downloading data from https://github.com/fchollet/deep-learning-models/releases/download/v0.4/xception_weights_tf_dim_ordering_tf_kernels.h5
91889664/91884032 [==============================] - 2s 0us/step

In [18]python · cell 14

python

X = np.array([x])

In [19]python · cell 15

python

X.shape

Output

(1, 299, 299, 3)

In [24]python · cell 16

python

X = preprocess_input(X)

In [27]python · cell 17

python

pred = model.predict(X)

In [31]python · cell 18

python

decode_predictions(pred)

Output

Downloading data from https://storage.googleapis.com/download.tensorflow.org/data/imagenet_class_index.json
40960/35363 [==================================] - 0s 0us/step

[[('n03595614', 'jersey', 0.6792451),
  ('n02916936', 'bulletproof_vest', 0.039600316),
  ('n04370456', 'sweatshirt', 0.035299566),
  ('n03710637', 'maillot', 0.010884117),
  ('n04525038', 'velvet', 0.0018057569)]]

8.4 Convolutional neural networks

Types of layers: convolutional and dense
Convolutional layers and filters
Dense layers

There are more layers. Read here: https://cs231n.github.io/

8.5 Transfer learning

Reading data with ImageDataGenerator
Train Xception on smaller images (150x150)

(Better to run it with a GPU)

In [6]python · cell 21

python

from tensorflow.keras.preprocessing.image import ImageDataGenerator

In [8]python · cell 22

python

train_gen = ImageDataGenerator(preprocessing_function=preprocess_input)

train_ds = train_gen.flow_from_directory(
    './clothing-dataset-small/train',
    target_size=(150, 150),
    batch_size=32
)

Output

Found 3068 images belonging to 10 classes.

In [10]python · cell 23

python

train_ds.class_indices

Output

{'dress': 0,
 'hat': 1,
 'longsleeve': 2,
 'outwear': 3,
 'pants': 4,
 'shirt': 5,
 'shoes': 6,
 'shorts': 7,
 'skirt': 8,
 't-shirt': 9}

In [14]python · cell 24

python

X, y = next(train_ds)

In [18]python · cell 25

python

y[:5]

Output

array([[0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],
       [0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],
       [0., 0., 0., 0., 0., 1., 0., 0., 0., 0.],
       [0., 0., 0., 0., 1., 0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0., 0., 0., 0., 0., 1.]], dtype=float32)

In [9]python · cell 26

python

val_gen = ImageDataGenerator(preprocessing_function=preprocess_input)

val_ds = val_gen.flow_from_directory(
    './clothing-dataset-small/validation',
    target_size=(150, 150),
    batch_size=32,
    shuffle=False
)

Output

Found 341 images belonging to 10 classes.

In [10]python · cell 27

python

base_model = Xception(
    weights='imagenet',
    include_top=False,
    input_shape=(150, 150, 3)
)

base_model.trainable = False



inputs = keras.Input(shape=(150, 150, 3))

base = base_model(inputs, training=False)

vectors = keras.layers.GlobalAveragePooling2D()(base)

outputs = keras.layers.Dense(10)(vectors)

model = keras.Model(inputs, outputs)

Output

Downloading data from https://github.com/fchollet/deep-learning-models/releases/download/v0.4/xception_weights_tf_dim_ordering_tf_kernels_notop.h5
83689472/83683744 [==============================] - 1s 0us/step

In [50]python · cell 28

python

learning_rate = 0.01
optimizer = keras.optimizers.Adam(learning_rate=learning_rate)

loss = keras.losses.CategoricalCrossentropy(from_logits=True)

model.compile(optimizer=optimizer, loss=loss, metrics=['accuracy'])

In [51]python · cell 29

python

history = model.fit(train_ds, epochs=10, validation_data=val_ds)

Output

WARNING:tensorflow:sample_weight modes were coerced from
  ...
    to  
  ['...']
WARNING:tensorflow:sample_weight modes were coerced from
  ...
    to  
  ['...']
Train for 96 steps, validate for 11 steps
Epoch 1/10
96/96 [==============================] - 20s 211ms/step - loss: 1.2568 - accuracy: 0.6698 - val_loss: 0.8537 - val_accuracy: 0.7449
Epoch 2/10
96/96 [==============================] - 16s 166ms/step - loss: 0.5745 - accuracy: 0.8188 - val_loss: 0.6850 - val_accuracy: 0.8065
Epoch 3/10
96/96 [==============================] - 16s 166ms/step - loss: 0.3261 - accuracy: 0.8885 - val_loss: 0.7734 - val_accuracy: 0.8152
Epoch 4/10
96/96 [==============================] - 16s 166ms/step - loss: 0.2215 - accuracy: 0.9244 - val_loss: 0.9123 - val_accuracy: 0.7918
Epoch 5/10
96/96 [==============================] - 16s 166ms/step - loss: 0.1433 - accuracy: 0.9475 - val_loss: 0.8244 - val_accuracy: 0.8094
Epoch 6/10
96/96 [==============================] - 16s 167ms/step - loss: 0.0897 - accuracy: 0.9707 - val_loss: 0.8517 - val_accuracy: 0.8035
Epoch 7/10
96/96 [==============================] - 16s 166ms/step - loss: 0.0855 - accuracy: 0.9703 - val_loss: 0.7886 - val_accuracy: 0.8299
Epoch 8/10
96/96 [==============================] - 16s 167ms/step - loss: 0.0907 - accuracy: 0.9661 - val_loss: 0.8840 - val_accuracy: 0.8123
Epoch 9/10
96/96 [==============================] - 16s 167ms/step - loss: 0.0458 - accuracy: 0.9876 - val_loss: 0.9531 - val_accuracy: 0.8035
Epoch 10/10
96/96 [==============================] - 16s 166ms/step - loss: 0.0465 - accuracy: 0.9876 - val_loss: 0.9695 - val_accuracy: 0.7889

In [59]python · cell 30

python

#plt.plot(history.history['accuracy'], label='train')
plt.plot(history.history['val_accuracy'], label='val')
plt.xticks(np.arange(10))
plt.legend()

Output

<matplotlib.legend.Legend at 0x7fe20fc3c860>

<Figure size 432x288 with 1 Axes>

8.6 Adjusting the learning rate

What's the learning rate
Trying different values

In [60]python · cell 32

python

def make_model(learning_rate=0.01):
    base_model = Xception(
        weights='imagenet',
        include_top=False,
        input_shape=(150, 150, 3)
    )

    base_model.trainable = False

    #########################################

    inputs = keras.Input(shape=(150, 150, 3))
    base = base_model(inputs, training=False)
    vectors = keras.layers.GlobalAveragePooling2D()(base)
    outputs = keras.layers.Dense(10)(vectors)
    model = keras.Model(inputs, outputs)
    
    #########################################

    optimizer = keras.optimizers.Adam(learning_rate=learning_rate)
    loss = keras.losses.CategoricalCrossentropy(from_logits=True)

    model.compile(
        optimizer=optimizer,
        loss=loss,
        metrics=['accuracy']
    )
    
    return model

In [61]python · cell 33

python

scores = {}

for lr in [0.0001, 0.001, 0.01, 0.1]:
    print(lr)

    model = make_model(learning_rate=lr)
    history = model.fit(train_ds, epochs=10, validation_data=val_ds)
    scores[lr] = history.history

    print()
    print()

Output

0.0001
WARNING:tensorflow:sample_weight modes were coerced from
  ...
    to  
  ['...']
WARNING:tensorflow:sample_weight modes were coerced from
  ...
    to  
  ['...']
Train for 96 steps, validate for 11 steps
Epoch 1/10
96/96 [==============================] - 20s 208ms/step - loss: 1.9262 - accuracy: 0.3403 - val_loss: 1.5806 - val_accuracy: 0.4897
Epoch 2/10
96/96 [==============================] - 16s 166ms/step - loss: 1.3832 - accuracy: 0.5613 - val_loss: 1.2368 - val_accuracy: 0.6012
Epoch 3/10
96/96 [==============================] - 16s 166ms/step - loss: 1.1397 - accuracy: 0.6463 - val_loss: 1.0575 - val_accuracy: 0.6716
Epoch 4/10
96/96 [==============================] - 16s 166ms/step - loss: 1.0022 - accuracy: 0.6858 - val_loss: 0.9497 - val_accuracy: 0.7214
Epoch 5/10
96/96 [==============================] - 16s 166ms/step - loss: 0.9087 - accuracy: 0.7141 - val_loss: 0.8773 - val_accuracy: 0.7566
Epoch 6/10
96/96 [==============================] - 16s 166ms/step - loss: 0.8401 - accuracy: 0.7376 - val_loss: 0.8281 - val_accuracy: 0.7713
Epoch 7/10
96/96 [==============================] - 16s 166ms/step - loss: 0.7886 - accuracy: 0.7565 - val_loss: 0.7873 - val_accuracy: 0.7801
Epoch 8/10
96/96 [==============================] - 16s 167ms/step - loss: 0.7453 - accuracy: 0.7637 - val_loss: 0.7537 - val_accuracy: 0.7830
Epoch 9/10
96/96 [==============================] - 16s 167ms/step - loss: 0.7094 - accuracy: 0.7757 - val_loss: 0.7299 - val_accuracy: 0.7859
Epoch 10/10
96/96 [==============================] - 16s 167ms/step - loss: 0.6791 - accuracy: 0.7806 - val_loss: 0.7060 - val_accuracy: 0.7889


0.001
WARNING:tensorflow:sample_weight modes were coerced from
  ...
    to  
  ['...']
WARNING:tensorflow:sample_weight modes were coerced from
  ...
    to  
  ['...']
Train for 96 steps, validate for 11 steps
Epoch 1/10
96/96 [==============================] - 20s 204ms/step - loss: 1.1293 - accuracy: 0.6199 - val_loss: 0.7179 - val_accuracy: 0.7625
Epoch 2/10
96/96 [==============================] - 16s 166ms/step - loss: 0.6555 - accuracy: 0.7761 - val_loss: 0.6018 - val_accuracy: 0.8035
Epoch 3/10
96/96 [==============================] - 16s 166ms/step - loss: 0.5239 - accuracy: 0.8230 - val_loss: 0.5708 - val_accuracy: 0.8094
Epoch 4/10
96/96 [==============================] - 16s 166ms/step - loss: 0.4409 - accuracy: 0.8553 - val_loss: 0.5574 - val_accuracy: 0.8035
Epoch 5/10
96/96 [==============================] - 16s 167ms/step - loss: 0.3796 - accuracy: 0.8820 - val_loss: 0.5373 - val_accuracy: 0.8152
Epoch 6/10
96/96 [==============================] - 16s 167ms/step - loss: 0.3307 - accuracy: 0.9029 - val_loss: 0.5273 - val_accuracy: 0.8270
Epoch 7/10
96/96 [==============================] - 16s 167ms/step - loss: 0.2973 - accuracy: 0.9153 - val_loss: 0.5111 - val_accuracy: 0.8152
Epoch 8/10
96/96 [==============================] - 16s 167ms/step - loss: 0.2657 - accuracy: 0.9316 - val_loss: 0.5245 - val_accuracy: 0.8006
Epoch 9/10
96/96 [==============================] - 16s 168ms/step - loss: 0.2394 - accuracy: 0.9426 - val_loss: 0.5233 - val_accuracy: 0.8299
Epoch 10/10
96/96 [==============================] - 16s 168ms/step - loss: 0.2137 - accuracy: 0.9505 - val_loss: 0.5017 - val_accuracy: 0.8240


0.01
WARNING:tensorflow:sample_weight modes were coerced from
  ...
    to  
  ['...']
WARNING:tensorflow:sample_weight modes were coerced from
  ...
    to  
  ['...']
Train for 96 steps, validate for 11 steps
Epoch 1/10
96/96 [==============================] - 20s 209ms/step - loss: 1.1734 - accuracy: 0.6701 - val_loss: 1.0102 - val_accuracy: 0.7390
Epoch 2/10
96/96 [==============================] - 16s 166ms/step - loss: 0.5969 - accuracy: 0.8129 - val_loss: 0.8572 - val_accuracy: 0.7771
Epoch 3/10
96/96 [==============================] - 16s 167ms/step - loss: 0.3137 - accuracy: 0.8902 - val_loss: 0.7792 - val_accuracy: 0.7947
Epoch 4/10
96/96 [==============================] - 16s 166ms/step - loss: 0.2034 - accuracy: 0.9273 - val_loss: 0.9025 - val_accuracy: 0.7859
Epoch 5/10
96/96 [==============================] - 16s 166ms/step - loss: 0.1677 - accuracy: 0.9423 - val_loss: 0.9980 - val_accuracy: 0.7713
Epoch 6/10
96/96 [==============================] - 16s 166ms/step - loss: 0.2104 - accuracy: 0.9205 - val_loss: 1.0246 - val_accuracy: 0.8123
Epoch 7/10
96/96 [==============================] - 16s 166ms/step - loss: 0.1228 - accuracy: 0.9615 - val_loss: 0.9585 - val_accuracy: 0.8094
Epoch 8/10
96/96 [==============================] - 16s 167ms/step - loss: 0.0779 - accuracy: 0.9743 - val_loss: 0.8759 - val_accuracy: 0.8123
Epoch 9/10
96/96 [==============================] - 16s 166ms/step - loss: 0.0651 - accuracy: 0.9791 - val_loss: 0.8844 - val_accuracy: 0.8211
Epoch 10/10
96/96 [==============================] - 16s 167ms/step - loss: 0.0619 - accuracy: 0.9817 - val_loss: 1.0551 - val_accuracy: 0.7889


0.1
WARNING:tensorflow:sample_weight modes were coerced from
  ...
    to  
  ['...']
WARNING:tensorflow:sample_weight modes were coerced from
  ...
    to  
  ['...']
Train for 96 steps, validate for 11 steps
Epoch 1/10
96/96 [==============================] - 20s 203ms/step - loss: 10.4920 - accuracy: 0.6327 - val_loss: 6.0144 - val_accuracy: 0.7537
Epoch 2/10
96/96 [==============================] - 16s 166ms/step - loss: 5.2193 - accuracy: 0.7744 - val_loss: 7.3142 - val_accuracy: 0.7566
Epoch 3/10
96/96 [==============================] - 16s 166ms/step - loss: 3.4999 - accuracy: 0.8387 - val_loss: 8.0551 - val_accuracy: 0.7683
Epoch 4/10
96/96 [==============================] - 16s 166ms/step - loss: 2.9493 - accuracy: 0.8625 - val_loss: 9.5917 - val_accuracy: 0.7507
Epoch 5/10
96/96 [==============================] - 16s 166ms/step - loss: 2.4901 - accuracy: 0.8814 - val_loss: 9.1683 - val_accuracy: 0.7507
Epoch 6/10
96/96 [==============================] - 16s 166ms/step - loss: 2.1291 - accuracy: 0.9016 - val_loss: 9.2850 - val_accuracy: 0.7654
Epoch 7/10
96/96 [==============================] - 16s 166ms/step - loss: 1.4794 - accuracy: 0.9185 - val_loss: 10.4883 - val_accuracy: 0.7566
Epoch 8/10
96/96 [==============================] - 16s 166ms/step - loss: 1.4156 - accuracy: 0.9211 - val_loss: 11.8833 - val_accuracy: 0.7742
Epoch 9/10
96/96 [==============================] - 16s 167ms/step - loss: 1.1350 - accuracy: 0.9400 - val_loss: 10.6944 - val_accuracy: 0.7625
Epoch 10/10
96/96 [==============================] - 16s 167ms/step - loss: 1.1251 - accuracy: 0.9475 - val_loss: 8.6116 - val_accuracy: 0.7889

In [65]python · cell 34

python

del scores[0.1]
del scores[0.0001]

In [68]python · cell 35

python

for lr, hist in scores.items():
    #plt.plot(hist['accuracy'], label=('train=%s' % lr))
    plt.plot(hist['val_accuracy'], label=('val=%s' % lr))

plt.xticks(np.arange(10))
plt.legend()

Output

<matplotlib.legend.Legend at 0x7fe0acceac88>

<Figure size 432x288 with 1 Axes>

In [ ]python · cell 36

python

learning_rate = 0.001

8.7 Checkpointing

Saving the best model only
Training a model with callbacks

In [70]python · cell 38

python

model.save_weights('model_v1.h5', save_format='h5')

In [21]python · cell 39

python

chechpoint = keras.callbacks.ModelCheckpoint(
    'xception_v1_{epoch:02d}_{val_accuracy:.3f}.h5',
    save_best_only=True,
    monitor='val_accuracy',
    mode='max'
)

In [79]python · cell 40

python

learning_rate = 0.001

model = make_model(learning_rate=learning_rate)

history = model.fit(
    train_ds,
    epochs=10,
    validation_data=val_ds,
    callbacks=[chechpoint]
)

Output

WARNING:tensorflow:sample_weight modes were coerced from
  ...
    to  
  ['...']
WARNING:tensorflow:sample_weight modes were coerced from
  ...
    to  
  ['...']
Train for 96 steps, validate for 11 steps
Epoch 1/10
96/96 [==============================] - 21s 216ms/step - loss: 1.1353 - accuracy: 0.6170 - val_loss: 0.7258 - val_accuracy: 0.7801
Epoch 2/10
96/96 [==============================] - 16s 168ms/step - loss: 0.6469 - accuracy: 0.7735 - val_loss: 0.6332 - val_accuracy: 0.7859
Epoch 3/10
96/96 [==============================] - 16s 169ms/step - loss: 0.5182 - accuracy: 0.8243 - val_loss: 0.5905 - val_accuracy: 0.8094
Epoch 4/10
96/96 [==============================] - 16s 170ms/step - loss: 0.4390 - accuracy: 0.8553 - val_loss: 0.5550 - val_accuracy: 0.8152
Epoch 5/10
96/96 [==============================] - 16s 170ms/step - loss: 0.3827 - accuracy: 0.8827 - val_loss: 0.5437 - val_accuracy: 0.8211
Epoch 6/10
96/96 [==============================] - 16s 170ms/step - loss: 0.3342 - accuracy: 0.8990 - val_loss: 0.5319 - val_accuracy: 0.8358
Epoch 7/10
96/96 [==============================] - 16s 166ms/step - loss: 0.2978 - accuracy: 0.9149 - val_loss: 0.5169 - val_accuracy: 0.8299
Epoch 8/10
96/96 [==============================] - 16s 166ms/step - loss: 0.2652 - accuracy: 0.9283 - val_loss: 0.5422 - val_accuracy: 0.8299
Epoch 9/10
96/96 [==============================] - 16s 166ms/step - loss: 0.2419 - accuracy: 0.9387 - val_loss: 0.5234 - val_accuracy: 0.8211
Epoch 10/10
96/96 [==============================] - 16s 166ms/step - loss: 0.2157 - accuracy: 0.9498 - val_loss: 0.5320 - val_accuracy: 0.8328

8.8 Adding more layers

Adding one inner dense layer
Experimenting with different sizes of inner layer

In [3]python · cell 42

python

def make_model(learning_rate=0.01, size_inner=100):
    base_model = Xception(
        weights='imagenet',
        include_top=False,
        input_shape=(150, 150, 3)
    )

    base_model.trainable = False

    #########################################

    inputs = keras.Input(shape=(150, 150, 3))
    base = base_model(inputs, training=False)
    vectors = keras.layers.GlobalAveragePooling2D()(base)
    
    inner = keras.layers.Dense(size_inner, activation='relu')(vectors)
    
    outputs = keras.layers.Dense(10)(inner)
    
    model = keras.Model(inputs, outputs)
    
    #########################################

    optimizer = keras.optimizers.Adam(learning_rate=learning_rate)
    loss = keras.losses.CategoricalCrossentropy(from_logits=True)

    model.compile(
        optimizer=optimizer,
        loss=loss,
        metrics=['accuracy']
    )
    
    return model

In [13]python · cell 43

python

learning_rate = 0.001

scores = {}

for size in [10, 100, 1000]:
    print(size)

    model = make_model(learning_rate=learning_rate, size_inner=size)
    history = model.fit(train_ds, epochs=10, validation_data=val_ds)
    scores[size] = history.history

    print()
    print()

Output

10
Downloading data from https://github.com/fchollet/deep-learning-models/releases/download/v0.4/xception_weights_tf_dim_ordering_tf_kernels_notop.h5
83689472/83683744 [==============================] - 1s 0us/step
[2021-11-03 13:35:43.005 ip-172-16-64-149:21634 INFO utils.py:27] RULE_JOB_STOP_SIGNAL_FILENAME: None
[2021-11-03 13:35:43.104 ip-172-16-64-149:21634 INFO profiler_config_parser.py:111] Unable to find config at /opt/ml/input/config/profilerconfig.json. Profiler is disabled.
WARNING:tensorflow:sample_weight modes were coerced from
  ...
    to  
  ['...']
WARNING:tensorflow:sample_weight modes were coerced from
  ...
    to  
  ['...']
Train for 96 steps, validate for 11 steps
Epoch 1/10
96/96 [==============================] - 42s 438ms/step - loss: 1.1849 - accuracy: 0.6033 - val_loss: 0.8076 - val_accuracy: 0.7243
Epoch 2/10
96/96 [==============================] - 16s 169ms/step - loss: 0.7314 - accuracy: 0.7474 - val_loss: 0.7305 - val_accuracy: 0.7507
Epoch 3/10
96/96 [==============================] - 16s 169ms/step - loss: 0.5834 - accuracy: 0.8018 - val_loss: 0.6308 - val_accuracy: 0.7889
Epoch 4/10
96/96 [==============================] - 16s 170ms/step - loss: 0.4878 - accuracy: 0.8370 - val_loss: 0.6715 - val_accuracy: 0.7859
Epoch 5/10
96/96 [==============================] - 16s 170ms/step - loss: 0.4128 - accuracy: 0.8647 - val_loss: 0.5720 - val_accuracy: 0.8094
Epoch 6/10
96/96 [==============================] - 16s 170ms/step - loss: 0.3512 - accuracy: 0.8980 - val_loss: 0.5757 - val_accuracy: 0.8152
Epoch 7/10
96/96 [==============================] - 16s 171ms/step - loss: 0.3043 - accuracy: 0.9091 - val_loss: 0.5484 - val_accuracy: 0.8152
Epoch 8/10
96/96 [==============================] - 16s 172ms/step - loss: 0.2564 - accuracy: 0.9306 - val_loss: 0.5492 - val_accuracy: 0.7918
Epoch 9/10
96/96 [==============================] - 16s 171ms/step - loss: 0.2220 - accuracy: 0.9459 - val_loss: 0.5523 - val_accuracy: 0.7977
Epoch 10/10
96/96 [==============================] - 16s 172ms/step - loss: 0.2006 - accuracy: 0.9462 - val_loss: 0.5594 - val_accuracy: 0.8152


100
WARNING:tensorflow:sample_weight modes were coerced from
  ...
    to  
  ['...']
WARNING:tensorflow:sample_weight modes were coerced from
  ...
    to  
  ['...']
Train for 96 steps, validate for 11 steps
Epoch 1/10
96/96 [==============================] - 20s 208ms/step - loss: 0.9623 - accuracy: 0.6819 - val_loss: 0.6446 - val_accuracy: 0.7801
Epoch 2/10
96/96 [==============================] - 17s 173ms/step - loss: 0.4784 - accuracy: 0.8341 - val_loss: 0.6494 - val_accuracy: 0.7918
Epoch 3/10
96/96 [==============================] - 17s 172ms/step - loss: 0.3548 - accuracy: 0.8752 - val_loss: 0.6583 - val_accuracy: 0.7830
Epoch 4/10
96/96 [==============================] - 17s 172ms/step - loss: 0.2430 - accuracy: 0.9166 - val_loss: 0.6221 - val_accuracy: 0.8006
Epoch 5/10
96/96 [==============================] - 17s 172ms/step - loss: 0.1873 - accuracy: 0.9413 - val_loss: 0.6345 - val_accuracy: 0.8035
Epoch 6/10
96/96 [==============================] - 17s 172ms/step - loss: 0.1431 - accuracy: 0.9566 - val_loss: 0.6523 - val_accuracy: 0.8065
Epoch 7/10
96/96 [==============================] - 16s 172ms/step - loss: 0.1153 - accuracy: 0.9700 - val_loss: 0.8347 - val_accuracy: 0.7713
Epoch 8/10
96/96 [==============================] - 17s 172ms/step - loss: 0.0802 - accuracy: 0.9795 - val_loss: 0.7221 - val_accuracy: 0.7859
Epoch 9/10
96/96 [==============================] - 17s 172ms/step - loss: 0.0647 - accuracy: 0.9870 - val_loss: 0.7199 - val_accuracy: 0.8152
Epoch 10/10
96/96 [==============================] - 17s 172ms/step - loss: 0.0518 - accuracy: 0.9922 - val_loss: 0.7012 - val_accuracy: 0.8123


1000
WARNING:tensorflow:sample_weight modes were coerced from
  ...
    to  
  ['...']
WARNING:tensorflow:sample_weight modes were coerced from
  ...
    to  
  ['...']
Train for 96 steps, validate for 11 steps
Epoch 1/10
96/96 [==============================] - 20s 208ms/step - loss: 1.5152 - accuracy: 0.6633 - val_loss: 0.9100 - val_accuracy: 0.7771
Epoch 2/10
96/96 [==============================] - 16s 172ms/step - loss: 0.5736 - accuracy: 0.8237 - val_loss: 0.7562 - val_accuracy: 0.7830
Epoch 3/10
96/96 [==============================] - 16s 171ms/step - loss: 0.4004 - accuracy: 0.8703 - val_loss: 0.9595 - val_accuracy: 0.7507
Epoch 4/10
96/96 [==============================] - 16s 171ms/step - loss: 0.2576 - accuracy: 0.9065 - val_loss: 0.7193 - val_accuracy: 0.8065
Epoch 5/10
96/96 [==============================] - 16s 171ms/step - loss: 0.1669 - accuracy: 0.9410 - val_loss: 0.8278 - val_accuracy: 0.7977
Epoch 6/10
96/96 [==============================] - 17s 174ms/step - loss: 0.1255 - accuracy: 0.9563 - val_loss: 0.8112 - val_accuracy: 0.8035
Epoch 7/10
96/96 [==============================] - 16s 171ms/step - loss: 0.1177 - accuracy: 0.9596 - val_loss: 0.8148 - val_accuracy: 0.8123
Epoch 8/10
96/96 [==============================] - 16s 171ms/step - loss: 0.1825 - accuracy: 0.9342 - val_loss: 0.9044 - val_accuracy: 0.7889
Epoch 9/10
96/96 [==============================] - 16s 171ms/step - loss: 0.1303 - accuracy: 0.9537 - val_loss: 1.0076 - val_accuracy: 0.7859
Epoch 10/10
96/96 [==============================] - 16s 172ms/step - loss: 0.0891 - accuracy: 0.9694 - val_loss: 1.1218 - val_accuracy: 0.8006

In [16]python · cell 44

python

for size, hist in scores.items():
    plt.plot(hist['val_accuracy'], label=('val=%s' % size))

plt.xticks(np.arange(10))
plt.yticks([0.78, 0.80, 0.82, 0.825, 0.83])
plt.legend()

Output

<matplotlib.legend.Legend at 0x7f03ada03b00>

<Figure size 432x288 with 1 Axes>

8.9 Regularization and dropout

Regularizing by freezing a part of the network
Adding dropout to our model
Experimenting with different values

In [8]python · cell 46

python

def make_model(learning_rate=0.01, size_inner=100, droprate=0.5):
    base_model = Xception(
        weights='imagenet',
        include_top=False,
        input_shape=(150, 150, 3)
    )

    base_model.trainable = False

    #########################################

    inputs = keras.Input(shape=(150, 150, 3))
    base = base_model(inputs, training=False)
    vectors = keras.layers.GlobalAveragePooling2D()(base)
    
    inner = keras.layers.Dense(size_inner, activation='relu')(vectors)
    drop = keras.layers.Dropout(droprate)(inner)
    
    outputs = keras.layers.Dense(10)(drop)
    
    model = keras.Model(inputs, outputs)
    
    #########################################

    optimizer = keras.optimizers.Adam(learning_rate=learning_rate)
    loss = keras.losses.CategoricalCrossentropy(from_logits=True)

    model.compile(
        optimizer=optimizer,
        loss=loss,
        metrics=['accuracy']
    )
    
    return model

In [13]python · cell 47

python

learning_rate = 0.001
size = 100

scores = {}

for droprate in [0.0, 0.2, 0.5, 0.8]:
    print(droprate)

    model = make_model(
        learning_rate=learning_rate,
        size_inner=size,
        droprate=droprate
    )

    history = model.fit(train_ds, epochs=30, validation_data=val_ds)
    scores[droprate] = history.history

    print()
    print()

Output

0.0
[2021-11-03 21:19:20.707 ip-172-16-13-140:18999 INFO utils.py:27] RULE_JOB_STOP_SIGNAL_FILENAME: None
[2021-11-03 21:19:20.814 ip-172-16-13-140:18999 INFO profiler_config_parser.py:111] Unable to find config at /opt/ml/input/config/profilerconfig.json. Profiler is disabled.
WARNING:tensorflow:sample_weight modes were coerced from
  ...
    to  
  ['...']
WARNING:tensorflow:sample_weight modes were coerced from
  ...
    to  
  ['...']
Train for 96 steps, validate for 11 steps
Epoch 1/30
96/96 [==============================] - 56s 583ms/step - loss: 0.9642 - accuracy: 0.6747 - val_loss: 0.6832 - val_accuracy: 0.7566
Epoch 2/30
96/96 [==============================] - 15s 158ms/step - loss: 0.5163 - accuracy: 0.8230 - val_loss: 0.6058 - val_accuracy: 0.7918
Epoch 3/30
96/96 [==============================] - 15s 160ms/step - loss: 0.3458 - accuracy: 0.8885 - val_loss: 0.5577 - val_accuracy: 0.8270
Epoch 4/30
96/96 [==============================] - 15s 161ms/step - loss: 0.2575 - accuracy: 0.9218 - val_loss: 0.5166 - val_accuracy: 0.8416
Epoch 5/30
96/96 [==============================] - 15s 157ms/step - loss: 0.2002 - accuracy: 0.9394 - val_loss: 0.5974 - val_accuracy: 0.8152
Epoch 6/30
96/96 [==============================] - 15s 159ms/step - loss: 0.1322 - accuracy: 0.9684 - val_loss: 0.5695 - val_accuracy: 0.8152
Epoch 7/30
96/96 [==============================] - 15s 158ms/step - loss: 0.0874 - accuracy: 0.9863 - val_loss: 0.6004 - val_accuracy: 0.8152
Epoch 8/30
96/96 [==============================] - 15s 160ms/step - loss: 0.0674 - accuracy: 0.9902 - val_loss: 0.5810 - val_accuracy: 0.8211
Epoch 9/30
96/96 [==============================] - 15s 157ms/step - loss: 0.0485 - accuracy: 0.9971 - val_loss: 0.6084 - val_accuracy: 0.8270
Epoch 10/30
96/96 [==============================] - 15s 158ms/step - loss: 0.0326 - accuracy: 0.9987 - val_loss: 0.6066 - val_accuracy: 0.8416
Epoch 11/30
96/96 [==============================] - 15s 158ms/step - loss: 0.0235 - accuracy: 0.9990 - val_loss: 0.6404 - val_accuracy: 0.8152
Epoch 12/30
96/96 [==============================] - 15s 159ms/step - loss: 0.0229 - accuracy: 0.9990 - val_loss: 0.6242 - val_accuracy: 0.8270
Epoch 13/30
96/96 [==============================] - 15s 158ms/step - loss: 0.0171 - accuracy: 0.9997 - val_loss: 0.6530 - val_accuracy: 0.8328
Epoch 14/30
96/96 [==============================] - 15s 157ms/step - loss: 0.0130 - accuracy: 0.9997 - val_loss: 0.6650 - val_accuracy: 0.8358
Epoch 15/30
96/96 [==============================] - 15s 160ms/step - loss: 0.0138 - accuracy: 0.9993 - val_loss: 0.6573 - val_accuracy: 0.8328
Epoch 16/30
96/96 [==============================] - 15s 158ms/step - loss: 0.0117 - accuracy: 0.9993 - val_loss: 0.7331 - val_accuracy: 0.8152
Epoch 17/30
96/96 [==============================] - 15s 161ms/step - loss: 0.0121 - accuracy: 0.9997 - val_loss: 0.6736 - val_accuracy: 0.8270
Epoch 18/30
96/96 [==============================] - 15s 157ms/step - loss: 0.0105 - accuracy: 0.9993 - val_loss: 0.6732 - val_accuracy: 0.8299
Epoch 19/30
96/96 [==============================] - 15s 157ms/step - loss: 0.0065 - accuracy: 0.9997 - val_loss: 0.7275 - val_accuracy: 0.8240
Epoch 20/30
96/96 [==============================] - 15s 158ms/step - loss: 0.0087 - accuracy: 0.9997 - val_loss: 0.6873 - val_accuracy: 0.8328
Epoch 21/30
96/96 [==============================] - 15s 158ms/step - loss: 0.0074 - accuracy: 0.9993 - val_loss: 0.7036 - val_accuracy: 0.8387
Epoch 22/30
96/96 [==============================] - 16s 162ms/step - loss: 0.0048 - accuracy: 0.9997 - val_loss: 0.7190 - val_accuracy: 0.8358
Epoch 23/30
96/96 [==============================] - 15s 159ms/step - loss: 0.0062 - accuracy: 0.9997 - val_loss: 0.7420 - val_accuracy: 0.8240
Epoch 24/30
96/96 [==============================] - 15s 158ms/step - loss: 0.0089 - accuracy: 0.9990 - val_loss: 0.7259 - val_accuracy: 0.8358
Epoch 25/30
96/96 [==============================] - 15s 159ms/step - loss: 0.0092 - accuracy: 0.9987 - val_loss: 0.7479 - val_accuracy: 0.8299
Epoch 26/30
96/96 [==============================] - 16s 163ms/step - loss: 0.0033 - accuracy: 0.9997 - val_loss: 0.7662 - val_accuracy: 0.8358
Epoch 27/30
96/96 [==============================] - 16s 165ms/step - loss: 0.0097 - accuracy: 0.9987 - val_loss: 0.8429 - val_accuracy: 0.8211
Epoch 28/30
96/96 [==============================] - 15s 159ms/step - loss: 0.0072 - accuracy: 0.9990 - val_loss: 0.8050 - val_accuracy: 0.8358
Epoch 29/30
96/96 [==============================] - 15s 159ms/step - loss: 0.0125 - accuracy: 0.9964 - val_loss: 0.8644 - val_accuracy: 0.8152
Epoch 30/30
96/96 [==============================] - 15s 157ms/step - loss: 0.0091 - accuracy: 0.9987 - val_loss: 0.8397 - val_accuracy: 0.8358


0.2
WARNING:tensorflow:sample_weight modes were coerced from
  ...
    to  
  ['...']
WARNING:tensorflow:sample_weight modes were coerced from
  ...
    to  
  ['...']
Train for 96 steps, validate for 11 steps
Epoch 1/30
96/96 [==============================] - 19s 201ms/step - loss: 1.0421 - accuracy: 0.6421 - val_loss: 0.6269 - val_accuracy: 0.8152
Epoch 2/30
96/96 [==============================] - 15s 157ms/step - loss: 0.5996 - accuracy: 0.7862 - val_loss: 0.5930 - val_accuracy: 0.7830
Epoch 3/30
96/96 [==============================] - 15s 159ms/step - loss: 0.4595 - accuracy: 0.8390 - val_loss: 0.5409 - val_accuracy: 0.8152
Epoch 4/30
96/96 [==============================] - 15s 159ms/step - loss: 0.3463 - accuracy: 0.8866 - val_loss: 0.4990 - val_accuracy: 0.8182
Epoch 5/30
96/96 [==============================] - 16s 162ms/step - loss: 0.2764 - accuracy: 0.9113 - val_loss: 0.4907 - val_accuracy: 0.8446
Epoch 6/30
96/96 [==============================] - 15s 160ms/step - loss: 0.2235 - accuracy: 0.9299 - val_loss: 0.5487 - val_accuracy: 0.8270
Epoch 7/30
96/96 [==============================] - 15s 159ms/step - loss: 0.1744 - accuracy: 0.9505 - val_loss: 0.5264 - val_accuracy: 0.8152
Epoch 8/30
96/96 [==============================] - 16s 162ms/step - loss: 0.1422 - accuracy: 0.9645 - val_loss: 0.5496 - val_accuracy: 0.8299
Epoch 9/30
96/96 [==============================] - 16s 162ms/step - loss: 0.1153 - accuracy: 0.9690 - val_loss: 0.5453 - val_accuracy: 0.8152
Epoch 10/30
96/96 [==============================] - 15s 159ms/step - loss: 0.0955 - accuracy: 0.9769 - val_loss: 0.5783 - val_accuracy: 0.8152
Epoch 11/30
96/96 [==============================] - 15s 160ms/step - loss: 0.0775 - accuracy: 0.9834 - val_loss: 0.5469 - val_accuracy: 0.8270
Epoch 12/30
96/96 [==============================] - 15s 158ms/step - loss: 0.0595 - accuracy: 0.9886 - val_loss: 0.5578 - val_accuracy: 0.8446
Epoch 13/30
96/96 [==============================] - 15s 159ms/step - loss: 0.0514 - accuracy: 0.9922 - val_loss: 0.6008 - val_accuracy: 0.8416
Epoch 14/30
96/96 [==============================] - 15s 161ms/step - loss: 0.0491 - accuracy: 0.9909 - val_loss: 0.5807 - val_accuracy: 0.8358
Epoch 15/30
96/96 [==============================] - 15s 160ms/step - loss: 0.0407 - accuracy: 0.9948 - val_loss: 0.6196 - val_accuracy: 0.8270
Epoch 16/30
96/96 [==============================] - 15s 159ms/step - loss: 0.0384 - accuracy: 0.9928 - val_loss: 0.6152 - val_accuracy: 0.8387
Epoch 17/30
96/96 [==============================] - 15s 158ms/step - loss: 0.0330 - accuracy: 0.9935 - val_loss: 0.5942 - val_accuracy: 0.8416
Epoch 18/30
96/96 [==============================] - 15s 158ms/step - loss: 0.0286 - accuracy: 0.9932 - val_loss: 0.7247 - val_accuracy: 0.8152
Epoch 19/30
96/96 [==============================] - 15s 160ms/step - loss: 0.0319 - accuracy: 0.9941 - val_loss: 0.6897 - val_accuracy: 0.8387
Epoch 20/30
96/96 [==============================] - 15s 160ms/step - loss: 0.0289 - accuracy: 0.9941 - val_loss: 0.6490 - val_accuracy: 0.8270
Epoch 21/30
96/96 [==============================] - 15s 161ms/step - loss: 0.0329 - accuracy: 0.9928 - val_loss: 0.7535 - val_accuracy: 0.8152
Epoch 22/30
96/96 [==============================] - 15s 160ms/step - loss: 0.0310 - accuracy: 0.9938 - val_loss: 0.6456 - val_accuracy: 0.8270
Epoch 23/30
96/96 [==============================] - 15s 161ms/step - loss: 0.0259 - accuracy: 0.9954 - val_loss: 0.7003 - val_accuracy: 0.8328
Epoch 24/30
96/96 [==============================] - 15s 159ms/step - loss: 0.0195 - accuracy: 0.9961 - val_loss: 0.7779 - val_accuracy: 0.8094
Epoch 25/30
96/96 [==============================] - 15s 160ms/step - loss: 0.0291 - accuracy: 0.9932 - val_loss: 0.8031 - val_accuracy: 0.8065
Epoch 26/30
96/96 [==============================] - 15s 159ms/step - loss: 0.0256 - accuracy: 0.9932 - val_loss: 0.7989 - val_accuracy: 0.8182
Epoch 27/30
96/96 [==============================] - 15s 161ms/step - loss: 0.0254 - accuracy: 0.9935 - val_loss: 0.7228 - val_accuracy: 0.8416
Epoch 28/30
96/96 [==============================] - 15s 161ms/step - loss: 0.0217 - accuracy: 0.9948 - val_loss: 0.7898 - val_accuracy: 0.8094
Epoch 29/30
96/96 [==============================] - 15s 161ms/step - loss: 0.0190 - accuracy: 0.9964 - val_loss: 0.8253 - val_accuracy: 0.8152
Epoch 30/30
96/96 [==============================] - 15s 158ms/step - loss: 0.0267 - accuracy: 0.9928 - val_loss: 0.8274 - val_accuracy: 0.8065


0.5
WARNING:tensorflow:sample_weight modes were coerced from
  ...
    to  
  ['...']
WARNING:tensorflow:sample_weight modes were coerced from
  ...
    to  
  ['...']
Train for 96 steps, validate for 11 steps
Epoch 1/30
96/96 [==============================] - 19s 201ms/step - loss: 1.3015 - accuracy: 0.5668 - val_loss: 0.7597 - val_accuracy: 0.7302
Epoch 2/30
96/96 [==============================] - 15s 159ms/step - loss: 0.8380 - accuracy: 0.7190 - val_loss: 0.6347 - val_accuracy: 0.7977
Epoch 3/30
96/96 [==============================] - 15s 160ms/step - loss: 0.6791 - accuracy: 0.7598 - val_loss: 0.5996 - val_accuracy: 0.8035
Epoch 4/30
96/96 [==============================] - 16s 163ms/step - loss: 0.5698 - accuracy: 0.8044 - val_loss: 0.5472 - val_accuracy: 0.8152
Epoch 5/30
96/96 [==============================] - 15s 159ms/step - loss: 0.4974 - accuracy: 0.8331 - val_loss: 0.5318 - val_accuracy: 0.8240
Epoch 6/30
96/96 [==============================] - 16s 162ms/step - loss: 0.4530 - accuracy: 0.8429 - val_loss: 0.5004 - val_accuracy: 0.8387
Epoch 7/30
96/96 [==============================] - 15s 161ms/step - loss: 0.3973 - accuracy: 0.8664 - val_loss: 0.5179 - val_accuracy: 0.8211
Epoch 8/30
96/96 [==============================] - 16s 162ms/step - loss: 0.3331 - accuracy: 0.8866 - val_loss: 0.5019 - val_accuracy: 0.8328
Epoch 9/30
96/96 [==============================] - 15s 160ms/step - loss: 0.3214 - accuracy: 0.8869 - val_loss: 0.5001 - val_accuracy: 0.8270
Epoch 10/30
96/96 [==============================] - 16s 162ms/step - loss: 0.2866 - accuracy: 0.9035 - val_loss: 0.5753 - val_accuracy: 0.8123
Epoch 11/30
96/96 [==============================] - 15s 161ms/step - loss: 0.2766 - accuracy: 0.9045 - val_loss: 0.5196 - val_accuracy: 0.8328
Epoch 12/30
96/96 [==============================] - 16s 163ms/step - loss: 0.2353 - accuracy: 0.9188 - val_loss: 0.5102 - val_accuracy: 0.8416
Epoch 13/30
96/96 [==============================] - 15s 161ms/step - loss: 0.2174 - accuracy: 0.9241 - val_loss: 0.5461 - val_accuracy: 0.8270
Epoch 14/30
96/96 [==============================] - 15s 159ms/step - loss: 0.1872 - accuracy: 0.9345 - val_loss: 0.5343 - val_accuracy: 0.8416
Epoch 15/30
96/96 [==============================] - 15s 159ms/step - loss: 0.1835 - accuracy: 0.9384 - val_loss: 0.5541 - val_accuracy: 0.8299
Epoch 16/30
96/96 [==============================] - 15s 158ms/step - loss: 0.1647 - accuracy: 0.9505 - val_loss: 0.5677 - val_accuracy: 0.8240
Epoch 17/30
96/96 [==============================] - 15s 160ms/step - loss: 0.1602 - accuracy: 0.9475 - val_loss: 0.5600 - val_accuracy: 0.8387
Epoch 18/30
96/96 [==============================] - 15s 158ms/step - loss: 0.1437 - accuracy: 0.9550 - val_loss: 0.5300 - val_accuracy: 0.8270
Epoch 19/30
96/96 [==============================] - 15s 160ms/step - loss: 0.1383 - accuracy: 0.9550 - val_loss: 0.5526 - val_accuracy: 0.8358
Epoch 20/30
96/96 [==============================] - 15s 159ms/step - loss: 0.1418 - accuracy: 0.9495 - val_loss: 0.5425 - val_accuracy: 0.8446
Epoch 21/30
96/96 [==============================] - 15s 159ms/step - loss: 0.1214 - accuracy: 0.9615 - val_loss: 0.5780 - val_accuracy: 0.8534
Epoch 22/30
96/96 [==============================] - 15s 159ms/step - loss: 0.0991 - accuracy: 0.9684 - val_loss: 0.6000 - val_accuracy: 0.8358
Epoch 23/30
96/96 [==============================] - 15s 158ms/step - loss: 0.1028 - accuracy: 0.9664 - val_loss: 0.5995 - val_accuracy: 0.8065
Epoch 24/30
96/96 [==============================] - 15s 160ms/step - loss: 0.0974 - accuracy: 0.9674 - val_loss: 0.6249 - val_accuracy: 0.8094
Epoch 25/30
96/96 [==============================] - 15s 160ms/step - loss: 0.1086 - accuracy: 0.9602 - val_loss: 0.5936 - val_accuracy: 0.8299
Epoch 26/30
96/96 [==============================] - 15s 158ms/step - loss: 0.0951 - accuracy: 0.9697 - val_loss: 0.6485 - val_accuracy: 0.8240
Epoch 27/30
96/96 [==============================] - 16s 162ms/step - loss: 0.0916 - accuracy: 0.9694 - val_loss: 0.5958 - val_accuracy: 0.8328
Epoch 28/30
96/96 [==============================] - 16s 163ms/step - loss: 0.0821 - accuracy: 0.9729 - val_loss: 0.6749 - val_accuracy: 0.8328
Epoch 29/30
96/96 [==============================] - 15s 159ms/step - loss: 0.1033 - accuracy: 0.9602 - val_loss: 0.6781 - val_accuracy: 0.8270
Epoch 30/30
96/96 [==============================] - 15s 161ms/step - loss: 0.0970 - accuracy: 0.9645 - val_loss: 0.6907 - val_accuracy: 0.8299


0.8
WARNING:tensorflow:Large dropout rate: 0.8 (>0.5). In TensorFlow 2.x, dropout() uses dropout rate instead of keep_prob. Please ensure that this is intended.
WARNING:tensorflow:sample_weight modes were coerced from
  ...
    to  
  ['...']
WARNING:tensorflow:sample_weight modes were coerced from
  ...
    to  
  ['...']
Train for 96 steps, validate for 11 steps
Epoch 1/30
WARNING:tensorflow:Large dropout rate: 0.8 (>0.5). In TensorFlow 2.x, dropout() uses dropout rate instead of keep_prob. Please ensure that this is intended.
WARNING:tensorflow:Large dropout rate: 0.8 (>0.5). In TensorFlow 2.x, dropout() uses dropout rate instead of keep_prob. Please ensure that this is intended.
96/96 [==============================] - 19s 199ms/step - loss: 1.8235 - accuracy: 0.3879 - val_loss: 1.1518 - val_accuracy: 0.6158
Epoch 2/30
96/96 [==============================] - 15s 157ms/step - loss: 1.4182 - accuracy: 0.5010 - val_loss: 0.9966 - val_accuracy: 0.7155
Epoch 3/30
96/96 [==============================] - 15s 159ms/step - loss: 1.2934 - accuracy: 0.5310 - val_loss: 0.8925 - val_accuracy: 0.7331
Epoch 4/30
96/96 [==============================] - 15s 159ms/step - loss: 1.2507 - accuracy: 0.5417 - val_loss: 0.8229 - val_accuracy: 0.7683
Epoch 5/30
96/96 [==============================] - 15s 158ms/step - loss: 1.1791 - accuracy: 0.5818 - val_loss: 0.8120 - val_accuracy: 0.7537
Epoch 6/30
96/96 [==============================] - 15s 159ms/step - loss: 1.1207 - accuracy: 0.5896 - val_loss: 0.7615 - val_accuracy: 0.7625
Epoch 7/30
96/96 [==============================] - 15s 159ms/step - loss: 1.0704 - accuracy: 0.6043 - val_loss: 0.7264 - val_accuracy: 0.7918
Epoch 8/30
96/96 [==============================] - 15s 158ms/step - loss: 1.0273 - accuracy: 0.6261 - val_loss: 0.6978 - val_accuracy: 0.7742
Epoch 9/30
96/96 [==============================] - 15s 158ms/step - loss: 1.0470 - accuracy: 0.6102 - val_loss: 0.7046 - val_accuracy: 0.7918
Epoch 10/30
96/96 [==============================] - 15s 158ms/step - loss: 0.9995 - accuracy: 0.6157 - val_loss: 0.6775 - val_accuracy: 0.7947
Epoch 11/30
96/96 [==============================] - 15s 161ms/step - loss: 0.9725 - accuracy: 0.6375 - val_loss: 0.6745 - val_accuracy: 0.7977
Epoch 12/30
96/96 [==============================] - 15s 158ms/step - loss: 0.9577 - accuracy: 0.6323 - val_loss: 0.6670 - val_accuracy: 0.7918
Epoch 13/30
96/96 [==============================] - 16s 163ms/step - loss: 0.9401 - accuracy: 0.6392 - val_loss: 0.6382 - val_accuracy: 0.7918
Epoch 14/30
96/96 [==============================] - 15s 161ms/step - loss: 0.9195 - accuracy: 0.6477 - val_loss: 0.6157 - val_accuracy: 0.8065
Epoch 15/30
96/96 [==============================] - 16s 163ms/step - loss: 0.8835 - accuracy: 0.6662 - val_loss: 0.6243 - val_accuracy: 0.8006
Epoch 16/30
96/96 [==============================] - 15s 158ms/step - loss: 0.8784 - accuracy: 0.6591 - val_loss: 0.6263 - val_accuracy: 0.7889
Epoch 17/30
96/96 [==============================] - 15s 158ms/step - loss: 0.8439 - accuracy: 0.6780 - val_loss: 0.6110 - val_accuracy: 0.8035
Epoch 18/30
96/96 [==============================] - 15s 159ms/step - loss: 0.8262 - accuracy: 0.6757 - val_loss: 0.6072 - val_accuracy: 0.8065
Epoch 19/30
96/96 [==============================] - 15s 159ms/step - loss: 0.8425 - accuracy: 0.6698 - val_loss: 0.6029 - val_accuracy: 0.8094
Epoch 20/30
96/96 [==============================] - 15s 159ms/step - loss: 0.8072 - accuracy: 0.6913 - val_loss: 0.6007 - val_accuracy: 0.7889
Epoch 21/30
96/96 [==============================] - 16s 164ms/step - loss: 0.8056 - accuracy: 0.6920 - val_loss: 0.5905 - val_accuracy: 0.8035
Epoch 22/30
96/96 [==============================] - 16s 163ms/step - loss: 0.7889 - accuracy: 0.6842 - val_loss: 0.5832 - val_accuracy: 0.8035
Epoch 23/30
96/96 [==============================] - 16s 164ms/step - loss: 0.7635 - accuracy: 0.6965 - val_loss: 0.6038 - val_accuracy: 0.8094
Epoch 24/30
96/96 [==============================] - 15s 159ms/step - loss: 0.7756 - accuracy: 0.6962 - val_loss: 0.5772 - val_accuracy: 0.8065
Epoch 25/30
96/96 [==============================] - 15s 159ms/step - loss: 0.7670 - accuracy: 0.6995 - val_loss: 0.6003 - val_accuracy: 0.8094
Epoch 26/30
96/96 [==============================] - 15s 158ms/step - loss: 0.7333 - accuracy: 0.7066 - val_loss: 0.5757 - val_accuracy: 0.8211
Epoch 27/30
96/96 [==============================] - 15s 161ms/step - loss: 0.7398 - accuracy: 0.7080 - val_loss: 0.5493 - val_accuracy: 0.8240
Epoch 28/30
96/96 [==============================] - 16s 162ms/step - loss: 0.7143 - accuracy: 0.7171 - val_loss: 0.5323 - val_accuracy: 0.8328
Epoch 29/30
96/96 [==============================] - 16s 162ms/step - loss: 0.7117 - accuracy: 0.7122 - val_loss: 0.5473 - val_accuracy: 0.8094
Epoch 30/30
96/96 [==============================] - 16s 165ms/step - loss: 0.6907 - accuracy: 0.7256 - val_loss: 0.5628 - val_accuracy: 0.8299

In [16]python · cell 48

python

for droprate, hist in scores.items():
    plt.plot(hist['val_accuracy'], label=('val=%s' % droprate))

plt.ylim(0.78, 0.86)
plt.legend()

Output

<matplotlib.legend.Legend at 0x7feda76a0b38>

<Figure size 432x288 with 1 Axes>

In [23]python · cell 49

python

hist = scores[0.0]
plt.plot(hist['val_accuracy'], label=0.0)

hist = scores[0.2]
plt.plot(hist['val_accuracy'], label=0.2)

plt.legend()
#plt.plot(hist['accuracy'], label=('val=%s' % droprate))

Output

<matplotlib.legend.Legend at 0x7feda74902e8>

<Figure size 432x288 with 1 Axes>

8.10 Data augmentation

Different data augmentations
Training a model with augmentations
How to select data augmentations?

In [17]python · cell 51

python

train_gen = ImageDataGenerator(
    preprocessing_function=preprocess_input,
#     vertical_flip=True,
)

train_ds = train_gen.flow_from_directory(
    './clothing-dataset-small/train',
    target_size=(150, 150),
    batch_size=32
)

val_gen = ImageDataGenerator(preprocessing_function=preprocess_input)

val_ds = val_gen.flow_from_directory(
    './clothing-dataset-small/validation',
    target_size=(150, 150),
    batch_size=32,
    shuffle=False
)

Output

Found 3068 images belonging to 10 classes.
Found 341 images belonging to 10 classes.

In [18]python · cell 52

python

learning_rate = 0.001
size = 100
droprate = 0.2

model = make_model(
    learning_rate=learning_rate,
    size_inner=size,
    droprate=droprate
)

history = model.fit(train_ds, epochs=50, validation_data=val_ds)

Output

WARNING:tensorflow:sample_weight modes were coerced from
  ...
    to  
  ['...']
WARNING:tensorflow:sample_weight modes were coerced from
  ...
    to  
  ['...']
Train for 96 steps, validate for 11 steps
Epoch 1/50
96/96 [==============================] - 19s 200ms/step - loss: 1.0738 - accuracy: 0.6346 - val_loss: 0.6670 - val_accuracy: 0.7859
Epoch 2/50
96/96 [==============================] - 16s 162ms/step - loss: 0.6210 - accuracy: 0.7846 - val_loss: 0.5732 - val_accuracy: 0.8182
Epoch 3/50
96/96 [==============================] - 16s 162ms/step - loss: 0.4573 - accuracy: 0.8387 - val_loss: 0.5620 - val_accuracy: 0.8182
Epoch 4/50
96/96 [==============================] - 15s 161ms/step - loss: 0.3637 - accuracy: 0.8797 - val_loss: 0.5376 - val_accuracy: 0.8094
Epoch 5/50
96/96 [==============================] - 16s 162ms/step - loss: 0.2951 - accuracy: 0.9042 - val_loss: 0.5465 - val_accuracy: 0.8065
Epoch 6/50
96/96 [==============================] - 16s 162ms/step - loss: 0.2348 - accuracy: 0.9283 - val_loss: 0.6016 - val_accuracy: 0.7918
Epoch 7/50
96/96 [==============================] - 16s 163ms/step - loss: 0.1910 - accuracy: 0.9446 - val_loss: 0.5404 - val_accuracy: 0.8299
Epoch 8/50
96/96 [==============================] - 16s 162ms/step - loss: 0.1439 - accuracy: 0.9589 - val_loss: 0.5493 - val_accuracy: 0.8182
Epoch 9/50
96/96 [==============================] - 16s 163ms/step - loss: 0.1177 - accuracy: 0.9690 - val_loss: 0.5867 - val_accuracy: 0.8328
Epoch 10/50
96/96 [==============================] - 16s 163ms/step - loss: 0.1066 - accuracy: 0.9736 - val_loss: 0.5887 - val_accuracy: 0.8240
Epoch 11/50
96/96 [==============================] - 16s 164ms/step - loss: 0.0842 - accuracy: 0.9821 - val_loss: 0.6101 - val_accuracy: 0.8240
Epoch 12/50
83/96 [========================>.....] - ETA: 1s - loss: 0.0721 - accuracy: 0.9832

[0;31m---------------------------------------------------------------------------[0m
[0;31mKeyboardInterrupt[0m                         Traceback (most recent call last)
[0;32m<ipython-input-18-e333a961bbf9>[0m in [0;36m<module>[0;34m[0m
[1;32m      9[0m )
[1;32m     10[0m [0;34m[0m[0m
[0;32m---> 11[0;31m [0mhistory[0m [0;34m=[0m [0mmodel[0m[0;34m.[0m[0mfit[0m[0;34m([0m[0mtrain_ds[0m[0;34m,[0m [0mepochs[0m[0;34m=[0m[0;36m50[0m[0;34m,[0m [0mvalidation_data[0m[0;34m=[0m[0mval_ds[0m[0;34m)[0m[0;34m[0m[0;34m[0m[0m
[0m
[0;32m~/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training.py[0m in [0;36mfit[0;34m(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_freq, max_queue_size, workers, use_multiprocessing, **kwargs)[0m
[1;32m    823[0m         [0mmax_queue_size[0m[0;34m=[0m[0mmax_queue_size[0m[0;34m,[0m[0;34m[0m[0;34m[0m[0m
[1;32m    824[0m         [0mworkers[0m[0;34m=[0m[0mworkers[0m[0;34m,[0m[0;34m[0m[0;34m[0m[0m
[0;32m--> 825[0;31m         use_multiprocessing=use_multiprocessing)
[0m[1;32m    826[0m [0;34m[0m[0m
[1;32m    827[0m   def evaluate(self,

[0;32m~/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_v2.py[0m in [0;36mfit[0;34m(self, model, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_freq, max_queue_size, workers, use_multiprocessing, **kwargs)[0m
[1;32m    340[0m                 [0mmode[0m[0;34m=[0m[0mModeKeys[0m[0;34m.[0m[0mTRAIN[0m[0;34m,[0m[0;34m[0m[0;34m[0m[0m
[1;32m    341[0m                 [0mtraining_context[0m[0;34m=[0m[0mtraining_context[0m[0;34m,[0m[0;34m[0m[0;34m[0m[0m
[0;32m--> 342[0;31m                 total_epochs=epochs)
[0m[1;32m    343[0m             [0mcbks[0m[0;34m.[0m[0mmake_logs[0m[0;34m([0m[0mmodel[0m[0;34m,[0m [0mepoch_logs[0m[0;34m,[0m [0mtraining_result[0m[0;34m,[0m [0mModeKeys[0m[0;34m.[0m[0mTRAIN[0m[0;34m)[0m[0;34m[0m[0;34m[0m[0m
[1;32m    344[0m [0;34m[0m[0m

[0;32m~/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_v2.py[0m in [0;36mrun_one_epoch[0;34m(model, iterator, execution_function, dataset_size, batch_size, strategy, steps_per_epoch, num_samples, mode, training_context, total_epochs)[0m
[1;32m    126[0m         step=step, mode=mode, size=current_batch_size) as batch_logs:
[1;32m    127[0m       [0;32mtry[0m[0;34m:[0m[0;34m[0m[0;34m[0m[0m
[0;32m--> 128[0;31m         [0mbatch_outs[0m [0;34m=[0m [0mexecution_function[0m[0;34m([0m[0miterator[0m[0;34m)[0m[0;34m[0m[0;34m[0m[0m
[0m[1;32m    129[0m       [0;32mexcept[0m [0;34m([0m[0mStopIteration[0m[0;34m,[0m [0merrors[0m[0;34m.[0m[0mOutOfRangeError[0m[0;34m)[0m[0;34m:[0m[0;34m[0m[0;34m[0m[0m
[1;32m    130[0m         [0;31m# TODO(kaftan): File bug about tf function and errors.OutOfRangeError?[0m[0;34m[0m[0;34m[0m[0;34m[0m[0m

[0;32m~/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_v2_utils.py[0m in [0;36mexecution_function[0;34m(input_fn)[0m
[1;32m     96[0m     [0;31m# `numpy` translates Tensors to values in Eager mode.[0m[0;34m[0m[0;34m[0m[0;34m[0m[0m
[1;32m     97[0m     return nest.map_structure(_non_none_constant_value,
[0;32m---> 98[0;31m                               distributed_function(input_fn))
[0m[1;32m     99[0m [0;34m[0m[0m
[1;32m    100[0m   [0;32mreturn[0m [0mexecution_function[0m[0;34m[0m[0;34m[0m[0m

[0;32m~/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/tensorflow_core/python/eager/def_function.py[0m in [0;36m__call__[0;34m(self, *args, **kwds)[0m
[1;32m    566[0m         [0mxla_context[0m[0;34m.[0m[0mExit[0m[0;34m([0m[0;34m)[0m[0;34m[0m[0;34m[0m[0m
[1;32m    567[0m     [0;32melse[0m[0;34m:[0m[0;34m[0m[0;34m[0m[0m
[0;32m--> 568[0;31m       [0mresult[0m [0;34m=[0m [0mself[0m[0;34m.[0m[0m_call[0m[0;34m([0m[0;34m*[0m[0margs[0m[0;34m,[0m [0;34m**[0m[0mkwds[0m[0;34m)[0m[0;34m[0m[0;34m[0m[0m
[0m[1;32m    569[0m [0;34m[0m[0m
[1;32m    570[0m     [0;32mif[0m [0mtracing_count[0m [0;34m==[0m [0mself[0m[0;34m.[0m[0m_get_tracing_count[0m[0;34m([0m[0;34m)[0m[0;34m:[0m[0;34m[0m[0;34m[0m[0m

[0;32m~/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/tensorflow_core/python/eager/def_function.py[0m in [0;36m_call[0;34m(self, *args, **kwds)[0m
[1;32m    597[0m       [0;31m# In this case we have created variables on the first call, so we run the[0m[0;34m[0m[0;34m[0m[0;34m[0m[0m
[1;32m    598[0m       [0;31m# defunned version which is guaranteed to never create variables.[0m[0;34m[0m[0;34m[0m[0;34m[0m[0m
[0;32m--> 599[0;31m       [0;32mreturn[0m [0mself[0m[0;34m.[0m[0m_stateless_fn[0m[0;34m([0m[0;34m*[0m[0margs[0m[0;34m,[0m [0;34m**[0m[0mkwds[0m[0;34m)[0m  [0;31m# pylint: disable=not-callable[0m[0;34m[0m[0;34m[0m[0m
[0m[1;32m    600[0m     [0;32melif[0m [0mself[0m[0;34m.[0m[0m_stateful_fn[0m [0;32mis[0m [0;32mnot[0m [0;32mNone[0m[0;34m:[0m[0;34m[0m[0;34m[0m[0m
[1;32m    601[0m       [0;31m# Release the lock early so that multiple threads can perform the call[0m[0;34m[0m[0;34m[0m[0;34m[0m[0m

[0;32m~/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py[0m in [0;36m__call__[0;34m(self, *args, **kwargs)[0m
[1;32m   2361[0m     [0;32mwith[0m [0mself[0m[0;34m.[0m[0m_lock[0m[0;34m:[0m[0;34m[0m[0;34m[0m[0m
[1;32m   2362[0m       [0mgraph_function[0m[0;34m,[0m [0margs[0m[0;34m,[0m [0mkwargs[0m [0;34m=[0m [0mself[0m[0;34m.[0m[0m_maybe_define_function[0m[0;34m([0m[0margs[0m[0;34m,[0m [0mkwargs[0m[0;34m)[0m[0;34m[0m[0;34m[0m[0m
[0;32m-> 2363[0;31m     [0;32mreturn[0m [0mgraph_function[0m[0;34m.[0m[0m_filtered_call[0m[0;34m([0m[0margs[0m[0;34m,[0m [0mkwargs[0m[0;34m)[0m  [0;31m# pylint: disable=protected-access[0m[0;34m[0m[0;34m[0m[0m
[0m[1;32m   2364[0m [0;34m[0m[0m
[1;32m   2365[0m   [0;34m@[0m[0mproperty[0m[0;34m[0m[0;34m[0m[0m

[0;32m~/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py[0m in [0;36m_filtered_call[0;34m(self, args, kwargs)[0m
[1;32m   1609[0m          if isinstance(t, (ops.Tensor,
[1;32m   1610[0m                            resource_variable_ops.BaseResourceVariable))),
[0;32m-> 1611[0;31m         self.captured_inputs)
[0m[1;32m   1612[0m [0;34m[0m[0m
[1;32m   1613[0m   [0;32mdef[0m [0m_call_flat[0m[0;34m([0m[0mself[0m[0;34m,[0m [0margs[0m[0;34m,[0m [0mcaptured_inputs[0m[0;34m,[0m [0mcancellation_manager[0m[0;34m=[0m[0;32mNone[0m[0;34m)[0m[0;34m:[0m[0;34m[0m[0;34m[0m[0m

[0;32m~/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py[0m in [0;36m_call_flat[0;34m(self, args, captured_inputs, cancellation_manager)[0m
[1;32m   1690[0m       [0;31m# No tape is watching; skip to running the function.[0m[0;34m[0m[0;34m[0m[0;34m[0m[0m
[1;32m   1691[0m       return self._build_call_outputs(self._inference_function.call(
[0;32m-> 1692[0;31m           ctx, args, cancellation_manager=cancellation_manager))
[0m[1;32m   1693[0m     forward_backward = self._select_forward_and_backward_functions(
[1;32m   1694[0m         [0margs[0m[0;34m,[0m[0;34m[0m[0;34m[0m[0m

[0;32m~/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py[0m in [0;36mcall[0;34m(self, ctx, args, cancellation_manager)[0m
[1;32m    543[0m               [0minputs[0m[0;34m=[0m[0margs[0m[0;34m,[0m[0;34m[0m[0;34m[0m[0m
[1;32m    544[0m               [0mattrs[0m[0;34m=[0m[0;34m([0m[0;34m"executor_type"[0m[0;34m,[0m [0mexecutor_type[0m[0;34m,[0m [0;34m"config_proto"[0m[0;34m,[0m [0mconfig[0m[0;34m)[0m[0;34m,[0m[0;34m[0m[0;34m[0m[0m
[0;32m--> 545[0;31m               ctx=ctx)
[0m[1;32m    546[0m         [0;32melse[0m[0;34m:[0m[0;34m[0m[0;34m[0m[0m
[1;32m    547[0m           outputs = execute.execute_with_cancellation(

[0;32m~/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/tensorflow_core/python/eager/execute.py[0m in [0;36mquick_execute[0;34m(op_name, num_outputs, inputs, attrs, ctx, name)[0m
[1;32m     59[0m     tensors = pywrap_tensorflow.TFE_Py_Execute(ctx._handle, device_name,
[1;32m     60[0m                                                [0mop_name[0m[0;34m,[0m [0minputs[0m[0;34m,[0m [0mattrs[0m[0;34m,[0m[0;34m[0m[0;34m[0m[0m
[0;32m---> 61[0;31m                                                num_outputs)
[0m[1;32m     62[0m   [0;32mexcept[0m [0mcore[0m[0;34m.[0m[0m_NotOkStatusException[0m [0;32mas[0m [0me[0m[0;34m:[0m[0;34m[0m[0;34m[0m[0m
[1;32m     63[0m     [0;32mif[0m [0mname[0m [0;32mis[0m [0;32mnot[0m [0;32mNone[0m[0;34m:[0m[0;34m[0m[0;34m[0m[0m

[0;31mKeyboardInterrupt[0m:

In [12]python · cell 53

python

hist = history.history
plt.plot(hist['val_accuracy'], label='val')
plt.plot(hist['accuracy'], label='train')

plt.legend()

Output

<matplotlib.legend.Legend at 0x7fc75018f438>

<Figure size 432x288 with 1 Axes>

8.11 Training a larger model

Train a 299x299 model

In [19]python · cell 55

python

def make_model(input_size=150, learning_rate=0.01, size_inner=100,
               droprate=0.5):

    base_model = Xception(
        weights='imagenet',
        include_top=False,
        input_shape=(input_size, input_size, 3)
    )

    base_model.trainable = False

    #########################################

    inputs = keras.Input(shape=(input_size, input_size, 3))
    base = base_model(inputs, training=False)
    vectors = keras.layers.GlobalAveragePooling2D()(base)
    
    inner = keras.layers.Dense(size_inner, activation='relu')(vectors)
    drop = keras.layers.Dropout(droprate)(inner)
    
    outputs = keras.layers.Dense(10)(drop)
    
    model = keras.Model(inputs, outputs)
    
    #########################################

    optimizer = keras.optimizers.Adam(learning_rate=learning_rate)
    loss = keras.losses.CategoricalCrossentropy(from_logits=True)

    model.compile(
        optimizer=optimizer,
        loss=loss,
        metrics=['accuracy']
    )
    
    return model

In [25]python · cell 56

python

input_size = 299

In [38]python · cell 57

python

train_gen = ImageDataGenerator(
    preprocessing_function=preprocess_input,
    shear_range=10,
    zoom_range=0.1,
    horizontal_flip=True
)

train_ds = train_gen.flow_from_directory(
    './clothing-dataset-small/train',
    target_size=(input_size, input_size),
    batch_size=32
)


val_gen = ImageDataGenerator(preprocessing_function=preprocess_input)

val_ds = train_gen.flow_from_directory(
    './clothing-dataset-small/validation',
    target_size=(input_size, input_size),
    batch_size=32,
    shuffle=False
)

Output

Found 3068 images belonging to 10 classes.
Found 341 images belonging to 10 classes.

In [39]python · cell 58

python

checkpoint = keras.callbacks.ModelCheckpoint(
    'xception_v4_1_{epoch:02d}_{val_accuracy:.3f}.h5',
    save_best_only=True,
    monitor='val_accuracy',
    mode='max'
)

In [40]python · cell 59

python

learning_rate = 0.0005
size = 100
droprate = 0.2

model = make_model(
    input_size=input_size,
    learning_rate=learning_rate,
    size_inner=size,
    droprate=droprate
)

history = model.fit(train_ds, epochs=50, validation_data=val_ds,
                   callbacks=[checkpoint])

Output

WARNING:tensorflow:sample_weight modes were coerced from
  ...
    to  
  ['...']
WARNING:tensorflow:sample_weight modes were coerced from
  ...
    to  
  ['...']
Train for 96 steps, validate for 11 steps
Epoch 1/50
96/96 [==============================] - 78s 816ms/step - loss: 1.0387 - accuracy: 0.6793 - val_loss: 0.5567 - val_accuracy: 0.8211
Epoch 2/50
96/96 [==============================] - 77s 805ms/step - loss: 0.5472 - accuracy: 0.8214 - val_loss: 0.4331 - val_accuracy: 0.8622
Epoch 3/50
96/96 [==============================] - 74s 772ms/step - loss: 0.4525 - accuracy: 0.8491 - val_loss: 0.4360 - val_accuracy: 0.8504
Epoch 4/50
96/96 [==============================] - 75s 778ms/step - loss: 0.4000 - accuracy: 0.8605 - val_loss: 0.3728 - val_accuracy: 0.8856
Epoch 5/50
96/96 [==============================] - 75s 785ms/step - loss: 0.3586 - accuracy: 0.8853 - val_loss: 0.3832 - val_accuracy: 0.8768
Epoch 6/50
96/96 [==============================] - 75s 777ms/step - loss: 0.3261 - accuracy: 0.8934 - val_loss: 0.3828 - val_accuracy: 0.8827
Epoch 7/50
96/96 [==============================] - 75s 778ms/step - loss: 0.3105 - accuracy: 0.8957 - val_loss: 0.3707 - val_accuracy: 0.8768
Epoch 8/50
96/96 [==============================] - 74s 772ms/step - loss: 0.2857 - accuracy: 0.9055 - val_loss: 0.3801 - val_accuracy: 0.8592
Epoch 9/50
96/96 [==============================] - 74s 773ms/step - loss: 0.2686 - accuracy: 0.9058 - val_loss: 0.3884 - val_accuracy: 0.8563
Epoch 10/50
96/96 [==============================] - 74s 776ms/step - loss: 0.2575 - accuracy: 0.9143 - val_loss: 0.3733 - val_accuracy: 0.8710
Epoch 11/50
96/96 [==============================] - 75s 778ms/step - loss: 0.2372 - accuracy: 0.9208 - val_loss: 0.3582 - val_accuracy: 0.8680
Epoch 12/50
96/96 [==============================] - 74s 776ms/step - loss: 0.2231 - accuracy: 0.9250 - val_loss: 0.3572 - val_accuracy: 0.8798
Epoch 13/50
96/96 [==============================] - 75s 783ms/step - loss: 0.2075 - accuracy: 0.9289 - val_loss: 0.3355 - val_accuracy: 0.9032
Epoch 14/50
96/96 [==============================] - 74s 772ms/step - loss: 0.2042 - accuracy: 0.9325 - val_loss: 0.3347 - val_accuracy: 0.8915
Epoch 15/50
96/96 [==============================] - 76s 796ms/step - loss: 0.1941 - accuracy: 0.9312 - val_loss: 0.3390 - val_accuracy: 0.8886
Epoch 16/50
96/96 [==============================] - 74s 773ms/step - loss: 0.1773 - accuracy: 0.9439 - val_loss: 0.3915 - val_accuracy: 0.8768
Epoch 17/50
96/96 [==============================] - 76s 789ms/step - loss: 0.1729 - accuracy: 0.9420 - val_loss: 0.3564 - val_accuracy: 0.8915
Epoch 18/50
96/96 [==============================] - 75s 777ms/step - loss: 0.1632 - accuracy: 0.9439 - val_loss: 0.3387 - val_accuracy: 0.8944
Epoch 19/50
96/96 [==============================] - 75s 783ms/step - loss: 0.1548 - accuracy: 0.9462 - val_loss: 0.3415 - val_accuracy: 0.8768
Epoch 20/50
96/96 [==============================] - 74s 775ms/step - loss: 0.1484 - accuracy: 0.9527 - val_loss: 0.3625 - val_accuracy: 0.8651
Epoch 21/50
96/96 [==============================] - 75s 781ms/step - loss: 0.1395 - accuracy: 0.9557 - val_loss: 0.3616 - val_accuracy: 0.8739
Epoch 22/50
25/96 [======>.......................] - ETA: 51s - loss: 0.1452 - accuracy: 0.9583

[0;31m---------------------------------------------------------------------------[0m
[0;31mKeyboardInterrupt[0m                         Traceback (most recent call last)
[0;32m~/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_v2.py[0m in [0;36mon_epoch[0;34m(self, epoch, mode)[0m
[1;32m    766[0m     [0;32mtry[0m[0;34m:[0m[0;34m[0m[0;34m[0m[0m
[0;32m--> 767[0;31m       [0;32myield[0m [0mepoch_logs[0m[0;34m[0m[0;34m[0m[0m
[0m[1;32m    768[0m     [0;32mfinally[0m[0;34m:[0m[0;34m[0m[0;34m[0m[0m

[0;32m~/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_v2.py[0m in [0;36mfit[0;34m(self, model, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_freq, max_queue_size, workers, use_multiprocessing, **kwargs)[0m
[1;32m    341[0m                 [0mtraining_context[0m[0;34m=[0m[0mtraining_context[0m[0;34m,[0m[0;34m[0m[0;34m[0m[0m
[0;32m--> 342[0;31m                 total_epochs=epochs)
[0m[1;32m    343[0m             [0mcbks[0m[0;34m.[0m[0mmake_logs[0m[0;34m([0m[0mmodel[0m[0;34m,[0m [0mepoch_logs[0m[0;34m,[0m [0mtraining_result[0m[0;34m,[0m [0mModeKeys[0m[0;34m.[0m[0mTRAIN[0m[0;34m)[0m[0;34m[0m[0;34m[0m[0m

[0;32m~/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_v2.py[0m in [0;36mrun_one_epoch[0;34m(model, iterator, execution_function, dataset_size, batch_size, strategy, steps_per_epoch, num_samples, mode, training_context, total_epochs)[0m
[1;32m    127[0m       [0;32mtry[0m[0;34m:[0m[0;34m[0m[0;34m[0m[0m
[0;32m--> 128[0;31m         [0mbatch_outs[0m [0;34m=[0m [0mexecution_function[0m[0;34m([0m[0miterator[0m[0;34m)[0m[0;34m[0m[0;34m[0m[0m
[0m[1;32m    129[0m       [0;32mexcept[0m [0;34m([0m[0mStopIteration[0m[0;34m,[0m [0merrors[0m[0;34m.[0m[0mOutOfRangeError[0m[0;34m)[0m[0;34m:[0m[0;34m[0m[0;34m[0m[0m

[0;32m~/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_v2_utils.py[0m in [0;36mexecution_function[0;34m(input_fn)[0m
[1;32m     97[0m     return nest.map_structure(_non_none_constant_value,
[0;32m---> 98[0;31m                               distributed_function(input_fn))
[0m[1;32m     99[0m [0;34m[0m[0m

[0;32m~/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/tensorflow_core/python/eager/def_function.py[0m in [0;36m__call__[0;34m(self, *args, **kwds)[0m
[1;32m    567[0m     [0;32melse[0m[0;34m:[0m[0;34m[0m[0;34m[0m[0m
[0;32m--> 568[0;31m       [0mresult[0m [0;34m=[0m [0mself[0m[0;34m.[0m[0m_call[0m[0;34m([0m[0;34m*[0m[0margs[0m[0;34m,[0m [0;34m**[0m[0mkwds[0m[0;34m)[0m[0;34m[0m[0;34m[0m[0m
[0m[1;32m    569[0m [0;34m[0m[0m

[0;32m~/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/tensorflow_core/python/eager/def_function.py[0m in [0;36m_call[0;34m(self, *args, **kwds)[0m
[1;32m    598[0m       [0;31m# defunned version which is guaranteed to never create variables.[0m[0;34m[0m[0;34m[0m[0;34m[0m[0m
[0;32m--> 599[0;31m       [0;32mreturn[0m [0mself[0m[0;34m.[0m[0m_stateless_fn[0m[0;34m([0m[0;34m*[0m[0margs[0m[0;34m,[0m [0;34m**[0m[0mkwds[0m[0;34m)[0m  [0;31m# pylint: disable=not-callable[0m[0;34m[0m[0;34m[0m[0m
[0m[1;32m    600[0m     [0;32melif[0m [0mself[0m[0;34m.[0m[0m_stateful_fn[0m [0;32mis[0m [0;32mnot[0m [0;32mNone[0m[0;34m:[0m[0;34m[0m[0;34m[0m[0m

[0;32m~/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py[0m in [0;36m__call__[0;34m(self, *args, **kwargs)[0m
[1;32m   2362[0m       [0mgraph_function[0m[0;34m,[0m [0margs[0m[0;34m,[0m [0mkwargs[0m [0;34m=[0m [0mself[0m[0;34m.[0m[0m_maybe_define_function[0m[0;34m([0m[0margs[0m[0;34m,[0m [0mkwargs[0m[0;34m)[0m[0;34m[0m[0;34m[0m[0m
[0;32m-> 2363[0;31m     [0;32mreturn[0m [0mgraph_function[0m[0;34m.[0m[0m_filtered_call[0m[0;34m([0m[0margs[0m[0;34m,[0m [0mkwargs[0m[0;34m)[0m  [0;31m# pylint: disable=protected-access[0m[0;34m[0m[0;34m[0m[0m
[0m[1;32m   2364[0m [0;34m[0m[0m

[0;32m~/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py[0m in [0;36m_filtered_call[0;34m(self, args, kwargs)[0m
[1;32m   1610[0m                            resource_variable_ops.BaseResourceVariable))),
[0;32m-> 1611[0;31m         self.captured_inputs)
[0m[1;32m   1612[0m [0;34m[0m[0m

[0;32m~/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py[0m in [0;36m_call_flat[0;34m(self, args, captured_inputs, cancellation_manager)[0m
[1;32m   1691[0m       return self._build_call_outputs(self._inference_function.call(
[0;32m-> 1692[0;31m           ctx, args, cancellation_manager=cancellation_manager))
[0m[1;32m   1693[0m     forward_backward = self._select_forward_and_backward_functions(

[0;32m~/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py[0m in [0;36mcall[0;34m(self, ctx, args, cancellation_manager)[0m
[1;32m    544[0m               [0mattrs[0m[0;34m=[0m[0;34m([0m[0;34m"executor_type"[0m[0;34m,[0m [0mexecutor_type[0m[0;34m,[0m [0;34m"config_proto"[0m[0;34m,[0m [0mconfig[0m[0;34m)[0m[0;34m,[0m[0;34m[0m[0;34m[0m[0m
[0;32m--> 545[0;31m               ctx=ctx)
[0m[1;32m    546[0m         [0;32melse[0m[0;34m:[0m[0;34m[0m[0;34m[0m[0m

[0;32m~/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/tensorflow_core/python/eager/execute.py[0m in [0;36mquick_execute[0;34m(op_name, num_outputs, inputs, attrs, ctx, name)[0m
[1;32m     60[0m                                                [0mop_name[0m[0;34m,[0m [0minputs[0m[0;34m,[0m [0mattrs[0m[0;34m,[0m[0;34m[0m[0;34m[0m[0m
[0;32m---> 61[0;31m                                                num_outputs)
[0m[1;32m     62[0m   [0;32mexcept[0m [0mcore[0m[0;34m.[0m[0m_NotOkStatusException[0m [0;32mas[0m [0me[0m[0;34m:[0m[0;34m[0m[0;34m[0m[0m

[0;31mKeyboardInterrupt[0m: 

During handling of the above exception, another exception occurred:

[0;31mKeyError[0m                                  Traceback (most recent call last)
[0;32m<ipython-input-40-89b8717352a9>[0m in [0;36m<module>[0;34m[0m
[1;32m     11[0m [0;34m[0m[0m
[1;32m     12[0m history = model.fit(train_ds, epochs=50, validation_data=val_ds,
[0;32m---> 13[0;31m                    callbacks=[checkpoint])
[0m
[0;32m~/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training.py[0m in [0;36mfit[0;34m(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_freq, max_queue_size, workers, use_multiprocessing, **kwargs)[0m
[1;32m    823[0m         [0mmax_queue_size[0m[0;34m=[0m[0mmax_queue_size[0m[0;34m,[0m[0;34m[0m[0;34m[0m[0m
[1;32m    824[0m         [0mworkers[0m[0;34m=[0m[0mworkers[0m[0;34m,[0m[0;34m[0m[0;34m[0m[0m
[0;32m--> 825[0;31m         use_multiprocessing=use_multiprocessing)
[0m[1;32m    826[0m [0;34m[0m[0m
[1;32m    827[0m   def evaluate(self,

[0;32m~/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_v2.py[0m in [0;36mfit[0;34m(self, model, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_freq, max_queue_size, workers, use_multiprocessing, **kwargs)[0m
[1;32m    395[0m                       total_epochs=1)
[1;32m    396[0m                   cbks.make_logs(model, epoch_logs, eval_result, ModeKeys.TEST,
[0;32m--> 397[0;31m                                  prefix='val_')
[0m[1;32m    398[0m [0;34m[0m[0m
[1;32m    399[0m     [0;32mreturn[0m [0mmodel[0m[0;34m.[0m[0mhistory[0m[0;34m[0m[0;34m[0m[0m

[0;32m~/anaconda3/envs/tensorflow2_p36/lib/python3.6/contextlib.py[0m in [0;36m__exit__[0;34m(self, type, value, traceback)[0m
[1;32m     97[0m                 [0mvalue[0m [0;34m=[0m [0mtype[0m[0;34m([0m[0;34m)[0m[0;34m[0m[0;34m[0m[0m
[1;32m     98[0m             [0;32mtry[0m[0;34m:[0m[0;34m[0m[0;34m[0m[0m
[0;32m---> 99[0;31m                 [0mself[0m[0;34m.[0m[0mgen[0m[0;34m.[0m[0mthrow[0m[0;34m([0m[0mtype[0m[0;34m,[0m [0mvalue[0m[0;34m,[0m [0mtraceback[0m[0;34m)[0m[0;34m[0m[0;34m[0m[0m
[0m[1;32m    100[0m             [0;32mexcept[0m [0mStopIteration[0m [0;32mas[0m [0mexc[0m[0;34m:[0m[0;34m[0m[0;34m[0m[0m
[1;32m    101[0m                 [0;31m# Suppress StopIteration *unless* it's the same exception that[0m[0;34m[0m[0;34m[0m[0;34m[0m[0m

[0;32m~/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_v2.py[0m in [0;36mon_epoch[0;34m(self, epoch, mode)[0m
[1;32m    769[0m       [0;32mif[0m [0mmode[0m [0;34m==[0m [0mModeKeys[0m[0;34m.[0m[0mTRAIN[0m[0;34m:[0m[0;34m[0m[0;34m[0m[0m
[1;32m    770[0m         [0;31m# Epochs only apply to `fit`.[0m[0;34m[0m[0;34m[0m[0;34m[0m[0m
[0;32m--> 771[0;31m         [0mself[0m[0;34m.[0m[0mcallbacks[0m[0;34m.[0m[0mon_epoch_end[0m[0;34m([0m[0mepoch[0m[0;34m,[0m [0mepoch_logs[0m[0;34m)[0m[0;34m[0m[0;34m[0m[0m
[0m[1;32m    772[0m       [0mself[0m[0;34m.[0m[0mprogbar[0m[0;34m.[0m[0mon_epoch_end[0m[0;34m([0m[0mepoch[0m[0;34m,[0m [0mepoch_logs[0m[0;34m)[0m[0;34m[0m[0;34m[0m[0m
[1;32m    773[0m [0;34m[0m[0m

[0;32m~/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/tensorflow_core/python/keras/callbacks.py[0m in [0;36mon_epoch_end[0;34m(self, epoch, logs)[0m
[1;32m    300[0m     [0mlogs[0m [0;34m=[0m [0mlogs[0m [0;32mor[0m [0;34m{[0m[0;34m}[0m[0;34m[0m[0;34m[0m[0m
[1;32m    301[0m     [0;32mfor[0m [0mcallback[0m [0;32min[0m [0mself[0m[0;34m.[0m[0mcallbacks[0m[0;34m:[0m[0;34m[0m[0;34m[0m[0m
[0;32m--> 302[0;31m       [0mcallback[0m[0;34m.[0m[0mon_epoch_end[0m[0;34m([0m[0mepoch[0m[0;34m,[0m [0mlogs[0m[0;34m)[0m[0;34m[0m[0;34m[0m[0m
[0m[1;32m    303[0m [0;34m[0m[0m
[1;32m    304[0m   [0;32mdef[0m [0mon_train_batch_begin[0m[0;34m([0m[0mself[0m[0;34m,[0m [0mbatch[0m[0;34m,[0m [0mlogs[0m[0;34m=[0m[0;32mNone[0m[0;34m)[0m[0;34m:[0m[0;34m[0m[0;34m[0m[0m

[0;32m~/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/tensorflow_core/python/keras/callbacks.py[0m in [0;36mon_epoch_end[0;34m(self, epoch, logs)[0m
[1;32m    990[0m           [0mself[0m[0;34m.[0m[0m_save_model[0m[0;34m([0m[0mepoch[0m[0;34m=[0m[0mepoch[0m[0;34m,[0m [0mlogs[0m[0;34m=[0m[0mlogs[0m[0;34m)[0m[0;34m[0m[0;34m[0m[0m
[1;32m    991[0m       [0;32melse[0m[0;34m:[0m[0;34m[0m[0;34m[0m[0m
[0;32m--> 992[0;31m         [0mself[0m[0;34m.[0m[0m_save_model[0m[0;34m([0m[0mepoch[0m[0;34m=[0m[0mepoch[0m[0;34m,[0m [0mlogs[0m[0;34m=[0m[0mlogs[0m[0;34m)[0m[0;34m[0m[0;34m[0m[0m
[0m[1;32m    993[0m     [0;32mif[0m [0mself[0m[0;34m.[0m[0mmodel[0m[0;34m.[0m[0m_in_multi_worker_mode[0m[0;34m([0m[0;34m)[0m[0;34m:[0m[0;34m[0m[0;34m[0m[0m
[1;32m    994[0m       [0;31m# For multi-worker training, back up the weights and current training[0m[0;34m[0m[0;34m[0m[0;34m[0m[0m

[0;32m~/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/tensorflow_core/python/keras/callbacks.py[0m in [0;36m_save_model[0;34m(self, epoch, logs)[0m
[1;32m   1009[0m                   int) or self.epochs_since_last_save >= self.period:
[1;32m   1010[0m       [0mself[0m[0;34m.[0m[0mepochs_since_last_save[0m [0;34m=[0m [0;36m0[0m[0;34m[0m[0;34m[0m[0m
[0;32m-> 1011[0;31m       [0mfilepath[0m [0;34m=[0m [0mself[0m[0;34m.[0m[0m_get_file_path[0m[0;34m([0m[0mepoch[0m[0;34m,[0m [0mlogs[0m[0;34m)[0m[0;34m[0m[0;34m[0m[0m
[0m[1;32m   1012[0m [0;34m[0m[0m
[1;32m   1013[0m       [0;32mtry[0m[0;34m:[0m[0;34m[0m[0;34m[0m[0m

[0;32m~/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/tensorflow_core/python/keras/callbacks.py[0m in [0;36m_get_file_path[0;34m(self, epoch, logs)[0m
[1;32m   1053[0m     if not self.model._in_multi_worker_mode(
[1;32m   1054[0m     ) or multi_worker_util.should_save_checkpoint():
[0;32m-> 1055[0;31m       [0;32mreturn[0m [0mself[0m[0;34m.[0m[0mfilepath[0m[0;34m.[0m[0mformat[0m[0;34m([0m[0mepoch[0m[0;34m=[0m[0mepoch[0m [0;34m+[0m [0;36m1[0m[0;34m,[0m [0;34m**[0m[0mlogs[0m[0;34m)[0m[0;34m[0m[0;34m[0m[0m
[0m[1;32m   1056[0m     [0;32melse[0m[0;34m:[0m[0;34m[0m[0;34m[0m[0m
[1;32m   1057[0m       [0;31m# If this is multi-worker training, and this worker should not[0m[0;34m[0m[0;34m[0m[0;34m[0m[0m

[0;31mKeyError[0m: 'val_accuracy'

8.12 Using the model

Loading the model
Evaluating the model
Getting predictions

In [3]python · cell 61

python

import tensorflow as tf
from tensorflow import keras

In [13]python · cell 62

python

from tensorflow.keras.preprocessing.image import ImageDataGenerator
from tensorflow.keras.preprocessing.image import load_img

from tensorflow.keras.applications.xception import preprocess_input

In [7]python · cell 63

python

test_gen = ImageDataGenerator(preprocessing_function=preprocess_input)

test_ds = test_gen.flow_from_directory(
    './clothing-dataset-small/test',
    target_size=(299, 299),
    batch_size=32,
    shuffle=False
)

Output

Found 372 images belonging to 10 classes.

In [11]python · cell 64

python

model = keras.models.load_model('xception_v4_1_13_0.903.h5')

In [10]python · cell 65

python

model.evaluate(test_ds)

Output

WARNING:tensorflow:sample_weight modes were coerced from
  ...
    to  
  ['...']
12/12 [==============================] - 8s 645ms/step - loss: 0.2939 - accuracy: 0.8978

[0.29389633300403756, 0.89784944]

In [12]python · cell 66

python

path = 'clothing-dataset-small/test/pants/c8d21106-bbdb-4e8d-83e4-bf3d14e54c16.jpg'

In [15]python · cell 67

python

img = load_img(path, target_size=(299, 299))

In [16]python · cell 68

python

import numpy as np

In [18]python · cell 69

python

x = np.array(img)
X = np.array([x])
X.shape

Output

(1, 299, 299, 3)

In [21]python · cell 70

python

X = preprocess_input(X)

In [23]python · cell 71

python

pred = model.predict(X)

In [25]python · cell 72

python

classes = [
    'dress',
    'hat',
    'longsleeve',
    'outwear',
    'pants',
    'shirt',
    'shoes',
    'shorts',
    'skirt',
    't-shirt'
]

In [26]python · cell 73

python

dict(zip(classes, pred[0]))

Output

{'dress': -1.4282539,
 'hat': -5.522186,
 'longsleeve': -3.1655293,
 'outwear': -2.201648,
 'pants': 9.294684,
 'shirt': -3.4289198,
 'shoes': -4.2395606,
 'shorts': 3.4339347,
 'skirt': -4.194675,
 't-shirt': -2.9939806}

8.13 Summary

We can use pre-trained models for general image classification
Convolutional layers let us turn an image into a vector
Dense layers use the vector to make the predictions
Instead of training a model from scratch, we can use transfer learning and re-use already trained convolutional layers
First, train a small model (150x150) before training a big one (299x299)
Learning rate - how fast the model trians. Fast learners aren't always best ones
We can save the best model using callbacks and checkpointing
To avoid overfitting, use dropout and augmentation

8.14 Explore more

Add more data, e.g. Zalando, etc (ADD LINKS)
Albumentations - another way of generating augmentations
Use PyTorch or MXNet instead of TensorFlow/Keras
In addition to Xception, there are others architectures - try them

Other projects:

cats vs dogs
Hotdog vs not hotdog
Category of images