Copyright 2019 The TensorFlow Authors.¶

In [ ]:

#@title Licensed under the Apache License, Version 2.0
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

TensorFlow Addons Optimizers: LazyAdam¶

View on TensorFlow.org

Run in Google Colab

View source on GitHub

Download notebook

Overview¶

This notebook will demonstrate how to use the lazy adam optimizer from the Addons package.

LazyAdam¶

LazyAdam is a variant of the Adam optimizer that handles sparse updates more efficiently.

The original Adam algorithm maintains two moving-average accumulators for
each trainable variable; the accumulators are updated at every step.
This class provides lazier handling of gradient updates for sparse
variables.  It only updates moving-average accumulators for sparse variable
indices that appear in the current batch, rather than updating the
accumulators for all indices. Compared with the original Adam optimizer,
it can provide large improvements in model training throughput for some
applications. However, it provides slightly different semantics than the
original Adam algorithm, and may lead to different empirical results.

Setup¶

In [ ]:

try:
  %tensorflow_version 2.x
except:
  pass

import tensorflow as tf

In [ ]:

!pip install --no-deps tensorflow-addons~=0.7

In [ ]:

import tensorflow_addons as tfa

In [ ]:

# Hyperparameters
batch_size=64
epochs=10

Build the Model¶

In [ ]:

model = tf.keras.Sequential([
    tf.keras.layers.Dense(64, input_shape=(784,), activation='relu', name='dense_1'),
    tf.keras.layers.Dense(64, activation='relu', name='dense_2'),
    tf.keras.layers.Dense(10, activation='softmax', name='predictions'),
])

Prepare the Data¶

In [ ]:

# Load MNIST dataset as NumPy arrays
dataset = {}
num_validation = 10000
(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()

# Preprocess the data
x_train = x_train.reshape(-1, 784).astype('float32') / 255
x_test = x_test.reshape(-1, 784).astype('float32') / 255

Train and Evaluate¶

Simply replace typical keras optimizers with the new tfa optimizer

In [ ]:

# Compile the model
model.compile(
    optimizer=tfa.optimizers.LazyAdam(0.001),  # Utilize TFA optimizer
    loss=tf.keras.losses.SparseCategoricalCrossentropy(),
    metrics=['accuracy'])

# Train the network
history = model.fit(
    x_train,
    y_train,
    batch_size=batch_size,
    epochs=epochs)

In [ ]:

# Evaluate the network
print('Evaluate on test data:')
results = model.evaluate(x_test, y_test, batch_size=128, verbose = 2)
print('Test loss = {0}, Test acc: {1}'.format(results[0], results[1]))