MNIST Example in fastai¶

This Jupyter notebook demonstrates using the fastai (version 1.0.38) deep learning framework with comet.ml.

In this example, we load a fastai model, called WideResNet, and train it on a small part of the MNIST_TINY dataset.

fastai is a framework built on top of the torch Python library.

To find out more, you might find these links helpful:

Let's get started!

0. Installation¶

This example uses fastai version 1.0.38. You can install a specific version of fastai (which should also install the correct version of torch) with this command at the terminal:

python -m pip install fastai==1.0.38

Once you have fastai and torch installed, we are ready to import them.

1. Imports¶

First, we import the comet_ml library, followed by the fastai library, and others if needed. The only requirement here is that comet_ml be imported first. If you forget, just restart the kernel, and import them in the proper order.

In [1]:

## Import this first:
from comet_ml import Experiment

## Import the deep learning framework:
import fastai
import fastai.vision

## Additional libraries for this example:
import glob
import os

2. Dataset¶

As a simple demo, we'll start with the the MNIST_TINY dataset, In fastai, we use the datasets.untar_data function to download and uncompress the data:

In [2]:

path = fastai.datasets.untar_data(fastai.datasets.URLs.MNIST_TINY)
print("data path:", path)

data path: /home/dblank/.fastai/data/mnist_tiny

The path returned by the untar_data function shows where the data was saved. Using the shell ! magic, we can explore the dataset in more detail:

In [3]:

! ls ~/.fastai/data/mnist_tiny/train/3/

715.png   7463.png  8360.png  9141.png	9303.png
7288.png  7626.png  8957.png  9192.png	9637.png

By poking around here, we can see that there are two categories 3 and 7 and each category has about 2,000 examples.

That is still too many for a CPU, so we trim it down to 10 in each category:

In [4]:

dirname = os.path.dirname(path)
for group in ["mnist_tiny/train/3/*.png",
              "mnist_tiny/train/7/*.png",
              "mnist_tiny/valid/3/*.png",
              "mnist_tiny/valid/7/*.png"]:
    for filename in glob.glob(os.path.join(dirname, group))[10:]:
        os.remove(filename)

Now, we check again to see if we have just 10 images:

In [5]:

! ls ~/.fastai/data/mnist_tiny/train/3/

715.png   7463.png  8360.png  9141.png	9303.png
7288.png  7626.png  8957.png  9192.png	9637.png

To see these images, we can use some tools from the IPython library:

In [6]:

from IPython.display import Image, display

Let's see what the training set looks like for the 3 and 7 categories:

In [7]:

dirname = os.path.dirname(path)
for group in ["mnist_tiny/train/3/*.png", 
              "mnist_tiny/train/7/*.png"]:
    for filename in glob.glob(os.path.join(dirname, group)):
        display(Image(filename))

So, all of the files under mnist_tiny/train/3/ are pictures of 3's, and mnist_tiny/train/7/ are pictures of 7's.

Now we get the image data from the folder. To train a dataset in fastai, we must create a DataBunch. In this case, we can use ImageDataBunch in the fastai.vision library. We pick 10 as the batch size (bs) because there are only 10 images in each category.

In [8]:

data = fastai.vision.ImageDataBunch.from_folder(path, bs=10) # bs: batch size

That's better for this simple test. Now we can create a model, and train the network:

3. Model¶

In this example, we will use the pre-designed WideresNet from fastai. The model is also known as wrn_22.

In [9]:

model = fastai.vision.models.WideResNet(num_groups=3,
                                        N=3,
                                        num_classes=10,
                                        k=6,
                                        drop_p=0.)

That's it! Often, you would probably build your own model, or adjust a default model. To see more on model building in fastai, see:

4. Experiment¶

In order for comet.ml to log your experiment and results, you need to create an Experiment instance. To do this, you'll need two items:

a Comet api_key
a project_name

You can find your Comet api_key when you log in to https://www.comet.ml and click on your project. You should see a screen that looks similar to:

comet login screen

Click on the API key to copy the key to your clipboard.

It is recommended that you put your COMET_API_KEY in a .env key in the current directory. You can do that using the following code. Put it in a cell, replace the ... with your key, and then delete the cell. That way your key stays private.

%%writefile .env

COMET_API_KEY=...

It is also recommended that you use your project_name in the cell, so you can match the results with this code. You can make up a new name, or add this experiment to a project that already exists.

In [10]:

experiment = Experiment(project_name="comet_notebooks")

COMET INFO: Experiment is live on comet.ml https://www.comet.ml/cometpublic/comet-notebooks/d21f94a1c71841d2961da1e6ddb5ab20

If you get the error that ends with:

ValueError: Comet.ml requires an API key. Please provide as the first argument to Experiment(api_key) or as an environment variable named COMET_API_KEY

then that means that either you don't have an .env file in this directory, or the key is invalid.

Otherwise, you should see the message:

COMET INFO: Experiment is live on comet.ml https://www.comet.ml/...

If you click the URL, then a new page will open up. But, even better, you can execute the following line to see the experiment in the current notebook:

In [11]:

experiment.display()

By the way, the line experiment.display() works when you are at the console too. It will open up a window in your browser.

Now, we are ready for training!

5. Training¶

In fastai, we can train differently depending on if we are running CPU or a GPU. To test, we can use the data.device.type property. This will create a fastai Learner:

In [12]:

if data.device.type == 'cpu':
    learn = fastai.basic_train.Learner(data, model, metrics=fastai.metrics.accuracy)
else: # GPU:
    learn = fastai.basic_train.Learner(data, model, metrics=fastai.metrics.accuracy).to_fp16()

Now we are ready to train the model. To tell Comet about the details, we put the call to fit or fit_one_cylce inside an indented block under experiment.train():

In [13]:

with experiment.train():
    learn.fit_one_cycle(2, 3e-3, wd=0.4, div_factor=10, pct_start=0.5)

Total time: 00:10
epoch  train_loss  valid_loss  accuracy
1      2.331475    2.306675    0.000000  (00:05)
2      1.933559    2.272467    0.000000  (00:05)

6. Logging¶

In fastai, Comet will automatically log:

the model description
the training loss
the training validation
the source code

To log other items manually, you can use any of the following:

experiment.log_html(HTML_STRING)
experiment.html_log_url(URL_STRING)
experiment.image(FILENAME)
experiment.log_dataset_hash(DATASET)
experiment.log_other(KEY, VALUE)
experiment.log_metric(NAME, VALUE)
experiment.log_parameter(PARAMETER, VALUE)
experiment.log_figure(NAME, FIGURE)

For complete details, please see:

https://www.comet.ml/docs/python-sdk/Experiment/#experiment

7. Finish¶

Finall, we are ready to tell Comet that our experiment is complete. You don't need to do this is a script that ends. But in Jupyter, we need to indicate that the experiment is finished. We do that with the experiment.end() method:

In [14]:

experiment.end()

COMET INFO: Uploading stats to Comet before program termination (may take several seconds)
COMET INFO: Experiment is live on comet.ml https://www.comet.ml/cometpublic/comet-notebooks/d21f94a1c71841d2961da1e6ddb5ab20

That's it! If you have any questions, please visit us on https://cometml.slack.com