Practical Deep Learning for Coders, v3¶
Lesson7_resnet_mnist¶
MNIST CNN¶
In [ ]:
%reload_ext autoreload
%autoreload 2
%matplotlib inline
In [ ]:
from fastai.vision import *
Data 数据¶
In [ ]:
path = untar_data(URLs.MNIST)
In [ ]:
path.ls()
Out[ ]:
[PosixPath('/home/ubuntu/.fastai/data/mnist_png/training'),
PosixPath('/home/ubuntu/.fastai/data/mnist_png/testing')]
In [ ]:
il = ImageList.from_folder(path, convert_mode='L')
In [ ]:
il.items[0]
Out[ ]:
PosixPath('/home/ubuntu/.fastai/data/mnist_png/training/4/44688.png')
In [ ]:
defaults.cmap='binary'
In [ ]:
il
Out[ ]:
ImageList (70000 items) [Image (1, 28, 28), Image (1, 28, 28), Image (1, 28, 28), Image (1, 28, 28), Image (1, 28, 28)]... Path: /home/ubuntu/.fastai/data/mnist_png
In [ ]:
il[0].show()
In [ ]:
sd = il.split_by_folder(train='training', valid='testing')
In [ ]:
sd
Out[ ]:
ItemLists; Train: ImageList (60000 items) [Image (1, 28, 28), Image (1, 28, 28), Image (1, 28, 28), Image (1, 28, 28), Image (1, 28, 28)]... Path: /home/ubuntu/.fastai/data/mnist_png; Valid: ImageList (10000 items) [Image (1, 28, 28), Image (1, 28, 28), Image (1, 28, 28), Image (1, 28, 28), Image (1, 28, 28)]... Path: /home/ubuntu/.fastai/data/mnist_png; Test: None
In [ ]:
(path/'training').ls()
Out[ ]:
[PosixPath('/home/ubuntu/.fastai/data/mnist_png/training/4'),
PosixPath('/home/ubuntu/.fastai/data/mnist_png/training/6'),
PosixPath('/home/ubuntu/.fastai/data/mnist_png/training/8'),
PosixPath('/home/ubuntu/.fastai/data/mnist_png/training/0'),
PosixPath('/home/ubuntu/.fastai/data/mnist_png/training/9'),
PosixPath('/home/ubuntu/.fastai/data/mnist_png/training/1'),
PosixPath('/home/ubuntu/.fastai/data/mnist_png/training/3'),
PosixPath('/home/ubuntu/.fastai/data/mnist_png/training/2'),
PosixPath('/home/ubuntu/.fastai/data/mnist_png/training/5'),
PosixPath('/home/ubuntu/.fastai/data/mnist_png/training/7')]
In [ ]:
ll = sd.label_from_folder()
In [ ]:
ll
Out[ ]:
LabelLists; Train: LabelList y: CategoryList (60000 items) [Category 4, Category 4, Category 4, Category 4, Category 4]... Path: /home/ubuntu/.fastai/data/mnist_png x: ImageList (60000 items) [Image (1, 28, 28), Image (1, 28, 28), Image (1, 28, 28), Image (1, 28, 28), Image (1, 28, 28)]... Path: /home/ubuntu/.fastai/data/mnist_png; Valid: LabelList y: CategoryList (10000 items) [Category 4, Category 4, Category 4, Category 4, Category 4]... Path: /home/ubuntu/.fastai/data/mnist_png x: ImageList (10000 items) [Image (1, 28, 28), Image (1, 28, 28), Image (1, 28, 28), Image (1, 28, 28), Image (1, 28, 28)]... Path: /home/ubuntu/.fastai/data/mnist_png; Test: None
In [ ]:
x,y = ll.train[0]
In [ ]:
x.show()
print(y,x.shape)
4 torch.Size([1, 28, 28])
In [ ]:
tfms = ([*rand_pad(padding=3, size=28, mode='zeros')], [])
In [ ]:
ll = ll.transform(tfms)
In [ ]:
bs = 128
In [ ]:
# not using imagenet_stats because not using pretrained model
data = ll.databunch(bs=bs).normalize()
In [ ]:
x,y = data.train_ds[0]
In [ ]:
x.show()
print(y)
4
In [ ]:
def _plot(i,j,ax): data.train_ds[0][0].show(ax, cmap='gray')
plot_multi(_plot, 3, 3, figsize=(8,8))
In [ ]:
xb,yb = data.one_batch()
xb.shape,yb.shape
Out[ ]:
(torch.Size([128, 1, 28, 28]), torch.Size([128]))
In [ ]:
data.show_batch(rows=3, figsize=(5,5))
Basic CNN with batchnorm 批量正规化（归一化）的基础版CNN模型¶
In [ ]:
def conv(ni,nf): return nn.Conv2d(ni, nf, kernel_size=3, stride=2, padding=1)
In [ ]:
model = nn.Sequential(
conv(1, 8), # 14
nn.BatchNorm2d(8),
nn.ReLU(),
conv(8, 16), # 7
nn.BatchNorm2d(16),
nn.ReLU(),
conv(16, 32), # 4
nn.BatchNorm2d(32),
nn.ReLU(),
conv(32, 16), # 2
nn.BatchNorm2d(16),
nn.ReLU(),
conv(16, 10), # 1
nn.BatchNorm2d(10),
Flatten() # remove (1,1) grid
)
In [ ]:
learn = Learner(data, model, loss_func = nn.CrossEntropyLoss(), metrics=accuracy)
In [ ]:
print(learn.summary())
====================================================================== Layer (type) Output Shape Param # Trainable ====================================================================== Conv2d [128, 8, 14, 14] 80 True ______________________________________________________________________ BatchNorm2d [128, 8, 14, 14] 16 True ______________________________________________________________________ ReLU [128, 8, 14, 14] 0 False ______________________________________________________________________ Conv2d [128, 16, 7, 7] 1168 True ______________________________________________________________________ BatchNorm2d [128, 16, 7, 7] 32 True ______________________________________________________________________ ReLU [128, 16, 7, 7] 0 False ______________________________________________________________________ Conv2d [128, 32, 4, 4] 4640 True ______________________________________________________________________ BatchNorm2d [128, 32, 4, 4] 64 True ______________________________________________________________________ ReLU [128, 32, 4, 4] 0 False ______________________________________________________________________ Conv2d [128, 16, 2, 2] 4624 True ______________________________________________________________________ BatchNorm2d [128, 16, 2, 2] 32 True ______________________________________________________________________ ReLU [128, 16, 2, 2] 0 False ______________________________________________________________________ Conv2d [128, 10, 1, 1] 1450 True ______________________________________________________________________ BatchNorm2d [128, 10, 1, 1] 20 True ______________________________________________________________________ Flatten [128, 10] 0 False ______________________________________________________________________ Total params: 12126 Total trainable params: 12126 Total non-trainable params: 0
In [ ]:
xb = xb.cuda()
In [ ]:
model(xb).shape
Out[ ]:
torch.Size([128, 10])
In [ ]:
learn.lr_find(end_lr=100)
LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.
In [ ]:
learn.recorder.plot()
In [ ]:
learn.fit_one_cycle(3, max_lr=0.1)
Total time: 00:30
| epoch | train_loss | valid_loss | accuracy |
|---|---|---|---|
| 1 | 0.223167 | 0.217859 | 0.930500 |
| 2 | 0.136179 | 0.078651 | 0.976400 |
| 3 | 0.072080 | 0.038664 | 0.988600 |
Refactor 重构¶
In [ ]:
def conv2(ni,nf): return conv_layer(ni,nf,stride=2)
In [ ]:
model = nn.Sequential(
conv2(1, 8), # 14
conv2(8, 16), # 7
conv2(16, 32), # 4
conv2(32, 16), # 2
conv2(16, 10), # 1
Flatten() # remove (1,1) grid
)
In [ ]:
learn = Learner(data, model, loss_func = nn.CrossEntropyLoss(), metrics=accuracy)
In [ ]:
learn.fit_one_cycle(10, max_lr=0.1)
Total time: 01:12
| epoch | train_loss | valid_loss | accuracy |
|---|---|---|---|
| 1 | 0.228332 | 0.206325 | 0.937500 |
| 2 | 0.189966 | 0.192558 | 0.940800 |
| 3 | 0.156765 | 0.092810 | 0.969100 |
| 4 | 0.135871 | 0.083914 | 0.973300 |
| 5 | 0.108844 | 0.071582 | 0.978000 |
| 6 | 0.105887 | 0.128586 | 0.960200 |
| 7 | 0.080699 | 0.052754 | 0.983200 |
| 8 | 0.066007 | 0.037588 | 0.988600 |
| 9 | 0.047513 | 0.030255 | 0.990200 |
| 10 | 0.044705 | 0.028373 | 0.991600 |
Resnet-ish¶
In [ ]:
class ResBlock(nn.Module):
def __init__(self, nf):
super().__init__()
self.conv1 = conv_layer(nf,nf)
self.conv2 = conv_layer(nf,nf)
def forward(self, x): return x + self.conv2(self.conv1(x))
In [ ]:
help(res_block)
Help on function res_block in module fastai.layers:
res_block(nf, dense:bool=False, norm_type:Union[fastai.layers.NormType, NoneType]=<NormType.Batch: 1>, bottle:bool=False, **kwargs)
Resnet block of `nf` features.
In [ ]:
model = nn.Sequential(
conv2(1, 8),
res_block(8),
conv2(8, 16),
res_block(16),
conv2(16, 32),
res_block(32),
conv2(32, 16),
res_block(16),
conv2(16, 10),
Flatten()
)
In [ ]:
def conv_and_res(ni,nf): return nn.Sequential(conv2(ni, nf), res_block(nf))
In [ ]:
model = nn.Sequential(
conv_and_res(1, 8),
conv_and_res(8, 16),
conv_and_res(16, 32),
conv_and_res(32, 16),
conv2(16, 10),
Flatten()
)
In [ ]:
learn = Learner(data, model, loss_func = nn.CrossEntropyLoss(), metrics=accuracy)
In [ ]:
learn.lr_find(end_lr=100)
learn.recorder.plot()
LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.
In [ ]:
learn.fit_one_cycle(12, max_lr=0.05)
Total time: 02:00
| epoch | train_loss | valid_loss | accuracy |
|---|---|---|---|
| 1 | 0.248807 | 0.121582 | 0.972800 |
| 2 | 0.120927 | 0.360583 | 0.890300 |
| 3 | 0.104013 | 0.074916 | 0.977800 |
| 4 | 0.081181 | 0.065717 | 0.980000 |
| 5 | 0.068514 | 0.096448 | 0.967200 |
| 6 | 0.061274 | 0.087955 | 0.971800 |
| 7 | 0.051673 | 0.033911 | 0.989400 |
| 8 | 0.048090 | 0.033234 | 0.988800 |
| 9 | 0.039095 | 0.024638 | 0.992400 |
| 10 | 0.023670 | 0.021215 | 0.993400 |
| 11 | 0.019128 | 0.016159 | 0.994500 |
| 12 | 0.021365 | 0.016120 | 0.995200 |
In [ ]:
print(learn.summary())
====================================================================== Layer (type) Output Shape Param # Trainable ====================================================================== Conv2d [128, 8, 14, 14] 72 True ______________________________________________________________________ ReLU [128, 8, 14, 14] 0 False ______________________________________________________________________ BatchNorm2d [128, 8, 14, 14] 16 True ______________________________________________________________________ Conv2d [128, 8, 14, 14] 576 True ______________________________________________________________________ ReLU [128, 8, 14, 14] 0 False ______________________________________________________________________ BatchNorm2d [128, 8, 14, 14] 16 True ______________________________________________________________________ Conv2d [128, 8, 14, 14] 576 True ______________________________________________________________________ ReLU [128, 8, 14, 14] 0 False ______________________________________________________________________ BatchNorm2d [128, 8, 14, 14] 16 True ______________________________________________________________________ MergeLayer [128, 8, 14, 14] 0 False ______________________________________________________________________ Conv2d [128, 16, 7, 7] 1152 True ______________________________________________________________________ ReLU [128, 16, 7, 7] 0 False ______________________________________________________________________ BatchNorm2d [128, 16, 7, 7] 32 True ______________________________________________________________________ Conv2d [128, 16, 7, 7] 2304 True ______________________________________________________________________ ReLU [128, 16, 7, 7] 0 False ______________________________________________________________________ BatchNorm2d [128, 16, 7, 7] 32 True ______________________________________________________________________ Conv2d [128, 16, 7, 7] 2304 True ______________________________________________________________________ ReLU [128, 16, 7, 7] 0 False ______________________________________________________________________ BatchNorm2d [128, 16, 7, 7] 32 True ______________________________________________________________________ MergeLayer [128, 16, 7, 7] 0 False ______________________________________________________________________ Conv2d [128, 32, 4, 4] 4608 True ______________________________________________________________________ ReLU [128, 32, 4, 4] 0 False ______________________________________________________________________ BatchNorm2d [128, 32, 4, 4] 64 True ______________________________________________________________________ Conv2d [128, 32, 4, 4] 9216 True ______________________________________________________________________ ReLU [128, 32, 4, 4] 0 False ______________________________________________________________________ BatchNorm2d [128, 32, 4, 4] 64 True ______________________________________________________________________ Conv2d [128, 32, 4, 4] 9216 True ______________________________________________________________________ ReLU [128, 32, 4, 4] 0 False ______________________________________________________________________ BatchNorm2d [128, 32, 4, 4] 64 True ______________________________________________________________________ MergeLayer [128, 32, 4, 4] 0 False ______________________________________________________________________ Conv2d [128, 16, 2, 2] 4608 True ______________________________________________________________________ ReLU [128, 16, 2, 2] 0 False ______________________________________________________________________ BatchNorm2d [128, 16, 2, 2] 32 True ______________________________________________________________________ Conv2d [128, 16, 2, 2] 2304 True ______________________________________________________________________ ReLU [128, 16, 2, 2] 0 False ______________________________________________________________________ BatchNorm2d [128, 16, 2, 2] 32 True ______________________________________________________________________ Conv2d [128, 16, 2, 2] 2304 True ______________________________________________________________________ ReLU [128, 16, 2, 2] 0 False ______________________________________________________________________ BatchNorm2d [128, 16, 2, 2] 32 True ______________________________________________________________________ MergeLayer [128, 16, 2, 2] 0 False ______________________________________________________________________ Conv2d [128, 10, 1, 1] 1440 True ______________________________________________________________________ ReLU [128, 10, 1, 1] 0 False ______________________________________________________________________ BatchNorm2d [128, 10, 1, 1] 20 True ______________________________________________________________________ Flatten [128, 10] 0 False ______________________________________________________________________ Total params: 41132 Total trainable params: 41132 Total non-trainable params: 0