This notebook demonstrates how you can find adversarial examples for a pre-trained example network on the MNIST dataset.
We suggest having the Gurobi
solver installed, since its performance is significantly faster. If this is not possible, the Cbc
solver is another option.
The Images
package is only necessary for visualizing the MNIST images.
using MIPVerify
using Gurobi
using Images
WARNING: Method definition midpoints(Base.Range{T} where T) in module Base at deprecated.jl:56 overwritten in module StatsBase at /home/vtjeng/.julia/v0.6/StatsBase/src/hist.jl:535. WARNING: Method definition midpoints(AbstractArray{T, 1} where T) in module Base at deprecated.jl:56 overwritten in module StatsBase at /home/vtjeng/.julia/v0.6/StatsBase/src/hist.jl:533.
mnist = MIPVerify.read_datasets("MNIST")
MNIST: `train`: {ImageDataset} `images`: 55000 images of size (28, 28, 1), with pixels in [0.0, 1.0]. `labels`: 55000 corresponding labels, with 10 unique labels in [0, 9]. `test`: {ImageDataset} `images`: 10000 images of size (28, 28, 1), with pixels in [0.0, 1.0]. `labels`: 10000 corresponding labels, with 10 unique labels in [0, 9].
mnist.train
{ImageDataset} `images`: 55000 images of size (28, 28, 1), with pixels in [0.0, 1.0]. `labels`: 55000 corresponding labels, with 10 unique labels in [0, 9].
size(mnist.train.images)
(55000, 28, 28, 1)
We can use Images.colorview
to preview these images.
colorview(Gray, mnist.train.images[1, :, :, 1])
mnist.train.labels
55000-element Array{Int64,1}: 7 3 4 6 1 8 1 0 9 8 0 3 1 ⋮ 7 8 9 2 9 5 1 8 3 5 6 8
We import a sample pre-trained neural network.
n1params = MIPVerify.get_example_network_params("MNIST.n1")
convolutional neural net MNIST.n1 `convlayer_params` [0]: (none) `fclayer_params` [2]: fully connected layer with 784 inputs and 40 output units, and a ReLU activation function. fully connected layer with 40 inputs and 20 output units, and a ReLU activation function. `softmax_params`: softmax layer with 20 inputs and 10 output units.
MIPVerify.frac_correct
allows us to verify that the network has a reasonable accuracy on the test set of 96.95%. (This step is crucial when working with your own neural net parameters; since the training is done outside of Julia, a common mistake is to transfer the parameters incorrectly.)
MIPVerify.frac_correct(n1params, mnist.test, 10000)
0.9695
We feed the first image into the neural net, obtaining the activations of the final softmax layer.
Note that the image must be specified as a 4-dimensional array with size (1, height, width, num_channels)
. We provide a helper function MIPVerify.get_image
that extracts the image from the dataset while preserving all four dimensions.
sample_image = MIPVerify.get_image(mnist.train.images, 1)
1×28×28×1 Array{Float32,4}: [:, :, 1, 1] = 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 … 0.0 0.0 0.0 0.0 0.0 0.0 0.0 [:, :, 2, 1] = 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 … 0.0 0.0 0.0 0.0 0.0 0.0 0.0 [:, :, 3, 1] = 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 … 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... [:, :, 26, 1] = 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 … 0.0 0.0 0.0 0.0 0.0 0.0 0.0 [:, :, 27, 1] = 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 … 0.0 0.0 0.0 0.0 0.0 0.0 0.0 [:, :, 28, 1] = 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 … 0.0 0.0 0.0 0.0 0.0 0.0 0.0
output_activations = sample_image |> n1params
10-element Array{Float32,1}: 0.15597 0.270125 1.49147 0.145112 0.281066 0.385918 0.228231 4.10202 0.905381 1.15467
The category that has the largest activation is category 8, corresponding to a label of 7.
(output_activations |> MIPVerify.get_max_index) - 1
7
This matches the true label.
MIPVerify.get_label(mnist.train.labels, 1)
7
We now try to find an adversarial example for the first image on n1params
, setting the target category as index 9
(corresponding to a true label of 8).
target_label_index = 9
d = MIPVerify.find_adversarial_example(n1params, sample_image, target_label_index, GurobiSolver())
[notice | MIPVerify]: Loading model from cache. [notice | MIPVerify]: Attempting to find adversarial example. Neural net predicted label is 8, target labels are [9] Optimize a model with 3385 rows, 3256 columns and 71132 nonzeros Variable types: 3196 continuous, 60 integer (60 binary) Coefficient statistics: Matrix range [2e-05, 7e+02] Objective range [1e+00, 1e+00] Bounds range [1e+00, 1e+02] RHS range [1e-02, 7e+02] Presolve removed 2860 rows and 2184 columns Presolve time: 0.21s Presolved: 525 rows, 1072 columns, 65472 nonzeros MIP start did not produce a new incumbent solution MIP start violates constraint R1024 by 1.000000000 Variable types: 1012 continuous, 60 integer (60 binary) Root relaxation: objective 0.000000e+00, 242 iterations, 0.01 seconds Nodes | Current Node | Objective Bounds | Work Expl Unexpl | Obj Depth IntInf | Incumbent BestBd Gap | It/Node Time 0 0 0.00000 0 9 - 0.00000 - - 0s Another try with MIP start H 0 0 55.3332559 0.00000 100% - 0s 0 0 0.00000 0 10 55.33326 0.00000 100% - 0s H 0 0 27.1696380 0.00000 100% - 0s 0 0 0.00000 0 10 27.16964 0.00000 100% - 0s 0 0 0.00000 0 11 27.16964 0.00000 100% - 0s H 0 0 9.8928387 0.00000 100% - 0s H 0 0 9.8622769 0.00000 100% - 0s 0 0 0.00000 0 1 9.86228 0.00000 100% - 0s 0 0 0.00000 0 1 9.86228 0.00000 100% - 1s 0 0 0.00000 0 1 9.86228 0.00000 100% - 1s 0 2 0.00000 0 1 9.86228 0.00000 100% - 1s H 1023 347 9.7630683 0.00000 100% 74.6 4s * 1027 347 27 9.7569972 0.00000 100% 74.3 4s 1063 364 1.42716 20 10 9.75700 0.00000 100% 73.5 5s 2669 651 cutoff 24 9.75700 0.59460 93.9% 75.5 10s 4772 477 cutoff 16 9.75700 4.59017 53.0% 68.7 15s 6523 117 cutoff 16 9.75700 6.92691 29.0% 66.1 20s Cutting planes: Gomory: 3 Cover: 2 MIR: 7 Flow cover: 11 Explored 7072 nodes (468031 simplex iterations) in 21.77 seconds Thread count was 8 (of 8 available processors) Solution count 6: 9.757 9.76307 9.86228 ... 55.3333 Pool objective bound 9.757 Optimal solution found (tolerance 1.00e-04) Best objective 9.756997241475e+00, best bound 9.756997241475e+00, gap 0.0000%
Dict{Symbol,Any} with 7 entries: :PerturbationParameters => additive :TargetIndexes => [9] :SolveStatus => :Optimal :Output => JuMP.GenericAffExpr{Float64,JuMP.Variable}[-0.0120… :Model => Minimization problem with:… :Perturbation => JuMP.Variable[__anon__ __anon__ __anon__ __anon__ … :PerturbedInput => JuMP.Variable[__anon__ __anon__ __anon__ __anon__ …
using JuMP
perturbed_sample_image = getvalue(d[:PerturbedInput])
1×28×28×1 Array{Float64,4}: [:, :, 1, 1] = 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 … 0.0 0.0 0.0 0.0 0.0 0.0 0.0 [:, :, 2, 1] = 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 … 0.0 0.0 0.0 0.0 0.0 0.0 0.0 [:, :, 3, 1] = 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 … 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... [:, :, 26, 1] = 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 … 0.0 0.0 0.0 0.0 0.0 0.0 0.0 [:, :, 27, 1] = 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 … 0.0 0.0 0.0 0.0 0.0 0.0 0.0 [:, :, 28, 1] = 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 … 0.0 0.0 0.0 0.0 0.0 0.0 0.0
As a sanity check, we feed the perturbed image into the neural net and inspect the activation in the final layer. We verify that the perturbed image does maximize the activation of the target label index, which is 9.
perturbed_sample_image |> n1params
10-element Array{Float64,1}: 0.257004 0.416757 0.692725 0.38037 0.295063 0.204749 0.488696 3.30817 3.30817 0.551406
We visualize the perturbed image and compare it to the original image. Since we are minimizing the L1-norm, changes are made to only a few pixels, but the magnitude of these changes are large (and noticeable).
colorview(Gray, perturbed_sample_image[1, :, :, 1])
colorview(Gray, sample_image[1, :, :, 1])
That concludes this quickstart! The next tutorial will introduce you to each of the layers, and show how you can import your own neural network parameters.