We use the same example as in the quickstart.
using MIPVerify
using Gurobi
mnist = MIPVerify.read_datasets("MNIST")
n1 = MIPVerify.get_example_network_params("MNIST.n1")
sample_image = MIPVerify.get_image(mnist.test.images, 1)
target_label_index = 10
10
Our package logs to STDOUT at the debug
, info
, notice
and warn
levels.
The default output level is to show logs only at the notice
level and above, which is also the recommended level.
d = MIPVerify.find_adversarial_example(n1, sample_image, target_label_index, GurobiSolver(), rebuild=true);
[notice | MIPVerify]: Attempting to find adversarial example. Neural net predicted label is 8, target labels are [10] [notice | MIPVerify]: Rebuilding model from scratch. This may take some time as we determine upper and lower bounds for the input to each non-linear unit.
Calculating upper bounds: 2%|█ | ETA: 0:00:06
Academic license - for non-commercial use only
Calculating upper bounds: 100%|███████████████████████| Time: 0:00:02 Calculating lower bounds: 100%|███████████████████████| Time: 0:00:00 Imposing relu constraint: 100%|███████████████████████| Time: 0:00:00 Calculating upper bounds: 100%|███████████████████████| Time: 0:01:48 Calculating lower bounds: 100%|███████████████████████| Time: 0:02:20
[notice | MIPVerify]: The model built will be cached and re-used for future solves, unless you explicitly set rebuild=true.
More information is available if we set the log level to debug
.
MIPVerify.setloglevel!("debug")
d = MIPVerify.find_adversarial_example(n1, sample_image, target_label_index, GurobiSolver(), rebuild=true)
Academic license - for non-commercial use only Optimize a model with 3385 rows, 3256 columns and 71132 nonzeros Variable types: 3196 continuous, 60 integer (60 binary) Coefficient statistics: Matrix range [2e-05, 7e+02] Objective range [1e+00, 1e+00] Bounds range [1e+00, 1e+02] RHS range [4e-03, 7e+02] MIP start did not produce a new incumbent solution MIP start violates constraint R1024 by 1.000000000 Presolve removed 2956 rows and 2227 columns Presolve time: 0.16s Presolved: 429 rows, 1029 columns, 61365 nonzeros Variable types: 969 continuous, 60 integer (60 binary) Root relaxation: objective 0.000000e+00, 223 iterations, 0.00 seconds Nodes | Current Node | Objective Bounds | Work Expl Unexpl | Obj Depth IntInf | Incumbent BestBd Gap | It/Node Time 0 0 0.00000 0 4 - 0.00000 - - 0s Another try with MIP start H 0 0 36.9304736 0.00000 100% - 0s 0 0 0.00000 0 6 36.93047 0.00000 100% - 0s H 0 0 32.4680084 0.00000 100% - 0s 0 0 0.00000 0 7 32.46801 0.00000 100% - 0s 0 0 0.00000 0 7 32.46801 0.00000 100% - 0s 0 0 0.00000 0 7 32.46801 0.00000 100% - 0s 0 2 0.00000 0 7 32.46801 0.00000 100% - 0s H 27 26 26.4992057 0.00000 100% 95.6 0s H 117 100 19.5583736 0.00000 100% 73.5 1s * 145 111 43 18.7741931 0.00000 100% 72.1 1s H 237 97 7.2802489 0.00000 100% 57.9 1s * 266 117 40 6.9633438 0.00000 100% 55.8 1s * 463 194 43 6.5385430 0.00000 100% 53.7 2s 901 439 4.46206 35 10 6.53854 0.00000 100% 58.2 5s H 1975 402 5.9198759 0.00000 100% 59.3 8s 2353 459 0.36395 32 9 5.91988 0.02050 100% 59.5 10s * 3375 661 49 5.7487698 0.42240 92.7% 56.1 12s * 3510 669 51 5.4974367 0.42240 92.3% 55.6 12s 4522 811 5.06223 33 10 5.49744 0.86529 84.3% 53.4 15s * 4731 846 46 5.2900951 1.22249 76.9% 52.3 15s * 5433 904 51 5.1129105 1.53829 69.9% 48.9 16s * 5719 936 45 4.9728818 1.65195 66.8% 47.7 17s * 5785 902 48 4.6601251 1.67856 64.0% 47.4 17s 7053 1025 3.63314 27 6 4.66013 2.04395 56.1% 44.8 20s * 8453 827 44 4.6418593 2.74366 40.9% 43.0 22s 9569 411 cutoff 25 4.64186 3.67506 20.8% 41.3 25s Cutting planes: Gomory: 3 Projected implied bound: 6 MIR: 2 Flow cover: 1 Explored 10523 nodes (420298 simplex iterations) in 26.66 seconds Thread count was 4 (of 4 available processors) Solution count 10: 4.64186 4.66013 4.97288 ... 6.96334 Optimal solution found (tolerance 1.00e-04) Best objective 4.641859316319e+00, best bound 4.641859316319e+00, gap 0.0000% [notice | MIPVerify]: Attempting to find adversarial example. Neural net predicted label is 8, target labels are [10] [notice | MIPVerify]: Rebuilding model from scratch. This may take some time as we determine upper and lower bounds for the input to each non-linear unit. [info | MIPVerify]: Applying Flatten() ... [info | MIPVerify]: Applying Linear(784 -> 40) ... [info | MIPVerify]: Applying ReLU() ... Academic license - for non-commercial use only [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δu = 0.0 [info | MIPVerify]: Applying Linear(40 -> 20) ... [info | MIPVerify]: Applying ReLU() ... [debug | MIPVerify]: Δu = 236.19499039114618 [debug | MIPVerify]: Δu = 347.3507384582369 [debug | MIPVerify]: Δu = 111.66031695101941 [debug | MIPVerify]: Δu = 110.09320929282381 [debug | MIPVerify]: Δu = 263.9686250131348 [debug | MIPVerify]: Δu = 170.58623081063135 [debug | MIPVerify]: Δu = 342.81111856390635 [debug | MIPVerify]: Δu = 322.1609214959067 [debug | MIPVerify]: Δu = 190.94181084213062 [debug | MIPVerify]: Δu = 120.56886946816377 [debug | MIPVerify]: Δu = 333.3978216299831 [debug | MIPVerify]: Δu = 156.25265560651098 [debug | MIPVerify]: Δu = 306.0828261286914 [debug | MIPVerify]: Δu = 176.38206258926618 [debug | MIPVerify]: Δu = 301.9881744933099 [debug | MIPVerify]: Δu = 325.9390651275078 [debug | MIPVerify]: Δu = 146.18208942669833 [debug | MIPVerify]: Δu = 292.4320734889739 [debug | MIPVerify]: Δu = 184.93657306973148 [debug | MIPVerify]: Δu = 256.17035614083665 [debug | MIPVerify]: Δu = 329.4706219653086 [debug | MIPVerify]: Δu = 490.33543193659597 [debug | MIPVerify]: Δu = 230.8574409869865 [info | MIPVerify]: Hit user limit during solve to determine bounds. Multiplicative gap was 0.7451049966147174. [debug | MIPVerify]: Δu = 240.98736242290283 [debug | MIPVerify]: Δu = 443.7377089444059 [debug | MIPVerify]: Δu = 546.9967283877565 [debug | MIPVerify]: Δu = 561.8499789605797 [debug | MIPVerify]: Δu = 252.88861567468402 [info | MIPVerify]: Hit user limit during solve to determine bounds. Multiplicative gap was 1.258999125666667. [debug | MIPVerify]: Δu = 368.8231767436639 [info | MIPVerify]: Hit user limit during solve to determine bounds. Multiplicative gap was 0.5343960501301519. [debug | MIPVerify]: Δu = 306.69327050758636 [debug | MIPVerify]: Δu = 395.6692979013308 [debug | MIPVerify]: Δu = 263.6311034209684 [debug | MIPVerify]: Δu = 259.3597215144569 [info | MIPVerify]: Hit user limit during solve to determine bounds. Multiplicative gap was 0.1117666858312083. [debug | MIPVerify]: Δu = 272.3179195538885 [debug | MIPVerify]: Δu = 363.8405178338219 [debug | MIPVerify]: Δu = 223.20192786473305 [debug | MIPVerify]: Δu = 397.6660964044127 [debug | MIPVerify]: Δu = 354.40977058244283 [debug | MIPVerify]: Δu = 347.3454124031803 [debug | MIPVerify]: Δu = 540.5128944047548 [info | MIPVerify]: Applying Linear(20 -> 10) ... [notice | MIPVerify]: The model built will be cached and re-used for future solves, unless you explicitly set rebuild=true.
Dict{Any,Any} with 10 entries: :PerturbedInput => JuMP.Variable[__anon__ __anon__ __anon__ __anon__ __an… :TargetIndexes => [10] :SolveStatus => :Optimal :TotalTime => 342.126 :TighteningApproach => "mip" :Output => JuMP.GenericAffExpr{Float64,JuMP.Variable}[-0.01206386… :PredictedIndex => 8 :Model => Minimization problem with:… :Perturbation => JuMP.Variable[__anon__ __anon__ __anon__ __anon__ __an… :PerturbationFamily => unrestricted
Academic license - for non-commercial use only Optimize a model with 3385 rows, 3256 columns and 71132 nonzeros Variable types: 3196 continuous, 60 integer (60 binary) Coefficient statistics: Matrix range [2e-05, 7e+02] Objective range [1e+00, 1e+00] Bounds range [1e+00, 1e+02] RHS range [4e-03, 7e+02] MIP start did not produce a new incumbent solution MIP start violates constraint R1024 by 1.000000000 Presolve removed 2956 rows and 2227 columns Presolve time: 0.17s Presolved: 429 rows, 1029 columns, 61365 nonzeros Variable types: 969 continuous, 60 integer (60 binary) Root relaxation: objective 0.000000e+00, 212 iterations, 0.02 seconds Nodes | Current Node | Objective Bounds | Work Expl Unexpl | Obj Depth IntInf | Incumbent BestBd Gap | It/Node Time 0 0 0.00000 0 3 - 0.00000 - - 0s Another try with MIP start H 0 0 36.9304736 0.00000 100% - 0s H 0 0 32.4680084 0.00000 100% - 0s 0 0 0.00000 0 7 32.46801 0.00000 100% - 0s 0 0 0.00000 0 7 32.46801 0.00000 100% - 0s 0 2 0.00000 0 7 32.46801 0.00000 100% - 0s H 106 61 16.8778596 0.00000 100% 61.7 1s H 150 81 6.3285068 0.00000 100% 64.0 1s * 772 401 41 6.2748878 0.00000 100% 51.5 2s 875 468 0.00000 28 9 6.27489 0.00000 100% 56.5 5s H 888 444 6.1453585 0.00000 100% 56.7 5s 2318 623 0.02050 30 12 6.14536 0.02050 100% 56.2 10s * 2577 683 52 6.0729847 0.02050 100% 54.6 10s * 3003 803 49 5.7567275 0.02050 100% 53.3 11s 4503 1259 0.72742 27 12 5.75673 0.33754 94.1% 50.0 15s 6702 1834 cutoff 36 5.75673 0.50354 91.3% 46.6 20s 8849 2358 4.79543 37 11 5.75673 0.66488 88.5% 46.4 25s * 9568 2095 50 4.6839523 0.66565 85.8% 46.4 26s 10924 2206 4.42918 31 4 4.68395 0.88975 81.0% 45.9 30s 12870 2458 3.07862 32 10 4.68395 1.11657 76.2% 46.1 35s *13768 2488 53 4.6601251 1.27564 72.6% 45.6 36s 14627 2653 cutoff 31 4.66013 1.35669 70.9% 45.2 41s H14661 2643 4.6418593 1.35669 70.8% 45.2 41s 16232 2662 3.78233 36 17 4.64186 1.66017 64.2% 44.4 45s 18398 2845 1.98944 23 8 4.64186 1.98944 57.1% 43.1 50s 20003 2806 cutoff 44 4.64186 2.28758 50.7% 42.6 55s 21951 2650 3.90135 27 15 4.64186 2.55568 44.9% 41.9 60s 24164 2427 3.11569 39 10 4.64186 2.98694 35.7% 40.6 65s 26303 2060 cutoff 36 4.64186 3.37603 27.3% 39.6 70s 28161 1541 cutoff 30 4.64186 3.69591 20.4% 39.1 75s 30006 846 3.98690 31 12 4.64186 3.97502 14.4% 38.6 80s Cutting planes: Gomory: 1 Projected implied bound: 41 MIR: 21 Flow cover: 19 Explored 31449 nodes (1197329 simplex iterations) in 84.54 seconds Thread count was 4 (of 4 available processors) Solution count 10: 4.64186 4.66013 4.68395 ... 32.468 Optimal solution found (tolerance 1.00e-04) Best objective 4.641859316319e+00, best bound 4.641859316319e+00, gap 0.0000%
When we select rebuild=false
, we do not do any of the work in determining upper and lower bounds, and none of the info
and debug
logs are produced.
d = MIPVerify.find_adversarial_example(n1, sample_image, target_label_index, GurobiSolver(), rebuild=false);
[notice | MIPVerify]: Attempting to find adversarial example. Neural net predicted label is 8, target labels are [10] [notice | MIPVerify]: Loading model from cache. Academic license - for non-commercial use only Optimize a model with 3385 rows, 3256 columns and 71132 nonzeros Variable types: 3196 continuous, 60 integer (60 binary) Coefficient statistics: Matrix range [2e-05, 7e+02] Objective range [1e+00, 1e+00] Bounds range [1e+00, 1e+02] RHS range [4e-03, 7e+02] MIP start did not produce a new incumbent solution MIP start violates constraint R1024 by 1.000000000 Presolve removed 2956 rows and 2227 columns Presolve time: 0.16s Presolved: 429 rows, 1029 columns, 61365 nonzeros Variable types: 969 continuous, 60 integer (60 binary) Root relaxation: objective 0.000000e+00, 212 iterations, 0.02 seconds Nodes | Current Node | Objective Bounds | Work Expl Unexpl | Obj Depth IntInf | Incumbent BestBd Gap | It/Node Time 0 0 0.00000 0 3 - 0.00000 - - 0s Another try with MIP start H 0 0 36.9304736 0.00000 100% - 0s H 0 0 32.4680084 0.00000 100% - 0s 0 0 0.00000 0 7 32.46801 0.00000 100% - 0s 0 0 0.00000 0 7 32.46801 0.00000 100% - 0s 0 2 0.00000 0 7 32.46801 0.00000 100% - 0s H 106 61 16.8778596 0.00000 100% 61.7 1s H 150 81 6.3285068 0.00000 100% 64.0 1s * 772 401 41 6.2748878 0.00000 100% 51.5 2s H 888 444 6.1453585 0.00000 100% 56.7 4s 953 466 0.00000 16 5 6.14536 0.00000 100% 58.6 5s 2465 667 5.66918 34 5 6.14536 0.02050 100% 55.1 10s * 2577 683 52 6.0729847 0.02050 100% 54.6 10s * 3003 803 49 5.7567275 0.02050 100% 53.3 10s 4740 1353 0.96158 29 8 5.75673 0.33754 94.1% 49.1 15s 7176 1971 3.40887 37 11 5.75673 0.55686 90.3% 46.5 20s 9473 2472 4.86502 32 5 5.75673 0.66565 88.4% 46.5 25s * 9568 2095 50 4.6839523 0.66565 85.8% 46.4 25s 11702 2322 2.86913 33 9 4.68395 0.97990 79.1% 46.1 30s *13768 2488 53 4.6601251 1.27564 72.6% 45.6 34s 13993 2555 cutoff 29 4.66013 1.33004 71.5% 45.3 35s H14661 2643 4.6418593 1.35669 70.8% 45.2 39s 14760 2643 cutoff 38 4.64186 1.37322 70.4% 45.1 40s 16838 2691 1.69505 24 8 4.64186 1.69505 63.5% 44.0 45s 18962 2868 cutoff 27 4.64186 2.07242 55.4% 43.0 50s 20514 2814 4.08882 31 5 4.64186 2.37951 48.7% 42.3 55s 22468 2601 cutoff 35 4.64186 2.71056 41.6% 41.6 60s 24747 2361 cutoff 23 4.64186 3.08461 33.5% 40.3 65s 26908 1883 3.51087 27 3 4.64186 3.49281 24.8% 39.4 70s 28747 1429 cutoff 36 4.64186 3.70938 20.1% 39.0 75s 30599 580 cutoff 40 4.64186 4.16933 10.2% 38.3 80s Cutting planes: Gomory: 1 Projected implied bound: 41 MIR: 21 Flow cover: 19 Explored 31449 nodes (1197329 simplex iterations) in 82.56 seconds Thread count was 4 (of 4 available processors) Solution count 10: 4.64186 4.66013 4.68395 ... 32.468 Optimal solution found (tolerance 1.00e-04) Best objective 4.641859316319e+00, best bound 4.641859316319e+00, gap 0.0000%
We use the logger provided by the Memento
package. Rather than just having logs written to STDOUT, you can choose to capture the output (for example, to write logs to file) by adding your own handlers to MIPVerify.LOGGER
. See the Memento documentation for more.
typeof(MIPVerify.LOGGER)
Memento.Logger