We use the same example as in the quickstart.
using MIPVerify
using Gurobi
mnist = MIPVerify.read_datasets("MNIST")
n1params = MIPVerify.get_example_network_params("MNIST.n1")
sample_image = MIPVerify.get_image(mnist.train.images, 1)
target_label_index = 9
9
Our package logs to STDOUT at the debug
, info
, notice
and warn
levels.
The default output level is to show logs only at the notice
level and above, which is also the recommended level.
d = MIPVerify.find_adversarial_example(n1params, sample_image, target_label_index, GurobiSolver(), rebuild=true);
[notice | MIPVerify]: Rebuilding model from scratch. This may take some time as we determine upper and lower bounds for the input to each non-linear unit. The model built will be cached and re-used for future solves, unless you explicitly set rebuild=false. [notice | MIPVerify]: Attempting to find adversarial example. Neural net predicted label is 8, target labels are [9] Optimize a model with 3385 rows, 3256 columns and 71132 nonzeros Variable types: 3196 continuous, 60 integer (60 binary) Coefficient statistics: Matrix range [2e-05, 7e+02] Objective range [1e+00, 1e+00] Bounds range [1e+00, 1e+02] RHS range [1e-02, 7e+02] Presolve removed 2860 rows and 2184 columns Presolve time: 0.11s Presolved: 525 rows, 1072 columns, 65472 nonzeros MIP start did not produce a new incumbent solution MIP start violates constraint R1024 by 1.000000000 Variable types: 1012 continuous, 60 integer (60 binary) Root relaxation: objective 0.000000e+00, 242 iterations, 0.01 seconds Nodes | Current Node | Objective Bounds | Work Expl Unexpl | Obj Depth IntInf | Incumbent BestBd Gap | It/Node Time 0 0 0.00000 0 9 - 0.00000 - - 0s Another try with MIP start H 0 0 55.3332559 0.00000 100% - 0s 0 0 0.00000 0 10 55.33326 0.00000 100% - 0s H 0 0 27.1696380 0.00000 100% - 0s 0 0 0.00000 0 10 27.16964 0.00000 100% - 0s 0 0 0.00000 0 11 27.16964 0.00000 100% - 0s H 0 0 9.8928387 0.00000 100% - 0s H 0 0 9.8622769 0.00000 100% - 0s 0 0 0.00000 0 1 9.86228 0.00000 100% - 0s 0 0 0.00000 0 1 9.86228 0.00000 100% - 0s 0 0 0.00000 0 1 9.86228 0.00000 100% - 0s 0 2 0.00000 0 1 9.86228 0.00000 100% - 0s * 354 156 25 9.7569972 0.00000 100% 68.9 1s 4777 468 8.90557 19 10 9.75700 4.59017 53.0% 58.4 5s Cutting planes: Gomory: 3 Cover: 3 MIR: 7 Flow cover: 10 Explored 6902 nodes (388477 simplex iterations) in 6.66 seconds Thread count was 8 (of 8 available processors) Solution count 5: 9.757 9.86228 9.89284 ... 55.3333 Pool objective bound 9.757 Optimal solution found (tolerance 1.00e-04) Best objective 9.756997241475e+00, best bound 9.756997241475e+00, gap 0.0000%
More information is available if we set the log level to debug
.
MIPVerify.setloglevel!("debug")
d = MIPVerify.find_adversarial_example(n1params, sample_image, target_label_index, GurobiSolver(), rebuild=true)
[notice | MIPVerify]: Rebuilding model from scratch. This may take some time as we determine upper and lower bounds for the input to each non-linear unit. The model built will be cached and re-used for future solves, unless you explicitly set rebuild=false. [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δl = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δl = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δl = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δl = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δl = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δl = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δl = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δl = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δl = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δl = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δl = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δl = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δl = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δl = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δl = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δl = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δl = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δl = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δl = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δl = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δl = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δl = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δl = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δl = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δl = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δl = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δl = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δl = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δl = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δl = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δl = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δl = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δl = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δl = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δl = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δl = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δl = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δl = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δl = 0.0 [debug | MIPVerify]: Δu = 0.0 [debug | MIPVerify]: Δl = 0.0 [debug | MIPVerify]: Δu = 236.19499039114618 [debug | MIPVerify]: Δl = 329.4706219653086 [debug | MIPVerify]: Δu = 347.3507384582369 [debug | MIPVerify]: Δl = 490.3354319365959 [debug | MIPVerify]: Δu = 111.66031695101941 [debug | MIPVerify]: Δl = 230.85744098698652 [debug | MIPVerify]: Δu = 110.09320929282381 [info | MIPVerify]: Hit user limit during solve to determine bounds. Multiplicative gap was 0.5778275495135097. [debug | MIPVerify]: Δl = 250.1943900717896 [debug | MIPVerify]: Δu = 263.9686250131348 [debug | MIPVerify]: Δl = 443.7377089444059 [debug | MIPVerify]: Δu = 170.58623081063135 [debug | MIPVerify]: Δl = 546.9967283877565 [debug | MIPVerify]: Δu = 342.81111856390635 [debug | MIPVerify]: Δl = 561.8499789605797 [debug | MIPVerify]: Δu = 322.1609214959067 [debug | MIPVerify]: Δl = 252.888615674684 [debug | MIPVerify]: Δu = 190.94181084213062 [info | MIPVerify]: Hit user limit during solve to determine bounds. Multiplicative gap was 0.9188018145812635. [debug | MIPVerify]: Δl = 389.16818001562916 [debug | MIPVerify]: Δu = 120.5688694681638 [info | MIPVerify]: Hit user limit during solve to determine bounds. Multiplicative gap was 0.43244015627890753. [debug | MIPVerify]: Δl = 314.1279773027621 [debug | MIPVerify]: Δu = 333.3978216299831 [debug | MIPVerify]: Δl = 395.66929790133076 [debug | MIPVerify]: Δu = 156.25265560651098 [debug | MIPVerify]: Δl = 263.63110342096843 [debug | MIPVerify]: Δu = 306.08282612869147 [debug | MIPVerify]: Δl = 259.3597215144569 [debug | MIPVerify]: Δu = 176.38206258926618 [debug | MIPVerify]: Δl = 279.1469821915614 [debug | MIPVerify]: Δu = 301.9881744933099 [debug | MIPVerify]: Δl = 363.84051783382193 [info | MIPVerify]: Hit user limit during solve to determine bounds. Multiplicative gap was 0.4156007693486492. [debug | MIPVerify]: Δu = 306.02212780906495 [debug | MIPVerify]: Δl = 223.20192365427127 [debug | MIPVerify]: Δu = 146.17861158898927 [debug | MIPVerify]: Δl = 397.6660964044127 [info | MIPVerify]: Hit user limit during solve to determine bounds. Multiplicative gap was 0.23529601270679312. [debug | MIPVerify]: Δu = 278.0674748427061 [debug | MIPVerify]: Δl = 354.40977058244283 [debug | MIPVerify]: Δu = 184.93657306973148 [debug | MIPVerify]: Δl = 347.3454124031803 [debug | MIPVerify]: Δu = 256.17035614083665 [debug | MIPVerify]: Δl = 540.5128944047548 [notice | MIPVerify]: Attempting to find adversarial example. Neural net predicted label is 8, target labels are [9] Optimize a model with 3385 rows, 3256 columns and 71132 nonzeros Variable types: 3196 continuous, 60 integer (60 binary) Coefficient statistics: Matrix range [2e-05, 7e+02] Objective range [1e+00, 1e+00] Bounds range [1e+00, 1e+02] RHS range [1e-02, 7e+02] Presolve removed 2860 rows and 2184 columns Presolve time: 0.13s Presolved: 525 rows, 1072 columns, 65472 nonzeros MIP start did not produce a new incumbent solution MIP start violates constraint R1024 by 1.000000000 Variable types: 1012 continuous, 60 integer (60 binary) Root relaxation: objective 0.000000e+00, 242 iterations, 0.01 seconds Nodes | Current Node | Objective Bounds | Work Expl Unexpl | Obj Depth IntInf | Incumbent BestBd Gap | It/Node Time 0 0 0.00000 0 9 - 0.00000 - - 0s Another try with MIP start H 0 0 55.3332559 0.00000 100% - 0s 0 0 0.00000 0 10 55.33326 0.00000 100% - 0s H 0 0 27.1696380 0.00000 100% - 0s 0 0 0.00000 0 10 27.16964 0.00000 100% - 0s 0 0 0.00000 0 11 27.16964 0.00000 100% - 0s H 0 0 9.8928387 0.00000 100% - 0s H 0 0 9.8622769 0.00000 100% - 0s 0 0 0.00000 0 1 9.86228 0.00000 100% - 0s 0 0 0.00000 0 1 9.86228 0.00000 100% - 0s 0 0 0.00000 0 1 9.86228 0.00000 100% - 0s 0 2 0.00000 0 1 9.86228 0.00000 100% - 0s * 839 309 27 9.7569972 0.00000 100% 70.8 1s 4504 347 5.30680 13 13 9.75700 5.30680 45.6% 62.8 5s Cutting planes: Gomory: 3 Cover: 2 MIR: 7 Flow cover: 11 Explored 6030 nodes (364445 simplex iterations) in 6.24 seconds Thread count was 8 (of 8 available processors) Solution count 5: 9.757 9.86228 9.89284 ... 55.3333 Pool objective bound 9.757 Optimal solution found (tolerance 1.00e-04) Best objective 9.756997241475e+00, best bound 9.756997241475e+00, gap 0.0000%
Dict{Symbol,Any} with 7 entries: :PerturbationParameters => additive :TargetIndexes => [9] :SolveStatus => :Optimal :Output => JuMP.GenericAffExpr{Float64,JuMP.Variable}[-0.0120… :Model => Minimization problem with:… :Perturbation => JuMP.Variable[__anon__ __anon__ __anon__ __anon__ … :PerturbedInput => JuMP.Variable[__anon__ __anon__ __anon__ __anon__ …
When we select rebuild=false
, we do not do any of the work in determining upper and lower bounds, and none of the info
and debug
logs are produced.
d = MIPVerify.find_adversarial_example(n1params, sample_image, target_label_index, GurobiSolver(), rebuild=false);
[notice | MIPVerify]: Loading model from cache. [notice | MIPVerify]: Attempting to find adversarial example. Neural net predicted label is 8, target labels are [9] Optimize a model with 3385 rows, 3256 columns and 71132 nonzeros Variable types: 3196 continuous, 60 integer (60 binary) Coefficient statistics: Matrix range [2e-05, 7e+02] Objective range [1e+00, 1e+00] Bounds range [1e+00, 1e+02] RHS range [1e-02, 7e+02] Presolve removed 2860 rows and 2184 columns Presolve time: 0.13s Presolved: 525 rows, 1072 columns, 65472 nonzeros MIP start did not produce a new incumbent solution MIP start violates constraint R1024 by 1.000000000 Variable types: 1012 continuous, 60 integer (60 binary) Root relaxation: objective 0.000000e+00, 242 iterations, 0.01 seconds Nodes | Current Node | Objective Bounds | Work Expl Unexpl | Obj Depth IntInf | Incumbent BestBd Gap | It/Node Time 0 0 0.00000 0 9 - 0.00000 - - 0s Another try with MIP start H 0 0 55.3332559 0.00000 100% - 0s 0 0 0.00000 0 10 55.33326 0.00000 100% - 0s H 0 0 27.1696380 0.00000 100% - 0s 0 0 0.00000 0 10 27.16964 0.00000 100% - 0s 0 0 0.00000 0 11 27.16964 0.00000 100% - 0s H 0 0 9.8928387 0.00000 100% - 0s H 0 0 9.8622769 0.00000 100% - 0s 0 0 0.00000 0 1 9.86228 0.00000 100% - 0s 0 0 0.00000 0 1 9.86228 0.00000 100% - 0s 0 0 0.00000 0 1 9.86228 0.00000 100% - 0s 0 2 0.00000 0 1 9.86228 0.00000 100% - 0s * 839 309 27 9.7569972 0.00000 100% 70.8 1s 4504 347 5.30680 13 13 9.75700 5.30680 45.6% 62.8 5s Cutting planes: Gomory: 3 Cover: 2 MIR: 7 Flow cover: 11 Explored 6030 nodes (364445 simplex iterations) in 6.24 seconds Thread count was 8 (of 8 available processors) Solution count 5: 9.757 9.86228 9.89284 ... 55.3333 Pool objective bound 9.757 Optimal solution found (tolerance 1.00e-04) Best objective 9.756997241475e+00, best bound 9.756997241475e+00, gap 0.0000%
typeof(MIPVerify.LOGGER)
Memento.Logger