Notebook

Demo of benchmarking a 4-qubit parity check¶

This notebook demonstrates how to compute the "weight-X disturbance" error metric between a reference and test data set using pyGSTi.

In [1]:

import numpy as np
import scipy as sp
import itertools
from matplotlib import pyplot as plt
import seaborn as sns
import time
import cvxpy
sns.set_style('white')
%matplotlib inline

import pygsti.extras.paritybenchmarking as pb
#Note: if you don't want to install pyGSTi to run this notebook,
# just grab the pygsti/extras/paritybenchmarking/disturbancecalc.py file, put it
# in the same directory as this notebook, and use this line instead of the one above:
# import disturbancecalc as pb

print(cvxpy.installed_solvers())
SOLVER='SCS'  # if you see 'MOSEK' as an option below, change this to SOLVER='MOSEK' 
              # as this is a better (but less widely available) solver

['CVXOPT', 'ECOS', 'GLPK', 'GLPK_MI', 'OSQP', 'SCS']

Setup¶

Here's some setup so we can easily work with probability distributions as arrays of probabilities, indexed by bitstring.

In [2]:

n_bits = 4
n_data_points = 10000

class ProbabilityDistribution(np.ndarray):
    @classmethod
    def from_array(cls, ar):
        ret = cls(int(round(np.log2(len(ar)))))
        ret[:] = ar[:]
        return ret

    def __new__(cls, n_bits):
        self = super().__new__(cls, (2**n_bits,), float, np.zeros(2**n_bits, 'd'))
        self.n_bits = n_bits
        self.bs = ["".join(x) for x in itertools.product(*([('0', '1')] * n_bits))]
        self.bi = {s: i for i,s in enumerate(self.bs)}
        return self
  
    def __getitem__(self, key):
        if isinstance(key, str):
            return super().__getitem__(self.bi[key])
        else:
            return super().__getitem__(key)
    
    def __setitem__(self, key, val):
        if isinstance(key, str):
            return super().__setitem__(self.bi[key], val)
        else:
            return super().__setitem__(key, val)
        
    def display(self, my_name='', other_distributions=None):
        if other_distributions is None: other_distributions = {}
        other_names = list(other_distributions.keys())
        print('     ', ', '.join([my_name] + other_names))
        print('\n'.join(["%s: %s" % (s, '  '.join([('%.3f' % x) for x in [v] + [other_distributions[nm][s] for nm in other_names]]))
                         for s,v in zip(self.bs, self)]))
    
    def __str__(self):
        return '\n'.join(["%s: %g" % (s, v) for s,v in zip(self.bs, self)]) + '\n'
    
    def __array_finalize__(self, obj):  # needed for creating new array instances
        if obj is None: return
        self.n_bits = getattr(obj, 'n_bits', None)
        self.bi = getattr(obj, 'bi', None)
        self.bs = getattr(obj, 'bs', None)
        
    def copy(self):
        ret = ProbabilityDistribution(self.n_bits)
        ret[:] = self[:]
        return ret

The main demo: a simple example¶

Let's start with an example problem, computing the weight-1 to weight-4 disturbances based on some data. This demonstrates the main interface to using the disturbancecalc.py code: the compute_disturbances function.

In [3]:

# Define the ideal distribution
p_ideal = ProbabilityDistribution(n_bits)
p_ideal['0000'] = .5
p_ideal['1111'] = .5


# Define the "reference" distribution.  In this case, we just set it equal to the ideal distribution
p_ref = p_ideal.copy()

# Define several elementary errors that can happen to a single bit:
ident1bit = np.identity(2)  # do nothing to a single bit
flip1bit = np.array([[0, 1],[1, 0]])  # flip a single bit
set1bit = np.array([[0, 0], [1, 1]])  # set a single bit (to 1)
reset1bit = np.array([[1, 1], [0, 0]])  # reset a single bit (to 0)

# Define, for example, and for use later on, several 4-bit errors.
def kron4(a, b, c, d):
    return np.kron(a, np.kron(b, np.kron(c, d)))
ident4bit = np.identity(16)
flip4bit = kron4(flip1bit, flip1bit, flip1bit, flip1bit)
fliplastbit = kron4(ident1bit, ident1bit, ident1bit, flip1bit)
setlastbit = np.kron(ident1bit, np.kron(ident1bit, np.kron(ident1bit, reset1bit)))
set4bit = kron4(set1bit, set1bit, set1bit, set1bit)

# Construct, for our example, a particular stochastic map that represents
# the "error" or "noise" that occurs during the parity check experiment.
parity_check_noise = 0.90 * ident4bit + 0.05 * set4bit + 0.05 * fliplastbit
#parity_check_noise = 0.90 * ident4bit + 0.1 * fliplastbit  # another possibility

# The "test" distribution is thus defined by applying the above map to p_ref:
p_test = ProbabilityDistribution.from_array(np.dot(parity_check_noise, p_ref))

# We can now simulate data corresponding to these probability distributions.
# Here we don't include proper finite sample error for simplicity, we just
# round to the nearest integer:
data_ref = np.array( p_ref * n_data_points, dtype='int')  # no finite sample error
data_test = np.array(p_test * n_data_points, dtype='int')  # no finite sample error

# This is what you would do to simulate finite sampling error:
# data_ref = np.random.multinomial(n_data_points, p_ref)
# data_test = np.random.multinomial(n_data_points, p_test)

# Before computing the disturbances, let's take a look at the probability distributions:
p_ideal.display('ideal', {'ref': p_ref, 'test': p_test})

      ideal, ref, test
0000: 0.500  0.500  0.450
0001: 0.000  0.000  0.025
0010: 0.000  0.000  0.000
0011: 0.000  0.000  0.000
0100: 0.000  0.000  0.000
0101: 0.000  0.000  0.000
0110: 0.000  0.000  0.000
0111: 0.000  0.000  0.000
1000: 0.000  0.000  0.000
1001: 0.000  0.000  0.000
1010: 0.000  0.000  0.000
1011: 0.000  0.000  0.000
1100: 0.000  0.000  0.000
1101: 0.000  0.000  0.000
1110: 0.000  0.000  0.025
1111: 0.500  0.500  0.500

In [4]:

# The actual disturbance computation
disturbances = pb.compute_disturbances(n_bits, data_ref, data_test, solver=SOLVER)

Computing base disturbances
Analyzing bootstrap sample 1 of 20...

/Users/enielse/.pyenv/versions/3.8.5/lib/python3.8/site-packages/cvxpy/problems/problem.py:1054: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
  warnings.warn(

 (0.5s)
Analyzing bootstrap sample 2 of 20... (0.4s)
Analyzing bootstrap sample 3 of 20... (0.4s)
Analyzing bootstrap sample 4 of 20... (0.4s)
Analyzing bootstrap sample 5 of 20... (0.5s)
Analyzing bootstrap sample 6 of 20... (1.2s)
Analyzing bootstrap sample 7 of 20... (0.5s)
Analyzing bootstrap sample 8 of 20... (0.7s)
Analyzing bootstrap sample 9 of 20... (0.5s)
Analyzing bootstrap sample 10 of 20... (0.6s)
Analyzing bootstrap sample 11 of 20... (0.5s)
Analyzing bootstrap sample 12 of 20... (0.6s)
Analyzing bootstrap sample 13 of 20... (0.5s)
Analyzing bootstrap sample 14 of 20... (0.6s)
Analyzing bootstrap sample 15 of 20... (0.5s)
Analyzing bootstrap sample 16 of 20... (0.5s)
Analyzing bootstrap sample 17 of 20... (0.5s)
Analyzing bootstrap sample 18 of 20... (0.5s)
Analyzing bootstrap sample 19 of 20... (0.4s)
Analyzing bootstrap sample 20 of 20... (0.5s)

In [5]:

# This shows how the returned `disturbances` list is interpreted (see the dostring too)
for i,d in enumerate(disturbances):
    print("Weight-%d disturbance = %g +/- %g" % (i+1,d[0],d[1]))

Weight-1 disturbance = 0.025 +/- 0.00463123
Weight-2 disturbance = -1.82279e-09 +/- 0.000378904
Weight-3 disturbance = 0.0249996 +/- 0.00471013
Weight-4 disturbance = 3.68479e-07 +/- 0.0045456

That's all there is to computing the disturbance metrics! From this point on, this notebook demonstrates the more recent OVD-based corrections to the disturbances and then shows some of the other things the QPL has done to test and verify that the disturbance metrics are computed correctly and behave as they're supposed to.

OVD-corrected disturbances¶

To make the disturbance metrics more robust to SPAM noise, you can apply a scaling factor that is the ratio of the original variation distance (OVD) to the usual total variational distance (TVD) between the reference and test distributions. The ratio can be computed independently, or, for convenience, methods exist which perform the scaling internally. Both are demonstrated below.

In [6]:

# Consider an example where the parity check circuit produces the same error
# (`parity_check_noise` from above) but with SPAM noise, so that p_ref is no longer perfect.

# We construct SPAM noise that is the tensor produce to a low-probability bit flip on the final qubit:
lowprobflip1bit = 0.95 * ident1bit + 0.05 * flip1bit
spam_noise = kron4(ident1bit, ident1bit, ident1bit, lowprobflip1bit)
            
# The "reference" distribution is then defined by applying the above map to p_ideal:
p_ref = ProbabilityDistribution.from_array(np.dot(spam_noise, p_ideal))

# And the "test" distribution is defined by applying the same parity_check_noise from before to p_ref:
p_test = ProbabilityDistribution.from_array(np.dot(parity_check_noise, p_ref))

#We can then simulate data and view these distributions like we did before:
data_ref = np.array( p_ref * n_data_points, dtype='int')  # no finite sample error
data_test = np.array(p_test * n_data_points, dtype='int')  # no finite sample error

# This is what you would do to simulate finite sampling error:
# data_ref = np.random.multinomial(n_data_points, p_ref)
# data_test = np.random.multinomial(n_data_points, p_test)

p_ideal.display('ideal', {'ref': p_ref, 'test': p_test})

      ideal, ref, test
0000: 0.500  0.475  0.429
0001: 0.000  0.025  0.046
0010: 0.000  0.000  0.000
0011: 0.000  0.000  0.000
0100: 0.000  0.000  0.000
0101: 0.000  0.000  0.000
0110: 0.000  0.000  0.000
0111: 0.000  0.000  0.000
1000: 0.000  0.000  0.000
1001: 0.000  0.000  0.000
1010: 0.000  0.000  0.000
1011: 0.000  0.000  0.000
1100: 0.000  0.000  0.000
1101: 0.000  0.000  0.000
1110: 0.000  0.025  0.046
1111: 0.500  0.475  0.479

In [7]:

# We next go on to compute the OVD/TVD ratio, the usual TVD-based disturbances, and the OVD-corrected disturbances.
r = ovd_over_tvd = pb.compute_ovd_over_tvd_ratio(n_bits, data_ref, data_test, p_ideal)
print("OVD/TVD ratio = ",r)

OVD/TVD ratio =  1.0526315789473677

In [8]:

# Usual TVD-based disturbances
disturbances = pb.compute_disturbances(n_bits, data_ref, data_test, solver=SOLVER)

Computing base disturbances
Analyzing bootstrap sample 1 of 20... (0.5s)
Analyzing bootstrap sample 2 of 20... (0.5s)
Analyzing bootstrap sample 3 of 20... (0.5s)
Analyzing bootstrap sample 4 of 20... (0.5s)
Analyzing bootstrap sample 5 of 20... (1.0s)
Analyzing bootstrap sample 6 of 20... (0.6s)
Analyzing bootstrap sample 7 of 20... (0.5s)
Analyzing bootstrap sample 8 of 20... (0.6s)
Analyzing bootstrap sample 9 of 20... (0.5s)
Analyzing bootstrap sample 10 of 20... (0.5s)
Analyzing bootstrap sample 11 of 20... (0.5s)
Analyzing bootstrap sample 12 of 20... (0.5s)
Analyzing bootstrap sample 13 of 20... (0.5s)
Analyzing bootstrap sample 14 of 20... (0.5s)
Analyzing bootstrap sample 15 of 20... (0.5s)
Analyzing bootstrap sample 16 of 20... (0.5s)
Analyzing bootstrap sample 17 of 20... (0.5s)
Analyzing bootstrap sample 18 of 20... (0.5s)
Analyzing bootstrap sample 19 of 20... (0.5s)
Analyzing bootstrap sample 20 of 20... (0.5s)

In [9]:

# This shows how the returned `disturbances` list is interpreted (see the dostring too)
for i,d in enumerate(disturbances):
    print("Weight-%d disturbance = %g +/- %g" % (i+1,d[0],d[1]))

Weight-1 disturbance = 0.0212092 +/- 0.00400825
Weight-2 disturbance = -5.56941e-08 +/- 0.000393251
Weight-3 disturbance = 0.0250039 +/- 0.00690518
Weight-4 disturbance = 1.18704e-06 +/- 0.000803861

In [10]:

ovd_corrected_disturbances = pb.compute_ovd_corrected_disturbances(n_bits, data_ref, data_test, p_ideal, solver=SOLVER)

Computing base disturbances
Analyzing bootstrap sample 1 of 20... (0.5s)
Analyzing bootstrap sample 2 of 20... (0.5s)
Analyzing bootstrap sample 3 of 20... (0.6s)
Analyzing bootstrap sample 4 of 20... (0.5s)
Analyzing bootstrap sample 5 of 20... (1.1s)
Analyzing bootstrap sample 6 of 20... (0.5s)
Analyzing bootstrap sample 7 of 20... (0.5s)
Analyzing bootstrap sample 8 of 20... (0.5s)
Analyzing bootstrap sample 9 of 20... (0.5s)
Analyzing bootstrap sample 10 of 20... (0.5s)
Analyzing bootstrap sample 11 of 20... (0.5s)
Analyzing bootstrap sample 12 of 20... (0.5s)
Analyzing bootstrap sample 13 of 20... (0.5s)
Analyzing bootstrap sample 14 of 20... (0.5s)
Analyzing bootstrap sample 15 of 20... (0.5s)
Analyzing bootstrap sample 16 of 20... (0.5s)
Analyzing bootstrap sample 17 of 20... (0.5s)
Analyzing bootstrap sample 18 of 20... (0.5s)
Analyzing bootstrap sample 19 of 20... (0.6s)
Analyzing bootstrap sample 20 of 20... (0.5s)

In [11]:

# This shows how the returned `disturbances` list is interpreted (see the dostring too)
for i,d in enumerate(ovd_corrected_disturbances):
    if i < 4:
        print("Weight-%d OVD-corrected disturbance = %g +/- %g" % (i+1,d[0],d[1]))
    else:
        print("OVD/TVD ratio = %g +/- %g" % (d[0],d[1]))

Weight-1 OVD-corrected disturbance = 0.0223255 +/- 0.00423156
Weight-2 OVD-corrected disturbance = -5.86253e-08 +/- 0.000411076
Weight-3 OVD-corrected disturbance = 0.0263199 +/- 0.00710352
Weight-4 OVD-corrected disturbance = 1.24952e-06 +/- 0.000836451
OVD/TVD ratio = 1.05263 +/- 0.0120014

In the above example, we note:

the OVD/ratio is close to 1, and so the OVD-corrected vs uncorrected numbers are similar. More precisely, the ratio is ~1.05, and so the corrected values are slightly larger than the uncorrected values.
the correction moves the disturbances closer to those the values given earlier when there was no SPAM error.

One more OVD-corrected disturbance example¶

It's also possible to have SPAM noise increase the disturbances rather than reduce them. In the example below, which uses SPAM noise that afflicts all the qubits equally, the presence of the SPAM noise actually makes the disturbance values larger than they would be without the noise. In this case r is (slightly) less than 1, and the OVD-corrected disturbances are smaller than the uncorrected ones.

In [12]:

#All the steps here are similar to the example above, except the SPAM noise is different
lowprobflip1bit = 0.98 * ident1bit + 0.02 * flip1bit
spam_noise = kron4(lowprobflip1bit, lowprobflip1bit, lowprobflip1bit, lowprobflip1bit)
            
p_ref = ProbabilityDistribution.from_array(np.dot(spam_noise, p_ideal))
p_test = ProbabilityDistribution.from_array(np.dot(parity_check_noise, p_ref))

data_ref = np.array( p_ref * n_data_points, dtype='int')  # no finite sample error
data_test = np.array(p_test * n_data_points, dtype='int')  # no finite sample error

p_ideal.display('ideal', {'ref': p_ref, 'test': p_test})

      ideal, ref, test
0000: 0.500  0.461  0.416
0001: 0.000  0.009  0.032
0010: 0.000  0.009  0.008
0011: 0.000  0.000  0.001
0100: 0.000  0.009  0.008
0101: 0.000  0.000  0.001
0110: 0.000  0.000  0.001
0111: 0.000  0.009  0.008
1000: 0.000  0.009  0.008
1001: 0.000  0.000  0.001
1010: 0.000  0.000  0.001
1011: 0.000  0.009  0.008
1100: 0.000  0.000  0.001
1101: 0.000  0.009  0.008
1110: 0.000  0.009  0.032
1111: 0.500  0.461  0.466

In [13]:

r = ovd_over_tvd = pb.compute_ovd_over_tvd_ratio(n_bits, data_ref, data_test, p_ideal)
print("OVD/TVD ratio = ",r)

#We don't do any bootstrap samples so the calculation is faster, but there are no error bars
disturbances = pb.compute_disturbances(n_bits, data_ref, data_test, solver=SOLVER, num_bootstrap_samples=0)
ovd_corrected_disturbances = pb.compute_ovd_corrected_disturbances(n_bits, data_ref, data_test, p_ideal, solver=SOLVER, num_bootstrap_samples=0)
print()

for i,d in enumerate(disturbances):
    print("Weight-%d disturbance = %g" % (i+1,d[0]))
    
print()
    
for i,d in enumerate(ovd_corrected_disturbances):
    if i < 4:
        print("Weight-%d OVD-corrected disturbance = %g" % (i+1,d[0]))
    else:
        print("OVD/TVD ratio = %g" % (d[0]))

OVD/TVD ratio =  0.9575080822905789
Computing base disturbances
Computing base disturbances

Weight-1 disturbance = 0.0256046
Weight-2 disturbance = 0.00328173
Weight-3 disturbance = 0.0191035
Weight-4 disturbance = 0.00365153

Weight-1 OVD-corrected disturbance = 0.0245166
Weight-2 OVD-corrected disturbance = 0.00314228
Weight-3 OVD-corrected disturbance = 0.0182917
Weight-4 OVD-corrected disturbance = 0.00349637
OVD/TVD ratio = 0.957508

Testing and verification¶

Test problems.¶

Below we put what we did above in the "main demo" section into a function so it's easier to run, and perform a bunch of simple tests to ensure that the disturbance behavior agrees with our intuition.

In [14]:

def gen_data(p_test_dict, p_ref_dict = None, include_finite_sample_error=False, seed=12345):
    # Define the reference distribution - in this case, it's equal to the target distribution
    p_ref = ProbabilityDistribution(n_bits)
    for k,v in p_ref_dict.items(): p_ref[k] = v
        
    if include_finite_sample_error:
        np.random.seed(seed)
        data_ref = np.random.multinomial(n_data_points, p_ref)
    else:
        data_ref = np.array( p_ref * n_data_points, dtype='int')  # no finite sample error
   
    # Define the reference distribution - we get this by moving 
    p_test = ProbabilityDistribution(n_bits)
    for k,v in p_test_dict.items(): p_test[k] = v
        
    if include_finite_sample_error:
        data_test = np.random.multinomial(n_data_points, p_test)
    else:
        data_test = np.array( p_test * n_data_points, dtype='int')  # no finite sample error
        
    return data_ref, data_test
        

def test(p_test_dict, p_ref_dict = None, include_finite_sample_error=False, seed=12345, confidence_percent=68):
    if p_ref_dict is None:
        p_ref_dict = {'0000': 0.5, '1111': 0.5}
    
    data_ref, data_test = gen_data(p_test_dict, p_ref_dict, include_finite_sample_error, seed)
    disturbances = pb.compute_disturbances(n_bits, data_ref, data_test, verbosity=0, solver=SOLVER)
    print("TEST (finite sampling = %s)" % include_finite_sample_error)
    print("p_ref = ", p_ref_dict)
    print("p_test = ", p_test_dict)
    for i,d in enumerate(disturbances):
        if d[1] is None:
            print("Weight-%d disturbance = %g" % (i+1,d[0]))
        else:
            print("Weight-%d disturbance = %g +/- %g" % (i+1,d[0],d[1]))
    print()

No finite sample error¶

The tests below have no sample error. In other words, we've just constructed fake "data" where the observed frequencies are equal to known probabilities for which we know what the disturbance metrics should be. So these tests are crafted to make it easy to interpret the outcome disturbances, and to make sure (for instance) that if we've constructed a pair of distributions that differ by a weight-2 error, we get a nonzero weight-2 disturbance (but approximately zero for the other disturbance metrics).

In [15]:

#Simple tests - no finite sample error
test({'0000': 0.45, '0001': 0.05, '1111': 0.5}) # weight 1 movement 
test({'0000': 0.45, '0011': 0.05, '1111': 0.5}) # weight 2 movement 
test({'0000': 0.45, '0111': 0.05, '1111': 0.5}) # weight 3 movement
test({'0000': 0.45, '1111': 0.55}) # weight 4 movement

TEST (finite sampling = False)
p_ref =  {'0000': 0.5, '1111': 0.5}
p_test =  {'0000': 0.45, '0001': 0.05, '1111': 0.5}
Weight-1 disturbance = 0.05 +/- 0.00495419
Weight-2 disturbance = 6.04827e-11 +/- 0.000394579
Weight-3 disturbance = 0 +/- 0.00460903
Weight-4 disturbance = 0 +/- 0.00481946

TEST (finite sampling = False)
p_ref =  {'0000': 0.5, '1111': 0.5}
p_test =  {'0000': 0.45, '0011': 0.05, '1111': 0.5}
Weight-1 disturbance = 0 +/- 0.000502418
Weight-2 disturbance = 0.05 +/- 0.00189544
Weight-3 disturbance = 3.37029e-09 +/- 0.000160759
Weight-4 disturbance = 0 +/- 0.00401039

TEST (finite sampling = False)
p_ref =  {'0000': 0.5, '1111': 0.5}
p_test =  {'0000': 0.45, '0111': 0.05, '1111': 0.5}
Weight-1 disturbance = 6.93889e-18 +/- 0.00558415
Weight-2 disturbance = -6.93889e-18 +/- 0.000411401
Weight-3 disturbance = 0.0499999 +/- 0.0053436
Weight-4 disturbance = 6.40582e-08 +/- 0.00422338

TEST (finite sampling = False)
p_ref =  {'0000': 0.5, '1111': 0.5}
p_test =  {'0000': 0.45, '1111': 0.55}
Weight-1 disturbance = 0 +/- 0.000301763
Weight-2 disturbance = 0 +/- 0.000383953
Weight-3 disturbance = 0 +/- 0.000321087
Weight-4 disturbance = 0.05 +/- 0.0062984

In [16]:

#More complex tests
test({'0000': 0.40, '0001': 0.05, '1111': 0.55}) # weight 1+4 movement 
test({'0000': 0.40, '0011': 0.05, '1111': 0.55}) # weight 2+4 movement 
test({'0000': 0.40, '0111': 0.05, '1111': 0.55}) # weight 3+4 movement

TEST (finite sampling = False)
p_ref =  {'0000': 0.5, '1111': 0.5}
p_test =  {'0000': 0.4, '0001': 0.05, '1111': 0.55}
Weight-1 disturbance = 0.05 +/- 0.00242677
Weight-2 disturbance = 0 +/- 0.00035457
Weight-3 disturbance = 0 +/- 0.000332917
Weight-4 disturbance = 0.05 +/- 0.00670465

TEST (finite sampling = False)
p_ref =  {'0000': 0.5, '1111': 0.5}
p_test =  {'0000': 0.4, '0011': 0.05, '1111': 0.55}
Weight-1 disturbance = 0 +/- 0.000313164
Weight-2 disturbance = 0.0499999 +/- 0.00182238
Weight-3 disturbance = 8.03188e-08 +/- 0.000408627
Weight-4 disturbance = 0.05 +/- 0.00655976

TEST (finite sampling = False)
p_ref =  {'0000': 0.5, '1111': 0.5}
p_test =  {'0000': 0.4, '0111': 0.05, '1111': 0.55}
Weight-1 disturbance = 0 +/- 0.000258968
Weight-2 disturbance = 0 +/- 0.000421522
Weight-3 disturbance = 0.05 +/- 0.00197274
Weight-4 disturbance = 0.05 +/- 0.00689537

In [17]:

#Different magnitudes of weight-2+4 movement
test({'0000': 0.30, '0011': 0.1, '1111': 0.6})
test({'0000': 0.35, '0011': 0.08, '1111': 0.57})
test({'0000': 0.40, '0011': 0.08, '1111': 0.52})

TEST (finite sampling = False)
p_ref =  {'0000': 0.5, '1111': 0.5}
p_test =  {'0000': 0.3, '0011': 0.1, '1111': 0.6}
Weight-1 disturbance = 0 +/- 0.000322861
Weight-2 disturbance = 0.1 +/- 0.00303067
Weight-3 disturbance = -1.23933e-08 +/- 0.000320724
Weight-4 disturbance = 0.1 +/- 0.00560226

TEST (finite sampling = False)
p_ref =  {'0000': 0.5, '1111': 0.5}
p_test =  {'0000': 0.35, '0011': 0.08, '1111': 0.57}
Weight-1 disturbance = 0 +/- 0.000316006
Weight-2 disturbance = 0.080008 +/- 0.00262233
Weight-3 disturbance = -1.86747e-07 +/- 0.00032341
Weight-4 disturbance = 0.0699572 +/- 0.00610212

TEST (finite sampling = False)
p_ref =  {'0000': 0.5, '1111': 0.5}
p_test =  {'0000': 0.4, '0011': 0.08, '1111': 0.52}
Weight-1 disturbance = 0 +/- 0.000313116
Weight-2 disturbance = 0.0799996 +/- 0.00216252
Weight-3 disturbance = 3.74311e-07 +/- 0.000366607
Weight-4 disturbance = 0.02 +/- 0.00644853

With finite sample error¶

Now we add finite sample error. In other words, we start with known probability distributions (constructed by hand like the ones above), then simulate N=10^4 samples from those distributions, and analyze the resulting empirical frequency distributions. The point here is to (1) make sure that the estimated weight-2 disturbances are fairly close to the true ones, and (2) check that the error bar calculations correctly quantify the estimation error (i.e., provide valid confidence regions).

We begin (next cell) by just making sure the results make sense.

In [18]:

#Simple tests - *with* finite sample error
test({'0000': 0.45, '0001': 0.05, '1111': 0.5}, include_finite_sample_error=True) # weight 1 movement 
test({'0000': 0.45, '0011': 0.05, '1111': 0.5}, include_finite_sample_error=True) # weight 2 movement 
test({'0000': 0.45, '0111': 0.05, '1111': 0.5}, include_finite_sample_error=True) # weight 3 movement
test({'0000': 0.45, '1111': 0.55}, include_finite_sample_error=True) # weight 4 movement

TEST (finite sampling = True)
p_ref =  {'0000': 0.5, '1111': 0.5}
p_test =  {'0000': 0.45, '0001': 0.05, '1111': 0.5}
Weight-1 disturbance = 0.0487 +/- 0.00488717
Weight-2 disturbance = 2.13164e-08 +/- 0.000387132
Weight-3 disturbance = 0.000699976 +/- 0.00471798
Weight-4 disturbance = 4.84389e-08 +/- 0.00441123

TEST (finite sampling = True)
p_ref =  {'0000': 0.5, '1111': 0.5}
p_test =  {'0000': 0.45, '0011': 0.05, '1111': 0.5}
Weight-1 disturbance = 1.38778e-17 +/- 0.000499835
Weight-2 disturbance = 0.0493999 +/- 0.00183174
Weight-3 disturbance = 8.23359e-08 +/- 0.000134797
Weight-4 disturbance = 0 +/- 0.00360386

TEST (finite sampling = True)
p_ref =  {'0000': 0.5, '1111': 0.5}
p_test =  {'0000': 0.45, '0111': 0.05, '1111': 0.5}
Weight-1 disturbance = 0.000698094 +/- 0.00559855
Weight-2 disturbance = 1.86611e-06 +/- 0.000397603
Weight-3 disturbance = 0.0486995 +/- 0.00529992
Weight-4 disturbance = 5.38069e-07 +/- 0.00386414

TEST (finite sampling = True)
p_ref =  {'0000': 0.5, '1111': 0.5}
p_test =  {'0000': 0.45, '1111': 0.55}
Weight-1 disturbance = 0 +/- 0.00029923
Weight-2 disturbance = 0 +/- 0.000385034
Weight-3 disturbance = 0 +/- 0.000324079
Weight-4 disturbance = 0.0487 +/- 0.00629815

Check error bars¶

Finally, we check that the error bars on the disturbances capture the "true" value of the disturbance an appropriate fraction of the time. The histograms at the end of each of the cells below should show that the mean deviation/errorbar-lengths is around 1.0 (it should be $\chi^2_1$-distributed). Note: these tests run a lot of calculations and can take a while to run. (Increasing num_checks will give better estimates of the distribution but will take longer.)

In [19]:

def test_errorbars(p_test_dict, p_ref_dict = None, num_checks=20, confidence_percent=68):
    if p_ref_dict is None:
        p_ref_dict = {'0000': 0.5, '1111': 0.5}
    
    data_ref, data_test = gen_data(p_test_dict, p_ref_dict, False)
    base_disturbances = pb.compute_disturbances(n_bits, data_ref, data_test, num_bootstrap_samples=0,
                                                verbosity=0, solver=SOLVER)
    for i,d in enumerate(base_disturbances):
        print("Base weight-%d disturbance = %g" % (i+1,d[0]))
    
    deviation_over_ebs = {i: [] for i in range(4)}
    seeds = [ np.random.randint(1000000000) for i in range(num_checks) ]
    for i,s in enumerate(seeds):
        print("Check %d of %d" % (i+1, num_checks))
        data_ref, data_test = gen_data(p_test_dict, p_ref_dict, True, seed=s)
        try:
            disturbances = pb.compute_disturbances(n_bits, data_ref, data_test, num_bootstrap_samples=10,
                                                   verbosity=0, solver=SOLVER)
        except Exception as e:
            print("EXCEPTION: ", str(e))
            continue

        for i,(d,dbase) in enumerate(zip(disturbances, base_disturbances)):
            deviation_over_eb = abs((d[0] - dbase[0]) / (d[1] + 1e-6))  # add 1e-6 to avoid divide-by-zero errors
            deviation_over_ebs[i].append(deviation_over_eb)
            print("Weight-%d disturbance = %g +/- %g  (deviation=%s)" % (i+1,d[0],d[1],deviation_over_eb))
    
    print("Percentanges within %d%% confidence region:" % confidence_percent)
    for i, pts in deviation_over_ebs.items():
        print("Weight %d: " % (i+1), pts)
        plt.figure()
        plt.title("Weight %d ResidualTVD deviation distribution" % (i+1))
        plt.xlabel("deviation / error_bar_length")
        plt.ylabel("counts")
        plt.hist(pts) 
    print()

In [20]:

test_errorbars({'0000': 0.45, '0001': 0.05, '1111': 0.5}) # weight 1 movement 

Base weight-1 disturbance = 0.05
Base weight-2 disturbance = 6.04827e-11
Base weight-3 disturbance = 0
Base weight-4 disturbance = 0
Check 1 of 20
Weight-1 disturbance = 0.0494 +/- 0.00460002  (deviation=0.13040749998957704)
Weight-2 disturbance = 7.6341e-09 +/- 0.000456376  (deviation=1.655883959978033e-05)
Weight-3 disturbance = 0.000799984 +/- 0.00463315  (deviation=0.17262814538110377)
Weight-4 disturbance = 1.56229e-08 +/- 0.00405963  (deviation=3.847421888800286e-06)
Check 2 of 20
Weight-1 disturbance = 0.0458 +/- 0.00138302  (deviation=3.034643466750717)
Weight-2 disturbance = 5.62315e-09 +/- 0.000451567  (deviation=1.2291357451368336e-05)
Weight-3 disturbance = -2.64545e-17 +/- 0.00139652  (deviation=1.8929638981497028e-14)
Weight-4 disturbance = 0.0028 +/- 0.00679048  (deviation=0.4122810123143864)
Check 3 of 20
Weight-1 disturbance = 0.0463998 +/- 0.00520716  (deviation=0.6912557196715029)
Weight-2 disturbance = 2.17257e-08 +/- 0.000483705  (deviation=4.469782196117712e-05)
Weight-3 disturbance = 0.00359972 +/- 0.00503248  (deviation=0.7151551398557452)
Weight-4 disturbance = 4.3058e-07 +/- 0.00275556  (deviation=0.00015620170156731812)
Check 4 of 20
Weight-1 disturbance = 0.0392 +/- 0.00630948  (deviation=1.711438568858322)
Weight-2 disturbance = 7.60645e-10 +/- 0.000411212  (deviation=1.698548931299419e-06)
Weight-3 disturbance = 0.0094996 +/- 0.00619116  (deviation=1.5341329993162802)
Weight-4 disturbance = 3.97051e-07 +/- 0.000907713  (deviation=0.0004369379544307312)
Check 5 of 20
Weight-1 disturbance = 0.0491 +/- 0.00400089  (deviation=0.22489948793603173)
Weight-2 disturbance = -1.33702e-07 +/- 0.000417098  (deviation=0.00031993188460831636)
Weight-3 disturbance = 1.28012e-07 +/- 0.00406901  (deviation=3.1452644912367964e-05)
Weight-4 disturbance = 0.000900029 +/- 0.00442147  (deviation=0.20351256664905235)
Check 6 of 20
Weight-1 disturbance = 0.0482 +/- 0.00479077  (deviation=0.3756442110481608)
Weight-2 disturbance = 1.92187e-09 +/- 0.000438221  (deviation=4.237937857391526e-06)
Weight-3 disturbance = 0.00499995 +/- 0.00440062  (deviation=1.1359328717951638)
Weight-4 disturbance = 5.32516e-08 +/- 0.00329567  (deviation=1.6153146291952415e-05)
Check 7 of 20
Weight-1 disturbance = 0.0410996 +/- 0.00605107  (deviation=1.4706299437122539)
Weight-2 disturbance = 3.50585e-07 +/- 0.000455917  (deviation=0.0007671528235064772)
Weight-3 disturbance = 0.00769991 +/- 0.00585666  (deviation=1.3145017691407308)
Weight-4 disturbance = 9.41379e-08 +/- 0.00130469  (deviation=7.209803865361063e-05)
Check 8 of 20
Weight-1 disturbance = 0.0437 +/- 0.00369769  (deviation=1.7033118048074751)
Weight-2 disturbance = -7.12836e-09 +/- 0.000446259  (deviation=1.6073118227795782e-05)
Weight-3 disturbance = 2.39058e-08 +/- 0.00382004  (deviation=6.256354525456113e-06)
Weight-4 disturbance = 0.0014 +/- 0.00449275  (deviation=0.3115434906305298)
Check 9 of 20
Weight-1 disturbance = 0.049 +/- 0.00151733  (deviation=0.6586340043958501)
Weight-2 disturbance = 2.25559e-08 +/- 0.000451989  (deviation=4.965998294268051e-05)
Weight-3 disturbance = -4.17616e-09 +/- 0.000209756  (deviation=1.9815109886722774e-05)
Weight-4 disturbance = 0.0172 +/- 0.00727269  (deviation=2.3646870966903855)
Check 10 of 20
Weight-1 disturbance = 0.051 +/- 0.00153494  (deviation=0.6510657351985843)
Weight-2 disturbance = -6.49282e-08 +/- 0.000445419  (deviation=0.0001455779927869709)
Weight-3 disturbance = 6.61187e-08 +/- 0.000244953  (deviation=0.00026882708498255655)
Weight-4 disturbance = 0.0084 +/- 0.00726491  (deviation=1.1560839016000193)
Check 11 of 20
Weight-1 disturbance = 0.0478 +/- 0.00366742  (deviation=0.5997134029708048)
Weight-2 disturbance = -2.04823e-07 +/- 0.000465137  (deviation=0.00043953378311990273)
Weight-3 disturbance = -3.01867e-07 +/- 0.00377937  (deviation=7.985115200956247e-05)
Weight-4 disturbance = 0.00150051 +/- 0.00444095  (deviation=0.33780404888116344)
Check 12 of 20
Weight-1 disturbance = 0.0505 +/- 0.00384426  (deviation=0.130029090095887)
Weight-2 disturbance = -6.47883e-08 +/- 0.000438218  (deviation=0.0001476457863412457)
Weight-3 disturbance = -1.98228e-08 +/- 0.00393113  (deviation=5.041241923229341e-06)
Weight-4 disturbance = 0.00120009 +/- 0.0044351  (deviation=0.2705277790730263)
Check 13 of 20
Weight-1 disturbance = 0.0470999 +/- 0.00527421  (deviation=0.5497516180655774)
Weight-2 disturbance = -5.6545e-08 +/- 0.000455973  (deviation=0.0001238704995554752)
Weight-3 disturbance = 0.0039995 +/- 0.00508321  (deviation=0.786650419911193)
Weight-4 disturbance = 6.12375e-07 +/- 0.002591  (deviation=0.0002362557262734456)
Check 14 of 20
Weight-1 disturbance = 0.0491 +/- 0.00151991  (deviation=0.5917537108130372)
Weight-2 disturbance = -6.51713e-09 +/- 0.000417749  (deviation=1.5707761375916745e-05)
Weight-3 disturbance = 8.06976e-09 +/- 0.000211464  (deviation=3.7981715807130665e-05)
Weight-4 disturbance = 0.0075 +/- 0.0072104  (deviation=1.040019507550239)
Check 15 of 20
Weight-1 disturbance = 0.0396 +/- 0.00593066  (deviation=1.753310579785569)
Weight-2 disturbance = 8.04692e-09 +/- 0.000439226  (deviation=1.814169758113655e-05)
Weight-3 disturbance = 0.00689996 +/- 0.00571055  (deviation=1.2080715585897506)
Weight-4 disturbance = 7.36639e-08 +/- 0.00151564  (deviation=4.857035400016532e-05)
Check 16 of 20
Weight-1 disturbance = 0.0515 +/- 0.00153019  (deviation=0.9796205525603602)
Weight-2 disturbance = -7.012e-07 +/- 0.000465682  (deviation=0.0015026512882456682)
Weight-3 disturbance = 7.17487e-07 +/- 0.000205005  (deviation=0.0034828684462739293)
Weight-4 disturbance = 0.0089 +/- 0.0072665  (deviation=1.224629892609855)
Check 17 of 20
Weight-1 disturbance = 0.0498 +/- 0.00145146  (deviation=0.1376978334792446)
Weight-2 disturbance = -1.90429e-08 +/- 0.00044312  (deviation=4.301395524442699e-05)
Weight-3 disturbance = -6.2634e-07 +/- 0.000232764  (deviation=0.0026793638831068725)
Weight-4 disturbance = 0.00670065 +/- 0.00713353  (deviation=0.9391849535197131)
Check 18 of 20
Weight-1 disturbance = 0.05 +/- 0.0013632  (deviation=3.762680271852993e-07)
Weight-2 disturbance = -1.85191e-07 +/- 0.000466908  (deviation=0.00039591463856024535)
Weight-3 disturbance = 3.14987e-08 +/- 0.000937694  (deviation=3.355586639363695e-05)
Weight-4 disturbance = 0.00410015 +/- 0.00685753  (deviation=0.5978183257153782)
Check 19 of 20
Weight-1 disturbance = 0.0491 +/- 0.00452  (deviation=0.199071287721814)
Weight-2 disturbance = -1.10186e-07 +/- 0.000466773  (deviation=0.00023568297034460035)
Weight-3 disturbance = 0.000200053 +/- 0.00458175  (deviation=0.04365349680628607)
Weight-4 disturbance = 5.75094e-08 +/- 0.00435963  (deviation=1.318831561575467e-05)
Check 20 of 20
Weight-1 disturbance = 0.0484 +/- 0.00261963  (deviation=0.6105394613122896)
Weight-2 disturbance = -1.3648e-08 +/- 0.000448692  (deviation=3.048427891202695e-05)
Weight-3 disturbance = 0.00129998 +/- 0.00273754  (deviation=0.47469993496888324)
Weight-4 disturbance = 2.93917e-08 +/- 0.00582643  (deviation=5.043684836920456e-06)
Percentanges within 68% confidence region:
Weight 1:  [0.13040749998957704, 3.034643466750717, 0.6912557196715029, 1.711438568858322, 0.22489948793603173, 0.3756442110481608, 1.4706299437122539, 1.7033118048074751, 0.6586340043958501, 0.6510657351985843, 0.5997134029708048, 0.130029090095887, 0.5497516180655774, 0.5917537108130372, 1.753310579785569, 0.9796205525603602, 0.1376978334792446, 3.762680271852993e-07, 0.199071287721814, 0.6105394613122896]
Weight 2:  [1.655883959978033e-05, 1.2291357451368336e-05, 4.469782196117712e-05, 1.698548931299419e-06, 0.00031993188460831636, 4.237937857391526e-06, 0.0007671528235064772, 1.6073118227795782e-05, 4.965998294268051e-05, 0.0001455779927869709, 0.00043953378311990273, 0.0001476457863412457, 0.0001238704995554752, 1.5707761375916745e-05, 1.814169758113655e-05, 0.0015026512882456682, 4.301395524442699e-05, 0.00039591463856024535, 0.00023568297034460035, 3.048427891202695e-05]
Weight 3:  [0.17262814538110377, 1.8929638981497028e-14, 0.7151551398557452, 1.5341329993162802, 3.1452644912367964e-05, 1.1359328717951638, 1.3145017691407308, 6.256354525456113e-06, 1.9815109886722774e-05, 0.00026882708498255655, 7.985115200956247e-05, 5.041241923229341e-06, 0.786650419911193, 3.7981715807130665e-05, 1.2080715585897506, 0.0034828684462739293, 0.0026793638831068725, 3.355586639363695e-05, 0.04365349680628607, 0.47469993496888324]
Weight 4:  [3.847421888800286e-06, 0.4122810123143864, 0.00015620170156731812, 0.0004369379544307312, 0.20351256664905235, 1.6153146291952415e-05, 7.209803865361063e-05, 0.3115434906305298, 2.3646870966903855, 1.1560839016000193, 0.33780404888116344, 0.2705277790730263, 0.0002362557262734456, 1.040019507550239, 4.857035400016532e-05, 1.224629892609855, 0.9391849535197131, 0.5978183257153782, 1.318831561575467e-05, 5.043684836920456e-06]

In [21]:

test_errorbars({'0000': 0.45, '0011': 0.05, '1111': 0.5}) # weight 2 movement 

Base weight-1 disturbance = 0
Base weight-2 disturbance = 0.05
Base weight-3 disturbance = 3.37029e-09
Base weight-4 disturbance = 0
Check 1 of 20
Weight-1 disturbance = 2.08167e-17 +/- 0.000320226  (deviation=6.48037959331721e-14)
Weight-2 disturbance = 0.0501997 +/- 0.00186102  (deviation=0.10723393001260678)
Weight-3 disturbance = 3.31702e-07 +/- 9.26582e-05  (deviation=0.0035056417084368204)
Weight-4 disturbance = 0 +/- 0.00130098  (deviation=0.0)
Check 2 of 20
Weight-1 disturbance = 0 +/- 0.00032646  (deviation=0.0)
Weight-2 disturbance = 0.0518 +/- 0.0021478  (deviation=0.8376683990201109)
Weight-3 disturbance = 2.12999e-08 +/- 0.000127726  (deviation=0.00013928557711311918)
Weight-4 disturbance = 0 +/- 0.0035868  (deviation=0.0)
Check 3 of 20
Weight-1 disturbance = 0 +/- 0.000292508  (deviation=0.0)
Weight-2 disturbance = 0.0490996 +/- 0.00207274  (deviation=0.43418023406088485)
Weight-3 disturbance = 3.81996e-07 +/- 0.000128251  (deviation=0.002929390154444396)
Weight-4 disturbance = 0 +/- 0.0034272  (deviation=0.0)
Check 4 of 20
Weight-1 disturbance = 0 +/- 0.000169689  (deviation=0.0)
Weight-2 disturbance = 0.0513991 +/- 0.00202709  (deviation=0.68986994068817)
Weight-3 disturbance = 8.83218e-07 +/- 0.000244732  (deviation=0.0035805186522389915)
Weight-4 disturbance = 0.0046 +/- 0.00757232  (deviation=0.6073955869545151)
Check 5 of 20
Weight-1 disturbance = 0 +/- 0.00032037  (deviation=0.0)
Weight-2 disturbance = 0.0485999 +/- 0.00186878  (deviation=0.748798227837998)
Weight-3 disturbance = 9.28049e-08 +/- 9.29751e-05  (deviation=0.0009516838963617885)
Weight-4 disturbance = 0 +/- 0.00234772  (deviation=0.0)
Check 6 of 20
Weight-1 disturbance = 1.38778e-17 +/- 0.000320292  (deviation=4.3193639925188406e-14)
Weight-2 disturbance = 0.0500997 +/- 0.00185449  (deviation=0.05374304319184878)
Weight-3 disturbance = 2.83644e-07 +/- 2.0211e-05  (deviation=0.01321364954710524)
Weight-4 disturbance = 0 +/- 0.000580414  (deviation=0.0)
Check 7 of 20
Weight-1 disturbance = 0 +/- 0.000154358  (deviation=0.0)
Weight-2 disturbance = 0.0508999 +/- 0.00197497  (deviation=0.4554288344616766)
Weight-3 disturbance = 8.8431e-08 +/- 0.000206231  (deviation=0.0004104628689849586)
Weight-4 disturbance = 0.0041 +/- 0.00755506  (deviation=0.5426113066627822)
Check 8 of 20
Weight-1 disturbance = 0 +/- 0.000189332  (deviation=0.0)
Weight-2 disturbance = 0.0526999 +/- 0.00216734  (deviation=1.2451642547686368)
Weight-3 disturbance = -3.21197e-08 +/- 0.00017192  (deviation=0.000205239637224678)
Weight-4 disturbance = 0.0072001 +/- 0.00621504  (deviation=1.1583097752696956)
Check 9 of 20
Weight-1 disturbance = 0 +/- 0.000154271  (deviation=0.0)
Weight-2 disturbance = 0.0468997 +/- 0.00187521  (deviation=1.6524227561140488)
Weight-3 disturbance = 2.96834e-08 +/- 0.000301676  (deviation=8.693521933442388e-05)
Weight-4 disturbance = 0.0148003 +/- 0.00773558  (deviation=1.9130239674864657)
Check 10 of 20
Weight-1 disturbance = 0 +/- 0.000287494  (deviation=0.0)
Weight-2 disturbance = 0.0502981 +/- 0.00203768  (deviation=0.1462356457145821)
Weight-3 disturbance = 1.87535e-06 +/- 0.00012714  (deviation=0.014608793365297608)
Weight-4 disturbance = 0 +/- 0.0050075  (deviation=0.0)
Check 11 of 20
Weight-1 disturbance = 6.93889e-18 +/- 0.000330877  (deviation=2.0908052497572605e-14)
Weight-2 disturbance = 0.0512999 +/- 0.00199652  (deviation=0.6507797438281614)
Weight-3 disturbance = 5.7407e-08 +/- 7.15794e-05  (deviation=0.0007445184141495582)
Weight-4 disturbance = 0 +/- 0.00410364  (deviation=0.0)
Check 12 of 20
Weight-1 disturbance = 6.93889e-18 +/- 0.000292524  (deviation=2.3639955192213346e-14)
Weight-2 disturbance = 0.0504 +/- 0.00204873  (deviation=0.1951397505805794)
Weight-3 disturbance = 1.86378e-08 +/- 0.000127964  (deviation=0.0001183861386504354)
Weight-4 disturbance = 0 +/- 0.00374556  (deviation=0.0)
Check 13 of 20
Weight-1 disturbance = 0 +/- 0.000380006  (deviation=0.0)
Weight-2 disturbance = 0.0514999 +/- 0.00203012  (deviation=0.7384725697895684)
Weight-3 disturbance = 7.5839e-08 +/- 1.44415e-05  (deviation=0.004693108147991298)
Weight-4 disturbance = 0 +/- 0.000647869  (deviation=0.0)
Check 14 of 20
Weight-1 disturbance = 0 +/- 0.000260267  (deviation=0.0)
Weight-2 disturbance = 0.0502 +/- 0.00206421  (deviation=0.09683471532737165)
Weight-3 disturbance = 3.84514e-09 +/- 0.000136579  (deviation=3.4515300156166684e-06)
Weight-4 disturbance = 0.00480002 +/- 0.00595947  (deviation=0.8053086482275309)
Check 15 of 20
Weight-1 disturbance = 4.85723e-17 +/- 0.000332456  (deviation=1.4566334150156198e-13)
Weight-2 disturbance = 0.0508991 +/- 0.00197841  (deviation=0.4542454811212361)
Weight-3 disturbance = 8.64566e-07 +/- 1.41023e-05  (deviation=0.057024011364594174)
Weight-4 disturbance = 0 +/- 0.000647268  (deviation=0.0)
Check 16 of 20
Weight-1 disturbance = 2.77556e-17 +/- 0.000358271  (deviation=7.725524425319541e-14)
Weight-2 disturbance = 0.0524 +/- 0.00201019  (deviation=1.1933056806059015)
Weight-3 disturbance = 3.64353e-08 +/- 1.45614e-05  (deviation=0.002124805283091678)
Weight-4 disturbance = 0 +/- 0.000647842  (deviation=0.0)
Check 17 of 20
Weight-1 disturbance = 0 +/- 0.000292737  (deviation=0.0)
Weight-2 disturbance = 0.0509999 +/- 0.00211602  (deviation=0.472329503596532)
Weight-3 disturbance = 7.29379e-08 +/- 0.000144074  (deviation=0.0004795323091813578)
Weight-4 disturbance = 0 +/- 0.00354282  (deviation=0.0)
Check 18 of 20
Weight-1 disturbance = 6.93889e-18 +/- 0.000292655  (deviation=2.362943609187102e-14)
Weight-2 disturbance = 0.0508994 +/- 0.00205872  (deviation=0.43666443355014845)
Weight-3 disturbance = 5.94748e-07 +/- 9.28559e-05  (deviation=0.006300903570748281)
Weight-4 disturbance = 0 +/- 0.0030315  (deviation=0.0)
Check 19 of 20
Weight-1 disturbance = 2.77556e-17 +/- 0.000320965  (deviation=8.620671012204619e-14)
Weight-2 disturbance = 0.0511999 +/- 0.00205324  (deviation=0.5841302285150249)
Weight-3 disturbance = 5.91564e-08 +/- 0.000127904  (deviation=0.00043277315884531007)
Weight-4 disturbance = 0 +/- 0.00490614  (deviation=0.0)
Check 20 of 20
Weight-1 disturbance = 0 +/- 0.000331929  (deviation=0.0)
Weight-2 disturbance = 0.043 +/- 0.00180785  (deviation=3.8698825657591676)
Weight-3 disturbance = 2.95117e-08 +/- 9.33766e-05  (deviation=0.000276990626300082)
Weight-4 disturbance = 0 +/- 0.00236553  (deviation=0.0)
Percentanges within 68% confidence region:
Weight 1:  [6.48037959331721e-14, 0.0, 0.0, 0.0, 0.0, 4.3193639925188406e-14, 0.0, 0.0, 0.0, 0.0, 2.0908052497572605e-14, 2.3639955192213346e-14, 0.0, 0.0, 1.4566334150156198e-13, 7.725524425319541e-14, 0.0, 2.362943609187102e-14, 8.620671012204619e-14, 0.0]
Weight 2:  [0.10723393001260678, 0.8376683990201109, 0.43418023406088485, 0.68986994068817, 0.748798227837998, 0.05374304319184878, 0.4554288344616766, 1.2451642547686368, 1.6524227561140488, 0.1462356457145821, 0.6507797438281614, 0.1951397505805794, 0.7384725697895684, 0.09683471532737165, 0.4542454811212361, 1.1933056806059015, 0.472329503596532, 0.43666443355014845, 0.5841302285150249, 3.8698825657591676]
Weight 3:  [0.0035056417084368204, 0.00013928557711311918, 0.002929390154444396, 0.0035805186522389915, 0.0009516838963617885, 0.01321364954710524, 0.0004104628689849586, 0.000205239637224678, 8.693521933442388e-05, 0.014608793365297608, 0.0007445184141495582, 0.0001183861386504354, 0.004693108147991298, 3.4515300156166684e-06, 0.057024011364594174, 0.002124805283091678, 0.0004795323091813578, 0.006300903570748281, 0.00043277315884531007, 0.000276990626300082]
Weight 4:  [0.0, 0.0, 0.0, 0.6073955869545151, 0.0, 0.0, 0.5426113066627822, 1.1583097752696956, 1.9130239674864657, 0.0, 0.0, 0.0, 0.0, 0.8053086482275309, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]

In [22]:

test_errorbars({'0000': 0.45, '0111': 0.05, '1111': 0.5}) # weight 3 movement 

Base weight-1 disturbance = 6.93889e-18
Base weight-2 disturbance = -6.93889e-18
Base weight-3 disturbance = 0.0499999
Base weight-4 disturbance = 6.40582e-08
Check 1 of 20
Weight-1 disturbance = 0.0107 +/- 0.00735355  (deviation=1.454881238330978)
Weight-2 disturbance = -2.63725e-09 +/- 0.000296416  (deviation=8.867232567089432e-06)
Weight-3 disturbance = 0.0395 +/- 0.00666818  (deviation=1.5744026062431984)
Weight-4 disturbance = 4.75707e-08 +/- 0.000755664  (deviation=2.178970913195027e-05)
Check 2 of 20
Weight-1 disturbance = 0.0037 +/- 0.00562772  (deviation=0.6573434556666093)
Weight-2 disturbance = -4.42276e-07 +/- 0.000307611  (deviation=0.001433119022133044)
Weight-3 disturbance = 0.0481001 +/- 0.00512665  (deviation=0.3705071575464912)
Weight-4 disturbance = 3.37135e-07 +/- 0.0035827  (deviation=7.619955664119643e-05)
Check 3 of 20
Weight-1 disturbance = 0.004 +/- 0.00563871  (deviation=0.7092557790160376)
Weight-2 disturbance = -3.25228e-08 +/- 0.000276748  (deviation=0.00011709444652051578)
Weight-3 disturbance = 0.0450998 +/- 0.00518115  (deviation=0.9455743967476032)
Weight-4 disturbance = 2.01269e-07 +/- 0.00338706  (deviation=4.049822637547276e-05)
Check 4 of 20
Weight-1 disturbance = 1.38778e-17 +/- 0.000890745  (deviation=7.781249194827357e-15)
Weight-2 disturbance = -1.38778e-17 +/- 0.000285596  (deviation=2.421143972593007e-14)
Weight-3 disturbance = 0.0513998 +/- 0.00302445  (deviation=0.4626835347068984)
Weight-4 disturbance = 0.00460024 +/- 0.00617985  (deviation=0.7442623519064119)
Check 5 of 20
Weight-1 disturbance = 0.0072 +/- 0.00648459  (deviation=1.1101533795093905)
Weight-2 disturbance = -2.27903e-07 +/- 0.000325502  (deviation=0.0006980145427165365)
Weight-3 disturbance = 0.0414001 +/- 0.00596273  (deviation=1.4420178335939444)
Weight-4 disturbance = 1.02746e-07 +/- 0.00186126  (deviation=2.0774927855684054e-05)
Check 6 of 20
Weight-1 disturbance = 0.0143 +/- 0.00747813  (deviation=1.9119880552459025)
Weight-2 disturbance = 1.15014e-10 +/- 0.000310339  (deviation=3.694165013705633e-07)
Weight-3 disturbance = 0.0357998 +/- 0.00673399  (deviation=2.108406861423403)
Weight-4 disturbance = 1.54797e-07 +/- 0.000741027  (deviation=0.00012228447492531905)
Check 7 of 20
Weight-1 disturbance = 0 +/- 0.00104302  (deviation=6.64629879376936e-15)
Weight-2 disturbance = 0 +/- 0.000268972  (deviation=2.5702256042809192e-14)
Weight-3 disturbance = 0.0508996 +/- 0.00304178  (deviation=0.2956564833181069)
Weight-4 disturbance = 0.00410045 +/- 0.00614243  (deviation=0.6674419554768625)
Check 8 of 20
Weight-1 disturbance = 0 +/- 0.00206695  (deviation=3.355447781054858e-15)
Weight-2 disturbance = 0 +/- 0.000280079  (deviation=2.4686665449615382e-14)
Weight-3 disturbance = 0.0527 +/- 0.00322692  (deviation=0.8364703003195016)
Weight-4 disturbance = 0.0072 +/- 0.00651895  (deviation=1.1042931567171468)
Check 9 of 20
Weight-1 disturbance = 1.38778e-17 +/- 0.00027688  (deviation=2.497085786409489e-14)
Weight-2 disturbance = -1.38778e-17 +/- 0.000282444  (deviation=2.4480618221926022e-14)
Weight-3 disturbance = 0.0469 +/- 0.00282885  (deviation=1.0954430024073278)
Weight-4 disturbance = 0.0148 +/- 0.00646437  (deviation=2.28910940161106)
Check 10 of 20
Weight-1 disturbance = 0.0006 +/- 0.00524845  (deviation=0.11429769894048605)
Weight-2 disturbance = 6.13371e-11 +/- 0.000295207  (deviation=2.0707503531370385e-07)
Weight-3 disturbance = 0.0496996 +/- 0.00486583  (deviation=0.06171302817649418)
Weight-4 disturbance = 4.10634e-07 +/- 0.00518835  (deviation=6.678595495834595e-05)
Check 11 of 20
Weight-1 disturbance = 0.0052 +/- 0.00445506  (deviation=1.1669508086394962)
Weight-2 disturbance = -2.31019e-08 +/- 0.000304199  (deviation=7.569464752946071e-05)
Weight-3 disturbance = 0.0461 +/- 0.00603516  (deviation=0.6460963895518306)
Weight-4 disturbance = 3.11564e-08 +/- 0.00253555  (deviation=1.297112487149386e-05)
Check 12 of 20
Weight-1 disturbance = 0.0033 +/- 0.00549937  (deviation=0.5999595201923575)
Weight-2 disturbance = -4.71161e-08 +/- 0.000295396  (deviation=0.00015896340588380405)
Weight-3 disturbance = 0.0470995 +/- 0.00507098  (deviation=0.571855715987774)
Weight-4 disturbance = 5.54578e-07 +/- 0.00376842  (deviation=0.00013013134362142417)
Check 13 of 20
Weight-1 disturbance = 0.0158 +/- 0.00753053  (deviation=2.0978467834104877)
Weight-2 disturbance = -8.05238e-08 +/- 0.000257282  (deviation=0.0003117671136982293)
Weight-3 disturbance = 0.0356999 +/- 0.00674715  (deviation=2.1191091295571263)
Weight-4 disturbance = 2.08135e-07 +/- 0.000740461  (deviation=0.0001943145857456431)
Check 14 of 20
Weight-1 disturbance = 0 +/- 0.00282355  (deviation=2.4566406668017653e-15)
Weight-2 disturbance = 0 +/- 0.000261132  (deviation=2.647100980784843e-14)
Weight-3 disturbance = 0.0502 +/- 0.00342669  (deviation=0.05835373857791067)
Weight-4 disturbance = 0.00480005 +/- 0.00609452  (deviation=0.7874609276683512)
Check 15 of 20
Weight-1 disturbance = 0.0145 +/- 0.00742954  (deviation=1.9514067045663785)
Weight-2 disturbance = -3.29946e-10 +/- 0.000310158  (deviation=1.0603801471510928e-06)
Weight-3 disturbance = 0.0363998 +/- 0.00693423  (deviation=1.9610144070025162)
Weight-4 disturbance = 1.50646e-07 +/- 0.000740431  (deviation=0.00011678510871164528)
Check 16 of 20
Weight-1 disturbance = 0.0146 +/- 0.00753976  (deviation=1.9361448038487765)
Weight-2 disturbance = 1.38778e-17 +/- 0.000312821  (deviation=6.633303683597947e-14)
Weight-3 disturbance = 0.0377999 +/- 0.0067226  (deviation=1.8145120169296816)
Weight-4 disturbance = 1.21068e-07 +/- 0.000740531  (deviation=7.688112907243159e-05)
Check 17 of 20
Weight-1 disturbance = 0.0038 +/- 0.00562518  (deviation=0.6754143141013569)
Weight-2 disturbance = 5.41529e-10 +/- 0.000298333  (deviation=1.8091205867831591e-06)
Weight-3 disturbance = 0.0471995 +/- 0.00513949  (deviation=0.5447710692745095)
Weight-4 disturbance = 4.54534e-07 +/- 0.00353124  (deviation=0.00011054618505305945)
Check 18 of 20
Weight-1 disturbance = 0.00509818 +/- 0.00589071  (deviation=0.8653133465883416)
Weight-2 disturbance = 1.58135e-06 +/- 0.000304676  (deviation=0.0051732862618611266)
Weight-3 disturbance = 0.0457997 +/- 0.00540828  (deviation=0.776488423711704)
Weight-4 disturbance = 5.50161e-07 +/- 0.00287436  (deviation=0.00016905825958112104)
Check 19 of 20
Weight-1 disturbance = 0.0008 +/- 0.00527499  (deviation=0.1516304117892074)
Weight-2 disturbance = -1.53444e-10 +/- 0.000289416  (deviation=5.283595940316372e-07)
Weight-3 disturbance = 0.0503993 +/- 0.00487028  (deviation=0.0819911338575766)
Weight-4 disturbance = 6.62272e-07 +/- 0.00510228  (deviation=0.00011722133137957083)
Check 20 of 20
Weight-1 disturbance = 0.007 +/- 0.00628485  (deviation=1.1136128520475674)
Weight-2 disturbance = -1.23083e-10 +/- 0.00027245  (deviation=4.501106154744604e-07)
Weight-3 disturbance = 0.0359999 +/- 0.00619172  (deviation=2.260730393447211)
Weight-4 disturbance = 1.2943e-07 +/- 0.00191162  (deviation=3.417898792828288e-05)
Percentanges within 68% confidence region:
Weight 1:  [1.454881238330978, 0.6573434556666093, 0.7092557790160376, 7.781249194827357e-15, 1.1101533795093905, 1.9119880552459025, 6.64629879376936e-15, 3.355447781054858e-15, 2.497085786409489e-14, 0.11429769894048605, 1.1669508086394962, 0.5999595201923575, 2.0978467834104877, 2.4566406668017653e-15, 1.9514067045663785, 1.9361448038487765, 0.6754143141013569, 0.8653133465883416, 0.1516304117892074, 1.1136128520475674]
Weight 2:  [8.867232567089432e-06, 0.001433119022133044, 0.00011709444652051578, 2.421143972593007e-14, 0.0006980145427165365, 3.694165013705633e-07, 2.5702256042809192e-14, 2.4686665449615382e-14, 2.4480618221926022e-14, 2.0707503531370385e-07, 7.569464752946071e-05, 0.00015896340588380405, 0.0003117671136982293, 2.647100980784843e-14, 1.0603801471510928e-06, 6.633303683597947e-14, 1.8091205867831591e-06, 0.0051732862618611266, 5.283595940316372e-07, 4.501106154744604e-07]
Weight 3:  [1.5744026062431984, 0.3705071575464912, 0.9455743967476032, 0.4626835347068984, 1.4420178335939444, 2.108406861423403, 0.2956564833181069, 0.8364703003195016, 1.0954430024073278, 0.06171302817649418, 0.6460963895518306, 0.571855715987774, 2.1191091295571263, 0.05835373857791067, 1.9610144070025162, 1.8145120169296816, 0.5447710692745095, 0.776488423711704, 0.0819911338575766, 2.260730393447211]
Weight 4:  [2.178970913195027e-05, 7.619955664119643e-05, 4.049822637547276e-05, 0.7442623519064119, 2.0774927855684054e-05, 0.00012228447492531905, 0.6674419554768625, 1.1042931567171468, 2.28910940161106, 6.678595495834595e-05, 1.297112487149386e-05, 0.00013013134362142417, 0.0001943145857456431, 0.7874609276683512, 0.00011678510871164528, 7.688112907243159e-05, 0.00011054618505305945, 0.00016905825958112104, 0.00011722133137957083, 3.417898792828288e-05]

In [23]:

test_errorbars({'0000': 0.45, '1111': 0.55}) # weight 4 movement 

Base weight-1 disturbance = 0
Base weight-2 disturbance = 0
Base weight-3 disturbance = 0
Base weight-4 disturbance = 0.05
Check 1 of 20
Weight-1 disturbance = 0 +/- 0.000223133  (deviation=0.0)
Weight-2 disturbance = 0 +/- 0.00027576  (deviation=0.0)
Weight-3 disturbance = 0 +/- 0.000312568  (deviation=0.0)
Weight-4 disturbance = 0.0395 +/- 0.00661217  (deviation=1.5877420124959105)
Check 2 of 20
Weight-1 disturbance = 0 +/- 0.000223139  (deviation=0.0)
Weight-2 disturbance = 0 +/- 0.000275264  (deviation=0.0)
Weight-3 disturbance = 0 +/- 0.000315445  (deviation=0.0)
Weight-4 disturbance = 0.0481 +/- 0.00661172  (deviation=0.28732501691130974)
Check 3 of 20
Weight-1 disturbance = 0 +/- 0.000222312  (deviation=0.0)
Weight-2 disturbance = 0 +/- 0.000275191  (deviation=0.0)
Weight-3 disturbance = 0 +/- 0.000315324  (deviation=0.0)
Weight-4 disturbance = 0.0451 +/- 0.00661149  (deviation=0.7410214778902107)
Check 4 of 20
Weight-1 disturbance = 0 +/- 0.000223168  (deviation=0.0)
Weight-2 disturbance = 0 +/- 0.000273215  (deviation=0.0)
Weight-3 disturbance = 0 +/- 0.000338002  (deviation=0.0)
Weight-4 disturbance = 0.056 +/- 0.00773358  (deviation=0.7757368190117534)
Check 5 of 20
Weight-1 disturbance = 0 +/- 0.000223199  (deviation=0.0)
Weight-2 disturbance = 0 +/- 0.000246704  (deviation=0.0)
Weight-3 disturbance = 0 +/- 0.000326236  (deviation=0.0)
Weight-4 disturbance = 0.0414 +/- 0.00661187  (deviation=1.3004947893168723)
Check 6 of 20
Weight-1 disturbance = 0 +/- 0.000223244  (deviation=0.0)
Weight-2 disturbance = 0 +/- 0.000275624  (deviation=0.0)
Weight-3 disturbance = 0 +/- 0.000312728  (deviation=0.0)
Weight-4 disturbance = 0.0358 +/- 0.0066118  (deviation=2.1473511479132927)
Check 7 of 20
Weight-1 disturbance = 0 +/- 0.000222007  (deviation=0.0)
Weight-2 disturbance = 0 +/- 0.000274695  (deviation=0.0)
Weight-3 disturbance = 0 +/- 0.00033473  (deviation=0.0)
Weight-4 disturbance = 0.055 +/- 0.00773436  (deviation=0.6463825878742214)
Check 8 of 20
Weight-1 disturbance = 0 +/- 0.000222213  (deviation=0.0)
Weight-2 disturbance = 2.77556e-17 +/- 0.000275795  (deviation=1.0027476856163692e-13)
Weight-3 disturbance = -2.77556e-17 +/- 0.000311755  (deviation=8.874555980810666e-14)
Weight-4 disturbance = 0.0599 +/- 0.00661126  (deviation=1.4972197269124894)
Check 9 of 20
Weight-1 disturbance = 0 +/- 0.000222464  (deviation=0.0)
Weight-2 disturbance = 0 +/- 0.000281361  (deviation=0.0)
Weight-3 disturbance = 0 +/- 0.000315589  (deviation=0.0)
Weight-4 disturbance = 0.0617 +/- 0.0077288  (deviation=1.513623578666049)
Check 10 of 20
Weight-1 disturbance = 0 +/- 0.00021946  (deviation=0.0)
Weight-2 disturbance = 0 +/- 0.000276702  (deviation=0.0)
Weight-3 disturbance = 0 +/- 0.000312045  (deviation=0.0)
Weight-4 disturbance = 0.0497 +/- 0.00661173  (deviation=0.045367032650432154)
Check 11 of 20
Weight-1 disturbance = 0 +/- 0.000223199  (deviation=0.0)
Weight-2 disturbance = 1.38778e-17 +/- 0.000265554  (deviation=5.206370696523145e-14)
Weight-3 disturbance = -1.38778e-17 +/- 0.000319565  (deviation=4.3291641409384156e-14)
Weight-4 disturbance = 0.0461 +/- 0.00774207  (deviation=0.5036761252635799)
Check 12 of 20
Weight-1 disturbance = 0 +/- 0.000222423  (deviation=0.0)
Weight-2 disturbance = 6.93889e-18 +/- 0.000276431  (deviation=2.5011276473195888e-14)
Weight-3 disturbance = -6.93889e-18 +/- 0.000312102  (deviation=2.2161793896502998e-14)
Weight-4 disturbance = 0.0471 +/- 0.00661158  (deviation=0.43855776440960176)
Check 13 of 20
Weight-1 disturbance = 0 +/- 0.000222348  (deviation=0.0)
Weight-2 disturbance = 0 +/- 0.000270045  (deviation=0.0)
Weight-3 disturbance = 0 +/- 0.000336146  (deviation=0.0)
Weight-4 disturbance = 0.0357 +/- 0.00661204  (deviation=2.1623953001266445)
Check 14 of 20
Weight-1 disturbance = 0 +/- 0.000223202  (deviation=0.0)
Weight-2 disturbance = 0 +/- 0.00027528  (deviation=0.0)
Weight-3 disturbance = 0 +/- 0.000315537  (deviation=0.0)
Weight-4 disturbance = 0.055 +/- 0.00661129  (deviation=0.7561681634414716)
Check 15 of 20
Weight-1 disturbance = 0 +/- 0.000222948  (deviation=0.0)
Weight-2 disturbance = 0 +/- 0.000301141  (deviation=0.0)
Weight-3 disturbance = 0 +/- 0.000301014  (deviation=0.0)
Weight-4 disturbance = 0.0364 +/- 0.00680892  (deviation=1.9970852638291023)
Check 16 of 20
Weight-1 disturbance = 0 +/- 0.000223246  (deviation=0.0)
Weight-2 disturbance = 0 +/- 0.000273596  (deviation=0.0)
Weight-3 disturbance = 0 +/- 0.00033148  (deviation=0.0)
Weight-4 disturbance = 0.0378 +/- 0.00661186  (deviation=1.8448900254282627)
Check 17 of 20
Weight-1 disturbance = 0 +/- 0.000223223  (deviation=0.0)
Weight-2 disturbance = 0 +/- 0.000275921  (deviation=0.0)
Weight-3 disturbance = 0 +/- 0.000312081  (deviation=0.0)
Weight-4 disturbance = 0.0472 +/- 0.00661175  (deviation=0.4234243292833587)
Check 18 of 20
Weight-1 disturbance = 0 +/- 0.000223079  (deviation=0.0)
Weight-2 disturbance = 2.77556e-17 +/- 0.000275356  (deviation=1.0043421246816683e-13)
Weight-3 disturbance = -2.77556e-17 +/- 0.000315514  (deviation=8.769159276775844e-14)
Weight-4 disturbance = 0.0458 +/- 0.0066118  (deviation=0.6351320581700877)
Check 19 of 20
Weight-1 disturbance = 0 +/- 0.000222484  (deviation=0.0)
Weight-2 disturbance = 0 +/- 0.00027661  (deviation=0.0)
Weight-3 disturbance = 0 +/- 0.000315487  (deviation=0.0)
Weight-4 disturbance = 0.0504 +/- 0.00661144  (deviation=0.06049207861490999)
Check 20 of 20
Weight-1 disturbance = 0 +/- 0.000223209  (deviation=0.0)
Weight-2 disturbance = 0 +/- 0.000301627  (deviation=0.0)
Weight-3 disturbance = 0 +/- 0.000298022  (deviation=0.0)
Weight-4 disturbance = 0.036 +/- 0.00680901  (deviation=2.0557975908568977)
Percentanges within 68% confidence region:
Weight 1:  [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
Weight 2:  [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0027476856163692e-13, 0.0, 0.0, 5.206370696523145e-14, 2.5011276473195888e-14, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0043421246816683e-13, 0.0, 0.0]
Weight 3:  [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 8.874555980810666e-14, 0.0, 0.0, 4.3291641409384156e-14, 2.2161793896502998e-14, 0.0, 0.0, 0.0, 0.0, 0.0, 8.769159276775844e-14, 0.0, 0.0]
Weight 4:  [1.5877420124959105, 0.28732501691130974, 0.7410214778902107, 0.7757368190117534, 1.3004947893168723, 2.1473511479132927, 0.6463825878742214, 1.4972197269124894, 1.513623578666049, 0.045367032650432154, 0.5036761252635799, 0.43855776440960176, 2.1623953001266445, 0.7561681634414716, 1.9970852638291023, 1.8448900254282627, 0.4234243292833587, 0.6351320581700877, 0.06049207861490999, 2.0557975908568977]

In [ ]: