In [1]:
import cabinetry
We start by configuring the output from cabinetry. It uses the logging module to send messages at different verbosity levels. This customization is optional, and you can also use the logging module directly for further customization. The set_logging function just sets up a verbose default.
In [2]:
cabinetry.set_logging()
The configuration file¶
The configuration file is the central place to configure cabinetry.
Let's have a look at the example configuration file used in this notebook.
In [3]:
cabinetry_config = cabinetry.configuration.load("config_ntuples.yml")
cabinetry.configuration.print_overview(cabinetry_config)
INFO - cabinetry.configuration - opening config file config_ntuples.yml INFO - cabinetry.configuration - the config contains: INFO - cabinetry.configuration - 3 Sample(s) INFO - cabinetry.configuration - 1 Regions(s) INFO - cabinetry.configuration - 1 NormFactor(s) INFO - cabinetry.configuration - 3 Systematic(s)
The configuration file is split into four different blocks of settings. There are general settings:
In [4]:
cabinetry_config["General"]
Out[4]:
{'Measurement': 'minimal_example',
'POI': 'Signal_norm',
'HistogramFolder': 'histograms/',
'InputPath': 'inputs/{SamplePath}'}
The list of phase space regions (channels), in this case we are considering just a single one:
In [5]:
cabinetry_config["Regions"]
Out[5]:
[{'Name': 'Signal_region',
'Variable': 'jet_pt',
'Filter': 'lep_charge > 0',
'Binning': [200, 300, 400, 500, 600]}]
A list of samples, including data:
In [6]:
cabinetry_config["Samples"]
Out[6]:
[{'Name': 'Data',
'Tree': 'pseudodata',
'SamplePath': 'data.root',
'Data': True},
{'Name': 'Signal',
'Tree': 'signal',
'SamplePath': 'prediction.root',
'Weight': 'weight',
'DisableStaterror': True},
{'Name': 'Background',
'Tree': 'background',
'SamplePath': 'prediction.root',
'Weight': 'weight'}]
A list of normalization factors:
In [7]:
cabinetry_config["NormFactors"]
Out[7]:
[{'Name': 'Signal_norm', 'Samples': 'Signal', 'Nominal': 1, 'Bounds': [0, 10]}]
And finally a list of systematic uncertainties. In this case there are three systematic uncertainties:
In [8]:
cabinetry_config["Systematics"]
Out[8]:
[{'Name': 'Luminosity',
'Up': {'Normalization': 0.05},
'Down': {'Normalization': -0.05},
'Type': 'Normalization'},
{'Name': 'Modeling',
'Up': {'SamplePath': 'prediction.root', 'Tree': 'background_varied'},
'Down': {'Symmetrize': True},
'Samples': 'Background',
'Type': 'NormPlusShape'},
{'Name': 'WeightBasedModeling',
'Up': {'Weight': 'weight_up'},
'Down': {'Weight': '0.7*weight'},
'Samples': 'Background',
'Type': 'NormPlusShape'}]
Regions, samples, normalization factors and systematics all can be identified by their names.
Creating template histograms from ntuples¶
We use the templates module to create all histograms needed to build the workspace defined in the configuration file.
In [9]:
cabinetry.templates.build(cabinetry_config, method="uproot")
DEBUG - cabinetry.route - in region Signal_region DEBUG - cabinetry.route - reading sample Data DEBUG - cabinetry.route - variation Nominal DEBUG - cabinetry.histo - saving histogram to histograms/Signal_region_Data.npz DEBUG - cabinetry.route - reading sample Signal DEBUG - cabinetry.route - variation Nominal WARNING - cabinetry.histo - Signal_region_Signal has empty bins: [0] DEBUG - cabinetry.histo - saving histogram to histograms/Signal_region_Signal.npz DEBUG - cabinetry.route - reading sample Background DEBUG - cabinetry.route - variation Nominal DEBUG - cabinetry.histo - saving histogram to histograms/Signal_region_Background.npz DEBUG - cabinetry.route - variation Modeling Up DEBUG - cabinetry.histo - saving histogram to histograms/Signal_region_Background_Modeling_Up.npz DEBUG - cabinetry.route - variation WeightBasedModeling Up DEBUG - cabinetry.histo - saving histogram to histograms/Signal_region_Background_WeightBasedModeling_Up.npz DEBUG - cabinetry.route - variation WeightBasedModeling Down DEBUG - cabinetry.histo - saving histogram to histograms/Signal_region_Background_WeightBasedModeling_Down.npz
The histograms are saved to the folder specified under HistogramFolder in the General settings in the configuration file.
In this case, this folder is histograms/:
In [10]:
!ls histograms/
Signal_region_Background.npz Signal_region_Background_Modeling_Up.npz Signal_region_Background_WeightBasedModeling_Down.npz Signal_region_Background_WeightBasedModeling_Up.npz Signal_region_Data.npz Signal_region_Signal.npz
It can be useful to apply additional post-processing after building template histograms.
Such processing can for example replace ill-defined statistical uncertainties in empty bins by zero.
It is also performed via the templates module:
In [11]:
cabinetry.templates.postprocess(cabinetry_config)
DEBUG - cabinetry.route - in region Signal_region DEBUG - cabinetry.route - reading sample Data DEBUG - cabinetry.route - variation Nominal DEBUG - cabinetry.histo - saving histogram to histograms/Signal_region_Data_modified.npz DEBUG - cabinetry.route - reading sample Signal DEBUG - cabinetry.route - variation Nominal WARNING - cabinetry.histo - Signal_region_Signal has empty bins: [0] DEBUG - cabinetry.histo - saving histogram to histograms/Signal_region_Signal_modified.npz DEBUG - cabinetry.route - reading sample Background DEBUG - cabinetry.route - variation Nominal DEBUG - cabinetry.histo - saving histogram to histograms/Signal_region_Background_modified.npz DEBUG - cabinetry.route - variation Modeling Up DEBUG - cabinetry.histo - saving histogram to histograms/Signal_region_Background_Modeling_Up_modified.npz DEBUG - cabinetry.route - variation WeightBasedModeling Up DEBUG - cabinetry.histo - saving histogram to histograms/Signal_region_Background_WeightBasedModeling_Up_modified.npz DEBUG - cabinetry.route - variation WeightBasedModeling Down DEBUG - cabinetry.histo - saving histogram to histograms/Signal_region_Background_WeightBasedModeling_Down_modified.npz
New histograms have now appeard in the histograms/ folder.
These "modified" histograms include the changes applied by the postprocessor.
In [12]:
!ls histograms/
Signal_region_Background.npz Signal_region_Background_Modeling_Up.npz Signal_region_Background_Modeling_Up_modified.npz Signal_region_Background_WeightBasedModeling_Down.npz Signal_region_Background_WeightBasedModeling_Down_modified.npz Signal_region_Background_WeightBasedModeling_Up.npz Signal_region_Background_WeightBasedModeling_Up_modified.npz Signal_region_Background_modified.npz Signal_region_Data.npz Signal_region_Data_modified.npz Signal_region_Signal.npz Signal_region_Signal_modified.npz
Optional: reading existing template histograms¶
Besides providing ntuples that first need to be turned into histograms, it is also possible to provide existing histograms to cabinetry.
The configuration options for this are slightly different, since less information is needed to read an existing histogram.
The following loads a cabinetry configuration using histogram inputs, collects all provided histograms (storing them in the format used internally by cabinetry for further processing) and applies post-processing.
The resulting histograms are equivalent to those created when reading the provided ntuples.
In [13]:
cabinetry_config_histograms = cabinetry.configuration.load("config_histograms.yml")
cabinetry.templates.collect(cabinetry_config_histograms, method="uproot")
cabinetry.templates.postprocess(cabinetry_config)
INFO - cabinetry.configuration - opening config file config_histograms.yml DEBUG - cabinetry.route - in region Signal_region DEBUG - cabinetry.route - reading sample Data DEBUG - cabinetry.route - variation Nominal DEBUG - cabinetry.histo - saving histogram to histograms/Signal_region_Data.npz DEBUG - cabinetry.route - reading sample Signal DEBUG - cabinetry.route - variation Nominal WARNING - cabinetry.histo - Signal_region_Signal has empty bins: [0] DEBUG - cabinetry.histo - saving histogram to histograms/Signal_region_Signal.npz DEBUG - cabinetry.route - reading sample Background DEBUG - cabinetry.route - variation Nominal DEBUG - cabinetry.histo - saving histogram to histograms/Signal_region_Background.npz DEBUG - cabinetry.route - variation Modeling Up DEBUG - cabinetry.histo - saving histogram to histograms/Signal_region_Background_Modeling_Up.npz DEBUG - cabinetry.route - variation WeightBasedModeling Up DEBUG - cabinetry.histo - saving histogram to histograms/Signal_region_Background_WeightBasedModeling_Up.npz DEBUG - cabinetry.route - variation WeightBasedModeling Down DEBUG - cabinetry.histo - saving histogram to histograms/Signal_region_Background_WeightBasedModeling_Down.npz DEBUG - cabinetry.route - in region Signal_region DEBUG - cabinetry.route - reading sample Data DEBUG - cabinetry.route - variation Nominal DEBUG - cabinetry.histo - saving histogram to histograms/Signal_region_Data_modified.npz DEBUG - cabinetry.route - reading sample Signal DEBUG - cabinetry.route - variation Nominal WARNING - cabinetry.histo - Signal_region_Signal has empty bins: [0] DEBUG - cabinetry.histo - saving histogram to histograms/Signal_region_Signal_modified.npz DEBUG - cabinetry.route - reading sample Background DEBUG - cabinetry.route - variation Nominal DEBUG - cabinetry.histo - saving histogram to histograms/Signal_region_Background_modified.npz DEBUG - cabinetry.route - variation Modeling Up DEBUG - cabinetry.histo - saving histogram to histograms/Signal_region_Background_Modeling_Up_modified.npz DEBUG - cabinetry.route - variation WeightBasedModeling Up DEBUG - cabinetry.histo - saving histogram to histograms/Signal_region_Background_WeightBasedModeling_Up_modified.npz DEBUG - cabinetry.route - variation WeightBasedModeling Down DEBUG - cabinetry.histo - saving histogram to histograms/Signal_region_Background_WeightBasedModeling_Down_modified.npz
Workspace building¶
Next, we build a pyhf workspace and serialize it to a file.
The workspace module takes care of this task.
In [14]:
workspace_path = "workspaces/example_workspace.json"
ws = cabinetry.workspace.build(cabinetry_config)
cabinetry.workspace.save(ws, workspace_path)
INFO - cabinetry.workspace - building workspace DEBUG - cabinetry.workspace - adding NormFactor Signal_norm to sample Signal in region Signal_region DEBUG - cabinetry.workspace - adding OverallSys Luminosity to sample Signal in region Signal_region DEBUG - cabinetry.workspace - adding OverallSys Luminosity to sample Background in region Signal_region DEBUG - cabinetry.workspace - adding OverallSys and HistoSys Modeling to sample Background in region Signal_region DEBUG - cabinetry.workspace - normalization impact of systematic Modeling on sample Background in region Signal_region is 0.800 DEBUG - cabinetry.workspace - adding OverallSys and HistoSys WeightBasedModeling to sample Background in region Signal_region INFO - pyhf.workspace - Validating spec against schema: workspace.json DEBUG - cabinetry.workspace - saving workspace to workspaces/example_workspace.json
Model structure¶
It can be helpful to visualize the modifier structure of the statistical model we have built to catch potential issues.
The visualize.modifier_grid function creates a figure showcasing the information about which modifiers (indicated by color) act on which region and sample when a given parameter (on the horizontal axis) is varied.
To split this visualization from one table per region to one table per sample, use split_by_sample=True.
We need the fit model (containing the probability density function) for the visualization, which we get from the workspace object. We will also extract data from it (observed bin yields and including auxiliary data for auxiliary measurements, see the HistFactory documentation), which we will use in the next step.
In [15]:
ws = cabinetry.workspace.load(workspace_path)
model, data = cabinetry.model_utils.model_and_data(ws)
cabinetry.visualize.modifier_grid(model)
INFO - pyhf.workspace - Validating spec against schema: workspace.json INFO - pyhf.pdf - Validating spec against schema: model.json INFO - pyhf.pdf - adding modifier Modeling (1 new nuisance parameters) INFO - pyhf.pdf - adding modifier WeightBasedModeling (1 new nuisance parameters) INFO - pyhf.pdf - adding modifier Signal_norm (1 new nuisance parameters) INFO - pyhf.pdf - adding modifier Luminosity (1 new nuisance parameters) INFO - pyhf.pdf - adding modifier staterror_Signal_region (4 new nuisance parameters) INFO - cabinetry.visualize.utils - saving figure as figures/modifier_grid.pdf
Out[15]:
Fitting¶
With the workspace built, we can perform a maximum likelihood fit.
The results for the fitted parameters are reported.
The cabinetry.model_utils.model_and_data function has an asimov keyword argument, which we can set to True to instead study the expected performance with an Asimov dataset.
In [16]:
fit_results = cabinetry.fit.fit(model, data)
INFO - cabinetry.fit - performing maximum likelihood fit INFO - cabinetry.fit - Migrad status: ┌─────────────────────────────────────────────────────────────────────────┐ │ Migrad │ ├──────────────────────────────────┬──────────────────────────────────────┤ │ FCN = 18.54 │ Nfcn = 330 │ │ EDM = 1.04e-06 (Goal: 0.0002) │ │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Valid Minimum │ No Parameters at limit │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Below EDM threshold (goal x 10) │ Below call limit │ ├───────────────┬──────────────────┼───────────┬─────────────┬────────────┤ │ Covariance │ Hesse ok │ Accurate │ Pos. def. │ Not forced │ └───────────────┴──────────────────┴───────────┴─────────────┴────────────┘ DEBUG - cabinetry.fit - -2 log(L) = 18.535252 at best-fit point INFO - cabinetry.fit - fit results (with symmetric uncertainties): INFO - cabinetry.fit - Modeling = -0.2753 +/- 0.5683 INFO - cabinetry.fit - WeightBasedModeling = -0.5277 +/- 0.6477 INFO - cabinetry.fit - Signal_norm = 1.5938 +/- 0.9757 INFO - cabinetry.fit - Luminosity = -0.0794 +/- 0.9910 INFO - cabinetry.fit - staterror_Signal_region[0] = 1.0012 +/- 0.0411 INFO - cabinetry.fit - staterror_Signal_region[1] = 0.9883 +/- 0.0384 INFO - cabinetry.fit - staterror_Signal_region[2] = 1.0216 +/- 0.0469 INFO - cabinetry.fit - staterror_Signal_region[3] = 0.9829 +/- 0.0610
We can also visualize the fit results. Below are the pulls:
In [17]:
cabinetry.visualize.pulls(fit_results, exclude=["Signal_norm"])
INFO - cabinetry.visualize.utils - saving figure as figures/pulls.pdf
Out[17]:
We excluded the "Signal_norm" parameter, which does not have an associated constraint term in our fit model. The result for it was reported above in the fit output:
INFO - cabinetry.fit - Signal_norm = 1.5938 +/- 0.9757
We can also look at the correlation between parameters:
In [18]:
cabinetry.visualize.correlation_matrix(fit_results)
INFO - cabinetry.visualize.utils - saving figure as figures/correlation_matrix.pdf
Out[18]:
These visualizations were also saved as .pdf figures in the figures/ folder.
Visualizing templates¶
What did we fit?
The visualize module also contains functionality to plot data/MC distributions: visualize.data_mc.
We first need to create a model prediction, which is achieved with model_utils.prediction.
By default this creates the pre-fit model, but the optional fit_results argument allows to create the model corresponding to a given best-fit configuration.
The config keyword argument of visualize.data_mc is optional, but required for correct horizontal axis labels, since the observable and bin edges are not part of the pyhf workspace.
Since this argument is optional, you can use cabinetry.visualize.data_mc with any workspace: it does not matter whether it was created with cabinetry or otherwise, since you do not need a configuration file.
visualize.data_mc returns a list of dictionaries, we can extract a figure from there to further customize it.
In [19]:
model_pred = cabinetry.model_utils.prediction(model)
figures = cabinetry.visualize.data_mc(model_pred, data, config=cabinetry_config)
DEBUG - cabinetry.model_utils - total stdev is [[69, 58.3, 38.2, 45.3]] DEBUG - cabinetry.model_utils - total stdev per channel is [137] INFO - cabinetry.visualize.utils - saving figure as figures/Signal_region_prefit.pdf
This figure is also again saved in the figures/ folder, like all figures in general.
To demonstrate figure customization, let's use $\LaTeX$ for the horizontal axis label. We can save the modified figure as well by using .savefig().
In [20]:
ratio_panel = figures[0]["figure"].get_axes()[1]
ratio_panel.set_xlabel("jet $p_T$")
figures[0]["figure"] # show figure again
Out[20]:
Yield tables can also be created from a model prediction, and compared to data.
Optional keyword arguments control whether yields per bin are shown (per_bin=True, default) and whether bins summed per region are shown (per_channel=True, disabled by default).
The yield table is also saved to disk by default, in a format customizable via the table_format argument.
In [21]:
_ = cabinetry.tabulate.yields(model_pred, data)
INFO - cabinetry.tabulate - yields per bin for pre-fit model prediction: ╒════════════╤═════════════════╤════════════════╤════════════════╤═══════════════╕ │ sample │ Signal_region │ │ │ │ │ │ bin 1 │ bin 2 │ bin 3 │ bin 4 │ ╞════════════╪═════════════════╪════════════════╪════════════════╪═══════════════╡ │ Background │ 112.74 ± 69.04 │ 128.62 ± 58.33 │ 88.11 ± 38.06 │ 55.25 ± 45.20 │ ├────────────┼─────────────────┼────────────────┼────────────────┼───────────────┤ │ Signal │ 0.00 ± 0.00 │ 1.59 ± 0.08 │ 23.62 ± 1.18 │ 24.55 ± 1.23 │ ├────────────┼─────────────────┼────────────────┼────────────────┼───────────────┤ │ total │ 112.74 ± 69.04 │ 130.21 ± 58.34 │ 111.72 ± 38.21 │ 79.79 ± 45.29 │ ├────────────┼─────────────────┼────────────────┼────────────────┼───────────────┤ │ data │ 112.00 │ 112.00 │ 124.00 │ 66.00 │ ╘════════════╧═════════════════╧════════════════╧════════════════╧═══════════════╛ INFO - cabinetry.tabulate - saving table as tables/yields_per_bin_pre-fit.txt
We can also have a look at the table we saved to disk.
Other supported formats include "html" and "latex".
In [22]:
!cat tables/yields_per_bin_pre-fit.txt
sample Signal_region Signal_region Signal_region Signal_region
bin 1 bin 2 bin 3 bin 4
---------- --------------- --------------- --------------- ---------------
Background 112.74 ± 69.04 128.62 ± 58.33 88.11 ± 38.06 55.25 ± 45.20
Signal 0.00 ± 0.00 1.59 ± 0.08 23.62 ± 1.18 24.55 ± 1.23
total 112.74 ± 69.04 130.21 ± 58.34 111.72 ± 38.21 79.79 ± 45.29
data 112.00 112.00 124.00 66.00
We can also take a look at the post-fit model.
In [23]:
model_pred_postfit = cabinetry.model_utils.prediction(model, fit_results=fit_results)
_ = cabinetry.visualize.data_mc(model_pred_postfit, data, config=cabinetry_config)
DEBUG - cabinetry.model_utils - total stdev is [[13.2, 7.2, 7.06, 7.73]] DEBUG - cabinetry.model_utils - total stdev per channel is [20.7] INFO - cabinetry.visualize.utils - saving figure as figures/Signal_region_postfit.pdf
Beyond simple maximum likelihood fitting¶
cabinetry provides a range of useful utilities for statistical inference besides simple maximum likelihood fitting.
To start, let's look at ranking nuisance parameters by their impact on the parameter of interest.
In [24]:
ranking_results = cabinetry.fit.ranking(model, data)
INFO - cabinetry.fit - Migrad status: ┌─────────────────────────────────────────────────────────────────────────┐ │ Migrad │ ├──────────────────────────────────┬──────────────────────────────────────┤ │ FCN = 18.54 │ Nfcn = 330 │ │ EDM = 1.04e-06 (Goal: 0.0002) │ │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Valid Minimum │ No Parameters at limit │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Below EDM threshold (goal x 10) │ Below call limit │ ├───────────────┬──────────────────┼───────────┬─────────────┬────────────┤ │ Covariance │ Hesse ok │ Accurate │ Pos. def. │ Not forced │ └───────────────┴──────────────────┴───────────┴─────────────┴────────────┘ DEBUG - cabinetry.fit - -2 log(L) = 18.535252 at best-fit point INFO - cabinetry.fit - calculating impact of Modeling on Signal_norm INFO - cabinetry.fit - Migrad status: ┌─────────────────────────────────────────────────────────────────────────┐ │ Migrad │ ├──────────────────────────────────┬──────────────────────────────────────┤ │ FCN = 38.93 │ Nfcn = 274 │ │ EDM = 2.61e-05 (Goal: 0.0002) │ │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Valid Minimum │ SOME Parameters at limit │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Below EDM threshold (goal x 10) │ Below call limit │ ├───────────────┬──────────────────┼───────────┬─────────────┬────────────┤ │ Covariance │ Hesse ok │ Accurate │ Pos. def. │ Not forced │ └───────────────┴──────────────────┴───────────┴─────────────┴────────────┘ DEBUG - cabinetry.fit - -2 log(L) = 38.933945 at best-fit point DEBUG - cabinetry.fit - POI is 0.000024, difference to nominal is -1.593742 INFO - cabinetry.fit - Migrad status: ┌─────────────────────────────────────────────────────────────────────────┐ │ Migrad │ ├──────────────────────────────────┬──────────────────────────────────────┤ │ FCN = 21.62 │ Nfcn = 265 │ │ EDM = 8.01e-09 (Goal: 0.0002) │ │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Valid Minimum │ No Parameters at limit │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Below EDM threshold (goal x 10) │ Below call limit │ ├───────────────┬──────────────────┼───────────┬─────────────┬────────────┤ │ Covariance │ Hesse ok │ Accurate │ Pos. def. │ Not forced │ └───────────────┴──────────────────┴───────────┴─────────────┴────────────┘ DEBUG - cabinetry.fit - -2 log(L) = 21.622209 at best-fit point DEBUG - cabinetry.fit - POI is 3.039836, difference to nominal is 1.446071 INFO - cabinetry.fit - Migrad status: ┌─────────────────────────────────────────────────────────────────────────┐ │ Migrad │ ├──────────────────────────────────┬──────────────────────────────────────┤ │ FCN = 21.51 │ Nfcn = 217 │ │ EDM = 7.95e-06 (Goal: 0.0002) │ │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Valid Minimum │ No Parameters at limit │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Below EDM threshold (goal x 10) │ Below call limit │ ├───────────────┬──────────────────┼───────────┬─────────────┬────────────┤ │ Covariance │ Hesse ok │ Accurate │ Pos. def. │ Not forced │ └───────────────┴──────────────────┴───────────┴─────────────┴────────────┘ DEBUG - cabinetry.fit - -2 log(L) = 21.505866 at best-fit point DEBUG - cabinetry.fit - POI is 0.440453, difference to nominal is -1.153312 INFO - cabinetry.fit - Migrad status: ┌─────────────────────────────────────────────────────────────────────────┐ │ Migrad │ ├──────────────────────────────────┬──────────────────────────────────────┤ │ FCN = 19.54 │ Nfcn = 245 │ │ EDM = 2.71e-06 (Goal: 0.0002) │ │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Valid Minimum │ No Parameters at limit │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Below EDM threshold (goal x 10) │ Below call limit │ ├───────────────┬──────────────────┼───────────┬─────────────┬────────────┤ │ Covariance │ Hesse ok │ Accurate │ Pos. def. │ Not forced │ └───────────────┴──────────────────┴───────────┴─────────────┴────────────┘ DEBUG - cabinetry.fit - -2 log(L) = 19.542408 at best-fit point DEBUG - cabinetry.fit - POI is 2.468905, difference to nominal is 0.875140 INFO - cabinetry.fit - calculating impact of WeightBasedModeling on Signal_norm INFO - cabinetry.fit - Migrad status: ┌─────────────────────────────────────────────────────────────────────────┐ │ Migrad │ ├──────────────────────────────────┬──────────────────────────────────────┤ │ FCN = 21.81 │ Nfcn = 253 │ │ EDM = 3.34e-05 (Goal: 0.0002) │ │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Valid Minimum │ SOME Parameters at limit │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Below EDM threshold (goal x 10) │ Below call limit │ ├───────────────┬──────────────────┼───────────┬─────────────┬────────────┤ │ Covariance │ Hesse ok │ Accurate │ Pos. def. │ Not forced │ └───────────────┴──────────────────┴───────────┴─────────────┴────────────┘ DEBUG - cabinetry.fit - -2 log(L) = 21.805618 at best-fit point DEBUG - cabinetry.fit - POI is 0.046114, difference to nominal is -1.547652 INFO - cabinetry.fit - Migrad status: ┌─────────────────────────────────────────────────────────────────────────┐ │ Migrad │ ├──────────────────────────────────┬──────────────────────────────────────┤ │ FCN = 20.84 │ Nfcn = 224 │ │ EDM = 2.04e-05 (Goal: 0.0002) │ │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Valid Minimum │ No Parameters at limit │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Below EDM threshold (goal x 10) │ Below call limit │ ├───────────────┬──────────────────┼───────────┬─────────────┬────────────┤ │ Covariance │ Hesse ok │ Accurate │ Pos. def. │ Not forced │ └───────────────┴──────────────────┴───────────┴─────────────┴────────────┘ DEBUG - cabinetry.fit - -2 log(L) = 20.838788 at best-fit point DEBUG - cabinetry.fit - POI is 2.781734, difference to nominal is 1.187969 INFO - cabinetry.fit - Migrad status: ┌─────────────────────────────────────────────────────────────────────────┐ │ Migrad │ ├──────────────────────────────────┬──────────────────────────────────────┤ │ FCN = 19.66 │ Nfcn = 201 │ │ EDM = 2.82e-05 (Goal: 0.0002) │ │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Valid Minimum │ No Parameters at limit │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Below EDM threshold (goal x 10) │ Below call limit │ ├───────────────┬──────────────────┼───────────┬─────────────┬────────────┤ │ Covariance │ Hesse ok │ Accurate │ Pos. def. │ Not forced │ └───────────────┴──────────────────┴───────────┴─────────────┴────────────┘ DEBUG - cabinetry.fit - -2 log(L) = 19.662341 at best-fit point DEBUG - cabinetry.fit - POI is 0.644792, difference to nominal is -0.948973 INFO - cabinetry.fit - Migrad status: ┌─────────────────────────────────────────────────────────────────────────┐ │ Migrad │ ├──────────────────────────────────┬──────────────────────────────────────┤ │ FCN = 19.51 │ Nfcn = 222 │ │ EDM = 6.61e-06 (Goal: 0.0002) │ │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Valid Minimum │ No Parameters at limit │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Below EDM threshold (goal x 10) │ Below call limit │ ├───────────────┬──────────────────┼───────────┬─────────────┬────────────┤ │ Covariance │ Hesse ok │ Accurate │ Pos. def. │ Not forced │ └───────────────┴──────────────────┴───────────┴─────────────┴────────────┘ DEBUG - cabinetry.fit - -2 log(L) = 19.507292 at best-fit point DEBUG - cabinetry.fit - POI is 2.407076, difference to nominal is 0.813311 INFO - cabinetry.fit - calculating impact of Luminosity on Signal_norm INFO - cabinetry.fit - Migrad status: ┌─────────────────────────────────────────────────────────────────────────┐ │ Migrad │ ├──────────────────────────────────┬──────────────────────────────────────┤ │ FCN = 19.54 │ Nfcn = 251 │ │ EDM = 9.08e-05 (Goal: 0.0002) │ │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Valid Minimum │ No Parameters at limit │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Below EDM threshold (goal x 10) │ Below call limit │ ├───────────────┬──────────────────┼───────────┬─────────────┬────────────┤ │ Covariance │ Hesse ok │ Accurate │ Pos. def. │ Not forced │ └───────────────┴──────────────────┴───────────┴─────────────┴────────────┘ DEBUG - cabinetry.fit - -2 log(L) = 19.540404 at best-fit point DEBUG - cabinetry.fit - POI is 1.434503, difference to nominal is -0.159262 INFO - cabinetry.fit - Migrad status: ┌─────────────────────────────────────────────────────────────────────────┐ │ Migrad │ ├──────────────────────────────────┬──────────────────────────────────────┤ │ FCN = 19.55 │ Nfcn = 251 │ │ EDM = 5.17e-06 (Goal: 0.0002) │ │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Valid Minimum │ No Parameters at limit │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Below EDM threshold (goal x 10) │ Below call limit │ ├───────────────┬──────────────────┼───────────┬─────────────┬────────────┤ │ Covariance │ Hesse ok │ Accurate │ Pos. def. │ Not forced │ └───────────────┴──────────────────┴───────────┴─────────────┴────────────┘ DEBUG - cabinetry.fit - -2 log(L) = 19.546177 at best-fit point DEBUG - cabinetry.fit - POI is 1.767688, difference to nominal is 0.173923 INFO - cabinetry.fit - Migrad status: ┌─────────────────────────────────────────────────────────────────────────┐ │ Migrad │ ├──────────────────────────────────┬──────────────────────────────────────┤ │ FCN = 19.52 │ Nfcn = 251 │ │ EDM = 9.19e-05 (Goal: 0.0002) │ │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Valid Minimum │ No Parameters at limit │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Below EDM threshold (goal x 10) │ Below call limit │ ├───────────────┬──────────────────┼───────────┬─────────────┬────────────┤ │ Covariance │ Hesse ok │ Accurate │ Pos. def. │ Not forced │ └───────────────┴──────────────────┴───────────┴─────────────┴────────────┘ DEBUG - cabinetry.fit - -2 log(L) = 19.522365 at best-fit point DEBUG - cabinetry.fit - POI is 1.435786, difference to nominal is -0.157979 INFO - cabinetry.fit - Migrad status: ┌─────────────────────────────────────────────────────────────────────────┐ │ Migrad │ ├──────────────────────────────────┬──────────────────────────────────────┤ │ FCN = 19.53 │ Nfcn = 251 │ │ EDM = 4.72e-06 (Goal: 0.0002) │ │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Valid Minimum │ No Parameters at limit │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Below EDM threshold (goal x 10) │ Below call limit │ ├───────────────┬──────────────────┼───────────┬─────────────┬────────────┤ │ Covariance │ Hesse ok │ Accurate │ Pos. def. │ Not forced │ └───────────────┴──────────────────┴───────────┴─────────────┴────────────┘ DEBUG - cabinetry.fit - -2 log(L) = 19.528062 at best-fit point DEBUG - cabinetry.fit - POI is 1.766161, difference to nominal is 0.172396 INFO - cabinetry.fit - calculating impact of staterror_Signal_region[0] on Signal_norm INFO - cabinetry.fit - Migrad status: ┌─────────────────────────────────────────────────────────────────────────┐ │ Migrad │ ├──────────────────────────────────┬──────────────────────────────────────┤ │ FCN = 19.59 │ Nfcn = 270 │ │ EDM = 6.15e-05 (Goal: 0.0002) │ │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Valid Minimum │ No Parameters at limit │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Below EDM threshold (goal x 10) │ Below call limit │ ├───────────────┬──────────────────┼───────────┬─────────────┬────────────┤ │ Covariance │ Hesse ok │ Accurate │ Pos. def. │ Not forced │ └───────────────┴──────────────────┴───────────┴─────────────┴────────────┘ DEBUG - cabinetry.fit - -2 log(L) = 19.586174 at best-fit point DEBUG - cabinetry.fit - POI is 1.461954, difference to nominal is -0.131811 INFO - cabinetry.fit - Migrad status: ┌─────────────────────────────────────────────────────────────────────────┐ │ Migrad │ ├──────────────────────────────────┬──────────────────────────────────────┤ │ FCN = 19.59 │ Nfcn = 270 │ │ EDM = 4.14e-06 (Goal: 0.0002) │ │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Valid Minimum │ No Parameters at limit │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Below EDM threshold (goal x 10) │ Below call limit │ ├───────────────┬──────────────────┼───────────┬─────────────┬────────────┤ │ Covariance │ Hesse ok │ Accurate │ Pos. def. │ Not forced │ └───────────────┴──────────────────┴───────────┴─────────────┴────────────┘ DEBUG - cabinetry.fit - -2 log(L) = 19.592124 at best-fit point DEBUG - cabinetry.fit - POI is 1.719186, difference to nominal is 0.125421 INFO - cabinetry.fit - Migrad status: ┌─────────────────────────────────────────────────────────────────────────┐ │ Migrad │ ├──────────────────────────────────┬──────────────────────────────────────┤ │ FCN = 19.53 │ Nfcn = 270 │ │ EDM = 5.76e-05 (Goal: 0.0002) │ │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Valid Minimum │ No Parameters at limit │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Below EDM threshold (goal x 10) │ Below call limit │ ├───────────────┬──────────────────┼───────────┬─────────────┬────────────┤ │ Covariance │ Hesse ok │ Accurate │ Pos. def. │ Not forced │ └───────────────┴──────────────────┴───────────┴─────────────┴────────────┘ DEBUG - cabinetry.fit - -2 log(L) = 19.532562 at best-fit point DEBUG - cabinetry.fit - POI is 1.465410, difference to nominal is -0.128355 INFO - cabinetry.fit - Migrad status: ┌─────────────────────────────────────────────────────────────────────────┐ │ Migrad │ ├──────────────────────────────────┬──────────────────────────────────────┤ │ FCN = 19.54 │ Nfcn = 270 │ │ EDM = 2.95e-06 (Goal: 0.0002) │ │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Valid Minimum │ No Parameters at limit │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Below EDM threshold (goal x 10) │ Below call limit │ ├───────────────┬──────────────────┼───────────┬─────────────┬────────────┤ │ Covariance │ Hesse ok │ Accurate │ Pos. def. │ Not forced │ └───────────────┴──────────────────┴───────────┴─────────────┴────────────┘ DEBUG - cabinetry.fit - -2 log(L) = 19.538020 at best-fit point DEBUG - cabinetry.fit - POI is 1.715952, difference to nominal is 0.122187 INFO - cabinetry.fit - calculating impact of staterror_Signal_region[1] on Signal_norm INFO - cabinetry.fit - Migrad status: ┌─────────────────────────────────────────────────────────────────────────┐ │ Migrad │ ├──────────────────────────────────┬──────────────────────────────────────┤ │ FCN = 19.6 │ Nfcn = 286 │ │ EDM = 3.77e-06 (Goal: 0.0002) │ │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Valid Minimum │ No Parameters at limit │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Below EDM threshold (goal x 10) │ Below call limit │ ├───────────────┬──────────────────┼───────────┬─────────────┬────────────┤ │ Covariance │ Hesse ok │ Accurate │ Pos. def. │ Not forced │ └───────────────┴──────────────────┴───────────┴─────────────┴────────────┘ DEBUG - cabinetry.fit - -2 log(L) = 19.599005 at best-fit point DEBUG - cabinetry.fit - POI is 1.747681, difference to nominal is 0.153916 INFO - cabinetry.fit - Migrad status: ┌─────────────────────────────────────────────────────────────────────────┐ │ Migrad │ ├──────────────────────────────────┬──────────────────────────────────────┤ │ FCN = 19.6 │ Nfcn = 269 │ │ EDM = 0.0001 (Goal: 0.0002) │ │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Valid Minimum │ No Parameters at limit │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Below EDM threshold (goal x 10) │ Below call limit │ ├───────────────┬──────────────────┼───────────┬─────────────┬────────────┤ │ Covariance │ Hesse ok │ Accurate │ Pos. def. │ Not forced │ └───────────────┴──────────────────┴───────────┴─────────────┴────────────┘ DEBUG - cabinetry.fit - -2 log(L) = 19.604940 at best-fit point DEBUG - cabinetry.fit - POI is 1.426444, difference to nominal is -0.167321 INFO - cabinetry.fit - Migrad status: ┌─────────────────────────────────────────────────────────────────────────┐ │ Migrad │ ├──────────────────────────────────┬──────────────────────────────────────┤ │ FCN = 19.53 │ Nfcn = 286 │ │ EDM = 3.65e-06 (Goal: 0.0002) │ │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Valid Minimum │ No Parameters at limit │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Below EDM threshold (goal x 10) │ Below call limit │ ├───────────────┬──────────────────┼───────────┬─────────────┬────────────┤ │ Covariance │ Hesse ok │ Accurate │ Pos. def. │ Not forced │ └───────────────┴──────────────────┴───────────┴─────────────┴────────────┘ DEBUG - cabinetry.fit - -2 log(L) = 19.532056 at best-fit point DEBUG - cabinetry.fit - POI is 1.742768, difference to nominal is 0.149002 INFO - cabinetry.fit - Migrad status: ┌─────────────────────────────────────────────────────────────────────────┐ │ Migrad │ ├──────────────────────────────────┬──────────────────────────────────────┤ │ FCN = 19.54 │ Nfcn = 269 │ │ EDM = 9.53e-05 (Goal: 0.0002) │ │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Valid Minimum │ No Parameters at limit │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Below EDM threshold (goal x 10) │ Below call limit │ ├───────────────┬──────────────────┼───────────┬─────────────┬────────────┤ │ Covariance │ Hesse ok │ Accurate │ Pos. def. │ Not forced │ └───────────────┴──────────────────┴───────────┴─────────────┴────────────┘ DEBUG - cabinetry.fit - -2 log(L) = 19.537490 at best-fit point DEBUG - cabinetry.fit - POI is 1.432287, difference to nominal is -0.161478 INFO - cabinetry.fit - calculating impact of staterror_Signal_region[2] on Signal_norm INFO - cabinetry.fit - Migrad status: ┌─────────────────────────────────────────────────────────────────────────┐ │ Migrad │ ├──────────────────────────────────┬──────────────────────────────────────┤ │ FCN = 19.58 │ Nfcn = 270 │ │ EDM = 2.15e-05 (Goal: 0.0002) │ │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Valid Minimum │ No Parameters at limit │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Below EDM threshold (goal x 10) │ Below call limit │ ├───────────────┬──────────────────┼───────────┬─────────────┬────────────┤ │ Covariance │ Hesse ok │ Accurate │ Pos. def. │ Not forced │ └───────────────┴──────────────────┴───────────┴─────────────┴────────────┘ DEBUG - cabinetry.fit - -2 log(L) = 19.578176 at best-fit point DEBUG - cabinetry.fit - POI is 1.401209, difference to nominal is -0.192556 INFO - cabinetry.fit - Migrad status: ┌─────────────────────────────────────────────────────────────────────────┐ │ Migrad │ ├──────────────────────────────────┬──────────────────────────────────────┤ │ FCN = 19.57 │ Nfcn = 271 │ │ EDM = 0.00011 (Goal: 0.0002) │ │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Valid Minimum │ No Parameters at limit │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Below EDM threshold (goal x 10) │ Below call limit │ ├───────────────┬──────────────────┼───────────┬─────────────┬────────────┤ │ Covariance │ Hesse ok │ Accurate │ Pos. def. │ Not forced │ └───────────────┴──────────────────┴───────────┴─────────────┴────────────┘ DEBUG - cabinetry.fit - -2 log(L) = 19.568416 at best-fit point DEBUG - cabinetry.fit - POI is 1.808117, difference to nominal is 0.214352 INFO - cabinetry.fit - Migrad status: ┌─────────────────────────────────────────────────────────────────────────┐ │ Migrad │ ├──────────────────────────────────┬──────────────────────────────────────┤ │ FCN = 19.54 │ Nfcn = 270 │ │ EDM = 2.3e-05 (Goal: 0.0002) │ │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Valid Minimum │ No Parameters at limit │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Below EDM threshold (goal x 10) │ Below call limit │ ├───────────────┬──────────────────┼───────────┬─────────────┬────────────┤ │ Covariance │ Hesse ok │ Accurate │ Pos. def. │ Not forced │ └───────────────┴──────────────────┴───────────┴─────────────┴────────────┘ DEBUG - cabinetry.fit - -2 log(L) = 19.539799 at best-fit point DEBUG - cabinetry.fit - POI is 1.404519, difference to nominal is -0.189246 INFO - cabinetry.fit - Migrad status: ┌─────────────────────────────────────────────────────────────────────────┐ │ Migrad │ ├──────────────────────────────────┬──────────────────────────────────────┤ │ FCN = 19.53 │ Nfcn = 270 │ │ EDM = 0.000111 (Goal: 0.0002) │ │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Valid Minimum │ No Parameters at limit │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Below EDM threshold (goal x 10) │ Below call limit │ ├───────────────┬──────────────────┼───────────┬─────────────┬────────────┤ │ Covariance │ Hesse ok │ Accurate │ Pos. def. │ Not forced │ └───────────────┴──────────────────┴───────────┴─────────────┴────────────┘ DEBUG - cabinetry.fit - -2 log(L) = 19.530558 at best-fit point DEBUG - cabinetry.fit - POI is 1.804686, difference to nominal is 0.210921 INFO - cabinetry.fit - calculating impact of staterror_Signal_region[3] on Signal_norm INFO - cabinetry.fit - Migrad status: ┌─────────────────────────────────────────────────────────────────────────┐ │ Migrad │ ├──────────────────────────────────┬──────────────────────────────────────┤ │ FCN = 19.52 │ Nfcn = 271 │ │ EDM = 8.59e-05 (Goal: 0.0002) │ │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Valid Minimum │ No Parameters at limit │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Below EDM threshold (goal x 10) │ Below call limit │ ├───────────────┬──────────────────┼───────────┬─────────────┬────────────┤ │ Covariance │ Hesse ok │ Accurate │ Pos. def. │ Not forced │ └───────────────┴──────────────────┴───────────┴─────────────┴────────────┘ DEBUG - cabinetry.fit - -2 log(L) = 19.518509 at best-fit point DEBUG - cabinetry.fit - POI is 1.726688, difference to nominal is 0.132923 INFO - cabinetry.fit - Migrad status: ┌─────────────────────────────────────────────────────────────────────────┐ │ Migrad │ ├──────────────────────────────────┬──────────────────────────────────────┤ │ FCN = 19.53 │ Nfcn = 269 │ │ EDM = 4.79e-05 (Goal: 0.0002) │ │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Valid Minimum │ No Parameters at limit │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Below EDM threshold (goal x 10) │ Below call limit │ ├───────────────┬──────────────────┼───────────┬─────────────┬────────────┤ │ Covariance │ Hesse ok │ Accurate │ Pos. def. │ Not forced │ └───────────────┴──────────────────┴───────────┴─────────────┴────────────┘ DEBUG - cabinetry.fit - -2 log(L) = 19.530219 at best-fit point DEBUG - cabinetry.fit - POI is 1.491218, difference to nominal is -0.102547 INFO - cabinetry.fit - Migrad status: ┌─────────────────────────────────────────────────────────────────────────┐ │ Migrad │ ├──────────────────────────────────┬──────────────────────────────────────┤ │ FCN = 19.53 │ Nfcn = 271 │ │ EDM = 8.25e-05 (Goal: 0.0002) │ │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Valid Minimum │ No Parameters at limit │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Below EDM threshold (goal x 10) │ Below call limit │ ├───────────────┬──────────────────┼───────────┬─────────────┬────────────┤ │ Covariance │ Hesse ok │ Accurate │ Pos. def. │ Not forced │ └───────────────┴──────────────────┴───────────┴─────────────┴────────────┘ DEBUG - cabinetry.fit - -2 log(L) = 19.529643 at best-fit point DEBUG - cabinetry.fit - POI is 1.727432, difference to nominal is 0.133667 INFO - cabinetry.fit - Migrad status: ┌─────────────────────────────────────────────────────────────────────────┐ │ Migrad │ ├──────────────────────────────────┬──────────────────────────────────────┤ │ FCN = 19.54 │ Nfcn = 269 │ │ EDM = 4.78e-05 (Goal: 0.0002) │ │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Valid Minimum │ No Parameters at limit │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Below EDM threshold (goal x 10) │ Below call limit │ ├───────────────┬──────────────────┼───────────┬─────────────┬────────────┤ │ Covariance │ Hesse ok │ Accurate │ Pos. def. │ Not forced │ └───────────────┴──────────────────┴───────────┴─────────────┴────────────┘ DEBUG - cabinetry.fit - -2 log(L) = 19.541530 at best-fit point DEBUG - cabinetry.fit - POI is 1.490819, difference to nominal is -0.102947
The previous cell ran a lot of maximum likelihood fits to calculate all the input needed to rank nuisance parameters. We will visualize them next.
In [25]:
cabinetry.visualize.ranking(ranking_results)
INFO - cabinetry.visualize.utils - saving figure as figures/ranking.pdf
Out[25]:
The results are contained in the ranking_results object. It is a simple named tuple, we can have a look at its content.
In [26]:
ranking_results
Out[26]:
RankingResults(bestfit=array([-0.27529071, -0.52768031, -0.07941828, 1.00123651, 0.9883001 ,
1.02156575, 0.98291109]), uncertainty=array([0.5683095 , 0.64774531, 0.99100884, 0.04114037, 0.03842981,
0.04690882, 0.06096864]), labels=['Modeling', 'WeightBasedModeling', 'Luminosity', 'staterror_Signal_region[0]', 'staterror_Signal_region[1]', 'staterror_Signal_region[2]', 'staterror_Signal_region[3]'], prefit_up=array([-1.59374151, -1.54765161, -0.15926218, -0.13181116, 0.1539161 ,
-0.19255584, 0.13292265]), prefit_down=array([ 1.44607054, 1.18796859, 0.17392294, 0.12542056, -0.16732123,
0.21435221, -0.10254726]), postfit_up=array([-1.15331231, -0.94897302, -0.15797906, -0.12835492, 0.14900244,
-0.18924588, 0.13366721]), postfit_down=array([ 0.87514006, 0.81331124, 0.1723961 , 0.12218725, -0.16147842,
0.2109206 , -0.10294653]))
We can also perform likelihood scans for parameters.
The example below performs a scan for the Modeling nuisance parameter.
In [27]:
scan_results = cabinetry.fit.scan(model, data, "WeightBasedModeling")
INFO - cabinetry.fit - Migrad status: ┌─────────────────────────────────────────────────────────────────────────┐ │ Migrad │ ├──────────────────────────────────┬──────────────────────────────────────┤ │ FCN = 18.54 │ Nfcn = 330 │ │ EDM = 1.04e-06 (Goal: 0.0002) │ │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Valid Minimum │ No Parameters at limit │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Below EDM threshold (goal x 10) │ Below call limit │ ├───────────────┬──────────────────┼───────────┬─────────────┬────────────┤ │ Covariance │ Hesse ok │ Accurate │ Pos. def. │ Not forced │ └───────────────┴──────────────────┴───────────┴─────────────┴────────────┘ DEBUG - cabinetry.fit - -2 log(L) = 18.535252 at best-fit point INFO - cabinetry.fit - performing likelihood scan for WeightBasedModeling in range (-1.823, 0.768) with 11 steps DEBUG - cabinetry.fit - performing fit with WeightBasedModeling = -1.823 INFO - cabinetry.fit - Migrad status: ┌─────────────────────────────────────────────────────────────────────────┐ │ Migrad │ ├──────────────────────────────────┬──────────────────────────────────────┤ │ FCN = 22.38 │ Nfcn = 225 │ │ EDM = 5.23e-05 (Goal: 0.0002) │ │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Valid Minimum │ No Parameters at limit │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Below EDM threshold (goal x 10) │ Below call limit │ ├───────────────┬──────────────────┼───────────┬─────────────┬────────────┤ │ Covariance │ Hesse ok │ Accurate │ Pos. def. │ Not forced │ └───────────────┴──────────────────┴───────────┴─────────────┴────────────┘ DEBUG - cabinetry.fit - -2 log(L) = 22.383546 at best-fit point DEBUG - cabinetry.fit - performing fit with WeightBasedModeling = -1.564 INFO - cabinetry.fit - Migrad status: ┌─────────────────────────────────────────────────────────────────────────┐ │ Migrad │ ├──────────────────────────────────┬──────────────────────────────────────┤ │ FCN = 21.01 │ Nfcn = 224 │ │ EDM = 2.34e-05 (Goal: 0.0002) │ │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Valid Minimum │ No Parameters at limit │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Below EDM threshold (goal x 10) │ Below call limit │ ├───────────────┬──────────────────┼───────────┬─────────────┬────────────┤ │ Covariance │ Hesse ok │ Accurate │ Pos. def. │ Not forced │ └───────────────┴──────────────────┴───────────┴─────────────┴────────────┘ DEBUG - cabinetry.fit - -2 log(L) = 21.008036 at best-fit point DEBUG - cabinetry.fit - performing fit with WeightBasedModeling = -1.305 INFO - cabinetry.fit - Migrad status: ┌─────────────────────────────────────────────────────────────────────────┐ │ Migrad │ ├──────────────────────────────────┬──────────────────────────────────────┤ │ FCN = 19.93 │ Nfcn = 224 │ │ EDM = 9.15e-06 (Goal: 0.0002) │ │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Valid Minimum │ No Parameters at limit │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Below EDM threshold (goal x 10) │ Below call limit │ ├───────────────┬──────────────────┼───────────┬─────────────┬────────────┤ │ Covariance │ Hesse ok │ Accurate │ Pos. def. │ Not forced │ └───────────────┴──────────────────┴───────────┴─────────────┴────────────┘ DEBUG - cabinetry.fit - -2 log(L) = 19.932185 at best-fit point DEBUG - cabinetry.fit - performing fit with WeightBasedModeling = -1.046 INFO - cabinetry.fit - Migrad status: ┌─────────────────────────────────────────────────────────────────────────┐ │ Migrad │ ├──────────────────────────────────┬──────────────────────────────────────┤ │ FCN = 19.16 │ Nfcn = 221 │ │ EDM = 5.01e-06 (Goal: 0.0002) │ │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Valid Minimum │ No Parameters at limit │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Below EDM threshold (goal x 10) │ Below call limit │ ├───────────────┬──────────────────┼───────────┬─────────────┬────────────┤ │ Covariance │ Hesse ok │ Accurate │ Pos. def. │ Not forced │ └───────────────┴──────────────────┴───────────┴─────────────┴────────────┘ DEBUG - cabinetry.fit - -2 log(L) = 19.158841 at best-fit point DEBUG - cabinetry.fit - performing fit with WeightBasedModeling = -0.787 INFO - cabinetry.fit - Migrad status: ┌─────────────────────────────────────────────────────────────────────────┐ │ Migrad │ ├──────────────────────────────────┬──────────────────────────────────────┤ │ FCN = 18.69 │ Nfcn = 219 │ │ EDM = 2.98e-06 (Goal: 0.0002) │ │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Valid Minimum │ No Parameters at limit │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Below EDM threshold (goal x 10) │ Below call limit │ ├───────────────┬──────────────────┼───────────┬─────────────┬────────────┤ │ Covariance │ Hesse ok │ Accurate │ Pos. def. │ Not forced │ └───────────────┴──────────────────┴───────────┴─────────────┴────────────┘ DEBUG - cabinetry.fit - -2 log(L) = 18.692966 at best-fit point DEBUG - cabinetry.fit - performing fit with WeightBasedModeling = -0.528 INFO - cabinetry.fit - Migrad status: ┌─────────────────────────────────────────────────────────────────────────┐ │ Migrad │ ├──────────────────────────────────┬──────────────────────────────────────┤ │ FCN = 18.54 │ Nfcn = 199 │ │ EDM = 4.69e-05 (Goal: 0.0002) │ │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Valid Minimum │ No Parameters at limit │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Below EDM threshold (goal x 10) │ Below call limit │ ├───────────────┬──────────────────┼───────────┬─────────────┬────────────┤ │ Covariance │ Hesse ok │ Accurate │ Pos. def. │ Not forced │ └───────────────┴──────────────────┴───────────┴─────────────┴────────────┘ DEBUG - cabinetry.fit - -2 log(L) = 18.535298 at best-fit point DEBUG - cabinetry.fit - performing fit with WeightBasedModeling = -0.269 INFO - cabinetry.fit - Migrad status: ┌─────────────────────────────────────────────────────────────────────────┐ │ Migrad │ ├──────────────────────────────────┬──────────────────────────────────────┤ │ FCN = 18.7 │ Nfcn = 201 │ │ EDM = 5.44e-06 (Goal: 0.0002) │ │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Valid Minimum │ No Parameters at limit │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Below EDM threshold (goal x 10) │ Below call limit │ ├───────────────┬──────────────────┼───────────┬─────────────┬────────────┤ │ Covariance │ Hesse ok │ Accurate │ Pos. def. │ Not forced │ └───────────────┴──────────────────┴───────────┴─────────────┴────────────┘ DEBUG - cabinetry.fit - -2 log(L) = 18.696625 at best-fit point DEBUG - cabinetry.fit - performing fit with WeightBasedModeling = -0.009 INFO - cabinetry.fit - Migrad status: ┌─────────────────────────────────────────────────────────────────────────┐ │ Migrad │ ├──────────────────────────────────┬──────────────────────────────────────┤ │ FCN = 19.22 │ Nfcn = 200 │ │ EDM = 7.7e-07 (Goal: 0.0002) │ │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Valid Minimum │ No Parameters at limit │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Below EDM threshold (goal x 10) │ Below call limit │ ├───────────────┬──────────────────┼───────────┬─────────────┬────────────┤ │ Covariance │ Hesse ok │ Accurate │ Pos. def. │ Not forced │ └───────────────┴──────────────────┴───────────┴─────────────┴────────────┘ DEBUG - cabinetry.fit - -2 log(L) = 19.219275 at best-fit point DEBUG - cabinetry.fit - performing fit with WeightBasedModeling = 0.250 INFO - cabinetry.fit - Migrad status: ┌─────────────────────────────────────────────────────────────────────────┐ │ Migrad │ ├──────────────────────────────────┬──────────────────────────────────────┤ │ FCN = 20.27 │ Nfcn = 201 │ │ EDM = 0.0001 (Goal: 0.0002) │ │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Valid Minimum │ No Parameters at limit │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Below EDM threshold (goal x 10) │ Below call limit │ ├───────────────┬──────────────────┼───────────┬─────────────┬────────────┤ │ Covariance │ Hesse ok │ Accurate │ Pos. def. │ Not forced │ └───────────────┴──────────────────┴───────────┴─────────────┴────────────┘ DEBUG - cabinetry.fit - -2 log(L) = 20.270438 at best-fit point DEBUG - cabinetry.fit - performing fit with WeightBasedModeling = 0.509 INFO - cabinetry.fit - Migrad status: ┌─────────────────────────────────────────────────────────────────────────┐ │ Migrad │ ├──────────────────────────────────┬──────────────────────────────────────┤ │ FCN = 22.13 │ Nfcn = 291 │ │ EDM = 5.14e-05 (Goal: 0.0002) │ │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Valid Minimum │ SOME Parameters at limit │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Below EDM threshold (goal x 10) │ Below call limit │ ├───────────────┬──────────────────┼───────────┬─────────────┬────────────┤ │ Covariance │ Hesse ok │ Accurate │ Pos. def. │ Not forced │ └───────────────┴──────────────────┴───────────┴─────────────┴────────────┘ DEBUG - cabinetry.fit - -2 log(L) = 22.125564 at best-fit point DEBUG - cabinetry.fit - performing fit with WeightBasedModeling = 0.768 INFO - cabinetry.fit - Migrad status: ┌─────────────────────────────────────────────────────────────────────────┐ │ Migrad │ ├──────────────────────────────────┬──────────────────────────────────────┤ │ FCN = 26.26 │ Nfcn = 256 │ │ EDM = 3.92e-06 (Goal: 0.0002) │ │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Valid Minimum │ SOME Parameters at limit │ ├──────────────────────────────────┼──────────────────────────────────────┤ │ Below EDM threshold (goal x 10) │ Below call limit │ ├───────────────┬──────────────────┼───────────┬─────────────┬────────────┤ │ Covariance │ Hesse ok │ Accurate │ Pos. def. │ Not forced │ └───────────────┴──────────────────┴───────────┴─────────────┴────────────┘ DEBUG - cabinetry.fit - -2 log(L) = 26.261757 at best-fit point
The resulting figure looks like this:
In [28]:
cabinetry.visualize.scan(scan_results)
INFO - cabinetry.visualize.utils - saving figure as figures/scan_WeightBasedModeling.pdf
Out[28]:
With cabinetry.fit.limit, we can evaluate observed and expected 95% confidence level upper parameter limits.
The implementation uses Brent bracketing to efficiently find the CLs=0.05 crossing points.
In [29]:
limit_results = cabinetry.fit.limit(model, data)
INFO - cabinetry.fit - calculating 95% confidence level upper limit for Signal_norm INFO - cabinetry.fit - determining observed upper limit DEBUG - cabinetry.fit - Signal_norm = 0.1000, observed CLs = 0.9184 DEBUG - cabinetry.fit - Signal_norm = 10.0000, observed CLs = 0.0000 DEBUG - cabinetry.fit - Signal_norm = 9.4610, observed CLs = 0.0000 DEBUG - cabinetry.fit - Signal_norm = 4.7805, observed CLs = 0.0001 DEBUG - cabinetry.fit - Signal_norm = 2.4403, observed CLs = 0.1975 DEBUG - cabinetry.fit - Signal_norm = 4.1886, observed CLs = 0.0012 DEBUG - cabinetry.fit - Signal_norm = 3.3144, observed CLs = 0.0287 DEBUG - cabinetry.fit - Signal_norm = 2.8773, observed CLs = 0.0862 DEBUG - cabinetry.fit - Signal_norm = 3.1526, observed CLs = 0.0446 DEBUG - cabinetry.fit - Signal_norm = 3.1049, observed CLs = 0.0504 DEBUG - cabinetry.fit - Signal_norm = 3.1099, observed CLs = 0.0498 INFO - cabinetry.fit - successfully converged after 11 steps INFO - cabinetry.fit - observed upper limit: 3.1099 INFO - cabinetry.fit - determining expected -2 sigma upper limit DEBUG - cabinetry.fit - Signal_norm = 0.1000, expected -2 sigma CLs = 0.7631 (cached) DEBUG - cabinetry.fit - Signal_norm = 2.4403, expected -2 sigma CLs = 0.0002 (cached) DEBUG - cabinetry.fit - Signal_norm = 2.2875, expected -2 sigma CLs = 0.0004 DEBUG - cabinetry.fit - Signal_norm = 1.1938, expected -2 sigma CLs = 0.0285 DEBUG - cabinetry.fit - Signal_norm = 0.6469, expected -2 sigma CLs = 0.1538 DEBUG - cabinetry.fit - Signal_norm = 1.0998, expected -2 sigma CLs = 0.0383 DEBUG - cabinetry.fit - Signal_norm = 0.9993, expected -2 sigma CLs = 0.0524 DEBUG - cabinetry.fit - Signal_norm = 1.0162, expected -2 sigma CLs = 0.0497 DEBUG - cabinetry.fit - Signal_norm = 1.0112, expected -2 sigma CLs = 0.0505 INFO - cabinetry.fit - successfully converged after 9 steps INFO - cabinetry.fit - expected -2 sigma upper limit: 1.0162 INFO - cabinetry.fit - determining expected -1 sigma upper limit DEBUG - cabinetry.fit - Signal_norm = 1.1938, expected -1 sigma CLs = 0.0839 (cached) DEBUG - cabinetry.fit - Signal_norm = 2.2875, expected -1 sigma CLs = 0.0034 (cached) DEBUG - cabinetry.fit - Signal_norm = 1.6542, expected -1 sigma CLs = 0.0261 DEBUG - cabinetry.fit - Signal_norm = 1.4637, expected -1 sigma CLs = 0.0434 DEBUG - cabinetry.fit - Signal_norm = 1.4025, expected -1 sigma CLs = 0.0506 DEBUG - cabinetry.fit - Signal_norm = 1.4079, expected -1 sigma CLs = 0.0500 INFO - cabinetry.fit - successfully converged after 6 steps INFO - cabinetry.fit - expected -1 sigma upper limit: 1.4079 INFO - cabinetry.fit - determining expected upper limit DEBUG - cabinetry.fit - Signal_norm = 1.6542, expected CLs = 0.1009 (cached) DEBUG - cabinetry.fit - Signal_norm = 2.2875, expected CLs = 0.0231 (cached) DEBUG - cabinetry.fit - Signal_norm = 2.0688, expected CLs = 0.0408 DEBUG - cabinetry.fit - Signal_norm = 1.9727, expected CLs = 0.0513 DEBUG - cabinetry.fit - Signal_norm = 1.9849, expected CLs = 0.0499 DEBUG - cabinetry.fit - Signal_norm = 1.9799, expected CLs = 0.0505 INFO - cabinetry.fit - successfully converged after 6 steps INFO - cabinetry.fit - expected upper limit: 1.9849 INFO - cabinetry.fit - determining expected +1 sigma upper limit DEBUG - cabinetry.fit - Signal_norm = 2.4403, expected +1 sigma CLs = 0.0902 (cached) DEBUG - cabinetry.fit - Signal_norm = 2.8773, expected +1 sigma CLs = 0.0332 (cached) DEBUG - cabinetry.fit - Signal_norm = 2.7485, expected +1 sigma CLs = 0.0457 DEBUG - cabinetry.fit - Signal_norm = 2.7087, expected +1 sigma CLs = 0.0503 DEBUG - cabinetry.fit - Signal_norm = 2.7137, expected +1 sigma CLs = 0.0497 INFO - cabinetry.fit - successfully converged after 5 steps INFO - cabinetry.fit - expected +1 sigma upper limit: 2.7087 INFO - cabinetry.fit - determining expected +2 sigma upper limit DEBUG - cabinetry.fit - Signal_norm = 3.3144, expected +2 sigma CLs = 0.0809 (cached) DEBUG - cabinetry.fit - Signal_norm = 4.1886, expected +2 sigma CLs = 0.0071 (cached) DEBUG - cabinetry.fit - Signal_norm = 3.6805, expected +2 sigma CLs = 0.0334 DEBUG - cabinetry.fit - Signal_norm = 3.5525, expected +2 sigma CLs = 0.0465 DEBUG - cabinetry.fit - Signal_norm = 3.5215, expected +2 sigma CLs = 0.0502 DEBUG - cabinetry.fit - Signal_norm = 3.5265, expected +2 sigma CLs = 0.0496 INFO - cabinetry.fit - successfully converged after 6 steps INFO - cabinetry.fit - expected +2 sigma upper limit: 3.5215 INFO - cabinetry.fit - total of 43 steps to calculate all limits INFO - cabinetry.fit - summary of 95% confidence level upper limits: INFO - cabinetry.fit - observed : 3.1099 INFO - cabinetry.fit - expected -2 sigma: 1.0162 INFO - cabinetry.fit - expected -1 sigma: 1.4079 INFO - cabinetry.fit - expected : 1.9849 INFO - cabinetry.fit - expected +1 sigma: 2.7087 INFO - cabinetry.fit - expected +2 sigma: 3.5215
Again, the results are visualized:
In [30]:
cabinetry.visualize.limit(limit_results)
INFO - cabinetry.visualize.utils - saving figure as figures/limit.pdf
Out[30]:
The observed limits are above the expected limits.
We can calculate the discovery significance with cabinetry.fit.significance:
In [31]:
significance_results = cabinetry.fit.significance(model, data)
INFO - cabinetry.fit - calculating discovery significance for Signal_norm INFO - cabinetry.fit - observed p-value: 4.889% INFO - cabinetry.fit - observed significance: 1.656 INFO - cabinetry.fit - expected p-value: 14.915% INFO - cabinetry.fit - expected significance: 1.040
In this case, we observe a 1.7 sigma excess (and expected 1.0 sigma).