T M V A Cross Validation

This macro provides an example of how to use TMVA for k-folds cross evaluation.

As input data is used a toy-MC sample consisting of two gaussian distributions.

The output file "TMVA.root" can be analysed with the use of dedicated macros (simply say: root -l <macro.C>), which can be conveniently invoked through a GUI that will appear at the end of the run of this macro. Launch the GUI via the command:

root -l -e 'TMVA::TMVAGui("TMVA.root")'

Cross Evaluation

Cross evaluation is a special case of k-folds cross validation where the splitting into k folds is computed deterministically. This ensures that the a given event will always end up in the same fold.

In addition all resulting classifiers are saved and can be applied to new data using MethodCrossValidation. One requirement for this to work is a splitting function that is evaluated for each event to determine into what fold it goes (for training/evaluation) or to what classifier (for application).

Split Expression

Cross evaluation uses a deterministic split to partition the data into folds called the split expression. The expression can be any valid TFormula as long as all parts used are defined.

For each event the split expression is evaluated to a number and the event is put in the fold corresponding to that number.

It is recommended to always use %int([NumFolds]) at the end of the expression.

The split expression has access to all spectators and variables defined in the dataloader. Additionally, the number of folds in the split can be accessed with NumFolds (or numFolds).

Example

 "int(fabs([eventID]))%int([NumFolds])"
  • Project : TMVA - a ROOT-integrated toolkit for multivariate data analysis
  • Package : TMVA
  • Root Macro: TMVACrossValidation

Author: Kim Albertsson (adapted from code originally by Andreas Hoecker)
This notebook tutorial was automatically generated with ROOTBOOK-izer from the macro found in the ROOT repository on Monday, December 06, 2021 at 10:50 AM.

In [1]:
%%cpp -d
#include <cstdlib>
#include <iostream>
#include <map>
#include <string>

#include "TChain.h"
#include "TFile.h"
#include "TTree.h"
#include "TString.h"
#include "TObjString.h"
#include "TSystem.h"
#include "TROOT.h"

#include "TMVA/CrossValidation.h"
#include "TMVA/DataLoader.h"
#include "TMVA/Factory.h"
#include "TMVA/Tools.h"
#include "TMVA/TMVAGui.h"

Helper function to load data into TTrees.

In [2]:
%%cpp -d
TTree *genTree(Int_t nPoints, Double_t offset, Double_t scale, UInt_t seed = 100)
{
   TRandom3 rng(seed);
   Float_t x = 0;
   Float_t y = 0;
   UInt_t eventID = 0;

   TTree *data = new TTree();
   data->Branch("x", &x, "x/F");
   data->Branch("y", &y, "y/F");
   data->Branch("eventID", &eventID, "eventID/I");

   for (Int_t n = 0; n < nPoints; ++n) {
      x = rng.Gaus(offset, scale);
      y = rng.Gaus(offset, scale);

      // For our simple example it is enough that the id's are uniformly
      // distributed and independent of the data.
      ++eventID;

      data->Fill();
   }

   // Important: Disconnects the tree from the memory locations of x and y.
   data->ResetBranchAddresses();
   return data;
}

Arguments are defined.

In [3]:
bool useRandomSplitting = false;

This loads the library

In [4]:
TMVA::Tools::Instance();

Load the data into ttrees. if you load data from file you can use a variant of

 TString filename = "/path/to/file";
 TFile * input = TFile::Open( filename );
 TTree * signalTree = (TTree*)input->Get("TreeName");
In [5]:
TTree *sigTree = genTree(1000, 1.0, 1.0, 100);
TTree *bkgTree = genTree(1000, -1.0, 1.0, 101);

Create a root output file where tmva will store ntuples, histograms, etc.

In [6]:
TString outfileName("TMVA.root");
TFile *outputFile = TFile::Open(outfileName, "RECREATE");

Dataloader definitions; we declare variables in the tree so that tmva can find them. For more information see TMVAClassification tutorial.

In [7]:
TMVA::DataLoader *dataloader = new TMVA::DataLoader("dataset");

Data variables

In [8]:
dataloader->AddVariable("x", 'F');
dataloader->AddVariable("y", 'F');

Spectator used for split

In [9]:
dataloader->AddSpectator("eventID", 'I');

Note: currently tmva treats all input variables, spectators etc as floats. Thus, if the absolute value of the input is too large there can be precision loss. This can especially be a problem for cross validation with large event numbers. A workaround is to define your splitting variable as: dataloader->AddSpectator("eventID := eventID % 4096", 'I'); where 4096 should be a number much larger than the number of folds you intend to run with.

Attaches the trees so they can be read from

In [10]:
dataloader->AddSignalTree(sigTree, 1.0);
dataloader->AddBackgroundTree(bkgTree, 1.0);
DataSetInfo              : [dataset] : Added class "Signal"
                         : Add Tree  of type Signal with 1000 events
DataSetInfo              : [dataset] : Added class "Background"
                         : Add Tree  of type Background with 1000 events

The cv mechanism of tmva splits up the training set into several folds. The test set is currently left unused. The nTest_ClassName=1 assigns one event to the the test set for each class and puts the rest in the training set. A value of 0 is a special value and would split the datasets 50 / 50.

In [11]:
dataloader->PrepareTrainingAndTestTree("", "",
                                       "nTest_Signal=1"
                                       ":nTest_Background=1"
                                       ":SplitMode=Random"
                                       ":NormMode=NumEvents"
                                       ":!V");

This sets up a CrossValidation class (which wraps a TMVA::Factory internally) for 2-fold cross validation.

The split type can be "Random", "RandomStratified" or "Deterministic". For the last option, check the comment below. Random splitting randomises the order of events and distributes events as evenly as possible. RandomStratified applies the same logic but distributes events within a class as evenly as possible over the folds.

In [12]:
UInt_t numFolds = 2;
TString analysisType = "Classification";

TString splitType = (useRandomSplitting) ? "Random" : "Deterministic";

One can also use a custom splitting function for producing the folds. The example uses a dataset spectator eventID.

The idea here is that eventID should be an event number that is integral, random and independent of the data, generated only once. This last property ensures that if a calibration is changed the same event will still be assigned the same fold.

This can be used to use the cross validated classifiers in application, a technique that can simplify statistical analysis.

If you want to run TMVACrossValidationApplication, make sure you have run this tutorial with Deterministic splitting type, i.e. with the option useRandomSPlitting = false

In [13]:
TString splitExpr = (!useRandomSplitting) ? "int(fabs([eventID]))%int([NumFolds])" : "";

TString cvOptions = Form("!V"
                         ":!Silent"
                         ":ModelPersistence"
                         ":AnalysisType=%s"
                         ":SplitType=%s"
                         ":NumFolds=%i"
                         ":SplitExpr=%s",
                         analysisType.Data(), splitType.Data(), numFolds,
                         splitExpr.Data());

TMVA::CrossValidation cv{"TMVACrossValidation", dataloader, outputFile, cvOptions};
<HEADER> Factory                  : You are running ROOT Version: 6.25/01, Mar 3, 2021
                         : 
                         : _/_/_/_/_/ _|      _|  _|      _|    _|_|   
                         :    _/      _|_|  _|_|  _|      _|  _|    _| 
                         :   _/       _|  _|  _|  _|      _|  _|_|_|_| 
                         :  _/        _|      _|    _|  _|    _|    _| 
                         : _/         _|      _|      _|      _|    _| 
                         : 
                         : ___________TMVA Version 4.2.1, Feb 5, 2015
                         : 

Books a method to use for evaluation

In [14]:
cv.BookMethod(TMVA::Types::kBDT, "BDTG",
              "!H:!V:NTrees=100:MinNodeSize=2.5%:BoostType=Grad"
              ":NegWeightTreatment=Pray:Shrinkage=0.10:nCuts=20"
              ":MaxDepth=2");

cv.BookMethod(TMVA::Types::kFisher, "Fisher",
              "!H:!V:Fisher:VarTransform=None");

Train, test and evaluate the booked methods. Evaluates the booked methods once for each fold and aggregates the result in the specified output file.

In [15]:
cv.Evaluate();
                         : Rebuilding Dataset dataset
                         : Building event vectors for type 2 Signal
                         : Dataset[dataset] :  create input formulas for tree 
                         : Building event vectors for type 2 Background
                         : Dataset[dataset] :  create input formulas for tree 
<HEADER> DataSetFactory           : [dataset] : Number of events in input trees
                         : 
                         : 
                         : Number of training and testing events
                         : ---------------------------------------------------------------------------
                         : Signal     -- training events            : 999
                         : Signal     -- testing events             : 1
                         : Signal     -- training and testing events: 1000
                         : Background -- training events            : 999
                         : Background -- testing events             : 1
                         : Background -- training and testing events: 1000
                         : 
<HEADER> DataSetInfo              : Correlation matrix (Signal):
                         : ------------------------
                         :                x       y
                         :       x:  +1.000  +0.075
                         :       y:  +0.075  +1.000
                         : ------------------------
<HEADER> DataSetInfo              : Correlation matrix (Background):
                         : ------------------------
                         :                x       y
                         :       x:  +1.000  +0.020
                         :       y:  +0.020  +1.000
                         : ------------------------
<HEADER> DataSetFactory           : [dataset] :  
                         : 
                         : 
                         : 
                         : ========================================
                         : Processing folds for method BDTG
                         : ========================================
                         : 
<HEADER> Factory                  : Booking method: BDTG_fold1
                         : 
<HEADER> BDTG_fold1               : #events: (reweighted) sig: 500 bkg: 500
                         : #events: (unweighted) sig: 500 bkg: 500
                         : Training 100 Decision Trees ... patience please
                         : Elapsed time for training with 1000 events: 0.0447 sec         
<HEADER> BDTG_fold1               : [dataset] : Evaluation of BDTG_fold1 on training sample (1000 events)
                         : Elapsed time for evaluation of 1000 events: 0.00332 sec       
                         : Creating xml weight file: dataset/weights/TMVACrossValidation_BDTG_fold1.weights.xml
                         : Creating standalone class: dataset/weights/TMVACrossValidation_BDTG_fold1.class.C
<HEADER> Factory                  : Test all methods
<HEADER> Factory                  : Test method: BDTG_fold1 for Classification performance
                         : 
<HEADER> BDTG_fold1               : [dataset] : Evaluation of BDTG_fold1 on testing sample (998 events)
                         : Elapsed time for evaluation of 998 events: 0.00325 sec       
<HEADER> Factory                  : Evaluate all methods
<HEADER> Factory                  : Evaluate classifier: BDTG_fold1
                         : 
<HEADER> BDTG_fold1               : [dataset] : Loop over test events and fill histograms with classifier response...
                         : 
                         : 
                         : Evaluation results ranked by best signal efficiency and purity (area)
                         : -------------------------------------------------------------------------------------------------------------------
                         : DataSet       MVA                       
                         : Name:         Method:          ROC-integ
                         : dataset       BDTG_fold1     : 0.973
                         : -------------------------------------------------------------------------------------------------------------------
                         : 
                         : Testing efficiency compared to training efficiency (overtraining check)
                         : -------------------------------------------------------------------------------------------------------------------
                         : DataSet              MVA              Signal efficiency: from test sample (from training sample) 
                         : Name:                Method:          @B=0.01             @B=0.10            @B=0.30   
                         : -------------------------------------------------------------------------------------------------------------------
                         : dataset              BDTG_fold1     : 0.575 (0.725)       0.947 (0.933)      0.981 (0.980)
                         : -------------------------------------------------------------------------------------------------------------------
                         : 
<HEADER> Factory                  : Thank you for using TMVA!
                         : For citation information, please visit: http://tmva.sf.net/citeTMVA.html
<HEADER> Factory                  : Booking method: BDTG_fold2
                         : 
<HEADER> BDTG_fold2               : #events: (reweighted) sig: 499 bkg: 499
                         : #events: (unweighted) sig: 499 bkg: 499
                         : Training 100 Decision Trees ... patience please
                         : Elapsed time for training with 998 events: 0.0449 sec         
<HEADER> BDTG_fold2               : [dataset] : Evaluation of BDTG_fold2 on training sample (998 events)
                         : Elapsed time for evaluation of 998 events: 0.00335 sec       
                         : Creating xml weight file: dataset/weights/TMVACrossValidation_BDTG_fold2.weights.xml
                         : Creating standalone class: dataset/weights/TMVACrossValidation_BDTG_fold2.class.C
<HEADER> Factory                  : Test all methods
<HEADER> Factory                  : Test method: BDTG_fold2 for Classification performance
                         : 
<HEADER> BDTG_fold2               : [dataset] : Evaluation of BDTG_fold2 on testing sample (1000 events)
                         : Elapsed time for evaluation of 1000 events: 0.00337 sec       
<HEADER> Factory                  : Evaluate all methods
<HEADER> Factory                  : Evaluate classifier: BDTG_fold2
                         : 
<HEADER> BDTG_fold2               : [dataset] : Loop over test events and fill histograms with classifier response...
                         : 
                         : 
                         : Evaluation results ranked by best signal efficiency and purity (area)
                         : -------------------------------------------------------------------------------------------------------------------
                         : DataSet       MVA                       
                         : Name:         Method:          ROC-integ
                         : dataset       BDTG_fold2     : 0.961
                         : -------------------------------------------------------------------------------------------------------------------
                         : 
                         : Testing efficiency compared to training efficiency (overtraining check)
                         : -------------------------------------------------------------------------------------------------------------------
                         : DataSet              MVA              Signal efficiency: from test sample (from training sample) 
                         : Name:                Method:          @B=0.01             @B=0.10            @B=0.30   
                         : -------------------------------------------------------------------------------------------------------------------
                         : dataset              BDTG_fold2     : 0.646 (0.696)       0.868 (0.930)      0.975 (0.976)
                         : -------------------------------------------------------------------------------------------------------------------
                         : 
<HEADER> Factory                  : Thank you for using TMVA!
                         : For citation information, please visit: http://tmva.sf.net/citeTMVA.html
<HEADER> Factory                  : Booking method: BDTG
                         : 
                         : Reading weightfile: dataset/weights/TMVACrossValidation_BDTG_fold1.weights.xml
                         : Reading weight file: dataset/weights/TMVACrossValidation_BDTG_fold1.weights.xml
                         : Reading weightfile: dataset/weights/TMVACrossValidation_BDTG_fold2.weights.xml
                         : Reading weight file: dataset/weights/TMVACrossValidation_BDTG_fold2.weights.xml
                         : 
                         : 
                         : ========================================
                         : Processing folds for method Fisher
                         : ========================================
                         : 
<HEADER> Factory                  : Booking method: Fisher_fold1
                         : 
<HEADER> Fisher_fold1             : Results for Fisher coefficients:
                         : -----------------------
                         : Variable:  Coefficient:
                         : -----------------------
                         :        x:       +0.449
                         :        y:       +0.436
                         : (offset):       +0.019
                         : -----------------------
                         : Elapsed time for training with 1000 events: 0.00047 sec         
<HEADER> Fisher_fold1             : [dataset] : Evaluation of Fisher_fold1 on training sample (1000 events)
                         : Elapsed time for evaluation of 1000 events: 7.61e-05 sec       
                         : Creating xml weight file: dataset/weights/TMVACrossValidation_Fisher_fold1.weights.xml
                         : Creating standalone class: dataset/weights/TMVACrossValidation_Fisher_fold1.class.C
<HEADER> Factory                  : Test all methods
<HEADER> Factory                  : Test method: Fisher_fold1 for Classification performance
                         : 
<HEADER> Fisher_fold1             : [dataset] : Evaluation of Fisher_fold1 on testing sample (998 events)
                         : Elapsed time for evaluation of 998 events: 0.000151 sec       
<HEADER> Factory                  : Evaluate all methods
<HEADER> Factory                  : Evaluate classifier: Fisher_fold1
                         : 
<HEADER> Fisher_fold1             : [dataset] : Loop over test events and fill histograms with classifier response...
                         : 
                         : 
                         : Evaluation results ranked by best signal efficiency and purity (area)
                         : -------------------------------------------------------------------------------------------------------------------
                         : DataSet       MVA                       
                         : Name:         Method:          ROC-integ
                         : dataset       Fisher_fold1   : 0.976
                         : -------------------------------------------------------------------------------------------------------------------
                         : 
                         : Testing efficiency compared to training efficiency (overtraining check)
                         : -------------------------------------------------------------------------------------------------------------------
                         : DataSet              MVA              Signal efficiency: from test sample (from training sample) 
                         : Name:                Method:          @B=0.01             @B=0.10            @B=0.30   
                         : -------------------------------------------------------------------------------------------------------------------
                         : dataset              Fisher_fold1   : 0.660 (0.665)       0.952 (0.923)      0.986 (0.985)
                         : -------------------------------------------------------------------------------------------------------------------
                         : 
<HEADER> Factory                  : Thank you for using TMVA!
                         : For citation information, please visit: http://tmva.sf.net/citeTMVA.html
<HEADER> Factory                  : Booking method: Fisher_fold2
                         : 
<HEADER> Fisher_fold2             : Results for Fisher coefficients:
                         : -----------------------
                         : Variable:  Coefficient:
                         : -----------------------
                         :        x:       +0.501
                         :        y:       +0.467
                         : (offset):       -0.000
                         : -----------------------
                         : Elapsed time for training with 998 events: 0.000273 sec         
<HEADER> Fisher_fold2             : [dataset] : Evaluation of Fisher_fold2 on training sample (998 events)
                         : Elapsed time for evaluation of 998 events: 7.1e-05 sec       
                         : Creating xml weight file: dataset/weights/TMVACrossValidation_Fisher_fold2.weights.xml
                         : Creating standalone class: dataset/weights/TMVACrossValidation_Fisher_fold2.class.C
<HEADER> Factory                  : Test all methods
<HEADER> Factory                  : Test method: Fisher_fold2 for Classification performance
                         : 
<HEADER> Fisher_fold2             : [dataset] : Evaluation of Fisher_fold2 on testing sample (1000 events)
                         : Elapsed time for evaluation of 1000 events: 0.000123 sec       
<HEADER> Factory                  : Evaluate all methods
<HEADER> Factory                  : Evaluate classifier: Fisher_fold2
                         : 
<HEADER> Fisher_fold2             : [dataset] : Loop over test events and fill histograms with classifier response...
                         : 
                         : 
                         : Evaluation results ranked by best signal efficiency and purity (area)
                         : -------------------------------------------------------------------------------------------------------------------
                         : DataSet       MVA                       
                         : Name:         Method:          ROC-integ
                         : dataset       Fisher_fold2   : 0.966
                         : -------------------------------------------------------------------------------------------------------------------
                         : 
                         : Testing efficiency compared to training efficiency (overtraining check)
                         : -------------------------------------------------------------------------------------------------------------------
                         : DataSet              MVA              Signal efficiency: from test sample (from training sample) 
                         : Name:                Method:          @B=0.01             @B=0.10            @B=0.30   
                         : -------------------------------------------------------------------------------------------------------------------
                         : dataset              Fisher_fold2   : 0.655 (0.645)       0.900 (0.928)      0.975 (0.977)
                         : -------------------------------------------------------------------------------------------------------------------
                         : 
<HEADER> Factory                  : Thank you for using TMVA!
                         : For citation information, please visit: http://tmva.sf.net/citeTMVA.html
<HEADER> Factory                  : Booking method: Fisher
                         : 
                         : Reading weightfile: dataset/weights/TMVACrossValidation_Fisher_fold1.weights.xml
                         : Reading weight file: dataset/weights/TMVACrossValidation_Fisher_fold1.weights.xml
                         : Reading weightfile: dataset/weights/TMVACrossValidation_Fisher_fold2.weights.xml
                         : Reading weight file: dataset/weights/TMVACrossValidation_Fisher_fold2.weights.xml
                         : 
                         : 
                         : ========================================
                         : Folds processed for all methods, evaluating.
                         : ========================================
                         : 
<HEADER> Factory                  : [dataset] : Create Transformation "I" with events from all classes.
                         : 
<HEADER>                          : Transformation, Variable selection : 
                         : Input : variable 'x' <---> Output : variable 'x'
                         : Input : variable 'y' <---> Output : variable 'y'
<HEADER> TFHandler_Factory        : Variable        Mean        RMS   [        Min        Max ]
                         : -----------------------------------------------------------
                         :        x:  -0.014284     1.4061   [    -4.1075     4.0969 ]
                         :        y: -0.0066370     1.4204   [    -4.8520     4.0761 ]
                         : -----------------------------------------------------------
                         : Ranking input variables (method unspecific)...
<HEADER> IdTransformation         : Ranking result (top variable is best ranked)
                         : --------------------------
                         : Rank : Variable  : Separation
                         : --------------------------
                         :    1 : x         : 5.429e-01
                         :    2 : y         : 5.230e-01
                         : --------------------------
                         : Elapsed time for training with 1998 events: 5.72e-06 sec         
<HEADER> BDTG                     : [dataset] : Evaluation of BDTG on training sample (1998 events)
                         : Elapsed time for evaluation of 1998 events: 0.00635 sec       
                         : Creating xml weight file: dataset/weights/TMVACrossValidation_BDTG.weights.xml
                         : Creating standalone class: dataset/weights/TMVACrossValidation_BDTG.class.C
<WARNING> <WARNING>                : MakeClassSpecificHeader not implemented for CrossValidation
<WARNING> <WARNING>                : MakeClassSpecific not implemented for CrossValidation
                         : Elapsed time for training with 1998 events: 4.05e-06 sec         
<HEADER> Fisher                   : [dataset] : Evaluation of Fisher on training sample (1998 events)
                         : Elapsed time for evaluation of 1998 events: 0.000346 sec       
                         : Creating xml weight file: dataset/weights/TMVACrossValidation_Fisher.weights.xml
                         : Creating standalone class: dataset/weights/TMVACrossValidation_Fisher.class.C
<WARNING> <WARNING>                : MakeClassSpecificHeader not implemented for CrossValidation
<WARNING> <WARNING>                : MakeClassSpecific not implemented for CrossValidation
<HEADER> Factory                  : Test all methods
<HEADER> Factory                  : Test method: BDTG for Classification performance
                         : 
<HEADER> BDTG                     : [dataset] : Evaluation of BDTG on testing sample (1998 events)
                         : Elapsed time for evaluation of 1998 events: 0.0061 sec       
<HEADER> Factory                  : Test method: Fisher for Classification performance
                         : 
<HEADER> Fisher                   : [dataset] : Evaluation of Fisher on testing sample (1998 events)
                         : Elapsed time for evaluation of 1998 events: 0.00031 sec       
<HEADER> Factory                  : Evaluate all methods
<HEADER> Factory                  : Evaluate classifier: BDTG
                         : 
<HEADER> BDTG                     : [dataset] : Loop over test events and fill histograms with classifier response...
                         : 
<HEADER> TFHandler_BDTG           : Variable        Mean        RMS   [        Min        Max ]
                         : -----------------------------------------------------------
                         :        x:  -0.014284     1.4061   [    -4.1075     4.0969 ]
                         :        y: -0.0066370     1.4204   [    -4.8520     4.0761 ]
                         : -----------------------------------------------------------
<HEADER> Factory                  : Evaluate classifier: Fisher
                         : 
<HEADER> Fisher                   : [dataset] : Loop over test events and fill histograms with classifier response...
                         : 
<HEADER> TFHandler_Fisher         : Variable        Mean        RMS   [        Min        Max ]
                         : -----------------------------------------------------------
                         :        x:  -0.014284     1.4061   [    -4.1075     4.0969 ]
                         :        y: -0.0066370     1.4204   [    -4.8520     4.0761 ]
                         : -----------------------------------------------------------
                         : 
                         : Evaluation results ranked by best signal efficiency and purity (area)
                         : -------------------------------------------------------------------------------------------------------------------
                         : DataSet       MVA                       
                         : Name:         Method:          ROC-integ
                         : dataset       Fisher         : 0.971
                         : dataset       BDTG           : 0.965
                         : -------------------------------------------------------------------------------------------------------------------
                         : 
                         : Testing efficiency compared to training efficiency (overtraining check)
                         : -------------------------------------------------------------------------------------------------------------------
                         : DataSet              MVA              Signal efficiency: from test sample (from training sample) 
                         : Name:                Method:          @B=0.01             @B=0.10            @B=0.30   
                         : -------------------------------------------------------------------------------------------------------------------
                         : dataset              Fisher         : 0.665 (0.665)       0.922 (0.922)      0.980 (0.980)
                         : dataset              BDTG           : 0.617 (0.617)       0.914 (0.914)      0.974 (0.974)
                         : -------------------------------------------------------------------------------------------------------------------
                         : 
<HEADER> Dataset:dataset          : Created tree 'TestTree' with 1998 events
                         : 
<HEADER> Dataset:dataset          : Created tree 'TrainTree' with 1998 events
                         : 
<HEADER> Factory                  : Thank you for using TMVA!
                         : For citation information, please visit: http://tmva.sf.net/citeTMVA.html
                         : Evaluation done.

Process some output programatically, printing the ROC score for each booked method.

In [16]:
size_t iMethod = 0;
for (auto && result : cv.GetResults()) {
   std::cout << "Summary for method " << cv.GetMethods()[iMethod++].GetValue<TString>("MethodName")
             << std::endl;
   for (UInt_t iFold = 0; iFold<cv.GetNumFolds(); ++iFold) {
      std::cout << "\tFold " << iFold << ": "
                << "ROC int: " << result.GetROCValues()[iFold]
                << ", "
                << "[email protected]=0.3: " << result.GetEff30Values()[iFold]
                << std::endl;
   }
}
Summary for method BDT
	Fold 0: ROC int: 0.972504, [email protected]=0.3: 0.981
	Fold 1: ROC int: 0.96115, [email protected]=0.3: 0.975
Summary for method Fisher
	Fold 0: ROC int: 0.976137, [email protected]=0.3: 0.986
	Fold 1: ROC int: 0.96584, [email protected]=0.3: 0.975

Save the output

In [17]:
outputFile->Close();

std::cout << "==> Wrote root file: " << outputFile->GetName() << std::endl;
std::cout << "==> TMVACrossValidation is done!" << std::endl;
==> Wrote root file: TMVA.root
==> TMVACrossValidation is done!

Launch the GUI for the root macros

In [18]:
if (!gROOT->IsBatch()) {
   // Draw cv-specific graphs
   cv.GetResults()[0].DrawAvgROCCurve(kTRUE, "Avg ROC for BDTG");
   cv.GetResults()[0].DrawAvgROCCurve(kTRUE, "Avg ROC for Fisher");

   // You can also use the classical gui
   TMVA::TMVAGui(outfileName);
}

return 0;