TMVA_CNN_Classification¶
TMVA Classification Example Using a Convolutional Neural Network
This is an example of using a CNN in TMVA. We do classification using a toy image data set that is generated when running the example macro
Author: Harshal Shende
This notebook tutorial was automatically generated with ROOTBOOK-izer from the macro found in the ROOT repository on Monday, March 27, 2023 at 09:58 AM.
TMVA Classification Example Using a Convolutional Neural Network
Helper function to create input images data we create a signal and background 2D histograms from 2d gaussians with a location (means in X and Y) different for each event The difference between signal and background is in the gaussian width. The width for the background gaussian is slightly larger than the signal width by few % values
In [1]:
import ROOT
Welcome to JupyROOT 6.29/01
witch off MT in OpenMP (BLAS)
In [2]:
ROOT.gSystem.Setenv("OMP_NUM_THREADS", "1")
TMVA = ROOT.TMVA
TFile = ROOT.TFile
import os
import importlib
TMVA.Tools.Instance()
def MakeImagesTree(n, nh, nw):
# image size (nh x nw)
ntot = nh * nw
fileOutName = "images_data_16x16.root"
nRndmEvts = 10000 # number of events we use to fill each image
delta_sigma = 0.1 # 5% difference in the sigma
pixelNoise = 5
sX1 = 3
sY1 = 3
sX2 = sX1 + delta_sigma
sY2 = sY1 - delta_sigma
h1 = ROOT.TH2D("h1", "h1", nh, 0, 10, nw, 0, 10)
h2 = ROOT.TH2D("h2", "h2", nh, 0, 10, nw, 0, 10)
f1 = ROOT.TF2("f1", "xygaus")
f2 = ROOT.TF2("f2", "xygaus")
sgn = ROOT.TTree("sig_tree", "signal_tree")
bkg = ROOT.TTree("bkg_tree", "background_tree")
f = TFile(fileOutName, "RECREATE")
x1 = ROOT.std.vector["float"](ntot)
x2 = ROOT.std.vector["float"](ntot)
# create signal and background trees with a single branch
# an std::vector<float> of size nh x nw containing the image data
bkg.Branch("vars", "std::vector<float>", x1)
sgn.Branch("vars", "std::vector<float>", x2)
sgn.SetDirectory(f)
bkg.SetDirectory(f)
f1.SetParameters(1, 5, sX1, 5, sY1)
f2.SetParameters(1, 5, sX2, 5, sY2)
ROOT.gRandom.SetSeed(0)
ROOT.Info("TMVA_CNN_Classification", "Filling ROOT tree \n")
for i in range(n):
if i % 1000 == 0:
print("Generating image event ...", i)
h1.Reset()
h2.Reset()
# generate random means in range [3,7] to be not too much on the border
f1.SetParameter(1, ROOT.gRandom.Uniform(3, 7))
f1.SetParameter(3, ROOT.gRandom.Uniform(3, 7))
f2.SetParameter(1, ROOT.gRandom.Uniform(3, 7))
f2.SetParameter(3, ROOT.gRandom.Uniform(3, 7))
h1.FillRandom("f1", nRndmEvts)
h2.FillRandom("f2", nRndmEvts)
for k in range(nh):
for l in range(nw):
m = k * nw + l
# add some noise in each bin
x1[m] = h1.GetBinContent(k + 1, l + 1) + ROOT.gRandom.Gaus(0, pixelNoise)
x2[m] = h2.GetBinContent(k + 1, l + 1) + ROOT.gRandom.Gaus(0, pixelNoise)
sgn.Fill()
bkg.Fill()
sgn.Write()
bkg.Write()
print("Signal and background tree with images data written to the file %s", f.GetName())
sgn.Print()
bkg.Print()
f.Close()
hasGPU = ROOT.gSystem.GetFromPipe("root-config --has-tmva-gpu") == "yes"
hasCPU = ROOT.gSystem.GetFromPipe("root-config --has-tmva-cpu") == "yes"
nevt = 1000 # use a larger value to get better results
opt = [1, 1, 1, 1, 1]
useTMVACNN = opt[0] if len(opt) > 0 else False
useKerasCNN = opt[1] if len(opt) > 1 else False
useTMVADNN = opt[2] if len(opt) > 2 else False
useTMVABDT = opt[3] if len(opt) > 3 else False
usePyTorchCNN = opt[4] if len(opt) > 4 else False
if (not hasCPU and not hasGPU) :
ROOT.Warning("TMVA_CNN_Classificaton","ROOT is not supporting tmva-cpu and tmva-gpu skip using TMVA-DNN and TMVA-CNN")
useTMVACNN = False
useTMVADNN = False
if ROOT.gSystem.GetFromPipe("root-config --has-tmva-pymva") != "yes":
useKerasCNN = False
usePyTorchCNN = False
else:
TMVA.PyMethodBase.PyInitialize()
tf_spec = importlib.util.find_spec("tensorflow")
if tf_spec is None:
useKerasCNN = False
ROOT.Warning("TMVA_CNN_Classificaton","Skip using Keras since tensorflow is not installed")
torch_spec = importlib.util.find_spec("torch")
if torch_spec is None:
usePyTorchCNN = False
ROOT.Warning("TMVA_CNN_Classificaton","Skip using PyTorch since torch is not installed")
if not useTMVACNN:
ROOT.Warning(
"TMVA_CNN_Classificaton",
"TMVA is not build with GPU or CPU multi-thread support. Cannot use TMVA Deep Learning for CNN",
)
writeOutputFile = True
num_threads = 4 # use default threads
max_epochs = 10 # maximum number of epochs used for training
do enable MT running
In [3]:
if num_threads >= 0:
ROOT.EnableImplicitMT(num_threads)
print("Running with nthreads = ", ROOT.GetThreadPoolSize())
outputFile = None
if writeOutputFile:
outputFile = TFile.Open("TMVA_CNN_ClassificationOutput.root", "RECREATE")
Running with nthreads = 4
Create TMVA Factory
Create the Factory class. Later you can choose the methods whose performance you'd like to investigate.
The factory is the major TMVA object you have to interact with. Here is the list of parameters you need to pass
- The first argument is the base of the name of all the output weight files in the directory weight/ that will be created with the method parameters
- The second argument is the output file for the training results
- The third argument is a string option defining some general configuration for the TMVA session.
For example all TMVA output can be suppressed by removing the "!" (not) in front of the "Silent" argument in the option string
- note that we disable any pre-transformation of the input variables and we avoid computing correlations between
input variables
In [4]:
factory = TMVA.Factory(
"TMVA_CNN_Classification",
outputFile,
V=False,
ROC=True,
Silent=False,
Color=True,
AnalysisType="Classification",
Transformations=None,
Correlations=False,
)
Declare DataLoader(s)
The next step is to declare the DataLoader class that deals with input variables
Define the input variables that shall be used for the MVA training note that you may also use variable expressions, which can be parsed by TTree::Draw( "expression" )]
In this case the input data consists of an image of 16x16 pixels. Each single pixel is a branch in a ROOT TTree
In [5]:
loader = TMVA.DataLoader("dataset")
Setup Dataset(s)
Define input data file and signal and background trees
In [6]:
imgSize = 16 * 16
inputFileName = "images_data_16x16.root"
if the input file does not exist create it
In [7]:
if ROOT.gSystem.AccessPathName(inputFileName):
MakeImagesTree(nevt, 16, 16)
inputFile = TFile.Open(inputFileName)
if inputFile is None:
ROOT.Warning("TMVA_CNN_Classification", "Error opening input file %s - exit", inputFileName.Data())
Generating image event ... 0 Signal and background tree with images data written to the file %s images_data_16x16.root ****************************************************************************** *Tree :sig_tree : signal_tree * *Entries : 1000 : Total = 1042365 bytes File Size = 932185 * * : : Tree compression factor = 1.12 * ****************************************************************************** *Br 0 :vars : vector<float> * *Entries : 1000 : Total Size= 1041947 bytes File Size = 931466 * *Baskets : 34 : Basket Size= 32000 bytes Compression= 1.12 * *............................................................................* ****************************************************************************** *Tree :bkg_tree : background_tree * *Entries : 1000 : Total = 1042373 bytes File Size = 932100 * * : : Tree compression factor = 1.12 * ****************************************************************************** *Br 0 :vars : vector<float> * *Entries : 1000 : Total Size= 1041947 bytes File Size = 931371 * *Baskets : 34 : Basket Size= 32000 bytes Compression= 1.12 * *............................................................................*
Info in <TMVA_CNN_Classification>: Filling ROOT tree
inputFileName = "tmva_class_example.root"
--- Register the training and test trees
In [8]:
signalTree = inputFile.Get("sig_tree")
backgroundTree = inputFile.Get("bkg_tree")
nEventsSig = signalTree.GetEntries()
nEventsBkg = backgroundTree.GetEntries()
global event weights per tree (see below for setting event-wise weights)
In [9]:
signalWeight = 1.0
backgroundWeight = 1.0
You can add an arbitrary number of signal or background trees
In [10]:
loader.AddSignalTree(signalTree, signalWeight)
loader.AddBackgroundTree(backgroundTree, backgroundWeight)
DataSetInfo : [dataset] : Added class "Signal"
: Add Tree sig_tree of type Signal with 1000 events
DataSetInfo : [dataset] : Added class "Background"
: Add Tree bkg_tree of type Background with 1000 events
add event variables (image) use new method (from ROOT 6.20 to add a variable array for all image data)
In [11]:
loader.AddVariablesArray("vars", imgSize)
Set individual event weights (the variables must exist in the original TTree) for signal : factory->SetSignalWeightExpression ("weight1weight2"); for background: factory->SetBackgroundWeightExpression("weight1weight2"); loader->SetBackgroundWeightExpression( "weight" );
Apply additional cuts on the signal and background samples (can be different)
In [12]:
mycuts = "" # for example: TCut mycuts = "abs(var1)<0.5 && abs(var2-0.5)<1";
mycutb = "" # for example: TCut mycutb = "abs(var1)<0.5";
Tell the factory how to use the training and testing events If no numbers of events are given, half of the events in the tree are used for training, and the other half for testing: loader.PrepareTrainingAndTestTree( mycut, "SplitMode=random:!V" ); It is possible also to specify the number of training and testing events, note we disable the computation of the correlation matrix of the input variables
In [13]:
nTrainSig = 0.8 * nEventsSig
nTrainBkg = 0.8 * nEventsBkg
build the string options for DataLoader::PrepareTrainingAndTestTree
In [14]:
loader.PrepareTrainingAndTestTree(
mycuts,
mycutb,
nTrain_Signal=nTrainSig,
nTrain_Background=nTrainBkg,
SplitMode="Random",
SplitSeed=100,
NormMode="NumEvents",
V=False,
CalcCorrelations=False,
)
DataSetInfo : [dataset] : Added class "Signal" : Add Tree sig_tree of type Signal with 10000 events DataSetInfo : [dataset] : Added class "Background" : Add Tree bkg_tree of type Background with 10000 events
signalTree.Print();
Booking Methods
Here we book the TMVA methods. We book a Boosted Decision Tree method (BDT)
Boosted Decision Trees
In [15]:
if useTMVABDT:
factory.BookMethod(
loader,
TMVA.Types.kBDT,
"BDT",
V=False,
NTrees=400,
MinNodeSize="2.5%",
MaxDepth=2,
BoostType="AdaBoost",
AdaBoostBeta=0.5,
UseBaggedBoost=True,
BaggedSampleFraction=0.5,
SeparationType="GiniIndex",
nCuts=20,
)
Factory : Booking method: BDT
:
: Rebuilding Dataset dataset
: Building event vectors for type 2 Signal
: Dataset[dataset] : create input formulas for tree sig_tree
: Using variable vars[0] from array expression vars of size 256
: Building event vectors for type 2 Background
: Dataset[dataset] : create input formulas for tree bkg_tree
: Using variable vars[0] from array expression vars of size 256
DataSetFactory : [dataset] : Number of events in input trees
:
:
: Number of training and testing events
: ---------------------------------------------------------------------------
: Signal -- training events : 800
: Signal -- testing events : 200
: Signal -- training and testing events: 1000
: Background -- training events : 800
: Background -- testing events : 200
: Background -- training and testing events: 1000
:
Booking Deep Neural Network¶
Here we book the DNN of TMVA. See the example TMVA_Higgs_Classification.C for a detailed description of the options
In [16]:
if useTMVADNN:
layoutString = ROOT.TString(
"DENSE|100|RELU,BNORM,DENSE|100|RELU,BNORM,DENSE|100|RELU,BNORM,DENSE|100|RELU,DENSE|1|LINEAR"
)
# Training strategies
# one can catenate several training strings with different parameters (e.g. learning rates or regularizations
# parameters) The training string must be concatenated with the `|` delimiter
trainingString1 = ROOT.TString(
"LearningRate=1e-3,Momentum=0.9,Repetitions=1,"
"ConvergenceSteps=5,BatchSize=100,TestRepetitions=1,"
"WeightDecay=1e-4,Regularization=None,"
"Optimizer=ADAM,DropConfig=0.0+0.0+0.0+0."
) # + "|" + trainingString2 + ...
trainingString1 += ",MaxEpochs=" + str(max_epochs)
# Build now the full DNN Option string
dnnMethodName = "TMVA_DNN_CPU"
# use GPU if available
dnnOptions = "CPU"
if hasGPU :
dnnOptions = "GPU"
dnnMethodName = "TMVA_DNN_GPU"
factory.BookMethod(
loader,
TMVA.Types.kDL,
dnnMethodName,
H=False,
V=True,
ErrorStrategy="CROSSENTROPY",
VarTransform=None,
WeightInitialization="XAVIER",
Layout=layoutString,
TrainingStrategy=trainingString1,
Architecture=dnnOptions
)
Factory : Booking method: TMVA_DNN_CPU
:
: Parsing option string:
: ... "!H:V:ErrorStrategy=CROSSENTROPY:VarTransform=None:WeightInitialization=XAVIER:Layout=DENSE|100|RELU,BNORM,DENSE|100|RELU,BNORM,DENSE|100|RELU,BNORM,DENSE|100|RELU,DENSE|1|LINEAR:TrainingStrategy=LearningRate=1e-3,Momentum=0.9,Repetitions=1,ConvergenceSteps=5,BatchSize=100,TestRepetitions=1,WeightDecay=1e-4,Regularization=None,Optimizer=ADAM,DropConfig=0.0+0.0+0.0+0.,MaxEpochs=10:Architecture=CPU"
: The following options are set:
: - By User:
: <none>
: - Default:
: Boost_num: "0" [Number of times the classifier will be boosted]
: Parsing option string:
: ... "!H:V:ErrorStrategy=CROSSENTROPY:VarTransform=None:WeightInitialization=XAVIER:Layout=DENSE|100|RELU,BNORM,DENSE|100|RELU,BNORM,DENSE|100|RELU,BNORM,DENSE|100|RELU,DENSE|1|LINEAR:TrainingStrategy=LearningRate=1e-3,Momentum=0.9,Repetitions=1,ConvergenceSteps=5,BatchSize=100,TestRepetitions=1,WeightDecay=1e-4,Regularization=None,Optimizer=ADAM,DropConfig=0.0+0.0+0.0+0.,MaxEpochs=10:Architecture=CPU"
: The following options are set:
: - By User:
: V: "True" [Verbose output (short form of "VerbosityLevel" below - overrides the latter one)]
: VarTransform: "None" [List of variable transformations performed before training, e.g., "D_Background,P_Signal,G,N_AllClasses" for: "Decorrelation, PCA-transformation, Gaussianisation, Normalisation, each for the given class of events ('AllClasses' denotes all events of all classes, if no class indication is given, 'All' is assumed)"]
: H: "False" [Print method-specific help message]
: Layout: "DENSE|100|RELU,BNORM,DENSE|100|RELU,BNORM,DENSE|100|RELU,BNORM,DENSE|100|RELU,DENSE|1|LINEAR" [Layout of the network.]
: ErrorStrategy: "CROSSENTROPY" [Loss function: Mean squared error (regression) or cross entropy (binary classification).]
: WeightInitialization: "XAVIER" [Weight initialization strategy]
: Architecture: "CPU" [Which architecture to perform the training on.]
: TrainingStrategy: "LearningRate=1e-3,Momentum=0.9,Repetitions=1,ConvergenceSteps=5,BatchSize=100,TestRepetitions=1,WeightDecay=1e-4,Regularization=None,Optimizer=ADAM,DropConfig=0.0+0.0+0.0+0.,MaxEpochs=10" [Defines the training strategies.]
: - Default:
: VerbosityLevel: "Default" [Verbosity level]
: CreateMVAPdfs: "False" [Create PDFs for classifier outputs (signal and background)]
: IgnoreNegWeightsInTraining: "False" [Events with negative weights are ignored in the training (but are included for testing and performance evaluation)]
: InputLayout: "0|0|0" [The Layout of the input]
: BatchLayout: "0|0|0" [The Layout of the batch]
: RandomSeed: "0" [Random seed used for weight initialization and batch shuffling]
: ValidationSize: "20%" [Part of the training data to use for validation. Specify as 0.2 or 20% to use a fifth of the data set as validation set. Specify as 100 to use exactly 100 events. (Default: 20%)]
: Will now use the CPU architecture with BLAS and IMT support !
Book Convolutional Neural Network in TMVA¶
For building a CNN one needs to define
- Input Layout : number of channels (in this case = 1) | image height | image width
- Batch Layout : batch size | number of channels | image size = (height*width)
Then one add Convolutional layers and MaxPool layers.
- For Convolutional layer the option string has to be:
- CONV | number of units | filter height | filter width | stride height | stride width | padding height | paddig
width | activation function
- note in this case we are using a filer 3x3 and padding=1 and stride=1 so we get the output dimension of the
conv layer equal to the input
- note we use after the first convolutional layer a batch normalization layer. This seems to help significantly the
convergence
- For the MaxPool layer:
- MAXPOOL | pool height | pool width | stride height | stride width
The RESHAPE layer is needed to flatten the output before the Dense layer
Note that to run the CNN is required to have CPU or GPU support
In [17]:
if useTMVACNN:
# Training strategies.
trainingString1 = ROOT.TString(
"LearningRate=1e-3,Momentum=0.9,Repetitions=1,"
"ConvergenceSteps=5,BatchSize=100,TestRepetitions=1,"
"WeightDecay=1e-4,Regularization=None,"
"Optimizer=ADAM,DropConfig=0.0+0.0+0.0+0.0"
)
trainingString1 += ",MaxEpochs=" + str(max_epochs)
## New DL (CNN)
cnnMethodName = "TMVA_CNN_CPU"
cnnOptions = "CPU"
# use GPU if available
if hasGPU:
cnnOptions = "GPU"
cnnMethodName = "TMVA_CNN_GPU"
factory.BookMethod(
loader,
TMVA.Types.kDL,
cnnMethodName,
H=False,
V=True,
ErrorStrategy="CROSSENTROPY",
VarTransform=None,
WeightInitialization="XAVIER",
InputLayout="1|16|16",
Layout="CONV|10|3|3|1|1|1|1|RELU,BNORM,CONV|10|3|3|1|1|1|1|RELU,MAXPOOL|2|2|1|1,RESHAPE|FLAT,DENSE|100|RELU,DENSE|1|LINEAR",
TrainingStrategy=trainingString1,
Architecture=cnnOptions,
)
Factory : Booking method: TMVA_CNN_CPU
:
: Parsing option string:
: ... "!H:V:ErrorStrategy=CROSSENTROPY:VarTransform=None:WeightInitialization=XAVIER:InputLayout=1|16|16:Layout=CONV|10|3|3|1|1|1|1|RELU,BNORM,CONV|10|3|3|1|1|1|1|RELU,MAXPOOL|2|2|1|1,RESHAPE|FLAT,DENSE|100|RELU,DENSE|1|LINEAR:TrainingStrategy=LearningRate=1e-3,Momentum=0.9,Repetitions=1,ConvergenceSteps=5,BatchSize=100,TestRepetitions=1,WeightDecay=1e-4,Regularization=None,Optimizer=ADAM,DropConfig=0.0+0.0+0.0+0.0,MaxEpochs=10:Architecture=CPU"
: The following options are set:
: - By User:
: <none>
: - Default:
: Boost_num: "0" [Number of times the classifier will be boosted]
: Parsing option string:
: ... "!H:V:ErrorStrategy=CROSSENTROPY:VarTransform=None:WeightInitialization=XAVIER:InputLayout=1|16|16:Layout=CONV|10|3|3|1|1|1|1|RELU,BNORM,CONV|10|3|3|1|1|1|1|RELU,MAXPOOL|2|2|1|1,RESHAPE|FLAT,DENSE|100|RELU,DENSE|1|LINEAR:TrainingStrategy=LearningRate=1e-3,Momentum=0.9,Repetitions=1,ConvergenceSteps=5,BatchSize=100,TestRepetitions=1,WeightDecay=1e-4,Regularization=None,Optimizer=ADAM,DropConfig=0.0+0.0+0.0+0.0,MaxEpochs=10:Architecture=CPU"
: The following options are set:
: - By User:
: V: "True" [Verbose output (short form of "VerbosityLevel" below - overrides the latter one)]
: VarTransform: "None" [List of variable transformations performed before training, e.g., "D_Background,P_Signal,G,N_AllClasses" for: "Decorrelation, PCA-transformation, Gaussianisation, Normalisation, each for the given class of events ('AllClasses' denotes all events of all classes, if no class indication is given, 'All' is assumed)"]
: H: "False" [Print method-specific help message]
: InputLayout: "1|16|16" [The Layout of the input]
: Layout: "CONV|10|3|3|1|1|1|1|RELU,BNORM,CONV|10|3|3|1|1|1|1|RELU,MAXPOOL|2|2|1|1,RESHAPE|FLAT,DENSE|100|RELU,DENSE|1|LINEAR" [Layout of the network.]
: ErrorStrategy: "CROSSENTROPY" [Loss function: Mean squared error (regression) or cross entropy (binary classification).]
: WeightInitialization: "XAVIER" [Weight initialization strategy]
: Architecture: "CPU" [Which architecture to perform the training on.]
: TrainingStrategy: "LearningRate=1e-3,Momentum=0.9,Repetitions=1,ConvergenceSteps=5,BatchSize=100,TestRepetitions=1,WeightDecay=1e-4,Regularization=None,Optimizer=ADAM,DropConfig=0.0+0.0+0.0+0.0,MaxEpochs=10" [Defines the training strategies.]
: - Default:
: VerbosityLevel: "Default" [Verbosity level]
: CreateMVAPdfs: "False" [Create PDFs for classifier outputs (signal and background)]
: IgnoreNegWeightsInTraining: "False" [Events with negative weights are ignored in the training (but are included for testing and performance evaluation)]
: BatchLayout: "0|0|0" [The Layout of the batch]
: RandomSeed: "0" [Random seed used for weight initialization and batch shuffling]
: ValidationSize: "20%" [Part of the training data to use for validation. Specify as 0.2 or 20% to use a fifth of the data set as validation set. Specify as 100 to use exactly 100 events. (Default: 20%)]
: Will now use the CPU architecture with BLAS and IMT support !
Book Convolutional Neural Network in Keras using a generated model¶
In [18]:
if usePyTorchCNN:
ROOT.Info("TMVA_CNN_Classification", "Using Convolutional PyTorch Model")
pyTorchFileName = str(ROOT.gROOT.GetTutorialDir())
pyTorchFileName += "/tmva/PyTorch_Generate_CNN_Model.py"
# check that pytorch can be imported and file defining the model exists
torch_spec = importlib.util.find_spec("torch")
if torch_spec is not None and os.path.exists(pyTorchFileName):
#cmd = str(ROOT.TMVA.Python_Executable()) + " " + pyTorchFileName
#os.system(cmd)
#import PyTorch_Generate_CNN_Model
ROOT.Info("TMVA_CNN_Classification", "Booking PyTorch CNN model")
factory.BookMethod(
loader,
TMVA.Types.kPyTorch,
"PyTorch",
H=True,
V=False,
VarTransform=None,
FilenameModel="PyTorchModelCNN.pt",
FilenameTrainedModel="PyTorchTrainedModelCNN.pt",
NumEpochs=max_epochs,
BatchSize=100,
UserCode=str(pyTorchFileName)
)
else:
ROOT.Warning(
"TMVA_CNN_Classification",
"PyTorch is not installed or model building file is not existing - skip using PyTorch",
)
if useKerasCNN:
ROOT.Info("TMVA_CNN_Classification", "Building convolutional keras model")
# create python script which can be executed
# create 2 conv2d layer + maxpool + dense
import tensorflow
from tensorflow.keras.models import Sequential
from tensorflow.keras.optimizers import Adam
# from keras.initializers import TruncatedNormal
# from keras import initializations
from tensorflow.keras.layers import Input, Dense, Dropout, Flatten, Conv2D, MaxPooling2D, Reshape
# from keras.callbacks import ReduceLROnPlateau
model = Sequential()
model.add(Reshape((16, 16, 1), input_shape=(256,)))
model.add(Conv2D(10, kernel_size=(3, 3), kernel_initializer="TruncatedNormal", activation="relu", padding="same"))
model.add(Conv2D(10, kernel_size=(3, 3), kernel_initializer="TruncatedNormal", activation="relu", padding="same"))
# stride for maxpool is equal to pool size
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Flatten())
model.add(Dense(64, activation="tanh"))
# model.add(Dropout(0.2))
model.add(Dense(2, activation="sigmoid"))
model.compile(loss="binary_crossentropy", optimizer=Adam(learning_rate=0.001), weighted_metrics=["accuracy"])
model.save("model_cnn.h5")
model.summary()
if not os.path.exists("model_cnn.h5"):
raise FileNotFoundError("Error creating Keras model file - skip using Keras")
else:
# book PyKeras method only if Keras model could be created
ROOT.Info("TMVA_CNN_Classification", "Booking convolutional keras model")
factory.BookMethod(
loader,
TMVA.Types.kPyKeras,
"PyKeras",
H=True,
V=False,
VarTransform=None,
FilenameModel="model_cnn.h5",
FilenameTrainedModel="trained_model_cnn.h5",
NumEpochs=max_epochs,
BatchSize=100,
GpuOptions="allow_growth=True",
) # needed for RTX NVidia card and to avoid TF allocates all GPU memory
running Torch code defining the model....
The PyTorch CNN model is created and saved as PyTorchModelCNN.pt
custom objects for loading model : {'optimizer': <class 'torch.optim.adam.Adam'>, 'criterion': BCELoss(), 'train_func': <function fit at 0x7f651468aca0>, 'predict_func': <function predict at 0x7f651468a8b0>}
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
reshape (Reshape) (None, 16, 16, 1) 0
conv2d (Conv2D) (None, 16, 16, 10) 100
conv2d_1 (Conv2D) (None, 16, 16, 10) 910
max_pooling2d (MaxPooling2D (None, 8, 8, 10) 0
)
flatten (Flatten) (None, 640) 0
dense (Dense) (None, 64) 41024
dense_1 (Dense) (None, 2) 130
=================================================================
Total params: 42,164
Trainable params: 42,164
Non-trainable params: 0
_________________________________________________________________
Factory : Booking method: PyTorch
:
: Using PyTorch - setting special configuration options
: Using PyTorch version 1
: Setup PyTorch Model for training
: Executing user initialization code from /home/sftnight/build/workspace/root-makedoc-master/rootspi/rdoc/src/master.build/tutorials/tmva/PyTorch_Generate_CNN_Model.py
: Loaded pytorch train function:
: Loaded pytorch optimizer:
: Loaded pytorch loss function:
: Loaded pytorch predict function:
: Loaded model from file: PyTorchModelCNN.pt
Factory : Booking method: PyKeras
:
: Setting up tf.keras
: Using TensorFlow version 2
: Use Keras version from TensorFlow : tf.keras
: Applying GPU option: gpu_options.allow_growth=True
: Loading Keras Model
: Loaded model from file: model_cnn.h5
Info in <TMVA_CNN_Classification>: Using Convolutional PyTorch Model Info in <TMVA_CNN_Classification>: Booking PyTorch CNN model Info in <TMVA_CNN_Classification>: Building convolutional keras model 2023-03-27 09:59:04.405090: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2023-03-27 09:59:06.335861: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /home/sftnight/build/workspace/root-makedoc-master/rootspi/rdoc/src/master.build/lib 2023-03-27 09:59:06.335984: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /home/sftnight/build/workspace/root-makedoc-master/rootspi/rdoc/src/master.build/lib 2023-03-27 09:59:06.335999: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly. 2023-03-27 09:59:08.131419: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /home/sftnight/build/workspace/root-makedoc-master/rootspi/rdoc/src/master.build/lib 2023-03-27 09:59:08.131464: W tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:265] failed call to cuInit: UNKNOWN ERROR (303) 2023-03-27 09:59:08.131544: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (root-ubuntu-2004-3): /proc/driver/nvidia/version does not exist 2023-03-27 09:59:08.131910: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. Info in <TMVA_CNN_Classification>: Booking convolutional keras model
Train Methods
In [19]:
factory.TrainAllMethods()
RecursiveScriptModule(
original_name=Sequential
(0): RecursiveScriptModule(original_name=Reshape)
(1): RecursiveScriptModule(original_name=Conv2d)
(2): RecursiveScriptModule(original_name=ReLU)
(3): RecursiveScriptModule(original_name=BatchNorm2d)
(4): RecursiveScriptModule(original_name=Conv2d)
(5): RecursiveScriptModule(original_name=ReLU)
(6): RecursiveScriptModule(original_name=MaxPool2d)
(7): RecursiveScriptModule(original_name=Flatten)
(8): RecursiveScriptModule(original_name=Linear)
(9): RecursiveScriptModule(original_name=ReLU)
(10): RecursiveScriptModule(original_name=Linear)
(11): RecursiveScriptModule(original_name=Sigmoid)
)
[1, 4] train loss: 1.169
[1, 8] train loss: 0.736
[1, 12] train loss: 0.697
[1] val loss: 0.689
[2, 4] train loss: 0.695
[2, 8] train loss: 0.698
[2, 12] train loss: 0.691
[2] val loss: 0.691
[3, 4] train loss: 0.688
[3, 8] train loss: 0.687
[3, 12] train loss: 0.679
[3] val loss: 0.691
[4, 4] train loss: 0.670
[4, 8] train loss: 0.666
[4, 12] train loss: 0.664
[4] val loss: 0.726
[5, 4] train loss: 0.641
[5, 8] train loss: 0.637
[5, 12] train loss: 0.633
[5] val loss: 0.734
[6, 4] train loss: 0.597
[6, 8] train loss: 0.597
[6, 12] train loss: 0.575
[6] val loss: 1.003
[7, 4] train loss: 0.540
[7, 8] train loss: 0.521
[7, 12] train loss: 0.568
[7] val loss: 0.740
[8, 4] train loss: 0.513
[8, 8] train loss: 0.525
[8, 12] train loss: 0.489
[8] val loss: 0.739
[9, 4] train loss: 0.470
[9, 8] train loss: 0.453
[9, 12] train loss: 0.469
[9] val loss: 0.796
[10, 4] train loss: 0.485
[10, 8] train loss: 0.453
[10, 12] train loss: 0.515
[10] val loss: 0.930
Finished Training on 10 Epochs!
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
reshape (Reshape) (None, 16, 16, 1) 0
conv2d (Conv2D) (None, 16, 16, 10) 100
conv2d_1 (Conv2D) (None, 16, 16, 10) 910
max_pooling2d (MaxPooling2D (None, 8, 8, 10) 0
)
flatten (Flatten) (None, 640) 0
dense (Dense) (None, 64) 41024
dense_1 (Dense) (None, 2) 130
=================================================================
Total params: 42,164
Trainable params: 42,164
Non-trainable params: 0
_________________________________________________________________
Epoch 1/10
13/13 [==============================] - ETA: 0s - loss: 0.7274 - accuracy: 0.5242
Epoch 1: val_loss improved from inf to 0.70090, saving model to trained_model_cnn.h5
13/13 [==============================] - 2s 73ms/step - loss: 0.7274 - accuracy: 0.5242 - val_loss: 0.7009 - val_accuracy: 0.5094
Epoch 2/10
13/13 [==============================] - ETA: 0s - loss: 0.6992 - accuracy: 0.4938
Epoch 2: val_loss improved from 0.70090 to 0.69866, saving model to trained_model_cnn.h5
13/13 [==============================] - 0s 21ms/step - loss: 0.6992 - accuracy: 0.4938 - val_loss: 0.6987 - val_accuracy: 0.4656
Epoch 3/10
12/13 [==========================>...] - ETA: 0s - loss: 0.6873 - accuracy: 0.5617
Epoch 3: val_loss improved from 0.69866 to 0.68974, saving model to trained_model_cnn.h5
13/13 [==============================] - 0s 23ms/step - loss: 0.6875 - accuracy: 0.5570 - val_loss: 0.6897 - val_accuracy: 0.5656
Epoch 4/10
9/13 [===================>..........] - ETA: 0s - loss: 0.6819 - accuracy: 0.5700
Epoch 4: val_loss improved from 0.68974 to 0.68789, saving model to trained_model_cnn.h5
13/13 [==============================] - 0s 20ms/step - loss: 0.6828 - accuracy: 0.5578 - val_loss: 0.6879 - val_accuracy: 0.5562
Epoch 5/10
10/13 [======================>.......] - ETA: 0s - loss: 0.6706 - accuracy: 0.6490
Epoch 5: val_loss improved from 0.68789 to 0.68667, saving model to trained_model_cnn.h5
13/13 [==============================] - 0s 20ms/step - loss: 0.6706 - accuracy: 0.6438 - val_loss: 0.6867 - val_accuracy: 0.5594
Epoch 6/10
11/13 [========================>.....] - ETA: 0s - loss: 0.6572 - accuracy: 0.6464
Epoch 6: val_loss improved from 0.68667 to 0.67853, saving model to trained_model_cnn.h5
13/13 [==============================] - 0s 19ms/step - loss: 0.6581 - accuracy: 0.6414 - val_loss: 0.6785 - val_accuracy: 0.5938
Epoch 7/10
12/13 [==========================>...] - ETA: 0s - loss: 0.6384 - accuracy: 0.6717
Epoch 7: val_loss improved from 0.67853 to 0.67365, saving model to trained_model_cnn.h5
13/13 [==============================] - 0s 34ms/step - loss: 0.6375 - accuracy: 0.6719 - val_loss: 0.6737 - val_accuracy: 0.5844
Epoch 8/10
12/13 [==========================>...] - ETA: 0s - loss: 0.6131 - accuracy: 0.6983
Epoch 8: val_loss did not improve from 0.67365
13/13 [==============================] - 0s 35ms/step - loss: 0.6124 - accuracy: 0.7008 - val_loss: 0.6936 - val_accuracy: 0.5344
Epoch 9/10
11/13 [========================>.....] - ETA: 0s - loss: 0.5899 - accuracy: 0.7200
Epoch 9: val_loss did not improve from 0.67365
13/13 [==============================] - 0s 17ms/step - loss: 0.5902 - accuracy: 0.7234 - val_loss: 0.6820 - val_accuracy: 0.5500
Epoch 10/10
11/13 [========================>.....] - ETA: 0s - loss: 0.5735 - accuracy: 0.7200
Epoch 10: val_loss improved from 0.67365 to 0.64810, saving model to trained_model_cnn.h5
13/13 [==============================] - 0s 19ms/step - loss: 0.5797 - accuracy: 0.7094 - val_loss: 0.6481 - val_accuracy: 0.6344
/usr/local/lib/python3.8/dist-packages/keras/engine/training_v1.py:2357: UserWarning: `Model.state_updates` will be removed in a future version. This property should not be used in TensorFlow 2.0, as `updates` are applied automatically. updates=self.state_updates,
Factory : Train all methods Factory : Train method: BDT for Classification : BDT : #events: (reweighted) sig: 800 bkg: 800 : #events: (unweighted) sig: 800 bkg: 800 : Training 400 Decision Trees ... patience please : Elapsed time for training with 1600 events: 2.91 sec BDT : [dataset] : Evaluation of BDT on training sample (1600 events) : Elapsed time for evaluation of 1600 events: 0.0403 sec : Creating xml weight file: dataset/weights/TMVA_CNN_Classification_BDT.weights.xml : Creating standalone class: dataset/weights/TMVA_CNN_Classification_BDT.class.C : TMVA_CNN_ClassificationOutput.root:/dataset/Method_BDT/BDT Factory : Training finished : Factory : Train method: TMVA_DNN_CPU for Classification : : Start of deep neural network training on CPU using MT, nthreads = 4 : : ***** Deep Learning Network ***** DEEP NEURAL NETWORK: Depth = 8 Input = ( 1, 1, 256 ) Batch size = 100 Loss function = C Layer 0 DENSE Layer: ( Input = 256 , Width = 100 ) Output = ( 1 , 100 , 100 ) Activation Function = Relu Layer 1 BATCH NORM Layer: Input/Output = ( 100 , 100 , 1 ) Norm dim = 100 axis = -1 Layer 2 DENSE Layer: ( Input = 100 , Width = 100 ) Output = ( 1 , 100 , 100 ) Activation Function = Relu Layer 3 BATCH NORM Layer: Input/Output = ( 100 , 100 , 1 ) Norm dim = 100 axis = -1 Layer 4 DENSE Layer: ( Input = 100 , Width = 100 ) Output = ( 1 , 100 , 100 ) Activation Function = Relu Layer 5 BATCH NORM Layer: Input/Output = ( 100 , 100 , 1 ) Norm dim = 100 axis = -1 Layer 6 DENSE Layer: ( Input = 100 , Width = 100 ) Output = ( 1 , 100 , 100 ) Activation Function = Relu Layer 7 DENSE Layer: ( Input = 100 , Width = 1 ) Output = ( 1 , 100 , 1 ) Activation Function = Identity : Using 1280 events for training and 320 for testing : Compute initial loss on the validation data : Training phase 1 of 1: Optimizer ADAM (beta1=0.9,beta2=0.999,eps=1e-07) Learning rate = 0.001 regularization 0 minimum error = inf : -------------------------------------------------------------- : Epoch | Train Err. Val. Err. t(s)/epoch t(s)/Loss nEvents/s Conv. Steps : -------------------------------------------------------------- : Start epoch iteration ... : 1 Minimum Test error found - save the configuration : 1 | 0.86444 0.982167 0.217822 0.0189736 6034.74 0 : 2 Minimum Test error found - save the configuration : 2 | 0.665743 0.804343 0.252836 0.0246082 5257.91 0 : 3 Minimum Test error found - save the configuration : 3 | 0.559295 0.724914 0.243865 0.0269991 5533.38 0 : 4 | 0.525703 0.792099 0.228501 0.0217753 5804.81 1 : 5 Minimum Test error found - save the configuration : 5 | 0.457889 0.713177 0.206954 0.0163815 6296.82 0 : 6 | 0.402496 0.765025 0.224198 0.0249116 6021.5 1 : 7 | 0.35982 0.775368 0.293707 0.0163662 4326.8 2 : 8 | 0.32168 0.718174 0.220982 0.0155485 5841.3 3 : 9 Minimum Test error found - save the configuration : 9 | 0.261654 0.67407 0.197752 0.0166592 6626.43 0 : 10 | 0.229271 0.728034 0.187634 0.0152467 6961.06 1 : : Elapsed time for training with 1600 events: 2.31 sec : Evaluate deep neural network on CPU using batches with size = 100 : TMVA_DNN_CPU : [dataset] : Evaluation of TMVA_DNN_CPU on training sample (1600 events) : Elapsed time for evaluation of 1600 events: 0.0805 sec : Creating xml weight file: dataset/weights/TMVA_CNN_Classification_TMVA_DNN_CPU.weights.xml : Creating standalone class: dataset/weights/TMVA_CNN_Classification_TMVA_DNN_CPU.class.C Factory : Training finished : Factory : Train method: TMVA_CNN_CPU for Classification : : Start of deep neural network training on CPU using MT, nthreads = 4 : : ***** Deep Learning Network ***** DEEP NEURAL NETWORK: Depth = 7 Input = ( 1, 16, 16 ) Batch size = 100 Loss function = C Layer 0 CONV LAYER: ( W = 16 , H = 16 , D = 10 ) Filter ( W = 3 , H = 3 ) Output = ( 100 , 10 , 10 , 256 ) Activation Function = Relu Layer 1 BATCH NORM Layer: Input/Output = ( 10 , 256 , 100 ) Norm dim = 10 axis = 1 Layer 2 CONV LAYER: ( W = 16 , H = 16 , D = 10 ) Filter ( W = 3 , H = 3 ) Output = ( 100 , 10 , 10 , 256 ) Activation Function = Relu Layer 3 POOL Layer: ( W = 15 , H = 15 , D = 10 ) Filter ( W = 2 , H = 2 ) Output = ( 100 , 10 , 10 , 225 ) Layer 4 RESHAPE Layer Input = ( 10 , 15 , 15 ) Output = ( 1 , 100 , 2250 ) Layer 5 DENSE Layer: ( Input = 2250 , Width = 100 ) Output = ( 1 , 100 , 100 ) Activation Function = Relu Layer 6 DENSE Layer: ( Input = 100 , Width = 1 ) Output = ( 1 , 100 , 1 ) Activation Function = Identity : Using 1280 events for training and 320 for testing : Compute initial loss on the validation data : Training phase 1 of 1: Optimizer ADAM (beta1=0.9,beta2=0.999,eps=1e-07) Learning rate = 0.001 regularization 0 minimum error = inf : -------------------------------------------------------------- : Epoch | Train Err. Val. Err. t(s)/epoch t(s)/Loss nEvents/s Conv. Steps : -------------------------------------------------------------- : Start epoch iteration ... : 1 Minimum Test error found - save the configuration : 1 | 2.09062 1.23817 1.47198 0.123887 890.148 0 : 2 Minimum Test error found - save the configuration : 2 | 0.841181 0.7274 1.55827 0.120912 834.863 0 : 3 Minimum Test error found - save the configuration : 3 | 0.711199 0.688134 1.5062 0.119959 865.65 0 : 4 | 0.704263 0.688568 1.55299 0.170355 867.909 1 : 5 Minimum Test error found - save the configuration : 5 | 0.681378 0.683751 1.71669 0.135752 759.042 0 : 6 | 0.679999 0.68816 1.52052 0.133917 865.425 1 : 7 Minimum Test error found - save the configuration : 7 | 0.665331 0.682981 1.59842 0.123999 813.88 0 : 8 Minimum Test error found - save the configuration : 8 | 0.636336 0.662822 1.60164 0.121625 810.8 0 : 9 Minimum Test error found - save the configuration : 9 | 0.609077 0.62137 1.60994 0.121365 806.14 0 : 10 | 0.583935 0.633779 1.63764 0.143019 802.877 1 : : Elapsed time for training with 1600 events: 15.9 sec : Evaluate deep neural network on CPU using batches with size = 100 : TMVA_CNN_CPU : [dataset] : Evaluation of TMVA_CNN_CPU on training sample (1600 events) : Elapsed time for evaluation of 1600 events: 0.77 sec : Creating xml weight file: dataset/weights/TMVA_CNN_Classification_TMVA_CNN_CPU.weights.xml : Creating standalone class: dataset/weights/TMVA_CNN_Classification_TMVA_CNN_CPU.class.C Factory : Training finished : Factory : Train method: PyTorch for Classification : : : ================================================================ : H e l p f o r M V A m e t h o d [ PyTorch ] : : : PyTorch is a scientific computing package supporting : automatic differentiation. This method wraps the training : and predictions steps of the PyTorch Python package for : TMVA, so that dataloading, preprocessing and evaluation : can be done within the TMVA system. To use this PyTorch : interface, you need to generatea model with PyTorch first. : Then, this model can be loaded and trained in TMVA. : : : <Suppress this message by specifying "!H" in the booking option> : ================================================================ : : Split TMVA training data in 1280 training events and 320 validation events : Print Training Model Architecture : Option SaveBestOnly: Only model weights with smallest validation loss will be stored : Elapsed time for training with 1600 events: 24 sec PyTorch : [dataset] : Evaluation of PyTorch on training sample (1600 events) : Elapsed time for evaluation of 1600 events: 0.437 sec : Creating xml weight file: dataset/weights/TMVA_CNN_Classification_PyTorch.weights.xml : Creating standalone class: dataset/weights/TMVA_CNN_Classification_PyTorch.class.C Factory : Training finished : Factory : Train method: PyKeras for Classification : : : ================================================================ : H e l p f o r M V A m e t h o d [ PyKeras ] : : : Keras is a high-level API for the Theano and Tensorflow packages. : This method wraps the training and predictions steps of the Keras : Python package for TMVA, so that dataloading, preprocessing and : evaluation can be done within the TMVA system. To use this Keras : interface, you have to generate a model with Keras first. Then, : this model can be loaded and trained in TMVA. : : : <Suppress this message by specifying "!H" in the booking option> : ================================================================ : : Split TMVA training data in 1280 training events and 320 validation events : Training Model Summary : Option SaveBestOnly: Only model weights with smallest validation loss will be stored : Getting training history for item:0 name = 'loss' : Getting training history for item:1 name = 'accuracy' : Getting training history for item:2 name = 'val_loss' : Getting training history for item:3 name = 'val_accuracy' : Elapsed time for training with 1600 events: 5.33 sec : Setting up tf.keras : Using TensorFlow version 2 : Use Keras version from TensorFlow : tf.keras : Applying GPU option: gpu_options.allow_growth=True : Disabled TF eager execution when evaluating model : Loading Keras Model : Loaded model from file: trained_model_cnn.h5 PyKeras : [dataset] : Evaluation of PyKeras on training sample (1600 events) : Elapsed time for evaluation of 1600 events: 0.293 sec : Creating xml weight file: dataset/weights/TMVA_CNN_Classification_PyKeras.weights.xml : Creating standalone class: dataset/weights/TMVA_CNN_Classification_PyKeras.class.C Factory : Training finished : : Ranking input variables (method specific)... BDT : Ranking result (top variable is best ranked) : -------------------------------------- : Rank : Variable : Variable Importance : -------------------------------------- : 1 : vars : 9.773e-03 : 2 : vars : 8.851e-03 : 3 : vars : 8.428e-03 : 4 : vars : 7.849e-03 : 5 : vars : 7.685e-03 : 6 : vars : 7.637e-03 : 7 : vars : 7.599e-03 : 8 : vars : 7.596e-03 : 9 : vars : 7.487e-03 : 10 : vars : 7.347e-03 : 11 : vars : 7.257e-03 : 12 : vars : 7.152e-03 : 13 : vars : 7.126e-03 : 14 : vars : 7.092e-03 : 15 : vars : 7.049e-03 : 16 : vars : 7.029e-03 : 17 : vars : 6.913e-03 : 18 : vars : 6.865e-03 : 19 : vars : 6.838e-03 : 20 : vars : 6.825e-03 : 21 : vars : 6.822e-03 : 22 : vars : 6.803e-03 : 23 : vars : 6.721e-03 : 24 : vars : 6.634e-03 : 25 : vars : 6.495e-03 : 26 : vars : 6.480e-03 : 27 : vars : 6.471e-03 : 28 : vars : 6.450e-03 : 29 : vars : 6.387e-03 : 30 : vars : 6.367e-03 : 31 : vars : 6.356e-03 : 32 : vars : 6.290e-03 : 33 : vars : 6.246e-03 : 34 : vars : 6.077e-03 : 35 : vars : 6.032e-03 : 36 : vars : 6.028e-03 : 37 : vars : 5.815e-03 : 38 : vars : 5.766e-03 : 39 : vars : 5.726e-03 : 40 : vars : 5.726e-03 : 41 : vars : 5.685e-03 : 42 : vars : 5.627e-03 : 43 : vars : 5.600e-03 : 44 : vars : 5.598e-03 : 45 : vars : 5.596e-03 : 46 : vars : 5.548e-03 : 47 : vars : 5.537e-03 : 48 : vars : 5.535e-03 : 49 : vars : 5.505e-03 : 50 : vars : 5.490e-03 : 51 : vars : 5.479e-03 : 52 : vars : 5.430e-03 : 53 : vars : 5.426e-03 : 54 : vars : 5.423e-03 : 55 : vars : 5.408e-03 : 56 : vars : 5.408e-03 : 57 : vars : 5.406e-03 : 58 : vars : 5.380e-03 : 59 : vars : 5.339e-03 : 60 : vars : 5.339e-03 : 61 : vars : 5.319e-03 : 62 : vars : 5.261e-03 : 63 : vars : 5.215e-03 : 64 : vars : 5.179e-03 : 65 : vars : 5.174e-03 : 66 : vars : 5.166e-03 : 67 : vars : 5.148e-03 : 68 : vars : 5.130e-03 : 69 : vars : 5.124e-03 : 70 : vars : 5.120e-03 : 71 : vars : 5.112e-03 : 72 : vars : 5.042e-03 : 73 : vars : 5.041e-03 : 74 : vars : 5.014e-03 : 75 : vars : 5.008e-03 : 76 : vars : 5.006e-03 : 77 : vars : 5.000e-03 : 78 : vars : 4.988e-03 : 79 : vars : 4.936e-03 : 80 : vars : 4.903e-03 : 81 : vars : 4.896e-03 : 82 : vars : 4.872e-03 : 83 : vars : 4.869e-03 : 84 : vars : 4.862e-03 : 85 : vars : 4.843e-03 : 86 : vars : 4.842e-03 : 87 : vars : 4.818e-03 : 88 : vars : 4.796e-03 : 89 : vars : 4.767e-03 : 90 : vars : 4.735e-03 : 91 : vars : 4.722e-03 : 92 : vars : 4.707e-03 : 93 : vars : 4.620e-03 : 94 : vars : 4.608e-03 : 95 : vars : 4.602e-03 : 96 : vars : 4.595e-03 : 97 : vars : 4.591e-03 : 98 : vars : 4.521e-03 : 99 : vars : 4.483e-03 : 100 : vars : 4.476e-03 : 101 : vars : 4.446e-03 : 102 : vars : 4.391e-03 : 103 : vars : 4.356e-03 : 104 : vars : 4.331e-03 : 105 : vars : 4.299e-03 : 106 : vars : 4.299e-03 : 107 : vars : 4.280e-03 : 108 : vars : 4.262e-03 : 109 : vars : 4.249e-03 : 110 : vars : 4.231e-03 : 111 : vars : 4.217e-03 : 112 : vars : 4.204e-03 : 113 : vars : 4.199e-03 : 114 : vars : 4.137e-03 : 115 : vars : 4.126e-03 : 116 : vars : 4.091e-03 : 117 : vars : 4.059e-03 : 118 : vars : 4.013e-03 : 119 : vars : 4.008e-03 : 120 : vars : 3.992e-03 : 121 : vars : 3.985e-03 : 122 : vars : 3.983e-03 : 123 : vars : 3.980e-03 : 124 : vars : 3.970e-03 : 125 : vars : 3.969e-03 : 126 : vars : 3.968e-03 : 127 : vars : 3.965e-03 : 128 : vars : 3.955e-03 : 129 : vars : 3.948e-03 : 130 : vars : 3.939e-03 : 131 : vars : 3.934e-03 : 132 : vars : 3.909e-03 : 133 : vars : 3.901e-03 : 134 : vars : 3.883e-03 : 135 : vars : 3.841e-03 : 136 : vars : 3.821e-03 : 137 : vars : 3.802e-03 : 138 : vars : 3.652e-03 : 139 : vars : 3.645e-03 : 140 : vars : 3.634e-03 : 141 : vars : 3.634e-03 : 142 : vars : 3.609e-03 : 143 : vars : 3.595e-03 : 144 : vars : 3.585e-03 : 145 : vars : 3.570e-03 : 146 : vars : 3.544e-03 : 147 : vars : 3.536e-03 : 148 : vars : 3.496e-03 : 149 : vars : 3.484e-03 : 150 : vars : 3.480e-03 : 151 : vars : 3.449e-03 : 152 : vars : 3.439e-03 : 153 : vars : 3.393e-03 : 154 : vars : 3.385e-03 : 155 : vars : 3.384e-03 : 156 : vars : 3.382e-03 : 157 : vars : 3.360e-03 : 158 : vars : 3.360e-03 : 159 : vars : 3.309e-03 : 160 : vars : 3.293e-03 : 161 : vars : 3.262e-03 : 162 : vars : 3.261e-03 : 163 : vars : 3.238e-03 : 164 : vars : 3.237e-03 : 165 : vars : 3.204e-03 : 166 : vars : 3.168e-03 : 167 : vars : 3.121e-03 : 168 : vars : 3.116e-03 : 169 : vars : 3.089e-03 : 170 : vars : 3.084e-03 : 171 : vars : 3.081e-03 : 172 : vars : 3.038e-03 : 173 : vars : 3.025e-03 : 174 : vars : 3.017e-03 : 175 : vars : 3.014e-03 : 176 : vars : 2.994e-03 : 177 : vars : 2.991e-03 : 178 : vars : 2.986e-03 : 179 : vars : 2.919e-03 : 180 : vars : 2.918e-03 : 181 : vars : 2.908e-03 : 182 : vars : 2.898e-03 : 183 : vars : 2.895e-03 : 184 : vars : 2.888e-03 : 185 : vars : 2.884e-03 : 186 : vars : 2.805e-03 : 187 : vars : 2.798e-03 : 188 : vars : 2.796e-03 : 189 : vars : 2.790e-03 : 190 : vars : 2.785e-03 : 191 : vars : 2.749e-03 : 192 : vars : 2.747e-03 : 193 : vars : 2.716e-03 : 194 : vars : 2.715e-03 : 195 : vars : 2.705e-03 : 196 : vars : 2.697e-03 : 197 : vars : 2.634e-03 : 198 : vars : 2.620e-03 : 199 : vars : 2.532e-03 : 200 : vars : 2.511e-03 : 201 : vars : 2.496e-03 : 202 : vars : 2.485e-03 : 203 : vars : 2.455e-03 : 204 : vars : 2.439e-03 : 205 : vars : 2.425e-03 : 206 : vars : 2.367e-03 : 207 : vars : 2.349e-03 : 208 : vars : 2.289e-03 : 209 : vars : 2.262e-03 : 210 : vars : 2.239e-03 : 211 : vars : 2.235e-03 : 212 : vars : 2.162e-03 : 213 : vars : 2.153e-03 : 214 : vars : 2.077e-03 : 215 : vars : 2.071e-03 : 216 : vars : 2.066e-03 : 217 : vars : 1.986e-03 : 218 : vars : 1.956e-03 : 219 : vars : 1.934e-03 : 220 : vars : 1.933e-03 : 221 : vars : 1.915e-03 : 222 : vars : 1.873e-03 : 223 : vars : 1.849e-03 : 224 : vars : 1.848e-03 : 225 : vars : 1.822e-03 : 226 : vars : 1.753e-03 : 227 : vars : 1.628e-03 : 228 : vars : 1.545e-03 : 229 : vars : 1.543e-03 : 230 : vars : 1.336e-03 : 231 : vars : 1.314e-03 : 232 : vars : 1.301e-03 : 233 : vars : 1.233e-03 : 234 : vars : 1.133e-03 : 235 : vars : 1.043e-03 : 236 : vars : 9.950e-04 : 237 : vars : 8.987e-04 : 238 : vars : 6.255e-04 : 239 : vars : 0.000e+00 : 240 : vars : 0.000e+00 : 241 : vars : 0.000e+00 : 242 : vars : 0.000e+00 : 243 : vars : 0.000e+00 : 244 : vars : 0.000e+00 : 245 : vars : 0.000e+00 : 246 : vars : 0.000e+00 : 247 : vars : 0.000e+00 : 248 : vars : 0.000e+00 : 249 : vars : 0.000e+00 : 250 : vars : 0.000e+00 : 251 : vars : 0.000e+00 : 252 : vars : 0.000e+00 : 253 : vars : 0.000e+00 : 254 : vars : 0.000e+00 : 255 : vars : 0.000e+00 : 256 : vars : 0.000e+00 : -------------------------------------- : No variable ranking supplied by classifier: TMVA_DNN_CPU : No variable ranking supplied by classifier: TMVA_CNN_CPU : No variable ranking supplied by classifier: PyTorch : No variable ranking supplied by classifier: PyKeras TH1.Print Name = TrainingHistory_TMVA_DNN_CPU_trainingError, Entries= 0, Total sum= 4.64799 TH1.Print Name = TrainingHistory_TMVA_DNN_CPU_valError, Entries= 0, Total sum= 7.67737 TH1.Print Name = TrainingHistory_TMVA_CNN_CPU_trainingError, Entries= 0, Total sum= 8.20332 TH1.Print Name = TrainingHistory_TMVA_CNN_CPU_valError, Entries= 0, Total sum= 7.31514 TH1.Print Name = TrainingHistory_PyKeras_'accuracy', Entries= 0, Total sum= 6.22344 TH1.Print Name = TrainingHistory_PyKeras_'loss', Entries= 0, Total sum= 6.54554 TH1.Print Name = TrainingHistory_PyKeras_'val_accuracy', Entries= 0, Total sum= 5.55312 TH1.Print Name = TrainingHistory_PyKeras_'val_loss', Entries= 0, Total sum= 6.8398 Factory : === Destroy and recreate all methods via weight files for testing === : : Reading weight file: dataset/weights/TMVA_CNN_Classification_BDT.weights.xml : Reading weight file: dataset/weights/TMVA_CNN_Classification_TMVA_DNN_CPU.weights.xml : Reading weight file: dataset/weights/TMVA_CNN_Classification_TMVA_CNN_CPU.weights.xml : Reading weight file: dataset/weights/TMVA_CNN_Classification_PyTorch.weights.xml : Reading weight file: dataset/weights/TMVA_CNN_Classification_PyKeras.weights.xml
0%, time left: unknown
6%, time left: 2 sec
12%, time left: 1 sec
19%, time left: 1 sec
25%, time left: 1 sec
31%, time left: 1 sec
37%, time left: 1 sec
44%, time left: 1 sec
50%, time left: 1 sec
56%, time left: 1 sec
62%, time left: 0 sec
69%, time left: 0 sec
75%, time left: 0 sec
81%, time left: 0 sec
87%, time left: 0 sec
94%, time left: 0 sec
0%, time left: unknown
7%, time left: 0 sec
13%, time left: 0 sec
19%, time left: 0 sec
25%, time left: 0 sec
32%, time left: 0 sec
38%, time left: 0 sec
44%, time left: 0 sec
50%, time left: 0 sec
57%, time left: 0 sec
63%, time left: 0 sec
69%, time left: 0 sec
75%, time left: 0 sec
82%, time left: 0 sec
88%, time left: 0 sec
94%, time left: 0 sec
2023-03-27 10:00:01.284328: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:357] MLIR V1 optimization pass is not enabled
2023-03-27 10:00:01.299098: W tensorflow/c/c_api.cc:291] Operation '{name:'dense_1/kernel/Assign' id:98 op device:{requested: '', assigned: ''} def:{{{node dense_1/kernel/Assign}} = AssignVariableOp[_has_manual_control_dependencies=true, dtype=DT_FLOAT, validate_shape=false](dense_1/kernel, dense_1/kernel/Initializer/stateless_random_uniform)}}' was changed by setting attribute after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-03-27 10:00:01.554491: W tensorflow/c/c_api.cc:291] Operation '{name:'dense_1/Sigmoid' id:109 op device:{requested: '', assigned: ''} def:{{{node dense_1/Sigmoid}} = Sigmoid[T=DT_FLOAT, _has_manual_control_dependencies=true](dense_1/BiasAdd)}}' was changed by setting attribute after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-03-27 10:00:01.577913: W tensorflow/c/c_api.cc:291] Operation '{name:'total/Assign' id:156 op device:{requested: '', assigned: ''} def:{{{node total/Assign}} = AssignVariableOp[_has_manual_control_dependencies=true, dtype=DT_FLOAT, validate_shape=false](total, total/Initializer/zeros)}}' was changed by setting attribute after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
Test and Evaluate Methods
In [20]:
factory.TestAllMethods()
factory.EvaluateAllMethods()
running Torch code defining the model....
The PyTorch CNN model is created and saved as PyTorchModelCNN.pt
custom objects for loading model : {'optimizer': <class 'torch.optim.adam.Adam'>, 'criterion': BCELoss(), 'train_func': <function fit at 0x7f66097b6f70>, 'predict_func': <function predict at 0x7f65e3e0d8b0>}
Factory : Test all methods
Factory : Test method: BDT for Classification performance
:
BDT : [dataset] : Evaluation of BDT on testing sample (400 events)
: Elapsed time for evaluation of 400 events: 0.00946 sec
Factory : Test method: TMVA_DNN_CPU for Classification performance
:
: Evaluate deep neural network on CPU using batches with size = 400
:
TMVA_DNN_CPU : [dataset] : Evaluation of TMVA_DNN_CPU on testing sample (400 events)
: Elapsed time for evaluation of 400 events: 0.0194 sec
Factory : Test method: TMVA_CNN_CPU for Classification performance
:
: Evaluate deep neural network on CPU using batches with size = 400
:
TMVA_CNN_CPU : [dataset] : Evaluation of TMVA_CNN_CPU on testing sample (400 events)
: Elapsed time for evaluation of 400 events: 0.159 sec
Factory : Test method: PyTorch for Classification performance
:
: Setup PyTorch Model for training
: Executing user initialization code from /home/sftnight/build/workspace/root-makedoc-master/rootspi/rdoc/src/master.build/tutorials/tmva/PyTorch_Generate_CNN_Model.py
: Loaded pytorch train function:
: Loaded pytorch optimizer:
: Loaded pytorch loss function:
: Loaded pytorch predict function:
: Loaded model from file: PyTorchTrainedModelCNN.pt
PyTorch : [dataset] : Evaluation of PyTorch on testing sample (400 events)
: Elapsed time for evaluation of 400 events: 0.139 sec
Factory : Test method: PyKeras for Classification performance
:
: Setting up tf.keras
: Using TensorFlow version 2
: Use Keras version from TensorFlow : tf.keras
: Applying GPU option: gpu_options.allow_growth=True
: Disabled TF eager execution when evaluating model
: Loading Keras Model
: Loaded model from file: trained_model_cnn.h5
PyKeras : [dataset] : Evaluation of PyKeras on testing sample (400 events)
: Elapsed time for evaluation of 400 events: 0.149 sec
Factory : Evaluate all methods
Factory : Evaluate classifier: BDT
:
BDT : [dataset] : Loop over test events and fill histograms with classifier response...
:
: Dataset[dataset] : variable plots are not produces ! The number of variables is 256 , it is larger than 200
Factory : Evaluate classifier: TMVA_DNN_CPU
:
TMVA_DNN_CPU : [dataset] : Loop over test events and fill histograms with classifier response...
:
: Evaluate deep neural network on CPU using batches with size = 1000
:
: Dataset[dataset] : variable plots are not produces ! The number of variables is 256 , it is larger than 200
Factory : Evaluate classifier: TMVA_CNN_CPU
:
TMVA_CNN_CPU : [dataset] : Loop over test events and fill histograms with classifier response...
:
: Evaluate deep neural network on CPU using batches with size = 1000
:
: Dataset[dataset] : variable plots are not produces ! The number of variables is 256 , it is larger than 200
Factory : Evaluate classifier: PyTorch
:
PyTorch : [dataset] : Loop over test events and fill histograms with classifier response...
:
: Dataset[dataset] : variable plots are not produces ! The number of variables is 256 , it is larger than 200
Factory : Evaluate classifier: PyKeras
:
PyKeras : [dataset] : Loop over test events and fill histograms with classifier response...
:
: Dataset[dataset] : variable plots are not produces ! The number of variables is 256 , it is larger than 200
:
: Evaluation results ranked by best signal efficiency and purity (area)
: -------------------------------------------------------------------------------------------------------------------
: DataSet MVA
: Name: Method: ROC-integ
: dataset BDT : 0.784
: dataset PyKeras : 0.699
: dataset TMVA_DNN_CPU : 0.686
: dataset TMVA_CNN_CPU : 0.670
: dataset PyTorch : 0.525
: -------------------------------------------------------------------------------------------------------------------
:
: Testing efficiency compared to training efficiency (overtraining check)
: -------------------------------------------------------------------------------------------------------------------
: DataSet MVA Signal efficiency: from test sample (from training sample)
: Name: Method: @B=0.01 @B=0.10 @B=0.30
: -------------------------------------------------------------------------------------------------------------------
: dataset BDT : 0.065 (0.325) 0.365 (0.757) 0.775 (0.915)
: dataset PyKeras : 0.040 (0.098) 0.333 (0.417) 0.595 (0.688)
: dataset TMVA_DNN_CPU : 0.055 (0.175) 0.285 (0.585) 0.574 (0.776)
: dataset TMVA_CNN_CPU : 0.065 (0.138) 0.285 (0.362) 0.520 (0.625)
: dataset PyTorch : 0.010 (0.014) 0.150 (0.158) 0.310 (0.320)
: -------------------------------------------------------------------------------------------------------------------
:
Dataset:dataset : Created tree 'TestTree' with 400 events
:
Dataset:dataset : Created tree 'TrainTree' with 1600 events
:
Factory : Thank you for using TMVA!
: For citation information, please visit: http://tmva.sf.net/citeTMVA.html
0%, time left: unknown
7%, time left: 0 sec
13%, time left: 0 sec
19%, time left: 0 sec
25%, time left: 0 sec
32%, time left: 0 sec
38%, time left: 0 sec
44%, time left: 0 sec
50%, time left: 0 sec
57%, time left: 0 sec
63%, time left: 0 sec
69%, time left: 0 sec
75%, time left: 0 sec
82%, time left: 0 sec
88%, time left: 0 sec
94%, time left: 0 sec
2023-03-27 10:00:02.954980: W tensorflow/c/c_api.cc:291] Operation '{name:'dense_1_1/bias/Assign' id:450 op device:{requested: '', assigned: ''} def:{{{node dense_1_1/bias/Assign}} = AssignVariableOp[_has_manual_control_dependencies=true, dtype=DT_FLOAT, validate_shape=false](dense_1_1/bias, dense_1_1/bias/Initializer/zeros)}}' was changed by setting attribute after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-03-27 10:00:03.237454: W tensorflow/c/c_api.cc:291] Operation '{name:'dense_1_1/Sigmoid' id:456 op device:{requested: '', assigned: ''} def:{{{node dense_1_1/Sigmoid}} = Sigmoid[T=DT_FLOAT, _has_manual_control_dependencies=true](dense_1_1/BiasAdd)}}' was changed by setting attribute after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-03-27 10:00:03.264644: W tensorflow/c/c_api.cc:291] Operation '{name:'total_1/Assign' id:503 op device:{requested: '', assigned: ''} def:{{{node total_1/Assign}} = AssignVariableOp[_has_manual_control_dependencies=true, dtype=DT_FLOAT, validate_shape=false](total_1, total_1/Initializer/zeros)}}' was changed by setting attribute after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
Plot ROC Curve
In [21]:
c1 = factory.GetROCCurve(loader)
c1.Draw()
close outputfile to save output file
In [22]:
outputFile.Close()
Draw all canvases
In [23]:
from ROOT import gROOT
gROOT.GetListOfCanvases().Draw()