TMVA_RNN_Classification¶
TMVA Classification Example Using a Recurrent Neural Network
This is an example of using a RNN in TMVA. We do classification using a toy time dependent data set that is generated when running this example macro
Author: Harshal Shende
This notebook tutorial was automatically generated with ROOTBOOK-izer from the macro found in the ROOT repository on Monday, March 27, 2023 at 09:58 AM.
TMVA Classification Example Using a Recurrent Neural Network
This is an example of using a RNN in TMVA.
We do the classification using a toy data set containing a time series of data sample ntimes
and with dimension ndim that is generated when running the provided function MakeTimeData (nevents, ntime, ndim)
In [1]:
import ROOT
num_threads = 4 # use max 4 threads
Welcome to JupyROOT 6.29/01
do enable MT running
In [2]:
if ROOT.gSystem.GetFromPipe("root-config --has-imt") == "yes":
ROOT.EnableImplicitMT(num_threads)
ROOT.gSystem.Setenv("OMP_NUM_THREADS", "1") # switch OFF MT in OpenBLAS
print("Running with nthreads = {}".format(ROOT.GetThreadPoolSize()))
else:
print("Running in serail mode since ROOT does not support MT")
TMVA = ROOT.TMVA
TFile = ROOT.TFile
import os
import importlib
TMVA.Tools.Instance()
TMVA.Config.Instance()
Running with nthreads = 4
Out[2]:
<cppyy.gbl.TMVA.Config object at 0xeb91d70>
Helper function to generate the time data set make some time data but not of fixed length. use a poisson with mu = 5 and truncated at 10
In [3]:
def MakeTimeData(n, ntime, ndim):
# ntime = 10;
# ndim = 30; // number of dim/time
fname = "time_data_t" + str(ntime) + "_d" + str(ndim) + ".root"
v1 = []
v2 = []
for i in range(ntime):
v1.append(ROOT.TH1D("h1_" + str(i), "h1", ndim, 0, 10))
v2.append(ROOT.TH1D("h2_" + str(i), "h2", ndim, 0, 10))
f1 = ROOT.TF1("f1", "gaus")
f2 = ROOT.TF1("f2", "gaus")
sgn = ROOT.TTree("sgn", "sgn")
bkg = ROOT.TTree("bkg", "bkg")
f = TFile(fname, "RECREATE")
x1 = []
x2 = []
for i in range(ntime):
x1.append(ROOT.std.vector["float"](ndim))
x2.append(ROOT.std.vector["float"](ndim))
for i in range(ntime):
bkg.Branch("vars_time" + str(i), "std::vector<float>", x1[i])
sgn.Branch("vars_time" + str(i), "std::vector<float>", x2[i])
sgn.SetDirectory(f)
bkg.SetDirectory(f)
ROOT.gRandom.SetSeed(0)
mean1 = ROOT.std.vector["double"](ntime)
mean2 = ROOT.std.vector["double"](ntime)
sigma1 = ROOT.std.vector["double"](ntime)
sigma2 = ROOT.std.vector["double"](ntime)
for j in range(ntime):
mean1[j] = 5.0 + 0.2 * ROOT.TMath.Sin(ROOT.TMath.Pi() * j / float(ntime))
mean2[j] = 5.0 + 0.2 * ROOT.TMath.Cos(ROOT.TMath.Pi() * j / float(ntime))
sigma1[j] = 4 + 0.3 * ROOT.TMath.Sin(ROOT.TMath.Pi() * j / float(ntime))
sigma2[j] = 4 + 0.3 * ROOT.TMath.Cos(ROOT.TMath.Pi() * j / float(ntime))
for i in range(n):
if i % 1000 == 0:
print("Generating event ... %d", i)
for j in range(ntime):
h1 = v1[j]
h2 = v2[j]
h1.Reset()
h2.Reset()
f1.SetParameters(1, mean1[j], sigma1[j])
f2.SetParameters(1, mean2[j], sigma2[j])
h1.FillRandom("f1", 1000)
h2.FillRandom("f2", 1000)
for k in range(ntime):
# std::cout << j*10+k << " ";
x1[j][k] = h1.GetBinContent(k + 1) + ROOT.gRandom.Gaus(0, 10)
x2[j][k] = h2.GetBinContent(k + 1) + ROOT.gRandom.Gaus(0, 10)
sgn.Fill()
bkg.Fill()
if n == 1:
c1 = ROOT.TCanvas()
c1.Divide(ntime, 2)
for j in range(ntime):
c1.cd(j + 1)
v1[j].Draw()
for j in range(ntime):
c1.cd(ntime + j + 1)
v2[j].Draw()
ROOT.gPad.Update()
if n > 1:
sgn.Write()
bkg.Write()
sgn.Print()
bkg.Print()
f.Close()
macro for performing a classification using a Recurrent Neural Network @param use_type use_type = 0 use Simple RNN network use_type = 1 use LSTM network use_type = 2 use GRU use_type = 3 build 3 different networks with RNN, LSTM and GRU
In [4]:
use_type = 1
ninput = 30
ntime = 10
batchSize = 100
maxepochs = 10
nTotEvts = 2000 # total events to be generated for signal or background
useKeras = True
useTMVA_RNN = True
useTMVA_DNN = True
useTMVA_BDT = False
tf_spec = importlib.util.find_spec("tensorflow")
if tf_spec is None:
useKeras = False
ROOT.Warning("TMVA_RNN_Classificaton","Skip using Keras since tensorflow is not installed")
rnn_types = ["RNN", "LSTM", "GRU"]
use_rnn_type = [1, 1, 1]
if 0 <= use_type < 3:
use_rnn_type = [0, 0, 0]
use_rnn_type[use_type] = 1
useGPU = True # use GPU for TMVA if available
useGPU = ROOT.gSystem.GetFromPipe("root-config --has-tmva-gpu") == "yes"
useTMVA_RNN = ROOT.gSystem.GetFromPipe("root-config --has-tmva-cpu") == "yes" or useGPU
if useTMVA_RNN:
ROOT.Warning(
"TMVA_RNN_Classification",
"TMVA is not build with GPU or CPU multi-thread support. Cannot use TMVA Deep Learning for RNN",
)
archString = "GPU" if useGPU else "CPU"
writeOutputFile = True
rnn_type = "RNN"
if ROOT.gSystem.GetFromPipe("root-config --has-tmva-pymva") == "yes":
TMVA.PyMethodBase.PyInitialize()
else:
useKeras = False
inputFileName = "time_data_t10_d30.root"
fileDoesNotExist = ROOT.gSystem.AccessPathName(inputFileName)
Warning in <TMVA_RNN_Classification>: TMVA is not build with GPU or CPU multi-thread support. Cannot use TMVA Deep Learning for RNN
if file does not exists create it
In [5]:
if fileDoesNotExist:
MakeTimeData(nTotEvts, ntime, ninput)
inputFile = TFile.Open(inputFileName)
if inputFile is None:
raise ROOT.Error("Error opening input file %s - exit", inputFileName.Data())
print("--- RNNClassification : Using input file: {}".format(inputFile.GetName()))
Generating event ... %d 0 Generating event ... %d 1000 --- RNNClassification : Using input file: time_data_t10_d30.root ****************************************************************************** *Tree :sgn : sgn * *Entries : 2000 : Total = 2693901 bytes File Size = 834468 * * : : Tree compression factor = 3.23 * ****************************************************************************** *Br 0 :vars_time0 : vector<float> * *Entries : 2000 : Total Size= 269305 bytes File Size = 83313 * *Baskets : 9 : Basket Size= 32000 bytes Compression= 3.23 * *............................................................................* *Br 1 :vars_time1 : vector<float> * *Entries : 2000 : Total Size= 269305 bytes File Size = 83331 * *Baskets : 9 : Basket Size= 32000 bytes Compression= 3.23 * *............................................................................* *Br 2 :vars_time2 : vector<float> * *Entries : 2000 : Total Size= 269305 bytes File Size = 83258 * *Baskets : 9 : Basket Size= 32000 bytes Compression= 3.23 * *............................................................................* *Br 3 :vars_time3 : vector<float> * *Entries : 2000 : Total Size= 269305 bytes File Size = 83324 * *Baskets : 9 : Basket Size= 32000 bytes Compression= 3.23 * *............................................................................* *Br 4 :vars_time4 : vector<float> * *Entries : 2000 : Total Size= 269305 bytes File Size = 83231 * *Baskets : 9 : Basket Size= 32000 bytes Compression= 3.23 * *............................................................................* *Br 5 :vars_time5 : vector<float> * *Entries : 2000 : Total Size= 269305 bytes File Size = 83380 * *Baskets : 9 : Basket Size= 32000 bytes Compression= 3.22 * *............................................................................* *Br 6 :vars_time6 : vector<float> * *Entries : 2000 : Total Size= 269305 bytes File Size = 83400 * *Baskets : 9 : Basket Size= 32000 bytes Compression= 3.22 * *............................................................................* *Br 7 :vars_time7 : vector<float> * *Entries : 2000 : Total Size= 269305 bytes File Size = 83261 * *Baskets : 9 : Basket Size= 32000 bytes Compression= 3.23 * *............................................................................* *Br 8 :vars_time8 : vector<float> * *Entries : 2000 : Total Size= 269305 bytes File Size = 83298 * *Baskets : 9 : Basket Size= 32000 bytes Compression= 3.23 * *............................................................................* *Br 9 :vars_time9 : vector<float> * *Entries : 2000 : Total Size= 269305 bytes File Size = 83276 * *Baskets : 9 : Basket Size= 32000 bytes Compression= 3.23 * *............................................................................* ****************************************************************************** *Tree :bkg : bkg * *Entries : 2000 : Total = 2693901 bytes File Size = 834661 * * : : Tree compression factor = 3.23 * ****************************************************************************** *Br 0 :vars_time0 : vector<float> * *Entries : 2000 : Total Size= 269305 bytes File Size = 83366 * *Baskets : 9 : Basket Size= 32000 bytes Compression= 3.22 * *............................................................................* *Br 1 :vars_time1 : vector<float> * *Entries : 2000 : Total Size= 269305 bytes File Size = 83300 * *Baskets : 9 : Basket Size= 32000 bytes Compression= 3.23 * *............................................................................* *Br 2 :vars_time2 : vector<float> * *Entries : 2000 : Total Size= 269305 bytes File Size = 83294 * *Baskets : 9 : Basket Size= 32000 bytes Compression= 3.23 * *............................................................................* *Br 3 :vars_time3 : vector<float> * *Entries : 2000 : Total Size= 269305 bytes File Size = 83304 * *Baskets : 9 : Basket Size= 32000 bytes Compression= 3.23 * *............................................................................* *Br 4 :vars_time4 : vector<float> * *Entries : 2000 : Total Size= 269305 bytes File Size = 83311 * *Baskets : 9 : Basket Size= 32000 bytes Compression= 3.23 * *............................................................................* *Br 5 :vars_time5 : vector<float> * *Entries : 2000 : Total Size= 269305 bytes File Size = 83319 * *Baskets : 9 : Basket Size= 32000 bytes Compression= 3.23 * *............................................................................* *Br 6 :vars_time6 : vector<float> * *Entries : 2000 : Total Size= 269305 bytes File Size = 83286 * *Baskets : 9 : Basket Size= 32000 bytes Compression= 3.23 * *............................................................................* *Br 7 :vars_time7 : vector<float> * *Entries : 2000 : Total Size= 269305 bytes File Size = 83387 * *Baskets : 9 : Basket Size= 32000 bytes Compression= 3.22 * *............................................................................* *Br 8 :vars_time8 : vector<float> * *Entries : 2000 : Total Size= 269305 bytes File Size = 83339 * *Baskets : 9 : Basket Size= 32000 bytes Compression= 3.22 * *............................................................................* *Br 9 :vars_time9 : vector<float> * *Entries : 2000 : Total Size= 269305 bytes File Size = 83357 * *Baskets : 9 : Basket Size= 32000 bytes Compression= 3.22 * *............................................................................*
Create a ROOT output file where TMVA will store ntuples, histograms, etc.
In [6]:
outfileName = "data_RNN_" + archString + ".root"
outputFile = None
if writeOutputFile:
outputFile = TFile.Open(outfileName, "RECREATE")
Declare Factory
Create the Factory class. Later you can choose the methods whose performance you'd like to investigate.
The factory is the major TMVA object you have to interact with. Here is the list of parameters you need to pass
- The first argument is the base of the name of all the output
weightfiles in the directory weight/ that will be created with the method parameters
The second argument is the output file for the training results
The third argument is a string option defining some general configuration for the TMVA session.
For example all TMVA output can be suppressed by removing the "!" (not) in front of the "Silent" argument in the option string
// Creating the factory object
In [7]:
factory = TMVA.Factory(
"TMVAClassification",
outputFile,
V=False,
Silent=False,
Color=True,
DrawProgressBar=True,
Transformations=None,
Correlations=False,
AnalysisType="Classification",
ModelPersistence=True,
)
dataloader = TMVA.DataLoader("dataset")
signalTree = inputFile.Get("sgn")
background = inputFile.Get("bkg")
nvar = ninput * ntime
add variables - use new AddVariablesArray function
In [8]:
for i in range(ntime):
dataloader.AddVariablesArray("vars_time" + str(i), ninput)
dataloader.AddSignalTree(signalTree, 1.0)
dataloader.AddBackgroundTree(background, 1.0)
DataSetInfo : [dataset] : Added class "Signal"
: Add Tree sgn of type Signal with 2000 events
DataSetInfo : [dataset] : Added class "Background"
: Add Tree bkg of type Background with 2000 events
check given input
In [9]:
datainfo = dataloader.GetDataSetInfo()
vars = datainfo.GetListOfVariables()
print("number of variables is {}".format(vars.size()))
for v in vars:
print(v)
nTrainSig = 0.8 * nTotEvts
nTrainBkg = 0.8 * nTotEvts
number of variables is 300 vars_time0[0] vars_time0[1] vars_time0[2] vars_time0[3] vars_time0[4] vars_time0[5] vars_time0[6] vars_time0[7] vars_time0[8] vars_time0[9] vars_time0[10] vars_time0[11] vars_time0[12] vars_time0[13] vars_time0[14] vars_time0[15] vars_time0[16] vars_time0[17] vars_time0[18] vars_time0[19] vars_time0[20] vars_time0[21] vars_time0[22] vars_time0[23] vars_time0[24] vars_time0[25] vars_time0[26] vars_time0[27] vars_time0[28] vars_time0[29] vars_time1[0] vars_time1[1] vars_time1[2] vars_time1[3] vars_time1[4] vars_time1[5] vars_time1[6] vars_time1[7] vars_time1[8] vars_time1[9] vars_time1[10] vars_time1[11] vars_time1[12] vars_time1[13] vars_time1[14] vars_time1[15] vars_time1[16] vars_time1[17] vars_time1[18] vars_time1[19] vars_time1[20] vars_time1[21] vars_time1[22] vars_time1[23] vars_time1[24] vars_time1[25] vars_time1[26] vars_time1[27] vars_time1[28] vars_time1[29] vars_time2[0] vars_time2[1] vars_time2[2] vars_time2[3] vars_time2[4] vars_time2[5] vars_time2[6] vars_time2[7] vars_time2[8] vars_time2[9] vars_time2[10] vars_time2[11] vars_time2[12] vars_time2[13] vars_time2[14] vars_time2[15] vars_time2[16] vars_time2[17] vars_time2[18] vars_time2[19] vars_time2[20] vars_time2[21] vars_time2[22] vars_time2[23] vars_time2[24] vars_time2[25] vars_time2[26] vars_time2[27] vars_time2[28] vars_time2[29] vars_time3[0] vars_time3[1] vars_time3[2] vars_time3[3] vars_time3[4] vars_time3[5] vars_time3[6] vars_time3[7] vars_time3[8] vars_time3[9] vars_time3[10] vars_time3[11] vars_time3[12] vars_time3[13] vars_time3[14] vars_time3[15] vars_time3[16] vars_time3[17] vars_time3[18] vars_time3[19] vars_time3[20] vars_time3[21] vars_time3[22] vars_time3[23] vars_time3[24] vars_time3[25] vars_time3[26] vars_time3[27] vars_time3[28] vars_time3[29] vars_time4[0] vars_time4[1] vars_time4[2] vars_time4[3] vars_time4[4] vars_time4[5] vars_time4[6] vars_time4[7] vars_time4[8] vars_time4[9] vars_time4[10] vars_time4[11] vars_time4[12] vars_time4[13] vars_time4[14] vars_time4[15] vars_time4[16] vars_time4[17] vars_time4[18] vars_time4[19] vars_time4[20] vars_time4[21] vars_time4[22] vars_time4[23] vars_time4[24] vars_time4[25] vars_time4[26] vars_time4[27] vars_time4[28] vars_time4[29] vars_time5[0] vars_time5[1] vars_time5[2] vars_time5[3] vars_time5[4] vars_time5[5] vars_time5[6] vars_time5[7] vars_time5[8] vars_time5[9] vars_time5[10] vars_time5[11] vars_time5[12] vars_time5[13] vars_time5[14] vars_time5[15] vars_time5[16] vars_time5[17] vars_time5[18] vars_time5[19] vars_time5[20] vars_time5[21] vars_time5[22] vars_time5[23] vars_time5[24] vars_time5[25] vars_time5[26] vars_time5[27] vars_time5[28] vars_time5[29] vars_time6[0] vars_time6[1] vars_time6[2] vars_time6[3] vars_time6[4] vars_time6[5] vars_time6[6] vars_time6[7] vars_time6[8] vars_time6[9] vars_time6[10] vars_time6[11] vars_time6[12] vars_time6[13] vars_time6[14] vars_time6[15] vars_time6[16] vars_time6[17] vars_time6[18] vars_time6[19] vars_time6[20] vars_time6[21] vars_time6[22] vars_time6[23] vars_time6[24] vars_time6[25] vars_time6[26] vars_time6[27] vars_time6[28] vars_time6[29] vars_time7[0] vars_time7[1] vars_time7[2] vars_time7[3] vars_time7[4] vars_time7[5] vars_time7[6] vars_time7[7] vars_time7[8] vars_time7[9] vars_time7[10] vars_time7[11] vars_time7[12] vars_time7[13] vars_time7[14] vars_time7[15] vars_time7[16] vars_time7[17] vars_time7[18] vars_time7[19] vars_time7[20] vars_time7[21] vars_time7[22] vars_time7[23] vars_time7[24] vars_time7[25] vars_time7[26] vars_time7[27] vars_time7[28] vars_time7[29] vars_time8[0] vars_time8[1] vars_time8[2] vars_time8[3] vars_time8[4] vars_time8[5] vars_time8[6] vars_time8[7] vars_time8[8] vars_time8[9] vars_time8[10] vars_time8[11] vars_time8[12] vars_time8[13] vars_time8[14] vars_time8[15] vars_time8[16] vars_time8[17] vars_time8[18] vars_time8[19] vars_time8[20] vars_time8[21] vars_time8[22] vars_time8[23] vars_time8[24] vars_time8[25] vars_time8[26] vars_time8[27] vars_time8[28] vars_time8[29] vars_time9[0] vars_time9[1] vars_time9[2] vars_time9[3] vars_time9[4] vars_time9[5] vars_time9[6] vars_time9[7] vars_time9[8] vars_time9[9] vars_time9[10] vars_time9[11] vars_time9[12] vars_time9[13] vars_time9[14] vars_time9[15] vars_time9[16] vars_time9[17] vars_time9[18] vars_time9[19] vars_time9[20] vars_time9[21] vars_time9[22] vars_time9[23] vars_time9[24] vars_time9[25] vars_time9[26] vars_time9[27] vars_time9[28] vars_time9[29]
Apply additional cuts on the signal and background samples (can be different)
In [10]:
mycuts = "" # for example: TCut mycuts = "abs(var1)<0.5 && abs(var2-0.5)<1";
mycutb = ""
build the string options for DataLoader::PrepareTrainingAndTestTree
In [11]:
dataloader.PrepareTrainingAndTestTree(
mycuts,
mycutb,
nTrain_Signal=nTrainSig,
nTrain_Background=nTrainBkg,
SplitMode="Random",
SplitSeed=100,
NormMode="NumEvents",
V=False,
CalcCorrelations=False,
)
print("prepared DATA LOADER ")
prepared DATA LOADER
Book TMVA recurrent models
Book the different types of recurrent models in TMVA (SimpleRNN, LSTM or GRU)
In [12]:
if useTMVA_RNN:
for i in range(3):
if not use_rnn_type[i]:
continue
rnn_type = rnn_types[i]
## Define RNN layer layout
## it should be LayerType (RNN or LSTM or GRU) | number of units | number of inputs | time steps | remember output (typically no=0 | return full sequence
rnnLayout = str(rnn_type) + "|10|" + str(ninput) + "|" + str(ntime) + "|0|1,RESHAPE|FLAT,DENSE|64|TANH,LINEAR"
## Defining Training strategies. Different training strings can be concatenate. Use however only one
trainingString1 = "LearningRate=1e-3,Momentum=0.0,Repetitions=1,ConvergenceSteps=5,BatchSize=" + str(batchSize)
trainingString1 += ",TestRepetitions=1,WeightDecay=1e-2,Regularization=None,MaxEpochs=" + str(maxepochs)
trainingString1 += "Optimizer=ADAM,DropConfig=0.0+0.+0.+0."
## define the inputlayout string for RNN
## the input data should be organize as following:
##/ input layout for RNN: time x ndim
## add after RNN a reshape layer (needed top flatten the output) and a dense layer with 64 units and a last one
## Note the last layer is linear because when using Crossentropy a Sigmoid is applied already
## Define the full RNN Noption string adding the final options for all network
rnnName = "TMVA_" + str(rnn_type)
factory.BookMethod(
dataloader,
TMVA.Types.kDL,
rnnName,
H=False,
V=True,
ErrorStrategy="CROSSENTROPY",
VarTransform=None,
WeightInitialization="XAVIERUNIFORM",
ValidationSize=0.2,
RandomSeed=1234,
InputLayout=str(ntime) + "|" + str(ninput),
Layout=rnnLayout,
TrainingStrategy=trainingString1,
Architecture=archString
)
Factory : Booking method: TMVA_LSTM
:
: Parsing option string:
: ... "!H:V:ErrorStrategy=CROSSENTROPY:VarTransform=None:WeightInitialization=XAVIERUNIFORM:ValidationSize=0.2:RandomSeed=1234:InputLayout=10|30:Layout=LSTM|10|30|10|0|1,RESHAPE|FLAT,DENSE|64|TANH,LINEAR:TrainingStrategy=LearningRate=1e-3,Momentum=0.0,Repetitions=1,ConvergenceSteps=5,BatchSize=100,TestRepetitions=1,WeightDecay=1e-2,Regularization=None,MaxEpochs=10Optimizer=ADAM,DropConfig=0.0+0.+0.+0.:Architecture=CPU"
: The following options are set:
: - By User:
: <none>
: - Default:
: Boost_num: "0" [Number of times the classifier will be boosted]
: Parsing option string:
: ... "!H:V:ErrorStrategy=CROSSENTROPY:VarTransform=None:WeightInitialization=XAVIERUNIFORM:ValidationSize=0.2:RandomSeed=1234:InputLayout=10|30:Layout=LSTM|10|30|10|0|1,RESHAPE|FLAT,DENSE|64|TANH,LINEAR:TrainingStrategy=LearningRate=1e-3,Momentum=0.0,Repetitions=1,ConvergenceSteps=5,BatchSize=100,TestRepetitions=1,WeightDecay=1e-2,Regularization=None,MaxEpochs=10Optimizer=ADAM,DropConfig=0.0+0.+0.+0.:Architecture=CPU"
: The following options are set:
: - By User:
: V: "True" [Verbose output (short form of "VerbosityLevel" below - overrides the latter one)]
: VarTransform: "None" [List of variable transformations performed before training, e.g., "D_Background,P_Signal,G,N_AllClasses" for: "Decorrelation, PCA-transformation, Gaussianisation, Normalisation, each for the given class of events ('AllClasses' denotes all events of all classes, if no class indication is given, 'All' is assumed)"]
: H: "False" [Print method-specific help message]
: InputLayout: "10|30" [The Layout of the input]
: Layout: "LSTM|10|30|10|0|1,RESHAPE|FLAT,DENSE|64|TANH,LINEAR" [Layout of the network.]
: ErrorStrategy: "CROSSENTROPY" [Loss function: Mean squared error (regression) or cross entropy (binary classification).]
: WeightInitialization: "XAVIERUNIFORM" [Weight initialization strategy]
: RandomSeed: "1234" [Random seed used for weight initialization and batch shuffling]
: ValidationSize: "0.2" [Part of the training data to use for validation. Specify as 0.2 or 20% to use a fifth of the data set as validation set. Specify as 100 to use exactly 100 events. (Default: 20%)]
: Architecture: "CPU" [Which architecture to perform the training on.]
: TrainingStrategy: "LearningRate=1e-3,Momentum=0.0,Repetitions=1,ConvergenceSteps=5,BatchSize=100,TestRepetitions=1,WeightDecay=1e-2,Regularization=None,MaxEpochs=10Optimizer=ADAM,DropConfig=0.0+0.+0.+0." [Defines the training strategies.]
: - Default:
: VerbosityLevel: "Default" [Verbosity level]
: CreateMVAPdfs: "False" [Create PDFs for classifier outputs (signal and background)]
: IgnoreNegWeightsInTraining: "False" [Events with negative weights are ignored in the training (but are included for testing and performance evaluation)]
: BatchLayout: "0|0|0" [The Layout of the batch]
: Will now use the CPU architecture with BLAS and IMT support !
Book TMVA fully connected dense layer models
In [13]:
if useTMVA_DNN:
# Method DL with Dense Layer
# Training strategies.
trainingString1 = ROOT.TString(
"LearningRate=1e-3,Momentum=0.0,Repetitions=1,"
"ConvergenceSteps=10,BatchSize=256,TestRepetitions=1,"
"WeightDecay=1e-4,Regularization=None,MaxEpochs=20"
"DropConfig=0.0+0.+0.+0.,Optimizer=ADAM:"
) # + "|" + trainingString2
# General Options.
trainingString1.Append(archString)
dnnName = "TMVA_DNN"
factory.BookMethod(
dataloader,
TMVA.Types.kDL,
dnnName,
H=False,
V=True,
ErrorStrategy="CROSSENTROPY",
VarTransform=None,
WeightInitialization="XAVIER",
RandomSeed=0,
InputLayout="1|1|" + str(ntime * ninput),
Layout="DENSE|64|TANH,DENSE|TANH|64,DENSE|TANH|64,LINEAR",
TrainingStrategy=trainingString1
)
Factory : Booking method: TMVA_DNN
:
: Parsing option string:
: ... "!H:V:ErrorStrategy=CROSSENTROPY:VarTransform=None:WeightInitialization=XAVIER:RandomSeed=0:InputLayout=1|1|300:Layout=DENSE|64|TANH,DENSE|TANH|64,DENSE|TANH|64,LINEAR:TrainingStrategy=LearningRate=1e-3,Momentum=0.0,Repetitions=1,ConvergenceSteps=10,BatchSize=256,TestRepetitions=1,WeightDecay=1e-4,Regularization=None,MaxEpochs=20DropConfig=0.0+0.+0.+0.,Optimizer=ADAM:CPU"
: The following options are set:
: - By User:
: <none>
: - Default:
: Boost_num: "0" [Number of times the classifier will be boosted]
: Parsing option string:
: ... "!H:V:ErrorStrategy=CROSSENTROPY:VarTransform=None:WeightInitialization=XAVIER:RandomSeed=0:InputLayout=1|1|300:Layout=DENSE|64|TANH,DENSE|TANH|64,DENSE|TANH|64,LINEAR:TrainingStrategy=LearningRate=1e-3,Momentum=0.0,Repetitions=1,ConvergenceSteps=10,BatchSize=256,TestRepetitions=1,WeightDecay=1e-4,Regularization=None,MaxEpochs=20DropConfig=0.0+0.+0.+0.,Optimizer=ADAM:CPU"
: The following options are set:
: - By User:
: V: "True" [Verbose output (short form of "VerbosityLevel" below - overrides the latter one)]
: VarTransform: "None" [List of variable transformations performed before training, e.g., "D_Background,P_Signal,G,N_AllClasses" for: "Decorrelation, PCA-transformation, Gaussianisation, Normalisation, each for the given class of events ('AllClasses' denotes all events of all classes, if no class indication is given, 'All' is assumed)"]
: H: "False" [Print method-specific help message]
: InputLayout: "1|1|300" [The Layout of the input]
: Layout: "DENSE|64|TANH,DENSE|TANH|64,DENSE|TANH|64,LINEAR" [Layout of the network.]
: ErrorStrategy: "CROSSENTROPY" [Loss function: Mean squared error (regression) or cross entropy (binary classification).]
: WeightInitialization: "XAVIER" [Weight initialization strategy]
: RandomSeed: "0" [Random seed used for weight initialization and batch shuffling]
: Architecture: "CPU" [Which architecture to perform the training on.]
: TrainingStrategy: "LearningRate=1e-3,Momentum=0.0,Repetitions=1,ConvergenceSteps=10,BatchSize=256,TestRepetitions=1,WeightDecay=1e-4,Regularization=None,MaxEpochs=20DropConfig=0.0+0.+0.+0.,Optimizer=ADAM" [Defines the training strategies.]
: - Default:
: VerbosityLevel: "Default" [Verbosity level]
: CreateMVAPdfs: "False" [Create PDFs for classifier outputs (signal and background)]
: IgnoreNegWeightsInTraining: "False" [Events with negative weights are ignored in the training (but are included for testing and performance evaluation)]
: BatchLayout: "0|0|0" [The Layout of the batch]
: ValidationSize: "20%" [Part of the training data to use for validation. Specify as 0.2 or 20% to use a fifth of the data set as validation set. Specify as 100 to use exactly 100 events. (Default: 20%)]
: Will now use the CPU architecture with BLAS and IMT support !
Book Keras recurrent models
Book the different types of recurrent models in Keras (SimpleRNN, LSTM or GRU)
In [14]:
if useKeras:
for i in range(3):
if use_rnn_type[i]:
modelName = "model_" + rnn_types[i] + ".h5"
trainedModelName = "trained_" + modelName
print("Building recurrent keras model using a", rnn_types[i], "layer")
# create python script which can be executed
# create 2 conv2d layer + maxpool + dense
from tensorflow.keras.models import Sequential
from tensorflow.keras.optimizers import Adam
# from keras.initializers import TruncatedNormal
# from keras import initializations
from tensorflow.keras.layers import Input, Dense, Dropout, Flatten, SimpleRNN, GRU, LSTM, Reshape, BatchNormalization
model = Sequential()
model.add(Reshape((10, 30), input_shape=(10 * 30,)))
# add recurrent neural network depending on type / Use option to return the full output
if rnn_types[i] == "LSTM":
model.add(LSTM(units=10, return_sequences=True))
elif rnn_types[i] == "GRU":
model.add(GRU(units=10, return_sequences=True))
else:
model.add(SimpleRNN(units=10, return_sequences=True))
# m.AddLine("model.add(BatchNormalization())");
model.add(Flatten()) # needed if returning the full time output sequence
model.add(Dense(64, activation="tanh"))
model.add(Dense(2, activation="sigmoid"))
model.compile(loss="binary_crossentropy", optimizer=Adam(learning_rate=0.001), weighted_metrics=["accuracy"])
model.save(modelName)
model.summary()
print("saved recurrent model", modelName)
if not os.path.exists(modelName):
useKeras = False
print("Error creating Keras recurrent model file - Skip using Keras")
else:
# book PyKeras method only if Keras model could be created
print("Booking Keras model ", rnn_types[i])
factory.BookMethod(
dataloader,
TMVA.Types.kPyKeras,
"PyKeras_" + rnn_types[i],
H=True,
V=False,
VarTransform=None,
FilenameModel=modelName,
FilenameTrainedModel="trained_" + modelName,
NumEpochs=maxepochs,
BatchSize=batchSize,
GpuOptions="allow_growth=True",
)
Building recurrent keras model using a LSTM layer
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
reshape (Reshape) (None, 10, 30) 0
lstm (LSTM) (None, 10, 10) 1640
flatten (Flatten) (None, 100) 0
dense (Dense) (None, 64) 6464
dense_1 (Dense) (None, 2) 130
=================================================================
Total params: 8,234
Trainable params: 8,234
Non-trainable params: 0
_________________________________________________________________
saved recurrent model model_LSTM.h5
Booking Keras model LSTM
Factory : Booking method: PyKeras_LSTM
:
: Setting up tf.keras
: Using TensorFlow version 2
: Use Keras version from TensorFlow : tf.keras
: Applying GPU option: gpu_options.allow_growth=True
: Loading Keras Model
: Loaded model from file: model_LSTM.h5
2023-03-27 09:59:08.331623: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2023-03-27 09:59:09.095615: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /home/sftnight/build/workspace/root-makedoc-master/rootspi/rdoc/src/master.build/lib 2023-03-27 09:59:09.095663: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine. 2023-03-27 09:59:10.507194: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /home/sftnight/build/workspace/root-makedoc-master/rootspi/rdoc/src/master.build/lib 2023-03-27 09:59:10.507378: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /home/sftnight/build/workspace/root-makedoc-master/rootspi/rdoc/src/master.build/lib 2023-03-27 09:59:10.507406: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly. 2023-03-27 09:59:12.532607: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /home/sftnight/build/workspace/root-makedoc-master/rootspi/rdoc/src/master.build/lib 2023-03-27 09:59:12.532643: W tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:265] failed call to cuInit: UNKNOWN ERROR (303) 2023-03-27 09:59:12.532681: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (root-ubuntu-2004-3): /proc/driver/nvidia/version does not exist 2023-03-27 09:59:12.533047: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
use BDT in case not using Keras or TMVA DL
In [15]:
if not useKeras or not useTMVA_BDT:
useTMVA_BDT = True
Book TMVA BDT
In [16]:
if useTMVA_BDT:
factory.BookMethod(
dataloader,
TMVA.Types.kBDT,
"BDTG",
H=True,
V=False,
NTrees=100,
MinNodeSize="2.5%",
BoostType="Grad",
Shrinkage=0.10,
UseBaggedBoost=True,
BaggedSampleFraction=0.5,
nCuts=20,
MaxDepth=2,
)
Factory : Booking method: BDTG
:
: the option NegWeightTreatment=InverseBoostNegWeights does not exist for BoostType=Grad
: --> change to new default NegWeightTreatment=Pray
: Rebuilding Dataset dataset
: Building event vectors for type 2 Signal
: Dataset[dataset] : create input formulas for tree sgn
: Using variable vars_time0[0] from array expression vars_time0 of size 30
: Using variable vars_time1[0] from array expression vars_time1 of size 30
: Using variable vars_time2[0] from array expression vars_time2 of size 30
: Using variable vars_time3[0] from array expression vars_time3 of size 30
: Using variable vars_time4[0] from array expression vars_time4 of size 30
: Using variable vars_time5[0] from array expression vars_time5 of size 30
: Using variable vars_time6[0] from array expression vars_time6 of size 30
: Using variable vars_time7[0] from array expression vars_time7 of size 30
: Using variable vars_time8[0] from array expression vars_time8 of size 30
: Using variable vars_time9[0] from array expression vars_time9 of size 30
: Building event vectors for type 2 Background
: Dataset[dataset] : create input formulas for tree bkg
: Using variable vars_time0[0] from array expression vars_time0 of size 30
: Using variable vars_time1[0] from array expression vars_time1 of size 30
: Using variable vars_time2[0] from array expression vars_time2 of size 30
: Using variable vars_time3[0] from array expression vars_time3 of size 30
: Using variable vars_time4[0] from array expression vars_time4 of size 30
: Using variable vars_time5[0] from array expression vars_time5 of size 30
: Using variable vars_time6[0] from array expression vars_time6 of size 30
: Using variable vars_time7[0] from array expression vars_time7 of size 30
: Using variable vars_time8[0] from array expression vars_time8 of size 30
: Using variable vars_time9[0] from array expression vars_time9 of size 30
DataSetFactory : [dataset] : Number of events in input trees
:
:
: Number of training and testing events
: ---------------------------------------------------------------------------
: Signal -- training events : 1600
: Signal -- testing events : 400
: Signal -- training and testing events: 2000
: Background -- training events : 1600
: Background -- testing events : 400
: Background -- training and testing events: 2000
:
Train all methods
In [17]:
factory.TrainAllMethods()
print("nthreads = {}".format(ROOT.GetThreadPoolSize()))
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
reshape (Reshape) (None, 10, 30) 0
lstm (LSTM) (None, 10, 10) 1640
flatten (Flatten) (None, 100) 0
dense (Dense) (None, 64) 6464
dense_1 (Dense) (None, 2) 130
=================================================================
Total params: 8,234
Trainable params: 8,234
Non-trainable params: 0
_________________________________________________________________
Epoch 1/10
19/26 [====================>.........] - ETA: 0s - loss: 0.7194 - accuracy: 0.5047
Epoch 1: val_loss improved from inf to 0.69546, saving model to trained_model_LSTM.h5
26/26 [==============================] - 4s 58ms/step - loss: 0.7139 - accuracy: 0.5090 - val_loss: 0.6955 - val_accuracy: 0.5078
Epoch 2/10
23/26 [=========================>....] - ETA: 0s - loss: 0.6946 - accuracy: 0.5174
Epoch 2: val_loss improved from 0.69546 to 0.68805, saving model to trained_model_LSTM.h5
26/26 [==============================] - 0s 15ms/step - loss: 0.6938 - accuracy: 0.5199 - val_loss: 0.6880 - val_accuracy: 0.5422
Epoch 3/10
23/26 [=========================>....] - ETA: 0s - loss: 0.6872 - accuracy: 0.5583
Epoch 3: val_loss improved from 0.68805 to 0.68795, saving model to trained_model_LSTM.h5
26/26 [==============================] - 0s 13ms/step - loss: 0.6862 - accuracy: 0.5617 - val_loss: 0.6880 - val_accuracy: 0.5297
Epoch 4/10
25/26 [===========================>..] - ETA: 0s - loss: 0.6805 - accuracy: 0.5688
Epoch 4: val_loss did not improve from 0.68795
26/26 [==============================] - 0s 13ms/step - loss: 0.6799 - accuracy: 0.5707 - val_loss: 0.6880 - val_accuracy: 0.5437
Epoch 5/10
24/26 [==========================>...] - ETA: 0s - loss: 0.6739 - accuracy: 0.5833
Epoch 5: val_loss improved from 0.68795 to 0.68428, saving model to trained_model_LSTM.h5
26/26 [==============================] - 0s 13ms/step - loss: 0.6733 - accuracy: 0.5844 - val_loss: 0.6843 - val_accuracy: 0.5641
Epoch 6/10
26/26 [==============================] - ETA: 0s - loss: 0.6682 - accuracy: 0.5926
Epoch 6: val_loss improved from 0.68428 to 0.68329, saving model to trained_model_LSTM.h5
26/26 [==============================] - 0s 13ms/step - loss: 0.6682 - accuracy: 0.5926 - val_loss: 0.6833 - val_accuracy: 0.5531
Epoch 7/10
21/26 [=======================>......] - ETA: 0s - loss: 0.6643 - accuracy: 0.5929
Epoch 7: val_loss improved from 0.68329 to 0.68269, saving model to trained_model_LSTM.h5
26/26 [==============================] - 0s 12ms/step - loss: 0.6628 - accuracy: 0.5984 - val_loss: 0.6827 - val_accuracy: 0.5516
Epoch 8/10
26/26 [==============================] - ETA: 0s - loss: 0.6582 - accuracy: 0.6129
Epoch 8: val_loss did not improve from 0.68269
26/26 [==============================] - 0s 12ms/step - loss: 0.6582 - accuracy: 0.6129 - val_loss: 0.6873 - val_accuracy: 0.5609
Epoch 9/10
20/26 [======================>.......] - ETA: 0s - loss: 0.6549 - accuracy: 0.6055
Epoch 9: val_loss did not improve from 0.68269
26/26 [==============================] - 0s 12ms/step - loss: 0.6543 - accuracy: 0.6070 - val_loss: 0.6933 - val_accuracy: 0.5688
Epoch 10/10
23/26 [=========================>....] - ETA: 0s - loss: 0.6509 - accuracy: 0.6200
Epoch 10: val_loss did not improve from 0.68269
26/26 [==============================] - 0s 14ms/step - loss: 0.6510 - accuracy: 0.6199 - val_loss: 0.6873 - val_accuracy: 0.5609
/usr/local/lib/python3.8/dist-packages/keras/engine/training_v1.py:2357: UserWarning: `Model.state_updates` will be removed in a future version. This property should not be used in TensorFlow 2.0, as `updates` are applied automatically. updates=self.state_updates,
nthreads = 4 Factory : Train all methods Factory : Train method: TMVA_LSTM for Classification : : Start of deep neural network training on CPU using MT, nthreads = 4 : : ***** Deep Learning Network ***** DEEP NEURAL NETWORK: Depth = 4 Input = ( 10, 1, 30 ) Batch size = 100 Loss function = C Layer 0 LSTM Layer: (NInput = 30, NState = 10, NTime = 10 ) Output = ( 100 , 10 , 10 ) Layer 1 RESHAPE Layer Input = ( 1 , 10 , 10 ) Output = ( 1 , 100 , 100 ) Layer 2 DENSE Layer: ( Input = 100 , Width = 64 ) Output = ( 1 , 100 , 64 ) Activation Function = Tanh Layer 3 DENSE Layer: ( Input = 64 , Width = 1 ) Output = ( 1 , 100 , 1 ) Activation Function = Identity : Using 2560 events for training and 640 for testing : Compute initial loss on the validation data : Training phase 1 of 1: Optimizer ADAM (beta1=0.9,beta2=0.999,eps=1e-07) Learning rate = 0.001 regularization 0 minimum error = 0.704951 : -------------------------------------------------------------- : Epoch | Train Err. Val. Err. t(s)/epoch t(s)/Loss nEvents/s Conv. Steps : -------------------------------------------------------------- : Start epoch iteration ... : 1 Minimum Test error found - save the configuration : 1 | 0.705989 0.694371 0.356304 0.0307382 7678.95 0 : 2 | 0.698472 0.699005 0.376711 0.0334483 7283.05 1 : 3 | 0.695936 0.699498 0.345582 0.0302009 7926.91 2 : 4 | 0.687913 0.70797 0.343871 0.0289042 7937.35 3 : 5 | 0.683866 0.696301 0.340926 0.0287914 8009.37 4 : 6 | 0.680696 0.697659 0.333539 0.0290075 8209.34 5 : 7 | 0.676794 0.697128 0.338384 0.0291015 8083.24 6 : : Elapsed time for training with 3200 events: 2.47 sec : Evaluate deep neural network on CPU using batches with size = 100 : TMVA_LSTM : [dataset] : Evaluation of TMVA_LSTM on training sample (3200 events) : Elapsed time for evaluation of 3200 events: 0.156 sec : Creating xml weight file: dataset/weights/TMVAClassification_TMVA_LSTM.weights.xml : Creating standalone class: dataset/weights/TMVAClassification_TMVA_LSTM.class.C Factory : Training finished : Factory : Train method: TMVA_DNN for Classification : : Start of deep neural network training on CPU using MT, nthreads = 4 : : ***** Deep Learning Network ***** DEEP NEURAL NETWORK: Depth = 4 Input = ( 1, 1, 300 ) Batch size = 256 Loss function = C Layer 0 DENSE Layer: ( Input = 300 , Width = 64 ) Output = ( 1 , 256 , 64 ) Activation Function = Tanh Layer 1 DENSE Layer: ( Input = 64 , Width = 64 ) Output = ( 1 , 256 , 64 ) Activation Function = Tanh Layer 2 DENSE Layer: ( Input = 64 , Width = 64 ) Output = ( 1 , 256 , 64 ) Activation Function = Tanh Layer 3 DENSE Layer: ( Input = 64 , Width = 1 ) Output = ( 1 , 256 , 1 ) Activation Function = Identity : Using 2560 events for training and 640 for testing : Compute initial loss on the validation data : Training phase 1 of 1: Optimizer ADAM (beta1=0.9,beta2=0.999,eps=1e-07) Learning rate = 0.001 regularization 0 minimum error = 0.959875 : -------------------------------------------------------------- : Epoch | Train Err. Val. Err. t(s)/epoch t(s)/Loss nEvents/s Conv. Steps : -------------------------------------------------------------- : Start epoch iteration ... : 1 Minimum Test error found - save the configuration : 1 | 0.796216 0.7346 0.201109 0.015894 13821.8 0 : 2 Minimum Test error found - save the configuration : 2 | 0.704905 0.717665 0.203066 0.015409 13641.9 0 : 3 Minimum Test error found - save the configuration : 3 | 0.688179 0.706443 0.198062 0.015442 14018.2 0 : 4 | 0.678704 0.707595 0.195428 0.015207 14204.8 1 : 5 | 0.672644 0.709861 0.202065 0.0155571 13726 2 : 6 Minimum Test error found - save the configuration : 6 | 0.669364 0.704199 0.202268 0.0155746 13712.3 0 : 7 | 0.667896 0.709128 0.216174 0.015084 12730.6 1 : 8 | 0.657214 0.710828 0.205342 0.015388 13476.9 2 : 9 | 0.656043 0.711339 0.207562 0.0155883 13335.1 3 : 10 | 0.653297 0.722629 0.198402 0.0154109 13989.8 4 : 11 | 0.648449 0.713304 0.198033 0.0155555 14029.2 5 : 12 | 0.643214 0.718417 0.273308 0.0160841 9952.4 6 : 13 | 0.643446 0.716826 0.197807 0.0156908 14056.9 7 : 14 | 0.644262 0.720914 0.199459 0.015622 13925.4 8 : 15 | 0.638272 0.733214 0.29173 0.0164953 9301.16 9 : 16 | 0.623468 0.725556 0.231793 0.0157688 11850.5 10 : 17 | 0.617931 0.713273 0.309709 0.0311426 9189.9 11 : : Elapsed time for training with 3200 events: 3.75 sec : Evaluate deep neural network on CPU using batches with size = 256 : TMVA_DNN : [dataset] : Evaluation of TMVA_DNN on training sample (3200 events) : Elapsed time for evaluation of 3200 events: 0.165 sec : Creating xml weight file: dataset/weights/TMVAClassification_TMVA_DNN.weights.xml : Creating standalone class: dataset/weights/TMVAClassification_TMVA_DNN.class.C Factory : Training finished : Factory : Train method: PyKeras_LSTM for Classification : : : ================================================================ : H e l p f o r M V A m e t h o d [ PyKeras_LSTM ] : : : Keras is a high-level API for the Theano and Tensorflow packages. : This method wraps the training and predictions steps of the Keras : Python package for TMVA, so that dataloading, preprocessing and : evaluation can be done within the TMVA system. To use this Keras : interface, you have to generate a model with Keras first. Then, : this model can be loaded and trained in TMVA. : : : <Suppress this message by specifying "!H" in the booking option> : ================================================================ : : Split TMVA training data in 2560 training events and 640 validation events : Training Model Summary : Option SaveBestOnly: Only model weights with smallest validation loss will be stored : Getting training history for item:0 name = 'loss' : Getting training history for item:1 name = 'accuracy' : Getting training history for item:2 name = 'val_loss' : Getting training history for item:3 name = 'val_accuracy' : Elapsed time for training with 3200 events: 8.25 sec : Setting up tf.keras : Using TensorFlow version 2 : Use Keras version from TensorFlow : tf.keras : Applying GPU option: gpu_options.allow_growth=True : Disabled TF eager execution when evaluating model : Loading Keras Model : Loaded model from file: trained_model_LSTM.h5 PyKeras_LSTM : [dataset] : Evaluation of PyKeras_LSTM on training sample (3200 events) : Elapsed time for evaluation of 3200 events: 0.341 sec : Creating xml weight file: dataset/weights/TMVAClassification_PyKeras_LSTM.weights.xml : Creating standalone class: dataset/weights/TMVAClassification_PyKeras_LSTM.class.C Factory : Training finished : Factory : Train method: BDTG for Classification : : : ================================================================ : H e l p f o r M V A m e t h o d [ BDTG ] : : : --- Short description: : : Boosted Decision Trees are a collection of individual decision : trees which form a multivariate classifier by (weighted) majority : vote of the individual trees. Consecutive decision trees are : trained using the original training data set with re-weighted : events. By default, the AdaBoost method is employed, which gives : events that were misclassified in the previous tree a larger : weight in the training of the following tree. : : Decision trees are a sequence of binary splits of the data sample : using a single discriminant variable at a time. A test event : ending up after the sequence of left-right splits in a final : ("leaf") node is classified as either signal or background : depending on the majority type of training events in that node. : : --- Performance optimisation: : : By the nature of the binary splits performed on the individual : variables, decision trees do not deal well with linear correlations : between variables (they need to approximate the linear split in : the two dimensional space by a sequence of splits on the two : variables individually). Hence decorrelation could be useful : to optimise the BDT performance. : : --- Performance tuning via configuration options: : : The two most important parameters in the configuration are the : minimal number of events requested by a leaf node as percentage of the : number of training events (option "MinNodeSize" replacing the actual number : of events "nEventsMin" as given in earlier versions : If this number is too large, detailed features : in the parameter space are hard to be modelled. If it is too small, : the risk to overtrain rises and boosting seems to be less effective : typical values from our current experience for best performance : are between 0.5(%) and 10(%) : : The default minimal number is currently set to : max(20, (N_training_events / N_variables^2 / 10)) : and can be changed by the user. : : The other crucial parameter, the pruning strength ("PruneStrength"), : is also related to overtraining. It is a regularisation parameter : that is used when determining after the training which splits : are considered statistically insignificant and are removed. The : user is advised to carefully watch the BDT screen output for : the comparison between efficiencies obtained on the training and : the independent test sample. They should be equal within statistical : errors, in order to minimize statistical fluctuations in different samples. : : <Suppress this message by specifying "!H" in the booking option> : ================================================================ : BDTG : #events: (reweighted) sig: 1600 bkg: 1600 : #events: (unweighted) sig: 1600 bkg: 1600 : Training 100 Decision Trees ... patience please : Elapsed time for training with 3200 events: 1.28 sec BDTG : [dataset] : Evaluation of BDTG on training sample (3200 events) : Elapsed time for evaluation of 3200 events: 0.0181 sec : Creating xml weight file: dataset/weights/TMVAClassification_BDTG.weights.xml : Creating standalone class: dataset/weights/TMVAClassification_BDTG.class.C : data_RNN_CPU.root:/dataset/Method_BDT/BDTG Factory : Training finished : : Ranking input variables (method specific)... : No variable ranking supplied by classifier: TMVA_LSTM : No variable ranking supplied by classifier: TMVA_DNN : No variable ranking supplied by classifier: PyKeras_LSTM BDTG : Ranking result (top variable is best ranked) : -------------------------------------------- : Rank : Variable : Variable Importance : -------------------------------------------- : 1 : vars_time8 : 2.214e-02 : 2 : vars_time8 : 2.209e-02 : 3 : vars_time7 : 2.192e-02 : 4 : vars_time7 : 2.128e-02 : 5 : vars_time0 : 2.089e-02 : 6 : vars_time8 : 2.056e-02 : 7 : vars_time9 : 1.864e-02 : 8 : vars_time0 : 1.858e-02 : 9 : vars_time0 : 1.813e-02 : 10 : vars_time6 : 1.773e-02 : 11 : vars_time6 : 1.762e-02 : 12 : vars_time9 : 1.664e-02 : 13 : vars_time8 : 1.595e-02 : 14 : vars_time9 : 1.572e-02 : 15 : vars_time9 : 1.556e-02 : 16 : vars_time8 : 1.514e-02 : 17 : vars_time1 : 1.499e-02 : 18 : vars_time5 : 1.443e-02 : 19 : vars_time1 : 1.440e-02 : 20 : vars_time0 : 1.433e-02 : 21 : vars_time5 : 1.417e-02 : 22 : vars_time0 : 1.401e-02 : 23 : vars_time9 : 1.328e-02 : 24 : vars_time5 : 1.317e-02 : 25 : vars_time4 : 1.304e-02 : 26 : vars_time8 : 1.302e-02 : 27 : vars_time6 : 1.251e-02 : 28 : vars_time1 : 1.249e-02 : 29 : vars_time0 : 1.244e-02 : 30 : vars_time6 : 1.222e-02 : 31 : vars_time5 : 1.220e-02 : 32 : vars_time3 : 1.219e-02 : 33 : vars_time1 : 1.193e-02 : 34 : vars_time2 : 1.191e-02 : 35 : vars_time6 : 1.190e-02 : 36 : vars_time1 : 1.189e-02 : 37 : vars_time4 : 1.188e-02 : 38 : vars_time7 : 1.173e-02 : 39 : vars_time4 : 1.148e-02 : 40 : vars_time1 : 1.138e-02 : 41 : vars_time4 : 1.127e-02 : 42 : vars_time7 : 1.116e-02 : 43 : vars_time7 : 1.115e-02 : 44 : vars_time3 : 1.114e-02 : 45 : vars_time6 : 1.099e-02 : 46 : vars_time1 : 1.082e-02 : 47 : vars_time3 : 1.073e-02 : 48 : vars_time9 : 1.043e-02 : 49 : vars_time5 : 1.018e-02 : 50 : vars_time8 : 1.005e-02 : 51 : vars_time0 : 9.970e-03 : 52 : vars_time5 : 9.933e-03 : 53 : vars_time2 : 9.765e-03 : 54 : vars_time6 : 9.505e-03 : 55 : vars_time4 : 9.342e-03 : 56 : vars_time1 : 9.224e-03 : 57 : vars_time2 : 9.190e-03 : 58 : vars_time5 : 9.097e-03 : 59 : vars_time7 : 9.058e-03 : 60 : vars_time2 : 9.012e-03 : 61 : vars_time8 : 8.856e-03 : 62 : vars_time4 : 8.790e-03 : 63 : vars_time3 : 8.490e-03 : 64 : vars_time7 : 8.372e-03 : 65 : vars_time9 : 8.358e-03 : 66 : vars_time9 : 8.276e-03 : 67 : vars_time5 : 8.255e-03 : 68 : vars_time6 : 7.862e-03 : 69 : vars_time8 : 7.845e-03 : 70 : vars_time3 : 7.626e-03 : 71 : vars_time7 : 7.325e-03 : 72 : vars_time4 : 7.268e-03 : 73 : vars_time5 : 6.971e-03 : 74 : vars_time6 : 6.922e-03 : 75 : vars_time3 : 6.907e-03 : 76 : vars_time3 : 6.733e-03 : 77 : vars_time1 : 6.709e-03 : 78 : vars_time2 : 6.613e-03 : 79 : vars_time8 : 6.608e-03 : 80 : vars_time0 : 6.364e-03 : 81 : vars_time2 : 6.150e-03 : 82 : vars_time2 : 6.073e-03 : 83 : vars_time4 : 5.334e-03 : 84 : vars_time5 : 5.247e-03 : 85 : vars_time1 : 5.178e-03 : 86 : vars_time9 : 5.172e-03 : 87 : vars_time3 : 4.574e-03 : 88 : vars_time2 : 3.562e-03 : 89 : vars_time0 : 0.000e+00 : 90 : vars_time0 : 0.000e+00 : 91 : vars_time0 : 0.000e+00 : 92 : vars_time0 : 0.000e+00 : 93 : vars_time0 : 0.000e+00 : 94 : vars_time0 : 0.000e+00 : 95 : vars_time0 : 0.000e+00 : 96 : vars_time0 : 0.000e+00 : 97 : vars_time0 : 0.000e+00 : 98 : vars_time0 : 0.000e+00 : 99 : vars_time0 : 0.000e+00 : 100 : vars_time0 : 0.000e+00 : 101 : vars_time0 : 0.000e+00 : 102 : vars_time0 : 0.000e+00 : 103 : vars_time0 : 0.000e+00 : 104 : vars_time0 : 0.000e+00 : 105 : vars_time0 : 0.000e+00 : 106 : vars_time0 : 0.000e+00 : 107 : vars_time0 : 0.000e+00 : 108 : vars_time0 : 0.000e+00 : 109 : vars_time0 : 0.000e+00 : 110 : vars_time0 : 0.000e+00 : 111 : vars_time1 : 0.000e+00 : 112 : vars_time1 : 0.000e+00 : 113 : vars_time1 : 0.000e+00 : 114 : vars_time1 : 0.000e+00 : 115 : vars_time1 : 0.000e+00 : 116 : vars_time1 : 0.000e+00 : 117 : vars_time1 : 0.000e+00 : 118 : vars_time1 : 0.000e+00 : 119 : vars_time1 : 0.000e+00 : 120 : vars_time1 : 0.000e+00 : 121 : vars_time1 : 0.000e+00 : 122 : vars_time1 : 0.000e+00 : 123 : vars_time1 : 0.000e+00 : 124 : vars_time1 : 0.000e+00 : 125 : vars_time1 : 0.000e+00 : 126 : vars_time1 : 0.000e+00 : 127 : vars_time1 : 0.000e+00 : 128 : vars_time1 : 0.000e+00 : 129 : vars_time1 : 0.000e+00 : 130 : vars_time1 : 0.000e+00 : 131 : vars_time2 : 0.000e+00 : 132 : vars_time2 : 0.000e+00 : 133 : vars_time2 : 0.000e+00 : 134 : vars_time2 : 0.000e+00 : 135 : vars_time2 : 0.000e+00 : 136 : vars_time2 : 0.000e+00 : 137 : vars_time2 : 0.000e+00 : 138 : vars_time2 : 0.000e+00 : 139 : vars_time2 : 0.000e+00 : 140 : vars_time2 : 0.000e+00 : 141 : vars_time2 : 0.000e+00 : 142 : vars_time2 : 0.000e+00 : 143 : vars_time2 : 0.000e+00 : 144 : vars_time2 : 0.000e+00 : 145 : vars_time2 : 0.000e+00 : 146 : vars_time2 : 0.000e+00 : 147 : vars_time2 : 0.000e+00 : 148 : vars_time2 : 0.000e+00 : 149 : vars_time2 : 0.000e+00 : 150 : vars_time2 : 0.000e+00 : 151 : vars_time2 : 0.000e+00 : 152 : vars_time2 : 0.000e+00 : 153 : vars_time3 : 0.000e+00 : 154 : vars_time3 : 0.000e+00 : 155 : vars_time3 : 0.000e+00 : 156 : vars_time3 : 0.000e+00 : 157 : vars_time3 : 0.000e+00 : 158 : vars_time3 : 0.000e+00 : 159 : vars_time3 : 0.000e+00 : 160 : vars_time3 : 0.000e+00 : 161 : vars_time3 : 0.000e+00 : 162 : vars_time3 : 0.000e+00 : 163 : vars_time3 : 0.000e+00 : 164 : vars_time3 : 0.000e+00 : 165 : vars_time3 : 0.000e+00 : 166 : vars_time3 : 0.000e+00 : 167 : vars_time3 : 0.000e+00 : 168 : vars_time3 : 0.000e+00 : 169 : vars_time3 : 0.000e+00 : 170 : vars_time3 : 0.000e+00 : 171 : vars_time3 : 0.000e+00 : 172 : vars_time3 : 0.000e+00 : 173 : vars_time3 : 0.000e+00 : 174 : vars_time3 : 0.000e+00 : 175 : vars_time4 : 0.000e+00 : 176 : vars_time4 : 0.000e+00 : 177 : vars_time4 : 0.000e+00 : 178 : vars_time4 : 0.000e+00 : 179 : vars_time4 : 0.000e+00 : 180 : vars_time4 : 0.000e+00 : 181 : vars_time4 : 0.000e+00 : 182 : vars_time4 : 0.000e+00 : 183 : vars_time4 : 0.000e+00 : 184 : vars_time4 : 0.000e+00 : 185 : vars_time4 : 0.000e+00 : 186 : vars_time4 : 0.000e+00 : 187 : vars_time4 : 0.000e+00 : 188 : vars_time4 : 0.000e+00 : 189 : vars_time4 : 0.000e+00 : 190 : vars_time4 : 0.000e+00 : 191 : vars_time4 : 0.000e+00 : 192 : vars_time4 : 0.000e+00 : 193 : vars_time4 : 0.000e+00 : 194 : vars_time4 : 0.000e+00 : 195 : vars_time4 : 0.000e+00 : 196 : vars_time4 : 0.000e+00 : 197 : vars_time5 : 0.000e+00 : 198 : vars_time5 : 0.000e+00 : 199 : vars_time5 : 0.000e+00 : 200 : vars_time5 : 0.000e+00 : 201 : vars_time5 : 0.000e+00 : 202 : vars_time5 : 0.000e+00 : 203 : vars_time5 : 0.000e+00 : 204 : vars_time5 : 0.000e+00 : 205 : vars_time5 : 0.000e+00 : 206 : vars_time5 : 0.000e+00 : 207 : vars_time5 : 0.000e+00 : 208 : vars_time5 : 0.000e+00 : 209 : vars_time5 : 0.000e+00 : 210 : vars_time5 : 0.000e+00 : 211 : vars_time5 : 0.000e+00 : 212 : vars_time5 : 0.000e+00 : 213 : vars_time5 : 0.000e+00 : 214 : vars_time5 : 0.000e+00 : 215 : vars_time5 : 0.000e+00 : 216 : vars_time5 : 0.000e+00 : 217 : vars_time6 : 0.000e+00 : 218 : vars_time6 : 0.000e+00 : 219 : vars_time6 : 0.000e+00 : 220 : vars_time6 : 0.000e+00 : 221 : vars_time6 : 0.000e+00 : 222 : vars_time6 : 0.000e+00 : 223 : vars_time6 : 0.000e+00 : 224 : vars_time6 : 0.000e+00 : 225 : vars_time6 : 0.000e+00 : 226 : vars_time6 : 0.000e+00 : 227 : vars_time6 : 0.000e+00 : 228 : vars_time6 : 0.000e+00 : 229 : vars_time6 : 0.000e+00 : 230 : vars_time6 : 0.000e+00 : 231 : vars_time6 : 0.000e+00 : 232 : vars_time6 : 0.000e+00 : 233 : vars_time6 : 0.000e+00 : 234 : vars_time6 : 0.000e+00 : 235 : vars_time6 : 0.000e+00 : 236 : vars_time6 : 0.000e+00 : 237 : vars_time6 : 0.000e+00 : 238 : vars_time7 : 0.000e+00 : 239 : vars_time7 : 0.000e+00 : 240 : vars_time7 : 0.000e+00 : 241 : vars_time7 : 0.000e+00 : 242 : vars_time7 : 0.000e+00 : 243 : vars_time7 : 0.000e+00 : 244 : vars_time7 : 0.000e+00 : 245 : vars_time7 : 0.000e+00 : 246 : vars_time7 : 0.000e+00 : 247 : vars_time7 : 0.000e+00 : 248 : vars_time7 : 0.000e+00 : 249 : vars_time7 : 0.000e+00 : 250 : vars_time7 : 0.000e+00 : 251 : vars_time7 : 0.000e+00 : 252 : vars_time7 : 0.000e+00 : 253 : vars_time7 : 0.000e+00 : 254 : vars_time7 : 0.000e+00 : 255 : vars_time7 : 0.000e+00 : 256 : vars_time7 : 0.000e+00 : 257 : vars_time7 : 0.000e+00 : 258 : vars_time7 : 0.000e+00 : 259 : vars_time7 : 0.000e+00 : 260 : vars_time8 : 0.000e+00 : 261 : vars_time8 : 0.000e+00 : 262 : vars_time8 : 0.000e+00 : 263 : vars_time8 : 0.000e+00 : 264 : vars_time8 : 0.000e+00 : 265 : vars_time8 : 0.000e+00 : 266 : vars_time8 : 0.000e+00 : 267 : vars_time8 : 0.000e+00 : 268 : vars_time8 : 0.000e+00 : 269 : vars_time8 : 0.000e+00 : 270 : vars_time8 : 0.000e+00 : 271 : vars_time8 : 0.000e+00 : 272 : vars_time8 : 0.000e+00 : 273 : vars_time8 : 0.000e+00 : 274 : vars_time8 : 0.000e+00 : 275 : vars_time8 : 0.000e+00 : 276 : vars_time8 : 0.000e+00 : 277 : vars_time8 : 0.000e+00 : 278 : vars_time8 : 0.000e+00 : 279 : vars_time8 : 0.000e+00 : 280 : vars_time9 : 0.000e+00 : 281 : vars_time9 : 0.000e+00 : 282 : vars_time9 : 0.000e+00 : 283 : vars_time9 : 0.000e+00 : 284 : vars_time9 : 0.000e+00 : 285 : vars_time9 : 0.000e+00 : 286 : vars_time9 : 0.000e+00 : 287 : vars_time9 : 0.000e+00 : 288 : vars_time9 : 0.000e+00 : 289 : vars_time9 : 0.000e+00 : 290 : vars_time9 : 0.000e+00 : 291 : vars_time9 : 0.000e+00 : 292 : vars_time9 : 0.000e+00 : 293 : vars_time9 : 0.000e+00 : 294 : vars_time9 : 0.000e+00 : 295 : vars_time9 : 0.000e+00 : 296 : vars_time9 : 0.000e+00 : 297 : vars_time9 : 0.000e+00 : 298 : vars_time9 : 0.000e+00 : 299 : vars_time9 : 0.000e+00 : 300 : vars_time9 : 0.000e+00 : -------------------------------------------- TH1.Print Name = TrainingHistory_TMVA_LSTM_trainingError, Entries= 0, Total sum= 4.82967 TH1.Print Name = TrainingHistory_TMVA_LSTM_valError, Entries= 0, Total sum= 4.89193 TH1.Print Name = TrainingHistory_TMVA_DNN_trainingError, Entries= 0, Total sum= 11.3035 TH1.Print Name = TrainingHistory_TMVA_DNN_valError, Entries= 0, Total sum= 12.1758 TH1.Print Name = TrainingHistory_PyKeras_LSTM_'accuracy', Entries= 0, Total sum= 5.77656 TH1.Print Name = TrainingHistory_PyKeras_LSTM_'loss', Entries= 0, Total sum= 6.74165 TH1.Print Name = TrainingHistory_PyKeras_LSTM_'val_accuracy', Entries= 0, Total sum= 5.48281 TH1.Print Name = TrainingHistory_PyKeras_LSTM_'val_loss', Entries= 0, Total sum= 6.87763 Factory : === Destroy and recreate all methods via weight files for testing === : : Reading weight file: dataset/weights/TMVAClassification_TMVA_LSTM.weights.xml : Reading weight file: dataset/weights/TMVAClassification_TMVA_DNN.weights.xml : Reading weight file: dataset/weights/TMVAClassification_PyKeras_LSTM.weights.xml : Reading weight file: dataset/weights/TMVAClassification_BDTG.weights.xml
2023-03-27 09:59:30.269923: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:357] MLIR V1 optimization pass is not enabled
2023-03-27 09:59:30.301207: W tensorflow/c/c_api.cc:291] Operation '{name:'dense_1/kernel/Assign' id:300 op device:{requested: '', assigned: ''} def:{{{node dense_1/kernel/Assign}} = AssignVariableOp[_has_manual_control_dependencies=true, dtype=DT_FLOAT, validate_shape=false](dense_1/kernel, dense_1/kernel/Initializer/stateless_random_uniform)}}' was changed by setting attribute after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-03-27 09:59:30.575369: W tensorflow/c/c_api.cc:291] Operation '{name:'dense_1/Sigmoid' id:311 op device:{requested: '', assigned: ''} def:{{{node dense_1/Sigmoid}} = Sigmoid[T=DT_FLOAT, _has_manual_control_dependencies=true](dense_1/BiasAdd)}}' was changed by setting attribute after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-03-27 09:59:30.614731: W tensorflow/c/c_api.cc:291] Operation '{name:'count/Assign' id:359 op device:{requested: '', assigned: ''} def:{{{node count/Assign}} = AssignVariableOp[_has_manual_control_dependencies=true, dtype=DT_FLOAT, validate_shape=false](count, count/Initializer/zeros)}}' was changed by setting attribute after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
1%, time left: unknown
8%, time left: 0 sec
14%, time left: 0 sec
20%, time left: 0 sec
26%, time left: 0 sec
33%, time left: 0 sec
39%, time left: 0 sec
45%, time left: 0 sec
51%, time left: 0 sec
58%, time left: 0 sec
64%, time left: 0 sec
70%, time left: 0 sec
76%, time left: 0 sec
83%, time left: 0 sec
89%, time left: 0 sec
95%, time left: 0 sec
0%, time left: unknown
7%, time left: 0 sec
13%, time left: 0 sec
19%, time left: 0 sec
25%, time left: 0 sec
32%, time left: 0 sec
38%, time left: 0 sec
44%, time left: 0 sec
50%, time left: 0 sec
57%, time left: 0 sec
63%, time left: 0 sec
69%, time left: 0 sec
75%, time left: 0 sec
82%, time left: 0 sec
88%, time left: 0 sec
94%, time left: 0 sec
---- Evaluate all MVAs using the set of test events
In [18]:
factory.TestAllMethods()
Factory : Test all methods
Factory : Test method: TMVA_LSTM for Classification performance
:
: Evaluate deep neural network on CPU using batches with size = 800
:
TMVA_LSTM : [dataset] : Evaluation of TMVA_LSTM on testing sample (800 events)
: Elapsed time for evaluation of 800 events: 0.0973 sec
Factory : Test method: TMVA_DNN for Classification performance
:
: Evaluate deep neural network on CPU using batches with size = 800
:
TMVA_DNN : [dataset] : Evaluation of TMVA_DNN on testing sample (800 events)
: Elapsed time for evaluation of 800 events: 0.022 sec
Factory : Test method: PyKeras_LSTM for Classification performance
:
: Setting up tf.keras
: Using TensorFlow version 2
: Use Keras version from TensorFlow : tf.keras
: Applying GPU option: gpu_options.allow_growth=True
: Disabled TF eager execution when evaluating model
: Loading Keras Model
: Loaded model from file: trained_model_LSTM.h5
PyKeras_LSTM : [dataset] : Evaluation of PyKeras_LSTM on testing sample (800 events)
: Elapsed time for evaluation of 800 events: 0.298 sec
Factory : Test method: BDTG for Classification performance
:
BDTG : [dataset] : Evaluation of BDTG on testing sample (800 events)
: Elapsed time for evaluation of 800 events: 0.0043 sec
2023-03-27 09:59:33.132945: W tensorflow/c/c_api.cc:291] Operation '{name:'dense_1_1/kernel/Assign' id:833 op device:{requested: '', assigned: ''} def:{{{node dense_1_1/kernel/Assign}} = AssignVariableOp[_has_manual_control_dependencies=true, dtype=DT_FLOAT, validate_shape=false](dense_1_1/kernel, dense_1_1/kernel/Initializer/stateless_random_uniform)}}' was changed by setting attribute after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-03-27 09:59:33.431927: W tensorflow/c/c_api.cc:291] Operation '{name:'dense_1_1/Sigmoid' id:844 op device:{requested: '', assigned: ''} def:{{{node dense_1_1/Sigmoid}} = Sigmoid[T=DT_FLOAT, _has_manual_control_dependencies=true](dense_1_1/BiasAdd)}}' was changed by setting attribute after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-03-27 09:59:33.529953: W tensorflow/c/c_api.cc:291] Operation '{name:'count_1/Assign' id:892 op device:{requested: '', assigned: ''} def:{{{node count_1/Assign}} = AssignVariableOp[_has_manual_control_dependencies=true, dtype=DT_FLOAT, validate_shape=false](count_1, count_1/Initializer/zeros)}}' was changed by setting attribute after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
0%, time left: unknown
7%, time left: 0 sec
13%, time left: 0 sec
19%, time left: 0 sec
25%, time left: 0 sec
32%, time left: 0 sec
38%, time left: 0 sec
44%, time left: 0 sec
50%, time left: 0 sec
57%, time left: 0 sec
63%, time left: 0 sec
69%, time left: 0 sec
75%, time left: 0 sec
82%, time left: 0 sec
88%, time left: 0 sec
94%, time left: 0 sec
----- Evaluate and compare performance of all configured MVAs
In [19]:
factory.EvaluateAllMethods()
Factory : Evaluate all methods Factory : Evaluate classifier: TMVA_LSTM : TMVA_LSTM : [dataset] : Loop over test events and fill histograms with classifier response... : : Evaluate deep neural network on CPU using batches with size = 1000 : : Dataset[dataset] : variable plots are not produces ! The number of variables is 300 , it is larger than 200 Factory : Evaluate classifier: TMVA_DNN : TMVA_DNN : [dataset] : Loop over test events and fill histograms with classifier response... : : Evaluate deep neural network on CPU using batches with size = 1000 : : Dataset[dataset] : variable plots are not produces ! The number of variables is 300 , it is larger than 200 Factory : Evaluate classifier: PyKeras_LSTM : PyKeras_LSTM : [dataset] : Loop over test events and fill histograms with classifier response... : : Dataset[dataset] : variable plots are not produces ! The number of variables is 300 , it is larger than 200 Factory : Evaluate classifier: BDTG : BDTG : [dataset] : Loop over test events and fill histograms with classifier response... : : Dataset[dataset] : variable plots are not produces ! The number of variables is 300 , it is larger than 200 : : Evaluation results ranked by best signal efficiency and purity (area) : ------------------------------------------------------------------------------------------------------------------- : DataSet MVA : Name: Method: ROC-integ : dataset BDTG : 0.654 : dataset PyKeras_LSTM : 0.564 : dataset TMVA_DNN : 0.545 : dataset TMVA_LSTM : 0.511 : ------------------------------------------------------------------------------------------------------------------- : : Testing efficiency compared to training efficiency (overtraining check) : ------------------------------------------------------------------------------------------------------------------- : DataSet MVA Signal efficiency: from test sample (from training sample) : Name: Method: @B=0.01 @B=0.10 @B=0.30 : ------------------------------------------------------------------------------------------------------------------- : dataset BDTG : 0.025 (0.055) 0.202 (0.353) 0.523 (0.670) : dataset PyKeras_LSTM : 0.028 (0.025) 0.160 (0.201) 0.378 (0.477) : dataset TMVA_DNN : 0.011 (0.016) 0.123 (0.166) 0.355 (0.394) : dataset TMVA_LSTM : 0.010 (0.013) 0.105 (0.104) 0.305 (0.311) : ------------------------------------------------------------------------------------------------------------------- : Dataset:dataset : Created tree 'TestTree' with 800 events : Dataset:dataset : Created tree 'TrainTree' with 3200 events : Factory : Thank you for using TMVA! : For citation information, please visit: http://tmva.sf.net/citeTMVA.html
check method
plot ROC curve
In [20]:
c1 = factory.GetROCCurve(dataloader)
c1.Draw()
if outputFile:
outputFile.Close()
Draw all canvases
In [21]:
from ROOT import gROOT
gROOT.GetListOfCanvases().Draw()