This notebook shows how to use XGBoost's Evaluation Metric (eval_metric
) with H2O.
Original (non-H2O) XGBoost library allows users to define one or more evaluation metrics that will be calculated on both training and validations datasets after each iteration. If no evaluation metric is defined by the user XGBoost will assign it based on the choice of the objective function. For example in the case of binary classification, XGBoost will use logloss to report performance on training and validation data. Logloss measures the extent to which predicted probabilities diverge from the actual class labels. You might want to consider evaluating the performance using a different metric, e.g. for imbalanced problems, where it is more important to correctly predict the positive minority class, you might want to use the Area under the PR curve and specify eval_metric="aucpr"
. Evaluation metric can easily be used for early stopping.
H2O's approach to calculating metrics is different. By default, H2O will calculate all appropritate metrics for the given problem. For example for binary classification, H2O will report logloss, AUC, AUCPR and also additional metrics. Users thus typically don't need to worry about selecting the appropriate metric before model training. When early stopping is used, users will need to chose from one built-in early stopping metrics. For consistency between different model types and/or algorithm implementations, these are always calculated by H2O itself and, in XGBoost's specific case independect of XGBoost's eval_metric implementation.
While you typically don't need to specify your custom eval_metric
, there are cases doing so would be beneficial.
Case 1: H2O doesn't provide a suitable built-in metric. Example: If you want to calculate classification error for a different threshold than the one automatically determined by H2O, you can do so only by specifying eval_metric="error@<your threshold>"
because H2O currently doesn't have this capability.
Case 2: Frequent scoring. By default H2O uses timing-triggered scoring, it is trying to make sure that majority of the time is used on model training as opposed to just model scoring. You can override this behavior and specify manually at what iterations you want model to be scored (eg. each iteration, every 5th iteration, ...). Because H2O calculates all possible metrics as opposed just few in native XGBoost, and needs to extract XGBoost model from the native memory, it can have a significant overhead when user desires to score very frequently. This can slow down the model building.
We will create a synthetic classification dataset, show XGBoost model training with default parameters and compare it to the output of model training with eval_metric used for early stopping.
import h2o
h2o.init(strict_version_check=False)
versionFromGradle='3.39.0',projectVersion='3.39.0.99999',branch='michalk_eval-metric-ntb',lastCommitHash='256ed83c89220493cca0574ebd517eb09e3611fd',gitDescribe='jenkins-master-5998-dirty',compiledOn='2022-10-28 14:15:26',compiledBy='kurkami' Checking whether there is an H2O instance running at http://localhost:54321 . connected.
H2O_cluster_uptime: | 5 hours 14 mins |
H2O_cluster_timezone: | America/New_York |
H2O_data_parsing_timezone: | UTC |
H2O_cluster_version: | 3.39.0.99999 |
H2O_cluster_version_age: | 1 day |
H2O_cluster_name: | kurkami |
H2O_cluster_total_nodes: | 1 |
H2O_cluster_free_memory: | 3.051 Gb |
H2O_cluster_total_cores: | 8 |
H2O_cluster_allowed_cores: | 8 |
H2O_cluster_status: | locked, healthy |
H2O_connection_url: | http://localhost:54321 |
H2O_connection_proxy: | {"http": null, "https": null} |
H2O_internal_security: | False |
Python_version: | 3.10.6 final |
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
X, y = make_classification(n_samples=100000,
n_informative=5,
n_classes=2,
random_state=42)
X_train, X_validation, y_train, y_validation = train_test_split(X, y, test_size=0.25)
from h2o import H2OFrame
train = H2OFrame(y_train, column_names=["y"]).cbind(H2OFrame(X_train))
valid = H2OFrame(y_validation, column_names=["y"]).cbind(H2OFrame(X_validation))
train["y"] = train["y"].asfactor()
valid["y"] = valid["y"].asfactor()
Parse progress: |████████████████████████████████████████████████████████████████| (done) 100% Parse progress: |████████████████████████████████████████████████████████████████| (done) 100% Parse progress: |████████████████████████████████████████████████████████████████| (done) 100% Parse progress: |████████████████████████████████████████████████████████████████| (done) 100%
train
y | C1 | C2 | C3 | C4 | C5 | C6 | C7 | C8 | C9 | C10 | C11 | C12 | C13 | C14 | C15 | C16 | C17 | C18 | C19 | C20 |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
1 | -0.460911 | 0.587072 | 1.27653 | 1.65893 | 0.82732 | 0.900837 | 1.27825 | -1.15342 | -1.10419 | 0.231584 | 1.42246 | 0.632382 | 1.6532 | 0.0807537 | 0.63636 | -0.0407703 | -0.712485 | -0.987504 | 0.529023 | 0.813985 |
0 | -0.122517 | 0.926967 | -3.27666 | 1.65648 | 1.26381 | 0.866601 | -0.112373 | 4.88467 | -2.56306 | 1.37596 | 1.66414 | 0.750439 | -0.712589 | 0.011264 | 0.338901 | -0.91639 | -1.37998 | -1.37817 | -0.702391 | -0.222837 |
1 | 1.07287 | 1.35581 | 0.504084 | -1.9799 | 0.860703 | -0.753541 | 1.17781 | 0.364174 | 2.49498 | -0.309902 | 0.52471 | -0.128492 | 0.133623 | 1.77568 | -1.70823 | 0.263603 | -1.69966 | 0.77875 | -0.578858 | 1.53087 |
1 | -0.497834 | 0.427476 | -1.67148 | -1.71013 | 1.11473 | 0.0850045 | -2.81089 | 3.2892 | 0.676264 | 1.04835 | 1.46596 | 0.818714 | -1.03108 | -1.65528 | 0.301107 | 1.19965 | 0.120588 | 0.820176 | -0.930966 | -0.82086 |
1 | 0.923038 | 1.51161 | 1.66059 | 1.45418 | -2.3419 | -0.757573 | 0.308842 | -3.57087 | -0.939606 | 0.369506 | 0.990806 | 0.67289 | 1.53918 | 1.79014 | 0.726961 | 0.0339329 | -0.542889 | -1.01856 | 1.97659 | 1.50693 |
0 | 0.575923 | 2.20724 | 0.595904 | -0.416675 | 0.21922 | 0.787082 | -0.0753548 | -0.413841 | -0.280823 | 0.756811 | -0.115625 | -1.03631 | 0.617746 | 1.15633 | -1.3063 | 1.83909 | 0.610228 | 0.62274 | -0.157638 | -1.19873 |
0 | -0.931317 | 3.38117 | -1.25809 | 0.323422 | 4.46615 | 0.366347 | -0.178002 | 4.59046 | -0.995683 | 0.248196 | -3.7437 | 1.22383 | -1.00021 | -0.754449 | 0.573945 | 1.94618 | -2.16485 | -0.285859 | -3.99979 | -1.39355 |
0 | 0.956366 | 1.02616 | -0.55144 | -0.66649 | 2.47247 | 1.14188 | 1.10357 | 2.61941 | 1.2818 | 0.528047 | -1.10656 | -0.664716 | -0.849829 | 1.49331 | -0.183118 | 0.618475 | -0.413351 | 0.410573 | -2.14432 | 2.35245 |
0 | -0.0523249 | 1.73885 | -1.47901 | 0.254941 | -1.37106 | -0.0549625 | -1.19023 | 0.848204 | -0.424188 | -0.0600663 | -0.781962 | 1.1958 | -0.456581 | -0.740832 | 2.26734 | 2.01902 | 1.89101 | 0.348762 | 0.260157 | 0.133197 |
0 | -0.818169 | -0.746954 | -1.28428 | -0.0786964 | 0.708029 | -0.252285 | -0.19201 | 1.33074 | 0.229397 | -0.599344 | -0.141543 | -0.721745 | 0.442618 | 0.953017 | 1.00111 | -0.862009 | 0.790705 | 1.56841 | 0.101824 | 1.48008 |
[75000 rows x 21 columns]
We are specifying 1000 trees to be built and providing stopping criteria to make the model stop reasonably early for the purpose of this example.
from h2o.estimators.xgboost import H2OXGBoostEstimator
model_def = H2OXGBoostEstimator(ntrees=1000, max_depth=6, score_each_iteration=True,
stopping_rounds=3, stopping_tolerance=0.05, stopping_metric="logloss")
model_def.train(y="y", training_frame=train, validation_frame=valid)
xgboost Model Build progress: |██████████████████████████████████████████████████| (done) 100%
Model Details ============= H2OXGBoostEstimator : XGBoost Model Key: XGBoost_model_python_1666962310297_518
number_of_trees | |
---|---|
22.0 |
ModelMetricsBinomial: xgboost ** Reported on train data. ** MSE: 0.03475774769265435 RMSE: 0.18643429859511995 LogLoss: 0.12901406890397824 Mean Per-Class Error: 0.04387881533186782 AUC: 0.9893340459497513 AUCPR: 0.988704085388505 Gini: 0.9786680918995025
0 | 1 | Error | Rate | |
---|---|---|---|---|
0 | 35155.0 | 2174.0 | 0.0582 | (2174.0/37329.0) |
1 | 1112.0 | 36559.0 | 0.0295 | (1112.0/37671.0) |
Total | 36267.0 | 38733.0 | 0.0438 | (3286.0/75000.0) |
metric | threshold | value | idx |
---|---|---|---|
max f1 | 0.5101676 | 0.9569918 | 214.0 |
max f2 | 0.3082806 | 0.9715081 | 263.0 |
max f0point5 | 0.7532607 | 0.9592084 | 147.0 |
max accuracy | 0.5561712 | 0.9564 | 201.0 |
max precision | 0.9974565 | 1.0 | 0.0 |
max recall | 0.0016028 | 1.0 | 399.0 |
max specificity | 0.9974565 | 1.0 | 0.0 |
max absolute_mcc | 0.5561712 | 0.9129277 | 201.0 |
max min_per_class_accuracy | 0.6239853 | 0.9548073 | 184.0 |
max mean_per_class_accuracy | 0.5561712 | 0.9563597 | 201.0 |
max tns | 0.9974565 | 37329.0 | 0.0 |
max fns | 0.9974565 | 37460.0 | 0.0 |
max fps | 0.0016028 | 37329.0 | 399.0 |
max tps | 0.0016028 | 37671.0 | 399.0 |
max tnr | 0.9974565 | 1.0 | 0.0 |
max fnr | 0.9974565 | 0.9943989 | 0.0 |
max fpr | 0.0016028 | 1.0 | 399.0 |
max tpr | 0.0016028 | 1.0 | 399.0 |
group | cumulative_data_fraction | lower_threshold | lift | cumulative_lift | response_rate | score | cumulative_response_rate | cumulative_score | capture_rate | cumulative_capture_rate | gain | cumulative_gain | kolmogorov_smirnov |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|
1 | 0.0101733 | 0.9961097 | 1.9909214 | 1.9909214 | 1.0 | 0.9967516 | 1.0 | 0.9967516 | 0.0202543 | 0.0202543 | 99.0921398 | 99.0921398 | 0.0202543 |
2 | 0.0200267 | 0.9957153 | 1.9909214 | 1.9909214 | 1.0 | 0.9958894 | 1.0 | 0.9963274 | 0.0196172 | 0.0398715 | 99.0921398 | 99.0921398 | 0.0398715 |
3 | 0.03068 | 0.9953964 | 1.9909214 | 1.9909214 | 1.0 | 0.9955495 | 1.0 | 0.9960573 | 0.0212099 | 0.0610815 | 99.0921398 | 99.0921398 | 0.0610815 |
4 | 0.0401067 | 0.9951944 | 1.9881054 | 1.9902595 | 0.9985856 | 0.9952708 | 0.9996676 | 0.9958724 | 0.0187412 | 0.0798227 | 98.8105385 | 99.0259523 | 0.0797959 |
5 | 0.0517733 | 0.9949297 | 1.9863707 | 1.9893832 | 0.9977143 | 0.9950084 | 0.9992274 | 0.9956777 | 0.0231743 | 0.1029970 | 98.6370721 | 98.9383216 | 0.1029166 |
6 | 0.1000133 | 0.9933097 | 1.9881700 | 1.9887980 | 0.9986180 | 0.9941657 | 0.9989335 | 0.9949484 | 0.0959093 | 0.1989063 | 98.8169987 | 98.8798032 | 0.1986920 |
7 | 0.15008 | 0.9904984 | 1.9840287 | 1.9872070 | 0.9965379 | 0.9920577 | 0.9981343 | 0.9939841 | 0.0993337 | 0.2982400 | 98.4028728 | 98.7206993 | 0.2976775 |
8 | 0.2000133 | 0.9849605 | 1.9824155 | 1.9860108 | 0.9957276 | 0.9879125 | 0.9975335 | 0.9924683 | 0.0989886 | 0.3972286 | 98.2415459 | 98.6010786 | 0.3962375 |
9 | 0.30004 | 0.9561046 | 1.9673021 | 1.9797737 | 0.9881365 | 0.9740387 | 0.9944007 | 0.9863243 | 0.1967827 | 0.5940113 | 96.7302096 | 97.9773725 | 0.5906359 |
10 | 0.4000133 | 0.8770613 | 1.9123254 | 1.9629167 | 0.9605228 | 0.9210076 | 0.9859338 | 0.9700000 | 0.1911815 | 0.7851929 | 91.2325408 | 96.2916704 | 0.7738880 |
11 | 0.5000133 | 0.6383476 | 1.6763558 | 1.9056061 | 0.842 | 0.8051186 | 0.9571478 | 0.9370246 | 0.1676356 | 0.9528284 | 67.6355817 | 90.5606055 | 0.9097788 |
12 | 0.6 | 0.0781245 | 0.4261140 | 1.6590569 | 0.2140285 | 0.2876185 | 0.8333111 | 0.8288047 | 0.0426057 | 0.9954342 | -57.3886006 | 65.9056923 | 0.7944912 |
13 | 0.7000133 | 0.0128222 | 0.0238879 | 1.4254346 | 0.0119984 | 0.0339001 | 0.7159673 | 0.7152339 | 0.0023891 | 0.9978233 | -97.6112128 | 42.5434648 | 0.5983483 |
14 | 0.80008 | 0.0057930 | 0.0122028 | 1.2486806 | 0.0061292 | 0.0084165 | 0.6271873 | 0.6268317 | 0.0012211 | 0.9990444 | -98.7797151 | 24.8680579 | 0.3997516 |
15 | 0.9 | 0.0034428 | 0.0045164 | 1.1105507 | 0.0022685 | 0.0045299 | 0.5578074 | 0.5577424 | 0.0004513 | 0.9994956 | -99.5483632 | 11.0550704 | 0.1999028 |
16 | 1.0 | 0.0001672 | 0.0050437 | 1.0 | 0.0025333 | 0.0026268 | 0.50228 | 0.5022308 | 0.0005044 | 1.0 | -99.4956332 | 0.0 | 0.0 |
ModelMetricsBinomial: xgboost ** Reported on validation data. ** MSE: 0.038600168581279404 RMSE: 0.1964692560714765 LogLoss: 0.14264211397729937 Mean Per-Class Error: 0.04874265748103474 AUC: 0.9860367984758349 AUCPR: 0.9841777019906762 Gini: 0.9720735969516698
0 | 1 | Error | Rate | |
---|---|---|---|---|
0 | 11844.0 | 840.0 | 0.0662 | (840.0/12684.0) |
1 | 385.0 | 11931.0 | 0.0313 | (385.0/12316.0) |
Total | 12229.0 | 12771.0 | 0.049 | (1225.0/25000.0) |
metric | threshold | value | idx |
---|---|---|---|
max f1 | 0.4899956 | 0.9511699 | 217.0 |
max f2 | 0.2381639 | 0.9684011 | 281.0 |
max f0point5 | 0.7637601 | 0.9521032 | 144.0 |
max accuracy | 0.4899956 | 0.951 | 217.0 |
max precision | 0.9974683 | 1.0 | 0.0 |
max recall | 0.0018354 | 1.0 | 398.0 |
max specificity | 0.9974683 | 1.0 | 0.0 |
max absolute_mcc | 0.4899956 | 0.9026291 | 217.0 |
max min_per_class_accuracy | 0.6303638 | 0.9494966 | 182.0 |
max mean_per_class_accuracy | 0.4899956 | 0.9512573 | 217.0 |
max tns | 0.9974683 | 12684.0 | 0.0 |
max fns | 0.9974683 | 12244.0 | 0.0 |
max fps | 0.0013435 | 12684.0 | 399.0 |
max tps | 0.0018354 | 12316.0 | 398.0 |
max tnr | 0.9974683 | 1.0 | 0.0 |
max fnr | 0.9974683 | 0.9941539 | 0.0 |
max fpr | 0.0013435 | 1.0 | 399.0 |
max tpr | 0.0018354 | 1.0 | 398.0 |
group | cumulative_data_fraction | lower_threshold | lift | cumulative_lift | response_rate | score | cumulative_response_rate | cumulative_score | capture_rate | cumulative_capture_rate | gain | cumulative_gain | kolmogorov_smirnov |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|
1 | 0.01028 | 0.9961244 | 2.0298798 | 2.0298798 | 1.0 | 0.9967313 | 1.0 | 0.9967313 | 0.0208672 | 0.0208672 | 102.9879831 | 102.9879831 | 0.0208672 |
2 | 0.02032 | 0.9958012 | 2.0298798 | 2.0298798 | 1.0 | 0.9959175 | 1.0 | 0.9963292 | 0.0203800 | 0.0412472 | 102.9879831 | 102.9879831 | 0.0412472 |
3 | 0.03016 | 0.9954615 | 2.0216283 | 2.0271877 | 0.9959350 | 0.9956102 | 0.9986737 | 0.9960946 | 0.0198928 | 0.0611400 | 102.1628287 | 102.7187683 | 0.0610611 |
4 | 0.04108 | 0.9951944 | 2.0224444 | 2.0259268 | 0.9963370 | 0.9952887 | 0.9980526 | 0.9958804 | 0.0220851 | 0.0832251 | 102.2444374 | 102.5926803 | 0.0830674 |
5 | 0.05256 | 0.9949297 | 2.0157343 | 2.0237006 | 0.9930314 | 0.9950085 | 0.9969559 | 0.9956900 | 0.0231406 | 0.1063657 | 101.5734327 | 102.3700593 | 0.1060503 |
6 | 0.1 | 0.9932859 | 2.0196106 | 2.0217603 | 0.9949410 | 0.9941552 | 0.996 | 0.9949619 | 0.0958103 | 0.2021760 | 101.9610625 | 102.1760312 | 0.2013876 |
7 | 0.15 | 0.9902694 | 2.0233842 | 2.0223016 | 0.9968 | 0.9919490 | 0.9962667 | 0.9939576 | 0.1011692 | 0.3033452 | 102.3384216 | 102.2301613 | 0.3022415 |
8 | 0.2 | 0.9840660 | 2.0136408 | 2.0201364 | 0.992 | 0.9873945 | 0.9952 | 0.9923168 | 0.1006820 | 0.4040273 | 101.3640792 | 102.0136408 | 0.4021351 |
9 | 0.3 | 0.9523897 | 1.9852225 | 2.0084984 | 0.978 | 0.9717051 | 0.9894667 | 0.9854462 | 0.1985222 | 0.6025495 | 98.5222475 | 100.8498430 | 0.5963212 |
10 | 0.4 | 0.8671057 | 1.9235141 | 1.9872524 | 0.9476 | 0.9142567 | 0.979 | 0.9676488 | 0.1923514 | 0.7949009 | 92.3514128 | 98.7252355 | 0.7783447 |
11 | 0.5 | 0.5822353 | 1.6174082 | 1.9132835 | 0.7968 | 0.7825059 | 0.94256 | 0.9306202 | 0.1617408 | 0.9566418 | 61.7408249 | 91.3283534 | 0.9000350 |
12 | 0.6 | 0.0667765 | 0.3775576 | 1.6573292 | 0.186 | 0.2480180 | 0.8164667 | 0.8168532 | 0.0377558 | 0.9943975 | -62.2442351 | 65.7329219 | 0.7773524 |
13 | 0.7 | 0.0125771 | 0.0186749 | 1.4232357 | 0.0092 | 0.0304482 | 0.7011429 | 0.7045096 | 0.0018675 | 0.9962650 | -98.1325106 | 42.3235744 | 0.5839345 |
14 | 0.80048 | 0.0057412 | 0.0185857 | 1.2469175 | 0.0091561 | 0.0081574 | 0.6142814 | 0.6171002 | 0.0018675 | 0.9981325 | -98.1414317 | 24.6917488 | 0.3895705 |
15 | 0.9014 | 0.0034592 | 0.0104592 | 1.1084846 | 0.0051526 | 0.0045211 | 0.5460839 | 0.5485164 | 0.0010555 | 0.9991880 | -98.9540849 | 10.8484633 | 0.1927390 |
16 | 1.0 | 0.0002609 | 0.0082348 | 1.0 | 0.0040568 | 0.0026446 | 0.49264 | 0.4946934 | 0.0008120 | 1.0 | -99.1765193 | 0.0 | 0.0 |
timestamp | duration | number_of_trees | training_rmse | training_logloss | training_auc | training_pr_auc | training_lift | training_classification_error | validation_rmse | validation_logloss | validation_auc | validation_pr_auc | validation_lift | validation_classification_error | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
2022-10-28 14:19:27 | 0.031 sec | 0.0 | 0.5 | 0.6931472 | 0.5 | 0.5022800 | 1.0 | 0.49772 | 0.5 | 0.6931472 | 0.5 | 0.4926400 | 1.0 | 0.50736 | |
2022-10-28 14:19:27 | 0.150 sec | 1.0 | 0.3996010 | 0.5084556 | 0.9599605 | 0.9528824 | 1.9699443 | 0.08508 | 0.4002939 | 0.5095766 | 0.9584754 | 0.9501212 | 2.0073256 | 0.08816 | |
2022-10-28 14:19:27 | 0.213 sec | 2.0 | 0.3370778 | 0.4032048 | 0.9693358 | 0.9638432 | 1.9749781 | 0.0786933 | 0.3378726 | 0.4043148 | 0.9689642 | 0.9627452 | 2.0180782 | 0.08072 | |
2022-10-28 14:19:27 | 0.272 sec | 3.0 | 0.2957135 | 0.3335651 | 0.9735764 | 0.9692971 | 1.9823583 | 0.07124 | 0.2972804 | 0.3355690 | 0.9727585 | 0.9672690 | 2.0164813 | 0.07328 | |
2022-10-28 14:19:27 | 0.337 sec | 4.0 | 0.2690565 | 0.2867050 | 0.9769386 | 0.9736661 | 1.9824404 | 0.0687733 | 0.2711930 | 0.2894303 | 0.9753680 | 0.9709177 | 2.0166126 | 0.07028 | |
2022-10-28 14:19:28 | 0.461 sec | 5.0 | 0.2529770 | 0.2548994 | 0.9782349 | 0.9753981 | 1.9830013 | 0.06936 | 0.2550437 | 0.2575265 | 0.9769549 | 0.9724197 | 2.0204531 | 0.06956 | |
2022-10-28 14:19:28 | 0.553 sec | 6.0 | 0.2430550 | 0.2328859 | 0.9792536 | 0.9767333 | 1.9826945 | 0.06864 | 0.2452571 | 0.2357295 | 0.9778085 | 0.9737537 | 2.0198804 | 0.06872 | |
2022-10-28 14:19:28 | 0.646 sec | 7.0 | 0.2366703 | 0.2173043 | 0.9801005 | 0.9779518 | 1.9840125 | 0.0655333 | 0.2388924 | 0.2202064 | 0.9787945 | 0.9751087 | 2.0189075 | 0.06696 | |
2022-10-28 14:19:28 | 0.742 sec | 8.0 | 0.2274282 | 0.2004642 | 0.9816894 | 0.9795849 | 1.9883425 | 0.06172 | 0.2302864 | 0.2040544 | 0.9803330 | 0.9772714 | 2.0219506 | 0.06516 | |
2022-10-28 14:19:28 | 0.831 sec | 9.0 | 0.2235048 | 0.1908878 | 0.9823444 | 0.9805219 | 1.9883948 | 0.0608 | 0.2267565 | 0.1951048 | 0.9808128 | 0.9779093 | 2.0222773 | 0.06432 | |
--- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
2022-10-28 14:19:28 | 1.227 sec | 13.0 | 0.2093299 | 0.1618817 | 0.9849208 | 0.9835753 | 1.9885342 | 0.0549333 | 0.2140217 | 0.1682975 | 0.9829089 | 0.9803703 | 2.0227574 | 0.0584 | |
2022-10-28 14:19:28 | 1.321 sec | 14.0 | 0.2080116 | 0.1587226 | 0.9852452 | 0.9839538 | 1.9883086 | 0.0544 | 0.2129618 | 0.1656925 | 0.9831326 | 0.9806934 | 2.0222773 | 0.05852 | |
2022-10-28 14:19:29 | 1.417 sec | 15.0 | 0.2065626 | 0.1556114 | 0.9856595 | 0.9846933 | 1.9909214 | 0.0538533 | 0.2118815 | 0.1633915 | 0.9833377 | 0.9810162 | 2.0227823 | 0.05792 | |
2022-10-28 14:19:29 | 1.542 sec | 16.0 | 0.2036096 | 0.1517939 | 0.9863474 | 0.9853808 | 1.9909214 | 0.0520267 | 0.2099121 | 0.1607776 | 0.9837030 | 0.9814828 | 2.0217603 | 0.0566 | |
2022-10-28 14:19:29 | 1.676 sec | 17.0 | 0.2001280 | 0.1470822 | 0.9868868 | 0.9859104 | 1.9909214 | 0.0503733 | 0.2068741 | 0.1565508 | 0.9842040 | 0.9821555 | 2.0298798 | 0.05524 | |
2022-10-28 14:19:29 | 1.797 sec | 18.0 | 0.1952364 | 0.1409688 | 0.9877667 | 0.9868323 | 1.9909214 | 0.0479067 | 0.2029770 | 0.1516665 | 0.9848696 | 0.9829746 | 2.0298798 | 0.05272 | |
2022-10-28 14:19:29 | 1.925 sec | 19.0 | 0.1943690 | 0.1391751 | 0.9880068 | 0.9871283 | 1.9909214 | 0.0476667 | 0.2025051 | 0.1505835 | 0.9849703 | 0.9829211 | 2.0298798 | 0.05232 | |
2022-10-28 14:19:29 | 2.069 sec | 20.0 | 0.1919489 | 0.1361213 | 0.9884007 | 0.9875789 | 1.9909214 | 0.0466933 | 0.2009170 | 0.1484152 | 0.9852986 | 0.9834889 | 2.0298798 | 0.05108 | |
2022-10-28 14:19:29 | 2.195 sec | 21.0 | 0.1886257 | 0.1319356 | 0.9889611 | 0.9882193 | 1.9909214 | 0.0444533 | 0.1983105 | 0.1448947 | 0.9857546 | 0.9838308 | 2.0217603 | 0.0496 | |
2022-10-28 14:19:29 | 2.322 sec | 22.0 | 0.1864343 | 0.1290141 | 0.9893340 | 0.9887041 | 1.9909214 | 0.0438133 | 0.1964693 | 0.1426421 | 0.9860368 | 0.9841777 | 2.0298798 | 0.049 |
[23 rows x 16 columns]
variable | relative_importance | scaled_importance | percentage |
---|---|---|---|
C11 | 70178.75 | 1.0 | 0.4261358 |
C2 | 35732.6640625 | 0.5091664 | 0.2169741 |
C13 | 21608.4863281 | 0.3079064 | 0.1312100 |
C8 | 15654.5498047 | 0.2230668 | 0.0950568 |
C3 | 12402.2607422 | 0.1767239 | 0.0753084 |
C19 | 5739.7641602 | 0.0817878 | 0.0348527 |
C5 | 2835.9191895 | 0.0404099 | 0.0172201 |
C6 | 88.9352493 | 0.0012673 | 0.0005400 |
C12 | 51.9876480 | 0.0007408 | 0.0003157 |
C17 | 49.1711044 | 0.0007007 | 0.0002986 |
C7 | 48.0521317 | 0.0006847 | 0.0002918 |
C14 | 46.5472183 | 0.0006633 | 0.0002826 |
C10 | 43.8788605 | 0.0006252 | 0.0002664 |
C4 | 42.4346542 | 0.0006047 | 0.0002577 |
C18 | 39.3952980 | 0.0005614 | 0.0002392 |
C1 | 32.5539932 | 0.0004639 | 0.0001977 |
C15 | 25.9528160 | 0.0003698 | 0.0001576 |
C20 | 23.1192322 | 0.0003294 | 0.0001404 |
C16 | 22.9983807 | 0.0003277 | 0.0001396 |
C9 | 18.9113159 | 0.0002695 | 0.0001148 |
[tips] Use `model.explain()` to inspect the model. -- Use `h2o.display.toggle_user_tips()` to switch on/off this section.
Scoring history will show us some of the metrics that were calculated on training and validation datasets. In our case, validation_logloss
was used as the metric for early stopping.
model_def.scoring_history()
timestamp | duration | number_of_trees | training_rmse | training_logloss | training_auc | training_pr_auc | training_lift | training_classification_error | validation_rmse | validation_logloss | validation_auc | validation_pr_auc | validation_lift | validation_classification_error | ||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
0 | 2022-10-28 14:19:27 | 0.031 sec | 0.0 | 0.500000 | 0.693147 | 0.500000 | 0.502280 | 1.000000 | 0.497720 | 0.500000 | 0.693147 | 0.500000 | 0.492640 | 1.000000 | 0.50736 | |
1 | 2022-10-28 14:19:27 | 0.150 sec | 1.0 | 0.399601 | 0.508456 | 0.959960 | 0.952882 | 1.969944 | 0.085080 | 0.400294 | 0.509577 | 0.958475 | 0.950121 | 2.007326 | 0.08816 | |
2 | 2022-10-28 14:19:27 | 0.213 sec | 2.0 | 0.337078 | 0.403205 | 0.969336 | 0.963843 | 1.974978 | 0.078693 | 0.337873 | 0.404315 | 0.968964 | 0.962745 | 2.018078 | 0.08072 | |
3 | 2022-10-28 14:19:27 | 0.272 sec | 3.0 | 0.295714 | 0.333565 | 0.973576 | 0.969297 | 1.982358 | 0.071240 | 0.297280 | 0.335569 | 0.972759 | 0.967269 | 2.016481 | 0.07328 | |
4 | 2022-10-28 14:19:27 | 0.337 sec | 4.0 | 0.269057 | 0.286705 | 0.976939 | 0.973666 | 1.982440 | 0.068773 | 0.271193 | 0.289430 | 0.975368 | 0.970918 | 2.016613 | 0.07028 | |
5 | 2022-10-28 14:19:28 | 0.461 sec | 5.0 | 0.252977 | 0.254899 | 0.978235 | 0.975398 | 1.983001 | 0.069360 | 0.255044 | 0.257526 | 0.976955 | 0.972420 | 2.020453 | 0.06956 | |
6 | 2022-10-28 14:19:28 | 0.553 sec | 6.0 | 0.243055 | 0.232886 | 0.979254 | 0.976733 | 1.982694 | 0.068640 | 0.245257 | 0.235729 | 0.977808 | 0.973754 | 2.019880 | 0.06872 | |
7 | 2022-10-28 14:19:28 | 0.646 sec | 7.0 | 0.236670 | 0.217304 | 0.980100 | 0.977952 | 1.984012 | 0.065533 | 0.238892 | 0.220206 | 0.978794 | 0.975109 | 2.018908 | 0.06696 | |
8 | 2022-10-28 14:19:28 | 0.742 sec | 8.0 | 0.227428 | 0.200464 | 0.981689 | 0.979585 | 1.988342 | 0.061720 | 0.230286 | 0.204054 | 0.980333 | 0.977271 | 2.021951 | 0.06516 | |
9 | 2022-10-28 14:19:28 | 0.831 sec | 9.0 | 0.223505 | 0.190888 | 0.982344 | 0.980522 | 1.988395 | 0.060800 | 0.226757 | 0.195105 | 0.980813 | 0.977909 | 2.022277 | 0.06432 | |
10 | 2022-10-28 14:19:28 | 0.940 sec | 10.0 | 0.220040 | 0.182967 | 0.983000 | 0.981210 | 1.988298 | 0.059227 | 0.223656 | 0.187705 | 0.981356 | 0.978627 | 2.021888 | 0.06240 | |
11 | 2022-10-28 14:19:28 | 1.039 sec | 11.0 | 0.214926 | 0.173463 | 0.983878 | 0.982224 | 1.988819 | 0.057027 | 0.218899 | 0.178748 | 0.982148 | 0.979434 | 2.023615 | 0.06084 | |
12 | 2022-10-28 14:19:28 | 1.135 sec | 12.0 | 0.212074 | 0.167261 | 0.984348 | 0.982791 | 1.988617 | 0.055987 | 0.216237 | 0.172957 | 0.982491 | 0.979711 | 2.022975 | 0.05932 | |
13 | 2022-10-28 14:19:28 | 1.227 sec | 13.0 | 0.209330 | 0.161882 | 0.984921 | 0.983575 | 1.988534 | 0.054933 | 0.214022 | 0.168298 | 0.982909 | 0.980370 | 2.022757 | 0.05840 | |
14 | 2022-10-28 14:19:28 | 1.321 sec | 14.0 | 0.208012 | 0.158723 | 0.985245 | 0.983954 | 1.988309 | 0.054400 | 0.212962 | 0.165693 | 0.983133 | 0.980693 | 2.022277 | 0.05852 | |
15 | 2022-10-28 14:19:29 | 1.417 sec | 15.0 | 0.206563 | 0.155611 | 0.985659 | 0.984693 | 1.990921 | 0.053853 | 0.211881 | 0.163391 | 0.983338 | 0.981016 | 2.022782 | 0.05792 | |
16 | 2022-10-28 14:19:29 | 1.542 sec | 16.0 | 0.203610 | 0.151794 | 0.986347 | 0.985381 | 1.990921 | 0.052027 | 0.209912 | 0.160778 | 0.983703 | 0.981483 | 2.021760 | 0.05660 | |
17 | 2022-10-28 14:19:29 | 1.676 sec | 17.0 | 0.200128 | 0.147082 | 0.986887 | 0.985910 | 1.990921 | 0.050373 | 0.206874 | 0.156551 | 0.984204 | 0.982155 | 2.029880 | 0.05524 | |
18 | 2022-10-28 14:19:29 | 1.797 sec | 18.0 | 0.195236 | 0.140969 | 0.987767 | 0.986832 | 1.990921 | 0.047907 | 0.202977 | 0.151666 | 0.984870 | 0.982975 | 2.029880 | 0.05272 | |
19 | 2022-10-28 14:19:29 | 1.925 sec | 19.0 | 0.194369 | 0.139175 | 0.988007 | 0.987128 | 1.990921 | 0.047667 | 0.202505 | 0.150583 | 0.984970 | 0.982921 | 2.029880 | 0.05232 | |
20 | 2022-10-28 14:19:29 | 2.069 sec | 20.0 | 0.191949 | 0.136121 | 0.988401 | 0.987579 | 1.990921 | 0.046693 | 0.200917 | 0.148415 | 0.985299 | 0.983489 | 2.029880 | 0.05108 | |
21 | 2022-10-28 14:19:29 | 2.195 sec | 21.0 | 0.188626 | 0.131936 | 0.988961 | 0.988219 | 1.990921 | 0.044453 | 0.198310 | 0.144895 | 0.985755 | 0.983831 | 2.021760 | 0.04960 | |
22 | 2022-10-28 14:19:29 | 2.322 sec | 22.0 | 0.186434 | 0.129014 | 0.989334 | 0.988704 | 1.990921 | 0.043813 | 0.196469 | 0.142642 | 0.986037 | 0.984178 | 2.029880 | 0.04900 |
We will use the same parameters as in the first case and add eval_metric="logloss"
. To use actually use the value of the eval_metric
for early stopping, we also need to specify stopping_metric="custom"
.
model_eval = H2OXGBoostEstimator(ntrees=1000, max_depth=6, score_each_iteration=True,
eval_metric="logloss",
stopping_rounds=3, stopping_tolerance=0.05, stopping_metric="custom")
model_eval.train(y="y", training_frame=train, validation_frame=valid)
xgboost Model Build progress: |██████████████████████████████████████████████████| (done) 100%
Model Details ============= H2OXGBoostEstimator : XGBoost Model Key: XGBoost_model_python_1666962310297_567
number_of_trees | |
---|---|
22.0 |
ModelMetricsBinomial: xgboost ** Reported on train data. ** MSE: 0.03475774769265435 RMSE: 0.18643429859511995 LogLoss: 0.12901406890397824 Mean Per-Class Error: 0.04387881533186782 AUC: 0.9893340459497513 AUCPR: 0.988704085388505 Gini: 0.9786680918995025
0 | 1 | Error | Rate | |
---|---|---|---|---|
0 | 35155.0 | 2174.0 | 0.0582 | (2174.0/37329.0) |
1 | 1112.0 | 36559.0 | 0.0295 | (1112.0/37671.0) |
Total | 36267.0 | 38733.0 | 0.0438 | (3286.0/75000.0) |
metric | threshold | value | idx |
---|---|---|---|
max f1 | 0.5101676 | 0.9569918 | 214.0 |
max f2 | 0.3082806 | 0.9715081 | 263.0 |
max f0point5 | 0.7532607 | 0.9592084 | 147.0 |
max accuracy | 0.5561712 | 0.9564 | 201.0 |
max precision | 0.9974565 | 1.0 | 0.0 |
max recall | 0.0016028 | 1.0 | 399.0 |
max specificity | 0.9974565 | 1.0 | 0.0 |
max absolute_mcc | 0.5561712 | 0.9129277 | 201.0 |
max min_per_class_accuracy | 0.6239853 | 0.9548073 | 184.0 |
max mean_per_class_accuracy | 0.5561712 | 0.9563597 | 201.0 |
max tns | 0.9974565 | 37329.0 | 0.0 |
max fns | 0.9974565 | 37460.0 | 0.0 |
max fps | 0.0016028 | 37329.0 | 399.0 |
max tps | 0.0016028 | 37671.0 | 399.0 |
max tnr | 0.9974565 | 1.0 | 0.0 |
max fnr | 0.9974565 | 0.9943989 | 0.0 |
max fpr | 0.0016028 | 1.0 | 399.0 |
max tpr | 0.0016028 | 1.0 | 399.0 |
group | cumulative_data_fraction | lower_threshold | lift | cumulative_lift | response_rate | score | cumulative_response_rate | cumulative_score | capture_rate | cumulative_capture_rate | gain | cumulative_gain | kolmogorov_smirnov |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|
1 | 0.0101733 | 0.9961097 | 1.9909214 | 1.9909214 | 1.0 | 0.9967516 | 1.0 | 0.9967516 | 0.0202543 | 0.0202543 | 99.0921398 | 99.0921398 | 0.0202543 |
2 | 0.0200267 | 0.9957153 | 1.9909214 | 1.9909214 | 1.0 | 0.9958894 | 1.0 | 0.9963274 | 0.0196172 | 0.0398715 | 99.0921398 | 99.0921398 | 0.0398715 |
3 | 0.03068 | 0.9953964 | 1.9909214 | 1.9909214 | 1.0 | 0.9955495 | 1.0 | 0.9960573 | 0.0212099 | 0.0610815 | 99.0921398 | 99.0921398 | 0.0610815 |
4 | 0.0401067 | 0.9951944 | 1.9881054 | 1.9902595 | 0.9985856 | 0.9952708 | 0.9996676 | 0.9958724 | 0.0187412 | 0.0798227 | 98.8105385 | 99.0259523 | 0.0797959 |
5 | 0.0517733 | 0.9949297 | 1.9863707 | 1.9893832 | 0.9977143 | 0.9950084 | 0.9992274 | 0.9956777 | 0.0231743 | 0.1029970 | 98.6370721 | 98.9383216 | 0.1029166 |
6 | 0.1000133 | 0.9933097 | 1.9881700 | 1.9887980 | 0.9986180 | 0.9941657 | 0.9989335 | 0.9949484 | 0.0959093 | 0.1989063 | 98.8169987 | 98.8798032 | 0.1986920 |
7 | 0.15008 | 0.9904984 | 1.9840287 | 1.9872070 | 0.9965379 | 0.9920577 | 0.9981343 | 0.9939841 | 0.0993337 | 0.2982400 | 98.4028728 | 98.7206993 | 0.2976775 |
8 | 0.2000133 | 0.9849605 | 1.9824155 | 1.9860108 | 0.9957276 | 0.9879125 | 0.9975335 | 0.9924683 | 0.0989886 | 0.3972286 | 98.2415459 | 98.6010786 | 0.3962375 |
9 | 0.30004 | 0.9561046 | 1.9673021 | 1.9797737 | 0.9881365 | 0.9740387 | 0.9944007 | 0.9863243 | 0.1967827 | 0.5940113 | 96.7302096 | 97.9773725 | 0.5906359 |
10 | 0.4000133 | 0.8770613 | 1.9123254 | 1.9629167 | 0.9605228 | 0.9210076 | 0.9859338 | 0.9700000 | 0.1911815 | 0.7851929 | 91.2325408 | 96.2916704 | 0.7738880 |
11 | 0.5000133 | 0.6383476 | 1.6763558 | 1.9056061 | 0.842 | 0.8051186 | 0.9571478 | 0.9370246 | 0.1676356 | 0.9528284 | 67.6355817 | 90.5606055 | 0.9097788 |
12 | 0.6 | 0.0781245 | 0.4261140 | 1.6590569 | 0.2140285 | 0.2876185 | 0.8333111 | 0.8288047 | 0.0426057 | 0.9954342 | -57.3886006 | 65.9056923 | 0.7944912 |
13 | 0.7000133 | 0.0128222 | 0.0238879 | 1.4254346 | 0.0119984 | 0.0339001 | 0.7159673 | 0.7152339 | 0.0023891 | 0.9978233 | -97.6112128 | 42.5434648 | 0.5983483 |
14 | 0.80008 | 0.0057930 | 0.0122028 | 1.2486806 | 0.0061292 | 0.0084165 | 0.6271873 | 0.6268317 | 0.0012211 | 0.9990444 | -98.7797151 | 24.8680579 | 0.3997516 |
15 | 0.9 | 0.0034428 | 0.0045164 | 1.1105507 | 0.0022685 | 0.0045299 | 0.5578074 | 0.5577424 | 0.0004513 | 0.9994956 | -99.5483632 | 11.0550704 | 0.1999028 |
16 | 1.0 | 0.0001672 | 0.0050437 | 1.0 | 0.0025333 | 0.0026268 | 0.50228 | 0.5022308 | 0.0005044 | 1.0 | -99.4956332 | 0.0 | 0.0 |
ModelMetricsBinomial: xgboost ** Reported on validation data. ** MSE: 0.038600168581279404 RMSE: 0.1964692560714765 LogLoss: 0.14264211397729937 Mean Per-Class Error: 0.04874265748103474 AUC: 0.9860367984758349 AUCPR: 0.9841777019906762 Gini: 0.9720735969516698
0 | 1 | Error | Rate | |
---|---|---|---|---|
0 | 11844.0 | 840.0 | 0.0662 | (840.0/12684.0) |
1 | 385.0 | 11931.0 | 0.0313 | (385.0/12316.0) |
Total | 12229.0 | 12771.0 | 0.049 | (1225.0/25000.0) |
metric | threshold | value | idx |
---|---|---|---|
max f1 | 0.4899956 | 0.9511699 | 217.0 |
max f2 | 0.2381639 | 0.9684011 | 281.0 |
max f0point5 | 0.7637601 | 0.9521032 | 144.0 |
max accuracy | 0.4899956 | 0.951 | 217.0 |
max precision | 0.9974683 | 1.0 | 0.0 |
max recall | 0.0018354 | 1.0 | 398.0 |
max specificity | 0.9974683 | 1.0 | 0.0 |
max absolute_mcc | 0.4899956 | 0.9026291 | 217.0 |
max min_per_class_accuracy | 0.6303638 | 0.9494966 | 182.0 |
max mean_per_class_accuracy | 0.4899956 | 0.9512573 | 217.0 |
max tns | 0.9974683 | 12684.0 | 0.0 |
max fns | 0.9974683 | 12244.0 | 0.0 |
max fps | 0.0013435 | 12684.0 | 399.0 |
max tps | 0.0018354 | 12316.0 | 398.0 |
max tnr | 0.9974683 | 1.0 | 0.0 |
max fnr | 0.9974683 | 0.9941539 | 0.0 |
max fpr | 0.0013435 | 1.0 | 399.0 |
max tpr | 0.0018354 | 1.0 | 398.0 |
group | cumulative_data_fraction | lower_threshold | lift | cumulative_lift | response_rate | score | cumulative_response_rate | cumulative_score | capture_rate | cumulative_capture_rate | gain | cumulative_gain | kolmogorov_smirnov |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|
1 | 0.01028 | 0.9961244 | 2.0298798 | 2.0298798 | 1.0 | 0.9967313 | 1.0 | 0.9967313 | 0.0208672 | 0.0208672 | 102.9879831 | 102.9879831 | 0.0208672 |
2 | 0.02032 | 0.9958012 | 2.0298798 | 2.0298798 | 1.0 | 0.9959175 | 1.0 | 0.9963292 | 0.0203800 | 0.0412472 | 102.9879831 | 102.9879831 | 0.0412472 |
3 | 0.03016 | 0.9954615 | 2.0216283 | 2.0271877 | 0.9959350 | 0.9956102 | 0.9986737 | 0.9960946 | 0.0198928 | 0.0611400 | 102.1628287 | 102.7187683 | 0.0610611 |
4 | 0.04108 | 0.9951944 | 2.0224444 | 2.0259268 | 0.9963370 | 0.9952887 | 0.9980526 | 0.9958804 | 0.0220851 | 0.0832251 | 102.2444374 | 102.5926803 | 0.0830674 |
5 | 0.05256 | 0.9949297 | 2.0157343 | 2.0237006 | 0.9930314 | 0.9950085 | 0.9969559 | 0.9956900 | 0.0231406 | 0.1063657 | 101.5734327 | 102.3700593 | 0.1060503 |
6 | 0.1 | 0.9932859 | 2.0196106 | 2.0217603 | 0.9949410 | 0.9941552 | 0.996 | 0.9949619 | 0.0958103 | 0.2021760 | 101.9610625 | 102.1760312 | 0.2013876 |
7 | 0.15 | 0.9902694 | 2.0233842 | 2.0223016 | 0.9968 | 0.9919490 | 0.9962667 | 0.9939576 | 0.1011692 | 0.3033452 | 102.3384216 | 102.2301613 | 0.3022415 |
8 | 0.2 | 0.9840660 | 2.0136408 | 2.0201364 | 0.992 | 0.9873945 | 0.9952 | 0.9923168 | 0.1006820 | 0.4040273 | 101.3640792 | 102.0136408 | 0.4021351 |
9 | 0.3 | 0.9523897 | 1.9852225 | 2.0084984 | 0.978 | 0.9717051 | 0.9894667 | 0.9854462 | 0.1985222 | 0.6025495 | 98.5222475 | 100.8498430 | 0.5963212 |
10 | 0.4 | 0.8671057 | 1.9235141 | 1.9872524 | 0.9476 | 0.9142567 | 0.979 | 0.9676488 | 0.1923514 | 0.7949009 | 92.3514128 | 98.7252355 | 0.7783447 |
11 | 0.5 | 0.5822353 | 1.6174082 | 1.9132835 | 0.7968 | 0.7825059 | 0.94256 | 0.9306202 | 0.1617408 | 0.9566418 | 61.7408249 | 91.3283534 | 0.9000350 |
12 | 0.6 | 0.0667765 | 0.3775576 | 1.6573292 | 0.186 | 0.2480180 | 0.8164667 | 0.8168532 | 0.0377558 | 0.9943975 | -62.2442351 | 65.7329219 | 0.7773524 |
13 | 0.7 | 0.0125771 | 0.0186749 | 1.4232357 | 0.0092 | 0.0304482 | 0.7011429 | 0.7045096 | 0.0018675 | 0.9962650 | -98.1325106 | 42.3235744 | 0.5839345 |
14 | 0.80048 | 0.0057412 | 0.0185857 | 1.2469175 | 0.0091561 | 0.0081574 | 0.6142814 | 0.6171002 | 0.0018675 | 0.9981325 | -98.1414317 | 24.6917488 | 0.3895705 |
15 | 0.9014 | 0.0034592 | 0.0104592 | 1.1084846 | 0.0051526 | 0.0045211 | 0.5460839 | 0.5485164 | 0.0010555 | 0.9991880 | -98.9540849 | 10.8484633 | 0.1927390 |
16 | 1.0 | 0.0002609 | 0.0082348 | 1.0 | 0.0040568 | 0.0026446 | 0.49264 | 0.4946934 | 0.0008120 | 1.0 | -99.1765193 | 0.0 | 0.0 |
timestamp | duration | number_of_trees | training_rmse | training_logloss | training_auc | training_pr_auc | training_lift | training_classification_error | training_custom | validation_rmse | validation_logloss | validation_auc | validation_pr_auc | validation_lift | validation_classification_error | validation_custom | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
2022-10-28 14:19:30 | 0.025 sec | 0.0 | 0.5 | 0.6931472 | 0.5 | 0.5022800 | 1.0 | 0.49772 | 0.693147 | 0.5 | 0.6931472 | 0.5 | 0.4926400 | 1.0 | 0.50736 | 0.693147 | |
2022-10-28 14:19:30 | 0.139 sec | 1.0 | 0.3996010 | 0.5084556 | 0.9599605 | 0.9528824 | 1.9699443 | 0.08508 | 0.508456 | 0.4002939 | 0.5095766 | 0.9584754 | 0.9501212 | 2.0073256 | 0.08816 | 0.509577 | |
2022-10-28 14:19:30 | 0.179 sec | 2.0 | 0.3370778 | 0.4032048 | 0.9693358 | 0.9638432 | 1.9749781 | 0.0786933 | 0.403205 | 0.3378726 | 0.4043148 | 0.9689642 | 0.9627452 | 2.0180782 | 0.08072 | 0.404315 | |
2022-10-28 14:19:30 | 0.228 sec | 3.0 | 0.2957135 | 0.3335651 | 0.9735764 | 0.9692971 | 1.9823583 | 0.07124 | 0.333565 | 0.2972804 | 0.3355690 | 0.9727585 | 0.9672690 | 2.0164813 | 0.07328 | 0.335569 | |
2022-10-28 14:19:30 | 0.280 sec | 4.0 | 0.2690565 | 0.2867050 | 0.9769386 | 0.9736661 | 1.9824404 | 0.0687733 | 0.286705 | 0.2711930 | 0.2894303 | 0.9753680 | 0.9709177 | 2.0166126 | 0.07028 | 0.28943 | |
2022-10-28 14:19:30 | 0.356 sec | 5.0 | 0.2529770 | 0.2548994 | 0.9782349 | 0.9753981 | 1.9830013 | 0.06936 | 0.254899 | 0.2550437 | 0.2575265 | 0.9769549 | 0.9724197 | 2.0204531 | 0.06956 | 0.257526 | |
2022-10-28 14:19:30 | 0.434 sec | 6.0 | 0.2430550 | 0.2328859 | 0.9792536 | 0.9767333 | 1.9826945 | 0.06864 | 0.232886 | 0.2452571 | 0.2357295 | 0.9778085 | 0.9737537 | 2.0198804 | 0.06872 | 0.235729 | |
2022-10-28 14:19:30 | 0.511 sec | 7.0 | 0.2366703 | 0.2173043 | 0.9801005 | 0.9779518 | 1.9840125 | 0.0655333 | 0.217304 | 0.2388924 | 0.2202064 | 0.9787945 | 0.9751087 | 2.0189075 | 0.06696 | 0.220206 | |
2022-10-28 14:19:30 | 0.587 sec | 8.0 | 0.2274282 | 0.2004642 | 0.9816894 | 0.9795849 | 1.9883425 | 0.06172 | 0.200464 | 0.2302864 | 0.2040544 | 0.9803330 | 0.9772714 | 2.0219506 | 0.06516 | 0.204054 | |
2022-10-28 14:19:31 | 0.676 sec | 9.0 | 0.2235048 | 0.1908878 | 0.9823444 | 0.9805219 | 1.9883948 | 0.0608 | 0.190888 | 0.2267565 | 0.1951048 | 0.9808128 | 0.9779093 | 2.0222773 | 0.06432 | 0.195105 | |
--- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
2022-10-28 14:19:31 | 1.058 sec | 13.0 | 0.2093299 | 0.1618817 | 0.9849208 | 0.9835753 | 1.9885342 | 0.0549333 | 0.161882 | 0.2140217 | 0.1682975 | 0.9829089 | 0.9803703 | 2.0227574 | 0.0584 | 0.168298 | |
2022-10-28 14:19:31 | 1.160 sec | 14.0 | 0.2080116 | 0.1587226 | 0.9852452 | 0.9839538 | 1.9883086 | 0.0544 | 0.158723 | 0.2129618 | 0.1656925 | 0.9831326 | 0.9806934 | 2.0222773 | 0.05852 | 0.165693 | |
2022-10-28 14:19:31 | 1.246 sec | 15.0 | 0.2065626 | 0.1556114 | 0.9856595 | 0.9846933 | 1.9909214 | 0.0538533 | 0.155611 | 0.2118815 | 0.1633915 | 0.9833377 | 0.9810162 | 2.0227823 | 0.05792 | 0.163391 | |
2022-10-28 14:19:31 | 1.345 sec | 16.0 | 0.2036096 | 0.1517939 | 0.9863474 | 0.9853808 | 1.9909214 | 0.0520267 | 0.151794 | 0.2099121 | 0.1607776 | 0.9837030 | 0.9814828 | 2.0217603 | 0.0566 | 0.160778 | |
2022-10-28 14:19:31 | 1.462 sec | 17.0 | 0.2001280 | 0.1470822 | 0.9868868 | 0.9859104 | 1.9909214 | 0.0503733 | 0.147082 | 0.2068741 | 0.1565508 | 0.9842040 | 0.9821555 | 2.0298798 | 0.05524 | 0.156551 | |
2022-10-28 14:19:31 | 1.566 sec | 18.0 | 0.1952364 | 0.1409688 | 0.9877667 | 0.9868323 | 1.9909214 | 0.0479067 | 0.140969 | 0.2029770 | 0.1516665 | 0.9848696 | 0.9829746 | 2.0298798 | 0.05272 | 0.151666 | |
2022-10-28 14:19:32 | 1.684 sec | 19.0 | 0.1943690 | 0.1391751 | 0.9880068 | 0.9871283 | 1.9909214 | 0.0476667 | 0.139175 | 0.2025051 | 0.1505835 | 0.9849703 | 0.9829211 | 2.0298798 | 0.05232 | 0.150583 | |
2022-10-28 14:19:32 | 1.802 sec | 20.0 | 0.1919489 | 0.1361213 | 0.9884007 | 0.9875789 | 1.9909214 | 0.0466933 | 0.136121 | 0.2009170 | 0.1484152 | 0.9852986 | 0.9834889 | 2.0298798 | 0.05108 | 0.148415 | |
2022-10-28 14:19:32 | 1.916 sec | 21.0 | 0.1886257 | 0.1319356 | 0.9889611 | 0.9882193 | 1.9909214 | 0.0444533 | 0.131936 | 0.1983105 | 0.1448947 | 0.9857546 | 0.9838308 | 2.0217603 | 0.0496 | 0.144895 | |
2022-10-28 14:19:32 | 2.057 sec | 22.0 | 0.1864343 | 0.1290141 | 0.9893340 | 0.9887041 | 1.9909214 | 0.0438133 | 0.129014 | 0.1964693 | 0.1426421 | 0.9860368 | 0.9841777 | 2.0298798 | 0.049 | 0.142642 |
[23 rows x 18 columns]
variable | relative_importance | scaled_importance | percentage |
---|---|---|---|
C11 | 70178.75 | 1.0 | 0.4261358 |
C2 | 35732.6640625 | 0.5091664 | 0.2169741 |
C13 | 21608.4863281 | 0.3079064 | 0.1312100 |
C8 | 15654.5498047 | 0.2230668 | 0.0950568 |
C3 | 12402.2607422 | 0.1767239 | 0.0753084 |
C19 | 5739.7641602 | 0.0817878 | 0.0348527 |
C5 | 2835.9191895 | 0.0404099 | 0.0172201 |
C6 | 88.9352493 | 0.0012673 | 0.0005400 |
C12 | 51.9876480 | 0.0007408 | 0.0003157 |
C17 | 49.1711044 | 0.0007007 | 0.0002986 |
C7 | 48.0521317 | 0.0006847 | 0.0002918 |
C14 | 46.5472183 | 0.0006633 | 0.0002826 |
C10 | 43.8788605 | 0.0006252 | 0.0002664 |
C4 | 42.4346542 | 0.0006047 | 0.0002577 |
C18 | 39.3952980 | 0.0005614 | 0.0002392 |
C1 | 32.5539932 | 0.0004639 | 0.0001977 |
C15 | 25.9528160 | 0.0003698 | 0.0001576 |
C20 | 23.1192322 | 0.0003294 | 0.0001404 |
C16 | 22.9983807 | 0.0003277 | 0.0001396 |
C9 | 18.9113159 | 0.0002695 | 0.0001148 |
[tips] Use `model.explain()` to inspect the model. -- Use `h2o.display.toggle_user_tips()` to switch on/off this section.
The scoring history for model with eval_metric="logloss"
will look similar to the scoring history of the first model. This is expected - we didn't actually changed the training behavior, we only changed the source of the values that trigger early stopping. In this case, we are stopping on values of validation_custom
. This value correspond to the value calculated and returned by XGBoost. It should be close to H2O's own validation_logloss
value, there can be only a small difference caused by a different precision in XGBoost and H2O (the values should be within absolute tolerance of 1e-5). This is, however, something to keep in mind. There can be edge cases where H2O metric will differ slightly from the conceptually same XGBoost metric and this might cause the models to stop at a different iteration.
The scoring history will also have value of eval_metric
for the training frame - see column training_custom
.
model_eval.scoring_history()
timestamp | duration | number_of_trees | training_rmse | training_logloss | training_auc | training_pr_auc | training_lift | training_classification_error | training_custom | validation_rmse | validation_logloss | validation_auc | validation_pr_auc | validation_lift | validation_classification_error | validation_custom | ||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
0 | 2022-10-28 14:19:30 | 0.025 sec | 0.0 | 0.500000 | 0.693147 | 0.500000 | 0.502280 | 1.000000 | 0.497720 | 0.693147 | 0.500000 | 0.693147 | 0.500000 | 0.492640 | 1.000000 | 0.50736 | 0.693147 | |
1 | 2022-10-28 14:19:30 | 0.139 sec | 1.0 | 0.399601 | 0.508456 | 0.959960 | 0.952882 | 1.969944 | 0.085080 | 0.508456 | 0.400294 | 0.509577 | 0.958475 | 0.950121 | 2.007326 | 0.08816 | 0.509577 | |
2 | 2022-10-28 14:19:30 | 0.179 sec | 2.0 | 0.337078 | 0.403205 | 0.969336 | 0.963843 | 1.974978 | 0.078693 | 0.403205 | 0.337873 | 0.404315 | 0.968964 | 0.962745 | 2.018078 | 0.08072 | 0.404315 | |
3 | 2022-10-28 14:19:30 | 0.228 sec | 3.0 | 0.295714 | 0.333565 | 0.973576 | 0.969297 | 1.982358 | 0.071240 | 0.333565 | 0.297280 | 0.335569 | 0.972759 | 0.967269 | 2.016481 | 0.07328 | 0.335569 | |
4 | 2022-10-28 14:19:30 | 0.280 sec | 4.0 | 0.269057 | 0.286705 | 0.976939 | 0.973666 | 1.982440 | 0.068773 | 0.286705 | 0.271193 | 0.289430 | 0.975368 | 0.970918 | 2.016613 | 0.07028 | 0.289430 | |
5 | 2022-10-28 14:19:30 | 0.356 sec | 5.0 | 0.252977 | 0.254899 | 0.978235 | 0.975398 | 1.983001 | 0.069360 | 0.254899 | 0.255044 | 0.257526 | 0.976955 | 0.972420 | 2.020453 | 0.06956 | 0.257526 | |
6 | 2022-10-28 14:19:30 | 0.434 sec | 6.0 | 0.243055 | 0.232886 | 0.979254 | 0.976733 | 1.982694 | 0.068640 | 0.232886 | 0.245257 | 0.235729 | 0.977808 | 0.973754 | 2.019880 | 0.06872 | 0.235729 | |
7 | 2022-10-28 14:19:30 | 0.511 sec | 7.0 | 0.236670 | 0.217304 | 0.980100 | 0.977952 | 1.984012 | 0.065533 | 0.217304 | 0.238892 | 0.220206 | 0.978794 | 0.975109 | 2.018908 | 0.06696 | 0.220206 | |
8 | 2022-10-28 14:19:30 | 0.587 sec | 8.0 | 0.227428 | 0.200464 | 0.981689 | 0.979585 | 1.988342 | 0.061720 | 0.200464 | 0.230286 | 0.204054 | 0.980333 | 0.977271 | 2.021951 | 0.06516 | 0.204054 | |
9 | 2022-10-28 14:19:31 | 0.676 sec | 9.0 | 0.223505 | 0.190888 | 0.982344 | 0.980522 | 1.988395 | 0.060800 | 0.190888 | 0.226757 | 0.195105 | 0.980813 | 0.977909 | 2.022277 | 0.06432 | 0.195105 | |
10 | 2022-10-28 14:19:31 | 0.773 sec | 10.0 | 0.220040 | 0.182967 | 0.983000 | 0.981210 | 1.988298 | 0.059227 | 0.182967 | 0.223656 | 0.187705 | 0.981356 | 0.978627 | 2.021888 | 0.06240 | 0.187705 | |
11 | 2022-10-28 14:19:31 | 0.854 sec | 11.0 | 0.214926 | 0.173463 | 0.983878 | 0.982224 | 1.988819 | 0.057027 | 0.173463 | 0.218899 | 0.178748 | 0.982148 | 0.979434 | 2.023615 | 0.06084 | 0.178748 | |
12 | 2022-10-28 14:19:31 | 0.943 sec | 12.0 | 0.212074 | 0.167261 | 0.984348 | 0.982791 | 1.988617 | 0.055987 | 0.167261 | 0.216237 | 0.172957 | 0.982491 | 0.979711 | 2.022975 | 0.05932 | 0.172957 | |
13 | 2022-10-28 14:19:31 | 1.058 sec | 13.0 | 0.209330 | 0.161882 | 0.984921 | 0.983575 | 1.988534 | 0.054933 | 0.161882 | 0.214022 | 0.168298 | 0.982909 | 0.980370 | 2.022757 | 0.05840 | 0.168298 | |
14 | 2022-10-28 14:19:31 | 1.160 sec | 14.0 | 0.208012 | 0.158723 | 0.985245 | 0.983954 | 1.988309 | 0.054400 | 0.158723 | 0.212962 | 0.165693 | 0.983133 | 0.980693 | 2.022277 | 0.05852 | 0.165693 | |
15 | 2022-10-28 14:19:31 | 1.246 sec | 15.0 | 0.206563 | 0.155611 | 0.985659 | 0.984693 | 1.990921 | 0.053853 | 0.155611 | 0.211881 | 0.163391 | 0.983338 | 0.981016 | 2.022782 | 0.05792 | 0.163391 | |
16 | 2022-10-28 14:19:31 | 1.345 sec | 16.0 | 0.203610 | 0.151794 | 0.986347 | 0.985381 | 1.990921 | 0.052027 | 0.151794 | 0.209912 | 0.160778 | 0.983703 | 0.981483 | 2.021760 | 0.05660 | 0.160778 | |
17 | 2022-10-28 14:19:31 | 1.462 sec | 17.0 | 0.200128 | 0.147082 | 0.986887 | 0.985910 | 1.990921 | 0.050373 | 0.147082 | 0.206874 | 0.156551 | 0.984204 | 0.982155 | 2.029880 | 0.05524 | 0.156551 | |
18 | 2022-10-28 14:19:31 | 1.566 sec | 18.0 | 0.195236 | 0.140969 | 0.987767 | 0.986832 | 1.990921 | 0.047907 | 0.140969 | 0.202977 | 0.151666 | 0.984870 | 0.982975 | 2.029880 | 0.05272 | 0.151666 | |
19 | 2022-10-28 14:19:32 | 1.684 sec | 19.0 | 0.194369 | 0.139175 | 0.988007 | 0.987128 | 1.990921 | 0.047667 | 0.139175 | 0.202505 | 0.150583 | 0.984970 | 0.982921 | 2.029880 | 0.05232 | 0.150583 | |
20 | 2022-10-28 14:19:32 | 1.802 sec | 20.0 | 0.191949 | 0.136121 | 0.988401 | 0.987579 | 1.990921 | 0.046693 | 0.136121 | 0.200917 | 0.148415 | 0.985299 | 0.983489 | 2.029880 | 0.05108 | 0.148415 | |
21 | 2022-10-28 14:19:32 | 1.916 sec | 21.0 | 0.188626 | 0.131936 | 0.988961 | 0.988219 | 1.990921 | 0.044453 | 0.131936 | 0.198310 | 0.144895 | 0.985755 | 0.983831 | 2.021760 | 0.04960 | 0.144895 | |
22 | 2022-10-28 14:19:32 | 2.057 sec | 22.0 | 0.186434 | 0.129014 | 0.989334 | 0.988704 | 1.990921 | 0.043813 | 0.129014 | 0.196469 | 0.142642 | 0.986037 | 0.984178 | 2.029880 | 0.04900 | 0.142642 |
In this example we will keep the same parameters as in the previous case and add flag score_eval_metric_only=True
. This flag will instruct H2O to disable its own scoring and solely rely on eval_metric
for early stopping and recording the scoring history.
model_eval_only = H2OXGBoostEstimator(ntrees=1000, max_depth=6, score_each_iteration=True,
eval_metric="logloss",
stopping_rounds=3, stopping_tolerance=0.05, stopping_metric="custom",
score_eval_metric_only=True)
model_eval_only.train(y="y", training_frame=train, validation_frame=valid)
xgboost Model Build progress: |██████████████████████████████████████████████████| (done) 100%
Model Details ============= H2OXGBoostEstimator : XGBoost Model Key: XGBoost_model_python_1666962310297_616
number_of_trees | |
---|---|
22.0 |
ModelMetricsBinomial: xgboost ** Reported on train data. ** MSE: 0.03475774769265435 RMSE: 0.18643429859511995 LogLoss: 0.12901406890397824 Mean Per-Class Error: 0.04387881533186782 AUC: 0.9893340459497513 AUCPR: 0.988704085388505 Gini: 0.9786680918995025
0 | 1 | Error | Rate | |
---|---|---|---|---|
0 | 35155.0 | 2174.0 | 0.0582 | (2174.0/37329.0) |
1 | 1112.0 | 36559.0 | 0.0295 | (1112.0/37671.0) |
Total | 36267.0 | 38733.0 | 0.0438 | (3286.0/75000.0) |
metric | threshold | value | idx |
---|---|---|---|
max f1 | 0.5101676 | 0.9569918 | 214.0 |
max f2 | 0.3082806 | 0.9715081 | 263.0 |
max f0point5 | 0.7532607 | 0.9592084 | 147.0 |
max accuracy | 0.5561712 | 0.9564 | 201.0 |
max precision | 0.9974565 | 1.0 | 0.0 |
max recall | 0.0016028 | 1.0 | 399.0 |
max specificity | 0.9974565 | 1.0 | 0.0 |
max absolute_mcc | 0.5561712 | 0.9129277 | 201.0 |
max min_per_class_accuracy | 0.6239853 | 0.9548073 | 184.0 |
max mean_per_class_accuracy | 0.5561712 | 0.9563597 | 201.0 |
max tns | 0.9974565 | 37329.0 | 0.0 |
max fns | 0.9974565 | 37460.0 | 0.0 |
max fps | 0.0016028 | 37329.0 | 399.0 |
max tps | 0.0016028 | 37671.0 | 399.0 |
max tnr | 0.9974565 | 1.0 | 0.0 |
max fnr | 0.9974565 | 0.9943989 | 0.0 |
max fpr | 0.0016028 | 1.0 | 399.0 |
max tpr | 0.0016028 | 1.0 | 399.0 |
group | cumulative_data_fraction | lower_threshold | lift | cumulative_lift | response_rate | score | cumulative_response_rate | cumulative_score | capture_rate | cumulative_capture_rate | gain | cumulative_gain | kolmogorov_smirnov |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|
1 | 0.0101733 | 0.9961097 | 1.9909214 | 1.9909214 | 1.0 | 0.9967516 | 1.0 | 0.9967516 | 0.0202543 | 0.0202543 | 99.0921398 | 99.0921398 | 0.0202543 |
2 | 0.0200267 | 0.9957153 | 1.9909214 | 1.9909214 | 1.0 | 0.9958894 | 1.0 | 0.9963274 | 0.0196172 | 0.0398715 | 99.0921398 | 99.0921398 | 0.0398715 |
3 | 0.03068 | 0.9953964 | 1.9909214 | 1.9909214 | 1.0 | 0.9955495 | 1.0 | 0.9960573 | 0.0212099 | 0.0610815 | 99.0921398 | 99.0921398 | 0.0610815 |
4 | 0.0401067 | 0.9951944 | 1.9881054 | 1.9902595 | 0.9985856 | 0.9952708 | 0.9996676 | 0.9958724 | 0.0187412 | 0.0798227 | 98.8105385 | 99.0259523 | 0.0797959 |
5 | 0.0517733 | 0.9949297 | 1.9863707 | 1.9893832 | 0.9977143 | 0.9950084 | 0.9992274 | 0.9956777 | 0.0231743 | 0.1029970 | 98.6370721 | 98.9383216 | 0.1029166 |
6 | 0.1000133 | 0.9933097 | 1.9881700 | 1.9887980 | 0.9986180 | 0.9941657 | 0.9989335 | 0.9949484 | 0.0959093 | 0.1989063 | 98.8169987 | 98.8798032 | 0.1986920 |
7 | 0.15008 | 0.9904984 | 1.9840287 | 1.9872070 | 0.9965379 | 0.9920577 | 0.9981343 | 0.9939841 | 0.0993337 | 0.2982400 | 98.4028728 | 98.7206993 | 0.2976775 |
8 | 0.2000133 | 0.9849605 | 1.9824155 | 1.9860108 | 0.9957276 | 0.9879125 | 0.9975335 | 0.9924683 | 0.0989886 | 0.3972286 | 98.2415459 | 98.6010786 | 0.3962375 |
9 | 0.30004 | 0.9561046 | 1.9673021 | 1.9797737 | 0.9881365 | 0.9740387 | 0.9944007 | 0.9863243 | 0.1967827 | 0.5940113 | 96.7302096 | 97.9773725 | 0.5906359 |
10 | 0.4000133 | 0.8770613 | 1.9123254 | 1.9629167 | 0.9605228 | 0.9210076 | 0.9859338 | 0.9700000 | 0.1911815 | 0.7851929 | 91.2325408 | 96.2916704 | 0.7738880 |
11 | 0.5000133 | 0.6383476 | 1.6763558 | 1.9056061 | 0.842 | 0.8051186 | 0.9571478 | 0.9370246 | 0.1676356 | 0.9528284 | 67.6355817 | 90.5606055 | 0.9097788 |
12 | 0.6 | 0.0781245 | 0.4261140 | 1.6590569 | 0.2140285 | 0.2876185 | 0.8333111 | 0.8288047 | 0.0426057 | 0.9954342 | -57.3886006 | 65.9056923 | 0.7944912 |
13 | 0.7000133 | 0.0128222 | 0.0238879 | 1.4254346 | 0.0119984 | 0.0339001 | 0.7159673 | 0.7152339 | 0.0023891 | 0.9978233 | -97.6112128 | 42.5434648 | 0.5983483 |
14 | 0.80008 | 0.0057930 | 0.0122028 | 1.2486806 | 0.0061292 | 0.0084165 | 0.6271873 | 0.6268317 | 0.0012211 | 0.9990444 | -98.7797151 | 24.8680579 | 0.3997516 |
15 | 0.9 | 0.0034428 | 0.0045164 | 1.1105507 | 0.0022685 | 0.0045299 | 0.5578074 | 0.5577424 | 0.0004513 | 0.9994956 | -99.5483632 | 11.0550704 | 0.1999028 |
16 | 1.0 | 0.0001672 | 0.0050437 | 1.0 | 0.0025333 | 0.0026268 | 0.50228 | 0.5022308 | 0.0005044 | 1.0 | -99.4956332 | 0.0 | 0.0 |
ModelMetricsBinomial: xgboost ** Reported on validation data. ** MSE: 0.038600168581279404 RMSE: 0.1964692560714765 LogLoss: 0.14264211397729937 Mean Per-Class Error: 0.04874265748103474 AUC: 0.9860367984758349 AUCPR: 0.9841777019906762 Gini: 0.9720735969516698
0 | 1 | Error | Rate | |
---|---|---|---|---|
0 | 11844.0 | 840.0 | 0.0662 | (840.0/12684.0) |
1 | 385.0 | 11931.0 | 0.0313 | (385.0/12316.0) |
Total | 12229.0 | 12771.0 | 0.049 | (1225.0/25000.0) |
metric | threshold | value | idx |
---|---|---|---|
max f1 | 0.4899956 | 0.9511699 | 217.0 |
max f2 | 0.2381639 | 0.9684011 | 281.0 |
max f0point5 | 0.7637601 | 0.9521032 | 144.0 |
max accuracy | 0.4899956 | 0.951 | 217.0 |
max precision | 0.9974683 | 1.0 | 0.0 |
max recall | 0.0018354 | 1.0 | 398.0 |
max specificity | 0.9974683 | 1.0 | 0.0 |
max absolute_mcc | 0.4899956 | 0.9026291 | 217.0 |
max min_per_class_accuracy | 0.6303638 | 0.9494966 | 182.0 |
max mean_per_class_accuracy | 0.4899956 | 0.9512573 | 217.0 |
max tns | 0.9974683 | 12684.0 | 0.0 |
max fns | 0.9974683 | 12244.0 | 0.0 |
max fps | 0.0013435 | 12684.0 | 399.0 |
max tps | 0.0018354 | 12316.0 | 398.0 |
max tnr | 0.9974683 | 1.0 | 0.0 |
max fnr | 0.9974683 | 0.9941539 | 0.0 |
max fpr | 0.0013435 | 1.0 | 399.0 |
max tpr | 0.0018354 | 1.0 | 398.0 |
group | cumulative_data_fraction | lower_threshold | lift | cumulative_lift | response_rate | score | cumulative_response_rate | cumulative_score | capture_rate | cumulative_capture_rate | gain | cumulative_gain | kolmogorov_smirnov |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|
1 | 0.01028 | 0.9961244 | 2.0298798 | 2.0298798 | 1.0 | 0.9967313 | 1.0 | 0.9967313 | 0.0208672 | 0.0208672 | 102.9879831 | 102.9879831 | 0.0208672 |
2 | 0.02032 | 0.9958012 | 2.0298798 | 2.0298798 | 1.0 | 0.9959175 | 1.0 | 0.9963292 | 0.0203800 | 0.0412472 | 102.9879831 | 102.9879831 | 0.0412472 |
3 | 0.03016 | 0.9954615 | 2.0216283 | 2.0271877 | 0.9959350 | 0.9956102 | 0.9986737 | 0.9960946 | 0.0198928 | 0.0611400 | 102.1628287 | 102.7187683 | 0.0610611 |
4 | 0.04108 | 0.9951944 | 2.0224444 | 2.0259268 | 0.9963370 | 0.9952887 | 0.9980526 | 0.9958804 | 0.0220851 | 0.0832251 | 102.2444374 | 102.5926803 | 0.0830674 |
5 | 0.05256 | 0.9949297 | 2.0157343 | 2.0237006 | 0.9930314 | 0.9950085 | 0.9969559 | 0.9956900 | 0.0231406 | 0.1063657 | 101.5734327 | 102.3700593 | 0.1060503 |
6 | 0.1 | 0.9932859 | 2.0196106 | 2.0217603 | 0.9949410 | 0.9941552 | 0.996 | 0.9949619 | 0.0958103 | 0.2021760 | 101.9610625 | 102.1760312 | 0.2013876 |
7 | 0.15 | 0.9902694 | 2.0233842 | 2.0223016 | 0.9968 | 0.9919490 | 0.9962667 | 0.9939576 | 0.1011692 | 0.3033452 | 102.3384216 | 102.2301613 | 0.3022415 |
8 | 0.2 | 0.9840660 | 2.0136408 | 2.0201364 | 0.992 | 0.9873945 | 0.9952 | 0.9923168 | 0.1006820 | 0.4040273 | 101.3640792 | 102.0136408 | 0.4021351 |
9 | 0.3 | 0.9523897 | 1.9852225 | 2.0084984 | 0.978 | 0.9717051 | 0.9894667 | 0.9854462 | 0.1985222 | 0.6025495 | 98.5222475 | 100.8498430 | 0.5963212 |
10 | 0.4 | 0.8671057 | 1.9235141 | 1.9872524 | 0.9476 | 0.9142567 | 0.979 | 0.9676488 | 0.1923514 | 0.7949009 | 92.3514128 | 98.7252355 | 0.7783447 |
11 | 0.5 | 0.5822353 | 1.6174082 | 1.9132835 | 0.7968 | 0.7825059 | 0.94256 | 0.9306202 | 0.1617408 | 0.9566418 | 61.7408249 | 91.3283534 | 0.9000350 |
12 | 0.6 | 0.0667765 | 0.3775576 | 1.6573292 | 0.186 | 0.2480180 | 0.8164667 | 0.8168532 | 0.0377558 | 0.9943975 | -62.2442351 | 65.7329219 | 0.7773524 |
13 | 0.7 | 0.0125771 | 0.0186749 | 1.4232357 | 0.0092 | 0.0304482 | 0.7011429 | 0.7045096 | 0.0018675 | 0.9962650 | -98.1325106 | 42.3235744 | 0.5839345 |
14 | 0.80048 | 0.0057412 | 0.0185857 | 1.2469175 | 0.0091561 | 0.0081574 | 0.6142814 | 0.6171002 | 0.0018675 | 0.9981325 | -98.1414317 | 24.6917488 | 0.3895705 |
15 | 0.9014 | 0.0034592 | 0.0104592 | 1.1084846 | 0.0051526 | 0.0045211 | 0.5460839 | 0.5485164 | 0.0010555 | 0.9991880 | -98.9540849 | 10.8484633 | 0.1927390 |
16 | 1.0 | 0.0002609 | 0.0082348 | 1.0 | 0.0040568 | 0.0026446 | 0.49264 | 0.4946934 | 0.0008120 | 1.0 | -99.1765193 | 0.0 | 0.0 |
timestamp | duration | number_of_trees | training_rmse | training_logloss | training_auc | training_pr_auc | training_lift | training_classification_error | training_custom | validation_rmse | validation_logloss | validation_auc | validation_pr_auc | validation_lift | validation_classification_error | validation_custom | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
2022-10-28 14:19:33 | 0.030 sec | 0.0 | nan | nan | nan | nan | nan | nan | 0.693147 | nan | nan | nan | nan | nan | nan | 0.693147 | |
2022-10-28 14:19:33 | 0.118 sec | 1.0 | nan | nan | nan | nan | nan | nan | 0.508456 | nan | nan | nan | nan | nan | nan | 0.509577 | |
2022-10-28 14:19:33 | 0.127 sec | 2.0 | nan | nan | nan | nan | nan | nan | 0.403205 | nan | nan | nan | nan | nan | nan | 0.404315 | |
2022-10-28 14:19:33 | 0.135 sec | 3.0 | nan | nan | nan | nan | nan | nan | 0.333565 | nan | nan | nan | nan | nan | nan | 0.335569 | |
2022-10-28 14:19:33 | 0.144 sec | 4.0 | nan | nan | nan | nan | nan | nan | 0.286705 | nan | nan | nan | nan | nan | nan | 0.28943 | |
2022-10-28 14:19:33 | 0.151 sec | 5.0 | nan | nan | nan | nan | nan | nan | 0.254899 | nan | nan | nan | nan | nan | nan | 0.257526 | |
2022-10-28 14:19:33 | 0.161 sec | 6.0 | nan | nan | nan | nan | nan | nan | 0.232886 | nan | nan | nan | nan | nan | nan | 0.235729 | |
2022-10-28 14:19:33 | 0.169 sec | 7.0 | nan | nan | nan | nan | nan | nan | 0.217304 | nan | nan | nan | nan | nan | nan | 0.220206 | |
2022-10-28 14:19:33 | 0.177 sec | 8.0 | nan | nan | nan | nan | nan | nan | 0.200464 | nan | nan | nan | nan | nan | nan | 0.204054 | |
2022-10-28 14:19:33 | 0.185 sec | 9.0 | nan | nan | nan | nan | nan | nan | 0.190888 | nan | nan | nan | nan | nan | nan | 0.195105 | |
--- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
2022-10-28 14:19:33 | 0.224 sec | 13.0 | nan | nan | nan | nan | nan | nan | 0.161882 | nan | nan | nan | nan | nan | nan | 0.168298 | |
2022-10-28 14:19:33 | 0.234 sec | 14.0 | nan | nan | nan | nan | nan | nan | 0.158723 | nan | nan | nan | nan | nan | nan | 0.165693 | |
2022-10-28 14:19:33 | 0.243 sec | 15.0 | nan | nan | nan | nan | nan | nan | 0.155611 | nan | nan | nan | nan | nan | nan | 0.163391 | |
2022-10-28 14:19:33 | 0.251 sec | 16.0 | nan | nan | nan | nan | nan | nan | 0.151794 | nan | nan | nan | nan | nan | nan | 0.160778 | |
2022-10-28 14:19:33 | 0.261 sec | 17.0 | nan | nan | nan | nan | nan | nan | 0.147082 | nan | nan | nan | nan | nan | nan | 0.156551 | |
2022-10-28 14:19:33 | 0.270 sec | 18.0 | nan | nan | nan | nan | nan | nan | 0.140969 | nan | nan | nan | nan | nan | nan | 0.151666 | |
2022-10-28 14:19:33 | 0.279 sec | 19.0 | nan | nan | nan | nan | nan | nan | 0.139175 | nan | nan | nan | nan | nan | nan | 0.150583 | |
2022-10-28 14:19:33 | 0.291 sec | 20.0 | nan | nan | nan | nan | nan | nan | 0.136121 | nan | nan | nan | nan | nan | nan | 0.148415 | |
2022-10-28 14:19:33 | 0.301 sec | 21.0 | nan | nan | nan | nan | nan | nan | 0.131936 | nan | nan | nan | nan | nan | nan | 0.144895 | |
2022-10-28 14:19:33 | 0.310 sec | 22.0 | 0.1864343 | 0.1290141 | 0.9893340 | 0.9887041 | 1.9909214 | 0.0438133 | 0.129014 | 0.1964693 | 0.1426421 | 0.9860368 | 0.9841777 | 2.0298798 | 0.049 | 0.142642 |
[23 rows x 18 columns]
variable | relative_importance | scaled_importance | percentage |
---|---|---|---|
C11 | 70178.75 | 1.0 | 0.4261358 |
C2 | 35732.6640625 | 0.5091664 | 0.2169741 |
C13 | 21608.4863281 | 0.3079064 | 0.1312100 |
C8 | 15654.5498047 | 0.2230668 | 0.0950568 |
C3 | 12402.2607422 | 0.1767239 | 0.0753084 |
C19 | 5739.7641602 | 0.0817878 | 0.0348527 |
C5 | 2835.9191895 | 0.0404099 | 0.0172201 |
C6 | 88.9352493 | 0.0012673 | 0.0005400 |
C12 | 51.9876480 | 0.0007408 | 0.0003157 |
C17 | 49.1711044 | 0.0007007 | 0.0002986 |
C7 | 48.0521317 | 0.0006847 | 0.0002918 |
C14 | 46.5472183 | 0.0006633 | 0.0002826 |
C10 | 43.8788605 | 0.0006252 | 0.0002664 |
C4 | 42.4346542 | 0.0006047 | 0.0002577 |
C18 | 39.3952980 | 0.0005614 | 0.0002392 |
C1 | 32.5539932 | 0.0004639 | 0.0001977 |
C15 | 25.9528160 | 0.0003698 | 0.0001576 |
C20 | 23.1192322 | 0.0003294 | 0.0001404 |
C16 | 22.9983807 | 0.0003277 | 0.0001396 |
C9 | 18.9113159 | 0.0002695 | 0.0001148 |
[tips] Use `model.explain()` to inspect the model. -- Use `h2o.display.toggle_user_tips()` to switch on/off this section.
Scoring history will show undefined values for H2O metrics for all scoring iteration except for the final one. Values of columns training_custom
and validation_custom
will be the only ones populated for all of the iterations.
In the final iteration H2O performs full scoring, that is why we see all values defined in the last row of the scoring history.
model_eval_only.scoring_history()
timestamp | duration | number_of_trees | training_rmse | training_logloss | training_auc | training_pr_auc | training_lift | training_classification_error | training_custom | validation_rmse | validation_logloss | validation_auc | validation_pr_auc | validation_lift | validation_classification_error | validation_custom | ||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
0 | 2022-10-28 14:19:33 | 0.030 sec | 0.0 | NaN | NaN | NaN | NaN | NaN | NaN | 0.693147 | NaN | NaN | NaN | NaN | NaN | NaN | 0.693147 | |
1 | 2022-10-28 14:19:33 | 0.118 sec | 1.0 | NaN | NaN | NaN | NaN | NaN | NaN | 0.508456 | NaN | NaN | NaN | NaN | NaN | NaN | 0.509577 | |
2 | 2022-10-28 14:19:33 | 0.127 sec | 2.0 | NaN | NaN | NaN | NaN | NaN | NaN | 0.403205 | NaN | NaN | NaN | NaN | NaN | NaN | 0.404315 | |
3 | 2022-10-28 14:19:33 | 0.135 sec | 3.0 | NaN | NaN | NaN | NaN | NaN | NaN | 0.333565 | NaN | NaN | NaN | NaN | NaN | NaN | 0.335569 | |
4 | 2022-10-28 14:19:33 | 0.144 sec | 4.0 | NaN | NaN | NaN | NaN | NaN | NaN | 0.286705 | NaN | NaN | NaN | NaN | NaN | NaN | 0.289430 | |
5 | 2022-10-28 14:19:33 | 0.151 sec | 5.0 | NaN | NaN | NaN | NaN | NaN | NaN | 0.254899 | NaN | NaN | NaN | NaN | NaN | NaN | 0.257526 | |
6 | 2022-10-28 14:19:33 | 0.161 sec | 6.0 | NaN | NaN | NaN | NaN | NaN | NaN | 0.232886 | NaN | NaN | NaN | NaN | NaN | NaN | 0.235729 | |
7 | 2022-10-28 14:19:33 | 0.169 sec | 7.0 | NaN | NaN | NaN | NaN | NaN | NaN | 0.217304 | NaN | NaN | NaN | NaN | NaN | NaN | 0.220206 | |
8 | 2022-10-28 14:19:33 | 0.177 sec | 8.0 | NaN | NaN | NaN | NaN | NaN | NaN | 0.200464 | NaN | NaN | NaN | NaN | NaN | NaN | 0.204054 | |
9 | 2022-10-28 14:19:33 | 0.185 sec | 9.0 | NaN | NaN | NaN | NaN | NaN | NaN | 0.190888 | NaN | NaN | NaN | NaN | NaN | NaN | 0.195105 | |
10 | 2022-10-28 14:19:33 | 0.194 sec | 10.0 | NaN | NaN | NaN | NaN | NaN | NaN | 0.182967 | NaN | NaN | NaN | NaN | NaN | NaN | 0.187705 | |
11 | 2022-10-28 14:19:33 | 0.202 sec | 11.0 | NaN | NaN | NaN | NaN | NaN | NaN | 0.173463 | NaN | NaN | NaN | NaN | NaN | NaN | 0.178748 | |
12 | 2022-10-28 14:19:33 | 0.213 sec | 12.0 | NaN | NaN | NaN | NaN | NaN | NaN | 0.167261 | NaN | NaN | NaN | NaN | NaN | NaN | 0.172957 | |
13 | 2022-10-28 14:19:33 | 0.224 sec | 13.0 | NaN | NaN | NaN | NaN | NaN | NaN | 0.161882 | NaN | NaN | NaN | NaN | NaN | NaN | 0.168298 | |
14 | 2022-10-28 14:19:33 | 0.234 sec | 14.0 | NaN | NaN | NaN | NaN | NaN | NaN | 0.158723 | NaN | NaN | NaN | NaN | NaN | NaN | 0.165693 | |
15 | 2022-10-28 14:19:33 | 0.243 sec | 15.0 | NaN | NaN | NaN | NaN | NaN | NaN | 0.155611 | NaN | NaN | NaN | NaN | NaN | NaN | 0.163391 | |
16 | 2022-10-28 14:19:33 | 0.251 sec | 16.0 | NaN | NaN | NaN | NaN | NaN | NaN | 0.151794 | NaN | NaN | NaN | NaN | NaN | NaN | 0.160778 | |
17 | 2022-10-28 14:19:33 | 0.261 sec | 17.0 | NaN | NaN | NaN | NaN | NaN | NaN | 0.147082 | NaN | NaN | NaN | NaN | NaN | NaN | 0.156551 | |
18 | 2022-10-28 14:19:33 | 0.270 sec | 18.0 | NaN | NaN | NaN | NaN | NaN | NaN | 0.140969 | NaN | NaN | NaN | NaN | NaN | NaN | 0.151666 | |
19 | 2022-10-28 14:19:33 | 0.279 sec | 19.0 | NaN | NaN | NaN | NaN | NaN | NaN | 0.139175 | NaN | NaN | NaN | NaN | NaN | NaN | 0.150583 | |
20 | 2022-10-28 14:19:33 | 0.291 sec | 20.0 | NaN | NaN | NaN | NaN | NaN | NaN | 0.136121 | NaN | NaN | NaN | NaN | NaN | NaN | 0.148415 | |
21 | 2022-10-28 14:19:33 | 0.301 sec | 21.0 | NaN | NaN | NaN | NaN | NaN | NaN | 0.131936 | NaN | NaN | NaN | NaN | NaN | NaN | 0.144895 | |
22 | 2022-10-28 14:19:33 | 0.310 sec | 22.0 | 0.186434 | 0.129014 | 0.989334 | 0.988704 | 1.990921 | 0.043813 | 0.129014 | 0.196469 | 0.142642 | 0.986037 | 0.984178 | 2.02988 | 0.049 | 0.142642 |
Models model_eval
and model_eval_only
are guaranteed to be identical in behavior (same trees, same thresholds,..). The only technical difference between them is that the first one doesn't have full scoring history.
We can also see that flag score_eval_metric_only=True
saved us some training time. Model model_eval_only
was built faster:
def total_duration(scoring_history):
return(sum(map(lambda x: float(x.strip().split(' ')[0]), scoring_history["duration"].tolist())))
print("Duration (s) with H2O scoring: %s" % total_duration(model_eval.scoring_history()))
print("Duration (s) with only eval_metric scored: %s" % total_duration(model_eval_only.scoring_history()))
Duration (s) with H2O scoring: 21.281 Duration (s) with only eval_metric scored: 4.67
model_eval.scoring_history()["duration"]
0 0.025 sec 1 0.139 sec 2 0.179 sec 3 0.228 sec 4 0.280 sec 5 0.356 sec 6 0.434 sec 7 0.511 sec 8 0.587 sec 9 0.676 sec 10 0.773 sec 11 0.854 sec 12 0.943 sec 13 1.058 sec 14 1.160 sec 15 1.246 sec 16 1.345 sec 17 1.462 sec 18 1.566 sec 19 1.684 sec 20 1.802 sec 21 1.916 sec 22 2.057 sec Name: duration, dtype: object
model_eval_only.scoring_history()["duration"]
0 0.030 sec 1 0.118 sec 2 0.127 sec 3 0.135 sec 4 0.144 sec 5 0.151 sec 6 0.161 sec 7 0.169 sec 8 0.177 sec 9 0.185 sec 10 0.194 sec 11 0.202 sec 12 0.213 sec 13 0.224 sec 14 0.234 sec 15 0.243 sec 16 0.251 sec 17 0.261 sec 18 0.270 sec 19 0.279 sec 20 0.291 sec 21 0.301 sec 22 0.310 sec Name: duration, dtype: object