import h2o
from h2o.estimators import H2OGradientBoostingEstimator
from h2o.utils.shared_utils import _locate # private function - used to find files within h2o git project directory
h2o.init(strict_version_check=False, port=54321)
Checking whether there is an H2O instance running at http://localhost:54321 . connected.
H2O_cluster_uptime: | 38 mins 14 secs |
H2O_cluster_timezone: | Europe/Berlin |
H2O_data_parsing_timezone: | UTC |
H2O_cluster_version: | 3.33.0.99999 |
H2O_cluster_version_age: | 47 minutes |
H2O_cluster_name: | mori |
H2O_cluster_total_nodes: | 1 |
H2O_cluster_free_memory: | 4.851 Gb |
H2O_cluster_total_cores: | 8 |
H2O_cluster_allowed_cores: | 8 |
H2O_cluster_status: | locked, healthy |
H2O_connection_url: | http://localhost:54321 |
H2O_connection_proxy: | {"http": null, "https": null} |
H2O_internal_security: | False |
H2O_API_Extensions: | Algos, Core V3, Core V4 |
Python_version: | 3.7.3 candidate |
cars = h2o.import_file(path=_locate("smalldata/junit/cars_20mpg.csv"))
features = ["displacement", "power", "weight", "acceleration", "year"]
response = "cylinders"
distribution = "multinomial"
cars[response] = cars[response].asfactor()
r = cars[0].runif()
train = cars[r > .3]
valid = cars[r <= .3]
# train model
gbm = H2OGradientBoostingEstimator(distribution="multinomial",
ntrees=100,
max_depth=3,
learn_rate=0.01,
auc_type="MACRO_OVR")
gbm.train(x =features,
y =response,
training_frame =train,
validation_frame=valid)
gbm.show()
Parse progress: |█████████████████████████████████████████████████████████| 100% gbm Model Build progress: |███████████████████████████████████████████████| 100% Model Details ============= H2OGradientBoostingEstimator : Gradient Boosting Machine Model Key: GBM_model_python_1606223609474_67 Model Summary:
number_of_trees | number_of_internal_trees | model_size_in_bytes | min_depth | max_depth | mean_depth | min_leaves | max_leaves | mean_leaves | ||
---|---|---|---|---|---|---|---|---|---|---|
0 | 100.0 | 500.0 | 67451.0 | 1.0 | 3.0 | 2.994 | 2.0 | 8.0 | 6.078 |
ModelMetricsMultinomial: gbm ** Reported on train data. ** MSE: 0.07963805354633585 RMSE: 0.28220215014477806 LogLoss: 0.32402399831181045 Mean Per-Class Error: 0.4120046082949308 AUC: 0.999511857555245 AUCPR: 0.998951571914844 Multinomial AUC values:
type | first_class_domain | second_class_domain | auc | |
---|---|---|---|---|
0 | 3 vs Rest | 3 | None | 1.000000 |
1 | 4 vs Rest | 4 | None | 0.998556 |
2 | 5 vs Rest | 5 | None | 1.000000 |
3 | 6 vs Rest | 6 | None | 0.999004 |
4 | 8 vs Rest | 8 | None | 1.000000 |
5 | Macro OVR | None | None | 0.999512 |
6 | Weighted OVR | None | None | 0.999032 |
7 | Class 3 vs. 4 | 3 | 4 | 0.994624 |
8 | Class 3 vs. 5 | 3 | 5 | 1.000000 |
9 | Class 3 vs. 6 | 3 | 6 | 1.000000 |
10 | Class 3 vs. 8 | 3 | 8 | 1.000000 |
11 | Class 4 vs. 5 | 4 | 5 | 0.975269 |
12 | Class 4 vs. 6 | 4 | 6 | 0.999482 |
13 | Class 4 vs. 8 | 4 | 8 | 1.000000 |
14 | Class 5 vs. 6 | 5 | 6 | 0.988095 |
15 | Class 5 vs. 8 | 5 | 8 | 1.000000 |
16 | Class 6 vs. 8 | 6 | 8 | 0.999752 |
17 | Macro OVO | None | None | 0.995722 |
18 | Weighted OVO | None | None | 0.995155 |
Multinomial auc_pr values:
type | first_class_domain | second_class_domain | auc_pr | |
---|---|---|---|---|
0 | 3 vs Rest | 3 | None | 1.000000 |
1 | 4 vs Rest | 4 | None | 0.998698 |
2 | 5 vs Rest | 5 | None | 1.000000 |
3 | 6 vs Rest | 6 | None | 0.996059 |
4 | 8 vs Rest | 8 | None | 1.000000 |
5 | Macro OVR | None | None | 0.998952 |
6 | Weighted OVR | None | None | 0.998538 |
7 | Class 3 vs. 4 | 3 | 4 | 0.999897 |
8 | Class 3 vs. 5 | 3 | 5 | 1.000000 |
9 | Class 3 vs. 6 | 3 | 6 | 1.000000 |
10 | Class 3 vs. 8 | 3 | 8 | 1.000000 |
11 | Class 4 vs. 5 | 4 | 5 | 0.999489 |
12 | Class 4 vs. 6 | 4 | 6 | 0.998871 |
13 | Class 4 vs. 8 | 4 | 8 | 1.000000 |
14 | Class 5 vs. 6 | 5 | 6 | 0.999350 |
15 | Class 5 vs. 8 | 5 | 8 | 1.000000 |
16 | Class 6 vs. 8 | 6 | 8 | 0.999689 |
17 | Macro OVO | None | None | 0.999730 |
18 | Weighted OVO | None | None | 0.999642 |
Confusion Matrix: Row labels: Actual class; Column labels: Predicted class
3 | 4 | 5 | 6 | 8 | Error | Rate | |
---|---|---|---|---|---|---|---|
0 | 0.0 | 3.0 | 0.0 | 0.0 | 0.0 | 1.000000 | 3 / 3 |
1 | 0.0 | 154.0 | 0.0 | 1.0 | 0.0 | 0.006452 | 1 / 155 |
2 | 0.0 | 2.0 | 0.0 | 1.0 | 0.0 | 1.000000 | 3 / 3 |
3 | 0.0 | 2.0 | 0.0 | 53.0 | 1.0 | 0.053571 | 3 / 56 |
4 | 0.0 | 0.0 | 0.0 | 0.0 | 72.0 | 0.000000 | 0 / 72 |
5 | 0.0 | 161.0 | 0.0 | 55.0 | 73.0 | 0.034602 | 10 / 289 |
Top-5 Hit Ratios:
k | hit_ratio | |
---|---|---|
0 | 1 | 0.965398 |
1 | 2 | 1.000000 |
2 | 3 | 1.000000 |
3 | 4 | 1.000000 |
4 | 5 | 1.000000 |
ModelMetricsMultinomial: gbm ** Reported on validation data. ** MSE: 0.08366695440378827 RMSE: 0.2892524060466711 LogLoss: 0.3345387893847581 Mean Per-Class Error: 0.2181318681318681 AUC: 0.7987492282997901 AUCPR: 0.7975377394397984 Multinomial AUC values:
type | first_class_domain | second_class_domain | auc | |
---|---|---|---|---|
0 | 3 vs Rest | 3 | None | 1.000000 |
1 | 4 vs Rest | 4 | None | 0.996154 |
2 | 5 vs Rest | 5 | None | 0.000000 |
3 | 6 vs Rest | 6 | None | 0.997592 |
4 | 8 vs Rest | 8 | None | 1.000000 |
5 | Macro OVR | None | None | 0.798749 |
6 | Weighted OVR | None | None | 0.997714 |
7 | Class 3 vs. 4 | 3 | 4 | 0.990385 |
8 | Class 3 vs. 5 | 3 | 5 | 0.500000 |
9 | Class 3 vs. 6 | 3 | 6 | 1.000000 |
10 | Class 3 vs. 8 | 3 | 8 | 1.000000 |
11 | Class 4 vs. 5 | 4 | 5 | 0.500000 |
12 | Class 4 vs. 6 | 4 | 6 | 0.995192 |
13 | Class 4 vs. 8 | 4 | 8 | 1.000000 |
14 | Class 5 vs. 6 | 5 | 6 | 0.500000 |
15 | Class 5 vs. 8 | 5 | 8 | 0.500000 |
16 | Class 6 vs. 8 | 6 | 8 | 0.998016 |
17 | Macro OVO | None | None | 0.798359 |
18 | Weighted OVO | None | None | 0.872818 |
Multinomial auc_pr values:
type | first_class_domain | second_class_domain | auc_pr | |
---|---|---|---|---|
0 | 3 vs Rest | 3 | None | 1.000000 |
1 | 4 vs Rest | 4 | None | 0.994834 |
2 | 5 vs Rest | 5 | None | 0.000000 |
3 | 6 vs Rest | 6 | None | 0.992854 |
4 | 8 vs Rest | 8 | None | 1.000000 |
5 | Macro OVR | None | None | 0.797538 |
6 | Weighted OVR | None | None | 0.995994 |
7 | Class 3 vs. 4 | 3 | 4 | 0.999817 |
8 | Class 3 vs. 5 | 3 | 5 | 0.500000 |
9 | Class 3 vs. 6 | 3 | 6 | 1.000000 |
10 | Class 3 vs. 8 | 3 | 8 | 1.000000 |
11 | Class 4 vs. 5 | 4 | 5 | 0.500000 |
12 | Class 4 vs. 6 | 4 | 6 | 0.996311 |
13 | Class 4 vs. 8 | 4 | 8 | 1.000000 |
14 | Class 5 vs. 6 | 5 | 6 | 0.500000 |
15 | Class 5 vs. 8 | 5 | 8 | 0.500000 |
16 | Class 6 vs. 8 | 6 | 8 | 0.997536 |
17 | Macro OVO | None | None | 0.799366 |
18 | Weighted OVO | None | None | 0.874012 |
Confusion Matrix: Row labels: Actual class; Column labels: Predicted class
3 | 4 | 5 | 6 | 8 | Error | Rate | |
---|---|---|---|---|---|---|---|
0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 1.000000 | 1 / 1 |
1 | 0.0 | 51.0 | 0.0 | 1.0 | 0.0 | 0.019231 | 1 / 52 |
2 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | NaN | 0 / 0 |
3 | 0.0 | 2.0 | 0.0 | 26.0 | 0.0 | 0.071429 | 2 / 28 |
4 | 0.0 | 0.0 | 0.0 | 0.0 | 36.0 | 0.000000 | 0 / 36 |
5 | 0.0 | 54.0 | 0.0 | 27.0 | 36.0 | 0.034188 | 4 / 117 |
Top-5 Hit Ratios:
k | hit_ratio | |
---|---|---|
0 | 1 | 0.965812 |
1 | 2 | 1.000000 |
2 | 3 | 1.000000 |
3 | 4 | 1.000000 |
4 | 5 | 1.000000 |
Scoring History:
timestamp | duration | number_of_trees | training_rmse | training_logloss | training_classification_error | training_auc | training_pr_auc | validation_rmse | validation_logloss | validation_classification_error | validation_auc | validation_pr_auc | ||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
0 | 2020-11-24 14:51:44 | 0.001 sec | 0.0 | 0.800000 | 1.609438 | 0.643599 | 0.500000 | 0.200000 | 0.800000 | 1.609438 | 0.615385 | 0.400000 | 0.200000 | |
1 | 2020-11-24 14:51:44 | 0.007 sec | 1.0 | 0.792456 | 1.572433 | 0.034602 | 0.996920 | 0.923202 | 0.792423 | 1.572271 | 0.034188 | 0.798394 | 0.797055 | |
2 | 2020-11-24 14:51:44 | 0.010 sec | 2.0 | 0.784944 | 1.536933 | 0.034602 | 0.998389 | 0.953458 | 0.784923 | 1.536834 | 0.034188 | 0.798834 | 0.798153 | |
3 | 2020-11-24 14:51:44 | 0.013 sec | 3.0 | 0.777463 | 1.502819 | 0.034602 | 0.998044 | 0.945566 | 0.777452 | 1.502770 | 0.034188 | 0.798834 | 0.798153 | |
4 | 2020-11-24 14:51:44 | 0.016 sec | 4.0 | 0.770014 | 1.470001 | 0.034602 | 0.998044 | 0.945566 | 0.769983 | 1.469861 | 0.034188 | 0.798834 | 0.798153 | |
5 | 2020-11-24 14:51:44 | 0.019 sec | 5.0 | 0.762600 | 1.438399 | 0.034602 | 0.998054 | 0.945574 | 0.762554 | 1.438200 | 0.034188 | 0.798834 | 0.798153 | |
6 | 2020-11-24 14:51:44 | 0.021 sec | 6.0 | 0.755219 | 1.407921 | 0.034602 | 0.998092 | 0.945606 | 0.755176 | 1.407739 | 0.034188 | 0.798834 | 0.798153 | |
7 | 2020-11-24 14:51:44 | 0.024 sec | 7.0 | 0.747874 | 1.378506 | 0.034602 | 0.998092 | 0.945606 | 0.747814 | 1.378260 | 0.034188 | 0.798834 | 0.798153 | |
8 | 2020-11-24 14:51:44 | 0.026 sec | 8.0 | 0.740564 | 1.350089 | 0.034602 | 0.998092 | 0.945606 | 0.740519 | 1.349903 | 0.034188 | 0.798834 | 0.798153 | |
9 | 2020-11-24 14:51:44 | 0.028 sec | 9.0 | 0.733293 | 1.322618 | 0.034602 | 0.998092 | 0.945606 | 0.733238 | 1.322396 | 0.034188 | 0.798834 | 0.798153 | |
10 | 2020-11-24 14:51:44 | 0.031 sec | 10.0 | 0.726058 | 1.296014 | 0.034602 | 0.998102 | 0.945614 | 0.726020 | 1.295866 | 0.034188 | 0.798834 | 0.798153 | |
11 | 2020-11-24 14:51:44 | 0.034 sec | 11.0 | 0.718861 | 1.270251 | 0.034602 | 0.998102 | 0.945614 | 0.718823 | 1.270106 | 0.034188 | 0.798834 | 0.798153 | |
12 | 2020-11-24 14:51:44 | 0.036 sec | 12.0 | 0.711704 | 1.245292 | 0.034602 | 0.998102 | 0.945614 | 0.711680 | 1.245194 | 0.034188 | 0.798834 | 0.798153 | |
13 | 2020-11-24 14:51:44 | 0.038 sec | 13.0 | 0.704588 | 1.221078 | 0.034602 | 0.998102 | 0.945614 | 0.704565 | 1.220990 | 0.034188 | 0.798834 | 0.798153 | |
14 | 2020-11-24 14:51:44 | 0.041 sec | 14.0 | 0.697512 | 1.197577 | 0.034602 | 0.999277 | 0.998774 | 0.697507 | 1.197554 | 0.034188 | 0.798834 | 0.798153 | |
15 | 2020-11-24 14:51:44 | 0.045 sec | 15.0 | 0.690478 | 1.174763 | 0.034602 | 0.999292 | 0.998837 | 0.690506 | 1.174846 | 0.034188 | 0.798834 | 0.798153 | |
16 | 2020-11-24 14:51:44 | 0.048 sec | 16.0 | 0.683489 | 1.152605 | 0.034602 | 0.999292 | 0.998837 | 0.683503 | 1.152644 | 0.034188 | 0.798834 | 0.798153 | |
17 | 2020-11-24 14:51:44 | 0.052 sec | 17.0 | 0.676542 | 1.131059 | 0.031142 | 0.999311 | 0.998856 | 0.676589 | 1.131201 | 0.034188 | 0.798834 | 0.798153 | |
18 | 2020-11-24 14:51:44 | 0.055 sec | 18.0 | 0.669640 | 1.110108 | 0.031142 | 0.999311 | 0.998856 | 0.669721 | 1.110354 | 0.034188 | 0.798834 | 0.798153 | |
19 | 2020-11-24 14:51:44 | 0.058 sec | 19.0 | 0.662783 | 1.089722 | 0.031142 | 0.999311 | 0.998856 | 0.662885 | 1.090031 | 0.034188 | 0.798834 | 0.798153 |
See the whole table with table.as_data_frame() Variable Importances:
variable | relative_importance | scaled_importance | percentage | |
---|---|---|---|---|
0 | displacement | 6259.065430 | 1.000000 | 0.979955 |
1 | power | 56.853493 | 0.009083 | 0.008901 |
2 | acceleration | 31.531071 | 0.005038 | 0.004937 |
3 | weight | 21.419443 | 0.003422 | 0.003354 |
4 | year | 18.227747 | 0.002912 | 0.002854 |
gbm.auc()
0.999511857555245
gbm.aucpr()
0.998951571914844
gbm.multinomial_auc_table()
Multinomial AUC values:
type | first_class_domain | second_class_domain | auc | |
---|---|---|---|---|
0 | 3 vs Rest | 3 | None | 1.000000 |
1 | 4 vs Rest | 4 | None | 0.998556 |
2 | 5 vs Rest | 5 | None | 1.000000 |
3 | 6 vs Rest | 6 | None | 0.999004 |
4 | 8 vs Rest | 8 | None | 1.000000 |
5 | Macro OVR | None | None | 0.999512 |
6 | Weighted OVR | None | None | 0.999032 |
7 | Class 3 vs. 4 | 3 | 4 | 0.994624 |
8 | Class 3 vs. 5 | 3 | 5 | 1.000000 |
9 | Class 3 vs. 6 | 3 | 6 | 1.000000 |
10 | Class 3 vs. 8 | 3 | 8 | 1.000000 |
11 | Class 4 vs. 5 | 4 | 5 | 0.975269 |
12 | Class 4 vs. 6 | 4 | 6 | 0.999482 |
13 | Class 4 vs. 8 | 4 | 8 | 1.000000 |
14 | Class 5 vs. 6 | 5 | 6 | 0.988095 |
15 | Class 5 vs. 8 | 5 | 8 | 1.000000 |
16 | Class 6 vs. 8 | 6 | 8 | 0.999752 |
17 | Macro OVO | None | None | 0.995722 |
18 | Weighted OVO | None | None | 0.995155 |
# early stopping
gbm = H2OGradientBoostingEstimator(distribution="multinomial",
ntrees=100,
max_depth=3,
learn_rate=0.01,
stopping_metric="AUCPR",
stopping_tolerance=0.01,
stopping_rounds=3,
auc_type="WEIGHTED_OVR")
gbm.train(x =features,
y =response,
training_frame =train,
validation_frame=valid)
gbm.scoring_history()
gbm Model Build progress: |███████████████████████████████████████████████| 100%
timestamp | duration | number_of_trees | training_rmse | training_logloss | training_classification_error | training_auc | training_pr_auc | validation_rmse | validation_logloss | validation_classification_error | validation_auc | validation_pr_auc | ||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
0 | 2020-11-24 14:51:44 | 0.003 sec | 0.0 | 0.800000 | 1.609438 | 0.643599 | 0.500000 | 0.387483 | 0.800000 | 1.609438 | 0.615385 | 0.500000 | 0.349551 | |
1 | 2020-11-24 14:51:44 | 0.016 sec | 1.0 | 0.792456 | 1.572433 | 0.034602 | 0.997761 | 0.993284 | 0.792423 | 1.572271 | 0.034188 | 0.996925 | 0.994890 | |
2 | 2020-11-24 14:51:44 | 0.020 sec | 2.0 | 0.784944 | 1.536933 | 0.034602 | 0.998140 | 0.995454 | 0.784923 | 1.536834 | 0.034188 | 0.997573 | 0.996420 | |
3 | 2020-11-24 14:51:44 | 0.024 sec | 3.0 | 0.777463 | 1.502819 | 0.034602 | 0.998134 | 0.995058 | 0.777452 | 1.502770 | 0.034188 | 0.997573 | 0.996420 | |
4 | 2020-11-24 14:51:44 | 0.026 sec | 4.0 | 0.770014 | 1.470001 | 0.034602 | 0.998134 | 0.995058 | 0.769983 | 1.469861 | 0.034188 | 0.997573 | 0.996420 | |
5 | 2020-11-24 14:51:44 | 0.029 sec | 5.0 | 0.762600 | 1.438399 | 0.034602 | 0.998160 | 0.995082 | 0.762554 | 1.438200 | 0.034188 | 0.997573 | 0.996420 | |
6 | 2020-11-24 14:51:44 | 0.031 sec | 6.0 | 0.755219 | 1.407921 | 0.034602 | 0.998264 | 0.995168 | 0.755176 | 1.407739 | 0.034188 | 0.997573 | 0.996420 |
# grid search
from h2o.grid.grid_search import H2OGridSearch
hyper_parameters = {'ntrees': [5, 10], 'max_depth': [10, 20]}
gs = H2OGridSearch(H2OGradientBoostingEstimator(distribution = "multinomial", auc_type="MACRO_OVR"), hyper_parameters)
gs.train(x=features, y=response, training_frame=train)
gs.auc(train=True)
gbm Grid Build progress: |████████████████████████████████████████████████| 100%
{'Grid_GBM_py_3_sid_b9cb_model_python_1606223609474_69_model_3': 0.9996445050710137, 'Grid_GBM_py_3_sid_b9cb_model_python_1606223609474_69_model_4': 0.9996445050710137, 'Grid_GBM_py_3_sid_b9cb_model_python_1606223609474_69_model_2': 0.9994134025192564, 'Grid_GBM_py_3_sid_b9cb_model_python_1606223609474_69_model_1': 0.9994134025192564}
# domain is too big
air = h2o.import_file(path=_locate("smalldata/airlines/AirlinesTrain.csv.zip"))
features = ["Origin", "Dest", "IsDepDelayed", "UniqueCarrier", "fMonth", "fDayofMonth", "fDayOfWeek"]
response = "Dest"
r = air[0].runif()
train = air[r < 0.8]
valid = air[r >= 0.8]
#Too many domains - AUC/PR AUC is not calculated
gbm = H2OGradientBoostingEstimator(distribution="multinomial",
ntrees=100,
max_depth=3,
learn_rate=0.01,
auc_type="MACRO_OVO")
gbm.train(x =features,
y =response,
training_frame =train,
validation_frame=valid)
gbm.show()
Parse progress: |█████████████████████████████████████████████████████████| 100% gbm Model Build progress: |███████████████████████████████████████████████| 100% Model Details ============= H2OGradientBoostingEstimator : Gradient Boosting Machine Model Key: GBM_model_python_1606223609474_70 Model Summary:
number_of_trees | number_of_internal_trees | model_size_in_bytes | min_depth | max_depth | mean_depth | min_leaves | max_leaves | mean_leaves | ||
---|---|---|---|---|---|---|---|---|---|---|
0 | 100.0 | 9800.0 | 2008108.0 | 1.0 | 3.0 | 2.99949 | 2.0 | 8.0 | 7.804898 |
ModelMetricsMultinomial: gbm ** Reported on train data. ** MSE: 0.430057090190663 RMSE: 0.6557873818477015 LogLoss: 1.3717578431630804 Mean Per-Class Error: 0.7578920698567744 AUC: NaN AUCPR: NaN Multinomial auc values: Table is not computed because it is disabled or due to domain size (maximum is 50 domains). Multinomial auc_pr values: Table is not computed because it is disabled or due to domain size (maximum is 50 domains). Confusion Matrix: Row labels: Actual class; Column labels: Predicted class
ABE | ABQ | ACY | ALB | ATL | AVP | BDL | BGM | BNA | BOS | ... | SNA | STL | SWF | SYR | TOL | TPA | TUS | UCA | Error | Rate | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 1.000000 | 102 / 102 |
1 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 1.000000 | 76 / 76 |
2 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 1.000000 | 15 / 15 |
3 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 1.000000 | 81 / 81 |
4 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 1.000000 | 18 / 18 |
5 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 1.000000 | 20 / 20 |
6 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 15.0 | 0.0 | 0.0 | 0.0 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.761905 | 48 / 63 |
7 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 10.0 | 8.0 | 0.0 | 0.0 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.794872 | 31 / 39 |
8 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 1.000000 | 41 / 41 |
9 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 114.0 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.673352 | 235 / 349 |
10 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 1.000000 | 30 / 30 |
11 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 36.0 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 8.0 | 0.859813 | 92 / 107 |
12 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | ... | 17.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 5.0 | 0.0 | 0.658046 | 229 / 348 |
13 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 9.0 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.591503 | 181 / 306 |
14 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 1.000000 | 21 / 21 |
15 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 1.000000 | 22 / 22 |
16 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 1.000000 | 1 / 1 |
17 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.000000 | 0 / 13 |
18 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 1.000000 | 40 / 40 |
19 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 4.0 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.421769 | 62 / 147 |
20 rows × 100 columns
See the whole table with table.as_data_frame() Top-10 Hit Ratios:
k | hit_ratio | |
---|---|---|
0 | 1 | 0.647989 |
1 | 2 | 0.772855 |
2 | 3 | 0.834294 |
3 | 4 | 0.872992 |
4 | 5 | 0.900168 |
5 | 6 | 0.920155 |
6 | 7 | 0.933615 |
7 | 8 | 0.943201 |
8 | 9 | 0.950798 |
9 | 10 | 0.956356 |
ModelMetricsMultinomial: gbm ** Reported on validation data. ** MSE: 0.45785658129894297 RMSE: 0.6766510040626135 LogLoss: 1.6072449485890636 Mean Per-Class Error: 0.794834751023698 AUC: NaN AUCPR: NaN Multinomial auc values: Table is not computed because it is disabled or due to domain size (maximum is 50 domains). Multinomial auc_pr values: Table is not computed because it is disabled or due to domain size (maximum is 50 domains). Confusion Matrix: Row labels: Actual class; Column labels: Predicted class
ABE | ABQ | ACY | ALB | ATL | AVP | BDL | BGM | BNA | BOS | ... | SNA | STL | SWF | SYR | TOL | TPA | TUS | UCA | Error | Rate | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 1.000000 | 28 / 28 |
1 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 1.000000 | 21 / 21 |
2 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 1.000000 | 4 / 4 |
3 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 1.000000 | 17 / 17 |
4 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 1.000000 | 5 / 5 |
5 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 1.000000 | 7 / 7 |
6 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 4.0 | 0.0 | 0.0 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 1.000000 | 18 / 18 |
7 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 3.0 | 0.0 | 0.0 | 0.0 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 1.000000 | 14 / 14 |
8 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 1.000000 | 8 / 8 |
9 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 27.0 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.712766 | 67 / 94 |
10 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 1.0 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 1.000000 | 7 / 7 |
11 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 7.0 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 4.0 | 0.950000 | 19 / 20 |
12 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | ... | 3.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.740260 | 57 / 77 |
13 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 3.0 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.712644 | 62 / 87 |
14 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 1.000000 | 3 / 3 |
15 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 1.000000 | 3 / 3 |
16 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 1.000000 | 2 / 2 |
17 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.000000 | 0 / 6 |
18 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 1.000000 | 12 / 12 |
19 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 1.0 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.444444 | 16 / 36 |
20 rows × 100 columns
See the whole table with table.as_data_frame() Top-10 Hit Ratios:
k | hit_ratio | |
---|---|---|
0 | 1 | 0.611065 |
1 | 2 | 0.737729 |
2 | 3 | 0.778910 |
3 | 4 | 0.814476 |
4 | 5 | 0.844218 |
5 | 6 | 0.864601 |
6 | 7 | 0.878744 |
7 | 8 | 0.891015 |
8 | 9 | 0.900999 |
9 | 10 | 0.909526 |
Scoring History:
timestamp | duration | number_of_trees | training_rmse | training_logloss | training_classification_error | training_auc | training_pr_auc | validation_rmse | validation_logloss | validation_classification_error | validation_auc | validation_pr_auc | ||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
0 | 2020-11-24 14:51:45 | 0.010 sec | 0.0 | 0.989796 | 4.584967 | 0.866007 | NaN | NaN | 0.989796 | 4.584967 | 0.867512 | NaN | NaN | |
1 | 2020-11-24 14:51:46 | 0.345 sec | 1.0 | 0.981584 | 4.064828 | 0.365268 | NaN | NaN | 0.981879 | 4.084868 | 0.386855 | NaN | NaN | |
2 | 2020-11-24 14:51:46 | 0.589 sec | 2.0 | 0.975317 | 3.822181 | 0.364095 | NaN | NaN | 0.975852 | 3.853578 | 0.387687 | NaN | NaN | |
3 | 2020-11-24 14:51:46 | 0.852 sec | 3.0 | 0.969404 | 3.645022 | 0.363840 | NaN | NaN | 0.970166 | 3.685062 | 0.387479 | NaN | NaN | |
4 | 2020-11-24 14:51:47 | 1.110 sec | 4.0 | 0.963683 | 3.503089 | 0.363432 | NaN | NaN | 0.964669 | 3.550449 | 0.387479 | NaN | NaN | |
5 | 2020-11-24 14:51:47 | 1.399 sec | 5.0 | 0.958093 | 3.383978 | 0.363636 | NaN | NaN | 0.959299 | 3.437732 | 0.388311 | NaN | NaN | |
6 | 2020-11-24 14:51:47 | 1.692 sec | 6.0 | 0.952597 | 3.280713 | 0.363636 | NaN | NaN | 0.954023 | 3.340154 | 0.388311 | NaN | NaN | |
7 | 2020-11-24 14:51:47 | 2.032 sec | 7.0 | 0.947184 | 3.189588 | 0.363636 | NaN | NaN | 0.948829 | 3.254264 | 0.388311 | NaN | NaN | |
8 | 2020-11-24 14:51:48 | 2.358 sec | 8.0 | 0.941846 | 3.107945 | 0.363636 | NaN | NaN | 0.943709 | 3.177402 | 0.388311 | NaN | NaN | |
9 | 2020-11-24 14:51:48 | 2.691 sec | 9.0 | 0.936570 | 3.033831 | 0.363738 | NaN | NaN | 0.938650 | 3.107704 | 0.387895 | NaN | NaN | |
10 | 2020-11-24 14:51:48 | 2.985 sec | 10.0 | 0.931338 | 2.965584 | 0.363687 | NaN | NaN | 0.933635 | 3.043692 | 0.387895 | NaN | NaN | |
11 | 2020-11-24 14:51:49 | 3.243 sec | 11.0 | 0.926156 | 2.902436 | 0.363687 | NaN | NaN | 0.928669 | 2.984427 | 0.387895 | NaN | NaN | |
12 | 2020-11-24 14:51:49 | 3.495 sec | 12.0 | 0.921028 | 2.843769 | 0.363534 | NaN | NaN | 0.923761 | 2.929714 | 0.388311 | NaN | NaN | |
13 | 2020-11-24 14:51:49 | 3.756 sec | 13.0 | 0.915957 | 2.789250 | 0.363687 | NaN | NaN | 0.918907 | 2.878823 | 0.387895 | NaN | NaN | |
14 | 2020-11-24 14:51:49 | 4.012 sec | 14.0 | 0.910934 | 2.738108 | 0.363126 | NaN | NaN | 0.914103 | 2.831229 | 0.388103 | NaN | NaN | |
15 | 2020-11-24 14:51:54 | 8.125 sec | 43.0 | 0.786695 | 1.924839 | 0.360424 | NaN | NaN | 0.796223 | 2.084962 | 0.389559 | NaN | NaN | |
16 | 2020-11-24 14:52:03 | 17.703 sec | 100.0 | 0.655787 | 1.371758 | 0.352011 | NaN | NaN | 0.676651 | 1.607245 | 0.388935 | NaN | NaN |
Variable Importances:
variable | relative_importance | scaled_importance | percentage | |
---|---|---|---|---|
0 | Origin | 156621.484375 | 1.000000 | 0.505359 |
1 | UniqueCarrier | 144643.531250 | 0.923523 | 0.466710 |
2 | fDayofMonth | 4113.584473 | 0.026264 | 0.013273 |
3 | fMonth | 3239.199219 | 0.020682 | 0.010452 |
4 | IsDepDelayed | 964.104858 | 0.006156 | 0.003111 |
5 | fDayOfWeek | 339.425415 | 0.002167 | 0.001095 |
gbm.multinomial_auc_table()
'Table is not computed because it is disabled or due to domain size (maximum is 50 domains).'