This notebook explores different metrics that can be used for evaluating various machine learning models. Here we use data obtained from functions provided by sklearn.
import numpy as np
import cuml
import sklearn
from cuml.cluster import KMeans
from cuml.ensemble import RandomForestRegressor as curfr
from cuml.ensemble import RandomForestClassifier as curfc
from cuml.manifold.umap import UMAP as cuUMAP
from sklearn import manifold
from sklearn.datasets import make_classification, make_regression
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.datasets.samples_generator import make_blobs
n_samples = 2**10
n_features = 100
n_info = 70
centers = round(n_samples*0.4)
X_blobs, y_blobs = make_blobs(n_samples=n_samples, centers=centers,
n_features=n_features)
X_class, y_class = make_classification(n_samples=n_samples, n_features=n_features,
n_clusters_per_class=1, n_informative=n_info,
random_state=123, n_classes=2)
X_class = X_class.astype(np.float32)
y_class = y_class.astype(np.int32)
X_class_train, X_class_test, y_class_train, y_class_test = train_test_split(X_class,
y_class, train_size=0.8,
random_state=10)
X_reg, y_reg = make_regression(n_samples=n_samples, n_features=n_features,
n_informative=n_info, random_state=123)
X_reg = X_reg.astype(np.float32)
y_reg = y_reg.astype(np.float32)
X_reg_train, X_reg_test, y_reg_train, y_reg_test = train_test_split(X_reg,
y_reg, train_size=0.8,
random_state=10)
It is a measure of the extent to which the local structure is retained in the embedding of the model. Therefore, if a sample predicted by the model lied within the unexpected region of the nearest neighbors, then those samples would be penalized. For more information on the trustworthiness metric please refer to: https://scikit-learn.org/dev/modules/generated/sklearn.manifold.t_sne.trustworthiness.html
the documentation for cuML's implementation of the trustworthiness metric is: https://rapidsai.github.io/projects/cuml/en/stable/api.html#cuml.metrics.trustworthiness.trustworthiness
X_embedded = cuUMAP(n_neighbors=10).fit_transform(X_blobs)
X_embedded = X_embedded.astype(np.float32)
cu_score = cuml.metrics.trustworthiness(X_blobs, X_embedded)
sk_score = sklearn.manifold.trustworthiness(X_blobs, X_embedded)
print(" cuml's trustworthiness score : ", cu_score)
print(" sklearn's trustworthiness score : ", sk_score)
This is a metrics which is used to measure the similarity between two data clusters, and it is adjusted to take into consideration the chance grouping of elements. For more information on Adjusted random index please refer to: https://en.wikipedia.org/wiki/Rand_index
cuml_kmeans = KMeans(n_clusters=10)
X_blobs = StandardScaler().fit_transform(X_blobs)
cu_y_pred = cuml_kmeans.fit_predict(X_blobs).to_array()
cu_adjusted_rand_index = cuml.metrics.cluster.adjusted_rand_score(y_blobs, cu_y_pred)
sk_adjusted_rand_index = sklearn.metrics.cluster.adjusted_rand_score(y_blobs, cu_y_pred)
print(" cuml's adjusted random index score : ", cu_adjusted_rand_index)
print(" sklearn's adjusted random index score : ", sk_adjusted_rand_index)
R^2 score is also known as the coefficient of determination. It is used as a metric for scoring regression models. It scores the output of the model based on the proportion of total variation of the model. For more information on the R^2 score metrics please refer to: https://en.wikipedia.org/wiki/Coefficient_of_determination
For more information on cuML's implementation of the r2 score metrics please refer to : https://rapidsai.github.io/projects/cuml/en/stable/api.html#cuml.metrics.regression.r2_score
cuml_reg_model = curfr(max_features=1.0, n_bins=8, max_depth=10,
split_algo=0, min_rows_per_node=2,
n_estimators=30)
cuml_reg_model.fit(X_reg_train,y_reg_train)
cu_preds = cuml_reg_model.predict(X_reg_test,y_reg_test)
cu_r2 = cuml.metrics.r2_score(y_reg_test, cu_preds)
sk_r2 = sklearn.metrics.r2_score(y_reg_test, cu_preds)
print("cuml's r2 score : ", cu_r2)
print("sklearn's r2 score : ", sk_r2)
Accuracy score is the ratio of correct predictions to the total number of predictions. It is used to measure the performance of classification models. For more information on the accuracy score metric please refer to: https://en.wikipedia.org/wiki/Accuracy_and_precision
For more information on cuML's implementation of accuracy score metrics please refer to: https://rapidsai.github.io/projects/cuml/en/0.10.0/api.html#cuml.metrics.accuracy.accuracy_score
cuml_class_model = curfc(max_features=1.0, n_bins=8, max_depth=10,
split_algo=0, min_rows_per_node=2,
n_estimators=30)
cuml_class_model.fit(X_class_train,y_class_train)
cu_preds = cuml_class_model.predict(X_class_test,y_class_test)
cu_accuracy = cuml.metrics.accuracy_score(y_class_test, cu_preds)
sk_accuracy = sklearn.metrics.accuracy_score(y_class_test, cu_preds)
print("cuml's accuracy score : ", cu_accuracy)
print("sklearn's accuracy score : ", sk_accuracy)