skpro introduction notebook¶

Set-up instructions: On binder, this should run out-of-the-box.

To run locally instead, ensure that skpro with basic dependency requirements is installed in your python environment.

skpro provides scikit-learn-like, scikit-base compatible interfaces to:

• tabular supervised regressors with probabilistic prediction modes - interval, quantile and distribution predictions
• performance metrics to evaluate probabilistic predictions, e.g., pinball loss, empirical coverage, CRPS
• reductions to turn non-probabilistic, scikit-learn regressors into probabilistic skpro regressors, such as bootstrap or conformal
• tools for building pipelines and composite machine learning models, including tuning via probabilistic performance metrics
• symbolic an lazy probability distributions with a value domain of pandas.DataFrame-s and a pandas-like interface

Section 1 provides an overview of common probabilistic supervised regression workflows supported by skpro.

Section 2 gives an more detailed introduction to prediction modes, performance metrics, and benchmarking tools.

Section 3 discusses advanced composition patterns, including various ways to add probabilistic capability to any sklearn regressor, pipeline building, tuning, ensembling.

Section 4 gives an introduction to how to write custom estimators compliant with the skpro interface.

1. Basic probabilistic supervised regression workflows ¶

skpro revolves around supervised probabilistic regressors:

• fit(X, y) with tabular features X, labels y, same rows, both pd.DataFrame
• predict_interval(X_test, coverage=0.90) for interval predictions of labels
• predict_quantiles(X_test, alpha=[0.05, 0.95]) for quantile predictions of labels
• predict_var(X_test) for variance predictions of labels
• predict(X_test) for mean predictions
• predict_proba(X_test) for distributional prediction

1.1 basic deployment workflow¶

skpro regressors are used via fit then predict_proba etc.

Same as sklearn regressors - X and y should be pd.DataFrame (numpy is also ok but not recommended)

1.1.1 distribution predictions¶

y_pred_proba is an skpro distribution - it has index and columns like pd.DataFrame

"we predict that true labels are distributed according to y_pred_proba"

(here: distribution marginal by row/columns)

skpro distribution objects are pandas-like

distribution objects have sample and methods such as mean, var:

more details on skpro distributions in the "distributions" tutorial!

1.1.2 interval predictions¶

interval prediction y_pred_interval is a pd.DataFrame:

• rows are the same as X_new
• columns indicate variables, nominal coverage, and bottom/upper bound

"we predict that value in row falls between bottom/upper with 90% chance"

multiple coverages can be passed:

predict_interval output spec:

pandas.DataFrame
Row index is as for X_new
Column has multi-index:
1st level = variable names from y in fit
2nd level = coverage fractions in coverage
3rd level = string "lower" or "upper"

Entries = interval prediction of lower/upper interval at nominal coverage in 2nd lvl, for var in 1st lvl, for data index in row

1.1.3 quantile predictions¶

quantile prediction y_pred_quantiles is a pd.DataFrame:

• rows are the same as X_new
• columns indicate variables, quantile points

"we predict the 5%, 50%, 95% quantile points for the row to be here"

multiple quantiles:

predict_quantiles output spec:

pandas.DataFrame
Row index is same as X_new
Column has multi-index:
1st level = variable names from y in fit
2nd level = quantile points in alpha

Entries = quantiles prediction at quantile point in 2nd lvl, for var in 1st lvl, for data index in row

1.1.4 mean and variance predictions¶

mean and variance predictions y_pred_mean, y_pred_var are pd.DataFrame-s:

• rows are the same as X_new
• columns are the same as X_new

entries are predictive mean and variance in row/column

this is the same as taking the distribution prediction and taking mean/variance

(for distribution objects that estimate these precisely)

1.2 simple evaluation workflow for probabilistic predictions¶

for simple evaluation:

1. split the data into train/test set
2. make predictions of either type for test features
3. compute metric on test set, comparing test predictions to hend out test labels

Note:

• metrics will compare tabular ground truth to probabilistic prediction
• the metric will needs to be of a compatible type, e.g., for proba predictions

how do we know that metric is of right type? Via scitype:y_pred tag

how do we find metrics for a prediction type?

extra note: quantile metrics can be applied to interval predictions as well

more details on metrics below

1.3 diagnostic visualisations¶

some useful diagnostic visualisations: variants of crossplots for probabilistic predictions

A. crossplot ground truth vs prediction intervals.

Works with both proba and interval predictions.

What to look for: intervals shouhld cut through the x = y line (green points)

B. crossplot residuals vs predictive standard deviation

Works with both proba and variance predictions.

What to look for: should be close to a line, high linear correlation

C. crossplot ground truth vs loss values

Loss and prediction type should agree.

What to look for: association between accuracy and ground truth value

Diagnostic of which values we can predict more accurately,

e.g., to inform modelling or identify unusual outliers

1.4 skpro objects - scikit-base interface, searching for regressors and metrics¶

• skpro objects - skbase interface points get_tags, get_params/set_params
• searching estimators and metrics via all_objects

1.4.1 primer on skpro object interface ¶

metrics and estimators are first-class citizens in skpro, with a scikit-base compatible interface

e.g., all have get_tags interface

the tag object_type indicates the type of object, e.g., metric or proba regressor

all objects also have the get_params/set_params interface known from scikit-learn

= reading or setting hyper-parameters

get_params returns dict {paramname: paramvalue}; set_params writes it

composite objects have the nested param interface, keys componentname__paramname

so reg_proba will have parameters coming from itself and either component:

further common interface points are get_config, set_config, and get_fitted_params (only fittable estimators)

1.4.2 searching for regressors and metrics ¶

as first-class citizens, all objects in skpro are indexed via the registry utility all_objects.

To find probabilistic supervised regressors, use all_objects with the type regressor_proba:

a full list can also be found in the online API reference.

for metrics, as seen above:

all tags can be printed by the all_tags utility:

filtering in search can be done with the filter_tags argument in all_objects, see docstring:

2. Prediction types, metrics, benchmarking ¶

This section gives more details on:

• different prediction types, including a methodological primer
• the API of metrics to compare probabilistic predictions to non-probabilistic actuals
• utilities for batch benchmarking of estimators and metrics

2.1 Probabilistic predictions - methodological primer ¶

readers familir with, or less interested in theory, may like to skip section 2.1

In supervised learning - probabilistic or not:

• we fit estimator to i.i.d samples $(X_1, Y_1), \dots, (X_N, Y_N) \sim (X_*, Y_*)$
• and want to predict $y$ given $x$ accurately, for $(x, y) \sim (X_*, Y_*)$

Let $y$ be the (true) value, for an observed feature $x$

(we consider $y$ a random variable)

More formal details & intuition:¶

let's consider the toy example again

• a "point prediction" is a prediction/estimate of the conditional expectation $\mathbb{E}[y|x]$.\

Intuition: "out of many repetitions/worlds, this value is the arithmetic average of all observations".

• a "variance prediction" is a prediction/estimate of the conditional expectation $Var[y|x]$.\

Intuition: "out of many repetitions/worlds, this value is the average squared distance of the observation to the perfect point prediction".

• a "quantile prediction", at quantile point $\alpha\in (0,1)$ is a prediction/estimate of the $\alpha$-quantile of $y'|y$, i.e., of $F^{-1}_{y|x}(\alpha)$, where $F^{-1}$ is the (generalized) inverse cdf = quantile function of the random variable y|x.\

Intuition: "out of many repetitions/worlds, a fraction of exactly $\alpha$ will have equal or smaller than this value."

• an "interval prediction" or "predictive interval" with (symmetric) coverage $c\in (0,1)$ is a prediction/estimate pair of lower bound $a$ and upper bound $b$ such that $P(a\le y \le b| x) = c$ and $P(y \gneq b| x) = P(y \lneq a| x) = (1 - c) /2$.\

Intuition: "out of many repetitions/worlds, a fraction of exactly $c$ will be contained in the interval $[a,b]$, and being above is equally likely as being below".

(similar - exercise left to the reader)

• a "distribution prediction" or "full probabilistic prediction" is a prediction/estimate of the distribution of $y|x$, e.g., "it's a normal distribution with mean 42 and variance 1".\

Intuition: exhaustive description of the generating mechanism of many repetitions/worlds.

note: the true distribution is unknown, and not accessible easily!

y_pred_proba is a distribution, but in general not equal to the true one!

that is, there are:

• true distribution y_pred_proba_true - unknown and unknowable but estimable
• y_pred_proba - our guess at y_pred_proba_true
• the actual data y_true is one y_pred_proba_true.sample()

• predict produces guess of y_pred_proba_true.mean()
• predict_var produces guess of y_pred_proba_true.var()
• predict_quantiles([0.05, 0.5, 0.95]) produces guess of y_pred_proba_true.quantiles([0.05, 0.5, 0.95])
• predict_proba produces guess of y_pred_proba_true

the guesses are algorithm specific, and some algorithms are more accurate than others, given data

2.2 probabilistic metrics - details ¶

General usage pattern same as for sklearn metrics:

1. get some actuals and predictions
2. specify the metric - similar to estimator specs
3. plug the actuals and predictions into metric to get metric values

but: need to use dedicated metric for probabilistic predictions

• ground truth: y_true samples
• prediction e.g., y_predict_proba, y_predict_interval
• so, match metric with type of prediction!
  • metric(y_true: 2D pd.DataFrame, y_pred: proba_prediction_type) -> float

Recall methods available for all probabilistic regressors:

• predict_interval produces interval predictions. Argument coverage (nominal interval coverage) must be provided.
• predict_quantiles produces quantile predictions. Argument alpha (quantile values) must be provided.
• predict_var produces variance predictions. Same args as predict.
• predict_proba produces full distributional predictions. Same args as predict.

let's produce some probabilistic predictions!

recall, all have their own output format:

we now need to apply a suitable metric, metric(y_test, y_pred)

IMPORTANT: sequence matters, y_test first; y_pred has very different type!

how do we find a metric that fits the prediction type?

answer: metrics are tagged

important tag: scitype:y_pred

• "pred_proba" - distributional, can applied to distributions, predict_proba output
• "pred_quantiles" - quantile forecast metric, can be applied to quantile predictions, interval predictions, distributional predictions
  • applicable to predict_quantiles, predict_interval, predict_proba outputs
• "pred_interval" - interval forecast metric, can be applied to interval predictions, distributional predictions
  • applicable to predict_interval, predict_proba outputs

listing metrics with the tag, filtering for probabilistic tags:

(let's try to find a quantile prediction metric!)

PinballLoss is a quantile forecast metric:

... this is by default an average (grand average, float)

• averages over samples in y_pred / y_test (rows)
• averages over variables (columns)
• average over alpha values, quantile points

what if we don't want these averages?

• variable (column) averaging is controlled by the multioutput arg.
  • "raw_values" prevents averaging, "uniform_average" computes arithmetic mean.
• quantile points (alpha) or coverage (coverage) is controlled by score_average arg
• evaluation by row via the evaluate_by_index method
  • can be useful for diagnostics or statistical tests

Caveat: not every metric is an average over time points, e.g., RMSE

In this case, evaluate_by_index computes jackknife pseudo-samples

(for mean statistics, jackknife pseudo-samples are equal to individual samples)

2.3 Benchmark evaluation of probabilistic regressors ¶

for quick evaluation and benchmarking,

the benchmarking.evaluate utility can be used:

3. Advanced composition patterns ¶

we introduce a number of composition patterns available in skpro:

• reducer-wrappers that turn sklearn regressors into probabilistic ones
• pipelines of sklearn transformers with skpro regressors
• tuning skpro probabilistic regressors via grid/random search, minimizing a probabilistic metric
• ensembling multiple skpro probabilistic regressors

data used in this section:

evaluation metric used in this section:

3.1 Reducers to turn sklearn regressors probabilistic ¶

there are many common algorithms that turn a non-probabilistic tabular regressor probabilistic

formally, this is a type of "reduction" - of probabilistic supervised tabular to non-probabilistic supervised tabular

Examples:

• predicting variance equal to training residual variance - ResidualDouble with standard settings
  • or other unconditional distribution estimate for residuals
• "squaring the residual" two-step prediction - ResidualDouble
• boostrap prediction intervals - BootstrapRegressor
• conformal prediction intervals - contributions appreciated :-)
• natural gradient boosting aka NGBoost - contributions appreciated :-)

3.1.1 constant variance prediction ¶

3.1.2 two-step residual prediction ¶

3.1.3 bootstrap prediction intervals ¶

3.2 Pipelines of skpro regressor and sklearn transformers ¶

skpro regressors can be pipelined with sklearn transformers, using the skpro pipeline.

This ensure presence of predict_proba etc in the pipeline object.

The syntax is exactly the same as for sklearn's pipeline.

the pipeline provides the familiar nested get_params, set_params interface:

nested parameters are keyed componentname__parametername

pipelines can also be created via simple lists of estimators,

in this case names are generated automatically:

3.3 Tuning of skpro regressors via grid and random search ¶

skpro provides grid and random search tuners to tune arbitrary probabilistic regressors,

using probabilistic metrics. Besides this, they function as the sklearn tuners do.

the grid search tuner behaves like any skpro probabilistic regressor:

random search is similar, except that instead of a grid a parameter sampler should be specified:

3.4 Bagging/mixture ensemble of probabilistic regressors ¶

Classical bagging does the following, for a wrapped estimator:

In fit:

1. subsample rows and/or columns of X, y to X_subs, y_subs
2. fit clone of wrapped estimator to X_subs, y_subs
3. Repeat 1-2 n_estimators times, store that many fitted clones.

In predict, for X_test:

1. for all fitted clones, obtain predictions on X_test - these are distributions
2. return the uniform mixture of these distributions, per test sample

4. Extension guide - implementing your own probabilistic regressor ¶

skpro is meant to be easily extensible, for direct contribution to skpro as well as for local/private extension with custom methods.

To get started:

• Follow the "implementing estimator" developer guide
• Use the probabilistic regressor template to get started

1. Read through the probabilistic regression extension template - this is a python file with todo blocks that mark the places in which changes need to be added.
2. Copy the proba regressor extension template to a local folder in your own repository (local/private extension), or to a suitable location in your clone of the skpro or affiliated repository (if contributed extension), inside skpro.regression; rename the file and update the file docstring appropriately.
3. Address the "todo" parts. Usually, this means: changing the name of the class, setting the tag values, specifying hyper-parameters, filling in __init__, _fit, and at least one of the probabilistic prediction methods, preferably _predict_proba (for details see the extension template). You can add private methods as long as they do not override the default public interface. For more details, see the extension template.
4. To test your estimator manually: import your estimator and run it in the worfklows in Section 1; then use it in the compositors in Section 3.
5. To test your estimator automatically: call skpro.utils.check_estimator on your estimator. You can call this on a class or object instance. Ensure you have specified test parameters in the get_test_params method, according to the extension template.

In case of direct contribution to skpro or one of its affiliated packages, additionally:

• Add yourself as an author to the code, and to the CODEOWNERS for the new estimator file(s).
• Create a pull request that contains only the new estimators (and their inheritance tree, if it's not just one class), as well as the automated tests as described above.
• In the pull request, describe the estimator and optimally provide a publication or other technical reference for the strategy it implements.
• Before making the pull request, ensure that you have all necessary permissions to contribute the code to a permissive license (BSD-3) open source project.

5. Summary¶

• skpro is a unified interface toolbox for probabilistic supervised regression, that is, for prediction intervals, quantiles, fully distributional predictions, in a tabular regression setting. The interface is fully interoperable with scikit-learn and scikit-base interface specifications.

• skpro comes with rich composition functionality that allows to build complex pipelines easily, and connect easily with other parts of the open source ecosystem, such as scikit-learn and individual algorithm libraries.

• skpro is easy to extend, and comes with user friendly tools to facilitate implementing and testing your own probabilistic regressors and composition principles.

Credits:¶

noteook creation: fkiraly

skpro: https://github.com/sktime/skpro/blob/main/CONTRIBUTORS.md