# Score and Predict Large Datasets¶

Sometimes you'll train on a smaller dataset that fits in memory, but need to predict or score for a much larger (possibly larger than memory) dataset. Perhaps your learning curve has leveled off, or you only have labels for a subset of the data.

In this situation, you can use ParallelPostFit to parallelize and distribute the scoring or prediction steps.

In [ ]:
from dask.distributed import Client, progress

# Scale up: connect to your own cluster with bmore resources
n_workers=1, memory_limit='2GB')
client

In [ ]:
import numpy as np
from sklearn.datasets import make_classification


We'll generate a small random dataset with scikit-learn.

In [ ]:
X_train, y_train = make_classification(
n_features=2, n_redundant=0, n_informative=2,
random_state=1, n_clusters_per_class=1, n_samples=1000)
X_train[:5]


And we'll clone that dataset many times with dask.array. X_large and y_large represent our larger than memory dataset.

In [ ]:
# Scale up: increase N, the number of times we replicate the data.
N = 100
X_large = da.concatenate([da.from_array(X_train, chunks=X_train.shape)
for _ in range(N)])
y_large = da.concatenate([da.from_array(y_train, chunks=y_train.shape)
for _ in range(N)])
X_large


Since our training dataset fits in memory, we can use a scikit-learn estimator as the actual estimator fit during traning. But we know that we'll want to predict for a large dataset, so we'll wrap the scikit-learn estimator with ParallelPostFit.

In [ ]:
from sklearn.linear_model import LogisticRegressionCV

In [ ]:
clf = ParallelPostFit(LogisticRegressionCV(cv=3), scoring="r2")


See the note in the dask-ml's documentation about when and why a scoring parameter is needed: https://ml.dask.org/modules/generated/dask_ml.wrappers.ParallelPostFit.html#dask_ml.wrappers.ParallelPostFit.

Now we'll call clf.fit. Dask-ML does nothing here, so this step can only use datasets that fit in memory.

In [ ]:
clf.fit(X_train, y_train)


Now that training is done, we'll turn to predicting for the full (larger than memory) dataset.

In [ ]:
y_pred = clf.predict(X_large)
y_pred


y_pred is Dask arary. Workers can write the predicted values to a shared file system, without ever having to collect the data on a single machine.

Or we can check the models score on the entire large dataset. The computation will be done in parallel, and no single machine will have to hold all the data.

In [ ]:
clf.score(X_large, y_large)