Example of task-graph speed-up?

[amueller]

This example shows how to use the dask gridsearch:
http://dask-ml.readthedocs.io/en/latest/examples/hyperparameter-search.html

but it only provides very modest speedups. Do you know why that is? I had seen experiments where the time using dask was just a fraction of the time with the vanilla version.

[amueller]

Oh in particular having an example that compares against sklearn with caching would be interesting.

[TomAugspurger]

Hmm I had a branch somewhere where I did some timings. Let me see if I can turn it up.

In the meantime, compared to scikit-learn grid search memory=None, the dask version will perform better when

1. There are relatively few hyper-parameters being searched at the start of the pipeline, and relatively more being searched towards the end
2. The fit calls are expensive
3. The transform calls are expensive.

I need to time it, but compared to scikit-learn's with memory=cachedir,

---

Let's stack the deck in favor of dask's version :) We'll write a pipeline where

1. The intermediate stage is very expensive (2 seconds to fit, 2 seconds to transform)
2. The second (final) stage is instant
3. Search across 3 parameters for the second stage

import time

import dask_ml.model_selection
import sklearn.model_selection

from sklearn.pipeline import make_pipeline
from sklearn.datasets import make_blobs
from sklearn.decomposition import PCA
from sklearn.base import BaseEstimator, TransformerMixin

n_samples = 100_000
n_features = 200

X, y = make_blobs(n_samples=n_samples, n_features=n_features)

class DummyTransformer(TransformerMixin, BaseEstimator):
    def __init__(self, fit_sleep=2, time_sleep=2):
        self.fit_sleep = fit_sleep
        self.time_sleep = time_sleep
        
    def fit(self, X, y=None):
        time.sleep(self.fit_sleep)
        return self
    
    def transform(self, X, y=None):
        time.sleep(self.time_sleep)
        return X

    
class NoOp(BaseEstimator):
    def __init__(self, parameter=0):
        self.parameter = parameter
        
    def fit(self, X, y=None):
        return self
    
    def score(self, X, y=None):
        return 0

Timings

Scikit-learn, no caching

%%time
>>> pipe = make_pipeline(DummyTransformer(), NoOp())
>>> gs = sklearn.model_selection.GridSearchCV(pipe, param_grid, verbose=2)
>>> gs.fit(X, y)
CPU times: user 366 ms, sys: 266 ms, total: 631 ms
Wall time: 1min 16s

Scikit-learn with caching

%%time
>>> pipe = make_pipeline(DummyTransformer(), NoOp(), memory='/tmp/cache')
>>> gs = sklearn.model_selection.GridSearchCV(pipe, param_grid)
>>> gs.fit(X, y)
CPU times: user 3.57 s, sys: 942 ms, total: 4.51 s
Wall time: 57.2 s

Dask, no caching

%%time
>>> pipe = make_pipeline(DummyTransformer(), NoOp())
>>> gs = dask_ml.model_selection.GridSearchCV(pipe, param_grid)
>>> gs.fit(X, y)
CPU times: user 457 ms, sys: 276 ms, total: 733 ms
Wall time: 16.3 s

Dask, caching

%%time
>>> pipe = make_pipeline(DummyTransformer(), NoOp(), memory='/tmp/cache')
>>> gs = dask_ml.model_selection.GridSearchCV(pipe, param_grid)
>>> gs.fit(X, y)
CPU times: user 747 ms, sys: 459 ms, total: 1.21 s
Wall time: 12.8 s

Results:

version	caching	time (s)
sklearn	no	76
sklearn	yes	57
dask_ml	no	16
dask_ml	yes	13

I'm running a more realistic example now, and will add that to the docs.

[TomAugspurger]

Sorry, I think those numbers are bogus since sckit-learn's default n_jobs=1, while dask's is -1. I'll update later today.

[amueller]

Did you get a chance to update this? Also: didn't realize you could do the details thing..

[TomAugspurger]

Just reran it with n_jobs=-1 for sklearn.model_selection.GridSearchCV

https://gist.github.com/TomAugspurger/9b7a2a7504c5f894b830349ef9059639

The grid search is over 2 * multiprocessing.cpu_count() items (16 in my case).

version	caching	time (s)
sklearn	no	57
sklearn	yes	33
dask_ml	no	33
dask_ml	yes	PicklingError

Not sure why the last one failed. I'll see if I can make a smaller example and make an issue.

It's interesting to see that scikit-learn w/ caching is the same as dask-ml's version. I suspect this is partly because the dataset is so small. Dumping X and y to disk with joblib takes ~300ms. I wonder how it would compare if data serialization took relatively long compared to compute.

That gist also includes a more realistic example with

pipe = make_pipeline(
    QuantileTransformer(),
    PCA(),
    RandomForestClassifier(),
)

param_grid = {
    'randomforestclassifier__max_features': [0.1, 0.5, 0.75, 1.0],
    'randomforestclassifier__max_depth': [None, 2, 10],
}

scikit-learn

CPU times: user 57 s, sys: 5.33 s, total: 1min 2s
Wall time: 4min 59s

dask-ml

CPU times: user 17min 5s, sys: 34.5 s, total: 17min 40s
Wall time: 3min 8s

[amueller]

Sweet, thanks! How come the CPU time for dask-ml is so much higher?

[TomAugspurger]

I think since I used threads for the dask version (client = Client(processes=False)), and I'm guessing the default for scikit-learn is to use processes? %%time must report the CPU time used just by the parent, and not the child proccesses.

CPU times: user 23.8 s, sys: 16.2 s, total: 40 s
Wall time: 4min 39s

So much closer to the scikit-learn version (4:59).

I see that I forgot memory for the realistic example. Re-running and updating the gist.

[stsievert]

I'm seeing similar times to #141 (comment) in dask/dask-ml-benchmarks#2, with a different pipeline:

function	time
sklearn GridSearchCV	~74s
dask-ml GridSearchCV	~55s