Scikit-learn example for random forest with Joblib

[mrocklin]

It would be nice to see an example using Scikit-Learn's RandomForest parallelized with Dask using the joblib/dask connection. This is a common question, and so probably has high value.

My understanding is that this will depend on the 0.20 release of scikit-learn first, though I could be mistaken about this.

It would also be useful for this to be smoothly runnable on dask-examples

[TomAugspurger]

My understanding is that this will depend on the 0.20 release of scikit-learn first, though I could be mistaken about this.

Yes, that's correct.

[mrocklin]

Are there common enhancements beyond just RandomForest? For example does it make sense to do RandomForest within a GridSearchCV?

[TomAugspurger]

does it make sense to do RandomForest within a GridSearchCV?

Yep, certainly. That will introduce two levels of parallelism though, which may not be well-suited for dask-examples.

There are other ensemble methods, like http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingClassifier.html#sklearn.ensemble.GradientBoostingClassifier, that aren't parallelized at the estimator level.

[js3711]

Does this need sklearn v0.20 to work? For some reason I thought there was (maybe used to be) some working examples in the documentation for grid-search

[TomAugspurger]

RandomForest (and a few others) currently hardcode their joblib backends, so they would ignore the parallel_backend context. scikit-learn/scikit-learn#9486, which should be in 0.20, includes the changes to remove the hardcoding.

[js3711]

It looks like there's already an example of something similar in the dask-ml documentation:
http://dask-ml.readthedocs.io/en/latest/examples/joblib-distributed.html
Should this be tagged as dependent on scitkit learn v0.20.0?

Also, I took a quick look at sklearn dev branch. It looks like they are still using threading:
https://github.com/scikit-learn/scikit-learn/blob/f158e2dfe2af1b23ae3f9d86c598013b2c155c3f/sklearn/ensemble/forest.py#L329
a call to parallel_backend now override this in the new joblib?

[TomAugspurger]

To clarify, the dask.distributed backend works fine with scikit-learn 0.19 for most estimators. http://dask-ml.readthedocs.io/en/latest/examples/joblib-distributed.html should probably be updated to use something like GridSearchCV, similar to http://dask-ml.readthedocs.io/en/latest/joblib.html The PR scikit-learn/scikit-learn#9486 isn't merged yet. Once that's in, it'll use the new joblib API for prefers / requires shared memory.

…

On Thu, Jul 19, 2018 at 8:50 AM js3711 ***@***.***> wrote: It looks like there's already an example of something similar in the dask-ml documentation: http://dask-ml.readthedocs.io/en/latest/examples/joblib-distributed.html Should this be tagged as dependent on scitkit learn v0.20.0 Also, I took a quick look at sklearn dev branch. It looks like they are still using threading: https://github.com/scikit-learn/scikit-learn/blob/f158e2dfe2af1b23ae3f9d86c598013b2c155c3f/sklearn/ensemble/forest.py#L329 a call to parallel_backend now override this in the new joblib? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#231 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABQHIr1mmQEWOfRfBidauhcYDKSvrbIBks5uII6sgaJpZM4U5iDW> .

[js3711]

Got it. Thanks Tom, so the dask.distributed backend should work for sklearn estimators that do not explicitly mention the backend argument in Parallel.

I was wondering why RandomForest did not appear to light up the dask scheduler :)

[ogrisel]

FYI, I just merged 0.12.2 into the master branch of scikit-learn.