Integration with deep learning frameworks

[lesteve]

Collecting related thoughts from #210.

@stsievert in #210 (comment)

Especially with PyTorch. It certainly feels like there should be an integration with PyTorch because it has torch.distributed and torch.multiprocessing

@stsievert in #210 (comment)

Optimization:

• I see these algs (e.g., Hogwild) as exploiting dask's distributed architecture
• These will require a parameter server. Can we make this general and integrate with (for example) CuPy/Chainer and PyTorch?

@TomAugspurger in #210 (comment)

Dask-Tensorflow

• Review new datasets API, anything we should do there?

[TomAugspurger]

Thanks @lesteve!

I'm relatively unfamiliar with these libraries, especially with their distributed runtimes, where Dask may be most useful.

I'd be curious to hear from people who have experience here.

[mrocklin]

I'm also unfamiliar with using any modern deep learning library. I'd love to hear from people who do use them, want to use them in a distributed way, and have experienced some pain while trying to do so. (also cc'ing @bnaul @seibert)

My current understanding is that people do the following:

1. Build up a model. For this they use Tensorflow, Keras, PyTorch, etc.. I don't think Dask has any role to play here

2. Choose an optimizer, like AdaGrad. Dask might play a role here, or it might modify this optimizer, similar to the approach taken in Hovorod

   # Build model...
   loss = ...
   opt = tf.train.AdagradOptimizer(0.01 * hvd.size())
   
   # Add Horovod Distributed Optimizer
   opt = hvd.DistributedOptimizer(opt)

3. Load and preprocess data. Numpy/Pandas is probably active here, maybe Dask as well

4. Hand that data to the model/optimizer for training

[mrocklin]

Personally, I'm curious to see what workflows that might potentially use Dask and a deep learning framework would look like. That is something productive that people can do now that might help to focus the discussion.

[lesteve]

I guess this is not really integration per se so I opened a different issue #281 about the use case that some people around me are trying to tackle by combining dask and deep learning frameworks.

[mrocklin]

Questions like that are welcome. It's nice to identify issues that come up in practice, even if they are less research-y.

…

On Wed, Jul 4, 2018 at 4:53 AM, Loïc Estève ***@***.***> wrote: I guess this is not really integration per se so I opened a different issue #281 <#281> about the use case that some people are trying to tackle by combining dask and deep learning frameworks. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#268 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AASszFNAxWdWyykdSZ8n1IKnFUuKBPgvks5uDIKMgaJpZM4VAn0f> .

[arthurmensch]

Currently pytorch only documented way of doing Hogwild training is through using Process() commands and inheritance of models. See for instance https://github.com/pytorch/examples/blob/master/mnist_hogwild/main.py.

This is not compatible with the modern approach of using dask pools, or even multiprocess pools. I don't really see a simple way around that.

[corrado9999]

My professional experience in which I believe daks-ml would be great.

I work with remote sensing data. If you are thinking of Google Maps that is only part of the story. Going to the point, we have images of several gigabytes (tens of thousands of pixel, tens of channels) acquired tens times each year. Thanks to NASA and ESA (European Space Agency), we have 4 great pools of images available for free: MODIS, LANDSAT-8, Sentinel-1 and Sentinel-2.

Now, I am facing the issue to train some deep network on these data. They are all available on AWS (https://registry.opendata.aws/?search=satellite%20imagery), so it makes perfect sense to avoid the downloading and train the model directly on the cloud. As features we use bands from one (or more!) datasource, but also time must be taken into account in some way (this not something dask can help with, but it could somehow affect the design).

The training is usually supervised, the target being classification (water, urban, grassland, cropland, ...). We are also interested in 3 main kind of classification:

1. Pure pixel based: each pixel is assigned to one class, without knowing where it is located
2. Pixel based with neighborhood: this is where CNN come to play, because we want to exploit information coming from near pixel in order to estimate some spatial features
3. Object based: we aggregate pixels based on some vector data (known a priori)

Please, take into account that we have a lot of data, and that data storage usually costs more than processing. So, in some cases it may be preferable to perform data preprocessing/normalization at each batch, in some other cases we may prefer to cache them (instance based). For the same reason, along and across epochs we would like to minimize data movement, but yet mix batches coming from different images in most possible ways.

Sorry for the long post, I hope to have been clear enough about my use case.

[joemcglinchy]

I also use primarily remote sensing data, but my use case is more on the model inference stage. Suppose you have already trained a model to your satisfaction. This is typically done with smaller samples, which absolutely could be supported by dask using a windowed approach. Once the model is trained I would like to apply it to a remote sensing dataset which is typically quite large, think 10k's of rows and columns. Loading that full dataset into memory is often times problematic, so I think dask could help here as well.

[TomAugspurger]

Thanks Joseph. Based on your description, dask/dask-examples#35 sounds quite similar to your workflow. It'd be nice if we could develop that into a fully-formed example.

…

On Thu, Jan 17, 2019 at 12:29 PM Joseph McGlinchy ***@***.***> wrote: I also use primarily remote sensing data, but my use case is more on the model inference stage. Suppose you have already trained a model to your satisfaction. This is typically done with smaller samples, which absolutely could be supported by dask using a windowed approach. Once the model is trained I would like to apply it to a remote sensing dataset which is typically quite large, think 10k's of rows and columns. Loading that full dataset into memory is often times problematic, so I think dask could help here as well. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#268 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABQHIo2MQ5OQimUMuaXHhXBgdJs6UN4eks5vEMEBgaJpZM4VAn0f> .

[stsievert]

dask/distributed#2581 is on training a PyTorch model with Dask.

[AlbertDeFusco]

I've made progress using a Dask DataFrame with the Keras .fit_generator() method to mimic dask_ml.wrappers.Incremental for Scikit-Learn.

https://anaconda.org/defusco/keras-dask/notebook

I'm close to getting dask_ml.wrappers.ParallelPostfit, and keras.wrappers.scikit_learn.KerasClassifier working with sklearn piplines.

[TomAugspurger]

Thanks Albert, looks interesting. I think providing a DaskGenerator, or at least documentation on how to write one, would be very useful. A couple questions 1. With your DaskGenerator, if I do `gen = DaskGenerator(X, y); gen[0]; gen[1]`, then I *think* that you'll end up redoing a bunch of computation. Does that sound correct? In this example I think you would end up refitting the entire StandardScaler, splitting, etc. The solution is to persist the transformed data, either in memory on the cluster, perhaps as part of DaskGenerator.__init__, or on disk. And then the data passed to DaskGenerator would be loaded from disk. (LMK if I'm not making sense) 2. Are you using distributed at all? It doesn't look like it. I ask because I've never gotten Keras / Tensorflow to work properly in multiple processes.

…

On Wed, Jun 12, 2019 at 12:10 PM Albert DeFusco ***@***.***> wrote: I've made progress using a Dask DataFrame with the Keras .fit_generator() method to mimic Incremental for Scikit-Learn. https://anaconda.org/defusco/keras-dask/notebook I'm close to getting dask_ml.wrappers.ParallelPostfit, and keras.wrappers.scikit_learn.KerasClassifier() working with sklearn piplines. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#268?email_source=notifications&email_token=AAKAOIQVSNPGVSZXW5LRCNTP2EUXPA5CNFSM4FICPUP2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXRETQI#issuecomment-501369281>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAKAOIRICB3JDRFN7LA7353P2EUXPANCNFSM4FICPUPQ> .