Add Pyodide support and CI jobs for Zarr

[agriyakhetarpal]

Description

This description is a stub and will be updated in due course.

TODO:

• Add unit tests and/or doctests in docstrings
• Add docstrings and API docs for any new/modified user-facing classes and functions
• New/modified features documented in docs/tutorial.rst
• Changes documented in docs/release.rst
• GitHub Actions have all passed
• Test coverage is 100% (Codecov passes)

Additional context

Here are a few xrefs to some other PRs that have been implemented for packages in the Scientific Python ecosystem:

• NumPy's Pyodide job: CI, BLD, TST: Re-enable Emscripten/Pyodide CI job for NumPy numpy/numpy#25894, by courtesy of @rgommers
• The same for PyWavelets (which is an optional dependency for scikit-image): Build and test PyWavelets Pyodide wheels in CI PyWavelets/pywt#701 and Upload dev wheels to Anaconda.org + revamp wheels publishing workflow PyWavelets/pywt#714
• For the pandas repository BLD, TST: Build and test Pyodide wheels for pandas in CI pandas-dev/pandas#57896
• Add out-of-tree Pyodide builds in CI for scikit-image scikit-image/scikit-image#7350
• and so on

The greater, long-term goal towards this change is to implement Sphinx-based interactive documentation via JupyterLite and WASM-powered, in-browser kernels, as referenced in Quansight-Labs/czi-scientific-python-mgmt#19, see also: Quansight-Labs/czi-scientific-python-mgmt#18. A pilot that can be readily tried out is available for the "API reference" pages under the PyWavelets documentation. This will be preceded by configuring a place to publish these WebAssembly/Emscripten wheels nightly or on a scheduled cadence for Zarr (which could be a third-party, PyPI-like index such as Anaconda.org) and then integrating Sphinx extensions such as jupyterlite-sphinx for hosted documentation.

[agriyakhetarpal]

Zarr v3 has crc32c and zstandard as dependencies, which were not present for v2. These packages contain compiled code and are not available in Pyodide yet (based on the logs from the workflow run). I shall try to add recipes for them Pyodide, but I am afraid I shall not have an answer for how long that can take.

[agriyakhetarpal]

Update: we added both crc32c and zstandard (please see the issues/PRs linked above). Pyodide 0.26.0 should be out very soon containing both packages plus with a bump to the Python version (to 3.12.1) and an ABI-breaking change (via Emscripten 3.1.58). I shall revisit this PR in a few days.

Next, when Zarr v3 (stable) comes out on PyPI in the next month or so, I shall bump the version/recipe for the in-tree Pyodide build as well. It would be great to have the out-of-tree build, i.e., this PR, ready for review/merging by then, and I will be committed to putting my best efforts in doing so – thanks!

[agriyakhetarpal]

The Pyodide 0.26.0 release is out, and we have been lucky to have got both crc32c and zstandard in time (but in any case, they would have got into a patch release anyway). Let me update this PR in a short moment.

[agriyakhetarpal]

The tests are now passing in https://github.com/zarr-developers/zarr-python/actions/runs/15322119791/job/43108023246?pr=1903 (except the extremely slow ones – also, they don't exhibit the same level of slowness at all if you run them locally); and I've added a release note highlighting Pyodide/WebAssembly support. Codecov is going to complain because it isn't aware of this WebAssembly environment, so I'm happy to either mock it to absurdly boost the coverage value or skip those lines with a # pragma: no cover comment. Thank you!

[hoodmane]

It would be nice to make coverage work with the webassembly tests too. Though we still haven't done it for Pyodide itself.

[agriyakhetarpal]

It would be nice to make coverage work with the webassembly tests too. Though we still haven't done it for Pyodide itself.

Yes, I'd assume merely collecting or operating on coverage data should already work, as coverage and pytest-cov themselves are able to generate reports, and we can combine them outside of Codecov to fix the missing values. It doesn't work for Codecov, though.

[ryanking13]

For Pyodide-wise, it looks good to me.

[agriyakhetarpal]

@ryanking13 – I've corrected the error message based on your suggestion, thanks!

@hoodmane, surfacing your conversation from above about solving the issue in webloop.py rather than just in the test code if it's possible: I've taken a more in-depth look at this, and I think it makes sense not to do much here. The idea is that shutdown_asyncgens closes tracked async generators (or the ones it's able to track with sys.set_asyncgen_hooks() – which allows the event loop to be notified of the generators that are created, through callbacks), by calling aclose() on them during shutdown. CPython uses _asyncgen_firstiter_hook() and _asyncgen_finalizer_hook() to manage them.

Next, what I understand here is that WebLoop's lifecycle management philosophy is fundamentally different from standard asyncio loops. We're not letting pyodide.webloop manage its lifecycle deliberately because it's delegated to the browser. At most, we could add a no-op async def shutdown_asyncgens() to webloop.py, but 1. it can't stop pytest-asyncio from raising this, and 2. I've realised that pytest-asyncio is raising only a RuntimeWarning here, which pytest has been filtering to an error based on Zarr's pytest configuration (!), so there was no need to add a fixture for this – there should be no harm in ignoring the warning. Please let me know if you had something else in mind; thanks!

[ryanking13]

At most, we could add a no-op async def shutdown_asyncgens() to webloop.py, but 1. it can't stop pytest-asyncio from raising this, and 2. I've realised that pytest-asyncio is raising only a RuntimeWarning here, which pytest has been filtering to an error based on Zarr's pytest configuration (!), so there was no need to add a fixture for this – there should be no harm in ignoring the warning. Please let me know if you had something else in mind; thanks!

Thanks for the investigation. If it only raises a runtime warning, I think it is okay to leave it as-is (without the fixture) in this PR.

Apart from that, adding no-op implementation of shutdown_asyncgens in Pyodide Webloop makes sense to me, as users (not just pytest-asyncio) who might call shutdown_asyncgens will face NotImplementedError. I think no-op implementation that does nothing is better than raising an error.

[hoodmane]

I think we could add an actual implementation of shutdown_asyncgens which is the same as the one for the normal event loop. We would keep track of the generators that are running, then iterate over that set and call close() on all of them.

[nenb]

test_set_orthogonal_selection_3d

Hey @agriyakhetarpal 👋

I had some questions related to your comments.

My background
I have a moderate amount of experience with the zarr-python codebase, but I am not an expert nor a maintainer. Treat my comments as possibly correct, but not definitely correct. 😄

My high-level understanding of zarr-python v3

• it has an event loop (from asyncio) at its core. This is useful for I/O operations e.g. read from object storage like S3.
• the event loop dispatches to a thread-pool for CPU-intensive calculations. This is useful for de/compression calculations like zstd.
• the event loop/async API has a synchronous wrapper for convenience. The synchronous wrapper runs all operations in a blocking manner in a separate thread to the asyncio event loop.

My questions on this PR

• Is the idea to try and let the user run zarr as normal in pyodide for the async API? If so, be aware that the de/compression logic ('codecs', see https://github.com/zarr-developers/zarr-python/tree/main/src/zarr/codecs) rely on asyncio.to_thread. This is because these are generally CPU-intensive calculations. I'm a little surprised that the CI is still passing for the current PR - I would have expected pyodide to throw an exception here.
• For this initial PR, it might also be useful experimenting with the concurrency value in the config. Its default is currently 10, but if everything will be run in a single thread for this initial PR, then my instinct is to set this to 1. But this is only speculation on my part!
• I know very little about the internals of pyodide. Is it possible (and if so, how much effort would it be) to create a web worker of some kind to handle the CPU-intensive calculations? My impression is that zarr v3 will run much more smoothly on pyodide if it was possible to have > 1 thread (it might require some tweaking of configuration defaults still, but fundamentally I think it has been designed with an assumption that there is at least 2 threads available)

Thanks for all the work here! Exciting to think that it may be possible to run zarr-python reading Zarr v3 in the browser in the not-so-distant future!