Indicating (technical) rendering hints for HDR content to viewers

[cmahnke]

Abstract

This proposal is about indicating technical rendering hints (for HDR images in the UltraHDR format) to Image API Clients.

Background

As a proof of concept I've created a static Image API endpoint delivering HDR enable content, you can read more about it and have a look in my blog.

Since it's only a PoC it shows a 70s wallpaper, since it's from my domain, there are other use cases for HDR scientific images, like:

• Image analysis
• Fluorescence photography

The tiles aren't just JPEGs, but UltraHDR images. This format allows to add a so called gainmap to the image. To be backwards compatible the format is still JPEG with an embedded image, as specified in MPO.

Keep in mind that you'll only see a SDR image, if the requirements (see below) aren't meet. This is actually a feature since it allows transparent fallback if those aren't provided (this is also a reason for this proposal: a Viewer needs to know whether it should hinting those requirements).

Displaying UltraHDR images needs to have the following requirements to be fulfilled:

• The device needs to have a HDR capable screen.
• The browser need to have HDR support - currently only recent Chromium-based browsers

Currently the combination of a Apple Silicon notebook or desktop and Google Chrome is the most likely working combination. Windows devices usually need to have HDR enabled explicitly. Have a look at the page created by photographer HDR evangelist Greg Benz.

Since the HDR support of the browser (currently) needs a specific initialisation of the viewer, some sort of indicator / flag is needed to:

• Init the client element (canvas)
• Check the requirements and indicate missing support to the user, maybe giving hints on browser and display settings.
• Expectation management of user experience

Possible solutions

Current implementation: Using supports of Image API 2.0

Since the current implementation is using the Python module iiif with a custom tiler backend the example uses the Image API 2.0. This allows to use supports with the URI https://christianmahnke.de/iiif-hdr/.

"supports": [
  "https://christianmahnke.de/iiif-hdr/"
]

Image API 3.0

In Image API 3.0 the best field is certainly extraFeatures.

Summary

• HDR on the web is already there - support will broaden over the next years
• Implementors might want to use this possibility to present either "spiced up" versions of their contents or the "real" (OpenEXR comes to mind) images.
• Currently it's not possible to hint the requirements for display devices in a standardised way.

Further questions / discussion

• Is there enough interest to define an extension?
• Since every UltraHDR image is a valid JPEG the example above didn't define contents as a format (or extraFormats), this is of cause open for discussion.
• Since every UltraHDR image is (technically, not in a viewer) a MPO is a JPEG - might it be worthwhile to consider adding support for both formats?
• Might it be worthwhile to abstract from the format and think about general HDR indication (in the Presentation API) which could also be used for 3D objects?
• Might it be worthwhile to abstract from the format and think about general indication of colorspace intents?

[azaroth42]

Thanks for such a well researched, well explained and well implemented suggestion!

Understanding that it's backwards compatible with jpg, and indeed doesn't have a separate media type or file extension (which I think is an odd choice, but not ours to make), I feel that in the IIIF view of the world, it's a different quality.

My rationale:

• extraFeatures are not features of the image delivered, but of the service that delivers the images. Hence the features are things like rotationArbitrary and sizeByW ... functionality that the server has, independent of the image formats.
• Quality is our abstraction for "colorspace" -- there's currently color (from which we might understand SDR color), gray (only black, grays, and white) and bitonal (only black and white). I would propose a ultrahdr quality.
• Format is about the set of encoding of the bytes to be delivered -- the encoding of the abstract image content having done the region, size and rotation and colorspace on the abstraction. The way to encode the ultrahdr improved colorspace is as a JPG, so the format remains as expected by implementers.
• Assuming there are other extensions to the base JPG structure, perhaps in competition with UltraHDR, a client wanting one extension rather than another would need a way to request UltraHDR vs RobsCrazyJpgThing. In the future an "upscaling" function might be possible (?) to interpolate the gainmap from a regular SDR image, and then it would be even more important to be able to explicitly request the original, or the enhanced version. However both are in the .jpg format, leaving quality as the parameter in the URL and the step in the processing for the HDR (or other) embedding.

If the above is accurate and convincing, it would mean

{
  "extraQualities": ["ultrahdr"]
}

Secondly, in thinking about the viewer requirements, it seems like this would need to be reflected into the Presentation API somehow, so that the viewer knows to do the UltraHDR tests and setup. I think this is fine without change to the API, as the functionality is tied to the image, and the annotation can embed information from the image service's info.json. Thus the above would be part of the ImageService block in the Manifest.
(e.g. per https://iiif.io/api/presentation/3.0/#56-annotation )

Thoughts?

[robcast]

I like @azaroth42 's idea of treating HDR as a quality rather than a feature.

I would really like to extend this to colorspaces outside of sRGB, as @cmahnke also hinted to. I recently implemented support for bigger colorspaces in digilib where it is currently only accessible through the native API: https://dh-tech.github.io/blog/2023/01/16/java-color-profiles/ (also viewers seem to have even less control about how the browser and the OS handle this)

I also don't know if "more-than-srgb" colorspace would be a separate quality from "ultrahdr" and what to do about about combinations of both...

[cmahnke]

@azaroth42: My rationale to use supports was thinking the other way around: The current approach can be considered to be "server push": The client requests JPEG but get's "more" by default. This way there is no need for additional content negotiation steps (and thus end user interaction). This has been a nuisance of mine when I've "tinkered" with qualities in the past.
But I don't have a strong opinion on this, since the reasons you gave are convincing.

Another remark would be that ultrahdr might be the wrong name, since other formats also support HDR and UltraHDR itself is to certain degree a marketing name (by Google).
Technically related names are Gain Map by Adobe or Apple HDR. Although sometimes other container formats are used, such as HEIF. But as almost always the official "name" or better identifier is rather hard to remember: ISO/CD 21496-1.

On the other end there are HDR initiatives for other formats, like PNG as well. Another, although made up, example would be TIFF with high bit depth (in Rec. 2020).

To be able to combine format and the proposed quality it might be only hdr or something similar.

As to the question regarding "upscaling" or how I called it "spicing images up": This is definitely possible, either programmatically or via a UltraHDR capable graphic program.

[ruven]

The gain map approach to SDR <-> HDR conversion looks really interesting. I'll look into adding support for this in IIPImage, especially as there will presumably be published specs for including gain maps into other formats once the ISO standardization process for gain maps has completed. Adding support will be non-trivial, though, as when you modify the image through resizing or rotation etc, you also need to modify the gain map image (which doesn't necessarily have the same resolution) in the same way.

For IIIF, hdr as a quality rather than a format makes sense as a number of formats can support both SDR and HDR either through gain maps (JPEG) or natively (AVIF, TIFF, EXR etc). However, there are some ambiguities as HDR is not just about the color space, but also about bit depth.

So, if we introduce a hdr quality, if the user does not specify hdr, by default you should presumably only get back an SDR version even if your source image is HDR. So, for JPEG, for example, this would be a standard JPEG (with no gain map as these take up quite a bit of space) and, say, for AVIF an 8 bit SDR version.

If you now request hdr quality: for JPEG output, the server has only one option, which is to include a gain map. However, for AVIF and other higher bit-depth formats, the server has, in theory, 2 options:

• an 8 bit SDR AVIF + gain map
• or a 10 bit or higher true HDR image.

I presume the user would prefer the latter if requesting an AVIF image. Perhaps, in practice, it will depend on what the source image consists of. If it's natively in 8 bit + gain map form, it will be easier for the server to output in the same form. Whereas if the source image is in native HDR form, the server will output to native HDR if the format supports this, or will have to transcode to 8 bit + gain map if it doesn't.

This raises the issue of bit depths in general, though. This is a separate discussion, but if the IIIF quality is concerned with color space, maybe there also needs to be a way to also specify output bit depth. JPEG is 8 bit only, but other formats that have native web browser support such as WebP, AVIF and PNG can all handle higher bit depths. Currently, there's no way to request, say, a 10 bit AVIF or 16bit PNG through IIIF. Perhaps this could be enabled by allowing avif10 or png16 as output formats? Or by including this somehow in the quality field: hdr10 or color16 (or with a separator color-16)? The native bit depth is also not currently included in the Image API info.json metadata, so it might be useful to add something like a depth field to this.

In fact, on the image server side, I can see the use of gain maps as being something useful for improving the handling of high bit-rate images in general. IIPImage, for example, simply scales 16 or 32 bit source images down linearly to 8 bits per channel so that they can be output as JPEG. If the image included a gain map (which can also be used to go from HDR to SDR), this would allow a more sophisticated tone-mapping to be done when going down to 8 bits.

I would really like to extend this to colorspaces outside of sRGB

The approach IIPImage takes here is simply to avoid unnecessary color space conversion: if the source image has a DCI-P3 ICC profile, just use the raw pixel values and copy the profile across (sure, ideally all image processing should be carried out in linear space, but this adds a lot of overhead). All current web browsers nowadays fully support ICC profiles, so images should be rendered correctly (in theory!).

If, however, we want IIIF servers to perform color space conversion, then, yes, we'll need a way specify this as a quality and we'll need a syntax to specify multiple qualities. Having said that, the only use-cases I can see for this would be if you want the server to downgrade a wider-gamut image to sRGB for a device that doesn't support ICC profiles. Or if your native image is in something that is not widely supported, such as CIELAB color space and you want to convert to a color space of you choice for output as JPEG or WebP etc.

[cmahnke]

Just a small update: Everyone knowing to have a working HDR setup but didn't get the example to work: I was under the misconception, that Chrome enabled this feature by default in June - this isn't the case.
One needs to enable the enable-experimental-web-platform-features flag in Chrome, I've updated the page.

[cmahnke]

Yesterday I had the opportunity to give a lightning talk essentially on this topic at the 2024 online meeting.
I focused heavily on use cases (most of them currently not implemented using IIIF yet) in order to promote this afford. You can find my slides here. I'm waiting for feedback by the participants and report here if I get any.

[cmahnke]

Jus to let anyone know who isn't reading the Slack Channel on a regular basis, @caitlinPerry posted this:

Hey everyone, before you all scatter for the holidays, please mark your calendars for our next community call.
Our first call of 2025, HDR Images via Image API, will be held January 15 at 9am PT / 12pm ET / 5pm UTC (this is a one-week deviation from our regular schedule). We’ll welcome
@Christian Mahnke
(Projektemacher), who will demo a proof of concept for a IIIF Image API endpoint with UltraHDR tiles and present a proposal to indicate technical rendering hints to Image API clients. Then, the IIIF editors will lead a discussion to elicit feedback and determine the community’s interest in supporting this new feature.
Hope many of you will join us!

I'll try to summarize this discussion there as well and be'll happy if the participants can also attend the meeting.

[cmahnke]

Hi everyone,
the notes of the call are available as a Google Doc, my slides are available here. Long story short:
The decision reached is a solution without any API change, by providing use case tailored profiles and to revisit the issue if this approach isn't sufficient. This would, of course, require that there be a certain number of users.

@ruven @robcast @azaroth42 @glenrobson : Everyone who was present is invited to add to the minutes if I have forgotten, misunderstood or abbreviated something.