Spatial and temporal search constraints, and interactions

[tomcrane]

In anticipation of future revisiting of #780

We dropped box or region from the first version of the Search spec for the reasons in that issue.

Now that Presentation 3 is ready, can we take t= and xywh= from P3 and use them in Search?

• xywh - spatial media fragment (as per media fragment spec). Server only returns annotations that intersect this region (rather than are wholly contained by this region). An extension flat could possibly toggle intersection/containment. The presence of this parameter implicitly limits annotations that can be returned to annotations that target space (but see note below).
• t: temporal fragment (as per media fragment spec). Server only returns annotations that intersect this time period.

Combinations of behaviours. Say you have annos on video that only have t targets (like those in https://openhypervideo.github.io/iiif-interactive-transcript/):

Example - https://tomcrane.github.io/bbctextav/iiif/ID194804400-transcript.json

...but the canvases are spatial (as they are here, it's a video). If I fire off xywh=100,100,200,200&=10,20 then I must get all the annos that overlap t=10,20, because they have an implicit spatial target of the whole canvas. But If it was a radio broadcast, where the transcription anno targets look the same but the target has no spatial dimensions, what happens? The search server that indexed the annos probably has no idea that the anno targets have a spatial extent (we can't require dereferencing of target and may not even be possible). In this particular case the targets of the indexed annos are implicitly t=t1,t2&xywh=0,0,1920,1080 but even that info alone doesn't tell you everything, you'd still need to know at query time that the target canvas itself isn't a 4K video - we don't know that 1920,1080 is the full extent of the target and not the top left quadrant, and therefore implicit in the query.

Do we need a param that identifies whether the spatial or temporal constraint should cause annos that lack targets with those dimensions to be excluded or included in the response.

Is that enough though to meet use cases?

All the search server indexes are the annos. These would look identical for a radio broadcast and a TV broadcast. https://tomcrane.github.io/bbctextav/iiif/ID194804400-transcript.json

• We might also want a point time as a search constraint (where intersect means overlap) - we run into same problem that causes Presentation API to introduce a point selector (there's no notion of points in W3C Media Fragments spec; a point is not a fragment).

[azaroth42]

(Rob to draft proposal)

[azaroth42]

I think this is the top priority new feature for 2.0, as it brings search up to date with Prezi3's new temporal dimension.
I also think @tomcrane has it pretty much right above.

To address the points:

• As the params aren't media fragments but search params that just use a superset of the syntax, I think we can overload them for this case more easily than overloading the actual fragments (which would have gotten us in trouble). So ?t=t1 should be valid as a point-in-time query.
• Multiple params as OR or AND ... I think they have to be AND, meaning that if you search a canvas for a dimension that it doesn't have, it doesn't match. If the query knows enough to get xywh from somewhere, then some agent has given it those coordinates (human or machine, we don't know or need to know). The client can also filter the results if necessary, or issue another search if there's too many / not enough. Relatedly, if you have q and t and xywh, then you'd be looking for whatever matches q and t and xywh. We wouldn't want to match things that didn't have text to match in q. Finally ... we don't want to create a complex search grammar, as that way lies madness.

We might want to re-address during #951 (e.g. search by target canvas might also impact searching spatio/temporal regions of the canvas) but I think the params are good enough for 2.0 draft.

[azaroth42]

Should this be included in Search 3.0? (thumbs up/down on the comment please)

[zimeon]

Following on from Rob's thought that AND is obvious meaning for multiple parameters, I think that would also mean that annos without a specified target dimension would be EXCLUDED. To include them would feel more like an OR behavior