add sdpi p history service option

[d-gregorczyk]

☑ Mandatory Tasks

The following aspects have been respected by the pull request assignee and at least one reviewer:

• Changelog update (necessity checked and entry added or not added respectively)
  • Pull Request Assignee
  • Reviewer

[PaulMartinsen]

Suggest changing R0512 to something like:

The provider shall transmit hm:HistoricMdib before transmitting each contiguous MDIB Sequence of Changes in the range requested by a consumer.
Note: this allows a provider to skip over gaps in its historical record while providing a MDIB Configuration for the consumer to apply changes to.

Possible need to tweak R0514 to something like:

For an MDIB Sequence of Changes that is requested from a SOMDS Provider, the SOMDS Provider shall provide exactly one MDIB Configuration in hm:ChangeSequence/hm:HistoricMdib for each contiguous MDIB sequence of changes.

Reasoning:

I was thinking more about the case where a provider starts providing history information but some of the history becomes unavailable before it is delivered. This could occur when:

1. the consumer requests all (or most) of the history available, and
2. the provider has limited memory and must discard old history records to make room for new changes to the mdib.

Some available options :

1. End subscription:

• The provider ends the history subscription, possibly with a (new) fault message.
• The consumer starts a new history subscription and gets the next lot of data.
• Repeat above steps each time items are lost from this history.

2. Send with gaps:

• The provider skips over the missing data and continues with what it has.
• This is (probably?) useless to the consumer who doesn't have a mdib starting point to apply changes to.

3. Send with gaps and starting mdib

• The provider sends a full mdib snapshot (if this was requested by the original subscription)
• Provider continues with change reports.

I quite like option 3 because:

• it avoids multiple subscriptions (with potential race conditions) and doesn't require any special handling of fault messages on the consumer end.
• the provider does its best to provide the information requested and while still letting the consumer create coherent mdib state.
• its more likely to avoid lost history than if the consumer has to re-subscribe. The consumer can unsubscribe if it doesn't want any more data following a gap.

If the consumer didn't request a baseline mdib, we could sensibly fallback to option 1: end the subscription.

[PaulMartinsen]

I would quite like to be able to omit waveform streams from the history because:

1. they use up a lot of memory,
2. an hour old waveform stream may often not be relevant anymore, and
3. a deeper history of states other than waveform streams may be more relevant than a shallow history with waveform streams.

But the general case, where providers may omit any information, seems quite complicated and could be problematic. That is, important context may be lost.

Could the provider be allowed to omit waveform streams either:

1. without disclosing this to the consumer (except perhaps in the documentation), or
2. by sending RealTimeSamleArrayMetricState with @Off.

Or maybe there is no value in waveform streams in the historic record and they could just be omitted completely? That is, remove msg:WaveformStream from the HistoricReportType in HistoryModel.xsd

The main problem seems to be missing state versions (R1005), although I think we need to get rid of that one anyway.

[PaulMartinsen]

I wonder if the <hm:LanguageFilter/> here could be more generic because histories are potentially large and some one to manage that may be helpful.

So, for example, <hm:Filter Dialect="urn:oid:language-oid"/>. With 0 or more allowed. And that providers must also support http://docs.oasis-open.org/ws-dd/ns/dpws/2009/01/Action and should support sdc.filter://Handle. (must and should for consistency with DWPS and 20701 R0039 ).

It would be nice if absence of the action or handle filter implied including everything. Is it a problem that's inconsistent with the language filter behaviour?

[PaulMartinsen]

From HistoryModel.xsd I think this should be something like:

                <hm:Range>
                    <hm:Version SequenceId="..." StartVersion="..." EndVersion="..."/>
                    <-- 'Range' may be alternatively supplied using time-stamps, though versions are preferred:
                   <hm:Time StartTime="4294967295" EndTime="4294967295"/>
                   -->
                </hm:Range>

[PaulMartinsen]

Do we need to deal with time epochs here? I can see three options:

1. drop support for requesting history by time, or use relative time like subscriptions.
2. leave the provider free to decide which time-reference frame it should use when comparing mdib timestamps and history query time-stamps.
3. add attributes to select epoch.

1 seems best to me, but I'm not sure what the use cases are for this one. 2 seems okay as well because time is supposed to be UTC time, which is independent of non-slewing time-adjustments. If 2, perhaps a note along the lines : "The provider may not reliably select the correct change reports for time ranges that span an abrupt time adjustment".

[PaulMartinsen]

Is there any value for consumers in allowing hm:Range and/or either range limit to be omitted to ask for:

• everything available (hm:Range omitted),
• everything up to a specific mdib version/time @StartVersion or @StartTime omitted,
• everything from a specific mdib version/time to the time the subscription was created (@EndVersion or @EndTime omitted)

[PaulMartinsen]

Suggest adding <xsd:attribute name="LastReport" default="false" type="xsd:boolean" use="optional" /> to signal end of reporting sequencing without needing an empty message.

[PaulMartinsen]

The sequence diagrams weren't showing up for me. It looks like the action isn't copying them into the supplement from .ci\asciidoc-converter\images when it makes the zip file I downloaded from https://github.com/IHE/DEV.SDPi/actions/runs/15761385158

[PaulMartinsen]

Suggest making the Consumer <-- Provider: SubscriptionEnd(SourceCancelling) optional for consistency with ws-eventing (where it is a should) :

opt Notification may not be sent
  Consumer <-- Provider: SubscriptionEnd(SourceCancelling)
end

Also suggest setting flag on last iteration since this makes life easier for consumers, something like:

loop until no more reports exist for @SequenceId=X
    alt Not last iteration
        Consumer <-- Provider: ChangeSequenceReport(ChangeSequence(LastReport=false, HistoricReport(@Time=tn>t)*))
    else Last iteration
        Consumer <-- Provider: ChangeSequenceReport(ChangeSequence(LastReport=true, HistoricReport(@Time=tn>t)*))
    end
end

[PaulMartinsen]

I think this should be:
Expires can be omitted as the subscription should be terminated by the event source when all data....

In WS-Eventing, a missing Expires means the subscriber is requesting an indefinite lifetime but I don't think the source is required to grant that. So the consumer probably needs to keep renewing the subscription if the subscribe response includes a wse:Expires and the subscription expires naturally before the consumer receives all data.

Also DEV-27 constrains wse:Expires to xml-durations, and doesn't allow it to be omitted. I wonder if we should just stick to including the expires element for the history service for simplicity?

[PaulMartinsen]

Is it worth pointing out that the consumer can also control the data rate it accepts the information using the HTTP protocol (i.e., delaying the response code)?

[PaulMartinsen]

Suggest setting hm:ChangeSequenceReport/@LastReport="true" for the last report (see other comments) instead of sending an empty report because this eliminates one message, which could fail to get through so including a flag on the message seems much more robust.

Also, ending the subscription seems okay to me. But I wonder, based "signify" and other text, if you mean that the provider must send a subscription end notification. That seems a problem because:

• "signify" to imply sending a notification may be a little opaque,
• consumers are not required to supply an wse:EndTo end-point in ws-eventing,
• providers are not required to send subscription end notifications by ws-eventing (its only should),
• the end notification could be lost and we haven't specified retry mechanisms.

It seems okay to me for the provider to end the subscription, and for them to should send the end-notification if they like to (that is, consistent with ws-eventing 2004 draft we use). And especially if the consumer could also detect the subscription has ended with hm:ChangeSequenceReport/@LastReport (I guess that's what they'd do in practice so everything else is just trying to stay consistent with the ws-eventing standards draft, which we might not want to over-specify? ).

But consumers could also detect failure of Renew operations from ws-eventing (which would return wse:UnableToRenew). Can't really use GetStatus because that's not specified by SDPi and its not clear from the ws-eventing draft what a source should do when the status of a subscription that doesn't exist is requested :(.

[PaulMartinsen]

I suggest that (for consistency with other profiles/standards) providers must support action based filtering and should support handle based filtering for the history service. So the protocol should support this somehow (see suggestion for one approach in another comment).

Because the history service is a foundational feature that may be used by large and small participants in the future, a little flexibility seems worthwhile even if the business cases aren't 100% clear at the beginning. I know low-resource providers would prefer that consumers only request what they need but providers have no control over that and best they can do is support the mechanisms.

Implementation doesn't seem hard to me if there is a mechanism in the protocol but I don't feel very strongly about this. Mostly because it requires consumers to use it for providers to benefit, which may not happen in practice.

[PaulMartinsen]

Is the history service mandatory or an option? I guess I think it should be an option because some devices (e.g., a portable pulse oximeter used for spot measurements) may not wish/have the resources to provide history. Devices with reliable connections to the network (e.g., ethernet) may not wish to maintain history either.

In that case, perhaps R0503 could be something like: A provider shall indicate support for the history service by implementing the port type with the QName hs:HistoryService`.

And R0504 would become something like A provider that implements the history service shall accept the ws-eventing...

[PaulMartinsen]

If the history service is mandatory, I think providers should be allowed (e.g., by R0504) to respond to subscriptions on the history service with a wse:FilteringRequestedUnavailable . Other wise they seem to have to send at least one history notification to signal the end of the subscription, which seems like a lot of work to implement and maintain for a null-response.

[PaulMartinsen]

R0507 and R0508 allow providers to send new mdib updates via the history service (they may respond with data that exceed [start, end]). They just keep sending change reports. I think this could be great because it saves splitting things up across multiple messages. However, when we last discussed that it wasn't something you wanted. Is that still the case?

[PaulMartinsen]

Also, perhaps we need to note that A provider may respond with a subset of data from the interval [start, end]

This could happen when:

1. the provider has discarded historic data to make room for current data (see suggestion in general comments for an idea to deal with gaps),
2. the filter requests measurements from the future (either in time or mdib version number).

[PaulMartinsen]

Suggest, for clarity, a note along the lines: the provider may omit hm:HistoricLocalizedTexts if it has no information matching the filter to transmit (e.g, doesn't support the language requested).

[PaulMartinsen]

I haven't looked at localization really, but reading this I wondered R0509 requires providers to send localized text even if they only have static localized text. Can't the consumer just get the localized text from the localized text service in this case? I think this is worth thinking through because:

1. many (most?) providers may only have static localized text
2. there might be a lot of localized text so it seems quite inefficient to send it to every history subscription if it never changes. Unless we trying to encourage everyone to support gzip :).

[PaulMartinsen]

As in other comments, I think we need to make this a should for consistency with ws-eventing, absence of retry and consumers are not required to provide an wse:EndTo address. As I understand it, the wse:Delivery can't be used for subscription end messages. I might be wrong about that though.

Then this requirement may not be needed, since it seems the same as ws-eventing. Though it does clarify expectations which is always helpful!