After N-1 ingester crashes, query results are unstable

[bboreham]

We replicate to three ingesters but after losing two of them in an Unfortunate Incident, queries would sometimes contain results from just the two restarted instances which didn't have recent history.

It rights itself once those chunks age out of the remaining ingester into the store, but not great in the meantime.

[csmarchbanks]

This should be fixed by #732 correct?

[bboreham]

Different problem - #732 is about dropping samples on writes, while this is about reads from ingesters that have crashed and restarted.

[bboreham]

To improve the user experience, we could get the remaining ingesters to flush those chunks for which they hold the only remaining copy.

So, if I'm ingester X, and I know ingesters A and B have restarted, I iterate over all series in memory, run the distributor hash, and every series that maps to (A, B, X) I should flush.

How do I get to know A and B have restarted? I guess that could be written as metadata into the ring, or we could just have a human type it in via an admin entrypoint.

Slight wrinkle: when using all-labels sharding we will get the wrong hash for series with a blank label value, since we discard that information on entry to the ingester.

[bboreham]

How about:

We have ingesters remember the time their data starts, and hand that time over on a transfer.
If an ingester gets a query covering further back than that time, they return http status 206 - partial content.
Change the querier to not cancel remaining requests on 206.

[bboreham]

I think we can get into the same bad situation if you add a new ingester and then one old ingester restarts - there are now two places to get an incomplete answer. And my suggestion at #731 (comment) helps there too.

[gouthamve]

Fixed by #1103

[gouthamve]

If you're having N ingesters, and 2 ingesters go down and come back, we can ignore a (we only wait for 2 replys) different ingester in different queries, hence will have unstable queries.

shard-by-all-labels makes this rarer, but it doesn't completely fix this.

Its not a trivial fix, and can be only fixed rebalancing data once the ingesters are back up.

[harry671003]

Fixed by #1103

@gouthamve Was this issues fixed only for blocks store?

[bboreham]

The WAL can be used for chunks, but I think Goutham’s subsequent comment is saying it isn’t fixed.

[khaines]

Even the WAL can't prevent this 100% of the time because the WAL can only recover the samples a given ingester has received. While it is offline or otherwise not receiving data, nothing is being written to the WAL, while its replication partners are potentially receiving data.