query-frontend: Response caching should work for subqueries and part queries.

[bwplotka]

AC:

• Subqueries should be split & cached.

I think that should not be that hard to implement, but we would need to finally parse the query and understand what is nested.

cc @brancz, @pracucci @tomwilkie

[pracucci]

@owen-d This is your area of expertise. What's your take, especially considering the recently merged #1878 ?

[bwplotka]

We are happy to help in free time, but ideas/suggestions are very welcome.

Or if any work happened already for this (:

[bwplotka]

Any update on this? 🤗

Any pointers for contributors, e.g if we would want to tackle it?

[owen-d]

I'm not sure if this will be straightforward to implement. We currently sequence a set of middlewares in the query frontend, of which a caching middleware is one. This makes it hard to pull out part of a query without dealing with the promql Engine, which is what we did in the sharding work (#1878). This could be another reason to support a refactoring of the query frontend though. There are a number of issues derived from sequencing middlewares this way, such as #1882

Caching middleware: https://github.com/cortexproject/cortex/blob/master/pkg/querier/queryrange/results_cache.go

Another thought - this is something we may want to solve via some sort of query planner.

[bwplotka]

Yes, definitely worth to look closer in to different design then. I would say initial promql parse might be necessary...

…

On Tue, 14 Apr 2020, 20:19 Owen Diehl, ***@***.***> wrote: I'm not sure if this will be straightforward to implement. We currently sequence a set of middlewares in the query frontend, of which a caching middleware is one. This makes it hard to pull out part of a query without dealing with the promql Engine, which is what we did in the sharding work ( #1878 <#1878>). This could be another reason to support a refactoring of the query frontend though. There are a number of issues derived from sequencing middlewares this way, such as #1882 <#1882> Caching middleware: https://github.com/cortexproject/cortex/blob/master/pkg/querier/queryrange/results_cache.go Another thought - this is something we may want to solve via some sort of query planner. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#2178 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABVA3OZ54MLVMWDAOPDDLOLRMSZKPANCNFSM4K2KDDZA> .

[bwplotka]

Something to add as well : thanos-io/thanos#2569

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. It will be closed in 15 days if no further activity occurs. Thank you for your contributions.

[pracucci]

Still valid