ES|QL: CCS check for FORK/RERANK/COMPLETION #129463

ioanatia · 2025-06-16T09:04:43Z

related #121950

These commands will be initially released as tech preview and CCS support is not added yet.

We will most likely need a manual backport for RERANK/COMPLETION for 8.19.

With FORK we just didn't get to make the cross cluster tests stable enough.
RERANK and COMPLETION use the inference APIs
COMPLETION allows you to send an LLM prompt and store the output in a column.
RERANK uses the inference rerank API which again uses a configured ML model.
Th the rerank API supports semantic reranking, e.g. you send a list of strings and query string and we return back a list of scores representing how similar the query string is to each of the strings we received.
The problem with both COMPLETION and RERANK is that in order for them to work properly with CCS we would have to enforce a mechanism where we check that the same inference ID is configured in all clusters.

If it's missing in a cluster or if for the same inference ID, you configured it with different LLMs in different clusters, at worst we would fail with a bad error, at best it might look like your query succeeded, but your results would not make much sense.

As part of GAing FORK/COMPLETION/RERANK we will need to support CCS.
But it's not a requirement for tech preview.

afoucret

🎉

ioanatia · 2025-06-17T07:03:30Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/plan/logical/Fork.java

@@ -32,7 +33,7 @@
 * A Fork is a n-ary {@code Plan} where each child is a sub plan, e.g.
 * {@code FORK [WHERE content:"fox" ] [WHERE content:"dog"] }
 */
-public class Fork extends LogicalPlan implements PostAnalysisPlanVerificationAware {
+public class Fork extends LogicalPlan implements PostAnalysisPlanVerificationAware, TelemetryAware {


I did sneak this one here, since I think it's the only change we require in order to include FORK in the telemetry payload and did not want to make it a separate PR.

elasticsearchmachine · 2025-06-17T09:19:06Z

Pinging @elastic/es-analytical-engine (Team:Analytics)

ioanatia · 2025-06-17T10:53:30Z

x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/parser/StatementParserTests.java

@@ -3124,7 +3124,7 @@ public void testInvalidJoinPatterns() {
            var joinPattern = randomIndexPattern(CROSS_CLUSTER, without(WILDCARD_PATTERN), without(INDEX_SELECTOR));
            expectError(
                "FROM " + fromPatterns + " | LOOKUP JOIN " + joinPattern + " ON " + randomIdentifier(),
-                "invalid index pattern [" + unquoteIndexPattern(joinPattern) + "], remote clusters are not supported in LOOKUP JOIN"
+                "invalid index pattern [" + unquoteIndexPattern(joinPattern) + "], remote clusters are not supported with LOOKUP JOIN"


I know I slightly changed the error message - but I wanted to reuse the same check for all commands cc @smalyshev
although I heard that for LOOKUP JOIN we might be able to remove these soon anyway 🤞

smalyshev · 2025-06-17T18:22:07Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/parser/LogicalPlanBuilder.java

            return new LookupJoin(source, p, right, joinFields);
        };
    }

+    private void checkForRemoteClusters(LogicalPlan plan, Source source, String commandName) {


It feels a bit weird that we're running the same code 3 times from 3 different places just to get different error message... I wonder if there isn't a way to somehow capture it better so we don't need to walk the whole (sub-)tree each time anew.

this is just reusing the same approach we had for LOOKUP JOIN.
even on main right now, if an ES|QL query has 3 LOOKUP JOIN commands, we are doing the same check 3 times.
I thought that if we had a better way to do it this would have been flagged in #120277
also it's just a temporary limitation, since FORK/COMPLETION/FORK are all going to be tech preview features first and CCS support will be done as part of the transition to GA.

it looks like we also have another place in LogicalPlanBuilder where walk the plan looking for UnresolvedRelation:

elasticsearch/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/parser/LogicalPlanBuilder.java

Lines 316 to 325 in 4e926ae

public PlanFactory visitStatsCommand(EsqlBaseParser.StatsCommandContext ctx) {

final Stats stats = stats(source(ctx), ctx.grouping, ctx.stats);

return input -> {

if (input.anyMatch(p -> p instanceof UnresolvedRelation ur && ur.indexMode() == IndexMode.TIME_SERIES)) {

return new TimeSeriesAggregate(source(ctx), input, stats.groupings, stats.aggregates, null);

} else {

return new Aggregate(source(ctx), input, stats.groupings, stats.aggregates);

}

};

}

just wanted to check that it's a pattern we use in some other places as well, not just for CCS check 🤔

Yeah we do such things repeatedly and I am not a big fan of this though I am not sure yet how to fix it. It will also become more complicated with CPS since we may not be able to know whether certain index pattern means remote or local without wider context, which is not available at that point. Not sure what to do with it, maybe these checks can be moved to a different place or checked in some other way. Needs more thinking. I guess it's ok for now.

I get it - there might be ways we could improve it, but as you said it's not clear how.
Hopefully we figure out CCS support before we need to go back and see how these check should be improved😄

elasticsearchmachine · 2025-06-18T11:25:02Z

💔 Backport failed

Status	Branch	Result
❌	8.19	Commit could not be cherrypicked due to conflicts

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 129463

ioanatia added 2 commits June 13, 2025 21:29

Add check for CCS

1d2f743

Add tests

f6faf5e

ioanatia added auto-backport Automatically create backport pull requests when merged :Analytics/ES|QL AKA ESQL v8.19.0 v9.1.0 Team:Search - Relevance The Search organization Search Relevance team labels Jun 16, 2025

afoucret approved these changes Jun 17, 2025

View reviewed changes

ioanatia added 3 commits June 17, 2025 08:53

Adjust test error check

1792ba5

Merge remote-tracking branch 'elasticsearch/main' into fork_ccs_check

23e7511

Include FORK in telemetry

225488e

ioanatia commented Jun 17, 2025

View reviewed changes

ioanatia marked this pull request as ready for review June 17, 2025 09:18

ioanatia requested review from afoucret and carlosdelest June 17, 2025 09:18

elasticsearchmachine added the Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) label Jun 17, 2025

elasticsearchmachine removed the Team:Search - Relevance The Search organization Search Relevance team label Jun 17, 2025

afoucret approved these changes Jun 17, 2025

View reviewed changes

ioanatia added the >non-issue label Jun 17, 2025

ioanatia requested a review from smalyshev June 17, 2025 10:45

ioanatia commented Jun 17, 2025

View reviewed changes

ioanatia mentioned this pull request Jun 17, 2025

Initial FORK with restrictions #121950

Open

23 tasks

smalyshev reviewed Jun 17, 2025

View reviewed changes

carlosdelest approved these changes Jun 18, 2025

View reviewed changes

Merge remote-tracking branch 'elasticsearch/main' into fork_ccs_check

5238977

ioanatia merged commit 0dea72b into elastic:main Jun 18, 2025
27 checks passed

ioanatia deleted the fork_ccs_check branch June 18, 2025 11:23

elasticsearchmachine added the backport pending label Jun 18, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

ES|QL: CCS check for FORK/RERANK/COMPLETION #129463

ES|QL: CCS check for FORK/RERANK/COMPLETION #129463

Uh oh!

ioanatia commented Jun 16, 2025 •

edited

Loading

Uh oh!

afoucret left a comment

Uh oh!

ioanatia Jun 17, 2025 •

edited

Loading

Uh oh!

elasticsearchmachine commented Jun 17, 2025

Uh oh!

ioanatia Jun 17, 2025

Uh oh!

smalyshev Jun 17, 2025

Uh oh!

ioanatia Jun 17, 2025

Uh oh!

ioanatia Jun 17, 2025

Uh oh!

smalyshev Jun 17, 2025

Uh oh!

ioanatia Jun 18, 2025

Uh oh!

Uh oh!

elasticsearchmachine commented Jun 18, 2025

Uh oh!

Uh oh!

	public PlanFactory visitStatsCommand(EsqlBaseParser.StatsCommandContext ctx) {
	final Stats stats = stats(source(ctx), ctx.grouping, ctx.stats);
	return input -> {
	if (input.anyMatch(p -> p instanceof UnresolvedRelation ur && ur.indexMode() == IndexMode.TIME_SERIES)) {
	return new TimeSeriesAggregate(source(ctx), input, stats.groupings, stats.aggregates, null);
	} else {
	return new Aggregate(source(ctx), input, stats.groupings, stats.aggregates);
	}
	};
	}

ES|QL: CCS check for FORK/RERANK/COMPLETION #129463

ES|QL: CCS check for FORK/RERANK/COMPLETION #129463

Uh oh!

Conversation

ioanatia commented Jun 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

afoucret left a comment

Choose a reason for hiding this comment

Uh oh!

ioanatia Jun 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elasticsearchmachine commented Jun 17, 2025

Uh oh!

ioanatia Jun 17, 2025

Choose a reason for hiding this comment

Uh oh!

smalyshev Jun 17, 2025

Choose a reason for hiding this comment

Uh oh!

ioanatia Jun 17, 2025

Choose a reason for hiding this comment

Uh oh!

ioanatia Jun 17, 2025

Choose a reason for hiding this comment

Uh oh!

smalyshev Jun 17, 2025

Choose a reason for hiding this comment

Uh oh!

ioanatia Jun 18, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

elasticsearchmachine commented Jun 18, 2025

💔 Backport failed

Uh oh!

Uh oh!

ioanatia commented Jun 16, 2025 •

edited

Loading

ioanatia Jun 17, 2025 •

edited

Loading