From 31214cb6ca100ec2f0d19a31ccba67c75d289dd1 Mon Sep 17 00:00:00 2001 From: Stanislav Malyshev Date: Wed, 25 Jun 2025 16:16:43 -0600 Subject: [PATCH 1/9] Improve skip_un description --- .../languages/esql-cross-clusters.md | 23 ++++++++++++++++--- 1 file changed, 20 insertions(+), 3 deletions(-) diff --git a/explore-analyze/query-filter/languages/esql-cross-clusters.md b/explore-analyze/query-filter/languages/esql-cross-clusters.md index 349501369d..ee5831fdc8 100644 --- a/explore-analyze/query-filter/languages/esql-cross-clusters.md +++ b/explore-analyze/query-filter/languages/esql-cross-clusters.md @@ -470,10 +470,27 @@ FROM my-index-000001,cluster*:my-index-*,cluster_three:-my-index-000001 ## Optional remote clusters [ccq-skip-unavailable-clusters] +```{applies_to} +stack: ga 9.1.0+ +``` + +If the remote cluster is configured with `skip_unavailable: true` (the default setting), the cluster would be set +to `skipped` or `partial` status but the query will not fail, if: + +* The remote cluster is disconnected from the querying cluster, either before or during the query. +* The remote cluster does not have the requested index. +* An error happened while processing the query on the remote cluster. + +The `partial` status will be used if the remote query was partially successful and some data was returned. -If a remote cluster disconnects from the querying cluster, {{ccs}} for {{esql}} will set it to `skipped` -and continue the query with other clusters, unless the remote cluster's `skip_unavailable` setting is set to `false`, -in which case the query will fail. +This however does not apply to the situation when the remote cluster is missing an index and this is the only index in the query, +or all the indices in the query are missing. For example, the following queries will fail: + +```esql +FROM cluster_one:missing-index | LIMIT 10 +FROM cluster_one:missing-index* | LIMIT 10 +FROM cluster_one:missing-index*,cluster_two:missing-index | LIMIT 10 +``` ## Query across clusters during an upgrade [ccq-during-upgrade] From 1c756067dbabbb9ea25955fe539ced02c2830713 Mon Sep 17 00:00:00 2001 From: Stanislav Malyshev Date: Wed, 25 Jun 2025 17:02:56 -0600 Subject: [PATCH 2/9] fix --- explore-analyze/query-filter/languages/esql-cross-clusters.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/explore-analyze/query-filter/languages/esql-cross-clusters.md b/explore-analyze/query-filter/languages/esql-cross-clusters.md index ee5831fdc8..6156710fc5 100644 --- a/explore-analyze/query-filter/languages/esql-cross-clusters.md +++ b/explore-analyze/query-filter/languages/esql-cross-clusters.md @@ -471,7 +471,7 @@ FROM my-index-000001,cluster*:my-index-*,cluster_three:-my-index-000001 ## Optional remote clusters [ccq-skip-unavailable-clusters] ```{applies_to} -stack: ga 9.1.0+ +stack: ga 9.1 ``` If the remote cluster is configured with `skip_unavailable: true` (the default setting), the cluster would be set From 3bcf5362a7370aeecaedd3afd8b5a2666bf4e804 Mon Sep 17 00:00:00 2001 From: Stanislav Malyshev Date: Thu, 26 Jun 2025 16:17:44 -0600 Subject: [PATCH 3/9] Use 9.0/9.1 tabs --- .../languages/esql-cross-clusters.md | 244 +++++++++++------- 1 file changed, 157 insertions(+), 87 deletions(-) diff --git a/explore-analyze/query-filter/languages/esql-cross-clusters.md b/explore-analyze/query-filter/languages/esql-cross-clusters.md index 6156710fc5..34386129ca 100644 --- a/explore-analyze/query-filter/languages/esql-cross-clusters.md +++ b/explore-analyze/query-filter/languages/esql-cross-clusters.md @@ -9,34 +9,40 @@ products: - id: elasticsearch --- - - # Use ES|QL across clusters [esql-cross-clusters] - ::::{warning} -{{ccs-cap}} for {{esql}} is in technical preview and may be changed or removed in a future release. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features. +{{ccs-cap}} for {{esql}} is in technical preview and may be changed or removed in a future release. Elastic will work to +fix any issues, but features in technical preview are not subject to the support SLA of official GA features. :::: - With {{esql}}, you can execute a single query across multiple clusters. - ## Prerequisites [esql-ccs-prerequisites] -* {{ccs-cap}} requires remote clusters. To set up remote clusters, see [*Remote clusters*](../../../deploy-manage/remote-clusters.md). - - To ensure your remote cluster configuration supports {{ccs}}, see [Supported {{ccs}} configurations](../../../solutions/search/cross-cluster-search.md#ccs-supported-configurations). +* {{ccs-cap}} requires remote clusters. To set up remote clusters, see [*Remote + clusters*](../../../deploy-manage/remote-clusters.md). -* For full {{ccs}} capabilities, the local and remote cluster must be on the same [subscription level](https://www.elastic.co/subscriptions). -* The local coordinating node must have the [`remote_cluster_client`](../../../deploy-manage/distributed-architecture/clusters-nodes-shards/node-roles.md#remote-node) node role. -* If you use [sniff mode](/deploy-manage/remote-clusters/remote-clusters-self-managed.md#sniff-mode), the local coordinating node must be able to connect to seed and gateway nodes on the remote cluster. + To ensure your remote cluster configuration supports {{ccs}}, + see [Supported {{ccs}} configurations](../../../solutions/search/cross-cluster-search.md#ccs-supported-configurations). - We recommend using gateway nodes capable of serving as coordinating nodes. The seed nodes can be a subset of these gateway nodes. +* For full {{ccs}} capabilities, the local and remote cluster must be on the + same [subscription level](https://www.elastic.co/subscriptions). +* The local coordinating node must have the [ + `remote_cluster_client`](../../../deploy-manage/distributed-architecture/clusters-nodes-shards/node-roles.md#remote-node) + node role. +* If you use [sniff mode](/deploy-manage/remote-clusters/remote-clusters-self-managed.md#sniff-mode), the local + coordinating node must be able to connect to seed and gateway nodes on the remote cluster. -* If you use [proxy mode](/deploy-manage/remote-clusters/remote-clusters-self-managed.md#proxy-mode), the local coordinating node must be able to connect to the configured `proxy_address`. The proxy at this address must be able to route connections to gateway and coordinating nodes on the remote cluster. -* {{ccs-cap}} requires different security privileges on the local cluster and remote cluster. See [Configure privileges for {{ccs}}](../../../deploy-manage/remote-clusters/remote-clusters-cert.md#remote-clusters-privileges-ccs) and [*Remote clusters*](../../../deploy-manage/remote-clusters.md). + We recommend using gateway nodes capable of serving as coordinating nodes. The seed nodes can be a subset of these + gateway nodes. +* If you use [proxy mode](/deploy-manage/remote-clusters/remote-clusters-self-managed.md#proxy-mode), the local + coordinating node must be able to connect to the configured `proxy_address`. The proxy at this address must be able to + route connections to gateway and coordinating nodes on the remote cluster. +* {{ccs-cap}} requires different security privileges on the local cluster and remote cluster. + See [Configure privileges for {{ccs}}](../../../deploy-manage/remote-clusters/remote-clusters-cert.md#remote-clusters-privileges-ccs) + and [*Remote clusters*](../../../deploy-manage/remote-clusters.md). ## Security model [esql-ccs-security-model] @@ -46,12 +52,11 @@ With {{esql}}, you can execute a single query across multiple clusters. * [API key authentication](#esql-ccs-security-model-api-key) ::::{tip} -To check which security model is being used to connect your clusters, run `GET _remote/info`. If you’re using the API key authentication method, you’ll see the `"cluster_credentials"` key in the response. +To check which security model is being used to connect your clusters, run `GET _remote/info`. If you’re using the API +key authentication method, you’ll see the `"cluster_credentials"` key in the response. :::: - - ### TLS certificate authentication [esql-ccs-security-model-certificate] ::::{admonition} Deprecated in 9.0.0. @@ -60,24 +65,36 @@ To check which security model is being used to connect your clusters, run `GET _ Use [API key authentication](#esql-ccs-security-model-api-key) instead. :::: +TLS certificate authentication secures remote clusters with mutual TLS. This could be the preferred model when a single +administrator has full control over both clusters. We generally recommend that roles and their privileges be identical +in both clusters. -TLS certificate authentication secures remote clusters with mutual TLS. This could be the preferred model when a single administrator has full control over both clusters. We generally recommend that roles and their privileges be identical in both clusters. - -Refer to [TLS certificate authentication](../../../deploy-manage/remote-clusters/remote-clusters-cert.md) for prerequisites and detailed setup instructions. - +Refer to [TLS certificate authentication](../../../deploy-manage/remote-clusters/remote-clusters-cert.md) for +prerequisites and detailed setup instructions. ### API key authentication [esql-ccs-security-model-api-key] -The following information pertains to using {{esql}} across clusters with the [**API key based security model**](../../../deploy-manage/remote-clusters/remote-clusters-api-key.md). You’ll need to follow the steps on that page for the **full setup instructions**. This page only contains additional information specific to {{esql}}. +The following information pertains to using {{esql}} across clusters with the [**API key based security model +**](../../../deploy-manage/remote-clusters/remote-clusters-api-key.md). You’ll need to follow the steps on that page for +the **full setup instructions**. This page only contains additional information specific to {{esql}}. -API key based cross-cluster search (CCS) enables more granular control over allowed actions between clusters. This may be the preferred model when you have different administrators for different clusters and want more control over who can access what data. In this model, cluster administrators must explicitly define the access given to clusters and users. +API key based cross-cluster search (CCS) enables more granular control over allowed actions between clusters. This may +be the preferred model when you have different administrators for different clusters and want more control over who can +access what data. In this model, cluster administrators must explicitly define the access given to clusters and users. You will need to: -* Create an API key on the **remote cluster** using the [Create cross-cluster API key](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-create-cross-cluster-api-key) API or using the [Kibana API keys UI](../../../deploy-manage/api-keys/elasticsearch-api-keys.md). -* Add the API key to the keystore on the **local cluster**, as part of the steps in [configuring the local cluster](../../../deploy-manage/remote-clusters/remote-clusters-api-key.md#remote-clusters-security-api-key-local-actions). All cross-cluster requests from the local cluster are bound by the API key’s privileges. +* Create an API key on the **remote cluster** using + the [Create cross-cluster API key](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-create-cross-cluster-api-key) + API or using the [Kibana API keys UI](../../../deploy-manage/api-keys/elasticsearch-api-keys.md). +* Add the API key to the keystore on the **local cluster**, as part of the steps + in [configuring the local cluster](../../../deploy-manage/remote-clusters/remote-clusters-api-key.md#remote-clusters-security-api-key-local-actions). + All cross-cluster requests from the local cluster are bound by the API key’s privileges. -Using {{esql}} with the API key based security model requires some additional permissions that may not be needed when using the traditional query DSL based search. The following example API call creates a role that can query remote indices using {{esql}} when using the API key based security model. The final privilege, `remote_cluster`, is required to allow remote enrich operations. +Using {{esql}} with the API key based security model requires some additional permissions that may not be needed when +using the traditional query DSL based search. The following example API call creates a role that can query remote +indices using {{esql}} when using the API key based security model. The final privilege, `remote_cluster`, is required +to allow remote enrich operations. ```console POST /_security/role/remote1 @@ -110,14 +127,25 @@ POST /_security/role/remote1 ``` 1. The `cross_cluster_search` cluster privilege is required for the *local* cluster. -2. Typically, users will have permissions to read both local and remote indices. However, for cases where the role is intended to ONLY search the remote cluster, the `read` permission is still required for the local cluster. To provide read access to the local cluster, but disallow reading any indices in the local cluster, the `names` field may be an empty string. -3. The indices allowed read access to the remote cluster. The configured [cross-cluster API key](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-create-cross-cluster-api-key) must also allow this index to be read. -4. The `read_cross_cluster` privilege is always required when using {{esql}} across clusters with the API key based security model. -5. The remote clusters to which these privileges apply. This remote cluster must be configured with a [cross-cluster API key](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-create-cross-cluster-api-key) and connected to the remote cluster before the remote index can be queried. Verify connection using the [Remote cluster info](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-remote-info) API. -6. Required to allow remote enrichment. Without this, the user cannot read from the `.enrich` indices on the remote cluster. The `remote_cluster` security privilege was introduced in version **8.15.0**. - - -You will then need a user or API key with the permissions you created above. The following example API call creates a user with the `remote1` role. +2. Typically, users will have permissions to read both local and remote indices. However, for cases where the role is + intended to ONLY search the remote cluster, the `read` permission is still required for the local cluster. To provide + read access to the local cluster, but disallow reading any indices in the local cluster, the `names` field may be an + empty string. +3. The indices allowed read access to the remote cluster. The + configured [cross-cluster API key](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-create-cross-cluster-api-key) + must also allow this index to be read. +4. The `read_cross_cluster` privilege is always required when using {{esql}} across clusters with the API key based + security model. +5. The remote clusters to which these privileges apply. This remote cluster must be configured with + a [cross-cluster API key](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-create-cross-cluster-api-key) + and connected to the remote cluster before the remote index can be queried. Verify connection using + the [Remote cluster info](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-remote-info) + API. +6. Required to allow remote enrichment. Without this, the user cannot read from the `.enrich` indices on the remote + cluster. The `remote_cluster` security privilege was introduced in version **8.15.0**. + +You will then need a user or API key with the permissions you created above. The following example API call creates a +user with the `remote1` role. ```console POST /_security/user/remote_user @@ -127,20 +155,22 @@ POST /_security/user/remote_user } ``` -Remember that all cross-cluster requests from the local cluster are bound by the cross cluster API key’s privileges, which are controlled by the remote cluster’s administrator. +Remember that all cross-cluster requests from the local cluster are bound by the cross cluster API key’s privileges, +which are controlled by the remote cluster’s administrator. ::::{tip} -Cross cluster API keys created in versions prior to 8.15.0 will need to replaced or updated to add the new permissions required for {{esql}} with ENRICH. +Cross cluster API keys created in versions prior to 8.15.0 will need to replaced or updated to add the new permissions +required for {{esql}} with ENRICH. :::: - - ## Remote cluster setup [ccq-remote-cluster-setup] Once the security model is configured, you can add remote clusters. -The following [cluster update settings](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-settings) API request adds three remote clusters: `cluster_one`, `cluster_two`, and `cluster_three`. +The +following [cluster update settings](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-settings) +API request adds three remote clusters: `cluster_one`, `cluster_two`, and `cluster_three`. ```console PUT _cluster/settings @@ -171,13 +201,14 @@ PUT _cluster/settings } ``` -1. Since `skip_unavailable` was not set on `cluster_three`, it uses the default of `true`. See the [Optional remote clusters](#ccq-skip-unavailable-clusters) section for details. - - +1. Since `skip_unavailable` was not set on `cluster_three`, it uses the default of `true`. See + the [Optional remote clusters](#ccq-skip-unavailable-clusters) section for details. ## Query across multiple clusters [ccq-from] -In the `FROM` command, specify data streams and indices on remote clusters using the format `:`. For instance, the following {{esql}} request queries the `my-index-000001` index on a single remote cluster named `cluster_one`: +In the `FROM` command, specify data streams and indices on remote clusters using the format +`:`. For instance, the following {{esql}} request queries the `my-index-000001` index on a +single remote cluster named `cluster_one`: ```esql FROM cluster_one:my-index-000001 @@ -194,17 +225,21 @@ FROM my-index-000001,cluster_one:my-index-000001,cluster_two:my-index-000001 | LIMIT 10 ``` -Likewise, this {{esql}} request queries the `my-index-000001` index from all remote clusters (`cluster_one`, `cluster_two`, and `cluster_three`): +Likewise, this {{esql}} request queries the `my-index-000001` index from all remote clusters (`cluster_one`, +`cluster_two`, and `cluster_three`): ```esql FROM *:my-index-000001 | LIMIT 10 ``` - ## Cross-cluster metadata [ccq-cluster-details] -Using the `"include_ccs_metadata": true` option, users can request that ES|QL {{ccs}} responses include metadata about the search on each cluster (when the response format is JSON). Here we show an example using the async search endpoint. {{ccs-cap}} metadata is also present in the synchronous search endpoint response when requested. If the search returns partial results and there are partial shard or remote cluster failures, `_clusters` metadata containing the failures will be included in the response regardless of the `include_ccs_metadata` parameter. +Using the `"include_ccs_metadata": true` option, users can request that ES|QL {{ccs}} responses include metadata about +the search on each cluster (when the response format is JSON). Here we show an example using the async search endpoint. +{{ccs-cap}} metadata is also present in the synchronous search endpoint response when requested. If the search returns +partial results and there are partial shard or remote cluster failures, `_clusters` metadata containing the failures +will be included in the response regardless of the `include_ccs_metadata` parameter. ```console POST /_query/async?format=json @@ -286,15 +321,28 @@ Which returns: ``` 1. How long the entire search (across all clusters) took, in milliseconds. -2. This section of counters shows all possible cluster search states and how many cluster searches are currently in that state. The clusters can have one of the following statuses: **running**, **successful** (searches on all shards were successful), **skipped** (the search failed on a cluster marked with `skip_unavailable`=`true`), **failed** (the search failed on a cluster marked with `skip_unavailable`=`false`) or **partial** (the search was [interrupted](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-esql) before finishing or has partially failed). +2. This section of counters shows all possible cluster search states and how many cluster searches are currently in that + state. The clusters can have one of the following statuses: **running**, **successful** (searches on all shards were + successful), **skipped** (the search failed on a cluster marked with `skip_unavailable`=`true`), **failed** (the + search failed on a cluster marked with `skip_unavailable`=`false`) or **partial** (the search + was [interrupted](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-esql) before finishing or has + partially failed). 3. The `_clusters/details` section shows metadata about the search on each cluster. -4. If you included indices from the local cluster you sent the request to in your {{ccs}}, it is identified as "(local)". -5. How long (in milliseconds) the search took on each cluster. This can be useful to determine which clusters have slower response times than others. -6. The shard details for the search on that cluster, including a count of shards that were skipped due to the can-match phase results. Shards are skipped when they cannot have any matching data and therefore are not included in the full ES|QL query. -7. The `is_partial` field is set to `true` if the search has partial results for any reason, for example due to partial shard failures, -failures in remote clusters, or if the async query was stopped by calling the [async query stop API](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-esql). - -The cross-cluster metadata can be used to determine whether any data came back from a cluster. For instance, in the query below, the wildcard expression for `cluster-two` did not resolve to a concrete index (or indices). The cluster is, therefore, marked as *skipped* and the total number of shards searched is set to zero. +4. If you included indices from the local cluster you sent the request to in your {{ccs}}, it is identified as "( + local)". +5. How long (in milliseconds) the search took on each cluster. This can be useful to determine which clusters have + slower response times than others. +6. The shard details for the search on that cluster, including a count of shards that were skipped due to the can-match + phase results. Shards are skipped when they cannot have any matching data and therefore are not included in the full + ES|QL query. +7. The `is_partial` field is set to `true` if the search has partial results for any reason, for example due to partial + shard failures, + failures in remote clusters, or if the async query was stopped by calling + the [async query stop API](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-esql). + +The cross-cluster metadata can be used to determine whether any data came back from a cluster. For instance, in the +query below, the wildcard expression for `cluster-two` did not resolve to a concrete index (or indices). The cluster is, +therefore, marked as *skipped* and the total number of shards searched is set to zero. ```console POST /_query/async?format=json @@ -360,19 +408,25 @@ Which returns: 2. Indicates that no shards were searched (due to not having any matching indices). 3. Since one of the clusters is skipped, the search result is marked as partial. - - ## Enrich across clusters [ccq-enrich] -Enrich in {{esql}} across clusters operates similarly to [local enrich](elasticsearch://reference/query-languages/esql/commands/processing-commands.md#esql-enrich). If the enrich policy and its enrich indices are consistent across all clusters, simply write the enrich command as you would without remote clusters. In this default mode, {{esql}} can execute the enrich command on either the local cluster or the remote clusters, aiming to minimize computation or inter-cluster data transfer. Ensuring that the policy exists with consistent data on both the local cluster and the remote clusters is critical for ES|QL to produce a consistent query result. +Enrich in {{esql}} across clusters operates similarly +to [local enrich](elasticsearch://reference/query-languages/esql/commands/processing-commands.md#esql-enrich). If the +enrich policy and its enrich indices are consistent across all clusters, simply write the enrich command as you would +without remote clusters. In this default mode, {{esql}} can execute the enrich command on either the local cluster or +the remote clusters, aiming to minimize computation or inter-cluster data transfer. Ensuring that the policy exists with +consistent data on both the local cluster and the remote clusters is critical for ES|QL to produce a consistent query +result. ::::{tip} -Enrich in {{esql}} across clusters using the API key based security model was introduced in version **8.15.0**. Cross cluster API keys created in versions prior to 8.15.0 will need to replaced or updated to use the new required permissions. Refer to the example in the [API key authentication](#esql-ccs-security-model-api-key) section. +Enrich in {{esql}} across clusters using the API key based security model was introduced in version **8.15.0**. Cross +cluster API keys created in versions prior to 8.15.0 will need to replaced or updated to use the new required +permissions. Refer to the example in the [API key authentication](#esql-ccs-security-model-api-key) section. :::: - -In the following example, the enrich with `hosts` policy can be executed on either the local cluster or the remote cluster `cluster_one`. +In the following example, the enrich with `hosts` policy can be executed on either the local cluster or the remote +cluster `cluster_one`. ```esql FROM my-index-000001,cluster_one:my-index-000001 @@ -380,7 +434,8 @@ FROM my-index-000001,cluster_one:my-index-000001 | LIMIT 10 ``` -Enrich with an {{esql}} query against remote clusters only can also happen on the local cluster. This means the below query requires the `hosts` enrich policy to exist on the local cluster as well. +Enrich with an {{esql}} query against remote clusters only can also happen on the local cluster. This means the below +query requires the `hosts` enrich policy to exist on the local cluster as well. ```esql FROM cluster_one:my-index-000001,cluster_two:my-index-000001 @@ -388,10 +443,11 @@ FROM cluster_one:my-index-000001,cluster_two:my-index-000001 | ENRICH hosts ON ip ``` - ### Enrich with coordinator mode [esql-enrich-coordinator] -{{esql}} provides the enrich `_coordinator` mode to force {{esql}} to execute the enrich command on the local cluster. This mode should be used when the enrich policy is not available on the remote clusters or maintaining consistency of enrich indices across clusters is challenging. +{{esql}} provides the enrich `_coordinator` mode to force {{esql}} to execute the enrich command on the local cluster. +This mode should be used when the enrich policy is not available on the remote clusters or maintaining consistency of +enrich indices across clusters is challenging. ```esql FROM my-index-000001,cluster_one:my-index-000001 @@ -405,13 +461,15 @@ Enrich with the `_coordinator` mode usually increases inter-cluster data transfe :::: - - ### Enrich with remote mode [esql-enrich-remote] -{{esql}} also provides the enrich `_remote` mode to force {{esql}} to execute the enrich command independently on each remote cluster where the target indices reside. This mode is useful for managing different enrich data on each cluster, such as detailed information of hosts for each region where the target (main) indices contain log events from these hosts. +{{esql}} also provides the enrich `_remote` mode to force {{esql}} to execute the enrich command independently on each +remote cluster where the target indices reside. This mode is useful for managing different enrich data on each cluster, +such as detailed information of hosts for each region where the target (main) indices contain log events from these +hosts. -In the below example, the `hosts` enrich policy is required to exist on all remote clusters: the `querying` cluster (as local indices are included), the remote cluster `cluster_one`, and `cluster_two`. +In the below example, the `hosts` enrich policy is required to exist on all remote clusters: the `querying` cluster (as +local indices are included), the remote cluster `cluster_one`, and `cluster_two`. ```esql FROM my-index-000001,cluster_one:my-index-000001,cluster_two:my-index-000001 @@ -420,7 +478,9 @@ FROM my-index-000001,cluster_one:my-index-000001,cluster_two:my-index-000001 | LIMIT 10 ``` -A `_remote` enrich cannot be executed after a [stats](elasticsearch://reference/query-languages/esql/commands/processing-commands.md#esql-stats-by) command. The following example would result in an error: +A `_remote` enrich cannot be executed after +a [stats](elasticsearch://reference/query-languages/esql/commands/processing-commands.md#esql-stats-by) command. The +following example would result in an error: ```esql FROM my-index-000001,cluster_one:my-index-000001,cluster_two:my-index-000001 @@ -430,10 +490,11 @@ FROM my-index-000001,cluster_one:my-index-000001,cluster_two:my-index-000001 | LIMIT 10 ``` - ### Multiple enrich commands [esql-multi-enrich] -You can include multiple enrich commands in the same query with different modes. {{esql}} will attempt to execute them accordingly. For example, this query performs two enriches, first with the `hosts` policy on any cluster and then with the `vendors` policy on the local cluster. +You can include multiple enrich commands in the same query with different modes. {{esql}} will attempt to execute them +accordingly. For example, this query performs two enriches, first with the `hosts` policy on any cluster and then with +the `vendors` policy on the local cluster. ```esql FROM my-index-000001,cluster_one:my-index-000001,cluster_two:my-index-000001 @@ -442,7 +503,8 @@ FROM my-index-000001,cluster_one:my-index-000001,cluster_two:my-index-000001 | LIMIT 10 ``` -A `_remote` enrich command can’t be executed after a `_coordinator` enrich command. The following example would result in an error. +A `_remote` enrich command can’t be executed after a `_coordinator` enrich command. The following example would result +in an error. ```esql FROM my-index-000001,cluster_one:my-index-000001,cluster_two:my-index-000001 @@ -451,40 +513,40 @@ FROM my-index-000001,cluster_one:my-index-000001,cluster_two:my-index-000001 | LIMIT 10 ``` - ## Excluding clusters or indices from {{esql}} query [ccq-exclude] -To exclude an entire cluster, prefix the cluster alias with a minus sign in the `FROM` command, for example: `-my_cluster:*`: +To exclude an entire cluster, prefix the cluster alias with a minus sign in the `FROM` command, for example: +`-my_cluster:*`: ```esql FROM my-index-000001,cluster*:my-index-000001,-cluster_three:* | LIMIT 10 ``` -To exclude a specific remote index, prefix the index with a minus sign in the `FROM` command, such as `my_cluster:-my_index`: +To exclude a specific remote index, prefix the index with a minus sign in the `FROM` command, such as +`my_cluster:-my_index`: ```esql FROM my-index-000001,cluster*:my-index-*,cluster_three:-my-index-000001 | LIMIT 10 ``` - ## Optional remote clusters [ccq-skip-unavailable-clusters] -```{applies_to} -stack: ga 9.1 -``` -If the remote cluster is configured with `skip_unavailable: true` (the default setting), the cluster would be set +::::{tab-set} +:::{tab-item} 9.1 +Remote clusters are configured with `skip_unavailable: true` setting by default. With this setting, the cluster would be +set to `skipped` or `partial` status but the query will not fail, if: -* The remote cluster is disconnected from the querying cluster, either before or during the query. -* The remote cluster does not have the requested index. +* The remote cluster is disconnected from the querying cluster, either before or during the query execution. +* The remote cluster does not have the requested index, or it is not accessible due to security settings. * An error happened while processing the query on the remote cluster. -The `partial` status will be used if the remote query was partially successful and some data was returned. +The `partial` status will be used if the remote query was partially successful and some data may have been returned. -This however does not apply to the situation when the remote cluster is missing an index and this is the only index in the query, -or all the indices in the query are missing. For example, the following queries will fail: +Queries will still fail when `skip_unavailable` is set `true`, if none of the specified indices exist. For example, the +following queries will fail: ```esql FROM cluster_one:missing-index | LIMIT 10 @@ -492,13 +554,21 @@ FROM cluster_one:missing-index* | LIMIT 10 FROM cluster_one:missing-index*,cluster_two:missing-index | LIMIT 10 ``` +::: +:::{tab-item} 9.0 +::: +If a remote cluster disconnects from the querying cluster, {{ccs}} for {{esql}} will set it to `skipped` +and continue the query with other clusters, unless the remote cluster's `skip_unavailable` setting is set to `false`, +in which case the query will fail. +:::: + ## Query across clusters during an upgrade [ccq-during-upgrade] -You can still search a remote cluster while performing a rolling upgrade on the local cluster. However, the local coordinating node’s "upgrade from" and "upgrade to" version must be compatible with the remote cluster’s gateway node. +You can still search a remote cluster while performing a rolling upgrade on the local cluster. However, the local +coordinating node’s "upgrade from" and "upgrade to" version must be compatible with the remote cluster’s gateway node. ::::{warning} Running multiple versions of {{es}} in the same cluster beyond the duration of an upgrade is not supported. :::: - For more information about upgrades, see [Upgrading {{es}}](../../../deploy-manage/upgrade/deployment-or-cluster.md). From 20aaf555ae959035b0388e025ca84a7b37e6c6e5 Mon Sep 17 00:00:00 2001 From: Stanislav Malyshev Date: Thu, 26 Jun 2025 16:18:46 -0600 Subject: [PATCH 4/9] Ws --- .../languages/esql-cross-clusters.md | 218 +++++++----------- 1 file changed, 78 insertions(+), 140 deletions(-) diff --git a/explore-analyze/query-filter/languages/esql-cross-clusters.md b/explore-analyze/query-filter/languages/esql-cross-clusters.md index 34386129ca..a61afe6e2a 100644 --- a/explore-analyze/query-filter/languages/esql-cross-clusters.md +++ b/explore-analyze/query-filter/languages/esql-cross-clusters.md @@ -9,40 +9,34 @@ products: - id: elasticsearch --- + + # Use ES|QL across clusters [esql-cross-clusters] + ::::{warning} -{{ccs-cap}} for {{esql}} is in technical preview and may be changed or removed in a future release. Elastic will work to -fix any issues, but features in technical preview are not subject to the support SLA of official GA features. +{{ccs-cap}} for {{esql}} is in technical preview and may be changed or removed in a future release. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features. :::: + With {{esql}}, you can execute a single query across multiple clusters. + ## Prerequisites [esql-ccs-prerequisites] -* {{ccs-cap}} requires remote clusters. To set up remote clusters, see [*Remote - clusters*](../../../deploy-manage/remote-clusters.md). +* {{ccs-cap}} requires remote clusters. To set up remote clusters, see [*Remote clusters*](../../../deploy-manage/remote-clusters.md). + + To ensure your remote cluster configuration supports {{ccs}}, see [Supported {{ccs}} configurations](../../../solutions/search/cross-cluster-search.md#ccs-supported-configurations). - To ensure your remote cluster configuration supports {{ccs}}, - see [Supported {{ccs}} configurations](../../../solutions/search/cross-cluster-search.md#ccs-supported-configurations). +* For full {{ccs}} capabilities, the local and remote cluster must be on the same [subscription level](https://www.elastic.co/subscriptions). +* The local coordinating node must have the [`remote_cluster_client`](../../../deploy-manage/distributed-architecture/clusters-nodes-shards/node-roles.md#remote-node) node role. +* If you use [sniff mode](/deploy-manage/remote-clusters/remote-clusters-self-managed.md#sniff-mode), the local coordinating node must be able to connect to seed and gateway nodes on the remote cluster. -* For full {{ccs}} capabilities, the local and remote cluster must be on the - same [subscription level](https://www.elastic.co/subscriptions). -* The local coordinating node must have the [ - `remote_cluster_client`](../../../deploy-manage/distributed-architecture/clusters-nodes-shards/node-roles.md#remote-node) - node role. -* If you use [sniff mode](/deploy-manage/remote-clusters/remote-clusters-self-managed.md#sniff-mode), the local - coordinating node must be able to connect to seed and gateway nodes on the remote cluster. + We recommend using gateway nodes capable of serving as coordinating nodes. The seed nodes can be a subset of these gateway nodes. - We recommend using gateway nodes capable of serving as coordinating nodes. The seed nodes can be a subset of these - gateway nodes. +* If you use [proxy mode](/deploy-manage/remote-clusters/remote-clusters-self-managed.md#proxy-mode), the local coordinating node must be able to connect to the configured `proxy_address`. The proxy at this address must be able to route connections to gateway and coordinating nodes on the remote cluster. +* {{ccs-cap}} requires different security privileges on the local cluster and remote cluster. See [Configure privileges for {{ccs}}](../../../deploy-manage/remote-clusters/remote-clusters-cert.md#remote-clusters-privileges-ccs) and [*Remote clusters*](../../../deploy-manage/remote-clusters.md). -* If you use [proxy mode](/deploy-manage/remote-clusters/remote-clusters-self-managed.md#proxy-mode), the local - coordinating node must be able to connect to the configured `proxy_address`. The proxy at this address must be able to - route connections to gateway and coordinating nodes on the remote cluster. -* {{ccs-cap}} requires different security privileges on the local cluster and remote cluster. - See [Configure privileges for {{ccs}}](../../../deploy-manage/remote-clusters/remote-clusters-cert.md#remote-clusters-privileges-ccs) - and [*Remote clusters*](../../../deploy-manage/remote-clusters.md). ## Security model [esql-ccs-security-model] @@ -52,11 +46,12 @@ With {{esql}}, you can execute a single query across multiple clusters. * [API key authentication](#esql-ccs-security-model-api-key) ::::{tip} -To check which security model is being used to connect your clusters, run `GET _remote/info`. If you’re using the API -key authentication method, you’ll see the `"cluster_credentials"` key in the response. +To check which security model is being used to connect your clusters, run `GET _remote/info`. If you’re using the API key authentication method, you’ll see the `"cluster_credentials"` key in the response. :::: + + ### TLS certificate authentication [esql-ccs-security-model-certificate] ::::{admonition} Deprecated in 9.0.0. @@ -65,36 +60,24 @@ key authentication method, you’ll see the `"cluster_credentials"` key in the r Use [API key authentication](#esql-ccs-security-model-api-key) instead. :::: -TLS certificate authentication secures remote clusters with mutual TLS. This could be the preferred model when a single -administrator has full control over both clusters. We generally recommend that roles and their privileges be identical -in both clusters. -Refer to [TLS certificate authentication](../../../deploy-manage/remote-clusters/remote-clusters-cert.md) for -prerequisites and detailed setup instructions. +TLS certificate authentication secures remote clusters with mutual TLS. This could be the preferred model when a single administrator has full control over both clusters. We generally recommend that roles and their privileges be identical in both clusters. + +Refer to [TLS certificate authentication](../../../deploy-manage/remote-clusters/remote-clusters-cert.md) for prerequisites and detailed setup instructions. + ### API key authentication [esql-ccs-security-model-api-key] -The following information pertains to using {{esql}} across clusters with the [**API key based security model -**](../../../deploy-manage/remote-clusters/remote-clusters-api-key.md). You’ll need to follow the steps on that page for -the **full setup instructions**. This page only contains additional information specific to {{esql}}. +The following information pertains to using {{esql}} across clusters with the [**API key based security model**](../../../deploy-manage/remote-clusters/remote-clusters-api-key.md). You’ll need to follow the steps on that page for the **full setup instructions**. This page only contains additional information specific to {{esql}}. -API key based cross-cluster search (CCS) enables more granular control over allowed actions between clusters. This may -be the preferred model when you have different administrators for different clusters and want more control over who can -access what data. In this model, cluster administrators must explicitly define the access given to clusters and users. +API key based cross-cluster search (CCS) enables more granular control over allowed actions between clusters. This may be the preferred model when you have different administrators for different clusters and want more control over who can access what data. In this model, cluster administrators must explicitly define the access given to clusters and users. You will need to: -* Create an API key on the **remote cluster** using - the [Create cross-cluster API key](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-create-cross-cluster-api-key) - API or using the [Kibana API keys UI](../../../deploy-manage/api-keys/elasticsearch-api-keys.md). -* Add the API key to the keystore on the **local cluster**, as part of the steps - in [configuring the local cluster](../../../deploy-manage/remote-clusters/remote-clusters-api-key.md#remote-clusters-security-api-key-local-actions). - All cross-cluster requests from the local cluster are bound by the API key’s privileges. +* Create an API key on the **remote cluster** using the [Create cross-cluster API key](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-create-cross-cluster-api-key) API or using the [Kibana API keys UI](../../../deploy-manage/api-keys/elasticsearch-api-keys.md). +* Add the API key to the keystore on the **local cluster**, as part of the steps in [configuring the local cluster](../../../deploy-manage/remote-clusters/remote-clusters-api-key.md#remote-clusters-security-api-key-local-actions). All cross-cluster requests from the local cluster are bound by the API key’s privileges. -Using {{esql}} with the API key based security model requires some additional permissions that may not be needed when -using the traditional query DSL based search. The following example API call creates a role that can query remote -indices using {{esql}} when using the API key based security model. The final privilege, `remote_cluster`, is required -to allow remote enrich operations. +Using {{esql}} with the API key based security model requires some additional permissions that may not be needed when using the traditional query DSL based search. The following example API call creates a role that can query remote indices using {{esql}} when using the API key based security model. The final privilege, `remote_cluster`, is required to allow remote enrich operations. ```console POST /_security/role/remote1 @@ -127,25 +110,14 @@ POST /_security/role/remote1 ``` 1. The `cross_cluster_search` cluster privilege is required for the *local* cluster. -2. Typically, users will have permissions to read both local and remote indices. However, for cases where the role is - intended to ONLY search the remote cluster, the `read` permission is still required for the local cluster. To provide - read access to the local cluster, but disallow reading any indices in the local cluster, the `names` field may be an - empty string. -3. The indices allowed read access to the remote cluster. The - configured [cross-cluster API key](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-create-cross-cluster-api-key) - must also allow this index to be read. -4. The `read_cross_cluster` privilege is always required when using {{esql}} across clusters with the API key based - security model. -5. The remote clusters to which these privileges apply. This remote cluster must be configured with - a [cross-cluster API key](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-create-cross-cluster-api-key) - and connected to the remote cluster before the remote index can be queried. Verify connection using - the [Remote cluster info](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-remote-info) - API. -6. Required to allow remote enrichment. Without this, the user cannot read from the `.enrich` indices on the remote - cluster. The `remote_cluster` security privilege was introduced in version **8.15.0**. - -You will then need a user or API key with the permissions you created above. The following example API call creates a -user with the `remote1` role. +2. Typically, users will have permissions to read both local and remote indices. However, for cases where the role is intended to ONLY search the remote cluster, the `read` permission is still required for the local cluster. To provide read access to the local cluster, but disallow reading any indices in the local cluster, the `names` field may be an empty string. +3. The indices allowed read access to the remote cluster. The configured [cross-cluster API key](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-create-cross-cluster-api-key) must also allow this index to be read. +4. The `read_cross_cluster` privilege is always required when using {{esql}} across clusters with the API key based security model. +5. The remote clusters to which these privileges apply. This remote cluster must be configured with a [cross-cluster API key](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-create-cross-cluster-api-key) and connected to the remote cluster before the remote index can be queried. Verify connection using the [Remote cluster info](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-remote-info) API. +6. Required to allow remote enrichment. Without this, the user cannot read from the `.enrich` indices on the remote cluster. The `remote_cluster` security privilege was introduced in version **8.15.0**. + + +You will then need a user or API key with the permissions you created above. The following example API call creates a user with the `remote1` role. ```console POST /_security/user/remote_user @@ -155,22 +127,20 @@ POST /_security/user/remote_user } ``` -Remember that all cross-cluster requests from the local cluster are bound by the cross cluster API key’s privileges, -which are controlled by the remote cluster’s administrator. +Remember that all cross-cluster requests from the local cluster are bound by the cross cluster API key’s privileges, which are controlled by the remote cluster’s administrator. ::::{tip} -Cross cluster API keys created in versions prior to 8.15.0 will need to replaced or updated to add the new permissions -required for {{esql}} with ENRICH. +Cross cluster API keys created in versions prior to 8.15.0 will need to replaced or updated to add the new permissions required for {{esql}} with ENRICH. :::: + + ## Remote cluster setup [ccq-remote-cluster-setup] Once the security model is configured, you can add remote clusters. -The -following [cluster update settings](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-settings) -API request adds three remote clusters: `cluster_one`, `cluster_two`, and `cluster_three`. +The following [cluster update settings](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-settings) API request adds three remote clusters: `cluster_one`, `cluster_two`, and `cluster_three`. ```console PUT _cluster/settings @@ -201,14 +171,13 @@ PUT _cluster/settings } ``` -1. Since `skip_unavailable` was not set on `cluster_three`, it uses the default of `true`. See - the [Optional remote clusters](#ccq-skip-unavailable-clusters) section for details. +1. Since `skip_unavailable` was not set on `cluster_three`, it uses the default of `true`. See the [Optional remote clusters](#ccq-skip-unavailable-clusters) section for details. + + ## Query across multiple clusters [ccq-from] -In the `FROM` command, specify data streams and indices on remote clusters using the format -`:`. For instance, the following {{esql}} request queries the `my-index-000001` index on a -single remote cluster named `cluster_one`: +In the `FROM` command, specify data streams and indices on remote clusters using the format `:`. For instance, the following {{esql}} request queries the `my-index-000001` index on a single remote cluster named `cluster_one`: ```esql FROM cluster_one:my-index-000001 @@ -225,21 +194,17 @@ FROM my-index-000001,cluster_one:my-index-000001,cluster_two:my-index-000001 | LIMIT 10 ``` -Likewise, this {{esql}} request queries the `my-index-000001` index from all remote clusters (`cluster_one`, -`cluster_two`, and `cluster_three`): +Likewise, this {{esql}} request queries the `my-index-000001` index from all remote clusters (`cluster_one`, `cluster_two`, and `cluster_three`): ```esql FROM *:my-index-000001 | LIMIT 10 ``` + ## Cross-cluster metadata [ccq-cluster-details] -Using the `"include_ccs_metadata": true` option, users can request that ES|QL {{ccs}} responses include metadata about -the search on each cluster (when the response format is JSON). Here we show an example using the async search endpoint. -{{ccs-cap}} metadata is also present in the synchronous search endpoint response when requested. If the search returns -partial results and there are partial shard or remote cluster failures, `_clusters` metadata containing the failures -will be included in the response regardless of the `include_ccs_metadata` parameter. +Using the `"include_ccs_metadata": true` option, users can request that ES|QL {{ccs}} responses include metadata about the search on each cluster (when the response format is JSON). Here we show an example using the async search endpoint. {{ccs-cap}} metadata is also present in the synchronous search endpoint response when requested. If the search returns partial results and there are partial shard or remote cluster failures, `_clusters` metadata containing the failures will be included in the response regardless of the `include_ccs_metadata` parameter. ```console POST /_query/async?format=json @@ -321,28 +286,15 @@ Which returns: ``` 1. How long the entire search (across all clusters) took, in milliseconds. -2. This section of counters shows all possible cluster search states and how many cluster searches are currently in that - state. The clusters can have one of the following statuses: **running**, **successful** (searches on all shards were - successful), **skipped** (the search failed on a cluster marked with `skip_unavailable`=`true`), **failed** (the - search failed on a cluster marked with `skip_unavailable`=`false`) or **partial** (the search - was [interrupted](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-esql) before finishing or has - partially failed). +2. This section of counters shows all possible cluster search states and how many cluster searches are currently in that state. The clusters can have one of the following statuses: **running**, **successful** (searches on all shards were successful), **skipped** (the search failed on a cluster marked with `skip_unavailable`=`true`), **failed** (the search failed on a cluster marked with `skip_unavailable`=`false`) or **partial** (the search was [interrupted](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-esql) before finishing or has partially failed). 3. The `_clusters/details` section shows metadata about the search on each cluster. -4. If you included indices from the local cluster you sent the request to in your {{ccs}}, it is identified as "( - local)". -5. How long (in milliseconds) the search took on each cluster. This can be useful to determine which clusters have - slower response times than others. -6. The shard details for the search on that cluster, including a count of shards that were skipped due to the can-match - phase results. Shards are skipped when they cannot have any matching data and therefore are not included in the full - ES|QL query. -7. The `is_partial` field is set to `true` if the search has partial results for any reason, for example due to partial - shard failures, - failures in remote clusters, or if the async query was stopped by calling - the [async query stop API](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-esql). - -The cross-cluster metadata can be used to determine whether any data came back from a cluster. For instance, in the -query below, the wildcard expression for `cluster-two` did not resolve to a concrete index (or indices). The cluster is, -therefore, marked as *skipped* and the total number of shards searched is set to zero. +4. If you included indices from the local cluster you sent the request to in your {{ccs}}, it is identified as "(local)". +5. How long (in milliseconds) the search took on each cluster. This can be useful to determine which clusters have slower response times than others. +6. The shard details for the search on that cluster, including a count of shards that were skipped due to the can-match phase results. Shards are skipped when they cannot have any matching data and therefore are not included in the full ES|QL query. +7. The `is_partial` field is set to `true` if the search has partial results for any reason, for example due to partial shard failures, +failures in remote clusters, or if the async query was stopped by calling the [async query stop API](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-esql). + +The cross-cluster metadata can be used to determine whether any data came back from a cluster. For instance, in the query below, the wildcard expression for `cluster-two` did not resolve to a concrete index (or indices). The cluster is, therefore, marked as *skipped* and the total number of shards searched is set to zero. ```console POST /_query/async?format=json @@ -408,25 +360,19 @@ Which returns: 2. Indicates that no shards were searched (due to not having any matching indices). 3. Since one of the clusters is skipped, the search result is marked as partial. + + ## Enrich across clusters [ccq-enrich] -Enrich in {{esql}} across clusters operates similarly -to [local enrich](elasticsearch://reference/query-languages/esql/commands/processing-commands.md#esql-enrich). If the -enrich policy and its enrich indices are consistent across all clusters, simply write the enrich command as you would -without remote clusters. In this default mode, {{esql}} can execute the enrich command on either the local cluster or -the remote clusters, aiming to minimize computation or inter-cluster data transfer. Ensuring that the policy exists with -consistent data on both the local cluster and the remote clusters is critical for ES|QL to produce a consistent query -result. +Enrich in {{esql}} across clusters operates similarly to [local enrich](elasticsearch://reference/query-languages/esql/commands/processing-commands.md#esql-enrich). If the enrich policy and its enrich indices are consistent across all clusters, simply write the enrich command as you would without remote clusters. In this default mode, {{esql}} can execute the enrich command on either the local cluster or the remote clusters, aiming to minimize computation or inter-cluster data transfer. Ensuring that the policy exists with consistent data on both the local cluster and the remote clusters is critical for ES|QL to produce a consistent query result. ::::{tip} -Enrich in {{esql}} across clusters using the API key based security model was introduced in version **8.15.0**. Cross -cluster API keys created in versions prior to 8.15.0 will need to replaced or updated to use the new required -permissions. Refer to the example in the [API key authentication](#esql-ccs-security-model-api-key) section. +Enrich in {{esql}} across clusters using the API key based security model was introduced in version **8.15.0**. Cross cluster API keys created in versions prior to 8.15.0 will need to replaced or updated to use the new required permissions. Refer to the example in the [API key authentication](#esql-ccs-security-model-api-key) section. :::: -In the following example, the enrich with `hosts` policy can be executed on either the local cluster or the remote -cluster `cluster_one`. + +In the following example, the enrich with `hosts` policy can be executed on either the local cluster or the remote cluster `cluster_one`. ```esql FROM my-index-000001,cluster_one:my-index-000001 @@ -434,8 +380,7 @@ FROM my-index-000001,cluster_one:my-index-000001 | LIMIT 10 ``` -Enrich with an {{esql}} query against remote clusters only can also happen on the local cluster. This means the below -query requires the `hosts` enrich policy to exist on the local cluster as well. +Enrich with an {{esql}} query against remote clusters only can also happen on the local cluster. This means the below query requires the `hosts` enrich policy to exist on the local cluster as well. ```esql FROM cluster_one:my-index-000001,cluster_two:my-index-000001 @@ -443,11 +388,10 @@ FROM cluster_one:my-index-000001,cluster_two:my-index-000001 | ENRICH hosts ON ip ``` + ### Enrich with coordinator mode [esql-enrich-coordinator] -{{esql}} provides the enrich `_coordinator` mode to force {{esql}} to execute the enrich command on the local cluster. -This mode should be used when the enrich policy is not available on the remote clusters or maintaining consistency of -enrich indices across clusters is challenging. +{{esql}} provides the enrich `_coordinator` mode to force {{esql}} to execute the enrich command on the local cluster. This mode should be used when the enrich policy is not available on the remote clusters or maintaining consistency of enrich indices across clusters is challenging. ```esql FROM my-index-000001,cluster_one:my-index-000001 @@ -461,15 +405,13 @@ Enrich with the `_coordinator` mode usually increases inter-cluster data transfe :::: + + ### Enrich with remote mode [esql-enrich-remote] -{{esql}} also provides the enrich `_remote` mode to force {{esql}} to execute the enrich command independently on each -remote cluster where the target indices reside. This mode is useful for managing different enrich data on each cluster, -such as detailed information of hosts for each region where the target (main) indices contain log events from these -hosts. +{{esql}} also provides the enrich `_remote` mode to force {{esql}} to execute the enrich command independently on each remote cluster where the target indices reside. This mode is useful for managing different enrich data on each cluster, such as detailed information of hosts for each region where the target (main) indices contain log events from these hosts. -In the below example, the `hosts` enrich policy is required to exist on all remote clusters: the `querying` cluster (as -local indices are included), the remote cluster `cluster_one`, and `cluster_two`. +In the below example, the `hosts` enrich policy is required to exist on all remote clusters: the `querying` cluster (as local indices are included), the remote cluster `cluster_one`, and `cluster_two`. ```esql FROM my-index-000001,cluster_one:my-index-000001,cluster_two:my-index-000001 @@ -478,9 +420,7 @@ FROM my-index-000001,cluster_one:my-index-000001,cluster_two:my-index-000001 | LIMIT 10 ``` -A `_remote` enrich cannot be executed after -a [stats](elasticsearch://reference/query-languages/esql/commands/processing-commands.md#esql-stats-by) command. The -following example would result in an error: +A `_remote` enrich cannot be executed after a [stats](elasticsearch://reference/query-languages/esql/commands/processing-commands.md#esql-stats-by) command. The following example would result in an error: ```esql FROM my-index-000001,cluster_one:my-index-000001,cluster_two:my-index-000001 @@ -490,11 +430,10 @@ FROM my-index-000001,cluster_one:my-index-000001,cluster_two:my-index-000001 | LIMIT 10 ``` + ### Multiple enrich commands [esql-multi-enrich] -You can include multiple enrich commands in the same query with different modes. {{esql}} will attempt to execute them -accordingly. For example, this query performs two enriches, first with the `hosts` policy on any cluster and then with -the `vendors` policy on the local cluster. +You can include multiple enrich commands in the same query with different modes. {{esql}} will attempt to execute them accordingly. For example, this query performs two enriches, first with the `hosts` policy on any cluster and then with the `vendors` policy on the local cluster. ```esql FROM my-index-000001,cluster_one:my-index-000001,cluster_two:my-index-000001 @@ -503,8 +442,7 @@ FROM my-index-000001,cluster_one:my-index-000001,cluster_two:my-index-000001 | LIMIT 10 ``` -A `_remote` enrich command can’t be executed after a `_coordinator` enrich command. The following example would result -in an error. +A `_remote` enrich command can’t be executed after a `_coordinator` enrich command. The following example would result in an error. ```esql FROM my-index-000001,cluster_one:my-index-000001,cluster_two:my-index-000001 @@ -513,24 +451,24 @@ FROM my-index-000001,cluster_one:my-index-000001,cluster_two:my-index-000001 | LIMIT 10 ``` + ## Excluding clusters or indices from {{esql}} query [ccq-exclude] -To exclude an entire cluster, prefix the cluster alias with a minus sign in the `FROM` command, for example: -`-my_cluster:*`: +To exclude an entire cluster, prefix the cluster alias with a minus sign in the `FROM` command, for example: `-my_cluster:*`: ```esql FROM my-index-000001,cluster*:my-index-000001,-cluster_three:* | LIMIT 10 ``` -To exclude a specific remote index, prefix the index with a minus sign in the `FROM` command, such as -`my_cluster:-my_index`: +To exclude a specific remote index, prefix the index with a minus sign in the `FROM` command, such as `my_cluster:-my_index`: ```esql FROM my-index-000001,cluster*:my-index-*,cluster_three:-my-index-000001 | LIMIT 10 ``` + ## Optional remote clusters [ccq-skip-unavailable-clusters] ::::{tab-set} @@ -564,11 +502,11 @@ in which case the query will fail. ## Query across clusters during an upgrade [ccq-during-upgrade] -You can still search a remote cluster while performing a rolling upgrade on the local cluster. However, the local -coordinating node’s "upgrade from" and "upgrade to" version must be compatible with the remote cluster’s gateway node. +You can still search a remote cluster while performing a rolling upgrade on the local cluster. However, the local coordinating node’s "upgrade from" and "upgrade to" version must be compatible with the remote cluster’s gateway node. ::::{warning} Running multiple versions of {{es}} in the same cluster beyond the duration of an upgrade is not supported. :::: + For more information about upgrades, see [Upgrading {{es}}](../../../deploy-manage/upgrade/deployment-or-cluster.md). From 60c591ff9c050d6502e398bab1888ce432c7f97b Mon Sep 17 00:00:00 2001 From: Stanislav Malyshev Date: Thu, 26 Jun 2025 16:24:30 -0600 Subject: [PATCH 5/9] ws --- explore-analyze/query-filter/languages/esql-cross-clusters.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/explore-analyze/query-filter/languages/esql-cross-clusters.md b/explore-analyze/query-filter/languages/esql-cross-clusters.md index a61afe6e2a..736acefa78 100644 --- a/explore-analyze/query-filter/languages/esql-cross-clusters.md +++ b/explore-analyze/query-filter/languages/esql-cross-clusters.md @@ -472,6 +472,7 @@ FROM my-index-000001,cluster*:my-index-*,cluster_three:-my-index-000001 ## Optional remote clusters [ccq-skip-unavailable-clusters] ::::{tab-set} + :::{tab-item} 9.1 Remote clusters are configured with `skip_unavailable: true` setting by default. With this setting, the cluster would be set @@ -491,8 +492,8 @@ FROM cluster_one:missing-index | LIMIT 10 FROM cluster_one:missing-index* | LIMIT 10 FROM cluster_one:missing-index*,cluster_two:missing-index | LIMIT 10 ``` - ::: + :::{tab-item} 9.0 ::: If a remote cluster disconnects from the querying cluster, {{ccs}} for {{esql}} will set it to `skipped` From e28d295e322c2905fde3d4986be73c37c9ec3236 Mon Sep 17 00:00:00 2001 From: Stanislav Malyshev Date: Thu, 26 Jun 2025 16:39:45 -0600 Subject: [PATCH 6/9] fix syntax --- explore-analyze/query-filter/languages/esql-cross-clusters.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/explore-analyze/query-filter/languages/esql-cross-clusters.md b/explore-analyze/query-filter/languages/esql-cross-clusters.md index 736acefa78..bc941a1585 100644 --- a/explore-analyze/query-filter/languages/esql-cross-clusters.md +++ b/explore-analyze/query-filter/languages/esql-cross-clusters.md @@ -495,10 +495,11 @@ FROM cluster_one:missing-index*,cluster_two:missing-index | LIMIT 10 ::: :::{tab-item} 9.0 -::: If a remote cluster disconnects from the querying cluster, {{ccs}} for {{esql}} will set it to `skipped` and continue the query with other clusters, unless the remote cluster's `skip_unavailable` setting is set to `false`, in which case the query will fail. +::: + :::: ## Query across clusters during an upgrade [ccq-during-upgrade] From c0a615bb766b5753f5f765fcf8f4097024bb6fa6 Mon Sep 17 00:00:00 2001 From: Stanislav Malyshev Date: Fri, 27 Jun 2025 12:31:29 -0600 Subject: [PATCH 7/9] Update explore-analyze/query-filter/languages/esql-cross-clusters.md Co-authored-by: Liam Thompson <32779855+leemthompo@users.noreply.github.com> --- explore-analyze/query-filter/languages/esql-cross-clusters.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/explore-analyze/query-filter/languages/esql-cross-clusters.md b/explore-analyze/query-filter/languages/esql-cross-clusters.md index bc941a1585..6ca60a578c 100644 --- a/explore-analyze/query-filter/languages/esql-cross-clusters.md +++ b/explore-analyze/query-filter/languages/esql-cross-clusters.md @@ -474,9 +474,7 @@ FROM my-index-000001,cluster*:my-index-*,cluster_three:-my-index-000001 ::::{tab-set} :::{tab-item} 9.1 -Remote clusters are configured with `skip_unavailable: true` setting by default. With this setting, the cluster would be -set -to `skipped` or `partial` status but the query will not fail, if: +Remote clusters are configured with the `skip_unavailable: true` setting by default. With this setting, clusters are marked as `skipped` or `partial` rather than causing queries to fail in the following scenarios: * The remote cluster is disconnected from the querying cluster, either before or during the query execution. * The remote cluster does not have the requested index, or it is not accessible due to security settings. From b0a78f3985963792ec5c0633bcf3f3b40dcaa1e9 Mon Sep 17 00:00:00 2001 From: Stanislav Malyshev Date: Tue, 1 Jul 2025 10:36:19 -0600 Subject: [PATCH 8/9] Update explore-analyze/query-filter/languages/esql-cross-clusters.md Co-authored-by: Liam Thompson <32779855+leemthompo@users.noreply.github.com> --- explore-analyze/query-filter/languages/esql-cross-clusters.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/explore-analyze/query-filter/languages/esql-cross-clusters.md b/explore-analyze/query-filter/languages/esql-cross-clusters.md index 6ca60a578c..2b13d22ed2 100644 --- a/explore-analyze/query-filter/languages/esql-cross-clusters.md +++ b/explore-analyze/query-filter/languages/esql-cross-clusters.md @@ -480,7 +480,7 @@ Remote clusters are configured with the `skip_unavailable: true` setting by defa * The remote cluster does not have the requested index, or it is not accessible due to security settings. * An error happened while processing the query on the remote cluster. -The `partial` status will be used if the remote query was partially successful and some data may have been returned. +The `partial` status means the remote query either has errors or was interrupted by an explicit user action, but some data may be returned. Queries will still fail when `skip_unavailable` is set `true`, if none of the specified indices exist. For example, the following queries will fail: From 14eabb9f93bfadfda99c7951167d3549452112ea Mon Sep 17 00:00:00 2001 From: Stanislav Malyshev Date: Tue, 1 Jul 2025 10:39:54 -0600 Subject: [PATCH 9/9] refine skip-un section --- explore-analyze/query-filter/languages/esql-cross-clusters.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/explore-analyze/query-filter/languages/esql-cross-clusters.md b/explore-analyze/query-filter/languages/esql-cross-clusters.md index 2b13d22ed2..e8fefe2c5f 100644 --- a/explore-analyze/query-filter/languages/esql-cross-clusters.md +++ b/explore-analyze/query-filter/languages/esql-cross-clusters.md @@ -469,7 +469,9 @@ FROM my-index-000001,cluster*:my-index-*,cluster_three:-my-index-000001 ``` -## Optional remote clusters [ccq-skip-unavailable-clusters] +## Skipping problematic remote clusters [ccq-skip-unavailable-clusters] + +{{ccs-cap}} for {{esql}} behavior when there are problems connecting to or running query on remote clusters differs between versions. ::::{tab-set}