Inconsistent behavior for all shards failure between local and remote clusters

[smalyshev]

Elasticsearch Version

8.18, 9.0

Installed Plugins

No response

Java Version

bundled

OS Version

Problem Description

When all shards fail on a cluster, ES returns an error even when allow_partial_search_results=true. However, if two clusters are involved, the behavior is inconsistent. If all shards on the remote fail, the result is partial with the non-failing results from local and remote set to skipped. However, if all shards on the local fail (but remote succeeds) the result is a failure and the remote results are ignored.

Note this only happens on MRT=true but MRT=false works as expected.

Steps to Reproduce

1. Set up two clusters, local and remote1
2. Create index "blogs" on both, preferably with multiple shards
3. Run this query:

POST /blogs,remote1:blogs/_search?allow_partial_search_results=true&ccs_minimize_roundtrips=true
{
       "size": 0,
       "query": {
         "error_query": {
           "indices": [
           {
             "name": "blogs",             
             "error_type": "exception",
             "message": "local cluster exception"
           }
         ]
         }
       },
     "aggs": {
       "indexgroup": {
          "terms": {
          "field": "_index"
          }
       }
      }
    }

4. Change index name to remote1:blogs and try again.

Logs (if relevant)

No response

[smalyshev]

@quux00 let's discuss this

[elasticsearchmachine]

Pinging @elastic/es-search-foundations (Team:Search Foundations)