[improve][broker] Deny removing local cluster from topic level replicated cluster policy

[poorbarcode]

Motivation

• When a topic detects that its cluster policies do not contain the local cluster, the topic will delete itself.
• This mechanism is designed for the following 3 scenarios

Scenario 1

• The user has two clusters enabled Geo-Replication with a global metadata store.
  • Namespace-level replicated cluster policy is shared by the global metadata store.
• The user wants to turn offline a cluster or split clusters by different namespaces, then remove the cluster that he wants to turn offline.
• The cluster that he wants to turn offline will delete all topics under the namespace.

Scenario 2
Set a topic-level replicated clusters that do not contain local cluster is not meaningful, except the following scenario: User has two clusters, which enabled Geo-Replication through a global metadata store, the resources named partitioned topic metadata and the resource namespace-level "replicated clusters" are shared between multi clusters. Pulsar can hardly delete a specify partitioned topic. To support this use case, the following steps can implement it:

1. set a global topic-level replicated clusters that do not contain local cluster.
2. the local cluster will remove the subtopics automatically, and remove the schemas and local topic policies. Just leave the global topic policies there, which prevents the namespace level replicated clusters policy taking affect.

TODO But the API "pulsar-admin topics set-replication-clusters" does not support global policy, to support this scenario, a PIP is needed.

Scenario 3

• The user has two clusters enabled Geo-Replication with a global metadata store.
• The user wants to delete topics under a namespace or delete a partitioned topic.
• The first step he needs to take is to disable Geo-Replication.
  • cluster-A will delete the topics/topic automatically.
• The user can call pulsar-admin topics delete-partitioned-topic <topic> on cluster-B.

Modifications

• Since no scenario needs to remove the local cluster from the topic-level replicated cluster policy, deny it.
• Add test cases
  • Manually modifying topic policies by the Rest API.
    • Global topic-level policies:
      • Add: replicate
      • Update: replicate
      • Delete a single policy(it is equivalent to specifying updating): delete both local and remote policies.
    • Local topic-level policies:
      • Add: never replicate
      • Update: never replicate
      • Delete a single policy(it is equivalent to specifying updating): delete local policies only.
    • Delete Topic triggers that both local and global policies will be deleted in the local cluster, but will not delete the remote cluster's global policies.

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

Matching PR in forked repository

PR in forked repository: x

[lhotari]

LGTM