Cluster Service Didn’t Work After Switching to External ETCD

[prasadkris]

We are managing a handful of PostgreSQL clusters using the Zalando operator. Generally, it functions well 👍🏻 but we occasionally encounter problems during brief disruptions with the kubernetes API service. These disruptions leads to adverse effects in the clusters, impacting our apps and sometime requires a postgresql cluster restart to fix issues. We get the ERROR: Error communicating with DCS message in the cluster's when this happens and I believe this is because Zalando uses the Kubernetes API as a Distributed Config Store (DCS) by default, which can cause problems during connectivity issues with the Kubernetes API service as described in postgres-operator/issues/354, postgres-operator/issues/1703

In order to get rid of this, we have decided to switch to an external etcd for the operator and have updated the PostgreSQL operator deployment with the etcd_host: environment variable.

  # etcd connection string for Patroni. Empty uses K8s-native DCS.
  etcd_host: "etcd.postgres-operator.svc.cluster.local"

This worked and performed a rolling restart of the existing clusters, but our applications were unable to connect to the database service afterward 🙁

We noticed the following errors in the operator logs (the operator logs are available here), and upon investigation, I found that the master service is somehow missing the endpoint, even though there is a pod with spilo-role=master label.

time="2023-12-19T09:30:33Z" level=warning msg="could not connect to Postgres database: dial tcp 10.111.90.85:5432: i/o timeout" cluster-name=test/ops-sentry-postgresql pkg=cluster
time="2023-12-19T09:30:48Z" level=warning msg="could not connect to Postgres database: dial tcp 10.111.90.85:5432: i/o timeout" cluster-name=test/ops-sentry-postgresql pkg=cluster
time="2023-12-19T09:31:03Z" level=warning msg="could not connect to Postgres database: dial tcp 10.111.90.85:5432: i/o timeout" cluster-name=test/ops-sentry-postgresql pkg=cluster
time="2023-12-19T09:31:18Z" level=warning msg="could not connect to Postgres database: dial tcp 10.111.90.85:5432: i/o timeout" cluster-name=test/ops-sentry-postgresql pkg=cluster

Name:              ops-sentry-postgresql
Namespace:         test
Labels:            application=spilo
                   cluster-name=ops-sentry-postgresql
                   spilo-role=master
                   team=ops-sentry
Annotations:       <none>
Selector:          <none>
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                10.111.90.85
IPs:               10.111.90.85
Port:              postgresql  5432/TCP
TargetPort:        5432/TCP
Endpoints:         <none>
Session Affinity:  None
Events:            <none>

kgpo -l spilo-role=master
NAME                      READY   STATUS    RESTARTS   AGE
ops-sentry-postgresql-0   2/2     Running   0          46m

The only way to fix this is either to roll back to the k8s native DCS or manually delete the database services and then restart the postgresql-operator deployment. This will recreate the database service with the correct endpoint IP pointing to the master pod

kubectl delete svc ops-sentry-postgresql 
kubectl -n postgres-operator rollout restart deployment postgres-operator

• Which image of the operator are you using?: registry.opensource.zalan.do/acid/postgres-operator:v1.10.0
• Where do you run it: Bare Metal K8s
• Are you running Postgres Operator in production?: Yes
• Type of issue?: Bug report

Any help with this would be much appreciated. We can definitely switch to external etcd and use the workaround (manually delete the database services and then restart the postgresql-operator deployment), but that will involve downtime, which is not desirable. Please feel free to let me know if you guys need any further details. Thanks! 🙏🏻

[FxKu]

I would recommend to you to enable Patroni failsafe mode. We use K8s as our DCS as well and face outages of it's API server from time to time. Without failsafe mode Patroni demotes the primaries. But when enabled, database stays online and apps continue to work.

Nevertheless, if we find time we should also investigate why switching to another DCS requires manual actions (at least there is a workaround).

[PedroCapdevila]

@FxKu, could you check this? We're encountering a similar issue.

In our case, the master service endpoint is pointing to the correct master pod IP, but it isn’t fully configured:

Master endpoint using external ETCD:

apiVersion: v1
kind: Endpoints
metadata:
  creationTimestamp: "2024-07-25T18:48:23Z"
  labels:
    application: spilo
    cluster-name: primary-postgres-cluster
    spilo-role: master
    team: primary
  name: primary-postgres-cluster
  namespace: app-postgres
  resourceVersion: "60436156"
  uid: c64c2804-3bed-4d79-9fd0-57563615ae11
subsets:
- addresses:
  - ip: 10.233.69.79
  ports:
  - name: postgresql
    port: 5432
    protocol: TCP

Master endpoint using native K8S API as DCS:

apiVersion: v1
kind: Endpoints
metadata:
  annotations:
    acquireTime: "2024-07-25T13:19:01.516723+00:00"
    leader: primary-postgres-cluster-1
    optime: "855639808"
    renewTime: "2024-07-25T18:53:01.678379+00:00"
    transitions: "1"
    ttl: "30"
  creationTimestamp: "2024-07-24T21:05:22Z"
  labels:
    application: spilo
    cluster-name: primary-postgres-cluster
    spilo-role: master
    team: primary
  name: primary-postgres-cluster
  namespace: app-postgres-2
  resourceVersion: "60436846"
  uid: 075f64a6-03da-4b2d-a1de-db8177fb67f5
subsets:
- addresses:
  - hostname: primary-postgres-cluster-1
    ip: 10.233.125.29
    nodeName: k8s-uat-worker02.dev.primary
    targetRef:
      kind: Pod
      name: primary-postgres-cluster-1
      namespace: app-postgres-2
      resourceVersion: "60377441"
      uid: 4c653b3b-eb59-49dc-9384-06804622b014
  ports:
  - name: postgresql
    port: 5432
    protocol: TCP

In this case, we can connect our apps to the database service, but we can't connect to the database externally using a LoadBalancer.