[FLINK-32033][Kubernetes-Operator] Fix Lifecycle Status of FlinkDeployment Resource in case of MISSING/ERROR JM status

[nishita-09]

What is the purpose of the change

Currently the lifecycle state shows STABLE even if an application deployment was deleted and stays in MISSING / reconciling state. This would fix the lifecycle status of the deployment which would be FAILED in cases when the JM is in MISSING/ERROR state with ERROR: configmaps have been deleted indicating TERMINAL error.

Brief change log

• Added logic to detect unrecoverable FlinkDeployment scenarios
• Inserted after existing job status FAILED check
• *Checks for JobManagerStatus:
  • MISSING JobManager Deployment with error ({"type":"org.apache.flink.kubernetes.operator.exception.UpgradeFailureException","message":"HA metadata not available to restore from last state. It is possible that the job has finished or terminally failed, or the configmaps have been deleted. ","additionalMetadata":{},"throwableList":[]}) → Return FAILED lifecycle state
  • MISSING JobManager Deployment (no error or recoverable error) → Return DEPLOYED lifecycle state
  • ERROR JobManager Deployment without terminal error -> Return DEPLOYED lifecycle state

Verifying this change

This change added tests and can be verified as follows:

• Added Unit Tests in ResourceLifecycleMetricsTest to validate Lifecycle Status for flinkDeployment
• Manually verified the change by running a cluster with 3 sets of Application Flink Clusters:
  • Application Deployment 1 -> with invalid image -> ERRORED JobManager Deployment Status -> DEPLOYED Lifecycle Status of flinkDeployment
  • Application Deployment 2 -> with valid image -> Deleted JobManager Deployment -> Caused Missing JM Status -> FAILED Lifecycle Status of flinkDeployment
  • Application Deployment 3 -> with valid image ->with invalid node selector -> Jobmanager Deployment Status: DEPLOYING -> Hence the Lifecycle Status is STABLE

Does this pull request potentially affect one of the following parts:

• Dependencies (does it add or upgrade a dependency): no
• The public API, i.e., is any changes to the CustomResourceDescriptors: no
• Core observer or reconciler logic that is regularly executed: yes

Documentation

• Does this pull request introduce a new feature? no
• If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

[gyfora]

Why are we checking this specific error?
In any case we are the ones triggering this error so please create a constant in the AbstractJobReconciler and use that here

[nishita-09]

@gyfora this seems to be the only case when we know that the cluster cannot recover on its own and needs a manual restore. hence used this. Will set this as a constant instead for a cleaner code.

[nishita-09]

@gyfora

1. There are multiple instances where HA metadata not available is written in different forms like HA metadata not available and HA data is not available. Should we maintain a uniformity in these by changing these exception messages using a constant (now that it is available).

2. Also currently flink-operator-api does not have flink-operator as a dependency -> to use the constants in AbstractJobReconciler we would have to import it as a dependency as the status change logic resides in flink-operator-api.
   Should I still go ahead with this?

[gyfora]

If possible let's use a single constant, and we can keep that constant in the operator api module so the reconciler can use it

[nishita-09]

@gyfora
I have added 3 constants for error messages which are frequently used and would mean that they are terminal, and referenced those in the reconcilers to maintain uniformity. I have also tried to keep the net changes minimum (Although a few error messages would differ slightly). Do let me know if this looks good?