OCPBUGS-57456: podman-etcd should keep the container for crash debugging #2062
Conversation
Replace embedded configuration arguments with external files: - etcd config file (/var/lib/etcd/config.yaml) - podman env file (/var/lib/etcd/env.yaml) This allows configuration changes to be applied via container restart rather than requiring full container recreation.
This commit modifies the podman-etcd resource agent to conditionally reuse existing containers, preventing their removal and preserving historical logs. The agent now: * Compares etcd-pod.yaml changes to decide whether to reuse an existing container or create a new one. * Retains containers after stops to ensure logs are always available for debugging.
No need to move further in the podman_start function if a container is already running. Signed-off-by: Carlo Lobrano <[email protected]>
|
Can one of the admins check and authorise this run please: https://ci.kronosnet.org/job/resource-agents/job/resource-agents-pipeline/job/PR-2062/1/input |
|
The existing container is deleted right before starting a new one. I could improve it to keep the last one only |
|
Can one of the admins check and authorise this run please: https://ci.kronosnet.org/job/resource-agents/job/resource-agents-pipeline/job/PR-2062/2/input |
clobrano
changed the title
|
As this PR also introduces configuration files for podman (env.yaml) and etcd (config.yaml), I also need to backup those files together with the podman container, otherwise we will be unable to check what data etcd is started with. /hold |
heartbeat/podman-etcd
Outdated
| FORCE_NEW_CLUSTER=false | ||
| fi | ||
|
|
||
| cat > "$OCF_RESKEY_podman_env_file" << EOF |
There was a problem hiding this comment.
For the collection of commented-out options, are these things we're leaving here because we expect to enable them down the line?
There was a problem hiding this comment.
Those were leftovers I forgot to remove. However I noticed that some env variables (e.g. etcd_data) were not correctly used, and, thinking more about it, the env file for podman is not necessary anyway, as the only thing that can really change is the etcd command line. This to say that I need to push a version without podman env file :)
clobrano
force-pushed
the
ocpbugs-57456-tnf-podman-etcd-keep-container-for-crash-debugging
branch
from
48e121c to
d5da5f6
Compare
|
Can one of the admins check and authorise this run please: https://ci.kronosnet.org/job/resource-agents/job/resource-agents-pipeline/job/PR-2062/3/input |
clobrano
force-pushed
the
ocpbugs-57456-tnf-podman-etcd-keep-container-for-crash-debugging
branch
from
d5da5f6 to
e6f68d0
Compare
|
Can one of the admins check and authorise this run please: https://ci.kronosnet.org/job/resource-agents/job/resource-agents-pipeline/job/PR-2062/4/input |
clobrano
force-pushed
the
ocpbugs-57456-tnf-podman-etcd-keep-container-for-crash-debugging
branch
from
e6f68d0 to
4c021cc
Compare
When recreating the container due to configuration changes, archive the
current stopped container and its config file by renaming them to
`${CONTAINER}-previous` and `/var/lib/etcd/config-previous.yaml`. This
preserves the container state for debugging while the new instance is
created.
Falls back to deletion if archiving fails. Only one archived copy is
maintained to limit disk usage.
Note: the use of an environment file for podman was removed as it is
an unnecessary complication. If the environment changes we are forced to
create a new container anyway.
|
Can one of the admins check and authorise this run please: https://ci.kronosnet.org/job/resource-agents/job/resource-agents-pipeline/job/PR-2062/5/input |
clobrano
force-pushed
the
ocpbugs-57456-tnf-podman-etcd-keep-container-for-crash-debugging
branch
from
4c021cc to
1353f02
Compare
|
Can one of the admins check and authorise this run please: https://ci.kronosnet.org/job/resource-agents/job/resource-agents-pipeline/job/PR-2062/6/input |
|
Let's keep it on hold again. Notice this line from etcd that should be investigated more |
|
/hold |
This change modifies the
podman-etcdresource agent to conditionally reuse existing containers, preventing their removal and preserving historical logs.As preliminary change, it replaces inline configuration arguments with external file:
This allows configuration changes to be applied via container restart rather than requiring full container recreation.
The agent can then
pod.yamlmanifest to decide whether to reuse an existing container or create a new one.-previoussuffix, together with the correspondentconfig.yaml, renamedconfig-previous.yaml, to allow further debugging.