Enable defaulting of nova in kuttle job #921

mrkisaolamb · 2025-02-05T10:50:26Z

Depends-On: openstack-k8s-operators/ci-framework#2735
Depends-On: #927
close: https://issues.redhat.com/browse/OSPRH-14126

softwarefactory-project-zuul · 2025-02-05T13:47:00Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/231ce90da53b4df8960ae8113279286b

✔️ openstack-meta-content-provider SUCCESS in 2h 55m 34s
❌ nova-operator-kuttl RETRY_LIMIT in 30m 30s
✔️ nova-operator-tempest-multinode SUCCESS in 2h 25m 47s
✔️ nova-operator-tempest-multinode-ceph SUCCESS in 2h 38m 15s

softwarefactory-project-zuul · 2025-02-10T13:45:48Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/d204e1b3b52f4e2f9da2ea9db4fbd04c

✔️ openstack-meta-content-provider SUCCESS in 2h 54m 11s
❌ nova-operator-kuttl RETRY_LIMIT in 31m 22s
✔️ nova-operator-tempest-multinode SUCCESS in 2h 36m 36s
❌ nova-operator-tempest-multinode-ceph FAILURE in 1h 27m 30s

softwarefactory-project-zuul · 2025-02-11T15:39:47Z

Zuul encountered a syntax error while parsing its
configuration in the repo openstack-k8s-operators/nova-operator on branch main. The
problem was:

extra keys not allowed @ data['post_deploy']

The problem appears in the the "nova-operator-kuttl" job stanza:

job:
name: nova-operator-kuttl
parent: cifmw-multinode-kuttl-operator-target
dependencies: ["openstack-meta-content-provider"]
roles:
- zuul: github.com/openstack-k8s-operators/ci-framework
description: |
This job deploy a basic "Compute Starter Kit" topology
https://www.openstack.org/software/sample-configs#compute-starter-kit
...

in "openstack-k8s-operators/nova-operator/.zuul.yaml@main", line 2

softwarefactory-project-zuul · 2025-02-11T15:44:19Z

Zuul encountered a syntax error while parsing its
configuration in the repo openstack-k8s-operators/nova-operator on branch main. The
problem was:

Invalid Ansible variable name 'post-run' for dictionary value @ data['vars']

The problem appears in the the "nova-operator-kuttl" job stanza:

job:
name: nova-operator-kuttl
parent: cifmw-multinode-kuttl-operator-target
dependencies: ["openstack-meta-content-provider"]
roles:
- zuul: github.com/openstack-k8s-operators/ci-framework
description: |
This job deploy a basic "Compute Starter Kit" topology
https://www.openstack.org/software/sample-configs#compute-starter-kit
...

in "openstack-k8s-operators/nova-operator/.zuul.yaml@main", line 2

softwarefactory-project-zuul · 2025-02-11T20:42:03Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/50f46ddde56e4db2bb5508e47e5c46d2

✔️ openstack-meta-content-provider SUCCESS in 3h 54m 57s
❌ nova-operator-kuttl RETRY_LIMIT in 21m 38s
✔️ nova-operator-tempest-multinode SUCCESS in 2h 23m 30s
✔️ nova-operator-tempest-multinode-ceph SUCCESS in 2h 41m 10s

mrkisaolamb · 2025-02-12T16:24:06Z

/retest

softwarefactory-project-zuul · 2025-02-12T20:50:25Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/ea87f35769074a5fb4cdff8ba7c7d70e

✔️ openstack-meta-content-provider SUCCESS in 5h 39m 34s
❌ nova-operator-kuttl FAILURE in 36m 02s
✔️ nova-operator-tempest-multinode SUCCESS in 2h 09m 58s
✔️ nova-operator-tempest-multinode-ceph SUCCESS in 2h 51m 48s

softwarefactory-project-zuul · 2025-02-13T12:53:32Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/355c453b32624f82acb9bd721bb030fc

✔️ openstack-meta-content-provider SUCCESS in 3h 20m 57s
❌ nova-operator-kuttl FAILURE in 34m 58s
❌ nova-operator-tempest-multinode FAILURE in 1h 24m 09s
✔️ nova-operator-tempest-multinode-ceph SUCCESS in 2h 48m 58s

softwarefactory-project-zuul · 2025-02-13T18:56:18Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/2d9be82c258b4f9ab501aa6d44a85fde

✔️ openstack-meta-content-provider SUCCESS in 3h 52m 12s
❌ nova-operator-kuttl POST_FAILURE in 33m 40s
✔️ nova-operator-tempest-multinode SUCCESS in 2h 19m 50s
✔️ nova-operator-tempest-multinode-ceph SUCCESS in 2h 56m 21s

softwarefactory-project-zuul · 2025-02-14T17:46:25Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/59e573377d4e43fb9056be89bf5a4ac2

✔️ openstack-meta-content-provider SUCCESS in 2h 38m 01s
❌ nova-operator-kuttl FAILURE in 35m 33s
✔️ nova-operator-tempest-multinode SUCCESS in 2h 17m 00s
❌ nova-operator-tempest-multinode-ceph FAILURE in 1h 34m 49s

softwarefactory-project-zuul · 2025-02-15T15:23:25Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/29b60a0ca6ec4aa2aa83ee4a0ba5715a

✔️ openstack-meta-content-provider SUCCESS in 1h 42m 07s
❌ nova-operator-kuttl FAILURE in 35m 29s
❌ nova-operator-tempest-multinode FAILURE in 1h 21m 18s
❌ nova-operator-tempest-multinode-ceph FAILURE in 1h 24m 48s

softwarefactory-project-zuul · 2025-02-16T14:38:45Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/d94f3c6b1d2b4a5fbd3f1ff5c92deaf8

✔️ openstack-meta-content-provider SUCCESS in 2h 53m 03s
❌ nova-operator-kuttl FAILURE in 32m 49s
✔️ nova-operator-tempest-multinode SUCCESS in 2h 21m 03s
✔️ nova-operator-tempest-multinode-ceph SUCCESS in 2h 35m 54s

softwarefactory-project-zuul · 2025-02-17T11:01:28Z

This change depends on a change that failed to merge.

Change openstack-k8s-operators/ci-framework#2735 is needed.

mrkisaolamb · 2025-02-18T07:04:23Z

recheck

SeanMooney

/lgtm

i agree with these updates we should merge this once the depnecy is merged

SeanMooney · 2025-02-20T14:10:54Z

test/kuttl/test-suites/default/cell-tests/01-assert.yaml

@@ -138,12 +138,10 @@ metadata:
  name: nova-kuttl-api-0
 status:
  containerStatuses:
-  - image: quay.io/podified-antelope-centos9/openstack-nova-api:current-podified


ah you removed this becuase it may come form the content provider i assume

ya that makes sense to remove

SeanMooney · 2025-02-20T14:15:12Z

im only adding hold to this because the depnecy has not merged

once that is merged feel free to drop the hold lable

mrkisaolamb · 2025-02-20T14:15:45Z

recheck

SeanMooney · 2025-02-20T14:15:58Z

we need to do one more rdo un with rdo-check too because the current one has #927 included
recheck wont work while its still running so we need to wait for it complete first unless you want to just rebase
that will kick it from the queue and retrigger the job

gibizer · 2025-02-20T14:39:25Z

.zuul.yaml

@@ -181,7 +189,8 @@
    github-check:
      jobs:
        - openstack-meta-content-provider
-        - nova-operator-kuttl
+        - nova-operator-kuttl:
+            override-checkout: main


what does this mean? Will this prevent testing the content of the nova-operator PR the job is triggered on?

IIUC https://zuul-ci.org/docs/zuul/latest/config/job.html#attr-job.override-checkout right, that picks the main branches for other involved projects

gibizer · 2025-02-20T14:42:29Z

Makefile

@@ -428,3 +429,11 @@ PHONY: crd-schema-check
 crd-schema-check: manifests
 	INSTALL_DIR=$(LOCALBIN) CRD_SCHEMA_CHECKER_VERSION=$(CRD_SCHEMA_CHECKER_VERSION) hack/build-crd-schema-checker.sh
 	INSTALL_DIR=$(LOCALBIN) BASE_REF="$${PULL_BASE_SHA:-$(BRANCH)}" hack/crd-schema-checker.sh
+
+
+.PHONY: nova


I would use a bit more descriptive name. We have a run target that runs the operator from the git repo. So this would be something like run_with_olm

gibizer · 2025-02-20T14:43:34Z

ci/nova-operator-kuttl/deploy_webhooks.yaml

+  tasks:
+    - name: Scale down deployment
+      ansible.builtin.shell: |
+        oc patch csv -n openstack-operators openstack-operator.v0.0.1 --type json -p="[{"op": "replace", "path": "/spec/install/spec/deployments/0/spec/replicas", "value": "0"}]"


I'm wondering if we can always rely on the csv name being v0.0.1

That is a good question, but CSV versioning questions are in flux as yet. Let's copy what other projects do for the time being.

so IMO we can go with openstack-operator.v0.0.1

gibizer · 2025-02-20T14:45:02Z

ci/nova-operator-kuttl/deploy_webhooks.yaml

+        chdir: "{{ ansible_user_dir }}/src/github.com/openstack-k8s-operators/nova-operator"
+        script: make nova
+        extra_args:
+          CATALOG_IMAGE: "{{ nova_catalog_image | default('quay.io/openstack-k8s-operators/nova-operator-index:latest') }}"


I guess nova_catalog_name is here to ensure that the nova-operator that is built from this PR will be used for the deployment. Who sets this variable?

gibizer · 2025-02-20T14:45:42Z

ci/nova-operator-kuttl/deploy_webhooks.yaml

+        chdir: "{{ ansible_user_dir }}/src/github.com/openstack-k8s-operators/nova-operator"
+        script: make nova
+        extra_args:
+          CATALOG_IMAGE: "{{ nova_catalog_image | default('quay.io/openstack-k8s-operators/nova-operator-index:latest') }}"


The default here seems to be a duplication as CATALOG_IMAGE is already defaulted in the makefile.

softwarefactory-project-zuul · 2025-02-20T15:17:19Z

This change depends on a change that failed to merge.

Change #927 is needed.

gibizer · 2025-02-20T17:25:57Z

Kuttl run looks green https://softwarefactory-project.io/zuul/t/rdoproject.org/build/453f64801ce749e49ef658a399731fa2/log/controller/controller/kuttl-test-run.out . We need to wait for the dependencies to land.

softwarefactory-project-zuul · 2025-02-20T18:42:12Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/f443106e47bd4d09819a39a07df484e6

✔️ openstack-meta-content-provider SUCCESS in 2h 46m 55s
✔️ nova-operator-kuttl SUCCESS in 44m 53s
❌ nova-operator-tempest-multinode FAILURE in 1h 28m 01s
❌ nova-operator-tempest-multinode-ceph FAILURE in 2h 23m 40s

SeanMooney · 2025-02-21T01:55:03Z

in #928 i have taken this pr and added a patch to drop the edpm josb and enable debug on the pipeline i also have removed #921

that should give us more info about why its not working without that.

SeanMooney · 2025-02-21T02:27:48Z

im not 100% sure but i think the actually issue is how we configure implied branches
so we could trigger jobs on opendev master
i belive will fix the issue for fr1 and main for now
#929

and we can then alter how we make the teampels work withopen dev on main
there are other ways to make that towrk i think but i belive that the cause of the current issue with fr1 job content running on main

we are declaring on fr1 that the job definition fr1 is actually for main

https://zuul-ci.org/docs/zuul/latest/config/pragma.html#attr-pragma.implied-branches

https://imgur.com/a/cjtmZL6

zuul should not have 2 job definition with the same name of rht same branch like that
the fr2 variant should have fr2 as the branch

I'm 99% sure that because of

pragma:
implied-branch-matchers: True
implied-branches:
- main
- master

mrkisaolamb · 2025-02-21T09:48:42Z

recheck

softwarefactory-project-zuul · 2025-02-21T12:44:35Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/fd567916c30b454c890cfd3804a211f4

✔️ openstack-meta-content-provider SUCCESS in 2h 54m 31s
✔️ nova-operator-kuttl SUCCESS in 41m 14s
❌ nova-operator-tempest-multinode FAILURE in 1h 24m 29s
✔️ nova-operator-tempest-multinode-ceph SUCCESS in 2h 37m 24s

mrkisaolamb · 2025-02-21T12:58:05Z

recheck

SeanMooney

/lgtm we agreed to adress the underlying issue in a follow up.

ill remove hold when the dependcies are merged

openshift-ci · 2025-02-24T16:50:32Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mrkisaolamb, SeanMooney

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [SeanMooney,mrkisaolamb]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci bot requested review from gibizer and kk7ds February 5, 2025 10:50

openshift-ci bot added the approved label Feb 5, 2025

mrkisaolamb force-pushed the kuttl_fix branch from 18d6ba3 to bae9492 Compare February 11, 2025 15:44

mrkisaolamb force-pushed the kuttl_fix branch 2 times, most recently from 4a6885f to 691568d Compare February 11, 2025 16:45

mrkisaolamb force-pushed the kuttl_fix branch 4 times, most recently from 379b8e1 to 67a7e0b Compare February 12, 2025 14:23

mrkisaolamb force-pushed the kuttl_fix branch from 67a7e0b to ce6dcaf Compare February 13, 2025 09:31

mrkisaolamb force-pushed the kuttl_fix branch from ce6dcaf to 681c104 Compare February 13, 2025 14:32

mrkisaolamb force-pushed the kuttl_fix branch from 681c104 to 85e4d2a Compare February 14, 2025 15:07

mrkisaolamb force-pushed the kuttl_fix branch from 85e4d2a to 04e7d6f Compare February 15, 2025 13:40

mrkisaolamb force-pushed the kuttl_fix branch from 04e7d6f to d4bf23d Compare February 16, 2025 11:44

mrkisaolamb force-pushed the kuttl_fix branch from d4bf23d to b561eb1 Compare February 17, 2025 08:47

cescgina mentioned this pull request Feb 20, 2025

Add hooks before running kuttl from operator target openstack-k8s-operators/ci-framework#2735

Merged

SeanMooney reviewed Feb 20, 2025

View reviewed changes

openshift-ci bot assigned SeanMooney Feb 20, 2025

openshift-ci bot added the lgtm label Feb 20, 2025

SeanMooney added the do-not-merge/hold label Feb 20, 2025

mrkisaolamb force-pushed the kuttl_fix branch from d8ff3d8 to df0a2f8 Compare February 20, 2025 14:19

openshift-ci bot removed the lgtm label Feb 20, 2025

SeanMooney added lgtm do-not-merge/hold and removed do-not-merge/hold labels Feb 20, 2025

gibizer reviewed Feb 20, 2025

View reviewed changes

Fix kuttl jobs by adding webhooks

c2dc4e3

mrkisaolamb force-pushed the kuttl_fix branch from df0a2f8 to c2dc4e3 Compare February 20, 2025 15:45

openshift-ci bot removed the lgtm label Feb 20, 2025

SeanMooney approved these changes Feb 24, 2025

View reviewed changes

openshift-ci bot added the lgtm label Feb 24, 2025

mrkisaolamb removed the do-not-merge/hold label Feb 24, 2025

openshift-merge-bot bot merged commit e1e2ba8 into openstack-k8s-operators:main Feb 24, 2025
7 checks passed

Enable defaulting of nova in kuttle job #921

Enable defaulting of nova in kuttle job #921

Uh oh!

Conversation

mrkisaolamb commented Feb 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

softwarefactory-project-zuul bot commented Feb 5, 2025

Uh oh!

softwarefactory-project-zuul bot commented Feb 10, 2025

Uh oh!

softwarefactory-project-zuul bot commented Feb 11, 2025

Uh oh!

softwarefactory-project-zuul bot commented Feb 11, 2025

Uh oh!

softwarefactory-project-zuul bot commented Feb 11, 2025

Uh oh!

mrkisaolamb commented Feb 12, 2025

Uh oh!

softwarefactory-project-zuul bot commented Feb 12, 2025

Uh oh!

softwarefactory-project-zuul bot commented Feb 13, 2025

Uh oh!

softwarefactory-project-zuul bot commented Feb 13, 2025

Uh oh!

softwarefactory-project-zuul bot commented Feb 14, 2025

Uh oh!

softwarefactory-project-zuul bot commented Feb 15, 2025

Uh oh!

softwarefactory-project-zuul bot commented Feb 16, 2025

Uh oh!

softwarefactory-project-zuul bot commented Feb 17, 2025

Uh oh!

mrkisaolamb commented Feb 18, 2025

Uh oh!

SeanMooney left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SeanMooney commented Feb 20, 2025

Uh oh!

mrkisaolamb commented Feb 20, 2025

Uh oh!

SeanMooney commented Feb 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bogdando Feb 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

softwarefactory-project-zuul bot commented Feb 20, 2025

Uh oh!

gibizer commented Feb 20, 2025

Uh oh!

softwarefactory-project-zuul bot commented Feb 20, 2025

Uh oh!

SeanMooney commented Feb 21, 2025

Uh oh!

SeanMooney commented Feb 21, 2025

mrkisaolamb commented Feb 5, 2025 •

edited

Loading

SeanMooney commented Feb 20, 2025 •

edited

Loading

bogdando Feb 20, 2025 •

edited

Loading