test failed in CI: `helios/deploy` (omdb permission error with zpool)

[bnaecker]

This test failed on a CI run on #8639 : https://github.com/oxidecomputer/omicron/pull/8639/checks?check_run_id=46275285185

Log showing the specific test failure: https://buildomat.eng.oxide.computer/wg/0/details/01K0F6KE4KR1W3G287K78J3NT2/eoGGAXXTlOvYKGZQPrrk6eBQ1lhiVpQvVq6II3yD03tvfVKb/01K0F6M43EAVBFYBDWMMKVGSAA#S1504

Excerpt from the log showing the failure:

2025-07-18T19:44:35.648Z    ++ pfexec zlogin oxz_switch /opt/oxide/omdb/bin/omdb db zpool list -i
2025-07-18T19:44:35.648Z    ++ wc -l
2025-07-18T19:44:35.780Z    note: database URL not specified.  Will search DNS.
2025-07-18T19:44:35.780Z    note: (override with --db-url or OMDB_DB_URL)
2025-07-18T19:44:35.780Z    note: using DNS server for subnet fd00:1122:3344::/48
2025-07-18T19:44:35.780Z    note: (if this is not right, use --dns-server to specify an alternate DNS server)
2025-07-18T19:44:35.804Z    note: using database URL postgresql://root@[fd00:1122:3344:101::4]:32221,[fd00:1122:3344:101::3]:32221,[fd00:1122:3344:101::7]:32221,[fd00:1122:3344:101::6]:32221,[fd00:1122:3344:101::5]:32221/omicron?sslmode=disable
2025-07-18T19:44:35.804Z    note: database schema version matches expected (165.0.0)
2025-07-18T19:44:35.941Z    Error: no latest inventory found!
2025-07-18T19:44:35.965Z    + ACTUAL_ZPOOL_COUNT='       0'
2025-07-18T19:44:35.965Z    + _exit_trap
2025-07-18T19:44:35.965Z    + local status=1
2025-07-18T19:44:35.965Z    + set +o errexit
2025-07-18T19:44:35.965Z    + [[ x != \x ]]
2025-07-18T19:44:35.965Z    + pfexec svcadm -Z disable -s cron
2025-07-18T19:44:37.879Z    + [[ 1 -eq 0 ]]
2025-07-18T19:44:37.879Z    + set -o xtrace
2025-07-18T19:44:37.879Z    + banner evidence

[askfongjojo]

There seems to be a permission issue with sleds, zpools, and perhaps other objects too. On dublin where I deployed commit 356e0b3:

root@oxz_switch0:~# /opt/oxide/omdb/bin/omdb db zpool list
Error: Forbidden

root@oxz_switch0:~# omdb db sleds
Error: listing sleds

Caused by:
    Forbidden

root@oxz_switch0:~# omdb db instances
Error: failed to look up sled

Caused by:
    Object (of type ById(6569d993-44cd-4791-b8f2-f59164bd7aac)) not found: sled

root@oxz_switch0:~# omdb db disks list
ID                                   SIZE   STATE    NAME                 ATTACHED_TO                          
5d77b663-8e97-462e-8811-431a31777ba6 4 GiB  detached noble-boot           -                                    
90621ce2-a859-4b96-aa2a-f995b8113ff5 10 GiB attached test-ubuntu24-8d0273 caa28082-77c1-4fea-ae11-c1d2e217fe87 

root@oxz_switch0:~# omdb db disks info 5d77b663-8e97-462e-8811-431a31777ba6
HOST_SERIAL DISK_NAME  INSTANCE_NAME PROPOLIS_ZONE VOLUME_ID                            DISK_STATE 
-           noble-boot -             -             6080bb74-bc2e-4c9a-a8c6-f16baa79e495 detached   
Error: failed to look up zpool

Caused by:
    Object (of type ById(f6cf0821-906f-4111-8130-ef67ec758580)) not found: zpool

[jgallagher]

These seem like two different (and unrelated?) issues to me, I think? The error @bnaecker originally reported is

Error: no latest inventory found!

which sounds like a typical CI race: we handed off to Nexus, but didn't wait for an inventory collection to complete before trying to run commands that need it.

The error you're seeing on dublin is long after this, and presumably there are inventory collections? That implies something different (and worse) is happening - maybe worth moving that to a new issue?

[askfongjojo]

Sure, I broke the default privileged user permissions while testing something in dublin. Please ignore the comment I put above.