Improve integration tests reliability

[andre-j3sus]

We've observed that the integration tests are occasionally flaky.

In some runs, they fail with errors related to Firecracker vsock communication. Here's one such example:

• https://github.com/vhive-serverless/vHive/actions/runs/15581822659/job/43936640621#step:9:234

A simple fix could be to improve test isolation. Currently, multiple tests share a single firecracker-containerd service, which might be leading to resource conflicts. Instead, we could run a separate instance of firecracker-containerd for each test. This would slightly increase the total test time, but it would likely improve reliability.

[joshjms]

Here's one of the failed run test artifacts for such error:

time="2025-07-07T19:07:33.240445368Z" level=info msg="creating new VM" runtime=aws.firecracker vmID=not-cld-0
time="2025-07-07T19:07:33.240674487Z" level=info msg="Called startVMM(), setting up a VMM on firecracker.sock" runtime=aws.firecracker
time="2025-07-07T19:07:33.252039667Z" level=info msg="refreshMachineConfiguration: [GET /machine-config][200] getMachineConfigurationOK  &{CPUTemplate: MemSizeMib:0xc00019b038 Smt:0xc00019b043 TrackDirtyPages:0xc00019b044 VcpuCount:0xc00019b030}" runtime=aws.firecracker
time="2025-07-07T19:07:33.252182214Z" level=info msg="PutGuestBootSource: [PUT /boot-source][204] putGuestBootSourceNoContent " runtime=aws.firecracker
time="2025-07-07T19:07:33.252192543Z" level=info msg="Attaching drive /var/lib/firecracker-containerd/runtime/default-rootfs.img, slot root_drive, root true." runtime=aws.firecracker
time="2025-07-07T19:07:33.252445537Z" level=info msg="Attached drive /var/lib/firecracker-containerd/runtime/default-rootfs.img: [PUT /drives/{drive_id}][204] putGuestDriveByIdNoContent " runtime=aws.firecracker
time="2025-07-07T19:07:33.252455886Z" level=info msg="Attaching drive /var/lib/firecracker-containerd/shim-base/firecracker-containerd#not-cld-0/ctrstub0, slot MN2HE43UOVRDA, root false." runtime=aws.firecracker
time="2025-07-07T19:07:33.252597251Z" level=info msg="Attached drive /var/lib/firecracker-containerd/shim-base/firecracker-containerd#not-cld-0/ctrstub0: [PUT /drives/{drive_id}][204] putGuestDriveByIdNoContent " runtime=aws.firecracker
time="2025-07-07T19:07:33.252607440Z" level=info msg="Attaching NIC tap0 (hwaddr 06:00:AC:10:00:02) at index 1" runtime=aws.firecracker
time="2025-07-07T19:07:33.260362158Z" level=info msg="startInstance successful: [PUT /actions][204] createSyncActionNoContent " runtime=aws.firecracker
time="2025-07-07T19:07:33.260393546Z" level=info msg="calling agent" runtime=aws.firecracker vmID=not-cld-0
time="2025-07-07T19:07:33.273853857Z" level=warning msg="firecracker exited: exit status 1" runtime=aws.firecracker
time="2025-07-07T19:07:33.361064725Z" level=error attempt=1 error="non-temporary vsock dial failure: failed to dial \"firecracker.vsock\" within 100ms: dial unix firecracker.vsock: connect: connection refused" runtime=aws.firecracker vmID=not-cld-0
time="2025-07-07T19:07:33.361097898Z" level=error msg="failed to create VM" error="failed to dial the VM over vsock: non-temporary vsock dial failure: failed to dial \"firecracker.vsock\" within 100ms: dial unix firecracker.vsock: connect: connection refused" runtime=aws.firecracker vmID=not-cld-0

The logs does seem to indicate a teardown/start race. I don't think we have to run separate instances of firecracker-containerd for each test, rather wait for the teardown to fully finish. I'll try to take a look.

Upd: I think I might be wrong. Also, looking at #1126 (one example being https://github.com/vhive-serverless/vHive/actions/runs/17098853438/job/48489765296?pr=1126), having a new firecracker-containerd instance for each test will still have the flake.

[joshjms]

I cannot reproduce this locally so I'll use the CI to test.

[leokondrashov]

@joshjms can you elaborate on how you arrived at a teardown/start race? In the logs that you provided, there is no concurrent teardown happening.

[joshjms]

It was a hypothesis, since there's this:

time="2025-07-07T19:07:33.273853857Z" level=warning msg="firecracker exited: exit status 1" runtime=aws.firecracker

One of the most probable cause for this is a conflicting resource. This is also supported with the fact that it cannot be reproduced in my local machine leading me to think that there might be slow IO contributing to the race.

I'd like to retract that though since adding a wall time sleep didn't seem to help, therefore I think it'll be helpful to get the firecracker logs first to debug further (exit status 1 isn't really helpful).

In firecracker-runtime.json, we can add the field log_fifo to create a named pipe inside each VM's shim_base directory. Then I'd propose for the Orchestrator to support reading these logs (that is to read the fifo file on each VM creation). Hopefully this will help with debugging the flake.