Improve debugging experience for k0s inttests #5878

juanluisvaladas · 2025-05-26T08:05:51Z

Description

Attaching a debugger or running the inttests with strace is super annoying and every time I need to do it I end up having to make a custom image and modifying the Bootloose suite or launching myself the k0s command. We should have a standardized way of doing this where we don't need to think and it's just a command to get that done.

This PR implements an optional, heavier debug image with delve and strace, a way to replace the command in the BootlooseSuite for a debugging command and documents it.

Type of change

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
Documentation update

How Has This Been Tested?

Manual test
Auto test added

Checklist

Implements an optional, heavier debug image with delve and strace, a way to replace the command in the BootlooseSuite for a debugging command and documents it. Signed-off-by: Juan-Luis de Sousa-Valadas Castaño <[email protected]>

juanluisvaladas · 2025-05-26T09:11:41Z

Closing and reopening to retrigger CI as github is having problems

juanluisvaladas · 2025-05-26T11:17:18Z

inttest/Makefile

+# Determine the bootloose stamp dependency based on K0S_DEBUG
+# Default to .bootloose-alpine.stamp (the "otherwise" case from the request)
+BOOTLOOSE_STAMP_DEP := .bootloose-alpine.stamp
+ifeq ($(K0S_DEBUG_TESTS),true)


Maybe we should rename this variable to DEBUG, which is used to build a non stripped k0s binary.

Sounds good to me. OTOH, we could have a richer syntax for automatically enabling delve, which would have a different syntax. We could then simply enable DEBUG, based on the presence/non-emptiness of the other variable.

github-actions · 2025-06-25T23:05:07Z

The PR is marked as stale since no activity has been recorded in 30 days

twz123 · 2025-06-26T10:15:33Z

inttest/bootloose-alpine/Dockerfile

  done

+RUN if [ "$INCLUDE_DEBUG_TOOLS" = "true" ]; then \
+  apk add --no-cache strace \
+  && go install github.com/go-delve/delve/cmd/dlv@latest; \


We could also just take delve from the Alpine repos: apk add --repository http://dl-cdn.alpinelinux.org/alpine/edge/community delve. In that case, we could even spare all the base image juggling.

twz123 · 2025-06-26T10:23:36Z

README.md

+It's possible to attach a debugger to a smoke test but it requires some modifications:
+
+1. k0s must be compiled with symbols: `DEBUG=true make k0s`
+2. The source code must be present in `go/src/github.com/k0sproject/` and


What's achieved by this second point? I was able debug stuff without doing this IIRC.

twz123 · 2025-06-26T10:26:36Z

README.md

+   ln -s <K0s source dir>/build/cache/go /run/k0s-build/go
+   ```
+
+3. Modify the inttest to include to run the command that you want to run:


Did you consider to support this without having to actively edit the code? Say, have a Make variable K0S_INTTEST_DEBUG_PORT=4000 which automatically selects the debug image and enables the debugger. We could also add something like "K0S_INTTEST_DEBUG=controller1:4000,controller2:5000" or sth. along those lines.

twz123 · 2025-06-26T10:28:24Z

README.md

+### Debugging smoke tests
+
+Every smoke test can be executed independently, the complete list of smoke tests is
+available in `inttest/Makefile.variables``


Suggested change

available in `inttest/Makefile.variables``

available in `inttest/Makefile.variables`.

Fun fact: You can print them to stdout via ./vars.sh FROM=inttest smoketests as well.

twz123 · 2025-06-26T10:30:45Z

README.md

+   s.Require().NoError(s.RunWorkers())
+   ```
+
+4. Launch the tests with `K0S_DEBUG_TEST=true` so that it uses the correct image:


Suggested change

4. Launch the tests with `K0S_DEBUG_TEST=true` so that it uses the correct image:

4. Launch the tests with `K0S_DEBUG_TESTS=true` so that it uses the correct image:

twz123 · 2025-06-26T10:45:22Z

inttest/Makefile

+BOOTLOOSE_STAMP_DEP := .bootloose-alpine.stamp
+ifeq ($(K0S_DEBUG_TESTS),true)
+  BOOTLOOSE_STAMP_DEP := .bootloose-debug.stamp
+  BOOTLOOSE_IMAGE ?= bootloose-debug


I'd suggest a lower case name to indicate this is not something that should come via the env.

twz123 · 2025-06-26T10:46:02Z

inttest/Makefile

 $(smoketests): K0S_PATH ?= $(realpath ../k0s)
 $(smoketests): K0S_IMAGES_BUNDLE ?= $(realpath ../airgap-image-bundle-linux-$(ARCH).tar)
-$(smoketests): .bootloose-alpine.stamp
+$(smoketests): $(BOOTLOOSE_STAMP_DEP)


I'd ditch that variable in favor of

Suggested change

$(smoketests): $(BOOTLOOSE_STAMP_DEP)

$(smoketests): .$(BOOTLOOSE_IMAGE).stamp

twz123 · 2025-06-26T10:48:46Z

inttest/Makefile

-	--build-arg GOLANG_IMAGE=$(golang_buildimage) \
-	--build-arg ALPINE_VERSION=$(alpine_patch_version) \
 	--build-arg ETCD_VERSION=$(etcd_version) \
 	--build-arg HELM_VERSION=$(helm_version) \
-	-t bootloose-alpine \
+	  --build-arg TARGETARCH=$(ARCH) \


This is breaking make check-bootloose-alpine-buildx. You can't include TARGETARCH here, and you need to include BOOTLOOSE_BASE.

twz123 · 2025-06-26T10:49:35Z

inttest/bootloose-alpine/Dockerfile

@@ -1,13 +1,13 @@
-ARG ALPINE_VERSION
-ARG GOLANG_IMAGE
+ARG BOOTLOOSE_BASE


Maybe just BASE_IMAGE?

twz123 · 2025-06-26T10:50:15Z

inttest/bootloose-alpine/Dockerfile

@@ -50,22 +50,27 @@ RUN curl --proto '=https' --tlsv1.2 -L https://get.helm.sh/helm-v$HELM_VERSION-l
 # Install etcd for smoke tests with external etcd
 # No arm or riscv64 binaries available (check-externaletcd won't work on ARMv7 or RISC-V)
 RUN if [ "$TARGETARCH" != arm ] && [ "$TARGETARCH" != riscv64 ]; then \
-    curl --proto '=https' --tlsv1.2 -L https://github.com/etcd-io/etcd/releases/download/v$ETCD_VERSION/etcd-v$ETCD_VERSION-linux-$TARGETARCH.tar.gz \


Unrelated whitespace change here and in the following lines.

juanluisvaladas requested review from a team as code owners May 26, 2025 08:05

juanluisvaladas requested review from ncopa and kke May 26, 2025 08:05

juanluisvaladas marked this pull request as draft May 26, 2025 08:09

juanluisvaladas force-pushed the dbg-inttests branch from 31e02c6 to 535b0c9 Compare May 26, 2025 08:13

juanluisvaladas added enhancement New feature or request area/smoke-tests labels May 26, 2025

juanluisvaladas marked this pull request as ready for review May 26, 2025 08:43

juanluisvaladas marked this pull request as draft May 26, 2025 09:05

juanluisvaladas force-pushed the dbg-inttests branch from 535b0c9 to 1d0a016 Compare May 26, 2025 09:06

Improve debugging experience for k0s inttests

d5eab45

Implements an optional, heavier debug image with delve and strace, a way to replace the command in the BootlooseSuite for a debugging command and documents it. Signed-off-by: Juan-Luis de Sousa-Valadas Castaño <[email protected]>

juanluisvaladas force-pushed the dbg-inttests branch from 1d0a016 to d5eab45 Compare May 26, 2025 09:07

juanluisvaladas closed this May 26, 2025

juanluisvaladas reopened this May 26, 2025

juanluisvaladas marked this pull request as ready for review May 26, 2025 09:14

juanluisvaladas commented May 26, 2025

View reviewed changes

github-actions bot added the Stale label Jun 25, 2025

twz123 reviewed Jun 26, 2025

View reviewed changes

github-actions bot removed the Stale label Jun 26, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Improve debugging experience for k0s inttests #5878

Improve debugging experience for k0s inttests #5878

Uh oh!

juanluisvaladas commented May 26, 2025

Uh oh!

juanluisvaladas commented May 26, 2025

Uh oh!

juanluisvaladas May 26, 2025

Uh oh!

twz123 Jun 26, 2025

Uh oh!

github-actions bot commented Jun 25, 2025

Uh oh!

twz123 Jun 26, 2025

Uh oh!

twz123 Jun 26, 2025

Uh oh!

twz123 Jun 26, 2025

Uh oh!

twz123 Jun 26, 2025

Uh oh!

twz123 Jun 26, 2025

Uh oh!

twz123 Jun 26, 2025

Uh oh!

twz123 Jun 26, 2025

Uh oh!

twz123 Jun 26, 2025

Uh oh!

twz123 Jun 26, 2025

Uh oh!

twz123 Jun 26, 2025

Uh oh!

Uh oh!

	available in `inttest/Makefile.variables``
	available in `inttest/Makefile.variables`.

	4. Launch the tests with `K0S_DEBUG_TEST=true` so that it uses the correct image:
	4. Launch the tests with `K0S_DEBUG_TESTS=true` so that it uses the correct image:

	$(smoketests): $(BOOTLOOSE_STAMP_DEP)
	$(smoketests): .$(BOOTLOOSE_IMAGE).stamp

Improve debugging experience for k0s inttests #5878

Are you sure you want to change the base?

Improve debugging experience for k0s inttests #5878

Uh oh!

Conversation

juanluisvaladas commented May 26, 2025

Description

Type of change

How Has This Been Tested?

Checklist

Uh oh!

juanluisvaladas commented May 26, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jun 25, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!