runc 1.1.15 OOMs in Kubernetes e2e tests with containerd, cgroup v2, and cgroupfs driver

[samuelkarp]

Description

In containerd/containerd#10795 we're attempting to update containerd 1.6 to use runc 1.1.15. This triggers a Kubernetes e2e CI run, which is consistently failing. The same update to runc 1.1.15 succeeds with containerd 1.7 and main (which will be 2.0).

Steps to reproduce the issue

We're currently attempting to narrow down differences between the test environments used in containerd's various branches, since it seems unlikely the actual containerd version will be the cause of the difference. So far, containerd 1.6's tests are using the cgroupfs driver for managing cgroups as opposed to systemd, while the 1.7 and main branch tests use systemd to manage cgroups.

All tests are running on COS M117 and appear to be using cgroup v2 as of now.

Describe the results you received and expected

Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel: runc:[2:INIT] invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=-998
Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel: CPU: 0 PID: 52432 Comm: runc:[2:INIT] Tainted: G           O       6.6.44+ #1
Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel: Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 09/13/2024
Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel: Call Trace:
Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel:  <TASK>
Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel:  dump_stack_lvl+0x5d/0x80
Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel:  dump_header+0x52/0x250
Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel:  oom_kill_process+0x10a/0x220
Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel:  out_of_memory+0x3bc/0x5a0
Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel:  ? mem_cgroup_iter+0x1ca/0x240
Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel:  try_charge_memcg+0x82a/0xa90
Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel:  charge_memcg+0x3f/0x1f0
Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel:  __mem_cgroup_charge+0x2f/0x80
Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel:  do_pte_missing+0x544/0xc40
Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel:  handle_mm_fault+0x7a6/0xa60
Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel:  do_user_addr_fault+0x21b/0x770
Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel:  exc_page_fault+0x7f/0x100
Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel:  asm_exc_page_fault+0x26/0x30
Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel: RIP: 0033:0x7966028a541e
Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel: Code: 0c 20 4c 8b 6d 98 49 39 d3 49 89 4b 60 0f 95 c2 48 83 c8 01 49 83 c0 10 0f b6 d2 48 c1 e2 02 4c 09 e2 48 83 ca 01 49 89 50 f8 <48> 89 41 08 e9 16 fb ff ff 48 8d 0d f2 ed 0f 00 ba 13 10 00 00 48
Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel: RSP: 002b:00007ffffc0bb770 EFLAGS: 00010202
Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel: RAX: 0000000000004f21 RBX: 00007966029d2b00 RCX: 00005d2a23d5a0e0
Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel: RDX: 00000000000000f1 RSI: 0000000000000000 RDI: 0000000000000004
Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel: RBP: 00007ffffc0bb830 R08: 00005d2a23d5a000 R09: 00005d2a23d59ed0
Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel: R10: 00000000000000f0 R11: 00007966029d2aa0 R12: 00000000000000f0
Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel: R13: 00000000000000e0 R14: 000000000000000f R15: ffffffffffffffb8
Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel:  </TASK>
Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel: memory: usage 15360kB, limit 15360kB, failcnt 49
Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel: swap: usage 0kB, limit 9007199254740988kB, failcnt 0
Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel: Memory cgroup stats for /kubepods/burstable/pod6e4665e8-ae02-48b1-aee9-fedeeb899d20:
Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel: anon 3665920
Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel: file 11612160
Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel: kernel 442368
Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel: kernel_stack 98304
Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel: pagetables 147456
Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel: sec_pagetables 0
Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel: percpu 6944
Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel: sock 0
Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel: vmalloc 0
Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel: shmem 11612160
Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel: file_mapped 4268032
Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel: file_dirty 0
Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel: file_writeback 0
Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel: swapcached 0
Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel: anon_thp 0
Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel: file_thp 0
Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel: shmem_thp 0
Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel: inactive_anon 15089664
Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel: active_anon 159744
Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel: inactive_file 0
Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel: active_file 0
Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel: unevictable 0
Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel: slab_reclaimable 71080
Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel: slab_unreclaimable 89832
Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel: slab 160912
Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel: workingset_refault_anon 0
Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel: workingset_refault_file 0
Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel: workingset_activate_anon 0
Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel: workingset_activate_file 0
Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel: workingset_restore_anon 0
Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel: workingset_restore_file 0
Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel: workingset_nodereclaim 0
Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel: pgscan 0
Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel: pgsteal 0
Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel: pgscan_kswapd 0
Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel: pgscan_direct 0
Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel: pgscan_khugepaged 0
Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel: pgsteal_kswapd 0
Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel: pgsteal_direct 0
Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel: pgsteal_khugepaged 0
Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel: pgfault 1252
Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel: pgmajfault 0
Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel: pgrefill 0
Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel: pgactivate 38
Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel: pgdeactivate 0
Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel: pglazyfree 0
Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel: pglazyfreed 0
Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel: thp_fault_alloc 0
Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel: thp_collapse_alloc 0
Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel: Tasks state (memory values in pages):
Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel: [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel: [  52432] 65535 52432   401515     2634   163840        0          -998 runc:[2:INIT]
Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel: oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=0eec9f496bc31187cd00a14d2debd8b4bb89cc81bfc2301ee425e30c07572398,mems_allowed=0,oom_memcg=/kubepods/burstable/pod6e4665e8-ae02-48b1-aee9-fedeeb899d20,task_memcg=/kubepods/burstable/pod6e4665e8-ae02-48b1-aee9-fedeeb899d20/0eec9f496bc31187cd00a14d2debd8b4bb89cc81bfc2301ee425e30c07572398,task=runc:[2:INIT],pid=52432,uid=65535
Oct 08 17:28:42 tmp-node-e2e-7aba7327-cos-beta-117-18613-0-76 kernel: Memory cgroup out of memory: Killed process 52432 (runc:[2:INIT]) total-vm:1606060kB, anon-rss:3496kB, file-rss:1152kB, shmem-rss:5888kB, UID:65535 pgtables:160kB oom_score_adj:-998

What version of runc are you using?

1.1.15

Host OS information

No response

Host kernel information

No response

[cyphar]

I'm not sure why it works with containerd 1.7, but this OOM issue with Kubernetes e2e tests is something we know about.

This is caused by #4392 (backport of #3931) which removed the code we added in #1984 to fix the same failing Kubernetes e2e test (reported in #1980). This code was removed because the bindfd logic is not as secure as using memfds and was causing significant performance problems on systems with lots of container churn. When I re-investigated the e2e issue (when working on #3931) I figured out that the OOM is happening because Kubernetes/kubelet/runc(?) is being run in a cgroup with a very low memory limit (this is not caused by the container memory limit as incorrectly claimed in #1980 because the extra memory used by memfds is never accounted by the container cgroup).

My suggestion is to change the Kubernetes e2e test to run Kubernetes in a cgroup with a more reasonable memory limit. I'm not quite sure what the purpose of a test where you run the entire Kubernetes/kubelet stack with such a tiny memory limit is -- sure, containers with a small memory limit makes some sense, but why run the management code in such a constrained setup?

[samuelkarp]

Do you have a link to where containerd or runc is being constrained to a low cgroup memory limit? containerd is run as a normal systemd unit with no memory limit and is itself responsible for directly launching runc (so runc should start inside containerd's own cgroup).

[cyphar]

I'm not super familiar with Kubernetes e2e tests, but I believe KubeReservedMemory is setting the limits of the cgroup that containerd runs in (or something equivalent). See runOomKillerTest:

func runOomKillerTest(f *framework.Framework, testCase testCase, kubeReservedMemory float64) {
	ginkgo.Context(testCase.name, func() {
		// Update KubeReservedMemory in KubeletConfig.
		if kubeReservedMemory > 0 {
			tempSetCurrentKubeletConfig(f, func(ctx context.Context, initialConfig *kubeletconfig.KubeletConfiguration) {
				if initialConfig.KubeReserved == nil {
					initialConfig.KubeReserved = map[string]string{}
				}
				// There's a race condition observed between system OOM and cgroup OOM if node alocatable
				// memory is equal to node capacity. Hence, reserving a fraction of node's memory capacity for
				// K8s components such that node allocatable memory is less than node capacity to
				// observe OOM kills at cgroup level instead of system OOM kills.
				initialConfig.KubeReserved["memory"] = fmt.Sprintf("%d", int(kubeReservedMemory*getLocalNode(context.TODO(), f).Status.Capacity.Memory().AsApproximateFloat64()))
			})
		}
/* ... */
}

The issue here is not the container memory limit, because we know that cannot cause this error (the kernel allocates memory against memcgs based on the cgroup you were in when the memory was allocated -- our memfds are allocated and filled while in the host cgroups -- and the kernel will never migrate memory accounting on cgroupv2 even if you change cgrouops, and even with cgroupv1 you need to enable an option that is disabled on all systems by default). This was not obvious in #1980 but upon re-examination in #3931 it became clear that this is about the host memory limits, not the container ones.

[samuelkarp]

I believe KubeReservedMemory is setting the limits of the cgroup that containerd runs in (or something equivalent).

I don't believe that's correct. My understanding is that KubeReservedMemory controls the amount of advertised Memory resource to the Kubernetes scheduler, not that it translates to any cgroup setting on the node itself. @yujuhong, is that correct?

[yujuhong]

Yes, kube-reserved memory was used to calculate the node allocatable resources. Kubelet doesn't actually set the cgroup values for it.
(BTW, for compressible resources like cpu, this PR was merged recently for kubelet to enforce the cgroup values if the cgroup path is provided: kubernetes/kubernetes#125982)

[samuelkarp]

The issue here is not the container memory limit

Yes, I saw your explanation before and that makes sense. I have also validated that our CI hosts are running cgroup v2. But I don't think we understand why it is failing; as far as I can tell there are no constraints on the cgroup runc is running in.

[rata]

It's EOD here and I'm just getting some progress with this. I'll expand more tomorrow, including the repro I found.

It's still not 100% clear to me what is causing this. I can fix the k8s tests locally with this patch to the k8s repo (github.com/kubernetes/kubernetes):

diff --git test/e2e_node/oomkiller_linux_test.go test/e2e_node/oomkiller_linux_test.go
index db84cb0f0cf..ebc99abfa19 100644
--- test/e2e_node/oomkiller_linux_test.go
+++ test/e2e_node/oomkiller_linux_test.go
@@ -108,7 +108,8 @@ func runOomKillerTest(f *framework.Framework, testCase testCase, kubeReservedMem
                                // memory is equal to node capacity. Hence, reserving a fraction of node's memory capacity for
                                // K8s components such that node allocatable memory is less than node capacity to
                                // observe OOM kills at cgroup level instead of system OOM kills.
-                               initialConfig.KubeReserved["memory"] = fmt.Sprintf("%d", int(kubeReservedMemory*getLocalNode(context.TODO(), f).Status.Capacity.Memory().AsApproximateFloat64()))
+                               //initialConfig.KubeReserved["memory"] = fmt.Sprintf("%d", int(kubeReservedMemory*getLocalNode(context.TODO(), f).Status.Capacity.Memory().AsApproximateFloat64()))
+                               initialConfig.KubeReserved["memory"] = "100Mi"
                        })
                }

I can also make the tests pass changing the memory limits of them from 15Mi to 20Mi. I'm not sure why is that. I guess that the kubelet makes sure that the cgroup it is in has at least the mem limit and that is why that approach also works.

However, it is not 100% clear to me yet what the issue is nor what the right fix is.

I've also isolated the config.json created for the container (k8s uses two containers, the pause container and the real container, I'm just using the real container as the pause container doesn't have any limits nor anything interesting in the config) and then run it with runc. However, the container runs just fine when runnning it with runc.

Due to that, I suspect the issue is as @cyphar was saying, something about the setup that allocates too little mem for the KubeReserved. If I allocate a static "100Mi" for it, I see it pass locally, and if I allocate too litle (like 10Mi), I see it fail for all tests.

Please note, though, that when it fails it shows the same error than in the containerd issue comment:

  [FAILED] pod: "initcontinar-oomkill-target-pod", container: "oomkill-target-init-container" has unexpected exitCode: '\u0080'
  Expected
      <int32>: 128
  to equal
      <int32>: 137
  In [It] at: k8s.io/kubernetes/test/e2e_node/oomkiller_linux_test.go:152 @ 10/09/24 18:42:39.812

Therefore, what is failing in the test is the exit code. It is expecting 137 but gets 128. When the kernel kills runc[init] it seems it gets 128, but when it kills the "dd" process running in the container, it gets the right exit code.

Indeed, it seems the two test that fail locally are because runc[INIT] is being killed by a OOM in a cgroup. It correlates perfectly with what I see in the kernel logs locally:

# First test gets OOM on the runc process

[Fri Oct 11 02:51:37 2024] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=8a6434811b9e6f4d415ea9e1cd3e66898d1257cc8947c67054c3092c1216df9d,mems_allowed=0,oom_memcg=/kubepods/burstable/poda6a4366a-76d7-46a0-b52e-c96317070a0e,task_memcg=/kubepods/burstable/poda6a4366a-76d7-46a0-b52e-c96317070a0e/8a6434811b9e6f4d415ea9e1cd3e66898d1257cc8947c67054c3092c1216df9d,task=runc:[2:INIT],pid=554398,uid=999
[Fri Oct 11 02:51:37 2024] Memory cgroup out of memory: Killed process 554398 (runc:[2:INIT]) total-vm:1605432kB, anon-rss:1744kB, file-rss:944kB, shmem-rss:4116kB, UID:999 pgtables:168kB oom_score_adj:-997
[Fri Oct 11 02:51:37 2024] Tasks in /kubepods/burstable/poda6a4366a-76d7-46a0-b52e-c96317070a0e/8a6434811b9e6f4d415ea9e1cd3e66898d1257cc8947c67054c3092c1216df9d are going to be killed due to memory.oom.group set
[Fri Oct 11 02:51:37 2024] Memory cgroup out of memory: Killed process 554398 (runc:[2:INIT]) total-vm:1605432kB, anon-rss:1744kB, file-rss:944kB, shmem-rss:4116kB, UID:999 pgtables:168kB oom_score_adj:-997
[Fri Oct 11 02:51:37 2024] runc:[2:INIT] invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=-997

# Second tests gets OOM on the runc process
[Fri Oct 11 02:51:37 2024] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=379e12af22e9387021abbb33d87ffd37862b563f7849158ddb64fd2b3b380256,mems_allowed=0,oom_memcg=/kubepods/burstable/podffbb6b04-1c85-4039-8804-a8b793230f23,task_memcg=/kubepods/burstable/podffbb6b04-1c85-4039-8804-a8b793230f23/379e12af22e9387021abbb33d87ffd37862b563f7849158ddb64fd2b3b380256,task=runc:[2:INIT],pid=554461,uid=999
[Fri Oct 11 02:51:37 2024] Memory cgroup out of memory: Killed process 554461 (runc:[2:INIT]) total-vm:1605944kB, anon-rss:2348kB, file-rss:912kB, shmem-rss:3432kB, UID:999 pgtables:168kB oom_score_adj:-997
[Fri Oct 11 02:51:37 2024] Tasks in /kubepods/burstable/podffbb6b04-1c85-4039-8804-a8b793230f23/379e12af22e9387021abbb33d87ffd37862b563f7849158ddb64fd2b3b380256 are going to be killed due to memory.oom.group set
[Fri Oct 11 02:51:37 2024] Memory cgroup out of memory: Killed process 554461 (runc:[2:INIT]) total-vm:1605944kB, anon-rss:2348kB, file-rss:912kB, shmem-rss:3432kB, UID:999 pgtables:168kB oom_score_adj:-997

So, two times the runc init has been killed by the cgroup OOM, the two tests that fail. The other OOM I see in my logs is for the "dd" process (that is the one allocating memory) and that is passing (killing dd with oom gets 137 and not 128 exit code, that makes the test pass).

So, it seems the k8s test is setting-up a cgroup with very low mem for runc to run, this is probably due to the KubeReserved thingy, as changing that to a higher number makes the tests pass just fine here.

The current allocation is a percentage of the node capacity, as @cyphar pointed out: https://github.com/kubernetes/kubernetes/blob/95ec69c16c76b5ee71fdbebc7a5dea2c39341eb3/test/e2e_node/oomkiller_linux_test.go#L111:

		if kubeReservedMemory > 0 {
			tempSetCurrentKubeletConfig(f, func(ctx context.Context, initialConfig *kubeletconfig.KubeletConfiguration) {
				if initialConfig.KubeReserved == nil {
					initialConfig.KubeReserved = map[string]string{}
				}
				// There's a race condition observed between system OOM and cgroup OOM if node alocatable
				// memory is equal to node capacity. Hence, reserving a fraction of node's memory capacity for
				// K8s components such that node allocatable memory is less than node capacity to
				// observe OOM kills at cgroup level instead of system OOM kills.
				initialConfig.KubeReserved["memory"] = fmt.Sprintf("%d", int(kubeReservedMemory*getLocalNode(context.TODO(), f).Status.Capacity.Memory().AsApproximateFloat64()))
			})
		}

My gut feeling is that the VMs running the CI for containerd 1.6 have less mem than the ones for 1.7 and main, and this calculation gets too low with that VM and the new runc binary is slightly bigger and makes it fail when it seals its on-memory copy of the binary (to fix a CVE, we really want that :)).

@samuelkarp some questions for you:

1. Can you confirm what is the memory allowed for the VMs running for containerd 1.6, 1.7 and main? I guess it's different size.
2. Can you check what is assigned to the kubeReserved memory in containerd 1.6 and 1.7 CI? It would be great to check what those numbers are for the CI.
3. Can you try to run the CI for 1.6 with the patch I pasted here for the k8s bits? I guess if we use 100Mi the CI will pass just fine.
4. Another fix might be to stop checking the return code. This fixes the tests too. The return code can change according to what process is OOM-Killed (not sure why yet, although it probably makes sense and can't be changed in any reasonable way), so stop checking it might be an option too.

My gut feeling is that this calculation based on percentage is not very good, we should at least add a max between that number and 100Mi or so. But I'd check out later what the right fix might be for this, I'm too tired now :-D

[samuelkarp]

I found the relevant difference between the containerd 1.6 CI and 1.7 + main CI is which cgroup driver is in use: cgroupfs vs. systemd. When moving containerd 1.6 to use the systemd cgroup driver, the tests no longer fail.

[dims]

@samuelkarp +1 to use systemd cgroup driver consistently!

[samuelkarp]

@dims this isn't quite about consistency. When both Kubelet and containerd are configured to use cgroupfs, the test fails. When both Kubelet and containerd are configured to use the systemd cgroup driver, the test passes. I didn't try to test a mixed mode (I don't think we'd consider that to be supported).

[dims]

@samuelkarp ah that sounds bad.

[akhilerm]

I didn't try to test a mixed mode (I don't think we'd consider that to be supported).

Using different driver on container runtime and kubelet will not work, as we have seen errors in kubernetes tests when done like that.