Log files in io.containerd.runtime.v2.task/*/log.json grow indefinitely and are never cleaned up

[Its-Alex]

Is your feature request related to a problem? Please describe.

Hello,

I have noticed that the log files located at io.containerd.runtime.v2.task/*/log.json are continuously growing and are never rotated or cleaned up. Over time, this leads to excessive disk usage on nodes running k0s.

$ find /run/k0s/containerd/io.containerd.runtime.v2.task/ -name log.json -size +100M
/run/k0s/containerd/io.containerd.runtime.v2.task/k8s.io/f254c117ea01089667f906f73aa085ab386bf8bddd90d727d6e56c03eff8bcb6/log.json
/run/k0s/containerd/io.containerd.runtime.v2.task/k8s.io/c62a974590a7c18b0c1ee44b002a382e442222a464abe037501fa8d36a26f921/log.json

Steps to reproduce:

1. Deploy k0s and run containers.
2. Observe the files in io.containerd.runtime.v2.task/*/log.json.
3. Notice that these files keep increasing in size and are never automatically cleaned up.

Expected behavior:

Log files should be rotated or cleaned up automatically to prevent disk space exhaustion.

Environment:

k0s version: v1.30.4+k0s.0
k0s sysinfo:

$ k0s config status
No resources found in kube-system namespace.
root@core-realtime:/home/xxii# k0s sysinfo
Total memory: 31.0 GiB (pass)
Disk space available for /var/lib/k0s: 297.0 GiB (pass)
Name resolution: localhost: [127.0.0.1] (pass)
Operating system: Linux (pass)
  Linux kernel release: 6.5.0-1019-nvidia (pass)
  Max. file descriptors per process: current: 1048576 / max: 1048576 (pass)
  AppArmor: active (pass)
  Executable in PATH: modprobe: /usr/sbin/modprobe (pass)
  Executable in PATH: mount: /usr/bin/mount (pass)
  Executable in PATH: umount: /usr/bin/umount (pass)
  /proc file system: mounted (0x9fa0) (pass)
  Control Groups: version 2 (pass)
    cgroup controller "cpu": available (is a listed root controller) (pass)
    cgroup controller "cpuacct": available (via cpu in version 2) (pass)
    cgroup controller "cpuset": available (is a listed root controller) (pass)
    cgroup controller "memory": available (is a listed root controller) (pass)
    cgroup controller "devices": available (device filters attachable) (pass)
    cgroup controller "freezer": available (cgroup.freeze exists) (pass)
    cgroup controller "pids": available (is a listed root controller) (pass)
    cgroup controller "hugetlb": available (is a listed root controller) (pass)
    cgroup controller "blkio": available (via io in version 2) (pass)
  CONFIG_CGROUPS: Control Group support: built-in (pass)
    CONFIG_CGROUP_FREEZER: Freezer cgroup subsystem: built-in (pass)
    CONFIG_CGROUP_PIDS: PIDs cgroup subsystem: built-in (pass)
    CONFIG_CGROUP_DEVICE: Device controller for cgroups: built-in (pass)
    CONFIG_CPUSETS: Cpuset support: built-in (pass)
    CONFIG_CGROUP_CPUACCT: Simple CPU accounting cgroup subsystem: built-in (pass)
    CONFIG_MEMCG: Memory Resource Controller for Control Groups: built-in (pass)
    CONFIG_CGROUP_HUGETLB: HugeTLB Resource Controller for Control Groups: built-in (pass)
    CONFIG_CGROUP_SCHED: Group CPU scheduler: built-in (pass)
      CONFIG_FAIR_GROUP_SCHED: Group scheduling for SCHED_OTHER: built-in (pass)
        CONFIG_CFS_BANDWIDTH: CPU bandwidth provisioning for FAIR_GROUP_SCHED: built-in (pass)
    CONFIG_BLK_CGROUP: Block IO controller: built-in (pass)
  CONFIG_NAMESPACES: Namespaces support: built-in (pass)
    CONFIG_UTS_NS: UTS namespace: built-in (pass)
    CONFIG_IPC_NS: IPC namespace: built-in (pass)
    CONFIG_PID_NS: PID namespace: built-in (pass)
    CONFIG_NET_NS: Network namespace: built-in (pass)
  CONFIG_NET: Networking support: built-in (pass)
    CONFIG_INET: TCP/IP networking: built-in (pass)
      CONFIG_IPV6: The IPv6 protocol: built-in (pass)
    CONFIG_NETFILTER: Network packet filtering framework (Netfilter): built-in (pass)
      CONFIG_NETFILTER_ADVANCED: Advanced netfilter configuration: built-in (pass)
      CONFIG_NF_CONNTRACK: Netfilter connection tracking support: module (pass)
      CONFIG_NETFILTER_XTABLES: Netfilter Xtables support: module (pass)
        CONFIG_NETFILTER_XT_TARGET_REDIRECT: REDIRECT target support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_COMMENT: "comment" match support: module (pass)
        CONFIG_NETFILTER_XT_MARK: nfmark target and match support: module (pass)
        CONFIG_NETFILTER_XT_SET: set target and match support: module (pass)
        CONFIG_NETFILTER_XT_TARGET_MASQUERADE: MASQUERADE target support: module (pass)
        CONFIG_NETFILTER_XT_NAT: "SNAT and DNAT" targets support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: "addrtype" address type match support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_CONNTRACK: "conntrack" connection tracking match support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_MULTIPORT: "multiport" Multiple port match support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_RECENT: "recent" match support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_STATISTIC: "statistic" match support: module (pass)
      CONFIG_NETFILTER_NETLINK: module (pass)
      CONFIG_NF_NAT: module (pass)
      CONFIG_IP_SET: IP set support: module (pass)
        CONFIG_IP_SET_HASH_IP: hash:ip set support: module (pass)
        CONFIG_IP_SET_HASH_NET: hash:net set support: module (pass)
      CONFIG_IP_VS: IP virtual server support: module (pass)
        CONFIG_IP_VS_NFCT: Netfilter connection tracking: built-in (pass)
        CONFIG_IP_VS_SH: Source hashing scheduling: module (pass)
        CONFIG_IP_VS_RR: Round-robin scheduling: module (pass)
        CONFIG_IP_VS_WRR: Weighted round-robin scheduling: module (pass)
      CONFIG_NF_CONNTRACK_IPV4: IPv4 connetion tracking support (required for NAT): unknown (warning)
      CONFIG_NF_REJECT_IPV4: IPv4 packet rejection: module (pass)
      CONFIG_NF_NAT_IPV4: IPv4 NAT: unknown (warning)
      CONFIG_IP_NF_IPTABLES: IP tables support: module (pass)
        CONFIG_IP_NF_FILTER: Packet filtering: module (pass)
          CONFIG_IP_NF_TARGET_REJECT: REJECT target support: module (pass)
        CONFIG_IP_NF_NAT: iptables NAT support: module (pass)
        CONFIG_IP_NF_MANGLE: Packet mangling: module (pass)
      CONFIG_NF_DEFRAG_IPV4: module (pass)
      CONFIG_NF_CONNTRACK_IPV6: IPv6 connetion tracking support (required for NAT): unknown (warning)
      CONFIG_NF_NAT_IPV6: IPv6 NAT: unknown (warning)
      CONFIG_IP6_NF_IPTABLES: IP6 tables support: module (pass)
        CONFIG_IP6_NF_FILTER: Packet filtering: module (pass)
        CONFIG_IP6_NF_MANGLE: Packet mangling: module (pass)
        CONFIG_IP6_NF_NAT: ip6tables NAT support: module (pass)
      CONFIG_NF_DEFRAG_IPV6: module (pass)
    CONFIG_BRIDGE: 802.1d Ethernet Bridging: module (pass)
      CONFIG_LLC: module (pass)
      CONFIG_STP: module (pass)
  CONFIG_EXT4_FS: The Extended 4 (ext4) filesystem: built-in (pass)
  CONFIG_PROC_FS: /proc file system support: built-in (pass)

OS: Ubuntu22.04
k0s config:

# generated-by-k0sctl 2025-06-09T11:25:28Z
apiVersion: k0s.k0sproject.io/v1beta1
kind: ClusterConfig
spec:
  api:
    address: 172.22.22.22
    extraArgs:
      feature-gates: HPAScaleToZero=true
    sans:
    - 172.22.22.22
    - 127.0.0.1
  controllerManager:
    extraArgs:
      horizontal-pod-autoscaler-tolerance: "0.001"
  images:
    default_pull_policy: Never
  network:
    provider: calico
  storage:
    etcd:
      peerAddress: 172.22.22.22
  telemetry:
    enabled: false
  workerProfiles:
  - name: xxii-k0s-profile
    values:
      imageGCHighThresholdPercent: 100
      imageMinimumGCAge: 876000h
      maxPods: 200
  - name: xxii-k0s-profile-with-cpu-optimization
    values:
      cpuManagerPolicy: static
      cpuManagerPolicyOptions:
        full-pcpus-only: "true"
      imageGCHighThresholdPercent: 100
      imageMinimumGCAge: 876000h
      maxPods: 200
      systemReserved:
        cpu: "4"
        memory: 1Gi

Container runtime: containerd

Thank you!

Describe the solution you would like

Is there a recommended way to manage or rotate these log files? If not, could you please consider implementing log rotation or cleanup for these files?

Describe alternatives you've considered

For now I used a CRON job:

0 0 * * * find /run/k0s/containerd/io.containerd.runtime.v2.task/k8s.io/*/ -maxdepth 1 -type f -name log.json -size +5M -exec truncate -s 0 {} \;

[jnummelin]

This is odd. I just checked on a cluster that's been running 12 days:

# find /run/k0s/containerd/io.containerd.runtime.v2.task \
  -type f -name log.json \
  -exec du -h {} \; | sort -h
0	/run/k0s/containerd/io.containerd.runtime.v2.task/k8s.io/2addf322fc03f8027855af3c2099149fe40e8834be28941e79d5f7047d897824/log.json
0	/run/k0s/containerd/io.containerd.runtime.v2.task/k8s.io/67a93c82be03a41671e02bd05a0383d124e5251d9bec4b0bdd978dc058341cfa/log.json
0	/run/k0s/containerd/io.containerd.runtime.v2.task/k8s.io/7ab85837f681bf16806deb0c5b489b8592ae3c79275e52d777bf431b7ca0c43f/log.json
0	/run/k0s/containerd/io.containerd.runtime.v2.task/k8s.io/898aa6044bd9f0c5cce15d93bff8698b701e38901c438672bc40ffb1cbca0f2f/log.json
0	/run/k0s/containerd/io.containerd.runtime.v2.task/k8s.io/8f681e2cb072de695998112ac4433722fa39a0f99c3bad6f1b8c7b75681949d0/log.json
0	/run/k0s/containerd/io.containerd.runtime.v2.task/k8s.io/966cd6279b2bdde38d6dfd088aa5e3de94ad837c015068283cad458e6949b5bf/log.json
0	/run/k0s/containerd/io.containerd.runtime.v2.task/k8s.io/aeffcfdc32970d3e2bc270970cbce8859b5f353d40829c0e3b5f7194c82b63be/log.json
0	/run/k0s/containerd/io.containerd.runtime.v2.task/k8s.io/b0103110ae212e2a32dd0c7fbf9c2135c657d152f5b538c18852157fbdd37b22/log.json

So in my case all those files are actually empty. As AFAIK they should be since kubelet is handling the logs, and rotation.

Can you check what is actually in those logs? Maybe that'll give us some clue why they are growing and why stuff even gets into those files.

GPT is telling me this:

Containerd’s built-in shim (runc v2) does not currently provide any native log-rotation for those per-task log.json files under /run/containerd/io.containerd.runtime.v2.task/.... By design the shim simply appends a JSON‐line on every exec or process stdout/stderr, so a noisy container (or a mis-configured liveness probe) can fill up your tmpfs.

So maybe it's some exec based probe that's gone haywire 🤔

[Its-Alex]

@jnummelin

As AFAIK they should be since kubelet is handling the logs, and rotation.

I checked the kubelet configuration, and the logs it manages are located in /var/log/pods/. For this folder, everything seems to be working perfectly:

$ find /var/log/pods/*/*   -type f   -exec du -h {} \; | sort -h
0       /var/log/pods/default_hivemq-operator-8555cb98f7-6mt2s_12a49d61-6a3a-49ef-ba70-3e5696c6f4dd/init-shared/0.log
...
11M     /var/log/pods/monitoring_monitoring-prometheus-adapter-5b467f8b67-6jfh5_a77e47b9-87fd-4934-b56a-478a5d522648/prometheus-adapter/0.log.20250619-052319
11M     /var/log/pods/monitoring_monitoring-prometheus-operator-grafana-84556f67c8-cbxg8_496ba5c4-1352-40c2-a3da-d930e93e7030/grafana/0.log.20250619-065150

Can you check what is actually in those logs? Maybe that'll give us some clue why they are growing and why stuff even gets into those files.

The logs contain entries from containerd (or possibly runc):

{"level":"info","msg":"Running with config:\n{\n  \"DisableRequire\": false,\n  \"SwarmResource\": \"\",\n  \"AcceptEnvvarUnprivileged\": true,\n  \"AcceptDeviceListAsVolumeMounts\": false,\n  \"SupportedDriverCapabilities\": \"compat32,compute,display,graphics,ngx,utility,video\",\n  \"NVIDIAContainerCLIConfig\": {\n    \"Root\": \"/run/nvidia/driver\",\n    \"Path\": \"/usr/local/nvidia/toolkit/nvidia-container-cli\",\n    \"Environment\": [],\n    \"Debug\": \"\",\n    \"Ldcache\": \"\",\n    \"LoadKmods\": true,\n    \"NoPivot\": false,\n    \"NoCgroups\": false,\n    \"User\": \"\",\n    \"Ldconfig\": \"@/run/nvidia/driver/sbin/ldconfig.real\"\n  },\n  \"NVIDIACTKConfig\": {\n    \"Path\": \"/usr/local/nvidia/toolkit/nvidia-ctk\"\n  },\n  \"NVIDIAContainerRuntimeConfig\": {\n    \"DebugFilePath\": \"/dev/null\",\n    \"LogLevel\": \"info\",\n    \"Runtimes\": [\n      \"docker-runc\",\n      \"runc\",\n      \"crun\"\n    ],\n    \"Mode\": \"auto\",\n    \"Modes\": {\n      \"CSV\": {\n        \"MountSpecPath\": \"/etc/nvidia-container-runtime/host-files-for-container.d\"\n      },\n      \"CDI\": {\n        \"SpecDirs\": [\n          \"/etc/cdi\",\n          \"/var/run/cdi\"\n        ],\n        \"DefaultKind\": \"management.nvidia.com/gpu\",\n        \"AnnotationPrefixes\": [\n          \"cdi.k8s.io/\"\n        ]\n      }\n    }\n  },\n  \"NVIDIAContainerRuntimeHookConfig\": {\n    \"Path\": \"/usr/local/nvidia/toolkit/nvidia-container-runtime-hook\",\n    \"SkipModeDetection\": true\n  }\n}","time":"2025-06-17T13:25:33Z"}
{"level":"info","msg":"Using low-level runtime /var/lib/k0s/bin/runc","time":"2025-06-17T13:25:33Z"}
{"level":"info","msg":"Using OCI specification file path: /run/k0s/containerd/io.containerd.runtime.v2.task/k8s.io/206cbf78799ba460432d969204ad46307e8c6c7f2f80f67e525bce219dee49bf/config.json","time":"2025-06-17T13:25:33Z"}
{"level":"info","msg":"Auto-detected mode as 'legacy'","time":"2025-06-17T13:25:33Z"}
{"level":"info","msg":"No graphics modifier required: no devices requested","time":"2025-06-17T13:25:33Z"}
{"level":"info","msg":"No modification required; no devices requested","time":"2025-06-17T13:25:33Z"}
{"level":"info","msg":"No modification required; no devices requested","time":"2025-06-17T13:25:33Z"}
{"level":"info","msg":"Using prestart hook path: /usr/local/nvidia/toolkit/nvidia-container-runtime-hook","time":"2025-06-17T13:25:33Z"}
{"level":"info","msg":"Applied required modification to OCI specification","time":"2025-06-17T13:25:33Z"}
{"level":"info","msg":"Forwarding command to runtime","time":"2025-06-17T13:25:33Z"}
{"level":"info","msg":"Running with config:\n{\n  \"DisableRequire\": false,\n  \"SwarmResource\": \"\",\n  \"AcceptEnvvarUnprivileged\": true,\n  \"AcceptDeviceListAsVolumeMounts\": false,\n  \"SupportedDriverCapabilities\": \"compat32,compute,display,graphics,ngx,utility,video\",\n  \"NVIDIAContainerCLIConfig\": {\n    \"Root\": \"/run/nvidia/driver\",\n    \"Path\": \"/usr/local/nvidia/toolkit/nvidia-container-cli\",\n    \"Environment\": [],\n    \"Debug\": \"\",\n    \"Ldcache\": \"\",\n    \"LoadKmods\": true,\n    \"NoPivot\": false,\n    \"NoCgroups\": false,\n    \"User\": \"\",\n    \"Ldconfig\": \"@/run/nvidia/driver/sbin/ldconfig.real\"\n  },\n  \"NVIDIACTKConfig\": {\n    \"Path\": \"/usr/local/nvidia/toolkit/nvidia-ctk\"\n  },\n  \"NVIDIAContainerRuntimeConfig\": {\n    \"DebugFilePath\": \"/dev/null\",\n    \"LogLevel\": \"info\",\n    \"Runtimes\": [\n      \"docker-runc\",\n      \"runc\",\n      \"crun\"\n    ],\n    \"Mode\": \"auto\",\n    \"Modes\": {\n      \"CSV\": {\n        \"MountSpecPath\": \"/etc/nvidia-container-runtime/host-files-for-container.d\"\n      },\n      \"CDI\": {\n        \"SpecDirs\": [\n          \"/etc/cdi\",\n          \"/var/run/cdi\"\n        ],\n        \"DefaultKind\": \"management.nvidia.com/gpu\",\n        \"AnnotationPrefixes\": [\n          \"cdi.k8s.io/\"\n        ]\n      }\n    }\n  },\n  \"NVIDIAContainerRuntimeHookConfig\": {\n    \"Path\": \"/usr/local/nvidia/toolkit/nvidia-container-runtime-hook\",\n    \"SkipModeDetection\": true\n  }\n}","time":"2025-06-17T13:25:34Z"}
{"level":"info","msg":"Using low-level runtime /var/lib/k0s/bin/runc","time":"2025-06-17T13:25:34Z"}

Our cluster remains operational for extended periods. Perhaps k0s could manage these logs and automatically remove them when they grow too large?