Enhance and Document Non-Root Operation for k0s controller --enable-worker (Single-Node)

[flip111]

Is your feature request related to a problem? Please describe.

Attempting to run a single-node k0s cluster (k0s controller --enable-worker) as a dedicated non-root system user (e.g., a k0s user) on Linux systems can lead to critical blockers preventing the cluster from becoming fully operational. These issues primarily stem from k0s components attempting to write to or use paths that are typically root-owned and not writable by the non-root service user, even when standard workarounds like setting TMPDIR are in place.

Specifically:

• Kubelet Bootstrap Kubeconfig Path: A significant issue arises when the k0s controller worker (kubelet component) fails to start due to an attempt to write a temporary bootstrap kubeconfig file to a path like /run/k0s-RANDOMID-kubelet-bootstrap-kubeconfig. This operation results in a "permission denied" error when the service runs as a non-root user, as /run is typically root-owned. Setting the TMPDIR environment variable for the k0s service to a user-writable path (e.g., /run/k0s) does not appear to be respected for this specific file creation, suggesting the path might be hardcoded or its generation logic doesn't honor TMPDIR.
• Flexvolume Plugin Directory Probing: The kube-controller-manager has been observed attempting to create/probe /usr/libexec/kubernetes/kubelet-plugins/volume/exec/, which is also a root-owned/immutable path on many systems, particularly those with non-standard FHS layouts like NixOS. While k0s.yaml provides mechanisms to configure flexVolumeDriverPath (for Calico), workerProfiles.values.volumePluginDir (for kubelet), and controllerManager.extraArgs.flex-volume-plugin-dir, ensuring these configurations are consistently respected and that no fallbacks to default FHS paths occur is crucial for non-root setups.
• Status Socket Path: The default k0s status socket path (e.g., /var/lib/k0s/run/status.sock) can differ from what the k0s status CLI and other components might expect (e.g., /run/k0s/status.sock). While configurable with --status-socket, clearer defaults or documentation for non-root service setups would enhance usability.

These behaviors make robust, secure non-root operation for single-node controller+worker deployments challenging, often necessitating running the entire k0s service as root, which may not align with desired security postures.

Describe the solution you would like

K0s should more robustly support and clearly document running the k0s controller --enable-worker (and k0s server) process as a non-root user with least privilege. This includes:

• Respect Standard Environment Variables: All temporary file creation, especially for critical bootstrap files like the kubelet bootstrap kubeconfig, should respect standard environment variables such as TMPDIR. If TMPDIR is set to a user-writable path, k0s should utilize it.
• Configurable Paths for Bootstrap/Runtime Files: If certain critical files cannot use TMPDIR for architectural reasons, k0s should provide clear and documented configuration options within k0s.yaml (e.g., under spec.storage or a new spec.runtimePaths) to define the base directory for such files. This would allow them to be placed within the k0s data directory (e.g., /var/lib/k0s/run) or another user-writable location.
• Consistent Flexvolume/Plugin Path Handling: Once volumePluginDir (for kubelet) and flex-volume-plugin-dir (for controller-manager) are configured in k0s.yaml, all internal k0s components must strictly adhere to these paths, avoiding any fallbacks or probes to default FHS locations like /usr/libexec.
• Improved Default for Status Socket with Non-Root Users: When k0s is run as a non-root user and a RuntimeDirectory like /run/k0s is established (owned by the k0s user via systemd), k0s could consider defaulting its --status-socket to within this directory (e.g., /run/k0s/status.sock) to align with CLI expectations and systemd best practices for runtime files.
• Comprehensive Documentation: The official k0s documentation should feature a dedicated section for "Running k0s as a Non-Root User," detailing:
  • Required filesystem permissions for data directories, runtime directories, and any specific files/sockets.
  • Necessary Linux capabilities (if any beyond standard user permissions).
  • Guidance on configuring TMPDIR and other relevant environment variables for the k0s service.
  • Best practices for paths for plugins, CNI, etc., in a non-root context.
  • Clear examples of systemd unit files for non-root operation, highlighting necessary overrides or configurations.

Describe alternatives you've considered

• Running k0s as root: This serves as a current workaround for the /run write permission issue but is suboptimal from a security perspective (principle of least privilege).
• Complex systemd tmpfiles.d rules or bind mounts: Attempting to make specific patterns under /run writable by the k0s user is fragile, system-specific, and potentially insecure.
• Setting TMPDIR in the service environment: This was attempted (TMPDIR=/run/k0s), but it was not respected for the kubelet bootstrap kubeconfig file path that k0s tried to create directly under /run.

The ideal solution involves k0s core being more flexible and configurable to better support non-root environments without requiring such workarounds.

Additional context

The primary motivation for desiring robust non-root operation of k0s is to enhance system security by adhering to the principle of least privilege. If the main k0s daemon process does not intrinsically require root privileges for its steady-state operation (post-initial setup which might require root, e.g., for CNI binary placement typically handled by k0s install), then running it as a dedicated, unprivileged user reduces the potential attack surface and impact of a compromised k0s process.

Testing on systems like NixOS (which have a more constrained FHS environment and clear ownership for paths like /run and /usr/libexec) clearly highlights these pathing and temporary file assumptions. However, these issues can affect non-root operation on any Linux distribution where the service user does not have write access to /run or default FHS plugin directories.

Ensuring k0s is "non-root friendly" by default, or at least easily and clearly configurable to be so, would significantly improve its deployability and security posture across a wider range of environments, particularly for single-node controller+worker setups where simplicity and security are often key.

The error logs observed for the primary issues are:

• Failed to start controller worker" error="failed to write kubelet bootstrap kubeconfig: open /run/k0s-RANDOMID-kubelet-bootstrap-kubeconfig: permission denied"
• (Previously observed before path corrections) Error initializing dynamic plugin prober" err="error (re-)creating driver directory: mkdir /usr/libexec: permission denied"

Improved support for non-root operation would be a valuable enhancement to k0s.

[twz123]

Thanks for the thorough write-up.

The TL;DR of this answer is "PRs highly appreciated".

Note that a bare k0s controller (i.e. one that's started without the --single or --enable-worker flags), should be able to run as a less privileged user, maybe with some rough edges, as this is not a scenario that's actively tested right now. However, running the worker components in rootless mode basically means to support rootless containers. This is a huge topic. So, anyone who wants to chime in to make this happen is very welcome!

• Kubelet Bootstrap Kubeconfig Path: A significant issue arises when the k0s controller worker (kubelet component) fails to start due to an attempt to write a temporary bootstrap kubeconfig file to a path like /run/k0s-RANDOMID-kubelet-bootstrap-kubeconfig. This operation results in a "permission denied" error when the service runs as a non-root user, as /run is typically root-owned.
• Setting the TMPDIR environment variable for the k0s service to a user-writable path (e.g., /run/k0s) does not appear to be respected for this specific file creation, suggesting the path might be hardcoded or its generation logic doesn't honor TMPDIR.

The handling of the runtime dir is not entirely consistent across k0s. The instance you're referring to is actually handled by looking up XDG_RUNTIME_DIR in cases where k0s is not run as root, and only if that's not yielding anything, falling back to /run. Other places will use <data-dir>/run for the non root case. PRs to improve this are welcome (including documentation, of course)!

These behaviors make robust, secure non-root operation for single-node controller+worker deployments challenging, often necessitating running the entire k0s service as root, which may not align with desired security postures.

By default, k0s tries to run some binaries as non-root users, if possible, to mitigate this. Note that running other binaries with less privileged users is currently only possible as root (or maybe CAP_SYS_ADMIN?!?). So if running as non-root user, all the binaries will be run under the same user.

K0s should more robustly support and clearly document running the k0s controller --enable-worker (and k0s server) process as a non-root user with least privilege. This includes:

• Respect Standard Environment Variables: All temporary file creation, especially for critical bootstrap files like the kubelet bootstrap kubeconfig, should respect standard environment variables such as TMPDIR. If TMPDIR is set to a user-writable path, k0s should utilize it.
• Configurable Paths for Bootstrap/Runtime Files: If certain critical files cannot use TMPDIR for architectural reasons, k0s should provide clear and documented configuration options within k0s.yaml (e.g., under spec.storage or a new spec.runtimePaths) to define the base directory for such files. This would allow them to be placed within the k0s data directory (e.g., /var/lib/k0s/run) or another user-writable location.

It does respect TMPDIR and XDG_RUNTIME_DIR, although this needs to be polished, unified and documented.

• Consistent Flexvolume/Plugin Path Handling: Once volumePluginDir (for kubelet) and flex-volume-plugin-dir (for controller-manager) are configured in k0s.yaml, all internal k0s components must strictly adhere to these paths, avoiding any fallbacks or probes to default FHS locations like /usr/libexec.

This is also a huge topic. K0s tries to not use FHS paths to a good extent, but upstream Kubernetes does a lot. This creates quite some friction, because folks are usually not prepared for the kubelet directory not being /var/lib/kubelet, especially when installing stuff like CSI drivers and such (the Helm charts often lack a config option for this). There's a lot of issues around pushing into the opposite direction. Maybe that's something that can be addressed in config v2 🙈 ...

• Improved Default for Status Socket with Non-Root Users: When k0s is run as a non-root user and a RuntimeDirectory like /run/k0s is established (owned by the k0s user via systemd), k0s could consider defaulting its --status-socket to within this directory (e.g., /run/k0s/status.sock) to align with CLI expectations and systemd best practices for runtime files.

Agree, PRs welcome. Note that I'd not depend too much on what systemd is doing, as k0s supports other init systems, too. And then there's Windows, of course ...