🚀 RKE2 + Cilium (Strict Mode) + LB-IPAM & L2 Announcements Setup

This document explains step-by-step how to run an RKE2 Kubernetes cluster without kube-proxy (strict mode) using Cilium, to:

• Assign specific static LoadBalancer IPs,
• Announce these IPs on the network via L2 Announcements using ARP/NDP,
• And control traffic with security policies.

💡 Goal: Create services directly accessible from devices in the Layer-2 segment without additional components like MetalLB.

---

📌 Table of Contents

1. RKE2 Installation
2. Cilium Installation
3. Metrics & Hubble UI Exposure
4. Network Restriction Example
5. Cilium LB-IPAM & L2 Announcements
6. Demo and Conclusion

---

🛠 RKE2 Installation

sudo mkdir -p /etc/rancher/rke2

sudo tee /etc/rancher/rke2/config.yaml > /dev/null << 'EOF'
write-kubeconfig-mode: "0644"
node-ip: "10.20.56.30"
disable:
  - rke2-canal
  - rke2-ingress-nginx
disable-kube-proxy: true
tls-san:
  - "10.20.56.30"
EOF

curl -sfL https://get.rke2.io | INSTALL_RKE2_TYPE=server sh -

systemctl enable rke2-server --now

sudo systemctl start rke2-server.service

export KUBECONFIG=/etc/rancher/rke2/rke2.yaml

Pods remain in Pending state because no CNI (Container Network Interface) plugin is installed. Since the default CNI (e.g., rke2-canal) has been disabled, pods cannot obtain the required network configuration.

---

🐝 Cilium Installation

Load the Prometheus CRDs:

kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/main/example/prometheus-operator-crd/monitoring.coreos.com_servicemonitors.yaml

Install Cilium via Helm:

helm upgrade --install cilium cilium/cilium \
  --version 1.18.0 \
  --namespace kube-system \
  -f cilium-values.yaml

---

📊 Metrics & Hubble UI Exposure

Remove default services and expose them via NodePort:

kubectl delete svc cilium-agent -n kube-system
kubectl delete svc hubble-metrics -n kube-system

kubectl apply -f cilium-metrics-nodeport.yaml -n kube-system
kubectl apply -f hubble-metrics-nodeport.yaml -n kube-system

• Metrics URLs:

  • http://10.20.56.30:30861/metrics
  • http://10.20.56.30:30965/metrics

(These endpoints can be added to Prometheus and visualized in Grafana.)

• Hubble UI: http://10.20.56.30:31647

---

🔒 Network Restriction Example

Create namespaces:

kubectl create ns client-ns
kubectl create ns nginx-ns

Deploy pods:

kubectl apply -f client.yaml -n client-ns
kubectl apply -f nginx.yaml -n nginx-ns

Apply restriction rule:

kubectl apply -f restrict-rule.yaml

Test from client:

kubectl -n client-ns exec -it client -- curl nginx.nginx-ns.svc.cluster.local

Now create an attacker namespace to test:

kubectl create ns attacker-ns

kubectl run -n attacker-ns attacker --rm -it --image=alpine -- sh
apk add curl
curl nginx.nginx-ns.svc.cluster.local

---

🌐 Cilium LB-IPAM & L2 Announcements

Create IP pool:

kubectl apply -f ip-pool.yaml

Add L2 announcement policy:

kubectl apply -f cilium-l2ann-policy.yaml

---

🧪 Demo and Conclusion

Create demo namespace:

kubectl create ns lb-demo
kubectl apply -f deploy.yaml -n lb-demo

Browser and curl outputs:

---

🎯 Summary

Using Cilium’s LB-IPAM feature, we can assign specific static LoadBalancer IPs to services. L2 Announcements make these IPs reachable in the same Layer-2 network via ARP/NDP. This eliminates the need for MetalLB and keeps full control in our hands.