Kubernetes AI Compute Cluster Setup Guide

1. Overview and architecture

1.1 Layering

The AI compute cluster uses a layered architecture so that the underlying hardware stays decoupled from the upper-layer AI frameworks:

• Infrastructure layer: GPU/CPU heterogeneous compute, RDMA high-speed network, NVMe/distributed storage.
• System and container layer: Linux Kernel, Containerd, NVIDIA Driver & CUDA.
• K8s scheduling layer: K8s Core, Device Plugin, Volcano/Kueue (batch scheduling), network/storage CSI.
• AI service layer: vLLM/Triton (inference), DeepSpeed/Megatron (training), Kubeflow/Ray (orchestration).

1.2 Recommended software version matrix

2. Hardware infrastructure planning

2.1 GPU node selection guidance

2.2 Network and storage planning

• Network:
  • Management/service network: 25Gbps/100Gbps Ethernet, used for the K8s control plane and service traffic.
  • Parameter network (RDMA): 100Gbps/200Gbps/400Gbps RoCE v2 or InfiniBand, used for multi-machine multi-card NCCL communication, SR-IOV and GPUDirect RDMA must be enabled.
• Storage:
  • System disk: 500GB NVMe SSD.
  • Model/data disk: distributed parallel file system (such as JuiceFS, Alluxio, WekaIO), requiring high throughput (>10GB/s) to accelerate checkpoint saving and model loading.

3. Operating system and kernel tuning

3.1 Base environment configuration

Recommends Ubuntu 22.04 LTS or Rocky Linux 9. Run the following on all nodes:

# 1. Disable Swap
swapoff -a
sed -i '/ swap / s/^$.*$$/#\1/g' /etc/fstab

# 2. Disable firewall and SELinux
systemctl stop firewalld && systemctl disable firewalld
setenforce 0
sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

# 3. Load the required kernel modules
cat <<EOF | tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
nvidia
nvidia_uvm
EOF
modprobe overlay br_netfilter nvidia nvidia_uvm

3.2 Kernel parameter tuning (sysctl)

Optimized for high-concurrency large models and network communication:

cat <<EOF | tee /etc/sysctl.d/99-k8s-ai.conf
# Network tuning
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-iptables = 1
net.core.somaxconn = 65535
net.core.netdev_max_backlog = 65535
net.ipv4.tcp_max_syn_backlog = 65535

# File descriptors
fs.file-max = 2097152
fs.inotify.max_user_watches = 524288
fs.inotify.max_user_instances = 8192

# GPU memory mapping tuning
vm.max_map_count = 262144
EOF
sysctl --system

4. Container runtime installation

Use containerd as the standard runtime.

# Install containerd
apt-get update && apt-get install -y containerd.io

# Generate the default configuration
mkdir -p /etc/containerd
containerd config default | tee /etc/containerd/config.toml

# Change the Cgroup driver to systemd (critical!)
sed -i 's/SystemdCgroup = false/SystemdCgroup = true/g' /etc/containerd/config.toml

# Configure NVIDIA Container Runtime (needed for a manual installation without GPU Operator)
# Using GPU Operator to inject it automatically is recommended; the manual nvidia runtime configuration in config.toml can be skipped here.

systemctl restart containerd
systemctl enable containerd

5. Kubernetes cluster setup

5.1 Install the kubeadm components

# Taking Ubuntu as an example, add the K8s 1.30 repository
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.30/deb/Release.key | gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.30/deb/ /' | tee /etc/apt/sources.list.d/kubernetes.list

apt-get update
apt-get install -y kubelet kubeadm kubectl
apt-mark hold kubelet kubeadm kubectl

5.2 Initialize the Master node

Create kubeadm-config.yaml:

apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
kubernetesVersion: v1.30.0
controlPlaneEndpoint: "192.0.2.100:6443" # High-availability VIP address
networking:
  podSubnet: "10.244.0.0/16"
  serviceSubnet: "10.96.0.0/12"
---
apiVersion: kubeadm.k8s.io/v1beta3
kind: InitConfiguration
nodeRegistration:
  criSocket: unix:///var/run/containerd/containerd.sock

Run the initialization:

kubeadm init --config=kubeadm-config.yaml --upload-certs

# Configure kubectl
mkdir -p $HOME/.kube
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

6. Network plugin deployment

Cilium is recommended: it is based on eBPF and outperforms traditional iptables/IPVS at large node scale and under high throughput.

# Install the Cilium CLI
CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
curl -L --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-amd64.tar.gz{,.sha256sum}
tar xzvfC cilium-linux-amd64.tar.gz /usr/local/bin

# Install Cilium (with Hubble observability enabled)
cilium install --version 1.15.0
cilium hubble enable
cilium status --wait

7. GPU compute resource onboarding

Use the NVIDIA GPU Operator to automate lifecycle management of the driver, Device Plugin, DCGM and other components.

7.1 Install the GPU Operator

# Add the NVIDIA Helm repository
helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update

# Install the GPU Operator (assuming the NVIDIA Driver is already installed on the nodes; drop --set driver.enabled=false if it is not)
helm install gpu-operator nvidia/gpu-operator \
  --namespace gpu-operator --create-namespace \
  --set driver.enabled=false \
  --set toolkit.enabled=true \
  --set dcgmExporter.enabled=true

7.2 Verify GPU detection

# View the GPU resources of the node
kubectl get nodes -o json | jq '.items[].status.allocatable."nvidia.com/gpu"'

# Run a test Pod to verify CUDA availability
kubectl run nvidia-smi --rm -it --image=nvidia/cuda:12.4.0-base-ubuntu22.04 -- nvidia-smi

8. Compute scheduling and isolation

8.1 Node labels and taint management

Isolate the GPU nodes to prevent ordinary workloads from being scheduled onto them by mistake:

# Label the GPU node
kubectl label nodes gpu-node-1 accelerator=nvidia-h800 gpu-mem=80G

# Set a taint (only Pods carrying the toleration may be scheduled there)
kubectl taint nodes gpu-node-1 dedicated=gpu:NoSchedule

8.2 Large-model inference Pod scheduling example

apiVersion: v1
kind: Pod
metadata:
  name: llm-inference
spec:
  tolerations:
  - key: "dedicated"
    operator: "Equal"
    value: "gpu"
    effect: "NoSchedule"
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: accelerator
            operator: In
            values: ["nvidia-h800"]
  containers:
  - name: vllm
    image: vllm/vllm-openai:latest
    resources:
      limits:
        nvidia.com/gpu: 2 # Request 2 GPUs
        memory: "128Gi"
        cpu: "32"

9. Storage solution

Loading large models (tens of GB up to hundreds of GB) and reading/writing checkpoints place extremely high demands on storage I/O. Deploying JuiceFS or NFS (test environments only) is recommended.

9.1 Shared mount for model files (alternative to HostPath/EmptyDir)

In production, mount distributed storage through a CSI driver:

volumes:
  - name: model-weights
    persistentVolumeClaim:
      claimName: juicefs-llm-pvc # Pre-created PVC pointing to the model directory in distributed storage
containers:
  - name: vllm
    volumeMounts:
      - name: model-weights
        mountPath: /models/Qwen2.5-72B

10. Model service packaging and deployment

Taking the deployment of vLLM (Qwen2.5-72B) as an example, the following shows a production-grade Deployment configuration.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: qwen2-72b-vllm
  labels:
    app: qwen2-72b
spec:
  replicas: 1
  selector:
    matchLabels:
      app: qwen2-72b
  template:
    metadata:
      labels:
        app: qwen2-72b
    spec:
      tolerations:
      - key: "dedicated"
        operator: "Equal"
        value: "gpu"
        effect: "NoSchedule"
      containers:
      - name: vllm
        image: vllm/vllm-openai:v0.5.4
        args:
          - "--model"
          - "/models/Qwen2.5-72B-Instruct"
          - "--tensor-parallel-size"
          - "4"
          - "--max-model-len"
          - "8192"
          - "--gpu-memory-utilization"
          - "0.9"
        ports:
        - containerPort: 8000
          name: http
        resources:
          limits:
            nvidia.com/gpu: 4
            memory: "256Gi"
            cpu: "48"
        volumeMounts:
        - name: model-storage
          mountPath: /models
        # Large models load slowly, so a StartupProbe is mandatory to prevent K8s from killing and restarting the container by mistake
        startupProbe:
          httpGet:
            path: /health
            port: 8000
          initialDelaySeconds: 60
          periodSeconds: 10
          failureThreshold: 60 # Allows a maximum loading time of 10 minutes
        readinessProbe:
          httpGet:
            path: /health
            port: 8000
          periodSeconds: 5
        livenessProbe:
          httpGet:
            path: /health
            port: 8000
          periodSeconds: 15
      volumes:
      - name: model-storage
        persistentVolumeClaim:
          claimName: llm-models-pvc

11. Autoscaling configuration

11.1 GPU utilization based HPA

Make sure the Prometheus Adapter is configured to map DCGM metrics to K8s custom metrics.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: qwen2-72b-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: qwen2-72b-vllm
  minReplicas: 1
  maxReplicas: 5
  metrics:
  - type: Pods
    pods:
      metric:
        name: DCGM_FI_DEV_GPU_UTIL # GPU utilization metric
      target:
        type: AverageValue
        averageValue: "75" # Scale out when the average GPU utilization exceeds 75%
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 60 # Scale-up cooldown
    scaleDown:
      stabilizationWindowSeconds: 300 # Scale-down cooldown, prevents flapping

12. Monitoring and observability

12.1 Monitoring stack deployment

kube-prometheus-stack is recommended:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install kube-prometheus prometheus-community/kube-prometheus-stack \
  --namespace monitoring --create-namespace

12.2 Key monitoring metrics (Grafana Dashboard)

The GPU Operator deploys dcgm-exporter automatically; monitor the following metrics closely:

• DCGM_FI_DEV_GPU_UTIL: GPU compute utilization
• DCGM_FI_DEV_FB_USED / FREE: VRAM usage
• DCGM_FI_DEV_GPU_TEMP: GPU temperature
• DCGM_FI_DEV_POWER_USAGE: power draw
• Inference service metrics: TTFT (Time To First Token), TPOT (Time Per Output Token), request queue length.

13. Security and multi-tenancy

13.1 Resource quota (ResourceQuota)

Limit a specific tenant (such as team-ai-research) to at most 8 GPUs:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: gpu-quota
  namespace: team-ai-research
spec:
  hard:
    requests.nvidia.com/gpu: 8
    limits.nvidia.com/gpu: 8
    requests.memory: "512Gi"
    limits.memory: "1024Gi"

13.2 Access control (RBAC)

Ensure ordinary developers can only view Pod logs and cannot delete nodes or change GPU scheduling policies.

14. Multi-cluster architecture (advanced)

For ultra-large compute pools spanning regions and multiple data centers, multi-cluster management is recommended:

• Control plane: Use Karmada or KubeFed to distribute workloads across clusters.
• Traffic scheduling: Combine Istio or a cloud provider’s GSLB to route intelligently based on the GPU idle state of each cluster.
• Federated scheduling: When cluster A’s GPUs are fully loaded, automatically overflow inference tasks to cluster B.

15. Troubleshooting and operations

15.1 Common troubleshooting commands

# 1. View GPU plugin logs (when Pods cannot be scheduled or GPUs are not detected)
kubectl logs -n gpu-operator -l app=nvidia-device-plugin-daemonset

# 2. Check node GPU hardware status (Xid error troubleshooting)
kubectl exec -it <gpu-pod-name> -- nvidia-smi -q | grep -i "xid"
# or check dmesg on the host
dmesg -T | grep -i nvidia | grep -i xid

# 3. NCCL communication timeout troubleshooting (during multi-machine multi-card training)
# Check RDMA network connectivity and confirm the NCCL_DEBUG=INFO environment variable is enabled to get detailed logs.

15.2 Graceful restart and upgrade

• Draining GPU nodes: Before upgrading the driver, run kubectl cordon <node> to block new scheduling, then kubectl drain <node> --ignore-daemonsets to migrate the running inference services.
• Rolling updates: Configure the RollingUpdate strategy for inference service Deployments to ensure the service is not interrupted during upgrades.

16. Environment acceptance checklist

Before handing the cluster over to business teams, verify the following acceptance criteria item by item: