Kubernetes AI Expert

Expert in deploying AI/ML workloads on Kubernetes with GPU scheduling, model serving frameworks, and MLOps patterns.

GPU Workload Scheduling

NVIDIA GPU Operator

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm install gpu-operator nvidia/gpu-operator

GPU Resource Requests

Full manifests: resources/manifests.yaml

Model Serving Frameworks

Framework Comparison

vLLM Deployment

Key configurations:

• --tensor-parallel-size - Multi-GPU inference
• --max-model-len - Context window
• --gpu-memory-utilization - Memory efficiency

Triton Inference Server

• Multi-model serving from S3/GCS
• HTTP (8000), gRPC (8001), Metrics (8002)
• Model polling for dynamic updates

Text Generation Inference (TGI)

• HuggingFace native
• Quantization support (bitsandbytes-nf4)
• Simple deployment

Deployment manifests: resources/manifests.yaml

Helm Chart Pattern

# values.yaml structure
inference:
  enabled: true
  replicas: 2
  framework: "vllm"  # vllm, tgi, triton
  resources:
    limits:
      nvidia.com/gpu: 1
  autoscaling:
    enabled: true
    minReplicas: 1
    maxReplicas: 10

vectorDB:
  enabled: true
  type: "qdrant"

monitoring:
  enabled: true

Auto-Scaling

Horizontal Pod Autoscaler (HPA)

Scale on:

• GPU utilization (DCGM_FI_DEV_GPU_UTIL)
• Inference queue length
• Custom metrics

KEDA Event-Driven Scaling

Scale on:

• Prometheus metrics
• Message queue depth (RabbitMQ, SQS)
• Custom external metrics

HPA/KEDA configs: resources/manifests.yaml

Networking

Ingress Configuration

• Rate limiting (nginx annotations)
• TLS with cert-manager
• Large body size for AI payloads
• Extended timeouts (300s+)

Network Policies

• Restrict pod-to-pod communication
• Allow only gateway → inference
• Permit DNS egress

Monitoring

Key Metrics

Setup

# Install DCGM Exporter
helm install dcgm-exporter nvidia/dcgm-exporter

# ServiceMonitor for Prometheus
# See resources/manifests.yaml

Managed Kubernetes

AWS EKS

• Instance types: g5.2xlarge, p4d.24xlarge
• AMI: AL2_x86_64_GPU
• GPU taints for isolation

Azure AKS

• VM sizes: Standard_NC*, Standard_ND*
• A100 support via NC24ads_A100_v4

OCI OKE

• Shapes: BM.GPU.A100-v2.8, VM.GPU.A10
• GPU node pools with taints

Terraform examples: ../terraform-iac/resources/modules.tf

Best Practices

Resource Management

• Always set GPU limits = requests
• Use node selectors for GPU types
• Implement tolerations for GPU taints
• PVC for model caching

High Availability

• Multiple replicas across zones
• Pod disruption budgets
• Readiness/liveness probes

Cost Optimization

• Spot instances for dev/test
• Auto-scaling to zero when idle
• Right-size GPU instances

Resources

• NVIDIA GPU Operator
• vLLM Documentation
• Triton Inference Server
• KEDA
• Kubernetes GPU Scheduling

Deploy AI workloads at scale with GPU-optimized Kubernetes.