## MLOps Maturity Levels for Local AI

### Level 0: Manual
- Manual deployment via scripts
- Ad-hoc monitoring
- No formal processes
- Suitable for: POC, small pilots

### Level 1: Repeatable
- Documented deployment procedures
- Basic monitoring (health checks)
- Manual scaling
- Suitable for: Small production deployments

### Level 2: Defined
- Infrastructure as Code (IaC)
- Comprehensive monitoring and alerting
- Runbooks for common operations
- Suitable for: Production deployments

### Level 3: Managed
- Automated deployment pipelines
- Predictive scaling
- SLO-based alerting
- Suitable for: Enterprise production

### Level 4: Optimizing
- Continuous optimization
- Automated incident response
- Cost optimization automation
- Suitable for: Large-scale enterprise

## SLO Framework for Local LLM

### Availability SLO
| Tier | Target | Monthly Downtime | Error Budget |
|------|--------|------------------|--------------|
| Standard | 99.5% | 3.6 hours | 216 minutes |
| Enhanced | 99.9% | 43 minutes | 43 minutes |
| Premium | 99.95% | 22 minutes | 22 minutes |

### Latency SLO
| Metric | Standard | Enhanced | Premium |
|--------|----------|----------|---------|
| TTFT P50 | <500ms | <300ms | <200ms |
| TTFT P95 | <2s | <1s | <500ms |
| Total P50 | <5s | <3s | <2s |
| Total P95 | <15s | <10s | <5s |

### Throughput SLO
| Metric | Standard | Enhanced | Premium |
|--------|----------|----------|---------|
| Tokens/sec (P50) | >50 | >100 | >200 |
| Requests/min | >10 | >30 | >100 |
| Queue depth max | <20 | <10 | <5 |

### Error Rate SLO
| Metric | Standard | Enhanced | Premium |
|--------|----------|----------|---------|
| Error rate | <2% | <1% | <0.5% |
| Timeout rate | <5% | <2% | <1% |

# prometheus.yml - LLM monitoring configuration
global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  # LLM inference server
  - job_name: 'vllm'
    static_configs:
      - targets: ['vllm:8000']
    metrics_path: /metrics
    scrape_interval: 10s

  # GPU metrics
  - job_name: 'dcgm'
    static_configs:
      - targets: ['dcgm-exporter:9400']
    scrape_interval: 5s

  # System metrics
  - job_name: 'node'
    static_configs:
      - targets: ['node-exporter:9100']

  # Custom application metrics
  - job_name: 'ai-gateway'
    static_configs:
      - targets: ['gateway:9090']

## Essential Metrics for LLM Operations

### Infrastructure Metrics
| Metric | Description | Alert Threshold |
|--------|-------------|-----------------|
| gpu_utilization | GPU compute % | <20% (idle), >95% (saturated) |
| gpu_memory_used_bytes | VRAM usage | >90% capacity |
| gpu_temperature_celsius | GPU temp | >80°C |
| node_cpu_seconds_total | CPU usage | >90% sustained |
| node_memory_used_bytes | RAM usage | >85% |
| node_disk_io_utilization | Disk I/O | >80% |

### Application Metrics
| Metric | Description | Alert Threshold |
|--------|-------------|-----------------|
| llm_requests_total | Total requests | Trend analysis |
| llm_requests_in_flight | Active requests | >80% max |
| llm_request_duration_seconds | Latency histogram | P95 >SLO |
| llm_tokens_generated_total | Output tokens | Trend analysis |
| llm_time_to_first_token_seconds | TTFT | P95 >SLO |
| llm_queue_depth | Pending requests | >10 sustained |
| llm_errors_total | Error count | >1% rate |

### Business Metrics
| Metric | Description | Purpose |
|--------|-------------|---------|
| requests_by_user | Per-user count | Cost allocation |
| tokens_by_model | Per-model usage | Capacity planning |
| daily_active_users | Unique users | Adoption tracking |
| requests_by_type | Task breakdown | Usage patterns |

# alerts/llm-operations.yml
groups:
  - name: llm-availability
    rules:
      - alert: LLMServiceDown
        expr: up{job="vllm"} == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "LLM service is down"
          runbook: "https://runbooks/llm-down"

      - alert: LLMHighLatency
        expr: histogram_quantile(0.95, rate(llm_request_duration_seconds_bucket[5m])) > 10
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "P95 latency exceeds 10 seconds"

      - alert: LLMHighErrorRate
        expr: rate(llm_errors_total[5m]) / rate(llm_requests_total[5m]) > 0.01
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Error rate exceeds 1%"

  - name: llm-infrastructure
    rules:
      - alert: GPUMemoryPressure
        expr: dcgm_fb_used / dcgm_fb_total > 0.9
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "GPU memory above 90%"

      - alert: GPUHighTemperature
        expr: dcgm_gpu_temp > 80
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "GPU temperature high"

  - name: llm-capacity
    rules:
      - alert: QueueBacklog
        expr: llm_queue_depth > 10
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "Request queue building up"

      - alert: CapacityNearLimit
        expr: avg_over_time(gpu_utilization[1h]) > 0.8
        for: 1h
        labels:
          severity: info
        annotations:
          summary: "Consider capacity expansion"

## LLM Service Start Runbook

### Pre-Start Checklist
```bash
# 1. Verify GPU availability
nvidia-smi --query-gpu=name,memory.total --format=csv

# 2. Check disk space
df -h /mnt/models /var/log

# 3. Verify model files
ls -la /mnt/models/$(cat /opt/llm/config/model-name)

# 4. Check dependencies
docker compose -f /opt/llm/docker-compose.yml config --quiet

# 1. Start infrastructure services
docker compose up -d prometheus grafana

# 2. Start LLM server
docker compose up -d vllm

# 3. Wait for model loading
timeout 600 bash -c 'until curl -s http://localhost:8000/health | grep -q "ok"; do sleep 5; done'

# 4. Start API gateway
docker compose up -d nginx

# 5. Run smoke test
curl -X POST http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"glm-4.6","messages":[{"role":"user","content":"test"}]}'

# Check all services running
docker compose ps

# Verify metrics collection
curl -s http://localhost:9090/api/v1/query?query=up | jq '.data.result'

# Check for errors in logs
docker compose logs --tail=50 vllm | grep -i error

docker compose down
docker compose -f docker-compose.backup.yml up -d

### Incident Response

```markdown
## Incident Response Runbook

### Severity Levels
| Level | Description | Response Time | Examples |
|-------|-------------|---------------|----------|
| SEV1 | Service down | 15 minutes | Complete outage |
| SEV2 | Major degradation | 30 minutes | >50% errors |
| SEV3 | Minor degradation | 2 hours | Elevated latency |
| SEV4 | No impact | 1 business day | Warning alerts |

### Initial Response (SEV1/SEV2)

### Diagnosis Checklist
```bash
# 1. Service health
curl http://localhost:8000/health

# 2. Recent changes
git -C /opt/llm log --oneline -5

# 3. Resource utilization
nvidia-smi
docker stats --no-stream

# 4. Error logs
docker logs vllm --since 10m 2>&1 | grep -i error

# 5. Network connectivity
curl -v http://localhost:8000/health

# 6. System resources
free -h
df -h

# Reduce batch size
docker compose down
# Edit config: --max-num-seqs 64
docker compose up -d

# Check GPU utilization
nvidia-smi
# If saturated, reduce load or add capacity
# Temporary: enable request shedding

# Check if service is running
docker ps | grep vllm
# Restart if needed
docker compose restart vllm

## Capacity Management

### Capacity Planning

```markdown
## Capacity Planning Framework

### Current State Assessment
```bash
# Collect metrics for planning
promtool query range \
  --start="-30d" \
  --end="now" \
  --step="1h" \
  'avg_over_time(gpu_utilization[1h])'

Current utilization: 60%
Growth rate: 10% monthly
Capacity threshold: 80%

Months until threshold:
(80 - 60) / (60 * 0.1) = 3.3 months

Action: Plan capacity expansion within 3 months

### Load Testing

```markdown
## Load Testing Procedures

### Pre-Production Load Test
```bash
# Using locust or custom script
python load_test.py \
  --target http://localhost:8000 \
  --users 50 \
  --spawn-rate 5 \
  --duration 10m \
  --output results.json

## Model Lifecycle Management

### Model Updates

```markdown
## Model Update Procedure

### Pre-Update
```bash
# 1. Download new model in staging
huggingface-cli download <provider>/<model> \
  --revision <REVISION_OR_TAG> \
  --local-dir /mnt/models/staging/<MODEL_DIR>-v2

# 2. Validate model
python validate_model.py /mnt/models/staging/<MODEL_DIR>-v2

# 3. Test in staging
docker run --gpus all -p 8001:8000 vllm/vllm-openai \
  --model /mnt/models/staging/<MODEL_DIR>-v2
pytest tests/model_acceptance.py --endpoint http://localhost:8001

# 1. Start new version
docker compose -f docker-compose-v2.yml up -d vllm-v2

# 2. Wait for ready
until curl -s http://localhost:8001/health; do sleep 5; done

# 3. Shift traffic gradually
# Update nginx upstream weights: v1=90%, v2=10%
# Monitor for errors
# Update weights: v1=50%, v2=50%
# Monitor
# Update weights: v1=0%, v2=100%

# 4. Decommission old version (after 24h)
docker compose down vllm-v1

# If issues detected:
# Revert nginx weights to v1=100%, v2=0%
# Document issue
# Stop v2
docker compose -f docker-compose-v2.yml down

### Model Versioning

```markdown
## Model Version Management

### Version Tracking
```yaml
# model-inventory.yml
models:
  - name: <MODEL_NAME>
    versions:
      - version: v1.0
        deployed: 2024-06-01
        status: deprecated
        path: /mnt/models/archive/<MODEL_DIR>-v1
      - version: v1.1
        deployed: 2024-09-01
        status: active
        path: /mnt/models/<MODEL_DIR>
      - version: v2.0
        deployed: null
        status: staging
        path: /mnt/models/staging/<MODEL_DIR>-v2

## Cost Optimization

### Cost Tracking

```markdown
## Cost Allocation Framework

### Infrastructure Costs
| Component | Monthly Cost | Allocation Method |
|-----------|--------------|-------------------|
| Hardware amortization | $5,000 | Fixed |
| Power | $400 | Metered |
| Cooling | $100 | Estimated |
| Network | $200 | Fixed |
| Maintenance | $500 | Fixed |
| **Total** | **$6,200** | |

### Per-Request Cost

### Cost Optimization Levers
1. **Increase utilization:** Process more with same hardware
2. **Right-size models:** Use smallest model that meets quality needs
3. **Quantization:** Reduce resource requirements
4. **Batch processing:** Higher efficiency for async workloads
5. **Off-peak scheduling:** Run intensive tasks during low usage

## Operational Efficiency Metrics

### Utilization Targets
| Resource | Target Range | Below Target | Above Target |
|----------|--------------|--------------|--------------|
| GPU | 60-80% | Overprovisioned | Underprovisioned |
| Memory | 70-85% | Normal | Add capacity |
| CPU | 50-70% | Normal | Check bottleneck |

### Efficiency Calculations

### Optimization Actions
| Efficiency | Utilization | Action |
|------------|-------------|--------|
| Low | Low | Reduce capacity or increase workload |
| Low | High | Optimize configuration |
| High | Low | Good - headroom available |
| High | High | Good - consider expansion |

# .github/workflows/model-update.yml
name: Model Update Pipeline

on:
  workflow_dispatch:
    inputs:
      model_name:
        description: 'Model to update'
        required: true
      model_version:
        description: 'Target version'
        required: true

jobs:
  validate:
    runs-on: self-hosted
    steps:
      - name: Download model
        run: |
          huggingface-cli download ${{ inputs.model_name }} \
            --revision ${{ inputs.model_version }} \
            --local-dir /tmp/model-staging

      - name: Validate model
        run: python scripts/validate_model.py /tmp/model-staging

      - name: Run acceptance tests
        run: |
          docker run -d --gpus all -p 8001:8000 \
            -v /tmp/model-staging:/model \
            vllm/vllm-openai --model /model
          sleep 120
          pytest tests/model_acceptance.py --endpoint http://localhost:8001

  deploy:
    needs: validate
    runs-on: self-hosted
    steps:
      - name: Deploy to staging
        run: |
          mv /tmp/model-staging /mnt/models/staging/
          ./scripts/deploy-staging.sh

      - name: Run smoke tests
        run: pytest tests/smoke.py --endpoint https://staging.llm.internal

      - name: Deploy to production
        run: ./scripts/deploy-production.sh
        if: success()

# Auto-remediation rules
rules:
  - name: restart-on-oom
    condition: container_restart_count > 3 in 10m
    action: |
      docker compose down
      docker system prune -f
      docker compose up -d
    cooldown: 30m

  - name: scale-on-queue
    condition: queue_depth > 20 for 5m
    action: |
      kubectl scale deployment vllm --replicas=+1
    cooldown: 15m
    max_replicas: 4

  - name: alert-on-degradation
    condition: error_rate > 0.05
    action: |
      slack-notify --channel ops --severity high \
        --message "LLM error rate elevated"