Production Monitoring Expertise

When This Applies

Apply this guidance when:

• Assessing whether a release is production-ready
• Setting up or reviewing health checks
• Evaluating system stability after a deployment
• Defining monitoring and alerting requirements
• Responding to production incidents

Health Check Design

Application Health Endpoint

Every service should expose a health endpoint:

GET /health
Response:
{
  "status": "healthy" | "degraded" | "unhealthy",
  "version": "1.2.3",
  "timestamp": "2026-02-28T10:00:00Z",
  "checks": {
    "database": "healthy",
    "cache": "healthy",
    "external_api": "degraded"
  }
}

Health Check Levels

Key Metrics to Monitor

The Four Golden Signals

1. Latency — Response time distribution (p50, p95, p99)
2. Traffic — Request rate (requests/second)
3. Errors — Error rate (5xx responses / total)
4. Saturation — Resource utilization (CPU, memory, disk, connections)

Service-Level Indicators (SLIs)

Post-Deployment Verification

After merging to main and deploying:

Immediate (0-5 minutes)

• Health endpoint returns healthy
• No error spike in logs
• Response times are within normal range
• Key user flows work (manual smoke test)

Short-term (5-30 minutes)

• Error rate is stable or improving
• No memory leaks (memory usage stable)
• No connection pool exhaustion
• No increase in support tickets or alerts

Medium-term (30 min - 2 hours)

• Performance metrics are within historical norms
• Background jobs are processing normally
• No data integrity issues reported
• All scheduled tasks ran successfully

Alerting Guidelines

Alert Severity

Alert Design Rules

1. Actionable — Every alert should have a clear response action
2. No noise — If an alert fires > 5 times/day without action needed, fix or remove it
3. Contextual — Include what's wrong, what threshold was breached, and where to look
4. Escalation — If not acknowledged within N minutes, escalate to next level

Incident Documentation

After any production incident, document in reports/INCIDENT_<YYYYMMDD>_<NNN>.md:

# Incident Report — <date>

## Summary
<One-line description of what happened>

## Timeline
- HH:MM — Issue detected
- HH:MM — Investigation started
- HH:MM — Root cause identified
- HH:MM — Fix applied
- HH:MM — Verified resolved

## Impact
- Duration: X minutes
- Users affected: N
- Data impact: None / Describe

## Root Cause
<What caused the issue>

## Resolution
<How it was fixed>

## Prevention
<What changes will prevent recurrence>