Agent skill

incident-response-rene-kuhm-opencode-ohmyopencod

Runbooks para respuesta a incidentes. Diagnóstico, mitigación y documentación de incidents en producción.

Stars 163
Forks 31

Install this agent skill to your Project

npx add-skill https://github.com/majiayu000/claude-skill-registry/tree/main/skills/other/other/incident-response-rene-kuhm-opencode-ohmyopencod

SKILL.md

Incident Response Skill

Eres un SRE experto en respuesta a incidentes. Tu rol es diagnosticar, mitigar y documentar incidentes de producción.

Severity Levels

Level Description Response Time Examples
P1 Critical - Service down < 15 min Complete outage, data loss
P2 Major - Degraded service < 30 min Partial outage, slow response
P3 Minor - Limited impact < 2 hours Single feature broken
P4 Low - Minimal impact < 24 hours Cosmetic issues

Fase 1: DIAGNOSE

1.1 Recopilar Información

bash
# Estado de servicios
systemctl status <service>
docker ps
kubectl get pods -A

# Logs recientes
tail -n 100 /var/log/app/error.log
kubectl logs <pod> --tail=100

# Métricas de sistema
top -bn1 | head -20
df -h
free -m

# Conexiones de red
netstat -an | grep ESTABLISHED | wc -l

1.2 Checklist de Diagnóstico

  • ¿Cuándo empezó el problema?
  • ¿Hubo deploys recientes?
  • ¿Hay errores en los logs?
  • ¿Recursos del sistema (CPU, RAM, Disk)?
  • ¿Conectividad de red OK?
  • ¿Base de datos respondiendo?
  • ¿Servicios externos OK?
  • ¿Hay patrones en los errores?

1.3 Comandos de Diagnóstico Rápido

bash
# Node.js app
curl -s http://localhost:3000/health | jq

# Database connectivity
pg_isready -h localhost -p 5432
redis-cli ping

# DNS resolution
dig api.example.com

# SSL certificate
echo | openssl s_client -connect example.com:443 2>/dev/null | openssl x509 -noout -dates

Fase 2: MITIGATE

2.1 Acciones Inmediatas por Tipo

Error 5xx (Server Error)

bash
# Reiniciar servicio
systemctl restart app
# o
kubectl rollout restart deployment/app

# Escalar temporalmente
kubectl scale deployment/app --replicas=5

# Rollback si deploy reciente
kubectl rollout undo deployment/app

Base de Datos Lenta

bash
# Identificar queries lentas
SELECT * FROM pg_stat_activity WHERE state = 'active' ORDER BY duration DESC;

# Terminar queries bloqueadas
SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE duration > interval '5 minutes';

# Verificar conexiones
SELECT count(*) FROM pg_stat_activity;

Out of Memory

bash
# Identificar proceso
ps aux --sort=-%mem | head -10

# Limpiar cache
sync; echo 3 > /proc/sys/vm/drop_caches

# Reiniciar con más memoria
docker update --memory="4g" container_id

Disk Full

bash
# Encontrar archivos grandes
du -sh /* | sort -rh | head -10

# Limpiar logs antiguos
find /var/log -name "*.log" -mtime +7 -delete

# Limpiar Docker
docker system prune -a

2.2 Comunicación

Template de comunicación de incidente:

🚨 INCIDENT ALERT - P[X]

Status: INVESTIGATING / MITIGATING / RESOLVED
Impact: [Descripción del impacto]
Start Time: [HH:MM UTC]
ETA Resolution: [Estimación]

Current Actions:
- [Acción 1]
- [Acción 2]

Updates: [Link a status page]

Fase 3: POSTMORTEM

3.1 Template de Postmortem

markdown
# Postmortem: [Título del Incidente]

**Date**: YYYY-MM-DD
**Duration**: X hours Y minutes
**Severity**: P[X]
**Author**: [Nombre]

## Summary
[1-2 párrafos describiendo el incidente]

## Timeline (UTC)
| Time | Event |
|------|-------|
| HH:MM | Alerta recibida |
| HH:MM | Investigación iniciada |
| HH:MM | Root cause identificado |
| HH:MM | Mitigación aplicada |
| HH:MM | Servicio restaurado |

## Root Cause
[Explicación técnica detallada]

## Impact
- Users affected: [número]
- Revenue impact: [si aplica]
- Data loss: [si aplica]

## What Went Well
- [Punto 1]
- [Punto 2]

## What Went Wrong
- [Punto 1]
- [Punto 2]

## Action Items
| Action | Owner | Due Date | Status |
|--------|-------|----------|--------|
| [Acción 1] | @person | YYYY-MM-DD | TODO |
| [Acción 2] | @person | YYYY-MM-DD | TODO |

## Lessons Learned
[Resumen de aprendizajes]

Runbooks por Servicio

Next.js / React App

bash
# Health check
curl -I http://localhost:3000

# Ver errores recientes
grep -i error logs/app.log | tail -20

# Rebuild si necesario
pnpm build && pnpm start

PostgreSQL

bash
# Conexiones activas
psql -c "SELECT count(*) FROM pg_stat_activity;"

# Queries lentas
psql -c "SELECT pid, now() - pg_stat_activity.query_start AS duration, query
FROM pg_stat_activity WHERE state = 'active' ORDER BY duration DESC LIMIT 5;"

# Replication lag (si aplica)
psql -c "SELECT client_addr, state, sent_lsn, write_lsn, flush_lsn, replay_lsn
FROM pg_stat_replication;"

Redis

bash
# Info general
redis-cli info

# Memoria usada
redis-cli info memory | grep used_memory_human

# Conexiones
redis-cli info clients

Kubernetes

bash
# Pods en error
kubectl get pods --field-selector=status.phase!=Running

# Eventos recientes
kubectl get events --sort-by='.lastTimestamp' | tail -20

# Describe pod problemático
kubectl describe pod <pod-name>

# Logs de pod
kubectl logs <pod-name> --previous

Output Esperado

🔍 INCIDENT DIAGNOSIS REPORT
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Severity: P2 - Service Degraded
Started: 2024-12-25 14:30 UTC

🔴 Issues Found:
1. High memory usage (92%)
2. Database connection pool exhausted
3. 500 errors on /api/users endpoint

📊 Metrics:
- Response time: 4500ms (normal: 200ms)
- Error rate: 15% (normal: 0.1%)
- Active connections: 100/100

🛠️ Recommended Actions:
1. [IMMEDIATE] Restart app pods to free memory
2. [IMMEDIATE] Increase DB connection pool to 150
3. [SHORT-TERM] Add connection pooler (PgBouncer)
4. [LONG-TERM] Investigate memory leak in /api/users

📝 Commands to Execute:
kubectl rollout restart deployment/api
kubectl patch configmap db-config -p '{"data":{"pool_size":"150"}}'

Expand your agent's capabilities with these related and highly-rated skills.

Didn't find tool you were looking for?

Be as detailed as possible for better results