Agent skill

monitoring-stack

Manage the homelab monitoring stack at ~/docker/stacks/monitoring/. Services: Prometheus, Grafana, Alertmanager (Telegram via DeLoNETBot), cadvisor, node-exporter, process-exporter, Loki, OTEL collector, Dockge, Uptime Kuma, health-monitor. Use when: (1) adding, editing, or debugging Prometheus alert rules, (2) managing or restarting monitoring services, (3) checking alert delivery or Telegram bot status, (4) diagnosing system performance issues (CPU hogs, memory bloat, swap pressure), (5) adding new scrape targets to Prometheus, (6) configuring Grafana dashboards or datasources, (7) any task referencing "monitoring", "alerts", "prometheus", "grafana", "cadvisor", "process-exporter", "alertmanager", "telegram alerts", or "DeLoNETBot".

View SKILL.md on GitHub Repository
Stars 9
Forks 0

Install this agent skill to your Project

npx add-skill https://github.com/delorenj/skills/tree/main/monitoring-stack

SKILL.md

Monitoring Stack

Stack root: ~/docker/stacks/monitoring/

Architecture

See references/architecture.md for full service map, ports, URLs, and data flow.

Common Operations

Restart the full stack

bash
cd ~/docker/stacks/monitoring && docker compose up -d

Restart a single service

bash
cd ~/docker/stacks/monitoring && docker compose up -d <service-name>

Hot-reload Prometheus rules (no restart needed)

bash
curl -s -X POST http://localhost:9472/-/reload

Hot-reload Alertmanager config (no restart needed)

bash
curl -s -X POST http://localhost:9784/-/reload

Verify process-exporter is scraping

bash
curl -s http://localhost:9256/metrics | grep namedprocess | head -5

Test Telegram alert delivery

python
import urllib.request, json
url = "https://api.telegram.org/bot<TOKEN>/sendMessage"
data = json.dumps({"chat_id": 7564050286, "text": "Test alert", "parse_mode": "HTML"}).encode()
req = urllib.request.Request(url, data=data, headers={"Content-Type": "application/json"})
urllib.request.urlopen(req)

Bot token is in ~/docker/stacks/monitoring/alertmanager/config.yml.

Adding New Alert Rules

See references/alert-rules-guide.md for rule file locations, PromQL patterns, severity conventions, and examples.

Quick path: Edit the appropriate rule file, then hot-reload Prometheus:

bash
curl -s -X POST http://localhost:9472/-/reload

Rule file locations

File Scope
prometheus/alert_rules.yml Container CPU, memory, availability
prometheus/system_alerts.yml Host system + per-process hogs
prometheus/rules/docker-health.yml Container health checks, restart loops

Alerting

Alerts route to Telegram via @DeLoNETBot (chat_id: 7564050286).

Config: alertmanager/config.yml

Severity routing:

  • critical: repeat every 1h
  • warning: repeat every 4h

Key Design Decisions

  • cadvisor is resource-capped: 1 CPU, 512M memory, 30s housekeeping interval, pruned metrics. Without these limits it will eat an entire core scanning 69+ containers.
  • process-exporter exists specifically to catch host-level hogs that cadvisor and node-exporter miss (per-process CPU/memory).
  • Duplicate rules were consolidated: docker-health.yml only has health-check and restart-loop rules. CPU/memory rules live in alert_rules.yml.

Diagnosing Performance Issues

When the system stutters, check in this order:

  1. ps aux --sort=-%cpu | head -15 - find CPU hogs
  2. free -h - check swap pressure
  3. sensors - check thermals (Tctl)
  4. docker stats --no-stream - find container hogs

The alert rules should catch most of these automatically now. If they don't fire, check:

  • Prometheus is up: curl http://localhost:9472/-/healthy
  • Alertmanager is up: curl http://localhost:9784/-/healthy
  • Rules loaded: curl http://localhost:9472/api/v1/rules | python3 -m json.tool | head -40

Recommended Agent Skills

Expand your agent's capabilities with these related and highly-rated skills.

delorenj/skills

openclaw-upgrade

Upgrade OpenClaw installations to the latest version. Handles global npm installs, local development setups, release channels (stable/beta/dev), configuration preservation, and platform-specific requirements. Use when updating OpenClaw, switching release channels, or troubleshooting update issues.

9 0
Explore
delorenj/skills

vercel-composition-patterns

React composition patterns that scale. Use when refactoring components with boolean prop proliferation, building flexible component libraries, or designing reusable APIs. Triggers on tasks involving compound components, render props, context providers, or component architecture. Includes React 19 API changes.

9 0
Explore
delorenj/skills

security-monitor

Real-time security monitoring for Clawdbot. Detects intrusions, unusual API calls, credential usage patterns, and alerts on breaches.

9 0
Explore
delorenj/skills

event-driven-architecture

Kafka, RabbitMQ, SQS/SNS, event sourcing, CQRS, saga patterns, dead letter queues, and idempotency. Use when designing asynchronous systems, implementing message-driven workflows, or building event streaming pipelines.

9 0
Explore
delorenj/skills

shadcn-ui

Expert guidance for integrating and building applications with shadcn/ui components, including component discovery, installation, customization, and best practices.

9 0
Explore
delorenj/skills

just-fucking-cancel

Find and cancel unwanted subscriptions by analyzing bank transactions. Detects recurring charges, calculates annual waste, and helps you cancel with direct URLs and browser automation. Use when: 'cancel subscriptions', 'audit subscriptions', 'find recurring charges', 'what am I paying for', 'save money', 'subscription cleanup', 'stop wasting money'. Supports CSV import (Apple Card, Chase, Amex, Citi, Bank of America, Capital One, Mint, Copilot) OR Plaid API for automatic transaction pull. Outputs interactive HTML audit with one-click cancel workflow. Pairs with Plaid integration for real-time transaction access without CSV exports.

9 0
Explore

Didn't find tool you were looking for?

Be as detailed as possible for better results