Application Metrics Instrumentation

Practical patterns for adding observability to applications.

Five Metric Types

Naming Convention

<system>.<component>.<operation>.<metric_type>

Examples:

• myapp.api.users.requests_total
• myapp.db.queries.duration_ms
• myapp.cache.items.hit_total

Component Checklists

API Endpoints

• Request count by endpoint and method
• Response time (p50, p95, p99)
• Error rate by status code
• Authentication failures
• Request/response payload sizes

Database

• Connection pool (active, idle, waiting)
• Query duration by operation type
• Slow query count (threshold-based)
• Error count by type (timeout, constraint, connection)
• Transaction commit/rollback rates

Message Queues

• Messages produced/consumed per topic
• Queue depth (current backlog)
• Processing latency (end-to-end)
• Consumer lag
• Dead letter queue size

Caching

• Hit/miss ratio
• Eviction count and reason
• Cache size (entries and bytes)
• TTL expiration rate
• Connection pool status

Locks/Synchronization

• Acquisition time
• Contention count (failed acquisitions)
• Hold duration
• Timeout count
• Deadlock occurrences

Anti-patterns to Avoid

1. Unbounded label cardinality - Never use user IDs, session tokens, or request IDs as labels
2. Missing failure paths - Always instrument errors alongside successes
3. No heartbeat metric - Add a constant gauge (e.g., app.up = 1) to verify instrumentation works
4. Inconsistent naming - Stick to one convention across the codebase

Full Reference

For detailed examples, patterns, and rationale, fetch the complete guide: https://pierrezemb.fr/posts/practical-guide-to-application-metrics/