NorthChoosing a monitoring stack you will not regret
Every monitoring stack looks fine in a demo. The regret arrives eighteen months later, when the bill is large, the dashboards are stale, and nobody trusts the alerts. Choosing well is mostly about optimising for the boring long term rather than the exciting first week.
Start from questions, not tools
Before naming a single product, write down the questions the system must answer at 3am. "Is the service up?" "Is it slower than usual, and where?" "What changed recently?" A stack that answers those crisply beats one with a hundred panels nobody opens.
Three signals, one home
Metrics, logs and traces each earn their place, but they should live in one coherent place with a shared notion of a service:
- Metrics tell you something is wrong and roughly where.
- Traces tell you which hop is responsible.
- Logs tell you exactly what happened at that hop.
If moving between them means copying identifiers by hand across three tools, engineers will stop doing it under pressure — which is exactly when they need it.
Alert on symptoms, not causes
Alerting on every cause produces a pager that everyone mutes. We alert on user-facing symptoms tied to service-level objectives, and let causes surface during investigation. A good rule of thumb: if an alert fires and the user would not have noticed, it should have been a dashboard, not a page.
Budget for retention
The cost of observability is dominated by how much you keep and for how long. Decide retention deliberately per signal — high-resolution metrics for weeks, sampled traces, and logs tiered by importance — rather than keeping everything forever by default.
Our baseline
For most teams we land on Prometheus and Grafana for metrics and dashboards, OpenTelemetry for traces, and a tiered log pipeline. It is not the only good answer, but it is well understood, portable, and cheap to reason about — which is what you want from the system that watches everything else.