Kubernetes observability, metrics, logs and traces

Latency on checkout tripled at nine this morning, every light on the board is green, and nothing shipped last night. Six people are now on a call where the most experienced engineer keeps saying that something must have changed, in the tone of someone who knows that is not an answer. They are asking a question the system was never built to answer, and staring harder at the dashboard will not produce one.

Observability is the property of a system that lets you answer a question you did not think to ask when you built it. That word property matters, because it means you cannot buy this. It depends on what your applications emit and how much of it you keep, and no tool creates data that was never sent.

The checks already on your board are doing honest work. Somebody watched a disk fill up at a genuinely inconvenient hour and wrote a check so nobody would learn that lesson twice, and every check you have encodes a failure someone already lived through. Those are known failures, and rules catch them reliably every time. Your nine o'clock latency problem is the other kind, where working out the shape of the failure is itself the work, and no rule written in advance was going to fire.

Kubernetes takes this from difficult to genuinely different, because every debugging instinct you built on long-lived servers assumed you could name the machine, log into it, and look around. The alert fires, you go to inspect the pod, and it was evicted 90 seconds ago along with its filesystem and its log file. The workload has been rescheduled onto a node you had not thought about. The service you are investigating is 40 identical replicas, the failure sits in three of them, and you have no way to tell which three. That is why everything has to be shipped somewhere central while it is happening rather than gathered at the moment you want it.

Your team is having a version of this argument every quarter. Someone came out of a bad incident wanting the full observability platform, with traces and dashboards and a retention tier reaching back a year. Someone else points out that when payments went down in March, the way anyone found out was a customer email, because the alert that should have caught it was never written. Both of them are correct, and they are arguing about two different layers as though those layers competed.

Monitoring asks whether a condition is true right now, feeds the answer to a rule engine, and wakes someone up when a rule breaks. Observability asks what changed, which is impossible without history, because a change only exists relative to a before. Those two questions look similar written down and they have wildly different appetites.

That appetite gap explains why one layer is cheap and the other is not. Monitoring needs a source of current values, a short window to hold them in, and something to evaluate thresholds against. Observability needs retention, and retention brings the whole supporting cast with it... long-term storage, downsampling so the invoice stays defensible, a query layer that can put this week beside the same week last quarter, aggregated logs, trace storage, and enough shared identity between the three to move between them. The signals are largely the same signals, and keeping them is what costs money.

Both layers stay, because the same metrics feed both of them. Alerting on a threshold and investigating a trend are two uses of one stream of numbers, so the real question is what order to build them in, and how far up to climb before the return stops justifying the invoice.

Why history is the proactive half

Real-time alerting catches the failure that has already happened, which is valuable and also inherently too late to prevent anything, while history catches the failure that is still on its way. A disk filling steadily across three weeks, latency creeping up four milliseconds with every release, memory that never quite returns to where it started after each restart... none of those trip a threshold today, and every one of them is obvious the moment you can look backward. That is the honest argument for paying to retain data, and it beats the usual one about debugging faster, because it turns the second layer into the thing that stops you being paged on a Sunday.

Who this is actually for

Observability assumes somebody is going to act on it, and that assumption is worth testing before you spend anything. The payoff is investigative, so it lands with whoever can both see the whole path a request took and change the code that made it slow. Where those are the same people, usually product teams carrying their own services in production, every rung above monitoring pays for itself quickly. Where they are different people, the data arrives with a team that cannot fix what it shows, and the team that can fix it never sees it.

Traces make this concrete faster than anything else in this primer. A trace crossing 11 services is only useful to somebody accountable for the request rather than for one box on the diagram. Hand that same trace to a team that owns one of those services and it becomes evidence the problem belongs to somebody else, which is a political artifact rather than a debugging one.

Monitoring is therefore unconditional, and observability has to earn its place. Every team running production needs to find out when it breaks, whatever the operating model, and that is rung two. The rungs above it earn their place once somebody owns an application end to end with both the access and the authority to change it. Buy the platform before that ownership exists and you get an expensive pipeline feeding dashboards nobody opens, so make the ownership change and let the tooling follow it.

Partial coverage fails in the same way for much the same reason. Traces across an estate where half the services drop the context produce fragments that mislead, logs from 200 services with no shared identifier produce volume rather than answers, and both hand your engineers more data and no more understanding. Depth across a few important request paths beats shallow coverage everywhere. What you defer here is the platform and the spend, never the instrumentation, because the structured logging and context propagation from the sharp edge below stay cheap today and expensive later regardless of who owns what.

The ladder, and the pain that pushes you up it

Each rung exists because the rung below stopped being able to answer something, so climbing faster than your failures demand is spending money to look responsible.

Rung one is metrics-server, the cheapest useful thing in this entire document. One manifest and a couple of megabytes of memory per node buys you kubectl top plus working horizontal and vertical autoscaling. Install it on day one, and know exactly what it is, because the project is refreshingly blunt on the subject. It holds a short in-memory window at roughly 15 second granularity, it has no alerting engine and nothing to feed one, and its own documentation tells you not to use it as a source for monitoring solutions or as an accurate record of resource usage. You will outgrow it within weeks, having got what you came for, which was a legible cluster and functioning autoscaling.

Rung two arrives the first time something breaks and no machine tells you. In-cluster Prometheus with Alertmanager and Grafana is the first layer that pages a human being. Prometheus scrapes your workloads and the cluster components, holds a couple of weeks of history locally by default, and gives you a query language to interrogate it with. Alertmanager turns queries into notifications, with grouping and silencing so one dead node stops generating 200 separate messages at half past two. Grafana is how anyone reads any of it, and it belongs at this rung because these three normally ship as a single bundle. Start with a small alert set aimed at symptoms a customer would notice, since a comprehensive alert set on its first day is a training program in ignoring alerts.

Rung three arrives roughly 40 seconds after rung two starts working, when an alert says payments is unhappy and nobody can say why. Centralized log aggregation is what an engineer reaches for in exactly that moment. Collection runs as one agent per node, sweeping up output from every container on that node and shipping it somewhere durable, which is what finally solves the evicted-pod problem from chapter 00.

Rung four arrives when someone asks whether this is worse than last quarter and the room goes quiet. Prometheus writes onward to something like Thanos or Mimir or a managed backend, and you gain trending, capacity conversations grounded in evidence, and an answer to that question. Cost also stops being an afterthought at this rung and becomes a design input.

Rung five arrives when metrics can tell you a request took 900 milliseconds and cannot tell you which of the 11 services involved is responsible. Distributed tracing answers attribution, usually with an OpenTelemetry collector consolidating collection across all three signals by the time you get here. Before that fan-out exists, tracing is a great deal of instrumentation work with very little to show for it.

The alert says payments is slow, which is more than anyone knew 20 minutes ago and considerably less than anyone needs. Extracting a cause from a latency graph means squinting at a number that already threw away everything useful, and watching a team attempt it is a familiar kind of painful.

So you open the trace for one slow request and the whole path is laid out end to end. The request hit the gateway, went to payments, and payments called the pricing service six times in sequence when once would have done, burning 780 milliseconds on calls that returned identical answers. Now you know which hop owns the problem, and you still have no idea why pricing was called six times.

For that you read the logs from that service over that minute, and there it is in the application's own words. A cache lookup was failing silently while the retry loop covered for it, dutifully, six times per request. Three signals carried that investigation because each one answers a question the other two answer badly.

What each one is, and what it costs

A metric is a number sampled over time, such as requests per second, memory in use, or queue depth. Metrics stay cheap to store and fast to query because each sample is tiny and compresses well, and they aggregate cleanly, so the 99th percentile across 40 replicas is a reasonable thing to ask for casually. The tradeoff is that a metric has already discarded the individual requests that produced it, which is why yours could report a tripling and could not name a single request involved.

A log is a record of one event, carrying the detail metrics threw away. Stack traces live here, along with the specific order ID and the sentence where your application explains what it was attempting. Logs are expensive at volume, because prose does not compress the way numbers do and a busy service produces vastly more log data than metric data over the same minute. Structure decides whether they are useful or merely present, since machine-readable fields can be filtered and grouped after the fact, while an English sentence has to be parsed by pattern matching that shatters the first time a developer rewords the message.

A trace follows one request across every service it touched, timing each hop and recording what called what. Each hop is a span, and the spans assemble into a tree showing the request's entire journey, which is the only way to answer attribution when 11 services are involved and one of them is quietly lying about being fine. Traces ask the most of you, because a trace only forms if every service passes the trace context along to the next one, and that work happens inside your code rather than in the collection layer. That is exactly why tracing is the rung most often postponed, usually until an incident makes postponing it untenable.

Connecting them is the actual skill

Three signals in three tools that share nothing gives you three separate investigations. The move from metric to trace to log lines in that story worked because a trace ID was written into the log lines, letting you pull the ones belonging to that request rather than the other 4,000 that arrived in the same minute. Get shared identifiers right early and every tooling decision afterward becomes easier, and skip it and you will have bought three products and assembled the correlation by hand, in your head, at three in the morning.

The cluster has opinions too

Kubernetes runs its own event stream describing what the cluster did, so scheduling decisions, image pull failures, evictions, and probe failures all appear there. It is the first place to look when a pod refuses to run and nobody can say why, and chapter 04 covers it properly alongside everything else the cluster tells you about itself.

Continuous profiling gets called the fourth signal now, and it answers which lines of code burned the CPU rather than which service did. Recognize the term when it comes up in conversation and leave it there for now, because very few teams need it before the other three are working properly.

Somebody on your team will add a single label to a metric, in a change too small to warrant review, and the monitoring bill will rise by a factor of five. They wanted the graph broken down per customer, which is a completely reasonable thing to want, and nothing about the change looked dangerous. Cardinality is the concept that punishes you hardest for not knowing it, and nobody explains it until after something is already broken.

Start with what a metric actually is, because the model most people arrive with is too simple. It is a name plus a set of labels, so http_requests_total shows up tagged with the cluster it came from, the namespace, the service, the HTTP method, the endpoint, and the response code. Those labels are the entire reason metrics are useful, taking you from a graph of everything, which tells you nothing, to a graph of the one endpoint on the one service ruining your afternoon.

Every distinct combination of those labels is stored as its own separate stream of data, and the combinations multiply rather than add.

Take three clusters, 10 namespaces, 40 endpoints, four methods and eight response codes, which is a modest environment by any standard. That is 38,400 separate series for a single metric name, and your cluster emits hundreds of metric names before you write a line of your own.

The labels that actually kill you

Multiplication stays manageable while every label holds a small, boring, stable set of values. Response codes qualify, since there are only so many of those, and methods and namespaces qualify too. The wheels come off the moment a label can take an unbounded number of values, and the culprits are consistent enough across teams that you can predict them.

A user ID gives you one series per user, scaling exactly as badly as your business succeeds. A request ID is the true catastrophe, one permanent series for every request ever received, each holding a single data point describing an event that will never recur. Full URL paths do the same thing more quietly, so /orders/8831 and /orders/8832 live forever as separate series when both should have been /orders/{id} from the start. Back to our helpful colleague from the opening... customer ID as a label is that same mistake wearing a suit, right up until you sign an enterprise deal and onboard 4,000 of them.

Kubernetes contributes a trap of its own, following directly from pods being ephemeral. Labeling metrics with the pod name feels like the obvious move, and it quietly rewrites your entire series set on every deployment. New pods bring new names and new series, while the old ones stop receiving data without having the decency to disappear. Deploy 10 times a week and you are carrying 10 fresh generations of series, all holding index memory, none letting you compare anything to anything. Label by the thing that outlives a rollout, which is the deployment or the service rather than whichever pod happens to be answering right now.

The rule that fits on a sticky note

Labels are for things you group by, and identity belongs somewhere else. If you can imagine wanting a graph broken down by it, and the list of possible values stays short and stable, make it a label. If it picks out one specific request, user, or object, it belongs in a log line or a trace, where individual records are the whole point and the storage was designed for exactly that. Metrics stay cheap precisely because they refuse to carry identity, which is the clearest argument for why three signals exist rather than one.

Kubernetes exposes a surprising amount of telemetry and stores almost none of it, which is the same pattern as networking and storage. The interfaces and the data are there, the durable layer is yours to provide, and knowing exactly where that line sits saves you from both installing something redundant and assuming something exists.

Three tiers are worth separating, because they differ enormously in what they ask of you. Some of this is already running on your cluster and needs nothing installed. Some is one small manifest away and costs almost nothing to keep. And some is real software with its own capacity planning, upgrades, storage bill, and failure modes. Most of the arguments about observability being too complex are actually arguments about the third tier, and a good deal of value sits in the first two.

Tier one, already running

Every kubelet exposes container resource usage through an embedded copy of cAdvisor, covering CPU, memory, filesystem, and network per container. The kubelet publishes its own operational metrics too, along with figures on probe results and runtime operations. Control plane components each expose their own metrics endpoint, so API server request latency, scheduler queue depth, and controller work rates are all published on an HTTP endpoint waiting for something to come and read them, which is the collection model the next chapter covers.

Kubernetes also reports on its own decisions through the event stream, and this is a genuinely different kind of data from resource usage. Events describe what the cluster did and why, so an image that failed to pull, a pod that could not be scheduled, and a container killed for exceeding its memory limit all show up here in plain terms. Events are held for about an hour and then discarded, so collect them into your log pipeline if you want to answer questions about last Tuesday.

The API audit log records who did what through the API server, and it is a security artifact rather than an operational one. It is a configuration flag rather than an installation, and enabling it and shipping it somewhere your security team controls is a compliance requirement in most regulated environments, since it is the only record of a human deleting a namespace at four in the afternoon.

Everything in this tier is exposed and none of it is kept, which is the sentence to hold on to. The data exists for as long as something is looking, and no longer.

Tier two, one manifest away

Three small components round out what the cluster tells you about itself. Each is a single deployment or DaemonSet, each does one job, and none of them stores anything, so the operational burden is close to zero.

Metrics-server and kube-state-metrics sound similar and do unrelated jobs, which is the confusion worth clearing up early. Metrics-server aggregates resource usage from kubelets into the metrics API, which is the rung one component from chapter 01 and the reason autoscaling works at all. Kube-state-metrics reads the Kubernetes API and emits metrics about object state, so it tells you a deployment wants five replicas and has three available, a pod has restarted 14 times, or a job failed. You want both, because a container using 90 percent of its memory limit and a deployment that has not reached its desired replica count for 20 minutes are different problems. Node-level metrics about the host itself, covering disk, filesystem, and hardware, come from a node exporter running as a DaemonSet, since the kubelet reports on containers rather than on the machine.

Tier three, real software you operate

Everything durable sits here, and the jump in commitment is the honest reason people find this topic heavy. There is no log aggregation, no trace storage, no dashboards, no alert routing, and no retention beyond the short windows above, so all of it arrives as software you install, size, upgrade, secure, and pay to store data for. The common starting bundle is a Prometheus stack packaged with Alertmanager, Grafana, kube-state-metrics, and node exporter together, deployed and managed through operators, which is why most teams meet these components as one installation rather than five. Beyond it sit log aggregation, trace storage, and long-term metric retention, each with its own capacity questions.

It is the same shape as the pluggable layers in the Kubernetes primer, where the interface ships and the implementation is your decision, and it is why the ladder in chapter 01 is a ladder rather than a shopping list. Tiers one and two are close to free and worth having on any cluster. Tier three is where you climb deliberately, in the order your failures demand.

Prometheus reaches out and collects from your applications rather than waiting for them to send anything, and that single design choice explains most of what you will encounter. Each application exposes a plain HTTP endpoint listing its current numbers as text, and Prometheus fetches that endpoint every 15 or 30 seconds and stores what it finds. Your application holds counters in memory and answers when asked, knowing nothing about where the monitoring lives.

Pulling wins on a cluster for reasons that become obvious once you have run both. A scrape that fails is itself a signal, telling you the target is unreachable without needing a separate check to notice silence. Applications need no configuration pointing at a monitoring endpoint, which matters when you are running 200 services and would otherwise be updating all of them to move one host name. You can point a second Prometheus at the same targets to test something without disturbing the first. The one that matters most is that Prometheus reads the Kubernetes API to discover what to scrape, so a new deployment starts being collected within seconds of existing, and pods disappearing causes no configuration change at all.

Push still exists for the handful of cases that pulling cannot reach. A batch job that runs for 30 seconds and exits will never be there when a scrape arrives, so it pushes to an intermediary that holds its final numbers. Long-term storage also works by push, with Prometheus forwarding a copy of everything onward as it arrives.

The four metric types

Counters only increase, counting things that have happened, such as requests served or errors returned. You never graph a counter directly, because a line that climbs forever tells you nothing, and instead you graph its rate of change, which is the requests per second you actually wanted. Gauges move in both directions and hold a current value, like memory in use or queue depth, and those you graph as they are.

Histograms count observations into buckets, so a latency histogram records how many requests fell under 10 milliseconds, under 50, under 100, and so on. That bucketing is what makes percentiles possible after the fact, and it aggregates correctly across instances, so you can compute the 99th percentile across all 40 replicas from their combined buckets. Summaries calculate percentiles inside the application instead, which sounds simpler and carries a real trap, because percentiles calculated separately per replica cannot be combined into a cluster-wide figure. Averaging a set of 99th percentiles produces a number that means nothing. Reach for histograms whenever you expect to aggregate across instances.

What a query language buys you

Storing numbers is the easy half, and the value arrives when you can ask questions across them. A query language lets you take a raw counter, convert it to a rate, sum that rate across every replica of a service, break the result down by response code, and compare the error fraction against the total, all in one expression evaluated at query time. That is how a single dashboard panel answers what percentage of requests are failing right now, per service, without anyone having precomputed it.

Queries that get run constantly or span long windows are worth precomputing, which is what recording rules do. They evaluate an expression on a schedule and store the answer as a new metric, so an expensive calculation runs once every 30 seconds rather than every time somebody opens a dashboard.

Your application writes to standard output, and everything after that is somebody else's problem in a chain most people have never traced. The container runtime captures that output and writes it to a file on the node, the kubelet manages those files, and kubectl logs reads them back from the node when you ask. That is the entire default arrangement, and it explains both why logs work at all out of the box and why they vanish.

Two things delete your logs before you get to them. Deleting the pod deletes the files, so an evicted or rescheduled workload takes its history with it, and kubectl logs against a pod that no longer exists returns nothing. Rotation deletes them too, and this one surprises people, because the kubelet rotates container log files once they reach a size limit and keeps only a handful of previous files, so a chatty service can lose this morning's logs by lunchtime while running perfectly happily. Both of those are why shipping logs off the node is table stakes rather than an advanced practice.

Collection

One agent per node, running as a DaemonSet, reads the log files for every container on that node and forwards them somewhere durable. Fluent Bit, Vector, and Alloy are the common choices, and all of them do roughly the same job of tailing files, attaching the Kubernetes metadata describing which pod and namespace and deployment each line came from, and shipping the result onward. That metadata enrichment is the part that matters, because a log line arriving without knowing which service produced it is nearly worthless.

Applications that insist on writing to a file inside the container instead of standard output need a sidecar container sharing the volume and reading that file. Treat that as a workaround for software you cannot change rather than a pattern to adopt.

Two storage models, and the cost difference

Full-text indexing builds a searchable index over every word in every log line, which is what Elasticsearch and OpenSearch do. Queries come back fast on any term you can think of, and you pay for that in storage, memory, and the compute needed to build the index continuously. The alternative indexes only the labels, such as the namespace and the service and the level, then compresses the log bodies into chunks and scans them by brute force when queried, which is the Loki approach. Storage costs drop enormously, and queries that narrow by label first stay quick, while a query hunting one string across everything for the last week takes real time.

Choose between the two based on how your team actually investigates. Teams that arrive at logs already knowing the service and the time window, having come from a metric or a trace, do well on the label-indexed model. Teams whose workflow is searching all logs for an unusual string get their money's worth from a full index.

Structure is the whole game

A log line emitted as machine-readable fields can be filtered, grouped, and counted after the fact by anyone. The same information written as an English sentence has to be recovered by pattern matching, and those patterns break silently the first time a developer rewords the message. Emitting structured output costs almost nothing at the point of writing the code, and converting an existing codebase to it is the miserable retrofit from chapter 01.

A trace exists only if every service in the path agrees to participate, which makes tracing the one signal your applications cannot be given from the outside. When a request arrives, the first service creates a trace ID, and every service after it must read that ID from the incoming request and pass it along to whatever it calls next. Any service that drops it breaks the chain, and you get two disconnected fragments instead of one journey.

Each unit of work inside that journey is a span, carrying a start time, a duration, the name of the operation, and a pointer to the span that caused it. Those parent pointers assemble the spans into a tree, and reading that tree is how you see the shape of a request rather than just its total time.

The context travels in an HTTP header, and the industry settled on the W3C trace context standard for its format, which is why cross-vendor tracing works at all now. Instrumentation libraries for most languages handle reading and forwarding that header automatically for common frameworks and clients, so the work is often a matter of adding a library rather than editing every call site. The gaps show up in message queues, background jobs, and any custom transport, where the context has to be carried deliberately.

Sampling, because keeping everything is not affordable

Tracing every request at high volume produces more data than anyone will pay to store, so you keep a fraction. Deciding at the start of the request is head sampling, which is cheap and simple and keeps a random one percent, and the flaw is obvious once you have needed it, because the slow request you want to examine is almost certainly among the 99 percent you threw away. Deciding at the end is tail sampling, where a collector buffers all the spans of a request until it completes and then keeps it if it was slow, or errored, or hit an interesting service. Tail sampling costs more to run and keeps the traces you actually want, which is why the collector in the next chapter usually ends up doing it.

What a service mesh gives you, and what it does not

A mesh sits in the network path and can generate spans for every call between services without touching your code, which sounds like tracing for free. It gets you the topology and the timings between services, and it cannot see inside them, so a slow function inside payments looks like time payments spent thinking. The larger catch is that a mesh still cannot connect the trace across a service unless that service forwards the incoming trace header to its outgoing calls. Your applications remain involved, and a mesh reduces the instrumentation work rather than removing it.

OpenTelemetry solved a problem that used to make every observability decision permanent. Instrumenting an application meant importing a specific vendor's library and emitting a specific vendor's format, so changing backend meant reinstrumenting the codebase, which nobody ever did. Teams stayed with tools they had outgrown because leaving cost more than tolerating. OpenTelemetry separates the instrumentation from the destination, giving you one vendor-neutral set of libraries and one wire format that every serious backend now accepts.

That separation is worth understanding as an architectural property rather than a feature. Your code emits OTLP, the OpenTelemetry protocol, and where that data lands becomes a configuration decision made outside the application. Switching from a self-hosted stack to a managed vendor, or running both during an evaluation, stops involving your developers entirely.

The collector is a standalone process that receives telemetry, does things to it, and forwards it onward. It is built from three kinds of component, and the model is simple enough to hold in your head. Receivers accept data, whether that is OTLP from your applications, Prometheus metrics scraped from endpoints, or logs read from files. Processors modify what passes through, batching it for efficiency, adding the Kubernetes metadata describing where it came from, dropping attributes you do not want, applying tail sampling, or stripping fields that should never leave the cluster. Exporters send the finished result onward to one or more destinations.

Two deployment patterns between them cover almost everything most teams need. An agent collector runs as a DaemonSet on every node, gathering from local workloads and enriching with node and pod metadata. A gateway collector runs as a scaled deployment receiving from those agents, and it is where anything requiring a complete view belongs, tail sampling in particular, since deciding whether to keep a trace requires all of that trace's spans in one place.

Where this leaves your existing stack

The collector can take over collection entirely, and a growing number of teams now run it that way rather than as an addition. Its Prometheus receiver scrapes the same endpoints Prometheus would, converts what it finds into OTLP, and keeps the internal scrape metrics intact, so the up signal that tells you a target went unreachable survives the switch. The OpenTelemetry operator ships a target allocator that handles discovery and shards targets across collector replicas, which removes the last operational reason to keep Prometheus in the collection path.

What the collector does not have is storage, a query language, rule evaluation, or alert routing. A pure OpenTelemetry pipeline therefore still ends at a backend, whether that is Mimir, Thanos, VictoriaMetrics, a commercial platform, or Prometheus itself, which has accepted OTLP natively since version 3.0 behind a flag. Scraping and shipping is the layer that gets replaced, and the layer that answers questions and pages people stays exactly where it was.

Running both at once is the migration pattern rather than the destination. Point a collector at the same targets your Prometheus already scrapes, validate your dashboards and alerts against the new path, then retire whichever collection layer you no longer want. Most teams arrive here from the tracing side, needing a collector that can tail sample, and consolidate the other two signals into it afterward because running one collection layer is easier than running three.

Semantic conventions are the quieter half of the project and matter more than they sound. They define standard attribute names, so HTTP status codes and service names and database systems get called the same thing everywhere, which is what lets a dashboard built against one service work against another without translation.

Most alerting setups fail in the same direction, generating volume nobody reads. A team wires up notifications for high CPU, high memory, restarting pods, and elevated disk usage, and within a month the channel receives 60 messages a day, every one of which turned out to be fine. The next real outage arrives in that channel and nobody looks, because looking has been unrewarding for weeks.

The fix is choosing what to alert on by asking whether a customer would notice. Symptoms are things users experience, such as requests failing, pages loading slowly, or a queue growing faster than it drains. Causes are the mechanisms behind those, and CPU sitting at 90 percent is a cause that may or may not be producing any symptom at all. Alerting on symptoms gives you one alert for a real problem instead of five for its possible explanations, and it catches failures whose cause you never anticipated, which is the entire point.

Causes still belong on dashboards, where they answer why once a symptom has told you that something is wrong. That division keeps the number of things capable of waking someone up small and honest.

Frameworks worth knowing by name

The four golden signals are latency, traffic, errors, and saturation, and covering those four for a service gets you most of the way to useful alerting. The RED method narrows it for request-driven services to rate, errors, and duration. The USE method applies to resources rather than services, covering utilization, saturation, and errors, and it fits nodes and disks and queues. These overlap heavily and exist because different things need watching differently, so use whichever fits the thing in front of you.

Objectives and budgets

An indicator is something you measure, such as the fraction of requests served successfully in under 300 milliseconds. An objective is the target you commit to internally, perhaps 99.9 percent of them over 30 days. An agreement is what you promise a customer contractually, and it should always be looser than your internal objective, so you find out about problems before they do.

The error budget follows from arithmetic and changes conversations more than any dashboard. An objective of 99.9 percent over 30 days permits roughly 43 minutes of failure, which is your budget to spend. Consuming it slowly through background errors is a legitimate use, and burning half of it in one incident tells you where to spend engineering time next. It also settles the argument about shipping speed with evidence, because a team comfortably inside budget can afford to move faster and a team that has blown through it has its answer.

Burn rate alerting comes directly out of that same arithmetic. Rather than paging when errors exceed a fixed threshold, you page when the budget is being consumed fast enough to exhaust it before the window ends, using a fast window to catch sudden breakage and a slow window to catch steady degradation.

Making an alert useful at three in the morning

Every alert that reaches a person should require that person to do something, and anything else belongs on a dashboard or in a weekly review. The alert should say what the user-visible impact is, link to the dashboard showing the relevant data, and link to a runbook describing what to check first. Grouping matters as much, since one failed node producing 200 notifications is an outage in your alerting rather than useful information.

Observability spend surprises people because it scales with things nobody sized it against. Teams budget from node count and pod count, then discover the invoice tracks label cardinality, log verbosity, and retention window, none of which appear on a capacity plan. The numbers involved get large enough that the second layer becomes a finance conversation, and going in with a design rather than a shrug is the difference between keeping useful history and having it cut arbitrarily.

Three drivers between them account for nearly all of that spend. Log volume usually leads, growing with traffic and multiplied by whatever level of detail your applications emit, and debug logging left on in production is the classic accelerant. Metric cardinality comes next, and chapter 03 covered why one label can move that number by a factor of five. Trace volume finishes the set, which is why sampling exists at all.

Tiering, and the two retention questions

The pattern that works separates recent data from historical data, because they get used completely differently. A hot tier holds days at full resolution, sitting in fast storage and answering the questions you ask during an incident. A warm tier holds weeks or months, usually in object storage, at full or slightly reduced resolution. A cold tier holds a year or more heavily downsampled, which is what serves capacity planning and quarterly comparisons.

Downsampling is what makes the long tail affordable, and it works because nobody asks about last March at 15 second granularity. Storing five minute aggregates instead cuts the volume by a factor of 20 while preserving every trend you would actually look for. Compaction and downsampling are built into the long-term storage systems for exactly this reason.

Two different requirements drive retention and conflating them causes trouble. Operational retention is how far back your engineers need to look, and a quarter is generous for most teams. Compliance retention applies to audit logs and sometimes access logs, and it is set by regulation rather than by usefulness, often running to years. Keeping those separate lets you retain a small volume for a long time and a large volume briefly, rather than applying the strictest rule to everything.

Levers worth pulling before buying more storage

Dropping metrics nobody queries is the most immediate saving available, and the query statistics in your own backend will tell you which those are. Dropping high-cardinality labels at the collector, before the data lands, costs nothing to implement. Sampling logs from high-volume services that emit near-identical lines preserves the signal at a fraction of the volume. Tail sampling traces keeps the interesting ones and discards the routine ones, typically cutting trace storage by an order of magnitude while improving what remains.

Cloud data transfer charges deserve a mention, because shipping telemetry between availability zones or out to a hosted vendor generates egress that appears on a different invoice from storage. Collecting locally, batching, and compressing before anything crosses a boundary is what the collector's batch processor exists to do.

Everything so far arrives in a specific order during a real problem, and walking that order once makes the shape of it stick. This is the payments incident from chapter 02, followed properly.

The page arrives at 09:12 and says the checkout success rate has dropped below the objective, burning error budget fast enough to exhaust the month inside four hours. It says nothing about a cause, and it came from a symptom alert watching what customers experience rather than from anything watching CPU.

The dashboard linked in the alert answers the first three questions in about 30 seconds. Failures are concentrated in the payments service rather than spread across everything, which rules out a cluster-wide problem. Latency at the 99th percentile went from 180 milliseconds to 2.4 seconds at 09:04, and errors are timeouts rather than rejections, so something is slow rather than refusing. Traffic is flat, which removes a load spike from consideration. Two minutes in, you know where the problem lives and roughly what kind of problem it is.

The trace is what explains the shape of what is happening. Filtering for slow requests through payments and opening one shows the tree, and payments is calling pricing six times in sequence rather than once, with each call taking around 130 milliseconds. Comparing that against a trace captured at 08:30 shows a single call. The extra five calls are new behavior, and the timing tells you they are sequential rather than parallel, so the delay is additive.

The logs are what finally explain the cause underneath all of it. Pulling the payments logs for that minute, filtered by the trace ID from the trace you just read, shows a cache lookup failing and a retry loop catching the failure and trying again, five times, silently. The log line naming the cache endpoint is the answer, and checking that dependency shows a connection pool exhausted after a configuration change deployed at 09:02.

Four things made that possible, and each one was a decision made months earlier. The alert watched a symptom, so it fired on a failure nobody had predicted. Traces existed for the paths that matter, so attribution took a minute rather than an afternoon of adding logging and waiting for a recurrence. The trace ID was written into log lines, so finding the relevant logs among thousands was a filter rather than a search. The logs were structured, so filtering by trace ID worked at all.

Consider how the same incident goes without any of those four things. The alert would have been a CPU threshold that never fired, so the first notification would have been a customer. With no traces, the investigation becomes reading payments logs and guessing which dependency to suspect. With unstructured logs and no trace ID, finding the relevant lines means grep across a busy service and reading by eye. The same root cause takes three hours instead of 12 minutes, and none of the difference is skill.

Your observability stack sees everything, which makes it both the most useful system you run and a genuine security concern. It holds cluster-wide read access, aggregates data from every namespace, and stores whatever your applications happened to write into a log line. Treating it as infrastructure rather than as a system holding sensitive data is how organizations end up with an incident whose cause is their monitoring.

Logs leak sensitive material in ways that are thoroughly predictable. Authentication tokens end up in URLs that get logged as request paths. Request bodies get logged wholesale during debugging and never get untangled. Stack traces carry variable values, which means customer records and card details land in the trace. Personal data arrives in logs simply because an application logged which user did what, and every one of these is ordinary rather than negligent.

Scrub at collection time, in the processor stage of your collector, before anything reaches storage. Redacting at query time leaves the data present for anyone who can query around the redaction, and cleaning a log store after the fact is expensive and rarely complete. Patterns for tokens, card numbers, and email addresses belong in the pipeline configuration, and dropping a field entirely beats masking it whenever the field has no operational use.

Personal data in logs also brings retention obligations that operational thinking misses. Data protection regimes treat identifiable log entries as personal data, which puts limits on how long you keep them and creates deletion obligations that immutable append-only log stores are poorly suited to satisfy. Logging a user identifier rather than an email address, and keeping the mapping elsewhere, avoids most of this and costs nothing at the point of writing the code.

Tenancy

A cluster serving several teams needs its observability split along the same lines, and the default is a single shared view of everything. Grafana handles this with folders and teams and data source permissions, so each team sees the dashboards and data belonging to its own namespaces. Log and metric backends built for multiple tenants enforce it at the storage layer, tagging data with a tenant identifier and filtering queries against it, which is stronger than filtering in the dashboard alone. Prometheus itself has no concept of tenancy, so multi-tenant enforcement is one of the reasons teams move to the larger long-term storage systems.

The audit log deserves separation from everything else described in this chapter. It records what your engineers did, which means the people it records should not be able to modify or delete it, and it belongs in a store your security team controls with its own access rules and its own retention. Routing it into the same log store as application output and granting everyone read access defeats the reason it exists.

A handful of these account for most of the trouble that shows up in the first year.

The fastest way to make this stick is to run the first two rungs on a cluster you can afford to break. Install a Prometheus stack, write three alerts on symptoms rather than causes, then break something deliberately and watch what reaches you and how long it takes. The gap between what you expected and what actually happened is where the learning is.

The Prometheus documentation is the authoritative reference for metric types, query semantics, and alerting rules, and the OpenTelemetry documentation covers the collector, the wire format, and the instrumentation libraries for each language. Both are worth trusting over any blog when details matter. Google's Site Reliability Engineering book is the original source for the golden signals, symptom-based alerting, and error budgets, and it remains the clearest writing on why alerting practice looks the way it does. The Kubernetes documentation covers the metrics pipeline, the audit log configuration, and the event stream.

For the layers underneath this one, Understanding Kubernetes explains the cluster this is all watching, how Helm works explains the packaging your workloads arrive in, and Understanding GitOps explains how they got there in the first place. And the aside on designing systems that solve the right problem steps outside the stack, covering how to establish which business measure any of this was supposed to move.

Once you can name which signal answers which question, size the cost of a retention decision before making it, and look at a team's alerting to judge which rung of the ladder they are actually on, you can hold your end of any conversation this topic produces. The harder judgment is the one from chapter 01, which is whether the people who would receive this data are the same people who can act on it, because that decides how far up the ladder your organization has any business climbing yet.