Designing systems that solve the right problem

Six things decide whether the thing you build gets used:

1. Design for your internal customers, the people who will use the system, and not for yourself.
2. Make it usable by them.. because you thinking something is simple does not mean your users do.
3. Design something supportable by many people rather than by specialists, including a vendor who answers in an emergency and an integrator you can hire when you need one.
4. Design something survivable, because nobody wants to rebuild their platform every year, and a product that has been around for a few years is that way for a reason.
5. Operational simplicity beats operational complexity every single day.
6. Understand the real requirement before you design anything, from the business point of view rather than yours, and understand the constraints you have to work inside.

Get them wrong and you ship something correct that nobody adopts, which costs the same as building it and returns nothing.

No business owner has ever asked for GitOps, a service mesh, or a second cluster. The tightest constraints on your design sit with people who do not work in technology, and are often forgotten about:

1. What an hour of outage actually costs, in orders, penalties or production.
2. Whether the team can run the process on paper for a day, or for a week.
3. When the real peak falls, and what causes it.
4. Which requirement is fixed by an obligation, and which is a habit nobody has questioned in years.

Getting the answers to these questions is what makes you a better engineer, and allows you to design a more fit-for-purpose solution.

The business gives you symptoms, and more often it gives you the solution it heard about somewhere. "We need GitOps" is a solution somebody already picked, and underneath it is a release process where changes reach production without a record of who approved them, which is the problem. Underneath that is an auditor who needs an attributable approval for every production change.. the only one of the three you can test a design against.

That bottom step is often satisfiable in a way the top step never suggested. An auditor who needs attribution can be satisfied by requiring pull request approval on the manifests you already keep in Git, with cluster access removed from everyone but the reconciler. You may well end up implementing GitOps, and now you are implementing it against a stated requirement rather than because it was the word in the room.

You get down the ladder by asking what the answer is for, again and again, then asking what happens if it is not there. The second question is where the numbers are, because a business owner who cannot give you a recovery time objective can still tell you what their team does on the morning the system is down. Ask how long they can keep that up, and what breaks first when they cannot.. a service desk that falls back to a spreadsheet and a phone until the next morning has just given you an eighteen hour recovery window without using the word recovery.

There is an order to this that architects have used for thirty years, and it is worth knowing even if you never use the vocabulary. What the business does comes first, then the information it needs, then the applications that handle it, and the technology comes fourth.. because each of those sets the constraints on the next. Start at the technology and you are working the sequence backwards.

The reverse-derived business case is the most common failure in platform work, and it is easy to spot once you know the shape. You pick the technology first, on criteria available to you at the technology layer, which are technical criteria, and then you write the requirement that the technology satisfies. That requirement cannot fail, because it came from the answer.. and the difference is usually apparent to everyone in the room.

It shows up in the language of the requirement itself, so a requirement that says the platform must support progressive delivery with automated canary analysis has a product in mind. A requirement that says a failed release must be detected and reverted within five minutes without a human decision does not, and any number of things can satisfy it, including a readiness probe and a rollback command in a runbook.

The fix is to state the capability you need so that it can be tested, before any product is named, and then evaluate products against it.

Every clause in that statement is checkable and no product is named, which means the evaluation produces an answer rather than a preference. A preference can be overruled by whoever is most senior in the room, and an answer cannot. The last clause is the one most requirement sets leave out and most projects are eventually defeated by, and chapters seven and eight are about that clause.

Where two candidates both meet the requirement, the choice between them does not matter, and the right output is to say so and stop. Both deliver the capability, so you can decide on price, on an existing relationship, or on which one your team already knows. Engineers find this uncomfortable because the technical differences are real and interesting, but they are differences that do not reach the requirement, and arguing them costs weeks the project does not get back.

The finding has to work in the other direction too. Where a capability has no product behind it, or has one nobody has demonstrated at your scale, the honest output is that the technology does not yet exist in a form you can operate.. there is no recommendation to make. What follows is a reduced requirement, a manual process kept deliberately, or a decision to wait.

Requirements tell you what the system has to do, and constraints tell you what you are allowed to do about it. Both are given to you rather than chosen by you, and a design that treats the second set as friction is correct in isolation and undeliverable in practice.

Fiscal constraints are the ones people expect, and they are rarely a single number. There is what you can spend this year against next year, what is capital against what is operating expense, and what has already been committed to a contract you cannot exit.. a cluster design that needs new hardware in a year when only operating spend is available is undeliverable, whatever it saves.

Operational constraints are the ones that end projects after go-live. How many people you have, what they already run, what hours they cover, and what else is on their plate during your rollout. A platform that needs a specialist on call is constrained by whether you have two of that specialist, because one is not a roster.

Then there are the constraints nobody thinks to tell you, and you find them by asking:

1. Change windows that only open twice a quarter.
2. A regulator that requires data to stay in one country, which decides your cluster topology before you draw it.
3. Hardware lead times that make a technically superior option undeliverable inside the window.
4. A support contract that expires in eight months, which is either a problem or the budget line your whole sequence can ride on.

Every platform has customers, and yours are the application teams who have to deploy onto it. Like any supplier, you learn quickly that a product the customers do not like is a product that does not get used.. and unlike a real supplier you often cannot tell, because nobody cancels an internal platform, they just route around it.

This is the failure that looks most like success from the inside. The cluster is up, it is patched, it is monitored, the architecture is defensible, and adoption stalls at four namespaces because the teams you built it for found the deployment path slower than what they were already doing.. you have built something excellent that nobody chose.

None of that is laziness, and the mechanism is simple enough. An application team is measured on shipping their application, so they will take whatever route ships it, and your platform competes with that route on the only axis they care about, which is time from commit to running. Add a two day access request to your path and you have lost to a cloud account with a credit card behind it, regardless of how much better your governance is.

So you talk to them before you design, the same way you talked to the business in chapter one:

1. How do you deploy today, and how long does it take.
2. What in that process do you hate.
3. When you need a new environment, who do you ask, and how long do you wait.
4. What did the last incident cost you in time.

Those answers are requirements, and they are as real as the auditor's date in chapter one.

You know what a namespace is, what a resource request does, and why the pod is pending. None of that is knowledge your users have, or should need, and a self-service portal that assumes it is not self-service.. it is a form in front of a system that still needs you standing behind it.

The test is whether a developer who has never seen your platform can get a working service into their own namespace without asking anyone. Walk the path yourself and count the places they stop.

Every one of those stops is a place where your platform loses to the alternative, and most of them are not technical problems. A field labelled "memory limit" with no default is a question a developer cannot answer, and they will either guess low and get evicted or guess high and waste half your cluster. Give it a sensible default, name the profiles in language they use, and the question disappears.

The same applies to what happens when it breaks, because a failed deployment that reports ImagePullBackOff has told an experienced operator everything and a developer nothing, so the useful platform translates it.. the image tag does not exist in the registry, here is the tag you asked for, here are the tags that do exist. That translation is a product feature, not a nicety, and it is the difference between a team that self-serves and a team that opens a ticket.

Usability also decides your support load, which makes it your problem rather than a favor you are doing them. A path with six places to stop generates six categories of ticket, all of which come to you, forever.

Working is the easy half, and a platform is finished when people choose it, which means you need a number for whether they are choosing it. Uptime does not tell you.. a platform can hold four nines while three quarters of the estate deploys somewhere else.

Four readouts tell you the truth:

1. How many teams and workloads are on the platform, against how many exist.
2. Time from commit to running in production, measured by the team rather than by you.
3. Unmanaged cloud spend, which finance can give you and which measures what got built elsewhere.
4. How many of your tickets are requests for things the platform was supposed to let people do themselves.

The third one is the most honest and the least comfortable, because shadow infrastructure is a vote. Every account somebody opened outside your platform is a team that judged your path slower than the alternative and acted on it, and finance can usually tell you what that judgment costs the company each month. That number moves conversations in a way an architecture diagram does not.

The fourth one measures the gap between what your platform claims and what it delivers. If a third of your tickets are "can you create a namespace for me" or "can you give me access to the logs", then self-service is a label rather than a capability, and every one of those tickets is a permanent tax on your own team.

Baseline all of it before you start, because you cannot demonstrate a benefit you never measured, and be willing to report it when it comes back negative. A platform team that reports a stalled adoption number and a plan is trusted. A platform team that reports uptime while the estate drifts elsewhere gets found out later, in a budget conversation.

The design that only its author can run is a liability on the balance sheet, however well it works, because the organization cannot patch it, cannot recover it, and cannot let that person take a holiday. Support depth is a ladder with three rungs, and they sit further apart than they look.

The first rung is running it from a runbook, rolling back a release, draining a node, following procedures somebody else wrote. The second is diagnosing something nobody has documented, a CNI problem, a scheduling failure, an admission webhook rejecting deployments at three in the morning.. that needs a working model of how the thing behaves rather than a list of steps. The third is changing the cluster architecture safely, which needs to know why it was built this way and what a change will disturb.

Most designs assume the third rung because the person drawing them is standing on it, and most organizations fund the first, because a runbook and a handover session are what a project plan pays for.

Prerequisite is the input most often assumed rather than checked, and training does not fix it. Training converts adjacent skill into operating skill, and it does not create the adjacent skill. A team that has run Linux at scale for a decade can be trained onto Kubernetes and reach the second rung inside a year. A team whose whole estate is Windows virtual machines, whose on-call staff have never held a shell prompt, is much further away than a five day course.. and virtual machine fluency is adjacent to Kubernetes rather than equivalent to it, because the failure modes are different and the debugging is different.

The cost moves with how deep a rung you need and how far the technology sits from what your team already runs. Take a platform whose steady operational load is 20 engineer days a year for a team that already runs the same class of thing. Something adjacent lands near 30 days in the first year while the team builds the model, settling back toward 22 or 24 once it is familiar. Something genuinely novel, where nobody holds the prerequisite, runs 50 to 60 days in the first year. It never settles while the knowledge sits with one person, because the day that person is unavailable your team drops to the first rung and every incident becomes an escalation.

Your own team is the innermost ring of support and it is never the only one you need, so a design has to answer where help comes from when the people who built it are out of ideas. That question has four answers, and you should be able to give all four before you commit.

The vendor ring is the one people assume and rarely test. A support contract is worth what its response model says under load, so ask what the target response is for a production-down case, whether it is engineering or a script on the other end, whether your time zone gets a human, and what has to be true for them to support your configuration.. because a vendor who supports the product but not the way you have deployed it is not support. Where the answer is a community forum, that is a real answer and it belongs in the design as a stated risk rather than a gap you hope nobody notices.

The integrator ring is the one nobody checks until it is needed. When a project stalls or somebody leaves, can you hire a firm who has done this before, and how many of them exist within reach. A technology chosen because it is elegant, with three consultancies in your country who know it and a six week wait for any of them, has a delivery risk that no architecture review will surface.

The labor market ring decides what happens over years rather than in an emergency. Can you replace the person who holds the knowledge, and what does that cost. Look at what the job ads in your city actually ask for, because that is a better signal than conference popularity, and a technology nobody local hires for is a technology you will be paying a premium to keep running for as long as you run it.

Nobody wants to rebuild their platform every year, so the useful question about any technology is whether it will still be here, and still supported, when you are three years into running it. A product that has been around for a few years is that way for a reason, and that reason is usually that enough people depend on it to keep it funded.

The signals are all public and none of them require a prediction. How old is the project, and how many major versions has it shipped. How many organizations contribute to it, because a project with a single corporate maintainer has one board meeting between you and an end of life notice. What is the release cadence, and does the upgrade path between versions get documented and tested or does each one become a migration. Is there a commercial entity selling support, and has it survived a funding round.

Version support windows matter more than people expect, because they set your floor for operational effort. A platform releasing minor versions every few months with a support window of about a year commits you to a permanent upgrade cadence for as long as you run it, and that cadence is a line in your operating budget rather than a one-off project cost.

None of this is an argument for adopting nothing, which fails in its own way, because an estate that never adopts anything ends up expensive too and much harder to hire for. The workable frame is a novelty budget, because every project can afford a small amount of unfamiliar technology, and you spend it where the unfamiliar thing is the reason the problem gets solved.. spend it anywhere else and you have bought risk the organization did not agree to carry and did not price.

Every component you add brings four things with it that have nothing to do with what it does:

1. A patch stream.
2. An upgrade cycle you have to test.
3. A monitoring integration.
4. A failure mode that interacts with the failure modes of everything already there.

That last one is why complexity compounds rather than adds. Two components have one relationship between them, four have six, and eight have twenty-eight.. and each relationship is a place where an upgrade to one thing breaks another and somebody has to work out which of them is responsible.

The arithmetic on duplication is the clearest case, because two platforms delivering the same capability is not twice the cost, because each carries its own patch stream, support agreement, upgrade testing, monitoring integration and scarce person, and the interaction between them adds a cost neither has alone. Where one costs 20 engineer days a year to operate, two comparable ones commonly run 50 to 60 rather than 40, so the second costs roughly 1.5 to 2 times what the first did while delivering nothing the first did not. Two GitOps reconcilers, or two observability stacks, is the most common version of this in Kubernetes estates and it usually arrives by accident rather than by decision.

Simplicity is not the same as doing less, and this is where the argument gets misread. A design with one reconciler, one ingress controller and one metrics stack, all of them boring and all of them understood by four people, delivers more capability in practice than a design with three of each that only one person can reason about. The question to ask of every addition is what it removes the need for, and if the answer is nothing, you are adding load rather than capability.

Every argument above comes down to a number somebody has to approve, and infrastructure assets live for five to seven years, so the number that matters is rarely the one on the quote. You show the derivation rather than the result, because a stakeholder who can follow it can defend the decision when you are not in the room, and one who cannot will fall back on the quote.

The cost curve of availability

Take a workload needing four nodes for its steady load. A single cluster with a tested etcd and volume restore path is 4 nodes, 4 node licenses where you pay per node, and roughly 4 engineer days a year of restore testing. A second cluster in another site, active-passive with replicated state, is 8 nodes, the interconnect, a second copy of the storage, and around 16 engineer days a year to exercise the failover quarterly. Multi-region with state replicated live carries the same 8 nodes plus the replication tooling, commonly another 30 to 50 percent on the software line, plus the testing burden of a much more complex failure model.. call it 24 engineer days.

Infrastructure roughly doubles from the first tier to the second and is flat from the second to the third, the software line doubles and then rises again, and the operational load runs 4, 16, 24, and the total lands near 1x, 2.3x and 2.9x. Show that curve and the business frequently declines the third tier and sometimes the second, once it is priced against the outage cost it buys down.

Build against run

Six weeks of engineering to build your own deployment pipeline is roughly 30 engineer days, or about 12,000 dollars fully loaded at 400 a day. Over five years that pipeline carries hosting, patching, upgrade testing when the cluster version moves, on-call load, and the cost of keeping two people current enough to change it. At a conservative one day a month of combined attention that is 12 x 5 x 400 = 24,000 dollars of labor alone, before any infrastructure, which puts run at twice build before the first invoice arrives.

Set that against an off-the-shelf reconciler that somebody else upgrades and documents, and the comparison the business is actually making becomes visible. Realistic estates land between five and fifteen times the build cost across the life, and almost every argument for building rather than adopting is made against the 12,000 and not against the 24,000 plus platform.

Naming what you assumed

Every figure above is illustrative and rests on stated assumptions, and the three inputs that move them most are the day rate, the testing cadence and the licensing model, none of which is constant across organizations. Do not reproduce these numbers.. run the derivation with your own, in front of the person approving it, naming what you assumed and what margin you applied. A number whose derivation is visible survives challenge, and one that arrives finished does not.

A design that nobody approves is a design that does not exist, and the reason competent designs stall at the approval stage is almost never that they are wrong. The design is right, the audience cannot evaluate it, and rather than say so they defer, ask for more detail, or approve it without understanding it.. all of which cost you time and none of which improve the design.

What you show somebody depends on what they are responsible for, and the same design produces several different pictures:

1. The security owner needs trust boundaries, what crosses them, and where authentication and authorization decisions get made.
2. The finance owner needs the cost lines, when they land, and what is capital against recurring.
3. The application teams need what changes for them, what they gain, and what they have to do differently.
4. The executive sponsor needs the capability being bought, the risk being retired, and the sequence.

Show everyone the same diagram and it is almost always the technology view, which answers a question only you were asking. Constructing a view is selection, so you decide what the viewer is responsible for, include everything relevant to it, and leave out everything that is not. That means a good security view can hold detail the technology view never had.

Then explain the design by the outcome it protects rather than by the mechanism it uses. The business hears that the release record satisfies the auditor and that a failed deployment reverts without anyone waking up, and they never needed to evaluate the mechanism to approve that. Explain the same design by its mechanism and you are asking them to trust you instead, which holds until the first thing goes wrong.

This is the second half of being a dual-linguist, and it is the half that gets a design funded. The first half is hearing the constraint in the language it arrives in, and this half is returning the answer in the language the listener already uses, so nobody has to learn your vocabulary in order to say yes.

The way to learn this is to run it on something small and reversible before you run it on something that matters. Take a change you are already planning, one you have already decided how to build, and work backwards. Write down which business capability it serves, find the person who owns that capability, and ask them what happens when it is not there. Derive the constraint from their answer, then see whether the design you already chose is still the one you would choose.

This lesson sits outside the Kubernetes sequence deliberately, and the sequence is where to go if you want the technology rather than the decision behind it. The primer on how Kubernetes is put together is the place to start, and the companion primers on Helm, GitOps and observability follow it.