Terraform, Kubernetes, GitOps, Crossplane and Service Mesh
You have a box (or a few). You SSH in. You install packages, copy files, start a process, maybe write a systemd unit so it restarts on reboot. If you need another server, you spin up a VM and do it again.
That works beautifully for a handful of machines. The pain begins when the numbers grow, and it grows along a few specific axes. Every tool below is a response to one of them, so hold onto these.
The first pain is repeatability – you SSH into box #2, try to reproduce box #1, and forget a step or the package version drifts. Your servers become snowflakes. The second is placement – five VMs, twelve services, and you're manually playing Tetris with CPU and memory. The third is recovery – a box dies at 3am and something has to notice, move its workloads, and repoint traffic. The fourth is change without downtime – shipping v2 while v1 keeps serving.
Keep one idea in mind as we go, because it's the thread running through everything: you declare desired state in version-controlled text, and a continuous loop keeps reality matching it. Every layer below is that same loop pointed at one more slice of reality.
The First Bridge: From a Process to a Container
Before Kubernetes, learn the step just below it. A container is basically a process on your Linux box – same kernel, same ps, same signals – but with its filesystem, network, and resources isolated via namespaces and cgroups, kernel features you already have. docker run is genuinely "start this process, but give it its own packaged filesystem."
That single step kills the repeatability pain. The image bundles the app and its dependencies, so "works on my box" becomes "works anywhere the kernel runs." The host no longer needs the right packages installed – they live inside the image.
Internalize one thing before touching k8s: a container is a process, an image is its packaged filesystem. Everything above sits on that.
Kubernetes: The Loop You'd Eventually Build Yourself
Imagine running an agent on every box that reports "here's my CPU/RAM, here's what I'm running," plus a central brain you talk to with statements like "I want 3 copies of this container, always." The brain picks boxes with room, starts the containers, and – the key part – keeps checking. If a container or a whole box dies, reality no longer matches your declaration, so the brain fixes it.
That's Kubernetes. The brain is the control plane, the agents are the kubelets, and your intent is a YAML manifest. You stop issuing commands ("start this") and start declaring facts ("this should be true"), and a reconciliation loop keeps them true.
That maps straight onto the pains: placement becomes the scheduler's job, recovery becomes automatic, zero-downtime deploys become a built-in rolling update, and repeatability comes from containers plus git-committed manifests.
Why it's great: one uniform, declarative control plane over everything; self-healing that stops a lot of 3am pages; and the individual machine abstracted away so you add nodes to grow capacity.
Why not: the complexity is enormous – a scheduling layer now sits between you and the process, so debugging needs both Linux and k8s. There's a real payoff floor: for three services on two VMs, systemd units and a deploy script are simpler and cheaper. Stateful workloads are the classic trap – databases care about this disk on this machine, so managed databases are often the wiser call. And the cluster itself needs patching and upgrading. "K8s for all infrastructure" is a slogan, not a rule: great for stateless services at scale, an over-engineered liability for small or stubbornly stateful things.
Terraform: The Same Idea, One Layer Down
Kubernetes manages what runs inside the cluster. But something has to create the cluster – the VMs, network, load balancer, DNS, IAM, the managed database beside it. Click around a console and you've reintroduced the snowflake pain one layer lower.
Terraform applies the reconciliation idea to everything underneath. You write, in text, "a network with these subnets, three VMs, a managed Postgres, and a cluster." Terraform diffs your declaration against what exists and makes reality match. Crucially, terraform plan shows you that diff before anything changes – the dry run your shell scripts never had.
The two are complementary layers of one idea: Terraform builds the foundation, Kubernetes runs the apps on it, and you get one declarative, git-tracked mental model from the metal up. The caveat is state – Terraform keeps a file of what it believes it created, and managing and locking that file is a real responsibility. Change things by hand behind its back and the states drift.
GitOps: Make Git the Source of Truth
There's still a gap: how do the manifests get applied? Probably someone runs kubectl apply from a laptop or CI. So the cluster's actual state can drift from git – someone kubectl edits a hotfix at 3am, forgets to commit, and now git lies.
GitOps closes that with a simple move: put an agent inside the cluster (Argo CD, Flux) whose only job is to watch a git repo and continuously make the cluster match it. You stop pushing changes in; the cluster pulls from git.
Your day changes shape. To deploy, you open a pull request; on merge, the agent reconciles the cluster within a minute. To roll back, you git revert. To know what's in production, you git log. Drift gets detected and corrected automatically, so git stays honest. Humans no longer need powerful kubectl credentials – they need write access to a repo, and the cluster pulls rather than exposing a push endpoint.
The caveats: another moving part, it wants everything declarative, and it governs desired state but not secrets – you add a companion like sealed-secrets or an external secrets operator.
Crossplane: Provision Cloud Resources the Kubernetes Way
Now the seam. GitOps has the cluster pulling everything from git – but your cloud resources live in Terraform, a separate tool with a separate apply and state file. Half your world is GitOps-driven, half isn't.
Crossplane erases the seam by teaching Kubernetes itself to manage cloud resources. You write a manifest saying "I want an RDS Postgres" as a Kubernetes object – same YAML, same control plane, same GitOps agent. The reconciliation loop now reaches outside the cluster. Where Terraform reconciles when you run it, Crossplane reconciles continuously: delete that database in the console and Crossplane recreates it, self-healing extended to your cloud account.
The bigger win is composition – a platform team defines a simplified PostgresInstance that expands into the database, networking, firewall rules, and credentials, so developers request a production-shaped database in two lines of YAML. You're building an internal self-service platform.
The caveats are real: Crossplane is younger than Terraform with a smaller ecosystem and a steeper composition learning curve, and it makes your cluster responsible for your cloud infrastructure, which couples blast radius. Many teams use both – Terraform to bootstrap the foundational, rarely-changing layer, Crossplane for the higher-level resources apps request daily.
Service Mesh: Reconcile the Network Between Services
The last corner. Once many services call each other, concerns appear in the space between them: encrypting traffic (mTLS), retries, timeouts, circuit breaking, sending 5% of traffic to a canary, and seeing who calls whom. Without a mesh, every one of these gets built into every service's code, in every language, consistently. That's the problem: cross-cutting network behavior scattered across every app.
A service mesh (Istio, Linkerd) pulls it out into the infrastructure. The classic mechanism is a sidecar proxy next to each app container; all traffic flows through it transparently. Your app makes a normal HTTP call; the proxy adds mTLS, retries, timeouts, traffic splitting, and metrics – none of it in your code, all configured declaratively and reconciled by the mesh. You get encryption everywhere, uniform reliability regardless of language, fine-grained traffic control for canaries, and a free map of who-calls-whom with latency and error rates.
On "we don't use one because it adds latency"
Half-right, half-outdated. The concern is real in origin: the sidecar model routes every call through two extra proxies. But that's typically single-digit milliseconds, often under one – usually negligible next to your actual database and network calls. If a request already spends 50ms in a database, a sub-millisecond hop rarely matters; the exceptions are ultra-low-latency hot paths, and even then it's specific paths, not the whole system.
Two nuances worth putting on the table. Not all meshes are equal – Linkerd's proxy is deliberately tiny and benchmarks far lighter than Istio, so "a mesh adds too much latency" is often really "Istio's footprint was too much." And the ecosystem has moved: ambient mesh (Istio's sidecar-less mode) restructures things specifically to cut the per-hop tax that drove objections like this. A blanket "meshes add latency" increasingly describes the 2019-era architecture, not today's options.
The sharper framing: a mesh is worth its cost when your number of services and need for uniform mTLS, reliability, and traffic control outgrow what you can sanely bake into each app. Below that threshold, skipping it is genuinely reasonable. Above it, the latency objection deserves re-testing against a modern lightweight mesh rather than being treated as settled – benchmark Linkerd or ambient mode on a representative path and see whether the real number matches the fear.
Where This Leaves the Map
Everything here is one reconciliation loop pointed at a new target. Kubernetes reconciles what runs inside the cluster, Terraform the cloud resources underneath, GitOps the cluster against git, Crossplane cloud resources from inside the cluster, and a Service Mesh the network between services. The through-line of the entire stack: declare intent in version-controlled text, and let continuous controllers keep reality matching it. Once that clicks, each new tool is mostly "which slice of reality does this one own."