Validating staging results safely, without touching real users.

Image by author

Performance testing in production on Kubernetes. The same load test runs in a staging cluster and a production cluster side by side, while real users are never touched.

In the previous article, I turned staging benchmarks into a capacity plan for Traefik, the ingress controller replacing NGINX on our platform. Every number in that plan came from a staging cluster.

Production isn’t staging. It’s a bigger cluster, with other workloads on the same nodes and far more network rules. So before moving any real traffic, I had one question: do the staging results hold in production, and can I find out without putting real users at risk?

They didn’t hold. Production delivered less than half of staging’s throughput. Finding out why taught me more than the benchmark itself, and my first explanation turned out to be wrong.

Here’s how I tested production safely, what the gap turned out to be, and how I decided whether it mattered.

Start With a Test That Can’t Hurt Anyone

The timing made a production test possible. Traefik was already deployed in production, but it wasn’t serving customers yet. Every real request still went through NGINX. That meant Traefik could be loaded hard without any user noticing, as long as the test stayed completely off the customer path.

Each safety measure had a specific job:

  • A separate path. k6, an open-source load-testing tool, ran inside the cluster and sent traffic straight to the Traefik pods. There was no DNS or firewall change, so real users kept going through NGINX.
  • A unique test hostname. No real user uses it, so the test route couldn’t collide with a real one.
  • A test certificate. The test route used a self-signed certificate, so it never requested a real one or used up certificate-authority rate limits.
  • A throwaway backend. A static page served the test traffic. I checked it on its own first, far above the test’s load, so it couldn’t become the limit.
  • A narrow network rule. Only the k6 pods were allowed to reach Traefik on the test path.
  • One label on everything. Every test resource carried the same label, so a single command deleted them all, and a second command confirmed nothing was left.
  • A quiet window. Traefik was idle, but it shared nodes with real workloads, so I ran the test at a low-traffic time and watched node CPU and memory throughout.
The customer path stays exactly as it is. The test runs on its own path, inside the cluster.

One more step protects the conclusion rather than the users: run the test in staging first. Before touching production, I ran it in staging again. It reproduced about 9,000 requests per second on 2 pods with 2 cores, close to the 8,400 measured in the last article. That proved the test itself was trustworthy, so any difference in production would belong to production.

Run the Same Test in Both Places

A comparison is only fair if everything else matches. Both clusters ran:

  • Traefik: the same version, the same Ingress rules, 2 pods with the autoscaler pinned, and a 2-core CPU limit per pod
  • Backend: the same static page, sized so it was never the limit
  • Load: the same k6 sweep from 4 to 1,600 concurrent users, a 30-second warm-up at each level, keepalive on, and three trials per level reported as the median

The only configuration difference was memory. Production pods had a larger memory limit, and memory never came close to being the issue.

I also set the pass mark before running anything. Results between test sessions already moved by 20 to 30% in staging, so production passed if it landed within that range of staging.

Traefik, 2 pods × 2 cores, same test in both clusters, median of 3 trials.

It didn’t pass. Production peaked at about 3,800 requests per second, 42% of staging’s 9,000. p95 latency at the peak, the response time that 95% of requests beat, rose from about 14 ms to 36 ms. Both clusters had zero errors at every level, and the curves had the same shape. Production simply topped out lower.

A gap that size is easy to panic over. Before chasing the cause, I checked what it meant.

Check the Gap Against What Matters

Even at 42% of staging, production’s peak was still more than 40 times our real peak traffic. It wasn’t a capacity risk, and it didn’t block the migration.

That didn’t make it safe to ignore. An unexplained gap is a risk in itself. It could be something that gets worse as traffic grows, or something that also slows down other services on the same nodes. And if the cause was Traefik, the capacity plan from the last article would be wrong.

So the migration kept moving, and the investigation started in parallel.

Find the Cause One Layer at a Time

First, I ruled out the obvious suspects. Each one was either identical in both clusters or had plenty of room:

  • Node CPU: production nodes were 13 to 42% busy, with most cores idle
  • Node memory: no out-of-memory kills and no evictions
  • Hardware and operating system: the same machine type, image and kernel
  • Backend: far above the test’s load
  • Traefik configuration: identical in both clusters

Then I made my first guess. At the peak, each production CPU core handled about a third of the work of a staging core, about 1,030 requests per second against 3,140. Production Traefik was also using 92% of its CPU limit, against 72% in staging. It looked like the production hosts had slower CPUs.

It was a reasonable guess, and it was wrong. The problem was the evidence. Throughput depends on the network as well as the CPU, so a throughput test can show that something is slow, but not where.

So I changed one layer at a time, and added the test that was missing: one that uses the CPU and nothing else.

  • CPU and memory benchmark on the node, no network at all: 11.3 seconds in production against 10.3 in staging. CPU steal, the time a virtual machine loses to other machines on the same host, was 0% in both, and the clock speed was the same.
  • Backend alone, plain HTTP, Traefik bypassed: 18,400 requests per second in production against 40,200 in staging, 2.2× slower with neither TLS nor Traefik involved.
  • Time per network call: 1.47 ms in production against 0.66 ms in staging, 2.2× slower again.
  • Network policies in the cluster: about 17,300 in production against 12,000 in staging, so production’s network rule set was 45% bigger.
How much worse production was than staging on each check. 0 errors in every test.

The CPU was fine. Everything that crossed the network was about twice as slow, even without Traefik.

The best explanation is the cost each connection pays to cross the cluster’s network layer. On these clusters, that layer is Cilium, the network plugin, which applies network policies and tracks connections using eBPF, small programs that run inside the Linux kernel. Traffic to a Kubernetes Service address also goes through kube-proxy, which turns that address into a pod address using routing rules. Both do more work per connection in a bigger cluster, with more policies, more workloads and more connections to track.

That’s our best explanation, not a proven one. The test couldn’t separate kube-proxy’s share from Cilium’s. A follow-up that sends traffic once to a pod address and once to a Service address would split them.

When a result looks like a CPU problem, add a test that uses the CPU and nothing else. That one test turned “slow hosts” into “slow network path” and pointed any future fix in the right direction. Faster CPUs wouldn’t have helped.

If the cost sits in the platform’s network path, it doesn’t belong to Traefik. It should hit any controller on that path. That was the last thing to check.

Compare Against What You Run Today

Comparing production Traefik with staging Traefik answers “is production slower?” It doesn’t answer the question that matters for a migration: will users notice the switch? For that, the fair comparison is Traefik against the NGINX serving customers today, in the same cluster, on the same path.

I ran both back to back, through each controller’s own Service, with the same domain, the same HTTPS requests and the same steady rate, set above our real peak traffic. Each run lasted three minutes after a short warm-up.

Production, same cluster, same HTTPS path, same domain, steady rate above our real peak, 0 errors in both runs.

Where almost every request lands, the two were effectively the same. The median was 29.6 ms for Traefik and 29.3 ms for NGINX, and p95 was within 8 ms. Neither returned a single error, and Traefik used less CPU per pod than NGINX at this load.

Traefik’s p99 was the one clear difference: 328 ms against 91 ms. It’s still well under a second, and it’s worth watching after the switch. At the very tail, both controllers hit the same worst case of about 2 seconds. A worst case that matches across two different controllers points to the shared network path again, not to either controller.

So Traefik was slower in production than in staging, but so was the platform under it. Against the NGINX already serving users, it was on par for almost every request.

What It Means for the Capacity Plan

Three conclusions came out of this:

  • Staging’s absolute numbers don’t transfer. Production’s peak was 42% of staging’s, so any plan built on staging’s raw throughput would overestimate production by more than double.
  • Ratios against the current controller are safer. The plan from the last article compared Traefik with NGINX, and both run on the same network path in production. A slower network path should slow both controllers alike, so the ratio is the safer number to plan with. Production’s lower peak still left more than 40 times our real peak traffic.
  • Fix the right layer. If more capacity is ever needed, the work belongs in the network path, such as kube-proxy and Cilium settings. Bigger CPUs wouldn’t move the number.

With those three in hand, the performance question was settled, and the migration could move on to the next risk.

Checklist

Before you trust a production performance test, check each of these. Each one either protects your users or protects your conclusion.

  • Keep the test off the customer path. Use a separate route, a unique hostname, a test certificate and a narrow network rule, with no DNS or firewall change.
  • Prove your test in staging first. If it reproduces staging’s earlier numbers, any difference in production belongs to production.
  • Set the pass mark before you run. Base it on the run-to-run variation you already know, so the result can’t move the goalposts.
  • Judge the gap against real traffic. A large gap with plenty of headroom is an investigation, not an emergency.
  • Change one layer at a time. Include a test with no network at all, because throughput alone can’t tell you where the time goes.
  • Compare against what you run today. The fair question for a migration is whether the new system is worse than the old one, on the same path.
  • Clean up with one command. Label every test resource, delete them together, and confirm nothing is left behind.

Together, they let you test production hard without asking your users to take the risk.

Final Thoughts

I expected the production test to be a formality. Staging had already answered the capacity question. Production disagreed by more than half, and my first explanation for it was wrong.

The test that settled it wasn’t a bigger load test. It was a small one that left the network out. And the comparison that mattered wasn’t staging against production. It was the new controller against the old one, on the same path.

So don’t assume staging tells the whole story. Staging tells you how your software behaves. Production tells you how your platform behaves. Test there safely, and compare against what you already run.

What’s Next

Performance turned out to be fine. The bigger surprises were in the configuration: certificates and settings the old controller had quietly tolerated. In the next article, I’ll show how to find those problems before a migration finds them for you.


Performance Testing in Production on Kubernetes: A Practical Guide was originally published in Level Up Coding on Medium, where people are continuing the conversation by highlighting and responding to this story.