The awkward number on NTNT's newest benchmark page is not 90,775 requests per second. It is minus six percent.
In a same-machine comparison, NTNT 0.5.5 served a plain-text response at a median 75,649 requests per second, down 5.9% from version 0.4.2. The JSON route fell 6.4%, from 81,038 to 75,841 requests per second.
Those are not catastrophic losses, but they are not flattering gains either. Numbers like these are easy to lose between a test report and a public release page. This time, they stayed.
The benchmark did not owe anyone a happy ending
Version 0.5.5 was mainly a reliability and operations release. It added multi-process Redis workers, monitoring start deadlines, OAuth routing fixes, and an ARMv7 build. None of that promises a faster hello-world response.
The broader benchmark still contains large numbers: nine frameworks, seven workloads, three measured runs per cell, and 90,775 requests per second for NTNT's plain-text route. That figure belongs to a fresh cross-framework run. It cannot be compared directly with an older result taken under different conditions, which the report says plainly.
So the project also ran a smaller controlled comparison. Versions 0.4.2 and 0.5.5 used the same little application, official binaries, identical containers, four HTTP workers, and alternating run order. That is where the two stable slowdowns appeared.
Publishing them changes the meaning of the page. It stops being an advertisement with a stopwatch and becomes a record someone can argue with.
Minus six percent is an observation, not a diagnosis
The controlled test was narrow. Each version got three five-second runs after a two-second warmup on four logical CPUs. It covered plain text, JSON, route parameters, and a small JSON request body, but no database workload. The parameter results varied too much on 0.4.2 to support a clean comparison, and the 0.5.5 request-body results were unstable by the run's own 10% spread threshold.
That leaves two stable observations. It does not identify the responsible commit, prove that production applications became 6% slower, or tell us whether a longer run would land on the same number.
This is the part performance charts often lose. A measurement can be valid for its setup and still be modest in what it says. "We measured a regression in two tiny routes" is useful. "The release is slower" would be broader than the evidence.
The footnotes are part of the result
The benchmark repository keeps the raw outputs, fixture revision, runtime versions, database seed, validation samples, and exact launch command. It also records a parser mistake in the original memory-soak report: wrk printed a duration in minutes, the parser expected seconds, and the request counts vanished. The project kept the original files, corrected the parser, and generated separate validated outputs rather than quietly replacing the evidence.
That correction is less glamorous than a requests-per-second trophy, but it tells a prospective user more about the benchmark. Numbers pass through software written by people, and that software can be wrong. Keeping the faulty report beside the corrected one makes the repair inspectable.
The same restraint appears in the two-minute memory test. NTNT's process set grew by 9 KiB while serving more than 8.5 million requests, but the report says a short test cannot prove the absence of a slow leak. A green result gets a boundary too.
Benchmarks are useful when they make future embarrassment possible. Publish the conditions, keep the raw measurements, label unstable runs, and a later release can fail against something real. Hide every regression and the chart may look better, but it stops functioning as an instrument.
NTNT 0.5.5 did not win every comparison with its younger self. The public record is better because the slower result survived editing.