Infrastructure Ladder

Claims, written before the code

• Held
  Of the 259 tests real Valkey 9.1.2 passes in unit/type/incr, unit/type/string, unit/keyspace, unit/expire (suite tags needs:debug, needs:repl, needs:reset excluded), ours passes ≥ 90%, and 100% of unit/type/incr

  Held. 251/259 (96.9%); incr 31/31

• Held
  GET on an existing key and INCR on an existing integer, including parse and reply, allocate 0 B per command

  Held. 0 B for 16 pipelined GETs and 16 INCRs

• Held
  valkey-benchmark -t set,get -n 1000000 -c 50 -P 16 reaches ≥ 50% of valkey-server's requests/sec

  Held. SET 91.7%, GET 82.1% (medians of 3 alternating rounds). But it uses about 4× the CPU. See below.

• Held
  After loading GeoNames cities15000 into both servers with valkey-cli --pipe, full dumps are byte-for-byte identical

  Held. 34,432 keys, identical

External judge: Valkey 9.1.2 test suite, run against both servers

Throughput (valkey-benchmark, 50 clients, pipeline 16) and CPU

What surprised us

• The judge found two real bugs before it found any missing features.
• One unsupported command can hide 58 passing tests. Without --durable, the first PSUBSCRIBE in string.tcl aborted the rest of the file, and the score read 68.7%. The flag is upstream and applies to both servers equally.
• Requests per second was the wrong headline. Ours reaches 82–92% of Valkey's throughput while using about 4× its CPU (3.2 µs vs 0.8 µs per request). Command execution is only 1% of the samples; thread-pool spinning (15%) and lock spinning (6%) are larger. Turning the spinning off saves 15% of the CPU; the rest is the cost of three async loops and thread hops per connection, against Valkey's single event loop.
• INCRBYFLOAT 10.5 then 0.1 returns 10.59999999999999964, and that's correct. Valkey formats with %.17Lf, and on Apple Silicon long double is a plain double. Real Valkey on the same Mac gives the same bytes.

Raw evidence

• README.md
• judge/geonames-load-6380.txt
• judge/geonames-load-6390.txt
• judge/geonames-sha256.txt
• judge/suite-ours.txt
• judge/suite-reference.txt
• judge/summary.txt
• evidence/benchmarks.md
• evidence/cpu-experiment.csv
• evidence/throughput-cpu.csv
• evidence/throughput-env.txt
• evidence/throughput-raw.csv
• evidence/throughput.md

Claims, written before the code

• Held
  Tokenizing an in-memory document allocates 0 bytes

  Held. BenchmarkDotNet Allocated = 0 B on all three files; Tokenizing_allocates_nothing unit test

• Held
  JSONTestSuite: 100% of y_ accepted, 100% of n_ rejected, no crash

  Held. 95/95 and 188/188, including n_structure_100000_opening_arrays

• Held
  Throughput within 2× of Utf8JsonReader

  Held, with margin. 0.51–0.64× of its time (ours is faster)

External judge: JSONTestSuite @ 1ef36fa

Benchmarks: Apple M1, .NET 10.0.13

Mutation testing: Stryker.NET

What surprised us

• System.Text.Json accepts invalid UTF-8 at parse time. On 20 of the 35 i_ files, JsonDocument.Parse succeeds and the error only appears later, as an InvalidOperationException when a string is read. Utf8JsonReader checks strings only for quotes, backslashes and control bytes. Ours rejects these files when it reads them.
• UTF-8 validation is nearly free. Utf8.IsValid over a whole file takes 40–64 µs, 2–7% of a tokenize pass. We expected the validation to be the reason we'd lose to STJ. It wasn't.
• We're faster than Utf8JsonReader, and the cause hasn't been measured. The likely reason is that STJ does work we don't: line and position tracking, support for streaming partial buffers, and comment and trailing-comma options. That is a hypothesis; no profile has confirmed it yet.
• STJ's timing on canada.json moved up to 20% between runs (3,267, 3,975 and 3,782 µs in three processes), while ours stayed within 1,904–1,938 µs. All runs are kept in the evidence.
• dotnet-counters reported allocation that wasn't ours. It showed 16–49 KB/s across two captures, which turned out to come from the in-process metrics listener it enables. A process-wide counter can't prove a per-call zero; BenchmarkDotNet and a unit test can.
• The tooling needed checking too. Stryker's first run reported a 0% score because it used the old VSTest runner while the repo runs on Microsoft.Testing.Platform. BenchmarkDotNet's disassembler doesn't run on macOS, so CI passes --disasm on Linux instead.
• The tests found a real bug. ValueSpan kept the previous token's bytes on [, {, ] and } tokens.

Raw evidence

• README.md
• judge/jsontestsuite.txt
• evidence/benchmarks.md
• evidence/counters.csv
• evidence/counters.md
• evidence/divergences.md
• evidence/stryker.md