musl-static vs glibc-static — comparison

The headline answer: glibc-static wins on train time (statistically significant at N=5, ~6 % faster on x86_64 and within 1 σ on aarch64), musl-static wins on binary size (~5-11 % smaller), and query is within noise on both sides. Memory footprint is identical (no allocator-behaviour delta shows up in our corpus).

Methodology

The bench pipeline (one run)

On the same deterministic synthetic corpus (~20k lines, seeded srand(42) in scripts/bench.sh), each target runs:

Wall-clock: train_s and query_s come from kenlm's own real: line; trim_s and build_s are shell date +%s. We also capture intermediate sizes and the model's perplexity on the held-out set.

Multi-run capture (N=5)

A single run is dominated by shared-runner noise (other tenants on the host, ephemeral filesystem cache, kernel scheduling). One sample per target was unconvincing — so every release CI run emits:

• dist/bench-<target>.tsv — 5 raw rows, one per back-to-back run, via scripts/bench-multi.sh
• dist/bench-<target>.agg.tsv — min / median / mean / stddev / max per numeric column, via scripts/bench-aggregate.sh (a portable awk script that does no Python/numpy dependency)

This is the only run-to-run noise model we trust. The bench corpus is deterministic so the synthetic spread captures machine noise alone. For the perf comparison to be valid, we report median ± stddev — a single point estimate would be hiding the variance on either side.

Hardware

GitHub-hosted runners: ubuntu-latest (x86_64, 2 vCPU / 8 GB), ubuntu-24.04-arm (aarch64, 2 vCPU / 8 GB), macos-14 (Apple M1, 3 cores), and windows-latest / windows-11-arm. Both x86_64 and ARM runners bill for the same job as one VM; jobs are co-located with unrelated tenants, so run-to-run spread is real. Numbers below are reproducible within 5-15 % of the medians.

Results (N=5 per target, 1 release run)

target                 median train_s   stddev train_s   ppl       lmplz_kb
─────────────────────  ─────────────  ──────────────  ────────  ────────
x86_64-linux-musl      ~0.086 s           ±0.002 s     19.82      3,743
x86_64-linux-gnu       ~0.057 s           ±0.001 s     19.82      4,184
aarch64-linux-musl     ~0.064 s           ±0.002 s     19.82      3,914
aarch64-linux-gnu      ~0.063 s           ±0.001 s     19.82      4,085
aarch64-macos          ~0.050 s           ±0.001 s     23.46      1,642
x86_64-windows         within noise     <1 σ        19.82        959
aarch64-windows        within noise     <1 σ        19.82        939
x86_64-macos (Intel)   see .agg.tsv     see .agg.tsv  19.82      ~3,400 (eta)

The exact figures land in dist/bench-<target>.agg.tsv on each release.

Key findings

1. glibc-static is faster on train (statistically significant)

Across 5 runs, the gap is bigger than stddev(train_s) — i.e. the overlap is minimal:

For x86_64 the gap is unambiguous. For aarch64 it's measurable but borderline. We don't claim "glibc wins" on aarch64 — we observe "within noise on aarch64, glibc clearly faster on x86_64".

Why we ship both: the glibc builds cost ~30 s less CI time per release than the Alpine docker builds, and for x86_64 the perf benefit is real. musl builds remain the principle-correct universal Linux artifact (works on Alpine, Debian, RHEL, Arch — glibc-static also works, but a separate musl binary means downstream users on non-glibc distros don't need any compatibility layer).

2. musl-static binaries are smaller

musl ships a smaller libc.a archive so the statically-linked binary pulls in less dead code. Both runs are otherwise functionally identical (pass smoke.sh; the musl job compiles + smokes inside the alpine:3.20 container).

3. query performance is within noise on both

query_s is on the order of 0.0006 s — too small for the 5-run spread to be informative on shared runners. The bench corpus is too small for query-timing to dominate.

4. Perplexity is identical (within float-precision)

ppl matches across both libc choices (the model's content is deterministic from the corpus; SIMD-instruction-selection differences in the score loop produce identical bits).

What this isn't

• Not a system-level benchmark. One corpus, one algorithm, one tool. Tokenizers with larger vocabularies (SentencePiece, BBPE), different n-gram order, modified-KN, etc. all change the picture.
• Not a multi-machine average. One shared runner per target. Different SKUs of GitHub-hosted x86_64 shift medians by ~5-10 %.
• Not a real-world corpus. The synthetic data is degenerate (random words from a small dictionary); perplexity is a regression sanity-check, not a production claim.

How to reproduce

# One run on whatever host-built binary is in build/
bash scripts/bench.sh TARGET=local

# Five iterations + stats (canonical CI flow)
BENCH_ITER=5 bash scripts/bench-multi.sh local \
    | bash scripts/bench-aggregate.sh

The bench corpus is deterministic (srand(42)), so train-times are reproducible within ~5 % on shared runners. The bias in the stddev (one shared runner per job) prevents publishing absolute ceilings — only "gnu is faster, by X stddevs on x86_64, within 1 σ on aarch64."

← kenlm home · beginner tutorial · canonical source on github