Tutorial: tuning, pruning, integrating kenlm

1. Choosing the order (-o)

The order is the single biggest knob. Trade-offs for a ~100M-token corpus:

Sweet spot: -o 4 for most production use, -o 5 when you have web-scale data and want every last bit of accuracy. Going past 5 almost never helps because data is too thin at high orders.

lmplz -o 4 -S 4G < corpus.txt > model.arpa

2. Memory tuning (-S) and threading

-S sets a single number: total memory budget in bytes (you can suffix with G/M). kenlm uses this for both the in-memory sort buffer and the hash table for counting.

Rule of thumb: -S >= 2× peak n-gram count in bytes, plus a margin. If you see Vocab hash size estimate exceeds total memory, raise it.

On modern machines, try -S 16G first and watch top:

build_binary trie -a 22 -q 8 -b 8 < model.arpa > model.bin
# -a 22   array-pointer compression
# -q 8    probability quantization (8 bits)
# -b 8    backoff quantization (8 bits)

3. Discounting strategies

By default kenlm uses modified interpolated Kneser-Ney, which is strong out of the box. Two flags you should know:

Unless you have a specific reason, just leave the default.

4. Vocabulary pruning with filter

If you only need a model that knows a specific vocabulary (e.g. machine-translation system's output vocab), filter strips the model:

# one token per line, plain text
sort -u corpus.txt > vocab.txt

filter file vocab.txt < model.arpa > model.pruned.arpa
build_binary model.pruned.arpa model.bin

Effect: model shrinks dramatically (often 5–50×) and perplexity on in-vocab text is usually unchanged. Out-of-vocab text starts penalising (kenlm can't score what it doesn't know).

Other filter modes worth knowing:

• filter grep PATTERN — keep only n-grams matching a regex (useful for keeping only lowercased words, or only ASCII, etc.)
• filter phrase_table_vocab source target — restrict to a translation phrase table's vocabulary (a common MT pattern)

5. Probing vs trie, and quantization

Reduce both with quantization:

build_binary trie \
    -a 22 \      # 22-bit array-pointer compression
    -q 8 \       # 8-bit probability quantization
    -b 8 \       # 8-bit backoff weight quantization
    < model.arpa > model.bin

# Typical: ~40 % smaller, <1 % perplexity hit

Quantization errors are bounded and small. Ship the quantized binary to users; keep one un-quantized copy for benchmarking.

6. mmap and instant loading

On Linux, kenlm auto-mmaps any model under about 2 GB when you construct lm::ngram::Model. This means:

• The model doesn't get copied into process memory
• First load looks like ~50 ms but is actually a mmap
• Memory usage grows only with the words you actually score

For a 100 MB model this is a 100× startup-time improvement. There is nothing to do — it's already on.

If you want to disable it (e.g. on Windows where mmap behaves differently) or force it:

// disable
lm::ngram::Config cfg;
cfg.load_method = util::LAZY;
// or fully into memory: cfg.load_method = util::POPULATE_OR_READ;
// or fully mmap:        cfg.load_method = util::READ;

lm::ngram::Model model("model.bin", cfg);

7. Embedding in C++ (a self-contained example)

// score.cc — score a stream of sentences against a kenlm model.
//   g++ -O2 -pthread score.cc -o score
#include <lm/model.hh>
#include <lm/lm_exception.hh>
#include <cstdio>
#include <cstring>
#include <string>

int main(int argc, char** argv) {
    if (argc < 2) { fprintf(stderr, "usage: %s model.bin < sentences.tsv\n", argv[0]); return 1; }
    lm::ngram::Model model(argv[1]);

    std::string line;
    while (std::getline(std::cin, line)) {
        lm::ngram::State st(model.BeginSentenceState());
        double total = 0.0;
        // expect: tokens separated by spaces, no </s> (we add it)
        char* save;
        for (char* tok = strtok_r(const_cast<char*>(line.data()), " ", &save);
             tok; tok = strtok_r(nullptr, " ", &save)) {
            auto id = model.GetVocabulary().Index(tok);
            if (id == lm::kUNK) total += model.OOVScore(st);
            else                 total += model.Score(st, id, st);
        }
        total += model.ScoreEndSentence(&st);
        std::printf("%f\n", total);
    }
}

Build and run:

g++ -O2 -pthread score.cc -o score -I upstream/kenlm
# Score a file of tokenized sentences (one per line):
./score model.bin < sentences.tsv > logprobs.txt

Headers are header-only — no -lkenlm step. The only external runtime deps are libpthread and libm.

8. Batch scoring millions of sentences

For large-scale evaluation, the per-construct Model cost is amortized; the bottleneck is scoring throughput. The single most expensive thing is the per-token Score lookup, so:

• Use multi-threading: feed the corpus to multiple threads sharing one Model. The model is read-only after load and safe for concurrent reads.
• Batch OOV lookups: model.GetVocabulary().Index(...) is a hash; cache sentence IDs once and reuse.
• Don't recreate State per token — use the existing in-place update pattern (Score(state, w, state)) as shown above.

Concrete numbers on a single thread on a mid-range x86_64 box: ~3 M tokens/sec scoring for a 4-gram probing model on disk.

9. Real-world recipes

9.1 Re-ranking N-best from a translation system

# inputs: kenlm model, a file of tokenized hypotheses (one per line, scored)
while read -r hyp; do
    score=$(echo "$hyp" | kenlm/score model.bin | tail -1)
    echo "$score $hyp"
done < hypotheses.tsv | sort -nr | head -1000

9.2 Perplexity of a held-out set

kenlm/query model.bin < held_out.txt | grep "^Perplexity"
# → single line: "Perplexity including OOVs: 42.7"

9.3 Streaming scoring in a long-running service

Construct the Model once at startup, then fork workers that share that one Model across processes (it's mmapped read-only). Memory cost on each worker: whatever it scores, not the full model.

9.4 Distribute a small model across many boxes

tar czf model.tgz model.bin. With a musl-static binary, the same archive runs on Alpine, Ubuntu, RHEL, macOS, Windows. Kenlm's v0.1.0 release ships exactly that — same source, 7 platform targets, one verified per arch.

← kenlm home