Tutorial: train your first language model in 5 minutes

beginner · copy-pasteable · <5 min

kenlm is an open-source project by Kenneth Heafield (canonical home: kpu/kenlm, 2011). This tutorial teaches you to use the upstream tool — for questions, bug reports, and contributions, please go to the upstream project. This site distributes pre-built static binaries; it does not maintain kenlm.

kenlm is CPU-only. Training runs on pthreads and a memory-resident sort buffer — no GPU is ever required. A free Colab CPU, a laptop, or a desktop are all plenty for typical corpora. (You may see old tutorials that claim kenlm uses Colab's GPU; that's a mistake — kenlm has no GPU code at all.)

1. What is kenlm?

kenlm is a fast, memory-efficient n-gram language model toolkit. Given a plain-text corpus, it produces a model that scores any new piece of text with a probability (and derives from it a perplexity, a "how surprising is this text" score).

It's used in:

Machine translation (the language model for the decoder)
Retrieval-augmented systems (re-ranking candidates)
Speech recognition (LM decoding)
Anywhere you need a "is this sequence plausible?" signal

You get four binaries and one optional header-only C++ library:

binary	what it does
`lmplz`	train: corpus → ARPA-format n-gram model
`build_binary`	compile: ARPA → compact binary model
`query`	score: sentences → log-probability / perplexity
`filter`	prune: restrict an ARPA model to a vocabulary

2. Install

kenlm releases are self-contained static binaries — no system libraries to install, no compiler needed. Pick your platform, download, extract, done.

From the v0.1.0 release page, grab the archive for your platform and extract it:

Linux (x86_64 or aarch64, including Alpine)

tar xzf kenlm-x86_64-linux-musl.tar.gz
cp kenlm-x86_64-linux-musl/bin/* /usr/local/bin/
which lmplz
# /usr/local/bin/lmplz

The -musl archive runs on every Linux distro — Alpine, Debian, Ubuntu, RHEL, Arch, … — because it's statically linked against musl. There's also a -gnu variant if you specifically need glibc linkage.

macOS (Apple Silicon)

tar xzf kenlm-aarch64-macos.tar.gz
cp kenlm-aarch64-macos/bin/* /usr/local/bin/

Windows (x64 or ARM64)

Expand-Archive kenlm-x86_64-windows.zip
# move bin\* somewhere on $PATH

Verify

lmplz --help

You should see kenlm's own usage banner. If yes, you're ready.

3. Core concepts

3.1 N-gram language models

Given a sequence of tokens w₁, w₂, …, wₙ, the model assigns a probability to it. N-gram models say: "the probability of token k depends only on the N − 1 tokens before it."

Big N → more accurate, more memory. Typical: 3 to 5.

3.2 ARPA format

The output of training. Human-readable, big on disk. Look at it once:

$ head -20 model.arpa
\data\
ngram 1=4
ngram 2=4
ngram 3=4

\1-grams:
-1.2345  word1  -0.5
...

\end\

You won't normally edit ARPA; you use it by compiling it.

3.3 Binary model

build_binary model.arpa model.bin compresses the ARPA into two formats you can pick from:

probing (default): fast queries, slightly more memory
trie: lower memory, slightly slower

Default to probing unless profiling shows trie fits your workload better. Switch with -a trie.

3.4 Perplexity

The score you usually report. Lower = better.

PPL = 2^(-1/N · Σ log₂ P(wᵢ))

Half the perplexity means the model is twice as good at predicting the text. Numbers above 1000 mean the model is nearly useless on that text; numbers in the dozens mean it knows what's coming.

4. Your first model (5 minutes)

Pick any directory. We're going to train a 3-gram model on a tiny hand-typed corpus, then score a sentence.

mkdir kenlm-demo && cd kenlm-demo

# --- 1. The corpus (one sentence per line, UTF-8 plain text)
cat > corpus.txt <<'EOF'
the cat sat on the mat
the cat ran fast and far
the dog sat down by the door
a small cat and a small dog
the cat and the dog sat together
EOF

# --- 2. Train (--discount_fallback is mandatory for small corpora;
#        Kneser-Ney discounts can otherwise be undefined)
lmplz -o 3 --discount_fallback < corpus.txt > model.arpa

# --- 3. Compile to a compact binary
build_binary model.arpa model.bin

# --- 4. Score a sentence
echo "the cat sat" | query model.bin

Expected: kenlm prints a multi-stage training log, ends with SUCCESS, then query prints something like:

Read 1 sentences
The command line was:
  echo "the cat sat" | query model.bin
   1 sentences, 1 words,  3 OOVs
   prob = -2.17280  ppl = 3.493

The two numbers you care about: prob (sum of log probabilities) and ppl (perplexity). Lower perplexity = the model finds this text more predictable.

5. Reading the output

From the training step:

=== 5/5 Writing ARPA model ===
lmplz      9650176 bytes  user:0.04s  sys:0.02s  CPU:0.06s  real:0.03s
SUCCESS

9650176 bytes ≈ how big the ARPA file is on disk, in bytes.
user:0.04s, sys:0.02s: CPU time in user / kernel space.
CPU:0.06s: total CPU time.
real:0.03s: wall-clock time. Use this to compare runs.
SUCCESS: the model wrote without error.

From build_binary:

Reading /home/.../model.arpa
----5---10---15---20---25---
SUCCESS
This binary file contains probing hash tables.
  type        B
  probing 1376 assuming -p 1.5
...

From query:

the=3 2    cat=4 3    sat=5 3    </s>=2 2
Total: -2.1727986 OOV: 0
Perplexity including OOVs:	3.493
Perplexity excluding OOVs:	3.493
query       1736704 bytes  user:0.001s  real:0.001s

Each token, the order it appeared at, its ID, and its context count.
Total: sum of log₁₀ probabilities across all words.
OOV: out-of-vocabulary count. 0 here because all words are from the training corpus.
Perplexity: the headline metric. Lower = better.

6. Bigger corpora & memory

For real corpora you need to think about memory. The -S flag caps the sort buffer lmplz uses for counting:

# A few GB of text needs a few GB of sort memory
lmplz -o 5 -S 8G < big_corpus.txt > model.arpa

If lmplz can't compute Kneser-Ney discounts (because your corpus is small for the order) you'll get:

BadDiscountException because 's.n[j] == 0'.
Could not calculate Kneser-Ney discounts for N-grams with adjusted count N
because we didn't observe any N-grams with adjusted count M.
Try deduplicating the input.  To override this error for e.g. a class-based
model, rerun with --discount_fallback

Two answers:

Use more data (the real fix).
Use --discount_fallback to use crude discounts on the tail. Acceptable for prototypes; don't ship it.

Another common error: Vocab hash size estimate exceeds total memory. Fix: raise -S (sort buffer is also the hash-table budget) or reduce -o.

7. Loading the model from C++

kenlm ships a header-only C++ API:

#include "lm/model.hh"
#include <cstdio>

int main() {
    lm::ngram::Model model("model.bin");

    lm::ngram::State state(model.BeginSentenceState());
    lm::FloatingPoint total = 0.0;

    const char* sentence = "</s> the cat sat </s>";
    for (const char* tok = strtok(const_cast<char*>(sentence), " ");
         tok; tok = strtok(nullptr, " ")) {
        auto id = model.GetVocabulary().Index(tok);
        lm::FloatingPoint score;
        if (id == lm::kUNK) {
            score = model.OOVScore(state);
        } else {
            score = model.Score(state, id, state);
        }
        total += score;
    }
    printf("log10 P(sentence) = %f\n", total);
}

The header that ships with kenlm is at upstream/kenlm/lm/model.hh. Wrap it in your build however you like — no autotools required, no link-time magic, just #include and link -lpthread -lm.

8. Common errors

error	cause / fix
`libstdc++.so.6: cannot open shared object file`	wrong ABI / dynamic build. Use the static binary from the v0.1.0 release — it's always self-contained.
`Vocab hash size estimate exceeds total memory`	`-S` too small. Raise it, or lower `-o`, or feed more data.
silent nonsense probabilities, NaN logs	likely double-encoded corpus (mojibake). Make sure your text is plain UTF-8, no BOM, no zero-width.
tiny model gives huge perplexity > 10⁶	almost always corpus too small for the order. Either lower `-o`, add `--discount_fallback`, or get more data.
`query: FileNotFoundException`	missing or wrong path to the binary model — `query` wants the `.bin`, not the `.arpa`.

9. Next steps

Larger corpora: CommonCrawl, Project Gutenberg, OSCAR, Wikipedia dumps.
Order tuning: try 4 or 5 once your corpus exceeds ~50M tokens.
Prune to a vocabulary: see filter in the next tutorial.
Quantize to shrink further: build_binary -q 8 -b 8 cuts memory ~2× with <1 % perplexity hit.
mmap for instant load: kenlm auto-mmaps models < ~2 GB on Linux. Just Model("model.bin") — no code change.
Score millions of sentences: pipe them into query, or call model.FullScore(...) in C++ for batch APIs.

Next: tuning & integration →