♞ Chess Notation Tokeniser

♟️ Nbd7 is one idea. Most tokenisers make it three.

N + bd + 7 scatters a single concept across three positions a model must learn to reassemble — for every move, in every game, forever. And the pieces are ambiguous alone: b is a file, and also the start of bxc4.

📉 Measured on 116,240 held-out tokens

general byte-BPE 328,516

this tokeniser 116,240

64.62% fewer

🔗 The separator is folded into the token that follows it. A token is " Nf3", not " " and "Nf3". Three things fall out at once:

The sequence doesn't spend half its length on spaces

The decoder is plain concatenation, so exact roundtrip is structural, not something to test for

Every move token carries a leading space, so a vocabulary entry can never match inside another move — " e4" cannot occur within " Nxe4"

The substring collision that plagues word-level chess vocabularies is designed out rather than patched around.

♜ Every game in the corpus is legal. Played out on a real board by python-chess, so SAN disambiguation (Nbd7, R1e2, exd5) is always correct for the position. Hand-written move strings would carry disambiguations that can never legally arise.

✅ Exact roundtrip. decode(encode(x)) == x, character for character, on every held-out sequence. Not a percentage — in this domain an altered character is a different object, so it's pass or fail. It passes.

🧰 What's in the box

tokenizer.json (loads with HuggingFace tokenizers and transformers, one line) · the corpus generator, seeded and reproducible byte-for-byte · the training script · the benchmark script · eval/report.json with every number · a full tokeniser card.

Vocabulary 3,500, of which 53.77% is actually reached on held-out data.

📊 Benchmark it against your own tokeniser

python src/benchmark.py --baseline-json /path/to/your/tokenizer.json

One command. The in-repo baseline is a byte-BPE trained here on mixed general text — a fair stand-in for "a tokeniser meeting this domain for the first time", but not necessarily yours. It is not GPT-2 or cl100k: those vocabularies couldn't be fetched in the build environment, and quoting a figure for a tokeniser that was never run would be inventing it.

⚠️ Limitations

99.87% of moves are single tokens, not 100%.

The remainder are rare disambiguated captures with check — Qh4xe1# — which decompose into three or four tokens. They roundtrip exactly, they're just less efficient. Getting to 100% would need ~25,000 enumerated entries: a large embedding table to carry for 0.13% of moves.

An earlier build did exactly that and measured worse — compression fell from 3.9 to 2.3 to buy 0.06 points of atomicity. The simpler design won and the card explains why.

Movetext only. PGN header tags, clock annotations and NAGs aren't modelled.

The corpus is synthetic, generated from documented rules, seeded at 20260904 and reproducible byte-for-byte. Nothing scraped, nothing sampled from a production system. No third-party licence attached, and no restriction on redistributing models you build with it.

🪪 Licence: use in production and in commercial products, ship models built with it, no attribution required. Don't resell the tokeniser itself.