🎼 ABC Music Notation Tokeniser

🎻 :| and || are not the same thing, and your tokeniser doesn't know that.

One is a repeat, one is a section end. A tokeniser that matches | first collapses |:, :|, || and |] into indistinguishable mush — and then a generative model produces notation that won't parse.

📉 Measured on 430,091 held-out tokens

general byte-BPE 1,218,217

this tokeniser 430,091

64.7% fewer

📋 The header is structural. Field tags (X:, T:, M:, L:, K:) mark boundaries a parser depends on. A token fusing K: to the key that follows destroys the field. Tags are isolated — 100% intact, measured.

🎵 Bar and repeat marks are isolated longest-first, so |: is never matched as | then :. 100% intact.

🎶 The body is left free. Notes, accidentals, octave marks, durations, chords, grace notes and slurs go to BPE, which learns the repeated figures — and traditional tunes are built from repeated figures, which is exactly why ABC compresses.

✅ Exact roundtrip. decode(encode(x)) == x, character for character, on every held-out sequence. Not a percentage — in this domain an altered character is a different object, so it's pass or fail. It passes.

🧰 What's in the box

tokenizer.json (loads with HuggingFace tokenizers and transformers, one line) · the corpus generator, seeded and reproducible byte-for-byte · the training script · the benchmark script · eval/report.json with every number · a full tokeniser card.

Vocabulary 4,096, of which 92.21% is actually reached on held-out data.

📊 Benchmark it against your own tokeniser

python src/benchmark.py --baseline-json /path/to/your/tokenizer.json

One command. The in-repo baseline is a byte-BPE trained here on mixed general text — a fair stand-in for "a tokeniser meeting this domain for the first time", but not necessarily yours. It is not GPT-2 or cl100k: those vocabularies couldn't be fetched in the build environment, and quoting a figure for a tokeniser that was never run would be inventing it.

⚠️ Limitations

2.8918 characters per token is the lowest of the five, and that's honest rather than disappointing. ABC is already terse, so there's less redundancy to remove than in a log line. The gain over a general tokeniser is still 64.7% fewer tokens.

The corpus is grammar-generated — correct bar filling for the stated metre, repeated phrases across parts, cadences on chord tones. Structurally realistic, musically unremarkable, and no copyrighted setting or arrangement appears anywhere in it.

Not covered: inline field changes mid-tune, multi-voice scores, lyric alignment (w: lines), ABC 2.2 extensions.

The corpus is synthetic, generated from documented rules, seeded at 20260904 and reproducible byte-for-byte. Nothing scraped, nothing sampled from a production system. No third-party licence attached, and no restriction on redistributing models you build with it.

🪪 Licence: use in production and in commercial products, ship models built with it, no attribution required. Don't resell the tokeniser itself.