🧬 Genomic Sequence Tokeniser

🔬 A general tokeniser reads DNA one base at a time.

Uniform random ACGT has no repeated substrings, so there's nothing for BPE to find — which is why tokenisers trained on random sequence are expensive character-level tokenisers wearing a hat. Real genomes are nothing like random, and this one is trained on structure: codon bias, promoter motifs, microsatellites, GC variation, soft-masking, sequencing gaps.

📉 Measured on 296,545 held-out tokens

general byte-BPE 1,444,609

this tokeniser 296,545

79.47% fewer

🔠 Case is preserved. Lowercase marks soft-masked repeat regions in reference FASTA. Folding case would halve the vocabulary and silently discard that. No token spans a masking boundary — 100% measured.

❓ Ambiguity codes are isolated. N, R, Y mark uncertain base calls. A merge that swallows an N into ordinary bases hides the one thing about that position you needed to know. 100% purity, measured.

📏 Nothing forces codon alignment, deliberately. A general DNA tokeniser can't know where a reading frame starts, and one that assumes frame 1 is wrong two thirds of the time. Coding-only work with known frame? Retrain with a codon pre-tokeniser — the script's included and it's a three-line change.

97.4% of the vocabulary is actually reached, so almost nothing in it is dead weight.

✅ Exact roundtrip. decode(encode(x)) == x, character for character, on every held-out sequence. Not a percentage — in this domain an altered character is a different object, so it's pass or fail. It passes.

🧰 What's in the box

tokenizer.json (loads with HuggingFace tokenizers and transformers, one line) · the corpus generator, seeded and reproducible byte-for-byte · the training script · the benchmark script · eval/report.json with every number · a full tokeniser card.

Vocabulary 4,096, of which 97.44% is actually reached on held-out data.

📊 Benchmark it against your own tokeniser

python src/benchmark.py --baseline-json /path/to/your/tokenizer.json

One command. The in-repo baseline is a byte-BPE trained here on mixed general text — a fair stand-in for "a tokeniser meeting this domain for the first time", but not necessarily yours. It is not GPT-2 or cl100k: those vocabularies couldn't be fetched in the build environment, and quoting a figure for a tokeniser that was never run would be inventing it.

⚠️ Limitations

The corpus models genomic structure but is generated, not sampled from any organism. Compression on a real reference will differ — likely lower on gene-poor regions, higher on repetitive ones.

DNA and RNA only. Protein sequences need a different tokeniser and this isn't it.

The corpus is synthetic, generated from documented rules, seeded at 20260904 and reproducible byte-for-byte. Nothing scraped, nothing sampled from a production system. No third-party licence attached, and no restriction on redistributing models you build with it.

🪪 Licence: use in production and in commercial products, ship models built with it, no attribution required. Don't resell the tokeniser itself.