⚗️ SMILES Chemistry Tokeniser

🧪 Cl split into C + l is not a small error.

It turns chlorine into carbon followed by a stray letter, and nothing downstream will tell you. Left to itself, BPE usually learns Cl because chlorine is common. Usually is not a property you can build a pipeline on.

📉 Measured on 14,996 held-out tokens

general byte-BPE 111,596

this tokeniser 14,996

86.56% fewer

🔒 Multi-character chemical units cannot be split. Two-letter elements, bracket atoms ([nH], [N+], [C@@H]) and two-digit ring closures are isolated before BPE ever sees them. Structurally incapable of splitting, not merely unlikely to.

🧬 Isolation alone isn't enough — and finding that out is why this works. A bracket atom never seen in training still gets shredded from the inside: [C@H] → [ + C + @ + H]. So a curated set of 71 bracket atoms (stereocentres, charged nitrogens, isotopes, metals) is added to the vocabulary explicitly.

🧫 The corpus is chemically valid. Every molecule passes through RDKit and is canonicalised; anything that doesn't parse is discarded. 40,000 unique structures, generated — nothing taken from ChEMBL, PubChem or ZINC, so there's no database licence attached to anything you build with it.

✅ Exact roundtrip. decode(encode(x)) == x, character for character, on every held-out sequence. Not a percentage — in this domain an altered character is a different object, so it's pass or fail. It passes.

🧰 What's in the box

tokenizer.json (loads with HuggingFace tokenizers and transformers, one line) · the corpus generator, seeded and reproducible byte-for-byte · the training script · the benchmark script · eval/report.json with every number · a full tokeniser card.

Vocabulary 4,167, of which 65.63% is actually reached on held-out data.

📊 Benchmark it against your own tokeniser

python src/benchmark.py --baseline-json /path/to/your/tokenizer.json

One command. The in-repo baseline is a byte-BPE trained here on mixed general text — a fair stand-in for "a tokeniser meeting this domain for the first time", but not necessarily yours. It is not GPT-2 or cl100k: those vocabularies couldn't be fetched in the build environment, and quoting a figure for a tokeniser that was never run would be inventing it.

⚠️ Limitations

The headline number is optimistic and the package says so.

Held-out compression is 7.58 characters per token. On 22 real, well-known drug molecules it's 4.09 — still 75% fewer tokens than baseline, but a long way off. Generated corpora share fragments more than a diverse screening library does.

That's why data/realworld.txt ships in the box and the benchmark reports it separately. Run it on your own library first.

Organic and medicinal chemistry. Inorganics, organometallics and polymers are under-represented; reaction SMILES and SMARTS aren't covered.

The corpus is synthetic, generated from documented rules, seeded at 20260904 and reproducible byte-for-byte. Nothing scraped, nothing sampled from a production system. No third-party licence attached, and no restriction on redistributing models you build with it.

🪪 Licence: use in production and in commercial products, ship models built with it, no attribution required. Don't resell the tokeniser itself.