📊 Observability Log Tokeniser

🔥 Logs are the most compressible text your system produces — and your tokeniser is throwing it away.

The same templates repeat millions of times with only the values changing. A general tokeniser has never seen "duration_ms": as a unit, so it spends five or six tokens on a key that appears in every single line. Multiply that by your ingest volume.

📉 Measured on 158,725 held-out tokens

tokens

general byte-BPE 740,876

this tokeniser 158,725

78.58% fewer

🎯 What it does differently

Structural fields are isolated before BPE runs: ISO-8601 timestamps, nginx CLF times, IPv4 addresses, UUIDs, HTTP methods, severity levels. Each is a field a parser or SIEM rule needs whole.

🎲 UUIDs are left to decompose, on purpose. They're random. No vocabulary can compress them, and any entry that appears to is memorising a value that will never recur. They're isolated so they can't contaminate their neighbours, then allowed to cost what they cost. A tokeniser claiming to compress random hex is reporting a bug.

📐 Measured by boundary alignment, not atomicity. A timestamp can't be one token — there are millions of distinct values. So what's measured is that no token straddles the field's edge: 100% on timestamps, IPv4 and UUIDs.

Four formats in realistic proportions: nginx combined, RFC5424 syslog, JSON application logs, logfmt.

✅ Exact roundtrip. decode(encode(x)) == x, character for character, on every held-out sequence. Not a percentage — in this domain an altered character is a different object, so it's pass or fail. It passes.

🧰 What's in the box

tokenizer.json (loads with HuggingFace tokenizers and transformers, one line) · the corpus generator, seeded and reproducible byte-for-byte · the training script · the benchmark script · eval/report.json with every number · a full tokeniser card.

Vocabulary 8,192, of which 87.45% is actually reached on held-out data.

📊 Benchmark it against your own tokeniser

python src/benchmark.py --baseline-json /path/to/your/tokenizer.json

One command. The in-repo baseline is a byte-BPE trained here on mixed general text — a fair stand-in for "a tokeniser meeting this domain for the first time", but not necessarily yours. It is not GPT-2 or cl100k: those vocabularies couldn't be fetched in the build environment, and quoting a figure for a tokeniser that was never run would be inventing it.

⚠️ Limitations

Your logs will differ, possibly a lot — field names, formats and cardinality vary enormously between stacks. This is the package where benchmarking on your own data matters most, and it's one command.

Not covered: Windows Event Log XML, CEF/LEEF, protobuf or otlp binary, multi-line stack traces.

The corpus is synthetic, generated from documented rules, seeded at 20260904 and reproducible byte-for-byte. Nothing scraped, nothing sampled from a production system. No third-party licence attached, and no restriction on redistributing models you build with it.

🪪 Licence: use in production and in commercial products, ship models built with it, no attribution required. Don't resell the tokeniser itself.