can1357
0cdd37fc15
- Replaced the `ctok` implementation with the `utok` universal tokenizer supporting multiple model families and UTF text encodings. - Added tokenizer support and embedding data for Qwen3, DeepSeek V3, Kimi K2, and GLM-5 model variants. - Added fixture generation scripts, vocabulary packers, and golden test suites for validating tokenization parity. - Updated dependency requirements and Bazel workspace definitions for new crates and tools.
utok vocabulary data
ctok_v3.bin.zst and ctok_v4_7.bin.zst are generated — do not
hand-edit. They are compacted from the measured vocabulary files of
sanderland/ctok v1.0.0 (revision
df3b59b5e645289a5eadc8e24036b99d39c333c4), MIT licensed — see
LICENSE.ctok. The vocabulary data is Sander Land's measurement work
("On the biology of Claude's tokenizer",
https://tokencontributions.substack.com/p/on-the-biology-of-claudes-tokenizer);
the Rust implementation in ../src/utok/claude/ is this repository's own.
Upstream ships every piece with a count_tokens witness probe; compaction
drops that metadata, parses the public ⟨bow⟩the⟨eow⟩ key notation into the
compact C0 marker alphabet (single bytes 0x01–0x05; safe because nfc
strips C0 controls from input), adds the glued contraction spellings, and
front-codes the sorted piece list into the version-2 binary format produced
by ../tools/gen-ctok-vocab.ts (~4.7 MB of upstream JSON →
~254 KB front-coded → ~106 KB after zstd -19).
Regenerate the front-coded binaries, then compress them here:
cd ../tools
bun gen-ctok-vocab.ts # fetch upstream, emit raw bins into cache/
bun pack-ctok.ts # zstd -19 into ../data/
If the upstream pin moves, also regenerate
../src/utok/claude/testdata/fixtures.json against the same ctok release
(see the fixture doc in ../src/utok/claude/mod.rs).
The other *.bin.zst files here are the UTOK1 BPE rank tables packed by
the per-family scripts in ../tools/ (container format and per-family
split specs: families.json in this directory).