feat(pi-natives/tools): implemented utok tokenizer for multiple models

Browse Source

- Replaced the `ctok` implementation with the `utok` universal tokenizer supporting multiple model families and UTF text encodings.
- Added tokenizer support and embedding data for Qwen3, DeepSeek V3, Kimi K2, and GLM-5 model variants.
- Added fixture generation scripts, vocabulary packers, and golden test suites for validating tokenization parity.
- Updated dependency requirements and Bazel workspace definitions for new crates and tools.

...

This commit is contained in:

can1357

2026-08-20 00:37:05 +02:00

parent 37fd2dbbe1

commit 0cdd37fc15

101 changed files with 746591 additions and 278 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download Patch File Download Diff File Expand all files Collapse all files

MODULE.bazel.lock

Generated

+71 -18

File diff suppressed because one or more lines are too long