can1357 0cdd37fc15 feat(pi-natives/tools): implemented utok tokenizer for multiple models ...

- Replaced the `ctok` implementation with the `utok` universal tokenizer supporting multiple model families and UTF text encodings.
- Added tokenizer support and embedding data for Qwen3, DeepSeek V3, Kimi K2, and GLM-5 model variants.
- Added fixture generation scripts, vocabulary packers, and golden test suites for validating tokenization parity.
- Updated dependency requirements and Bazel workspace definitions for new crates and tools.

2026-08-20 01:45:04 +02:00

398 KiB

Raw History

View Raw