smashimo/oh-my-pi
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: implemented native ctok tokenization engine with model scopes

Browse Source 
- Implemented the `ctok` Rust native tokenization engine with offline support for Claude V3, V47, V5, and V5Sonnet families.
- Replaced global token estimation with model-scoped `Tokenizer` instances and provider-anchored transcript accounting across packages.
- Added vocabulary generation scripts, test fixtures, and comprehensive unit tests for tokenizer routing and matching modes.
This commit is contained in:
can1357
2026-08-19 23:27:29 +02:00
parent c39004063e
commit 12238f55ca
63 changed files with 6745 additions and 664 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Cargo.toml
+2
View File
@@ -233,6 +233,8 @@ pdf-inspector = "1"
regex = "1"
similar = "3.1.0"
unicode-segmentation = "1.13"
unicode-normalization = "0.1"
unicode-properties = "0.1"
unicode-width = "0.2"
fontdue = { version = "0.9", default-features = false }