can1357
0cdd37fc15
- Replaced the `ctok` implementation with the `utok` universal tokenizer supporting multiple model families and UTF text encodings. - Added tokenizer support and embedding data for Qwen3, DeepSeek V3, Kimi K2, and GLM-5 model variants. - Added fixture generation scripts, vocabulary packers, and golden test suites for validating tokenization parity. - Updated dependency requirements and Bazel workspace definitions for new crates and tools.
20 lines
1.7 KiB
JSON
20 lines
1.7 KiB
JSON
[
|
|
"",
|
|
" ",
|
|
" \t\t\n\n\r\n ",
|
|
"hello world",
|
|
"Hello, World! It's a test. We'll see; don't worry, y'all've been warned.",
|
|
"The quick brown fox jumps over the lazy dog. 1234567890 12 345 6789",
|
|
"fn main() {\n let x: Vec<u32> = (0..10).map(|i| i * 2).collect();\n println!(\"{x:?}\");\n}",
|
|
"{\"key\": [1, 2.5, -3e8], \"nested\": {\"a\": null, \"b\": true}}",
|
|
"东京は日本の首都であり、世界で最も人口の多い都市圏の一つです。深度求索发布了新一代基座模型。",
|
|
"한국어 텍스트도 테스트합니다. 안녕하세요!",
|
|
"Многоязычный текст: русский, ελληνικά, עברית, العربية.",
|
|
"naïve café résumé — em-dash…ellipsis",
|
|
"naïve café",
|
|
"👍🏽 emoji test 👨👩👧👦 family, flags 🇹🇷🇯🇵, math 𝕏≈∑∫",
|
|
"https://example.com/path?query=value&other=%20escaped#fragment",
|
|
"supercalifragilisticexpialidocious supercalifragilisticexpialidocious supercalifragilisticexpialidocious supercalifragilisticexpialidocious supercalifragilisticexpialidocious supercalifragilisticexpialidocious supercalifragilisticexpialidocious supercalifragilisticexpialidocious supercalifragilisticexpialidocious supercalifragilisticexpialidocious supercalifragilisticexpialidocious supercalifragilisticexpialidocious supercalifragilisticexpialidocious supercalifragilisticexpialidocious supercalifragilisticexpialidocious supercalifragilisticexpialidocious supercalifragilisticexpialidocious supercalifragilisticexpialidocious supercalifragilisticexpialidocious supercalifragilisticexpialidocious",
|
|
" indented\n\tmixed whitespace runs\n\n\n",
|
|
"CamelCaseIdentifier snake_case_name SCREAMING_SNAKE kebab-case-name"
|
|
] |