can1357 0cdd37fc15 feat(pi-natives/tools): implemented utok tokenizer for multiple models ...

- Replaced the `ctok` implementation with the `utok` universal tokenizer supporting multiple model families and UTF text encodings.
- Added tokenizer support and embedding data for Qwen3, DeepSeek V3, Kimi K2, and GLM-5 model variants.
- Added fixture generation scripts, vocabulary packers, and golden test suites for validating tokenization parity.
- Updated dependency requirements and Bazel workspace definitions for new crates and tools.

2026-08-20 01:45:04 +02:00

..

skills

refactor: migrated packages to internal utility modules and removed external dependencies

2026-08-05 13:39:09 +02:00

toolconv

chore: update docs + rename reset to clear

2026-08-03 18:37:23 +02:00

tools

Merge PR #8572: fix(lsp): gate rust-analyzer/reloadWorkspace behind rust-analyzer check (@roboomp)

2026-08-16 02:43:33 +02:00

adding-a-provider.md

chore: update stale docs

2026-08-03 16:39:23 +02:00

advisor-watchdog.md

feat(nix): implemented wayland screencast support and restructured package builder

2026-08-13 05:25:22 +02:00

agent-hub.md

docs(agent-hub): documented subagent observability

2026-08-07 05:39:05 +00:00

ai-schema-normalize.md

refactor: migrated packages to internal utility modules and removed external dependencies

2026-08-05 13:39:09 +02:00

approval-mode.md

docs: clarify bash.patterns gates bash tool only, not eval

2026-08-19 08:46:21 +00:00

auth-broker-gateway.md

chore: update docs + rename reset to clear

2026-08-03 18:37:23 +02:00

bash-tool-runtime.md

docs: clarify bash.patterns gates bash tool only, not eval

2026-08-19 08:46:21 +00:00

blob-artifact-architecture.md

chore: update stale docs

2026-08-03 16:39:23 +02:00

collab.md

docs(collab): clarified relay distribution status

2026-08-10 14:37:54 +00:00

compaction.md

chore: update docs + rename reset to clear

2026-08-03 18:37:23 +02:00

computer-use.md

fix(computer): lazily requested wayland input permission

2026-08-07 06:50:16 +00:00

config-usage.md

refactor: migrated packages to internal utility modules and removed external dependencies

2026-08-05 13:39:09 +02:00

context-files.md

fix: preserve AGENTS boundary behavior

2026-08-05 11:51:30 +01:00

custom-tools.md

refactor: migrated packages to internal utility modules and removed external dependencies

2026-08-05 13:39:09 +02:00

environment-variables.md

feat: introduced dynamic version discovery and rate limit parsing for google

2026-08-13 23:59:37 +02:00

ERRATA-GPT5-HARMONY.md

chore: update stale docs

2026-08-03 16:39:23 +02:00

extension-loading.md

chore: update docs + rename reset to clear

2026-08-03 18:37:23 +02:00

extensions.md

Merge branch 'main' into fix/onpayload-replacement-completions-bedrock-cursor

2026-08-18 09:13:09 +08:00

fs-scan-cache-architecture.md

chore: update stale docs

2026-08-03 16:39:23 +02:00

gemini-manifest-extensions.md

chore: update docs + rename reset to clear

2026-08-03 18:37:23 +02:00

handoff-generation-pipeline.md

fix(session): preserved harness handoff abort reasons

2026-08-08 09:02:28 +00:00

hooks.md

refactor: migrated packages to internal utility modules and removed external dependencies

2026-08-05 13:39:09 +02:00

install-id.md

chore: update stale docs

2026-08-03 16:39:23 +02:00

keybindings.md

docs(agent-hub): documented subagent observability

2026-08-07 05:39:05 +00:00

local-models.md

chore: update stale docs

2026-08-03 16:39:23 +02:00

lsp-config.md

chore: update stale docs

2026-08-03 16:39:23 +02:00

macos-signing-notarization.md

chore: update docs + rename reset to clear

2026-08-03 18:37:23 +02:00

magic-keywords.md

chore: update stale docs

2026-08-03 16:39:23 +02:00

marketplace.md

chore: update docs + rename reset to clear

2026-08-03 18:37:23 +02:00

mcp-config.md

docs(mcp): clarify translated config precedence

2026-08-05 01:11:58 +02:00

mcp-protocol-transports.md

chore: update stale docs

2026-08-03 16:39:23 +02:00

mcp-runtime-lifecycle.md

fix(mcp): keep Exa MCP servers that request non-native tools

2026-08-15 00:39:58 +05:30

mcp-server-tool-authoring.md

chore: update docs + rename reset to clear

2026-08-03 18:37:23 +02:00

memory.md

fix(coding-agent): lowercase the hindsight project label

2026-08-10 13:21:02 +02:00

mnemosyne-memory-backend.md

chore: update docs + rename reset to clear

2026-08-03 18:37:23 +02:00

models.md

feat(pi-natives/tools): implemented utok tokenizer for multiple models

2026-08-20 01:45:04 +02:00

native-crates.md

build: configured docker build pipelines for cargo native addons

2026-08-14 07:09:48 +02:00

natives-addon-loader-runtime.md

chore: update stale docs

2026-08-03 16:39:23 +02:00

natives-architecture.md

chore: update stale docs

2026-08-03 16:39:23 +02:00

natives-binding-contract.md

chore: update stale docs

2026-08-03 16:39:23 +02:00

natives-build-release-debugging.md

fix(natives): pin cmake opus build to static MSVC CRT (/MT)

2026-08-13 16:21:23 +00:00

natives-media-system-utils.md

chore: update docs + rename reset to clear

2026-08-03 18:37:23 +02:00

natives-rust-task-cancellation.md

chore: update docs + rename reset to clear

2026-08-03 18:37:23 +02:00

natives-shell-pty-process.md

chore: update stale docs

2026-08-03 16:39:23 +02:00

natives-text-search-pipeline.md

chore: update docs + rename reset to clear

2026-08-03 18:37:23 +02:00

non-compaction-retry-policy.md

fix(session): resume Cursor idle stalls after unmarked MCP results

2026-08-18 14:48:45 +02:00

notebook-tool-runtime.md

chore: update stale docs

2026-08-03 16:39:23 +02:00

omptype-guide.md

refactor: migrated packages to internal utility modules and removed external dependencies

2026-08-05 13:39:09 +02:00

plugin-manager-installer-plumbing.md

fix(extensions): preserved unsafe schemas during install

2026-08-10 07:37:28 +00:00

porting-from-pi-mono.md

refactor: migrated packages to internal utility modules and removed external dependencies

2026-08-05 13:39:09 +02:00

porting-to-natives.md

chore: update docs + rename reset to clear

2026-08-03 18:37:23 +02:00

provider-compat-reference.md

refactor: generalized thinking loop guard for multiple model families

2026-08-13 06:15:00 +02:00

provider-endpoint-constraints.md

fix(catalog): omit Responses penalties on all first-party xAI models

2026-08-14 22:02:52 -07:00

provider-quirks.md

fix(cursor): answer interactionQuery so hosted fetch can continue

2026-08-18 14:48:45 +02:00

provider-streaming-internals.md

chore: update docs + rename reset to clear

2026-08-03 18:37:23 +02:00

providers.md

chore: update docs + rename reset to clear

2026-08-03 18:37:23 +02:00

python-repl.md

fix(coding-agent/eval): remove per-call model override from agent() (#6438)

2026-08-04 08:35:01 -04:00

resolve-tool-runtime.md

chore: update stale docs

2026-08-03 16:39:23 +02:00

rpc.md

chore: update docs + rename reset to clear

2026-08-03 18:37:23 +02:00

rulebook-matching-pipeline.md

chore: update stale docs

2026-08-03 16:39:23 +02:00

sdk.md

chore: update docs + rename reset to clear

2026-08-03 18:37:23 +02:00

secrets.md

chore: update stale docs

2026-08-03 16:39:23 +02:00

session-operations-export-share-fork-resume.md

chore: update docs + rename reset to clear

2026-08-03 18:37:23 +02:00

session-switching-and-recent-listing.md

docs: reconcile session dir layout with the 17.2.9 hashed-scheme revert

2026-08-12 21:03:00 +01:00

session-tree-plan.md

chore: update stale docs

2026-08-03 16:39:23 +02:00

session.md

docs: reconcile session dir layout with the 17.2.9 hashed-scheme revert

2026-08-12 21:03:00 +01:00

settings.md

docs: clarify bash.patterns gates bash tool only, not eval

2026-08-19 08:46:21 +00:00

skills.md

fix: preserve AGENTS boundary behavior

2026-08-05 11:51:30 +01:00

slash-command-internals.md

chore: update docs + rename reset to clear

2026-08-03 18:37:23 +02:00

system-prompt-customization.md

fix(coding-agent): keep the system prompt byte-stable across date/cwd changes

2026-08-14 13:16:54 +03:00

task-agent-discovery.md

fix(task): refresh model roles before agent discovery

2026-08-17 20:43:06 -04:00

theme.md

chore: update stale docs

2026-08-03 16:39:23 +02:00

tree.md

chore: update stale docs

2026-08-03 16:39:23 +02:00

ttsr-injection-lifecycle.md

chore: update stale docs

2026-08-03 16:39:23 +02:00

tui-core-renderer.md

fix(tui): stopped direct herdr pane flicker

2026-08-13 14:34:03 +00:00

tui-runtime-internals.md

fix(tui): stopped direct herdr pane flicker

2026-08-13 14:34:03 +00:00

tui.md

docs(agent-hub): documented subagent observability

2026-08-07 05:39:05 +00:00

user-facing-packages.md

refactor: migrated packages to internal utility modules and removed external dependencies

2026-08-05 13:39:09 +02:00

vibe-mode.md

chore: update stale docs

2026-08-03 16:39:23 +02:00