- Implemented the `ctok` Rust native tokenization engine with offline support for Claude V3, V47, V5, and V5Sonnet families.
- Replaced global token estimation with model-scoped `Tokenizer` instances and provider-anchored transcript accounting across packages.
- Added vocabulary generation scripts, test fixtures, and comprehensive unit tests for tokenizer routing and matching modes.
- Introduce `@oh-my-pi/omptype` as a new ArkType-compatible schema validation package featuring a lazy JIT runtime, JSON Schema emission, and compatibility adapters.
- Replace `arktype` across workspace packages and test utilities with `@oh-my-pi/omptype`.
- Add benchmark suites, tests, and documentation for the new validation engine and adapters.
- Update workspace build, test runner, and release configurations to include the new package.
computeNonMessageBreakdown estimated Skills tokens from the unfiltered
session.skills registry and subtracted that from the first system-prompt
block, which only ever contained the rendered (filtered) skills. Hidden
explicit-only skills (hide/disable-model-invocation) and all skills when
the read tool was absent inflated Skills and clamped System prompt to 0.
Add a renderedSkills helper mirroring buildSystemPrompt's filter and use
it for the Skills estimate so the category split matches the provider
prompt.
Fixes#6498
computeNonMessageTokens / computeNonMessageBreakdown re-tokenize the system
prompt and every tool's wire schema (per-tool JSON.stringify) on each call,
but the per-turn compaction and context-threshold paths call them several
times (getContextBreakdown twice, #estimateStoredContextTokens once) over
inputs that change at most once per turn. Memoize on the identity of
(systemPrompt, tools, skills) -- the same stable refs the StatusLineComponent
cache already trusts -- so the expensive parts run at most once per input
change instead of per call.
- Migrated all wire protocol, schema definitions, and tools validation from Zod to ArkType across multiple packages.
- Updated extension runtimes, custom tools loader, and TypeBox compatibility shim to expose and use ArkType instances.
- Added a comprehensive ArkType migration guide, validation parity tests, and helper utilities.
- Removed redundant PDF asset routing and parsing implementations from the read tool.
- Fixed `/dump` output (`session-dump-format.ts`) and the RPC `get_state` `dumpTools` payload (`rpc-mode.ts`) serializing Zod schema instances — leaking `def`/`shape` internals and stringified methods — by converting parameters through `zodToWireSchema` like providers receive.
- Fixed context-usage token estimation stringifying the Zod `def` tree, which overcounted tool schema tokens in the status line.
- Fixed tool-discovery indexing (`tool-index.ts`) returning empty `schemaKeys` for Zod tools, weakening BM25 ranking; keys are now recovered via wire conversion.
- Fixed the extension inspector panel rendering "(no arguments)" for Zod tools by reading properties off the converted wire schema.
- Added coverage in `tool-index.test.ts` plus new `context-usage.test.ts` and `session-dump-format.test.ts`, and a changelog entry.
can1357