18 Commits

Author SHA1 Message Date
can1357 e9888367d1 refactor: migrated packages to internal utility modules and removed external dependencies
- Implemented in-house, zero-dependency utility modules in `pi-utils` covering DOM manipulation, markdown parsing, templating, browser automation helpers, and terminal buffers.
- Migrated packages across the repository to consume the new internal utilities and `omptype` schema validators instead of external dependencies.
- Removed multiple external runtime and development dependencies including Zod, Marked, LRU cache, Turndown, and Puppeteer browser packages.
2026-08-05 13:39:09 +02:00
can1357 7c295faa52 feat: added native support for ArkType schemas alongside Zod
- Added native support for parsing, normalized validation, and serialization of ArkType schemas throughout the agent pipeline.
- Implemented pruning of unconstrained union branches and normalization helpers to handle unrepresentable strict-mode branches.
- Patched ArkType's schema package to preserve declared object key order during serialization.
- Integrated ArkType schemas into `agent-loop` and migrated coding agent tool parameters to ArkType format.
2026-06-18 00:59:56 +02:00
can1357 c341fe99a8 Merge PR #1837: fix(ai): preserve nullable enum tool schemas 2026-06-15 19:46:10 +02:00
can1357 64aa558e62 chore: consistency 2026-06-13 00:03:27 +02:00
can1357 e40c28146b fix(ai): reduced strict-mode retries by disabling strict tools after schema errors
- Tracked whether strict tool schemas were actually applied and used that flag when deciding strict-to-nonstrict retries.
- Recorded strict-tool failures in session state so later requests skip strict mode retries and avoid extra error rounds.
- Hardened strict schema normalization by flattening nested pure anyOf unions and expanded schema error matching for additional invalid-schema rejections.
2026-06-10 22:52:59 +02:00
roboomp 4428942b84 fix(ai): preserved constrained nullable unions
Only flatten optional anyOf schemas when the union wrapper has no sibling constraints that would exclude null.

Added a strict-schema regression for constrained anyOf properties so null remains an outer branch.
2026-06-10 20:18:39 +00:00
roboomp 4ccc9d7e88 fix(ai): flattened nullable strict unions
Flattened optional union tool schemas during strict enforcement so OpenRouter DeepSeek V4 no longer receives nested anyOf branches without a type.

Added schema-level and OpenRouter DeepSeek payload regressions for optional string-or-array tool parameters.

Fixes #2270
2026-06-10 20:13:54 +00:00
roboomp 89a47c8567 fix(ai): preserved nullable enum tool schemas
Distributed enum constraints across strict-mode type-array branches and kept null in collapsed nullable scalar enum wire schemas. Added regression coverage for the OpenAI Responses MCP nullable enum path.\n\nFixes #1835
2026-06-04 10:09:43 +00:00
can1357 54a60a7702 fix(ai/schema): hoisted description to anyOf wrapper in strict-mode unions
- Extracted `description` from type-array and nullable branches so it lives on the wrapper, not duplicated onto each variant.
- Replaced inline enum-type inference with `inferStrictPrimitiveTypeFromEnumOrConst`, covering both `enum` and `const` in sanitize and enforce paths.
- Mixed-primitive enums and non-primitive consts now fall back to non-strict instead of producing a typeless schema that OpenAI rejects on the wire.
2026-05-16 20:22:39 +02:00
can1357 9bd4d0099b fix(ai): resolved strict-mode local-ref inlining and single-item allOf
- Added strict-mode ref preprocessing to resolve local `#/` refs with sibling-key precedence.
- Added single-item allOf handling that inlines the sole schema entry and retains multi-item allOf.
- Added Anthropic schema normalization to retain whitelisted metadata and enforce stricter key handling.
- Added Google schema sanitizer options for snake_case renaming, null-collapse behavior, and ordering.
- Expanded schema tests for Anthropic, Google, and strict-mode parity across refs, arrays, and recursion.
2026-05-16 18:33:25 +02:00
can1357 5c7c5eccb8 fix(ai): corrected AI schema normalization to draft-2020-12 prefixItems
- Adopted draft-2020-12 tuple validation with `prefixItems`, rejecting array-valued `items`.
- Expanded strict-mode handling to recurse `prefixItems` entries and infer `array` when tuple prefixes exist.
- Normalized Anthropic schemas through `prefixItems`, keeping supported tuple constraints and dropping unsupported fields.
- Updated coding-agent schema metadata and tests to draft-2020-12 `$schema` targets, including MCP/theme fixtures.
- Added `trimTrailingWhitespace()` to strip trailing spaces/tabs and keep the original line when none exist.
2026-05-15 23:46:23 +02:00
can1357 734091f6f2 fix(ai/schema): fail validation when primitive ref recursion exceeds cap
Primitive $ref recursion now emits a validation issue when the depth cap is exceeded instead of returning true. This preserves termination without accepting invalid values at the end of a long ref chain.
2026-05-15 18:31:12 +02:00
can1357 82c24a3f74 fix(ai/schema): enforce conditional keywords, fix $ref recursion, preserve Zod root extras
- JSON Schema validator now enforces propertyNames, patternProperties, dependentRequired, dependencies, if/then/else, contains, and prefixItems instead of silently accepting values that violate them; unevaluated* still permissive but warns once.
- Recursive $ref no longer short-circuits to true on revisit: cycle detection keys on (ref, value-identity) with a primitive depth cap, so nested sub-schema violations are caught.
- Meta-validator now structurally validates if/then/else/dependencies sub-schemas and accepts draft-07 dependencies as either schemas or string-array dependent keys.
- Wire-schema null normalization no longer strips null-valued unknown root fields before preserveUnknownRootFields snapshots them, so task.simple and similar callers still see disallowed null arguments.
2026-05-15 17:47:20 +02:00
can1357 2867e1f4e3 feat(deps): added pi.zod exports and removed TypeBox package exports
- Added canonical `pi.zod` schema API exports and removed TypeBox package exports/imports.
- Migrated Tool schema typing from TypeBox to shared `TSchema`/Zod flow with legacy TypeBox compatibility.
- Updated AI provider adapters and MCP/agent builders to convert tool params through `toolWireSchema()`.
- Reworked schema validation from AJV to Zod-safe parsing with `fromTypeBox`, `toolWireSchema`, and meta schema checks.
2026-05-15 14:46:54 +02:00
Miroslav Drbal 416c5494fd Compress system prompt and tool descriptions for token efficiency
Reduces per-turn token cost of prompt and tool metadata by ~2,160
tokens (~8.6 KB across 12 prompt files) without losing instructional
signal. Moves literal default values from description text into
TypeBox's native `default:` keyword, and updates the strict-mode
sanitizer to preserve default info for providers that strip the
keyword.

Prompt compression

|File                                      |before |after  |delta |
|------------------------------------------|------:|------:|-----:|
|system-prompt.md                          |25,605 |21,530 |−16%  |
|prompts/tools/ast-grep.md                 | 4,892 | 3,880 |−21%  |
|prompts/tools/ast-edit.md                 | 4,141 | 3,549 |−14%  |
|prompts/tools/bash.md                     | 3,770 | 3,093 |−18%  |
|prompts/tools/task.md                     | 7,297 | 6,711 |−8%   |
|prompts/tools/read.md                     | 2,701 | 2,309 |−15%  |
|prompts/tools/debug.md                    | 2,791 | 2,412 |−14%  |
|prompts/tools/find.md                     |   860 |   561 |−35%  |
|prompts/tools/grep.md                     | 1,226 | 1,038 |−15%  |
|prompts/tools/todo-write.md               | 2,600 | 2,402 |−8%   |
|prompts/tools/python.md                   | 2,013 | 1,916 |−5%   |
|prompts/tools/hashline.md                 | 4,280 | 4,135 |−3%   |

Changes are textual compression only — grammar-scaffolding removed,
redundant bullets stripped, repeated facts consolidated. All
behavioral contracts, safety rules, worked examples, and
tool-precedence directives are preserved. The system prompt retains
RFC 2119 invocation, XML tag semantics, adversarial-caller guidance,
persistence doctrine, unit-of-change and no-forwarding-addresses
rules, completeness contract, DRY-at-2, earn-every-line, trust-
internal-code, tool precedence, AST tool priority, pattern-syntax
cheatsheet (via ast-grep.md/ast-edit.md), tool persistence,
outside-in code integrity, default follow-through, and procedural
steps 1-7. AST tool docs retain the class-wrapper +
method_definition sel example (the most error-prone usage pattern).
bash.md critical section retains MUST-weight on the ast_grep /
ast_edit directives.

Schema: defaults as first-class metadata

Moved 18 literal default values from `(default: X)` description
text into TypeBox's native `default:` keyword across:
ast-edit.ts, ast-grep.ts, bash.ts, browser.ts, find.ts, gh.ts,
grep.ts, python.ts, read.ts, ssh.ts. Runtime-resolved placeholders
(cwd, pr-<number>) remain as text since they cannot be literal
TypeBox defaults.

Added:
- `minItems: 1` on ast-edit.ts `ops` array — machine-enforces what
  the .md previously stated only in prose; handler already rejects
  empty ops arrays, this adds the schema-level constraint.
- `find.ts` `pattern` description enriched with facts previously
  only in find.md (comma-separated lists, simple patterns recurse
  from cwd).

Strict-mode sanitizer: inline `default` into `description`

OpenAI's Structured Outputs strict mode rejects schemas containing
`default` with HTTP 422 ("default is not permitted"). Affects
openai, azure, github-copilot, openrouter, cerebras, together,
zenmux, and deepseek providers.

`sanitizeSchemaForStrictMode` in packages/ai/src/utils/schema/
strict-mode.ts now appends ` (default: X)` to the sibling
`description` before stripping the `default` keyword. Non-strict
providers (Anthropic, Google) still see the native keyword.

Rules:
- Inline is skipped when description already contains `(default:`
  (prevents double-inlining on recursive calls)
- Inline is skipped when no sibling description exists (no
  synthesis)
- Formatting: strings as-is (`cwd`), other values via
  `JSON.stringify` (matches the conventional text form)

CONSTRAINTS.md documents the inlining rule alongside the existing
keyword-strip rule.

Regression tests

packages/ai/test/schema-strict-mode.test.ts gains 7 `it` blocks:
- number/bool/string default types inline correctly
- falsy defaults (`false`, `""`, `0`) are not confused with absent
- `null` default goes through JSON.stringify branch
- double-inline prevention when description already says `(default:`
- no synthesis when no description exists
- nested object property with default (recursion + cache path)
- type-array `[T, null]` branch with default on outer schema
  (variant-materialization path)

23/23 schema tests pass, 541/541 ai-package tests pass,
`bun check` clean.

Rationale

|Metric                          |Value          |
|--------------------------------|--------------:|
|Per-turn prompt savings         |~2,160 tok     |
|Files touched                   |25             |
|Lines changed                   |+325 / −296    |
|Schemas migrated to `default:`  |18             |
|New regression tests            |7              |
2026-04-23 22:59:37 +02:00
can1357 212d56bc11 feat: added strict-mode fallback for OpenAI tool calls with all_strict
- Added `toolStrictMode` support with `all_strict`/`none`/`mixed` options to OpenAI compatibility.
- Fixed OpenAI-completion strict-mode flows by capturing failed HTTP responses and retrying once as non-strict.
- Fixed completion error reporting by surfacing captured status, headers, and JSON `type`/`param`/`code` details.
- Improved strict-schema enforcement with WeakMap memoization and circular-schema detection in sanitization.
- Fixed OpenRouter provider lookup by resolving fallback model IDs for suffix and date variants in registry resolution.
- Refactored benchmark tooling and added async RPC error-window tracking for scheduled run execution.
2026-04-13 15:46:06 +02:00
can1357 cba80c79c3 fix(schema): harden all provider schema normalizers with cycle detection, fixpoint iteration, and correctness fixes
## New: schema compatibility validation API

Add `validateSchemaCompatibility(schema, provider)` in
`packages/ai/src/utils/schema/compatibility.ts` that performs a static
audit of a JSON Schema against three provider targets:

- `openai-strict`: checks forbidden keys, required/properties symmetry,
  additionalProperties constraint, and that every node declares a type,
  combinator, or $ref
- `google`: checks unsupported keyword set and array-valued type
- `cloud-code-assist-claude`: checks forbidden keywords, array type,
  null type, nullable keyword, and combiner presence; also validates via
  AJV 2020 draft

Add `validateStrictSchemaEnforcement(original, result)` to assert the
fail-open contract: when strict enforcement succeeds the output must pass
openai-strict validation; when it fails the output must be the original
schema object (same reference).

Export both functions and their types from `./utils/schema/index.ts`.

## New: shared constants in fields.ts

Extract `COMBINATOR_KEYS` (`anyOf`, `allOf`, `oneOf`) and add
`CCA_UNSUPPORTED_SCHEMA_FIELDS` as exported constants, eliminating the
local duplicate in `strict-mode.ts` and providing a canonical field set
for Cloud Code Assist (much narrower than the Google set — CCA supports
validation keywords like `additionalProperties`, `minLength`,
`pattern`, etc.).

## Fix: cycle detection in all recursive schema traversals

All recursive walkers now carry a `WeakSet<object>` guard. Previously any
schema with a reference cycle (or a schema object that appears at two
nodes in the tree) would cause an infinite loop or a stack overflow:

- `sanitizeSchemaForStrictMode` / `enforceStrictSchema`
- `normalizeSchemaForCloudCodeAssistClaude`
- `normalizeNullablePropertiesForCloudCodeAssist`
- `stripResidualCombiners`
- `sanitizeSchemaImpl` (Google sanitizer)
- `hasResidualCloudCodeAssistIncompatibilities`

`hasResidualCloudCodeAssistIncompatibilities` previously returned `true`
for already-visited nodes, producing false positives that forced the CCA
fallback schema on valid (but multiply-referenced) schemas. It now
correctly returns `false`.

## Fix: stripResidualCombiners iterates to fixpoint

The previous single-pass approach missed chained combiner reductions
where one collapsed variant exposed another reducible combiner. The
rewriter now loops until no further reduction occurs.

## Fix: mergeObjectCombinerVariants required-field computation

The merged object schema now takes the intersection of all variants'
`required` arrays, then unions in own-level required properties that
exist in the merged schema. Previously the `required` field was silently
dropped from the flattened schema, making all properties effectively
optional.

## Fix: sanitizeSchemaForGoogle improvements

- Type inference for const-collapsed enums: type is derived from all
  variants (must unanimously agree), falling back to inference from enum
  values; mixed null/non-null infers the non-null scalar type and sets
  `nullable: true`
- Const→enum deduplication now uses deep structural equality instead of
  `Object.is`
- Recursion spreads the full options object so new fields (`unsupportedFields`,
  `seen`) are not silently dropped when descending into sub-schemas
- Array-valued `type` is filtered to strings before processing
- Removed incorrect stripping of `additionalProperties: false` (the
  field is valid and should be preserved)
- Parameterized `unsupportedFields` in `SanitizeSchemaOptions` enables
  code reuse between the Google and CCA sanitizers

## Fix: sanitizeSchemaForStrictMode / enforceStrictSchema

- `nullable: true` is now stripped during sanitization and expanded into
  `anyOf: [schema, {type: "null"}]` in the enforcer output, matching
  what OpenAI strict mode requires
- Type inference: `type: "array"` is inferred when `items` is present;
  a scalar type is inferred from uniform `enum` values
- Const→enum merge uses deep equality to avoid duplicate entries when
  both `const` and `enum` exist with the same value
- `additionalProperties` is now dropped unconditionally in sanitization
  (previously only object-valued `additionalProperties` was recursed;
  non-object values were passed through)
- `enforceStrictSchema` recurses into `$defs` and `definitions` blocks
- `enforceStrictSchema` handles tuple-style `items` arrays
- `enforceStrictSchema` skips double-wrapping: optional properties
  already expressed as `anyOf: [..., {type: "null"}]` are not wrapped again
- `tryEnforceStrictSchema` now caches results in a `WeakMap` keyed on
  the input schema object to avoid redundant work on repeated calls

## Fix: mergeCompatibleEnumSchemas deep equality

Uses `areJsonValuesEqual` instead of `Object.is` when deduplicating
enum members, so structurally equal objects are not duplicated.

## New: test coverage

- `packages/ai/test/schema-normalization.test.ts`: comprehensive unit
  tests for strict mode, Google, and Cloud Code Assist normalization
- `packages/ai/test/schema-compatibility.test.ts`: unit tests for all
  three provider targets in the new compatibility validator
- `packages/coding-agent/test/tools/provider-schema-compatibility.test.ts`:
  integration test that instantiates every builtin and hidden tool, runs
  their parameter schemas through all three provider pipelines, and
  asserts zero compatibility violations
2026-02-28 18:41:10 +01:00
can1357 a44f8f1f48 refactor(ai): restructured schema utilities into modular utils/schema package with unified strict mode enforcement
- Extracted schema utilities from typebox-helpers and google-shared into new modular utils/schema package with 17 exported functions.
- Consolidated OpenAI strict mode schema enforcement across codex, completions, and responses providers using unified adaptSchemaForStrict() helper.
- Refactored credential ranking from hardcoded Codex-specific logic to pluggable CredentialRankingStrategy pattern with provider implementations.
- Migrated 500+ lines of Google schema sanitization and normalization logic from google-shared.ts to dedicated utils/schema modules with expanded functionality.
2026-02-28 18:38:29 +01:00