can1357
121 lines
8.6 KiB
Markdown
121 lines
8.6 KiB
Markdown
# inspect_image
|
|
|
|
> Send a local image file to a vision-capable model and return text analysis.
|
|
|
|
## Source
|
|
- Entry: `packages/coding-agent/src/tools/inspect-image.ts`
|
|
- Model-facing prompt: `packages/coding-agent/src/prompts/tools/inspect-image.md`
|
|
- Key collaborators:
|
|
- `packages/coding-agent/src/tools/inspect-image-renderer.ts` — TUI call/result rendering.
|
|
- `packages/coding-agent/src/utils/image-loading.ts` — path resolution, type detection, size gate, optional resize.
|
|
- `packages/coding-agent/src/utils/image-resize.ts` — downscale and recompress oversized images.
|
|
- `packages/coding-agent/src/tools/path-utils.ts` — resolve input path relative to session cwd.
|
|
- `packages/utils/src/mime.ts` — detect supported image formats from file bytes.
|
|
|
|
## Inputs
|
|
|
|
| Field | Type | Required | Description |
|
|
| --- | --- | --- | --- |
|
|
| `path` | `string` | Yes | Image path passed to `loadImageInput`; resolved relative to `session.cwd` by `resolveReadPath(...)`. |
|
|
| `question` | `string` | Yes | User prompt sent as a text content block alongside the image. |
|
|
|
|
## Outputs
|
|
The tool returns a single `AgentToolResult`:
|
|
|
|
- `content`: one text block, `[{ type: "text", text }]`, where `text` is the concatenated assistant text content from the model response.
|
|
- `details`:
|
|
- `model`: `<provider>/<id>` of the selected model.
|
|
- `imagePath`: resolved filesystem path returned by `loadImageInput(...)`.
|
|
- `mimeType`: MIME type actually sent to the model after optional resize/re-encode.
|
|
|
|
Model-visible output is single-shot, not streamed by this tool.
|
|
|
|
TUI rendering adds presentation-only truncation from `packages/coding-agent/src/tools/inspect-image-renderer.ts`:
|
|
|
|
- call preview truncates `question` to 100 columns,
|
|
- result view shows 4 lines collapsed or 16 lines expanded,
|
|
- each rendered output line is truncated to 120 columns,
|
|
- footer metadata shows `model · mimeType` when present.
|
|
|
|
## Flow
|
|
1. `InspectImageTool.execute(...)` rejects immediately if `images.blockImages` is enabled in session settings.
|
|
2. It reads `session.modelRegistry`; missing registry, empty registry, missing API key, or unresolved model each raise `ToolError` from `packages/coding-agent/src/tools/inspect-image.ts`.
|
|
3. Model selection tries, in order, `pi/vision`, `pi/default`, the active model string from the session, then `availableModels[0]`. `expandRoleAlias(...)` and `resolveModelFromString(...)` handle each lookup.
|
|
4. The chosen model must advertise `input.includes("image")`; otherwise execution fails before reading the file.
|
|
5. `loadImageInput(...)` in `packages/coding-agent/src/utils/image-loading.ts` resolves the path with `resolveReadPath(...)`, detects MIME type with `readImageMetadata(...)`, and rejects files larger than `MAX_IMAGE_INPUT_BYTES` (`20 * 1024 * 1024`, 20 MiB) using `ImageInputTooLargeError`.
|
|
6. `readImageMetadata(...)` in `packages/utils/src/mime.ts` inspects file headers only. Supported detected MIME types are `image/png`, `image/jpeg`, `image/gif`, and `image/webp`.
|
|
7. If `images.autoResize` is true, `loadImageInput(...)` calls `resizeImage(...)`. Resize failures are swallowed there and the original bytes are kept.
|
|
8. If MIME detection returned no supported image type, `execute(...)` throws `ToolError("inspect_image only supports PNG, JPEG, GIF, and WEBP files detected by file content.")`.
|
|
9. The tool calls `completeSimple(...)` with one user message containing two content parts in order:
|
|
- `{ type: "image", data: imageInput.data, mimeType: imageInput.mimeType }`
|
|
- `{ type: "text", text: params.question }`
|
|
10. `systemPrompt` is a one-element array rendered from `packages/coding-agent/src/prompts/tools/inspect-image-system.md`.
|
|
11. If the model response stop reason is `error` or `aborted`, the tool maps that to `ToolError`.
|
|
12. `extractResponseText(...)` concatenates only `text` content blocks from the assistant message, trims the result, and fails if nothing remains.
|
|
13. Success returns the text plus `details`; `inspectImageToolRenderer` formats the result for the TUI.
|
|
|
|
## Modes / Variants
|
|
- **Original image path**: `images.autoResize` disabled. The original file bytes are base64-encoded and sent with the detected MIME type.
|
|
- **Auto-resized path**: `images.autoResize` enabled. `resizeImage(...)` may downscale and re-encode the image before upload.
|
|
- **Unsupported image path**: file exists but header sniffing does not identify PNG/JPEG/GIF/WEBP. The tool returns a `ToolError` before any model call.
|
|
- **Oversize image path**: file size exceeds 20 MiB before upload. The tool returns a `ToolError` before any model call.
|
|
|
|
## Side Effects
|
|
- Filesystem
|
|
- Resolves and reads the target image from disk.
|
|
- Stats the file once with `Bun.file(...).stat()` and reads it fully with `fs.readFile(...)`.
|
|
- Network
|
|
- Sends the final base64 image payload plus question text to the selected model through `completeSimple(...)`.
|
|
- Session state
|
|
- Reads session settings, active model preferences, cwd, and model registry.
|
|
- Background work / cancellation
|
|
- Passes the caller `AbortSignal` into `completeSimple(...)`.
|
|
- Image preprocessing is local and not cancellation-aware in these helpers.
|
|
|
|
## Limits & Caps
|
|
- Supported detected input formats: `image/png`, `image/jpeg`, `image/gif`, `image/webp` (`SUPPORTED_IMAGE_MIME_TYPES` in `packages/utils/src/mime.ts`).
|
|
- Metadata sniff cap: `DEFAULT_IMAGE_METADATA_HEADER_BYTES = 256 * 1024` bytes. Format detection only reads up to 256 KiB from the file header.
|
|
- Upload input cap: `MAX_IMAGE_INPUT_BYTES = 20 * 1024 * 1024` bytes (20 MiB) in `packages/coding-agent/src/utils/image-loading.ts`.
|
|
- Auto-resize defaults in `packages/coding-agent/src/utils/image-resize.ts`:
|
|
- `maxWidth: 1568`
|
|
- `maxHeight: 1568`
|
|
- `maxBytes: 500 * 1024` bytes (500 KiB target)
|
|
- `jpegQuality: 75`
|
|
- Resize fast path: if the original image is already within `1568x1568` and within `maxBytes / 4` (125 KiB by default), `resizeImage(...)` returns the original bytes unchanged.
|
|
- Resize quality ladder: after the first encode pass, lossy retries use qualities `[70, 60, 50, 40]`.
|
|
- Resize dimension ladder: if quality reduction still misses the byte target, retries scale dimensions by `[1.0, 0.75, 0.5, 0.35, 0.25]` and stop if either dimension would fall below `100` pixels.
|
|
- First resize pass encodes PNG, JPEG, and WebP, then keeps the smallest encoded buffer. Fallback passes encode JPEG and WebP only, again keeping the smaller output.
|
|
- Renderer caps:
|
|
- `INSPECT_QUESTION_PREVIEW_WIDTH = 100`
|
|
- `INSPECT_OUTPUT_COLLAPSED_LINES = 4`
|
|
- `INSPECT_OUTPUT_EXPANDED_LINES = 16`
|
|
- `INSPECT_OUTPUT_LINE_WIDTH = 120`
|
|
|
|
## Errors
|
|
- Settings gate:
|
|
- `Image submission is disabled by settings (images.blockImages=true). Disable it to use inspect_image.`
|
|
- Model resolution / capability:
|
|
- `Model registry is unavailable for inspect_image.`
|
|
- `No models available for inspect_image.`
|
|
- `Unable to resolve a model for inspect_image.`
|
|
- `Resolved model <provider>/<id> does not support image input. Configure a vision-capable model for modelRoles.vision.`
|
|
- `No API key available for <provider>/<id>. Configure credentials for this provider or choose another vision-capable model.`
|
|
- Input file:
|
|
- `Image file too large: <size> exceeds <limit> limit.` from `ImageInputTooLargeError`, remapped to `ToolError`.
|
|
- `inspect_image only supports PNG, JPEG, GIF, and WEBP files detected by file content.` when header sniffing fails.
|
|
- Model call:
|
|
- `inspect_image request failed.` if the response stop reason is `error` without a provider message.
|
|
- Provider `errorMessage` is passed through when present.
|
|
- `inspect_image request aborted.` on aborted responses.
|
|
- `inspect_image model returned no text output.` when the assistant message contains no text blocks after filtering.
|
|
|
|
Failures surface as thrown `ToolError`s from `execute(...)`; the normal success return shape is not used for error reporting.
|
|
|
|
## Notes
|
|
- The model-facing prompt path on disk is `packages/coding-agent/src/prompts/tools/inspect-image.md`; the assignment's underscore form does not exist.
|
|
- Format support is based on file content, not filename extension. Renaming a non-image file to `.png` does not make it valid.
|
|
- `resolveReadPath(...)` tries macOS-specific path variants: shell-unescaped spaces, AM/PM narrow no-break-space filenames, NFD normalization, and curly-quote variants.
|
|
- `loadImageInput(...)` also computes `textNote`, `dimensionNote`, and final `bytes`, but `inspect_image` does not include those in tool output.
|
|
- Auto-resize can change the MIME type sent to the model. A JPEG or GIF input may be uploaded as PNG, JPEG, or WebP depending on which encoder output is smallest.
|
|
- If `resizeImage(...)` throws or cannot decode the image, `loadImageInput(...)` silently keeps the original base64 payload instead of failing.
|