oh-my-pi/docs/natives-media-system-utils.md

Natives media + system utilities

This document covers the media/system/conversion exports in @oh-my-pi/pi-natives: image processing, HTML conversion, clipboard access, token counting, macOS appearance/power helpers, ProjFS helpers, and work profiling.

Implementation files

• crates/pi-natives/src/image.rs
• crates/pi-natives/src/html.rs
• crates/pi-natives/src/clipboard.rs
• crates/pi-natives/src/tokens.rs
• crates/pi-natives/src/appearance.rs
• crates/pi-natives/src/power.rs
• crates/pi-natives/src/projfs_overlay.rs
• crates/pi-natives/src/prof.rs
• crates/pi-natives/src/task.rs
• packages/natives/native/index.d.ts

Note: there is no crates/pi-natives/src/work.rs; work profiling is implemented in prof.rs and fed by instrumentation in task.rs.

JS API ↔ Rust export/module mapping

JS export	Rust N-API export	Rust module
PhotonImage.parse(bytes)	PhotonImage::parse	image.rs
PhotonImage#resize(width, height, filter)	PhotonImage::resize	image.rs
PhotonImage#encode(format, quality)	PhotonImage::encode	image.rs
encodeSixel(bytes, targetWidthPx, targetHeightPx)	encode_sixel	image.rs
htmlToMarkdown(html, options?)	html_to_markdown	html.rs
copyToClipboard(text)	copy_to_clipboard	clipboard.rs
readImageFromClipboard()	read_image_from_clipboard	clipboard.rs
countTokens(input, encoding?)	count_tokens	tokens.rs
detectMacOSAppearance()	detect_mac_os_appearance	appearance.rs
MacAppearanceObserver.start(callback)	MacAppearanceObserver::start	appearance.rs
MacOSPowerAssertion.start(options?)	MacOSPowerAssertion::start	power.rs
projfsOverlayProbe/start/stop	ProjFS exports	projfs_overlay.rs
getWorkProfile(lastSeconds)	get_work_profile	prof.rs

Data format boundaries and conversions

Image (image)

• JS input boundary: Uint8Array encoded image bytes for PhotonImage.parse and encodeSixel.
• Rust decode boundary: bytes are copied/read, format is guessed with ImageReader::with_guessed_format(), then decoded to DynamicImage.
• In-memory state: PhotonImage stores Arc<DynamicImage>.
• Output boundary:
  • PhotonImage#encode(format, quality) returns a promise for encoded bytes (Vec<u8> in Rust; generated TS currently declares Promise<Array<number>>).
  • encodeSixel(...) returns a SIXEL escape string synchronously.

Format IDs:

• 0: PNG
• 1: JPEG
• 2: WebP
• 3: GIF

Encoding behavior:

• JPEG uses the provided quality with JpegEncoder::new_with_quality.
• WebP uses the webp crate encoder with quality as f32 in the same 0..=100 range.
• PNG/GIF ignore quality.
• Invalid dimensions for SIXEL (0 width or height) fail with Target SIXEL dimensions must be greater than zero.

HTML conversion (html)

• JS input boundary: HTML string + optional { cleanContent?: boolean; skipImages?: boolean }.
• Rust conversion boundary: conversion is scheduled through task::blocking("html_to_markdown", (), ...).
• Output boundary: Markdown string promise.

Conversion behavior:

• cleanContent defaults to false.
• When cleanContent=true, preprocessing uses PreprocessingPreset::Aggressive and hard-removal flags for navigation/forms.
• skipImages defaults to false.

Clipboard (clipboard)

• copyToClipboard(text) is a synchronous native call using arboard::Clipboard::set_text.
• readImageFromClipboard() runs in task::blocking("clipboard.read_image", (), ...).
• Image read returns null/undefined when arboard reports ContentNotAvailable.
• Successful image read re-encodes clipboard RGBA data as PNG and returns { data: Uint8Array, mimeType: "image/png" }.
• Clipboard access or image encoding failures reject/throw as native errors.

There is no current packages/natives TS wrapper that emits OSC52, handles Termux, or suppresses native clipboard failures. Any best-effort clipboard policy must live in consumers.

Tokens (tokens)

• countTokens(input, encoding?) accepts a single string or an array of strings.
• Arrays return one aggregate token count; encoding work is parallelized in Rust.
• Default encoding is O200kBase; Cl100kBase is also exported.
• The implementation uses ordinary encoding, not special-token handling.

macOS appearance and power helpers

• detectMacOSAppearance() returns "dark", "light", or null on non-macOS.
• MacAppearanceObserver.start(callback) returns a handle with stop(); on macOS it uses distributed notifications plus a 2-second polling fallback, and on non-macOS it is a no-op observer.
• MacOSPowerAssertion.start(options?) returns a handle with stop(); on macOS it acquires an IOKit assertion, and on other platforms it is a no-op handle.

Windows ProjFS helpers

• projfsOverlayProbe() reports whether ProjFS APIs are available.
• projfsOverlayStart(lowerRoot, projectionRoot) starts an overlay.
• projfsOverlayStop(projectionRoot) stops an overlay session.

These helpers are platform-specific; availability must be checked before relying on overlay behavior.

Work profiling (work)

• Collection boundary: profiling samples are produced by profile_region(tag) guards in task::blocking and task::future.
• Storage format: fixed-size circular buffer (MAX_SAMPLES = 10_000) storing stack path, duration, and timestamp.
• Output boundary: getWorkProfile(lastSeconds) returns:
  • folded: folded-stack text (flamegraph input)
  • summary: markdown table summary
  • svg: optional flamegraph SVG
  • totalMs, sampleCount

Lifecycle and state transitions

Image lifecycle

1. PhotonImage.parse(bytes) schedules a blocking decode task (image.decode).
2. On success, a native PhotonImage handle exists in JS.
3. resize(...) creates a new native handle (image.resize); old and new handles can coexist.
4. encode(...) schedules image.encode and materializes bytes without mutating image dimensions.
5. encodeSixel(...) decodes, optionally resizes to exact target dimensions with Lanczos3, and returns SIXEL text synchronously.

Failure transitions:

• Format detection/decode failure rejects parse promise or throws from SIXEL encoding.
• Encode failure rejects encode promise.
• Invalid SIXEL dimensions throw.

HTML lifecycle

1. htmlToMarkdown(html, options) schedules a blocking conversion task.
2. Conversion runs with defaulted options (cleanContent=false, skipImages=false) unless specified.
3. Returns markdown string or rejects.

Clipboard lifecycle

• Text copy constructs an arboard::Clipboard and calls set_text synchronously.
• Image read constructs an arboard::Clipboard, calls get_image, encodes PNG on success, maps ContentNotAvailable to None, and rejects other errors.

Work profiling lifecycle

1. No explicit start: profiling is active when task helpers execute.
2. Every instrumented task scope records one sample on guard drop.
3. Samples overwrite oldest entries after buffer capacity is reached.
4. getWorkProfile(lastSeconds) reads a time window and derives folded/summary/svg artifacts.

Failure transitions:

• SVG generation failure is soft (svg omitted/undefined), while folded and summary still return.
• Empty sample windows return empty folded data and no SVG, not an error.

Unsupported operations and error propagation

Image

• Unsupported decode input or corrupted bytes: strict failure.
• Invalid SIXEL target dimensions: strict failure.
• No JS fallback path in the natives package.

HTML

• Conversion errors are strict failures.
• Option omission is defaulting, not failure.

Clipboard

• Text copy is strict at the native API surface.
• Image read distinguishes "no image" (null/undefined) from operational failure (rejection).

Work profiling

• Retrieval is strict for the function call itself.
• Flamegraph SVG generation is nullable/optional.
• Buffer truncation is expected ring-buffer behavior.

Platform caveats

• Clipboard access depends on OS/session support exposed through arboard.
• macOS appearance and power helpers intentionally return no-op/null behavior on unsupported platforms.
• ProjFS helpers are Windows-specific and should be gated by projfsOverlayProbe().