The non-PTY bash streaming bridge queued every decoded chunk into
flume::unbounded and fired ThreadsafeFunction callbacks NonBlocking
with no budget, so a producer outrunning the JS event loop grew the
native queue (and the napi queue behind it) without bound — measured
33.5 MB queued for a 32 MiB stream with a stalled consumer, and
multi-GB RSS on longer runs. The downstream OutputSink caps sit after
the N-API boundary and cannot bound either queue.
Bound the pipeline end to end without dropping data:
- pi-natives: bridge_chunks now creates flume::bounded(64) and the
drain task (extracted as pump_chunks) awaits on_chunk.call_async per
coalesced <=64 KiB batch, so at most one batch sits in the napi
queue and the JS event loop's real consumption rate backpressures
the whole pipeline. If the JS side is gone, the pump exits and
drops the receiver so senders fail fast.
- pi-shell: emit_chunk sends with send_async().await — a full bridge
queue parks the pipe reader, which parks the child on its
stdout/stderr pipe (ordinary pipe backpressure) instead of
buffering; a disconnected receiver fails immediately so child pipes
always keep draining.
Unlike a drop-after-cap design, every byte still reaches JS: the
rolling tail view, lossless [raw output: artifact://…] capture, and
totalBytes accounting keep working for outputs past the display cap.
E2E (darwin-arm64 addon): 32 MiB through a JS callback stalling 1 ms
per call — lossless, 472 coalesced callbacks, peak RSS +21.8 MiB.
Fixes#4078
can1357
5d1e25f834