From 4df68d60438423b384b2b47fb3d6835641624757 Mon Sep 17 00:00:00 2001 From: can1357 Date: Thu, 30 Jul 2026 18:06:49 +0200 Subject: [PATCH] fix(ci): serialized native addon builds to avoid kata pod OOM - The aggregate //:natives-linux-all build links all six addon cdylibs concurrently; rustc RSS peaks OOMed the pod and the kernel killed the bazel server (exit 37, runs 30556752623 / 30557524371, twice at the same spot). - Build one addon target per invocation so the persistent server shares analysis and cached actions while the heavy links run one at a time; a final aggregate build stays as a completeness no-op. --- .github/workflows/ci.yml | 21 ++++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 31b2878c0..483db794b 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -250,7 +250,26 @@ jobs: run: | set -eo pipefail if [ "$(ulimit -Sn)" != unlimited ] && [ "$(ulimit -Sn)" -lt 65536 ]; then ulimit -Sn 65536 || true; fi - bazelisk --bazelrc="${{ steps.cache.outputs.rc }}" build //:natives-linux-all 2>&1 | tee "$RUNNER_TEMP/bazel-build.log" + # The addon cdylib links each peak at several GiB of rustc RSS; + # the aggregate //:natives-linux-all build runs all six + # concurrently, which OOMs the kata pod and takes the bazel + # server with it (exit 37 "Server terminated abruptly", runs + # 30556752623 / 30557524371). Build one addon per invocation — + # analysis and cached actions are shared through the persistent + # server, so only the heavy links serialize — then assemble the + # aggregate as a no-op (also catches targets added to the + # filegroup but missing from this list). + : > "$RUNNER_TEMP/bazel-build.log" + for target in \ + natives-linux-arm64 \ + natives-linux-musl-arm64 \ + natives-linux-musl-x64-baseline \ + natives-linux-x64-baseline \ + natives-linux-x64-modern \ + natives-win32-x64-baseline; do + bazelisk --bazelrc="${{ steps.cache.outputs.rc }}" build "//:$target" 2>&1 | tee -a "$RUNNER_TEMP/bazel-build.log" + done + bazelisk --bazelrc="${{ steps.cache.outputs.rc }}" build //:natives-linux-all 2>&1 | tee -a "$RUNNER_TEMP/bazel-build.log" # Cache-hit visibility: a supposedly warm build that executes # thousands of actions is the failure mode that made CI slow — make # it visible in the run summary instead of discovering it weeks in.