perf(ci): cut warm CI time via download-skipping, splat reuse, PR gating

Four levers on top of the green pipeline: - kata jobs pass --remote_download_toplevel, so fully cache-hit builds stay metadata-only instead of pulling every intermediate artifact from bazel-remote (the bulk of the previous 6-minute TS-only main runs). - the xwin MSVC splat caches its ~1GiB CDN payload on the runner-cache PVC (OMP_XWIN_CACHE_DIR), instead of re-downloading per ephemeral pod. - main-push rust jobs export their bazel disk cache to the GitHub cache (once per lockfile change, shared linux scope). GitHub only shares default-branch caches across PRs, and main runs on kata where actions/cache never saved — so every fresh PR was building cold. - TS-only pull requests skip Rust validation entirely (gh pr diff path gate); their test jobs restore addons from the main-exported cache. Export runs disable top-level-only downloading: remote hits would otherwise export action entries whose blobs were never materialized.
2026-07-27 14:31:24 +02:00
parent ae01a76136
commit a7abeff1b7
4 changed files with 114 additions and 23 deletions
@@ -2,21 +2,35 @@ name: "Compose bazel cache config"
 description: >
   Single source of truth for how a CI job caches bazel work, emitted as a
   bazelrc fragment (rc output) consumers pass via `bazelisk --bazelrc=...`.
+
   omp-kata pods (detected via BAZEL_REMOTE_USER/BAZEL_REMOTE_PASSWORD from the
   bazel-remote-ci secret) get read-write gRPC access to the in-cluster
   bazel-remote service — an address that only resolves inside the cluster, so
-   nothing about the infrastructure leaks from this public repo. GitHub-hosted
-   runners never talk to that infrastructure: they use a local bazel disk cache
-   persisted with actions/cache, keyed on the crate lockfile and module
-   definition.
+   nothing about the infrastructure leaks from this public repo. With
+   `export: true` (the main-push rust job), the kata job additionally writes a
+   bazel disk cache and saves it to the GitHub Actions cache at job end under
+   the main branch scope — that is what makes pull requests warm: PR-created
+   caches are never shared across PRs, main-created ones are readable by every
+   PR.
+
+   GitHub-hosted runners never talk to the cluster: they restore the
+   main-exported disk cache (plus their own branch-scoped refresh saves).

 inputs:
   scope:
      description: >
-         Disk-cache key discriminator for GitHub-hosted runners; jobs building
-         different target sets (linux pair, darwin-all, msvc, validation) use
-         separate scopes so they don't evict each other's entries.
+         Disk-cache key discriminator. Every linux-family consumer (validation,
+         TS test jobs, linux/musl/win32 release jobs) shares the `linux` scope
+         so PRs hit the cache exported from main's rust job; darwin release
+         jobs keep per-target scopes and self-populate.
      required: true
+   export:
+      description: >
+         Write a disk cache during the build and save it to the GitHub cache
+         at job end (main-push rust job only). No-op when the key already
+         exists for the current lockfiles.
+      required: false
+      default: "false"

 outputs:
   rc:
@@ -37,9 +51,23 @@ runs:
           restore-keys: |
              bazel-disk-${{ inputs.scope }}-${{ runner.os }}-${{ runner.arch }}-

+      # Combined restore+save: restore is a no-op on the first run for these
+      # lockfiles (that's exactly when we want to build the export), and the
+      # post-job save only fires on a primary-key miss — so the export is
+      # written once per lockfile change, from a trusted main push.
+      - name: Prepare disk cache export (omp-kata)
+        if: env.BAZEL_REMOTE_USER != '' && inputs.export == 'true'
+        id: export
+        uses: actions/cache@v4
+        with:
+           path: ~/.cache/omp-bazel-disk
+           key: bazel-disk-${{ inputs.scope }}-Linux-X64-${{ hashFiles('Cargo.Bazel.lock', 'MODULE.bazel', 'rust-toolchain.toml') }}
+
      - name: Compose cache config
        id: compose
        shell: bash
+        env:
+           EXPORT_DISK: ${{ inputs.export == 'true' && steps.export.outputs.cache-hit != 'true' && 'true' || 'false' }}
        run: |
           set -euo pipefail
           rc="$RUNNER_TEMP/bazel-cache.rc"
@@ -58,8 +86,22 @@ runs:
               echo "common --tls_certificate=infra/bazel-remote/ca.crt"
               echo "common --remote_header='authorization=Basic ${auth}'"
               # PVC-backed shared repository cache (pods are ephemeral; without
-               # it every job re-downloads toolchains + crate archives).
+               # it every job re-downloads toolchains + crate archives). The
+               # xwin MSVC splat reuses the same PVC via OMP_XWIN_CACHE_DIR
+               # (its ~1GiB CDN payload is not repository-cacheable).
               echo "common --repository_cache=/opt/bazel-repo-cache"
+               echo "common --repo_env=OMP_XWIN_CACHE_DIR=/opt/bazel-repo-cache/xwin"
+               if [ "$EXPORT_DISK" = "true" ]; then
+                 # Export runs (once per lockfile change) write the disk cache
+                 # PRs restore. They must download everything: with top-level-
+                 # only downloading, remote hits would export action entries
+                 # whose blobs were never materialized.
+                 echo "common --disk_cache=$HOME/.cache/omp-bazel-disk"
+               else
+                 # Cache-hit work stays metadata-only; only requested top-level
+                 # outputs (the .node addons) are actually downloaded.
+                 echo "common --remote_download_toplevel"
+               fi
             } > "$rc"
           else
             {
@@ -131,12 +131,37 @@ jobs:
      runs-on: ${{ github.event_name == 'pull_request' && 'ubuntu-22.04' || 'omp-kata' }}
      steps:
         - uses: actions/checkout@v4
-         - uses: ./.github/actions/bun-install
+         # TS-only PRs skip Rust validation entirely; addons for the TS test
+         # jobs come from the main-exported disk cache. Pushes always run.
+         - name: Detect Rust-affecting changes
+           id: changes
+           shell: bash
+           env:
+              GH_TOKEN: ${{ github.token }}
+           run: |
+              if [ "${{ github.event_name }}" != "pull_request" ]; then
+                echo "rust=true" >> "$GITHUB_OUTPUT"
+                exit 0
+              fi
+              if gh pr diff ${{ github.event.pull_request.number }} --name-only \
+                 | grep -qE '^(crates/|Cargo\.(toml|lock)|Cargo\.Bazel\.lock|MODULE\.bazel|BUILD\.bazel|\.bazelrc|\.bazelversion|bazel/|rust-toolchain\.toml|rustfmt\.toml|scripts/bazel-natives|\.github/actions/bazel-|\.github/workflows/ci\.yml)'; then
+                echo "rust=true" >> "$GITHUB_OUTPUT"
+              else
+                echo "No Rust-affecting changes; skipping validation."
+                echo "rust=false" >> "$GITHUB_OUTPUT"
+              fi
+         - if: steps.changes.outputs.rust == 'true'
+           uses: ./.github/actions/bun-install
         - id: cache
+           if: steps.changes.outputs.rust == 'true'
           uses: ./.github/actions/bazel-cache
           with:
-              scope: validation
+              scope: linux
+              # Main pushes export the disk cache PRs restore (once per
+              # lockfile change; no-op otherwise).
+              export: ${{ github.event_name != 'pull_request' }}
         - name: Rust tests
+           if: steps.changes.outputs.rust == 'true'
           # The ulimit guard runs in the step that launches the bazel server
           # (limits are per-process and the server persists across steps).
           run: |
@@ -147,14 +172,17 @@ jobs:
         # `[lints] workspace = true` get the workspace policy, the vendored
         # brush fork is exempt (same as run-rs-task.ts's cargo excludes).
         - name: Clippy (workspace lint policy on opted-in crates)
+           if: steps.changes.outputs.rust == 'true'
           run: |
              bazelisk query "kind('rust_library|rust_shared_library', //crates/pi-ast/... + //crates/pi-iso/... + //crates/pi-natives/... + //crates/pi-shell/... + //crates/pi-walker/...)" \
                | xargs bazelisk --bazelrc="${{ steps.cache.outputs.rc }}" build --config=clippy-strict --
         - name: Clippy (default lints elsewhere)
+           if: steps.changes.outputs.rust == 'true'
           run: |
              bazelisk query "kind('rust_library|rust_shared_library', //crates/... - (//crates/pi-ast/... + //crates/pi-iso/... + //crates/pi-natives/... + //crates/pi-shell/... + //crates/pi-walker/...) - //crates/vendor/brush-core/... - //crates/vendor/brush-builtins/...)" \
                | xargs bazelisk --bazelrc="${{ steps.cache.outputs.rc }}" build --config=clippy --
         - name: Rustfmt
+           if: steps.changes.outputs.rust == 'true'
           run: bazelisk --bazelrc="${{ steps.cache.outputs.rc }}" build --config=rustfmt //crates/...
         - name: Warm native addon cache (main push)
           if: github.event_name != 'pull_request'
@@ -175,7 +203,7 @@ jobs:
         - uses: ./.github/actions/bazel-natives
           with:
              targets: linux-x64-baseline linux-x64-modern
-              cache-scope: linux-x64-pair
+              cache-scope: linux
         - name: Test workspace packages and repo scripts (TS)
           env:
              OMP_TEST_CONCURRENCY: "4"
@@ -194,7 +222,7 @@ jobs:
         - uses: ./.github/actions/bazel-natives
           with:
              targets: linux-x64-baseline linux-x64-modern
-              cache-scope: linux-x64-pair
+              cache-scope: linux
         - name: Test coding-agent singleton/global-state bucket
           # Keep global Settings/env/fake-timer tests serial; native addon
           # artifacts are still available like every other coding-agent bucket.
@@ -213,7 +241,7 @@ jobs:
         - uses: ./.github/actions/bazel-natives
           with:
              targets: linux-x64-baseline linux-x64-modern
-              cache-scope: linux-x64-pair
+              cache-scope: linux
         - name: Test native/TUI/browser-ish packages (TS)
           env:
              OMP_TEST_CONCURRENCY: "4"
@@ -232,7 +260,7 @@ jobs:
         - uses: ./.github/actions/bazel-natives
           with:
              targets: linux-x64-baseline linux-x64-modern
-              cache-scope: linux-x64-pair
+              cache-scope: linux
         - name: Test coding-agent UI/TUI bucket
           env:
              OMP_TEST_CONCURRENCY: "2"
@@ -251,7 +279,7 @@ jobs:
         - uses: ./.github/actions/bazel-natives
           with:
              targets: linux-x64-baseline linux-x64-modern
-              cache-scope: linux-x64-pair
+              cache-scope: linux
         - name: Test coding-agent runtime bucket
           # Runtime/session tests import native-backed barrels too; keep this
           # separate for concurrency, not as a native-free guardrail.
@@ -272,7 +300,7 @@ jobs:
         - uses: ./.github/actions/bazel-natives
           with:
              targets: linux-x64-baseline linux-x64-modern
-              cache-scope: linux-x64-pair
+              cache-scope: linux
         - name: Test coding-agent native/unit bucket
           env:
              OMP_TEST_CONCURRENCY: "4"
@@ -291,7 +319,7 @@ jobs:
         - uses: ./.github/actions/bazel-natives
           with:
              targets: linux-x64-baseline linux-x64-modern
-              cache-scope: linux-x64-pair
+              cache-scope: linux
         - name: CLI smoke test
           run: bun run ci:test:smoke

@@ -307,7 +335,7 @@ jobs:
         - uses: ./.github/actions/bazel-natives
           with:
              targets: linux-x64-baseline linux-x64-modern
-              cache-scope: linux-x64-pair
+              cache-scope: linux
         - name: Install method smoke tests
           env:
              OMP_INSTALL_TEST_SKIP_NATIVE_BUILD: "1"
@@ -421,7 +449,7 @@ jobs:
           uses: ./.github/actions/bazel-natives
           with:
              targets: ${{ matrix.native_targets }}
-              cache-scope: release-${{ matrix.target_id }}
+              cache-scope: ${{ startsWith(matrix.target_id, 'darwin') && format('release-{0}', matrix.target_id) || 'linux' }}
         - name: Build release binary
           env:
              RELEASE_TARGETS: ${{ matrix.target_id }}
@@ -588,7 +616,7 @@ jobs:
           uses: ./.github/actions/bazel-natives
           with:
              targets: linux-x64-baseline linux-x64-modern
-              cache-scope: linux-x64-pair
+              cache-scope: linux
         - name: Publish to npm
           env:
              # Fallback auth: setup-node wrote an .npmrc referencing
@@ -103,6 +103,16 @@ def _xwin_sysroot_impl(rctx):
        stripPrefix = prefix,
    )
    rctx.report_progress("Splatting MSVC CRT + Windows SDK via xwin (first fetch ~1 GiB from the Microsoft CDN)")
+
+    # OMP_XWIN_CACHE_DIR points at a persistent location (CI: the runner-cache
+    # PVC via --repo_env) so ephemeral pods reuse the ~1 GiB CDN download the
+    # repository cache cannot hold. Unset (dev machines): a repo-local cache,
+    # deleted after the --copy splat.
+    cache_dir = rctx.os.environ.get("OMP_XWIN_CACHE_DIR", "")
+    ephemeral_cache = cache_dir == ""
+    if ephemeral_cache:
+        cache_dir = ".xwin-cache"
+
    result = rctx.execute(
        [
            rctx.path("xwin"),
@@ -114,7 +124,7 @@ def _xwin_sysroot_impl(rctx):
            "--manifest-version",
            _XWIN_MANIFEST_VERSION,
            "--cache-dir",
-            ".xwin-cache",
+            cache_dir,
            "splat",
            "--copy",
            "--output",
@@ -130,7 +140,8 @@ def _xwin_sysroot_impl(rctx):
        ))

    # Drop the download cache (the splat is a --copy, not links into it).
-    rctx.delete(".xwin-cache")
+    if ephemeral_cache:
+        rctx.delete(cache_dir)

    # xwin drops `<sdk-version> -> .` alias symlinks (e.g. sdk/lib/10.0.26100)
    # so version-qualified include/lib paths resolve. They are self-referential
@@ -153,4 +164,7 @@ def _xwin_sysroot_impl(rctx):
 xwin_sysroot_repository = repository_rule(
    implementation = _xwin_sysroot_impl,
    doc = "MSVC CRT + Windows SDK sysroot splatted by a pinned xwin release.",
+    # Persistent splat cache location; changing it only changes where the CDN
+    # payload lands, not the splat contents, but Bazel still refetches.
+    environ = ["OMP_XWIN_CACHE_DIR"],
 )
@@ -412,7 +412,14 @@ bazel build \
 On omp-kata the credentials come from the injected pod env
 (`bazel-remote-ci` secret) and `.github/actions/bazel-cache` composes the rc
 fragment. GitHub-hosted jobs get the disk-cache branch of the same action —
-no remote endpoint, no credentials, no infrastructure knowledge.
+no remote endpoint, no credentials, no infrastructure knowledge. The bridge
+between the two worlds is the **disk-cache export**: main-push rust jobs
+write a bazel disk cache alongside the remote cache and save it to the
+GitHub Actions cache (once per lockfile change, `linux` scope). GitHub only
+shares caches from the default branch across pull requests, so this export
+is what keeps PR builds warm; kata jobs otherwise skip artifact downloads
+entirely (`--remote_download_toplevel`), and the xwin MSVC splat persists on
+the runner-cache PVC (`OMP_XWIN_CACHE_DIR`).

 **(b) Cargo registry cache** - the scale-set pod template mounts only the
 immutable download cache and sparse index at