From ae8a3be9585d0918fe38fa199c963aeb80ee4f55 Mon Sep 17 00:00:00 2001 From: Michael Bolin Date: Sat, 28 Mar 2026 14:08:19 -0700 Subject: [PATCH] bazel: refresh the expired macOS SDK pin (#16128) ## Why macOS BuildBuddy started failing before target analysis because the Apple CDN object pinned in [`MODULE.bazel`](https://github.com/openai/codex/blob/fce0f76d577b5070f1e2b4a2abaa8350acfc38ff/MODULE.bazel#L28-L36) now returns `403 Forbidden`. The failure report that triggered this change was this [BuildBuddy invocation](https://app.buildbuddy.io/invocation/c57590e0-1bdb-4e19-a86f-74d4a7ded228). This repo uses `@llvm//extensions:osx.bzl` via `osx.from_archive(...)`, and that API does not discover a current SDK URL for us. It fetches exactly the `urls`, `sha256`, and `strip_prefix` we pin. Once Apple retires that `swcdn.apple.com` object, `@macos_sdk` stops resolving and every downstream macOS build fails during external repository fetch. This is the same basic failure mode we hit in [b9fa08ec619c96617a9ae2041c9ddb02d2c02434](https://github.com/openai/codex/commit/b9fa08ec619c96617a9ae2041c9ddb02d2c02434): the pin itself aged out. ## How I tracked it down 1. I started from the BuildBuddy error and copied the exact `swcdn.apple.com/.../CLTools_macOSNMOS_SDK.pkg` URL from the failure. 2. I reproduced the issue outside CI by opening that URL directly in a browser and by running `curl -I` against it locally. Both returned `403 Forbidden`, which ruled out BuildBuddy as the root cause. 3. I searched the repo for that URL and found it hardcoded in `MODULE.bazel`. 4. I inspected the `llvm` Bzlmod `osx` extension implementation to confirm that `osx.from_archive(...)` is just a literal fetch of the pinned archive metadata. There is no automatic fallback or catalog lookup behind it. 5. I queried Apple's software update catalogs to find the current Command Line Tools package for macOS 26.x. The useful catalog was: - `https://swscan.apple.com/content/catalogs/others/index-26-15-14-13-12-10.16-10.15-10.14-10.13-10.12-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog.gz` This is scriptable; it does not require opening a website in a browser. The catalog is a gzip-compressed plist served over HTTP, so the workflow is just: 1. fetch the catalog, 2. decompress it, 3. search or parse the plist for `CLTools_macOSNMOS_SDK.pkg` entries, 4. inspect the matching product metadata. The quick shell version I used was: ```shell curl -L \ | gzip -dc \ | rg -n -C 6 'CLTools_macOSNMOS_SDK\.pkg|PostDate|English\.dist' ``` That is enough to surface the current product id, package URL, post date, and the matching `.dist` file. If we want something less grep-driven next time, the same catalog can be parsed structurally. For example: ```python import gzip import plistlib import urllib.request url = "https://swscan.apple.com/content/catalogs/others/index-26-15-14-13-12-10.16-10.15-10.14-10.13-10.12-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog.gz" with urllib.request.urlopen(url) as resp: catalog = plistlib.loads(gzip.decompress(resp.read())) for product_id, product in catalog["Products"].items(): for package in product.get("Packages", []): package_url = package.get("URL", "") if package_url.endswith("CLTools_macOSNMOS_SDK.pkg"): print(product_id) print(product.get("PostDate")) print(package_url) print(product.get("Distributions", {}).get("English")) ``` In practice, `curl` was only the transport. The important part is that the catalog itself is a machine-readable plist, so this can be automated. 6. That catalog contains the newer `047-96692` Command Line Tools release, and its distribution file identifies it as [Command Line Tools for Xcode 26.4](https://swdist.apple.com/content/downloads/32/53/047-96692-A_OAHIHT53YB/ybtshxmrcju8m2qvw3w5elr4rajtg1x3y3/047-96692.English.dist). 7. I downloaded that package locally, computed its SHA-256, expanded it with `pkgutil --expand-full`, and verified that it contains `Payload/Library/Developer/CommandLineTools/SDKs/MacOSX26.4.sdk`, which is the correct new `strip_prefix` for this pin. The core debugging loop looked like this: ```shell curl -I rg 'swcdn\.apple\.com|osx\.from_archive' MODULE.bazel curl -L | gzip -dc | rg 'CLTools_macOSNMOS_SDK.pkg' pkgutil --expand-full CLTools_macOSNMOS_SDK.pkg expanded find expanded/Payload/Library/Developer/CommandLineTools/SDKs -maxdepth 1 -mindepth 1 ``` ## What changed - Updated `MODULE.bazel` to point `osx.from_archive(...)` at the currently live `047-96692` `CLTools_macOSNMOS_SDK.pkg` object. - Updated the pinned `sha256` to match that package. - Updated the `strip_prefix` from `MacOSX26.2.sdk` to `MacOSX26.4.sdk`. ## Verification - `bazel --output_user_root="$(mktemp -d /tmp/codex-bazel-sdk-fetch.XXXXXX)" build @macos_sdk//sysroot` ## Notes for next time As long as we pin raw `swcdn.apple.com` objects, this will likely happen again. When it does, the expected recovery path is: 1. Reproduce the `403` against the exact URL from CI. 2. Find the stale pin in `MODULE.bazel`. 3. Look up the current CLTools package in the relevant Apple software update catalog for that macOS major version. 4. Download the replacement package and refresh both `sha256` and `strip_prefix`. 5. Validate the new pin with a fresh `@macos_sdk` fetch, not just an incremental Bazel build. The important detail is that the non-`26` catalog did not surface the macOS 26.x SDK package here; the `index-26-15-14-...` catalog was the one that exposed the currently live replacement. --- MODULE.bazel | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/MODULE.bazel b/MODULE.bazel index 8163c0b01..e71f92ca7 100644 --- a/MODULE.bazel +++ b/MODULE.bazel @@ -27,11 +27,11 @@ register_toolchains("@llvm//toolchain:all") osx = use_extension("@llvm//extensions:osx.bzl", "osx") osx.from_archive( - sha256 = "6a4922f89487a96d7054ec6ca5065bfddd9f1d017c74d82f1d79cecf7feb8228", - strip_prefix = "Payload/Library/Developer/CommandLineTools/SDKs/MacOSX26.2.sdk", + sha256 = "1bde70c0b1c2ab89ff454acbebf6741390d7b7eb149ca2a3ca24cc9203a408b7", + strip_prefix = "Payload/Library/Developer/CommandLineTools/SDKs/MacOSX26.4.sdk", type = "pkg", urls = [ - "https://swcdn.apple.com/content/downloads/26/44/047-81934-A_28TPKM5SD1/ps6pk6dk4x02vgfa5qsctq6tgf23t5f0w2/CLTools_macOSNMOS_SDK.pkg", + "https://swcdn.apple.com/content/downloads/32/53/047-96692-A_OAHIHT53YB/ybtshxmrcju8m2qvw3w5elr4rajtg1x3y3/CLTools_macOSNMOS_SDK.pkg", ], ) osx.frameworks(names = [