Skip to content

CI: make sccache resilient to transient cache-backend failures #3273

Open

Description

@leofang

The sccache daemon currently aborts the build when the remote cache backend returns a transient error at startup. Example from the recent GitHub/Azure outage in build-wheel · PR 3239, step 23 L881:

sccache: error: Server startup failed: cache storage failed to read:
Unexpected (temporary) at read =>
<Error><Code>ServerBusy</Code><Message>Egress is over the account limit.

The compile would have succeeded without the cache; the daemon's pre-flight check is what killed the job. We want sccache to degrade gracefully to a cacheless build in this case.

Proposal — try the env vars suggested by @trxcllnt offline

  • SCCACHE_SKIP_CACHE_CHECK=true — skip the pre-flight .sccache_check read/write that is the actual thing failing here. Documented in mozilla/sccache docs/Configuration.md; parsed in src/config.rs (skip_cache_check_from_env). Per Paul, cache errors after startup already do not fail the build; the pre-flight is the only hard gate.
  • SCCACHE_CACHE_RATE_LIMIT_ON_ERROR_COUNT=5 and SCCACHE_CACHE_RATE_LIMIT_ON_ERROR_WINDOW_SIZE_SECS=5 — temporarily disables cache I/O for 5s after 5 errors in a 5s window. Parsed in the rapidsai fork at src/config/client.rs:730-731 (confirmed present at the v0.18.0-rapids.7 tag that install-sccache pins). Not currently in upstream mozilla/sccache.

Where to apply

  • .github/workflows/build-wheel.yml — the failing path uses the GHA backend; this is where the Azure-blob-storage read fired. Add the env vars alongside the existing SCCACHE_GHA_ENABLED=true / SCCACHE_PREPROCESSOR_CACHE_GHA_ENABLED=true block.
  • .github/workflows/test-sdist-linux.yml — mirror if the behavior is useful for the local-disk sccache path too.

Related: #1061 (original sccache rollout).

-- Leo's bot

Activity

  1. added theissue type on Oct 28, 2026
  2. added this to the cuda.core 1.3.0 milestone on Oct 29, 2026
  3. removed theissue type on Oct 29, 2026
  4. added
    P1Medium priority - Should do
    and removed
    triageNeeds the team's attention
    on Oct 29, 2026
  5. trxcllnt commented on Oct 29, 2026

    @trxcllnt
    Member

    By the way, v0.18.0-rapids.8 has a fix related to using -fmodules-ts. CMake defaults to scanning for modules now, so bump to that version if you're seeing failures configuring any CMake projects.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    CI/CDCI/CD infrastructureP1Medium priority - Should do

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions