GitHub’s 8-hour outage was a retry bug. Check yours tonight

The Aug 17 GitHub incident is what every team is talking about this week. The RCA is not “GitHub is down again.” It is optimistic retries, a VS Code 10x storm, and an autoscaler watching the wrong process.

On 17 August 2026 GitHub was degraded for 7 hours 47 minutes. Issues, PRs, APIs, Actions, Copilot, and enterprise SSO all hurt. Peak web/API errors sat around 20%. Archive and raw downloads hit about 50%. Two days later the RCA landed, and the industry conversation is not “switch hosts.” It is “we all ship this class of bug.”

The trigger was network saturation on load balancers in Central US. An Istio sidecar hit its concurrency limit. Autoscaling watched the host service, not the sidecar, so capacity never arrived. Optimistic retries then piled onto already-dead HAProxy nodes on the auth path. When GitHub failed some traffic over to Northern Virginia, delayed replies from one internal endpoint tripped a latent retry bug in VS Code that amplified Copilot token traffic by about 10x and delayed recovery of the Copilot Token Service until 21:02 UTC.

Step 1: Steal the postmortem, not the drama

Step 2: Find every “retry forever” in your HTTP clients

Axios, fetch wrappers, Stripe SDKs, npm, GitHub Actions, VS Code extensions — if a 502 makes the client fire the same request immediately, you are GitHub’s VS Code in miniature. Search for retry, axios-retry, and undici Agent options before you argue about hosting.

type RetryOptions = {
  retries: number;
  baseMs: number;
  capMs: number;
};

export async function withBackoff<T>(
  fn: () => Promise<T>,
  { retries = 3, baseMs = 200, capMs = 5_000 }: RetryOptions,
): Promise<T> {
  let attempt = 0;
  for (;;) {
    try {
      return await fn();
    } catch (err) {
      if (attempt >= retries) throw err;
      const jitter = Math.random() * baseMs;
      const wait = Math.min(capMs, baseMs * 2 ** attempt + jitter);
      await new Promise((r) => setTimeout(r, wait));
      attempt += 1;
    }
  }
}

Step 3: Cap retries and fail closed on 429 / 503

Do not retry every status. Timeouts and 502s can retry with jitter. 401, 403, and 404 should not. 429 and 503 need a Retry-After if the server sent one. GitHub had to temporarily return 403s and cut Copilot token traffic to recover. Your API should be allowed to say no.

export function shouldRetry(status: number): boolean {
  if (status === 429 || status === 502 || status === 503) return true;
  if (status >= 400 && status < 500) return false;
  return status >= 500;
}

export function retryAfterMs(header: string | null, fallback: number) {
  if (!header) return fallback;
  const seconds = Number(header);
  if (Number.isFinite(seconds)) return seconds * 1000;
  const date = Date.parse(header);
  return Number.isFinite(date) ? Math.max(0, date - Date.now()) : fallback;
}

Step 4: Point the autoscaler at the thing that actually saturates

GitHub’s policy watched the app and ignored Istio sidecar concurrency. If you run sidecars, Envoy, or a Node cluster behind HAProxy, the metric that pages you has to be the one that hits the wall — queue depth, outstanding requests, not CPU of the wrong container.

Step 5: Have a plan for when GitHub or npm is the delayed dependency

You cannot merge, you cannot npm install, Copilot is dead. That is a product incident for every team, not only GitHub’s. Cache the last green lockfile. Keep a tagged image you can redeploy without cloning. If you use AI editors, remember the other thread this month: ChainDrop (4 Aug) poisoned keyv and 400+ npm packages via preinstall hooks and IDE startup files. Outages and supply-chain worms are the same lesson — your laptop and CI retry and install more aggressively than your brain does.