Postmortem ยท Sample: an illustrative incident, not a real event
Exports hung for 47 minutes on a confined Chromium
Image exports from CI runners that picked up a snap-packaged Chromium waited until the 60-second protocol timeout, then failed, for 47 minutes. 312 export jobs failed. A preference order that tried snap Chromium last, and a clearer error, fixed it.
Impact
| Measure | Value | Source |
|---|---|---|
| Duration | 47 minutes | CI job timestamps (sample) |
| Export jobs failed | 312 | CI job log (sample) |
| Users affected | Teams exporting on Ubuntu runners | Support tickets (sample) |
Timeline (UTC)
| Time | Event |
|---|---|
| 09:12 | Runner image updated; /snap/bin/chromium is now the first browser found. |
| 09:20 | First export job fails with a protocol timeout. |
| 09:41 | On-call notices the failure rate and starts investigating. |
| 09:52 | Snap confinement identified: the browser can't read the hidden .open-design/ folder. |
| 09:59 | Mitigated by setting OPEN_DESIGN_BROWSER_PATH on the runners. |
Root cause
- Browser discovery returned the first executable it found.
- Snap's confinement blocks hidden directories under
$HOMEand gives the browser a private/tmp. - The page never loaded, so export waited for the full protocol timeout instead of failing fast.
Contributing factors
- The timeout error didn't say which browser was used.
- No test covered a browser that launches but can't read the workspace.
What went well
- The
OPEN_DESIGN_BROWSER_PATHoverride let on-call mitigate without a release.
What went badly
- Detection relied on a person noticing the failure rate.
Action items
| Action | Owner | Priority | Due |
|---|---|---|---|
| Try snap Chromium last in discovery | Export | P1 | Done |
| Name the browser path in every export error | Export | P1 | Done |
| Alert on export failure rate in CI | Platform | P2 | 2 weeks |
Sources
This is a sample written to show the format. A real postmortem lists its incident notes, chat timeline, dashboards and the commits that caused and fixed the issue. The snap-confinement behaviour itself is real and is documented in packages/core/src/export/browserDiscovery.ts.