What we promise, and how we've done.

incidents

Every one, with a postmortem.

1. 2026-10-14minorDelivery to GCS buckets paused for 47 minutesread postmortem +

   38 finished jobs waited 47 minutes for upload to Google Cloud Storage buckets after a credential rotation on our side. Every job had at least 2h 10m of slack left, so no deadline was missed.

   CauseOur rotation script updated the secret before confirming every delivery worker had reloaded it.

   What changed

   • Rotation now runs in two phases: add the new key, confirm all workers use it, then revoke the old one.
   • Delivery queue age alert lowered from 15 to 5 minutes.

   duration

   47 min

   jobs affected

   38

   deadlines missed

   0

   1. 14:02 UTC Scheduled rotation of the service account used for GCS delivery.
   2. 14:04 UTC First upload fails with 403. Retries back off as designed.
   3. 14:21 UTC Alert fires on delivery queue age > 15 min. On-call engineer paged.
   4. 14:38 UTC Cause found: new key not yet propagated to two of five delivery workers.
   5. 14:49 UTC Workers restarted with the new key. Queue drains in 7 minutes.
2. 2026-08-21majorTokyo capacity reclaimed during heatwave; 3 deadlines missedread postmortem +

   A grid emergency in Tokyo let our provider reclaim 1,400 unsold GPUs with 10 minutes' notice. 38 jobs moved to Seoul and Sydney in time; 3 jobs on the 6h tier finished 12 to 31 minutes late. All three were refunded in full and credited 2× their cost.

   CauseThe scheduler treated each region's reclaim risk as independent. A heat emergency reclaimed every interruptible GPU in the region at once, and the 6h tier had only a 22% slack margin.

   What changed

   • 6h jobs now keep 35% slack and never place more than half their GPU-hours in one grid zone.
   • Grid-emergency feeds for Japan, Korea and Australia are now an input to placement.
   • Missed deadline policy made automatic: full refund plus 2× credit, no ticket required.

   duration

   186 min

   jobs affected

   41

   deadlines missed

   3

   refunded $4,812.40 + 2× credit

   1. 05:40 UTC Provider notice: interruptible capacity in Tokyo to be reclaimed at 05:50.
   2. 05:51 UTC 41 running jobs checkpointed. Scheduler re-plans across Seoul and Sydney.
   3. 06:30 UTC 38 jobs resumed elsewhere with slack intact.
   4. 08:46 UTC Last of 3 late jobs delivered, 31 minutes after its deadline.
3. 2026-07-03minorReceipts issued without a root hash for 22 minutesread postmortem +

   19 receipts were issued with field digests but no root hash after a deploy changed the order fields were serialised in. The verifier flagged all 19. We re-issued them with correct roots within the hour.

   CauseA refactor sorted field digests alphabetically instead of in canonical order.

   What changed

   • The public verifier now runs against every receipt in CI before a receipt-service deploy can ship.

   duration

   22 min

   jobs affected

   19

   deadlines missed

   0

   1. 09:12 UTC Deploy of receipt service v1.8.0.
   2. 09:15 UTC Public verifier reports root mismatch on new receipts.
   3. 09:34 UTC Rollback to v1.7.4. New receipts verify.
   4. 10:05 UTC 19 affected receipts re-issued; customers notified by Slack.