vendel.xi2ix.com/xi2ix.com-website

Fork 0

rollback-drill result: rollback-drill.yaml #17

New issue

Closed

opened 2026-07-26 03:47:31 +00:00 by vendel.xi2ix.com · 1 comment

vendel.xi2ix.com commented

2026-07-26 03:47:31 +00:00

Owner

Workflow: rollback-drill.yaml
Job: rollback-drill
Run: https://forgejo.lab.xi2ix.de/vendel.xi2ix.com/xi2ix.com-website/actions/runs/265
Commit: 571b976463
Branch/ref: main
Triggered by: forgejo-actions

This is a scheduled/manual rollback-drill result, NOT a production incident report.

What happened

All 3 real SSE-lifecycle assertions genuinely passed, but the rollback mechanism itself did NOT fire at the threshold -- the counter-to-rollback mechanism is broken.

Cycle outcomes: 1=success, 2=success, 3=success. Rollback fired: false.

Captured output (tail, per failing-eligible step)

/tmp/gsd-failhook-rollback-drill-rollback-drill-cycle1-test.log


Running 1 test using 1 worker

  ✓  1 tests/prod-smoke-sse-lifecycle.spec.ts:35:5 › production smoke: SSE lifecycle — reload, idle, concurrent-reopen-while-answering (zero 429s) (24.9s)

  1 passed (27.4s)

/tmp/gsd-failhook-rollback-drill-rollback-drill-cycle2-test.log


Running 1 test using 1 worker

  ✓  1 tests/prod-smoke-sse-lifecycle.spec.ts:35:5 › production smoke: SSE lifecycle — reload, idle, concurrent-reopen-while-answering (zero 429s) (23.8s)

  1 passed (26.3s)

/tmp/gsd-failhook-rollback-drill-rollback-drill-cycle3-test.log


Running 1 test using 1 worker

  ✓  1 tests/prod-smoke-sse-lifecycle.spec.ts:35:5 › production smoke: SSE lifecycle — reload, idle, concurrent-reopen-while-answering (zero 429s) (21.9s)

  1 passed (23.5s)

**Workflow:** rollback-drill.yaml **Job:** rollback-drill **Run:** https://forgejo.lab.xi2ix.de/vendel.xi2ix.com/xi2ix.com-website/actions/runs/265 **Commit:** 571b9764639bc5cd5f968f0987420526cf8e22f1 **Branch/ref:** main **Triggered by:** forgejo-actions **This is a scheduled/manual rollback-drill result, NOT a production incident report.** ### What happened All 3 real SSE-lifecycle assertions genuinely passed, but the rollback mechanism itself did NOT fire at the threshold -- the counter-to-rollback mechanism is broken. Cycle outcomes: 1=success, 2=success, 3=success. Rollback fired: false. ### Captured output (tail, per failing-eligible step) **/tmp/gsd-failhook-rollback-drill-rollback-drill-cycle1-test.log** ``` Running 1 test using 1 worker ✓ 1 tests/prod-smoke-sse-lifecycle.spec.ts:35:5 › production smoke: SSE lifecycle — reload, idle, concurrent-reopen-while-answering (zero 429s) (24.9s) 1 passed (27.4s) ``` **/tmp/gsd-failhook-rollback-drill-rollback-drill-cycle2-test.log** ``` Running 1 test using 1 worker ✓ 1 tests/prod-smoke-sse-lifecycle.spec.ts:35:5 › production smoke: SSE lifecycle — reload, idle, concurrent-reopen-while-answering (zero 429s) (23.8s) 1 passed (26.3s) ``` **/tmp/gsd-failhook-rollback-drill-rollback-drill-cycle3-test.log** ``` Running 1 test using 1 worker ✓ 1 tests/prod-smoke-sse-lifecycle.spec.ts:35:5 › production smoke: SSE lifecycle — reload, idle, concurrent-reopen-while-answering (zero 429s) (21.9s) 1 passed (23.5s) ```

vendel.xi2ix.com added the

rollback-drill

label

2026-07-26 03:47:31 +00:00

vendel.xi2ix.com commented

2026-07-28 00:41:52 +00:00

Author

Owner

Corrected: the mechanism was never broken. This report's own verdict was the defect.

Resolved by quick task 260727-nvm (commits 4c5edad, 1a3a28f, 5d771bf, b6475e5, 8378509).

What this issue claimed

All 3 real SSE-lifecycle assertions genuinely passed, but the rollback mechanism itself
did NOT fire at the threshold -- the counter-to-rollback mechanism is broken.

That verdict was wrong. The mechanism fired correctly. The verification reported the opposite.

What actually happened on 2026-07-26

The shared streak counter (ConfigMap xi2ix-prod-smoke-streak) stood at 2 when the drill
started, from two earlier REAL prod-smoke failures on 2026-07-21 (this repo's Issue #12
comments 543 and 558 record streak=1 then streak=2, runs #262/#264). No deploy ran in
between to reset it. So the drill's three simulated failures ran:

Cycle	Counter before	Result
1	2	2+1 = 3 >= THRESHOLD -> real `helm rollback` FIRED, counter reset to 0
2	0	1 -> `rollback_fired=false`
3	1	2 -> `rollback_fired=false`

The Verify the rollback actually fired step read only steps.drill-3-streak.outputs.rollback_fired,
saw false, and concluded the mechanism was broken.

Independently confirmed by infra from the live cluster: revision 66, description
literally Rollback to 64, created 74 seconds after this drill started. Production then
served the content of revision 64 for 30 hours 45 minutes, until revision 67 landed on
2026-07-27T10:31:37Z. Nobody noticed — not us, not infra. This report said the rollback had
not fired, while it had.

What was fixed

Deterministic precondition — the drill now resets the streak before cycle 1, so the
progression no longer depends on production's prior state. This is the actual root-cause fix.
Verification checks threshold BEHAVIOUR, not just cycle 3 — the full progression, plus a
new rollback_ok flag, plus an independent helm history cross-check (revision
increased, description ^Rollback to , status deployed) that never consults the script's
own self-report. Previously a failed helm rollback would also have been reported green,
since the script printed rollback_fired=true regardless of exit code.
Production-neutral by construction — the drill first creates a content-identical revision
from the live release's own values, so the real rollback lands on identical content. A
fail-closed preflight aborts before touching production if byte-equality cannot be proven,
or if the live release status is not exactly deployed.
The streak steps' own stdout now reaches this issue. It never did, which is exactly why
this run was undiagnosable from this report alone.
Alert-on-fire — any rollback crossing the threshold now announces itself immediately from
both callers (drill and real deploy), naming the before/after revisions and the
Rollback to N target, marked DRILL or PRODUCTION. The 30h45m silence above is the reason.

Live-fire proof

Two real dispatches, runs #270 and #271 (2026-07-28), both green, both independently verified
by infra against the cluster rather than against our own report:

run #271:  rev 70 (baseline) -> 71 identical upgrade -> ROLLBACK -> 72 "Rollback to 70"
manifests: 70 == 71 == 72, all sha256 a70a5b8e02747f91, 6274 bytes
pod:       restartCount 0, untouched by either operation
cycles:    25.5s / 27.7s / 40.5s apart -- three genuinely separate cycles
alert:     Issue #18 filed at the moment of firing, marked DRILL

A real helm rollback fired and production served byte-identical content throughout.

One correction found by the verification itself

infra's watch-stream recording found no count=3 event in the ConfigMap. Confirmed from
source and it is by construction: rollback-streak.sh computes n=3 in memory, fires the
rollback, then unconditionally writes 0. The persisted sequence is 0,1,2,0; the sequence
reported on stdout is 1,2,3. Our assertions read stdout and were always correct — but our
documentation said "the recorded progression is always 0->1->2->3", which reads as the
ConfigMap and is false of it. Fixed in 8378509.

That mattered: had the recording been less careful, "no count=3 observed" would have read as
"the threshold branch did not run" — the same true-but-misleading signal shape as this very
issue, reproduced inside the verification of its own fix.

Residual, deliberately left open

Production's live revision is now described Rollback to 70. Content-identical and therefore
harmless, but a human reading helm history during a future incident could reasonably conclude
a real rollback happened and was never resolved. Logged as a follow-up; the likely fix is a
marker in the rollback's own description rather than another document.

Closing this issue. The weekly Sunday 03:45 UTC drill continues unchanged.

## Corrected: the mechanism was never broken. This report's own verdict was the defect. **Resolved by quick task `260727-nvm` (commits `4c5edad`, `1a3a28f`, `5d771bf`, `b6475e5`, `8378509`).** ### What this issue claimed > All 3 real SSE-lifecycle assertions genuinely passed, but the rollback mechanism itself > did NOT fire at the threshold -- the counter-to-rollback mechanism is broken. **That verdict was wrong.** The mechanism fired correctly. The verification reported the opposite. ### What actually happened on 2026-07-26 The shared streak counter (ConfigMap `xi2ix-prod-smoke-streak`) stood at **2** when the drill started, from two earlier REAL prod-smoke failures on 2026-07-21 (this repo's Issue #12 comments 543 and 558 record `streak=1` then `streak=2`, runs #262/#264). No deploy ran in between to reset it. So the drill's three simulated failures ran: | Cycle | Counter before | Result | |---|---|---| | 1 | 2 | 2+1 = 3 >= THRESHOLD -> **real `helm rollback` FIRED**, counter reset to 0 | | 2 | 0 | 1 -> `rollback_fired=false` | | 3 | 1 | 2 -> `rollback_fired=false` | The `Verify the rollback actually fired` step read **only** `steps.drill-3-streak.outputs.rollback_fired`, saw `false`, and concluded the mechanism was broken. **Independently confirmed by infra from the live cluster:** revision **66**, description literally `Rollback to 64`, created **74 seconds** after this drill started. Production then served the content of revision 64 for **30 hours 45 minutes**, until revision 67 landed on 2026-07-27T10:31:37Z. Nobody noticed — not us, not infra. This report said the rollback had not fired, while it had. ### What was fixed 1. **Deterministic precondition** — the drill now resets the streak before cycle 1, so the progression no longer depends on production's prior state. This is the actual root-cause fix. 2. **Verification checks threshold BEHAVIOUR**, not just cycle 3 — the full progression, plus a new `rollback_ok` flag, plus an **independent** `helm history` cross-check (revision increased, description `^Rollback to `, status `deployed`) that never consults the script's own self-report. Previously a *failed* `helm rollback` would also have been reported green, since the script printed `rollback_fired=true` regardless of exit code. 3. **Production-neutral by construction** — the drill first creates a content-identical revision from the live release's own values, so the real rollback lands on identical content. A fail-closed preflight aborts *before* touching production if byte-equality cannot be proven, or if the live release status is not exactly `deployed`. 4. **The streak steps' own stdout now reaches this issue.** It never did, which is exactly why this run was undiagnosable from this report alone. 5. **Alert-on-fire** — any rollback crossing the threshold now announces itself immediately from *both* callers (drill and real deploy), naming the before/after revisions and the `Rollback to N` target, marked DRILL or PRODUCTION. The 30h45m silence above is the reason. ### Live-fire proof Two real dispatches, **runs #270 and #271** (2026-07-28), both green, both independently verified by infra against the cluster rather than against our own report: ``` run #271: rev 70 (baseline) -> 71 identical upgrade -> ROLLBACK -> 72 "Rollback to 70" manifests: 70 == 71 == 72, all sha256 a70a5b8e02747f91, 6274 bytes pod: restartCount 0, untouched by either operation cycles: 25.5s / 27.7s / 40.5s apart -- three genuinely separate cycles alert: Issue #18 filed at the moment of firing, marked DRILL ``` A real `helm rollback` fired and production served byte-identical content throughout. ### One correction found by the verification itself infra's watch-stream recording found **no `count=3` event** in the ConfigMap. Confirmed from source and it is by construction: `rollback-streak.sh` computes `n=3` in memory, fires the rollback, then unconditionally writes `0`. The persisted sequence is `0,1,2,0`; the sequence reported on stdout is `1,2,3`. Our assertions read stdout and were always correct — but our *documentation* said "the recorded progression is always 0->1->2->3", which reads as the ConfigMap and is false of it. Fixed in `8378509`. That mattered: had the recording been less careful, "no `count=3` observed" would have read as "the threshold branch did not run" — the same true-but-misleading signal shape as this very issue, reproduced inside the verification of its own fix. ### Residual, deliberately left open Production's live revision is now described `Rollback to 70`. Content-identical and therefore harmless, but a human reading `helm history` during a future incident could reasonably conclude a real rollback happened and was never resolved. Logged as a follow-up; the likely fix is a marker in the rollback's own description rather than another document. Closing this issue. The weekly Sunday 03:45 UTC drill continues unchanged.

vendel.xi2ix.com closed this issue

2026-07-28 00:41:52 +00:00

vendel.xi2ix.com referenced this issue from a commit

2026-07-28 00:43:01 +00:00

docs(quick-260721-gwo): close out the rollback-testing task, seven days late