vendel.xi2ix.com/xi2ix.com-website
1
0
Fork 0
Code Issues 11 Pull requests Projects Releases Packages Wiki Activity Actions

rollback-drill result: rollback-drill.yaml #20

New issue
Open
opened 2026-09-17 13:53:44 +00:00 by vendel.xi2ix.com · 0 comments
vendel.xi2ix.com commented 2026-09-17 13:53:44 +00:00
Owner
Copy link

Workflow: rollback-drill.yaml
Job: rollback-drill
Run: https://forgejo.lab.xi2ix.de/vendel.xi2ix.com/xi2ix.com-website/actions/runs/358
Commit: ee0eb9ff3d
Branch/ref: main
Triggered by: forgeadmin

This is a scheduled/manual rollback-drill result, NOT a production incident report.

What happened

The drill cycles and the rollback behaved as designed; some OTHER step in this job failed. Read the captured output below rather than assuming a provisioning or mechanism problem.

Drill state

Field Value
Streak counter before the drill's reset 0
Identical-revision preflight success
Helm revision before the drill's upgrade 116
Helm revision created by the drill's upgrade 117
Cycle 1 outcome=success, streak=1, rollback_fired=false
Cycle 2 outcome=success, streak=2, rollback_fired=false
Cycle 3 outcome=success, streak=3, rollback_fired=true
Cycle 3 rollback_ok true
Post-drill health check success

Expected progression is cycle1 streak=1/fired=false, cycle2 streak=2/fired=false, cycle3 streak=3/fired=true/rollback_ok=true.

Captured output (tail, per failing-eligible step)

/tmp/gsd-failhook-rollback-drill-rollback-drill-cycle1-streak.log

Cycle 1's real SSE-lifecycle assertions genuinely PASSED -- deliberately recording this as a simulated failure to drive the drill.
streak=1
rollback_fired=false
rollback_ok=n/a

/tmp/gsd-failhook-rollback-drill-rollback-drill-cycle1-test.log


Running 1 test using 1 worker

  ✓  1 tests/prod-smoke-sse-lifecycle.spec.ts:35:5 › production smoke: SSE lifecycle — reload, idle, concurrent-reopen-while-answering (zero 429s) (19.2s)

  1 passed (21.5s)

/tmp/gsd-failhook-rollback-drill-rollback-drill-cycle2-streak.log

Cycle 2's real SSE-lifecycle assertions genuinely PASSED -- deliberately recording this as a simulated failure to drive the drill.
streak=2
rollback_fired=false
rollback_ok=n/a

/tmp/gsd-failhook-rollback-drill-rollback-drill-cycle2-test.log


Running 1 test using 1 worker

  ✓  1 tests/prod-smoke-sse-lifecycle.spec.ts:35:5 › production smoke: SSE lifecycle — reload, idle, concurrent-reopen-while-answering (zero 429s) (19.5s)

  1 passed (21.2s)

/tmp/gsd-failhook-rollback-drill-rollback-drill-cycle3-streak.log

Cycle 3's real SSE-lifecycle assertions genuinely PASSED -- deliberately recording this as a simulated failure to drive the drill. This SHOULD cross the threshold and fire a real helm rollback.
Streak threshold reached (n=3 >= THRESHOLD=3) — firing helm rollback xi2ix in namespace xi2ix
helm history xi2ix (before rollback):
REVISION	UPDATED                 	STATUS    	CHART      	APP VERSION	DESCRIPTION     
108     	Sun Sep 13 03:47:17 2026	superseded	xi2ix-0.1.0	0.1.0      	Rollback to 106 
109     	Tue Sep 15 12:18:03 2026	superseded	xi2ix-0.1.0	0.1.0      	Upgrade complete
110     	Wed Sep 16 22:55:54 2026	superseded	xi2ix-0.1.0	0.1.0      	Upgrade complete
111     	Wed Sep 16 23:02:06 2026	superseded	xi2ix-0.1.0	0.1.0      	Upgrade complete
112     	Wed Sep 16 23:12:35 2026	superseded	xi2ix-0.1.0	0.1.0      	Upgrade complete
113     	Thu Sep 17 00:04:55 2026	superseded	xi2ix-0.1.0	0.1.0      	Upgrade complete
114     	Thu Sep 17 00:10:38 2026	superseded	xi2ix-0.1.0	0.1.0      	Upgrade complete
115     	Thu Sep 17 12:26:50 2026	superseded	xi2ix-0.1.0	0.1.0      	Upgrade complete
116     	Thu Sep 17 12:28:13 2026	superseded	xi2ix-0.1.0	0.1.0      	Rollback to 114 
117     	Thu Sep 17 13:51:44 2026	deployed  	xi2ix-0.1.0	0.1.0      	Upgrade complete
Rollback was a success! Happy Helming!
helm rollback xi2ix succeeded
streak=3
rollback_fired=true
rollback_ok=true
Rollback notification appended to open issue #18 (label rollback-fired:drill)

/tmp/gsd-failhook-rollback-drill-rollback-drill-cycle3-test.log


Running 1 test using 1 worker

  ✓  1 tests/prod-smoke-sse-lifecycle.spec.ts:35:5 › production smoke: SSE lifecycle — reload, idle, concurrent-reopen-while-answering (zero 429s) (20.1s)

  1 passed (21.7s)

/tmp/gsd-failhook-rollback-drill-rollback-drill-drift-heartbeat.log

Drift detector heartbeat: 2026-09-17T10:13:06Z (13237s ago); outcome recorded on that run: success
::error::The hourly drift detector last ran 13237s ago (2026-09-17T10:13:06Z), which is more than two hours -- it has missed at least two hourly ticks and is very likely not running at all. Its own start-of-run staleness check CANNOT report this, because a dead schedule never executes it; that blind spot is precisely why this assertion lives in a different workflow on a different schedule.

/tmp/gsd-failhook-rollback-drill-rollback-drill-post-drill-health.log


Running 1 test using 1 worker

  ✓  1 tests/prod-smoke-sse-lifecycle.spec.ts:35:5 › production smoke: SSE lifecycle — reload, idle, concurrent-reopen-while-answering (zero 429s) (21.4s)

  1 passed (23.0s)

/tmp/gsd-failhook-rollback-drill-rollback-drill-preflight.log

jq: /usr/bin/jq — jq-1.6
Live release latest revision: 116 (status: deployed)
--- live user-supplied values ---
image:
  tag: ee0eb9ff3da98195affb78ab4de3d1a0c590e52c
---------------------------------
Resolved live image tag: ee0eb9ff3da98195affb78ab4de3d1a0c590e52c
Preflight diff is EMPTY — the proposed revision renders byte-identically to live revision 116.
Release "xi2ix" has been upgraded. Happy Helming!
NAME: xi2ix
LAST DEPLOYED: Thu Sep 17 13:51:44 2026
NAMESPACE: xi2ix
STATUS: deployed
REVISION: 117
DESCRIPTION: Upgrade complete
TEST SUITE: None
Post-upgrade latest revision: 117 (status: deployed)
Revisions 116 and 117 have byte-identical stored manifests — the rollback fired at cycle 3 will land on content identical to what was already live.
REVISION	UPDATED                 	STATUS    	CHART      	APP VERSION	DESCRIPTION     
108     	Sun Sep 13 03:47:17 2026	superseded	xi2ix-0.1.0	0.1.0      	Rollback to 106 
109     	Tue Sep 15 12:18:03 2026	superseded	xi2ix-0.1.0	0.1.0      	Upgrade complete
110     	Wed Sep 16 22:55:54 2026	superseded	xi2ix-0.1.0	0.1.0      	Upgrade complete
111     	Wed Sep 16 23:02:06 2026	superseded	xi2ix-0.1.0	0.1.0      	Upgrade complete
112     	Wed Sep 16 23:12:35 2026	superseded	xi2ix-0.1.0	0.1.0      	Upgrade complete
113     	Thu Sep 17 00:04:55 2026	superseded	xi2ix-0.1.0	0.1.0      	Upgrade complete
114     	Thu Sep 17 00:10:38 2026	superseded	xi2ix-0.1.0	0.1.0      	Upgrade complete
115     	Thu Sep 17 12:26:50 2026	superseded	xi2ix-0.1.0	0.1.0      	Upgrade complete
116     	Thu Sep 17 12:28:13 2026	superseded	xi2ix-0.1.0	0.1.0      	Rollback to 114 
117     	Thu Sep 17 13:51:44 2026	deployed  	xi2ix-0.1.0	0.1.0      	Upgrade complete

/tmp/gsd-failhook-rollback-drill-rollback-drill-streak-reset.log

Shared prod-smoke streak counter BEFORE this drill's reset: 0
streak=0
rollback_fired=false
rollback_ok=n/a

/tmp/gsd-failhook-rollback-drill-rollback-drill-verify-progression.log

Observed progression:
  cycle 1: streak=1 rollback_fired=false
  cycle 2: streak=2 rollback_fired=false
  cycle 3: streak=3 rollback_fired=true rollback_ok=true
Preflight revisions: pre=116 post=117
Independent cross-check -- latest revision: 118, status: deployed, description: Rollback to 116
Helm recorded the rollback target as revision 116.
PROVEN: the threshold BEHAVIOUR is correct end-to-end -- the streak REPORTED on the script's stdout progressed 1,2,3 (the ConfigMap only ever holds 0,1,2,0; it never holds 3 -- see the header comment, this is by construction and not a defect), the rollback fired at cycle 3 and ONLY at cycle 3, the helm rollback command itself SUCCEEDED, and an independent helm history read confirms a new 'Rollback to ' revision with status deployed. Not merely that 'something fired'.
**Workflow:** rollback-drill.yaml **Job:** rollback-drill **Run:** https://forgejo.lab.xi2ix.de/vendel.xi2ix.com/xi2ix.com-website/actions/runs/358 **Commit:** ee0eb9ff3da98195affb78ab4de3d1a0c590e52c **Branch/ref:** main **Triggered by:** forgeadmin **This is a scheduled/manual rollback-drill result, NOT a production incident report.** ### What happened The drill cycles and the rollback behaved as designed; some OTHER step in this job failed. Read the captured output below rather than assuming a provisioning or mechanism problem. ### Drill state | Field | Value | | --- | --- | | Streak counter before the drill's reset | 0 | | Identical-revision preflight | success | | Helm revision before the drill's upgrade | 116 | | Helm revision created by the drill's upgrade | 117 | | Cycle 1 | outcome=success, streak=1, rollback_fired=false | | Cycle 2 | outcome=success, streak=2, rollback_fired=false | | Cycle 3 | outcome=success, streak=3, rollback_fired=true | | Cycle 3 rollback_ok | true | | Post-drill health check | success | Expected progression is cycle1 streak=1/fired=false, cycle2 streak=2/fired=false, cycle3 streak=3/fired=true/rollback_ok=true. ### Captured output (tail, per failing-eligible step) **/tmp/gsd-failhook-rollback-drill-rollback-drill-cycle1-streak.log** ``` Cycle 1's real SSE-lifecycle assertions genuinely PASSED -- deliberately recording this as a simulated failure to drive the drill. streak=1 rollback_fired=false rollback_ok=n/a ``` **/tmp/gsd-failhook-rollback-drill-rollback-drill-cycle1-test.log** ``` Running 1 test using 1 worker ✓ 1 tests/prod-smoke-sse-lifecycle.spec.ts:35:5 › production smoke: SSE lifecycle — reload, idle, concurrent-reopen-while-answering (zero 429s) (19.2s) 1 passed (21.5s) ``` **/tmp/gsd-failhook-rollback-drill-rollback-drill-cycle2-streak.log** ``` Cycle 2's real SSE-lifecycle assertions genuinely PASSED -- deliberately recording this as a simulated failure to drive the drill. streak=2 rollback_fired=false rollback_ok=n/a ``` **/tmp/gsd-failhook-rollback-drill-rollback-drill-cycle2-test.log** ``` Running 1 test using 1 worker ✓ 1 tests/prod-smoke-sse-lifecycle.spec.ts:35:5 › production smoke: SSE lifecycle — reload, idle, concurrent-reopen-while-answering (zero 429s) (19.5s) 1 passed (21.2s) ``` **/tmp/gsd-failhook-rollback-drill-rollback-drill-cycle3-streak.log** ``` Cycle 3's real SSE-lifecycle assertions genuinely PASSED -- deliberately recording this as a simulated failure to drive the drill. This SHOULD cross the threshold and fire a real helm rollback. Streak threshold reached (n=3 >= THRESHOLD=3) — firing helm rollback xi2ix in namespace xi2ix helm history xi2ix (before rollback): REVISION UPDATED STATUS CHART APP VERSION DESCRIPTION 108 Sun Sep 13 03:47:17 2026 superseded xi2ix-0.1.0 0.1.0 Rollback to 106 109 Tue Sep 15 12:18:03 2026 superseded xi2ix-0.1.0 0.1.0 Upgrade complete 110 Wed Sep 16 22:55:54 2026 superseded xi2ix-0.1.0 0.1.0 Upgrade complete 111 Wed Sep 16 23:02:06 2026 superseded xi2ix-0.1.0 0.1.0 Upgrade complete 112 Wed Sep 16 23:12:35 2026 superseded xi2ix-0.1.0 0.1.0 Upgrade complete 113 Thu Sep 17 00:04:55 2026 superseded xi2ix-0.1.0 0.1.0 Upgrade complete 114 Thu Sep 17 00:10:38 2026 superseded xi2ix-0.1.0 0.1.0 Upgrade complete 115 Thu Sep 17 12:26:50 2026 superseded xi2ix-0.1.0 0.1.0 Upgrade complete 116 Thu Sep 17 12:28:13 2026 superseded xi2ix-0.1.0 0.1.0 Rollback to 114 117 Thu Sep 17 13:51:44 2026 deployed xi2ix-0.1.0 0.1.0 Upgrade complete Rollback was a success! Happy Helming! helm rollback xi2ix succeeded streak=3 rollback_fired=true rollback_ok=true Rollback notification appended to open issue #18 (label rollback-fired:drill) ``` **/tmp/gsd-failhook-rollback-drill-rollback-drill-cycle3-test.log** ``` Running 1 test using 1 worker ✓ 1 tests/prod-smoke-sse-lifecycle.spec.ts:35:5 › production smoke: SSE lifecycle — reload, idle, concurrent-reopen-while-answering (zero 429s) (20.1s) 1 passed (21.7s) ``` **/tmp/gsd-failhook-rollback-drill-rollback-drill-drift-heartbeat.log** ``` Drift detector heartbeat: 2026-09-17T10:13:06Z (13237s ago); outcome recorded on that run: success ::error::The hourly drift detector last ran 13237s ago (2026-09-17T10:13:06Z), which is more than two hours -- it has missed at least two hourly ticks and is very likely not running at all. Its own start-of-run staleness check CANNOT report this, because a dead schedule never executes it; that blind spot is precisely why this assertion lives in a different workflow on a different schedule. ``` **/tmp/gsd-failhook-rollback-drill-rollback-drill-post-drill-health.log** ``` Running 1 test using 1 worker ✓ 1 tests/prod-smoke-sse-lifecycle.spec.ts:35:5 › production smoke: SSE lifecycle — reload, idle, concurrent-reopen-while-answering (zero 429s) (21.4s) 1 passed (23.0s) ``` **/tmp/gsd-failhook-rollback-drill-rollback-drill-preflight.log** ``` jq: /usr/bin/jq — jq-1.6 Live release latest revision: 116 (status: deployed) --- live user-supplied values --- image: tag: ee0eb9ff3da98195affb78ab4de3d1a0c590e52c --------------------------------- Resolved live image tag: ee0eb9ff3da98195affb78ab4de3d1a0c590e52c Preflight diff is EMPTY — the proposed revision renders byte-identically to live revision 116. Release "xi2ix" has been upgraded. Happy Helming! NAME: xi2ix LAST DEPLOYED: Thu Sep 17 13:51:44 2026 NAMESPACE: xi2ix STATUS: deployed REVISION: 117 DESCRIPTION: Upgrade complete TEST SUITE: None Post-upgrade latest revision: 117 (status: deployed) Revisions 116 and 117 have byte-identical stored manifests — the rollback fired at cycle 3 will land on content identical to what was already live. REVISION UPDATED STATUS CHART APP VERSION DESCRIPTION 108 Sun Sep 13 03:47:17 2026 superseded xi2ix-0.1.0 0.1.0 Rollback to 106 109 Tue Sep 15 12:18:03 2026 superseded xi2ix-0.1.0 0.1.0 Upgrade complete 110 Wed Sep 16 22:55:54 2026 superseded xi2ix-0.1.0 0.1.0 Upgrade complete 111 Wed Sep 16 23:02:06 2026 superseded xi2ix-0.1.0 0.1.0 Upgrade complete 112 Wed Sep 16 23:12:35 2026 superseded xi2ix-0.1.0 0.1.0 Upgrade complete 113 Thu Sep 17 00:04:55 2026 superseded xi2ix-0.1.0 0.1.0 Upgrade complete 114 Thu Sep 17 00:10:38 2026 superseded xi2ix-0.1.0 0.1.0 Upgrade complete 115 Thu Sep 17 12:26:50 2026 superseded xi2ix-0.1.0 0.1.0 Upgrade complete 116 Thu Sep 17 12:28:13 2026 superseded xi2ix-0.1.0 0.1.0 Rollback to 114 117 Thu Sep 17 13:51:44 2026 deployed xi2ix-0.1.0 0.1.0 Upgrade complete ``` **/tmp/gsd-failhook-rollback-drill-rollback-drill-streak-reset.log** ``` Shared prod-smoke streak counter BEFORE this drill's reset: 0 streak=0 rollback_fired=false rollback_ok=n/a ``` **/tmp/gsd-failhook-rollback-drill-rollback-drill-verify-progression.log** ``` Observed progression: cycle 1: streak=1 rollback_fired=false cycle 2: streak=2 rollback_fired=false cycle 3: streak=3 rollback_fired=true rollback_ok=true Preflight revisions: pre=116 post=117 Independent cross-check -- latest revision: 118, status: deployed, description: Rollback to 116 Helm recorded the rollback target as revision 116. PROVEN: the threshold BEHAVIOUR is correct end-to-end -- the streak REPORTED on the script's stdout progressed 1,2,3 (the ConfigMap only ever holds 0,1,2,0; it never holds 3 -- see the header comment, this is by construction and not a defect), the rollback fired at cycle 3 and ONLY at cycle 3, the helm rollback command itself SUCCEEDED, and an independent helm history read confirms a new 'Rollback to ' revision with status deployed. Not merely that 'something fired'. ```
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
plan/phase-06-grounded-ai-conversation
fix/ci-runner-dedicated-dind
v1.0
Labels
Clear labels   
ci-failure:ci.yaml-gates

Automated CI/CD failure-dedup label (gsd 260708-g89)

  
ci-failure:deploy.yaml-build-push-deploy

Automated CI/CD failure-dedup label (gsd 260708-g89)

  
ci-failure:drift-check.yaml-drift-check

Automated CI/CD failure-dedup label (gsd 260708-g89)

  
rollback-drill

Scheduled/manual rollback-drill result -- NEVER a real production incident report (gsd 260721-gwo)

  
rollback-fired:drill

A helm rollback fired at the prod-smoke failure threshold — emitted by scripts/rollback-streak.sh (gsd 260727-nvm)

  
rollback-fired:production

A REAL 3-strike production rollback fired (NOT a drill) — emitted by scripts/rollback-streak.sh at the moment of firing

No labels
ci-failure:ci.yaml-gates
ci-failure:deploy.yaml-build-push-deploy
ci-failure:drift-check.yaml-drift-check
rollback-drill
rollback-fired:drill
rollback-fired:production
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
vendel.xi2ix.com/xi2ix.com-website#20
No description provided.