ADR-0001 — Retire blue/green deployment

Status: not proceeding, as of 14 September 2026

Nothing has been changed on any deployment group. The record below is the original proposal of 13 September, kept as written. Three of its supporting points did not survive review and measurement; they are corrected in Update — 14 September 2026 at the end, which carries the current state. Read the update before acting on anything above it.

Replace blue/green with in-place deployment with traffic control on the three blue/green CodeDeploy groups, because the traffic volume does not justify the complexity.

Raised 2026-09-13, out of CU-869enhrxj — a ticket that had already cut UAT from 23.7 to 10.7 minutes and production from 21.5 to about 13.5 by tuning blue/green rather than removing it. The observation that prompted this: at this traffic level the machinery is not earning its keep.

The load-bearing fact

From AWS/ApplicationELB RequestCount, 351 hourly samples across fourteen days:

	backend production	CRM production
busiest hour in 14 days	0.60 req/s	—
p90 hour	0.37 req/s	—
median hour	0.16 req/s	—
daily total	~19,000 req/day	~4,000 req/day
daily average	0.22 req/s	0.04 req/s

The busiest hour production has seen in two weeks is 0.6 requests per second. Blue/green exists to make deploys invisible under load. There is effectively no load to hide behind.

Production also deploys roughly once or twice a week — about 17 successful production deploys across the last 30 workflow runs, clustered on release days.

The target state already ran here

This is not a theoretical migration.

group	style	typical duration
ODH-UAT-CRM/uat	in-place, no load balancer	~1 min
ODH-UAT/…-Supervisor-deployment-group	in-place, no load balancer	~1 min
ODH-Prod-Supervisor/prod	in-place, no load balancer	~1 min
ODH-Pdf/{uat,prod}	in-place container swap	seconds
Orascom-Prod-app/orascom-crm-deploy-group	in-place + traffic control, until 2026-08-24	~4 min
ODH-UAT/ODH-UAT-Backend-deployment-group	blue/green	~10.7 min (was 21.6)
Orascom-Prod-app/orascom-backend-deploy-group	blue/green	~13.5 min (was 21.5)

CRM production ran in-place with traffic control, CodeDeployDefault.OneAtATime, against its ALB target group — successfully, at about four minutes a deploy — until 24 August. It was switched to blue/green after that and has not run a blue/green deploy since. Most of the rest of the estate is already in-place.

Zero downtime is not what is at stake

This is the point most likely to be misread, so it is worth stating directly.

In-place with traffic control on a two-instance fleet is a rolling deploy. CodeDeploy deregisters one instance, deploys to it, re-registers it, then does the other. Traffic never stops. Removing blue/green does not mean accepting downtime.

The actual trade

	blue/green	in-place + traffic control
downtime	none	none (rolling, two instances)
deploy duration	~13.5 min production	~4–5 min
rollback	instant — old fleet still live	redeploy previous revision (minutes)
fleet during deploy	2×	1×
green-fleet leak risk	open	none
configuration surface	large	small

What blue/green is costing: roughly nine of every 13.5 minutes on production; fleet duplication on every deploy; an open leak path where a green Auto Scaling group that survives a failed deploy keeps billing (one did, for five weeks); a large configuration surface — ASG copies, traffic-shift options, termination waits, alarm-gated rollback; and an untested path on CRM production right now.

What it genuinely buys: instant rollback, because the old fleet is still running. That is the one real loss, and it should not be minimised — in-place recovery is a redeploy of the previous revision, so minutes rather than seconds.

The honest counter-argument

In-place rollback leaves the bad revision on the instances while you recover. At 0.6 requests per second that is minutes of degraded service rather than seconds, which is cheap — but if a service has a business reason that is unacceptable, that service keeps blue/green. This decision does not have to be uniform.

Automatic rollback on DEPLOYMENT_FAILURE still works in-place: CodeDeploy redeploys the last known-good revision. What is lost specifically is the instant path, and DEPLOYMENT_STOP_ON_ALARM — which only became functional on 2026-09-13.

Proposed migration order

1. CRM production first. It is configured blue/green but has never run one, and ran in-place successfully until August. Reverting removes a live untested-path risk and validates the pattern on the lower-traffic service.
2. UAT backend second. A mistake is cheap there, and it produces a real duration measurement for the backend workload before production is touched.
3. Backend production last, once steps 1 and 2 have each run clean at least twice.

Each step: capture the deployment-group JSON first, change one group, diff it, deploy once, measure, then check for a leaked Auto Scaling group.

Use the correct leak predicate

An extra CodeDeploy_* Auto Scaling group is not the leak signal — normal cleanup produces one transiently, at zero instances and zero cost. The real predicate is length(Instances) > 0 on a group whose deployment has reached a terminal state. A check written on the name alone raises false alarms on healthy deploys, which is how a real check gets muted.

Open questions

These are why this is a proposal rather than a change.

1. Is minutes-not-seconds rollback acceptable for backend production? A business call, not a technical one, and the only real loss.
2. What happens to the CodeDeploy_*-named Auto Scaling groups? Reverting to in-place freezes the fleet on whatever green copy is current — the live production fleet is presently named after a deployment id, which is an artifact rather than a fleet name. Cleaner to create a properly-named group as part of the work.
3. How does this interact with the plan to retire UAT CodeDeploy entirely in favour of Docker Compose over SSM? If that is going ahead, step 2 above may be wasted work — or this record may supersede it. The two should not proceed independently; that is how an estate ends up running three deployment models at once.
4. Does DEPLOYMENT_STOP_ON_ALARM still earn its place? It only started working on 2026-09-13, and it is worth less without a live blue fleet to fall back to.

What has changed, so this is not mistaken for a change record

Nothing in this record has been executed. The only related change made on 2026-09-13 was terminationWaitTimeInMinutes 10 → 2 on both production blue/green groups — tuning within the existing model, and independent of this decision.

---

Update — 14 September 2026

Outcome: the change is not proceeding on any of the three groups. Everything above is the original proposal, left as written. This section records what changed.

The review was Ahmad Hamed’s, plus a three-part study (simplicity, cost, topology) run afterwards.

1. The CRM precedent does not hold

The section The target state already ran here reads CRM production’s in-place deployments as evidence that the proposed end state has already run successfully. It is not evidence of that.

Ahmad raised it in review: the last four of those deployments each ran on a single instance, so they were downtime deploys rather than rolling ones. Checking the deployment records against creator confirms a stronger version of the same point — every one of them was created by autoscaling, not by a person. They are Auto Scaling launch-hook installs onto newly launched instances, each replaying one revision zip, which is why none of them carries a BlockTraffic event. The date given in that section, 24 August, is the date of the last such install.

Counting every deployment on the three groups since 1 June 2026:

group	user, blue-green	user, in-place	autoscaling, in-place
orascom-backend-deploy-group	25	0	12
orascom-crm-deploy-group	10	0	23
ODH-UAT-Backend-deployment-group	34	0	17

No user-created in-place deployment has run on any of the three. Every group on the estate that does run in-place — both supervisors, CRM UAT, both PDF groups — runs it without traffic control. So the estate has experience with in-place deployment, but not with the specific configuration this record proposes.

Worth carrying forward: read deploymentStyle and creator per deployment. A deployment group’s current configuration does not describe what its past deployments did.

2. Fleet size changes the downtime answer

Zero downtime is not what is at stake is correct for a fleet of two or more, where in-place with traffic control is a rolling deploy. It does not hold at one instance, where there is no second instance to keep serving.

group	fleet	in-place without downtime?
orascom-backend-deploy-group (production)	2 × t4g.small	yes, rolling
orascom-crm-deploy-group (production)	1 × t4g.micro	no
ODH-UAT-Backend-deployment-group	1 × t4g.small	no

Two of the three are single-instance. The “none” in the downtime row of The actual trade therefore applies to production backend only. The proposed migration order, which put CRM production first as the lowest-risk case, had it backwards: single-instance makes it the hardest case, not the easiest.

3. The cost moves the wrong way

The proposal was raised under the constraint of cutting cost, or at least not increasing it. Priced from Cost Explorer actuals and the Pricing API, by two independent methods that agree:

	monthly, ex-VAT
All blue/green on this estate, today	$0.15 – $0.35 (under $4.20/year)
Preserving zero downtime after removal — UAT	+$3.81
Preserving zero downtime after removal — CRM production	+$10.82
Both	+$14.40

Blue/green needs no steady-state instances. Production backend runs two because MinSize=2, which is an availability choice rather than a deployment-style cost.

Separately, tuning within the existing model has already delivered most of the speed the proposal was after: UAT went from 23.73 to 10.66 minutes median, and CRM production still runs the untuned deregistration_delay=300 and health-check interval of 30 that production backend has already moved off.

4. Open question 3 is answered

CU-869eu4h16, which consolidates UAT onto a single instance running Docker Compose, is active and in design, and its phase 6 deletes the UAT CodeDeploy applications. Migrating a group that has an assigned deletion ticket would be wasted work. That ticket also saves $49.85 a month, of which $0.10 is attributable to blue/green — so that saving belongs to the consolidation, not to this proposal.

5. A claim made during the study, corrected

An interim draft of the study stated that in-place downtime would trip the 5xx alarm on every deploy and cause alert fatigue. That was written without reading the thresholds and is wrong. The alarm requires two consecutive 60-second periods at five or more, with treatMissingData: notBreaching. A 30–60 second outage falls inside one period. At CRM production’s 0.021 requests per second — about 1.3 a minute — five in one minute is not reachable. The unhealthy-hosts alarm would not fire either, because under traffic control the target is deregistered rather than unhealthy.

The corrected version is the more serious one: recurring 502s during deploys would be largely invisible to current monitoring rather than noisy. That is the same shape as the July 2026 incident, which ran for 71 hours before anyone noticed.

6. Still open from review: in-place without traffic control

Ahmad asked whether the single-instance groups could run in-place without traffic control, the way the Supervisor and CRM-UAT groups already do — the instance stays registered, so the exposure is however long the deploy leaves the application broken, and a release-folder plus symlink switch makes that very short.

The topology study reached the same option independently and ranked it first for those two groups. Measured against keeping traffic control on a single-instance group, the comparison favours it clearly: keeping traffic control means deregistration_delay 300 + install 66 + re-register 156, roughly 528 seconds of zero healthy targets and about 21 failed requests, whereas dropping it leaves a window of roughly 45 seconds during which about 1.8 requests are exposed.

This is not adopted here, and it is not closed either. Two things need settling first:

• groupmap.sh keys on deployment group name, not deployment style, and BeforeAllowTraffic does not exist without traffic control. Changing a group without editing that file stops migrations, stops the fail-closed Passport check, and stops the php-fpm reload — each silently, with the deployment still reporting success. It is a one-line change per group, but it has to ride in the same change.
• The exposure figure assumes a failing request fails quickly. AbstractClient builds every backend-to-CRM call with timeout(300) and connectTimeout(300) and no retry, so a hung dependency could hold php-fpm workers for five minutes rather than erroring out. That would make a clean 503 the better outcome and invert the comparison. Shortening that timeout is worth doing regardless.

It is recorded in the study rather than decided here.

What is being done instead

1. Explain the 227-second AllowTraffic bucket. It is the largest non-boot component of the deploy window and is application readiness, so it is a floor under every topology including in-place. It should be understood before any deployment-model change is judged.
2. Tune CRM production’s target group: deregistration_delay 300 → 120, health-check interval 30 → 20. Two fields, precedent already set on production backend, roughly three minutes per deploy, no migration.

Estate changes to date

terminationWaitTimeInMinutes went from 10 to 2 on both production blue/green groups on 13 September. That is tuning within the existing model and is independent of this record. Nothing else has changed.