How deployment works

You push to a branch, GitHub Actions builds a complete deploy revision, and AWS CodeDeploy rolls it out. This page is the single “how it ships” picture. For which environment a branch reaches and what each one is for, see Environments; for the AWS estate underneath, see Infrastructure.

One-sentence version

Open a PR against uat, get it past the required check and one approval → merge → GitHub Actions builds a self-contained revision → CodeDeploy deploys it; production shifts traffic automatically once the new fleet is healthy, and every deploy posts a status card to Teams.

The rail, end-to-end

flowchart LR
DEV(Developer):::person -->|PR → uat, then uat → main| GH[GitHub repo<br/>OrascomDevelopment]:::src
GH -->|push on uat or main| GHA[GitHub Actions<br/>reusable laravel-deploy.yml]:::ci
GHA -->|build self-contained<br/>revision .zip| S3[(S3<br/>orascom-deploy-bucket)]:::store
GHA -->|create deployment| CD[AWS CodeDeploy]:::deploy
S3 --> CD
CD -->|blue/green backend<br/>in-place elsewhere| ASG[EC2 Auto Scaling<br/>Graviton t4g · behind ALB]:::compute
classDef person fill:#e7eef6,stroke:#2563A8,color:#0b1b2b;
classDef src fill:#e7eef6,stroke:#2563A8,color:#0b1b2b;
classDef ci fill:#efe7f7,stroke:#7A4FB5,color:#0b1b2b;
classDef deploy fill:#dff0f0,stroke:#0C7C84,color:#0b1b2b;
classDef store fill:#eceff1,stroke:#5B6B7A,color:#0b1b2b;
classDef compute fill:#f7efdc,stroke:#B7791F,color:#0b1b2b;

Deploy rail — GitHub Actions → CodeDeploy (backend & CRM)

1. Merge a pull request into uat (or promote uat → main) in the GitHub org OrascomDevelopment. Those are the only two branches that deploy anywhere.
2. GitHub Actions builds the revision. For the backend and the CRM this is a reusable workflow (portals-workflows/laravel-deploy.yml, called at a pinned commit SHA): it installs dependencies and packages a self-contained artifact — the built code plus the CodeDeploy appspec.yml and its lifecycle hooks — then uploads it to S3 (orascom-deploy-bucket). This is the key property of the rail: the revision carries the code, so a box never has to git pull to get it.
3. CodeDeploy deploys the revision. On the box, lifecycle hooks render the environment file from AWS Secrets Manager, run database migrations under a fleet-safe lock, and clear caches — then health-check before any traffic moves.

Not every app uses the same builder

Only the backend and the CRM call the reusable laravel-deploy.yml. The PDF service has its own standalone workflow, and the frontend and this docs site have no deploy workflow at all — they build from a git integration (AWS Amplify and Cloudflare Pages respectively). Treat “the reusable workflow” as the backend/CRM rail, not as the whole platform.

Per-app × environment

Derived from the workflow files on each repo’s default branch. UAT changed on 2026-09-22 and the old UAT fleet was deleted on 2026-10-01: every UAT row below is now the single UAT host (one EC2 instance running the backend, its workers and scheduler, the CRM middleware and the PDF service as containers), deployed by each repo’s uat-image.yml / uat-host-deploy.yml through portals-workflows/.github/workflows/uat-host.yml. CodeDeploy is Production only. The host’s runbook is portals-workflows/uat-host/README.md.

App	Source repo (GitHub)	UAT · uat	Production · main
Backend	Orascom-Portals-Backend	UAT host container, uat-image.yml	Orascom-Prod-app / orascom-backend-deploy-group blue/green
Backend workers (queues/schedule)	(same repo, same revision)	UAT host containers (worker, worker-pdf, scheduler)	ODH-Prod-Supervisor / prod in-place
CRM (crms-middleware)	Orascom-Portals-CRM	UAT host container, uat-image.yml	Orascom-Prod-app / orascom-crm-deploy-group blue/green
PDF service	Orascom-Portals-pdf-service	UAT host container, pdf-deploy → uat-host-deploy.yml	ODH-Pdf / prod in-place container swap
Frontend (4 React apps)	Orascom-Portals-Frontend	AWS Amplify app dojdzrb1syifv	AWS Amplify app dndjdmne9kvk6
This docs site	orascom-portals-docs	—	Cloudflare Pages

• The CRM has no workers — its queue runs synchronously (QUEUE=sync).
• Amplify has one branch per app (uat and main respectively) with branch-level autoBuild enabled. It is not on the CodeDeploy rail.
• crm-mock deploys nowhere. See the CRM mock.

What gates a pull request

Checking branch protection the obvious way returns a false clean

GET /repos/<repo>/branches/<branch>/protection returns 404 “Branch not protected” on every branch of every repo here. That does not mean the branches are unprotected — the gating is a repository ruleset, which that endpoint cannot see. Query /rulesets instead.

All four application repos carry an active ruleset named no prod push on refs/heads/main and refs/heads/uat, with zero bypass actors: no force-push, no branch deletion, a pull request is required, and one approval is required, plus one required status check.

Repo	Required check	What runs but does not block
Backend	tests (PHPUnit)	quality (php-cs-fixer, PHPStan and composer audit are advisory; shellcheck, php -l and the LOCK_NAME assertion can fail it) and semgrep
CRM	tests (PHPUnit)	same
PDF service	tests (vitest + build)	—
Frontend	checks	within checks, lint and build block; the test target runs with continue-on-error

Two things worth knowing before you rely on a green tick. First, only the one required check per repo can stop a merge — but that is because the other jobs are not required checks, not because everything in them is advisory. Inside quality, php-cs-fixer, PHPStan and composer audit are suffixed || true, while shellcheck (over the deploy hooks), php -l and the MigrateLocked LOCK_NAME assertion can genuinely fail the job — so a red quality is worth reading, not dismissing. That assertion is the guard that stops a CRM deploy cross-blocking the backend on the shared database. Second, strict_required_status_checks_policy is false, so a PR does not have to be up to date with the base branch before merging.

This docs site has no ruleset

orascom-portals-docs has no branch ruleset at all — a direct push to main publishes to the live documentation site with no review and no check. Work on a branch and open a PR.

Automated traffic shift, and the limits of the safety net

All three blue/green deployment groups are configured deploymentReadyOption = CONTINUE_DEPLOYMENT with waitTimeInMinutes = 0: CodeDeploy stands up the new “green” fleet (greenFleetProvisioningOption = COPY_AUTO_SCALING_GROUP), health-checks it behind the load balancer, and shifts traffic automatically — there is no human step. The old “blue” fleet is kept for terminationWaitTimeInMinutes = 10 before termination, and that ten-minute window is the rollback window. The manual approval gate used on the first cutover deploys was removed in July and nothing has been re-added.

Alarm rollback is production-only, and it fails open

The safety net is weaker than “CloudWatch alarms roll back a bad deploy” suggests, in two specific ways:

• UAT has no alarm rollback at all. The UAT backend deployment group sets alarmConfiguration.enabled = false.
• In production the gate fails open. Both production groups set ignorePollAlarmFailure = true, so if CloudWatch cannot be polled the deployment proceeds rather than stopping.

Users are still protected on a deploy failure — but “rollback” is the wrong word for what happens. If a lifecycle hook errors or health checks do not pass before the traffic shift, blue never stopped serving, so CodeDeploy records that a rollback was not required and takes no action. The user-facing outcome is fine; the side effect is that the green fleet is left running with no owner. See Known defects for that, and Infrastructure for the alarms themselves, including one that currently flaps.

Blue/green and rollback

flowchart LR
REV[New revision<br/>from S3]:::store --> GREEN[Green fleet<br/>new EC2 ASG]:::compute
GREEN -->|health-check + migrate| READY{Green healthy<br/>behind ALB?}:::deploy
READY -->|auto-continue| SHIFT[ALB shifts traffic<br/>blue → green]:::ok
READY -->|hook error / health-check fail| RB[No traffic shift<br/>blue keeps serving<br/>green fleet ORPHANED]:::bad
SHIFT --> TERM[Blue terminated<br/>after 10-min window]:::compute
classDef store fill:#eceff1,stroke:#5B6B7A,color:#0b1b2b;
classDef compute fill:#f7efdc,stroke:#B7791F,color:#0b1b2b;
classDef deploy fill:#dff0f0,stroke:#0C7C84,color:#0b1b2b;
classDef ok fill:#dff0e6,stroke:#2f855a,color:#0b1b2b;
classDef bad fill:#f7e0e0,stroke:#c53030,color:#0b1b2b;

Blue/green deploy with automated traffic shift (production)

• Blue/green runs for the backend and the CRM in production (UAT stopped rehearsing it on 2026-09-22: the host does a stop–migrate–start, and the migration-safety CI check guards expand/contract on every PR instead). CodeDeploy stands up a new Auto Scaling group running the new revision alongside the live one, shifts traffic at the ALB, and keeps blue briefly so a rollback is a traffic flip rather than a rebuild.
• In-place runs everywhere else in production — the supervisor box and the PDF container.
• ⚠️ A deploy that fails before the traffic shift is safe for users but leaves a fleet running. Blue never stops serving, so CodeDeploy records that a rollback was “not required” — and nothing then owns the green Auto Scaling group. It keeps running until someone deletes it. See Known defects for the detection rule, including why the obvious version of it would delete production.
• To roll back, redeploy the previous revision. Because the revision carries the code, that genuinely restores the previous build. This is the rollback lever; there is no pipeline to re-run.
• Database migrations run inside a deploy hook under a fleet-safe lock. The lock serialises, it does not de-duplicate: whichever box gets there first migrates, and the others wait and then re-run migrate, which is a no-op. Seeding is opt-in and off by default. Migrations are expand/contract by discipline so blue and green can run against the same schema during the shift — nothing automated enforces that, so a reviewer has to check the pending diff on every production deploy.
• Two hooks guard correctness, and they are not equivalent: the CRM’s validate_passport runs at BeforeAllowTraffic and is a genuine traffic gate, while the backend’s validate_pdf_token runs at AfterInstall — it fails the deployment early if the backend↔PDF shared token has drifted, but it is not a traffic gate.

How long a deploy takes

Measured across successful runs since the 2026-08-13 cutover:

Rail	Typical duration
In-place (CRM on UAT)	≈ 1 minute
Blue/green	21–61 minutes, median ≈ 25
PDF container swap	6–8 minutes
Frontend (Amplify)	4–6 minutes
This docs site	≈ 1 minute

The blue/green figure is the traffic shift, not the build — it is the same artifact for the whole window. The rail is safe by default, and the safety is what costs the time.

Where config comes from

Secrets → .env on the box, at deploy time

There are no committed .env files in the CodeDeploy rail. Each app’s runtime configuration lives in AWS Secrets Manager under portals/<env>/<app> (for example portals/prod/backend, portals/prod/crms). A render_env deploy hook reads the secret and writes the .env on the box before the app starts. Rotating a secret is a Secrets-Manager edit plus a redeploy — nothing is baked into an image or a repo.

The frontend is the exception, and it matters: its environment variables live in AWS Amplify, not in Secrets Manager. That is a second configuration plane, and it is where the build-time secrets described in Environment variables come from.

Deploy notifications

Every deployment posts a status card (✅ / ❌) to the Portals deployments feed in Microsoft Teams — app, environment, status, commit, and a link to the build. (The workflows post to an opaque webhook URL and name no channel, so the destination is described here by function rather than by channel name.)

⚠️ A backend deploy posts two cards, not one. The notify step lives inside the shared reusable workflow, and the backend invokes that workflow twice — once for web, once for workers — so each stage posts its own result, and the two can disagree. Seeing two cards does not mean something ran twice. Tracked as 869enhrxj. Every other app posts one.

Deploy	How the card fires	Where the webhook lives
Backend & CRM	a notify step in the deploy workflow	Secrets Manager portals/shared/notify-webhook
Frontend	an EventBridge rule on Amplify build events → the portals-amplify-teams-notify Lambda	Secrets Manager portals/shared/notify-webhook
PDF service	a notify step in its own workflow	GitHub repo secret NOTIFY_WEBHOOK
This docs site	a workflow that fires when Cloudflare’s build check completes	GitHub repo secret NOTIFY_WEBHOOK

The two GitHub-secret cases are deliberate, not drift: reading the webhook from a repo secret means the card still fires when an early step fails before AWS credentials are available. The practical consequence is that rotating the webhook means touching three places: the Secrets Manager entry, and the NOTIFY_WEBHOOK repo secret in each of the two repositories that carry their own.

What replaced what

The old rail, and why the change mattered

Before July 2026 there was no single pipeline — there were two paths. Most services used a GitLab runner installed on the target VM, whose entire job was systemctl restart pull-<app>.service; CodePipeline was never involved. Production backend and CRM used a manual GitLab job that started an AWS CodePipeline, which drove CodeDeploy, whose appspec.yml contained exactly one hook that also restarted a pull service. Both paths ended the same way: the box ran git pull on the branch tip and fetched its own code.

That is why the current rail is a real change rather than a tooling swap. The old revision carried no files, so there was nothing to roll back to; redeploying a previous revision simply re-pulled whatever the branch tip was at that moment.

The old production CodePipelines were kept — disconnected but not deleted — as a manual rollback path for about four weeks after the cutover, which is what earlier versions of this page described. They were deleted on 2026-08-16, along with their artifact bucket. There is no pipeline to fall back to; the rollback lever is a CodeDeploy redeploy of the previous revision.

`main` and `uat` carry different copies of the deploy workflow

GitHub runs a workflow from the ref you pushed, and the two branches have diverged. On the backend, uat’s deploy.yml is 52 lines and main’s is 76: main still triggers on develop, still maps a staging case to the deleted ODH-Staging-Backend application, and carries an extra api-tests job gated on env == 'staging'. The CRM differs too; the PDF service does not.

Production behaviour is unaffected — a main push takes the main) case, which is byte-identical to uat’s, so the matrix above holds. But two consequences are worth knowing: a main deploy run shows a job that does not exist in the uat workflow anyone reads, and — the real risk — promoting uat to main does not carry workflow changes unless someone merges them. A fix made to the deploy rail on uat silently does not apply to production until it is promoted.

GitLab has no runtime role. The one repository never migrated is crm-mock, which deploys nowhere at all.

Operational runbooks

The blow-by-blow operator procedures (rebake steps, exact deployment-group and instance identifiers, rollback levers) live in the repo, not on this site. They are in handoff/handoff-vault/ — note the path, there is no top-level docs/ directory in this repository:

• runbook-prod-cutover.md — the production cutover procedure (backend + CRM).
• runbook-uat-backend-bluegreen.md — the UAT backend blue/green deploy. Historical since 2026-10-01: the old UAT fleet it describes was deleted; UAT is the host, see portals-workflows/uat-host/README.md.
• deploy-logic-decoded.md — the old on-box pull logic, decoded.
• runbook-staging-deploy.md — historical only. Staging was decommissioned on 2026-08-16.

Where to go next

EnvironmentsWhich branch reaches which environment, the URLs, and what each environment is for.

InfrastructureThe AWS estate underneath — compute, data, secrets, alarms, and what is retired.

ArchitectureThe whole system on one page — components, CRM shape, and the production topology.