How deployment works

The platform moved off Robusta’s old deploy system onto a new, self-contained rail: you push to GitHub, GitHub Actions builds a complete deploy revision, and AWS CodeDeploy rolls it out — blue/green in production and UAT, in-place in staging. The backend and CRM production apps cut over to this rail on 2026-07-20 (blue/green, live and verified). This page is the single “how it ships” picture; the granular, step-by-step operator procedures live in the runbooks under docs/handoff-vault/ (see Operational runbooks below).

One-sentence version

Push to a branch → GitHub Actions builds a self-contained revision → CodeDeploy deploys it; production shifts traffic automatically once the new fleet passes its health checks, a failed deploy or a CloudWatch alarm rolls back automatically, and every deploy posts a status card to Teams.

The new rail, end-to-end

flowchart LR
DEV(Developer):::person -->|push / merge PR| GH[GitHub repo<br/>OrascomDevelopment]:::src
GH -->|branch → environment| GHA[GitHub Actions<br/>reusable laravel-deploy.yml]:::ci
GHA -->|build self-contained<br/>revision .zip| S3[(S3<br/>orascom-deploy-bucket)]:::store
GHA -->|create deployment| CD[AWS CodeDeploy]:::deploy
S3 --> CD
CD -->|blue/green prod+uat<br/>in-place staging| ASG[EC2 Auto Scaling<br/>Graviton t4g · behind ALB]:::compute
classDef person fill:#e7eef6,stroke:#2563A8,color:#0b1b2b;
classDef src fill:#e7eef6,stroke:#2563A8,color:#0b1b2b;
classDef ci fill:#efe7f7,stroke:#7A4FB5,color:#0b1b2b;
classDef deploy fill:#dff0f0,stroke:#0C7C84,color:#0b1b2b;
classDef store fill:#eceff1,stroke:#5B6B7A,color:#0b1b2b;
classDef compute fill:#f7efdc,stroke:#B7791F,color:#0b1b2b;

Deploy rail — GitHub Actions → CodeDeploy (backend & CRM)

1. Push / merge a pull request to an environment branch (develop → staging, uat → UAT, main → production) in the GitHub org OrascomDevelopment.
2. GitHub Actions builds the revision. A reusable workflow (portals-workflows/laravel-deploy.yml) installs dependencies and packages a self-contained deploy artifact — the built code plus the CodeDeploy appspec.yml and lifecycle hooks — then uploads it to S3 (orascom-deploy-bucket). This is the key change from the old rail: the revision carries the code, so a box never has to git pull to get it.
3. CodeDeploy deploys the revision. On the box, lifecycle hooks render the environment file from AWS Secrets Manager (see Where config comes from), run database migrations under a fleet-safe lock, and clear caches — then health-check before any traffic moves.

What replaced what

The old rail was GitLab CI + AWS CodePipeline → CodeDeploy, where CodeDeploy only told the box to git pull the branch tip and the real work lived in on-box pull-*.service systemd units. The new rail removes GitLab and CodePipeline from prod/UAT entirely: GitHub Actions builds a real, versioned artifact and CodeDeploy deploys that. The old CodePipelines are kept disconnected as a manual rollback lever, not deleted.

Per-app × environment

Each app has three environment branches. The rail and cutover status as of 2026-07-21:

App	Source repo (GitHub)	Staging · develop	UAT · uat	Production · main
Backend	Orascom-Portals-Backend	Actions → CodeDeploy in-place	Actions → CodeDeploy blue/green	Actions → CodeDeploy blue/green live 2026-07-20
Backend workers (queues/schedule)	(same repo)	in-place, supervisor	in-place, supervisor	ODH-Prod-Supervisor / prod in-place live 2026-07-20
CRM (crms-middleware)	Orascom-Portals-CRM	Actions → CodeDeploy in-place	Actions → CodeDeploy in-place	Actions → CodeDeploy blue/green live 2026-07-20
Frontend (4 React apps)	Orascom-Portals-Frontend	AWS Amplify autoBuild	AWS Amplify autoBuild	AWS Amplify autoBuild live
PDF service	Orascom-Portals-pdf-service	not on this rail yet	Actions → CodeDeploy in-place (supervisor)	Actions → CodeDeploy in-place live 2026-07-21

• Backend production = CodeDeploy app Orascom-Prod-app / group orascom-backend-deploy-group (blue/green), with queue workers deployed in-place to ODH-Prod-Supervisor / prod.
• CRM production = Orascom-Prod-app / orascom-crm-deploy-group (blue/green). The CRM has no workers — its queue runs synchronously (QUEUE=sync).
• Frontend is AWS Amplify, sourced from GitHub Orascom-Portals-Frontend with autoBuild=true — one Amplify app for production (main) and one for the staging/UAT branches. Amplify builds and deploys on push; it is not on the CodeDeploy rail. (The one-time GitLab “Fresh copy” commits in July were the migration onto GitHub, not an ongoing GitLab dependency.)

Automated traffic shift + alarm rollback

Production shifts traffic automatically, guarded by alarms

Production and UAT deployments are fully automated (deploymentReadyOption = CONTINUE_DEPLOYMENT): CodeDeploy brings up the new “green” fleet, health-checks it behind the load balancer, and then shifts traffic automatically — no human step. The old “blue” fleet is kept for a short window (terminationWaitTimeInMinutes = 10) before it is terminated, so a bad shift can be reverted immediately.

The safety net is CloudWatch alarms, not a person: a 5xx spike or unhealthy-host alarm during a deploy triggers an automatic rollback (blue keeps serving), and any deploy failure rolls back too. The earlier first-cutover deploys paused at a manual approval gate (STOP_DEPLOYMENT + aws deploy continue-deployment); that gate was removed on 2026-07-21 once the rail had proven itself, so deploys are now push-and-forget with alarm-driven rollback.

Blue/green and rollback

flowchart LR
REV[New revision<br/>from S3]:::store --> GREEN[Green fleet<br/>new EC2 ASG]:::compute
GREEN -->|health-check + migrate| READY{Green healthy<br/>behind ALB?}:::deploy
READY -->|auto-continue| SHIFT[ALB shifts traffic<br/>blue → green]:::ok
READY -->|deploy fails / 5xx / unhealthy alarm| RB[Auto-rollback<br/>blue keeps serving]:::bad
SHIFT --> TERM[Blue terminated<br/>after 10-min window]:::compute
classDef store fill:#eceff1,stroke:#5B6B7A,color:#0b1b2b;
classDef compute fill:#f7efdc,stroke:#B7791F,color:#0b1b2b;
classDef deploy fill:#dff0f0,stroke:#0C7C84,color:#0b1b2b;
classDef ok fill:#dff0e6,stroke:#2f855a,color:#0b1b2b;
classDef bad fill:#f7e0e0,stroke:#c53030,color:#0b1b2b;

Blue/green deploy with automated traffic shift + alarm rollback (prod & UAT)

• Blue/green (production + UAT): CodeDeploy stands up a new Auto Scaling group (“green”) running the new revision alongside the live one (“blue”), shifts traffic at the ALB, and keeps blue briefly so rollback is a traffic flip, not a rebuild.
• In-place (staging): the revision is deployed onto the existing boxes. Faster, cheaper, and fine for a non-production environment.
• Auto-rollback on failure is enabled. If a deployment fails (a hook errors, health checks don’t pass), CodeDeploy automatically rolls back and blue keeps serving — no manual intervention needed.
• Database migrations run inside a deploy hook under a fleet-safe lock (only one box in the group runs them), and are migrate-only — additive/expand-contract by discipline, so blue and green can run against the same schema during the shift.

Where config comes from

Secrets → .env on the box, at deploy time

There are no committed .env files in the deploy rail. Each app’s runtime configuration lives in AWS Secrets Manager under portals/<env>/<app> (for example portals/prod/backend, portals/prod/crms). A render_env deploy hook reads the secret and writes the .env on the box before the app starts. Rotating a secret is a Secrets-Manager edit plus a redeploy — nothing is baked into an image or a repo.

Deploy notifications

Every deploy announces itself in Teams

Every deployment posts a status card (✅ / ❌) to the “Orascom Portals Deployments” Microsoft Teams channel — app, environment, status, commit, and a link to the build. The card fires for:

• Backend & CRM — a notify step in the GitHub Actions deploy workflow (success / failure / cancelled).
• PDF service — a notify step in its own GitHub Actions deploy workflow (build → ECR → CodeDeploy).
• Frontend (Amplify) — an EventBridge rule on Amplify build events → a small Lambda that posts the card.
• This docs site (Cloudflare Pages) — a GitHub Actions workflow that fires when Cloudflare’s build check completes.

All cards share one format and one Teams webhook (kept in AWS Secrets Manager as portals/shared/notify-webhook).

The mixed state, told honestly

The cutover is substantially done. As of 2026-07-21 most of the transition is complete; here is the honest status of each moving part:

• PDF service — on its own rail (uat + prod). live 2026-07-21 The PDF service now auto-deploys via its own GitHub Actions workflow in Orascom-Portals-pdf-service: a push to uat or main runs OIDC → a docker buildx --platform linux/arm64 cross-build (QEMU) → push to Amazon ECR (backend-pdf-service) → a tiny CodeDeploy revision → CodeDeploy app ODH-Pdf, in-place on the supervisor box. The deploy hook pdf_run.sh is fail-safe, build-before-swap: it pulls the image, renders the env from Secrets Manager, asserts the pdf API_TOKEN equals the backend PDF_SERVICE_TOKEN, runs the new container on a temp port, runs a real Chromium render smoke test, and only then swaps it onto :3001 with --restart=always — so a bad image leaves the old container running (empirically proven in UAT), and --restart=always means the container now survives a reboot (it was --restart=no before). It touches only the Docker container, never the supervisord pdf-service-queue worker, and posts a Teams card. Still honest: staging/develop PDF is not on this rail yet — it runs on the combined ODH-Staging box; only uat + main are wired. Remaining follow-up: the pdf↔backend shared-token rotation (F-F) — rotate portals/<env>/pdf.API_TOKEN and backend.PDF_SERVICE_TOKEN together at a redeploy.
• GitLab is fully retired. done 2026-07-21 Production and UAT never touched GitLab; on 2026-07-21 the last holdout — the staging box’s crm-mock, pdf-service and crms-middleware git remotes — were re-pointed to GitHub (OrascomDevelopment, read-only deploy keys). All Portals repos now live in GitHub and GitLab has zero runtime role.
• Prod gate removed — deploys are automated. done 2026-07-21 The manual continue-deployment gate was removed on all production groups (and the UAT backend group); traffic now shifts automatically once green is healthy, with CloudWatch-alarm and deploy-failure auto-rollback as the safety net.
• Old CodePipelines kept as rollback. The previous prod pipelines are disconnected (webhooks deregistered, deploy transitions disabled) but not deleted, so they remain a manual rollback path until the new rail has proven itself over several deploys.

Cutover status in one line

Backend + CRM production are on the new GitHub-Actions → CodeDeploy blue/green rail (done 2026-07-20, verified live). Backend/CRM staging + UAT are on the rail too. Frontend runs on Amplify. The PDF service is now on its own rail (uat + prod, live 2026-07-21, in-place with a fail-safe swap). The only PDF piece still pending is staging PDF, plus the pdf↔backend shared-token rotation follow-up.

The unchanged infrastructure underneath

The deploy rail changed; the compute, data, and network topology did not. It still runs in AWS account 907815021625, region eu-central-1 (Frankfurt):

• Compute: EC2 Auto Scaling groups (Graviton t4g) for backend and CRM, behind a shared Application Load Balancer (orascom-prod-ecommerce-lb) with AWS WAF on the prod ALB.
• Data: RDS MySQL, Multi-AZ (orascom-prod-rds); a dedicated Redis EC2 instance for cache/queues.
• Storage / CDN: S3 for media, served via CloudFront. Deploy revisions live in a separate bucket, orascom-deploy-bucket.
• Config: AWS Secrets Manager (portals/<env>/<app>), rendered to .env on-box at deploy time.

See the Architecture page for the full component and production-topology diagrams.

Operational runbooks

The blow-by-blow operator procedures (rebake steps, gate validation commands, exact deployment-group and instance identifiers, rollback levers) live in the repo, not on this site. They are in docs/handoff-vault/:

• runbook-prod-cutover.md — the production cutover procedure (backend + CRM).
• runbook-uat-backend-bluegreen.md — the UAT backend blue/green deploy.
• runbook-staging-deploy.md — the staging (in-place) deploy.
• deploy-logic-decoded.md — the old on-box pull logic, decoded (the behaviour the new rail replicates and improves on).

Where to go next

ArchitectureThe whole system on one page — components, CRM shape, and the AWS production topology.

KT trackerKnown issues and the remaining environment unknowns.