Infrastructure

This is the platform infrastructure — the AWS estate and the deploy control plane that runs it. It complements How deployment works (how a change ships), Environments (which branch reaches what), and Observability & infrastructure (the application integrations — Sentry, GA4, S3, Redis).

Everything runs in one AWS account — 907815021625, region eu-central-1 (Frankfurt). The account is standalone: it left the outgoing vendor’s AWS Organization, and billing, authentication and deployment no longer depend on them.

As-built as of 2026-09-06, from a read-only sweep of the account.

The estate on one diagram

flowchart TB
subgraph GH["GitHub · OrascomDevelopment"]
  REPO[App repos]:::src
  GHA[GitHub Actions]:::ci
end
subgraph AWS["AWS · account 907815021625 · eu-central-1 · standalone"]
  OIDC{{OIDC · gh-deploy-prod / gh-deploy-uat<br/>no long-lived keys}}:::sec
  CD[CodeDeploy · 3 applications, production only<br/>Orascom-Prod-app · ODH-Prod-Supervisor · ODH-Pdf]:::deploy
  UATH[UAT host · one EC2 t4g.medium<br/>backend · workers · scheduler · CRM · PDF · MySQL · Redis as containers<br/>stack odh-uat-host]:::compute
  S3REV[(S3 orascom-deploy-bucket<br/>revisions)]:::store
  ECR[(ECR backend-pdf-service)]:::store
  SM[Secrets Manager<br/>portals/&lt;env&gt;/&lt;app&gt; · CMK]:::sec
  subgraph SHARED["Shared by prod AND UAT — same subnets, same ALB"]
    ALB[Shared ALB + WAF<br/>orascom-prod-ecommerce-lb]:::compute
    BEASG[Backend ASG · t4g<br/>blue/green]:::compute
    CRMASG[CRM ASG · t4g]:::compute
    SUP[Supervisor boxes<br/>workers · scheduler · PDF container]:::compute
  end
  RDSP[(RDS orascom-prod-rds<br/>MySQL 8.4.11 · db.t3.small · Multi-AZ)]:::data
  REDIS[(Redis EC2)]:::data
  MEDIA[(S3 media)]:::store
  OBS[CloudWatch · CloudTrail · GuardDuty<br/>Access Analyzer · Teams notify Lambda]:::sec
end
AMP[AWS Amplify · 2 apps<br/>outside the VPC · not behind the ALB or WAF]:::compute
CF[CloudFront ×2<br/>prod media · UAT media<br/>no WAF on either]:::store
REPO --> GHA -->|assume via OIDC| OIDC --> CD
GHA -->|build revision| S3REV --> CD
GHA -->|buildx arm64 push| ECR --> SUP
REPO -->|autoBuild| AMP
CD -->|blue/green| BEASG & CRMASG
CD -->|in-place| SUP
ALB --> BEASG & CRMASG
BEASG & CRMASG --> RDSP
SUP --> RDSP
BEASG --> REDIS & MEDIA
CF --> MEDIA
CD -.render env.-> SM
BEASG & CRMASG & SUP -.-> OBS
classDef src fill:#e7eef6,stroke:#2563A8,color:#0b1b2b;
classDef ci fill:#efe7f7,stroke:#7A4FB5,color:#0b1b2b;
classDef deploy fill:#dff0f0,stroke:#0C7C84,color:#0b1b2b;
classDef compute fill:#f7efdc,stroke:#B7791F,color:#0b1b2b;
classDef data fill:#eceff1,stroke:#5B6B7A,color:#0b1b2b;
classDef store fill:#eceff1,stroke:#5B6B7A,color:#0b1b2b;
classDef sec fill:#f6e7dc,stroke:#C4622D,color:#0b1b2b;

Orascom Portals — AWS estate + deploy control plane (as-built)

Compute and network

• Backend & CRM run on EC2 Auto Scaling groups (AWS Graviton t4g) behind one shared Application Load Balancer (orascom-prod-ecommerce-lb) with AWS WAF attached.
• The supervisor box (ODH-Supervisor, production) is a stable pet, not an ASG. It runs, under supervisord: the backend queue workers, the Laravel scheduler (its only home, so scheduled commands cannot duplicate across a scaled-out fleet), the pdf-service-queue worker, and the PDF service Docker container. Deploys to it are in-place.
• UAT is one EC2 host (ODH-UAT-Host, t4g.medium, CloudFormation stack odh-uat-host, since 2026-09-22) running the backend, its workers and scheduler, the CRM middleware, the PDF service, MySQL 8.4 and Redis as Docker Compose containers on an encrypted data volume, behind the shared ALB through the ODH-UAT-Host-Backend and ODH-UAT-Host-Middleware target groups. It is switched off by an EventBridge schedule outside 08:00–20:00 Cairo, every day (UAT answers 503 then), backs its databases up nightly at 17:17 Cairo to S3 odh-uat-host/backups/, and is operated through the SSM documents UAT-Deploy, UAT-Data and UAT-Logs. The old UAT fleet (ODH-UAT-Backend-ASG, ODH-UAT-Middleware, ODH-UAT-Supervisor, ODH-UAT-Redis and odh-uat-rds) was deleted on 2026-10-01. Runbook: portals-workflows/uat-host/README.md.
• Frontend = AWS Amplify — two apps, one per environment. Amplify is AWS-managed serverless hosting: it is not inside the VPC, and it is not behind the ALB or the WAF. If the frontend is down, the load balancer and target groups are not the place to look.

UAT and production share the same subnets and the same load balancer

There is no network boundary between UAT and production. One subnet holds the production CRM, Redis and supervisor and their UAT counterparts, and the production and UAT backend Auto Scaling groups declare an identical set of subnets. The two environments are separated only by ALB host rules and security groups — there is no separate VPC, no separate subnet, and no separate load balancer.

Two consequences worth holding on to: a security-group or ALB-rule change has a blast radius that spans both environments, and “we will try it on UAT first” is a weaker safety guarantee here than it sounds.

Security groups are clean: only orascom-prod-alb-sg has a 0.0.0.0/0 rule, and only on ports 80/443. No SSH, database or Redis port is exposed to the internet.

Data

	orascom-prod-rds	UAT (on the host)
Engine	MySQL 8.4.11	MySQL 8.4 container
Class	db.t3.small (Intel)	part of the t4g.medium host
Multi-AZ	Yes	No
Storage	80 GB gp2	the host’s encrypted data volume

There are two databases, not one. Production RDS is the shared relational store for the production backend and CRM. UAT’s MySQL runs as a container on the UAT host since 2026-09-22 (odh-uat-rds, its former RDS counterpart, was deleted on 2026-10-01); it is backed up nightly to S3 and restore-drilled monthly. Production automated backups and point-in-time recovery are on. Production migrations run once per deploy under a fleet-safe GET_LOCK mutex on the relevant instance.

The 8.0 → 8.4.11 upgrade completed on 2026-08-23, evidenced by pre-upgrade snapshots captured at 8.0.45, and was rehearsed on a throwaway instance the same day before production was touched. It ended a paid Extended Support enrolment that MySQL 8.0 had triggered automatically when it left standard support.

Note that compute is Graviton but the production database is not — it is Intel db.t3.small. Only UAT is on t4g.

A dedicated Redis EC2 instance per environment serves cache / queues / sessions where an environment routes them to Redis; the app also supports file and database drivers (see Observability & infrastructure).

Storage, CDN and registry

• S3 (media) → CloudFront — two distributions, one for production media (media.my.orascomdh.com) and one for UAT (media.odh-uat.my.orascomdh.com).
• Amazon ECR — backend-pdf-service — the private registry for the PDF service’s arm64 image. GitHub Actions cross-builds the image (buildx --platform linux/arm64) and pushes it here; the supervisor box pulls it during a PDF deploy. Its lifecycle policy expires untagged images — but every image carries a SHA tag, so the rule can never actually fire.
• S3 — orascom-deploy-bucket — holds the CodeDeploy revisions, env-prefixed (uat/, prod/). A staging/ prefix also survives with ~710 MB of dead revisions; no lifecycle rule removes it.

The production media CDN is configured more permissively than UAT's

The two distributions are not symmetric, and production is the weaker of the two:

	Production	UAT
Viewer protocol	allow-all — plain HTTP is not redirected to HTTPS	redirect-to-https
Allowed methods	GET/HEAD/OPTIONS plus PUT/POST/PATCH/DELETE, with a bucket policy granting CloudFront s3:PutObject	GET/HEAD/OPTIONS only
WAF	none	none

The web ACL is attached to the ALB only, so neither distribution is behind the WAF. Anything on this page describing WAF coverage means the ALB, not the CDN.

Config and secrets — two planes, not one

No committed .env files on the CodeDeploy rail

Each backend/CRM/PDF app’s runtime configuration lives in AWS Secrets Manager under portals/<env>/<app> (portals/prod/backend, portals/prod/crms, portals/prod/pdf, …) plus portals/<env>/files (Passport keys, Firebase JSON, GA PEM) and portals/shared/notify-webhook. A render_env deploy hook reads the secret and writes the .env (and secret files) onto the box before the app starts, failing the deploy on any missing key. Secrets are encrypted with a customer-managed KMS key; instance roles get scoped kms:Decrypt and read access on their own environment’s secrets only.

What that separation does and does not guarantee

One customer-managed key encrypts both environments — it is named portals-prod, but UAT secrets use it too. So the key is not the separation boundary; the IAM resource ARNs are (portals/prod/* versus portals/uat/*). If those were ever widened to portals/*, the shared key would provide no protection and this page would still read as though it did. Note also that portals/shared/* is deliberately readable from both environments, so “its own environment’s secrets” is true of the portals/<env>/ prefix rather than of the whole namespace.

The frontend does not use this plane. Its environment variables — 53 of them — live in AWS Amplify. That is a second configuration plane with different access control and a different rotation path, and it is where the build-time secrets described in Environment variables originate.

Deploy control plane

• GitHub Actions builds every deploy; AWS CodeDeploy rolls it out. No long-lived AWS keys exist in GitHub — Actions assumes an OIDC role (gh-deploy-prod / gh-deploy-uat), each trust-bound to the exact repo:…:environment:<env> claim of the deploying repository.

• CodeDeploy applications — there are exactly three, all production (the UAT ones, ODH-UAT, ODH-UAT-CRM and the uat group of ODH-Pdf, were deleted on 2026-10-01; UAT deploys to the host without CodeDeploy):

  App	Groups	Style	Runs
  Orascom-Prod-app	orascom-backend-deploy-group, orascom-crm-deploy-group	blue/green	prod backend + CRM
  ODH-Prod-Supervisor	prod	in-place	prod queue workers / scheduler
  ODH-Pdf	prod	in-place	PDF service container swap

• The PDF service has its own rail: push → buildx arm64 → ECR → ODH-Pdf in-place → a fail-safe container swap (pull the image, render the env, assert the shared token matches, run the new container on a temp port, run a real Chromium render smoke test, and only then swap it onto the live port with --restart=always). A bad image leaves the old container running.

Full detail on How deployment works.

Observability and audit

Seven CloudWatch alarms exist. Four cover the two production target groups (prod-backend-ALB-target-5xx, prod-backend-ALB-unhealthy-hosts, prod-crm-ALB-target-5xx, prod-crm-ALB-unhealthy-hosts); two cover backend CPU and memory; the seventh is a legacy low-CPU alarm. All seven notify a single SNS topic whose name is misleading — it is named after one of the alarms rather than after its purpose.

Two alarm problems worth knowing before you trust the dashboard

A legacy low-CPU alarm is in a permanent alarm state, and its action is not a notification. Oraacom-Backend-CPU-Utilization-lower-20 (note the misspelling in the name) fires below 20% CPU and has been in ALARM continuously since 2026-09-03. Its alarm action is an Auto Scaling scale-in policy, not an SNS topic — so it is not paging anyone, it is continuously asking production to scale in, and only the group’s minimum size is holding capacity. It cannot be fixed by deleting it: CodeDeploy re-creates the scaling policies and their alarms on every blue/green deployment, so the fix has to be to the configuration it is re-created from.

A deploy-gating alarm flaps. prod-backend-ALB-unhealthy-hosts cycles into and out of alarm within 60–120 seconds, several times a week. It gates deployment rollback on both production groups, so a deploy that starts mid-flap can roll back spuriously. A contributing factor is that the production backend target group health-checks /admin/login — a full application route rather than a dedicated liveness endpoint — where UAT uses /up, Laravel’s purpose-built health route. A route that renders the admin login page exercises more of the stack than a liveness check needs to.

Coverage gaps. There are no alarms on RDS, none on the CRM’s CPU or memory, none on the supervisor boxes, none on the NAT gateway, none on any UAT resource, and no billing alarm. Production RDS has no Performance Insights. Given that the single largest cost item in this account appeared automatically on a support calendar, a billing alarm is a notable absence.

Target-group health checks differ between environments, and not in the direction you would expect:

Target group	Path	Interval	Timeout	Unhealthy threshold
orascom-prod-backend-target	/up	20 s	10 s	5
orascom-prod-crm-target	/up	20 s	10 s	5
ODH-UAT-Host-Backend (the host, since 2026-09-22)	/up	20 s	5 s	2
ODH-UAT-Host-Middleware (the host)	/	30 s	5 s	2

(Values read from the target groups on 2026-10-01; the old ODH-UAT-Backend and ODH-UAT-Middleware groups pointed at the deleted fleet.)

UAT marks the host unhealthy in 40 s (backend) or 60 s (middleware), against 100 s for either production group — and it uses a 5-second health-check timeout where production allows 10, so a response taking 5–10 seconds counts as a failure in UAT and passes in production. With a single host there is nothing to eject to: an unhealthy target means UAT answers 503 until the host recovers, which is the deliberate trade of one machine against a fleet. slow_start is 0 on all four, so a target takes full traffic with no ramp.

CloudTrail (multi-region, own encrypted bucket), GuardDuty, and IAM Access Analyzer provide the account-level audit and threat-detection floor.

DNS and certificates

• Root orascomdh.com is on Orascom IT’s Cloudflare. my.orascomdh.com is served from a Route 53 zone in a different Orascom AWS account — this account holds zero hosted zones.
• ACM provides the TLS certificates for the *.my.orascomdh.com family (ALB and CloudFront). The production media certificate media.my.orascomdh.com expires 2026-10-29 and auto-renews via DNS validation; the validation record was confirmed passing on 2026-09-06. No action is currently required — but this is worth re-checking rather than assuming, because the renewal is automatic only for as long as that record stays resolvable, and the zone is not in this account.
• Certificate hygiene: the account also holds several certificates that timed out validation and two that have expired outright, all unused.

What is retired

• Staging — decommissioned 2026-08-16. The instance was terminated, the CodeDeploy applications deleted, and the Amplify branch removed. See Environments.
• AWS CodePipeline — the old production pipelines were kept, disconnected, as a manual rollback path for about four weeks after the July cutover, then deleted on 2026-08-16 along with their artifact bucket. Only orphaned IAM roles remain. There is no pipeline to fall back to.
• GitLab — no runtime role. The one repository never migrated is crm-mock, whose .gitlab-ci.yml is inert (no GitLab, no runner, no staging box).
• Vendor-hosted tooling (SonarQube, Dependency-Track, self-hosted Sentry, Grafana, Mattermost) — replaced by GitHub-native checks, CloudWatch, and Orascom’s own Sentry.

Debris still present and worth clearing rather than documenting forever: a retained Elastic IP from the staging box (deliberately kept — stale DNS records still point at it, some of them in a zone the outgoing vendor controls); an orphan Amplify domain association with no branch behind it; the IAM roles odh_staging_role and gh-deploy-staging — the latter still carrying a live GitHub OIDC trust; an empty staging log group; the staging/ revision prefix; staging entries in the deploy scripts’ group map; and portals/staging/* secrets scheduled for deletion on 2026-09-21.

Where to go next

EnvironmentsWhich branch reaches which environment, the URLs, and what each is for.

How deployment worksThe GitHub Actions → CodeDeploy rail end-to-end, the PR gates, and the rollback story.

ArchitectureThe application components, the CRM integration shape, and the tech stack.