Known defects & findings register

What this page is

While writing the rest of this documentation, every claim was traced to current source (backend backend @ 8434e475, 2026-07). That process surfaced a stack of real defects and oddities. This page collects them — source-cited, grouped by kind — so the engineering team has a single actionable list. It is not a description of intended behaviour; it is a bug/finding register. Severities are a rough triage, not a formal risk assessment.

Infrastructure & cost exposure

🔴 A five-day bot flood in July 2026, and no rate limiting today

Between 2026-07-09 and 07-13 the platform absorbed roughly 384 million WAF requests against a ~30,000/day baseline — about 2,500× normal. It made AWS WAF the single largest line on that month’s bill (~$616, against a June WAF of ~$22 measured from its own request volume). (No share-of-the-month figure is quoted: Cost Explorer holds nothing before 2026-08-12, so no July denominator exists.)

The platform itself was never affected. The load balancer’s :443 listener routes on host header, the flood matched no rule, and the listener’s default fixed-response 503 answered it without ever consulting a target group: ELB 503s went from 4,116 to 87,364,520 while requests actually reaching targets stayed flat (22,413 → 21,805) and target 5XX errors stayed at zero. This was a cost incident, not an availability incident — worth stating, because “384 million requests” reads like an outage.

And it was self-inflicted, not an attack. A UAT hostname migration at 2026-07-09T00:37:41Z removed the retired hosts from the production ALB listener rules. That listener’s default action is fixed-response 503, so every client still calling the old names received an ALB-generated 503 that never reached a target — and retried. Onset was under three minutes, and the traffic ramped over 45 minutes then plateaued, which is retry backoff rather than an attack arriving at its rate.

The operational lesson, and it generalises: retiring a hostname on an ALB whose default action is fixed-response 503 silently converts every stale client into billable, inspected traffic — and nothing in this estate watches for it.

It went undetected for 71 hours. The flood began 2026-07-09T00:40Z; the first human action against it was 2026-07-12T09:43:57Z, and WAF blocking only took hold at 10:03:18Z — ~81 hours after onset. The HTTPCode_ELB_503_Count metric went ~21,000× and triggered nothing.

Five things about it are still true today:

• There is no alarm on HTTPCode_ELB_503_Count — the one signal that would have caught this in minutes rather than days. It costs nothing.
• A retired hostname is not kept on a 301/410 rule through a cutover, so it falls to the listener default. That is the cause, and preventing it is free.
• There is no rate-based rule on the Web ACL. RateBasedStatement count: zero. Worth ~$1/month as a blast-radius cap on any volumetric event, whatever its origin.
• There is no AWS Budget and no billing alarm. Cost Anomaly Detection exists and would probably fire for an event this size, but not for a smaller recurring step.
• Every mitigating change was made by an account that no longer has access (the outgoing vendor’s). Nobody currently on the account has operated this failure mode.

No request-level forensics exist and none can be reconstructed — WAF logging was enabled on 2026-07-22, ten days after the event.

Why this matters for the Bot Control decision

AWS WAF Bot Control looks like poor value in steady state — it blocks about 0.04% of requests, while a free rule ahead of it blocks around 90%. During the flood it blocked 95 million requests, more than every other rule group combined. It is simultaneously the most expensive item under load and the only volumetric defence present. If it is ever removed, a rate-based rule should be added first.

🔴 A failed or stopped blue/green deploy orphans its green fleet, and nothing cleans it up

Green-fleet teardown is wired only to success. All three blue/green deployment groups set terminateBlueInstancesOnDeploymentSuccess and provision green with COPY_AUTO_SCALING_GROUP. There is no equivalent on any other terminal state, so a deployment that ends before the traffic shift leaves a running Auto Scaling group behind with no owner.

Auto-rollback does not cover this, and CodeDeploy says so itself. The failed production deployment’s rollbackInfo.rollbackMessage reads:

“Automatic rollback isn’t required because traffic hasn’t begun rerouting to the replacement environment.”

Auto-rollback governs traffic, not fleet lifecycle: blue never stopped serving, so there is nothing to roll back. A stopped deployment does not even reach that check — it returns an empty rollbackInfo.

Terminal state	Traffic shifted?	Rollback fires?	Green fleet
Succeeded	yes	n/a	becomes the live fleet
Failed before the shift	no	no — “not required”	orphaned
Stopped	no	no — never evaluated	orphaned

Both branches have already fired. A stalled CRM deploy on 2026-07-20 leaked a fleet that billed for five weeks before anyone noticed; a failed backend deploy on 2026-08-24 leaked one that was force-deleted by hand 35 minutes later. Since the cutover, 56 production blue/green deployments reached a terminal state, exactly two were non-success — and both leaked. (A failure only leaks if it happens after the green fleet is created and before the traffic shift; failures that end earlier — no instances, missing IAM permissions — never create one. Both of these were in that window, and both leaked.)

The obvious detection rule would delete production

“A CodeDeploy_* ASG with no in-progress deployment against its group is an orphan” matches all three live fleets in this account. Every ASG here is named CodeDeploy_* — a promoted green fleet keeps the name of the deployment that created it — and normally no deployment is in progress.

The discriminator that actually works is target-group attachment, plus a fourth condition that is easy to miss. Sweep on the conjunction: TargetGroupARNs empty and no in-progress deployment for the group and created more than two hours ago and no deployment for that group reached a terminal state in the last ~5 minutes.

Only the first of those separates an orphan from production. The age check does far less than it looks: a promoted fleet stays live until the next deployment replaces it, so it is routinely days old — one was 4 days 2 hours old when its successor finally deleted it, and today’s live CRM fleet has been running since 2026-08-22. Age only rules out a fleet that is mid-deploy. Target-group attachment is the whole discriminator, and a leaked fleet is leaked precisely because traffic never reached it.

That fourth one is a race margin, and it is deliberately small. A plausible-sounding argument says it needs to be much larger: on success CodeDeploy holds the outgoing blue fleet for terminationWaitTimeInMinutes = 10, so blue should spend ten minutes looking exactly like an orphan. Measured, it does not. Across all 47 automated blue deletions in CloudTrail’s window, the deletion lands between 0.4 and 1.3 seconds before its own deployment’s completion time — it is the deployment’s last act, not a follow-up. The ten-minute hold happens while the deployment is still in progress, where the second condition already suppresses the sweep.

So the only thing left to guard is the sub-second gap between the sweep reading the ASG list and reading the deployment list. A long guard is not free: it blinds the sweep to a genuine orphan for its whole duration — including the case it most needs to catch, a deploy that fails after the traffic shift.

Before this is allowed to delete anything, run the query against a healthy estate and confirm it returns zero rows. This rule has been written wrong several times over, matching anything from the entire production estate to the wrong time window, and a single describe-auto-scaling-groups call would have caught every one of them. A sweep that has never been seen returning empty has not been tested.

Report it; do not auto-delete, for two reasons. An orphan costs about $0.30/day while a false positive is an outage — and the orphaned group’s name carries the deployment id, which makes it the only durable trace of a deploy that ended badly with nothing watching. A silent auto-deleter destroys the evidence.

An emergency control can outlive its emergency

Two Allow-* rules sit at the top of the Web ACL and let matching traffic skip the entire managed rule set. Neither is an arbitrary mis-ordering: both are carve-outs added under incident pressure — one created mid-flood so an uptime monitor would not be caught by newly-aggressive blocking — and neither has been revisited since.

This is worth generalising. The CRM outage of 2026-07-22 has been recorded as caused by “pre-cutover” WAF hardening. It was not: it was the mitigation for this flood, and a service was later moved behind that mitigation without anyone knowing it was one. Check for emergency controls before a cutover — they become invisible constraints on the next change.

Security

Finding	Where	Impact
Paymob webhook HMAC verification is entirely commented out — the callback accepts any body and trusts obj.success with no signature check	app/Services/PaymobService.php:146-152 (guard call), :153 (trust)	Critical — forged payment-success webhooks are accepted
Even the dead HMAC code uses == not hash_equals (non-constant-time)	app/Services/PaymobService.php:210	Timing side-channel (moot while disabled, wrong if re-enabled)
No replay / idempotency guard on payment callbacks — a replayed success re-marks the transaction, re-submits to the CRM, and re-fires notifications	app/Services/PaymentService.php:36-73	High — duplicate CRM writes / notifications
Payment callback routes are public / unauthenticated	routes/api.php:31-38	High — combined with the above
check-whitelisted-emails is an unauthenticated GET that leaks whitelist membership and the brokerage source_id	app/Http/Controllers/Api/Broker/UserController.php:116-121, routes/api/broker.php:72	Medium — enumeration / info leak
OTP throttles are keyed by the phone field value, not IP — rotating the phone value bypasses the bucket	shopper OTP routes (routes/api/shopper.php)	Medium — OTP brute-force / spam
EOI details GET forwards the raw {eoiSourceId} to the CRM with no local existence check (not a route-model binding)	routes/api/shopper.php (EOI details)	Low/Medium — CRM probing
Thawani secret can be sourced from a DB column payment_api_secret_key; Paymob integration id written to the debug log	app/Services/ThawaniService.php:25, PaymobService.php:100	Low — secret handling

Broken routes & dead code

Finding	Where	Impact
getSalesOfferPdf route points at a controller method that does not exist — dead route, 500 on hit	routes/api/broker.php:132 and routes/api/sales-man.php:79 (Base\SalesOfferPdfController has no such method)	Broken endpoint (superseded by the async generate/status/download trio)
Sales-man logout resolves to a non-existent class (Api\SalesMan\Auth\MicrosoftController) — 500 “Target class does not exist”; the real class is App\Http\Controllers\SalesMan\Auth\MicrosoftController@logout	routes/api/sales-man.php:22	Broken logout (SSO login works via routes/web.php)
Broker GET documents returns 4 hardcoded rows with document: "#"	app/Http/Controllers/Api/Broker/UserController.php:24-52	Mock in production surface
Shopper GET user/referrals (“my referrals”) returns hardcoded mock data behind a // TODO: call CRM	app/Http/Controllers/.../ReferralController.php:12-51	Mock in production surface
UnitClient::createResale() reads config('services.crms.unit.create_resale'), a key not declared in the config block → resolves to null	crms-middleware/app/Clients/Crms/UnitClient.php:61 vs config/services.php:177-186	Resale-create silently broken
PreDeliveryUnitSaleResource and the UnitAddonPackageType enum are defined but referenced nowhere in app code	app/Http/Resources/PreDeliveryUnitSaleResource.php; app/Enums/UnitAddonPackageType.php	Dead code
Taskeen’s Broker code branch is dead (Taskeen is sales-man-only)	app/Actions/Taskeen/GetTaskeenList.php:25-27	Dead code

Config, auth & doc drift

Finding	Where	Impact
Shopper auth is Sanctum, not Passport (as backend/CLAUDE.md claims); config/auth.php sets the api guard driver to passport but routes gate on auth:sanctum	app/Models/Shopper.php:11,16; config/auth.php	Documentation + config mismatch
backend/CLAUDE.md §9 references app/Services/Payment/*, a directory that does not exist (payment code is flat under app/Services/)	backend/CLAUDE.md:§9	Stale doc — misdirects searches
Frontend declares a @orascom/api tsconfig path alias pointing at libs/api/src/index.ts that does not exist	frontend/tsconfig.base.json	Dangling alias
PDF service fails closed on auth (unset API_TOKEN → 401 for all requests) even though its README/CLAUDE claim “no token ⇒ open”	backend-pdf-service/src/routes/sales-offer.ts:9-12	Doc contradicts code
middleware-package config file is git-ignored / absent, so its provider’s config() reads resolve to null — it would fail to boot as checked out; its provider also auto-registers routes + webhook ingress against its own CLAUDE rules	middleware-package/ (.gitignore last line; CrmIntegrationServiceProvider.php:48)	Library unusable as-is (it is also unused: 0 imports platform-wide)
crm-mock PROJECT-OVERVIEW.md documents a POST /api/oman/brokers/documents route that is not in routes/api.php	crm-mock/	Stale doc
PDF_SERVICE_URL default is http://localhost:3001 in config but docs cite the Docker host http://pdf-service:3001	backend/config/services.php:69	Minor drift

Integration reality-checks

Finding	Where	Impact
There is no separate Payload CMS — all content is served by the Laravel Voyager admin over the same DB (the handover note “Payload CMS” is a mislabel)	grep across repos (0 payload.config, 0 dep)	Corrects a false architectural assumption
Firebase is auth-only, not FCM push — Kreait Firebase Auth verifies shopper tokens; there is no CloudMessage/device-token/push-send code anywhere	app/Services/FirebaseAuthService.php:16; shopper LoginController.php:25	“Push notifications” don’t exist server-side
OpenTelemetry is not wired (no open-telemetry/* / @opentelemetry/* deps) — Sentry tracing covers the need	composer.json / package.json	Corrects an assumed integration
Pusher is config-scaffold only, not wired — no pusher-php-server dep, no ShouldBroadcast, default driver null/log, no frontend pusher-js	config/broadcasting.php	Broadcasting is inert
Egypt is hardcoded in several write paths: broker register and EOI submit (Country::where('slug','egypt')), add-on payment (Egypt-only), and Taskeen (Egypt-only CRM URL)	UserController.php:74; AddonPayment.php:19; Taskeen middleware	Non-Egypt markets silently unsupported on these paths
Currency-Layer is wired with a deliberate CURRENY_LAYER typo “workaround”; the frontend NX_CURRENY_LAYER_* keys are vestigial (unused)	config/currency.php; frontend env	Confusing but functional

Logic & data bugs

Finding	Where	Impact
Precedence bug: if (! $response->getStatusCode() == 200) — ! binds tighter than ==, so the guard is never true and the intended exception can’t fire	app/Services/PaymobService.php:69, ThawaniService.php:52	Error path unreachable
Resell-assistance reads $requestData['service_slug'], a key the FormRequest never collects → PHP 8.2 warning + service_type = null on every unit_resale row	app/Http/Controllers/.../CustomerRequestController.php:102	Data always null
A referral is classified referral locally off portal_page, but as is_referral in the CRM off the referral_* fields — the two conditions can disagree	app/Http/Controllers/.../LeadController.php:48-51 vs ShopperSubmitNewLead.php:41	Inconsistent classification
The unit hold is placed at create-sale (DRAFT), not at the reserve click, and share-reservation-link creates no hold — there is no local reservation TTL (expiry is CRM-owned)	app/Actions/Sale/CreateSaleCrm.php:35; ReserveSaleCrm.php:31	Holds behave unintuitively
create_unit_type and mark_pipeline_as_lost have no egypt entry in the per-country validator map — a valid egypt header still 422s on those actions	app/Http/Requests/SalesMan/Country/CountryRequestValidator.php:44-61	Egypt agents blocked on 2 actions
finish-sale is a PATCH that force-sets status = DONE with no FormRequest / body validation on the state transition	routes/api/shopper.php (finish-sale)	Unvalidated state change
Infobip non-2xx errors are swallowed; SMS silently no-ops when ALLOW_SEND_SMS=false even though the OTP row is stored/verifiable	app/Services/SmsService.php; app/Notifications/SmsNotification.php	Silent delivery failures
OTP code space is only ~10k with two fixed leading zeros (e.g. 009999): str_pad((string) random_int(0, 9999), 6, '0', ...)	app/Actions/Otp/SendOtp.php:16-18	Weak OTP entropy
The commission-analytics enum case CACHED is a misspelling of “cashed” (= paid)	app/Enums/BrokerCommissionAnalyticsStatus.php	Cosmetic but confusing
handlePaymobResponseCallback (serves all Paymob callbacks — sale / add-on / EOI) contains a hardcoded sleep(10)	app/Services/PaymentService.php:102	Blocks the request 10s

One thing that looked like a bug but isn't

The SetLocale middleware runs after next(), which looks wrong, but it is correct by design: FormatApiResponse re-serializes the resource on the way out after SetLocale has applied the header locale, so translated fields resolve correctly. Documented here so nobody “fixes” it.