Runbook: a broker's lead isn't reaching the CRM

Symptom: a broker submitted a deal, but it doesn’t appear in the CRM (or the broker got an error). This runbook takes you from the report to the layer at fault, then to the tool and the owner. It assumes the flow is understood.

First question: what did the broker actually see?

The response message tells you the half of the system at fault before you open a log: "Success!" (2xx) means the CRM accepted it — look at the read/list side. "Failure!" (non-2xx) means it was rejected — the status code tells you where.

The status codes you’ll actually see

The middleware normalizes every CRM/Logic-App error to HTTP 400 (an upstream 401/403/500 is delivered to the backend as a 400 with the upstream JSON inside the inner message). So on this path the broker only ever sees one of: 200, the local 422, a 400, a 404, or a 500.

Status	What it means	Owner
200, but lead not visible in My Deals	CRM accepted the write; the read side hasn’t surfaced it (or a per-country read was silently dropped — see step 5), or the returned source ids were empty	Read path / CRM team
200, lead_source_id/customer_source_id empty	Logic App returned 200 without Lead.leadid/Lead.customerid	CRM team (Logic App)
422, top-level data is a string (“maximum number of leads…”)	Local leads-limit gate — never reached the CRM	Product / config
422, top-level data is a string (“not linked to a brokerage company…”)	The broker has no brokerage company, so there is nothing to count the limit against — never reached the CRM	Admin: link the broker to a company
422, top-level errors object (no data)	SubmitLeadRequest validation failed (bypasses the success envelope)	Frontend payload / backend rules
400 with a CRM/Azure JSON blob in the inner message	Logic App rejected the payload, or the middleware’s Azure-AD auth failed (stale token → upstream 401, delivered as 400)	Middleware ↔ CRM team
500, only on a destination with a leads limit, only for a broker with no company	The broker_company_id lookup dereferences a company that is not there. Fixed by Backend#133 — before that lands this is what these brokers get, and it looks like an infrastructure fault when it is not	Backend
500	Middleware unreachable from the backend / a 300s connect/read timeout, or the middleware’s “Unsupported Country” guard fired (it throws UnsupportedCountryException, which renders as 500 — see note) — the latter is rare on the broker form (country is derived from a validated destination)	Infra / Middleware / data

Decision tree

1. Was it a local refusal? A 422 whose top-level data is a plain string is the backend’s GetLeadsCount gate (added May 2026, ORASCOM-3526) — not a CRM problem. Read the string; there are two, and they need opposite actions:

   • “You have reached the maximum number of leads for this destination.” — the brokerage’s per-destination monthly limit. Resolution is product/config.
   • “Your account is not linked to a brokerage company, so leads cannot be created for this destination. Please contact your administrator.” — the broker has no brokerage_company_id. The limit is counted per company, so there is nothing to count against. Do not send this broker to look at their lead count — there isn’t one, and nothing they do will change it. An administrator has to link the account to a brokerage company.

   A validation 422 looks different again: a top-level errors object, no data, no Success!/Failure! wrapper (validation errors bypass the envelope). Stop here if any of these.

   Caution

   Before Backend#133 the second case is a 500, not a 422 — it reads as a middleware or infrastructure fault and will send you down step 4 for nothing. Nine production brokers are in that state, and only on a destination that has a limit configured. If you are looking at a 500 on the broker lead form, check brokerage_company_id on that broker before anything else.

   Caution

   Whatever the broker reported seeing may not be the string above. The reason travels in data while message is the generic "Failure!", and the broker app showed message for everything except the limit case until Frontend#74. A broker saying “it just said Failure” is consistent with any of these — get the response, not the retelling.

2. Did the backend even try the CRM? Check user_requests. On an exact 200 the backend writes one user_requests row (user_id = broker, request_type = 'lead', recent date).

   • Row exists → the CRM returned 200; the write succeeded. If the lead still isn’t visible, it’s the read/list path or the CRM’s own processing — jump to step 5.
   • No row → the submit did not get a clean 200 (limit gate, validation, or CRM rejection).

     Caution

     Loose == 200: a 201 from the CRM creates the lead but skips the local row — so “no row” doesn’t strictly prove “not created”. Confirm against the CRM.

3. Read the backend log. Channel LOG_CHANNEL (default stack); the detailed lines need LOG_LEVEL=debug. LeadController@submitLead logs, in order: LeadController submitLead: Start (info) → request (the full payload, debug) → SubmitNewLead response (the raw middleware Response — status + body, debug) → mappedResponse (debug) → : End (info). The SubmitNewLead response line tells you exactly what the middleware returned.

4. Interpret the middleware’s answer (it’s a 400 for every CRM error):

   • 400 with an Azure/Dynamics JSON blob → either the Logic App rejected the payload (bad/missing field, or a Dynamics validation — compare against the Dynamics field map), or the middleware’s Link-Dev Azure-AD token was stale/invalid. On an auth failure the middleware sends the request with no Authorization header (empty-token behaviour, token cached as linkdev-crm-token); the Logic App replies 401, which the middleware delivers to you as a 400 with the auth blob in message. There is no line in the backend log for the auth failure — but the middleware debug log does contain the failed auth response (LinkDevAuthenticatedClient::authenticate logs it at LOG_LEVEL=debug).
   • 500 → the middleware is unreachable from the backend, or a 300s connect/read timeout (no retries). A Logic-App outage surfaces as a 400/500 response from the middleware, not as a thrown exception.

5. If the write succeeded but the lead is invisible in My Deals: the list is a separate read path — per-country direct Dynamics OData v9.1/v9.2, live on every page load (no cache). Two causes, in order:

   • A per-country read failed silently. If a country’s Dynamics read returns non-200, GetLeads sets that country’s result to null, logs Leads list … Status: fail / Error: …, and the endpoint still returns 200 Success! with that country’s leads missing. Grep the backend log for Leads list failures first — a green My-Deals response does not mean every country was read.
   • Slow or lagging Dynamics. Otherwise it’s a slow Dynamics read or Dynamics hasn’t finished processing. Confirm by querying the CRM directly (see Access, below) — the lead’s PII is not stored locally, so the portal DB can’t answer “does this lead exist”.

Access & escalation

Where to reach each tool. Environment-specific links (log sink URL, Sentry project, Tempo, the per-country Dynamics org URLs, on-call handles) are not in the repo — fill them in for your environment with your ops lead; the structure is fixed.

Signal	How to reach it	Environment-specific bit to fill in
Backend log	LOG_CHANNEL (stack); set LOG_LEVEL=debug for the lead lines	Where UAT/prod logs ship
Sentry	Backend and frontend report; filter to the lead endpoint. ⚠ The middleware is not in Sentry at all, and a 500 reaches backend Sentry only when the middleware is unreachable (Logic-App/CRM failures are invisible to Sentry — see below)	Sentry project URL
Distributed trace	OpenTelemetry is enabled in the backend only (OTEL_SERVICE_NAME=orascom-backend, OTLP → Grafana Tempo). The trace covers the backend; the middleware and CRM are not instrumented, so a slow middleware/CRM shows as one long backend HTTP-client span, not separate hops	Tempo/collector endpoint (repo only shows a local-dev one)
The CRM	Reads go via per-country Dynamics OData: {orgBaseUrl}/api/data/v9.2/leads?$filter=…. To confirm a lead exists, query the country’s Dynamics org or ask the CRM team	The Dynamics org URL per country

Three things the docs used to get wrong — now corrected

1. Sentry does not catch Logic-App outages. The backend HTTP client never ->throw()s, so a middleware 400/500 is an ordinary response, not an exception — nothing is sent to Sentry. Only a backend-can’t-reach-middleware ConnectionException lands in Sentry.
2. The trace stops at the backend. There is no orascom-middleware span and no Dynamics span.
3. “Unsupported Country” is a 500, not a 404. The middleware’s UnsupportedCountryException passes HTTP_NOT_FOUND as the exception code and has no render(), so Laravel emits a generic 500. Don’t wait for a 404 that never comes — see the KT tracker.

Who owns the fix

Fault	Owner
Validation, limit gate, response shape	Backend team (backend)
Field mapping, Azure-AD auth, timeout/retry policy, country header handling	Middleware team (crms-middleware)
Logic App rejects/errors, Dynamics field not accepted, lead created but not returned	CRM team (Link Dev — Logic Apps + Dynamics orgs)
Broker’s per-destination monthly limit	Product / config

(Add the concrete team owners / Slack channels / on-call for your org next to each row.)

No retries, 300s timeout

The middleware makes a single attempt with a 300-second connect+read timeout and no retry. A transient CRM blip surfaces as a hard 400/500 to the broker, not a silent retry. Adding resilience is a middleware change (AbstractClient).