Event-Driven Pub/Sub Broadcast

The obvious endpoints in this flow lie to you. Order Service publishing an event and Push Notification Service delivering a message are both easy to observe and easy to assert against, which is exactly why they attract testing attention they don't deserve. The real complexity sits in the Order Events Topic and the Notification Adapter, because that is where a single publish stops being a single thing. The topic decouples the producer from the consumer entirely: Order Service gets no answer about whether anything consumed its event, no answer about how many times, and no answer about when. The Notification Adapter, not the publisher, decides what an order event *means* as a notification — which events warrant a push, how a domain event's shape maps onto a delivery payload, and what happens when it can't map cleanly. A green test at each endpoint tells you the producer produced and the deliverer delivered, and says nothing at all about whether the right event became the right notification.

So a QA engineer's first questions belong in the middle. Because the topic is a broadcast, the Notification Adapter is one subscriber among potentially many, and its failure to consume is invisible to Order Service — meaning a lost or unprocessed notification produces no upstream error signal anywhere. Because the adapter re-consumes on retry or redelivery, the interesting defect is not "no notification" but "three identical push notifications for one order," a bug that only exists at the adapter and only appears under conditions endpoint tests don't create. Ordering is a middle problem too: two order events published in sequence carry no guarantee that the adapter processes them in that sequence, so a status notification can arrive describing a state the order has already left. Test the adapter's consumption semantics — duplicate handling, out-of-order tolerance, unmappable-event behavior — before you spend another hour asserting that Order Service publishes and Push Notification Service pushes.

The failure that actually blindsides teams is the silent gap: a change saved without an integration event ever being emitted. Every other item on this list leaves evidence somewhere. Duplicates on retry show up as double-applied updates. Dead-lettered messages sit in a queue, unalerted but countable. A black-holed event type, a filter that discards a subtype, a mapping error — all of these mean a message existed and went somewhere you can go look. A silent gap produces nothing: no message, no dead letter, no retry, no log line to correlate. The source system reports success, because from its perspective the write succeeded. The target simply never learns, and stays quietly, plausibly wrong until someone compares the two systems by hand.

The assumption that makes it surprising is that persisting the change and publishing the event are one operation — that if the record is in the database, the event is on the bus. It's a reasonable belief because that's the intent of the design, and because everything downstream is built to trust it: retry logic assumes a message to retry, dead-letter monitoring assumes a message that failed delivery, schema and filter checks assume a message to inspect. For testing, this means end-to-end assertions that start from a published event validate nothing about the gap, and neither do broker-side metrics. You have to test at the boundary the assumption spans: exercise the write path, especially its transient-failure and rollback branches, and assert that a corresponding event was emitted — then reconcile source records against delivered events as a standing check, because absence is the only symptom this failure has.

Picture the Order Service committing order `ORD-88421` — customer `CUST-3390`, total `142.50 USD`, status moving from `PENDING` to `CONFIRMED` — and then publishing a single message to the Order Events Topic: `{"eventType":"OrderConfirmed","eventVersion":"1.2","eventId":"evt-7f3a91c2","orderId":"ORD-88421","customerId":"CUST-3390","totalAmount":142.50,"currency":"USD","occurredAt":"2024-06-11T14:22:07Z"}`. That one publish is the only thing CP1 owns: the database write and the emission of the integration event onto the topic. From there the broker broadcasts the identical message to all three independent subscribers — notification, analytics, and audit — each reading the same stream for its own purpose, with the Order Service knowing nothing about any of them.

The things that go wrong here are all invisible from the producer's side. `ORD-88421` can be committed to the orders table while the publish silently fails or is never called at all, so the row says CONFIRMED and no subscriber ever hears about it. A transient broker error followed by a retry can put `evt-7f3a91c2` on the topic twice, so all three consumers process the confirmation twice. If the service's own model changes and someone renames `totalAmount` to `orderTotal` or bumps `eventVersion` to `2.0`, that's schema drift landing on a topic three unrelated teams read. Worse, a subscription filter written for `eventType = "OrderConfirmed"` will silently drop the message if someone starts emitting `"ORDER_CONFIRMED"`, and an entirely new type like `OrderPartiallyConfirmed` may match no subscription at all and be black-holed with no error anywhere; failed deliveries can also land in a dead-letter queue that nobody is watching. To test this, confirm `ORD-88421` and assert both that the order row is CONFIRMED *and* that exactly one message with `eventId` `evt-7f3a91c2` reached the topic — then force a broker timeout mid-publish, retry, and assert the topic still holds exactly one `evt-7f3a91c2`. Validate the emitted payload against the registered `OrderConfirmed` v1.2 schema so a rename of `totalAmount` fails the build, assert each of the three subscriptions actually received the `ORD-88421` message rather than just checking the publish succeeded, publish a deliberately unknown `eventType` and assert the test fails loudly rather than passing on silence, and assert the dead-letter queue depth is zero at the end of the run.

The notification subscriber picks up `evt-7f3a91c2` off the Order Events Topic and hands it to the Notification Adapter, whose whole job is to turn that generic event into a call against the email provider's API. It maps `customerId` `CUST-3390` to a recipient lookup, `totalAmount` `142.50` plus `currency` `USD` into a rendered `"$142.50"` in the template body, `orderId` `ORD-88421` into the subject line, and `eventType` `OrderConfirmed` into template id `tmpl_order_confirmed_v3`, then POSTs to the provider with a bearer token from a cached session. Analytics and audit are reading the identical message off the same topic at the same moment and know nothing about this — if the adapter fails here, `ORD-88421` is still committed, still on the topic, and still counted by analytics; only the customer's email is missing, and only this subscription is wrong.

The failure modes are all local to this adapter and invisible to the other two. A subtype the mapper doesn't know — `OrderPartiallyConfirmed` — falls through to no template or to `tmpl_order_confirmed_v3` with an unmapped `totalAmount`, and `CUST-3390` gets an email claiming the full `142.50` was confirmed. The cached bearer token can expire between messages, so the POST comes back 401 and, if the adapter treats that as retryable, it hammers the provider with the same `evt-7f3a91c2` in a retry storm that never succeeds because the token is never refreshed. And because `evt-7f3a91c2` can legitimately arrive twice from CP1, or because a later status event for `ORD-88421` can be processed concurrently with this one, two in-flight handlers can race on the same order's notification record and send two emails or write a last-writer-wins state that disagrees with the order. To test it, feed the adapter the exact `evt-7f3a91c2` payload and assert the outbound request carries `tmpl_order_confirmed_v3`, recipient resolved from `CUST-3390`, and a body rendering `$142.50 USD`; feed it an `OrderPartiallyConfirmed` variant and assert it fails loudly rather than falling back to the confirmed template. Expire the session token, replay `evt-7f3a91c2`, and assert exactly one refresh-and-retry rather than an unbounded retry count against a persistent 401. Deliver `evt-7f3a91c2` twice and assert one email for `ORD-88421`, then deliver it concurrently with a second event for the same order and assert the notification record ends in a single consistent state. Finally, stall the analytics consumer on the same message and assert the notification path still delivers `ORD-88421` on time — the subscriptions must not block each other.

Downstream of the mapping, CP3 is the moment the adapter's rendered payload actually leaves for the provider and becomes a delivered push/email for `ORD-88421`. Concretely: the POST body carries `template_id: tmpl_order_confirmed_v3`, `to: <recipient resolved from CUST-3390>`, `subject: "Your order ORD-88421 is confirmed"`, `body_vars: {amount: "$142.50", currency: "USD"}`, and `idempotency_key: evt-7f3a91c2`, with the bearer token from the cached session on the Authorization header. The provider answers `202 Accepted` with its own `provider_message_id: pm_4471c9`, and the adapter marks the notification record for `ORD-88421` as sent. Nothing here is visible to the analytics or audit subscribers reading the same message off the Order Events Topic — they have already counted and recorded `ORD-88421` regardless of whether `pm_4471c9` ever exists, so a failure at this sink is a correctness failure of exactly one subscription and of nothing else.

Two things go wrong at the sink specifically. First, alert storms from flapping values: if the order's status oscillates and each swing republishes to the topic, the adapter will keep firing sends for `ORD-88421` — `evt-7f3a91c2` followed by a confirm/unconfirm churn — and `CUST-3390` gets a burst of contradictory "$142.50 confirmed" pushes, each individually valid and collectively spam, especially if the provider treats a fresh `idempotency_key` per event as a distinct legitimate send. Second, missed alerts when evaluation lags ingestion: the adapter can fall behind the topic, and by the time `evt-7f3a91c2` is evaluated the notification is stale or superseded, so the send is either dropped as no-longer-relevant or delivered late while analytics has long since shown the order as confirmed. To test, drive a flap sequence for `ORD-88421` — confirm, revert, confirm again within a short window — and assert the sink emits a single coalesced or suppressed delivery to `CUST-3390` rather than one POST per swing, with the storm bounded by an explicit rule rather than by provider rate limits. Then hold the adapter back so it lags the topic, release `evt-7f3a91c2`, and assert it is still delivered (or explicitly and observably dropped with a reason) rather than silently lost, and confirm the recorded `provider_message_id` for `ORD-88421` is exactly one — while analytics and audit, reading the same message, remain unaffected either way.