Omnichannel Order Orchestration

The obvious endpoints in this chain are the ones everyone tests first: place an order at the Storefront Checkout Service, then confirm the Store POS Inventory API decremented. Both are easy to assert against and both are misleading, because neither of them is where the order actually gets decided. The real complexity sits in the middle, at the Order Orchestrator's dependence on the Inventory Availability API and the Ship-from-Store Adapter, because those two hops answer different questions. The Availability API tells the orchestrator what it believes is stocked; the Ship-from-Store Adapter is what commits that belief against a specific store's POS. Between the read and the commit there is a window, and the orchestrator owns it. A checkout that succeeds and a POS that ends up wrong are not contradictory outcomes here — they are the expected shape of a failure that lives entirely in the orchestration layer and shows up nowhere in the endpoints you were watching.

That means a QA engineer's first attention belongs on the orchestrator's contract with its two downstream dependencies, not on checkout happy paths. The questions worth asking are what the orchestrator does when the Inventory Availability API says yes and the Ship-from-Store Adapter's call to the Store POS Inventory API then fails or comes back short, whether the orchestrator retries the adapter and what a retry does to a POS that may have already applied the first attempt, and whether availability is re-checked at commit time or trusted from the earlier read. Testing the Storefront Checkout Service in isolation validates that an order can be accepted; testing the Store POS Inventory API in isolation validates that a decrement can be applied. Neither tells you whether the orchestrator correctly turned one into the other, and that translation is the only part of this flow where the system can be internally inconsistent while every individual component reports success.

The failure most likely to blindside a team is routing logic silently defaulting to warehouse when the store-availability call times out. Every other item on this list eventually announces itself: duplicate events show up as duplicate records, schema drift and field mapping errors break consumers loudly, validation rejects edge-case records with a rejection, throttling and retry storms show up as load, auth expiry fails a call. Even the two nastiest quiet ones — a success response with no persisted record, and changes saved without an emitting integration event — leave a detectable discrepancy between two systems that a reconciliation check can find. The warehouse default leaves nothing to reconcile. The order routes, fulfills, and closes successfully. There is no failed record, no error event, no stuck message. The only artifact is a fulfillment decision that was made on absence of data rather than presence of it, and it looks identical to one made on good data.

The assumption that makes it surprising is that a timeout is a failure — that when a dependency doesn't answer, the flow either errors, retries, or at minimum marks the decision as degraded. Here the timeout is swallowed and converted into a business decision, so the availability call's reliability never appears in any success metric the team watches. That has a direct testing consequence: you cannot validate this path by asserting on order outcomes, because the outcome is valid. You have to test the store-availability call as an injected timeout and assert on *why* the route was chosen, which means the routing decision needs to carry its own provenance — availability-confirmed versus defaulted. If the service doesn't record that distinction, no test downstream of routing can tell the two apart, and neither can production. The same blind spot compounds with the stale/cached availability read and the two-orders-to-one-last-unit race: all three produce a confidently wrong routing decision through a green path, and only the last one is likely to generate a complaint that traces back.

Picture order ORD-88412 leaving the Storefront Checkout Service at 14:02:17 UTC: a single line item, SKU `TSH-BLK-M` ("Ridgeline Tee, black, medium"), quantity 1, ship-to postal code 98104, payment authorized. Checkout persists the order row and emits an `OrderPlaced` integration event carrying `orderId: ORD-88412`, `sku: TSH-BLK-M`, `qty: 1`, `destinationZip: 98104`, `channel: web`, and `eventVersion: 2`. The Order Orchestrator picks it up and, before committing anything, reads availability: store #114 (Seattle Downtown) shows `onHand: 1`, the Reno warehouse shows `onHand: 340`. Store #114 is two hours from the customer, so the orchestrator picks ship-from-store and only then commits the decrement. Everything downstream — which bulkheaded lane decrements, which fulfillment path runs — is decided by that pre-commit read.

The dangerous version is that store #114's `onHand: 1` came from a cache refreshed forty seconds ago, and a walk-in customer bought the last black medium at 14:02:05; ORD-88412 gets routed to a store that has nothing to give, and the customer is promised same-day stock that does not exist. Two other shapes of the same checkpoint bite just as hard: checkout saves ORD-88412 but the `OrderPlaced` event never publishes, so the orchestrator never sees the order at all and no downstream system will ever re-surface it; or the publish times out, retries, and the orchestrator routes ORD-88412 twice, decrementing store #114 to negative. Test it by holding the availability read: freeze the cached store #114 value at `onHand: 1`, sell that unit in-store, then replay ORD-88412 and assert the orchestrator does not route to store #114. Fire two orders, ORD-88412 and ORD-88413, at the same last unit within the same second and assert exactly one is routed to store #114. Replay the identical `OrderPlaced` for ORD-88412 twice and assert one routing decision and one decrement. Kill the checkout-to-orchestrator publish after the order row is written and assert ORD-88412 is detected as an unemitted gap rather than sitting silently. And force the store-availability call to time out, then assert ORD-88412 surfaces the timeout explicitly rather than quietly landing in the warehouse lane as if store stock had been checked and found wanting.

Once the orchestrator has settled on ship-from-store for ORD-88412, the handoff at CP2 is narrow and concrete: the Inventory Availability API's answer for store #114 (Seattle Downtown, `onHand: 1`) becomes the input to a commit call into the Ship-from-Store Adapter — something like `{orderId: "ORD-88412", locationId: "114", sku: "TSH-BLK-M", qty: 1, decrementReason: "SFS_COMMIT", eventVersion: 2}` — and the adapter is what actually turns that decision into a decrement against store #114's stock and starts the ship-from-store fulfillment path. This is the moment the pre-commit read stops being advisory and becomes a promise: the same call both takes the unit off store #114's book and puts ORD-88412 into the store-pick lane, separate from the Reno warehouse lane by design so a failure on one side cannot stall the other.

What goes wrong here is that the adapter can say yes without meaning it. A `200 OK` with `commitStatus: "ACCEPTED"` that never lands the decrement leaves store #114 still showing `onHand: 1` for the next order while ORD-88412 walks off with the unit; an expired store-system session can turn the same call into a `401` that a naive retry loop hammers into a storm; a `qty: 1` against an `onHand: 1` edge boundary is exactly the value validation tends to reject or round oddly; and if ORD-88412 and ORD-88413 both commit against store #114 in the same second, the adapter's update to that one SKU row is a plain concurrency race. To test it, commit ORD-88412 through the adapter and then re-read store #114's `onHand` from the source of record rather than trusting the response body — assert it moved 1→0, and that a stubbed success-with-no-persist is caught, not celebrated. Expire the adapter's session token mid-flight on ORD-88412 and assert one re-auth and a bounded retry, not an unbounded loop. Send ORD-88412 and ORD-88413 concurrently at the last unit and assert exactly one decrement lands and the loser gets an explicit rejection. Then fail the Ship-from-Store Adapter outright for ORD-88412 and assert the Reno warehouse lane keeps flowing for other orders, and that ORD-88412 surfaces the commit failure explicitly rather than being reported as fulfilled.

Downstream of the adapter's commit, the decrement has to actually land somewhere, and for store #114 that somewhere is the Store POS Inventory API — the register-side system of record that a Seattle Downtown associate also sells against. The adapter translates the commit into a write that looks roughly like `{storeId: "114", sku: "TSH-BLK-M", adjustment: -1, newOnHand: 0, sourceRef: "ORD-88412", reason: "SFS_COMMIT"}`, and the POS API's response is the last word on whether ORD-88412's unit is really off store #114's book. This is the target delivery for CP2's promise: everything upstream — the availability read that said `onHand: 1`, the routing decision, the adapter's `commitStatus: "ACCEPTED"` — is only as true as this write. If the POS row still reads `onHand: 1` after the call, the store floor will happily sell TSH-BLK-M to a walk-in customer that ORD-88412 has already claimed.

The failure modes cluster tightly here. `newOnHand: 0` is precisely the edge value POS validation likes to reject or coerce — a schema that treats zero as unset, or a rule that refuses adjustments driving stock to floor, and ORD-88412's write silently no-ops. Nightly or peak-hour batches of these decrements against store #114 invite throttling, where the POS API sheds ORD-88412's write under load and returns something that reads like acceptance. And the nastiest case is the same one as upstream, one layer deeper: a `200` from the POS API whose row never changed, which the adapter faithfully reports as success and the orchestrator marks fulfilled. To test, write ORD-88412's `-1` adjustment and then read store #114's TSH-BLK-M row back from the POS system directly — assert `onHand` is 0, not that the response said so. Repeat the write at the zero boundary specifically, asserting the `newOnHand: 0` case is accepted rather than validation-rejected or rounded. Drive a batch of a few hundred decrements alongside ORD-88412's and assert its write either lands or returns an explicit throttle error the adapter can surface, never a false success. And stub the POS API to return `200` while persisting nothing, then assert ORD-88412 ends in a visible commit-failure state rather than being reported fulfilled to the customer.