CQRS with Event Sourcing

The obvious endpoints in this flow — the Command API and the Query API — are the least interesting places to spend test effort, because both are synchronous, request/response surfaces where a wrong answer announces itself immediately. The real complexity sits in the seam between the Event Store and the Read Model Store, specifically in the Projection Updater. Everything upstream of the append is a decision: the Command Validator either accepts a command or rejects it, and once an event is appended to an append-only log, that decision is permanent and unrevisable. Everything downstream is an interpretation: the Projection Updater reads that permanent log and derives a Read Model Store that the Query API will present as truth. The log and the read model are two separate stores holding two representations of the same history, and only one of them is authoritative. A defect in the validator produces a bad event that is at least honestly recorded; a defect in the projection produces a read model that disagrees with an event store that is still, by construction, correct.

That asymmetry is why a QA engineer's attention belongs at the projection first. The append-only property means the Event Store cannot be repaired in place to cover for a projection bug — the correction has to come from re-deriving the read model, which makes the Projection Updater's determinism a testable property in its own right: the same event sequence must yield the same Read Model Store every time, including on replay. It also means the Query API can return a confidently wrong answer while every component the tester can easily observe looks healthy, because the Command API accepted, the validator passed, and the event was appended. Test the validator for what it lets into the log, since that is irreversible, and test the projection for whether the read model it builds still agrees with the log after arbitrary event sequences. The two APIs at the ends are worth checking, but they will mostly tell you what the middle already decided.

The failure most likely to blindside a team is the read model silently diverging from the event store because a projection bug goes undetected. Every other item on the list announces itself: a poison message halts processing, a timeout on an oversized batch throws, validation rejects edge-case values loudly, a crash mid-update leaves visible partial state. Divergence produces none of that. The command side keeps accepting commands, events keep appending durably, the projector keeps consuming without error — and the answers it serves keep drifting from what the events actually say. There is no exception to catch, no dead letter to inspect, no lag metric spiking, because the projector is not behind; it is up to date and wrong. Teams typically discover it the way they discover a stale read window with no staleness signal — through a customer, months later, arguing about a number.

The assumption that makes this surprising is the one that sells event sourcing in the first place: that because the event store is the immutable source of truth and the read model is merely derived from it, the read model is always reconstructible and therefore effectively self-correcting. It is reconstructible, but reconstruction reruns the same projection logic against the same events and reproduces the same defect — and the rebuild itself briefly serves an incomplete read model to live queries during the rebuild window, so the remedy has its own blast radius. For QA this means correctness of the projector cannot be inferred from the health of the pipeline. Testing has to compare derived state against state independently recomputed from the event stream, run continuously rather than at release time, and cover exactly the inputs that break projections elsewhere in this list — nulls, unexpected enums, older event schema versions, events arriving out of causal order across partitions. Absence of errors is not evidence of agreement, and only an explicit reconciliation check turns divergence into something you find before your users do.

A `SubmitPolicyEndorsement` command arrives at the write-side API from the broker portal carrying `commandId: "cmd-8f2a1c04"`, `policyId: "POL-2291-A"`, `effectiveDate: "2024-03-01"`, `endorsementType: "COVERAGE_INCREASE"`, and a batch of three line items: `LI-1` (`limitDelta: 250000`), `LI-2` (`limitDelta: 0`, `reasonCode: null`), and `LI-3` (`endorsementType: "COVERAGE_REINSTATE"`). The Command Validator is the gate everything crosses before anything durable happens: it checks required fields, coerces `endorsementType` against the known enum set, and confirms the batch is well-formed. Only if all of that passes does the command proceed to be written as domain events to the event store, which is the single source of truth; the read model that the portal later queries is just an asynchronous, rebuildable projection of those events, so nothing this checkpoint rejects will ever appear on the query side at all.

The dangerous outcomes here are the quiet ones. `LI-2`'s `limitDelta: 0` and null `reasonCode` are exactly the edge-case values that get rejected outright or silently mapped to a default, and `LI-3`'s `COVERAGE_REINSTATE` may not be in the validator's enum set, producing a mapping error that either fails the whole `cmd-8f2a1c04` or drops that one line. Worse, splitting the three line items into separate events can lose or duplicate one, and because the API can return `202 Accepted` for `cmd-8f2a1c04` while the event store write never actually landed, the client sees success and then blames projection lag when the endorsement never shows up on the read side — a stale-read story that is actually permanent data loss. Under batch submission the portal can also trip throttling, or an oversized batch can time out mid-validation, and a single unparseable command can sit as a poison message blocking the ones behind it. To test, submit `cmd-8f2a1c04` exactly as above and assert per-line-item outcomes rather than a single overall status: confirm `LI-2` is accepted with `limitDelta: 0` preserved and `reasonCode` still null rather than defaulted, and that `LI-3`'s unknown enum yields an explicit, addressable rejection naming the field. Then, for every command the API acknowledges, read the event store directly — not the query model — and assert exactly three events for `POL-2291-A` with matching line-item ids and no duplicates, since a success response with no corresponding event is the failure that ordinary read-side checks cannot distinguish from lag. Finish by replaying a few hundred endorsements at once to see whether throttling surfaces as a clear retryable error, whether a 500-item batch completes or times out partway leaving some line items persisted, and whether a deliberately malformed command wedged into the stream stops `cmd-8f2a1c04` from ever being processed.

Picture `cmd-8f2a1c04` clearing the Command Validator and arriving at the event store as an append: three domain events for aggregate `POL-2291-A` — `CoverageLimitIncreased` for `LI-1` with `limitDelta: 250000`, `CoverageLimitIncreased` for `LI-2` with `limitDelta: 0` and `reasonCode: null`, and (assuming the enum question is resolved in its favor) `CoverageReinstated` for `LI-3` — each stamped with an expected aggregate version, say appending at versions 7, 8, and 9 after the endorsement stream's last known version 6. That append is the only durable act in the whole flow: once those three records are in the log at those offsets, the endorsement exists, and the read model the broker portal queries is just a replayable derivative that will catch up when the projector reads offsets 7 through 9. Nothing else in the system holds authoritative state, so a write that does not land here did not happen, regardless of what the API told the portal.

Three things go quietly wrong at this seam. If the API returns `202 Accepted` for `cmd-8f2a1c04` before the append is fsynced and acknowledged by the store, a node loss between ack and durability deletes all three events while the portal believes the coverage increase is filed — indistinguishable from projection lag from the outside. If a second endorsement for `POL-2291-A` is submitted concurrently and both writers assume last-version 6, one append can overwrite or interleave ahead of the other, so `LI-3`'s reinstatement lands before `LI-1`'s increase and the projector folds them in the wrong causal order. And if `CoverageLimitIncreased` later gains a required field, the projector rebuilding from offset 0 may fail to deserialize the version-7 event written today, stalling silently while the store keeps accepting writes. Test it by killing the event store process in the window between the API ack for `cmd-8f2a1c04` and the append confirmation, then reading the `POL-2291-A` stream directly and asserting either all three events at versions 7–9 or none, never a partial two; submit two endorsements for `POL-2291-A` simultaneously and assert one is rejected with a version-conflict error rather than both succeeding with overlapping versions; and replay the stored `LI-1`/`LI-2`/`LI-3` events through the current projector after every schema change, asserting `limitDelta: 0` and `reasonCode: null` still deserialize intact instead of throwing or defaulting.

Now follow `cmd-8f2a1c04`'s three events past the log and into the projector. The updater polls the `POL-2291-A` endorsement stream, sees offsets 7, 8, and 9, and folds each into the broker portal's read-model row — call it `policy_coverage_summary` keyed on `POL-2291-A`, holding a per-line `currentLimit`, a `status`, and a `lastAppliedVersion`. Offset 7 lifts `LI-1`'s `currentLimit` from 750000 to 1000000 by applying `limitDelta: 250000`; offset 8 applies `LI-2`'s `limitDelta: 0`, which changes no number but must still advance `lastAppliedVersion` to 8; offset 9 flips `LI-3`'s `status` from `LAPSED` to `ACTIVE` per `CoverageReinstated`. When all three land, `lastAppliedVersion` reads 9 and the portal's endorsement screen matches the log. Until then the portal serves the pre-endorsement row — `LI-1` still at 750000 — with no marker that a newer version exists in the store, which is expected staleness inside the disclosed window and not a defect.

What goes wrong is that the row can settle into states the log never described. If the updater crashes after writing `LI-1`'s new `currentLimit` but before stamping `lastAppliedVersion: 7`, the restart replays offset 7 and applies `limitDelta: 250000` a second time, leaving `LI-1` at 1250000 — a number no sequence of events ever produced. If offsets are consumed out of causal order across partitions and offset 9 lands before 7, the portal briefly shows `LI-3` reinstated while `LI-1` is still at its old limit, a point-in-time snapshot that never existed on the write side. And a burst of endorsements behind `cmd-8f2a1c04`, or a rebuild replaying `POL-2291-A` from offset 0, can leave the row stale or half-built with the store happily accepting writes and no error raised anywhere. Test it by killing the projector process between the `currentLimit` write and the `lastAppliedVersion` stamp for offset 7, restarting, and asserting `LI-1` reads 1000000 and not 1250000; by feeding offsets 9, 8, 7 in that order and asserting the projector either buffers or rejects rather than publishing a row where `LI-3` is `ACTIVE` while `lastAppliedVersion` is still 6; by asserting the no-op offset 8 still advances `lastAppliedVersion` to 8 so `LI-2` is not silently reprocessed forever; and by alerting on the gap between the `POL-2291-A` stream head (9) and the row's `lastAppliedVersion`, so a stalled projector surfaces as a lag alarm instead of a broker quietly quoting 750000 on a policy the log says is at 1000000.

Downstream of the projector sits the Query API, and this is where the broker actually meets `POL-2291-A`. A portal request — `GET /policies/POL-2291-A/coverage-summary` — reads the `policy_coverage_summary` row and returns the three lines with their `currentLimit` and `status` values plus the `lastAppliedVersion` the projector last stamped. Once offsets 7, 8, and 9 have all folded in, the response reads `LI-1: 1000000`, `LI-2` unchanged, `LI-3: ACTIVE`, `lastAppliedVersion: 9`, and the endorsement screen matches the log. Mid-catch-up, the same call legitimately returns `LI-1: 750000` with `lastAppliedVersion: 6`, and that is the disclosed staleness window doing its job, not a defect — the event store is still the source of truth and the row is just a rebuildable projection of it.

What goes wrong is that the read path can misrepresent an otherwise-correct row. `LI-2`'s `limitDelta: 0` is exactly the edge-case value a validator chokes on: a serializer or response schema that treats a zero-delta or an unchanged `currentLimit` as missing can drop `LI-2` from the payload entirely, so the broker sees a two-line policy where the log says three. Under a batch load — a whole book of endorsements behind `cmd-8f2a1c04` plus the portal refreshing — the Query API can throttle, and a 429 or truncated response is easy to render as "no coverage found" rather than "try again," which reads to a broker as a lapsed policy. Worst is the write-then-read illusion: the endorsement command returned success at the event store, so the portal shows a confirmation, yet a query moments later still reports `750000` with no `lastAppliedVersion` exposed, and a user reasonably concludes the write was lost. Test it by querying `POL-2291-A` after only offset 8 has been folded and asserting `LI-2` appears in the response with its unchanged limit and `lastAppliedVersion: 8`, never omitted; by hammering the coverage-summary endpoint for `POL-2291-A` under a synthetic burst and asserting throttled responses surface as an explicit retryable status the portal distinguishes from an empty result; and by submitting `cmd-8f2a1c04`, immediately querying, and asserting the response carries `lastAppliedVersion: 6` so the staleness is visible to the caller rather than looking like a silently dropped endorsement.