HIE Federated Query

The obvious places to point a test plan are the two ends of this flow: the Requesting Provider System, where a clinician asks for records, and Member EHR A, where the records actually live. Both are tempting because both are legible — you can see a request go out and a document come back, and you can assert on it. But neither end is where this architecture earns its name. A federated query is federated precisely because the requester does not know where the patient's data is, and Member EHR A does not know it is one of several possible answers. That knowledge lives entirely in the middle, in the Patient Pointer Index and the Federated Query Router, and everything that can go subtly and silently wrong in this flow is a property of those two components rather than of the systems on either side.

The Patient Pointer Index is the first place a QA engineer should sit, because it makes a claim that nothing downstream re-verifies: that this patient, as identified by the Requesting Provider System, corresponds to a record held at Member EHR A. If that correlation is wrong, the Router will still route, Member EHR A will still respond, and the requester will still receive a well-formed clinical document — for the wrong person. Nothing in the flow raises an error, because every component did its job correctly given the input it was handed. The Federated Query Router is the second place, because it is the only component that sees the fan-out as a whole; the requester sees one request and Member EHR A sees one query, so partial results, a silent non-response from a member, or a pointer that resolves to nothing are all conditions that only the Router can distinguish from success. Test the index for correlation correctness and the Router for how it represents incomplete answers, and the endpoints mostly take care of themselves. Test the endpoints first, and you will confirm that a broken query works beautifully.

The failure mode most likely to blindside a team is the record that is accepted but sits in a staged state forever. Every other item on the list announces itself: timeouts on oversized batches, poison messages halting processing, connection pool exhaustion cascading into timeouts elsewhere, lock contention from a long-running transaction — these all produce errors, stalls, or alarms that someone notices. A staged record produces none of that. The submitting site got its acknowledgment, the pipeline reported no error, and the row exists. Nothing in the system is complaining, so nothing in the system is watched, and the gap only surfaces when a clinician queries for data that was successfully accepted weeks earlier and isn't there.

The assumption that makes it surprising is that acceptance implies eventual completion — that once a record clears intake and is durably staged, the only remaining outcomes are "landed" or "failed loudly." Federated query environments quietly break this, because acceptance is asynchronous and the party that acknowledged the record is not the party responsible for advancing it. The same assumption is what lets duplicate staged records from upstream redelivery and silent schema drift after a migration persist undetected: all three leave the data in a state the system considers valid. For testing, this means the meaningful assertion is not "did submission return success" but "did this specific record reach its terminal state within a bounded window" — a reconciliation check with an age threshold on staged records, run against real ingest, rather than a happy-path submit-and-ack test that will pass indefinitely while the backlog grows.

An ED physician at Mercy General pulls a record for a patient presenting unconscious, and her EHR sends a query to the HIE carrying `patient_mrn: "MG-4471902"`, `given_name: "Rosalind"`, `family_name: "Okafor-Reyes"`, `dob: "1978-03-14"`, `gender: "F"`, and `requesting_facility_oid: "2.16.840.1.113883.3.9101.7"`. The Patient Pointer Index takes those demographics and resolves them to an internal `hie_patient_link_id: "HL-0093311"`, and against that link returns three pointer rows — `src_system: "NorthValley-Epic"`, `src_system: "Coastline-Cerner"`, and `src_system: "Bayside-FamilyPractice"` — each with a `last_seen_encounter_date` and an endpoint OID. The Federated Query Router receives exactly those three pointers and nothing else; it has no copy of Rosalind's allergies or med list, only the addresses of who claims to. Everything downstream of CP1 is only as good as that three-row projection: a pointer that never gets emitted here is a member system that is never contacted, and the physician will never know the difference.

The concrete risk is in the projection itself. Suppose Bayside-FamilyPractice was acquired eight months ago and renamed, but its pointer row for HL-0093311 still carries the old facility OID — the router will dutifully fan out to an endpoint that resolves to nothing, and the aggregated response for Rosalind comes back with two sources' data and no marker that a third was attempted and failed. Equally plausible: the demographic match on `Okafor-Reyes` is written to the index by one site as a hyphenated family name and by another as `family_name: "Reyes"` with `Okafor` in a middle-name segment, so only two of three pointers project and the third source silently drops out of the fan-out. Add a stale replica read on the pointer index and the newest pointer — written when Rosalind was seen at Coastline last Tuesday — may not be visible at all. To test this, assert on the projection as a countable artifact rather than on the final clinical display: seed HL-0093311 with three known pointers, fire the query from `requesting_facility_oid 2.16.840.1.113883.3.9101.7`, and assert the router received exactly three fan-out targets with the expected OIDs, not "at least two." Then deliberately point Bayside's row at a dead OID and assert the aggregated response for HL-0093311 carries an explicit per-source status (attempted, no-answer) rather than collapsing to the same shape as a source that genuinely had nothing. Re-run the same query against a forced replica-lag window and against a variant record where family name is stored unhyphenated, and confirm the pointer count for HL-0093311 stays at three in both cases.

CP2 is where the router's first fan-out target actually takes delivery: NorthValley-Epic receives an inbound query carrying `hie_patient_link_id: "HL-0093311"` plus the demographic set the Patient Pointer Index resolved it from — `given_name: "Rosalind"`, `family_name: "Okafor-Reyes"`, `dob: "1978-03-14"`, `gender: "F"` — stamped with `requesting_facility_oid: "2.16.840.1.113883.3.9101.7"` so NorthValley can decide whether Mercy General is authorized to ask. NorthValley matches it to its own local chart, `src_mrn: "NV-88213"`, and returns an allergy and medication payload with a `response_status: "complete"` and its own `last_updated` timestamps. Nothing about Rosalind persists anywhere in the HIE as a result of this — the payload is assembled live and handed straight back for aggregation, so the only artifact of a successful delivery is a timely, well-formed response from this one endpoint OID.

The failure modes bite precisely because a non-answer looks like a clean empty answer. NorthValley can accept the query, queue it internally against `HL-0093311`, and never emit a response — from Mercy's side that is indistinguishable from a source that genuinely had no allergies on file for Rosalind. Target-side defaults are just as quiet: if NorthValley's interface engine fills an absent `severity` with a local default of `"unknown"` or coerces `gender: "F"` into a site-specific code before matching, the physician sees a softened or missing allergy where the source chart has "anaphylaxis." And because the router may retry a slow endpoint, NorthValley can answer twice for the same `HL-0093311` query, producing duplicate med rows that read as two active prescriptions rather than one. Test it by asserting on the per-target response as its own artifact: fire the HL-0093311 query at NorthValley-Epic with a forced hang on the endpoint and confirm the aggregated response marks that source as attempted-no-answer within the timeout rather than returning an empty-but-complete shape; seed `NV-88213` with an allergy whose `severity: "anaphylaxis"` is explicitly populated and assert the returned value is byte-identical, not defaulted; then replay the identical query twice within the retry window and assert the aggregation for HL-0093311 de-duplicates to a single med list rather than doubling it.