
In short
- A test checks what you told it to check, on the day you wrote it. It fails loudly when something breaks and stays silent when a rule disappears.
- Removed validation, loosened limits and features switched off by a vendor update all leave the happy path working, so the suite stays green.
- Negative tests, checking after every change, and comparing today’s app with yesterday’s catch most of what a green suite misses.
On this page
Picture a Monday morning. Over the weekend, the platform your online store runs on pushed a routine update. At 7am the regression suite runs. 412 tests. All green. You close the report and get on with the week.
On Wednesday, support asks why a few hundred orders have no email address on them. Nobody can send those customers a receipt, a shipping update or a refund.
The update had made the email field at checkout optional. The tests didn’t notice, because nothing broke. Checkout still worked. It just stopped checking.
If you have tested software for a while, you have probably seen a version of this. This guide explains why it happens, why it is getting more common, and what to change in how you test so the next one is caught on Monday.
01Tests notice what breaks, not what disappears
A test is a frozen opinion. It encodes what someone believed the software should do, on the day they wrote it. That makes it very good at one kind of change and blind to another.
| What changed | What the suite does | How much it matters |
|---|---|---|
| A button is renamed from "Buy now" to "Place order" | Fails loudly: the test can’t find its button | Usually nothing. It’s a label. |
| A required field becomes optional | Passes: every step still works | Bad data in production, discovered days later |
| A limit is raised, say from 5 coupon uses to unlimited | Passes | Real money, quietly |
| A format check is removed, like a ZIP code pattern | Passes, unless a test types something invalid | Bad data, failed deliveries |
The harmless change gets all the attention. Someone spends twenty minutes fixing the button test, and nothing was ever wrong. The dangerous change gets none, because a missing rule never makes a test fail on its own. The steps still succeed. They just succeed for input that should have been stopped.
02Why it is happening more often
Ten years ago, business software mostly changed when you decided to upgrade it. An upgrade was a project with a date, a test cycle and someone who signed it off. The suite got updated because the project forced it to. Two things have changed since.
Vendors update the app for you
Most business platforms now ship on their own schedule. Salesforce has three releases a year that customers can’t opt out of. Dynamics 365 Finance and Operations has service updates through the year plus smaller fix builds in between. Shopify ships a new API version every quarter. Some platforms don’t publish a schedule at all, and many ship features switched off so that two customers on the same product run different apps. "Which version are we on?" no longer has one answer.
More of the code is written by AI
In October 2024 Google said about a quarter of its new code was written by AI. In April 2026 it said 75% of new code at Google is AI-generated and approved by engineers. Your vendors use the same tools. Google’s own DORA research found most engineers now use AI at work and teams are shipping faster, while delivery stability isn’t keeping pace. More change means more chances for a rule to slip, and the part of the pipeline meant to catch that, testing, mostly still works the way it did ten years ago.
03Write tests that expect to be stopped
Most suites test the happy path: fill in the form, submit, see the confirmation. Those tests prove the feature works. They can’t prove a rule is still there. For that you need the unhappy path: tests that send something the app should refuse, and fail if it doesn’t.
| Rule | Happy-path test | Test that catches the rule disappearing |
|---|---|---|
| Email is required | Submit with a valid email, expect success | Submit without an email, expect an error and no order |
| ZIP code is 5 digits | Submit 94107, expect success | Submit "abc" and "1234", expect a validation message |
| One coupon per order | Apply one coupon, expect the discount | Apply a second coupon, expect it to be rejected |
| Max 10 items per line | Order 2, expect success | Order 11, expect it to be refused or capped |
If you already use boundary value analysis, you have most of these tests already: every invalid partition is a rule-still-exists test. If you don’t, start with the rules that cost money or lose data when they vanish.
// Playwright: fails if checkout stops requiring an email.
test('checkout refuses an order without an email', async ({ page }) => {
await page.goto('/checkout');
await fillShippingAddress(page); // everything valid except email
await page.getByLabel('Email').fill('');
await page.getByRole('button', { name: /place order/i }).click();
await expect(page.getByText(/email is required/i)).toBeVisible();
await expect(page).not.toHaveURL(/order-confirmed/);
});04Check after every change, not every release
If the app changes every two weeks, a quarterly regression cycle is testing a version that no longer exists. Tie your checks to whatever knows something changed: a deploy, a merged pull request, a vendor’s release note, a configuration change. When nothing tells you, run the critical ones every day. A small suite that runs daily catches more than a big one that runs quarterly.
05Compare today’s app with yesterday’s
The strongest habit is keeping a record of what the app looked like: its pages, its fields, what each field demanded, and what each page said. When something changes, the difference tells you where to look before any test has a chance to fail or, worse, pass.
- Schema and contract diffs. Compare API schemas (OpenAPI, GraphQL) between releases. A field that went from required to optional is visible in the diff even when every test passes.
- Form and validation snapshots. Record each form field’s
required,maxlength,patternand type. Diff them after every deploy. - Business rules written as sentences. "Free shipping on orders of $50 or more" can change to $75 without a single line of the page’s code changing. Track the text too.
- Configuration and feature flags. Many vendor changes arrive as switched-off features. Record which flags are on.
Then rank what changed by what got weaker. A rule that stopped being enforced deserves more attention than a broken button, because the broken button will announce itself and the missing rule never will.
| Severity | What changed |
|---|---|
| High | Something stopped being enforced: a required flag dropped, a pattern removed, a limit raised, a field or page gone |
| Medium | Something got stricter or new: a tighter limit, a new required field, a new form nobody tests yet |
| Low | Cosmetic: new labels, new titles, a different success code |
06Before you fix a failing test, ask who is wrong
A failing test means one of two things: the test is out of date, or the app has a bug. Auto-fixing assumes it’s always the first. Make someone answer the question before the test changes, and write the answer down in the commit message. That record is usually the first thing anyone asks for after a bad release.
Habits that catch the bugs a green suite misses
- Every rule that costs money or loses data has a test that expects to be refused
- Critical checks run after every deploy or vendor update, and daily otherwise
- API schemas and form validation are diffed between releases
- Business rules stated in page text are tracked as well as code
- Changes are ranked by what got weaker, not by what broke
- No test is "healed" until someone decides whether the test or the app is wrong
Frequently asked questions
Why do tests pass when there is a bug?
Because a test only checks what it was written to check. If a rule is removed, the steps the test follows still succeed, so it passes. Only a test that deliberately sends invalid input, and expects to be stopped, notices the rule is gone.
What is the difference between positive and negative testing?
A positive test sends valid input and expects success. A negative test sends invalid input and expects the software to refuse it with a clear error. Negative tests are what catch removed or weakened validation.
Are self-healing tests a bad idea?
They save time on harmless changes like renamed buttons. The risk is healing a test when the app changed for a bad reason, which hides the bug. Use them with a review step that asks whether the test or the app is wrong.
How often should regression tests run?
After every change that could affect the app: deploys, merged pull requests, vendor updates and configuration changes. If you can’t trigger on changes, run the critical tests daily.
How do vendor SaaS updates affect testing?
They change your app on the vendor’s schedule, often without a version you control. Keep a record of the app’s fields, rules and flags, and compare it after each update to see what changed before relying on test results.
Written by
Vihari Panyam
QualityAIQ