Every failure is put through this catalogue before the AI is asked anything. Each rule states what it detected, why it matched, how confident it is, and what to check next — the same answer, every time, with no model involved.
20 deterministic rules establish what happened.
AI interprets what it means.
The rules below produce the evidence. The AI never sees the raw test output — only what these rules established, and it must cite that evidence for every claim it makes. That ordering is what keeps the verdict grounded.
8 categories
Across 78 analysed failures
Five concrete checks per rule
2 rules refuse to guess
Refusing to guess is a designed outcome, not a gap.
This test failed with an error that doesn't match any known Playwright pattern. Manual investigation with traces and screenshots is the best next step.
Fired 9 times here — each one routed to a person rather than given a confident cause.
This test fails in almost every run. It is not flaky — it has a real, consistent bug or the test itself is broken.
Fired 6 times here — each one routed to a person rather than given a confident cause.
This test failed with an error that doesn't match any known Playwright pattern. Manual investigation with traces and screenshots is the best next step.
A network call failed during the test. This could be a transient infrastructure issue, rate limiting, or the target service being unavailable.
This test fails in almost every run. It is not flaky — it has a real, consistent bug or the test itself is broken.
net::ERR_ errors indicate the browser could not establish or complete a network connection. This is typically an infrastructure issue rather than an application bug.
The server rejected the request as unauthenticated. The test's authentication state (tokens, cookies, storageState) may have expired or not been set up correctly.
Playwright timed out waiting for a condition — the page, element, or network request did not complete within the configured timeout.
The element may load asynchronously, be conditionally rendered, or have a dynamic selector that changed between test runs.
Playwright located the element in the DOM but it was hidden (display:none, visibility:hidden, zero dimensions, or outside viewport). This is typically a timing issue where the element hasn't finished rendering.
The test assertion failed. The application produced different output than expected — this could be a real application change or a timing issue where the value wasn't settled yet.
When a test flips between PASS and FAIL with no discernible pattern, the most common cause is a race condition between the test's actions and the application's async behavior.
Playwright auto-waits for elements to be stable, visible, enabled, and not covered. The click timed out because one of these conditions was not met within the timeout.
Playwright's strict mode requires locators to resolve to exactly one element. When multiple elements match, it throws this error to prevent accidental interaction with the wrong element.
The browser's page title did not match what the test expected. This can happen after redirects, SPA navigation, or when the title is set asynchronously.
The server accepted the authentication but denied access. The test user has insufficient permissions or the resource has access controls that the test does not satisfy.
The server could not find the requested resource. This could be a broken URL, a removed endpoint, or a resource that doesn't exist in the test data.
The backend server threw an unhandled error. This is a server-side issue that requires investigation of backend logs and service health.
The TCP connection was forcibly closed by the remote host. This is usually a transient network issue or the server terminating the connection unexpectedly.
The browser page or context was closed before Playwright finished interacting with it. This can happen due to memory pressure, crashes, or incorrect test cleanup.
The element was found in the DOM but a framework re-render replaced it before Playwright could complete the action. The element reference became stale.
The test expected specific text content but the element contained different text. This is often a UI update that wasn't reflected in the test.