Playwright Testing KPIs and Metrics: What Engineering Leaders Should Actually Track
Summarize With:
Harsh Goel
Here's the uncomfortable truth about Playwright dashboards: the suite can be green and the release can still be risky. Same run, both true.
That's not Playwright's fault. It's what happens when teams treat pass rate as quality, coverage as confidence, and automatic retries as harmless. The dashboard says "ship it." The support queue, two weeks later, disagrees.
This article is the KPI stack we use with client teams to separate false signals from real ones. By the end, you'll know which metrics belong on a leadership dashboard, how to pull each one out of Playwright, and when the right move isn't better measurement, it's pushing checks down into faster unit, API, or contract tests.
But first, the trap that makes all the other numbers lie.
The Trap: Retries Turn Failures Into Green Builds
Retries are automatic reruns. Set retries: 2 in playwright.config.ts and a failed test gets up to three chances to pass. Fail once, pass on the second try and the run still ends green.
Useful? Sometimes. Dangerous? Definitely. Because now your "green" build might be a stack of first-attempt failures wearing a trench coat.
Playwright's reporters see this differently, and the difference matters. The HTML report shows what the run looked like at the end. The JSON reporter keeps the attempt-level data underneath, which tests failed first and recovered on retry.
Your dashboard needs the second one, because the question isn't "did the suite end green?" It's "did green depend on a failed first attempt?"
In the Playwright programs we run, we've found trust in CI starts collapsing around 3-4% sustained flakiness - the same instability showing up run after run, not a one-off. Past that point, developers start treating red builds as noise to work around. The CI gate still exists. It just doesn't gate anything.
That's why the first real KPI is the one that measures exactly this.
The 17 KPIs at a Glance
| KPI | Formula | Healthy | Worry | |
| 1 | Flaky test rate | first-attempt fails that pass on retry / total executions × 100 | <2% | >2% suite-wide; >10% per test = chronic |
| 2 | Defect escape rate | production defects / (production + pre-production defects) × 100 | Trending down | Rising hotfixes with stable pass rate |
| 3 | Test maintenance burden | QA hours on existing tests / total QA hours × 100 | <30% | >50% |
| 4 | Mean-time-to-signal (MTTS) | avg(first failure notification − CI trigger) | <15 min (PR gate) | >30 min (full suite) |
| 5 | Pass rate (final + first-attempt) | final passed / total × 100 | Both stable, close together | Final stable while first-attempt falls |
| 6 | Workflow coverage % | critical workflows covered / total critical workflows × 100 | Grouped by business flow | Stable % while new paths ship uncovered |
| 7 | Parallel worker utilization | busy time / available time × 100 | 60–95% | <60% (waste) or >95% (queueing) |
| 8 | Retry rate | tests with ≥1 retry / total × 100 | Trending flat/down | Rising with stable pass rate |
| 9 | Suite run time | finish − start timestamps | PR <15 min | Full suite >30 min |
| 10 | Cross-browser parity | highest browser failure rate − lowest | <2% delta | >5% risk; >10% block release |
| 11 | E2E suite share | highest browser failure rate − lowest | Balanced with API/unit layers | >1,500 E2E tests + >30 min runs |
Plus six metrics almost nobody tracks (full section below):
| Differentiated metric | Formula | Act when | |
| 12 | Setup tax | setup time / total runtime × 100 | >40%; fix architecture before buying workers |
| 13 | Hard-wait count | count of waitForTimeout in suite | Rising, every sleep is deferred flake |
| 14 | Quarantine dwell time | avg days quarantined before fixed/deleted | >14 days; repair shop became a graveyard |
| 15 | Coverage lag | median days from feature ship to first coverage | >1 sprint; launches run unprotected |
| 16 | Assertion blind-spot rate | escapes in covered flows / total escapes × 100 | High; tests walk paths but check nothing |
| 17 | AI-draft rejection rate | escapes in covered flows / total escapes × 100 | Not falling; AI produces volume, not protection |
Now each one, what it measures, how to pull it from Playwright, and the threshold that should make you act.
1. Flaky Test Rate
Flaky tests fail first and pass on rerun. Google's engineering research puts the practical target below 2% - above that, recovered failures pile up fast enough that nobody trusts a green run anymore.
Flaky rate = tests failing on first attempt but passing on retry / total test executions × 100
The instrumentation is specific in Playwright: use the JSON reporter and count tests with status === "flaky", or passed results where retry > 0.
Track two views separately. Suite-level flake rate tells you whether the release gate is losing credibility. Per-test flake frequency tells you which tests are doing the damage. A test above 10% flake frequency over a rolling 30 days is a chronic flake - quarantine it, give it an owner, fix the cause. Leave it in and it trains your whole team to ignore first failures.
Flake rate tells you the gate stayed credible. It can't tell you whether the gate caught what your customers would have found. Different question, next KPI.
Read Also: How to reduce flaky test automation
2. Defect Escape Rate
This is the one your CFO would pick. It measures the share of defects that customers or production found instead of your tests.
Defect escape rate = production defects / (production defects + pre-production defects) × 100
Playwright can't emit this directly, a browser test only knows what happened before release. You need two datasets speaking the same feature language: pre-release findings from Playwright and QA, and post-release defects from incidents, monitoring, support tickets, and hotfixes.
The first version can be crude. Tag Playwright failures by feature area (test file naming or Playwright tags work fine). Tag production defects in Jira, Linear, PagerDuty, or Sentry to the same areas. Compare. If post-deploy hotfixes are rising while pass rate holds steady, your pass rate has stopped describing customer risk.
One sharper note for regulated teams: in healthcare workflows, the tolerance for critical escapes is effectively zero. A missed defect there isn't a bug ticket, it's an audit gap or a patient-facing error.
3. Test Maintenance Burden
Here's the KPI that explains why coverage looks fine while protection quietly erodes: how much QA capacity goes into keeping existing tests alive versus protecting new product work?
Maintenance burden % = QA hours modifying, fixing, rewriting, or deleting existing tests / total QA engineering hours × 100
We see this constantly in client Playwright programs - sprints consumed by selector updates, flaky-spec triage, and test-data repairs while new customer paths ship thin. The coverage number never moves. The risk does.
Under 30% is healthy, the team has room for new coverage and exploratory work. Above 50%, maintenance is eating more than half the team, and new coverage gets squeezed no matter what the roadmap says. This is also where test automation ROI quietly dies: you're paying automation prices for standing still.
No hour tracking? Use git as a proxy:
git log --diff-filter=M -- tests/
git log --diff-filter=A -- tests/
Compare modifications to additions over 30 days. A ratio above 2:1 isn't the real KPI, but it's an early warning the suite demands more upkeep than expansion.
4. Mean-Time-to-Signal (MTTS)
Even when Playwright catches a failure, does the developer find out while the change is still fresh? That's MTTS.
MTTS = average(first failure notification timestamp − CI trigger timestamp)
Track P50 and P95. P50 is the typical experience. P95 is the slow edge: the runs where a developer waits long enough to context-switch, merge something else, and lose the thread entirely.
Thresholds: a pull-request gate under 15 minutes is healthy, because most developers will actually wait for it. A full suite past 30 minutes is a warning that the architecture is the bottleneck, and the fix is usually test selection, splitting the suite across jobs, or moving checks down the pyramid. Not another parallel worker.
Timestamps come from GitHub Actions annotations (--reporter=github), the Playwright JSON reporter's timing fields, or Azure Playwright Testing run data. The tool matters less than the question: does the failure arrive while it's still cheap to fix?
DORA's 2025 research sharpens the stakes: delivery speed doesn't help when change failure and rework are rising. A slow signal gives broken code more time to travel.
5. Pass Rate - Useful, Just Not the Way You're Using It
Pass rate belongs on the dashboard. As a diagnostic. Not as the headline.
Final pass rate = tests with final status passed / total tests run × 100
The mistake is reading final pass rate without its sibling: first-attempt pass rate. Final answers "did the suite eventually pass?" First-attempt answers "did it pass without Playwright's recovery mechanisms?" A 96% final pass rate looks healthy. Add a 5% flake rate and your clean first-attempt number is closer to 91% - a very different dashboard.
Pull final status, retry count, and flaky status from the JSON reporter and show both numbers side by side. If final holds steady while first-attempt falls, your suite isn't getting better. It's getting better at hiding.
6. Workflow Coverage %
Coverage here means workflow coverage, not code coverage : how many critical user paths, roles, browsers, and integrations have real browser-test protection.
Workflow coverage % = critical workflows covered by Playwright tests / total critical workflows in release scope × 100
The failure mode: the percentage stays stable while new checkout flows, admin actions, or compliance paths ship with thin protection. The denominator moved; the dashboard didn't notice. Pull the denominator from release scope, product analytics, or your risk register, and group coverage by business flow, not just by count. A healthy-looking global number can hide an untested payments path, and only one of those two facts will make it into the incident review.
Coverage also breaks when maintenance burden climbs (see KPI #3), the team is too busy keeping old tests alive to extend protection. These test coverage metrics work as a pair.
7. Parallel Worker Utilization
A capacity KPI, not a quality KPI, but leaders pay for the capacity, so it earns a row.
Utilization % = worker busy time / total available worker time × 100
The practical range is 60-95%. Below 60%, you're paying for idle machines. Above 95%, there's no slack: new jobs queue and MTTS stretches. CI job timing plus Playwright shard timing gives you both inputs; managed services like Azure Playwright Testing expose the same pattern.
Read it next to MTTS, because utilization alone misleads. High utilization with fast signals is efficiency. High utilization with rising MTTS means you need either more capacity or a smarter split between PR tests and full-suite tests. And before buying capacity, ask the cheaper question: should all these tests be running here at all? (KPI #11.)
8. Retry Rate
The trend-line version of the trap from the top of this article.
Retry rate % = tests with one or more retries / total tests run × 100
Every retry means a first attempt failed. Per-test retry rate above 10% over 30 days is chronic flake: quarantine, assign, redesign. But the trend matters more than any single run: rising retries with stable pass rate means the dashboard is getting greener because retries are recovering more failures, not because the suite got more stable.
Group retry counts by test, browser, and environment (all in the JSON reporter). By test finds the unstable spec. By browser finds engine-specific trouble. By environment finds the CI runner or data setup that's actually to blame. Without the split, retry data is just a pile of failures nobody owns.
9. Suite Run Time
Suite run time = finished timestamp − started timestamp
Track PR run time and full-suite run time separately, because they answer different questions: can developers wait before merging, and can regression finish inside the release window?
Keep PR suites under 15 minutes when Playwright gates the merge. Treat 30 minutes as the point where full-suite architecture needs scrutiny. If run time grows while coverage stays flat, the suite is probably carrying duplicate checks, slow setup, or browser tests that belong in cheaper layers.
Read it beside MTTS and flaky rate. A fast suite that hides flake protects nothing. A slow suite that finds real failures may still report too late to matter.
10. Cross-Browser Parity
Playwright's superpower is running the same suite across Chromium, Firefox, and WebKit. This KPI checks whether you're actually using it.
Cross-browser delta = highest browser failure rate − lowest browser failure rate
Pull project-level results from the JSON reporter and compare failure rates for the same workflow and branch. Under 2% is normal variation. Above 5% is browser-specific application risk. Above 10%? That should block release if you have meaningful Safari or Firefox traffic.
This matters double if you migrated from Cypress, where suites historically centered on Chrome. A Chrome-heavy suite can make the app look stable while WebKit quietly breaks layout, uploads, auth, or payments for every Safari user. (Weighing that migration? Our Playwright vs Selenium vs Cypress comparison covers it.)
Track parity by workflow, not just globally - "checkout is 12% worse on WebKit" is actionable; "WebKit is weaker" is a shrug.
11. E2E Suite Share
The last core KPI asks the question the other ten can't: is Playwright carrying too much of the testing load?
E2E share % = end-to-end tests / total automated tests × 100
E2E tests are valuable and expensive; slower to run, costlier to maintain than unit, API, or contract tests. Martin Fowler's test pyramid and Google's "Just Say No to More End-to-End Tests" converge on the same rule: cheaper checks should catch lower-level problems before the browser suite has to. If two services exchange bad data, a contract test should catch it before a browser walks the whole user journey to find it.
When it becomes a leadership problem: past roughly 1,500 E2E tests, past 30-minute full runs, or past 50% maintenance burden. Then the question isn't "is Playwright working?" It's "why is everything being forced through the slowest layer?"
The sequencing rule: if MTTS is under 15 minutes and maintenance burden under 30%, instrument these KPIs first and migrate duplicated checks later. If MTTS is over 30 and maintenance over 60%, shrink the E2E share first - measurement can wait; the fire can't.
Six Playwright Automation Metrics Almost Nobody Tracks (and Should)
The eleven above will put you ahead of most teams. These six put you ahead of most vendors, we track them in client programs because each one catches a failure mode the standard stack misses.
12. Setup Tax
How much of your suite's runtime is spent getting ready to test versus actually testing?
Setup tax % = time in login, navigation, and data setup / total suite runtime × 100
Nobody measures this, and it's routinely the biggest number on the profile. When we rebuilt Centerbase's suite, a cached login strategy alone cut roughly 60% of execution time, the tests weren't slow, the re-authentication before every test was. Pull step-level timing from Playwright traces or wrap setup in fixtures and time them. Above 40% setup tax, fix architecture before buying workers: cached auth state (storageState), API-based data seeding instead of UI-driven setup, shared fixtures.
13. Hard-Wait Count
The crudest metric on this list, and weirdly one of the most predictive.
Hard-wait count = occurrences of waitForTimeout (and fixed sleeps) in the suite
One grep -r "waitForTimeout" tests/ | wc -l and you have it. Playwright auto-waits by design — every hard-coded sleep is a place where someone gave up on understanding a timing issue and papered over it. In our experience, suites with rising hard-wait counts are flake generators wearing a delay costume: the sleep works until the environment slows by 200ms, then it doesn't. Target: zero in new tests, trending down in old ones. This number is also the fastest health check we run when auditing an inherited suite, before we've seen a single dashboard.
14. Quarantine Dwell Time
Everyone says "quarantine flaky tests." Nobody measures what happens next.
Quarantine dwell time = average days a test stays quarantined before being fixed or deleted
A quarantine list is supposed to be a repair shop. Without this metric it becomes a graveyard — tests go in, protection quietly exits the suite, and the coverage number never registers the loss. Healthy dwell time is under 14 days with a named owner per test. If your quarantine list only grows, you're not managing flake; you're archiving it. Pair with a hard cap (e.g., quarantine can't exceed 2% of the suite) so the pressure to fix stays real.
15. Coverage Lag
Workflow coverage (KPI #6) tells you what's covered. Coverage lag tells you how long new features ship unprotected.
Coverage lag = median days between a feature shipping and its first meaningful Playwright coverage
This is the metric that catches the "coverage looked stable while risk grew" failure in the act. Every release, list shipped user-facing changes; record when each got its first real browser test. A lag under one sprint means QA is keeping pace. A lag of three sprints means every launch enjoys weeks of zero regression protection, precisely when it's changing fastest and breaking most. Leadership loves this one because it's a time number, not a testing number.
16. Assertion Blind-Spot Rate
Defect escape rate (KPI #2) counts what escaped. This one asks the more embarrassing question: how many escapes happened in flows that had a passing test?
Blind-spot rate = production defects in flows covered by passing tests / total production defects × 100
The distinction matters because the fixes are opposite. An escape in an uncovered flow means write more tests. An escape in a covered flow means your tests walk the path but don't check the right things: weak assertions, happy-path-only data, missing state validation. In our client audits this regularly explains the "we have 2,000 tests, why is production still breaking?" mystery. If blind-spot rate is high, adding tests makes the suite slower without making it safer.
17. AI-Draft Rejection Rate
New in 2026, and no one else is publishing thresholds for it yet. If your team uses AI to draft tests : Playwright MCP, Copilot, or otherwise track how much of it survives review.
AI-draft rejection rate = AI-drafted tests requiring significant rework at review / total AI-drafted tests × 100
This is the quality gate for AI-assisted authoring. A falling rejection rate means your prompting patterns and generation guidelines are maturing; a stubbornly high one means AI is producing volume, not protection; brittle selectors, assertion-free walkthroughs, duplicated coverage. We treat everything AI-drafted as a first draft behind a human review gate (the workflow from our Playwright MCP vs CLI guide), and this metric tells you whether that gate is a formality or load-bearing. Expect it high in the first month, then trending toward the rejection rate of human-written tests as your generation guidelines codify.
The Dashboard, Reduced to Three Questions
Every KPI above serves one of three leadership questions:
- Did we test the workflows customers actually risk? (coverage, coverage lag, E2E share, cross-browser parity)
- Did green depend on retries? (flaky rate, retry rate, first-attempt pass rate, hard-wait count)
- Did failures arrive in time to act? (MTTS, run time, setup tax, worker utilization)
If your dashboard answers those three, it's doing its job. If it shows a 96% pass rate and nothing else, it's decoration.
This is the KPI stack ThinkSys uses in Playwright automation engagements: assess the current suite, map the gaps to release risk and QA cost, and build measurable release protection - the same discipline behind cutting Centerbase's regression cycle in half (with an 85% flaky-failure reduction, KPI #1 in action). Our Zero Critical Bugs Guarantee exists for teams that want that protection contractual.
Get a free Playwright KPI assessment, we'll show you where your dashboard is lying to you.