Basal John

Case study

Making browser automation dependable

How I approach Playwright architecture, locator strategy and CI feedback so that a red build means something and a green one can be trusted.

Context

End-to-end browser suites across web applications, run on every change and nightly, in pipelines shared by several teams. Playwright and, before it, CodeceptJS and Selenium. The recurring problem in every one of these environments is the same: a suite that fails often enough that people stop reading it.

The constraint

A flaky suite is worse than no suite. It costs the run time, it costs the attention of whoever triages it, and it teaches the team that red means retry. Once a team has learnt that, a genuine regression gets the same shrug as a timing race, and the suite has become a tax rather than a control.

So the goal is not more coverage. It is a suite whose verdict people believe.

My role and decision scope

Framework architecture, locator conventions, pipeline design and the review standard that keeps them in place. At LogMeIn I introduced and built out a testing framework on CodeceptJS and Playwright. In the quality engineering function I now lead, I set the Playwright and CI approach. The specifics below are my practice, not a description of any one employer’s internals.

What changed in the system

  • Intent-based locators

    A test finds things the way a person does, by role, label and text, so a layout change is not a failure.

  • Deterministic state

    Every test creates the data it needs, scoped to its worker, and never queries for whatever happens to be latest.

  • Web-first assertions

    Waiting is a property of the assertion, retried until it holds. No fixed sleep encodes a guess about someone else’s machine.

  • Useful failure evidence

    Trace, video and reporter output shaped for a CI log, so triage is one read rather than an afternoon.

A suite whose verdict engineers trust.

Coverage is not what makes a suite believable. These four properties are, and a suite missing any one of them teaches the team that red means retry.

Locators describe intent, not markup. Playwright’s role-, label- and text-based locators tie a test to what a user perceives. A CSS chain tied to div > div:nth-child(3) ties it to a layout that a designer will change on Thursday, and the resulting failure is not a defect report, it is noise. Where a role is genuinely ambiguous, an explicit test id beats a clever selector, but it should be a deliberate contract with the application, added to the code on purpose, not scavenged from the DOM.

Waiting is a property of the assertion, not a step in the test. Playwright’s web-first assertions retry until they hold or the deadline passes. A test with an explicit sleep in it has encoded a guess about someone else’s machine. Every fixed wait I have removed has made a suite both faster and steadier, which is a rare combination and a good signal that the wait was hiding a real synchronisation bug.

State is created, never discovered. The single most common cause of cross-worker flake I have seen is a test that queries for “the latest record” or picks an item from a shared pool. Under parallelism, two workers race for the same row and one of them loses, intermittently, usually only in CI. Tests should create the data they need, scoped to the worker, and be indifferent to what else is running.

A failure has to explain itself in one read. Trace, video and a screenshot on first retry, and reporter output shaped for the terminal a person is actually looking at, which in CI is a plain-text log with no scrollback. This is why I wrote and published playwright-ci-reporter: a Playwright reporter that colourises results, surfaces the slowest tests and timeout warnings, and prints failure detail in a form that reads correctly in a CI log rather than assuming an interactive terminal.

Quarantine is a decision with an owner and a date. Skipping a test is sometimes right. Skipping it without a ticket, an owner and an expiry is how a suite quietly loses its coverage while its badge stays green.

Evidence and outcome

Public evidence

Versions, licences and dates read from the npm registry and the GitHub API on 26 August 2026.

The rest of this is a practice rather than a metric. I have not published flake-rate or runtime reductions, because the figures I have are internal, and each one needs its baseline and its suite size to mean anything. I would rather give you the reasoning and let you judge whether it is sound.

What I would change

I spent too long treating flakiness as a property of individual tests. Most of it was structural: shared fixtures, module-scope test data, and setup that belonged in a worker-scoped fixture instead of a file. Auditing the suite for shared state would have found in an afternoon what test-by-test triage took months to find.

I would also introduce the diagnosis tooling earlier. Traces and a readable reporter feel like polish when the suite is new. They are what determines whether anyone trusts it a year later.

Back to selected work