Bug0 Releases QA Agent to Automate Regression Testing With Cached Playwright Replays

Linearbytes released Bug0 QA Agent, a source-available Playwright library for agentic browser regression testing. The AI agent associates the test instructions expressed in natural language to actions on the user interface, caches those when appropriate for faster CI replays, remapping them to test specifications when the user interface changes. Verification is based on consensus from up to three LLMs and evaluations from vision models.

At one end of the spectrum, legacy approaches to web application testing, while largely automated, do not incorporate AI as a key productivity factor. At the other extremity, innovative teams follow a fully agent-based automated approach to testing where the agent is front and center, handling every browser interaction dynamically.

The legacy approaches often falls prey to known pain points. Test suites commonly depend on implementation details (CSS classes, attributes, data-testid) that may change even when the specifications remain constant, causing unnecessary test rewriting and refactoring. Flaky tests result from the inherently inpredictible nature of the complex, asynchronous and distributed systems underlying web applications (race conditions, latency, dropped calls).

AI-first testing approaches on the other hand suffer from high latency that makes them impractical for large regression suites. LLMs can routinely add several seconds of latency per step, escalate API token costs, and introduce their own brand of flakyness due to the non-determinism of the actions performed. On the bright side, test generation resists well to changes in the implementation; image and video models may provide visual verification with some success.

Bug0 proposes an hybrid approach in which its autonomous agent decouples the initial visual exploration from routine regression execution. Developers write declarative test steps in plain English inside standard Playwright specs.

test("Shopping cart tests", async ({ page }) => {
  await runSteps({
    page,
    test,
    expect,
    userFlow: "Add product to cart",
    steps: [
      { description: "Navigate to https://demo.vercel.store" },
      { description: "Click Acme Circles T-Shirt" },
      { description: "Select color", data: { value: "White" } },
      { description: "Select size", data: { value: "M" } },
      { description: "Add to cart", waitUntil: "My Cart is visible" },
    ],
    assertions: [{ assertion: "Cart shows the Acme Circles T-Shirt" }],
  });
});

During an initial execution, the agent inspects the accessibility tree and screenshots to understand the user interface and perform the requested actions. Once a single-action step executes successfully, the framework compiles the interaction into concrete browser operations and caches them into an in-memory Redis store. On following runs, the suite replays the cached action directly through Playwright, eschewing the latency cost of a new LLM inference.

When implementation details of the user interface change (e.g., button coordinates updated, form hierarchies modified), the replayed Playwright action may as a result throw an execution failure. Rather than failing the entire test run, Bug0 QA Agent detects the break, invalidates the action cache, and re-invokes the multimodal agent loop. Important caveat: caching applies only to single-action steps; complex, multi-action directives re-execute through the agent on every run.

Unlike action steps, assertion verification is not cached. Assertions execute through vision models on every pass. Claude and Gemini independently evaluate every assertion, with a third model breaking ties. Video models allow catching transient UI defects (e.g., auto-dismissing toast alerts, flashing banner errors).

The framework embeds disposable email testing via mailsink and community-built adapters like Mailinator, allowing tests to extract one-time passwords (OTPs) and magic links via natural language prompts like

{{email.otp:Extract the 6-digit code}}

.

Chevrot et al. (Are Autonomous Web Agents Good Testers?) reports that their best-performing autonomous testing architecture (ATA) agreed with human professional testers for only about 60% of test runs, identifying assertion verifiability and grounding (identification of the relevant elements of the target application for the test action) as primary drivers of improvement to scale usage of autonomous testing agents. The paper was published in April 2025 and used GPT-4o, Sonnet, and Gemini. As language models and harnesses regularly improve, so do the efforts to use AI to improve the productivy of testing in all its aspects (generation, self-healing, execution, triage, analytics).

Bug0 QA Agent is released under the Functional Source License, Version 1.1, ALv2 Future License. The Functional Source License is a source-available license (i.e., not open-source), that imposes non-compete restrictions during the first two years of usage. The license came of existence to, in the words of Sentry, its first proponent, “grant freedom without harmful free-riding.”