We Taught an AI Agent to QA Our Cloud Console — in Plain English

Cerebras is known for speed. Our platform serves the fastest tokens in the industry — but speed isn't only silicon; it's also how fast we can ship the console, playground, and billing flows developers touch every day. And the faster we ship, the louder one question gets: how do we know we didn't break anything?

The traditional answer is brittle machinery. A human writes a script full of CSS selectors, the frontend team renames a div, and overnight half the suite goes red — not because the product broke, but because the test did. That tax scales with how fast you move.

So we asked a different question: what if a test read like a QA engineer's notebook, ran itself like a QA engineer, and fixed itself when the UI drifts? That's console-prompt-tests — an internal framework where every test is a paragraph of plain English, run by an LLM agent driving a real browser. Here's what we learned building it.

A few months in, the payoff is concrete — each stat below reads before → after:

A test is a paragraph, not a script

Here is a complete, real test from our suite. Not pseudocode — this is the actual file the system runs:

A .pt file — a prompt test. No data-testid, no getByRole(), no network-idle waits. Just intent.

There isn't a selector in sight — no data-testid, no getByRole('button', { name: /github/i }), no waiting on a network-idle event that may never fire. Just intent, written the way you'd describe the flow to a teammate.

An agent reads that intent, looks at the actual page in front of it, and finds the GitHub button the way a human would. Rename a class, reshuffle the DOM — the test shrugs. It was never looking at the markup; it was looking at the page.

That single property — tests that describe behavior, not markup — is what makes the rest of the system possible.

How it works: an agent, a browser, and a protocol

Under the hood, three pieces do the work.

1. The prompt runner. A Python harness (run_prompt_with_agent.py) loads a .pt file, hands it to an LLM coding agent, and runs the show: logging, timeouts, a fresh browser session, and a per-test screen recording — so every run leaves a .mov you can watch later.

2. Playwright MCP. The agent doesn't get raw access to the browser. It drives a Playwright instance through the Model Context Protocol (MCP) — a tool interface exposing exactly what a tester needs: navigate, click, type, wait for text, screenshot. The agent plans, MCP executes, the browser responds, the agent looks again. A perception–action loop, not a fixed script.

3. A real, signed-in browser. These tests run against a live deployment — a staging environment or a local Docker stack — with real auth. We handled the annoying parts (GitHub's first-login device-verification email, NextAuth session minting for headless CI), so the agent lands in an authenticated console and gets straight to the flow under test.

The payoff of that loop is resilience: the suite bends with the UI instead of shattering against it.

From test plan to test suite, automatically

Writing dozens of end-to-end tests by hand is exactly the toil we were trying to escape. So we mostly don't.

• From a PDF test plan. generate_tests_from_pdf.py ingests a QA test plan and emits ready-to-run .pt files, one per scenario. The test plan becomes the test suite.
• From the live UI. generate_tests_from_baseline.py captures a DOM baseline of each page and generates sanity tests from what's actually there — so coverage tracks the product as it evolves, and DOM diffs flag when a page changes out from under us.

Today the suite covers dozens of critical user journeys — the CUJ catalog — across the whole console: get-started, playground, API keys, analytics, logs, members, billing, org settings, and the chat app. Each is a row in an auto-generated catalog describing exactly what it checks, tagged with markers (Smoke, Billing, release-gate…) so a 90-second smoke pass or the full regression sweep is a single command.

Knowing what to test on every PR

A test suite is only as good as your confidence that it covers the change in front of you. So we pointed the same intelligence at our own pull requests.

A PR coverage agent cross-references a diff against the CUJ catalog and asks the question a release manager asks: does an existing test already exercise this change — and if not, what would a new test need to add? Because every CUJ carries a plain-English description of what it verifies, the agent can reason about coverage semantically instead of grepping for file paths. Gaps become visible before a release, not after an incident.

This closes a loop that most test suites leave open. It's not enough to run the tests you have; you need to know when the tests you have aren't enough.

The part we're most proud of: tests that heal themselves

Here's the reality of any large E2E suite: overnight, some tests go red. The expensive part isn't running the suite — it's the morning triage. Real bug? Flaky network blip? Did the UI just move a button? Or did the infrastructure fall over? So we built the triage into the system.

Every night, a batch runs and a router skill — heal-batch — reads the results and classifies each failure by type: UI/flow drift, flaky, infrastructure/session death, a real product bug, or a gap in the runner's own capabilities. Each type routes to a dedicated handler. For the most common case — the UI drifted and a stale .pt step no longer matches — a debugging agent reconstructs the failure from the screen recording, screenshots, and logs, then launches the UI and reproduces the flow live to confirm its hypothesis before proposing a fix.

The guardrail is what makes this trustworthy, and it's worth stating precisely:

The result is a suite that absorbs the routine churn of a fast-moving frontend on its own, and escalates to an engineer only for the failures that actually deserve one: real bugs.

From a test suite to a release gate

Running and repairing a suite is one thing; deciding whether a whole release is safe to ship is another. Before the journeys even run, the pipeline reads every diff and scores its risk — flagging the patterns that actually cause outages: database migrations, feature-flag flips, ordering dependencies, breaking API changes. A deterministic floor the model can only raise, never lower, keeps the one risky PR in thirty from being quietly waved through.

Once the tests finish, the pipeline drafts the release doc — PRs grouped into plain-English features, risk and test results attached — and ends with a recommendation: Go, No-Go, or Go-with-checks. The write-up that used to eat a Monday morning is now a ~20-minute run, and nothing ships that hasn't been risk-scored, tested, and proven green in a real environment.

Skills on a runner that scales out

None of this is one monolithic program. Every capability — score risk, run a journey, classify a failure, heal a stale test, publish the notes — is an independent skill, and a single containerized runner executes any of them. Adding a new capability means adding a skill, not rewriting the pipeline.

That design is what lets it scale. Each run is a pod with the skill runner inside it; the skills chain together, and that chain runs across many pull requests at once, each in its own pod. More features means more skills; more work means more pods, and EKS scales out to fit. The pods use Pod Identity rather than static keys — each can write exactly one S3 bucket and has no path to production.

The pipeline is cloud-native: runs on EKS, orchestrated by an AWS Step Function, where the light stages — resolving which commits are in a release, recording a run — are Lambdas measured in seconds, while the heavy stages that need a real browser or run long — run the journeys, classify failures, publish, open heal PRs — run as Kubernetes Jobs. Outputs land in S3 and DynamoDB.

- One containerized runner, many pods

Is it safe to let AI do this?

It's the right question, and the honest answer starts with the obvious: a language model isn't deterministic — you can't ship customer software on "the model probably got it right." So we drew a hard line between what the AI decides and what it proposes. The rules that gate a release are fixed and apply the same way every time: the risk floor the model can only raise, and a pass that only a real, live re-run can set. What the AI does is propose — it drafts the risk rationale, writes the test, suggests the fix, drafts the notes. It never merges, never writes to production, and holds no standing credentials, and its output is structured and checked before anything downstream acts on it.

On top of that sit the ordinary guardrails you'd expect: least-privilege access — Pod Identity rather than static keys, each pod scoped to a single namespace and one S3 bucket with no path to production — plus sandboxed per-run containers, locked-down egress, secrets in a vault, and an audit trail for everything. And the one rule that never changes: a human approves every merge.

What we learned

1. Describe intent, not implementation. The moment your tests stop referencing markup, the single largest source of E2E flakiness disappears. An agent that perceives the page is fundamentally more robust than a script that addresses it.
2. Let the agent see, don't just let it act. The screen recording and screenshots aren't just for debugging — they're the ground truth that keeps a self-healing loop honest. A fix proposed from evidence and confirmed live is a fix you can trust.
3. Determinism where it counts. LLMs are the judgment layer; they should never be the apply layer. Keeping "propose" (model) and "apply/verify" (mechanical) strictly separate is what lets us automate healing without ever automating a hallucinated change into main.
4. Point the intelligence at the whole loop. The biggest wins weren't in running tests — they were in generating them, scoping them to each PR, and repairing them. The test run is the easy part.

Where this goes next

This is a preview of what QA looks like when fast models are cheap enough to live inside the development loop, not beside it. When a test is a paragraph, anyone can write one. When an agent can see the page, the suite stops fighting the frontend. And when it can triage and heal itself, engineers spend their days on real bugs instead of red builds.

And it isn't staying on the Console. We're already rolling the same framework out to our API Gateway and API Server, so the platform's core request path gets the same paragraph-sized, self-healing coverage.

Speed has always been the Cerebras story. Now it includes the speed of knowing we shipped it right — powered by the same fast models we serve on our platform.

Interested in building agentic workflows on the fastest inference in the world?