← All records

No. 02 — AI UX Auditor

An agent that finds UX and accessibility bugs on a live site, fixes them, and proves each fix worked.

The problem

Audits are manual and late, scanners skip menus and modals, and nobody checks that a fix actually fixed anything.

What I built

A Playwright agent that walks real user flows, opens pop-ups, pins issues, applies fixes and re-checks them before exporting a patch.

The result

54 of 54 auto-fixes verified on the demo site, and the accessibility score went from 17 to 50 in one pass.

RoleSolo: design, engineering, evaluation
StackNext.js 15, Playwright, Claude vision, SSE, TypeScript
Year2026

01See it work

Redline auditing a demo page: numbered red pins mark issues across the page, with the score and issue list on the right. Demo video coming soon
01 The audit view: every issue is pinned on the page, and the score updates as fixes land.

Video coming soon: 2-minute walkthrough: paste a URL, watch the agent audit, fix and export.

54/54
auto-fixes verified after re-render
17 → 50
accessibility score on the demo page
~13s
to audit 4 views and find 70 issues
22% → 1%
false visual change after row alignment

02Why it matters

Accessibility audits happen at the end of a project, by hand, from a checklist. Automated scanners help, but they only look at the page as it loads. The worst bugs are in menus, modals and checkout steps that nobody opens. And when a tool suggests a fix, you're left to find out on your own whether it worked or broke the layout.

03The idea

I treated auditing as an agent loop with a verifier: detect, fix, re-render, re-check. A fix only counts if the agent can prove it in a real browser, so every claim in the report has evidence behind it.

04How it works

  1. DeterministicPaste a URLStart from any live page
  2. DeterministicWalk & openPlaywright follows sign-up, login and checkout flows and clicks open menus and modals
  3. AI modelDetectRule engine plus an optional Claude vision pass
  4. AI modelFixApplies a candidate fix in a sandboxed browser
  5. Verification gateRe-checkRe-renders the page, re-runs the rules and diffs the layout
  6. OutputExportJS/CSS patch, HTML report and issue list
Deterministic AI model Verification gate Output

05Decisions that mattered

A

Prove it, don't just suggest it

Fixes are applied and re-tested in Redline's own browser. Only the verified ones end up in the exported patch, which reproduces 54 of 54 fixes on its own.

B

Open the hidden layers

The agent finds likely triggers, opens each pop-up and checks it as its own view: focus moves in, Esc closes it, and focus returns afterwards.

C

A diff that understands layout

A naive pixel diff reported a 22% change for a one-line fix. Aligning page rows with banded Needleman–Wunsch first cut that false change to about 1%.

D

Fixes that don't move the page

Tap-target fixes use padding plus a negative margin, so surrounding content moves 0px. That's asserted in tests, because reflowing a client's layout is a bug too.

06How I know it works

A smoke suite runs against a seeded demo site: 16 known rule types, 2 pages and 2 pop-ups. A real university site exposed a contrast bug (white text over a photo was 'fixed' to grey), which is now a regression test.

16/16 seeded issue types are caught. Next is a labeled set of ~50 real screens, scored against axe-core for precision and recall.

07What's next

  1. "Open PR" mode: map each fix back to the source component
  2. Labeled eval set with precision/recall against axe-core
Next recordBillTrace