Automating QA with AI agents
Updated Jan 2026 • 5 min read
QA has always been a bottleneck. You build a feature, deploy it to staging, and then check it against a spreadsheet of 50 manual steps. If you update a button, you have to do it all again.
We solved this by turning LLMs into "Virtual Users." These aren't just unit tests; they are agents given a persona (e.g., "Impatient user with a slow network") and a goal ("Sign up and create a story").
This playbook covers
- Agentic QA workflows: using LLM agents as virtual users for UI testing
- Automated regression testing: repeatable safety checks before merges
- Edge-case discovery: inputs and behavior humans typically miss
- Structured prompts + UI context: getting reliable actions from models
Structured prompts & UI context
The secret isn't a complex framework; it's providing the AI with the right DOM context. We dump a simplified version of the current HTML accessibility tree to the model and ask it: "Based on this screen, what is the CSS selector to click to achieve X?"
This turns vague “test the app” requests into structured, repeatable LLM testing runs. Each run produces an action plan, a set of selectors, and a trace you can compare across releases—making UI regression tests far easier to maintain than brittle scripts.
Testing edge cases
AI excels at trying things humans forget:
- Input validation: Typing emojis into phone number fields.
- Race conditions: Clicking “Submit” 5 times rapidly.
- Navigation: Using the Back button mid-checkout.
- State handling: Refreshing the page during a multi-step flow.
- Network reality: Slow connections and timeouts during key actions.
The result
We now run a "Safety Check" suite before every merge. It costs ~$0.50 per run in diverse API tokens and catches 80% of UI regressions that unit tests miss. It allows a 1-person team to ship with the confidence of a 5-person QA department.
The biggest win isn’t just speed—it’s consistency. When the same automated QA suite runs on every release, you catch regressions early, keep velocity high, and avoid the hidden cost of “small” UI bugs compounding into support load.
Key takeaway
Treat LLMs like virtual users, not magic. With structured prompts and solid UI context, AI QA automation becomes repeatable regression coverage that catches real-world behavior before customers do.