Slack's engineering team ran more than 200 automated agent-driven end-to-end test executions. Playwright MCP failed 0% of runs on a simple thread-reply flow and about 12% on a medium-complexity search flow, against about 12% and 20% for the Playwright CLI. Agent-generated Playwright tests were fastest, at about 3 minutes per run including generation versus 5-8 minutes for MCP and 9-11 for CLI, but their failure rate rose from about 8% to about 48% on the harder flow. Agent-driven runs cost $15-30 each.
"Tests enforce journeys. Agents verify goals." — Sergii Gorbachov, Staff Software Engineer at Slack.
Setup. Three execution models were compared: an agent driving the browser through Playwright MCP, an agent running Playwright CLI commands one step at a time, and an agent writing deterministic Playwright code that it runs and refines until it passes. Agent-driven runs took either detailed natural-language instructions or the same workflow as structured YAML. Two fixed flows were used throughout, thread reply (~15-20 steps) and search discovery (~25-30 steps), with each configuration run 20 times. Agents ran non-interactively in Claude Code (`claude -p`), on Claude Sonnet 4.5 for MCP and CLI and Claude Opus 4.6 for test generation; environments were set up through the Slack Dev API MCP in test workspaces with non-production data.
Reliability. The gap widened as flows got longer. CLI failures came mostly from the execution layer (authentication, navigation timing, session instability) rather than model reasoning. The team's likely explanations are that MCP keeps a live view of the app while CLI rebuilds state from snapshots at each step, and that MCP runs appear to reuse earlier successful interactions, which was not measured. Only about 20% of agent runs followed an identical action sequence while still reaching the same end state.
Cost. Claude Code re-sends the full system prompt and history on every turn, so cost tracks turn count and context growth, not output. On the search-discovery flow, MCP used about 3.5M tokens on Sonnet 4.5, 3.8M on Opus 4.6 and 5.7M on Haiku 4.5, against about 6M for CLI and 7M for code generation, over roughly 40, 40, 60, 85 and 70 turns. CLI needs more turns because it splits each browser interaction into separate action, wait, snapshot and lookup commands, while MCP returns the action and resulting accessibility-tree snapshot in one round trip. MCP also ran concurrently, while CLI was hard to parallelise in Slack's setup. Token use was not optimised; prompt caching, context compaction and fewer snapshots are the named next levers.
Conclusion. Slack places agentic testing as a fourth layer above unit, integration and E2E tests, for exploratory UI behaviour, debugging flaky workflows and reproducing production bugs, while deterministic tests stay the CI foundation. At current cost it suits targeted debugging better than high-frequency CI.