Navigating My Own AI Revolution: Testing My Site with Playwright MCP

At this year’s PSConfEU, Jeffrey Snover — the creator of PowerShell — gave a session called Navigating the AI Revolution. One idea stuck with me: engineering is how we take something that isn’t perfect yet and make it useful anyway. With AI specifically, the answer isn’t to trust it blindly — it’s to build verification layers around it. Validate the output. Trust, but verify.

That hit a nerve, because if I’m honest I’m a bit of a perfectionist and a bit of a control-freak. AI velocity is real and I want it — but handing work to something that’s confidently wrong sometimes goes against every instinct I have. So I’ve been trying to let the reins go a little, and the thing that makes that bearable is the same thing that’s always made me comfortable shipping code: tests. This post is about one testing layer in particular — Playwright MCP — and how it lets me let go without flying blind.

A green build isn’t a working site

Here’s the trap I keep falling into with my own site. It’s a Hugo blog, and when I ask an AI assistant to change something, the feedback loop I want to trust is the build: hugo runs, exits 0, no errors. Done, right?

No. A passing build tells you the site compiled. It tells you nothing about whether the thing actually works in a browser — whether search still returns results, whether the table of contents renders, whether a theme change quietly broke the dark-mode toggle, or whether the console is throwing errors on every page.

That gap is exactly where AI velocity can hurt you. The assistant confidently makes a change, the build stays green, and you happily move on — while something user-facing is silently broken. I needed a verification layer that checks the rendered site, not just the build. That’s Playwright MCP.

Enter Playwright MCP

Model Context Protocol servers give an AI assistant new capabilities. Playwright MCP gives it a real browser. Once it’s wired into Claude Code, Claude can:

• Navigate to any page on my local site
• Click things — menus, buttons, the search box
• Screenshot pages so I can eyeball them
• Read the browser console for JavaScript errors

In other words, it can use my site the way a visitor would, and report back what’s actually happening — not what the build log claims is happening. It’s the difference between “the code compiled” and “I loaded the page and search works.”

A real thing it caught

This isn’t hypothetical. The first time I pointed Playwright MCP at my own site, it found that my search was completely broken — and had been for a while.

The story: my theme had moved on from its old search engine, but my config still asked for the old one. Hugo quietly fell back to a different search backend whose index was never being generated, so the browser was requesting a file that returned a 404. Search returned nothing. And here’s the kicker — every hugo build had been passing the whole time. Nothing in the build log even hinted at it. It took a browser actually clicking the search box and Claude reading the failed request off the console to surface it. A quick config fix later, search worked again.

The second win was bigger. I’d been putting off a scary upgrade — jumping Hugo up sixteen minor versions and moving my theme across a major version bump at the same time. Normally that’s the kind of change I’d poke at nervously for a week. Instead I captured a baseline “known-good” pass with Playwright MCP first, did the upgrade, and ran the exact same checks again to diff before-and-after. Every page, every interactive feature, the console on each — checked in minutes. The upgrade went from “pray and click around” to “here’s the evidence it still works.” That’s the velocity-with-a-safety-net feeling I was after.

This is why I love Pester, too

If you’ve read this blog before, you know I’m a fan of the Pester testing framework. Playwright MCP scratches the exact same itch, and I think that’s why it clicked for me so fast.

Pester lets me change PowerShell quickly because I trust the tests to catch me. It’s a verification layer for my code. Playwright MCP is a verification layer for my rendered site. Same instinct, different layer: let the fast thing be fast, and let a test hold the line so you’re not the one holding your breath. The more I lean on AI, the more I want these layers — not fewer of them. Snover’s point, basically, applied to my little corner of the internet.

How to set it up

This is the setup on Windows, which is what I run. You need Node.js 18+ (it provides npx, which runs the server) — I install it with winget to keep it version-managed:

# 0. Prerequisite (skip if `node -v` already shows 18+)
winget install --id OpenJS.NodeJS --exact --source winget

# 1. Register the Playwright MCP server with Claude Code
claude mcp add playwright -- npx -y @playwright/mcp@latest --browser chromium

# 2. Download the browser build the MCP expects
npx -y @playwright/mcp@latest install-browser chrome-for-testing

# 3. Restart Claude Code so it loads the MCP

# 4. Verify it connected
claude mcp list   # → playwright: ... ✔ Connected

Two gotchas that bit me, so they don’t bite you:

• Use --browser chromium. Without it, the server defaults to your installed Google Chrome — which needs admin rights to install if it’s missing. The bundled Chromium needs neither, and it’s the one flag that makes this “just work” on a locked-down machine.
• Use the MCP’s own install-browser command, not npx playwright install chromium. The generic installer pulls a different Chromium build number than the MCP’s bundled Playwright expects, and you’ll get a Browser "chrome-for-testing" is not installed error that’s genuinely confusing until you know why.

How I actually use it

Day-to-day it’s simple. I start the dev server:

hugo serve --disableFastRender -D

Then I ask Claude to walk through the site — load the home page, a blog post, the taxonomy pages, click the search box, toggle the theme — screenshotting as it goes and checking the console on each page. I’ve written the whole thing down as a small baseline checklist in my repo, so “run the baseline test” is now a single instruction. Before a risky change I run it to capture a known-good state; after the change I run it again and compare. That checklist is my verification layer, and it doubles as living documentation of what “working” means for my site.

The honest caveats

Because nothing is free:

• It’s smoke testing, not a full test suite. Playwright MCP tells me the site renders and behaves correctly in a browser. It’s not a replacement for real unit tests on the code that generates it — it’s the layer on top.
• hugo serve runs in dev mode, which disables things like the PWA, the comment system, CDN and fingerprinting. Those only exist in a production build, so don’t try to validate them locally.
• Search is the sneaky one — the search index is a post-build step, so hugo serve can’t fully exercise it. Ironically, the exact feature it caught as broken is one you have to test against a production-style build. Know that going in.
• Headed vs headless. By default the browser is visible (headed), which is oddly reassuring the first few times — you watch it click around. Add --headless once you trust it and just want it quiet and fast.

Wrapping up

I’m not going to pretend I’ve fully made peace with letting AI drive. But Snover’s framing helped: the move isn’t to slow the AI down, it’s to build the verification layers that let you speed it up safely. Playwright MCP is one of those layers for me — the one that turned “the build passed, I guess it’s fine” into “I watched it work.” For a recovering control-freak, that’s the difference between using AI and trusting it.

If you’ve got a static site and you’re already working with Claude Code, this is a couple of commands and a genuinely eye-opening first run. Point it at your own site and see what a green build has been hiding.

Happy scripting 😊