Promptwatch — a Regression
Harness for Prompts

// the idea

Prompts change. Nobody tracks why.

Anyone running a model in production has a prompt somewhere that has been edited forty times by six people. Each edit fixed something. Nobody can say what the other thirty-nine fixed, or whether edit forty quietly undid three of them. The prompt is the most load-bearing piece of text in the system and it is treated like a Slack message.

Code solved this problem decades ago. You diff the change, you run the tests, you ship. There is no reason a prompt should get less rigour than a CSS file, and yet the standard workflow is still "try it in the playground, looks fine, paste it in."

Promptwatch is the smallest tool that closes that gap. Paste the prompt you run today and the edit you want to ship. It shows you every word that moved and classifies what kind of change each one is. Then you give it a few test inputs and a handful of rules the answer must obey, and it runs both prompts against a real model side by side. A rule that passed with the old prompt and fails with the new one is a regression, and it gets flagged before it reaches a customer.

A concrete example

Think of baseline and candidate as before and after. Say you run a small assistant that summarises meeting notes. The prompt you've used for months is the baseline:

Summarise the following meeting notes for the team. Be clear and concise.

Notes: {{input}}

Someone complains the summaries are long and keep missing the action items, so you rewrite it. The rewrite is the candidate:

Summarise the following meeting notes for the team.
- Keep it under 150 words.
- End with a bulleted list of action items, each with an owner.
- Do not include anything that wasn't in the notes.

Notes: {{input}}

Same job, different instructions. The test inputs are two or three real sets of meeting notes. The assertions are rules like max words 150 and contains "action items". Promptwatch runs the same notes through both prompts and checks the rules against both sets of answers.

What you're looking for is a surprise. Maybe B now produces action items — two assertions newly passing — but one set of notes that A handled fine now comes back with an invented owner for a task. That is a regression, and it is the thing this tool exists to catch: before you switch the prompt over, not a week later when someone asks why Dave owns a task Dave never agreed to.

The distinction it is trying to make is between prompt engineering — the craft of writing a good prompt once — and prompt operations: the discipline of keeping it good as it changes. The second is where the real cost lives, and almost nobody has tooling for it.

// what it does

Six panels, one loop

A · baseline     7/12   avg 612 ms
B · candidate   12/12   avg 598 ms
regressions     0      fixes  5
▶ B improves on A — 5 assertions newly passing, nothing broken

// how it was built

Build notes

The whole thing is one HTML file, no framework, no build step, under 700 lines including the CSS. That was a constraint, not an accident: a tool meant to be dropped onto any static host should not need a toolchain. Four decisions are worth recording.

The diff is a plain LCS on word tokens

Prompts are short, so there is no need for Myers' algorithm or any of the clever linear-space variants. The text is tokenised into words with their trailing whitespace attached, a longest-common-subsequence table is filled in an Int32Array, and the edit script is walked back out and merged into runs. Keeping whitespace attached to the preceding word means the rendered diff preserves line breaks and paragraph structure for free. It's capped at 2,500 tokens a side, which is already a prompt that needs a different kind of help.

// tokens keep their trailing whitespace so layout survives the diff
function tokenize(s){ return s.match(/\S+\s*|\s+/g) || []; }

// LCS table, then walk it back out into =, +, - ops
for (let i = n-1; i >= 0; i--)
  for (let j = m-1; j >= 0; j--)
    dp[i*W+j] = an[i] === bn[j]
      ? dp[(i+1)*W+j+1] + 1
      : Math.max(dp[(i+1)*W+j], dp[i*W+j+1]);

The change summary is heuristics, and that's the point

It would have been easy to send both prompts to a model and ask it to describe the difference. It would also have been slower, cost money, needed a key before the user had seen anything useful, and given a different answer every time. Instead the classifier is a short ordered list of regular expressions: sentences with never / must / do not / under N are constraints, sentences mentioning JSON / markdown / bullet / plain text are format rules, you are / act as is persona, and so on. It is wrong sometimes. It is also instant, deterministic, and legible — you can read the rules in the source and disagree with them, which is exactly the property a review tool should have.

API calls go straight from the browser

There is no server, so the page talks to the provider directly. For OpenAI that is an ordinary fetch with a bearer token. For Anthropic, browser-originated calls are refused by default; they go through only if the request carries the anthropic-dangerous-direct-browser-access: true header, which is the provider's way of making you say out loud that you understand the key is in the client. The key lives in page memory, and in localStorage only if the user ticks a box. The privacy note under the run bar states exactly where it goes and recommends a key with a spend limit. That note is doing real work: a tool that asks for API keys has to earn them.

Demo mode is a fake model that obeys instructions

The hardest problem with a tool like this isn't the code; it's that a visitor without a key bounces before they understand what it does. So the demo provider is a few dozen lines of deterministic JavaScript that reads the prompt for the obvious instructions — don't promise refunds, don't name competitors, stay under a word limit, no markdown — and generates replies that obey them when present and violate them when absent. It isn't an LLM and doesn't pretend to be. It is enough that the example run produces a baseline that over-promises a refund in bullet points and a candidate that doesn't, which makes the whole regression idea click in one screen.

// what this demonstrates

Why this matters

The useful shift in AI work over the last year has been from can the model do it to can we keep it doing it. Evals, observability and change control are where production teams are actually spending their time, and the vocabulary of that work — baseline, candidate, assertion, regression — is borrowed straight from software engineering because the problem is the same one.

Promptwatch is a deliberately small statement of that view. It doesn't try to be a platform. It takes the one moment where most prompt regressions are born — someone edits a prompt and ships it on vibes — and puts a diff and a test run in the way. If it makes one person run three test cases before pasting into production, it has done its job.

It also sits naturally next to earlier work here: the agent primer is about the boundary where a model asks to act and your code decides; the RAG red-team is about what goes wrong when you trust inputs you didn't check. This is the same instinct pointed at the prompt itself.

Where it could go next