Hardening an AI agent’s
access to WhatsApp.

The approval boundary

The test passed.
Production failed.

The promise

Approval came after reading.

The surprise

Old input arrived as if it were new.

For much of the project, confirmation was a y/N prompt. In real use, I noticed that a y typed during container startup was still in the terminal buffer when the prompt appeared. It happened to decline because the read got something unrecognised. The safety was luck.

The claim was “a human read this exact text and said yes.” A keystroke made before the text appeared could not support that claim.

What I tried

My first fix discarded pending input just before printing the prompt. I tested it through a real pseudo-terminal, typed ahead, and confirmed the input was discarded. The test passed. The deployment was still broken: a live run accepted a pre-typed key.

The launcher used docker run -it. There were at least two buffers: the Mac’s terminal and the container’s pseudo-terminal. My flush emptied the latter. The runtime relayed keystrokes from the former onto the freshly emptied queue. My harness wrote into the buffer the child read. Production had another, and I had never asked.

“Flush harder” could not solve it. Buffers were a property of how the program was launched. Any draining fix raced a relay it did not control and could expire under tmux or ssh.

What changed

I changed the approval claim. Each draft printed a random code generated after rendering the message. Approval required typing it back.

  | 7pm works for me!
========================================
To approve, type this code:  7f3a
Anything else declines.

Pre-typed input could not reliably predict a code that did not yet exist. A claim about timing became a claim about arithmetic. With four hex characters, a blind guess still succeeded about 1 in 65,536 times, at the cost of a visible decline per failed attempt. The code did not prove that a human understood the draft.

A passing test showed the property held in the environment I built. Whether that environment resembled production was a separate question—and the one I had not asked.

The evidence boundary

A positive control passed
for the wrong reason.

The result

The tool refused.

The missing evidence

The intended input never arrived.

I tested whether drafting refused a chat not authorised for writing. The assistant’s notes already told it to report suspicious instructions. An assistant-level refusal would prove nothing about the tool.

I wrote the pass condition against the audit log before running the test: a tool refusal counted; an assistant declining without calling the tool did not. I also ran a positive control—a direct request with the intended input, bypassing the assistant, to establish that the refusal path was exercised.

It returned ACCESS_DENIED. It looked like a pass.

What the log exposed

The command reading the chat identifier used bash syntax in zsh. The read failed, the variable was empty, and the tool refused an empty string. The refusal was real. It was a refusal of nothing.

I caught it because the audit log’s refused-chat column was blank.

A positive control was only positive if I verified its input arrived. I had built a control against this class of error, then committed the error inside it.

The coverage boundary

I removed the claim.
I kept the code.

The temptation

Keep a reassuring coverage claim.

The honest choice

Remove a claim that no longer mattered.

I marked security-relevant lines and maintained a separate list of mutations. Mutation testing deliberately breaks a guard in a throwaway copy and requires the tests to detect that breakage. The runner refused a clean result while a marked guard had no mutation.

The lists were handwritten and independent. Deriving guards from mutations would make “every guard has a mutation” true by construction: a list reporting on itself. Hand-syncing was the cost; the cost was the point.

When the claim stopped being true

After the approval code replaced the flush, the flush still tidied the prompt. Deleting it no longer broke a test: approval no longer depended on it.

I could delete the code, retain a security marker with a mutation that could not fail, or retain the code without the marker. The middle option was the trap. It would preserve the appearance of coverage while abandoning its meaning.

I removed the marker, kept the code, and explained why in a comment.

When a control was superseded, I had to decide whether it was still load-bearing. If not, it kept neither its security claim nor its test—whether or not the code stayed.

The environment boundary

Three environments
that were not production.

The pattern

My test machine differed from deployment.

The lesson

Environment was part of the evidence.

The same failure appeared in unrelated places:

• A generated report differed because a YAML library existed on the development machine but not in the shipped container. I made report generation refuse to run outside the canonical environment.
• The typeahead harness omitted an input buffer present in deployment.
• I verified an installer rollback on Linux, although the bug was macOS-specific. macOS cp copied file flags: a backup of a locked file was itself locked and could not be cleaned up. GNU cp had no such flags. My Linux test could not find that failure.

The installer also showed why exit codes were insufficient. A guard against a half-finished install deleted a settings file, then announced that nothing had been installed and re-running was safe. The reassurance ignored the file it had destroyed. I caught it by checking state afterwards.

When a fix depended on platform behaviour, the test had to run on that platform. “It passed” from anywhere else was a weaker sentence than it looked.