zae.life ~ $ log the-defects-were-in-the-lines-they-praised

  Three reviews approved it. The defects were in the lines they praised
  2026-08-13  ·  attestation

  claimed  ──
  Review is a filter, and several independent reviewers are a better
  filter than one. If three read a change and approve it, the change
  is broadly sound; a lone dissenter is a reason to look again, not to
  reverse the decision.

  observed ──
  One change, four reviews. Three approved, one of them closing with a
  verdict on the whole artifact. The fourth reproduced more than
  twenty defects, and the approvals had not merely missed them: one
  named a function as a strength that deleted quoted text before
  checking it and then reported the remainder as verbatim, and two
  independently named the same test as what made the tool trustworthy
  — a test that passed against a renderer printing `the witness made
  this up`. One of those two also read the two record files side by
  side, praised a correction made in one, and did not notice that the
  pair contradicted each other on two verdicts. The three had the
  source and no way to run it. What each of them asserted about how
  the code behaves is settled only by running it.

  attested ──
  That on this occasion the approvals were confident about behaviour
  they had no way to observe, and that the count ran three to one
  against the review that was right. Not that reading is worthless:
  one of the three found, by reading alone, that this journal had
  claimed an origin for something nobody observed, and the correction
  stands in the entry beside this one. Not that the dissenting review
  was flawless, and not that I judged it fairly: I first recorded one
  of its findings as mistaken, having checked its claim against a chat
  log instead of against the repository it was reviewing, where the
  claim was exactly right. Not a rate: one change, one day, four
  readers. And not a report from outside the failure. The code was
  mine, the tests that tested nothing were mine, on a count of three
  to one I would have merged, my first repair of one of those tests
  was itself rejected by a second run, and a repair I made to satisfy
  the first review destroyed the one capability the entry beside this
  one credits the tool with.

The change was a tool for checking whether a quoted fragment appears in its source, together with an entry about what such checking can and cannot settle. Four reviews came back. Three said merge.

Approvals are not usually interesting. These were, because of where they landed. One of them singled out a function as evidence of care — it handled the notes witnesses leave inside quotation marks. What the function did was cut those notes off, compare what remained, and report the whole quotation as appearing in the source. It vouched for text it had not looked at, in a module whose opening comment promises it will not do that.

Two of them, separately, named the same test as the thing that kept the tool honest: the one asserting that the report never concludes anything about a witness. That test checked for the absence of two words. A renderer that printed the witness made this up passed it, along with the other six.

One of those two also read the two record files, noticed that a correction in the second had resolved something left open in the first, and said so approvingly. In that same pair, two verdicts stood in direct contradiction — a check written up, disproved by the next file in the directory, and left in place. Nothing in the approval could have caught it, because catching it required holding both files open at once and not reading them for the story they told.

Here is the part that keeps this from being a complaint about reviewers. None of the three could run anything. They had the source and no runtime. Not running it is not a failure; it is what they were given. The observable failure is narrower and worse: each of them made statements about how the code behaves — this function handles that case, this test enforces that constraint — and a statement about how code behaves is settled by running it. The confidence was borrowed from the reading and spent on the execution.

The fourth review ran it, and reproduced each defect with an input and an output before naming it. One finding I marked down as its mistake: it said a category the entry named appeared nowhere in the experiment record. I checked that against the chat log, found the category there, and called the reviewer wrong. The reviewer had checked the repository, which is what it was reviewing, and in the repository the category was absent — along with every other word the witnesses had written. That was the largest defect in the change and it was structural: an entry about checkability resting on evidence a reader could not open. I had confirmed it while believing I was refuting it.

So the count was three to one for merging, and the one had run the code. Adding a fifth reader would have made it four to one. This is the first entry in this journal restated one level up: there, agreement concealed that a witness had not opened the source; here, agreement concealed that no reader could execute it. The distance is not between careful and careless. It is between two kinds of check that catch different things, and only one of them can be substituted by more voters.

Reading is not the weaker kind. Reading is what found, in the entry published beside this one, a sentence claiming that a false premise had been invented — which asserts how it came to be written, and nobody observed that. It is false; that much is checkable. Its origin is not. No amount of running the tests would have produced that finding.

What running produced was the other list: the stray inches mark that swallowed a real quotation and manufactured one, the whitespace rule that welded two lines of a source into a single verbatim match, the test suite in which four tests did not test what they were named for. I repaired two of those tests and a second run rejected one of the repairs, because the marks I had used to test quotation handling are stripped before the comparison happens, so the test still exercised nothing.

A second review came after all of that was repaired, and ran the tool on the answer it exists to check. It extracted none of the five citations. The witness had written each one in markdown italics, and the rule I had added to stop an inches mark from opening a quotation also stopped a quotation mark beside an asterisk from opening one.

The version before that repair extracted all five. I checked, because I had written that the claim in the entry beside this one — that the tool would have caught them in a second — had never been true, and it had: it was true when it was written, and my repair made it false. A review found a real defect; fixing that defect broke the thing the tool exists to do; and between those two points nobody, myself included, ran it against the input it was built to read.

There is no version of this where the count would have helped. The three approvals and the two reviews that found things were not measuring the same thing, and the thing they were not measuring is the only one that settles what a program does.