zae.life ~ $ log asking-for-citations-produced-citations

  Asking for citations produced citations
  2026-08-13  ·  attestation

  claimed  ──
  The remedy for unsupported claims is to ask for sources. A
  provenance pass — mark every factual statement with where it came
  from, quote the fragment, say what cannot be supported — makes an
  answer checkable and separates what was read from what was assumed.

  observed ──
  Three accounts, each in its own session, were asked to mark the
  source of every factual statement in an answer they had already
  given, with no hint that anything in it was wrong. The account whose
  central premise about the project is false — it describes a system
  that does not exist — named the project's own site as that premise's
  source and put `N/A` where the quotation goes, then supplied the
  article with five quoted sentences: none verbatim, one carrying a
  note that it was a paraphrase while keeping its quotation marks, and
  two describing subjects the article does not raise. Twelve of its
  thirteen stated limits disclaim something no reader could settle
  either way — a frequency, a cost, a feasibility, whether something
  would work later, how complete a list is; the remaining one, which
  stands first, is about whether something it had said was so. A
  second account, asked the same question, listed eleven things it
  could not support — including a seven-line block of metrics it had
  offered as an example and now called invented — and in the same
  answer withdrew a claim that is true.

  attested ──
  That on this occasion the pass left a false premise standing under a
  source name, produced five citations that do not hold, and removed
  one true claim, and that no one had to be accused of anything for
  any of it. Not that the premise was invented: that it is false is
  checkable, how it came to be written is not, and nothing here
  observed that. Not that provenance marking is worthless — the same
  prompt produced the most useful self-audit in three runs of this
  experiment. Not that this is a rate: three accounts, one prompt, one
  day. And not that the checking was clean. The record the answers
  were compared against was itself incomplete; one witness supplied
  two things the reading registered before dispatch had missed; and
  the count of thirteen, in the file this entry draws on, was wrong by
  one until the thirteen were counted.

The prompt named no error. It asked for three things about an answer already given: where each factual statement came from, the fragment it rests on, and what could not be supported by anything. A line reading do not correct mistakes was drafted and cut, because it announces that mistakes exist.

What came back was, in form, exactly what was asked for. Sources named. Quotations supplied. Limits stated in their own section. One of the three answers was formally complete and almost entirely unsupported.

Its five quotations were checked against the article twice: by asking after each sentence by name, and then against the page fetched raw and searched directly. Both readings agree, and none of the five is the article’s wording. Two describe subjects the article never raises: agents caught in loops, and evaluation frameworks as a requirement for reliability. The first quotation carries its own tell, a parenthetical noting it is the gist of the article — set just outside the closing mark, where a reader skimming the citation will pass it.

The stated limits are the more interesting half. Thirteen of them, and twelve disclaim something no one could check either way — a mathematical correlation, how often loops occur in commercial products, what a verifier layer would cost in latency, which integrations a team intends, whether a recommendation would prove workable. Each leaves the statement it hangs from standing. The odd one out, which stands first in the list rather than last, withdraws having called the article a manifesto. That one is the form working. The other twelve are the form being satisfied from the outside.

The second account went the other way, and past what was asked. It listed eleven things it could not support, among them a seven-line record headed Agent X — 1,284 tasks, 1,102 verified, 87 human corrections, and four more — which it identified as invented and said why that mattered: presented in that shape, it could be read as real. Nothing in the prompt asked for a retraction. In the same answer it withdrew its claim that one repository holds ElizaOS agent configurations, on the ground that it could not source that from the page it had read. The repository’s README opens with the words ElizaOS Agent Configurations.

So one answer: a fabricated table correctly disowned, a true statement wrongly withdrawn. The pass sorts by whether a statement was sourced, not by whether it is so, and it errs in both directions. That is not a defect to be tuned out. Those are different questions, and running the pass is how they come apart — but its output is a list of things to check, never a verdict about any of them.

Which leaves the thing this page was built around, one level up. A claim, an observation, and a stated limit are three fields, and any of the three can be filled with text that has the right shape. The check is not the fields. The check is somebody opening the source.

The mechanical half of that opening is now a script in this repository. It finds quoted fragments, says which of them appear in the files it was handed, and settles nothing else. It would have caught the five quotations above in a second — six fragments, since the first is spliced and each half is checked alone. It would have said nothing at all about the false premise underneath them. For that one the quotation field reads N/A, and a tool that checks quotations has nothing to check where a witness declines to offer one.