This morning, our AI agents reported a result we did not want: a promising repair for damaged memory healed 0 of 10 cases. Not five. Not two. Zero.

We published it. Here is the entry, verbatim, from our public research log:

An honest zero, published verbatim — claim VERIFY.MEMORY.ERODED_MINT_RETEACH_RECOVERY.01, outcome Failed.

That sounds like basic scientific hygiene. In an autonomous lab, it is the whole game and it is exactly what Pawl is built to make routine.

Pawl in one card — claims, a registry as ground truth, and honest negatives.
What you get: frozen gates, honest negatives, queryable memory, multi-agent safety.

AI agents can now propose hypotheses, change code, run experiments on real hardware overnight, and write fluent, confident reports about what happened. The bottleneck is no longer whether they can do the work. It is whether anyone can believe the morning report. An agent that wants to please you will find a metric that improved. Five agents can each produce locally plausible work while collectively turning the lab into very expensive fiction.

We did not need another intelligent agent. We needed an institution encoded in software and then we extracted it so other teams can adopt the same ratchet without copying a whole research repo. That institution is Pawl.

Where Pawl was forged

Hylaean is our attempt to build an AI system that behaves more like a physical field than a language model. Knowledge is geometry. A question is a disturbance. An answer is a settled state. That ambition creates a brutal measurement problem: there is no single training-loss number that tells you whether tonight went well. On any given day there are hundreds of open questions like does memory survive a restart? does the system refuse honestly when it does not know? ... and each needs its own experiment, its own pass criteria, its own controls. A false "it works!" becomes a foundation other work builds on, and tearing it out later costs weeks.

Pawl is the verification discipline we built underneath that lab, now open-sourced after 590 claims in 18 days (404 verified, 167 failed, 17 planned; 4–5 parallel agent sessions, zero corrupted records). The spine of every claim in Pawl:

The life of every claim — register, freeze, run, verify or fail, remember, publish.

Rule 1 — Every claim is a sealed bet

Before an experiment runs, its success criteria are written down and frozen — an envelope sealed in a drawer. Then the experiment runs, on real hardware, at full scale. Then the envelope is opened. There are exactly two outcomes: Verified or Failed. There is no "promising direction," no reinterpreting the data into a softer story, no moving goalposts after seeing the numbers. This morning's zero survived precisely because the bar it missed had been fixed the evening before.

The sealed bet — criteria frozen before the run, an outcome either way, a public entry either way.
Tech inside — three commands (register, init_folder, run) move a claim from Planned to Verified or Failed.

Rule 2 — Failure becomes memory

A failed claim does not disappear. It must name the exact missing capability, this morning that name was "the protective mechanism is a one-way valve, not a pump" and it permanently changes the research graph: every planned experiment that depended on the failed idea is immediately re-decided. Redirected onto the named missing piece, re-scoped, or closed.

This matters more with agents than with humans, because agent swarms forget faster than people do, every session starts blank. Without institutional memory, agents rediscover old dead ends with fresh enthusiasm. So before any agent registers a new experiment, it must ask the memory index: has anyone tried this? What happened? Is there already a plan? It answers in a fraction of a second, across every experiment ever run and every strategy paper ever written, with the connections between them.

Failure becomes memory — experiments and plans, connected; failures stay visible to everything that depended on them.
Tech inside — a BM25 index over claims and plans; the mandatory duplicate check before any new claim.

Rule 3 — The lab coordinates itself

Five agent sessions. Three GPUs. One shared research ledger. That sentence sounds mundane until you watch it fail: two agents grabbing the same GPU and crashing each other's experiments, one overwriting the ledger page another is writing, a dead agent leaving an experiment half-recorded forever. A door lock does not sound impressive either until five researchers all believe GPU 2 belongs to them.

So Pawl includes the boring articles. A booking system for the machines: an agent reserves a GPU before running anything heavy, and two agents can share one machine when the memory budget fits. A single ledger with a lock: one writer at a time, every write completed in full or not at all. And a crash rule: if an agent dies mid-experiment, its locks are detected as stale and released, and another agent resumes exactly where the work stopped.

The lab coordinates itself — five sessions, three GPUs, one host, every result through the locked ledger.
Tech inside — flock + atomic replace on the ledger, per-claim run locks with stale-PID takeover.
Tech inside — why speed is an honesty feature: faster runs let agents afford the full-scale check.

Rule 4 — Uncomfortable results go on the record

Everything above would still allow a subtle kind of dishonesty: quietly not mentioning the experiments that failed. So the lab publishes its log. At hylaean.ai/status an entry appears when an agent session finishes — generated from the experiment's own recorded results, whether it is a verified capability or an honest negative, so the log cannot drift away from the data. A human still decides when the entry goes live. But nobody rewrites the outcome into a happier story.

18 days, 590 claims, zero corrupted records — 29% of completed claims are public failures.

Look at the failures number. 167 of ~571 completed claims — just over 29 percent — are public failures. Good. A lab with no visible failures may be excellent. Or merely excellent at editing the story.

The ledger's honest shape — bars proportional to real counts, and the lab is proud of the red one.
Tech inside — only no-teach runs count; mandatory negative controls; "a benchmark won by a smarter decoder is invalid."
Tech inside — the log is generated from ledger rows, idempotent, publish opt-in.

The obvious ancestor

If this pattern — agents running a research loop overnight, humans programming the organization instead of the experiments — sounds familiar, it should. Andrej Karpathy's open-source autoresearch (March 2026) distilled it beautifully: a small, real language-model training setup, one file the agent may edit, roughly five minutes of training per attempt, one number that decides — improved? keep : discard.

Karpathy showed the elegance of programming the research organization instead of scripting every experiment. We share that thesis — and built Pawl to generalize it. Where autoresearch discards a failed attempt and moves on, Pawl publishes it, names what was missing, and lets it permanently block its dead end. Where one agent edits one file, five sessions negotiate locks and machine bookings. Where one number decides, a sealed envelope of criteria decides.

Same autonomy thesis, different accountability — discard versus a ledger-backed outcome, published either way.

Why an agentic-commerce lab built this

Pawl is maintained by Agentic Commerce Lab — Shopware's open initiative for agent-driven commerce.

Pawl is maintained by Agentic Commerce Lab — Shopware's open initiative for the protocols and tooling of AI-driven commerce (agentic-commerce.dev). The Lab builds open standards for agent-driven discovery, transactions and order management — UCP, AXP, WebMCP, sales-agent harnesses and storefront bridges — alongside partners across the commerce and AI industry.

That work has the same shape as the research lab above: many parallel agent sessions editing real code, where a false "it works" is expensive and in commerce, a wrong "checkout succeeded" or "payment verified" is not just costly, it is a trust breach. Agentic commerce only works if every capability an agent relies on is actually implemented and provable, not merely reported. That is precisely the guarantee Pawl encodes: frozen gates, a registry the runner alone can write to, and failures that stay on the record. Verification is the substrate agent-driven commerce has to stand on, so the Lab open-sourced the discipline it runs on internally — usable in any repo, commerce or not.

Adopt Pawl in five minutes

Pawl ships as a copyable folder — no install step, no dependencies. Bootstrap verify.json and a registry, register your first claim, freeze its gates, run the official pass. Rules and skills for Cursor agents are included. A worked demo project shows one Verified claim and one honest-negative Failed claim end to end.

Get started — repo, lab, case-study log, and the autoresearch ancestor.

What still breaks

Pawl has honest gaps of its own. When many agents finish at once, they queue at the ledger's lock. The memory index is keyword-based. And we gave up something real: autoresearch's beautiful simplicity. We traded it for a framework whose word you can check.

Autonomy scales execution. Accountability scales truth. The night can belong to the agents. The morning report still has to survive scrutiny.

So here is the question we would ask any team heading into this decade: what evidence would you require before trusting research, or shipping features, produced entirely by agents?

Pawl · github.com/agentic-commerce-lab/pawl · agentic-commerce.dev · case study: hylaean.ai · baseline: karpathy/autoresearch

#AgenticAI #AICodingAgents #AgentDrivenDevelopment #Verification #OpenSource