← blog

Multi-agent debate for diagnostics

Jul 24, 2026 · ~14 min read

multi-agent langgraph architecture

Picture a familiar failure mode. A team runs a sequence of batches — experiments, jobs, deployments, production runs. The first few look fine. Somewhere around batch five or six, yield collapses, latency spikes, or a metric that used to be stable goes wrong. The people responsible already have the past runs sitting in PDFs, spreadsheets, and CSVs. What they need is not another chat box that “summarizes the folder.” They need a system that can turn that mess into structured evidence, hold competing explanations accountable to the numbers, and produce a recommendation they can actually act on.

I built that kind of system at Lemnisca for fermentation diagnostics: scientists uploaded run files, asked why a late batch failed, and got a structured report from a panel of specialist agents that had debated hypotheses against the data. This post is not a product tour of that domain. It is a writeup of the architecture choices that made the system trustworthy enough to ship — and why those choices transfer to any diagnostic workflow with the same shape: messy multi-run inputs, uncertain root cause, and a need for auditable reasoning.

The shape of the problem

Most “ask an LLM about my data” products collapse three different jobs into one prompt:

  1. Turn heterogeneous files into reliable numbers.
  2. Separate what was measured from what was inferred.
  3. Argue among plausible causes until one survives contact with the evidence.

Those jobs have different failure modes. Extraction invents cells. Analysis mixes measured yields with literature constants and then forgets which was which. Diagnosis becomes a confident monologue because nothing in the loop is forced to disagree. If you are building for manufacturing QA, clinical trial protocol review, incident postmortems, or batch experiment debugging, you hit the same three jobs. Treating them as one agent is how you get fluent wrong answers.

So I split the system into two phases with a hard boundary between them.

Two phases, disk as the contract

upload + question
        │
        ▼
┌─────────────────────┐
│ Phase 1: extract &  │
│ analyze             │  → data_packages/<id>/
└─────────────────────┘
        │  (read from disk only)
        ▼
┌─────────────────────┐
│ Phase 2: diagnostic │
│ panel / debate      │  → report.json + debate.json
└─────────────────────┘

Phase 1 is one-shot per upload: triage files, extract tables, run analysis tiers, audit, chart, package. Phase 2 is one-shot per debate: read the package, fan out specialists, moderate cross-examination, vote, synthesize, write the report.

The Phase 1 analysis agents are not stuck in pure text. They get a real tool surface: a full Python environment they can drive via execute_python (pandas, numpy, whatever the computation needs), and Gemini CLI as a shell process for PDF/DOCX extraction when the file is not already tabular. That matters. Diagnostic quality upstream is limited by whether the agent can actually compute and parse, not just narrate. Every Python invocation is traced — code, stdout, stderr, duration — so you can later see whether a bad number came from bad math or bad input.

The important part is the handoff. There is no in-memory state shared between phases. Everything Phase 2 needs lives on disk under a single bundle directory. That sounds boring until you need to re-run only the debate after fixing a specialist prompt, or re-extract after fixing a PDF table, without paying for the other phase again. Independently re-runnable stages are also how you get crash recovery for free: a mid-debate failure still leaves completed turns on disk.

General pattern: if your pipeline has a “make the world structured” step and a “reason over the structure” step, make the artifact between them the source of truth. Memory handoffs hide bugs; directories make them inspectable.

Label confidence before anyone debates

Phase 1’s analysis is not one agent that computes everything it can think of. It is three sequential ReAct agents with different trust mandates:

  • Tier A — measured / direct. Things that come off the measurement surface: volumes, feeds, growth rates from observed curves, cross-run KPI tables.
  • Tier B — derived. One step of math from measurements: per-batch rates from timecourses, apparent yields, offgas-derived rates when the sensors exist.
  • Tier C — modeled. Assumption-heavy estimates that use literature constants. Useful, but dangerous if cited as if they were measured.

Why sequential, not parallel? Tier B reads Tier A’s derived tables. Tier C reads both. Each tier is told not to recompute what the previous one already produced. That saves tokens, prevents drift, and makes the dependency graph explicit in the prompts and the checklists.

Why three tiers at all? Because later, when a specialist cites a number, the tier label is the warning label. “This yield was measured” and “this oxygen uptake was estimated under RQ = 1.0” should not carry the same epistemic weight. In any domain with calculations on top of sensors — energy metering, clinical labs, manufacturing MES, A/B experiment platforms — you want the same split: measured, derived, modeled. Without it, debate agents confidently treat assumptions as facts.

Capture everything; feed almost nothing

Every Phase 1 run leaves a paper trail: cell-level lineage for extracted CSVs, a JSONL of every execute_python call with code and stdout, and an agent trace of reasoning plus tool calls. Phase 2 does the same for every specialist turn and every moderator decision.

There is an invariant that matters more than the file layout: the audit directories are never sent back into the LLMs at runtime. They exist for humans. Capture can be rich and full-fidelity; production prompts stay lean. If you mix those two goals, you either lose the ability to debug, or you drown the agent in its own logs until it starts citing its previous hallucinations as evidence.

Same idea applies outside this stack. Your observability pipeline and your agent context window are different products. Do not pretend they are the same buffer.

A panel beats a polymath

Phase 2 is a diagnostic panel: seven specialists with narrow domains, plus a moderator that owns the shared hypothesis board. In our case the domains were strain, kinetics, media, contamination, scale-up, downstream, and data consistency. The names are domain-specific. The structure is not.

For an incident postmortem you might have networking, storage, deploy, client, and capacity specialists. For manufacturing yield loss: process, raw materials, equipment, measurement systems, and scheduling. The rule that transfers is: give each agent a lane, a “stay in your lane” system prompt, and the same tool surface over the evidence bundle. Depth comes from specialization; coverage comes from the panel.

The pipeline looks like this: 

intake
  → initial_analysis   (7 specialists in parallel, ReAct)
  → consolidation      (moderator clusters claims → hypotheses)
  → cross_examination  (loop: moderator picks round type + panel)
  → stance_vote        (per-hypothesis support / refute / insufficient)
  → synthesis
  → report_writer / debate_writer

Initial analysis is a fanout. Consolidation is deliberately a single moderator call: free-form specialist prose becomes a small set of hypotheses on a shared board. Without that clustering step, debate is seven people talking past each other.

Debate needs a board, not a chat log

The mutable shared state in Phase 2 is a HypothesisBoard. Specialists do not “win” by writing longer paragraphs. They emit typed actions: add evidence, update stance, propose a merge, argue for rejection, request more data. Actions apply immediately so the next speaker sees the consequences.

The moderator picks round types on purpose:

  • cross-exam — argue one or two specific hypotheses
  • generative — propose missing hypotheses
  • reconciliation — decide whether two hypotheses are the same claim
  • challenge — force someone to argue against an emerging consensus
  • call vote — terminate the loop

Loop exit conditions matter too: consensus or rejection on every hypothesis, moderator calls the vote, round budget hit, or several rounds with no new board actions. Without those stops, multi-agent systems burn money performing disagreement theater.

If you are building any deliberative agent system — code review panels, security threat modeling, medical differential diagnosis assist — typed shared state beats appending messages to a transcript. A transcript is history. A board is the object under dispute.

Make evidence fetching non-optional

Specialists run as ReAct agents with tools scoped to the bundle: tier reports, assumptions sections, cross-run KPI tables, single metrics, timecourses, derived volume tables. Hard rules per turn include a tool-call budget, a wall-clock budget, and one rule that paid for itself immediately:

If a specialist emits zero tool calls, retry once with a stern “you must fetch evidence” addendum.

The most common failure mode in diagnostic agents is not bad math. It is confident fabrication of a number that looked plausible in training data. Forcing at least one fetch before a claim is allowed to land on the board is cheap insurance — roughly ten percent extra model calls in our runs, and a large drop in invented numerics.

One asymmetry I kept on purpose: initial analysis and discussion turns get tools; the final stance vote sees only the rendered board. Voting is meant to be calibration over already-gathered evidence, not a second research phase. That choice is still debatable — if votes only see evidence titles, consensus can become too easy. The general lesson is still useful: decide which nodes are allowed to gather facts and which nodes are only allowed to weigh them.

Schema design is part of the agent architecture

A quieter set of decisions lived in the Pydantic models. Model providers do not all honor the JSON Schema you think you wrote. In our stack, Gemini’s schema translator stripped oneOf and discriminator, which silently broke discriminated unions into “last arm wins.” Tuple tool args became prefixItems, which the API rejected outright.

The fix was unglamorous and correct: flatten evidence references into one model with optional fields and a type literal; use list[float] instead of tuples; pin the SDK versions that actually work. The wire JSON still looks clean after model_dump(exclude_none=True), so the UI never cared.

If your multi-agent system depends on structured outputs, treat schema compatibility as a first-class design constraint, not a serialization detail. The agent graph is only as reliable as the types it can round-trip.

Do not swallow real bugs as “no contribution”

Early versions had the classic agent-service pattern: catch Exception, return a sentinel, keep the panel moving. That hides AttributeError and TypeError behind polite “this specialist had nothing to say” outcomes. I narrowed the retryable set to transient LLM failures — timeouts, recursion limits, validation errors, connection issues — and let real bugs raise.

Multi-agent systems amplify this mistake because one quiet failure looks like healthy disagreement. Prefer loud engineering bugs and soft model failures, not the reverse.

What transfers to systems like yours

Strip away fermentation and the reusable architecture is:

  1. Split extract/analyze from diagnose/debate. Put a durable bundle between them.
  2. Tier evidence by trust. Measured, derived, modeled — and force later agents to read the label.
  3. Specialize the panel; centralize the board. Parallel opinions, one mutable object of dispute.
  4. Moderate with round types and stop conditions. Disagreement is a tool, not a vibe.
  5. Enforce tool use when numerics matter. Retry empty-handed turns.
  6. Persist every call incrementally. Aggregate the viewer transcript at the end.
  7. Keep audit trails out of model context. Humans get fidelity; agents get lean prompts.
  8. Design schemas for the model you actually call. Flat often beats elegant.

That list is useful whether you are debugging a failed bioreactor batch, a flaky CI component, a production incident, or a clinical study arm that diverged from protocol. The domain experts change. The control plane does not have to.

Honest limits

Architecture does not remove product failure modes. Early runs could still collapse into all-consensus too quickly, especially when challenge rounds were weak and votes saw evidence titles rather than full bodies. Some evidence objects landed with empty numeric bodies until the schema forced fields. Data-request actions could queue with nothing to service them. Those are all fixable — tighter challenge prompts, tools on the vote phase, schema-enforced numerics, a worker for request queues — and naming them is part of shipping a diagnostic system people will trust.

The goal was never a panel that always agrees. It was a panel that cannot quietly invent the data it claims to have read, and a report that shows how it got there.

Closing

Multi-agent debate is easy to demo and hard to operationalize. The version that worked for us was less about clever personas and more about boring systems work: a disk-backed contract between phases, confidence tiers on every number, a typed hypothesis board, hard tool-use rules, and traces that humans can audit without stuffing those traces back into the next prompt.

If your problem looks like “late run failed, past runs exist, root cause is contested,” start there. The specialists are the skin. The pipeline is the skeleton.