Scaling Agent Goodput: Improving Complex Agent Effective Throughput by 96.3×

Under identical quality thresholds and an 8-minute delivery SLA, we reduced our prototyping Agent’s average single-task cost from $21.80 to $0.88, compressed P95 end-to-end latency from 55.4 minutes to 7.6 minutes, and achieved a 96.3× improvement in Agent Goodput (qualified deliverables produced per $100 compute spend).

This evolutionary journey began with a pervasive production pain point: a user would make a minor adjustment to a button style or API contract on a single page, but an unconstrained Agent exploring an open action space would trigger a destructive full-codebase refactoring. A minor tweak that should have taken minutes degenerated into a cascading LLM re-entrancy loop lasting 60 minutes and consuming over $30 in raw API calls.

This single local refactoring incident exposes a systemic vulnerability in complex LLM Agents: when task graph interconnectivity scales with system complexity, unconstrained non-deterministic reasoning inevitably traps the workflow in oscillation and entropy.

Optimizing complex Agent architectures requires defining three core system boundaries:

1. Terminal Boundary: Defining objective criteria for when a task is truly completed;
2. Reasoning Boundary: Identifying state transitions that genuinely require non-deterministic LLM reasoning;
3. Control Boundary: Determining how deterministic computation and rule-based state transitions execute efficiently.

This article details the architectural evolution of our prototyping Agent system over several months: moving from open-ended ReAct exploration toward a high-throughput architecture centered around contract-driven decomposition, deterministic workflows, and context topology optimization.

1. The Post-Hoc Review Trap: Latency and Cost Inflation

In early iterations, the system relied on an open ReAct loop to drive the entire generation lifecycle. The Agent dynamically reasoned about its next action given current state, called tools to emit or test code, and used feedback to decide whether to continue generating, repair errors, or finalize delivery.

To enforce a quality floor, engineers overlaid multi-tier review checkpoints within the ReAct loop. Every generated page was routed through mandatory LLM evaluation; detected defects triggered code repairs, followed by re-evaluation.

flowchart LR
    A["Implement Page"] --> B["Review Checkpoint"]
    B -->|"Defect Found"| C["Repair Module"]
    C --> B
    B -->|"Pass"| D["Deliver Prototype"]

While this design successfully intercepted sub-par code before final delivery, heavy reliance on post-hoc review introduced severe engineering drawbacks:

• Compute Inflation: Every review iteration reloaded massive system contexts, executed fresh LLM inference, and ran tool checkers. Repairs frequently introduced subtle regressions in previously passing modules, triggering endless secondary review loops.
• Severe Tail Latency: On multi-page prototypes, single-task model costs routinely exceeded $20, while end-to-end wall-clock time surpassed 60 minutes.
• Degraded SLA Determinism: Evaluation variance in the reviewer LLM caused even simple tasks to get stuck in multi-turn review cycles, making SLA commitments impossible to guarantee.

Evaluating across a representative benchmark with uniform task complexity, the review-heavy baseline exhibited extreme tail latency and cost inflation:

These empirical results highlighted a critical counter-intuitive reality: Adding more post-hoc reviews raised the quality floor, but degraded effective delivery capacity. The architecture was spending exponentially more compute to yield increasingly unpredictable completion times.

This shifted our engineering goal from maximizing naive success rate in isolation to maximizing useful throughput under strict latency and cost SLAs.

2. Quantitative Baseline: Engineering Definition of Agent Goodput

To rigorously quantify architectural efficiency, we adapted throughput benchmarking principles from computer networking and distributed systems, establishing Agent Goodput as our primary engineering metric.

In network engineering:

• Throughput measures raw data packets transmitted across the wire, including retransmissions, protocol headers, and dropped frames.
• Goodput measures useful application-layer payload delivered successfully without errors.

Mapped to LLM / Agent architecture:

• Raw Token/Request Throughput merely reflects total compute spend or API interaction frequency;
• Agent Goodput isolates the rate of SLA-compliant, quality-verified end-to-end deliveries produced per unit compute spend.

We formally define Agent Goodput as:

In practical terms, it answers: For every $100 spent on LLM compute, how many qualified product prototypes can the system deliver within its promised SLA window?

The metric enforces strict constraints on both sides of the ratio:

• Numerator: Includes only tasks that simultaneously meet objective product-completeness criteria and finish within the end-to-end SLA (e.g., 8 minutes);
• Denominator: Accounts for all compute costs across the system, including prompt/completion tokens, tool execution overhead, intermediate evaluation calls, retries, and abandoned/failed runs.

Agent Goodput establishes clear optimization priorities: Quality must first cross the acceptance threshold, end-to-end latency must then be bounded within SLA limits, and total compute cost per task must subsequently be minimized.

3. Architectural Decoupling: Contract-Driven Front-Loaded Planning

The inefficiency of the legacy architecture stemmed from dispersed decision density. In open ReAct loops, each page generation node independently attempted to interpret global requirements, scattering architectural choices across subtasks and triggering style drift and multi-turn refactoring loops.

Our solution shifts non-deterministic design reasoning upstream of execution by introducing an explicit Planning Node. Before any code implementation begins, the planner freezes three system-level contracts:

1. Decomposition Contract: Locks the page tree topology, routing relations, and component boundaries;
2. Shared Constraint Contract: Standardizes global design tokens, UI component primitives, and state-management conventions;
3. Acceptance Contract: Formulates deterministic pass/fail rules and validation gates.

flowchart TD
    subgraph NEW["Contract-Driven Plan-First Architecture"]
        N1["Planning Node generates global contracts"] --> N2["Implement Page A inside boundary"]
        N1 --> N3["Implement Page B inside boundary"]
        N1 --> N4["Implement Page C inside boundary"]
        N2 --> N5["Deterministic Gate Validation"]
        N3 --> N5
        N4 --> N5
    end

By front-loading planning contracts, global architecture design is decoupled from local implementation. The Planning Node absorbs top-level non-deterministic reasoning, while downstream subtasks operate within bounded deterministic contracts. Heavy LLM review loops simplify into fast deterministic contract validation.

4. Topology Restructuring: Parallel Execution and Prefix Cache Sharing

Formalizing global planning contracts unlocked independent parallel execution for subtasks. However, naive parallel execution exposes two new bottlenecks: context token inflation and branch latency accumulation.

We restructured the pipeline to address both:

4.1 Critical Path Decoupling and Branch Concurrency

Because parallel page generation subtasks share identical top-level design facts from the planning contract, end-to-end execution time shifts from serial accumulation to being bounded by the single slowest critical path:

• Serial Execution Time: \( T_{\text{serial}} \approx T_{\text{plan}} + \sum T_{\text{page}} + T_{\text{gate}} \)
• Parallel Execution Time: \( T_{\text{parallel}} \approx T_{\text{plan}} + \max(T_{\text{page}}) + T_{\text{gate}} \)

Segmenting benchmark results by page count demonstrates the performance gains:

While legacy latency scaled linearly with page count, the restructured architecture absorbs additional pages by expanding concurrency width without lengthening the critical path. Completion rates remain stable, proving that front-loading planning captures quality specifications far more effectively than tail reviews.

4.2 Tree-Shaped Prompt Topology and KV Cache Reuse

Naive branch concurrency causes token costs to explode because every branch redundantly reloads global requirements, layout trees, and visual tokens.

We solved this by restructuring prompt inputs into a Tree-Shaped Context Topology, separating inputs into a Global Static Prefix (requirements, contracts, design system) and a Local Dynamic Subtask Context:

flowchart TB
    P["Global Static Prefix<br/>Requirements · Structure · Visual Tokens · Rules"]
    P -->|"Shared Prefix Cache"| A["Page A<br/>Local Subtask"]
    P -->|"Shared Prefix Cache"| B["Page B<br/>Local Subtask"]
    P -->|"Shared Prefix Cache"| C["Page C<br/>Local Subtask"]

The inference engine maintains the Global Static Prefix as an in-memory KV cache, reused seamlessly across parallel branches. Once the initial cache write is amortized across parallel branches, context processing costs drop dramatically:

By converting branch context costs from additive (\(O(N)\)) to single-write read-reused (\(O(1)\)), context topology optimization alone reduced total task costs on 5-to-6 page prototypes by 56%.

5. Bounded Control: Deterministic State Machines and Rule Gates

Having eliminated decision drift and context bloat, we resolved the final bottleneck: execution entropy in open ReAct loops.

While ReAct excels at open exploration, letting an LLM decide when to continue reviewing once contracts are fixed introduces unnecessary execution entropy and long-tail latency.

We extracted control logic out of LLM prompts into a Deterministic State Machine Workflow:

flowchart LR
    P["Plan Phase<br/>Agent Exploration"] --> E["Parallel Branches<br/>Workflow Control"]
    E --> G["Contract Gate<br/>Rule Validation"]
    G -->|"Pass"| D["Delivery"]
    G -->|"Fail"| R["Exception Repair<br/>Bounded Local Agent"]
    R --> G

Under this bounded control model:

• The Workflow Engine governs branch concurrency width, execution timeouts, and retry budgets;
• Rule Gates supply fast, objective pass/fail validation signals;
• LLM Agents are restricted to upfront planning decisions and bounded local repairs when a gate fails.

When a gate passes, the workflow terminates immediately. When it fails, repair context is strictly scoped to the failing component, preventing full-graph rebuilds.

Deterministic workflows established strict latency and cost bounds across all benchmark workloads:

The 8-minute ceiling is not an empirical coincidence—it is an architectural guarantee enforced by concurrency limits, scoped repairs, and explicit termination conditions.

6. Throughput Overview and Task-Graph Restructuring Paradigms

Across our standardized 200-sample benchmark suite (2–6 page prototypes, 8-minute SLA), these three paradigm shifts produced a step-function increase in Agent Goodput:

Surveying industry literature on Agent optimization highlights distinct architectural perspectives:

• OpenAI emphasizes establishing quantitative evaluation baselines (Evals) and applying model cascading to route lightweight tasks to smaller models;
• Anthropic advocates keeping control flows minimal and invoking LLM reasoning only at non-deterministic choice points;
• EvoRoute and AI21 Maestro focus on dynamic model routing and multi-agent configuration space search.

We incorporated core insights from these approaches, but identified a fundamental distinction: Model routing optimizes which model executes a given node, whereas task-graph restructuring determines whether that reasoning node needs to exist at all.

By front-loading contracts, open exploration collapses into bounded workflows and rule gates. Routing strategies select compute carriers, but task-graph restructuring eliminates redundant inference at the source.

7. Conclusion: From In-Band Blocking to Out-of-Band Quality Flywheels

The engineering evolution of complex Agent systems spans three distinct stages: from early unbounded exploration relying on post-hoc reviews, to structural decomposition via contract-driven planning, and finally to bounded control governed by deterministic state machines.

This journey reflects three core engineering principles:

In this bounded workflow, synchronous in-band reviews are replaced by deterministic rule gates, locking latency firmly inside SLA bounds. However, failure traces, component misalignments, and repair logs captured during review remain rich diagnostic assets.

Decoupling review mechanisms from the synchronous critical path enables a new paradigm: offloading review diagnostics to an asynchronous out-of-band observer loop—powering specialized model fine-tuning, preference alignment (DPO/RLHF), and dynamic rule-gate evolution.

References & Further Reading

1. Anthropic, Building Effective Agents: https://www.anthropic.com/research/building-effective-agents
2. OpenAI, OpenAI Evals Framework: https://github.com/openai/evals
3. EvoRoute Research, Dynamic Model Routing for LLM Agent Workflows
4. AI21 Labs, Maestro: Multi-Agent Orchestration Engine