The Agent Optimizer: Using AI to Build Reliable AI

Hi all,

📌 If this email lands in Promotions, please drag it to Primary so you don’t miss future updates.

---

The Agent Optimizer: Using AI to Build Reliable AI

Everyone wants AI agents right now. But deploying them reveals a harsh reality: agents are non-deterministic, prone to hallucinations, slow, and expensive.

How do we optimize software that doesn’t behave predictably? Simple: We use AI to build AI.

This week, we break down the engineering framework shared by Alfonso Graziano (Tech Lead at Nearform and O’Reilly author of Learning AI Native Software Engineering) on how to build autonomous optimization loops that fix agents without human hair-pulling.

The Anatomy of an Agent

At its first-principles core, an AI agent consists of three elements operating inside a continuous loop:

1. The Brain: The core Large Language Model (LLM).

2. Tools: The APIs and functions the LLM can trigger.

3. Context: The data and retrieval systems the LLM can query.

When agents fail, they typically fall into two categories: bad performance on your evaluation suites (evals) or bad performance on live user data. Here is how to conquer both.

Battleground 1: Crushing Evals with “Auto Agent”

When an agent fails its initial evaluations, developers usually spend hours manually tweaking system prompts or adding tools. Instead, we can automate this entire cycle.

1. Build a “Golden Dataset” and Scorer

You cannot optimize what you do not measure. Work with Subject Matter Experts (SMEs) to build a Golden Dataset that outlines specific inputs and expected outputs (ground truth). Pair this with a Scorer to evaluate the agent’s output and generate a baseline accuracy metric.

• The Cold Hard Truth: A bare-minimum agent relying solely on its internal LLM weights might only hit an 18% pass rate on complex tasks.

2. The Auto Agent Loop

Inspired by Andrej Karpathy’s “auto research” loop, Alfonso built Auto Agent—an autonomous loop designed to optimize AI agents:

[Markdown Objective] ➔ [Run Baseline Eval] ➔ [Create Hypothesis Branch] ➔ [Run Evals] ➔ [Keep or Rollback]

• The Blueprint: Define the optimization objective, target repository, and performance metrics in a Markdown file.

• The Baseline: The system runs evals once to generate a comprehensive baseline report of what is broken.

• The Branch & Execution: An optimizer coding agent opens a new git branch, proposes a single hypothesis to tackle a failure class, and updates the target agent’s system prompt or tool logic.

• The Verdict: The system re-runs evals. If accuracy metrics improve, the loop continues from this new branch. If a regression occurs, it automatically rolls back to the previous stable branch.

• The Memory: The system updates a global memory.md file across all runs so the coding agent learns from its previous successes and failures.

• The Human Guardrail: You must explicitly instruct the coding agent not to modify the golden dataset or the scorers just to force a passing grade.

Real-World Impact: Using this loop, Nearform boosted a naive baseline agent from 18% to 83% accuracy, and even squeezed an additional 10% improvement out of a heavily human-optimized production agent by catching edge cases humans missed.

Battleground 2: Fixing Messy, Live Production Data

Evals are clean; production data is chaotic, wide, and messy. To optimize your agent based on real-world interactions, you need a continuous engineering pipeline.

The Live Data Optimization Pipeline

1. Collect Detailed Traces: Log user interactions, tool usage, execution latency, and token consumption, not just the final output.

2. Gather Multi-Source Feedback: Collect explicit user feedback (thumbs up/down) and have SMEs annotate raw traces to log how the agent should have behaved.

3. Automate Cluster & Root Cause Analysis: Feed these traces to a coding agent to run a cluster analysis on the negative feedback. Because it has codebase access, it can trace failures down to the code level and conduct a root cause analysis.

4. Triage and Validate: The agent generates an executive markdown summary detailing failure clusters, matching trace IDs, and proposed code fixes. Humans review this to triage issues into three buckets: fix now with the agent, fix later, or discard.

5. Close the Feedback Loop: Once a failure mode is resolved, turn that specific production failure into a permanent fixture of your Golden Dataset to prevent future regressions.

• Cadence Tip: Running this pipeline once per sprint is typically the sweet spot for keeping production agents highly optimized without drowning in data.

The Core Philosophy: Harness Engineering

All of this automation relies on Harness Engineering, the practice of building a robust, highly constrained environment around your coding agents so they can operate reliably and safely.

A world-class engineering harness requires four distinct layers:

• Spec-Driven Environments: Every production failure mode must be translated into a clear technical specification before the agent attempts a fix.

• Quality Gates: Tight integration of code linters, unit tests, automated evaluation suites, and secondary LLM code reviews.

• Context Engineering: Providing the coding agent with precise, scoped access to the exact files and documentation it needs, while restricting everything else.

• Observability: Deep logging of thinking traces so both humans and agents can pinpoint exactly where a logic chain shattered.

By wrapping your development workflow in a strict harness, you transform AI from an unpredictable assistant into an autonomous, self-healing engineering engine.

New Jobs This Week

🤖 AI, Machine Learning & Agentic Systems (Remote)

1. Senior Distributed Systems Engineer (AI Infrastructure) – Lambda Labs

• Apply here

2. Senior Software Engineer, Cluster Management – Lambda Labs

• Apply here

3. Senior Technical Product Manager, Deep Learning Platform – Lambda Labs

• Apply here

4. Developer Relations – Maxim AI

• Apply here

5. Senior Software Engineer, Platform (Model Deployment) – Cohere

• Apply here

6. Member of Technical Staff, Toolkit Automation – Cohere

• Apply here

7. Member of Technical Staff, Machine Learning Platforms – Cohere

• Apply here

8. Senior Infrastructure Engineer (Data Platform) – Cohere

• Apply here

9. Senior Full Stack Engineer (Internal Tools & AI Workflows) – Cohere

• Apply here

Notes from Team:

The landscape is shifting faster than a prime Cristiano Ronaldo counterattack. Look at Cohere and Lambda Labs opening wide pipelines for platform engineers, distributed cluster management, and automation specialists this week. If you want to dominate the engineering field like the GOAT dominates the pitch on the world stage, focus on becoming the engineer who scales the system, not just the one sitting on the bench chatting with it. Siuuu

Wishing you a productive week ahead,

The Fursah Jobs Team