Over about six months I built and rebuilt a harness for a tool-using AI agent — first in Python, then as a full rewrite in Rust with a desktop client. Looking back at the commit history, the story isn't "requirements then implementation." It's a sequence of operational surprises that kept forcing hidden assumptions into explicit contracts. Here's what stuck.
Why evidence, not just answers, was the whole point
This is worth pulling out on its own, because it's the reason a harness existed at all rather than a thin wrapper around an off-the-shelf assistant: in the kind of investigative, high-stakes work this was built for, a tool result isn't just an input to the next reasoning step — it's potential evidence. Months later, someone has to be able to answer exactly what was queried, through which tool, when, and whether the conclusion the model reached actually follows from what came back. An assistant that summarizes nicely and discards the raw output underneath is pleasant to use and close to useless for that standard of accountability.
That single requirement is what pulled evidence handling into the center of the design instead of leaving it as a logging afterthought: raw tool output is saved before the model ever sees a summarized version of it, compaction has to leave a pointer back to what it condensed rather than replacing it outright, and “the model believes it saw X” has to be a claim someone can independently verify against “the tool actually returned X.” Lessons 4, 8, and 11 below are really different failure modes of that same underlying obligation: if you can't reconstruct and defend how an answer was reached, you don't actually have an answer — you have a guess that happened to sound right.
Lesson 1: tool naming is a policy decision, not a detail
The very first prototype was barely a day old before tool names collided and had to be namespaced. It seemed like a trivial fix at the time. It wasn't. A harness doesn't just expose "some tools" to a model — it publishes an API surface, and that surface needs names that are stable, unique across providers, and reversible back to whatever actually executes the call. Tool discovery without a naming policy doesn't remove the collision problem, it just postpones it to a worse moment.
Lesson 2: your plugin categories aren't interchangeable, even if your code treats them that way
As integrations piled up — background jobs, long-running external calls, OAuth-gated actions, scriptable commands — a pattern of bugs kept recurring: something that behaved like a "skill" got treated like a "tool," or a "command" got silently coerced into "script" semantics. Each of those categories carries different assumptions about execution, auth, and lifecycle. Collapsing them for convenience works right up until one of them needs to behave differently, and then you're debugging a category-boundary bug that looks like a random one-off.
Lesson 3: keep the core smaller than the product
Higher-level workflow features — templated multi-step recipes, natural-language workflow extraction, conditional branching — moved fast and taught the project as much by failing as by succeeding. Some of that functionality got ripped back out a few weeks after shipping. The eventual replacement was a proper skills system: small, spec-defined capabilities (one file per skill, following the open Agent Skills format) covering both step-by-step playbooks for recurring categories of work and lower-level technical lookups, composed by the agent instead of hard-coded into the orchestrator. That churn wasn't wasted motion; it drew a durable line: orchestration primitives (turns, tools, evidence, memory) need to stay small, stable, and inspectable, while product-level workflow ideas should be free to change quickly precisely because they aren't fused into the core loop.
Lesson 4: a bad or empty tool response is worse than an error
The single biggest source of model hallucination wasn't a bad prompt — it was a tool that failed silently or handed back an empty result instead of a real error. Faced with nothing, the model would confidently fabricate a plausible-sounding answer rather than say it didn't know. The fix was to treat "the tool failed" as a first-class response, not an edge case: every failure mode needs a clear, structured message that says what went wrong and what the model should do about it (retry, ask the user, try a different tool, give up gracefully). An empty string or a bare null is an invitation to hallucinate; a real error with next-step guidance isn't.
Lesson 5: context is a managed resource, not a string buffer
The context window turned into its own battlefield. Automatic compaction shipped with a classic timing bug — it ran after a turn when it needed to protect the next one, so the trigger point had to move. A stale default meant the harness was budgeting against the wrong context size entirely. One integration's tool schemas alone were eating a majority of the available prompt space before the conversation even started.
The first fix was blunt but effective: a simple enable/disable toggle per tool integration, so a user only pays the context cost for the tools they actually turned on for that conversation. That's a stopgap, not a real solution — the honest answer is something smarter than a manual switch: activating tool definitions progressively, based on what the conversation actually needs, rather than loading every schema up front. That's future work, but the toggle alone made an immediate, measurable dent.
The deeper mental model shift: truncation is a provenance operation, not a string operation. Save the full output. Show the model a bounded, clearly-labeled preview. State explicitly that it's partial. Tell the model how to retrieve the rest. Cap recovery attempts. A silent slice of a huge tool result isn't "handling" it, it's just hiding the failure mode.
Lesson 6: a rewrite doesn't reset your threat model
The move wasn't a language-preference exercise: the Python version had hit a real wall as a desktop tool, with no good story for packaging a native app, OS-level file dialogs, or system integration. Rust plus a native app framework got us an actual installable desktop client, and that requirement — not performance, not taste — is what drove the rewrite. Porting the system from Python to Rust (with a native desktop client) meant re-implementing storage, config, OAuth, conversation state — deliberately mirroring the same on-disk formats so users could move between both versions freely. That parity was the right call, but it also meant every old invariant needed a second, independent implementation, plus a new set of native-app privileges the Python version never had. Unsurprisingly, the very first commit that introduced the new native storage layer was already a path-traversal fix. Rewrites don't buy you a clean slate; they buy you a second copy of every assumption you made the first time, running in an environment with more permissions.
Lesson 7: sandboxing the agent sounds safer than it is practical
For a while the agent ran with a deliberately jailed filesystem and shell — the reasoning being that a tool-using model should have the smallest possible blast radius. In practice, that constraint fought the actual job the tool was for: day-to-day analyst work routinely needs to read arbitrary local files, run ad hoc commands, and move data between locations that don't fit neatly inside a sandbox. The jail kept getting worked around, one exception at a time, until the exceptions were doing more work than the sandbox.
The lesson isn't "don't sandbox" — it's that the sandbox boundary has to match how the tool is actually used, or it just becomes friction that gets punched through anyway. The better trade turned out to be: fewer, sharper boundaries (validated paths, explicit allowlists, audited execution) rather than one blanket jail everyone quietly bypasses. In practice that meant trading a blanket filesystem jail for a narrower gate: routine reads and lookups run freely, but anything destructive or high-privilege stops for an explicit human approval before it executes. The constraint moved from “can the agent reach this at all” to “does a person sign off on this specific action” — safer in practice, and far less friction to work around.
Lesson 8: if a result doesn't carry the identity of the state that produced it, treat it as already stale
Once background jobs, schedulers, config watchers, and concurrent UI updates were all writing to shared state, the bugs stopped being "sometimes it's slow" and became "sometimes it's just wrong." A newly created item could collide with an in-flight one. A read-modify-write could clobber a concurrent write. An async reconnect could overwrite state that had already moved on. The fix was a set of boring, unglamorous primitives: per-item mutation locks, atomic file replacement, atomic admission checks instead of check-then-act, and generation counters so a late-arriving result can recognize it's obsolete and discard itself.
Lesson 9: an agent needs to know about its own memory, not just the current turn
A single conversation turn is a bad unit of introspection. The agent needed to know what else it (or the user) had already tracked: other open conversations touching the same case, a persistent to-do list of pending actions instead of intentions buried in prose, and structured case facts (key entities, findings, open questions) that survive well past whatever fits in one context window. Bolting that on after the fact was much harder than it needed to be — a durable, queryable memory layer (distinct from conversation history, and distinct from tool output) turned out to be a core primitive, not a nice-to-have feature.
Lesson 10: don't let a refactor and a behavior change share a commit
At one point several core modules had quietly grown oversized from months of accumulated feature work, and splitting them into focused modules became necessary. Doing that split at the same time as landing security fixes and race-condition repairs made both much harder to verify independently. A pure structural refactor and a behavior change are two different kinds of risk, and reviewing them together means you can't cleanly attribute a regression to either one.
Lesson 11: "it looks successful" is not the same claim as "it succeeded"
The subtlest failure mode showed up last, and it's the one I'd flag as most important: a system can report success when the underlying execution record doesn't actually support that conclusion. Concretely, that meant things like malformed tool arguments getting silently replaced with an empty object instead of failing loudly, provider-side error flags getting lost before they reached the orchestration layer, truncated results not carrying an exact enough pointer back to the full evidence, and models hitting output limits in ways that were indistinguishable from a normal, clean completion.
The fix touches every layer: validate every tool call and preserve invalid input as invalid (don't paper over it), carry error/truncation/evidence state through the full pipeline as structured data rather than prose, emit an explicit "incomplete" status instead of guessing, and make grounding in real evidence a non-optional contract rather than a nice-to-have. Truthfulness, it turns out, isn't a tone you can prompt for. It's end-to-end preservation of what actually happened during execution.
What I'd tell someone starting a harness from scratch
If I were doing this again, I'd design the execution record for a tool call before I designed the prompt. It should be able to answer: what did the model actually request, was the input valid, did policy evaluation run to completion, which system actually got called, did that system flag an error, was the output partial or truncated, where does the raw evidence live, and can all of that be correlated back to whatever the model ultimately claimed happened.
Everything else — retry policy, state ownership for background work, budgeting the repeated cost of tool schemas across every turn of a loop (not just the first prompt), testing the actual packaged app instead of just the source checkout, giving the agent real memory instead of a single-turn view — falls out naturally once that record exists. The core lesson underneath all the others: build the boring parts (evidence, validation, identity, bounded failure, honest errors) before the interesting parts (prompts, workflows, UI), because the interesting parts are what everyone notices, but the boring parts are what keeps the whole thing honest.
None of this was academic. The harness is in daily production use now, and the payoff analysts will actually mention isn't any of the architecture above — it's that a lot of the first-pass triage and routine lookups that used to eat their day now happen automatically, evidence trail attached, leaving them the time to focus on the part of an investigation that actually needs a human.
Comments
Post a Comment