Teaching a debugging agent where to look first

I work as a Solutions Engineer on parasitic extraction: debugging runs, automation and data inspection around a PEX tool. The agent I am building debugs extraction runs and answers product questions, grounded in what the team already knows.

That knowledge lives in curated units called pills (symptom, root cause, fix, evidence), plus extraction options, error entries, bug records and debug learnings. The original search read a JSON index, filtered it and used fuzzy matching to survive typos. It worked up to a few hundred pills.

I wanted a tool that could hold everything without saturating the context window, and that always knew what was available and what to use.

The tempting order is to improve search first. I defined evidence first. A manual states expected behaviour, a bug record describes a failure in specific builds, a run shows what actually happened, and a model inference is only an inference. The agent has to know which one it is holding.

Every entry carries its source type, process, mode, affected builds, fixed-in build, validation status and content hash. That metadata also decides when two claims really conflict: "release A requires option X" and "release B no longer requires it" are not a contradiction. They apply to different builds.

Before relations, manuals or agentic behaviour, the search had to be measurable. The golden set mixes exact codes, typos, options, partial names, natural-language symptoms, build-specific and multi-hop questions, plus questions the knowledge base cannot answer, because a good search must sometimes come back empty.

The first run scored 1.000 on every metric. I did not read that as proof: 28 queries can show that nothing obvious is broken, not that nothing is. I expanded the set and kept the score as a baseline, the same way I treat a backtest that looks too good.

Relations came next, as a GraphRAG-lite with no graph database: errors link to pills, pills to bug records, bug records to options. One hop by default, two for complex troubleshooting, never everything.

Official manuals only arrived after that, starting with a single one. The original is kept, hashed and tagged with its release, then split along its own structure (chapter, section, subsection, option, procedure, table) with page ranges, instead of fixed 500-word chunks.

Then citations. Every result carries a source id, so a conclusion can keep apart what the manual says should happen, what a bug record says happens in a given build and what the current run shows, before concluding that the run matches a known bug.

By August 2026 the agent retrieved knowledge well. Pointed at a run directory, though, it did not know which files to open first: it had no picture of the flows that produced them.

I added two layers. User-intent search returns a procedure for the requested kind of task alongside the knowledge results. Version one matched requests lexically against synonym groups, with stopwords and a threshold, over 11 procedures stored as procedure pills.

Flow file knowledge is a search over each flow's files and documentation. When the agent finds something, it knows which flow document it belongs to, which artifacts the flow produces (setup script, run summary, extraction outputs, SPEF or DSPF netlists) and what each one helps diagnose.

Version one drifted towards rails: an intent lookup on every task, a fixed checklist, the procedure repeated on every search. I reviewed it against Anthropic's Agent Skills (progressive disclosure), Anthropic's "Building effective agents" (workflows versus agents, routing), Agent Workflow Memory and ReAct, then applied the changes through one consolidated prompt.

The main change separates two memories. Declarative knowledge answers "what do we know about this error?". Procedural knowledge answers "given this run, what do I look at first, where, and in what order?". A capability is what can be done, a procedure how to do it, a tool what executes it, and file knowledge where the evidence lives.

A procedure is now a playbook: purpose, first checks, expected artifacts, first actions, stop conditions and a fallback. Its decision points branch on evidence: a failed extraction leads to the errors, a completed one to output validation, a netlist anomaly to the netlist procedure.

Retrieval metrics cannot judge this layer, so it has its own: top-1 intent accuracy, top-3 recall, precision when no intent applies, and wrong-procedure activations. The last one matters most. Activating the wrong procedure is worse than finding none.

A workflow benchmark scores what the agent does next: filesystem calls, broad recursive searches, knowledge-search calls, repeated searches, step count and whether the first action was useful. The main prompt keeps only behaviour, and its token count is tracked before and after.

This layer is work in progress, and those numbers will judge it. The goal is not to tell the agent exactly what to do, but to give it a road map so it stops driving across the fields.