This post describes the agent, what had to change in the software to make it possible, and how we test it with a language model acting as a judge, which also repairs most of the failures it finds.
We have previously described two AI components in EcoRouter, our voyage optimisation software. The data steward keeps emissions and regulatory data up to date. It has since moved from a laptop into the application as a service. The chatbot explains how the software works, including what a control does and what an error message means.
Neither of them can tell you why your result looks the way it does. The solution analyser can, and that changes how it has to be built.
The question worth answering
A user can read a chart. If one route option is slower and cheaper than another, the chart shows this. What it cannot explain is why the complete set of route options has the pattern it does.
Consider a run (one calculation of route options for a voyage) in which all the alternative routes come out looking very similar. An example is shown in Figure 2. There are several possible reasons. The voyage itself may allow only a few really different ways to sail it. Alternatively, a tight deadline, or a search that was not detailed enough, may have left little room for the routes to differ. Each cause requires a different response, yet the chart looks the same in every case. The information that distinguishes these causes is stored with the run, but it is not shown to the user, who cannot be expected to search through a technical configuration file to find it.
The agent therefore reads the result and the set-up together. It reports which conditions actually applied in this run, and which figures should not be relied on.
What the agent is given
Each completed run is summarised in a short digest. This contains the software’s checks of its own output, the settings used for the run, and information that the solver identified during the search but would previously have discarded. The digest is produced by ordinary code, not by a language model, so the same run always gives exactly the same digest. A language model might word it differently each time.
The digest is kept short on purpose. When the agent needs more detail, it looks it up in the saved run, using tools that can read the run but cannot change it. These tools are connected through MCP (Model Context Protocol), a standard way of linking AI models to software. The agent keeps looking things up, one step at a time, until it has evidence from the run for everything it says. Most questions depend on several values at once, and each of them may look perfectly normal on its own; the explanation lies in the combination. The agent quotes these values from the run rather than calculating them itself, so every number in an answer can be checked against the stored result.
A knowledge base that could not be built from questions
A product chatbot’s knowledge base can be built around the questions users ask, because its subject is the software itself, which is fixed and can be documented. Computed results are different. A result does not exist until someone runs the software, and it can go wrong in ways nobody has foreseen. So there is no list of questions to build the knowledge base from.
The documentation therefore describes what the software can observe about its output and settings, rather than answering anticipated questions. Whatever the user asks, the agent first reads the pages about the conditions that were actually found in the run. It does not choose pages just because their wording resembles the question.
The documentation on settings has a second use. It allows the agent to point out choices at the narrow, most restrictive end of their usual range, and to suggest possible changes. These suggestions explain what the documentation says a setting normally does; they do not predict what will happen in a particular run.
What the software had to start recording
Much of the work revolved around changing the optimisation software itself. Many values needed by the agent were calculated during the search and then discarded. For example, if no route to the destination exists, the solver knows why as soon as the search fails, but that reason used to be thrown away. So the user would end up with no result and no explanation as to why. Cases like this had to be identified individually, and the reason for each failure had to be stored with the result.
What it may not say
The agent tells the user what it found in the run and which settings were used. It is not allowed to claim that a particular setting caused the result, except in two cases: when the software recorded the connection during the search, or when the connection follows directly from the way the algorithm works and is therefore true for every run.
There are two reasons for this restriction. First, such claims cannot be proven. The various inputs to an optimisation process interact, so it is not possible to isolate the effect of a single setting. Changing one setting and running the software again does not prove a cause, because it produces a second, different result. Second, a false explanation of this kind cannot be detected afterwards. If the agent quotes a number incorrectly, testing can compare it with the correct value stored in the run. If the agent invents a convincing explanation, there is nothing to compare it against. Such claims must therefore be ruled out in advance.
Testing and the repair loop
Testing is done during development by a second AI agent, separate from the one in EcoRouter. It asks the analyser questions, scores the answers and fixes what fails. The analyser is tested using a set of questions drafted by a language model and corrected by people. Each question includes an example of a good answer and several examples of incorrect answers. Every answer is compared with the stored result and these examples, so the assessment is based on known values rather than a reviewer’s personal preference. Scoring by a language model in this way is known as LLM-as-a-judge, here guided by reference answers.
The loop does not end with a pass or fail decision. The model examines each failed answer and identifies the likely cause. There are four possibilities: 1) the solver did not store a value it had already calculated; 2) a documentation page was incorrect or missing; 3) a retrieval tool could not access the required detail; or 4) the test question was unclear. The model repairs most of these problems itself. A developer reviews changes that require decisions about the system’s design and decides when the loop has run for long enough.
An unclear question is the one type of problem that the scoring process cannot detect, because the answers are assessed against the question itself. And unclear questions do occur, because the people who write them also built the system. They know what they intended, and may not spot how a question could be interpreted otherwise.
What we found
The data steward needed instruments: ways to measure and check what the software holds. The chatbot needed documentation. The analyser needed both, and its documentation had to describe what the software can measure rather than provide answers to expected questions. We believe this applies to any software that produces a computed result. It is impossible to list every question users may ask in advance, but the values needed to answer those questions are already inside the software, and most of them are never shown to the user.
The research was conducted through the MarTrans project and the Dynaport project.
Comments
No comments yet. Be the first to comment!