The EcoRouter chatbot: Building an AI guide and why It differs from the data agent

The chatbot is also compared with the data agent covered in Keeping a maritime emissions model up to date and From a local prototype to a cloud service. Both run on the same platform and use the same language model, but they are built differently because they solve different problems.

How the chatbot works

The chatbot is defined by three things.

1. A general-purpose language model (GPT in our case)
2. a page of written instructions
3. a knowledge base of documentation

The model was not trained on EcoRouter. Many people assume that product chatbots are trained on product documentation, but that is rarely the case. Training a model on your documentation may influence its tone and style, but it does not provide a reliable way to maintain factual accuracy. It also makes it difficult to trace answers back to their sources and requires retraining whenever the software changes. For a product under active development, that approach is impractical.

Instead, the model searches the documentation itself when a question calls for it, and answers from the passages it finds. If the first search brings back nothing useful, it can search again with different wording.

This technique, known as retrieval-augmented generation (RAG), is the standard way of providing a general-purpose language model with knowledge about a specific product.

The search is based on meaning rather than exact wording. Each document is transformed into a numerical representation of its content and indexed in Azure AI Search, allowing the system to find relevant information even when users phrase questions differently from the documentation. We use our own Azure AI Search service rather than the managed knowledge feature available in Foundry.

The chatbot also has access, via MCP, to a small set of read-only tools connected to EcoRouter’s live emissions and regulatory configuration. This allows it to report current values rather than rely solely on information captured in documentation. It has no ability to modify data or settings.

Building the knowledge base

Setting up the search and integrating the widget into the application was the small part of work. Building the knowledge base took several weeks and remains an ongoing effort. It currently consists of 57 documents. Four principles proved particularly important during its development.

1. Depth. Documentation written for colleagues turned out to be far too thin. What works as a reminder for an experienced user often provides too little information for a chatbot. To give the model the context it needs, every screen is documented control by control, including when controls are greyed out, the exact wording of error messages, and which actions cannot be undone.
2. Questions written the way users ask them. The largest part of the knowledge base consists of 639 Q&A pairs, each headed by the question itself. This is not just a stylistic choice but a retrieval strategy. The search compares a user’s query with the indexed content, so a heading such as “Why did my run take so long?” is far more likely to match what someone types than a section titled “Performance”. Framing headings as questions improved answer quality more than any parameter we changed.
3. Procedures, not just controls. Users are more likely to ask how to complete a task than what an individual control does. We therefore created nine documents describing end-to-end procedures that span multiple screens, from preparing input data and launching a run to interpreting the results. Each guide also explains what to do when the first attempt fails.
4. Limitations. “Can the software do X?” is one of the most common questions users ask, and the answer is often no. Three documents therefore describe, in plain terms, what EcoRouter cannot do. Without this information, the chatbot risks sending users in search of features that do not exist.

Where the knowledge came from

Roughly half of the knowledge base could be derived directly from the source code. By examining the application, we could determine what each control does, why it might be greyed out, what different status messages mean, and which combinations of inputs are rejected. This provided the basis for most of the Q&A content without requiring interviews or workshops.

The other half was not in the code at all. The application does not explain where input data originates, which parts of the underlying datasets are less reliable, how long a realistic run is likely to take, or the mistakes users most commonly make. That knowledge resides with the people who built the tool. Capturing it required around a hundred written questions and answers collected over several weeks. These records were kept alongside the documentation so that every statement in the knowledge base could be traced back to its source.

Comparing the chatbot and the data agent

data agent reads live regulatory data through tools and stages a change for a person to approve.

The data agent keeps EcoRouter’s emissions and regulatory data up to date. It reads the values currently used by the software, compares them with published sources, and submits proposed updates for administrator approval.

The two systems differ from the second step onwards. The data agent starts with the data itself. Carbon prices change, regulations are amended, and sea areas can acquire new legal statuses. Documentation cannot keep pace with these changes. The agent therefore needs a way to inspect the current state, compare it with authoritative sources, and ensure that no change takes effect without human approval.

For the chatbot, the challenge was different. The information was already contained within EcoRouter, but much of it had never been documented. The key question was whether the topic had been captured in the knowledge base. If it had not, there was no other source the chatbot could rely on.

Finding what is missing

Every interaction is logged, together with the documents retrieved to answer it. Administrators can review these logs at any time.

This is the only reliable way to identify gaps in the knowledge base. Users who do not receive a useful answer are unlikely to report the problem; they simply close the chatbot and move on. The logs turn these missed answers into a list of topics and pages that need to be written.

Recording the retrieved documents alongside each response also helps distinguish between two very different types of failure. If the correct document was retrieved but the model failed to interpret it correctly, the content may need to be rewritten. If the relevant document was never retrieved, its wording or structure may need to be revised to make it easier for the search system to find.

The knowledge base is never truly finished. Every change to the software requires corresponding updates to the documentation, and the logs continually reveal new questions that nobody thought to document.

The research was conducted through the MarTrans project and the Dynaport project.