Fault diagnosis from machine logs and technician notes

Fault diagnosis at this plant begins the moment a machine throws a fault and writes structured data to the historian database. A maintenance technician then manually reviews those logs, searches their own memory and a decade of free-text repair notes stored in the maintenance system, and mentally connects the current fault to patterns they have seen before. Once they have a working hypothesis, they propose a root cause, open a ticket listing the required parts, and record their reasoning in the maintenance file for audit purposes. The whole cycle runs roughly 2,400 times a year and takes an average of 55 minutes per case, consuming around 2,200 hours of skilled technician time annually.

The process sits at the intersection of two very different data types. The historian database is structured and queryable, giving a clean picture of what happened electrically or mechanically at the moment of the fault. The maintenance system holds ten years of repair notes written in plain language by multiple technicians, full of institutional knowledge but inconsistent in terminology and format. Bridging those two sources quickly and reliably is the core challenge the team faces every time a machine goes down.

A small but significant minority of cases, somewhere between ten and fifteen percent, do not match any standard pattern and require genuine expert judgement. Compliance rules make this distinction critical: safety-critical decisions must be made by a qualified technician, and the reasoning behind every diagnosis must be recorded in the maintenance file. The current process depends heavily on the knowledge of two or three experienced people, which creates concentration risk and limits the team's ability to scale as fault volume grows.

The recommended path is a Hybrid AI workflow with human-in-the-loop gates, and the case for it rests on a combination of factors that rarely align this cleanly in a single process. The volume is high enough, around 2,400 cases per year, that even modest per-case time savings translate into a compelling return. The data is rich enough, structured logs plus a decade of repair history, to give an AI system genuine material to work with. And the process has a natural complexity threshold built into it: the majority of faults are pattern-matching exercises that a well-designed system can handle reliably, while a defined minority genuinely require a trained technician. A Hybrid architecture is not a compromise between automation and manual work; it is the design that fits this process most precisely.

The client's preferred tech stack is Python, and the automation engineer already has live integrations against both the historian database and the maintenance system. This matters more than it might initially appear. Many automation projects of this type spend their first month on infrastructure: standing up middleware, negotiating API access, dealing with authentication across systems. Here, that work is largely done. The Hybrid AI workflow can be built directly on the existing Python foundation, which reduces build risk, keeps the team working in a language they already know, and avoids introducing a new vendor whose platform the automation engineer would need to learn and maintain. The recommendation fits the stack precisely, and that alignment is worth preserving.

The alternative paths were considered seriously. A fully automated AI Workflow scores well on the routine majority but fails on the compliance requirement: safety-critical decisions must stay with a qualified technician, and a system with no mandatory human gate cannot satisfy that constraint by design. An AI Agent could handle the retrieval and reasoning steps capably, particularly the semantic search over free-text repair notes, but an autonomous agent without hard escalation controls carries the same compliance problem. Traditional Code could support rule-based pattern matching against known fault codes from the historian database, but it cannot reason across ten years of unstructured text, which is precisely where the diagnostic value lies for complex cases. RPA is the wrong tool entirely: the bottleneck here is semantic reasoning, not UI navigation or data copying between fixed screens.

The Hybrid path satisfies the compliance requirements by design rather than by workaround. Every AI inference step is logged, which means the maintenance file always contains recorded reasoning whether the case was handled automatically or escalated. The human gate for safety-critical and low-confidence cases is not an optional feature; it is the architectural boundary that makes the rest of the automation safe to deploy. Defining the confidence threshold, the point at which the system routes a case to a technician rather than proceeding automatically, is the most important design decision in the whole project, and it needs to be agreed with the plant manager and a senior technician before go-live, then reviewed quarterly as the model accumulates more labelled data.

The honest tradeoff is this: a Hybrid system is more complex to build and govern than a simple automated pipeline. The confidence threshold requires ongoing calibration. The RAG pipeline over ten years of free-text notes will need a data cleaning sprint before it produces reliable results. And the system will concentrate diagnostic knowledge in a model that the automation engineer maintains, which creates a succession risk that needs to be managed explicitly. None of these tradeoffs change the recommendation, but they all need to be planned for. The projected payback period of around five months, based on 65 percent automation of current manual effort, is realistic if the data quality work is done properly and the pilot phase is used honestly to refine the model rather than treated as a formality before go-live.

The two closest alternatives to the recommended Hybrid path are a fully automated AI Workflow and an autonomous AI Agent. Both would handle the routine majority of fault diagnoses reasonably well, and either could serve as the backbone of a workable system. The reason neither is sufficient on its own comes down to two things that are specific to this process: the explicit regulatory requirement that safety-critical decisions remain with a qualified technician, and the reality that 10 to 15 percent of faults do not match any standard pattern and genuinely require contextual human judgement. A fully automated AI Workflow has no mechanism for routing those cases to a person before a ticket is actioned. An autonomous AI Agent could query logs, retrieve repair history and draft a ticket without any human involvement, which is precisely what the compliance requirement prohibits for the harder cases. The Hybrid path resolves this not by weakening the automation but by adding a hard routing gate: high-confidence routine diagnoses flow straight to ticket creation, while anything below the agreed confidence threshold, or flagged as safety-critical, lands in a technician review queue before any action is taken.

What makes the Hybrid path the clear winner for this specific process is that it matches the actual shape of the work. The majority of the 2,400 annual cases are pattern-matching exercises against structured logs and historical notes, which an AI system can do faster and more consistently than a person searching from memory. The minority are genuinely difficult, and those are exactly the cases where a skilled technician adds irreplaceable value. The Hybrid design does not try to replace that judgement; it clears the routine workload away so that technicians can spend their time on the cases that actually need them. Traditional Code and RPA were both considered and scored significantly lower, for straightforward reasons. Traditional rule-based code could handle a subset of known fault codes in the structured historian data, but it cannot reason across ten years of free-text repair notes, and that unstructured history is where a large part of the diagnostic value sits. RPA is well suited to navigating fixed interfaces and copying structured data between systems, but the diagnostic step here is a reasoning problem, not a navigation problem, and RPA contributes nothing to it.

The build starts with a data audit before any model work begins. The automation engineer queries the historian database via its existing Python interface and takes a representative sample of the free-text repair notes from the maintenance system to assess consistency, completeness and the range of terminology in use. Ten years of notes written by multiple technicians will contain abbreviations, shorthand and gaps, and the audit surfaces exactly where cleaning and normalisation are needed before those notes become reliable retrieval material. The output of this phase is a cleaned, embedded corpus of historical repair records stored in a vector database, with a retrieval-augmented generation pipeline in Python that can accept a live fault event and return the most semantically relevant past cases as context. LangChain or LlamaIndex are natural choices for orchestrating this pipeline given the automation engineer's existing Python fluency, and a hosted embedding model such as OpenAI's text-embedding-3-small or a self-hosted alternative via Ollama keeps the data within the organisation's control if that is a requirement.

With the retrieval layer in place, the next phase builds the inference engine. When a machine throws a fault, the historian database emits the structured log data through the existing Python integration. The system pulls that data, constructs a prompt that combines the structured log context with the top retrieved repair cases, and calls a language model to produce a ranked list of probable root causes with supporting evidence drawn from both sources. GPT-4o or a comparable model is appropriate here given the reasoning complexity involved. Every inference step, the retrieved cases, the prompt, the model's reasoning and the proposed root cause, is written to a structured log that feeds directly into the maintenance file, satisfying the audit requirement by design rather than as an afterthought.

The human-in-the-loop gate sits between the inference engine and ticket creation. A confidence score is derived from the model's output and matched against a threshold agreed in advance with the plant manager and a senior technician. Cases above the threshold and not flagged as safety-critical are passed automatically to the ticket creation step, where the maintenance system integration writes the parts list and recorded reasoning without further human input. Cases below the threshold, or carrying a safety-critical flag based on fault code classification, are pushed to a technician review queue, implemented as a lightweight web interface or a structured email notification depending on what the team will actually use. The technician sees the AI's proposed diagnosis and the supporting evidence, approves or corrects it, and the corrected outcome is written back to the maintenance system and added to the training corpus for future improvement.

The system runs in shadow mode for three to four weeks before go-live, processing live faults in parallel with the existing manual process without replacing it. Every case where the AI proposal and the technician's manual diagnosis diverge is reviewed together to refine the confidence threshold and identify retrieval gaps. This shadow period is also when the team builds trust in the system: technicians who can see the AI's reasoning and check it against their own judgement are far more likely to rely on it confidently once it goes live. After shadow mode, the manual email and spreadsheet workarounds are retired and the hybrid system becomes the single point of record for fault diagnosis across all four technicians and the plant manager.