XAIminer: Understanding AI Decisions Interactively

XAIminer is a browser-based tool that makes it possible to trace what an AI model bases its decision on. In the XRAISE research project it is applied to a question from automated rail operations: are there people close to the track or not?

Why interactive rather than prescribed?

How useful an XAI method is depends on the task, the AI model, its performance and the image. A method that carries in one context can be uninformative or even misleading in another. This is why the XRAISE project prescribes no single method: assessors should be able to choose, switch and place results side by side as the situation demands. Whether that works is examined in an experimental user study and a focus group with safety assessors.

In the image above both models detect the person reliably (100 % and 91 %). The Grad-CAM maps show, however, that only ConvNeXt-T (centre) looks at the person — VGG16 (right) relies on the lower right edge of the image. Right answer, questionable reasoning.

What the system does

XAIminer runs in the browser and is modular: any number of panels side by side, each with its own AI model, training state, XAI method (Grad-CAM, LRP, CRAFT, CRP) and filters. Panels can be linked so that all of them show the same image. For every image the classification probabilities and the true class are shown; anything noticeable can be annotated.

In the image above, the same network type in two training states: one reports "person" (91 %) although there is none — the LRP map (centre) shows that it responds to the building at the edge of the image (Clever Hans effect). The clean training state (right) decides correctly.

Background: XRAISE

In automated train operation at grades GoA3 and GoA4, a technical system has to monitor the track ahead and detect obstacles — today the train driver's task. That cannot be done without machine learning. Yet the safety case under EN 50716 cannot be applied to deep neural networks, because their components are not interpretable the way conventional software is. XRAISE examines whether explainable AI methods can narrow this gap — and what risks arise when people interpret XAI results. XAIminer is the tool assessors use to compare these results.

Partners: TU Dresden (Chair of Fundamentals of Electrical Engineering; Chair of Engineering Psychology and Applied Cognitive Research), EYYES Deutschland GmbH, PECS-WORK GmbH.

Demo version: XAIminer online
Duration: 2024-2026
Funding: DZSF beim Eisenbahn-Bundesamt
Contact: Sascha Weber, Carsten Knoll