Current Projects
[XRAISE] Explainable AI for Railway Safety Evaluations
Can the computer vision (CV) system of an autonomous train detect with sufficient certainty whether a deviation in the captured image data is caused by a shadow or perhaps by a child playing on the tracks? And during the safety assessment, how can a person be sure that the CV system is not falsely overconfident? How can he ensure that his safety assessment is sufficiently secure - especially when the technology of the CV system makes it fundamentally impossible to provide the relevant evidence?
In the development of future CV systems for collision avoidance in rail traffic, it must be possible to evaluate their safety based on the results of deep neural networks (DNN). However, standards from the railway sector for the development of functionally safe systems (e.g. EN 5012x) cannot be applied to statistical statements from DNN. The methodology of 100% test coverage is not affordable for artificial intelligence (AI). The dimensionality of the property spaces within the networks and possible chaotic behavior prevent a deterministic approach. In addition, these complex interactions are opaque and the way the networks work is like a black box. To make this black box more transparent, explainable artificial intelligence (XAI) methods can be applied.
XAI offers great potential for safety testing of railway-specific CV systems, but also carries the risk of cognitive biases. The following application scenario is assumed for their experimental testing: For a selection of images, XAI highlights indicate which areas the DNN has taken into account in order to decide on the presence of collision objects. Based on this information, safety auditors should evaluate whether and under what conditions the CV system makes correct decisions and when problems arise. The following factors are varied in six experiments: (1) the XAI method used, (2) three sensor types and their multimodal combination, (3) various test aspects in the form of collision objects and environmental conditions and (4) the reliability of the CV system. The following explains which work steps are necessary to make this evaluation possible. In order to clarify which test aspects need to be taken into account and which technological foundations are required, the first step is to carry out a requirements analysis and define the architecture of the CV system. In order to ensure that the most suitable XAI methods are used effectively, a literature review is carried out in the second step. XAI methods suitable for the railway context are being researched, as well as their effects on human users, their side effects and possible measures to combat distortions. Based on this basis, an evaluation framework is specified in the third step. This also includes the implementation of the technical basics (CV system with DNN and XAI). In the fourth step, six experiments are carried out in which the XAI methods are compared and their effects are critically examined. At the end of the project, the results are converted into recommendations for action.
As part of the XRAISE project, an interactive, web-based assistance system is being developed to evaluate AI decisions: XAIminer
Duration: 2024-2026
Funding: Eisenbahn-Bundesamt
Contact: Romy Müller
[INAK] Interaktives nutzerzentriertes Assistenzsystem für die Knochenchirurgie im Kopfbereich
Noise is generated when milling the skull bone. This noise cannot be avoided during cochlear implant surgery, but it can certainly be reduced so that the patient's remaining hearing is not damaged. However, this presents surgeons with the challenging task of adapting various process parameters to suit the situation. How can AI be used to assist surgeons in this task?
The project is developing a self-learning AI-based assistance system for ear surgery. This system supports surgeons in carrying out milling work on the skull bone in such a way that patients are protected from damage caused by noise, heat or injuries to nerves and vessels. To do this, the system uses modern sensor technology to record various process values and uses AI methods to combine these values in conjunction with individual patient data to predict when the situation threatens to become dangerous for the patient. The results of the AI algorithms should be communicated to surgeons in such a way that they can understand the current situation more precisely and are supported in their decision-making.
Duration: 2024-2026
Funding: SAB
Contact: Romy Müller
[MoCa-Dia] Modelling and visualising causal relations in process chains: How do multi-linear, systemic, and combined methods affect human fault diagnosis?
Machine faults in the processing and packaging industry often result from previous production steps. Such causes usually are unknown to operators, making it hard for them to correctly diagnose faults. Operator support systems could foster an understanding of causal relations in process chains, for instance by presenting causal diagrams. However, such multi-linear fault models do not adequately represent the complex interactions in a system. This can be achieved by systemic modelling methods. They reveal how different system functions interact and how this can give rise to undesirable, emergent effects. However, these models might be hard to understand and use for fault diagnosis due to their visual complexity and their lack of diagnostically relevant information about observable symptoms. Therefore, method combinations are needed, but it is unclear what information they should represent in what ways, and how this would affect human fault diagnosis. These issues are investigated in the present project.
In the modelling part of the project, we model the causal relations in a process chain, using a multi-linear, a systemic, and two combined methods. The combinations are either based on causal diagrams or on networks of functional relations, in which they integrate information from the respective other method. Moreover, they provide information on different levels of abstraction, thus supporting the recognition of general principles as well as the integration of functional knowledge with concrete observations of symptoms. Subsequently, we test the generalisability of the models by transferring them to three other packaging lines that vary in their similarity to the first one. Finally, we formalise the models in a domain-specific ontology of causal relations.
In the empirical part, we investigate how visualisations of the four models affect human diagnostic processes and performance, as well as their learning and understanding of causal relations. Four experiments use one of the model visualisations each, and vary two situational factors that might affect the impacts of model visualisations (i.e., complexity of faults and availability of visual highlighting). A fifth experiment compares the models in one and the same experiment. We measure participants’ speed and accuracy of selecting fault causes, their examination of relevant process parameters, and their recall and inference of causal information. The following hypotheses are tested: (1) for simple faults, causal diagrams are useful but the systemic model is not, (2) for interactions, both basic models are deficient (because causal diagrams do not adequately represent them, while systemic models are too complex and lack diagnostically relevant information), (3) the combined visualisations retain the benefits of both methods and mitigate the problems. The results serve to adapt our models and integrate the best one in a concept for operator support, which we evaluate with machine operators.
Duration: 2024-2027
Funding: DFG
Contact: Romy Müller
[XAI-Dia] Explainable artificial intelligence for fault diagnosis: Impacts on human diagnostic processes and performance
Fault diagnosis in industrial settings is a challenging task. Although it can be supported by machine learning (ML), human-machine cooperation is essential to monitor and evaluate ML algorithms. However, this is hampered by the fact that ML relies on black box models. To increase its transparency, explainable artificial intelligence (XAI) can indicate which inputs a ML has used to compute a solution. For instance, this can be achieved by highlighting the specific areas in images that were attended by the ML algorithm. Previous research has revealed benefits and pitfalls of XAI in other task contexts, but it is unclear how XAI affects human diagnostic processes and performance during fault diagnosis. Specifically, it needs to be investigated under what conditions XAI helps people to critically evaluate the ML results or leads them to over-rely on incorrect explanations.
The present project investigates how diagnostic processes and performance are affected by XAI that explains ML outcomes on three levels: anomaly detection, fault classification, and fault diagnosis. XAI for detection and classification is implemented by highlighting areas in product images that were attended by the algorithm. XAI for diagnosis informs people which process parameters from the previous production step it has used. In a computer-based chocolate production scenario, participants either receive XAI information or are only informed about the results of ML algorithms. Their task is to evaluate these results. Besides the presence of XAI, we vary the correctness of ML and the difficulty of the task.
To assess diagnostic performance, we analyse solution times and correctness of participants’ response. To assess diagnostic processes, we analyse eye movements and diagnostic actions aimed at cross-checking the ML results. We hypothesise that XAI improves diagnostic speed when ML results are correct, and diagnostic accuracy when they are incorrect. However, we expect these effects to depend on the type of ML error. Specifically, we hypothesise that participants tend to over-rely XAI when additional faults besides the highlighted one are present. Moreover, we expect the effects of XAI to vary with task difficulty. To test these hypotheses, we conduct three experiments, one for each level of ML and XAI (i.e., detection, classification, diagnosis). In addition to these experiments, we conduct a pilot study to select suitable stimuli and a user study to investigate the interpretability of XAI outputs.
Finally, in a field study we investigate to what degree our experimental results can be transferred to expert fault diagnosis in a real plant. Taken together, these studies describe how diagnostic performance is affected by XAI and explain these effects by providing insights into the underlying diagnostic processes.
Duration: 2022-2025
Funding: DFG
Contact: Romy Müller
[HyTec] Hypotheses During Diagnostic Problem Solving in Technical Domains: Mental Basis, Process, and Outcome
In diagnostic problem solving, hypotheses about the underlying fault causes play a central role and determine the quality of subsequent diagnoses. In a previous project, it was found that apprentices of car mechatronics use different strategies (case-based, computer-based, and model-based), and especially for difficult problems the reliance on mental models of the system is crucial. However, it is not sufficiently understood whether the strategies indeed have a causal effect on hypotheses generation, how the model-based strategy is applied to generate hypotheses, and whether this process differs between technical domains. Therefore, the aim of the present project is to study hypotheses generation and the reliance on mental models in depth.
Based on the method of the previous project, we suggest four studies. Apprentices of mechatronics specialized on either cars or packaging machines diagnose fault causes from their respective domain, either using a realistic car simulation or a real packaging machine. In subsequent phases, they first generate hypotheses and later test them to diagnose the cause of the fault. Log data, eye tracking data, and verbal protocols are used to characterize the process and outcomes of hypotheses generation. Moreover, in two studies participants are required to generate concept maps reflecting their mental models of the system, and the contents of these models are related to hypotheses generation behaviours.
Study 1 serves as a replication of the findings from our previous project on the consequences of three spontaneously applied strategies in car mechatronics, and complements them with a detailed characterization of the hypotheses generation process. Study 2 manipulates strategies in car mechatronics experimentally to allow for an examination of causal relations between strategies and the quality of hypotheses generation. In Study 3, the hypotheses generation of experts is investigated in both domains and expert mental models of the car or packaging machine are elicited. In Study 4, these expert models are used to evaluate the mental models of apprentices. It is investigated how the quality of mental models affects hypotheses generation processes and outcomes. In combination, the studies enable a theoretically and empirically founded description of hypotheses generation processes and the role of mental models in different technical domains. Based on this description, the studies further provide implications for how to support hypotheses generation processes during vocational education.
Duration: 2021-2024
Funding: DFG
Contact: Romy Müller
[CeTI] Centre for Tactile Internet with Human-in-the-Loop
As part of the digitalization men and machines become ever more interconnected. This does not only provide us with an ample source of information – rather, men and machines both access processes that need to be manipulated and controlled together. A well-functioning cooperative interaction between humans and cyber-physical systems (CPS) is a prerequisite and – at the same time – a chance to participate equally in society, e.g. regardless of age or any physical limitation.
The Cluster of Excellence CeTI pursues unique and interdisciplinary research with six central goals to enable people to interact in real time with interconnected, automated systems and set up mutual learning during human-machine interaction. These include the development of (1) intelligent networks and (2) a fundamental understanding of human learning and goal-directed behavior reliably linking people and making cooperative learning and interaction between humans and CPS possible without (noticeable) delay. Furthermore, (3) technological solutions extending the human sensor and actuator systems, (4) new haptic coding schemes coping with the augmented sensory information as well as (5) suitable electronics (flexible, fast, reconfigurable) will be designed and ultimately (6) these new developments will be transferred into the fields of robot-assisted medicine, industry (co-working) and innovative teaching and learning applications.
Contributing to the cluster‘s goals, we investigate how augmented multisensory perception and interaction between humans and CPS, including visual, haptic and auditory information, is affected by temporal factors and task context. For instance, in many use cases it will be necessary to consider the effects of long delays between the initiation of a control action and its execution by the system. In addition, adaptive interfaces will be developed for integrating multimodal perceptual information and optimizing augmented interaction between humans and CPS.
Duration: since 01/2019
Funding: DFG (Cluster of Excellence 2050)
Further information: TUD - Clusters of Excellence
Contact: Annika Dix