Text: Anna Gullers, KTH.
An AI system may perform well in testing and still encounter a situation it was never designed to handle. How can we tell when it is approaching its limits? And know when not to trust its decisions? These questions are at the heart of Jenni Reuben’s research into trustworthy autonomous systems.
As researcher at Saab and a WASP industrial postdoctoral researcher at KTH, Jenni Reuben investigates how explainable AI and real-time monitoring can help people understand an autonomous system’s decisions and recognise when its behaviour may no longer be reliable.
What initially sparked your interest in AI, and in trustworthy AI in particular?
Early in my career as a research scientist, I worked with machine learning models designed to detect when sensitive data was being transferred out of a protected system without authorisation. The models learned to recognise patterns in data that could indicate such an attack. Seeing how a computer could learn to recognise patterns and spot suspicious activity made the potential of machine learning tangible to me.
My PhD in privacy and security gave me a good understanding of how much we depend on complex computer systems. Can we justifiably rely on system’s outputs?
I brought that perspective to autonomous systems that act in the physical world, where an incorrect decision can have irreversible consequences beyond the machine. Reinforcement learning (RL) interests me because teaches systems to develop strategies for achieving a goal. I want to understand when we can trust those strategies and what evidence stakeholders need to judge their reliability.
For someone who doesn’t work with reinforcement learning, how would you explain what you research in simple terms?
Reinforcement learning is a way of teaching a machine to make decisions by learning from the outcomes of its actions. For autonomous aircraft, we can train it through repeated simulated missions, giving it feedback on how well it achieves the mission’s objectives.
Imagine an autonomous aircraft helping to search for a missing person. It must make a series of decisions, which area to search for next, whether to investigate something its camera has detected, and when to return before its battery runs too low. Spending more time investigating one location might be useful, but it leaves less time and energy to search elsewhere.
Reinforcement learning offers a way to learn a strategy, or policy – in this case, a search strategy that considers those longer-term consequences. My research asks how dependable such learned strategies are when conditions change, and how explanations and runtime monitoring can help humans judge when to rely on them.
You lead a research project at Saab on trustworthy AI, with a focus on explainable AI. Why is it important to understand how an AI system arrives at a decision, rather than just whether the decision is correct?
Yes, the research project is called ASTRA-COA. In addition to safety questions in autonomous systems trained using RL, developers, operators and regulators need evidence to assess an autonomous system’s behaviour and limitations. Explainable AI (XAI) can help make a system’s decisions more interpretable by showing which inputs appear to have influenced them.
Imagine a simulated air-traffic scenario in which two autonomous aircraft are approaching the same intersection at the same altitude. An AI-assisted controller recommends slowing one aircraft to maintain safe separation. An explanation could show the operator which aircraft positions and speeds influenced that recommendation.
The key questions are whether these explanations help different stakeholders know when to trust the system, and when to question its decisions. Does an explanation help an operator recognise when the system’s decision deserves closer scrutiny? How do you evaluate whether the explanation faithfully reflects the system’s decision? Does an explanation help the decision maker using it to take appropriate action?
You have commented on the challenge of recognising when an AI agent is operating outside the conditions it was designed for. How does this connect to your work on explainable AI?
An RL policy may perform well in training and testing but encounter unfamiliar conditions in operation. My research investigates how runtime monitoring can detect such deviations and how explanations can help people assess their significance.
Imagine weather-related diversions creating an air-traffic situation very different from those used to train an autonomous aircraft system. A runtime monitor could flag the unfamiliar situation and trigger a safety mechanism or prompt a human operator to question the system’s recommendation. My research explores how explanations and monitoring can work together to support those judgement.
Explainable AI could make the warning actionable by showing which observations or changes in behaviour contributed to it. Together, monitoring and explanations could help people judge when to rely on the system and when a fallback response may be needed.
You work at Saab while also being a postdoc at KTH and affiliated with TECoSA. What does moving between academia and industry bring to your work?
Industry brings concrete engineering constraints, operational challenges and the needs of different stakeholders. Academia gives me the opportunity to examine the assumptions behind those questions and evaluate the methods systematically.
At KTH, working with Martin Törngren and being affiliated with TECoSA gives me a setting in which to connect interpretable machine learning methods with the behaviour and safety questions of the overall system. I want to use that connection to understand how an explanation and runtime monitor fits into a wider engineering process.
Your postdoc is funded through WASP. What possibilities will this open for you as a researcher?
WASP funding allows me to pursue a focused research project at KTH while maintaining a strong connection to applied autonomy research at Saab. It gives me the opportunity to develop an independent research direction, publish scientific contributions and build collaborations.
The wider WASP community give me the opportunity to strengthen my publication record, broaden my academic collaborations, and develop my research leadership.
Looking five years ahead, what do you hope we will have become better at when it comes to making AI systems work safely in the real world?
For autonomous systems, I would like to see clearer connections between human judgement, the monitoring of changing operating environment and an engineered response like graceful degradation when the system approaches its limits.
As humans and autonomous systems increasingly work together on challenges such as wildfire management and national defence, we need better safety mechanisms, ways to help different stakeholders critically assess system behaviour, and infrastructure that makes systems’ intentions and limitations easier to understand.
You spend quite a lot of time on the train between Stockholm and Linköping. What do you usually do during those two and a half hours?
I usually catch up on emails, get some rest or read. On one journey, I came across a paper on AI safety that changed how I think about the field. It highlighted risks that can emerge even when individual AI models perform as intended, and it sparked a growing interest in the wider societal and organisational challenges of deploying autonomous systems.
About Jenni Reuben
Education: PhD in computer science and engineering from Karlstad University, with research in privacy and security in big data processing.
Research: Trustworthy autonomous systems, explainable AI and reinforcement learning.
Work: Researcher at Saab, working on autonomous aircraft systems and co-leading its internal autonomy Research and Technology (R&T) cluster.
Research at WASP and KTH: WASP-funded industrial postdoctoral researcher at KTH.
Research environment: Affiliated with TECoSA, KTH, https://www.tecosa.center.kth.
Published: November 4th, 2026
[addtoany]