Bálint Gyevnár

Autonomous multi-agent systems (MAS) are useful for automating complex tasks but raise trust concerns due to risks such as miscoordination or goal misalignment. Explainability is vital for users’ trust calibration, but explainable MAS face challenges due to complex environments, the human factor, and non-standardised evaluation. Leveraging the counterfactual effect size model and LLMs, we propose Agentic eXplanations via Interrogative Simulation (AXIS). AXIS generates human-centred action explanations for multi-agent policies by having an LLM interrogate an environment simulator using prompts like ’whatif’ and ’remove’ to observe and synthesise counterfactual information over multiple rounds. We evaluate AXIS on autonomous driving across ten scenarios for five LLMs with a comprehensive methodology combining robustness, subjective preference, correctness, and goal/action prediction with an external LLM as evaluator. Compared to baselines, AXIS improves perceived explanation correctness by at least 7.7% across all models and goal prediction accuracy by 23% for four models, with comparable action prediction accuracy, achieving the highest scores overall. Our code is open-sourced at this https URL.

Narratives about AI safety have recently ignited significant debate, with profound implications for policy decisions and resource allocation in AI development and regulation. Two distinct perspectives have emerged: one views AI safety primarily as a project for minimizing existential threats of advanced AI, while the other sees it as a natural extension of existing technological safety practices, focusing on immediate and concrete risks of current AI systems. This paper conducts a systematic literature review of primarily peer-reviewed research on AI safety to empirically investigate the diversity of studied safety risks associated with AI systems and their corresponding mitigation strategies. Our review shows a vast array of concrete research, including numerous AI system engineering studies, that address safety concerns irrespective of existential risk considerations. This research includes areas like adversarial robustness and interpretability, highlighting the immediate relevance and importance of AI safety for current AI systems. Based on these empirical findings, we argue that primarily tying AI safety research to efforts for minimizing the existential risks of advanced AI presents an overly narrow perspective on the breadth and scope of AI safety research. Instead, the perception of the scope and aims of AI safety research must be more epistemically inclusive, embracing the diverse concerns and aspirations that shape the ongoing discussion about safe AI. We recognize that the focus on existential risks remains a contested topic within the AI community and more broadly, with many researchers actively engaged in safe AI development and deployment holding neutral or dissenting views about existential risks from advanced AI. Consequently, given our findings, we advocate for a conception of AI safety that is pluralistic and incorporates a wider range of safety considerations, motivations, and perspectives.

It is often argued that effective human-centered explainable artificial intelligence (XAI) should resemble human reasoning. However, empirical investigations of how concepts from cognitive science can aid the design of XAI are lacking. Based on insights from cognitive science, we propose a framework of explanatory modes to analyze how people frame explanations, whether mechanistic, teleological, or counterfactual. Using autonomous driving, a complex safety-critical domain, we conduct an experiment consisting of two studies on (i) how people explain the behavior of a vehicle in 14 unique scenarios (N1=54), and (ii) how they perceive these explanations (N2=382). Our main finding is that participants deem teleological explanations significantly better quality than counterfactual ones, with perceived teleology being the best predictor of perceived quality. Based on our results, we argue that explanatory modes are an important axis of analysis when designing and evaluating XAI and highlight the need for a principled and empirically grounded understanding of the cognitive mechanisms of explanation.

We present CEMA: Causal Explanations in Multi-A gent systems; a framework for creating causal natural language explanations of an agent’s decisions in dynamic sequential multi-agent systems to build more trustworthy autonomous agents. Unlike prior work that assumes a fixed causal structure, CEMA only requires a probabilistic model for forward-simulating the state of the system. Using such a model, CEMA simulates counterfactual worlds that identify the salient causes behind the agent’s decisions. We evaluate CEMA on the task of motion planning for autonomous driving and test it in diverse simulated scenarios. We show that CEMA correctly and robustly identifies the causes behind the agent’s decisions, even when a large number of other agents is present, and show via a user study that CEMA’s explanations have a positive effect on participants’ trust in autonomous vehicles and are rated as high as high-quality baseline explanations elicited from other participants. We release the collected explanations with annotations as the HEADD dataset.

The European Union has proposed the Artificial Intelligence Act which introduces detailed requirements of transparency for AI systems. Many of these requirements can be addressed by the field of explainable AI (XAI), however, there is a fundamental difference between XAI and the Act regarding what transparency is. The Act views transparency as a means that supports wider values, such as accountability, human rights, and sustainable innovation. In contrast, XAI views transparency narrowly as an end in itself, focusing on explaining complex algorithmic properties without considering the socio-technical context. We call this difference the “transparency gap”. Failing to address the transparency gap, XAI risks leaving a range of transparency issues unaddressed. To begin to bridge this gap, we overview and clarify the terminology of how XAI and European regulation – the Act and the related General Data Protection Regulation (GDPR) – view basic definitions of transparency. By comparing the disparate views of XAI and regulation, we arrive at four axes where practical work could bridge the transparency gap: defining the scope of transparency, clarifying the legal status of XAI, addressing issues with conformity assessment, and building explainability for datasets.