From Regulation to Interaction: Expert Views on Aligning Explainable AI with the EU AI Act (2025)

Abstract: Explainable AI (XAI) aims to support people who interact with high-stakes AI-driven decisions, and the EU AI Act mandates that users must be able to interpret system outputs appropriately. Although the Act requires users to interpret outputs and mandates human oversight, it offers no technical guidance for implementing explainability, leaving interpretability methods opaque to non-experts and compliance obligations unclear. To address these gaps, we interviewed eight experts to explore (1) how explainability is defined and perceived under the Act, (2) the practical and regulatory obstacles to XAI implementation, and (3) recommended solutions and future directions. Our findings reveal that experts view explainability as context- and audience-dependent, face challenges from regulatory vagueness and technical trade-offs, and advocate for domain-specific rules, hybrid methods, and user-centered explanations. These insights provide a basis for a potential framework to align XAI methods—particularly for AI and Natural Language Processing (NLP) systems—with regulatory requirements, and suggest actionable steps for policymakers and practitioners

## introduction and background with the increasing deployment of large ai models (e.g., pretrained language models and large language models (llms)) in high-stakes domains, their inherent "black-box" nature offers limited transparency into decision-making processes, posing significant risks. consequently, regulations such as the gdpr and the eu ai act require both transparency and explainability. this has paved the way for the development of extensive research in the explainable ai (xai) field. research in xai has been increasing, with numerous surveys focused on critical domains such as healthcare, finance, and law @xcite @xcite . moreover, information and knowledge management systems are increasingly integrating explainable ai methods to support trustworthy decision-making in different domains @xcite @xcite @xcite the eu ai act (commission, 2021) mandates transparency for high-risk ai systems via: article 13(1): systems must enable users to interpret outputs appropriately and article 14(4): human oversight must prevent undue reliance on ai decisions. however, the act provides no technical guidance for implementing explainability methods @xcite . existing xai techniques focus on algorithmic transparency but often remain opaque to non-experts and difficult to operationalize for intended users @xcite . moreover, the high-risk classification (art. 6(2)) is ambiguous, creating uncertainty about compliance obligations @xcite . consequently, many xai methods offer mathematical insight without satisfying legal standards for human interpretability @xcite . previous research shows that no standardized framework exists to assess whether xai methods satisfy the eu ai act's transparency mandates @xcite . second, legal expectations for explainability remain ambiguous, with no consensus on what qualifies as an acceptable explanation @xcite . third, research rarely integrates explainability techniques with compliance mechanisms, as most studies address xai methods and regulatory frameworks in isolation @xcite . despite extensive xai research, little work has engaged experts on how current methods align with the eu ai act's transparency and explainability requirements. with key compliance deadlines approaching (timeline, 2024), understanding practitioners' views on the feasibility and challenges of meeting these mandates is critical yet largely unexplored. aiming to address these gaps, we conducted an interview study with 8 domain experts from europe, addressing the following research questions (rqs): this work investigates domain experts' perspectives on the intersection of xai and the eu ai act, focusing on perceptions and the challenges of implementing xai to achieve compliance. this paper provides an empirical perspective grounded in expert interviews and offers evidence-based and actionable insights for aligning xai with regulatory demands. the remainder of the paper is structured as follows: section 2 introduces the research methodology; section 3 presents the results of our study; section 4 reviews related work in the literature. finally, section 5 discusses our findings, presents our conclusions, and outlines future work and study limitations. ## research methodology we followed a qualitative research approach and drew on the experiences of experts by conducting semi-structured interviews (ssi), which offer flexibility while retaining a structure for data collecting and enable a full analysis of participants' experiences @xcite . to examine the potential gaps and trends in the literature, we first conducted an initial literature review @xcite @xcite @xcite @xcite to build our knowledge base and as the foundation for the questionnaire. then, we interviewed experts and professionals to collect input on the intersection between xai and the eu ai act, focusing on understanding the context, interventions, mechanisms, and outcomes of xai implementations while addressing regulatory compliance and practical challenges. we describe below the study design, data collection, and data analysis of our research. ## related work recent studies have examined the intersection of xai and the eu ai act, highlighting some regulatory and technical challenges. @xcite conducted an interdisciplinary analysis of how the act addressed black-box ai systems, reviewing existing xai techniques and noting persistent limitations in meeting the act's objectives. using a case study on ai-based exam proctoring, they illustrated how transparency and human oversight could be implemented even when decisions were not inherently interpretable. similarly, @xcite analysed explainability requirements under european and german law, finding that different legal frameworks demanded distinct xai properties and that current techniques often failed to meet expectations, particularly regarding fidelity and confidence estimates. these shortcomings reflected technical trade-offs, as many post-hoc methods lacked faithfulness guarantees, posing compliance challenges. pavlidis (2024) explored various approaches to advance xai and discussed the difficulties of embedding explainability into governance and policy, with attention to standard-setting, oversight, and enforcement. @xcite further emphasized the context-dependent nature of explainability, noting that the appropriate form and degree of transparency depended on specific circumstances. they also observed that the act's obligations were largely aimed at professional users of high-risk ai systems rather than the general public, which could result in transparency measures that overwhelm non-expert users with overly technical details. while prior studies have examined the eu ai act's transparency provisions from legal, policy, or interdisciplinary perspectives @xcite @xcite , they have primarily focused on conceptual analyses, legal interpretations, or technical reviews of existing xai methods. while they provide valuable insights, they don't systematically engage with practitioners and domain experts to understand how explainability is interpreted in real-world settings, nor how the act's high-level mandates translate into concrete implementation practices. our study fills this gap by using semistructured interviews with eight experts in highstakes ai to examine how they interpret explainability under the eu ai act, the factors influencing their views, and the practical barriers to compliance, revealing nuances often overlooked in legal or technical analyses. our work contributes actionable insights by synthesizing experts' recommendations into potential solution concepts. ## discussion and conclusion our interview study findings with eight experts reveal that explainability is fundamentally contextand user-specific, challenging the eu ai act's broad transparency mandates. in practice, organizations struggle with three main obstacles: (1) regulatory vagueness, which leaves compliance criteria undefined, (2) the inherent accuracy-interpretability trade-off, forcing a choice between performance and clarity, and (3) limited technical expertise, which delays effective integration of xai tools. together, these factors could lead to superficial "box-checking" rather than genuine explainability. experts propose a threefold strategy to bridge these gaps. (1) domain-specific regulations would adapt transparency requirements to each sector's risk profile. (2) hybrid approaches combining highaccuracy models with simpler explainers can reconcile predictive performance with comprehensibility. our findings have several implications for different stakeholders. researchers are encouraged to develop evaluation frameworks that jointly assess legal compliance and end-user comprehensibility. policymakers and regulators should consider refining the eu ai act by adding sector-tailored criteria and establishing regulatory sandboxes for safe testing. on the other hand, practitioners can use these results as a basis to invest in interpretability training and adopt flexible xai pipelines. by synthesizing expert insights on definitions, challenges, and solution concepts, this work lays the groundwork for aligning xai methods with regulatory requirements and advancing trustworthy ai in high-risk domains. beyond regulatory alignment, our findings have implications for the design of explanation interfaces and interaction modalities, particularly those relying on natural language @xcite @xcite . incorporating user-centered design principles into explanation generation @xcite and presentation can help bridge the gap between legal compliance and meaningful human understanding, an intersection of growing interest in human-computer interaction and nlp research. unlike prior work, which has mainly offered legal or conceptual analyses of the eu ai act's explainability requirements, our study provides an empirical perspective grounded in expert interviews. by capturing how practitioners interpret explainability, the challenges they face in achieving compliance, and their proposed solutions, we offer evidence-based, actionable insights for aligning xai methods with regulatory demands. we present this study as work in progress and hope it catalyzes dialogue between hci and xai/nlp researchers, laying a foundation for further work on explainability under regulatory mandates. with this study, we aim to advance efforts to bridge the gap between human-and regulatororiented explainability and technical explainability in ai/nlp. future work in our current sample, interviewees are primarily based in europe, reflecting the particular relevance of the eu ai act in this region. as future work, we plan to conduct additional interviews with experts from both european and non-european countries to expand the sample and enable comparative analyses between eu and non-eu perspectives. a complementary direction is to field a large-scale survey to validate and generalize our qualitative findings, quantify key insights, and broaden their applicability. together, these extensions aim to support more concrete recommendations and implications for the design of explanation systems, including natural languagebased approaches. limitations this study's modest sample size, while comparable with relevant research @xcite , may limit generalizability beyond the examined contexts. however, thematic saturation appeared to be reached, as no new themes emerged in later interviews. all interviews were conducted in english, and although the sample in-cluded participants from diverse linguistic backgrounds, conducting them in additional languages could have broadened participation. we invite future research to build on and validate these findings through larger, more diverse samples. while the interviews were designed to follow a consistent and impartial approach, and reflexivity was actively maintained throughout the analysis, the findings may still carry the inherent biases and limitations of self-reported data @xcite . in addition, one limitation of these types of works (e.g., prior related work we discussed and our work) could be the evolving nature of the eu ai act, driven by the rapid development of ai research and practice. the act has undergone several key milestones. in our study, we referred to the latest version of the act available as of december 2024.