Multilingual Datasets for Custom Input Extraction and Explanation Requests Parsing in Conversational XAI Systems (2025)

Abstract: Conversational explainable artificial intelligence (ConvXAI) systems based on large language models (LLMs) have garnered considerable attention for their ability to enhance user comprehension through dialogue-based explanations. Current ConvXAI systems often are based on intent recognition to accurately identify the user’s desired intention and map it to an explainability method. While such methods offer great precision and reliability in discerning users’ underlying intentions for English, a significant challenge in the scarcity of training data persists, which impedes multilingual generalization. Besides, the support for free-form custom inputs, which are user-defined data distinct from pre-configured dataset instances, remains largely limited. To bridge these gaps, we first introduce MultiCoXQL, a multilingual extension of the CoXQL dataset spanning five typologically diverse languages, including one low-resource language. Subsequently, we propose a new parsing approach aimed at enhancing multilingual parsing performance, and evaluate three LLMs on MultiCoXQL using various parsing strategies. Furthermore, we present Compass, a new multilingual dataset designed for custom input extraction in ConvXAI systems, encompassing 11 intents across the same five languages as MultiCoXQL. We conduct monolingual, cross-lingual, and multilingual evaluations on Compass, employing three LLMs of varying sizes alongside BERT-type models.

## introduction to improve the transparency of llms while ensuring efficiency and user comprehension, conversational xai systems have recently emerged @xcite @xcite @xcite @xcite . several systems have since been developed, e.g., talktomodel @xcite , inter-rolang @xcite and llmcheckup @xcite . these systems include user interfaces that facilitate users to interact in natural language with a system and rely on intent recognition foot_1 . intent recognition is a key upstream component in convxai systems, focusing on accurately interpreting user inputs from multiple perspectives @xcite and mapping user intents to the corresponding explainability methods, which enables explanations that are as faithful as the underlying method allows @xcite . nevertheless, intent recognition remains challenging for convxai, due to the scarcity of training data, particularly multilingual data, and the specialized nature of the xai domain, which involves mapping requests across a diverse range of xai methods. @xcite presents the first and, to-date, largest dataset, coxql, for intent recognition in convxai, but it is limited to english. entering queries in other languages may either result in undesired explanations or prevent users from fully accessing the convxai capabilities, highlighting the need to extend intent recognition to multiple languages for effective use in multilingual scenarios @xcite . furthermore, coxql is restricted to an existing dataset with fixed data points ( §4) that can be queried by the user and explained, as in talk-tomodel, interrolang, and llmcheckup. the deficiency is that users are unable to freely explore custom input based on their preferences with the explained llms, which hinders the generalizability of convxai systems. one goal of convxai systems is to support more adaptive usage ( §4), allowing personalization @xcite , user engagement @xcite , and figure 1 : example parallel utterances from compass, including the user question, corresponding intent, and extracted custom input, are provided in english (en), russian (ru), chinese (zh), german (de), and telugu (te). the example question is requesting reasoning (parsed as "rationale", which should provide a natural language explanation) and the corresponding extracted custom input comes from the sst2 dataset. efficient use of systems @xcite . this can be realized by allowing custom input (figure 1 ). nonetheless, the current custom input extraction in the context of convxai has been largely constrained by the lack of suitable datasets. to address these gaps, we first extend coxql @xcite to support multiple languages, called multicoxql (figure 2 ). multicoxql is created by machine translating coxql instances, while preserving their intent annotations, covering 5 languages: german, chinese, russian, telugu and english. we assess the quality of machine translation through human annotators who assess meaningfulness and correctness. the chinese and german translations are of high quality (figure 5 ), and the similarity between the original english text and the translated text improves after human correction. secondly, we evaluate the effectiveness of one baseline and three state-of-the-art parsing approaches in convxai on multicoxql. to improve upon the limited cross-lingual generalization of existing approaches, we propose guided multi-prompt parsing (gmp), which combines existing methods and noticeably enhances multilingual parsing accuracy. thirdly, we present the compass dataset foot_2 (figure 1 , figure 3 ) for enabling custom input in convxai. it includes user questions, extracted custom inputs, and corresponding intents across the five aforementioned languages. we conduct monolingual, cross-lingual, and multilingual evaluations on compass using (m)bert and three decoder-only llms. for intent recognition, fine-tuned bert performs comparably to that of the llms and outperforms them in chinese, german, and telugu. for custom input extraction, out of four approaches, gollie @xcite performs best with smaller llms, while naïve few-shot prompting presents the best results with larger llms. ## related work parsing in convxai systems in most prior con-vxai systems @xcite @xcite , parsing is achieved by comparing the semantic similarity between user queries and a predefined set of example utterances, often resulting in relatively low parsing accuracy. in contrast, talktomodel @xcite converts user questions into sql-like queries for parsing and employs guided decoding (gd) by defining a grammar to constrain the output vocabulary. similarly, interrolang (feldhus et al., 2023) uses finetuned models for slot tagging to perform parsing, while wang et al. (2024a) introduces multi-prompt parsing (mp), which hierarchically parses user intent from coarse-grained to fine-grained slots. to ensure compatibility with various xai operations, mp with template checking (mp+) (wang et al., 2024b) is proposed to validate the output and fulfill operational requirements. our work further investigates the effectiveness of gd, mp, and mp+ in multilingual settings and proposes a new approach, guided multi-prompt parsing, which substantially enhances multilingual parsing performance. multilingual dataset annotation xnli (conneau et al., 2018) extend multinli (williams et al., 2018) to 15 languages, including low-resource languages, to facilitate cross-lingual natural language inference. min et al. (2019), tuan nguyen et al. 2020), and bakshandaeva et al. (2022 broaden spider dataset @xcite , a widely recognized text-to-sql dataset in english that encompasses queries of varying complexity, by translating it into target languages. multispider is subsequently created as a multilingual text-to-sql dataset, covering seven popularly used languages @xcite . @xcite introduce the multitacred dataset, created by translating tacred @xcite , a dataset for information extraction, into 12 typologically diverse languages from nine language families. our work is closely aligned with prior research on multilingual dataset creation and annotation, and adheres to established best practices in the field. ## the multicoxql dataset coxql @xcite is a text-to-sql dataset for intent recognition in convxai systems and comprises user questions and gold labels (sql-like queries) in english. coxql covers 31 operations, including explainability and supplementary operations foot_3 ( the multicoxql dataset introduced in this work encompasses five languages: english, german, russian, chinese, and telugu (figure 2 ). these languages are selected for their typological diversity, representing a spectrum from widely spoken to low-resource languages that use different scripts. following @xcite and @xcite , we translate the entire train and test splits of coxql into the target languages using gemini-1.5-pro foot_5 @xcite and name it mul-ticoxql (figure 2 ), whose translation quality is evaluated in §7.1. note we only translate the user questions, not the gold labels, in order to maintain consistency in the label space. finally, we store the translated instances in the same json format as the original coxql english dataset. ## the compass dataset custom input (figure 1 ) refers to user-defined or task-specific data provided to convxai, distinct from instances found in pre-configured datasets (figure 2 ). in previous convxai systems, users can only query instances from pre-configured datasets using their dataset id, while the main challenge in custom input is for llms to explicitly interpret and extract relevant information from the user's question. custom input allows users to explore convxai systems according to their individual preferences, thereby enhancing system flexibility, generalizability, and extensibility @xcite . however, no publicly available dataset currently addresses this type of input for information extraction tasks within convxai systems. compass is therefore constructed to address this gap and serves as a synergistic complement to (multi)coxql, offering users diverse input text formats. to maintain consistent language selection, compass adopts the same five languages as multicoxql ( §3). ## models we select three open-source, state-of-the-art decoder-only llms with increasing parameter sizes from distinct model families: llama3-8b (ai@meta, 2024), phi4-14b (abdin et al., 2024), and qwen2.5-72b @xcite to evaluate the 9 further details on gmp can be found in appendix c. 10 prompts for each of the following four approaches are provided in appendix e. multicoxql and compass datasets. these models are selected because they have been trained on at least one of the non-english target languages we chose ( §3). in contrast to few-shot prompting, for fine-tuning, we employ bert and mbert models @xcite to conduct monolingual, crosslingual, and multilingual evaluations ( §7.2) foot_8 . ## conclusion in this work, we first extend the coxql dataset for intent recognition in convxai to a multilingual version, multicoxql, covering five languages, in-cluding one low-resource language, using machine translation followed by human evaluation and correction. we benchmark state-of-the-art explanation request parsing approaches on multicoxql using three different llms. second, we propose a new approach, guided multi-prompt parsing, which integrates the strengths of existing methods and substantially improves parsing accuracy in multilingual settings. third, we introduce the compass dataset for coarse-grained intent recognition and custom input extraction in convxai, incorporating the same five languages as multicoxql. we conduct comprehensive experiments using three llms, along with (m)bert, on compass, to evaluate performance in monolingual, cross-lingual, and multilingual scenarios. we observe that cross-lingual mbert underperforms compared to monolingual mbert, whereas multilingual mbert outperforms both. for the task of custom input extraction, gol-lie proves to be more effective for smaller llms, while naïve few-shot prompting yields better results with larger llms.