MASSIVE Multilingual A bstract M eaning R epresentation: A Dataset and Baselines for Hallucination Detection (2024)

Abstract: Abstract Meaning Representation (AMR) is a semantic formalism that captures the core meaning of an utterance. There has been substantial work developing AMR corpora in English and more recently across languages, though the limited size of existing datasets and the cost of collecting more annotations are prohibitive. With both engineering and scientific questions in mind, we introduce MASSIVE-AMR, a dataset with more than 84,000 text-to-graph annotations, currently the largest and most diverse of its kind: AMR graphs for 1,685 information-seeking utterances mapped to 50+ typologically diverse languages. We describe how we built our resource and its unique features before reporting on experiments using large language models for multilingual AMR and SPARQL parsing as well as applying AMRs for hallucination detection in the context of knowledge base question answering, with results shedding light on persistent issues using LLMs for structured parsing.

## introduction knowledge base question answering (kbqa) has a long history in natural language processing, with the task of retrieving an answer from a knowledge base such as wikidata or dbpedia @xcite integral to many large-scale question answering systems @xcite . in kbqa, a question is converted into a structured query language such as sparql, an executable semantic parse. however, data to train models is expensive, few multilingual resources are available, and performance is limited for long-tail queries, a problem compounded by arbitrary variability in formmeaning mappings across languages @xcite . most notably, research in multilingual kbqa is hindered by lack of data @xcite @xcite . following work using meaning representations for this problem, we create a dataset 20 times larger and with @xcite , massive-amr covers more languages, has more utterances, and has localized or translated entities for each language (see exs. our research contributions thus include: (1) creation of the largest-scale multilingual amr question corpus to date; (2) evaluation of llms on parsing of sparql and amrs structures across languages; and (3) design, development, and evaluation of generative models leveraging amrs for sparql relation hallucination detection. utterance amr massive-amr when was obama born (b / bear-02 :arg1 (o / "obama") :time (u / unknown)) quand est né sarkozy (b / bear-02 :arg1 (s / "sarkozy") :time (u / unknown)) +50 langs. +50 amrs, local entities qald9-amr who developed skype? (d / develop-02 qui a développé skype? :arg0 (u / unknown) :arg1 (s / "skype")) 9+ langs. same amr, all langs. we present related work in qa, knowledge base question answering (kbqa), the amr formalism, amrs for kbqa, and hallucination detection. ## data: corpus creation to create a corpus of multilingual amr graphs, we started with an existing dataset of qa utterances, tailored amr 3.0 guidelines to our use case, trained a team of professional annotators to create amrs for english utterances, and then made automatic mappings to multilingual utterances using existing entity mention spans, a process which from start to finish took three months. in this section, we report details about the data we started with, guidelines, and annotation agreement scores. acquiring scaleable multilingual data. we wanted a resource targeting a wide distribution of qa utterances and thus selected 1685 english examples from massive @xcite including entity annotations like in the multilingual examples in lang. example utterance en-us what is the population of [place: new york] sl-sl koliko prebivalcev ima [place: ljubljana] it-it qual è la popolazione di [place: roma] sq-al cila është popullësia e [place: tiranës] cy-gb beth yw poblogaeth [place: efrog newydd] af-za wat is die bevolking van [place: kaapstad] is-is hver er íbúafjöldi [place: reykjavíkur] az-az [place: sumqayıtın] @halisi n@q@d@rdir long-tail qa. many utterances in massive are described as long-tail, that is, associated with low user feedback in interactions with a digital assistant. in some cases, it is clear what increases friction (an incomplete utterance, or a speech-totext error). examining translations of english utterances provides insight (appendix a.2). localized entities. in comparable datasets @xcite , entities are shared across languages (e.g., english where did abraham lincoln die? corresponds to german wo starb abraham lincoln?). to address challenges of large-scale qa, massive entities are mostly language-specific, e.g. german questions target german entities (wo starb otto von bismarck foot_2 ). amr datasets differ in composition: amr 3.0 @xcite is based on news and other written discourse and consists of relatively few factoid or information-seeking questions (less than 10%). in contrast, massive-amr includes requests about currency conversions, quantities, comparative and superlatives, and simple arithmetic. for more details about how the corpora compare, see appendix a and annotation principles: canonical forms. in keeping with original amr guidelines, an amr captures meaning, not form @xcite . we hence prefer canonical forms for utterances like currency conversion and arithmetic: e.g., 'how much is the euro versus the dollar' and 'what is the euro worth compared to the dollar' map to similar graphs. likewise, arithmetic questions are associated with top node 'equal-01' even without token 'equal' present ('how much is two plus two' and 'sum of two and two' treated like 'what does two and two equal'). question-imperative continuum. it proved difficult to reach agreement for annotations of question versus imperative forms. in english, 'could you tell me the price of google', 'what is the price of google', and 'tell me the price of google' share the same meaning. however, treating the imperative (e.g., an embedded question 'tell me what the price is') as a question is out-of-line with amr 3.0. the guideline we adopt is to preserve imperative form and treat polite questions (e.g., english 'could you tell me the price') the same as base question forms (e.g., 'what is the price'). annotation agreement scores. 4-5 trained annotators created amrs for 1685 utterances, examining differences in batches of 200 weekly, with inter-annotator agreement ranging from 78-82% smatch, comparable to reported agreement for amr experts @xcite . we note that massive-amr consists of many similar questions and simple utterances, with on average 50% fewer tokens compared to amr 3.0 ( for non-english entities, we replace amr node labels using massive annotations. we note that not all utterances have annotations, and that a lack of entity alignments adds noise since often word order matters (e.g., currency conversion). to improve data quality, we manually curate validation and test sets (25% of total). ## sparql hallucination detection our original motivation for creating a multilingual amr dataset ( §3) was to help improve largescale qa systems. scaleable qa systems often utilize structured representations (e.g., sparql) for knowledge base retrieval, pairing a natural language utterance with an executable semantic query. the sparql in the wikidata or dbpedia case is straightforward: we get a question in, the system produces an answer out. however, in practice we simply need a system capable of judging if a given answer is correct, which using generative methods we study as hallucination detection. ## conclusion we present massive-amr, the largest and most diverse dataset of multilingual questions paired with abstract meaning representation (amr) graphs, which we publicly release for research purposes. we discuss the origins of the data, and detail the processes of dataset creation, curation, and quality control. to examine the utility of our dataset in controlled experimentation using large language models, we first consider the task of structure parsing, showing results for both amr graph and sparql query parsing across languages. overall, performance for amr parsing with in-context learning is less effective compared with reported state-of-the-art using fine-tuning; still, qualitative assessment of produced structures reveals many coherent, correct graphs despite low similarity with a ground truth. in comparison, sparql parsing performance is high across languages, at least in small studies using the qald9-amr dataset. one motivating factor behind the creation of massive-amr was to be able to test the utility of amrs for knowledge base question answering (kbqa), specifically ascertaining whether amrs can help detect incongruous sparql queries, essentially serving as a proxy confidence measure for the correctness of an answer suggested by a qa system. in these experiments, we first confirm that the gpt models do indeed hallucinate semantic relations, and then discover that 'easy' hallucination detection-asking a model to verify relations are allowed-is actually quite challenging, even for gpt-4. further, 'hard' hallucination detection-the identification of utterances that are likely ambiguous-is also challenging, with a joint amr-sparql model only detecting 1 in 5 cases. beyond the amr-for-kbqa investigations we performed in this work, we hope that the release of massive-amr will support additional research into using structured meaning representations for multilingual qa and model interpretability.