Question-parsing with A bstract M eaning R epresentation enhanced by adding small datasets (2025)

Abstract: Abstract Meaning Representation (AMR) is a graph-based formalism for representing meaning in sentences. As the annotation is quite complex, few annotated corpora exist. The most well-known and widely-used corpora are LDC’s AMR 3.0 and the datasets available on the new AMR website. Models trained on the LDC corpora work fine on texts with similar genre and style: sentences extracted from news articles, Wikipedia articles. However, other types of texts, in particular questions, are less well processed by models trained on this data. We analyse how adding few sentence-type specific annotations can steer the model to improve parsing in the case of questions in English.

## introduction abstract meaning representation @xcite provides a framework to model the meaning of a sentence, notably actions, events or states and their participants. amr relies heavily on (verbal) concepts defined in propbank @xcite , e.g. bear-02 in figure 1, propbank's sense -02 for the verb "to bear". instances are indicated by a following "/", e.g., p being an instance of the concept person. the names of the variables do not have any other semantics than being distinct. relations are indicated by an initial colon (e.g. :arg1, :time). literals (strings and numbers) lack a preceding instance and "/" (c.f. "elizabeth" and 1926 in the example in figure 1 ). this serialised format, shown in figure 1 left, is called penman @xcite . the largest available corpus used to train models capable of parsing sentences from natural languages into amr graphs, called amr 3.0, (b / bear-02 :arg1 (p / person :name (n / name :op1 "queen" :op2 "elizabeth")) :time (d / date-entity :year 1926)) figure 1: amr graph for "queen elizabeth was born in 1926" in penman format. ldc2020t02 foot_0 , is provided by the linguistic data consortium (ldc). this corpus is composed of nearly 59 000 sentences and corresponding amr graphs. the data contains discussions from forums (partly technical), news reels, translations to english of chinese news broadcasts, along with a part originating from english wikipedia pages and aesop's fables (see ldc2020t02 documentation). the problem we address in this article is the following: the gold data currently available for amr parsing is very homogeneous in form as it is composed of declarative, informative sentences. training models on such data yields lower-thanexpected results for parsing of questions in amr. we add a small dataset of questions to the training data to bypass this problem. even if we were intuitively expecting this kind of result, we were able to confirm it and measure improvement. ## related work domain type adaptation research for amr has been attempted in several contexts and perspectives, one of the most well-known leading to the development of bio-amr foot_1 . bio-amr includes texts from the biomedical domain, extracted from pubmed foot_2 . @xcite conducted a research on amr of data outside news article excerpts, focusing on the legal documents domain, using a gold dataset. the parsing results were not very conclusive, and the authors provide a detailed discussion of this result. among the explanations for the models not-so-good performances, two stand out: first, legal documents contain mostly sentences longer than the ones from ldc datasets; then, the models faced out-of-vocabulary (oov) issues, as some concepts, specific to the legal domain, were not defined in propbank. this latter issue comes from the semantic difference between the news and the legal documents domains. lee et al. (2022) experimented on sentencetype adaptation through both algorithmic and databased research. they created and released the qald-9-amr corpus, built on top of qald-9 data @xcite . it contains amr annotations for natural language questions in english, originally provided for executable semantic parsing. @xcite further mention one unavoidable difficulty for domain adaptation which is outof-vocabulary named entities and their types, that cannot be solved without using domain-specific corpora. the authors compare the usage of silver data with that of human annotation for qald-9. ## amr parsing of short questions in this section, we present our data and our parsing methods, followed by first observations and hypotheses. corpora the amr 3.0 corpus mainly contains sentences from newspapers and a small part of wikipedia. there is almost no real question in this corpus (apart from a few rhetorical ones). our hypothesis is that a model trained on this data will not perform well on question parsing. thus, our research question is to see whether it is possible to improve the model's performance on questions by adding a small corpus of short questions to its training data (i.e. amr 3.0 train). the data we used in this article is the following: a. amr 3.0, about 55 000 sentences for training, 1 722 sentences for validation and 1 898 sentences for test. b. qald-9 lee et al. (2022) 4 , contains 400 (train) and 150 (test) questions taken from the qald-9 project and annotated using amr. ## https://github.com/ibm/amr-annotations the test set of qald-9 contains 13 sentences which are also in the train corpus and in the qald-7 and qald-8 data which served as input for quereo. we deleted them from the qald-9 test set, and use only the 137 remaining sentences. (c.f. fig. 2 ). about 25% of the questions and all answers in quereo were formulated prior to the amr annotation by human annotators from our team foot_3 (cf. fig. 3 and 4 ). figure 3 : example question from our corpus amr 3.0 corpus, which we augment with datasets containing short questions. we trained the models using either t5 @xcite , flan-t5 (chung et al., 2022) or mt5 @xcite as underlying language model (base size in all four cases). for evaluation, we use the smatch package @xcite 7 . ## effect of adding questions to the training data after our first observations, we trained models using different combinations of augmented data. ## discussion in this paper we showed that even minor additions to the standard amr 3.0 training corpus can have big impacts on the performance of an amr parser for a new sentence type, syntactic in the case of questions. next, we plan on taking our studies further by annotating a domain specific corpus in the domain of artificial intelligence) or noisy data. we are aware of the problems of smatch-based evaluation, and we follow the other algorithms for amr comparison that have been proposed, in particular semantic smatch such as s2match @xcite . in future work, we would like to broaden our exploration of the benefits of adding a small corpus of specialised examples through different dimension of amr, using different types of evaluation metrics. the exploration conducted in this paper has fo-cused on one method of parsing, the one provided by our extended version of amrlib. it would be interesting for us to test whether the data augmentation results presented here are coherent throughout the different parsing methods, in particular in using the most efficient parsing methods for amr such as mbse (see @xcite ). we would also like to conduct an exploration of errors similar to the one presented in @xcite to be able to quantify and qualify the remaining percentages of mistakes. the goal then would be to use pre-and postprocessing methods to accommodate these errors when possible. we only worked with short questions, quiz-like, since we were not (yet) able to annotate corpora with longer questions or more complex types of questions. the questions in the additional corpora (qald-9 and quereo) are given without a proper context. if we were to parse dialogues, coreference resolution and ellipsis resolution should be considered. ## further work: beyond english the work presented in the current article only concerns english, since gold amr data is only available for this language. another problem is that the amr3.0 training corpus is translated to other languages using machine translation, so errors in this translation may influence the results. even though amr has explicitly not been developed to be an interlingua for multi-lingual processing, it is in fact used exactly for this. a manual translation of the amr test corpus sentences into chinese, german, italian and spanish is provided by ldc (ldc2020t07 foot_6 ). in order to annotate non-english text in amr, two variants for multi-lingual amr can be found in the literature: 1) annotating non-english sentences using a language specific set of concepts, i.e. instead of the (english) propbank, concepts from language specific thesauri are used (e.g. chinese amr, @xcite , spanish @xcite , turkish @xcite amongst others) or 2) english amr graphs represent the meaning of non-english sentences @xcite @xcite @xcite . we followed the latter approach by machine-translating the sentences of the amr 3.0 corpus into french and training baseline models using this translation. we used google machine translation @xcite and no language left behind @xcite ). for the training of the french corpus we only finetuned mt5. in addition we created a multilingual model (based on mt5) by concatenating and shuffling the english and french training and validation corpora. in this case we have an english and a french sentence for each amr graph. the first results look very similar to the results described in this paper as shown in another approach we would like to explore is the transition from amr to uniform meaning representation (umr) @xcite . as umr is designed to be "cross-linguistically plausible", the multilanguage considerations are inherent to the umr annotations, making them particularly interesting for our type of investigations.