Cross-domain Generalization for AMR Parsing (2022)

Abstract: Abstract Meaning Representation (AMR) parsing aims to predict an AMR graph from textual input. Recently, there has been notable growth in AMR parsing performance. However, most existing work focuses on improving the performance in the specific domain, ignoring the potential domain dependence of AMR parsing systems. To address this, we extensively evaluate five representative AMR parsers on five domains and analyze challenges to cross-domain AMR parsing. We observe that challenges to cross-domain AMR parsing mainly arise from the distribution shift of words and AMR concepts. Based on our observation, we investigate two approaches to reduce the domain distribution divergence of text and AMR features, respectively. Experimental results on two out-of-domain test sets show the superiority of our method.

## introduction abstract meaning representation (amr; @xcite ) is a broad-coverage semantic structure formalism that represents the meaning of a text in a rooted directed graph. as shown in figure 1 , the nodes in an amr graph represent concepts such as entities and predicates, and the edges indicate their semantic relations. amr parsing @xcite @xcite @xcite @xcite @xcite is the task of transforming natural language into amr graphs. this is a fundamental task in semantics, which can also benefit downstream use. amr has been proven to be useful for many downstream tasks, such as information extraction @xcite @xcite @xcite ), text * work done as an intern at tencent ai lab seattle. summarization @xcite @xcite @xcite , machine translation @xcite @xcite ), text generation @xcite @xcite @xcite , and dialogue systems @xcite @xcite . to benefit such a diverse set of tasks that covers various domains, an ideal amr parser should generalize well across different domains. however, most existing work only focuses on improving the in-domain parsing accuracy, ignoring the performances on other domains. though state-of-the-art amr parsers can obtain a smatch score of over 84% on an indomain test set, we observe that their cross-domain performance is still weak (e.g., lower than 65% on the biomedical domain). it remains an openquestion how well different types of amr parsers generalize to out-of-domain (ood) data. in this work, we take the first step to study the cross-domain generalization ability of a range of typical amr parsers, investigating three main research questions: 1) how well do different amr parsers perform on out-of-domain test sets? 2) what are the main challenges to cross-domain amr parsing? and 3) how to improve the performance of cross-domain amr parsing? we empirically choose five major amr parsers for comparison, including a two-stage statistical parser @xcite , a graph-based parser @xcite , a transition-based parser @xcite , a seq2seq-based parser @xcite , and an amrspecific pre-training parser @xcite . the test domains cover news, biomedical, novel, and wiki questions. we conduct experiments under the zero-shot setting, where a model is trained on the source domain and evaluated on the target domain without using any target-domain labeled data. our results show that 1) all models give relatively lower (up to 45.5%) performances on out-of-domain test sets, with the most dramatic drop on named entities and wiki links; 2) the graph pretraining-based parser is stronger in domain transfer than the other parsers; 3) the transitionbased parser is more robust than the seq2seqbased parser. we further analyze the impact of a set of linguistic features, and the results suggest that the performance degradation is positively correlated with the distribution shifts of words and amr concepts. compared with the distribution divergences of the input features, those of the output features are more challenging to crossdomain amr parsing. based on our analysis, we investigate two approaches to bridge the domain gap for improving cross-domain amr parsing. we first continually pre-train a bart model on target domain raw text to reduce the distribution gap of words. to further bridge the domain gap of output features, we adopt a pre-trained amr parser to construct silver amr graphs on the target domain, which potentially reduces the output features divergence. experimental results show that the proposed methods consistently improve the parsing performance on out-of-domain test sets. to our knowledge, this is the first systematic study on cross-domain amr parsing. our code and results will be available at https://github.com/ goodbai-nlp/amr-domainadaptation . ## compared models we choose the representative or top-performing parser of two-stage, graph-based, transition-based, seq2seq-based as well as a pre-trained parser for evaluation. in particular, the following amr parsing systems are considered: jamr @xcite , as shown in figure 2 (a), is a two-stage parsing model which predicts concepts and relations in a pipeline. jamr identifies concepts and predicts the relations using two discriminatively-trained linear structured predictors, which use rich features like part-of-speech tagging (pos), named entities recognition (ner), lemmatization, etc. in addition, jamr relies on an external aligner to construct supervision signals for both stages. amrgs (cai and lam, 2020a) is a graph-based parser which builds a semantic graph incrementally. as shown in figure 2(b) , at every step, the graphbased parser predicts one node and its connection to existing graph. amrgs learns mutual causalities between text and graph by updating the sentence and graph representations iteratively. amrgs obtains word-level representation from a pretrained language model (i.e., bert @xcite ) and uses pos, ner and lemmatization as external knowledge to make predictions. structbart @xcite , as shown in figure 2 (c), is a transition-based parser which generates an amr graph through a sequence of transition actions. in particular, the transition actions are: structbart takes a pre-trained bart model as the backbone and extends the original vocabulary with transition actions. additionally, structbart requires an external aligner to obtain oracle transition actions for training. spring @xcite , as shown in figure 2(d ), is a sequence-to-sequence parser which transforms a text sequence into a linearized amr sequence. spring adopts a depth-first algorithm to transform amr graphs into a sequence where concepts and relations are treated equally. to deal with co-referring nodes, spring adds special tokens to the vocabulary. same with structbart, spring also initializes model parameters with bart. amrbart @xcite ) is a continually pre-trained bart model on amr graphs and text. it uses three graph-based pre-training tasks to improve the structure awareness of the encoder and decoder and another four tasks that jointly learns on text and amr graph to capture the correspondence between amr and text. amrbart is pre-trained on 250k training instances, which lie in the same domain as amr2.0. in addition, jamr uses complicated rulebased pre-processing and post-processing steps to simplify the input and reconstruct the amr graphs. amrgs uses rule-based graph re-categorization for pre-processing and recovers concept sense tags, wiki links, and polarities during post-processing. structbart, spring, and amrbart do not require pre-processing steps and use the blink entity linker @xcite to handle wiki links during post-processing. ## experiments experimental configurations and our adopted datasets are shown in sections 4.1 and 4.2, respectively. to study the cross-domain generalization ability of current amr parsers, we first quantify the difference between in-domain training data and out-of-domain test data (section 4.3), and then evaluate the cross-domain performance of 5 typical amr parsers (section 4.4). ## key challenges to ood amr parsing based on the results above, we further study two important questions: which amr components are the most challenging for cross-domain amr parsing (in section 5.1); and what contributes most to the performance degradation on ood test sets (in section 5.2)? ## bridging the domain gap according to our analysis in section 5.2, we investigate two approaches to improve the model performance on ood datasets by bridging the distribution gap between the training and test domains without modifying model structures. alleviating input feature divergence. we employ raw text from target domain to enrich the model with domain-specific input features. specifically, we collect raw text from the biomedical and fairy tales domain, which are then used as extra knowledge for training. inspired by previous work @xcite , we add an intermediate pre-training step to adapt the pre-trained model to the target domain, which refers to domain-adaptive pre-training. we take bart @xcite as the backbone and continually pre-train bart on the collected dataset using the standard self-supervised learning training objective. we randomly mask text spans, replacing 15% tokens. the adaptively trained model is used for initialization during fine-tuning. alleviating output feature divergence. we investigate silver data as pseudo target domain training data to fine-tune the amr parsers. in results. ## conclusion we investigated the cross-domain generalization challenges for amr parsing by analyzing the performance of five representative models. empirically, we found that all amr parsers give lower performance on ood test sets, and the difficulty lies more in output features divergences, including concept and relation, compared with input features. based on our analysis, we investigated two approaches to bridge the domain gap of input and output features, respectively, which achieve higher scores on out-of-domain test sets than previous work. in the future, we would like to investigate more methods, such as vocabulary adaptation @xcite and knearest-neighbor (knn, @xcite , to improve cross-domain amr parsing.