B i BL : AMR Parsing and Generation with Bidirectional B ayesian Learning (2022)

Abstract: Abstract Meaning Representation (AMR) offers a unified semantic representation for natural language sentences. Thus transformation between AMR and text yields two transition tasks in opposite directions, i.e., Text-to-AMR parsing and AMR-to-Text generation. Existing AMR studies only focus on one-side improvements despite the duality of the two tasks, and their improvements are greatly attributed to the inclusion of large extra training data or complex structure modifications which harm the inference speed. Instead, we propose data-efficient Bidirectional Bayesian learning (BiBL) to facilitate bidirectional information transition by adopting a single-stage multitasking strategy so that the resulting model may enjoy much lighter training at the same time. Evaluation on benchmark datasets shows that our proposed BiBL outperforms strong previous seq2seq refinements without the help of extra data which is indispensable in existing counterpart models. We release the codes of BiBL at: https://github.com/KHAKhazeus/BiBL .

## introduction abstract meaning representation (amr) @xcite is a formalism that could capture the semantic meaning of a natural language sentence in a graph representation. an amr graph example is shown in figure 1 . leaves in the amr graph denote concepts in the text sentence while other nodes within the graph are entities. two entities could be connected with relations defined under a common standard, and instance relations connect entities and corresponding concepts. the flexibility and semantic invariance characteristics make amr applicable to various natural language processing (nlp) downstream tasks and achieve solid performance. the transformation between amr graphs and texts forms two basic tasks offering helpful amr information for the downstream tasks. traditional approaches treat amr-to-text generation and textto-amr parsing separately. however, utilizing graph linearization and the power of pre-trained language models, spring @xcite casts two tasks into a unified sequence-to-sequence (seq2seq) structure and outperforms all traditional approaches proposed before. such progress consolidates the prevalent position of seq2seq approaches tackling the amr transition tasks. recently, many methods have been proposed to further enhance the seq2seq model. in order to improve the model performance on text-to-amr parsing, @xcite incorporates an actionpointer mechanism. @xcite adds ancestor information of amr graphs into the model. @xcite introduces a data augmentation approach by transforming data excerpted from semantic role labeling and dependency parsing into pseudo-amr data, and then constructing extra training tasks using these data. @xcite proposes graph ensembling to combine different models in order to achieve better results. for the amr-to-text generation task, @xcite designs two auxiliary training objectives, i.e., relationship prediction and distance prediction of nodes in amr graphs. @xcite utilizes data from the kg2text task, which is transforming knowledge graphs into texts, and masked language modeling to pre-train the model. @xcite proposes six graph pre-training tasks to obtain a better pre-trained model. though the above-mentioned refinements are effective, applying auxiliary techniques, including ensembling and adding graph information during training, could greatly harm the training or inference speed. meanwhile, obtaining a better pre-training model using extra silver data or data augmentation techniques greatly increases the time consumption and design complexity of the whole model training phase. instead of using extra data or complicated techniques which harm the training or inference speed, we propose a novel solution for amr transition tasks. the solution consists of two auxiliary tasks to help the model grasp the joint probability of the amr graphs and texts. by interweaving the two auxiliary tasks with the main transduction task, the extra knowledge learned in the two auxiliary tasks could boost the model understanding of the main task. therefore, single-stage multitask learning without fine-tuning could be adopted for bibl. based on our proposed bibl, the training paradigm could be greatly simplified, and under the same model settings, our implemented models achieve new state-of-the-art performances in amr-to-text generation. compared with previous refinements, further comparison experiments show that bibl is more efficient both during the training and inference phases. ## related work there are two tasks when concerning the transformation between amr graphs and texts. however, it is not always that these two tasks are jointly considered. ## method as the seq2seq model only receives and emits data in sequential forms, we have to take the necessary pre-processing and post-processing to perform conversions between amr graphs and their sequential forms. then, the seq2seq transduction is completed using sequence generation models of encoder-decoder structure. on top of the seq2seq model design, we enhance bidirectional model performance through two auxiliary tasks which facilitate the model's understanding of the joint probability between texts and amr graphs. ## experimental setup based on the evaluation system of previous works @xcite @xcite , we inspect the performance of bibl in two settings: in-distribution and outof-distribution. the in-distribution setting shows the power of bibl on standard benchmarks. since the bidirectional transitions between amr graphs and texts should be modeled as a unified process without dependence on language contexts, it is also important to evaluate our model in out-ofdistribution settings. the experiment benchmark system uses nvidia titan rtx as gpu and intel xeon e5-2637@3.50ghz as cpu. the system is built with 256gb ram. ## ablation studies since amr 3.0 is the superset of amr 2.0, our ablation studies are based on the amr 3.0 dataset so that a more comprehensive investigation could be conducted. through a thorough analysis of bibl enables the model to effectively capture the underlying information in the joint probabilistic space of amr graphs and texts. bibl is trained with single-stage multitask learning. further experiments are conducted to compare the chosen paradigm with the two-stage training paradigm. for two-stage training, we pre-train bibl with auxiliary tasks and then fine-tune bibl without them. results listed in ## conclusion in this paper, we propose bibl which is capable of enhancing the seq2seq approach to tackle the bidirectional transition task between amr graphs and texts.