GPT -too: A Language-Model-First Approach for AMR -to-Text Generation (2020)

Abstract: Abstract Meaning Representations (AMRs) are broad-coverage sentence-level semantic graphs. Existing approaches to generating text from AMR have focused on training sequence-to-sequence or graph-to-sequence models on AMR annotated data only. In this paper, we propose an alternative approach that combines a strong pre-trained language model with cycle consistency-based re-scoring. Despite the simplicity of the approach, our experimental results show these models outperform all previous techniques on the English LDC2017T10 dataset, including the recent use of transformer architectures. In addition to the standard evaluation metrics, we provide human evaluation experiments that further substantiate the strength of our approach.

## introduction abstract meaning representation (amr) @xcite is a rooted, directed, acyclic graph with labeled edges (relations) and nodes (concepts) expressing "who is doing what to whom". amr-to-text generates sentences representing the semantics underlying an amr graph. initial works in amr-to-text used transducers @xcite , phrase-based machine translation @xcite and neural sequence-to-sequence (seq2seq) models with linearized graphs @xcite . @xcite leverage constituency parsing for generation. @xcite improve upon prior rnn graph encoding @xcite with levi graph transformations. @xcite compare multiple representations and find graph encoders to be the best. @xcite use rnn graph encoders with dense graph convolutional encoding. @xcite use rnn encoders with dual graph representations. transformer-based seq2seq @xcite was first applied to amr-to-text in @xcite . @xcite greatly improve over the prior state-of-the-art by modifying self-attention to account for amr graph structure. using transformers has also been recently explored by @xcite who propose a mutli-head graph attention mechanism. pre-trained transformer representations @xcite @xcite use transfer learning to yield powerful language models that considerably outperform the prior art. they have also shown great success when fine-tuned to particular text generation tasks @xcite @xcite . given their success, it would be desirable to apply pre-trained transformer models to a graph-to-text task like amr-to-text, but the need for graph encoding precludes in principle that option. feeding the network with some sequential representation of the graph, such as a topological sorting, looses some of the graphs representational power. complex graph annotations, such as amr, also contain many special symbols and special constructs that departure from natural language and may by not interpretable by a pretrained language model. in this paper we explore the possibility of directly fine-tuning a pre-trained transformer language model on a sequential representation of amr graphs, despite the expected difficulties listed above. for this we re-purpose a gpt-2 language model @xcite to yield an amr-to-text system. we show that it is surprisingly easy to fine-tune gpt-2 to learn amr graph to text mapping that outperforms the previous state-of-the-art on automatic evaluation metrics. since a single graph amr, graph corresponds to multiple sentences with the same meaning, we also provide human evaluation and semantic similarity metric results @xcite which are less dependent on reference text. human evaluation and semantic similarity results highlight the positive impact of a strong language model strategy. finally we also introduce a simple re-scoring technique based on cycle-consistency that further improves performance. ## fine-tuning gpt-for conditional language generation in order to fine-tune a generative model (gpt-2; @xcite al. (2019) ) for conditional text generation, prior works fine-tune the language model to predict target text starting from the additional source text as context. in our experiments, we found it beneficial to fine-tune on the joint distribution of amr and text instead i.e. also reconstruct the source. given a tokenized sentence w 1 • • • w n and the sequential amr representation a 1 • • • a m we maximized the joint probability a special separator token is added to mark the end of the sequential amr representation. special amr symbols that should not be interpreted literally are assigned tokens from the gpt-2 unused token list. in addition to this, we also observed that freezing the input embeddings when fine-tuning had positive impact in performance. at test time, we provide the amr as context as in conventional conditional text generation: ## re-scoring via cycle consistency the general idea of cycle consistency is to assess the quality of a system's output based on how well an external 'reverse' system can reconstruct the input from it. in previous works, cycle-consistency based losses have been used as part of the training objective in machine translation @xcite and speech recognition @xcite . it has also been used for filtering synthetic training data for question answering @xcite . here we propose the use of a cycle consistency measure to re-score the system outputs. in particular, we take the top k sentences generated by our system from each gold amr graph and parse them using an off-the-shelf parser to obtain a second amr graph. we then re-score each sentence using the standard amr parsing metric smatch @xcite by comparing the gold and parsed amrs. ## experimental setup following previous works on amr-to-text, we use the standard ldc2017t10 amr corpus for evaluation of the proposed model. this corpus contains 36,521 training instances of amr graphs in penman notation and the corresponding texts. it also includes 1368 and 1371 development and test instances, respectively. we tokenize each input text using the jamr toolkit @xcite . the concatenation of an amr graph and the corresponding text is split into words, special symbols and sub-word units using the gpt-2 tokenizer. we add all arc labels seen in the training set and the root node :root to the vocabulary of the gpt-2model, but we freeze the embedding layer for training. we use the hugging face implementation of @xcite for gpt-2 small (gpt-2s), medium (gpt-2m) and large (gpt-2l). fine-tuning converges after 6 epochs, which takes just a few hours on a v100 gpu foot_0 . for cycle-consistency re-scoring we use an implementation of @xcite in py-torch. for re-scoring experiments, we use a beam size of 15. amr input representation. we test three variants of amr representation. first, a depth-first search (dfs) through the graph following @xcite , where the input sequence is the path followed in the graph. second, to see if gpt-2 is in fact learning from the graph structure, we remove all the edges from the dfs, keeping only the concept nodes. this has the effect of removing the relation information between concepts, such as subject/object relations. as a third option, we use the penman representation without any modification. the three input representations are illustrated below: nodes recommend advocate-01 it vigorous dfs recommend :arg1 advocate-01 :arg1 it :manner vigorous penman (r / recommend-01 :arg1 (a / advocate-01 :arg1 (i / it) :manner (v / vigorous))) decoding. for generation, we experiment with greedy decoding, beam search, and nucleus sampling @xcite . for beam search, we explore beam sizes of 5, 10 and 15. as the system, in some cases, produces repetitive output at the end of the text, we additionally perform a post-processing step to remove these occurrences. metrics. we considered the three automatic evaluation metrics commonly used in previous works. we compute bleu @xcite using sacrebleu @xcite . we compute chrf++ @xcite using both sacrebleu and the scripts used by authors of the baseline systems. we compute meteor @xcite with the default values for english of the cmu implementation. 2 in addition to the standard automatic metrics, we also carry out human evaluation experiments and use the semantic similarity metric bertscore @xcite . both metrics arguably have less dependency on the surface symbols of the reference text used for evaluation. this is particularly relevant for the amr-to-text task, since one single amr graph corresponds to multiple sentences with the same semantic meaning. conventional metrics for amr-to-text are are strongly influenced by surface symbols and thus do not capture well the ability of the system to produce a diverse sentences with same underlying semantics. human evaluations are carried out by three professional annotators on 51 randomly selected sentences from the 1371 test sentences, on a 6 point scale, ranging from 0 to 5. for each system, scores from all annotators are averaged to compute a single score. inter-annotator agreement was 0.7 when measured by pearson correlation coefficient. our system produces de-tokenized cased output after bpe decoding, whereas previous systems produce traditional tokenized lower-cased output. therefore, we lowercase and tokenize our system outputs to have fair comparisons with previous systems. ## conclusions in this work, we present a language model-based approach for the amr-to-text generation task. we show that a strong pre-trained transformer language model (gpt-2) can be fine-tuned to generate text directly from the penman notation of an amr graph. comparison with state-of-the-art models in blue, chrf++, meteor as well as semsim and human evaluation metrics show that while simple, this approach can outperform existing methods including methods training transformers from scratch. we also show that cycle consistency-based re-scoring using a conventional amr parser and the smatch metric can notably improve the results. future work will focus on incorporating better encoding of the amr graph into the current system and exploring data augmentation techniques leveraging the proposed approach.