Exploring a POS -based Two-stage Approach for Improving Low-Resource AMR -to-Text Generation (2022)

Abstract: This work presents a two-stage approach for tackling low-resource AMR-to-text generation for Brazilian Portuguese. Our approach consists of (1) generating a masked surface realization in which some tokens are masked according to its Part-of-Speech class and (2) infilling the masked tokens according to the AMR graph and the previous masked surface realization. Results show a slight improvement over the baseline, mainly in BLEU (1.63) and METEOR (0.02) scores. Moreover, we evaluate the pipeline components separately, showing that the bottleneck of the pipeline is the masked surface realization. Finally, the human evaluation suggests that models still suffer from hallucinations, and some strategies to deal with the problems found are proposed.

## introduction abstract meaning representation (amr) is a semantic formalism that encodes the meaning of a sentence into a labeled directed and rooted graph @xcite . this representation comprises semantic information related to semantic roles, named entities, and co-references, among others. amr is a widely-studied research topic in the semantic representation field and has been proven helpful in many natural language processing tasks @xcite . its success is partially based on its broad use of mature linguistic resources, such as propbank @xcite , and its attempt to abstract away from syntax. figure 1 shows an example of an amr graph and its corresponding penman notation for the sentence "the boy must not go.". amr-to-text generation aims to "translate" an abstract meaning representation graph into its corresponding text. this task has been widely tackled by diverse approaches, starting from statistical, transducer-based and transition-based ones (pourdamghani @xcite @xcite , until end-to-end neural ones @xcite , recently. in particular, end-to-end neural models -mainly those based on pre-trained models-have largely outperformed the initial methods, achieving stateof-the-art results @xcite . these models can generate fluent text. however, they are prone to generate hallucinations, i.e., texts that are irrelevant to or contradicted with the input @xcite . another drawback is that these models are usually data-hungry, i.e., they need to be trained on a large dataset to achieve a good performance. it can be a problem when we deal with low-resource domains, languages, or tasks @xcite . even when the results may be better than those obtained by statistical methods, they are still far from good results. for example, @xcite show that fine-tuning t5 @xcite on a small portion of a big dataset (∼500 instances) produces a ∼10-15 bleu score. in general, an approach to have more control over the decoding process (and avoid hallucinations) is to use a pipeline-based method in which the model of each pipeline's module is implemented with neural models @xcite @xcite . another alternative is to use templates, infill concepts in these templates, and then define a strategy to transform them into sentences/paragraphs @xcite . both approaches have proven to be helpful in text generation tasks. however, the main issue for the latter one is that it only can be applied in restricted domains as it is necessary to define a set of templates. in this work, we approach the amr-to-text generation task in two stages. firstly, generating a masked surface realization in which some tokens are masked according to its part-of-speech (pos) classes. then, finally, infilling the masked tokens according to the amr graph and the previous masked surface realization 1 . the intuition for masking some tokens this way is that some pos classes are more difficult to be predicted during text decoding and can harm the performance. on the other hand, filling-in-theblank is commonly used on current sota architectures, such as t5 @xcite during the pre-training phase. this way, we can leverage the learned knowledge to infill the masked tokens in the previous stage adequately. experiments are conducted on low-resource an amr-to-text generation task for brazilian portuguese @xcite to show how this method behaves even when a large dataset is unavailable. in general, our main contributions are: amr is more open-ended than other datasets such as webnlg @xcite . this way, extracting templates can be a complex task. some attempts to get templates in the form of rules are presented by @xcite and @xcite . however, these approaches need some manually created rules and have been surpassed by current models. on the other hand, current neural models have achieved sota results. however, they need a large dataset to get high performance. on the contrary, a small portion of an extensive dataset produces lower scores @xcite foot_1 . ## results and discussion overall, results show a slight improvement over the baseline when we use the model trained on all main classes, mainly in bleu (+1.63) and me-teor (+0.02) scores. moreover, the best main classes to be masked seem to be "verb" and "qualifier". on the other hand, masking nouns and other pos classes harm the decoding performance. we might interpret this result as the characters in a sentence, and some connections between chunks are the most important in the realization of a sentence. another point to note is that it is better to train models on all main classes together instead of separately. a possible explanation is that more data can lead to better results. also, examples from other main classes serve as negative examples for a specific main class, and it helps to improve its performance. in order to verify which stage of the pipeline is affecting the overall performance, we evaluate each module separately. in addition, we evaluate meteor. concerning the mask surface realization task, it is worth noting that, in general, the bottleneck of the whole pipeline is the masked surface realization task, as values are similar to the overall performance. having the worst performance on the word infilling task, achieves the highest performance because the previous task gets the best one. a possible explanation for this problem is how the generation is performed. we use an encoder-decoder architecture in which the generation of a token depends on the previously generated tokens. this way, adding mask tokens in training could make it more difficult as the pre-trained model never saw these tokens in a generation task (these were used for training the blank infilling task). among the alternatives to solve this issue, we could explore other strategies to determine the less confident tokens in a generated sentence and mask them for the next stage. also, we could try a non-autoregressive model that can overcome the problem of dependency mentioned before @xcite . ## manual revision we conduct a manual revision of the outputs for each model in order to check the main and most common errors. in particular, we select the two best models in our experiments, i.e., the ones trained on all main classes but focusing on masking/filling verbs and qualifiers. we analyze 35 instances from the test set and classify the outputs into four classes: (1) accurate ("acc"), for accurate outputs,(2) hallucination ("hall"), for outputs that are not related to the reference, (3) cut chunk, for outputs that contains only a portion of the reference, and (4) small changes, for outputs with slightly different from the reference (some tokens are different). in general, the model trained on all main classes, but focusing on verbs got the best results. it is worth noting the high number of hallucinations in all models, mainly when longer sentences are evaluated. also, the cut chunks happen in the same cases. moreover, there are several instances where only changing a simple word (or two) would be necessary to make the output similar to the reference. this problem happens mainly with connectors such as "em" ("in" or "at") or "de" ("of") (words highlighted in red in figure 3 ) and with bad conjugations in the case of the verbs. figure 3 shows three examples. the first example shows that the model focused on verbs gets an accurate output (example 1). the second example shows that the outputs for models focused on verbs and qualifiers can generate paraphrases instead of ## conclusion and further work this work presents a simple two-stage approach to the low-resource amr-to-text generation task. the approach consists of generating a masked surface realization in which some tokens are masked according to a pos class criteria and infilling the masked tokens according to the amr graph and the previous masked surface realization. results show a slight improvement over the baseline, mainly in bleu (1.63) and meteor (0.02) scores. however, it is necessary to fine-tune the model on all the sub-corpus created together. besides, we can note that verb masking seems to be the best strategy in this approach. on the other hand, we note that the bottleneck of this approach is the masked surface realization model, as some generated tokens are different and unrelated to the original reference (hallucinations), and some tokens are omitted from the original reference. some possible explanations for this problem are how the generation is performed -as each output word is conditioned on previously generated outputs-and the need to constrain the decoding process. as further work, we plan to explore strategies to enforce the model to cover all the amr concepts in the masked generated sentence and non-autoregressive text generation with pre-trained models, similar to @xcite . besides, we plan to explore other strategies to mask tokens according to its confidence in decoding instead of using a pos-based one as the later can add more complexity to the task. finally, we plan to extend this work to english amr corpus, in order to make a better comparison in terms of generalization.