Pre-Trained Language-Meaning Models for Multilingual Parsing and Generation (2023)

Abstract: Pre-trained language models (PLMs) have achieved great success in NLP and have recently been used for tasks in computational semantics. However, these tasks do not fully benefit from PLMs since meaning representations are not explicitly included. We introduce multilingual pre-trained language-meaning models based on Discourse Representation Structures (DRSs), including meaning representations besides natural language texts in the same model, and design a new strategy to reduce the gap between the pre-training and fine-tuning objectives. Since DRSs are language neutral, cross-lingual transfer learning is adopted to further improve the performance of non-English tasks. Automatic evaluation results show that our approach achieves the best performance on both the multilingual DRS parsing and DRS-to-text generation tasks. Correlation analysis between automatic metrics and human judgements on the generation task further validates the effectiveness of our model. Human inspection reveals that out-of-vocabulary tokens are the main cause of erroneous results.

## introduction there are two common tasks in computational semantics: mapping a text to a meaning representation (semantic parsing), and its reverse, producing a text from a meaning representation (semantic generation). these tasks generally rely on corpora that contain texts aligned with meaning representations. while in recent years large pre-trained language models (plms), both monolingual as well as multilingual, have brought nlp tasks to a new level, semantic parsing and generation cannot fully benefit from them since the meaning representations are not included in plms explicitly. our goal in this work is to leverage the principle of pre-trained models and explore the benefit of multilingual semantic parsing and generation of including in the same model meaning representations aside from natural language. this would make it possible not only to operate multilingually, thanks to representation neutrality, but also to leverage the bidirectionality of language-meaning alignment. figure 1 illustrates our idea. semantic parsing and generation (in different languages) are clearly related, but traditionally they are studied and developed independently of one another, usually focusing on a single language (often english). this results in having to train separate models from scratch for each task and language, and progress has been hampered by data scarcity. this is especially true for languages other than english, where data scarcity is even more severe. our proposal to incorporate meaning representations in plms and to concurrently embrace a multilingual approach breaks with this tradition yielding a twofold advantage. first, multilingual plms enable different languages to be represented in one universal space making it possible to benefit from cross-lingual knowledge transfer in semantic parsing and generation. second, joining the formal and natural language representations in training makes it possible to leverage one and the same model for parsing and generation. for this approach to work, we need a meaning representation frame-work where (i) the formalism is language-neutral, (ii) there is aligned data both in terms of meaninglanguage(s), but also multilingually across different languages, and (iii) there is enough expressivity to cover for a wide range of language phenomena. discourse representation structure (drs), which satisfies our requirements well, is the formal meaning representation proposed in discourse representation theory @xcite @xcite @xcite . it covers a large variety of linguistic phenomena, including anaphors, presuppositions, temporal expressions and multisentence discourses and captures the semantics of negation, modals and quantification. furthermore, drs provides a language-neutral meaning representation: the same meaning representation associated with text that can be expressed in various languages. while abstract meaning representations (amr, @xcite ) have been proposed for this task, we believe drs is more suitable because of its multi-lingual representation capability (all predicates are interpreted), its expressive power (proper treatment of negation and universal quantification), and the comparable annotated data available for multiple languages. as a first step, we consider drs as an additional abstract language that will complement the natural languages in our pre-trained model. we take the multilingual plm mbart @xcite and further pre-train it with all of our language data, thus both the four natural languages we use as well as the language neutral meaning representations, so that the drss and texts are learnt in the same semantic space. as a second step, we introduce a supervised denoising training that exploits more explicitly the relationship between drs and each corresponding text as well as between the parallel texts in the different languages; we do this combined with denoising training to reduce the gap between the pre-training and fine-tuning objectives. at this point, we have at our disposal a single multilingual language-meaning model which can then be fine-tuned for either parsing (text-to-drs) or generation (drs-to-text), in a monolingual or multilingual fashion. overall, our main contributions include: (i) a novel task of multilingual drs-to-text generation, and a framework for a mixed language-meaning modelling in a multilingual setting, serving both parsing and generation. (ii) a pre-training strat-egy, with self-supervised training followed by supervised training, to reduce the gap between pretraining and fine-tuning; we also employ multilingual transfer techniques to boost performance in languages other than english exploiting language neutrality in drss. (iii) extensive experiments for both parsing and generation across different languages, including both automatic and human evaluation to understand how multilingual models perform. foot_0 ## background and related work this work employs intensive multilingual pretraining techniques for language-meaning modelling for both parsing and generation. in this section, we briefly introduce the concept of drs, which serves as our meaning representation tool, and relevant background and related work. ## method we use mbart as our backbone to jointly model natural language and meaning representation in a multilingual manner, thereby enabling the drs representations and the texts to be learnt in the same semantic space. this one model is then fine-tuned for parsing and generation. ## experiments for all experiments we use pmb release 4.0.0, which contains texts in english, german, dutch and italian for three levels of annotation (gold, silver and bronze). ## conclusion and future work using drs-based meaning representations as an additional language aside four different natural languages yields a novel multilingual pre-trained language-meaning model that can be fine-tuned for both semantic parsing and generation from formal meaning representations. by doing so, we achieve state-of-the-art performance on both tasks. exploiting parallel data and drs language neutrality is key to boost performance in lesser-resourced languages. we believe our approach can benefit from improvements in its current form, but also opens up to further research in language-meaning models. regarding future modelling directions, the contribution of graph structures should be further explored in future work. specifically, it could be possibly to leverage the graph structure to mask tokens in a more meaningful and principled way, designing a denoising training using the rich linguistic phenomena expressed by drss.