Better Translation + Split and Generate for Multilingual RDF -to-Text ( W eb NLG 2023) (2023)

Abstract: This paper presents system descriptions of our submitted outputs for WebNLG Challenge 2023. We use mT5 in multi-task and multilingual settings to generate more fluent and reliable verbalizations of the given RDF triples. Furthermore, we introduce a partial decoding technique to produce more elaborate yet simplified outputs. Additionally, we demonstrate the significance of employing better translation systems in creating training data.

## introduction state-of-the-art in natural language generation (nlg) recently has benefited from self-supervised pretrained language models (plms) such as bart, t5, and gpt-2 @xcite @xcite , which provided performance improvements on many nlg tasks @xcite @xcite . one of the most prominent nlg tasks is data-to-text generation @xcite , with multiple public datasets, including webnlg @xcite , @xcite , or dart @xcite . a significant problem of these datasets is their limitation to english only, which effectively rules out their application to other languages; incorporating low-resource languages in nlg is generally an open problem @xcite . researchers addressed this by multilingual plms like mt5 @xcite and mbart @xcite pre-trained on massively multilingual corpora like common crawl 101. 1 unfortunately, given the disproportionate representation of low-resource languages in multilingual datasets, 2 the efficacy of plms on these languages is compromised. efforts are being made to improve plm performance for low-resource languages via crosslingual transfer @xcite . to assess the state-of-the-art performance and spur development in data-to-text nlg techniques for lowresource languages, the webnlg 2023 challenge features five non-english languages -four entirely new low-resource ones: maltese (mt), welsh (cy), irish (ga) and breton (br), with the addition of russian (ru) from the 2020 iteration @xcite . while development and test sets for the low-resource languages were created by hand, training sets are machine-translated. in our submission of webnlg 2023 challenge, we use the power of mt5 @xcite , a multilingual transformer model trained in a multi-task fashion. to boost performance on low-resource languages, we employ several additional steps in finetuning and inference: (1) we improve webnlg 2023 training data by re-translating with a stronger machine translation (mt) model @xcite . (2) we finetune either individual models for each language (single-task setting), or a single model for all languages (multi-task setting). (3) we generate text by splitting the input rdf triples and decoding a subset of triples at a time. we publish our code and submitted outputs on github. 3 ## related works a lot of non-english nlg works addresses either surface realization @xcite ) or text-to-text tasks such as summarization @xcite @xcite or question generation @xcite . in data-to-text generation, @xcite trained a rnn-based model for czech, van der lee et al. (2020) apply a similar architecture for dutch. a rnn decoder has also been applied to japanese in a multimodal setup @xcite . recent works mostly use transformer-based architectures. kale and roy (2020) applied a custom transformer pretrained for mt to a czech data-to-text task. several systems targeted webnlg 2020's russian task @xcite . @xcite 's system features a custom-trained bilingual t5-derived transformer, the works of @xcite and @xcite build on the standard mbart model. our approach combines the use of mt for preprocessing with standard multilingual plms and adds multi-task learning and sentence-level generation. ## webnlg 202task description the webnlg challenge focuses on generating text from a set of rdf triples. input rdf triples are extracted from dbpedia @xcite and the corresponding reference texts are gathered through crowdsourcing. previous iterations @xcite focused on english and russian. the 2023 webnlg challenge includes 4 low-resource languages (mt, cy, ga, br), along with russian (ru). the development and test sets are manually translated from english, but the training data is a result of mt by the edinburgh zero translation system @xcite . ## proposed approaches our own approach builds on multilingual transformer plms, but improves input data processing in the low-resource languages by better translation or filtering (section 4.1) and employs two simple yet effective strategies to improve generation: multitask learning (section 4.2) and splitting complex inputs for decoding (section 4.3). ## results unlike ru, mt ( nllb (tables 1c and 1d ), similar to ru. there is a slight improvement in the performance of sag decoding over mtl, but not in the mtm case. while our direct translation models outperform the challenge baseline, they fare worse than generate-andtranslate with nllb. the scores for br are shown in ## conclusion in this work, we describe our submitted systems for the webnlg 2023 challenge. the provided silver training dataset for the under-resourced languages sometimes contains incomplete translations. to address this, we create alternate parallel data using nllb, which seems to provide the biggest performance boost. furthermore, we gain additional minor improvements by using a multi-task and multilingual training settings, coupled with a split-and-generate decoding method, which produces simpler and more verbose outputs. our systems mostly score in the middle of the challenge scoreboard; our br submission was the only competitor in the challenge. for ru, ga and cy, our submitted systems perform substantially better than the organizers' baseline. a non-submitted variant of our system for br also surpasses the baseline. however, in mt, ga, and cy, our approaches underperform a simple generate-and-translate baseline with the improved nllb mt system.