High-quality Data-to-Text Generation for Severely Under-Resourced Languages with Out-of-the-box Large Language Models (2024)

Abstract: The performance of NLP methods for severely under-resourced languages cannot currently hope to match the state of the art in NLP methods for well resourced languages. We explore the extent to which pretrained large language models (LLMs) can bridge this gap, via the example of data-to-text generation for Irish, Welsh, Breton and Maltese. We test LLMs on these under-resourced languages and English, in a range of scenarios. We find that LLMs easily set the state of the art for the under-resourced languages by substantial margins, as measured by both automatic and human evaluations. For all our languages, human evaluation shows on-a-par performance with humans for our best systems, but BLEU scores collapse compared to English, casting doubt on the metric’s suitability for evaluating non-task-specific systems. Overall, our results demonstrate the great potential of LLMs to bridge the performance gap for under-resourced languages.

## introduction automatically generating text for a given data set (e.g. a textual summary) is a much bigger challenge for severely under-resourced languages than for well resourced languages like english. creating a rule-based system by hand is one option: slow but faster if language-independent resources can be used @xcite ). an alternative is taskspecific finetuning and collecting training data for it (partly) by hand and/or by collecting/generating silver training data which may be good enough to achieve a desired performance level. these methods all take varying but considerable amounts of manual work and time. in contrast, using large language models (llms) in their 'out of the box' state has next to no such overheads. however, at this point their zero-shot ability to generate correct text of sufficient quality (e.g. in terms of minimum real-world usefulness where first-draft plus post-editing takes less time than from-scratch) for severely under-resourced languages is untested. given that by definition llms will have seen very little text in under-resourced languages during training, using them in zero-shot mode for text generation in such languages may not seem a promising idea. in this paper, we explore the extent to which it is possible for data-to-text generation, in so doing shedding light on the potential of llms to bridge performance gaps between under-resourced languages (the vast majority of the world's languages) and well resourced languages like english. all code and results are available on github: https://github.com/michelalorandi/ d2t-gen-for-under-res-lang-w-llms . ## related research a large number of papers in the past year have reported work on using llms, and gpt in particular, in zero or few-shot mode for a wide range of different tasks, including both system development @xcite @xcite @xcite and evaluation @xcite @xcite @xcite . because the performance of zero-shot llms depends on the quality of the prompt, there has been a corresponding flurry of research on prompt engineering, including plan-and-solve prompting @xcite , tree-of-thought prompting @xcite , and automatic prompt fixing @xcite . webnlg 2023 (see below) included a first attempt @xcite to perform datato-text generation for under-resourced languages using out-of-the-box gpt-3.5 plus google translate which outperformed other participating systems by considerable margins. we take the same approach but test four llms and three mt systems (two closed source and one open source) in a wider range of scenarios, and additionally test our best system on english where the tough state-of-the-art outperforms humans. ## data and task webnlg 2023 is the third iteration of the webnlg shared task series and focuses on the severely under-resourced european languages irish, breton, welsh and maltese foot_0 @xcite . the webnlg 2023 data consists of 1,778 test items for each language, 1,399 dev items for breton, and 1,665 dev items for welsh, irish and maltese. the test sets were manually translated by professional translators from the english originals. additionally 13,211 training items are provided where texts were automatically translated from english. webnlg 2023 systems map from rdf triples to a suitable output text, as in the example from the webnlg'23 website foot_1 in figure 1 . the complete shared-task data is available from the same website. ## models we test four different pretrained llms (paid-for gpt-3.5, and open-source bloom, llama2-chat, and falcon-chat), each in two modes: (i) direct generation into the target language, and (ii) generation into english followed by translation into the target language with one of three machine translation (mt) engines (google translate, alibaba translate, and no language left behind system (costa-jussà et al., 2022)). gpt-3.5 or instructgpt @xcite is gpt-3 plus supervision fine-tuning on instruction data, reward model training and reinforcement learning with human feedback (rlhf) with the reward model. bloom (scao et al., 2022) is trained on the roots corpus, a collection of 498 huggingface datasets. llama2-chat @xcite builds on the pretrained llama2 model (trained only on publicly available datasets) fine-tuned in two steps similar to gpt-3.5, but instead of using one reward model for helpfulness and safety, two separately optimised reward models are used. falcon-chat @xcite builds on falcon-base, which is trained on the re-finedweb dataset @xcite . falconbase is then fine-tuned on chat and instruction datasets with a mix of large-scale conversational datasets. ## experimental set-up in this section we describe the main aspects of the experimental set-up. hyperparameters and api access are provided in section a.1 in the appendix. ## results this section reports the main human and metric evaluation results. details of cost in section a.2. ## discussion and conclusion one striking aspect of the metric results for the under-resourced languages is that bleu scores are far lower across the board than for english. at the same time, human evaluations show on-a-par-withhuman performance for both the under-resourced languages and english. this shows a significant performance failure for bleu that is not reflected in chrf++, ter, bertscore or comet. this bleu failure may be due to two aspects: for one, bleu is a word n-gram overlap metric, while chrf++ and ter are character f-score and character edit distance based, respectively. bertscore computes cosine similarity for each token in candidate and reference sentences using the pre-trained contextual embeddings from bert, and comet uses a pretrained multilingual model trained to mimic human judgement. two, the gpt training data is likely to have contained the english webnlg data in its entirety (albeit not as input/output pairs), but not any of the underresourced language outputs. it seems that under these circumstances, where system outputs and reference texts have not been sampled from the same narrow distribution, bleu simply does not work. the systems that we introduce and test here are generic, non-task-specifically trained systems. all of the systems we compare them against are taskspecifically supervision-trained systems, and in one case @xcite , hand-crafted to perform a single specific task. it is yet another piece of evidence showcasing the astonishing out-of-the-box abilities of the latest generation of llms. similarly to previous evidence, we see that absence of instruction tuning (bloom) and smaller size (llama2) are associated with poorer performance. it is also unclear how such systems can be utilised in real-world application scenarios. however, we show the incredible ability of llms to generate texts on-a-par performance with humans for our best systems in all languages tested.