Data-to-text Generation for Severely Under-Resourced Languages with GPT -3.5: A Bit of Help Needed from G oogle T ranslate ( W eb NLG 2023) (2023)

Abstract: LLMs are great at tasks involving English which dominates in their training data. We explore their ability to address tasks involving languages that are severely under-represented in their training data. More specifically, we do this in the context of data-to-text generation for Irish, Maltese, Welsh and Breton. During the prompt-engineering phase we tested GPT-3.5 and~4 with a range of prompt types and formats on a small sample of example input/output pairs. We then fully evaluated the two most promising prompts in two scenarios: (i) direct generation into the under-resourced languages, and (ii) generation into English followed by translation into the under-resourced languages. We find that few-shot prompting works better for direct generation into under-resourced languages, but that the difference disappears when pivoting via English. The few-shot + translation system variants were submitted to the WebNLG 2023 shared task where they outperformed all other systems by substantial margins in all languages on all automatic metrics. We conclude that good performance can be achieved with state-of-the-art LLMs out-of-the box for under-resourced languages. However, best results (for Welsh) of BLEU 25.12, ChrF++ 0.55, and TER 0.64 are well below the lowest ranked English system at WebNLG’20 with BLEU 0.391, ChrF++ 0.579, and TER 0.564.

## introduction pretrained large language models (llms) are everywhere, achieving state-of-the-art-results across many application areas. llms display strong outof-the-box abilities at a wide range of different tasks far beyond language-based tasks. however, these abilities have been explored mostly for english which dominates text online, hence the data on which llms are trained. in the work reported here, we explore to what extent state-of-the-art llms can help with tasks involving severely underresourced languages which by definition would have been very scarce in their training data. more specifically, we address the challenging task of data-to-text generation for four severely under-resourced languages: irish, maltese, welsh and breton. developing any new system is difficult in a severely under-resourced scenario, more so where the task has language-specific input/output combinations as in data-to-text and text-to-text. the paper is structured as follows. section 2 describes data and task, section 3 presents the general approach, prompt engineering, and the specific systems we fully evaluated. experimental set-up and results are outlined in section 4, and section 5 provides discussion and conclusions. all code and results are available on github: https://github.com/dcu-nlg/dcu-nlg-pbn . ## data and task the webnlg'23 (?) data consists of 1,778 dev items for each language, 1,399 test items for breton, and 1,665 for welsh, irish and maltese, that were manually translated by professional translators from the english originals. additionally 13,211 training items are provided where texts were automatically translated from english. as in all webnlg shared tasks, webnlg 2023 systems must map from the data (rdf triples) to a suitable output text, as in the example from the webnlg'23 website foot_0 in figure 2 . this is a datato-text generation task which is well established in the natural language generation (nlg) field. ## general approach and specific systems the general approach we take is to use a state-ofthe-art llm in its out-of-the-box state, to either (i) generate texts directly from the data in the target language, or (ii) generate texts in english and then translate them into the target language. more specifically, the llm we used was the instruct-gpt model identified as text-davinci-003, and the mt system was google translate which covers all webnlg'23 languages except breton. in all cases, we fed a prompt representing the generation task to text-davinci-003, and collected the system completion as the output which we then cleaned up by simple post-processing procedures. ## experimental set-up and results we executed our experiments using the paid-for openai api to access text-davinci-003, foot_2 and the free google translate api, foot_3 in early june 2023. in all experiments, we set text-davinci-003 parameters to @xmath0, top @xmath1, frequency @xmath2, best @xmath3, and maximum length to 500. all generated texts are postprocessed as described above. comparing zero-shot results with corresponding few-shot results, it is clear that the addition of examples to the prompt helps the model perform the task better in all languages: bleu and ter are always higher for the few-shot variant than the zero-shot variant, while chrf++ is always higher for few-shot where translation is not used, and virtually identical where translation is used. when generating the texts in english plus translation, we don't see the same strong benefit achieved by the addition of few-shot examples noted above. even where there is a benefit, it is small. in fact, we observe similar performance between zero-shot + mt and few-shot + mt throughout, while both clearly outperform versions without translation; this is the case in all three languages where we have translation (irish, maltese, and welsh). it might be speculated that providing few-shot examples partially compensates for the comparatively smaller amount of knowledge that the llm has about the under-resourced languages. this would explain why the benefit disappears when pivoting via english, where examples can add little to the knowledge the llm already has. ## discussion and conclusion we submitted the three few-shot + mt systems to the webnlg challenge 2023, 6 in the welsh, irish and maltese tracks. based on the metric evaluation results (test set) shared by webnlg organisers, these systems outperform all other submitted systems by sizeable margins; moreover, few-shot breton without translation ( * ) also outperforms the one submitted system substantially: few-shot + gtransl nearest system bleu chrf++ ter bleu chrf++ ter irish 20.4 0.51 0.69 16.66 0.44 0.75 maltese 21.27 0.52 0.65 16.49 0.47 0.7 welsh 25.12 0.55 0.64 20.97 0.49 0.67 breton * 17.21 0.44 0.74 13.11 0.33 0.76 at the time of writing, we do not know what kinds of systems produced the runner-up results, except that the irish runner-up is the handcrafted rulebased forge system extended for irish (?). there are interesting differences between forge and our system. forge took a considerable amount of person time, but uses no paid-for resources. to 6 https://synalp.gitlabpages.inria.fr/webnlg-c hallenge/challenge_2023/docs create and test our system took next to no time, and us$91.82 in api costs. for forge, the system and how it was created is known. for our system, we don't have access to the model itself, and only incomplete knowledge about its exact technical details at the particular time we access it; on another day, results may differ. this makes experiments non-repeatable, and generalising to future use of the same approach difficult. moreover, it's not a system that can be deployed in this form, because of the changing nature of what's at the other end of the llm and mt apis. this raises important general questions about (i) how scientific research is to be conducted with llms served as-a-service that deliver best performance, but whose technical details aren't known and moreover change over time; and (ii) how such research results can be utilised in developing systems for real-world deployment. in this paper we have explored different prompt types and formats for data-to-text generation for under-resourced languages, with out-of-the-box llms with and without machine translation. we found that few-shot + mt achieves new state-ofthe-art metric performance at the webnlg'23 rdf-to-text generation task for the severely underresourced languages irish, maltese, welsh and breton. however, given the caveats above, results presented here must be treated as a rough indication of what is achievable with an llm+mt approach. triple 1: """ {set of triples in the format subject predicate object and each triple in a new line} """ text 1: {verbalisation of triple 1} ## triple 2: """ {set of triples in the format subject predicate object and each triple in a new line} """ text 2: {verbalisation of triple 2} ## triple 3: """ {set of triples in the format subject predicate object and each triple in a new line} """ text 3: [model] example prompt: write the following triples as fluent english text. triple 1: """ adolfo_suárez_madrid-barajas_airport runwayname "14r/32l" """ text 1: 14r/32l is the runway name of adolfo suárez madrid-barajas airport. ## triple 2: """ american_journal_of_mathematics abbreviation "am. j. math." american_journal_of_mathematics firstpublicationyear 1878 american_journal_of_mathematics issnnumber "1080-6377" """ text 2: the american journal of mathematics was first published in 1878 and is also known by the abbreviated title of am. j. math. it has an issn number of 1080-6377. ## triple 3: """ ac_hotel_bella_sky_copenhagen owner bella_center ac_hotel_bella_sky_copenhagen tenant marriott_international ac_hotel_bella_sky_copenhagen architect 3xn ac_hotel_bella_sky_copenhagen floorcount 23 """ text 3: model output: the ac hotel bella sky copenhagen is owned by bella center and is tenanted by marriott international. it was designed by 3xn and has 23 floors.