DCU - NLG -Small at the GEM ’24 Data-to-Text Task: Rule-based generation and post-processing with T5-Base (2024)

Abstract: Our submission to the GEM data-to-text shared task aims to assess the quality of texts produced by the combination of a rule-based system with a language model of reduced size, by first using a rule-based generator to convert input triples into semantically correct English text, and then a language model to paraphrase these texts to make them more fluent. The texts are translated to languages other than English with the NLLB machine translation system.

## introduction on the one hand, very large language models are able to produce human-like texts from structured data but require enormous amounts of energy and computational resources to be trained, fine-tuned and run; on the other hand, resource-efficient techniques such as rule-based systems generally output texts that are less than optimally fluent. for our submission, we used three components: (i) a rulebased generator, forge @xcite to generate all inputs in english, (ii) a small-sized language model, t5-base @xcite , finetuned for rephrasing the rule-based outputs in a more fluent way, and (iii) an off-the-shelf machine translation system, nllb @xcite , for producing outputs in languages other than english. our hypothesis is that using a language model for paraphrasing textual output produced by a reliable rule-based generator, rather than for directly mapping from triples to text, will make the system (i) more accurate in term of contents, i.e. less prone to omissions and additions (since all the contents of the input triples are already verbalised in the input of the language model), and (ii) generalise better to out-of-domain data, which represents five out of the six test sets of the gem d2t task (since for the language model, instead of verbalising, the task is 1 our code and data is available at https://github.com/ dcu-nlg/gem24-dcu-nlg-small . ## the gem d2t shared task in gem d2t @xcite , the task is to generate texts in various languages starting from input triples extracted from dbpedia (subtask 1) or wikidata (subtask 2) triples; see figure 1 for an example of an input/output pair. each subtask has three test sets: (i) a factual dataset (fa), which contains only factually correct information; (ii) a counterfactual dataset (cfa), which is the factual dataset but with entities (subjects and objects, see figure 1 ) replaced by other entities of the same category (e.g. a person is replaced by another person, a date by another date, etc.); and (iii) a fictional dataset (fi), in which all subject and object values are fictional names made up by a language model. the d2t-1 data is derived from webnlg data @xcite , while the d2t2 data was created for the present task using the method proposed by @xcite (i.e. collection of new wikidata triples sets for a list of entities, and then replacement of entities according to steps (ii) and (iii) above). no training data was provided to the participants, and apart from the english factual webnlg data (i.e. the original test set in @xcite ), no reference texts were available for any test set or language. the gem organisers encouraged submissions in multiple languages, namely english (en), chinese (zh), german (de), russian (ru), spanish (es), korean (ko), hindi (hi), swahili (sw), and arabic (ar), without saying beforehand which languages were going to be assessed. ## rule-based generator for our rule-based system, we use the forge generator @xcite , which was partly developed on the webnlg data. forge is implemented as a pipeline of modules that perform subtasks such as text planning, lexicalisation, sentence structuring and surface realisation. each module consists of a set of rules (called grammars), which use dictionaries that describe the semantic and syntactic behaviours of the lexical units used in the verbalisations. the generator takes as input abstract predicate-argument structures manually crafted for each property, as shown in figure 2 . forge already has such predicate-argument structures for the whole webnlg 2020 dataset in english, which means that we were able to use forge off-the-shelf for subtask 1; no modification was performed to address new entity names of the fictional test set. for subtask 2, properties in the dataset built by the organisers come from the wikidata vocabulary, which is different from the dbpedia vocabulary used in the webnlg dataset. there are 74 different wikidata properties, 17 of which have a direct mapping to a dbpedia property. for these 17 properties, we use the existing predicateargument templates, while for the remaining 57 properties, new predicate-argument templates were crafted, referring to the wikipedia pages of the en-tities used along each property to make sure we captured the correct semantics of each property. crafting the 57 templates took approximately 2 hours. minor updates to the generator's grammars were implemented to account for the specific aspects of the wikidata test sets, in which the subject is always the same, unlike in the webnlg-based inputs. ## finetuning datasets our objective in the paraphrasing component is to improve the fluency of the rule-based generator without sacrificing its semantic accuracy (i.e. avoiding what is commonly reported as omissions and hallucinations). for this, we collected parallel textual data, with on one side accurate but possibly disfluent texts (text dis ), and on the other side accurate and fluent texts (text f lu ). in this section, we describe the three different datasets we created for the experiments; section 5 reports on how we used this data for fine-tuning t5. ## machine translation with nllb the combination of resources we need for applying our approach (rule-based generator + parallel textual data) is currently only available in english. for producing outputs in other languages, we used the freely available nllb machine translation tool @xcite . nllb is a pre-trained model that covers translation between numerous languages; it is available through huggingface foot_6 and can be executed on various types of runtimes, including cpus. each english text was split into sentences, and sentences were processed one at a time by nllb; the translated sentences were then brought back together as a text and stored in the same format as the english outputs, in a .txt file with one text per line. we ran nllb-200-distilled-1.3b on a t4 gpu on google colab, which generally needed between 30 and 60 minutes to translate a file. with 48 files to translate (8 target languages, 6 test sets per language) and several server interruptions, we had to finish the translations on a local hpc cluster to finish the translations on time. we did not try to improve the translation quality, and did not perform any systematic qualitative analysis of the translated texts; for a few languages (spanish, hindi, german), we asked native speakers to browse through a few translations to have an idea of the general quality, which was judged sufficient to submit the outputs. ## results and submitted systems in this section, we present the results of evaluating our 12 models on the d2t-1-fa test data, using the webnlg 2020 reference texts for calculating bleu @xcite and me-teor @xcite , based on which we selected the model for our submission. we then briefly discuss the official results of the gem d2t task as provided by the organisers, which at this point are restricted to the metrics scores for english (we report bleu, meteor and bertscore-f1 @xcite ). ## conclusions we have presented the dcu-nlg-small submission to the gem'24 data-to-text shared task. our system combines a rule-based generator that converts triples into english text, with a small language model that paraphrases the text to improve its fluency. an off-the-shelf mt system is used for producing outputs in the other languages. our system performs better than a purely rule-based system according to metrics on an existing english test set, but generally undergoes substantial score decreases when confronted with different types of out-of-domain data. we hope that the human evaluation results will allow us to draw more definitive conclusions.