DCU - ADAPT -mod PB at the GEM ’24 Data-to-Text Generation Task: Model Hybridisation for Pipeline Data-to-Text Natural Language Generation (2024)

Abstract: In this paper, we present our approach to the GEM Shared Task at the INLG’24 Generation Challenges, which focuses on generating data-to-text in multiple languages, including low-resource languages, from WebNLG triples. We employ a combination of end-to-end and pipeline neural architectures for English text generation. To extend our methodology to Hindi, Korean, Arabic, and Swahili, we leverage a neural machine translation model. Our results demonstrate that our approach achieves competitive performance in the given task.

## introduction the gem 2024 shared task @xcite aims to advance summarisation and data-to-text (d2t) generation, with a particular focus on enhancing multilingual capabilities. the d2t task @xcite involves generating coherent natural language text from structured data in the form of wikidata and webnlg datasets, which are organised as triples consisting of a subject, predicate, and object. the goal of the tasks is to comprehensively evaluate and improve the ability of systems to interpret and generate text from rdf triples, assess their general knowledge, and produce texts in factual (fa), counterfactual (cfa), and fictional (fi) scenarios. the dominance of english in d2t generation presents a considerable challenge, highlighting the need for research to support effective multilingual generation, particularly for languages with diverse morphological structures and distinct word order characteristics. the gem 2024 shared task addresses this challenge by including english alongside other languages such as chinese, german, russian, spanish, korean, hindi, swahili, and arabic, which are low-resource in the d2t setting. this task aims to enhance the adaptability and robustness of different systems across varied linguistic frameworks for text generation from structured data. in this submission, we focus on the d2t generation aspect of the task using the webnlg dataset @xcite . our approach combines end-to-end and pipeline neural architectures to generate english text, while also fine-tuning a state-of-the-art open-source flan-t5 and mistral-7b large language models (llms) for generating text in low-resource languages. our approach aims to further the understanding of how various architectures can be optimised for multilingual d2t generation. our methodology demonstrates competitive performance and contributes substantial insights and advancements to the field of multilingual d2t generation. the code and results are available foot_1 . ## related work the field of data-to-text generation has undergone significant transformations, evolving from traditional pre-neural approaches that relied on handcrafted rules, templates, and statistical models @xcite to modern deep learning architectures. these advanced models are trained to identify and replicate the relationships between structured data and its corresponding textual outputs. the introduction of end-to-end systems, particularly pre-trained language models (plms), has substantially improved the processing of textual sequences in data-to-text tasks @xcite . however, despite their advanced capabilities, these systems often struggle with content selection and maintaining fidelity due to the opaqueness and complexity inherent to deep learning models and the data-totext generation task @xcite . a recent example of methodological advancement in this field is showcased in the 2023 webnlg shared task on low resource languages, where many participants employed nlg+mt (natural language generation plus machine translation) pipeline approach @xcite . for instance, some participants implemented systems which generate english text from rdf graphs using a plm fine-tuned on the webnlg 2020 dataset, followed by translation into various languages using a machine translation (mt) model @xcite . this approach showcases the potential of combining nlg and mt models for effective multilingual data-to-text generation. similarly, @xcite proposed a novel approach that utilises large language models (gpt-3.5 and gpt-4) for prompt-based generation without additional training. they tested two methods: direct generation in under-resourced languages and generation in english followed by translation using google translate. in our research, we build upon these methodologies and incorporate a 3-stage pipeline neural architecture, as in figure 1 , inspired by @xcite . however, we modify the approach by integrating only the first two stages of ordering and structuring, followed by the final stage of surface realisation. this approach aims to optimise the use of large language models for multilingual data-to-text generation. ## methodology in this section, we outline the methodologies employed to address the generation challenge for the languages english (en), hindi (hi), korean (ko), arabic (ar), and swahili (sw). our experimental setup is as follows: ## results in our results' naming convention, "struct" denotes the pipeline architecture system that utilises structured triples for generation. "finetunedmistral" refers to the fine-tuned mistral-7b-instruct system, while systems without these acronyms represent direct generation using the base models within the end-to-end architecture. furthermore, we employed the comet metric @xcite , a neural evaluation model specifically designed to predict quality scores for translations. comet is known for demonstrating a strong correlation with human judgement and is capable of performing reference-less evaluations. this capability makes comet particularly well-suited for assessing our results in non-english languages within the fa dataset, as well as for all languages in the cfa and fi datasets, where reference translations are not yet available. the results of our evaluation using comet are presented in ## analysis and discussion in this analysis, we highlight the factors which may have contributed to the varying performances of the models in our experiments. first, the underlying architecture and training data play a critical role. we observe that our gpt4based systems benefits from extensive training on a large and diverse dataset, which likely contributes to its consistent performance across different languages and domains. the robustness of its architecture allows it to handle a wide range of tasks effectively. however, we observed a decline in performance within the fi dataset. upon manual inspection, we found that the system generated text with the correct entities but often rejected certain entity claims in the dataset, leading to its overall poor performance in this category. second, the fine-tuning process and the nature of the tasks significantly influence performance. structgpt4, for instance, is fine-tuned with a focus on specific tasks (i.e., ordering and structuring) requiring precision and the handling of complex or nuanced content, which explains its superior performance in bleu and ter, especially in fa english text generation. third, language-specific optimisations or model adaptations can lead to better performance in certain languages. mistral shows strong results in korean and swahili, which may indicate that it has been trained or optimised for these specific languages, allowing it to outperform gpt4 and structgpt4 in these contexts. fourth, the evaluation metrics themselves might favour certain models depending on how they align with the strengths of each model. for example, structgpt4 performs better in bleu and ter, metrics that emphasise precision and reduced errors, while gpt4 excels in meteor and chrf++, which also account for semantic accuracy and fluency. these factors highlight the importance of select-ing models based on the specific requirements of the task, considering not only the general capabilities of the model but also how well it has been optimised or fine-tuned for particular languages and tasks. to fully harness the aggregate benefits of the various factors influencing the performance of models as identified in our experiment, future work should focus on conducting a comprehensive exploration of each aspect. this may involve: ## conclusion and future directions in conclusion, this paper presents the methodologies and automatic evaluation results of our submission to the gem 2024 tasks. the evaluation results highlight the strengths of different models across various metrics and languages. structgpt4 stands out in producing precise translations with fewer errors, especially in english, outperforming gpt4 in metrics like bleu and ter. gpt4, however, proves to be the most versatile and high-performing model across multiple languages and domains, excelling in meteor, chrf++, bert_f1, and comet metrics, although it shows limitations in generating text within the fi task. mistral demonstrates strong performance in languages such as korean, hindi, and arabic, particularly within the fi task, while structmistral excels in swahili cfa tasks. these findings suggest that while gpt4 is the most reliable general-purpose model, structgpt4, due to its incorporation of task splitting and pipelining, is better suited for tasks requiring minimal errors, high accuracy, and attention to detail. meanwhile, mistral and structmistral offer valuable performance in specific applications, indicating their potential for specialised use cases. in order to gain a more comprehensive understanding of our systems' performance, we look forward to the availability of human evaluation results, which will provide valuable insights and enable us to draw further conclusions. moreover, we plan to further explore the impact of advanced fine-tuning methods with preference-based learning, such as recent state-of-the-art frameworks like dpo @xcite , kto @xcite , sppo @xcite and the rein-force @xcite preference optimisation. these methods have shown promise in improving model alignment and generation performance, and we believe they could be valuable additions to our existing systems. we will also investigate the possible impact of data selection and prompt engineering methods on optimising our existing systems. studies, for example in @xcite have shown that carefully selecting and preparing high-quality data for llm finetuning often leads to improvement in model performance. this is because high-quality data allows the model to learn from relevant and accurate examples, which is crucial for fine-tuning the model's parameters and achieving optimal performance. lastly, we are keen on investigating the development of an end-to-end framework that encompasses ordering, structuring, and text generation collectively. this would allow us to streamline our pipeline and potentially improve the overall performance of our systems.