Fine-tuning with HED - IT : The impact of human post-editing for dialogical language models (2024)

Abstract: Automatic methods for generating and gathering linguistic data have proven effective for fine-tuning Language Models (LMs) in languages less resourced than English. Still, while there has been emphasis on data quantity, less attention has been given to its quality. In this work, we investigate the impact of human intervention on machine-generated data when fine-tuning dialogical models. In particular, we study (1) whether post-edited dialogues exhibit higher perceived quality compared to the originals that were automatically generated; (2) whether fine-tuning with post-edited dialogues results in noticeable differences in the generated outputs; and (3) whether post-edited dialogues influence the outcomes when considering the parameter size of the LMs. To this end we created HED-IT, a large-scale dataset where machine-generated dialogues are paired with the version post-edited by humans. Using both the edited and unedited portions of HED-IT, we fine-tuned three different sizes of an LM. Results from both human and automatic evaluation show that the different quality of training data is clearly perceived and it has an impact also on the models trained on such data. Additionally, our findings indicate that larger models are less sensitive to data quality, whereas this has a crucial impact on smaller models. These results enhance our comprehension of the impact of human intervention on training data in the development of high-quality LMs.

## introduction the landscape of language technology is constantly being populated by new instruction-based and conversational models, thanks to the increased availability of foundational large language models (llms) and easy access to fine-tuning support. especially for lesser-resourced languages, the standard procedure is to automatically translate existing english datasets and use those to fine-tune one figure 1 : data creation pipeline. the author (machine) provides initial dialogues, the reviewer (human) postedits them. these two datasets are then used to train different sizes of an lm. automatic and human evaluation is performed both on training data and corresponding trained models to assess the impact of curation. still, this proliferation of models is not matched by progress in evaluation efforts aimed at understanding exact differences, especially in terms of data and model quality as perceived by humans (rather than just overall model performance). gaining such insights would help in optimising the balance between cheaper and more costly choices, such as pure automatic translation vs manual intervention, in the creation of datasets for developing high-quality models. this is the aim of the present contribution, focusing on italian dialogic models, but applicable to other under-resourced languages. by building on previous research revolving around the concept of data quality, we adopted an authorreviewer architecture @xcite to create the hed-it (human edited dialogues for italian) dataset foot_0 where automatically generated dialogues are post-edited by humans. we then use such edited dialogues, as well as the unedited, to fine-tune an llm in three different sizes. this setup enables us to study the impact of human intervention on training data (vs its absence) on the final resulting models. we do so in terms of (i) quality assessment by humans of llm output; and (ii) behaviour assessment across model sizes. figure 1 provides an overview of our approach. the research questions posed and addressed in this paper can be summarised as follows: q1 we investigate whether post-edited dialogues to be used for fine-tuning are (perceived as) better in quality than the original (automatically generated) ones. automatic metrics and dedicated manual evaluation suggest that this is the case. ## dataset creation to create the hed-it dataset, we used the authorreviewer pipeline proposed by @xcite , in which an llm (the author component) produces novel data, while humans (the reviewer component) filter and eventually post-edit them. we tested several author configurations and introduced two adaptations to the original proposal: (i) following @xcite , rather than finetuning a gpt model, we opted for an instructionbased llm (i.e., chatgpt (openai, 2022)) that does not require fine-tuning; (ii) we extend the concept of a machine-based author in this setting to include also the automatic extraction of excerpts from existing human-produced text. to create the dataset, we used non-real data sourced from movie scripts and synthetic data to address potential privacy and ephemerality concerns associated with real data @xcite . the following sections provide a detailed overview of the sources and methodologies used to construct the hed-it dataset. ## dataset analysis following the author-reviewer process, two versions of our dataset were collected: one with the original dialogues (hed-it orig ) and another with the post-edited versions (hed-it p-e ). hed-it orig comprises over 9000 dialogues and more than 66000 turns, whereas hed-it p-e comprises over 7000 dialogues and almost 48000 turns. on average, dialogues contain 8 turns. more details are reported in to assess the impact of post-editing both quantitatively and qualitatively, we use automatic metrics and a human evaluation. ## experimental design using the hed-it dataset, our objective is to understand the impact of post-editing on the performances of an llm. we evaluated the models automatically using the test set gold standard, and ran human evaluations of models' outputs. ## results in this section, we will outline the results obtained on the six configurations we tested. ## conclusions machine-generated data is commonly used to finetune llms in low-resourced languages, while improving data quality has received little attention. to this end, we extensively investigated the impact of human intervention on data used for fine-tuning dialogical models. we provided a corpus of machinegenerated italian dialogues paired with post-edited counterparts, and trained an llm in three different sizes using this corpus. results clearly showed that post-edited dialogues had a higher perceived quality compared to their original counterparts. model evaluation also indicates an overall quality improvement for the models fine-tuned with post-edited data. still, results suggest that the effect of postediting is stronger for smaller models while slowly fading for bigger ones, and this can be a critical aspect when running a data collection campaign. also the intervention of machine-rewriting before human post-editing can be beneficial when available training data, even if human-written, is silver with respect to the target domain.