OSU C omp L ing at the GEM ’24 Data-to-Text Task (2024)

Abstract: This paper details experiments conducted for completing the GEM 2024 Data-to-Text task for a WebNLG dataset (Gardent et al., 2017). We show that model performance varies greatly across English, Spanish, Chinese, and Russian. Data filtering was done with automatic model judgments via error detection, which performs differently per language. We report English and Spanish dev set results for a data filtering and knowledge distillation approach to generating natural language outputs for sets of triples across a variety of domains. Specifically, we compare three generation conditions: 1) few-shot prompting with ChatGPT (GPT4), 2) fine-tuning LLama2 on the unfiltered dataset, and 3) fine-tuning Llama2 on a filtered version of the dataset. Russian and Chinese efforts did not result in submissions due to inconsistent or incoherent translations being produced in either the data synthesis or final generation stages. We provide details on these shortcomings but largely focus on Spanish and English efforts that align with our task submissions. We ultimately submitted outputs in English and Spanish that were generated using a version of Llama2 fine-tuned on a filtered dataset.

## introduction in the webnlg 2020 challenge, the osu system @xcite and others achieved nearly flawless english performance by carefully fine-tuning a pretrained model on the training set, though performance in russian was remarkably poor by comparison. since then, large language models (llms) like openai's chatgpt, anthropic's claude, and google's gemini have exhibited increasingly remarkable performance on a wide variety of tasks in multiple languages using in-context learning (i.e., few-shot generation), potentially obviating the need for human annotated training data. in this work, we first examine chatgpt's few-shot performance on this data-to-text task -converting webnlg-style logical triples to text -in a variety of languages and find several limitations in cases with lower-resource languages. coupling these limitations with the high computational and financial costs, as well as lack of consistent behavior across time, makes chatgpt difficult to rely upon for such tasks across languages. in this work, we leverage the robust capabilities of chatgpt but attempt to offset the aforementioned downsides via knowledge distillation with smaller, more economical open source models. following the example of @xcite , we leverage few-shot prompting to first generate training data using the chatgpt api, then to detect errors in the generated data and filter them out. the filtered and unfiltered datasets are then used to fine-tune a llama-2 @xcite model for the data-to-text task. this strategy was employed for english, spanish, russian, and chinese data but we ultimately found that it was not effective for the latter two. specifically, chinese fine-tuning llama2 (pretrained on chinese) on the initial filtered and unfiltered datasets yielded inconsistent outputs. this paper therefore describe alternative data filtering approaches and initial dev set results. on the other hand, russian experimentation was the least fruitful as there were numerous errors in the data synthesis and filtering stages. this paper describes some of the obstacles faces for this language and details our efforts to resolve these issues. this work shows that filtering synthesized training data (for spanish and english conditions) is a promising strategy for generating fluent, informative, and concise outputs. initial dev set results indicate that improvements to our filtration methods would ultimately maximize performance. by employing a knowledge distillation approach, we capitalize on chatgpt's expertise while lowering overall cost of generation by using an open-source model like llama2. our work also describes shortcomings of llms in completing this task for chinese and russian. by describing various data filtering efforts, we show potential paths forward in generating fluent and grammatically correct outputs across languages. ## related work the strategy of knowledge distillation in which we use chatgpt to generate synthetic data, apply automatic data filtering methods, and use the filtered data to train a downstream model is most closely related to the work of @xcite in which they use the same general approach to create a virtual tour guide for a museum. @xcite use gpt-3 to construct a conversational dataset for distilling a smaller t5 conversational model in a similar fashion, focusing on social conversation filtered for safety. @xcite setting involves retrieval augmented generation (rag) and both of these works involve chat functions, while our work here applies knowledge distillation and filtering in a data-to-text task. @xcite explore the viability of using llms, both with few-shot prompting alone and fine-tuning, for the task of semantic parsing in a conversational setting and find that the recent models are able to perform reasonably well on the task, though they investigate english only. our work also explores performance on other languages. @xcite use a self-refine approach to refine synthetic data, which, while not immediately relevant to this work, aligns with our efforts to refine synthetic data to improve the overall training set. other works that have utilized knowledge distillation for seq2seq models include @xcite and @xcite , who both explore distilling bert's bidirectional encoder knowledge into a seq2seq model for generation. our method can also be viewed as the first step in an approach combining knowledge distillation with self-training, not unlike @xcite , who implement self-training using an acceptability classifier @xcite and ultimately distill fine-tuned bart @xcite seq2seq models. more recent approaches also explores using llms in self-training with various refinements to how acceptability classifiers are incorporated @xcite . ## methodology we used the webnlg dataset @xcite to synthesize a training set, including factual, counterfactual, and fictional examples. we then experimented with few-shot prompting, data filtering, and knowledge distillation via fine-tuning. target languages were english, spanish, chinese, and russian. this section provides an overview of methods that apply to all target languages. section 4 details language-specific efforts that vary depending on problem cases. section 4 for spanish and english also provides automatic evaluations on dev set results and initial insights based on those findings. ## experiments this section details language-specific efforts and offers further insights as to how model performance differed per language. ## preliminary english results the task organizers provided preliminary results for the english data set @xcite . results are for factual, counterfactual, and fictional cases for the webnlg test set (i.e., the 'seen' subtask) and the wikidata test set (i.e., the 'unseen' subtask). as expected, our team's best performing category was webnlg fa with a bleu score of 30.03. webnlg fi performed worse than webnlg cfa with bleu scores of 21.44 and 24.45, respectively. the seen results for fa and fi results were higher than the unseen ones in the same categories with bleu scores of 24.97 for unseen fa and 16.9 for unseen fi cases. given aforementioned challenges with our cfa training data in terms of potentially nonsensical triple sets, it is not surprising that cfa unseen bleu score (27.06) outperformed the seen bleu score (24.45). fictional cases were our lowest performing for both subtasks. this may be due to the system's desire to adhere to real-world facts. as mentioned in section 4.1.2, more explicit prompting could have led to improved performance. compared to other systems that completed this data-to-text task, our system ranked in the mid-dle across categories. a difference that potentially led other systems to outperform our system is that we opted out of using existing supervised training data. instead, we chose to use a limited amount of few-shot synthetic data generated by chatgpt. because our efforts expanded across english, spanish, chinese, and russian, we avoided tactics that may give english an advantage over the other languages of interest (for which the same supervised training data was not available). we could have also experimented with generating more few-shot synthetic data and using self-training methods to improve our system's performance but did not due to expense and lack of time. while we did not have access to spanish results at the time of publishing, we expect overall improved performance due to increased training data set size compared to the english training set. ## discussion and conclusion for all languages, newer state of the art models such as llama3 (ai@meta, 2024) could have improved output performance -specifically in the cases of russian and chinese where incorrect translations were a large portion of errors qualitatively found. we also chose to not mention in the evaluation prompts how to handle non-factual triple sets and corresponding output being tested. the evaluations for counterfactual and fictional cases could potentially have been improved if the prompt included explicit instructions to ignore any non-factual information and focus on the representation of the triple sets in the output. a data synthesis and knowledge distillation approach yielded promising results in english and spanish. chatgpt was successful in synthesizing training data, but was less-than-ideal in acting as an evaluator -particularly for chinese. chatgpt was less successful in generating usable synthetic data for russian. further work could focus on refining our approach to data filtering and experiment with selftraining, which could potentially yield results comparable to or even exceeding few-shot prompting with chatgpt using a cheaper and more reliable open source model. based on the experiments presented in this paper, we see that llms perform better on data-to-text tasks for higher resource languages (english and spanish) and struggle with others (chinese and russian).