W eb NLG -Interno: Utilizing FRED -T5 to address the RDF -to-text problem ( W eb NLG 2023) (2023)

Abstract: We present our solution for the Russian RDF002 to-text generation task of the WebNLG Challenge 2023. We use the pretrained large language model named FRED-T5 (Zmitrovich et al., 2023) to finetune on the train dataset. Also, we propose several types of prompt and run experiments to analyze their effectiveness. Our submission achieves 0.373 TER on the test dataset, taking the first place according to the results of the automatic evaluation and outperforming the best result of the previous challenge by 0.025. The code of our solution is available at the following link: https://github.com/Ivan30003/webnlg_interno

## introduction recently released large language models (a.k.a. llms) like @xcite , bloom @xcite , llama @xcite and @xcite proved the ability of deep neural networks to generate realistic texts, maintain a human-like conversation and answer factual questions based on the information contained in the training data. however, real-life applications of llms could benefit from extracting relevant information from external databases to provide a user with a proper answer. data in databases typically have a structured form, and one of the main challenges is to present extracted information in the natural sentences. the rdf-to-text track of the webnlg challenge aims to address that particular problem. given the data presented in a common rdf format, the task is to generate a natural utterance that conveys information from the structured data. the rdf data format operates with three entities -subject, object and predicate, where the latter represents a type of relations between subject and object. such entities form a triple, and each data sample is represented as a tripleset consisting of one or several triples ( it was mentioned by @xcite , such tripleset representation in english can be seen as a noisy version of a target utterance, and denoising autoencoders like t5 @xcite and @xcite ) may provide a stable solution to the problem. in russian track of the challenge we additionally expect generated utterances to be in russian while the triples remain in english. that can be handled either by a two stage solution where translation (of input triples or generated utterances) performed as a separate step, or by an end-to-end solution -for the such case, encoder-decoder architectures are suitable as well @xcite . the above assumptions are supported by the results of the previous challenge @xcite , where 3 out of 6 solutions for the rdf-to-text russian track were based on a pretrained multilingual bart model @xcite @xcite , and the winning solution was based on a t5 model pretrained on a large bilingual (en, ru) corpus of structured data @xcite . in contrast to the previous solutions, we focus solely on the russian track and use the state-ofthe-art llm for russian language named fred-t5 @xcite , assuming that pretraning on a large corpus of russian texts can benefit the model's ability to generate more realistic utterances. we also experiment with prompt con- ## data the original dataset for rdf-to-text task is presented in the xml-file format. it is split into train, dev and test sets. rdf-triples were extracted from dbpedia and have the subject-predicate-object structure linearized with vertical bars as separators ( category names and triples are initially presented in english. however, links between english and russian entities from subjects and objects of rdf-triples are provided @xmath0. these links are encoded by two kinds of relations: "includes" and "sameas", the latter of which can be considered as an explicit translation. the model's input is the prompt, and we discuss the prompt construction in the next section. as an output, we expect to receive a generated text similar to the text from the label. however, each sample may contain multiple labels that represent a natural language text corresponding to triples ( ## results in this section we provide the results of our experiments. we finetuned fred-t5 model on the training dataset with different prompts. in order to understand what advantages fred-t5 has over multilingual llms, we finetuned mt5-large and mt5-xl models on the train dataset with "full translated" prompt. from at the same time, mt5-large is closer to fred-t5 in terms of model size, but converges with lower accuracy, especially on the dev dataset. considering the fact that from architecture perspective all models are based on t5, we may assume that fred-t5 benefits from its pretraining scheme and the target language corpora. in our submission we used fred-t5 model finetuned on the train dataset with "full translated" prompt. ## conclusion we presented a solution for the rdf-to-text problem in russian. our solution is based on fred-t5 large language model, utilizes additional information from raw xml data and is aided by machine translation. the developed solution achieves 0.373 ter on the webnlg-2023 test dataset and outperforms existing solutions by a large margin. furthermore, conducted experiments demonstrated that translated data is not crucial for the solution and provides only a small gain, while a proper pretraining plays a major role.