XF 2 T : Cross-lingual Fact-to-Text Generation for Low-Resource Languages (2023)

Abstract: Multiple business scenarios require an automated generation of descriptive human-readable text from structured input data. This has resulted into substantial work on fact-to-text generation systems recently. Unfortunately, previous work on fact-to-text (F2T) generation has focused primarily on English mainly due to the high availability of relevant datasets. Only recently, the problem of cross-lingual fact-to-text (XF2T) was proposed for generation across multiple languages alongwith a dataset, XAlign for eight languages. However, there has been no rigorous work on the actual XF2T generation problem. We extend XAlign dataset with annotated data for four more languages: Punjabi, Malayalam, Assamese and Oriya. We conduct an extensive study using popular Transformer-based text generation models on our extended multi-lingual dataset, which we call XAlignV2. Further, we investigate the performance of different text generation strategies: multiple variations of pretraining, fact-aware embeddings and structure-aware input encoding. Our extensive experiments show that a multi-lingual mT5 model which uses fact-aware embeddings with structure-aware input encoding leads to best results (30.90 BLEU, 55.12 METEOR and 59.17 chrF++) across the twelve languages. We make our code, dataset and model publicly available, and hope that this will help advance further research in this critical area.

## introduction fact-to-text (f2t) is a natural language generation (nlg) task where input is structured data (like facts 2 ) and output is its natural language description. f2t systems have been shown to be effective in many applications like automated dialog figure 1 : xf2t example from xalignv2: generating english, hindi, bengali, gujarati, tamil and punjabi sentences to capture semantics from english facts. across many business domains, there is abundance of facts (or key-value stores) in english, and consumers want to access that information in their own regional languages. for example, users want product descriptions, weather report, match report, financial report in various lrls. another related problem is to automatically populate first sentence for lrl wikipedia pages using facts from english wikidata. if such facts were in lrls and there were models to do f2t in those lrls, we could leverage those. however, neither exist. even lrl facts on wikidata are very sparse. another approach could be to do f2t in english and then translate the output to lrls. but our experiments show that this leads to poor quality primarily due to lack of robust translation systems for lrls. specifically, we focus on the f2t problem of in @xcite , we proposed transfer learning and distance supervision based methods for cross-lingual alignment for aligning english wikidata facts with equivalent text from lrl wikipedia pages. in that paper, we used such alignment methods to contribute the xalign dataset which consists of sentences from lr language wikipedia aligned with english fact triples from wikidata. it contains data for the following eight languages: hindi (hi), telugu (te), bengali (bn), gujarati (gu), marathi (mr), kannada (kn), tamil (ta) and english (en). in that paper, we focused on dataset creation and not much on the xf2t task. in this paper, we extend this dataset to four more lr languages: punjabi (pa), malayalam (ml), assamese (as) and oriya (or). fig. 1 shows an xf2t example from our extended dataset, xalignv2. further, we rigorously investigate models for the xf2t problem. first, we experiment with standard existing transformer-based multi-lingual encoderdecoder models like the vanilla transformer, in-dicbart and mt5. next, we explore performance across various training setups: bi-lingual, translateoutput, translate-input and multi-lingual. further, we systematically explore various strategies for improving xf2t generation like multi-lingual datato-text pre-training, fact-aware embeddings, and structure-aware encoding. overall, we make the following contributions in this work. ## related work multi-lingual and cross-lingual nlg: recently there has been a lot of work on multi-lingual and cross-lingual nlg tasks like machine translation @xcite , question generation @xcite , news title generation @xcite , and summarization @xcite thanks to models like xnlg @xcite , mbart @xcite , mt5 (xue et al., 2021), etc. in this work, we investigate effectiveness of multiple modeling techniques for the xf2t task. further, from a knowledge graph (kg) and text linking perspective, our work is related to tasks like entity linking (link mention in a sentence to a kg entity) @xcite and fact linking (linking sentence to a set of facts) @xcite . as against this, xf2t is the problem of generating a sentence given a set of facts. xf2t is also related to graph-to-text @xcite where our fact triples about an entity can be mapped to a star-like graph, but no cross-lingual graph-to-text methods exist unfortunately. f2t datasets: several f2t datasets have been proposed in the literature: wikibio @xcite , e2e @xcite , webnlg 2017 @xcite , webnlg 2020 (ferreira et al., 2020), fr-de bio @xcite , kelm @xcite , wita @xcite , wikitablet @xcite , gen-wiki @xcite , trex @xcite and xalign @xcite . these datasets contain text from various domains like people, sports, restaurants, airports, politicians, artists, etc. also, these datasets vary widely in terms of statistics like the number of instances, number of facts per instance, number of unique relations and average number of tokens per instance. all of these are english only except fr-de bio (which has french and german), webnlg 2020 (which has english and russian) and xalign (which has english and 7 other lr languages). both fr-de bio and webnlg 2020 propose multi-lingual but not cross-lingual f2t tasks. unlike other datasets, xalign and our dataset, xalignv2 are cross-lingual. our proposed dataset, xalignv2, contains 12 languages, has 0.55m instances, 374 unique relations, avg 19.7 tokens/instance and avg 1.98 facts/instance. ## xf2t approaches in this section, we first discuss our input representation. next, we discuss various transformerbased methods, different training setups, multiple pretraining methods, and discussion on fact-aware embeddings. structure-aware input encoding: each input instance consists of multiple facts @xmath1, f 2 , . . . , f n } and a section title t. a fact f i is a tuple composed of subject s i , relation r i , object o i and m qualifiers @xmath2, q 2 , . . . , q m . each qualifier provides more information about the fact. each of the qualifiers {q j } m @xmath3. for example, consider the sentence: "narendra modi was the chief minister of gujarat from 7 october 2001 to 22 may 2014, preceded by keshubhai patel and succeeded by anandiben patel." this can be represented by a fact where subject is "narendra modi", relation is "position held", object is "chief minister of gujarat" and there are 4 qualifiers each with their qualifier relations as follows: (1) q 1 ="7 october 2001", qr 1 ="start time", (2) q 2 ="22 may 2014", qr 2 ="end time", (3) q 3 ="keshubhai patel", qr 3 ="replaces", and (4) q 4 ="anandiben patel", qr 4 ="replaced by". 0 0.1 0.2 0.3 0.4 0.5 0.6 1 2 3 4 5+ fraction of total sentences frequency of facts as bn en gu hi kn ml mr or pa ta te 0 0.1 0.2 0.3 0.4 0.5 1 2 3 4 5+ fraction of total sentences frequency of facts train validation test figure 3: fact count distribution across data subsets each fact f i is encoded as a string and the overall input consists of a concatenation of such strings across all facts in f . the string representation for a fact f i is "⟨s⟩s i ⟨r⟩r i ⟨o⟩o i ⟨r⟩qr i 1 ⟨o⟩q i 1 ⟨r⟩qr i 2 ⟨o⟩q i 2 . . . ⟨r⟩qr im ⟨o⟩q im " where ⟨s⟩, ⟨r⟩, ⟨o⟩ are special tokens. finally, the overall input with n facts is obtained as follows: "generate [language] f 1 f 2 . . . f n ⟨t ⟩[t]" where "[language]" is one of our 12 languages, ⟨t ⟩ is the section title delimiter token, and t is the section title. ## results and analysis since xf2t is a very recently proposed task, there are not many baseline methods to compare with. in this section, we will present results using methods described in section 4. due to lack of space, we show per language results only for our best model, standard transformer-based models: ## conclusion in this paper, we worked on the xf2t problem. we contributed the xalignv2 dataset which has instances with english facts aligned to 12 languages. we investigated several multi-lingual transformer methods with different training setups, pretraining setups and input representations. we obtained models with best metrics of 30.90 bleu, 55.12 me-teor and 59.17 chrf++ for xf2t. we make our code and dataset 1 publicly available to empower future research in this critical area.