T a TA : A Multilingual Table-to-Text Dataset for A frican Languages (2023)

Abstract: Existing data-to-text generation datasets are mostly limited to English. To address this lack of data, we create Table-to-Text in African languages (TaTA), the first large multilingual table-to-text dataset with a focus on African languages. We created TaTA by transcribing figures and accompanying text in bilingual reports by the Demographic and Health Surveys Program, followed by professional translation to make the dataset fully parallel. TaTA includes 8,700 examples in nine languages including four African languages (Hausa, Igbo, Swahili, and Yorùbá) and a zero-shot test language (Russian). We additionally release screenshots of the original figures for future research on multilingual multi-modal approaches. Through an in-depth human evaluation, we show that TaTA is challenging for current models and that less than half the outputs from an mT5-XXL-based model are understandable and attributable to the source data. Our results highlight a) the need for validating metrics; and b) the importance of domain-specific metrics.

## introduction generating text based on structured data is a classic natural language generation (nlg) problem that still poses significant challenges to current models. despite the recent increase in work focusing on creating multilingual and cross-lingual resources for nlp @xcite @xcite , data-to-text datasets are mostly limited to english and a small number of other languages. data-to-text generation presents important opportunities in multilingual settings, e.g., the expansion of widely used knowledge sources, such as wikipedia to under-represented languages @xcite . data-to-text tasks are also an effective testbed to assess reasoning capabilities of models @xcite . however, creating challenging, high-quality datasets for nlg is difficult. datasets frequently suffer from outputs that are not attributable to the inputs or are unnatural, and overly simple tasks fail to identify model limitations @xcite @xcite . to provide a high-quality dataset for multilingual datato-text generation, we introduce table -to-text in african languages (tata). tata contains multiple references for each example, which require selecting important content, reasoning over multiple cells, and realizing it in the respective language (see fig. 1 ). the dataset is parallel and covers nine languages, eight of which are spoken in africa: arabic, english, french, hausa, igbo, portuguese, swahili, yorùbá, and russian. 2 we create tata by transcribing 3 and translating charts and their descriptions in informational reports by the demographic and health surveys (dhs) program, which publishes population, health, and nutrition data through more than 400 surveys in over 90 countries in pdf format. in an analysis of tata using professional annotators, we find that over 75% of collected sentences require reasoning over and comparing multiple cells, which makes the dataset challenging for current models. even our best baseline model generates attributable language less than half of the time, i.e., over half of model outputs are not faithful to the source. moreover, we demonstrate that popular automatic metrics achieve very low correlations with human judgments and are thus unreliable. to mitigate this issue, we train our own metrics on human annotations, which we call stata, and we use them to investigate the cross-lingual transfer properties of monolingually trained models. this setup identifies swahili as the best transfer language whereas traditional metrics would have falsely indicated other languages. overall, our experiments highlight that a) metrics need to always be validated in how they are used; and b) that domain-specific metrics may be necessary. consequently, as our field increasingly relies on trained metrics, e.g., reward models for rlhf @xcite , we must ensure that metrics are appropriate for the chosen task. dealing with noisy data while creating more challenging datasets is necessary to keep up with modeling advances, their creation process can introduce noise. noise in simpler datasets can often be detected and filtered out through regular expressions @xcite , as done by @xcite for the e2e dataset @xcite . however, the larger output space in complex datasets requires more involved approaches and researchers have thus devised strategies to ensure that references are of sufficient quality. for example, totto @xcite used an annotation scheme in which annotators were asked to remove non-attributed information from crawled text. synthbio @xcite followed a similar strategy but started with text generated by a large language model @xcite . the downside of involving crowdworkers in the language generation steps is that outputs can be unnaturally phrased compared to naturally occurring descriptions; studies on translationese in machine translation @xcite highlight potential negative effects on the final model and its evaluation @xcite . our approach aims to mitigate these issues by transcribing naturally occurring descriptions. multilingual generation at present, there exist only few data-to-text datasets that cover languages beyond english @xcite @xcite . none of these datasets offers parallel data in two or more languages. 4 in contrast, tata supports eight fully parallel languages focusing on african languages, and russian as an additional zero-shot language. each source info-graphic covers at least two of these languages; we provide data for the remaining languages using professional translations, leading to a fully parallel corpus while minimizing the drawbacks of only relying on translated texts. existing datasets available in african languages mainly focus on classic nlp applications such as machine translation @xcite and summarization @xcite . among others. tata enables data-to-text generation as a new task for these languages and-due to its parallel nature-also supports the development of mt models that generalize to new domains and multi-modal settings @xcite . in addition, due to the focus on surveys in african countries, the topics and entities in tata are distinctly africacentric. this is in contrast to other african language datasets where western-centric entities are over-represented @xcite . ## conclusion in this paper, we introduce tata, a table-to-text dataset covering nine different languages with a focus on african languages and languages spoken in africa. tata is the first multilingual table-to-text dataset among many existing english-only datasets and it is also fully parallel, enabling research into cross-lingual transfer. we experiment with differ-ent monolingual and filtered and augmented multilingual training strategies for various models. our extensive automatic and human evaluation identifies multiple avenues for future improvements in terms of understandability, attribution, and faithfulness of neural generation models and their metrics. we develop the metric stata based on additional data collected on the development set and demonstrate that it has a much better agreement with human ratings than existing metrics, which we consider unreliable and which we show lead to misleading results when analyzing transfer between languages.