LOWRECORP : the Low-Resource NLG Corpus Building Challenge (2023)

Abstract: Most languages in the world do not have sufficient data available to develop neural-network-based natural language generation (NLG) systems. To alleviate this resource scarcity, we propose a novel challenge for the NLG community: low-resource language corpus development (LOWRECORP). We present an innovative framework to collect a single dataset with dual tasks to maximize the efficiency of data collection efforts and respect language consultant time. Specifically, we focus on a text-chat-based interface for two generation tasks – conversational response generation grounded in a source document and/or image and dialogue summarization (from the former task). The goal of this shared task is to collectively develop grounded datasets for local and low-resourced languages. To enable data collection, we make available web-based software that can be used to collect these grounded conversations and summaries. Submissions will be assessed for the size, complexity, and diversity of the corpora to ensure quality control of the datasets as well as any enhancements to the interface or novel approaches to grounding conversations.

## introduction around the world, people speak about 7000 different languages and nearly all of these have very weak support in language technologies. while about 100 languages are included in recent large language models (e.g. @xcite , most languages do not have good resources. the situation is especially dire when we examine task-specific datasets, such as for response generation, summarisation, and other forms of natural language generation (nlg). to address this problem, we propose a new shared task on dataset creation for nlg: lowrecorp 1 challenge, which invites participants to collect a new dataset combining dialogue 1 pronounced @xmath0, @xmath1. grounded in documents and/or images and dialogue summarisation for a low-resource language (lrl) of their choice. although conversational response generation and summarization have distinct objectives, they both aim to generate coherent output based on a given context. drawing from this symbiosis, a new task is proposed that combines the challenges of both tasks into a single framework. the first task is grounded response generation and the second is dialogue summarization. grounded response generation is the task of generating a conversational response grounded in a context such as documents, images, or other modality, to provide more contextually relevant information @xcite @xcite . similarly, dialogue summarization aims to generate a summary of a conversation while preserving its main ideas, and it is particularly useful in scenarios such as meeting notes and doctor-patient conversations where there may be a lot of irrelevant exchange that can obscure informative segments, making the task challenging for traditional approaches @xcite . to facilitate this challenge, we developed a new interface for slurk @xcite which allows paired participants to view different grounding materials such as text, vision, audio, etc., relating to the same topics to engage in a conversation and dialogue summarization task (figures 1 and 2 ). combining these tasks enables the efficient use of participant time and energy, which is especially important when working with lrls which often have small communities of use, and will serve broader research efforts in linguistic analysis, language documentation, and preservation efforts (anastasopoulos, 2019). in contrast to mitigation strategies such as data augmentation and multilingual language models @xcite , our dual-task design focuses on efficiency during corpus creation. to our knowledge, this is the first work combining both grounded response generation and dialogue summarization to maximize benefits for lrls data collection. ## task description engaging human participants to build or annotate corpora typically takes one of two forms. in the first, bulk annotations 2 for a single task are collected via crowdsourcing, but this is challenging to replicate in lrl settings as recruiting, training, and maintaining qualified annotators is more difficult. in the second, annotation tasks are built incrementally, which requires recruiting multiple pools of participants or having the same participants return for multiple tasks -for example multiple sessions to collect annotations for retrieval, localization, and comparison (e.g. @xcite . in contrast, we propose a third approach that combines two tasks within the same experimental session to alleviate challenges in recruiting proficient speakers of low-resourced target languages. the two tasks are: task 1: grounded response generation given an image or document and the conversation history as context, respond to the previous utterance. task 2: (dialogue) summarization given the full grounded conversation, summarize the important points with the goal to maximize reconstructing the original grounding context. these 2 tasks of short and long form generation complement each other well, for instance, using 2 throughout, we will use 'annotations' equally to refer to annotations on or based upon existing texts as well as the creation of new texts for a corpus and other tasks performed by participants in the process of creating and annotating a corpus. ## implementation strategies the proposed dual-task design provides a framework for efficiently collecting complementary datasets. noting that many large language models today are trained on a substantial proportion of the data found online in any given language and that these models will likely be used as components in future systems trained on the datasets our participants create, we encourage participants to focus on sources of grounding material that are less likely to be in the training data for such models already (e.g. information about museum exhibits, children's stories, etc.). we invite participants to leverage this framework to gather datasets in indigenous and local languages grounded on topics of local interest, deciding exactly what data to collect and how to implementation strategy technological access/literacy data volume quality control online across network high high low in lab or field moderate moderate high offline in field can be low moderate high online across the network: in this approach, researchers use a webserver to deliver contextual materials along with an audio, video, or text chat interface. this allows researchers to potentially run multiple pairs of participants simultaneously and provides digital representations of the resulting dialogues and summaries from the start. however, this method relies on linguistic literacy to provide instructions (and potentially for data collection, if using a text-based chat interface) and on technological literacy, since participants must be familiar with using a computer, tablet, or smartphone and feel comfortable interacting with the software. computer-based in the lab or field: with this approach, researchers are no longer limited to participants with access to technology at home and can be on-hand to answer questions from participants about the interface or troubleshoot any problems.researchers can use the same kind of software that they would normally deploy online, hosted on a single laptop. in cases where the aim is to collect spoken dialogues, this also allows the researcher to control the quality of the audio equipment instead of relying on participants to use their own hardware. this method also ensures that the data is immediately available in a digital format and may help address management and/or privacy concerns since data will not need to be transmitted over the internet. this approach requires access to a lab or another controlled space and incurs travel costs either for the participants (to come to a lab of the researcher's choosing) or the researchers (to meet the participants where they are). offline in the field: developing corpora for lrls can be limited by factors such as participant literacy @xcite , lack of availability of technology, systemic societal issues @xcite , and unrepresentative user bases for crowdsourcing. therefore, it is possible to implement this dialogue+summarization task fully offline, using in-person methods. researchers can provide contextual materials (keyword explanations and source materials) to participants on printed sheets of paper and use a microphone to record conversa-tions and summaries. this method may be most helpful when working with participants with very low technological literacy or in communities where electricity or connectivity is limited. in order for participants to use the source materials to answer questions, however, they will need to be literate or to have materials provided to them in a visual format. this approach will also generally require transcription in addition to the kind of normalization and data cleaning required by the other approaches. recruitment considerations recruiting participants for lrls for co-designing, corpus collection, and system evaluation can be challenging due to the small number of speakers. for example, scottish gaelic (gàidhlig) has about 57, 000 speakers living in scotland, 3 implying the difficulty even for a wealthy country with good internet access. in such cases, it can be helpful to collaborate with researchers and institutions that are already involved with the target community. these contacts provide access to community leaders and information exchange structures like community centers and newspapers to promote experiments to potential participants. being able to reach audiences using spoken or signed media may be especially crucial for lrls with lower literacy rates @xcite . ## submissions and assessment submission details more details about the task and the interface are available at lowrecorp. github.io. researchers interested in participating in the challenge can contact us at lowrecorp@googlegroups.com, where participating teams can interact and receive updates and support from the organizers. researchers will upload their data in may 2024 to allow sufficient time for reporting at inlg 2024. each team is expected to submit a paper to a special session that discusses their chosen subject, target language, any innovations in their approach, and key corpus statistics along with a data card @xcite . assessment a strict evaluation of the diverse submissions expected for this challenge would limit the creativity and flexibility of the teams. hence, we intend to focus on open evaluations aimed primarily at quality control and description of the data, adapting the data-to-text corpus description guidelines of perez-beltrachini and gardent (2017) to our dialogue and summarisation tasks.for exam- we will recognize and celebrate submissions based on a variety of dimensions, such as corpus size, lexical diversity, language rarity, most creative grounding source, etc. (cf. ## conclusion this challenge aims to bring together researchers interested in corpus building for lrls to work on a shared, streamlined, and vetted protocol (tested in scottish gaelic) to build interesting corpora. we hope that our challenge will contribute toward recent efforts in addressing geographically diverse nlp @xcite @xcite by aiding in the creation of new datasets for nlg tasks from a wider variety of languages. we aim to encourage cross-pollination of ideas and ideally set up for future generation challenges in a variety of low-resourced languages which make use of the data collected. the resulting (future) task will serve as a text-and-imagegrounded complementary task to efforts like the semantic-web-grounded webnlg challenge foot_1 and the syntactically-grounded multilingual surface realization challenge @xcite .