Building Representative Corpora from Illiterate Communities: A Reviewof Challenges and Mitigation Strategies for Developing Countries (2021)

Abstract: Most well-established data collection methods currently adopted in NLP depend on the as- sumption of speaker literacy. Consequently, the collected corpora largely fail to represent swathes of the global population, which tend to be some of the most vulnerable and marginalised people in society, and often live in rural developing areas. Such underrepresented groups are thus not only ignored when making modeling and system design decisions, but also prevented from benefiting from development outcomes achieved through data-driven NLP. This paper aims to address the under-representation of illiterate communities in NLP corpora: we identify potential biases and ethical issues that might arise when collecting data from rural communities with high illiteracy rates in Low-Income Countries, and propose a set of practical mitigation strategies to help future work.

## introduction the exponentially increasing popularity of supervised machine learning (ml) in the past decade has made the availability of data crucial to the development of the natural language processing (nlp) field. as a result, much nlp research has focused on developing rigorous processes for collecting large corpora suitable for training ml systems. we observe, however, that many best practices for quality data collection make two implicit assumptions: that speakers have internet access and that they are literate (i.e. able to read and often write text effortlessly foot_0 ). such assumptions might be reasonable in the context of most high-income countries (hics) (unesco, 2018). however, in low-income countries (lics), and especially in sub-saharan africa (ssa), such assumptions may not hold, particularly in rural developing areas where the bulk of the population lives @xcite , figure 1 ). as a consequence, common data collection techniques -designed for use in hics -fail to capture data from a vast portion of the population when applied to lics. such techniques include, for example, crowdsourcing @xcite , scraping social media @xcite or other websites @xcite , collecting articles from local newspapers @xcite , or interviewing experts from international organizations @xcite . while these techniques are important to easily build large corpora, they implicitly rely on the abovementioned assumptions (i.e. internet access and literacy), and might result in demographic misrepresentation @xcite . in this paper, we make a first step towards addressing how to build representative corpora in lics from illiterate speakers. we believe that this is a currently unaddressed topic within nlp research. it aligns with previous work investigating sources of bias resulting from the under-representation of specific demographic groups in nlp corpora (such as women @xcite , youth @xcite , or ethnic minorities @xcite ). in this paper, we make the following contributions: (i) we introduce the challenges of collecting data from illiterate speakers in §2; figure 1 : literacy, urban population, and internet usage in african countries. note that countries with more rural populations tend to have less literacy and less internet users. these countries are likely to be under-represented in corpora generated using common data collection methods that assume literacy and internet access (grey: no data). in recent years, developing corpora that encompasses as many human languages as possible has been recognised as important in the nlp community. in this context, widely translated texts (such as the bible @xcite or the human rights declaration (king, 2015)) are often used as a source of data. however, these texts tend to be quite short and domain-specific. moreover, while the internet constitutes a powerful data collection tool which is more representative of real language use than the previously-mentioned texts, it excludes illiterate communities, as well as speakers which lack reliable internet access (as is often the case in rural developing settings, figure 1 ). given the obstacles to using these common language data collection methods in lic contexts, the nlp community can learn from methodologies adopted in other fields. researchers from fields such as sustainable development @xcite ), african studies @xcite , and ethnology @xcite , tend to rely heavily on qualitative data from oral interviews, transcribed verbatim. collecting such data in rural developing areas is considerably more difficult than in developed or urban contexts. in addition to high illiteracy levels, researchers face challenges such as seasonal roads and low population densities. to our knowledge, there are very few nlp works which explicitly focus on building corpora from rural and illiterate communities: of those works that exist, some present clear priming effect issues @xcite , while others focus on application @xcite . a detailed description of best practices for data collection remains a notable research gap. ## definitions and challenges guided by research in medicine (pannucci and wilkins, 2010), sociology @xcite , and psychology @xcite , nlp has experienced increasing interest in ethics and bias mitigation to minimise unintentional demographic misrepresentation and harm @xcite . while there are many stages where bias may enter the nlp pipeline @xcite , we focus on those pertinent to data collection from rural illiterate communities in lics, leaving the study of biases in model development for future work foot_1 . ## countermeasures drawing on existing literature and years of field experience collecting spoken data in lics, below we outline a number of practical data collection strategies to minimise previously-outlined challenges ( §3), enabling the collection of high-quality, minimally-biased data from illiterate speakers in lics suitable for use in nlp models. while these measures have primarily been applied in ssa, we have also successfully tested them in projects focusing on refugees in the middle east and rural communities in south asia. ## conclusion in this paper, we provided a first step towards defining best practices in data collection in rural and illiterate communities in low-income countries to create globally representative corpora. we proposed a comprehensive classification of sources of bias and unethical practices that might arise in the data collection process, and discussed practical steps to minimise their negative effects. we hope that this work will motivate nlp practitioners to include input from rural illiterate communities in their research, and facilitate smooth and respectful interaction with communities during data collection. importantly, despite the challenges that working in such contexts might bring, the effort to build substantial and high-quality corpora which represent this subset of the population can result in considerable sd outcomes.