Unsung Challenges of Building and Deploying Language Technologies for Low Resource Language Communities (2019)

Abstract: In this paper, we examine and analyze the challenges associated with developing and introducing language technologies to low-resource language communities. While doing so we bring to light the successes and failures of past work in this area, challenges being faced in doing so, and what have they achieved. Throughout this paper, we take a problem-facing approach and describe essential factors which the success of such technologies hinges upon. We present the various aspects in a manner which clarify and lay out the different tasks involved, which can aid organizations looking to make an impact in this area. We take the example of Gondi, an extremely-low resource Indian language, to reinforce and complement our discussion.

## introduction technology pervades all aspects of society and continues to change the way people access and share information, learn and educate, as well as provide and access services. language is the main medium through which such transformational technology can be integrated into the socioeconomic processes of a community. natural language processing (nlp) and speech systems, therefore, break down barriers and enable users and whole communities with easy access to information and services. however, the current trend in building language technology is designed to work on languages with very high resources in terms of data and infrastructure. also, as machine learning (ml) and nlp practitioners, we get caught up in an informationtheoretic view of the problem, e.g., focusing on incremental improvements of performance on benchmarks or capturing accurate distributions over data, and tend to forget that the raison d'łtre of nlp is to build systems that add value to its users @xcite . we want to build models that enable people to read the news that was not written in their language, ask questions about their health when they do not have access to a doctor, etc. and while these technology applications are more and more ubiquitous for languages with a lot of data, a larger majority of languages remain resource-poor and bereft of such systems. as discussed in the united nations e-government survey (nations, 2014), one of the most important obstacles to e-inclusion, particularly among vulnerable groups with little education, is language. thus, by excluding these languages from reaping the benefits of the advancements in language technology, we marginalize the already vulnerable groups even further. india is a highly multilingual society and home to some of the largest language communities in the world. 6 out of 20 most-spoken (native) languages in the world are indic. ethnologue @xcite records 461 tongues in india out of 6912 worldwide (6%), the 4th largest belonging to any single country in the world. 122 of these languages are spoken by more than 10,000 people. 29 languages have more than 1 million speakers, which include indigenous tribal languages like gondi and mundari, some without a supported writing system or script. despite the large numbers of users, most of these languages have very little data available. figure 1 shows that as compared to some of the much lesser spoken languages like german, indic languages are severely low resourced. in a vast country like india, access to information thus becomes a huge concern. this lack of information means that not only do these communities not have information in domains like agriculture, health, weather etc., which could improve their quality of lives, but they may also not be aware of their basic rights as citizens of the country. in this paper, we take the position that the current direction of advanced language technology towards extremely high data requirements can have severe socio-economic implications for a majority of language communities in the world. we focus on specific aspects of designing and building systems and applications for low resource languages and their speech communities to exemplify viable social impact through language technology. we begin by discussing the aspect of information exchange, which is the core motivation behind enabling low-resource language communities. we then steer our analysis towards the design and creation of an interface for people in these communities to simplify and enrich the process of information exchange. finally, we gather insights about how to deploy these technologies to ensure extensive impact by studying and taking inspiration from existing technological deployments. we use gondi, a south-central dravidian language in the vulnerable category on unesco's atlas of the worlds languages in danger (moseley, 2010), as an example wherever possible. spoken by nearly 3 million people (india, 2011) in the indian states of chhattisgarh, andhra, odisha, maharashtra and karnataka, it is heavily influenced by the dominant state language. however, it is also one of the least resourced languages in india, with very little available data and technology. we believe that the components discussed in the sections below encapsulate the spectrum of issues surrounding this field and that all future discussions in this area will also fall under the umbrella of these categories. we believe that by focusing on gondi, we will not only empower the gondi community but more importantly, understand and create a pipeline or framework which can serve as a clear guide for potential ventures which plan on introducing disruptive language technologies in under-served communities. ## information exchange the primary element in communication is information exchange. people living in less connected areas are often unable to get the kind of information they need, due to various socio-economical and technological barriers. as a result, they miss out on crucial knowledge required to improve their well-being. there are three co-dependent aspects woven into the fabric of information exchangeaccess of information, quality and coverage of the information and methods to create and digitize available knowledge (generation). ## interface the design of a user-friendly interface plays a very crucial role in ensuring that the deployed technology encompasses all strata of society. it is often seen that a majority of target users have not had the privilege of education, and show varying levels of literacy, both foundational and digital. in such scenarios, text-based modalities pose several limitations from both the user and designer perspectives, and graphical user interfaces have been the preferred choice in these applications. @xcite reports that text-based interfaces were completely redundant for illiterate users and severely error-prone for literate but novice users. further, several languages do not have unique key-board standards or fonts, and some do not have a script at all @xcite . to overcome these issues with text, speech as a modality has also been deployed with varying success. 'cgnet swara', a citizen-run journalism portal, uses a phone-based ivr system to educate illiterate users @xcite . avaaj utalo allows users to make simple phone calls to ask questions or browse questions and answers asked on agricultural topics @xcite . spoken web is another application wherein users can create voice sites analogous to websites which can then be easily accessed through voice interaction on mobile phones @xcite . these serve to provide farmers with relevant crop and market information. an attempt to leverage the complementarity of voice and graphic-based inputs was made by videokheti, a mobile system with a multi-modal interface for low-literate farmers providing agricultural extension videos on command in their own language or dialect @xcite . they report that people in these communities find it difficult to use softkey type keyboards that are extremely common on modern smartphones. instead, they proposed a system comprising of large buttons, graphics and some voice input. such a system for delivering information to farmers was made and they showed that the farmers were very comfortable using it. their results also show that a speech interface alone was not enough for that scenario, except in cases where the search list was long and the results were dependent on keywords or short phrases. similarly, the adivasi radio app foot_3 , based on text-to-speech (tts) technology, is developed to read out written reports in gondi, one of the main tribal languages in chhattisgarh. bolo is another mobile application which uses a very simple interface to improve children's literacy in india. project karya also proposes to divide massive digital tasks into microwork and crowdsource this work to millions of people in rural india via phones foot_4 . while voice might solve the foundational literacy problems, the lack of digital literacy is often more challenging to overcome. @xcite demonstrate the use of an app to teach the mundari language to children. the app comprised of a series of games designed with the help of the community. the content was delivered in the bangla script, which was what the children were taught in school. their study noted that children from such communities found the usage of a smartphone to be difficult. relying on voice-based systems also poses a few challenges. it is not easy to build robust asr systems for these languages due to severe lack of data, dialect variations and several such constraints. an attempt to resolve this was made with the development of the salaam asr @xcite which uses the acoustic model of an existing asr and performs a cross-lingual phoneme mapping between the source and target language. this, however, is limited to recognition of a very small set of vocabulary, but finds use due to its' cost-effective and low resource setting. ## deployment and impact after developing technologies to provide information, and ensuring that the applications are designed in such a way that they are accessible to the population, the technology must be effectively deployed. specialized applications are useless if they are not deployed properly in a way that accesses their end-users. when deploying a specially developed technology, the application must be deployed with consideration of the existing community dynamics. for any deployment to be successful, it usually must be able to be purposefully integrated into the lifestyle of community members -or have strong utilization incentives if it is a transformative technology. in this section, we will review examples of technology dissemination to low-resource/rural communities, and the impacts of effective deployment. while some technologies that we examine are not deployed utilizing low-resource languages specifically, the types of rural communities and villages in which they are deployed are analogous to the contexts in which low-resource languages exist, and clear parallels can be drawn. integrating the usage of a language technology intervention into a community in a low-resource context requires much more simply introducing the technology. unlike hardware interventions and innovations like solar panels or new agricultural tools, language technologies often rely on the delivery, exchange, and utilization of information, which is much less tangible than physical solutions. this is especially for people with limited previous exposure to digital technology. upon observing a selection of language-based interventions that were deployed in low-resource contexts, we observed that the most successful deployments of technologies tended to have three components of success. they: 1) initially launched by seeding with target communities, 2) worked closely to engage the community itself with the technology and information, and 3) provided a strong incentive structure to adapt the technology -this incentive could be as simple as payments or as complex as communicated benefits from the technology. ## conclusion the boost in recent advancements in nlp research has started breaking down communication and information barriers. this, coupled with in-depth studies on the socio-economic benefits of enabling less-connected communities with technology, provides a strong argument for increasing investment in this area. it is promising to observe increased innovation and steady progress in the empowerment of rural communities using language tools. increased exposure to the challenges and works in this area can catalyse developments in improving inclusion and information dissemination. we hope that this paper will provide pointers in the right direction for potential ventures that plan on introducing disruptive language technologies to marginalized communities.