Learning From Failure: Data Capture in an A ustralian Aboriginal Community (2022)

Abstract: Most low resource language technology development is premised on the need to collect data for training statistical models. When we follow the typical process of recording and transcribing text for small Indigenous languages, we hit up against the so-called “transcription bottleneck.” Therefore it is worth exploring new ways of engaging with speakers which generate data while avoiding the transcription bottleneck. We have deployed a prototype app for speakers to use for confirming system guesses in an approach to transcription based on word spotting. However, in the process of testing the app we encountered many new problems for engagement with speakers. This paper presents a close-up study of the process of deploying data capture technology on the ground in an Australian Aboriginal community. We reflect on our interactions with participants and draw lessons that apply to anyone seeking to develop methods for language data collection in an Indigenous community.

## introduction for decades, the work of collecting data for indigenous languages has been the province of documentary and descriptive linguistics @xcite @xcite . this work has involved various kinds of elicitation, e.g. of word lists, phrases, etc, to support description of the phonology, morphosyntax, and grammar of the language. it has also involved the collection of unrestricted text, through recording and transcription. in most cases, the result is audio with aligned text. many software tools have been developed for supporting these activities @xcite @xcite . within the field of natural language processing, established practice is to support the linguist's work @xcite @xcite . in some cases, this includes the participation of speakers in activities using apps controlled by linguists @xcite @xcite . however, the premise is basically the same: obtain a substantial quantity of audio and transcribe it, or post-edit the output of an automatic transcription system. we believe that these approaches do not adequately address a fundamental reality of small languages: they are oral. there may be an official orthography, but it has no place in the local language ecology where any written business takes place in a language of wider communication. as a result, local people are usually not confident in the orthography of the language. furthermore, there may be low confidence in using computers and text editors, and inadequate support for the language in terms of keyboarding and spelling correction. add to all this the fact that the whole space of rendering an oral language into standardised orthography can be alienating @xcite . there is no particular reason for nlp approaches to indigenous languages to follow the long-established practices of linguists. after all, there is an equally long history of algorithmic approaches being profoundly different to the human tasks they replicate. for instance, a human sorting a hand of cards may use insertion sort, but a machine might use quicksort, with better average-case complexity @xcite . computational approaches may be inspired by analogy, e.g. simulated annealing, genetic algorithms, neural networks, but they are not required to adhere to the human defined process. accordingly, we can ask, what is an idiomatic computational approach to collecting data for indigenous languages that is a better fit to the capabilities of human participants? in the case of associating text and speech, we believe that the answer might be keyword spotting. this is because, in our experience, speakers and learners are attuned to identifying whole words, rather than obsessing about the idiosyncratic phonetic makeup of individual tokens as required for phone transcription (cf. @xcite . accordingly, we investigate an approach to transcription based on word spotting known as "sparse transcription" @xcite . this would seem to be an easier, less specialised task than direct, contiguous transcription. if more people can participate, we can hope to establish a virtuous circle with more data, better models, less correction, even more data, and so on. the idea is that transcription can be accelerated by identifying the tokens of high-frequency terms all at once, then playing them back in quick succession for confirmation by participants. this paper reports on the deployment of a lexical confirmation app which supports human confirmation of system hypotheses. we begin by describing the background to this work (sec. 2), including related work on designing technology for use in indigenous places. we also describe the site where we work and the design of the lexical verification app. next, we report what happened when we deployed the app in two field tests, including detailed accounts of interactions with participants (sec. 3). in the discussion section, we reflect on the field experience from a variety of perspectives, trying to draw out lessons that may be applicable to other places where nlp researchers seek to design technologies for language data collection (sec. 4). the paper concludes with a summary and prospects for further research. ## fieldwork we tested our approach with two trials in two aboriginal towns, with three people in each place. while the number of participants seems small, larger trials are difficult to arrange in aboriginal contexts due to the small number of speakers. at the beginning of each elicitation session, the first author explained our intention to teach a machine to transcribe the language automatically, and that we wanted help to correct system guesses. there is actually no direct translation of transcription in kunwok and the concept is usually given by the formulation karribimbun kure djurra, "we're drawing on paper". in both places, we recruited the participants with the support of two local institutions, the art centre in gunbalanya and the ranger organisation in manmoyi. at the start of our trips, the first author introduced himself to the communities and explained that he was looking for people to support him for language work. then the people interested came to find him throughout the day. each session lasted approximately 15 minutes and was part of other language work including recordings or language learning. each participant was paid at the regular rate for language work. ## discussion there were many issues in the design and conduct of this elicitation activity, and it is clear that our approach need to be completely rethought. in this section, we analyse the above interactions and try to identify some principles to inform nlp elicitation methodology, hoping to avoid such problems occurring in future. task motivation. sb, rb and lb understood us to be interested in interpreting the content. sb thought we were testing his memory. tm offered detailed explanations. lb said things that we interpret as asserting authority. it appears that our attempt to explain our purpose in automatic transcription, and the activity of confirming or refuting system guesses, was unsuccessful. task definition. participants were not clear about what we were asking of them. the notion of "word" was not clearly defined, and there were a variety of responses when the query term was not identical yet morphologically or semantically similar to a word in the corresponding utterance. naturalness of the task. when it comes to collaboration with western language workers, aboriginal people in these communities are accustomed to participating in interviews, recordings, transcription, and translation activities. this may explain people's readiness to respeak or interpret the content or supply additional cultural information. we entered with a different task, one where the overt activity of human confirmation/rejection of system guesses was not transparently related to a recognisable transcription task. we explained and demonstrated the activity, but tm was the only participant to instantly grasp this task. even so, he provided extensive explanations when the system guess was wrong in an effort to teach us. utterance context. from our perspective, the components of the device were clear. we have a query term that needs to be detected, and an utterance that should contain the query term. from this, we just need two feedback buttons to confirm whether the query term is included in the utterance. however, to the participant listening to the audio produced by the app and not following our use of the controls, the query term and utterance may be perceived as a single utterance. everything put into the aural space appears to be concatenated by listeners, and our non-conventional metalinguistic context is not interpretable. when endeavouring to explain the task in kunwok, we were hampered by the lack of words for "word" and "sentence". teaching. the participants generally provided much more information than the simple yes/no response we requested. each instance was another opportunity to teach us about the language or the country. the design of the task only limited the space for this style of participation. the activity itself was not particularly engaging, taking utterances out of context and asking for a mechanical response to a seemingly pointless question. it seems to be a kind of resilience that participants made the most of the opportunity to pursue their own ends of educating newcomers. further discussion with community members highlighted their concerns about knowledge preservation, access to archival recordings, and learning literacy. knowledge transmission. @xcite explains that the way in which westerners and australian aboriginal people transmit their knowledge varies in that one extracts, identifies and, categorizes while the other needs the information to be embedded in a system of kinship relationships. for example, in bininj country, every individual has a kinship relationship to every other individual, and they address each other accordingly @xcite . stories do not exist in isolation but are connected to an individual who tells them, and the country it comes from. we ran up against this when participants needed to connect isolated utterances back to their rightful cultural context, not just consider them as arbitrary linguistic material for which they can answer an unmotivated question: "does this utterance contain this word?" we can see this in trial 2 where lb ignores the utterance and uses his knowledge of the speaker of the query term to link the content back to the story. ## further reflection our first attempt in this space was unsuccessful on many levels. most superficially were issues with the task definition and the app interface. the task focused on the notion of "word" and on deciding whether a given word occurred in a given utterance. yet the notion of word was not established; as an oral language, there was no a priori shared understanding between the participant's notion of spoken word and our notion of orthographic word. throughout our interactions with participants, our attempts to explain the method and the purpose were unsuccessful. local perception was fixed on the idea that we had entered the community to learn the language and culture, and that the purpose of participating in the study was to teach us and to interpret the texts for us. consequently, the narrow focus of our activity on eliciting a binary, thumbs up/down response was unsuccessful. this is hardly surprising as many people have noted that engaging aboriginal people with direct questions requiring a yes or no response is seen as testing people's knowledge or memory, and potentially irritating @xcite . we observed this ourselves, when sb reported that he felt like he was being tested, or when lb responded as if his authority was being questioned. clearly, our style of engagement was not the expected kind of collaboration on a linguistic task. aside from one participant (tm), no one would participate in the abstract and apparently pointless task of confirming whether a word was present in a sentence. instead, all participants sought to create meaning from any language fragments they were presented with. on the basis of an isolated word, and person, place or story would be detected, and people would seek to teach us about these aspects of their lifeworld. this took various forms: repeating, paraphrasing, translating, interpreting, or offering extensive cultural commentaries. in retrospect, this response to our approach comes across as resilient and generous. in comparison, our narrow focus on data collection, and on getting across the specialised task of lexical confirmation may have come across as disconnected from local interests, and potentially disrespectful. of course, we can hope to recruit more people like tm. however, the story about scalable creation of language resources involves working with whoever is available. the tasks need to be locally comprehensible and motivating. in moving forward, we believe it is necessary to rethink the collaborative transcription task. the starting point is to understand local participants as teachers and cultural guides, occupied with their own knowledge practices and with passing these on. special focus need to be given on the creation of a third space between the several stakeholders of a project with benefits that serve both indigenous participants and external actors @xcite . could we view the task of putting an audio recording into textual form as a way to help a newcomer make progress with the language and culture, and with getting the pronunciations and meanings correct? the answer to this question depends on further research. ## conclusion outside the major languages, the development of language technologies is considered to be held up by the general lack of data @xcite . in the case of the world's small, oral languages, the usual approach has been to follow the long-established practice of linguists and record and transcribe audio and elicit wordlists and paradigms. many computational tools were developed to support this approach. however, algorithmic approaches to working with small languages do not need to be limited by these past practices, and so we believe it is worth considering other approaches to data collection that might simultaneously support computational methods while engaging effectively with members of the speech community. accordingly, we took a recently proposed approach to transcription based on keyword spotting, and developed an app for confirming system guesses. we anticipated that this app would be more accessible to local participants than the conventional linguist-driven tasks. we ran trials in two aboriginal towns, with speakers of the kunwok language. in this paper, we report the description of the several interactions we had with locals around a lexical verification activity. we present the many challenges we encountered, including a reflection around the technical and cultural issues of the task design, and the flaws around our approach in terms of collaborative language work. for the present, we offer our findings as a candid report on the experience of deploying data capture technology in an indigenous community, in the hope that others will succeed where we have failed. we hope others will also follow our lead and share their own experiences of data collection, and make visible more of the real work of @xcite . perhaps it is possible for an externallydefined task such as transcription to be aligned to local agendas. just as often, we expect that it will be necessary to let go of such tasks and do something different. something that makes sense locally.