Spoken Document Retrieval for an Unwritten Language: A Case Study on Gormati (2025)

Abstract: Speakers of unwritten languages have the potential to benefit from speech-based automatic information retrieval systems. This paper proposes a speech embedding technique that facilitates such a system that we can be used in a zero-shot manner on the target language. After conducting development experiments on several written Indic languages, we evaluate our method on a corpus of Gormati – an unwritten language – that was previously collected in partnership with an agrarian Banjara community in Maharashtra State, India, specifically for the purposes of information retrieval. Our system achieves a Top 5 retrieval rate of 87.9% on this data, giving the hope that it may be useable by unwritten language speakers worldwide.

## introduction introducing and integrating well-designed digital systems into communities, particularly those with low digital participation, such as oral communities, can enhance their exposure to digital technologies and could reduce inequalities arising from their limited digital use or presence @xcite . one application of advancements in language technology is in the application of speech-based search and information retrieval (ir). this task, commonly known as spoken document retrieval (sdr) has been investigated over several decades, most notably in darpa and iarpa programmes such as bolt, gale and babel @xcite @xcite . early work, e.g., @xcite took the form of simple keyword-spotting tasks (sometimes referred to as spoken term detection), but more sophisticated search capabilities have also been developed @xcite . in a standard setting, sdr operates over spoken documents (i.e., audio and video files contain-* equal contributions ing speech) but input queries remain text-based @xcite . however, in an alternative setting the input query may be in the form of speech as well. it is this latter formulation that is, of course, most relevant to unwritten languages. whilst sdr systems are typically developed for a specific target language, often using significant quantities of transcribed speech data for model training, this is not possible for an unwritten language. in this case, work to date has adopted a significant simplification of the ir task to that of query-by-example (qbe), essentially a form of keyword-spotting in which spoken documents are ranked based on the estimated occurrence of an arbitrary spoken input phrase. qbe systems have been developed in a zero-shot manner @xcite , meaning that no transcribed data from the target language is required. a simple approach is to perform pattern matching at the acoustic level, usually requiring a variant of dynamic time warping (dtw). however, the advent of unsupervised methods for neural network based acoustic word embedding raises the potential that such embeddings could be used for qbe, or even more sophisticated ir tasks for languages without a written form, or even for languages whose speakers would benefit from voice interfaces but where speech transcription tools are unreliable. in this paper, we leverage speech embeddings similar to @xcite and extend inference techniques from @xcite to support arbitrary-length queries and cases with known phone boundaries. we conduct a comprehensive set of development experiments in which we compare the technique to common competing methods -including both dtw and a discrete string search -on a qbe proxy task that we create for several indic languages. we go on to evaluate the method on a recently-collected corpus of gormati @xcite , an actual unwritten language. this data was collected specifically with ir in mind, and enables us to evaluate the performance of our method against metrics that are directly relevant for the community in question. the main contributions of this work are that, to our knowledge, this is the first successful approach on a real-world ir task for an unwritten language. as well as this, our method extends and enhances existing sdr speech embedding-based inference techniques to support arbitrary length queries and to incorporate predicted phone boundaries. ## awe model in this work, sdr is performed using a database of spoken documents and a set of spoken queries. we refer to our approach as the awe model. the general system pipeline is as follows: queries and documents are first passed through mhubert. these mhubert representations are then split into spans of mhubert vectors using one of two inference methods, which will be described in section 3.3. these spans are used to obtain awe representations of queries and documents via a trained learned pooling model. we perform retrieval by comparing the awe representation of each query with that of each document and ranking documents for each query based on their similarity. ## discussion the best results for each model on the mucs languages are shown in on gormati, the awe model performs the best with a top 5 score of 87.9%, exceeding the best score of 74% from @xcite . note that the dtw model cannot readily be applied to the more complex gormati document retrieval task. unlike @xcite , our model requires no target language training data and thus can operate on classes that have just one document. similarly, the success of our transfer learning approach shows that this method can be extended to other low-resource languages using only one hour of data from a related higher-resource language. for best performance, this data must be labelled. however, we showed that a good model can be trained using mpr labels. this shows that a successful model can be produced without any supervised data from the target language, a critical requirement for an unwritten language. since awes are derived from mhubert vectors, which are known to encode semantic information, it is likely that awes also carry some semantic content, though the extent of this remains uncertain. a method that captures more semantic content might produce better results given the nature of the gormati data. jacobs and kamper (2024) present such a method, but it requires knowledge of word boundaries, making it unsuitable for unwritten languages. in the future, adapting this or similar methods to unwritten languages could in a case where there is no data from related languages or resources are unavailable for training, then the mhubert or dtw models could be used. they perform worse than the awe model, but require no training and so can be applied directly to the target language. ## conclusion we have presented a successful unsupervised method for developing a purely speech-based ir system. however, there remain several avenues for future work. improving our inference method by experimenting with window lengths, strides and overlaps could be valuable. optimising model architectures could also lead to improvements, as might combining the awe model with the dtw model. our model development with the mucs data was geared towards the specific task of returning documents that directly contained a singleword query. this type of retrieval is insufficient when it is necessary to return semantically similar results to the query, or for multi-word queries that might benefit from partial matching. when applying speech technology to an unwritten local language, care must be taken to prioritise the specific needs of the community being served @xcite . our current work builds upon @xcite collaboration with the banjara community, which developed an ir system for recordings in the agricultural domain. on this same data, our awe model performs well based on the automatic top 5 metric, but could benefit from in-situ evaluation by community members. the awe model could also be used for retrieval in other domains that community members have expressed interest in, such as recipes and religion @xcite , allowing gormati speakers easier access to cultural information.