W av2 G loss: Generating Interlinear Glossed Text from Speech (2024)

Abstract: Thousands of the world’s languages are in danger of extinction—a tremendous threat to cultural identities and human language diversity. Interlinear Glossed Text (IGT) is a form of linguistic annotation that can support documentation and resource creation for these languages’ communities. IGT typically consists of (1) transcriptions, (2) morphological segmentation, (3) glosses, and (4) free translations to a majority language. We propose Wav2Gloss: a task in which these four annotation components are extracted automatically from speech, and introduce the first dataset to this end, Fieldwork: a corpus of speech with all these annotations, derived from the work of field linguists, covering 37 languages, with standard formatting, and train/dev/test splits. We provide various baselines to lay the groundwork for future research on IGT generation from speech, such as end-to-end versus cascaded, monolingual versus multilingual, and single-task versus multi-task approaches.

## introduction working against the overwhelming tide of social and historical forces, linguists and community activists from around the world have set out to record endangered languages while they are still actively spoken. as a first step, these documentary efforts involve-quite literally-recordings, then transcriptions, translations, and other annotations. the ultimate goal of such efforts is often to take a large volume of recorded speech and annotate it with interlinear glossed text (igt). igt is the lingua franca of documentary linguistics. most igt now follows a set of conventions called the leipzig glossing rules @xcite but other formats are in use. an example of igt from @xcite is shown in figure 1 . it consists of an unsegmented transcription (wd), underlying (ur) and surface forms (sr) segmented into morphemes, morpheme labels (glosses; gl), and free translation (tr) aligned with one another and the source audio recording. the lines labeled sr, ur, and gl are most important to linguists and language teachers. together, they tell the user how linguistic form maps into linguistic function, enabling the creation of many valuable resources and a variety of useful analyses. most linguistic field recordings, though, never make it to igt @xcite . simply transcribing field data (without other annotations) can take up to one hour per minute of recorded speech @xcite . adding additional annotations is even more expensive. this bottleneck keeps vast collections of field recordings from achieving their full documentary potential. producing igt from audio is a tractable problem. while linguists can do little or nothing to address the underlying causes of language endangermentwhich form an intersecting lattice of political, cul-tural, and economic factors-speech and natural language processing researchers can do even less. technologists can, however, facilitate the efforts of field linguists and language workers to document endangered languages by developing technologies that make the mammoth tasks of annotating field data surmountable @xcite . for example, they can develop models that automatically generate first-pass transcriptions from raw speech. research has demonstrated that such models can speed up transcription dramatically @xcite . in direct support of the goal of language documentation, we propose a new speech and language processing task: wav2gloss. this task assumes recorded speech as the only input. the output consists of aligned annotations for transcription (with and without segmentation), glossing, and translation. in order to allow the research community to participate in this task, we introduce the following: ## dataset we present the fieldwork dataset, a collection of linguistic field recordings with audio that has been transcribed and glossed in igt. we build upon doreco @xcite and multi-cast, which are curated collections of field data, as well as data released through the cocoon repository, foot_0 and data produced by the inel project foot_1 and nin-jal @xcite . our main contributions are selecting data for which there is both audio and gloss; compiling this data into a single structured, computer accessible dataset; and providing transcription and glossing benchmarks for each language in the dataset. our work would not have been possible without the dedicated work of expert field linguists and speaker communities. our first step in adapting the fieldwork corpora for our four subtasks is to select languages where both audio and igt are available. there are some overlaps between doreco, multi-cast, and inel. for each language that appears more than once in those three sources, we only use the data from the source that contains the highest number of utterances. we do not attempt to merge corpora of the same language from multiple sources. we also require all languages we select to have a permissive cc license without a no derivatives (nd) restriction. 4 the list of language corpora selected along with their license information, and the number of hours of data available in training and combined dev and test splits is shown in annotated data come in a variety of formats such as json or xml. most of our source data come in xml-based formats-with @xcite being the most popular-as well as exmaralda @xcite and pangloss dtd @xcite . 5 in elan and exmaralda, annotations are text strings with beginning and end time stamps, organized into tiers including underlying form, surface form, transcription, gloss, and unique id. we extract annotations associated with utterances along with their corresponding audio spans, convert all audio files to wav format with a single channel and 16 khz sampling rate, and store annotations in an intermediate yaml based format @xcite that is easier to process, read, and edit. we manually inspect the annotations for conversion errors and non-speech markers. finally, we partition the corpus into train/dev/test splits. we create partitions that contain full documents in order to preserve the contextual information. assuming each document covers a different topic and has slightly different recording conditions, using full documents will make modeling more challenging and realistic, since the dev and test splits will be out-of-distribution, with minimal overlap in content between the splits. for each language, we look at the number of utterances in total to determine the splits. if there are fewer than 200 utterances, all of them are assigned to the test set; if there are between 200 and 1,000 utterances, we assign 25% of the data to the dev set, and the rest to the test set; if there are more than 1,000 utterances, we assign 250 utterances to the dev set, 750 utterances to the test set, and the rest to the training set. with those partitions determined, we use a knapsack solver foot_4 to optimally assign documents to splits based on the number of utterances within each document. 7 we then apply final text cleaning by normalizing punctuations and removing special symbols on the transcriptions and translations, and convert the corpora into a hugging face dataset which can be readily used to train and test speechto-text models. in the following paragraphs, we present information on the specific data sources and unique processing required for each. ## experiments we provide benchmarks for automatically generating igt by fine-tuning pre-trained speech and text models. we choose commonly used models, methods, and finetuning settings with their corresponding codebases to provide a solid baseline for future research. see appendix a for details on models sizes and hyper-parameter settings. ## results we compare average model performance on seen languages (the 22 languages with training sets) and unseen languages (the 15 languages with only dev and test sets) in owsm model, on the other hand, is better at generating gloss and translation. since xls-r is our best model for transcription, we employ it for the cascade setting so that its transcription outputs become the inputs for the text annotation model. the cascade approach produces better translation than all end-to-end models, but fails to improve on the underlying or glossing tasks. pretraining the text model on odin slightly improves glossing performance, but not enough to surpass end-to-end. unsurprisingly, models generally perform better on seen than unseen languages. the overall performance of the models is low in absolute numbers across most of the tasks, with asr being the easiest task, and gloss and translation the hardest. qualitatively, from inspecting the models' outputs on a few languages, it seems that they are not able to generate coherent and relevant translations. this highlights the challenges with building nlp resources for low-resource languages, with minimal data spread across many languages. even models that were pretrained with multilingual datasets are hard to adapt to languages in fieldwork. ## related work our study is informed by a richness of previous work in automatic glossing, including a number of proposed systems to predict igt from segmented or unsegmented transcriptions. in the recent sig-morphon shared task on interlinear glossing, the best performing models included hard attention, transformer, and lstm @xcite . @xcite showed that byt5 models were not best at glossing, but had higher tolerance to noise and could benefit from noisy fine-tuning data such as odin (informing our decision to use them here). fieldwork is made possible by the work of field linguists. previously, there have been many text-only igt datasets from linguistic field works available for machine learning use, such as odin @xcite , imtvault @xcite , or the sigmorphon igt shared task data @xcite . however, to our knowledge, fieldwork represents the first multilingual machine readable dataset focused on speech and interlinear gloss. our work is also informed by prior study in low-resource asr. previous low-resource asr methodologies include fine-tuning high-resource asr models @xcite or self-supervised speech models @xcite . later researchers built on these methods by continuous pre-training @xcite @xcite , model adaptation @xcite , and data augmentation @xcite . these prior works informed our use of pre-trained models; however our work is the first to explore their adaptability to predict morphological, glossing, and translation annotations along with transcription. since many low-resource languages need these annotations, fieldwork can provide a valuable benchmark to facilitate future asr and igt prediction. ## conclusion and future work in this work, we make a number of strides to advance technology-assisted documentation needed by language communities. we define a new task, wav2gloss, which is to predict igt annotations directly from speech. we present fieldwork, the first dataset for this task, comprised of audio files and expert annotations that were cleaned and formatted. we provide benchmarks across various training methodologies for transcription, underlying form prediction, glossing, and translation. and we analyze various prominent trends from experimental results. these data, benchmarks, and preliminary experimental insights provide a strong foundation for future wav2gloss breakthroughs, to expedite creation of needed language resources for communities of dying languages. this work may be continued in a number of ways. the models presented in this work represent baseline approaches, and we hope this work will spur on more experimentation on the optimal hyperparameters for this task. in particular, the choice of model size and the tokenization method can be further explored. moreover, we think multimodal setups such as @xcite where both speech and text inputs can be fed into the model at the same time can be very promising for wav2gloss. another possible avenue of research is the use of cascades consisting of more than two models, for example, asr into morpheme segmentation into glossing. future researchers may also further normalize igt labels, to expand fieldwork to more diverse languages and phenomena. this is labor intensive, as previous research @xcite and we ourselves observed, given inconsistent use of labels and language-specific phenomena across igt collections. expansion of fieldwork to more languages may also come through community-driven projects to benefit low-resource language communities and academics. researchers may also focus on improving our work's modeling capabilitiese.g., adapting models to zero-shot performance. in our own work, all models perform notably better on seen than on unseen languages. future work may mitigate this by mapping all transcriptions to a shared vocabulary, such as ipa, to minimize superficial orthographic differences.