Gamli - I celandic Oral History Corpus: Design, Collection and Evaluation (2023)

Abstract: We present Gamli, an ASR corpus for Icelandic oral histories, the first of its kind for this language, derived from the Ísmús ethnographic collection. Corpora for oral histories differ in various ways from corpora for general ASR, they contain spontaneous speech, multiple speakers per channel, noisy environments, the effects of historic recording equipment, and typically a large proportion of elderly speakers. Gamli contains 146 hours of aligned speech and transcripts, split into a training set and a test set. We describe our approach for creating the transcripts, through both OCR of previous transcripts and post-editing of ASR output. We also describe our approach for aligning, segmenting, and filtering the corpus and finally training a Kaldi ASR system, which achieves 22.4% word error rate (WER) on the Gamli test set, a substantial improvement from 58.4% word error rate from a baseline general ASR system for Icelandic.

## introduction icelandic open-licensed speech corpora have in recent years grown in volume and numbers, there are now talrómur @xcite , málrómur @xcite , samrómur @xcite and the althingi's parliamentary speeches corpus @xcite to name a few. however both historical speech and older speakers are underrepresented in these corpora. for instance, regarding older speakers, in samrómur, the largest open-licensed asr corpus for icelandic (2233 hours in the latest release, @xcite , only 4.8% of speakers are over 60 years old. gamli, the oral history speech corpus presented in this paper differs from that in many ways. firstly, it contains, predominantly, spontaneous speech in the form of interviews, secondly, it has a very high ratio of older speakers (94.8% of speakers are over 60 years old), thirdly, background noise is common as well as noise artefacts from historical recording equipment and lastly, historic dialects (word choice and accent) are much more prevalent than in existing corpora. the corpus contains 146 hours of aligned speech and transcripts split into a training set and a test set. this data, based on valuable historical 20th century recordings stored at the department of ethnology and folklore at the árni magnússon institute for icelandic studies, is therefore an important addition to the existing icelandic speech corpora. 1 the custom asr system presented in this paper along with the corpus will in due course be used to automatically transcribe all of the ethnographic audio recordings stored at the institute. the transcripts will then be made available on the online portal ísmús foot_1 and paired with the respective recording. ## related work for many years, asr systems have been trained on unaligned transcriptions @xcite and even approximate transcriptions of spontaneous speech @xcite . in the case of icelandic asr for spontaneous speech, there has been an ongoing project @xcite , @xcite to align and filter icelandic parliamentary transcripts for asr in order to reduce the manual work involved in transcribing parliamentary proceedings. creating the corpora involves text normalization, time-alignment, and filtering utterances. while asr for oral histories is new for icelandic, it is already being used in other languages. for example, the first large project was the malach project @xcite in 2002, where asr transcriptions were used for indexing oral history archives and making them more searchable. however, some authors still consider oral history speech recognition an open problem @xcite and a recent study @xcite found that human word error rate was 8.7% on a german oral history corpus (taking into account case-sensitivity and annotation of hesitations). whereas lippmann (1997) found a human word error rate of less than 4% on the switchboard corpus of spontaneous telephony speech and less than 0.4% on the wall street journal corpus of clear read speech. this suggests that the minimum possible word error rate for asr might be much higher on oral histories than it is for cleaner speech corpora. one other factor that makes oral history asr an interesting challenge is the particularly high ratio of older speakers. it has been noted by @xcite that for general asr models, wer correlates strongly with age, even throughout a single speakers lifetime. this could be caused by multiple changes in aging voices, such as slower speaking rate, changes in f0 (decrease for males and increase for females), increase in jitter and shimmer (all from @xcite ), some of which could be mitigated by increasing the number of older speakers in the training set. however, other changes might not be so easily solved, such as a reduction of tongue and jaw strength and an increase in breathiness (all from @xcite ) which could reduce articulatory precision. there are three main use-cases for oral history speech recognition. first, to index oral archives for spoken document retrieval. second, to provide transcripts to aid listeners. third, as a hypothesis transcript for post-editing. for each of these use-cases, it's important to determine the minimum acceptable asr performance. for the first use-case, indexing, @xcite found that using asr output significantly improves spoken document retrieval performance compared to only using the accompanying text meta data, even when wer is as high as 50%. more recently, fan-jiang et al. (2020) used a bert-based retrieval model with query reformulation and managed to get impressive results for document retrieval of mandarin news when using erroneous recognition transcripts (35%). the accuracy was 0.594 with the erroneous transcripts and 0.597 with reference transcripts. this suggests that for indexing, acceptable wer may be even higher than 35%. for the second use-case, to provide transcriptions as an aid to listeners, @xcite found that transcripts with a 25% wer improved listeners' understanding more than listening to audio without a transcript, however they found that understanding was reduced when the transcripts had 45% wer, suggesting that a maximum acceptable wer is somewhere between 25% and 45%. for the third use-case, post-editing, @xcite found that for recordings of ted talks, asr transcriptions with less than 30% wer sped up the transcription process but if the wer was higher than 30% it slowed transcribers down. ## origin of the corpus the ethnography collection of the department of ethnology and folklore at the árni magnússon institute for icelandic studies contains more than 2,300 hours of audio recordings of oral heritage and traditions, with a little less than 2,500 interviewees. the oldest material are recordings made on wax cylinders in the early 20th century and the collection is continually expanding with new material being added every year. the bulk of the collection, however, consists of recordings from the 1960's and 1970's, mainly the work of three collectors. their focus was to gather ethnographic material from all of iceland, first and foremost from older generations -the majority of the informants were born before or around the turn of the 20th century, this resulted in an extensive collection of legends and fairy tales, accounts of beliefs and customs, poems, hymns, nursery rhymes, icelandic ballads (rímur), occasional verses and more, with the material being variously spoken, sung or chanted. apart from recited verse and that which is sung or chanted the speech is spontaneous. accompanying the recordings is detailed metadata on the speaker, time and location of recording, as well as various other parameters such as genre (for different kinds of verse or prose material, e.g. poems or nursery rhymes, fairy tales or legends etc.), mode of performance (sung, chanted, spoken), key words, content (short summary, description), taletypes and motifs (in folktales and legends). ## corpus content gamli contains 146 hours of transcribed audio broken down into 1. ∼ 111 hours from optical character recognition (ocr) of previous transcriptions in various formats 2. ∼ 35 hours of new transcriptions (post-edited from asr output) the 111 hours include 9 hours defined as a test set, which was manually reviewed and corrected and annotated with speaker id and time alignments in the annotation tool elan. the test set contains recordings with 10 speakers, 5 women and 5 men, plus the interviewers (4 men) and serves for evaluating the system's performance. a validation set has not been defined for the corpus as the acoustic model training in kaldi @xcite used a random sample of the training corpus for validation. ## models (and out-of-domain data) we trained a hybrid asr system in kaldi. that is, the language model and acoustic model were trained separately as opposed to an end-to-end system. for the acoustic and language models in the custom asr system, we expanded the training sets with various out-of-domain data, which will be described in the following sections. ## evaluation to assess the final asr system's performance on the test set, we compare it to two baselines. the first is the out-of-domain system, which was trained in the same way as the final system but only on the out-of-domain data detailed in sections 5.1 and 5.2, not on the gamli training set. the second baseline is the samrómur "base" system 14 . this is a kaldi-trained system from a wellknown dataset of read icelandic speech, the acoustic mode is a tdnn chain model the language model is an n-gram model. while the asr baseline systems achieved 36.7% and 58.4% respectively on the gamli test set, the final asr system performed better, achieving 22.4% wer on the same set, as shown in to investigate the differences in the systems, we also compare the performance when taking demographic information into account in figure 3 . as stated earlier, the test set contains 10 speakers and a total of 9 hours of audio. to separate the effect that the gamli training set had on acoustic model adaptation and language model adaptation, in it seems that acoustic model adaptation had a larger impact than language model adaptation for wer on the gamli test set. this is an interesting finding, seeing as language model adaptation is generally more commonly performed, at least in kaldi, where it takes less computing power than acoustic model adaptation. though, the results from ## conclusion and future work in this paper we have presented gamli, a corpus suitable for training speech recognition systems, we have aligned and segmented icelandic oral histories from manual transcriptions (both ocr from typewritten transcripts and post-edited from asr output), and filtered out unintelligible segments. we have described the compilation of the corpus, which has been published under an open license, the origins of the data and evaluation of an asr system trained on the corpus. we have shown that using the corpus along with other relevant datasets can substantially lower wer for historical speech data, from 58.4% from a baseline system (samrómur "base" system) to 22.4%. we also draw the conclusion that it could be combined with other asr training sets which lack in historical recordings and speech from older speakers in order to improve robustness to such audio. our final asr system will be used to automatically transcribe the entire ethnographic audio data stored in ísmús, i.e. 2,300 hours of audio. we expect the outcome of that process to be in line with the results presented in this paper, with verse, nursery rhymes, singing etc. still remaining a challenge for the customised model, but accuracy for spontaneous speech to be more reliant on audio quality and clarity of speech. where the quality of these two factors is high, we expect the system to perform well. even though the wer may differ substantially for some files, the general outcome will nonetheless be a somewhat readable version of the ísmús ethnographic collection. as outlined in 1, that output can subsequently be used in a number of ways: first, indexing the ísmús ethnographic collection for search queries (useful for longer audio files where the description can not do the entire content justice). second, presenting transcripts alongside the audio as a listening aid and to increase accessibility. third, as a hypothesis transcript for post-editing of more transcripts. the gamli corpus itself should provide an interesting challenge to linguists and asr researchers interested in spontaneous speech, older speakers, noisy audio, historical recordings and historical dialects.