Towards the speech recognition for L ivonian (2024)

Abstract: This article outlines the path toward the development of speech synthesis and speech recognition technologies for Livonian, a critically endangered Uralic language with around 20 contemporary fluent speakers. It presents the rationale behind the creation of these technologies and introduces the hypotheses and planned approaches to achieve this goal. The article discusses the four-stage approach of leveraging existing data and multiplying voice data through speech synthesis and voice cloning to generate the necessary data for building and training speech recognition for Livonian.

## introduction in october 2024 university of latvia livonian institute has launched a new project aimed at developing speech synthesis for livonian. this project has a broader goal of laying the groundwork for future speech recognition technology for the language. this article outlines the rationale behind the creation of speech synthesis and speech recognition technology for livonian, and presents the hypotheses and approaches intended to achieve this goal. recently, as technological advancements progress, significant efforts have been made to develop speech-recognition technologies for underrepresented and critically under-resourced languages, including uralic languages (e.g. @xcite . this work is also being undertaken by two key partners of the ul livonian institute in this project -ailab 1 and the university of tartu nlp group. it is worth noting that the current article focuses on the broader context, data production and handling approaches, and hypothetical methods for acquiring data and achieving the desired quality. the article does not cover the actual development of the technologies (speech synthesis and recognition) themselves, which will be conducted later in the project by the ul livonian institute's partners. ## broader context the world's linguistic and cultural diversity is in urgent danger. to safeguard endangered languages and intangible cultural heritage, major policy documents and programmes have been introduced, e.g., the unesco convention for the safeguarding of the intangible cultural heritage (2003), the un international decade of indigenous languages (2022-2032; further -idil) and others. a large part of this work in the digital era has been finding ways to narrow the digital gap between languages and cultures with extensive resources and data and the -mostly endangered -ones lacking resources and especially the data needed for creating digital technologies. with every new technology, this gap only grows, and closing it is crucial for keeping endangered languages competitive and vital. thus, in 2024, the ad-hoc group on digital equality and domains was created by the global task force of the idil to tackle these issues. another crucial aspect of this work is ensuring access to existing data and archives, especially materials only in analogue format. such archives may be a great resource for academia and communities themselves, e.g., for language acquisition or expanding the language environment, but not all analogue materials can be easily digitised and utilised, as some are scattered ## creating digital resources for livonian like many other uralic languages, livonian with around 20 contemporary fluent speakers @xcite ) is highly constrained in terms of its domains of use and available resources. while many digital tools developed in the 21st century can potentially offer support, they first require large amounts of digital data. this presents a significant challenge not only for livonian but also for most uralic languages and the majority of the world's languages. since its founding in 2018, the university of latvia's livonian institute has been developing a range of digital resources and methods for livonian, focusing on data acquisition, efficient data use, resource-saving workflows, and improved accessibility. what began as a digital dictionary has now expanded into a cluster of interconnected databases, accessible through livonian.tech (lt). this includes an up-to-date lexical database with translations of livonian lemmas and example sentences in estonian, latvian, and english; a morphology database; a partially annotated corpus of written livonian with translations; and a geospatially linked place-name database. new databases focused on documentation sources and individuals are currently in development. to meet the specific data extraction and management needs of extremely low-resource languages, tailored data processing approaches have been implemented in the construction of this cluster. in collaboration with partners, the institute has also explored opportunities to introduce various digital technologies for livonian. however, livonian continues to face significant obstacles in obtaining new data, particularly due to the extremely small number of contemporary speakers and the limited domains in which the language is actively used. the language is also nearing a "glass ceiling" of available written data foot_0 , much of which has already been digitized. ## data extraction or data synthesis? at the core of every technological advancement in the digital field lies the availability of sufficient data. for languages like livonian, which have very few contemporary speakers, extremely limited domains, and is entirely overshadowed by dominant language, the natural production of new data is an enormous challenge. therefore, to obtain the additional data necessary for developing language technologies, one must either rely on extracting data from earlier documentation (if available) or turn to synthetic data produced through the use of technology. however, both approaches come with their own flaws and risks. in terms of using earlier documentation, written records are generally more accessible. converting analog documentation into digitally usable form requires only basic technologies (such as ocr for printed materials) or staff with basic language skills, knowledge of orthography, and an understanding of transcription methods used in the original documentation. however, the amount of pre-existing written documentation is usually quite limited, as is the case with livonian. transliterating spoken language poses a much greater challenge. not only is it significantly more time-consuming, but it also requires full proficiency in the language to accurately capture what is being spoken. consequently, manual transliteration can deplete the limited human resources proficient in the language for an indefinite period of time. when it comes to synthesizing language data with the help of technology, a major risk is that the data produced may be of poor quality, due to the lack of sufficient data for building or training the necessary models. poor-quality synthetic data may even pollute the digital environment of the language @xcite , particularly given the ability of many technologies to generate large amounts of data in a short time-amounts that could far exceed the natural language data available for languages like livonian. a relevant example is machine translation, which is often one of the first technologies mentioned in discussions about endangered and low-resource languages. as with human translation, the quality of machine translation depends on its ability to generate accurate output in the target language, which, in turn, relies on the availability of training data and the methods used. consequently, it is easier-and more beneficialto develop machine translation from the endangered language to provide access to texts written in it, rather than to create machine translation into the endangered language that can synthesize high-quality data for further use. experiments with machine translation have been conducted also for livonian (e.g., @xcite , but the evaluation of these results has shown that even with the use of all available aligned and monolingual corpora and databases, the data was still insufficient to build a high-quality machine translation model capable of synthesizing quality livonian data. however, these two approaches-data extraction and data synthesis-can be combined to generate new language data and significantly enhance the data production capabilities for languages like livonian. ## choosing speech synthesis and recognition analyzing technologies that can be most beneficial for livonian from the perspective of obtaining data and best serving the community, it has been concluded that speech recognition may actually be the most advantageous @xcite at this stage. as mentioned, livonian corpus building is nearing the point where no major written sources remain to be added, but there are extensive livonian speech recordings in several archives and private collections. the contents of these recordings could significantly expand the livonian language corpus, but to access information beyond phonetic features (to get to vocabulary, morphology, syntax, etc.), these recordings need to be transcribed. currently, the only way to achieve this is through the involvement of fully proficient speakers, which is neither viable nor effective. speech recognition, on the other hand, would significantly accelerate this process. moreover, it would be a powerful tool for rapidly expanding the livonian information space (and corpus) by enabling speakers to record and transcribe large volumes of information in livonian. however, speech recognition is quite challenging to develop and requires a significant amount of data for training. conversely, speech synthesis-a technology that is much easier to develop compared to speech recognition-is another technology urgently needed by the livonian community. the need for speech synthesis arises from the fact that a natural livonian-speaking environment no longer exists, and most of the data available to the general public and those learning the language is in written form. this means that livonian speakers and learners primarily read livonian rather than hear it. to address this issue, from 2022 to 2024, the livonian lexicographic database (lt) has been supplemented with audio recordings of lemmas and example sentences spoken by contemporary livonian speakers, giving users the opportunity to hear livonian. thus, the collection of audio data necessary for developing speech technologies has already begun. successful voice synthesis would give the general public, and especially the livonian community, the opportunity to hear all digitized livonian texts immediately, although having golden-standard level voice recordings should clearly be prefferable. speech synthesis would also greatly expand the livonian audio information space, offering the community opportunities to create content or even develop audio and video media, thereby allowing livonian to be encountered in new language domains. additionally, speech synthesis could provide the conditions and data necessary for creating and training speech recognition technology. ## how is it planned? the approach proposed by the researchers of the ul livonian institute for developing speech recognition for livonian involves four stages and is based on using existing data from the livonian.tech database cluster (lt) and both alligned and non-aligned speech recordings from various informants found in archives. in the first stage, initial aligned datasets are created using pre-existing audio data from the livonian.tech databases and the 'gold standard' natural speech data from contemporary livonian speakers reading texts from written corpora. subsequently, based on this data, speech synthesis is created. in the second stage, voice cloning takes place, creating additional synthetic voices by using the natural speech data of previously recorded informants from archives. initially, the focus is on informants who have been recorded more extensively and whose speech has also been transcribed and published as language samples. in the third stage, livonian written data is used to generate a synthetic voice corpus using all available voices (both contemporary voices and those from archive recordings), in order to obtain voice data from all existing written sources. this process effectively multiplies the corresponding data by the number of synthetic voices. in the fourth stage, both natural and synthesized speech data are used to train the speech recognition model. in the final stage, voice data manipulations are planned, including techniques such as voice merging, adding disturbances based on recording quality and phonetic peculiarities, and using nonaligned data from both living and deceased speakers for training. this point is particularly important considering the varying time periods and quality of many livonian recordings. as the current project is relatively short (a little over a year), only the first stage will be fully completed, which involves creating speech synthesis (text to speech), along with some data multiplication and initial isolated speech recognition (speech to text) experiments. however, this will lay the groundwork for a continuation project, which will focus more specifically on speech recognition. the tasks to be completed within this stage are: the project is expected to conclude with the first preliminary results of the experiments on the creation of speech recognition in february 2026. ## final notes all the necessary preconditions exist for the proposed approach to be successful, at least to some extent. even a partial acceleration of the transcription process would offer significant benefits for extracting data from audio sources. gradually expanding the capabilities of livonian in the sound environment would undoubtedly strengthen the language and make it more competitive in the long run. moreover, the methods and approaches developed during this research could serve as a foundation for other research teams seeking opportunities to apply speech technologies to extremely low-resource languages. if methods for providing high-quality speech synthesis and recognition to languages with extremely limited data are discovered, this would be beneficial to all endangered languages, particularly those lacking domains for everyday language use (speech synthesis) or without a written tradition, where the language is primarily oral (speech recognition). furthermore, this would provide the varied and abundant data essential for developing other technologies, such as machine translation, chatbots, caption generation, and other solutions powered by advancing ai technologies.