Speech Database (Speech- DB ) – An on-line platform for storing, validating, searching, and recording spoken language data (2023)

Abstract: The Speech Database (Speech-DB: URL: https://speech-db.altlab.app ) is an on-line platform for language documentation, written and spoken language validation, and speech exploration; its code-base is available as open source. In its current state, Speech-DB has expanded to contain content for several Indigenous languages spoken in Western Canada, having started with audio for the dialect of Plains Cree spoken in Maskwacîs, Alberta, Canada. Currently, it is used primarily for validation and storage. It can be accessed by anyone with an internet connection in six levels of access rights. What follows is the rationale for the development of speech-DB, an exploration of its features, and a description of usage scenarios, as well as initial user feedback on the application.

## introduction the speech database (speech-db: https:// speech-db.altlab.app ) is an online platform of spoken language data intended for use in the preservation and documentation of less-resourced languages. with dual function as a searchable database for transcribed and translated audio data and as a validation interface for editing spoken dictionary entries, it is available online for anyone to use, and it is easily adaptable for use in various language pairs. while speech-db has been used to store spoken data for multiple indigenous languages and their dialects spoken in western canada, here we exemplify its use in language documentation primarily for plains cree (nêhiyawêwin, iso: crk), an algonquian language spoken throughout western canada, specifically the dialect spoken in maskwacîs, @xcite . this paper will review the objectives of speech-db, both at the time it was developed as well as how they evolved, the current and future features of the service, the means by which it was developed, and some key technical features. furthermore, we discuss how the speech-db differs from other, similar services available online, as well as describe the practical usage of speech-db through a selection of qualitative user evaluations. ## background the origins of speech-db may be traced to an earlier language documentation project; namely, the spoken dictionary of maskwacîs cree / nêhiyawêwi-pîkiskwêwina maskwacîsihk @xcite . this joint endeavor between miyo wahkohtowin education (now part of the maskwacîs education schools commission (mesc: https://maskwacised.ca ) and the alberta language technology lab (alt-lab: https://altlab.ualberta.ca ), sought to achieve three primary goals: 1) to record audio for all entries (@xmath0, the maskwacîs dictionary of cree words / nêhiyaw pîkiskwêwinisa @xcite , as spoken by multiple native speakers from maskwacîs, alberta (canada); 2) to fill lexical gaps in the content of this dictionary; and 3) to elicit and record example sentences for as many of these entries as possible @xcite this project resulted in the accumulation of 341 approximately 2-hour recording sessions, each of which involved twoto-four fluent native speakers of cree and at least one linguist. these sessions were recorded at intervals in maskwacîs between june 2014 and may 2018, and ultimately resulted in the elicitation of 20,299 cree words and sentences, with anywhere between one and several tens of pronunciation tokens of the same entry by one or more speakers. in 2019-2020, these recording sessions were annotated by undergraduate students to isolate the cree vocabulary items therein and align them with the transcriptions and english translations provided in the field elicitation sheets. ## objectives and their evolution the original objective for the development of speech-db was to construct a centralized database for the maskwacîs cree audio entries (and their associated metadata) in a format that was easily accessible from other services, such as itwêwina ( https: //itwewina.altlab.app ), an online, morphologically intelligent plains cree -english dictionary @xcite . the process of validating the recording quality, cree transcriptions, english translations, and metadata (e.g. speaker id codes) of the database's audio recordings was initially planned to take place in-person in maskwacîs. however, when in-person activities became all but impossible in early 2020, a new approach was needed to enable this validation task to take place virtually. the speech-db was subsequently expanded to support this task. in the planning and organization of the various subtasks within the validation work, we aimed to optimize the impact of, and minimize the time commitment for, our native speaker consultants, who (in maskwacîs) were predominantly elderly individuals whose time was in high demand for various other language documentation and instruction tasks. thus, we divided the validation tasks into activities which categorically required the participation of a native speaker of cree and tasks which could be accomplished by a linguist knowledgeable in the language. therefore, the native speakers (or 'language experts') would be categorically needed for 1) judging the accuracy of english translations for all the cree entries in the database (and providing corrections to these), 2) judging the accuracy and naturalness of cree sentences (word choice, word order), as well as 3) judging the quality of each individual spoken token, in particular their exemplariness. consequently, the supporting linguists could undertake preparatory standardization work, such as 1) reviewing and fixing any apparent inconsistencies in the cree transcriptions, which is coupled with 2) reviewing the linguistic analyses of the transcriptions (including the lemma, stem, and other lexical information). this workflow is described in detail in section 5. in addition, given that there were roughly 150,000 individual unvalidated recording tokens at the beginning of the validation process, provisionally made available through itwêwina, we also made it possible for any person to flag recordings in speech-db that were in any respect problematic. (i.e. poor recording quality, unusual transcription or translation) for review by linguists or language experts. as the validation process proceeded to take place online (both asynchronously and synchronously through teleconference with cree elders and other speakers in maskwacîs), new features were added to ease the workflow, in effect extending the use of speech-db to (new) language documentation. one such additional feature was the ability to record entries directly into the speech-db. previously, recordings had to be done on a separate computer or with a separate software, annotated by a linguist to segment relevant snippets from larger recordings, and then uploaded to the speech-db using a custom script written by the software developer. with the addition of this feature, any authorized user (see section 4.1) can add a new recording directly to the speech-db, so long as they know the transcription and translation of the entry. these recordings are then subject to review and approval by a linguist prior to being made available to the general public. the database is also structured in such a way that new language groups can be added with minimal technical effort. all that is required for the addition of a new language on speech-db is for the site administrator to enter in the new language family; users can then immediately begin adding and viewing entries. this new language family is then presented on the introductory page as a new section of the speech-db. new sections can be instantiated with no recordings as an empty version of the speech-db; with recordings supplied by a linguist or community member in a format that can be parsed and uploaded by the software developer. alternatively, sections containing only prompts for future recordings can be created. in the case of the last option, these prompts may be taken from handwritten, gestalt lists of entries, or, more effectively, from an existing, codified semantic domain set, such as that used in the sil rapid word collection method @xcite , which would both provide an overall structure for entries and ensure a relatively balanced coverage of the lexicon. consequently, besides audio for plains cree spoken in maskwacîs, speech-db has expanded to incorporate content for another cree dialect spoken in môswacîhk, saskatchewan, as well as selected outputs from a plains cree synthesizer @xcite . additionally, extensive audio exists for the dene language tsuut'ina, imported into speech-db, as well as for three areal variants of the siouan language nakoda. speech-db's search functionality was initially very basic, featuring only the option to search for entries matching a search string. however, as needs evolved, an advanced search feature was added, allowing users to search by a variety of attributes, including recording quality ('good' or 'bad'), speaker, morphological analysis, transcription, translation, and semantic classification. the last of these attributes, semantic classification, is based on the semantic domain assigned to the entry according to the aforementioned sil rapid word collection methodology, which was used in the initial recordings sessions to collate similar vocabulary to be covered per each session. however, this semantic classification search functionality is not yet fully operational. ## description of the application 4.1 user types and functionality the speech-db supports six distinct user types, implemented so as to segment permissions and authorizations. the first of these user types are unauthorized users; that is, users who are not logged in to a speech-db account. such users can see all publicly available language families and can view and listen to all recordings belonging to those families. they have zero permissions to provide feedback or make any changes to the database, and are shown minimal metadata information for each entry (figure 1 ). the second user type is designated as 'learners'. these users, who must be logged into an account, have access to all features available to unauthorized users, with the addition of being able to flag entries for review. this allows learner users, who are assumed to be neither fluent speakers nor linguists, to provide feedback on entries without making any direct changes to the database. an internal issue is created for each flagged entry, storing the feedback from the user. issues can then be reviewed and addressed by more advanced users. in addition, learners can record new audio directly into the speech-db, subject to review by linguists. the third user type is that of the 'instructor'. currently, instructors have the same privileges as learners. in the future, instructors will receive access to specific layouts and displays intended for instructing the language, such as the option to view entries grouped by lesson type or complexity. the fourth level of access is the 'language expert'. language experts, assumed to be native or otherwise fluent speakers, have more available options than instructors, and are shown additional metadata for each entry (figure 2 ). in addition to flagging entries for review and adding new entries to the database, language experts can validate the recording quality of existing entries. validation is done through a series of steps, each involving its own button or pair of buttons on the entry. firstly, the language expert can indicate if the transcription and translation are both spelled correctly and if the meanings are correct through the use of "yes", "no", and "i don't know" buttons. the last option is provided so as not to oblige users to accept or reject entries with which they are not familiar. this option also informs the site administrators which entries require further review. next, the language expert can listen to each recording for the entry, marking them as "good" or "bad" based both on recording quality and quality of pronunciation. these changes are directly reflected in the database. while listening to the recordings, language experts can note if the cree word(s) in the recording do not match the transcription (but are otherwise valid), or if a recording is assigned to the incorrect speaker, using a series of buttons on each entry. these issues are logged as issue items and can be reviewed by either language expert users or linguists. have identical permissions as language experts, with the additional option to view and alter interlinear glosses, translations, and transcriptions directly using a button labeled "more options" (figure 3 ). this option allows 'linguist' users to make changes directly to the database. when available, transcription, translation, and analysis suggestions are provided through the use of a finite-state morphological model (for plains cree, described in @xcite and @xcite ) and dictionary content from the sister application itwêwina. suggestions are ranked by modified edit distance (med), which the service calculates itself. an entry's med is the number of changes needed for the suggestion to match the current input. the med assigns a lesser penalty to some common inconsistencies in the spelling of plains cree words that we are aware of; for other spelling divergences the regular edit distance penalty is applied. for example, adding or removing an 'i' or an 'h' has a distance of 0.5, thus 0.5 is added to the total med for every 'i' or 'h' that is added or removed from the original entry in order to match the suggested entry. adding or removing a diacritic from a character has a cost of 0, whereas adding or removing any letter other than 'i' or 'h' has a cost of 1. all these changes are calculated and summed up to present the total med between the current transcription and the suggested spelling. lastly, this linguist-specific view contains a table listing all previous changes made to an entry, when those changes were made, and by whom the changes were made (figure 4 ). this table can then be used to revert an entry to a previous state in the case that it was incorrectly changed at some point in history. the sixth and final user type is the 'administrator', a role reserved for one or two softwareeducated users who update the database in the backend and make changes to the service using django's administrator interface. the role has no special privileges on the front-end and has total control over the backend, with the ability to change any and all aspects of any given entry. as previously mentioned, many user-types have the ability to record new entries directly into the speech-db. this can be done either from the entry itself, which then adds a new provisional recording to the database containing the transcription and translation of that entry, or through the page directly intended for recording new entries. in the latter option, the transcription and translation are added in text fields before the user records as many entries as desired, saving only the ones that meet their standards of pronunciation and audio quality. if this user has recorded entries in the past, there will be a "speaker" object associated with the user and this speaker object will be used. otherwise, a new speaker object is made using the name and username provided when the account was created. ## work processes by current users as mentioned, the basic standardization and validation processes for which speech-db was designed may be exemplified in the undertaking of these processes for the maskwacîs cree audio, which (at the time that it was initially uploaded to speech-db) was aligned with transcriptions and translations taken from the field elicitation sheets and notes produced by the linguists who collected the audio. these elicitation notes were organized under semantic groupings following rapid words, and combined semantically classified content from the maskwacîs cree dictionary (which did not adhere to standard roman orthography @xcite , as well as "new" cree words and sentences in response to prompt questions and words throughout the elicitation session. as the multiple linguists recording the sessions were not fluent speakers of plains cree, the written records represented their best approximation of the phonological form of what they heard, rather than the orthographically standard form. the resultant transcriptions therefore required comprehensive orthographic standardization. furthermore, the english translations varied, either following conventions in the maskwacîs cree dictionary (maskwachees cultural college, 2009) or the larger cree: words dictionary @xcite , or some hybrid of both; these too were to be standardized. to facilitate this standardization, the maskwacîs cree audio clips were uploaded to the speech-db and grouped by the elicitation session in which they were collected (which would concern words mainly from related semantic domains). these sets of recordings were then manually reviewed by a linguist with knowledge of plains cree morphosyntax and orthographic conventions (initially the third author, and then primarily the second author). for each entry, the linguist would, using the provided recordings, verify that the word or sentence spoken in the audio was the word or sentence provided in the gloss. the linguist would subsequently standardize the spelling of the cree words in the 'transcription' field to sro conventions, render the definition in the 'translation' field to a format closely resembling that used in the largest currently existing plains cree dictionary (wolvengrey, 2011), and provide an interlinear gloss detailing the inflectional characteristics of the word(s) present in the 'analysis' field, making use of the computationally generated suggestions when suitable. a fourth field, the 'comments' field, was used in instances in which the entry in question was notable or unusual in some respect; typically, in the process of standardization, this was reserved for alternative spellings, derivational breakdowns of semantically non-compositional terms, and morphosyntactic irregularities. however, this 'comments' field (which was added to the site by the request of linguists working with speech-db) was also used as a miscellaneous repository for additional information on entries. after being manually standardized and interlinearized by a linguist, the quality of the recordings and translations for these entries were also validated by rose makinaw, an l1 cree-speaking elder from maskwacîs, in collaboration with linguists (second and third author) and the software developer (first author). across 162 validation sessions, totalling 262 hours, these audio validations have covered 50% of the total contents of the speech-db, as well as having provided 500 novel words to the database with multiple recordings of each. ## feedback from current users in total, using speech-db as an editing interface, the second author has been able to standardize roughly 63% of the 20,299 entries of maskwacîs cree over the course of 21 months of sporadic work. he has noted no significant structural deficits with speech-db as a platform (with the exception of occasional server errors), and deemed the general layout as "intuitive" and as "not requiring a great deal of training to use". furthermore, the aforementioned native cree speaker (who has no formal training in linguistics and a self-professed lack of tact in the use of computers and digital interfaces) reported no complaints regarding the practical usage of the site, and commented that she was "comfortable with it" after having been exposed to it for a time. when asked about how she would explain the interface to a new user, she commented that it would be sufficient to have them "sit beside me" during a validation session, and described her own experience of learning to use the site as "not that bad". when asked what skills a potential validation annotator using speech-db would need to begin their work, she mentioned only literacy in the cree standard roman orthography and for the annotator to be "fluent enough to know when . . . the speakers [on the database] are saying it wrong"; no mention of specialized computational or linguistic knowledge was mentioned. the software developer (first author) has participated in a large proportion of the validation sessions, from their beginning in march 2021 until the time of writing, in order to directly observe any erroneous or otherwise undesirable functionality, and consequently to resolve such issues as swiftly as possible. several of the linguists involved in the initial recordings have also participated, and have consistently judged that the validation and associated standardization activities currently undertaken in speech-db are being accomplished as efficiently as can be reasonably expected while still giving each and every recording, transcription, and translation a sufficient amount of attention for proper quality assurance. indeed, while the very first 10 validation sessions involved a learning process and covered on average 13 entries per hour, at the end of that period the rate had already increased to 25 entries/hour, having now doubled to 60 entries/hour. as for the standardization work, that has always progressed faster than validation, and has now reached a rate of 110 entries/hour. ## comparison with other relevant similar applications although other applications similar to the speech-db exist, none of the ones we are aware of are able to fill all of the aforementioned usage roles. ## conclusion the speech-db is an online platform for spoken language data available to the public in varying degrees of access, depending on the user's familiarity with the language. it serves as a service for documentation, exploration, and validation, with its functionalities having expanded over time to accommodate user needs. the primary users of speech-db regard it as easy to use and generally have no complaints about how it operates. the speech-db differs from other similar platforms primarily in its ability to grow and adapt with the language, easily add new language families, and easily add new recordings.