Applications of Natural Language Processing in Bilingual Language Teaching: An I ndonesian- E nglish Case Study (2020)

Abstract: Multilingual corpora are difficult to compile and a classroom setting adds pedagogy to the mix of factors which make this data so rich and problematic to classify. In this paper, we set out methodological considerations of using automated speech recognition to build a corpus of teacher speech in an Indonesian language classroom. Our preliminary results (64% word error rate) suggest these tools have the potential to speed data collection in this context. We provide practical examples of our data structure, details of our piloted computer-assisted processes, and fine-grained error analysis. Our study is informed and directed by genuine research questions and discussion in both the education and computational linguistics fields. We highlight some of the benefits and risks of using these emerging technologies to analyze the complex work of language teachers and in education more generally.

## introduction using quantitative methods to understand language learning and teaching is difficult work as limitations in the recording, transcribing, and analyzing of data continue to constrain the size of datasets. it is not surprising then, that quantitative studies looking at second language foot_0 acquisition have been critiqued for their low statistical power @xcite . usage-based analyses of teacher corpora are an important next stage in understanding language acquisition @xcite . given the magnitude of worldwide investment in l2 teaching and learning, drawing on developments in automated methods of compiling this kind of speech data is timely. consequently, we sought to address the following main research question in this paper: how can automated speech recognition (asr) be adapted for this use? more specifically, i) how well do these speech-to-text tools perform on this type of data? and ii) how do these tools and datasets relate to the overall purpose of opening a window into the practice of language teachers? such an endeavor requires careful consideration of how asr models are built, and what the underlying training data and desired output of such models might be. in this paper we use the term asr model to refer to statistical models used to map speech sequences and sounds to respective text sequences @xcite . our study is drawn from a project investigating the teaching of indonesian. data was collected from a tertiary indonesian language program at an australian university. a single teacher's speech was recorded throughout one semester of a secondyear language program. in investigating indonesian language teaching, ideally various instances of linguistic features and non-standard indonesian would be annotated to allow for analyzing various topics, including, for instance, the comprehensibility of teachers' speech, movement between the l2 and assumed first language (l1), representations of regional indonesian languages, and non-standard varieties and loanwords. yet, the tools tend to restrict the data structures for annotating the audio. as an example from the conclusions of this paper, the classification of data as belonging to the l2 (indonesian) or the l1 (english) quickly emerged as a very significant issue. the paper is organized as follows: we begin by presenting an overview of related work in transcription and asr before describing our methodological approach, with subsections on bilingual and classroom teacher data. this is followed by a more detailed description of our materials and methods to train and evaluate asr models. finally, we present experimental results of our machine transcription, discuss them, and conclude the study. ## background transcription is a complex task traditionally seen by linguists from the perspective of linguistic theory and documentation of complex language structures and phenomena. linguists and their research teams become extremely familiar with their data during the process of transcribing, and their publications usually make reference to data-specific guidelines developed for their transcription teams @xcite . often these are adapted from generic guidelines or rules for annotating language which aim to record the "most basic transcription information: the words and who they were spoken by, the division of the stream of speech into turns and intonation units, the truncation of intonation units and words, intonation contours, medium and long pauses, laughter, and uncertain hearings or indecipherable words" @xcite . most teams use sophisticated software tools, which provide a method for rich interlinear annotation of speech data by humans. 2 these annotations allow linguists to record more than 'just' the words used in human communication, but obviously cannot represent all characteristics of the audio data. acknowledging the time constraints and subjectivity or bias that enter the transcription process as transcription guidelines are developed is important. the purpose of these guidelines -namely, to create uniformity of practice from individual, and teams of transcribers -may not be achievable @xcite . in fact, experiments looking at the subjectivity of transcription led @xcite to conclude that "the choices researchers make about transcription enact the theories they hold and constrain the interpretations they draw from their educational practice". moreover, a transcription survey carried out by the centre of excellence for the dynamics of language, transcription acceleration project (coedl tap) team documented a significant variety in the way linguists go about transcribing their data. the survey also found that each minute of data takes, on average, 39 minutes for a linguist to transcribe, creating the well-known 'transcription bottleneck' @xcite . advances in asr and other natural language processing (nlp) bring researchers closer to overcoming this bottleneck, but many open challenges remain @xcite . asr tools can help by providing a first-pass hypothesis of audio for languages with large datasets to train the underlying models @xcite . 3 however, financial and ethical restrictions may prevent a study from using these offthe-shelf systems and cloud computing services. 4 existing solutions may also have insufficient coverage of the domain-specific language used by speakers or not support a given l2. recognizing the potential benefit that integrating asr tools into a linguist's workflow could have, the coedl tap team has been building elpis @xcite , an accessible interface for researchers to use the powerful but complex kaldi asr toolkit. 5 according to @xcite "compared to the other recognizers, the outstanding performance of kaldi can be seen as a revolution in open-source @xcite technology". this project constitutes an early use of the elpis pipeline to prepare training data, ready for kaldi to build asr models, which can then be used to "infer" a hypothesis for un-transcribed audio. the interdisciplinary work involved in this project shines a spotlight on the limitations on the type of data used by asr systems; human transcribers often face difficult decisions as to what should and what should not be recorded in the training data. 6 since bias and error may multiply in asr models and create unreliable and undesirable outcomes, sharing the best practices and having transparent processes for creating training and evaluation data and protocols is of utmost importance @xcite . asr trained on carefully compiled data can then be scientifically tested and variations to the training data analysed for their impact @xcite . ## methodological approach our methodological considerations addressed the following: which asr tool to use, how to prepare training data for this tool, and how to best manage the bias of the training data inherent in all transcription processes. kaldi's orthographic transcription capabilities and elpis' processing and output of time-aligned elan files were a good fit with the broader research goals in lexical analysis, including dispersion analysis. in general, we took a pragmatic approach to managing the loss of data from audio recordings, viewing information such as rising intonation 7 as something that was unnecessary to our lexical focus and which could be added later if the data were used for different research purposes. we were able to minimize some loss using a tier structure in the elan training data and this allowed us to maintain syntax relationships and other information used by kaldi in the data. entwined in the issue of data loss, was the management of subjectivity in transcription. indonesian and english native speakers, linguists, and an indonesian language teacher worked together to transcribe our training data and we used extended discussions of specific samples to develop our transcription guidelines, including some discussions with our teacher participant. meanwhile the tier structure allowed consideration of the teacher's behavior from an alternative framework discussed below; that is, translanguaging. ## materials and methods the audio data was recorded in a second year tertiary indonesian language program at an australian university (ethics approval no. 2017/889 of the australian national university human research committee for the speech recognition; building datasets from indonesian language classrooms and resources protocol). the teacher, who was recorded over the course of one semester, grew up using indonesian in school and public places, and a regional language at home. a semester of over 32 hours of class was recorded. 13 the teacher wore a head-mounted microphone and wireless bodypack linked to a zoom recorder set to record 44.1 khz, 36-bit wav format audio. because students were not the target of the study, the microphone settings were optimized to exclude their voices. three lessons of approximately 50 minutes were chosen for transcription as training and test data for the asr. the lessons were selected to contain a range of content, instructional styles, and activities. the remaining audio recordings were held out from training and testing. 14 ## preliminary results from automated speech recognition and their analysis the wer of three models was from 64% to 89% ( the wer of the large commercial services is typically related monolingual tasks, usually on english data, and outside the classroom context. in a monolingual spanish classroom environment, an impressively small wer of 10% was reported using a tailored, commercial asr system with test data of two 50-minute university lectures and one 50-minute seminar with 10-16 year-old students @xcite . in contrast, for monolin-gual us english-speaking teachers' speech, a wer from 44% to 100% was reported for five asr systems, which were free of cost to use and required no additional supervised learning to train the asr model @xcite . in our results, we analyzed teacher speech phenomena, such as emphasized articulation. for example, an instance of 'sma', an acronym for a senior high school produced as the indonesian names of the letters, [es em ah] was hyper-articulated. our bilingual_3g and indonesian_3g models produced reasonable approximations: 'ah sma aha' and 'hasan aha', respectively. given the variation this sort of phenomena introduced into lexical items, teacher speech characteristics seem likely to have impacted our asr performance. 2019 )). many of these studies note their shortage of training data and some report success in using training data from high resource languages to work with low resource languages. for example, @xcite experimented with a new south african soap opera corpus in which five languages were present and found that the incorporation of monolingual, out-of-domain training data reduced their wer. working with the same corpus, @xcite first trained bilingual systems and a unified five-lingual system, and then experimented with adding convolutional neural network layers to these models. overall they achieved wers ranging from 43% to 64% for paired languages and from 26% to 78% for single languages. while performance between different language pairs might not be suitable for comparison due to the interplay of language typologies interacting in distinctive ways, wers from codeswitch bilingual data were more similar to our wer, especially given our small amount of training data. @xcite studied indonesian, iban, and malay codeswitching in written work, however, to the best of our knowledge, our work was the first work on spoken indonesian-english data. wer rates were useful in relating our results with the overall progress being made in asr, but given our goal to expedite human transcription, for us, it was more fruitful to analyze the number and length of correctly recognized text spans in the asr-based transcription. we theorized that these tools could begin to change workflow or decrease cognitive load for human transcribers by generating a draft transcript for revision. the two 4-word and one 3-word correct text spans produced by our bilingual_1g model would probably be the most useful in speeding transcription ( proceeding to a more detailed study of the performance of the models, we undertook an error analysis to elucidate the type of errors occurring. we analyzed them as segments, from multiple perspectives (figure 4 ). there was a high incidence of resyllabification foot_17 in the machine transcription, as words were split, concatenated with the preceding or succeeding word(s), a middle consonant was omitted, and/or an initial consonant was omitted. for example, 'perguruan' in the reference transcript and 'per keren' by asr accumulated three errors: resyllabification and two counts of substitution. 22 another example is, 'it' in the reference transcript produced as 'old' in the asr output. this error was coded for a vowel change and consonant change. 23 given the small test set, using this error analysis, we made the tentative note that the bilingual_3g model seemed slightly less likely to make errors of insertion and deletion, indicating that the errors were perhaps less 'disruptive' than the errors in the other models. thus, despite the model's worse overall performance, it might improve rapidly with more training data. 24 ## discussion as our principal result, we concluded that kaldi, in conjunction with the elpis interface, can expedite the transcription of teacher corpora. the time taken to transcribe speech can be extreme; in our project, transcribers spent months familiarizing themselves with the participants speech and setting up extensive transcription guidelines. our final 51 minutes of test data took approximately 1, 024 minutes 25 to transcribe. 26 however, the use of kaldi and elpis was also time-consuming and required significant training and expertise. the continued development of elpis may make the tool more viable for asrassisted transcription in research. our detailed discussion of methodological issues arising during human transcription of training data cannot prescribe a solution for all language teacher corpora; as helm and dooly (2017, p. 170) say of their own methodology paper examining the transcription of online language classroom data, their methods necessarily reflect "the research questions and the situated context of the study". however, we do hope to provide a baseline of discussion for those developing training datasets with this kind of complex speech. similarly to @xcite , we hope to "highlight how we can try to be reflexive and critical in our research practices, increasing the transparency and accountability of our work and opening it up for discussion with others". this is especially pertinent as we debeyond the goals of this study, our findings contribute to expanding bodies of research into the use of asr with small datasets @xcite , in educational and classroom settings, as well as asr of multilingual data. our results gave some indication that while developing an initial (small) training dataset, using a simpler unigram model with less lexical information is better. of course, asr could be enhanced with a larger training dataset and supplementary text corpora from teaching resources. data loss was inevitable when we converted enacted classroom interactional phenomena into the linear, rather two-dimensional written format of orthographic transcription. this loss of complexity causes us to raise a cautionary flag; datasets produced through these methods can be used to support teacher reflection on their practice, but should never be taken as the entirety of a teacher's work and metrics derived from them should be viewed with a careful understanding of how much they reduce the complexity of the phenomena they record. losing the context of data is not an "obscure problem apparent to a few philosophers focused on cy-bernetics" @xcite . in this paper, we highlight the importance of decisions about 'what data to lose' when transcribing, making tactical decisions that are justified by research questions @xcite , and how transcription bias can be multiplied in unknown ways by computational processes. investments are necessary to convert the tools we used to a useable workflow for practicing teachers. elpis is likely to make significant headway in this area, but the complex nature of transcribing bilingual teaching data requires specialized skills. training teachers to do this work seems a useful area of technological investment in languages education. it could incorporate established uses of teacher corpora for teacher training and professional development with new goals of elucidating the language input teachers provide in the classroom. teachers who transcribe a small training dataset of their own speech may gain deep insight into their own language use. using asr to accelerate transcription could lead to teachers having the capacity to build larger datasets, analyze their own teaching, and thereby progress their practice. given the workload issues often associated with teaching, asking teachers to transcribe their own lessons may be unrealistic in the initial development of this tool but could be more appropriate in teacher training settings where it could be included as part of their studies. engaging with concerns in education about the use of teaching technologies as performance management tools @xcite , this tool in teachers' hands could advance action research and protect teachers from it being used as a supervision/performance management tool. ## conclusion having trained and applied asr in the form of kaldi and elpis to a dataset of carefully prepared indonesian language teaching data, it is clear that the applicability of these technologies is limited with such a small set of training data. yet, further investigation and development toward the goal of expedited transcription is warranted because of the virtuous cycle of asr-assisted human-workflow. the limitations and risks of these technologies must be considered if we hope to use them to gain real insight into the practice of language teachers. however, it is crucial that education is not excluded from technological advances. empirical information about teacher practice for teachers, curriculum writers, educational researchers, and policy makers could be used to inform and advance the education sector the same way as these computational advancements are already routinely used in industry and other sectors.