An Effective Automated Speaking Assessment Approach to Mitigating Data Scarcity and Imbalanced Distribution (2024)

Abstract: Automated speaking assessment (ASA) typically involves automatic speech recognition (ASR) and hand-crafted feature extraction from the ASR transcript of a learner’s speech. Recently, self-supervised learning (SSL) has shown stellar performance compared to traditional methods. However, SSL-based ASA systems are faced with at least three data-related challenges: limited annotated data, uneven distribution of learner proficiency levels and non-uniform score intervals between different CEFR proficiency levels. To address these challenges, we explore the use of two novel modeling strategies: metric-based classification and loss re-weighting, leveraging distinct SSL-based embedding features. Extensive experimental results on the ICNALE benchmark dataset suggest that our approach can outperform existing strong baselines by a sizable margin, achieving a significant improvement of more than 10% in CEFR prediction accuracy.

## introduction with the unprecedented advancements in computer technology and the growing number of secondlanguage (l2) learners worldwide, automated speaking assessment (asa) has aroused much attention, figuring prominently in computerassisted language learning (call). as shown in figure 1 1 , asa systems are designed to provide timely feedback on learners' speaking quality, enabling them to improve their spoken language skills in a stress-free and self-directed manner. what is more, asa systems can alleviate the embeddings that have been successfully integrated into various language assessment tasks like sentence assessment @xcite , grading of essays @xcite @xcite , spoken monologues @xcite , and many others. on a separate front, the advent of speech-based ssl features has introduced another regime of modeling sophistication and capability to asa system developments. these features have been particularly effective in specialized tasks within the call tasks, such as mispronunciation detection and diagnosis (mdd) @xcite @xcite , automatic pronunciation assessment (apa) @xcite @xcite and asa @xcite @xcite @xcite . while the work presented in @xcite made pioneering attempts to employ ssl features (bert and wav2vec 2.0) for asa, it falls short when faced with three critical issues which our study aims to address: 1) relatively small amount of annotated data, 2) the imbalanced distribution of cefr proficiency levels @xcite , and 3) the nonuniform score gaps between different cefr levels (e.g., b2 -b1 ≠ b1 -a2 ). to address these challenges, we first utilize text-and speech-based encoders (bert and wav2vec 2.0) pre-trained on large-scale datasets. on top of this, this paper introduces effective modeling strategies that are underexplored in previous asa work, including metric-based classification @xcite @xcite and loss re-weighting @xcite . in particular, we draw on a unique set of prototypical embeddings for each cefr level and use various similarity functions to mitigate the imbalanced distribution. it is our hypothesis that metric-based learning not only addresses data imbalance but also efficiently tackles the nonuniform score gaps between cefr levels. all variants of our approach are evaluated on the icnale corpus @xcite , an opensource l2 english benchmark dataset, using both text-and speech-based classifiers. empirical results indicate that our approach can effectively mitigate the issue of data imbalance and achieve significant improvements in accuracy, rising from 77.88% to 92.63% compared to the state-of-the-art baselines @xcite . finally, we also conduct a series of analytical experiments to look into the impacts of our modeling strategies on asa performance, highlighting their practical potential for assessing learners' proficiency. this paper has three-fold contributions: we explore novel and effective modeling strategies for ssl feature extraction and classification in asa 2. we demonstrate through experiments on the icnale corpus that our bestperforming instantiation establishes a new state-of-the-art for asa on this corpus. ## related work in general, asa is deployed for assessing speaking proficiency with respect to the responses from an l2 learner, predicting the corresponding level of overall proficiency (holistic score) or specific aspects of proficiency (analytic scores). it is currently common practice to treat asa as a classification problem using either text-based or speech-based classifiers. nevertheless, in the early days, researchers used to tackle the asa problem with standard classifiers in conjunction with handcrafted features pertinent to specific facets of language proficiency, such as pronunciation, fluency, prosody, grammar, and others. these features are extracted from the utterances and associated transcripts of an l2 learner and taken as input to a meticulously selected classifier to predict analytic scores @xcite @xcite @xcite . for example, @xcite utilized vowel space characteristics for asa. @xcite ventured into syntactic analysis by employing part-of-speech tag-based complexity measures. @xcite scrutinized the efficacy of the redshift parser in processing non-native spoken english, finding proficiency in discerning grammatical relations but limitations in detecting speech disfluencies. @xcite also concentrated on assessing prosodic and spectral features. nonetheless, @xcite suggest that hand-crafted features may not always effectively capture important information about proficiency, as their efficacy heavily depends on the underlying assumptions of feature curation. as for the feature representations of text-based classification, the nlp community has witnessed a significant trend of transitioning from using static token (e.g., word, subword, and others) embeddings to contextualized word embeddings, such as those derived by bert @xcite , with ssl paradigms. consequently, there has been a surge of research on adopting these contextualized embeddings into automated assessments, such as essays @xcite @xcite and spoken monologues @xcite . @xcite initiated the use of contextualized embeddings for essay grading. @xcite applied these embeddings in conjunction with prototypical embedding for readability assessment. @xcite innovatively employed them in evaluating spoken monologues, highlighting the embeddings' versatility in linguistic assessments. on the other hand, the use of speech-based ssl features emerges as a promising approach. recent studies have shown good promise in various downstream tasks, such as asr, and speaker identification @xcite . moreover, contextualized representations derived from pretrained models can capture a diverse range of acoustic and linguistic information for l1 and l2 speech @xcite . this finding adds another dimension to the potential of ssl-based asa systems. with the increasing availability of annotated speech data and advancements in ssl techniques, there is immense scope for further research and development in the speech process. despite the promising use of speech-based ssl features in various call tasks such as mispronunciation detection and diagnosis (mdd) @xcite @xcite and automatic pronunciation assessment (apa) @xcite @xcite , there is still a dearth of research specifically focused on their application in automated speaking assessment (asa) @xcite @xcite @xcite . this situation presents significant research space and ample opportunity for further exploration and investigation. ## dataset to evaluate our proposed approach and corresponding methods, we employed the international corpus network of asian learners of english (icnale) corpus @xcite , which is a publicly available dataset consisting of written and spoken responses from both native speakers and asian learners from japan, china, hong kong, south korea, taiwan, singapore, indonesia, pakistan, philippines and thailand, at various cefr (common european framework of reference for language) levels ranging from a2 to b2. prior to data collection, the icnale team assigned cefr levels to the learners based on their l2 vocabulary size and proficiency scores in english proficiency tests such as ielts and toefl. for our experiments, we made exclusive use of the monologue section of the corpus, consisting of 4,332 speaking responses. in this setup, learners were prompted to describe their opinions on smoking in restaurants and the importance of engaging in part-time employment. following the methodological practice adopted by (banno and matassoni, 2022), this curated collection was divided into a training set of 3,898 responses, as well as a validation set and a test set of 217 responses for each. to evaluate the proficiency of l2 speakers, we will frame the asa task as a classification problem with five proficiency levels, as illustrated in ## methodology cefr levels generally follow an ordinal scale where, for example, the b1 level is considered lower than the b2 level. while it may seem reasonable to approach the asa task as a regression problem, the non-uniform gaps between the proficiency levels may lead to challenges in interpreting regression outputs @xcite . as such, we adopt a classification regime to design and implement assessment methods for cefr-oriented asa. a bit of terminology: a training set with 𝑁 samples {(𝐱 ! , 𝑦 ! ), (𝐱 " , 𝑦 " ),•••, (𝐱 #@xmath0" )} are given, where 𝐱 % is an utterance embedding extracted from either text-or speech-based encoder; 𝑦 % ∈ {0, 1, … , 𝐽 -1} indicates the index 𝑖 of the corresponding level (𝐽 = 5 in the icnale corpus). in addition, we utilize two types of ssl-based neural encoders separately as our backbone architectures, namely, a text-based encoder (bert) and a speech-based encoder (wav2vec 2.0), as schematically depicted in figure 2 . on top of this, we employ two modeling strategies: metric-based classification (cf. section 4.2) and loss reweighting (cf. section 4.3). these are used to train both text-and speech-based classifiers, with the goal to address the scarcity of learner data at basic (e.g., a1 and a2 speakers) and highly proficient (e.g., c1, c2, and native speakers) levels. ## conclusion this paper has put forward two innovative asa modeling strategies, namely metric-based classification and loss re-weighting, to enhance the performance of self-supervised learning (ssl) models for use in asa. both strategies work well with pre-trained embeddings, meanwhile addressing the challenging issues of data scarcity and imbalanced distribution. extensive experiments on the icnale dataset have demonstrated the practical utility of our methods in relation to previous methods in terms of accuracy and various metrics for measuring l2 learners' speaking proficiency. the corresponding results also provide valuable insights for discussing the efficacy of ssl made inroads into asa. for future work, we plan to delve deeper into the investigation of diverse features and traits, pre-trained models and fine-tuning strategies to mitigate the impact of imbalanced data distribution. additionally, we envisage extending the scope of our proposed modeling approach to other corpora and tasks, for the purpose of further generalizing its applicability.