Automatic Assessment Of Spoken E nglish Proficiency Based on Multimodal and Multitask Transformers (2023)

Abstract: This paper describes technology developed to automatically grade students on their English spontaneous spoken language proficiency with common european framework of reference for languages (CEFR) level. Our automated assessment system contains two tasks: elicited imitation and spontaneous speech assessment. Spontaneous speech assessment is a challenging task that requires evaluating various aspects of speech quality, content, and coherence. In this paper, we propose a multimodal and multitask transformer model that leverages both audio and text features to perform three tasks: scoring, coherence modeling, and prompt relevancy scoring. Our model uses a fusion of multiple features and multiple modality attention to capture the interactions between audio and text modalities and learn from different sources of information.

## introduction language proficiency testing is an increasingly important part of our society. the need to demonstrate language abilities through standardized testing is required in many situations for access to higher education and employment opportunities. this paper presents an automatic system to address the assessment of english spoken proficiency with cefr level. our framework contains two tasks: elicited imitation and spontaneous speech assessment. the elicited imitation task taps into reading and speaking skills by requiring examinees to say a sentence out loud. test takers must be able to process the input and are evaluated on their fluency, accuracy, and ability to use complex language orally @xcite . we employ statistical machine learning (ml) and natural language processing (nlp) using a transformer-based classifier to directly estimate item difficulties for a large item bank. for spontaneous speech assessment, the candidates are asked to talk about a prompt/questionrelated topic. our spontaneous speech system is based on ef standard english test (efset) dataset. in the proposed system, the students' spoken answers are first transcribed by a state-of-theart automatic speech recognition (asr) system and then scored using a multimodal and multitask framework. this work argues that audio and text features are complementary for a valid automatic spoken assessment system @xcite . the contributions in this paper are threefold: 1) we propose the use of test items for elicited imitation that can be automatically created and graded using a bert transformer; 2) a multimodal and multitask framework for spontaneous speech assessment combining audio and text is proposed; 3) a complete automated assessment framework was built and evaluated using a calibrated dataset. in the pages that follow, we first summarize the state-of-the-art in automated speech assessment and then describe our approach to assess language proficiency. we then present evidence for the validity and reliability of our approach using efset validation set and a calibration dataset. finally, we will give a conclusion. ## related works a number of approaches have been proposed to assess different aspects of a learner's spoken language proficiency. most automatic assessment systems contain an asr system, with the success of deep neural networks (dnn) in speech recognition @xcite , a number of automatic assessment systems that deploy dnn-based speech recognition systems have been proposed. the extracted features are then used to train a grader to give a score. all existing automatic assessment sys-tems are learning-based and can be classified based on whether they are feature-based, end-to-end or multitask approaches. ## proposed approach in this section, we are going to describe our system which combines elicited imitation and spontaneous speech assessment. ## ef speak oral english test the ef speak oral english test is an online assessment initially created using the methods in this paper. the elicited imitation task contains 9 items ranked by difficulty using our bert classifier. the spontaneous speech assessment task contains 6 prompts. figure 4 shows examples of items. finally, each part is scored by our framework and the final value is mapped to the corresponding cefr level. figure 4 : example of test items for spontaneous speech assessment. ## conclusion this paper has described an automatic assessment system for spontaneous english based focused on elicited imitation and spontaneous speech assessment. this system uses a multimodal and multitask framework to leverage both audio and text features. the performance of the proposed system has been evaluated using pcc and qwk measures and the best combination of features gives a pcc of 0.86 and a qwk of 0.84 when compared with expert scores.