An Interpretable and Crosslingual Method for Evaluating Second-Language Dialogues (2025)

Abstract: We analyse the cross-lingual transferability of a dialogue evaluation framework that assesses the relationships between micro-level linguistic features (e.g. backchannels) and macro-level interactivity labels (e.g. topic management), originally designed for English-as-a-second-language dialogues. To this end, we develop CNIMA (**C**hinese **N**on-Native **I**nteractivity **M**easurement and **A**utomation), a Chinese-as-a-second-language labelled dataset with 10K dialogues. We found the evaluation framework to be robust across languages, revealing language-specific and language-universal relationships between micro-level and macro-level features. Next, we propose an automated, interpretable approach with low data requirements that scores the overall quality of a second-language dialogue based on the framework. Our approach is interpretable in that it reveals the key linguistic and interactivity features that contributed to the overall quality score. As our approach does not require labelled data, it can also be adapted to other languages for second-language dialogue evaluation.

## introduction in the context of second language (sl) assessment, speaking has always been considered an essential ability @xcite @xcite , but prior studies have predominantly focused on written correction @xcite or pronunciation from asr @xcite . given the lack of researches that capture the unique linguistic features of sl speakers in dialogues (especially in open-domain interactive conversations), this leaves conversational interaction assessment under-explored and ultimately contributes to a limited understanding of conversational fluency and interactivity for sl speakers from different languages. an exception is @xcite , who propose a two-level framework for assessing the interactivity ability of english-as-a-second-language (esl) speakers in open-domain conversations. the framework introduces micro-level word/utterance features (e.g. backchannels) and macro-level interactivity labels (e.g. topic management) and they find that the micro-level features are highly predictive of the macro-level labels. however, they conduct the study only in the context of esl, raising questions about the transferability of the evaluation framework to other languages besides english. furthermore, they do not introduce an automated approach for this evaluation, as the analysis relies on human annotations. our work aims to address these shortcomings by: (1) testing the cross-lingual transferability of the evaluation framework by validating it on an annotated chinese-as-a-second-language (csl) dialogue dataset; and (2) introducing an automated, interpretable approach with low data requirement that scores the overall quality of a second-language dialogue. our automated approach is interpretable as it highlights the key features that lead to the overall score, and it has a low data requirement because it does not require labelled data. as it can be adapted to other languages easily, our method paves the way for a new tool for assessing sl conversations. 1 to summarise our contributions: ## evaluation framework we extend the evaluation framework of @xcite for assessing second-language dialogues, which considers micro-level features and macrolevel interactivity labels. one addition we have made is to introduce an overall dialogue quality score. figure 1 shows an example of csl dialogue annotation with the evaluation framework. the framework has 17 micro-level features, and it can be further broken into token-level (such as 'reference word': she and 'backchannel': hmm) and utterance-level features (such as 'formulaic response': how's going). the micro-level features are annotated as spans (i.e. annotators mark text spans corresponding to a feature). for macro-level interactivity labels, there are four: topic management, tone choice appropriateness, conversation opening and conversation closing, and they are annotated as (dialogue) labels. briefly, topic management refers to the speaker's ability to control and navigate the flow of topics; tone choice the suitability of the tone; and conversation opening/-closing the naturalness in the initial exchange and conclusion of the discussion. a more elaborate definition can be found in appendix macro-level interactivity label, the score ranges from 1 to 5 (categorical), and higher scores indicate a more natural and active interactivity quality. note that for tone choice appropriateness, higher scores denote a casual tone, and lower scores indicate a formal tone. 2 in addition to micro-and macrolevel features, we introduce an overall score for each dialogue to measure second language speakers' interaction abilities in open-domain conversation. like the macro-level labels, it is scored from 1 to 5 (categorical). the overall score is designed to capture the holistic quality of a conversation, integrating elements like contextualisation, responsiveness, and communicative purposes across the whole conversation, and a high score reflects the speaker's ability to engage in fluid and meaningful interactions. the full description of each score is detailed in appendix ## automated pipeline we now propose automating the prediction of the micro-level features (step 1), macro-level interactivity labels (step 2), and overall score (step 3); see figure 2 for an illustration. we experiment with classical machine learning models (logistic regression (lr), random forest (rf), and naïve bayes (nb)), fine-tuned chinese bert @xcite , and gpt-4o (openai, 2024) with prompts to automate these steps. note that these predictions are done in sequence, where the output from the previous step is used as input for the next step (e.g. when predicting the macro-level labels, we use the predicted micro-level features as input). we partition cnima into train, development and test sets (@xmath0:1:2) for these experiments. step 1 this step predicts the spans of the 17 micro-level features given an input dialogue, and we experiment with bert and gpt-4o here. for bert, we fine-tune 17 span classifiers (one for each feature) to classify the presence or absence of a feature for each word in the dialogue. for gpt-4o, we do one-shot prompting (i.e. 1 dialogue with the expected output as a demonstration) with for bert and gpt-4o, the input is the dialogue concatenated with the micro-level feature spans. bert is fine-tuned as a document classifier, and gpt-4o is one-shot prompted with instructions. step 3 this step predicts the overall dialogue quality score. as this is a document classification task like step 2, we follow the same process for building lr, rf, nb, bert and gpt-4o. note that the classical models (lr, rf, nb) use only the normalised macro-level interactivity score as input features (in other words, they have only 4 input features), while bert and gpt-4o uses a concatenation of the dialogue, micro-level feature spans, and macro-level interactivity scores as input. for more details on configurations and prompts, see appendix a.5 for bert and a.6 for gpt-4o. ## experiments we first replicate the esl experiments of @xcite for csl and compare their results in section 6.1. we then assess the performance of our proposed automated approach for sl assessment in section 6.2. ## conclusion we introduce cnima, a large-scale annotated dataset for chinese-as-a-second-language (csl) dialogues based on the evaluation framework introduced by @xcite . this allows us to test the cross-lingual transferability of the framework, as it is originally designed for english. our experiments demonstrate that the framework works well for csl dialogues, suggesting that it is robust across languages and that it is a viable framework for evaluating any second-language conversations. our analyses also reveal language-specific and language-universal relationships between microlevel linguistic features (such as backchannels) and macro-level interactivity features (such as topic management). next, we propose an automated, interpretable approach that predicts the micro-and macro-level features and the overall quality of second-language dialogues. to this end, we explore a range of models, from classical machine learning models to llms. our approach is interpretable because it does not only predict the overall dialogue quality score, but also reveals the key microlevel and macro-level features that contribute to the score. our llm-based approach performs very well, and as it does not require any labelled data for training, it can be adapted to other languages easily. ultimately, our method paves the way for an interpretable and practical system for automatic second-language dialogue assessment.