Towards Human Evaluation of Mutual Understanding in Human-Computer Spontaneous Conversation: An Empirical Study of Word Sense Disambiguation for Naturalistic Social Dialogs in A merican E nglish (2022)

Abstract: Current evaluation practices for social dialog systems, dedicated to human-computer spontaneous conversation, exclusively focus on the quality of system-generated surface text, but not human-verifiable aspects of mutual understanding between the systems and their interlocutors. This work proposes Word Sense Disambiguation (WSD) as an essential component of a valid and reliable human evaluation framework, whose long-term goal is to radically improve the usability of dialog systems in real-life human-computer collaboration. The practicality of this proposal is proved via experimentally investigating (1) the WordNet 3.0 sense inventory coverage of lexical meanings in spontaneous conversation between humans in American English, assumed as an upper bound of lexical diversity of human-computer communication, and (2) the effectiveness of state-of-the-art WSD models and pretrained transformer-based contextual embeddings on this type of data.

## introduction as surveyed in @xcite , current evaluation practices for human-computer spontaneous conversation, including open domain dialog systems and chatbots, exclusively focus on the quality of system responses, e.g. how well the responses match ground truth human responses (based on certain automated metrics) or whether they are ontopic with the immediate dialog history (judged by a human). these evaluation practices potentially drive researchers into the race of generating better surface text while undermining or ignoring the ultimate goal of capturing mutual understanding between the systems and humans throughout the conversation (cf. the great misalignment problem raised by @xcite . consequently, current systems are unable to effectively function in real-life human-computer collaboration tasks. for example, the lack of genuine conceptual alignment with users leads to language learning chatbots being used only as reactive systems, even though theoretically they could provide the learners with the opportunity for free and flexible meaningful conversation @xcite , and consequently play a key role in supporting autonomous language learning beyond the classroom. to improve the usability of dialog systems for humancomputer spontaneous conversation, their evaluation should include human-verifiable aspects of language competence which facilitate mutual understanding (instead of treating them as black box functions). moreover, breaking down the evaluation into such concrete components would allow users' participation in system evaluation from early development stages @xcite . currently, talking to social chatbots without knowing which sense of a semantically ambiguous word foot_1 the chatbots have in their internal interpretation, human evaluators cannot identify the root cause of a problematic conversational move performed by the chatbots to provide more useful feeback. for example, examining the dialog shown in figure 1 , we can agree that the last utterance produced by the chatbot is not appropriate. however, we cannot know for sure if that is due to the chatbot's inadequate interpretation of "bank" foot_2 in the preceding question "what do you do at a river bank?", or its complete ignorance of the meaning of this word by just generating the most probable utterance according to the dataset it is trained on. arguably, one of the most natural ways for social chatbots to enhance the quality of their interaction with humans is explicitly assigning semantically ambiguous words specific senses, aka word sense disambiguation (wsd), and using these senses for further reasoning foot_3 to demonstrate the chatbots' understanding capability with human-readable aspects of grounding @xcite in the course of spontaneous conversation. this would improve human-computer communication in collaborative tasks by allowing the human partners to directly access the interpretable form of computers' model of conversation anytime they need to so that they can make adequate on-the-fly conversational adjustments. in addition, being able to access the computer's human-readable representation of conversational context in the evaluation regime, a human evaluator does not need to construct different interpretation alternatives and therefore can be confident that they are on the same page with other evaluators (cf. appendix a -a small experiment that shows a wide divergence in human interpretation of a word token in spontaneous conversation). this transparency definitely reduces the subjectivity of the evaluation task, and therefore improves its reliability and reproducibility @xcite . this work proposes and evaluates wsd as an essential component of a novel human evaluation framework intended for human-computer mutual understanding in spontaneous conversation in english, but also sensible for any tasks involving natural language interpretation. specifically, based on the state of the art in wsd @xcite , it addresses the following research questions: to address (1-3), i first automatically annotated wordnet senses of ambiguous words in newt-sbcsae, a publicly accessible corpus of naturally occurring spontaneous dialogs in american english @xcite @xcite , using both a sota wsd model and a simple baseline model directly based on contextual embeddings of pretrained transformers (section 2.2). next, i collected human judgements on the outputs of these models as well as the appropriate senses of the target words (section 2.3). these judgments were then used to assess the coverage of the wordnet sense inventory (section 3) and the efficacy of wsd models, including both models used in automatic sense annotation (section 4.1) and variants of the baseline model based on various pretrained transformers (section 4.2). ## experimental setup the experiment reflects the proposed wsd-based evaluation protocol: ambiguous words in spontaneous dialog are first disambiguated by dialog systems and then evaluated by humans (or, less interactively, against predefined gold standard data). ## wordnet sense coverage wordnet senses cover 96.3% of ambiguous words as shown in ## discussion future work next, i will perform a detailed data analysis to gain insights into (1) what the annotators disagreed about, (2) what kinds of errors the wsd models made, and (3) how good incorrect senses are, taking into account the distinction between polysemous and homonymous senses, which is not available in wordnet @xcite @xcite . these insights will help improve the design of the annotation task and the performance of the wsd models. i will also study the effect of manipulation of input utterances, by taking into account the linguistic and discourse information about the target words, on the performance of the pretrained transformers. this can shed light on how to create optimal contextual embeddings of ambiguous words for wsd. limitations and challenges exclusively relying on pre-existing sense inventories such as wordnet, the proposed evaluation method would not only miss semantically ambiguous words that do not have multiple senses in these sense inventories, but also inherit their limitations, due to the fact that their senses have different degrees of granularity and cannot keep up with the continuously involving character of natual languages @xcite . the proposed evaluation method may not easily be adopted by the developers of end-to-end dialog models, the most popular approach to open-domain dialog systems @xcite , as the "black box" nature of these systems does not facilitate human-readable word-level interpretations. ## conclusion this work proposes wsd, an established nlp task, as a required component of a valid and reliable human evaluation framework for mutual understanding in human-computer spontaneous conversation. the conducted experiments demonstrate the practicality of this proposal for english. to sufficiently evaluate human-computer mutual understanding, i envision that the wsd component will be necessarily coupled with a reasoning judgment component in which human evaluators assess the appropriateness of conversation moves made by a dialog system, including clarifying and adjusting their interpretations, based on the disambiguated word senses in those moves. this setting will help human evaluation become more grounded and therefore more objective than the current common practices, in which human evaluators are asked to rate system responses using vaguely defined criteria and inconsistent numeric scales @xcite . linguistic annotation types with arbitrary tagsets, and annotated with flat foot_11 , folia's web-based annotation tool whose user-interface can show different linguistic annotation layers at the same time @xcite .