Exploring the Naturalness of Cognitive Status-Informed Referring Form Selection Models (2023)

Abstract: Language-capable robots must be able to efficiently and naturally communicate about objects in the environment. A key part of communication is Referring Form Selection (RFS): the process of selecting a form like it, that, or the N to use when referring to an object. Recent cognitive status-informed computational RFS models have been evaluated in terms of goodness-of-fit to human data. But it is as yet unclear whether these models actually select referring forms that are any more natural than baseline alternatives, regardless of goodness-of-fit. Through a human subject study designed to assess this question, we show that even though cognitive status-informed referring selection models achieve good fit to human data, they do not (yet) produce concrete benefits in terms of naturality. On the other hand, our results show that human utterances also had high variability in perceived naturality, demonstrating the challenges of evaluating RFS naturality.

## introduction referring is a critical part of human communication, especially in situated, task-based interactions. humans use a variety of referring forms during reference production, including both definite descriptions (e.g., the red box on the table) and concise referring forms (e.g., this box, that, or it). while more concise referring forms are less informationrich, they allow speakers to express their intentions more quickly, and allow their listeners in turn to more quickly and effectively infer those intentions @xcite . the process of choosing what type of referring form to use, known as referring form selection, is an important first step in the production of referring language @xcite . * the first three authors contributed equally to this work. figure 1 : to investigate the naturalness of referring forms, we conducted a study where participants watched videos of human-human instruction tasks. after each video, participants were shown a referring utterance that could have followed as the next line in the dialogue, and were asked to assess the naturalness of that utterance. yet despite the wide variety of referring forms observed in human-human interaction, and the critical role of referring form selection in language production, most research on generating referring language has focused solely on definite descriptions @xcite . while generating effective definite descriptions is a critical task, a speaker solely relying on this referring form would be an inefficient, unnatural, and annoying speaker. this discrepancy is critical not just for the psycholinguistics community, who seek to understand the cognitive dynamics of language production, but also for the artificial intelligence and human-robot interaction communities, who seek to enable efficient, natural, and humanlike communication in task-based, situated domains @xcite @xcite @xcite . as such, we argue that more attention to the problem of referring form selection is needed across multiple areas of cognitive science. recently, a number of researchers have begun to give the referring form selection problem additional consideration, using a variety of experimental and machine learning research techniques @xcite @xcite @xcite . for example, @xcite examined the features learned by deep learning models of referring form selection; @xcite addressed the ecological validity of the contexts in which referring form selection is studied; @xcite studied how document planning of task instructions could be designed for optimal referring form selection; and @xcite and @xcite studied how models of cognitive status could be used to enable cognitively informed models of referring form selection. this last set of work is of particular interest: @xcite and @xcite leveraged the well-validated @xcite givenness hierarchy theory @xcite , a linguistic theory that captures the relation between different referring forms and the cognitive status of referents in listeners' minds. for example, per this theory, when a speaker uses this, one can infer they assume their target referent to be activated in their listener's mind; when a speaker uses it, one can infer they assume their target referent to be in focus. previous work on cognitive status-informed models of referring form selection have largely been evaluated in terms of fit to human data using objective metrics like accuracy and notably human evaluations of these computational model in live human-robot interactions by @xcite . that is, previous researchers have only assessed whether the referring forms predicted by their models match the referring forms that people actually use in human-human interactions. while assessing fit to human data supports these models as cognitive models, it obfuscates a key dimension of referring form selection: when a human selects a referring form during referring form selection, there is no one "correct" form for them to select. in many contexts, for example, the n' and that n' may be relatively equally appropriate. even when an object is truly in focus, warranting the use of the extremely concise it, the use of the-n is not wrong; and in fact, in some such cases, the use of the-n may be advantageous as it is simply more natural sounding. as such, while cognitive status-informed models of referring form selection have been shown to achieve good fit to human data, (1) high goodness of fit may be an unnecessarily aggressive benchmark, and (2) it is unclear whether the referring forms selected by these models are actually any more natural than those that would be produced if simpler baseline models were used. in this work, we thus compared the naturalness of referring forms selected by cognitive statusinformed referring form selection models (specifically, that presented by @xcite ) to those that would be selected by a variety of baselines, including a random baseline, and a definite description baseline (in which a definite noun phrase the n is always used). to do so, we conducted an experiment in which we modified a dataset of task-based referring expressions, systematically varied the referring forms shown to participants, allowing us to collect naturalness ratings for all possible referring forms that could have been used in those referring expressions. we then consider, for each of those referring expressions, what referring form each of the compared models would have predicted, and thus what the perceived naturalness would have been. averaging these naturalness predictions for each model, we are able to compare the overall naturalness of the considered models. as we will show, our results suggest that even though cognitive status-informed referring selection models achieve good fit to human data, they do not (yet) produce concrete benefits in terms of naturality. but our results also demonstrate the challenges of performing this type of evaluation, as even the utterances produced by humans had high variability in perceived naturality. ## related work arnold and zerkle (2019) argues that linguistic referring form selection models generally fall into two categories: rational and pragmatic. rational models (e.g. @xcite ; frank and goodman (2012)) could explain the use of pronouns from an egocentric perspective, i.e., in terms of their ease of use in conversations. pragmatic models, on the other hand, could explain the use of pronouns from an allocentric perspective, i.e., in terms of the assumptions about interlocutors that lead to their use. these allocentric accounts are typically grounded in theoretical constructs like cognitive status @xcite . although these pragmatic models vary in terms of the constructs they use to explain referring form choice (e.g., givenness @xcite , and focus @xcite @xcite , these models are all centered around the assumption that referring form selection is based on the status a referent has in a conversation or in the mind of conversational participants. while both of these models make important contributions to the literature, neither performs at exceptional levels when it comes to predicting which specific referring forms to use. as arnold and zerkle (2019) pointed out, rational models suggest using reduced forms vastly more often than seen in practice, and fail to predict referring forms that are equally short. furthermore, arnold and zerkle (2019) and @xcite note that both kinds of models focus on individual events or factors, such as recency in conversation @xcite , instead of developing a fully comprehensive model for all of what reference production entails. artificial intelligence (ai) researchers developing referring form selection models have the same problems @xcite @xcite @xcite @xcite . ai rfs models can be broadly categorized as multi-factorial process modeling, where the prediction of referring forms is approached as a problem of classification based on various linguistic and contextual features. much like previous models, those discussed by @xcite and @xcite opt for predicting pronoun use in general, as opposed to predicting the use of specific referring forms. additionally, models like those listed above tend to be trained using purely textual domains (e.g., @xcite 's) that are very different from situated domains. situated domains are highly ambiguous, with large numbers of nearly identical objects, and require speakers to make run-time decisions based on linguistic features (like prosody) and non-linguistic features (like physical distance) that may be assessed using noisy sensors. to fix these problems, @xcite presented dynamic models of cognitive status based on the givenness hierarchy @xcite , which they used to produce cognitive status-informed rfs models. in addition to cognitive status, these models included situated features like physical distance, leading to promising results @xcite . @xcite solved a number of external ecological issues in the task environment that referring form data was collected from, e.g., including repeated and non-present objects. this led to a wider variety of referring forms collected. however, both @xcite and @xcite only evaluated their work in terms of model goodness-of-fit to human data foot_0 ; that is, how well the model predicts the referring forms used by others. while goodness-of-fit is a valuable metric, it obscures the fact that when choosing a referring form, there are often multiple referring forms that might be appropriate. as such, focusing on goodness-of-fit simultaneously risks underselling the performance of a model and overselling the benefits of the model. because of the ecological validity and the wide range of referring forms from @xcite 's model, we used it in this work. ## hypothesis due to the incorporation of cognitive statuses of objects in listeners' mind, we believe that referring forms predicted by cognitive status-informed models will have higher perceived naturalness, compared to random selection and compared to use of definite descriptions alone. ## method in order to evaluate the naturalness of different referring forms, we used a novel experimental design in which we (1) collected naturalness ratings for each of a large set of referring forms across a large set of referring contexts, and (2) used these ratings to determine the overall naturalness of several competing models, by seeing what the naturalness ratings would have been under the referring forms selected by those models across those referring contexts. to collect naturalness ratings, we performed an online experiment in which participants watched a series of videos from a dataset of human-human task based interactions. after each video, the participant was shown, in text, the next utterance spoken in the video, and asked to evaluate its naturalness. we will now step through (1) the specific stimuli used in this experiment; (2) the experimental design that determined how these videos were shown to participants; (3) how we systematically varied the referring forms shown to participants and assessed the naturalness of those referring forms; (4) the overall experimental procedure; and (5) our participant demographics. ## conclusions to go beyond the focus on goodness-of-fit in cognitive status-informed computational referring form selection model evaluation, we conducted a humansubjects study to explore the naturalness ratings of the predictions. surprisingly, results did not reveal an improvement in naturalness over random baselines, and in fact suggest that human perceptions of even human-generated referring forms are incredibly varied and not significantly different from those random baselines. our results suggest several directions for future work, and new technical and