Resolving Indirect Referring Expressions for Entity Selection (2023)

Abstract: Recent advances in language modeling have enabled new conversational systems. In particular, it is often desirable for people to make choices among specified options when using such systems. We address the problem of reference resolution, when people use natural expressions to choose between real world entities. For example, given the choice ‘Should we make a Simnel cake or a Pandan cake¿ a natural response from a non-expert may be indirect: ‘let’s make the green one‘. Reference resolution has been little studied with natural expressions, thus robustly understanding such language has large potential for improving naturalness in dialog, recommendation, and search systems. We create AltEntities (Alternative Entities), a new public dataset of entity pairs and utterances, and develop models for the disambiguation problem. Consisting of 42K indirect referring expressions across three domains, it enables for the first time the study of how large language models can be adapted to this task. We find they achieve 82%-87% accuracy in realistic settings, which while reasonable also invites further advances.

## introduction natural dialog often requires resolving referring expressions (res), not only within and across texts, but also for grounding natural language expressions to specific entities or images. we focus on a specific conversational setting where a speaker's utterance intends to disambiguate between known named entities. while many aspects of re resolution have been studied extensively, past work has focused on pragmatic reasoning @xcite , influence of discourse @xcite , and multimodal (e.g., image) context @xcite ). 1 our dataset can be found at https://github.com/ google-research-datasets/altentities did you mean a simnel or pandan cake? it looks surprisingly green in color without any frosting or fruit it is made from some leaf comes from indonesia isn't the easter one in the specific case of dialog, when people make choices, the natural res are not always item names, spatial locations or attributes present in the question. for instance when the choice is among items with similar names (perhaps disambiguating automatic speech recognition errors), or items with difficult to pronounce names, or where the user does not even recall which name is correct but instead recalls some higher level attribute, the user may choose an indirect expression ( in this work, we created the altentities corpus by a multi-step process, soliciting crowdworkers to provide diverse yet realistic natural expressions for selecting entities in three domains: books, recipes, and music. to obtain natural and casual dialogic language, we introduce a novel cartoon-based annotation approach (figure 1 ). altentities consists of 6,247 alternative questions (presenting two entities) along with 42,529 res. in this context, res are typically definite noun phrases with a pronominal head and a restrictive relative phrase or one of its reduced variants. our experiments are based on fine-tuned bert @xcite and t5 @xcite lms. we assess the representation of entity names as well as other sources of entity information. we find that the results depend significantly on the type of entity information provided to the models alongside the res: if a lm only has access to the entity names but no other information, a case that might happen especially for long tail entities, accuracy is around 60%. on the other hand, if a lm is (unrealistically) given entity information that is identical to that shown to annotators producing the res, accuracy is very high (up to 95%). however, if the model (more realistically) only has access to generic information that may or may not overlap with annotators' knowledge (section 5), accuracy of our models is only 82%-87%, leaving significant room for methodological improvements. our work adds to recent efforts to allow users to speak more naturally to conversational systems. here, we present the most related studies focusing on the properties of res as well as their resolution. alternative questions. our questions belong to the class of alternative questions (e.g. 'are you staying or leaving?'). several studies have focused on the form and semantics of such questions, and differences from yes/no questions particularly on the basis of prosody @xcite @xcite . this paper focuses on the deep understanding of answers to such alternative questions when they are posed for selecting between two entities. speaker-listener cooperation. the research in this space follow the rational speech act theory @xcite , where the way speakers and listeners reason about each others' intentions and beliefs explains which attributes speakers pick to describe an entity, and how listeners disambiguate the entity. @xcite ; @xcite focus on the pragmatic reasoning involved during the conversation which helps in reaching a common understanding of the topic. wilkes-gibbs and clark (1992) study how res change as the conversation proceeds. in an experiment, they show that participants start from long and indefinite descriptions of images, but end up with short and definite references. jordan and @xcite study the subproblem of content and attribute selection for generating object descriptions. in our data collection, we assume a conversation between two humans in three dialog turns, where the first two turns prime the re produced in the last turn (section 3). common ground. in addition to the interlocutors' intentions, their prior or shared knowledge also plays an important role in how they understand each other's utterances. sometimes the common knowledge arises from a shared situation, e.g., in navigation dialog @xcite @xcite or the presence of a visual space @xcite . in the latter, the common ground is given, i.e., it is assumed the image is what all participants in the interaction see in the same way. in many other situations, e.g., in a dialog between two friends about a movie or a book, the common ground is hidden and we can only make assumptions of what information participants share. in this work, during data collection, we assume that annotators have access to rich common ground involving multiple modalities such as text, image, and video (section 3.3). during model training inference, we explore performance with varying levels of background information (sectoin 5.2). implicature understanding. this paper advances the broad area of understanding implicature in dialog. for example, a few recent papers developed datasets and models for indirect boolean responses (without saying 'yes' or 'no') @xcite @xcite . interestingly, @xcite shows that llms cannot solve such implicatures in a zero-shot setting. ## collecting rich referring expressions to maximize generalizability, we collect data in three domains: books, recipes, and music. these were selected to cover a diverse variety of entity types with different kinds of available information -e.g. plot summaries for books, images for recipes, and lyrics and videos for songs. we performed careful and detailed annotations, and explain the annotation steps in this section. ## the altentities corpus our annotations were carried out using a pool of around 60 in-house crowdworkers. 5 they were all native english speakers recruited from u.s., u.k., canada, and australia so as to obtain a diverse set of perspectives. 6 each question was shown to two workers to get multiple inputs per question. around 2k entity pairs were annotated for each domain resulting in around 42k expressions in total. the res for books were on average a word longer than for other domains. they also contained more named entities per expression. each domain contains some repeated res (e.g., the pop song), that are often high-level responses, e.g., a song's genre. the books domain contains the most unique responses. the number of contrastives, estimated as res starting with "not the", are from 8% in music up to 20% in books. 7 for music and recipes, we manually checked 200 random res for references to modalities other than text. around 10% multi-modal res were present in the recipes domain (mostly color), and 20% in the music domain (mostly beat, speed, and mood). we estimated the re error rate by manually inspecting 40 question samples (around 250 to 300 expressions) per domain. the error rate is between 4.5% to 6.8% for the three domains. 78% of these errors were due to the re applying to both items, not just the target entity. the remaining errors were mostly due to confusing the two entities. we also the annotators were inspired by the provided stylistic cues in the instructions (e.g., starting with the one or i meant the), but followed our guidelines to vary their responses as well. we observed that the content of res (e.g., timeline, lyrics, singer or band information, instrument) included both the categories covered by the provided examples (e.g., timeline for books and songs) and novel categories (e.g., background information on books and songs such as the one inspired by a rolling stones song). ## task and models indirect reference resolution can be defined as follows: given an alternative question with k choices foot_6 @xmath0, . . . , c k }, and a re r, models should disambiguate the choice c * ∈ c intended by r. we assume r does not directly mention c * by its name or position, but does uniquely refer to c * . ## experiments we split the questions in the altentities corpus in each domain into training (70%), development (15%), and test (15%) sets. to avoid information leaking between the sets, we allow each target item to be in only one of the sets. for the use and entailment models, we do not tune any hyperparameters. for supervised models, we tune the learning rate, batch size, and number of epochs using a grid search on the development data (96 configurations for bert and 24 configurations for t5). we report the hyper-parameter details in appendix d. ## conclusion we have revisited re resolution with a new focus on indirect expressions, introducing altentities, a new large dataset for this task -covering books, recipes, and music examples. the dataset was collected using a novel cartoon completion approach to encourage conversational and causal expressions while avoiding name or position expressions. the experimental results show that in a realistic setting, lms adapted for this task achieve 82%-87% accuracy. while an improvement on existing approaches, this also encourages further research on this important problem. moreover, we showed that the models' performance does not drop when trained and tested on different domains, suggesting that models can learn the semantic task well and generalize to new domains. it is notable that in practice, many entities do not have textual descriptions or rich meta-data. future research could study resolving res with minimal information, e.g., when we only have access to their names or limited meta-data. future research could also use multi-modal input for training and inference. further, to handle more complex res such as the newer one, or the happy song, one could decompose a re into simpler expressions and then perform the comparison. similar data collection methodologies could be applied to collect a dataset with more number of choices and also cases where neither or multiple choices match the re.