R e ALM : Reference Resolution as Language Modeling (2024)

Abstract: Reference resolution is an important problem, one that is essential to understand and successfully handle contexts of different kinds. This context includes both previous turns and context that pertains to non-conversational entities, such as entities on the user’s screen or those running in the background. While LLMs have been shown to be extremely powerful for a variety of tasks, their use in reference resolution, particularly for non-conversational entities, remains underutilized. This paper demonstrates how LLMs can be used to create an effective system to resolve references of various types, by showing how reference resolution can be converted into a language modeling problem, despite involving forms of entities like those on screen that are not traditionally conducive to being reduced to a text-only modality. We demonstrate large improvements over an existing system with similar functionality across different types of references, with our smallest model obtaining absolute gains of over 5% for on-screen references. We also benchmark against GPT-3.5 and GPT-4, with our smallest model achieving performance comparable to that of GPT-4, and our larger models substantially outperforming it.

## introduction human speech typically contains ambiguous references such as "they" or "that", whose meaning is obvious (to other humans) given the context. being able to understand context, including references like these, is essential for a conversational assistant that aims to allow a user to naturally communicate their requirements to an agent, or to have a conversation with it @xcite . in addition, enabling the user to issue queries about what they see on their screen is a crucial step in ensuring a true hands-free experience in voice assistants. for instance, consider * equal contribution the following interactions between a user and an agent shown in recent large language models (llms) @xcite @xcite have often enabled end-to-end experiences, perhaps even obviating the need of a traditional multi-stage pipeline that includes reference resolution @xcite . there are, however, still several realworld cases where a pipeline is valuable, perhaps even essential, and an end-to-end approach falls short. first, when a framework runs completely on-device (for example, for privacy and efficiency reasons) on a system such as a smartphone that has relatively limited computing power, due to the low-power nature of the system and latency constraints, using a single, large, end-to-end model is infeasible: using a single llm for this task would usually require the use of a large model with long prompts for true end-to-end experiences @xcite . second, consider the case when the model has to integrate with apis, has to consume information from components upstream, or has to provide information to be consumed downstream: while in these cases it is possible to have an end-to-end approach having the llm write api calls @xcite , this often requires a large language model and a complete overhaul of existing pipelines, which might be cumbersome or completely infeasible. third, the use of a focused model would allow for an existing reference resolution module to be swapped with improved versions in a transparent way, while providing improved ability to hill-climb and improved interpretability, by virtue of the system being modular. finally, for the task under consideration in this paper, reference resolution does not include solely conversational references, but also includes the ability to reference an on-screen and/or a background entity that is part of what the user currently perceives in their interaction with a device, but has not been a part of the conversational history that results from their direct interaction with the virtual agent in question. there thus continues to be utility in exploring "traditional" nlp tasks such as reference resolution, despite some of the larger language models being able to handle them implicitly. in this work, we thus advocate the use of (relatively) smaller language models, but fine-tuned for specifically and explicitly for the task of reference resolution. along similar lines, relying on language modeling alone @xcite @xcite has recently shown great promise in being able to handle a variety of tasks @xcite @xcite @xcite , such as causal reasoning, linguistic acceptability, question answering, textual entailment and even coreference resolution: using language models (lms) does exceedingly well on tasks that can be modeled in a sequence-to-sequence fashion. however, the biggest challenge with adopting this technique for the general reference resolution task in the context of a voice assistant lies in resolving references to entities on the screen and using their properties, in other words, getting the lm to, informally speaking, "see". in particular, it is non-obvious how to encode entities on a screen in a manner that is conducive to being resolved by an lm, while also being consistent enough with how conversational entities are encoded to enable the lm to successfully perform reference resolution on both types of entities. in this work, we propose reconstructing the screen using parsed entities and their locations to generate a purely textual representation of the screen that is visually representative of the screen content. the parts of the screen that are entities are then tagged, so that the lm has context around where entities appear, and what the text surrounding them is (eg: call the business number). to the best of our knowledge, this is the first work using a large language model that aims to encode context from a screen. ## realated work and motivation while traditional reference resolution systems have explored conversational and visual/deictic references in great depth @xcite @xcite , resolving on-screen references is a domain that has been relatively under-explored. however, as shown above, conversational agents on a mobile device need to understand references to the screen, and to support such experiences, to be truly natural. on screen references differ from visual and deictic references for several reasons: they tend to be more structured and highly textual, which enables the use of a lighter model to treat this as a text-only problem without a visual component; further, user queries around on-screen elements often tend to be more action-oriented rather than qa based; finally, they use synthetic screens rather than natural real-world images, which are much easier to parse, but whose distribution completely differs from that on which larger pre-trained image-based systems (such as clip @xcite ) tend to be trained. further, jointly being able to perform conversational and on-screen reference resolution has been even less explored, with prior work often focusing on images and graphics @xcite , or ui elements @xcite . vision transformers @xcite @xcite and other pre-trained models have recently gained prominence as a popular first step in tasks that require visual understanding. however, these tend to be trained on natural, real-world images rather than screenshots of on-screen layouts, which have a very different distribution. in addition, these can be extremely expensive to (pre-)train, requiring a very large number of images and several hundred gpu hours (or more). further, they tend to not perform as well on images heavily embedded with text, and dedicated textual understanding approaches @xcite @xcite tend to heavily rely on multiple modules such as bounding box detection and ocr while also relying on good image quality. joint vision+text models are also substantially more expensive with respect to parameters and computational cost. finally, these models would need to parse text to be able to perform function (eg: "call the business number" needs to extract the number associated with the business landline from the raw image), a process which can be complex and compute intensive when bearing in mind that the underlying text and its location on the screen has been referred by the system, and as a consequence can be relatively easily extracted without large, complex models. the most closely related work which we are aware of, and which we consequently use as our baseline, is that of @xcite , an extension of @xcite which deals purely with on-screen references; however, it suffers from several drawbacks, which we address in this work. first, these approaches rely on a dedicated "category module" to deal with type-based references. this module often requires manually on-boarding entities every time a new type is created (a common occurrence in voice assistants, as the supported functionality of the assistant is expanded over time). in addition, such modules often treat each type as distinct, with the similarity of different types ignored. this, in turn, leaves on the table the potential positive transfer that could have happened between semantically related classes (such as "phone number" and "contact") when data is added for one of those classes. this approach is thus difficult to scale to new entity types and use cases. second, these systems rely on the use of hand-crafted rule-based textual overlap features, which require heavy feature engineering and tend not to be robust. in addition, these heuristics often do not account for semantic similarity, and are not able to encode real-world understanding or commonsense reasoning. finally, these methods effectively classify how related each entity is to the query in question independently of all other entities and later threshold them, whereas our current approach directly picks out the most relevant option (or options), while also allowing for no entities to be relevant. our approach thus additionally has the advantage of removing the reliance on a set threshold, while also providing all the functionality supported in the previous approaches. ## task we formulate our task as follows: given relevant entities and a task the user wants to perform, we wish to extract the entity (or entities) that are pertinent to the current user query. the relevant entities are of 3 different types: we pose the task of reference resolution as a multiple choice task for the llm, where the intended output is a single option (or multiple options) from the entities shown on the user's screen. in some cases, the answer could also be "none of these", in which case the model needs to predict "0". to evaluate this task, we check if the predicted set of options matches the ground truth set; in other words, we allow the model to output the relevant entities in any order, i.e. if the ground truth is entities 8, 7, and 4, then we accept any permutation of these three correct entities while evaluating the performance of the model. note that as in ates et al. (2023); bhargava et al. (2023), we assume that entities along with their types come in from an upstream system (for example, through a mechanism involving entity pullers which are able to extract entities in a high recall manner or through a donation from a device app, as in @xcite ). ## datasets our datasets comprise data that was either synthetically created, or created with the help of annotators. each data point contains the user query and a list of entities, along with the ground-truth entity (or set of entities) that are relevant to the corresponding user query. each entity, in turn, contains information about its type and other properties such as the name and other textual details associated with the entity (the label and time of an alarm, for example). for data points where relevant on-screen context exists, this context is available in the form of the bounding box of the entity, and the list of objects surrounding it along with properties of these surrounding objects such as their types, textual contents and locations. note that our data collection follows that of @xcite ; ates et al. (2023); we present an overview here and direct the interested reader to the aforementioned papers for a more detailed description. note also that each dataset below is somewhat representative of one of our tasks of interest (with our synthetic data bucket being used for both conversational and background entity resolution). ## models we compare our proposed model realm, described in detail in section 5.3 below, with two baseline approaches: one based on the reference resolver proposed in marrs (section 5.1), which is non-llm based, and one based on chatgpt (both gpt-3.5 and gpt-4; section 5.2). show me pharmacies near me which one? walgreens 225 rainbow st, san jose, ca 94088 cvs 105 e el camino real, sunnyvale, ca 94087 qwark 1287 hammerwood ave, sunnyvale, ca 94088 call the one on rainbow st. ## conclusion and future work in this work, we demonstrate how large language models, which are typically trained on text alone, can be also be adapted to perform reference resolution to items in an extra-linguistic context. we do this by encoding entity candidates as natural text; we demonstrate how entities that are present on the screen can be passed into an llm using a novel textual representation that effectively summarizes the user's screen while retaining relative spatial positions of these entities. our proposed system is thus able to resolve references in a variety of human-computer interaction settings, such as those involving on-screen, conversational and background entities; we note, however, that our proposed approach focuses primarily on anaphoric and deictic references, and we leave the extension of our system to handle other types of references, such as bridging references, to future work. in addition, we show that realm outperforms previous approaches, and performs roughly as well as the current state-of-the-art llm, gpt-4, despite consisting of far fewer parameters, even for onscreen references despite being purely in the textual domain. it also outperforms gpt-4 for domainspecific user queries, thus making realm an ideal choice for a practical reference resolution system that can exist on-device without compromising on performance. while our approach is effective in encoding the position of entities on the screen, we find that it may not be able to resolve complex user queries that rely on nuanced positional understanding. we thus believe that exploring more complex approaches such as splitting the screen into a grid and encoding these relative spatial positions into text, while challenging, is a promising avenue of future exploration. in addition, in contrast to a critical assumption of our proposed system, not all on-screen entities are textual. while extending this paper to cover onscreen images, graphics and ui elements is beyond the scope of this work, this is certainly another extension that merits further investigation.