Generating Quantified Referring Expressions through Attention-Driven Incremental Perception (2020)

Abstract: We model the production of quantified referring expressions (QREs) that identity collections of visual items. A previous approach, called Perceptual Cost Pruning, modeled human QRE production using a preference-based referring expression generation algorithm, first removing facts from the input knowledge base based on a model of perceptual cost. In this paper, we present an alternative model that incrementally constructs a symbolic knowledge base through simulating human visual attention/perception from raw images. We demonstrate that this model produces the same output as Perceptual Cost Pruning. We argue that this is a more extensible approach and a step toward developing a wider range of process-level models of human visual description.

## introduction modeling the generation of human-like referring expressions in visual contexts is an ongoing challenge in the field of natural language generation (nlg). one key aspect of this challenge is the disparity between how humans and current computational approaches operate at a process-level. humans have perceptual and cognitive limitations; they can not mentally represent visual scenes down to the exact detail of every visual object or collection of objects. thus, people tend to selectively attend to visual scenes in order to acquire enough information to complete their task @xcite . in contrast, current referring expression generation (reg) algorithms generally assume a fullyspecified, symbolic knowledge base @xcite . essentially, these approaches abstract away the process of perception and seek to model patterns of human language primarily at the content selection phase. one task where this approach breaks down is quantified reference expression (qre) generation. @xcite . qre tasks involve referring to collections of visual items by communicating information about the quantities contained in each collection. initial experimental work in human-produced qres showed regularities in responses that were not easily explained by content selection processes @xcite . to illustrate this, consider the two examples of qre problems found in figure 1 . in problem a, it was found that participants favored relative quantity expressions (e.g., "the box with the most circles") over exact number expressions (e.g., "the box with thirty-one circles"). in contrast, participants favored exact expressions in problem b @xcite . to account for this finding, the experimenters appealed to a principle of least perceptual effort. determining the exact number of approximately 30 objects is effortful and time-consuming, whereas determining that there are exactly three objects is quick and requires little effort. thus, the findings can be explained by people generating exact qres so long as determining the exact quantity did not exceed some threshold of effort. this explanation was validated using a computational method called perceptual cost pruning, proposed previously by @xcite . perceptual cost pruning was able to model human qre production by: (1) starting with a complete symbolic knowledge base representing the visual scene; (2) removing facts from the input knowledge base based on a model of the time cost of exact enumeration; (3) using a preference-based referring expression generation algorithm (i.e., the incremental algorithm) on this reduced knowledge base. we call this approach a destructive approach, wherein a full knowledge base is degraded to be sparser. in this paper, we present a constructive approach to modeling the production of qres, wherein a sparse, symbolic scene representation is built up from nothing. specifically, we present an alternative model that incrementally builds a symbolic knowledge base through simulating human visual attention/perception from raw images. we demonstrate that this model produces the same output as perceptual cost pruning. furthermore, we argue that this is a more extensible approach and a step toward developing a wider range of process-level models of human visual description. output in the original study @xcite , participants' qres were annotated as following into different categories, and these annotations were refined in subsequent work @xcite . we use the latest annotation categories, which are: exact number (num) (e.g., "the box with 31 dots"); relative quantity (rel) (e.g., "the box with the most dots"); and absolute description (abs) (e.g., "the box with dots"). the qre generation algorithm presented in this paper is designed to predict which of these categories is included given a particular qre task image. ## a constructive model of qre generation our model of qre generation was developed within the arcadia cognitive framework @xcite . arcadia provides an ideal framework upon which to implement a model of incremental perception and quantified description for the following reasons: (1) attention and its strategic control is the central organizing feature of the system; (2) it provides the representational flexibility necessary for modeling both an approximate number system that supports group quantity estimation and an object-tracking system that supports exact enumeration (i.e., counting); (3) it aims to implement a cognitively plausible model of human visual processing; and (4) it operates in discrete cycles, allowing for modeling the time course of perceptual and cognitive processes. in the interest of space, we omit more precise technical details regarding arcadia models, which can be found in other work @xcite .