Decoupling Pragmatics: Discriminative Decoding for Referring Expression Generation (2021)

Abstract: The shift to neural models in Referring Expression Generation (REG) has enabled more natural set-ups, but at the cost of interpretability. We argue that integrating pragmatic reasoning into the inference of context-agnostic generation models could reconcile traits of traditional and neural REG, as this offers a separation between context-independent, literal information and pragmatic adaptation to context. With this in mind, we apply existing decoding strategies from discriminative image captioning to REG and evaluate them in terms of pragmatic informativity, likelihood to ground-truth annotations and linguistic diversity. Our results show general effectiveness, but a relatively small gain in informativity, raising important questions for REG in general.

## introduction in recent years, neural models have become the workhorses for referring expression generation (reg, e.g. @xcite @xcite , as in other tasks in the vision and language (v&l) domain @xcite . in reg, this was accompanied by a major shift in how the task was conceptualized. classical approaches (e.g. @xcite mostly investigated rule-based procedures to determine combinations of properties that distinguish target objects from distractors, based on knowledge bases of objects and associated attributes @xcite . recent work in reg has shifted to more natural settings (e.g. objects in photographs, cf. figure 1 ), but at the expense of interpretability: since continuous representations have replaced knowledge bases as the input, pragmatic processes in neural reg no longer operate on symbolic properties, but are deeply woven into model architectures and training schemes. decoding and reasoning methods for discriminative image captioning @xcite could represent a middle ground in this regard: during inference, predictions from a captioning model are reranked according to pragmatic principles, boosting contextually informative and inhibiting ambiguous utterances. this offers interesting similarities to traditional reg, as it is carried out through explicit algorithms and targets symbolic representations (e.g. word tokens). discriminative decoding has been shown to be effective for image captioning @xcite @xcite . in this work, we investigate discriminative decoding for reg, adapting the methods from @xcite and @xcite . we compare them to standard greedy and beam search decoding, in terms of informativity, likelihood to ground-truth annotations, and linguistic diversity. we show that discriminative decoding increases informativity and diversity, although the results are less clear than expected. we attribute this, in part, to the way human annotations are collected, highlighting implications for reg research in general. ## background traditional and neural reg in reg, the goal is to generate descriptions for entities, which al-low their identification in a given context @xcite ; i.e. generating expressions with sufficient, but not too much information, following a gricean notion of pragmatics @xcite . in classic work, target and distractor objects were defined in terms of symbolic attributes and associated values (e.g. color -red). the full reg task was conceived as involving different levels of processing, i.e. lexicalization, content selection and surface realization @xcite . however, foundational work in reg has mostly focused on algorithms for finding distinguishing sets of attribute-value pairs, which apply to the target, but rule out distractor objects, such as the incremental algorithm (ia, @xcite . this algorithm iterates over the attribute set in a pre-defined order, selects an attribute if it rules out objects from the set of distractors and terminates when the set is empty. it has been refined, extended and tested in subsequent work @xcite @xcite . in recent years, neural models have enabled reg set-ups based on real-world images @xcite @xcite @xcite @xcite @xcite , representing scenes with many different types of real-world objects. most commonly, neural reg models follow the encoderdecoder scheme and are trained end-to-end. based on low-level visual representations as the input, various aspects of the task are modeled jointly, e.g. lexicalization and content selection. ## discussion and conclusion discriminative decoding is appealing for reg, as it combines traits from traditional reg with neural generation models. our results confirm findings previously reported for image captioning: discriminative decoding decreases likelihood to groundtruth annotations, but increases informativity and diversity. for informativity, the margin of gain was surprisingly low in our experiments. we attributed this, in part, to the high informativity of underlying model expressions. while this is of importance especially in our setup (both es and rsa assume a basis of pragmatically neutral descriptions), the question of whether pragmatic informativity has to be explicitly modelled or is implicitly learned from the data is relevant for reg research in general, and should be investigated in subsequent work. to this end, controlling the impact of pragmatic processing as in discriminative decoding could be a valuable instrument. beyond this, future work should investigate discriminative decoding in reg in more detail, e.g. to see whether pragmatic reasoning leads to the generation of different or more specific attributes.