Diversity as a By-Product: Goal-oriented Language Generation Leads to Linguistic Variation (2021)

Abstract: The ability for variation in language use is necessary for speakers to achieve their conversational goals, for instance when referring to objects in visual environments. We argue that diversity should not be modelled as an independent objective in dialogue, but should rather be a result or by-product of goal-oriented language generation. Different lines of work in neural language generation investigated decoding methods for generating more diverse utterances, or increasing the informativity through pragmatic reasoning. We connect those lines of work and analyze how pragmatic reasoning during decoding affects the diversity of generated image captions. We find that boosting diversity itself does not result in more pragmatically informative captions, but pragmatic reasoning does increase lexical diversity. Finally, we discuss whether the gain in informativity is achieved in linguistically plausible ways.

## introduction when speakers converse, for instance, in and about a visual environment, their utterances are remarkably diverse: analyzing a corpus of human descriptions of @xcite find that 99% of the image captions are unique. more generally, it is well known that word usage in language data follows a zipfian distribution @xcite . in this paper, we take a closer look at linguistic diversity in image captioning, following @xcite 's notion of corpus-level global diversity as "the ability to use (many different combinations of) many different words". reproducing the diversity of natural language remains a key challenge in neural generation, despite all progress in recent years. neural generation systems in various tasks, but most notably in image captioning @xcite and conversation * work done while at friedrich schiller university jena modeling @xcite have been found to produce bland, generic and repetitive utterances @xcite @xcite . this lack of diversity in neural sequence-to-sequence models is often attributed to their standard training and decoding objective, i.e. likelihood, and the corresponding decoding method, i.e. beam search, which seems too biased towards highly probable and generic output @xcite @xcite @xcite . a commonly adopted solution is to relax the likelihood objective and sample candidate words during decoding, thereby introducing randomness into the generation process at testing time @xcite @xcite @xcite @xcite . in this paper, we take a different perspective on diversity and argue that it should not result from randomness but from principles of intentional and goal-oriented language use, as formulated by e.g. @xcite or @xcite . in particular, we hypothesize that linguistic variation in image descriptions should arise as a by-product from reasoning about different ways of referring to objects and scenes in coordination with an interlocutor. this builds upon a long tradition of linguistic research showing that speakers consider the pragmatic informativity of their lexical choices @xcite @xcite . for example, the more specific word "collie" might be preferred over the more common word "dog" when speakers need to unambiguously identify an entity in a context with other, similar entities @xcite . hence, in different contexts, the same types of entities could be described differently, resulting in higher diversity when considering all generated utterances. with this in mind, we investigate whether linguistic diversity is triggered by simulating pragmatic objectives during the decoding of neural language models. we use recent approaches from discriminative and pragmatically informative captioning @xcite that generate unambiguous descriptions of a target image in the context of distractor images and compare them to sampling-and search-based generation. to the best of our knowledge, no detailed comparison has yet been made between decoding strategies maximising diversity on the one and informativity on the other hand. we assess the effect of decoding along three dimensions: (i) likelihood, i.e. overlap with ground-truth captions, (ii) lexical diversity as in @xcite and (iii) pragmatic informativity measured in terms of the performance of a pre-trained image retrieval model @xcite . we show that neither sampling methods nor beam search lead to higher pragmatic informativity compared to a greedy baseline, despite the higher diversity or likelihood to annotated ground-truth captions. conversely, however, incorporating pragmatic objectives leads to increased diversity. finally, we show that even simple pragmatic constraints lead to variation which is linguistically plausible. ## background criteria for high-quality and human-like descriptions of images have been discussed much in work on image captioning, pragmatics and dialogue. besides conformity with ground truth annotations, suggestions include, for example, that descriptions should exhibit human-like diversity, sufficiently distinguish their target image from others and exhibit human-like strategies for referring (e.g. @xcite @xcite @xcite . diverse outputs are desirable in both open-ended dialogue and more constrained tasks like image captioning @xcite , and needed for, e.g., generating entertaining responses in chit-chat dialogues @xcite , responses with certain personality traits @xcite , or accounting for variation in referring expressions @xcite . in neural image captioning @xcite , various approaches have been presented to generate more diverse captions (e.g. @xcite @xcite @xcite @xcite @xcite @xcite . @xcite describe different decoding methods for increasing diversity in image captioning, e.g. diverse beam search @xcite or sampling from sets of candidate tokens. not all methods are applicable in our setting, since the authors focus on local diversity, i.e., generating diverse sets of descriptions for individual stimuli @xcite . hence, for this group of methods, we focus on the widely used sampling approaches top-k @xcite and nucleus sampling @xcite , cf. section 3.2. apart from diversity, recent work focused on generating more specific, accurate or detailed, yet (more or less) neutral descriptions @xcite @xcite . other works have extended the task to pragmatically informative captioning, given a specific context @xcite @xcite . here, neural captioning models are trained on standard image description datasets and decoded, at testing time, to produce captions that discriminate target images from a given set of distractor images. this setting, which we adopt for our evaluation of pragmatic informativity, is very similar to the referring expression generation (reg) task @xcite @xcite . in our experiments we use the methods proposed by @xcite and @xcite (adapted to word level decoding), cf. section 3.3. to the best of our knowledge, recent work on pragmatics in neural generation has not looked explicitly at lexical diversity, although the ability to use a rich, human-like vocabulary and control lexical choice seems an important prerequisite to being able to discriminate a referent in a given context @xcite . inversely, most of the literature on diversity in image captioning does not explicitly analyze the underlying linguistic phenomena that cause diversity in image descriptions. however, some work discusses whether increased diversity facilitates the selection of the corresponding referent image from a large number of potential targets @xcite @xcite . in particular, @xcite bears certain similarities to our work, as the authors suggest that more specific captions lead to higher diversity. we differ from this line of work in the following aspects: a) we focus on the decoding stage, b) our approach is linked more closely to pragmatic theory, as we generate captions that are not more specific in general, but more informative in a particular context, and c) we examine the relationship between informativity and diversity in more detail by systematically varying the contextual pressure through rationality parameters and inspecting further properties of the resulting captions. ## decoding methods a large number of decoding strategies for neural nlg has been developed recently (cf. section 2). we focus on several representative decoding methods that target conceptually very different aspects of language use: likelihood, diversity and pragmatic informativity. these dimensions will be the basis of our analysis, as reflected in our evaluation criteria (see section 4). technically, the decoding methods are very generic and should be compatible with most neural nlg models. ## linguistic strategies in pragmatic decoding the results discussed above show that pragmatic reasoning during decoding results in both increased diversity and informativity of captions. this suggests that the phenomenon of linguistic diversity can be integrated, at least to some extent, into well-established theories of intentional and goaloriented language use @xcite . figure 1 shows two different ways, in which variation of literal image descriptions leads to higher informativity: re-conceptualizing and re-describing entities mentioned in the literal caption in a way that distinguishes them from similar entities in distractor images, or describing further objects and elements, which are present in the target image but not in the distractor images. changing "clock tower" to "tall clock tower" can be seen as refining the description; switching "desk" to "office cubicle" as re-conceptualizing parts of the scene in favour of more informative categories. the inclusion of "trees in the background" states an example of additional distinctive elements. in human annotations, the informativity of unambiguous referring expression is achieved e.g. by increasing lexical specificity or adding descriptive modifiers @xcite . to explore those strategies in our captions, we measure the average distance of generated nouns from the wordnet root, as a rough approximation of specificity, and accumulate the pos tags for the generated captions, both using off-the-shelf models from the spacy library. the results are shown in regarding lexical specificity, beam search appears to generate more general nouns in comparison to the greedy baseline. in contrast, samplingbased methods lead to a more specific vocabulary. however, neither does this specificity translate to improved retrieval results (cf. section 5.2), nor does changing the parameters seem to have much impact. for es and rsa, higher α or lower λ settings systematically lead to a higher specificity for nouns, as well as improved retrieval results. the average specificity for rsa with low rationality is surprisingly low, which could be due to the beam search scheme in which reasoning is integrated. whereas there doesn't seem to be a systematic relation between rationality and the ratio of nouns and verbs, we observe a higher ratio for adjectives if rationality is increased. however, we should note that e.g. es (λ = 0.3) generates more ungrammatical sentences, which may affect the pos tagger. also, this extends to sampling-based methods, where more adjectives are produced if the parameters are tuned towards higher diversity. taken together, the higher average specificity of nouns and greater proportion of adjectives are consistent with the linguistic devices described by @xcite . although future work should explore this in more detail, this suggests that linguistic variation in es and rsa corresponds, at least to some degree, to plausible strategies for achieving communicative goals. ## discussion and conclusion our findings show that pragmatic reasoning in neural generation adds an interesting dimension to the analysis and modeling of lexical diversity in neural image captioning. although not aiming at diversity itself, es and rsa lead to linguistic variation through simulated coordination with interlocutors, which in turn leads to increased lexical diversity (section 5.1). whereas this variation translates to improved informativity, this is not the case for sampling-based methods like nucleus and top-k sampling (section 5.2). further exploration revealed that discriminative decoding results in a higher rate of generated adjectives and a higher average specifity for nouns (section 6), resembling linguistic strategies found in human annotations @xcite . therefore, pragmatic reasoning leads to linguistically meaningful variation, resulting in higher informativity due to linguistically plausible devices, and, from a global perspective, increased diversity. in this regard, linguistic diversity arises naturally from conversational goals and adaptations to contextual constraints. we see great potential for future work in exploring linguistic variation in tasks related to and going beyond image captioning. first, the human annotations used here were produced in a relatively neutral communicative context. hence, they differ from generated captions in terms of their communicative purpose and possibly do not reflect the full range of variation that speakers might use in more challenging tasks. thus, similar studies could be made on e.g. referring expressions @xcite or other datasets that record longer interactions centered on images @xcite . second, as discriminative image captioning captures only partial aspects of natural conversation, it could be investigated whether our findings apply to other dialogue tasks. finally, other sources of variation should be considered, e.g. formality or individual characteristics of speakers @xcite .