Decoding, Fast and Slow: A Case Study on Balancing Trade-Offs in Incremental, Character-level Pragmatic Reasoning (2021)

Abstract: Recent work has adopted models of pragmatic reasoning for the generation of informative language in, e.g., image captioning. We propose a simple but highly effective relaxation of fully rational decoding, based on an existing incremental and character-level approach to pragmatically informative neural image captioning. We implement a mixed, ‘fast’ and ‘slow’, speaker that applies pragmatic reasoning occasionally (only word-initially), while unrolling the language model. In our evaluation, we find that increased informativeness through pragmatic decoding generally lowers quality and, somewhat counter-intuitively, increases repetitiveness in captions. Our mixed speaker, however, achieves a good balance between quality and informativeness.

## character-level experiment our experiments compare three different speakers: the literal (or 'fast') speaker s 0 , which simply decodes the language model, the rational (or 'slow') speaker s 1 , which reasons at every time step, and the mixed (or 'fast and slow') speaker s x . ## conclusion in this paper we have replicated @xcite 's approach to character-level pragmatic image captioning and have evaluated it not only with respect to informativeness, but also quality, repetitiveness, and global diversity of the model outputs. our results show various trade-offs in characterlevel captioning: models that are more informative and rational produce captions that are of lower quality and contain more out-of-vocabulary words. in terms of diversity, we find that character-level rsa increases the amount of word repetitions within captions. interestingly, at the same time, it also increases global diversity and leads to a larger vocabulary being exploited by the underlying language model. this analysis fully confirms and extends findings on word-level decoding methods facing different types of trade-offs as a result of additional objectives introduced into the generation process at the decoding stage (e.g., @xcite @xcite . our analysis also shows that these trade-offs can be countered by our mixed, 'fast' and 'slow', speaker that applies reasoning in a simple, wordinitial fashion. future work could explore further ways of controlling when reasoning is needed during decoding and generalize this to word-level decoding. as character-level image captioning is arguably not state-of-the-art, some of the effects that we report in this case study might not generalize to more powerful -especially word-level -models. nevertheless, we believe that the trade-offs observed in this pilot study could be explored in other task that require the generation of long sequences (e.g., image paragraphs, longer responses in an in-teraction) and that the effectiveness of mixed pragmatic decoding might be an interesting avenue for such tasks.