How do decoding algorithms distribute information in dialogue responses? (2023)

Abstract: Humans tend to follow the Uniform Information Density (UID) principle by distributing information evenly in utterances. We study if decoding algorithms implicitly follow this UID principle, and under what conditions adherence to UID might be desirable for dialogue generation. We generate responses using different decoding algorithms with GPT-2 on the Persona-Chat dataset and collect human judgments on their quality using Amazon Mechanical Turk. We find that (i) surprisingly, model-generated responses follow the UID principle to a greater extent than human responses, and (ii) decoding algorithms that promote UID do not generate higher-quality responses. Instead, when we control for surprisal, non-uniformity of information density correlates with the quality of responses with very low/high surprisal. Our findings indicate that encouraging non-uniform responses is a potential solution to the “likelihood trap” problem (quality degradation in very high-likelihood text). Our dataset containing multiple candidate responses per dialog history along with human-annotated quality ratings is available at: https://huggingface.co/datasets/saranya132/dialog_uid_gpt2 .

## introduction the uniform information density (uid) hypothesis states that humans distribute information in their utterances evenly for optimal communication @xcite . consequently, language generation has benefitted from uid-based objectives and regularization @xcite . specifically, @xcite argued that uid can be optimized for machine translation using beam search. yet, the effect of different decoding algorithms on information density distributions of generated text are unknown, as is uid's broader role in neural response generation in the special case of dialogue dialog history speaker a: hello, how are you today? speaker a: that is cool! i just like to skateboard! speaker b: good. yourself? listening to classical music. speaker b: candidate responses human annotation scores candidate responses can you do a kick flip haha. i love skating too, but i have a truck to keep as well. type reference text nucleus sampling top-k greedy related {3,3,3} {.,.,.} {.,.,.} {.,.,.} furthering {3,3,3} {.,.,.} {.,.,.} {.,.,.} interesting {3,2,2} {.,.,.} {.,.,.} {.,.,.} figure 1: our dataset contains 4 candidate responses for every dialog history, along with human annotations for 3 qualitative measures. models. here, we investigate (i) if different decoding algorithms follow the uid principle, and (ii) if following the uid principle is beneficial for dialogue response generation, and (iii) collect human annotations of qualitative measures for multiple candidate responses to dialog histories generated using different decoding algorithms (figure 1 ) to study the relationship of dialog response quality and uid. we operationalize uid as the variance of surprisal and measure its correlation with automatic metrics (e.g., bleu, meteor, bertscore) as well as human judgments on qualitative measures of response quality and find that adherence to uid correlates negatively with human judgments when the responses have very low/high surprisal. language production in humans. spreading information content evenly in utterances is a marker of optimally strategized responses, and humans follow this uid principle as a means to state their thoughts clearly and to make themselves intelligible @xcite . the probability of a sentence has been associated with the cognitive load it incurs @xcite . as a means to avoid salient variations in the information content (surprisal, i.e., negative log probability) of responses, speakers maintain uid through linguistic choices such as that at the phonetic @xcite , syntactic @xcite and lexical level @xcite . ## discussion contrary to our expectations, we find nonuniformity to be a more desirable property in machine-generated responses. overall, uid scores and surprisal do not correlate with human judgments ( ## limitations while we present a study of multiple decoding settings, we generate all machine responses using the same transformers based model architecture. thus, the presented work does not yet explore individual differences between different model architectures. additionally, due to limited resources we were not able to collect large-scale human annotations across multiple corpora and acknowledge the same as part of future efforts. ## ethical considerations in this work, we collected human annotations on dialogue response quality using mturk. each hit in our mturk study contained one dialogue history and four candidate responses. the annotators could read the history and rate the responses that followed using mouse clicks on their response choices. we provided an additional feedback field for annotators to write comments in. we received very positive feedback on the task from all the annotators who used this feature. there were no restrictions on the minimum or maximum number of examples the annotators had to rate. from a pilot study on mturk, we found the average time to complete one hit to be slightly under 2.5 minutes. after considering the average time required and the task difficulty (expressed to be clearly and easily understood by annotators in their comments) we set the payment amount to @xmath012 per hour. (a) detailed instructions that mturk raters could expand at any time. (b) examples responses for each measure and rating category shown to mturk raters.