Beyond the Bias: Unveiling the Quality of Implicit Causality Prompt Continuations in Language Models (2023)

Abstract: Recent studies have used human continuations of Implicit Causality (IC) prompts collected in linguistic experiments to evaluate discourse understanding in large language models (LLMs), focusing on the well-known IC coreference bias in the LLMs’ predictions of the next word following the prompt. In this study, we investigate how continuations of IC prompts can be used to evaluate the text generation capabilities of LLMs in a linguistically controlled setting. We conduct an experiment using two open-source GPT-based models, employing human evaluation to assess different aspects of continuation quality. Our findings show that LLMs struggle in particular with generating coherent continuations in this rather simple setting, indicating a lack of discourse knowledge beyond the well-known IC bias. Our results also suggest that a bias congruent continuation does not necessarily equate to a higher continuation quality. Furthermore, our study draws upon insights from the Uniform Information Density hypothesis, testing different prompt modifications and decoding procedures and showing that sampling-based methods are particularly sensitive to the information density of the prompts.

## introduction there is currently a growing interest in probing the performance of large language models (llms) on carefully controlled linguistic test suites and experimental datasets to get a deeper understanding of specific linguistic capabilities captured in these models (e.g., @xcite . while a lot of previous work focused on analyzing the syntactic competence of llms (e.g, @xcite , recent studies also started to investigate the abilities of llms on the level of semantics and pragmatic discourse processing. one promising diagnostic for probing discourse knowledge in llms has turned out to be the use of implicit causality (ic) prompts. ic refers to a property of a broad range of interpersonal verbs that exhibit strong preferences for establishing coreference to one of the verb's arguments over the other in explanations. for instance, when asked to provide a continuation after "..." in a sentence like (1), humans display strong nextmention preferences towards the stimulus (he/tom in this case): as the ic bias has been extensively researched in psycholinguistics and psychology across various languages and populations (e.g., @xcite @xcite , investigating this bias in llms has gained significant interest. a range of recent studies investigated llms' predictions of the next mention in examples like (1) and whether these mentions (i.e. pronouns) follow the same coreference biases as can be found in human data (e.g., @xcite @xcite . these studies predominantly indicated that llms are not generally congruent with the human ic bias, which has been interpreted as evidence for llms struggling with certain aspects of discourse understanding (but see @xcite . in this work, we propose that experimentally elicited data of human continuations of ic prompts cannot only be used for analyzing comprehension in llms, but constitutes an excellent basis for analyzing llms' discourse-level generation capabilities, i.e. going beyond the prediction of the next mention. while discourse-level downstream tasks in nlg, e.g. story generation or summarization, are complex and notoriously difficult to evaluate systematically with respect to targeted linguistic capacities of nlg systems, ic continuations provide a well-controlled diagnostic of discourse knowledge and, at the same time, rather simple sentences whose quality can be easily assessed in human evaluation. yet, to date, only little consideration has been given to the extent to which ic continuations generated by language models maintain semantic appropriateness and coherence with respect to the given ic prompts, cf. @xcite to what extent congruency with the human coreference bias is related to the quality and coherence of the generated continuation. we expand prior work on discourse knowledge captured by llms in ic contexts and investigate their ability to generate not only bias-congruent but also sensible continuations. in contrast to most previous studies, we are not only interested in the llms predictions of the first word following the prompt (i.e. the pronoun), but in the quality of the generated sequences and their comparison against human continuations. we would like to emphasize that ic bias can be violated without any loss of discourse coherence. as an example, consider the following generated sequences in (2), where the first sentence is not congruent with the human bias but coherent, whereas the second sentence is congruent with the bias but not coherent: in order to assess the quality of generated ic continuations, we carry out an evaluation study with human ratings of naturalness, coherence, and informativity. we aim to investigate to what extent continuation quality is associated with bias congruency and identify the factors that may influence this interaction. thus, we manipulate two different types of conditions. first, we evaluate the predicted continuations of language models on two types of ic prompts: "standard" ic prompts (such as those in ( 1 ) and ( 2 )) and ic prompts that are extended with adverbial modifiers. second, we evaluate the performance of three different decoding procedures. in the following, section 2 will present background on the set up of our study, section 3 describes the hypotheses of our evaluation experiment and section 4 describes the results. ## experimental setup data. we ground our study on german ic data from bott and solstad, 2021 and use their experimental items to generate german prompts to be completed by the llms. these data also provide us with human-generated sentence continuations for both prompt conditions, offering a valuable reference point for evaluating the model-generated continuations. prompts consist of simple sentences introducing the verb, the verb's arguments and the connective weil ('because') (as in (1) and ( 2 )). to further investigate the effect of causal modification, we designed the following four conditions: as in previous studies, we vary and balance prompts for the names and gender of verb arguments. to ensure that the prompts' informational content remains unaffected by the verb arguments, we deliberately keep the proper names consistent across the conditions. in addition, we also consider the sentiment of the verbs, ensuring a balance between those that are positively and negatively connotated. see ## results naturalness and coherence. generally, when comparing the two language models, gpt-2 and mgpt, the automatic metrics presented in informativity. referring to relation of bias congruency and continuation quality. next, let us consider figure 2 , which depicts the relationship between bias congruency and the human evaluation criteria for each decoding method, aggregated across the two prompt conditions and across the generations of both llms (see figure 12 in the appendix for a visualiziation separated according to conditions). the higher green bars consistently observed across all prompt conditions and decoding methods indicate that each decoding method generated a greater number of bias-congruent continuations than bias-incongruent ones. furthermore, for all three decoding methods, it is evident that non-bias congruent continuations receive lower ratings in terms of coherence, as indicated by the descending purple bars for this criterion. however, a preference for bias-congruent continuations being more natural than bias-incongruent continuations is primarily observed for the diverse beam search decoding method. further, it is interesting to note that although typical sampling tends to generate the most bias-congruent continuations, this does not necessarily result in better scores on the evaluation metrics. these results, thus, indicate that a bias congruent continuation does not equate to a qualitatively better continuation. correlation between automatic and human evaluation. ## conclusion this paper investigated how continuations of ic prompts can be used to evaluate the text generation capabilities of language models, expanding prior work on discourse knowledge captured by llms in ic contexts by investigating their ability to generate not only bias-congruent but also sensible continuations. our study reveals that llms face challenges in generating coherent continuations for relatively simple prompts, highlighting their struggle with discourse-level coherence. further, our findings show that both the decoding method and the information density of the prompt have a substan-tial impact on the quality of generated text, even surpassing the influence of the specific language model used. our results further indicate that modifying the standard ic prompts has a notable effect on the llms' capability to capture the ic bias, depending on the decoding strategy employed. at the same time, the results indicate that a bias congruent continuation does not necessarily equate to a higher quality of the continuation. intriguingly, we observe a surprisingly low correlation between automatic evaluation metrics and human judgments. this poses an interesting challenge for natural language generation, calling for future research to enhance the evaluation methodologies and metrics used in nlg systems. another potential future direction for our study is to expand the scope beyond german data, as this is a major limitation of this work. while ic is known to be cross-linguistically stable, the inclusion of other languages in our investigation should be performed to validate our findings. also, it would be interesting to more extensively investigate the models' strategy for choosing the first word of the continuation (i.e. the pronoun), which may simply consist in selecting the most recently mentioned discourse element. further, it could be valuable to investigate the effects of additional prompt modifications on bias congruency and the quality of continuations. for this, one possible foundation could be the research conducted by @xcite , for instance, where ic prompts are integrated within a larger pre-context, making them arguably a more natural option for evaluating llms compared to the prompts investigated in this study. also, @xcite , for example, investigated ic in the context of relative clauses, which could provide another compelling starting point for further examination of llms in this context. gpt2 mgpt bleu rouge-l bertscore bleu rouge-l bertscore figure 9: diverse beam search & modified ic prompts figure 10: nucleus sampling & modified ic prompts figure 11: typical sampling & modified ic prompts figure 12: relationship between bias congruency and the text continuation criteria for each decoding method and for each prompt condition, aggregated across se verbs and es verbs and across the generations of both llms. bias congruent continuations are depicted in green, bias incongruent continuations are depicted in purple.