How Useful is Context, Actually? Comparing LLM s and Humans on Discourse Marker Prediction (2024)

Abstract: This paper investigates the adverbial discourse particle actually. We compare LLM and human performance on cloze tests involving actually on examples sourced from the Providence Corpus of speech around children. We explore the impact of utterance context on cloze test performance. We find that context is always helpful, though the extent to which additional context is helpful, and what relative placement of context (i.e. before or after the masked word) is most helpful differs for individual models and humans. The best-performing LLM, GPT-4, narrowly outperforms humans. In an additional experiment, we explore cloze performance on synthetic LLM-generated examples, and find that several models vastly outperform humans.

## introduction natural human language utterances can be described as containing different levels of information. the most obvious is the main message or topic of the utterance, but speakers often also aim to convey information that is about the main message, e.g., reflecting their beliefs about or stance on the message, and its relation to other utterances in the discourse @xcite . this pragmatic information is an essential component of human linguistic interactions. with the recent advent of highly capable large language models (llms), it is also becoming a key focus in research on computational language generation. in this paper we focus on the english adverbial discourse marker actually, a word with pragmatic functions. actually serves to 1) highlight unexpectedness by conveying contrast and contradiction @xcite @xcite , and 2) express reality, truth, certainty, and evidentiality @xcite @xcite . we expect these functions to result in statistically robust properties in the surrounding linguistic context that could potentially allow an llm to correctly predict a missing actually. our experiments examine how the relative placement of contextual information affects prediction success of a variety of llms. here we provide models context in the form of preceding and following utterances. we also compare the llms' performance to that of humans on the same task. to evaluate prediction success, we utilize standard cloze tests, which consist of masking a word in an utterance or sequence of utterances and asking a model/human to predict the missing word. our cloze test items are drawn from the providence corpus @xcite of transcribed everyday conversational speech around children, which represents an under-explored text-type in computational studies. we recruit human participants through the mturk platform @xcite and compare their performance to artificial language models: gpt-3.5 and gpt-4 @xcite , bert @xcite , roberta @xcite and electra @xcite . humans' guessing accuracy on cloze-type tasks has been shown to improve with increased surrounding context and to depend on the placement of that context @xcite . we, therefore, vary the amount and type of context available in the cloze examples. our results show that gpt-4 prediction performance is on par with (in fact higher than) human subjects' performance, echoing very recent findings @xcite on other pragmatic language processing tasks. performance depends crucially on surrounding utterance context. some amount of context is always helpful though different models and humans benefit from different placement of context. for humans and the highest-performing model, gpt-4, preceding context is more helpful than following context. in an additional experiment, we compare cloze performance on examples sourced from the provi-dence corpus to synthetic examples generated by gpt-3.5. surprisingly, while human and model performance is quite similar for the corpus-sourced examples, three of the models vastly outperform human subjects on the synthetic examples. this result corroborates earlier findings that llm-generated data differs in crucial ways from natural data @xcite and that llms demonstrate a preference for synthetic text @xcite . related work pragmatic llm language use is an active research area. @xcite investigate llm capacity for pragmatically motivated interpretations, finding that humans and models utilize similar cues for pragmatic language use. sravanthi et al. (2024) present a benchmark of ten pragmatic language use tasks, showing that llms achieve near comparable performance with human subjects on many tasks. our findings lend additional support for this result. lake and murphy (2023) raise two important points: 1. current models are strongly linked to text-based patterns and 2. if the aim is human-like language capacity, models should benefit from context in similar ways to humans. we address these observations by targeting (transcribed) spoken language and investigating the impact of context. several studies investigate llm cloze test performance. @xcite compare cloze test performance of bert and lstm language models @xcite . @xcite compare lstms to human subjects on quantifier prediction in context, observing that humans benefit from broad context, while models do not. our findings do not agree with this observation because we found that models also benefit from broad context. this possibly reflects differences between lstms and llms. closely related to our approach, @xcite investigate llm cloze performance on discourse markers, finding that model performance does not mirror humans on causal connectives. however, in contrast to our approach, they force models to choose a completion from among a set of 66 discourse particles. we instead allow models and humans to freely generate the masked word. we believe our approach to be preferable because artificially restricting the pool of answers makes the task substantially easier. this complicates interpretation of the experimental results and risks inflating performance for models and/or humans. all actually-containing utterances in our dataset are spoken by adults, but many context utterances are child speech, and children are always present or nearby. 70% of target utterances (utterances containing actually) are directed at children while the remaining 30% are adult-directed, with the result that the speech style is best described as "speech around children". preceding and following utterances could be spoken by the same speaker as the target utterance or by a different speaker. in addition to the actually examples, 37 distractor examples containing one of three other words, drawn from the same corpus, are included in the dataset. see appendix a for additional details. in our experiment with human participants (conducted using amazon mechanical turk @xcite , participants control how many cloze items they complete. because actually is always a possible answer, there is a risk that participants will realize that it is the intended completion regardless of the example. to counteract this possibility, we generated 295 synthetic examples using gpt-3.5. these are engineered to resemble the actually examples, but target a variety of other words (selected based on which word in the synthetic target utterance was predicted with the lowest confidence by bert). see appendix a for additional details. cloze test to investigate the effect of surrounding context, we created four examples for every actually utterance, each accompanied by some combination of the preceding and following context utterance as demonstrated in figure 1 . the four conditions were: t (target utterance only); t+n (target plus next utterance); p+t (preceding utterance plus target); and p+t+n (preceding, target and next). in total, this results in 2360 examples of which 1180 are actually examples and the rest synthetic. we asked human subjects to fill in the "5 most likely english words" for the target utterance. we chose to ask for five answers because there are normally several reasonable answers for any given example, and overly limiting the number of responses (say, to one or two) would mean our dataset would fail to include words that participants might believe are equally likely. on the other hand, asking for more than five responses could make the task too difficult and time-consuming. all examples and all conditions were randomized such that any given participant received a random mix of actually and synthetic examples across a random distribution of conditions. each example was only completed by one participant (although participants were free to complete as many items as they wanted). in order to ensure the validity of the results, we limited participants to those completing the task in an englishspeaking country, and we removed responses which contained high proportions of repeated answers or answers that occurred in the prompt. in total, there were 255 mturk participants. given that there were 2360 examples in total, each annotator annotated 8.1 examples on average. see appendix c for additional details. we experiment with two types of llms: 1) encoder-only large language models: bert, roberta and electra, and 2) generative large language models: gpt-3.5 and gpt-4. for encoder-only models, which are trained on a masked language modeling objective, we can straightforwardly frame the cloze task as masked prediction. we extracted the five most probable words in the given context. for generative models (gpt-3.5 and gpt-4), we cannot directly frame the cloze task as masked prediction. instead, we prepared a prompt which asks the model to predict the missing word in the example (see appendix b for details on the prompts and generation process). for both models and humans, we evaluate recall@5, i.e., we computed how often the correct word is found among the five completions. ## results experimental results for human subjects and llms are presented in figure 2 . the information is shown in tabular format in actually examples across all settings, humans do well in comparison to most models on actually prediction but gpt-4 outperforms humans in all settings apart from t, where only the target utterance is provided as context. in general, context is helpful for both models and humans, but the effect of quantity and placement differs across humans and different models. purely based on the target utterance (t), human recall@5 (0.291) narrowly beats both gpt-4 (0.237) and roberta (0.224), while bert, gpt-3.5 and electra deliver substantially lower performance. in the presence of additional supervision in the form of the following utterance (t+n), the general picture remains largely unchanged, although roberta and gpt-4 now narrowly outperform human annotators. providing the previous utterance as context instead of the following one (p+t), results in a substantial boost in recall for gpt-4 (+0.08) and gpt-3.5 (+0.06), while changes for humans and other models remain small. in this setting, gpt-4 clearly outperforms all other models and humans. finally, full context (p+t+n) delivers the best performance for gpt-4. in this context, humans' recall@5 gains +0.04 and ends up close to, though still slightly below, gpt-4. ## discussion both models and humans achieved moderate success in the cloze task for actually and notably higher rates of success on the synthetic examples. humans' ability to predict actually in a variety of contexts generally fell within the range of accuracy of the best-performing models. this suggests that the models were able to generalize to the type of speech from which we drew our examples (largely comprising child-directed speech). gpt-4, in particular, outperformed humans. this is not wholly unexpected given that it is one of the largest llms to date. context is generally helpful. the preceding context utterance seems to be crucial-the best performance is always achieved either in the condition p+t or p+t+n. for the actually examples, the generative models gpt-3.5 and gpt-4, along with humans, derive the largest gains in accuracy from added context, while other models saw smaller improvements. this might indicate that a generative training objective better helps models condition on contextual information in a human-like way compared to a masked language modeling objective. on the other hand, gpt-4 outperforms humans, so it is in fact using context more effectively than our human subjects. interestingly, although roberta sees little improvement with context, it nevertheless outperforms gpt-3.5 in all conditions. ## conclusion and future work in our data, the performance of models and humans fell within the same range. this suggests that models, especially those performing closest to humans, are able to predict the occurrence of the pragmatically-sensitive item actually, possibly based on similar aspects of the surrounding context as humans. however, important differences emerged between models and humans in overall accuracy, use of context, and the effect of context location, suggesting differences in how, and how effectively, models and humans utilize context. our work raises some important questions: where models outperform humans, are they picking up on contextual cues that humans are not sensitive to? if so, what are these cues? is outperforming humans a desirable goal, or is emulating human behavior-interpreting contextual cues in a humanlike way, including failing to make use of certain cues even if they might be useful-more aligned with the goals of language modeling? finally, our experiments on synthetic examples demonstrate a stark contrast between llm performance on natural and synthetic data. consequently, we urge caution when using synthetic data in experiments, especially when comparing human and llm performance.