How can large language models become more human? (2024)

Abstract: Psycholinguistic experiments reveal that efficiency of human language use is founded on predictions at both syntactic and lexical levels. Previous models of human prediction exploiting LLMs have used an information theoretic measure called surprisal , with success on naturalistic text in a wide variety of languages, but under-performance on challenging text such as garden path sentences. This paper introduces a novel framework that combines the lexical predictions of an LLM with the syntactic structures provided by a dependency parser. The framework gives rise to an Incompatibility Fraction . When tested on two garden path datasets, it correlated well with human reading times, distinguished between easy and hard garden path, and outperformed surprisal.

## introduction psycholinguistic research develops models of human language understanding using experimental techniques such as self-paced reading and eyetracking. natural language processing research develops algorithms that enable machines solve human language tasks. novel lines of research bringing these two fields together have emerged, where a question of interest has been whether machines are able to process language in ways similar to humans. the goal of this paper is to show that the answer can be yes, but only when they are equipped with human capabilities that enable them to predict with a combination of both syntactic structure and lexical statistics. in order to model these characteristics, one needs a computational framework with at least two levels (more if we take pragmatics and other language features into account). we work with presheaves and specific instances of them, which consist of (1) a base that models linear structure, and (2) data that encode the statistics of different interpretations of the base. the data can be manifold recording outcomes of events, which can themselves be binary or many-valued, and their probabilities. for these reasons, presheaves provide a good candidate framework for modelling features of human language understanding. we use a simple topological space as the base of our presheaf: that of a pre-ordered set. the elements of this set are sub-phrases of a sentence. the pre-order relation over the elements is the prefix relation between the sub-phrases. this relation will be used to the represent the incrementality of the parsing process. our data is the probabilities of syntactic structures of sub-phrases. first, we obtained completions and their statistical information from the predictions of the large language model gpt-2. then, to get the syntactic structures of the sub-phrases, we use the dependency parser spacy. our sheaf theoretic framework gives rise to a schematic fraction that measures how incompatible is the syntactic probability of a phrase from its completions. we refer to this fraction as the incompatibility fraction (if). well known distance measures between probability distributions exist and can be used when instantiating if; we worked with kullback-leibler divergence (kl), jensen-shannon divergence (js), and a measure similar to earth movers (em). deep learning algorithms, especially attentionbased ones, have made impressive advances in predicting the next words of a sentence. a statistical quantity known as "surprisal" has been found to correlate with human reading times @xcite @xcite . this, however, has only been the case for naturalistic text such as news paper articles. the jury is still out regarding a class of challenging sentences known as garden path (gp) sentences @xcite @xcite . psycholinguistic research has shown that humans experience processing difficulty and show longer reading times when processing gp sentences. further, different types of syntactic ambiguities have been shown to result in different levels of processing difficulty @xcite . so far, surprisal has not been able to accurately predict the human reading times of gp sentences and more importantly has not been able to distinguish between easy versus hard sentences @xcite @xcite . in order to test the applicability of our framework, we tested it on two gp datasets @xcite , with hard (i.e., subordinate clause) and easy (i.e., complement clause) ambiguities. both datasets had a disambiguated control for each of their gp sentences. they also had variants of them which were either semantically plausible or implausible. if was measured for all these sentences and its predictions compared with with human reading times and surprisal. all of the instances we worked with, i.e. kl, js, and em, correlated well with human reading times and had very low errors, predicted the differences between gp sentences and their disambiguated controls well, could distinguish between easy and hard garden path, and outperformed surprisal. on the semantic front, all the measures including surprisal validated one of the hypotheses, that a semantically implausible sub-phrase take longer to read. the other hypothesis was about shorter gp effects in implausible sentences, which could not be detected by any of the measures. dealing with these needs an explicit encoding of the semantic structure of sentences and we believe presheaves can also help. working out the details is left to future work. ## related work inspired by applications of information theory to psycholinguistics @xcite , hale argued that suprisal is a good measure for the cognitive load faced by humans during sentence processing @xcite @xcite . surprisal measures the degree of unpredictability of a word w given its prefix context w 1 • • • w n and is computed via the following formula: hale argued in favour of the use of surprisal in incremental parsing procedures. building on this, @xcite and later smith and @xcite showed that surprisal can also model the cognitive load modelled by constraint-based theories. the focus of hale's work was on gp sentences, but he only provided experimental data for a couple of examples. large scale validations on large datasets @xcite and eleven different languages from five different language families followed suit @xcite . these only considered naturalistic text such as wikipedia and news articles. large scale data for gp sentences were not taken into account until more recent times @xcite @xcite , where it was found out that surprisal does not provide good correlation. this has been the case for the surprisal computed over either syntactic predictions of a probabilistic parser or the lexical predictions of a statistical language model. in either case, the predictions largely underestimated human reading times. weighted combinations of the syntactic and lexical surprisal were also computed but still underestimated @xcite . another drawback of surprisal is that it has been unable to distinguish between easy and hard gp sentences. much of the original work on gp sentences focused on structural ambiguities. here we have the original work, insights and examples of bever @xcite , which was followed by the indepth analysis of frazier @xcite @xcite . later work brought the role of semantics into the forefront. since humans process language incrementally, it was expected that the existence of relevant semantic information would increase the speed of recovery from a local ambiguity. in this regard, @xcite ; @xcite studied the role of referential information, @xcite worked on the tenses of the verbs, and pickering and traxler (1998) on the lexical information encoded in sentential sub-phrases such as subject-verb and verbobject. most of this work has only been verified by psycholinguistic experiments on human subjects, but some of it was also verified using statistical machine learning methods such as clustering @xcite . presheaves and sheaves are general mathematical models introduced to formalise and reason about abstract notions of global consistency. they originate from the work of jean leray @xcite , whose aim was to study partial differential equations from a purely topological perspective. subsequent work then extended the use of sheaf theory to other areas of mathematics, such as algebraic geometry @xcite @xcite and logic @xcite . more recently, sheaves and presheaves have been applied to formalise the consistency of different forms of concrete data. here we have examples of data coming from quantum mechanics @xcite , signal processing @xcite , graph neural networks @xcite , and natural language @xcite @xcite @xcite . notably, measures similar to if were developed for physical experiments to compute the amount of unsharpness of experimental data @xcite . these were preliminarily also tested on linguistic data, e.g. for the interpretations of phrases with semantic and anaphoric ambiguities @xcite @xcite . a recent paper explores their applicability to ambiguities arising in garden path sentences but does not consider the general case nor the range of instantiations we offer here, works with the masked feature of bert and has not been tested on semantic plausibility @xcite . ## methodology we use topological spaces and their associated data to model the sub-phrases of a sentence and their interpretations. the topological spaces model the relation between the sub-phrases as they are read by a human subject from a piece of text, i.e. incrementally and according to the linear flow of time. this order is also known as the prefix order or the information order. the data associated to each subphrase models the possible different interpretations of each sub-phrase and their probabilities. here, we work with the completions of sub-phrases into a sentence and the probability of their syntactic structures. this is obtained via a combination of gpt-2 and spacy (with transformers). in what follows, we first go over the abstract model, then instantiate it to the concrete data of natural language, finally develop a set of measure that compute the differences between the different interpretations, giving rise to the notion of an incompatibility fraction. ## results and analysis we trained a regression model between human first pass and total reading times for all of the regions in all sentences and each of our distance measures. the individual regression equations, their resulting degrees of correlations and corresponding p-values are presented in in the gpe of hard versus easy sentences are presented in columns "np/z" and "np/s" of so far we have only considered syntactic effects. in order to evaluate whether our model is able to detect some semantic effects, we study the predictions for plausible and implausible sentences. the reading times for plausible and implausible sentences are in tables 3 and 4 . the results in ## conclusions and future work our work highlights the importance of combining syntactic structure and lexical statistics when modelling human language understanding. for syntactic structure we worked with the linear prefix ordering between sub-phrases of a sentence and their dependency structures. for lexical statistics, we worked with sub-phrase completions and their probabilities provided by an llm. an incompatibility fraction was developed to measure the distance between probability distributions of sub-phrases and their completions. we experimented with known relative entropy distances (kl and js) and earth movers, all of which showed a strong correlation with human behaviour in syntactic gp sentences and outperformed surprisal. none of the measures however, neither any of ours nor surprisal, were successful when it came to gp sentences with semantic implausibilities. we believe these sentences are too complex and in order to deal with them, one needs to explicitly model semantic structure. as it is, the predictions of the parser are over shadowed by the probabilities provided by the llm, which predicts very high incompatibility and surprisal for implausible phrases. kullback-leibler has a long history of applications in natural language tasks, e.g. in measuring the semantic content of words @xcite and deriving objective functions for language models @xcite . notably, levy showed that under certain assumptions it equates surprisal @xcite . earth movers has also been applied in natural language processing, e.g. to compute the relationship between a document and its words @xcite and the distance between bilingual lexicons @xcite . the main difference between the modelling part of these works and ours is the measurement events. we work with sub-phrases and their syntactic structures, whereas the other measures only consider word co-occurrence. despite these, we believe there should be a relationship between the incompatibility of two phrases and their degree of surprisal. formalising this relation is work in progress. there are four other directions that we aim to pursue in future work. these are as follows: (i) the focus of the paper was on garden path sentences. more work is required to test the performance of our measures on a wider range of naturally occurring sentences. (ii) the plausibility element of the dataset used in this work may not be representative of the garden-path effect as a whole. we therefore also plan to replicate our results using different datasets, notably the ones of @xcite ) (iii) as structure, we only considered syntax. modelling semantic structure of sub-phrases and sentences, e.g. as agentpatient relations or event structures and/or the thematic information associated with verbs needs to be done. (iv) our framework is by default only forward looking; experimenting with regression and back tracking to model repair and recovery is left to future work.