S ent S pace: Large-Scale Benchmarking and Evaluation of Text using Cognitively Motivated Lexical, Syntactic, and Semantic Features (2022)

Abstract: SentSpace is a modular framework for streamlined evaluation of text. SentSpacecharacterizes textual input using diverse lexical, syntactic, and semantic features derivedfrom corpora and psycholinguistic experiments. Core sentence features fall into three primaryfeature spaces: 1) Lexical, 2) Contextual, and 3) Embeddings. To aid in the analysis of computed features, SentSpace provides a web interface for interactive visualization and comparison with text from large corpora. The modular design of SentSpace allows researchersto easily integrate their own feature computation into the pipeline while benefiting from acommon framework for evaluation and visualization. In this manuscript we will describe thedesign of SentSpace, its core feature spaces, and demonstrate an example use case by comparing human-written and machine-generated (GPT2-XL) sentences to each other. We findthat while GPT2-XL-generated text appears fluent at the surface level, psycholinguistic normsand measures of syntactic processing reveal key differences between text produced by humansand machines. Thus, SentSpace provides a broad set of cognitively motivated linguisticfeatures for evaluation of text within natural language processing, cognitive science, as wellas the social sciences.

## introduction natural language processing (nlp) researchers and language scientists alike rely heavily on numeric representations of text in order to better understand how machines and humans process language. consider the following text generated by a large pre-trained language model, gpt2: the scientist named the population, after their distinctive horn, ovid's unicorn. these four-horned, silver-white unicorns were previously unknown to science. ## structure and design at the core of sentspace there are features associated with a sentence: f : @xmath0. the core features are organized into three core modules based on the nature of their characterization of a sentence. (1) the lexical module acts at the individual lexical item (token) level. sentence-level lexical features are computed by aggregating over the tokens of a sentence. (2) contextual features are sentence-level features and are obtained as a result of some computation at the sentence level, such as, constructing and then processing a syntax parse tree. finally, (3) embeddings computes pooled vector representations from one of many popular embedding models, from glove @xcite to transformer architec-tures @xcite @xcite . these three modules cover a wide range of features-derived from text corpora or behavioral experiments-that have some demonstrated relevance to language processing. the lexical module consists of features that pertain to individual lexical items, words, regardless of the context in which they appear. these features include, for example, lexical frequency, concreteness, and valence. because sentspace is built to work with sentences, lexical-level features are aggregated across the sentence (cf. @xcite ). as a default, sentspace aggregates over all words with available norms in the sentence by computing the arithmetic mean across words. the contextual module consists of features that quantify contextual and combinatorial inter-word relations that are not captured by individual lexical items. this module encompasses features that relate to the syntactic structure of the sentence (con-textual_syntax features) and features that apply to the sentence context but are not (exclusively) related to syntactic structure (contextual_misc features). contextual_syntax include features related to syntactic complexity, instantiated as e.g., surprisal or integration cost, based on leading theoretical proposals @xcite @xcite . some syntactic features are computed for each word in the sentence and then subsequently aggregated; other features are computed for multi-word sequences or the entire sentence. contextual_misc include features like lexical density or sentence sentiment. the embedding module consists of highdimensional vectors derived from pre-trained language models. sentspace also provides functionality that allows users to contribute novel features and modules. a user may design their own features and plug-and-play into sentspace to achieve a more streamlined analysis pipeline and integrated benchmarking and visualization (figure 1 ). in order to contribute a module, users must adhere to the module call api, accepting a sentence batch and returning a dataframe whose columns consist of features. users may make use of parallelism and other utils provided as a part of sentspace. users may also plug in their computed features in the visualization module and use the web interface. ## benchmarking against large corpora to understand where a sentence stands relative to other text, we facilitate comparison with sentences figure 6 : we use sentspace to visualize sentences from two sources of interest. in one case, humans generated paragraphs, and in the other, a gpt2-xl language model did. we find several points of differences between the two sources, verified using statistical tests comparing the two distributions. p-values were obtained using two-tailed independent samples t-tests: concreteness (@xmath1.24, p≪0.001), polysemy (@xmath2.27, @xmath3.s.), lexical frequency (@xmath4.91, @xmath5.005), n-gram surprisal (3-gram) (@xmath6.91, @xmath7.005), syntactic integration cost (@xmath8.34, @xmath9.05), syntactic embedding depth (@xmath10.81, @xmath11.s.). *@xmath12.05, **@xmath13.01, ***@xmath14.005. from large corpora of human-generated text (both written and spoken). we allow this by subsampling from large corpora to include an approximately equal number (≈ 500) of sentences from each corpus. we pre-computed and cached sentspace features for each of 8 corpora (≈ 4000 sentences in total), enabling quick and streamlined comparison with sentences from existing corpora. in the sentspace[.vis] module, these corpora are loaded by default in addition to user-supplied input. corpora benchmarking can be disabled to allow visualizing user input in isolation. we provide a list of corpora used in appendix c. ## system demonstration and results in this section we provide an example of how sentspace can be used to compare and visualize sets of sentences to one another using features from the lexical (@xmath15. for this example demonstration, we compare two sets of materials: human-generated text versus gpt2-xl-generated text. the texts consisted of 52 unique paragraphs written by multiple human writers. the first 10 words of each paragraph were used as a prompt to a pretrained gpt2-xl autoregressive language model @xcite . prompt completions were extracted across multiple random seeds using topp sampling @xcite with generation parameters @xmath16.9 and @xmath17. we selected 5 completions per prompt that most closely matched the human-generated prompt in word length (@xmath18. as a result, we had one human-generated and 5 gpt2-xl-generated paragraphs per prompt, yielding a total of @xmath19, see appendix b). features were averaged across sentences within each paragraph. for statistical tests, features for the @xmath20. in figure 6 , we demonstrate that our feature measures can reveal subtle quantitative differences between machine-generated (blue) and human-generated (red) texts that may not be subjectively apparent. figure 6a demonstrates three features from the lexical module: i) concreteness @xcite ; a behavioral measure of the extent to which the concept denoted by the word refers to a perceptible entity, ii) polysemy @xcite , and iii) log lexical frequency from the subtlex-us database @xcite . as evident, gpt2-xl produces sentences that on average have less concrete words compared to human sentences (p≪.001). lexical frequency reflects how often a given word is used in language. lexical frequency is known to affect language comprehension, for instance more frequent words are read faster (e.g., @xcite ; @xcite and articulated faster (e.g., @xcite ). we can see this as being a trend towards gpt2-xl's use of more frequent wording compared to humans (p≪.001). figure 6b demonstrates three contextual features: i) n-gram surprisal (3-gram), ii) average syntactic integration cost according to the dependency locality theory (dlt, @xcite ; integration cost is roughly proportional to dependency length), and iii) average syntactic center-embedding depth in a left-corner phrase-structure parser (van @xcite . although gpt2-xl usually generates sentences that are syntactically wellformed, their syntactic features differ on average from human-generated text. as shown, texts generated by gpt2-xl show lower 3-gram surprisal (@xmath21.91, p≪.001), tend to be less syntactically complex on average than human-generated ones, with shorter syntactic dependencies (@xmath22.33, @xmath23.02) and numerically shallower center-embedded tree structures (@xmath24.8, @xmath25.09, n.s.). so, these findings might suggest gpt2-xl makes use of 'simpler' wording compared to humans. the remaining sentspace features obtained for the comparison between human-and gpt2-xlgenerated text (@xmath26, the comparison between human-and machinegenerated text is a demonstration of one of the use cases of sentspace: comparing and visualizing texts to one another. the sentspace framework streamlines the process of obtaining corpora-backed features, parsing and syntactically analyzing texts, simplifying and accelerating such analyses for natural language generation. ## related work related work include @xcite who collected behavioral visual lexical decision and speeded naming reaction times and provided these along with a set of word-level, psycholinguistic features (the english lexicon project). @xcite provide a meta-base of word-level, psycholinguistic features. a different alley of related work includes visualization tools for highdimensional embeddings obtained from pre-trained language models (e.g., van aken et al. (2020); ope-nai). a large body of work focuses on characterizing bias in text, particularly that either used in training language models, or that generated by language models @xcite . related work also focuses on methods to mitigate bias in existing language models using debiasing methods. in the future we hope to include norms that characterize bias as one of the many features that will be added to sentspace. we also hope that outputs from sentspace will inform what data goes into training large language models to make them more human-like. ## conclusion sentspace is a system for obtaining numerical representations of sentences. our core feature modules span lexical, semantic, and syntactic features from corpora and behavioral experiments. we provide an interface for comparing textual inputs to one another or to large normative distributions based on natural language corpora. within the last few years, contextualized embeddings obtained from large pre-trained language models have revolutionized and dominated the field of natural language processing. however, despite these embeddings being useful for diverse applications, it is unclear precisely which information is embedded in these high-dimensional feature representations. we view sentspace as a complementary resource that can provide interpretability and grounding to these pre-trained high-dimensional embeddings. a major limitation of sentspace is that we currently only support english. part of the limiting factor is the relative lack of behavioral and psycholinguistic experimental data for other languages, as well as mature linguistic features tai-lored to other languages. we envision sentspace as a dynamic platform with continuous collaboration across research labs for the addition of new features and we hope to make this framework valuable for a number of applications within natural language processing, cognitive science, psychology, linguistics, and social sciences.