G is P y: A Tool for Measuring Gist Inference Score in Text (2022)

Abstract: Decision making theories such as Fuzzy-Trace Theory (FTT) suggest that individuals tend to rely on gist, or bottom-line meaning, in the text when making decisions. In this work, we delineate the process of developing GisPy, an opensource tool in Python for measuring the Gist Inference Score (GIS) in text. Evaluation of GisPy on documents in three benchmarks from the news and scientific text domains demonstrates that scores generated by our tool significantly distinguish low vs. high gist documents. Our tool is publicly available to use at: https: //github.com/phosseini/GisPy.

## introduction according to fuzzy-trace theory (ftt) @xcite , when individuals read text, they encode multiple mental representations of the text in parallel in their mind. these mental representations vary along a continuum ranging from 1) gist to 2) verbatim. while verbatim representations are related to surface-level information, gist represents the bottom-line meaning of the text, given its context. ftt sees the word gist in much the same way as everyday usage, as the essence or main part, the substance or pith of a matter. gist representations are important to assess because they influence judgments and decision making more than verbatim representations @xcite . knowing gist helps us measure the capability of a document (e.g., news article, social media post, etc.) in creating a clear and actionable mental representation in readers' mind and the degree to which a document can communicate its message. the majority of existing natural language processing (nlp) tools and models focus on measuring coherence, cohesion, and readability in text @xcite @xcite @xcite @xcite . it is worth mentioning that even though coherence promotes gist extraction, these two are not the same. and gist can be viewed as a mechanism that allows coherence apprehension @xcite . to the best of our knowledge, there is no publicly available tool for directly measuring gist in text. @xcite ; @xcite ; @xcite are the only studies that introduced a theoretically motivated method to measure gist inference score (gis) using a subset of coh-metrix indices. coh-metrix @xcite is a tool for producing linguistic and discourse representations of a text including measures of cohesion and readability. coh-metrix, even though useful and inspiring, has several limitations. for example, its public version does not allow batch processing of documents, is only available via a web interface, and its cohesion indices focus on local and overall cohesion @xcite . in this work, inspired by @xcite and definition of a subset of indices in coh-metrix, we develop a new open-source tool to automatically compute gis for a collection of text documents. we leverage the state-of-the-art nlp tools and models such as contextual language model embeddings to further improve the quality of indices in our tool. our contributions can be summarized as follows: ## methods in this section, we explain how we implement each of the indices in gispy and compute gis. we start by explaining common implementation features among indices followed by specific details about each of them. ## experiments to test whether gispy can correctly group and measure the level of gist in documents, we run our tool on a collection of datasets with known gist levels -low or high. we selected three benchmarks including two introduced by @xcite and one introduced by @xcite to test the quality of scores in our tool. we give more detail about these benchmarks in the following subsections. before running gispy, we also run coh-metrix on each dataset and compute gis using the original coh-metrix indices. our goal for doing so is to: 1) make sure we have a reliable gold standard that we can compare gispy scores with and 2) reproduce the results from @xcite . once we computed the gis score using gispy, to compare low vs. high gist groups, we compare the mean of their gis scores. moreover, we run a student's t-test with the null hypothesis that there is no difference between the two groups in terms of the level of gist. the goal of running the t-test is to see whether our scores can significantly distinguish groups with lower and higher levels of gist. also, since for five indices including referential cohesion, verb overlap based on embeddings, verb overlap using wordnet, concreteness, and imageability we have multiple implementations, we compute the final gis based on all possible combinations of these indices (320 sets of indices for each benchmark). our goal is to find out what implementation of each index contributes better to distinguishing low vs. high gist documents. in a separate analysis, we also run two robustness tests to ensure our results are not biased by seeing all possible combinations of indices. ## results and discussion results of running gispy on three benchmarks are shown in tables 2, 3, and 4. for each benchmark, we listed the top 10 combinations that most significantly distinguish low vs. high gist documents. as can be seen, for indices that we have paragraph-level vs. non-paragraph-level implementations, in the majority of cases, paragraph-level indices achieve better results. we do not necessarily observe a strong difference between local vs. global implementations. also, for concreteness and imageability indices, almost all the time we see better performance when we use megahr scores by @xcite . we leveraged megahr as a replacement for mrc that was originally used by coh-metrix. comparisons of individuals indices for low vs. high gist documents from the best combination on each benchmark are shown in figures 3 , 4 , and 5 . in ## next steps and future work despite achieving significant improvements and solid results from robustness tests on three benchmarks from two domains, there is still great room to further improve the quality of gispy indices. in this section, we list challenges in the current implementation of gispy and explain what we think can be a proper next step and direction in addressing them. we hope these insights inspire the community to keep working on this exciting line of research. we did our best to bring three different benchmarks for measuring gist inference score to life by aggregating, standardizing, and making them very also, our pcdc index, even though based on strong causal connective markers, mainly covers the explicit causal relations while not all causal relations are expressed explicitly in text. it would be interesting to think how we can enhance the quality of this index by also including implicit relations and disambiguating causal connectives that can also be non-causal (e.g., temporal markers such as since or after) or leveraging discourse parsers such as discopy @xcite . we initially hypothesized that utilizing coreference resolution chains (coref index) may also help us improve the referential cohesion index. by looking at the most significant combinations of indices in each benchmark, we noticed that coref appeared in 0/38, 53/281, 1/110 combinations for report vs. editorials, methods vs. discussion, and disney benchmarks, respectively. as a followup, it would be interesting to see how coreference resolution can be leveraged in a different wayindividually or in combination with other implementations of referential cohesion-to further improve this index. ## conclusion in this work, we introduced gispy, a new opensource tool for measuring gist inference score (gis) in text. evaluation of gispy and robustness tests on three different benchmarks of low and high gist documents demonstrate that our tool can significantly distinguish documents with different levels of gist. we hope making gispy publicly available inspires the research community to further improve indices of measuring gist inference in text.