CIPHE : A Framework for Document Cluster Interpretation and Precision from Human Exploration (2024)

Abstract: Document clustering models serve unique application purposes, which turns model quality into a property that depends on the needs of the individual investigator. We propose a framework, Cluster Interpretation and Precision from Human Exploration (CIPHE), for collecting and quantifying human interpretations of cluster samples. CIPHE tasks survey participants to explore actual document texts from cluster samples and records their perceptions. It also includes a novel inclusion task that is used to calculate the cluster precision in an indirect manner. A case study on news clusters shows that CIPHE reveals which clusters have multiple interpretation angles, aiding the investigator in their exploration.

## introduction automatically structuring large text collections into clusters is a common research method for its timesaving potential and aiding in discovering patterns. in digital humanities, clustering methods like topic modeling are frequently used for many applications @xcite @xcite @xcite . topic models are optimized for structuring texts into coherent themes. however, modern clustering methods powered by sophisticated language models can organize the documents beyond themes. it may be semantic, pragmatic, or other valuable stylistic features. validating the cluster quality in these cases, or merely discovering such features, requires moving beyond the traditional measures of topic coherence based on keywords @xcite . humans possess a remarkable ability to find patterns, and the discovery of patterns in collections of texts is no exception. unfortunately, patterns can even be "found" where there are none, a phenomenon called apophenia @xcite 1 . fur-1 apophenia -the tendency to perceive a connection or ther, given an overall collection of documents, it is infeasible to objectively define the most appropriate level of granularity in dividing it into topics. whether sports is one news topic or basketball, football, and tennis should be viewed as individual topics depends on the research and application context. additionally, the background, knowledge, and prior experiences of a reader make it infeasible to establish an objective truth of the cluster properties @xcite . thus, researchers and practitioners often focus on specific aspects with carefully formulated questions and have a need to evaluate individual models on unique data. the connection between topic model output and human interpretation is a topic of debate @xcite @xcite . thus, trusting models based on their automatic coherence benchmarking scores may not be good practice for making scientific claims about the data. moreover, limited quality checks of the topics and apophenia could lead to researchers projecting their own bias to the interpretation of topics, especially if only working with the keyword representation of the topics. to get around this, we suggest performing manual quality validation checks on the actual documents making up a topic. by having human validation of cluster interpretation and precision, there is a stronger basis for making claims based on clustering model results. we propose a framework for collecting data and calculating descriptive metrics for comparing clusters. the framework is aimed toward investigators who either want to systematically validate a model for a specific research question, or who want to use crowdsourcing to collect a general interpretation of a context made up of multiple documents. a qualitative approach to validating cluster coherence and gaining an understanding of the clusters is to extract a sample of texts from each cluster meaningful pattern between unrelated or random things (such as objects or ideas) and inspect them manually, exemplified in @xcite . the inspector would then provide an interpretation of each cluster by 1) free-text naming a characteristic feature defining the cluster (i.e., a common theme the majority of its articles belong to), and 2) identifying texts that do not fit into the cluster according to this definition. we propose cluster interpretation and precision from human exploration (ciphe) as a framework for recording human interpretation of clusters built around these two tasks. this acknowledges the richness of documents and the possible features to which they can be clustered, and leverages the human ability to recognize patterns to discover cluster properties. this aligns with research requiring interpretative depth and contextual understanding. this paper consists of two parts. first, we introduce ciphe as such. second, we report on a case study with crowdsourcing participants in which ciphe is examined with respect to its ability to generate valuable insights via crowdsourcing. for this, we conducted a ciphe survey on ten news article clusters created with different topic models (section 3). one task of the crowdsource workers is to come up with a descriptive name for each cluster. for this, three sets of instructions were applied and their influence on the task complexity and outcome was discussed. we further analyze the survey results according to the various framework metrics and discuss which insights into the clusters they reveal. ## ciphe to bring structure to the perception of multiple evaluators, we introduce cluster interpretation and precision from human exploration (ciphe, figure 1 ) as a framework for recording and analyzing human interpretations of document clusters. provided that a cluster can be characterized by a feature that most of the texts have in common (which may but does not necessarily have to be a general theme), we expect that a human exploring a sample of the cluster will be able to 1) name this feature, and 2) identify which of the articles do not share it and have thus wrongly been placed into the cluster. the central idea of ciphe is to exploit these abilities to obtain insights into the quality and properties of text clusters. for this, ciphe lets a group of survey participants perform both of the mentioned tasks and compares the individual interpretations and quality assessments. a ciphe survey can be performed by either a small group of experts or a larger number of participants. the large number of participants accessible via crowdsourcing allows human evaluation to be based on a broader interpretation of texts and can mitigate certain biases @xcite while an expert survey can be used if the clustered documents or the demands on the clustering require expert knowledge. the quality assessments are indirect, which makes them more comparable: rather than asking a participant explicitly to assess the quality of a cluster on a subjective scale, quality is inferred from the number of documents they exclude from the cluster. in general, different participants may name clusters differently, and their segmentation, i.e. which articles they choose to include in their interpretation of the cluster, will also vary. rather than viewing this as a problem, ciphe builds upon it. for a given cluster, the degree of agreement between participants and their individual assessments of the task complexity are converted to comparable metrics for cluster quality (section 2.1). a ciphe survey consists of three tasks that collect responses reflecting the interpretation of the cluster by the participant. inclusion: the participant is asked to explore the cluster and decide which articles, according to them, belong to the cluster. intuitively, the fewer articles are excluded, the better the cluster in the eyes of the participant. naming: the participant is asked to give the cluster a descriptive free text name. the precise instructions for how to do this may differ. in our case study, we compare three different instruction sets; see section 3.3. complexity assessment: the participant answers likert-scale questions about their experience exploring the cluster @xcite . this provides information about both the participant and the perceived simplicity of interpreting the cluster. a survey platform (figure 2 ) was implemented in django foot_0 to be able to manage the survey in detail and to have control over how the different elements were displayed to participants. ## case study a case study was conducted to validate the ability of ciphe to quantitatively differentiate clusters in terms of interpretation and precision using human interpretation data collected via crowdsourcing. ## results and discussion ciphe is evaluated on its ability to capture the varying quality of the clusters and human interpretation of them (section 4.1). in our analysis, we use the fact that some of the manually selected clusters, the artificial cluster, and the random cluster should be more difficult to interpret. further, we compare the instructions and analyze their effect on the responses (section 4.2). we also discuss adaptions that can be made to suit different usage purposes (section 4.3). ## conclusion we have presented cluster interpretation and precision from human exploration (ciphe), a method for collecting human perception data of document clusters. ciphe is based on the assumption that humans, when presented with a random sample of texts from a given cluster, can identify a majority feature of the texts, and also determine which texts should be excluded from the cluster. this is called the inclusion task and it shows promise for being an indirect measurement of cluster quality that can be used for algorithmic improvement. the case study on ten clusters using crowdsource workers showed that participants generally saw similar coarse themes and that ciphe highlighted when a cluster contained multiple interpretation angles. the framework is flexible enough to support a variety of research questions and practical applications. it was designed to be applicable even with only one dataset and model. future work involves conducting larger-scale experiments with varying document styles to assess statistical properties. rate the complexity of the task: