Goal-Driven Explainable Clustering via Language Descriptions (2023)

Abstract: Unsupervised clustering is widely used to explore large corpora, but existing formulations neither consider the users’ goals nor explain clusters’ meanings. We propose a new task formulation, “Goal-Driven Clustering with Explanations” (GoalEx), which represents both the goal and the explanations as free-form language descriptions. For example, to categorize the errors made by a summarization system, the input to GoalEx is a corpus of annotator-written comments for system-generated summaries and a goal description “cluster the comments based on why the annotators think the summary is imperfect.”; the outputs are text clusters each with an explanation (“this cluster mentions that the summary misses important context information.”), which relates to the goal and accurately explains which comments should (not) belong to a cluster. To tackle GoalEx, we prompt a language model with “[corpus subset] + [goal] + Brainstorm a list of explanations each representing a cluster.”; then we classify whether each sample belongs to a cluster based on its explanation; finally, we use integer linear programming to select a subset of candidate clusters to cover most samples while minimizing overlaps. Under both automatic and human evaluation on corpora with or without labels, our method produces more accurate and goal-related explanations than prior methods.

## introduction text clustering is widely used to explore large corpora @xcite . however, existing formulations cannot adapt to different users' goals, which might be clustering based on sentiment, genre, or other properties @xcite ; as a result, the desired output is underspecified. furthermore, since the output clusters are not immediately interpretable, users must manually examine the clusters to gain insights. this can be time-consuming, especially when some clusters are semantically incoherent @xcite . where the input is a set of texts (corpus) and a goal, and the output constitutes a set of corpus subsets (clusters) each with an explanation. given the goal, a successful goalex algorithm should cluster based on sentiment instead of topic and for each cluster explain which samples should (not) belong to it. to address these weaknesses, we propose a new task formulation, goalex, "goal-driven clustering with explanations" (section 2). as illustrated in figure 1 , the input to the task is a text corpus with multiple attributes (e.g., sports and arts related texts with different sentiments) and a goal description in natural language ("clustering based on sentiment"). the output of the task constitutes a set of corpus subsets (clusters), each with a natural language explanation of which text samples should or should not belong to the cluster (e.g. "contains positive sentiment"). the output should satisfy three desiderata: 1) the explanations are goal-related, 2) each cluster is accurately described by its explanation, and 3) the clusters should overlap minimally while their union should cover most of the corpus. to tackle goalex, we develop a three-stage algorithm propose-assign-select (pas, figure 2 , section 3), each designed to address one of the desiderata. at the proposal stage we address the 1st desideratum that the explanation should be goalrelated; we do this by prompting a language model (lm) to generate a list of goal-related explanations ❶ propose: given a goal and a set of texts, propose a list of explanations each representing a candidate cluster. for candidate clusters based on the goal and a subset of the corpus. at the assignment stage we address the 2nd desideratum that the explanations should accurately explain the clusters; we do this by assigning text samples only to the explanations that support them. at the selection stage we address the 3rd desideratum on maximizing coverage while minimizing overlap; we do this by using integer linear programming to search for a subset of candidate explanations so that each sample is roughly supported once. at last, we output the selected explanations and their supported samples as clusters. we benchmarked pas in two ways: 1) automatically evaluating its ability to recover known clusters from corpora (section 4), and 2) manually evaluating its clusters and explanations on open-ended corpora (section 5). for automatic evaluation, we first compared pas to prior methods on recovering topic clusters underlying news and dbpedia articles and found that pas is competitive while additionally providing accurate explanations. to test whether pas is goal-driven, we used an lm to synthesize a corpus, syngoalex, where each text has three known attributes: topic, language, and style; pas effectively adapts to different goals such as "clustering by topic / language / style", while prior methods fail catastrophically. for open-ended evaluation, we constructed opengoalex, a collection of 12 open-ended goalex problems from various nlp papers. we compared pas to previous clustering methods such as lda and found that pas's explanations are more accurate and goal-related under human evaluation. finally, we applied pas hierarchically to create progressively finer-grained clusters on opengoalex, inducing taxonomies over debate arguments, model errors, and customer reviews. 1 our contributions are summarized as follows. ## defining goalex we formalize the input-output space of goalex and introduce the desiderata for an output. ## the propose-assign-select algorithm each section between 3.1 and 3.3 describes one stage of the pas algorithm (outlined in figure 2 ). ## automatic evaluation following the evaluation protocol from prior works, we evaluated pas by applying it to corpora that are mixtures of known clusters, treating the known clusters and their explanations as the reference solutions, and checking how well the outputs of pas can recover the references. we evaluated pas under two settings: traditional topic clustering and goal-driven non-topic clustering. in both settings, we compared 1) the similarity between the reference and the output clusters automatically and 2) the similarity between the explanations manually. we found that pas is comparable to previous methods for topic clustering and recovers most of the reference explanations; additionally, since pas is goal-driven, it performs significantly better when there are multiple ways to cluster a corpus. the following sections will present the datasets (section 4.1), the clustering methods we evaluated (section 4.2), the evaluation protocol (section 4.3), and the performance of each method (section 4.4). in addition, we evaluated the quality of each stage of pas in appendix f. ## open-ended advanced applications while pas achieves high performance on benchmarks with cluster labels, it does not necessary imply high performance under real applications. therefore, we constructed opengoalex, a collections of 12 open-ended realistic goalex problems to evaluate pas. since these problems do not have cluster labels, we evaluated pas with the three metrics introduced in section 2: (1) explanation accuracy, (2) goal-relevance, and (3) coverage and overlap. as (1) and ( 2 ) require human annotators and are hence expensive to conduct repeatedly, we used them to test the limit of pas to inform future research: we applied pas with the highest quality models under our budgetary constraints and challenged it to generate taxonomies by producing trees of progressively finer-grained clusters on opengoalex. we evaluated pas quantitatively with human annotators for the first layer of the taxonomy and qualitatively analyzed the rest. ## conclusion we proposed goalex, a new formulation for text clustering that adapts to a user's goal and outputs explanations for each cluster. to tackle goalex, we developed the propose-assign-select (pas) algorithm; under automatic evaluation with known references and open-ended applications, pas can generate accurate and goal-related explanations for the clusters. finally, we applied goalex hierarchically to produce taxonomies over debate arguments, customer complaints, and model errors, thus assisting users to explore large corpora. future works can improve on discovering minority clusters, following the goal better, and resolving global inconsistency when applying pas recursively.