Moving beyond word lists: towards abstractive topic labels for human-like topics of scientific documents (2022)

Abstract: Topic models represent groups of documents as a list of words (the topic labels). This work asks whether an alternative approach to topic labeling can be developed that is closer to a natural language description of a topic than a word list. To this end, we present an approach to generating human-like topic labels using abstractive multi-document summarization (MDS). We investigate our approach with an exploratory case study. We model topics in citation sentences in order to understand what further research needs to be done to fully operationalize MDS for topic labeling. Our case study shows that in addition to more human-like topics there are additional advantages to evaluation by using clustering and summarization measures instead of topic model measures. However, we find that there are several developments needed before we can design a well-powered study to evaluate MDS for topic modeling fully. Namely, improving cluster cohesion, improving the factuality and faithfulness of MDS, and increasing the number of documents that might be supported by MDS. We present a number of ideas on how these can be tackled and conclude with some thoughts on how topic modeling can also be used to improve MDS in general.

## introduction topic modeling, a common approach for extracting themes from scientific documents, is currently facing many challenges: methodological validity @xcite , validity of automated evaluation @xcite , and utility of classical approaches @xcite . we propose an additional challenge: are lists of words the best we can do for topic labels? topic models have tended to represent a topic as a list of words. traditional topic labels are supposed to be "a set of terms, when viewed together, enable human recognition of an identifiable category" @xcite . however, a set of terms do not align with our intuitive understandings of what a topic is: a common theme or concept explicated as a word, phrase, or natural language description @xcite . in this paper, we present an exploratory case study using multi-document summaries (mds) as labels for clusters of citations in order to understand current limitations and future work needed for using abstractive topic labels for human-like topics of scientific documents. to our knowledge, it is the first work that proposes to use mds for topic labeling on top of topic clusters constructed with contextualized embeddings. in addition to word lists not aligning with natural understanding of what a topic is, @xcite has presented an extensive criticism of why traditional topic models based on lexical overlap measures lead to problematic topic models. namely that they they fail to understand word sense and capture context. recent approaches have relaxed these restrictions when constructing topic clusters @xcite by using contextualized word embeddings. however topic labels in those models are still constructed as word lists drawn from documents such as through tf-idf. some work has anticipated this challenge by developing topic representations with phrases @xcite and summaries @xcite @xcite . but those works tend to be extractive, drawing the phrase or summary from a single document in the cluster foot_0 . in the extractive setting, there may be no existing and fluent phrase or sentence that is capable of describing all documents in the cluster or there may be multiple and even conflicting subtopics in the cluster that require a longer abstractive representation for producing a factual summary. ## experimental setup we apply the method described in section 2 in order to identify clusters of citations and provide labels for those clusters without any supervision. specifically, we present a case study of what this looks like on a single paper to illustrate the potential of our approach and try to assess future work needed in order to make mds a good solution for topic labeling in general and citation summarization in particular. for this study, we used scite.ai @xcite to extract in-text passages which contained citations (citation statements) to the paper (lau et al., 2014), a well known paper that introduces the npmi metric in topic modeling. this resulted in 183 citation statements which is the corpus we will use for topic modeling. in order to identify meaningful groups of clusters we use contextualized topic models (ctm) @xcite since this method uses contextualized word embeddings (we used specter for constructing embeddings @xcite ). we selected ctm since we still get word lists as topic labels which we used for evaluation. in order to select the number of topics hyperparameter, we trained ctm several times steadily increasing the number of topics from 3 to 50 and selected the best model according to coherence (npmi) resulting in a 10 topic model (see appendix a for more details) over 183 citation statements. the models selected for generating abstractive mds are outlined in ## results we randomly sampled 3 topics to explore their representations. as an example, ## discussion in order to ensure downstream topic labels are coherent, document clusters must represent meaningful and well separated clusters. grootendorst (2022); sia et al. (2020); zhang et al. (2022) have shown that traditional clustering methods might provide good candidates for moving beyond topic models like lda that suffer from lack of contextualized natural language understanding due to their use of word co-occurence statistics for constructing topic clusters. however in order to fully replace traditional methods we would like to see: (1) the demonstration of effective mixed-membership approaches in abstractive topic modeling to recover the ability for documents to belong to multiple topic clusters, (2) the demonstration of cluster evaluation measures that correlate well with how hu-topic 0 npmi and topic coherence are measures used to measure the semantic coherence of topics. topic 4 topic model quality and interpretability are two different metrics used to measure the semantic interpretability of a topic. topic 2 evaluation metrics: log predictive probability (lpp) and topic interpretability in order to make this approach work for a wide variety of application and analysis scenarios, controllable summarization (such as @xcite ) should be investigated so that users can control for length of summaries (such as question, phrase, sentence, or paragraph) or style of summary (such as in the style of a paper title, abstract, citation, or literature review). additional controls such as the ones suggested in shadrova (2021) like granularity of topic label can also be developed in a controllable summarization framework in such a way as to make topic representations better fit for user's needs. finally while methods like longformer @xcite enable the use of transformers with multiple documents as input, more research needs to be done to enable a method like the one we proposed on large sets of documents. in the scientific domain, where we might want to model hundreds or even thousands of full-text articles belonging to a single cluster, the approaches presented would be intractable without further development of longattention transformer models. one advantage of our approach is that since we are breaking topic models out into clustering and mds as separate steps we can rely on a established work for evaluation of document clusters and summaries to assess models performance. while we'd need to validate the application of these metrics in end-to-end topic modeling scenarios, if text clustering and summarization metrics do correlate with human judgements of topic cluster and representation quality then we can avoid using topic modeling metrics which have come into question repeatedly @xcite ; @xcite ; doogan and buntine (2021)). however, we will not know this until we design robust human studies to validate the approach we have proposed above. ## conclusion in this paper, we presented a reframing of topic modeling as document clustering with mds applied to produce topic representations that might (1) align more intuitively with what humans understand as topics and (2) overcome some of the issues with topic models using bag of word assumptions such as inability to capture context. an initial case study on using this approach for unsupervised discovery of citation intents was explored. we found that while cohesive alternatives to topic representations can be produced using mds in a variety of styles (short and long summaries), there are still many obstacles that need to be overcome before we can fully evaluate whether this approach could provide a viable alternative to traditional topic modeling and representation. namely, improving cluster cohesion, improving the factuality and faithfulness of mds, and increasing the num-ber of documents that might be supported by mds. while there might be an advantage in utilizing well validated approaches for evaluating clustering and summarization as measures of our approach, future studies will need to validate those with human studies. it is our hope that further work in this area can use our discussion as a roadmap towards what needs to be done if we want to move past word lists as topic representations.