Unsupervised Keyphrase Extraction via Interpretable Neural Networks (2023)

Abstract: Keyphrase extraction aims at automatically extracting a list of “important” phrases representing the key concepts in a document. Prior approaches for unsupervised keyphrase extraction resorted to heuristic notions of phrase importance via embedding clustering or graph centrality, requiring extensive domain expertise. Our work presents a simple alternative approach which defines keyphrases as document phrases that are salient for predicting the topic of the document. To this end, we propose INSPECT—an approach that uses self-explaining models for identifying influential keyphrases in a document by measuring the predictive impact of input phrases on the downstream task of the document topic classification. We show that this novel method not only alleviates the need for ad-hoc heuristics but also achieves state-of-the-art results in unsupervised keyphrase extraction in four datasets across two domains: scientific publications and news articles.

## introduction keyphrase extraction is crucial for processing and analysis of long documents in specialized (e.g., scientific, medical) domains @xcite @xcite . the task is challenging, as the notion of phrase importance is context-and domain-dependent. therefore, developing domain-agnostic keyphrase annotation guidelines and curating representative handlabeled datasets is not feasible. this motivates the need for generalizable unsupervised approaches to keyphrase extraction. unsupervised keyphrase extraction methods have used heuristic notions of phrase importance @xcite @xcite . popular proxies for phrase importance include phrase clustering based on statistical features like word density @xcite figure 1 : a comprehensive set of keyphrases should highlight important phrases for all major topics in a document. inspect identifies such keyphrases using interpretable neural models by measuring how use phrases are for predicting the topic of a text. @xcite and structural features like graph centrality @xcite or more recently neural embedding clustering techniques @xcite @xcite . however, such methods do not generalize to new domains as they require experts to carefully construct domain-specific heuristics @xcite . historically, topic models @xcite @xcite have relied on salient words and phrases in a document, which are similar to the notion of keyphrases, although to the best of our knowledge there is no prior work that identified keyphrases using topic models. in this work, we hypothesize that end-to-end neural models for topic classification latently rely on salient phrases for document representation and topic classification. consequently, if we can interpret model decisions via highlighting salient and influential features (phrases) used for neural topic prediction, we can identify such keyphrases. inspired by this intuition, we propose in-spect-a novel and simple framework to identify keyphrases by leveraging interpretable text classi-fiers to highlight phrases important for predicting the topics in a text. specifically, we adapt an interpretable classifier selfexplain @xcite to jointly predict the topic of an input document and to identify the salient phrases influencing the prediction. the model is distantly supervised using topic labels from off-the-shelf topic-models, eliminating the need for any human/expert annotations. we consider selfexplain's output interpretations as keyphrases for the input document ( §2). inspect can be trained on documents of any domain without keyphrase annotations and using distant topic supervision, making them easily adaptable to new domains. we contribute two versions of our method: i) inspect-individual models trained for topic-classification for each target dataset. ii) inspect-gen-a more general model pre-trained on a large in-domain corpus, without finetuning on pre-specified target datasets. we evaluate inspect and inspect-gen on four benchmark datasets across two domains: scientific documents and news articles ( §3). our results in §4 show that inspect improves keyphrase extraction performance over strong baselines by 0.8% f1 on average, without any domain-specific processing. inspect-gen further improves the performance, outperforming the state of the art in unsupervised keyphrase extraction on 3 out of 4 datasets by 2.7% f1 on average. our experiments suggest that inspect-gen has strong generalization capabilities, and can be used out-of-the-box without finetuning on individual datasets. importantly, inspect alleviates the need for heuristics and expert-labelled annotations, and thus can be applied to a wide range of domains and problems where keyphrase extraction is important. our results confirm that the latent keyphrases obtained from an interpretable model correlate with human annotated keyphrases, opening new avenues for research on interpretable models for information extraction. ## the inspect framework the goal of the inspect framework is to extract important keyphrases in long documents. following the hypothesis that neural text classifiers latently leverage important keyphrases for predicting topics in text, inspect extracts keyphrases through interpreting topic classification decisions. it builds upon an interpretable model, selfexplain @xcite , which learns to attribute text classification decisions to relevant phrases in the input. however, selfexplain was designed and tested in supervised settings and for single-sentence classification; in this work we explore its extension to unsupervised keyphrase extraction from long documents. in what follows, we describe the base selfexplain model ( §2.1) and the distant supervision setup for topic classification ( §2.4). we outline the training mechanism to jointly predict topics and highlight salient phrases in the document as model interpretations ( §2.2) and finally extract the resulting phrase interpretations as important keyphrases in the document ( §2.3). ## discussion here, we present an analysis on the common error types in inspect and discuss the strengths and weaknesses of inspect using qualitative examples. ## related work unsupervised keyphrase extraction is typically treated as a ranking problem, given a set of candidate phrases @xcite @xcite . broadly, prior approaches can be categorized as statistical, graph-based, embedding-based, or language model based methods; @xcite provide a detailed survey. statistical methods exploit notions of information theory directly. common approaches include tf-idf based scoring @xcite of phrases with other co-occurrence statistics to enhance performance @xcite . @xcite shows the importance of incorporating statistical information of the context of each phrase to improve performance. statistical approaches typically treat different instances of a phrase equally, which is a limitation. graph-based techniques, on the other hand, broadly aim to form a graph of candidate phrases connected based on similarity to each other. then core components of the graph are chosen as key phrases. amongst these, pagerank @xcite and textrank @xcite assign scores to nodes based on their influence. a common extension is to use weights on the edges denoting the strength of connection @xcite @xcite . position rank @xcite and sgrank @xcite combine the ideas from statistical, word co-occurrence and positional information. some approaches, especially applied in the scientific document setting, make use of citation graphs @xcite , and external knowledge bases @xcite to improve keyphrase extraction. in this work, we focus our approach on a general unsupervised keyphrase extraction setting applicable to any domain where such external resources may not be present. finally, embedding based techniques @xcite @xcite ) make use of word-document similarity using word embeddings @xcite , while language-model based techniques use word prediction uncertainty to decide informativeness @xcite . @xcite uses attention scores to calculate phrase importance 1114 with the document in an unsupervised manner. ## conclusion and future work in this work, we introduced inspect, a novel approach to unsupervised keyphrase extraction. our framework uses a neural model that explains text classification decisions to extract keyphrases via phrase-level feature attribution. using four standard datasets in two domains, we show that in-spect outperforms prior methods and establishes state-of-art results in 3 out of 4 datasets.. through qualitative and quantitative analysis, we show that inspect can produce high-quality and relevant keyphrases. inspect presents applications of interpretable models beyond explanations for humans.