Explainable and Sparse Representations of Academic Articles for Knowledge Exploration (2020)

Abstract: We focus on a recently deployed system built for summarizing academic articles by concept tagging. The system has shown great coverage and high accuracy of concept identification which could be contributed by the knowledge acquired from millions of publications. Provided with the interpretable concepts and knowledge encoded in a pre-trained neural model, we investigate whether the tagged concepts can be applied to a broader class of applications. We propose transforming the tagged concepts into sparse vectors as representations of academic documents. The effectiveness of the representations is analyzed theoretically by a proposed framework. We also empirically show that the representations can have advantages on academic topic discovery and paper recommendation. On these applications, we reveal that the knowledge encoded in the tagging system can be effectively utilized and can help infer additional features from data with limited information.

## introduction efficiently exploring knowledge is an active research topic in this era. in this work, we focus on a deployed system @xcite which is built for summarizing academic publications via explainable concepts. an example 1 of the system output is shown in figure 1 , where the system tags relevant concepts such as "word2vec" and "word order" for summarizing the given paper. notably, the concepts are called fields-of-study (fos). to ensure concepts covered by various articles can be recognized, the system acquired knowledge from 170 million academic publications via deep learning models. the learned concepts are further organized by a hierarchical structure. specifically, the hierarchy is a 6-level tree. the fos in level 0 are the most coarse-grained concepts such as "computer science" and "chemistry". on the contrary, fos in level 5 are the most fine-grained concepts such as "convolutional deep belief networks" and "phosphatase". in the tagging process, the system would normally tag both high-level and low-level fos for helping users efficiently and thoroughly understand ideas of the given articles. the accuracy of tagging has been carefully evaluated by human judge. however, whether the tagged fos can be beneficial to broader applications is still unknown. we propose that this problem is worth studying as the tagging system natively possesses two advantages. the first one is the guaranteed interpretability. it potentially helps models leveraging the identified concepts be more explainable. secondly, the tagging system has learned abundant knowledge from millions of academic papers. ideally, the learned knowledge can be utilized for solving diverse natural language processing (nlp) problems as long as academic data are involved. in this work, we reveal that the tagged fos can be transformed into sparse and explainable representations. we then theoretically and empirically show that the proposed representations indeed inherit the two advantages mentioned above. this paper is organized as follows. in section 2, we first describe how we obtain sparse representations via the fos tagging system with slight modifications. afterwards, we provide theoretical analyses on the representations in section 3 by modeling the tagging system as a framework with a replaceable neural model. in the analyses, we reveal that the tagging process leveraging concept hierarchy potentially helps the sparse vectors capture more thorough semantics for measuring document similarities. in section 4 and 5, we evaluate representations via two applications related to knowledge exploration and can benefit from the fos-based methods. the first one is document clustering for topic discovery. it could help users quickly classify and identify documents in a massive text corpus. we empirically show that the proposed representations and similarity measurements can gather documents with common topics more accurately. we also leverage the interpretability of fos to explain the generated clusters. another application is document ranking for paper recommendation. as the queries for paper searching are usually short, we demonstrate that our tagging system is able to tag abundant fos on extremely short documents. particularly, the tagged fos could have similar effects to query expansion. we then show that the fos sparse representations can constantly help improve ranking performance especially on searching by short queries. ## conclusion in this work, we investigated whether the concepts summarizing academic publications can be effective sparse representations for diverse applications. to examine the effectiveness, we provided theoretical analyses and empirical studies for verification. we presented that the tagging method with the concept hierarchy could have an effect on capturing more thorough semantics. we also revealed several advantages of the proposed representations. the intrinsic advantages would be interpretability and compactness, helping people understand the underlying knowledge more efficiently. when coming to downstream tasks, the knowledge learned from millions of publications can show the impact. as reported, the tagging system can infer additional features from data with limited information. also, the additional features can be easily combined with common ranking methods and achieve better performance. for future work, a direction would be focusing on the neural model providing the dense word and fos representations. as the current system has a limitation that dynamic contextual information is not included, it could be interesting to investigate whether similar conclusions can be made on contextualized embeddings.