Unsupervised Concept Representation Learning for Length-Varying Text Similarity (2021)

Abstract: Measuring document similarity plays an important role in natural language processing tasks. Most existing document similarity approaches suffer from the information gap caused by context and vocabulary mismatches when comparing varying-length texts. In this paper, we propose an unsupervised concept representation learning approach to address the above issues. Specifically, we propose a novel Concept Generation Network (CGNet) to learn concept representations from the perspective of the entire text corpus. Moreover, a concept-based document matching method is proposed to leverage advances in the recognition of local phrase features and corpus-level concept features. Extensive experiments on real-world data sets demonstrate that new method can achieve a considerable improvement in comparing length-varying texts. In particular, our model achieved 6.5% better F1 Score compared to the best of the baseline models for a concept-project benchmark dataset.

## introduction measuring the similarity between documents is a fundamental problem in several natural language tasks such as information retrieval @xcite , paraphrase identification @xcite and question routing @xcite . a wide range of document similarity approaches @xcite have been proposed to handle the fundamental problem; however, most of them are based on the assumption that the documents being compared have similar document length. however, varying-length document matching tasks are ubiquitous in many real-world scenarios. for instance, in the news categorization task, the news articles may include both short reports for breaking news or narrative reports with cumbersome details. the document matching in varying length may introduce the information gap between two documents in the following two aspects: (i) context mismatch, which is caused by the long-length documents usually provide more detailed context to support the key information while the short-length documents contain limited context information. the issue renders the existing pre-trained natural language representation models @xcite pay more attention to the long but less important contexts, which makes their document representations distinct from the short-length documents with little context information. (ii) vocabulary mismatch, which is usually caused by the different terms usage between short and long texts, which leads them do not share majority terms. existing document distance such as word mover's distance @xcite focus on comparing the local features. still, the vocabulary mismatch issue makes the local features hard to be matched while the majority of vocabulary is not shared. to address the above challenges, our approach proposes a concept-based document matching method that incorporates both local phrase features and corpus-level concepts in an unsupervised setting, where concepts can be interpreted as a group of representative features that are interpretable for humans. the main contributions of this paper can be summarized as follows: (i) a novel unsupervised concept generation network is proposed to learn corpus-level concepts in the perspective of entire text corpus. specifically, each concept and its phrase assignment is iteratively optimized by the reconstruction loss between local phrase features and global concept representations. (ii) a new concept-based document comparison method is proposed to measure the similarity between two text documents based on augmented concept representations, which leverages the advances of local phrases and corpus-level concepts. moreover, an enhanced concept-weight constraint is proposed to improve the performance in optimizing the concept-based document similarity. (iii) extensive experiments on several length-varying text matching datasets demonstrate that the effectiveness of our proposed approach consistently outperforms existing state-of-the-art methods. in particular, our method improved 7.1% accuracy and 6.5% f1-score in concept-project dataset compared to the best baseline method. the rest of this paper is organized as follows. section 2 reviews related work, and section 3 provides a detailed description of our proposed model. the experiments on multiple real-world data sets are presented in section 4. the paper concludes with a summary of the research in section 5. ## related work in this section, we briefly describe recent advances in document similarity research. we start our discussion with recent progress in supervised methods, and then we shift our focus to unsupervised settings. ## model we now describe our approach to calculate the document similarity for length-varying texts. we begin by introducing the overview of our model in section 3.1. then we provide details of the concept generation and document matching components in section 3.2 and 3.3. last, the implementation details are described in section 3.4. ## experiment in this section, we evaluate the performance of the model described in section 3 on document matching task for length-varying texts. ## conclusion in this paper, an unsupervised concept representation learning method is proposed to address the length-varying text comparison problem. to achieve this, we propose a deep neural network based model to generate corpus-level concept representation and design a concept-based document matching method based on augmented concept representation that leverages the advances of both local phrase features and global concept features. extensive experiments on real-world datasets demonstrated that our proposed method dramatically outperforms competing methods, exhibiting a significant improvement in all the metrics in different length-vary text comparison tasks.