UCT opic: Unsupervised Contrastive Learning for Phrase Representations and Topic Mining (2022)

Abstract: High-quality phrase representations are essential to finding topics and related terms in documents (a.k.a. topic mining). Existing phrase representation learning methods either simply combine unigram representations in a context-free manner or rely on extensive annotations to learn context-aware knowledge. In this paper, we propose UCTopic, a novel unsupervised contrastive learning framework for context-aware phrase representations and topic mining. UCTopic is pretrained in a large scale to distinguish if the contexts of two phrase mentions have the same semantics. The key to the pretraining is positive pair construction from our phrase-oriented assumptions. However, we find traditional in-batch negatives cause performance decay when finetuning on a dataset with small topic numbers. Hence, we propose cluster-assisted contrastive learning (CCL) which largely reduces noisy negatives by selecting negatives from clusters and further improves phrase representations for topics accordingly. UCTopic outperforms the state-of-the-art phrase representation model by 38.2% NMI in average on four entity clustering tasks. Comprehensive evaluation on topic mining shows that UCTopic can extract coherent and diverse topical phrases.

## introduction topic modeling discovers abstract 'topics' in a collection of documents. a topic is typically modeled as a distribution over terms. highquality phrase representations help topic models understand phrase semantics in order to find well-separated topics and extract coherent phrases. some phrase representation methods @xcite @xcite learn context-free representations by unigram embedding combination. context-free representations tend to extract similar phrases mentions (e.g. "great food" and "good food", see section 4.3). contextaware methods such as densephrase @xcite the united states is a federation of 50 individual states. irving washington's book was popular in the united states. the @xcite [mask] is a federation of 50 individual states. irving washington's book was popular in the @xcite . in this work, we seek to advance state-of-theart phrase representation methods and demonstrate that a contrastive objective can be extremely effective at learning phrase semantics in sentences. we present uctopic, an unsupervised contrastive learning framework for phrase representations and topic mining, which can produce superior phrase embeddings and have topic-specific finetuning for topic mining. to conduct contrastive learning for phrase representations, we first seek to produce contrastive pairs. existing data augmentation methods for natural language processing (nlp) such as back translation @xcite , synonym replacement @xcite and text mix up @xcite are not designed for phrase-oriented noise, and thus cannot produce training pairs for phrase representation learning. in uctopic, we propose two assumptions about phrase semantics to obtain contrastive pairs: for large-scale pre-training, we follow previous works @xcite @xcite and adopt in-batch negatives for training. however, we find in-batch negatives undermine the representation performance as finetuning (see based on the two assumptions and clusterassisted negative sampling introduced in this paper, we pre-train phrase representations on a large-scale dataset and then finetune on a specific dataset for topic mining in an unsupervised way. in our experiments, we select luke @xcite as our backbone phrase representation model and pre-train it on wikipedia foot_0 english corpus. to evaluate the quality of phrase representations, we conduct entity clustering on four datasets and find that pre-trained uctopic achieves 53.1% (nmi) improvement compared to luke. after learning data-specific features with ccl, uctopic outperforms luke by 73.2% (nmi) in average. we perform topical phrase mining on three datasets and comprehensive evaluation indicates uctopic extracts coherent and diverse topical phrases. overall, our contributions are three-fold: • we propose uctopic which produces superior phrase representations by unsupervised contrastive learning based on positive pairs from our phrase-oriented assumptions. ## background in this section, we introduce background knowledge about contrastive learning and our phrase encoder luke @xcite . ## uctopic uctopic is an unsupervised contrastive learning method for phrase representations and topic mining. our goal is to learn a phrase encoder as well as topic representations, so we can represent phrases effectively for general settings and find topics from documents in an unsupervised way. in this section, we introduce uctopic from two aspects: (1) constructing positive pairs for phrases; ## related work many attempts have been made to extract topical phrases via lda @xcite . @xcite incorporated a bigram language model into lda by a hierarchical dirichlet generative probabilistic model to share the topic across each word within a bigram. tng @xcite applied additional latent variables and word-specific multinomials to model bi-grams and combined bi-grams to form n-gram phrases. pd-lda @xcite used a hierarchical pitman-yor process to share the same topic among all words in a given n-gram. @xcite ranked the resultant phrases based on four heuristic metrics. top-mine @xcite proposed to restrict all constituent terms within a phrase to share the same latent topic and assign a phrase to the topic of its constituent words. compared to previous topic mining methods, uctopic builds on the success of pre-trained language models and unsupervised contrastive learning on a large-scale dataset. therefore, uctopic provides high-quality pre-trained phrase representations and state-of-the-art finetun-ing for topic mining. early works in phrase representation build upon a composition function that combines component word embeddings together into simple phrase embedding. @xcite implemented the function by rule-based composition over word vectors. @xcite applied a pair-wise gru model and datasets such as ppdb @xcite to learn phrase representations. phrase-bert @xcite composed token embeddings from bert and pretrained on positive instances produced by gpt-2-based diverse paraphrasing model @xcite . @xcite learned phrase representations from the supervision of reading comprehension tasks and applied representations on open-domain qa. other works learned phrase embeddings for specific tasks such as semantic parsing @xcite and machine translation @xcite . in this paper, we present unsupervised contrastive learning method for pre-training phrase representations of general purposes and for finetuning to topicspecific phrase representations. ## conclusion in this paper, we propose uctopic, a contrastive learning framework that can effectively learn phrase representations without supervision. to finetune on topic mining datasets, we propose clusterassisted contrastive learning which reduces noise by selecting negatives from clusters. during finetuning, our phrase representations are optimized for topics in the document hence the representations are further improved. we conduct comprehensive experiments on entity clustering and topical phrase mining. results show that uctopic largely improves phrase representations. objective metrics and a user study indicate uctopic can extract coherent and diverse topical phrases.