An Intent-based and Annotation-free Method for Duplicate Question Detection in CQA Forums (2023)

Abstract: With the advent of large language models (LLMs), Community Question Answering (CQA) forums offer well-curated questions and answers that can be utilized for instruction-tuning, effectively training LLMs to be aligned with human intents. However, the issue of duplicate questions arises as the volume of content within CQA continues to grow, posing a threat to content quality. Recent research highlights the benefits of detecting and eliminating duplicate content. It not only enhances the LLMs’ ability to generalize across diverse intents but also improves the efficiency of training data utilization while addressing concerns related to information leakage. However, existing methods for detecting duplicate questions in CQA typically rely on generic text-pair matching models, overlooking the intent behind the questions. In this paper, we propose a novel intent-based duplication detector named Intent-DQD that comprehensively leverages intent information to address the problem of duplicate question detection in CQA. Intent-DQD first leverages the characteristics in CQA forums and extracts training labels to recognize and match intents without human annotation. Intent-DQD then effectively aggregates intent-level relations and establishes question-level relations to enable intent-aware duplication detection. Experimental results on fifteen distinct domains from both CQADupStack and Stack Overflow datasets demonstrate the effectiveness of Intent-DQD. Reproducible codes and datasets will be released upon publication of the paper.

## introduction with the rapid development of web technology, community question answering (cqa) forums have emerged as crucial platforms for individuals to seek solutions to their problems. these forums serve as high-quality knowledge bases, offering * corresponding author well-curated questions and answers that can be effectively utilized for instruction-tuning for large language models (llms). instruction tuning plays a vital role in the reinforcement learning from human feedback (rlhf) process employed in building systems like chatgpt and claude @xcite . the purpose of instruction tuning is to train llms to better follow human intents. for instance, when a human posts a question to an llm, instruction tuning helps the llm learn to provide an answer instead of merely imitating the act of asking a question. as the volume of content in cqa forums continues to increase, the issue of duplicate questions becomes more prevalent, thereby posing a significant challenge to content quality. recent research emphasizes the significance of detecting and removing duplicate content in the instruction-tuning of llms. it promotes generalization across diverse intents @xcite , reduces overlap between training and test sets @xcite , mitigates information leakage risks @xcite , enhances learning stability, and improves training data efficiency. inspiringly, @xcite discovers that even a carefully selected set of 1,000 unique questions can effectively train an llm to follow intents and achieve competitive performance when compared to openai's gpt3-davinci003 model. despite the impressive results of previous studies in detecting duplication @xcite ; @xcite ; @xcite , these methods do not consider intents in question, which may lead to sub-optimal results when deduplicating content. figure 1 shows two question pairs that illustrate the importance of considering intents in duplication detection for instruction-tuning. from a generic text-pair matching perspective, both the exemplar duplicate and neutral pairs are partially semantically matched. despite being mismatched in the non-intent part, the first pair is labeled as duplica-i am making a space shooter game. how to fire bullets in the direction a ship is facing? i have a space shooter game i am making. i want to play a sound every time a user hits the button fire. tion due to the matched question intent. meanwhile, the second pair only matches in non-intent content and thus should be treated as neutral. therefore, our goal is to highlight intents in duplicate question detection. however, this task remains challenging as follows: (1) accurately recognizing intent from users' posts is difficult due to the high diversity of intent in different qa forums, especially without sufficient human annotation. (2) determining whether two lexically different intents have the same semantics can be difficult because most annotations are at question-level instead of intent-level. (3) one question posted in online forums often carries more than one intent, and it is unclear how to handle multiple intents for duplicate question detection. to address the above challenges, we first leverage the characteristics in cqa forums and extract abundant training labels to both recognize and match intents without human annotation. based on matched intent pairs, we leverage supervised contrastive learning @xcite and properly learn intent representations in a low-dimensional space. finally, we design a similarity matrix on the question pair level, based on inter and intra similarities between multiple intents for each question to detect duplicate questions. the key contributions of this paper are summarized as follows: ## annotation-free label collection the primary challenge is the absence of labels for (1) intent recognition, which involves determining whether a sentence expresses an intent or not; ## method after label collection, we proceed to present our method for detecting duplicate questions. intent-dqd consists of three key components: intent recognition module, intent matching module, and duplicate question detection module. figure 3 shows the overall structure of our method. in the intent recognition module, we exploit the inner features of intent sentences to augment recognition accuracy. in the intent matching module, we leverage supervised contrastive learning @xcite and properly learn intent representations in a low-dimensional space. in the duplicate question detection module, we design a similarity matrix on the question pair level to integrate inter and intra-similarities between multiple intents. ## related work instruction tuning is commonly regarded as the first step in the rlhf training process for training human-aligned llms including gpt-3.5, gpt-4, claude @xcite . the goal of instruction tuning is to train llm in following human intents expressed in instructions @xcite @xcite . for instance, when a human asks a question to a llm, their intent is to receive a proper answer. before instruction tuning, a pre-trained llm may continue the question or generate similar questions, whereas an aligned llm trained through instruction tuning is more likely to generate an answer based on the question. collecting high-quality training data is fundamental for effective instruction tuning. cqa forums have millions of well-aligned q&a data, making them ideal sources for instruction tuning @xcite . however, the issue of duplicate questions arises as the volume of content within cqa continues to grow, posing a threat to content quality. recent research highlights the importance of detecting and removing duplication to optimize for llm instruction tuning. @xcite find deduplication increases generalization ability and learning stability. it can also save computation and decrease train-test set overlap @xcite . deduplication also reduces privacy risks because llm is vulnerable to privacy attacks when sensitive information is duplicated in the instruct-tuning dataset @xcite . existing duplication detection studies mainly build text-pair classifiers based on bi-encoders or cross-encoders. bi-encoders @xcite @xcite , encode each question individually while cross-encoders @xcite @xcite , use cross-attention between questions or self-attention by concatenating multiple questions. despite the popularity of crossencoders, they typically have high computational cost @xcite and inconsistent performance @xcite . in this work, we focus on bi-encoders, which are more computationally efficient and have shown consistent performance in duplicate question detection. however, the above-mentioned methods are generally generic text-pair matching and do not consider intents in question, which can lead to sub-optimal results in deduplicating datasets for instruction tuning. therefore, we incorporate intent features in intent-dqd to detect duplicate questions. unlike existing research on intents @xcite @xcite which typically rely on human annotation, our proposed method is annotation-free, which leverages characteristics of cqa forums and automatically mine labels for intents. ## conclusion in this work, we propose an intent-based duplicate question detection method intent-dqd. we design intent recognition to identify question intent without human annotation. intent matching employs contrastive learning to enhance the embeddings. duplicate question detection effectively aggregates intent-level relations and obtains question-level relations. evaluation on fifteen domains from two real-world datasets shows that our method outperforms existing state-of-the-art methods.