MQDD : Pre-training of Multimodal Question Duplicity Detection for Software Engineering Domain (2023)

Abstract: This work proposes a new pipeline for leveraging data collected on the Stack Overflow website for pre-training a multimodal model for searching duplicates on question answering websites. Our multimodal model is trained on question descriptions and source codes in multiple programming languages. We design two new learning objectives to improve duplicate detection capabilities. The result of this work is a mature, fine-tuned Multimodal Question Duplicity Detection (MQDD) model, ready to be integrated into a Stack Overflow search system, where it can help users find answers for already answered questions. Alongside the MQDD model, we release two datasets related to the software engineering domain. The first Stack Overflow Dataset (SOD) represents a massive corpus of paired questions and answers. The second Stack Overflow Duplicity Dataset (SODD) contains data for training duplicate detection models.

## introduction the benefits of question-answer (qa) networks for software developers such as the stack overflow website are widely exploited by professionals and beginners alike during the software creation process. many solutions to various problems, short tutorials, and other helpful tips can be found on these networks. however, access to this valuable source of information highly depends on users' ability to search for the answers. in our paper, we introduce a multimodal method for detecting duplicate questions. apart from the primary use to prevent posting duplicate questions, this technique can be directly used for better search. when users are posting already answered questions, they can get the answer immediately without the necessity to wait until someone else links the duplicate post or answers their question. the duplicate question detection task aims to classify whether two questions share the same intent. in other words, if two questions are duplicates, they relate to the same answer. the duplicate detection task is quite challenging since the classifier needs to distinguish tiny semantic nuances that can significantly change the desired answer. the posts in the qa networks for software development often intermix natural language with source code snippets. the great success of neural networks for natural language processing (nlp) encourages us to build a bi-modal natural language (nl) and programming language (pl) encoder for duplicate detection @xcite on questionanswering platforms such as stack overflow. current state-of-the-art nlp methods build on large pre-trained models, leveraging transformer architecture @xcite . the transformer-based models such as bert @xcite , @xcite , roberta @xcite , or t5 @xcite are usually pre-trained on massive unlabeled corpora and applied to a task with much less training data afterward. we follow this idea and introduce the pre-training phase into our solution. to achieve the best possible results, we design duplicate-detectionspecific pre-training objectives (see section 3.3). since the source code snippets present in the stack overflow questions may be relatively long, we choose to base our model on the longformer architecture @xcite ; whose modified attention scheme scales linearly with the sequence length. the resulting model with ≈146m parameters is firstly pre-trained on a large semi-supervised corpus of stack overflow questions and answers. for detailed information about the dataset and pretraining, see section 3. afterward, in section 4, we fine-tune the obtained model on the duplicate detection task and compare our model with codebert @xcite , which represents another nl-pl multimodal encoder. we also compare our model to a randomly initialized longformer @xcite and pre-trained roberta @xcite to see whether the pre-training of both models brings a significant improvement of the achieved results. the previously described experiments are visualized in figure 1 . at the end of this paper, we explore how well our model generalizes to other tasks by applying our model to the codesearchnet dataset @xcite in section 5. our main contributions are: 1) we release a finetuned multimodal question duplicity detection (mqdd) model for duplicate question detection. the model is mature enough to be deployed to stack overflow, where it can automatically link duplicate questions and, therefore, improve users' ability to search for desired answers. furthermore, we release the pre-trained version of the encoder, so other researchers may reuse the most computationally intensive phase of our model training. 2) we present and explore the effect of entirely new pre-training objectives specially designed for duplicate detection. 3) we release a stack overflow dataset (sod) that can be used for pre-training models in a software engineering domain. furthermore, we release a novel stack overflow duplicity dataset (sodd) for duplicate question detection, enabling other researchers to follow up on our work seamlessly. ## related work the naturally collected massive amounts of data in software management systems, issue tracker tools, and versioning systems makes the software development an ideal domain to apply deep models to increase work effectiveness. codex @xcite represents a large pretrained neural network model that can generate source code for the software engineering domain. it is designated for source code generation. its slightly modified form is also integrated with the github copilot foot_0 system, a digital pair programmer. codet5 @xcite is another model that also works with source code. it demonstrates the capability of solving multiple tasks thanks to converting all problems into a unified sequence-tosequence form. different approach is introduced in the paper by @xcite , which translates source codes into a natural language to retrieve similar code snippets. the previous papers build upon the architecture of the transformer @xcite , which can be pre-trained on a massive corpus on unlabeled data, and applied on a downstream task only with much less demanding fine-tuning. this approach is used by bert @xcite , which employs the transformer encoder to produce contextual representations of input tokens. these contextual embeddings @xcite can then be utilized for various tasks, including the classification of entire sequences @xcite or individual tokens @xcite . such success can probably be attributed to a welldesigned attention mechanism @xcite , which allows the model to capture contextual information from the entire sequence being processed. the results obtained using large pre-trained model can be significantly influenced by the correct choice of training objective. adapting the pre-training phase and finding a proper objective allows the model to exploit useful features from large source of data. for example, roberta @xcite slightly modifies the masked language modeling (mlm) objective and abandons the next sentence prediction (nsp) to improve the achieved results. different way of improving results is represented by the changes in the architecture of the model. for example, longformer @xcite model significantly modifies the attention mechanism to mitigate the o(n 2 ) complexity of a vanilla attention enabling processing of longer sequences. the whole concept of pre-trained encoders laid out by bert @xcite is often applied to multimodal data as well. this enables, for example, a unified processing source codes and natural texts. the produced contextual embeddings of source code and text @xcite is then directly applicable to downstream tasks such as code similarity, code search, or code fixing @xcite . the cubert @xcite ) is an example of a multimodal encoder for python source codes and texts. the model outperforms bilstm @xcite and randomly initialized transformer @xcite approach in five different tasks, including classification of variable misuse, wrong binary operator usage, swapped operands, and functiondocstring match. another representative of multimodal source code encoders is the codebert model @xcite pre-trained on a multilingual corpus of source codes from six different programming languages. the codebert builds upon the roberta @xcite and follows the generator-discriminator approach laid out in electra @xcite . the resulting model shows superior results in code search, natural language-programming language (nl-pl) probing, and documentation generation. our work differs from the previous multimodal source code encoders in the following points: 1) our model is trained using novel pre-training objectives targeting specifically the duplicate detection task. 2) unlike the cubert, explicitly designated for python and codebert, pre-trained on six different programming languages, our model is capable of processing inputs from an arbitrary programming language enabling it to be deployed to real-world question-answering platforms. 3) our mqdd model employs a transformer-based architecture with an attention scheme scaling linearly with sequence length allowing it to process long sequences in a reasonable time. ## model pre-training this section describes the pre-training procedure, including the construction of the new dataset from the stack overflow, the definition of the learning objectives, and the model itself. ## duplicate question detection following the pre-training phase, this section focuses on applying the obtained model to the task of duplicate detection. in the first part, we describe the construction of a new dataset for duplicate detection. the next part presents how we integrate the pre-trained model into a two-tower neural network. at the end of this section, we describe the concluded experiments and present the results. ## generalization to other tasks to explore how well our model generalizes to other tasks, we choose the code search task. the information retrieval seems to be close to our pretraining tasks. for all the experiments, we use the codesearchnet dataset @xcite containing approximately 2.3m examples from six different programming languages extracted from github repositories. ## discussion as the results stated in sections 4.4 and 5.3 suggest, our model excels in detecting duplicates but lags in source code retrieval. we expected the dominance of our model in the duplication detection task. however, an interesting observation is that the pretraining of the codebert, whose author's @xcite initialized it using the roberta's @xcite weights, does not bring any improvement when applied to the duplicate detection. on the other hand, it is surprising that our mqdd model does not perform comparably well as the codebert on the code search as our pre-training objectives require the model to build a deep understanding of the processed source code. this can be explained by the fact that the datasets used for pre-training of both models have very different characteristics. the sod does not contain source code from a constrained set of six programming languages (see however, the offered explanation does not cover that roberta, whose pre-training dataset did not contain any source code, outperforms our model in the code search task. we speculate that this can be caused by the mqdd model being trapped in its local optimum due to its pre-training designed especially for the duplicate detection. this can make it difficult to get out of this local optimum when fine-tuned on a slightly different dataset and task. this phenomenon is often referred to as a negative transfer @xcite and can be caused, among other things, by the discrepancy between the pre-training and fine-tuning domains. given that our research aimed to build a model designed directly for the detection of duplicates on platforms such as stack overflow, it can be stated that the results we achieve are satisfactory. our model far exceeds the results achieved by competitive work on a task that can be perceived as more demanding due to the need to process a general source language and distinguish seemingly insignificant semantic nuances. for example, questions "how to implement a producer-consumer in java" and "how to implement a producer-consumer in c++" must be identified as different since the answers would significantly differ. ## future work our work opens up further opportunities to build on our current research. first of all, it would be interesting to explore methods that would eliminate the effect of negative transfer and thus allow the use of our pre-trained model in other tasks. furthermore, the follow-up work can integrate our model into a production-ready duplicate detection system employing a fast vector space search library such as faiss. the proposed system can be further extended by a duplicate detection model that jointly processes both questions allowing the attention mechanism to attend across both inputs. such a model can potentially achieve better results and be deployed along with our two-tower-based model. our two-tower model would then be used to filter out candidate duplicate questions. afterward, the cross-attention model could verify that the candidate questions are indeed duplicates more accurately. ## conclusion this work presents a new pre-trained bert-like model that detects duplicate posts on programmingrelated discussion platforms. based on the longformer architecture, the presented model is pretrained on our novel pre-training objectives (qa and sp) that aim to target the duplicate detection task. the comparison with the competitive code-bert model shows that our model outperforms other approaches, suggesting the effectiveness of our learning objectives. furthermore, we investigated the generalization capabilities of our model by applying it to a code retrieval task. in this task, it turned out that our model does not exceed the results achieved with either codebert or the more general roberta model. we attribute these findings to the significant differences between our pretraining dataset and the evaluation dataset for the code search task. therefore, we consider our model an excellent choice for solving duplicate detection. however, it seems to be too specialized to solve other tasks well. our models are publicly available for research purposes in our hugging face foot_8 and github foot_9 repositories.