R epo4 QA : Answering Coding Questions via Dense Retrieval on G it H ub Repositories (2022)

Abstract: Open-source platforms such as GitHub and Stack Overflow both play significant roles in current software ecosystems. It is crucial but time-consuming for developers to raise programming questions in coding forums such as Stack Overflow and be navigated to actual solutions on GitHub repositories. In this paper, we dedicate to accelerating this activity. We find that traditional information retrieval-based methods fail to handle the long and complex questions in coding forums, and thus cannot find suitable coding repositories. To effectively and efficiently bridge the semantic gap between repositories and real-world coding questions, we introduce a specialized dataset named Repo4QA, which includes over 12,000 question-repository pairs constructed from Stack Overflow and GitHub. Furthermore, we propose QuRep, a CodeBERT-based model that jointly learns the representation of both questions and repositories. Experimental results demonstrate that our model simultaneously captures the semantic features in both questions and repositories through supervised contrastive loss and hard negative sampling. We report that our approach outperforms existing state-of-art methods by 3%-8% on MRR and 5%-8% on P@1.

## introduction with the increasing popularity of software development, stack overflow and github (two large-scale communities widely recognized in open source ecosystems) have attracted growing research interests. as the idiom goes, "don't reinvent the wheel". tackling programming problems with existing codes and documents is more effective and economical, since various resources in repositories can provide more helpful information than textformed answers. specifically, developers can get help and advice to solve the technical challenges they face, and be provided a variety of solutions and tools in repositories on github. as a vast number of challenges have already been considered and settled by the community, sophisticated schemes posted on github can help satisfy requirements or solve problems discussed in stack overflow. naturally, many answers in stack overflow provide links to github repositories, and a large amount of these answers are acknowledged to be high-quality and helpful by the community. this phenomenon is worth investigating to determine its contribution to the efficiency improvement and the code reuse of the open source ecosystem. motivated by the interplay between stack overflow and github, we introduce a novel questionrepository matching task. given a naturallanguage-formed question in the programming domain, we aim to search for the most relevant github repository from all candidate repositories as the answer. figure 1 illustrates the interaction between a question and a repository, where the key information for problem-solving is framed. to this end, we introduce repo4qa, a dataset consisting of 12,995 question-repository pairs foot_0 for complex coding question solving. the questions are collected from stack overflow, and the repositories are crawled from github through the hyperlinks provided in corresponding stack overflow answers. each repository is instrumental for troubleshooting confirmed by forum users with upvotes. such that proper repositories can be used as answers to coding questions. different from previous information retrieval (ir) tasks, the challenge of the proposed task lies in the long-form questions. typical ir methods and tricks, such as query expansion, are designed for short and keyword-based user inputs, while our task focuses on questions written in natural languages containing semantic information, which is harder to resolve. different from code searching task @xcite , our task is supposed to find a reasonable semantic alignment between two long-form sentences. questions and repositories have more complex structures and richer information than short-form web queries and code snippets. compared with communitybased qa task @xcite , our task is a cross-platform task resulting in semantic gaps between questions and answers, which is more challenging for traditional ir-based methods such as @xcite . besides, unlike common questions, questions about programming are more challenging and specialized. the terminology of programming is widely used to form questions; there are even some questions described with codes. moreover, complex models considering more interactions between qa pairs are more computationally expensive in our task. finally, the interactions inside queries or documents corpus should not be neglected. the relationship of the elements in the same semantic category is informative while discriminating positive samples from negatives. to address the aforementioned issues, we propose a novel jointly learning scheme, qurep, for question and repository pairs. qurep employs codebert @xcite as the initial weight of its transformer-based encoder. our work leverages the supervised contrastive loss for better separating elements from each other in the representation space. an unsupervised hard negative sampling strategy enables better discrimination performance. the shared weights between encoders reduce the computation cost, while the relevance between questions and answers can be ranked by the similarity score of embedding vectors. with experimental evaluation and comprehensive analyses, we show that our method strongly outperforms baselines. to summarize, the main contributions of this paper are concluded into three points respectively. ## conclusion in this paper, we introduce an automatically collected novel dataset, repo4qa, for the proposed task. furthermore, we propose qurep, a contrastive learning method to fine-tune codebert for our task. experimental results demonstrate that our method outperforms baseline models in effectiveness and efficiency. detailed analyses are conducted to investigate the impact on performance brought by components of our model. we look forward to other applications and more research interest in our task. moving forward, we plan to deploy our dataset and method to solve program-ming questions in software engineering practice and consider code stored in repositories for better modeling of multi-modal github repositories. zhou zhao, hanqing lu, vincent w zheng, deng cai, xiaofei he, and yueting zhuang. 2017. communitybased question answering via asymmetric multifaceted ranking network learning. in thirty-first aaai conference on artificial intelligence.