Code and Named Entity Recognition in S tack O verflow (2020)

Abstract: There is an increasing interest in studying natural language and computer code together, as large corpora of programming texts become readily available on the Internet. For example, StackOverflow currently has over 15 million programming related questions written by 8.5 million users. Meanwhile, there is still a lack of fundamental NLP techniques for identifying code tokens or software-related named entities that appear within natural language sentences. In this paper, we introduce a new named entity recognition (NER) corpus for the computer programming domain, consisting of 15,372 sentences annotated with 20 fine-grained entity types. We trained in-domain BERT representations (BERTOverflow) on 152 million sentences from StackOverflow, which lead to an absolute increase of +10 F1 score over off-the-shelf BERT. We also present the SoftNER model which achieves an overall 79.10 F-1 score for code and named entity recognition on StackOverflow data. Our SoftNER model incorporates a context-independent code token classifier with corpus-level features to improve the BERT-based tagging model. Our code and data are available at: https://github.com/jeniyat/StackOverflowNER/

## introduction recently there has been significant interest in modeling human language together with computer code @xcite @xcite , as more data becomes available on websites such as stackoverflow and github. this is an ambitious yet promising direction for scaling up language understanding to richer domains. access to domain-specific nlp tools could help a wide range of downstream applications. for example, extracting software knowledge bases from 1 our code and data are available at: https:// github.com/jeniyat/stackoverflowner/ text @xcite , developing better quality measurements of stackoverflow posts @xcite , finding similar questions (amirreza shirani, 2019) and more. however, there is a lack of nlp resources and techniques for identifying software-related named entities (e.g., variable names or application names) within natural language texts. in this paper, we present a comprehensive study that investigates the unique challenges of named entity recognition in the social computer programming domain. these named entities are often ambiguous and have implicit reliance on the accompanied code snippets. for example, the word 'list' commonly refers to a data structure, but can also be used as a variable name (figure 1 ). in order to recognize these entities, we propose a software-related named entity recognizer (soft-ner) that utilizes an attention network to combine the local sentence-level context with corpuslevel information extracted from the code snippets. using our newly annotated corpus of 15,372 sentences in stackoverflow, we rigorously test our proposed softner model, which outperforms bilstm-crf model and fine-tuned bert model for identifying 20 types of software-related named entities. our key contributions are the following: overall, our named entity tagger (softner) achieves a 79.10% f 1 score on stackoverflow and 61.08% f 1 score on github data for extracting the 20 software related named entity types. we believe this performance is sufficiently strong to be practically useful. we have released our data and code, including the named entity tagger, our annotated corpus, annotation guideline, a specially designed tokenizer, and pre-trained stackoverflow bert and elmo embeddings. ## annotated stackoverflow corpus in this section, we describe the construction of our stackoverflow ner corpus. we randomly selected 1,237 question-answer threads from stack-overflow 10-year archive (from september 2008 to march 2018) and manually annotated them with 20 types of entities. for each question, four answers were annotated, including the accepted answer, the most upvoted answer, as well as two randomly selected answers (if they exist). ## named entity recognition models the extraction of software-related named entities imposes significant challenges as it requires resolving a significant amount of unseen tokens, inherent polysemy, and salient reliance on context. unlike news or biomedical data, spelling patterns and long-distance dependencies are more crucial in the software domain to resolve ambiguities and categorize unseen words. taken in isolation, many tokens are highly ambiguous and can refer to either programming concepts or common english words, such as: 'go', 'react', 'spring', 'while', 'if ', 'select'. to address these challenges, we design the softner model that leverages sentential context to disambiguate and domain-specific character representations to handle rare words. figure 2 shows the architecture of our model, which consists of primarily three components: ## evaluation in this section, we show that our softner model outperforms all the previous ner approaches on the stackoverflow and github data. we also discuss the factors pivotal to the performance of our model, namely pre-trained in-domain bert embeddings and two domain-specific auxiliary tasks. ## domain adaptation to github data the code recognizer and segmenter vectors to the github elmo embeddings. we think one likely explanation is that github data contains less coderelated tokens when compared to stackoverflow. the percentage of code-related entity tokens is 63.20% in github and 77.21% in stackoverflow. overall, we observe a drop of our softner tagger from 79.10 f 1 on stackoverflow ( the conll 2003 dataset @xcite ) is a widely used benchmark for named entity recognition, which contains annotated newswire text from the reuters rcv1 corpus. state-of-the-art approaches on this dataset @xcite use a bidirectional lstm @xcite with conditional random field @xcite and contextualized word representations @xcite @xcite . named entity recognition has been explored for new domains and languages, such as social media @xcite @xcite @xcite , biomedical texts @xcite @xcite , multilingual texts @xcite and codeswitched corpora @xcite . various methods have been investigated for handling rare entities, for example incorporating external context @xcite or approaches that make use of distant supervision @xcite @xcite . there has been relatively little prior work on named entity recognition in the software engineering domain. @xcite annotated 4,646 sentences from stackoverflow with five named entity types (programming language, platform, api, tool-library-framework and software standard). the authors used a traditional feature-based crf to recognize these entities. in contrast, we present a much larger annotated corpus consisting of 15,372 sentences labeled with 20 fine-grained entity types. we also develop a novel attention based neural ner model to extract those finegrained entities. ## conclusion in this work, we investigated the task of named entity recognition in the social computer programming domain. we developed a new ner corpus of 15,372 sentences from stackoverflow and 6,510 sentences from github annotated with 20 fine-grained named entities. we demonstrate that this new corpus is an ideal benchmark dataset for contextual word representations, as there are many challenging ambiguities that often require long-distance context to resolve. we also proposed a novel attention based model, named soft-ner, that outperforms the state-of-the-art ner models on this dataset. furthermore, we investigated the important sub-task of code recognition. our code recognition model captures additional spelling information beyond then contextual word representations and consistently helps to improve the ner performance. we believe our corpus, stackoverflow-specific bert embeddings and named entity tagger will be useful for various language-and-code tasks, such as code retrieval, software knowledge base extraction and automated question-answering.