Understanding Programs by Exploiting (Fuzzing) Test Cases (2023)

Abstract: Semantic understanding of programs has attracted great attention in the community. Inspired by recent successes of large language models (LLMs) in natural language understanding, tremendous progress has been made by treating programming language as another sort of natural language and training LLMs on corpora of program code. However, programs are essentially different from texts after all, in a sense that they are normally heavily structured and syntax-strict. In particular, programs and their basic units (i.e., functions and subroutines) are designed to demonstrate a variety of behaviors and/or provide possible outputs, given different inputs. The relationship between inputs and possible outputs/behaviors represents the functions/subroutines and profiles the program as a whole. Hence, we propose to incorporate such a relationship into learning, for achieving a deeper semantic understanding of programs. To obtain inputs that are representative enough to trigger the execution of most part of the code, we resort to fuzz testing and propose fuzz tuning to boost the performance of program understanding and code representation learning, given a pre-trained LLM. The effectiveness of the proposed method is verified on two program understanding tasks including code clone detection and code classification, and it outperforms current state-of-the-arts by large margins. Code is available at https://github.com/rabbitjy/FuzzTuning .

## introduction code intelligence powered by machine learning has attracted considerable attention in both the ai and software engineering community. particularly, code representation learning, which aims to encode functional semantics of source code, lays the foundation for achieving the intelligence and is of * equal contribution † corresponding author great interest. the learned representation can be applied to various downstream tasks, including code classification @xcite , code summarization @xcite , clone detection @xcite , etc. many efforts inspired by the developments of natural language understanding have been devoted to learning code representations, among which it has been increasingly popular to adopt large language models (llms) that are capable of learning contextual information from data at scale @xcite . the llms can then be fine-tuned on domain-specific code to achieve superior performance compared with tradition models. despite being effective, these natural language processing methods do not fit perfectly for handling programs. specifically, programs are heavily structured and syntax-strict (to be understood by compilers or interpreters), while natural language corpus is not. as basic units of programs, functions and subroutines can take a variety of argument values to demonstrate different logical behaviors or return different results. that being said, the relationship between inputs and possible outputs/behaviors essentially represents the functions/subroutines and further the whole programs. in this paper, we propose to incorporate such a relationship into learning for a deeper understanding of programs. in fact, given enough inputs to execute all pieces of the code, then the outputs would include enough runtime information we need to profile and understand the program. however, it is nontrivial to generate a limited number of inputs that are representative enough to execute every part of the code. without a proper strategy, we may end up with a large number of inputs that execute similar parts of codes. to address the issue, we opt to utilize fuzz testing (also known as fuzzing) @xcite , which is a common software testing practice and dynamic analysis tool whose original goal is to find software bugs by executing as much code as possible. more specifically, we repurpose fuzz testing to generate input and output data for assisting code representation learning, and we demonstrate how the input and output data (i.e., test cases) can be appropriately incorporated into existing llms to achieve superior program understanding performance. the contributions of this paper are three-fold. first, by recognizing the essence code representation learning, we propose to take advantage of the relationship between inputs and possible outputs for achieving a deeper understanding of programs. second, we, for the first time, repurpose fuzz testing to assist code presentation learning, marrying these two concepts from different communities for achieving more powerful ai. third, we obtain stateof-the-art results on typical program understanding tasks including clone detection and code classification, in comparison to prior arts. ## related work in this section, we introduce related work on code understanding (from the natural language understanding community) and fuzzing (from the software engineering community). ## method as mentioned in section 1, programs show strict syntax. to inspire deeper understanding of the syntax and logical behaviors of a program or functions/subroutines (which are the building blocks of the program), we attempt to exploit the relationship between their inputs and possible outputs/behaviors for achieving improved understanding of programs and code, akin to how engineers understands thirdparty code. however, with existing learning techniques, it seems nontrivial to generate inputs that could lead to execution of sufficient part of the code, thus we resort to fuzzing to achieve this goal. ## experimental results in this section, we report experimental results to verify the effectiveness of our fuzz tuning. we consider popular tasks (i.e., clone detection and code classification) and datasets involving mainstream languages including c, c++, java, and python. experiments were performed on nvidia v100 gpus using pytorch 1.7.0 @xcite implementations. ## conclusion in this paper, we have pointed out that exploiting informative test cases helps to understanding of programs. we have developed fuzz tuning, a novel method which takes advantage of fuzzing together with prior large-scale pre-training effort to achieve this goal. our fuzz tuning repurposes traditional fuzzers to generate informative test cases that wellrepresent the functionality of programs and it introduces appropriate cloze prompts to incorporate the test cases into being processed. by performing comprehensive experiments on two datasets and two program understanding tasks, we have verified the effectiveness of the method and achieved new state-of-the-arts.