F uzz A ug: Data Augmentation by Coverage-guided Fuzzing for Neural Test Generation (2025)

Abstract: Testing is essential to modern software engineering for building reliable software.Given the high costs of manually creating test cases,automated test case generation, particularly methods utilizing large language models,has become increasingly popular.These neural approaches generate semantically meaningful tests that are more maintainable compared with traditional automated testing methods such as fuzzing.However, the diversity and volume of unit tests in current datasets are limited, especially for newer but important languages.In this paper, we present a novel data augmentation technique, *FuzzAug*,that brings the benefits of fuzzing to large language models by incorporating valid testing semantics and providing diverse coverage-guided inputs.Doubling the size of training datasets,FuzzAug improves performance over the baselines significantly.This technique demonstrates the potential of introducing prior knowledge from dynamic software analysisto improve neural test generation,offering significant enhancements in this task.Our code is open-sourced at https://github.com/SecurityLab-UCD/FuzzAug.

## introduction testing is one of the most important processes in software engineering, ensuring the quality and reliability of large software applications. unit tests are example-based self-assessment tests written and executed by the developer to demonstrate that the software works correctly as described in the design specification @xcite . however, despite its importance, developers do not always contribute new tests due to the difficulty of identifying which code to test, isolating them as finegrained units, and finding relevant inputs @xcite . heuristic-based automatic unit test generation @xcite is one solution to these issues, but the resulting tests are unsatisfactory in read-ability, correctness, and diversity of relevant inputoutput pairs @xcite . other popular automatic randomized testing methods, e.g. fuzzing @xcite , often ignores readability and focuses only on generating inputs to find new program behaviors, i.e. new coverage or crashes. however, these randomized testing methods only provide the input that triggers the bug with no valid semantics. these reported input seeds are usually not as informative as unit test functions in practice @xcite . therefore, finding semantically meaningful test cases correctly and effectively remains an unsolved problem. more recently, people have attempted to overcome these issues by leveraging the power of generative language models @xcite @xcite . large language models (llms) trained on large code corpora can write meaningful programs given text descriptions @xcite @xcite . therefore, with sufficiently large code and test datasets, we expect that llms could generate high-quality unit tests to assist human software engineers. however, testing functions typically occupy a minor fraction of a software repository, compared with regular functions for software features. @xcite found that in popular python and java repositories, test files comprise fewer than 20% of all code files. this deficiency in training data hampers the ability of llms to generate practical tests for production environments for two reasons: 1. the imbalance in training data causes the model to miss critical details in the units under test. 2. the insufficient amount of testing code presents a significant challenge in learning the representations of unit tests adequately. previous work addressed the imbalance issue by aligning code and tests into pairs @xcite . however, the second issue remains unsolved, and is further amplified by the trend of switching to newer pro-gramming languages for better maintainability and reliability, e.g. redesigning software in rust. a promising strategy to further enhance the existing state-of-the-art unit test datasets is designing a new specialized data augmentation (da) method for llm-based test generation. in computer vision, data augmentation typically involves applying randomized geometric or color transformations or injecting random noise to images in the training set. however, these methods are unsuitable for programming languages (pls) due to their formal grammar and strict semantics. limited research @xcite on da for pl is not suitable for test generation, as they do not introduce new test cases that explore the behavior of the program. unit test functions provide correct setups to invoke the functions under test (focal functions), and test inputs are fed to the focal functions to explore their functionality at run-time. consequently, a valid data augmentation method for test generation must incorporate semantically meaningful unit test functions, coupled with randomized yet valid testing inputs tailored to the specific functions under test. to address these challenges, we propose fuz-zaug. fuzzaug, as depicted in figure 1 , is a direct and effective data augmentation technique utilizing fuzzing data to enhance test generation with llms. fuzzing identifies vulnerabilities in software by randomly generating inputs to trigger new execution paths in software. these inputs capture the program's runtime behavior and thus can enhance the code understanding capabilities of llms @xcite . for implementing fuzzing data as a form of data augmentation, we perform code transformations on fuzz targets in libfuzzer @xcite to create new unit test functions. fuzzaug nearly doubles the limited amount of testing code in training datasets and provides a richer diversity of accurate and executable inputs for the focal functions. training llm-based test generation models with fuzzaug addresses the aforementioned issues by automatically providing unit test functions with high-quality test inputs. thus, fuzzaug is a novel approach in training practical llm-based unit test assistance, enhancing software robustness and maintaining test readability. to assess the effectiveness of fuzzaug, we conducted experiments with three different state-ofthe-art 7b open-source code generation models. each model was trained on two datasets: on the original unitsyn @xcite dataset and its fuzzaug-augmented counterpart. all three models trained with fuzzaug consistently outperformed their counterparts trained on only unitsyn, and outperformed the pre-trained/instruction-tuned baseline significantly. they demonstrated significant improvements in generating accurate test cases (assertions) and useful test functions that achieved higher code coverage. our contributions. 1. we introduce fuzzaug, a novel data augmentation method specifically designed for neural test generation llms to address the limitations of existing training datasets. 2. we build and release the rust version of unitsyn, aiming at training test generation models for rust programs. furthermore, we apply fuzzaug to this dataset and release the resulting augmented dataset, enhancing its utility for advanced model training. 3. we validate the efficacy of fuzzaug by training generative llms on the unitsyn dataset augmented by it. the notable improvement underscores the necessity and advantages of incorporating fuzzing-augmented testing functions into the training corpus, demonstrating the practical benefits of our approach. ## evaluation results we report our experimental results on the performance of neural test generation in this section. we categorize the models into three groups: pre-trained (pt), instruction-tuned (it), and fine-tuned (ft) models. pt and it models are the baselines, while ft models are further trained with unitsyn and fuzzaug. ## conclusion we developed fuzzaug, a data augmentation method for unit test function generation. fuz-zaug combines the advantages of coverage-guided fuzzing and generative large language models to generate tests that are not only semantically meaningful but also strategically comprehensive. we applied fuzzaug to fine-tune three state-of-the-art 7b open-source code generation models, demonstrating the effectiveness of fuzzaug. we collected our experimental dataset on rust crates that have pre-defined fuzzers as a rust extension to unitsyn. our method can be generalized to all languages that oss-fuzz supports with slight modifications. our results show the effectiveness of employing dynamic program analysis to generate high-quality inputs to augment the code-test datasets in training language models for unit test generation. we believe fuzzaug can spur the development of unit test generation by large language models and contribute to the field of ai for software testing.