Representation and Generation of Machine Learning Test Functions (2024)

Abstract: Writing tests for machine learning (ML) code is a crucial step towards ensuring the correctness and reliability of ML software. At the same time, Large Language Models (LLMs) have been adopted at a rapid pace for various code generation tasks, making it a natural choice for many developers who need to write ML tests. However, the implications of using these models, and how the LLM-generated tests differ from human-written ones, are relatively unexplored. In this work, we examine the use of LLMs to extract representations of ML source code and tests in order to understand the semantic relationships between human-written test functions and LLM-generated ones, and annotate a set of LLM-generated tests for several important qualities including usefulness, documentation, and correctness. We find that programmers prefer LLM-generated tests to those selected using retrieval-based methods, and in some cases, to those written by other humans.

## introduction as ai and ml become more and more integrated into everyday processes, ensuring the quality and reliability of these ml models is mandatory, and a critical part of ensuring ml models' performance in production is having good, representative test cases. traditionally, these tests have been written by engineers and programmers, a process that, while valuable, can be time-consuming and requires extensive experience and expertise in ml methodology. recognizing the challenges posed by the intricacies of ml code, particularly the distinct nature of ml testing involving both pre-training and post-training tests, our research takes a deliberate focus on this specific domain. this choice serves to constrain the scope of our investigation and allows us to address the unique complexities associated with ml testing, which often deviates from conventional software testing. one possible way to aid programmers is to retrieve existing functions that have been previously implemented, similarly to what has been done for test case selection within a test suite @xcite . for the purpose of writing tests, relevant test cases could be retrieved from other projects that are written to test functions that are semantically similar to the programmers' target functions. these retrieved functions might serve as references for programmers to consider when developing their own tests. however, with the recent advent of powerful code-generating llms such as codex @xcite and llama @xcite , those seeking to develop ml test cases are now able to prompt the model given the source function and instructions required to produce the appropriate test case. this has the potential to revolutionize the way that ml tests are developed, and it is therefore important to analyze how ai-generated tests compare to those written by humans and how developers may consider using these methods. in this work, we make initial steps toward comparing the ml test functions that are generated by llms with those generated by human programmers to better anticipate the consequences of a growing number of ml test functions being generated automatically by llms. using a set of approximately 10,000 pairs of ml functions and their tests, we use code embedding methods to explore the semantic relationships between functions and their tests. we then experiment with semantic retrieval-based approaches to find relevant ml tests given an input test function, and finally, we compare several models' ability to generate useful ml test functions and evaluate them using expert human annotations. an overview of the process that we used is presented in figure 1 . focusing on the specific domain of ml allowed us to make the focal methods more comparable and facilitated the recruitment of participants familiar with ml algorithms. our core contributions include: (1) exploration of semantic representations for ml functions and their tests; (2) experiments with retrieval of ml tests; and it's also important to mention the effort on benchmarking datasets like codesearchnet @xcite and codexglue @xcite , which offer robust platforms for training and evaluating numerous models in this area. however, none of the previous evaluation datasets checked for the quality of ml function and test matching, that is, evaluating which approaches are best able to retrieve a test case given an input function (rather than a search query written in natural language). also, the aforementioned methods are not full language models that can generate code for any language. some only work on one language and are not necessarily applicable to the ml domain. ## conclusion in this work, we employed state-of-the-art nlp techniques to generate effective representations for ml source and test code. we developed a heuristic method to build a good-quality dataset of ml function-to-test mappings, forming the basis for generating these representations. we have studied these representations through visualization by leveraging a couple of dimensionality reduction methods, and we have successfully captured some patterns, that we later confirmed. our findings revealed an interesting insight: the codebert model struggled to capture test case semantics compared to other recent gpt embeddings. we also explored the practicality of these representations for retrieving an ml test case given an ml method. surprisingly, even state-of-the-art nlp models faced challenges in this task. we also assessed the performance of llms in automatically generating test cases, which revealed that some of these models outperformed human-generated tests in terms of helpfulness.