OOP : Object-Oriented Programming Evaluation Benchmark for Large Language Models (2024)

Abstract: Advancing automated programming necessitates robust and comprehensive code generation benchmarks, yet current evaluation frameworks largely neglect object-oriented programming (OOP) in favour of functional programming (FP), e.g., HumanEval and MBPP. To address this, our study introduces a pioneering OOP-focused benchmark, featuring 431 Python programs that encompass essential OOP concepts and features like classes and encapsulation methods. We propose a novel evaluation metric, pass@o, tailored for OOP, enhancing traditional pass@k metric. Our evaluation of 23 leading large language models (LLMs), including both general and code-specialized models, reveals three key insights: 1) pass@o offers a more relevant and comprehensive assessment for OOP code generation; 2) Despite excelling in FP, code-specialized LLMs like WizardCoder lag in OOP compared to models like ChatGPT; 3) The poor performance of all advanced LLMs on our OOP benchmark highlights a critical need for improvements in this field. Our benchmark and scripts will be publicly released at GitHub.

## introduction large language models @xcite , consisting of billions or even trillions of parameters' transformer blocks @xcite , have emerged like mushrooms after the rain, especially since the emergence of chatgpt 1 . in comparison to small models, llms exhibit stronger generalization and reasoning capabilities @xcite . currently, llms are playing a crucial role in various tasks, e.g., code generation @xcite @xcite , language understanding @xcite , human-computer interac- figure 1 : the performance comparison of widelyused code language models on functional programming (fp) and object-oriented programming (oop) code generation benchmarks, in terms of pass@1 scores. we see that all models perform relatively well on fp benchmarks, i.e., humaneval @xcite and mbpp @xcite , while exhibiting poor performance on our oop benchmark. humaneval @xcite 164 en python function programming mbpp @xcite 974 en python function programming apps @xcite 5000 en python function programming codecontests @xcite 165 en multi function programming multipl-mbpp @xcite 974 * multi multi function programming humaneval-x @xcite 164 * en multi function programming multipl-humaneval @xcite 164 * en multi function programming mtpb @xcite 115 en python function programming odex @xcite 945 multi python function programming pandaseval @xcite 101 en python function programming big-bench @xcite 32 en python function programming codeapex ⋆ @xcite 476 * zh&en c++ function programming classeval @xcite 100 en python class-level programming oop (our) 431 en python object-oriented programming ## related work code evaluation benchmark in the early days of llms, researchers from google and openai launched artificial handwritten code evaluation benchmarks, namely mbpp @xcite and humaneval @xcite , respectively. mbpp and humaneval are currently the mainstream code generation evaluation benchmarks, but both of them are based on the python programming language. subsequently, multipl-mbpp (cassano et al., 2023) and multipl-humaneval (cassano et al., 2023) expanded upon these two benchmarks by translating the python programming language into eighteen other programming languages, e.g., java, c++, php, etc, to evaluate the performance of llms across others programming languages. additionally, humaneval-x (zheng et al., 2023) incorporated multiple test cases into the hu-maneval benchmark. apart from the extensions made to these two benchmarks, other benchmarks like codeapex (fu et al., 2023) and odex (wang et al., 2022) exhibit distinctive features across different natural languages and task types. classeval (du et al., 2023) focuses on class tasks. unlike existing code evaluation benchmarks, our proposed oop benchmark primarily focuses on the concepts and features of oop, e.g., class, inheritance, etc. these works are summarized in code evaluation metrics existing evaluation metrics can be broadly categorized into two types: dynamic evaluation metrics and static evaluation metrics. dynamic evaluation metrics evaluate the executability of generated codes by using test cases, with pass@k @xcite serving as the primary representative. the calculation process for pass@k is shown in appendix a. additionally, this category of metrics includes n@k @xcite . static evaluation metrics calculate bleu @xcite , rouge @xcite , codescore @xcite and @xcite among manually written examples and generated programs. however, these code evaluation metrics do not specifically focus on evaluating the concepts and features of oop. therefore, we further optimized the pass@k metric based on the evaluation benchmark for oop. ## discussion in this section, we will explore the reasons behind the generally lower scores of llms in oop, as well as the applicability of the chain-of-thoughts similarly, it can be deduced that "def _" and "def __" serve as indicators for private function names; "def" signifies a public function name; and "def __init__" represents the indicator for attribute variables name. moreover, in our oop benchmark, the llm should ideally generate at least 86,200 "class", 36,000 "def __" or "def _", 86,200 "def", and 70,800 "def __init__".) why llms score lower in oop benchmarks? we use the experimental results of chatgpt and codellama-34b on pass@1 as examples for analysis. as we instruct llms to generate relevant class names, private function names, public function names, etc., we conducted searches using simple keywords, e.g., "class", "def _", "def __", "def __init__", and "def" on both codellama-34b and chatgpt results. a detailed description of the retrieval process is provided in appendix f. we compiled and analyzed the distribution of retrieval "class", "def _", "def __", "def __init__", and "def", as shown in figure 5 , concluding that: 1) weak knowledge, e.g., class, encapsulation methods, etc, of oop in llms; 2) llms particularly lack cognition of private functions; 3) there is a certain degree of gap between codellama-34b and chatgpt. specific example is shown in figure 11 . the scores of the codellama-13b, starcoder, and wizardcoder-15b models on few-shot cot are also lower than their scores on few-shot. we analyzed the experimental results of zero-shot and zero-shot cot and found that using the cot method introduces an illusion to the model, preventing it from directly generating the corresponding code, as illustrated in figure 12 . therefore, it is necessary to integrate the concepts and features of oop to design appropriate cot strategies in order to enhance the effectiveness of generating oop by llms. appendix g provides detailed prompts for few-shot, zero-shot cot, and few-shot cot. ## conclusion in this paper, we propose the first oop evaluation benchmark based on python, consisting of 431 python programs, encompassing key concepts and features of oop, e.g., class, encapsulation methods, etc. simultaneously, we propose the evaluation metric pass@o for the oop benchmark. pass@o improves upon the limitations of pass@k by matching keyword points between natural language with program language. we evaluate 23 mainstream llms using the proposed oop benchmark and pass@o metric. experimental results show that the current oop of llms is far from ideal, which also reveals that llms have room for further improvement. furthermore, existing llms have a certain gap with chatgpt in oop. moreover, we also investigate that applying some of the current improvement strategies directly to the oop benchmark does not show significant improvement. in the future, we need to further strengthen the oop knowledge of llms, especially regarding private functions. at the same time, we also hope that more researchers can contribute to the advancement of research in oop.