MLA lgo-Bench: Can Machines Implement Machine Learning Algorithms? (2025)

Abstract: As machine learning (ML) application continues to expand across diverse fields, there is a rising demand for ML code generation. In this paper, we aim at a critical research question: Can machines autonomously generate ML code for sophisticated, human-designed algorithms or solutions? To answer this question, we introduce a novel benchmark, MLAlgo-Bench, which includes two challenging tasks: 1) Generating code for ML algorithms including both traditional ML and modern deep learning-based methods, and 2) Giving humans solution sketches, writing ML code for solving practical tasks in Kaggle competitions. This benchmark is unique in its focus on the challenges of interpreting intricate human instructions and producing multi-step, high-complexity code, offering a rigorous test for current Large Language Model (LLM) capabilities. We introduce an automatic evaluation framework with comprehensive metrics such as task pass rate, relative performance metric, and time overhead. Currently, the top-performing models (Claude3.5-Sonet) achieve a 48.8% task completion rate on realizing machine learning algorithms, and a 21.6% rate for completing Kaggle competitions. Further analysis suggests substantial room for improvement.

## introduction recent advancements in large language models (llms) @xcite @xcite have demonstrated remarkable performance across various domains. in this context, the code generation capabilities of llms @xcite @xcite have attracted significant attention thanks to their potential to greatly enhance the efficiency of profession-* equal contribution. recently, new interest has emerged in benchmarking the capabilities of llm agents to perform ml tasks @xcite . this area of research holds great promise, as ml is closely linked to the rapidly advancing field of artificial intelligence. while recent ml benchmarks @xcite @xcite provide valuable insights, they primarily evaluate llms' ability to solve ml tasks autonomously without human-designed solutions or algorithms. in this paper, we contend that the development of effective ml agents fundamentally depends on their capacity to follow sophisticated, human-designed solutions. mastery of this ability opens new opportunities for llms to assist human practitioners and researchers in translating highlevel ideas-such as those found in ml research papers-into practical implementations, significantly reducing the effort required for experimentation. to comprehensively evaluate the capabilities of llms in the implementation of ml methods, we propose a benchmark called mlalgo-bench. our benchmark consists of two settings. setting 1 is designed to assess whether llms can successfully implement ml modules based on detailed algorithm descriptions. it includes a total of 121 tasks, covering not only traditional machine learning algorithms like random forest and svm but also a wide range of deep learning algorithms, such as transformer modules and their various variants. setting 2 focuses on evaluating whether llms can successfully complete machine learning competition by generating the full workflow code based on a task description and the corresponding solution ideas, which are provided by human top competitors. for each task in our benchmark, we provide benchmark time source avg input lens num metrics conala @xcite 2018 stack overflow 9.6 2879 bleu humaneval @xcite 2021 -131.3 164 pass@k mbpp @xcite 2021 -16.1 974 pass rate pandaseval @xcite 2022 stackoverflow 68.3 101 pass@k numpyeval @xcite 2022 stackoverflow 69.1 101 pass@k ds1000 @xcite 2022 stackoverflow 282.4 1000 pass@k + sf-contrains swe-bench @xcite 2023 github 480.9 2294 pass rate + apply classeval @xcite 2024 manual 123.7 100 pass@k ml-bench @xcite 2023 github 781.6 9641 pass rate/ pass@k mlagentbench @xcite 2023 kaggle and others 108.7 13 pass rate mlgym-bench @xcite 2025 canonical tasks 387.5 13 aup scores @xcite 2024 kaggle 2106.7 76 pass@k / kaggle ranks mlalgo-bench (ours) 2024 scikit-learn/ labml-github/ kaggle 1449.8 218 pass rate/ performance/ time overhead / instruction-following compared to existing code generation benchmarks @xcite @xcite , mlalgo-bench presents greater challenges for llms. first, previous benchmarks typically involve generating only short code snippets (function-level code) such as humaneval @xcite , ml-bench @xcite . in contrast, each task in our benchmark requires module-level implementation, with more lengthy output code. second, the tasks in our benchmark generally involve more complex code instructions. for setting 1, the instructions often include intricate workflows and mathematical formulas. in setting 2, our instructions encompass detailed descriptions of machine learning tasks, datasets, and solutions -a composite ml method with feature engineering. in other words, mlalgo-bench places higher demands on their ability to understand and follow technical instructions. please refer to we evaluate the code generation capabilities of several llms on our benchmark. in setting 1, the best-performing llm is claude-3.5-sonnet, achieving a pass rate of 48.8%. in setting 2, the top performer is also claude-3.5-sonnet, with a pass rate of 21.6%. we provide a detailed analysis of the common error types made by llms, and conduct an in-depth investigation into several factors: 1) the impact of methodology specification on ml method implementation; 2) a de-tailed evaluation and analysis was conducted on the instruction-following capabilities of llms; 3) the impact of an agentic framework on the ability to correctly implement a given solution. in summary, the paper makes the following contributions: ## conclusion this paper introduces mlalgo-bench for evaluating llms in the task of implementing machine learning (ml) methods. we provide detailed instruction annotations, an automated evaluation framework, as well as code and performance reference to facilitate future research. additionally, we evaluate the performance of current mainstream llms on mlalgo-bench, offering a comprehensive analysis of their capabilities in this domain. a deeper analysis reveals several areas for improvement, including the enhancement of mathematical understanding, better comprehension of data descriptions, improved function call capabilities, and more robust instruction-following abilities. addressing these issues would enable llms to generate more precise and reliable ml implementations. such improvements could significantly benefit ml agents @xcite , allowing them to complete ml tasks more effectively with fewer interactive steps.