MLC opilot: Unleashing the Power of Large Language Models in Solving Machine Learning Tasks (2024)

Abstract: The field of machine learning (ML) has gained widespread adoption, leading to significant demand for adapting ML to specific scenarios, which is yet expensive and non-trivial. The predominant approaches towards the automation of solving ML tasks (e.g., AutoML) are often time-consuming and hard to understand for human developers. In contrast, though human engineers have the incredible ability to understand tasks and reason about solutions, their experience and knowledge are often sparse and difficult to utilize by quantitative approaches. In this paper, we aim to bridge the gap between machine intelligence and human knowledge by introducing a novel framework MLCopilot, which leverages the state-of-the-art large language models to develop ML solutions for novel tasks. We showcase the possibility of extending the capability of LLMs to comprehend structured inputs and perform thorough reasoning for solving novel ML tasks. And we find that, after some dedicated design, the LLM can (i) observe from the existing experiences of ML tasks and (ii) reason effectively to deliver promising results for new tasks. The solution generated can be used directly to achieve high levels of competitiveness.

## introduction past decades have witnessed a great advance and rapid development of machine learning (ml), but ml algorithms are still notoriously hard to configure @xcite . for specific tasks, configuring and conducting corresponding ml solutions is non-trivial, which thus requires extensive human labor. many challenges arise in developing practical ml solutions. first, it is of large human efforts considering the large space of ml solutions, such as feature engineering, model design, optimization details, etc. second, ml algorithms are sensitive to even minor changes of the task context. as a result, even the same algorithm may need to be reconfigured for different application tasks. last but not least, transferring successful experiences across different tasks is also intractable, which demands high-level reasoning abilities of human experts to derive reasonable solutions for novel tasks. the predominant approaches that relieve the human effort of algorithm configuration have been some automation mechanisms such as automl (automated machine learning) @xcite . one major branch of automl formulates the problem as black-box optimization, and resorts to some optimization approaches such as bayesian optimization (bo) @xcite to solve it. though the obtained results are shown to be promising, it is time-consuming to spawn multiple trials, especially for large datasets and complex tasks. moreover, automl does not follow the natural pattern of ml development that humans are accustomed to, which leaves a huge gap for humans to understand and control the whole process. specifically, it is either difficult to explain the behavior of auto-tuning, or intractable to incorporate human prior such as the knowledge of the model architectures into the process, making it less flexible for human developers. furthermore, the ml solutions derived by these optimization-based methods may only fit to the specific domains, and the transferring ability of these results also remains an open problem @xcite . contrarily, we notice two tendencies in how humans approach an ml task. instead of jumping into solving the new task directly, humans often try to comprehend the task at hand and draw from their past experiences on relevant tasks. additionally, humans recall their knowledge, which may have came from a textbook or prior experiences. this process differs significantly from the auto-mated approach mentioned earlier, which leads us to a natural question: can we leverage both machine intelligence and human design patterns to improve our ability to solve ml tasks? the advances of large language models (llm) @xcite @xcite have illustrated tremendous promising performance in mimicking human behaviors on conversationbased tasks. it seems plausible to utilize the power of llm to address ml problems in a more humanlike way. nevertheless, several challenges remain when incorporating llms to achieve this goal. first, we discovered that llms have trouble performing ml tasks based solely on the task description, in which case the performance is no better than random generation. attempting to leverage historical ml experience, we found that the data often reside in heterogeneous formats (e.g., code, configs and logs), which need to be canonicalized into formats that are acceptable to llms. moreover, the amount of information that can be incorporated into in-context learning @xcite is quite limited, and thus some retrieval strategy is desired to make the best out of it. finally, deriving a ml solution based on historical experience is in its essence a mathematical thinking and logical reasoning problem @xcite , which necessitates some mechanisms to reasoning over knowledge. in this paper, we explore and present a novel framework mlcopilot, which leverages llms to suggest solutions for novel real-world ml tasks, based on the existing experiences from historical tasks. we decompose the problem into offline and online stages. in the offline stage, mlcopilot canonicalizes historical data and creates an experience pool. llms are then used to elicit valuable knowledge from historical experience. in the online stage, mlcopilot retrieves experiences from the most relevant tasks from the experience pool, given the description of the target task. it then interacts with llms to obtain multiple suggested ml solutions in one round. we demonstrate that, with a well-designed framework, llms can not only elicit meaningful knowledge from historical experiences but also provide reasonable and competitive ml solutions for novel tasks. our work presents a three-fold contribution, which can be summarized as follows. (i) to the best of our knowledge, we are the first to utilize llms as a tool to generate solutions for new ml tasks. (ii) a novel retrieve-and-prompt framework has been proposed to solve ml tasks almost instantaneously, without any time-consuming searching or optimization. (iii) we leverage the text understanding and generation capabilities of llms to produce interpretable foot_0 results for ml tasks. this approach has shown comparable or even better performance on a variety of real-world ml benchmarks. ## preliminaries the goal of mlcopilot is to assist humans in solving complex ml problems. generally speaking, given a task which is a real-world problem for ml models to tackle, the goal of ml development is to conduct a concrete solution. the solution can be either a pipeline, configuration, or code snippet, based upon which a concrete ml model could be learned to handle the target task. the solution is also a particular sample within a complicated solution space that involves various design choices. these choices are mutually correlated and the outcome of different alternatives often influences the others and eventually affects the final performance of the overall ml solution. to create reasonable ml solutions for new tasks, we can draw on experiences from previous relevant tasks. mlcopilot is designed to use historical experiences for knowledge elicitation and effectively conduct effective solutions for the given novel ml task (details in § 4). to improve comprehension and clarity, we summarize the terminologies with descriptions and examples in ## mlcopilot in this section, we present mlcopilot, with the formulation of the main problem and the overall architecture of our method. then, we will describe some key components of mlcopilot in detail, including target task description, retrieval, canonicalization, and knowledge elicitation. ## experiment we evaluate mlcopilot on a series of benchmarks, aiming to answer the following research questions: (i) can mlcopilot outperform traditional approaches or simple interactions with llms? (ii) how important are individual techniques in ml-copilot, e.g., knowledge and experience? (iii) is the elicited knowledge informative and reasonable? ## conclusion in conclusion, this paper proposes mlcopilot, a framework that unleashes the power of llms to solve practical ml tasks. mlcopilot showcases the versitility of llms, that it can handle not only text-related tasks, but also tasks involving heterogeneous inputs and intricate reasoning. we believe this represents a significant advancement in expanding the scope of llm applications to a broader spectrum of complex problems.