Automatic Discovery of Heterogeneous Machine Learning Pipelines: An Application to Natural Language Processing (2020)

Abstract: This paper presents AutoGOAL, a system for automatic machine learning (AutoML) that uses heterogeneous techniques. In contrast with existing AutoML approaches, our contribution can automatically build machine learning pipelines that combine techniques and algorithms from different frameworks, including shallow classifiers, natural language processing tools, and neural networks. We define the heterogeneous AutoML optimization problem as the search for the best sequence of algorithms that transforms specific input data into the desired output. This provides a novel theoretical and practical approach to AutoML. Our proposal is experimentally evaluated in diverse machine learning problems and compared with alternative approaches, showing that it is competitive with other AutoML alternatives in standard benchmarks. Furthermore, it can be applied to novel scenarios, such as several NLP tasks, where existing alternatives cannot be directly deployed. The system is freely available and includes in-built compatibility with a large number of popular machine learning frameworks, which makes our approach useful for solving practical problems with relative ease and effort.

## introduction the continued development of new machine learning algorithms and techniques, and the widely available tools and datasets have brought new opportunities and challenges for researchers and practitioners in both academia and industry. selecting the best possible strategy to solve a machine learning problem is increasingly difficult partly because it requires long experimentation times and deep technical knowledge. in this scenario, automatic machine learning (automl) has risen to prominence as it provides tools based on specific technologies to efficiently search large spaces of machine learning pipelines, such as auto-weka @xcite , auto-sklearn @xcite and auto-keras @xcite . however, practical problems often require combining and comparing heterogeneous algorithms implemented with different underlying technologies. natural language processing is one scenario where the space of possible techniques to apply varies widely between different tasks, from preprocessing to representation and actual classification. performing automl in an heterogeneous scenario like this is complex because the necessary solution could comprise non-compatible tools and libraries. this would require all algorithms to agree on a common protocol that enables the output of an algorithm to be shared as inputs to any other. ## problem statement in this section, we formally define the problem of heterogeneous automl that this paper addresses. our definition includes not only supervised learning problems, but more general scenarios including paradigms such as unsupervised learning, information retrieval, etc. we consider a as the space of all "atomic" machine learning algorithms, i.e., specific techniques such as logistic regression, k-means, or tf-idf representation, which can intervene in any machine learning process. an algorithm @xmath0: t @xmath1. t i are all the possible types of data that machine learning algorithms deal with, e.g., natural language text, embedding vectors, categories, word stems, etc. for example, tokenization algorithms can be seen as any a : t @xmath2, where these names have a semantic interpretation that corresponds to their usual usage in machine learning. we define s as the space of all possible "pipelines", i.e., sequences of algorithms from a. a pipeline @xmath3, . . . , a @xmath4, or formally as the composition of the corresponding algorithms, i.e., p( in and output t n out . not all possible algorithms can be combined sequentially, but only those a i y a j such that their corresponding output and input types are compatible, e.g., t i @xmath5. in general we can define a partial ordering function t @xmath6. let s be the space of all "valid" pipelines in the previous sense. to select the best pipeline we require a metric ϕ(p) : @xmath7. finally, we define the problem of heterogeneous automl as the optimization problem of finding the best pipeline (given an arbitrary ϕ) that transforms a specific input @xmath8. formally: this definition applies directly to supervised and unsupervised learning scenarios. in supervised learning, the input will consist of some data in feature space (e.g., documents, images, numerical features) plus a prediction target (e.g., a vector of classes or values to perform regression). in unsupervised learning only the features data is assumed as input. in both cases, the output consists of some prediction target, e.g., classes for supervised learning and cluster labels for unsupervised learning. ## computational implementation the heterogeneous automl problem can be computationally represented as a graph g a where all known algorithms a i are nodes, and edges exist between all pairs a i , a j such that t i @xmath9, i.e., their corresponding output and input types are compatible. to solve the optimization problem, we insert two nodes in this graph, input and output, connected correspondingly to all algorithms compatible with the specific input t * in and all algorithms that output the desired t * out . a pipeline that solves our problem is any path in g a between input and output. a suitable computational implementation of this process requires solving the following problems: this research proposes a theoretical solution to the aforementioned problems, and a concrete computational implementation in the python programming language, i.e., the autogoal library. figure 1 illustrates one key design idea of autogoal, i.e., the automatic discovery of machine learning pipelines in a graph of heterogeneous algorithms. ## results and discussion in this section, we evaluate the adaptability and performance of autogoal in dealing with heterogeneous machine learning problems. for this purpose, we select three distinct problems with increasing levels of complexity, respectively: classic supervised benchmarks, section 4.1; text classification, section 4.2; and, named entity recognition, section 4.3. regarding the first group in section 4.1, our system is evaluated for seven classic supervised learning benchmarks in the automl literature. the purpose of this experimentation is to show that autogoal is competitive with other automl systems. nevertheless, our purpose is not to supersede other automl tools, but rather to provide solutions in novel problem settings where existing systems cannot be directly deployed. for this reason, sections 4.2 and 4.3 deal with more challenging problems in natural language processing tasks. in these more complex scenarios, autogoal will be compared with state-of-the-art solutions that are carefully crafted by domain experts, since existing automl tools cannot be directly applied in these scenarios. source code and data to reproduce these experimental results is available online foot_3 . in terms of code, the three experiments follow the same logic, as exemplified in figure 3 . figure 3 : example source code for running autogoal on a specific dataset. for clarity, only key aspects are included. extra code such as irrelevant imports, logging, time and memory constraints, etc., are not shown. the input and output types used in this example are specific for supervised nlp problems and must be adapted to the problem definition. ## discussion regarding the experimental results, it is important to note that our system is deployed using the same code in all the experiments (see figure 3 ), varying only the definition of the input and output types. for example, setting up haha and meddocan challenges took approximately one and four hours respectively, fundamentally devoted to preparing the corpora in a suitable format. the total running time in our approach is considerably higher than running a single hand-crafted pipeline, since several different pipelines are trained and tested (e.g., 20 total hours on the haha dataset). however, the time necessary for setting up on a novel problem is considerably smaller than coding a hand-crafted solution from scratch, since several pre-made components are provided. hence, there is a trade-off between computational cost and human cost, which allows researchers to focus more on the conceptualization and modeling, i.e., deciding what is the space of possible pipelines to evaluate, and less on actually implementing those pipelines. autogoal enables researchers and practitioners to quickly develop strong baselines in diverse machine learning problems. in some scenarios, the solution provided by ours and other automl systems could already be good enough. however, automl systems should not attempt only to replace human experts, but rather to serve as complementary tools that allow researchers to quickly obtain better baselines and insights on the most promising strategies. we envision a future in which humans and computers work together to solve the most challenging and complex problems in artificial intelligence, each bringing their core competences. much like compilers brought a significant improvement in the efficiency of software development, we think automl opens the door to revolutionizing the way machine learning research and practice is performed. ## conclusions this paper presents autogoal, a system for automatic machine learning (automl) that seamlessly combines heterogeneous technologies and can be applied to a wide variety of machine learning scenarios. our proposal is competitive with other automl frameworks on standard benchmarks, and it can be applied to novel scenarios for which several existing automl tools are not directly applicable. an experimental evaluation in two different nlp problems shows that this system can obtain competitive results with human-designed pipelines, without any domain or problem-specific considerations. a computational prototype of this system with pre-packaged implementations of over 100 algorithms from popular machine learning frameworks is made available for the research community.