Original: B EYOND D EEP L EARNING : A N E VOLUTIONARY F EATURE E NGINEERING A PPROACH TO T ABULAR D ATA C LASSIFICATION
Revised: B EYOND D EEP L EARNING : A N E VOLUTIONARY F EATURE E NGINEERING A PPROACH TO T ABULAR D ATA C LASSIFICATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: In recent years, deep learning has achieved impressive performance in the computer vision and natural language processing domains.
Revised: In recent years, deep learning has achieved impressive performance in the computer vision and natural language processing domains.

Original: In the tabular data classification scenario, with the emergence of the transformer architecture, a number of algorithms have been reported to yield better results than conventional tree-based models.
Revised: In the tabular data classification scenario, with the emergence of the transformer architecture, a number of algorithms have been reported to yield better results than conventional tree-based models.

Original: Most of these methods attribute the success of deep learning methods to the expressive feature construction capability of neural networks.
Revised: Most of these methods attribute the success of deep learning methods to the expressive feature construction capability of neural networks.

Original: Nonetheless, in real practice, manually designed high-order features with traditional machine learning methods are still widely used because neural-network-based features can be easy to over-fitting.
Revised: Nonetheless, in real practice, manually designed high-order features with traditional machine learning methods are still widely used because neural-network-based features can be easy to over-fitting.

Original: In this paper, we propose an evolution-based feature engineering algorithm to imitate the manual feature construction process through trial and improvement.
Revised: In this paper, we propose an evolution-based feature engineering algorithm to imitate the manual feature construction process through trial and improvement.

Original: Importantly, the evolutionary method provides an opportunity to optimize cross-validation loss, where gradient methods fail to do so.
Revised: Importantly, the evolutionary method provides an opportunity to optimize cross-validation loss, where gradient methods fail to do so.

Original: On a large-scale classification benchmark of 119 datasets, the experimental results demonstrate that the proposed method outperforms existing fine-tuned state-of-the-art tree-based and deep-learning-based classification algorithms.
Revised: On a large-scale classification benchmark of 119 datasets, the experimental results demonstrate that the proposed method outperforms existing fine-tuned state-of-the-art tree-based and deep-learning-based classification algorithms.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Tabular data is a typical way of storing information in a computer system, and the tabular data learning methodology has been widely employed in recommendation systems (Wang et al., 2021), advertising (Yang et al., 2021), and the medical industry (Spooner et al., 2020) to automate the decision process.
Revised: Tabular data is a typical way of storing information in a computer system, and the tabular data learning methodology has been widely employed in recommendation systems (Wang et al., 2021), advertising (Yang et al., 2021), and the medical industry (Spooner et al., 2020) to automate the decision process.

Original: Formally, tabular data learning techniques capture the association between some explanatory variables { x 1 , . . .
Revised: Formally, tabular data learning techniques capture the association between some explanatory variables { x 1 , . . .

Original: , x m } and a response variable y for a given dataset { ( { x 11 , . . .
Revised: , x m } and a response variable y for a given dataset { ( { x 11 , . . .

Original: , x 1 m } , y 1 ) , . .
Revised: , x 1 m } , y 1 ) , . .

Original: , ( { x n 1 , . . .
Revised: , ( { x n 1 , . . .

Original: , x nm } , y n ) } , where n is the number of data items.
Revised: , x nm } , y n ) } , where n is the number of data items.

Original: There are various machine learning methods that could address the tabular data learning problem, such as decision trees, linear models, and support vector machines.
Revised: There are various machine learning methods that could address the tabular data learning problem, such as decision trees, linear models, and support vector machines.

Original: However, these learning algorithms have a limited learning capability to fully tackle this problem.
Revised: However, these learning algorithms have a limited learning capability to fully tackle this problem.

Original: For example, the linear model assumes that the relationship between explanatory variables and response variables is linear, whereas the decision tree posits that an axis-parallel decision boundary can divide distinct categories of samples.
Revised: For example, the linear model assumes that the relationship between explanatory variables and response variables is linear, whereas the decision tree posits that an axis-parallel decision boundary can divide distinct categories of samples.

Original: These assumptions are not always correct in the real world, and thus machine learning practitioners always employ feature engineering techniques to construct a set of high-order features {{ ϕ 11 . . .
Revised: These assumptions are not always correct in the real world, and thus machine learning practitioners always employ feature engineering techniques to construct a set of high-order features {{ ϕ 11 . . .

Original: ϕ 1 k } , . .
Revised: ϕ 1 k } , . .

Original: , { ϕ n 1 . . .
Revised: , { ϕ n 1 . . .

Original: ϕ nk }} to improve the learning capability of existing algorithms.
Revised: ϕ nk }} to improve the learning capability of existing algorithms.

Original: In real-world applications, many feature engineering methods can be used by industrial practitioners to construct high-order features before training a machine learning model.
Revised: In real-world applications, many feature engineering methods can be used by industrial practitioners to construct high-order features before training a machine learning model.

Original: The conventional feature engineering strategies include manually designed features, dimensionality reduction methods (Colange et al., 2020), kernel methods (Allen-Zhu & Li, 2019), and so on.
Revised: The conventional feature engineering strategies include manually designed features, dimensionality reduction methods (Colange et al., 2020), kernel methods (Allen-Zhu & Li, 2019), and so on.

Original: However, these methods have drawbacks.
Revised: However, these methods have drawbacks.

Original: Manual feature construction approaches require a large amount of human labor, kernel methods are hard to be combined with tree-based methods, and dimensionality reduction algorithms are insufficient to enhance state-of-the-art learning algorithms, especially for unsupervised ones (Lensen et al., 2022).
Revised: Manual feature construction approaches require a large amount of human labor, kernel methods are hard to be combined with tree-based methods, and dimensionality reduction algorithms are insufficient to enhance state-of-the-art learning algorithms, especially for unsupervised ones (Lensen et al., 2022).

Original: With the success of deep learning, there is a growing interest in using neural networks to build high-order features automatically, particularly in the field of recommendation systems (Song et al., 2019; Lian et al., 2018).
Revised: With the success of deep learning, there is a growing interest in using neural networks to build high-order features automatically, particularly in the field of recommendation systems (Song et al., 2019; Lian et al., 2018).

Original: However, it is debatable whether deep learning techniques can learn highorder features effectively (Wang et al., 2021).
Revised: However, it is debatable whether deep learning techniques can learn highorder features effectively (Wang et al., 2021).

Original: Despite the fact that various neural network-based tabular learning algorithms have shown better predictive accuracy than some tree-based ensemble models (Arık & Pfister, 2021), most of these methods only validate their algorithms on datasets with a large number of training samples and homogeneous features.
Revised: Despite the fact that various neural network-based tabular learning algorithms have shown better predictive accuracy than some tree-based ensemble models (Arık & Pfister, 2021), most of these methods only validate their algorithms on datasets with a large number of training samples and homogeneous features.

Original: Nowadays, it is still controversial whether deep learning methods outperform tree-based methods on datasets with heterogeneous features and a limited amount of data (Gorishniy et al., 2021), and we hope to further explore this problem in this paper.
Revised: Nowadays, it is still controversial whether deep learning methods outperform tree-based methods on datasets with heterogeneous features and a limited amount of data (Gorishniy et al., 2021), and we hope to further explore this problem in this paper.

Original: In real-world cases, machine learning engineers typically start from some randomly designed features, and then evaluate these features to get cross-validation loss and feature importance value.
Revised: In real-world cases, machine learning engineers typically start from some randomly designed features, and then evaluate these features to get cross-validation loss and feature importance value.

Original: The feedback information can be used by engineers to design better features.
Revised: The feedback information can be used by engineers to design better features.

Original: To imitate this workflow, we design a feature engineering method named evolutionary feature engineering automation tool (EvoFeat) 1 , which constructs nonlinear features for enhancing an ensemble of decision tree and logistic regression models, and thus achieving better results than conventional ensemble learning methods.
Revised: To imitate this workflow, we design a feature engineering method named evolutionary feature engineering automation tool (EvoFeat) 1 , which constructs nonlinear features for enhancing an ensemble of decision tree and logistic regression models, and thus achieving better results than conventional ensemble learning methods.

Original: The three objectives of our paper are summarized as follows:
Revised: The three objectives of our paper are summarized as follows:

Original: • Considering cross-validation losses and feature importance values are widely used by machine learning engineers in the feature engineering process, we propose an evolutionary feature construction method that uses both information to guide the search process.
Revised: • Considering cross-validation losses and feature importance values are widely used by machine learning engineers in the feature engineering process, we propose an evolutionary feature construction method that uses both information to guide the search process.

Original: • Given that the performance of the constructed features is highly bonded to the base learner, we propose a mechanism that allows the evolutionary algorithm to automatically determine whether to use random decision trees or logistic regression models as base learners.
Revised: • Given that the performance of the constructed features is highly bonded to the base learner, we propose a mechanism that allows the evolutionary algorithm to automatically determine whether to use random decision trees or logistic regression models as base learners.

Original: • We conduct experiments on 119 datasets and compare our method to six traditional machine learning algorithms, four deep learning algorithms and two automated feature construction methods.
Revised: • We conduct experiments on 119 datasets and compare our method to six traditional machine learning algorithms, four deep learning algorithms and two automated feature construction methods.

Original: Experiment results show that deep learning methods are not silver bullets for classification tasks.
Revised: Experiment results show that deep learning methods are not silver bullets for classification tasks.

Original: The remainder of the paper is structured as follows.
Revised: The remainder of the paper is structured as follows.

Original: Section 2 introduces related work of feature construction.
Revised: Section 2 introduces related work of feature construction.

Original: Section 3 discusses the motivation behind our work.
Revised: Section 3 discusses the motivation behind our work.

Original: Section 4 then explains the details of the proposed method, followed by the experimental results provided in Section 5.
Revised: Section 4 then explains the details of the proposed method, followed by the experimental results provided in Section 5.

Original: Finally, Sectionsummarizes our paper and recommends some intriguing future avenues.
Revised: Finally, Sectionsummarizes our paper and recommends some intriguing future avenues.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Evolutionary feature construction methods have received a lot of attention in the evolutionary learning domain.
Revised: Evolutionary feature construction methods have received a lot of attention in the evolutionary learning domain.

Original: Nonetheless, the majority of these methods focus on simple learners, such as a single decision tree (Tran et al., 2019) or a single support vector machine (Nag & Pal, 2019).
Revised: Nonetheless, the majority of these methods focus on simple learners, such as a single decision tree (Tran et al., 2019) or a single support vector machine (Nag & Pal, 2019).

Original: Research on how to apply evolutionary feature construction methods to improve state-of-the-art classification algorithms is still lacking.
Revised: Research on how to apply evolutionary feature construction methods to improve state-of-the-art classification algorithms is still lacking.

Original: In recent years, ensemble-based evolutionary feature construction methods have produced encouraging results on both the symbolic regression (Zhang et al., 2021) and image classification (Bi et al., 2020; 2021a) tasks.
Revised: In recent years, ensemble-based evolutionary feature construction methods have produced encouraging results on both the symbolic regression (Zhang et al., 2021) and image classification (Bi et al., 2020; 2021a) tasks.

Original: However, feature construction on these tasks are different than tabular data classification tasks because domain knowledge for designing primitive functions are available for these tasks, such as a function named analytical quotient Ni et al.
Revised: However, feature construction on these tasks are different than tabular data classification tasks because domain knowledge for designing primitive functions are available for these tasks, such as a function named analytical quotient Ni et al.

Original: (2012) for symbolic regression and the histogram of oriented gradients (HOG) Dalal & Triggs (2005) for image classification.
Revised: (2012) for symbolic regression and the histogram of oriented gradients (HOG) Dalal & Triggs (2005) for image classification.

Original: In addition to the evolution-based method, beam search could also be used to construct high-order features (Katz et al., 2016;
Revised: In addition to the evolution-based method, beam search could also be used to construct high-order features (Katz et al., 2016;

Original: Luo et al., 2019; Shi et al., 2020).
Revised: Luo et al., 2019; Shi et al., 2020).

Original: The main idea behind these methods is to start with some low-order features and iteratively generate higher-order features based on important low-order features.
Revised: The main idea behind these methods is to start with some low-order features and iteratively generate higher-order features based on important low-order features.

Original: A meta learner (Katz et al., 2016), logistic regression accuracy (Luo et al., 2019), or XGBoost feature importance (Shi et al., 2020) can be used to determine the feature importance of each generated feature.
Revised: A meta learner (Katz et al., 2016), logistic regression accuracy (Luo et al., 2019), or XGBoost feature importance (Shi et al., 2020) can be used to determine the feature importance of each generated feature.

Original: However, these methods lack an effective mechanism to prevent over-fitting and thus may have restricted feature construction capabilities.
Revised: However, these methods lack an effective mechanism to prevent over-fitting and thus may have restricted feature construction capabilities.

Original: In the recommendation system domain, deep learning methods have been shown to be an effective way to construct high-order features for tabular data classification tasks (Guo et al., 2017).
Revised: In the recommendation system domain, deep learning methods have been shown to be an effective way to construct high-order features for tabular data classification tasks (Guo et al., 2017).

Original: Even though the feed-forward neural network (FNN) is considered a universal function approximator (Kidger & Lyons, 2020), its ability to model high-order feature interactions has been questioned (Mudgal et al., 2018).
Revised: Even though the feed-forward neural network (FNN) is considered a universal function approximator (Kidger & Lyons, 2020), its ability to model high-order feature interactions has been questioned (Mudgal et al., 2018).

Original: In order to efficiently construct high-order features, a cross network is designed in
Revised: In order to efficiently construct high-order features, a cross network is designed in

Original: Deep & Cross Network (DCN) to explicitly model the feature interaction (Mudgal et al., 2018).
Revised: Deep & Cross Network (DCN) to explicitly model the feature interaction (Mudgal et al., 2018).

Original: Following that, xDeepFM employs a field-wise feature cross method to effectively (Lian et al., 2018) model the feature interaction in a bit-wise manner.
Revised: Following that, xDeepFM employs a field-wise feature cross method to effectively (Lian et al., 2018) model the feature interaction in a bit-wise manner.

Original: AutoInt (Song et al., 2019) goes a step further, learning feature interaction using the attention mechanism rather than a simple outer product.
Revised: AutoInt (Song et al., 2019) goes a step further, learning feature interaction using the attention mechanism rather than a simple outer product.

Original: Due to useless feature interaction can introduce noise, AutoFIS (Liu et al., 2020) employs a generalized regularized dual averaging (GRDA)
Revised: Due to useless feature interaction can introduce noise, AutoFIS (Liu et al., 2020) employs a generalized regularized dual averaging (GRDA)

Original: (Chao & Cheng, 2019) method to remove unimportant feature interaction.
Revised: (Chao & Cheng, 2019) method to remove unimportant feature interaction.

Original: With the increasing popularity of the neural network architecture search (NAS) technique, an algorithm known as AutoPI (Meng et al., 2021) employs the NAS technique to automatically discover better feature interaction components.
Revised: With the increasing popularity of the neural network architecture search (NAS) technique, an algorithm known as AutoPI (Meng et al., 2021) employs the NAS technique to automatically discover better feature interaction components.

Original: This method is inspired by differentiable architecture search (DARTS)
Revised: This method is inspired by differentiable architecture search (DARTS)

Original: (Liu et al., 2018), and it uses the temperature anneal (Liu et al., 2018) technique to find the best subnet from a supernet using the gradient descent method.
Revised: (Liu et al., 2018), and it uses the temperature anneal (Liu et al., 2018) technique to find the best subnet from a supernet using the gradient descent method.

Original: However, the effectiveness of these feature cross modules over FNN is still debatable (Wang et al., 2021), let alone the lack of a comprehensive study of whether these approaches can outperform XGBoost on classical classification problems (Gorishniy et al., 2021).
Revised: However, the effectiveness of these feature cross modules over FNN is still debatable (Wang et al., 2021), let alone the lack of a comprehensive study of whether these approaches can outperform XGBoost on classical classification problems (Gorishniy et al., 2021).

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: First of all, let us analyze how a machine learning engineer constructs features for a given dataset { ( { x 11 , . . .
Revised: First of all, let us analyze how a machine learning engineer constructs features for a given dataset { ( { x 11 , . . .

Original: , x 1 m } , y 1 ) , . .
Revised: , x 1 m } , y 1 ) , . .

Original: , ( { x n 1 , . . .
Revised: , ( { x n 1 , . . .

Original: , x nm } , y n ) } in the absence of domain expertise.
Revised: , x nm } , y n ) } in the absence of domain expertise.

Original: Figure 1 depicts an example of the feature engineering workflow.
Revised: Figure 1 depicts an example of the feature engineering workflow.

Original: It is divided into three phases:
Revised: It is divided into three phases:

Original: • Feature Initialization : First, a machine learning engineer without any domain knowledge can randomly construct some features Φ = { ϕ 1 . . .
Revised: • Feature Initialization : First, a machine learning engineer without any domain knowledge can randomly construct some features Φ = { ϕ 1 . . .

Original: ϕ k } based on original features { x 1 . . .
Revised: ϕ k } based on original features { x 1 . . .

Original: x m } .
Revised: x m } .

Original: • Feature Evaluation : Second, using the cross-validation approach, the generated features { ϕ 1 . . .
Revised: • Feature Evaluation : Second, using the cross-validation approach, the generated features { ϕ 1 . . .

Original: ϕ k } are evaluated on the training set and the target learning algorithm.
Revised: ϕ k } are evaluated on the training set and the target learning algorithm.

Original: The trained model produces cross-validation losses and feature importance values to help an engineer determine useful features.
Revised: The trained model produces cross-validation losses and feature importance values to help an engineer determine useful features.

Original: • Feature Improvement : Finally, the engineer removes some useless features and replaces them with new features derived from important features.
Revised: • Feature Improvement : Finally, the engineer removes some useless features and replaces them with new features derived from important features.

Original: During the feature construction process, the second and third phases are repeated until the engineer is satisfied with the result.
Revised: During the feature construction process, the second and third phases are repeated until the engineer is satisfied with the result.

Original: Formally, if F is a search space of machine learning models and L ( f ( x ) , y ) is the loss function of a model f ∈ F , the feature construction process is to find a set of features Φ that can minimize the objective function min f ∗ ∈ F L ( f ∗ (Φ( x )) , y ) .
Revised: Formally, if F is a search space of machine learning models and L ( f ( x ) , y ) is the loss function of a model f ∈ F , the feature construction process is to find a set of features Φ that can minimize the objective function min f ∗ ∈ F L ( f ∗ (Φ( x )) , y ) .

Original: In real-world applications, some machine learning engineers prefer to only use feature importance values to detect useful features (Shi et al., 2020).
Revised: In real-world applications, some machine learning engineers prefer to only use feature importance values to detect useful features (Shi et al., 2020).

Original: However, this is risky because the feature importance values do not reflect the generalization performance of the constructed features.
Revised: However, this is risky because the feature importance values do not reflect the generalization performance of the constructed features.

Original: In other words, a feature may play an important role in an over-fitted model, and we must have a strategy to avoid sticking at this local optima.
Revised: In other words, a feature may play an important role in an over-fitted model, and we must have a strategy to avoid sticking at this local optima.

Original: Constructing numerous sets of features at the same time is a plausible idea.
Revised: Constructing numerous sets of features at the same time is a plausible idea.

Original: The engineer can then detect the over-fitting issue based on the cross-validation performance of multiple sets of features and make a more informed decision.
Revised: The engineer can then detect the over-fitting issue based on the cross-validation performance of multiple sets of features and make a more informed decision.

Original: The feature engineering paradigm depicted in Figure 1 is general.
Revised: The feature engineering paradigm depicted in Figure 1 is general.

Original: Even for a machine learning algorithm that does not provide an internal feature importance report, it can still be employed in this feature engineering method based on some black-box feature importance calculation approaches, such as SHAP (Lundberg & Lee, 2017).
Revised: Even for a machine learning algorithm that does not provide an internal feature importance report, it can still be employed in this feature engineering method based on some black-box feature importance calculation approaches, such as SHAP (Lundberg & Lee, 2017).

Original: Under this paradigm, the most critical problem is to determine how to leverage information based on feature importance values and cross-validation loss to generate better features.
Revised: Under this paradigm, the most critical problem is to determine how to leverage information based on feature importance values and cross-validation loss to generate better features.

Original: Here, we employ tree-based genetic programming (TGP) to do so.
Revised: Here, we employ tree-based genetic programming (TGP) to do so.

Original: TGP is an evolutionary algorithm that focuses on evolving symbolic trees.
Revised: TGP is an evolutionary algorithm that focuses on evolving symbolic trees.

Original: Its gradient free nature and variable length representation that make it suitable to generate expressive features based on non-differentiable objective values.
Revised: Its gradient free nature and variable length representation that make it suitable to generate expressive features based on non-differentiable objective values.

Original:  T HE P ROPOSED A LGORITHM 4. F EATURE R EPRESENTATION
Revised:  T HE P ROPOSED A LGORITHM 4. F EATURE R EPRESENTATION

Original: First of all, unlike the traditional evolutionary algorithm, which utilizes an array to denote a solution, we employ a set of symbolic trees to represent a set of constructed features.
Revised: First of all, unlike the traditional evolutionary algorithm, which utilizes an array to denote a solution, we employ a set of symbolic trees to represent a set of constructed features.

Original: Each symbolic tree consists of two parts: non-leaf nodes and leaf nodes.
Revised: Each symbolic tree consists of two parts: non-leaf nodes and leaf nodes.

Original: Non-leaf nodes are also known as functions, which can transform or combine basic features, such as { + , −} or { log, sin } .
Revised: Non-leaf nodes are also known as functions, which can transform or combine basic features, such as { + , −} or { log, sin } .

Original: The leaf nodes are also known as terminals, representing original features, such as { x 1 , x 2 } .
Revised: The leaf nodes are also known as terminals, representing original features, such as { x 1 , x 2 } .

Original: Each symbolic tree can be used to construct a feature ϕ , and all symbolic trees in an individual Φ together could form a new feature space S .
Revised: Each symbolic tree can be used to construct a feature ϕ , and all symbolic trees in an individual Φ together could form a new feature space S .

Original: In addition to a set of symbolic trees, each individual is also accompanied by a base learner, which is a decision tree or a linear regression model in this research.
Revised: In addition to a set of symbolic trees, each individual is also accompanied by a base learner, which is a decision tree or a linear regression model in this research.

Original: This base learner is used to evaluate the predictive performance of each feature space and also used to give a prediction of unseen data.
Revised: This base learner is used to evaluate the predictive performance of each feature space and also used to give a prediction of unseen data.

Original: 4. A LGORITHM F RAMEWORK
Revised: 4. A LGORITHM F RAMEWORK

Original: The proposed evolutionary algorithm consists of five parts: initialization, evaluation, selection, generation, and archive update.
Revised: The proposed evolutionary algorithm consists of five parts: initialization, evaluation, selection, generation, and archive update.

Original: The pseudocode for our algorithm is shown in Algorithm 1.
Revised: The pseudocode for our algorithm is shown in Algorithm 1.

Original: Initialization:
Revised: Initialization:

Original: In the initialization stage, the algorithm randomly initializes N individuals { Φ 1 . . .
Revised: In the initialization stage, the algorithm randomly initializes N individuals { Φ 1 . . .

Original: Φ N } to form the initial population.
Revised: Φ N } to form the initial population.

Original: Each individual consists of k symbolic trees { ϕ 1 . . .
Revised: Each individual consists of k symbolic trees { ϕ 1 . . .

Original: ϕ k } , which could be used to construct a k -dimensional feature space.
Revised: ϕ k } , which could be used to construct a k -dimensional feature space.

Original: Evaluation: At the evaluation stage, we construct training data based on each individual Φ , and evaluate them on their corresponding base learners f .
Revised: Evaluation: At the evaluation stage, we construct training data based on each individual Φ , and evaluate them on their corresponding base learners f .

Original: The fitness value of each individual can be calculated using the five-fold cross-validation loss, and the fitness value of each feature can be obtained using the average feature importance values across five folds.
Revised: The fitness value of each individual can be calculated using the five-fold cross-validation loss, and the fitness value of each feature can be obtained using the average feature importance values across five folds.

Original: Selection: The selection operator selects promising individuals { Φ A , Φ B } from the population P for crossover and mutation.
Revised: Selection: The selection operator selects promising individuals { Φ A , Φ B } from the population P for crossover and mutation.

Original: In this paper, we use the lexicase selection operator (La Cava et al., 2019) to select parent individuals based on the cross-validation losses.
Revised: In this paper, we use the lexicase selection operator (La Cava et al., 2019) to select parent individuals based on the cross-validation losses.

Original: Generation:
Revised: Generation:

Original: The generation operator generates new individuals based on selected parent features.
Revised: The generation operator generates new individuals based on selected parent features.

Original: We use self-competitive based crossover (Nguyen et al., 2021) and guided mutation operators to increase search efficiency by leveraging feature importance values.
Revised: We use self-competitive based crossover (Nguyen et al., 2021) and guided mutation operators to increase search efficiency by leveraging feature importance values.

Original: Archive Update: At the end of each iteration, we dynamically update an archive A to store historically top-performing models, depending on the performance of archived models A and newly generated models { f 1 . . .
Revised: Archive Update: At the end of each iteration, we dynamically update an archive A to store historically top-performing models, depending on the performance of archived models A and newly generated models { f 1 . . .

Original: f N } .
Revised: f N } .

Original: The archive update method employed in this research is a greedy ensemble selection method named reduce-error pruning (Caruana et al., 2004).
Revised: The archive update method employed in this research is a greedy ensemble selection method named reduce-error pruning (Caruana et al., 2004).

Original: The selection, generation, evaluation and archive update operators are executed repeatedly until the maximum number of generations is reached.
Revised: The selection, generation, evaluation and archive update operators are executed repeatedly until the maximum number of generations is reached.

Original: Finally, all base learners in archive A could form an ensemble model based on the weights assigned by the ensemble selection method.
Revised: Finally, all base learners in archive A could form an ensemble model based on the weights assigned by the ensemble selection method.

Original: 4. F EATURE I NITIALIZATION
Revised: 4. F EATURE I NITIALIZATION

Original: The proposed evolutionary feature construction method is based on TGP, and thus we use a rampedhalf-and-half (Burke et al., 2003) symbolic tree initialization strategy from TGP to initialize each tree ϕ .
Revised: The proposed evolutionary feature construction method is based on TGP, and thus we use a rampedhalf-and-half (Burke et al., 2003) symbolic tree initialization strategy from TGP to initialize each tree ϕ .

Original: In particular, half of the symbolic trees are built as full symbolic trees, while the other half are initialized with a random height.
Revised: In particular, half of the symbolic trees are built as full symbolic trees, while the other half are initialized with a random height.

Original: In addition, each individual is randomly assigned a random decision tree (RDT) or logistic regression model (LR) with equal probability.
Revised: In addition, each individual is randomly assigned a random decision tree (RDT) or logistic regression model (LR) with equal probability.

Original: This allows the evolutionary algorithm to determine the optimal base learner to form an ensemble model.
Revised: This allows the evolutionary algorithm to determine the optimal base learner to form an ensemble model.

Original: 4. F EATURE S ELECTION
Revised: 4. F EATURE S ELECTION

Original: The main purpose of the feature selection operator is to select two individuals from the population based on the cross-validation loss.
Revised: The main purpose of the feature selection operator is to select two individuals from the population based on the cross-validation loss.

Original: In the existing EA literature, numerous selection operators are available, such as age-fitness pareto (AFP) (Schmidt & Lipson, 2011), double tournament selection (Luke & Panait, 2002), lexicase selection (La Cava et al., 2019), and so on.
Revised: In the existing EA literature, numerous selection operators are available, such as age-fitness pareto (AFP) (Schmidt & Lipson, 2011), double tournament selection (Luke & Panait, 2002), lexicase selection (La Cava et al., 2019), and so on.

Original: In this research, we employ lexicase selection as the selection operator because it is helpful to conserve population diversity, which is advantageous in the ensemble learning scenario (Zhang et al., 2021). 
Revised: In this research, we employ lexicase selection as the selection operator because it is helpful to conserve population diversity, which is advantageous in the ensemble learning scenario (Zhang et al., 2021). Lexicase Selection:

Revised: For a given vector of validation losses, the lexicase selection operator constructs a set of filters based on each loss value across all individuals.

Revised: Each filter randomly selects one dimension

Original: Input: Population Size N , Number of Features k , Number of Generations max _ gen
Revised: Input: Population Size N , Number of Features k , Number of Generations max _ gen

Original: Output: Archive A 1: Randomly initialize N set of features P = { Φ 1 . . .
Revised: Output: Archive A 1: Randomly initialize N set of features P = { Φ 1 . . .

Original: Φ N } ▷ Feature Initialization 2: for i = 1 , . . .
Revised: Φ N } ▷ Feature Initialization 2: for i = 1 , . . .

Original: N do ▷ Feature Evaluation 3: f i ← cross-validation( f i , Φ i ) 4: A ← ∅ 5: A ← archive update ( { f 1 . . .
Revised: N do ▷ Feature Evaluation 3: f i ← cross-validation( f i , Φ i ) 4: A ← ∅ 5: A ← archive update ( { f 1 . . .

Original: f N } , A )
Revised: f N } , A )

Original: ▷ Archive Update 6: gen ←7: while gen < max _
Revised: ▷ Archive Update 6: gen ←7: while gen < max _

Original: gen do ▷ Main loop 8:
Revised: gen do ▷ Main loop 8:

Original: P ′ = ∅ 9: for i = 1 , · · · , N do 10:
Revised: P ′ = ∅ 9: for i = 1 , · · · , N do 10:

Original: P LR , P DT ← population division( P ) 11: if rand () < 0 .
Revised: P LR , P DT ← population division( P ) 11: if rand () < 0 .

Original: 5 then ▷ Base Learner Selection 12: Φ A ← lexicase selection( P LR ) 13: Φ B ← lexicase selection( P DT ) 14: else 15:
Revised: 25 then ▷ Base Learner Selection 12: Φ A ← lexicase selection( P LR ) 13: Φ B ← lexicase selection( P DT ) 14: else 15:

Original: P ′ ← random selection( P LR , P DT ) 16: Φ A , Φ B ← lexicase selection( P ′ )
Revised: P ′ ← random selection( P LR , P DT ) 16: Φ A , Φ B ← lexicase selection( P ′ )

Original: ▷ Individual Selection 17: for j = 1 , · · · , ⌊ k 2 ⌋ do ▷ Crossover (Feature Generation) 18: if rand () <
Revised: ▷ Individual Selection 17: for j = 1 , · · · , ⌊ k 2 ⌋ do ▷ Crossover (Feature Generation) 18: if rand () <

Original: p c then 19: ϕ a , ϕ b ← self-competitive selection( Φ A )
Revised: p c then 19: ϕ a , ϕ b ← self-competitive selection( Φ A )

Original: ▷ Feature Selection 20: ϕ c , ϕ d ← self-competitive selection( Φ B ) 21: ϕ a ← crossover( ϕ a , ϕ d ) 22: ϕ c ← crossover( ϕ c , ϕ b ) 23: for j = 1 , · · · , k do ▷ Mutation (Feature Generation) 24: if rand () <
Revised: ▷ Feature Selection 20: ϕ c , ϕ d ← self-competitive selection( Φ B ) 21: ϕ a ← crossover( ϕ a , ϕ d ) 22: ϕ c ← crossover( ϕ c , ϕ b ) 23: for j = 1 , · · · , k do ▷ Mutation (Feature Generation) 24: if rand () <

Original: p m then 25: ϕ a ← random selection( Φ A ) 26: ϕ
Revised: p m then 25: ϕ a ← random selection( Φ A ) 26: ϕ

Original: i ← mutation( ϕ i ) 27:
Revised: i ← mutation( ϕ i ) 27:

Original: P ′ =
Revised: P ′ =

Original: P ′ ∪ { Φ A , Φ B } 28:
Revised: P ′ ∪ { Φ A , Φ B } 28:

Original: P = P ′ 29: for k = 1 , . . .
Revised: P = P ′ 29: for k = 1 , . . .

Original: N do ▷ Feature Evaluation 30:
Revised: N do ▷ Feature Evaluation 30:

Original: f k ← cross-validation( f i , Φ k ) 31:
Revised: f k ← cross-validation( f i , Φ k ) 31:

Original: A ← archive update ( { f 1 . . .
Revised: A ← archive update ( { f 1 . . .

Original: f N } , A )
Revised: f N } , A )

Original: ▷ Archive Update 32: gen ← gen +33: return A Lexicase Selection:
Revised: ▷ Archive Update 32: gen ← gen +33: return A j ∈ [1 , n ] to do filtering, and the threshold τ j is dynamically set as τ j = min i mse

Original: For a given vector of validation losses, the lexicase selection operator constructs a set of filters based on each loss value across all individuals.

Original: Each filter randomly selects one dimension j ∈ [1 , n ] to do filtering, and the threshold τ j is dynamically set as τ j = min i mse ij + mad j , where mad j = median i ( | mse ij − median k ( mse kj ) | ) represents the median absolute deviation.
Revised: ij + mad j , where mad j = median i ( | mse ij − median k ( mse k j ) | ) represents the median absolute deviation.

Original: Based on this selection operator, we can select a suitable number of parent individuals for crossover and selection.
Revised: Based on this selection operator, we can select a suitable number of parent individuals for crossover and selection.

Original: Base Learner Selection: We employ either a decision tree or logistic regression model as the base learner to evaluate each group of features.
Revised: Base Learner Selection: We employ either a decision tree or logistic regression model as the base learner to evaluate each group of features.

Original: However, decision trees and logistic regression models may not produce comparable predictive results.
Revised: However, decision trees and logistic regression models may not produce comparable predictive results.

Original: For example, because a single decision tree cannot produce a categorical distribution, misclassifying only one example will result in an infinite crossentropy loss.
Revised: For example, because a single decision tree cannot produce a categorical distribution, misclassifying only one example will result in an infinite crossentropy loss.

Original: Thus, we borrow an idea from the multitask evolutionary optimization domain (Wei et al., 2021) to select two parent individuals from a heterogeneous population.
Revised: Thus, we borrow an idea from the multitask evolutionary optimization domain (Wei et al., 2021) to select two parent individuals from a heterogeneous population.

Original: First, we divide the entire population into two subgroups: one with individuals exclusively using decision trees, and the other with individuals only using logistic regression.
Revised: First, we divide the entire population into two subgroups: one with individuals exclusively using decision trees, and the other with individuals only using logistic regression.

Original: Then we define a hyperparameter called random mating probability rmp = 0 . 5 , which signifies the probability of selecting two parents from two subgroups.
Revised: Then we define a hyperparameter called random mating probability rmp = 0 . 25 , which signifies the probability of selecting two parents from two subgroups.

Original: In the remaining fifty percent of cases, we select two parents from either subgroup.
Revised: In the remaining seventy five percent of cases, we select two parents from either subgroup.

Original: 4. F EATURE G ENERATION
Revised: 4. F EATURE G ENERATION

Original: There are two operators for transforming two given parent individuals into new individuals, namely a self-competitive crossover operator and a guided mutation operator.
Revised: There are two operators for transforming two given parent individuals into new individuals, namely a self-competitive crossover operator and a guided mutation operator.

Original: The self-competitive crossover operator was originally developed to evolve multiple job scheduling rules (Nguyen et al., 2021).
Revised: The self-competitive crossover operator was originally developed to evolve multiple job scheduling rules (Nguyen et al., 2021).

Original: It keeps effective rules and modifies the less effective rules, which imitates the manual rule design process and is suitable for our feature construction scenario.
Revised: It keeps effective rules and modifies the less effective rules, which imitates the manual rule design process and is suitable for our feature construction scenario.

Original: Guided mutation was proposed to solve maximum clique problems (Zhang et al., 2005) by eliminating less useful variables, which is also important for classification problem.
Revised: Guided mutation was proposed to solve maximum clique problems (Zhang et al., 2005) by eliminating less useful variables, which is also important for classification problem.

Original: Thus, we decide to use these two operators in our algorithm.
Revised: Thus, we decide to use these two operators in our algorithm.

Original: Self-Competitive Crossover: First, we utilize the self-competitive crossover operator to exchange beneficial materials of two parent individuals.
Revised: Self-Competitive Crossover: First, we utilize the self-competitive crossover operator to exchange beneficial materials of two parent individuals.

Original: In general, the goal of the self-competitive operator is to move important materials from the best feature to the worst feature while keeping the best feature intact.
Revised: In general, the goal of the self-competitive operator is to move important materials from the best feature to the worst feature while keeping the best feature intact.

Original: For a given set of feature importance values { θ 1 . . .
Revised: For a given set of feature importance values { θ 1 . . .

Original: θ k } provided by the base learner, we can use a softmax function, P ( θ i ) =e
Revised: θ k } provided by the base learner, we can use a softmax function, P ( θ i ) =

Original: [Equation]
Revised: [Equation]

Original: ,  to determine the likelihood of selecting each feature θ i as the donor, and use a negative softmax function to determine the acceptor.
Revised: , where T represents temperature, to determine the likelihood of selecting each feature θ i as the donor, and use a negative softmax function to determine the acceptor.

Original: Given the preceding probability distribution, we begin by sampling the acceptor individual Φ acceptor negatively proportionate to P ( θ i ) , i.e., P ( − θ i ) .
Revised: Given the preceding probability distribution, we begin by sampling the acceptor individual Φ acceptor negatively proportionate to P ( θ i ) , i.e., P ( − θ i ) .

Original: Then we sample another individual Φ donor as the donor individual according to the probability P ( θ i ) .
Revised: Then we sample another individual Φ donor as the donor individual according to the probability P ( θ i ) .

Original: Thereafter, the crossover operator selects a subtree from Φ donor at random to replace a subtree of Φ acceptor .
Revised: Thereafter, the crossover operator selects a subtree from Φ donor at random to replace a subtree of Φ acceptor .

Original: This crossover operator ensures that the top-performing features are rarely destroyed, while the worst features have a high probability of being improved.
Revised: This crossover operator ensures that the top-performing features are rarely destroyed, while the worst features have a high probability of being improved.

Original: Guided Mutation: In addition to the self-competitive operator, the guided mutation operator is utilized to generate new features.
Revised: Guided Mutation: In addition to the self-competitive operator, the guided mutation operator is utilized to generate new features.

Original: The guided mutation operator is similar to the random mutation operator in that it replaces a subtree of a parent individual with a randomly generated subtree.
Revised: The guided mutation operator is similar to the random mutation operator in that it replaces a subtree of a parent individual with a randomly generated subtree.

Original: However, the guided mutation operator uses a guided probability vector p to determine the probability of using each terminal variable in a newly generated subtree.
Revised: However, the guided mutation operator uses a guided probability vector p to determine the probability of using each terminal variable in a newly generated subtree.

Original: For each terminal variable x , the guided probability { p 1 . . .
Revised: For each terminal variable x , the guided probability { p 1 . . .

Original: p m } is defined as the percentage of occurrences of variable x in all features in the archive, weighted by the feature importance of each feature.
Revised: p m } is defined as the percentage of occurrences of variable x in all features in the archive, weighted by the feature importance of each feature.

Original: The feature importance values of a decision tree are calculated by counting the total reduction of Gini impurity with each feature ϕ , whereas the feature importance values in a linear model are obtained by calculating the absolute value of model coefficients.
Revised: The feature importance values of a decision tree are calculated by counting the total reduction of Gini impurity with each feature ϕ , whereas the feature importance values in a linear model are obtained by calculating the absolute value of model coefficients.

Original: All features are standardized before constructing the linear model.
Revised: All features are standardized before constructing the linear model.

Original: This ensures that the magnitude of each feature will not have an impact on the model coefficient.
Revised: This ensures that the magnitude of each feature will not have an impact on the model coefficient.

Original: 4. F EATURE E VALUATION
Revised: 4. F EATURE E VALUATION

Original: For each group of features, feature evaluation is conducted on its corresponding classification model.
Revised: For each group of features, feature evaluation is conducted on its corresponding classification model.

Original: To avoid over-fitting, we validate the effectiveness of each set of features using a cross-validation approach.
Revised: To avoid over-fitting, we validate the effectiveness of each set of features using a cross-validation approach.

Original: For any given training set, we partition it into five subsets, i.e., { X 1 . . .
Revised: For any given training set, we partition it into five subsets, i.e., { X 1 . . .

Original: X 5 } .
Revised: X 5 } .

Original: Then, for each round, we utilize four folds to train a model and the remaining folds to validate the learned model.
Revised: Then, for each round, we utilize four folds to train a model and the remaining folds to validate the learned model.

Original: Instead of summing all losses into a scalar, we keep the original loss vector because it contains more detailed and useful information which the selection operator can exploit and utilize.
Revised: Instead of summing all losses into a scalar, we keep the original loss vector because it contains more detailed and useful information which the selection operator can exploit and utilize.

Original: For each training instance, we employ cross entropy as the fitness function.
Revised: For each training instance, we employ cross entropy as the fitness function.

Original: If the problem contains C classes, the cross entropy loss can be defined as (cid:80) c ∈ C p c ∗ log ( q c ) , where q c refers to the probability of the model prediction belonging to a certain class c as q c and the ground truth probability distribution is p c .
Revised: If the problem contains C classes, the cross entropy loss can be defined as (cid:80) c ∈ C p c ∗ log ( q c ) , where q c refers to the probability of the model prediction belonging to a certain class c as q c and the ground truth probability distribution is p c .

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We conduct extensive experiments on synthetic and real-world datasets to answer the following three questions:
Revised: We conduct extensive experiments on synthetic and real-world datasets to answer the following three questions:

Original: • Q1 :
Revised: • Q1 :

Original: How does the proposed method perform compared with traditional machine learning (ML) methods (Section 5.2.1), deep learning-based (DL) methods (Section 5.2.2), and open-source feature engineering (FE) tools (Section 5.2.3)?
Revised: How does the proposed method perform compared with traditional machine learning (ML) methods (Section 5.2.1), deep learning-based (DL) methods (Section 5.2.2), and open-source feature engineering (FE) tools (Section 5.2.3)?

Original: • Q2 :
Revised: • Q2 :

Original: How does dynamically selecting base learners during the evolutionary process perform compared with determining them in advance (Section 5.3.1)?
Revised: How does dynamically selecting base learners during the evolutionary process perform compared with determining them in advance (Section 5.3.1)?

Revised: • Q3 :

Revised: Is it more efficient to use both cross-validation losses and feature importance values to guide the search than using either one alone (Section 5.3.2)?

Original: [Figure]
Revised: [Figure]

Original: LightGBM
Revised: LightGBM

Original: XGBoost RF
Revised: XGBoost RF

Original: FEW AutoFeat
Revised: FEW AutoFeat

Original: FTTransformer MLP
Revised: FTTransformer MLP

Original: DT DCN V
Revised: DT DCN V

Original: ResNet LR
Revised: ResNet LR

Original: KNN
Revised: KNN

Original: Algorithm
Revised: Algorithm

Original: Mean
Revised: Mean

Original: Imp.
Revised: Imp.

Original: ↑
Revised: ↑

Original: EvoFeat
Revised: EvoFeat

Original: [Table]
Revised: [Table]

Original: LightGBM
Revised: LightGBM

Original: [Table]
Revised: [Table]

Original: XGBoost
Revised: XGBoost

Original: [Table]
Revised: [Table]

Original: RF
Revised: RF

Original: [Table] [Table]
Revised: [Table] [Table]

Original: FEW
Revised: FEW

Original: [Table]
Revised: [Table]

Original: AutoFeat
Revised: AutoFeat

Original: [Table]
Revised: [Table]

Original: FTTransformer ‡ 0.
Revised: FTTransformer ‡ 0.

Original: MLP ‡ 0.
Revised: MLP ‡ 0.

Original: DT † 0.
Revised: DT † 0.

Original: DCN V2 ‡ 0.
Revised: DCN V2 ‡ 0.

Original: ResNet ‡ 0.
Revised: ResNet ‡ 0.

Original: LR † 0.
Revised: LR † 0.

Original: KNN † 0.
Revised: KNN † 0.

Original: • Q3 :

Original: Is it more efficient to use both cross-validation losses and feature importance values to guide the search than using either one alone (Section 5.3.2)?

Original: 5. E XPERIMENTAL S ETTINGS
Revised: 5. E XPERIMENTAL S ETTINGS

Original: Experimental Datasets: The experimental datasets include all datasets in DIGEN (Orzechowski & Moore, 2021) 2 and all PMLB (Romano et al., 2021)
Revised: Experimental Datasets: The experimental datasets include all datasets in DIGEN (Orzechowski & Moore, 2021) 2 and all PMLB (Romano et al., 2021)

Original: 3 classification datasets between 200 and 10000 instances.
Revised: 3 classification datasets between 200 and 10000 instances.

Original: DIGEN is a diverse set of datasets synthesized by the genetic programming method, whereas PMLB includes real-world datasets collected from OpenML (Vanschoren et al., 2014).
Revised: DIGEN is a diverse set of datasets synthesized by the genetic programming method, whereas PMLB includes real-world datasets collected from OpenML (Vanschoren et al., 2014).

Original: In large-scale experiments, we test the performance of our approach against other machine learning algorithms on all 119 datasets in PMLB with up to 10000 instances.
Revised: In large-scale experiments, we test the performance of our approach against other machine learning algorithms on all 119 datasets in PMLB with up to 10000 instances.

Original: In ablation studies, we use all datasets in DIGEN and all datasets in PMLB ranging from 200 to 2000 instances due to limited computational resources.
Revised: In ablation studies, we use all datasets in DIGEN and all datasets in PMLB ranging from 200 to 2000 instances due to limited computational resources.

Original: Evaluation Protocol: For the evaluation protocol, we randomly sample 80% data as the training set to train a model and use the remaining 20% data as the test set to calculate the testing balanced accuracy of the trained model.
Revised: Evaluation Protocol: For the evaluation protocol, we randomly sample 80% data as the training set to train a model and use the remaining 20% data as the test set to calculate the testing balanced accuracy of the trained model.

Original: To obtain a reliable result, we repeat all experiments with ten random seeds and take the average score as the final score.
Revised: To obtain a reliable result, we repeat all experiments with ten random seeds and take the average score as the final score.

Original: For all baseline algorithms, we tune them by Heteroscedastic
Revised: For all baseline algorithms, we tune them by Heteroscedastic

Original: Evolutionary Bayesian Optimization (HEBO) (Cowen-Rivers et al., 2020) or heuristic methods.
Revised: Evolutionary Bayesian Optimization (HEBO) (Cowen-Rivers et al., 2020) or heuristic methods.

Original: The details of the tuning methods and all hyperparameter search spaces are shown in the supplementary material.
Revised: The details of the tuning methods and all hyperparameter search spaces are shown in the supplementary material.

Original: 5. L ARGE - SCALE E XPERIMENT 5.2. C OMPARISON WITH T RADITIONAL M ACHINE L EARNING M ETHODS
Revised: 5. L ARGE - SCALE E XPERIMENT 5.2. C OMPARISON WITH T RADITIONAL M ACHINE L EARNING M ETHODS

Original: Baseline Algorithms: First of all, we compare our algorithm with traditional machine learning algorithms to demonstrate how feature construction improves the performance.
Revised: Baseline Algorithms: First of all, we compare our algorithm with traditional machine learning algorithms to demonstrate how feature construction improves the performance.

Original: We compare our algorithm to six traditional machine learning classification algorithms, including XGBoost (Chen &
Revised: We compare our algorithm to six traditional machine learning classification algorithms, including XGBoost (Chen &

Original: Guestrin, 2016), LightGBM (Ke et al., 2017), Random Forest (RF), Decision Tree (DT),
Revised: Guestrin, 2016), LightGBM (Ke et al., 2017), Random Forest (RF), Decision Tree (DT),

Original: Logistic
Revised: Logistic

Original: Regression (LR), and K-Nearest Neighbors (KNN).
Revised: Regression (LR), and K-Nearest Neighbors (KNN).

Original: Among these methods, XGBoost and LightGBM are regarded as state-of-the-art algorithms (Gorishniy et al., 2021).
Revised: Among these methods, XGBoost and LightGBM are regarded as state-of-the-art algorithms (Gorishniy et al., 2021).

Original: Experimental Results: Figure 2 illustrates the testing accuracy distribution of all algorithms across 119 datasets.
Revised: Experimental Results: Figure 2 illustrates the testing accuracy distribution of all algorithms across 119 datasets.

Original: It is clear that gradient boosting methods, such as XGBoost and LightGBM, are top-performing classification algorithms.
Revised: It is clear that gradient boosting methods, such as XGBoost and LightGBM, are top-performing classification algorithms.

Original: However, our approach achieves a better average accuracy2 https than these algorithms.
Revised: However, our approach achieves a better average accuracy than these algorithms.

Original: The numerical results shown in Table 1 confirm this, as our method achieves an average accuracy of 83%, which has more than 1% improvement over the average accuracies of
Revised: The numerical results shown in Table 1 confirm this, as our method achieves an average accuracy of 83%, which has more than 1% improvement over the average accuracies of

Original: XGBoost and LightGBM.
Revised: XGBoost and LightGBM.

Original: A further analysis shows that EvoFeat ranks first on 51 out of 119 datasets,  while LightGBM only ranks first on 26 datasets.
Revised: A further analysis shows that EvoFeat ranks first on 51 out of 119 datasets, https:/ while LightGBM only ranks first on 26 datasets.

Original: These results demonstrate that the evolution-based feature construction method enables an ensemble learning model composed of simple base learners to outperform state-of-the-art machine learning models.
Revised: These results demonstrate that the evolution-based feature construction method enables an ensemble learning model composed of simple base learners to outperform state-of-the-art machine learning models.

Original: It is worth mentioning that EvoFeat ranks first does not mean it performs the best on every dataset.
Revised: It is worth mentioning that EvoFeat ranks first does not mean it performs the best on every dataset.

Original: Due to space constraints, we discuss this topic in the supplementary material using the instance space analysis technique.
Revised: Due to space constraints, we discuss this topic in the supplementary material using the instance space analysis technique.

Original: 5.2. C OMPARISON WITH D EEP L EARNING M ETHODS
Revised: 5.2. C OMPARISON WITH D EEP L EARNING M ETHODS

Original: Baseline Algorithms: There are a lot of deep learning-based methods that have been proposed in recent years for tabular data learning.
Revised: Baseline Algorithms: There are a lot of deep learning-based methods that have been proposed in recent years for tabular data learning.

Original: We select four models from the existing literature for comparison, i.e., multilayer perceptron (MLP), ResNet, DCN V2 (Wang et al., 2021) and FTTransformer (Gorishniy et al., 2021).
Revised: We select four models from the existing literature for comparison, i.e., multilayer perceptron (MLP), ResNet, DCN V2 (Wang et al., 2021) and FTTransformer (Gorishniy et al., 2021).

Original: The first two architectures are commonly used baselines in the existing literature.
Revised: The first two architectures are commonly used baselines in the existing literature.

Original: The third one is the state-of-the-art model for click-through rate (CTR) prediction, and the fourth model is the state-of-the-art model for tabular data learning.
Revised: The third one is the state-of-the-art model for click-through rate (CTR) prediction, and the fourth model is the state-of-the-art model for tabular data learning.

Original: Experimental Results: The experimental results in Table 1 reveal that existing deep learning approaches cannot compete with XGBoost or LightGBM because the best one, FT-Transformer, only achieves 78% average accuracy.
Revised: Experimental Results: The experimental results in Table 1 reveal that existing deep learning approaches cannot compete with XGBoost or LightGBM because the best one, FT-Transformer, only achieves 78% average accuracy.

Original: Surprisingly, all deep learning algorithms perform similarly to a fine-tuned decision tree, which has an average accuracy of 77%.
Revised: Surprisingly, all deep learning algorithms perform similarly to a fine-tuned decision tree, which has an average accuracy of 77%.

Original: One probable explanation is that these deep learning algorithms require a large amount of training data and thus fail to learn a robust feature representation on datasets with less than ten thousand items, which is however typical for tabular data.
Revised: One probable explanation is that these deep learning algorithms require a large amount of training data and thus fail to learn a robust feature representation on datasets with less than ten thousand items, which is however typical for tabular data.

Original: 5.2. C OMPARISON WITH F EATURE C ONSTRUCTION M ETHODS
Revised: 5.2. C OMPARISON WITH F EATURE C ONSTRUCTION M ETHODS

Original: Baseline Algorithms: In this paper, we compare our method to two open-source feature construction methods: Feature engineering wrapper (FEW) (La Cava & Moore, 2017b) and AutoFeat (Horn et al., 2019).
Revised: Baseline Algorithms: In this paper, we compare our method to two open-source feature construction methods: Feature engineering wrapper (FEW) (La Cava & Moore, 2017b) and AutoFeat (Horn et al., 2019).

Original: FEW is a genetic programming based feature construction method, whereas AutoFeat is an exhaustive search based feature engineering method.
Revised: FEW is a genetic programming based feature construction method, whereas AutoFeat is an exhaustive search based feature engineering method.

Original: They are used to construct features for the best model among Extra Trees (ET), RF, XGBoost and LightGBM.
Revised: They are used to construct features for the best model among Extra Trees (ET), RF, XGBoost and LightGBM.

Original: The details of experiment settings are given in the supplementary material.
Revised: The details of experiment settings are given in the supplementary material.

Original: Experimental Results: Experiment results in Table 1 show that both FEW and AutoFeat cannot achieve top performance, with accuracy scores of 80% and 79%, respectively.
Revised: Experimental Results: Experiment results in Table 1 show that both FEW and AutoFeat cannot achieve top performance, with accuracy scores of 80% and 79%, respectively.

Original: FEW uses the coefficient of determination and feature importance values to search for high-order features, and uses the cross-validation loss to determine the historically best set of features.
Revised: FEW uses the coefficient of determination and feature importance values to search for high-order features, and uses the cross-validation loss to determine the historically best set of features.

Original: However, generating features without considering the cross-validation loss may mislead the search process and diminish the likelihood of uncovering expressive features.
Revised: However, generating features without considering the cross-validation loss may mislead the search process and diminish the likelihood of uncovering expressive features.

Original: AutoFeat employs L1-regularized LR to determine superfluous high-order features.
Revised: AutoFeat employs L1-regularized LR to determine superfluous high-order features.

Original: However, features useful on LR may not work well on RF, and therefore its feature construction capability is also limited.
Revised: However, features useful on LR may not work well on RF, and therefore its feature construction capability is also limited.

Original: 5. A BLATION S TUDIES 5.3. B ASE L EARNERS
Revised: 5. A BLATION S TUDIES 5.3. B ASE L EARNERS

Original: Due to the difficulties of generating a diverse set of linear models, most ensemble learning systems employ decision trees as base learners rather than linear regression models.
Revised: Due to the difficulties of generating a diverse set of linear models, most ensemble learning systems employ decision trees as base learners rather than linear regression models.

Original: However, the construction of non-linear features may lead to different results.
Revised: However, the construction of non-linear features may lead to different results.

Original: Therefore, we perform an ablation study in which we only use linear regression models or decision trees during the evolutionary feature construction process to investigate the benefit of simultaneously constructing features for linear regression models and decision trees.
Revised: Therefore, we perform an ablation study in which we only use linear regression models or decision trees during the evolutionary feature construction process to investigate the benefit of simultaneously constructing features for linear regression models and decision trees.

Original: Figure 3 illustrates the balanced testing accuracy with various base learners over 20 datasets in PMLB.
Revised: Figure 3 illustrates the balanced testing accuracy with various base learners over 20 datasets in PMLB.

Original: The experimental results reveal that there is no free lunch when it comes to addressing classification problems, and each method has its own set of niching tasks.
Revised: The experimental results reveal that there is no free lunch when it comes to addressing classification problems, and each method has its own set of niching tasks.

Original: Nonetheless, the combination of random decision tree and logistic regression allows our system to automatically select the appropriate model on the fly, resulting in improved average performance.
Revised: Nonetheless, the combination of random decision tree and logistic regression allows our system to automatically select the appropriate model on the fly, resulting in improved average performance.

Original: Table 2 shows the mean accuracy of different base learners over 89 PMLB datasets and 40 DIGEN datasets.
Revised: Table 2 shows the mean accuracy of different base learners over 89 PMLB datasets and 40 DIGEN datasets.

Original: This table illustrates that the combination of DT and LR achieves an accuracy score of 86%, which is more than 1.
Revised: This table illustrates that the combination of DT and LR achieves an accuracy score of 86%, which is more than 1.2% higher than dropping either of them, confirming the effectiveness of using heterogeneous base learners.

Original:  p r nn _
Revised: In this paper, we propose to use an evolutionary approach to exploit the information provided by feature importance and cross-validation loss. It is worthwhile investigating whether the evolutionary p r nn _

Original: p r o f b s a h e a s c h
Revised: p r o f b s a h e a s c h

Original: i z s o l a r s o l a r s o n a r
Revised: i z s o l a r s o l a r s o n a r

Original: s
Revised: s

Original: o
Revised: o

Original: y b
Revised: y b

Original: e s p e c t s p e c t t h r ee t i c _ t t o k y o v e h i c
Revised: e s p e c t s p e c t t h r ee t i c _ t t o k y o v e h i c

Original: v o t e v o w e l w db c w i n e _ x d 6 y
Revised: v o t e v o w e l w db c w i n e _ x d 6 y

Original: e
Revised: e

Original: a s t
Revised: a s t

Original: Dataset 0.
Revised: Dataset 0.

Original: S c
Revised: S c

Original: o r e
Revised: o r e

Original: [Figure]
Revised: [Figure]

Original: LR
Revised: LR

Original: [Figure]
Revised: [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 5.3. E VOLUTIONARY A LGORITHMS

Original: In this paper, we propose to use an evolutionary approach to exploit the information provided by feature importance and cross-validation loss.

Original: It is worthwhile investigating whether the evolutionary algorithm effectively uses this information during the search process.
Revised: algorithm effectively uses this information during the search process.

Original: Otherwise, random search would be a preferable option because it is much simpler.
Revised: Otherwise, random search would be a preferable option because it is much simpler.

Original: Therefore, this section presents an ablation experiment to demonstrate the efficacy of combined selection operators, which is the driving force of an evolutionary algorithm.
Revised: Therefore, this section presents an ablation experiment to demonstrate the efficacy of combined selection operators, which is the driving force of an evolutionary algorithm.

Original: Four selection operators: random selection, only automatic lexicase selection (AL), only selfcompetitive selection (SC), and both AL and SC (AL+SC) are compared in this section.
Revised: Four selection operators: random selection, only automatic lexicase selection (AL), only selfcompetitive selection (SC), and both AL and SC (AL+SC) are compared in this section.

Original: Figurepresents the distribution of testing accuracy with respect to three selection mechanisms on
Revised: Figurepresents the distribution of testing accuracy with respect to three selection mechanisms on

Original: DIGEN datasets.
Revised: DIGEN datasets.

Original: From this figure, it is clear that using two selection operators simultaneously can effectively improve the predictive accuracy of the final searched features.
Revised: From this figure, it is clear that using two selection operators simultaneously can effectively improve the predictive accuracy of the final searched features.

Original: Table 3 displays the mean testing classification accuracy across 129 datasets for different selection operators.
Revised: Table 3 displays the mean testing classification accuracy across 129 datasets for different selection operators.

Original: These results confirm the effectiveness of using both selection operators in the searching process, where the accuracy score of combining two operators is more than 0.32% higher than that of using only one of these operators.
Revised: These results confirm the effectiveness of using both selection operators in the searching process, where the accuracy score of combining two operators is more than 0.32% higher than that of using only one of these operators.

Original: Therefore, we can conclude that leveraging both cross-validation loss and feature-importance information is beneficial for obtaining better final results.
Revised: Therefore, we can conclude that leveraging both cross-validation loss and feature-importance information is beneficial for obtaining better final results.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: Automated feature engineering is a critical component of an automated machine learning system.
Revised: Automated feature engineering is a critical component of an automated machine learning system.

Original: In this research, we develop and present an evolution-based feature construction method for augmenting standard machine learning algorithms.
Revised: In this research, we develop and present an evolution-based feature construction method for augmenting standard machine learning algorithms.

Original: To make the technique applicable to a wide range of tasks, we simultaneously evolve features for decision trees and linear regression models and form an ensemble of top-performing models.
Revised: To make the technique applicable to a wide range of tasks, we simultaneously evolve features for decision trees and linear regression models and form an ensemble of top-performing models.

Original: To increase the search capability of our algorithm, we use both feature importance values and cross-validation losses to identify promising features.
Revised: To increase the search capability of our algorithm, we use both feature importance values and cross-validation losses to identify promising features.

Original: Experimental results on 119 datasets show that the proposed method outperforms both state-of-the-art tree-based ensemble learning methods and deep learning methods.
Revised: Experimental results on 119 datasets show that the proposed method outperforms both state-of-the-art tree-based ensemble learning methods and deep learning methods.

Original: For future study, considering the current base learner is confined to decision trees and logistic regression models, it is worth investigating whether other learners, such as neural networks or
Revised: For future study, considering the current base learner is confined to decision trees and logistic regression models, it is worth investigating whether other learners, such as neural networks or

Original: GBDTs, could be added to this framework.
Revised: GBDTs, could be added to this framework.

Original: Furthermore, exploring how to adapt our algorithm to large-scale datasets is an intriguing issue; instance selection methods (Bi et al., 2021b) may be required to reduce computational complexity.
Revised: Furthermore, exploring how to adapt our algorithm to large-scale datasets is an intriguing issue; instance selection methods (Bi et al., 2021b) may be required to reduce computational complexity.