Original: Completing the Model Optimization Process by Correcting Patterns of Failure in Regression Tasks
Revised: Completing the Model Optimization Process by Correcting Patterns of Failure in Regression Tasks

Original: Abstract
Revised: Abstract

Original: Model selection and hyper-parameter optimization sometimes prove to be complex and costly processes with unfinished outcomes.
Revised: Model selection and hyper-parameter optimization sometimes prove to be complex and costly processes with unfinished outcomes.

Original: In fact, a so-called optimized model can still suffer from patterns of failure when predicting on new data, affecting the generalization error.
Revised: In fact, a so-called optimized model can still suffer from patterns of failure when predicting on new data, affecting the generalization error.

Original: In this paper, we focus on regression tasks and introduce an additional stage to the model optimization process in order to render it more reliable.
Revised: In this paper, we focus on regression tasks and introduce an additional stage to the model optimization process in order to render it more reliable.

Original: This new step aims to correct error patterns when the model makes predictions on unlabeled data.
Revised: This new step aims to correct error patterns when the model makes predictions on unlabeled data.

Original: To that end, our method includes two techniques.
Revised: To that end, our method includes two techniques.

Original: AutoCorrect Rules leverage the model under/overestimation bias and applies simple rules to adjust predictions.
Revised: AutoCorrect Rules leverage the model under/overestimation bias and applies simple rules to adjust predictions.

Original: AutoCorrect
Revised: AutoCorrect

Original: Model is a supervised approach which exploits different representations to predict residuals in order to revise model predictions.
Revised: Model is a supervised approach which exploits different representations to predict residuals in order to revise model predictions.

Original: We empirically prove the relevance of our method on the outcome of an AutoML tool using different time budgets, and on a specific optimization case leveraging a pre-trained model for an image regression task.
Revised: We empirically prove the relevance of our method on the outcome of an AutoML tool using different time budgets, and on a specific optimization case leveraging a pre-trained model for an image regression task.

Original: 1. Introduction
Revised: 1. Introduction

Original: As the learner selection and hyper-parameter tuning process can prove to be complex various Automated Machine Learning (AutoML) libraries have been proposed to address this task for a given training dataset and optimization metric.
Revised: As the learner selection and hyper-parameter tuning process can prove to be complex, various Automated Machine Learning (AutoML) libraries have been proposed to address this task for a given training dataset and optimization metric.

Original: Those tools have specificities with different search strategies, such as Bayesian optimization with meta-learning and model ensembles [12], combining the strengths of Bayesian optimization and Hyperband [11] or ECI-based (Estimated Cost for Improvement) learner choice and cost-effective hyper-parameter tuning [29].
Revised: Those tools have specificities with different search strategies, such as Bayesian optimization with meta-learning and model ensembles [12], combining the strengths of Bayesian optimization and Hyperband [11], or ECI-based (Estimated Cost for Improvement) learner choice and cost-effective hyper-parameter tuning [29].

Original: Still, Machine Learning (ML) models are inclined to failure modes when error patterns surface during inference [28].
Revised: Still, Machine Learning (ML) models are inclined to failure modes when error patterns surface during inference [28].

Original: Even though a model displays good overall performance, some regions of the feature space may be prone to higher error rates.
Revised: Even though a model displays good overall performance, some regions of the feature space may be prone to higher error rates.

Original: For example, a regression model could be subject to an overestimation bias in a given area of the feature space.
Revised: For example, a regression model could be subject to an overestimation bias in a given area of the feature space.

Original: These failure patterns turn out to be harmful in regression applications such as price prediction in finance or dysmorphic facial sign detection in the medical field.
Revised: These failure patterns turn out to be harmful in regression applications such as price prediction in finance or dysmorphic facial sign detection in the medical field.

Original: Scope and contribution
Revised: Scope and contribution

Original: In this paper, the focus is on regression tasks.
Revised: In this paper, the focus is on regression tasks.

Original: We consider the output of an optimization process, i.e. an ML model M ∗ , based on an initial configuration χ = ( M , H , D t , T, m, e ) , where M and H denote the algorithm space and the hyper-parameter space, respectively.
Revised: We consider the output of an optimization process, i.e. an ML model M ∗ , based on an initial configuration χ = ( M , H , D t , T, m, e ) , where M and H denote the algorithm space and the hyper-parameter space, respectively.

Original: D t is the training dataset.
Revised: D t is the training dataset.

Original: T is the time budget.
Revised: T is the time budget.

Original: m denotes an objective metric to be minimized (e.g. mean squared error MSE) and e is an evaluation method (e.g. cross-validation).
Revised: m denotes an objective metric to be minimized (e.g. mean squared error MSE) and e is an evaluation method (e.g. cross-validation).

Original: M ∗ has been trained on D t ( X , Y ) , with the features X and target variable Y , and is ready to be deployed in order to make predictions on new (unlabeled) data.
Revised: M ∗ has been trained on D t ( X , Y ) , with the features X and target variable Y , and is ready to be deployed in order to make predictions on new (unlabeled) data.

Original: We aim to extend the model optimization process with an additional stage in order to give unbiased and reliable generalization properties to M ∗ .
Revised: We aim to extend the model optimization process with an additional stage in order to give unbiased and reliable generalization properties to M ∗ .

Original: This new step is intended for correcting potential patterns of failure surfacing during inference.
Revised: This new step is intended for correcting potential patterns of failure surfacing during inference.

Original: We do not alter the parameters of M ∗ , but we adjust its predictions based on error patterns detected by our method.
Revised: We do not alter the parameters of M ∗ , but we adjust its predictions based on error patterns detected by our method.

Original: To that end, we propose two techniques.
Revised: To that end, we propose two techniques.

Original: In order to assess the ML model under/overestimation bias, AutoCorrect Rules first calibrates rules based on the model prediction values on a dataset with residuals ( Y − ˆ Y ).
Revised: In order to assess the ML model under/overestimation bias, AutoCorrect Rules first calibrates rules based on the model prediction values on a dataset with residuals ( Y − ˆ Y ).

Original: The prediction is then adjusted during inference to counteract any potential error bias.
Revised: The prediction is then adjusted during inference to counteract any potential error bias.

Original: AutoCorrect Model leverages X and prediction ˆ Y =
Revised: AutoCorrect Model leverages X and prediction ˆ Y =

Original: M ∗ ( X ) as features, and the residuals as target variable.
Revised: M ∗ ( X ) as features, and the residuals as target variable.

Original: When the ML model predicts based on new data, AutoCorrect Model then estimates the residual values in order to adjust the original
Revised: When the ML model predicts based on new data, AutoCorrect Model then estimates the residual values in order to adjust the original

Original: ML model predictions.
Revised: ML model predictions.

Original: We experimentally show the relevance of those techniques on two types of use cases: (i) On tabular data where the ML model has been optimized via an AutoML tool with various time budgets.
Revised: We experimentally show the relevance of those techniques on two types of use cases: (i) On tabular data where the ML model has been optimized via an AutoML tool with various time budgets.

Original: (ii) On image data where the ML model includes a frozen pre-trained model and additional top layers tuned on new data.
Revised: (ii) On image data where the ML model includes a frozen pre-trained model and additional top layers tuned on new data.

Original: This second application is thus a specific optimization use case based on transfer learning.
Revised: This second application is thus a specific optimization use case based on transfer learning.

Original: 2. AutoCorrect method Notations 
Revised: 2. AutoCorrect method Notations and assumptions

Original: and assumptions

Original: We consider a training dataset D t with x i , y i ∈
Revised: We consider a training dataset D t with x i , y i ∈

Original: X × Y for i = 1 , ..., N of i.i.d.
Revised: X × Y for i = 1 , ..., N of i.i.d.

Original: realizations of random variables X , Y ∼ P , where the joint distribution P is unknown.
Revised: realizations of random variables X , Y ∼ P , where the joint distribution P is unknown.

Original: The ML model is a predictor M ∗ : X → Y with X ⊆ R d and Y ⊆ R .
Revised: The ML model is a predictor M ∗ : X → Y with X ⊆ R d and Y ⊆ R .

Original: M ∗ is the outcome of an optimization process (e.g. AutoML): argmin M h , h ( m ( M h , D t )) , subject to budget
Revised: M ∗ is the outcome of an optimization process (e.g. AutoML): argmin M h , h ( m ( M h , D t )) , subject to budget

Original: T and evaluation method e , and where algorithm M h ∈ M and hyper-parameters h ∈ H , as defined in the previous section.
Revised: T and evaluation method e , and where algorithm M h ∈ M and hyper-parameters h ∈ H , as defined in the previous section.

Original: At the time of inference, the evaluation set { x i } i>N is unlabeled.
Revised: At the time of inference, the evaluation set { x i } i>N is unlabeled.

Original: Prerequisite
Revised: Prerequisite

Original: AutoCorrect Rules and Model both require a calibration dataset including instances x i , y i , the corresponding ML model predictions
Revised: AutoCorrect Rules and Model both require a calibration dataset including instances x i , y i , the corresponding ML model predictions

Original: ˆ y i , and thus the residuals r i
Revised: ˆ y i , and thus the residuals r i

Original: =
Revised: =

Original: y
Revised: y

Original: i − ˆ y i .
Revised: i − ˆ y i .

Original: These should be computed on data which has not been not used to train M ∗ .
Revised: These should be computed on data which has not been not used to train M ∗ .

Original: Similarly to cross-validation, we randomly partition D t into k mutually exclusive folds D t 1 , ..., D tk of equal size.
Revised: Similarly to cross-validation, we randomly partition D t into k mutually exclusive folds D t 1 , ..., D tk of equal size.

Original: M ∗ is repeatedly trained k times on D t \ D tj , with j ∈ 1 , ..., k .
Revised: M ∗ is repeatedly trained k times on D t \ D tj , with j ∈ 1 , ..., k .

Original: The model predictions, and thus the residuals, are computed based on D tj .
Revised: The model predictions, and thus the residuals, are computed based on D tj .

Original: We thus obtain D t + , the whole dataset with residuals.
Revised: We thus obtain D t + , the whole dataset with residuals.

Original: Lastly, we train M ∗ on the whole dataset D t .
Revised: Lastly, we train M ∗ on the whole dataset D t .

Original: We note that this process can be naturally included in a model selection and hyper-parameter optimization process (i.e. AutoML tool).
Revised: We note that this process can be naturally included in a model selection and hyper-parameter optimization process (i.e. AutoML tool).

Original: According to Kohavi [18], if the ML model remains stable under the perturbations created by removing each fold, the cross-validation estimate of the generalization error will remain unbiased.
Revised: According to Kohavi [18], if the ML model remains stable under the perturbations created by removing each fold, the cross-validation estimate of the generalization error will remain unbiased.

Original: Better stability should be observed with a large number of subsets.
Revised: Better stability should be observed with a large number of subsets.

Original: Further, for most models, the variance of the estimate of the error rate will either be constant or decreasing with k .
Revised: Further, for most models, the variance of the estimate of the error rate will either be constant or decreasing with k .

Original: We follow the author’s suggestion to use ten folds as a reasonable trade-off between stability and computational cost.
Revised: We follow the author’s suggestion to use ten folds as a reasonable trade-off between stability and computational cost.

Original: 2.1. AutoCorrect Rules
Revised: 2.1. AutoCorrect Rules

Original: The residual plot, showing the residuals r i against the fitted values ˆ y i , is a diagnostic graph employed to check whether the variance of residuals is constant in linear regression models [1].
Revised: The residual plot, showing the residuals r i against the fitted values ˆ y i , is a diagnostic graph employed to check whether the variance of residuals is constant in linear regression models [1].

Original: We exploit this representation and apply it to ML models, in order to assess any potential bias in the residuals, based on the ML model prediction values.
Revised: We exploit this representation and apply it to ML models, in order to assess any potential bias in the residuals, based on the ML model prediction values.

Original: r i > 0 amounts to a model underestimation bias as y i >
Revised: r i > 0 amounts to a model underestimation bias as y i >

Original: ˆ y i .
Revised: ˆ y i .

Original: To this end, we use the dataset with residuals D t + and define P ∗ interval bins B p .
Revised: To this end, we use the dataset with residuals D t + and define P ∗ interval bins B p .

Original: Each B p is the set of samples whose prediction values ˆ y fall into the interval I p .
Revised: Each B p is the set of samples whose prediction values ˆ y fall into the interval I p .

Original: I p are equally-spaced intervals of ˆ y : I p =
Revised: I p are equally-spaced intervals of ˆ y : I p =

Original: [ m − + ( m + − m − ) × p − 1 P ∗ , m − + ( m + − m − ) × pP ∗ ) , where m − =
Revised: [ m − + ( m + − m − ) × p − 1 P ∗ , m − + ( m + − m − ) × pP ∗ ) , where m − =

Original: min (ˆ y i ) and m + = max (ˆ y i ) , with 1 ≤ p ≤ P ∗ and 1 ≤ i ≤ N .
Revised: min (ˆ y i ) and m + = max (ˆ y i ) , with 1 ≤ p ≤ P ∗ and 1 ≤ i ≤ N .

Original: We compute the mean bias error MBE ( B p )
Revised: We com- pute the mean bias error MBE ( B p )

Original: = 1 N p (cid:80) N p j =1 r j for each B p , where N p denotes the number of samples in B p .
Revised: = 1 N p (cid:80) N p j =1 r j for each B p , where N p denotes the number of samples in B p .

Original: Figure 1 shows two types of residual plots with different degrees of error pattern.
Revised: Figure 1 shows two types of residual plots with different degrees of error pattern.

Original: During inference on a new sample x ′ , the prediction is adjusted based on M ∗ ( x ′ ) value:
Revised: During inference on a new sample x ′ , the prediction is adjusted based on M ∗ ( x ′ ) value:

Original: ˆ y ′ adj =
Revised: ˆ y ′ adj =

Original: M ∗ ( x ′ )
Revised: M ∗ ( x ′ )

Original: + MBE ( B p )
Revised: + MBE ( B p )

Original: M ∗ ( x ′ ) ∈
Revised: M ∗ ( x ′ ) ∈

Original: I p (with potential boundaries on ˆ y ′ adj depending on the use case, e.g. positive amounts).
Revised: I p (with potential boundaries on ˆ y ′ adj depending on the use case, e.g. positive amounts).

Original: If M ∗ ( x ′ ) < min (ˆ y i ) or M ∗ ( x ′ ) > max (ˆ y i ) , then the applied correction is MBE ( B 1 ) or MBE ( B P ∗ ) , respectively.
Revised: If M ∗ ( x ′ ) < min (ˆ y i ) or M ∗ ( x ′ ) > max (ˆ y i ) , then the applied correction is MBE ( B 1 ) or MBE ( B P ∗ ) , respectively.

Original: Lastly, the optimal number of bins is computed with D t + using cross-validation.
Revised: Lastly, the optimal number of bins is computed with D t + using cross-validation.

Original: 2.2. AutoCorrect Model
Revised: 2.2. AutoCorrect Model

Original: While rules can be relevant for some use cases, predicting residual values with a regression model may prove to be more effective.
Revised: While rules can be relevant for some use cases, predicting residual values with a regression model may prove to be more effective.

Original: D t + is thus used to train a new regression model A ∗ : X ×Y → Z , with X ⊆ R d , Y ⊆ R and
Revised: D t + is thus used to train a new regression model A ∗ : X ×Y → Z , with X ⊆ R d , Y ⊆ R and

Original: Z ⊆ R : A ∗ = argmin
Revised: Z ⊆ R : A ∗ = argmin

Original: A ( 1 N (cid:80) Ni
Revised: A ( 1 N (cid:80) Ni

Original: =1 L ( r i , A
Revised: =1 L ( r i , A

Original: ( x i , ˆ y i ))) .
Revised: ( x i , ˆ y i ))) .

Original: A is chosen in
Revised: A is chosen in

Original: M , and L is a loss function consistent with the objective metric m : for example, if m is the mean squared error, L will be the squared loss.
Revised: M , and L is a loss function consistent with the objective metric m : for example, if m is the mean squared error, L will be the squared loss.

Original: A ∗ ( X , ˆ Y ) is an estimate of E ( R | X , ˆ Y ) , with R =
Revised: A ∗ ( X , ˆ Y ) is an estimate of E ( R | X , ˆ Y ) , with R =

Original: Y − ˆ Y .
Revised: Y − ˆ Y .

Original: During inference on a new sample x ′ , the prediction is adjusted: ˆ y ′ adj =
Revised: During inference on a new sample x ′ , the prediction is adjusted: ˆ y ′ adj =

Original: M ∗ ( x ′ )
Revised: M ∗ ( x ′ )

Original: + A ∗ ( x ′ , ˆ y ′ ) (with potential boundaries on ˆ y ′ adj depending on the use case), with ˆ y ′ =
Revised: + A ∗ ( x ′ , ˆ y ′ ) (with potential boundaries on ˆ y ′ adj depending on the use case), with ˆ y ′ =

Original: M ∗ ( x ′ ) .
Revised: M ∗ ( x ′ ) .

Original: A ∗ can leverage various representations as features, such as X and/or ˆ Y .
Revised: A ∗ can leverage various representations as features, such as X and/or ˆ Y .

Original: When X is sparse or if d >> N , more parsimonious representations can be adequate.
Revised: When X is sparse or if d >> N , more parsimonious representations can be adequate.

Original: For instance, in a multi-target regression setting where Y ⊆ R k with k > 1 , any dependency between the targets could be exploited.
Revised: For instance, in a multi-target regression setting where Y ⊆ R k with k > 1 , any dependency between the targets could be exploited.

Original: A ∗ ( ˆ Y ) may then be helpful to learn error patterns and correct the predictions at inference time.
Revised: A ∗ ( ˆ Y ) may then be helpful to learn error patterns and correct the predictions at inference time.

Original: In the multitarget case, A ∗ could be either a specialized model correcting predictions related to one target Y j (i.e. r j is the target used to train A ∗ ) or a general model with several targets Y .
Revised: In the multitarget case, A ∗ could be either a specialized model correcting predictions related to one target Y j (i.e. r j is the target used to train A ∗ ) or a general model with several targets Y .

Original: Algorithm 1 gives an outline of the whole AutoCorrect learning approach.
Revised: Algorithm 1 gives an outline of the whole AutoCorrect learning approach.

Original: for
Revised: for

Original: K do
Revised: K do

Original: Train
Revised: Train

Original: Predict ˆ
Revised: Predict ˆ

Original: M
Revised: M

Original: Compute r
Revised: Compute r

Original: i
Revised: i

Original: y
Revised: y

Original: i − ˆ y i on D tj ;
Revised: i − ˆ y i on D tj ;

Original: D t + j = D tj ∪ { ˆ y i , r i } i ∈ D tj ; end
Revised: D t + j = D tj ∪ { ˆ y i , r i } i ∈ D tj ; end

Original: D t + =
Revised: D t + =

Original: D t + j ) 1 ≤ j ≤ K ;
Revised: D t + j ) 1 ≤ j ≤ K ;

Original: Optimize number using
Revised: Optimize number using

Original: Compute interval bins B p p ≤
Revised: Compute interval bins B p p ≤

Original: ; for
Revised: ; for

Original: ∗ do
Revised: ∗ do

Original: MBE ( B p ) =
Revised: MBE ( B p ) =

Original: 1 N p (cid:80) N p j =1 r j ; end
Revised: 1 N p (cid:80) N p j =1 r j ; end

Original: A ∗ = argmin
Revised: A ∗ = argmin

Original: A ( 1 N
Revised: A ( 1 N

Original: Ni =1 L ( r i , A x
Revised: Ni =1 L ( r i , A x

Original: ˆ
Revised: ˆ

Original: Output: MBE
Revised: Output: MBE

Original: B p ) 1 ≤ p ≤ P ∗ ; A ∗
Revised: B p ) 1 ≤ p ≤ P ∗ ; A ∗

Original: 3. Experiments 3.1. Assessing the outcome of an AutoML tool Datasets
Revised: 3. Experiments 3.1. Assessing the outcome of an AutoML tool Datasets

Original: With the Ames Housing Dataset (use case 1), we aim to predict the sale price of residential properties in Iowa, with 80 explanatory variables [6, 7].
Revised: With the Ames Housing Dataset (use case 1), we aim to predict the sale price of residential properties in Iowa, with 80 explanatory variables [6, 7].

Original: We use the 1,460 instances for which the target is available.
Revised: We use the 1,460 instances for which the target is available.

Original: The Seoul Bike Sharing Demand Dataset (use case 2) is a tabular dataset with 8,760 instances.
Revised: The Seoul Bike Sharing Demand Dataset (use case 2) is a tabular dataset with 8,760 instances.

Original: The target is the count of public bikes rented every hour in Seoul [9, 10, 24].
Revised: The target is the count of public bikes rented every hour in Seoul [9, 10, 24].

Original: In each case, 20% of the original dataset is kept for testing and the rest is used as D t .
Revised: In each case, 20% of the original dataset is kept for testing and the rest is used as D t .

Original: Modeling settings
Revised: Modeling settings

Original: The dataset with residuals is constructed with a 10-fold strategy.
Revised: The dataset with residuals is constructed with a 10-fold strategy.

Original: The number of bins in AutoCorrect Rules is optimized using 5-fold cross-validation with values in [2 , 10] .
Revised: The number of bins in AutoCorrect Rules is optimized using 5-fold cross-validation with values in [2 , 10] .

Original: Experiments are run over 4 seeds (distinct dataset splits).
Revised: Experiments are run over 4 seeds (distinct dataset splits).

Original: M ∗ and A ∗ are both selected and optimized using Fast Lightweight AutoML (FLAML)
Revised: M ∗ and A ∗ are both selected and optimized using Fast Lightweight AutoML (FLAML)

Original: [22, 29] with various time budgets, and with the MSE optimization metric.
Revised: [22, 29] with various time budgets, and with the MSE optimization metric.

Original: Time budget includes learner selection and hyper-parameter tuning.
Revised: Time budget includes learner selection and hyper-parameter tuning.

Original: A ∗ budget is set to min( M ∗ budget , 300 s ) .
Revised: A ∗ budget is set to min( M ∗ budget , 300 s ) .

Original: Experiments are run on an Intel i7-7500U CPU with 12 GB RAM.
Revised: Experiments are run on an Intel i7-7500U CPU with 12 GB RAM.

Original: Learners are selected among Random Forest [3], LGBM [16], XGBoost [5], CatBoost [23], and
Revised: Learners are selected among Random Forest [3], LGBM [16], XGBoost [5], CatBoost [23], and

Original: Extra Trees [13].
Revised: Extra Trees [13].

Original: FLAML includes a specific cost-related algorithm for tuning hyper-parameters [30], called CostFrugal Optimization (CFO) method.
Revised: FLAML includes a specific cost-related algorithm for tuning hyper-parameters [30], called CostFrugal Optimization (CFO) method.

Original: This randomized search algorithm starts in a low-cost region, moves gradually towards a low-loss area, and runs until the end of the budget.
Revised: This randomized search algorithm starts in a low-cost region, moves gradually towards a low-loss area, and runs until the end of the budget.

Original: h denotes the hyperparameter configuration, f and g are the respective loss and cost function, and δ refers to the step size.
Revised: h denotes the hyperparameter configuration, f and g are the respective loss and cost function, and δ refers to the step size.

Original: The initial point h 0 has a small g ( h 0 ) .
Revised: The initial point h 0 has a small g ( h 0 ) .

Original: At each iteration, a vector u is sampled uniformly at random from the unit sphere:
Revised: At each iteration, a vector u is sampled uniformly at random from the unit sphere:

Original: – if f ( h + δ u ) < f ( h ) , then h ← h + δ u , – else if f ( h − δ u ) <
Revised: – if f ( h + δ u ) < f ( h ) , then h ← h + δ u , – else if f ( h − δ u ) <

Original: f ( h ) , then h ← h − δ u .
Revised: f ( h ) , then h ← h − δ u .

Original: If no progress is being made or in order to avoid getting confined in a local optimum, δ can be dynamically adjusted or the process can restart from a randomized starting point.
Revised: If no progress is being made or in order to avoid getting confined in a local optimum, δ can be dynamically adjusted or the process can restart from a randomized starting point.

Original: This approach provides guarantees for a convergence rate of O ( √ d/ √ K ) , with d denoting the search space dimensionality and K the number of iterations, and for an upper bounded cost.
Revised: This approach provides guarantees for a convergence rate of O ( √ d/ √ K ) , with d denoting the search space dimensionality and K the number of iterations, and for an upper bounded cost.

Original: Lastly, we also use A ∗ RF , a simple Autocorrect model based on a random forest with 100 trees, without optimizing its hyper-parameters.
Revised: Lastly, we also use A ∗ RF , a simple Autocorrect model based on a random forest with 100 trees, without optimizing its hyper-parameters.

Original: [Table]
Revised: [Table]

Original: Results
Revised: Results

Original: M ∗ is considered as the baseline.
Revised: M ∗ is considered as the baseline.

Original: Table 1 shows the evolution of the test root mean squared error (RMSE) after applying AutoCorrect Rules and 2 different AutoCorrect Models:
Revised: Table 1 shows the evolution of the test root mean squared error (RMSE) after applying AutoCorrect Rules and 2 different AutoCorrect Models:

Original: A ∗ ( x , ˆ y ) and A ∗ RF ( x , ˆ y ) .
Revised: A ∗ ( x , ˆ y ) and A ∗ RF ( x , ˆ y ) .

Original: The relative change in error is, on average, almost always negative (a few minor exceptions in red), which demonstrates a reliable behavior.
Revised: The relative change in error is, on average, almost always negative (a few minor exceptions in red), which demonstrates a reliable behavior.

Original: In use case 1, although M ∗
Revised: In use case 1, although M ∗

Original: RMSE tends to converge with increasing time budget, AutoCorrect Models reveal that there is still potential for further optimization.
Revised: RMSE tends to converge with increasing time budget, AutoCorrect Models reveal that there is still potential for further optimization.

Original: On the other hand, in use case 2, with budgets greater than 360s, this potential proves to be thinner.
Revised: On the other hand, in use case 2, with budgets greater than 360s, this potential proves to be thinner.

Original: In that case, the AutoML tool seems to have produced an adequate outcome.
Revised: In that case, the AutoML tool seems to have produced an adequate outcome.

Original: Lastly, A ∗ RF ( x , ˆ y ) mean training time is 4.8s (case 1) and 8.3s (case 2).
Revised: Lastly, A ∗ RF ( x , ˆ y ) mean training time is 4.8s (case 1) and 8.3s (case 2).

Original: This simple model proves to be a useful low-cost alternative.
Revised: This simple model proves to be a useful low-cost alternative.

Original: 3.2. Optimizing a pre-trained model for image regression Dataset
Revised: 3.2. Optimizing a pre-trained model for image regression Dataset

Original: The Facial Key Points Detection (use case 3) is a face image dataset based on 96 ×pixels [2].
Revised: The Facial Key Points Detection (use case 3) is a face image dataset based on 96 ×pixels [2].

Original: The objective is to predict the (x,y) coordinates for 15 key points (e.g. nose tip), for applications such as dysmorphic facial sign detection.
Revised: The objective is to predict the (x,y) coordinates for 15 key points (e.g. nose tip), for applications such as dysmorphic facial sign detection.

Original: There are thus 30 target variables.
Revised: There are thus 30 target variables.

Original: We retain 2,140 entries, i.e. images without missing target values.
Revised: We retain 2,140 entries, i.e. images without missing target values.

Original: 20% of the original dataset is kept for testing and the rest is used as D t .
Revised: 20% of the original dataset is kept for testing and the rest is used as D t .

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Modeling settings
Revised: Modeling settings

Original: M ∗ is trained on all the key points.
Revised: M ∗ is trained on all the key points.

Original: It has the following architecture: pretrained MobileNet v2 [26] with non-trainable weights, completed with global max pooling, a dense layer (128 units), batch normalization, ReLu activation, and a final layer for the 30 outputs.
Revised: It has the following architecture: pretrained MobileNet v2 [26] with non-trainable weights, completed with global max pooling, a dense layer (128 units), batch normalization, ReLu activation, and a final layer for the 30 outputs.

Original: The
Revised: The

Original: MSE is minimized with Adam [17].
Revised: MSE is minimized with Adam [17].

Original: The number of epochs is tuned with an early stopping strategy (patience of 5) based on validation loss.
Revised: The number of epochs is tuned with an early stopping strategy (patience of 5) based on validation loss.

Original: We use AutoCorrect to optimize predictions on two target values: (x,y) for the left eye center.
Revised: We use AutoCorrect to optimize predictions on two target values: (x,y) for the left eye center.

Original: A ∗ ( ˆ y ) leverages the dependencies between the 30 coordinates to predict the residuals.
Revised: A ∗ ( ˆ y ) leverages the dependencies between the 30 coordinates to predict the residuals.

Original: We use one specific AutoCorrect Model ( Y → Z with Y ⊆ R 30 and Z ⊆ R ) for each coordinate of the left eye center in order to predict the corresponding residuals.
Revised: We use one specific AutoCorrect Model ( Y → Z with Y ⊆ R 30 and Z ⊆ R ) for each coordinate of the left eye center in order to predict the corresponding residuals.

Original: Each A ∗ is optimized using FLAML, with T = 60 s and m = MSE .
Revised: Each A ∗ is optimized using FLAML, with T = 60 s and m = MSE .

Original: Lastly, we also use A ∗ RF ( ˆ y ) , a simple Autocorrect model using a random forest with 100 trees, without hyper-parameter optimization.
Revised: Lastly, we also use A ∗ RF ( ˆ y ) , a simple Autocorrect model using a random forest with 100 trees, without hyper-parameter optimization.

Original: Results
Revised: Results

Original: Table 2 demonstrates the efficacy of AutoCorrect Rules and especially AutoCorrect
Revised: Table 2 demonstrates the efficacy of AutoCorrect Rules and especially AutoCorrect

Original: Model.
Revised: Model.

Original: There are significant patterns in the residuals that are well exploited by both approaches.
Revised: There are significant patterns in the residuals that are well exploited by both approaches.

Original: Figure 2 exhibits the effect of correcting with A ∗ on 3 test instances.
Revised: Figure 2 exhibits the effect of correcting with A ∗ on 3 test instances.

Original: Lastly, A ∗ RF mean training time is 2.9s (left eye center x) and 3.0s (left eye center y).
Revised: Lastly, A ∗ RF mean training time is 2.9s (left eye center x) and 3.0s (left eye center y).

Original: This model turns out to be a valuable low-cost alternative to A ∗ .
Revised: This model turns out to be a valuable low-cost alternative to A ∗ .

Original: 4. Conclusion and future work
Revised: 4. Conclusion and future work

Original: In this paper, we presented AutoCorrect which aims to complete the optimization process and render it more reliable for a moderate additional time budget.
Revised: In this paper, we presented AutoCorrect which aims to complete the optimization process and render it more reliable for a moderate additional time budget.

Original: On average, AutoCorrect in fact improves the generalization properties of a model by detecting patterns in the residuals.
Revised: On average, AutoCorrect in fact improves the generalization properties of a model by detecting patterns in the residuals.

Original: AutoCorrect could also be used in validation during the optimization process to assess whether an ML model requires further optimization.
Revised: AutoCorrect could also be used in validation during the optimization process to assess whether an ML model requires further optimization.

Original: Moreover, additional AutoML tools could be tested and compared for given time budgets.
Revised: Moreover, additional AutoML tools could be tested and compared for given time budgets.

Original: More runs could be performed to better assess the reliability of the methodology.
Revised: More runs could be performed to better assess the reliability of the methodology.

Original: The practicality of AutoCorrect should also be assessed when the data size is large.
Revised: The practicality of AutoCorrect should also be assessed when the data size is large.

Original: Extending the optimization pipeline obviously adds complexity and computational costs.
Revised: Extending the optimization pipeline obviously adds complexity and computational costs.

Original: On the other hand, AutoCorrect is also a promise for potential improvement in performance and additional transparency.
Revised: On the other hand, AutoCorrect is also a promise for potential improvement in performance and additional transparency.

Original: In fact, AutoCorrect can reveal patterns in the residuals whose explainability could be performed with methods such as Shapley values [21, 27].
Revised: In fact, AutoCorrect can reveal patterns in the residuals whose explainability could be performed with methods such as Shapley values [21, 27].

Original: Lastly, we could study the efficacy of AutoCorrect when distribution shifts occur.
Revised: Lastly, we could study the efficacy of AutoCorrect when distribution shifts occur.

Original: Appendix A. Related work
Revised: Appendix A. Related work

Original: The focus of research related to error detection and error rate estimation has mainly been on neural networks and classification tasks.
Revised: The focus of research related to error detection and error rate estimation has mainly been on neural networks and classification tasks.

Original: Singla et al.
Revised: Singla et al.

Original: [28] adapted the Fault Tree Analysis [20] to deep neural networks by representing the model failure modes with interpretable features in classification tasks.
Revised: [28] adapted the Fault Tree Analysis [20] to deep neural networks by representing the model failure modes with interpretable features in classification tasks.

Original: Leveraging the model confidence, such as the maximum softmax probability in classification tasks, is a first step to detecting incorrect predictions during inference [15].
Revised: Leveraging the model confidence, such as the maximum softmax probability in classification tasks, is a first step to detecting incorrect predictions during inference [15].

Original: However, modern neural networks have proved to be poorly calibrated [14].
Revised: However, modern neural networks have proved to be poorly calibrated [14].

Original: Kuleshov et al.
Revised: Kuleshov et al.

Original: [19] extend calibration to regression in order to produce uncertainty estimates.
Revised: [19] extend calibration to regression in order to produce uncertainty estimates.

Original: According to them, a forecaster M is calibrated if the empirical cumulative function (CDF), based on the target values y t , matches the predicted CDF, based on the outputs of M , when the dataset size goes to infinity.
Revised: According to them, a forecaster M is calibrated if the empirical cumulative function (CDF), based on the target values y t , matches the predicted CDF, based on the outputs of M , when the dataset size goes to infinity.

Original: The problem of accuracy estimation with unlabeled data has received more attention on classification than on regression tasks.
Revised: The problem of accuracy estimation with unlabeled data has received more attention on classification than on regression tasks.

Original: For instance, Chen et al.
Revised: For instance, Chen et al.

Original: [4] employ an ensemble of models to detect errors and estimate a classifier’s accuracy.
Revised: [4] employ an ensemble of models to detect errors and estimate a classifier’s accuracy.

Original: With regard to regression, Donmez et al.
Revised: With regard to regression, Donmez et al.

Original: [8] estimate the error rate assuming that the target distribution p ( y ) is known, in order to evaluate the maximum likelihood of the predictor outputs.
Revised: [8] estimate the error rate assuming that the target distribution p ( y ) is known, in order to evaluate the maximum likelihood of the predictor outputs.

Original: With regard to model uncertainty in regression tasks, Romano et al.
Revised: With regard to model uncertainty in regression tasks, Romano et al.

Original: suggest combining the strengths of conformal prediction and quantile regression [25].
Revised: suggest combining the strengths of conformal prediction and quantile regression [25].

Original: This method, called conformalized quantile regression , produces valid and shorter prediction intervals, with lengths varying according to heteroskedasticity in the data.
Revised: This method, called conformalized quantile regression , produces valid and shorter prediction intervals, with lengths varying according to heteroskedasticity in the data.

Original: Appendix B. Facial key point detection: local explainability of A ∗ outputs
Revised: Appendix B. Facial key point detection: local explainability of A ∗ outputs

Original: [Table]
Revised: [Table]