Original: PowerGraph: Using neural networks and principal components to determine multivariate statistical power trade-offs
Revised: PowerGraph: Using neural networks and principal components to determine multivariate statistical power trade-offs

Original: Anonymous Authors

Original: Abstract
Revised: Abstract

Original: Statistical power estimation for studies with multiple model parameters is inherently a multivariate problem.
Revised: Statistical power estimation for studies with multiple model parameters is inherently a multivariate problem.

Original: Power for individual parameters of interest cannot be reliably estimated univariately since correlation and variance explained relative to one parameter will impact the power for another parameter, all usual univariate considerations being equal.
Revised: Power for individual parameters of interest cannot be reliably estimated univariately since correlation and variance explained relative to one parameter will impact the power for another parameter, all usual univariate considerations being equal.

Original: Explicit solutions in such cases, especially for models with many parameters, are either impractical or impossible to solve, leaving researchers to the prevailing method of simulating power.
Revised: Explicit solutions in such cases, especially for models with many parameters, are either impractical or impossible to solve, leaving researchers to the prevailing method of simulating power.

Original: However, the point estimates for a vector of model parameters are uncertain, and the impact of inaccuracy is unknown.
Revised: However, the point estimates for a vector of model parameters are uncertain, and the impact of inaccuracy is unknown.

Original: In such cases, sensitivity analysis is recommended such that multiple combinations of possible observable parameter vectors are simulated to understand power trade-offs.
Revised: In such cases, sensitivity analysis is recommended such that multiple combinations of possible observable parameter vectors are simulated to understand power trade-offs.

Original: A limitation to this approach is that it is computationally expensive to generate sufficient sensitivity combinations to accurately map the power trade-off function in increasingly high-dimensional spaces for the models that social scientists estimate.
Revised: A limitation to this approach is that it is computationally expensive to generate sufficient sensitivity combinations to accurately map the power trade-off function in increasingly high-dimensional spaces for the models that social scientists estimate.

Original: This paper explores the efficient estimation and graphing of statistical power for a study over varying model parameter combinations.
Revised: This paper explores the efficient estimation and graphing of statistical power for a study over varying model parameter combinations.

Original: We propose a simple and generalizable machine learning inspired solution to cut the computational cost to less than 10% of the brute force method while providing F1 scores above 90%.
Revised: We propose a simple and generalizable machine learning inspired solution to cut the computational cost to less than 10% of the brute force method while providing F1 scores above 90%.

Original: We further motivate the impact of transfer learning in learning power manifolds across varying distributions.
Revised: We further motivate the impact of transfer learning in learning power manifolds across varying distributions.

Original: 1. Introduction
Revised: 1. Introduction

Original: Statistical power is quantitatively equal to the probability of rejecting a null hypothesis.
Revised: Statistical power is quantitatively equal to the probability of rejecting a null hypothesis.

Original: Thus, it plays a crucial role in finding an effect of hypothesized interest in any study.
Revised: Thus, it plays a crucial role in finding an effect of hypothesized interest in any study.

Original: Historically, the simplest and most direct way to improve power is to increase the study’s sample size, as mathematically, it has the most significant impact concerning what the researcher can control (Cohen, 1992).
Revised: Historically, the simplest and most direct way to improve power is to increase the study’s sample size, as mathematically, it has the most significant impact concerning what the researcher can control (Cohen, 1992).

Original: However, there are practical considerations to make when increasing the sample size.
Revised: However, there are practical considerations to make when increasing the sample size.

Original: In the case of a rare disease study, the total population itself might be small, difficult to locate, and even more challenging to engage, as in early- and late-stage neuro-genetic syndromes (Button et al., 2013; Szucs & Ioannidis, 2017).
Revised: In the case of a rare disease study, the total population itself might be small, difficult to locate, and even more challenging to engage, as in early- and late-stage neuro-genetic syndromes (Button et al., 2013; Szucs & Ioannidis, 2017).

Original: On the other hand, there could be access or funding issues with increasing the number of participants a researcher wishes to obtain.
Revised: On the other hand, there could be access or funding issues with increasing the number of participants a researcher wishes to obtain.

Original: In this article, we empirically show that tuning the model parameters ( i.e. , weights) can dramatically change the study’s power.
Revised: In this article, we empirically show that tuning the model parameters ( i.e. , weights) can dramatically change the study’s power.

Original: Thus, the change in model parameters can push the survey from a mid-powered study to well-powered research even for a constant sample size.
Revised: Thus, the change in model parameters can push the survey from a mid-powered study to well-powered research even for a constant sample size.

Original: For the rest of the article, we assume that well-powered research is one where the power is higher than 0.8 .
Revised: For the rest of the article, we assume that well-powered research is one where the power is higher than 0.8 .

Original: Fig.
Revised: Fig.

Original: 1 demonstrates that even for a fixed sample size, the power can considerab1y vary due to a change in model parameters.
Revised: 1 demonstrates that even for a fixed sample size, the power can considerab1y vary due to a change in model parameters.

Original: In Fig.
Revised: In Fig.

Original: 1a we compute power gradients by considering the directional difference between clusters derived by KMeans, where the power and the squared L2-norm of the model parameters represent the point coordinates.
Revised: 1a we compute power gradients by considering the directional difference between clusters derived by KMeans, where the power and the squared L2-norm of the model parameters represent the point coordinates.

Original: Thus, generating such a manifold, as shown in Fig.
Revised: Thus, generating such a manifold, as shown in Fig.

Original: 1a can exceptionally aid researchers in identifying areas of high power.
Revised: 1a can exceptionally aid researchers in identifying areas of high power.

Original: Enabling access to these plots is thus a simple way of reducing the ideal sample size without sacrificing power.
Revised: Enabling access to these plots is thus a simple way of reducing the ideal sample size without sacrificing power.

Original: However, as described in Algorithm 1, computing power even once requires 200-1000 simulations.
Revised: However, as described in Algorithm 1, computing power even once requires 200-1000 simulations.

Original: Further, the manifold parameter space, including the model weights, sample size, and additional hyperparameter choices, is enormous.
Revised: Further, the manifold parameter space, including the model weights, sample size, and additional hyperparameter choices, is enormous.

Original: For instance, for a seven predictor model, with 11 choices per predictor and 11 choices for the N , we have a parameter space of 11 7+1 .
Revised: For instance, for a seven predictor model, with 11 choices per predictor and 11 choices for the N , we have a parameter space of 11 7+1 .

Original: Computation for this vast space might even take days.
Revised: Computation for this vast space might even take days.

Original: Thus, in this work, we look at cheaper alternatives to simulating the entire power manifold.
Revised: Thus, in this work, we look at cheaper alternatives to simulating the entire power manifold.

Original: We present three different contributions
Revised: We present three different contributions

Original: • We develop a neural network-based approach to predict power over a high-dimensional manifold for classification and regression purposes.
Revised: • We develop a neural network-based approach to predict power over a high-dimensional manifold for classification and regression purposes.

Original: Such an approach provides significant performance with even 10% of the training data.
Revised: Such an approach provides significant performance with even 10% of the training data.

Original: • We provide four baselines for alternatively predicting power and test all our approaches on three models-= regression, logistic and Repeated Measures ANOVA.
Revised: • We provide four baselines for alternatively predicting power and test all our approaches on three models-= regression, logistic and Repeated Measures ANOVA.

Original: • To leverage the transfer of knowledge from one model’s power manifold to another model’s manifold, we first demonstrate the intuition behind such a transfer.
Revised: • To leverage the transfer of knowledge from one model’s power manifold to another model’s manifold, we first demonstrate the intuition behind such a transfer.

Original: Next, we show that such a transfer can enable the same or better results in every case.
Revised: Next, we show that such a transfer can enable the same or better results in every case.

Original: We organize the rest of the article as follows.
Revised: We organize the rest of the article as follows.

Original: We first detail the relevant literature and highlight the background information required to build our algorithm.
Revised: We first detail the relevant literature and highlight the background information required to build our algorithm.

Original: We provide pseudocodes for the core and auxiliary algorithms in this section.
Revised: We provide pseudocodes for the core and auxiliary algorithms in this section.

Original: Next we present our empirical results, highlight the significant wins, and report the limitations.
Revised: Next, we present our empirical results, highlight the significant wins, and report the limitations.

Original: Finally, we also present potential future work.
Revised: Finally, we also present potential future work.

Original: [Figure]
Revised: [Figure]

Original: 2. Related Work
Revised: 2. Related Work

Original: Previous articles ((Bakker et al., 2012), (Bakker et al., 2016), (Maxwell, 2004), (Cohen, 1992)) have pointed out that studies are frequently underpowered and thus lead to statistically insignificant results.
Revised: Previous articles ((Bakker et al., 2012), (Bakker et al., 2016), (Maxwell, 2004), (Cohen, 1992)) have pointed out that studies are frequently underpowered and thus lead to statistically insignificant results.

Original: Underpowered studies are primarily a result of a lack of a formal power analysis.
Revised: Underpowered studies are primarily a result of a lack of a formal power analysis.

Original: Furthermore, in (Card et al., 2020) the authors demonstrated that numerous
Revised: Furthermore, in (Card et al., 2020) the authors demonstrated that numerous

Original: NLP models are typically underpowered.
Revised: NLP models are typically underpowered.

Original: They mention that datasets with sentences less than 2000 lead to a power of only 75% and provide best practices to perform power analyses.
Revised: They mention that datasets with sentences less than 2000 lead to a power of only 75% and provide best practices to perform power analyses.

Original: Next, the work in (Koehn, 2004) provides ways to measure the statistical significance of the performance improvement due to a system change in the domain of machine translation.
Revised: Next, the work in (Koehn, 2004) provides ways to measure the statistical significance of the performance improvement due to a system change in the domain of machine translation.

Original: The authors argue that just a change in the system metric (ex., accuracy or F1-score) is not enough, and we need to demonstrate that the change in the metric is statistically significant.
Revised: The authors argue that just a change in the system metric (ex., accuracy or F1-score) is not enough, and we need to demonstrate that the change in the metric is statistically significant.

Original: This work further broadens the need for thorough power analysis.
Revised: This work further broadens the need for thorough power analysis.

Original: Finally, (Koehn, 2004) takes into consideration the effects of model complexity in extracting the maximum information from a given dataset.
Revised: Finally, (Koehn, 2004) takes into consideration the effects of model complexity in extracting the maximum information from a given dataset.

Original: Under a given class of models, the authors provide an algorithm to identify the right-fit model.
Revised: Under a given class of models, the authors provide an algorithm to identify the right-fit model.

Original: The right-fit model avoids both underfitting and overfitting and has predictive power asymptotically close to the best model in the given model class.
Revised: The right-fit model avoids both underfitting and overfitting and has predictive power asymptotically close to the best model in the given model class.

Original: Such a method can replace cross-validation and signifies the importance of improving predictive power.
Revised: Such a method can replace cross-validation and signifies the importance of improving predictive power.

Original: (Bakker et al., 2012) suggested power simulations for studies using Questionable Research Practices (QRPs) (John et al., 2012) and lower sample sizes with more trials.
Revised: (Bakker et al., 2012) suggested power simulations for studies using Questionable Research Practices (QRPs) (John et al., 2012) and lower sample sizes with more trials.

Original: QRPs could include running multiple tests with a much smaller sample size (underpowered), rerunning analyses after adding more subjects, or (even selectively) removing outliers.
Revised: QRPs could include running multiple tests with a much smaller sample size (underpowered), rerunning analyses after adding more subjects, or (even selectively) removing outliers.

Original: The results show that such practices can significantly inflate statistical power and provide misleading evidence about the effect size while hampering the study’s reproducibility.
Revised: The results show that such practices can significantly inflate statistical power and provide misleading evidence about the effect size while hampering the study’s reproducibility.

Original: (Bakker et al., 2012) suggests that to avoid such underpowered studies, we should use sample sizes derived after a formal power analysis.
Revised: (Bakker et al., 2012) suggests that to avoid such underpowered studies, we should use sample sizes derived after a formal power analysis.

Original: (Baker et al., 2021) solved the power issue by providing an online tool with power contours to demonstrate the effect of sample size ( N ) and trials per participant on statistical power ( k ).
Revised: (Baker et al., 2021) solved the power issue by providing an online tool with power contours to demonstrate the effect of sample size ( N ) and trials per participant on statistical power ( k ).

Original: With the results provided in the article, it is clear that changes to N or even k can dramatically change the power region and convert an underpowered study to an appropriately powered one.
Revised: With the results provided in the article, it is clear that changes to N or even k can dramatically change the power region and convert an underpowered study to an appropriately powered one.

Original: (Rast & Hofer, 2014) also demonstrates the strong (inversely correlated) impact of sample size on both the effect size and study design.
Revised: (Rast & Hofer, 2014) also demonstrates the strong (inversely correlated) impact of sample size on both the effect size and study design.

Original: (Lane & Hennes, 2018) and (Lane & Hennes, 2019) provide a clear guide to conducting formal power analysis based on simulation methods (rather than a formula-based approach).
Revised: (Lane & Hennes, 2018) and (Lane & Hennes, 2019) provide a clear guide to conducting formal power analysis based on simulation methods (rather than a formula-based approach).

Original: Even though the simulation approach is universal, it often requires many computational resources.
Revised: Even though the simulation approach is universal, it often requires many computational resources.

Original: The resource usage increases exponentially with a linear increase in the number of predictors or other factors impacting power.
Revised: The resource usage increases exponentially with a linear increase in the number of predictors or other factors impacting power.

Original: Thus, in this work, we highlight machine learning-based approaches that can seriously reduce resource usage with accuracies greater than 95%.
Revised: Thus, in this work, we highlight machine learning-based approaches that can seriously reduce resource usage with accuracies greater than 95%.

Original: 3. Proposed Work 3.1. Computing Power
Revised: 3. Proposed Work 3.1. Computing Power

Original: We present a power computation algorithm in Algorithmfor t-tests in a linear regression model.
Revised: We present a power computation algorithm in Algorithmfor t-tests in a linear regression model.

Original: We can extend this algorithm to the f-test by replacing the t-test in Line 9 with an f-test or a partial f-test.
Revised: We can extend this algorithm to the f-test by replacing the t-test in Line 9 with an f-test or a partial f-test.

Original: For extensions to other models, we only need to modify the data generation process on Line 5.
Revised: For extensions to other models, we only need to modify the data generation process on Line 5.

Original: Computing power for multiple model weights requires us to call the COMPUTE-POWER function each time.
Revised: Computing power for multiple model weights requires us to call the COMPUTE-POWER function each time.

Original: Thus, the cost of computing power boils down to the number of calls made to COMPUTE-POWER .
Revised: Thus, the cost of computing power boils down to the number of calls made to COMPUTE-POWER .

Original: We wish to reduce the number of calls while still predicting power for the entire parameter space in our work.
Revised: We wish to reduce the number of calls while still predicting power for the entire parameter space in our work.

Original: 3.2. Feature Engineering with PCA
Revised: 3.2. Feature Engineering with PCA

Original: Complex datasets include several features, and it often becomes necessary to reduce data dimensionality to conserve resources or speed up training.
Revised: Complex datasets include several features, and it often becomes necessary to reduce data dimensionality to conserve resources or speed up training.

Original: Furthermore, we wish to remove redundancy in features.
Revised: Furthermore, we wish to remove redundancy in features.

Original: Removing highly correlated features can improve training speeds or data processing speeds, reduce bias, and improve the interpretability of our dataset.
Revised: Removing highly correlated features can improve training speeds or data processing speeds, reduce bias, and improve the interpretability of our dataset.

Original: A common way to achieve these goals is to use
Revised: A common way to achieve these goals is to use

Original: Principal Component Analysis (PCA) (Wold et al., 1987).
Revised: Principal Component Analysis (PCA) (Wold et al., 1987).

Original: For brevity, we add further details of PCA to the appendix.
Revised: For brevity, we add further details of PCA to the appendix.

Original: We already know that increasing the sample size N can increase the power of a study.
Revised: We already know that increasing the sample size N can increase the power of a study.

Original: To visualize the impact of each parameter in the model, we take a look at a partial f-test.
Revised: To visualize the impact of each parameter in the model, we take a look at a partial f-test.

Original: The data follows the distribution of the first three variables from Table 5.
Revised: The data follows the distribution of the first three variables from Table 5.

Original: We compute the power of the partial f-test to test for the significance of the first and third predictors.
Revised: We compute the power of the partial f-test to test for the significance of the first and third predictors.

Original: To visualize the importance of each parameter, we draw a correlation plot in Figure 13.
Revised: To visualize the importance of each parameter, we draw a correlation plot in Figure 13.

Original: We can easily verify that as expected β 1 and β 3 have higher correlation with the power (as the hypothesis depends on them).
Revised: We can easily verify that as expected β 1 and β 3 have higher correlation with the power (as the hypothesis depends on them).

Original: N has a high correlation as we would expect.
Revised: N has a high correlation as we would expect.

Original: Further, we define a new feature scaled weight computed as Nσ , wherein σ = (cid:113)(cid:80) k
Revised: Further, we define a new feature scaled weight computed as Nσ , wherein σ = (cid:113)(cid:80) ki

Original: i =1 β 2 i ( k denotes the number of predictors).
Revised: =1 β 2 i ( k denotes the number of predictors).

Original: We observe that the
Revised: We observe that the

Original: Nσ feature has the highest correlation amongst non-PCA features.
Revised: Nσ feature has the highest correlation amongst non-PCA features.

Original: After including the principal components the correlation substantially increases to ≥ 0 .
Revised: After including the principal components the correlation substantially increases to ≥ 0 .

Original: 90 for PC 1 .
Revised: 90 for PC 1 .

Original: We believe that transforming the data with PCA can further increase the variance while ignoring the redundant features in our dataset.
Revised: We believe that transforming the data with PCA can further increase the variance while ignoring the redundant features in our dataset.

Original: We, provide exact details about our dataset in 4.2.
Revised: We, provide exact details about our dataset in 4.2.

Original: 3.3. Primary Approach: The Neural Network Approach
Revised: 3.3. Primary Approach: The Neural Network Approach

Original: We employ a simple neural network as described in Tabletrained on the new dataset.
Revised: We employ a simple neural network as described in Tabletrained on the new dataset.

Original: With the flexibility of a neural network, we can train for both a classification and a regression task.
Revised: With the flexibility of a neural network, we can train for both a classification and a regression task.

Original: We denote this approach by POWER-NETWORK or POWER-Neural Network ( PNN )).
Revised: We denote this approach by POWER-NETWORK or POWER-Neural Network ( PNN )).

Original: PNN also allows for an easy extension to a multi-class classifier.
Revised: PNN also allows for an easy extension to a multi-class classifier.

Original: However, we do not explore this domain in this article.
Revised: However, we do not explore this domain in this article.

Original: 3.3.1. L EARNING F ASTER WITH T RANSFER L EARNING
Revised: 3.3.1. Learning Faster with Transfer Learning

Original: The primary goal of our article is to reduce the number of calls to the power function and thus reduce the simulation overhead for computing power.
Revised: The primary goal of our article is to reduce the number of calls to the power function and thus reduce the simulation overhead for computing power.

Original: Thus, we now describe a commonly used technique fine-tuning (Houlsby et al., 2019) in Natural Language Processing (NLP) domains.
Revised: Thus, we now describe a commonly used technique fine-tuning (Houlsby et al., 2019) in Natural Language Processing (NLP) domains.

Original: Rather than cold-starting model training with random weights, we take the help of a previously trained model to initialize the model weights.
Revised: Rather than cold-starting model training with random weights, we take the help of a previously trained model to initialize the model weights.

Original: Thus, most layers are already trained, and our new smaller dataset can be used to tune the neural net for our new task.
Revised: Thus, most layers are already trained, and our new smaller dataset can be used to tune the neural net for our new task.

Original: Note that when the number of input dimensions is different we remove the unused dimensions (from the pre-trained model) by specifying zero input columns for those features.
Revised: Note that when the number of input dimensions is different we remove the unused dimensions (from the pre-trained model) by specifying zero input columns for those features.

Original: Formally, the transfer learning algorithm P-Transfer only deviates from PNN in the fact that line 6 of the algorithm 4 has an additional step of initializing the model (before training) with the weights of a larger pre-trained model.
Revised: Formally, the transfer learning algorithm P-Transfer only deviates from PNN in the fact that line 6 of the algorithm 4 has an additional step of initializing the model (before training) with the weights of a larger pre-trained model.

Original: Note that the original pre-trained model needs more features than the new model.
Revised: Note that the original pre-trained model needs more features than the new model.

Original: Finally, the extra features in the larger model are turned off by passing zero vectors to these features.
Revised: Finally, the extra features in the larger model are turned off by passing zero vectors to these features.

Original: Thus, even a larger model can be used for our smaller models.
Revised: Thus, even a larger model can be used for our smaller models.

Original: 3.3.2. I NTUITION BEHIND T RANSFER L EARNING
Revised: 3.3.2. Intuition behind Transfer Learning

Original: The intuition behind transfer learning may be sought from this article (Chinn, 2000).
Revised: The intuition behind transfer learning may be sought from this article (Chinn, 2000).

Original: We first note that each model directly interacts with the linear model.
Revised: We first note that each model directly interacts with the linear model.

Original: However, due to the difference in their composition, the effect size might differ.
Revised: However, due to the difference in their composition, the effect size might differ.

Original: The authors in(Chinn, 2000) empirically demonstrate that given the standard deviation, we can find the correlation between, say, the effect size of the logistic regression model and the regression model.
Revised: The authors in(Chinn, 2000) empirically demonstrate that given the standard deviation, we can find the correlation between, say, the effect size of the logistic regression model and the regression model.

Original: Thus, we would have the same standard deviation and thus correlated/comparable effect sizes for standardized predictors.
Revised: Thus, we would have the same standard deviation and thus correlated/comparable effect sizes for standardized predictors.

Original: Thus, we might be able to use transfer learning to boost a model with similar traits as long as the predictors are standardized and they both have the same hypothesis.
Revised: Thus, we might be able to use transfer learning to boost a model with similar traits as long as the predictors are standardized and they both have the same hypothesis.

Original: 3.4. Baselines: Power Cluster, Power Label Propagation and KNeighbors Classifier
Revised: 3.4. Baselines: Power Cluster, Power Label Propagation and KNeighbors Classifier

Original: POWERCLUSTER also referred to as P-CLUSTER ) simply runs K-Means on PC 1 and Nσ for identifying two clusters with two vectors from our dataset.
Revised: POWERCLUSTER also referred to as P-CLUSTER ) simply runs K-Means on PC 1 and Nσ for identifying two clusters with two vectors from our dataset.

Original: We provide its pseudocode in Algorithm 2.
Revised: We provide its pseudocode in Algorithm 2.

Original: K-Means identifies clusters of points that minimize the intra-cluster distances.
Revised: K-Means identifies clusters of points that minimize the intra-cluster distances.

Original: Since both of our feature vectors are independent of power, we can cluster our data points without computing the true power.
Revised: Since both of our feature vectors are independent of power, we can cluster our data points without computing the true power.

Original: The intuition behind deploying P-CLUSTER is that it can quickly identify appropriately varying power domains, as seen from Fig.
Revised: The intuition behind deploying P-CLUSTER is that it can quickly identify appropriately varying power domains, as seen from Fig.

Original: If we compare the clustering to the standard definition of a high-powered study ( i.e., power is more significant than 0.8 ), we can segregate power with high performance.
Revised: If we compare the clustering to the standard definition of a high-powered study ( i.e., power is more significant than 0.8 ), we can segregate power with high performance.

Original: We demonstrate the results in Figures 3a, 3b.
Revised: We demonstrate the results in Figures 3a, 3b.

Original: We can see that P-CLUSTER performs well arbitrarily.
Revised: We can see that P-CLUSTER performs well arbitrarily.

Original: Thus, we can conclude that not including label information will lead to poor performance for models.
Revised: Thus, we can conclude that not including label information will lead to poor performance for models.

Original: We provide two other strong baselines for power predictionthe label propagation algorithm from (Zhu & Ghahramani, 2002), and the KNeighbors Classifier (Nelli, 2018).
Revised: We provide two other strong baselines for power predictionthe label propagation algorithm from (Zhu & Ghahramani, 2002), and the KNeighbors Classifier (Nelli, 2018).

Original: We use the Label Propagation version implemented in ScikitLearn (Pedregosa et al., 2011).
Revised: We use the Label Propagation version implemented in ScikitLearn (Pedregosa et al., 2011).

Original: Label propagation works by first creating a fully-connected graph between all data points wherein each edge weight is directly dependent on the euclidean distance between the two points.
Revised: Label propagation works by first creating a fully-connected graph between all data points wherein each edge weight is directly dependent on the euclidean distance between the two points.

Original: These edge weights dictate the probability of propagating a label onto another label.
Revised: These edge weights dictate the probability of propagating a label onto another label.

Original: We direct the reader to section 2.2 of (Zhu &
Revised: We direct the reader to section 2.2 of (Zhu &

Original: Ghahramani, 2002) for further details on the Label Propagation algorithm.
Revised: Ghahramani, 2002) for further details on the Label Propagation algorithm.

Original: We also refer to it as PL-PROP .
Revised: We also refer to it as PL-PROP .

Original: KNeighbors classifier (Nelli, 2018) also provides good performance, and we thus include it for completeness.
Revised: KNeighbors classifier (Nelli, 2018) also provides good performance, and we thus include it for completeness.

Original: Overall,
Revised: Overall,

Original: Label Propagation performs consistently better than the
Revised: Label Propagation performs consistently better than the

Original: KNeighbors classifier.
Revised: KNeighbors classifier.

Revised: 4. Experiments 4.1. Code

Revised: We test the efficacy of the baseline algorithms and 4 with three kinds of statistical models.

Revised: For simulation, we use

Revised: Google Colaboratory with the standard run-time.

Revised: 1 Other experiment details are deferred to the appendix in section B.1.1.

Original: 1: Input: Distributions of columns in the dataset- D
Revised: 1: Input: Distributions of columns in the dataset- D

Original: = { D X 1 , D X 2 , ..., D X p } , Sample Size- N , Model Weightβ ∈ R p , Number of predictors- p , sensitivity-
Revised: = { D X 1 , D X 2 , ..., D X p } , Sample Size- N , Model Weightβ ∈ R p , Number of predictors- p , sensitivity-

Original: α ( default: 0.05), number of simulations- sims ( default: 1000 ),
Revised: α ( default: 0.05), number of simulations- sims ( default: 1000 ),

Original: Error Distribution-
Revised: Error Distribution-

Original: E 2: Output: Power of t-test 3: procedure COMPUTE-POWER( X, k ) 4: significance ←5: for 1: sims do 6:
Revised: E 2: Output: Power of t-test 3: procedure COMPUTE-POWER( X, k ) 4: significance ←5: for 1: sims do 6:

Original: X ← generate N data samples from 7: distribution( D ) 8: e ← generate error from distribution( E ) 9: y ← ( X × β ) + e 10:
Revised: X ← generate N data samples from 7: distribution( D ) 8: e ← generate error from distribution( E ) 9: y ← ( X × β ) + e 10:

Original: M ← Fit ( X, y ) to a linear regression model 11: if p-value of t-test for model M ≤ α then 12: significance +=13: end if 14: end for 15: power ← significancesims 16: return power 17: end procedure
Revised: M ← Fit ( X, y ) to a linear regression model 11: if p-value of t-test for model M ≤ α then 12: significance +=13: end if 14: end for 15: power ← significancesims 16: return power 17: end procedure

Original: 1: Input: Length of training set-
Revised: 1: Input: Length of training set-

Original: L , model parameter spaceS of length L (each parameter consists of the weight vector β and sample size N ), power sensitivity-
Revised: L , model parameter spaceS of length L (each parameter consists of the weight vector β and sample size N ), power sensitivity-

Original: α ( default 0.05 ), number of simulations- sims ( default 200 ),
Revised: α ( default 0.05 ), number of simulations- sims ( default 200 ),

Original: PCA variance (in %) to be retained- var , Number of clusters- k 2: Output: Power Surface Graph 3: procedure POWERCLUSTER( S , α , sims ) 4:
Revised: PCA variance (in %) to be retained- var , Number of clusters- k 2: Output: Power Surface Graph 3: procedure POWERCLUSTER( S , α , sims ) 4:

Original: S PCA ← PCA-FIT-TRANSFORM(S, variance= var ) 5:
Revised: S PCA ← PCA-FIT-TRANSFORM(S, variance= var ) 5:

Original: C k ← KMeans( S PCA , num clusters = k ) 6: return clusters C k derived from KMeans 7: end procedure 
Revised: C k ← KMeans( S PCA , num clusters = k ) 6: return clusters C k derived from KMeans 7: end procedure 4.2. Dataset

Revised: For sampling parameters we use P-SAMPLER (algorithm 5) to generate the parameter sample space and in conjunction with algorithm 3 to compute power.

Revised: Both of these algorithms together make up the power computation black box.

Revised: After collecting the dataset, we pre-process it using PCA, as explained below.

Revised: P-SAMPLER is inspired by the fungible weights described in (Waller, 2008).

Original:  1: Input: Length of training set-
Revised: Next, we provide exact details about the data we use for the methods introduced in this work. For both supervised and unsupervised tasks we use the same dataset albeit the unsupervised one does not contain the labels or true power information. The dataset 1: Input: Length of training set-

Original: L , model parameter spaceS of length L (each parameter consists of the weight vector β and sample size N ), power sensitivity-
Revised: L , model parameter spaceS of length L (each parameter consists of the weight vector β and sample size N ), power sensitivity-

Original: α ( default 0.05 ), number of simulations- sims ( default 200 ) 2: Output: Training Set S with parameters and corresponding powers 3: procedure GENERATE-DATA( S , α , sims ) 4: for parameter in S do 5: ( β, N ) ← parameter 6: power ← COMPUTE-POWER( β, N, α, sims ) 7:
Revised: α ( default 0.05 ), number of simulations- sims ( default 200 ) 2: Output: Training Set S with parameters and corresponding powers 3: procedure GENERATE-DATA( S , α , sims ) 4: for parameter in S do 5: ( β, N ) ← parameter 6: power ← COMPUTE-POWER( β, N, α, sims ) 7:

Original: S [powers] ← power 8: end for 9: end procedure
Revised: S [powers] ← power 8: end for 9: end procedure

Original: 1: Input: Length of training set-
Revised: 1: Input: Length of training set-

Original: L , model parameter spaceS of length L (each parameter consists of the weight vector β and sample size N ), power sensitivity-
Revised: L , model parameter spaceS of length L (each parameter consists of the weight vector β and sample size N ), power sensitivity-

Original: α ( default 0.05 ), number of simulations- sims ( default 200 ),
Revised: α ( default 0.05 ), number of simulations- sims ( default 200 ),

Original: PCA variance (in %) to be retained- var 2: Output: Power Surface Graph 3: procedure POWERNETWORK( S , α , sims ) 4:
Revised: PCA variance (in %) to be retained- var 2: Output: Power Surface Graph 3: procedure POWERNETWORK( S , α , sims ) 4:

Original: S ∗ ← GENERATE-DATA( S , α , sims ) 5:
Revised: S ∗ ← GENERATE-DATA( S , α , sims ) 5:

Original: S PCA ← PCA-FIT-TRANSFORM( S , variance= var ) 6:
Revised: S PCA ← PCA-FIT-TRANSFORM( S , variance= var ) 6:

Original: Train Neural Network Model M on ( S ∗ ∪ S PCA ) 7: return trained model M 8: end procedure 
Revised: Train Neural Network Model M on ( S ∗ ∪ S PCA ) 7: return trained model M 8: end procedure σ

Original: We test the efficacy of the baseline algorithms and 4 with three kinds of statistical models.
Revised: : S

Original: For simulation, we use
Revised: 3: procedure P-SAMPLER( N

Original: Google Colaboratory with the standard run-time. 1 Other experiment details are deferred to the appendix in section B.1.1. 1: Input: no. of training points- N s , domain of the k predictors and the sample size- D ∈ R k , no. of local training points- n s , local sigma for Gaussian samplingσ l 2: Output: S model parameter space 3: procedure P-SAMPLER( N s ) 4: n ←5: while n < N s do 6:
Revised: ) 4: n ←5: while n < N s do 6:

Original: Uniformly randomly pick a centroid β from D 7:
Revised: Uniformly randomly pick a centroid β from D 7:

Original: S l ← Sample n s datapoints from a Gaussian distribution N ( µ = β, σ = σ l ) 8:
Revised: S l ← Sample n s datapoints from a Gaussian distribution N ( µ = β, σ = σ l ) 8:

Original: S ← S ∪ S l 9: n ← n + 1 + n s 10: end while return parameter space S 11: end procedure 4.2. Dataset
Revised: S ← S ∪ S l 9: n ← n + 1 + n s 10: end while return parameter space S 11: end procedure details are included in Table 1.

Original: For sampling parameters we use P-SAMPLER (algorithm 5) to generate the parameter sample space and in conjunction with algorithm 3 to compute power.

Original: Both of these algorithms together make up the power computation black box.

Original: After collecting the dataset, we pre-process it using PCA, as explained below.

Original: P-SAMPLER is inspired by the fungible weights described in (Waller, 2008).

Original: Next, we provide exact details about the data we use for the methods introduced in this work.

Original: For both supervised and unsupervised tasks we use the same dataset albeit the unsupervised one does not contain the labels or true power information.

Original: The dataset details are included in Table 1.

Original: We assume that our original linear model is denoted as
Revised: We assume that our original linear model is denoted as

Original: [Equation]
Revised: [Equation]

Original: + x k β k + e wherein, e ∼ N (0 , 1) is the error term, k is the number of predictors, y is the model predictions, x = { x 1 , ..., x k } and β = { β 1 , ..., β k } is our linear model parameter.
Revised: wherein, e ∼ N (0 , 1) is the error term, k is the number of predictors, y is the model predictions, x = { x 1 , ..., x k } and β = { β 1 , ..., β k } is our linear model parameter.

Original: Our modified dataset includes X = { β, N, Nσ } .
Revised: Our modified dataset includes X = { β, N, Nσ } .

Original: After including the PCA induced features our final dataset is X PCA
Revised: After including the PCA induced features our final dataset is X PCA

Original: = { β, N, Nσ, PC } .
Revised: = { β, N, Nσ, PC } .

Original: 4.2.1. M ODELS
Revised: 4.2.1. Models

Original: Three models are considered for understanding how the power manifold is impacted by model complexity.
Revised: Three models are considered for understanding how the power manifold is impacted by model complexity.

Revised: x below

Original: x below signifies the number of predictors in the model.
Revised: signifies the number of predictors in the model

Original: • Linear Model (REG-x): We run an f-test to understand whether a simpler model works better than the full model.
Revised: • Linear Model (REG-x): We run an f-test to understand whether a simpler model works better than the full model.

Original: Feature distribution listed in Table 5.
Revised: Feature distribution listed in Table 5.

Original: • Logistic Regression (LOGIT-x
Revised: • Logistic Regression (LOGIT-x

Original: Here, we run a Wald’s test to understand how much a reduced model impacts the predictive power.
Revised: Here, we run a Wald’s test to understand how much a reduced model impacts the predictive power.

Original: This is the only non-linear model that we consider.
Revised: This is the only non-linear model that we consider.

Original: Feature distribution listed in Table 6.
Revised: Feature distribution listed in Table 6.

Original: • Repeated Measures ANOVA (RMANOVA-x): We run a within-subjects RMANOVA to measure the impact of different factors (two) on the subjects.
Revised: • Repeated Measures ANOVA (RMANOVA-x): We run a within-subjects RMANOVA to measure the impact of different factors (two) on the subjects.

Original: Feature distribution listed in Table 5.
Revised: Feature distribution listed in Table 5.

Original: Throughout the paper, we use the same hypothesis H O where we test whether the model parameters β 1 , β 3 are zero (in the form of a partial F-test or a Wald’s test).
Revised: Throughout the paper, we use the same hypothesis H O where we test whether the model parameters β 1 , β 3 are zero (in the form of a partial F-test or a Wald’s test).

Original: However, for testing the robustness of our transfer learning algorithm
Revised: However, for testing the robustness of our transfer learning algorithm

Original: P-TRANSFER , we use the alternative hypothesis H
Revised: P-TRANSFER , we use the alternative hypothesis H

Original: ′ O where we test whether the model parameters β 1 , β 7 , β 8 are zero or not for both an F-test and a Wald’s test.
Revised: ′ O where we test whether the model parameters β 1 , β 7 , β 8 are zero or not for both an F-test and a Wald’s test.

Original: [Table]
Revised: [Table]

Original: For the 3 -predictor network we have 9 total features,features for the 5 -predictor network, 23 for the 10 -predictor network and 43 for the 20 -predictor network.
Revised: For the 3 -predictor network we have 9 total features,features for the 5 -predictor network, 23 for the 10 -predictor network and 43 for the 20 -predictor network.

Original: 4.3. Metrics
Revised: 4.3. Metrics

Original: In our proposed algorithm, we solve two kinds of problemsbinary power classification and a regression problem with actual power values.
Revised: In our proposed algorithm, we solve two kinds of problemsbinary power classification and a regression problem with actual power values.

Original: 4.3.1. C LASSIFICATION
Revised: 4.3.1. Classification

Original: For binary classification, we normally simply use the he accuracy.
Revised: For binary classification, we normally simply use the he accuracy.

Original: However, since we might have unbalanced datasets, we instead report the F1 score over the test set.
Revised: However, since we might have unbalanced datasets, we instead report the F1 score over the test set.

Original: While using multiple binary classifiers we could still use the F1-score albeit in an one-vs-rest fashion.
Revised: While using multiple binary classifiers we could still use the F1-score albeit in an one-vs-rest fashion.

Original: For multi-class classification, we are first required to run the more restrictive binary classifier, compute its performance, and then for the second binary classifier we only compute its performance over the remaining data points.
Revised: For multi-class classification, we are first required to run the more restrictive binary classifier, compute its performance, and then for the second binary classifier we only compute its performance over the remaining data points.

Original: Thus, suppose we have two binary classifiers-
Revised: Thus, suppose we have two binary classifiers-

Original: C 1 ( power > 0 . 8 and power ≤ 0 . 8 ) and
Revised: C 1 ( power > 0 . 8 and power ≤ 0 . 8 ) and

Original: C 2 ( power > 0 .
Revised: C 2 ( power > 0 .

Original: 6 and power ≤ 0 . 6 ).
Revised: 6 and power ≤ 0 . 6 ).

Original: We first run C 1 to find all data points with power > 0 . 

Revised: We first run C 1 to find all data points with power > 0 .

Original: and then run C 2 to find data points with 0 .
Revised: and then run C 2 to find data points with 0 .

Original: 8 ≥ power > 0 .
Revised: 8 ≥ power > 0 .

Original: 6 and power ≤ 0 .
Revised: 6 and power ≤ 0 .

Original: Note that currently in the results we only provide separate performance metrics for C 1 and C 2 .
Revised: Note that currently in the results we only provide separate performance metrics for C 1 and C 2 .

Original: 4.3.2. R EGRESSION
Revised: 4.3.2. Regression

Original: We use a Mean Squared Error loss function for the regression problem between the ground truth power and the simulated power values.
Revised: We use a Mean Squared Error loss function for the regression problem between the ground truth power and the simulated power values.

Original: However, power is the probability of successfully rejecting a null hypothesis, and thus we can consider the ground truth power P g and the simulated power values P s as probability distributions.
Revised: However, power is the probability of successfully rejecting a null hypothesis, and thus we can consider the ground truth power P g and the simulated power values P s as probability distributions.

Original: Note that the
Revised: Note that the

Original: KL Divergence between A and B is shown in equation 1.
Revised: KL Divergence between A and B is shown in equation 1.

Original: [Equation]
Revised: [Equation]

Original: Here, x denotes the feature space associated with the probability distribution ( i.e., actual and simulated power), and
Revised: Here, x denotes the feature space associated with the probability distribution ( i.e., actual and simulated power), and

Original: A, B represents the ground truth and the simulated powers themselves.
Revised: A, B represents the ground truth and the simulated powers themselves.

Original: However, KL divergence is non-symmetric and not a distance measure.
Revised: However, KL divergence is non-symmetric and not a distance measure.

Original: Thus, we use the Jensen-Shannon (JS) divergence which is an extension of the equation.
Revised: Thus, we use the Jensen-Shannon (JS) divergence which is an extension of the equation.

Original: The JS divergence between two probability distributions A and B is given by the equation 2.
Revised: The JS divergence between two probability distributions A and B is given by the equation 2.

Original: [Equation]
Revised: [Equation]

Original: [Figure]

Original: 4.4. A note about current baselines for predicting power
Revised: 4.4. A note about current baselines for predicting power

Original: From literature, previous approaches for computing a power manifold used a brute-force approach by essentially running the black box for computing power.
Revised: From literature, previous approaches for computing a power manifold used a brute-force approach by essentially running the black box for computing power.

Original: Thus, given unlimited time, they could achieve 100% accuracy by just literally computing the power manifold.
Revised: Thus, given unlimited time, they could achieve 100% accuracy by just literally computing the power manifold.

Original: Specific greedy approaches attempted to find optimal power but not the entire manifold.
Revised: Specific greedy approaches attempted to find optimal power but not the entire manifold.

Original: Since our work aims to minimize the effort required to  compute the power manifold, we do not compare our work directly to these brute-force approaches.
Revised: Since our work aims to minimize the effort required to [Figure] compute the power manifold, we do not compare our work directly to these brute-force approaches.

Original: Instead, we elect to create our baselines using traditional predictive machine learning.
Revised: Instead, we elect to create our baselines using traditional predictive machine learning.

Original: 5. Results 5.1. P-Transfer
Revised: 5. Results 5.1. P-Transfer

Original: We evaluate P-Transfer by first pre-training a REG-20 model and then testing it over three different scenarios-(i)
Revised: We evaluate P-Transfer by first pre-training a REG-20 model and then testing it over three different scenarios-(i)

Original: Same feature distribution, Same Hypothesis, (ii) Different feature distribution, Same Hypothesis, and (iii) Different feature distribution, Different Hypothesis.
Revised: Same feature distribution, Same Hypothesis, (ii) Different feature distribution, Same Hypothesis, and (iii) Different feature distribution, Different Hypothesis.

Original: We denote these cases by TF 1 , TF 2 , TF 3 respectively.
Revised: We denote these cases by TF 1 , TF 2 , TF 3 respectively.

Original: Note that we always use a model smaller or equal in size to the original pre-trained model for transfer learning.
Revised: Note that we always use a model smaller or equal in size to the original pre-trained model for transfer learning.

Original: Our preliminary observations indicate that transfer learning never hurts model performance.
Revised: Our preliminary observations indicate that transfer learning never hurts model performance.

Original: However, for the cases where both the models have the same hypothesis ( i.e., cases (i) and (ii)), we see significant gains in the model performance.
Revised: However, for the cases where both the models have the same hypothesis ( i.e., cases (i) and (ii)), we see significant gains in the model performance.

Original: Table 2 summarizes these results.
Revised: Table 2 summarizes these results.

Original: [Figure] [Table] Thus, for comparison to PNN , we use our baselines - PLRand, P-CLUSTER, PL-PROP and the PK-neighbors approach.
Revised: Thus, for comparison to PNN , we use our baselines - PLRand, P-CLUSTER, PL-PROP and the PK-neighbors approach.

Original: Note that to accommodate the various trends in our model, we only report the best performing baseline (over all four baselines) in the plots.
Revised: Note that to accommodate the various trends in our model, we only report the best performing baseline (over all four baselines) in the plots.

Original: For brevity and due to many baseline performance combinations, we skip the point-wise results of the baselines.
Revised: For brevity and due to many baseline performance combinations, we skip the point-wise results of the baselines.

Original: However, the code provided hosts with all the information required to recreate these baselines.
Revised: However, the code provided hosts with all the information required to recreate these baselines.

Original: We capture three primary trends in our experiments, the performance over the change in model complexity, the number of predictors and the amount of training data used for predicting the power manifold.
Revised: We capture three primary trends in our experiments, the performance over the change in model complexity, the number of predictors, and the amount of training data used for predicting the power manifold.

Revised: [Table]

Original: The trend of increased performance with increased training data usage is reported via Figures 3, 4.
Revised: The trend of increased performance with increased training data usage is reported via Figures 3, 4.

Original: Interestingly, even at lower training data proportions, we receive good performance from the classifiers.
Revised: Interestingly, even at lower training data proportions, we receive good performance from the classifiers.

Original: From the exact figures, we can see that predictions struggle over the RMANOVA model while they are much easier for the REG model and moderate for the LOGIT model.
Revised: From the exact figures, we can see that predictions struggle over the RMANOVA model while they are much easier for the REG model and moderate for the LOGIT model.

Original: A similar trend can be observed for regression tasks as seen in the Figs.
Revised: A similar trend can be observed for regression tasks as seen in the Figs.

Original: 7, 8, 9 in the appendix.
Revised: 7, 8, 9 in the appendix.

Original: Furthermore, with a change in the number of predictors as seen in Figs. 10, 11 and 12, the model performance deteriorates for change in the number of predictors for REG and
Revised: Furthermore, with a change in the number of predictors as seen in Figs. 10, 11 and 12, the model performance deteriorates for change in the number of predictors for REG and

Original: LOGIT.
Revised: LOGIT.

Original: However, for the RMANOVA model, we see an arbitrary performance plot indicating our sub-optimal model choice might be the reason.
Revised: However, for the RMANOVA model, we see an arbitrary performance plot indicating our sub-optimal model choice might be the reason.

Original: Here sub-optimal indicates the model derived without using the validation set.
Revised: Here sub-optimal indicates the model derived without using the validation set.

Original: It seems that for both regression and classification tasks, PNN struggles the most for the RMANOVA model family.
Revised: It seems that for both regression and classification tasks, PNN struggles the most for the RMANOVA model family.

Original: However, given that we see this is only the models with low access to training data, we can say that these classifiers are biased.
Revised: However, given that we see this is only the models with low access to training data, we can say that these classifiers are biased.

Original: Finally, we can observe the number of calls v/s the performance in Fig. 15.
Revised: Finally, we can observe the number of calls v/s the performance in Fig. 15.

Original: Clearly, for multiple models using fewer data points can still provide high utility.
Revised: Clearly, for multiple models using fewer data points can still provide high utility.

Original: Even though we do not fully report the baselines, we would like to highlight a few notable observations.
Revised: Even though we do not fully report the baselines, we would like to highlight a few notable observations.

Original: As expected,
Revised: As expected,

Original: P-RAND performs the worst, followed by P-CLUSTER .
Revised: P-RAND performs the worst, followed by P-CLUSTER .

Original: Depending on the simulated data’s distribution P-CLUSTER surprisingly beats both PK-neighbors and PL-PROP when fewer training data points are used.
Revised: Depending on the simulated data’s distribution P-CLUSTER surprisingly beats both PK-neighbors and PL-PROP when fewer training data points are used.

Original: Amongst, PL-PROP and
Revised: Amongst, PL-PROP and

Original: PK-neighbors , the PL-PROP approach is the more robust one and it is the closest one to PNN . 
Revised: PK-neighbors , the PL-PROP approach is the more robust one and it is the closest one to PNN . [Figure]

Original: 6. Limitations and Future Work
Revised: 6. Limitations and Future Work

Original: This work primarily investigates the utility of training a neural network for predicting statistical power over a manifold.
Revised: This work primarily investigates the utility of training a neural network for predicting statistical power over a manifold.

Original: Furthermore, we also provide baselines-
Revised: Furthermore, we also provide baselines-

Original: Random (PRand), K Neighbors Classifier (PK-Neighbors), Label Propagation (PL-Prop), and an unsupervised clustering method (P-CLUSTER).
Revised: Random (PRand), K Neighbors Classifier ( PK-Neighbors), Label Propagation (PL-Prop), and an unsupervised clustering method (P-CLUSTER).

Original: Note that each of the baselines fails to generalize for complex models.
Revised: Note that each of the baselines fails to generalize for complex models.

Original: • P-RAND :
Revised: • P-RAND :

Original: The random mechanism works as advertised and provides the least performance throughout.
Revised: The random mechanism works as advertised and provides the least performance throughout.

Original: • P-CLUSTER :
Revised: • P-CLUSTER :

Original: This approach can be compelling for visualizing the power manifold in 2 dimensions space effectively.
Revised: This approach can be compelling for visualizing the power manifold in 2 dimensions space effectively.

Original: However, we note that its performance is inconsistent, and the linear classifier might not always be able to predict exactly between high and low power regions.
Revised: However, we note that its performance is inconsistent, and the linear classifier might not always be able to predict exactly between high and low power regions.

Original: Further, the lack of label information makes this approach useful for pre-processing but not for predicting power classes.
Revised: Further, the lack of label information makes this approach useful for pre-processing but not for predicting power classes.

Original: Alternatively, we might be able to identify and eliminate low powered regions using this approach, and we again propose this as an open problem.
Revised: Alternatively, we might be able to identify and eliminate low powered regions using this approach, and we again propose this as an open problem.

Original: • PK-neighbors : This approach is a supervised clustering alternative to P-CLUSTER .
Revised: • PK-neighbors : This approach is a supervised clustering alternative to P-CLUSTER .

Original: However, it does not necessarily perform consistently.
Revised: However, it does not necessarily perform consistently.

Original: We, however, note[Figure] that we have not fine-tuned this approach for the number of neighbors.
Revised: We, however, note that we have not fine-tuned this approach for the number of neighbors.

Original: We leave this investigation as part of our future work.
Revised: We leave this investigation as part of our future work.

Original: • PL-PROP : The most promising baseline is PL-PROP with its rbf kernel.
Revised: • PL-PROP : The most promising baseline is PL-PROP with its rbf kernel.

Original: PL-PROP is better than the rest but fails to generalize for complex models or more predictors.
Revised: PL-PROP is better than the rest but fails to generalize for complex models or more predictors.

Original: Furthermore, it requires much more data compared to the neural network.
Revised: Furthermore, it requires much more data compared to the neural network.

Original: We, however, have not used any guidance for tuning this algorithm’s parameters, and we leave this aspect for future work.
Revised: We, however, have not used any guidance for tuning this algorithm’s parameters, and we leave this aspect for future work.

Original: • Model Limitations: PNN provides excellent empirical performance and generalizes well over increasing model complexity while providing competitive performance even with only 10% of the data.
Revised: • Model Limitations: PNN provides excellent empirical performance and generalizes well over increasing model complexity while providing competitive performance even with only 10% of the data.

Original: We note that PNN is not yet trained using methods like early stopping or an adaptive learning rate .
Revised: We note that PNN is not yet trained using methods like early stopping or an adaptive learning rate .

Original: We would pursue these optimizations in subsequent work.
Revised: We would pursue these optimizations in subsequent work.

Original: Particularly we do not use a validation dataset to improve the model generalization, and we set 500 epochs as an arbitrary stopping point.
Revised: Particularly we do not use a validation dataset to improve the model generalization, and we set 500 epochs as an arbitrary stopping point.

Original: We have seen at least one instance of such an approach leading to a sub-optimal model choice, thus leaving a gap in the empirical performance.
Revised: We have seen at least one instance of such an approach leading to a sub-optimal model choice, thus leaving a gap in the empirical performance.

Original: • Dataset Limitations: Our most complex model currently is the RMANOVA with two within-subjects factors.
Revised: • Dataset Limitations: Our most complex model currently is the RMANOVA with two within-subjects factors.

Original: To further understand the generalization error, we hope to extend our approach to higher complexity models, including other non-linear models.
Revised: To further understand the generalization error, we hope to extend our approach to higher complexity models, including other non-linear models.

Original: Further, we would like to explore multi-class classifications to provide the user with more options in terms of choosing the power manifold.
Revised: Further, we would like to explore multi-class classifications to provide the user with more options in terms of choosing the power manifold.

Original: • Theoretical Limitation: Finally, the current implementations do not provide any formal theoretical guarantees for the convergence of any of these methods.
Revised: • Theoretical Limitation: Finally, the current implementations do not provide any formal theoretical guarantees for the convergence of any of these methods.

Original: We do provide intuition about why our transfer learning methods work.
Revised: We do provide intuition about why our transfer learning methods work.

Original: However, we would like to explore how different distributions with the same hypothesis impact the power manifold in greater detail.
Revised: However, we would like to explore how different distributions with the same hypothesis impact the power manifold in greater detail.

Original: 7. Conclusion
Revised: 7. Conclusion

Original: In this work, we show that PCA-derived features are beneficial for exploring the high-dimensional manifolds of the power surface.
Revised: In this work, we show that PCA-derived features are beneficial for exploring the high-dimensional manifolds of the power surface.

Original: We provide multiple algorithms that provide reasonable alternatives to the original brute force method of a power analysis.
Revised: We provide multiple algorithms that provide reasonable alternatives to the original brute force method of a power analysis.

Original: We show that using a simple, fully connected neural network, we can generalize across complex and even non-linear models to consistently predict power.
Revised: We show that using a simple, fully connected neural network, we can generalize across complex and even non-linear models to consistently predict power.

Original: We showcase that even using 10% training data can lead to high prediction accuracy for regression and classification tasks.
Revised: We showcase that even using 10% training data can lead to high prediction accuracy for regression and classification tasks.

Original: Finally, with transfer learning, we can learn a single model and boost the performance of other models again with only 10% training data.
Revised: Finally, with transfer learning, we can learn a single model and boost the performance of other models again with only 10% training data.

Original: A. Appendix B. PNN Performance
Revised: A. Appendix B. PNN Performance

Original: We demonstrate additional results about the model’s performance across changes in the number of predictors, for different classification boundaries (0.6 instead of 0.8), and the performance for regression tasks.
Revised: We demonstrate additional results about the model’s performance across changes in the number of predictors, for different classification boundaries (0.6 instead of 0.8), and the performance for regression tasks.

Original: We first consider changes in the classification boundary from 0.8 to 0.6 to observe models with lower power as potentially interesting.
Revised: We first consider changes in the classification boundary from 0.8 to 0.6 to observe models with lower power as potentially interesting.

Original: We observe from Figs.
Revised: We observe from Figs.

Original: 5 and 6.
Revised: 5 and 6.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: For all regression tasks in Figs. 7, 8, 9 (even over the change in the classification boundary from 0.8 to 0.6) we can observe that the JS divergence over the test set is always better than the random baseline by at least 10x and going up to even 200x performance. 
Revised: For all regression tasks in Figs. 7, 8, 9 (even over the change in the classification boundary from 0.8 to 0.6) we can observe that the JS divergence over the test set is always better than the random baseline by at least 10x and going up to even 200x performance. [Figure]

Original: The performance over the change in the number of predictors as seen in Figs.
Revised: The performance over the change in the number of predictors as seen in Figs.

Original: 10, 11 and 12 shows a downward trend for the
Revised: 10, 11 and 12 shows a downward trend for the

Original: REG and LOGIT models and so the classifier consistently provides poor performance with an increase in model complexity due to the number of predictors.
Revised: REG and LOGIT models and so the classifier consistently provides poor performance with an increase in model complexity due to the number of predictors.

Original: This can be explained by the massive increase in the dimensionality of the space.
Revised: This can be explained by the massive increase in the dimensionality of the space.

Original: However,[Figure] for RMANOVA we do not see such a trend at all, and rather the models can have higher performance for even the complex models.
Revised: However, for RMANOVA we do not see such a trend at all, and rather the models can have higher performance for even the complex models.

Original: We believe this is a current limitation due to sub-optimal model choice (not driven by the validation set).
Revised: We believe this is a current limitation due to sub-optimal model choice (not driven by the validation set)

Original: [Figure]
Revised: [Figure]

Original: B.1. Parameter Tuning
Revised: B.1. Parameter Tuning

Original: Tuning the algorithms used in this work is critical to improving the performance of the Power Network and Power Cluster.
Revised: Tuning the algorithms used in this work is critical to improving the performance of the Power Network and Power Cluster.

Original: It turns out that choosing the variance captured by PCA directly affects the Power Network.
Revised: It turns out that choosing the variance captured by PCA directly affects the Power Network.

Original: Moreover, PCA variance and the number of clusters selected in Power Clustering can improve its performance.
Revised: Moreover, PCA variance and the number of clusters selected in Power Clustering can improve its performance [Figure] [Figure]

Original: B.1.1. E XPERIMENT D ETAILS
Revised: B.1.1. Experiment Details

Original: We have standardized the parameter distribution for data collection to provide a consistent comparison.
Revised: We have standardized the parameter distribution for data collection to provide a consistent comparison.

Original: These parameters can be found in Table 3.
Revised: These parameters can be found in Table 3.

Original: For tuning the neural network’s learning rate, we use the library Optuna ((Akiba et al., 2019)).[Figure] [Figure]
Revised: For tuning the neural network’s learning rate, we use the library Optuna ((Akiba et al., 2019)).

Original: Experimentation Details

Original: Values

Original: No. of power simulations for each record

Original: [Figure]

Original: Total Datapoints per model

Original: [Figure]

Original: No. of training epochs

Original: [Figure]

Original: Training Data Split 10, 20, 40, 60, 80 (%)

Original: Power Classification Boundary 0.

Original: [Figure]

Original: Optimizer

Original: Adam

Original: Model Performance confidence interval 95%

Original: [Figure]

Original: [Figure]

Original: B.1.2. PCA
Revised: B.1.2. PCA

Original: Usually, increasing the sample size improves power.
Revised: Usually, increasing the sample size improves power.

Original: Furthermore, our feature engineered dimension denoted by Nσ i.e., (scaled weight) also affects the power significantly.
Revised: Furthermore, our feature engineered dimension denoted by Nσ i e., (scaled weight) also affects the power significantly.

Original: We can observe the correlation values concerning power in Fig. 13.
Revised: We can observe the correlation values concerning power in Fig. 13.

Original: Thus, we expect the variance covered in PCA transformed data to require at least two dimensions. 
Revised: Thus we expect the variance covered in PCA transformed data to require at least two dimensions. [Figure] [Figure] [Figure]

Original: We consistently observe excellent performance after selecting 99% variance from PCA.
Revised: We consistently observe excellent performance after selecting 99% variance from PCA.

Original: Furthermore, we concatenate these new features to the original matrix and use this to train the Neural Network.
Revised: Furthermore, we concatenate these new features to the original matrix and use this to train the Neural Network.

Original: As we see from figure 13, the impact of the
Revised: As we see from figure [Figure] the impact of the

Original: PCA features is sizeable.
Revised: [Figure]

Original: B.1.3. C HOICE OF CLUSTERS IN P OWER C LUSTER
Revised: B.1.3. Choice of clusters in Power Cluster

Original: Depending on how the data is clustered, the Choice of several clusters can directly impact the classification performance.
Revised: Depending on how the data is clustered, the Choice of several clusters can directly impact the classification performance.

Original: However, empirically this method is unreliable and requires a visual inspection of the dataset to be helpful.[Figure]
Revised: However, empirically this method is unreliable and requires a visual inspection of the dataset to be helpful.

Original: B.1.4. O THER C ONSIDERATIONS
Revised: B.1.4. Other Considerations

Original: We employ a few techniques during neural network training with increasingly complex models and parameter spaces.
Revised: We employ a few techniques during neural network training with increasingly complex models and parameter spaces.

Original: The training is sensitive to the choice of the learning rate.
Revised: The training is sensitive to the choice of the learning rate.

Original: Tuning needs to be achieved after using a hyperparameter tuner like Optuna (Akiba et al., 2019).
Revised: Tuning needs to be achieved after using a hyperparameter tuner like Optuna (Akiba et al., 2019).

Original: (2) Training tends to fluctuate a lot, and a higher number of epochs and early stopping seem to help.
Revised: (2) Training tends to fluctuate a lot, and a higher number of epochs and early stopping seem to help.

Original: B.1.5. N EURAL N ETWORK A RCHITECTURE
Revised: B.1.5. Neural Network Architecture

Original: Layer
Revised: Layer

Original: Parameters
Revised: Parameters

Original: [Table] 32 units + ReLUDense (
Revised: Dense (input) 64 units + ReLU

Original: [Table]

Original: ) 1 unit + Sigmoid

Original: B.1.6. L IST OF M ODEL P ARAMETERS

Original: D X

Original: D X D X D [Table]
Revised: Dense (hidden) 32 units + ReLU

Original: D X

Original: CAT[-1, 1]

Original: N (0 , 1)

Original: X 1 × X

Original: N (0 , 2)

Original: X 4 × X

Original: D X

Original: D X

Original: D X

Original: D X

Original: D X

Original: CAT[0, 1, 2]

Original: N (0 , 2)

Original: X 6 × X N (0 , 1) X 2 × X D X
Revised: Dense (output) 1 unit + Sigmoid

Original: D X

Original: D X

Original: D X

Original: D X 15 N (0 , 3)

Original: N (0 , 1)

Original: X 1 × X

Original: N (0 , 2)

Original: X 11 × X

Original: D X

Original: D X

Original: D X

Original: D X

Original: D X 20 N (0 , 2)

Original: N (0 , 2) X 11 × X N (0 , 1) X 6 × X
Revised: B.1.6. List of Model Parameters

Original: D

Original: [Table]
Revised: [Table]

Original: D X

Original: D

Original: [Table]

Original: D

Original: [Table]

Original: D

Original: [Table]

Original: N (0 , 1)

Original: CAT[-1, 1]

Original: N (0 , 1)

Original: [Table]
Revised: [Table]

Original: C. Computational Costs C.1. Details about Principal Component Analysis
Revised: C. Computational Costs C.1. Details about Principal Component Analysis

Original: PCA works by finding the direction with maximal variance.
Revised: PCA works by finding the direction with maximal variance.

Original: Next, it finds the direction with the maximal variance such that this direction is uncorrelated with the previous direction.
Revised: Next, it finds the direction with the maximal variance such that this direction is uncorrelated with the previous direction.

Original: We continue in this fashion so that any pair of directions are uncorrelated.
Revised: We continue in this fashion so that any pair of directions are uncorrelated.

Original: We then reorient our dataset with these new components to compute our new dataset.
Revised: We then reorient our dataset with these new components to compute our new dataset.

Original: Suppose our original dataset is X ∈ R N × p and v ∈ R p × k are the derived Principal Components (PCs) where N is the total samples in X , p = dim( X ) and k is the number of PCs to be selected.
Revised: Suppose our original dataset is X ∈ R N × p and v ∈ R p × k are the derived Principal Components (PCs) where N is the total samples in X , p = dim( X ) and k is the number of PCs to be selected.

Original: Then the new dataset is given by,
Revised: Then the new dataset is given by,

Original: [Equation]
Revised: [Equation]