Original: AutoML Adoption in ML Software
Revised: AutoML Adoption in ML Software

Original: Abstract
Revised: Abstract

Original: Machine learning (ML) has become essential to a vast range of applications, while ML experts are in short supply.
Revised: Machine learning (ML) has become essential to a vast range of applications, while ML experts are in short supply.

Original: To alleviate this problem, AutoML aims to make ML easier and more eﬃcient to use.
Revised: To alleviate this problem, AutoML aims to make ML easier and more eﬃcient to use.

Original: Even so, it is not clear to which extent AutoML techniques are actually adopted in an engineering context, nor what facilitates or inhibits adoption.
Revised: Even so, it is not clear to which extent AutoML techniques are actually adopted in an engineering context, nor what facilitates or inhibits adoption.

Original: To study this, we deﬁne AutoML engineering practices, measure their adoption through surveys, and distil ﬁrst insights into factors inﬂuencing adoption from two initial interviews.
Revised: To study this, we deﬁne AutoML engineering practices, measure their adoption through surveys, and distil ﬁrst insights into factors inﬂuencing adoption from two initial interviews.

Original: Depending on the practice, results show around 20 to 30% of the respondents have not adopted it at all and many more only partially, leaving substantial room for increases in adoption.
Revised: Depending on the practice, results show around 20 to 30% of the respondents have not adopted it at all and many more only partially, leaving substantial room for increases in adoption.

Original: The interviews indicate adoption may in part be inhibited by usability issues with AutoML frameworks and the increased computational resources needed for adoption.
Revised: The interviews indicate adoption may in part be inhibited by usability issues with AutoML frameworks and the increased computational resources needed for adoption.

Original: 1. Introduction
Revised: 1. Introduction

Original: Recent advances in machine learning (ML) have resulted in increasingly widespread use of
Revised: Recent advances in machine learning (ML) have resulted in increasingly widespread use of

Original: ML methods in a wide range of applications.
Revised: ML methods in a wide range of applications.

Original: AutoML promises to decrease development time to enable adoption of ML on a larger scale and also to empower developers less experienced with ML.
Revised: AutoML promises to decrease development time to enable adoption of ML on a larger scale and also to empower developers less experienced with ML.

Original: By automating key stages in creating ML tools and pipelines, AutoML trades human time and experience for compute time.
Revised: By automating key stages in creating ML tools and pipelines, AutoML trades human time and experience for compute time.

Original: However, it is not yet clear to which extent
Revised: However, it is not yet clear to which extent

Original: developers adopt AutoML techniques, and what factors inﬂuence their decisions.
Revised: developers adopt AutoML techniques and what factors inﬂuence their decisions.

Original: To measure adoption we formulateAutoML practices based on a selection of AutoML techniques.
Revised: To measure adoption we formulate practices based on a selection of AutoML techniques.

Original: For the three AutoML practices we consider, the results from two surveys with a combined total of over 360 developers are presented.
Revised: For the three AutoML practices we consider, the results from two surveys with a combined total of over 360 development teams are presented.

Original: Moreover, we present preliminary results from interviewing ML practitioners, which add depth to our survey results.
Revised: Moreover, we present preliminary results from interviewing ML practitioners, which add depth to our survey results.

Original: We investigate the adoption of AutoML practices in the broader context of software engineering for ML (SE for ML), where AutoML practices automate diﬀerent stages of the software development life cycle for ML.
Revised: We investigate the adoption of AutoML practices in the broader context of software engineering for ML (SE for ML), where AutoML practices automate diﬀerent stages of the software development life cycle for ML.

Original: Automation is known to bring multiple beneﬁts in traditional software development.
Revised: Automation is known to bring multiple beneﬁts in traditional software development.

Original: However, its impact on the development of ML systems is still unknown.
Revised: However, its impact on the development of ML systems is still unknown.

Original: Along with other results, we previously collected data about automatic hyperparameter optimisation from over 300 developers (Serban et al., 2020).
Revised: Along with other results, we previously collected data about automatic hyperparameter optimisation from over 300 developers (Serban et al., 2020).

Original: With an extended survey, additionally covering trustworthy ML, and also including additional practices on automatic feature generation and model conﬁguration, we gathered more than
Revised: With an extended survey, additionally covering trustworthy ML, and also including additional practices on automatic feature generation and model conﬁguration, we gathered more than

Original: 60 responses so far (Serban et al., 2021).
Revised: 60 responses so far (Serban et al., 2021).

Original: Here, we analyse the AutoML data from these quantitative studies to obtain a ﬁne-grained view of AutoML practices and initiate qualitative interviews with practitioners to situate the survey results.
Revised: Here, we analyse the AutoML data from these quantitative studies to obtain a ﬁne-grained view of AutoML practices and initiate qualitative interviews with practitioners to situate the survey results.

Original: Overall, we observe the adoption of AutoML practices is relatively low, and the preliminary interviews suggest this may in part be motivated by usability issues with AutoML frameworks and the increase in computational resources required by AutoML approaches.
Revised: Overall, we observe the adoption of AutoML practices is relatively low, and the preliminary interviews suggest this may in part be motivated by usability issues with AutoML frameworks and the increase in computational resources required by AutoML approaches.

Original: To position this study, Section 2 covers related work.
Revised: To position this study, Section 2 covers related work.

Original: Following that, in Section 3, we describe the AutoML practices considered in our work, and discuss the survey design as well as our interview setup.
Revised: Next, Section 3 describes the AutoML practices considered in our work, and discusses the survey design and interview setup.

Original: Section 4then analysesboth the AutoML results from Serban et al.
Revised: Section 4 analyses the AutoML results from Serban et al.

Original: (2020) as well as from the extended survey from Serban et al.
Revised: (2020) and from the extended survey from Serban et al.

Original: (2021), and relates them to observations from the interviews.
Revised: (2021), and relates them to observations from the interviews.

Original: Lastly, in Section 5, we brieﬂy summarise our main ﬁndings and discuss directions for future work.
Revised: Lastly, in Section 5, we brieﬂy summarise our main ﬁndings and discuss directions for future work.

Original: 2. Related Work
Revised: 2. Related Work

Original: Best practices have been deﬁned for research and for benchmarking of AutoML techniques and frameworks.
Revised: Best practices have been deﬁned for research and for benchmarking of AutoML techniques and frameworks.

Original: For instance, Eggensperger et al.
Revised: For instance, Eggensperger et al.

Original: (2019) deﬁne best practices for the use of automatic algorithm conﬁguration.
Revised: (2019) deﬁne best practices for the use of automatic algorithm conﬁguration.

Original: Similar work exists for other techniques, such as neural architecture search (Lindauer and Hutter, 2020).
Revised: Similar work exists for other techniques, such as neural architecture search (Lindauer and Hutter, 2020).

Original: Best practices to compare AutoML frameworks have also been proposed, e.g. by Gijsbers et al.
Revised: Best practices to compare AutoML frameworks have also been proposed, e.g. by Gijsbers et al.

Original: combined with a set of benchmark problems.
Revised: combined with a set of benchmark problems.

Original: However, none of this prior work considers the practical application of AutoML, or AutoML practices used by developers of applications with ML components.
Revised: However, none of this prior work considers the practical application of AutoML, or AutoML practices used by developers of applications with ML components.

Original: In the ﬁeld of human computer interaction, there is recent work with a focus on the adoption and accessibility of AutoML.
Revised: In the ﬁeld of human computer interaction, there is recent work with a focus on the adoption and accessibility of AutoML.

Original: Wang et al.
Revised: Wang et al.

Original: (2019) postulate that users may not adopt AutoML, because they distrust AutoML systems – for instance, because they do not know whether the system was run suﬃciently long.
Revised: (2019) postulate that users may not adopt AutoML, because they distrust AutoML systems – for instance, because they do not know whether the system was run suﬃciently long.

Original: To tackle this problem, they introduce a system to visualise and analyse results to help the user understand what the system does, and to guide it.
Revised: To tackle this problem, they introduce a system to visualise and analyse results to help the user understand what the system does, and to guide it.

Original: Lee et al.
Revised: Lee et al.

Original: (2019) suggest that AutoML adoption may be low due to usability issues rather than a lack of awareness, but do not indicate any evidence for this.
Revised: (2019) suggest that AutoML adoption may be low due to usability issues rather than a lack of awareness, but do not indicate any evidence for this.

Original: They propose a mixed-initiative ML framework with the aim of improving the usability which should, among other things, result in a more transparent framework that users are more likely to trust.
Revised: They propose a mixed-initiative ML framework with the aim of improving the usability which should, among other things, result in a more transparent framework that users are more likely to trust.

Original: Drozdal et al.
Revised: Drozdal et al.

Original: (2020) investigate which features are important to gain the trust of AutoML users, and found that visualising the process and providing performance metrics are the most important.
Revised: (2020) investigate which features are important to gain the trust of AutoML users, and found that visualising the process and providing performance metrics are the most important.

Original: While these works aim to improve usability or trustworthiness in order to facilitate increased adoption, they do not actually establish that adoption is low, or that these are the primary factors limiting adoption.
Revised: While these works aim to improve usability or trustworthiness in order to facilitate increased adoption, they do not actually establish that adoption is low, or that these are the primary factors limiting adoption.

Original: In addition to the usability of AutoML frameworks themselves, there is also work on making AutoML frameworks more usable by simplifying the interaction with them, or to extend the automation to earlier or later steps than handled by the current frameworks.
Revised: In addition to the usability of AutoML frameworks themselves, there is also work on making AutoML frameworks more usable by simplifying the interaction with them, or to extend the automation to earlier or later steps than handled by the current frameworks.

Original: For instance, Cambronero et al.
Revised: For instance, Cambronero et al.

Original: (2020) introduce a tool to help deﬁne the search space for AutoML frameworks.
Revised: (2020) introduce a tool to help deﬁne the search space for AutoML frameworks.

Original: With another tool, Narkar et al.
Revised: With another tool, Narkar et al.

Original: (2021) aim to help users with the analysis of results and to let them choose a ﬁnal model based on more than just a performance metric optimised by the AutoML framework.
Revised: (2021) aim to help users with the analysis of results and to let them choose a ﬁnal model based on more than just a performance metric optimised by the AutoML framework.

Original: As far as actual AutoML adoption goes, Lee et al.
Revised: As far as actual AutoML adoption goes, Lee et al.

Original: (2019) observe, based on preliminary results, that fewer than 2% of the workﬂows available on OpenML (Vanschoren et al., 2013) adopt AutoML.
Revised: (2019) observe, based on preliminary results, that fewer than 2% of the workﬂows available on OpenML (Vanschoren et al., 2013) adopt AutoML.

Original: However, since OpenML is primarily a research platform, this may not be a very good indicator for adoption in a wider context; in addition, it gives no detailed view of adoption of the various AutoML stages.
Revised: However, since OpenML is primarily a research platform, this may not be a very good indicator for adoption in a wider context; in addition, it gives no detailed view of adoption of the various AutoML stages.

Original: Xin et al.
Revised: Xin et al.

Original: (2021) study for which parts of the ML process AutoML adopters apply the automated methods and which they prefer to do manually.
Revised: (2021) study for which parts of the ML process AutoML adopters apply the automated methods and which they prefer to do manually.

Original: Their ﬁndings also cover the beneﬁts and shortcomings the users see in the current AutoML frameworks.
Revised: Their ﬁndings also cover the beneﬁts and shortcomings the users see in the current AutoML frameworks.

Original: For the steps where AutoML is used most (hyperparameter tuning, model- and feature selection), however, they go into the least detail, merely stating that these are the most used; insights into what drives or holds back adoption of these steps are not reported.
Revised: For the steps where AutoML is used most (hyperparameter tuning, model- and feature selection), however, they go into the least detail, merely stating that these are the most used; insights into what drives or holds back adoption of these steps are not reported.

Original: In a similar study Crisan and Fiore-Gartland (2021) look more closely at the use and adoption of visualisation techniques for AutoML, and how desirable users actually ﬁnd them.
Revised: In a similar study Crisan and Fiore-Gartland (2021) look more closely at the use and adoption of visualisation techniques for AutoML, and how desirable users actually ﬁnd them.

Original: Both Xin et al.
Revised: Both Xin et al.

Original: and Crisan and Fiore-Gartland (2021) gather qualitative data from interviews and suggest that human experts (currently) demand a place in the loop because AutoML does not suﬃciently address all their needs.
Revised: and Crisan and Fiore-Gartland (2021) gather qualitative data from interviews and suggest that human experts (currently) demand a place in the loop because AutoML does not suﬃciently address all their needs.

Original: 3. AutoML Practices Practice
Revised: 3. AutoML Practices Practice

Original: mining.
Revised: mining.

Original: To extract the practices, we followed a similar process to Serban et al.
Revised: To extract the practices, we followed a similar process to Serban et al.

Original: (2020), resulting in three AutoML practices, which overlap with the pre-processing and modelling tasks from Xin et al.
Revised: (2020), resulting in three AutoML practices, which overlap with the pre-processing and modelling tasks from Xin et al.

Original: (2021) that are well established in academia.
Revised: (2021) that are well established in academia.

Original: These practices were derived from grey literature as follows.
Revised: These practices were derived from grey literature as follows.

Original: Dean (2019), Tunguz (2020) and ZelrosAI (2019) all consider various AutoML stages, which can be categorised as revolving around (a) feature engineering and selection, (b) hyperparameter optimisation (HPO) and model or algorithm selection, and (c) the conﬁguration of algorithms or model structures and architectures.
Revised: Dean (2019), Tunguz (2020) and ZelrosAI (2019) all consider various AutoML stages, which can be categorised as revolving around (a) feature engineering and selection, (b) hyperparameter optimisation (HPO) and model or algorithm selection, and (c) the conﬁguration of algorithms or model structures and architectures.

Original: This categorisation resulted in the following practices:
Revised: This categorisation resulted in the following practices:

Original: Automate feature generation or selection.
Revised: Automate feature generation or selection.

Original: Automate model selection and hyperparameter optimisation.
Revised: Automate model selection and hyperparameter optimisation.

Original: Automate conﬁguration of algorithms or model structure (e.g., NAS).
Revised: Automate conﬁguration of algorithms or model structure (e g., NAS).

Original: Practice adoption
Revised: Practice adoption.

Original: To measure the adoption of these AutoML practices, we ran two surveys in the general context of software engineering for ML, i.e., the surveys contained both questions about software engineering for ML and AutoML.
Revised: To measure the adoption of these AutoML practices, we ran two surveys in the general context of software engineering for ML, i.e., the surveys contained both questions about software engineering for ML and AutoML.

Original: The questionnaire was designed following the recommendations from Kitchenham and Pﬂeeger (2002) as a crosssectional observational study, asking participants at the moment of ﬁlling the questionnaire if they adopted the practices.
Revised: The questionnaire was designed following the recommendations from Kitchenham and Pﬂeeger (2002) as a cross-sectional observational study, asking participants at the moment of ﬁlling the questionnaire if they adopted the practices.

Original: Several preliminary questions provided the basis for assigning participants to pre-deﬁned groups.
Revised: Several preliminary questions provided the basis for assigning participants to pre-deﬁned groups.

Revised: Participants were instructed to answer from the perspective of their team, rather than based on their individual practices.

Original: The questions  used standard answers on a Likert scale with four possible answers reﬂecting degrees of adoption rather than levels of agreement.
Revised: The questions (see Appendix A) used standard answers on a Likert scale with four possible answers reﬂecting degrees of adoption rather than levels of agreement.

Original: This allowed the practices to be expressed impartially and avoided the null-point bias strategy.
Revised: This allowed the practices to be expressed impartially and avoided the null-point bias strategy.

Original: The four possible answers were “Not at all”, “Partially”, “Mostly” and “Completely”.
Revised: The four possible answers were “Not at all”, “Partially”, “Mostly” and “Completely”.

Original: The ﬁrst survey covered only the second AutoML practice mentioned above, while the second survey was speciﬁcally designed to cover all practices.
Revised: The ﬁrst survey covered only the second AutoML practice mentioned above, while the second survey was speciﬁcally designed to cover all practices.

Original: For information about other software engineering practices related to machine learning, we refer the reader to  Serban et al.  and  Serban et al.
Revised: Details on how the survey was distributed and a demographic analysis to identify possible biases can be found in Serban et al. Other SE for ML practices are also available there and in Serban et al.

Original: Interviews.
Revised: Interviews.

Original: To complement our survey results, but also to validate that the practices we identiﬁed are indeed relevant for practitioners and developers of AutoML frameworks, we conducted interviews.
Revised: To complement our survey results, but also to validate that the practices we identiﬁed are indeed relevant for practitioners and developers of AutoML frameworks, we conducted interviews.

Original: The interview protocol was designed using the guidelines from Hove and Anda (2005) and consisted of 15 questions designed to support a natural conversation.
Revised: The interview protocol was designed using the guidelines from Hove and Anda (2005) and consisted of 15 questions designed to support a natural conversation.

Original: The interviews were structured as follows.
Revised: The interviews were structured as follows.

Original: First, we described the goals and background of our research.
Revised: First, we described the goals and background of our research.

Original: Next, we asked participants to share information about their background and demographics.
Revised: Next, we asked participants to share information about their background and demographics.

Original: We then asked participants about the way they use AutoML and about the challenges faced by them in adopting AutoML, the perceived beneﬁts and the risks associated with the use of AutoML tools and techniques.
Revised: We then asked participants about the way they use AutoML and about the challenges faced by them in adopting AutoML, the perceived beneﬁts and the risks associated with the use of AutoML tools and techniques.

Original: Finally, we asked participants open-ended questions designed to elicit additional thoughts and feedback from them.
Revised: Finally, we asked participants open-ended questions designed to elicit additional thoughts and feedback from them.

Original: We recruited participants using purposeful sampling (Palinkas et al., 2015), by contacting practitioners which we knew were using or interested in using AutoML techniques.
Revised: We recruited participants using purposeful sampling (Palinkas et al., 2015), by contacting practitioners which we knew were using or interested in using AutoML techniques.

Original: We present results from two interviews.
Revised: We present results from two interviews.

Original: Participant P1 works on applications with ML components at a European technology company, while participant P2 works on AutoML at a
Revised: Participant P1 works on applications with ML components at a European technology company, while participant P2 works on AutoML at a

Original: European university (not our own).
Revised: European university (not our own).

Original: Both have ﬁve or more years of experience with ML.
Revised: Both have ﬁve or more years of experience with ML.

Original: 4. Results
Revised: 4. Results

Original: Since the two surveys were open during separate time frames, and adoption may have changed, we also analyse their results separately.
Revised: Since the two surveys were open during separate time frames, and adoption may have changed, we also analyse their results separately.

Original: Furthermore, we discuss the interviews in relation to the survey results whenever there is a clear connection, followed by further observations from the interviews.
Revised: Furthermore, we discuss the interviews in relation to the survey results whenever there is a clear connection, followed by further observations from the interviews.

Original: Additional survey results are available in the appendices.
Revised: Additional survey results are available in the appendices.

Original: Survey 1.
Revised: Survey 1.

Original: The initial survey resulted in a total of 307 valid responses 2 to the question on model selection and hyperparameter optimisation.
Revised: The initial survey resulted in a total of 307 valid responses 2 to the question on model selection and hyperparameter optimisation.

Original: Out of these respondents, 8% reported to have completely adopted these AutoML techniques, 26% mostly, 35% partially, and 31% not at all (see Table 1 in Appendix A).
Revised: Out of these respondents, 8% reported to have completely adopted these AutoML techniques, 26% mostly, 35% partially, and 31% not at all (see Table 1 in Appendix B).

Original: Evidently, although these techniques appear to be widely known, there is still signiﬁcant room to increase adoption.
Revised: Evidently, although these techniques appear to be widely known, there is still signiﬁcant room to increase adoption.

Original: Adoption levels between geographic regions (Figure 3a, Appendix B) are quite similar, with only South America showing higher rates of adoption, which are likely the result of the comparatively small size of this particular sample (6% of the respondents).
Revised: Adoption levels between geographic regions (Figure 3a, Appendix C) are quite similar, with only South America showing higher rates of adoption, which are likely the result of the comparatively small size of this particular sample (6% of the respondents).

Original: Figure 1a breaks down adoption by organisation type.
Revised: Figure 1a breaks down adoption by organisation type.

Original: Adoption is highest for research laboratories, followed closely by technology companies.
Revised: Adoption is highest for research laboratories, followed closely by technology companies.

Original: Non-tech companies as well as governmental organisations show substantially lower adoption.
Revised: Non-tech companies as well as governmental organisations show substantially lower adoption.

Original: Part of the explanation for this diﬀerence may be the proximity of research and tech organisations to the latest technological developments.
Revised: Part of the explanation for this diﬀerence may be the proximity of research and tech organisations to the latest technological developments.

Original: However, it seems unlikely that this is the only factor.
Revised: However, it seems unlikely that this is the only factor.

Original: Non-tech companies and governmental organisations may work with sensitive data more frequently, which could make them more cautious about adopting AutoML techniques.
Revised: Non-tech companies and governmental organisations may work with sensitive data more frequently, which could make them more cautious about adopting AutoML techniques.

Original: Usability issues are another factor that could play a role, as also postulated by Lee et al.
Revised: Usability issues are another factor that could play a role, as also postulated by Lee et al.

Original: and these may be easier to overcome for people with a strong technical background.
Revised: and these may be easier to overcome for people with a strong technical background.

Original: There appears to be a positive correlation between experience and adoption (Figure 1b).
Revised: There appears to be a positive correlation between experience and adoption (Figure 1b).

Original: Speciﬁcally, a substantial change in adoption appears after two years of experience.
Revised: Speciﬁcally, a substantial change in adoption appears after two years of experience.

Original: Having more than ﬁve years of experience does not seem to result in an additional increase in adop-
Revised: Having more than ﬁve years of experience does not seem to result in an additional increase in adop-

Original: [Figure]
Revised: [Figure]

Original: tion.
Revised: tion.

Original: These results are somewhat at odds with the notion that AutoML may democratise
Revised: These results are somewhat at odds with the notion that AutoML may democratise

Original: ML
Revised: ML

Original: by making it more accessible to users with less ML expertise.
Revised: by making it more accessible to users with less ML expertise.

Original: A comment by interview participant P2 (academic) could provide part of the explanation why more experienced teams may have higher adoption.
Revised: A comment by interview participant P2 (academic) could provide part of the explanation why more experienced teams may have higher adoption.

Original: They indicated that it is diﬃcult to judge how long
Revised: They indicated that it is diﬃcult to judge how long

Original: AutoML frameworks should be allowed to run to get good results while avoiding overﬁtting, something that might be learned with experience.
Revised: AutoML frameworks should be allowed to run to get good results while avoiding overﬁtting, something that might be learned with experience.

Original: This was echoed by P1 (tech), who indicated being more concerned about overﬁtting as a result of using AutoML techniques compared to using a manual approach based on domain knowledge.
Revised: This was echoed by P1 (tech), who indicated being more concerned about overﬁtting as a result of using AutoML techniques compared to using a manual approach based on domain knowledge.

Original: The respondents who just started using ML appear to be outside the perceived trend, but since they represent a relatively small group of the answers (7%), this result may be statistically less reliable.
Revised: The respondents who just started using ML appear to be outside the perceived trend, but since they represent a relatively small group of the answers (7%), this result may be statistically less reliable.

Original: Survey 2.
Revised: Survey 2.

Original: So far, the second survey resulted in 67 valid responses 3 to the AutoML questions.
Revised: So far, the second survey resulted in 67 valid responses 3 to the AutoML questions.

Original: Although the number of responses is much lower than in our ﬁrst survey, the breakdown into three questions concerning AutoML already allows us to gain additional insights.
Revised: Although the number of responses is much lower than in our ﬁrst survey, the breakdown into three questions concerning AutoML already allows us to gain additional insights.

Original: In Figure 2, the adoption per practice is shown.
Revised: In Figure 2, the adoption per practice is shown.

Original: Automated conﬁguration of algorithms and the structure of models is clearly less adopted than the other two practices.
Revised: Automated conﬁguration of algorithms and the structure of models is clearly less adopted than the other two practices.

Original: For feature generation and selection, it is interesting to note that more than 20% of the cases do this implicitly – for instance, by using deep learning.
Revised: For feature generation and selection, it is interesting to note that more than 20% of the cases do this implicitly – for instance, by using deep learning.

Original: The number of respondents that adopted model selection and hyperparameter optimisation at least partially is higher than in the ﬁrst survey, leaving only around 20% that do not adopt this practice at all.
Revised: The number of respondents that adopted model selection and hyperparameter optimisation at least partially is higher than in the ﬁrst survey, leaving only around 20% that do not adopt this practice at all.

Original: Due to the lower number of responses from our second survey, it is still diﬃcult to draw meaningful conclusions regarding adoption diﬀerences between organisation types and other demographics.
Revised: Due to the lower number of responses from our second survey, it is still diﬃcult to draw meaningful conclusions regarding adoption diﬀerences between organisation types and other demographics.

Original: Interviews.
Revised: Interviews.

Original: In addition to the factors inﬂuencing AutoML adoption mentioned so far, the interviewees also raised some other interesting points.
Revised: In addition to the factors inﬂuencing AutoML adoption mentioned so far, the interviewees also raised some other interesting points.

Original: Speciﬁc to automated feature selection, P1 (tech) saw few issues, raising that they can look back at what the selected features mean and whether the selection makes sense.
Revised: Speciﬁc to automated feature selection, P1 (tech) saw few issues, raising that they can look back at what the selected features mean and whether the selection makes sense.

Original: P1 also indicated that the initial required investment to adopt new AutoML techniques is high, and at odds with short
Revised: P1 also indicated that the initial required investment to adopt new AutoML techniques is high, and at odds with short

Original: [Figure]
Revised: [Figure]

Original: production times to ship to customers.
Revised: production times to ship to customers.

Original: However, P1 did see potential beneﬁts in the long run, where AutoML might provide savings in the time spent by human experts.
Revised: However, P1 did see potential beneﬁts in the long run, where AutoML might provide savings in the time spent by human experts.

Original: Since with deep learning, explainability is already an issue, P1 was not too concerned about possible explainability issues stemming from AutoML use in this context.
Revised: Since with deep learning, explainability is already an issue, P1 was not too concerned about possible explainability issues stemming from AutoML use in this context.

Original: P2 (academic) mentioned that in cases where AutoML frameworks don’t work, it is generally diﬃcult to ﬁnd out why it is not working.
Revised: P2 (academic) mentioned that in cases where AutoML frameworks don’t work, it is generally diﬃcult to ﬁnd out why it is not working.

Original: In which component is the issue?
Revised: In which component is the issue?

Original: Is it a problem with the dataset or a bug?
Revised: Is it a problem with the dataset or a bug?

Original: For P1, neural architecture search on the full search space would be too expensive, making the manual design with expert knowledge more practical.
Revised: For P1, neural architecture search on the full search space would be too expensive, making the manual design with expert knowledge more practical.

Original: P1 further indicated that the use of ML (without AutoML) often already led to improvements for their customers; in combination with limited production time, in their view, that left relatively little opportunity and motivation for using AutoML techniques to possibly achieve improved performance.
Revised: P1 further indicated that the use of ML (without AutoML) often already led to improvements for their customers; in combination with limited production time, in their view, that left relatively little opportunity and motivation for using AutoML techniques to possibly achieve improved performance.

Original: 5. Conclusion
Revised: 5. Conclusion

Original: Aiming to measure AutoML adoption by practitioners working on applications with ML components, we identiﬁed three engineering practices for AutoML, and measured their adoption through surveys.
Revised: Aiming to measure AutoML adoption by practitioners working on applications with ML components, we identiﬁed three engineering practices for AutoML, and measured their adoption through surveys.

Original: In addition, a preliminary set of interviews was held to gain insight into the factors driving and inhibiting adoption.
Revised: In addition, a preliminary set of interviews was held to gain insight into the factors driving and inhibiting adoption.

Original: Our ﬁndings indicate that there is substantial room to increase adoption across the board.
Revised: Our ﬁndings indicate that there is substantial room to increase adoption across the board.

Original: Based on the preliminary interviews, the relatively low adoption might result from usability issues with AutoML frameworks and the substantial computational resources required for AutoML.
Revised: Based on the preliminary interviews, the relatively low adoption might result from usability issues with AutoML frameworks and the substantial computational resources required for AutoML.

Original: In the near future, we plan to conduct additional interviews.
Revised: In the near future, we plan to conduct additional interviews.

Original: So far, our two interviews only provided ﬁrst insights into possible reasons for limited adoption of AutoML practices.
Revised: So far, our two interviews only provided ﬁrst insights into possible reasons for limited adoption of AutoML practices.

Original: It would be of particular interest to interview people that have not yet adopted AutoML, since both the interviews we present here and those by Xin et al.
Revised: It would be of particular interest to interview people that have not yet adopted AutoML, since both the interviews we present here and those by Xin et al.

Original: (2021) targeted users with at least partial adoption.
Revised: (2021) targeted users with at least partial adoption.

Original: We expect that this will help us in testing and reﬁning our preliminary ideas regarding the reasons why some ML practitioners refrain from adopting
Revised: We expect that this will help us in testing and reﬁning our preliminary ideas regarding the reasons why some ML practitioners refrain from adopting

Original: AutoML tools and techniques.
Revised: AutoML tools and techniques.

Revised: Acknowledgements

Revised: This research was partially supported by TAILOR, a project funded by EU Horizonresearch and innovation programme under GA No

Original: Appendix A. Survey 1 – Overall Model Selection or Hyperparameter Optimisation
Revised: Appendix A. Survey Questions about AutoML A.1 Survey

Revised: We can perform model selection and hyper-parameter optimization in an automated way. A.2 Survey

Revised: We use automated methods to generate or select features from input data.

Revised: We perform model selection and hyper-parameter optimisation in an automated way.

Revised: We use automated methods to conﬁgure our algorithms or the structure of our models.

Revised: Appendix B. Survey 1 – Overall Model Selection or Hyperparameter Optimisation

Revised: Answer

Original: [Figure]
Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Original: [Table] 
Revised: [Table] [Figure] [Table] [Figure]

Original: Appendix B. Survey 1 – Model Selection or Hyperparameter Optimisation by Demographic
Revised: Appendix C. Survey 1 – Model Selection or Hyperparameter Optimisation by Demographic

Original: [Figure]
Revised: [Figure]

Original: Appendix C. Survey 2 – Mean Adoption by Demographic
Revised: Appendix D. Survey 2 – Mean Adoption by Demographic

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Appendix D. Survey 2 – Feature Generation/Selection by Demographic
Revised: Appendix E. Survey 2 – Feature Generation/Selection by Demographic

Original: [Figure]
Revised: [Figure]

Original: Appendix E. Survey 2 – Model Selection or Hyperparameter Optimisation by Demographic
Revised: Appendix F. Survey 2 – Model Selection or Hyperparameter Optimisation by Demographic

Original: [Figure]
Revised: [Figure]

Original: Appendix F. Survey 2 – Algorithm/Model Conﬁguration by Demographic
Revised: Appendix G. Survey 2 – Algorithm/Model Conﬁguration by Demographic

Original: [Figure]
Revised: [Figure]