Original: I N D EFENSE OF THE P APER
Revised: I N D EFENSE OF THE P APER

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The machine learning publication process is broken, of that there can be no doubt.
Revised: The machine learning publication process is broken, of that there can be no doubt.

Original: Many of these ﬂaws are attributed to the current workﬂow: L A TEX to PDFsubject to review before the published PDF.
Revised: Many of these ﬂaws are attributed to the current workﬂow: L A TEX to PDF to reviewers to camera ready PDF.

Original: This has driven the desire for new forms of publications; ones that can increase inclusively, accessibility and pedagogical strength.
Revised: This has understandably resulted in the desire for new forms of publications; ones that can increase inclusively, accessibility and pedagogical strength.

Original: However, this venture fails to address the origins of these inadequacies in the contemporary paper workﬂow.
Revised: However, this venture fails to address the origins of these inadequacies in the contemporary paper workﬂow.

Original: The paper, being the basic unit of academic research, is merely how problems in the publication and research ecosystem manifest; but is not itself responsible for them.
Revised: The paper, being the basic unit of academic research, is merely how problems in the publication and research ecosystem manifest; but is not itself responsible for them.

Original: Not only will simply replacing or augmenting papers with different formats not ﬁxthe existing problems; when used as a band-aid without systemic changes, will likely exacerbate the existing inequities.
Revised: Not only will simply replacing or augmenting papers with different formats not ﬁx existing problems when used as a band-aid without systemic changes, will likely exacerbate the existing inequities.

Original: In this work, we argue that the root cause of hindrances in accessibility of machine learning research lies not in the paper workﬂow but within the misaligned incentives behind the publishing and research process.
Revised: In this work, we argue that the root cause of hindrances in accessibility of machine learning research lies not in the paper workﬂow but within the misaligned incentives behind the publishing and research process.

Original: We discuss these problems, and argue  the paper is the optimal workﬂow.
Revised: We discuss these problems and argue that the paper is the optimal workﬂow.

Original: We also highlight some potential solutions for the incentivization problems.
Revised: We also highlight some potential solutions for the incentivization problems.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Machine Learning is one of the most popular ﬁelds of research, with tens of thousands of papers being produced every year 1 .
Revised: Machine Learning is one of the most popular ﬁelds of research, with tens of thousands of papers being produced every year 1 .

Original: Contemporary machine learning research is dominated by deep neural networks (LeCun et al., 2015), which have achieved well publicized successes in a variety of ﬁelds including (but not limited to): Computer Vision (Krizhevsky et al., 2012), (Redmon et al., 2016),
Revised: Contemporary machine learning research is dominated by deep neural networks (LeCun et al., 2015), which have achieved well publicized successes in a variety of ﬁelds including (but not limited to): Computer Vision (Krizhevsky et al., 2012), (Redmon et al., 2016),

Original: Generative Modelling (Goodfellow et al., 2014), (Karras et al., 2019), Natural Language Processing (Sutskever et al., 2014), (Devlin et al., 2018), and Reinforcement Learning (Mnih et al., 2015), (Silver et al., 2016),(Redmon et al., 2016), (Vinyals et al., 2019).
Revised: Generative Modelling (Goodfellow et al., 2014), (Karras et al., 2019), Natural Language Processing (Sutskever et al., 2014), (Devlin et al., 2018), and Reinforcement Learning (Mnih et al., 2015), (Silver et al., 2016), (Vinyals et al., 2019).

Original: However, as the ﬁeld grows massively, so too do the problems.
Revised: Parallel to this explosion in research volume are the potentially problematic implications of this research.

Original: These range from privacy and discrimination concerns of large language models(Bender et al., 2021), (Carlini et al., 2020), (McGufﬁe & Newhouse, 2020),  to environmental effects (Strubell et al., 2019), to algorithmic bias (Buolamwini & Gebru, 2018), (Hooker et al., 2020), (Bagdasaryan & Shmatikov, 2019), and further ethical implications of widespread adoption of machine learning systems (A¨ıvodji et al., 2019).
Revised: These range from privacy and discrimination concerns of large language models (Carlini et al., 2020), (McGufﬁe & Newhouse, 2020), (Bender et al., 2021) to environmental effects (Strubell et al., 2019), to algorithmic bias (Buolamwini & Gebru, 2018), (Hooker et al., 2020), (Bagdasaryan & Shmatikov, 2019), and further ethical implications of widespread adoption of machine learning systems (A¨ıvodji et al., 2019).

Original: Theseworks are  concerned with the greater social implications of machine learning research, as opposed to the machine learning research process itself.
Revised: These are important works concerned with the greater social implications of machine learning research, which differs from the focus of this work: the machine learning research process itself.

Original: When evaluating the publication system as a whole, not only can the  problems be explained, but we can begin to understand the mechanisms of whyexactly machine learning research has these problems.
Revised: When evaluating the publication system as a whole, not only can the existing problems be explained, but we can begin to understand the mechanisms of why machine learning research has these problems.

Original: While the symptoms of systemic problems are often discussed, e.g.  reproducibility problems (Pineau et al., 2020), hypothesizing after experiments (Gencoglu et al., 2019), etc.
Revised: While the symptoms of systemic problems are often discussed, e.g. the reproducibility crisis (Pineau et al., 2020), hypothesizing after experiments (Gencoglu et al., 2019), etc.

Original: the root cause of these issues are often ignored or misattributed.
Revised: the root cause of these issues are often ignored or misattributed.

Original: Misaligned incentives were previously mentioned by Sculley et al.
Revised: Misaligned incentives were previously mentioned by Sculley et al.

Original: (2018); however, it was only touched on brieﬂy and their solutions failed to address  systemic incentivation problems.
Revised: (2018); however, it was only touched on brieﬂy and their solutions did not address any systemic incentivation problems.

Revised: Lipton & Steinhardt (2018) provides an excellent overview of existing problems in machine learning publications, but offers little in terms of the origin of these problems (once agian only brieﬂy mentioning misaligned incentives) or what to do about them.

Original: The rationale for this workshop is understandable, contemporary machine learning papers are often far from the pinnacle of inclusivity, explainability, and interpretability.
Revised: The rationale for this workshop is understandable and important, contemporary machine learning papers are often far from the pinnacle of inclusivity, explainability, and interpretability.

Original: The idea that a new conceptualization of the research publication  is a natural one.
Revised: The idea that a new conceptualization of the research publication will help alleviate some of these problems is a natural one.

Original: However, it fails to consider the problem as a whole, missing the forest for the trees.
Revised: However, it fails to consider the problem as a whole, missing the forest for the trees.

Original: These new formats will fail to meaningfully solve any of the existing problems in accessibility and will likely decrease inclusivity because they will fail tomeaningfully address any of the primary  causes of these problems.
Revised: These new formats will fail to meaningfully solve any of the existing problems in accessibility and will likely decrease inclusivity because they will fail to address any of the primary root causes of these problems.

Original:  M ISALIGNED I NCENTIVES
Revised:  M ISALIGNED I NCENTIVES

Original: Here we highlight the problems that exist in the publication system that decrease accessibility, explainability and interpretability.
Revised: Here we highlight the problems that exist in the publication system that decrease accessibility, explainability and interpretability.

Original: These are often attributed to the paper workﬂow; however, we argue that these problems are the result of misaligned incentives that merely present themselves in the paper (the basic unit of contemporary research).
Revised: These are often attributed to the paper workﬂow; however, we argue that these problems are the result of misaligned incentives that merely present themselves in the paper (the basic unit of contemporary research).

Original: 2. D EANS D ON ’ T R EAD , T HEY O NLY C OUNT
Revised: 2. D EANS D ON ’ T R EAD , T HEY O NLY C OUNT

Original: The ﬁrst problem is one that decreases inclusivity in the research production process.
Revised: The ﬁrst problem is one that decreases inclusivity in the research production process.

Original: The expected pace of producing research, especially for graduate students and early career researchers, is a major problem.
Revised: The expected pace of producing research, especially for graduate students and early career researchers, is a major problem.

Original: Not only is this pace of research extremely exclusive, those able to keep up with it pay the price in mental stability (Chirikov et al., 2020). 
Revised: Not only is this pace of research extremely exclusive, those able to keep up with it pay the price in mental stability (Chirikov et al., 2020). This problem has only been exacerbated by the

Original:  There are usually 5 major conferences per year that publications are expected in: Association for the Advancement of Artiﬁcial Intelligence (AAAI) Conference, Conference on Neural Information Processing Systems (NeurIPS),
Revised: COVID-19 pandemic (Gewin, 2021). There are usually 5 major conferences per year that publications are expected in: Association for the Advancement of Artiﬁcial Intelligence (AAAI) Conference on Artiﬁcial Intelligence, Conference on Neural Information Processing Systems (NeurIPS),

Original: International Conference on Machine Learning (ICML) with the other two being subﬁeld speciﬁc.
Revised: International Conference on Machine Learning (ICML) with the other two being subﬁeld speciﬁc.

Original: E.g. for Natural Language Processing: Association for Computational Linguistics (ACL) Conference, Empirical Methods in Natural Language Processing (EMNLP), for Reinforcement Learning:
Revised: E.g. for Natural Language Processing: Association for Computational Linguistics (ACL) Conference, Empirical Methods in Natural Language Processing (EMNLP), for Reinforcement Learning:

Original: International Conference on Learning Representations (ICLR), for Computer Vision: Conference on Computer Vision and Pattern Recognition (CVPR), European Conference on Computer Vision (ECCV), International Conference on Computer Vision (ICCV), etc.
Revised: International Conference on Learning Representations (ICLR), for Computer Vision: Conference on Computer Vision and Pattern Recognition (CVPR), European Conference on Computer Vision (ECCV), International Conference on Computer Vision (ICCV), etc.

Original: We list all of these to emphasize the magnitude of the publishing expectations.
Revised: We list all of these to emphasize the magnitude of the publishing expectations.

Original: How does this problem manifest itself in papers?
Revised: How does this problem manifest itself in papers?

Original: Via the (in)famous concept of the “Salami Publication” (ˇSupak Smol ˇ ci ´ c, 2013), also called the Minimum Publishable Unit (MPU).
Revised: Via the (in)famous concept of the “Salami Publication” ( ˇ Supak Smol ˇ ci ´ c, 2013), also called the Minimum Publishable Unit (MPU).

Original: The MPU is  the smallest amount of work that can result in a publication.
Revised: The MPU is exactly what the name indicates: the smallest amount of work/research that can result in a publication.

Original: Given the publication expectations, why would one publish a “better” paper in one venue, when one could publish two lower quality papers in two venues?
Revised: Given the publication expectations, why would one publish a “better” paper in one venue, when one could publish two lower quality papers in two venues?

Original: If you could get one paper in NeurIPS or one paper in AAAI and NeurIPS, one would always chose the latter.
Revised: If you could get one paper in NeurIPS or one paper in AAAI and NeurIPS, one would always chose the latter.

Revised: This isn’t meant to blame the researchers, it is exactly what a rational actor would do.

Original: Because existing incentives (e.g. funding, promotions, academic/social captial, etc.) rewards the number of publications, something that can be objectively measured, not the quality of papers, something that cannot be objectively determined.
Revised: Existing incentives (e.g. funding, promotions, academic/social capital, etc.) rewards the number of publications and having a publication in the major conferences, something that can be objectively measured, not the quality of papers, something that cannot be objectively determined.

Original: This inevitably pushes researchers towards the MPU (Edwards & Roy, 2017).
Revised: This inevitably pushes researchers towards the MPU (Edwards & Roy, 2017).

Original: While this is undoubtedly problematic and is a considerable source of exclusionary practices in research production, it is not attributable to the paper workﬂow.
Revised: While this is undoubtedly problematic and is a considerable source of exclusionary practices in research production, it is not attributable to the paper workﬂow.

Original: Although we can (subjectively) observe papers becoming more “salami”, it would be mistaken to assume that this is a result of the paper.
Revised: Although we can (subjectively) observe papers becoming more “salami”, it would be mistaken to assume that this is a result of the paper.

Original: It is merely because the paper is the basic unit of research expression,  ﬂaws in the system will naturally express themselves through it.
Revised: It is merely because the paper is the basic unit of research expression, thus ﬂaws in the system will naturally express themselves through it.

Original: 2. W INNING I SN ’ T E VERYTHING , I T ’ S T HE O NLY T HING
Revised: 2. W INNING I SN ’ T E VERYTHING , I T ’ S T HE O NLY T HING

Original: The second problem is the constant push for State of the Art (SotA) results on a set of standard benchmarks.
Revised: The second problem is the constant push for State of the Art (SotA) results on a set of standard benchmarks.

Original: It is generally understood in research that when presenting an empirical method, SotA results are important to getting published.
Revised: It is generally understood in research that when presenting an empirical method, SotA results are important to getting published.

Original: Thus, this incentives SotA-hacking, the machine learning equivalent to p-hacking (Head et al., 2015).
Revised: Thus, this incentives SotA-hacking, the machine learning equivalent to p-hacking (Head et al., 2015).

Original: This limits the explainability of papers and is a severe pedagogical weakness.
Revised: This limits the explainability of papers and is a severe pedagogical weakness.

Original: To understand why this is the case, we ﬁrst need to evaluate the results of this incentivation.
Revised: To understand why this is the case, we ﬁrst need to evaluate the results of this incentivation.

Original: Because SotA results are key to successful publications, it quickly became the goal of machine learning research.
Revised: Because SotA results are key to successful publications, it quickly became the goal of machine learning research; a publication version of Goodhart’s law 2 (Goodhart, 1984).

Original: When the goal is SotA, as opposed to thorough and impactful contributions to human knowledge, it results in poor scientiﬁc practices.
Revised: When the goal is SotA, as opposed to thorough and impactful contributions to human knowledge, it results in poor scientiﬁc practices.

Original: One such example is Hypothesizing 
Revised: One such example is Hypothesizing After Results Are Known (HARK)

Original: After Results Are Known (HARK) (Kerr, 1998), conducting experiments and once you have empirical results that are publishable, working backwards to ﬁll in why this technique works.
Revised: (Kerr, 1998): conducting experiments and once you have empirical results that are publishable, working backwards to ﬁll in why this technique works.

Original: A common variation of HARKing in machine learning research is called “Grad Student Descent” (Gencoglu et al., 2019), i.e. running experiments in which you change hyperparameters (largely at random) that result in a mild increase in performance which is then (mis)attributed to a seemingly substantial theoretical change.
Revised: A common variation of HARK in machine learning research is called “Grad Student Descent” (Gencoglu et al., 2019), i.e. running experiments in which you change hyperparameters (largely at random) that result in a mild increase in performance which is then (mis)attributed to a seemingly substantial theoretical change.

Original: This backﬁll is necessary because random hyperparameter changes that result in SotA results is  a publishable methodology.
Revised: This backﬁll is necessary because random hyperparameter changes that result in SotA results is not a publishable methodology.

Original: This misattribution of the source of improvements is a severe problem for explainability.
Revised: This misattribution of the source of improvements is a severe problem for explainability.

Original: It is not uncommon for SotA results to be attributed to algorithmic differences, when the real origin of the improvement is something far more banal, such as random seeds (Henderson et al., 2018) or code level changes (e.g. learning rate, gradient clipping, etc.)
Revised: It is not uncommon for SotA results to be attributed to algorithmic differences, when the real origin of the improvement is something far more banal, such as random seeds (Henderson et al., 2018) or code level changes (e.g. learning rate, gradient clipping, etc.)

Original: (Engstrom et al., 2020).
Revised: (Engstrom et al., 2020), (Tucker et al., 2018).

Original: This SotA misincentivation manifests itself via the paper, resulting in publications that are severely ﬂawed and obfuscatory (to hide the true origin of their performance gains).
Revised: This SotA misincentivation manifests itself via the paper, resulting in publications that are severely ﬂawed and obfuscatory (to hide the true origin of their performance gains).

Original: Here, again, we can see it is easy to blame the paper for lacking explainability; however, this is not a result of the paper workﬂow, but the incentives behind it.
Revised: Here, again, we can see it is easy to blame the paper for lacking explainability; however, this is not a result of the paper workﬂow, but the incentives behind it.

Original: 2. D ON ’ T H ATE T HE P LAYER , H ATE T HE G AME
Revised: 2. D ON ’ T H ATE T HE P LAYER , H ATE T HE G AME

Original: A third, and ﬁnal, problem in machine learning research publications is the theoretical/mathematical foundations, or, lack-thereof.
Revised: A third problem in machine learning research publications is the theoretical/mathematical foundations, or lack-thereof.

Original: The meaningless (and not infrequently incorrect or irrelevant) mathematics and equation packing included in machine learning papers is a severe hindrance to the interpretability of papers.
Revised: The meaningless (and not infrequently incorrect or irrelevant) mathematics and equation packing included in machine learning papers is a severe hindrance to the interpretability of papers.

Original: The general form of many research papers is: introduction, background, theoretical (mathematical) foundation, empirical results, conclusion.
Revised: The general form of many research papers is: introduction, background, theoretical (mathematical) foundation, empirical results, conclusion.

Original: This is not inherently a bad way to structure a paper.
Revised: This is not inherently a bad way to structure a paper.

Original: The problem is that because there is an expectation for a thorough mathematical background, papers that do not naturally ﬁt this format must mutilate themselves into it.
Revised: The problem is that because there is an expectation for a thorough mathematical background, papers that do not naturally ﬁt this format must mutilate themselves into it.

Revised: Even papers that do ﬁt this format are not immune mathematical ﬂaws (Reddi et al., 2019).

Original: The faulty incentives here act on both reviewer and researcher.
Revised: The faulty incentives here act on both reviewer and researcher.

Original: Reviewer incentives encourage acceptance of papers that conform to the status quo of the existing normal science paradigm (Kuhn, 2012), expected extensive mathematical/theoretical justiﬁcations for the empirical results.
Revised: Reviewer incentives encourage acceptance of papers that conform to the status quo of the existing normal science paradigm (Kuhn, 2012), expecting extensive mathematical/theoretical justiﬁcations for the empirical results.

Original: Researcher incentives encourage publication, leading to the reviewers being  an obstacleto publications (rather than an important part of the process) and seen with some deal of animosity (Peterson, 2020).
Revised: Researcher incentives encourage publications, leading to the reviewers being seen as an obstacle (rather than an important part of the process) and can be subject to animosity (Peterson, 2020).

Original: Rather than reviewer and researcher sharing the same goal of improving research, of making papers as understandable and interpretable as possible, their incentivations encourage the opposite.
Revised: Rather than reviewer and researcher sharing the same goal of improving research, sharing important work with the scientiﬁc community, making papers as understandable and interpretable as possible, their incentivations encourage the opposite.

Original: These expectations result in mathematical foundations that are intentionally long and obfuscatory, often irrelevant and, sometimes, outright wrong (Lipton & Steinhardt, 2018).
Revised: These expectations result in mathematical foundations that are intentionally long and obfuscatory, often irrelevant and, sometimes, outright wrong (Lipton & Steinhardt, 2018).

Original: What could be said in a few clear and coherent set of sentences is said in unhelpful lemmas and proof-ﬁlled appendices (note this is not a complaint of math in papers, but of unnecessary math in papers that hurt interpretability).
Revised: What could be said in a few clear and coherent set of sentences is said in unhelpful lemmas and proof-ﬁlled appendices (note this is not a complaint of math in papers, but of unnecessary math in papers that hurt interpretability).

Original: Typically, this math offers some “intuition” into why the method works, but offers little in terms of mathematical rigour or relevance.
Revised: Typically, this math offers some “intuition” into why the method works, but offers little in terms of mathematical rigour or relevance.

Original: This intuition often comes in the form of proofs of error bounds or convergence properties, which may be mathematically valid, but are irrelevant when their assumptions get thrown out the window because of the usage of massively non-convex deep neural networks.
Revised: This intuition often comes in the form of proofs of error bounds or convergence properties, which may be mathematically valid, but are irrelevant when their assumptions get thrown out the window because of the usage of massively non-convex deep neural networks.

Original: Consider the famous reinforcement learning paper Haarnoja et al.
Revised: Consider the famous reinforcement learning paper Haarnoja et al.

Original: (2018) 2 , in which convergence proofs are given for tabular soft policy iteration (taking up nearly a page)
Revised: (2018) 3 , in which convergence proofs are given for tabular soft policy iteration (taking up nearly a page)

Original: but none of the proofs actually apply to their algorithm which uses neural networks.
Revised: but none of the proofs actually apply to their algorithm which uses neural networks.

Revised: This technique of proving the tabular case but using neural networks for the main algorithm (making the tabular proofs largely moot) is far from rare in RL research (Lan et al., 2020), (Dabney et al., 2018).

Original: This  “intuition” is often wrong.
Revised: This is problematic because this “intuition” is often wrong when it comes to neural networks.

Original: It is known that Q-learning (Watkins & Dayan, 1992) has guaranteed convergence properties; but when a neural network is used, these properties become meaningless (convergence being one of the largest problems of Deep Q Networks)
Revised: It is known that Q-learning (Watkins & Dayan, 1992) has guaranteed convergence properties; but when a neural network is used, these properties become meaningless (convergence being one of the largest problems of Deep Q Networks)

Original: (Mnih et al., 2013).
Revised: (Mnih et al., 2013).

Original: Once again, when reading contemporary research and seeing thisproblem of meaningless math, it is easy to blame this on the paper.
Revised: Once again, when reading contemporary research and seeing this meaningless math, it is easy to blame this on the paper.

Original: But the uninterpretable mess is not a result of the paper process, but the incentivation behind it.
Revised: But the uninterpretable mess is not a result of the paper process, but the incentivation behind it.

Original:  W HY N OT C HANGE ?
Revised:  W HY N OT C HANGE ?

Original: It is here that we would like to offer a rebuttal to the work being submitted to this workshop.
Revised: It is here that we would like to offer a rebuttal to the work being submitted to this workshop.

Original: While the submissions will surely be high quality pieces of research, they will ultimately fail to solve any of the problems previously laid out and fail to meet the desires goals of increasing inclusivity, explainability, and interpretability.
Revised: While the submissions will surely be high quality pieces of research, they will ultimately fail to solve any of the problems previously laid out and fail to meet the desires goals of increasing inclusivity, explainability, and interpretability.

Original: This seems like a bold claim, given that we cannot see what has been submitted; however, we are able to make it because ultimately any change to the paper workﬂow is just that: a workﬂow change.
Revised: This seems like a bold claim, given that we cannot see what has been submitted; however, we are able to make it because ultimately any change to the paper workﬂow is just that: a workﬂow change.

Original: Any change that doesn’t fundamentally change the publication process, instituting systemic changes, cannot succeed in addressing the primary concerns of this workshop.
Revised: Any change that doesn’t fundamentally alter the publication process, instituting systemic changes, cannot succeed in addressing the primary concerns of this workshop.

Original: 3. T HE D OUBLE -E DGED S WORD
Revised: 3. T HE D OUBLE -E DGED S WORD

Original: Let us ﬁrst consider  the tradeoff of inclusivity vs explainability.
Revised: Let us ﬁrst consider what we argue is the tradeoff of inclusivity vs explainability.

Original: Increasing inclusivity requires that the machine learning research ﬁeld is more open(e.g. to previously underrepresented/disadvantaged groups).
Revised: Increasing inclusivity requires that the machine learning research ﬁeld is more open to those contributing to research (especially to underrepresented/disadvantaged groups).

Original: However, increasing the pedagogical strength of work acts counteracts inclusivity efforts.
Revised: However, increasing the pedagogical strength of work can act to counteract inclusivity efforts.

Original: Consider the existing example of distill.pub.
Revised: First, consider the existing example of distill.pub.

Original: These works represent a step up in pedagogical strength (increasing explainability andespecially visualization).
Revised: These works represent a step up in pedagogical strength (increasing explainability and visualization).

Original: However, this requires substantially more time and skills than traditional publications (as the journal notes).
Revised: However, this requires substantially more time and skills than traditional publications (as the journal notes).

Original: We cannot increase inclusivity in research while also increasing the number of  skills required for a publication.
Revised: We cannot increase inclusivity in research while also increasing the number of (non-research) skills required for a publication.

Original: Other solutions that have better visualizations inevitably increase barriers, as machine learning research is not about visualizations (thus requiring additional skills).
Revised: Other solutions that have better visualizations inevitably increase barriers, as machine learning research is not about visualizations (thus requiring additional skills).

Revised: If someone has done research worth publishing, but there are barriers to publication (visualization, advanced explainability techniques, etc.), it limits both the researcher and the community.

Original: Here we can see the paper as the optimal form of publication.
Revised: Here we can see the paper as the optimal form of publication.

Original: The paper represents the lowest entry requirements when it comes to inclusivity.
Revised: The paper represents the lowest entry requirements when it comes to inclusivity.

Original: If one has completed some research, writing it down represents the lowest conceivable barrier (the bare minimum of communication).
Revised: If one has completed some research, writing it down represents the lowest conceivable barrier (the bare minimum of communication).

Original: It requires no further technical or research skills, only the ability to write (which is an implicit prerequesite for machine learning research).
Revised: It requires no further technical or research skills, only the ability to write (which is an implicit prerequisite for machine learning research).

Original: Any other publication workﬂow increases this barrier.
Revised: Any other publication workﬂow increases this barrier.

Original: Whatever the reconceptualization is, it is deﬁnitionally not the paper and therefore will require more skills (increasing the barrier), be in in web design, visualization, rendering,  etc.
Revised: Whatever the reconceptualization is, it is deﬁnitionally not the paper and therefore will require more skills (increasing the barrier), be in in web design, visualization, rendering, video communication etc.

Original: Not only is the paper optimally inclusive (lowest barrier to entry)
Revised: Not only is the paper optimally inclusive (lowest barrier to entry)

Original: it also has an extremely high explainability ceiling.
Revised: it also has an extremely high explainability ceiling.

Original: Papers have the ability to be extremely effective methods of communicating machine learning research.
Revised: Papers have the ability to be extremely effective methods of communicating machine learning research.

Original: That most papers  aren’tcurrently very well written does not mean it is an ineffective format, merely that the incentives need to be changed to reward explainability.
Revised: The fact that paper also has a low explainability ﬂoor, resulting in many papers that aren’t very well written does not mean it is an ineffective format, merely that the incentives need to be changed to reward explainability.

Original: One might, understandably, challenge this and say something along the lines of, “we would never require every single paper to instantly transition to our new style and use (whatever new techniques are introduced).
Revised: One might, understandably, challenge this and say something along the lines of, “we would never require every single paper to instantly transition to our new style and use [whatever new techniques are introduced].

Original: The traditional paper workﬂow will still exist, we are merely augmenting it”.
Revised: The traditional paper workﬂow will still exist, we are merely augmenting it”.

Original: However, this once again fails to address the fundamental problems causing the inaccessibility.
Revised: However, this once again fails to address the fundamental problems causing the inaccessibility.

Original: Augmenting the paper workﬂow sounds promising but even that would decrease inclusivity if the other problems are not ﬁxed.
Revised: Augmenting the paper workﬂow sounds promising but even that would decrease inclusivity if the other problems are not ﬁxed.

Original: Without changing the sheer number of publications, these “augmented” publications would naturally rise to the top.
Revised: Without changing the sheer number of publications, these “augmented” publications would naturally rise to the top (as existing publications from prominent labs/groups already do).

Original: When reading (and citing) research, we reach not the work that has the most inclusive practices, but the easiest to cite and read.
Revised: When reading (and citing) research, we reach not the work that has the most inclusive practices, but the easiest to cite and read.

Original: Second, we can see that, in reality, most of these augmented publications would change little.
Revised: Second, we can see that, in reality, most of these augmented publications would change little.

Original: There are already existing “augmented” publications in the form of blogs and talks and demos that go along with some papers.
Revised: There are already existing “augmented” publications in the form of blogs and talks and demos that go along with some papers.

Original: These are undoubtedly improving the visualization and the communication of complex ideas.
Revised: These are undoubtedly improving the visualization and the communication of complex ideas.

Original: However, the same groups that currently use these techniques would also be the only groups to use the new publication methods as well: large companies and academic labs.
Revised: However, the same groups that currently use these techniques would also be the only groups to use the new publication methods as well: large companies and academic labs.

Original: This offers little in the way of increasing inclusivity.
Revised: This offers little in the way of increasing inclusivity.

Original: 3. E XPLAIN , BUT NOT TOO MUCH
Revised: 3. E XPLAIN , BUT NOT TOO MUCH

Original: What exactly is the purpose of publications?
Revised: What exactly is the purpose of publications?

Original: One of the core questions that this workshop asks is: “How do we communicate ML research and theory more effectively?”
Revised: One of the core questions that this workshop asks is: “How do we communicate ML research and theory more effectively?”

Original: As we previously discussed, there is no doubt a problem with explainability in contemporary papers.
Revised: As we previously discussed, there is no doubt a problem with explainability in contemporary papers.

Original: However, it is important to recognize that ultimate goal of a research publication is to convey the information to a (relatively) specialized audience.
Revised: However, it is important to recognize that ultimate goal of a research publication is to convey the information to a (relatively) specialized audience.

Original: While scientiﬁc communication to larger audiences/the general public is important, that is not the role of the publication, so we must improve explainability while keeping the focus on the audience (i.e. who we are explaining to).
Revised: While scientiﬁc communication to larger audiences/the general public is important, that is not the role of the publication. Therefore, when seeking to improve explainability, it is important to remember the audience (i.e. who we are explaining to).

Original: There needs to be some expectation of background knowledge in order to effectively and concisely convey research to ones peers.
Revised: There needs to be some expectation of background knowledge in order to effectively and concisely convey research to ones peers.

Original:  W HAT IS MACHINE LEARNING ?
Revised:  W HAT IS MACHINE LEARNING ?

Original: Before getting to potential solutions, we will brieﬂy touch on an important philosophical concept that often goes undiscussed in machine learning research.
Revised: Before getting to potential solutions, we will brieﬂy touch on an important philosophical concept that often goes undiscussed in machine learning research.

Original: That is the question of the epistemological basis of machine learning, or given the crossroads we are at as a ﬁeld, what the basis of machine learning research should be.
Revised: That is the question of the epistemological basis of machine learning, or given the crossroads we are at as a ﬁeld, what the basis of machine learning research ought to be.

Original: Should what the majority of the ﬁeld is doing be part of the publication process?
Revised: Should what much of the ﬁeld is doing be part of the publication process?

Original: As we previously discussed, much of machine learning research is deﬁned by a hyper focus on benchmarks (Wagstaff, 2012).
Revised: As we previously discussed, much of machine learning research is deﬁned by a hyper focus on benchmarks (Wagstaff, 2012).

Original: While random hyperparameter optimization isn’t really “research”, that doesn’t mean it isn’t of any use.
Revised: While random hyperparameter optimization isn’t a theoretically sound method of research, that doesn’t mean it isn’t of any use.

Original: Better hyperparameters can help withindustrial and machine learning models used inthe real world.
Revised: Better hyperparameters can help with machine learning models used in real world applications.

Original: Even if incentives are ﬁxed, these results are still important, but they need a  place to exist outside of conferences.
Revised: Even if incentives are ﬁxed, these results are still important, but they perhaps there is a better place for them than the proceedings of conferences.

Original: When asking what machine learning should be, we have to consider what the goal of machine learning research is.
Revised: When asking what machine learning should be, we have to consider what the goal of machine learning research is.

Original: Machine learning is in a unique position, given it’s unusually strong ties to industry, At NeurIPS’20, a single company (Google) was involved in more than 12% of all papers 3 .Data
Revised: Machine learning is in a unique position, given it’s unusually strong ties to industry. At NeurIPS’20, a single company (Google) was involved in more than 12% of all papers 4 .

Original: While corporations can contribute meaningful research to scientiﬁc ﬁelds, the substantial presence begs the question: is machine learning research’s goal really a scientiﬁc one?
Revised: While corporations can contribute meaningful research to scientiﬁc ﬁelds, the substantial presence begs the question: is machine learning research’s goal really a scientiﬁc one?

Original: While the “goal of science” is difﬁcult to philosophically deﬁne and there is a long history of debate (Leplin, 1984), we are able to sidestep that dilemma by simply contrasting it with the goal of  the industrial paradigm.
Revised: While the “goal of science” is difﬁcult to philosophically deﬁne and there is a long history of debate (Leplin, 1984), we are able to sidestep that dilemma by simply contrasting it with the goal of what we call the “industrial paradigm”.

Original: Industrial paradigm machine learning research attempts to improve techniques used in practice, without concern for fundamental knowledge.
Revised: Industrial paradigm machine learning research attempts to improve techniques used in practice, without concern for fundamental knowledge.

Original: This breakdown might, at ﬁrst, sound like a simple division of theory vs. applied work.
Revised: This breakdown might, at ﬁrst, sound like a simple division of theory vs. applied work.

Original: However, the focus of the industrial paradigm is more speciﬁc than applied work.
Revised: However, the focus of the industrial paradigm is more speciﬁc than applied work.

Original: It isn’t just about applying machine learning to a real world problem, it’s about applying machine learning to an actual corporate problems, the solution to which results in increased revenue for the company.
Revised: It isn’t just about applying machine learning to a real world problem, it’s about applying machine learning to an actual corporate problems, the solution to which results in increased revenue for the company.

Original: Mnih et al.
Revised: Mnih et al.

Original: (2015) represents an example of applied research, done in an industry lab, but is not within the industrial paradigm.
Revised: (2015) represents an example of applied research, done in an industry lab, but is not within the industrial paradigm.

Original: This also represents the idea that industrial research is not just research that comes from corporate labs and can originate in any lab.
Revised: This also represents the idea that industrial research is not just research that comes from corporate labs and can originate in any lab.

Revised: This paradigm can also help explain machine learning’s atychiphobia (fear of negative results).

Revised: Negative results are important contributions in science, with a number of scientiﬁc journals dedicated purely to them (e.g. the Journal of Negative Results collection).

Revised: However, negative result publications have been dropping (even before the machine learning research boom)

Revised: (Fanelli, 2012).

Revised: Negative results are substantially less important to industrial research (given the focus on commercial solutions); hence, negative machine learning publications are not just less common, but almost unheard of.

Original: While this type of research may occupy a sizeable fraction of publication (even with incentive changes) due to the nature of industry involvement, it is important to consider whether this is what we, as a ﬁeld, want.  
Revised: While this industrial paradigm of research may occupy a sizeable fraction of publication (even with incentive changes) due to the nature of industry involvement, it is important to consider whether this is what we, as a ﬁeld, want.  S OLUTIONS

Revised: We highlight how existing solutions ﬁt within our incentivization model and how they can be expanded further, in addition to providing some of our own solutions.

Original: S OLUTIONS 5. S HORT T ERM S OLUTIONS
Revised: 5. S HORT T ERM S OLUTIONS

Original: What can be done about these problems today?
Revised: What can be done about these problems today?

Original: Institutional, large-scale changes inevitably require signiﬁcant time.
Revised: Institutional, large-scale changes inevitably require signiﬁcant time.

Original: While we want to enact these changes, we also want to increase accessibility in near term research.
Revised: While we want to enact these changes, we also want to increase accessibility in near term research.

Original: There are several steps we can take to increase accessibility for near term research (many of which are already underway).
Revised: There are several steps we can take to increase accessibility for near term research (many of which are already underway).

Original: None of these changes require any modiﬁcation to the current paper workﬂow and will be able to meet the goals better than any workﬂow modiﬁcation.
Revised: None of these changes require any modiﬁcation to the current paper workﬂow and will be able to meet the goals better than any workﬂow modiﬁcation.

Original: One example, that the world was forced into, is online conferences.
Revised: One example, that the world was forced into, is online conferences.

Original: For post-pandemic conferences maintaining hybrid options can enable those without the ability to travel contribute to the research at a conference.
Revised: For post-pandemic conferences maintaining hybrid options can enable those without the ability to travel contribute to the research at a conference.

Original: This supports the globalization of research and underfunded researchers.
Revised: This supports the globalization of research and underfunded researchers.

Original: Another important step is the democratization of computational resources.
Revised: Another important step is the democratization of computational resources.

Original: Even with advancements in computational power and techniques, deep learning research is not very accessible to those lacking resources.
Revised: Even with advancements in computational power and techniques, deep learning research is not very accessible to those lacking resources.

Original: In the entire continent of Africa, there exists only one supercomputer on the Top 500 list (the Toubkal) 4 .
Revised: In the entire continent of Africa, there exists only one supercomputer on the Top 500 list (the Toubkal) 5 .

Original: Government and corporate funding is essential to increase the accessibility of computational resources.
Revised: Government and corporate funding is essential to increase the accessibility of computational resources.

Original: This is might seem like a lofty goal, but it is entirely doable in the immediate future because it doesn’t require any value realignment of funding agencies.
Revised: This is might seem like a lofty goal, but it is entirely doable in the immediate future because it doesn’t require any value realignment of funding agencies.

Original: I.e. we do not have to convince any government or corporation to act more egalitarian, we just have to show how it is in their rational self interest to increase funding to these potentially massive sources of research 5 .
Revised: I.e. we do not have to convince any government or corporation to act more egalitarian, we just have to show how it is in their rational self interest to increase funding to these potentially massive sources of research 6 .

Original: These are just a few ideas; however, these near term steps are not the focus of this paper as they aren’t making any changes to the incentives.
Revised: These are just a few ideas; however, these near term steps are not the focus of this paper as they aren’t making any changes to the incentives.

Original: 5. L ONG T ERM S OLUTIONS
Revised: 5. L ONG T ERM S OLUTIONS

Original: Although the short term solutions are promising, we need substantial changes to machine learning research incentives.
Revised: Although the short term solutions are promising, we need substantial changes to machine learning research incentives.

Original: Here we address each of the misaligned incentives previously discussed and present potential solutions, showing how the paper workﬂow can be used to increase accessibility.
Revised: Here we address each of the misaligned incentives previously discussed and present potential solutions, showing how the paper workﬂow can be used to increase accessibility.

Original: The ﬁrst incentive discussed was the problem of “publish or perish”.
Revised: The ﬁrst incentive discussed was the problem of “publish or perish”.

Original: Speciﬁcally, the expectation is an unreasonably large number of publications that results in more, lower quality, publications.
Revised: Speciﬁcally, the expectation is an unreasonably large number of publications that results in more, lower quality, publications.

Original: It is clear that we need change, and while this is a common sentiment (Bengio, 2020), actual policies are far less common.
Revised: It is clear that we need change, and while this is a common sentiment (Bengio, 2020), actual policies are far less common.

Original: The reason policies are so rare is because most try to address the actual publication cycle, or try to change something about the publication process.
Revised: The reason policies are so rare is because most try to address the actual publication cycle, or try to change something about the publication process.

Original: However, this fails to address the real problem, which isnot that there are so many conferences and venues of publication, but that there is the expectation to publish is so many conferences.
Revised: However, this fails to address the real problem, which is that there is the expectation to publish is so many conferences.

Original: Therefore, the only effective policy will be one that changes this expectation.
Revised: Therefore, the only effective policy will be one that changes this expectation.

Original: One example is a cultural change that has been promoted for some time now: the concept of “Slow Science” (Lutz, 2012), (Stengers, 2018).
Revised: One example is a cultural change that has been promoted for some time now: the concept of “Slow Science” (Lutz, 2012), (Stengers, 2018).

Original: The idea of Slow Science is to encourage cultural shifts towards publishing slower, more impactfulAs Gould put it, “I am, somehow, less interested in publications, which would enable research to ﬂourish when it would otherwise be stiﬂed by the pace of contemporary publications (Aitkenhead, 2013).
Revised: The idea of Slow Science is to encourage cultural shifts towards publishing slower, more impactful publications, which would enable research to ﬂourish when it would otherwise be stiﬂed by the pace of contemporary publications (Aitkenhead, 2013).

Original: One could also consider adjustments to the reward systems for publications.
Revised: One could also consider adjustments to the reward systems for publications.

Original: The number of publications should be signiﬁcantly less considered when determining funding, academic capital, etc.
Revised: The number of publications should be signiﬁcantly less considered when determining funding, academic capital, etc.

Revised: One could also consider a more radical maneuver of greatly reducing or ending the large (non-specialized) ML conferences (NeurIPS,

Revised: AAAI, ICML), thus eliminating the possibility of the unhealthy publication expectations.

Revised: Given the number of papers in these conferences, it is not feasible to engage with (much less read) even a small fraction of the accepted works.

Revised: If these massive conferences just naturally fracture into the different specializations (NLP, RL, CV, theory, etc.)

Original:  These maintain the paper workﬂow, but increase accessibility by changing the incentives such that the number of publications expected is decreased.
Revised: why not just have smaller more niche conferences? This would help to reduce the publication drive (by reducing the number of conferences) and could make the conferences better at achieving their goals of scientiﬁc communication. These maintain the paper workﬂow, but increase accessibility by changing the incentives such that the number of publications expected is decreased.

Original: The second problem is the SotA expectation.
Revised: The second problem is the SotA expectation.

Original: The incentive for SotA-hacking stems from the accepted knowledge that SotA results in publications.
Revised: The incentive for SotA-hacking stems from the accepted knowledge that SotA results in publications.

Original: If you want to get published in big conferences with empirical work, SotA results are expected.
Revised: If you want to get published in big conferences with empirical work, SotA results are expected.

Original: While reducing the publication demand will help with this problem, it is not a complete solution.
Revised: While reducing the publication demand will help with this problem, it is not a complete solution.

Original: In addition, the conference/publication expectations need to change.
Revised: In addition, the conference/publication expectations need to change.

Original: This will likely be much easier to change than the previous incentive, because it is merely a result of the cultural norms rather than any actual regulations.
Revised: This will likely be much easier to change than the previous incentive, because it is merely a result of the cultural norms rather than any actual regulations.

Original: On the speciﬁc problem of
Revised: On the speciﬁc problem of

Original: HARKing, one potential solution is to require hypothesises prior to experiments being conducted, requiring well deﬁned ideas rather than random hyperparameter tuning.
Revised: HARKing, one potential solution is to require hypothesises prior to experiments being conducted, requiring well deﬁned ideas rather than random hyperparameter tuning.

Original: This is something that has already seen some interest 6 .
Revised: This is something that has already seen some interest 7 .

Original: These shifts will result in better pedagogical approaches, as the SotA requirements are relaxed enabling better science to be conducted.
Revised: These shifts will result in better pedagogical approaches, as the SotA requirements are relaxed enabling better science to be conducted.

Original: This will enable more explainable papers, as researcher’s starting point will not longer be SotA results, but proper scientiﬁc investigations, eliminating the need to backﬁll. 
Revised: This will enable more explainable papers, as researcher’s starting point will not longer be SotA results, but proper scientiﬁc investigations, eliminating the need to backﬁll. As Gould

Original: The ﬁnal concern has to do with reviewers.
Revised: The third concern has to do with reviewers.

Original: Speciﬁcally, the incentives for reviewers are broken.
Revised: Speciﬁcally, the incentives for reviewers are broken.

Original: Reviewers expect mathematics (leading to equation packing) and yet do not check or verify the correctness of this math leading to well publicized instances of faulty mathematics (Lipton & Steinhardt, 2018).
Revised: Reviewers expect mathematics (leading to equation packing) and yet do not check or verify the correctness of this math leading to well publicized instances of faulty mathematics (Lipton & Steinhardt, 2018).

Original: Simply put, there is almost no external incentive for reviewers to actually be good reviewers.
Revised: Simply put, there is almost no external incentive for reviewers to actually be good reviewers.

Original: The ﬂaws of the review system has seen a fair amount of (unofﬁcial, i.e. twitter) discourse.
Revised: The ﬂaws of the review system has seen a fair amount of (unofﬁcial, e.g. twitter) discourse.

Original: Some ideas include paying reviewers, creating an anonymous rating system of reviewers, eliminating double blind peer review and shifting towards a continuous arXiv review, etc.
Revised: Some ideas include paying reviewers, creating an anonymous rating system of reviewers, eliminating double blind peer review and shifting towards a continuous arXiv review, etc.

Original: We have little to add to this topic, other than to note that this is an example of solutions that might actually work, because they address the root cause of the problem: the incentives.
Revised: We have little to add to this topic, other than to note that this is an example of solutions that might actually work, because they address the root cause of the problem: the incentives.

Original: By changing review incentives so that papers are valued for their interpretability, not their ability to appear impressive behind a wall of incomprehensible math, will be of substantial beneﬁt (and once again requires no paper process changes). 
Revised: By changing review incentives so that papers are valued for their interpretability, not their ability to appear impressive behind a wall of incomprehensible math, will be of substantial beneﬁt (and once again requires no paper process changes). Finally, we want to brieﬂy address a potential solution to the industrial paradigm.

Revised: This may not universally be seen as a problem, but we believe that it has substantial limitations.

Revised: Common in industry is the practice of patenting algorithmic developments.

Revised: These patents are not conducive to open research.

Revised: Machine learning patents are not patenting the software, but the ideas behind the algorithms.

Original: C ONCLUSION
Revised: Examples include Google patenting “Methods ... for asynchronous deep reinforcement learning ... wherein each worker is conﬁgured to operate independently of each other worker ... each worker is associated with a respective actor that interacts ... the environment during the training of the deep neural network” (Mnih et al., 2021), “Methods ... for training an actor neural network ... methods includes obtaining a minibatch of experience tuples and updating current values of the parameters of the actor neural network ...” (Lillicrap et al., 2020), and Amazon patenting “a system for capturing and processing portions of a spoken utterance command that may occur before a wakeword” (Yasa et al., 2021).

Revised: Patenting machine learning algorithms is representative of industrial research and in opposition with the global and open scientiﬁc initiatives.

Revised: Expanding patent laws to further limit algorithmic patenting may help to curb these practices, something the Supreme Court of the United States has been trending towards.

Revised: In Diamond vs. Diehr (1981) the ruling stated “a mathematical formula, like a law of nature, cannot be the subject of a patent” and SAP America vs.

Revised: Investpic LLC (2018) stated that “mathematical calculations and formulas are not patent eligible”.

Revised: Fundamentally machine learning algorithms are automated mathematical calculations, so legally clarifying this may help to enforce open research practices.

Revised:  C ONCLUSION

Original: Contemporary machine learning research is problematic.
Revised: Contemporary machine learning research is problematic.

Original: The way forward is not through temporary band-aids of publication reconceptualizations, but through thorough institutional and systematic changes.
Revised: The way forward is not through temporary band-aids of publication reconceptualizations, but through thorough institutional and systematic changes.

Original: Fixing these incentives will enable machine learning research to become a more inclusive research space, enabling better pedagogy.
Revised: Fixing these incentives will enable machine learning research to become a more inclusive research space, enabling better pedagogy.

Revised: We recognize that our solutions are limited, and we often highlighted existing techniques; however, we are hopeful that this work will spark further discussion on solutions to be reached as a community (as they ought to be).

Original: The main faulty incentives are that of research production expectations, SotA expectations and reviewer status quo expectations.
Revised: The main faulty incentives are that of research production expectations, SotA expectations and reviewer status quo expectations.

Original: All three of these reduces the accessibility of the paper workﬂow.
Revised: All three of these reduces the accessibility of the paper workﬂow.

Original: However, any other publication form would be subject to the same ﬂaws.
Revised: However, any other publication form would be subject to the same ﬂaws.

Original: We can directly incorporate the goals of inclusivity, explainability, and interpretability within the publication process thus resulting in increased accessibility, all within the paper process.
Revised: We can directly incorporate the goals of inclusivity, explainability, and interpretability within the publication process thus resulting in increased accessibility, all within the paper process.