Original: SPICE S : S URVEY P APERS AS I NTERACTIVE C HEATSHEET E MBEDDINGS
Revised: SPICE S : S URVEY P APERS AS I NTERACTIVE C HEATSHEET E MBEDDINGS

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Papers are hard to write.
Revised: Papers are hard to write.

Original: Survey papers are just that much harder.
Revised: Survey papers are just that much harder.

Original: From the authors’ perspective, challenges include the responsibility to not erase out important work being done by (sometimes) adversarially aligned research groups, ﬁnding the right semantic clustering to sub-categorize individual contributions, controlling for the verbosity and length of the ﬁnal paper, ensuring an optimal mixing of personal opinion and the innate narratives in the paper(s) being cited, version controlling, ease of updating, and also the aesthetics of presentation.
Revised: From the authors’ perspective, challenges include the responsibility to not erase out important work being done by (sometimes) adversarially aligned research groups, ﬁnding the right semantic clustering to sub-categorize individual contributions, controlling for the verbosity and length of the ﬁnal paper, ensuring an optimal mixing of personal opinion and the innate narratives in the paper(s) being cited, version controlling, ease of updating, and also the aesthetics of presentation.

Original: From the reader’s viewpoint, challenges include ease of reading, single-snapshot summarizability, portability, and being given the agency to edit or fork their own copies.
Revised: From the reader’s viewpoint, challenges include ease of reading, single-snapshot summarizability, portability, and being given the agency to edit or fork their own copies.

Original: Taking cues from the emergence of the cheat-sheet culture in machine learning and the virtues of living editable documentation and version control, we propose an interactive and live SVG format based methodology that we term SPICE: Survey Papers as Interactive Cheat-sheet Embedding.
Revised: Taking cues from the emergence of the cheat-sheet culture in machine learning and the virtues of living editable documentation and version control, we propose an interactive and live SVG format based methodology that we term SPICE: Survey Papers as Interactive Cheat-sheet Embedding.

Original: We cover the technical details behind constructing SPICEs and present an example gallery covering ‘hot button’ areas in machine learning such as Out of distribution detection, the ‘All you need’ histrionics and Transformer architectures.
Revised: We cover the technical details behind constructing SPICEs and present an example gallery covering ‘hot button’ areas in machine learning such as Out of distribution detection, the ‘All you need’ histrionics and Transformer architectures.

Original: We have open-sourced all of the code with regards to this project here: https://anonymous.4open.science/ r/7c5ee736-c876-4b90-97bd-49870eb6b63f/ .
Revised: We have open-sourced all of the code with regards to this project here: https://anonymous.4open.science/ r/7c5ee736-c876-4b90-97bd-49870eb6b63f/ .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: We, the denizens of this machine learning world, are all grasping at straws desperately trying to gracefully drink from the ﬁre-hose of publications that ﬂood the arXiv servers every day.
Revised: We, the denizens of this machine learning world, are all grasping at straws desperately trying to gracefully drink from the ﬁre-hose of publications that ﬂood the arXiv servers every day.

Original: The intense democratization of all facets of machine learning research: papers, code, teaching materials and the computing resources (to an extent), has meant that the state-of-the-art is evolving at breakneck speeds and hence, it is natural that the literature landscape might seem chaotic and cacophonous at times.
Revised: The intense democratization of all facets of machine learning research: papers, code, teaching materials and the computing resources (to an extent), has meant that the state-of-the-art is evolving at breakneck speeds and hence, it is natural that the literature landscape might seem chaotic and cacophonous at times.

Original: It is in the backdrop of this observation that we’d like to contextualize and celebrate the utility of well-authored survey papers.
Revised: It is in the backdrop of this observation that we’d like to contextualize and celebrate the utility of well-authored survey papers.

Original: Besides the obvious good of providing a comprehensive bird’s-eye view of the ﬁeld, they serve ﬁve other important roles that are often ignored:
Revised: Besides the obvious good of providing a comprehensive bird’s-eye view of the ﬁeld, they serve ﬁve other important roles that are often ignored:

Original: These are high quality invitation notes to researchers from different domains to contribute.
Revised: These are high quality invitation notes to researchers from different domains to contribute.

Original: They serve as collections of important open problems waiting to be solved.
Revised: They serve as collections of important open problems waiting to be solved.

Original: They are immensely helpful as source materials that enable faster, better, and up-to-date teaching course design.
Revised: They are immensely helpful as source materials that enable faster, better, and up-to-date teaching course design.

Original: They are instrumental in setting the agenda for the research directions in the near future.
Revised: They are instrumental in setting the agenda for the research directions in the near future.

Original: They help ease the burden of lengthy citation lists, especially for short communication papers.
Revised: They help ease the burden of lengthy citation lists, especially for short communication papers.

Revised: A survey paper author

Revised: Their papers My papers

Original: 1. T HE CHALLENGES AND A NEW DISRUPTIVE - MEDIUM
Revised: 1. T HE CHALLENGES AND A NEW DISRUPTIVE - MEDIUM

Original: As one might fathom, it is not easy to author and publish survey papers.
Revised: As one might fathom, it is not easy to author and publish survey papers.

Original: To begin with, it is natural for a researcher to have a stronger leaning towards working on new research rather than comprehensively summarizing the legacy work in the ﬁeld.
Revised: To begin with, it is natural for a researcher to have a stronger leaning towards working on new research rather than comprehensively summarizing the legacy work in the ﬁeld.

Original: While the allure of getting to buff up one’sA survey paper author Their papers My papers citation-counts (and h -index-like metrics) does act of a counter-balancing factor, the sheer volume of labor involved and the unacceptability of such work in mainstream conference venues such as
Revised: While the allure of getting to buff up one’s citation-counts (and h -index-like metrics) does act of a counter-balancing factor, the sheer volume of labor involved and the unacceptability of such work in mainstream conference venues such as

Original: NeurIPS, ICML, ICLR etc.
Revised: NeurIPS, ICML, ICLR etc.

Original: acts as a strong detraction point.
Revised: acts as a strong detraction point.

Original: Also, we opine that gate-keeping constraints such as the demand for a co-author to be a de facto “expert” 1 ) only makes things worse.
Revised: Also, we opine that gate-keeping constraints such as the demand for a co-author to be a de facto “expert” 1 ) only makes things worse.

Original: Survey papers also elicit a uniquely speciﬁc challenge: The challenge of toeing the proverbial pareto-optimal walk between one’s own contributions to the ﬁeld and the contribution of the other researchers and research groups (that are in some cases, adversarially situated).
Revised: Survey papers also elicit a uniquely speciﬁc challenge: The challenge of toeing the proverbial pareto-optimal walk between one’s own contributions to the ﬁeld and the contribution of the other researchers and research groups (that are in some cases, adversarially situated).

Original: This necessitates imbibing a subtle personal tonality on the research ideas being explored whilst being respectful and considerate towards the work of others, an art-form in itself.
Revised: This necessitates imbibing a subtle personal tonality on the research ideas being explored whilst being respectful and considerate towards the work of others, an art-form in itself.

Original: The authors would like to unabashedly admit that this Buridan’s ass-like quandary Weintraub (2012) has killed many of survey paper-writing efforts that they themselves initiated but never ﬁnished (See Figure 1.1).
Revised: The authors would like to unabashedly admit that this Buridan’s ass-like quandary Weintraub (2012) has killed many of survey paper-writing efforts that they themselves initiated but never ﬁnished (See Figure 1.1).

Original: Besides this, there is also the challenge to stitch together a coherent narrative to ensure that the paper does not have the feel of a hastily copy-pasted cacophonous mix of all the paper abstracts available in that area.
Revised: Besides this, there is also the challenge to stitch together a coherent narrative to ensure that the paper does not have the feel of a hastily copy-pasted cacophonous mix of all the paper abstracts available in that area.

Original: This sometimes does lead to what we term as the “monolithic-blob effect” in papers where all the different works covered are propelled into a single lengthy section of the paper as sub-sections without an over-arching narrative that helps categorize and contextualize the sub-sets of ideas that resulted in the constituent papers.
Revised: This sometimes does lead to what we term as the “monolithic-blob effect” in papers where all the different works covered are propelled into a single lengthy section of the paper as sub-sections without an over-arching narrative that helps categorize and contextualize the sub-sets of ideas that resulted in the constituent papers.

Original: As example, we’d like to draw the attention of the reader towards
Revised: As example, we’d like to draw the attention of the reader towards

Original: Figure 5 from Efﬁcient Transformers: A Survey Tay et al.
Revised: Figure 5 from Efﬁcient Transformers: A Survey Tay et al.

Original: (2020) (which we found to be otherwise extremely well-authored), where oddly, a certain Section 3.2 contained a stock-pile of all the efﬁcient Transformer models that had been proposed in recent literature.
Revised: (2020) (which we found to be otherwise extremely well-authored), where oddly, a certain Section 3.2 contained a stock-pile of all the efﬁcient Transformer models that had been proposed in recent literature.

Original: Lastly, there is the need to ensure that the focus is not only to include the hit-papers emanating from the usual-suspect corners of the industry and academia (the FAANG -US news-Top- K -schools nexus and the other “Grey-hoodie miscreants” Abdalla & Abdalla (2020) ).
Revised: Lastly, there is the need to ensure that the focus is not only to include the hit-papers emanating from the usual-suspect corners of the industry and academia (the FAANG -US news-Top- K -schools nexus and the other “Grey-hoodie miscreants” Abdalla & Abdalla (2020) ).

Original: This responsibility also takes speciﬁc importance in the context of being a vanguard against citation erasure that elicits paying strong heed to the “Citational Desires” Murphy (2020) of contributors such as black women scholars whose contributions have been systematically under-emphasized or erased 2 .
Revised: This responsibility also takes speciﬁc importance in the context of being a vanguard against citation erasure that elicits paying strong heed to the “Citational Desires” Murphy (2020) of contributors such as black women scholars whose contributions have been systematically under-emphasized or erased 2 .

Original: 1.1. T HE RISE AND RISE OF CHEAT - SHEET COMPENDIA
Revised: 1.1. T HE RISE AND RISE OF CHEAT - SHEET COMPENDIA

Original: “ First they ignore you, then they laugh at you, then they ﬁght you, then you win ”.
Revised: “ First they ignore you, then they laugh at you, then they ﬁght you, then you win ”.

Original: - Mahatma
Revised: - Mahatma

Original: Gandhi .
Revised: Gandhi .

Original: With these challenges in mind, the increasingly prose-averse machine learning community seems to have gravitated towards a rather intriguing hack in the form of cheat-sheets.
Revised: With these challenges in mind, the increasingly prose-averse machine learning community seems to have gravitated towards a rather intriguing hack in the form of cheat-sheets.

Original: The cheat-sheet titled A mostly complete chart of neural networks van Veen (September); Leijnen & Veen (2020)call- -survey- trended on every social-media platform upon it’s release in September 2016 3 .
Revised: The cheat-sheet titled A mostly complete chart of neural networks van Veen (September); Leijnen & Veen (2020) trended on every social-media platform upon it’s release in September 2016 3 .

Original: Similarly, one of the more popular pages on a certain popular online knowledge portal 4 is literally titled Top 28 Cheat
Revised: Similarly, one of the more popular pages on a certain popular online knowledge portal 4 is literally titled Top 28 Cheat

Original: Sheets for Machine Learning, Data Science, Probability, SQL and Big Data .
Revised: Sheets for Machine Learning, Data Science, Probability, SQL and Big Data .

Original: The GitHub repository, awesome-machine-learning has over 49,200 stars and 12000 forks 5 .
Revised: The GitHub repository, awesome-machine-learning has over 49,200 stars and 12000 forks 5 .

Original: The Awesome
Revised: The Awesome

Original: Pytorch list 6 has 11.2k stars and 2.5k forks.
Revised: Pytorch list 6 has 11.2k stars and 2.5k forks.

Original: Stanford’s CS 229 - Machine Learning tips and tricks cheat-sheet is similarly well known and has also been translated into dozens of languages 7 .
Revised: Stanford’s CS 229 - Machine Learning tips and tricks cheat-sheet is similarly well known and has also been translated into dozens of languages 7 .

Original: Robbie
Revised: Robbie

Original: Allen’s compendium titled Cheat Sheet of Machine Learning and Python (and Math)
Revised: Allen’s compendium titled Cheat Sheet of Machine Learning and Python (and Math)

Original: Cheat Sheetsattracted 7.3K “claps”!
Revised: Cheat Sheetsattracted 7.3K “claps”!

Original: While it is easy to don the cloak of academic snobbery and pooh-pooh away these developments as “shortcut-hacks”, we implore the academic priesthood of the high “cloisters” to pay heed to this development from the humble denizens of the “bazaar” (See The cathedral and the bazaar Raymond (1999) ).
Revised: While it is easy to don the cloak of academic snobbery and pooh-pooh away these developments as “shortcut-hacks”, we implore the academic priesthood of the high “cloisters” to pay heed to this development from the humble denizens of the “bazaar” (See The cathedral and the bazaar Raymond (1999) ).

Original: Belonging squarely to the “bazaar” camp, we champion the cause of this cheat-sheet phenomenon, and propose a template versioning of the cheat-sheets that we term
Revised: Belonging squarely to the “bazaar” camp, we champion the cause of this cheat-sheet phenomenon, and propose a template versioning of the cheat-sheets that we term

Original: SPICEs and motivate its utility in the context of authoring and sharing survey papers.
Revised: SPICEs and motivate its utility in the context of authoring and sharing survey papers.

Original: The rest of the paper is organized as follows.
Revised: The rest of the paper is organized as follows.

Original: In Section 2, we cover the related work from the viewpoint of the WISE framework.
Revised: In Section 2, we cover the related work from the viewpoint of the WISE framework.

Original: In Section 3, we present the procedure to construct the SVGs.
Revised: In Section 3, we present the procedure to construct the SVGs.

Original: In
Revised: In

Original: Section 4, we present the SPICE gallery with some examples and conclude the paper in Section 5.
Revised: Section 4, we present the SPICE gallery with some examples and conclude the paper in Section 5.

Original:  A BRIEF OVERVIEW OF RELATED WORK
Revised:  A BRIEF OVERVIEW OF RELATED WORK

Original: There have been several high proﬁle calls for revisiting the current publication model targeting both the authorship-and-reviewing facets as well the formatting facet.
Revised: There have been several high proﬁle calls for revisiting the current publication model targeting both the authorship-and-reviewing facets as well the formatting facet.

Original: Authorship-and-reviewing facets : With speciﬁc emphasis on challenging the review-process and the publish-or-perish culture, Parnas’ 2007 call to “Stop the numbers game” Parnas (2007) begins with the stark warning that “counting papers slows the rate of scientiﬁc progress”.
Revised: Authorship-and-reviewing facets : With speciﬁc emphasis on challenging the review-process and the publish-or-perish culture, Parnas’ 2007 call to “Stop the numbers game” Parnas (2007) begins with the stark warning that “counting papers slows the rate of scientiﬁc progress”.

Original: Yoshua Bengio’s more recent monologue Time to rethink the publication process in machine learning Bengio (2020) imbibes a call to embrace the “Slow science manifesto” and claimed that “... the race to put out more papers is humanly crushing” while advocating for the journal-conference hybrid model.
Revised: Yoshua Bengio’s more recent monologue Time to rethink the publication process in machine learning Bengio (2020) imbibes a call to embrace the “Slow science manifesto” and claimed that “... the race to put out more papers is humanly crushing” while advocating for the journal-conference hybrid model.

Original: Formatting facets : One of the central gripes of anti-pdf-paper-ness lies the non-intersecting worlds of static text and dynamic software.
Revised: Formatting facets : One of the central gripes of anti-pdf-paper-ness lies the non-intersecting worlds of static text and dynamic software.

Original: In experimentation-heavy ﬁelds such as Deep learning, this can be incredibly frustrating.
Revised: In experimentation-heavy ﬁelds such as Deep learning, this can be incredibly frustrating.

Original: With the goal to bridge the worlds of interactive data, dynamic software and the static text, Mathematica notebooks 9 emerged on the scene, fueling works such as
Revised: With the goal to bridge the worlds of interactive data, dynamic software and the static text, Mathematica notebooks 9 emerged on the scene, fueling works such as

Original: A New Kind of Science Wolfram (2002).
Revised: A New Kind of Science Wolfram (2002).

Original: Seeking to break-free from the proprietary constraints of Mathematica notebooks, the IPython notebook revolution unfolded Shen (2014).
Revised: Seeking to break-free from the proprietary constraints of Mathematica notebooks, the IPython notebook revolution unfolded Shen (2014).

Original: It is indeed noteworthy that the dissemination of the ﬁrst ever conﬁrmed detection of gravitational waves was made with supplemental IPython notebook 10 .
Revised: It is indeed noteworthy that the dissemination of the ﬁrst ever conﬁrmed detection of gravitational waves was made with supplemental IPython notebook 10 .

Original: In 2014, the IPython-notebook morphed into the
Revised: In 2014, the IPython-notebook morphed into the

Original: Project Jupyter Kluyver et al.
Revised: Project Jupyter Kluyver et al.

Original: (2016) that now supports over 100 programming languages and has an entire ecosystem of support built around it to auto-generate PDFs ( The notebook-as-pdf package)
Revised: (2016) that now supports over 100 programming languages and has an entire ecosystem of support built around it to auto-generate PDFs ( The notebook-as-pdf package)

Original: https://pypi.org/project/notebook-as-pdf/ or blog-posts (Ex: Jupyterto-medium https://pypi.org/project/jupyter-to-medium/ or Fastpages  https://fastpages.fast.ai/ ).
Revised: https://pypi.org/project/notebook-as-pdf/ or blog-posts (Ex: Jupyterto-medium https:/ pypi.org/project/jupyter-to-medium/ or Fastpages net https://fastpages.fast.ai/ ).

Original: We’d like to encourage the reader to sift through the article on The science paper is obsolete Somers (2018) for a more ﬁne-grained journey of this timeline.
Revised: We’d like to encourage the reader to sift through the article on The science paper is obsolete Somers (2018) for a more ﬁne-grained journey of this timeline.

Original: Recently, Stanford hosted an entire workshop titled Scientiﬁc Publication Beyond the Text: Sharing
Revised: Recently, Stanford hosted an entire workshop titled Scientiﬁc Publication Beyond the Text: Sharing

Original: Research Objects 11 that showcased non-paper dissemination such as live-sites, ORBIS -The
Revised: Research Objects 11 that showcased non-paper dissemination such as live-sites, ORBIS -The

Original: Stan- net ford Geospatial Network Model of the Roman World 12 , PanGeo Hamman et al.
Revised: Stanford Geospatial Network Model of the Roman World 12 , PanGeo Hamman et al.

Original: (2018), Binder Forde et al.
Revised: (2018), Binder Forde et al.

Original: (2018), Registered Reports Chambers (2013) and OpenfMRI 13 emanating from different disciplines such as Geo-data analysis, psychology and the Life sciences.
Revised: (2018), Registered Reports Chambers (2013) and OpenfMRI 13 emanating from different disciplines such as Geo-data analysis, psychology and the Life sciences.

Original: (The associated Video-playlist can be found here 14 ).
Revised: (The associated Video-playlist can be found here 14 ).

Original: With regards to the machine learning community, we’d like to highlight the following important works.
Revised: With regards to the machine learning community, we’d like to highlight the following important works.

Original: Firstly, we have the Codalab Worksheets initiative 15 designed to “Run reproducible experiments and create executable papers”.
Revised: Firstly, we have the Codalab Worksheets initiative 15 designed to “Run reproducible experiments and create executable papers”.

Original: Secondly, given the “metrics chasing” fetish that has besieged the ﬁeld, Papers-with-code 16 emerges as an invaluable community resource that hosts “Machine
Revised: Secondly, given the “metrics chasing” fetish that has besieged the ﬁeld, Papers-with-code 16 emerges as an invaluable community resource that hosts “Machine

Original: Learning papers, code and evaluation tables” and helps keep track of what is SoTA and what is not in the top-k accuracy hunting rat-race.
Revised: Learning papers, code and evaluation tables” and helps keep track of what is SoTA and what is not in the top-k accuracy hunting rat-race.

Original: Thirdly, with the emergence of document2vec frameworks and semanticity-preserving dimensionality reduction algorithms such as tSNE and UMAP, we have seen interactive web-pages providing a bird’s-eye view of conference proceedings come to fruition.
Revised: Thirdly, with the emergence of document2vec frameworks and semanticity-preserving dimensionality reduction algorithms such as tSNE and UMAP, we have seen interactive web-pages providing a bird’s-eye view of conference proceedings come to fruition.

Original: A brilliant example of this can be seen in Fig A which is from the ICLR paper explorer Rush &
Revised: A brilliant example of this can be seen in Fig A which is from the ICLR paper explorer Rush &

Original: Strobelt (2020) project (See Fig A for example).
Revised: Strobelt (2020) project (See Fig A for example).

Original: Lastly, we’d like to mention other efforts such as
Revised: Lastly, we’d like to mention other efforts such as

Original: DLPaper2Code Sethi et al.
Revised: DLPaper2Code Sethi et al.

Original: (2018), Multimodal Knowledge Graph for Deep Learning Papers-andCode Akrotirianakis et al.
Revised: (2018), Multimodal Knowledge Graph for Deep Learning Papers-andCode Akrotirianakis et al.

Original: (2018) and SOLE Pham et al.
Revised: (2018) and SOLE Pham et al.

Original: (2012) that offered a novel solution for linking research papers with science objects.
Revised: (2012) that offered a novel solution for linking research papers with science objects.

Original: Lastly, the web-page of Rethinking ML Papers has a survey of other workﬂows such as Distill.pub Odena et al.
Revised: Lastly, the web-page of Rethinking ML Papers has a survey of other workﬂows such as Distill.pub Odena et al.

Original: (2016) and Hypothesis
Revised: (2016) and Hypothesis

Original:  M ETHODOLOGY
Revised:  M ETHODOLOGY

Original: Inspired by the above body of work, we formulated a simple Google-slides-SVG export methodology whose details we present in this section.
Revised: Inspired by the above body of work, we formulated a simple Google-slides-SVG export methodology whose details we present in this section.

Original: The rest of this section describes a workﬂow used to generate the eventual deliverable: A hosted hyperlinked SVG.
Revised: The rest of this section describes a workﬂow used to generate the eventual deliverable: A hosted hyperlinked SVG.

Original: Where possible, we also present alternatives to some steps.
Revised: Where possible, we also present alternatives to some steps.

Original: 3. D ESIGNING THE D IAGRAM
Revised: 3. D ESIGNING THE D IAGRAM

Original: The taxonomy does not need to be designed from the start in Google Slides.
Revised: The taxonomy does not need to be designed from the start in Google Slides.

Original: The information may originate in Google Slides 18 , or it may be copied from a draft.
Revised: The information may originate in Google Slides 18 , or it may be copied from a draft.

Original: The draft may be designed on paper, or in another program like Roam Research 19 , draw.io 20 , or yEd 21 .
Revised: The draft may be designed on paper, or in another program like Roam Research 19 , draw.io 20 , or yEd 21 .

Original: Ultimately, the information that the ﬁnal SVG retains will need to be present in the Google Slides slide.
Revised: Ultimately, the information that the ﬁnal SVG retains will need to be present in the Google Slides slide.

Original: 3. G OOGLE S LIDES ( OR OTHER P RESENTATION P ROGRAMS )
Revised: 3. G OOGLE S LIDES ( OR OTHER P RESENTATION P ROGRAMS )

Original: Once the diagram is laid out, it can be assembled in Google Slides.
Revised: Once the diagram is laid out, it can be assembled in Google Slides.

Original: Other programs with similar functionality (e.g., Microsoft PowerPoint, Apple’s Keynote) can be useful as a substitute for this step, though this workﬂow has been veriﬁed using Google Slides.
Revised: Other programs with similar functionality (e.g., Microsoft PowerPoint, Apple’s Keynote) can be useful as a substitute for this step, though this workﬂow has been veriﬁed using Google Slides.

Original: Google slides has the advantage of being shareable among researchers working remotely, regardless of local operating systems or even browsers.
Revised: Google slides has the advantage of being shareable among researchers working remotely, regardless of local operating systems or even browsers.

Original: While it is true that other design programs like Figma 22 also share these properties, we believe most engineers not typically involved in front-end engineering will be more familiar with
Revised: While it is true that other design programs like Figma 22 also share these properties, we believe most engineers not typically involved in front-end engineering will be more familiar with

Original: Google Slides.
Revised: Google Slides.

Original: For creating a hyperlinked diagram in Google slides one can use either a default diagram (e.g., through selecting Insert > Diagram in the top menu) or assemble it from scratch using shapes, lines, images, or text boxes (all of which can be selected from the Insert section of the top menu, or from the menu below).
Revised: For creating a hyperlinked diagram in Google slides one can use either a default diagram (e.g., through selecting Insert > Diagram in the top menu) or assemble it from scratch using shapes, lines, images, or text boxes (all of which can be selected from the Insert section of the top menu, or from the menu below).

Original: 12 https://orbis.stanford.edu/https://reproducibility stanford.edu/https://bit.ly/3t6I0WQhttps://worksheets.codalab.org/https://paperswithcode.com/

Original: See https://rethinkingmlpapers.github.io/submit/https://www.google.com/slides/about/https://roamresearch.com/ 20 https://draw.io/ 21 https://www.yworks.com/products/yedhttps://www.figma.com/

Original: When creating diagrams in Google slides, there are a few additional features of the UI to be mindful of. 1)
Revised: When creating diagrams in Google slides, there are a few additional features of the UI to be mindful of. 1)

Original: 3. A PPLICABLE D ESIGN P RINCIPLES
Revised: 3. A PPLICABLE D ESIGN P RINCIPLES

Original: Before exporting the slide to SVG format, we recommend a few design principles making the output both navigable and aesthetically pleasing.
Revised: Before exporting the slide to SVG format, we recommend a few design principles making the output both navigable and aesthetically pleasing.

Original: These have been selected both by practical value, as well as their extolling elsewhere by designers such as Lima (2019).
Revised: These have been selected both by practical value, as well as their extolling elsewhere by designers such as Lima (2019).

Original: Give every design choice a purpose To communicate you must reduce cognitive load on the part of the viewer and emphasize what matters.
Revised: Give every design choice a purpose To communicate you must reduce cognitive load on the part of the viewer and emphasize what matters.

Original: Every action, color, and visual element should support data insights and understanding.
Revised: Every action, color, and visual element should support data insights and understanding.

Original: Ideally, one should maintain a consistent style for conveying types of information (even better if you can provide a legend or key in your diagram).
Revised: Ideally, one should maintain a consistent style for conveying types of information (even better if you can provide a legend or key in your diagram).

Original: At a high level, consider whether you want the diagram to convey to the user a hierarchy, data orientation, schema, or relationships.
Revised: At a high level, consider whether you want the diagram to convey to the user a hierarchy, data orientation, schema, or relationships.

Original: Even with the SVG being hosted digitally, minimize the ink one would use if they had to print it out (i.e., maximize the data-ink ratio).
Revised: Even with the SVG being hosted digitally, minimize the ink one would use if they had to print it out (i.e., maximize the data-ink ratio).

Original: When creating shapes or the diagram, keep in mind that any feature of the shape can be used to convey information or lack thereof.
Revised: When creating shapes or the diagram, keep in mind that any feature of the shape can be used to convey information or lack thereof.

Original: This extends to labels, highlights, line connections, groupings of multiple shapes, or even the exact measurements of the shapes (it’s good to make a habit of copying and pasting text and shapes if you want to conserve information among data subsets).
Revised: This extends to labels, highlights, line connections, groupings of multiple shapes, or even the exact measurements of the shapes (it’s good to make a habit of copying and pasting text and shapes if you want to conserve information among data subsets).

Original: The same principles for designing a consistent map legend apply here, including the design of inconsistent parts.
Revised: The same principles for designing a consistent map legend apply here, including the design of inconsistent parts.

Original: If you want to emphasize some part of the diagram with a design change, the strength of the emphasis will be inversely proportional to how common that design style is in the rest of the diagram.
Revised: If you want to emphasize some part of the diagram with a design change, the strength of the emphasis will be inversely proportional to how common that design style is in the rest of the diagram.

Original: Since the motivation behind using an SVG is partly the use of hyperlinks, one shouldn’t neglect the style of the links.
Revised: Since the motivation behind using an SVG is partly the use of hyperlinks, one shouldn’t neglect the style of the links.

Original: When adding hyperlinks to text in the diagram, you can also remove the default underline and change the default color.
Revised: When adding hyperlinks to text in the diagram, you can also remove the default underline and change the default color.

Original: That said, make sure that the hyperlinked text follows a consistent color/font so that viewers can recognize where to point.
Revised: That said, make sure that the hyperlinked text follows a consistent color/font so that viewers can recognize where to point.

Original: Make use of color-coding while compensating for its limitations Color-coding parts of the diagram, either for selective emphasis or as part of a consistent labelling schema, can be effective.
Revised: Make use of color-coding while compensating for its limitations Color-coding parts of the diagram, either for selective emphasis or as part of a consistent labelling schema, can be effective.

Original: However, one should not rely solely on color changes as it has its limitations.
Revised: However, one should not rely solely on color changes as it has its limitations.

Original: The ﬁrst major limitations is the number of possible contrasting color combinations.
Revised: The ﬁrst major limitations is the number of possible contrasting color combinations.

Original: Anyone that’s created a scatter-plot knows the challenge of plotting more data categories than there are easily discernible colors.
Revised: Anyone that’s created a scatter-plot knows the challenge of plotting more data categories than there are easily discernible colors.

Original: Richardson et al.
Revised: Richardson et al.

Original: (2014) provides some further examples of contrasting color combinations, including appropriate backgrounds.
Revised: (2014) provides some further examples of contrasting color combinations, including appropriate backgrounds.

Original: The Color Usage Research Lab at NASA Ames provides additional guidelines on color selection ( https://colorusage.arc.nasa.gov/ discrim.php ).
Revised: The Color Usage Research Lab at NASA Ames provides additional guidelines on color selection ( https://colorusage.arc.nasa.gov/ discrim.php ).

Original: The second major limitation is the prevalence of color vision deﬁciency.
Revised: The second major limitation is the prevalence of color vision deﬁciency.

Original: Large random population surveys such as Birch (2012) show that the prevalence of red-green deﬁciency in European
Revised: Large random population surveys such as Birch (2012) show that the prevalence of red-green deﬁciency in European

Original: Caucasians is about 8% in men, and about 0 .
Revised: Caucasians is about 8% in men, and about 0 .

Original: 4% in women and between 4% and 6 .
Revised: 4% in women and between 4% and 6 .

Original: 5% in men of
Revised: 5% in men of

Original: Chinese and Japanese ethnicity.
Revised: Chinese and Japanese ethnicity.

Original: If part of the intended viewing audience is color vision deﬁcient, similar contrast effects can be achieved in greyscale.
Revised: If part of the intended viewing audience is color vision deﬁcient, similar contrast effects can be achieved in greyscale.

Original: Like with the emphasis principles, keep in mind that some parts of the audience may be color vision deﬁcient.
Revised: Like with the emphasis principles, keep in mind that some parts of the audience may be color vision deﬁcient.

Original: Using features such as shape or font or contrasting shades of grey can get around this.
Revised: Using features such as shape or font or contrasting shades of grey can get around this.

Original: Mind where the SVG is being hosted Bear in mind the intended audience or hosting place of the ﬁnal product.
Revised: Mind where the SVG is being hosted Bear in mind the intended audience or hosting place of the ﬁnal product.

Original: Whether the ﬁnal SVG is going to be a standalone page or embedded within a larger website or e-book should inform limits on some design choices.
Revised: Whether the ﬁnal SVG is going to be a standalone page or embedded within a larger website or e-book should inform limits on some design choices.

Original: For example, if the SVG is going to be embedded within a larger web page, the chosen font will not automatically scale to the text of the web page.
Revised: For example, if the SVG is going to be embedded within a larger web page, the chosen font will not automatically scale to the text of the web page.

Original: Setting a lower limit on text size (such as 16pt or 18pt ) can prevent viewers from getting lost.
Revised: Setting a lower limit on text size (such as 16pt or 18pt ) can prevent viewers from getting lost.

Original: Of course, one should be mindful that any hyperlinks will not be conserved in printed media.
Revised: Of course, one should be mindful that any hyperlinks will not be conserved in printed media.

Original: If the links are truly necessary and the desired printed format is necessary, we recommend redesigning the diagrams to incorporate QR codes instead of hyperlinks.
Revised: If the links are truly necessary and the desired printed format is necessary, we recommend redesigning the diagrams to incorporate QR codes instead of hyperlinks.

Original: URLs, v-cards, social media information, emails, SMS links, images, documents, sound ﬁles, and app stores can all be embedded in a QR code.
Revised: URLs, v-cards, social media information, emails, SMS links, images, documents, sound ﬁles, and app stores can all be embedded in a QR code.

Original: Multiple tools exist for generating QR codes 23 .
Revised: Multiple tools exist for generating QR codes 23 .

Original: Displaying standalone urls in a diagram in lieu of QR codes is acceptable, as long as the url is both short and memorable.
Revised: Displaying standalone urls in a diagram in lieu of QR codes is acceptable, as long as the url is both short and memorable.

Original: QR codes are preferable if short domain names haven’t been secured or if there is an abundance of links.
Revised: QR codes are preferable if short domain names haven’t been secured or if there is an abundance of links.

Original: 3. E XPORTING THE SVG
Revised: 3. E XPORTING THE SVG

Original: When you no longer want to make changes to your diagram, you can export the SVG.
Revised: When you no longer want to make changes to your diagram, you can export the SVG.

Original: If you’re using Google Slides, go to File > Download > Scalable Vector Graphics (.svg, current slide) in the top dropdown menu.
Revised: If you’re using Google Slides, go to File > Download > Scalable Vector Graphics (.svg, current slide) in the top dropdown menu.

Original: If any images are included in the Google Slide, make sure that they’re included in the same directory as the SVG, wherever it is hosted.
Revised: If any images are included in the Google Slide, make sure that they’re included in the same directory as the SVG, wherever it is hosted.

Original: 3. H OSTING THE SVG
Revised: 3. H OSTING THE SVG

Original: Once the SVG is exported, it can be hosted on a web page or blog, or even embedded in a PDF paper.
Revised: Once the SVG is exported, it can be hosted on a web page or blog, or even embedded in a PDF paper.

Original: Basic, Temporary Hosting
Revised: Basic, Temporary Hosting

Original: If the SVG is being for a temporary presentation, it can take up the entirety of the web page.
Revised: If the SVG is being for a temporary presentation, it can take up the entirety of the web page.

Original: The advantage this can be cheaply hosted with single-line HTML ﬁle, as well as a free hosting service (such as GitHub Pages).
Revised: The advantage this can be cheaply hosted with single-line HTML ﬁle, as well as a free hosting service (such as GitHub Pages).

Original: For embeding the hyperlinked SVG in
Revised: For embeding the hyperlinked SVG in

Original: HTML, one can create an object tag with a "image/svg+xml" type as follows: 24 :
Revised: HTML, one can create an object tag with a "image/svg+xml" type as follows: 24 :

Original: Listing 1: Single-line index.html ﬁle for fast [Equation] data [Equation]
Revised: Listing 1: Single-line index.html ﬁle for fast < object type =” image / svg+xml” data =” SVG filename .

Original:  > < [Equation] >
Revised: svg ” > < / object >

Original: The reason for this choice is because SVG is XML based and not HTML based.
Revised: The reason for this choice is because SVG is XML based and not HTML based.

Original: You can’t use a normal <a> or <image> tag outside of the SVG ﬁle itself, and instead have to include links using
Revised: You can’t use a normal <a> or <image> tag outside of the SVG ﬁle itself, and instead have to include links using

Original: Xlink.
Revised: Xlink.

Original: If you want to change links within the SVG ﬁle itself, anchor tags are usable (though xlink:href should be used in place of href , as deﬁned in DeRose et al.
Revised: If you want to change links within the SVG ﬁle itself, anchor tags are usable (though xlink:href should be used in place of href , as deﬁned in DeRose et al.

Original: You can do this if you want to link to pages or images that are no longer hosted in the same directory as the SVG itself.
Revised: You can do this if you want to link to pages or images that are no longer hosted in the same directory as the SVG itself.

Original: Long-term, link-rot-resistant Hosting One of the requirements of multimedia conference presentations is that the material be accessible for a long period of time.
Revised: Long-term, link-rot-resistant Hosting One of the requirements of multimedia conference presentations is that the material be accessible for a long period of time.

Original: Contrary to popular belief, content posted to the internet is not eternal.
Revised: Contrary to popular belief, content posted to the internet is not eternal.

Original: Much of the internet from the 1990s and early 2000s is no longer accessible (a phenomenon sometimes referred to as link-rot, such as in works like Gomes & Silva (2006)).
Revised: Much of the internet from the 1990s and early 2000s is no longer accessible (a phenomenon sometimes referred to as link-rot, such as in works like Gomes & Silva (2006)).

Original: While tools like the Wayback Machine 25 are extremely useful, its pages are often slow to load, missing sizeable amounts of content like images, and not discoverable through many search engines.
Revised: While tools like the Wayback Machine 25 are extremely useful, its pages are often slow to load, missing sizeable amounts of content like images, and not discoverable through many search engines.

Original: If the SVG is to be stored on a blog, or within a website created with a more advanced framework, there needs to be a way of storing it beyond the horizon of 10 years beyond which most content struggles to persist.
Revised: If the SVG is to be stored on a blog, or within a website created with a more advanced framework, there needs to be a way of storing it beyond the horizon of 10 years beyond which most content struggles to persist.

Original: Given the long-term storage requirements, we propose a two-part framework for long-term hosting:
Revised: Given the long-term storage requirements, we propose a two-part framework for long-term hosting:

Original: Rendering the text and multimedia may be done using any technology.
Revised: Rendering the text and multimedia may be done using any technology.

Original: This may be a static site generator like Gatsby.js, the JAM stack, or a custom CMS.
Revised: This may be a static site generator like Gatsby.js, the JAM stack, or a custom CMS.

Original: The text and multimedia must be stored as simply as possible.
Revised: The text and multimedia must be stored as simply as possible.

Original: The SVG format solves part of this problem, but this extends to using a tool like Markdown and Git for storing everything else (even if it’s just the object tag that references the SVG ﬁle).
Revised: The SVG format solves part of this problem, but this extends to using a tool like Markdown and Git for storing everything else (even if it’s just the object tag that references the SVG ﬁle).

Original: As long as the SVGs and associated content are stored on a long-lived, erosion-resistant host, one can use any technology they please for the front-end 26 .
Revised: As long as the SVGs and associated content are stored on a long-lived, erosion-resistant host, one can use any technology they please for the front-end 26 .

Original: Most Markdown-rendering tools should be able to use the object-tag-based embedding described in Listing 1.
Revised: Most Markdown-rendering tools should be able to use the object-tag-based embedding described in Listing 1.

Original:  SPICE GALLERY
Revised:  SPICE GALLERY

Original: In this section, we present a gallery of examples that we constructed using the technique detailed in Section-3.
Revised: In this section, we present a gallery of examples that we constructed using the technique detailed in Section-3.

Original: In doing so, we want to highlight two issues.
Revised: In doing so, we want to highlight two issues.

Original: The ﬁrst is the utility of the svg latex package 27 built with a goal to easily include and extract SVG pictures in LaTeX documents.
Revised: The ﬁrst is the utility of the svg latex package 27 built with a goal to easily include and extract SVG pictures in LaTeX documents.

Original: The second is the breadth of the topic-space that is being surveyed and SPICEd up.
Revised: The second is the breadth of the topic-space that is being surveyed and SPICEd up.

Original: To this end, we pick one example each covering : A research topic (OOD failures), A model survey (Transformer models from Tay et al.
Revised: To this end, we pick one example each covering : A research topic (OOD failures), A model survey (Transformer models from Tay et al.

Original: and the wild garden of X is all you need papers that have emerged in past few years.
Revised: and the wild garden of X is all you need papers that have emerged in past few years.

Original: To begin with, in Figure 2, we address our presentation addressing the previously highlighted stockpiling of all the efﬁcient Transformer architectures in Tay et al.
Revised: To begin with, in Figure 2, we address our presentation addressing the previously highlighted stockpiling of all the efﬁcient Transformer architectures in Tay et al.

Original: (2020) by means of a SPICEyrendition of the same.
Revised: (2020) by means of a SPICEyrendition of the same.

Original: Secondly, in Figure 3, we present a landscape of the ideas that populate the Out-of-distribution-detection corner of the machine learning world.
Revised: Secondly, in Figure 3, we present a landscape of the ideas that populate the Out-of-distribution-detection corner of the machine learning world.

Original: The success enjoyed by the
Revised: The success enjoyed by the

Original: Attention is all you need paper Vaswani et al.
Revised: Attention is all you need paper Vaswani et al.

Original: (2017) spurred an avalanche of papers whose title or content banked off of the X is all you need quip.
Revised: (2017) spurred an avalanche of papers whose title or content banked off of the X is all you need quip.

Original: In Figure 6, we present a SPICEd version of the survey of all these papers that allows the reader to summarize where and why these quips occur.
Revised: In Figure 6, we present a SPICEd version of the survey of all these papers that allows the reader to summarize where and why these quips occur.

Original:  C ONCLUSIONS AND FUTURE WORK
Revised:  C ONCLUSIONS AND FUTURE WORK

Original: The institution of the scientiﬁc paper has received a lot of challenges in recent times, especially in high ﬂux research areas such as machine learning.
Revised: The institution of the scientiﬁc paper has received a lot of challenges in recent times, especially in high ﬂux research areas such as machine learning.

Original: Researchers and community thought-leaders are cleverly harnessing non-traditional media such as  podcasts, YouTube,Instagram, Twitch streams, and apps such as Clubhouse and TikTok, to get their message across.
Revised: Researchers and community contributors alike are cleverly harnessing non-traditional media such as Twitter handles 28 ,Instagram 29 , podcasts, YouTube, Twitch streams, and apps such as Clubhouse and TikTok, to get their message across.

Original: We feel it is inevitable that there will soon be an infusion of NFT(Non-Fungible Token)-like ideas thrown into the mix, especially given ﬂag-posting or anti-scooping motivations.
Revised: We feel it is inevitable that there will soon be an infusion of NFT(Non-Fungible Token)-like ideas thrown into the mix, especially given ﬂag-posting or anti-scooping motivations.

Original: Survey papers, speciﬁcally, are facing a unique set of challenges.
Revised: Survey papers, speciﬁcally, are facing a unique set of challenges.

Original: To begin with, they straddle this grey-zone between knowledge and wisdom in the DIKW pyramid Frick ´ e (2019).
Revised: To begin with, they straddle this grey-zone between knowledge and wisdom in the DIKW pyramid Frick´e (2019).

Original: Too much personal opinion, and you have a parochial preachy wisdom-heavy rendering that might be a disservice for someone seeking a broad-based lay-of-the-land perspective of the topic being surveyed.
Revised: Too much personal opinion, and you have a parochial preachy wisdom-heavy rendering that might be a disservice for someone seeking a broad-based lay-of-the-land perspective of the topic being surveyed.

Original: Similarly, mere copy-pasting and google-scholar-parsing can turn it into just a citation-mining knowledgeconsolidation device.
Revised: Similarly, mere copy-pasting and google-scholar-parsing can turn it into just a citation-mining knowledgeconsolidation device.

Original: Inspired by the emergence of cheat-sheets, we present a framework that we term SPICE (Survey Papers as Interactive Cheat-sheet embedding).
Revised: Inspired by the emergence of cheat-sheets, we present a framework that we term SPICE (Survey Papers as Interactive Cheat-sheet embedding).

Original: In this paper, we have covered the technical know-how of quickly creating SPICEs and also presented an example SPICE-gallery .
Revised: In this paper, we have covered the technical know-how of quickly creating SPICEs and also presented an example SPICE-gallery .

Original: This is a work in progress and we anticipate reﬁning and improving this framework in multiple ways based on community feedback.
Revised: This is a work in progress and we anticipate reﬁning and improving this framework in multiple ways based on community feedback.

Original:  T HE W HYNESS OF THE SVG: A N ACCESSIBILITY STATEMENT
Revised:  T HE W HYNESS OF THE SVG: A N ACCESSIBILITY STATEMENT

Original: As stated in Gardner & Bulatov (2008), the inclusion of ” title and description attributes in the SVG speciﬁcation along with other features promoting accessibility make SVG a nearly ideal language for creating accessible graphics ”.
Revised: As stated in Gardner & Bulatov (2008), the inclusion of ” title and description attributes in the SVG speciﬁcation along with other features promoting accessibility make SVG a nearly ideal language for creating accessible graphics ”.

Original: Also, the SVG format has a slew of accessibility features with regards to soniﬁcation, braille displays and haptic feedback screens built in that made it an attractive proposition for us to adopt.
Revised: Also, the SVG format has a slew of accessibility features with regards to soniﬁcation, braille displays and haptic feedback screens built in that made it an attractive proposition for us to adopt.

Original: The W3C Note 7( Charles McCathieNevile (2000)) speciﬁcally states: ”SVG offers a number of features to make graphics on the Web more accessible than is currently possible, to a wider group of users.
Revised: The W3C Note 7( Charles McCathieNevile (2000)) speciﬁcally states: ”SVG offers a number of features to make graphics on the Web more accessible than is currently possible, to a wider group of users.

Original: Users who beneﬁt include users with low vision, color vision deﬁcient or sight impaired users, and users of assistive technologies.” .
Revised: Users who beneﬁt include users with low vision, color vision deﬁcient or sight impaired users, and users of assistive technologies.” .

Original: Further, we are using the guidelines from the Tree of graphic section in the ”SVG Accessibility/People and Issues” sectionand formatting the surveys using the following order of : title, axes, legends, data and reference elements adhering to the National Center for Accessible Media (NCAM) suggestions.
Revised: Further, we are using the guidelines from the Tree of graphic section in the ”SVG Accessibility/People and Issues” sectionand formatting the surveys using the following order of : title, axes, legends, data and reference elements adhering to the National Center for Accessible Media (NCAM) suggestions.

Original: Many of these recommendations were taken into consideration during our design process (See section-3 3.3).
Revised: Many of these recommendations were taken into consideration during our design process (See section-3 3.3).

Original: Further, with regards to our initial draft, we have received feedback on adding aural tones to the individual data elements (so the reader can hear when they mouse over the data element) which we plan to implement during our demo.
Revised: Further, with regards to our initial draft we have received feedback on adding aural tones to the individual data elements (so the reader can hear when they mouse over the data element) which we plan to implement during our demo.

Original: Last but not the least, we stand committed to enacting the changes that the reviewers and the workshop participants might suggest.
Revised: Last but not the least, we stand committed to enacting the changes that the reviewers and the workshop participants might suggest

Original: A A PPENDIX -A: P LOTS
Revised: A A PPENDIX -A: P LOTS

Original: Eﬃcient Transformers: A Survey
Revised: Eﬃcient Transformers: A Survey

Original: Contents 1 Introduction 2 Background on Transformers 2.
Revised: Contents 1 Introduction 2 Background on Transformers 2.

Original: Multi-Head Self-Attention . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Multi-Head Self-Attention . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: Position-wise Feed-forward Layers . .
Revised: Position-wise Feed-forward Layers . .

Original: Putting it all together . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Putting it all together . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: Transformer Mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Transformer Mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: 3 A Survey of Eﬃcient Transformer Models
Revised: 3 A Survey of Eﬃcient Transformer Models

Original: A Taxonomy of Eﬃcient Transformers . . . . . . . . . . . . . . . . . . . . .
Revised: A Taxonomy of Eﬃcient Transformers . . . . . . . . . . . . . . . . . . . . .

Original: Detailed Walk-through of Eﬃcient Transformer Models . . . . . . . . . . .
Revised: Detailed Walk-through of Eﬃcient Transformer Models . . . . . . . . . . .

Original: Memory Compressed Transformer . . . . . . . . . . . . . . . . . . .
Revised: Memory Compressed Transformer . . . . . . . . . . . . . . . . . . .

Original: Image Transformer . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Image Transformer . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: Set Transformer . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Set Transformer . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: Sparse Transformer . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Sparse Transformer . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: Axial Transformer . . . .
Revised: Axial Transformer . . . .

Original: Longformer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Longformer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: Extended Transformer Construction (ETC) . . . . . . . . . . . . . .
Revised: Extended Transformer Construction (ETC) . . . . . . . . . . . . . .

Original: BigBird . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: BigBird . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: Routing Transformer . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Routing Transformer . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: 3.2.10 Reformer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: 3.2.10 Reformer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: 3.2.11 Sinkhorn Transformers . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: 3.2.11 Sinkhorn Transformers . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: 3.2.12 Linformer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: 3.2.12 Linformer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: 3.2.13 Linear Transformer . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: 3.2.13 Linear Transformer . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: 3.2.14 Performer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: 3.2.14 Performer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: 3.2.15 Synthesizers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: 3.2.15 Synthesizers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: 3.2.16 Transformer-XL . . . .
Revised: 3.2.16 Transformer-XL . . . .

Original: 3.2.17 Compressive Transformers . . . . . . . . . . . . . . . . . . . . . . . .
Revised: 3.2.17 Compressive Transformers . . . . . . . . . . . . . . . . . . . . . . . .

Original:  Discussion
Revised: 4 Discussion

Original: On Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: On Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: On Model Design Trends . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: On Model Design Trends . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: Brief Discussion on Orthogonal Eﬃciency Eﬀorts . . . . . . . . . . . . . . .
Revised: Brief Discussion on Orthogonal Eﬃciency Eﬀorts . . . . . . . . . . . . . . .

Original: 5 Conclusion
Revised: 5 Conclusion