Original: H ARDWARE - AWARE COMPRESSION WITH R ANDOM O PERATION A CCESS S PECIFIC T ILE (ROAST) HASHING
Revised: H ARDWARE - AWARE COMPRESSION WITH R ANDOM O PERATION A CCESS S PECIFIC T ILE (ROAST) HASHING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Advancements in deep learning are often associated with increasing model sizes.
Revised: Advancements in deep learning are often associated with increasing model sizes.

Original: Training and deploying large models require sophisticated hardware and incur significantly higher costs.
Revised: Training and deploying large models require sophisticated hardware and incur significantly higher costs.

Original: Thus, model compression is a widely explored approach to solving the problem.
Revised: Thus, model compression is a widely explored approach to solving the problem.

Original: However, SOTA techniques fall short in one or more desirable aspects of compression - for instance, pruning does not reduce memory for training, quantization can only provide up to 32x compression, HashedNet is cache-inefficient, etc.
Revised: However, SOTA techniques fall short in one or more desirable aspects of compression - for instance, pruning does not reduce memory for training, quantization can only provide up to 32x compression, HashedNet is cache-inefficient, etc.

Original: This paper proposes a model-agnostic, cache-friendly, and hardware-aware model compression approach: Random Operation Access
Revised: This paper proposes a model-agnostic, cache-friendly, and hardware-aware model compression approach: Random Operation Access

Original: Specific Tile (ROAST) hashing.
Revised: Specific Tile (ROAST) hashing.

Original: ROAST collapses the parameters by clubbing them through a lightweight mapping.
Revised: ROAST collapses the parameters by clubbing them through a lightweight mapping.

Original: While clubbing these parameters, ROAST utilizes cache hierarchies by aligning the memory access pattern with the parameter access pattern.
Revised: While clubbing these parameters, ROAST utilizes cache hierarchies by aligning the memory access pattern with the parameter access pattern.

Original: ROAST is up to ∼ 25 × faster to train and ∼ 50 × faster to infer than the popular parameter sharing method HashedNet.
Revised: ROAST is up to ∼ 25 × faster to train and ∼ 50 × faster to infer than the popular parameter sharing method HashedNet.

Original: Additionally, ROAST introduces global weight sharing, which is empirically and theoretically superior to local weight sharing in HashedNet, and can be of independent interest.
Revised: Additionally, ROAST introduces global weight sharing, which is empirically and theoretically superior to local weight sharing in HashedNet, and can be of independent interest.

Original: With ROAST, we can efficiently train and deploy the model using a much smaller memory footprint ( ∼ 10 − 100 × lesser) in text and image classification tasks.
Revised: With ROAST, we can efficiently train and deploy the model using a much smaller memory footprint ( ∼ 10 − 100 × lesser) in text and image classification tasks.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Models across different domains, including Natural Language Processing (NLP), Computer Vision (CV), and Information Retrieval (IR), are exploding in size.
Revised: Models across different domains, including Natural Language Processing (NLP), Computer Vision (CV), and Information Retrieval (IR), are exploding in size.

Original: State-of-the-art (SOTA) results in these domains are being obtained at a disproportionate increase in model sizes, questioning the sustainability of deep learning (Thompson et al., 2021).
Revised: State-of-the-art (SOTA) results in these domains are being obtained at a disproportionate increase in model sizes, questioning the sustainability of deep learning (Thompson et al., 2021).

Original: For instance, SOTA architectures for vision include VGG (Simonyan & Zisserman, 2014) (150M params, 0.6GB) and ViT (Dosovitskiy et al., 2020) (up to 304M params, 1.2GB).
Revised: For instance, SOTA architectures for vision include VGG (Simonyan & Zisserman, 2014) (150M params, 0.6GB) and ViT (Dosovitskiy et al., 2020) (up to 304M params, 1.2GB).

Original: Additionally, SOTA NLP models range from BERT (Devlin et al., 2018)
Revised: Additionally, SOTA NLP models range from BERT (Devlin et al., 2018)

Original: (340M params, 1.36GB) to GShard (Lepikhin et al., 2020) (600B params, 2.4TB).
Revised: (340M params, 1.36GB) to GShard (Lepikhin et al., 2020) (600B params, 2.4TB).

Original: Similarly, industrial-scale recommendation models such as DLRM (Naumov et al., 2019; Mudigere et al., 2021) can have up to 10s of trillions of parameters (50TB).
Revised: Similarly, industrial-scale recommendation models such as DLRM (Naumov et al., 2019; Mudigere et al., 2021) can have up to 10s of trillions of parameters (50TB).

Original: Large models, such as the above, come with various challenges.
Revised: Large models, such as the above, come with various challenges.

Original: They need high-end distributed hardware for training and deployment, incurring higher costs.
Revised: They need high-end distributed hardware for training and deployment, incurring higher costs.

Original: Additionally, the required modelparallel setup has higher inference and training-iteration latency for these models.
Revised: Additionally, the required modelparallel setup has higher inference and training-iteration latency for these models.

Original: Model compression is a research direction that aims to resolve these issues by reducing the memory footprint of the model.
Revised: Model compression is a research direction that aims to resolve these issues by reducing the memory footprint of the model.

Original: Compression of the order of 100 × can eliminate the need for model-parallel setup for many SOTA models like GPT(Radford et al., 2019), Gshard(Lepikhin et al., 2020), DLRM (Naumov et al., 2019) which now can fit on a single GPU.
Revised: Compression of the order of 100 × can eliminate the need for model-parallel setup for many SOTA models like GPT(Radford et al., 2019), Gshard(Lepikhin et al., 2020), DLRM (Naumov et al., 2019) which now can fit on a single GPU.

Original: Furthermore, compressing large models to small sizes come with immediate latency benefits.
Revised: Furthermore, compressing large models to small sizes come with immediate latency benefits.

Original: For example, Desai et al.
Revised: For example, Desai et al.

Original: (2022) showed that by compressing the DLRM model 1000 × and using 1 GPU instead of 8 GPUs, we could get 3 × faster inference at a lower cost.
Revised: (2022) showed that by compressing the DLRM model 1000 × and using 1 GPU instead of 8 GPUs, we could get 3 × faster inference at a lower cost.

Original: Also, in the case of CPU inference, a smaller model is efficient.
Revised: Also, in the case of CPU inference, a smaller model is efficient.

Original: For example, (Diamos et al., 2016) showed that if a single RNN layer can fit in registers, it leads to 146 × faster inference.
Revised: For example, (Diamos et al., 2016) showed that if a single RNN layer can fit in registers, it leads to 146 × faster inference.

Original: Thus, the ML community has heavily invested in model compression.
Revised: Thus, the ML community has heavily invested in model compression.

Original: A variety of model compression paradigms now exist in literature like pruning (Han et al., 2016b), quantisation (Han et al., 2016b), knowledge distillation (Buciluˇa et al., 2006), parameter-sharing (Chen et al., 2015; Desai et al., 2022), and low rank decomposition (Hrinchuk et al., 2020; Yin et al., 2021).
Revised: A variety of model compression paradigms now exist in literature like pruning (Han et al., 2016b), quantisation (Han et al., 2016b), knowledge distillation (Buciluˇa et al., 2006), parameter-sharing (Chen et al., 2015; Desai et al., 2022), and low rank decomposition (Hrinchuk et al., 2020; Yin et al., 2021).

Original: Table 1 compares these approaches on three considerations (1) if the model memory is reduced for training.
Revised: Table 1 compares these approaches on three considerations (1) if the model memory is reduced for training.

Original: (2) if the memory size can be controlled independently of the model, and (3) if the approach considers the underlying
Revised: (2) if the memory size can be controlled independently of the model, and (3) if the approach considers the underlying

Original: [Table]
Revised: [Table]

Original: pruning that are tuned to the underlying hardware and are cache-efficient
Revised: pruning that are tuned to the underlying hardware and are cache-efficient

Original: [Table]
Revised: [Table]

Original: memory hierarchies and is cache-efficient.
Revised: memory hierarchies and is cache-efficient.

Original: We want the techniques to fare positively in these three aspects.
Revised: We want the techniques to fare positively in these three aspects.

Original: However, techniques like pruning, QAT, and knowledge distillation require us to use the memory of the original model while training and only reduce inference time memory.
Revised: However, techniques like pruning, QAT, and knowledge distillation require us to use the memory of the original model while training and only reduce inference time memory.

Original: Additionally, there are limits to compression obtained by quantization and pruning depending on which component we are compressing.
Revised: Additionally, there are limits to compression obtained by quantization and pruning depending on which component we are compressing.

Original: For example, we cannot prune an embedding table ( N × d ) more than d × as we do not want any embedding vector to have all zeros.
Revised: For example, we cannot prune an embedding table ( N × d ) more than d × as we do not want any embedding vector to have all zeros.

Original: HashedNet provides memory reduction during training and arbitrary control over memory.
Revised: HashedNet provides memory reduction during training and arbitrary control over memory.

Original: However, the look-ups in HashedNet are randomly and independently distributed across the total memory.
Revised: However, the look-ups in HashedNet are randomly and independently distributed across the total memory.

Original: This makes HashedNet cache-inefficient.
Revised: This makes HashedNet cache-inefficient.

Original: This paper presents Random Operation Access Specific Tile (ROAST) hashing, a parameter-sharing approach that provides cache-efficiency and arbitrary control over memory during training as well as inference.
Revised: This paper presents Random Operation Access Specific Tile (ROAST) hashing, a parameter-sharing approach that provides cache-efficiency and arbitrary control over memory during training as well as inference.

Original: ROAST does not change the model’s functional form and can be applied to all computational modules of a model, such as MLP layers, attention blocks, convolution layers, and embedding tables.
Revised: ROAST does not change the model’s functional form and can be applied to all computational modules of a model, such as MLP layers, attention blocks, convolution layers, and embedding tables.

Original: ROAST is hardware aware: it proposes a tile-based hashing scheme tuned to the memory access pattern of the algorithmic implementation of the operation being performed.
Revised: ROAST is hardware aware: it proposes a tile-based hashing scheme tuned to the memory access pattern of the algorithmic implementation of the operation being performed.

Original: ROAST uses this hash function to recover blocks of the model from a single array of parameters - ROAST array.
Revised: ROAST uses this hash function to recover blocks of the model from a single array of parameters - ROAST array.

Original: ROAST is superior to HashedNet due to two factors (1) Unlike HashedNet, ROAST proposes global weight-sharing where parameters are shared across the different computational modules.
Revised: ROAST is superior to HashedNet due to two factors (1) Unlike HashedNet, ROAST proposes global weight-sharing where parameters are shared across the different computational modules.

Original: As we shall see, global weight-sharing is empirically and theoretically superior to local weight-sharing and might be of independent interest.
Revised: As we shall see, global weight-sharing is empirically and theoretically superior to local weight-sharing and might be of independent interest.

Original: (2) ROAST uses block-based hashing, which is theoretically superior to count-sketch hashing used in HashedNet.
Revised: (2) ROAST uses block-based hashing, which is theoretically superior to count-sketch hashing used in HashedNet.

Original: (Desai et al., 2022)
Revised: (Desai et al., 2022)

Original: We show that with ROAST, we can train a BERT-2-2 ( 2 layers, 2 attention heads) model on the largest available text-classification datasets (amazon-polarity, yelp-polarity) using 100 × lesser memory without loss of quality.
Revised: We show that with ROAST, we can train a BERT-2-2 ( 2 layers, 2 attention heads) model on the largest available text-classification datasets (amazon-polarity, yelp-polarity) using 100 × lesser memory without loss of quality.

Original: In cases where the model is overly parameterized, like using BERT-12-12 in the text classification task above, we can still obtain similar compression of 100 × .
Revised: In cases where the model is overly parameterized, like using BERT-12-12 in the text classification task above, we can still obtain similar compression of 100 × .

Original: Thus it is a good alternative to neural architecture search.
Revised: Thus it is a good alternative to neural architecture search.

Original: The results extend to CV datasets as well.
Revised: The results extend to CV datasets as well.

Original: Specifically, we can train a ResNet-9 model with 10 × lesser memory for the CIFAR10 dataset.
Revised: Specifically, we can train a ResNet-9 model with 10 × lesser memory for the CIFAR10 dataset.

Original: Importantly, we show that ROAST, due to its hardware-aware nature, is significantly faster than HashedNet: ROAST is up to ∼ 25 × faster to train and ∼ 50 × faster to infer than HashedNet for large matrix multiplications.
Revised: Importantly, we show that ROAST, due to its hardware-aware nature, is significantly faster than HashedNet: ROAST is up to ∼ 25 × faster to train and ∼ 50 × faster to infer than HashedNet for large matrix multiplications.

Original: Our current implementation of ROAST matrix multiplication is about 1.34 × slower than full matrix multiplication in pytorch.
Revised: Our current implementation of ROAST matrix multiplication is about 1.34 × slower than full matrix multiplication in pytorch.

Original: This is a testament to how optimized CUBLAS libraries are.
Revised: This is a testament to how optimized CUBLAS libraries are.

Original: We believe, with enough investigation, we can make ROAST-MM comparably efficient to pytorch-MM as well.
Revised: We believe, with enough investigation, we can make ROAST-MM comparably efficient to pytorch-MM as well.

Original: Limitations of ROAST:
Revised: Limitations of ROAST:

Original: One of the goals of model compression, apart from reducing memory usage, is to reduce computational workload for deployment.
Revised: One of the goals of model compression, apart from reducing memory usage, is to reduce computational workload for deployment.

Original: ROAST, currently, is not devised to decrease computation; it only decreases the memory footprint of a model.
Revised: ROAST, currently, is not devised to decrease computation; it only decreases the memory footprint of a model.

Original: Reducing computation with a small memory is left for future work.
Revised: Reducing computation with a small memory is left for future work.

Original: However, it should be noted that reducing the memory footprint can significantly reduce computation latency and power consumption.
Revised: However, it should be noted that reducing the memory footprint can significantly reduce computation latency and power consumption.

Original: As shown in (Han et al., 2016a), accessing memory from RAM is 6400 × costlier than 32bit INT ADD and 128 × more expensive than on-chip SRAM access in terms of energy consumption.
Revised: As shown in (Han et al., 2016a), accessing memory from RAM is 6400 × costlier than 32bit INT ADD and 128 × more expensive than on-chip SRAM access in terms of energy consumption.

Original: Additionally, RAM access generally is ∼ 100 × slower than a floating-point operation.
Revised: Additionally, RAM access generally is ∼ 100 × slower than a floating-point operation.

Original: So, this model compression with ROAST, in our opinion, is an important step for efficient training and inference.
Revised: So, this model compression with ROAST, in our opinion, is an important step for efficient training and inference.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: This section briefly reviews the rich history of model compression paradigms.
Revised: This section briefly reviews the rich history of model compression paradigms.

Original: Model compression can be generally classified into two categories: (1) Compressing a learned model and (2) Learning a compressed model.
Revised: Model compression can be generally classified into two categories: (1) Compressing a learned model and (2) Learning a compressed model.

Original: ROAST lies in the second category.
Revised: ROAST lies in the second category.

Original: Compressing learned models: 1)
Revised: Compressing learned models: 1)

Original: Pruning: Pruning (Zhu & Gupta, 2017) is a technique to remove parts of a large model, including weights, blocks, and layers, to make the model lighter.
Revised: Pruning: Pruning (Zhu & Gupta, 2017) is a technique to remove parts of a large model, including weights, blocks, and layers, to make the model lighter.

Original: Pruning can be performed as a one-time operation or gradually interspersed with training.
Revised: Pruning can be performed as a one-time operation or gradually interspersed with training.

Original: 2) Quantization:
Revised: 2) Quantization:

Original: Quantization can involve reducing the precision of the parameters of a model.
Revised: Quantization can involve reducing the precision of the parameters of a model.

Original: Mixed precision models are sometimes used where different precision is used with different weights.
Revised: Mixed precision models are sometimes used where different precision is used with different weights.

Original: KMeans quantization is another type of quantization, where models’ weights are clustered using KMeans, and each cluster’s centroid is used for all cluster weights.
Revised: KMeans quantization is another type of quantization, where models’ weights are clustered using KMeans, and each cluster’s centroid is used for all cluster weights.

Original: Model compression, in this case, is achieved by reducing the number of distinct weights.
Revised: Model compression, in this case, is achieved by reducing the number of distinct weights.

Original: 3) Knowledge distillation: Knowledge distillation (Buciluˇa et al., 2006) is widely applied in model compression with a focus on distilled architectures.
Revised: 3) Knowledge distillation: Knowledge distillation (Buciluˇa et al., 2006) is widely applied in model compression with a focus on distilled architectures.

Original: Knowledge distillation involves training a teacher model (large original model); then, a student model is trained using the logits of the teacher model.
Revised: Knowledge distillation involves training a teacher model (large original model); then, a student model is trained using the logits of the teacher model.

Original: Empirically, the student model trained under this paradigm generalizes better than the student model trained standalone.
Revised: Empirically, the student model trained under this paradigm generalizes better than the student model trained standalone.

Original: Many variations exist on this basic idea of knowledge distillation.
Revised: Many variations exist on this basic idea of knowledge distillation.

Original: While these techniques have successfully reduced memory for inference, one of the drawbacks of this line of compression is that the memory usage while training the model is not reduced.
Revised: While these techniques have successfully reduced memory for inference, one of the drawbacks of this line of compression is that the memory usage while training the model is not reduced.

Original: ROAST, however, provides a solution that reduces the model’s memory during training and inference.
Revised: ROAST, however, provides a solution that reduces the model’s memory during training and inference.

Original: Learning compressed models 1) Low-rank decomposition:
Revised: Learning compressed models 1) Low-rank decomposition:

Original: In this method, matrices in the model are decomposed into a product of two low-rank matrices, thus saving memory per matrix.
Revised: In this method, matrices in the model are decomposed into a product of two low-rank matrices, thus saving memory per matrix.

Original: A generalization of low-rank decomposition to tensors is called tensor-train decomposition 2)
Revised: A generalization of low-rank decomposition to tensors is called tensor-train decomposition 2)

Original: Parameter sharing:
Revised: Parameter sharing:

Original: Parameter sharing approaches such as HashedNet (Chen et al., 2015) are generally used for matrix compression.
Revised: Parameter sharing approaches such as HashedNet (Chen et al., 2015) are generally used for matrix compression.

Original: These approaches randomly share weights among different parameters, reducing the model’s memory usage.
Revised: These approaches randomly share weights among different parameters, reducing the model’s memory usage.

Original: This line of research provides model reduction even during training.
Revised: This line of research provides model reduction even during training.

Original: However, Low-rank decomposition does not offer arbitrary control over memory footprint, and HashedNets are inefficient due to heavy cache-trashing caused by non-local lookups.
Revised: However, Low-rank decomposition does not offer arbitrary control over memory footprint, and HashedNets are inefficient due to heavy cache-trashing caused by non-local lookups.

Original: Conversely, ROAST is a model-agnostic parameter-sharing approach that can arbitrarily reduce the model size without affecting the functional form while keeping the model recovery efficient.
Revised: Conversely, ROAST is a model-agnostic parameter-sharing approach that can arbitrarily reduce the model size without affecting the functional form while keeping the model recovery efficient.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: HashedNet: Compressing MLP matrices
Revised: HashedNet: Compressing MLP matrices

Original: Previous work (Chen et al., 2015) introduced a weight sharing method to compress weight matrices of MLP models.
Revised: Previous work (Chen et al., 2015) introduced a weight sharing method to compress weight matrices of MLP models.

Original: They map each matrix parameter to a shared parameter array using a random hash function xxhash (Collet, 2016).
Revised: They map each matrix parameter to a shared parameter array using a random hash function xxhash (Collet, 2016).

Original: In the forward pass, this mapping is used to recover a weight matrix and perform matrix multiplication for each
Revised: In the forward pass, this mapping is used to recover a weight matrix and perform matrix multiplication for each

Original: MLP layer.
Revised: MLP layer.

Original: In the backward pass, the gradients of each weight matrix are mapped to the shared compressed array and aggregated using the sum operation.
Revised: In the backward pass, the gradients of each weight matrix are mapped to the shared compressed array and aggregated using the sum operation.

Original: It should also be noted that each MLP layer uses an independent array of parameters.
Revised: It should also be noted that each MLP layer uses an independent array of parameters.

Original: One of the main concerns with HashedNet is that memory accesses on the compressed array are non-coalesced.
Revised: One of the main concerns with HashedNet is that memory accesses on the compressed array are non-coalesced.

Original: Thus, fetching a compressed matrix via HashedNet requires significantly more memory read transactions than fetching an uncompressed matrix for which memory accesses can coalesce.
Revised: Thus, fetching a compressed matrix via HashedNet requires significantly more memory read transactions than fetching an uncompressed matrix for which memory accesses can coalesce.

Original: Our evaluation shows that uncoalesced memory accesses lead to high latency, especially for large matrices.
Revised: Our evaluation shows that uncoalesced memory accesses lead to high latency, especially for large matrices.

Original: Random Block Offset Embedding Array (ROBE) for embedding compression In ROBE (Desai et al., 2022), the embedding table is generated using an array of parameters.
Revised: Random Block Offset Embedding Array (ROBE) for embedding compression In ROBE (Desai et al., 2022), the embedding table is generated using an array of parameters.

Original: The embedding of a token is obtained by drawing chunks of the embedding from the ROBE array.
Revised: The embedding of a token is obtained by drawing chunks of the embedding from the ROBE array.

Original: The locations of the chunks are decided randomly via light-weight universal hash functions.
Revised: The locations of the chunks are decided randomly via light-weight universal hash functions.

Original: Authors of ROBE showed that ROBE hashing is theoretically superior to feature hashing used in HashedNet.
Revised: Authors of ROBE showed that ROBE hashing is theoretically superior to feature hashing used in HashedNet.

Original: Also, the use of chunks causes memory accesses to coalesce, making embedding lookup efficient.
Revised: Also, the use of chunks causes memory accesses to coalesce, making embedding lookup efficient.

Original: ROAST proposes a component agnostic, global parameter sharing approach that tunes the hashing function to match memory accesses of algorithmic implementation of operation over available hardware, thus giving a superior parameter sharing scheme.
Revised: ROAST proposes a component agnostic, global parameter sharing approach that tunes the hashing function to match memory accesses of algorithmic implementation of operation over available hardware, thus giving a superior parameter sharing scheme.

Original:  R ANDOM O PERATION A CCESS S PECIFIC T ILE (ROAST) HASHING
Revised:  R ANDOM O PERATION A CCESS S PECIFIC T ILE (ROAST) HASHING

Original: Let M be the compressed memory from which parameters will be used, f be the model or the function that we want to run using M , and W be the recovered weights used in f .
Revised: Let M be the compressed memory from which parameters will be used, f be the model or the function that we want to run using M , and W be the recovered weights used in f .

Original: f can be considered as a composition of operations {O i ( X i , W i ) } .
Revised: f can be considered as a composition of operations {O i ( X i , W i ) } .

Original: By operation, we mean the smaller functions that, when composed together, give us the model f .
Revised: By operation, we mean the smaller functions that, when composed together, give us the model f .

Original: Here X i is the input to the operation, and W i is the weights (i.e., learnable parameters) that O i uses.
Revised: Here X i is the input to the operation, and W i is the weights (i.e., learnable parameters) that O i uses.

Original: Generally, W i s are distinct and do not share parameters.
Revised: Generally, W i s are distinct and do not share parameters.

Original: Random Operation Access Specific Tile (ROAST) hashing is a way to perform efficient modelagnostic parameter sharing-based compression.
Revised: Random Operation Access Specific Tile (ROAST) hashing is a way to perform efficient modelagnostic parameter sharing-based compression.

Original: The following distinct aspects of ROAST set it apart from previous parameter sharing-based methods.
Revised: The following distinct aspects of ROAST set it apart from previous parameter sharing-based methods.

Original: (1) ROAST is a generic technique applicable to all computational modules.
Revised: (1) ROAST is a generic technique applicable to all computational modules.

Original: (2) ROAST proposes to tune its mapping from W i to M in a way that coalesces memory accesses according to how memory is accessed during the operation.
Revised: (2) ROAST proposes to tune its mapping from W i to M in a way that coalesces memory accesses according to how memory is accessed during the operation.

Original: This makes
Revised: This makes

Original: ROAST efficient and up to 45 × faster than competing approaches like HashedNet.
Revised: ROAST efficient and up to 45 × faster than competing approaches like HashedNet.

Original: (3) ROAST proposes Global Memory Sharing (GMS) as opposed to Local Memory Sharing (LMS) used in
Revised: (3) ROAST proposes Global Memory Sharing (GMS) as opposed to Local Memory Sharing (LMS) used in

Original: HashedNet.
Revised: HashedNet.

Original: We show GMS to be theoretically and empirically superior to LMS in Section 5 and 6.
Revised: We show GMS to be theoretically and empirically superior to LMS in Section 5 and 6.

Original: 4. ROAST OPERATIONS IN DEEP LEARNING
Revised: 4. ROAST OPERATIONS IN DEEP LEARNING

Original: Any model f can be considered as a composition of smaller functions {O i ( X i , W i ) } .
Revised: Any model f can be considered as a composition of smaller functions {O i ( X i , W i ) } .

Original: There are multiple ways to perform this decomposition depending upon what we consider a valid (or small enough) operation.
Revised: There are multiple ways to perform this decomposition depending upon what we consider a valid (or small enough) operation.

Original: In ROAST, we consider three types of operations: (1) L ( l, W ) , lookup that accesses M and recovers l th element of W , say w .
Revised: In ROAST, we consider three types of operations: (1) L ( l, W ) , lookup that accesses M and recovers l th element of W , say w .

Original: By element, we mean some particular part of
Revised: By element, we mean some particular part of

Original: W that is identifiable by an integer.
Revised: W that is identifiable by an integer.

Original: An example with embedding tables is given in figure 1.
Revised: An example with embedding tables is given in figure 1.

Original: MM ( X, W ) , matrix multiplication that multiplies X with W and returns the result, and (3) N ( X ) , various operations that only act on the input but do not interact with M .
Revised: MM ( X, W ) , matrix multiplication that multiplies X with W and returns the result, and (3) N ( X ) , various operations that only act on the input but do not interact with M .

Original: In ROAST, in order to limit the memory usage, we make sure that L is used only on a small w and MM is performed without recovering the entire matrix.
Revised: In ROAST, in order to limit the memory usage, we make sure that L is used only on a small w and MM is performed without recovering the entire matrix.

Original: We find that most deep learning models, if not all, can be written as a composition of operations N , MM and L , where L is only applied on small parameters.
Revised: We find that most deep learning models, if not all, can be written as a composition of operations N , MM and L , where L is only applied on small parameters.

Original: Let us discuss how ROAST implements L and MM operations in the following paragraphs.
Revised: Let us discuss how ROAST implements L and MM operations in the following paragraphs.

Original: Lookup ( L ( l, W ) )
Revised: Lookup ( L ( l, W ) )

Original: We recover a parameter weight w of any shape in a row-major format.
Revised: We recover a parameter weight w of any shape in a row-major format.

Original: Thus, we can consider w = W ( l ) to be a 1D vector without loss of generality.
Revised: Thus, we can consider w = W ( l ) to be a 1D vector without loss of generality.

Original: ROAST recovers w from M in a blocked fashion.
Revised: ROAST recovers w from M in a blocked fashion.

Original: Consider w to be composed of chunks of size Z .
Revised: Consider w to be composed of chunks of size Z .

Original: Each chunk c is located in M using a universal hash function h 1 and is recovered from the location h 1 ( c ) in M .
Revised: Each chunk c is located in M using a universal hash function h 1 and is recovered from the location h 1 ( c ) in M .

Original: Let C ( i ) give the chunk number of index i and O ( i ) give the offset of i in this chunk.
Revised: Let C ( i ) give the chunk number of index i and O ( i ) give the offset of i in this chunk.

Original: [Equation]
Revised: [Equation]

Original: The recovered W has λ as a scaling factor discussed in section 4.2.
Revised: The recovered W has λ as a scaling factor discussed in section 4.2.

Original: The hash function hashes to a range { 0 , ..., |M| − Z } to avoid overflows while reading the memory.
Revised: The hash function hashes to a range { 0 , ..., |M| − Z } to avoid overflows while reading the memory.

Original: For example, Figure(right) illustrates the embedding lookup using L with chunk size of 2.
Revised: For example, Figure(right) illustrates the embedding lookup using L with chunk size of 2.

Original: ROAST uses L to implement computational modules such as embeddings, bias vectors, and so on.
Revised: ROAST uses L to implement computational modules such as embeddings, bias vectors, and so on.

Original: We generalize the embedding lookup kernel from ROBE (Desai et al., 2022) to implement our L kernel.
Revised: We generalize the embedding lookup kernel from ROBE (Desai et al., 2022) to implement our L kernel.

Original: Matrix multiplication ( MM ( X i , W i ) )
Revised: Matrix multiplication ( MM ( X i , W i ) )

Original: 2D matrix multiplication is one of the most widely used operations in deep learning.
Revised: 2D matrix multiplication is one of the most widely used operations in deep learning.

Original: We implement our ROAST-MM kernel with parameter sharing performed in a way that the algorithm for matrix multiplication accesses coalesced pieces of M .
Revised: We implement our ROAST-MM kernel with parameter sharing performed in a way that the algorithm for matrix multiplication accesses coalesced pieces of M .

Original: An efficient implementation of matrix multiplication on GPU follows a block multiplication algorithm to use the on-chip shared memory efficiently.
Revised: An efficient implementation of matrix multiplication on GPU follows a block multiplication algorithm to use the on-chip shared memory efficiently.

Original: While computing C =
Revised: While computing C =

Original: A × B , A, B and C are divided in tiles of size Z 0 ×
Revised: A × B , A, B and C are divided in tiles of size Z 0 ×

Original: Z 1 , Z 1 × Z 2 and Z 0 × Z 2 respectively.
Revised: Z 1 , Z 1 × Z 2 and Z 0 × Z 2 respectively.

Original: Thus, we divide our 2D weight matrix into tiles of size Z 1 × Z 2 .
Revised: Thus, we divide our 2D weight matrix into tiles of size Z 1 × Z 2 .

Original: The tile, ( x, y ) , where x and y are the coordinates of the tile, is located in
Revised: The tile, ( x, y ) , where x and y are the coordinates of the tile, is located in

Original: M in a row-major format via a universal hash function h 2 ( x, y ) .
Revised: M in a row-major format via a universal hash function h 2 ( x, y ) .

Original: Let C 1 ( i, j ) and C 2 ( i, j ) give the x -coordinate and y -coordinate of the tile to which i , j belongs.
Revised: Let C 1 ( i, j ) and C 2 ( i, j ) give the x -coordinate and y -coordinate of the tile to which i , j belongs.

Original: Similarly, let O 

Revised: Similarly, let O

Original: ( i, j ) and O 2 ( i, j ) give the x -offset and y -offset of a location ( i, j ) on the tile.
Revised: ( i, j ) and O 2 ( i, j ) give the x -offset and y -offset of a location ( i, j ) on the tile.

Original: Then, we use the following mapping for
Revised: Then, we use the following mapping for

Original: ROAST-MM,
Revised: ROAST-MM,

Original: [Equation]
Revised: [Equation]

Original: Again, λ is the scaling factor discussed in section 4.2.
Revised: Again, λ is the scaling factor discussed in section 4.2.

Original: The hash function hashes to a range { 0 , ..., |M| − Z 1 Z 2 } to avoid overflows while reading the chunk.
Revised: The hash function hashes to a range { 0 , ..., |M| − Z 1 Z 2 } to avoid overflows while reading the chunk.

Original: Figure 1 (left) illustrates ROASTMM with a chunk size of 2 × 2 .
Revised: Figure 1 (left) illustrates ROASTMM with a chunk size of 2 × 2 .

Original: The above mapping is used whenever a 2D tile is accessed in the matrix multiplication algorithm.
Revised: The above mapping is used whenever a 2D tile is accessed in the matrix multiplication algorithm.

Original: The pseudo code for ROAST-MM is shown in algorithm 1.
Revised: The pseudo code for ROAST-MM is shown in algorithm 1.

Original: We talk about implementation of this kernel and its evaluation in the later part of the paper.
Revised: We talk about implementation of this kernel and its evaluation in the later part of the paper.

Original: ROAST uses
Revised: ROAST uses

Original: ROAST-MM kernel to implement computational modules such as MLP layers, attention blocks, etc.
Revised: ROAST-MM kernel to implement computational modules such as MLP layers, attention blocks, etc.

Original: Each module invoking ROAST kernels uses independent hash functions.
Revised: Each module invoking ROAST kernels uses independent hash functions.

Original: Require:
Revised: Require:

Original: X ∈ R I × H , M , λ , h : N 2 → { 0 , ..., |M| − Z 1 Z 2 }
Revised: X ∈ R I × H , M , λ , h : N 2 → { 0 , ..., |M| − Z 1 Z 2 }

Original: Ensure: output = MM ( X, M
Revised: Ensure: output = MM ( X, M

Original: [ h (: , :)]) value ← TILE ( Z 0 , Z 2 )
Revised: [ h (: , :)]) value ← TILE ( Z 0 , Z 2 )

Original: ▷ Allocate a 2D tile of size Z 0 ×
Revised: ▷ Allocate a 2D tile of size Z 0 ×

Original: Z 2 to accumulate results for i ∈ { 0 , 1 , ..., ⌈ I/Z 0 ⌉ − 1 } do for j ∈ { 0 , 1 , ..., ⌈ O/Z 2 ⌉ − 1 } do value [: , :] ←for k ∈ { 0 , 1 , ..., ⌈ H/Z 1 ⌉ − 1 } do value ← value + MM ( X [ i : i + Z 0 , k : k
Revised: Z 2 to accumulate results for i ∈ { 0 , 1 , ..., ⌈ I/Z 0 ⌉ − 1 } do for j ∈ { 0 , 1 , ..., ⌈ O/Z 2 ⌉ − 1 } do value [: , :] ←for k ∈ { 0 , 1 , ..., ⌈ H/Z 1 ⌉ − 1 } do value ← value + MM ( X [ i : i + Z 0 , k : k

Original: + Z 1 ] , M ( h ( k : k + Z 1 , j : j + Z 2 )))
Revised: + Z 1 ] , M ( h ( k : k + Z 1 , j : j + Z 2 )))

Original: ▷ Access to the weight tile passes through the hash function end for output [ i : i + Z 0 , j : j + Z 2 ] ← λ ∗ value end for end for
Revised: ▷ Access to the weight tile passes through the hash function end for output [ i : i + Z 0 , j : j + Z 2 ] ← λ ∗ value end for end for

Original: Apart from scaling each recovered parameter with module-specifc λ , we can also multiply it with another independent hash function g : N k → {± 1 } ( k =1 or k =2).
Revised: Apart from scaling each recovered parameter with module-specifc λ , we can also multiply it with another independent hash function g : N k → {± 1 } ( k =1 or k =2).

Original: 4. G LOBAL MEMORY SHARING (GMS)
Revised: 4. G LOBAL MEMORY SHARING (GMS)

Original: HashedNet uses local memory sharing (LMS), which states that each layer will have independent compressed memory.
Revised: HashedNet uses local memory sharing (LMS), which states that each layer will have independent compressed memory.

Original: In contrast, ROAST proposes global memory sharing (GMS), wherein we share memory across modules.
Revised: In contrast, ROAST proposes global memory sharing (GMS), wherein we share memory across modules.

Original: However, modules cannot directly use the parameters stored in M as each module’s weights requires initialization and optimization at different scales.
Revised: However, modules cannot directly use the parameters stored in M as each module’s weights requires initialization and optimization at different scales.

Original: For instance, in the Xavier’s initialization (Glorot & Bengio, 2010), weights are initialized with distribution
Revised: For instance, in the Xavier’s initialization (Glorot & Bengio, 2010), weights are initialized with distribution

Original: Uniform ( − 1 / √ n, 1 / √ n ) where n is size of the input to the module.
Revised: Uniform ( − 1 / √ n, 1 / √ n ) where n is size of the input to the module.

Original: In GMS, we must ensure that each module gets weights at the required scale.
Revised: In GMS, we must ensure that each module gets weights at the required scale.

Original: To achieve this, we first initialize the entire
Revised: To achieve this, we first initialize the entire

Original: ROAST parameter array with values from the distribution Uniform ( − 1 /C, 1 /C )
Revised: ROAST parameter array with values from the distribution Uniform ( − 1 /C, 1 /C )

Original: for some constant
Revised: for some constant

Original: C .
Revised: C .

Original: Then, for each module, we scale the weights retrieved from the ROAST array by a factor of λ = C/ √ n .
Revised: Then, for each module, we scale the weights retrieved from the ROAST array by a factor of λ = C/ √ n .

Original: One can understand the benefit of GMS over LMS in terms of the number of distinct functions in f that can be expressed using a fixed M .
Revised: One can understand the benefit of GMS over LMS in terms of the number of distinct functions in f that can be expressed using a fixed M .

Original: Consider a family of functions with n parameters.
Revised: Consider a family of functions with n parameters.

Original: GMS can potentially express |M| n functions across different random mappings.
Revised: GMS can potentially express |M| n functions across different random mappings.

Original: In LMS, let separate parameters be of sizes n 1 , n 2 , ..n k and each of them is mapped into memories
Revised: In LMS, let separate parameters be of sizes n 1 , n 2 , ..n k and each of them is mapped into memories

Original: M 1 , M 2 , ..., M k .
Revised: M 1 , M 2 , ..., M k .

Original: Thus, n = (cid:80) i n i and |M| =
Revised: Thus, n = (cid:80) i n i and |M| =

Original: (cid:80) i |M i | .
Revised: (cid:80) i |M i | .

Original: Then LMS can only express |M 1 | n 1 |M 2 | n 2 ....
Revised: Then LMS can only express |M 1 | n 1 |M 2 | n 2 ....

Original: |M k | n k different functions.
Revised: |M k | n k different functions.

Original: Thus expressivity of LMS is strictly less than that of GMS and can be orders of magnitude less depending on exact values of n i and |M i | .
Revised: Thus expressivity of LMS is strictly less than that of GMS and can be orders of magnitude less depending on exact values of n i and |M i | .

Original: We also show that GMS is superior to
Revised: We also show that GMS is superior to

Original: LMS in terms of dimensionality reduction (feature hashing) in Section 5.
Revised: LMS in terms of dimensionality reduction (feature hashing) in Section 5.

Original: 4. F ORWARD AND BACKWARD PASSES
Revised: 4. F ORWARD AND BACKWARD PASSES

Original: Recall that in ROAST, operations are of three types L , MM and N .
Revised: Recall that in ROAST, operations are of three types L , MM and N .

Original: The forward pass proceeds by applying each operation in sequence.
Revised: The forward pass proceeds by applying each operation in sequence.

Original: If an operation is of type N , we directly apply its function on the input.
Revised: If an operation is of type N , we directly apply its function on the input.

Original: For L and MM operations, outputs are computed according to the procedure described in
Revised: For L and MM operations, outputs are computed according to the procedure described in

Original: Section 4.1.
Revised: Section 4.1.

Original: The gradient of the loss w.r.t a weight w i in M is the λ -scaled aggregation of gradients of loss w.r.t all the parameters that map to this weight.
Revised: The gradient of the loss w.r.t a weight w i in M is the λ -scaled aggregation of gradients of loss w.r.t all the parameters that map to this weight.

Original: For simplicity of notation, consider θ as the complete parameter, λ ( j ) as the scaling factor we use for the module that θ j belongs to, and h be the mapping from θ to M .
Revised: For simplicity of notation, consider θ as the complete parameter, λ ( j ) as the scaling factor we use for the module that θ j belongs to, and h be the mapping from θ to M .

Original: See Appendix B.1 for more details.
Revised: See Appendix B.1 for more details.

Original: [Equation]
Revised: [Equation]

Original: 4. I MPLEMENTATION OF ROAST-MM
Revised: 4. I MPLEMENTATION OF ROAST-MM

Original: The high-performance community has heavily investigated the fast implementation of the General
Revised: The high-performance community has heavily investigated the fast implementation of the General

Original: Matrix Multiplication (GEMM) kernel, a fundamental operation in many computational workloads, including deep learning.
Revised: Matrix Multiplication (GEMM) kernel, a fundamental operation in many computational workloads, including deep learning.

Original: Optimized implementations of GEMM kernels are available in vendor libraries such as cuBLAS (NVIDIA Corporation, 2022a) and CUTLASS (NVIDIA Corporation, 2022b).
Revised: Optimized implementations of GEMM kernels are available in vendor libraries such as cuBLAS (NVIDIA Corporation, 2022a) and CUTLASS (NVIDIA Corporation, 2022b).

Original: Unfortunately, these implementations do not support custom tile loading operations, which is the key of ROAST-MM.
Revised: Unfortunately, these implementations do not support custom tile loading operations, which is the key of ROAST-MM.

Original: To implement ROAST-MM to a reasonable level of efficiency, we used
Revised: To implement ROAST-MM to a reasonable level of efficiency, we used

Original: Triton (Tillet et al., 2019): an intermediate language for tiled neural network computations.
Revised: Triton (Tillet et al., 2019): an intermediate language for tiled neural network computations.

Original: Triton abstracts out the shared memory management to make it helpful in customizing tiled operations with high efficiency.
Revised: Triton abstracts out the shared memory management to make it helpful in customizing tiled operations with high efficiency.

Original: In our implementation of ROAST-MM, the optimal size of coalesced tiles is a parameter that depends on the shape of the weight matrix.
Revised: In our implementation of ROAST-MM, the optimal size of coalesced tiles is a parameter that depends on the shape of the weight matrix.

Original: Therefore, different tile sizes can lead to different parallelism, occupancy, and shared memory efficiency, resulting in different execution times.
Revised: Therefore, different tile sizes can lead to different parallelism, occupancy, and shared memory efficiency, resulting in different execution times.

Original: We autotune this parameter to obtain the best performance for particular matrix shapes.
Revised: We autotune this parameter to obtain the best performance for particular matrix shapes.

Original: We propose two strategies for autotuning each ROAST-MM layer - (1) Optimize the inference workload by autotuning the forward kernel and sharing the tile size with the backward kernels.
Revised: We propose two strategies for autotuning each ROAST-MM layer - (1) Optimize the inference workload by autotuning the forward kernel and sharing the tile size with the backward kernels.

Original: (2) Optimize the training workload by autotuning the forward and backward kernels together.
Revised: (2) Optimize the training workload by autotuning the forward and backward kernels together.

Original: Extensive evaluation of this kernel is presented in appendix C.2.
Revised: Extensive evaluation of this kernel is presented in appendix C.2.

Original:  F EATURE HASHING QUALITY : GLOBAL MEMORY SHARING ADVANTAGE OVER LOCAL MEMORY SHARING
Revised:  F EATURE HASHING QUALITY : GLOBAL MEMORY SHARING ADVANTAGE OVER LOCAL MEMORY SHARING

Original: We can consider model compression as dimensionality reduction of a parameter vector (a one dimensional vector of all parameters in a model) of size n into a vector of size |M|
Revised: We can consider model compression as dimensionality reduction of a parameter vector (a one dimensional vector of all parameters in a model) of size n into a vector of size |M|

Original: = m .
Revised: = m .

Original: Quality of inner-product preservation is used as a metric to measure the quality of dimensionality reduction.
Revised: Quality of inner-product preservation is used as a metric to measure the quality of dimensionality reduction.

Original: In terms of dimensionality reduction, ROAST uses ROBE hashing, which shows that chunk based hashing is theoretically better than hashing individual elements.
Revised: In terms of dimensionality reduction, ROAST uses ROBE hashing, which shows that chunk based hashing is theoretically better than hashing individual elements.

Original: In this section, we compare ROAST’s
Revised: In this section, we compare ROAST’s

Original: GMS proposal against HashedNet’s LMS using a chunck size of one.
Revised: GMS proposal against HashedNet’s LMS using a chunck size of one.

Original: Consider two parameter vectors x, y ∈ R n , we are interested in how the inner product of parameter vectors are preserved under hashing.
Revised: Consider two parameter vectors x, y ∈ R n , we are interested in how the inner product of parameter vectors are preserved under hashing.

Original: Let x =
Revised: Let x =

Original: [ x 1 , x 2 , ..., x k ] and y =
Revised: [ x 1 , x 2 , ..., x k ] and y =

Original: [ y 1 , y 2 , ..., y k ] be composed of k vectors of sizes n 1 , n 2 , ...n k where [] denotes concatentation.
Revised: [ y 1 , y 2 , ..., y k ] be composed of k vectors of sizes n 1 , n 2 , ...n k where [] denotes concatentation.

Original: In LMS, let each piece map to memory of size f
Revised: In LMS, let each piece map to memory of size f

Original: i m where (cid:80)
Revised: i m where (cid:80)

Original: i f i
Revised: i f i

Original: = 1 .
Revised: = 1 .

Original: The estimated inner product with GMS is
Revised: The estimated inner product with GMS is

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: The estimated inner product with LMS can be written as
Revised: The estimated inner product with LMS can be written as

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: Theorem 1 Let x, y ∈ R n and be composed of k vectors x = [ x 1 , x 2 , ..., x k ] and y =
Revised: Theorem 1 Let x, y ∈ R n and be composed of k vectors x = [ x 1 , x 2 , ..., x k ] and y =

Original: [ y 1 , y 2 , ..., y k ] .
Revised: [ y 1 , y 2 , ..., y k ] .

Original: Then the inner product estimation of global and local weight sharing are unbiased.
Revised: Then the inner product estimation of global and local weight sharing are unbiased.

Original: [Equation]
Revised: [Equation]

Original: The variance for inner product estimation can be written as,
Revised: The variance for inner product estimation can be written as,

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: b
Revised: b

Original: [Equation]
Revised: [Equation]

Original: 
Revised: 

Original:  , where x l = ( a 1 , a 2 ..., a n l ) and y l = ( b 1 , b 2 ..., b n l ) (8) where V L is local memory sharing variance and V G is global memory sharing variance.
Revised:  , where x l = ( a 1 , a 2 ..., a n l ) and y l = ( b 1 , b 2 ..., b n l ) (8) where V L is local memory sharing variance and V G is global memory sharing variance.

Original: Intuition: The two terms in V G can be understood as follows:
Revised: Intuition: The two terms in V G can be understood as follows:

Original: The first term is the local variance with individual terms reduced by a factor of f i .
Revised: The first term is the local variance with individual terms reduced by a factor of f i .

Original: This is because each piece of the vector is being distributed in a memory that is 1 /f i × larger.
Revised: This is because each piece of the vector is being distributed in a memory that is 1 /f i × larger.

Original: However, in GMS, there is a possibility of more collisions across pieces.
Revised: However, in GMS, there is a possibility of more collisions across pieces.

Original: This leads to the second term in V G .
Revised: This leads to the second term in V G .

Original: Note that, for a given x, y and a finite value for m , V G is always bounded.
Revised: Note that, for a given x, y and a finite value for m , V G is always bounded.

Original: At the same time, V L is unbounded due to 0 < f i < 1 in the denominator.
Revised: At the same time, V L is unbounded due to 0 < f i < 1 in the denominator.

Original: So if the number of pieces increases or particular f i grows smaller, V L increases.
Revised: So if the number of pieces increases or particular f i grows smaller, V L increases.

Original: While we cannot prove that V G is strictly less than V L , we can investigate the equation under some assumptions on the data.
Revised: While we cannot prove that V G is strictly less than V L , we can investigate the equation under some assumptions on the data.

Original: Practically, each piece of the parameter vector is a computational block like a matrix for multiplication or embedding table lookup.
Revised: Practically, each piece of the parameter vector is a computational block like a matrix for multiplication or embedding table lookup.

Original: These blocks are initialized at a scale proportional to the square root of their size.
Revised: These blocks are initialized at a scale proportional to the square root of their size.

Original: So the norms of these vectors are similar.
Revised: So the norms of these vectors are similar.

Original: Let us assume the norm of each piece to be √ α .
Revised: Let us assume the norm of each piece to be √ α .

Original: Also, let us assume that over random data distributions over x and y , all the inner products to be β in expectation.
Revised: Also, let us assume that over random data distributions over x and y , all the inner products to be β in expectation.

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: Thus, V L is greater than V G , and it can be much greater depending on the exact values of f i .
Revised: Thus, V L is greater than V G , and it can be much greater depending on the exact values of f i .

Original: The proof of the theorem and other details are presented in Appendix B.
Revised: The proof of the theorem and other details are presented in Appendix B.

Original:  E XPERIMENTAL EVALUATION
Revised:  E XPERIMENTAL EVALUATION

Original: Setup:
Revised: Setup:

Original: In this section, we evaluate the ROAST compression approach on two types of tasks.
Revised: In this section, we evaluate the ROAST compression approach on two types of tasks.

Original: The details of the tasks, datasets and models used are mentioned in table 2. .
Revised: The details of the tasks, datasets and models used are mentioned in table 2. .

Original: For image-classification tasks, we choose the cifar-10 dataset and the leader for the DawnBenchmark (Coleman et al., 2017) - a ResNet-9 model 1 for cifar-10.
Revised: For image-classification tasks, we choose the cifar-10 dataset and the leader for the DawnBenchmark (Coleman et al., 2017) - a ResNet-9 model 1 for cifar-10.

Original: The target accuracy for this benchmark is 94% and hence we perform hyper-parameter tuning to get a test accuracy of ≥ 94% .
Revised: The target accuracy for this benchmark is 94% and hence we perform hyper-parameter tuning to get a test accuracy of ≥ 94% .

Original: We stop the tuning once we
Revised: We stop the tuning once we

Original: [Table] [Table]
Revised: [Table] [Table]

Original: reach this accuracy and hence the results for CIFAR-10 should be compared w.r.t whether it crosses 94.0%.
Revised: reach this accuracy and hence the results for CIFAR-10 should be compared w.r.t whether it crosses 94.0%.

Original: For NLP tasks, we use two largest available text-classification datasets on huggingface (HuggingFace, 2022).
Revised: For NLP tasks, we use two largest available text-classification datasets on huggingface (HuggingFace, 2022).

Original: For the model, we use BERT-x-y
Revised: For the model, we use BERT-x-y

Original: (x:number of layers, y:number of attention heads) architecture with classification head.
Revised: (x:number of layers, y:number of attention heads) architecture with classification head.

Original: On both NLP datasets, using models larger than BERT-22 lead to similar test accuracy and hence we choose BERT-2-2 as the base model.
Revised: On both NLP datasets, using models larger than BERT-22 lead to similar test accuracy and hence we choose BERT-2-2 as the base model.

Original: The other hyper parameters for NLP tasks are { batch 64 for amazon-polarity and 32 for yelp-polarity, learning rate 2e-5, AdamW optimizer, Linear scheduler}
Revised: The other hyper parameters for NLP tasks are { batch 64 for amazon-polarity and 32 for yelp-polarity, learning rate 2e-5, AdamW optimizer, Linear scheduler}

Original: Roast for compression As we can see in tables 3 and 4 , with ROAST, we can achieve similar quality of model in much smaller space.
Revised: Roast for compression As we can see in tables 3 and 4 , with ROAST, we can achieve similar quality of model in much smaller space.

Original: Specifically, for text-classification, we see that we can train and deploy the BERT-2-2 model in 100 × lesser space.
Revised: Specifically, for text-classification, we see that we can train and deploy the BERT-2-2 model in 100 × lesser space.

Original: Similarly, we can train and deploy ResNet model in 10 × lesser space for same target test accuracy.
Revised: Similarly, we can train and deploy ResNet model in 10 × lesser space for same target test accuracy.

Original: Thus, ROAST is an effective method for training and deploying models on memory-constrained systems.
Revised: Thus, ROAST is an effective method for training and deploying models on memory-constrained systems.

Original: Managing excess parameters It is clear from table 3, that BERT-base architecture is highly over parameterized for the tasks under consideration.
Revised: Managing excess parameters It is clear from table 3, that BERT-base architecture is highly over parameterized for the tasks under consideration.

Original: However, even in this case, ROAST can be used to control the memory footprint while maintaining the functional form of the larger model.
Revised: However, even in this case, ROAST can be used to control the memory footprint while maintaining the functional form of the larger model.

Original: Pruning and ROAST We perform unstructured iterative-magnitude pruning (Han et al., 2016b) on
Revised: Pruning and ROAST We perform unstructured iterative-magnitude pruning (Han et al., 2016b) on

Original: ResNet model and find that pruning gives upto 100 × compression.
Revised: ResNet model and find that pruning gives upto 100 × compression.

Original: However note that pruning requires us to train the model using memory required to store the original model.
Revised: However note that pruning requires us to train the model using memory required to store the original model.

Original: However, compression with ROAST means using lesser memory even for training.
Revised: However, compression with ROAST means using lesser memory even for training.

Original: Additionally, pruning can be used in conjunction with ROAST to obtain smaller models using smaller memory.
Revised: Additionally, pruning can be used in conjunction with ROAST to obtain smaller models using smaller memory.

Original: In table 4, we see that we can prune 90% of weights in 10 × compressed ROAST array and still achieve the same quality.
Revised: In table 4, we see that we can prune 90% of weights in 10 × compressed ROAST array and still achieve the same quality.

Original: Local vs. Global memory sharing In the figure 3, we show that the quality of the model while using global memory sharing is, indeed, better than local memory sharing.
Revised: Local vs. Global memory sharing In the figure 3, we show that the quality of the model while using global memory sharing is, indeed, better than local memory sharing.

Original: This supports our theoretical observation about these memory sharing schemes.
Revised: This supports our theoretical observation about these memory sharing schemes.

Original: Efficiency of ROAST operators as compared to HashedNet Table 7 shows the inference performance of a simple model using ROAST-MM for matrix multiplication on compressed memory.
Revised: Efficiency of ROAST operators as compared to HashedNet Table 7 shows the inference performance of a simple model using ROAST-MM for matrix multiplication on compressed memory.

Original: Our model linearly transforms the input vector and computes its norm.
Revised: Our model linearly transforms the input vector and computes its norm.

Original: We optimized the ROAST-MM kernel for this experiment using the inference-optimal strategy.
Revised: We optimized the ROAST-MM kernel for this experiment using the inference-optimal strategy.

Original: We make the following observations
Revised: We make the following observations

Original: [Table] [Table]
Revised: [Table] [Table]

Original: from Table 7: (1) ROAST-MM outperforms HashedNet kernel consistently across the different multiplication workloads. On an average over different workloads, ROAST-MM is up to 45 × faster than
Revised: 128MB 0.

Original: HashedNet. (2) ROAST-MM is 1 .
Revised: 256MB 0. ROAST 512MB 0.

Original: 34 × slower than PyTorch-MM. This is expected as Pytorch-MM uses extremely optimized libraries for matrix multiplication and ROAST-MM implementation is comparatively under-optimized.
Revised: [Figure]

Original: It is stillinteresting tonote thatROAST-MM’s performance better in terms of scaling efficiency than PyTorch-MM with the increase in workload.
Revised: comparatively still to that terms of ROAST

Original: As the workload increases 1600 × (from 512 × 512 to 20480 × 20480), PyTorch-MM takes 39 × time, HashedNet takes 106 × time whereas ROAST-MM only takes around16 × time.
Revised: MM only takes around

Original: We present moredetailed measurements across different optimizers for training-optimal strategy in the appendingC.
Revised: We present more measurements across different optimizers for training optimal strategy in the appending

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Traditionally model compression has focused on memory reduction during inference.
Revised: Traditionally model compression has focused on memory reduction during inference.

Original: However, model memory during training is also an important consideration.
Revised: However, model memory during training is also an important consideration.

Original: While some of the existing methods such as HashedNet and Low-rank factorisation provide model reduction during training, these methods either do not provide cache-efficient model recovery or have implicit cap on memory reduction.
Revised: While some of the existing methods such as HashedNet and Low-rank factorisation provide model reduction during training, these methods either do not provide cache-efficient model recovery or have implicit cap on memory reduction.

Original: ROAST overcomes these obstacles and provides a cache-efficient, arbitrary control over the memory footprint of model during training and inference.
Revised: ROAST overcomes these obstacles and provides a cache-efficient, arbitrary control over the memory footprint of model during training and inference.

Original: ROAST, essentially provides a practical parameter sharing method.
Revised: ROAST, essentially provides a practical parameter sharing method.

Original: ROAST is theoretically better than HashedNet in terms of dimensionality reduction due to block based hashing and global memory sharing.
Revised: ROAST is theoretically better than HashedNet in terms of dimensionality reduction due to block based hashing and global memory sharing.

Original: We empirically validate the efficiency advantage of ROAST over HashedNet and that we can achieve high compression with ROAST.R A
Revised: We empirically validate the efficiency advantage of ROAST over HashedNet and that we can achieve high compression with ROAST.

Original: A DDITIONAL D ATA FOR R EVIEWERS - PARTS OF WHICH WILL GO IN MAIN PAPER A. E XTENDED
Revised: WITH

Original: [Table]
Revised: [Table]

Original: We add a lot of information and new results to the table.
Revised: We add a lot of information and new results to the table.

Original: Specifically,
Revised: Specifically,

Original: • We add the GMS and LMS results to the table separately.
Revised: • We add the GMS and LMS results to the table separately.

Original: So that readers can get an idea of each of the method on the task.
Revised: So that readers can get an idea of each of the method on the task.

Original: • We add unstructured pruning (best pruning quality wise) resutls for NLP tasks as well.
Revised: • We add unstructured pruning (best pruning quality wise) resutls for NLP tasks as well.

Original: The pruning results are obtained in the following manner.
Revised: The pruning results are obtained in the following manner.

Original: With the full-9-1 schedule, we start from the fully trained model, perform iterative pruning during next 9 epochs and then tune the final pruned model for 1 more epoch.
Revised: With the full-9-1 schedule, we start from the fully trained model, perform iterative pruning during next 9 epochs and then tune the final pruned model for 1 more epoch.

Original: Whereas in the full-1-9 schedule, we again start from the fully trained model, perform pruning in the next 1 epoch and then tune the model further for 9 epochs.
Revised: Whereas in the full-1-9 schedule, we again start from the fully trained model, perform pruning in the next 1 epoch and then tune the model further for 9 epochs.

Original: We note the best achieved accuracy with the final model structure and the epoch at which this accuracy is reached.
Revised: We note the best achieved accuracy with the final model structure and the epoch at which this accuracy is reached.

Original: • For each result, we note the number of epoch when the best accuracy was reached.
Revised: • For each result, we note the number of epoch when the best accuracy was reached.

Original: • We append an additional small table which notes the number of epochs required to reach a target accuracy to compare the convergence of different models.
Revised: • We append an additional small table which notes the number of epochs required to reach a target accuracy to compare the convergence of different models.

Original: We make the following observations.
Revised: We make the following observations.

Original: • GMS reaches better accuracy than LMS for the same amount of compression for both the datasets.
Revised: • GMS reaches better accuracy than LMS for the same amount of compression for both the datasets.

Original: Additionally, GMS reaches the same target accuracy faster than the LMS.
Revised: Additionally, GMS reaches the same target accuracy faster than the LMS.

Original: • The ROAST approach is more effective than pruning approaches in NLP tasks of textclassification for architectures like BERT.
Revised: • The ROAST approach is more effective than pruning approaches in NLP tasks of textclassification for architectures like BERT.

Original: • It is interesting that GMS-10 × converges faster than original model on both datasets.
Revised: • It is interesting that GMS-10 × converges faster than original model on both datasets.

Original: We leave investigating this as future work.
Revised: We leave investigating this as future work.

Original: A. GMS VS LMS FOR Y ELP
Revised: A. GMS VS LMS FOR Y ELP

Original: As can be seen from the two plots in figure4, it is clear the GMS performs superior to LMS in both the compression settings.
Revised: As can be seen from the two plots in figure4, it is clear the GMS performs superior to LMS in both the compression settings.

Original: [Figure]
Revised: [Figure]

Original: B T HEORY
Revised: B T HEORY

Original: ROAST is a generalized model compression which performs operation specific system-friendly lookup and global memory sharing.
Revised: ROAST is a generalized model compression which performs operation specific system-friendly lookup and global memory sharing.

Original: This raises some interesting theoretical questions
Revised: This raises some interesting theoretical questions

Original: B. B ACKWARD PASS FOR MODEL SHARING WEIGHTS ACROSS DIFFERENT COMPONENTS
Revised: B. B ACKWARD PASS FOR MODEL SHARING WEIGHTS ACROSS DIFFERENT COMPONENTS

Original: A general function sharing a weight, say x across different components can be written as , f ( x, g ( x ))
Revised: A general function sharing a weight, say x across different components can be written as , f ( x, g ( x ))

Original: The interpretation is that x was used in g(.) and then again used ahead in f. (In case of MLP, we can think of x being used in multiple layers)
Revised: The interpretation is that x was used in g(.) and then again used ahead in f. (In case of MLP, we can think of x being used in multiple layers)

Original: Let f ( g 1 , g 2 ) where both g 1 and g 2 are functions of x .
Revised: Let f ( g 1 , g 2 ) where both g 1 and g 2 are functions of x .

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: g 1 = x and g 2 = g ( x )
Revised: g 1 = x and g 2 = g ( x )

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Renaming,
Revised: Renaming,

Original: [Equation]
Revised: [Equation]

Original: Thus, we can essentially consider each place where x appears as new variables and then gradient w.r.t x is just summation of partial derivatives of the function w.r.t these new variables.
Revised: Thus, we can essentially consider each place where x appears as new variables and then gradient w.r.t x is just summation of partial derivatives of the function w.r.t these new variables.

Original: Thus, it is easy to implement this in the backward pass.
Revised: Thus, it is easy to implement this in the backward pass.

Original: In order to make sure that the memory utilization in backward pass is not of the order of the recovered model size, we do not use the auto-differentiation of tensorflow/pytorch.
Revised: In order to make sure that the memory utilization in backward pass is not of the order of the recovered model size, we do not use the auto-differentiation of tensorflow/pytorch.

Original: We implement our own backward pass and it can be found in the code.
Revised: We implement our own backward pass and it can be found in the code.

Original: B. G LOBAL FEATURE HASHING VS LOCAL FEATURE HASHING .
Revised: B. G LOBAL FEATURE HASHING VS LOCAL FEATURE HASHING .

Original: We can consider model compression techniques as dimensionality reduction of the parameter vector (a one dimensional vector of all parameters in a model) of size n into a vector of size |M|
Revised: We can consider model compression techniques as dimensionality reduction of the parameter vector (a one dimensional vector of all parameters in a model) of size n into a vector of size |M|

Original: = m .
Revised: = m .

Original: Quality of inner-product preservation is used as a metric to measure the quality of dimensionality reduction.
Revised: Quality of inner-product preservation is used as a metric to measure the quality of dimensionality reduction.

Original: In terms of dimensionality reduction, ROAST uses ROBE hashing Desai et al.
Revised: In terms of dimensionality reduction, ROAST uses ROBE hashing Desai et al.

Original: (2022), which showed that chunk based hashing is theoretically better than hashing individual elements.
Revised: (2022), which showed that chunk based hashing is theoretically better than hashing individual elements.

Original: In this section, we analyse GMS proposal of ROAST against LMS of HashedNet.
Revised: In this section, we analyse GMS proposal of ROAST against LMS of HashedNet.

Original: For the purpose of this comparison we assume a chunk size of 1.
Revised: For the purpose of this comparison we assume a chunk size of 1.

Original: Consider two parameter vectors x, y ∈ R n .
Revised: Consider two parameter vectors x, y ∈ R n .

Original: We are interested in how inner product between these parameter vectors are preserved under hashing.
Revised: We are interested in how inner product between these parameter vectors are preserved under hashing.

Original: Let x =
Revised: Let x =

Original: [ x 1 x 2 ...x k ] and y =
Revised: [ x 1 x 2 ...x k ] and y =

Original: [ y 1 y 2 ...
Revised: [ y 1 y 2 ...

Original: y k ] be composed of k pieces of sizes n 1 , n 2 , ...n k .
Revised: y k ] be composed of k pieces of sizes n 1 , n 2 , ...n k .

Original: In LMS, let each piece be mapped into memory of size f i m where (cid:80) i f i
Revised: In LMS, let each piece be mapped into memory of size f i m where (cid:80) i f i

Original: = 1 .
Revised: = 1 .

Original: The estimators of inner product in the GMS case can be written as ,
Revised: The estimators of inner product in the GMS case can be written as ,

Original: [Equation]
Revised: [Equation]

Original: The estimate of inner product with LMS can be written as,
Revised: The estimate of inner product with LMS can be written as,

Original: [Equation]
Revised: [Equation]

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: The GMS estimator is the standard feature hashing estimator and the LMS is essentially sum of GMS estimators for each of the piece.
Revised: The GMS estimator is the standard feature hashing estimator and the LMS is essentially sum of GMS estimators for each of the piece.

Original: as E [ g ( i )]
Revised: as E [ g ( i )]

Original: = 0 , it is easy to check by linearity of expectations that
Revised: = 0 , it is easy to check by linearity of expectations that

Original: Expectation The suffix L refers to local hashing and G refers to global hashing.
Revised: Expectation The suffix L refers to local hashing and G refers to global hashing.

Original: [Equation]
Revised: [Equation]

Original: Let us now look at the variance.
Revised: Let us now look at the variance.

Original: Let us follow the following notation,
Revised: Let us follow the following notation,

Original: • V G = V ( (cid:91) ⟨ x, y ⟩ G,m ) .
Revised: • V G = V ( (cid:91) ⟨ x, y ⟩ G,m ) .

Original: GMS variance
Revised: GMS variance

Original: of entire vectors
Revised: of entire vectors

Original: • V L = V ( (cid:91) ⟨ x, y ⟩ L,m,⃗f ) .
Revised: • V L = V ( (cid:91) ⟨ x, y ⟩ L,m,⃗f ) .

Original: LMS variance of entire vectors • V l = V ( (cid:92) ⟨ x l , y l ⟩ G,f l m ) .
Revised: LMS variance of entire vectors • V l = V ( (cid:92) ⟨ x l , y l ⟩ G,f l m ) .

Original: variance of each piece
Revised: variance of each piece

Original: we can write V l as follows.
Revised: we can write V l as follows.

Original: The following equation is easy to derive and it can be found the lemmaof Weinberger et al.
Revised: The following equation is easy to derive and it can be found the lemmaof Weinberger et al.

Original: [Equation]
Revised: [Equation]

Original: j ) where x l = ( a 1 , a 2 ...a n l ) and y l = ( b 1 , b 2 ...
Revised: j ) where x l = ( a 1 , a 2 ...a n l ) and y l = ( b 1 , b 2 ...

Original: b n l )
Revised: b n l )

Original: [Equation]
Revised: [Equation]

Original: As, each of the piece is independently hashed in LSM, we can see
Revised: As, each of the piece is independently hashed in LSM, we can see

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Let us now look at V G .
Revised: Let us now look at V G .

Original: Again, using lemma 2 from Weinberger et al.
Revised: Again, using lemma 2 from Weinberger et al.

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: The expression can be split into terms that belong to same pieces and those across pieces
Revised: The expression can be split into terms that belong to same pieces and those across pieces

Original: [Equation]
Revised: [Equation]

Original: piece
Revised: piece

Original: [Equation]
Revised: [Equation]

Original: l
Revised: l

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Observation 1:
Revised: Observation 1:

Original: In V L we can see that there are terms with 1 f l which makes it unbounded.
Revised: In V L we can see that there are terms with 1 f l which makes it unbounded.

Original: It makes sense as if number of pieces increase a lot a lot of compressions will not work for example if number of peices > |M| .
Revised: It makes sense as if number of pieces increase a lot a lot of compressions will not work for example if number of peices > |M| .

Original: Also, it will affect V L a lot when some f l is very small which can often be the case.
Revised: Also, it will affect V L a lot when some f l is very small which can often be the case.

Original: For example, generally embedding tables in DLRM model are much larger than that of matrix multiplciation modules (MLP) .
Revised: For example, generally embedding tables in DLRM model are much larger than that of matrix multiplciation modules (MLP) .

Original: which can make f ≈ 0 .
Revised: which can make f ≈ 0 .

Original: 001 for MLP components.
Revised: 001 for MLP components.

Original: Observation 2: Practically we can assume each piece, no matter the size of the vector, to be of same norm.
Revised: Observation 2: Practically we can assume each piece, no matter the size of the vector, to be of same norm.

Original: The reason lies in initialization.
Revised: The reason lies in initialization.

Original: According to Xavier’s initialization the weights of a particular node are initialized with norm 1.
Revised: According to Xavier’s initialization the weights of a particular node are initialized with norm 1.

Original: So for now lets assume a more practical case of all norms being equal to √ α .
Revised: So for now lets assume a more practical case of all norms being equal to √ α .

Original: Also, in order to make the comparisons we need to consider some average case over the data.
Revised: Also, in order to make the comparisons we need to consider some average case over the data.

Original: So let us assume that under independent randomized data assumption, the expected value of all inner products are β .
Revised: So let us assume that under independent randomized data assumption, the expected value of all inner products are β .

Original: With this , in expectation over randomized data, we have
Revised: With this , in expectation over randomized data, we have

Original: [Equation]
Revised: [Equation]

Original: Now note that,
Revised: Now note that,

Original: [Equation]
Revised: [Equation]

Original: b
Revised: b

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: x l = ( a 1 , a 2 ...a n l ) and y l
Revised: x l = ( a 1 , a 2 ...a n l ) and y l

Original: = ( b 1 , b 2 ...
Revised: = ( b 1 , b 2 ...

Original: b n l )
Revised: b n l )

Original: [Equation]
Revised: [Equation]

Original: 24) (dropping the subscript "l" below)
Revised: 24) (dropping the subscript "l" below)

Original: [Equation]
Revised: [Equation]

Original: Note that for each negative term, there are n l positive terms.
Revised: Note that for each negative term, there are n l positive terms.

Original: To simplify we disregard this term in the equation above.
Revised: To simplify we disregard this term in the equation above.

Original: This is an approximation which is practical and only made to get a sense of V L and V G relation.
Revised: This is an approximation which is practical and only made to get a sense of V L and V G relation.

Original: [Equation]
Revised: [Equation]

Original: Note that we ignored a term which reduces the V L a bit, Let the error be ϵ
Revised: Note that we ignored a term which reduces the V L a bit, Let the error be ϵ

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The above equation shows even for the best case, V G might be slightly more than V L .
Revised: The above equation shows even for the best case, V G might be slightly more than V L .

Original: However for general case where harmonic mean is much worse than arithmetic mean, V L will be much larger depending on exact f l s
Revised: However for general case where harmonic mean is much worse than arithmetic mean, V L will be much larger depending on exact f l s

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 128MB 0.
Revised: 128MB 0.

Original: 256MB 0.
Revised: 256MB 0.

Original: ROAST 512MB 0.
Revised: ROAST 512MB 0.

Original: C ROAST-MM LATENCY MEASUREMENTS C. I NFERENCE OPTIMIZATION C. T RAINING OPTIMIZATION
Revised: C ROAST-MM LATENCY MEASUREMENTS C. I NFERENCE OPTIMIZATION C. T RAINING OPTIMIZATION

Original: See tables 8, 9, 10,forward(ms) (optimized for forward + backward) dim (Matrix dimension = dim x dim)
Revised: See tables 8, 9, 10,forward(ms) (optimized for forward + backward) dim (Matrix dimension = dim x dim)

Original: Memory (mb)
Revised: Memory (mb)

Original: Average
Revised: Average

Original: Full (uncompressed) 0.
Revised: Full (uncompressed) 0.

Original: HashedNet 0.
Revised: HashedNet 0.

Original: [Table]
Revised: [Table]

Original: D V ARIANCE IN QUALITY OVER DIFFERENT RUNS
Revised: D V ARIANCE IN QUALITY OVER DIFFERENT RUNS

Original: The figure 5 shows three runs of ROASTed BERT and BERT models
Revised: The figure 5 shows three runs of ROASTed BERT and BERT models

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: Full 0.
Revised: Full 0.

Original: HashedNet 7.
Revised: HashedNet 7.

Original: adam
Revised: adam

Original: ROAST 7.
Revised: ROAST 7.

Original: Full 0.
Revised: Full 0.

Original: HashedNet
Revised: HashedNet

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: Full 0.
Revised: Full 0.

Original: HashedNet 10.
Revised: HashedNet 10.

Original: adam
Revised: adam

Original: ROAST 9.
Revised: ROAST 9.

Original: Full 0.
Revised: Full 0.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: measurements are taken with
Revised: measurements are taken with