Original: E SPRESSO : E FFICIENT F ORWARD P ROPAGATION FOR B INARY D EEP N EURAL N ETWORKS
Revised: E SPRESSO : E FFICIENT F ORWARD P ROPAGATION FOR B INARY D EEP N EURAL N ETWORKS

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: There are many applications scenarios for which the computational performance and memory footprint of the prediction phase of Deep Neural Networks (DNNs) need to be optimized.
Revised: There are many applications scenarios for which the computational performance and memory footprint of the prediction phase of Deep Neural Networks (DNNs) need to be optimized.

Original: Binary Deep Neural Networks (BDNNs) have been shown to be an effective way of achieving this objective.
Revised: Binary Deep Neural Networks (BDNNs) have been shown to be an effective way of achieving this objective.

Original: In this paper, we show how Convolutional Neural Networks (CNNs) can be implemented using binary representations.
Revised: In this paper, we show how Convolutional Neural Networks (CNNs) can be implemented using binary representations.

Original: Espresso is a compact, yet powerful library written in C/CUDA that features all the functionalities required for the forward propagation of CNNs, in a binary ﬁle less than 400KB, without any external dependencies.
Revised: Espresso is a compact, yet powerful library written in C/CUDA that features all the functionalities required for the forward propagation of CNNs, in a binary ﬁle less than 400KB, without any external dependencies.

Original: Although it is mainly designed to take advantage of massive GPU parallelism, Espresso also provides an equivalent CPU implementation for CNNs.
Revised: Although it is mainly designed to take advantage of massive GPU parallelism, Espresso also provides an equivalent CPU implementation for CNNs.

Original: Espresso provides special convolutional and dense layers for BCNNs, leveraging bit-packing and bit-wise computations for efﬁcient execution.
Revised: Espresso provides special convolutional and dense layers for BCNNs, leveraging bit-packing and bitwise computations for efﬁcient execution.

Original: These techniques provide a speed-up of matrixmultiplication routines, and at the same time, reduce memory usage when storing parameters and activations.
Revised: These techniques provide a speed-up of matrix-multiplication routines, and at the same time, reduce memory usage when storing parameters and activations.

Original: We experimentally show that Espresso is signiﬁcantly faster than existing implementations of optimized binary neural networks ( ≈orders of magnitude).
Revised: We experimentally show that Espresso is signiﬁcantly faster than existing implementations of optimized binary neural networks ( ≈ 2 orders of magnitude).

Original: Espresso is released under the Apache 2.0 license and is available at http://github.com/organization/project .
Revised: Espresso is released under the Apache 2. license and is available at http://github.com/fpeder/espresso .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Convolutional Neural Networks have revolutionized computer vision, pushing the task of object recognition beyond human capabilities (Krizhevsky et al., 2012; Simonyan & Zisserman, 2014;
Revised: Convolutional Neural Networks have revolutionized computer vision, pushing the task of object recognition beyond human capabilities (Krizhevsky et al., 2012; Simonyan & Zisserman, 2014;

Original: Szegedy et al., 2015).
Revised: Szegedy et al., 2015).

Original: Deep Neural Networks (DNN), have also been successfully applied in other ﬁelds, such as speech recognition (Graves et al., 2013; Hinton et al., 2012) and automated translation (Bahdanau et al., 2014; Sutskever et al., 2014).
Revised: Deep Neural Networks (DNN), have also been successfully applied in other ﬁelds, such as speech recognition (Graves et al., 2013; Hinton et al., 2012) and automated translation (Bahdanau et al., 2014; Sutskever et al., 2014).

Original: Despite achieving impressive classiﬁcation accuracy results, DNNs require too much memory and power to be used effectively on embedded or low-power devices.
Revised: Despite achieving impressive classiﬁcation accuracy results, DNNs require too much memory and power to be used effectively on embedded or low-power devices.

Original: Many networks consume a considerable amount of memory.
Revised: Many networks consume a considerable amount of memory.

Original: Memory remains a very limited resource on mobile platforms making harder the usage of trained DNNs 1 .
Revised: Memory remains a very limited resource on mobile platforms making harder the usage of trained DNNs 1 .

Original: Even when memory is not an issue, DNNs remain very computationally intensive, and can quickly drain the battery.
Revised: Even when memory is not an issue, DNNs remain very computationally intensive, and can quickly drain the battery.

Original: Reducing the computational load does not only improve energy efﬁciency, but can also enable further applications.
Revised: Reducing the computational load does not only improve energy efﬁciency, but can also enable further applications.

Original: For example, when processing real-time object classiﬁcation on mobile, being able to perform faster predictions frees up computational resources that can be spent on tasks such as speech recognition and analysis.
Revised: For example, when processing real-time object classiﬁcation on mobile, being able to perform faster predictions frees up computational resources that can be spent on tasks such as speech recognition and analysis.

Original: Therefore, there is a substantial interest in reducing the computational and memory requirements of DNNs.
Revised: Therefore, there is a substantial interest in reducing the computational and memory requirements of DNNs.

Original: Efﬁcient deep neural networks One way to achieve this target is to use specialized hardware for
Revised: Efﬁcient deep neural networks One way to achieve this target is to use specialized hardware for

Original: DNNs.
Revised: DNNs.

Original: Another strategy is to reduce the network’s memory footprint and associated computation, hence increasing its efﬁciency.
Revised: Another strategy is to reduce the network’s memory footprint and associated computation, hence increasing its efﬁciency.

Original: Such solutions are preferable as they can be implemented in software without requiring specialized hardware.
Revised: Such solutions are preferable as they can be implemented in software without requiring specialized hardware.

Original: In our research we follow the software approach, and focus our attention to quantized networks.
Revised: In our research we follow the software approach, and focus our attention to quantized networks.

Original: In this case, the parameters are stored as “small” integers (typically less than 8 -bit) instead of single precision ﬂoating point numbers ( 32 -bit).
Revised: In this case, the parameters are stored as “small” integers (typically less than 8 -bit) instead of single precision ﬂoating point numbers ( 32 -bit).

Original: In particular, we consider the binary deep neural networks (BDNN) proposed by Hubara et al.
Revised: In particular, we consider the binary deep neural networks (BDNN) proposed by Hubara et al.

Original: (2016) where parameters and activations are 1 -bit integers: {− 1 , +1 } .
Revised: (2016) where parameters and activations are 1 -bit integers: {− 1 , +1 } .

Original: At the expense of a relatively small decrease in accuracy, BDNNs can considerably reduce memory usage, and result in faster execution time (i.e. forward propagation).
Revised: At the expense of a relatively small decrease in accuracy, BDNNs can considerably reduce memory usage, and result in faster execution time (i.e. forward propagation).

Original: Further, note that potential hardware implementation of BDNNs would also be cheaper due to the reduced number of required FPUs.
Revised: Further, note that potential hardware implementation of BDNNs would also be cheaper due to the reduced number of required FPUs.

Original: While these results are highly promising, currently only proof-of-concept implementations of BinaryNets have been published (Hubara et al., 2016).
Revised: While these results are highly promising, currently only proof-of-concept implementations of BinaryNets have been published (Hubara et al., 2016).

Original: Therefore, the availability of a ﬂexible end-to-end framework, with particular emphasis placed on computational efﬁciency, can enable further research on BDNNs, as well as its application to practical scenarios.
Revised: Therefore, the availability of a ﬂexible end-to-end framework, with particular emphasis placed on computational efﬁciency, can enable further research on BDNNs, as well as its application to practical scenarios.

Original: Contributions With Espresso we provide an optimized framework for BDNNs capable of achieving state-of-the-art run-time performance with minimal memory footprint while being numerical equivalent to their non-optimized binary counterpart.
Revised: Contributions With Espresso we provide an optimized framework for BDNNs capable of achieving state-of-the-art run-time performance with minimal memory footprint while being numerical equivalent to their non-optimized binary counterpart.

Original: Espresso provides a complete optimized framework for BDNNs supporting both the dense and the convolutional layer.
Revised: Espresso provides a complete optimized framework for BDNNs supporting both the dense and the convolutional layer.

Original: Current state-ofthe-art optimized BDNNs implementations are limited to the fully connected layer, with the serious drawback of not being able to run optimized state-of-art convolutional BDNNs (BCNNs).
Revised: Current state-ofthe-art optimized BDNNs implementations are limited to the fully connected layer, with the serious drawback of not being able to run optimized state-of-art convolutional BDNNs (BCNNs).

Original: While our work is a necessary stepping stone towards optimization of training routines, in this paper we focus on the optimization of forward-propagation (i.e. testing), rather than back-propagation (i.e. training).
Revised: While our work is a necessary stepping stone towards optimization of training routines, in this paper we focus on the optimization of forward-propagation (i.e. testing), rather than back-propagation (i.e. training).

Original: Espresso is designed to have no external dependencies.
Revised: Espresso is designed to have no external dependencies.

Original: This not only results in a highly optimized implementation of BDNNs, but also substantially simpliﬁes its deployment in practical applications, such as those executing on mobile or embedded devices.
Revised: This not only results in a highly optimized implementation of BDNNs, but also substantially simpliﬁes its deployment in practical applications, such as those executing on mobile or embedded devices.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Improving the performance of DNNs can be achieved at either the hardware or software level.
Revised: Improving the performance of DNNs can be achieved at either the hardware or software level.

Original: At the hardware level, chipsets that are dedicated to DNN execution can outperform general-purpose
Revised: At the hardware level, chipsets that are dedicated to DNN execution can outperform general-purpose

Original: CPUs/GPUs (Jouppi, 2017; Han et al., 2016).
Revised: CPUs/GPUs (Jouppi, 2017; Han et al., 2016).

Original: At software level one approach is to design simpler architectures, in terms of overall ﬂoating point operations, that can offer the same accuracy as the original model (Iandola et al., 2016).
Revised: At software level one approach is to design simpler architectures, in terms of overall ﬂoating point operations, that can offer the same accuracy as the original model (Iandola et al., 2016).

Original: Another approach is to prune the weights (Guo et al., 2016), or even entire ﬁlters (Li et al., 2016), that have low impact on the activations such that a simpler model can be derived.
Revised: Another approach is to prune the weights (Guo et al., 2016), or even entire ﬁlters (Li et al., 2016), that have low impact on the activations such that a simpler model can be derived.

Original: These simpliﬁed models can be further compressed by weight sharing (Han et al., 2015).
Revised: These simpliﬁed models can be further compressed by weight sharing (Han et al., 2015).

Original: Finally instead of removing connections, another approach is to quantize the network weights (Courbariaux et al., 2014) such that computations can be executed more efﬁciently.
Revised: Finally instead of removing connections, another approach is to quantize the network weights (Courbariaux et al., 2014) such that computations can be executed more efﬁciently.

Original: Quantized networks In quantized networks, the objective is to train DNNs whose (quantized) weights do not signiﬁcantly impact the network’s classiﬁcation accuracy.
Revised: Quantized networks In quantized networks, the objective is to train DNNs whose (quantized) weights do not signiﬁcantly impact the network’s classiﬁcation accuracy.

Original: For example, Courbariaux et al.
Revised: For example, Courbariaux et al.

Original: (2014) show that 10 -bits are enough for Maxout Networks, and how more efﬁcient multiplications can be performed with ﬁxed-point arithmetic.
Revised: (2014) show that 10 -bits are enough for Maxout Networks, and how more efﬁcient multiplications can be performed with ﬁxed-point arithmetic.

Original: Continuing this trend, more aggressive quantization schemes, up to ternary (Zhu et al., 2016), have also been studied.
Revised: Continuing this trend, more aggressive quantization schemes, up to ternary (Zhu et al., 2016), have also been studied.

Original: Binary Deep Neural Networks (BDNN)
Revised: Binary Deep Neural Networks (BDNN)

Original: Recently, Courbariaux et al.
Revised: Recently, Courbariaux et al.

Original: (2015) showed that a network with binary {− 1 , +1 } weights can achieve near state-of-the-art results on several standard datasets.
Revised: (2015) showed that a network with binary {− 1 , +1 } weights can achieve near state-of-the-art results on several standard datasets.

Original: Binary DNNs (BDNNs) were shown to perform effectively on datasets with relatively small images, such as the permutation-invariant MNIST (LeCun et al., 1998), CIFAR-10 (Krizhevsky et al., 2009) and SVHN (Netzer et al., 2011).
Revised: Binary DNNs (BDNNs) were shown to perform effectively on datasets with relatively small images, such as the permutation-invariant MNIST (LeCun et al., 1998), CIFAR-10 (Krizhevsky et al., 2009) and SVHN (Netzer et al., 2011).

Original: Recently, Rastegari et al.
Revised: Recently, Rastegari et al.

Original: (2016) show that binarized
Revised: (2016) show that binarized

Original: CNNs can perform well even on massive datasets such as ImageNet (Deng et al., 2009) using binarized versions of well-known DNN architectures such as AlexNet (Krizhevsky et al., 2012), ResNet18 (He et al., 2016), and GoogLenet (Szegedy et al., 2015).
Revised: CNNs can perform well even on massive datasets such as ImageNet (Deng et al., 2009) using binarized versions of well-known DNN architectures such as AlexNet (Krizhevsky et al., 2012), ResNet18 (He et al., 2016), and GoogLenet (Szegedy et al., 2015).

Original: Similarly interesting results can be achieved by binarizing both DNN weights and activations as showed by Hubara et al.
Revised: Similarly interesting results can be achieved by binarizing both DNN weights and activations as showed by Hubara et al.

Original: In this work, the authors introduce BinaryNet , a technique to effectively train DNNs where both weights and activations are constrained to {− 1 , +1 } .
Revised: In this work, the authors introduce BinaryNet , a technique to effectively train DNNs where both weights and activations are constrained to {− 1 , +1 } .

Original: BinaryNet achieves nearly state-of-the-art accuracy for MLP training on MNIST and CNN training on CIFAR-10.
Revised: BinaryNet achieves nearly state-of-the-art accuracy for MLP training on MNIST and CNN training on CIFAR-10.

Original: The authors also propose a binary optimized implementation of matrix multiplication which result in 7 × faster performance than the base-line non optimized implementation, and, almost 2 × faster than Theano (Bergstra et al., 2010).
Revised: The authors also propose a binary optimized implementation of matrix multiplication which result in 7 × faster performance than the base-line non optimized implementation, and, almost 2 × faster than Theano (Bergstra et al., 2010).

Original: Their core contributions, namely to replace Floating-point Multiply and Add operations (FMAs) with XNORs and bit-counts , represent the cornerstone over which we build our research.
Revised: Their core contributions, namely to replace Floating-point Multiply and Add operations (FMAs) with XNORs and bit-counts , represent the cornerstone over which we build our research.

Original:  T HE E SPRESSO F RAMEWORK
Revised:  T HE E SPRESSO F RAMEWORK

Original: Espresso provides the user with the necessary tools for executing forward-propagation of DNNs, with particular emphasis placed on convolutional neural networks due to their ubiquitousness in computer vision applications.
Revised: Espresso provides the user with the necessary tools for executing forward-propagation of DNNs, with particular emphasis placed on convolutional neural networks due to their ubiquitousness in computer vision applications.

Original: As the complexity of these networks is cubic to the size of the problem, they are less memory efﬁcient and more computationally intensive than traditional machinelearning algorithms.
Revised: As the complexity of these networks is cubic to the size of the problem, they are less memory efﬁcient and more computationally intensive than traditional machinelearning algorithms.

Original: Identifying the memory and computational bottlenecks of DNNs is therefore essential to enable their practical application.
Revised: Identifying the memory and computational bottlenecks of DNNs is therefore essential to enable their practical application.

Original: In particular, our primary focus is GPU-optimized
Revised: In particular, our primary focus is GPU-optimized

Original: BDNN architectures, which we refer to as GPU opt , but we also support the equivalent ﬂoatingpoint counterparts on heterogeneous architectures, which we refer to as CPU and GPU .
Revised: BDNN architectures, which we refer to as GPU opt , but we also support the equivalent ﬂoatingpoint counterparts on heterogeneous architectures, which we refer to as CPU and GPU .

Original: The CPU and GPU implementations of Espresso do not feature binary optimizations because the data is encoded as single precision ﬂoating point numbers.
Revised: The CPU and GPU implementations of Espresso do not feature binary optimizations because the data is encoded as single precision ﬂoating point numbers.

Original: However they still utilize an optimized library for matrix multiplication.
Revised: However they still utilize an optimized library for matrix multiplication.

Original: Hybrid DNNs The Espresso ’s implementations of tensors and layers come in three variants { CPU , GPU , GPU opt } .
Revised: Hybrid DNNs The Espresso ’s implementations of tensors and layers come in three variants { CPU , GPU , GPU opt } .

Original: A CPU-tensor is allocated in CPU memory, and is processed on the CPU using sequential code.
Revised: A CPU-tensor is allocated in CPU memory, and is processed on the CPU using sequential code.

Original: A GPU-tensor is allocated on GPU main memory and is processed by CUDA kernels.
Revised: A GPU-tensor is allocated on GPU main memory and is processed by CUDA kernels.

Original: Espresso provides functions for converting tensors and layers from one variant to the other, and different variants can also be interconnected with each other.
Revised: Espresso provides functions for converting tensors and layers from one variant to the other, and different variants can also be interconnected with each other.

Original: Consequently, Espresso enables the design of hybrid DNNs consisting of a combination of { CPU , GPU , GPU opt } layers.
Revised: Consequently, Espresso enables the design of hybrid DNNs consisting of a combination of { CPU , GPU , GPU opt } layers.

Original: The computational bottleneck: dot products Dense linear algebra is at the heart of deeplearning as deep networks can be viewed as a composition of matrix-matrix , matrix-vector and elementwise matrix-matrix or vector-vector multiplications.
Revised: The computational bottleneck: dot products Dense linear algebra is at the heart of deeplearning as deep networks can be viewed as a composition of matrix-matrix , matrix-vector and elementwise matrix-matrix or vector-vector multiplications.

Original: The implementation of these dense linear algebra operations relies heavily on the efﬁcient computation of the dot-product .
Revised: The implementation of these dense linear algebra operations relies heavily on the efﬁcient computation of the dot-product .

Original: The execution of this operator consists of (single precision) Floating-point Multiply and Add (FMA) operations.
Revised: The execution of this operator consists of (single precision) Floating-point Multiply and Add (FMA) operations.

Original: In modern architectures, ﬂoating-point multiplications executing on the FPU dominate the complexity of FMAs, and BDNNs address these concerns by replacing FMAs with simpler bitwise operations; see Section 4.
Revised: In modern architectures, ﬂoating-point multiplications executing on the FPU dominate the complexity of FMAs, and BDNNs address these concerns by replacing FMAs with simpler bitwise operations; see Section 4.

Original: Technical highlights The superior computational performance of Espresso derives from three main technical contributions: (1) the use of bit-packing in network layers, (2) better memory layout and management, and (3) the use of custom optimized CUDA kernels.
Revised: Technical highlights The superior computational performance of Espresso derives from three main technical contributions: (1) the use of bit-packing in network layers, (2) better memory layout and management, and (3) the use of custom optimized CUDA kernels.

Original: Through the use of bitpacked layers, Espresso can execute a forward operation without the need for expensive memory re-arrangements employed by existing implementations.
Revised: Through the use of bitpacked layers, Espresso can execute a forward operation without the need for expensive memory re-arrangements employed by existing implementations.

Original: As dynamic memory allocation on GPUs is a performance bottleneck, Espresso implements a custom memory allocator that pre-allocates memory at start-up, and replaces the traditional malloc and free system calls.
Revised: As dynamic memory allocation on GPUs is a performance bottleneck, Espresso implements a custom memory allocator that pre-allocates memory at start-up, and replaces the traditional malloc and free system calls.

Original: Finally, matrix multiplications are performed with CUDA kernels that have been adapted to bit-packing, and only resort to XNORs and bit-counts.
Revised: Finally, matrix multiplications are performed with CUDA kernels that have been adapted to bit-packing, and only resort to XNORs and bit-counts.

Original:  B INARY D EEP N EURAL N ETWORKS (BDNN S )
Revised:  B INARY D EEP N EURAL N ETWORKS (BDNN S )

Original: In this section, we overview the fundamental characteristics of BDNNs (Hubara et al., 2016) that inform the basics of Espresso ’s design.
Revised: In this section, we overview the fundamental characteristics of BDNNs (Hubara et al., 2016) that inform the basics of Espresso ’s design.

Original: In BDNNs, computationally intensive FMA operations are replaced by XNOR (for multiplications) and bit-count (for additions), enabling signiﬁcant computational speed-ups.
Revised: In BDNNs, computationally intensive FMA operations are replaced by XNOR (for multiplications) and bit-count (for additions), enabling signiﬁcant computational speed-ups.

Original: In particular, XNOR is a simpler machine instruction compared to ﬂoating point multiplication, and therefore achieves much higher throughput on many architectures.
Revised: In particular, XNOR is a simpler machine instruction compared to ﬂoating point multiplication, and therefore achieves much higher throughput on many architectures.

Original: More importantly, a single XNOR step can execute multiple 64 -bit wide blocks of dot-products, further increasing the overall computational efﬁciency.
Revised: More importantly, a single XNOR step can execute multiple 64 -bit wide blocks of dot-products, further increasing the overall computational efﬁciency.

Original: In what follows, we describe how a network is binarized, detail a compressed memory layout enabling efﬁcient execution of dot-products, show how to re-interpret input data to allow execution on ﬁxed-precision input (e.g. images), and provide a few notes regarding the training procedure.
Revised: In what follows, we describe how a network is binarized, detail a compressed memory layout enabling efﬁcient execution of dot-products, show how to re-interpret input data to allow execution on ﬁxed-precision input (e.g. images), and provide a few notes regarding the training procedure.

Original: 4. N ETWORK BINARIZATION
Revised: 4. N ETWORK BINARIZATION

Original: A BDNN is composed of a sequence of k = 1 , . . .
Revised: A BDNN is composed of a sequence of k = 1 , . . .

Original: , L layers whose weights W bk and activations a bk are binarized to the values {− 1 , +1 } .
Revised: , L layers whose weights W bk and activations a bk are binarized to the values {− 1 , +1 } .

Original: The superscript b in the notation indicates binary quantities.
Revised: The superscript b in the notation indicates binary quantities.

Original: Weights and activations are {− 1 , +1 } , but at the hardware level they must be encoded as { 0 , 1 } .
Revised: Weights and activations are {− 1 , +1 } , but at the hardware level they must be encoded as { 0 , 1 } .

Original: Our convention is to encode − 1 → 0 and +1 → 1 .
Revised: Our convention is to encode − 1 → 0 and +1 → 1 .

Original: Amongst many possible choices, e.g. stochastic binarization (Courbariaux et al., 2015), we employ the following activation function due to its efﬁcient implementation:
Revised: Amongst many possible choices, e.g. stochastic binarization (Courbariaux et al., 2015), we employ the following activation function due to its efﬁcient implementation:

Original: [Equation]
Revised: [Equation]

Original: 4. B IT - PACKING
Revised: 4. B IT - PACKING

Original: The weights of a BDNN can be stored in the bits of a 64 -bit word.
Revised: The weights of a BDNN can be stored in the bits of a 64 -bit word.

Original: One immediate advantage of bit-packing is to drastically reduce the memory usage by a 32 × factor.
Revised: One immediate advantage of bit-packing is to drastically reduce the memory usage by a 32 × factor.

Original: An even more signiﬁcant advantage is the ability to process multiple values at the same time using registers.
Revised: An even more signiﬁcant advantage is the ability to process multiple values at the same time using registers.

Original: This is particularly useful for dot-products: with bit-packing we can compute a dot-product of 64 element vectors by using just one XNOR and one bit-count.
Revised: This is particularly useful for dot-products: with bit-packing we can compute a dot-product of 64 element vectors by using just one XNOR and one bit-count.

Original: Furthermore, modern computer architectures provide a hardware instruction for counting the number of bits set to 1 in a given word.
Revised: Furthermore, modern computer architectures provide a hardware instruction for counting the number of bits set to 1 in a given word.

Original: Assuming binary vectors a, b ∈ B 1 × N where N is a multiple of 64 , the dot-product is then equivalent to:
Revised: Assuming binary vectors a, b ∈ B 1 × N where N is a multiple of 64 , the dot-product is then equivalent to:

Original: [Equation]
Revised: [Equation]

Original: where (cid:28) represents the bit-shift operator.
Revised: where (cid:28) represents the bit-shift operator.

Original: This simple computation becomes the building block of optimized BDNNs as binary matrix-matrix or matrix-vector operations are computed in this fashion.
Revised: This simple computation becomes the building block of optimized BDNNs as binary matrix-matrix or matrix-vector operations are computed in this fashion.

Original: 4. I NPUT DATA BINARIZATION
Revised: 4. I NPUT DATA BINARIZATION

Original: BDNNs require binary input data, which is not typically available at the ﬁrst layer of the network.
Revised: BDNNs require binary input data, which is not typically available at the ﬁrst layer of the network.

Original: However, the input data usually comes in a ﬁxed precision format (e.g. 8 -bit/channel in RGB images).
Revised: However, the input data usually comes in a ﬁxed precision format (e.g. 8 -bit/channel in RGB images).

Original: Therefore, the optimized computation of dot-products can still be applied if we split the input data according to bit-planes, and then sum back each contribution according to the corresponding weight.
Revised: Therefore, the optimized computation of dot-products can still be applied if we split the input data according to bit-planes, and then sum back each contribution according to the corresponding weight.

Original: For instance, if with (cid:104) a (cid:105) n we indicate the n -th bit of a ﬁxed precision vector, and with i the corresponding bit-plane, we obtain:
Revised: For instance, if with (cid:104) a (cid:105) n we indicate the n -th bit of a ﬁxed precision vector, and with i the corresponding bit-plane, we obtain:

Original: [Equation]
Revised: [Equation]

Original: 4. T RAINING
Revised: 4. T RAINING

Original: When training a BDNN, it is important to note that the gradient is computed with the binary weights, but is accumulated with ﬂoating point precision (Hubara et al., 2016).
Revised: When training a BDNN, it is important to note that the gradient is computed with the binary weights, but is accumulated with ﬂoating point precision (Hubara et al., 2016).

Original: That is because the optimizer needs sufﬁcient precision to make a reliable update.
Revised: That is because the optimizer needs sufﬁcient precision to make a reliable update.

Original: In addition, the derivative of the sign function, which is zero almost everywhere, cannot be used for back-propagation.
Revised: In addition, the derivative of the sign function, which is zero almost everywhere, cannot be used for back-propagation.

Original: To overcome these issues, the straight-through estimator (Bengio et al., 2013) is employed, where 1 is back-propagated if the ﬂoating point argument | x | ≤ 1 , and 0 otherwise.
Revised: To overcome these issues, the straight-through estimator (Bengio et al., 2013) is employed, where 1 is back-propagated if the ﬂoating point argument | x | ≤ 1 , and 0 otherwise.

Original: Finally, during training weights are clipped to [ − 1 , 1] to avoid a large growth of the ﬂoating point weights that would not have an impact on the binary weights.
Revised: Finally, during training weights are clipped to [ − 1 , 1] to avoid a large growth of the ﬂoating point weights that would not have an impact on the binary weights.

Original:  E SPRESSO ARCHITECTURE
Revised:  E SPRESSO ARCHITECTURE

Original: The principal components of our framework are tensors , layers and the network .
Revised: The principal components of our framework are tensors , layers and the network .

Original: These components are organized as a hierarchy.
Revised: These components are organized as a hierarchy.

Original: Tensors are n dimensional matrices used for storing inputs , weights and activations (outputs).
Revised: Tensors are n dimensional matrices used for storing inputs , weights and activations (outputs).

Original: A layer processes an input tensor and produces an output tensor, while a network consists of a concatenation of layers.
Revised: A layer processes an input tensor and produces an output tensor, while a network consists of a concatenation of layers.

Original: 5. T ENSORS
Revised: 5. T ENSORS

Original: In Espresso , each element of a tensor A ∈ R M × N × L is identiﬁed by the triplet m, n, l , where m ∈ [0 , M ) indicates the row, n ∈ [0 , N ) indicates the column, and l ∈ [0 , L ) indicates the channel.
Revised: In Espresso , each element of a tensor A ∈ R M × N × L is identiﬁed by the triplet m, n, l , where m ∈ [0 , M ) indicates the row, n ∈ [0 , N ) indicates the column, and l ∈ [0 , L ) indicates the channel.

Original: A tensor is stored in memory using row-major order with interleaved channels.
Revised: A tensor is stored in memory using row-major order with interleaved channels.

Original: Therefore, according to this layout, the element A m,n,l is found at position ( mN + n ) L + l in linear memory.
Revised: Therefore, according to this layout, the element A m,n,l is found at position ( mN + n ) L + l in linear memory.

Original: [Equation]
Revised: [Equation]

Original: We use the notation A m,n, : , to indicate all the channels of the ( m, n ) -th element.
Revised: We use the notation A m,n, : , to indicate all the channels of the ( m, n ) -th element.

Original: Using the same storing scheme Espresso also deﬁnes bit-packed tensors for GPU opt implementations but with the following changes to further increase its performance.
Revised: Using the same storing scheme Espresso also deﬁnes bit-packed tensors for GPU opt implementations but with the following changes to further increase its performance.

Original: Bit-packing is performed according to the number of channels: when L > 1 bit-packing is done along the l dimension; when L = 1 bitpacking is done along the n dimension.
Revised: Bit-packing is performed according to the number of channels: when L > 1 bit-packing is done along the l dimension; when L = 1 bitpacking is done along the n dimension.

Original: For convolutional layers this packing direction enables
Revised: For convolutional layers this packing direction enables

Original: [Table]
Revised: [Table]

Original: efﬁcient memory access when unrolling/lifting a tensor, which would have not been possible if either m or n had been chosen instead.
Revised: efﬁcient memory access when unrolling/lifting a tensor, which would have not been possible if either m or n had been chosen instead.

Original: More speciﬁcally, this layout is optimal for retrieving a pixel neighborhood as needed by convolution without requiring the layout to be changed.
Revised: More speciﬁcally, this layout is optimal for retrieving a pixel neighborhood as needed by convolution without requiring the layout to be changed.

Original: Further, typically a large number of ﬁlters are used resulting in an increase of tensor dimension in the l direction, while the m and n dimensions are progressively shrunk by pooling layers.
Revised: Further, typically a large number of ﬁlters are used resulting in an increase of tensor dimension in the l direction, while the m and n dimensions are progressively shrunk by pooling layers.

Original: For other layer types, n is the most efﬁcient packing direction, as neurons are stored along rows and their number decreases as we move toward later stages in the network.
Revised: For other layer types, n is the most efﬁcient packing direction, as neurons are stored along rows and their number decreases as we move toward later stages in the network.

Original: 5. L AYERS
Revised: 5. L AYERS

Original: Espresso provides the following layer types: Input , Convolutional , Pooling , Dense (i.e. fully connected) and Batch-normalization .
Revised: Espresso provides the following layer types: Input , Convolutional , Pooling , Dense (i.e. fully connected) and Batch-normalization .

Original: Each layer is characterized by its size, tensor parameters and output.
Revised: Each layer is characterized by its size, tensor parameters and output.

Original: The Espresso API deﬁnes for each layer a forward function that computes the output of a layer given an input tensor, and a function for applying non-linearity to the outputs of convolutional and dense layers.
Revised: The Espresso API deﬁnes for each layer a forward function that computes the output of a layer given an input tensor, and a function for applying non-linearity to the outputs of convolutional and dense layers.

Original: Moreover, the convolutional layer features additional functions for pooling and unrolling .
Revised: Moreover, the convolutional layer features additional functions for pooling and unrolling .

Original: [Figure]
Revised: [Figure]

Original: Convolutional layers
Revised: Convolutional layers

Original: In our framework, 2D convolutions are computed through matrix multiplications – an operation involving a very high reuse of data.
Revised: In our framework, 2D convolutions are computed through matrix multiplications – an operation involving a very high reuse of data.

Original: For both CPU and GPU , this computation is performed by sectioning data in amounts that are cache-friendly (Dongarra et al., 1990), resulting in implementations attaining close to peak computational performance.
Revised: For both CPU and GPU , this computation is performed by sectioning data in amounts that are cache-friendly (Dongarra et al., 1990), resulting in implementations attaining close to peak computational performance.

Original: However, in order to express convolution as matrix multiplication we need to re-organize the input memory appropriately.
Revised: However, in order to express convolution as matrix multiplication we need to re-organize the input memory appropriately.

Original: This is achieved through the unrolling procedure; see Figure 1.
Revised: This is achieved through the unrolling procedure; see Figure 1.

Original: It consists of transforming a tensor into a matrix where each row is formed by unrolling the tensor data contained in each convolution sliding volume.
Revised: It consists of transforming a tensor into a matrix where each row is formed by unrolling the tensor data contained in each convolution sliding volume.

Original: The unrolled matrix is then multiplied by the ﬁlter matrix.
Revised: The unrolled matrix is then multiplied by the ﬁlter matrix.

Original: Finally, the result of the convolution is reordered back to tensor by using the lifting procedure.
Revised: Finally, the result of the convolution is reordered back to tensor by using the lifting procedure.

Original: In Espresso we do need to manually lift the convolution result in order to undo the unrolling: thanks to our tensor representation this happens automatically and at zero cost.
Revised: In Espresso we do need to manually lift the convolution result in order to undo the unrolling: thanks to our tensor representation this happens automatically and at zero cost.

Original: Espresso provides CUDA kernels for the unrolling and pooling of tensors for both GPU and GPU opt implementations.
Revised: Espresso provides CUDA kernels for the unrolling and pooling of tensors for both GPU and GPU opt implementations.

Original: Efﬁcient Matrix multiplication Matrix-vector multiplications are fundamental operations of both dense and CNN layers.
Revised: Efﬁcient Matrix multiplication Matrix-vector multiplications are fundamental operations of both dense and CNN layers.

Original: For the CPU architecture, we use the OpenBLAS library (Xianyi et al.)
Revised: For the CPU architecture, we use the OpenBLAS library (Xianyi et al.)

Original: to implement these operations.
Revised: to implement these operations.

Original: For GPU and GPU opt architectures, the CUDA kernels are based on
Revised: For GPU and GPU opt architectures, the CUDA kernels are based on

Original: MAGMA(sgemm) (Abdelfattah et al., 2016), modiﬁed to make it compatible with our binary data representation.
Revised: MAGMA(sgemm) (Abdelfattah et al., 2016), modiﬁed to make it compatible with our binary data representation.

Original: These kernels for matrix multiplication feature register blocking optimization: since the introduction of Fermi architectures the number of registers have been increased, while register access latency has been substantially reduced compared to shared-memory; hence caching at the register-memory level results in considerably faster throughput (Nath et al., 2010).
Revised: These kernels for matrix multiplication feature register blocking optimization: since the introduction of Fermi architectures the number of registers have been increased, while register access latency has been substantially reduced compared to shared-memory; hence caching at the register-memory level results in considerably faster throughput (Nath et al., 2010).

Original: Espresso ﬁrst fetches the tiles of the two matrices into shared-memory and then process sub-tiles using registers.
Revised: Espresso ﬁrst fetches the tiles of the two matrices into shared-memory and then process sub-tiles using registers.

Original: In the GPU opt variant, we modify the code by replacing blocks of 64 (or blocks of 32 for GPU opt 32 ) single precision multiply and add (FMA) operations with XNOR and bit-count using packed tensors.
Revised: In the GPU opt variant, we modify the code by replacing blocks of 64 (or blocks of 32 for GPU opt 32 ) single precision multiply and add (FMA) operations with XNOR and bit-count using packed tensors.

Original: We also re-tune the kernel block size parameters for improving the performance on reduced size matrices.
Revised: We also re-tune the kernel block size parameters for improving the performance on reduced size matrices.

Original: Zero-padding for convolutions Typical CNN implementations apply a tensor convolution in a “same” conﬁguration, where the sizes of input and output tensors matches.
Revised: Zero-padding for convolutions Typical CNN implementations apply a tensor convolution in a “same” conﬁguration, where the sizes of input and output tensors matches.

Original: This is achieved by zeropadding input tensors, but in convolutional GPU opt layers the zero-padding of the input introduces the side-effect of making the data ternary {− 1 , 0 , +1 } .
Revised: This is achieved by zeropadding input tensors, but in convolutional GPU opt layers the zero-padding of the input introduces the side-effect of making the data ternary {− 1 , 0 , +1 } .

Original: We deal with this problem by treating the data as if it was binary (zero is considered a minus one) and ﬁx the results of the convolution at these corner-cases in post-processing.
Revised: We deal with this problem by treating the data as if it was binary (zero is considered a minus one) and ﬁx the results of the convolution at these corner-cases in post-processing.

Original: This allows us to leave the convolution kernel code – the computational bottleneck of the code – untouched.
Revised: This allows us to leave the convolution kernel code – the computational bottleneck of the code – untouched.

Original: The corner-cases are ﬁxed using a highly efﬁcient kernel which executes an element-wise sum between the results of the convolution and the correction matrix.
Revised: The corner-cases are ﬁxed using a highly efﬁcient kernel which executes an element-wise sum between the results of the convolution and the correction matrix.

Original: The correction matrix is computed once, when the GPU opt layer is loaded, and it simply consists of the convolution of the layer’s weights with a (+1) -padded zero-tensor.
Revised: The correction matrix is computed once, when the GPU opt layer is loaded, and it simply consists of the convolution of the layer’s weights with a (+1) -padded zero-tensor.

Original: Converting a network to Espresso A DNN in Espresso is deﬁned as a combination of layers, which is loaded at run-time by reading its parameters ﬁle.
Revised: Converting a network to Espresso A DNN in Espresso is deﬁned as a combination of layers, which is loaded at run-time by reading its parameters ﬁle.

Original: The parameters ﬁle speciﬁes the storage format of all the layers, as well as their weights.
Revised: The parameters ﬁle speciﬁes the storage format of all the layers, as well as their weights.

Original: Therefore, it completely speciﬁes a DNN as layers are stored sequentially.
Revised: Therefore, it completely speciﬁes a DNN as layers are stored sequentially.

Original: Training of the network is done by BinaryNet (Hubara et al., 2016); the resulting parameters are converted to the Espresso format by utility script distributed together with our sources.
Revised: Training of the network is done by BinaryNet (Hubara et al., 2016); the resulting parameters are converted to the Espresso format by utility script distributed together with our sources.

Original:  E VALUATION
Revised:  E VALUATION

Original: The performance of our framework is evaluated in terms of average computational time needed to perform a particular task.
Revised: The performance of our framework is evaluated in terms of average computational time needed to perform a particular task.

Original: The execution times, averaged over 100 experiments, are obtained on a machine equipped with an NVIDIA GeForce GTX 960 with 2 GB of RAM, and a Intel R (cid:13) dual-Xeon R (cid:13) X5660 @ 2 .
Revised: The execution times, averaged over 100 experiments, are obtained on a machine equipped with an NVIDIA GeForce GTX 960 with 2 GB of RAM, and a Intel R (cid:13) dual-Xeon R (cid:13) X5660 @ 2 .

Original: 80 GHz.
Revised: 80 GHz.

Original: In CPU mode, we conﬁgure the OpenBLAS library for matrix multiplication to use all the 24 available cores.
Revised: In CPU mode, we conﬁgure the OpenBLAS library for matrix multiplication to use all the 24 available cores.

Original: Experimental design We perform three quantitative evaluations: (Section 6.1) matrix multiplications of two dense square matrices of size 8192 × 8192 ; (Section 6.2) forward-propagations of a Multi-Layer Perceptron (MLP) trained on the MNIST dataset (LeCun et al., 1998); (Section 6.3) forward-propagations of a Convolutional Neural Network (CNN) trained on the CIFARdataset (Krizhevsky et al., 2009).
Revised: Experimental design We perform three quantitative evaluations: (Section 6.1) matrix multiplications of two dense square matrices of size 8192 × 8192 ; (Section 6.2) forward-propagations of a Multi-Layer Perceptron (MLP) trained on the MNIST dataset (LeCun et al., 1998); (Section 6.3) forward-propagations of a Convolutional Neural Network (CNN) trained on the CIFARdataset (Krizhevsky et al., 2009).

Original: By Using the same models and datasets, we compare Espresso with: (1) the author provided optimized implementation of BinaryNet (Courbariaux et al., 2015); (2) the optimized BDNN implemented in the Intel Nervana neon framework (NervanaSystems); (3) a self-comparison across { CPU , GPU , GPU opt } as no binary-optimized implementations of convolutional layers are publicly available.
Revised: By Using the same models and datasets, we compare Espresso with: (1) the author provided optimized implementation of BinaryNet (Courbariaux et al., 2015); (2) the optimized BDNN implemented in the Intel Nervana neon framework (NervanaSystems); (3) a self-comparison across { CPU , GPU , GPU opt } as no binary-optimized implementations of convolutional layers are publicly available.

Original: Espresso is numerically equivalent to BinaryNet in terms of classiﬁcation accuracy.
Revised: Espresso is numerically equivalent to BinaryNet in terms of classiﬁcation accuracy.

Original: Therefore our evaluation focuses on computation speed.
Revised: Therefore our evaluation focuses on computation speed.

Original: Public datasets The MNIST dataset (LeCun et al., 1998) consists of 60 K instances for training and, 10 K instances for testing.
Revised: Public datasets The MNIST dataset (LeCun et al., 1998) consists of 60 K instances for training and, 10 K instances for testing.

Original: Each instance is a 28 × 28 grayscale image that depicts digits ranging from 0 to 9 .
Revised: Each instance is a 28 × 28 grayscale image that depicts digits ranging from 0 to 9 .

Original: The CIFAR-10 dataset (Krizhevsky et al., 2009), consists of 50 K training instances and 10 K testing instances of 32 × 32 × 3 color images.
Revised: The CIFAR-10 dataset (Krizhevsky et al., 2009), consists of 50 K training instances and 10 K testing instances of 32 × 32 × 3 color images.

Original: Images are subdivided into 10 classes (airplanes, automobiles, birds, cats, deers, dogs, frogs, horses, ships and trucks).
Revised: Images are subdivided into 10 classes (airplanes, automobiles, birds, cats, deers, dogs, frogs, horses, ships and trucks).

Original: Since our interest is to asses the real-time performance of binary optimized DNNs, in those experiment we use a batch-size of one, and measure the averaged forward time for each image of the testing-sets for each dataset.
Revised: Since our interest is to asses the real-time performance of binary optimized DNNs, in those experiment we use a batch-size of one, and measure the averaged forward time for each image of the testing-sets for each dataset.

Original: 6. B INARY DENSE MATRIX MULTIPLICATION
Revised: 6. B INARY DENSE MATRIX MULTIPLICATION

Original: [Table]
Revised: [Table]

Original: In computing dense matrix multiplication, Espresso outperforms BinaryNet by a ≈ 8 × factor.
Revised: In computing dense matrix multiplication, Espresso outperforms BinaryNet by a ≈ 8 × factor.

Original: Much of the gain can be attributed to our optimized kernels, and the use of register blocking: by fetching bigger data from main memory and shared memory, our kernel increases the bandwidth utilization by decreasing the number of memory fetch instructions.
Revised: Much of the gain can be attributed to our optimized kernels, and the use of register blocking: by fetching bigger data from main memory and shared memory, our kernel increases the bandwidth utilization by decreasing the number of memory fetch instructions.

Original: The use of 64 -bit packing instead of the 32 -bit (such as that of BinaryNet), introduces an additional performance improvement.
Revised: The use of 64 -bit packing instead of the 32 -bit (such as that of BinaryNet), introduces an additional performance improvement.

Original: The 64 -bit kernel achieves a memory DRAM throughput of 40 GB s − 1 for reads and 5 GB s − 1 for writes, while the 32 -bit kernel obtain 29 .
Revised: The 64 -bit kernel achieves a memory DRAM throughput of 40 GB s − 1 for reads and 5 GB s − 1 for writes, while the 32 -bit kernel obtain 29 .

Original: 6 GB s − 1 for reads and 3 .
Revised: 6 GB s − 1 for reads and 3 .

Original: 6 GB s − 1 for writes.
Revised: 6 GB s − 1 for writes.

Original: This translates into the resulting ≈ 25% speed improvement.
Revised: This translates into the resulting ≈ 25% speed improvement.

Original: 6. M ULTI - LAYER PERCEPTRON ON MNISTBinaryNet
Revised: 6. M ULTI - LAYER PERCEPTRON ON MNIST

Original: [Table]
Revised: [Table]

Original: We evaluate the average classiﬁcation execution time over the MNIST dataset, where we trained the MLP architecture from (Courbariaux et al., 2016, Sec 2.1) with author-provided sources, and then converted it to Espresso ’s format.
Revised: We evaluate the average classiﬁcation execution time over the MNIST dataset, where we trained the MLP architecture from (Courbariaux et al., 2016, Sec 2.1) with author-provided sources, and then converted it to Espresso ’s format.

Original: In Table 2, Espresso achieves a consistent speed-up of ≈ 68 × when compared to BinaryNet.
Revised: In Table 2, Espresso achieves a consistent speed-up of ≈ 68 × when compared to BinaryNet.

Original: As the Nervana/neon implementation of binary network is a BinaryNet derivative, it is affected by the same drawbacks of BinaryNet, and hence achieves comparable performance.
Revised: As the Nervana/neon implementation of binary network is a BinaryNet derivative, it is affected by the same drawbacks of BinaryNet, and hence achieves comparable performance.

Original: Both alternatives have the additional cost of running CUDA through
Revised: Both alternatives have the additional cost of running CUDA through

Original: Python/Theano which may introduce further latency in the process.
Revised: Python/Theano which may introduce further latency in the process.

Original: In Table 2, the evaluation over the three variants of Espresso shows the expected outcome, with the GPU opt implementation leading the ranking.
Revised: In Table 2, the evaluation over the three variants of Espresso shows the expected outcome, with the GPU opt implementation leading the ranking.

Original: Note that we are able to achieve a speedup of ≈ 12 × on an NVIDIA GTX 960 ( ≈ 2 . 5 TFLOPs), although this device has only roughly four times more throughput than the Xeon
Revised: Note that we are able to achieve a speedup of ≈ 12 × on an NVIDIA GTX 960 ( ≈ 2 . 5 TFLOPs), although this device has only roughly four times more throughput than the Xeon

Original: X5660 ( ≈ 500 GFLOPs without turbo-boost).
Revised: X5660 ( ≈ 500 GFLOPs without turbo-boost).

Original: Through binary optimization, we are able to further increase the performance to ≈ 15 × with respect to the GPU implementation.
Revised: Through binary optimization, we are able to further increase the performance to ≈ 15 × with respect to the GPU implementation.

Original: We attribute our computational gains to (1) the use of binary-optimized layers, (2) our use of optimized kernels for matrix multiplication and (3) Espresso ’s ability to perform binary optimization of the ﬁrst layer.
Revised: We attribute our computational gains to (1) the use of binary-optimized layers, (2) our use of optimized kernels for matrix multiplication and (3) Espresso ’s ability to perform binary optimization of the ﬁrst layer.

Original: Binary optimized layers
Revised: Binary optimized layers

Original: An evident drawback of Binary-Net is the need for binarizing/packing the layer’s parameters every time a forward method is called.
Revised: An evident drawback of Binary-Net is the need for binarizing/packing the layer’s parameters every time a forward method is called.

Original: In the case of binary optimized networks, the cost of packing the parameters is closely related to the cost of multiplication itself.
Revised: In the case of binary optimized networks, the cost of packing the parameters is closely related to the cost of multiplication itself.

Original: There- fore, the reduction of bit-packing function calls leads to a consistent improvement.
Revised: There- fore, the reduction of bit-packing function calls leads to a consistent improvement.

Original: This motivates our choice of designing speciﬁc layers, where bit-packing is done once during network loading.
Revised: This motivates our choice of designing speciﬁc layers, where bit-packing is done once during network loading.

Original: Optimized kernels BinaryNet employs two bit-packing kernels: one for row-packing, the other for column-packing.
Revised: Optimized kernels BinaryNet employs two bit-packing kernels: one for row-packing, the other for column-packing.

Original: Although BinaryNet’s pack-by-rows kernel is slightly slower than ours ( 8% ), the pack-by-columns kernel is signiﬁcantly slower ( ≈ 4 × ) due to non-coalesced accesses to global memory.
Revised: Although BinaryNet’s pack-by-rows kernel is slightly slower than ours ( 8% ), the pack-by-columns kernel is signiﬁcantly slower ( ≈ 4 × ) due to non-coalesced accesses to global memory.

Original: An additional performance gain of ≈ 15% is achieved by swapping matrix-vector in favour of matrix-matrix multiplication kernels when appropriate (i.e. Dense layers with batch size equal to 1 ); for this reason, Espresso also includes the binary-optimized MAGMA(sgemv) kernel.
Revised: An additional performance gain of ≈ 15% is achieved by swapping matrix-vector in favour of matrix-matrix multiplication kernels when appropriate (i.e. Dense layers with batch size equal to 1 ); for this reason, Espresso also includes the binary-optimized MAGMA(sgemv) kernel.

Original: First-layer binary optimization
Revised: First-layer binary optimization

Original: Another important advantage offered by Espresso is the ability to leverage binary optimization in the ﬁrst layer.
Revised: Another important advantage offered by Espresso is the ability to leverage binary optimization in the ﬁrst layer.

Original: Since the ﬁrst stage of a network processes nonbinary data, BinaryNet does not feature binary optimization for this layer.
Revised: Since the ﬁrst stage of a network processes nonbinary data, BinaryNet does not feature binary optimization for this layer.

Original: However if the input data is split into its constituent bit-planes, binary optimization can still be applied.
Revised: However if the input data is split into its constituent bit-planes, binary optimization can still be applied.

Original: In particular, we split the input vector in a matrix of 8 rows, and recombine the result after multiplication by a weighted sum.
Revised: In particular, we split the input vector in a matrix of 8 rows, and recombine the result after multiplication by a weighted sum.

Original: Our experimental results report an overall ≈ 3 × performance boost when comparing the full binary optimized network with one in which the ﬁrst layer is not binary optimized.
Revised: Our experimental results report an overall ≈ 3 × performance boost when comparing the full binary optimized network with one in which the ﬁrst layer is not binary optimized.

Original: Finally, in terms of memory the GPU opt implementation requires 4 .
Revised: Finally, in terms of memory the GPU opt implementation requires 4 .

Original: 57 MB instead of 140 .
Revised: 57 MB instead of 140 .

Original: 6 MB as in the case of non binary optimized implementation, resulting in a saving ≈ 31 × amount of memory.
Revised: 6 MB as in the case of non binary optimized implementation, resulting in a saving ≈ 31 × amount of memory.

Original: 6. C ONVOLUTIONAL N EURAL N ETWORK ON CIFAR- 
Revised: 6. C ONVOLUTIONAL N EURAL N ETWORK ON CIFAR-

Original: [Table]
Revised: [Table]

Original: To the best of our knowledge, no BDNN implementation of binary-optimized CNN layers is publicly available.
Revised: To the best of our knowledge, no BDNN implementation of binary-optimized CNN layers is publicly available.

Original: Our self-evaluation implements the VGGNet -like CNN architecture from Hubara et al.
Revised: Our self-evaluation implements the VGGNet -like CNN architecture from Hubara et al.

Original: (Hubara et al., 2016, Sec. 2.3), and evaluates it across our three modalities: as expected the
Revised: (Hubara et al., 2016, Sec. 2.3), and evaluates it across our three modalities: as expected the

Original: GPU opt implementation achieves signiﬁcantly better performance.
Revised: GPU opt implementation achieves signiﬁcantly better performance.

Original: Unrolling and pooling Note how the GPU implementation offers a slightly better improvement over CPU with respect to the MLP test, with an ≈ 16 × speed-up.
Revised: Unrolling and pooling Note how the GPU implementation offers a slightly better improvement over CPU with respect to the MLP test, with an ≈ 16 × speed-up.

Original: In this experiment, the inherent parallelism of unrolling and pooling, and the GPU higher memory throughput explain the behavior.
Revised: In this experiment, the inherent parallelism of unrolling and pooling, and the GPU higher memory throughput explain the behavior.

Original: Gains are marginal as FMA still represents the computational bottleneck.
Revised: Gains are marginal as FMA still represents the computational bottleneck.

Original: Bit-packing The GPU opt implementation results in a ≈ 5 × performance gain with the respect to GPU .
Revised: Bit-packing The GPU opt implementation results in a ≈ 5 × performance gain with the respect to GPU .

Original: These gains, to binary optimizations, are slightly smaller than those discussed for MLP in
Revised: These gains, to binary optimizations, are slightly smaller than those discussed for MLP in

Original: Section 6.2.
Revised: Section 6.2.

Original: The output of convolutional layers is signiﬁcantly larger than those of MLP’s dense layers, therefore, the computation of bit-packing sign-activation requires more computational effort.
Revised: The output of convolutional layers is signiﬁcantly larger than those of MLP’s dense layers, therefore, the computation of bit-packing sign-activation requires more computational effort.

Original: Finally, in terms of memory the GPU opt implementation requires 1 .
Revised: Finally, in terms of memory the GPU opt implementation requires 1 .

Original: 73 MB instead of 53 .
Revised: 73 MB instead of 53 .

Original: 54 MB as in the case of non binary optimized implementation, resulting in a saving ≈ 31 × amount of memory.
Revised: 54 MB as in the case of non binary optimized implementation, resulting in a saving ≈ 31 × amount of memory.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: In this paper we presented Espresso , a highly optimized forward-propagation framework for both traditional DNNs as well as BCNNs, that supports heterogeneous deployment on CPU and GPU.
Revised: In this paper we presented Espresso , a highly optimized forward-propagation framework for both traditional DNNs as well as BCNNs, that supports heterogeneous deployment on CPU and GPU.

Original: While
Revised: While

Original: BinaryNet and Nervana/neon BDNN implementations are limited to MLP networks, our framework also supports the popular CNN while simultaneously outperforming state-of-the-art implementations of MLP networks.
Revised: BinaryNet and Nervana/neon BDNN implementations are limited to MLP networks, our framework also supports the popular CNN while simultaneously outperforming state-of-the-art implementations of MLP networks.

Original: Espresso is highly-efﬁcient, light-weight and self-contained.
Revised: Espresso is highly-efﬁcient, light-weight and self-contained.

Original: Computation on the GPU side is done though speciﬁcally designed CUDA kernels, which, combined with a more careful handling of memory allocation and bit-packing, allows us to obtain considerable performance improvements.
Revised: Computation on the GPU side is done though speciﬁcally designed CUDA kernels, which, combined with a more careful handling of memory allocation and bit-packing, allows us to obtain considerable performance improvements.

Original: In future work we would like to add training capabilities, and perform additional performance comparisons on larger standard datasets.
Revised: In future work we would like to add training capabilities, and perform additional performance comparisons on larger standard datasets.