B I P OINT N ET : B INARY N EURAL N ETWORK FOR P OINT C LOUDS
A BSTRACT
To alleviate the resource constraint for real-time point cloud applications that run on edge devices, in this paper we present BiPointNet, the ﬁrst model binarization approach for efﬁcient deep learning on point clouds.
We discover that the immense performance drop of binarized models for point clouds mainly stems from two challenges: aggregation-induced feature homogenization that leads to a degradation of information entropy, and scale distortion that hinders optimization and invalidates scale-sensitive structures.
With theoretical justiﬁcations and in-depth analysis, our BiPointNet introduces Entropy-Maximizing Aggregation (EMA) to modulate the distribution before aggregation for the maximum information entropy, and Layer-wise Scale Recovery (LSR) to efﬁciently restore feature representation capacity.
Extensive experiments show that BiPointNet outperforms existing binarization methods by convincing margins, at the level even comparable with the full precision counterpart.
We highlight that our techniques are generic, guaranteeing signiﬁcant improvements on various fundamental tasks and mainstream backbones.
Moreover, BiPointNet gives an impressive 14 .
7 × speedup and 18 .
9 × storage saving on real-world resource-constrained devices.
I NTRODUCTION
With the advent of deep neural networks that directly process raw point clouds (PointNet (Qi et al., 2017a) as the pioneering work), great success has been achieved in learning on point clouds (Qi et al., 2017b;
Li et al., 2018; Wang et al., 2019a; Wu et al., 2019; Thomas et al., 2019; Liu et al., 2019b;
Zhang et al., 2019b).
Point cloud applications, such as autonomous driving and augmented reality, often require real-time interaction and fast response.
However, computation for such applications is usually deployed on resource-constrained edge devices.
To address the challenge, novel algorithms, such as Grid-GCN (Xu et al., 2020b), RandLA-Net (Hu et al., 2020), and PointVoxel (Liu et al., 2019d), have been proposed to accelerate those point cloud processing networks.
While signiﬁcant speedup and memory footprint reduction have been achieved, these works still rely on expensive ﬂoating-point operations, leaving room for further optimization of the performance from the model quantization perspective.
Model binarization (Rastegari et al., 2016; Bulat & Tzimiropoulos, 2019;
Hubara et al., 2016; Wang et al., 2020; Zhu et al., 2019; Xu et al., 2019) emerged as one of the most promising approaches to optimize neural networks for better computational and memory usage efﬁciency.
Binary Neural Networks (BNNs) leverage 1) compact binarized parameters that take small memory space, and 2) highly efﬁcient bitwise operations which are far less costly compared to the ﬂoating-point counterparts.
Despite that in 2D vision tasks (Krizhevsky et al., 2012; Simonyan & Zisserman, 2014; Szegedy et al., 2015; Girshick et al., 2014; Girshick, 2015; Russakovsky et al., 2015; Wang et al., 2019b;
[Figure]
Zhang et al., 2021) has been studied extensively by the model binarization community, the methods developed are not readily transferable for 3D point cloud networks due to the fundamental differences between 2D images and 3D point clouds.
First, to gain efﬁciency in processing unordered 3D points, many point cloud learning methods rely heavily on pooling layers with large receptive ﬁeld to aggregate point-wise features.
As shown in PointNet (Qi et al., 2017b), global pooling provides a strong recognition capability.
However, this practice poses challenges for binarization.
Our analyses show that the degradation of feature diversity, a persistent problem with binarization (Liu et al., 2019a; Qin et al., 2020b; Xie et al., 2017), is signiﬁcantly ampliﬁed by the global aggregation function (Figure 2), leading to homogenization of global features with limited discriminability.
Second, the binarization causes immense scale distortion at the point-wise feature extraction stage, which is detrimental to model performance in two ways: the saturation of forward-propagated features and backward-propagated gradients hinders optimization, and the disruption of the scale-sensitive structures (Figure 3) results in the invalidation of their designated functionality.
In this paper, we provide theoretical formulations of the above-mentioned phenomenons and obtain insights through in-depth analysis.
Such understanding allows us to propose a method that turns fullprecision point cloud networks into extremely efﬁcient yet strong binarized models (see the overview in Figure 1).
To tackle the homogenization of the binarized features after passing the aggregation function, we study the correlation between the information entropy of binarization features and the performance of point cloud aggregation functions.
We thus propose Entropy-Maximizing Aggregation (EMA) that shifts the feature distribution towards the statistical optimum, effectively improving expression capability of the global features.
Moreover, given maximized information entropy, we further develop Layer-wise Scale Recovery (LSR) to efﬁciently restore the output scale that enhances optimization, which allows scale-sensitive structures to function properly.
LSR uses only one learnable parameter per layer, leading to negligible storage increment and computation overhead.
Our BiPointNet is the ﬁrst binarization approaches to deep learning on point clouds, and it outperforms existing binarization algorithms for 2D vision by convincing margins.
It is even almost on par (within ∼ 1-2%) with the full-precision counterpart.
Although we conduct most analysis on the
PointNet baseline, we show that our methods are generic and can be readily extendable to other popular backbones, such as PointNet++ (Qi et al., 2017b), PointCNN (Li et al., 2018), DGCNN (Wang et al., 2019a), and PointConv (Wu et al., 2019), which are the representatives of mainstream categories of point cloud feature extractors.
Moreover, extensive experiments on multiple fundamental tasks on the point cloud, such as classiﬁcation, part segmentation, and semantic segmentation, highlight that our BiPointNet is task-agnostic.
Besides, we highlight that our EMA and LSR are efﬁcient and easy to implement in practice: in the actual test on popular edge devices, BiPointNet achieves 14 .
7 × speedup and 18 .
9 × storage savings compared to the full-precision PointNet.
Our code is released at https://github.com/htqin/BiPointNet.
R ELATED W ORK
Network Binarization.
Recently, various quantization methods for neural networks have emerged, such as uniform quantization (Gong et al., 2019;
Zhu et al., 2020), mixed-precision quantization (Wu et al., 2018; Yu et al., 2020), and binarization.
Among these methods, binarization enjoys compact binarized parameters and highly efﬁcient bitwise operations for extreme compression and acceleration (Rastegari et al., 2016; Qin et al., 2020a).
In general, the forward and backward propagation of binarized models in the training process can be formulated as:
[Equation]
where x denotes the element in ﬂoating-point weights and activations, b denotes the element in binarized weights B w and activations B a .
g x , and g b donate the gradient ∂C∂x and ∂C∂b , respectively, where C is the cost function for the minibatch.
In forward propagation, sign function is directly applied to obtain the binary parameters.
In backward propagation, the Straight-Through Estimator (STE) (Bengio et al., 2013) is used to obtain the derivative of the sign function, avoiding getting all zero gradients.
The existing binarization methods are designed to obtain accurate binarized networks by minimizing the quantization error (Rastegari et al., 2016; Zhou et al., 2016; Lin et al., 2017), improving loss function (Ding et al., 2019;
Hou et al., 2017), reducing the gradient error (Liu et al., 2018; 2020), and designing novel structures and pipelines (Martinez et al., 2020).
Unfortunately, we show in Sec 3 that these methods, designed for 2D vision tasks, are not readily transferable to 3D point clouds.
Deep Learning on Point Clouds.
PointNet (Qi et al., 2017a) is the ﬁrst deep learning model that processes raw point clouds directly.
The basic building blocks proposed by PointNet such as MLP for point-wise feature extraction and max pooling for global aggregation (Guo et al., 2020) have become the popular design choices for various categories of newer backbones: 1) the pointwise
MLP-based such as PointNet++ (Qi et al., 2017b); 2) the graph-based such as DGCNN (Xu et al., 2020b); 3) the convolution-based such as PointCNN (Li et al., 2018), PointConv (Wu et al., 2019)
RS-CNN (Liu et al., 2019c) and KP-Conv (Thomas et al., 2019).
Recently, methods are proposed for efﬁcient deep learning on point clouds through novel data structuring (Xu et al., 2020b), faster sampling (Hu et al., 2020), adaptive ﬁlters
(Xu et al., 2020a), efﬁcient representation (Liu et al., 2019d) or convolution operation (Zhang et al., 2019b) .
However, they still use expensive ﬂoatingpoint parameters and operations, which can be improved by binarization.
M ETHODS
Binarized models operate on efﬁcient binary parameters, but often suffer large performance drop.
Moreover, the unique characteristics of point clouds pose even more challenges.
We observe there are two main problems: ﬁrst, aggregation of a large number of points leads to a severe loss of feature diversity; second, binarization induces an immense scale distortion, that undermines the functionality of scale-sensitive structures.
In this section, we discuss our observations, and propose our BiPointNet with theoretical justiﬁcations.
3. B INARIZATION F RAMEWORK
We ﬁrst give a brief introduction to our framework that binarizes a ﬂoating-point network.
For example, deep learning models on point clouds typically contain multi-layer perceptrons (MLPs) for feature extraction.
In contrast, the binarized models contain binary MLPs (BiMLPs), which are composed of binarized linear (bi-linear) layers.
Bi-linear layers perform the extremely efﬁcient bitwise operations (XNOR and Bitcount) on the lightweight binary weight/activation.
Speciﬁcally, the activation of the bi-linear layer is binarized to B a , and is computed with the binarized weight
B w to obtain the output Z :
[Equation]
where (cid:12) denotes the inner product for vectors with bitwise operations XNOR and Bitcount.
When
B w and B a denote the random variables in B w and B a , we represent their probability mass function as p B w ( b w ) , and p B a ( b a ) .
Moreover, we divide the BiPointNet into units for detailed discussions.
In BiPointNet, the original data or feature X ∈ R n × c ﬁrst enters the symmetric function Ω , which represents a composite function built by stacking several permutation equivariant and permutation invariant layers (e.g., nonlinear layer, bi-linear layer, max pooling).
And then, the output Y ∈ R n × k is binarized to obtain
[Figure]
[Equation]
Similarly, when B , Y and X denote the random variables sampled from B , Y and X , we represent their probability mass function as p B ( b ) , p Y ( y ) and p X ( x ) .
3. E NTROPY - MAXIMIZING A GGREGATION
Unlike images pixels that are arranged in regular lattices, point clouds are sets of points without any speciﬁc order.
Hence, features are usually processed in a point-wise manner and aggregated explicitly through pooling layers.
Our study shows that the aggregation function is a performance bottleneck of the binarized model, due to severe homogenization as shown in Figure 2.
We apply information theory (Section 3.2.1) to quantify the effect of the loss of feature diversity, and ﬁnd that global feature aggregation leads to a catastrophic loss of information entropy.
In Section 3.2.2, we propose the concept of Entropy-Maximizing Aggregation (EMA) that gives the statistically maximum information entropy to effectively tackle the feature homogenization problem.
3.2. A GGREGATION - INDUCED F EATURE H OMOGENIZATION
Ideally, the binarized tensor B should reﬂect the information in the original tensor Y as much as possible.
From the perspective of information, maximizing mutual information can maximize the information ﬂow from the full-precision to the binarized parameters.
Hence, our goal is equivalent to maximizing the mutual information
I ( Y ; B ) of the random variables Y and B :
[Equation]
where H ( B ) is the information entropy, and H ( B | Y ) is the conditional entropy of B given Y .
H ( B | Y ) = 0 as we use the deterministic sign function as the quantizer in binarization (see Section
A.1 for details).
Hence, the original objective function Eq.
(4) is equivalent to:
[Equation]
where B is the set of possible values of B .
We then study the information properties of max pooling, which is a common aggregation function used in popular point cloud learning models such as
PointNet.
Let the max pooling be the last layer φ of the multi-layer stacked Ω , and the input of φ is deﬁned as X φ .
The data ﬂow of Eq.
(3) can be further expressed as B = sign( φ ( X φ )) , and the information entropy H B of binarized feature B can be expressed as
[Equation]
where n is the number of elements aggregated by the max pooling, and X φ is the random variable sampled from X φ .
The brief derivation of Eq (6) is shown in Appendix A.2.
Theorem 1 shows the information properties of max pooling with the normal distribution input on the binarized network architecture.
Theorem 1 For input X φ of max pooling φ with arbitrary distribution, the information entropy of the binarized output goes to zero as n goes to inﬁnity, i.e., lim n → + ∞ H B = 0 .
And there exists a constant c , for any n 1 and n 2 , if n 1 > n 2 > c , we have H B,n 1 < H B,n 2 , where n is the number of elements to be aggregated.
The proof of Theorem 1 is included in Appendix A.2, which explains the severe feature homogenization after global feature pooling layers.
As the number of points is typically large (e.g.points by convention in ModelNet40 classiﬁcation task), it signiﬁcantly reduces the information entropy H B of binarized feature B , i.e., the information of Y is hardly retained in B , leading to highly similar output features regardless of the input features to pooling layer as shown in Figure 2.
Furthermore, Theorem 1 provides a theoretical justiﬁcation for the poor performance of existing binarization methods, transferred from 2D vision tasks to point cloud applications.
In 2D vision, the aggregation functions are often used to gather local features with a small kernel size n (e.g. n = 4 in
ResNet (He et al., 2016; Liu et al., 2018) and VGG-Net (Simonyan & Zisserman, 2014) which use 2 × 2 pooling kernels).
Hence, the feature homogenization problem on images is not as signiﬁcant as that on point clouds.
3.2. EMA FOR M AXIMUM I NFORMATION E NTROPY
Therefore, we need a class of aggregation functions that maximize the information entropy of B to avoid the aggregation-induced feature homogenization.
We study the correlation between the information entropy H B of binary random variable B and the distribution of the full-precision random variable Y .
We notice that the sign function used in binarization has a ﬁxed threshold and decision levels, so we get Proposition 1 about information entropy of B and the distribution of Y .
Proposition 1 When the distribution of the random variable Y satisﬁes (cid:80) y<
[Equation]
Y
[Equation]
y )
=
[Equation]
y ≥
p Y ( y ) = 0 .
5 , the information entropy H B is maximized.
The proof of Proposition 1 is shown in Appendix A.3.
Therefore, theoretically, there is a distribution of Y that can maximize the mutual information of Y and B by maximizing the information entropy of the binary tensor B , so as to maximally retain the information of Y in B .
To maximize the information entropy H B , we propose the EMA for feature aggregation in BiPointNet.
The EMA is not one, but a class of binarization-friendly aggregation layers.
Modifying the aggregation function in the full-precision neural network to a EMA keeps the entropy maximized by input transformation.
The deﬁnition of EMA is
[Equation]
where ϕ denotes the aggregation function (e.g. max pooling and average pooling) and τ denotes the transformation unit.
Note that a standard normal distribution N (0 , 1) is assumed for X φ because batch normalization layers are placed prior to the pooling layers by convention.
τ can take many forms; we discover that a simple constant offset is already effective.
The offset shifts the input so that the output distribution satisﬁes (cid:80) y< 0 p Y ( y ) = 0 .
5 , to maximize the information entropy of binary feature B .
The transformation unit τ in our BiPointNet can be deﬁned as τ ( X φ ) =
X φ − δ ∗ .
When max pooling is applied as ϕ , we obtain the distribution offset δ ∗ for the input X φ that maximizes the information entropy H B by solving the objective function
[Equation]
[Table]
where n denotes the number of elements in each batch.
For each n , we can obtain an optimized δ ∗ max for Eq.
(8), we include the pseudo code in the Appendix A.5.
Moreover, we derive in the Appendix A.6 that when average pooling is used as ϕ , the solution to its objective function is expressed as δ = 0 .
We thus obtain δ ∗ avg = 0 .
This means the solution
[Figure]
In a nutshell, we provide two possible variants of ϕ : ﬁrst, we show that a simple shift is sufﬁcient to turn a max pooling layer into an EMA (EMA-max); second, average pooling can be directly used (EMA-avg) without modiﬁcation as a large number of points does not undermine its information entropy, making it adaptive to the dynamically changing number of point input.
Note that modifying existing aggregation functions is only one way to achieve EMA; the theory also instructs the development of new binarization-friendly aggregation functions in the future.
3. O NE - SCALE - FITS - ALL : L AYER - WISE S CALE R ECOVERY
In this section, we show that binarization leads to feature scale distortion and study its cause.
We conclude that the distortion is directly related to the number of feature channels.
More importantly, we discuss the detriments of scale distortion from the perspectives of the functionality of scalesensitive structures and the optimization.
To address the severe scale distortion in feature due to binarization, we propose the Layer-wise Scale
Recovery (LSR).
In LSR, each bi-linear layer is added only one learnable scaling factor to recover the original scales of all binarized parameters, with negligible additional computational overhead and memory usage.
3.3. S CALE D ISTORTION
[Figure]
The scale of parameters is deﬁned as the standard deviation σ of their distribution.
As we mentioned in Section 3.2, balanced binarized weights are used in the bi-linear layer aiming to maximize the entropy of the output after binarization, i.e., p B w (1) = 0 .and
p B a (1) = 0 .
Theorem 2 When we let p B w (1) = 0 .
5 and p B a (1) = 0 .in bi-linear layer to maximize the mutual information, for the binarized weight B w ∈ {− 1 , +1 } m × k and activation B a ∈ {− 1 , +1 } n × m , the probability mass function for the distribution of output Z can be represented as p Z (2 i − m )
= 0 .
5 m C im , i ∈ { 0 , 1 , 2 , ..., m } .
The output has approximately a normal distribution N (0 , m ) .
The proof of Theorem 2 is found in Appendix A.4.
Theoremshows that given the maximized information entropy, the scale of the output features is directly related to the number of feature channels.
Hence, scale distortion is pervasive as a large number of channels is the design norm of deep learning neural networks for effective feature extraction.
We discuss two major impacts of the scale distortion on the performance of binarized point cloud learning models.
First, the scale distortion invalidates structures designed for 3D deep learning that
[Table]
are sensitive to the scale of values.
For example
[Table]
the T-Net in PointNet is designed to predict an orthogonal transformation matrix for canonicalization of input and intermediate features (Qi et al., 2017a).
The predicted matrix is regularized by minimizing the loss term L reg =
(cid:13)(cid:13)
I − ZZ T (cid:13)(cid:13) 2 .
However, this regularization is ineffective for the Z with huge variance as shown in Figure 3.
Second, the scale distortion leads to a saturation of forward-propagated activations and backwardpropagated gradients (Ding et al., 2019).
In the binary neural networks, some modules (such as sign and Hardtanh ) rely on the Straight-Through Estimator (STE) Bengio et al.
for feature binarization or feature balancing.
When the scale of their input is ampliﬁed, the gradient is truncated instead of increased proportionally.
Such saturation, as shown in Fig 4(c), hinders learning and even leads to divergence.
3.3. LSR FOR O UTPUT S CALE R ECOVERY
To recover the scale and adjustment ability of output, we propose the LSR for bi-linear layers in our
BiPointNet.
We design a learnable layer-wise scaling factor α in our LSR.
α is initialized by the ratio of the standard deviations between the output of bi-linear and full-precision counterpart:
[Equation]
where σ denotes as the standard deviation.
And the α is learnable during the training process.
The calculation and derivative process of the bi-linear layer with our LSR are as follows:
[Equation]
where g α and g Z denotes the gradient ∂C∂α and ∂C∂ Z , respectively.
By applying the LSR in BiPointNet, we mitigate the scale distortion of output caused by binarization.
Compared to existing methods, the advantages of LSR is summarized in two folds.
First, LSR is efﬁcient.
It not only abandons the adjustment of input activations to avoid expensive inference time computation, but also recovers the scale of all weights parameters in a layer collectively instead of expensive restoration in a channel-wise manner (Rastegari et al., 2016).
Second, LSR serves the purpose of scale recovery that we show is more effective than other adaptation such as minimizing quantization errors (Qin et al., 2020b; Liu et al., 2018).
E XPERIMENTS
In this section, we conduct extensive experiments to validate the effectiveness of our proposed BiPointNet for efﬁcient learning on point clouds.
We ﬁrst ablate our method and demonstrate the contributions of EMA and LSR on three most fundamental tasks: classiﬁcation on ModelNet40 (Wu et al., 2015), part segmentation on ShapeNet (Chang et al., 2015), and semantic segmentation on
S3DIS (Armeni et al., 2016).
Moreover, we compare BiPointNet with existing binarization methods where our designs stand out.
Besides, BiPointNet is put to the test on real-world devices with limited computational power and achieve extremely high speedup ( 14 . 7 × ) and storage saving ( 18 . 9 × ).
The details of the datasets and the implementations are included in the Appendix E. are equipped with our
LSR performs others
[Table] [Table]
The techniques in our are generic to point cloud learning.
Hence they are easily extendable
[Table]
4. A BLATION S TUDY
As shown in Table 1, the binarization model baseline suffers a catastrophic performance drop in the classiﬁcation task.
EMA and LSR improve performance considerably when used alone, and they further close the gap between the binarized model and the full-precision counterpart when used together.
In Figure 4, we further validate the effectiveness of EMA and LSR.
We show that BiPointNet with EMA has its information entropy maximized during training, whereas the vanilla binarized network with max pooling gives limited and highly ﬂuctuating results.
Also, we make use of the regularization loss L reg =
(cid:13)(cid:13)
I − ZZ T (cid:13)(cid:13) F for the feature transformation matrix of T-Net in PointNet as an indicator, the L reg of the BiPointNet with LSR is much smaller than the vanilla binarized network, demonstrating LSR’s ability to reduce the scale distortion caused by binarization, allowing proper prediction of orthogonal transformation matrices.
Moreover, we also include the results of two challenging tasks, part segmentation, and semantic segmentation, in Table 1.
As we follow the original PointNet design for segmentation, which concatenates pointwise features with max pooled global feature, segmentation suffers from the information loss caused by the aggregation function.
EMA and LSR are proven to be effective: BiPointNet is approaching the full precision counterpart with only ∼ 4% mIoU difference on part segmentation and ∼ 10 .
4% mIoU gap on semantic segmentation.
The full results of segmentation are presented in Appendix E.6.
4. C OMPARATIVE E XPERIMENTS
In Table 2, we show that our BiPointNet outperforms other binarization methods such as
BNN (Hubara et al., 2016), XNOR (Rastegari et al., 2016), Bi-Real
(Liu et al., 2018), ABC-Net (Lin
ARM Devices
PointNet
BiPointNet
S t
o r a g e U sa g e (
M B ) 18.9 × 67.
A A
PointNet
BiPointNet
T i m e
C o s t ( m s ) 14.7 × 12.1 ×
[Figure]
acy
[Figure]
IR
[Figure]
Net
Ours XNOR++
Bi
[Figure]
BNN
[Figure]
FP
[Figure]
et al., 2017), XNOR++ (Bulat & Tzimiropoulos, 2019), and IR-Net (Qin et al., 2020b).
Although these methods have been proven effective in 2D vision, they are not readily transferable to point clouds due to aggregation-induced feature homogenization.
Even if we equip these methods with our EMA to mitigate information loss, our BiPointNet still performs better.
We argue that existing approaches, albeit having many scaling factors, focus on minimizing quantization errors instead of recovering feature scales, which is critical to effective learning on point clouds.
Hence, BiPointNet stands out with a negligible increase of parameters that are designed to restore feature scales.
The detailed analysis of the performance of XNOR is found in
Appendix C. Moreover, we highlight that our EMA and LSR are generic, and Tableshows improvements across several mainstream categories of point cloud deep learning models, including PointNet (Qi et al., 2017a), PointNet++ (Qi et al., 2017b), PointCNN (Li et al., 2018),
DGCNN (Wang et al., 2019a), and PointConv (Wu et al., 2019).
4. D EPLOYMENT E FFICIENCY ON R EAL - WORLD D EVICES
To further validate the efﬁciency of BiPointNet when deployed into the real-world edge devices, we further implement our BiPointNet on Raspberry Pi 4B with 1.5 GHz 64-bit quad-core ARM CPU
Cortex-A72 and Raspberry Pi 3B with 1.2 GHz 64-bit quad-core ARM CPU Cortex-A53.
We compare our BiPointNet with the PointNet in Figure 5(a) and Figure 5(b).
We highlight that
BiPointNet achieves 14 .
7 × inference speed increase and 18 .
9 × storage reduction over PointNet, which is recognized as a fast and lightweight model itself.
Moreover, we implement various binarization methods over PointNet architecture and report their real speed performance on ARM A
CPU device.
As Figure 5(c), our BiPointNet surpasses all existing binarization methods in both speed and accuracy.
Note that all binarization methods adopt our EMA and report their best accuracy, which is the important premise that they can be reasonably applied to binarize the PointNet.
C ONCLUSION
We propose BiPointNet, the ﬁrst binarization approach for efﬁcient learning on point clouds.
We build a theoretical foundation to study the impact of binarization on point cloud learning models, and proposed EMA and LSR in BiPointNet to improve the performance.
BiPointNet outperforms existing binarization methods, and it is easily extendable to a wide range of tasks and backbones, giving an impressive 14 .
7 × speedup and 18 .
9 × storage saving on resource-constrained devices.
Our work demonstrates the great potential of binarization.
We hope our work can provide directions for future research.
Acknowledgement This work was supported by National Natural Science Foundation of China (62022009, 61872021), Beijing Nova Program of Science and Technology (Z191100001119050), and State Key Lab of Software Development Environment (SKLSDE-2020ZX-06).
A PPENDIX FOR B I P OINT N ET A M AIN P ROOFS AND D ISCUSSION A. P ROOF OF Z ERO C ONDITIONAL E NTROPY
In our BiPointNet, we hope that the binarized tensor B reﬂects the information in the original tensor
Y as much as possible.
From the perspective of information, our goal is equivalent to maximizing the mutual information
I ( Y ; B ) of the random variables Y and B :
[Equation] [Equation]
[Equation] [Equation]
[Equation]
where p ( Y,B ) and p Y , p B are the joint and marginal probability mass functions of these discrete variables.
H ( B ) is the information entropy, and H ( B | Y ) is the conditional entropy of B given Y .
According to the Eq.
(15) and Eq.
(18), the conditional entropy H ( Y | X ) can be expressed as
[Equation]
Since we use the deterministic sign function as the quantizer in binarization, the value of B fully depends on the value of Y , p B | Y
=
y ( b ) = 0 or 1 in Eq.
(4), i.e., every value y has a ﬁxed mapping to a binary value b .
Then we have
[Equation]
Hence, the original objective function is equivalent to maximizing the information entropy H ( B ) :
[Equation]
A. P ROOFS OF T HEOREM
Theorem 1 For input X φ of max pooling φ with arbitrary distribution, the information entropy of the binarized output to zero as n to inﬁnity, i.e., lim n → + ∞ H B = 0 .
And there is a constant c , for any nand n 2 , if n 1 > n 2 > c , we have H B,n 1 < H B,n 2 , where n is the number of aggregated elements.
Proof.
We obtain the correlation between the probability mass function of input X φ and output
Y of max pooling, intuitively, all values are negative to give a negative maximum value:
[Equation]
[Equation]
Since the sign function is applied as the quantizer, the H B ( B ) of binarized feature can be expressed as Eq.
(1) When X φ obeys a arbitrary distribution, the probability mass function p X φ ( x φ ) must satisﬁes (cid:80) x φ < 0 p X φ ( x φ ) ≤ 1 .
According to Eq.
(6), let t = (cid:80) x φ < 0 p X φ ( x φ ) , we have
[Equation]
(2) For any n ≥ 1 , we can obtain the representation of the information entropy H B,n ( X φ ) :
[Equation]
Let p n = (cid:16) (cid:80) x φ < 0 p X φ ( x φ ) (cid:17) n , the H B,n ( p n ) can be expressed as
[Equation] [Equation]
and the derivative of H B,n ( p n ) is
[Equation]
, (29) the H B,n ( p n ) is maximized when p n takes 0.5, and is positive correlation with p n when p n < 0 .since
the d H B,n ( p n ) d p B ( p n ) >
when p n < 0 .
Therefore, when the constant c satisﬁes p c = (cid:16) (cid:80) x φ < 0 p X φ ( x φ ) (cid:17) c ≥ 0 .
5 , given the n 1 > n 2 > c , we have p n 1 < p n 2 < p c , and H B,n 1 ( X φ ) <
H B,n 2 ( X φ ) <
H B,c ( X φ ) .
(cid:3)
A. P ROOFS OF P ROPOSITION
Proposition 1 When the distribution of the random variable Y satisﬁes (cid:80) y< 0 p Y ( y )
=
(cid:80) y ≥
p Y ( y ) = 0 .
5 , the information entropy H B is maximized.
Proof.
According to Eq (5), we have
[Equation] [Equation]
[Equation]
− (1 − p B ( − 1) log (1 − p B ( − 1))) .
Then we can get the derivative of H B ( B ) with respect to p B ( − 1)
[Equation]
When we let d H B ( B ) d p B ( − 1) = 0 to maximize the H B ( B ) , we have p B ( − 1)
= 0 .
Sine the deterministic sign function with the zero threshold is applied as the quantizer, the probability mass function of B is represented as
[Equation]
) and when the information entropy is maximized, we have
[Equation]
A.
D ISCUSSION AND P ROOFS OF T HEOREM
The bi-linear layers are widely used in our BiPointNet to model each point independently, and each linear layer outputs an intermediate feature.
The calculation of the bi-linear layer is represented as
Eq.
Since the random variable B is sampled from B w or B a obeying Bernoulli distribution, the probability mass function of B can be represented as
[Equation]
(38) where p is the probability of taking the value +1.
The distribution of output Z can be represented by the probability mass function of B w and B a .
Proposition 2 In bi-linear layer, for the binarized weight B w ∈ {− 1 , +1 } m × k and activation
B a ∈ {− 1 , +1 } n × m with probability mass function p B w (1) =
p w and p B a (1) =
p a , the probability mass function for the distribution of output Z can be represented as p Z (2 i − m )
=
C
im (1 − p w − p a
+ 2 p w p a ) i ( p w + p a − 2 p w p a ) m − i , i ∈ { 0 , 1 , 2 , ..., m } .
Proof.
To simplify the notation in the following statements, we deﬁne A
= B a and W = B w .
Then, for each element Z i,j in output Z ∈ {− 1 , +1 } n × k , we have
[Equation]
Observe that A i,k is independent to W k,j and the value of both variables are either − 1 or +1 .
Therefore, the discrete probability distribution of A i,k × W k,j can be deﬁned as p
[Equation]
if
[Equation]
p
[Equation] [Equation]
p
[Equation]
p a , if x = −0 , otherwise .
Simplify the above equation
[Equation]
Notice that x i,j can be parameterized as a binomial distribution.
Then we have
[Equation]
Observe that p Z obeys the same distribution as x i,j .
Finally, we have
[Equation]
Proposition 2 shows that the output distribution of the bi-linear layer depends on the probability mass functions of binarized weight and activation.
Then we present the proofs of Theorem 2.
Theorem 2 When we let p B w (1) = 0 .
5 and p B a (1) = 0 .
5 in bi-linear layer to maximize the mutual information, for the binarized weight B w ∈ {− 1 , +1 } m × k and activation B a ∈ {− 1 , +1 } n × m , the probability mass function for the distribution of output Z can be represented as p Z (2 i − m )
= 0 .
5 m C im , i ∈ { 0 , 1 , 2 , ..., m } .
The distribution of output is approximate normal distribution
N (0 , m ) .
Proof.
First, we prove that the distribution of Z can be approximated as a normal distribution.
For bi-linear layers in our BiPointNet, all weights and activations are binarized, which can be represented as B w and B a , respectively.
And the value of an element z ( i,j ) in Z can be expressed as
[Equation]
=
[Equation]
(cid:0) b w ( i,k )
× b a ( k,
[Equation]
) (cid:1) , and the value of the element b w ( i,k )
× b a ( k,j ) can be expressed as
[Equation]
The b w ( i,k ) ×
b a ( k,j ) only can take from two values and its value can be considered as the result of one
Bernoulli trial.
Thus for the random variable Z sampled from the output tensor Z , the probability mass function, p Z can be expressed as
[Equation]
where p e denotes the probability that the element b w ( i,k ) ×
b a ( k,j ) takes 1 .
Note that the Eq.
(45) is completely equivalent to the representation in Proposition 2.
According to the De Moivre–Laplace theorem, the normal distribution N ( µ, σ 2 ) can be used as an approximation of the binomial distribution under certain conditions, and the p Z (2 i − m ) can be approximated as
[Equation]
and then, we can get the mean µ = 0 and variance σ = √ m of the approximated distribution N with the help of equivalent representation of p Z in Proposition 2.
Now we give proof of this below.
According to Proposition 2, when p w = p a = 0 .
5 , we can rewrite the equation as
[Equation]
Then we move to calculate the mean and standard variation of this distribution.
The mean of this distribution is deﬁned as
[Equation]
By the virtue of binomial coefﬁcient
[Equation]
we have
[Equation]
Besides
[Equation]
when m is an even number
[Equation]
we have (2 i − m )0 .
5 m C im = 0 , i = m 2 .
These equations prove the symmetry of function (2 i − m )0 .
5 m C i m .
Finally, we have
[Equation]
The standard variation of p Z is deﬁned as
[Equation]
To calculate the standard variation of p Z
[Equation]
we use Binomial Theorem and have several identical equations:
[Equation]
These identical equations help simplify Eq.
[Equation]
Now we proved that, the distribution of output is approximate normal distribution N (
[Equation]
m ) .
[Equation]
A. D ISCUSSION OF THE O PTIMAL δ FOR EMA- MAX
When the X φ ∼ N (0 , 1) , the objective function of EMA-max to obtain optimal δ ∗ is represented as Eq.
It is difﬁcult to directly solve the objective function.
To circumvent this issue, we use
Monte Carlo simulation to approximate the value of the optimal δ ∗ max as shown Algorithm 1.
Algorithm 1 Monte
Input: The number n of points to be aggregated; the number of simulations m (e.g. 10000 )
Output: Estimated optimal δ ∗ max for EMA-max 1: Creating an empty list F (represents elements sampled form distribution of aggregated feature) 2: for i = 0 to m do 3:
Creating an empty list T i (representing one channel of input feature) 4: for j = 0 to n do 5:
Sampling an element e ij from the distribution N (0 , 1) 6:
Adding the sampled element e ij to the list T i 7: end for 8:
Adding an element represents the aggregated feature MAX( T i ) to F 9: end for 10:
Estimating the optimal δ ∗ max as δ ∗ max = Median( F ) (follow Proposition 1)
A. D ISCUSSION OF THE O PTIMAL δ FOR EMA- AVG
When the X φ ∼ N ( δ, 1) , the Y ∼ N ( δ, n − 1 ) and the objective function of EMA-avg for obtaining optimal δ ∗ avg can be represented as
[Equation]
The solution of Eq.
(66) is expressed as δ = 0 , we thus obtain δ ∗ avg = 0 .
This means the solution is not related to n .
B I MPLEMENTATION OF B I P OINT N ET ON ARM D EVICES B. O VERVIEW
We further implement our BiPointNet on Raspberry Pi 4B with 1.5 GHz 64-bit quad-core ARM
Cortex-A72 and Raspberry Pi 3B with 1.2 GHz 64-bit quad-core ARM Cortex-A53, and test the real speed that one can obtain in practice.
Although the PointNet is a recognized high-efﬁciency model, the inference speed of BiPointNet is much faster.
Compared to PointNet, BiPointNet enjoys up to 14 .
7 × speedup and 18 .
9 × storage saving.
We utilize the SIMD instruction SSHL on ARM NEON to make inference framework daBNN (Zhang et al., 2019a) compatible with our BiPointNet and further optimize the implementation for more efﬁcient inference.
B. I MPLEMENTATION D ETAILS
Figure 6 shows the detailed structures of six PointNet implementations.
In Full-Precision version (a), BN is merged into the later fully connected layer for speedup, which is widely chosen for deployment in real-world applications.
In Binarization version (b)(c)(d)(e), we have to keep BN unmerged due to the binarization of later layers.
Instead, we merge the scaling factor of LSR into
BN layers.
The HardTanh function is removed because it does not affect the binarized value of input for the later layers.
We test the quantization for the ﬁrst layer and last layer in the variants (b)(c)(d)(e).
In the last variant(f), we drop the BN layers during training.
The scaling factor is ignored during deployment because it does not change the sign of the output.
[Figure]
B. A BLATION ANALYSIS OF TIME COST AND QUANTIZATION SENSITIVITY
[Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
All the models use PointNet as the base model and EMA-max as the aggregation function.
The accuracy performance is reported on the point cloud classiﬁcation task with the ModelNet40 dataset.
[Table]
Table 4 shows the detailed conﬁguration including overall accuracy, storage usage, and time cost of the above-mentioned six implementations.
The result shows that binarization of the middle fully connected layers can extremely speed up the original model.
We achieve 18 .
9 × storage saving, 14 .
7 × speedup on A
[Table]
and 12 .
1 × speed on A53.
The quantization of the last layer further helps save storage consumption and improves the speed with a slight performance drop.
However, the quantization of the ﬁrst layer causes a drastic drop in accuracy without discernible computational cost reduction.
The variant (f) without BN achieves comparable performance with variant (b).
It suggests that our LSR method could be an ideal alternative to the original normalization layers to achieve a fully quantized model except for the ﬁrst layer.
C C OMPARISON BETWEEN L AYER -W ISE S CALE R ECOVERY AND OTHER M ETHODS
In this section, we will analyze the difference between the LSR method with other model binarization methods.
Theorem 2 shows the signiﬁcance of recovering scale in point cloud learning.
However,
IRNet and BiReal only consider the scale of weight but ignore the scale of input features.
Therefore, these two methods cannot recover the scale of output due to scale distortion on the input feature.
A major difference between these two methods is that LSR opts for layer-wise scaling factor while
XNOR opts for point-wise one.
Point-wise scale recovery needs dynamical computation during inference while our proposed LSR only has a layer-wise global scaling factor, which is independent of the input.
As a result, our method can achieve higher speed in practice.
[Figure]
Table 3 shows that XNOR can alleviate the aggregation-induced feature homogenization.
The pointwise scaling factor helps the model to achieve comparable adjustment capacity as full-precision linear layers.
Therefore, although XNOR suffers from feature homogenization at the beginning of the training process, it can alleviate this problem with the progress of training and achieve acceptable performance, as shown in Figure 7.
D C OMPARISON WITH O THER E FFICIENT L EARNING M ETHODS
We compare our computation speedup and storage savings with several recently proposed methods to accelerate deep learning models on point clouds.
Note that the comparison is for reference only; tests are conducted on different hardware, and for different tasks.
Hence, direct comparison cannot give any meaningful conclusion.
In Table 5, we show that BiPointNet achieves the most impressive acceleration.
E E XPERIMENTS E. D ATASETS
ModelNet40: ModelNet40 (Wu et al., 2015) for part segmentation.
The ModelNet40 dataset is the most frequently used dataset for shape classiﬁcation.
ModelNet is a popular benchmark for point cloud classiﬁcation.
It contains 12,311 CAD models from 40 representative classes of objects.
[Table]
ShapeNet Parts: ShapeNet Parts (Chang et al., for
S3DIS: S3DIS for semantic segmentation (Armeni et al., 2016). includes 3D point clouds for areas
272 rooms in point belongs to
We follow the ofﬁcial
(Qi et al., 2017a training
E. I MPLEMENTATION D ETAILS OF B I P OINT N ET
We follow the popular PyTorch implementation of PointNet and the recent geometric deep learning codebase (Fey & Lenssen, 2019) for the implementation of PointNet baselines.
Our BiPointNet is built by binarizing the full-precision PointNet.
All linear layers in PointNet except the ﬁrst and last one are binarized to bi-linear layer, and we select Hardtanh as our activation function instead of
ReLU when we binarize the activation before the bi-linear layer.
For the part segmentation task, we follow the convention (Wu et al., 2014; Yi et al., 2016) to train a model for each of the 16 classes.
We also provide our PointNet baseline under this setting.
Following previous works, we train 200 epochs, 250 epochs, 128 epochs on point cloud classiﬁcation, part segmentation, semantic segmentation respectively.
To stably train the binarized models, we use a learning rate of 0.001 with Adam and Cosine Annealing learning rate decay for all binarized models on all three tasks.
E. M ORE B ACKBONES
We also propose four other models: BiPointCNN, BiPointNet++, BiDGCCN, and BiPointConv, which are binarized versions of PointCNN (Li et al., 2018), PointNet++ (Qi et al., 2017b),
DGCNN (Wang et al., 2019a), and PointConv (Wu et al., 2019), respectively.
This is attributed to the fact that all these variants have characteristics in common, such as linear layers for point-wise feature extraction and global pooling layers for feature aggregation (except PointConv, which does not have explicit aggregators).
In PointNet++, DGCNN, and PointConv, we keep the ﬁrst layer and the last layer full-precision and binarize all the other layers.
In PointCNN, we keep every ﬁrst layer of XConv full precision and keep the last layer of the classiﬁer full precision.
E. B INARIZATION M ETHODS
For comparison, we implement various representative binarization methods for 2D vision, including
BNN (Hubara et al., 2016), XNOR-Net (Rastegari et al., 2016), Bi-Real Net (Liu et al., 2018),
XNOR++ (Bulat & Tzimiropoulos, 2019), ABC-Net (Lin et al., 2017), and IR-Net (Qin et al., 2020b), to be applied on 3D point clouds.
Note that the Case 1 version of XNOR++ is used in our experiments for a fair comparison, which applies layerwise learnable scaling factors to minimize the quantization error.
These methods are implemented according to their open-source code or the description in their papers, and we take reference of their 3x3 convolution design when implementing the corresponding bi-linear layers.
We follow their training process and hyperparameter settings, but note that the speciﬁc shortcut structure in Bi-Real and IR-Net is ignored since it only applies to the ResNet architecture.
E. T RAINING D ETAILS
Our BiPointNet is trained from scratch (random initialization) without leveraging any pre-trained model.
Amongst the experiments, we apply Adam as our optimizer and use the cosine annealing learning rate scheduler to stably optimize the networks.
To evaluate our BiPointNet on various network architectures, we mostly follow the hyper-parameter settings of the original papers (Qi et al., 2017a; Li et al., 2018;
Qi et al., 2017b; Wang et al., 2019a).
E. D ETAILED R ESULTS OF S EGMENTATION
We present the detailed results of part segmentation on ShapeNet Part in Table 6 and semantic segmentation on S3DIS in Table 7.
The detailed results further prove the conclusion of Section 4.
as EMA and LSR improve performance considerably in most of the categories (instead of huge performance in only a few categories).
This validates the effectiveness and robustness of our method.
T a b l e 6 :
D e t a il e d r e s u lt s o f ou r B i
P
o
i n t
N e t f o r p a r t s e g m e n t a ti on on
S h a p e N e t
P a r t s .
a gg r .
m ea n ae r o b a g ca p ca r c h a i r ea r phon e gu it a r kn i f e l a m p l a p t op m o t
o r m ug p
i s t o l r
o c k e t s k a t e bo a r d t
a b l e # s h a p e s
FP m a x 84 .83 .79 .
FP a vg 84 .83 .78 .90 .
B
NN m a x 54 .
B
NN e m aa vg 53 .39 .46 .
[Table] [Table]
B
NN e m am a x 47 .37 .46 .
[Table] [Table]
[Table] [Table]
[Table]
L S R m a x 58 .
[Table]
[Table] [Table]
[Table] [Table]
O u
r s e
m aa vg 80 .79 .
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
O u r s e m am a x 80 .79 .69 .8
[Table]
[Table] [Table]
[Table] [Table]
T a b l e 7 :
D e t a il e d r e s u lt s o f ou r B i
P
o
i n t
N e t f
o r s e m a n ti c s e g m e n t a ti on o n
S 3 D
I S .
m e t hod a gg r ov e r a ll m I
o U ov e
r a ll acc .
a r ea ( m I o
U / acc . )
a r ea ( m I o
U / a cc . )
a r ea ( m I o
U / acc . )
a r ea ( m I o
U / acc . )
a r ea ( m I o
U / acc . )
a r ea ( m I o
U / acc . )
ce ili ng
I o U ﬂ oo r
I o U w a ll
I o U b ea m
I o U c
o l u m n
I o U w i ndo w
I o U doo r
I o U t a b l e
I o U c
h a
i r
I o U s o f a
I o U book ca s e
I o U bo a r d
I o U
FP m a x 54 .83 .61 .
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
FP a vg 51 .81 .59 .
B NN m a x 9 .45 .9 .
B NN e m aa vg 9 .46 .7 .
B NN e m am a x 8 .47 .7 .
L S R m a x 2 .25 .2 .
O u
r s e
m aa vg 40 .74 .47 .
O u r s e m am a x 44 .76 .50 .