A UTOMATED M OBILE A TTENTION KPC ONV N ETWORKS VIA A W IDE
D EEP P REDICTOR
Anonymous authors
Paper under double-blind review
A BSTRACT
Kernel Point Convolution (KPConv) achieves cutting-edge performance on 3D point cloud applications.
Unfortunately, the large size of KPConv network limits its usage in mobile scenarios.
In addition, we observe that KPConv ignores the kernel relationship and treats each kernel point equally when formulating neighbor-kernel correlation via Euclidean distance.
This leads to a weak representation power.
To mitigate the above issues, we propose a module named Mobile
Attention Kernel Point Convolution (MAKPConv) to improve the efﬁciency and quality of KPConv.
MAKPConv employs a depthwise kernel to reduce resource consumption, and re-calibrates the contribution of kernel points towards each neighbor point via Neighbor-Kernel attention to improve representation power.
Furthermore, we capitalize Inverted Residual Bottleneck (IRB) to craft a design space and employ a predictor-based Neural Architecture Search (NAS) approach to automate the design of efﬁcient 3D networks based on MAKPConv.
To fully exploit the immense design space via an accurate predictor, we identify the importance of carrying feature engineering on searchable features to improve neural architecture representations, and propose a Wide & Deep Predictor to unify dense and sparse neural architecture representations for lower error in performance prediction.
Experimental evaluations show that our NAS-crafted MAKPConv network uses 96% fewer parameters on 3D point cloud classiﬁcation and segmentation benchmarks with better performance.
Compared with state-of-the-art NAScrafted model SPVNAS, our NAS-crafted MAKPConv network achieves ∼ 1% better mIOU with 83% fewer parameters and 78% fewer Multiply-Accumulates.
I NTRODUCTION
3D applications such as object recognition (Wu et al., 2015), shape segmentation (Yu et al., 2004), and indoor-scene segmentation (Armeni et al., 2016) are growing popularity thanks to the rise of 3D acquisition technologies and the increasing amount of geometric data collected from 3D sensors.
Deep learning models are becoming one of the most common and promising ways to represent 3D data, especially 3D point cloud data, thanks to its high performance (Zhou et al., 2020;
Zhao et al., 2020) and good efﬁciency (Cortinhal et al., 2020; Hu et al., 2020a).
Starting from PointNet (Qi et al., 2017a), point-based methods are becoming one of the most popular choices to extract features and make predictions for 3D input point clouds.
Such an approach typically learns an individual representation for each data point and aggregates different representations to formulate a signature for all global points, usually via end-to-end point convolutions (Xu et al., 2018;
Atzmon et al., 2018).
Kernel point convolution (KPConv) (Thomas et al., 2019) is the ﬁrst work that maps the input points to the inﬂuence of kernel points via linear correlation, and then further convolves the mapped features over the kernel points to compute weighted features.
Given a point y i , the kernel function g over neighbor points y i in KPConv carries g ( y i ) =
(cid:80)
k<K h ( y i , ˜ x k )@
W k to learn representations via K kernel points, with the neighbor-kernel correlation h
( y i , x k ) .
However, KPConv results in very large model sizes for 3D point cloud recognition due to inefﬁcient local feature aggregation via point convolution over kernel weights W k .
As a result, KPConv takes up to ∼ 200G Multiply-Accumulates (MACs) to perform inference on a single scene at the cost of ∼ 200ms on a TITAN XP GPU.
Such resource consumption and inference cost are too high for real-time execution in mobile scenarios.
In addition, the above Neighbor-Kernel cor- relation function has a strong hidden assumption: the contribution of all kernel points are measured equally towards the neighbor points.
As a result of ignoring cross-kernel information from other kernel points to accommodate the joint evolution of different kernel points, such an assumption fails to enhance representation learning on important kernel points and thus yields sub-optimal results.
In this paper, we propose a novel 3D attention module, named Mobile Attention Kernel Point Convolution (MAKPConv), to address the above issues.
MAKPConv employs two key design elements to improve the performance and efﬁciency.
First, we propose a depthwise kernel to split feature extraction on point-wise features from channel-wise features during point convolution.
Such design allows signiﬁcant savings in both parameter count and amount of computation, without performance degradation.
Second, we introduce Neighbor-Kernel attention (NK attention) to enable the kernelwise information exchange between different kernel points via learnable attention networks.
The joint evolution of all kernel points encourages representation learning on more important kernel points while suppressing less important ones, thus beneﬁts from a better learning adaptability of kernel point contributions towards neighbor point distributions in KPConv networks.
Inspired by Sandler et al.
(2018), we capitalize Inverted Residual Bottleneck (IRB) to encapsulate
MAKPConv module into MAKPConv block, and craft MAKPConv design space to explore a wider range of choices in 3D models.
As 3D architectures are usually high-dimensional, unstructured and unordered, tailoring a good neural architecture to address such complex point cloud data requires a delicate and precise evaluation protocol.
This is difﬁcult to achieve with weight-sharing oneshot NAS (e.g., SPVNAS (Tang et al., 2020)) because co-adaptation between sub-networks may degrade the quality of evaluation and lead to sub-optimal results.
Thus, we pioneer the usage of predictor-based Neural Architecture Search (NAS) (Wen et al., 2020) that utilizes a sample-based mechanism to reduce evaluation noise and seek better models on 3D point cloud.
Speciﬁcally, we train a performance predictor based on exploitable samples, and probe the whole MAKPConv design space to obtain the best NAS model based on predictive performance.
We observe that existing predictor-based methods adopt a ﬁxed scheme (e.g., one-hot encoding) to encode different types of features, leading to sub-optimal neural architecture representations with inaccurate priors.
This is because real-valued features (e.g., block width) and categorical features (e.g., optional NK attention) have different priors and thus they should not be encoded in the same way.
To address this issue, we formulate dense representations for features that have quasi-linear priors on model performance (e.g., depth, width), and formulate sparse representation for features that have nonlinear sophisticated priors on model performance (e.g., optional attention).
Intuited by Cheng et al.
(2016), we devise a Wide & Deep predictor to interact dense and sparse representations such that the combination of memorization and generalization reduces the prediction error.
By utilizing the Wide & Deep predictor to perform NAS under the MAKPConv design space, our results on various benchmarks show promising results of the automatically discovered models in terms of both performance and efﬁciency.
In particular, our MAKPConv networks outperform the original KPConv on both classiﬁcation and semantic segmentation benchmarks with 92% ∼ 93% fewer parameters, thanks to the efﬁcient design of MAKPConv block.
Empowered by our Wide &
Deep predictor driven NAS, the NAS-crafted networks outperform the state-of-the-art NAS work,
SPVNAS, by ∼ 1 mIOU with 83% smaller size and 78% fewer MACs.
We highlight the contribution of this paper as follows:
• We propose Mobile Attention Kernel Point Convolution (MAKPConv) module that combines depthwise kernel with Neighbor-Kernel attention to achieve efﬁcient and highperforming architectures dedicated to learning on 3D point clouds.
• We identify the importance of carrying feature engineering on neural architecture representations to improve predictor-based NAS.
Speciﬁcally, we encode different searchable features with dense and/or sparse representations given their priors, and utilize a Wide &
Deep neural predictor to interact these representations for better accuracy prediction.
R ELATED W ORK
3D Point Convolution Networks.
Point convolution networks (Xu et al., 2018; Hu et al., 2020a;
Lang et al., 2020) extends convolutional operations towards irregular point set grids to beneﬁt from learnable radial function and spherical harmonics in 3D point clouds.
Later work (Thomas et al.,
2019) performs point convolution over radius neighbors and aggregates convolved results to produce learned representation without additional intermediate operators.
Recent work on point convolutions improves point sampling efﬁciency (Hu et al., 2020a;
Lang et al., 2020), local feature aggregations (Zhao et al., 2020; Lin et al., 2020), manual architecture fabrications (Hu et al., 2020b;
Lei et al., 2020a), and/or improved training protocols (Goyal et al., 2021) to seek higher model performance and efﬁciency.
However, these works neither simplify the structure of point networks to improve model efﬁciency, nor specify an accurate neighbor-kernel correlation.
Our work extends kernel point convolution (Thomas et al., 2019) and addresses the above caveats: we improve the efﬁciency of local feature aggregation via a depthwise kernel on point convolution, and further introduce Neighbor-Kernel attention to enhance the representation power of a depthwise kernel.
Predictor-based Neural Architecture Search.
Predictor-based NAS (Wen et al., 2020; Dudziak et al., 2020) is a popularized method that trains a performance predictor on limited exploitable samples.
The trained predictor serves as a surrogate model of the ground-truth performance and is utilized to guide architecture search on the full design space.
Existing achievements on predictorbased NAS lies in improving sample efﬁciency (Dudziak et al., 2020) and augmenting architecture samples (Liu et al., 2021), yet they overlook the importance of carrying feature engineering on neural architecture features.
Our work identiﬁes the importance of improving neural architecture representation, and assigns different priors to a wide set of searchable features and yield dense representations and sparse representations.
Our work further designs a Wide & Deep predictor to leverage and interact these representations for better prediction quality on unseen architectures.
M OBILE A TTENTION K ERNEL P OINT C ONVOLUTION
We start with the formulation of KPConv (Thomas et al., 2019). Given a center point X ∈ R 3 and its corresponding features ˆ F ∈ R D in , a kernel point convolution (KPConv) function carries all N neighbor points X n = { x (cid:48) : || x (cid:48) − x || < r } ∈ R N × 3 and their features ¯ F ∈ R N × D in within radius r over a set of K kernel points ˜ X ∈ R K × 3 and accompanying weights W k . Speciﬁcally, a correlation function: h : R N × D in → R K × D in maps the unordered neighbor point features ¯ F to a set of features F on kernel points.
Usually, the correlation function is represented as H ∈ R N × K and applied via F = H ¯ F .
The correlation function is followed by a kernel g : R K × D in → R D out that convolves these features to obtain the aggregated output F (cid:48) .
Here, D in denotes the dimension of input, and D out denotes the dimension of output.
The original KPConv overlooks an important opportunity to improve the efﬁciency by reducing the redundancy of kernel g that is due to the simultaneously performed point-wise and channel-wise feature extractions.
In addition, the original
KPConv adopts a linear function as correlation: it treats the impact of each kernel point equally and ignores the adaptation of neighbor points towards kernel dispositions.
For example, the center kernel point usually attracts more centralized neighbor points with a higher density, and edge kernel points perform critical jobs in recognizing boundary neighbor points with a lower density in the radius sampling neighborhood.
Thus, the original KPConv may have sub-optimal performance and efﬁciency on 3D applications.
[Figure]
⨀ I npu t s
O u t pu t s 𝐹𝐹 :[ 𝐾𝐾 , 𝐷𝐷 ] 𝑊𝑊 :[ 𝐾𝐾 , 𝐷𝐷 ] Σ 𝐹𝐹′ :[ 𝐷𝐷 ] 𝐷𝐷 𝑖𝑖𝑖𝑖
= 𝐷𝐷 𝑜𝑜𝑜𝑜𝑜𝑜 = 𝐷𝐷 𝑁𝑁𝐾𝐾 𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐷 𝐿𝐿𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐿𝐿 𝐶𝐶𝐶𝐶𝐿𝐿𝐿𝐿𝐷𝐷𝐶𝐶𝐷𝐷𝐷𝐷𝐷𝐷𝐶𝐶𝐷𝐷 [1, 𝐾𝐾 ] 𝑀𝑀𝐿𝐿𝑀𝑀 [1, 𝐾𝐾 ] 𝜎𝜎 𝑁𝑁𝐾𝐾 𝐶𝐶𝐶𝐶𝐿𝐿𝐿𝐿𝐷𝐷𝐶𝐶𝐷𝐷𝐷𝐷𝐷𝐷𝐶𝐶𝐷𝐷 0. MAKPConv
NK Attention 𝐿𝐿𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐿𝐿𝐹𝐹𝐹𝐹𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐶𝐶𝐷𝐷 [ 𝑁𝑁 , 𝐾𝐾 ] [ 𝑁𝑁 , 𝐾𝐾 ] [
[Equation] [Equation] ] 𝐺𝐺𝐶𝐶𝐶𝐶𝑜𝑜𝐷𝐷𝐶𝐶 𝐶𝐶𝐶𝐶𝐿𝐿𝐿𝐿𝐷𝐷𝐶𝐶𝐷𝐷𝐷𝐷𝐷𝐷𝐶𝐶𝐷𝐷 To overcome the above limitations, we propose Mobile Attention Kernel Point Convolution (MAKPConv) module as an in-place improvement of KPConv block to seek higher quality and efﬁciency.
Figure 1 provides an overview of MAKPConv.
Speciﬁcally, we design a depthwise kernel to address the computation cost: A depthwise kernel ˆ g carries element-wise multiplication in place of matrix multiplication between aggregated features and kernel weights, thus achieves memory and computation reduction.
Considering a depthwise kernel has limited representation power, we introduce Neighbor-Kernel attention (NK attention) as an improved correlation function ˆ h to emphasize representation learning on important depthwise kernels and fully utilize the capacity of kernel points. This compensates for the reduced representation power: the joint evolution of kernel points via learnable attention encourages effective representation learning on important depthwise kernels and best exploit their potential.
As a result, the fusion of depthwise kernel and NK attention into MAKPConv block has the potential to achieve high-performing yet efﬁcient 3D models.
3. D EPTHWISE K ERNEL
In original kernel function of KPConv, a regular kernel takes an input F ∈ R K × D in over K kernel points and uses a weight kernel W ∈ R K × D in × D out to simultaneously perform point-wise transformation and channel-wise transformation.
A regular kernel g produces an output F (cid:48)
∈ R D out :
[Equation]
where k is a kernel index, and d is a channel index.
A regular kernel leads to K × D in × D out parameters and D in × D out × K MACs during inference, and leads to large computation and memory for the deep layers of the 3D architectures.
To favor model efﬁciency, we design a depthwise kernel ˆ g that only performs point-wise feature extractions over input point clouds, see Figure 1(a).
The proposed depthwise kernel adopts learnable weights W ∈ R K × D in , extracts point-wise information on each channel, and obtains an output F (cid:48)
∈ R D out as follows:
[Equation]
where k is a kernel index, and d is a channel index.
Here, D out = D in = D as a depthwise kernel does not operate on channel-wise features.
As a result, a depthwise kernel point convolution only leads to K × D in parameters and only D in × K MACs during inference, and reduces D out × parameters and MACs during convolution compared to a regular kernel.
The parameter saving from a depthwise kernel is signiﬁcant when D out is large, especially in deeper layers of the network.
3. N EIGHBOR -K ERNEL A TTENTION
Although depthwise kernel ˆ g adds to model efﬁciency, such design lacks representation power to ﬁt complex point cloud structures due to inactive channel-wise communication.
To compensate for the loss, we propose Neighbor-Kernel attention (NK attention)
ˆ h : R N × D in → R K × D in that re-weights the contribution of kernel points by learning a joint representation ˆ H ∈ R N × K on Neighbor-Kernel distances (NK distance) via an attention network, see the lower part of Figure 1.
Such representation, when being applied to neighbor point features ¯ F , can fully harness the capacity provided by different kernel points, thus provide better aggregated features F for the depthwise kernel.
NK attention ﬁrst adopts a linear function to transform NK distance into linear NK correlation, and applies average pooling to obtain global neighborhood correlation.
This global information captures rich information over the statistics of the locations of neighbor points and thus gives a highlevel abstraction on the contribution of each kernel point towards the global neighborhood.
Then, the global neighborhood correlation is fed into a 2-layer attention network to enable cross-kernel interaction and learn a good representation to quantify the contribution of kernel points, followed by a sigmoid gating to normalize the learned representation.
Unlike Squeeze-and-excitation (Hu et al., 2018) , NK attention adopts a full-rank transformation to preserve the information needed towards every kernel with marginal parameter cost.
Finally, the learned representation is scaled to the original NK linear correlation, followed by a bypass (i.e., addition) of the original linear NK correlation for easier optimization.
As a result, NK attention grasps a richer representation ˆ H that incorporates the joint evolution of kernel points and encourages representation learning on important depthwise kernels, This leads to concrete performance improvement on 3D point-cloud benchmarks.
N ERUAL A RCHITECTURE S EARCH WITH W IDE & D EEP P REDICTOR
In this section, we introduce a predictor-based Neural Architecture Search (NAS) driven by a novel
Wide & Deep predictor, see Figure 2.
MAKPConv module provides efﬁcient motifs for processing point-wise features.
To incorporate channel-wise feature extraction into MAKPConv module, we exploit the design of Inverted Residual Bottleneck (IRB) (Sandler et al., 2018) and sandwich
MAKPConv module with 2 linear transformation to construct MAKPConv block.
To design efﬁcient yet high performing 3D models, we utilize MAKPConv block as an architecture motif and exploit predictor-based neural architectures with a wide set of features such as positional organizations, structural settings, and accessories.
As formulating neural architecture representations from these features is the key in predictor-based NAS, we identify the importance of carrying engineering on searchable features.
Thus, we assign different priors to these features to formulate dense architecture representations and sparse architecture representations, and propose a novel Wide & Deep neural predictor to effectively interact different sets of neural architecture representations for accurate predictions.
Wide & Deep predictor trades-off generalization and memorization via dense deep models and sparse wide embeddings, thus reduce prediction MSE on unseen architectures.
[Figure]
Accessories Structural
Positional
[Figure]
⋮ Dense [Figure] Sparse
[Figure]
Dot
[Figure]
Embedding ( 𝐾𝐾 , 𝐴𝐴𝐴𝐴𝐴𝐴 ) ⨁
[Figure]
Linear [ 𝒃𝒃 , 𝑾𝑾 × 𝑬𝑬 ] [ 𝒃𝒃 , 𝑾𝑾 ] 𝑫𝑫 =D [ 𝒃𝒃 , 𝑾𝑾 × 𝑬𝑬 ] [ 𝒃𝒃 , 𝑾𝑾 ] [ 𝒃𝒃 , 𝑾𝑾 ]
[Figure]
4. MAKPC ONV B LOCK
Inspired by Sandler et al.
(2018), we sandwich MAKPConv by 2 linear transformation layers to derive MAKPConv block as a promising architecture motif to carry efﬁcient and high performing 3D point cloud learning.
It has a similar structure as Inverted Residual Bottleneck (IRB) and is described by four key parameters: expansion factor E , kernel size K , output width W , and optional choice of NK attention Atn .
We follow the positional setting of MobileNetV2 to manually craft a MAKPConv network as a hand-crafted baseline to verify its performance and efﬁciency.
We compare the performance and size between hand-crafted MAKPConv network and KPConv under the native KPConv protocol in Table 1.
Compared with KPConv, the hand-crafted MAKPConv achieves on-par Overall Accuracy (OA) with at least 92% smaller size on ModelNet40, and achieves 0.4% higher mIOU with 93% smaller size on SemanticKITTI, even without NK attention.
Such advantage is attributed to the efﬁcient depthwise kernel.
With NK attention, MAKPConv yields ∼ 0.4% OA gain on classiﬁcation and ∼ 0.5% mIOU gain on semantic segmentation with marginal increase in parameter count.
These results suggest that re-weighing the contribution of kernel points via learnable attention network enhances depthwise kernels thus improves performance
[Table] [Table]
4. MAKPC ONV D ESIGN S PACE
We utilize MAKPConv blocks to construct a ﬂexible network design space.
Speciﬁcally, we divide the overall 3D network into a stem KPConv for initial feature extraction, a head MLP for carrying ﬁne-grained classiﬁcation/segmentation, and several searchable stages.
In each searchable stage, we jointly search for the optimal conﬁguration of MAKPConv blocks, including the positional organizations, structural settings, and accessories.
We elaborate these searchable features below.
Positional Organizations.
The block width and depth of MAKPConv has a direct impact on the latency of MAKPConv models by affecting both computation-bounded operations (e.g., linear transformation) and memory-bounded operations (e.g., indexing).
As deeper and wider architectures lead to better performance in 3D point cloud classiﬁcation and segmentation following the Pareto frontiers, positional organizations have a positive quasi-linear impact towards performance and take continuous values on any positive integers.
Structural Settings.
The expansion ratio affects the inner-structure of MAKPConv block that has a positive quasi-linear correlation with performance.
However, co-adaption of expansion factor towards block width may induce challenges when learning a mapping rule to accurately predict architectures in the search space.
Structural settings take continuous values on any positive integers.
Accessories.
We mark the search of optional attentions and kernel size as accessories in the MAKPConv search space.
Accessories have sophisticated impact on model performance.
For example, as kernel point convolution utilize different dispositions for different number of kernels, it is unclear whether the change in the number of kernel points affects the quality of kernel disposition and the model performance.
Both accessories can only take categorical values.
Considering the high variance in 3D point cloud tasks and the robustness to noisy evaluations, we adopt predictor-based NAS to explore MAKPConv design space.
However, MAKPConv design space is prohibitively large: a total of 11 stages and 7 searchable features within each stage leads a total search space size of 1.8 × 10 19 possible architectures.
Such a large and complex search space calls for a high-performing predictor-based NAS to accurately model the design space.
4. W IDE & D EEP P REDICTOR
Neural architecture representation is the key in the performance of predictor-based NAS due to its strong connection with predictor performance.
Speciﬁcally, better neural architecture representation links searchable features towards priors that are more reﬂective to the ﬁnal performance, and thus increases the chance to discover high-performing NAS models.
To improve neural architecture representations, We focus on carrying feature engineering to distinguish searchable features with different priors.
Motivated by the aforementioned relationship between searchable features and the model performance, we assign different priors to a wide set of searchable features and craft dense and sparse neural architecture representations for corresponding features.
Speciﬁcally, we formulate dense neural architecture representation for searchable features that can take continuous values (e.g., positional organizations, structural settings), and formulate sparse neural architecture representation for searchable features that can only take discrete values (e.g., accessories).
With dense and sparse neural architecture representations, we follow the idea of
Wide and Deep Models (Cheng et al., 2016) to design a Wide & Deep predictor that trades-off memorization and generalization in performance prediction, see Figure 2.
Wide & Deep model adopts a separate MLP to obtain high-level dense outputs based on each of the 2 dense representations, and adopts an embedding table to convert sparse representations to continuous sparse outputs.
To interact dense and sparse outputs, we leverage the DotProduct block in DLRM (Naumov et al., 2019) to fuse learned dense representations and sparse representations.
Speciﬁcally, given a dense representation
D ∈ R B × dim and a sparse representation S ∈ R B × N × dim , DotProduct carries cross-representation communication by carrying the following transformation:
[Equation]
where dim represents the dimension of both dense and sparse representations, N denotes the number of sparse inputs, and B denotes the batch size.
Eq.
(3) ﬁrst vectorizes the dense tensor D to have the same dimension as the sparse tensor S , and extracts the upper triangular matrix upon self matrix factorization.
We carry 2 DotProducts on 2 separate pairs of dense and sparse representations to encourage the learning of independent interactions.
This is because positional organizations and
[Table]
structural settings may not necessarily have the same inﬂuence over the choice over accessories.
After fusing dense and sparse representations via DotProduct
[Table]
we utilize a MLP to learn predictions based on the fused representations in the Wide & Deep predictor.
4. T RAINING AND E VALUATION OF THE W IDE & D EEP P REDICTOR
Architecture Sampling.
We randomly sample ∼ 1K architectures on both ModelNet40 and SemanticKITTI.
Following the common NAS pipeline
[Table]
we split the training dataset of ModelNet40/SemanticKITTI into mini-train and mini-val.
We train the sampled architectures on mini-train until convergence
[Table]
and evaluate these architectures on mini-val to obtain architecture performance.
Predictor Training.
Mean-Squared Error (MSE) is the training objective of the Wide & Deep Predictor on 1K sampled architectures.
To ensure a consistent comparison of predictor performance over multiple benchmarks, we carry optimization on predictors with normalized performance ranging from 0 to 1.
Since the Wide & Deep Predictor can perform gradient-based optimization, we incorporate predictor pretraining on MACs (Dai et al., 2021) and rank loss (Dudziak et al., 2020).
We conduct extensive experiments on various top-performing NAS predictors collected from
NASBench-301 (Siems et al., 2020).
To justify that interacting dense and sparse representations beneﬁts predictor performance, we establish “Wide” as a MLP Predictor that accepts sparse neural architecture representations transformed from categorical encoding of features, and establish “Deep” as a MLP predictor that accepts dense neural architecture representations transformed from realvalued encoding features.
These two predictors also leverage rank loss (Dudziak et al., 2020) and predictor pretraining (Dai et al., 2021), thus are utilized as strong baselines for gradient-based NAS predictors to justify the signiﬁcance of dense-sparse feature interaction.
We list the supported training heuristics and compare the performance (i.e., MSE and Kendall τ ) of Wide & Deep Predictor over various NAS predictors under 10 different runtimes.
Compared with all non-gradient predictors (Chen & Guestrin, 2016; Ke et al., 2017; Duan et al., 2020), Wide & Deep predictor achieves lower MSE and higher Kendall τ in accuracy prediction, thanks to the potential of ranking-aware training and predictor pretraining via gradient-based optimization.
Compared to gradient-based predictors using either dense or sparse features, Wide & Deep predictor achieves better prediction quality thanks to better architecture representation.
Such improvement in predictor precision eventually yields the discovery of better NAS models, see the elaborations in Section 6.2.
E XPERIMENTS
In this section, we evaluate MAKPConv over classiﬁcation (ModelNet40 (Wu et al., 2015)) and semantic segmentation (SemanticKITTI (Behley et al., 2019)) benchmark.
5. S EARCH C ONFIGURATION
MAKPConv Design Space.
Each block can choose from kernel size { 5, 7, 13 } to perform MAKPConv operation.
Each block has up to 3 different choices for block depth, block width, and expansion factors.
In addition, each block has the choice of an optional NK attention to balance performance and efﬁciency.
We describe other details of the MAKPConv design space in Appendix.
Regularized Evolution.
Following Real et al.
(2019), we employ regularized evolution and our
Wide & Deep predictor to effectively probe the MAKPConv design space.
Given a candidate architecture A with predicted performance ˆ P and MACs M , we use S ( A ) =
ˆ P − β × log M as our search objective, and empirically set β to 0.5 to balance performance and resource consumption on both ModelNet40 and SemanticKITTI.
We use a population of 200 and sample size of 150 to carry regularized evolution with 360 rounds over MAKPConv design space to craft NAS models.
5. E VALUATION ON 3D BENCHMARKS
We use our Wide & Deep predictor to automate the design of MAKPConv blocks and discover promising models on ModelNet40 and SemanticKITTI 3D applications.
To ensure a fair comparison with larger models on SemanticKitti, we apply width scaling (Sandler et al., 2018) on the NAScrafted models and attach a m × sufﬁx to denote the application of a m × width multiplier.
We compare the performance and efﬁciency metrics such as parameter count and MACs.
[Table]
[Table]
ModelNet40.
We follow the RSCNN protocol in SimpleView (Goyal et al., 2021) and show the evaluation results on the testing dataset of
ModelNet40 in Table 3.
For fair comparison, we report the mean accuracy across 5 runs and observe a 0.2% OA variance.
Compared to KPConv, the hand-crafted MAKPConv achieves on-par OA with ∼ 92% fewer parameters thanks to the proposed MAKPConv module.
Our
NAS approach explores more efﬁcient model designs: the NAS-crafted MAKPConv achieves 0.5% higher OA than hand-crafted counterpart while being 55% smaller.
Compared with efﬁcient 3D models (Lei et al., 2020b; Goyal et al., 2021; Lin et al., 2020; Liu et al., 2019; Wang et al., 2019), NAS-crafted MAKPConv achieves onpar performance with ∼ 30% fewer parameters.
By further scaling up the model, our NAS-crafted
MAKPConv outperforms its hand-crafted counterpart by 0.8% OA with a similar parameter count.
SemanticKITTI.
Table 4 shows the mIOU comparison on sequence 08 of the SemanticKITTI dataset.
Our hand-crafted MAKPConv outperforms point-based, projection-based and voxel-based methods by at least 0.9%, 1.1% and 1.2% mIOU respectively, with signiﬁcant parameter and MAC reduction.
The NAS-crafted MAKPConv achieves ∼ 1% higher mIOU than the state-of-the-art SPVNAS while saving % parameters and
Applying width multiplier on
-crafted model further boosts the performance.
We elaborate on the latency comparison in Appendix A.7.
[Table]
A BLATION S TUDIES
In this section [Table] we ﬁrst demonstrate how NK attention re-calibrates the contribution of each kernel point by visualizing the relationship between NK attention and NK linear correlation.
Then, we discuss the relationship between predictor MSE reduction and quality improvement of the discovered
NAS models to demonstrate the importance of predictor design in predictor-based NAS [Table]
6. K ERNEL C ONTRIBUTION VIA NK A TTENTION
Empirical evaluations reﬂect that NK attention leads to 0.4% OA gain and 0.5% mIOU gain on 3D object classiﬁcation and semantic segmentation.
To verify the source of improvement of MAKPConv over its non-attention counterpart and justify the re-calibration of kernel points, we plot the learned NK attention on 7 kernel points in Figure 3 and identify the contribution of each kernel via the slope.
We observe that the learned attention for each kernel point has a strong relationship with kernel dispositions (i.e., 1-4-2 groups among symmetrical axis) in the Octahedron.
[Figure]
The two vertices that are the most distant from the center point shows the most contribution towards neighbor points.
This is because these kernel points are involved in addressing the boundary effects of input point clouds for more robust recognition results.
We also observe that the 4 vertices connecting the square plane has the least contribution, illustrating that such high concentration of kernels causes redundancy and suggests a future direction in optimizing kernel location to fully capitalize the kernel points to improve representation power.
The center kernel point admits most information from centralized neighbor points, thus co-adapts to global neighbor information and yields an moderate contribution towards neighbor points.
6. E FFECTIVENESS OF W IDE & D EEP P REDICTOR
We show that even marginal MSE reduction (i.e., 0.07 × 10 − 2 ) leads to signiﬁcant quality improvement of NAS-crafted models, which justiﬁes the importance of feature and architecture engineering in predictors.
We establish random search baseline by selecting the top-5 models from the outcome of 1K random samples, and establish predictor-based NAS baseline by selecting top-5 models using the top-performing Wide NN predictor in Table 5.
All models are evaluated on classiﬁcation and segmentation benchmarks using the same 100-epoch training pipeline.
We report the mean and standard deviation of performance on ModelNet40 and SemanticKITTI in Table 5.
[Table]
Results demonstrate that: (1) without introducing extra architecture samples, predictor-based NAS can outperform random search by up to 1.4% mIOU on SemanticKITTI.
(2) Even with 0.07 × 10 −lower MSE, the NAS-crafted models via Wide & Deep predictor can outperform NAS-crafted models via Wide predictor by a large margin (i.e., up to 1.2% mIOU on SemanticKITTI).
The above results emphasize the need to improve predictor-based NAS via a delicate neural predictor design.
C ONCLUSION
In this work, we present MAKPConv module to improve both efﬁciency and performance of KPConv.
MAKPConv employs depthwise KPConv that separates point-wise feature extraction from channel-wise feature extraction to reduce computation cost and memory consumption, and NK attention that re-weighs the contribution of different kernel points via interactions of neighbor-kernel correlation through an learnable attention network.
We further incorporate the IRB design to construct MAKPConv block and propose a new predictor-based NAS driven Wide & Deep predictor to automate the model design.
Experiments and ablation study over 3D classiﬁcation and 3D semantic segmentation benchmarks verify
R EPRODUCIBILITY S TATEMENT .
We make tremendous efforts to ensure reproducibility of our proposed method.
Speciﬁcally, we mention the detailed conﬁguration of predictor-based NAS in Section 5.1, introduce the design of hand-crafted MAKPConv design in Appendix A.1/A.2, describe the detailed conﬁguration of Wide & Deep predictor in Appendix A.3, show the transferability of searched architecture in Appendix
A.4, cover the training protocols in Appendix A.5, and use Appendix A.6 to justify orthogonal contribution of our improving performance predictor and other line of research in improving search algorithms.
In addition, we use Appendix A.7 to give an analysis on the latency breakdown of searched models and points an direction on further improving the efﬁciency of MAKPConv kernel to seek higher efﬁciency in 3D point cloud applications.
Code that covers other details of this work is available via this link
A A PPENDIX
In this appendix
[Table]
we elaborate the detailed structure of hand-crafted MAKPConv via MAKPConv blocks.
We also verify the transferability of NAS discovered architecture on S3DIS (Armeni et al., 2016) indoor scene segmentation benchmark.
Furthermore, we discuss detailed hyperparameter settings for training and evaluating 3D architectures on ModelNet40, S3DIS, and SemanticKITTI benchmarks, and visualize the optimization curve in the search process for both vanilla predictorbased NAS (i.e., via Wide Predictor) and our improved predictor-based NAS (i [Table] e., via Wide & Deep predictor). We also discuss the latency breakdown of our searched models and propose some future work in improving the efﬁciency of MAKPConv series models.
A. H AND - CRAFTED MAKPC ONV NETWORK
[Table]
[Table]
[Table]
Table 6 illustrates the detailed architecture of our hand-crafted MAKPConv.
The hand-crafted MAKPConv contains a total of 17 MAKPConv blocks for classiciation, and 4 additional MAKPConv blocks for semantic segmentation.
The positional setting (i [Table] e., depth, block width) of hand-crafted MAKPConv strictly follows MobileNetV2, which is considered as the state-of-the-art handcrafted model with remarkable efﬁciency.
For structural settings, we employ an expansion factor of 3 for all MAKPConv layers.
In addition, we leverage a kernel size of 7 for each MAKPConv block for design efﬁciency.
As a result, MAKPConv serves as a strong baseline to compare with when evaluating the performance of NAS-crafted
MAKPConv models.
A. MAKPC ONV D ESIGN S PACE
We present the detailed settings of the MAKPConv design space in Table 7.
Despite the search of block depth, block width, expansion factor and kernel size, we also incorporate the search of optional attention into MAKPConv design space to trade-off the overhead in Neighbor-Kernel Attention and its performance gain
[Table]
[Table]
A. D ESIGN OF W IDE & D EEP PREDICTOR
We conﬁgure the architectural hyperparameters of our wide & deep predictor via intuition without delicate architecture engineering.
Speciﬁcally, we employ a 3-layer MLP with 64-128-256 layers as the dense architecture to extract dense neural architecture representations for both positional organizations and structural settings.
The overall architecture that is responsible for processing fused features after dense-sparse interaction is a 2-layer MLP with 256-256 layers.
We utilize ReLU as the activation function and apply Dropout of 0.5 before the ﬁnal regression head to mitigate overﬁtting.
To ensure fair comparison, both the Wide predictor and the Deep predictor also adopt a 2-layer MLP with 256-128 units to keep the same maximum projection dimension as the Wide & Deep predictor.
A. A RCHITECTURE T RANSFERABILITY ON S3DIS
We further verify the transferability of NAS-crafted MAKPConv architectures on S3DIS to evaluate its performance on indoor scene segmentation.
Following the established protocols in 3D point cloud segmentation, we report both 6-fold cross-validation mIOU across all Area 1 ∼ Area 6 splits, see Table 8.
By convention, we also report the validation mIOU and mIOU per class on Area 5 to compare with existing methods on 3D point cloud, see Table 9.
Though our NAS-crafted model is not optimized in S3DIS dataset, it shows a signiﬁcant margin on S3DIS 6-fold crossvalidation benchmark compared with existing approaches such as Qiu et al.
; Zhao et al.
(2020) and achieves 74.5 mIOU, a new state-of-the-art result using only 1.35M parameters.
On Area 5 of the S3DIS dataset, we observe that: (1) Our NAS-crafted MAKPConv achieves 1.8% higher mIOU than original KPConv with 11 × parameter efﬁciency.
(2) Our NAS-crafted MAKPConv achieves competitive performance over different classes on S3DIS indoor semantic segmentation and demonstrate ∼ 1% higher mIOU over 9 out of 13 classes compared to existing approaches.
Thus, our NAS-crafted models have a good transferability towards alternative benchmarks, and our
NAS approach can be further incorporated into other 3D point cloud tasks such as nuScenes (Caesar et al., 2019), KITTI (Geiger et al., 2013) to improve the current state-of-the-art models. Architecture
[Table] [Table]
A. H YPERPARAMETER S ETTINGS
The implementation of our work is based upon the ofﬁcial PyTorch repository of KPConv (Thomas et al., 2019).
We follow the exact evaluation protocol in this repo to produce our validation results.
For training, we demonstrate the detailed hyperparameter settings on 3 different benchmarks (i.e., ModelNet40 (Wu et al., 2015), SemanticKITTI (Behley et al., 2019) and S3DIS (Armeni et al., 2016)) in the following context.
ModelNet40.
A single training batch in ModelNet40 contains 16 sub-sampled point clouds.
We adopt a similar training pipeline following the original KPConv (Thomas et al., 2019) paper yet incorporates SimpleView (Goyal et al., 2021)
RSCNN training protocol for fair comparison.
Specifically, the SimpleView-RSCNN protocol adopts random scaling & translation for data augmentation, employ cross-entropy loss to optimize, and utilize a voting scheme on the best test model to best exploit the potential of searched model.
We train our NAS-crafted model for 250 epochs with an initial learning rate of 0.02 and cosine learning rate schedule (Loshchilov & Hutter, 2016).
To combat overﬁtting, we also adopt a dropout rate of 0.5, a L2 weight decay of 3e-4 and employ early stopping to select the best model for testing.
S3DIS.
A single training batch in S3DIS contains 8 sub-sampled point clouds.
We use a downsample rate of dl 0 = 0 .
04 m following the original KPConv paper (Thomas et al., 2019).
Speciﬁcally, we train our best model for 250 epochs with an initial learning rate of 0.04 and cosine learning rate schedule (Loshchilov & Hutter, 2016).
To combat overﬁtting, We also adopt a L2 weight decay of 3e-4 and the default data augmentation in KPConv paper.
SemanticKITTI.
A single training batch in SemanticKITTI contains 10 sub-sampled point clouds.
On SemanticKITTI, we measure Multiply-Accumulates (MACs) over a scene with an average of ∼ 12.3K points, which gives similar MAC count for KPConv architecture as is shown in (Tang et al., 2020).
We use a downsample rate of dl 0 = 0 .
06 m following the original KPConv paper (Thomas et al., 2019).
Speciﬁcally, we train our best model for 250 epochs with an initial learning rate of 0.04 and cosine learning rate schedule (Loshchilov & Hutter, 2016).
To combat overﬁtting, We also adopt a L2 weight decay of 3e-4 and the default data augmentation in KPConv paper.
A. O PTIMIZATION CURVE ON D IFFERENT P REDICTORS
In this section, we demonstrate that the improvements of the proposed predictors has no interference over the improvements in search algorithms.
More speciﬁcally, our contribution towards leveraging a better neural architecture representation is orthogonal to the contribution in introducing a better search algorithm.
We apply 3 popular search algorithms: Random Search, Regularized Evolution (Real et al., 2019) and Reinforcement-Learning via Proximal Policy Optimization (PPO) (Zoph & Le, 2016; Schulman et al., 2017) on both Wide predictor and Wide & Deep predictor, and plot the normalized performance for visualization, see Figure 4.
Despite the greater complexity of Wide & Deep predictor that may potentially formulate a curvy performance surface, we observe that the superiority of EA on Wide predictor is successfully preserved on Wide & Deep predictor.
This indicates that the improvement of Wide & Deep predictor is consistent with the evaluation of different search algorithm, and sets up a new direction of improving the design of neural predictors to enhance predictor-based NAS. A. L ATENCY A NALYSIS OF MAKPC ONV K ERNEL
In Table 4, we notice that our NAS-crafted MAKPConv model is 24% faster than original KPConv model despite of its higher mIOU.
However, we also observe that the latency reduction is less signiﬁcant compared to MAC reduction of MAKPConv models, especially compared to voxel-based methods.
This is because voxel-based methods (e.g., SPVNAS (Tang et al., 2020)) beneﬁt from 1) reduced overhead in point-based neighboring mechanisms 2) mature software-hardware co-design that enables inference with high throughput.
To dive deep into this issue, we proﬁle the latency breakdown of the NAS-crafted MAKPConv model and demonstrate the proﬁling result in Table 10.
Here, the preprocessing operations (including radius sampling, grid sub-sampling etc.)
consumes 33.7% of the inference cost, neighbor gathering contributes to the memory-bounded operation (i.e., aten :: gather ), which takes 10% of the total inference latency.
Computation-bounded parts in
KPConv such as depthwise convolution (i.e., aten :: mul and aten :: sum ), MLPs (i.e., aten :: bmm )
[Table]
and point-wise local neighborhood movement (i
[Table]
e., aten :: sub ) contribute to ∼ 50% of the total inference cost.
Although radius sampling still consumes moderate cost in MAKPConv
[Table]
there is still a need to leverage design automation algorithm (e
[Table]
g., our proposed MAKPConv algorithm) to perform optimization on the overall architecture on MAKPConv and seek better inference efﬁciency by improving computation-bounded operations.
Besides the optimization of architecture
[Table]
we also discover some potential improvements that harnesses the hardware-software co-design to improve efﬁciency of MAKPConv blocks, such as yet not limited to:
• Efﬁcient radius sampling implementation.
The current implementation lies in CPU and is not optimal for parallelization.
A possible CUDA version of the radius sampling can speedup the inference speed of MAKPConv networks.
• The current MAKPConv does not utilize a fused version of convolution that aggregates the feature outputs on each kernel point efﬁciently.
Instead, the output features on each kernel point is ﬁrstly computed via aten :: bmm and then aggregated via aten :: sum .
• The current process of moving points towards their local neighborhoods controlled by aten :: sub is far from efﬁcient and consumes ∼ 20% latency.
Thus, a specialized hardware for efﬁcient batch points movement is potentially beneﬁcial for efﬁcient inference on point-based networks.
• As lower radius leads to more efﬁcient sampling, we consider incorporating the choice of radius for each MAKPConv layer into the design space and harness such option to discover more efﬁcient point-based 3D networks.
As such, with heavy optimization on MAKPConv design, we can reduce the uncertainty and/or complexity in proﬁling MAKPConv blocks and thus push forward latency-aware search for better hardware specialization.
We leave this as our future work to improve the design efﬁciency of 3D architectures on processing point cloud data.