P2P: Tuning Pre-trained Image Models for Point Cloud Analysis with Point-to-Pixel Prompting
Abstract
Nowadays, pre-training big models on large-scale datasets has become a crucialtopic in deep learning.
The pre-trained models with high representation ability andtransferability are tuned on downstream tasks and achieve a great success in naturallanguage processing and computer vision.
However, it is non-trivial to promotesuch a pretraining-tuning paradigm to the 3D vision, given the limited trainingdata that are relatively inconvenient to collect.
In this paper, we propose a newperspective of leveraging pre-trained 2D knowledge in 3D domain to tackle thisproblem, tuning pre-trained image models with the novel Point-to-Pixel promptingfor point cloud analysis.
Following the principle of prompting engineering, wetransform point clouds into colorful images with geometry-preserved projectionand geometry-aware coloring to adapt to pre-trained image models, whose weights are kept frozen during the end-to-end optimization of point cloud analysis tasks.
Weconduct extensive experiments to demonstrate that cooperating with our proposed Point-to-Pixel Prompting, better pre-trained image model will lead to consistentlybetter performance in 3D vision. Therefore, by leveraging prosperous developmentfrom image pre-training ﬁeld, our proposed framework achieves competitive resultswith previous methods on point cloud classiﬁcation and part segmentation.
Introduction
With the rapid development of deep learning and computing hardware neural networks are experiencing explosive growth in model size and representation capacity.
Nowadays, pre-training big models has become an important research topic in both natural language processing [11, 48, 4, 55] andcomputer vision [47, 49, 17, 52] and has achieved a great success when transferred to downstreamtasks with ﬁne-tuning [18, 9, 8, 2, 17] or prompt-tuning [47, 43, 62, 54, 27, 33] strategies.
Fine-tuningis a traditional tuning strategy that requires a large amount of trainable parameters, while prompttuning is a recently emerged lightweight scheme to convert downstream tasks into the similar form asthe pre-training task.
However, such prevalence of the pretraining-tuning pipeline cannot be obtainedwithout the support of numerous training data in pre-training stage.
Language pre-training leadingwork Megatron-Turing NLG [55] with 530 billion parameters is trained on 15 datasets containingover 338 billion tokens, while Vision MoE [52] with 14.7 billion parameters is trained on JFT-300Mdataset [57] including 305 million training images.
Unfortunately, the aforementioned convention of pre-training big models on large-scale datasets andtuning on downstream tasks has encountered obstacles when we refer to the 3D vision realm.
3Dvisual perception is gaining more and more attention given its superiority in many emerging researchﬁelds including autonomous driving [26, 75], robotics vision [7, 73] and virtual reality [39, 66].
However, obtaining abundant 3D data such as point clouds from LiDAR is neither convenient norinexpensive.
For example, the largest object-level point cloud dataset ShapeNet [6] only contains 50 thousand synthetic samples.
Therefore, pre-training fundamental 3D models with limited dataremains an open question.
There are some previous works [72, 63, 76] that attempts to developspeciﬁc pre-training strategies on point clouds with limited training data, such as Point Contrast [72],
OcCo [63] and Point-BERT [76].
Although they prove that the pretraining-ﬁnetuning pipeline workswell in the 3D domain, the imbalance between numerous trainable parameters and limited trainingdata may lead to insufﬁcient optimization or overﬁtting problems.
Different from the previous works that directly pre-train models on 3D data, we propose to transferthe pre-trained knowledge from image ﬁeld to 3D domain with appropriate prompting engineering,since images and point clouds display the same visual world and share some common knowledge.
In this way, we address the data-starvation problem in the 3D domain, given that the pre-trainingstrategy is well-studied in the 2D ﬁeld with abundant training data and that prompt-tuning on 3Dtasks does not require much 3D training data.
To the best of our knowledge, we are the ﬁrst workto transfer knowledge in pre-trained image models to 3D vision with a novel prompting approach.
More speciﬁcally, we propose an innovative Point-to-Pixel Prompting mechanism that transformspoint clouds into colorful images with geometry-preserved projection and geometry-aware coloring.
Examples of produced colorful images are shown in Figure 1.
Then the colorful images are fedinto the pre-trained image model with frozen weights to extract representative features, which arefurther deployed to downstream task-speciﬁc heads.
The end-to-end optimization pipeline and thestrategy of freezing the pre-trained image model promote the bidirectional knowledge ﬂow betweenpoints and pixels.
The geometric information from point clouds is mostly retained in projectedimages via our geometry-preserved projection, while the color information of natural images fromthe pre-trained image model is transmitted back to colorless point clouds via the cooperation betweenthe geometry-aware coloring module and the ﬁxed pre-trained image model.
We conduct extensive experiments demonstrate that with our Point-to-Pixel Prompting, enlarging thescale of the same image model will result in higher point cloud classiﬁcation performance, which isconsistent with the observations in image classiﬁcation. This suggest that we can take advantage ofthe successful researches in pre-training big image model, opening up a new avenue for point cloudanalysis.
With much fewer trainable parameters, we achieve comparable results with the best objectclassiﬁcation methods on both synthetic ModelNet40 [69] and obtain state-of-the-art performanceon real-world ScanObjectNN [60].
We also demonstrate the potential of our method to performdense predictions like part segmentation on ShapeNetPart [74].
In conclusion, our Point-to-Pixel Prompting (P2P) framework explores the feasibility and superiority of transferring image pre-trainedknowledge to the point cloud domain, bridging the gap between the two most decisive vision ﬁeldsand promoting a new pre-training paradigm in 3D point cloud analysis.
Related Work2.1 Visual Pre-training
Pre-training visual models has been studied thoroughly in the image domain and there are mainlythree kinds of pre-training strategies.
Supervised pre-training [12, 77, 5] on classiﬁcation task with large-scale dataset is a traditional practice and is stimulated by the boosting development of basevision models [19, 20, 12, 35].
Weakly-supervised pre-training methods [58, 3, 71, 44] use lessannotations while unsupervised pre-training works [18, 9, 8, 2, 17, 14] introduces no task-relatedbias and brings higher transferability to various downstream tasks.
Different from the prosperity of pre-training image models, pre-training 3D models is still underdevelopment.
Many works have developed self-supervised learning mechanisms with various pretexttasks such as solving jigsaw puzzles [53] and deformation reconstruction [1].
Inspired by pre-trainingstrategies in image domain, Point Contrast [72] adopts contrastive learning principle while OcCo [63]and Point-BERT [76] introduce reconstruction pretext tasks for pre-training.
2.2 Prompt Tuning
Prompt tuning is an important mechanism whose principle is to adapt downstream tasks withlimited annotated data to the original pre-training task at a minimum cost, thus exploiting thepre-trained knowledge to solve downstream problems.
It is ﬁrst proposed in the natural languageprocessing community [31].
At ﬁrst, hand-crafted prompting methods [43, 4] are promoted and theirfollowers [62, 54] develop automated searching algorithms for discrete prompt tokens.
Recently,continuous prompting methods [29, 27] are becoming the mainstream given their high performance.
On the contrary, the development in prompting visual pre-training models lags behind.
L2P
[65]proposes a prompt pool for continual learning problem while VPT [21] ﬁrst introduce continuousprompt tuning framework to visual classiﬁcation and segmentation, inspired by P-Tuning [33, 32].
As far as we are concerned, there are no previous works like this paper to discuss tuning pre-trainedimage models for point cloud analysis with an appropriate prompting mechanism.
2.3 Object-level Point Cloud Analysis
Given the unordered data structure of point clouds, early literature has developed voxel-based andpoint-based methods to construct structural representations for point cloud object analysis.
Voxelbased methods [41, 24, 51] partition the 3D space into ordered voxels and perform 3D convolutions forfeature extraction.
Point-based methods [45, 46, 40, 10, 30, 68, 59, 64, 28] directly process unorderedpoints and introduce various approaches to aggregate local information.
Recently, attention-based Transformer [61, 15, 78] architecture has prevailed over other frameworks in vision community andachieved competitive performance in point cloud object analysis.
Besides the aforementioned methods that perform representation learning in the 3D space, there areprojection-based works[56, 22, 67, 16, 13] that leverage multi-view images to represent synthetic3D objects.
Recently, MVTN [16] introduces the differentiable rendering technique to build anend-to-end learning pipeline and regress the optimal projection view.
Different from theirs, our workdesigns a novel prompting engineering scheme, utilizing 2D color knowledge from pre-trained imagemodels that is absent in colorless point clouds.
Moreover, our framework is implemented in a fastersingle-view pattern, as we only select one random projection view during training and don’t developany aggregation strategy to explicitly fuse multi-view knowledge.
Approach
3.1 Overview
The overall framework of our P2P framework is illustrated in Figure 2.
The network architectureconsists of three components: 1) a Point-to-Pixel Prompting module to produce colorful images frompoint clouds, 2) a pre-trained image model to leverage pre-trained knowledge from image domain,and 3) a task-speciﬁc head to perform various kinds of point cloud tasks.
We will introduce the Point-to-Pixel Prompting module and task-speciﬁc heads in detail in the following sections.
With the proposed architecture that can be optimized in an end-to-end manner, we are able to exploit2D pre-trained knowledge for point cloud analysis from two perspectives.
In the forward process, thepoint clouds are projected into images with preserved geometry information and the resulting imagescan be recognized and handled by the pre-trained 2D ViT model.
In the backward optimization,the frozen pre-trained ViT weights act as an anchor and guide the learnable coloring module tolearn extra color knowledge for colorless point clouds, without explicit manual interference and only [Figure] under the weak supervision from the overall target functions of downstream tasks.
Therefore, theresulting colorful images are expected to mimic patterns in 2D images and to be distinguishable forthe pre-trained ViT model in downstream tasks.
3.2 Point-to-Pixel Prompting
[Figure] Following the principle of prompt tuning mechanism introduced in Section 2.2, we propose Point-toPixel Prompting to adapt point cloud analysis to image representation learning tasks, on which the2D ViT model is initially pre-trained.
As illustrated in Figure 2, we ﬁrst perform point cloud featureencoding to extract geometric information, obtaining a rough perception of the input point cloud.
Then we introduce geometry-preserved projection to transform the 3D point cloud into 2D images,rearranging 3D geometric features according to the projection correspondences.
Finally, we proposea geometric-aware coloring module to dye projected images, transferring 2D color knowledge in thefrozen ViT model to the colorless point cloud and obtaining more distinguishable images that can bebetter recognized by the pre-trained ViT model.
3.2.1 Point Cloud Feature Encoding
One of the most signiﬁcant advantages of 3D point clouds over 2D images is that point clouds containmore spatial and geometric information that is compressed or even lost in ﬂat images.
Therefore, weﬁrst extract geometry features from point clouds for better spatial comprehension, implementing alightweight DGCNN [64] to focus on the local region of each point.
Given an input point cloud P ∈ R N × 3 with N points, we ﬁrst locate k -nearest neighbors N ∈ 143 R N × k × 3 of each point.
Then for each local region, we implement a small neural network h Θ toencode the relative position relations between the central point p i and the local neighbor points N p i ,resulting in geometric features F = { f i , 0 ≤ i < N } ∈ R N × C with dimension C :
[Equation]
where x i , x j are coordinates of p i , p j respectively, maxpool N pi and concat N pi stand for max-poolingand concatenation within all points p j in local neighbor region N p i respectively.
3.2.2 Geometry-Preserved Projection
Once obtaining geometric features F ∈ R N × C of the input point cloud P , we further rearrange theminto an image-style layout ˆ F ∈ R H × W × C to prepare for producing colorful image I , where H, Ware height and width of the target image I .
We elaborately design a geometry-preserved projection toavoid information loss when casting 3D point clouds to 2D images.
The ﬁrst step is to ﬁnd spatial correspondence between point coordinates X ∈ R N × 3 and image pixelcoordinates Y ∈ R N × 2 .
Since there is a dimensional diminishing during the projection process,we randomly select a projection view during training to construct a stereoscopic space with ﬂatimage components.
Equivalently, we rotate the input point cloud with rotation matrix R ∈ R 3 × 3to get 3D coordinates ˜X after rotation: ˜X = X R T .
The rotation matrix R is constructed throughtwo steps: ﬁrst rotating around the axis u
θ
=
(0 , 0 , 1) by angle θ , then rotating around the axis u φ = (sin θ, − cos θ, 0) by angle φ , where θ ∈ [ − π, π ] and φ ∈ [ − π/ 2 , π/ 2] are random rotationangles during training and ﬁx-selected angles during inference.
Then we just omit the ﬁnal dimension˜X : , 2 and evenly split the ﬁrst two dimensions into 2D grids:
y i,d = (cid:98) ˜x i,d /g d (cid:99) , where 0 ≤ i < Ndenotes point index, d = 0 , 1 denotes coordinate dimension, g d denotes grid size at dimension d .
The second step is to rearrange per-point geometric features F into per-pixel ˆ F according to coordinates correspondence between X and Y .
If there are multiple points S h,w = { p j } falling in the samepixel at ( h, w ) , which is a common situation, we add the features of these points altogether to producethe pixel-level feature: ˆ f h,w = (cid:80) p
j ∈S h,w f j .
The summation operation brings two advantages relatedto our geometry-preserved design.
On the one hand, we take all points in one pixel into considerationinstead of the common practice that only keeps the foremost point according to depth and occlusionrelation.
In this way, we are able to represent and optimize all points in one image and produceimages containing semitransparent objects with richer geometric information as shown in Figure 1.
On the other hand, we conduct a summation operation instead of taking the average, resulting in largerfeature values when there are more points in one pixel.
Such design maintains the spatial densityinformation of point clouds during the projection process, which is lacked in image representationsand is critical in preserving geometry knowledge.
In conclusion, our proposed geometry-preserved projection is able to produce geometry-aware imagefeature map ˆ F that contains plentiful spatial knowledge of the object.
Note that we only utilize oneprojection view during training and do not explicitly design any aggregation functions for multi-viewfeature fusion.
Therefore, we follow a single-view projection pipeline which is much more efﬁcientthan its multi-view counterpart.
3.2.3 Geometry-Aware Coloring
Despite that 3D point cloud contains richer geometric knowledge than 2D images, colorful picturesembrace more texture and color information than colorless point clouds, which is also decisive invisual comprehension.
The frozen ViT pre-trained on abundant images learns to perceive the visualworld not only based on object shape and outlines, but also heavily relied on discriminative colorsand textures.
Therefore, the image feature map ˆ F that contains only geometry knowledge and lackscolor information is not most suitable for the pre-trained ViT to understand and process.
In orderto better leverage pre-trained 2D knowledge of the frozen ViT model, we propose to predict colorsfor each pixel, explicitly encouraging the network to migrate color knowledge in the pre-trained ViTmodel to ˆ F via the end-to-end optimization.
Since the input ˆ F contains rich geometry informationthat will heavily affect the coloring process, the resulting images are expected to display differentcolors on different geometry parts, which has been veriﬁed in Figure 1.
Therefore, we referred to ourdyeing process as geometry-aware coloring.
More speciﬁcally, we design a lightweight 2D neural network g Φ to predict RGB colors
C
= { c h,w } ∈
R H
× W × 3 for each pixel ( h, w ) :
c h,w = g Φ ( ˆ f h,w ) .
We implement several 3 × 3 convolutions in g Φfor image smoothing, as the initial projected image feature ˆ F are relatively discontinuous due tothe sparsity of the original point cloud.
Therefore, the smoothing operation is critical in producingmore realistic images that the pre-trained ViT can recognize.
The resulting colorful images are thenprepared for further image-level feature extraction through the pre-trained ViT model.
3.3 Optimization on Downstream Tasks
Take Vision Transformer (ViT)
[12] as the pre-trained image model for example.
The outputsfrom the pre-trained ViT model are image token features ¯
F ∈ R N t × C t and one class token feature¯ f cls ∈ R 1 × C t , where N t is the number of image patches and C t is the token feature dimension.
Fordifferent downstream tasks, we design different task-speciﬁc heads and optimization strategies.
Object Classiﬁcation For object classiﬁcation, we follow the common protocol in image Transformer models to utilize the class token ¯ f cls as the input to the classiﬁer CLS implemented as onelinear layer: p = softmax(CLS(¯
f cls ))) .
We use the CrossEntropy loss as the optimization target.
Part Segmentation For part segmentation, we rearrange the token features ¯ F into image layoutsand upsample them to H × W .
Then we design a lightweight 2D segmentation head SEG based on
[Table]
[Table]
[Table]
[Table]
[Table]
[Table] V-T V-S V-B
C-T
C-SC-
[Table]
C-L
RR-
RS-T
S-S S-
[Table]
ViT
[Table]
ResNet Swin
[Table]
84 ImageNet Top-
[Table]
[Table]
[Table]
[Table]
V
[Table]
T
V-S
V-B
C
[Table]
L
R
[Table]
[Table]
S-T
S-S S-
[Table]
ViT
[Table]
ResNet Swin
SemanticFPN [23] to predict per-pixel segmentation logits: p h,w = softmax(SEG(¯
f h , w )) .
Insteadof re-projecting 2D segmentation results back to 3D points and supervising with 3D labels, weproject 3D labels into 2D image-style labels, exactly as how the point cloud is projected.
Then weimplement a per-pixel multi-label CE loss as there may be points from multiple classes projectedto the same pixel: L seg =
(cid:80) h,w (cid:80)
k − y h,w,k log p h,w,k .
The values of multi-hot 2D label y areassigned according to projection correspondences, satisfying (cid:80) k y h,w,k = 1 .
Supervision in 2Ddomain speeds up the training procedure without much information loss, since we keep all featuresof points in one pixel and the optimization target is accordingly based on their category distributions.
During inference, we select multiple projection views and re-project 2D per-pixel segmentationresults back to 3D points, fusing multi-view predictions.
Therefore, the per-point segmentation isdecided by the most evident predictions from the most distinguishable projection directions.
Experiments
Datasets and Experiment Settings
Datasets.
We conduct classiﬁcation on ModelNet40 [69] and ScanObjectNN [60], while ShapeNetPart [60] is utilized for part segmentation.
ModelNet40 is a synthetic 3D dataset containing 12,
CAD models from 40 categories.
ScanObjectNN samples from real-world scans with backgroundand occlusions.
It contains 2,902 samples from 15 categories, and we choose the perturbed (PB-T50RS) variant.
ShapeNetPart samples 16,881 objects covering 16 shape categories from the synthetic ShapeNet and annotates each object with part-level labels from 50 classes.
Implementation Details.
We implement our proposed P2P architecture with PyTorch [42] andutilize AdamW [38] optimizer with learning rate 5 e − 4 and weight decay 5 e − 2 , accompanied bythe CosineAnnealing learning rate scheduler [37].
We optimize the parameters of Point-to-Pixel
[Table]
[Table]
[Table]
Prompting module, task-speciﬁc head and normalization layers in ViT, freezing the remaining pretrained weights of the ViT model.
The model is trained for 300 epochs with a batch size of 64.
During training, the rotation angle θ is randomly selected from [ − π, π ] and φ is randomly selectedfrom [ − 0 .
4 π, − 0 .
2 π ] to keep the objects standing upright in the images.
During inference, we evenlychoose 10 values of θ and 4 values of φ to produce 40 views for majority voting.
Please refer to thesupplementary material for architectural details.
4.2 Object Classiﬁcation
Main Results.
We implement our P2P framework with different image models of different scales,ranging from convolution-based ResNet [19] and ConvNeXt [36] to attention-based Vision Transformer [12] and Swin Transformer [34]. These image models are pre-trained on ImageNet-1k [25]with supervised classiﬁcation. We report the image classiﬁcation performance of the original imagemodel, the number of trainable parameters after Point-to-Pixel Prompting, and the classiﬁcationaccuracy on ModelNet40 and ScanObjectNN datasets, as shown in Table 1.
From the quantitative results and accuracy curve, we can conclude that enlarging the scale of thesame image model will result in higher classiﬁcation performance, which is consistent with theobservations in image classiﬁcation. Therefore, our proposed P2P prompting can beneﬁt 3D domaintasks by leveraging the prosperous development of 2D visual domain, including abundant trainingdata, various pre-training strategies and superiors fundamental architectures. Comparisons with Previous Methods. Comparisons with previous methods on the ModelNet40and ScanobjectNN are shown in Table 2 and Table 3.
For baseline comparisons, we select architecturebased methods [46, 59, 64, 40, 16] and traditional pre-training works [76, 63] in 3D domain. For our
P2P framework, we show results on two versions: (1) base version with ResNet-101 as image model,(2) advanced version with ConvNeXt-L pre-trained on ImageNet-21k dataset [50] as image model,additionally replacing the linear head with a multi-layer perceptron (MLP).
From the results we can draw three conclusions.
Firstly, our P2P outperforms traditional 3D pretraining methods.
This suggests that the pre-trained knowledge from 2D domain is useful for solving3D recognition problems and is better than directly pre-training on 3D datasets with limited data.
Secondly, we obtain competitive performance with architecture-based works which require muchmore trainable parameters than ours and achieve state-of-the-art performance on ScanObjectNNdataset.
Therefore, our P2P framework fully exploits the potential of pre-training knowledge fromimage domain and open a new avenue for point cloud analysis.
Finally, our P2P framework performsrelatively better on real-world ScanObjectNN dataset than synthetic ModelNet dataset.
This maybe caused by the data distribution of ScanObjectNN dataset being more similar to the pre-trained ImageNet dataset, as they both contain objects from the natural world.
This prosperity reveals thepotential of our model in real-world applications.
Visualization Analysis.
The visualizations of our projected colorful images are shown in Figure 1.
The ﬁrst line shows several point cloud samples from the synthetic ModelNet40 dataset and real-world ScanObjectNN dataset, while the second and third lines illustrate the colorful images from Point-to- [Table] [Table]
[Table]
Pixel Prompting module from different projection views.
Two conclusions can be drawn from theillustration.
Firstly, our geometry-preserved projection design maintains most spatial information,resulting in images of semitransparent objects that avoid occlusion problems, such as the chair legin the second row 5 th column.
Secondly, the colors of the objects are reasonable and even suggestpart-level distinctions to some extent, making projected images more similar to natural images thatcan be better processed by the pre-trained image model.
4.2.1 Ablation Studies
Introducing 2D Pre-trained Knowledge to 3D Analysis.
In Section 4.2, we demonstrate thatleveraging pre-trained knowledge from the image domain with appropriate prompting can obtaincomparable performance with state-of-the-art methods.
Therefore, we conduct ablation studies todiscuss whether the pre-trained weights are crucial in our P2P framework, as shown in Table 4a.
In model A, we keep the same architecture as our P2P framework and train all parameters withoutloading pre-trained ViT weights.
However, the classiﬁcation accuracy decreases dramatically dueto the imbalance between 82.0M trainable parameters and the limited training data of ModelNet40dataset.
Therefore, in model B, we additionally replace ViT-base with ViT-tiny to decrease themodel scale.
However, its classiﬁcation accuracy is still far worse than P2P with pre-trainingknowledge, even though model B still contains much more trainable parameters than P2P.
Therefore,the pre-trained knowledge from the image domain is the key component of our P2P framework.
Tuning Extent of the Pre-trained ViT Model.
During the training process of our P2P framework,we froze most of the ViT parameters and only tune weights of Normalization layers.
Therefore,we also conduct ablation studies on the extent of tuning ViT model: (1) Model C: tuning all ViTparameters, which is the most severe tuning degree.
(2) P2P: tuning only normalization parameters,which is a modest tuning solution.
(3) Model D: ﬁx all ViT parameters without any tuning, which isthe weakest tuning degree.
According to the results in Table 4b, tuning normalization parametersis the most suitable solution, avoiding 2D information lost during massive tuning (model C) whileadapting the model to point cloud data distribution that model D variant fails to accomplish.
Point-to-Pixel Prompting.
In Section 3.2, we propose Point-to-Pixel Prompting mechanism, whichis much different from the prompting approaches in NLP ﬁeld.
Therefore, we conduct ablationson prompting choices to illustrate that the proposed Point-to-Pixel Prompting is the most suitableprompting method for adapting pre-trained image knowledge to point cloud analysis.
As the baselineexperiment, we implement one of the best architecture-based works PointMLP [40] with the samelinear classiﬁcation head as ours and lower the feature dimension of the third block, obtaining a PointMLP-tiny (PT) version with similar trainable parameters as our P2P framework. Then we add
[Table]
ViT-base model with pre-trained weights from image domain and ﬁne-tune only normalization layersas P2P, denoted as model PT-A.
Finally, in model PT-B, we implement the advanced continuousprompting mechanism P-Tuning [33, 32] that adds learnable prompt tokens to the input of Transformerblocks, which has also been adopted in VPT [21].
Comparing results between model PT and model
PT-A in Table 4d, we ﬁnd that adding pre-trained ViT to PointMLP-tiny without any promptingmechanism will decrease the performance due to the difference between point clouds and naturalimages.
Comparing results between model PT and model PT-B, we conclude that VPT does not workwell when tuning the pre-trained 2d ViT model to point cloud classiﬁcation, probably because thehigh-level features encoded by PointMLP-tiny cannot be understood by pre-trained ViT even withthe learnable prompt tokens.
Therefore, our proposed Point-to-Pixel Prompting is a better solution totuning pre-trained 2D knowledge for point cloud analysis.
Image Coloring.
In our Point-to-Pixel Prompting, we propose to produce colorful images to adaptto the pre-trained image ViT.
In Section 3.2.3, we have discussed our motivation and advantagesof obtaining colorful images.
Here we further prove the statement via ablation studies in Table 4c.
Model E processes per-pixel feature ˆ F from Section 3.2.2 to directly generate image tokens, whichare the inputs to the ViT blocks.
In this variant, we bypass the explicit image generation process anddirectly adopt patch embedding layers on feature map ˆ F .
Model F generates binary black-and-whiteimages conditioned only on the geometric projection from the point cloud, without predicting pixelcolors as in P2P.
According to the results from Table 4c, Model E introduces much more trainableparameters due to the trainable patch embedding projection convolution layer with kernel size 16,while producing inferior classiﬁcation results than P2P.
On the other hand, even though Model Frequires fewer trainable parameters, its performance lags far behind.
Therefore, producing colorfulimages as the prompting mechanism can best communicate knowledge between the image domainand point cloud domain, fully exploiting pre-trained image knowledge from the frozen ViT model.
4.3 Part Segmentation
The quantitative part segmentation results on ShapeNetPart dataset are shown in Table 5.
We implement the base version of ConvNeXt [36] as image model and SemanticFPN [23] as 2D segmentationhead for baseline comparison. We further implement the large version of ConvNeXt as image modeland more complex UPerNet [70] as 2D segmentation head to obtain better results. Our P2P frameworkcan achieve better performance than classical point-based methods, which demonstrates its potentialin performing 3D dense prediction tasks based on 2D pre-trained image models.
We leave it forfuture work to develop advanced segmentation heads and supervision strategies to better leveragepre-trained 2D knowledge in object-level or even scene-level point cloud segmentation.
Conclusion
In this paper, we propose a point-to-pixel prompting method to tune pre-trained image models forpoint cloud analysis.
The pre-trained knowledge in image domain can be efﬁcaciously adapted to 3Dtasks at a low trainable parameter cost and achieve the state-of-the-art performance on ScanObjectNNclassiﬁcation.
The proposed Point-to-Pixel Prompting builds a bridge between 2D and 3D domains,preserving the geometry information of point clouds in projected images while transferring colorinformation from the pre-trained image model back to the colorless point cloud.
Experimental resultson object classiﬁcation and part segmentation demonstrate P2P’s superiority and potential.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Section 4.3.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
There
isn’t any potential negative societal impacts of our work.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] See Section 4.
and supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Section 4.1 and supplemental material.
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[N/A] We follow the common practice of the previous paper,
where they didn’t report the error bars.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See supplemental material.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See Section 4.1.
(b) Did you mention the license of the assets?
[N/A]
[Table]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
[Table]
We doesn’t use any new assets.
[Table] (d) Did you discuss whether and how consent was obtained from people whose data you’re [Table]
using/curating?
[N/A]
[Table]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
[Table]
information or offensive content?
[N/A]
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[N/A]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[N/A]
[Table]