Joint Learning of 2D-3D Weakly Supervised Semantic Segmentation
Anonymous Author(s)
Abstract
The aim of weakly supervised semantic segmentation (WSSS) is to learn semanticsegmentation without using dense annotations.
WSSS has been intensively stuidedfor 2D images and 3D point clouds.
However, the existing WSSS studies havefocused on a single domain, i .
e .
2D or 3D, even when multi-domain data is available.
In this paper, we propose a novel joint 2D-3D WSSS framework taking advantageof WSSS in different domains, using classification labels only.
Via projection, weleverage the 2D class activation map as self-supervision to enhance the 3D semanticperception.
Conversely, we exploit the similarity matrix of point cloud featuresfor training the image classifier to achieve more precise 2D segmentation.
In bothdirections, we devise a confidence-based scoring method to reduce the effect ofinaccurate self-supervision.
With extensive quantitative and qualitative experiments,we verify that the proposed joint WSSS framework effectively transfers the benefitof each domain to the other domain, and the resulting semantic segmentationperformance is remarkably improved in both 2D and 3D domains.
On ScanNetV2benchmark, our framework significantly outperforms the prior WSSS approaches,suggesting a new research direction for WSSS.
Introduction
Learning-based semantic segmentation methods have achieved remarkable performance in recent.
However, they usually rely on strong supervision such as pixel-wise (2D) or point-wise (3D) dense semantic labels.
Since dense labels are time-consuming and labor-intensive to annotate, the labelingprocess has remained the main obstacle to applying semantic segmentation to real problems.
Weaklysupervised semantic segmentation (WSSS) has flourished to learn semantic segmentation by usinginexpensive weak labels ( e g .
bounding boxes or class labels).
WSSS using class labels has been widely studied in the 2D domain (image) and has recently also beenscaled into the 3D domain (point cloud).
For acquiring pixel- or point-wise class predictions withoutany spatial supervision, the existing 2D and 3D WSSS studies have employed Class Activation Maps(CAM) [35] extracted from a classifier.
Since CAM usually localizes the discriminative regionsof each class, one can acquire pseudo-labels for semantic segmentation from the CAM.
While thestudies have shown promising results, they focus on using the data of a single domain and place lessemphasis on the information of other domains that might have complementary properties.
With rich and dense features from a CNN-based architecture, 2D WSSS succeeds in the semanticperception of images.
However, the 2D CAM is usually not sharp at the object boundary and thesegmentation result tends to be imprecise as shown in (b3) of Fig 1.
In terms of segmentation, 3D
[Figure]
Ground
[Figure]
WSSS is advantageous since the input point cloud contains the 3D geometry of the scene.
On theother hand, 3D WSSS suffers from class imbalance problems and has difficulty in predicting thesemantics from sparse point clouds, as in (b1) and (b2).
The results of 2D and 3D WSSS showdifferent tendencies originating from the distinct domains, and they are complementary for achievingsemantic segmentation of the scene.
Upon this background, we propose a novel 2D-3D WSSS framework that jointly targets both domainsof image and point cloud.
For unifying the complementary strengths of both domains, we propagatethe benefit of one domain to the other domain via self-supervision, and vice versa.
It enables ourframework to individually perform 2D and 3D WSSS, without requiring the paired input data in theinference phase.
We obtain a 2D pseudo-label from the 2D CAM of the image classifier and make3D CAM follow it, which enhances the semantic perception of the point cloud classifier.
On the otherhand, to transfer the segmentation capability of 3D CAM to 2D, we construct a correlation matrixbetween the 3D features.
The image classifier is then trained to generate 2D features having a similarcorrelation matrix to that of the 3D, which relieves the impreciseness problem of 2D CAM.
In bothdirections, to prevent the networks from being spoiled by inaccurate self-supervision, the frameworkobtains a confidence score for each region and rejects erroneous predictions.
With extensive qualitative and quantitative experiments, we verify that the proposed joint frameworkeffectively transfers the benefit of each domain to the other domain as visualized in (c1-c3) of Fig.
As a result, in both image and point cloud domains, the semantic segmentation performance isremarkably enhanced simultaneously.
For example, on the ScanNetV2 [5] benchmark, our methodsignificantly outperforms the existing WSSS studies, demonstrating its superiority.
The proposedframework is the first multi-domain approach for the field of WSSS, and we believe that 2D-3D jointlearning is a promising direction for conquering semantic segmentation with class labels only.
[Table]
Related works
In this section, we introduce the existing studies on 2D/3D WSSS and clarify the uniqueness of theproposed WSSS method that targets both domains in a simultaneous manner.
Table 1 shows thesetting of the weak labels we used and those of existing studies.
Note that the proposed framework isthe first method that exploits the complementary strengths of WSSS in the 2D and 3D domains, usingimage- and scene-level class labels only.
WSSS in 2D domain 2D WSSS with image-level class labels only [11, 13, 1, 32, 24, 16, 3, 7, 25]has been widely studied for the last decade.
Since the class labels do not provide any spatialsupervision, most WSSS studies have exploited a CAM for localizing the regions of each class.
Although the CAM can serve as an acceptable estimate for localization, they are usually activated inthe most discriminative regions only, rather than capturing whole object regions.
Moreover, the CAMis sensitive to the scale variance of the input image and less precise at the boundary of the objects.
Torelieve these issues, existing works have proposed various techniques such as refining [11, 13, 1],adversarial
erasing [32, 24, 16], sub-categorical classification [3], and cross-image attention [7, 25].
However, their problem definition and benchmark dataset [6, 18] mainly focus on the 2D domain andless consider 3D structures of the scene, which are valuable for solving the aforementioned issues.
WSSS in 3D domain Point-wise dense semantic segmentation labeling for a 3D point cloud isnotorious for being difficult to annotate (22.3 min) [29].
To resolve this, several studies [20, 31,8, 33, 34] have been conducted to learn semantic segmentation with only partially labeled points.
However, since acquiring partial point-wise annotation is still expensive even under the “One Thing
One Click" setting (2 min) [20], a few studies have been conducted for exploiting less expensive classlabels (15 sec) [29].
MPRM [29] samples subclouds from the full scene and annotates them, sinceusing subcloud-level class labels can relieve the class imbalance issue ( e .
g . wall or floor ) commonlyappearing in almost every scene).
Recently, WyPR [22] jointly targets semantic segmentation andobject detection of a point cloud with scene-level class labels only.
The studies have shown thefeasibility of WSSS in the 3D domain; however, due to the sparse and irregular nature of the pointcloud, the performance of 3D WSSS is limited compared to the 2D WSSS.
Semantic segmentation in combined 2D-3D domains To exploit both 2D and 3D data for semanticsegmentation, several semantic segmentation studies have been conducted for uni-directional[12, 4]or bi-directional[10] feature projection between the 2D and 3D domains, in a fully-supervised setting .[28, 15] leveraged the 2D semantic segmentation for learning the 3D semantic segmentation withmulti-view or virtual-view settings; however, they still require a dense 2D GT.
[26] proposed touse 3D information as guidance for enhancing 2D WSSS using bounding boxes.
Recently, [9, 17]incorporated 3D priors for self-supervised learning of 2D networks to achieve 3D-aware scenerecognition.
Unlike the existing studies, our framework is the first WSSS approach that jointly unifiesthe complementary benefits of 2D and 3D domains, by using the image-level and scene-level classlabels only.
Method
In this section, we introduce the proposed 2D-3D joint framework for learning WSSS in both domains.
Figure 2 shows our overall framework
3.1 Acquiring CAM of each dimension
[Figure]
Following the prior works with the class labels only, we exploit a CAM to localize the class regionsfrom the image or point cloud.
We assume that a set of K images I = { I 1 , · · · , I K } and a pointcloud P are obtained from a certain scene.
I k is an RGB image with a size of H × W , and P is a setof
N points where each point is represented by its 3D coordinate (XYZ) and color (RGB), as in [22].
A view matrix V k (perspective projection matrix) of the camera for the image I k is also given, asin many point cloud datasets [5, 2].
As weak supervision, we use the image-level class label t 2 dand scene-level class label t 3 d , where both labels are represented as C -dimensional binary vectors.
C denotes the total number of classes, and the classes are non-exclusive since multiple classes canco-exist in a single image (or point cloud).
As visualized in Fig.
2, we train an image classifier F to extract a 2D feature map Z 2 d ∈ R D × H × W .
A single linear layer having D input channels and C output channels then serves as a classificationhead for making pixel-wise class prediction A 2 d ∈ R C × H × W .
Similarly, a point cloud classifier G istrained to provide point-wise features Z 3 d ∈ R D × N and point-wise class prediction A 3 d ∈ R C × N .
Here, we directly employ A 2 d and A 3 d as the CAM of each domain, and refer to them as 2D CAMand 3D CAM throughout this paper.
For both networks, we use the GAP
(Global Average Pooling)layer to average the pixel- or point-wise predictions along the spatial dimension as follows:
[Equation]
where y 2 d ∈ R C and y 3 d ∈ R C denote image-level and scene-level class prediction, respectively.
Topromote the 2D and 3D CAMs to be activated on the corresponding regions of classes, the classifiersare trained with classification loss as following, where ℓ bce denotes the binary cross entropy loss.
[Equation]
3.2 From 2D to 3D: Transferring semantic capability
We are inspired by an observation that the information contained in 2D and 3D data is not only distinctbut also complementary.
The proposed framework aims to jointly learn WSSS in both domains andunify their strengths.
For joint learning of both domains, we could directly provide informationfrom one domain to the other domain via feature paths using projection, as in [10].
However, in theinference phase, a model including the feature path requires the paired set of data of both domains,which greatly limits the practical use case.
Therefore, instead of directly swapping the features, wedevise our framework to provide information from one domain to the other domain as a form ofself-supervision during the training process only.
This enables the proposed 2D-3D framework toseparately process the data of each domain – our framework can perform 2D WSSS and 3D WSSSindividually in the inference phase, without using the input data from the other domain.
We use the view matrices of images for bridging the 2D and 3D domains, as visualized in Fig. 2.
The projection from 3D homogeneous coordinates to the 2D pixel coordinates in I k is formulated asfollows:
[Equation]
where ( x n , y n , z n ) is the coordinate of the n th point in the point cloud P and ( u kn , v kn ) is thedestination of the projected point in I k .
We denote the projection function with the view matrix V kas proj ( · , V k ) .
For simplicity, we omit k throughout this paper.
With a given V , the 3D point features Z 3 d and 3D CAM A 3 d are projected onto the image plane.
Here, we discard the points located outside of the image grid.
Also, we filter the points of theoccluded object which should not exist on the image.
The projected 3D features Z 3 d,p ∈ R D × N v and3D CAM A 3 d,p ∈ R C × N v can be obtained as follows: Z 3 d,p = proj ( Z 3 d , V ) and A 3 d,p = proj
[Equation]
where N v denotes the number of preserved points.
Note that the projection process is differentiable,and therefore the gradients can be back-propagated (see blue dashed lines in Fig. 2).
Once the projected 3D CAM is acquired, we bridge the 2D CAM and 3D CAM at the pixels wherethe projected points exist.
To leverage the semantic advantages of the 2D domain for improving3D CAM, we obtain a pseudo-label map M 2 d from the 2D CAM.
Since there is usually no explicitbackground class in indoor scenes, we regard the class with maximum prediction score as a predictedclass at each pixel as follows:
[Equation]
where the non-existing classes are ignored by using the given image-level class label t 2 d .
We apply point-wise classification loss to the 3D CAM to follow the acquired 2D pseudo-label.
However, unlike GT supervision, the predicted 2D pseudo-labels may include inaccurate signals andcannot be fully trusted.
To address this, we regard the maximum score of class prediction of eachpixel as a metric for the confidence of that pixel.
The confidence map W 2 d is defined as follows:
[Equation]
[Equation]
We define the 2D-to-3D loss that transfers 2D semantic capability to 3D network as
[Equation]
where the ( u n , v n ) is a pixel coordinate of the n th projected point as we denoted.
Using maximumlogit as a confidence score can reduce the contributions of such less confident pixels to the loss, andprevent 3D CAM from learning with erroneous self-supervision.
3.3 From 3D to 2D: Transferring segmentation capability
Thanks to the well-preserved geometrical structures of the point cloud, the 3D WSSS can be advantageous for segmenting the scenes compared with 2D WSSS.
However, unlike 2D-to-3D loss utilizing2D pseudo-labels, this benefit cannot be directly leveraged to mitigate the impreciseness of 2D CAMdue to the class imbalance issue and low semantic perception capability of the 3D CAM.
In this paper,to unlock the segmentation potentials of the point cloud classifier, we train the image classifier togenerate 2D features having a similar correlation matrix with that of the 3D features.
Motivated by [19], we construct a correlation matrix S 3 d,p from the projected 3D features Z 3 d,p , byusing the following equation:
[Equation]
where ( a, b ) denotes a pair of the projected points.
Similarly, for the pixels corresponding to theprojected points, S 2 d is computed from 2D features Z 2 d as follows:
[Equation]
where the ( u a , v a ) and ( u b , v b ) denote pixel coordinates of the a th and b th projected point, respectively.
Note that we map the features on a unit hypersphere by normalization, before computingthe correlation matrices.
Similar to the 2D-to-3D loss, we construct W 3 d,p , the confidence of thecorrelation matrix of projected 3D features, by multiplying the confidence at each point as follows:
[Equation]
The confidence-based 3D-to-2D loss is then defined as follows:
[Equation]
In summary, we apply the sum of the classification and the joint loss functions for training ourframework as following equation, where λ is a weighting parameter.
[Equation]
Experiments
4.1 Implementation details
Framework The proposed framework is implemented with PyTorch.
ResNet
[30] and PointNet++ [21] are employed as backbones for the image classifier and point cloud classifier, respectively.
The image classifier is initialized with the weights pre-trained on ImageNet [23].
We augment theinput point cloud with sub-sampling, random flipping, and random rotation.
For the image, randomresizing/cropping, horizontal flipping, and color jittering [14] are applied.
We set λ = 1 in Eq.
The model is trained on two Tesla V100 GPUs with batch size 16 for 200 epochs.
The initial learningrate is set to 0.003, and be decayed by 0.1 at epoch 120, 160, 180 as in [22].
More details can befound in Supplementary Material .
Dataset and metric
Following the existing 3D WSSS studies [29, 22], we conduct experiments on
ScanNetV2 [5] dataset (MIT license, we agreed to the terms of use), which contains indoor scenessuch as offices or bathrooms.
ScanNetV2 is composed of 1513 scans with annotations (semanticsegmentation GT, camera poses, etc.) and 100 test scans serving as a benchmark.
We follow theofficial split, where there exist 1201 train scans and 312 val scans.
For the 2D data, we use theprovided RGB frames and the corresponding camera view matrices.
We sample only 1% of allprovided frames, which is equivalent to using around 17 frames per scene on average.
Note thatthe proposed label setting is less expensive than that of MPRM [29] (using around 17 class-labeledsubclouds per scene), since annotating class-level labels is even easier on images than on point clouds.
For evaluating the semantic segmentation performance, we use the mean intersection over union(mIoU) as a metric on both the 2D and 3D domains.
Since the GT for the test set is not publiclyavailable, test performance is evaluated on the ScanNet evaluation server.
[Table]
2D CAMs – (b1) uses image-level classification loss only, (b2) uses 3D-to-2D loss, and (b3) uses bidirectional joint loss.
Additional 2D/3D CAMs can be found in Supplementary Material .
4.2 Ablation studies
To demonstrate the advantage of the proposed joint learning framework, we conduct an ablationstudy, as shown in Table 2.
For each domain, the first row shows the result of our baseline setting,where only the image- or scene-level classification loss of each domain ( L 2 dcls or L 3 dcls ) is applied.
Asreported in prior works [29, 22], we also observe that the 3D WSSS suffers from the class imbalanceof scene-level class labels and thereby completely fails to understand the semantics of the objects, asvisualized in (a1) of Fig.
On the other hand, as shown in (b1), 2D WSSS shows the capability toroughly localize the region of each class on the image.
However, due to the lack of geometrical 3Dinformation, the 2D CAM fails to be precise at the object boundary.
This illustrates the motivation ofthe proposed 2D-3D joint framework.
A benefit of the 2D-to-3D loss ( L 3 d → 2 d joint ) is shown in the second row of the 3D domain in Table 2.
The train and val performance of 3D CAM are significantly increased from 17.9% and 16.2% to51.3% and 43.8%, respectively.
The 3D CAMs in Fig.
3 (a2) show much more accurate resultswith the help of the 2D-to-3D joint loss.
Here, to clarify the benefit of the 2D-to-3D loss, we alsoconduct an additional experiment.
In this experiment, instead of the joint loss, we apply an additionalclassification loss to the point cloud classifier by using the image-level class labels.
This settingis similar to that of PCAM [29]; however, the classification is done on the image level, not on thesubcloud level.
It achieves 43.9% and 39.8% on train and val split respectively, which are the almostequivalent performance to PCAM (43.1% and 38.1%) but clearly lower than using our 2D-to-3D loss(51.3% and 43.8%).
The result indicates that the performance gain of the proposed 2D-to-3D lossnot only comes from relieving the class imbalance problem of the 3D domain but also effectivelytransferring the better semantics of the 2D domain to the 3D domain.
[Table]
The results also show that the 3D-to-2D loss ( L 3 d → 2 d joint ) enhances the performance of 2D WSSS, from38.0% and 33.3% to 47.8% and 39.2%, respectively.
Thanks to the geometrical information providedby the 3D domain in the form of a correlation matrix, as visualized in Fig.
3 (b2), the generated 2D
CAMs are much more precise at the object boundary when being trained with the 3D-to-2D loss.
Theresults strongly support our design intentions: the 2D-to-3D loss provides rich semantics from 2D to3D, while the 3D-to-2D loss transfers the segmentation capability of 3D to 2D.
We quantitatively verify that the confidence-based scoring strategy is effective in both the 2D-to-3Dand 3D-to-2D directions, by suppressing the self-supervision at inaccurate regions while preservingreliable guidance.
Finally, by using the bidirectional (2D-to-3D and 3D-to-2D) joint losses, we evenfurther improve the semantic segmentation performance in both domains (the fourth row of eachdomain in Table 2 and (a3)/(b3) in Fig. 3).
It shows the superiority of the proposed joint framework,which unlocks the complementary potential of 2D and 3D domains and unifies them.
Here, we canobserve that the performance of 2D WSSS is quite lower than that of 3D WSSS after we apply jointlosses.
One of the main reasons is the limited FoV (Field of View) of the image.
When only thecorner of an object is captured in the image, it is almost impossible to classify the object and therebythe 2D CAM is also inaccurate.
To address the issue, during the training process, we filter 2D CAMwith ground truth image-level class labels and suppress confusing regions; however, misclassificationin the inference phase is difficult to relieve.
Extending the proposed framework using the cameraswith large FoV ( e .
g .
omnidirectional camera) can be an interesting future direction.
4.3 Segmentation results
In Table.
3, we compare the point cloud semantic segmentation performance of the proposed framework with that of the other 3D WSSS methods [29, 22].
This shows the superiority of the proposed2D-3D joint framework, surpassing the other methods by more than 6% in every split.
Althoughwe use additional 2D image data and image-level class labels compared to WyPR [22], annotatingadditional image-level class labels is not very expensive, particularly when we consider the significantperformance gain.
Compared with MPRM [29] using subcloud-level labels, the annotating cost israther inexpensive at the image-level compared to the subcloud-level, while the proposed frameworkoutperforms MPRM by a significant margin.
It is also noteworthy that the proposed frameworkdoes not employ the two-stage retraining technique or dense conditional random field (CRF), unlike MPRM.
In summary, the proposed 2D-3D joint learning method outperforms the existing 3D WSSSmethods in both terms of label efficiency and performance.
Figure 4 visualizes qualitative comparisons of the proposed framework and baselines, in both 2Dand 3D domains.
Compared with the result learned with a single 2D or 3D domain, our frameworkshows significantly better results in terms of accuracy of semantics and preciseness of segmentation.
With our 2D-3D joint framework for WSSS, the complementary advantages of both domains areeffectively combined.
We can observe that the unified benefits of multi-domains achieve substantialsemantic segmentation results, compared to uni-domain baselines.
There are a few limitations suchas thin objects ( e . g .
legs of a table) in the 3D domain, fine details in the 2D domain, and sensitivity tothe pose quality. Nevertheless, our framework achieves a strong baseline for the 2D-3D joint learningof WSSS.
Additional semantic segmentation results are provided in Supplementary Material .
[Figure]
Table
[Figure]
Window
[Figure]
Bathtub
[Figure]
Conclusion
[Figure] Although WSSS has been widely studied for relieving the annotation cost of semantic segmentation inboth 2D and 3D domains, the existing studies have been focused on domain-specific methods.
In thispaper, we propose a novel 2D-3D joint framework for WSSS in both 2D and 3D domains.
Motivatedby the complementary nature of the 2D and 3D domains, we devise our framework to effectivelytransfer the benefit of one domain to the other domain in a simultaneous manner.
We bridge the 2Dpixels and 3D points via projection and leverage the 2D CAM as self-supervision for improving thesemantic perception of the 3D CAM.
To transfer the geometrical structure of the 3D scene to the2D domain, we exploit a correlation matrix of 3D features for training the 2D features.
For bothdirections, we propose a confidence-based scoring strategy to suppress less accurate self-supervisionand make the framework learn from the reliable regions only.
With extensive experiments, we verifythat the proposed 2D-to-3D and 3D-to-2D joint loss effectively unifies the distinct benefits of bothdomains, while relieving the disadvantages.
The proposed framework significantly outperforms thesingle-domain baselines and the other state-of-the-art methods.
We believe that 2D-3D joint learningis a new and promising research direction for WSSS, and this paper provides a strong baseline for it.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] We clarify our contributions in Section 1, and they areexperimentally verified in Section 4.
(b) Did you describe the limitations of your work?
[Yes] See Section 4.3.
(c) Did you discuss any potential negative societal impacts of your work?
[No] As far aswe know, WSSS would not make meaningful negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] We carefully read the guidelines and ensure it.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[No] But all the codeand instructions will be publicly available, soon.(b)
Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See Section 4.1.
(c) Did you report error bars (e.g., with respect to the random seed after running exper-
iments multiple times)?
[No] But we empirically check that the proposed method is
robust to random seed, and we are willing to provide the results regarding it if required.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Section 4.1.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] We use ScanNetV
dataset, and cite the paper
(b) Did you mention the license of the assets?
[Yes] See Section 4.1.
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes] See Section 4.1.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[Yes] See Section 4.1.
If you used crowdsourcing or conducted research with
human subjects...
(a) Did you include the full text of instructions given to participants and screenshots if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]