3D S EGMENTER : 3D T RANSFORMER BASED S EMANTIC S EGMENTATION VIA 2D P ANORAMIC D ISTILLATION
A BSTRACT
Recently, 2D semantic segmentation has witnessed a significant advancement thanks to the huge amount of 2D image datasets available.
Therefore, in this work, we propose the first 2D-to-3D knowledge distillation strategy to enhance 3D semantic segmentation model with knowledge embedded in the latent space of powerful 2D models.
Specifically, unlike standard knowledge distillation, where teacher and student models take the same data as input, we use 2D panoramas properly aligned with corresponding 3D rooms to train the teacher network and use the learned knowledge from 2D teacher to guide 3D student.
To facilitate our research, we create a large-scale, fine-annotated 3D semantic segmentation benchmark, containing voxel-wise semantic labels and aligned panoramas of 5175 scenes.
Based on this benchmark, we propose a 3D volumetric semantic segmentation network, which adapts Video Swin Transformer as backbone and introduces a skip connected linear decoder.
Achieving a state-of-the-art performance, our 3D Segmenter is computationally efficient and only requires 3 .
8% of the parameters compared to the prior art.
Our code is released on https://github.com/swwzn714/3DSegmenter .
I NTRODUCTION
Semantic segmentation assigns each 2D pixel(Long et al., 2015) or 3D point(Qi et al., 2017a)/voxel(C¸ic¸ek et al., 2016) to a separate category label representing the corresponding object class.
As a fundamental computer vision technique, semantic segmentation has been widely applied to medical image analysis(Ronneberger et al., 2015), autonomous driving(Cordts et al., 2016) and robotics(Ainetter & Fraundorfer, 2021).
Most of the existing efforts are invested in 2D settings thanks to great amount of public 2D semantic segmentation datasets(Zhou et al., 2017; Cordts et al., 2016; Nathan Silberman & Fergus, 2012).
Nowadays, the wide availability of consumer 3D sensors also largely promotes the need for 3D semantic segmentation (Han et al., 2020; Choy et al., 2019;
Thomas et al., 2019; Graham et al., 2018b; Nekrasov et al., 2021).
We have seen great success in semantic segmentation in 2D images.
However, such success does not fully transfer to the 3D domain.
One reason is that, given the same scene, processing the 3D data usually requires orders of magnitude more computations than processing a 2D image.
E.g. using a volumetric 3D representation, the number of voxels grows O ( n 3 ) with the size of the scene.
As a result, existing 3D semantic segmentation methods usually need to use smaller receptive fields or shallower networks than 2D models to handle large scenes.
This motivates us to facilitate 3D semantic segmentation using 2D models.
Specifically, we propose a novel 2D-to-3D knowledge distillation method to enhance 3D semantic segmentation by leveraging the knowledge embedded in a 2D semantic segmentation network.
Unlike traditional knowledge distillation approaches, where student and teacher models should take the same input, in our case, the 2D teacher model is pre-trained on a large-scale 2D image repository and finetuned by panoramas rendered from 3D scenes.
The panorama image is rendered at the center of the 3D scans with 360 ◦ receptive field, i.e. it contains the global context of the environment.
Our 3D student model takes the 3D scene as input and outputs 3D volumetric semantic segmentation.
Through differentiable panoramic rendering of 3D semantic segmentation prediction, we could obtain the mapping from the 2D teacher’s prediction to the 3D student’s prediction.
Then we transfer the class distributions of the pixel produced from the 2D teacher to the corresponding voxel of the 3D student.
To the best of our knowledge, this is the first solution to distill from a pre-trained 2D model to enhance the 3D computer vision task.
To facilitate this 2D-to-3D knowledge distillation design, we create a large-scale indoor dataset called PanoRooms3D .
PanoRooms3D contains 5917 rooms diversely furnished by 3D CAD objects and augmented by randomly sampled floor and wall textures.
We manually filter out scenes that contain unlabeled furniture.
We prepare for each room a dense 3D volume, and a corresponding 2D panorama that is rendered in the center of the room with 360 ◦ receptive field, both containing clean semantic labels.
Upon finding panorama-to-volume correspondence, the knowledge distillation from 2D to 3D is then enabled.
As our 3D student backbone, we propose a novel efficient architecture, 3D Segmenter, for semantic segmentation of 3D volumetric data.
Inspired by Video Swin Transformer(Liu et al., 2022a), 3D
Segmenter employ a pure-transformer structure with an efficient linear decoder.
We demonstrate that our 3D Segmenter achieves superior performance and only requires 3 .
8% of the parameters compared to the prior art that uses 3D convolutional backbones.
Our contributions are three-fold:
• We propose the first 2D-to-3D knowledge distillation method to utilize the data-abundant and pretrain-ready 2D semantic segmentation to improve 3D semantic segmentation.
• We propose PanoRooms3D, a large-scale 3D volumetric dataset with a clean voxel-wise semantic label and well-aligned corresponding 2D panoramic renderings with pixel-wise semantic labels.
• We propose a novel efficient pure-transformer-based network for the 3D semantic segmentation task.
Our 3D Segmenter outperforms prior art that uses 3D convolutional backbone with only 3 . 8% of the parameters.
We proved through experiments that our baseline variants have already achieved SoTA performance.
The distilled knowledge from 2D further widened our lead.
Considering the difficulty of 3D data collection, and the cubic memory consumption of 3D models, our model paves the way to bridge 2D and 3D vision tasks.
R ELATED W ORKS 2. 2D SEMANTIC SEGMENTATION
The success of deep convolutional neural networks (Simonyan & Zisserman, 2014; He et al., 2016) for object classification led researchers to exploit the possibility of working out dense prediction problems.
Fully Convolutional Networks (FCN)(Long et al., 2015) based encoder-decoder architectures have dominated the research of semantic segmentation since 2015.
Later works extend FCN on different facets.
Follow-up approaches (Badrinarayanan et al., 2017; Lin et al., 2017; Pohlen et al., 2017; Ronneberger et al., 2015; Zhao et al., 2017) leverage multi-scale feature aggregation.
(Fu et al., 2019;
Yin et al., 2020; Yu et al., 2020; Yuan et al., 2018; Zhao et al., 2018) apply attention mechanism to model long-range dependencies.
Recently, ConNeXt(Liu et al., 2022b) modernizes the CNNs to achieve higher performance than transformers, suggesting the effectiveness of convolutional methods.
On the other hand, ViT(Dosovitskiy et al., 2020) successfully introduced Transformer(Vaswani et al., 2017) to computer vision.
Later, SETR(Zheng et al., 2021) demonstrates the feasibility of using Transformer-based semantic segmentation.
PVT(Pyramid Vision Transformer)(Wang et al., 2021) combines pyramid structures and ViT for dense prediction.
SegFormer(Xie et al., 2021) proposes a hierarchical Transformer encoder and a lightweight MLP decoder to fuse multi-level features and predict the semantic segmentation mask.
Using vanilla ViT and DeiT(Touvron et al., 2021) as backbone, Segmenter(Strudel et al., 2021) designed a trainable mask decoder to decode a patch-level embedding and interpolate to the original resolution.
Combine with UPerNet(Xiao et al., 2018) structure, Swin Transformer (Liu et al., 2021) shows excellent performance on semantic segmentation.
2. 3D S EMANTIC S EGMENTATION
PointNet and PointNet++ (Qi et al., 2017a;b) are the pioneer works using MLP (Multi-layer Perceptron) to directly process 3D coordinates for classification and semantic segmentation.
Later works specially design convolution operations to extract features inside the neighborhood for the point clouds.
Kpconv(Thomas et al., 2019) designs both rigid and deformable kernel point convolution operators for 3D point clouds nearing a set of kernel points.
Minkowski Engine (Choy et al., 2019) proposes a sparse convolution to process high-dimensional data efficiently.
Occuseg(Han et al., 2020) voxelized the input and uses occupancy signal to achieve instance segmentation.
The aforementioned 3D semantic segmentation works generally use 3D point cloud as representation.
Apart from point cloud, volumetric representation is another widely used 3D shape representation.
3D U-Net (C¸ic¸ek et al., 2016) extends convolutional U-Net(Ronneberger et al., 2015) to 3D biomedical data segmentation.
SSCNet (Song et al., 2017a) proposed to jointly do semantic segmentation and completion on RGB-D images.
Scancomplete (Dai et al., 2018) propose a coarseto-fine structure to complete scene and predict semantics.
U-net structure with 3D CNN kernels is the dominant backbone in this research field.
2. T RANSFORMERS IN C OMPUTER V ISION
Transformer(Vaswani et al., 2017) was proposed to model long-range dependence between words in
NLP tasks.
Vision Transformer (Dosovitskiy et al., 2020) firstly introduce transformer to computer vision research by embedding image into non-overlapping patches, which serve as tokens for selfattention computation.
Variants of ViT such as T2T-ViT(Yuan et al., 2021) , TNT(Han et al., 2021),
CrossViT (Chen et al., 2021), LocalViT (Li et al., 2021),Msg-TransformerFang et al. (2022), CvT (Wu et al., 2021), and XCiT(Ali et al., 2021) optimized vanilla ViT for further image classification performance gain.
Apart from image classification, PVT (Wang et al., 2021), Swin Transformer (Liu et al., 2021), CoaT (Xu et al., 2021), LeViT(Graham et al., 2021) and Twins (Chu et al., 2021) refer to the hierarchical design of FCN(Long et al., 2015), tailor transformer model for dense prediction tasks.
Because of the remarkable performance of transformers in modeling long-range dependence, we adopt a pure-transformer architecture in this work.
2. K NOWLEDGE D ISTILLATION
The idea of model compression first came out in 2006 where (Buciluˇa et al., 2006) proposed to transfer knowledge from large model to smaller model without much performance drop.
(Hinton et al., 2015) systematically summarized existing knowledge distillation ideas.
They demonstrated the effectiveness of student-teacher strategy and response-based knowledge distillation.
Following
Hinton, many works (Ge et al., 2020; Pham et al., 2021;
Li et al., 2020; Xie et al., 2020; Mirzadeh et al., 2020;
Yang et al., 2019; Zhang et al., 2020; Wu & Gong, 2021; Touvron et al., 2021) further explore the technique of training response-based knowledge distillation models.
Since the knowledge conveyed by output layers is limited, many researchers focus on distillation through intermediate layers.
Fitnets(Romero et al., 2014) firstly introduce intermediate layer distillation by using hints from the teachers’ hidden layers to guide the training of the student.
the idea of feature-based distillation was carried forward by following researches (Zagoruyko & Komodakis, 2016; Tung & Mori, 2019;
Guan et al., 2020; Ahn et al., 2019; Heo et al., 2019; Kim et al., 2018).
Recently, BP-Net(Hu et al., 2021) utilizes a bidirectional projection module with symmetric 2D and 3D architectures to achieve mutual enhancement between 2D and 3D. Compared to our distillation design, BP-Net relies on 2D-3D paired input in both training and testing process.
3D S EGMENTER 3. E NCODER
An overview of the 3D Segmenter pipeline is shown in Figure 1.
We adapt Video Swin Transformer(Liu et al., 2022a) as our backbone.
The encoder takes a Truncated
Signed Distance Function (Curless & Levoy, 1996)(TSDF) block x = [ x tsdf , x rgb ] ∈ R X × Y × Z ×as input ,
where X × Y ×
Z specifies the size of the block and 4 channels consist of one TSDF channel and three RGB color channels.
TSDF is an implicit regular 3D shape representation that stores distance to the nearest surface in each voxel.
It is proficient in producing high-quality real-time rendering.
The encoder split input into small overlappable patches before outputting a sequence of feature embedding.
Different from 2D images, when applying transformer models for 3D tasks, the length of the token sequence grows cubically with the size of the input.
The shifted window design proposed by Swin
Transformer(Liu et al., 2021) is efficient in limiting the sequence length of each Multi-head Self
Attention (MSA) computation, which makes it has linear computational complexity to image size.
For efficient and scalable 3D computation, we choose Video Swin Transformer (Liu et al., 2022a) as our encoder backbone.
Our model is trained on fixed-sized 3D blocks cropped from 3D scenes.
Receiving an input block x = [ x tsdf ,
x rgb ] ∈ R X
× Y ×
Z × 4 , we regard each 3D patch of spatial size P × P × P as a token.
Therefore, each 3D block x is partitioned into XP × YP × ZP tokens.
A single 3D convolution layer embeds each token from 4 channels to feature dimension of size denoted by D .
To formulate hierarchical architecture, Video Swin Transformer reduces the number of tokens through patch merging between blocks.
They use a 3D patch merging operation that concatenates the C dimensional features of 2 × 2 spatially neighboring patches to 4 C .
The token size on the temporal dimension is kept the same during merging.
This reduces the number of tokens by a multiple of 2 × 2 = 4 .
To keep the resolution consistent with typical convolutional networks, the 4 C dimensional concatenated feature is downsampled to 2 C using a linear layer.
Following this design, we use a patch merging layer to merge neighboring 2 ×
2 × 2 patches into 8 C and downsample it to 2 C .
Our encoder is comprised of h Swin Transformer blocks.
Patch merging is performed between every two blocks.
Therefore, the output of encoder is a sequence with N =
[Equation]
tokens.
The dimension of each token is D × 2 h − 1 .
3. D ECODER
The decoder maps tokens coming from the encoder to voxel-wise prediction ˆ
y sem ∈ R X × Y ×
Z × K , where K is the number of categories in this task.
In this work, we propose a decoder with only two linear layers.
Accepting token sequence of shape N ×
( D × 2 h − 1 ) from the encoder, a token-wise linear layer is applied to the sequence, getting a N × K class embedding.
The class embedding is then rearranged to patch-wise class embedding
X P × 2 h − 1 × Y P × 2 h − 1 ×
Z P × 2 h − 1 × K and trilinearly interpolated to original resolution to get y f ∈ R X
× Y ×
Z × K .
We use a skip connection here to ’remind’ the model of the input x .
By adding a small number of parameters, the skip connection effectively enhances our performance.
Concretely, we concatenate x and y f and feed them through a linear layer to predict the output
ˆ y sem ∈ R X
× Y ×
Z × K =
Linear ( Concatenate ( y f , x ))
The model is trained using cross entropy loss L CE between ˆ y sem and ground truth y sem .
2D TO 3D D ISTILLATION
As shown in Figure 2, We use pairs of panoramic renderings I r and 3D rooms r =
[ r tsdf , r rgb ] ∈
R X ′
× Y ′ ×
Z ′ × 4 as our input for 2D-to-3D distillation.
The 2D teacher is pretrained on large scale 2D repository and finetuned on 2D panoramas I r and 2D semantic ground truth rendered from 3D rooms.
The 3D Segmenter, as the student model in distillation, is first trained to converge using fix-sized block data.
Then, the knowledge distillation from 2D teacher to 3D student is performed on the scene level.
Suppose we have a input room r =
[ r tsdf , r rgb ] ∈ R X ′
× Y ′ ×
Z ′ × 4 , we feed it to the student model and get voxel-level semantic segmentation ˆ y r ∈ R X ′
×
Y ′ ×
Z ′ × K .
The 2D panoramic image could be acquired from panoramic camera placed at any position that is not occupied by objects.
Here, we place it at the center of the room for sake of simplification.
Combining 3D semantic segmentation prediction ˆ y r and input geometry r tsdf , a mapping between pixels and voxels is required to project ˆ y r to 2D.
The mapping is determined by raycasting.
For each pixel in the image, we construct a ray from the view and march along the ray through r tsdf .
Trilinear interpolation is used to determine
TSDF values along the ray.
The surface voxel is located when zero-crossing is detected.
In this way, we establish a mapping between a pixel and a voxel.
Through this mapping, 6 squared semantic segmentation maps are rendered from + X , − X , + Y , − Y , + Z , − Z directions with 90 ◦ field of view.
The 6 views are then stitched into a panorama through pixel mapping and interpolation.
The total process is differentiable, making it possible for gradient backpropagates from 2D to 3D. We project a panoramic view of ˆ y r .
[Equation]
where StudentSegMap ∈ R H × W × K , H and W is the height and width of rendered panorama.
After forwarding I r through the teacher model, we receive a TeacherSegMap ∈
R H × W × K which contains the teacher model’s learned knowledge.
We follow (Hinton et al., 2015) to formulate a Kullback–Leibler divergence loss to measure the difference of distribution between StudentSegMap and TeacherSegMap .
We have
[Equation]
Where τ refers to distillation temperature.
The model is trained using the sum of 2D-to-3D distillation loss L Dist and 3D cross-entropy loss L CE .
[Figure]
[Equation]
[Figure]
[Equation] [Figure] [Equation] To facilitate the knowledge distillation from the advanced 2D segmentation models to 3D models, we need a large-scale dataset of 3D scans with their corresponding panorama images.
Existing datasets such as ScanNet (Dai et al., 2017) and Matterport3D (Chang et al., 2017) usually require intensive annotation effort, contain noisy labels, and do not cover complete scene geometry, while
SUNCG (Song et al., 2017b) does have clean annotation but is limited to narrow field-of-view RGBD images.
To this end, we propose a novel large TSDF-based semantic segmentation dataset called
PanoRooms3D.
To achieve a human-like understanding of the 3D environment, we use professional designed room layouts from 3D-FRONT (Fu et al., 2021a), furnished by 3D-FUTURE(Fu et al., 2021b), a largescale 3D CAD shapes with high-resolution informative textures.
To diversify the generation, we augment each scene with randomly sampled floor/wall textures.
To obtain the TSDF representation together with semantic labels, we randomly sample camera poses inside the rooms with a height between 1.4m and 1.8m. This process simulates users holding RGBD cameras to capture indoor scenes like (Chang et al., 2017;
Dai et al., 2017; Hua et al., 2016;
Armeni et al., 2017).
RGB-D plus semantic frames are rendered at all camera poses and later fused into 3D scenes and semantic ground truth using TSDF fusion.
A total of 2,509,873 RGB-D images and corresponding pixel-wise semantic segmentation maps are rendered.
We set each voxel a 4 cm cube.
We use 120 NVIDIA Tesla V100 GPU to render for 48 hours.
3D-FRONT includes 99 semantic categories.
In practice, considering the semantical consistency between labels, we consolidate similar labels, merging the original 99 classes into 30 hyper classes.
Since many scenes contain furniture that is left unlabeled, we manually filter out noisy rooms.
5917 TSDF-based living rooms and bedrooms are created.
We split 5325 rooms for training, 297 for validation, and 295 for [Figure] testing.
For the convenience of window partition, we pad the all spatial dimensions of scenes to the multiples of 128 .
To get panoramic images for the training of 2D teachers, we place a camera at the center of each room.
The camera renders 6 square images from + X , − X , + Y , − Y , + Z , − Z directions with a field of view of 90 ◦ , which constitute a skybox.
The skybox is then converted to an equirectangular panorama through pixel mapping and bilinear interpolation.
A sample of PanoRooms3D is visualized in Figure 3.
E XPERIMENTS 6. I MPLEMENTATION D ETAIL
In our training of 3D Segmenters on our PanoRooms3D dataset, we slice the 5325 rooms for training into 128 × 128 × 128 cubes as the training set.
The patch size P is set to 4 , resulting 1284 × 1284 × 1284 = 32768 tokens at the beginning.
The number of Swin Transformer blocks h is set to 2 , with h − 1 =patch merging layer in between.
The decoder receives N =
h − 1 ) 3 = 4096 tokens from encoder.
The number of class K is 30 .
We change the number of layers and number of MSA heads within each Swin Transformer block and embedding dimension D to create different sizes of variants.
Variants of encoder architecture include:
• 3DSeg-T(iny), D = 48 , Swin Layers [2 , 2] , Num Heads [3 , 3] , #PARAM 0.38M • 3DSeg-B(ase), D = 48 , Swin Layers [2 , 4] , Num Heads [3 , 3] , #PARAM 0.62M • 3DSeg-M(iddle),
D = 64 , Swin Layers [3 , 6] , Num Heads [4 , 4] , #PARAM 1.55M • 3DSeg-L(arge), D = 96 , Swin Layers [4 , 8] , Num Heads [6 , 6] , #PARAM 4.43M
We train our 3D Segmenter on NVIDIA Ampere A100 80GB, using the Adam optimizer (Kingma & Ba, 2014) with a learning rate of 0.001 and batch size of 8.
It takes 48 hours for our models to converge.
For 2D to 3D knowledge distillation, we operate the experiments taking room as a unit.
we render the 3D rooms for train/val/test sets to get 2D panoramas and 2D ground truth segmentation for train/val/test sets respectively.
The 2D training set is used to finetune the 2D teacher model.
The size of rendered panorama is set to ( H, W ) = (512 , 1024) .
We manually filter out noisy renderings in 5323 training rooms caused by objects that block the view.
We use the 5175 panoramas after filtration paired up with their original 3D rooms to do distillation.
We choose 2D Segmenter(Strudel et al., 2021)
Seg-B-Mask/16 and Seg-L-Mask/16 as the 2D teacher models.
Pretrained on ImageNet-21K(Steiner et al., 2021), the models are finetuned on our rendered 2D panoramas training set for 200 epochs.
Because the scale of rooms varies, we train/validate the distillation with a batch size of 1 .
We train the distillation for 45000 iterations and validate every 1500 iterations.
The distillation temperature τ is set to 20 in all of our experiments.
Apart from our PanoRoom3D dataset, we also include the Scannet (Dai et al., 2017) for baseline comparison.
We random split the 1513 rooms into train/val/test 1361/77/75 respectively.
All rooms are fused with a voxel resolution of 4 cm .All
spatial dimensions of scenes are padded to the multiples of 128.
For training, the rooms are sliced into 128 × 128 × 128 cubes.
6. B ASELINE C OMPARISON
[Table]
[Table]
Sparseconv
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table] [Table]
64M
VT
[Table] [Table]
20.89M 3DSeg
[Table]
0.38M
[Table]
0.38M 3DSeg
[Table]
0.62M 3DSeg
[Table]
Dist 98.
0.62M 3DSeg
[Table]
1.55M
[Table] [Table]
1.55M
[Table] [Table]
[Table]
[Table] [Table]
[Table] [Table] [Table] [Table]
16.32M
[Table]
[Table]
[Table]
[Table]
All of the quantitative metrics are run on a room-level.
Metrics evaluated from different rooms are weighted with the corresponding rooms’ volume.
Here we compare our four variants and four distilled models with Fully Convolutional Network 3D U-Net(C¸ic¸ek et al., 2016), sparse convolution(Graham et al., 2018a) based model and recent transformer-based model VT-UNet(Peiris et al., 2022) .
We Report 7 metrics used in semantic segmentation: overall accuracy(aAcc), mean intersection over union(mIoU), mean accuracy(Acc), Dice, Fscore, Precision, Recall.
Among these metrics, mIoU is the most representative one.
3DSeg-T Dist distills knowledge from 2D Segmenter (Strudel et al., 2021)
Seg-B-Mask/16.
3DSeg-B Dist, 3DSeg-M Dist and 3D Seg-L Dist distill knowledge from Seg-L-Mask/16.
See the quantitative comparison in table 1.
3DSeg-B already outperforms 3D
U-Net even with only 3 . 8% parameters.
The distillation schemes achieve a higher mIoU score without adding extra parameters.
This implies that our 2D-to-3D panoramic distillation demonstrates the feasibility of training powerful 3D networks with 2D knowledge.
Our baseline comparison on
Scannet (Dai et al., 2017) is shown in table 2.
The result shows that our proposed 3D Segmenter also achieves the best performance on real world data.
We also conduct a complexity comparison of our four baseline variants and 3D U-Net.
We randomly choose 20 scenes to calculate the total inference time.
Since some of the scenes are too large to be run on GPU for 3D U-Net, we run our inference time comparison on CPU.
The total inference time of our four variants is 43 .
62 s , 44 .
79 s , 61 .
08 s , 95 .
s from Tiny to Large, and 254 .
75 s for 3D
U-Net.
6. A BLATION S TUDY
The effectiveness of 2D-to-3D distillation.
Our 3D Segmenter is pretrained on fixed-size blocks and then distilled at room level.
To control variables and validate the effectiveness of our proposed distillation strategy, we conduct two experiments by finetuning 3DSeg-T and 3DSeg-B at room level with 3D cross-entropy loss only.
[Table]
We can see from Table 3 that compared with directly finetune the model on rooms, our panoramic distillation design can steadily improve the mIoU performance.
The necessity of skip connection.
While introducing the decoder of 3D Segmenter, we use a skip connection to ’remind’ our model of the input x .
We validate the necessity of the design in this experiment.
[Table]
We can see in Table 4 by adding a skip connection before output significantly improves the overall performance by adding a small number of parameters.
D ISCUSSIONS
In this work, we propose the first 2D-to-3D knowledge distillation method to utilize the data abundant and pretrain-ready 2D semantic segmentation to improve 3D semantic segmentation.
Experiments demonstrate our technique significantly improves the 3D model over the baseline.
We have also introduced a PanoRooms3D dataset that contains a large variety of indoor scenes with dense 3D volumes and their corresponding panorama renderings.
A CKNOWLEDGMENTS
This work is supported by JST Moonshot R&D Grant Number JPMJMS2011 and JST ACT-X Grant Number JPMJAX190D, Japan and partially supported by the Shanghai Committee of Science and Technology (Grant No. 21DZ1100100) A A PPENDIX
Limitations.
However, a few limitations are yet to be addressed.
First, though the panorama images could provide the global context of the environment, the single-camera rendering inevitably suffers from occlusion, a promising direction is to directly leverage the full 2D textures from the surface of the 3D scans, as done in TextureNet (Huang et al., 2019).
Second, currently, we do not consider the similarity between different models, we leave this as future work.
[Figure]