Original: Look Around and Refer: 2D Synthetic Semantics Knowledge Distillation for 3D Visual Grounding
Revised: Look Around and Refer: 2D Synthetic Semantics Knowledge Distillation for 3D Visual Grounding

Original: Abstract
Revised: Abstract

Original: The 3D visual grounding task has been explored with visual and language streamsto comprehend referential language to identify targeted objects in 3D scenes.
Revised: The 3D visual grounding task has been explored with visual and language streamsto comprehend referential language to identify targeted objects in 3D scenes.

Original: However, most existing methods devote the visual stream to capturing the 3D visualclues using off-the-shelf point clouds encoders.
Revised: However, most existing methods devote the visual stream to capturing the 3D visualclues using off-the-shelf point clouds encoders.

Original: The main question we address inthis paper is “can we consolidate the 3D visual stream by 2D clues synthesizedfrom point clouds and efficiently utilize them in training and testing?” .
Revised: The main question we address inthis paper is “can we consolidate the 3D visual stream by 2D clues synthesizedfrom point clouds and efficiently utilize them in training and testing?” .

Original: The mainidea is to assist the 3D encoder by incorporating rich 2D object representationswithout requiring extra 2D inputs.
Revised: The mainidea is to assist the 3D encoder by incorporating rich 2D object representationswithout requiring extra 2D inputs.

Original: To this end, we leverage 2D clues, syntheticallygenerated from 3D point clouds, and empirically show their aptitude to boost thequality of the learned visual representations.
Revised: To this end, we leverage 2D clues, syntheticallygenerated from 3D point clouds, and empirically show their aptitude to boost thequality of the learned visual representations.

Original: We validate our approach throughcomprehensive experiments on Nr3D, Sr3D, and ScanRefer datasets, and we showconsistent performance gains compared to the state-of-the-art.
Revised: We validate our approach throughcomprehensive experiments on Nr3D, Sr3D, and ScanRefer datasets, and we showconsistent performance gains compared to the state-of-the-art.

Original: Our proposedmodule, dubbed as Look Around and Refer (LAR), significantly outperforms thestate-of-the-art 3D visual grounding techniques on these three benchmarks.
Revised: Our proposedmodule, dubbed as Look Around and Refer (LAR), significantly outperforms thestate-of-the-art 3D visual grounding techniques on these three benchmarks.

Original: Ourcode will be made publicly available.
Revised: Ourcode will be made publicly available.

Original:  Introduction
Revised:  Introduction

Original: Visiolinguistic modeling has attracted much attentionover the past decade due to impact on various areas,including visual question answering [3, 36], 3D scenecaptioning [7], image-text matching [26, 43], visualrelation detection [25] and scenegraph generation[41].
Revised: Visiolinguistic modeling has attracted much attentionover the past decade due to impact on various areas,including visual question answering [3, 36], 3D scenecaptioning [7], image-text matching [26, 44], visualrelation detection [25] and scenegraph generation[42].

Original: Consequently, understanding natural languageutterances and grounding them to real-world scenarios is attractive due to their importance in many realapplications, such as robot navigation and interactionin indoor environment [22] [39].
Revised: Consequently, understanding natural languageutterances and grounding them to real-world scenarios is attractive due to their importance in many realapplications, such as robot navigation and interactionin indoor environment [22] [40].

Original: Visual Grounding (VG) has been explored in recentyears in the context of 2D [16, 30, 45, 54, 56], extending object detection to a more complex task.
Revised: Visual Grounding (VG) has been explored in recentyears in the context of 2D [16, 30, 46, 55, 57], extending object detection to a more complex task.

Original: However, for real robots to intelligently ground a givenutterance on a natural scene in a challenging environment like a crowded indoor or outdoor scene, 3D representations are needed to understand the surroundingenvironment better.
Revised: However, for real robots to intelligently ground a givenutterance on a natural scene in a challenging environment like a crowded indoor or outdoor scene, 3D representations are needed to understand the surroundingenvironment better.

Original: Therefore, 3D visual groundingtask have been proposed by ReferIt3D [2] and ScanRefer [5].
Revised: Therefore, 3D visual groundingtask have been proposed by ReferIt3D [2] and ScanRefer [5].

Original: The ReferIt3D [2] introduces two datasets
Revised: The ReferIt3D [2] introduces two datasets

Original: [Table]
Revised: [Table]

Original: containing natural and synthetic language utterances, namely Nr3D and Sr3D, respectively, whichaims to determine the referent object from pre-defined objects set for each scene.
Revised: containing natural and synthetic language utterances, namely Nr3D and Sr3D, respectively, whichaims to determine the referent object from pre-defined objects set for each scene.

Original: In contrast, ScanRefer [5] aims to predict 3D bounding boxes for the referent object based on a given languagedescription and 3D point clouds.
Revised: In contrast, ScanRefer [5] aims to predict 3D bounding boxes for the referent object based on a given languagedescription and 3D point clouds.

Original: Both ScanRefer [5] and ReferIt3D [2] are built on the ScanNetdataset [8].
Revised: Both ScanRefer [5] and ReferIt3D [2] are built on the ScanNetdataset [8].

Original: The critical difference, which made ReferIt3D [2] more challenging, is that in each scene,several object instances belong to the same fine-grained category as the referent, namely distractors.
Revised: The critical difference, which made ReferIt3D [2] more challenging, is that in each scene,several object instances belong to the same fine-grained category as the referent, namely distractors.

Original: Due to the sparsity and disorganization of the 3D point clouds, SAT [51] explores utilizing the 2Dimages provided by the ScanNet dataset [8] during training to assist point-cloud-language jointrepresentation learning.
Revised: Due to the sparsity and disorganization of the 3D point clouds, SAT [52] explores utilizing the 2Dimages provided by the ScanNet dataset [8] during training to assist point-cloud-language jointrepresentation learning.

Original: SAT [51] proposes three variants; 1) The baseline, non-SAT, relies only onthe 3D point clouds while tackling the grounding task.
Revised: SAT [52] proposes three variants; 1) The baseline, non-SAT, relies only onthe 3D point clouds while tackling the grounding task.

Original: Utilizing 2D semantics during the trainingphase only and masking them while inference, we termed it SAT.
Revised: Utilizing 2D semantics during the trainingphase only and masking them while inference, we termed it SAT.

Original: 3) Utilizing the 2D semantics inboth phases, i.e., training and testing, we termed it SAT-GT.
Revised: 3) Utilizing the 2D semantics inboth phases, i.e., training and testing, we termed it SAT-GT.

Original: However, requiring extra 2D inputs intraining or inference limits potential application scenarios.
Revised: However, requiring extra 2D inputs intraining or inference limits potential application scenarios.

Original: To tackle this drawback, we hypothesizedthat incorporating synthetic 2D semantics will consolidate the 3D learned representation while notrequiring any additional info.
Revised: To tackle this drawback, we hypothesizedthat incorporating synthetic 2D semantics will consolidate the 3D learned representation while notrequiring any additional info.

Original: To this end, we propose a novel 2D Synthetic Images Generator (SIG)that projects the 3D point clouds into multi-view 2D images, as shown in Figure 1.
Revised: To this end, we propose a novel 2D Synthetic Images Generator (SIG)that projects the 3D point clouds into multi-view 2D images, as shown in Figure 1.

Original: To the bestof our knowledge, LAR is the first method that augment learning of 3D tasks with synthetic 2Dsemantics.
Revised: To the bestof our knowledge, LAR is the first method that augment learning of 3D tasks with synthetic 2Dsemantics.

Original: In addition, we propose a novel multi-modal transformer-based architecture.
Revised: In addition, we propose a novel multi-modal transformer-based architecture.

Original: Our proposedarchitecture, LAR, on Nr3D outperforms SAT [51] by boosting the accuracy by 1.6 % while exceedingall existing approaches that did not use any additional information than the 3D point clouds by awider margin, i.e., 5.0 % to the closest one.
Revised: Our proposedarchitecture, LAR, on Nr3D outperforms SAT [52] by boosting the accuracy by 1.6 % while exceedingall existing approaches that did not use any additional information than the 3D point clouds by awider margin, i.e., 5.0 % to the closest one.

Original: Also, LAR architecture achieves state-of-the-art resultson Sr3D and ScanRefer benchmarks, 59.35% and 54.6%, respectively.
Revised: Also, LAR architecture achieves state-of-the-art resultson Sr3D and ScanRefer benchmarks, 59.35% and 54.6%, respectively.

Original: Another variant is tailoredfrom our architecture to be compared against SAT-GT, where geometry information and ground-truthclass labels are concatenated to the 2D images; synthetic in our case and original images in SAT case.
Revised: Another variant is tailoredfrom our architecture to be compared against SAT-GT, where geometry information and ground-truthclass labels are concatenated to the 2D images; synthetic in our case and original images in SAT case.

Original: Our LAR architecture outperforms SAT-GT by 11.7 % , where we achieved 62% while SAT achieves50.3%.
Revised: Our LAR architecture outperforms SAT-GT by 11.7 % , where we achieved 62% while SAT achieves50.3%.

Original: Our contributions are summarized as follows:
Revised: Our contributions are summarized as follows:

Original: • We propose a novel End-to-End multi-modal transformer-based architecture, LAR, for the3D visual grounding task that relies only on 3D point clouds, which exploits the synthetic2D images generated by our SIG module.• We verify the effectiveness of our LAR architecture throughout extensive experiments with
Revised: • We propose a novel End-to-End multi-modal transformer-based architecture, LAR, for the3D visual grounding task that relies only on 3D point clouds, which exploits the synthetic2D images generated by our SIG module.• We verify the effectiveness of our LAR architecture throughout extensive experiments with

Original: various benchmarks, which show that utilizing our 2D synthetic semantics significantly
Revised: various benchmarks, which show that utilizing our 2D synthetic semantics significantly

Original: enhances the results.
Revised: enhances the results.

Original: • Through detailed analysis, we examine our method’s internal behavior and validity.
Revised: • Through detailed analysis, we examine our method’s internal behavior and validity.

Original:  Related Work
Revised:  Related Work

Original: Our novel architecture draws success from several areas, including 2D visual grounding, 3D visualgrounding, and 2D semantics in 3D tasks.
Revised: Our novel architecture draws success from several areas, including 2D visual grounding, 3D visualgrounding, and 2D semantics in 3D tasks.

Original: – 2D
Revised: – 2D

Original: visual grounding.
Revised: visual grounding.

Original: 2D visual grounding is locating an object based on a natural languagedescription.
Revised: 2D visual grounding is locating an object based on a natural languagedescription.

Original: Flickr30k [32] and ReferItGame [20] generated referring expressions for a particularobject in an image used to identify which object is being referred to.
Revised: Flickr30k [32] and ReferItGame [20] generated referring expressions for a particularobject in an image used to identify which object is being referred to.

Original: Prior works [55, 27, 29]have attempted to localize these referred objects based on the given expressions by focusing on thebounding-box level.
Revised: Prior works [56, 27, 29]have attempted to localize these referred objects based on the given expressions by focusing on thebounding-box level.

Original: In contrast, [52, 10] focus on the pixels level.
Revised: In contrast, [53, 10] focus on the pixels level.

Original: Another 2D visual groundingtaxonomy is the one-stage, also known as a single-stage framework, versus the two-stage framework.
Revised: Another 2D visual groundingtaxonomy is the one-stage, also known as a single-stage framework, versus the two-stage framework.

Original: The single-stage methods [38, 48, 49], fuse the linguistic query with each pixel/patches of theimage, and generate a set of possible bounding box candidates.
Revised: The single-stage methods [38, 49, 50], fuse the linguistic query with each pixel/patches of theimage, and generate a set of possible bounding box candidates.

Original: In contrast, the two-stage framework[44, 53, 55] uses the visual image semantics to generate object proposals.
Revised: In contrast, the two-stage framework[45, 54, 56] uses the visual image semantics to generate object proposals.

Original: Afterward, the grounding prediction is selected by comparing each proposal to the language query.
Revised: Afterward, the grounding prediction is selected by comparing each proposal to the language query.

Original: Consequently, 2D visualgrounding methods can guide learning 3D relationships.
Revised: Consequently, 2D visualgrounding methods can guide learning 3D relationships.

Original: Driven by this, we explore using 2D syntheticsemantics to consolidate the 3D visual stream.
Revised: Driven by this, we explore using 2D syntheticsemantics to consolidate the 3D visual stream.

Original: – 3D visual grounding.
Revised: – 3D visual grounding.

Original: Table 1 summarizes the existing 3D visual grounding models whether aground truth object proposal is used or a pre-trained detector is utilized to generate the proposals.
Revised: Table 1 summarizes the existing 3D visual grounding models whether aground truth object proposal is used or a pre-trained detector is utilized to generate the proposals.

Original: Also, it summarizes whether extra info is used alongside the object’s 3D point cloud or not.
Revised: Also, it summarizes whether extra info is used alongside the object’s 3D point cloud or not.

Original: 3D visual grounding field has undergone extensive research and made considerable progress over the last few years [2, 5, 12, 17, 58, 1, 37, 13, 57, 19, 51, 6, 19].
Revised: 3Dvisual grounding field has undergone extensive research and made considerable progress over thelast few years [2, 5, 12, 17, 59, 1, 37, 13, 58, 19, 52, 6, 19].

Original: A 3D visual grounding task useslanguage instruction along with a 3D scene [8] as input in order to estimate a bounding box thatallocates the most relevant object to the provided expression.
Revised: A 3D visual grounding task useslanguage instruction along with a 3D scene [8] as input in order to estimate a bounding box thatallocates the most relevant object to the provided expression.

Original: However, the literature offers severalextensions, such as using auxiliary input to improve localizing 3D objects [51].
Revised: However, the literature offers severalextensions, such as using auxiliary input to improve localizing 3D objects [52].

Original: Previous work[2, 12, 17, 58, 1, 13, 57, 19], focuses on improving the language and 3D scene fusion by adopting a two-stage framework.
Revised: Previous work[2, 12, 17, 59, 1, 13, 58, 19], focuses on improving the language and 3D scene fusion by adopting atwo-stage framework.

Original: The first stage outputs 3D object proposals given a 3D point cloud representing a scene.
Revised: The first stage outputs 3D object proposals given a 3D point cloud representinga scene.

Original: Secondly, visual and linguistic features are fused to compute a confidence score for each 3D object proposal.
Revised: Secondly, visual and linguistic features are fused to compute a confidence score for each3D object proposal.

Original: The first stage could be skipped by directly using the ground-truth proposalsinstead of predicting them [51] [13] [2].
Revised: The first stage could be skipped by directly using the ground-truth proposalsinstead of predicting them [52] [13] [2].

Original: In contrast, [6, 19, 58] do not assume the access to theground-truth objects and use an object detector to extract the proposals.
Revised: In contrast, [6, 19, 59] do not assume the access to theground-truth objects and use an object detector to extract the proposals.

Original: Intuitively, if the detectorfails, the referring performance will significantly decrease.
Revised: Intuitively, if the detectorfails, the referring performance will significantly decrease.

Original: Therefore, we focus on the main trainingobjective related to the grounding task, which is learning 3D visual-language joint representations byassuming access to the ground-truth boxes.
Revised: Therefore, we focus on the main trainingobjective related to the grounding task, which is learning 3D visual-language joint representations byassuming access to the ground-truth boxes.

Original: Although we are assuming access to the ground-truthboxes, LAR is compatible with the setting that utilizes detector-generated proposals.
Revised: Although we are assuming access to the ground-truthboxes, LAR is compatible with the setting that utilizes detector-generated proposals.

Original: – 2D semantics in 3D tasks.
Revised: – 2D semantics in 3D tasks.

Original: 2D semantics contains useful information that can be used to detect3D objects.
Revised: 2D semantics contains useful information that can be used to detect 3D objects.

Original: The representations of visual scenes consist of point clouds, meshes, or volumes,while 2D images consist of pixel grids
Revised: The representations of visual scenes consist of point clouds, meshes, or volumes,while 2D images consist of pixel grids.

Original: Combining the best of two worlds can enhance 3D objectlocalization.
Revised: Combining the best of two worlds can enhance 3D object localization.

Original: Researchers have attempted to solve the 3D detection problem by adopting differentways of representing RGB-D data [34, 47, 23, 15, 40, 24].
Revised: Researchers have attempted to solve the 3D detection problem by adopting different ways of representing RGB-D data [34, 48, 23, 15, 41, 24].

Original: By either projecting the 2D image onthe 3D scene [34, 47, 24] or fuse both 2D features with the 3D [23, 15, 42, 40], [33].
Revised: By either projecting the 2D image on the 3D scene [34, 48, 24] or fuse both 2D features with the 3D [23, 15, 43, 41], [33].

Original: SAT [51]leveraged 2D semantics to perform the 3D VG task.
Revised: SAT [52] leveraged 2D semantics to perform the 3D VG task.

Original: However, requiring extra 2D inputs limitspotential application scenarios.
Revised: However, requiring extra 2D inputs limits potential application scenarios.

Original: In contrast, in this work, we generate synthetic 2D images duringthe training phase, which is also used in the inference phase.
Revised: In contrast, in this work, we generate synthetic 2D images during the training phase, which is also used in the inference phase.

Original: The synthetic images enable us toconsolidate the 3D visual stream by 2D semantics without requiring extra 2D inputs.
Revised: The synthetic images enable us to consolidate the 3D visual stream by 2D semantics without requiring extra 2D inputs.

Original:  Look Around and Refer (LAR)
Revised:  Look Around and Refer (LAR)

Original: LAR aims to enhance the visual module, which in turn enhances the final objective,i.e., referentialaccuracy.
Revised: LAR aims to enhance the visual module, which in turn enhances the final objective,i.e., referentialaccuracy.

Original: By incorporating the synthetic 2D semantics alongside the 3D point clouds, LAR inherentlycaptures the correlations between the 2D and the 3D streams.
Revised: By incorporating the synthetic 2D semantics alongside the 3D point clouds, LAR inherentlycaptures the correlations between the 2D and the 3D streams.

Original: To design a general frameworkto fit in any 3D application, when the input is 3D point clouds, the need for extra information,such as 2D semantics, should be bypassed.
Revised: To design a general frameworkto fit in any 3D application, when the input is 3D point clouds, the need for extra information,such as 2D semantics, should be bypassed.

Original: To this end, a 2D synthetic images generator, namely
Revised: To this end, a 2D synthetic images generator, namely

Original: SIG, is introduced, which is discussed in Section 3.1.
Revised: SIG, is introduced, which is discussed in Section 3.1.

Original: Then, we demonstrate how to learn robustrepresentations by aligning the generated 2D semantics with the 3D ones by incorporating our SIGmodule into our proposed architecture, depicted in Section 3.2.
Revised: Then, we demonstrate how to learn robustrepresentations by aligning the generated 2D semantics with the 3D ones by incorporating our SIGmodule into our proposed architecture, depicted in Section 3.2.

Original: 3.1 Look Around: 2D Synthetic Images Generator
Revised: 3.1 Look Around: 2D Synthetic Images Generator

Original: LAR exploits synthetic 2D semantics to reinforce the learned 3D representations.
Revised: LAR exploits synthetic 2D semantics to reinforce the learned 3D representations.

Original: Consequently, thissection will demonstrate our 2D Synthetic Images Generator (SIG) module that projects arbitrary 3D
Revised: Consequently, thissection will demonstrate our 2D Synthetic Images Generator (SIG) module that projects arbitrary 3D

Original: Point clouds representing an object to a 2D image.
Revised: Point clouds representing an object to a 2D image.

Original: First, we will show the essential operations of ourmodule on a single image, then show how we extend it to fulfill a multi-view setup.
Revised: First, we will show the essential operations of ourmodule on a single image, then show how we extend it to fulfill a multi-view setup.

Original: A 3D scene, S ∈ R N × 6 , is represented by N points with spatial and color information, i.e., XYZ,and RGB, respectively.
Revised: A 3D scene, S ∈ R N × 6 , is represented by N points with spatial and color information, i.e., XYZ,and RGB, respectively.

Original: Following the previous works [2] [1] [37] [51], the object proposals x pci foreach scene are known, either generated by one of the off-the-shelf 3D object detectors or manuallyannotated.
Revised: Following the previous works [2] [1] [37] [52], the object proposals x pci foreach scene are known, either generated by one of the off-the-shelf 3D object detectors or manuallyannotated.

Original: The object proposal x pci ∈ R N ′ × 6 is a subset of the whole scene, where N ′ < N and N ′represents the number of points that represent the object i .
Revised: The object proposal x pci ∈ R N ′ × 6 is a subset of the whole scene, where N ′ < N and N ′represents the number of points that represent the object i .

Original: Given a scene S and an object proposal x pci , our SIG module generates a 2D synthetic image x imgirepresents the object i .
Revised: Given a scene S and an object proposal x pci , our SIG module generates a 2D synthetic image x imgirepresents the object i .

Original: The main idea is to place a virtualpinhole camera in front of the fed object proposal, then project its points x pci into the 2D plane to generate the corresponding 2D image x imgi .
Revised: The main idea is to place a virtual camera in front of the fed object proposal,then project its points x pci into the 2D plane to generate the corresponding 2D image x imgi .

Original: The camera’s location should be set carefully to generate a decent image representation for an objecti .
Revised: Thecamera’s location should be set carefully to generate a decent image representation for an object i .

Original: As shown in part (A) in Figure 2, after excluding the upper and lower faces from the fed objectproposal, we get the center of the four remaining faces, then calculate the distance between each facecenter O li , and the center of the scene O s , where l ranging from 0 to 3.
Revised: Asshown in part (A) in Figure 2, after excluding the upper and lower faces from the fed object proposal,we get the center of the four remaining faces, then calculate the distance between each face center O li ,and the center of the scene O s , where l ranging from 0 to 3.

Original: The nearest face l ′ from thescene center O s is the prominent one, colored with blue in Figure 2, which guarantees that we look atthe object i from inside the room.
Revised: The nearest face l ′ from the scene center O s is the prominent one, colored with blue in Figure 2, which guarantees that we look at the objecti from inside the room.

Original: After selecting the prominent face l ′ , the camera is located at adistance d f from that face  and at a distance d up from the room’s floor; thus, the camera position isinterpreted as Cam p =
Revised: The camera is located at a distance d f from the prominent face l ′ and at adistance d up from the room’s floor; thus, the camera position is interpreted as Cam p =

Original: [ d fx , d fy , d up ] , as shown in part (B) in Figure 2.
Revised: [ d fx , d fy , d up ] ,as shown in part (B) in Figure 2.

Revised: The orientation of the camera frame is defined as follows.

Revised: The z axisis aligned with the principal axis of the camera, and two other orthogonal directions (x, y) align withthe corresponding axes of the image plane.

Original: Failure cases are shown in Figure 5, when the camera is located outside the room.
Revised: Failure cases are shown in Figure 5, when the camera islocated outside the room.

Original: To project the points x pci , representing object i , from the scene coordinates to the camera coordinates,a transformation matrix M ∈ R 4 × 4 needs to be defined.
Revised: To project the points x pci , representing object i , from the scene coordinates to the camera coordinates,a transformation matrix M ∈ R 4 × 4 needs to be defined.

Original: Given the camera position Cam p and the center of the prominent face O l ′ , the transformation matrix M can be interpreted as, 
Revised: Given the camera position Cam p and the center of the prominent face O l ′ , the transformation matrix M can be interpreted as, (cid:20) V r V up V

Original: [Figure]   (cid:21)
Revised: f Cam p   (cid:21)

Original: Where, the camera direction is defined by three vectors;forward vector V f∈ R 3 × 1 ,right vector V r∈ R 3 × 1 , andup vector V up∈ R 3 × 1 . 
Revised: Where, the camera direction is defined by three vectors; V f , V r , and V up . We define our virtualcamera using the Unified Camera Model (UCM), which has five parameters I =

Original: To direct the camera to the center of the prominent face
Revised: [ γ x , γ y , c x , c y , ζ ]

Original: O l ′ , the forward vector V f is determined by ∥ Cam p − O l ′ ∥ , where ∥ x ∥ means the normal of thevector x .
Revised: Following the UCM, projection is defined as follows: π ( x pci , I ) =

Original: While, the right vector V r is set to ∥ V f ⊗ [0 , 0 , 1] ∥ , where ⊗ refer to cross product.
Revised: (cid:20) γ [Equation] x

Original: As theforward vector V f and the right vector V r are orthogonal, thus the cross product between these twovectors will result in the camera up direction V up =
Revised: [Equation] + [Equation]

Original: ∥
Revised: γ

Original: V r ⊗ V f ∥ .
Revised: [Equation] +

Original: Consequently, projecting the 3D point clouds to a 2D image can be interpreted as three operations;converting points from scene coordinates to camera coordinates, then the mapping from cameracoordinates to image coordinates, and finally the transformation from image coordinates to pixelcoordinates.
Revised: [Equation] (cid:21) + (cid:20) c [Equation]

Original: Thus, by performing a dot product between the 3D point clouds and the transformationmatrix M , transforming from scene coordinates x pci to camera coordinates x cami is achieved.
Revised: c [Equation]

Original: Then,followed by another dot product between the 3D points in the camera coordinates x cami and theintrinsic pinhole camera parameters I ,the points are projected into image space x img i . Finally, tofurther transform points from image space to the pixel space, we divide the 2D spatial coordinates; x,y, for each point by its height information; z, to get homogeneous coordinates in the image plane.
Revised: (cid:21) ,

Original: Afterwards, the projected 3D point clouds are assigned to a 2D grid map.
Revised: Afterwards, the projected 3D point clouds are assigned to a 2D grid map.

Original: To mitigate the overfittingeffect, we randomly extend the region of interest by ϵ as demonstrated in part (C) at Figure 2.
Revised: To mitigate the overfittingeffect, we randomly extend the region of interest by ϵ as demonstrated in part (C) at Figure 2.

Original: Point cloud differs from 2D images, where every pixel contains spatial and texture details.
Revised: Point cloud differs from 2D images, where every pixel contains spatial and texture details.

Original: Moreover,the object is represented in a view-invariant way, unlike the images that are view-dependent.
Revised: Moreover,the object is represented in a view-invariant way, unlike the images that are view-dependent.

Original: Tomitigate this drawback in the image representation, a multi-view setup is constructed as shown in
Revised: To mitigate this drawback in the image representation, a multi-view setup is constructed as shown in

Original: Figure 1.
Revised: Figure 1.

Original: Where, v cameras are mounted in a circle around the prominent face l ′ .
Revised: Where, v cameras are mounted in a circle around the prominent face l ′ .

Original: Algorithm 1,attached in the supplemental materials, demonstrates a pseudo-code for our SIG module.
Revised: Algorithm 1,attached in the supplemental materials, demonstrates a pseudo-code for our SIG module.

Original: 3.2 Refer: Visiolinguistic Transformer Architecture
Revised: 3.2 Refer: Visiolinguistic Transformer Architecture

Original: We present our approach, depicted in Figure 3, for learning multi-modal representations of a 3Dscene associated with language instruction.
Revised: We present our approach, depicted in Figure 3, for learning multi-modal representations of a 3Dscene associated with language instruction.

Original: First, the proposed visual module enriches the 3Drepresentations by utilizing the 2D synthetic images generator, namely the SIG module, described in
Revised: First, the proposed visual module enriches the 3Drepresentations by utilizing the 2D synthetic images generator, namely the SIG module, described in

Original: Section 3.1.
Revised: Section 3.1.

Original: Thanks to it, we do not require access to any imagery sensor during training or testing.
Revised: Thanks to it, we do not require access to any imagery sensor during training or testing.

Original: Then, the multi-modal module learns joint representations targeted to the final task, i.e., referring toan object in the scene based on the language instruction.
Revised: Then, the multi-modal module learns joint representations targeted to the final task, i.e., referring toan object in the scene based on the language instruction.

Original: – Visual Module.
Revised: – Visual Module.

Revised: Given a

Revised: 3D

Revised: scene

Revised: S

Revised: ∈ R N

Original: Given a 3D scene S ∈ R N × 6 combined with agnostic multiple 3D object proposalsx pci ∈ R N ′ × 6 , we aim to classify eachobject proposal by predicting the semantic labels z .
Revised: × 6 combined with agnostic multiple 3D object proposalsx pci ∈ R N ′ × 6 , we aim to classify each

Revised: object

Revised: proposal by predicting

Revised: the

Revised: semantic

Revised: labels

Original:  First,for each object i in the scene S , we generate multi-view synthetic 2D images x imgi ∈ R v × H ×W × 3from the 3D object’s point cloud x pci , where v is the number of views for each object, H and Ware the spatial dimensions for the generated images.
Revised: z . First,for each object i in the scene S , we generate multi view synthetic 2D images x imgi ∈ R v × H ×

Revised: W

Revised: × 3from the

Revised: 3D object’s

Revised: point

Revised: cloud x pci

Original:  Second, we employ PointNet++ [35] as a 3Dencoder, producing 3D features E pci ∈ R 1 × dim pc .
Revised: , where v is the number of views for each object, H and Ware the spatial dimensions for the generated images. Second, we employ PointNet++ [35] as a 3Dencoder, producing 3D features E pc i ∈ R 1 × dim pc .

Original: Hence, the Tiny-ConvNext [28] is adopted as a2D encoder, that produce y imgi ∈ R v × H ′ × W ′ × dim img .
Revised: Hence, the Tiny-ConvNext [28] is adopted as a2D encoder, that produce y imgi ∈ R v × H ′ × W ′ × dim img .

Original: Where H ′ and W ′ represent a coarse spatialgrid of spatial features, each represented by a dim img .
Revised: Where H ′ and W ′ represent a coarse spatialgrid of spatial features, each represented by a dim img .

Original: Then, the spatial dimensions are squeezedproducing o imgi ∈ R v × dim img .
Revised: Then, the spatial dimensions are squeezedproducing o imgi ∈ R v × dim img .

Original: Since our 2D encoder encodes multiple views v for the same objecti , a fusing mechanism is needed to fuse the views’ features.
Revised: Since our 2D encoder encodes multiple views v for the same objecti , a fusing mechanism is needed to fuse the views’ features.

Original: Various possibilities are discussed tosettle down on the best fusion mechanism.
Revised: Various possibilities are discussed tosettle down on the best fusion mechanism.

Original: We can interpret this as a fully connected layer to fullycapture the views and cross-channel interactions.
Revised: We can interpret this as a fully connected layer to fullycapture the views and cross-channel interactions.

Original: In contrast, to learn the views’ interactions andneglect the cross-channel relations, a 1D-Conv. could be utilized to avoid involving tremendousparameters as in the fully connected case.
Revised: In contrast, to learn the views’ interactions andneglect the cross-channel relations, a 1D-Conv. could be utilized to avoid involving tremendousparameters as in the fully connected case.

Original: Using the 1D-Conv
Revised: Using the 1D-Conv.

Original: with k size = v as a fusion layerproduce E imgi ∈ R 1 × dim img , where k size is the kernel size.
Revised: with k size = v as a fusion layerproduce E imgi ∈ R 1 × dim img , where k size is the kernel size.

Original: The fused 2D features E imgi and the3D features E pci are concatenated and fed to a visual transformer, composed of several stackedtransformer layers.
Revised: The fused 2D features E imgi and the3D features E pci are concatenated and fed to a visual transformer, composed of several stackedtransformer layers.

Original: The visual transformer is utilized to capture the cross-correlation between the 2Dand the 3D features outputting refined features, V pc and V img , guided by three weighted auxiliarylosses, i.e., alignment loss between V pc and V img , and two visual classification losses.
Revised: The visual transformer is utilized to capture the cross-correlation between the 2Dand the 3D features outputting refined features, V pc and V img , guided by three weighted auxiliarylosses, i.e., alignment loss between V pc and V img , and two visual classification losses.

Original: – 2D semantics.
Revised: – 2D semantics.

Original: Inspired by SAT [51], we explore different 2D semantic information types combinations.
Revised: Inspired by SAT [52], we explore different 2D semantic information types combinations.

Original: Where we formulate our 2D semantics as follows:
Revised: Where we formulate our 2D semantics as follows:

Original: [Equation]
Revised: [Equation]

Original: Where, z ′ i is the predicted class labels, W 1 are learned projection matrices, LN is layer normalization,and ϕ indicates concatenation.
Revised: where, z ′ i is the predicted class labels, W 1 are learned projection matrices, LN is layer normalization,and ϕ indicates concatenation.

Original: The geometry semantics, x geoi ∈ R 1 × 30  represents the bounding boxcoordinates of object i in the scene space and encode the virtual camera parameters, i.e., intrinsicand extrinsic parameters, used while generating the 2D image x imgi .
Revised: We encode each object individually using different semantic information types, i.e., visual semantics E imgi , and geometry semantics x geoi ∈ R 1 × 30 . The geometrysemantics, x geoi represents the bounding box coordinates of object i in the scene space and encodethe virtual camera parameters, i.e., intrinsic and extrinsic parameters, used while generating the 2Dimage x img i .

Revised: Thus, it can be interpreted as a positional encoding for the vision transformer, where theinteraction among objects is captured.

Original: In contrast, SAT [51] utilizes aground-truth 2D semantics, where the z ′ i denotes the manually annotated class labels, and the E imgirepresents the embedding generated from real 2D images captured by imagery sensor[8].
Revised: In contrast, SAT [52] utilizes a ground-truth 2D semantics,where the z ′ i denotes the manually annotated class labels, and the E imgi represents the embeddinggenerated from real 2D images captured by imagery sensor

Original: – Multi-Modal Module.
Revised: – Multi-Modal Module.

Original: Using a pre-trained BERT model [11] followed by a language classificationhead adopted from Referit3D [2], we embed words query Q .
Revised: Using a pre-trained BERT model [11] followed by a language classification head adopted from Referit3D [2], we embed words query Q .

Original: After embedding each of the visualfeatures using the aforementioned visual module, we fuse the input modalities Q , V pc , V imgsimultaneously by using two different transformers; 3D Multi-Modal Transformer and 2D MultiModal Transformer.
Revised: After embedding each of the visual features using the aforementioned visual module, we fuse the input modalities Q , V pc , V img simultaneously by using two different transformers; 3D Multi-Modal Transformer and 2D MultiModal Transformer.

Original: In 3D Multi-Modal Transformer, we jointly encode the two input modalitiesV pc and Q to understand better the referring expression that captures co-occurrence relationships
Revised: In 3D Multi-Modal Transformer, we jointly encode the two input modalitiesV pc and Q to understand better the referring expression that captures co-occurrence relationships

Original: between different objects in the scene.
Revised: between different objects in the scene.

Original: Meanwhile, the same words query Q are encoded with V imgto enhance context-aware representation.
Revised: Meanwhile the same words query Q are encoded with V img to enhance context-aware representation.

Original: A separate 2D and 3D referring heads are adopted from
Revised: A separate 2D and 3D referring heads are adopted from

Original: Referit3D, which are auxiliary tasks.
Revised: Referit3D which are auxiliary tasks.

Original: Finally, the output of the multi-modal transformers isprocessedby a Fusion Transformer that chooses the referred object from the fused features, F pc and F img .
Revised: Finally the output of the multi- modal transformers is

Revised: processed

Revised: by

Revised: a

Revised: Fusion

Revised: Transformer

Revised: that chooses the referred object from the fused features

Revised: F pc and F img

Original: – Losses.
Revised: – Losses.

Original: Two auxiliary losses are employed to optimize the visual module; 3Dclassification loss L 3 Dcls and 2D classification loss
Revised: Our architecture, termed LAR, consists of two visual streams, i.e., 3D and 2D, in additionto one language stream.

Original: L 2 Dcls .
Revised: To refer to the object correctly guided by the input utterance, we must first classify the objects accurately.

Original:  We adopt Referit3D [2] language classification loss L langcls . Inaddition, we explore the effect of adding multiple grounding losses from different semantic levels,i.e., L 3 DEref , L 2 DEref , L 3 Dref and L 2 Dref and find that adding losses from both level boosts the groundingaccuracy.
Revised: Thus, two auxiliary losses are employed to optimize the visual module;3D classification loss L 3 Dcls and 2D classification loss L 2 Dcls . In addition, We adopted Referit3D [2]auxiliary loss for language classification loss L langcls to cooperate in the visual grounding task by recognizing the referred object class according to the description.

Original: Where, L 3 DEref and L 2 DEref indicates the early referring heads losses , before the fusiontransformer. L 3 Dref and L 2 Dref refers to the final referring losses.
Revised: Hence, two auxiliary grounding losses( L 3 DEref and L 2 DEref , termed as early referring losses, are exploited before the fusion transformer.

Original: The object correspondence loss L coris adopted from SAT [51].
Revised: As aresult of these early referring losses, the model learns the correlation between language descriptionand the visual features separately, which enhances the performance. Additionally, we adopted objectcorrespondence loss L cor from SAT to encourage the 3D model to distill the knowledge from the 2Dstream. Finally, two grounding losses, i.e., L 3 Dref and L 2 Dref were added after the fusion transformer topredict the scores of each proposal.

Original: We optimize the whole model in an End-to-End manner, unlike SAT whichlearns the 2D features E imgi separately, with the following loss function:
Revised: We optimize the whole model in an End-to-End manner, unlike SAT which learns the 2D features E imgi separately, with the following loss function:

Original: [Equation]
Revised: [Equation]

Original:  where, λ cls is the object classification loss weight, λ Eref is the early referring loss weight, λ ref is thefinal referring loss weight, λ cor is the object correspondence loss weight, and λ lang is the languageclassification loss weight.
Revised: ) where, λ cls is the object classification loss weight, λ Eref is the early referring loss weight, λ ref is thefinal referring loss weight, λ cor is the object correspondence loss weight, and λ lang is the languageclassification loss weight.

Original: 3.3 Implementation Details
Revised: 3.3 Implementation Details

Original: – Input Configuration.
Revised: – Input Configuration.

Original: For the 3D input, we randomly sample 1024 points per proposal from its pointcloud segment.
Revised: For the 3D input, we randomly sample 1024 points per proposal from its pointcloud segment.

Original: Furthermore, for the synthetic 2D input, we follow Algorithm 1, which is scrutinizedin Section3.1.
Revised: Furthermore, for the synthetic 2D input, we follow Algorithm , which is scrutinized in Section

Original: During the testing phase, the camera is set 2 meters away from the prominent facel ′ , i.e., d
Revised: During the testing phase, the camera is set 2 meters away from the prominent face

Original:  f = 2 , and 1 meter away from the floor, i.e., d up = 1 . 
Revised: l ′ , i.e., d f = 2 , and 1 meter away from the floor, i.e., d up = 1 . In addition, the region of

Original: In addition, the region of interestextension ratio ϵ , depicted in Figure 2 part (C), is set to 2%.
Revised: interest extension ratio ϵ , depicted in Figure 2 part (C), is set to 2%.

Original: The generated image is a square with asize of 32 pixels.
Revised: The generated image is a square with asize of 32 pixels.

Original: Five views are generated per object, where the angle θ between each camera is 30 ◦ ,demonstrated in Figure 1.
Revised: Five views are generated per object, where the angle θ between each camera is 30 ◦ ,demonstrated in Figure 1.

Original: Thus the camera is set to the following angles ( 0 ◦ , 30 ◦ , − 30 ◦ , 60 ◦ , − 60 ◦ ).
Revised: Thus the camera is set to the following angles ( 0 ◦ , 30 ◦ , − 30 ◦ , 60 ◦ , − 60 ◦ ).

Original: – Network and Training Configuration.
Revised: – Network and Training Configuration.

Original: A pre-trained model on the ImageNet dataset [9], TinyConvNext [28], is used to extract the features of our synthetic 2D images.
Revised: A pre-trained model on the ImageNet dataset [9], TinyConvNext [28], is used to extract the features of our synthetic 2D images.

Original: Our 2D backbone is sharedacross different views.
Revised: Our 2D backbone is sharedacross different views.

Original: We adopt the same 3D encoder in the previous works [1] [2] [13] [37] [18][51], PointNet++ [35], to encode the sampled points per proposal.
Revised: We adopt the same 3D encoder in the previous works [1] [2] [13] [37] [18][52], PointNet++ [35], to encode the sampled points per proposal.

Original: For the language encoder, weadopt the BERT encoder [11] with three layers.
Revised: For the language encoder, weadopt the BERT encoder [11] with three layers.

Original: We adopted the same transformer architecture used in[51] [50] for our visual, multi-modal, and fusion transformers.
Revised: We adopted the same transformer architecture used in[52] [51] for our visual, multi-modal, and fusion transformers.

Original: All models are trained for 100 epochsfrom scratch, using the weight initialization strategy described in [14].
Revised: All models are trained for 100 epochsfrom scratch, using the weight initialization strategy described in [14].

Original: As mentioned in SAT [51],the initial learning rate is set to 10 − 4 and decreases by 0.65 every ten epochs .
Revised: As mentioned in SAT [52],the initial learning rate is set to 10 − 4 and decreases by 0.65 every ten epochs .

Original: Adam optimizer [21]and mini-batch size of 32 per GPU are used for training all our models.
Revised: Adam optimizer [21]and mini-batch size of 32 per GPU are used for training all our models.

Original: We set the losses weights asfollows, λ cls = 5 , λ Eref = 0 .
Revised: We set the losses weights asfollows, λ cls = 5 , λ Eref = 0 .

Original: 5 , λ ref = 5 , λ cor = 10 , and λ lang = 0 .
Revised: 5 , λ ref = 5 , λ cor = 10 , and λ lang = 0 .

Original: – Software and Hardware Details.
Revised: – Software and Hardware Details.

Original: We use the python augmentation framework, Albumentations [4],for image augmentation.
Revised: We use the python augmentation framework, Albumentations [4],for image augmentation.

Original: In order to perform an online augmentation while training, we implementedour 2D Synthetic Images Generator (SIG), described at Section 3.1, in a pure Pythonic approachwithout using any 3D libraries which also fasten the training.
Revised: In order to perform an online augmentation while training, we implementedour 2D Synthetic Images Generator (SIG), described at Section 3.1, in a pure Pythonic approachwithout using any 3D libraries which also fasten the training.

Original: Our module is implemented in Pythonusing the PyTorch framework and four Nvidia V100 GPUs.
Revised: Our module is implemented in Pythonusing the PyTorch framework and four Nvidia V100 GPUs.

Original:  Experiments
Revised:  Experiments

Original: Datasets.
Revised: Datasets.

Original: We evaluate our proposed 3D visual grounding approach, LAR, on three benchmarkingdatasets, Nr3D [2], Sr3D [2], and ScanRefer [5].
Revised: We evaluate our proposed 3D visual grounding approach, LAR, on three benchmarkingdatasets, Nr3D [2], Sr3D [2], and ScanRefer [5].

Original: Nr3D, 3D Natural reference dataset [2], consistsof 41.5K natural, free-form utterances collected by humans using a referring game between twohumans.
Revised: Nr3D, 3D Natural reference dataset [2], consistsof 41.5K natural, free-form utterances collected by humans using a referring game between twohumans.

Original: As opposed to Sr3D [2], which consists of 83,5K synthetic utterances.
Revised: As opposed to Sr3D [2], which consists of 83,5K synthetic utterances.

Original: Unlike Nr3D and
Revised: Unlike Nr3D and

Original: Sr3D datasets, ScanRefer does not have the distractors concept making it relatively easier.
Revised: Sr3D datasets, ScanRefer does not have the distractors concept making it relatively easier.

Original: In addition,it provides 51.5K utterances of 11K objects for 800 3D indoor scenes.
Revised: In addition,it provides 51.5K utterances of 11K objects for 800 3D indoor scenes.

Original: 4.1 Ablation studies
Revised: Evaluation Metrics. we followed the same metrics followed by the previous work [2] [52], wherethree evaluation metrics are mainly used, i.e., the referring accuracy, the visual classification accuracy,and the language classification accuracy. In the case of Nr3d and Sr3d datasets, we followed theprevious work convention, assuming the proposals are generated with GT objects. Following thisassumption, the 3D grounding problem is reformulated as a classification problem, where the modelobjective is to classify the referred object among the other distractors correctly, termed referring accuracy.

Revised: The referring accuracy is calculated based on whether the model picks the correct proposalfrom a set of X proposals.

Revised: Alongside the main objective, which is the referring accuracy, twoauxiliary metrics are defined to evaluate the performance of each stream individually, i.e., the visualclassification accuracy and the language classification accuracy.

Revised: Given agnostic GT-proposals, thevisual classification accuracy is measured based on whether the model predicts the object’s classcorrectly or not from a predefined set of possible classes.

Revised: Similarly, language classification accuracyis another auxiliary metric that assesses the performance of the linguistic branch.

Revised: This branch aims tocorrectly predict the class of the referred object based on the input utterance.

Revised: 4.1 Ablation studies

Original: We conducted five ablation studies to assess the proposed architecture and to validate the contributionof each component to achieve good performance.
Revised: We conducted five ablation studies to assess the proposed architecture and to validate the contributionof each component to achieve good performance.

Original: These components are: 1) 2D multi-views and 2D resolution. Types of 2D semantic information.
Revised: These components are: 1) 2D multi-views and 2Dresolution. 2) Types of 2D semantic information.

Original: 3) Camera vs. image augmentation 4) Differentmulti-modal fusion techniques.
Revised: 3) Camera vs. image augmentation 4) Differentmulti-modal fusion techniques.

Original: Different 2D semantic fusion techniques.
Revised: Different 2D semantic fusion techniques.

Original: Where the last two arestated in Appendix 5.
Revised: Where the last two arestated in Appendix 5.

Original: [Table]
Revised: [Table]

Revised: [Table]

Original: – 2D multi-views and 2D resolution.
Revised: – 2D multi-views and 2D resolution.

Original: To assess our 2D
Revised: To assess our 2D

Original: Synthetic Images Generator (SIG), we have deliberatelyremoved the 3D stream, colored with yellow in Figure 3;3D encoder, 3D heads, and rely only on our generated 2Dimages only while tackling the referring task.
Revised: Synthetic Images Generator (SIG), we have deliberatelyremoved the 3D stream, colored with yellow in Figure 3;3D encoder, 3D heads, and rely only on our generated 2Dimages only while tackling the referring task.

Original: Thus, our architecture is simplified to include only the 2D encoder, 2Dmulti-modal transformer, and 2D heads.
Revised: Thus, our architecture is simplified to include only the 2D encoder, 2Dmulti-modal transformer, and 2D heads.

Original: We trained this2D architecture variant with different image resolutionsand multi-view setup combinations.
Revised: We trained this2D architecture variant with different image resolutionsand multi-view setup combinations.

Original: As shown in Table 2,
Revised: As shown in Table 2,

Original: The referring accuracy increased linearly while increasingthe image size or increasing the number of views for theprojected images.
Revised: The referring accuracy increased linearly while increasingthe image size or increasing the number of views for theprojected images.

Original: However, our multi-view setup has thedominant effect, whereas using only 64 image resolutionwith five views surpassing one shot with 128 resolution.
Revised: However, our multi-view setup has thedominant effect, whereas using only 64 image resolutionwith five views surpassing one shot with 128 resolution.

Original: Also, we found that using the combination of five different views, each has 128 resolution, surpassing
Revised: Also, we found that using the combination of five different views, each has 128 resolution, surpassing

Original: Referit3D and non-SAT variant, where both use 3D semantics only, which shows the quality of thelearned representation via our novel synthetic 2D images generator (SIG).
Revised: Referit3D and non-SAT variant, where both use 3D semantics only, which shows the quality of thelearned representation via our novel synthetic 2D images generator (SIG).

Original: [Table]
Revised: [Table]

Original: – Types of 2D semantic information.
Revised: – Types of 2D semantic information.

Original: Inspired by
Revised: Inspired by

Original: SAT [51], we dissect the role of different 2D semantic,following Eq. 1.
Revised: SAT [52], we dissect the role of different 2D semantic,following Eq. 1.

Original: As shown in Table 3, the first row(a) represents a 3D variant of our architecture whereonly the 3D encoder, 3D multi-modal transformer,and 3D heads are included.
Revised: As shown in Table 3, the first row(a) represents a 3D variant of our architecture where only the 3D encoder, 3D multi-modal transformer,and 3D heads are included.

Original: Therefore, this variantmimics the Non-SAT [51] configurations.
Revised: Therefore, this variant mimics the Non-SAT [52] configurations

Revised: Startingfrom

Revised: row b

Revised: to

Original: Startingfrom row b to row e, our synthetic 2D semantics areincorporated alongside the 3D semantics.
Revised: row e , our synthetic 2D semantics areincorporated alongside the 3D semantics.

Original: Also, wecompare against SAT [51] to highlight the criticaldifferences in the meaning of each semantic information.
Revised: Also, wecompare against SAT [52] to highlight the critical differences in the meaning of each semantic information.

Original: ROI in our architecture is the synthetic2D image generated from the 3d point clouds using our SIG module.
Revised: ROI in our architecture is the synthetic2D image generated from the 3d point clouds using our SIG module.

Original: In contrast, SAT [51] usesthe original ScanNet images during the training phase.
Revised: In contrast, SAT [52] usesthe original ScanNet images during the training phase.

Original: Unlike SAT, which uses the ground-truthclassification labels during the training and masks them during testing, we use the predicted labelsfrom the 2D classification head z ′ .
Revised: Unlike SAT, which uses the ground-truthclassification labels during the training and masks them during testing, we use the predicted labelsfrom the 2D classification head z ′ .

Original: Accordingly, we do not need to mask the 2D semantics, as we areusing synthetic ones.
Revised: Accordingly, we do not need to mask the 2D semantics, as we areusing synthetic ones.

Original: As shown in row (e), the best accuracy is achieved when we jointly incorporatethe three semantic types.
Revised: As shown in row (e), the best accuracy is achieved when we jointly incorporate the three semantic types.

Original: We also notice a significant superiority in classification accuracywhere oursynthetic 2D images assist the 3D classifier in both phases; training and testing.
Revised: We also notice a significant superiority in classification accuracy

Original: [Table]
Revised: where our synthetic 2D images assist the 3D classifier in both phases; training and testing.

Original: – Camera vs Image Augmentation.
Revised: – Camera vs. Image Augmentation.

Original: We have two kinds ofaugmentation during training: 1) Image augmentation.
Revised: We have two kinds of augmentation during training: 1) Imageaugmentation.

Original: 2) Camera augmentation.
Revised: 2) Camera augmentation.

Original: For image augmentation, we followed theconventional techniques used by the major computer visionapplications like horizontal and vertical flipping, blurring, color[Table] jitter, random scaling, random shifting, and random cropping.
Revised: For image augmentation, we followed the conventional techniques used by the major computervision applications like horizontal and vertical flipping, blurring, color jitter, random scaling, randomshifting, and random cropping.

Original: For camera augmentation, we range d f from 1.5 to 4 meters,d up from 0.5 to 2.5 meters, and ϵ from 2% to 15%.
Revised: For camera augmentation, we range d f from 1.5 to 4 meters, d upfrom 0.5 to 2.5 meters, and ϵ from 2% to 15%.

Original: By changingthe variables mentioned above, the  extrinsic camera parametersare augmented.
Revised: By changing the variables mentioned above, the [Table] extrinsic camera parameters are augmented.

Original: In addition, random noise is added to the intrinsic camera parameters.
Revised: In addition, random noise is added to the intrinsic cameraparameters.

Original: Table 4 demonstrates the effect of eachaugmentation type alone and the effect of fusing both of them.
Revised: Table 4 demonstrates the effect of each augmentation type alone and the effect of fusingboth of them.

Original: The best accuracy is achieved byutilizing the two augmentation types.
Revised: The best accuracy is achieved by utilizing the two augmentation types.

Original: 4.2 Comparison to State-of-the-art
Revised: [Table]

Original: Moreover, we achieve state-of-the-art results on the three well-known 3D visual grounding benchmarks, i.e., Nr3D [2], Sr3D [2], and ScanRefer [5].
Revised: [Table] – SIG Module vs. 2D real clues. To validate our SIG module, we tailored a variant of our architecture, where our SIG module is replaced by the actual 2Dclues produced by Faster R-CNN [39]. As shown in Table 5,replacing our SIG module with the features produced by Faster

Original: By effectively utilizing our 2D synthetic
Revised: R-CNN [39] degrades the performance by 2%. In addition,even when we compared unfairly to methods that use additional2D information, even the variant of SAT, which utilize realimages during training and testing (row #2), we still outperformit by 1%. 4.2 Comparison to State-of-the-art

Original: [Table]
Revised: [Table]

Revised: Moreover, we achieve state-of-the-art results on the threewell-known 3D visual grounding benchmarks, i.e., Nr3D[2], Sr3D [2], and ScanRefer [5].

Original:  images, LAR outperforms all the existing methods, which do not use any additional trainingdataset, by a large margin of 5.0 % and 1.9 % on Nr3D and Sr3D, respectively, depicted in Table 6. [Table]
Revised: By effectively utilizing our 2D synthetic images, LAR outperforms allthe existing methods, which do not use any additionaltraining dataset, by a large margin of 5.0 % and 1.9 %on Nr3D and Sr3D, respectively, depicted in Table 7.

Original: Where, LanguageRefer [37] achieves 43.9%, while LAR achieves 48.9% on Nr3D. For Sr3D, our model achieves 59.35%, which is a 1.9 % increase over TransRefer3D [13].
Revised: Where, LanguageRefer [37] achieves 43.9%, while LARachieves 48.9% on Nr3D. For Sr3D, our model achieves59.35%, which is a 1.9 % increase over TransRefer3D [13].

Original: In addition, for the ScanRefer [2] benchmark, we alsoachieve state-of-the-art results, by boosting the accuracy by 2.3 % from theclosest counterpart architecture; SAT [51], mentioned in Table 5.
Revised: In addition, for the ScanRefer [2] benchmark, we also achieve state-of-the-art results, by boosting theaccuracy by 2.3 % from the closest counterpart architecture; SAT [52], mentioned in Table 6. [Table]

Original: We retrained SAT [51] for a fair comparison, as we notice difficulties reproducing and verifying SAT results 1 .
Revised: We retrained SAT [52] for a fair comparison, as we notice difficulties reproducing and verifying SAT results 1 .

Original: Although we do not use an extra input duringtraining as SAT, which uses the original ScanNet 2D images [8], we surpass
Revised: Although we do not use an extra input duringtraining as SAT, which uses the original ScanNet 2D images [8], we surpass

Original: SAT by 1.6 % , 2.7 % , and 2.3 % on Nr3D, Sr3D, and ScanRefer, respectively.
Revised: SAT by 1.6 % , 2.7 % , and 2.3 % on Nr3D, Sr3D, and ScanRefer, respectively.

Original: 4.3 Discussion and Analysis
Revised: 4.3 Discussion and Analysis

Original: First, we will demonstrate our qualitative analysis.
Revised: First, we will demonstrate our qualitative analysis.

Original: Then, we have conducted two experiments toprobe the robustness of the representations learned by our proposed module LAR.
Revised: Then, we have conductedtwo experiments to probe the robustness of the representations learned by ourproposed module LAR.

Original: – Qualitative Analysis.
Revised: – Qualitative Analysis.

Original: Figure 5 depicts two corner cases in our projection module (SIG).
Revised: Figure 5 depicts two corner cases in our projectionmodule (SIG).

Original: Whileassigning the projected 3D points into a 2D grid, more than one point may share the exact spatiallocation on the grid, which means only one will be stored in the grid, and the rest of the points will bediscarded.
Revised: While assigning the projected 3D points into a 2D grid, morethan one point may share the exact spatial location on the grid, which means only one will be storedin the grid, and the rest of the points will be discarded.

Original: As shown in part (A) in Figure 5, on the left column, the floor overlaps the other objectsdue to the conflict between points during the assignation step.
Revised: As shown in part (A) in Figure 5, on the leftcolumn, the floor overlaps the other objects due to the conflict between points during the assignationstep.

Original: To overcome this behavior, a simplepriority rank is given for each point based on its height.
Revised: To overcome this behavior, a simple priority rank is given for each point based on its height.

Original: Part (B) shows another failure case, where theprominent face is falsely localized, which causes placing the camera outside the room or inside otherobjects.
Revised: Part (B) shows another failure case, where the prominent face is falsely localized, which causesplacing the camera outside the room or inside other objects.

Original: Therefore, we localize the prominent face relative to the room center to mitigate this effect.
Revised: Therefore, we localize the prominentface relative to the room center to mitigate this effect.

Original: An example of an ambiguous description is shown in Figure 6 part A.
Revised: An example of an ambiguous description isshown in Figure 6 part A.

Original: The model incorrectly predicts"the chair on the right rather than the chair on the left.
Revised: The model incorrectly predicts "the chair on the right rather than the chairon the left."

Original: " The model and the human annotations areoccasionally confusing to referring to the objects in the view-dependent description.
Revised: The model and the human annotations are occasionally confusing to referring to theobjects in the view-dependent description.

Original: While successfulcases can be seen in Figure 6 part B. For more qualitative visualization, please refer to Appendix 5.
Revised: While successful cases can be seen in Figure 6 part B. Formore qualitative visualization, please refer to Appendix 5.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: – Evaluation with Ground-Truth Class Labels.
Revised: – Evaluation with Ground-Truth Class Labels.

Original: Table 7 shows anothervariant, which is tailored from our architecture to be compared against SATGT.
Revised: Table 8 shows anothervariant, which is tailored from our architecture to be compared against SATGT.

Original: Where the predicted class labels are replaced by the ground-truth ones.
Revised: Where the predicted class labels are replaced by the ground-truth ones.

Original: Our LAR architecture outperforms SAT-GT by 12%.
Revised: Our LAR architecture outperforms SAT-GT by 12%.

Original: – Zero-shot Cameras Dying Testing.
Revised: – Zero-shot Cameras Dying Testing.

Original: We deliberately drop the cameras duringtesting to verify the robustness of our learned representations against camera variations.
Revised: We deliberately drop the cameras duringtesting to verify the robustness of our learned representations against cameravariations.

Original: As shown in Figure 4, starting from our best model, which is trainedusing five cameras, the referring accuracy drops to 44.6% then 40.15%, whenwe only use three cameras then one camera during testing, respectively.
Revised: As shown in Figure 4, starting from our best model, which is trainedusing five cameras, the referring accuracy drops to 44.6% then 40.15%, whenwe only use three cameras then one camera during testing, respectively.

Original: Driven by these results, camera dropout techniques can be invoked to overcome the performance drop, by
Revised: Drivenby these results, camera dropout techniques can be invoked to overcome the performance drop, bydeliberately drop a

Original: deliberately drop a random camera during the training; replace its input by aduplicate view of anothercamera.
Revised: random camera during the training; replace its input by a

Revised: duplicate

Revised: view of another

Original:  – Noisy Cameras Extrinsic and Intrinsic.
Revised: camera – Noisy Cameras Extrinsic and Intrinsic.

Original: Due to the significant improvement introduced by utilizingthe camera augmentation, as shown in Table 4, a detailed analysis is conducted to explore therobustness against the camera’s variations.
Revised: Due to the significant improvement introduced by utilizingthe camera augmentation, as shown in Table 4, a detailed analysis is conducted to explore therobustness against the camera’s variations.

Original: Consequently, our 2D synthetic branch learns how to fuseinformation across different cameras in a data-driven way.
Revised: Consequently, our 2D synthetic branch learns how to fuseinformation across different cameras in a data-driven way.

Original: Therefore, we can train the model to berobust to noise camera models, as shown in Table 8.
Revised: Therefore, we can train the model to berobust to noise camera models, as shown in Table 9.

Original: [Table]
Revised: [Table]

Original: Both cameras dying and noisy parameters experiments couldbe thought as irrelevant as LAR utilizes virtual cameras.
Revised: Both cameras dying and noisy parameters experiments could [Table] be thought as irrelevant as LAR utilizes virtual cameras.

Original: Thus,no camera dropout nor noise can be introduced.
Revised: Thus,no camera dropout nor noise can be introduced.

Original: However, our
Revised: However, our

Original: SIG module, depicted in Section 3.1, can be utilized in a widerange of applications when these robustness characteristicsagainst such noise are essential for these applications.
Revised: SIG module, depicted in Section 3.1, can be utilized in a widerange of applications when these robustness characteristicsagainst such noise are essential for these applications.

Original: Forinstance, in the autonomous driving world, our SIG module canbe employed to convert dense lidar point clouds into syntheticimages, which will be used to train an arbitrary neural network that tackles specific tasks (e.g., objectsegmentation in cocoon setup [31] [46]).
Revised: Forinstance, in the autonomous driving world, our SIG module canbe employed to convert dense lidar point clouds into syntheticimages, which will be used to train an arbitrary neural network that tackles specific tasks (e.g., objectsegmentation in cocoon setup [31] [47]).

Original: Then, the model will be tested using real cameras instead ofgenerating synthetic imagery input from the lidar.
Revised: Then, the model will be tested using real cameras instead ofgenerating synthetic imagery input from the lidar.

Original: – Limitations.
Revised: – Limitations.

Original: Following the standard setup of Referit3D [2], we assume access to the object proposalsof each 3D scene, similar to SAT [51], LanguageRefer [37], TransRefer3D [13], and 3DVG-Trans[58].
Revised: Following the standard setup of Referit3D [2], we assume access to the object proposalsof each 3D scene, similar to SAT [52], LanguageRefer [37], TransRefer3D [13], and 3DVG-Trans[59].

Original: Therefore, we use the ground-truth proposals in our experiments.
Revised: Therefore, we use the ground-truth proposals in our experiments.

Original: Following this assumption,the 3D grounding problem is reformulated as a classification problem, where the model objective isto classify the referred object among the other distractors correctly.
Revised: Following this assumption,the 3D grounding problem is reformulated as a classification problem, where the model objective isto classify the referred object among the other distractors correctly.

Original: However, LAR is compatiblewith the setting that uses detected-based proposals.
Revised: However, LAR is compatiblewith the setting that uses detected-based proposals.

Original:  Conclusion
Revised:  Conclusion

Original: We propose an efficient 3D grounding architecture that leverages the 2D knowledge to reinforce thelearned 3D representations.
Revised: We propose an efficient 3D grounding architecture that leverages the 2D knowledge to reinforce thelearned 3D representations.

Original: To avoid requiring 2D semantics neither during training nor testing, weintroduce a 2D synthetic images generator, termed SIG.
Revised: To avoid requiring 2D semantics neither during training nor testing, weintroduce a 2D synthetic images generator, termed SIG.

Original: Our SIG module generates high-quality 2Dsynthetic images.
Revised: Our SIG module generates high-quality 2Dsynthetic images.

Original: LAR, 2D Synthetic Semantics Knowledge Distillation for 3D Visual Grounding,consists of two stages, i.e., Visual and Multi-Modal modules.
Revised: LAR, 2D Synthetic Semantics Knowledge Distillation for 3D Visual Grounding,consists of two stages, i.e., Visual and Multi-Modal modules.

Original: LAR preserves the advantages ofdistilling the synthetic 2D knowledge to the 3D stream without requiring any additional information.
Revised: LAR preserves the advantages ofdistilling the synthetic 2D knowledge to the 3D stream without requiring any additional information.

Original: Our experiments show a significant superior performance, where LAR achieves state-of-the-art resultsacross three different 3D grounding benchmarks by outperforming the existing approaches, whichdo not rely on extra training data, by a wide margin, i.e., 5.0 % , 1.9 % and 2.3 % on Nr3D, Sr3Dand ScanRefer, respectively.
Revised: Our experiments show a significant superior performance, where LAR achieves state-of-the-art results across three different 3D grounding benchmarks by outperforming the existing approaches, whichdo not rely on extra training data, by a wide margin, i e., 5.0 % , 1.9 % and 2.3 % on Nr3D, Sr3Dand ScanRefer, respectively.

Original: Hence, despite the unfair comparison with SAT, which uses extra 2Dimages, LAR surpasses SAT by 1.6 % , 2.7 %, and 2.3 % on Nr3D, Sr3D, and ScanRefer, respectively.
Revised: Hence, despite the unfair comparison with SAT, which uses extra 2D images, LAR surpasses SAT by 1.6 % , 2.7 % and 2.3 % on Nr3D, Sr3D and ScanRefer, respectively.

Original: Furthermore, we verify the robustness of LAR and its generalization ability via zero-shot experimentsto intrinsic and extrinsic camera parameters and dying cameras.
Revised: Furthermore, we verify the robustness of LAR and its generalization ability via zero-shot experiments to intrinsic and extrinsic camera parameters and dying cameras.

Original: More qualitative results are attachedin the supplementary materials.
Revised: More qualitative results are attached in the supplementary materials.

Original: Checklist
Revised: Checklist

Original: For all authors...( a
Revised: For all authors...

Original: ) Do the main claims made

Original: in

Original: the abstract

Original:  and introduction accurately reflect the paper’scontributions and 
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?

Original: scope? [Yes b) Did you describe the limitations of your work?
Revised: [Yes](b) Did you describe the limitations of your work?

Original: [Yes] Please refer to Section 4.
Revised: [Yes] Please refer to Section 4.3(c)

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: Did you discuss any potential negative societal impacts of your work

Original: [N/A]
Revised: N /A]

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?
Revised: them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?
Revised: mental results (either in the supplemental material or as a URL)?

Original: [Yes] Our codes are
Revised: [Yes] Our codes are

Original: attached in the supplemental material combined with a detailed ’readme’ file.
Revised: attached in the supplemental material combined with a detailed ’readme’ file.

Original: Also,
Revised: Also,

Original: refer to Section 3.3 for the implementation details.
Revised: refer to Section 3.3 for the implementation details.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters,
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters,

Original: how they
Revised: how they

Original: were chosen)?
Revised: were chosen)?

Original: [Yes] Our codes are attached in the supplemental material combined
Revised: [Yes] Our codes are attached in the supplemental material combined

Original: with a detailed ’readme’ file.
Revised: with a detailed ’readme’ file.

Original: Also, refer to Section 3.3 for the implementation details.
Revised: Also, refer to Section 3.3 for the implementation details.

Original: (c ) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: (c) Did you report error bars (e g., with respect to the random seed after running experi- ments multiple times)?

Original: [Yes] Refer to Table 6. [Table]
Revised: [Yes] Refer to Table 7.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type

Original: [Table]

Original: of GPUs, internal cluster, or cloud provider)?
Revised: of GPUs, internal cluster, or cloud provider)?

Original: [Yes] Refer to Section 3.3.
Revised: [Yes] Refer to Section 3.3.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e g., code, data, models) or curating/releasing new assets...

Original: [Table]

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes]
Revised: [Yes]

Original: [Table]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes]
Revised: [Yes]

Original: [Table]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: [Table]

Original: using/curating?
Revised: using/curating?

Original: [Yes]
Revised: [Yes]

Original: [Table]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: [Table]

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: [Table]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: [Table]

Original: a) Did you include the full text of instructions given to participants and screenshots, if
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?
Revised: applicable?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A]
Revised: [N/A]

Original: A Appendix
Revised: A Appendix

Original: Our Supplemental Material contains the following sections:
Revised: Our Supplemental Material contains the following sections:

Original: • Demonstrate our synthetic images generator algorithm (SIG).
Revised: • Demonstrate our synthetic images generator algorithm (SIG).

Original: • More ablations.
Revised: • More ablations.

Original: • Qualitative results.
Revised: • Qualitative results.

Original: • Our source code.
Revised: • Our source code.

Original: A.1 Synthetic Images Generator Algorithm 
Revised: A.

Revised: Synthetic Images Generator Algorithm

Original: Algorithm 1, demonstrates a pseudo-code for our
Revised: Algorithm 1, demonstrates a pseudo-code for our

Original: SIG module thatprojects arbitrary 3D
Revised: SIG module thatprojects arbitrary 3D

Original: Point clouds representing an object to a 2D image.
Revised: Point clouds representing an object to a 2D image.

Original: do 7: end
Revised: 11: V f

Original: 8: end Cam p d d fy d ] 10: // Build transformation matrix M ∈
Revised: 12: V

Original: 11: V f ← ∥ Cam p − O l ′ ∥ 12: V r ← ∥ V f ⊗ [0 , 0 , 1] ∥ 13: V up ←∥ V r ⊗ V f ∥ 14: M ← (cid:20) V r V up
Revised: ←

Original: V f Cam p
Revised: V f ⊗

Original: 15: x cami ← Dot
Revised: 13: V up ← V

Original:  Mpci ) 16: x imgi ← Dot 
Revised: ⊗ 14: M Dot ←

Original: ] 18: return

Original: A.2 Ablations Studies
Revised: A.2 Ablations Studies

Revised: A.2.1 Different 2D semantic fusion techniques.

Original: Driven by the aforementioned results in Table 2, we use five virtual cameras set in a cocoon setuparound each object as depicted in Figure 1, and the image resolution is 128.
Revised: Driven by the aforementioned results in Table 2, we use five virtual cameras set in a cocoon setuparound each object as depicted in Figure 1, and the image resolution is 128.

Original: In Table 9, we studytwo fusion techniques to combine the features generated from the five views.
Revised: In Table 10, we studytwo fusion techniques to combine the features generated from the five views.

Original: The first method is asimple addition operation, and the second one is a learnable fusion mechanism where a 1D-Conv.is used where its kernel size is set to the number of the views used, i.e., in our case set to five.
Revised: The first method is asimple addition operation, and the second one is a learnable fusion mechanism where a 1D-Conv.is used where its kernel size is set to the number of the views used, i.e., in our case set to five.

Original: Asexpected, the learnable fusion scheme; 1D-Conv., performs better than the none learnable scheme; addition.
Revised: Asexpected, the learnable fusion scheme; 1D-Conv., performs better than the none learnable scheme;addition.

Original: Our 2D variant built on top of Referit3D [2] is used in this ablation, where the 2D stream isonly used.
Revised: Our 2D variant built on top of Referit3D [2] is used in this ablation, where the 2D stream isonly used.

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: A.2.2 Different multi-modal fusion techniques.
Revised: A.2.2 Different multi-modal fusion techniques.

Original: [Table]

Original: In addition, we study three fusion techniques for the 3D, and 2D multi-modal features, M pc and
Revised: In addition, we study three fusion techniques for the 3D, and 2D multi-modal features, M pc and

Original: M img , respectively.
Revised: M img , respectively.

Original: In this ablation, we only utilize the ROI as the 2D semantic information, set theimage resolution to 32, and set the number of multi-views to five.
Revised: In this ablation, we only utilize the ROI as the 2D semantic information, set theimage resolution to 32, and set the number of multi-views to five.

Original: As shown in Table 10, using atransformer while fusing both multi-modal features achieves the best accuracy.
Revised: As shown in Table 11, using atransformer while fusing both multi-modal features achieves the best accuracy.

Original: [Table] Optionally include extra information (complete proofs, additional experiments and plots) in the [Table] appendix.
Revised: Optionally include extra information (complete proofs, additional experiments and plots) in theappendix.

Original: This section will often be part of the supplemental material.
Revised: This section will often be part of the supplemental material.

Original: A.3 Quantitative Results
Revised: A.3 Quantitative Results

Original: This section analyzes the qualitative results from the successful and the failure cases on the Nr3ddataset.
Revised: This section analyzes the qualitative results from the successful and the failure cases on the Nr3ddataset.

Original: In addition to Figure 7 that shows successful scenarios and Figure 8 that shows failurescenarios, we attach a short video to demonstrate more scenarios
Revised: In addition to Figure 7 that shows successful scenarios and Figure 8 that shows failurescenarios, we attach a short video to demonstrate more scenarios.

Original:  .A.3.1 Successful Scenarios
Revised:  A.3.1 Successful Scenarios

Original: LAR achieves top-ranking results on both the Nr3D and Sr3D datasets.
Revised: LAR achieves top-ranking results on both the Nr3D and Sr3D datasets.

Original: The results in Figure 7demonstrate the correct instance localization on the Nr3D datasets only.
Revised: The results in Figure 7demonstrate the correct instance localization on the Nr3D datasets only.

Original: The ground truth boxesare marked green and our model prediction marked in blue.
Revised: The ground truth boxesare marked green and our model prediction marked in blue.

Original: Our analysis shows that LAR correctlyunderstands the global context based on natural language description.
Revised: Our analysis shows that LAR correctlyunderstands the global context based on natural language description.

Original: For example, in Figure 7 (a-f),
Revised: For example, in Figure 7 (a-f),

Original: LAR understands relationships, e.g. (behind it, middle of, furthest from .. ), which proves that ourproposed model understands instance-to-instance and instance-to-background relations.
Revised: LAR understands relationships, e.g. (behind it, middle of, furthest from .. ), which proves that ourproposed model understands instance-to-instance and instance-to-background relations.

Original: As well asunderstanding colors and shapes related to language, LAR also can determine the meaning of termssuch as “green,” Figure 7 (e).
Revised: As well asunderstanding colors and shapes related to language, LAR also can determine the meaning of termssuch as “green,” Figure 7 (e).

Original: A.3.2 Failure Scenarios
Revised: A.3.2 Failure Scenarios

Original: Figure 8 shows representative failure cases of LAR.
Revised: Figure 8 shows representative failure cases of LAR.

Original: Despise the incorrect instance localization LARmanaged to allocate the instance with the same class as the target.
Revised: Despise the incorrect instance localization LARmanaged to allocate the instance with the same class as the target.

Original: As shown in Figure 8 (d, e), weobserved that failure cases require an understanding of "against the wall", ’Facing the windows,"which is determined by camera view dependency.
Revised: As shown in Figure 8 (d, e), weobserved that failure cases require an understanding of "against the wall", ’Facing the windows,"which is determined by camera view dependency.

Original: Despite LAR’s progress in view-dependent andindependent samples, view understanding remains a challenge.
Revised: Despite LAR’s progress in view-dependent andindependent samples, view understanding remains a challenge.

Original: Besides, other state-of-the-art modelssuffer from the same issue.
Revised: Besides, other state-of-the-art modelssuffer from the same issue.

Original: In addition, some LAR failure cases are caused by ambiguous queries.
Revised: In addition, some LAR failure cases are caused by ambiguous queries.

Original: Figure 8 (a,b,c,e, f) shows a complex and compound language description to describe the targetinstance, which in some cases, both models and human annotators mix up similar objects.
Revised: Figure 8 (a,b,c,e, f) shows a complex and compound language description to describe the targetinstance, which in some cases, both models and human annotators mix up similar objects.

Original: To summarize, We show the effect of using synthetic 2D images, which improves the model instanceand context-aware representation.
Revised: To summarize, We show the effect of using synthetic 2D images, which improves the model instanceand context-aware representation.

Original: However, our model has limitations due to ambiguous and complexqueries.
Revised: However, our model has limitations due to ambiguous and complexqueries.

Original: Although some cases remained unsolved, LAR improved progress in the camera viewdependency samples.
Revised: Although some cases remained unsolved, LAR improved progress in the camera viewdependency samples.

Original: Therefore, we believe our model LAR surpasses existing state-of-the-art by asignificant margin.
Revised: Therefore, we believe our model LAR surpasses existing state-of-the-art by asignificant margin.

Original: A.4 Source Code
Revised: A.4 Source Code

Original: our source code is attached in the supplementary material, where it contains the source code for our
Revised: our source code is attached in the supplementary material, where it contains the source code for our

Original: Synthetic Images Generator (SIG), our architecture, training functions, and evaluation functions.
Revised: Synthetic Images Generator (SIG), our architecture, training functions, and evaluation functions.

Original: Formore details regarding our implementation, refer to the Readme file attached with the source code.
Revised: Formore details regarding our implementation, refer to the Readme file attached with the source code.