Original: Look Around and Refer: 2D Synthetic Semantics Knowledge Distillation for 3D Visual Grounding
Revised: Look Around and Refer: 2D Synthetic Semantics Knowledge Distillation for 3D Visual Grounding

Original: Abstract
Revised: Abstract

Original: The 3D visual grounding task has been explored with visual and language streamsto comprehend referential language to identify targeted objects in 3D scenes.
Revised: The 3D visual grounding task has been explored with visual and language streamsto comprehend referential language to identify targeted objects in 3D scenes.

Original: However, most existing methods devote the visual stream to capturing the 3D visualclues using off-the-shelf point clouds encoders.
Revised: However, most existing methods devote the visual stream to capturing the 3D visualclues using off-the-shelf point clouds encoders.

Original: The main question we address inthis paper is “can we consolidate the 3D visual stream by 2D clues synthesizedfrom point clouds and efficiently utilize them in training and testing?” .
Revised: The main question we address inthis paper is “can we consolidate the 3D visual stream by 2D clues synthesizedfrom point clouds and efficiently utilize them in training and testing?” .

Original: The mainidea is to assist the 3D encoder by incorporating rich 2D object representationswithout requiring extra 2D inputs.
Revised: The mainidea is to assist the 3D encoder by incorporating rich 2D object representationswithout requiring extra 2D inputs.

Original: To this end, we leverage 2D clues, syntheticallygenerated from 3D point clouds, and empirically show their aptitude to boost thequality of the learned visual representations.
Revised: To this end, we leverage 2D clues, syntheticallygenerated from 3D point clouds, and empirically show their aptitude to boost thequality of the learned visual representations.

Original: We validate our approach throughcomprehensive experiments on Nr3D, Sr3D, and ScanRefer datasets, and we showconsistent performance gains compared to the state-of-the-art.
Revised: We validate our approach throughcomprehensive experiments on Nr3D, Sr3D, and ScanRefer datasets, and we showconsistent performance gains compared to the state-of-the-art.

Original: Our proposedmodule, dubbed as Look Around and Refer (LAR), significantly outperforms thestate-of-the-art 3D visual grounding techniques on these three benchmarks.
Revised: Our proposedmodule, dubbed as Look Around and Refer (LAR), significantly outperforms thestate-of-the-art 3D visual grounding techniques on these three benchmarks.

Original: Ourcode will be made publicly available.
Revised: Ourcode will be made publicly available.

Original:  Introduction
Revised:  Introduction

Original: Visiolinguistic modeling has attracted much attentionover the past decade due to impact on various areas,including visual question answering [3, 36], 3D scenecaptioning [7], image-text matching [26, 44], visualrelation detection [25] and scenegraph generation[42].
Revised: Visiolinguistic modeling has attracted much attentionover the past decade due to impact on various areas,including visual question answering [3, 36], 3D scenecaptioning [7], image-text matching [26, 44], visualrelation detection [25] and scenegraph generation[42].

Original: Consequently, understanding natural languageutterances and grounding them to real-world scenarios is attractive due to their importance in many realapplications, such as robot navigation and interactionin indoor environment [22] [40].
Revised: Consequently, understanding natural languageutterances and grounding them to real-world scenarios is attractive due to their importance in many realapplications, such as robot navigation and interactionin indoor environment [22] [40].

Original: Visual Grounding (VG) has been explored in recentyears in the context of 2D [16, 30, 46, 55, 57], extending object detection to a more complex task.
Revised: Visual Grounding (VG) has been explored in recentyears in the context of 2D [16, 30, 46, 55, 57], extending object detection to a more complex task.

Original: However, for real robots to intelligently ground a givenutterance on a natural scene in a challenging environment like a crowded indoor or outdoor scene, 3D representations are needed to understand the surroundingenvironment better.
Revised: However, for real robots to intelligently ground a givenutterance on a natural scene in a challenging environment like a crowded indoor or outdoor scene, 3D representations are needed to understand the surroundingenvironment better.

Original: Therefore, 3D visual groundingtask have been proposed by ReferIt3D [2] and ScanRefer [5].
Revised: Therefore, 3D visual groundingtask have been proposed by ReferIt3D [2] and ScanRefer [5].

Original: The ReferIt3D [2] introduces two datasets
Revised: The ReferIt3D [2] introduces two datasets

Original: [Table]
Revised: [Table]

Original: containing natural and synthetic language utterances, namely Nr3D and Sr3D, respectively, whichaims to determine the referent object from pre-defined objects set for each scene.
Revised: containing natural and synthetic language utterances, namely Nr3D and Sr3D, respectively, whichaims to determine the referent object from pre-defined objects set for each scene.

Original: In contrast, ScanRefer [5] aims to predict 3D bounding boxes for the referent object based on a given languagedescription and 3D point clouds.
Revised: In contrast, ScanRefer [5] aims to predict 3D bounding boxes for the referent object based on a given languagedescription and 3D point clouds.

Original: Both ScanRefer [5] and ReferIt3D [2] are built on the ScanNetdataset [8].
Revised: Both ScanRefer [5] and ReferIt3D [2] are built on the ScanNetdataset [8].

Original: The critical difference, which made ReferIt3D [2] more challenging, is that in each scene,several object instances belong to the same fine-grained category as the referent, namely distractors.
Revised: The critical difference, which made ReferIt3D [2] more challenging, is that in each scene,several object instances belong to the same fine-grained category as the referent, namely distractors.

Original: Due to the sparsity and disorganization of the 3D point clouds, SAT [52] explores utilizing the 2Dimages provided by the ScanNet dataset [8] during training to assist point-cloud-language jointrepresentation learning.
Revised: Due to the sparsity and disorganization of the 3D point clouds, SAT [52] explores utilizing the 2Dimages provided by the ScanNet dataset [8] during training to assist point-cloud-language jointrepresentation learning.

Original: SAT [52] proposes three variants; 1) The baseline, non-SAT, relies only onthe 3D point clouds while tackling the grounding task.
Revised: SAT [52] proposes three variants; 1) The baseline, non-SAT, relies only onthe 3D point clouds while tackling the grounding task.

Original: Utilizing 2D semantics during the trainingphase only and masking them while inference, we termed it SAT.
Revised: Utilizing 2D semantics during the trainingphase only and masking them while inference, we termed it SAT.

Original: 3) Utilizing the 2D semantics inboth phases, i.e., training and testing, we termed it SAT-GT.
Revised: 3) Utilizing the 2D semantics inboth phases, i.e., training and testing, we termed it SAT-GT.

Original: However, requiring extra 2D inputs intraining or inference limits potential application scenarios.
Revised: However, requiring extra 2D inputs intraining or inference limits potential application scenarios.

Original: To tackle this drawback, we hypothesizedthat incorporating synthetic 2D semantics will consolidate the 3D learned representation while notrequiring any additional info.
Revised: To tackle this drawback, we hypothesizedthat incorporating synthetic 2D semantics will consolidate the 3D learned representation while notrequiring any additional info.

Original: To this end, we propose a novel 2D Synthetic Images Generator (SIG)that projects the 3D point clouds into multi-view 2D images, as shown in Figure 1.
Revised: To this end, we propose a novel 2D Synthetic Images Generator (SIG)that projects the 3D point clouds into multi-view 2D images, as shown in Figure 1.

Original: To the bestof our knowledge, LAR is the first method that augment learning of 3D tasks with synthetic 2Dsemantics.
Revised: To the bestof our knowledge, LAR is the first method that augment learning of 3D tasks with synthetic 2Dsemantics.

Original: In addition, we propose a novel multi-modal transformer-based architecture.
Revised: In addition, we propose a novel multi-modal transformer-based architecture.

Original: Our proposedarchitecture, LAR, on Nr3D outperforms SAT [52] by boosting the accuracy by 1.6 % while exceedingall existing approaches that did not use any additional information than the 3D point clouds by awider margin, i.e., 5.0 % to the closest one.
Revised: Our proposedarchitecture, LAR, on Nr3D outperforms SAT [52] by boosting the accuracy by 1.6 % while exceedingall existing approaches that did not use any additional information than the 3D point clouds by awider margin, i.e., 5.0 % to the closest one.

Original: Also, LAR architecture achieves state-of-the-art resultson Sr3D and ScanRefer benchmarks, 59.35% and 54.6%, respectively.
Revised: Also, LAR architecture achieves state-of-the-art resultson Sr3D and ScanRefer benchmarks, 59.35% and 54.6%, respectively.

Original: Another variant is tailoredfrom our architecture to be compared against SAT-GT, where geometry information and ground-truthclass labels are concatenated to the 2D images; synthetic in our case and original images in SAT case.
Revised: Another variant is tailoredfrom our architecture to be compared against SAT-GT, where geometry information and ground-truthclass labels are concatenated to the 2D images; synthetic in our case and original images in SAT case.

Original: Our LAR architecture outperforms SAT-GT by 11.7 % , where we achieved 62% while SAT achieves50.3%.
Revised: Our LAR architecture outperforms SAT-GT by 11.7 % , where we achieved 62% while SAT achieves50.3%.

Original: Our contributions are summarized as follows:
Revised: Our contributions are summarized as follows:

Original: • We propose a novel End-to-End multi-modal transformer-based architecture, LAR, for the3D visual grounding task that relies only on 3D point clouds, which exploits the synthetic2D images generated by our SIG module.• We verify the effectiveness of our LAR architecture throughout extensive experiments with
Revised: • We propose a novel End-to-End multi-modal transformer-based architecture, LAR, for the3D visual grounding task that relies only on 3D point clouds, which exploits the synthetic2D images generated by our SIG module.• We verify the effectiveness of our LAR architecture throughout extensive experiments with

Original: various benchmarks, which show that utilizing our 2D synthetic semantics significantly
Revised: various benchmarks, which show that utilizing our 2D synthetic semantics significantly

Original: enhances the results.
Revised: enhances the results.

Original: • Through detailed analysis, we examine our method’s internal behavior and validity.
Revised: • Through detailed analysis, we examine our method’s internal behavior and validity.

Original:  Related Work
Revised:  Related Work

Original: Our novel architecture draws success from several areas, including 2D visual grounding, 3D visualgrounding, and 2D semantics in 3D tasks.
Revised: Our novel architecture draws success from several areas, including 2D visual grounding, 3D visualgrounding, and 2D semantics in 3D tasks.

Original: – 2D
Revised: – 2D

Original: visual grounding.
Revised: visual grounding.

Original: 2D visual grounding is locating an object based on a natural languagedescription.
Revised: 2D visual grounding is locating an object based on a natural languagedescription.

Original: Flickr30k [32] and ReferItGame [20] generated referring expressions for a particularobject in an image used to identify which object is being referred to.
Revised: Flickr30k [32] and ReferItGame [20] generated referring expressions for a particularobject in an image used to identify which object is being referred to.

Original: Prior works [56, 27, 29]have attempted to localize these referred objects based on the given expressions by focusing on thebounding-box level.
Revised: Prior works [56, 27, 29]have attempted to localize these referred objects based on the given expressions by focusing on thebounding-box level.

Original: In contrast, [53, 10] focus on the pixels level.
Revised: In contrast, [53, 10] focus on the pixels level.

Original: Another 2D visual groundingtaxonomy is the one-stage, also known as a single-stage framework, versus the two-stage framework.
Revised: Another 2D visual groundingtaxonomy is the one-stage, also known as a single-stage framework, versus the two-stage framework.

Original: The single-stage methods [38, 49, 50], fuse the linguistic query with each pixel/patches of theimage, and generate a set of possible bounding box candidates.
Revised: The single-stage methods [38, 49, 50], fuse the linguistic query with each pixel/patches of theimage, and generate a set of possible bounding box candidates.

Original: In contrast, the two-stage framework[45, 54, 56] uses the visual image semantics to generate object proposals.
Revised: In contrast, the two-stage framework[45, 54, 56] uses the visual image semantics to generate object proposals.

Original: Afterward, the grounding prediction is selected by comparing each proposal to the language query.
Revised: Afterward, the grounding prediction is selected by comparing each proposal to the language query.

Original: Consequently, 2D visualgrounding methods can guide learning 3D relationships.
Revised: Consequently, 2D visualgrounding methods can guide learning 3D relationships.

Original: Driven by this, we explore using 2D syntheticsemantics to consolidate the 3D visual stream.
Revised: Driven by this, we explore using 2D syntheticsemantics to consolidate the 3D visual stream.

Original: – 3D visual grounding.
Revised: – 3D visual grounding.

Original: Table 1 summarizes the existing 3D visual grounding models whether aground truth object proposal is used or a pre-trained detector is utilized to generate the proposals.
Revised: Table 1 summarizes the existing 3D visual grounding models whether aground truth object proposal is used or a pre-trained detector is utilized to generate the proposals.

Original: Also, it summarizes whether extra info is used alongside the object’s 3D point cloud or not.
Revised: Also, it summarizes whether extra info is used alongside the object’s 3D point cloud or not.

Original: 3Dvisual grounding field has undergone extensive research and made considerable progress over thelast few years [2, 5, 12, 17, 59, 1, 37, 13, 58, 19, 52, 6, 19].
Revised: 3Dvisual grounding field has undergone extensive research and made considerable progress over thelast few years [2, 5, 12, 17, 59, 1, 37, 13, 58, 19, 52, 6, 19].

Original: A 3D visual grounding task useslanguage instruction along with a 3D scene [8] as input in order to estimate a bounding box thatallocates the most relevant object to the provided expression.
Revised: A 3D visual grounding task useslanguage instruction along with a 3D scene [8] as input in order to estimate a bounding box thatallocates the most relevant object to the provided expression.

Original: However, the literature offers severalextensions, such as using auxiliary input to improve localizing 3D objects [52].
Revised: However, the literature offers severalextensions, such as using auxiliary input to improve localizing 3D objects [52].

Original: Previous work[2, 12, 17, 59, 1, 13, 58, 19], focuses on improving the language and 3D scene fusion by adopting atwo-stage framework.
Revised: Previous work[2, 12, 17, 59, 1, 13, 58, 19], focuses on improving the language and 3D scene fusion by adopting atwo-stage framework.

Original: The first stage outputs 3D object proposals given a 3D point cloud representinga scene.
Revised: The first stage outputs 3D object proposals given a 3D point cloud representinga scene.

Original: Secondly, visual and linguistic features are fused to compute a confidence score for each3D object proposal.
Revised: Secondly, visual and linguistic features are fused to compute a confidence score for each3D object proposal.

Original: The first stage could be skipped by directly using the ground-truth proposalsinstead of predicting them [52] [13] [2].
Revised: The first stage could be skipped by directly using the ground-truth proposalsinstead of predicting them [52] [13] [2].

Original: In contrast, [6, 19, 59] do not assume the access to theground-truth objects and use an object detector to extract the proposals.
Revised: In contrast, [6, 19, 59] do not assume the access to theground-truth objects and use an object detector to extract the proposals.

Original: Intuitively, if the detectorfails, the referring performance will significantly decrease.
Revised: Intuitively, if the detectorfails, the referring performance will significantly decrease.

Original: Therefore, we focus on the main trainingobjective related to the grounding task, which is learning 3D visual-language joint representations byassuming access to the ground-truth boxes.
Revised: Therefore, we focus on the main trainingobjective related to the grounding task, which is learning 3D visual-language joint representations byassuming access to the ground-truth boxes.

Original: Although we
Revised: Although we

Original: are assuming access to the ground-truthboxes, LAR is compatible with the setting that utilizes detector-generated proposals.
Revised: are assuming access to the ground-truthboxes, LAR is compatible with the setting that utilizes detector-generated proposals.

Original: – 2D semantics in 3D tasks.
Revised: – 2D semantics in 3D tasks.

Original: 2D semantics contains useful information that can be used to detect
Revised: 2D semantics contains useful information that can be used to detect

Original: 3D objects.
Revised: 3D objects.

Original: The representations of visual scenes consist of point clouds, meshes, or volumes,while 2D images consist of pixel grids.
Revised: The representations of visual scenes consist of point clouds, meshes, or volumes,while 2D images consist of pixel grids.

Original: Combining the best of two worlds can enhance 3D object
Revised: Combining the best of two worlds can enhance 3D object

Original: localization.
Revised: localization.

Original: Researchers have attempted to solve the 3D detection problem by adopting different
Revised: Researchers have attempted to solve the 3D detection problem by adopting different

Original: ways of representing RGB-D data [34, 48, 23, 15, 41, 24].
Revised: ways of representing RGB-D data [34, 48, 23, 15, 41, 24].

Original: By either projecting the 2D image on
Revised: By either projecting the 2D image on

Original: the 3D scene [34, 48, 24] or fuse both 2D features with the 3D [23, 15, 43, 41], [33].
Revised: the 3D scene [34, 48, 24] or fuse both 2D features with the 3D [23, 15, 43, 41], [33].

Original: SAT [52]
Revised: SAT [52]

Original: leveraged 2D semantics to perform the 3D VG task.
Revised: leveraged 2D semantics to perform the 3D VG task.

Original: However, requiring extra 2D inputs limits
Revised: However, requiring extra 2D inputs limits

Original: potential application scenarios.
Revised: potential application scenarios.

Original: In contrast, in this work, we generate synthetic 2D images during
Revised: In contrast, in this work, we generate synthetic 2D images during

Original: the training phase, which is also used in the inference phase.
Revised: the training phase, which is also used in the inference phase.

Original: The synthetic images enable us to
Revised: The synthetic images enable us to

Original: consolidate the 3D visual stream by 2D semantics without requiring extra 2D inputs.
Revised: consolidate the 3D visual stream by 2D semantics without requiring extra 2D inputs.

Original:  Look Around and Refer (LAR)
Revised:  Look Around and Refer (LAR)

Original: LAR aims to enhance the visual module, which in turn enhances the final objective,i.e., referentialaccuracy.
Revised: LAR aims to enhance the visual module, which in turn enhances the final objective,i.e., referentialaccuracy.

Original: By incorporating the synthetic 2D semantics alongside the 3D point clouds, LAR inherentlycaptures the correlations between the 2D and the 3D streams.
Revised: By incorporating the synthetic 2D semantics alongside the 3D point clouds, LAR inherentlycaptures the correlations between the 2D and the 3D streams.

Original: To design a general frameworkto fit in any 3D application, when the input is 3D point clouds, the need for extra information,such as 2D semantics, should be bypassed.
Revised: To design a general frameworkto fit in any 3D application, when the input is 3D point clouds, the need for extra information,such as 2D semantics, should be bypassed.

Original: To this end, a 2D synthetic images generator, namely
Revised: To this end, a 2D synthetic images generator, namely

Original: SIG, is introduced, which is discussed in Section 3.1.
Revised: SIG, is introduced, which is discussed in Section 3.1.

Original: Then, we demonstrate how to learn robustrepresentations by aligning the generated 2D semantics with the 3D ones by incorporating our SIGmodule into our proposed architecture, depicted in Section 3.2.
Revised: Then, we demonstrate how to learn robustrepresentations by aligning the generated 2D semantics with the 3D ones by incorporating our SIGmodule into our proposed architecture, depicted in Section 3.2.

Original: 3.1 Look Around: 2D Synthetic Images Generator
Revised: 3.1 Look Around: 2D Synthetic Images Generator

Original: LAR exploits synthetic 2D semantics to reinforce the learned 3D representations.
Revised: LAR exploits synthetic 2D semantics to reinforce the learned 3D representations.

Original: Consequently, thissection will demonstrate our 2D Synthetic Images Generator (SIG) module that projects arbitrary 3D
Revised: Consequently, thissection will demonstrate our 2D Synthetic Images Generator (SIG) module that projects arbitrary 3D

Original: Point clouds representing an object to a 2D image.
Revised: Point clouds representing an object to a 2D image.

Original: First, we will show the essential operations of ourmodule on a single image, then show how we extend it to fulfill a multi-view setup.
Revised: First, we will show the essential operations of ourmodule on a single image, then show how we extend it to fulfill a multi-view setup.

Original: A 3D scene, S ∈ R N × 6 , is represented by N points with spatial and color information, i.e., XYZ,and RGB, respectively.
Revised: A 3D scene, S ∈ R N × 6 , is represented by N points with spatial and color information, i.e., XYZ,and RGB, respectively.

Original: Following the previous works [2] [1] [37] [52], the object proposals x pci foreach scene are known, either generated by one of the off-the-shelf 3D object detectors or manuallyannotated.
Revised: Following the previous works [2] [1] [37] [52], the object proposals x pci foreach scene are known, either generated by one of the off-the-shelf 3D object detectors or manuallyannotated.

Original: The object proposal x pci ∈ R N ′ × 6 is a subset of the whole scene, where N ′ < N and N ′represents the number of points that represent the object i .
Revised: The object proposal x pci ∈ R N ′ × 6 is a subset of the whole scene, where N ′ < N and N ′represents the number of points that represent the object i .

Original: Given a scene S and an object proposal x pci , our SIG module generates a 2D synthetic image x imgirepresents the object i .
Revised: Given a scene S and an object proposal x pci , our SIG module generates a 2D synthetic image x imgirepresents the object i .

Original: The main idea is to place a virtual camera in front of the fed object proposal,then project its points x pci into the 2D plane to generate the corresponding 2D image x imgi .
Revised: The main idea is to place a virtual camera in front of the fed object proposal,then project its points x pci into the 2D plane to generate the corresponding 2D image x imgi .

Original: Thecamera’s location should be set carefully to generate a decent image representation for an object i .
Revised: Thecamera’s location should be set carefully to generate a decent image representation for an object i .

Original: Asshown in part (A) in Figure 2, after excluding the upper and lower faces from the fed object proposal,we get the center of the four remaining faces, then calculate the distance between each face center O li ,and the center of the scene O s , where l ranging from 0 to 3.
Revised: Asshown in part (A) in Figure 2, after excluding the upper and lower faces from the fed object proposal,we get the center of the four remaining faces, then calculate the distance between each face center O li ,and the center of the scene O s , where l ranging from 0 to 3.

Original: The nearest face l ′ from the scene center
Revised: The nearest face l ′ from the scene center

Original: O s is the prominent one, colored with blue in Figure 2, which guarantees that we look at the objecti from inside the room.
Revised: O s is the prominent one, colored with blue in Figure 2, which guarantees that we look at the objecti from inside the room.

Original: The camera is located at a distance d f from the prominent face l ′ and at adistance d up from the room’s floor; thus, the camera position is interpreted as Cam p =
Revised: The camera is located at a distance d f from the prominent face l ′ and at adistance d up from the room’s floor; thus, the camera position is interpreted as Cam p =

Original: [ d fx , d fy , d up ] ,as shown in part (B) in Figure 2.
Revised: [ d fx , d fy , d up ] ,as shown in part (B) in Figure 2.

Original: The orientation of the camera frame is defined as follows.
Revised: The orientation of the camera frame is defined as follows.

Original: The z axisis aligned with the principal axis of the camera, and two other orthogonal directions (x, y) align withthe corresponding axes of the image plane.
Revised: The z axisis aligned with the principal axis of the camera, and two other orthogonal directions (x, y) align withthe corresponding axes of the image plane.

Original: Failure cases are shown in Figure 5, when the camera islocated outside the room.
Revised: Failure cases are shown in Figure 5, when the camera islocated outside the room.

Original: To project the points x pci , representing object i , from the scene coordinates to the camera coordinates,a transformation matrix M ∈ R 4 × 4 needs to be defined.
Revised: To project the points x pci , representing object i , from the scene coordinates to the camera coordinates,a transformation matrix M ∈ R 4 × 4 needs to be defined.

Original: Given the camera position Cam p and the center of the prominent face O l ′ , the transformation matrix M can be interpreted as, (cid:20) V r V up V
Revised: Given the camera position Cam p and the center of the prominent face O l ′ , the transformation matrix M can be interpreted as, (cid:20) V r V up V

Original: f Cam p   (cid:21)
Revised: f Cam p   (cid:21)

Original: Where, the camera direction is defined by three vectors; V f , V r , and V up .
Revised: Where, the camera direction is defined by three vectors; V f , V r , and V up .

Original: We define our virtualcamera using the Unified Camera Model (UCM), which has five parameters I =
Revised: We define our virtualcamera using the Unified Camera Model (UCM), which has five parameters I =

Original: [ γ x , γ y , c x , c y , ζ ]
Revised: [ γ x , γ y , c x , c y , ζ ]

Original: Following the UCM, projection is defined as follows: π ( x pci , I )
Revised: Following the UCM, projection is defined as follows: π ( x pci , I )

Original: =
Revised: =

Original: (cid:20) γ
Revised: (cid:20) γ

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: γ
Revised: γ

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: (cid:21) + (cid:20) c
Revised: (cid:21) + (cid:20) c

Original: [Equation]
Revised: [Equation]

Original: c
Revised: c

Original: [Equation]
Revised: [Equation]

Original: (cid:21) ,
Revised: (cid:21) ,

Original: Afterwards, the projected 3D point clouds are assigned to a 2D grid map.
Revised: Afterwards, the projected 3D point clouds are assigned to a 2D grid map.

Original: To mitigate the overfittingeffect, we randomly extend the region of interest by ϵ as demonstrated in part (C) at Figure 2.
Revised: To mitigate the overfittingeffect, we randomly extend the region of interest by ϵ as demonstrated in part (C) at Figure 2.

Original: Point cloud differs from 2D images, where every pixel contains spatial and texture details.
Revised: Point cloud differs from 2D images, where every pixel contains spatial and texture details.

Original: Moreover,the object is represented in a view-invariant way, unlike the images that are view-dependent.
Revised: Moreover,the object is represented in a view-invariant way, unlike the images that are view-dependent.

Original: To mitigate this drawback in the image representation, a multi-view setup is constructed as shown in
Revised: To mitigate this drawback in the image representation, a multi-view setup is constructed as shown in

Original: Figure 1.
Revised: Figure 1.

Original: Where, v cameras are mounted in a circle around the prominent face l ′ .
Revised: Where, v cameras are mounted in a circle around the prominent face l ′ .

Original: Algorithm 1,attached in the supplemental materials, demonstrates a pseudo-code for our SIG module.
Revised: Algorithm 1,attached in the supplemental materials, demonstrates a pseudo-code for our SIG module.

Original: 3.2 Refer: Visiolinguistic Transformer Architecture
Revised: 3.2 Refer: Visiolinguistic Transformer Architecture

Original: We present our approach, depicted in Figure 3, for learning multi-modal representations of a 3Dscene associated with language instruction.
Revised: We present our approach, depicted in Figure 3, for learning multi-modal representations of a 3Dscene associated with language instruction.

Original: First, the proposed visual module enriches the 3Drepresentations by utilizing the 2D synthetic images generator, namely the SIG module, described in
Revised: First, the proposed visual module enriches the 3Drepresentations by utilizing the 2D synthetic images generator, namely the SIG module, described in

Original: Section 3.1.
Revised: Section 3.1.

Original: Thanks to it, we do not require access to any imagery sensor during training or testing.
Revised: Thanks to it, we do not require access to any imagery sensor during training or testing.

Original: Then, the multi-modal module learns joint representations targeted to the final task, i.e., referring toan object in the scene based on the language instruction.
Revised: Then, the multi-modal module learns joint representations targeted to the final task, i.e., referring toan object in the scene based on the language instruction.

Original: – Visual Module.
Revised: – Visual Module.

Original: Given a
Revised: Given a

Original: 3D
Revised: 3D

Original: scene
Revised: scene

Original: S
Revised: S

Original: ∈ R N
Revised: ∈ R N

Original: ×
Revised: ×

Original: 6 combined with agnostic multiple 3D
Revised: 6 combined with agnostic multiple 3D

Original: object
Revised: object

Original: proposalsx pci ∈ R N ′ × 6 , we aim to classify each
Revised: proposalsx pci ∈ R N ′ × 6 , we aim to classify each

Original: object
Revised: object

Original: proposal by predicting
Revised: proposal by predicting

Original: the
Revised: the

Original: semantic
Revised: semantic

Original: labels
Revised: labels

Original: z .
Revised: z .

Original: First,for
Revised: First,for

Original: each object i in the scene S
Revised: each object i in the scene S

Original: , we generate multi view synthetic 2D images x imgi ∈ R v × H ×
Revised: , we generate multi view synthetic 2D images x imgi ∈ R v × H ×

Original: W
Revised: W

Original: × 3from the
Revised: × 3from the

Original: 3D object’s
Revised: 3D object’s

Original: point
Revised: point

Original: cloud x pci
Revised: cloud x pci

Original: , where
Revised: , where

Original: v
Revised: v

Original: is the number of views for each object, H and Ware the spatial dimensions for the generated images.
Revised: is the number of views for each object, H and Ware the spatial dimensions for the generated images.

Original: Second, we employ PointNet++ [35] as a 3Dencoder, producing 3D features E pc i ∈ R 1 × dim pc .
Revised: Second, we employ PointNet++ [35] as a 3Dencoder, producing 3D features E pc i ∈ R 1 × dim pc .

Original: Hence, the Tiny-ConvNext [28] is adopted as a2D encoder, that produce y imgi ∈ R v × H ′ × W ′ × dim img .
Revised: Hence, the Tiny-ConvNext [28] is adopted as a2D encoder, that produce y imgi ∈ R v × H ′ × W ′ × dim img .

Original: Where H ′ and W ′ represent a coarse spatialgrid of spatial features, each represented by a dim img .
Revised: Where H ′ and W ′ represent a coarse spatialgrid of spatial features, each represented by a dim img .

Original: Then, the spatial dimensions are squeezedproducing o imgi ∈ R v × dim img .
Revised: Then, the spatial dimensions are squeezedproducing o imgi ∈ R v × dim img .

Original: Since our 2D encoder encodes multiple views v for the same objecti , a fusing mechanism is needed to fuse the views’ features.
Revised: Since our 2D encoder encodes multiple views v for the same objecti , a fusing mechanism is needed to fuse the views’ features.

Original: Various possibilities are discussed tosettle down on the best fusion mechanism.
Revised: Various possibilities are discussed tosettle down on the best fusion mechanism.

Original: We can interpret this as a fully connected layer to fullycapture the views and cross-channel interactions.
Revised: We can interpret this as a fully connected layer to fullycapture the views and cross-channel interactions.

Original: In contrast, to learn the views’ interactions andneglect the cross-channel relations, a 1D-Conv. could be utilized to avoid involving tremendousparameters as in the fully connected case.
Revised: In contrast, to learn the views’ interactions andneglect the cross-channel relations, a 1D-Conv. could be utilized to avoid involving tremendousparameters as in the fully connected case.

Original: Using the 1D-Conv.
Revised: Using the 1D-Conv.

Original: with k size = v as a fusion layerproduce E imgi ∈ R 1 × dim img , where k size is the kernel size.
Revised: with k size = v as a fusion layerproduce E imgi ∈ R 1 × dim img , where k size is the kernel size.

Original: The fused 2D features E imgi and the3D features E pci are concatenated and fed to a visual transformer, composed of several stackedtransformer layers.
Revised: The fused 2D features E imgi and the3D features E pci are concatenated and fed to a visual transformer, composed of several stackedtransformer layers.

Original: The visual transformer is utilized to capture the cross-correlation between the 2Dand the 3D features outputting refined features, V pc and V img , guided by three weighted auxiliarylosses, i.e., alignment loss between V pc and V img , and two visual classification losses.
Revised: The visual transformer is utilized to capture the cross-correlation between the 2Dand the 3D features outputting refined features, V pc and V img , guided by three weighted auxiliarylosses, i.e., alignment loss between V pc and V img , and two visual classification losses.

Original: – 2D semantics.
Revised: – 2D semantics.

Original: Inspired by SAT [52], we explore different 2D semantic information types combinations.
Revised: Inspired by SAT [52], we explore different 2D semantic information types combinations.

Original: Where we formulate our 2D semantics as follows:
Revised: Where we formulate our 2D semantics as follows:

Original: [Equation]
Revised: [Equation]

Original: where, z ′ i is the predicted class labels, W 1 are learned projection matrices, LN is layer normalization,and ϕ indicates concatenation.
Revised: where, z ′ i is the predicted class labels, W 1 are learned projection matrices, LN is layer normalization,and ϕ indicates concatenation.

Original: We encode each object individually using different semantic information types, i.e., visual semantics E imgi , and geometry semantics x geoi ∈ R 1 × 30 .
Revised: We encode each object individually using different semantic information types, i.e., visual semantics E imgi , and geometry semantics x geoi ∈ R 1 × 30 .

Original: The geometrysemantics, x geoi represents the bounding box coordinates of object i in the scene space and encodethe virtual camera parameters, i.e., intrinsic and extrinsic parameters, used while generating the 2Dimage x img i .
Revised: The geometrysemantics, x geoi represents the bounding box coordinates of object i in the scene space and encodethe virtual camera parameters, i.e., intrinsic and extrinsic parameters, used while generating the 2Dimage x img i .

Original: Thus, it can be interpreted as a positional encoding for the vision transformer, where theinteraction among objects is captured.
Revised: Thus, it can be interpreted as a positional encoding for the vision transformer, where theinteraction among objects is captured.

Original: In contrast, SAT [52] utilizes a ground-truth 2D semantics,where the z ′ i denotes the manually annotated class labels, and the E imgi represents the embeddinggenerated from real 2D images
Revised: In contrast, SAT [52] utilizes a ground-truth 2D semantics,where the z ′ i denotes the manually annotated class labels, and the E imgi represents the embeddinggenerated from real 2D images

Original: captured by
Revised: captured by

Original: imagery sensor
Revised: imagery sensor

Original: – Multi-Modal Module.
Revised: – Multi-Modal Module.

Original: Using a pre-trained BERT model [11] followed by a language classification
Revised: Using a pre-trained BERT model [11] followed by a language classification

Original: head adopted from Referit3D [2], we embed words query Q .
Revised: head adopted from Referit3D [2], we embed words query Q .

Original: After embedding each of the visual
Revised: After embedding each of the visual

Original: features using the aforementioned visual module, we fuse the input modalities Q , V pc , V img
Revised: features using the aforementioned visual module, we fuse the input modalities Q , V pc , V img

Original: simultaneously by using two different transformers; 3D Multi-Modal Transformer and 2D MultiModal Transformer.
Revised: simultaneously by using two different transformers; 3D Multi-Modal Transformer and 2D MultiModal Transformer.

Original: In 3D Multi-Modal Transformer, we jointly encode the two input modalitiesV pc and Q to understand better the referring expression that captures co-occurrence relationships
Revised: In 3D Multi-Modal Transformer, we jointly encode the two input modalitiesV pc and Q to understand better the referring expression that captures co-occurrence relationships

Original: between different objects in the scene.
Revised: between different objects in the scene.

Original: Meanwhile
Revised: Meanwhile

Original: the same words query Q are encoded with V img
Revised: the same words query Q are encoded with V img

Original: to enhance context-aware representation.
Revised: to enhance context-aware representation.

Original: A separate 2D and 3D referring heads are adopted from
Revised: A separate 2D and 3D referring heads are adopted from

Original: Referit3D
Revised: Referit3D

Original: which are auxiliary tasks.
Revised: which are auxiliary tasks.

Original: Finally
Revised: Finally

Original: the output of the multi-
Revised: the output of the multi-

Original: modal
Revised: modal

Original: transformers is
Revised: transformers is

Original: processed
Revised: processed

Original: by
Revised: by

Original: a
Revised: a

Original: Fusion
Revised: Fusion

Original: Transformer
Revised: Transformer

Original: that chooses the referred object from the fused features
Revised: that chooses the referred object from the fused features

Original: F pc and F img
Revised: F pc and F img

Original: – Losses.
Revised: – Losses.

Original: Our architecture, termed LAR, consists of two visual streams, i.e., 3D and 2D, in additionto one language stream.
Revised: Our architecture, termed LAR, consists of two visual streams, i.e., 3D and 2D, in additionto one language stream.

Original: To refer to the object correctly guided by the input utterance, we must first
Revised: To refer to the object correctly guided by the input utterance, we must first

Original: classify the objects accurately.
Revised: classify the objects accurately.

Original: Thus, two auxiliary losses are employed to optimize the visual module;3D classification loss L 3 Dcls and 2D classification loss L 2 Dcls .
Revised: Thus, two auxiliary losses are employed to optimize the visual module;3D classification loss L 3 Dcls and 2D classification loss L 2 Dcls .

Original: In addition, We adopted Referit3D [2]auxiliary loss for language classification loss L langcls to cooperate in the visual grounding task by recognizing the referred object class according to the description.
Revised: In addition, We adopted Referit3D [2]auxiliary loss for language classification loss L langcls to cooperate in the visual grounding task by recognizing the referred object class according to the description.

Original: Hence, two auxiliary grounding losses( L 3 DEref and L 2 DEref , termed as early referring losses, are exploited before the fusion transformer.
Revised: Hence, two auxiliary grounding losses( L 3 DEref and L 2 DEref , termed as early referring losses, are exploited before the fusion transformer.

Original: As aresult of these early referring losses, the model learns the correlation between language descriptionand the visual features separately, which enhances the performance.
Revised: As aresult of these early referring losses, the model learns the correlation between language descriptionand the visual features separately, which enhances the performance.

Original: Additionally, we adopted objectcorrespondence loss L cor from SAT to encourage the 3D model to distill the knowledge from the 2Dstream.
Revised: Additionally, we adopted objectcorrespondence loss L cor from SAT to encourage the 3D model to distill the knowledge from the 2Dstream.

Original: Finally, two grounding losses, i.e., L 3 Dref and L 2 Dref were added after the fusion transformer topredict the scores of each proposal.
Revised: Finally, two grounding losses, i.e., L 3 Dref and L 2 Dref were added after the fusion transformer topredict the scores of each proposal.

Original: We optimize the whole model in an End-to-End manner, unlike
Revised: We optimize the whole model in an End-to-End manner, unlike

Original: SAT which learns the 2D features E imgi separately, with the following loss function:
Revised: SAT which learns the 2D features E imgi separately, with the following loss function:

Original: [Equation]
Revised: [Equation]

Original: ) where, λ cls is the object classification loss weight, λ Eref is the early referring loss weight, λ ref is thefinal referring loss weight, λ cor is the object correspondence loss weight, and λ lang is the languageclassification loss weight.
Revised: ) where, λ cls is the object classification loss weight, λ Eref is the early referring loss weight, λ ref is thefinal referring loss weight, λ cor is the object correspondence loss weight, and λ lang is the languageclassification loss weight.

Original: 3.3 Implementation Details
Revised: 3.3 Implementation Details

Original: – Input Configuration.
Revised: – Input Configuration.

Original: For the 3D input, we randomly sample 1024 points per proposal from its pointcloud segment.
Revised: For the 3D input, we randomly sample 1024 points per proposal from its pointcloud segment.

Original: Furthermore, for the synthetic 2D input, we follow Algorithm
Revised: Furthermore, for the synthetic 2D input, we follow Algorithm

Original: , which is scrutinized
Revised: , which is scrutinized

Original: in
Revised: in

Original: Section
Revised: Section

Original: During the testing phase, the camera is set 2 meters away from the prominent face
Revised: During the testing phase, the camera is set 2 meters away from the prominent face

Original: l ′ , i.e., d f = 2 , and 1 meter away from the floor, i.e., d up = 1 .
Revised: l ′ , i.e., d f = 2 , and 1 meter away from the floor, i.e., d up = 1 .

Original: In addition, the region
Revised: In addition, the region

Original: of
Revised: of

Original: interest
Revised: interest

Original: extension ratio ϵ , depicted in
Revised: extension ratio ϵ , depicted in

Original: Figure
Revised: Figure

Original: 2 part (C), is set to 2%.
Revised: 2 part (C), is set to 2%.

Original: The generated image is a square with asize of 32 pixels.
Revised: The generated image is a square with asize of 32 pixels.

Original: Five views are generated per object, where the angle θ between each camera is 30 ◦ ,demonstrated in Figure 1.
Revised: Five views are generated per object, where the angle θ between each camera is 30 ◦ ,demonstrated in Figure 1.

Original: Thus the camera is set to the following angles ( 0 ◦ , 30 ◦ , − 30 ◦ , 60 ◦ , − 60 ◦ ).
Revised: Thus the camera is set to the following angles ( 0 ◦ , 30 ◦ , − 30 ◦ , 60 ◦ , − 60 ◦ ).

Original: – Network and Training Configuration.
Revised: – Network and Training Configuration.

Original: A pre-trained model on the ImageNet dataset [9], TinyConvNext [28], is used to extract the features of our synthetic 2D images.
Revised: A pre-trained model on the ImageNet dataset [9], TinyConvNext [28], is used to extract the features of our synthetic 2D images.

Original: Our 2D backbone is sharedacross different views.
Revised: Our 2D backbone is sharedacross different views.

Original: We adopt the same 3D encoder in the previous works [1] [2] [13] [37] [18][52], PointNet++ [35], to encode the sampled points per proposal.
Revised: We adopt the same 3D encoder in the previous works [1] [2] [13] [37] [18][52], PointNet++ [35], to encode the sampled points per proposal.

Original: For the language encoder, weadopt the BERT encoder [11] with three layers.
Revised: For the language encoder, weadopt the BERT encoder [11] with three layers.

Original: We adopted the same transformer architecture used in[52] [51] for our visual, multi-modal, and fusion transformers.
Revised: We adopted the same transformer architecture used in[52] [51] for our visual, multi-modal, and fusion transformers.

Original: All models are trained for 100 epochsfrom scratch, using the weight initialization strategy described in [14].
Revised: All models are trained for 100 epochsfrom scratch, using the weight initialization strategy described in [14].

Original: As mentioned in SAT [52],the initial learning rate is set to 10 − 4 and decreases by 0.65 every ten epochs .
Revised: As mentioned in SAT [52],the initial learning rate is set to 10 − 4 and decreases by 0.65 every ten epochs .

Original: Adam optimizer [21]and mini-batch size of 32 per GPU are used for training all our models.
Revised: Adam optimizer [21]and mini-batch size of 32 per GPU are used for training all our models.

Original: We set the losses weights asfollows, λ cls = 5 , λ Eref = 0 .
Revised: We set the losses weights asfollows, λ cls = 5 , λ Eref = 0 .

Original: 5 , λ ref = 5 , λ cor = 10 , and λ lang = 0 .
Revised: 5 , λ ref = 5 , λ cor = 10 , and λ lang = 0 .

Original: – Software and Hardware Details.
Revised: – Software and Hardware Details.

Original: We use the python augmentation framework, Albumentations [4],for image augmentation.
Revised: We use the python augmentation framework, Albumentations [4],for image augmentation.

Original: In order to perform an online augmentation while training, we implementedour 2D Synthetic Images Generator (SIG), described at Section 3.1, in a pure Pythonic approachwithout using any 3D libraries which also fasten the training.
Revised: In order to perform an online augmentation while training, we implementedour 2D Synthetic Images Generator (SIG), described at Section 3.1, in a pure Pythonic approachwithout using any 3D libraries which also fasten the training.

Original: Our module is implemented in Pythonusing the PyTorch framework and four Nvidia V100 GPUs.
Revised: Our module is implemented in Pythonusing the PyTorch framework and four Nvidia V100 GPUs.

Original:  Experiments
Revised:  Experiments

Original: Datasets.
Revised: Datasets.

Original: We evaluate our proposed 3D visual grounding approach, LAR, on three benchmarkingdatasets, Nr3D [2], Sr3D [2], and ScanRefer [5].
Revised: We evaluate our proposed 3D visual grounding approach, LAR, on three benchmarkingdatasets, Nr3D [2], Sr3D [2], and ScanRefer [5].

Original: Nr3D, 3D Natural reference dataset [2], consistsof 41.5K natural, free-form utterances collected by humans using a referring game between twohumans.
Revised: Nr3D, 3D Natural reference dataset [2], consistsof 41.5K natural, free-form utterances collected by humans using a referring game between twohumans.

Original: As opposed to Sr3D [2], which consists of 83,5K synthetic utterances.
Revised: As opposed to Sr3D [2], which consists of 83,5K synthetic utterances.

Original: Unlike Nr3D and
Revised: Unlike Nr3D and

Original: Sr3D datasets, ScanRefer does not have the distractors concept making it relatively easier.
Revised: Sr3D datasets, ScanRefer does not have the distractors concept making it relatively easier.

Original: In addition,it provides 51.5K utterances of 11K objects for 800 3D indoor scenes.
Revised: In addition,it provides 51.5K utterances of 11K objects for 800 3D indoor scenes.

Original: Evaluation Metrics.
Revised: Evaluation Metrics.

Original: we followed the same metrics followed by the previous work [2] [52], wherethree evaluation metrics are mainly used, i.e., the referring accuracy, the visual classification accuracy,and the language classification accuracy.
Revised: we followed the same metrics followed by the previous work [2] [52], wherethree evaluation metrics are mainly used, i.e., the referring accuracy, the visual classification accuracy,and the language classification accuracy.

Original: In the case of Nr3d and Sr3d datasets, we followed theprevious work convention, assuming the proposals are generated with GT objects.
Revised: In the case of Nr3d and Sr3d datasets, we followed theprevious work convention, assuming the proposals are generated with GT objects.

Original: Following thisassumption, the 3D grounding problem is reformulated as a classification problem, where the modelobjective is to classify the referred object among the other distractors correctly, termed referring accuracy.
Revised: Following thisassumption, the 3D grounding problem is reformulated as a classification problem, where the modelobjective is to classify the referred object among the other distractors correctly, termed referring accuracy.

Original: The referring accuracy is calculated based on whether the model picks the correct proposalfrom a set of X proposals.
Revised: The referring accuracy is calculated based on whether the model picks the correct proposalfrom a set of X proposals.

Original: Alongside the main objective, which is the referring accuracy, twoauxiliary metrics are defined to evaluate the performance of each stream individually, i.e., the visualclassification accuracy and the language classification accuracy.
Revised: Alongside the main objective, which is the referring accuracy, twoauxiliary metrics are defined to evaluate the performance of each stream individually, i.e., the visualclassification accuracy and the language classification accuracy.

Original: Given agnostic GT-proposals, thevisual classification accuracy is measured based on whether the model predicts the object’s classcorrectly or not from a predefined set of possible classes.
Revised: Given agnostic GT-proposals, thevisual classification accuracy is measured based on whether the model predicts the object’s classcorrectly or not from a predefined set of possible classes.

Original: Similarly, language classification accuracyis another auxiliary metric that assesses the performance of the linguistic branch.
Revised: Similarly, language classification accuracyis another auxiliary metric that assesses the performance of the linguistic branch.

Original: This branch aims tocorrectly predict the class of the referred object based on the input utterance.
Revised: This branch aims tocorrectly predict the class of the referred object based on the input utterance.

Original: 4.1 Ablation studies
Revised: 4.1 Ablation studies

Original: We conducted five ablation studies to assess the proposed architecture and to validate the contributionof each component to achieve good performance.
Revised: We conducted five ablation studies to assess the proposed architecture and to validate the contributionof each component to achieve good performance.

Original: These components are: 1) 2D multi-views and 2Dresolution. 2) Types of 2D semantic information.
Revised: These components are: 1) 2D multi-views and 2Dresolution. 2) Types of 2D semantic information.

Original: 3) Camera vs. image augmentation 4) Differentmulti-modal fusion techniques.
Revised: 3) Camera vs. image augmentation 4) Differentmulti-modal fusion techniques.

Original: Different 2D semantic fusion techniques.
Revised: Different 2D semantic fusion techniques.

Original: Where the last two arestated in Appendix 5.
Revised: Where the last two arestated in Appendix 5.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: – 2D multi-views and 2D resolution.
Revised: – 2D multi-views and 2D resolution.

Original: To assess our 2D
Revised: To assess our 2D

Original: Synthetic Images Generator (SIG), we have deliberatelyremoved the 3D stream, colored with yellow in Figure 3;3D encoder, 3D heads, and rely only on our generated 2Dimages only while tackling the referring task.
Revised: Synthetic Images Generator (SIG), we have deliberatelyremoved the 3D stream, colored with yellow in Figure 3;3D encoder, 3D heads, and rely only on our generated 2Dimages only while tackling the referring task.

Original: Thus, our architecture is simplified to include only the 2D encoder, 2Dmulti-modal transformer, and 2D heads.
Revised: Thus, our architecture is simplified to include only the 2D encoder, 2Dmulti-modal transformer, and 2D heads.

Original: We trained this2D architecture variant with different image resolutionsand multi-view setup combinations.
Revised: We trained this2D architecture variant with different image resolutionsand multi-view setup combinations.

Original: As shown in Table 2,
Revised: As shown in Table 2,

Original: The referring accuracy increased linearly while increasingthe image size or increasing the number of views for theprojected images.
Revised: The referring accuracy increased linearly while increasingthe image size or increasing the number of views for theprojected images.

Original: However, our multi-view setup has thedominant effect, whereas using only 64 image resolutionwith five views surpassing one shot with 128 resolution.
Revised: However, our multi-view setup has thedominant effect, whereas using only 64 image resolutionwith five views surpassing one shot with 128 resolution.

Original: Also, we found that using the combination of five different views, each has 128 resolution, surpassing
Revised: Also, we found that using the combination of five different views, each has 128 resolution, surpassing

Original: Referit3D and non-SAT variant, where both use 3D semantics only, which shows the quality of thelearned representation via our novel synthetic 2D images generator (SIG).
Revised: Referit3D and non-SAT variant, where both use 3D semantics only, which shows the quality of thelearned representation via our novel synthetic 2D images generator (SIG).

Original: [Table]
Revised: [Table]

Original: – Types of 2D semantic information.
Revised: – Types of 2D semantic information.

Original: Inspired by
Revised: Inspired by

Original: SAT [52], we dissect the role of different 2D semantic,following Eq. 1.
Revised: SAT [52], we dissect the role of different 2D semantic,following Eq. 1.

Original: As shown in Table 3, the first row(a) represents a 3D variant of our architecture where
Revised: As shown in Table 3, the first row(a) represents a 3D variant of our architecture where

Original: only the 3D encoder, 3D multi-modal transformer,and 3D heads are included.
Revised: only the 3D encoder, 3D multi-modal transformer,and 3D heads are included.

Original: Therefore,
Revised: Therefore,

Original: this variant
Revised: this variant

Original: mimics the Non-SAT [52] configurations
Revised: mimics the Non-SAT [52] configurations

Original: Startingfrom
Revised: Startingfrom

Original: row b
Revised: row b

Original: to
Revised: to

Original: row e
Revised: row e

Original: , our
Revised: , our

Original: synthetic 2D
Revised: synthetic 2D

Original: semantics areincorporated alongside the 3D semantics.
Revised: semantics areincorporated alongside the 3D semantics.

Original: Also, wecompare against SAT [52] to highlight the
Revised: Also, wecompare against SAT [52] to highlight the

Original: critical
Revised: critical

Original: differences in the meaning of each semantic information.
Revised: differences in the meaning of each semantic information.

Original: ROI in our architecture is the synthetic2D image generated from the 3d point clouds using our SIG module.
Revised: ROI in our architecture is the synthetic2D image generated from the 3d point clouds using our SIG module.

Original: In contrast, SAT [52] usesthe original ScanNet images during the training phase.
Revised: In contrast, SAT [52] usesthe original ScanNet images during the training phase.

Original: Unlike SAT, which uses the ground-truthclassification labels during the training and masks them during testing, we use the
Revised: Unlike SAT, which uses the ground-truthclassification labels during the training and masks them during testing, we use the

Original: predicted
Revised: predicted

Original: labelsfrom the 2D classification head z ′ .
Revised: labelsfrom the 2D classification head z ′ .

Original: Accordingly, we do not need to mask the 2D semantics, as we areusing synthetic ones.
Revised: Accordingly, we do not need to mask the 2D semantics, as we areusing synthetic ones.

Original: As shown in row (e), the best accuracy is achieved when we jointly incorporate
Revised: As shown in row (e), the best accuracy is achieved when we jointly incorporate

Original: the
Revised: the

Original: three
Revised: three

Original: semantic types.
Revised: semantic types.

Original: We
Revised: We

Original: also
Revised: also

Original: notice a significant
Revised: notice a significant

Original: superiority in classification accuracy
Revised: superiority in classification accuracy

Original: where our
Revised: where our

Original: synthetic
Revised: synthetic

Original: 2D images assist the 3D classifier in both phases; training and testing.
Revised: 2D images assist the 3D classifier in both phases; training and testing.

Original: – Camera vs. Image Augmentation.
Revised: – Camera vs. Image Augmentation.

Original: We have two kinds of augmentation during training: 1) Imageaugmentation.
Revised: We have two kinds of augmentation during training: 1) Imageaugmentation.

Original: 2) Camera augmentation.
Revised: 2) Camera augmentation.

Original: For image augmentation, we followed the conventional techniques used by the major computervision applications like horizontal and vertical flipping, blurring, color jitter, random scaling, randomshifting, and random cropping.
Revised: For image augmentation, we followed the conventional techniques used by the major computervision applications like horizontal and vertical flipping, blurring, color jitter, random scaling, randomshifting, and random cropping.

Original: For camera augmentation, we range d f from 1.5 to 4 meters, d upfrom 0.5 to 2.5 meters, and ϵ from 2% to 15%.
Revised: For camera augmentation, we range d f from 1.5 to 4 meters, d upfrom 0.5 to 2.5 meters, and ϵ from 2% to 15%.

Original: By changing the variables mentioned above, the
Revised: By changing the variables mentioned above, the

Original: [Table]
Revised: [Table]

Original: extrinsic camera parameters are augmented.
Revised: extrinsic camera parameters are augmented.

Original: In addition, random noise is added to the intrinsic cameraparameters.
Revised: In addition, random noise is added to the intrinsic cameraparameters.

Original: Table 4 demonstrates the effect of each augmentation type alone and the effect of fusingboth of them.
Revised: Table 4 demonstrates the effect of each augmentation type alone and the effect of fusingboth of them.

Original: The best accuracy is achieved by utilizing the two augmentation types.
Revised: The best accuracy is achieved by utilizing the two augmentation types.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: – SIG Module vs. 2D real clues.
Revised: – SIG Module vs. 2D real clues.

Original: To validate our SIG module, we tailored a variant of our architecture, where our SIG module is replaced by the actual 2Dclues produced by Faster R-CNN [39].
Revised: To validate our SIG module, we tailored a variant of our architecture, where our SIG module is replaced by the actual 2Dclues produced by Faster R-CNN [39].

Original: As shown in Table 5,replacing our SIG module with the features produced by Faster
Revised: As shown in Table 5,replacing our SIG module with the features produced by Faster

Original: R-CNN [39] degrades the performance by 2%.
Revised: R-CNN [39] degrades the performance by 2%.

Original: In addition,even when we compared unfairly to methods that use additional2D information, even the variant of SAT, which utilize realimages during training and testing (row #2), we still outperformit by 1%.
Revised: In addition,even when we compared unfairly to methods that use additional2D information, even the variant of SAT, which utilize realimages during training and testing (row #2), we still outperformit by 1%.

Original: 4.2 Comparison to State-of-the-art
Revised: 4.2 Comparison to State-of-the-art

Original: [Table]
Revised: [Table]

Original: Moreover, we achieve state-of-the-art results on the threewell-known 3D visual grounding benchmarks, i.e., Nr3D[2], Sr3D [2], and ScanRefer [5].
Revised: Moreover, we achieve state-of-the-art results on the threewell-known 3D visual grounding benchmarks, i.e., Nr3D[2], Sr3D [2], and ScanRefer [5].

Original: By effectively utilizing our 2D synthetic images, LAR outperforms allthe existing methods, which do not use any additionaltraining dataset, by a large margin of 5.0 % and 1.9 %on Nr3D and Sr3D, respectively, depicted in Table 7.
Revised: By effectively utilizing our 2D synthetic images, LAR outperforms allthe existing methods, which do not use any additionaltraining dataset, by a large margin of 5.0 % and 1.9 %on Nr3D and Sr3D, respectively, depicted in Table 7.

Original: Where, LanguageRefer [37] achieves 43.9%, while LARachieves 48.9% on Nr3D. For Sr3D, our model achieves59.35%, which is a 1.9 % increase over TransRefer3D [13].
Revised: Where, LanguageRefer [37] achieves 43.9%, while LARachieves 48.9% on Nr3D. For Sr3D, our model achieves59.35%, which is a 1.9 % increase over TransRefer3D [13].

Original: In addition, for the ScanRefer [2] benchmark, we also achieve state-of-the-art results, by boosting theaccuracy by 2.3 % from the closest counterpart architecture; SAT [52], mentioned in Table 6.
Revised: In addition, for the ScanRefer [2] benchmark, we also achieve state-of-the-art results, by boosting theaccuracy by 2.3 % from the closest counterpart architecture; SAT [52], mentioned in Table 6.

Original: [Table]
Revised: [Table]

Original: We retrained SAT [52] for a fair comparison, as we notice difficulties reproducing and verifying SAT results 1 .
Revised: We retrained SAT [52] for a fair comparison, as we notice difficulties reproducing and verifying SAT results 1 .

Original: Although we do not use an extra input duringtraining as SAT, which uses the original ScanNet 2D images [8], we surpass
Revised: Although we do not use an extra input duringtraining as SAT, which uses the original ScanNet 2D images [8], we surpass

Original: SAT by 1.6 % , 2.7 % , and 2.3 % on Nr3D, Sr3D, and ScanRefer, respectively.
Revised: SAT by 1.6 % , 2.7 % , and 2.3 % on Nr3D, Sr3D, and ScanRefer, respectively.

Original: 4.3 Discussion and Analysis
Revised: 4.3 Discussion and Analysis

Original: First, we will demonstrate our qualitative analysis.
Revised: First, we will demonstrate our qualitative analysis.

Original: Then, we have conductedtwo experiments to probe the robustness of the representations learned by ourproposed module LAR.
Revised: Then, we have conductedtwo experiments to probe the robustness of the representations learned by ourproposed module LAR.

Original: – Qualitative Analysis.
Revised: – Qualitative Analysis.

Original: Figure 5 depicts two corner cases in our projectionmodule (SIG).
Revised: Figure 5 depicts two corner cases in our projectionmodule (SIG).

Original: While assigning the projected 3D points into a 2D grid, morethan one point may share the exact spatial location on the grid, which means only one will be storedin the grid, and the rest of the points will be discarded.
Revised: While assigning the projected 3D points into a 2D grid, morethan one point may share the exact spatial location on the grid, which means only one will be storedin the grid, and the rest of the points will be discarded.

Original: As shown in part (A) in Figure 5, on the leftcolumn, the floor overlaps the other objects due to the conflict between points during the assignationstep.
Revised: As shown in part (A) in Figure 5, on the leftcolumn, the floor overlaps the other objects due to the conflict between points during the assignationstep.

Original: To overcome this behavior, a simple priority rank is given for each point based on its height.
Revised: To overcome this behavior, a simple priority rank is given for each point based on its height.

Original: Part (B) shows another failure case, where the prominent face is falsely localized, which causesplacing the camera outside the room or inside other objects.
Revised: Part (B) shows another failure case, where the prominent face is falsely localized, which causesplacing the camera outside the room or inside other objects.

Original: Therefore, we localize the prominentface relative to the room center to mitigate this effect.
Revised: Therefore, we localize the prominentface relative to the room center to mitigate this effect.

Original: An example of an ambiguous description isshown in Figure 6 part A.
Revised: An example of an ambiguous description isshown in Figure 6 part A.

Original: The model incorrectly predicts "the chair on the right rather than the chairon the left."
Revised: The model incorrectly predicts "the chair on the right rather than the chairon the left."

Original: The model and the human annotations are occasionally confusing to referring to theobjects in the view-dependent description.
Revised: The model and the human annotations are occasionally confusing to referring to theobjects in the view-dependent description.

Original: While successful cases can be seen in Figure 6 part B. Formore qualitative visualization, please refer to Appendix 5.
Revised: While successful cases can be seen in Figure 6 part B. Formore qualitative visualization, please refer to Appendix 5.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: – Evaluation with Ground-Truth Class Labels.
Revised: – Evaluation with Ground-Truth Class Labels.

Original: Table 8 shows anothervariant, which is tailored from our architecture to be compared against SATGT.
Revised: Table 8 shows anothervariant, which is tailored from our architecture to be compared against SATGT.

Original: Where the predicted class labels are replaced by the ground-truth ones.
Revised: Where the predicted class labels are replaced by the ground-truth ones.

Original: Our LAR architecture outperforms SAT-GT by 12%.
Revised: Our LAR architecture outperforms SAT-GT by 12%.

Original: – Zero-shot Cameras Dying Testing.
Revised: – Zero-shot Cameras Dying Testing.

Original: We deliberately drop the cameras duringtesting to verify the robustness of our learned representations against cameravariations.
Revised: We deliberately drop the cameras duringtesting to verify the robustness of our learned representations against cameravariations.

Original: As shown in Figure 4, starting from our best model, which is trainedusing five cameras, the referring accuracy drops to 44.6% then 40.15%, whenwe only use three cameras then one camera during testing, respectively.
Revised: As shown in Figure 4, starting from our best model, which is trainedusing five cameras, the referring accuracy drops to 44.6% then 40.15%, whenwe only use three cameras then one camera during testing, respectively.

Original: Drivenby these results, camera dropout
Revised: Drivenby these results, camera dropout

Original: techniques can be invoked to overcome the performance drop, bydeliberately drop
Revised: techniques can be invoked to overcome the performance drop, bydeliberately drop

Original: a
Revised: a

Original: random camera during
Revised: random camera during

Original: the
Revised: the

Original: training;
Revised: training;

Original: replace its input by a
Revised: replace its input by a

Original: duplicate
Revised: duplicate

Original: view of another
Revised: view of another

Original: camera
Revised: camera

Original: – Noisy Cameras Extrinsic and Intrinsic.
Revised: – Noisy Cameras Extrinsic and Intrinsic.

Original: Due to the significant improvement introduced by utilizingthe camera augmentation, as shown in Table 4, a detailed analysis is conducted to explore therobustness against the camera’s variations.
Revised: Due to the significant improvement introduced by utilizingthe camera augmentation, as shown in Table 4, a detailed analysis is conducted to explore therobustness against the camera’s variations.

Original: Consequently, our 2D synthetic branch learns how to fuseinformation across different cameras in a data-driven way.
Revised: Consequently, our 2D synthetic branch learns how to fuseinformation across different cameras in a data-driven way.

Original: Therefore, we can train the model to berobust to noise camera models, as shown in Table 9.
Revised: Therefore, we can train the model to berobust to noise camera models, as shown in Table 9.

Original: [Table]
Revised: [Table]

Original: Both cameras dying and noisy parameters experiments could
Revised: Both cameras dying and noisy parameters experiments could

Original: [Table]
Revised: [Table]

Original: be thought as irrelevant as LAR utilizes virtual cameras.
Revised: be thought as irrelevant as LAR utilizes virtual cameras.

Original: Thus,no camera dropout nor noise can be introduced.
Revised: Thus,no camera dropout nor noise can be introduced.

Original: However, our
Revised: However, our

Original: SIG module, depicted in Section 3.1, can be utilized in a widerange of applications when these robustness characteristicsagainst such noise are essential for these applications.
Revised: SIG module, depicted in Section 3.1, can be utilized in a widerange of applications when these robustness characteristicsagainst such noise are essential for these applications.

Original: Forinstance, in the autonomous driving world, our SIG module canbe employed to convert dense lidar point clouds into syntheticimages, which will be used to train an arbitrary neural network that tackles specific tasks (e.g., objectsegmentation in cocoon setup [31] [47]).
Revised: Forinstance, in the autonomous driving world, our SIG module canbe employed to convert dense lidar point clouds into syntheticimages, which will be used to train an arbitrary neural network that tackles specific tasks (e.g., objectsegmentation in cocoon setup [31] [47]).

Original: Then, the model will be tested using real cameras instead ofgenerating synthetic imagery input from the lidar.
Revised: Then, the model will be tested using real cameras instead ofgenerating synthetic imagery input from the lidar.

Original: – Limitations.
Revised: – Limitations.

Original: Following the standard setup of Referit3D [2], we assume access to the object proposalsof each 3D scene, similar to SAT [52], LanguageRefer [37], TransRefer3D [13], and 3DVG-Trans[59].
Revised: Following the standard setup of Referit3D [2], we assume access to the object proposalsof each 3D scene, similar to SAT [52], LanguageRefer [37], TransRefer3D [13], and 3DVG-Trans[59].

Original: Therefore, we use the ground-truth proposals in our experiments.
Revised: Therefore, we use the ground-truth proposals in our experiments.

Original: Following this assumption,the 3D grounding problem is reformulated as a classification problem, where the model objective isto classify the referred object among the other distractors correctly.
Revised: Following this assumption,the 3D grounding problem is reformulated as a classification problem, where the model objective isto classify the referred object among the other distractors correctly.

Original: However, LAR is compatiblewith the setting that uses detected-based proposals.
Revised: However, LAR is compatiblewith the setting that uses detected-based proposals.

Original:  Conclusion
Revised:  Conclusion

Original: We propose an efficient 3D grounding architecture that leverages the 2D knowledge to reinforce thelearned 3D representations.
Revised: We propose an efficient 3D grounding architecture that leverages the 2D knowledge to reinforce thelearned 3D representations.

Original: To avoid requiring 2D semantics neither during training nor testing, weintroduce a 2D synthetic images generator, termed SIG.
Revised: To avoid requiring 2D semantics neither during training nor testing, weintroduce a 2D synthetic images generator, termed SIG.

Original: Our SIG module generates high-quality 2Dsynthetic images.
Revised: Our SIG module generates high-quality 2Dsynthetic images.

Original: LAR, 2D Synthetic Semantics Knowledge Distillation for 3D Visual Grounding,consists of two stages, i.e., Visual and Multi-Modal modules.
Revised: LAR, 2D Synthetic Semantics Knowledge Distillation for 3D Visual Grounding,consists of two stages, i.e., Visual and Multi-Modal modules.

Original: LAR preserves the advantages ofdistilling the synthetic 2D knowledge to the 3D stream without requiring any additional information.
Revised: LAR preserves the advantages ofdistilling the synthetic 2D knowledge to the 3D stream without requiring any additional information.

Original: Our experiments show a significant superior performance, where LAR achieves state-of-the-art results across three different 3D grounding benchmarks by outperforming the existing approaches, whichdo not rely on extra training data, by a wide margin, i e., 5.0 % , 1.9 % and 2.3 % on Nr3D, Sr3Dand ScanRefer, respectively.
Revised: Our experiments show a significant superior performance, where LAR achieves state-of-the-art results across three different 3D grounding benchmarks by outperforming the existing approaches, whichdo not rely on extra training data, by a wide margin, i e., 5.0 % , 1.9 % and 2.3 % on Nr3D, Sr3Dand ScanRefer, respectively.

Original: Hence, despite the unfair comparison with SAT, which uses extra 2D images, LAR surpasses SAT by 1.6 % , 2.7 % and 2.3 % on Nr3D, Sr3D and ScanRefer, respectively.
Revised: Hence, despite the unfair comparison with SAT, which uses extra 2D images, LAR surpasses SAT by 1.6 % , 2.7 % and 2.3 % on Nr3D, Sr3D and ScanRefer, respectively.

Original: Furthermore, we verify the robustness of LAR and its generalization ability via zero-shot experiments to intrinsic and extrinsic camera parameters and dying cameras.
Revised: Furthermore, we verify the robustness of LAR and its generalization ability via zero-shot experiments to intrinsic and extrinsic camera parameters and dying cameras.

Original: More qualitative results are attached in the supplementary materials.
Revised: More qualitative results are attached in the supplementary materials.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?

Original: [Yes](b) Did you describe the limitations of your work?
Revised: [Yes](b) Did you describe the limitations of your work?

Original: [Yes] Please refer to Section 4.3(c)
Revised: [Yes] Please refer to Section 4.3(c)

Original: Did you discuss any potential negative societal impacts of your work
Revised: Did you discuss any potential negative societal impacts of your work

Original: N
Revised: N

Original: /A]
Revised: /A]

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?
Revised: them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?
Revised: mental results (either in the supplemental material or as a URL)?

Original: [Yes] Our codes are
Revised: [Yes] Our codes are

Original: attached in the supplemental material combined with a detailed ’readme’ file.
Revised: attached in the supplemental material combined with a detailed ’readme’ file.

Original: Also,
Revised: Also,

Original: refer to Section 3.3 for the implementation details.
Revised: refer to Section 3.3 for the implementation details.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters,
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters,

Original: how they
Revised: how they

Original: were chosen)?
Revised: were chosen)?

Original: [Yes] Our codes are attached in the supplemental material combined
Revised: [Yes] Our codes are attached in the supplemental material combined

Original: with a detailed ’readme’ file.
Revised: with a detailed ’readme’ file.

Original: Also, refer to Section 3.3 for the implementation details.
Revised: Also, refer to Section 3.3 for the implementation details.

Original: (c) Did you report error bars (e
Revised: (c) Did you report error bars (e

Original: g., with respect to the random seed after running experi-
Revised: g., with respect to the random seed after running experi-

Original: ments multiple times)?
Revised: ments multiple times)?

Original: [Yes] Refer to Table 7.
Revised: [Yes] Refer to Table 7.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type

Original: of GPUs, internal cluster, or cloud provider)?
Revised: of GPUs, internal cluster, or cloud provider)?

Original: [Yes] Refer to Section 3.3.
Revised: [Yes] Refer to Section 3.3.

Original: If you are using existing assets (e
Revised: If you are using existing assets (e

Original: g., code, data, models) or curating/releasing new assets...
Revised: g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes]
Revised: [Yes]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [Yes]
Revised: [Yes]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?
Revised: applicable?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A]
Revised: [N/A]

Original: A Appendix
Revised: A Appendix

Original: Our Supplemental Material contains the following sections:
Revised: Our Supplemental Material contains the following sections:

Revised: • Detector-Generated 3D Proposal.

Revised: • Computational Cost.

Original: • Demonstrate our synthetic images generator algorithm (SIG). • More ablations.
Revised: • Demonstrate our synthetic images generator algorithm (SIG).• More ablations.

Original: • Qualitative results.
Revised: • Qualitative results.

Original: • Our source code.
Revised: • Our source code.

Revised: A.1 Detector-Generated 3D Proposal

Revised: Following the standard setup of Referit3D [2], we assume access to the object proposals of each3D scene, similar to SAT [52], LanguageRefer [37], TransRefer3D [13], and 3DVG-Trans [59].

Revised: Therefore, we use the ground-truth proposals in our experiments.

Original: A.
Revised: Following this assumption, the3D grounding problem is reformulated as a classification problem, where the model objective is toclassify the referred object among the other distractors correctly. However, LAR is compatible withthe setting that uses detected-based proposals. To this end, we experimented our LAR module withthe detector-generated 3D proposals using VoteNet detector. As shown in Figure 7, we adapted the

Revised: Scanrefer architecture as follows: 1) 2D Encoder: Integrating a 2D encoder right after the predictedbox proposals, i.e., SIG module and Tiny-ConvNext in our case, and Faster R-CNN [1] in SATcase.

Revised: 2) Fusion Transformer:

Revised: In LAR case, we added our visual transformer only as we aim toemphasize the contribution of our SIG module only.

Revised: While in SAT case, we convert their multi-modaltransformer to vision transformer by excluding the language embedding from it.

Revised: The same 3Dprediction modules and language modules are adopted from Scanrefer, i.e., P++ and Voting module,

Revised: [Table]

Revised: and GloVe and GRU, respectively.

Revised: We train three models: 1) The best variant of Scanrefer. 2)

Revised: SAT-Detection-based architecture.

Revised: 3) LAR-Detection-based architecture.

Revised: We first train the Scanrefer for 50 epochs then freeze their 3D prediction modules and languagemodules for two epochs to train the new blocks added in SAT and LAR detection based architectures.

Revised: Then, we train the whole architecture for two more epochs.

Revised: For more details, please refer to the"Detector-Generated 3D Proposal" section in Appendix A in the revised paper version, where weadded a figure to demonstrate the adaptations done on the ScanRefer architecture.

Revised: As shown in Table 10, the tailored LAR-Detection-based architecture outperforms both the original

Revised: ScanRefer and the SAT-Detection-based architectures.

Revised: Also, the results show that SAT is morevulnerable to the distortions in the detected proposals.

Revised: A.2 Computational Cost

Revised: We compared our complexity against SAT, during both the training and testing.

Revised: As shown in Table11, our training and inference time are 3.6 x and 0.7 x compared to SAT.

Revised: All the reported results aremeasured on single GTX-1080 GPU.

Revised: [Table]

Revised: SAT 47.

Revised: 237 M 81 M 0.317 FPS 5.97 FPS

Revised: LAR 48.

Revised: 118 M 118 M 1.

Revised: [Table]

Revised: A.

Original: Synthetic Images Generator Algorithm
Revised: Synthetic Images Generator Algorithm

Original: Algorithm 1, demonstrates a pseudo-code for our
Revised: Algorithm 1, demonstrates a pseudo-code for our

Original: SIG module thatprojects arbitrary 3D
Revised: SIG module thatprojects arbitrary 3D

Original: Point clouds representing an object to a 2D image.
Revised: Point clouds representing an object to a 2D image.

Original: 11: V f 12: V
Revised: l 9: Cam d up ] 10:

Original: ← V f ⊗
Revised: Build transformation matrix

Original: 13: V up ← 
Revised: 11: V f ← ∥

Original:  V 
Revised: Cam p − O ∥ 12: V r ← ∥ V f ⊗ 0 , 0 , 1] ∥ 13: V up ← ∥ V r ⊗ V f ∥ 14: M ← (cid:20) V V up

Original: ⊗
Revised: 15: x cami ← Dot

Original: 14: M
Revised: 16: x

Original: Dot ← 
Revised: ← Dot

Revised: I x

Revised: 17: x imgi ← x cami x

Revised: 18: return

Original: A.2 Ablations Studies
Revised: A.4 Ablations Studies

Original: A.2.1 Different 2D semantic fusion techniques.
Revised: A.4.1 Different 2D semantic fusion techniques.

Original: Driven by the aforementioned results in Table 2, we use five virtual cameras set in a cocoon setuparound each object as depicted in Figure 1, and the image resolution is 128.
Revised: Driven by the aforementioned results in Table 2, we use five virtual cameras set in a cocoon setuparound each object as depicted in Figure 1, and the image resolution is 128.

Original: In Table 10, we studytwo fusion techniques to combine the features generated from the five views.
Revised: In Table 12, we studytwo fusion techniques to combine the features generated from the five views.

Original: The first method is asimple addition operation, and the second one is a learnable fusion mechanism where a 1D-Conv.is used where its kernel size is set to the number of the views used, i.e., in our case set to five.
Revised: The first method is asimple addition operation, and the second one is a learnable fusion mechanism where a 1D-Conv.is used where its kernel size is set to the number of the views used, i.e., in our case set to five.

Original: Asexpected, the learnable fusion scheme; 1D-Conv., performs better than the none learnable scheme;addition.
Revised: Asexpected, the learnable fusion scheme; 1D-Conv., performs better than the none learnable scheme;addition.

Original: Our 2D variant built on top of Referit3D [2] is used in this ablation, where the 2D stream isonly used.
Revised: Our 2D variant built on top of Referit3D [2] is used in this ablation, where the 2D stream isonly used.

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: A.2.2 Different multi-modal fusion techniques.
Revised: A.4.2 Different multi-modal fusion techniques.

Original: In addition, we study three fusion techniques for the 3D, and 2D multi-modal features, M pc and
Revised: In addition, we study three fusion techniques for the 3D, and 2D multi-modal features, M pc and

Original: M img , respectively.
Revised: M img , respectively.

Original: In this ablation, we only utilize the ROI as the 2D semantic information, set theimage resolution to 32, and set the number of multi-views to five.
Revised: In this ablation, we only utilize the ROI as the 2D semantic information, set theimage resolution to 32, and set the number of multi-views to five.

Original: As shown in Table 11, using atransformer while fusing both multi-modal features achieves the best accuracy.
Revised: As shown in Table 13, using atransformer while fusing both multi-modal features achieves the best accuracy.

Original: Optionally include extra information (complete proofs, additional experiments and plots) in theappendix.
Revised: Optionally include extra information (complete proofs, additional experiments and plots) in theappendix.

Original: This section will often be part of the supplemental material.
Revised: This section will often be part of the supplemental material.

Original: A.3 Quantitative Results
Revised: A.5 Quantitative Results

Original: This section analyzes the qualitative results from the successful and the failure cases on the Nr3ddataset.
Revised: This section analyzes the qualitative results from the successful and the failure cases on the Nr3ddataset.

Original: In addition to Figure 7 that shows successful scenarios and Figure 8 that shows failurescenarios, we attach a short video to demonstrate more scenarios.
Revised: In addition to Figure 8 that shows successful scenarios and Figure 9 that shows failurescenarios, we attach a short video to demonstrate more scenarios.

Original: A.3.1 Successful Scenarios
Revised: A.5.1 Successful Scenarios

Original: LAR achieves top-ranking results on both the Nr3D and Sr3D datasets.
Revised: LAR achieves top-ranking results on both the Nr3D and Sr3D datasets.

Original: The results in Figure 7demonstrate the correct instance localization on the Nr3D datasets only.
Revised: The results in Figure 8demonstrate the correct instance localization on the Nr3D datasets only.

Original: The ground truth boxesare marked green and our model prediction marked in blue.
Revised: The ground truth boxesare marked green and our model prediction marked in blue.

Original: Our analysis shows that LAR correctlyunderstands the global context based on natural language description.
Revised: Our analysis shows that LAR correctlyunderstands the global context based on natural language description.

Original: For example, in Figure 7 (a-f),
Revised: For example, in Figure 8 (a-f),

Original: LAR understands relationships, e.g. (behind it, middle of, furthest from .. ), which proves that ourproposed model understands instance-to-instance and instance-to-background relations.
Revised: LAR understands relationships, e.g. (behind it, middle of, furthest from .. ), which proves that ourproposed model understands instance-to-instance and instance-to-background relations.

Original: As well asunderstanding colors and shapes related to language, LAR also can determine the meaning of termssuch as “green,” Figure 7 (e).
Revised: As well asunderstanding colors and shapes related to language, LAR also can determine the meaning of termssuch as “green,” Figure 8 (e).

Original: A.3.2 Failure Scenarios
Revised: A.5.2 Failure Scenarios

Original: Figure 8 shows representative failure cases of LAR.
Revised: Figure 9 shows representative failure cases of LAR.

Original: Despise the incorrect instance localization LARmanaged to allocate the instance with the same class as the target.
Revised: Despise the incorrect instance localization LARmanaged to allocate the instance with the same class as the target.

Original: As shown in Figure 8 (d, e), weobserved that failure cases require an understanding of "against the wall", ’Facing the windows,"which is determined by camera view dependency.
Revised: As shown in Figure 9 (d, e), weobserved that failure cases require an understanding of "against the wall", ’Facing the windows,"which is determined by camera view dependency.

Original: Despite LAR’s progress in view-dependent andindependent samples, view understanding remains a challenge.
Revised: Despite LAR’s progress in view-dependent andindependent samples, view understanding remains a challenge.

Original: Besides, other state-of-the-art modelssuffer from the same issue.
Revised: Besides, other state-of-the-art modelssuffer from the same issue.

Original: In addition, some LAR failure cases are caused by ambiguous queries.
Revised: In addition, some LAR failure cases are caused by ambiguous queries.

Original: Figure 8 (a,b,c,e, f) shows a complex and compound language description to describe the targetinstance, which in some cases, both models and human annotators mix up similar objects.
Revised: Figure 9 (a,b,c,e, f) shows a complex and compound language description to describe the targetinstance, which in some cases, both models and human annotators mix up similar objects.

Original: To summarize, We show the effect of using synthetic 2D images, which improves the model instanceand context-aware representation.
Revised: To summarize, We show the effect of using synthetic 2D images, which improves the model instanceand context-aware representation.

Original: However, our model has limitations due to ambiguous and complexqueries.
Revised: However, our model has limitations due to ambiguous and complexqueries.

Original: Although some cases remained unsolved, LAR improved progress in the camera viewdependency samples.
Revised: Although some cases remained unsolved, LAR improved progress in the camera viewdependency samples.

Original: Therefore, we believe our model LAR surpasses existing state-of-the-art by asignificant margin.
Revised: Therefore, we believe our model LAR surpasses existing state-of-the-art by asignificant margin.

Original: A.4 Source Code
Revised: A.6 Source Code

Original: our source code is attached in the supplementary material, where it contains the source code for our
Revised: our source code is attached in the supplementary material, where it contains the source code for our

Original: Synthetic Images Generator (SIG), our architecture, training functions, and evaluation functions.
Revised: Synthetic Images Generator (SIG), our architecture, training functions, and evaluation functions.

Original: Formore details regarding our implementation, refer to the Readme file attached with the source code.
Revised: Formore details regarding our implementation, refer to the Readme file attached with the source code.