Original: Look Around and Refer: 2D Synthetic Semantics Knowledge Distillation for 3D Visual Grounding
Revised: Look Around and Refer: 2D Synthetic Semantics Knowledge Distillation for 3D Visual Grounding

Original: Abstract
Revised: Abstract

Original: The 3D visual grounding task has been explored with visual and language streamsto comprehend referential language to identify targeted objects in 3D scenes.
Revised: The 3D visual grounding task has been explored with visual and language streams to comprehend referential language to identify targeted objects in 3D scenes.

Original: However, most existing methods devote the visual stream to capturing the 3D visualclues using off-the-shelf point clouds encoders.
Revised: However, most existing methods devote the visual stream to capturing the 3D visual clues using off-the-shelf point clouds encoders.

Original: The main question we address inthis paper is “can we consolidate the 3D visual stream by 2D clues synthesizedfrom point clouds and efficiently utilize them in training and testing?” .
Revised: The main question we address in this paper is “can we consolidate the 3D visual stream by 2D clues synthesized from point clouds and efficiently utilize them in training and testing?” .

Original: The mainidea is to assist the 3D encoder by incorporating rich 2D object representationswithout requiring extra 2D inputs.
Revised: The main idea is to assist the 3D encoder by incorporating rich 2D object representations without requiring extra 2D inputs.

Original: To this end, we leverage 2D clues, syntheticallygenerated from 3D point clouds, and empirically show their aptitude to boost thequality of the learned visual representations.
Revised: To this end, we leverage 2D clues, synthetically generated from 3D point clouds, and empirically show their aptitude to boost the quality of the learned visual representations.

Original: We validate our approach throughcomprehensive experiments on Nr3D, Sr3D, and ScanRefer datasets, and we showconsistent performance gains compared to the state-of-the-art.
Revised: We validate our approach through comprehensive experiments on Nr3D, Sr3D, and ScanRefer datasets, and we show consistent performance gains compared to the state-of-the-art.

Original: Our proposedmodule, dubbed as Look Around and Refer (LAR), significantly outperforms thestate-of-the-art 3D visual grounding techniques on these three benchmarks.
Revised: Our proposed module, dubbed as Look Around and Refer (LAR), significantly outperforms the state-of-the-art 3D visual grounding techniques on these three benchmarks.

Original: Ourcode will be made publicly available.
Revised: The code is available at https://github.com/eslambakr/LAR-Look-Around-and-Refer.

Original:  Introduction
Revised:  Introduction

Original: Visiolinguistic modeling has attracted much attentionover the past decade due to impact on various areas,including visual question answering [3, 36], 3D scenecaptioning [7], image-text matching [26, 44], visualrelation detection [25] and scenegraph generation[42].
Revised: Visiolinguistic modeling has attracted much attention over the past decade due to impact on various areas, including visual question answering [3, 36], 3D scene captioning [7], image-text matching [26, 44], visual relation detection [25] and scenegraph generation [42].

Original: Consequently, understanding natural languageutterances and grounding them to real-world scenarios is attractive due to their importance in many realapplications, such as robot navigation and interactionin indoor environment [22] [40].
Revised: Consequently, understanding natural language utterances and grounding them to real-world scenarios is attractive due to their importance in many real applications, such as robot navigation and interaction in indoor environment [22] [40].

Original: Visual Grounding (VG) has been explored in recentyears in the context of 2D [16, 30, 46, 55, 57], extending object detection to a more complex task.
Revised: Visual Grounding (VG) has been explored in recent years in the context of 2D [16, 30, 46, 55, 57], extending object detection to a more complex task.

Original: However, for real robots to intelligently ground a givenutterance on a natural scene in a challenging environment like a crowded indoor or outdoor scene, 3D representations are needed to understand the surroundingenvironment better.
Revised: However, for real robots to intelligently ground a given utterance on a natural scene in a challenging environment like a crowded indoor or outdoor scene, 3D representations are needed to understand the surrounding environment better.

Original: Therefore, 3D visual groundingtask have been proposed by ReferIt3D [2] and ScanRefer [5].
Revised: Therefore, 3D visual grounding task have been proposed by ReferIt3D [2] and ScanRefer [5].

Original: The ReferIt3D [2] introduces two datasets 
Revised: The ReferIt3D [2] introduces two datasets containing natural and synthetic language utterances, namely Nr3D and Sr3D, respectively, which

Original: [Table]
Revised: [Table]

Original: containing natural and synthetic language utterances, namely Nr3D and Sr3D, respectively, whichaims to determine the referent object from pre-defined objects set for each scene.
Revised: aims to determine the referent object from pre-defined objects set for each scene.

Original: In contrast, ScanRefer [5] aims to predict 3D bounding boxes for the referent object based on a given languagedescription and 3D point clouds.
Revised: In contrast, ScanRefer [5] aims to predict 3D bounding boxes for the referent object based on a given language description and 3D point clouds.

Original: Both ScanRefer [5] and ReferIt3D [2] are built on the ScanNetdataset [8].
Revised: Both ScanRefer [5] and ReferIt3D [2] are built on the ScanNet dataset [8].

Original: The critical difference, which made ReferIt3D [2] more challenging, is that in each scene,several object instances belong to the same fine-grained category as the referent, namely distractors.
Revised: The critical difference, which made ReferIt3D [2] more challenging, is that in each scene, several object instances belong to the same fine-grained category as the referent, namely distractors [Table]

Original: Due to the sparsity and disorganization of the 3D point clouds, SAT [52] explores utilizing the 2Dimages provided by the ScanNet dataset [8] during training to assist point-cloud-language jointrepresentation learning.
Revised: Due to the sparsity and disorganization of the 3D point clouds [Table] SAT [52] explores utilizing the 2D images provided by the ScanNet dataset [8] during training to assist point-cloud-language joint representation learning.

Original: SAT [52] proposes three variants; 1) The baseline, non-SAT, relies only onthe 3D point clouds while tackling the grounding task.
Revised: SAT [52] proposes three variants; 1) The baseline, non-SAT, relies only on the 3D point clouds while tackling the grounding task.

Original: Utilizing 2D semantics during the trainingphase only and masking them while inference, we termed it SAT.
Revised: Utilizing 2D semantics during the training phase only and masking them while inference, we termed it SAT.

Original: 3) Utilizing the 2D semantics inboth phases, i.e., training and testing, we termed it SAT-GT.
Revised: Utilizing the 2D semantics in both phases, i.e., training and testing, we termed it SAT-GT.

Original: However, requiring extra 2D inputs intraining or inference limits potential application scenarios.
Revised: However, requiring extra 2D inputs in training or inference limits potential application scenarios.

Original: To tackle this drawback, we hypothesizedthat incorporating synthetic 2D semantics will consolidate the 3D learned representation while notrequiring any additional info.
Revised: To tackle this drawback, we hypothesized that incorporating synthetic 2D semantics will consolidate the 3D learned representation while not requiring any additional info.

Original: To this end, we propose a novel 2D Synthetic Images Generator (SIG)that projects the 3D point clouds into multi-view 2D images, as shown in Figure 1.
Revised: To this end, we propose a novel 2D Synthetic Images Generator (SIG) that projects the 3D point clouds into multi-view 2D images, as shown in Figure 1.

Original: To the bestof our knowledge, LAR is the first method that augment learning of 3D tasks with synthetic 2Dsemantics.
Revised: To the best of our knowledge, LAR is the first method that augment learning of 3D tasks with synthetic 2D semantics.

Original: In addition, we propose a novel multi-modal transformer-based architecture.
Revised: In addition, we propose a novel multi-modal transformer-based architecture.

Original: Our proposedarchitecture, LAR, on Nr3D outperforms SAT [52] by boosting the accuracy by 1.6 % while exceedingall existing approaches that did not use any additional information than the 3D point clouds by awider margin, i.e., 5.0 % to the closest one.
Revised: Our proposed architecture, LAR, on Nr3D outperforms SAT [52] by boosting the accuracy by 1.6 % while exceeding all existing approaches that did not use any additional information than the 3D point clouds by a wider margin, i.e., 5.0 % to the closest one.

Original: Also, LAR architecture achieves state-of-the-art resultson Sr3D and ScanRefer benchmarks, 59.35% and 54.6%, respectively.
Revised: Also, LAR architecture achieves state-of-the-art results on Sr3D and ScanRefer benchmarks, 59.35% and 54.6%, respectively.

Original: Another variant is tailoredfrom our architecture to be compared against SAT-GT, where geometry information and ground-truthclass labels are concatenated to the 2D images; synthetic in our case and original images in SAT case.
Revised: Another variant is tailored from our architecture to be compared against SAT-GT, where geometry information and ground-truth class labels are concatenated to the 2D images; synthetic in our case and original images in SAT case.

Original: Our LAR architecture outperforms SAT-GT by 11.7 % , where we achieved 62% while SAT achieves50.3%.
Revised: Our LAR architecture outperforms SAT-GT by 11.7 % , where we achieved 62% while SAT achieves 50.3%.

Original: Our contributions are summarized as follows:
Revised: Our contributions are summarized as follows:

Original: • We propose a novel End-to-End multi-modal transformer-based architecture, LAR, for the3D visual grounding task that relies only on 3D point clouds, which exploits the synthetic2D images generated by our SIG module.• We verify the effectiveness of our LAR architecture throughout extensive experiments with various benchmarks, which show that utilizing our 2D synthetic semantics significantly enhances the results.
Revised: • We propose a novel End-to-End multi-modal transformer-based architecture, LAR, for the 3D visual grounding task that relies only on 3D point clouds, which exploits the synthetic 2D images generated by our SIG module. • We verify the effectiveness of our LAR architecture throughout extensive experiments with various benchmarks, which show that utilizing our 2D synthetic semantics significantly enhances the results.

Original: • Through detailed analysis, we examine our method’s internal behavior and validity.
Revised: • Through detailed analysis, we examine our method’s internal behavior and validity.

Original:  Related Work
Revised:  Related Work

Original: Our novel architecture draws success from several areas, including 2D visual grounding, 3D visualgrounding, and 2D semantics in 3D tasks.
Revised: Our novel architecture draws success from several areas, including 2D visual grounding, 3D visual grounding, and 2D semantics in 3D tasks.

Original: – 2D visual grounding.
Revised: – 2D visual grounding.

Original: 2D visual grounding is locating an object based on a natural languagedescription.
Revised: 2D visual grounding is locating an object based on a natural language description.

Original: Flickr30k [32] and ReferItGame [20] generated referring expressions for a particularobject in an image used to identify which object is being referred to.
Revised: Flickr30k ] and ReferItGame [ ] generated referring expressions for a particular object in an image used to identify which object is being referred to.

Original: Prior works [56, 27, 29]have attempted to localize these referred objects based on the given expressions by focusing on thebounding-box level.
Revised: Prior works [56, 27, 29] have attempted to localize these referred objects based on the given expressions by focusing on the bounding-box level.

Original: In contrast, [53, 10] focus on the pixels level.
Revised: In contrast, [53, 10] focus on the pixels level.

Original: Another 2D visual groundingtaxonomy is the one-stage, also known as a single-stage framework, versus the two-stage framework.
Revised: Another 2D visual grounding taxonomy is the one-stage, also known as a single-stage framework, versus the two-stage framework.

Original: The single-stage methods [38, 49, 50], fuse the linguistic query with each pixel/patches of theimage, and generate a set of possible bounding box candidates.
Revised: The single-stage methods [38, 49, 50], fuse the linguistic query with each pixel/patches of the image, and generate a set of possible bounding box candidates.

Original: In contrast, the two-stage framework[45, 54, 56] uses the visual image semantics to generate object proposals.
Revised: In contrast, the two-stage framework [45, 54, 56] uses the visual image semantics to generate object proposals.

Original: Afterward, the grounding prediction is selected by comparing each proposal to the language query.
Revised: Afterward, the grounding prediction is selected by comparing each proposal to the language query.

Original: Consequently, 2D visualgrounding methods can guide learning 3D relationships.
Revised: Consequently, 2D visual we grounding methods can guide learning 3D relationships.

Original: Driven by this, we explore using 2D syntheticsemantics to consolidate the 3D visual stream.
Revised: Driven by this, we explore using 2D synthetic semantics to consolidate the 3D visual stream.

Original: – 3D visual grounding.
Revised: – 3D visual grounding.

Original: Table 1 summarizes the existing 3D visual grounding models whether aground truth object proposal is used ora pre-trained detector is utilized to generate the proposals.
Revised: Table 1 summarizes the existing 3D visual grounding models whether a ground truth object proposal is used or

Revised: a pre-trained detector is utilized

Revised: to generate the proposals.

Original: Also, it summarizes whether extra info is used alongside the object’s 3D point cloud or not.
Revised: Also, it summarizes whether extra info is used alongside the object’s 3D point cloud or not.

Original: 3Dvisual grounding field has undergone extensive research and made considerable progress over thelast few years [2, 5, 12, 17, 59, 1, 37, 13, 58, 19, 52, 6, 19].
Revised: 3D visual grounding field has undergone extensive research and made considerable progress over the last few years [2, 5, 12, 17, 59, 1, 37, 13, 58, 19, 52, 6, 19].

Original: A 3D visual grounding task useslanguage instruction along with a 3D scene [8] as input in order to estimate a bounding box thatallocates the most relevant object to the provided expression.
Revised: A 3D visual grounding task uses language instruction along with a 3D scene [8] as input in order to estimate a bounding box that allocates the most relevant object to the provided expression.

Original: However, the literature offers severalextensions, such as using auxiliary input to improve localizing 3D objects [52].
Revised: However, the literature offers several extensions, such as using auxiliary input to improve localizing 3D objects [52].

Original: Previous work[2, 12, 17, 59, 1, 13, 58, 19], focuses on improving the language and 3D scene fusion by adopting atwo-stage framework.
Revised: Previous work [2, 12, 17, 59, 1, 13, 58, 19], focuses on improving the language and 3D scene fusion by adopting a two-stage framework.

Original: The first stage outputs 3D object proposals given a 3D point cloud representinga scene.
Revised: The first stage outputs 3D object proposals given a 3D point cloud representing a scene.

Original: Secondly, visual and linguistic features are fused to compute a confidence score for each3D object proposal.
Revised: Secondly, visual and linguistic features are fused to compute a confidence score for each 3D object proposal.

Original: The first stage could be skipped by directly using the ground-truth proposalsinstead of predicting them [52] [13] [2].
Revised: The first stage could be skipped by directly using the ground-truth proposals instead of predicting them [52] [13] [2].

Original: In contrast, [6, 19, 59] do not assume the access to theground-truth objects and use an object detector to extract the proposals.
Revised: In contrast, [6, 19, 59] do not assume the access to the ground-truth objects and use an object detector to extract the proposals.

Original: Intuitively, if the detectorfails, the referring performance will significantly decrease.
Revised: Intuitively, if the detector fails, the referring performance will significantly decrease.

Original: Therefore, we focus on the main trainingobjective related to the grounding task, which is learning 3D visual-language joint representations byassuming access to the ground-truth boxes.
Revised: Therefore, we focus on the main training objective related to the grounding task, which is learning 3D visual-language joint representations by assuming access to the ground-truth boxes.

Original: Although we are assuming access to the ground-truthboxes, LAR is compatible with the setting that utilizes detector-generated proposals.
Revised: Although we are assuming access to the ground-truth boxes, LAR is compatible with the setting that utilizes detector-generated proposals.

Original: – 2D semantics in 3D tasks.
Revised: – 2D semantics in 3D tasks.

Original: 2D semantics contains useful information that can be used to detect 3D objects.
Revised: 2D semantics contains useful information that can be used to detect 3D objects.

Original: The representations of visual scenes consist of point clouds, meshes, or volumes,while 2D images consist of pixel grids.
Revised: The representations of visual scenes consist of point clouds, meshes, or volumes, while 2D images consist of pixel grids.

Original: Combining the best of two worlds can enhance 3D object localization.
Revised: Combining the best of two worlds can enhance 3D object localization.

Original: Researchers have attempted to solve the 3D detection problem by adopting different ways of representing RGB-D data [34, 48, 23, 15, 41, 24].
Revised: Researchers have attempted to solve the 3D detection problem by adopting different ways of representing RGB-D data [34, 48, 23, 15, 41, 24].

Original: By either projecting the 2D image on the 3D scene [34, 48, 24] or fuse both 2D features with the 3D [23, 15, 43, 41], [33].
Revised: By either projecting the 2D image on the 3D scene [34, 48, 24] or fuse both 2D features with the 3D [23, 15, 43, 41], [33].

Original: SAT [52] leveraged 2D semantics to perform the 3D VG task.
Revised: SAT [52] leveraged 2D semantics to perform the 3D VG task.

Original: However, requiring extra 2D inputs limits potential application scenarios.
Revised: However, requiring extra 2D inputs limits potential application scenarios.

Original: In contrast, in this work, we generate synthetic 2D images during the training phase, which is also used in the inference phase.
Revised: In contrast, in this work, we generate synthetic 2D images during the training phase, which is also used in the inference phase.

Original: The synthetic images enable us to consolidate the 3D visual stream by 2D semantics without requiring extra 2D inputs.
Revised: The synthetic images enable us to consolidate the 3D visual stream by 2D semantics without requiring extra 2D inputs.

Original:  Look Around and Refer (LAR)
Revised:  Look Around and Refer (LAR)

Original: LAR aims to enhance the visual module, which in turn enhances the final objective,i.e., referentialaccuracy.
Revised: LAR aims to enhance the visual module, which in turn enhances the final objective,i.e., referential accuracy.

Original: By incorporating the synthetic 2D semantics alongside the 3D point clouds, LAR inherentlycaptures the correlations between the 2D and the 3D streams.
Revised: By incorporating the synthetic 2D semantics alongside the 3D point clouds, LAR inherently captures the correlations between the 2D and the 3D streams.

Original: To design a general frameworkto fit in any 3D application, when the input is 3D point clouds, the need for extra information,such as 2D semantics, should be bypassed.
Revised: To design a general framework to fit in any 3D application, when the input is 3D point clouds, the need for extra information, such as 2D semantics, should be bypassed.

Original: To this end, a 2D synthetic images generator, namely
Revised: To this end, a 2D synthetic images generator, namely

Original: SIG, is introduced, which is discussed in Section 3.1.
Revised: SIG, is introduced, which is discussed in Section 3.1.

Original: Then, we demonstrate how to learn robustrepresentations by aligning the generated 2D semantics with the 3D ones by incorporating our SIGmodule into our proposed architecture, depicted in Section 3.2.
Revised: Then, we demonstrate how to learn robust representations by aligning the generated 2D semantics with the 3D ones by incorporating our SIG module into our proposed architecture, depicted in Section 3.2.

Revised: The 2D synthetic images are first extracted by SIG then processed by a shared ConvNext backbone.

Revised: Simultaneously, language descriptions are transformed into tokens and embedded into a feature vector.

Revised: Then, the Multi-Modal Module fuses the output of the Visual Module separately by two Transformers.

Revised: Finally, the output of the Multi-Modal Transformers is processed by a Fusion Transformers.

Original: 3.1 Look Around: 2D Synthetic Images Generator
Revised: 3.1 Look Around: 2D Synthetic Images Generator

Original: LAR exploits synthetic 2D semantics to reinforce the learned 3D representations.
Revised: LAR exploits synthetic 2D semantics to reinforce the learned 3D representations.

Original: Consequently, thissection will demonstrate our 2D Synthetic Images Generator (SIG) module that projects arbitrary 3D
Revised: Consequently, this section will demonstrate our 2D Synthetic Images Generator (SIG) module that projects arbitrary 3D

Original: Point clouds representing an object to a 2D image.
Revised: Point clouds representing an object to a 2D image.

Original: First, we will show the essential operations of ourmodule on a single image, then show how we extend it to fulfill a multi-view setup.
Revised: First, we will show the essential operations of our module on a single image, then show how we extend it to fulfill a multi-view setup.

Original: A 3D scene, S ∈ R N × 6 , is represented by N points with spatial and color information, i.e., XYZ,and RGB, respectively.
Revised: A 3D scene, S ∈ R N × 6 , is represented by N points with spatial and color information, i.e., XYZ, and RGB, respectively.

Original: Following the previous works [2] [1] [37] [52], the object proposals x pci foreach scene are known, either generated by one of the off-the-shelf 3D object detectors or manuallyannotated.
Revised: Following the previous works [2] [1] [37] [52], the object proposals x pci for each scene are known, either generated by one of the off-the-shelf 3D object detectors or manually annotated.

Original: The object proposal x pci ∈ R N ′ × 6 is a subset of the whole scene, where N ′ < N and N ′represents the number of points that represent the object i .
Revised: The object proposal x pci ∈ R N ′ × 6 is a subset of the whole scene, where N ′ < N and N ′ represents the number of points that represent the object i .

Original: Given a scene S and an object proposal x pci , our SIG module generates a 2D synthetic image x imgirepresents the object i .
Revised: Given a scene S and an object proposal x pci , our SIG module generates a 2D synthetic image x imgi represents the object i .

Original: The main idea is to place a virtual camera in front of the fed object proposal,then project its points x pci into the 2D plane to generate the corresponding 2D image x imgi .
Revised: The main idea is to place a virtual camera in front of the fed object proposal, then project its points x pci into the 2D plane to generate the corresponding 2D image x imgi .

Original: Thecamera’s location should be set carefully to generate a decent image representation for an object i .
Revised: The camera’s location should be set carefully to generate a decent image representation for an object i .

Original: Asshown in part (A) in Figure 2, after excluding the upper and lower faces from the fed object proposal,we get the center of the four remaining faces, then calculate the distance between each face center O li ,and the center of the scene O s , where l ranging from 0 to 3.
Revised: As shown in part (A) in Figure 2, after excluding the upper and lower faces from the fed object proposal, we get the center of the four remaining faces, then calculate the distance between each face center O li , and the center of the scene O s , where l ranging from 0 to 3.

Original: The nearest face l ′ from the scene center
Revised: The nearest face l ′ from the scene center

Original: O s is the prominent one, colored with blue in Figure 2, which guarantees that we look at the objecti from inside the room.
Revised: O s is the prominent one, colored with blue in Figure 2, which guarantees that we look at the object i from inside the room.

Original: The camera is located at a distance d f from the prominent face l ′ and at adistance d up from the room’s floor; thus, the camera position is interpreted as Cam p =
Revised: The camera is located at a distance d f from the prominent face l ′ and at a distance d up from the room’s floor; thus, the camera position is interpreted as Cam p =

Original: [ d fx , d fy , d up ] ,as shown in part (B) in Figure 2.
Revised: [ d fx , d fy , d up ] , as shown in part (B) in Figure 2.

Original: The orientation of the camera frame is defined as follows.
Revised: The orientation of the camera frame is defined as follows.

Original: The z axisis aligned with the principal axis of the camera, and two other orthogonal directions (x, y) align withthe corresponding axes of the image plane.
Revised: The z axis is aligned with the principal axis of the camera, and two other orthogonal directions (x, y) align with the corresponding axes of the image plane.

Original: Failure cases are shown in Figure 5, when the camera islocated outside the room.
Revised: Failure cases are shown in Figure 5, when the camera is located outside the room.

Original: To project the points x pci , representing object i , from the scene coordinates to the camera coordinates,a transformation matrix M ∈ R 4 × 4 needs to be defined.
Revised: To project the points x pci , representing object i , from the scene coordinates to the camera coordinates, a transformation matrix M ∈ R 4 × 4 needs to be defined.

Original: Given the camera position Cam p and the center of the prominent face O l ′ , the transformation matrix M can be interpreted as, (cid:20) V r V up V
Revised: Given the camera position Cam p and the center of the prominent face O l ′ , the transformation matrix M can be interpreted as, (cid:20) V r V up [Table]

Original: f Cam p (cid:21)
Revised: f Cam p [Equation]

Original: Where, the camera direction is defined by three vectors; V f , V r , and V up .
Revised: Where, the camera direction is defined by three vectors; V f , V r , and V up .

Original: We define our virtualcamera using the Unified Camera Model (UCM), which has five parameters I =
Revised: We define our virtual camera using the Unified Camera Model (UCM), which has five parameters I =

Original: [ γ x , γ y , c x , c y , ζ ]
Revised: [ γ x , γ y , c x , c y , ζ ]

Original: Following the UCM, projection is defined as follows:π ( x pci , I ) = (cid:20) γ [Equation]
Revised: Following the UCM, projection is defined as follows:

Original:  x[Equation] + [Equation]
Revised: π ( x

Original: γ

Original: [Equation]

Original: +

Original: [Equation]

Original: (cid:21) + (cid:20) c

Original: [Equation]
Revised: [Equation]

Original: c

Original: [Equation]

Original: (cid:21) ,

Original: Afterwards, the projected 3D point clouds are assigned to a 2D grid map.
Revised: Afterwards, the projected 3D point clouds are assigned to a 2D grid map.

Original: To mitigate the overfittingeffect, we randomly extend the region of interest by ϵ as demonstrated in part (C) at Figure 2.
Revised: To mitigate the overfitting effect, we randomly extend the region of interest by ϵ as demonstrated in part (C) at Figure 2.

Original: Point cloud differs from 2D images, where every pixel contains spatial and texture details.
Revised: Point cloud differs from 2D images, where every pixel contains spatial and texture details.

Original: Moreover,the object is represented in a view-invariant way, unlike the images that are view-dependent.
Revised: Moreover, the object is represented in a view-invariant way, unlike the images that are view-dependent.

Original: To mitigate this drawback in the image representation, a multi-view setup is constructed as shown in
Revised: To mitigate this drawback in the image representation, a multi-view setup is constructed as shown in

Original: Figure 1.
Revised: Figure 1.

Original: Where, v cameras are mounted in a circle around the prominent face l ′ .
Revised: Where, v cameras are mounted in a circle around the prominent face l ′ .

Original: Algorithm 1,attached in the supplemental materials, demonstrates a pseudo-code for our SIG module.
Revised: Algorithm 1, attached in the supplemental materials, demonstrates a pseudo-code for our SIG module.

Original: 3.2 Refer: Visiolinguistic Transformer Architecture
Revised: 3.2 Refer: Visiolinguistic Transformer Architecture

Original: We present our approach, depicted in Figure 3, for learning multi-modal representations of a 3Dscene associated with language instruction.
Revised: We present our approach, depicted in Figure 3, for learning multi-modal representations of a 3D scene associated with language instruction.

Original: First, the proposed visual module enriches the 3Drepresentations by utilizing the 2D synthetic images generator, namely the SIG module, described in
Revised: First, the proposed visual module enriches the 3D representations by utilizing the 2D synthetic images generator, namely the SIG module, described in

Original: Section 3.1.
Revised: Section 3.1.

Original: Thanks to it, we do not require access to any imagery sensor during training or testing.
Revised: Thanks to it, we do not require access to any imagery sensor during training or testing.

Original: Then, the multi-modal module learns joint representations targeted to the final task, i.e., referring toan object in the scene based on the language instruction.
Revised: Then, the multi-modal module learns joint representations targeted to the final task, i.e., referring to an object in the scene based on the language instruction.

Original: – Visual Module.
Revised: – Visual Module.

Original: Givena
Revised: Given

Original: 3D

Original: scene

Original:  S ∈ R N × 6 combined with agnosticmultiple 3D
Revised: a 3D scene S ∈ R N × 6 combined with agnostic

Original: object
Revised: multiple

Original: proposalsx pci ∈ R N ′ × 6 , we aim to classify each object proposal by predicting the 
Revised: 3D object proposals x pci ∈ R N ′ × 6 , we aim to classify each object proposal by predicting the semantic labels z .

Original: semantic

Original: labels

Original: z . First,for each object i in the scene S , we generate multi view synthetic 2D images x imgi ∈ R v × H × 
Revised: First, for each object i in the scene S , we generate multi-view synthetic 2D images x imgi ∈ R v × H × W ×from the 3D object’s point cloud x pc i , where v is the number of views for each object, H and W are the spatial dimensions for the generated images.

Original: W

Original: × 3from the

Original: 3D object’s

Original: point

Original: cloud x pci

Original: , where

Original: v

Original: is the number of views for each object, H and Ware the spatial dimensions for the generated images.

Original: Second, we employ PointNet++ [35] as a 3Dencoder, producing 3D features E pc i ∈ R 1 × dim pc .
Revised: Second, we employ PointNet++ [35] as a 3D encoder, producing 3D features E pci ∈ R 1 × dim pc .

Original: Hence, the Tiny-ConvNext [28] is adopted as a2D encoder, that produce y imgi ∈ R v × H ′ × W ′ × dim img .
Revised: Hence, the Tiny-ConvNext [28] is adopted as a 2D encoder, that produce y imgi ∈ R v × H ′ × W ′ × dim img .

Original: Where H ′ and W ′ represent a coarse spatialgrid of spatial features, each represented by a dim img .
Revised: Where H ′ and W ′ represent a coarse spatial grid of spatial features, each represented by a dim img .

Original: Then, the spatial dimensions are squeezedproducing o imgi ∈ R v × dim img .
Revised: Then, the spatial dimensions are squeezed producing o imgi ∈ R v × dim img .

Original: Since our 2D encoder encodes multiple views v for the same objecti , a fusing mechanism is needed to fuse the views’ features.
Revised: Since our 2D encoder encodes multiple views v for the same object i , a fusing mechanism is needed to fuse the views’ features.

Original: Various possibilities are discussed tosettle down on the best fusion mechanism.
Revised: Various possibilities are discussed to settle down on the best fusion mechanism.

Original: We can interpret this as a fully connected layer to fullycapture the views and cross-channel interactions.
Revised: We can interpret this as a fully connected layer to fully capture the views and cross-channel interactions.

Original: In contrast, to learn the views’ interactions andneglect the cross-channel relations, a 1D-Conv. could be utilized to avoid involving tremendousparameters as in the fully connected case.
Revised: In contrast, to learn the views’ interactions and neglect the cross-channel relations, a 1D-Conv. could be utilized to avoid involving tremendous parameters as in the fully connected case.

Original: Using the 1D-Conv.
Revised: Using the 1D-Conv.

Original: with k size = v as a fusion layerproduce E imgi ∈ R 1 × dim img , where k size is the kernel size.
Revised: with k size = v as a fusion layer produce E imgi ∈ R 1 × dim img , where k size is the kernel size.

Original: The fused 2D features E imgi and the3D features E pci are concatenated and fed to a visual transformer, composed of several stackedtransformer layers.
Revised: The fused 2D features E imgi and the 3D features E pci are concatenated and fed to a visual transformer, composed of several stacked transformer layers.

Original: The visual transformer is utilized to capture the cross-correlation between the 2Dand the 3D features outputting refined features, V pc and V img , guided by three weighted auxiliarylosses, i.e., alignment loss between V pc and V img , and two visual classification losses.
Revised: The visual transformer is utilized to capture the cross-correlation between the 2D and the 3D features outputting refined features, V pc and V img , guided by three weighted auxiliary losses, i.e., alignment loss between V pc and V img , and two visual classification losses.

Original: – 2D semantics.
Revised: 2D semantics.

Original: Inspired by SAT [52], we explore different 2D semantic information types combinations.
Revised: Inspired by SAT [52], we explore different 2D semantic information types combinations.

Original: Where we formulate our 2D semantics as follows:
Revised: Where we formulate our 2D semantics as follows:

Original: [Equation]
Revised: [Equation]

Original: where, z ′ i is the predicted class labels, W 1 are learned projection matrices, LN is layer normalization,and ϕ indicates concatenation.
Revised: where, z ′ i is the predicted class labels, W 1 are learned projection matrices, LN is layer normalization, and ϕ indicates concatenation.

Original: We encode each object individually using different semantic information types, i.e., visual semantics E imgi , and geometry semantics x geoi ∈ R 1 × 30 .
Revised: We encode each object individually using different semantic information types, i.e., visual semantics E img i , and geometry semantics x geo i ∈ R 1 × 30 .

Original: The geometrysemantics, x geoi represents the bounding box coordinates of object i in the scene space and encodethe virtual camera parameters, i.e., intrinsic and extrinsic parameters, used while generating the 2Dimage x img i .
Revised: The geometry semantics, x geoi represents the bounding box coordinates of object i in the scene space and encode the virtual camera parameters, i.e., intrinsic and extrinsic parameters, used while generating the 2D image x imgi .

Original: Thus, it can be interpreted as a positional encoding for the vision transformer, where theinteraction among objects is captured.
Revised: Thus, it can be interpreted as a positional encoding for the vision transformer, where the interaction among objects is captured.

Original: In contrast, SAT [52] utilizes a ground-truth 2D semantics,where the z ′ i denotes the manually annotated class labels, and the E imgi represents the embeddinggenerated from real 2D images captured by imagery sensor 
Revised: In contrast, SAT [52] utilizes a ground-truth 2D semantics, where the z ′ i denotes the manually annotated class labels, and the E imgi represents the embedding generated from real 2D images captured by imagery sensor [8].

Original: – Multi-Modal Module.
Revised: – Multi-Modal Module.

Original: Using a pre-trained BERT model [11] followed by a language classification 
Revised: Using a pre-trained BERT model [11] followed by a language classification head adopted from Referit3D

Original: head adopted from Referit3D [2], we embed words query Q .
Revised: [2], we embed words query Q .

Original: After embedding each of the visual features using the aforementioned visual module, we fuse the input modalities Q , V pc , V img simultaneously by using two different transformers; 3D Multi-Modal Transformer and 2D MultiModal Transformer.
Revised: After embedding each of the visual features using the aforementioned visual module, we fuse the input modalities Q , V pc , V img simultaneously by using two different transformers; 3D Multi-Modal Transformer and 2D MultiModal Transformer.

Original: In 3D Multi-Modal Transformer, we jointly encode the two input modalitiesV pc and Q to understand better the referring expression that captures co-occurrence relationships between different objects in the scene.
Revised: In 3D Multi-Modal Transformer, we jointly encode the two input modalities V pc and Q to understand better the referring expression that captures co-occurrence relationships between different objects in the scene.

Original: Meanwhile the same words query Q are encoded with V img to enhance context-aware representation.
Revised: Meanwhile, the same words query Q are encoded with V img to enhance context-aware representation.

Original: A separate 2D and 3D referring heads are adopted from
Revised: A separate 2D and 3D referring heads are adopted from

Original: Referit3D which are auxiliary tasks.
Revised: Referit3D, which are auxiliary tasks.

Original: Finally

Original: the output of the multi-

Original: modal

Original: transformers is

Original: processed

Original: by

Original: a

Original:  Fusion Transformer that chooses the referred object from the fused features F pc and F img 
Revised: Finally, the output of the multi-modal transformers is processed by a Fusion Transformer that chooses the referred object from the fused features, F pc and F img .

Original: – Losses.
Revised: 3.3 Losses

Original: Our architecture, termed LAR, consists of two visual streams, i.e., 3D and 2D, in additionto one language stream.
Revised: Our architecture, termed LAR, consists of two visual streams, i.e., 3D and 2D, in addition to one language stream.

Original: To refer to the object correctly guided by the input utterance, we must first classify the objects accurately.
Revised: To refer to the object correctly guided by the input utterance, we must first classify the objects accurately.

Original: Thus, two auxiliary losses are employed to optimize the visual module;3D classification loss L 3 Dcls and 2D classification loss L 2 Dcls .
Revised: Thus, two auxiliary losses are employed to optimize the visual module; 3D classification loss L 3 Dcls and 2D classification loss L 2 Dcls .

Original: In addition, We adopted Referit3D [2]auxiliary loss for language classification loss L langcls to cooperate in the visual grounding task by recognizing the referred object class according to the description.
Revised: In addition, We adopted Referit3D [2] auxiliary loss for language classification loss L langcls to cooperate in the visual grounding task by recognizing the referred object class according to the description.

Original: Hence, two auxiliary grounding losses( L 3 DEref and L 2 DEref , termed as early referring losses, are exploited before the fusion transformer.
Revised: Hence, two auxiliary grounding losses ( L 3 DEref and L 2 DEref , termed as early referring losses, are exploited before the fusion transformer.

Original: As aresult of these early referring losses, the model learns the correlation between language descriptionand the visual features separately, which enhances the performance.
Revised: As a result of these early referring losses, the model learns the correlation between language description and the visual features separately, which enhances the performance.

Original: Additionally, we adopted objectcorrespondence loss L cor from SAT to encourage the 3D model to distill the knowledge from the 2Dstream.
Revised: Additionally, we adopted object correspondence loss L cor from SAT to encourage the 3D model to distill the knowledge from the 2D stream.

Original: Finally, two grounding losses, i.e., L 3 Dref and L 2 Dref were added after the fusion transformer topredict the scores of each proposal.
Revised: Finally, two grounding losses, i.e., L 3 Dref and L 2 Dref were added after the fusion transformer to predict the scores of each proposal.

Original: We optimize the whole model in an End-to-End manner, unlike
Revised: We optimize the whole model in an End-to-End manner, unlike

Original: SAT which learns the 2D features E imgi separately, with the following loss function:
Revised: SAT which learns the 2D features E imgi separately, with the following loss function:

Original: [Equation]
Revised: [Equation]

Original: ) where, λ cls is the object classification loss weight, λ Eref is the early referring loss weight, λ ref is thefinal referring loss weight, λ cor is the object correspondence loss weight, and λ lang is the languageclassification loss weight.
Revised: where, λ cls is the object classification loss weight, λ Eref is the early referring loss weight, λ ref is the final referring loss weight, λ cor is the object correspondence loss weight, and λ lang is the language classification loss weight.

Original: 3.3 Implementation Details
Revised: Our implementation details are detailed in Appendix  Experiments 4.1 Datasets We evaluate our proposed 3D visual grounding approach, LAR, on three benchmarking datasets, Nr3D [2], Sr3D [2], and ScanRefer [5].

Revised: Nr3D, 3D Natural reference dataset [2], consists of 41.5K natural, free-form utterances collected by humans using a referring game between two humans.

Revised: As opposed to Sr3D [2], which consists of 83,5K synthetic utterances.

Revised: Unlike Nr3D and Sr3D datasets,

Revised: ScanRefer does not have the distractors concept making it relatively easier.

Revised: In addition, it provides 51.5K utterances of 11K objects for 800 3D indoor scenes.

Revised: 4.2 Evaluation Metrics

Revised: we followed the same metrics followed by the previous work [2] [52], where three evaluation metrics are mainly used, i.e., the referring accuracy, the visual classification accuracy, and the language classification accuracy.

Revised: In the case of Nr3d and Sr3d datasets, we followed the previous work convention, assuming the proposals are generated with GT objects.

Revised: Following this assumption, the 3D grounding problem is reformulated as a classification problem, where the model objective is to classify the referred object among the other distractors correctly, termed referring accuracy.

Revised: The referring accuracy is calculated based on whether the model picks the correct proposal from a set of

Revised: X proposals.

Revised: Alongside the main objective, which is the referring accuracy, two auxiliary metrics are defined to evaluate the performance of each stream individually, i.e., the visual classification accuracy and the language classification accuracy.

Revised: Given agnostic GT-proposals, the visual classification accuracy is measured based on whether the model predicts the object’s class correctly or not from a predefined set of possible classes.

Revised: Similarly, language classification accuracy is another auxiliary metric that assesses the performance of the linguistic branch.

Revised: This branch aims to correctly predict the class of the referred object based on the input utterance.

Revised: 4.3 Implementation Details

Original: – Input Configuration.
Revised: – Input Configuration.

Original: For the 3D input, we randomly sample 1024 points per proposal from its pointcloud segment.
Revised: For the 3D input, we randomly sample 1024 points per proposal from its point cloud segment.

Original: Furthermore, for the synthetic 2D input, we follow Algorithm , which is scrutinized in Section 
Revised: Furthermore, for the synthetic 2D input, we follow Algorithm 1, which is scrutinized in Section 3.1.

Original: During the testing phase, the camera is set 2 meters away from the prominent face l ′ , i.e., d f = 2 , and 1 meter away from the floor, i.e., d up = 1 .In addition, the region of
Revised: During the testing phase, the camera is set 2 meters away from the prominent face l ′ , i.e., d f = 2 , and 1 meter away from the floor, i.e., d up = 1 .

Original:  interest extension ratio ϵ , depicted in Figure 2 part (C), is set to 2%.
Revised: In addition, the region of interest extension ratio ϵ , depicted in Figure 2 part (C), is set to 2%.

Original: The generated image is a square with asize of 32 pixels.
Revised: The generated image is a square with a size of 32 pixels.

Original: Five views are generated per object, where the angle θ between each camera is 30 ◦ ,demonstrated in Figure 1.
Revised: Five views are generated per object, where the angle θ between each camera is 30 ◦ , demonstrated in Figure 1.

Original: Thus the camera is set to the following angles ( 0 ◦ , 30 ◦ , − 30 ◦ , 60 ◦ , − 60 ◦ ). 
Revised: Thus the camera is set to the following angles ( 0 ◦ , 30 ◦ , − 30 ◦ , 60 ◦ , − 60 ◦ ). [Table] [Table]

Original: – Network and Training Configuration.
Revised: – Network and Training Configuration.

Original: A pre-trained model on the ImageNet dataset [9], TinyConvNext [28], is used to extract the features of our synthetic 2D images.
Revised: A pre-trained model on the ImageNet dataset [9], TinyConvNext [28], is used to extract the features of our synthetic 2D images.

Original: Our 2D backbone is sharedacross different views.
Revised: Our 2D backbone is shared across different views.

Original: We adopt the same 3D encoder in the previous works [1] [2] [13] [37] [18][52], PointNet++ [35], to encode the sampled points per proposal.
Revised: We adopt the same 3D encoder in the previous works [1] [2] [13] [37] [18] [52], PointNet++ [35], to encode the sampled points per proposal.

Original: For the language encoder, weadopt the BERT encoder [11] with three layers.
Revised: For the language encoder, we adopt the BERT encoder [11] with three layers.

Original: We adopted the same transformer architecture used in[52] [51] for our visual, multi-modal, and fusion transformers.
Revised: We adopted the same transformer architecture used in [52] [51] for our visual, multi-modal, and fusion transformers.

Original: All models are trained for 100 epochsfrom scratch, using the weight initialization strategy described in [14].
Revised: All models are trained for 100 epochs from scratch, using the weight initialization strategy described in [14].

Original: As mentioned in SAT [52],the initial learning rate is set to 10 − 4 and decreases by 0.65 every ten epochs .
Revised: As mentioned in SAT [52], the initial learning rate is set to 10 − 4 and decreases by 0.65 every ten epochs .

Original: Adam optimizer [21]and mini-batch size of 32 per GPU are used for training all our models.
Revised: Adam optimizer [21] and mini-batch size of 32 per GPU are used for training all our models.

Original: We set the losses weights asfollows, λ cls = 5 , λ Eref = 0 .
Revised: We set the losses weights as follows, λ cls = 5 , λ Eref = 0 .

Original: 5 , λ ref = 5 , λ cor = 10 , and λ lang = 0 . 
Revised: 5 , λ ref = 5 , λ cor = 10 , and λ lang = 0 . 5 .

Original: – Software and Hardware Details.
Revised: – Software and Hardware Details.

Original: We use the python augmentation framework, Albumentations [4],for image augmentation.
Revised: We use the python augmentation framework, Albumentations [4], for image augmentation.

Original: In order to perform an online augmentation while training, we implementedour 2D Synthetic Images Generator (SIG), described at Section 3.1, in a pure Pythonic approachwithout using any 3D libraries which also fasten the training.
Revised: In order to perform an online augmentation while training, we implemented our 2D Synthetic Images Generator (SIG), described at Section 3.1, in a pure Pythonic approach without using any 3D libraries which also fasten the training.

Original: Our module is implemented in Pythonusing the PyTorch framework and four Nvidia V100 GPUs.Experiments
Revised: Our module is implemented in Python using the PyTorch framework and four Nvidia V100 GPUs.

Original: Datasets.

Original: We evaluate our proposed 3D visual grounding approach, LAR, on three benchmarkingdatasets, Nr3D [2], Sr3D [2], and ScanRefer [5].

Original: Nr3D, 3D Natural reference dataset [2], consistsof 41.5K natural, free-form utterances collected by humans using a referring game between twohumans.

Original: As opposed to Sr3D [2], which consists of 83,5K synthetic utterances.

Original: Unlike Nr3D and

Original: Sr3D datasets, ScanRefer does not have the distractors concept making it relatively easier.

Original: In addition,it provides 51.5K utterances of 11K objects for 800 3D indoor scenes.

Original: Evaluation Metrics.

Original: we followed the same metrics followed by the previous work [2] [52], wherethree evaluation metrics are mainly used, i.e., the referring accuracy, the visual classification accuracy,and the language classification accuracy.

Original: In the case of Nr3d and Sr3d datasets, we followed theprevious work convention, assuming the proposals are generated with GT objects.

Original: Following thisassumption, the 3D grounding problem is reformulated as a classification problem, where the modelobjective is to classify the referred object among the other distractors correctly, termed referring accuracy.

Original: The referring accuracy is calculated based on whether the model picks the correct proposalfrom a set of X proposals.

Original: Alongside the main objective, which is the referring accuracy, twoauxiliary metrics are defined to evaluate the performance of each stream individually, i.e., the visualclassification accuracy and the language classification accuracy.

Original: Given agnostic GT-proposals, thevisual classification accuracy is measured based on whether the model predicts the object’s classcorrectly or not from a predefined set of possible classes.

Original: Similarly, language classification accuracyis another auxiliary metric that assesses the performance of the linguistic branch.

Original: This branch aims tocorrectly predict the class of the referred object based on the input utterance.

Original: 4.1 Ablation studies
Revised: 4.4 Ablation studies

Original: We conducted five ablation studies to assess the proposed architecture and to validate the contributionof each component to achieve good performance.
Revised: We conducted five ablation studies to assess the proposed architecture and to validate the contribution of each component to achieve good performance.

Original: These components are: 1) 2D multi-views and 2Dresolution. 2) Types of 2D semantic information.
Revised: These components are: 1) 2D multi-views and 2D resolution. Types of 2D semantic information.

Original: 3) Camera vs. image augmentation 4) Differentmulti-modal fusion techniques. Different 2D semantic fusion techniques.
Revised: 3) Camera vs. image augmentation 4) SIG Module vs. 2D real clues. 5) Noisy Cameras Extrinsic and Intrinsic.

Original: Where the last two arestated in Appendix 5.[Table] [Table]
Revised: More ablations are stated in Appendix 5.

Original: – 2D multi-views and 2D resolution.
Revised: – 2D multi-views and 2D resolution.

Original: To assess our 2D Synthetic Images Generator (SIG), we have deliberatelyremoved the 3D stream, colored with yellow in Figure 3;3D encoder, 3D heads, and rely only on our generated 2Dimages only while tackling the referring task.
Revised: To assess our 2D Synthetic Images Generator (SIG), we have deliberately removed the 3D stream, colored with yellow in Figure 3; 3D encoder, 3D heads, and rely only on our generated 2D images only while tackling the referring task.

Original: Thus, our architecture is simplified to include only the 2D encoder, 2Dmulti-modal transformer, and 2D heads.
Revised: Thus, our architecture is simplified to include only the 2D encoder, 2D multi-modal transformer, and 2D heads.

Original: We trained this2D architecture variant with different image resolutionsand multi-view setup combinations.
Revised: We trained this 2D architecture variant with different image resolutions and multi-view setup combinations

Original: As shown in Table 2, The referring accuracy increased linearly while increasingthe image size or increasing the number ofviews for theprojected images.
Revised: As shown in Table 2, The referring accuracy increased linearly while increasing the image size or increasing the number of

Revised: views

Original:  However, our multi-view setup has thedominant effect, whereas using only 64 image resolutionwith five views surpassing oneshot with 128 resolution.
Revised: for the projected images. However, our multi-view setup has the dominant effect, whereas using only 64 image resolution with five views surpassing one

Original:  Also, we found that using the combination of five different views, each has 128 resolution, surpassing Referit3D and non-SAT variant, where both use 3D semantics only, which shows the quality of thelearned representation via our novel synthetic 2D images generator (SIG).
Revised: shot with 128 resolution. Also, we found that using the combination of five different views, each hasresolution, surpassing Referit3D and non-SAT variant, where both use 3D semantics only, which shows the quality of the learned representation via our novel synthetic 2D images generator (SIG).

Original: [Table]

Original: – Types of 2D semantic information.
Revised: – Types of 2D semantic information.

Original: Inspired by SAT [52], we dissect the role of different 2D semantic,following Eq. 1.
Revised: Inspired by SAT [52], we dissect the role of different 2D semantic, following Eq. 1.

Original: As shown in Table 3, the first row(a) represents a 3D variant of our architecture where only the 3D encoder, 3D multi-modal transformer,and 3D heads are included.
Revised: As shown in Table 3, the first row (a) represents a 3D variant of our architecture where only the 3D encoder, 3D multi-modal transformer, and 3D heads are included.

Original: Therefore, this variant mimics the Non-SAT [52] configurations
Revised: Therefore, this variant mimics the Non-SAT [52] configurations.

Original: Startingfrom

Original: row b

Original: to

Original:  row e , our synthetic 2D semantics areincorporated alongside the 3D semantics.
Revised: Starting from row b to row e, our synthetic 2D semantics are incorporated alongside the 3D semantics.

Original: Also, wecompare against SAT [52] to highlight the critical differences in the meaning of each semantic information.
Revised: Also, we compare against SAT [52] to highlight the critical differences in the meaning of each semantic information.

Original: ROI in our architecture is the synthetic2D image generated from the 3d point clouds using our SIG module.
Revised: ROI in our architecture is the synthetic 2D image generated from the 3d point clouds using our SIG module.

Original: In contrast, SAT [52] usesthe original ScanNet images during the training phase.
Revised: In contrast, SAT [52] uses the original ScanNet images during the training phase.

Original: Unlike SAT, which uses the ground-truthclassification labels during the training and masks them during testing, we use the
Revised: Unlike SAT, which uses the ground-truth classification labels during the training and masks them during testing, we use [Table] [Table]

Original: predicted
Revised: [Table] [Table] [Table] [Table]

Original: labelsfrom the 2D classification head z ′ .
Revised: the predicted labels from the 2D classification head z ′ .

Original: Accordingly, we do not need to mask the 2D semantics, as we areusing synthetic ones.
Revised: Accordingly, we do not need to mask the 2D semantics, as we are using synthetic ones.

Original: As shown in row (e), the best accuracy is achieved when we jointly incorporate the three semantic types.
Revised: As shown in row (e), the best accuracy is achieved when we jointly incorporate the three semantic types.

Original: We

Original: also

Original: notice a significant

Original:  superiority in classification accuracy where our synthetic 2D images assist the 3D classifier in both phases; training and testing.
Revised: We also notice a significant superiority in classification accuracy where our synthetic 2D images assist the 3D classifier in both phases; training and testing.

Original: – Camera vs. Image Augmentation.
Revised: – Camera vs. Image Augmentation.

Original: We have two kinds of augmentation during training: 1) Imageaugmentation.
Revised: We have two kinds of augmentation during training: 1) Image augmentation.

Original: 2) Camera augmentation.
Revised: 2) Camera augmentation.

Original: For image augmentation, we followed the conventional techniques used by the major computervision applications like horizontal and vertical flipping, blurring, color jitter, random scaling, randomshifting, and random cropping.
Revised: For image augmentation, we followed the conventional techniques used by the major computer vision applications like horizontal and vertical flipping, blurring, color jitter, random scaling, random shifting, and random cropping.

Original: For camera augmentation, we range d f from 1.5 to 4 meters, d upfrom 0.5 to 2.5 meters, and ϵ from 2% to 15%.
Revised: For camera augmentation, we range d f from 1.5 to 4 meters, d up from 0.5 to 2.5 meters, and ϵ from 2% to 15%.

Original: By changing the variables mentioned above, the[Table] extrinsic camera parameters are augmented.
Revised: By changing the variables mentioned above, the extrinsic camera parameters are augmented.

Original: In addition, random noise is added to the intrinsic cameraparameters.
Revised: In addition, random noise is added to the intrinsic camera parameters.

Original: Table 4 demonstrates the effect of each augmentation type alone and the effect of fusingboth of them.
Revised: Table 4 demonstrates the effect of each augmentation type alone and the effect of fusing both of them.

Original: The best accuracy is achieved by utilizing the two augmentation types.
Revised: The best accuracy is achieved by utilizing the two augmentation types.

Original: [Table]

Original: [Table]

Original: – SIG Module vs. 2D real clues.
Revised: – SIG Module vs. 2D real clues.

Original: To validate our SIG module, we tailored a variant of our architecture, where our SIG module is replaced by the actual 2Dclues produced by Faster R-CNN [39].
Revised: To validate our SIG module, we tailored a variant of our architecture, where our SIG module is replaced by the actual 2D clues produced by Faster R-CNN [39].

Original: As shown in Table 5,replacing our SIG module with the features produced by Faster R-CNN [39] degrades the performance by 2%.
Revised: As shown in Table 5, replacing our SIG module with the features produced by Faster R-CNN [39] degrades the performance by 2%.

Original: In addition,even when we compared unfairly to methods that use additional2D information, even the variant of SAT, which utilize realimages during training and testing (row #2), we still outperformit by 1%.
Revised: In addition, even when we compared unfairly to methods that use additional 2D information, even the variant of SAT, which utilize real images during training and testing (row #2), we still outperform it by 1%.

Revised: – Noisy Cameras Extrinsic and Intrinsic.

Original: 4.2 Comparison to State-of-the-art
Revised: Due to the significant improvement introduced by utilizing the camera augmentation, as shown in Table 4, a detailed analysis is conducted to explore the robustness against the camera’s variations. Consequently, our 2D synthetic branch learns how to fuse information across different cameras in a data-driven way. Therefore, we can train the model to be robust to noise camera models, as shown in Table Both cameras dying and noisy parameters experiments could be thought as irrelevant as LAR utilizes virtual cameras.

Revised: Thus, no camera dropout nor noise can be introduced.

Revised: However, our SIG module, depicted in Section 3.1, can be utilized in a wide range of applications when these robustness characteristics against such noise are essential for these applications.

Revised: For instance, in the autonomous driving world, our SIG module can be employed to convert dense lidar point clouds into synthetic images, which will be used to train an arbitrary neural network that tackles specific tasks (e.g., object segmentation in cocoon setup [31] [47]).

Revised: Then, the model will be tested using real cameras instead of generating synthetic imagery input from the lidar.

Original: [Table]
Revised: [Table]

Revised: 4.5 Comparison to State-of-the-art

Revised: Moreover

Original: Moreover, we achieve state-of-the-art results on the threewell-known 3D visual grounding benchmarks, i.e., Nr3D[2], Sr3D [2], and ScanRefer [5].
Revised: [Table] we achieve state-of-the-art results on the three well-known 3D visual grounding benchmarks, i [Table] e., Nr3D [2], Sr3D [2]

Original:  By effectively utilizing our 2D synthetic images, LAR outperforms allthe existing methods, which do not use any additionaltraining dataset, by a large margin of 5.0 % and 1.9 %on Nr3D and Sr3D, respectively, depicted in Table 7.
Revised: [Table] and ScanRefer [5]. By effectively utilizing our 2D synthetic images, LAR outperforms all the existing methods, which do not use any additional training dataset, by a large margin of 5.0 % and 1.9 % on Nr3D and Sr3D, respectively, depicted in Table 9.

Original: Where, LanguageRefer [37] achieves 43.9%, while LARachieves 48.9% on Nr3D. For Sr3D, our model achieves59.35%, which is a 1.9 % increase over TransRefer3D [13].
Revised: Where, LanguageRefer [37] achieves 43.9%, while LAR achieves 48.9% on Nr3D. For Sr3D, our model achieves 59.35%, which is a 1.9 % increase over TransRefer3D [13].

Original: In addition, for the ScanRefer [2] benchmark, we also achieve state-of-the-art results, by boosting theaccuracy by 2.3 % from the closest counterpart architecture; SAT [52], mentioned in Table 6. [Table]
Revised: In addition, for the ScanRefer [2] benchmark, we also achieve state-of-the-art results, by boosting the accuracy by 2.3 % from the closest counterpart architecture; SAT [52], mentioned in Table 7.

Original: We retrained SAT [52] for a fair comparison, as we notice difficulties reproducing and verifying SAT results 1 .
Revised: We retrained SAT [52] for a fair comparison, as we notice difficulties reproducing and verifying SAT results 1 .

Original: Although we do not use an extra input duringtraining as SAT, which uses the original ScanNet 2D images [8], we surpass SAT by 1.6 % , 2.7 % , and 2.3 % on Nr3D, Sr3D, and ScanRefer, respectively.
Revised: Although we do not use an extra input during training as SAT, which uses the original ScanNet 2D images [8], we surpass SAT by 1.6 % , 2.7 % , and 2.3 % on Nr3D, Sr3D, and ScanRefer, respectively.

Original: 4.3 Discussion and Analysis
Revised: 4.6 Discussion and Analysis

Original: First, we will demonstrate our qualitative analysis.
Revised: First, we will demonstrate our qualitative analysis.

Original: Then, we have conductedtwo experiments to probe the robustness of the representations learned by ourproposed module LAR.
Revised: Then, we have conducted two experiments to probe the robustness of the representations learned by our proposed module LAR.

Original: – Qualitative Analysis.
Revised: – Qualitative Analysis.

Original: Figure 5 depicts two corner cases in our projectionmodule (SIG).
Revised: Figure 5 depicts two corner cases in our projection module (SIG).

Original: While assigning the projected 3D points into a 2D grid, morethan one point may share the exact spatial location on the grid, which means only one will be storedin the grid, and the rest of the points will be discarded.
Revised: While assigning the projected 3D points into a 2D grid, more than one point may share the exact spatial location on the grid, which means only one will be stored in the grid, and the rest of the points will be discarded.

Original: As shown in part (A) in Figure 5, on the leftcolumn, the floor overlaps the other objects due to the conflict between points during the assignationstep.
Revised: As shown in part (A) in Figure 5, on the left column, the floor overlaps the other objects due to the conflict between points during the assignation step.

Original: To overcome this behavior, a simple priority rank is given for each point based on its height.
Revised: To overcome this behavior, a simple priority rank is given for each point based on its height.

Original: Part (B) shows another failure case, where the prominent face is falsely localized, which causesplacing the camera outside the room or inside other objects.
Revised: Part (B) shows another failure case, where the prominent face is falsely localized, which causes placing the camera outside the room or inside other objects

Original: Therefore, we localize the prominentface relative to the room center to mitigate this effect.
Revised: Therefore, we localize the prominent face relative to the room center to mitigate this effect

Original: An example of an ambiguous description isshown in Figure 6 part A.
Revised: An example of an ambiguous description is shown in Figure 6 part A.

Original: The model incorrectly predicts "the chair on the right rather than the chairon the left."
Revised: The model incorrectly predicts "the chair on the right rather than the chair on the left."

Original: The model and the human annotations are occasionally confusing to referring to theobjects in the view-dependent description.
Revised: The model and the human annotations are occasionally confusing to referring to the objects in the view-dependent description.

Original: While successful cases can be seen in Figure 6 part B. Formore qualitative visualization, please refer to Appendix 5.[Table] [Table]
Revised: While successful cases can be seen in Figure 6 part B. For more qualitative visualization, please refer to Appendix 5.

Original: – Evaluation with Ground-Truth Class Labels.
Revised: – Evaluation with Ground-Truth Class Labels.

Original: Table 8 shows anothervariant, which is tailored from our architecture to be compared against SATGT.
Revised: Table 8 shows another variant, which is tailored from our architecture to be compared against SAT-GT.

Original: Where the predicted class labels are replaced by the ground-truth ones.
Revised: Where the predicted class labels are replaced by the ground-truth ones.

Original: Our LAR architecture outperforms SAT-GT by 12%.
Revised: Our LAR architecture outperforms SAT-GT by 12%.

Original: – Zero-shot Cameras Dying Testing.
Revised: – Zero-shot Cameras Dying Testing.

Original: We deliberately drop the cameras duringtesting to verify the robustness of our learned representations against cameravariations.
Revised: We deliberately drop the cameras during testing to verify the robustness of our learned representations against camera variations.

Original: As shown in Figure 4, starting from our best model, which is trainedusing five cameras, the referring accuracy drops to 44.6% then 40.15%, whenwe only use three cameras then one camera during testing, respectively.
Revised: As shown in Figure 4, starting from our best model, which is trained using five cameras, the referring accuracy drops to 44.6% then 40.15%, when we only use three cameras then one camera during testing, respectively.

Original: Drivenby these results, camera dropout techniques can be invoked to overcome the performance drop, bydeliberately drop a random camera during 
Revised: Driven by these results, camera dropout techniques can be invoked to overcome the performance drop, by deliberately drop a random camera during the training; replace its input by a duplicate view of another camera.

Original: the

Original: training;

Original: replace its input by a

Original: duplicate

Original: view of another

Original: camera

Original: – Noisy Cameras Extrinsic and Intrinsic.

Original: Due to the significant improvement introduced by utilizingthe camera augmentation, as shown in Table 4, a detailed analysis is conducted to explore therobustness against the camera’s variations.

Original: Consequently, our 2D synthetic branch learns how to fuseinformation across different cameras in a data-driven way.

Original: Therefore, we can train the model to berobust to noise camera models, as shown in Table 9.

Original: [Table]

Original: Both cameras dying and noisy parameters experiments could

Original: [Table]

Original: be thought as irrelevant as LAR utilizes virtual cameras.

Original: Thus,no camera dropout nor noise can be introduced.

Original: However, our

Original: SIG module, depicted in Section 3.1, can be utilized in a widerange of applications when these robustness characteristicsagainst such noise are essential for these applications.

Original: Forinstance, in the autonomous driving world, our SIG module canbe employed to convert dense lidar point clouds into syntheticimages, which will be used to train an arbitrary neural network that tackles specific tasks (e.g., objectsegmentation in cocoon setup [31] [47]).

Original: Then, the model will be tested using real cameras instead ofgenerating synthetic imagery input from the lidar.

Original: – Limitations.
Revised: – Limitations.

Original: Following the standard setup of Referit3D [2], we assume access to the object proposalsof each 3D scene, similar to SAT [52], LanguageRefer [37], TransRefer3D [13], and 3DVG-Trans[59].
Revised: Following the standard setup of Referit3D [2], we assume access to the object proposals of each 3D scene, similar to SAT [52], LanguageRefer [37], TransRefer3D [13], and 3DVG-Trans [59].

Original: Therefore, we use the ground-truth proposals in our experiments.
Revised: Therefore, we use the ground-truth proposals in our experiments.

Original: Following this assumption,the 3D grounding problem is reformulated as a classification problem, where the model objective isto classify the referred object among the other distractors correctly.
Revised: Following this assumption, the 3D grounding problem is reformulated as a classification problem, where the model objective is to classify the referred object among the other distractors correctly.

Original: However, LAR is compatiblewith the setting that uses detected-based proposals.
Revised: However, LAR is compatible with the setting that uses detected-based proposals.

Original:  Conclusion
Revised:  Conclusion

Original: We propose an efficient 3D grounding architecture that leverages the 2D knowledge to reinforce thelearned 3D representations.
Revised: We propose an efficient 3D grounding architecture that leverages the 2D knowledge to reinforce the learned 3D representations.

Original: To avoid requiring 2D semantics neither during training nor testing, weintroduce a 2D synthetic images generator, termed SIG.
Revised: To avoid requiring 2D semantics neither during training nor testing, we introduce a 2D synthetic images generator, termed SIG.

Original: Our SIG module generates high-quality 2Dsynthetic images.
Revised: Our SIG module generates high-quality 2D synthetic images.

Original: LAR, 2D Synthetic Semantics Knowledge Distillation for 3D Visual Grounding,consists of two stages, i.e., Visual and Multi-Modal modules.
Revised: LAR, 2D Synthetic Semantics Knowledge Distillation for 3D Visual Grounding, consists of two stages, i.e., Visual and Multi-Modal modules.

Original: LAR preserves the advantages ofdistilling the synthetic 2D knowledge to the 3D stream without requiring any additional information.
Revised: LAR preserves the advantages of distilling the synthetic 2D knowledge to the 3D stream without requiring any additional information.

Original: Our experiments show a significant superior performance, where LAR achieves state-of-the-art results across three different 3D grounding benchmarks by outperforming the existing approaches, whichdo not rely on extra training data, by a wide margin, i e., 5.0 % , 1.9 % and 2.3 % on Nr3D, Sr3Dand ScanRefer, respectively.
Revised: Our experiments show a significant superior performance, where LAR achieves state-of-the-art results across three different 3D grounding benchmarks by outperforming the existing approaches, which do not rely on extra training data, by a wide margin, i.e., 5.0 % , 1.9 % and 2.3 % on Nr3D, Sr3D and ScanRefer, respectively.

Original: Hence, despite the unfair comparison with SAT, which uses extra 2D images, LAR surpasses SAT by 1.6 % , 2.7 %  and 2.3 % on Nr3D, Sr3D and ScanRefer, respectively.
Revised: Hence, despite the unfair comparison with SAT, which uses extra 2D images, LAR surpasses SAT by 1.6 % , 2.7 % , and 2.3 % on Nr3D, Sr3D, and ScanRefer, respectively.

Original: Furthermore, we verify the robustness of LAR and its generalization ability via zero-shot experiments to intrinsic and extrinsic camera parameters and dying cameras.
Revised: Furthermore, we verify the robustness of LAR and its generalization ability via zero-shot experiments to intrinsic and extrinsic camera parameters and dying cameras.

Original: More qualitative results are attached in the supplementary materials.
Revised: More qualitative results are attached in the supplementary materials.

Revised: Acknowledgments and Disclosure of Funding

Revised: This work is funded by KAUST BAS/1/1685-01-0 and is also partially supported by the SDAIAKAUST Center of Excellence in Data Science and Artificial Intelligence (SDAIA-KAUST AI).

Original: Checklist
Revised: Checklist

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes](b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] Please refer to Section 4.3(c)
Revised: [Yes] Please refer to Section 4.

Original:  Did you discuss any potential negative societal impacts of your work
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: N /A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: [N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A] (b) Did you include complete proofs of all theoretical results?
Revised: [N/A] (b) Did you include complete proofs of all theoretical results?

Original: [N/A] 
Revised: [N/A] 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] Our codes are attached in the supplemental material combined with a detailed ’readme’ file.
Revised: [Yes] Our codes are attached in the supplemental material combined with a detailed ’readme’ file.

Original: Also, refer to Section 3.3 for the implementation details.
Revised: Also, refer to Section 4.3 for the implementation details.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] Our codes are attached in the supplemental material combined with a detailed ’readme’ file.
Revised: [Yes] Our codes are attached in the supplemental material combined with a detailed ’readme’ file.

Original: Also, refer to Section 3.3 for the implementation details.
Revised: Also, refer to Section 4.3 for the implementation details.

Original: (c) Did you report error bars (e g., with respect to the random seed after running experi- ments multiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] Refer to Table 7.
Revised: [Yes] Refer to Table 9.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] Refer to Section 3.3.
Revised: [Yes] Refer to Section 4.3.

Original: If you are using existing assets (e g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] (b) Did you mention the license of the assets?
Revised: [Yes] (b) Did you mention the license of the assets?

Original: [Yes] (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: [Yes] (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
Revised: [Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [Yes] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Revised: [Yes] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]
Revised: [N/A]

Original: A Appendix
Revised: A Appendix

Original: Our Supplemental Material contains the following sections:
Revised: Our Supplemental Material contains the following sections:

Original: • Detector-Generated 3D Proposal.
Revised: • Detector-Generated 3D Proposal.

Original: • Computational Cost.
Revised: • Computational Cost.

Original: • Demonstrate our synthetic images generator algorithm (SIG).• More ablations.
Revised: • Demonstrate our synthetic images generator algorithm (SIG). • Implementation Details.

Revised: • Evaluation Metrics • More ablations.

Original: • Qualitative results.
Revised: • Qualitative results.

Original: • Our source code.

Original: A.1 Detector-Generated 3D Proposal
Revised: A.1 Detector-Generated 3D Proposal

Original: Following the standard setup of Referit3D [2], we assume access to the object proposals of each3D scene, similar to SAT [52], LanguageRefer [37], TransRefer3D [13], and 3DVG-Trans [59].
Revised: Following the standard setup of Referit3D [2], we assume access to the object proposals of each 3D scene, similar to SAT [52], LanguageRefer [37], TransRefer3D [13], and 3DVG-Trans [59].

Original: Therefore, we use the ground-truth proposals in our experiments.
Revised: Therefore, we use the ground-truth proposals in our experiments.

Original: Following this assumption, the3D grounding problem is reformulated as a classification problem, where the model objective is toclassify the referred object among the other distractors correctly.
Revised: Following this assumption, the 3D grounding problem is reformulated as a classification problem, where the model objective is to classify the referred object among the other distractors correctly.

Original: However, LAR is compatible withthe setting that uses detected-based proposals.
Revised: However, LAR is compatible with the setting that uses detected-based proposals.

Original: To this end, we experimented our LAR module withthe detector-generated 3D proposals using VoteNet detector.
Revised: To this end, we experimented our LAR module with the detector-generated 3D proposals using VoteNet detector.

Original: As shown in Figure 7, we adapted the
Revised: As shown in Figure 7, we adapted the

Original: Scanrefer architecture as follows: 1) 2D Encoder: Integrating a 2D encoder right after the predictedbox proposals, i.e., SIG module and Tiny-ConvNext in our case, and Faster R-CNN [1] in SATcase.
Revised: Scanrefer architecture as follows: 1) 2D Encoder: Integrating a 2D encoder right after the predicted box proposals, i.e., SIG module and Tiny-ConvNext in our case, and Faster R-CNN [1] in SAT case.

Original: 2) Fusion Transformer:
Revised: 2) Fusion Transformer:

Original: In LAR case, we added our visual transformer only as we aim toemphasize the contribution of our SIG module only.
Revised: In LAR case, we added our visual transformer only as we aim to emphasize the contribution of our SIG module only.

Original: While in SAT case, we convert their multi-modaltransformer to vision transformer by excluding the language embedding from it.
Revised: While in SAT case, we convert their multi-modal transformer to vision transformer by excluding the language embedding from it.

Original: The same 3Dprediction modules and language modules are adopted from Scanrefer, i.e., P++ and Voting module,[Table] and GloVe and GRU, respectively.
Revised: The same 3D prediction modules and language modules are adopted from Scanrefer, i.e., P++ and Voting module, and GloVe and GRU, respectively.

Original: We train three models: 1) The best variant of Scanrefer. 2)
Revised: We train three models: 1) The best variant of Scanrefer. 2)

Original: SAT-Detection-based architecture.
Revised: SAT-Detection-based architecture.

Original: 3) LAR-Detection-based architecture.
Revised: 3) LAR-Detection-based architecture.

Original: We first train the Scanrefer for 50 epochs then freeze their 3D prediction modules and languagemodules for two epochs to train the new blocks added in SAT and LAR detection based architectures.
Revised: We first train the Scanrefer for 50 epochs then freeze their 3D prediction modules and language modules for two epochs to train the new blocks added in SAT and LAR detection based architectures.

Original: Then, we train the whole architecture for two more epochs.
Revised: Then, we train the whole architecture for two more epochs.

Original: For more details, please refer to the"Detector-Generated 3D Proposal" section in Appendix A in the revised paper version, where weadded a figure to demonstrate the adaptations done on the ScanRefer architecture.
Revised: For more details, please refer to the "Detector-Generated 3D Proposal" section in Appendix A in the revised paper version, where we added a figure to demonstrate the adaptations done on the ScanRefer architecture.

Original: As shown in Table 10, the tailored LAR-Detection-based architecture outperforms both the original
Revised: As shown in Table 10, the tailored LAR-Detection-based architecture outperforms both the original

Original: ScanRefer and the SAT-Detection-based architectures.
Revised: ScanRefer and the SAT-Detection-based architectures.

Original: Also, the results show that SAT is morevulnerable to the distortions in the detected proposals. 
Revised: Also, the results show that SAT is more vulnerable to the distortions in the detected proposals. [Table] [Table]

Original: A.2 Computational Cost
Revised: A.2 Computational Cost

Original: We compared our complexity against SAT, during both the training and testing.
Revised: We compared our complexity against SAT, during both the training and testing.

Original: As shown in Table11, our training and inference time are 3.6 x and 0.7 x compared to SAT.
Revised: As shown in Table 11, our training and inference time are 3.6 x and 0.7 x compared to SAT.

Original: All the reported results aremeasured on single GTX-1080 GPU.[Table] SAT 47.
Revised: All the reported results are measured on single GTX-1080 GPU.

Original: 237 M 81 M 0.317 FPS 5.97 FPS

Original: LAR 48.

Original: 118 M 118 M 1.

Original: [Table]

Original: A.
Revised: A.3 Synthetic Images Generator Algorithm

Original: Synthetic Images Generator Algorithm

Original: Algorithm 1, demonstrates a pseudo-code for our
Revised: Algorithm 1, demonstrates a pseudo-code for our

Original: SIG module thatprojects arbitrary 3D
Revised: SIG module that projects arbitrary 3D

Original: Point clouds representing an object to a 2D image.
Revised: Point clouds representing an object to a 2D image.

Original: l
Revised: Input : x pci : Point-cloud for an object i in the scene, I : The pinhole camera intrinsic parameters,

Original: 9: Cam d up ] 10:
Revised: O s : Scene center.

Original: Build transformation matrix
Revised: Output : x imgi : 2D synthetic image that represents the object i . 1: // Get the prominent face: 2: minDist ← Dist ( O l 0 , O s ) 3: for each face do 4: if Dist ( O l k , O s ) < = minDist then 5: l ′ ← l k 6: minDist ← Dist ( O l k , O s ) 7: end if 8: end for 9: Cam p =

Original: 11: V f ← ∥
Revised: [ d fx , d fy , d up ] 10: // Build transformation matrix M ∈ R 4 ×11: V f ← ∥

Original: Cam p − O  ∥ 12: V r ← ∥V f ⊗
Revised: Cam p − O l ′ ∥ 12: V r ← ∥

Original: 0 , 0 , 1] ∥ 13: V up ← ∥ V r ⊗ V f ∥ 14: M ← (cid:20) V 
Revised: V f ⊗ [0 , 0 , 1] ∥ 13: V up ← ∥ V r ⊗ V f ∥ 14: M ← (cid:20) V r

Original: V up
Revised: V up

Original: 15: x cami ← Dot 16: x ← Dot I x
Revised: V f

Original: 17: x imgi ← x cami x 18: return
Revised: Cam p   (cid:21) .

Original: A.4 Ablations Studies
Revised: A.4 Ablations Studies

Original: A.4.1 Different 2D semantic fusion techniques.
Revised: A.4.1 Different 2D semantic fusion techniques.

Original: Driven by the aforementioned results in Table 2, we use five virtual cameras set in a cocoon setuparound each object as depicted in Figure 1, and the image resolution is 128.
Revised: Driven by the aforementioned results in Table 2, we use five virtual cameras set in a cocoon setup around each object as depicted in Figure 1, and the image resolution is 128.

Original: In Table 12, we studytwo fusion techniques to combine the features generated from the five views.
Revised: In Table 12, we study two fusion techniques to combine the features generated from the five views.

Original: The first method is asimple addition operation, and the second one is a learnable fusion mechanism where a 1D-Conv.is used where its kernel size is set to the number of the views used, i.e., in our case set to five.
Revised: The first method is a simple addition operation, and the second one is a learnable fusion mechanism where a 1D-Conv. is used where its kernel size is set to the number of the views used, i.e., in our case set to five.

Original: Asexpected, the learnable fusion scheme; 1D-Conv., performs better than the none learnable scheme;addition.
Revised: As expected, the learnable fusion scheme; 1D-Conv., performs better than the none learnable scheme; addition.

Original: Our 2D variant built on top of Referit3D [2] is used in this ablation, where the 2D stream isonly used.
Revised: Our 2D variant built on top of Referit3D [2] is used in this ablation, where the 2D stream is only used.

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: A.4.2 Different multi-modal fusion techniques.

Original: In addition, we study three fusion techniques for the 3D, and 2D multi-modal features, M pc and
Revised: In addition, we study three fusion techniques for the 3D, and 2D multi-modal features, M pc and

Original: M img , respectively.
Revised: M img , respectively.

Original: In this ablation, we only utilize the ROI as the 2D semantic information, set theimage resolution to 32, and set the number of multi-views to five.
Revised: In this ablation, we only utilize the ROI as the 2D semantic information, set the image resolution to 32, and set the number of multi-views to five.

Original: As shown in Table 13, using atransformer while fusing both multi-modal features achieves the best accuracy.
Revised: As shown in Table 13, using a transformer while fusing both multi-modal features achieves the best accuracy.

Original: Optionally include extra information (complete proofs, additional experiments and plots) in theappendix.
Revised: Optionally include extra information (complete proofs, additional experiments and plots) in the appendix.

Original: This section will often be part of the supplemental material.
Revised: This section will often be part of the supplemental material.

Original: A.5 Quantitative Results
Revised: A.5 Quantitative Results

Original: This section analyzes the qualitative results from the successful and the failure cases on the Nr3ddataset.
Revised: This section analyzes the qualitative results from the successful and the failure cases on the Nr3d dataset.

Original: In addition to Figure 8 that shows successful scenarios and Figure 9 that shows failurescenarios, we attach a short video to demonstrate more scenarios.
Revised: In addition to Figure 8 that shows successful scenarios and Figure 9 that shows failure scenarios, we attach a short video to demonstrate more scenarios.

Original: A.5.1 Successful Scenarios
Revised: A.5.1 Successful Scenarios

Original: LAR achieves top-ranking results on both the Nr3D and Sr3D datasets.
Revised: LAR achieves top-ranking results on both the Nr3D and Sr3D datasets.

Original: The results in Figure 8demonstrate the correct instance localization on the Nr3D datasets only.
Revised: The results in Figuredemonstrate the correct instance localization on the Nr3D datasets only.

Original: The ground truth boxesare marked green and our model prediction marked in blue.
Revised: The ground truth boxes are marked green and our model prediction marked in blue.

Original: Our analysis shows that LAR correctlyunderstands the global context based on natural language description.
Revised: Our analysis shows that LAR correctly understands the global context based on natural language description.

Original: For example, in Figure 8 (a-f),
Revised: For example, in Figure 8 (a-f),

Original: LAR understands relationships, e.g. (behind it, middle of, furthest from .. ), which proves that ourproposed model understands instance-to-instance and instance-to-background relations.
Revised: LAR understands relationships, e.g. (behind it, middle of, furthest from .. ), which proves that our proposed model understands instance-to-instance and instance-to-background relations.

Original: As well asunderstanding colors and shapes related to language, LAR also can determine the meaning of termssuch as “green,” Figure 8 (e).
Revised: As well as understanding colors and shapes related to language, LAR also can determine the meaning of terms such as “green,” Figure 8 (e).

Original: A.5.2 Failure Scenarios
Revised: A.5.2 Failure Scenarios

Original: Figure 9 shows representative failure cases of LAR.
Revised: Figure 9 shows representative failure cases of LAR.

Original: Despise the incorrect instance localization LARmanaged to allocate the instance with the same class as the target.
Revised: Despise the incorrect instance localization LAR managed to allocate the instance with the same class as the target.

Original: As shown in Figure 9 (d, e), weobserved that failure cases require an understanding of "against the wall", ’Facing the windows,"which is determined by camera view dependency.
Revised: As shown in Figure 9 (d, e), we observed that failure cases require an understanding of "against the wall", ’Facing the windows, "which is determined by camera view dependency.

Original: Despite LAR’s progress in view-dependent andindependent samples, view understanding remains a challenge.
Revised: Despite LAR’s progress in view-dependent and independent samples, view understanding remains a challenge.

Original: Besides, other state-of-the-art modelssuffer from the same issue.
Revised: Besides, other state-of-the-art models suffer from the same issue.

Original: In addition, some LAR failure cases are caused by ambiguous queries.
Revised: In addition, some LAR failure cases are caused by ambiguous queries.

Original: Figure 9 (a,b,c,e, f) shows a complex and compound language description to describe the targetinstance, which in some cases, both models and human annotators mix up similar objects.
Revised: Figure 9 (a,b,c,e, f) shows a complex and compound language description to describe the target instance, which in some cases, both models and human annotators mix up similar objects.

Original: To summarize, We show the effect of using synthetic 2D images, which improves the model instanceand context-aware representation.

Original: However, our model has limitations due to ambiguous and complexqueries.

Original: Although some cases remained unsolved, LAR improved progress in the camera viewdependency samples.

Original: Therefore, we believe our model LAR surpasses existing state-of-the-art by asignificant margin.

Original: A.6 Source Code

Original: our source code is attached in the supplementary material, where it contains the source code for our

Original: Synthetic Images Generator (SIG), our architecture, training functions, and evaluation functions.

Original: Formore details regarding our implementation, refer to the Readme file attached with the source code.