Original: Look Around and Refer: 2D Synthetic Semantics Knowledge Distillation for 3D Visual Grounding
Revised: Look Around and Refer: 2D Synthetic Semantics Knowledge Distillation for 3D Visual Grounding

Original: Abstract
Revised: Abstract

Original: The 3D visual grounding task has been explored with visual and language streams to comprehend referential language to identify targeted objects in 3D scenes.
Revised: The 3D visual grounding task has been explored with visual and language streams comprehending referential language to identify target objects in 3D scenes.

Original: However, most existing methods devote the visual stream to capturing the 3D visual clues using off-the-shelf point clouds encoders.
Revised: However, most existing methods devote the visual stream to capturing the 3D visual clues using off-the-shelf point clouds encoders.

Original: The main question we address in this paper is “can we consolidate the 3D visual stream by 2D clues synthesized from point clouds and efficiently utilize them in training and testing?” .
Revised: The main question we address in this paper is “can we consolidate the 3D visual stream by 2D clues synthesized from point clouds and efficiently utilize them in training and testing?” .

Original: The main idea is to assist the 3D encoder by incorporating rich 2D object representations without requiring extra 2D inputs.
Revised: The main idea is to assist the 3D encoder by incorporating rich 2D object representations without requiring extra 2D inputs.

Original: To this end, we leverage 2D clues, synthetically generated from 3D point clouds, and empirically show their aptitude to boost the quality of the learned visual representations.
Revised: To this end, we leverage 2D clues, synthetically generated from 3D point clouds, and empirically show their aptitude to boost the quality of the learned visual representations.

Original: We validate our approach through comprehensive experiments on Nr3D, Sr3D, and ScanRefer datasets, andwe show consistent performance gains compared to the state-of-the-art.
Revised: We validate our approach through comprehensive experiments on Nr3D, Sr3D, and ScanRefer datasets and show consistent performance gains compared to existing methods.

Original: Our proposed module, dubbed as Look Around and Refer (LAR), significantly outperforms the state-of-the-art 3D visual grounding techniques onthese three benchmarks.
Revised: Our proposed module, dubbed as Look Around and Refer (LAR), significantly outperforms the state-ofthe-art 3D visual grounding techniques on three benchmarks, i.e., Nr3D, Sr3D, and

Original:  The code is available at https://github.com/eslambakr/LAR-Look-Around-and-Refer.
Revised: ScanRefer. The code is available at https://eslambakr.github.io/LAR.github.io/.

Original:  Introduction
Revised:  Introduction

Original: Visiolinguistic modeling has attracted much attention over the past decade due to impact on various areas, including visual question answering [3, 36], 3D scene captioning [7], image-text matching [26, 44], visual relation detection [25] and scenegraph generation [42].
Revised: Visiolinguistic modeling has attracted much attention over the past decade due to impact on various areas, including visual question answering [3, 36], 3D scene captioning [7], image-text matching [26, 44], visual relation detection [25], and scene graph generation [42].

Original: Consequently, understanding natural language utterances and grounding them to real-world scenarios is attractive due to their importance in many real applications, such as robot navigation and interaction in indoor environment [22] [40].
Revised: Consequently, understanding natural language utterances and grounding them to real-world scenarios is attractive due to their importance in many real applications, such as robot navigation and interaction in indoor environment [22] [40].

Original: Visual Grounding (VG) has been explored in recent years in the context of 2D [16, 30, 46, 55, 57], extending object detection to a more complex task.
Revised: Visual Grounding (VG) has been explored in recent years in the context of 2D [16, 30, 46, 55, 57], extending object detection to a more complex task.

Original: However, for real robots to intelligently ground a given utterance on a natural scene in a challenging environment like a crowded indoor or outdoor scene, 3D representations are needed to understand the surrounding environment better.
Revised: However, for real robots to intelligently ground a given utterance on a natural scene in a challenging environment like a crowded indoor or outdoor scene, 3D representations are needed to understand the surrounding environment better.

Original: Therefore, 3D visual grounding task have been proposed by ReferIt3D [2] and ScanRefer [5].
Revised: Therefore, 3D visual grounding task have been proposed by ReferIt3D [2] and ScanRefer [5].

Original: The ReferIt3D [2] introduces two datasets containing natural and synthetic language utterances,  namely Nr3D and Sr3D, respectively, which
Revised: The ReferIt3D [2] introduces two datasets containing natural and synthetic language utterances, 36th Conference on Neural Information Processing Systems (NeurIPS 2022). [Table] namely Nr3D and Sr3D, respectively

Original: [Table]
Revised: [Table]

Original:  aims to determine the referent object from pre-defined objects set for each scene.
Revised: which aims to determine the referent object from pre-defined objects set for each scene

Original: In contrast, ScanRefer [5] aims to predict 3D bounding boxes for the referent object based on a given language description and 3D point clouds.
Revised: In contrast, ScanRefer [5] aims to predict 3D bounding boxes for the referent object based on a given language description and 3D point clouds.

Original: Both ScanRefer [5] and ReferIt3D [2] are built on the ScanNet dataset [8].
Revised: Both ScanRefer [5] and ReferIt3D [2] are built on the ScanNet dataset [8].

Original: The critical difference, which made ReferIt3D [2] more challenging, is that in each scene, several object instances belong to the same fine-grained category as the referent, namely distractors
Revised: The critical difference, which made ReferIt3D [2] more challenging [Table] is that in each scene, several object instances belong to the same fine-grained category as the referent, namely distractors

Original: [Table]
Revised: [Table]

Original: Due to the sparsity and disorganization of the 3D point clouds
Revised: Due to the sparsity and disorganization of the 3D point clouds

Original: [Table]
Revised: [Table]

Original: SAT [52] explores utilizing the 2D images provided by the ScanNet dataset [8] during training to assist point-cloud-language joint representation learning.
Revised: SAT [52] explores utilizing the 2D images provided by the ScanNet dataset [8] during training to assist point-cloud-language joint representation learning.

Original: SAT [52] proposes three variants; 1) The baseline, non-SAT, relies only on the 3D point clouds while tackling the grounding task.
Revised: SAT [52] proposes three variants; 1) The baseline, non-SAT, relies only on the 3D point clouds while tackling the grounding task.

Original: Utilizing 2D semantics during the training phase only and masking them while inference, we termed it SAT.
Revised: Utilizing 2D semantics during the training phase only and masking them while inference, we termed it SAT.

Original: Utilizing the 2D semantics in both phases, i.e., training and testing, we termed it SAT-GT.
Revised: Utilizing the 2D semantics in both phases, i.e., training and testing, we termed it SAT-GT.

Original: However, requiring extra 2D inputs in training or inference limits potential application scenarios.
Revised: However, requiring extra 2D inputs in training or inference limits potential application scenarios.

Original: To tackle this drawback, we hypothesized that incorporating synthetic 2D semantics will consolidate the 3D learned representation while not requiring any additional info.
Revised: To tackle this drawback, we hypothesized that incorporating synthetic 2D semantics will consolidate the 3D learned representation while not requiring any additional info.

Original: To this end, we propose a novel 2D Synthetic Images Generator (SIG) that projects the 3D point clouds into multi-view 2D images, as shown in Figure 1.
Revised: To this end, we propose a novel 2D Synthetic Images Generator (SIG) that projects the 3D point clouds into multi-view 2D images, as shown in Figure 1.

Original: To the best of our knowledge, LAR is the first method that augment learning of 3D tasks with synthetic 2D semantics.
Revised: To the best of our knowledge, LAR is the first method that augment learning of 3D tasks with synthetic 2D semantics.

Original: In addition, we propose a novel multi-modal transformer-based architecture.
Revised: In addition, we propose a novel multi-modal transformer-based architecture.

Original: Our proposed architecture, LAR, on Nr3D outperforms SAT [52] by boosting the accuracy by 1.6 % while exceeding all existing approaches that did not use any additional information than the 3D point clouds by a wider margin, i.e., 5.0 % to the closest one.
Revised: Our proposed architecture, LAR, on Nr3D outperforms SAT [52] by boosting the accuracy by 1.6 % while exceeding all existing approaches that did not use any additional information than the 3D point clouds by a wider margin, i.e., 5.0 % to the closest one.

Original: Also, LAR architecture achieves state-of-the-art results on Sr3D and ScanRefer benchmarks, 59.35% and 54.6%, respectively.
Revised: Also, LAR architecture achieves state-of-the-art results on Sr3D and ScanRefer benchmarks, 59.35% and 54.6%, respectively.

Original: Another variant is tailored from our architecture to be compared against SAT-GT, where geometry information and ground-truth class labels are concatenated to the 2D images; synthetic in our case and original images in SAT case.
Revised: Another variant is tailored from our architecture to be compared against SAT-GT, where geometry information and ground-truth class labels are concatenated to the 2D images; synthetic in our case and original images in SAT case.

Original: Our LAR architecture outperforms SAT-GT by 11.7 % , where we achieved 62% while SAT achieves 50.3%.
Revised: Our LAR architecture outperforms SAT-GT by 11.7 % ( 62% for our method and 50.3% for that).

Original: Our contributions are summarized as follows:
Revised: Our contributions are summarized as follows:

Original: • We propose a novel End-to-End multi-modal transformer-based architecture, LAR, for the 3D visual grounding task that relies only on 3D point clouds, which exploits the synthetic 2D images generated by our SIG module.
Revised: • We propose a novel End-to-End multi-modal transformer-based architecture, LAR, for the 3D visual grounding task that relies only on 3D point clouds, which exploits the synthetic 2D images generated by our SIG module.

Original: • We verify the effectiveness of our LAR architecture throughout extensive experiments with various benchmarks,  which show that utilizing our 2D synthetic semantics significantly enhances the results.
Revised: • We verify the effectiveness of our LAR architecture throughout extensive experiments with various benchmarks, i e., Nr3D, Sr3D, and ScanRefer, which show that utilizing our 2D synthetic semantics significantly enhances the results.

Original: • Through detailed analysis, we examine our method’s internal behavior and validity.
Revised: • Through detailed analysis, we examine our method’s internal behavior and validity.

Original:  Related Work
Revised:  Related Work

Original: Our novel architecture draws success from several areas, including 2D visual grounding, 3D visual grounding, and 2D semantics in 3D tasks.
Revised: Our novel architecture draws success from several areas, including 2D visual grounding, 3D visual grounding, and 2D semantics in 3D tasks.

Original: – 2D visual grounding.
Revised: – 2D visual grounding.

Original: 2D visual grounding is locating an object based on a natural language description.
Revised: 2D visual grounding is locating an object based on a natural language description.

Original: Flickr30k 
Revised: Flickr30k [32] and

Original: ] and ReferItGame [ ] generated referring expressions for a particular object in an image used to identify which object is being referred to.
Revised: ReferItGame [20] generated referring expressions for a particular object in an image used to identify which object is being referred to.

Original: Prior works [56, 27, 29] have attempted to localize these referred objects based on the given expressions by focusing on the bounding-box level.
Revised: Prior works [56, 27, 29] have attempted to localize these referred objects based on the given expressions by focusing on the bounding-box level.

Original: In contrast, [53, 10] focus on the pixels level.
Revised: In contrast, [53, 10] focus on the pixel- level predictions.

Original: Another 2D visual grounding taxonomy is the one-stage, also known as a single-stage framework, versus the two-stage framework.
Revised: Another 2D visual grounding taxonomy is the one-stage, also known as a single-stage framework, versus the two-stage framework.

Original: The single-stage methods  [38, 49, 50], fuse the linguistic query with each pixel/patches of the image, and generate a set of possible bounding box candidates.
Revised: Single-stage methods (e.g., [38, 49, 50]) fuse the linguistic query with each pixel/patch of the image, and generate a set of possible bounding box candidates.

Original: In contrast, the two-stage framework [45, 54, 56] uses the visual image semantics to generate object proposals.
Revised: In contrast, the two-stage framework [45, 54, 56] uses the visual image semantics to generate object proposals.

Original: Afterward, the grounding prediction is selected by comparing each proposal to the language query.
Revised: Afterward, the grounding prediction is selected by comparing each proposal to the language query.

Original: Consequently, 2D visualwe grounding methods can guide learning 3D relationships.
Revised: Consequently, 2D visual grounding methods can guide learning 3D relationships.

Original: Driven by this, we explore using 2D synthetic semantics to consolidate the 3D visual stream.
Revised: Driven by this, we explore using 2D synthetic semantics to consolidate the 3D visual stream.

Original: – 3D visual grounding.
Revised: – 3D visual grounding.

Original: Table 1 summarizes the existing 3D visual grounding  models whether a ground truth object proposal is used or 
Revised: Table 1 summarizes the existing 3D visual grounding (3D VG) models whether a ground truth object proposal is used or a pre-trained detector is utilized to generate the proposals.

Original: a pre-trained detector is utilized

Original: to generate the proposals.

Original: Also, it summarizes whether extra info is used alongside the object’s 3D point cloud or not.
Revised: Also, it summarizes whether extra info is used alongside the object’s 3D point cloud or not.

Original: 3D visual grounding field has undergone extensive research and made considerable progress over the last few years [2, 5, 12, 17, 59, 1, 37, 13, 58, 19, 52, 6, 19].
Revised: 3D visual grounding field has undergone extensive research and made considerable progress over the last few years [2, 5, 12, 17, 59, 1, 37, 13, 58, 19, 52, 6, 19].

Original: A 3D visual grounding task uses language instruction along with a 3D scene [8] as input in order to estimate a bounding box that allocates the most relevant object to the provided expression.
Revised: A 3D visual grounding task uses language instruction along with a 3D scene [8] as input in order to estimate a bounding box that allocates the most relevant object to the provided expression.

Original: However, the literature offers several extensions, such as using auxiliary input to improve localizing 3D objects [52].
Revised: However, the literature offers several extensions, such as using auxiliary input to improve localizing 3D objects [52].

Original: Previous work [2, 12, 17, 59, 1, 13, 58, 19], focuses on improving the language and 3D scene fusion by adopting a two-stage framework.
Revised: Previous work [2, 12, 17, 59, 1, 13, 58, 19], focuses on improving the language and 3D scene fusion by adopting a two-stage framework.

Original: The first stage outputs 3D object proposals given a 3D point cloud representing a scene.
Revised: The first stage outputs 3D object proposals given a 3D point cloud representing a scene.

Original: Secondly, visual and linguistic features are fused to compute a confidence score for each 3D object proposal.
Revised: Secondly, visual and linguistic features are fused to compute a confidence score for each 3D object proposal.

Original: The first stage could be skipped by directly using the ground-truth proposals instead of predicting them [52] [13] [2].
Revised: The first stage could be skipped by directly using the ground-truth proposals instead of predicting them [52] [13] [2].

Original: In contrast, [6, 19, 59] do not assume the access to the ground-truth objects and use an object detector to extract the proposals.
Revised: In contrast, [6, 19, 59] do not assume the access to the ground-truth objects and use an object detector to extract the proposals.

Original: Intuitively, if the detector fails, the referring performance will significantly decrease.
Revised: Intuitively, if the detector fails, the referring performance will significantly decrease.

Original: Therefore, we focus on the main training objective related to the grounding task, which is learning 3D visual-language joint representations by assuming access to the ground-truth boxes.
Revised: Therefore, we focus on the main training objective related to the grounding task, which is learning 3D visual-language joint representations by assuming access to the ground-truth boxes.

Original: Although we are assuming access to the ground-truth boxes, LAR is compatible with the  setting that utilizes detector-generated proposals.
Revised: In addition, we explore the other setting that utilizes detector-generated proposals, which is mentioned in Appendix 5.

Original: – 2D semantics in 3D tasks.
Revised: – 2D semantics in 3D tasks.

Original: 2D semantics contains useful information that can be used to detect 3D objects.
Revised: 2D semantics contains useful information that can be used to detect 3D objects.

Original: The representations of visual scenes consist of point clouds, meshes, or volumes, while 2D images consist of pixel grids.
Revised: The representations of visual scenes consist of point clouds, meshes, or volumes, while 2D images consist of pixel grids.

Original: Combining the best of two worlds can enhance 3D object localization.
Revised: Combining the best of two worlds can enhance 3D object localization.

Original: Researchers have attempted to solve the 3D detection problem by adopting different ways of representing RGB-D data [34, 48, 23, 15, 41, 24].
Revised: Researchers have attempted to solve the 3D detection problem by adopting different ways of representing RGB-D data [34, 48, 23, 15, 41, 24].

Original: By either projecting the 2D image on the 3D scene [34, 48, 24] or fuse both 2D features with the 3D [23, 15, 43, 41], [33].
Revised: By either projecting the 2D image on the 3D scene [34, 48, 24] or fuse both 2D features with the 3D [23, 15, 43, 41], [33].

Original: SAT [52] leveraged 2D semantics to perform the 3D VG task.
Revised: SAT [52] leveraged 2D semantics to perform the 3D VG task.

Original: However, requiring extra 2D inputs limits potential application scenarios.
Revised: However, requiring extra 2D inputs limits potential application scenarios.

Original: In contrast, in this work, we generate synthetic 2D images during the training phase, which is also used in the inference phase.
Revised: In contrast, in this work, we generate synthetic 2D images during the training phase, which is also used in the inference phase.

Original: The synthetic images enable us to consolidate the 3D visual stream by 2D semantics without requiring extra 2D inputs.
Revised: The synthetic images enable us to consolidate the 3D visual stream by 2D semantics without requiring extra 2D inputs.

Original:  Look Around and Refer (LAR)
Revised:  Look Around and Refer (LAR)

Original: LAR aims to enhance the visual module, which in turn enhances the final objective,i.e., referential accuracy.
Revised: LAR aims to enhance the visual module, which in turn enhances the final objective,i.e., referential accuracy.

Original: By incorporating the synthetic 2D semantics alongside the 3D point clouds, LAR inherently captures the correlations between the 2D and the 3D streams.
Revised: By incorporating the synthetic 2D semantics alongside the 3D point clouds, LAR inherently captures the correlations between the 2D and the 3D streams.

Original: To design a general framework to fit in any 3D application, when the input is 3D point clouds, the need for extra information, such as 2D semantics, should be bypassed.
Revised: To design a general framework to fit in any 3D application, when the input is 3D point clouds, the need for extra information, such as 2D semantics, should be bypassed.

Original: To this end, a 2D synthetic images generator, namely
Revised: To this end, a 2D synthetic images generator, namely

Original: SIG, is introduced, which is discussed in Section 3.1.
Revised: SIG, is introduced, which is discussed in Section 3.1.

Original: Then, we demonstrate how to learn robust representations by aligning the generated 2D semantics with the 3D ones by incorporating our SIG module into our proposed architecture, depicted in Section 3.2.
Revised: Then, we demonstrate how to learn robust representations by aligning the generated 2D semantics with the 3D ones by incorporating our SIG module into our proposed architecture, depicted in Section 3.2.

Original: The 2D synthetic images are first extracted by SIG then processed by a shared ConvNext backbone.

Original: Simultaneously, language descriptions are transformed into tokens and embedded into a feature vector.
Revised: Simultaneously, language descriptions are transformed into tokens and embedded into a feature vector.

Original: Then, the Multi-Modal Module fuses the output of the Visual Module separately by two Transformers.
Revised: Then, the Multi-Modal Module fuses the output of the Visual Module separately by two Transformers.

Original: Finally, the output of the Multi-Modal Transformers is processed bya Fusion Transformers.
Revised: Finally, the output of the Multi-Modal Transformers is processed by Fusion Transformers.

Original: 3.1 Look Around: 2D Synthetic Images Generator
Revised: 3.1 Look Around: 2D Synthetic Images Generator

Original: LAR exploits synthetic 2D semantics to reinforce the learned 3D representations.
Revised: LAR exploits synthetic 2D semantics to reinforce the learned 3D representations.

Original: Consequently, this section will demonstrate our 2D Synthetic Images Generator (SIG) module that projects arbitrary 3D
Revised: Consequently, this section will demonstrate our 2D Synthetic Images Generator (SIG) module that projects arbitrary 3D

Original: Point clouds representing an object to a 2D image.
Revised: Point clouds representing an object to a 2D image.

Original: First, we will show the essential operations of our module on a single image, then show how we extend it to fulfill a multi-view setup.
Revised: First, we introduce the essential operations of our module on a single image, then show how we extend it to fulfill a multi-view setup.

Original: A 3D scene, S ∈ R N × 6 , is represented by N points with spatial and color information, i.e., XYZ, and RGB, respectively.
Revised: A 3D scene, S ∈ R N × 6 , is represented by N points with spatial and color information, i.e., XYZ, and RGB, respectively.

Original: Following the previous works [2] [1] [37] [52], the object proposals x pci for each scene are known, either generated by one of the off-the-shelf 3D object detectors or manually annotated.
Revised: Following the previous works [2] [1] [37] [52], the object proposals x pci for each scene are known, either generated by one of the off-the-shelf 3D object detectors or manually annotated.

Original: The object proposal x pci ∈ R N ′ × 6 is a subset of the whole scene, where N ′ < N and N ′ represents the number of points that represent the object i .
Revised: The object proposal x pci ∈ R N ′ × 6 is a subset of the whole scene, where N ′ < N and N ′ represents the number of points that represent the object i .

Original: Given a scene S and an object proposal x pci , our SIG module generates a 2D synthetic image x imgi represents the object i .
Revised: Given a scene S and an object proposal x pci , our SIG module generates a 2D synthetic image x imgi represents the object i .

Original: The main idea is to place a virtual camera in front of the fed object proposal, then project its points x pci into the 2D plane to generate the corresponding 2D image x imgi .
Revised: The main idea is to place a virtual camera in front of the fed object proposal, then project its points x pci into the 2D plane to generate the corresponding 2D image x imgi .

Original: The camera’s location should be set carefully to generate a decent image representation for an object i .
Revised: The camera’s location should be set carefully to generate a decent image representation for an object i .

Original: As shown in part (A) in Figure 2, after excluding the upper and lower faces from the fed object proposal, we get the center of the four remaining faces, then calculate the distance between each face center O li , and the center of the scene O s , where l ranging from 0 to 3.
Revised: As shown in part (A) in Figure 2, after excluding the upper and lower faces from the fed object proposal, we get the center of the four remaining faces, then calculate the distance between each face center O li , and the center of the scene O s , where l ranging from 0 to 3.

Original: The nearest face l ′ from the scene center
Revised: The nearest face l ′ from the scene center

Original: O s is the prominent one, colored with blue in Figure 2, which guarantees that we look at the object i from inside the room.
Revised: O s is the prominent one, colored with blue in Figure 2, which guarantees that we look at the object i from inside the room.

Original: The camera is located at a distance d f from the prominent face l ′ and at a distance d up from the room’s floor; thus, the camera position is interpreted as Cam p =
Revised: The camera is located at a distance d f from the prominent face l ′ and at a distance d up from the room’s floor; thus, the camera position is interpreted as Cam p =

Original: [ d fx , d fy , d up ] , as shown in part (B) in Figure 2.
Revised: [ d fx , d fy , d up ] , as shown in part (B) in Figure 2.

Original: The orientation of the camera frame is defined as follows.
Revised: The orientation of the camera frame is defined as follows.

Original: The z axis is aligned with the principal axis of the camera, and two other orthogonal directions (x, y) align with the corresponding axes of the image plane.
Revised: The z-axis is aligned with the principal axis of the camera, and two other orthogonal directions (x, y) align with the corresponding axes of the image plane.

Original: Failure cases are shown in Figure 5, when the camera is located outside the room.
Revised: Failure cases are shown in Figure 5 when the camera is located outside the room.

Original: To project the points x pci , representing object i , from the scene coordinates to the camera coordinates, a transformation matrix M ∈ R 4 × 4 needs to be defined.
Revised: To project the points x pci , representing object i , from the scene coordinates to the camera coordinates, a transformation matrix M ∈ R 4 × 4 needs to be defined.

Original: Given the camera position Cam p and the center of the prominent face O l ′ , the transformation matrix M can be interpreted as, (cid:20) V r V up [Table]
Revised: Given the camera position Cam p and the center of the prominent face O l ′ , the transformation matrix M can be interpreted as, (cid:20) V r V up V

Original: f Cam p   [Equation]
Revised: f Cam p   (cid:21)

Original: Where, the camera direction is defined by three vectors; V f , V r , and V up .
Revised: Where the camera direction is defined by three vectors; V f , V r , and V up .

Original: We define our virtual camera using the Unified Camera Model (UCM), which has five parameters I =
Revised: We define our virtual camera using the Unified Camera Model (UCM), which has five parameters I =

Original: [ γ x , γ y , c x , c y , ζ ]
Revised: [ γ x , γ y , c x , c y , ζ ]

Original: Following the UCM, projection is defined as follows:
Revised: Following the UCM, projection is defined as follows:

Original: π ( x
Revised: π ( x

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Afterwards, the projected 3D point clouds are assigned to a 2D grid map.
Revised: Afterward, the projected 3D point clouds are assigned to a 2D grid map.

Original: To mitigate the overfitting effect, we randomly extend the region of interest by ϵ as demonstrated in part (C) at Figure 2.
Revised: To mitigate overfitting, we randomly extend the region of interest by ϵ as demonstrated in part (C) in Figure 2.

Original: Point cloud differs from 2D images, where every pixel contains spatial and texture details.
Revised: Point clouds differ from 2D images, where every pixel contains spatial and texture details.

Original: Moreover, the object is represented in a view-invariant way, unlike the images that are view-dependent.
Revised: Moreover, the object is represented in a view-invariant way, unlike the images that are view-dependent.

Original: To mitigate this drawback in the image representation, a multi-view setup is constructed as shown in
Revised: To mitigate this drawback in the image representation, a multi-view setup is constructed as shown in

Original: Figure 1. Where, v cameras are mounted in a circle around the prominent face l ′ .
Revised: Figure 1 where v cameras are mounted in a circle around the prominent face l ′ .

Original: Algorithm 1, attached in the supplemental materials, demonstrates a pseudo-code for our SIG module.
Revised: Algorithm 1, attached in the supplemental materials, demonstrates a pseudo-code for our SIG module.

Original: 3.2 Refer: Visiolinguistic Transformer Architecture
Revised: 3.2 Refer: Visiolinguistic Transformer Architecture

Original: We present our approach, depicted in Figure 3, for learning multi-modal representations of a 3D scene associated with language instruction.
Revised: We present our approach, depicted in Figure 3, for learning multi-modal representations of a 3D scene associated with language instruction.

Original: First, the proposed visual module enriches the 3D representations by utilizing the 2D synthetic images generator, namely the SIG module, described in
Revised: First, the proposed visual module enriches the 3D representations by utilizing the 2D synthetic images generator, namely the SIG module, described in

Original: Section 3.1.
Revised: Section 3.1.

Original: Thanks to it, we do not require access to any imagery sensor during training or testing.
Revised: Thanks to it, we do not require access to any imagery sensor during training or testing.

Original: Then, the multi-modal module learns joint representations targeted to the final task, i.e., referring to an object in the scene based on the language instruction.
Revised: Then, the multi-modal module learns joint representations targeted to the final task, i.e., referring to an object in the scene based on the language instruction.

Original: – Visual Module.
Revised: – Visual Module.

Original: Given a 3D scene S ∈ R N × 6 combined with agnostic multiple 3D object proposals x pci ∈ R N ′ × 6 , we aim to classify each object proposal by predicting the semantic labels z .
Revised: Given a 3D scene S ∈ R N × 6 combined with agnostic multiple 3D object proposals x pci ∈ R N ′ × 6 , we aim to classify each object proposal by predicting the semantic labels z .

Original: First, for each object i in the scene S , we generate multi-view synthetic 2D images x imgi ∈ R v × H × W ×from the 3D object’s point cloud x pc i , where v is the number of views for each object, H and W are the spatial dimensions for the generated images.
Revised: First, for each object i in the scene S , we generate multi-view synthetic 2D images x imgi ∈ R v × H × W ×from the 3D object’s point cloud x pci , where v is the number of views for each object, H and W are the spatial dimensions for the generated images.

Original: Second, we employ PointNet++ [35] as a 3D encoder, producing 3D features E pci ∈ R 1 × dim pc .
Revised: Second, we employ PointNet++ [35] as a 3D encoder, producing 3D features E pci ∈ R 1 × dim pc .

Original: Hence, the Tiny-ConvNext [28] is adopted as a 2D encoder, that produce y imgi ∈ R v × H ′ × W ′ × dim img .
Revised: Hence, the Tiny-ConvNext [28] is adopted as a 2D encoder, that produce y imgi ∈ R v × H ′ × W ′ × dim img .

Original: Where H ′ and W ′ represent a coarse spatial grid of spatial features, each represented by a dim img .
Revised: Where H ′ and W ′ represent a coarse spatial grid of spatial features, each represented by a dim img .

Original: Then, the spatial dimensions are squeezed producing o imgi ∈ R v × dim img .
Revised: Then, the spatial dimensions are squeezed producing o imgi ∈ R v × dim img .

Original: Since our 2D encoder encodes multiple views v for the same object i , a fusing mechanism is needed to fuse the views’ features.
Revised: Since our 2D encoder encodes multiple views v for the same object i , a fusing mechanism is needed to fuse the views’ features.

Original: Various possibilities are discussed to settle down on the best fusion mechanism.
Revised: Various possibilities are explored to settle down on the best fusion mechanism.

Original: We can interpret this as a fully connected layer to fully capture the views and cross-channel interactions.
Revised: We can interpret this as a fully connected layer to fully capture the views and cross-channel interactions.

Original: In contrast, to learn the views’ interactions and neglect the cross-channel relations, a 1D-Conv. could be utilized to avoid involving tremendous parameters as in the fully connected case.
Revised: In contrast, to learn the views’ interactions and neglect the cross-channel relations, a 1D-Conv. could be utilized to avoid involving tremendous parameters as in the fully connected case.

Original: Using the 1D-Conv.
Revised: Using the 1D-Conv.

Original: with k size = v as a fusion layer produce E imgi ∈ R 1 × dim img , where k size is the kernel size.
Revised: with k size = v as a fusion layer produce E imgi ∈ R 1 × dim img , where k size is the kernel size.

Original: The fused 2D features E imgi and the 3D features E pci are concatenated and fed to a visual transformer,  composed of several stacked transformer layers.
Revised: The fused 2D features E imgi and the 3D features E pci are concatenated and fed to a visual transformer, which is composed of several stacked transformer layers.

Original: The visual transformer is utilized to capture the cross-correlation between the 2D and the 3D features outputting refined features, V pc and V img , guided by three weighted auxiliary losses, i.e., alignment loss between V pc and V img , and two visual classification losses.
Revised: The visual transformer is utilized to capture the cross-correlation between the 2D, and the 3D features outputting refined features, V pc and V img , guided by three weighted auxiliary losses, i.e., alignment loss between V pc and V img , and two visual classification losses.

Original: 2D semantics.
Revised: 2D semantics.

Original: Inspired by SAT [52], we explore different 2D semantic information types combinations.
Revised: Inspired by SAT [52], we explore different 2D semantic information types combinations.

Original: Where we formulate our 2D semantics as follows:
Revised: Where we formulate our 2D semantics as follows:

Original: [Equation]
Revised: [Equation]

Original: where, z ′ i is the predicted class labels, W 1 are learned projection matrices, LN is layer normalization, and ϕ indicates concatenation.
Revised: where, z ′ i is the predicted class labels, W 1 are learned projection matrices, LN is layer normalization, and ϕ indicates concatenation.

Original: We encode each object individually using different semantic information types, i.e., visual semantics E img i , and geometry semantics x geo i ∈ R 1 × 30 .
Revised: We encode each object individually using different semantic information types, i.e., visual semantics E imgi , and geometry semantics x geoi ∈ R 1 × 30 .

Original: The geometry semantics, x geoi represents the bounding box coordinates of object i in the scene space and encode the virtual camera parameters, i.e., intrinsic and extrinsic parameters, used while generating the 2D image x imgi .
Revised: The geometry semantics, x geo i represents the bounding box coordinates of object i in the scene space and encode the virtual camera parameters, i.e., intrinsic and extrinsic parameters, used while generating the 2D image x imgi .

Original: Thus, it can be interpreted as a positional encoding for the vision transformer, where the interaction among objects is captured.
Revised: Thus, it can be interpreted as a positional encoding for the vision transformer, where the interaction among objects is captured.

Original: In contrast, SAT [52] utilizes a ground-truth 2D semantics, where the z ′ i denotes the manually annotated class labels, and the E imgi represents the embedding generated from real 2D images captured by imagery sensor [8].
Revised: In contrast, SAT [52] utilizes a ground-truth 2D semantics, where the z ′ i denotes the manually annotated class labels, and the E imgi represents the embedding generated from real 2D images captured by imagery sensor [8].

Original: – Multi-Modal Module.
Revised: – Multi-Modal Module.

Original: Using a pre-trained BERT model [11] followed by a language classification head adopted from Referit3D
Revised: Using a pre-trained BERT model [11] followed by a language classification head adopted from Referit3D

Original: [2], we embed words query Q .
Revised: [2], we embed words query Q .

Original: After embedding each of the visual features using the aforementioned visual module, we fuse the input modalities Q , V pc , V img simultaneously by using two different transformers; 3D Multi-Modal Transformer and 2D MultiModal Transformer.
Revised: After embedding each of the visual features using the aforementioned visual module, we fuse the input modalities Q , V pc , V img simultaneously by using two different transformers; 3D Multi-Modal Transformer and 2D MultiModal Transformer, as showen in Figure 3.

Original: In 3D Multi-Modal Transformer, we jointly encode the two input modalities V pc and Q to understand better the referring expression that captures co-occurrence relationships between different objects in the scene.
Revised: In 3D Multi-Modal Transformer, we jointly encode the two input modalities V pc and Q to understand better the referring expression that captures cooccurrence relationships between different objects in the scene.

Original: Meanwhile, the same words query Q are encoded with V img to enhance context-aware representation.
Revised: Meanwhile, the same words query Q are encoded with V img to enhance context-aware representation.

Original: A separate 2D and 3D referring heads are adopted from Referit3D, which are auxiliary tasks.
Revised: A separate 2D and 3D referring heads are adopted from Referit3D, which are auxiliary tasks.

Original: Finally, the output of the multi-modal transformers is processed by a Fusion Transformer that chooses the referred object from the fused features, F pc and F img .
Revised: Finally, the output of the multi-modal transformers is processed by a Fusion Transformer that chooses the referred object from the fused features, F pc and F img .

Original: 3.3 Losses
Revised: 3.3 Losses

Original: Our architecture, termed LAR, consists of two visual streams, i.e., 3D and 2D, in addition to one language stream.
Revised: Our architecture, termed LAR, consists of two visual streams, i.e., 3D and 2D, in addition to one language stream.

Original: To refer to the object correctly guided by the input utterance, we must first classify the objects accurately.
Revised: To refer to the object correctly guided by the input utterance, we must first classify the objects accurately.

Original: Thus, two auxiliary losses are employed to optimize the visual module; 3D classification loss L 3 Dcls and 2D classification loss L 2 Dcls .
Revised: Thus, two auxiliary losses are employed to optimize the visual module; 3D classification loss L 3 Dcls and 2D classification loss L 2 Dcls .

Original: In addition, We adopted Referit3D [2] auxiliary loss for language classification loss L langcls to cooperate in the visual grounding task by recognizing the referred object class according to the description.
Revised: In addition, We adopted Referit3D [2] auxiliary loss for language classification loss L langcls to cooperate in the visual grounding task by recognizing the referred object class according to the description.

Original: Hence, two auxiliary grounding losses ( L 3 DEref and L 2 DEref , termed as early referring losses, are exploited before the fusion transformer.
Revised: Hence, two auxiliary grounding losses ( L 3 DEref and L 2 DEref , termed as early referring losses, are exploited before the fusion transformer.

Original: As a result of these early referring losses, the model learns the correlation between language description and the visual features separately, which enhances the performance.
Revised: As a result of these early referring losses, the model learns the correlation between language description and the visual features separately, which enhances the performance.

Original: Additionally, we adopted object correspondence loss L cor from SAT to encourage the 3D model to distill the knowledge from the 2D stream.
Revised: Additionally, we adopted object correspondence loss L cor from SAT to encourage the 3D model to distill the knowledge from the 2D stream.

Original: Finally, two grounding losses, i.e., L 3 Dref and L 2 Dref were added after the fusion transformer to predict the scores of each proposal.
Revised: Finally, two grounding losses, i.e., L 3 Dref and L 2 Dref were added after the fusion transformer to predict the scores of each proposal.

Original: We optimize the whole model in an End-to-End manner, unlike
Revised: We optimize the whole model in an End-to-End manner, unlike

Original: SAT which learns the 2D features E imgi separately, with the following loss function:
Revised: SAT which learns the 2D features E imgi separately, with the following loss function:

Original: [Equation]
Revised: [Equation]

Original: where, λ cls is the object classification loss weight,
Revised: where, λ cls is the object classification loss weight,

Original: λ Eref is the early referring loss weight, λ ref is the final referring loss weight, λ cor is the object correspondence loss weight, and λ lang is the language classification loss weight.
Revised: λ Eref is the early referring loss weight, λ ref is the final referring loss weight, λ cor is the object correspondence loss weight, and λ lang is the language classification loss weight.

Original: Our implementation details are detailed in Appendix
Revised: Our implementation details are detailed in Appendix

Original:  Experiments 4.1 Datasets
Revised:  Experiments 4.1 Datasets

Original: We evaluate our proposed 3D visual grounding approach, LAR, on three benchmarking datasets,
Revised: We evaluate our proposed 3D visual grounding approach, LAR, on three benchmarking datasets,

Original: Nr3D [2], Sr3D [2], and ScanRefer [5].
Revised: Nr3D [2], Sr3D [2], and ScanRefer [5].

Original: Nr3D, 3D Natural reference dataset [2], consists of 41.5K natural, free-form utterances collected by humans using a referring game between two humans.
Revised: Nr3D, 3D Natural reference dataset [2], consists of 41.5K natural, free-form utterances collected by humans using a referring game between two humans.

Original: As opposed to Sr3D [2], which consists of 83,5K synthetic utterances.
Revised: As opposed to Sr3D [2], which consists of 83,5K synthetic utterances.

Original: Unlike Nr3D and Sr3D datasets,

Original: ScanRefer does not have the distractors concept making it relatively easier.

Original: In addition, it provides 51.5K utterances of 11K objects for 800 3D indoor scenes.
Revised: Consequently, ScanRefer provides 51.5K utterances of 11K objects for 800 3D indoor scenes.

Original: 4.2 Evaluation Metrics
Revised: 4.2 Evaluation Metrics

Original: we followed the same metrics followed by the previous work [2] [52], where three evaluation metrics are mainly used, i.e., the referring accuracy,the visual classification accuracy, andthe language classification accuracy.
Revised: we followed the same metrics followed by the previous work [2] [52], where three evaluation metrics are mainly used, i.e., the referring accuracy, visual classification accuracy, and language classification accuracy.

Original: In the case of Nr3d and Sr3d datasets, we followed the previous work convention, assuming the proposals are generated with GT objects.
Revised: In the case of Nr3d and Sr3d datasets, we followed the previous work convention, assuming the proposals are generated with GT objects.

Original: Following this assumption, the 3D grounding problem is reformulated as a classification problem, where the model objective is to classify the referred object among the other distractors correctly, termed referring accuracy.
Revised: Following this assumption, the 3D grounding problem is reformulated as a classification problem, where the model objective is to classify the referred object among the other objects correctly, termed referring accuracy.

Original: The referring accuracy is calculated based on whether the model picks the correct proposal from a set of X proposals.
Revised: The referring accuracy is calculated based on whether the model picks the correct proposal from a set of X proposals.

Original: Alongside the main objective, which is the referring accuracy, two auxiliary metrics are defined to evaluate the performance of each stream individually, i.e., the visual classification accuracy and the language classification accuracy.
Revised: Alongside the main objective, which is the referring accuracy, two auxiliary metrics are defined to evaluate the performance of each stream individually, i.e., the visual classification accuracy and the language classification accuracy.

Original: Given agnostic GT-proposals, the visual classification accuracy is measured based on whether the model predicts the object’s class correctly or not from a predefined set of possible classes.
Revised: Given agnostic GT-proposals, the visual classification accuracy is measured based on whether the model predicts the object’s class correctly or not from a predefined set of possible classes.

Original: Similarly, language classification accuracy is another auxiliary metric that assesses the performance of the linguistic branch.
Revised: Similarly, language classification accuracy is another auxiliary metric that assesses the performance of the linguistic branch.

Original: This branch aims to correctly predict the class of the referred object based on the input utterance.
Revised: This branch aims to correctly predict the class of the referred object based on the input utterance.

Original: 4.3 Implementation Details
Revised: 4.3 Implementation Details

Original: – Input Configuration.
Revised: – Input Configuration.

Original: For the 3D input, we randomly sample 1024 points per proposal from its point cloud segment.
Revised: For the 3D input, we randomly sample 1024 points per proposal from its point cloud segment.

Original: Furthermore, for the synthetic 2D input, we follow Algorithm 1, which is scrutinized in Section 3.1.
Revised: Furthermore, for the synthetic 2D input, we follow Algorithm 1, which is scrutinized in Section 3.1.

Original: During the testing phase, the camera is set 2 meters away from the prominent face l ′ , i.e., d f = 2 , and 1 meter away from the floor, i.e., d up = 1 .
Revised: During the testing phase, the camera is set 2 meters away from the prominent face l ′ , i.e., d f = 2 , and 1 meter away from the floor, i.e., d up = 1 .

Original: In addition, the region of interest extension ratio ϵ , depicted in Figure 2 part (C), is set to 2%.
Revised: In addition, the region of interest extension ratio ϵ , depicted in Figure 2 part (C), is set to 2%.

Original: The generated image is a square with a size of 32 pixels.
Revised: The generated image is a square with a size of 32 pixels.

Original: Five views are generated per object, where the angle θ between each camera is 30 ◦ , demonstrated in Figure 1.
Revised: Five views are generated per object, where the angle θ between each camera is 30 ◦ , demonstrated in Figure 1.

Original: Thus the camera is set to the following angles ( 0 ◦ , 30 ◦ , − 30 ◦ , 60 ◦ , − 60 ◦ ).
Revised: Thus the camera is set to the following angles ( 0 ◦ , 30 ◦ , − 30 ◦ , 60 ◦ , − 60 ◦ ).

Original: [Table] [Table]
Revised: [Table] [Table]

Original: – Network and Training Configuration.
Revised: – Network and Training Configuration.

Original: A pre-trained model on the ImageNet dataset [9], TinyConvNext [28], is used to extract the features of our synthetic 2D images.
Revised: A pre-trained model on the ImageNet dataset [9], TinyConvNext [28], is used to extract the features of our synthetic 2D images.

Original: Our 2D backbone is shared across different views.
Revised: Our 2D backbone is shared across different views.

Original: We adopt the same 3D encoder in the previous works [1] [2] [13] [37] [18] [52], PointNet++ [35], to encode the sampled points per proposal.
Revised: We adopt the same 3D encoder in the previous works [1] [2] [13] [37] [18] [52], PointNet++ [35], to encode the sampled points per proposal.

Original: For the language encoder, we adopt the BERT encoder [11] with three layers.
Revised: For the language encoder, we adopt the BERT encoder [11] with three layers.

Original: We adopted the same transformer architecture used in [52] [51] for our visual, multi-modal, and fusion transformers.
Revised: We adopted the same transformer architecture used in [52] [51] for our visual, multi-modal, and fusion transformers.

Original: All models are trained for 100 epochs from scratch, using the weight initialization strategy described in [14].
Revised: All models are trained for 100 epochs from scratch, using the weight initialization strategy described in [14].

Original: As mentioned in SAT [52], the initial learning rate is set to 10 − 4 and decreases by 0.65 every ten epochs .
Revised: As mentioned in SAT [52], the initial learning rate is set to 10 − 4 and decreases by 0.65 every ten epochs .

Original: Adam optimizer [21] and mini-batch size of 32 per GPU are used for training all our models.
Revised: Adam optimizer [21] and mini-batch size of 32 per GPU are used for training all our models.

Original: We set the losses weights as follows,
Revised: We set the losses weights as follows,

Original: λ cls = 5 , λ Eref = 0 .
Revised: λ cls = 5 , λ Eref = 0 .

Original: 5 , λ ref = 5 , λ cor = 10 , and λ lang = 0 .5 .
Revised: 5 , λ ref = 5 , λ cor = 10 , and λ lang = 0 .

Original: – Software and Hardware Details.
Revised: – Software and Hardware Details

Original: We use the python augmentation framework, Albumentations [4], for image augmentation.
Revised: We use the python augmentation framework, Albumentations [ ], for image augmentation.

Original: In order to perform an online augmentation while training, we implemented our 2D Synthetic Images Generator (SIG), described at Section 3.1, in a pure Pythonic approach without using any 3D libraries which also fasten the training.
Revised: In order to perform an online augmentation while training, we implemented our 2D Synthetic Images Generator (SIG), described in Section 3.1, in a pure Pythonic approach without using any 3D libraries, which also speedup the training.

Original: Our module is implemented in Python using the PyTorch framework and four Nvidia V100 GPUs.
Revised: Our module is implemented in Python using the PyTorch framework and four Nvidia V100 GPUs.

Original: 4.4 Ablation studies
Revised: 4.4 Ablation studies

Original: We conducted five ablation studies to assess the proposed architecture and to validate the contribution of each component to achieve good performance.
Revised: We conducted five ablation studies to assess the proposed architecture and to validate the contribution of each component to achieve good performance.

Original: These components are: 1) 2D multi-views and 2D resolution.
Revised: These components are 1) 2D multi-views and 2D resolution.

Original: Types of 2D semantic information.
Revised: Types of 2D semantic information.

Original: 3) Camera vs. image augmentation 4) SIG Module vs. 2D real clues.
Revised: 3) Camera vs. image augmentation 4) SIG Module vs. 2D real clues.

Original: 5) Noisy Cameras Extrinsic and Intrinsic.
Revised: 5) Noisy Cameras Extrinsic and Intrinsic.

Original: More ablations are stated in Appendix 5.
Revised: More ablations are stated in Appendix 5.

Original: – 2D multi-views and 2D resolution.
Revised: – 2D multi-views and 2D resolution.

Original: To assess our 2D Synthetic Images Generator (SIG), we have deliberately removed the 3D stream, colored with yellow in Figure 3; 3D encoder, 3D heads, and rely only on our generated 2D images only while tackling the referring task.
Revised: To assess our 2D Synthetic Images Generator (SIG), we have deliberately removed the 3D stream, colored with yellow in Figure 3; 3D encoder, 3D heads, and rely only on our generated 2D images only while tackling the referring task.

Original: Thus, our architecture is simplified to include only the 2D encoder, 2D multi-modal transformer, and 2D heads.
Revised: Thus, our architecture is simplified to include only the 2D encoder, 2D multi-modal transformer, and 2D heads.

Original: We
Revised: We

Original: trained this 2D architecture variant with different image resolutions and multi-view setup combinations
Revised: trained this 2D architecture variant with different image resolutions and multi-view setup combinations

Original: As shown in Table 2, The referring accuracy increased linearly while increasing the image size or increasing
Revised: As shown in Table 2, The referring accuracy increased linearly while increasing the image size or increasing

Original: the
Revised: the

Original: number of
Revised: number of

Original: views
Revised: views

Original: for the projected images.
Revised: for the projected images.

Original: However, our multi-view setup has the dominant effect, whereas using only 64 image resolution with five
Revised: However, our multi-view setup has the dominant effect, whereas using only 64 image resolution with five

Original: views
Revised: views

Original: surpassing one
Revised: surpassing one

Original: shot
Revised: shot

Original: with 128 resolution.
Revised: with 128 resolution.

Original: Also, we found that using the combination of five different views, each hasresolution, surpassing Referit3D and non-SAT variant, where both use 3D semantics only, which shows the quality of the learned representation via our novel synthetic 2D images generator (SIG).
Revised: Also, we found that using the combination of five different views, each hasresolution, surpassing Referit3D and non-SAT variant, where both use 3D semantics only, which shows the quality of the learned representation via our novel synthetic 2D images generator (SIG).

Original: – Types of 2D semantic information.
Revised: – Types of 2D semantic information.

Original: Inspired by SAT [52], we dissect the role of different 2D semantic, following Eq. 1.
Revised: Inspired by SAT [52], we dissect the role of different 2D semantic, following Eq. 1.

Original: As shown in Table 3, the first row (a) represents a 3D variant of our architecture where only the 3D encoder, 3D multi-modal transformer, and 3D heads are included.
Revised: As shown in Table 3, the first row (a) represents a 3D variant of our architecture where only the 3D encoder, 3D multi-modal transformer, and 3D heads are included.

Original: Therefore, this variant mimics the Non-SAT [52] configurations.
Revised: Therefore, this variant mimics the Non-SAT [52] configurations.

Original: Starting from row b to row e, our synthetic 2D semantics are incorporated alongside the 3D semantics.
Revised: Starting from row b to row e, our synthetic 2D semantics are incorporated alongside the 3D semantics.

Original: Also, we compare against SAT [52] to highlight the critical differences in the meaning of each semantic information.
Revised: Also, we compare against SAT [52] to highlight the critical differences in the meaning of each semantic information.

Original: ROI in our architecture is the synthetic 2D image generated from the 3d point clouds using our SIG module.
Revised: ROI in our architecture is the synthetic 2D image generated from the 3d point clouds using our SIG module.

Original: In contrast, SAT [52] uses the original ScanNet images during the training phase.
Revised: In contrast, SAT [52] uses the original ScanNet images during the training phase.

Original: Unlike SAT, which uses the ground-truth classification labels during the training and masks them during testing, we use
Revised: Unlike SAT, which uses the ground-truth classification labels during the training and masks them during testing, we use

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: the predicted labels from the 2D classification head z ′ .
Revised: the predicted labels from the 2D classification head z ′ .

Original: Accordingly, we do not need to mask the 2D semantics, as we are using synthetic ones.
Revised: Accordingly, we do not need to mask the 2D semantics, as we are using synthetic ones.

Original: As shown in row (e), the best accuracy is achieved when we jointly incorporate the three semantic types.
Revised: As shown in row (e), the best accuracy is achieved when we jointly incorporate the three semantic types.

Original: We also notice a significant superiority in classification accuracy where our synthetic 2D images assist the 3D classifier in both phases; training and testing.
Revised: We also notice a significant superiority in classification accuracy where our synthetic 2D images assist the 3D classifier in both phases; training and testing.

Original: – Camera vs. Image Augmentation.
Revised: – Camera vs. Image Augmentation.

Original: We have two kinds of augmentation during training: 1) Image augmentation.
Revised: We have two kinds of augmentation during training: 1) Image augmentation.

Original: 2) Camera augmentation.
Revised: 2) Camera augmentation.

Original: For image augmentation, we followed the conventional techniques used by the major computer vision applications like horizontal and vertical flipping, blurring, color jitter, random scaling, random shifting, and random cropping.
Revised: For image augmentation, we followed the conventional techniques used by the major computer vision applications like horizontal and vertical flipping, blurring, color jitter, random scaling, random shifting, and random cropping.

Original: For camera augmentation, we range d f from 1.5 to 4 meters, d up from 0.5 to 2.5 meters, and ϵ from 2% to 15%.
Revised: For camera augmentation, we range d f from 1.5 to 4 meters, d up from 0.5 to 2.5 meters, and ϵ from 2% to 15%.

Original: By changing the variables mentioned above, the extrinsic camera parameters are augmented.
Revised: By changing the variables mentioned above, the extrinsic camera parameters are augmented.

Original: In addition, random noise is added to the intrinsic camera parameters.
Revised: In addition, random noise is added to the intrinsic camera parameters.

Original: Table 4 demonstrates the effect of each augmentation type alone and the effect of fusing both of them.
Revised: Table 4 demonstrates the effect of each augmentation type alone and the effect of fusing both of them.

Original: The best accuracy is achieved by utilizing the two augmentation types.
Revised: The best accuracy is achieved by utilizing the two augmentation types.

Original: – SIG Module vs. 2D real clues.
Revised: – SIG Module vs. 2D real clues.

Original: To validate our SIG module, we tailored a variant of our architecture, where our SIG module is replaced by the actual 2D clues produced by Faster R-CNN [39].
Revised: To validate our SIG module, we tailored a variant of our architecture, where our SIG module is replaced by the actual 2D clues produced by Faster R-CNN [39].

Original: As shown in Table 5, replacing our
Revised: As shown in Table 5, replacing our

Original: SIG module with the features produced by Faster R-CNN [39] degrades the performance by 2%.
Revised: SIG module with the features produced by Faster R-CNN [39] degrades the performance by 2%.

Original: In addition, even when we compared unfairly to methods that use additional 2D information, even the variant of SAT, which utilize real images during training and testing (row #2), we still outperform it by 1%.
Revised: In addition, even when we compared unfairly to methods that use additional 2D information, even the variant of SAT, which utilize real images during training and testing (row #2), we still outperform it by 1%.

Original: – Noisy Cameras Extrinsic and Intrinsic.
Revised: – Noisy Cameras Extrinsic and Intrinsic.

Original: Due to the significant improvement introduced by utilizing the camera augmentation, as shown in Table 4, a detailed analysis is conducted to explore the robustness against the camera’s variations.
Revised: Due to the significant improvement introduced by utilizing the camera augmentation, as shown in Table 4, a detailed analysis is conducted to explore the robustness against the camera’s variations.

Original: Consequently, our 2D synthetic branch learns how to fuse information across different cameras in a data-driven way.
Revised: Consequently, our 2D synthetic branch learns how to fuse information across different cameras in a data-driven way.

Original: Therefore, we can train the model to be robust to noise camera models, as shown in Table
Revised: Therefore, we can train the model to be robust to noise camera models, as shown in Table

Original: Both cameras dying and noisy parameters experiments could be thought as irrelevant as LAR utilizes virtual cameras.
Revised: Both cameras dying and noisy parameters experiments could be thought as irrelevant as LAR utilizes virtual cameras.

Original: Thus, no camera dropout nor noise can be introduced.
Revised: Thus, no camera dropout nor noise can be introduced.

Original: However, our SIG module, depicted in Section 3.1, can be utilized in a wide range of applications when these robustness characteristics against such noise are essential for these applications.
Revised: However, our SIG module, depicted in Section 3.1, can be utilized in a wide range of applications when these robustness characteristics against such noise are essential for these applications.

Original: For instance, in the autonomous driving world, our SIG module can be employed to convert dense lidar point clouds into synthetic images, which will be used to train an arbitrary neural network that tackles specific tasks (e.g., object segmentation in cocoon setup [31] [47]).
Revised: For instance, in the autonomous driving world, our SIG module can be employed to convert dense lidar point clouds into synthetic images, which will be used to train an arbitrary neural network that tackles specific tasks (e.g., object segmentation in cocoon setup [31] [47]).

Original: Then, the model will be tested using real cameras instead of generating synthetic imagery input from the lidar.
Revised: Then, the model will be tested using real cameras instead of generating synthetic imagery input from the lidar.

Original: [Table]
Revised: [Table]

Original: 4.5 Comparison to State-of-the-art
Revised: 4.5 Comparison to State-of-the-art

Original: Moreover
Revised: Moreover

Original: [Table]
Revised: [Table]

Original: we achieve state-of-the-art results on the three well-known 3D visual grounding benchmarks, i
Revised: we achieve state-of-the-art results on the three well-known 3D visual grounding benchmarks, i

Original: [Table]
Revised: [Table]

Original: e., Nr3D [2], Sr3D [2]
Revised: e., Nr3D [2], Sr3D [2]

Original: [Table]
Revised: [Table]

Original: and ScanRefer [5].
Revised: and ScanRefer [5].

Original: By effectively utilizing our 2D synthetic images,
Revised: By effectively utilizing our 2D synthetic images,

Original: LAR outperforms all the existing methods, which do not use any additional training dataset, by a large margin of 5.0 % and 1.9 % on Nr3D and Sr3D, respectively, depicted in Table 9.
Revised: LAR outperforms all the existing methods, which do not use any additional training dataset, by a large margin of 5.0 % and 1.9 % on Nr3D and Sr3D, respectively, depicted in Table 9.

Original: Where,
Revised: Where,

Original: LanguageRefer [37] achieves 43.9%, while LAR achieves 48.9% on Nr3D. For Sr3D, our model achieves 59.35%, which is a 1.9 % increase over TransRefer3D [13].
Revised: LanguageRefer [37] achieves 43.9%, while LAR achieves 48.9% on Nr3D. For Sr3D, our model achieves 59.35%, which is a 1.9 % increase over TransRefer3D [13].

Original: In addition, for the ScanRefer [2] benchmark, we also achieve state-of-the-art results, by boosting the accuracy by 2.3 % from the closest counterpart architecture; SAT [52], mentioned in Table 7.
Revised: In addition, for the ScanRefer [2] benchmark, we also achieve state-of-the-art results, by boosting the accuracy by 2.3 % from the closest counterpart architecture; SAT [52], mentioned in Table 7.

Original: We retrained SAT [52] for a fair comparison, as we notice difficulties reproducing and verifying SAT results 1 .
Revised: We retrained SAT [52] for a fair comparison, as we notice difficulties reproducing and verifying SAT results 1 .

Original: Although we do not use an extra input during training as SAT, which uses the original
Revised: Although we do not use an extra input during training as SAT, which uses the original

Original: ScanNet 2D images [8], we surpass SAT by 1.6 % , 2.7 % , and 2.3 % on Nr3D, Sr3D, and ScanRefer, respectively.
Revised: ScanNet 2D images [8], we surpass SAT by 1.6 % , 2.7 % , and 2.3 % on Nr3D, Sr3D, and ScanRefer, respectively.

Original: 4.6 Discussion and Analysis
Revised: 4.6 Discussion and Analysis

Original: First, we will demonstrate our qualitative analysis.
Revised: First, we will demonstrate our qualitative analysis.

Original: Then, wehave conducted two experiments to probe the robustness of the representations learned by our proposed module LAR.
Revised: Then, we conducted two experiments to probe the robustness of the representations learned by our proposed approach LAR.

Original: – Qualitative Analysis.
Revised: – Qualitative Analysis.

Original: Figure 5 depicts two corner cases in our projection module (SIG).
Revised: Figure 5 depicts two corner cases in our projection module (SIG).

Original: While assigning the projected 3D points into a 2D grid, more than one point may share the exact spatial location on the grid, which means only one will be stored in the grid, and the rest of the points will be discarded.
Revised: While assigning the projected 3D points into a 2D grid, more than one point may share the exact spatial location on the grid, which means only one will be stored in the grid, and the rest of the points will be discarded.

Original: As shown in part (A) in Figure 5, on the left column, the floor overlaps the other objects due to the conflict between points during the assignation step.
Revised: As shown in part (A) in Figure 5, on the left column, the floor overlaps the other objects due to the conflict between points during the assignation step.

Original: To overcome this behavior, a simple priority rank is given for each point based on its height.
Revised: To overcome this behavior, a simple priority rank is given for each point based on its height.

Original: Part (B) shows another failure case, where the prominent face is falsely localized, which causes placing the camera outside the room or inside other objects
Revised: Part (B) shows another failure case, where the prominent face is falsely localized, which causes placing the camera outside the room or inside other objects.

Original: Therefore, we localize the prominent face relative to the room center to mitigate this effect
Revised: Therefore, we localize the prominent face relative to the room center to mitigate this effect.

Original: An example of an ambiguous description is shown in Figure 6 part A.
Revised: An example of an ambiguous description is shown in Figure 6 part A.

Original: The model incorrectly predicts "the chair on the right rather than the chair on the left."
Revised: The model incorrectly predicts "the chair on the right rather than the chair on the left."

Original: The model and the human annotations are occasionally confusing to referring to the objects in the view-dependent description.
Revised: The model and the human annotations are occasionally confusing for referring to the objects in the view-dependent description.

Original: While successful cases can be seen in Figure 6 part B. For more qualitative visualization, please refer to Appendix 5.
Revised: While successful cases can be seen in Figure 6 part B. For more qualitative visualization, please refer to Appendix 5.

Original: – Evaluation with Ground-Truth Class Labels.
Revised: – Evaluation with Ground-Truth Class Labels.

Original: Table 8 shows another variant, which is tailored from our architecture to be compared against SAT-GT.
Revised: Table 8 shows another variant, which is tailored from our architecture to be compared against SAT-GT.

Original: Where the predicted class labels are replaced by the ground-truth ones.
Revised: Where the predicted class labels are replaced by the ground-truth ones.

Original: Our LAR architecture outperforms SAT-GT by 12%.
Revised: Our LAR architecture outperforms SAT-GT by 12%.

Original: – Zero-shot Cameras Dying Testing.
Revised: – Zero-shot Cameras Dying Testing.

Original: We deliberately drop the cameras during testing to verify the robustness of our learned representations against camera variations.
Revised: We deliberately drop the cameras during testing to verify the robustness of our learned representations cases.

Revised: Part A shows failure in assignation step for the projected 3D points the 2D grid

Revised: Part B demonstrates failure in the prominent face against camera variations.

Original: As shown in Figure 4, starting from our best model, which is trained using five cameras, the referring accuracy drops to 44.6% then 40.15%, when we only use three cameras then one camera during testing, respectively.
Revised: As shown in Figure 4, starting from our best model, which is trained using five cameras, the referring accuracy drops to 44.6% then 40.15%, when we only use three cameras then one camera during testing, respectively.

Original: Driven by these results, camera dropout techniques can be invoked to overcome the performance drop, by deliberately drop a random camera during the training; replace its input by a duplicate view of another camera.
Revised: Driven by these results, camera dropout techniques can be invoked to overcome the performance drop, by deliberately drop a random camera during the training; replace its input by a duplicate view of another camera.

Original: – Limitations.
Revised: – Limitations.

Original: Following the standard setup of Referit3D [2], we assume access to the object proposals of each 3D scene, similar to SAT [52], LanguageRefer [37], TransRefer3D [13], and 3DVG-Trans [59].
Revised: Following the standard setup of Referit3D [2], we assume access to the object proposals of each 3D scene, similar to SAT [52], LanguageRefer [37], TransRefer3D [13], and 3DVG-Trans [59].

Original: Therefore, we use the ground-truth proposals in our experiments.
Revised: Therefore, we use the ground-truth proposals in our experiments.

Original: Following this assumption, the 3D grounding problem is reformulated as a classification problem, where the model objective is to classify the referred object among the other distractors correctly.
Revised: Following this assumption, the 3D grounding problem is reformulated as a classification problem, where the model objective is to classify the referred object among the other distractors correctly.

Original: However, LAR is compatible with the setting that uses detected-based proposals.
Revised: However, LAR is compatible with the setting that uses detected-based proposals.

Original:  Conclusion
Revised:  Conclusion

Original: We propose an efficient 3D grounding architecture that leveragesthe 2D knowledge to reinforce the learned 3D representations.
Revised: We propose an efficient 3D grounding architecture that leverages 2D knowledge to reinforce the learned 3D representations.

Original: To avoid requiring 2D semantics neither during training nor testing, we introduce a 2D synthetic images generator, termed SIG.
Revised: To avoid requiring 2D semantics neither during training nor testing, we introduce a 2D synthetic image generator, termed SIG.

Original: Our SIG module generates high-quality 2D synthetic images.
Revised: Our SIG module generates high-quality 2D synthetic images.

Original: LAR, 2D Synthetic Semantics Knowledge Distillation for 3D Visual Grounding, consists of two stages, i.e., Visual and Multi-Modal modules.
Revised: LAR, 2D Synthetic Semantics Knowledge Distillation for 3D Visual Grounding, consists of two stages, i.e., Visual and Multi-Modal modules.

Original: LAR preserves the advantages of distilling the synthetic 2D knowledge to the 3D stream without requiring any additional information.
Revised: LAR preserves the advantages of distilling the synthetic 2D knowledge to the 3D stream without requiring any additional information.

Original: Our experiments show a significant superior performance, where LAR achieves state-of-the-art results across three different 3D grounding benchmarks by outperforming the existing approaches, which do not rely on extra training data, by a wide margin, i.e., 5.0 % , 1.9 % and 2.3 % on Nr3D, Sr3D and ScanRefer, respectively.
Revised: Our experiments show a significant superior performance, where LAR achieves state-of-the-art results across three different 3D grounding benchmarks by outperforming the existing approaches, which do not rely on extra training data, by a large margin, i.e., 5.0 % , 1.9 % and 2.3 % on Nr3D, Sr3D and ScanRefer, respectively.

Original: Hence, despite the unfair comparison with SAT, which uses extra 2D images, LAR surpasses SAT by 1.6 % , 2.7 % , and 2.3 % on Nr3D, Sr3D, and ScanRefer, respectively.
Revised: Hence, despite the unfair comparison with SAT, which uses extra 2D images, LAR surpasses SAT by 1.6 % , 2.7 % , and 2.3 % on Nr3D, Sr3D, and ScanRefer, respectively.

Original: Furthermore, we verify the robustness of LAR and its generalization ability via zero-shot experiments to intrinsic and extrinsic camera parameters and dying cameras.
Revised: Furthermore, we verify the robustness of LAR and its generalization ability via zero-shot experiments to intrinsic and extrinsic camera parameters and dying cameras.

Original: More qualitative results are attached in the supplementary materials.
Revised: More qualitative results are attached in the supplementary materials.

Original: Acknowledgments and Disclosure of Funding
Revised: Acknowledgments and Disclosure of Funding

Original: This work is funded by KAUST BAS/1/1685-01-0 and is also partially supported by the SDAIAKAUST Center of Excellence in Data Science and Artificial Intelligence (SDAIA-KAUST AI).
Revised: This work is funded by KAUST BAS/1/1685-01-0 and is also partially supported by the SDAIAKAUST Center of Excellence in Data Science and Artificial Intelligence (SDAIA-KAUST AI).

Original: Checklist
Revised: Checklist

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] Please refer to Section 4.
Revised: [Yes] Please refer to Section 4.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: [N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 2.
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A] (b) Did you include complete proofs of all theoretical results?
Revised: [N/A] (b) Did you include complete proofs of all theoretical results?

Original: [N/A] 3.
Revised: [N/A] 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] Our codes are attached in the supplemental material combined with a detailed ’readme’ file.
Revised: [Yes] Our codes are attached in the supplemental material combined with a detailed ’readme’ file.

Original: Also, refer to Section 4.3 for the implementation details.
Revised: Also, refer to Section 4.3 for the implementation details.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] Our codes are attached in the supplemental material combined with a detailed ’readme’ file.
Revised: [Yes] Our codes are attached in the supplemental material combined with a detailed ’readme’ file.

Original: Also, refer to Section 4.3 for the implementation details.
Revised: Also, refer to Section 4.3 for the implementation details.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] Refer to Table 9.
Revised: [Yes] Refer to Table 9.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] Refer to Section 4.3.
Revised: [Yes] Refer to Section 4.3.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] (b) Did you mention the license of the assets?
Revised: [Yes] (b) Did you mention the license of the assets?

Original: [Yes] (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: [Yes] (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
Revised: [Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [Yes] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Revised: [Yes] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]
Revised: [N/A]

Original: A Appendix
Revised: A Appendix

Original: Our Supplemental Material contains the following sections:
Revised: Our Supplemental Material contains the following sections:

Original: • Detector-Generated 3D Proposal.
Revised: • Detector-Generated 3D Proposal.

Original: • Computational Cost.
Revised: • Computational Cost.

Original: • Demonstrate our synthetic images generator algorithm (SIG).
Revised: • Demonstrate our synthetic images generator algorithm (SIG).

Original: • Implementation Details.
Revised: • Implementation Details.

Original: • Evaluation Metrics • More ablations.
Revised: • Evaluation Metrics • More ablations.

Original: • Qualitative results.
Revised: • Qualitative results.

Original: A.1 Detector-Generated 3D Proposal
Revised: A.1 Detector-Generated 3D Proposal

Original: Following the standard setup of Referit3D [2], we assume access to the object proposals of each 3D scene, similar to SAT [52], LanguageRefer [37], TransRefer3D [13], and 3DVG-Trans [59].
Revised: Following the standard setup of Referit3D [2], we assume access to the object proposals of each 3D scene, similar to SAT [52], LanguageRefer [37], TransRefer3D [13], and 3DVG-Trans [59].

Original: Therefore, we use the ground-truth proposals in our experiments.
Revised: Therefore, we use the ground-truth proposals in our experiments.

Original: Following this assumption, the 3D grounding problem is reformulated as a classification problem, where the model objective is to classify the referred object among the other distractors correctly.
Revised: Following this assumption, the 3D grounding problem is reformulated as a classification problem, where the model objective is to classify the referred object among the other distractors correctly.

Original: However, LAR is compatible with the setting that uses detected-based proposals.
Revised: However, LAR is compatible with the setting that uses detected-based proposals.

Original: To this end, we experimented our LAR module with the detector-generated 3D proposals using VoteNet detector.
Revised: To this end, we experimented our LAR module with the detector-generated 3D proposals using VoteNet detector.

Original: As shown in Figure 7, we adapted the
Revised: As shown in Figure 7, we adapted the

Original: Scanrefer architecture as follows: 1) 2D Encoder: Integrating a 2D encoder right after the predicted box proposals, i.e., SIG module and Tiny-ConvNext in our case, and Faster R-CNN [1] in SAT case.
Revised: Scanrefer architecture as follows: 1) 2D Encoder: Integrating a 2D encoder right after the predicted box proposals, i.e., SIG module and Tiny-ConvNext in our case, and Faster R-CNN [1] in SAT case.

Original: 2) Fusion Transformer:
Revised: 2) Fusion Transformer:

Original: In LAR case, we added our visual transformer only as we aim to emphasize the contribution of our SIG module only.
Revised: In LAR case, we added our visual transformer only as we aim to emphasize the contribution of our SIG module only.

Original: While in SAT case, we convert their multi-modal transformer to vision transformer by excluding the language embedding from it.
Revised: While in SAT case, we convert their multi-modal transformer to vision transformer by excluding the language embedding from it.

Original: The same 3D prediction modules and language modules are adopted from Scanrefer, i.e., P++ and Voting module, and GloVe and GRU, respectively.
Revised: The same 3D prediction modules and language modules are adopted from Scanrefer, i.e., P++ and Voting module, and GloVe and GRU, respectively.

Original: We train three models: 1) The best variant of Scanrefer. 2)
Revised: We train three models: 1) The best variant of Scanrefer. 2)

Original: SAT-Detection-based architecture.
Revised: SAT-Detection-based architecture.

Original: 3) LAR-Detection-based architecture.
Revised: 3) LAR-Detection-based architecture.

Original: We first train the Scanrefer for 50 epochs then freeze their 3D prediction modules and language modules for two epochs to train the new blocks added in SAT and LAR detection based architectures.
Revised: We first train the Scanrefer for 50 epochs then freeze their 3D prediction modules and language modules for two epochs to train the new blocks added in SAT and LAR detection based architectures.

Original: Then, we train the whole architecture for two more epochs.
Revised: Then, we train the whole architecture for two more epochs.

Original: For more details, please refer to the "Detector-Generated 3D Proposal" section in Appendix A in the revised paper version, where we added a figure to demonstrate the adaptations done on the ScanRefer architecture.
Revised: For more details, please refer to the "Detector-Generated 3D Proposal" section in Appendix A in the revised paper version, where we added a figure to demonstrate the adaptations done on the ScanRefer architecture.

Original: As shown in Table 10, the tailored LAR-Detection-based architecture outperforms both the original
Revised: As shown in Table 10, the tailored LAR-Detection-based architecture outperforms both the original

Original: ScanRefer and the SAT-Detection-based architectures.
Revised: ScanRefer and the SAT-Detection-based architectures.

Original: Also, the results show that SAT is more vulnerable to the distortions in the detected proposals.
Revised: Also, the results show that SAT is more vulnerable to the distortions in the detected proposals.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: A.2 Computational Cost
Revised: A.2 Computational Cost

Original: We compared our complexity against SAT, during both the training and testing.
Revised: We compared our complexity against SAT, during both the training and testing.

Original: As shown in Table 11, our training and inference time are 3.6 x and 0.7 x compared to SAT.
Revised: As shown in Table 11, our training and inference time are 3.6 x and 0.7 x compared to SAT.

Original: All the reported results are measured on single GTX-1080 GPU.
Revised: All the reported results are measured on single GTX-1080 GPU.

Original: A.3 Synthetic Images Generator Algorithm
Revised: A.3 Synthetic Images Generator Algorithm

Original: Algorithm 1, demonstrates a pseudo-code for our
Revised: Algorithm 1, demonstrates a pseudo-code for our

Original: SIG module that projects arbitrary 3D
Revised: SIG module that projects arbitrary 3D

Original: Point clouds representing an object to a 2D image.
Revised: Point clouds representing an object to a 2D image.

Original: Algorithm
Revised: Algorithm

Original: Input : x pci : Point-cloud for an object i in the scene,
Revised: Input : x pci : Point-cloud for an object i in the scene,

Original: I : The pinhole camera intrinsic parameters,
Revised: I : The pinhole camera intrinsic parameters,

Original: O s : Scene center.
Revised: O s : Scene center.

Original: Output : x imgi : 2D synthetic image that represents the object i .
Revised: Output : x imgi : 2D synthetic image that represents the object i .

Original: 1: // Get the prominent face: 2: minDist ← Dist ( O l 0 , O s ) 3: for each face do 4: if Dist ( O l k , O s ) <
Revised: 1: // Get the prominent face: 2: minDist ← Dist ( O l 0 , O s ) 3: for each face do 4: if Dist ( O l k , O s ) <

Original: = minDist then 5: l ′ ← l k 6: minDist ← Dist ( O l k , O s ) 7: end if 8: end for 9: Cam p =
Revised: = minDist then 5: l ′ ← l k 6: minDist ← Dist ( O l k , O s ) 7: end if 8: end for 9: Cam p =

Original: [ d fx , d fy , d up ] 10: // Build transformation matrix M ∈ R 4 ×11: V f ← ∥
Revised: [ d fx , d fy , d up ] 10: // Build transformation matrix M ∈ R 4 ×11: V f ← ∥

Original: Cam p − O l ′ ∥ 12: V r ← ∥
Revised: Cam p − O l ′ ∥ 12: V r ← ∥

Original: V f ⊗ [0 , 0 , 1] ∥ 13: V up ← ∥ V r ⊗ V f ∥ 14: M ← (cid:20) V r
Revised: V f ⊗ [0 , 0 , 1] ∥ 13: V up ← ∥ V r ⊗ V f ∥ 14: M ← (cid:20) V r

Original: V up
Revised: V up

Original: V f
Revised: V f

Original: Cam p   (cid:21) .
Revised: Cam p   (cid:21) .

Original: A.4 Ablations Studies
Revised: A.4 Ablations Studies

Original: A.4.1 Different 2D semantic fusion techniques.
Revised: A.4.1 Different 2D semantic fusion techniques.

Original: Driven by the aforementioned results in Table 2, we use five virtual cameras set in a cocoon setup around each object as depicted in Figure 1, and the image resolution is 128.
Revised: Driven by the aforementioned results in Table 2, we use five virtual cameras set in a cocoon setup around each object as depicted in Figure 1, and the image resolution is 128.

Original: In Table 12, we study two fusion techniques to combine the features generated from the five views.
Revised: In Table 12, we study two fusion techniques to combine the features generated from the five views.

Original: The first method is a simple addition operation, and the second one is a learnable fusion mechanism where a 1D-Conv. is used where its kernel size is set to the number of the views used, i.e., in our case set to five.
Revised: The first method is a simple addition operation, and the second one is a learnable fusion mechanism where a 1D-Conv. is used where its kernel size is set to the number of the views used, i.e., in our case set to five.

Original: As expected, the learnable fusion scheme; 1D-Conv., performs better than the none learnable scheme; addition.
Revised: As expected, the learnable fusion scheme; 1D-Conv., performs better than the none learnable scheme; addition.

Original: Our 2D variant built on top of Referit3D [2] is used in this ablation, where the 2D stream is only used.
Revised: Our 2D variant built on top of Referit3D [2] is used in this ablation, where the 2D stream is only used.

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: In addition, we study three fusion techniques for the 3D, and 2D multi-modal features, M pc and
Revised: In addition, we study three fusion techniques for the 3D, and 2D multi-modal features, M pc and

Original: M img , respectively.
Revised: M img , respectively.

Original: In this ablation, we only utilize the ROI as the 2D semantic information, set the image resolution to 32, and set the number of multi-views to five.
Revised: In this ablation, we only utilize the ROI as the 2D semantic information, set the image resolution to 32, and set the number of multi-views to five.

Original: As shown in Table 13, using a transformer while fusing both multi-modal features achieves the best accuracy.
Revised: As shown in Table 13, using a transformer while fusing both multi-modal features achieves the best accuracy.

Original: Optionally include extra information (complete proofs, additional experiments and plots) in the appendix.
Revised: Optionally include extra information (complete proofs, additional experiments and plots) in the appendix.

Original: This section will often be part of the supplemental material.
Revised: This section will often be part of the supplemental material.

Original: A.5 Quantitative Results
Revised: A.5 Quantitative Results

Original: This section analyzes the qualitative results from the successful and the failure cases on the Nr3d dataset.
Revised: This section analyzes the qualitative results from the successful and the failure cases on the Nr3d dataset.

Original: In addition to Figure 8 that shows successful scenarios and Figure 9 that shows failure scenarios, we attach a short video to demonstrate more scenarios.
Revised: In addition to Figure 8 that shows successful scenarios and Figure 9 that shows failure scenarios, we attach a short video to demonstrate more scenarios.

Original: A.5.1 Successful Scenarios
Revised: A.5.1 Successful Scenarios

Original: LAR achieves top-ranking results on both the Nr3D and Sr3D datasets.
Revised: LAR achieves top-ranking results on both the Nr3D and Sr3D datasets.

Original: The results in Figuredemonstrate the correct instance localization on the Nr3D datasets only.
Revised: The results in Figuredemonstrate the correct instance localization on the Nr3D datasets only.

Original: The ground truth boxes are marked green and our model prediction marked in blue.
Revised: The ground truth boxes are marked green and our model prediction marked in blue.

Original: Our analysis shows that LAR correctly understands the global context based on natural language description.
Revised: Our analysis shows that LAR correctly understands the global context based on natural language description.

Original: For example, in Figure 8 (a-f),
Revised: For example, in Figure 8 (a-f),

Original: LAR understands relationships, e.g. (behind it, middle of, furthest from .. ), which proves that our proposed model understands instance-to-instance and instance-to-background relations.
Revised: LAR understands relationships, e.g. (behind it, middle of, furthest from .. ), which proves that our proposed model understands instance-to-instance and instance-to-background relations.

Original: As well as understanding colors and shapes related to language, LAR also can determine the meaning of terms such as “green,” Figure 8 (e).
Revised: As well as understanding colors and shapes related to language, LAR also can determine the meaning of terms such as “green,” Figure 8 (e).

Original: A.5.2 Failure Scenarios
Revised: A.5.2 Failure Scenarios

Original: Figure 9 shows representative failure cases of LAR.
Revised: Figure 9 shows representative failure cases of LAR.

Original: Despise the incorrect instance localization LAR managed to allocate the instance with the same class as the target.
Revised: Despise the incorrect instance localization LAR managed to allocate the instance with the same class as the target.

Original: As shown in Figure 9 (d, e), we observed that failure cases require an understanding of "against the wall", ’Facing the windows, "which is determined by camera view dependency.
Revised: As shown in Figure 9 (d, e), we observed that failure cases require an understanding of "against the wall", ’Facing the windows, "which is determined by camera view dependency.

Original: Despite LAR’s progress in view-dependent and independent samples, view understanding remains a challenge.
Revised: Despite LAR’s progress in view-dependent and independent samples, view understanding remains a challenge.

Original: Besides, other state-of-the-art models suffer from the same issue.
Revised: Besides, other state-of-the-art models suffer from the same issue.

Original: In addition, some LAR failure cases are caused by ambiguous queries.
Revised: In addition, some LAR failure cases are caused by ambiguous queries.

Original: Figure 9 (a,b,c,e, f) shows a complex and compound language description to describe the target instance, which in some cases, both models and human annotators mix up similar objects.
Revised: Figure 9 (a,b,c,e, f) shows a complex and compound language description to describe the target instance, which in some cases, both models and human annotators mix up similar objects.