Original: HUM3DIL: Semi-supervised Multi-modal 3D Human Pose Estimation for Autonomous Driving
Revised: HUM3DIL: Semi-supervised Multi-modal 3D Human Pose Estimation for Autonomous Driving

Original: Anonymous Author(s)

Original: AfﬁliationAddress email

Original: Abstract: Autonomous driving is an exciting new industry, posing important research questions.
Revised: Abstract: Autonomous driving is an exciting new industry, posing important research questions.

Original: Within the perception module, 3D human pose estimation is anemerging technology, which can enable the autonomous vehicle to perceive andunderstand the subtle and complex behaviors of pedestrians.
Revised: Within the perception module, 3D human pose estimation is an emerging technology, which can enable the autonomous vehicle to perceive and understand the subtle and complex behaviors of pedestrians.

Original: While hardware systems and sensors have dramatically improved over the decades – with cars potentially boasting complex LiDAR and vision systems and with a growing expansionof the available body of dedicated datasets for this newly available information – not much work has been done to harness these novel signals for the core problemof 3D human pose estimation.
Revised: While hardware systems and sensors have dramatically improved over the decades – with cars potentially boasting complex LiDAR and vision systems and with a growing expansion of the available body of dedicated datasets for this newly available information – not much work has been done to harness these novel signals for the core problem of 3D human pose estimation.

Original: Our method, which we coin HUM3DIL (HUMan3D from Images and LiDAR), efﬁciently makes use of these complementary signals, in a semi-supervised fashion and outperforms existing methods with a largemargin.
Revised: Our method, which we coin HUM3DIL (HUMan 3D from Images and LiDAR), efﬁciently makes use of these complementary signals, in a semi-supervised fashion and outperforms existing methods with a large margin.

Original: It is a fast and compact model for onboard deployment.
Revised: It is a fast and compact model for onboard deployment.

Original: Speciﬁcally,we embed LiDAR points into pixel-aligned multi-modal features, which we passthrough a sequence of Transformer reﬁnement stages.
Revised: Speciﬁcally, we embed LiDAR points into pixel-aligned multi-modal features, which we pass through a sequence of Transformer reﬁnement stages.

Original: Quantitative experimentson the Waymo Open Dataset support these claims, where we achieve state-of-theart results on the task of 3D pose estimation.
Revised: Quantitative experiments on the Waymo Open Dataset support these claims, where we achieve state-of-theart results on the task of 3D pose estimation.

Original:  Introduction
Revised:  Introduction

Original: Robotic systems which operate in environments with humans are required to avoid collisions withpeople and beneﬁt from analysing their actions and forecasting future behaviors.
Revised: Robotic systems which operate in environments with humans are required to avoid collisions with people and beneﬁt from analysing their actions and forecasting future behaviors.

Original: Human pose understanding is a well established research direction in computer vision with numerous industrialapplications [1, 2, 3, 4].
Revised: Human pose understanding is a well established research direction in computer vision with numerous industrial applications [1, 2, 3, 4].

Original: In this work we focus on 3D human pose understanding for the autonomousvehicle (AV) industry and robotic applications in general.
Revised: In this work we focus on 3D human pose understanding for the autonomous vehicle (AV) industry and robotic applications in general.

Original: Safety is the top priority for the AV industry.
Revised: Safety is the top priority for the AV industry.

Original: Many robotic platforms use sensors in different modalities (e.g. cameras, LiDARs, radars, audio, etc.) to improve safety by analyzing more signals aboutthe environment.
Revised: Many robotic platforms use sensors in different modalities (e.g. cameras, LiDARs, radars, audio, etc.) to improve safety by analyzing more signals about the environment.

Original: Using RGB cameras coupled with LiDAR sensors could be considered as a standard sensor suite for most robotic platforms.
Revised: Using RGB cameras coupled with LiDAR sensors could be considered as a standard sensor suite for most robotic platforms.

Original: While many studies have shown impressive results forestimating human poses using RGB imagery, there is a paucity of methods which can effectively useboth modalities [5].
Revised: While many studies have shown impressive results for estimating human poses using RGB imagery, there is a paucity of methods which can effectively use both modalities [5].

Original: Recent studies have made great headway into estimating 3D human pose in controlled environments[1], but many real-world and safety-critical robotic applications require estimating human poses inuncontrolled environments, where subjects may be captured under different levels of occlusions,from various perspectives and across any ranges.
Revised: Recent studies have made great headway into estimating 3D human pose in controlled environments [1], but many real-world and safety-critical robotic applications require estimating human poses in uncontrolled environments, where subjects may be captured under different levels of occlusions, from various perspectives and across any ranges.

Original: There are several approaches to estimate 3D human poses in uncontrolled environments [5], but they have insufﬁcient accuracy to unlock their fullpotential, especially for AV applications.
Revised: There are several approaches to estimate 3D hu- man poses in uncontrolled environments [5], but they have insufﬁcient accuracy to unlock their full potential, especially for AV applications.

Original: Another limiting factor for robotic applications of available methods of 3D pose detection is theircomputational complexity.
Revised: Another limiting factor for robotic applications of available methods of 3D pose detection is their computational complexity.

Original: There are fast methods for detecting a 3D bounding box which containsa single person [6, 7].
Revised: There are fast methods for detecting a 3D bounding box which contains a single person [6, 7].

Original: Thus most robotic applications represent humans with 3D bounding boxes.
Revised: Thus most robotic applications represent humans with 3D bounding boxes.

Original: While this crude representation of people allows such applications to meet basic safety requirements and avoid collisions, it is not sufﬁcient for understanding complex human body gestures.
Revised: While this crude representation of people allows such applications to meet basic safety requirements and avoid collisions, it is not sufﬁcient for understanding complex human body gestures.

Original: There aremethods which output feature rich representations and estimate parameters of full body meshes [8],but they are relatively slow.
Revised: There are methods which output feature rich representations and estimate parameters of full body meshes [8], but they are relatively slow.

Original: Representing 3D human body pose as a sequence of locations of 3D keypoints inside the body could be considered as a balanced trade off between fast to compute boxesand slow full body models.
Revised: Representing 3D human body pose as a sequence of locations of 3D key points inside the body could be considered as a balanced trade off between fast to compute boxes and slow full body models.

Original: The main goal of this work is to provide a fast method for human poseestimation in uncontrolled environments which efﬁciently uses sensor modalities common for AVindustry and outputs representations rich enough to enable analysis of complex human behaviors.
Revised: The main goal of this work is to provide a fast method for human pose estimation in uncontrolled environments which efﬁciently uses sensor modalities common for AV industry and outputs representations rich enough to enable analysis of complex human behaviors.

Original: One of the key factors enabling research and development of methods for human pose understandingis the availability of high-quality ground truth 3D data with human poses and sensor data.
Revised: One of the key factors enabling research and development of methods for human pose understanding is the availability of high-quality ground truth 3D data with human poses and sensor data.

Original: There arefew ways to collect such data: marker or markerless camera based motion capture systems (suitablefor controlled indoor environments); IMU based motion capture systems (suitable for both indoorand outdoor, but also controlled environments) [1] and manual human labeling (suitable for allenvironments, but expensive and error prone).
Revised: There are few ways to collect such data: marker or markerless camera based motion capture systems (suitable for controlled indoor environments); IMU based motion capture systems (suitable for both indoor and outdoor, but also controlled environments) [1] and manual human labeling (suitable for all environments, but expensive and error prone).

Original: To the best of our knowledge, there is no large-scaleoutdoor dataset with human poses collected in an uncontrolled environment with ground truth 2Dand 3D keypoints with RGB and LiDAR for a fully supervised training mode.
Revised: To the best of our knowledge, there is no large-scale outdoor dataset with human poses collected in an uncontrolled environment with ground truth 2D and 3D keypoints with RGB and LiDAR for a fully supervised training mode.

Original: Waymo recentlyreleased a version of their Waymo Open Dataset (WOD v1.3.2) with a large amount of camera (2D)keypoints and small amount of laser (3D) keypoints (enough for ﬁne-tuning and evaluation purposes)which is suitable for weakly and/or semi-supervised training modes.
Revised: Waymo recently released a version of their Waymo Open Dataset (WOD v1.3.2) with a large amount of camera (2D) keypoints and small amount of laser (3D) keypoints (enough for ﬁne-tuning and evaluation purposes) which is suitable for weakly and/or semi-supervised training modes.

Original: In this work we use the WODv1.3.2 dataset to demonstrate that our method can reliably predict 3D human pose in uncontrolledand challenging AV scenarios, and we compare our approach with several state-of-the-art methods[9, 10, 11] after adapting them for multi-modal applications.
Revised: In this work we use the WOD v1.3.2 dataset to demonstrate that our method can reliably predict 3D human pose in uncontrolled and challenging AV scenarios, and we compare our approach with several state-of-the-art methods [9, 10, 11] after adapting them for multi-modal applications.

Original: We propose HUM3DIL , a light-weight 3D human joints prediction network, that leverages RGBinformation with LiDAR points, in a novel fashion, by computing pixel-aligned [12] multi-modalfeatures with the 3D positions of the LiDAR signal.
Revised: We propose HUM3DIL , a light-weight 3D human joints prediction network, that leverages RGB information with LiDAR points, in a novel fashion, by computing pixel-aligned [12] multi-modal features with the 3D positions of the LiDAR signal.

Original: These features are then used by subsequent
Revised: These features are then used by subsequent

Original: Transformer-based reﬁnement stages, to produce the desired 3D joints.
Revised: Transformer-based reﬁnement stages, to produce the desired 3D joints.

Original: We train our model in asemi-supervised manner, to maximize the utility of both 2D annotations (less expensive to collect,available in larger volumes) and 3D labels (expensive to collect, accurate, but with limited coverage).
Revised: We train our model in a semi-supervised manner, to maximize the utility of both 2D annotations (less expensive to collect, available in larger volumes) and 3D labels (expensive to collect, accurate, but with limited coverage).

Original: Quantitative results on Waymo Open Dataset indicate state-of-the-art performance.
Revised: Quantitative results on Waymo Open Dataset indicate state-of-the-art performance.

Original: Being accurate,fast and lightweight, HUM3DIL can be deployed into onboard autonomous driving systems to provide real-time perception signals of human road users.
Revised: Being accurate, fast and lightweight, HUM3DIL can be deployed into onboard autonomous driving systems to provide real-time perception signals of human road users.

Original: We believe downstream tasks can greatlybeneﬁt from these ﬁne-grained signals.
Revised: We believe downstream tasks can greatly beneﬁt from these ﬁne-grained signals.

Original: Related Work: There are considerable amount of prior works in 3D human pose reconstruction,mostly focused on estimation from RGB images alone.
Revised: Related Work: There are considerable amount of prior works in 3D human pose reconstruction, mostly focused on estimation from RGB images alone.

Original: There are two main classes of methods, theﬁrst of which is model based [13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23] and relies on statisticalhuman body models like SMPL [24] or GHUM [25].
Revised: There are two main classes of methods, the ﬁrst of which is model based [13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23] and relies on statistical human body models like SMPL [24] or GHUM [25].

Original: These methods do not estimate 3D posedirectly, but instead regress the parameters of a statistical body model, which has built-in anatomicaland kinematic constraints.
Revised: These methods do not estimate 3D pose directly, but instead regress the parameters of a statistical body model, which has built-in anatomical and kinematic constraints.

Original: This leads to more natural predictions, with body shape usually estimatedas well, even for poses not encountered during training.
Revised: This leads to more natural predictions, with body shape usually estimated as well, even for poses not encountered during training.

Original: The second class of methods is skeletonbased [26, 27, 28, 22], where the 3D pose is represented by 3D joint positions and these are tobe regressed or detected directly from the input.
Revised: The second class of methods is skeletonbased [26, 27, 28, 22], where the 3D pose is represented by 3D joint positions and these are to be regressed or detected directly from the input.

Original: These methods have the advantage of usuallybeing more accurate and faster, but they are not guaranteed to produce anatomically correct humanskeletons (e.g. the left arm may be reconstructed with different length than the right arm).
Revised: These methods have the advantage of usually being more accurate and faster, but they are not guaranteed to produce anatomically correct human skeletons (e.g. the left arm may be reconstructed with different length than the right arm).

Original: Ourproposed model falls in the latter category, as our goal is to estimate pedestrians as accurately aspossible in real-time.
Revised: Our proposed model falls in the latter category, as our goal is to estimate pedestrians as accurately as possible in real-time.

Original: Mixed approaches have started to recently emerge, like in e.g. [9], where 3Dpositions are inferred directly, but anatomically regularized through a statistical model.
Revised: Mixed approaches have started to recently emerge, like in e.g. [9], where 3D positions are inferred directly, but anatomically regularized through a statistical model.

Original: There are works that use depth information, either separately or in combination with an RGB image,to reconstruct the 3D pose [29, 30, 31].
Revised: There are works that use depth information, either separately or in combination with an RGB image, to reconstruct the 3D pose [29, 30, 31].

Original: Approaches that utilize LiDAR information are hard to comeby, mostly because of the lack of ground-truth 3D human pose paired with LiDAR data.
Revised: Approaches that utilize LiDAR information are hard to come by, mostly because of the lack of ground-truth 3D human pose paired with LiDAR data.

Original: There area few datasets that, to different degrees, do provide 3D annotations.
Revised: There are a few datasets that, to different degrees, do provide 3D annotations.

Original: The PedX dataset [32] offers14 , 000 3D automatic pedestrian annotations obtained using model ﬁtting on different modalities,gathered from three different real-world scenes.
Revised: The PedX dataset [32] offers 14 , 000 3D automatic pedestrian annotations obtained using model ﬁtting on different modalities, gathered from three different real-world scenes.

Original: The Waymo Open Dataset [33] has a similar amountof 3D annotations as [32], but it features many more different environments (2,030 real scenes, 7,650different people) with high-quality 2D and 3D manual annotations.
Revised: The Waymo Open Dataset [33] has a similar amount of 3D annotations as [32], but it features many more different environments (2,030 real scenes, 7, different people) with high-quality 2D and 3D manual annotations.

Original: Even with the existence of thesedatasets, the few works on 3D pose reconstruction published in this space mostly rely on weaksupervision, by lifting 2D pose information in 3D. [34] trains on 2D ground-truth pose annotationsand uses a reprojection loss for the 3D pose regression task.
Revised: Even with the existence of these datasets, the few works on 3D pose reconstruction published in this space mostly rely on weak supervision, by lifting 2D pose information in 3D. [34] trains on 2D ground-truth pose annotations and uses a reprojection loss for the 3D pose regression task.

Original: [11] creates pseudo ground-truth 3D joint positions from the projection of annotated 2D joints, by considering neighboring LiDAR points[Figure] in the projection space.
Revised: [11] creates pseudo ground-truth 3D joint positions from the projection of annotated 2D joints, by considering neighboring LiDAR points in the projection space.

Original: During training, they directly compare the predicted 3D positions againstthis pseudo ground-truth.
Revised: During training, they directly compare the predicted 3D positions against this pseudo ground-truth.

Original:  Methodology 
Revised:  Methodology 2.1 Problem Formulation

Original: [Figure] 2.1 Problem Formulation

Original: [Figure] We focus on the task of key points localization and formulate the problem as estimating 3D locationsfor a set of key points Y ∈ R N j × 3 inside human body, given the ground truth or predicted boundingbox of a human as well as multi-modal inputs from camera and LiDAR sensors: an RGB cameraimage I ∈ [0 , 1] H × W × 3 and a point cloud P ∈ R N p × 3 , consisting of N p
Revised: We focus on the task of key points localization and formulate the problem as estimating 3D locations for a set of key points Y ∈ R N j × 3 inside human body, given the ground truth or predicted bounding box of a human as well as multi-modal inputs from camera and LiDAR sensors: an RGB camera image I ∈ [0 , 1] H × W × 3 and a point cloud P ∈ R N p × 3 , consisting of N p

Original: LiDAR points from asingle scan.
Revised: LiDAR points from a single scan.

Original: Camera Model.
Revised: Camera Model.

Original: For correct LiDAR points to camera image projections, we use a differentialimplementation of the Waymo Open Dataset [33] camera model, with rolling shutter effect compensated.
Revised: For correct LiDAR points to camera image projections, we use a differential implementation of the Waymo Open Dataset [33] camera model, with rolling shutter effect compensated.

Original: We denote the intrinsic information (e.g. lens distortion, shutter speed, focal length, focalpoint etc.)
Revised: We denote the intrinsic information (e.g. lens distortion, shutter speed, focal length, focal point etc.)

Original: as a vector K i , and the extrinsic parameters (e.g. vehicle pose, camera pose, linear and rotation speed) as K e .
Revised: as a vector K i , and the extrinsic parameters (e.g. vehicle pose, camera pose, linear and rotation speed) as K e .

Original: The complete camera information will be denoted as K =
Revised: The complete camera information will be denoted as K =

Original: [ K e | K i ] ∈ R 1 × N k .
Revised: [ K e | K i ] ∈ R 1 × N k .

Original: The associated camera image projection operator will be denoted by Π( ∗ , K ) , where a 3D input willbe correctly projected into the image space.
Revised: The associated camera image projection operator will be denoted by Π( ∗ , K ) , where a 3D input will be correctly projected into the image space.

Original: 2.2 HUM3DIL
Revised: 2.2 HUM3DIL

Original: [Figure] Our network, deemed HUM3DIL (see ﬁg. 1), receives as input the RGB camera image I , LiDARpoint cloud P , camera intrinsics K , the paired 2D and 3D bounding boxes, and outputs the predicted 3D human keypoints Y .
Revised: Our network, deemed HUM3DIL (see ﬁg. 1), receives as input the RGB camera image I , LiDAR point cloud P , camera intrinsics K , the paired 2D and 3D bounding boxes, and outputs the predicted 3D human keypoints Y .

Original: Our goal is to use image input to better exploit or disambiguate the structure present in the 3D point cloud, while at the same time use the 3D points to anchor imageryevidence.
Revised: Our goal is to use image input to better exploit or disambiguate the structure present in the 3D point cloud, while at the same time use the 3D points to anchor imagery evidence.

Original: Because we have access to the ground-truth camera intrinsics, we can move between 3Dspace and 2D image space by projection, and vice-versa, by back-projection.
Revised: Because we have access to the ground-truth camera intrinsics, we can move between 3D space and 2D image space by projection, and vice-versa, by back-projection.

Original: Motivated by recentadvancements in 3D pose estimation and point-cloud processing [36, 37, 9], we use a Transformerbased architecture [38] to process 3D and image-based structural information at the same time,compared to typical approaches which use disconnected PointNet 3D [39, 40] embeddings and/orimage features.
Revised: Motivated by recent advancements in 3D pose estimation and point-cloud processing [36, 37, 9], we use a Transformerbased architecture [38] to process 3D and image-based structural information at the same time, compared to typical approaches which use disconnected PointNet 3D [39, 40] embeddings and/or image features.

Original: Enriching LiDAR points with image evidence We construct a depth-image projection[-D ∈ 
-]
Revised: Enriching LiDAR points with image evidence We construct a depth-image projection[-D ∈ 
-]

Original:  R H×
Revised: D ∈ R H

Original:  W × 1 into the space of LiDAR point cloud P .
Revised: × W × 1 into the space of LiDAR point cloud P .

Original: We concatenate it with the raw RGB image,[ I ; D ] , and pass the derived tensor through a convolutional architecture.
Revised: We concatenate it with the raw RGB image, [ I ; D ] , and pass the derived tensor through a convolutional architecture.

Original: Thus, we simultaneouslyinform the convolutional layers of the regions of interest in the image, i.e. sparse locations on thesilhouette of the person, and make depth information available from the start.
Revised: Thus, we simultaneously inform the convolutional layers of the regions of interest in the image, i.e. sparse locations on the silhouette of the person, and make depth information available from the start.

Original: Adding the depthmap channel helps disambiguate the task of key point prediction in cases with heavy occlusions orin crowded scenes where multiple people are in the frame - depth map channel will have non zerovalues only for a person of interest.
Revised: Adding the depth map channel helps disambiguate the task of key point prediction in cases with heavy occlusions or in crowded scenes where multiple people are in the frame - depth map channel will have non zero values only for a person of interest.

Original: For the convolutional architecture, we employ a lightweight
Revised: For the convolutional architecture, we employ a lightweight

Original: U-Net network [41], that outputs a dense feature map representation F ∈ R H
Revised: U-Net network [41], that outputs a dense feature map representation F ∈ R H

Original: × W × D f .
Revised: × W × D f .

Original: This map isused to query from, based on the projection of the LiDAR points, by bilinear interpolation:
Revised: This map is used to query from, based on the projection of the LiDAR points, by bilinear interpolation:

Original: [Equation]
Revised: [Equation]

Original: ) for each point P i ∈ P .
Revised: for each point P i ∈ P .

Original: We thus obtain pixel-aligned image features for LiDAR points.
Revised: We thus obtain pixel-aligned image features for LiDAR points.

Original: LiDAR points embedding Aside from the per-point depth-infused image features, we also process the initial 3D LiDAR points.
Revised: LiDAR points embedding Aside from the per-point depth-infused image features, we also process the initial 3D LiDAR points.

Original: We use Random Fourier Features (RFF) [42] to embed P ina higher-dimensional space, capturing high-frequency behaviour of the signal.
Revised: We use Random Fourier Features (RFF) [42] to embed P in a higherdimensional space, capturing high-frequency behaviour of the signal.

Original: We use a random Gaussian matrix B ∈ R 3 × D p / 2 , with each entry independently drawn from a normal distribution N (0 , σ 2 ) .
Revised: We use a random Gaussian matrix B ∈ R 3 × D p / 2 , with each entry independently drawn from a normal distribution N (0 , σ 2 ) .

Original: The transformed points will have the form (cid:101) P =
Revised: The transformed points will have the form (cid:101) P =

Original: [ cos (2 π PB ); sin (2 π PB )] ∈ R N p × D p . 
Revised: [ cos (2 π PB ); sin (2 π PB )] ∈ R N p × D p . [Figure] L x ( T r an s f o r m e r E n c ode r) M L P

Original:  Transforming the LiDAR points In order to regress Y , we employ a Transformer architecture[43].
Revised: Predicted [Figure] Transforming the LiDAR points In order to regress Y , we employ a Transformer architecture [43].

Original: We deﬁne an input i-
Revised: We deﬁne an input i-

Original: th token M i ∈
Revised: th token M i ∈

Original: R 1 × D , with D = D f + D p + N
Revised: R 1 × D , with D = D f + D p + N

Original: k
Revised: k

Original: , as:
Revised: , as:

Original: [Equation]
Revised: [Equation]

Original: i = [Equation] , F i , (cid:101) P i ] , (2) the concatenation of the camera intrinsics, the per-point image feature and the per-point Fourierrepresentation.
Revised: the concatenation of the camera intrinsics, the per-point image feature and the per-point Fourier representation.

Original: We will apply the transformer on a ﬁxed sequence of N p tokens .
Revised: We will apply the Transformer on a ﬁxed sequence of N p tokens .

Original: In order to workwith a variable number of tokens, we use a ﬁxed maximum size for the token sequence, N maxp .
Revised: In order to work with a variable number of tokens, we use a ﬁxed maximum size for the token sequence, N maxp .

Original: Weshufﬂe and trim excess points, and pad with zeros if we have a fewer number of points.
Revised: We shufﬂe and trim excess points, and pad with zeros if we have a fewer number of points.

Original: The completeinput sequence is M ∈ R N maxp × D .
Revised: The complete input sequence is M ∈ R N maxp × D .

Original: This sequence is at ﬁrst linearly embedded by using a learnablematrix E ∈ R D × D 0 .
Revised: This sequence is at ﬁrst linearly embedded by using a learnable matrix E ∈ R D × D 0 .

Original: Here, D 0 is the operating dimensionality of the transformer architecture.
Revised: Here, D 0 is the operating dimensionality of the Transformer architecture.

Original: Weadditionally concatenate learnable joints tokens, M J ∈ R N j × D 0 .
Revised: We additionally concatenate learnable joints tokens, M J ∈ R N j × D 0 .

Original: Similar to [9], we use a cascadedblock of L transformer encoder layers, and collect the predicted 3D keypoints ˜Y from an MLPapplied on the transformed joints tokens:
Revised: Similar to [9], we use a cascaded block of L Transformer encoder layers, and collect the predicted 3D keypoints ˜Y from an MLP applied on the transformed joints tokens:

Original: [Equation]
Revised: [Equation]

Original: Losses Labeling 3D keypoints is signiﬁcantly more expensive and slower than 2D keypoints inuncontrolled real-world environments.
Revised: Losses Labeling 3D keypoints is signiﬁcantly more expensive and slower than 2D keypoints in uncontrolled real-world environments.

Original: As a result, we usually collect a dataset containing manymore 2D annotations than 3D labels.
Revised: As a result, we usually collect a dataset containing many more 2D annotations than 3D labels.

Original: To maximize the utility of all available labels (2D and 3D) andboost performance, we use a mixture of weakly and fully supervised losses to train our model.
Revised: To maximize the utility of all available labels (2D and 3D) and boost performance, we use a mixture of weakly and fully supervised losses to train our model.

Revised: We denote the ground-truth 2D joints keypoints as y ∈ R N j × 2 and deﬁne the 2D reprojection, and 3D

Original: We
Revised: [Table]

Original: 3D reconstruction losses as:
Revised: reconstruction losses as:

Original: [Equation]
Revised: [Equation]

Original: where (cid:107)∗(cid:107) 2 is the (cid:96) 2 vector norm – i.e. the more robust euclidean distance between predictions.
Revised: where (cid:107)∗(cid:107) 2 is the (cid:96) 2 vector norm – i.e. the more robust euclidean distance between predictions.

Original: Weadd a small (cid:15) during training, as the function is not differentiable at 0 .
Revised: We add a small (cid:15) during training, as the function is not differentiable at 0 .

Original: Our ﬁnal loss is given by:
Revised: Our ﬁnal loss is given by:

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: where λ is a scalar factor used to weigh the two losses.
Revised: where λ is a scalar factor used to weigh the two losses.

Original: Semi-supervised support In order to efﬁciently train with mixed 2D and 3D labels, we also set atraining batch to contain a pre-deﬁned fraction of 2D-to-3D annotations.
Revised: Semi-supervised support In order to efﬁciently train with mixed 2D and 3D labels, we also set a training batch to contain a pre-deﬁned fraction of 2D-to-3D annotations.

Original: This will allow the networkto not forget about 3D, when the dataset is drastically biased towards 2D annotations.
Revised: This will allow the network to not forget about 3D, when the dataset is drastically biased towards 2D annotations.

Original: We show astudy on the effect of the percentage of 3D and the loss balancing factor in ﬁgure 4, left.
Revised: We show a study on the effect of the percentage of 3D and the loss balancing factor in ﬁgure 4, left.

Original:  Experimental Results
Revised:  Experimental Results

Original: Datasets Waymo Open Dataset v1.3.2 (WOD) [33] contains RGB and LiDAR range images capturing various road users.
Revised: Datasets Waymo Open Dataset v1.3.2 (WOD) [33] contains RGB and LiDAR range images capturing various road users.

Original: Recently camera (2D) and laser (3D) key points annotations on a portionof human subjects (pedestrians and cyclists) in WOD have been released, namely Waymo Human
Revised: Recently camera (2D) and laser (3D) key points annotations on a portion of human subjects (pedestrians and cyclists) in WOD have been released, namely Waymo Human

Original: Key Points dataset v1.3.2 (WHKP).
Revised: Key Points dataset v1.3.2 (WHKP).

Original: We benchmark HUM3DIL and other baselines on the WHKP.
Revised: We benchmark HUM3DIL and other baselines on the WHKP.

Original: As the ofﬁcial WOD/WHKP testing subset is hidden from the public, we randomly select 50% ofsubjects from the WOD validation subset as our validation split, and the rest 50% fall into the testingsplit for benchmarking.
Revised: As the ofﬁcial WOD/WHKP testing subset is hidden from the public, we randomly select 50% of subjects from the WOD validation subset as our validation split, and the rest 50% fall into the testing split for benchmarking.

Original: In our experiments, we use the ground-truth camera and LiDAR boundingboxes during training and evaluation, for two reasons: a) disentangling the evaluation of the keypoints localization from the object detection task; b) setting an easy-to-reproduce baseline for futureresearch works.
Revised: In our experiments, we use the ground-truth camera and LiDAR bounding boxes during training and evaluation, for two reasons: a) disentangling the evaluation of the key points localization from the object detection task; b) setting an easy-to-reproduce baseline for future research works.

Original: Metrics Where applicable, we will report two metrics: the mean per-joint position error (i.e.
Revised: Metrics Where applicable, we will report two metrics: the mean per-joint position error (i.e.

Original: MPJPE) between predicted and ground-truth 3D joints, and a similar one, for the 2D case.
Revised: MPJPE) between predicted and ground-truth 3D joints, and a similar one, for the 2D case.

Original: Asthe ground-truth is not available for every frame, or even for every joint, we will use a visibilityindicator v ji ∈ { 0 , 1 } .
Revised: As the ground-truth is not available for every frame, or even for every joint, we will use a visibility indicator v ji ∈ { 0 , 1 } .

Original: This signals if we have a ground-truth annotation for a particular joint i , of aparticular testing sample j .
Revised: This signals if we have a ground-truth annotation for a particular joint i , of a particular testing sample j .

Original: The MPJPE over a particular dataset will then have the value:
Revised: The MPJPE over a particular dataset will then have the value:

Original: [Equation]
Revised: [Equation]

Original: Evaluation on WHKP We train three different methods on the WHKP training subset: ours (i.e.
Revised: We train three different methods on the WHKP training subset: ours (i.e.

Original: HUM3DIL), THUNDR [9] reimplemented following the original paper, THUNDR with also an additional depth image as input, ContextPose [10] using publicly available code, and the multi-modalapproach of [11], for which we report their number on a similar version of the dataset.
Revised: HUM3DIL), THUNDR [9] reimplemented following the original paper, THUNDR with also an additional depth image as input, ContextPose [10] using publicly available code, and the multi-modal approach of [11], for which we report their number on a similar version of the dataset.

Original: These are allstate-of-the-art methods in 3D pose estimation for single persons, from an RGB image.
Revised: These are all state-of-the-art methods in 3D pose estimation for single persons, from an RGB image.

Original: We tried tomake comparisons against pure RGB image methods as fair as possible, but there is a modeling gapthat cannot be breached – our architecture naturally exploits LiDAR signal with ease.
Revised: We tried to make comparisons against pure RGB image methods as fair as possible, but there is a modeling gap that cannot be breached – our architecture naturally exploits LiDAR signal with ease.

Original: Also note thatwe have 1 / 5 − 1 / 14 th of the number of parameters of competing methods.
Revised: Also note that we have 1 / 5 − 1 / 14 th of the number of parameters of competing methods.

Original: We report results on thetest split in Table.
Revised: We report results on the test split in Table.

Original: Implementation details In all our experiments we use a U-Net backbone [41], with randomlyinitialized weights.
Revised: Implementation details In all our experiments we use a U-Net backbone [41], with randomly initialized weights.

Original: The encoder/decoder convolutional ﬁlter sequences are [32 , 64 , 128 , 256] and[256 , 128 , 64 , 32] , respectively.
Revised: The encoder/decoder convolutional ﬁlter sequences are [32 , 64 , 128 , 256] and [256 , 128 , 64 , 32] , respectively.

Original: The backbone has 2 , 095 , 392 parameters.
Revised: The backbone has 2 , 095 , 392 parameters.

Original: For the transformer architecture, we use L = 4 stages, an embedding size 256 and 8 heads for the MultiHeadAttentionlayer.
Revised: For the Transformer architecture, we use L = 4 stages, an embedding size 256 and 8 heads for the MultiHeadAttention layer.

Original: We train the network for 50 epochs, with batch size of 16 , base learning rate of 1 e − 4 and[Table] exponential decay 0 . 
Revised: We train the network for 50 epochs, with batch size of 16 , base learning rate of 1 e − 4 and exponential decay 0 . 99 .

Original: We set the maximum number of LiDAR points to 1024 .
Revised: We set the maximum number of LiDAR points to 1024 .

Original: Our transformerarchitecture consists of 3 , 229 , 696 parameters, with a ﬁnal MLP of 771 neurons.
Revised: Our Transformer architecture consists of 3 , 229 , 696 parameters, with a ﬁnal MLP of 771 neurons.

Original: We validate σ = 10and λ = 1 e − 2 .
Revised: We validate σ =and λ = 1 e − 2 .

Original: The complete architecture has 5 , 325 , 859 parameters.
Revised: The complete architecture has 5 , 325 , 859 parameters.

Original: All of our networks weretrained on a single V100 GPU with 16GB of memory.
Revised: All of our networks were trained on a single V100 GPU with 16GB of memory.

Original: Our code is implemented in TensorFlow.
Revised: Our code is implemented in TensorFlow.

Original: Wetest the network in inference mode on an Nvidia RTX 2080 GPU, for a batch of a single example.
Revised: We test the network in inference mode on an Nvidia RTX 2080 GPU, for a batch of a single example.

Original: One pass is done in 8 milliseconds.
Revised: One pass is done in 8 milliseconds.

Original: The main performance/memory bottleneck resides in computingthe attention matrix (which is ≈ 1000 × 1000 ) in the Transformer architecture.
Revised: The main performance/memory bottleneck resides in computing the attention matrix (which is ≈ 1000 × 1000 ) in the Transformer architecture.

Original: [Table]
Revised: [Table]

Original: 3.1 Ablation studies
Revised: 3.1 Ablation studies

Original: In table 3, we ablate different high-level methodological choices in our proposed architecture andreport results on the validation set of WHKP.
Revised: In table 3, we ablate different high-level methodological choices in our proposed architecture and report results on the validation set of WHKP.

Original: First, we disable the weakly-supervised loss by settingλ = 0 .
Revised: First, we disable the weakly-supervised loss by setting λ = 0 .

Original: We notice that the error increases, as expected, substantially in the 2D MPJPE, but also inthe 3D MPJPE.
Revised: We notice that the error increases, as expected, substantially in the 2D MPJPE, but also in the 3D MPJPE.

Original: This showcases the importance of using the available 2D training signal, even forinferring 3D joints.
Revised: This showcases the importance of using the available 2D training signal, even for inferring 3D joints.

Original: Next, we disable the Random Fourier Embedding for the 3D LiDAR points, byreplacing it with an identity embedding.
Revised: Next, we disable the Random Fourier Embedding for the 3D LiDAR points, by replacing it with an identity embedding.

Original: This has the most substantial impact on the performance,as the error more than doubles, from 6 . 91 cm to 15 . 19 cm.
Revised: This has a fairly low impact on the performance.

Original: We also disable the depth input, leavingonly the RGB image through the U-Net backbone.
Revised: We also disable the depth input, leaving only the RGB image through the U-Net backbone.

Original: The performance drop is not as dramatic in thiscase, as LiDAR point positions are already available as tokens.
Revised: The performance drop is not as dramatic in this case, as LiDAR point positions are already available as tokens.

Original: However, this signals the fact thatthe feature processing done by the backbone is not redundant, as a gap in performance still exists.
Revised: However, this signals the fact that the feature processing done by the backbone is not redundant, as a gap in performance still exists.

Original: When we disable the RGB, we get ≈ 1  cm drop in performance.
Revised: When we disable the RGB, we get ≈ 1 . 3 cm drop in performance.

Revised: We also ablate with the PointNet[40] and PointNet++[39] architectures, instead of a Tranformer, and performance is worse.

Original: The best performing method hasall the components activated, showing their complementary impact.
Revised: The best performing method has all the components activated, showing their complementary impact.

Original: [Table]
Revised: [Table]

Original:  Limitations
Revised:  Limitations

Original: Failure cases.
Revised: Failure cases.

Original: In ﬁgure 3, we randomly select and show six examples of results where the errorexceeds 15 cm.
Revised: In ﬁgure 3, we randomly select and show six examples of results where the error exceeds 15 cm.

Original: Extreme occlusions and partial views are the most difﬁcult cases to handle.
Revised: Extreme occlusions and partial views are the most difﬁcult cases to handle.

Original: Pleasenote that the results are still, generally, anatomically plausible.
Revised: Please note that the results are still, generally, anatomically plausible.

Original: Performance decay.
Revised: Performance decay.

Original: We also showthe performance decay w.r.t.
Revised: We also show the performance decay w.r.t.

Original: the distance to the LiDAR human point-cloud (which also controls thesparsity of the LiDAR signal).
Revised: the distance to the LiDAR human point-cloud (which also controls the sparsity of the LiDAR signal).

Original: As we can see from ﬁgure 4, bottom-right, our performance dropswhen the subject is too near or too far, but still produces reasonable results.
Revised: As we can see from ﬁgure 4, bottom-right, our performance drops when the subject is too near or too far, but still produces reasonable results.

Original: We are seeing a ≈ 3 cmerror gap between the best and worst conditions (as captured by the dataset).
Revised: We are seeing a ≈ 3 cm error gap between the best and worst conditions (as captured by the dataset).

Original: We also see a performance degradation (see ﬁgure 4, top-right) when the target subject is heavily occluded, due to partialviews or self-occlusions.
Revised: We also see a performance degradation (see ﬁgure 4, top-right) when the target subject is heavily occluded, due to partial views or self-occlusions.

Original: The error increases almost two-fold when going from full-view to severelyoccluded.
Revised: The error increases almost two-fold when going from full-view to severely occluded.

Original: General applicability.
Revised: General applicability.

Original: For now, our network can only process each human instance byindividually cropping, so it cannot use information about multiple humans at once (e.g. [17, 44, 45]),to improve the error and processing speed.
Revised: For now, our network can only process each human instance by individually cropping, so it cannot use information about multiple humans at once (e.g. [17, 44, 45]), to improve the error and processing speed.

Original: Also, we do not utilize temporal information (e.g. [46])which could further stabilize predictions and improve errors under occlusions.
Revised: Also, we do not utilize temporal information (e.g. [46]) which could further stabilize predictions and improve errors under occlusions.

Original:  Conclusions
Revised:  Conclusions

Original: We have presented a novel deep neural network architecture, which has been tailored for the needs ofmodern autonomous driving vehicles: fast, lightweight and accurate, for the problem of 3D humanpose estimation from color and 3D signals.
Revised: We have presented a novel deep neural network architecture, which has been tailored for the needs of modern autonomous driving vehicles: fast, lightweight and accurate, for the problem of 3D human pose estimation from color and 3D signals.

Original: Our novel architecture, deemed HUM3DIL, makes usageof both RGB and LiDAR data, by gathering pixel-aligned multi-modal features, that are then fedinto a sequence of Transformer stages.
Revised: Our novel architecture, deemed HUM3DIL, makes usage of both RGB and LiDAR data, by gathering pixel-aligned multi-modal features, that are then fed into a sequence of Transformer stages.

Original: The network is thus informed by multi-modal signals, whichcomplement each other in achieving state-of-the-art performance.
Revised: The network is thus informed by multi-modal signals, which complement each other in achieving state-of-the-art performance.

Original: We also train in a semi-supervisedregime, with limited annotated 3D data, but with an abundance of 2D labels, almost 2 orders ofmagnitude more.
Revised: We also train in a semi-supervised regime, with limited annotated 3D data, but with an abundance of 2D labels, almost 2 orders of magnitude more.

Original: This makes data collection and annotation easier, as 3D signals are non-trivialand tedious to annotate precisely.
Revised: This makes data collection and annotation easier, as 3D signals are non-trivial and tedious to annotate precisely.

Original: The performance of the network is supported by a quantitativeevaluation on one of the largest relevant datasets in the literature, with methodical ablation studies.
Revised: The performance of the network is supported by a quantitative evaluation on one of the largest relevant datasets in the literature, with methodical ablation studies.

Original: For future work, we will explore temporal consistencies between the predictions and include themin motion forecasting and analysis.
Revised: For future work, we will explore temporal consistencies between the predictions and include them in motion forecasting and analysis.