3D-R EL N ET : J OINT O BJECT AND R ELATIONAL N ETWORK FOR 3D P REDICTION
Anonymous authors
Paper under double-blind review
A BSTRACT
We propose an approach to predict the 3D shape and pose for the objects present in a scene.
Existing learning based methods that pursue this goal make independent predictions per object, and do not leverage the relationships amongst them.
We argue that reasoning about these relationships is crucial, and present an approach to incorporate these in a 3D prediction framework.
In addition to independent per-object predictions, we predict pairwise relations in the form of relative 3D pose, and demonstrate that these can be easily incorporated to improve object level estimates.
We report performance across different datasets (SUNCG, NYUv2), and show that our approach signiﬁcantly improves over independent prediction approaches while also outperforming alternate implicit reasoning methods.
I NTRODUCTION
A single 2D image can induce a rich 3D perception.
When we look at an image, we can reason about its 3D layout, the objects in the image, their shape, extent, relationships etc. .
This is really surprising given that going from a 2D projection to a 3D model is inherently ill-posed.
How are we able to solve this problem?
Humans rely on regularities in the 3D world in order to do so – this helps us discard many improbable solutions in 3D and reason about more likely ones.
This regularity exists at the scene level - indoor scenes have roughly perpendicular walls; object levelchairs have similar shapes; and in local object relationships - chairs are close to tables, monitors are on top of tables etc. .
A decade ago, a lot of work in computer vision focused on using all the three levels of regularities.
For example, a lot of work focused on object-centered 3D prediction (Fidler et al., 2012), scene-level 3D prediction (Hedau et al., 2009), and multi-object 3D reasoning (Jiang et al., 2012).
However, in recent years, since the advent of ConvNets, a vast majority of computer vision approaches do not leverage these object-object relationships, and instead reason about each object independently.
In this paper, we attempt to take a holistic view of the 3D prediction problem and note that solving the 3D prediction problem would require incorporation of all the three cues.
We believe there are three fundamental questions that need to be answered to design this holistic architecture: (a) What is the right representation for object level 3D prediction?
; (b) How do we represent object-object relationships and how do we predict them from pixels?
; (c) Finally, how to incorporate objectobject relationships with object-level modules.
This paper builds upon the recent success in (a) and investigates how to model relationships and incorporate them into our 3D prediction framework.
So, how do we model relationships and estimate them from pixels?
There is a whole spectrum of possible approaches.
On one end of the spectrum is a complete end-to-end approach.
Some examples of these include Interaction Networks (Battaglia et al., 2016) or Graph Convolutional
Networks (Kipf & Welling, 2016).
Both these methods provide a mechanism for object features in the scene to effect each other, thereby allowing an implicit modeling of relationships among them.
However, as we show in our experiments, these end-to-end approaches disregard the structural information which might be crucial for modeling the relationships.
The other end of spectrum is to use category-based image-agnostic pairwise priors (Zhao & Zhu, 2011) to model relationships.
A drawback is that these priors are often too strong to generalize and it is better to learn them (Jiang et al., 2012).
When it comes to the ﬁnal question of how does one incorporate relationships to improve 3D prediction, the answer is even murkier.
One classical approach is to use graphical models such as
[Figure]
CRFs (Lafferty et al., 2001; Schwing et al., 2012).
However, these classical approaches have usually provided little improvements over object-based approaches.
Our key insight is to incorporate structural information in end-to-end systems.
Speciﬁcally, we model and predict pairwise relationships in the translation, rotation and scale space.
One advantage of using this structured relationship space is that the incorporation of relationships into object-level estimates is simple yet effective.
But how do we predict these pairwise relationships from pixels?
Our paper investigates several design choices and proposes a simple architecture.
Our method demonstrates signiﬁcant improvement in performance across multiple metrics and datasets.
As we show in our experiments, this modeling of relationships in this structured space provides a huge 6 point AP improvement in detection settings over current state-of-the-art 3D approaches.
We will release our code for reproducibility.
R ELATED W ORK
Dating back to the ﬁrst thesis written on computer vision (Roberts, 1963), inferring the underlying 3D structure of an image has been a long-standing goal in the ﬁeld.
While we have explored several 3D scene representations over the years, e.g. , depth (Saxena et al., 2009; Eigen et al., 2014), qualitative 3D (Hoiem et al., 2005; Gupta et al., 2010), manifolds (Osadchy et al., 2007) or volumetric 3D (Choy et al., 2016; Girdhar et al., 2016; Song et al., 2017) , the prevalent paradigm is still the one followed in Roberts’ seminal work – that of inferring a 3D scene in terms of the shape and pose of the underlying objects.
The initial attempts under this paradigm (Guzm´an, 1968; Huttenlocher & Ullman, 1990) focused on placing known object instances to match image evidence, relying on matching edges, corners etc.
to ﬁt the known shape templates to images.
Subsequent approaches have focused on a more general setting of reconstructing scenes comprising of novel objects, and leverage either explicit or implicitly learned category level priors for pose and shape estimation, typically relying on a deformable shape space (Cashman & Fitzgibbon, 2013; Kar et al., 2015) or template CAD models (Lim et al., 2013;
Aubry et al., 2014; Li et al., 2015; Bansal et al., 2016; Izadinia et al., 2017) for the latter inference.
Current CNN based incarnations of these approaches, driven by the abundant success of deep learning and availability of annotated data, have further improved the results for pose estimation (Tulsiani & Malik, 2015; Pavlakos et al., 2017), and have also been extended to joint shape and pose inference of the objects present in a scene (Kundu et al., 2018; Tulsiani et al., 2018).
A common characteristic amongst these approaches is the reasoning at a per-object level.
While the object-centric nature is certainly desirable as a representation, we argue that reasoning independently for each object to infer this representation is not, as it does not allow leveraging the relationships between the entities in a scene.
We propose a method that also uses object-centric representations but goes beyond independent reasoning per object.
We are of course not the ﬁrst to pursue reasoning about relationships between entities in a scene.
Several previous approaches focus on the goal of predicting various relations e.g. human-object interactions (Gupta & Malik, 2015; Gkioxari et al., 2018), object-object interactions (Gupta et al., 2010; Lu et al., 2016), object-scene interactions (Lee et al., 2010) etc.
While these works pursue
[Figure]
relation inference as the end goal, we instead aim to leverage these for a per-instance prediction task.
In the context of incorporating relations for such per-instance prediction, there are two alternate ideologies.
On the one hand, approaches pursuing 3D scene inference or generation (Zhao & Zhu, 2011; Lin et al., 2013; Choi et al., 2013; Fisher et al., 2012; 2011; Huang et al., 2018; Jiang et al., 2018) typically incorporate pairwise (or higher order) relations via explicit class-based priors regarding possible conﬁgurations and optimize predictions to adhere to these.
This approach of explicitly modeling relations as a prior imposes the same constraints across all scenes, independent of the structure in the image, and is therefore not ﬂexible enough and has difﬁculties in scaling up to arbitrary relations across arbitrary objects.
The alternate ideology for incorporating relationships is to eschew any explicit structure for these relations, and instead implicitly capture these via architectural changes to the CNNs, thereby allowing the features of objects to inﬂuence one another (Battaglia et al., 2016; Kipf & Welling, 2016).
While this a generally applicable mechanism, it does not leverage several aspects regarding the structure of the problem – for 3D inference, speciﬁc relations like relative position, orientation are very relevant, and can be used in speciﬁc ways to inﬂuence per-instance predictions.
Our approach leverages some aspects of both these ideologies – unlike the classical prior based approaches, we learn and infer these relations in a image-dependent context via a CNN, and unlike the purely implicit methods, we are more explicit about the structure and meaning of these relations.
A PPROACH
Our goal is to predict the 3D pose and shape for all the objects in a scene.
We observe that in addition to the visual cues per object, reasoning about relationships across them can further help our predictions, in particular for the 3D pose – a chair would be in front of a table, and of a compatible relative size, and therefore even if we are uncertain about the pose of one of these objects, e.g. , due to occlusion, these relationships can enable us to make accurate predictions.
We operationalize this insight in our method (see ﬁgure 1) that leverages both - independent perobject predictions alongwith predictions regarding relationships between them.
We infer the ﬁnal estimates for all the objects in the scene by integrating these two.
We ﬁrst formally describe the object-centric representations pursued and brieﬂy review a recent per-instance prediction approach in section 3.1.
We then introduce the relative representations in section 3.2 and present our network architecture that enables predicting these in section 3.3.
In section 3.4 we discuss how these relative predictions are combined with the independent per-object predictions to yield the ﬁnal 3D estimates for the objects.
We show that optimizing the combination of these estimates ( i.e. , ﬁnal estimate) in a differentiable end-to-end framework helps improve the ﬁnal per-object predictions.
3. I NSTANCE S PECIFIC R EPRESENTATIONS AND I NFERENCE
We output the 3D pose of an object by predicting its shape in a canonical frame, and its scale, translation and rotation in camera frame.
The shape is parametrized as a 32 3 volumetric occupancy grid in a canonical space where the objects are upright, front-facing, normalized to a unit cube.
The translation t ∈ R 3 and (logarithm of) anisotropic scale s ∈ R 3 , and normalized quaternion q indicate the position, size and orientation of the object respectively.
Following prior work (Tulsiani et al., 2018), we parametrize the rotation prediction as a classiﬁcation task among ﬁxed bins.
We build upon the recent work by (Tulsiani et al., 2018): they pursue a similar per-object representation, but make independent predictions across objects.
We use their approach to obtain the independent predictions for each object.
We brieﬂy review their prediction framework but refer the reader to the paper for details about the representation and architecture.
Their method uses an architecture similar to Fast R-CNN (Girshick, 2015), where the input image is encoded via convolutional layers, and for each object bounding box, an RoI pooling layer crops the corresponding features.
These per-object features, in conjunction with coarse image feature and an encoding of the bounding box coordinates, are encoded to an instance-speciﬁc bottleneck representation from which the corresponding shape and pose are predicted.
We adopt a similar architecture, illustrated in ﬁgure 2, with the introduction of spatial coordinates as additional feature channels (see section 3.3) to obtain per-object (unary) predictions.
We note that these predictions are obtained independently across objects, and do not incorporate reasoning across them.
However, unlike previous work, these are subsequently integrated with relative predictions (section 3.4) to obtain ﬁnal estimates.
3. R ELATIONS AS R ELATIVE R EPRESENTATIONS
Given an image of a scene, we can infer that two chairs nearby might be of a similar size, a laptop kept above a desk, and a television facing the bed etc. .
Thus, relative pose between objects captures an important aspect of their relationship in the scene.
Concretely, given two objects, say A and B , we infer the relative pose from A to B .
The relative pose, akin to the absolute 3D pose, is factored into the relative translation, scale, and direction.
The relative translation is deﬁned as the difference of the absolute locations of the object in the camera frame t AB = t B − t A .
Similarly, the relative scale is simply the ratio of the two object sizes, or equivalently a difference in logarithms s AB = s B − s A .
Finally, we predict a relative direction ˆ d AB , i.e. , the direction of object B in the frame of object A : ˆ d AB ∝ ( ¯ R A ) t AB , where ˆ d AB is normalized to unit norm.
We note that this parametrization, unlike relative rotation, helps us overcome some ambiguities due to symmetries, e.g. , the relative direction from a chair to a table in front of it is unambiguous, even if the table is symmetric.
3. N ETWORK A RCHITECTURE
Our network has two branches – a per-object (unary) prediction module, and a relative prediction module.
The architecture of both these branches is shown in ﬁgure 2.
The former, as described in section 3.1, simply computes a per-object encoding and subsequently makes per-object predictions.
The relative prediction module is then tasked with inferring the relative pose (section 3.2) for every pair of objects in the scene.
As depicted in ﬁgure 2, the relative prediction module takes as input: a) the encoding of both objects, and b) an encoding of the larger (union) bounding box containing both objects.
Since this larger box may contain several additional objects beyond the ones of interest, we additionally give as input two channels with binary masks indicating the source and target bounding box extents respectively (denoted as ‘object location image’).
We also ﬁnd it beneﬁcial to concatenate the normalized per-pixel spatial coordinates as additional features, as it allows the network to easily reason about the absolute spatial location of the bounding box(es) under consideration.
Finally, similar to the parametrization in section 3.1, we frame the relative direction prediction as a classiﬁcation task among 24 bins, where the bins are computed by clustering relative directions across the training instances.
3. C OMBINING P ER - OBJECT AND R ELATIVE P REDICTIONS
We saw in section 3.1 that we can obtain independent per-object predictions for the 3D shape and pose, and introduced the relative pose predictions and architecture in section 3.2 and section 3.3.
We denote by ( t n , s n , q n ) the per-object (unary) predictions for the translation, log-scale, and rotation respectively for the n th object, and similarly ( t mn , s mn , d mn ) denote the relative pose predictions from the m th to the n th object.
Using these, we show that we can obtain ﬁnal per-instance predictions ( t ∗ n , s ∗ n q
∗ n that incorporate both, the unary and relative predictions
Translation and Scale Prediction.
The relative predictions give us linear constraints that the ﬁnal predictions should ideally satisfy.
As an example, we would want t ∗ n − t ∗ m =
t mn .
This can equivalently be expressed as A mn t ∗ =
t mn , where A mn is a sparse row-matrix with the m th and n th entries as ( − 1 , 1) , and t ∗ denoting the ﬁnal translations for all the N objects.
We can similarly express all pairwise linear constraints as A t ∗ = t rel , where t rel denotes all the relative predictions, and A is the appropriate sparse matrix.
In addition to satisfying these linear constraints, we would also like the ﬁnal estimates to be close to the unary predictions.
We can therefore incorporate both, the relative and the unary constraints via a system of linear equations, and solve these to obtain the ﬁnal estimates.
[Equation]
Here X + denotes the Moore-Penrose inverse of matrix X , and λ indicates the relative importance of the unary estimates.
We can therefore obtain the ﬁnal translation predictions t ∗ that integrate both, unary and relative estimates.
We note that the ﬁnal estimates are simply a linear function of the unary predictions t and the relative predictions t rel , and it is therefore straightforward to propagate learning signal from supervision for t ∗ to these predictions.
While the description here focused predicting translation, as we represent scale using logarithm of the sizes, similar linear constraints apply.
We can therefore similarly compute ﬁnal predictions for the scales across objects s ∗ .
Rotation Prediction.
While the incorporation of unary and relative predictions can be expressed via linear constraints in the case of translation and scale, a similar closed form update does not apply for rotation because of the framing as a classiﬁcation task and non-linearity of the manifold.
Instead, we update the likelihood of the unary predictions based on how consistent each rotation bin is with the relative estimates.
We denote as R b the rotation matrix corresponding to the b th rotation bin, and use ∆( R, d , t ) to measure how inconsistent a predicted rotation R is w.r.t the predicted relative direction distribution d and relative translation t (see appendix for details).
Using these, we can compute the (unnormalized) negative-log likelihood distribution over possible rotations as follows:
[Equation]
This update to compute q ∗ can equivalently be viewed as a single round of message passing, with the messages being of an explicit rather than an implicit form.
Training Details.
We described above how the independent per-object predictions are analytically combined with the relative pose predictions to obtain the ﬁnal estimates, and note that this integration process allows us to propagate learning signal from supervision on the ﬁnal estimates back to the unary and relative predictions.
Our training happens in two steps.
In the ﬁrst step, we train both unary and relative predictions independent of each other.
We formulate the loss-function for each network similar to (Tulsiani et al., 2018).
Speciﬁcally, we use regression losses for shape encoding, the (absolute and relative) translation and scale, and classiﬁcation losses for the rotation and relative direction prediction.
Note that as some objects might be rotationally symmetric, we allow multiple ‘correct’ bins for these and maximize the maximum probability across these.
In the second step, after a few epochs, we train the whole model in joint manner.
We add similar losses for the ﬁnal pose predictions that are computed using both, unary and relative estimates.
During inference, given the unary and relative predictions, we simply compute the ﬁnal pose predictions via the optimization process described above.
Additional details on optimization are provided in the appendix.
E XPERIMENTS 4. E XPERIMENTAL S ETUP
Datasets: We use the SUNCG dataset (Song et al., 2017) which provides many diverse and detailed 3D models for houses.
Following (Zhang et al., 2017; Tulsiani et al., 2018), we use the 2D renderings of the houses and the corresponding parsed 3D house model information to get roughly 550k image and 3D supervision pairs.
We follow the setup in (Tulsiani et al., 2018) and split this data into train (70%), val (10%) and test (20%); with objects classes - bed, chair, desk, sofa, table, and tv.
We also use the NYUv2 (Silberman et al., 2012) dataset which consists of 1449 real world images, and use the annotations by (Guo & Hoiem, 2013) to ﬁnetune the network trained on SUNCG using the same subset of object categories.
This dataset has lower resolution images and serves well to check the generalization of our approach to real data.
As the NYUv2 annotations use the same small set of 3D shapes across train and test images, we do not evaluate shape prediction.
Metrics: (Tulsiani et al., 2018) propose different metrics to measure the quality of various prediction components - detection, rotation, translation and scale.
They also propose different thresholds δ for each of these components which are used to count a prediction as a true positive in the detection setting.
We use these metrics and refer the reader to the appendix for a summary review.
Baselines: We use the following baseline methods:
• Factored3D : This method from (Tulsiani et al., 2018) reasons about each object independently and serves as a baseline to see how our relationship reasoning can improve performance.
• GCN : We use Graph Convolutional Networks (Kipf & Welling, 2016) to perform implicit relational reasoning.
We use the Object Encoder (Figure 2) to obtain object embeddings o i for each object and then use a 2-layer GCN to obtain the ﬁnal object embeddings for each object.
These are then used to predict the 3D pose and shape for the objects.
• InteractionNet : We use the method from (Battaglia et al., 2016) as an alternate way to perform implicit reasoning over the object embeddings.
We compute an ‘effect’ embedding e AB for each ordered tuple ( o A , o B ) via a learned MLP,
and update the object embeddings by aggregating these as o A + max B ( e AB ) , and use these for per-object predictions.
Note that while the latter two baselines can implicitly reason about relationships, they ignore the underlying structure (how relative translations affect translation etc. ).
[Figure]
4. U SING G ROUND T RUTH B OXES
We ﬁrst analyze all methods in the setting where we are given the ground truth bounding boxes.
In this setting, we can analyze just the 3D prediction quality without the additional variance introduced due to imperfect detection.
During training, we train all the methods on ground-truth boxes as well as object proposals (obtained using (Zitnick & Doll´ar, 2014)) which have an IOU ≥ 0 .
7 and at test time, we evaluate them only on ground truth boxes.
We evaluate the ﬁnal (combined) output of our method.
As shown in Table 1, our method generates higher quality predictions across both translation and scale where it can outperform the baselines on the mean, median and % error.
It is
[Figure]
GT
[Figure]
Ours
[Figure]
Rotation (degrees)
Scale
[Figure]
Method
Median Mean (
[Table] [Table]
Median Mean
[Table]
Mean
[Table]
SUNCG
Factored3D
[Table]
InteractionNet
[Table]
also worth noting that adding any form of relationships modeling, like in GCN or InteractionNet, gives a boost over doing per-object predictions like in Factored3D. Our structured reasoning and inference about object relationships provides a boost over other pairwise models such as GCN and
InteractionNet.
Our performance gain over the baseline holds across SUNCG and NYUv2 datasets demonstrating the generalization of our method.
Qualitative Results:
In ﬁgure 3, we show a few results of our method and the baseline (Factored3D).
We see that our method can correct many error modes (relative positions and poses of objects) compared to the baseline.
We observe similar trends on the NYUv2 dataset (ﬁgure 4).
[Figure]
4. D ETECTION S ETTING
In this setting, we test the learned models using detections from a pre-trained object detector (taken from (Tulsiani et al., 2018)).
Thus, we can now evaluate the robustness of these methods when the
[Table] [Table]
input object boxes are not pristine.
In Table 2, we see the mean Average Precision values for different criteria used to deﬁne true positives (see appendix for metrics).
As an example, in the ﬁrst column a true positive satisﬁes IOU (box2d) threshold with the ground truth, is close in scale, rotation, translation, shape thresholds.
Each subsequent column examines one criteria so we can analyze their accuracy with detections.
We see (in the ﬁrst column) that our method provides a signiﬁcant gain ofpoints in mAP over the baseline.
Relationship modeling (GCN and InteractionNet) provides beneﬁt in the detection setting too.
However, our structured reasoning outperforms these methods, with most of the gains coming from predicting higher quality scale and translation.
We visualize some predictions showing the difference between our approach and the baseline in ﬁgure 5.
Furthermore, in Table 4 we evaluate the our method on the NYUv2 in the detection setting, and we achieve similar trends in performance with respect to the baseline.
Due to increased difﬁculty of the task on NYU we observe relative lower mAP scores.
4. E FFECT OF C OMBINATION AND O PTIMIZATION
Combining the unary and relative predictions provides beneﬁt at the training time because it allows the model to modify its unary and relative predictions so that they are more ‘compatible’ with each other.
We quantify this in Table 3 using the SUNCG dataset.
We report the performance of a purely multi-task version (‘MT’) that only predicts the unaries and relative pose values, without ever combining them.
The ‘MT + combine test only’ combines these during inference (but not training).
Finally, the ‘MT + combine train only’.
We also compare to a method that only uses this combination at train time, and ﬁnally report our full method as a reference.
We see that combining at either train or test alone performs better than pure multi-task learning.
This shows the importance of the unary and relative predictions interacting with each other.
Combining these at both train and test time, and jointly optimizing like in our method performs the best. [Table]
D ISCUSSION A ND F UTURE W ORK
We proposed a method to incorporate relationship based reasoning in the form to relative pose estimates for the task of 3D scene inference.
While this allowed us to signiﬁcantly improve over existing approaches that reason independently across objects, numerous challenges still remain to be addressed.
In our approach, we only leveraged pairwise relations among the objects in a scene, and it would be interesting to pursue incorporating higher order relations.
We also relied on a synthetic dataset with full 3D supervision to train our prediction networks, thereby limiting direct applicability to datasets without 3D supervision.
Towards overcoming this, it might be desirable to combine our approach with parallel efforts in the community to use 2D reprojection losses (Garg et al., 2016) or leverage domain adaptation techniques (Hoffman et al., 2017).
A PPENDIX 6. M ETRICS
We use the metrics from (Tulsiani et al., 2018) and summarize them below.
• Translation ( t ): Euclidean distance between prediction and ground-truth (cid:107) t p − t gt (cid:107) .
δ t ≤ 0 .
5 m. • Scale ( s ):
We measure the average unsigned difference in log-scale, i.e. , ∆( s p , s gt ) = 13 (cid:80)
i =1 | log 2 ( s i p ) − log 2 ( s i gt ) | .
We threshold at δ s ≤ 0 .
• Rotation ( q ): Geodesic distance between rotations 1 √ 2 (cid:107) log( R (cid:124) p R gt ) (cid:107) .
δ q ≤ 30 ◦ .
For objects that exhibit rotational symmetry, we use the lowest error across the different possible values of R gt .
• Shape ( V ):
Following (Choy et al., 2016), we measure the intersection over union (IoU) and use threshold δ V = 0 . 25 .
As a higher IOU is better, so we use δ V ≥ 0 .
25 for true positive.
• Bounding Box overlap ( b ): The bounding box overlap is measured using IOU.
δ b ≥ 0 .
• Detection: A prediction is considered a true positive when it satisﬁes the thresholds for each of the above components ( δ t , δ s , δ q , δ V , δ b ) .
We use Average Precision (AP) to measure the ﬁnal detection performance.
6. T RAINING D ETAILS
Optimization.
We train our network in two stages.
In the ﬁrst stage of training we use ground truth boxes.
We train for 8 epochs by using adam optimizer with a learning rate of 10 − 4 .
During the ﬁrst 4 epochs of the training we train for relative and object speciﬁc predictions independently and during next 4 epochs of the training we optimize the whole model jointly by combining the relative and object speciﬁc estimates.
In the next stage we consider overlapping proposals with IOU of over 0.7 with respect to ground truth boxes and the ground truth boxes as positive proposals to further make the model robust in the detection setting.
In the NYUv2 setting we start with a network trained on the SUNCG dataset and ﬁnetune the network for 16 epochs on the NYU train + val split and evaluate method on the test split.
Rotation Prediction.
We deﬁned ∆( R, d, t ) as a measure of how inconsistent a predicted rotation
R is w.r.t the predicted relative direction distribution d and relative translation t .
Given a predicted rotation R , we would expect the predicted direction to align with the vector ¯ R ˆ t , where ˆ t is unitnormalized.
Note that the predicted d is a probability distribution over possible directions, and let d ∗ denote the bin that aligns maximally with ¯ R ˆ t .
We measure ∆( R, d, t ) by combining measures of how likely this bin is with how well it agrees with the rotation and translation: ∆( R, d, t )
=
− log p ( d ∗ ) + (1 − cos( d ∗ , ¯ R ˆ t )) .
Relative Importance.
We use lambda for unary importance to get t ∗ and s ∗ as 1.
In case of rotation we use we weight for the relative predictions as min(5 .
0 /n, 1) where n represents number of neighbours of the object.
In the detection setting we create set of valid objects which are allowed to inﬂuence the ﬁnal predictions for other objects based upon the detection score.
We consider objects with a score above 0.3 to be part of the valid set and only use them to get ﬁnal predictions for other objects.
6. A RCHITECTURE A NALYSIS 6. A DDITIONAL V ISUALIZATIONS AND R ESULTS
We visualize the precision-recall curves in the detection setting using the SUNCG dataset in ﬁgure 6.
We also visualize predictions for randomly sampled images in the setting with known bounding boxes in ﬁgure 7.
[Figure] [Figure]
GT
Factored3D
Ours
Image
GT
Factored3D
Ours