Original: Iterative Scene Graph Generation
Revised: Iterative Scene Graph Generation

Original: Anonymous Author(s)

Original: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: The task of scene graph generation entails identifying object entities and theircorresponding interaction predicates in a given image (or video).
Revised: The task of scene graph generation entails identifying object entities and their corresponding interaction predicates in a given image (or video).

Original: Due to thecombinatorially large solution space, existing approaches to scene graph generationassume certain factorization of the joint distribution to make the estimation feasible( e.g ., assuming that objects are conditionally independent of predicate predictions).
Revised: Due to the combinatorially large solution space, existing approaches to scene graph generation assume certain factorization of the joint distribution to make the estimation feasible ( e.g ., assuming that objects are conditionally independent of predicate predictions).

Original: However, this ﬁxed factorization is not ideal under all scenarios ( e.g ., for imageswhere an object entailed in interaction is small and not discernible on its own).
Revised: However, this ﬁxed factorization is not ideal under all scenarios ( e.g ., for images where an object entailed in interaction is small and not discernible on its own).

Original: In this work, we propose a novel framework for scene graph generation thataddresses this limitation, as well as introduces dynamic conditioning on the image,using message passing in a Markov Random Field.
Revised: In this work, we propose a novel framework for scene graph generation that addresses this limitation, as well as introduces dynamic conditioning on the image, using message passing in a Markov Random Field.

Original: This is implemented as aniterative reﬁnement procedure wherein each modiﬁcation is conditioned on thegraph generated in the previous iteration.
Revised: This is implemented as an iterative reﬁnement procedure wherein each modiﬁcation is conditioned on the graph generated in the previous iteration.

Original: This conditioning across reﬁnementsteps allows joint reasoning over entities and relations.
Revised: This conditioning across reﬁnement steps allows joint reasoning over entities and relations.

Original: This framework is realizedvia a novel and end-to-end trainable transformer-based architecture.
Revised: This framework is realized via a novel and end-to-end trainable transformer-based architecture.

Original: In addition,the proposed framework can improve existing approach performance.
Revised: In addition, the proposed framework can improve existing approach performance.

Original: Throughextensive experiments on Visual Genome [29] and Action Genome [24] benchmarkdatasets we show improved performance on the scene graph generation.
Revised: Through extensive experiments on Visual Genome [30] and Action Genome [25] benchmark datasets we show improved performance on the scene graph generation task.

Revised: The code is available at github.com/ubc-vision/IterativeSG .

Original:  Introduction
Revised:  Introduction

Original: Scene graphs allow for structured understanding of objects and their interactions within a scene.
Revised: Scene graphs allow for structured understanding of objects and their interactions within a scene.

Original: Ascene graph is a graph where nodes represent objects within the scene, each detailed by the class labeland spatial location and the edges capture the relationships between object pairs.
Revised: A scene graph is a graph where nodes represent objects within the scene, each detailed by the class label and spatial location, and the edges capture the relationships between object pairs.

Original: These relationshipsare usually represented by a <subject , predicate , object> triple.
Revised: These relationships are usually represented by a <subject , predicate , object> triple.

Original: Effectively generating suchgraphs, from either images or videos, has emerged as a core problem in computer vision [13, 34, 38,44, 48, 50, 52].
Revised: Effectively generating such graphs, from either images or videos, has emerged as a core problem in computer vision [14, 35, 39, 46, 50, 52, 54].

Original: Scene graph representations can be leveraged to improve performance on a variety ofcomplex high-level tasks like VQA [23, 45], Image Captioning [16, 51] and Image Generation [25].
Revised: Scene graph representations can be leveraged to improve performance on a variety of complex high-level tasks like VQA [24, 47], Image Captioning [17, 53], and Image Generation [26].

Original: The task of scene graph generation involves estimating the conditional distribution of the relationship triplets given an image.
Revised: The task of scene graph generation involves estimating the conditional distribution of the relationship triplets given an image.

Original: Naively modelling this distribution is often infeasible as the space ofpossible relationship triplets is considerably larger than the space of possible subjects, objects, andpredicates.
Revised: Naively modelling this distribution is often infeasible as the space of possible relationship triplets is considerably larger than the space of possible subjects, objects, and predicates.

Original: To circumvent this issue, existing methods factorize the aforementioned distributioninto easy-to-estimate conditionals.
Revised: To circumvent this issue, existing methods factorize the aforementioned distribution into easy-to-estimate conditionals.

Original: For example, two-stage approaches, like [10, 45, 53], follow thegraphical model image Ñ r subject , object s Ñ predicate , wherein the subjects and objects are31 independently obtained via a pre-trained detector like Faster-RCNN [40].
Revised: For example, two-stage approaches, like [11, 47, 55], follow the graphical model image Ñ r subject , object s Ñ predicate , wherein the subjects and objects are independently obtained via a pre-trained detector like Faster-RCNN [41].

Original: These are then consumedby a downstream network to estimate predicates.
Revised: These are then consumed by a downstream network to estimate predicates.

Original: Any such factorization induces conditional dependencies (and independencies) that heavily inﬂuence model characteristics.
Revised: Any such factorization induces conditional dependencies (and independencies) that heavily inﬂuence model characteristics.

Original: For example, in theaforesaid graphical model, errors made during the estimation of the subjects and objects are naturallypropagated towards the predicate distribution, which makes the estimation of predicates involvingclasses with poor detectors challenging.
Revised: For example, in the aforesaid graphical model, errors made during the estimation of the subjects and objects are naturally propagated towards the predicate distribution, which makes the estimation of predicates involving 36th Conference on Neural Information Processing Systems (NeurIPS 2022). classes with poor detectors challenging.

Original: Furthermore, the assumed ﬁxed factorization might notalways be optimal.
Revised: Furthermore, the assumed ﬁxed factorization might not always be optimal.

Original: Having information about predicates in an image can help narrow down the spaceof possible subjects/objects, e.g ., predicate wearing makes it likely that the subject is a person .
Revised: Having information about predicates in an image can help narrow down the space of possible subjects/objects, e.g ., predicate wearing makes it likely that the subject is a person .

Original: Additionally, due to the two-stage nature of most existing scene graph approaches (barring veryrecent methods like [13, 30, 34]), the image feature representations obtained from a pre-trainedtask-oblivious detector might not be optimally catered towards the scene graph generation problem.
Revised: Additionally, due to the two-stage nature of most existing scene graph approaches (barring very recent methods like [14, 31, 35]), the image feature representations obtained from a pre-trained task-oblivious detector might not be optimally catered towards the scene graph generation problem.

Original: Intuitively, one can imagine that the information required to accurately localize an object might notnecessarily be sufﬁcient for predicate prediction, and by extension, accurate scene graph generation.
Revised: Intuitively, one can imagine that the information required to accurately localize an object might not necessarily be sufﬁcient for predicate prediction, and by extension, accurate scene graph generation.

Original: Morever, two-stage approaches often suffer with efﬁciency issues as the detected objects are requiredto be paired up before predicate assignment.
Revised: Morever, two-stage approaches often suffer with efﬁciency issues as the detected objects are required to be paired up before predicate assignment.

Original: Doing so naively, by pairing all possible objects [48],results in quadratic number of pairs that need to be considered.
Revised: Doing so naively, by pairing all possible objects [50], results in quadratic number of pairs that need to be considered.

Original: Traditional approaches deal with thisusing heuristics, such as IoU-based threshold-based pairing [45, 53].
Revised: Traditional approaches deal with this using heuristics, such as IoU-based threshold-based pairing [47, 55].

Original: our proposed approach essentially models an iterative reﬁnement process (b) wherein each modiﬁcation step is conditioned on the previously generated graph estimate.
Revised: our proposed approach essentially models an iterative reﬁnement process (b) wherein each modiﬁcation step is conditioned on the previously generated graph estimate.

Original: In this work, we aim to alleviate the previously mentioned issues arising from a ﬁxed factorization andpotentially limited object-centric feature representations by proposing a general framework wherein thesubject, objects, and predicates can be inferred jointly( i.e ., depend on one another), while simultaneouslyavoiding the complexity of exponential search space ofrelational triplets.
Revised: In this work, we aim to alleviate the previously mentioned issues arising from a ﬁxed factorization and potentially limited object-centric feature representations by proposing a general framework wherein the subject, objects, and predicates can be inferred jointly ( i.e ., depend on one another), while simultaneously avoiding the complexity of exponential search space of relational triplets.

Original: This is achieved by performing message passing within a Markov Random Field (MRF)deﬁned by components of a relational triplet (Figure1(a)).
Revised: This is achieved by performing message passing within a Markov Random Field (MRF) deﬁned by components of a relational triplet (Figure 1(a)).

Original: Unrolling this message passing is equivalent toan iterative reﬁnement procedure (Figure 1(b)), whereeach message passing stage takes in the estimate fromthe previous step.
Revised: Unrolling this message passing is equivalent to an iterative reﬁnement procedure (Figure 1(b)), where each message passing stage takes in the estimate from the previous step.

Original: Our proposed framework modelsthis iterative reﬁnement strategy by ﬁrst producing ascene graph estimate using a traditional factorization,and then systematically improving it over multiplereﬁnement steps, wherein each reﬁnement is conditioned on the graph generated in the previous iteration.
Revised: Our proposed framework models this iterative reﬁnement strategy by ﬁrst producing a scene graph estimate using a traditional factorization, and then systematically improving it over multiple reﬁnement steps, wherein each reﬁnement is conditioned on the graph generated in the previous iteration.

Original: This conditioning across reﬁnement steps compensates for the conditional independence assumptionsand lets our framework jointly reason over the subjects ( s ), objects ( o ), and predicates ( p ).
Revised: This conditioning across reﬁnement steps compensates for the conditional independence assumptions and lets our framework jointly reason over the subjects ( s ), objects ( o ), and predicates ( p ).

Original: Contributions.
Revised: Contributions.

Original: To realize the aforementioned iterative framework, we propose a novel and intuitivetransformer [46] based architecture.
Revised: To realize the aforementioned iterative framework, we propose a novel and intuitive transformer [48] based architecture.

Original: On a technical level, our model deﬁnes three separate multi-layermulti-output synchronized decoders, wherein each decoder layer is tasked with modeling either thesubject, object, or predicate components of a relationship triplets.
Revised: On a technical level, our model deﬁnes three separate multi-layer multi-output synchronized decoders, wherein each decoder layer is tasked with modeling either the subject, object, or predicate components of a relationship triplets.

Original: Therefore, the combined outputsfrom each layer of the three decoders generates a scene graph estimate.
Revised: Therefore, the combined outputs from each layer of the three decoders generates a scene graph estimate.

Original: The inputs to each decoderlayer are conditioned to enable joint reasoning across decoders and effective reﬁnement of previouslayer estimates.
Revised: The inputs to each decoder layer are conditioned to enable joint reasoning across decoders and effective reﬁnement of previous layer estimates.

Original: This conditioning is achieved implicitly via a novel joint loss, and explicitly via crossdecoder and layer-wise attention.
Revised: This conditioning is achieved implicitly via a novel joint loss, and explicitly via crossdecoder and layer-wise attention.

Original: Additionally, each decoder layer is also conditioned on the imagefeatures, which are provided by a shared encoder.
Revised: Additionally, each decoder layer is also conditioned on the image features, which are provided by a shared encoder.

Original: As our proposed model is end-to-end trainable, itaddresses the limitation of two-stage approaches, allowing image features to directly adopt to thescene graph generation task.
Revised: As our proposed model is end-to-end trainable, it addresses the limitation of two-stage approaches, allowing image features to directly adopt to the scene graph generation task.

Original: Finally, to tackle the long-tail nature of the scene graph predicate classes[10], we employ a loss weighting strategy to enable ﬂexible trade-off between dominant (head) andunderrepresented (tail) predicate classes in the long-tail distribution.
Revised: Finally, to tackle the long-tail nature of the scene graph predicate classes [11], we employ a loss weighting strategy to enable ﬂexible trade-off between dominant (head) and underrepresented (tail) predicate classes in the long-tail distribution.

Original: In contrast to data samplingstrategies [10, 31], this has a beneﬁt of not requiring additional ﬁne-tuning of models with sampleddata post training.
Revised: In contrast to data sampling strategies [11, 32], this has a beneﬁt of not requiring additional ﬁne-tuning of models with sampled data post training.

Original: We illustrate that our proposed architecture achieves state-of-the-art performanceon two benchmark datasets – Visual Genome [29] and Action Genome [24]; and thoroughly analyzeeffectiveness of the approach as a function of the reﬁnement steps, design choices employed and as ageneric add-on to an existing, MOTIF [53], architecture.
Revised: We illustrate that our proposed architecture achieves state-of-the-art performance on two benchmark datasets – Visual Genome [30] and Action Genome [25]; and thoroughly analyze effectiveness of the approach as a function of the reﬁnement steps, design choices employed and as a generic add-on to an existing, MOTIF [55], architecture.

Original:  Related Work
Revised:  Related Work

Original: Scene Graph Generation.
Revised: Scene Graph Generation.

Original: Scene graph generation has emerged as a popular research area in thevision community [10, 27, 30, 34, 38, 39, 44, 45, 48, 50, 52, 53].
Revised: Scene graph generation has emerged as a popular research area in the vision community [11, 28, 31, 35, 39, 40, 46, 47, 50, 52, 54, 55].

Original: Existing scene graph generationmethods can be broadly categorized as either one-stage or two-stage approaches.
Revised: Existing scene graph generation methods can be broadly categorized as either one-stage or two-stage approaches.

Original: The ﬁrst step ofthe predominant approach – the two-stage methods – involves pre-training a strong object detectorfor all object classes in the dataset, usually using detector like Faster-RCNN [40].
Revised: The ﬁrst step of the predominant approach – the two-stage methods – involves pre-training a strong object detector for all object classes in the dataset, usually using detector like Faster-RCNN [41].

Original: The graphgeneration network is then built on top of the object information (bounding boxes and correspondingfeatures) obtained from the detector.
Revised: The graph generation network is then built on top of the object information (bounding boxes and corresponding features) obtained from the detector.

Original: This second step entails freezing the detector parameters andsolely training the graph generation network.
Revised: This second step entails freezing the detector parameters and solely training the graph generation network.

Original: The graph generation network is realized via different architectures such as Bi-directional LSTMs [53], Tree-LSTMs [45], Graph Neural Networks [50],and other message passing algorithms [39, 48].
Revised: The graph generation network is realized via different architectures such as Bi-directional LSTMs [55], Tree-LSTMs [47], Graph Neural Networks [52], and other message passing algorithms [40, 50].

Original: The limitation of all two-stage approaches is apparent– the graph generation network has no inﬂuence over the detector and object features.
Revised: The limitation of all two-stage approaches is apparent – the graph generation network has no inﬂuence over the detector and object features.

Original: One stageapproaches overcome this obstacle by employing architectures that allow end-to-end training, suchas fully convolutional networks [34] and transformer based models [13, 30].
Revised: One stage approaches overcome this obstacle by employing architectures that allow end-to-end training, such as fully convolutional networks [35] and transformer based models [7, 14, 31, 43].

Original: The joint optimizationenables interaction between the detection and the graph generation modules, leading to better scenegraphs.
Revised: The joint optimization enables interaction between the detection and the graph generation modules, leading to better scene graphs.

Original: However, all the aforementioned methods operate under the assumption of a ﬁxed factorizedmodel, inducing conditional independencies that might not be ideal under all scenarios.
Revised: However, all the aforementioned methods operate under the assumption of a ﬁxed factorized model, inducing conditional independencies that might not be ideal under all scenarios.

Original: Long-Tail Recognition.
Revised: Long-Tail Recognition.

Original: The task of scene graph generation suffers from the challenge of longtail recognition due to the combinatorial nature of relational triplets (both object and predicatedistributions are skewed).
Revised: The task of scene graph generation suffers from the challenge of longtail recognition due to the combinatorial nature of relational triplets (both object and predicate distributions are skewed).

Original: Long-tail recognition, in general, is a well studied problem in literature.
Revised: Long-tail recognition, in general, is a well studied problem in literature.

Original: Data sampling is a popular strategy [3,19, 20, 26, 36, 41, 56] wherein the training data is modiﬁed toeither over-represent tail classes (oversampling) or under-represent the head classes (undersampling).
Revised: Data sampling is a popular strategy [3, 20, 21, 27, 37, 42, 58] wherein the training data is modiﬁed to either over-represent tail classes (oversampling) or under-represent the head classes (undersampling).

Original: Speciﬁc to the task of scene graph generation, [10, 31] have explored the use of data samplingstrategies to improve performance on tail predicate classes.
Revised: Speciﬁc to the task of scene graph generation, [11, 32] have explored the use of data sampling strategies to improve performance on tail predicate classes.

Original: On the contrary, Loss re-weightingstrategies [1, 8, 12, 26, 33] assign higher weights or impose larger decision boundaries for tailclasses.
Revised: On the contrary, Loss re-weighting strategies [1, 9, 13, 27, 34] assign higher weights or impose larger decision boundaries for tail classes.

Original: [49] recently adopted this paradigm for the task of scene graph generation by proposinga re-weighting strategy based on correlations between predicate classes.
Revised: [51] recently adopted this paradigm for the task of scene graph generation by proposing a re-weighting strategy based on correlations between predicate classes.

Original: Conceptually similar to[49], we propose a novel re-weighting strategy and illustrate its effectiveness both on its own, and incombination with data resampling.
Revised: Conceptually similar to [51], we propose a novel re-weighting strategy and illustrate its effectiveness both on its own, and in combination with data resampling.

Original: Transformer Models.
Revised: Transformer Models.

Original: In [46], authors introduced a new attention-based architecture called transformers for the task of machine translation, doing away with recurrent or convolutional architectures.
Revised: In [48], authors introduced a new attention-based architecture called transformers for the task of machine translation, doing away with recurrent or convolutional architectures.

Original: Transformers have since widely been adopted for a variety of tasks, such as object detection [2, 37, 55],image captioning [7, 22], and image generation [11].
Revised: Transformers have since widely been adopted for a variety of tasks, such as object detection [2, 38, 57], image captioning [8, 23], and image generation [12].

Original: More recently, scene graph generation methodshave also adopted transformer architectures [13, 30], owing to their end-to-end trainable nature andparallelism.
Revised: More recently, scene graph generation methods have also adopted transformer architectures [7, 14, 31, 43], owing to their end-to-end trainable nature and parallelism.

Original: Perhaps conceptually closest, [13], which focuses on visual relation detection (VRD) andnot scene graph prediction, leverages the composite nature of relationships to simultaneously decodean entire relationship triplet alongside its constituent subject, object, and predicate components.
Revised: Perhaps conceptually closest, [14], which focuses on visual relation detection (VRD) and not scene graph prediction, leverages the composite nature of relationships to simultaneously decode an entire relationship triplet alongside its constituent subject, object, and predicate components.

Original: Unlike [13], we do not explicitly model subject-predicate-object "sum" composites, resulting in asimpler architecture, and leverage iterative reﬁnement procedure that relies on a different factorizedattention mechanism.
Revised: Unlike [14], we do not explicitly model subject-predicate-object "sum" composites, resulting in a simpler architecture, and leverage iterative reﬁnement procedure that relies on a different factorized attention mechanism.

Original: In the latest, concurrent work, [30] generates a set of entities and predicatesseparately, and then utilizes a graph assembly procedure to match the predicates to a pair of entities.
Revised: In the latest, concurrent work, [31] generates a set of entities and predicates separately, and then utilizes a graph assembly procedure to match the predicates to a pair of entities.

Original: In contrast to our formulation, this precludes conditioning of entities on predicate estimates; thereforerelying on accuracy of less contextualized predictions for subsequent pairing.
Revised: In contrast to our formulation, this precludes conditioning of entities on predicate estimates; therefore relying on accuracy of less contextualized predictions for subsequent pairing.

Original:  Formulation
Revised:  Formulation

Original: For a given image I , a scene graph G can be represented as a set of triplets G “ t r i u i § n “tp s i , p i , o i qu i § n , where r i denotes the i -th triplet p
Revised: For a given image I , a scene graph G can be represented as a set of triplets G “ t r i u i § n “ tp s i , p i , o i qu i § n , where r i denotes the i -th triplet p

Original: s i , p i , o i q , and n denotes the total number of133 triplets.
Revised: s i , p i , o i q , and n denotes the total number of triplets.

Original: The subject s i denotes a tuple p s i,c , s i,b q , where s i,c P R ⌘ is the one-hot class label, ands i.b P R 4 is the corresponding bounding box coordinates.
Revised: The subject s i denotes a tuple p s i,c , s i,b q , where s i,c P R ⌘ is the one-hot class label, and s i.b P R 4 is the corresponding bounding box coordinates.

Original: ⌘ is the total number of possible entityclasses in the dataset.
Revised: ⌘ is the total number of possible entity classes in the dataset.

Original: The object o i and predicate p i can similarly be represented as tuples p
Revised: The object o i and predicate p i can similarly be represented as tuples p

Original: o i,c , o i,b q136 and p p i,c , p i,b q respectively.
Revised: o i,c , o i,b q and p p i,c , p i,b q respectively.

Original: Note that, p i,b corresponds to the box formed by the centers’ of s i,b and137 o i,b as the diagonally opposite coordinates.
Revised: Note that, p i,b corresponds to the box formed by the centers’ of s i,b and o i,b as the diagonally opposite coordinates.

Original: Additionally, p i,c P R  represents the correspondingone-hot predicate label between the pair p s i , o i q , where  is the total number of possible predicateclasses in the dataset.
Revised: Additionally, p i,c P R  represents the corresponding one-hot predicate label between the pair p s i , o i q , where  is the total number of possible predicate classes in the dataset.

Original: Then the task of scene graph generation can be thought of as modelling theconditional distribution Pr p G | I q .
Revised: Then the task of scene graph generation can be thought of as modelling the conditional distribution Pr p G | I q .

Original: This distribution can be expressed as a product of conditionals,
Revised: This distribution can be expressed as a product of conditionals,

Original: [Equation]
Revised: [Equation]

Original: where t .
Revised: where t .

Original: u denotes a set.
Revised: u denotes a set.

Original: For brevity, we omit explicitly mentioning the total number of set elements142 n throughout the paper.
Revised: For brevity, we omit explicitly mentioning the total number of set elements n throughout the paper.

Original: Existing approaches model this product of conditionals by making someunderlying assumptions.
Revised: Existing approaches model this product of conditionals by making some underlying assumptions.

Original: For example, [10, 45, 53] assume conditional independence between s i ando i as they rely on heuristics to obtain the entity pairs.
Revised: For example, [11, 47, 55] assume conditional independence between s i and o i as they rely on heuristics to obtain the entity pairs.

Original: However, modelling the conditional in Equation1 (or any other equivalent factorization) in such a “one-shot” manner makes certain assumptions onthe ﬂow of information, which, in this case, is from s i Ñ
Revised: However, modelling the conditional in Equation 1 (or any other equivalent factorization) in such a “one-shot” manner makes certain assumptions on the ﬂow of information, which, in this case, is from s i Ñ

Original: o i
Revised: o i

Original: Ñ p
Revised: Ñ p

Original: i .
Revised: i .

Original: Therefore, any errors madeduring the estimation of s i are naturally propagated towards the estimation of o i and p i .
Revised: Therefore, any errors made during the estimation of s i are naturally propagated towards the estimation of o i and p i .

Original: Additionally,the subject (or object) estimation procedure Pr pt s i u | I q is completely oblivious to the estimated149 predicate p i .
Revised: Additionally, the subject (or object) estimation procedure Pr pt s i u | I q is completely oblivious to the estimated predicate p i .

Original: Having access to such information can help the subject (or object) predictor update itsbeliefs and signiﬁcantly narrow down the space of feasible entity pairs.
Revised: Having access to such information can help the subject (or object) predictor update its beliefs and signiﬁcantly narrow down the space of feasible entity pairs.

Original: Contrary to existing works, our proposed formulation moves away from the “one-shot” generationideology described in Equation 1.
Revised: Contrary to existing works, our proposed formulation moves away from the “one-shot” generation ideology described in Equation 1.

Original: We instead argue for modelling the task of scene graph generationas an iterative reﬁnement procedure, wherein the scene graph estimate at step t , G t , is dependenton the previous estimates t G t 1 u t 1 † t .
Revised: We instead argue for modelling the task of scene graph generation as an iterative reﬁnement procedure, wherein the scene graph estimate at step t , G t , is dependent on the previous estimates t G t 1 u t 1 † t .

Original: Formally, our aim is to model the conditional distribution
Revised: Formally, our aim is to model the conditional distribution

Original: Pr
Revised: Pr

Original: [Equation]
Revised: [Equation]

Original: ¯ .
Revised: ¯ .

Original: Assuming Markov property holds, this can be conveniently factorized as,
Revised: Assuming Markov property holds, this can be conveniently factorized as,

Original: [Equation]
Revised: [Equation]

Original: Object Predictor
Revised: Subject Predictor

Revised: [Equation]

Revised: ˘

Revised: [Equation]

Revised: Object Predictor hkkkkkkkkkkkkkkikkkkkkkkkkkkkkj

Original: [Equation]
Revised: [Equation]

Original: Note that even though we assume the ﬂow of information to be from s ti Ñ o ti Ñ p ti for G t , conditioning on the previous graph estimate G t ´ 1 allows the subject, object, and predicate predictors to jointlyreason and update their beliefs, leading to better predictions.
Revised: Note that even though we assume the ﬂow of information to be from s ti Ñ o ti Ñ p ti for G t , conditioning on the previous graph estimate G t ´ 1 allows the subject, object, and predicate predictors to jointly reason and update their beliefs, leading to better predictions.

Original: Additionally, the framework describedin Equation 2 is model agnostic and can be implemented using any of the existing architectures.
Revised: Additionally, the framework described in Equation 2 is model agnostic and can be implemented using any of the existing architectures.

Original:  Transformer Based Iterative Generation
Revised:  Transformer Based Iterative Generation

Original: As described in Section 3, our proposed iterative scene graph generation formulation is agnostic tothe architectural choices used to realise the subject, object, and predicate predictors.
Revised: As described in Section 3, our proposed iterative scene graph generation formulation is agnostic to the architectural choices used to realise the subject, object, and predicate predictors.

Original: In this sectionwe provide a realization of the proposed formulation in Equation 2 using Transformer networks [46].
Revised: In this section we provide a realization of the proposed formulation in Equation 2 using Transformer networks [48].

Original: The choice of using transformer networks is motivated by their natural tendency to model iterativereﬁnement behaviour under the Markov property, wherein each layer of the transformer decoder takesas input the output of the previous layer.
Revised: The choice of using transformer networks is motivated by their natural tendency to model iterative reﬁnement behaviour under the Markov property, wherein each layer of the transformer decoder takes as input the output of the previous layer.

Original: Our novel end-to-end trainable transformer based iterativegeneration architecture builds on top of the DETR [2] framework, which is shown to be effective forthe task of object detection.
Revised: Our novel end-to-end trainable transformer based iterative generation architecture builds on top of the DETR [2] framework, which is shown to be effective for the task of object detection.

Original: Our proposed model architecture is shown in Figure 3.
Revised: Our proposed model architecture is shown in Figure 3.

Original: Given an image, our proposed approach ﬁrst obtains corresponding image features using a combination of a convolutional backbone and a multi-layer transformer encoder, akin to DETR [2].
Revised: Given an image, our proposed approach ﬁrst obtains corresponding image features using a combination of a convolutional backbone and a multi-layer transformer encoder, akin to DETR [2].

Original: Theseimage features are used as inputs to the subject, object, and predicate predictors, each implemented asa multi-layer Transformer decoder.
Revised: These image features are used as inputs to the subject, object, and predicate predictors, each implemented as a multi-layer Transformer decoder.

Original: To generate a scene graph estimate p
Revised: To generate a scene graph estimate p

Original: G t at step t in accordance with
Revised: G t at step t in accordance with

Original: Equation 2, the queries used in each predictor decoder are appropriately conditioned.
Revised: Equation 2, the queries used in each predictor decoder are appropriately conditioned.

Original: For example, inthe case of the predicate decoder, its input queries are conditioned on the subject ( t p s ti u ) and object175 ( t p o ti u ) estimates at step t .Additionally, the input to each predictor decoder is infused with all decoderestimates tp p s [Equation] , p p
Revised: For example, in the case of the predicate decoder, its input queries are conditioned on the subject ( t p s ti u ) and object ( t p o ti u ) estimates at step t .

Original: [Equation] , p o t [Equation] 1 i qu from the previous step t ´ 1 via a structured attentional mechanism.
Revised: Additionally, the input to each predictor decoder is infused with all decoder estimates tp p s t ´ 1 i , p p t ´ 1 i , p o t ´ 1 i qu from the previous step t ´ 1 via a structured attentional mechanism.

Original: The entire model is trained end-to-end, with a novel joint loss applied at each step t to ensure thegeneration of a valid scene graph at every level.
Revised: The entire model is trained end-to-end, with a novel joint loss applied at each step t to ensure the generation of a valid scene graph at every level.

Original: This section describes these components in detail.
Revised: This section describes these components in detail.

Original: 4.1 Image Encoder
Revised: 4.1 Image Encoder

Original: Similar to DETR [2], for each image I , our proposed architecture uses a deep convolutional network(like ResNet [21]) to obtain image level spatial map
Revised: Similar to DETR [2], for each image I , our proposed architecture uses a deep convolutional network (like ResNet [22]) to obtain image level spatial map

Original: ¯ I P R c
Revised: ¯ I P R c

Original: ˆ w ˆ h , where c is the number of channels,and w, h correspond to the spatial dimensions.
Revised: ˆ w ˆ h , where c is the number of channels, and w, h correspond to the spatial dimensions.

Original: A multi-layer encoder f e then transforms ¯ I into aposition-aware ﬂattened image feature representation Z P R d ˆ wh , where d † c .
Revised: A multi-layer encoder f e then transforms ¯ I into a position-aware ﬂattened image feature representation Z P R d ˆ wh , where d † c .

Original: 4.2 Predictor Decoders
Revised: 4.2 Predictor Decoders

Original: Our approach models each of the subject, object, and predicate predictors using a multi-layertransformer decoder [2], which are denoted by f s , f o , and f p respectively.
Revised: Our approach models each of the subject, object, and predicate predictors using a multi-layer transformer decoder [2], which are denoted by f s , f o , and f p respectively.

Original: The t -th layer of eachdecoder, denoted as f t x ; x P t s , o , p u , is tasked with generating the step t scene graph G t .
Revised: The t -th layer of each decoder, denoted as f t x ; x P t s , o , p u , is tasked with generating the step t scene graph G t .

Original: Therefore,at each step t , the decoders output a set of n feature representations t q t x ,i u ; x P t s , o , p u , which aretransformed into a set of triplet estimates tp p s ti , p p ti , p o ti qu via fully-connected feed forward layers.
Revised: Therefore, at each step t , the decoders output a set of n feature representations t q t x ,i u ; x P t s , o , p u , which are transformed into a set of triplet estimates tp p s ti , p p ti , p o ti qu via fully-connected feed forward layers.

Original: Speciﬁcally, for a decoder f x ; x P t s , o , p u , an arbitrary layer t takes as input a set of queries t q t ´ 1 x ,i uand a set of learnable positional encodings t p x ,i u , where q t ´ 1 x ,i ; p x ,i P R d .
Revised: Speciﬁcally, for a decoder f x ; x P t s , o , p u , an arbitrary layer t takes as input a set of queries t q t ´ 1 x ,i u and a set of learnable positional encodings t p x ,i u , where q t ´ 1 x ,i ; p x ,i P R d .

Original: The output representationst q t x ,i u are then obtained via a combination of self-attention between the input queries, and encoderdecoder attention across the encoder output Z .
Revised: The output representations t q t x ,i u are then obtained via a combination of self-attention between the input queries, and encoderdecoder attention across the encoder output Z .

Original: These attention modules allow the decoder to jointlyreason across all queries, while simultaneously incorporating context from the input image.
Revised: These attention modules allow the decoder to jointly reason across all queries, while simultaneously incorporating context from the input image.

Original: At a given step t , naively using the inputs t q t ´ 1 x ,i u and t p x ,i u to generate G t forgoes leveraging the196 compositional property of relations.
Revised: At a given step t , naively using the inputs t q t ´ 1 x ,i u and t p x ,i u to generate G t forgoes leveraging the compositional property of relations.

Original: As described Equation 2, for any arbitrary step t , our proposedformulation entails two types of conditioning for better scene graph estimation.
Revised: As described Equation 2, for any arbitrary step t , our proposed formulation entails two types of conditioning for better scene graph estimation.

Original: The ﬁrst involvesconditioning decoders on the step t outputs, speciﬁcally the object decoder f t o on the subject decoderf t s , and the predicate decoder f t p on both f t s and f t o .
Revised: The ﬁrst involves conditioning decoders on the step t outputs, speciﬁcally the object decoder f t o on the subject decoder f t s , and the predicate decoder f t p on both f t s and f t o .

Original: The second requires all three decoder layersat step t to be conditioned on the outputs generated at step t ´ 1 .
Revised: The second requires all three decoder layers at step t to be conditioned on the outputs generated at step t ´ 1 .

Original: To effectively implement thisdesign, we modify the inputs to each decoder layer f t x .
Revised: To effectively implement this design, we modify the inputs to each decoder layer f t x .

Original: Speciﬁcally, the positional encodings t p x ,i uare modiﬁed to condition them on step t outputs, and the queries q t ´ 1 x ,i are updated to incorporateinformation from the previous step t ´ 1 .
Revised: Speciﬁcally, the positional encodings t p x ,i u are modiﬁed to condition them on step t outputs, and the queries q t ´ 1 x ,i are updated to incorporate information from the previous step t ´ 1 .

Original: Modifying the positional encoding and queries separatelyallows the model to easily disentangle and differentiate between the two conditioning types.
Revised: Modifying the positional encoding and queries separately allows the model to easily disentangle and differentiate between the two conditioning types.

Original: Conditional Positional Encodings.
Revised: Conditional Positional Encodings.

Original: At a particular step t , the conditional positional encodings forthe three decoders are obtained as,
Revised: At a particular step t , the conditional positional encodings for the three decoders are obtained as,

Original: [Equation]
Revised: [Equation]

Original: where MultiHead(Q, K, V) is the Multi-Head Attention module introduced in [46], FFN(.) is afully-connected feed forward network, and ‘ is the concatenation operation.
Revised: where MultiHead(Q, K, V) is the Multi-Head Attention module introduced in [48], FFN(.) is a fully-connected feed forward network, and ‘ is the concatenation operation.

Original: Additionally, r q t x ,i “q t x ,i ` p t x ,i ; x P t s , o , p u is the position-aware query.
Revised: Additionally, r q t x ,i “ q t x ,i ` p t x ,i ; x P t s , o , p u is the position-aware query.

Original: Conditional Queries.
Revised: Conditional Queries.

Original: Similarly, for a step t , conditional queries for the subject decoder are deﬁned, 
Revised: Similarly, for a step t , conditional queries for the subject decoder are deﬁned, t

Original: [Equation]
Revised: [Equation]

Original: t q t ´ 1 s ,i u ` FFN `[-MultiHead ` t r q t ´ 1 s ,i u , t r q t ´ 1 s ,i ‘ r q t ´ 
 o ,i ‘ r q t ´ 1 p ,i u , t q t ´ 1 s ,i ‘ q t ´ 1 o ,i ‘ q t ´ 1 p ,i u ˘˘ (4) p q t ´ 
-]
Revised: `[-MultiHead ` t r q t ´ 1 s ,i u , t r q t ´ 1 s ,i ‘ r q t ´ 
 o ,i ‘ r q t ´ 1 p ,i u , t q t ´ 1 s ,i ‘ q t ´ 1 o ,i ‘ q t ´ 1 p ,i u ˘˘ (4) p q t ´ 
-]

Original: o ,
Revised: [Equation]

Original: i and p q t ´ 1 p ,i are deﬁned identically.
Revised: p q t ´ 1 o ,i and p q t ´ 1 p ,i are deﬁned identically.

Original: For a decoder layer f t x ; x P t s , o , p u we use the conditionedpositional encodings t p p t x ,i u and queries t p q t ´ 1 x ,i u as input.
Revised: For a decoder layer f t x ; x P t s , o , p u we use the conditioned positional encodings t p p t [Equation] ,i [Equation]

Revised: and queries t

Revised: [Equation]

Original:  4.3 End-To-End Learning
Revised: as input. 4.3 End-To-End Learning

Original: Our proposed transformer based reﬁnement architecture can be trained in an end-to-end fashion.
Revised: Our proposed transformer based reﬁnement architecture can be trained in an end-to-end fashion.

Original: Toensure that a valid scene graph is generated at every level, we propose a novel joint loss that is appliedat each step t .
Revised: To ensure that a valid scene graph is generated at every level, we propose a novel joint loss that is applied at each step t .

Original: Therefore, the combined loss L can be expressed as L “∞ t L t “ ∞
Revised: Therefore, the combined loss L can be expressed as L “

Original: t
Revised: [Equation]

Original: L t s ` L t o ` L t p ,where L t x ; x P t s , o , p u represents the loss applied to the t -th layer of the decoder f x .
Revised: where L t x ; x P t s , o , p u represents the loss applied to the t -th layer of the decoder f x .

Original: Our approachgenerates a ﬁxed-size set of n triplet estimates t p r ti u “ tp p s ti , p p ti , p o ti qu at each step t , where n is largerthan the number of ground truth relations for a given image.
Revised: Our approach generates a ﬁxed-size set of n triplet estimates t [Equation] p o ti qu at each step t , where n is larger than the number of ground truth relations for a given image.

Original: Therefore, in order to effectively optimizethe proposed model, we obtain an optimal bipartite matching between the predicted and ground truthtriplets.
Revised: Therefore, in order to effectively optimize the proposed model, we obtain an optimal bipartite matching between the predicted and ground truth triplets.

Original: Note that, contrary to the matching algorithm in [2], our proposed matching is deﬁned overtriplets rather than individual entities.
Revised: Note that, contrary to the matching algorithm in [2], our proposed matching is deﬁned over triplets rather than individual entities.

Original: Additionally, instead of independently computing the loss overeach decoder layer as in [2], our loss computes a joint matching across all reﬁnement layers.
Revised: Additionally, instead of independently computing the loss over each decoder layer as in [2], our loss computes a joint matching across all reﬁnement layers.

Original: Let G “ t r i u “ tp s i , p i , o i qu denote the ground truth scene graph for an image I .
Revised: Let G “ t r i u “ tp s i , p i , o i qu denote the ground truth scene graph for an image I .

Original: Note that, as the225 number of ground truth relations is less than n , we convert G to a n -sized set by padding it with
Revised: Note that, as the number of ground truth relations is less than n , we convert G to a n -sized set by padding it with

Original: H226 (no relation).
Revised: H (no relation).

Original: The goal then is to ﬁnd a bipartite matching between the ground truth graph G and theset of all graph estimates t p
Revised: The goal then is to ﬁnd a bipartite matching between the ground truth graph G and the set of all graph estimates t p

Original: G t u that minimizes the joint matching cost .
Revised: G t u that minimizes the joint matching cost .

Original: Speciﬁcally, assuming  to bea valid permutation of n elements,
Revised: Speciﬁcally, assuming  to be a valid permutation of n elements,

Original: [Equation]
Revised: [Equation]

Original: where the pair-wise relation matching cost L rel is deﬁned as,
Revised: where the pair-wise relation matching cost L rel is deﬁned as,

Original: [Equation]
Revised: [Equation]

Original: where ‚ is the vector dot product, and L box is a combination of the L-1 and generalized IoU losses.
Revised: where ‚ is the vector dot product, and L box is a combination of the L-1 and generalized IoU losses.

Original: Please refer to Section 3 for clariﬁcation on the notations.
Revised: Please refer to Section 3 for clariﬁcation on the notations.

Original: The optimal permutation p  can then becomputed using the Hungarian algorithm.
Revised: The optimal permutation p  can then be computed using the Hungarian algorithm.

Original: The loss L t s is then deﬁned as,
Revised: The loss L t s is then deﬁned as,

Original: [Equation]
Revised: [Equation]

Original: L t o and L t p are deﬁned identically.
Revised: L t o and L t p are deﬁned identically.

Original: Note that as we use the same permutation p  for all reﬁnementlayers t , it induces strong implicit dependencies between the subject, object, and predicate decoders.
Revised: Note that as we use the same permutation p  for all reﬁnement layers t , it induces strong implicit dependencies between the subject, object, and predicate decoders.

Original: The potency of the aforementioned implicit conditioning is highlighted in the experiment section.
Revised: The potency of the aforementioned implicit conditioning is highlighted in the experiment section.

Original: 4.4 Loss Re-Weighting
Revised: 4.4 Loss Re-Weighting

Original: Due to the inherent long-tail nature of the scene graph generation task, using an unbiased loss oftenleads to the model prioritizing the most common ( a.k.a., head) predicate classes like has and on ,which have abundant training examples.
Revised: Due to the inherent long-tail nature of the scene graph generation task, using an unbiased loss often leads to the model prioritizing the most common ( a.k.a., head) predicate classes like has and on , which have abundant training examples.

Original: To afford our proposed model the ﬂexibility to achieve thetrade-off between head/tail classes, we integrate a loss-reweighing scheme into the model trainingprocedure.
Revised: To afford our proposed model the ﬂexibility to achieve the trade-off between head/tail classes, we integrate a loss-reweighing scheme into the model training procedure.

Original: Note that, contrary to existing methods that do this post-hoc via ﬁnetuning the ﬁnal layerof the trained network (see Section 2), we instead train the model with this weighting to allow theinternal feature representations to reﬂect the desired trade-off.
Revised: Note that, contrary to existing methods that do this post-hoc via ﬁnetuning the ﬁnal layer of the trained network (see Section 2), we instead train the model with this weighting to allow the internal feature representations to reﬂect the desired trade-off.

Original: Note, to the best of our knowledge, ourpaper is the ﬁrst to illustrate effectiveness of such a strategy for the task of scene graph generation.
Revised: Note, to the best of our knowledge, our paper is the ﬁrst to illustrate effectiveness of such a strategy for the task of scene graph generation.

Original: Fora particular predicate class c P r 1 ,  s , we deﬁne the class weight w c as max
Revised: For a particular predicate class c P r 1 ,  s , we deﬁne the class weight w c as max

Original: [Equation]
Revised: [Equation]

Original: , wheref c is the frequency of the predicate class c in the training set, and t ↵ ,  u are scaling parameters.
Revised: , where f c is the frequency of the predicate class c in the training set, and t ↵ ,  u are scaling parameters.

Original: Notethat this weighting scheme is similar to the data sampling strategy described in [18, 31].
Revised: Note that this weighting scheme is similar to the data sampling strategy described in [19, 32].

Original: However,instead of modifying the training set, we scale each class weight by the factor w c when computingthe predicate classiﬁcation loss L t p .
Revised: However, instead of modifying the training set, we scale each class weight by the factor w c when computing the predicate classiﬁcation loss L t p .

Original: Therefore, L t p can be deﬁned similarly to Equation 7,
Revised: Therefore, L t p can be deﬁned similarly to Equation 7,

Original: [Equation]
Revised: [Equation]

Original:  Experiments
Revised:  Experiments

Original: We demonstrate the effectiveness of our proposed approach on two datasets,
Revised: We demonstrate the effectiveness of our proposed approach on two datasets,

Original: Visual Genome [29].
Revised: Visual Genome [30].

Original: This is a benchmark for scene graph generation.
Revised: This is a benchmark for scene graph generation.

Original: We use the common processedsubset from [48], which contains 108 k images, with 150 object and 50 predicate categories.
Revised: We use the common processed subset from [50], which contains 108 k images, with 150 object and 50 predicate categories.

Original: Action Genome [24].
Revised: Action Genome [25].

Original: This dataset provides scene graph annotations over videos in the Charadesdataset [42] for the task of human-object interaction.
Revised: This dataset provides scene graph annotations over videos in the Charades dataset [44] for the task of human-object interaction.

Original: It contains 9 , 848 videos across 35 object(excluding class person ) and 25 relation categories.
Revised: It contains 9 , 848 videos across 35 object [Table] (excluding class person ) and 25 relation categories.

Original: As not all video frames are annotated, consistentwith prior work [15], we use the annotated frames provides by [24].
Revised: As not all video frames are annotated, consistent with prior work [16], we use the annotated frames provides by [25].

Original: Additionally, as in [15], weremove frames without any person or object annotations as they do not provide usable scene graphs.
Revised: Additionally, as in [16], we remove frames without any person or object annotations as they do not provide usable scene graphs.

Original: Implementation Details ( transformer-based approach ).
Revised: Implementation Details ( transformer-based approach ).

Original: We use ResNet-101 [21] as the backbonenetwork for image feature extraction.
Revised: We use ResNet-101 [22] as the backbone network for image feature extraction.

Original: Each of the subject, object, and predicate decoders have 6layers, with a feature size of 256 .
Revised: Each of the subject, object, and predicate decoders havelayers, with a feature size of 256 .

Original: The decoders use 300 queries.
Revised: The decoders use 300 queries.

Original: For training we use a batch size

Original: [Table]

Original: FP N

Original: F a s t e r

Original: RC NN [Table] of 12 and initial learning rate of 10 ´ 4 , which is gradually decayed.
Revised: For training we use a batch size of 12 and initial learning rate of 10 ´ 4 , which is gradually decayed.

Original: Note that although our modelpredicts individual relation triplets, a graph is obtained by applying a non-maximum suppression(NMS) strategy [14, 40] to group the predicted subjects and objects into entity instances.
Revised: Note that although our model predicts individual relation triplets, a graph is obtained by applying a non-maximum suppression (NMS) strategy [15, 41] to group the predicted subjects and objects into entity instances.

Original: Implementation Details ( MOTIF ).
Revised: Implementation Details ( MOTIF ).

Original: For the re-implementation of the MOTIF [53] baseline and thesubsequent augmentation of our proposed framework, we follow the same training procedure as [53].
Revised: For the re-implementation of the MOTIF [55] baseline and the subsequent augmentation of our proposed framework, we follow the same training procedure as [55].

Original: Speciﬁcally, we assume the Faster-RCNN detector [40] with the ResNeXt-101-FPN [47] backbone.
Revised: Speciﬁcally, we assume the Faster-RCNN detector [41] with the ResNeXt-101-FPN [49] backbone.

Original: The detector is ﬁrst pre-trained on the Visual Genome dataset [29].
Revised: The detector is ﬁrst pre-trained on the Visual Genome dataset [30].

Original: As is the case with two-stageapproaches, when learning the scene graph generator, the detector parameters are freezed.
Revised: As is the case with two-stage approaches, when learning the scene graph generator, the detector parameters are freezed.

Original: Thespeciﬁcs on how our approach is augmented to MOTIF is described in the supplementary (Sec. A).
Revised: The speciﬁcs on how our approach is augmented to MOTIF is described in the supplementary (Sec. A).

Original: Evaluation Metrics.
Revised: Evaluation Metrics.

Original: To measure performance, we report results using standard scene graph evaluation metrics, namely Recall (R@K) [48] and Mean Recall (mR@K) [5, 45].
Revised: To measure performance, we report results using standard scene graph evaluation metrics, namely Recall (R@K) [50] and Mean Recall (mR@K) [5, 47].

Original: While recall isclass agnostic, mean recall averages the recalls computed for each predicate category independently.
Revised: While recall is class agnostic, mean recall averages the recalls computed for each predicate category independently.

Original: Usually, higher R@K is indicative of better performance on dominant (head) classes, where as highermR@K suggests better tail class performance.
Revised: Usually, higher R@K is indicative of better performance on dominant (head) classes, where as higher mR@K suggests better tail class performance.

Original: Recent methods [10, 31] have argued for the useof mean recall as it reduces inﬂuence of dominant relationships such as on and has on the metric.
Revised: Recent methods [11, 32] have argued for the use of mean recall as it reduces inﬂuence of dominant relationships such as on and has on the metric.

Original: However, even for the same model architecture, one can trade-off R@K for mR@K by using long tailrecognition techniques described in Section 2.
Revised: However, even for the same model architecture, one can trade-off R@K for mR@K by using long tail recognition techniques described in Section 2.

Original: This trade-off is seldom analyzed or reported, makingit difﬁcult to compare performance across methods.
Revised: This trade-off is seldom analyzed or reported, making it difﬁcult to compare performance across methods.

Original: Therefore, inspired by the generalized zero-shotlearning harmonic average metric [9], we propose a new metric – harmonic recall (hR@K), deﬁnedas the harmonic mean of mR@K and R@K.
Revised: Therefore, inspired by the generalized zero-shot [Table] [Table] We vary

Revised: [Table]

Revised: [Table]

Revised: and on

Revised: [Table]

Revised: [Table]

Revised: learning harmonic average metric [10], we propose a new metric – harmonic recall (hR@K), deﬁned as the harmonic mean of mR@K and R@K.

Original: Such a metric encourages methods to have a healthybalance between the head and tail class performance.
Revised: Such a metric encourages methods to have a healthy balance between the head and tail class performance.

Original: Additionally, we report the performance of ourmodel for different values of the loss weighing parameters described in Section 4.4.
Revised: Additionally, we report the performance of our model for different values of the loss weighing parameters described in Section 4.4.

Original: Furthermore, wealso report the mR@100 on each long-tail category subset, namely head , body , and tail , as in [31].
Revised: Furthermore, we also report the mR@100 on each long-tail category subset, namely head , body , and tail , as in [32].

Original: 5.1 Ablation Study
Revised: 5.1 Ablation Study

Original: All the subsequent ablation studies are done on the Visual Genome dataset [29]. [Table] [Table]
Revised: All the subsequent ablation studies are done on the Visual Genome dataset [30].

Original: Model Components.
Revised: Model Components.

Original: We analyze the importance of each of our model components, namely theconditioning within a particular step t (CWS; Equation 3), the conditioning across steps (CAS;
Revised: We analyze the importance of each of our model components, namely the conditioning within a particular step t (CWS; Equation 3), the conditioning across steps (CAS;

Original: Equation 4), and the joint loss (JL; Section 4.3) in Table 2.
Revised: Equation 4) [Table] and the joint loss (JL; Section 4.3) in Table 2.

Original: For a fair comparison, all models aretrained using the same loss weighing parameters, ↵ “ 0 . 14 ,  “ 0 . 75 .
Revised: For a fair comparison, all models are trained using the same loss weighing parameters, ↵ “ 0 . 14 ,  “ 0 . 75 .

Original: It can be seen that ourproposed novel joint loss provides a signiﬁcant improvement in performance ( 3 ) when comparedto the ablated model that uses separate losses (as in [2]; 2 ).
Revised: It can be seen that our proposed novel joint loss provides a signiﬁcant improvement in performance ( 3 ) when compared to the ablated model that uses separate losses (as in [2]; 2 ).

Original: This is largely due to the joint lossbeing able to induce strong implicit dependencies between the three decoders.
Revised: This is largely due to the joint loss being able to induce strong implicit dependencies between the three decoders.

Revised: As a consequence, even without CAS and CWS, the proposed joint loss by itself enables reﬁnement.

Original: The two forms ofconditioning ( 4 - 5 ) provide a structured pathway to incorporating knowledge from previouslygenerated estimates or other triplet components in an explicit fashion, leading to improved scenegraphs.
Revised: The two forms of conditioning ( 4 - 5 ) provide a structured pathway to incorporating knowledge from previously generated estimates or other triplet components in an explicit fashion, leading to improved scene graphs.

Revised: CAS ( 5 ) builds upon the joint loss, integrating information that the aforementioned implicit conditioning is unable to capture.

Revised: CWS ( 4 ) is complementary to the reﬁnement process, and allows for more consistent graph generation within a step.

Original: However, these two types of conditionings capture complementary information, and usingthem together ( 1 )  improves on the performance of the aforementioned models that use only one.
Revised: As these two types of conditionings capture complementary information, using them together ( 1 ) further improves on performance.

Original: Reﬁnement Across Steps.
Revised: Reﬁnement Across Steps.

Original: To further analyze the effectiveness of our reﬁnement procedure, wecontrast the quality of scene graphs generated at each reﬁnement step for a particular trained model( ↵ “ 0 .
Revised: To further analyze the effectiveness of our reﬁnement procedure, we contrast the quality of scene graphs generated at each reﬁnement step for a particular trained model ( ↵ “ 0 .

Original: 14 ,  “ 0 . 75 ).
Revised: 14 ,  “ 0 . 75 ).

Original: We provide a detailed analysis in the supplementary (Table A1).
Revised: We provide a detailed analysis in the supplementary (Table A1).

Original: Visually,
Revised: Visually,

Original: Figure 3 highlights steady improvements in the graph quality over reﬁnement steps.
Revised: Figure 3 highlights steady improvements in the graph quality over reﬁnement steps.

Original: Owing toour structured conditioning, the model is able to improve on both predicate detection and entitylocalization over reﬁnement iterations.
Revised: Owing to our structured conditioning, the model is able to improve on both predicate detection and entity localization over reﬁnement iterations.

Original: Loss Re-Weighting.
Revised: Loss Re-Weighting.

Original: As mentioned in Section 4.4, we employ a simple loss re-weighting scheme,allowing for an easy trade-off between recall for mean recall.
Revised: As mentioned in Section 4.4, we employ a simple loss re-weighting scheme, allowing for an easy trade-off between recall for mean recall.

Original: Some results with different weightingsare reported in Table 1 ( 17 – 20 ).
Revised: Some results with different weightings are reported in Table 1 ( 17 – 20 ).

Original: Detailed analysis and discussion is in  the supplementary (Table A2).
Revised: To investigate this further, we train our proposed model with different values for the parameters ↵ and  , and report the performance in Table 4. We ﬁx  in the top half of the table (ﬁrst 4 rows), and ↵ in the bottom half (last 4 rows). It can be seen that increasing ↵ and  generally biases the model more towards the tail classes (higher mean recall), whereas lower values tend to favour the head classes (higher recall). As a consequence, our proposed approach is ﬂexibly able to function in a wide range of recall and mean recall values.

Revised: However, as is the case with every long-tail recognition approach, choosing values that are too high ( e.g . ↵ “ 0 . 21 ,  “ 0 . 75 ) lead to poor performance due to the model being extremely biased and therefore unable to learn accurate feature representations on the popular classes.

Revised: Model Parameters.

Revised: We additionally argue that the improvements obtained by our proposed reﬁnement procedure are not a direct consequence of having an increased number of parameters.

Revised: To corroborate this, we contrast our proposed model with alternatives with similar number of parameters but devoid of reﬁnement in Table 5.

Revised: 1 corresponds to our transformer-based approach with 6 decoder layers that uses the proposed joint loss and both forms of conditioning.

Revised: 2 is a similar model withdecoder layers but bereft of the aforementioned proposed implicit and explicit conditionings.

Revised: 3 , on the other hand, is a single decoder layer transformer model without reﬁnement, wherein the number of parameters are increased to be comparable to our proposed approach.

Revised: The inferior performance of 2 and 3 highlight that increasing model capacity either via making each layer larger or via adding more layers (increasing model depth) does not emulate reﬁnement.

Revised: Our proposed reﬁnement procedure, realized by the joint loss and explicit conditionings, allows for better learning, and the performance gains observed cannot be attributed to having additional parameters.

Original: 5.2 Comparison to Existing Methods
Revised: 5.2 Comparison to Existing Methods

Original: Visual Genome.
Revised: Visual Genome.

Original: We report the mean recall, recall, and harmonic recall values contrasting ourproposed model with existing scene graph methods, including the concurrent work in [30], in Table1.
Revised: We report the mean recall, recall, and harmonic recall values contrasting our proposed model with existing scene graph methods, including the concurrent work in [31], in Table 1.

Original: Contrary to existing approaches, depending on the loss weighting parameters used, our proposedapproach is able to easily operate on a wide spectrum of recall and mean recall values ( 17 - 20 ).
Revised: Contrary to existing approaches, depending on the loss weighting parameters used, our proposed approach is able to easily operate on a wide spectrum of recall and mean recall values ( 17 - 20 ).

Original: Moreconcretely, compared to the most competitive baseline in SGTR [30] ( 14 - 16 ) and other one stagemethods in AS-Net [4] ( 12 ) and HOTR [28] ( 13 ), our approach is able to achieve a considerably betterrecall on the head classes ( 17 ; 3 .
Revised: More concretely, compared to the most competitive baseline in SGTR [31] ( 14 - 16 ) and other one stage methods in AS-Net [4] ( 12 ) and HOTR [29] ( 13 ), our approach is able to achieve a considerably better recall on the head classes ( 17 ; 3 .

Original: mR@100 better on head classes compared to [30]).
Revised: mR@100 better on head classes compared to [31]).

Original: Additionally,when comparing a variant of our approach that has a similar R@100 value to SGTR ( 15 and 20 ),we do signiﬁcantly better on mean recall ( 3 . 8 higher mR@50) highlighting the proﬁciency of ourmethod to generalize to tail classes wherein the training data is limited.
Revised: Additionally, when comparing a variant of our approach that has a similar R@100 value to SGTR ( 15 and 20 ), we do signiﬁcantly better on mean recall ( 3 . 8 higher mR@50) highlighting the proﬁciency of our method to generalize to tail classes wherein the training data is limited.

Original: Furthermore, our proposedmodel in 20 achieves the best performance across all models (including two-stage methods that use asuperior backbone [47]) on hR@50/100, underlining the capability of our method to perform well onboth the head and tail classes simultaneously .
Revised: Furthermore, our proposed model in 20 achieves the best performance across all models (including two-stage methods that use a superior backbone [49]) on hR@50/100, underlining the capability of our method to perform well on both the head and tail classes simultaneously .

Original: SGTR [30] additionally reports numbers by utilizing a[Table] data sampling approach BGNN [31] to bias the model towards the tail classes ( 16 ).
Revised: SGTR [31] additionally reports numbers by utilizing a data sampling approach BGNN [32] to bias the model towards the tail classes ( 16 ).

Original: We highlight that,compared to 16 , our model in 20 performs similarly on mR@50 but much better R@50/100 ( 5 .[-
-]
Revised: We highlight that, compared to 16 , our model in 20 performs similarly on mR@50 but much better R@50/100 ( 5 .[-
-]

Original: R@50 higher).
Revised: R@50 higher).

Original: The poorer performance on mR@100 is largely due to SGTR [30] selecting top- 3subjects and objects for each predicate, leading to more relationship triplets being generated.
Revised: The poorer performance on mR@100 is largely due to SGTR [31] selecting topsubjects and objects for each predicate, leading to more relationship triplets being generated.

Original: Onthe contrary, our approach only uses 1 predicate per subject-object pair.
Revised: On the contrary, our approach only uses 1 predicate per subject-object pair.

Original: For a fairer comparison, weevaluate our model in 20 using a similar top- k strategy, wherein we select the top- 3 predicates foreach subject-object pair.
Revised: For a fairer comparison, we evaluate our model in 20 using a similar top- k strategy, wherein we select the top- 3 predicates for each subject-object pair.

Original: The resulting model ( 22 ) outperforms the closest baseline by a signiﬁcantmargin ( 3 . 3 higher mR@100, 10 . 6 higher R@100 compared to 16 ).
Revised: The resulting model ( 22 ) outperforms the closest baseline by a signiﬁcant margin ( 3 . 3 higher mR@100, 10 . 6 higher R@100 compared to 16 ).

Original: Additionally, we show that ourloss re-weighting strategy is complementary to existing data sampling approaches by ﬁne-tuning ourtrained model in 20 using BGNN [31] ( 21 ).
Revised: Additionally, we show that our loss re-weighting strategy is complementary to existing data sampling approaches by ﬁne-tuning our trained model in 20 using BGNN [32] ( 21 ).

Original: For further analysis, see supplementary Section C.
Revised: For further analysis, see supplementary Section C.2. [Table] t [Table]

Revised: r

Original:  Action Genome.
Revised: RC NN [Table] Action Genome.

Original: We additionally report recall, mean recall, and harmonic recall values on the Action
Revised: We additionally report recall, mean recall, and harmonic recall values on the Action

Original: Genome dataset [24] in Table 4.
Revised: Genome dataset [25] in Table 6.

Original: Similar to the observations on Visual Genome, we observe 3 .
Revised: Similar to the observations on Visual Genome, we observe 3 .

Original: 1 higher
Revised: 1 higher

Original: R@20 and 2 .
Revised: R@20 and 12 .

Original: 6 higher mR@20 ( 8 ) when compared to the closest baseline in [15] ( 7 ) despite usingan inferior backbone [47].
Revised: 6 higher mR@20 ( 8 ) when compared to the closest baseline in [16] ( 7 ) despite using an inferior backbone [49].

Original: Furthermore, by effectively biasing our model towards the tail classes, weare able to achieve 11 .
Revised: Furthermore, by effectively biasing our model towards the tail classes, we are able to achieve 29 .

Original: 4 better mR@20 while having better R@20/50 values ( 10 ) compared to 7 .
Revised: 6 better mR@20 while having better R@20/50 values ( 10 ) compared to 7 .

Original: We provide per class relation recall comparisons for both datasets in the supplementary (Section C).
Revised: We provide per class relation recall and object AP for both datasets in the supplementary (Section C).

Original: 5.3 Generality of Proposed Formulation
Revised: 5.3 Generality of Proposed Formulation

Original: Although our transformer model effectively generates better scene graphs, the proposed iterativereﬁnement formulation (Equation 2) can be applied to any existing method.
Revised: Although our transformer model effectively generates better scene graphs, the proposed iterative reﬁnement formulation (Equation 2) can be applied to any existing method.

Original: To demonstrate this, weaugment this framework to the simple two-stage MOTIF [53] architecture.
Revised: To demonstrate this, we augment this framework to the simple two-stage MOTIF [55] architecture.

Original: The details on how this isachieved is deferred to the supplementary (Section A).
Revised: The details on how this is achieved is deferred to the supplementary (Section A).

Original: We contrast the performance of the baselineand our augmented approach on the Visual Genome dataset [29] in Table 3.
Revised: We contrast the performance of the baseline and our augmented approach on the Visual Genome dataset [30] in Table 3.

Original: It can be seen that witheach reﬁnement step, the model is able to generate better scene graphs ( „ 6% higher on [-mR@
 andmR@50-] after 3 steps).
Revised: It can be seen that with each reﬁnement step, the model is able to generate better scene graphs ( „ 6% higher on [-mR@
 andmR@50-]mR@20 and mR@50 after 3 steps).

Original: Note that, owing to the two-stage nature of MOTIF [53], the image featuresused by our reﬁnement framework are ﬁxed and oblivious to the task of scene graph generation (asthe detector parameters are frozen).
Revised: Note that, owing to the two-stage nature of MOTIF [55], the image features used by our reﬁnement framework are ﬁxed and oblivious to the task of scene graph generation (as the detector parameters are frozen).

Original: Contrary to the proposed transformer model that can query theimage at each layer, the performance gains when using the augmented MOTIF variant are largelylimited due to the image features being the same across the reﬁnement steps.
Revised: Contrary to the proposed transformer model that can query the image at each layer, the performance gains when using the augmented MOTIF variant are largely limited due to the image features being the same across the reﬁnement steps.

Original:  Conclusion, Limitations, and Societal Impact
Revised:  Conclusion, Limitations, and Societal Impact

Original: In this work we propose a novel and general framework for iterative reﬁnement that overcomeslimitations arising from a ﬁxed factorization assumption in existing methods.
Revised: In this work we propose a novel and general framework for iterative reﬁnement that overcomes limitations arising from a ﬁxed factorization assumption in existing methods.

Original: We demonstrate itseffectiveness through a transformer based end-to-end trainable architecture.
Revised: We demonstrate its effectiveness through a transformer based end-to-end trainable architecture.

Original: Limitations.
Revised: Limitations.

Original: Limitations of the proposed approach include using a shared image encoder, whichimproves efﬁciency, but may limit diversity of representations available for different decoder layers;and limited ability to model small objects inherited from the DETR object detection design [30].
Revised: Limitations of the proposed approach include using a shared image encoder, which improves efﬁciency, but may limit diversity of representations available for different decoder layers; and limited ability to model small objects inherited from the DETR object detection design [31].

Original: Societal Impact.
Revised: Societal Impact.

Original: While our model does not have any direct serious societal implications, its impactcould be consequential if it is used in speciﬁc critical applications ( e.g ., autonomous driving).
Revised: While our model does not have any direct serious societal implications, its impact could be consequential if it is used in speciﬁc critical applications ( e.g ., autonomous driving).

Original: In suchcases both the overall performance and biases in object / predicate predictions would require carefulanalysis and further calibration.
Revised: In such cases both the overall performance and biases in object / predicate predictions would require careful analysis and further calibration.

Original: Our loss re-weighting strategy is a step in that direction.
Revised: Our loss re-weighting strategy is a step in that direction.

Original: Checklist

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope? [Yes](b) Did you describe the limitations of your work? [Yes] see Section 6.(c)
Revised: Acknowledgments and Disclosure of Funding

Original: Did you discuss any potential negative societal impacts of your work?

Original: [Yes] see Section6.

Original: d

Original: ) Have you read the ethics review guidelines and ensured that your paper conforms tothem?

Original: [Yes]2.

Original: If you are including theoretical results

Original: (a) Did you state the full set of assumptions of all theoretical results? [N/A] (b) Did you include complete proofs of all theoretical results? [N/A]
Revised: This work was funded, in part, by the Vector Institute for AI, Canada CIFAR AI Chair, NSERC CRC and an NSERC DG and Accelerator Grants.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] We include main
Revised: Hardware resources used in preparing this research were provided, in part, by the Province of Ontario, the Government of Canada through CIFAR, and companies sponsoring the Vector Institute 1 .

Original: code ﬁles for our model in the supplementary.

Original: All the data we use in our experiments

Original: is publicly available.

Original: We will also make the code public upon acceptance.

Original: b) Did you specify all the training details (e

Original: g., data splits, hyperparameters,

Original: how they

Original: were chosen)?

Original: [Yes] We use widely popular data splits for our experiments.

Original: We brieﬂy

Original: describe

Original: this in Section 5.

Original: The hyperparameters andadditional data details are also mentioned in the supplementary (Section B). c ) Did
Revised: Additional support was provided by JELF CFI grant and

Original: you

Original: report

Original: error

Original: bars

Original: (e

Original: g., with respect to the random

Original: seed

Original: after

Original: running

Original: experi

Original: -ments

Original: multiple times)?

Original: [No] As scene graph models are computationally expensive to train
Revised: Compute Canada under the RAC award.

Original: , none of the existing literature report error bars.

Original: However, we do report the perfor-

Original: mance of our method on different hyperparameter values to enable better understanding

Original: of the performance improvements.

Original: (d) Did you include the total amount of compute and the type of resources used (e

Original: g.,

Original: type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] This is described in the

Original: supplementary (Section B).

Original: If you are using existing assets (e

Original: g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] All the datasets

Original: we

Original: [Table]

Original: use are publicly available.

Original: We cite the creators of Visual Genome (Krishna et al. [Table] and Action Genome (Ji et al. [24]) throughout the paper.
Revised: Finally, we sincerely thank Bicheng Xu and Muchen Li for their valuable feedback on the paper draft.

Original: [Table]

Original: (b) Did you mention the license of the assets?

Original: [Yes] Visual Genome [29] is licensed under

Original: [Table]

Original: the Creative Commons Attribution 4.0 International License.

Original: Action Genome [24] is

Original: [Table]

Original: licensed under the MIT license.

Original: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A]

Original: [Table]

Original: We don’t introduce any new assets in the paper.

Original: [Table]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: [Table]

Original: using/curating?

Original: [N/A] Both Visual Genome [29] and Action Genome [24] are publicly

Original: [Table]

Original: available.

Original: [Table]

Original: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable

Original: [Table]

Original: information or offensive content?

Original: [N/A]

Original: [Table]

Original: If you used crowdsourcing or conducted research with human subjects...

Original: [Table]

Original: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: [Table]

Original: applicable?

Original: [N/A]

Original: [Table]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: [Table]

Original: Board (IRB) approvals, if applicable?

Original: [N/A]

Original: [Table]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: [Table]

Original: spent on participant compensation?

Original: [N/A]

Original: [Table]