Original: V IDEO S CENE G RAPH G ENERATION FROM S INGLE -F RAME W EAK S UPERVISION
Revised: V IDEO S CENE G RAPH G ENERATION FROM S INGLE -F RAME W EAK S UPERVISION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Video scene graph generation (VidSGG) aims to generate a sequence of graphstructure representations for the given video.
Revised: Video scene graph generation (VidSGG) aims to generate a sequence of graphstructure representations for the given video.

Original: However, all existing VidSGG methods are fully-supervised , i .
Revised: However, all existing VidSGG methods are fully-supervised , i .

Original: e ., they need dense and costly manual annotations.
Revised: e ., they need dense and costly manual annotations.

Original: Inthis paper, we propose the first weakly-supervised VidSGG task with only singleframe weak supervision: SF-VidSGG.
Revised: Inthis paper, we propose the first weakly-supervised VidSGG task with only singleframe weak supervision: SF-VidSGG.

Original: By “weakly-supervised”, we mean that
Revised: By “weakly-supervised”, we mean that

Original: SF-VidSGG relaxes the training supervision from two different levels: 1) It onlyprovides single-frame annotations instead of all-frame annotations.
Revised: SF-VidSGG relaxes the training supervision from two different levels: 1) It onlyprovides single-frame annotations instead of all-frame annotations.

Original: The singleframe ground-truth annotation is still a weak image SGG annotation, i .
Revised: The singleframe ground-truth annotation is still a weak image SGG annotation, i .

Original: e
Revised: e

Original: ., an unlocalized scene graph.
Revised: ., an unlocalized scene graph.

Original: To solve this new task, we also propose a novel Pseudo
Revised: To solve this new task, we also propose a novel Pseudo

Original: Label Assignment based method, dubbed as PLA .
Revised: Label Assignment based method, dubbed as PLA .

Original: PLA is a two-stage method,which generates pseudo visual relation annotations for the given video at the firststage, and then trains a fully-supervised VidSGG model with these pseudo labels.
Revised: PLA is a two-stage method,which generates pseudo visual relation annotations for the given video at the firststage, and then trains a fully-supervised VidSGG model with these pseudo labels.

Original: Specifically, PLA consists of three modules: an object PLA module, a predicate
Revised: Specifically, PLA consists of three modules: an object PLA module, a predicate

Original: PLA module, and a future predicate prediction (FPP) module.
Revised: PLA module, and a future predicate prediction (FPP) module.

Original: Firstly, in the object PLA, we localize all objects for every frame.
Revised: Firstly, in the object PLA, we localize all objects for every frame.

Original: Then, in the predicate PLA,we design two different teachers to assign pseudo predicate labels.
Revised: Then, in the predicate PLA,we design two different teachers to assign pseudo predicate labels.

Original: Lastly, in the
Revised: Lastly, in the

Original: FPP module, we fusion these two predicate pseudo labels by the regularity of relation transition in videos.
Revised: FPP module, we fusion these two predicate pseudo labels by the regularity of relation transition in videos.

Original: Extensive ablations and results on the benchmark Action
Revised: Extensive ablations and results on the benchmark Action

Original: Genome have demonstrated the effectiveness of our PLA.
Revised: Genome have demonstrated the effectiveness of our PLA.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Scene graph (Johnson et al., 2015) is a type of visually-aligned graph-structured representation thatsummarizes all the object instances ( e .
Revised: Scene graph (Johnson et al., 2015) is a type of visually-aligned graph-structured representation thatsummarizes all the object instances ( e .

Original: g ., “ person ”, “ chair ”) as nodes and their pairwise visualrelations (or predicates, e .
Revised: g ., “ person ”, “ chair ”) as nodes and their pairwise visualrelations (or predicates, e .

Original: g .
Revised: g .

Original: , “ sitting on ”) as directed edges.
Revised: , “ sitting on ”) as directed edges.

Original: As a bridge to connect the visionand language modalities, scene graphs have been widely used in many different downstream visuallanguage tasks, such as visual captioning (Yang et al., 2019; 2020), grounding (Jing et al.
Revised: As a bridge to connect the visionand language modalities, scene graphs have been widely used in many different downstream visuallanguage tasks, such as visual captioning (Yang et al., 2019; 2020), grounding (Jing et al.

Original: 2020),question answering (Hudson & Manning, 2019), and retrieval (Johnson et al., 2015).
Revised: 2020),question answering (Hudson & Manning, 2019), and retrieval (Johnson et al., 2015).

Original: Early Scene Graph Generation (SGG) work mainly focuses on generating scene graphs for the givenimage , dubbed as ImgSGG
Revised: Early Scene Graph Generation (SGG) work mainly focuses on generating scene graphs for the givenimage , dubbed as ImgSGG

Original: (Xu et al., 2017; Zellers et al., 2018; Chen et al., 2019).
Revised: (Xu et al., 2017; Zellers et al., 2018; Chen et al., 2019).

Original: However,due to its static nature, ImgSGG fails to represent numerous dynamic visual relations that takeplace over a period of time, such as “ walking with ” and “ running away ” (vs. static relation“ standing ”).
Revised: However,due to its static nature, ImgSGG fails to represent numerous dynamic visual relations that takeplace over a period of time, such as “ walking with ” and “ running away ” (vs. static relation“ standing ”).

Original: Meanwhile, it is hard or impossible to identify these dynamic visual relations withonly a single frame, because these visual relations can only be well classified by considering thetemporal context.
Revised: Meanwhile, it is hard or impossible to identify these dynamic visual relations withonly a single frame, because these visual relations can only be well classified by considering thetemporal context.

Original: Therefore, another more meaningful but challenging video-based SGG task wasproposed: VidSGG (Shang et al., 2017; 2019).
Revised: Therefore, another more meaningful but challenging video-based SGG task wasproposed: VidSGG (Shang et al., 2017; 2019).

Original: Since the complex and dense annotations of a scene graph ( cf. , Figure 1(a)), fully-supervised SGGmethods always require lots of manual annotations, and the case is even worse for video data.
Revised: Since the complex and dense annotations of a scene graph ( cf. , Figure 1(a)), fully-supervised SGGmethods always require lots of manual annotations, and the case is even worse for video data.

Original: Meanwhile, several prior SGG works (Li et al., 2022a) have found that even carefully manually-annotatedlabels are still quite noisy, i .
Revised: Meanwhile, several prior SGG works (Li et al., 2022a) have found that even carefully manually-annotatedlabels are still quite noisy, i .

Original: e ., the annotated positive labels may not be the best matched, and numerous negative labels are just missing annotated.
Revised: e ., the annotated positive labels may not be the best matched, and numerous negative labels are just missing annotated.

Original: Thus, a surge of recent ImgSGG work (Zareianet al., 2020; Zhong et al., 2021; Shi et al., 2021; Li et al., 2022c) start to generate scene graphs forimages with only weak supervision.
Revised: Thus, a surge of recent ImgSGG work (Zareianet al., 2020; Zhong et al., 2021; Shi et al., 2021; Li et al., 2022c) start to generate scene graphs forimages with only weak supervision.

Original: By “weak supervision”, we mean that the annotations for modeltraining are not complete localized scene graphs.
Revised: By “weak supervision”, we mean that the annotations for modeltraining are not complete localized scene graphs.

Original: For example, a typical type of weak supervision is unlocalized scene graphs .
Revised: For example, a typical type of weak supervision is unlocalized scene graphs .

Original: As illustrated in Figure 1(b), an unlocalized scene graph only containsimage-level object and relation categories without corresponding object bounding boxes (bboxes).a)
Revised: As illustrated in Figure 1(b), an unlocalized scene graph only containsimage-level object and relation categories without corresponding object bounding boxes (bboxes).

Original: [Figure]
Revised: [Figure]

Original: person sit on look at on b) Unlocalized scene graph phone table chair person sit on look at on phone table chair c) SF-VidSGG weakly annotated frame unannotated frames unannotated frames time person hold sit on on bed floor towel
Revised: floor towel

Original: Although recent weakly-supervised ImgSGGhas achieved good performance and receivedunprecedented attention, to the best our knowledge, there is no existing work about generating video scene graphs from weak supervision.
Revised: Although recent weakly-supervised ImgSGGhas achieved good performance and receivedunprecedented attention, to the best our knowledge, there is no existing work about generating video scene graphs from weak supervision.

Original: To put forward the research on this criticaltopic, we propose the first weakly-supervised
Revised: To put forward the research on this criticaltopic, we propose the first weakly-supervised

Original: VidSGG task with single-frame weak supervision, called SF-VidSGG .
Revised: VidSGG task with single-frame weak supervision, called SF-VidSGG .

Original: Specifically, given aninput video, SF-VidSGG aims to generate a localized scene graph for each frame in the video,but the only supervision for training is an unlocalized scene graph for the middle frame ofeach training video .
Revised: Specifically, given aninput video, SF-VidSGG aims to generate a localized scene graph for each frame in the video,but the only supervision for training is an unlocalized scene graph for the middle frame ofeach training video .

Original: As the example shown in
Revised: As the example shown in

Original: Figure 1(c), the supervision is an unlocalizedscene graph for the third frame.
Revised: Figure 1(c), the supervision is an unlocalizedscene graph for the third frame.

Original: Obviously, SFVidSGG task tries to relieve the intensive annotation issues from two levels: 1) Video-level:
Revised: Obviously, SFVidSGG task tries to relieve the intensive annotation issues from two levels: 1) Video-level:

Original: For each video, we only need single-frame annotations instead of all-frame annotations as thefully-supervised setting ( i .
Revised: For each video, we only need single-frame annotations instead of all-frame annotations as the fullysupervised setting ( i .

Original: e ., reduce the number of annotated frames).
Revised: e ., reduce the number of annotated frames).

Original: 2) Frame-level: The singleframe annotation is an unlocalized scene graph ( i .
Revised: 2) Frame-level: The single-frameannotation is an unlocalized scene graph ( i .

Original: e ., avoid annotating object locations).
Revised: e ., avoid annotating object locations).

Original: A straightforward solution for SF-VidSGG is: Using all the weakly annotated frames to train aweakly-supervised ImgSGG model first, and then detecting scene graphs on each frame with the
Revised: A straightforward solution for SF-VidSGG is: Using all the weakly annotated frames to train aweakly-supervised ImgSGG model first, and then detecting scene graphs on each frame with the

Original: ImgSGG model.
Revised: ImgSGG model.

Original: Apparently, this naive ImgSGG-based method has overlooked the temporal context in the video data.
Revised: Apparently, this naive ImgSGG-based method has overlooked the temporal context in the video data.

Original: To this end, we propose a novel Pseudo Label Assignment strategy PLA ,which can serve as the first baseline for SF-VidSGG.
Revised: To this end, we propose a novel Pseudo Label Assignment strategy PLA ,which can serve as the first baseline for SF-VidSGG.

Original: Since PLA is agnostic to different VidSGGarchitectures, it can be easily incorporated into any advanced VidSGG model.
Revised: Since PLA is agnostic to different VidSGGarchitectures, it can be easily incorporated into any advanced VidSGG model.

Original: Specifically, PLAdecouples the problem into two steps: The first step is to assign a pseudo localized scene graphfor every frame in the video, and the second step is to train a fully-supervised VidSGG model bythe pseudo localized scene graphs.
Revised: Specifically, PLAdecouples the problem into two steps: The first step is to assign a pseudo localized scene graphfor every frame in the video, and the second step is to train a fully-supervised VidSGG model bythe pseudo localized scene graphs.

Original: PLA consists of three modules: object pseudo label assignmentmodule (Obj-PLA), relation pseudo label assignment module (Rel-PLA), and future predicate prediction module (FPP).
Revised: PLA consists of three modules: object pseudo label assignmentmodule (Obj-PLA), relation pseudo label assignment module (Rel-PLA), and future predicate prediction module (FPP).

Original: In the Obj-PLA module, we detect object region proposals for all the frames.
Revised: In the Obj-PLA module, we detect object region proposals for all the frames.

Original: In the Rel-PLA module, we propose two relation pseudo label assignment teachers and they generate two different pseudo labels for each frame.
Revised: In the Rel-PLA module, we propose two relation pseudo label assignment teachers and they generate two different pseudo labels for each frame.

Original: In the FPP module, we determine adapted weightsto fuse these labels from two different teachers.
Revised: In the FPP module, we determine adapted weightsto fuse these labels from two different teachers.

Original: To effectively obtain optimal adapted weights forfusing different teacher knowledge, the FPP module exploits the temporal context based on the relation transition in videos.
Revised: To effectively obtain optimal adapted weights forfusing different teacher knowledge, the FPP module exploits the temporal context based on the relation transition in videos.

Original: The relation transition means how the predicates change between the samesubject-object pairs in different frames.
Revised: The relation transition means how the predicates change between the samesubject-object pairs in different frames.

Original: In summary, we make three main technique contributions in this paper:
Revised: In summary, we make three main technique contributions in this paper:

Original: We propose the first weakly-supervised VidSGG task.
Revised: We propose the first weakly-supervised VidSGG task.

Original: Compared to its fully-supervised coun- terpart, we try to mitigate the intensive annotations from both video-level and frame-level.2.
Revised: Compared to its fully-supervised counterpart, we try to mitigate the intensive annotations from both video-level and frame-level.

Original: We propose a novel method PLA for SF-VidSGG.
Revised: We propose a novel method PLA for SF-VidSGG.

Original: It utilizes two teachers to assign pseudo labelfor unlabeled data and refines the pseudo labels from both teachers by knowledge distillation.3.
Revised: It utilizes two teachers to assign pseudo labelfor unlabeled data and refines the pseudo labels from both teachers by knowledge distillation.3.

Original: We propose a future predicate prediction module that leverages temporal dependencies in video.
Revised: We propose a future predicate prediction module that leverages temporal dependencies in video.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Image Scene Graph Generation (ImgSGG).
Revised: Image Scene Graph Generation (ImgSGG).

Original: ImgSGG aims to generate semantic graph structures— scene graphs — as the representation of images.
Revised: ImgSGG aims to generate semantic graph structures— scene graphs — as the representation of images.

Original: In each scene graph, every node represents anobject instance and every edge represents a visual relation between two objects.
Revised: In each scene graph, every node represents anobject instance and every edge represents a visual relation between two objects.

Original: Early ImgSGGmethods always directly predict all pairwise visual relations (Lu et al., 2016; Zhang et al., 2017).
Revised: Early ImgSGGmethods always directly predict all pairwise visual relations (Lu et al., 2016; Zhang et al., 2017).

Original: Later, ImgSGG methods mainly focus on designing different context encoding architectures, such as message passing (Xu et al., 2017; Li et al., 2017), recurrent network (Zellers et al., 2018), treestructure model (Tang et al., 2019), agent communication (Chen et al., 2019), and Transformer-basedmodels (Lu et al., 2021).
Revised: Later, ImgSGG methods mainly focus on designing different context encoding architectures, suchas message passing (Xu et al., 2017; Li et al., 2017), recurrent network (Zellers et al., 2018), treestructure model (Tang et al., 2019), agent communication (Chen et al., 2019), and Transformer-based models (Lu et al., 2021).

Original: Recently, a surge of ImgSGG work starts to explore the long-tailed issuein predicate or triplet classification.
Revised: Recently, a surge of ImgSGG work starts to explore the long-tailed issuein predicate or triplet classification.

Original: Existing methods for debiased SGG can be roughly devided tofour types: class-aware re-sampling (Li et al., 2021;
Revised: Existing methods for debiased SGG can be roughly devided tofour types: class-aware re-sampling (Li et al., 2021;

Original: Desai et al., 2021), loss re-weighting (Lin et al.,2017; Yan et al., 2020; Knyazev et al., 2020), biased-model-based (Tang et al., 2020; Chiou et al.
Revised: Desai et al., 2021), loss re-weighting (Lin et al.,2017; Yan et al., 2020; Knyazev et al., 2020), biased-model-based (Tang et al., 2020; Chiou et al.

Original: Yu et al., 2020) and noisy label correction (Li et al., 2022a;b).
Revised: Yu et al., 2020) and noisy label correction (Li et al., 2022a;b).

Original: Video Scene Graph Generation (VidSGG).
Revised: Video Scene Graph Generation (VidSGG).

Original: Beyond static images, VidSGG aims to detect dynamic visual relations in the videos.
Revised: Beyond static images, VidSGG aims to detect dynamic visual relations in the videos.

Original: Compared to ImgSGG, VidSGG is more challenging becausethey need to consider the spatio-temporal context in adjacent frames.
Revised: Compared to ImgSGG, VidSGG is more challenging becausethey need to consider the spatio-temporal context in adjacent frames.

Original: Based on the formats ofrelation triplet annotations, existing VidSGG work can be categorized into two groups: 1) Trackletbased :
Revised: Based on the formats ofrelation triplet annotations, existing VidSGG work can be categorized into two groups: 1) Trackletbased :

Original: Each graph node is an object tracklet in a video clip (Shang et al., 2017; 2019) or the wholevideo (Liu et al., 2020; Gao et al., 2021; 2022).
Revised: Each graph node is an object tracklet in a video clip (Shang et al., 2017; 2019) or the wholevideo (Liu et al., 2020; Gao et al., 2021; 2022).

Original: 2) Frame-based : Each graph node is an object bboxas in ImgSGG, but these visual relation triplets are dynamic in the whole video sequence (Ji et al.
Revised: 2) Frame-based : Each graph node is an object bboxas in ImgSGG, but these visual relation triplets are dynamic in the whole video sequence (Ji et al.

Original: Feng et al., 2021;
Revised: Feng et al., 2021;

Original: Cong et al., 2021; Li et al., 2022d).
Revised: Cong et al., 2021; Li et al., 2022d).

Original: In this paper, we follow the frame-based
Revised: In this paper, we follow the frame-based

Original: VidSGG setting, and propose the first weakly-supervised VidSGG setting and model.
Revised: VidSGG setting, and propose the first weakly-supervised VidSGG setting and model.

Original: Weakly-Supervised ImgSGG.
Revised: Weakly-Supervised ImgSGG.

Original: A ground-truth visual relation triplet annotation consists of two object locations, their object categories, and pairwise visual relations.
Revised: A ground-truth visual relation triplet annotation consists of two object locations, their object categories, and pairwise visual relations.

Original: Thus, it is extremely laborintensive to obtain large-scale fully-annotated scene graph benchmarks.
Revised: Thus, it is extremely laborintensive to obtain large-scale fully-annotated scene graph benchmarks.

Original: To reduce the labelingcosts, several ImgSGG work proposed to generate scene graphs under weak supervision.
Revised: To reduce the labelingcosts, several ImgSGG work proposed to generate scene graphs under weak supervision.

Original: Currently,there are two main types of weak supervision for ImgSGG: 1) Unlocalized Scene Graphs (Zareianet al., 2020;
Revised: Currently,there are two main types of weak supervision for ImgSGG: 1) Unlocalized Scene Graphs (Zareianet al., 2020;

Original: Shi et al., 2021; Li et al., 2022c):
Revised: Shi et al., 2021; Li et al., 2022c):

Original: It only consists of image-level labels that describethe object and visual relation categories without object locations.
Revised: It only consists of image-level labels that describethe object and visual relation categories without object locations.

Original: Aligned Captions (Shi et al.
Revised: Aligned Captions (Shi et al.

Original: Ye & Kovashka, 2021):
Revised: Ye & Kovashka, 2021):

Original: It provides an entailment caption for the given image.
Revised: It provides an entailment caption for the given image.

Original: In this paper,we only utilize an unlocalized scene graph of a middle frame as the supervision for VidSGG.
Revised: In this paper,we only utilize an unlocalized scene graph of a middle frame as the supervision for VidSGG.

Original: Knowledge Distillation (KD) .
Revised: Knowledge Distillation (KD) .

Original: KD is first used for model compression (Hinton et al., 2015).
Revised: KD is first used for model compression (Hinton et al., 2015).

Original: Then,
Revised: Then,

Original: KD becomes a prevalent method to transfer knowledge from a (larger) teacher model to a (smaller)student model.
Revised: KD becomes a prevalent method to transfer knowledge from a (larger) teacher model to a (smaller)student model.

Original: Benefiting from the soft targets generated by teacher models, the student model caneven outperform their teachers through appropriate training strategies (Zhang et al., 2018; Furlanelloet al., 2018).
Revised: Benefiting from the soft targets generated by teacher models, the student model caneven outperform their teachers through appropriate training strategies (Zhang et al., 2018; Furlanelloet al., 2018).

Original: However, these traditional KD methods are single-teacher.
Revised: However, these traditional KD methods are single-teacher.

Original: To reduce the limitation ofthe data diversity and single-teacher knowledge, recent work tries to distill knowledge from multipleteachers.
Revised: To reduce the limitation ofthe data diversity and single-teacher knowledge, recent work tries to distill knowledge from multipleteachers.

Original: The idea of multi-teacher KD has been widely applied to numerous vision tasks, i .
Revised: The idea of multi-teacher KD has been widely applied to numerous vision tasks, i .

Original: e ., objectdetection (Chen et al., 2017; Wang et al., 2019), visual question answering (Niu & Zhang, 2021;
Revised: e ., objectdetection (Chen et al., 2017; Wang et al., 2019), visual question answering (Niu & Zhang, 2021;

Original: Chen et al., 2022) and ImgSGG (Li et al., 2022c).
Revised: Chen et al., 2022) and ImgSGG (Li et al., 2022c).

Original: In this paper, we propose two relation pseudolabel assignment teachers and use multi-teacher KD to fuse knowledge of multi-teachers.
Revised: In this paper, we propose two relation pseudolabel assignment teachers and use multi-teacher KD to fuse knowledge of multi-teachers.

Original:  P ROPOSED A PPROACH
Revised:  P ROPOSED A PPROACH

Original: Task Definition.
Revised: Task Definition.

Original: In this paper, we propose a new VidSGG task: weakly-supervised VidSGG withsingle-frame weak supervision (SF-VidSGG).
Revised: In this paper, we propose a new VidSGG task: weakly-supervised VidSGG withsingle-frame weak supervision (SF-VidSGG).

Original: Specifically, given an input video V = { I 1 , ..., I T }with T frames, SF-VidSGG aims to generate a scene graph G t for each frame I t , and then stack themalong the time axis to obtain the final video scene graph G = { G 1 , ..., G T } , where G t =
Revised: Specifically, given an input video V = { I 1 , ..., I T }with T frames, SF-VidSGG aims to generate a scene graph G t for each frame I t , and then stack themalong the time axis to obtain the final video scene graph G = { G 1 , ..., G T } , where G t =

Original: ( N t , E t ) ,
Revised: ( N t , E t ) ,

Original: N t and E t denote the set of graph nodes and edges in G t , respectively.
Revised: N t and E t denote the set of graph nodes and edges in G t , respectively.

Original: In the training stage, we onlyuse the weakly-supervision as mentioned in Sec.
Revised: In the training stage, we onlyuse the weakly-supervision as mentioned in Sec.

Original: 1, i .
Revised: 1, i .

Original: e ., the only supervision for SF-VidSGG is anunlocalized scene graph for a middle frame of the video V , denoted as (cid:101)
Revised: e ., the only supervision for SF-VidSGG is anunlocalized scene graph for a middle frame of the video V , denoted as (cid:101)

Original: G m =
Revised: G m =

Original: ( (cid:101) N m , (cid:101) E m ) , and I mis the only frame in the video with annotation.
Revised: ( (cid:101) N m , (cid:101) E m ) , and I mis the only frame in the video with annotation.

Original: 3. M ODEL O VERVIEW
Revised: 3. M ODEL O VERVIEW

Original: To tackle the SF-VidSGG task, we propose a novel framework: Pseudo Label Assignment ( PLA ).
Revised: To tackle the SF-VidSGG task, we propose a novel framework: Pseudo Label Assignment ( PLA ).

Original: The whole pipeline of PLA is illustrated in Figure 2.
Revised: The whole pipeline of PLA is illustrated in Figure 2.

Original: Specifically, it consists of three components:object pseudo label assignment (Obj-PLA) module, relation pseudo label assignment (Rel-PLA)module, and future predicate prediction (FPP) module.
Revised: Specifically, it consists of three components:object pseudo label assignment (Obj-PLA) module, relation pseudo label assignment (Rel-PLA)module, and future predicate prediction (FPP) module.

Original: In the Obj-PLA, we detect object regionproposals for all the frames.
Revised: In the Obj-PLA, we detect object regionproposals for all the frames.

Original: For the frame with weak annotation, we further complete its unlocalized scene graph (cid:101) G to obtain a pseudo localized scene graph G pseudo .
Revised: For the frame with weak annotation, we further complete its unlocalized scene graph (cid:101) G to obtain a pseudo localized scene graph G pseudo .

Original: In the Rel-PLA, we proposetwo relation pseudo label assignment teachers: model-based and model-free.
Revised: In the Rel-PLA, we proposetwo relation pseudo label assignment teachers: model-based and model-free.

Original: The model-based
Revised: The model-basedteacher assigns “soft” predicate pseudo labels P soft by an ImgSGG model, while the model-freeteacher assigns “hard” predicate pseudo labels P hard by some heuristic rules. After P soft and P hard

Original: [Figure]
Revised: [Figure]

Original: 3. O BJECT P ESUDO L ABEL A SSIGNMENT (O BJ -PLA)
Revised: O BJECT P ESUDO L ABEL A SSIGNMENT (O BJ -PLA)

Original: In this module [Figure] we aim to generate pseudo object category annotations for all frames.
Revised: In this module, we aim to generate pseudo object category annotations for all frames.

Original: For eachframe I t in input video, we first use an off-the-shelf detector to generate a set of proposal O t .
Revised: For eachframe I t in input video, we first use an off-the-shelf detector to generate a set of proposal O t .

Original: Eachproposal o ti ∈
Revised: Eachproposal o ti ∈

Original: O t has a corresponding bounding box position b ti and an initial object category ˆ c ti ,i .
Revised: O t has a corresponding bounding box position b ti and an initial object category ˆ c ti ,i .

Original: e ., o ti =
Revised: e ., o ti =

Original: ( b ti , ˆ c ti ) .
Revised: ( b ti , ˆ c ti ) .

Original: Since the differences in object category ontology between the pretrained detectorand VidSGG benchmarks, we then map the detected object categories to the object categories in the
Revised: Since the differences in object category ontology between the pretrained detectorand VidSGG benchmarks, we then map the detected object categories to the object categories in the

Original: VidSGG dataset if there is overlapping between their synonyms in WordNet (Miller, 1995), e .
Revised: VidSGG dataset if there is overlapping between their synonyms in WordNet (Miller, 1995), e .

Original: g .,“ woman ” is mapped to “ person ”, “ pizza box ” is mapped to “ box ”.
Revised: g .,“ woman ” is mapped to “ person ”, “ pizza box ” is mapped to “ box ”.

Original: For the annotated frame I m , Obj-PLA has an additional step: matching these detected boundingboxes with entities in the unlocalized scene graph to obtain a pseudo localized scene graph.
Revised: For the annotated frame I m , Obj-PLA has an additional step: matching these detected boundingboxes with entities in the unlocalized scene graph to obtain a pseudo localized scene graph.

Original: Herewe use a category-based strategy: a bounding box and a entity will be matched if they have the samecategory.
Revised: Herewe use a category-based strategy: a bounding box and a entity will be matched if they have the samecategory.

Original: This strategy has a weakness that it cannot distinguish bounding boxes with the sameobject category.
Revised: This strategy has a weakness that it cannot distinguish bounding boxes with the sameobject category.

Original: When the detector finds multiple bounding boxes with the same object category,we just randomly choose one of them to match.
Revised: When the detector finds multiple bounding boxes with the same object category,we just randomly choose one of them to match.

Original: To solve this problem, Li et al. (2022c) utilizespretrained VL models to calculate relevant scores between entities with bounding boxes.
Revised: To solve this problem, Li et al. (2022c) utilizespretrained VL models to calculate relevant scores between entities with bounding boxes.

Original: The finalmatches are decided by the object category labels and the relevant scores.
Revised: The finalmatches are decided by the object category labels and the relevant scores.

Original: We leave these advancedmethods for the future work.
Revised: We leave these advancedmethods for the future work.

Original: 3. R ELATION P ESUDO L ABEL A SSIGNMENT (R EL -PLA)
Revised: 3. R ELATION P ESUDO L ABEL A SSIGNMENT (R EL -PLA)

Original:  After Obj-PLA, we obtain a pseudo localize scene graph for the annotated Frame and object bboxesfor all other unannotated frames.
Revised: [Figure] After Obj-PLA, we obtain a pseudo localize scene graphfor the annotated Frame and object bboxes for all otherunannotated frames.

Original: In Rel-PLA, we design two strategies (teachers) to generate pseudopredicate labels for the unannotated frames.
Revised: In Rel-PLA, we design two strategies (teachers) to generate pseudo predicate labels for theunannotated frames.

Original: Model-based Teacher.
Revised: Model-based Teacher.

Original: Model-based Teacher assigns “soft” predicate pseudo labels by an ImgSGGmodel.
Revised: Model-based Teacher assigns“soft” predicate pseudo labels by an ImgSGG model.

Original: It first obtains pseudo localized scene graphs G pseudo from the annotated frame set by ObjPLA, then train a ImgSGG model by these scene graphs.
Revised: Itfirst obtains pseudo localized scene graphs G pseudo fromthe annotated frame set by Obj-PLA, then train a ImgSGGmodel by these scene graphs.

Original: After training, each un-annotated frame I t is fed to the network to obtain G t soft =
Revised: After training, each unannotated frame I t is fed to the network to obtain G t soft =

Original: M img ( I t ) , the scene graph for I t .
Revised: M img ( I t ) , the scene graph for I t .

Original: Here we representthe predicate in G t soft by the category distribution of predicate, besides one-hot labels.
Revised: Here we represent thepredicate in G t soft by the category distribution of predicate, besides one-hot labels.

Original: Inspired byknowledge distillation believe that the soft pseudo label can preserve more information than thehard pseudo label Hinton al [Figure]
Revised: Inspired by knowledge dis- tillation, we believe that the soft pseudo label can preserve more information than the hard pseudo label (Hinton et al., 2015).

Original: Model-free Teacher.
Revised: Model-free Teacher.

Original: Model-free teacher assigns “hard” pseudo prediacte labels by someheuristic rules.
Revised: Model-free teacher assigns “hard” pseudo prediacte labels by some heuristicrules.

Original: The first step is object sequence matching .
Revised: The first step is object sequence matching .

Original: We adopt the object category and the IoU to match the objects detectedin the different frames.
Revised: We adopt the object category and the IoU tomatch the objects detected in the different frames.

Original: Two object o ti and o t +1 jin adjacent frames can be matched if they havesame object categories and IoU ( b ti , b t +1 j ) >
Revised: Two object o ti and o t +1 j in adjacent frames can bematched if they have same object categories and IoU ( b ti , b t +1 j ) >

Original: η ,where η is the IoU matching threshold to filter objects.
Revised: η , where η is the IoU matchingthreshold to filter objects.

Original: We frame-by-framely match objectsfrom the middle frame to both ends.
Revised: We frame-by-framely match objects from the middle frame to both ends.

Original: As shownin Figure 3, “ book ” and “ towel ” in I m +2 arenot matched with any object in I m
Revised: As shown in Figure 3, “ book ” and “ towel ” in I m +2 are not matched with any object in I m

Original: [-+
 becausethere-] is no object detected in I m +1 with category “ book ” and IoU of bounding boxes withcategory “ towel ” in I m
Revised: [-+
 becausethere-]+1because there is no object detected in I m +1 with category “ book ” and IoU of bounding boxes withcategory “ towel ” in I m

Original: +

Revised: +

Original: and I m +2 is lowerthan η .
Revised: and I m +2 is lower than η .

Original: The second step is relation propagation .
Revised: The second step is relation propagation .

Original: We simply propagate the predicate of asubject-object pair in the annotated frame to the same subject-object pair in the un-annotated frames.
Revised: Wesimply propagate the predicate of a subject-object pair in the annotated frame to the same subjectobject pair in the un-annotated frames.

Original: 3. F UTURE P REDICATE P REDICTION (FPP) M ODULE
Revised: 3. F UTURE P REDICATE P REDICTION (FPP) M ODULE

Original: Multi-Teacher Fusion Strategy.
Revised: Multi-Teacher Fusion Strategy.

Original: After obtaining P soft and P hard by two teachers, the simplest fusingstrategy is treating each teacher equally and averaging the two pseudo labels.
Revised: After obtaining P soft and P hard by two teachers, the simplest fusingstrategy is treating each teacher equally and averaging the two pseudo labels.

Original: However, this strategyoverlooks the noise in the model-based teacher.
Revised: However, this strategyoverlooks the noise in the model-based teacher.

Original: The model-based teacher sometimes assign wrongpseudo labels due to the incorrect predictions in model M img .
Revised: The model-based teacher sometimes assign wrongpseudo labels due to the incorrect predictions in model M img .

Original: On the other hand, pseudo labelsassigned by the model-free teacher can be seen as the default predictions because these labels havealready appeared in the video.
Revised: On the other hand, pseudo labelsassigned by the model-free teacher can be seen as the default predictions because these labels havealready appeared in the video.

Original: To some extent, the model-free teacher can be seen as a complementwhen the model-based teacher assign a wrong pseudo label.
Revised: To some extent, the model-free teacher can be seen as a complementwhen the model-based teacher assign a wrong pseudo label.

Original: Therefore, a better strategy is assigningadaptive weights for P soft and P hard and then calculating the final pseudo labels by the weighted sum:
Revised: Therefore, a better strategy is assigningadaptive weights for P soft and P hard and then calculating the final pseudo labels by the weighted sum:

Original: [Equation]
Revised: [Equation]

Original: where w i soft and w i hard denotes the weights of the soft and hard predicate labels for video i , respectively.
Revised: where w i soft and w i hard denotes the weights of the soft and hard predicate labels for video i , respectively.

Original: In the FPP module, we calculate w i soft and w i hard by regularity of relation transition.
Revised: In the FPP module, we calculate w i soft and w i hard by regularity of relation transition.

Original: Relation Transition.
Revised: Relation Transition.

Original: Relation transition means how the predicates change between the samesubject-object pairs in different frames.
Revised: Relation transition means how the predicates change between the samesubject-object pairs in different frames.

Original: Mi et al.
Revised: Mi et al.

Original: (2021) has proved that relation transition has thetemporal tendency, which means we can infer their relationship in the next frame through theirrelationship in the current frame for the same subject-object pair.
Revised: (2021) has proved that relation transition hasthe temporal tendency, which means we can infer their relation in the next frame according totheir relation in the current frame for the same subject-object pair.

Original: As the example shown in Figure 4, we can found two main conclusions: 1) Relation transitions have some regular patterns,e . g ., for subject-object pair “ people-chair ”, relation “ leaning on ” is mainly transferredto “ sitting on ”, but hardly transferred to “ lying on ” and “ standing on ”.
Revised: As the example shown in Figure 4, we can find two main conclusions: 1) Relation transitions have some regular patterns,e . g ., for subject-object pair “ people-chair ”, relation “ leaning on ” is mainly transferred to“ sitting on ”, but hardly transferred to “ lying on ” and “ standing on ”.

Original: 2) Relationtransitions with different subject-object pairs have different regular patterns, e .
Revised: 2) Relation transitions with different subject-object pairs have different regular patterns, e .

Original: g ., the statistical distributions for “ people-chair ” and “ people-food ” are different.
Revised: g ., the statistical distributions for “ people-chair ” and “ people-food ” are different.

Original: Therefore, we can evaluatethe quality of predicate pseudo labels by regularity of relation transition in video.
Revised: Therefore, we can evaluate thequality of predicate pseudo labels by regularity of relation transition in video.

Original: Specifically, irregular relation transitions ( e . g .
Revised: Specifically, irregular relation transitions ( e . g .

Original: , “ leaning on ” → “ lying on ” for “ people-chair ”) are oflow quality, then we assign low weights to them.
Revised: , “ leaning on ” → “ lying on ” for “ people-chair ”) are of lowquality, then we assign low weights to them.

Original: Proposed FPP.
Revised: Proposed FPP.

Original: As shown in Figure 5, We take two steps to calculate weights w soft and w hard . 1)
Revised: As shown in Figure 5, We take two steps to calculate weights w soft and w hard . 1)

Original: Predicate prediction : For each predicate p t in I t assigned by the model-based teacher, we predictˆ p t +1 as the predicate that p t will transition to in I t
Revised: Predicate prediction : For each predicate p t in I t assigned by the model-based teacher, we predictˆ p t +1 as the predicate that p t will transition to in I t

Original: +1 .
Revised: +1 .

Original: For each ˆ p t +1 , we find its correspondingpredicate p t +1 in I t
Revised: For each ˆ p t +1 , we find its correspondingpredicate p t +1 in I t

Original: +1 that actually assigned by the model-based teacher.
Revised: +1 that actually assigned by the model-based teacher.

Original: 2) Adapted weight assignment :
Revised: 2) Adapted weight assignment :

Original: We first calculate KL-divergence between ˆ p t +1 and p t +1 as the inconsistency score.
Revised: We first calculate KL-divergence between ˆ p t +1 and p t +1 as the inconsistency score.

Original: Then, we assign low weights to pseudo labels with high inconsistency scores and high weights topseudo labels with low inconsistency scores.
Revised: Then, we assign low weights to pseudo labels with high inconsistency scores and high weights topseudo labels with low inconsistency scores.

Original: In the following, we detailed introduce these two steps.
Revised: In the following, we detailed introduce these two steps.

Original: Predicate Prediction.
Revised: Predicate Prediction.

Original: To predict the future predicates, we construct a relation transition matrix
Revised: To predict the future predicates, we construct a relation transition matrix

Original: T ∈ R N o ∗ N r ∗ N r , where N o , N r are the number of objects and relations in the dataset, respectively.
Revised: T ∈ R N o ∗ N r ∗ N r , where N o , N r are the number of objects and relations in the dataset, respectively.

Original: The relation transition matrix indicates the statistical distribution of relation transition in the VidSGGdataset.
Revised: The relation transition matrix indicates the statistical distribution of relation transition in the VidSGGdataset.

Original: Given a relation p t in frame I t with object category c
Revised: Given a predicate p t in frame I t with object category c

Original: o , we predictthe future predicate ˆ p t[-+
 [Figure] [Figure] for the same subject-object pair in I t-] +1 as:
Revised: o , we predict ˆ p t[-+
 [Figure] [Figure] for the same subject-object pair in I t-] +1 as:

Original: [Equation]
Revised: [Equation]

Original: where id ( c o ) is the id of c o in the vocabulary C go .
Revised: where id ( c o ) is the id of c o in the vocabulary C go .

Original: There is a small problem that ˆ p t +1 sometimesdoes not have its corresponding predicate p t +1 inframe I t +1  because the detector can not detectthe subject-object pair.
Revised: Then, for each ˆ p t +1 , we try to find its corresponding predicate p t +1 in I t +1 that actually assigned by the model-based teacher. Sometimes p t +1 is notexisted because the detector can not detect the subject-object pair of ˆ p t +1 in frame I t +1 .

Original: Similar to object sequence matching in the model-free teacher, we adopt theobject category and the IoU to filter predicates without corresponding subject-object pair in the nextframe.
Revised: To handle [Figure] [Figure] this problem, we adopt the object category and the IoU to filter predicates without corresponding subject-object pair in the next frame, as we did in object sequence matching of the model-freeteacher.

Original: The set of ˆ p t +1 and p t +1 after filtering in the whole video are denoted as ˆ P filter and P filter ,respectively.
Revised: The set of ˆ p t +1 and p t +1 after filtering in the whole video are denoted as ˆ P filter and P filter ,respectively.

Original: Adapted Weight Assignment.
Revised: Adapted Weight Assignment.

Original: We first calculate the inconsistency score by KL-divergence as below: 
Revised: We first calculate the inconsistency score by KL-divergence as below: [Equation]

Original: [Equation]
Revised: [Equation]

Original: We then calculate the inconsistency score for P soft , denoted as S softincon .
Revised: Then, we calculate the inconsistency score for P soft , denoted as S softincon .

Original: A high inconsistency scoreindicates that these predicate pseudo labels do not satisfy the regularity of relation transition, so weshould assign low weights to them.
Revised: A high inconsistency scoreindicates that these predicate pseudo labels do not satisfy the regularity of relation transition, so weshould assign low weights to them.

Original: To the end, we define the function as:
Revised: To the end, we define the function as:

Original: [Equation]
Revised: [Equation]

Original: T RAINING O BJECTIVES
Revised: T RAINING O BJECTIVES

Original: FPP module.
Revised: FPP module.

Original: We assume that the relation transition matrix T in FPP module indicates the statisticaldistribution of relation transition in the VidSGG dataset.
Revised: We assume that the relation transition matrix T in FPP module indicates the statisticaldistribution of relation transition in the VidSGG dataset.

Original: However, in the SF-VidSGG task, we cannot count the distribution of relation transition directly because we only have single-frame weaksupervision for each video.
Revised: However, in the SF-VidSGG task, we cannot count the distribution of relation transition directly because we only have single-frame weaksupervision for each video.

Original: An alternative approach is generating pseudo labels by the model-basedteacher and then learning the distribution of these pseudo labels.
Revised: An alternative approach is generating pseudo labels by the model-basedteacher and then learning the distribution of these pseudo labels.

Original: To accomplish this, we formulatethe loss function as below to train the FPP module:
Revised: To accomplish this, we formulatethe loss function as below to train the FPP module:

Original: [Equation]
Revised: [Equation]

Original: Student Model.
Revised: Student Model.

Original: We adopt the KL-divergence loss for training like most knowledge distillationmethods, which can be formulated as follows:
Revised: We adopt the KL-divergence loss for training like most knowledge distillationmethods, which can be formulated as follows:

Original: [Equation]
Revised: [Equation]

Original:  where P pred denotes the output of the student model, and P final denotes the predicate pseudo labels.
Revised: ) where P pred denotes the output of the student model, and P final denotes the predicate pseudo labels.

Original: Finally, the total training loss of PLA is:
Revised: Finally, the total training loss of PLA is:

Original: [Equation]
Revised: [Equation]

Original:  E XPERIMENTS4. E XPERIMENTAL S ETTINGS
Revised:  E XPERIMENTS

Original: Datasets. We evaluated our method PLA on the challenging VidSGG benchmark: Action Genome(AG) (Ji et al., 2020). AG is a prevalent VidSGG dataset providing human-object relationships with frame-level annotations.
Revised: Due to the limited space, the details about experimental settings (including the evaluation datasetand metrics) and implementation details are left in the appendix.

Original: Specifically, AG consists of 9,201 videos, which annotated for 234,253frames by 476,299 bounding boxes of 36 object categories and 1,715,568 instances of 25 relationcategories. All relation categories in AG are split into three types: 3 attention relations, 6 spatialrelations and 16 contact relations. Each subject-object pair has one attention relation but may hasmultiple spatial or contact relations. In our SF-VidSGG setting, we used the official splits as fullysupervised work, i .
Revised: 4. A BLATION S TUDIES

Original: e ., 7,464 videos for training, 1,737 videos for testing. For each video in thetraining set, we only used the unlocalized scene graph of the middle frame as supervision. Evaluation Metrics. Following the fully-supervised setting, we evaluated PLA on three differentsettings: 1) Scene Graph Detection ( SGDet ): Given a video, the model needs to detect the objectsand predict predicate categories of object pairs for all frame.
Revised: In this section, we validated the effectiveness of our PLA by answering the following questions:

Original: In SGDet, an object is considered tobe successfully detected if the IoU between the predicted bbox and its ground-truth bbox is largerthan 0.5.

Original: 2) Scene Graph Classification ( SGCls ): Given a video and object bboxes, the model needsto predict object and predicate categories for all frames. Predicate Classification ( PredCls ): Given a video, object bboxes and object labels, the model needs to predict predicate categories forall frames. Since the ground-truth bboxes are not provided in SF-VidSGG, we only trained modelsunder SGDet settings, then evaluated them under these three settings.
Revised: Q1 :

Original: We used Recall@K ( R@K , K= [10,20,50]) as our evaluation metric, which measures the ratio of the ground-truth relation tripletsamong the top-K predicted relation triplets.

Original: Furthermore, we adopt two typical strategies to generatedynamic scene graphs: 1) With Constraint : It only allows each subject-object pair to have at mostone predicate.

Original: No Constraint : It allows each subject-object pair tohave multiple predicates. Implementation Details. In the Obj-PLA module, we used the pretrained VinVL (Li et al., 2020; Zhang et al., 2021) with backbone ResNeXt-152 C4 as the off-the-shelf detector.
Revised: How does each component in PLA contribute to the performance on the SF-VidSGG task?

Original: Specifically,it is pre-trained on multiple detection benchmarks, including COCO (Lin et al., 2014), OpenImages (Kuznetsova et al., 2020), Objects365 (Shao et al., 2019), and Visual Genome (Krishna et al.,2017). This detector is capable of detecting 1,594 general object categories. We kept objects whichconfidence is higher than 0.2 and extracted the 2048-D region features from the detector.
Revised: Q2 :

Original: In the Rel-PLA module, we modified STTran (Cong et al., 2021) to an ImgSGG model that process oneframe at a time, denoted as STTran † . We used STTran † as the teacher model to assign pseudo softlabels in Rel-PLA.
Revised: How far are the gaps between PLA and fully-supervised VidSGG approaches?

Original: For the model-free teacher, we set the IOU matching threshold η = 0 .

Original: In FPPmodule, we set the initial learning rate 1 e − 3 .

Original: We used STTran as the student model, and followedthe same training settings ( e .

Original: g ., learning rate and batch size) of the original paper.

Original: 4. A BLATION S TUDIES

Original: In this section, we validated the effectiveness of our PLA by answering the following questions:

Original: Q1 :
Revised: Q3 :

Original: How does each component in PLA contribute to  the performance on the SF-VidSGG task?
Revised: Can a strongerstudent model help to improve the performance of PLA?

Original: Q2 :
Revised: Q4 :

Original: What is the upper-bound under our pseudo label assignment training paradigm?
Revised: What is the upper-bound under ourpseudo label assignment training paradigm?

Original: Q3 :
Revised: Q5 :

Original: How do wechoose hyperparameters in our model?
Revised: What is the influence of different annotation framechoosing strategies? [Table] [Table] [Table]

Original: 4.2. E FFECTIVENESS OF E ACH C OMPONENT (Q1)
Revised: E FFECTIVENESS OF E ACH C OMPONENT (Q1)

Original: Settings.
Revised: Settings.

Original: To analyze the importance of each component in PLA, we implemented the straightforward model mentioned in Sec.
Revised: To analyze the importance of each component in PLA, we implemented the straightforward model mentioned in Sec.

Original: 1 as the baseline model.
Revised: 1 as the baseline model.

Original: It first obtained pseudo localized scenegraphs G pseudo from the annotated frame set by Obj-PLA, then trained a STTran † model by thesescene graphs.
Revised: It first obtained pseudo localized scenegraphs G pseudo from the annotated frame set by Obj-PLA, then trained a STTran † model by thesescene graphs.

Original: The gray lines in Table 1 show the results of the baseline model (model A ).
Revised: The gray lines in Table 1 show the results of the baseline model (model A ).

Original: We alsotrained the models with different components of PLA, which is shown in Table 1.
Revised: We alsotrained the models with different components of PLA, which is shown in Table 1.

Original: Effectiveness of Rel-PLA.
Revised: Effectiveness of Rel-PLA.

Original: Compared to the baseline model A , models with model-based teacher ormodel-free teacher (model B and model C ) has a significant improvement.
Revised: Compared to the baseline model A , models with model-based teacher ormodel-free teacher (model B and model C ) has a significant improvement.

Original: The experimental resultsdemonstrate that by utilizing the unannotated frames in the video, model B and model C are ableto outperform the baseline model
Revised: The experimental resultsdemonstrate that by utilizing the unannotated frames in the video, model B and model C are ableto outperform the baseline model

Original: A across almost all the metrics, only except for no constrained
Revised: A across almost all the metrics, only except for no constrained

Original: PredCls-R@50 of model B .
Revised: PredCls-R@50 of model B .

Original: Particularly, model B and model C have outperformed the baseline by5.24% (15.07 v .
Revised: Particularly, model B and model C have outperformed the baseline by5.24% (15.07 v .

Original: s .14.32) and 2.58% (14.69 v .
Revised: s .14.32) and 2.58% (14.69 v .

Original: s .14.32)
Revised: s .14.32)

Original: relatively in terms of R@10 with constraintcriteria for SGDet, respectively.
Revised: relatively in terms of R@10 with constraintcriteria for SGDet, respectively.

Original: This is owing to that the two teachers have the ability to assignhigh-quality pseudo labels to the unannotated frames.the equally [Table] [Table]
Revised: This is owing to that the two teachers have the ability to assignhigh-quality pseudo labels to the unannotated frames.

Original: %) for SGDet on AG of the model only uses model-free teacher with different IoU matching threshold η .

Original: [Table]

Original: Effectiveness of FPP.
Revised: Effectiveness of FPP.

Original: As shown in Table 1, model D and model E are all use both teachers, butmodel E uses the FPP module to set adapted weights.
Revised: As shown in Table 1, model D and model E are all use both teachers, butmodel E uses the FPP module to set adapted weights.

Original: We can observe that model E outperformsmodel D across all the metrics.
Revised: We can observe that model E outperformsmodel D across all the metrics.

Original: Especially, model E has achieved about 2% relative improvementover model D on R@10 metric under the no constraint criteria for SGDet.
Revised: Especially, model E has achieved about 2% relative improvementover model D on R@10 metric under the no constraint criteria for SGDet.

Original: The improvement provesthat FFP has the ability to assign more accurate pseudo labels than the simple fixed weight strategy.
Revised: The improvement provesthat FFP has the ability to assign more accurate pseudo labels than the simple fixed weight strategy.

Original: 4.2. B EST C ASE : G ROUND - TRUTH U NLOCALIZED S CENE G RAPHS (Q2)
Revised: [Table] [Table] [Table] G AP BETWEEN PLA AND FULLY S UPERVISED A PPROACHS (Q2)

Original: Settings.
Revised: Settings.

Revised: We compared our PLA with three state-of-the-art fully-supervised VidSGG methods:

Revised: STTran (Cong et al., 2021), APT (Li et al., 2022d), and DSG-DETR (Feng et al., 2021).

Revised: Results.

Revised: As shown in Table 2, we can observe that there is still a gap between the performanceof PLA and other fully-supervised approaches ( e .

Revised: g ., 15.4 v .

Original: {+s .25.
+}
Revised: {+s .25.
+}

Original: To address this question, we trained a model with the ground-truth unlocalized scenegraph annotations of all frames as the upper-bound model, denoted as GTRel.
Revised: for STTran in terms of R@10with constraint criteria for SGDet). However, considering that the annotation cost of our methodis extremely lower ( < 5%) than that of full supervision. we think it is still a good start for weaklysupervised frame-level VidSGG. We leave the task to reduce the gap as the future work.

Revised: 4.1. E FFECTS OF D IFFERENT S TUDENT M ODELS

Original: More specifically, wefirst utilized the Obj-PLA module togenerate pseudo localized scene graphs from the ground-truthunlocalized scene graphs.
Revised: (Q3) Settings. We used DSG-DETR as the student model of PLA. Since PLA is agnostic to the specific

Original: Then we used these pseudo localized scene graphs to train a VidSGGmodel.
Revised: VidSGG architecture, it can be easily incorporated into any advanced VidSGG models. We usedthe same teachers as for STTran to generate pseudo labels for DSG-DETR, then trained a fullysupervised VidSGG model by these pseudo labels. Results. As shown in Table 2, PLA with DSG-DETR also achieves similar performance with

Revised: STTran across all the metrics.

Original: Here we fixed the off-shelf detector in Obj-PLA and the final student model, because PLAis agnostic to them.
Revised: These results show that PLA is a general framework and can easily extend to advanced fully supervised VidSGG methods. 4.1. B EST C ASE : G ROUND - TRUTH U NLOCALIZED S CENE G RAPHS (Q4) Settings. Like most existing SGG methods, PLA has two main steps: the first step is to detect objectsin each frame, and the second step is to predict relations between these objects.

Original: We compared baseline model, PLA and GTRel in Table 2.
Revised: The performanceof PLA is mainly affected by the first step due to the poor performance of the off-shelf detectorin Action Genome. Under this object detection strategy, the upper-bound performance is not high. We can train an upper-bound model as follows: first detect objects by Obj-PLA, then assign groundtruth relation labels between them, finally use these labels to train the student model. We named thisupper-bound model as GTRel.

Original: Results.
Revised: Results.

Original: As shown in Table 1, we observe that GTRel significantly improves the performanceover the baseline.
Revised: As shown in Table 3, we observe that GTRel significantly improves the performanceover the baseline.

Original: Specifically, GTRel has outperformed the baseline by 13.19% (16.73 v .
Revised: Specifically, GTRel has outperformed the baseline by 13.19% (16.73 v .

Original: s .14.72)relatively
Revised: s .14.72)relatively

Original: in terms of R@10 no constraint criteria for SGDet.
Revised: in terms of R@10 no constraint criteria for SGDet.

Original: We also demonstrate that PLA hasability to reduce the gap between the baseline model and GTRel.
Revised: We also demonstrate that PLA hasability to reduce the gap between the baseline model and GTRel.

Original: For example, GTRel improvesby 12.43% (16.10 v .
Revised: For example, GTRel improvesby 12.43% (16.10 v .

Original: s .14.32)
Revised: s .14.32)

Original: relatively over baseline in terms of R@10 with constraint criteria for
Revised: relatively over baseline in terms of R@10 with constraint criteria for

Original: SGDet, but only improves by 4.61% (16.10 v .
Revised: SGDet, but only improves by 4.61% (16.10 v .

Original: s .15.39) relatively over PLA on same metric.
Revised: s .15.39) relatively over PLA on same metric.

Original: Theseresults show that PLA is able to reduce about half performance gap between the baseline model andthe upper-bound model GTRel for the SF-VidSGG task.
Revised: Theseresults show that PLA is able to reduce about half performance gap between the baseline modeland the upper-bound model GTRel for the SF-VidSGG task.

Revised: Besides, the method to detect objectsunder unlocalized scene graph setting is not our main contribution.

Revised: As a general framework, if bettermethods to detect objects are proposed, PLA can easily extend to them.

Original: 4.2. A BLATION S TUDY ON H YPERPARAMETER S ETTINGS (Q3)
Revised: 4.1. E FFECTS OF D IFFERENT A NNOTATION F RAME -C HOOSING S TRATEGIES . (Q5)

Original: Settings.
Revised: Settings.

Original: We analyzed the influence of the hyperparameter η , which denote the IoU matching threshold in Rel-PLA. We set different η for the model only uses model-free teacher (model C ).
Revised: We compared four simple annotation frame choosing strategies.

Original: All resultsare in Table 3. Particularly, when the threshold η is set to 0, we match the objects in different framesonly by the object category without the IoU of the bounding boxes in adjacent frames.
Revised: Specifically, “First”,“Last”, and “Middle” means annotating the “first”, “last”, and “middle” frame with an unlocalizedscene graph in each video, respectively.

Original: [Figure] (Weakly [Figure] [Figure]
Revised: And “Random” means choosing one frame randomly.

Original: Results.
Revised: Results.

Original: As shown in Table 3, we can observe thatthe model  with η = 0 has a weak performancedue to more wrong objects matching without IoU information.
Revised: As shown in Table 4, We can observe that model trained by the middle annotation frame hasthe best performance, eg., in terms of R@10 with constraint criteria for SGDet, strategy “Middle” [Figure] [Figure]

Original: And the model with η = 0 . 5 slightlyoutperforms the model with η = 0 . Therefore, we set η = 0.5 for all experiments.
Revised: outperforms “First”, “Last”, and “Random” by 1.01, 1.14, and 0.48, respectively. It also proves thatthe middle frame usually contains more information.

Original: 4. Q UALITATIVE R ESULTS
Revised: 4. Q UALITATIVE R ESULTS

Original: Pseudo Label Visualization.
Revised: Pseudo Label Visualization.

Original: Figure 6 shows a qualitative comparison of the pseudo labels generated by different variants of PLA.
Revised: Figure 6 shows a qualitative comparison of the pseudo labels generated by different variants of PLA.

Original: Results indicates that the adapted weights given by FPP modulecan generate more accurate pseudo labels than fixed weights.
Revised: Results indicates that the adapted weights given by FPP modulecan generate more accurate pseudo labels than fixed weights.

Original: For example, relation in T m [-+
-] (“ notcontacting ”) is different from that in T m (“ touching ” and “ sitting on ”), so the modelfree teacher give totally wrong relations (“ touching ” and “ sitting on ”) and the model-basedteacher predict an almost accurate relation (98%  for “ not contacting ”).
Revised: Specifically, relation in T m [-+
-]+1 (“ notcontacting ”) is different from that in T m (“ touching ” and “ sitting on ”), so the modelfree teacher give totally wrong relations (“ touching ” and “ sitting on ”) and the model-basedteacher predict an almost accurate relation (98% confidence for “ not contacting ”).

Original: Compared to the fixedweight, the FPP module gives more weight to the model-based teacher, and generates more accuratepseudo labels.
Revised: Compared to fixed weight strategy, the FPP module gives more weight to the model-based teacher (79%v . s .49% confidence for “ not contacting ”) and generates more accurate pseudo labels.

Original: Relation Transition Distribution Visualization.
Revised: Relation Transition Distribution Visualization.

Original: Figure 7 demonstrates the statistical distributionsof relation transition counted by ground truth and learned by PLA.
Revised: Figure 7 demonstrates the statistical distributionsof relation transition counted by ground truth and learned by PLA.

Original: From Figure 7, we can observethat PLA can learn a distribution similar to the ground truth.
Revised: From Figure 7, we can observethat PLA can learn a distribution similar to the ground truth.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: In this paper, we presented the first weakly-supervised VidSGG task with only single-frame weak supervision: SF-VidSGG.
Revised: In this paper, we presented the first weakly-supervised VidSGG task with only single-frame weak supervision: SF-VidSGG.

Original: Unlike VidSGG, SF-VidSGG only require a weak annotaion without bounding box for one frame in each video.
Revised: Unlike VidSGG, SF-VidSGG only require a weak annotaion without bounding box for one frame in each video.

Original: To the end, we proposed a novel and efficient method named
Revised: To the end, we proposed a novel and efficient method named

Original: PLA for SF-VidSGG, which based on pseudo label assignment for the video.
Revised: PLA for SF-VidSGG, which based on pseudo label assignment for the video.

Original: We validated the effectiveness of each component of PLA through extensive experiments.
Revised: We validated the effectiveness of each component of PLA through extensive experiments.

Original: In future work, we wouldlike to explore weaker supervision in VidSGG or more accurate pseudo label assignment.
Revised: In future work, we wouldlike to explore weaker supervision in VidSGG or more accurate pseudo label assignment.

Original: Ethics Statement
Revised: Ethics Statement

Original: Video scene generation models can be utilized in unauthorized human monitoring and surveillance, which leads to some ethical and privacy issues.
Revised: Video scene generation models may be utilized in unauthorized human monitoring and surveillance,which leads to some ethical and privacy issues.

Original: Apart from these general issues that already exist inthe video scene graph generation task, our paper has no additional ethical issues.
Revised: Apart from these general issues that already exist in the video scene graph generation task, our paper has no additional ethical issues

Original: Reproducibility Statement
Revised: .Reproducibility Statement

Original: PLA is mainly implemented based on the released code of STTran (Cong et al., 2021) and evaluatedon the Action Genome (AG)
Revised: PLA is mainly implemented based on the released code of STTran (Cong et al., 2021) and evaluatedon the Action Genome (AG)

Original: (Ji et al., 2020) dataset, which is publicly available with a license thatallows free usage for research purposes.
Revised: (Ji et al., 2020) dataset, which is publicly available with a license thatallows free usage for research purposes.

Original: We will also release our code of PLA. 
Revised: We will also release our code of PLA. A A PPENDIX

Revised: This Appendix has the following contents:

Revised: • The details about the valuation dataset are given in Sec.

Revised: A.1•

Revised: The evaluation metrics are given in Sec.

Revised: A.2.• The implementation details are given in Sec.

Revised: A.3.• The detailed hyperparameter setting are given in Sec.

Revised: A.4.• Ablation studies on imperfect object detector in Sec.

Revised: A.5.• Ablation studies on the different proportions of the weakly supervised labels in Sec.

Revised: A.6.

Revised: A. D ATASET

Revised: We evaluated our method PLA on the challenging VidSGG benchmark: Action Genome (AG) (Jiet al., 2020).

Revised: AG is a prevalent VidSGG dataset providing human-object relationships with framelevel annotations.

Revised: Specifically, AG consists of 9,201 videos, which annotated for 234,253 frames by476,299 bounding boxes of 36 object categories and 1,715,568 instances of 25 relation categories.

Revised: All relation categories in AG are split into three types: 3 attention relations, 6 spatial relations and16 contact relations.

Revised: Each subject-object pair has one attention relation but may has multiple spatialor contact relations.

Revised: In our SF-VidSGG setting, we used the official splits as fully-supervised work,i .

Revised: e ., 7,464 videos for training, 1,737 videos for testing.

Revised: For each video in the training set, we onlyused the unlocalized scene graph of the middle frame as supervision.

Revised: A. E VALUATION M ETRICS

Revised: Following the fully-supervised setting, we evaluated PLA on three different settings: 1) Scene Graph

Revised: Detection ( SGDet ): Given a video, the model needs to detect the objects and predict predicatecategories of object pairs for all frame.

Revised: In SGDet, an object is considered to be successfully detectedif the IoU between the predicted bbox and its ground-truth bbox is larger than 0.5.

Revised: 2) Scene Graph

Revised: Classification ( SGCls ): Given a video and object bboxes, the model needs to predict object andpredicate categories for all frames.

Revised: Predicate Classification ( PredCls ): Given a video, object bboxes and object labels, the model needs to predict predicate categories for all frames.

Revised: Since theground-truth bboxes are not provided in SF-VidSGG, we only trained models under SGDet settings,then evaluated them under these three settings.

Revised: We used Recall@K ( R@K , K = [10,20,50]) as ourevaluation metric, which measures the ratio of the ground-truth relation triplets among the top-Kpredicted relation triplets.

Revised: Furthermore, we adopt two typical strategies to generate dynamic scenegraphs: 1) With Constraint : It only allows each subject-object pair to have at most one predicate.2) No Constraint

Revised: It allows each subject-object pair to have multiple predicates.

Revised: A. I MPLEMENTATION D ETAILS

Revised: In the Obj-PLA module, we used the pretrained VinVL (Li et al., 2020; Zhang et al., 2021) withbackbone ResNeXt-152 C4 as the off-the-shelf detector.

Revised: Specifically, it is pre-trained on multipledetection benchmarks, including COCO (Lin et al., 2014), OpenImages (Kuznetsova et al., 2020),

Revised: Objects365 (Shao et al., 2019), and Visual Genome (Krishna et al., 2017).

Revised: This detector is capableof detecting 1,594 general object categories.

Revised: We kept objects which confidence is higher than 0.2and extracted the 2048-D region features from the detector.

Revised: In the Rel-PLA module, we modified

Revised: STTran (Cong et al., 2021) to an ImgSGG model that process one frame at a time, denoted as

Revised: STTran † .

Revised: We used STTran † as the teacher model to assign pseudo soft labels in Rel-PLA.

Revised: For themodel-free teacher, we set the IOU matching threshold η = 0 .

Revised: In FPP module, we set the initiallearning rate 1 e − 3 .

Revised: We used STTran as the student model, and followed the same training settings( e .

Revised: g ., learning rate and batch size) of the original paper.

Revised: A. H YPERPARAMETER S ETTINGS

Revised: We analyzed the influence of the hyperparameter η , which denote the IoU matching threshold in

Revised: Rel-PLA.

Revised: We set different η for the model only uses model-free teacher (model C ).

Revised: All results are in

Revised: Table 5.

Revised: Particularly, when the threshold η is set to 0, we match the objects in different frames onlyby the object category without the IoU of the bounding boxes in adjacent frames.

Revised: [Table]

Revised: Results.

Revised: As shown in Table

Revised: [Table]

Revised: we can observe that the model with η = 0 has a weak performancedue to more wrong objects matching without IoU information.

Revised: And the model with η = 0 .

Revised: 5 slightlyoutperforms the model with η = 0 .

Revised: Therefore, we set η = 0.5 for all experiments.

Revised: [Table]

Revised: [Table]

Revised: A. E FFECTS OF I MPERFECT O BJECT D ETECTORS

Revised: Settings.

Revised: To simulate an imperfect object detector, we randomly dropped a certain proportion ofbounding boxes given by the object detector.

Revised: Without loss of generality, we set five dropped proportion as 0%, 10%, 20%, 40% and 80%.

Revised: Results.

Revised: As shown in Table 6, we can observe that the bigger dropped proportion, the worse theperformance of PLA, eg.

Revised: the setting with 0% dropped proportion outperforms settings with 10%,20%, 40% and 80% dropped proportion by 1.23, 2.21, 3.91 and 10.55, respectively.

Revised: When dropping10%, 20% and 40% bboxes, the performance has not decreased much, but the performance decreasedsignificantly if dropping 80% bboxes.

Revised: Results show that PLA is affected by the preformance of theobject detector, but it is the weakness for all two-stage (first detect objects, then predict relationsbetween them) SGG method.

Revised: A. E FFECTS OF THE S IZE OF W EAKLY S UPERVISED L ABELS

Revised: Settings.

Revised: Although PLA is designed for single-frame weak supervision, it can be easily extendedto multi-frame weak supervision.

Revised: For the model-based teacher, we just train the model with moreframes.

Revised: For the model-free teacher, we take each annotated frame as a starting point to propagaterelations.

Revised: Without loss of generality, we set four annotated proportion as 10%, 20%, 40% and 80%.

Revised: We annotated frame evenly in temporal axis, for example, if a 20-frame video with 20% annotatedproportion, we will annotate its 1st, 7th, 14th and 20th frame.

Revised: Results.

Revised: As shown in 7, we can observe that the more annotated frames, the better the performanceof PLA, e.g., model with 80% annotation frames outperforms models with 10%, 20% and 40%annotation frames 1.16, 0.26 and 0.03.

Revised: It proves that PLA can easily adapt to multi-frame weaksupervision setting.