Original: V IDEO S CENE G RAPH G ENERATION FROM S INGLE -F RAME W EAK S UPERVISION
Revised: V IDEO S CENE G RAPH G ENERATION FROM S INGLE -F RAME W EAK S UPERVISION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Video scene graph generation (VidSGG) aims to generate a sequence of graphstructure representations for the given video.
Revised: Video scene graph generation (VidSGG) aims to generate a sequence of graphstructure representations for the given video.

Original: However, all existing VidSGG methods are fully-supervised , i .
Revised: However, all existing VidSGG methods are fully-supervised , i .

Original: e ., they need dense and costly manual annotations.
Revised: e ., they need dense and costly manual annotations.

Original: Inthis paper, we propose the first weakly-supervised VidSGG task with only singleframe weak supervision: SF-VidSGG.
Revised: Inthis paper, we propose the first weakly-supervised VidSGG task with only singleframe weak supervision: SF-VidSGG.

Original: By “weakly-supervised”, we mean that
Revised: By “weakly-supervised”, we mean that

Original: SF-VidSGG relaxes the training supervision from two different levels: 1) It onlyprovides single-frame annotations instead of all-frame annotations.
Revised: SF-VidSGG relaxes the training supervision from two different levels: 1) It onlyprovides single-frame annotations instead of all-frame annotations.

Original: The singleframe ground-truth annotation is still a weak image SGG annotation, i .
Revised: The singleframe ground-truth annotation is still a weak image SGG annotation, i .

Original: e
Revised: e

Original: ., an unlocalized scene graph.
Revised: ., an unlocalized scene graph.

Original: To solve this new task, we also propose a novel Pseudo
Revised: To solve this new task, we also propose a novel Pseudo

Original: Label Assignment based method, dubbed as PLA .
Revised: Label Assignment based method, dubbed as PLA .

Original: PLA is a two-stage method,which generates pseudo visual relation annotations for the given video at the firststage, and then trains a fully-supervised VidSGG model with these pseudo labels.
Revised: PLA is a two-stage method,which generates pseudo visual relation annotations for the given video at the firststage, and then trains a fully-supervised VidSGG model with these pseudo labels.

Original: Specifically, PLA consists of three modules: an object PLA module, a predicate
Revised: Specifically, PLA consists of three modules: an object PLA module, a predicate

Original: PLA module, and a future predicate prediction (FPP) module.
Revised: PLA module, and a future predicate prediction (FPP) module.

Original: Firstly, in the object PLA, we localize all objects for every frame.
Revised: Firstly, in the object PLA, we localize all objects for every frame.

Original: Then, in the predicate PLA,we design two different teachers to assign pseudo predicate labels.
Revised: Then, in the predicate PLA,we design two different teachers to assign pseudo predicate labels.

Original: Lastly, in the
Revised: Lastly, in the

Original: FPP module, we fusion these two predicate pseudo labels by the regularity of relation transition in videos.
Revised: FPP module, we fusion these two predicate pseudo labels by the regularity of relation transition in videos.

Original: Extensive ablations and results on the benchmark Action
Revised: Extensive ablations and results on the benchmark Action

Original: Genome have demonstrated the effectiveness of our PLA.
Revised: Genome have demonstrated the effectiveness of our PLA.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Scene graph (Johnson et al., 2015) is a type of visually-aligned graph-structured representation thatsummarizes all the object instances ( e .
Revised: Scene graph (Johnson et al., 2015) is a type of visually-aligned graph-structured representation thatsummarizes all the object instances ( e .

Original: g ., “ person ”, “ chair ”) as nodes and their pairwise visualrelations (or predicates, e .
Revised: g ., “ person ”, “ chair ”) as nodes and their pairwise visualrelations (or predicates, e .

Original: g .
Revised: g .

Original: , “ sitting on ”) as directed edges.
Revised: , “ sitting on ”) as directed edges.

Original: As a bridge to connect the visionand language modalities, scene graphs have been widely used in many different downstream visuallanguage tasks, such as visual captioning (Yang et al., 2019; 2020), grounding (Jing et al.
Revised: As a bridge to connect the visionand language modalities, scene graphs have been widely used in many different downstream visuallanguage tasks, such as visual captioning (Yang et al., 2019; 2020), grounding (Jing et al.

Original: 2020),question answering (Hudson & Manning, 2019), and retrieval (Johnson et al., 2015).
Revised: 2020),question answering (Hudson & Manning, 2019), and retrieval (Johnson et al., 2015).

Original: Early Scene Graph Generation (SGG) work mainly focuses on generating scene graphs for the givenimage , dubbed as ImgSGG
Revised: Early Scene Graph Generation (SGG) work mainly focuses on generating scene graphs for the givenimage , dubbed as ImgSGG

Original: (Xu et al., 2017; Zellers et al., 2018; Chen et al., 2019).
Revised: (Xu et al., 2017; Zellers et al., 2018; Chen et al., 2019).

Original: However,due to its static nature, ImgSGG fails to represent numerous dynamic visual relations that takeplace over a period of time, such as “ walking with ” and “ running away ” (vs. static relation“ standing ”).
Revised: However,due to its static nature, ImgSGG fails to represent numerous dynamic visual relations that takeplace over a period of time, such as “ walking with ” and “ running away ” (vs. static relation“ standing ”).

Original: Meanwhile, it is hard or impossible to identify these dynamic visual relations withonly a single frame, because these visual relations can only be well classified by considering thetemporal context.
Revised: Meanwhile, it is hard or impossible to identify these dynamic visual relations withonly a single frame, because these visual relations can only be well classified by considering thetemporal context.

Original: Therefore, another more meaningful but challenging video-based SGG task wasproposed: VidSGG (Shang et al., 2017; 2019).
Revised: Therefore, another more meaningful but challenging video-based SGG task wasproposed: VidSGG (Shang et al., 2017; 2019).

Original: Since the complex and dense annotations of a scene graph ( cf. , Figure 1(a)), fully-supervised SGGmethods always require lots of manual annotations, and the case is even worse for video data.
Revised: Since the complex and dense annotations of a scene graph ( cf. , Figure 1(a)), fully-supervised SGGmethods always require lots of manual annotations, and the case is even worse for video data.

Original: Meanwhile, several prior SGG works (Li et al., 2022a) have found that even carefully manually-annotatedlabels are still quite noisy, i .
Revised: Meanwhile, several prior SGG works (Li et al., 2022a) have found that even carefully manually-annotatedlabels are still quite noisy, i .

Original: e ., the annotated positive labels may not be the best matched, and numerous negative labels are just missing annotated.
Revised: e ., the annotated positive labels may not be the best matched, and numerous negative labels are just missing annotated.

Original: Thus, a surge of recent ImgSGG work (Zareianet al., 2020; Zhong et al., 2021; Shi et al., 2021; Li et al., 2022c) start to generate scene graphs forimages with only weak supervision.
Revised: Thus, a surge of recent ImgSGG work (Zareianet al., 2020; Zhong et al., 2021; Shi et al., 2021; Li et al., 2022c) start to generate scene graphs forimages with only weak supervision.

Original: By “weak supervision”, we mean that the annotations for modeltraining are not complete localized scene graphs.
Revised: By “weak supervision”, we mean that the annotations for modeltraining are not complete localized scene graphs.

Original: For example, a typical type of weak supervision is unlocalized scene graphs .
Revised: For example, a typical type of weak supervision is unlocalized scene graphs .

Original: As illustrated in Figure 1(b), an unlocalized scene graph only containsimage-level object and relation categories without corresponding object bounding boxes (bboxes).
Revised: As illustrated in Figure 1(b), an unlocalized scene graph only containsimage-level object and relation categories without corresponding object bounding boxes (bboxes).

Original: [Figure]
Revised: [Figure]

Original: floor towel
Revised: floor towel

Original: Although recent weakly-supervised ImgSGGhas achieved good performance and receivedunprecedented attention, to the best our knowledge, there is no existing work about generating video scene graphs from weak supervision.
Revised: Although recent weakly-supervised ImgSGGhas achieved good performance and receivedunprecedented attention, to the best our knowledge, there is no existing work about generating video scene graphs from weak supervision.

Original: To put forward the research on this criticaltopic, we propose the first weakly-supervised
Revised: To put forward the research on this criticaltopic, we propose the first weakly-supervised

Original: VidSGG task with single-frame weak supervision, called SF-VidSGG .
Revised: VidSGG task with single-frame weak supervision, called SF-VidSGG .

Original: Specifically, given aninput video, SF-VidSGG aims to generate a localized scene graph for each frame in the video,but the only supervision for training is an unlocalized scene graph for the middle frame ofeach training video .
Revised: Specifically, given aninput video, SF-VidSGG aims to generate a localized scene graph for each frame in the video,but the only supervision for training is an unlocalized scene graph for the middle frame ofeach training video .

Original: As the example shown in
Revised: As the example shown in

Original: Figure 1(c), the supervision is an unlocalizedscene graph for the third frame.
Revised: Figure 1(c), the supervision is an unlocalizedscene graph for the third frame.

Original: Obviously, SFVidSGG task tries to relieve the intensive annotation issues from two levels: 1) Video-level:
Revised: Obviously, SFVidSGG task tries to relieve the intensive annotation issues from two levels: 1) Video-level:

Original: For each video, we only need single-frame annotations instead of all-frame annotations as the fullysupervised setting ( i .
Revised: For each video, we only need single-frame annotations instead of all-frame annotations as the fullysupervised setting ( i .

Original: e ., reduce the number of annotated frames).
Revised: e ., reduce the number of annotated frames).

Original: 2) Frame-level: The single-frameannotation is an unlocalized scene graph ( i .
Revised: 2) Frame-level: The single-frameannotation is an unlocalized scene graph ( i .

Original: e ., avoid annotating object locations).
Revised: e ., avoid annotating object locations).

Original: A straightforward solution for SF-VidSGG is: Using all the weakly annotated frames to train aweakly-supervised ImgSGG model first, and then detecting scene graphs on each frame with the
Revised: A straightforward solution for SF-VidSGG is: Using all the weakly annotated frames to train aweakly-supervised ImgSGG model first, and then detecting scene graphs on each frame with the

Original: ImgSGG model.
Revised: ImgSGG model.

Original: Apparently, this naive ImgSGG-based method has overlooked the temporal context in the video data.
Revised: Apparently, this naive ImgSGG-based method has overlooked the temporal context in the video data.

Original: To this end, we propose a novel Pseudo Label Assignment strategy PLA ,which can serve as the first baseline for SF-VidSGG.
Revised: To this end, we propose a novel Pseudo Label Assignment strategy PLA ,which can serve as the first baseline for SF-VidSGG.

Original: Since PLA is agnostic to different VidSGGarchitectures, it can be easily incorporated into any advanced VidSGG model.
Revised: Since PLA is agnostic to different VidSGGarchitectures, it can be easily incorporated into any advanced VidSGG model.

Original: Specifically, PLAdecouples the problem into two steps: The first step is to assign a pseudo localized scene graphfor every frame in the video, and the second step is to train a fully-supervised VidSGG model bythe pseudo localized scene graphs.
Revised: Specifically, PLAdecouples the problem into two steps: The first step is to assign a pseudo localized scene graphfor every frame in the video, and the second step is to train a fully-supervised VidSGG model bythe pseudo localized scene graphs.

Original: PLA consists of three modules: object pseudo label assignmentmodule (Obj-PLA), relation pseudo label assignment module (Rel-PLA), and future predicate prediction module (FPP).
Revised: PLA consists of three modules: object pseudo label assignmentmodule (Obj-PLA), relation pseudo label assignment module (Rel-PLA), and future predicate prediction module (FPP).

Original: In the Obj-PLA module, we detect object region proposals for all the frames.
Revised: In the Obj-PLA module, we detect object region proposals for all the frames.

Original: In the Rel-PLA module, we propose two relation pseudo label assignment teachers and they generate two different pseudo labels for each frame.
Revised: In the Rel-PLA module, we propose two relation pseudo label assignment teachers and they generate two different pseudo labels for each frame.

Original: In the FPP module, we determine adapted weightsto fuse these labels from two different teachers.
Revised: In the FPP module, we determine adapted weightsto fuse these labels from two different teachers.

Original: To effectively obtain optimal adapted weights forfusing different teacher knowledge, the FPP module exploits the temporal context based on the relation transition in videos.
Revised: To effectively obtain optimal adapted weights forfusing different teacher knowledge, the FPP module exploits the temporal context based on the relation transition in videos.

Original: The relation transition means how the predicates change between the samesubject-object pairs in different frames.
Revised: The relation transition means how the predicates change between the samesubject-object pairs in different frames.

Original: In summary, we make three main technique contributions in this paper:
Revised: In summary, we make three main technique contributions in this paper:

Original: We propose the first weakly-supervised VidSGG task.
Revised: We propose the first weakly-supervised VidSGG task.

Original: Compared to its fully-supervised counterpart, we try to mitigate the intensive annotations from both video-level and frame-level.
Revised: Compared to its fully-supervised counterpart, we try to mitigate the intensive annotations from both video-level and frame-level.

Original: We propose a novel method PLA for SF-VidSGG.
Revised: We propose a novel method PLA for SF-VidSGG.

Original: It utilizes two teachers to assign pseudo labelfor unlabeled
Revised: It utilizes two teachers to assign pseudo labelfor unlabeled

Original: data and refines the pseudo labels from both teachers by knowledge distillation.3.
Revised: data and refines the pseudo labels from both teachers by knowledge distillation.3.

Original: We propose a future predicate prediction module that leverages temporal dependencies in video.
Revised: We propose a future predicate prediction module that leverages temporal dependencies in video.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Image Scene Graph Generation (ImgSGG).
Revised: Image Scene Graph Generation (ImgSGG).

Original: ImgSGG aims to generate semantic graph structures— scene graphs — as the representation of images.
Revised: ImgSGG aims to generate semantic graph structures— scene graphs — as the representation of images.

Original: In each scene graph, every node represents anobject instance and every edge represents a visual relation between two objects.
Revised: In each scene graph, every node represents anobject instance and every edge represents a visual relation between two objects.

Original: Early ImgSGGmethods always directly predict all pairwise visual relations (Lu et al., 2016; Zhang et al., 2017).
Revised: Early ImgSGGmethods always directly predict all pairwise visual relations (Lu et al., 2016; Zhang et al., 2017).

Original: Later, ImgSGG methods mainly focus on designing different context encoding architectures, suchas message passing (Xu et al., 2017; Li et al., 2017), recurrent network (Zellers et al., 2018), treestructure model (Tang et al., 2019), agent communication (Chen et al., 2019), and Transformer-based models (Lu et al., 2021).
Revised: Later, ImgSGG methods mainly focus on designing different context encoding architectures, suchas message passing (Xu et al., 2017; Li et al., 2017), recurrent network (Zellers et al., 2018), treestructure model (Tang et al., 2019), agent communication (Chen et al., 2019), and Transformer-based models (Lu et al., 2021).

Original: Recently, a surge of ImgSGG work starts to explore the long-tailed issuein predicate or triplet classification.
Revised: Recently, a surge of ImgSGG work starts to explore the long-tailed issuein predicate or triplet classification.

Original: Existing methods for debiased SGG can be roughly devided tofour types: class-aware re-sampling (Li et al., 2021;
Revised: Existing methods for debiased SGG can be roughly devided tofour types: class-aware re-sampling (Li et al., 2021;

Original: Desai et al., 2021), loss re-weighting (Lin et al.,2017; Yan et al., 2020; Knyazev et al., 2020), biased-model-based (Tang et al., 2020; Chiou et al.
Revised: Desai et al., 2021), loss re-weighting (Lin et al.,2017; Yan et al., 2020; Knyazev et al., 2020), biased-model-based (Tang et al., 2020; Chiou et al.

Original: Yu et al., 2020) and noisy label correction (Li et al., 2022a;b).
Revised: Yu et al., 2020) and noisy label correction (Li et al., 2022a;b).

Original: Video Scene Graph Generation (VidSGG).
Revised: Video Scene Graph Generation (VidSGG).

Original: Beyond static images, VidSGG aims to detect dynamic visual relations in the videos.
Revised: Beyond static images, VidSGG aims to detect dynamic visual relations in the videos.

Original: Compared to ImgSGG, VidSGG is more challenging becausethey need to consider the spatio-temporal context in adjacent frames.
Revised: Compared to ImgSGG, VidSGG is more challenging becausethey need to consider the spatio-temporal context in adjacent frames.

Original: Based on the formats ofrelation triplet annotations, existing VidSGG work can be categorized into two groups: 1) Trackletbased :
Revised: Based on the formats ofrelation triplet annotations, existing VidSGG work can be categorized into two groups: 1) Trackletbased :

Original: Each graph node is an object tracklet in a video clip (Shang et al., 2017; 2019) or the wholevideo (Liu et al., 2020; Gao et al., 2021; 2022).
Revised: Each graph node is an object tracklet in a video clip (Shang et al., 2017; 2019) or the wholevideo (Liu et al., 2020; Gao et al., 2021; 2022).

Original: 2) Frame-based : Each graph node is an object bboxas in ImgSGG, but these visual relation triplets are dynamic in the whole video sequence (Ji et al.
Revised: 2) Frame-based : Each graph node is an object bboxas in ImgSGG, but these visual relation triplets are dynamic in the whole video sequence (Ji et al.

Original: Feng et al., 2021;
Revised: Feng et al., 2021;

Original: Cong et al., 2021; Li et al., 2022d).
Revised: Cong et al., 2021; Li et al., 2022d).

Original: In this paper, we follow the frame-based
Revised: In this paper, we follow the frame-based

Original: VidSGG setting, and propose the first weakly-supervised VidSGG setting and model.
Revised: VidSGG setting, and propose the first weakly-supervised VidSGG setting and model.

Original: Weakly-Supervised ImgSGG.
Revised: Weakly-Supervised ImgSGG.

Original: A ground-truth visual relation triplet annotation consists of two object locations, their object categories, and pairwise visual relations.
Revised: A ground-truth visual relation triplet annotation consists of two object locations, their object categories, and pairwise visual relations.

Original: Thus, it is extremely laborintensive to obtain large-scale fully-annotated scene graph benchmarks.
Revised: Thus, it is extremely laborintensive to obtain large-scale fully-annotated scene graph benchmarks.

Original: To reduce the labelingcosts, several ImgSGG work proposed to generate scene graphs under weak supervision.
Revised: To reduce the labelingcosts, several ImgSGG work proposed to generate scene graphs under weak supervision.

Original: Currently,there are two main types of weak supervision for ImgSGG: 1) Unlocalized Scene Graphs (Zareianet al., 2020;
Revised: Currently,there are two main types of weak supervision for ImgSGG: 1) Unlocalized Scene Graphs (Zareianet al., 2020;

Original: Shi et al., 2021; Li et al., 2022c):
Revised: Shi et al., 2021; Li et al., 2022c):

Original: It only consists of image-level labels that describethe object and visual relation categories without object locations.
Revised: It only consists of image-level labels that describethe object and visual relation categories without object locations.

Original: Aligned Captions (Shi et al.
Revised: Aligned Captions (Shi et al.

Original: Ye & Kovashka, 2021):
Revised: Ye & Kovashka, 2021):

Original: It provides an entailment caption for the given image.
Revised: It provides an entailment caption for the given image.

Original: In this paper,we only utilize an unlocalized scene graph of a middle frame as the supervision for VidSGG.
Revised: In this paper,we only utilize an unlocalized scene graph of a middle frame as the supervision for VidSGG.

Original: Knowledge Distillation (KD) .
Revised: Knowledge Distillation (KD) .

Original: KD is first used for model compression (Hinton et al., 2015).
Revised: KD is first used for model compression (Hinton et al., 2015).

Original: Then,
Revised: Then,

Original: KD becomes a prevalent method to transfer knowledge from a (larger) teacher model to a (smaller)student model.
Revised: KD becomes a prevalent method to transfer knowledge from a (larger) teacher model to a (smaller)student model.

Original: Benefiting from the soft targets generated by teacher models, the student model caneven outperform their teachers through appropriate training strategies (Zhang et al., 2018; Furlanelloet al., 2018).
Revised: Benefiting from the soft targets generated by teacher models, the student model caneven outperform their teachers through appropriate training strategies (Zhang et al., 2018; Furlanelloet al., 2018).

Original: However, these traditional KD methods are single-teacher.
Revised: However, these traditional KD methods are single-teacher.

Original: To reduce the limitation ofthe data diversity and single-teacher knowledge, recent work tries to distill knowledge from multipleteachers.
Revised: To reduce the limitation ofthe data diversity and single-teacher knowledge, recent work tries to distill knowledge from multipleteachers.

Original: The idea of multi-teacher KD has been widely applied to numerous vision tasks, i .
Revised: The idea of multi-teacher KD has been widely applied to numerous vision tasks, i .

Original: e ., objectdetection (Chen et al., 2017; Wang et al., 2019), visual question answering (Niu & Zhang, 2021;
Revised: e ., objectdetection (Chen et al., 2017; Wang et al., 2019), visual question answering (Niu & Zhang, 2021;

Original: Chen et al., 2022) and ImgSGG (Li et al., 2022c).
Revised: Chen et al., 2022) and ImgSGG (Li et al., 2022c).

Original: In this paper, we propose two relation pseudolabel assignment teachers and use multi-teacher KD to fuse knowledge of multi-teachers.
Revised: In this paper, we propose two relation pseudolabel assignment teachers and use multi-teacher KD to fuse knowledge of multi-teachers.

Original:  P ROPOSED A PPROACH
Revised:  P ROPOSED A PPROACH

Original: Task Definition.
Revised: Task Definition.

Original: In this paper, we propose a new VidSGG task: weakly-supervised VidSGG withsingle-frame weak supervision (SF-VidSGG).
Revised: In this paper, we propose a new VidSGG task: weakly-supervised VidSGG withsingle-frame weak supervision (SF-VidSGG).

Original: Specifically, given an input video V = { I 1 , ..., I T }with T frames, SF-VidSGG aims to generate a scene graph G t for each frame I t , and then stack themalong the time axis to obtain the final video scene graph G = { G 1 , ..., G T } , where G t =
Revised: Specifically, given an input video V = { I 1 , ..., I T }with T frames, SF-VidSGG aims to generate a scene graph G t for each frame I t , and then stack themalong the time axis to obtain the final video scene graph G = { G 1 , ..., G T } , where G t =

Original: ( N t , E t ) ,
Revised: ( N t , E t ) ,

Original: N t and E t denote the set of graph nodes and edges in G t , respectively.
Revised: N t and E t denote the set of graph nodes and edges in G t , respectively.

Original: In the training stage, we onlyuse the weakly-supervision as mentioned in Sec.
Revised: In the training stage, we onlyuse the weakly-supervision as mentioned in Sec.

Original: 1, i .
Revised: 1, i .

Original: e ., the only supervision for SF-VidSGG is anunlocalized scene graph for a middle frame of the video V , denoted as (cid:101)
Revised: e ., the only supervision for SF-VidSGG is anunlocalized scene graph for a middle frame of the video V , denoted as (cid:101)

Original: G m =
Revised: G m =

Original: ( (cid:101) N m , (cid:101) E m ) , and I mis the only frame in the video with annotation.
Revised: ( (cid:101) N m , (cid:101) E m ) , and I mis the only frame in the video with annotation.

Original: 3. M ODEL O VERVIEW
Revised: 3. M ODEL O VERVIEW

Original: To tackle the SF-VidSGG task, we propose a novel framework: Pseudo Label Assignment ( PLA ).
Revised: To tackle the SF-VidSGG task, we propose a novel framework: Pseudo Label Assignment ( PLA ).

Original: The whole pipeline of PLA is illustrated in Figure 2.
Revised: The whole pipeline of PLA is illustrated in Figure 2.

Original: Specifically, it consists of three components:object pseudo label assignment (Obj-PLA) module, relation pseudo label assignment (Rel-PLA)module, and future predicate prediction (FPP) module.
Revised: Specifically, it consists of three components:object pseudo label assignment (Obj-PLA) module, relation pseudo label assignment (Rel-PLA)module, and future predicate prediction (FPP) module.

Original: In the Obj-PLA, we detect object regionproposals for all the frames.
Revised: In the Obj-PLA, we detect object regionproposals for all the frames.

Original: For the frame with weak annotation, we further complete its unlocalized scene graph (cid:101) G to obtain a pseudo localized scene graph G pseudo .
Revised: For the frame with weak annotation, we further complete its unlocalized scene graph (cid:101) G to obtain a pseudo localized scene graph G pseudo .

Original: In the Rel-PLA, we proposetwo relation pseudo label assignment teachers: model-based and model-free.
Revised: In the Rel-PLA, we proposetwo relation pseudo label assignment teachers: model-based and model-free.

Original: The model-basedteacher assigns “soft” predicate pseudo labels P soft by an ImgSGG model, while the model-freeteacher assigns “hard” predicate pseudo labels P hard by some heuristic rules.
Revised: The model-basedteacher assigns “soft” predicate pseudo labels P soft by an ImgSGG model, while the model-freeteacher assigns “hard” predicate pseudo labels P hard by some heuristic rules.

Original: After P soft and P hard
Revised: After P soft and P hard

Original: [Figure]
Revised: [Figure]

Original: O BJECT P ESUDO L ABEL A SSIGNMENT (O BJ -PLA)
Revised: O BJECT P ESUDO L ABEL A SSIGNMENT (O BJ -PLA)

Original: In this module, we aim to generate pseudo object category annotations for all frames.
Revised: In this module, we aim to generate pseudo object category annotations for all frames.

Original: For eachframe I t in input video, we first use an off-the-shelf detector to generate a set of proposal O t .
Revised: For eachframe I t in input video, we first use an off-the-shelf detector to generate a set of proposal O t .

Original: Eachproposal o ti ∈
Revised: Eachproposal o ti ∈

Original: O t has a corresponding bounding box position b ti and an initial object category ˆ c ti ,i .
Revised: O t has a corresponding bounding box position b ti and an initial object category ˆ c ti ,i .

Original: e ., o ti =
Revised: e ., o ti =

Original: ( b ti , ˆ c ti ) .
Revised: ( b ti , ˆ c ti ) .

Original: Since the differences in object category ontology between the pretrained detectorand VidSGG benchmarks, we then map the detected object categories to the object categories in the
Revised: Since the differences in object category ontology between the pretrained detectorand VidSGG benchmarks, we then map the detected object categories to the object categories in the

Original: VidSGG dataset if there is overlapping between their synonyms in WordNet (Miller, 1995), e .
Revised: VidSGG dataset if there is overlapping between their synonyms in WordNet (Miller, 1995), e .

Original: g .,“ woman ” is mapped to “ person ”, “ pizza box ” is mapped to “ box ”.
Revised: g .,“ woman ” is mapped to “ person ”, “ pizza box ” is mapped to “ box ”.

Original: For the annotated frame I m , Obj-PLA has an additional step: matching these detected boundingboxes with entities in the unlocalized scene graph to obtain a pseudo localized scene graph.
Revised: For the annotated frame I m , Obj-PLA has an additional step: matching these detected boundingboxes with entities in the unlocalized scene graph to obtain a pseudo localized scene graph.

Original: Herewe use a category-based strategy: a bounding box and a entity will be matched if they have the samecategory.
Revised: Herewe use a category-based strategy: a bounding box and a entity will be matched if they have the samecategory.

Original: This strategy has a weakness that it cannot distinguish bounding boxes with the sameobject category.
Revised: This strategy has a weakness that it cannot distinguish bounding boxes with the sameobject category.

Original: When the detector finds multiple bounding boxes with the same object category,we just randomly choose one of them to match.
Revised: When the detector finds multiple bounding boxes with the same object category,we just randomly choose one of them to match.

Original: To solve this problem, Li et al. (2022c) utilizespretrained VL models to calculate relevant scores between entities with bounding boxes.
Revised: To solve this problem, Li et al. (2022c) utilizespretrained VL models to calculate relevant scores between entities with bounding boxes.

Original: The finalmatches are decided by the object category labels and the relevant scores.
Revised: The finalmatches are decided by the object category labels and the relevant scores.

Original: We leave these advancedmethods for the future work.
Revised: We leave these advancedmethods for the future work.

Original: 3. R ELATION P ESUDO L ABEL A SSIGNMENT (R EL -PLA)
Revised: 3. R ELATION P ESUDO L ABEL A SSIGNMENT (R EL -PLA)

Original: [Figure]
Revised: [Figure]

Original: After Obj-PLA, we obtain a pseudo localize scene graphfor the annotated Frame and object bboxes for all otherunannotated frames.
Revised: After Obj-PLA, we obtain a pseudo localize scene graphfor the annotated Frame and object bboxes for all otherunannotated frames.

Original: In Rel-PLA, we design two strategies (teachers) to generate pseudo predicate labels for theunannotated frames.
Revised: In Rel-PLA, we design two strategies (teachers) to generate pseudo predicate labels for theunannotated frames.

Original: Model-based Teacher.
Revised: Model-based Teacher.

Original: Model-based Teacher assigns“soft” predicate pseudo labels by an ImgSGG model.
Revised: Model-based Teacher assigns“soft” predicate pseudo labels by an ImgSGG model.

Original: Itfirst obtains pseudo localized scene graphs G pseudo fromthe annotated frame set by Obj-PLA, then train a ImgSGGmodel by these scene graphs.
Revised: Itfirst obtains pseudo localized scene graphs G pseudo fromthe annotated frame set by Obj-PLA, then train a ImgSGGmodel by these scene graphs.

Original: After training, each unannotated frame I t is fed to the network to obtain G t soft =
Revised: After training, each unannotated frame I t is fed to the network to obtain G t soft =

Original: M img ( I t ) , the scene graph for I t .
Revised: M img ( I t ) , the scene graph for I t .

Original: Here we represent thepredicate in G t soft by the category distribution of predicate, besides one-hot labels.
Revised: Here we represent thepredicate in G t soft by the category distribution of predicate, besides one-hot labels.

Original: Inspired by knowledge dis- tillation, we believe that the soft pseudo label can preserve more information than the hard pseudo label (Hinton et al., 2015).
Revised: Inspired by knowledge dis- tillation, we believe that the soft pseudo label can preserve more information than the hard pseudo label (Hinton et al., 2015).

Original: Model-free Teacher.
Revised: Model-free Teacher.

Original: Model-free teacher assigns “hard” pseudo prediacte labels by some heuristicrules.
Revised: Model-free teacher assigns “hard” pseudo prediacte labels by some heuristicrules.

Original: The first step is object sequence matching .
Revised: The first step is object sequence matching .

Original: We adopt the object category and the IoU tomatch the objects detected in the different frames.
Revised: We adopt the object category and the IoU tomatch the objects detected in the different frames.

Original: Two object o ti and o t +1 j in adjacent frames can bematched if they have same object categories and IoU ( b ti , b t +1 j ) >
Revised: Two object o ti and o t +1 j in adjacent frames can bematched if they have same object categories and IoU ( b ti , b t +1 j ) >

Original: η , where η is the IoU matchingthreshold to filter objects.
Revised: η , where η is the IoU matchingthreshold to filter objects.

Original: We frame-by-framely match objects from the middle frame to both ends.
Revised: We frame-by-framely match objects from the middle frame to both ends.

Original: As shown in Figure 3, “ book ” and “ towel ” in I m +2 are not matched with any object in I m
Revised: As shown in Figure 3, “ book ” and “ towel ” in I m +2 are not matched with any object in I m

Original: +1because there is no object detected in I m +1 with category “ book ” and IoU of bounding boxes withcategory “ towel ” in I m
Revised: +1because there is no object detected in I m +1 with category “ book ” and IoU of bounding boxes withcategory “ towel ” in I m

Original: +

Revised: +

Original: and I m +2 is lower than η .
Revised: and I m +2 is lower than η .

Original: The second step is relation propagation .
Revised: The second step is relation propagation .

Original: Wesimply propagate the predicate of a subject-object pair in the annotated frame to the same subjectobject pair in the un-annotated frames.
Revised: Wesimply propagate the predicate of a subject-object pair in the annotated frame to the same subjectobject pair in the un-annotated frames.

Original: 3. F UTURE P REDICATE P REDICTION (FPP) M ODULE
Revised: 3. F UTURE P REDICATE P REDICTION (FPP) M ODULE

Original: Multi-Teacher Fusion Strategy.
Revised: Multi-Teacher Fusion Strategy.

Original: After obtaining P soft and P hard by two teachers, the simplest fusingstrategy is treating each teacher equally and averaging the two pseudo labels.
Revised: After obtaining P soft and P hard by two teachers, the simplest fusingstrategy is treating each teacher equally and averaging the two pseudo labels.

Original: However, this strategyoverlooks the noise in the model-based teacher.
Revised: However, this strategyoverlooks the noise in the model-based teacher.

Original: The model-based teacher sometimes assign wrongpseudo labels due to the incorrect predictions in model M img .
Revised: The model-based teacher sometimes assign wrongpseudo labels due to the incorrect predictions in model M img .

Original: On the other hand, pseudo labelsassigned by the model-free teacher can be seen as the default predictions because these labels havealready appeared in the video.
Revised: On the other hand, pseudo labelsassigned by the model-free teacher can be seen as the default predictions because these labels havealready appeared in the video.

Original: To some extent, the model-free teacher can be seen as a complementwhen the model-based teacher assign a wrong pseudo label.
Revised: To some extent, the model-free teacher can be seen as a complementwhen the model-based teacher assign a wrong pseudo label.

Original: Therefore, a better strategy is assigningadaptive weights for P soft and P hard and then calculating the final pseudo labels by the weighted sum:
Revised: Therefore, a better strategy is assigningadaptive weights for P soft and P hard and then calculating the final pseudo labels by the weighted sum:

Original: [Equation]
Revised: [Equation]

Original: where w i soft and w i hard denotes the weights of the soft and hard predicate labels for video i , respectively.
Revised: where w i soft and w i hard denotes the weights of the soft and hard predicate labels for video i , respectively.

Original: In the FPP module, we calculate w i soft and w i hard by regularity of relation transition.
Revised: In the FPP module, we calculate w i soft and w i hard by regularity of relation transition.

Original: Relation Transition.
Revised: Relation Transition.

Original: Relation transition means how the predicates change between the samesubject-object pairs in different frames.
Revised: Relation transition means how the predicates change between the samesubject-object pairs in different frames.

Original: Mi et al.
Revised: Mi et al.

Original: (2021) has proved that relation transition hasthe temporal tendency, which means we can infer their relation in the next frame according totheir relation in the current frame for the same subject-object pair.
Revised: (2021) has proved that relation transition hasthe temporal tendency, which means we can infer their relation in the next frame according totheir relation in the current frame for the same subject-object pair.

Original: As the example shown in
Revised: As the example shown in

Original: Figure 4, we can find two main conclusions: 1) Relation transitions have some regular patterns,e .
Revised: Figure 4, we can find two main conclusions: 1) Relation transitions have some regular patterns,e .

Original: g ., for subject-object pair “ people-chair ”, relation “ leaning on ” is mainly transferred to“ sitting on ”, but hardly transferred to “ lying on ” and “ standing on ”.
Revised: g ., for subject-object pair “ people-chair ”, relation “ leaning on ” is mainly transferred to“ sitting on ”, but hardly transferred to “ lying on ” and “ standing on ”.

Original: 2) Relation transitions with different subject-object pairs have different regular patterns, e .
Revised: 2) Relation transitions with different subject-object pairs have different regular patterns, e .

Original: g ., the statistical distributions for “ people-chair ” and “ people-food ” are different.
Revised: g ., the statistical distributions for “ people-chair ” and “ people-food ” are different.

Original: Therefore, we can evaluate thequality of predicate pseudo labels by regularity of relation transition in video.
Revised: Therefore, we can evaluate thequality of predicate pseudo labels by regularity of relation transition in video.

Original: Specifically, irregular relation transitions ( e . g .
Revised: Specifically, irregular relation transitions ( e . g .

Original: , “ leaning on ” → “ lying on ” for “ people-chair ”) are of lowquality, then we assign low weights to them.
Revised: , “ leaning on ” → “ lying on ” for “ people-chair ”) are of lowquality, then we assign low weights to them.

Original: Proposed FPP.
Revised: Proposed FPP.

Original: As shown in Figure 5, We take two steps to calculate weights w soft and w hard . 1)
Revised: As shown in Figure 5, We take two steps to calculate weights w soft and w hard . 1)

Original: Predicate prediction : For each predicate p t in I t assigned by the model-based teacher, we predictˆ p t +1 as the predicate that p t will transition to in I t
Revised: Predicate prediction : For each predicate p t in I t assigned by the model-based teacher, we predictˆ p t +1 as the predicate that p t will transition to in I t

Original: +1 .
Revised: +1 .

Original: For each ˆ p t +1 , we find its correspondingpredicate p t +1 in I t
Revised: For each ˆ p t +1 , we find its correspondingpredicate p t +1 in I t

Original: +1 that actually assigned by the model-based teacher.
Revised: +1 that actually assigned by the model-based teacher.

Original: 2) Adapted weight assignment :
Revised: 2) Adapted weight assignment :

Original: We first calculate KL-divergence between ˆ p t +1 and p t +1 as the inconsistency score.
Revised: We first calculate KL-divergence between ˆ p t +1 and p t +1 as the inconsistency score.

Original: Then, we assign low weights to pseudo labels with high inconsistency scores and high weights topseudo labels with low inconsistency scores.
Revised: Then, we assign low weights to pseudo labels with high inconsistency scores and high weights topseudo labels with low inconsistency scores.

Original: In the following, we detailed introduce these two steps.
Revised: In the following, we detailed introduce these two steps.

Original: Predicate Prediction.
Revised: Predicate Prediction.

Original: To predict the future predicates, we construct a relation transition matrix
Revised: To predict the future predicates, we construct a relation transition matrix

Original: T ∈ R N o ∗ N r ∗ N r , where N o , N r are the number of objects and relations in the dataset, respectively.
Revised: T ∈ R N o ∗ N r ∗ N r , where N o , N r are the number of objects and relations in the dataset, respectively.

Original: The relation transition matrix indicates the statistical distribution of relation transition in the VidSGGdataset.
Revised: The relation transition matrix indicates the statistical distribution of relation transition in the VidSGGdataset.

Original: Given a predicate p t in frame I t with object category c
Revised: Given a predicate p t in frame I t with object category c

Original: o , we predict ˆ p t +1 as:
Revised: o , we predict ˆ p t +1 as:

Original: [Equation]
Revised: [Equation]

Original: where id ( c o ) is the id of c o in the vocabulary C go .
Revised: where id ( c o ) is the id of c o in the vocabulary C go .

Original: Then, for each ˆ p t +1 , we try to find its corresponding predicate p t +1 in I
Revised: Then, for each ˆ p t +1 , we try to find its corresponding predicate p t +1 in I

Original: t +1 that actually assigned by the model-based teacher.
Revised: t +1 that actually assigned by the model-based teacher.

Original: Sometimes p t +1 is notexisted because the detector can not detect the subject-object pair of ˆ p t +1 in frame I t
Revised: Sometimes p t +1 is notexisted because the detector can not detect the subject-object pair of ˆ p t +1 in frame I t

Original: +1 .
Revised: +1 .

Original: To handle
Revised: To handle

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: this problem, we adopt the object category and the IoU to filter predicates without corresponding subject-object pair in the next frame, as we did in object sequence matching of the model-freeteacher.
Revised: this problem, we adopt the object category and the IoU to filter predicates without corresponding subject-object pair in the next frame, as we did in object sequence matching of the model-freeteacher.

Original: The set of ˆ p t +1 and p t +1 after filtering in the whole video are denoted as ˆ P filter and P filter ,respectively.
Revised: The set of ˆ p t +1 and p t +1 after filtering in the whole video are denoted as ˆ P filter and P filter ,respectively.

Original: Adapted Weight Assignment.
Revised: Adapted Weight Assignment.

Original: We first calculate the inconsistency score by KL-divergence as below:
Revised: We first calculate the inconsistency score by KL-divergence as below:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Then, we calculate the inconsistency score for P soft , denoted as S softincon .
Revised: Then, we calculate the inconsistency score for P soft , denoted as S softincon .

Original: A high inconsistency scoreindicates that these predicate pseudo labels do not satisfy the regularity of relation transition, so weshould assign low weights to them.
Revised: A high inconsistency scoreindicates that these predicate pseudo labels do not satisfy the regularity of relation transition, so weshould assign low weights to them.

Original: To the end, we define the function as:
Revised: To the end, we define the function as:

Original: [Equation]
Revised: [Equation]

Original: T RAINING O BJECTIVES
Revised: T RAINING O BJECTIVES

Original: FPP module.
Revised: FPP module.

Original: We assume that the relation transition matrix T in FPP module indicates the statisticaldistribution of relation transition in the VidSGG dataset.
Revised: We assume that the relation transition matrix T in FPP module indicates the statisticaldistribution of relation transition in the VidSGG dataset.

Original: However, in the SF-VidSGG task, we cannot count the distribution of relation transition directly because we only have single-frame weaksupervision for each video.
Revised: However, in the SF-VidSGG task, we cannot count the distribution of relation transition directly because we only have single-frame weaksupervision for each video.

Original: An alternative approach is generating pseudo labels by the model-basedteacher and then learning the distribution of these pseudo labels.
Revised: An alternative approach is generating pseudo labels by the model-basedteacher and then learning the distribution of these pseudo labels.

Original: To accomplish this, we formulatethe loss function as below to train the FPP module:
Revised: To accomplish this, we formulatethe loss function as below to train the FPP module:

Original: [Equation]
Revised: [Equation]

Original: Student Model.
Revised: Student Model.

Original: We adopt the KL-divergence loss for training like most knowledge distillationmethods, which can be formulated as follows:
Revised: We adopt the KL-divergence loss for training like most knowledge distillationmethods, which can be formulated as follows:

Original: [Equation]
Revised: [Equation]

Original: ) where P pred denotes the output of the student model, and P final denotes the predicate pseudo labels.
Revised: ) where P pred denotes the output of the student model, and P final denotes the predicate pseudo labels.

Original: Finally, the total training loss of PLA is:
Revised: Finally, the total training loss of PLA is:

Original: [Equation]
Revised: [Equation]

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Due to the limited space, the details about experimental settings (including the evaluation datasetand metrics) and implementation details are left in the appendix.
Revised: Due to the limited space, the details about experimental settings (including the evaluation datasetand metrics) and implementation details are left in the appendix.

Original: 4. A BLATION S TUDIES
Revised: 4. A BLATION S TUDIES

Original: In this section, we validated the effectiveness of our PLA by answering the following questions:
Revised: In this section, we validated the effectiveness of our PLA by answering the following questions:

Original: Q1 :
Revised: Q1 :

Original: How does each component in PLA contribute to the performance on the SF-VidSGG task?
Revised: How does each component in PLA contribute to the performance on the SF-VidSGG task?

Original: Q2 :
Revised: Q2 :

Original: How far are the gaps between PLA and fully-supervised VidSGG approaches?
Revised: How far are the gaps between PLA and fully-supervised VidSGG approaches?

Original: Q3 :
Revised: Q3 :

Original: Can a strongerstudent model help to improve the performance of PLA?
Revised: Can a strongerstudent model help to improve the performance of PLA?

Original: Q4 :
Revised: Q4 :

Original: What is the upper-bound under ourpseudo label assignment training paradigm?
Revised: What is the upper-bound under ourpseudo label assignment training paradigm?

Original: Q5 :
Revised: Q5 :

Original: What is the influence of different annotation framechoosing strategies?
Revised: What is the influence of different annotation framechoosing strategies?

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: E FFECTIVENESS OF E ACH C OMPONENT (Q1)
Revised: E FFECTIVENESS OF E ACH C OMPONENT (Q1)

Original: Settings.
Revised: Settings.

Original: To analyze the importance of each component in PLA, we implemented the straightforward model mentioned in Sec.
Revised: To analyze the importance of each component in PLA, we implemented the straightforward model mentioned in Sec.

Original: 1 as the baseline model.
Revised: 1 as the baseline model.

Original: It first obtained pseudo localized scenegraphs G pseudo from the annotated frame set by Obj-PLA, then trained a STTran † model by thesescene graphs.
Revised: It first obtained pseudo localized scenegraphs G pseudo from the annotated frame set by Obj-PLA, then trained a STTran † model by thesescene graphs.

Original: The gray lines in Table 1 show the results of the baseline model (model A ).
Revised: The gray lines in Table 1 show the results of the baseline model (model A ).

Original: We alsotrained the models with different components of PLA, which is shown in Table 1.
Revised: We alsotrained the models with different components of PLA, which is shown in Table 1.

Original: Effectiveness of Rel-PLA.
Revised: Effectiveness of Rel-PLA.

Original: Compared to the baseline model A , models with model-based teacher ormodel-free teacher (model B and model C ) has a significant improvement.
Revised: Compared to the baseline model A , models with model-based teacher ormodel-free teacher (model B and model C ) has a significant improvement.

Original: The experimental resultsdemonstrate that by utilizing the unannotated frames in the video, model B and model C are ableto outperform the baseline model
Revised: The experimental resultsdemonstrate that by utilizing the unannotated frames in the video, model B and model C are ableto outperform the baseline model

Original: A across almost all the metrics, only except for no constrained
Revised: A across almost all the metrics, only except for no constrained

Original: PredCls-R@50 of model B .
Revised: PredCls-R@50 of model B .

Original: Particularly, model B and model C have outperformed the baseline by5.24% (15.07 v .
Revised: Particularly, model B and model C have outperformed the baseline by5.24% (15.07 v .

Original: s .14.32) and 2.58% (14.69 v .
Revised: s .14.32) and 2.58% (14.69 v .

Original: s .14.32)
Revised: s .14.32)

Original: relatively in terms of R@10 with constraintcriteria for SGDet, respectively.
Revised: relatively in terms of R@10 with constraintcriteria for SGDet, respectively.

Original: This is owing to that the two teachers have the ability to assignhigh-quality pseudo labels to the unannotated frames.
Revised: This is owing to that the two teachers have the ability to assignhigh-quality pseudo labels to the unannotated frames.

Original: Effectiveness of FPP.
Revised: Effectiveness of FPP.

Original: As shown in Table 1, model D and model E are all use both teachers, butmodel E uses the FPP module to set adapted weights.
Revised: As shown in Table 1, model D and model E are all use both teachers, butmodel E uses the FPP module to set adapted weights.

Original: We can observe that model E outperformsmodel D across all the metrics.
Revised: We can observe that model E outperformsmodel D across all the metrics.

Original: Especially, model E has achieved about 2% relative improvementover model D on R@10 metric under the no constraint criteria for SGDet.
Revised: Especially, model E has achieved about 2% relative improvementover model D on R@10 metric under the no constraint criteria for SGDet.

Original: The improvement provesthat FFP has the ability to assign more accurate pseudo labels than the simple fixed weight strategy.
Revised: The improvement provesthat FFP has the ability to assign more accurate pseudo labels than the simple fixed weight strategy.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: G AP BETWEEN PLA AND FULLY S UPERVISED A PPROACHS (Q2)
Revised: G AP BETWEEN PLA AND FULLY S UPERVISED A PPROACHS (Q2)

Original: Settings.
Revised: Settings.

Original: We compared our PLA with three state-of-the-art fully-supervised VidSGG methods:
Revised: We compared our PLA with three state-of-the-art fully-supervised VidSGG methods:

Original: STTran (Cong et al., 2021), APT (Li et al., 2022d), and DSG-DETR (Feng et al., 2021).
Revised: STTran (Cong et al., 2021), APT (Li et al., 2022d), and DSG-DETR (Feng et al., 2021).

Original: Results.
Revised: Results.

Original: As shown in Table 2, we can observe that there is still a gap between the performanceof PLA and other fully-supervised approaches ( e .
Revised: As shown in Table 2, we can observe that there is still a gap between the performanceof PLA and other fully-supervised approaches ( e .

Original: g ., 15.4 v .
Revised: g ., 15.4 v .

Original: s .25.

Revised: s .25.

Original: for STTran in terms of R@10with constraint criteria for SGDet).
Revised: for STTran in terms of R@10with constraint criteria for SGDet).

Original: However, considering that the annotation cost of our methodis extremely lower ( < 5%) than that of full supervision.
Revised: However, considering that the annotation cost of our methodis extremely lower ( < 5%) than that of full supervision.

Original: we think it is still a good start for weaklysupervised frame-level VidSGG.
Revised: we think it is still a good start for weaklysupervised frame-level VidSGG.

Original: We leave the task to reduce the gap as the future work.
Revised: We leave the task to reduce the gap as the future work.

Original: 4.1. E FFECTS OF D IFFERENT S TUDENT M ODELS
Revised: 4.1. E FFECTS OF D IFFERENT S TUDENT M ODELS

Original: (Q3)
Revised: (Q3)

Original: Settings.
Revised: Settings.

Original: We used DSG-DETR as the student model of PLA.
Revised: We used DSG-DETR as the student model of PLA.

Original: Since PLA is agnostic to the specific
Revised: Since PLA is agnostic to the specific

Original: VidSGG architecture, it can be easily incorporated into any advanced VidSGG models.
Revised: VidSGG architecture, it can be easily incorporated into any advanced VidSGG models.

Original: We usedthe same teachers as for STTran to generate pseudo labels for DSG-DETR, then trained a fullysupervised VidSGG model by these pseudo labels.
Revised: We usedthe same teachers as for STTran to generate pseudo labels for DSG-DETR, then trained a fullysupervised VidSGG model by these pseudo labels.

Original: Results.
Revised: Results.

Original: As shown in Table 2, PLA with DSG-DETR also achieves similar performance with
Revised: As shown in Table 2, PLA with DSG-DETR also achieves similar performance with

Original: STTran across all the metrics.
Revised: STTran across all the metrics.

Original: These results show that PLA is a general framework and can easily extend to advanced fully supervised VidSGG methods.
Revised: These results show that PLA is a general framework and can easily extend to advanced fully supervised VidSGG methods.

Original: 4.1. B EST C ASE : G ROUND - TRUTH U NLOCALIZED S CENE G RAPHS (Q4)
Revised: 4.1. B EST C ASE : G ROUND - TRUTH U NLOCALIZED S CENE G RAPHS (Q4)

Original: Settings.
Revised: Settings.

Original: Like most existing SGG methods, PLA has two main steps: the first step is to detect objectsin each frame, and the second step is to predict relations between these objects.
Revised: Like most existing SGG methods, PLA has two main steps: the first step is to detect objectsin each frame, and the second step is to predict relations between these objects.

Original: The performanceof PLA is mainly affected by the first step due to the poor performance of the off-shelf detectorin Action Genome.
Revised: The performanceof PLA is mainly affected by the first step due to the poor performance of the off-shelf detectorin Action Genome.

Original: Under this object detection strategy, the upper-bound performance is not high.
Revised: Under this object detection strategy, the upper-bound performance is not high.

Original: We can train an upper-bound model as follows: first detect objects by Obj-PLA, then assign groundtruth relation labels between them, finally use these labels to train the student model.
Revised: We can train an upper-bound model as follows: first detect objects by Obj-PLA, then assign groundtruth relation labels between them, finally use these labels to train the student model.

Original: We named thisupper-bound model as GTRel.
Revised: We named thisupper-bound model as GTRel.

Original: Results.
Revised: Results.

Original: As shown in Table 3, we observe that GTRel significantly improves the performanceover the baseline.
Revised: As shown in Table 3, we observe that GTRel significantly improves the performanceover the baseline.

Original: Specifically, GTRel has outperformed the baseline by 13.19% (16.73 v .
Revised: Specifically, GTRel has outperformed the baseline by 13.19% (16.73 v .

Original: s .14.72)relatively
Revised: s .14.72)relatively

Original: in terms of R@10 no constraint criteria for SGDet.
Revised: in terms of R@10 no constraint criteria for SGDet.

Original: We also demonstrate that PLA hasability to reduce the gap between the baseline model and GTRel.
Revised: We also demonstrate that PLA hasability to reduce the gap between the baseline model and GTRel.

Original: For example, GTRel improvesby 12.43% (16.10 v .
Revised: For example, GTRel improvesby 12.43% (16.10 v .

Original: s .14.32)
Revised: s .14.32)

Original: relatively over baseline in terms of R@10 with constraint criteria for
Revised: relatively over baseline in terms of R@10 with constraint criteria for

Original: SGDet, but only improves by 4.61% (16.10 v .
Revised: SGDet, but only improves by 4.61% (16.10 v .

Original: s .15.39) relatively over PLA on same metric.
Revised: s .15.39) relatively over PLA on same metric.

Original: Theseresults show that PLA is able to reduce about half performance gap between the baseline modeland the upper-bound model GTRel for the SF-VidSGG task.
Revised: Theseresults show that PLA is able to reduce about half performance gap between the baseline modeland the upper-bound model GTRel for the SF-VidSGG task.

Original: Besides, the method to detect objectsunder unlocalized scene graph setting is not our main contribution.
Revised: Besides, the method to detect objectsunder unlocalized scene graph setting is not our main contribution.

Original: As a general framework, if bettermethods to detect objects are proposed, PLA can easily extend to them.
Revised: As a general framework, if bettermethods to detect objects are proposed, PLA can easily extend to them.

Original: 4.1. E FFECTS OF D IFFERENT A NNOTATION F RAME -C HOOSING S TRATEGIES . (Q5)
Revised: 4.1. E FFECTS OF D IFFERENT A NNOTATION F RAME -C HOOSING S TRATEGIES . (Q5)

Original: Settings.
Revised: Settings.

Original: We compared four simple annotation frame choosing strategies.
Revised: We compared four simple annotation frame choosing strategies.

Original: Specifically, “First”,“Last”, and “Middle” means annotating the “first”, “last”, and “middle” frame with an unlocalizedscene graph in each video, respectively.
Revised: Specifically, “First”,“Last”, and “Middle” means annotating the “first”, “last”, and “middle” frame with an unlocalizedscene graph in each video, respectively.

Original: And “Random” means choosing one frame randomly.
Revised: And “Random” means choosing one frame randomly.

Original: Results.
Revised: Results.

Original: As shown in Table 4, We can observe that model trained by the middle annotation frame hasthe best performance, eg., in terms of R@10 with constraint criteria for SGDet, strategy “Middle”
Revised: As shown in Table 4, We can observe that model trained by the middle annotation frame hasthe best performance, eg., in terms of R@10 with constraint criteria for SGDet, strategy “Middle”

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: outperforms “First”, “Last”, and “Random” by 1.01, 1.14, and 0.48, respectively.
Revised: outperforms “First”, “Last”, and “Random” by 1.01, 1.14, and 0.48, respectively.

Original: It also proves thatthe middle frame usually contains more information.
Revised: It also proves thatthe middle frame usually contains more information.

Original: 4. Q UALITATIVE R ESULTS
Revised: 4. Q UALITATIVE R ESULTS

Original: Pseudo Label Visualization.
Revised: Pseudo Label Visualization.

Original: Figure 6 shows a qualitative comparison of the pseudo labels generated by different variants of PLA.
Revised: Figure 6 shows a qualitative comparison of the pseudo labels generated by different variants of PLA.

Original: Results indicates that the adapted weights given by FPP modulecan generate more accurate pseudo labels than fixed weights.
Revised: Results indicates that the adapted weights given by FPP modulecan generate more accurate pseudo labels than fixed weights.

Original: Specifically, relation in T m +1 (“ notcontacting ”) is different from that in T m (“ touching ” and “ sitting on ”), so the modelfree teacher give totally wrong relations (“ touching ” and “ sitting on ”) and the model-basedteacher predict an almost accurate relation (98% confidence for “ not contacting ”).
Revised: Specifically, relation in T m +1 (“ notcontacting ”) is different from that in T m (“ touching ” and “ sitting on ”), so the modelfree teacher give totally wrong relations (“ touching ” and “ sitting on ”) and the model-basedteacher predict an almost accurate relation (98% confidence for “ not contacting ”).

Original: Compared to fixed weight strategy, the FPP module gives more weight to the model-based teacher (79%v .
Revised: Compared to fixed weight strategy, the FPP module gives more weight to the model-based teacher (79%v .

Original: s .49% confidence for “ not contacting ”) and generates more accurate pseudo labels.
Revised: s .49% confidence for “ not contacting ”) and generates more accurate pseudo labels.

Original: Relation Transition Distribution Visualization.
Revised: Relation Transition Distribution Visualization.

Original: Figure 7 demonstrates the statistical distributionsof relation transition counted by ground truth and learned by PLA.
Revised: Figure 7 demonstrates the statistical distributionsof relation transition counted by ground truth and learned by PLA.

Original: From Figure 7, we can observethat PLA can learn a distribution similar to the ground truth.
Revised: From Figure 7, we can observethat PLA can learn a distribution similar to the ground truth.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: In this paper, we presented the first weakly-supervised VidSGG task with only single-frame weak supervision: SF-VidSGG.
Revised: In this paper, we presented the first weakly-supervised VidSGG task with only single-frame weak supervision: SF-VidSGG.

Original: Unlike VidSGG, SF-VidSGG only require a weak annotaion without bounding box for one frame in each video.
Revised: Unlike VidSGG, SF-VidSGG only require a weak annotaion without bounding box for one frame in each video.

Original: To the end, we proposed a novel and efficient method named
Revised: To the end, we proposed a novel and efficient method named

Original: PLA for SF-VidSGG, which based on pseudo label assignment for the video.
Revised: PLA for SF-VidSGG, which based on pseudo label assignment for the video.

Original: We validated the effectiveness of each component of PLA through extensive experiments.
Revised: We validated the effectiveness of each component of PLA through extensive experiments.

Original: In future work, we wouldlike to explore weaker supervision in VidSGG or more accurate pseudo label assignment.
Revised: In future work, we wouldlike to explore weaker supervision in VidSGG or more accurate pseudo label assignment.

Original: Ethics Statement
Revised: Ethics Statement

Original: Video scene generation models may be utilized in unauthorized human monitoring and surveillance,which leads to some ethical and privacy issues.
Revised: Video scene generation models may be utilized in unauthorized human monitoring and surveillance,which leads to some ethical and privacy issues.

Original: Apart from these general issues that already exist in the video scene graph generation task, our paper has no additional ethical issues
Revised: Apart from these general issues that already exist in the video scene graph generation task, our paper has no additional ethical issues

Original: .Reproducibility Statement
Revised: .Reproducibility Statement

Original: PLA is mainly implemented based on the released code of STTran (Cong et al., 2021) and evaluatedon the Action Genome (AG)
Revised: PLA is mainly implemented based on the released code of STTran (Cong et al., 2021) and evaluatedon the Action Genome (AG)

Original: (Ji et al., 2020) dataset, which is publicly available with a license thatallows free usage for research purposes.
Revised: (Ji et al., 2020) dataset, which is publicly available with a license thatallows free usage for research purposes.

Original: We will also release our code of PLA.
Revised: We will also release our code of PLA.

Original: A A PPENDIX
Revised: A A PPENDIX

Original: This Appendix has the following contents:
Revised: This Appendix has the following contents:

Original: • The details about the valuation dataset are given in Sec.
Revised: • The details about the valuation dataset are given in Sec.

Original: A.1•
Revised: A.1•

Original: The evaluation metrics are given in Sec.
Revised: The evaluation metrics are given in Sec.

Original: A.2.• The implementation details are given in Sec.
Revised: A.2.• The implementation details are given in Sec.

Original: A.3.• The detailed hyperparameter setting are given in Sec.
Revised: A.3.• The detailed hyperparameter setting are given in Sec.

Original: A.4.• Ablation studies on imperfect object detector in Sec.
Revised: A.4.• Ablation studies on imperfect object detector in Sec.

Original: A.5.• Ablation studies on the different proportions of the weakly supervised labels in Sec. 
Revised: A.5.• Ablation studies on the different proportions of the weakly supervised labels in Sec. A.6.• More

Revised: qualitative results are in Sec.

Original: A.6.
Revised: A.7.

Original: A. D ATASET
Revised: A D ATASET

Original: We evaluated our method PLA on the challenging VidSGG benchmark: Action Genome (AG) (Jiet al., 2020).
Revised: We evaluated our method PLA on the challenging VidSGG benchmark: Action Genome (AG) (Jiet al., 2020).

Original: AG is a prevalent VidSGG dataset providing human-object relationships with framelevel annotations.
Revised: AG is a prevalent VidSGG dataset providing human-object relationships with framelevel annotations.

Original: Specifically, AG consists of 9,201 videos, which annotated for 234,253 frames by476,299 bounding boxes of 36 object categories and 1,715,568 instances of 25 relation categories.
Revised: Specifically, AG consists of 9,201 videos, which annotated for 234,253 frames by476,299 bounding boxes of 36 object categories and 1,715,568 instances of 25 relation categories.

Original: All relation categories in AG are split into three types: 3 attention relations, 6 spatial relations and16 contact relations.
Revised: All relation categories in AG are split into three types: 3 attention relations, 6 spatial relations and16 contact relations.

Original: Each subject-object pair has one attention relation but may has multiple spatialor contact relations.
Revised: Each subject-object pair has one attention relation but may has multiple spatialor contact relations.

Original: In our SF-VidSGG setting, we used the official splits as fully-supervised work,i .
Revised: In our SF-VidSGG setting, we used the official splits as fully-supervised work,i .

Original: e ., 7,464 videos for training, 1,737 videos for testing.
Revised: e ., 7,464 videos for training, 1,737 videos for testing.

Original: For each video in the training set, we onlyused the unlocalized scene graph of the middle frame as supervision.
Revised: For each video in the training set, we onlyused the unlocalized scene graph of the middle frame as supervision.

Original: A. E VALUATION M ETRICS
Revised: A. E VALUATION M ETRICS

Original: Following the fully-supervised setting, we evaluated PLA on three different settings: 1) Scene Graph
Revised: Following the fully-supervised setting, we evaluated PLA on three different settings: 1) Scene Graph

Original: Detection ( SGDet ): Given a video, the model needs to detect the objects and predict predicatecategories of object pairs for all frame.
Revised: Detection ( SGDet ): Given a video, the model needs to detect the objects and predict predicatecategories of object pairs for all frame.

Original: In SGDet, an object is considered to be successfully detectedif the IoU between the predicted bbox and its ground-truth bbox is larger than 0.5.
Revised: In SGDet, an object is considered to be successfully detectedif the IoU between the predicted bbox and its ground-truth bbox is larger than 0.5.

Original: 2) Scene Graph
Revised: 2) Scene Graph

Original: Classification ( SGCls ): Given a video and object bboxes, the model needs to predict object andpredicate categories for all frames.
Revised: Classification ( SGCls ): Given a video and object bboxes, the model needs to predict object andpredicate categories for all frames.

Original: Predicate Classification ( PredCls ): Given a video, object bboxes and object labels, the model needs to predict predicate categories for all frames.
Revised: Predicate Classification ( PredCls ): Given a video, objectbboxes and object labels, the model needs to predict predicate categories for all frames.

Original: Since theground-truth bboxes are not provided in SF-VidSGG, we only trained models under SGDet settings,then evaluated them under these three settings.
Revised: Since the ground-truth bboxes are not provided in SF-VidSGG, we only trained models under SGDet settings,then evaluated them under these three settings.

Original: We used Recall@K ( R@K , K = [10,20,50]) as ourevaluation metric, which measures the ratio of the ground-truth relation triplets among the top-Kpredicted relation triplets.
Revised: We used Recall@K ( R@K , K = [10,20,50]) as ourevaluation metric, which measures the ratio of the ground-truth relation triplets among the top-Kpredicted relation triplets.

Original: Furthermore, we adopt two typical strategies to generate dynamic scenegraphs: 1) With Constraint : It only allows each subject-object pair to have at most one predicate.2) No Constraint 
Revised: Furthermore, we adopt two typical strategies to generate dynamic scenegraphs : 1) With Constraint : It only allows each subject-object pair to have at most one predicate. No Constraint :

Original: It allows each subject-object pair to have multiple predicates.
Revised: It allows each subject-object pair to have multiple predicates.

Original: A. I MPLEMENTATION D ETAILS
Revised: A. I MPLEMENTATION D ETAILS

Original: In the Obj-PLA module, we used the pretrained VinVL (Li et al., 2020; Zhang et al., 2021) withbackbone ResNeXt-152 C4 as the off-the-shelf detector.
Revised: In the Obj-PLA module, we used the pretrained VinVL (Li et al., 2020; Zhang et al., 2021) withbackbone ResNeXt-152 C4 as the off-the-shelf detector.

Original: Specifically, it is pre-trained on multipledetection benchmarks, including COCO (Lin et al., 2014), OpenImages (Kuznetsova et al., 2020),
Revised: Specifically, it is pre-trained on multipledetection benchmarks, including COCO (Lin et al., 2014), OpenImages (Kuznetsova et al., 2020),

Original: Objects365 (Shao et al., 2019), and Visual Genome (Krishna et al., 2017).
Revised: Objects365 (Shao et al., 2019), and Visual Genome (Krishna et al., 2017).

Original: This detector is capableof detecting 1,594 general object categories.
Revised: This detector is capableof detecting 1,594 general object categories.

Original: We kept objects which confidence is higher than 0.2and extracted the 2048-D region features from the detector.
Revised: We kept objects which confidence is higher than 0.2and extracted the 2048-D region features from the detector.

Original: In the Rel-PLA module, we modified
Revised: In the Rel-PLA module, we modified

Original: STTran (Cong et al., 2021) to an ImgSGG model that process one frame at a time, denoted as
Revised: STTran (Cong et al., 2021) to an ImgSGG model that process one frame at a time, denoted as

Original: STTran † .
Revised: STTran † .

Original: We used STTran † as the teacher model to assign pseudo soft labels in Rel-PLA.
Revised: We used STTran † as the teacher model to assign pseudo soft labels in Rel-PLA.

Original: For themodel-free teacher, we set the IOU matching threshold η = 0 .
Revised: For themodel-free teacher, we set the IOU matching threshold η = 0 .

Original: In FPP module, we set the initiallearning rate 1 e − 3 .
Revised: In FPP module, we set the initiallearning rate 1 e − 3 .

Original: We used STTran as the student model, and followed the same training settings( e .
Revised: We used STTran as the student model, and followed the same training settings( e .

Original: g ., learning rate and batch size) of the original paper.
Revised: g ., learning rate and batch size) of the original paper.

Original: A. H YPERPARAMETER S ETTINGS
Revised: A. H YPERPARAMETER S ETTINGS

Original: We analyzed the influence of the hyperparameter η , which denote the IoU matching threshold in
Revised: We analyzed the influence of the hyperparameter η , which denote the IoU matching threshold in

Original: Rel-PLA.
Revised: Rel-PLA.

Original: We set different η for the model only uses model-free teacher (model C ).
Revised: We set different η for the model only uses model-free teacher (model C ).

Original: All results are in
Revised: All results are in

Original: Table 5.
Revised: Table 5.

Original: Particularly, when the threshold η is set to 0, we match the objects in different frames onlyby the object category without the IoU of the bounding boxes in adjacent frames.
Revised: Particularly, when the threshold η is set to 0, we match the objects in different frames onlyby the object category without the IoU of the bounding boxes in adjacent frames.

Original: [Table]
Revised: [Table]

Original: Results.
Revised: Results.

Original: As shown in Table
Revised: As shown in Table

Original: [Table]
Revised: [Table]

Original: we can observe that the model with η = 0 has a weak performancedue to more wrong objects matching without IoU information.
Revised: we can observe that the model with η = 0 has a weak performancedue to more wrong objects matching without IoU information.

Original: And the model with η = 0 .
Revised: And the model with η = 0 .

Original: 5 slightlyoutperforms the model with η = 0 .
Revised: 5 slightlyoutperforms the model with η = 0 .

Original: Therefore, we set η = 0.5 for all experiments.
Revised: Therefore, we set η = 0.5 for all experiments.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: A. E FFECTS OF I MPERFECT O BJECT D ETECTORS
Revised: A. E FFECTS OF I MPERFECT O BJECT D ETECTORS

Original: Settings.
Revised: Settings.

Original: To simulate an imperfect object detector, we randomly dropped a certain proportion ofbounding boxes given by the object detector.
Revised: To simulate an imperfect object detector, we randomly dropped a certain proportion ofbounding boxes given by the object detector.

Original: Without loss of generality, we set five dropped proportion as 0%, 10%, 20%, 40% and 80%.
Revised: Without loss of generality, we set five dropped proportion as 0%, 10%, 20%, 40% and 80%.

Original: Results.
Revised: Results.

Original: As shown in Table 6, we can observe that the bigger dropped proportion, the worse theperformance of PLA, eg.
Revised: As shown in Table 6, we can observe that the bigger dropped proportion, the worse theperformance of PLA, eg.

Original: the setting with 0% dropped proportion outperforms settings with 10%,20%, 40% and 80% dropped proportion by 1.23, 2.21, 3.91 and 10.55, respectively.
Revised: the setting with 0% dropped proportion outperforms settings with 10%,20%, 40% and 80% dropped proportion by 1.23, 2.21, 3.91 and 10.55, respectively.

Original: When dropping10%, 20% and 40% bboxes, the performance has not decreased much, but the performance decreasedsignificantly if dropping 80% bboxes.
Revised: When dropping10%, 20% and 40% bboxes, the performance has not decreased much, but the performance decreasedsignificantly if dropping 80% bboxes.

Original: Results show that PLA is affected by the preformance of theobject detector, but it is the weakness for all two-stage (first detect objects, then predict relationsbetween them) SGG method.
Revised: Results show that PLA is affected by the preformance of theobject detector, but it is the weakness for all two-stage (first detect objects, then predict relationsbetween them) SGG method.

Original: A. E FFECTS OF THE S IZE OF W EAKLY S UPERVISED L ABELS
Revised: A. E FFECTS OF THE S IZE OF W EAKLY S UPERVISED L ABELS

Original: Settings.
Revised: Settings.

Original: Although PLA is designed for single-frame weak supervision, it can be easily extendedto multi-frame weak supervision.
Revised: Although PLA is designed for single-frame weak supervision, it can be easily extendedto multi-frame weak supervision.

Original: For the model-based teacher, we just train the model with moreframes.
Revised: For the model-based teacher, we just train the model with moreframes.

Original: For the model-free teacher, we take each annotated frame as a starting point to propagaterelations.
Revised: For the model-free teacher, we take each annotated frame as a starting point to propagaterelations.

Original: Without loss of generality, we set four annotated proportion as 10%, 20%, 40% and 80%.
Revised: Without loss of generality, we set four annotated proportion as 10%, 20%, 40% and 80%.

Original: We annotated frame evenly in temporal axis, for example, if a 20-frame video with 20% annotatedproportion, we will annotate its 1st, 7th, 14th and 20th frame.
Revised: We annotated frame evenly in temporal axis, for example, if a 20-frame video with 20% annotatedproportion, we will annotate its 1st, 7th, 14th and 20th frame.

Original: Results.
Revised: Results.

Original: As shown in 7, we can observe that the more annotated frames, the better the performanceof PLA, e.g., model with 80% annotation frames outperforms models with 10%, 20% and 40%annotation frames 1.16, 0.26 and 0.03.
Revised: As shown in 7, we can observe that the more annotated frames, the better the performanceof PLA, e.g., model with 80% annotation frames outperforms models with 10%, 20% and 40%annotation frames 1.16, 0.26 and 0.03.

Original: It proves that PLA can easily adapt to multi-frame weaksupervision setting.
Revised: It proves that PLA can easily adapt to multi-frame weaksupervision setting.

Revised:  A. M ORE QUALITATIVE RESULTS ON A CTION G ENOME DATASET

Revised: More qualitative results of PLA on Action Genome are shown in Figure 8.

Revised: From Figure 8, we canobserve that PLA has the person

Revised: Standing on

Revised: Holding shoe floor person

Revised: Standing on

Revised: Not contanting shoe floor person

Revised: Standing on

Revised: Not contanting shoe floor shoe

Revised: Not contanting person

Revised: Sitting on

Revised: Not contacting table chair person

Revised: Sitting on

Revised: Not contacting table chair bag

Revised: Touching bag

Revised: Holding person person

Revised: Not contacting

Revised: Touching laptop table person

Revised: Not contacting

Revised: Touching laptop table chair

Revised: Sitting on person

Revised: Touching

Revised: Sitting on chair table table

Revised: Touching