Original: V IDEO S CENE G RAPH G ENERATION FROM S INGLE -F RAME W EAK S UPERVISION
Revised: V IDEO S CENE G RAPH G ENERATION FROM S INGLE -F RAME W EAK S UPERVISION

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Video scene graph generation (VidSGG) aims to generate a sequence of graphstructure representations for the given video.
Revised: Video scene graph generation (VidSGG) aims to generate a sequence of graphstructure representations for the given video.

Original: However, all existing VidSGG methods are fully-supervised , i .
Revised: However, all existing VidSGG methods are fully-supervised , i .

Original: e ., they need dense and costly manual annotations.
Revised: e ., they need dense and costly manual annotations.

Original: Inthis paper, we propose the first weakly-supervised VidSGG task with only singleframe weak supervision: SF-VidSGG.
Revised: In this paper, we propose the first weakly-supervised VidSGG task with only singleframe weak supervision: SF-VidSGG.

Original: By “weakly-supervised”, we mean that
Revised: By “weakly-supervised”, we mean that

Original: SF-VidSGG relaxes the training supervision from two different levels: 1) It onlyprovides single-frame annotations instead of all-frame annotations.
Revised: SF-VidSGG relaxes the training supervision from two different levels: 1) It only provides single-frame annotations instead of all-frame annotations.

Original: The singleframe ground-truth annotation is still a weak image SGG annotation, i .
Revised: The singleframe ground-truth annotation is still a weak image SGG annotation, i .

Original: e
Revised: e

Original: ., an unlocalized scene graph.
Revised: ., an unlocalized scene graph.

Original: To solve this new task, we also propose a novel Pseudo
Revised: To solve this new task, we also propose a novel Pseudo

Original: Label Assignment based method, dubbed as PLA .
Revised: Label Assignment based method, dubbed as PLA .

Original: PLA is a two-stage method,which generates pseudo visual relation annotations for the given video at the firststage, and then trains a fully-supervised VidSGG model with these pseudo labels.
Revised: PLA is a two-stage method, which generates pseudo visual relation annotations for the given video at the first stage, and then trains a fully-supervised VidSGG model with these pseudo labels.

Original: Specifically, PLA consists of three modules: an object PLA module, a predicate
Revised: Specifically, PLA consists of three modules: an object PLA module, a predicate

Original: PLA module, and a future predicate prediction (FPP) module.
Revised: PLA module, and a future predicate prediction (FPP) module.

Original: Firstly, in the object PLA, we localize all objects for every frame.
Revised: Firstly, in the object PLA, we localize all objects for every frame.

Original: Then, in the predicate PLA,we design two different teachers to assign pseudo predicate labels.
Revised: Then, in the predicate PLA, we design two different teachers to assign pseudo predicate labels.

Original: Lastly, in the
Revised: Lastly, in the

Original: FPP module, we fusion these two predicate pseudo labels by the regularity of relation transition in videos.
Revised: FPP module, we fusion these two predicate pseudo labels by the regularity of relation transition in videos.

Original: Extensive ablations and results on the benchmark Action
Revised: Extensive ablations and results on the benchmark Action

Original: Genome have demonstrated the effectiveness of our PLA.
Revised: Genome have demonstrated the effectiveness of our PLA.

Revised: Code is available at https://github.com/zjucsq/PLA.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Scene graph (Johnson et al., 2015) is a type of visually-aligned graph-structured representation thatsummarizes all the object instances ( e .
Revised: Scene graph (Johnson et al., 2015) is a type of visually-aligned graph-structured representation that summarizes all the object instances ( e .

Original: g ., “ person ”, “ chair ”) as nodes and their pairwise visualrelations (or predicates, e .
Revised: g ., “ person ”, “ chair ”) as nodes and their pairwise visual relations (or predicates, e .

Original: g .
Revised: g .

Original: , “ sitting on ”) as directed edges.
Revised: , “ sitting on ”) as directed edges.

Original: As a bridge to connect the visionand language modalities, scene graphs have been widely used in many different downstream visuallanguage tasks, such as visual captioning (Yang et al., 2019; 2020), grounding (Jing et al. 2020),question answering (Hudson & Manning, 2019), and retrieval (Johnson et al., 2015).
Revised: As a bridge to connect the vision and language modalities, scene graphs have been widely used in many different downstream visuallanguage tasks, such as visual captioning (Yang et al., 2019; 2020), grounding (Jing et al., 2020), question answering (Hudson & Manning, 2019), and retrieval (Johnson et al., 2015).

Original: Early Scene Graph Generation (SGG) work mainly focuses on generating scene graphs for the givenimage , dubbed as ImgSGG
Revised: Early Scene Graph Generation (SGG) work mainly focuses on generating scene graphs for the given image , dubbed as ImgSGG

Original: (Xu et al., 2017; Zellers et al., 2018; Chen et al., 2019).
Revised: (Xu et al., 2017; Zellers et al., 2018; Chen et al., 2019).

Original: However,due to its static nature, ImgSGG fails to represent numerous dynamic visual relations that takeplace over a period of time, such as “ walking with ” and “ running away ” (vs. static relation“ standing ”).
Revised: However, due to its static nature, ImgSGG fails to represent numerous dynamic visual relations that take place over a period of time, such as “ walking with ” and “ running away ” (vs. static relation “ standing ”).

Original: Meanwhile, it is hard or impossible to identify these dynamic visual relations withonly a single frame, because these visual relations can only be well classified by considering thetemporal context.
Revised: Meanwhile, it is hard or impossible to identify these dynamic visual relations with only a single frame, because these visual relations can only be well classified by considering the temporal context.

Original: Therefore, another more meaningful but challenging video-based SGG task wasproposed: VidSGG (Shang et al., 2017; 2019).
Revised: Therefore, another more meaningful but challenging video-based SGG task was proposed: VidSGG (Shang et al., 2017; 2019).

Original: Since the complex and dense annotations of a scene graph ( cf. , Figure 1(a)), fully-supervised SGGmethods always require lots of manual annotations, and the case is even worse for video data.
Revised: Since the complex and dense annotations of a scene graph ( cf. , Figure 1(a)), fully-supervised SGG methods always require lots of manual annotations, and the case is even worse for video data.

Original: Meanwhile, several prior SGG works (Li et al., 2022a) have found that even carefully manually-annotatedlabels are still quite noisy, i .
Revised: Meanwhile, several prior SGG works (Li et al., 2022a) have found that even carefully manually-annotated labels are still quite noisy, i .

Original: e ., the annotated positive labels may not be the best matched, and numerous negative labels are just missing annotated.
Revised: e ., the annotated positive labels may not be the best matched, and numerous negative labels are just missing annotated.

Original: Thus, a surge of recent ImgSGG work (Zareianet al., 2020; Zhong et al., 2021; Shi et al., 2021; Li et al., 2022c) start to generate scene graphs forimages with only weak supervision.
Revised: Thus, a surge of recent ImgSGG work (Zareian et al., 2020; Zhong et al., 2021; Shi et al., 2021; Li et al., 2022c) start to generate scene graphs for images with only weak supervision.

Original: By “weak supervision”, we mean that the annotations for modeltraining are not complete localized scene graphs.
Revised: By “weak supervision”, we mean that the annotations for model training are not complete localized scene graphs.

Original: For example, a typical type of weak supervision is unlocalized scene graphs .
Revised: For example, a typical type of weak supervision is unlocalized scene graphs .

Original: As illustrated in Figure 1(b), an unlocalized scene graph only containsimage-level object and relation categories without corresponding object bounding boxes (bboxes).
Revised: As illustrated in Figure 1(b), an unlocalized scene graph only contains image-level object and relation categories without corresponding object bounding boxes (bboxes).

Original: [Figure]
Revised: [Figure]

Original: floor towel

Original: Although recent weakly-supervised ImgSGGhas achieved good performance and receivedunprecedented attention, to the best our knowledge, there is no existing work about generating video scene graphs from weak supervision.
Revised: Although recent weakly-supervised ImgSGG has achieved good performance and received unprecedented attention, to the best our knowledge, there is no existing work about generating video scene graphs from weak supervision.

Original: To put forward the research on this criticaltopic, we propose the first weakly-supervised
Revised: To put forward the research on this critical topic, we propose the first weakly-supervised

Original: VidSGG task with single-frame weak supervision, called SF-VidSGG .
Revised: VidSGG task with single-frame weak supervision, called SF-VidSGG .

Original: Specifically, given aninput video, SF-VidSGG aims to generate a localized scene graph for each frame in the video,but the only supervision for training is an unlocalized scene graph for the middle frame ofeach training video .
Revised: Specifically, given an input video, SF-VidSGG aims to generate a localized scene graph for each frame in the video, but the only supervision for training is an unlocalized scene graph for the middle frame of each training video .

Original: As the example shown in
Revised: As the example shown in

Original: Figure 1(c), the supervision is an unlocalizedscene graph for the third frame.
Revised: Figure 1(c), the supervision is an unlocalized scene graph for the third frame.

Original: Obviously, SFVidSGG task tries to relieve the intensive annotation issues from two levels: 1) Video-level:
Revised: Obviously, SFVidSGG task tries to relieve the intensive annotation issues from two levels: 1) Video-level:

Original: For each video, we only need single-frame annotations instead of all-frame annotations as the fullysupervised setting ( i .
Revised: For each video, we only need single-frame annotations instead of all-frame annotations as the fullysupervised setting ( i .

Original: e ., reduce the number of annotated frames).
Revised: e ., reduce the number of annotated frames).

Original: 2) Frame-level: The single-frameannotation is an unlocalized scene graph ( i .
Revised: 2) Frame-level: The single-frame annotation is an unlocalized scene graph ( i .

Original: e ., avoid annotating object locations).
Revised: e ., avoid annotating object locations).

Original: A straightforward solution for SF-VidSGG is: Using all the weakly annotated frames to train aweakly-supervised ImgSGG model first, and then detecting scene graphs on each frame with the
Revised: A straightforward solution for SF-VidSGG is: Using all the weakly annotated frames to train a weakly-supervised ImgSGG model first, and then detecting scene graphs on each frame with the

Original: ImgSGG model.
Revised: ImgSGG model.

Original: Apparently, this naive ImgSGG-based method has overlooked the temporal context in the video data.
Revised: Apparently, this naive ImgSGG-based method has overlooked the temporal context in the video data.

Original: To this end, we propose a novel Pseudo Label Assignment strategy PLA ,which can serve as the first baseline for SF-VidSGG.
Revised: To this end, we propose a novel Pseudo Label Assignment strategy PLA , which can serve as the first baseline for SF-VidSGG.

Original: Since PLA is agnostic to different VidSGGarchitectures, it can be easily incorporated into any advanced VidSGG model.
Revised: Since PLA is agnostic to different VidSGG architectures, it can be easily incorporated into any advanced VidSGG model.

Original: Specifically, PLAdecouples the problem into two steps: The first step is to assign a pseudo localized scene graphfor every frame in the video, and the second step is to train a fully-supervised VidSGG model bythe pseudo localized scene graphs.
Revised: Specifically, PLA decouples the problem into two steps: The first step is to assign a pseudo localized scene graph for every frame in the video, and the second step is to train a fully-supervised VidSGG model by the pseudo localized scene graphs.

Original: PLA consists of three modules: object pseudo label assignmentmodule (Obj-PLA), relation pseudo label assignment module (Rel-PLA), and future predicate prediction module (FPP).
Revised: PLA consists of three modules: object pseudo label assignment module (Obj-PLA), relation pseudo label assignment module (Rel-PLA), and future predicate prediction module (FPP).

Original: In the Obj-PLA module, we detect object region proposals for all the frames.
Revised: In the Obj-PLA module, we detect object region proposals for all the frames.

Original: In the Rel-PLA module, we propose two relation pseudo label assignment teachers and they generate two different pseudo labels for each frame.
Revised: In the Rel-PLA module, we propose two relation pseudo label assignment teachers and they generate two different pseudo labels for each frame.

Original: In the FPP module, we determine adapted weightsto fuse these labels from two different teachers.
Revised: In the FPP module, we determine adapted weights to fuse these labels from two different teachers.

Original: To effectively obtain optimal adapted weights forfusing different teacher knowledge, the FPP module exploits the temporal context based on the relation transition in videos.
Revised: To effectively obtain optimal adapted weights for fusing different teacher knowledge, the FPP module exploits the temporal context based on the relation transition in videos.

Original: The relation transition means how the predicates change between the samesubject-object pairs in different frames.
Revised: The relation transition means how the predicates change between the same subject-object pairs in different frames.

Original: In summary, we make three main technique contributions in this paper:
Revised: In summary, we make three main technique contributions in this paper:

Original: We propose the first weakly-supervised VidSGG task.
Revised: We propose the first weakly-supervised VidSGG task.

Original: Compared to its fully-supervised counterpart, we try to mitigate the intensive annotations from both video-level and frame-level.
Revised: Compared to its fully-supervised counterpart, we try to mitigate the intensive annotations from both video-level and frame-level.

Original: We propose a novel method PLA for SF-VidSGG.
Revised: We propose a novel method PLA for SF-VidSGG.

Original: It utilizes two teachers to assign pseudo labelfor unlabeled data and refines the pseudo labels from both teachers by knowledge distillation.3.
Revised: It utilizes two teachers to assign pseudo label for unlabeled data and refines the pseudo labels from both teachers by knowledge distillation.

Original: We propose a future predicate prediction module that leverages temporal dependencies in video.
Revised: We propose a future predicate prediction module that leverages temporal dependencies in video.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Image Scene Graph Generation (ImgSGG).
Revised: Image Scene Graph Generation (ImgSGG).

Original: ImgSGG aims to generate semantic graph structures— scene graphs — as the representation of images.
Revised: ImgSGG aims to generate semantic graph structures — scene graphs — as the representation of images.

Original: In each scene graph, every node represents anobject instance and every edge represents a visual relation between two objects.
Revised: In each scene graph, every node represents an object instance and every edge represents a visual relation between two objects.

Original: Early ImgSGGmethods always directly predict all pairwise visual relations (Lu et al., 2016; Zhang et al., 2017).
Revised: Early ImgSGG methods always directly predict all pairwise visual relations (Lu et al., 2016; Zhang et al., 2017).

Original: Later, ImgSGG methods mainly focus on designing different context encoding architectures, suchas message passing (Xu et al., 2017; Li et al., 2017), recurrent network (Zellers et al., 2018), treestructure model (Tang et al., 2019), agent communication (Chen et al., 2019), and Transformer-based models (Lu et al., 2021).
Revised: Later, ImgSGG methods mainly focus on designing different context encoding architectures, such as message passing (Xu et al., 2017; Li et al., 2017), recurrent network (Zellers et al., 2018), treestructure model (Tang et al., 2019), agent communication (Chen et al., 2019), and Transformer-based models (Lu et al., 2021).

Original: Recently, a surge of ImgSGG work starts to explore the long-tailed issuein predicate or triplet classification.
Revised: Recently, a surge of ImgSGG work starts to explore the long-tailed issue in predicate or triplet classification.

Original: Existing methods for debiased SGG can be roughly devided tofour types: class-aware re-sampling (Li et al., 2021;  Desai et al., 2021), loss re-weighting (Lin et al.,2017; Yan et al., 2020; Knyazev et al., 2020), biased-model-based (Tang et al., 2020; Chiou et al. Yu et al., 2020) and noisy label correction (Li et al., 2022a;b).
Revised: Existing methods for debiased SGG can be roughly devided to four types: class-aware re-sampling (Li et al., 2021; 2022d; Desai et al., 2021), loss re-weighting (Lin et al., 2017; Yan et al., 2020; Knyazev et al., 2020), biased-model-based (Tang et al., 2020; Chiou et al., 2021; Yu et al., 2021) and noisy label correction (Li et al., 2022a;b).

Original: Video Scene Graph Generation (VidSGG).
Revised: Video Scene Graph Generation (VidSGG).

Original: Beyond static images, VidSGG aims to detect dynamic visual relations in the videos.
Revised: Beyond static images, VidSGG aims to detect dynamic visual relations in the videos.

Original: Compared to ImgSGG, VidSGG is more challenging becausethey need to consider the spatio-temporal context in adjacent frames.
Revised: Compared to ImgSGG, VidSGG is more challenging because they need to consider the spatio-temporal context in adjacent frames.

Original: Based on the formats ofrelation triplet annotations, existing VidSGG work can be categorized into two groups: 1) Trackletbased :
Revised: Based on the formats of relation triplet annotations, existing VidSGG work can be categorized into two groups: 1) Trackletbased :

Original: Each graph node is an object tracklet in a video clip (Shang et al., 2017; 2019) or the wholevideo (Liu et al., 2020; Gao et al., 2021; 2022).
Revised: Each graph node is an object tracklet in a video clip (Shang et al., 2017; 2019) or the whole video (Liu et al., 2020; Gao et al., 2021; 2022; 2023).

Original: 2) Frame-based : Each graph node is an object bboxas in ImgSGG, but these visual relation triplets are dynamic in the whole video sequence (Ji et al.
Revised: 2) Frame-based : Each graph node is an object bbox as in ImgSGG, but these visual relation triplets are dynamic in the whole video sequence (Ji et al., 2020;

Original: Feng et al., 2021;
Revised: Feng et al., 2021;

Original: Cong et al., 2021; Li et al., 2022d).
Revised: Cong et al., 2021; Li et al., 2022e).

Original: In this paper, we follow the frame-based VidSGG setting, and propose the first weakly-supervised VidSGG setting and model.
Revised: In this paper, we follow the frame-based VidSGG setting, and propose the first weakly-supervised VidSGG setting and model.

Original: Weakly-Supervised ImgSGG.
Revised: Weakly-Supervised ImgSGG.

Original: A ground-truth visual relation triplet annotation consists of two object locations, their object categories, and pairwise visual relations.
Revised: A ground-truth visual relation triplet annotation consists of two object locations, their object categories, and pairwise visual relations.

Original: Thus, it is extremely laborintensive to obtain large-scale fully-annotated scene graph benchmarks.
Revised: Thus, it is extremely laborintensive to obtain large-scale fully-annotated scene graph benchmarks (Zhang et al., 2020; Liu et al., 2022; Song et al., 2023; Liu et al., 2016).

Original: To reduce the labelingcosts, several ImgSGG work proposed to generate scene graphs under weak supervision.
Revised: To reduce the labeling costs, several ImgSGG work proposed to generate scene graphs under weak supervision.

Original: Currently,there are two main types of weak supervision for ImgSGG: 1) Unlocalized Scene Graphs (Zareianet al., 2020; Shi et al., 2021;Li et al., 2022c):
Revised: Currently, there are two main types of weak supervision for ImgSGG: 1) Unlocalized Scene Graphs (Zareian et al., 2020; Shi et al., 2021;

Original:  It only consists of image-level labels that describethe object and visual relation categories without object locations.
Revised: Li et al., 2022c): It only consists of image-level labels that describe the object and visual relation categories without object locations.

Original: Aligned Captions (Shi et al. Ye & Kovashka, 2021):
Revised: Aligned Captions (Shi et al., 2021; Ye & Kovashka, 2021):

Original: It provides an entailment caption for the given image.
Revised: It provides an entailment caption for the given image.

Original: In this paper,we only utilize an unlocalized scene graph of a middle frame as the supervision for VidSGG.
Revised: In this paper, we only utilize an unlocalized scene graph of a middle frame as the supervision for VidSGG.

Original: Knowledge Distillation (KD) .
Revised: Knowledge Distillation (KD) .

Original: KD is first used for model compression (Hinton et al., 2015).
Revised: KD is first used for model compression (Hinton et al., 2015).

Original: Then,
Revised: Then,

Original: KD becomes a prevalent method to transfer knowledge from a (larger) teacher model to a (smaller)student model.
Revised: KD becomes a prevalent method to transfer knowledge from a (larger) teacher model to a (smaller) student model.

Original: Benefiting from the soft targets generated by teacher models, the student model caneven outperform their teachers through appropriate training strategies (Zhang et al., 2018; Furlanelloet al., 2018).
Revised: Benefiting from the soft targets generated by teacher models, the student model can even outperform their teachers through appropriate training strategies (Zhang et al., 2018; Furlanello et al., 2018).

Original: However, these traditional KD methods are single-teacher.
Revised: However, these traditional KD methods are single-teacher.

Original: To reduce the limitation ofthe data diversity and single-teacher knowledge, recent work tries to distill knowledge from multipleteachers.
Revised: To reduce the limitation of data diversity and single-teacher knowledge, recent work tries to distill knowledge from multiple teachers.

Original: The idea of multi-teacher KD has been widely applied to numerous vision tasks, i .
Revised: The idea of multi-teacher KD has been widely applied to numerous vision tasks, i .

Original: e ., objectdetection (Chen et al., 2017; Wang et al., 2019), visual question answering (Niu & Zhang, 2021;
Revised: e ., object detection (Chen et al., 2017; Wang et al., 2019), visual question answering (Niu & Zhang, 2021;

Original: Chen et al., 2022) and ImgSGG (Li et al., 2022c).
Revised: Chen et al., 2022) and ImgSGG (Li et al., 2022c).

Original: In this paper, we propose two relation pseudolabel assignment teachers and use multi-teacher KD to fuse  knowledge of multi-teachers.
Revised: In this paper, we propose two relation pseudo label assignment teachers and use multi-teacher KD to fuse the knowledge of multi-teachers.

Original:  P ROPOSED A PPROACH
Revised:  P ROPOSED A PPROACH

Original: Task Definition.
Revised: Task Definition.

Original: In this paper, we propose a new VidSGG task: weakly-supervised VidSGG withsingle-frame weak supervision (SF-VidSGG).
Revised: In this paper, we propose a new VidSGG task: weakly-supervised VidSGG with single-frame weak supervision (SF-VidSGG).

Original: Specifically, given an input video V = { I 1 , ..., I T }with T frames, SF-VidSGG aims to generate a scene graph G t for each frame I t , and then stack themalong the time axis to obtain the final video scene graph G = { G 1 , ..., G T } , where G t =
Revised: Specifically, given an input video V = { I 1 , ..., I T } with T frames, SF-VidSGG aims to generate a scene graph G t for each frame I t , and then stack them along the time axis to obtain the final video scene graph G = { G 1 , ..., G T } , where G t =

Original: ( N t , E t ) ,
Revised: ( N t , E t ) ,

Original: N t and E t denote the set of graph nodes and edges in G t , respectively.
Revised: N t and E t denote the set of graph nodes and edges in G t , respectively.

Original: In the training stage, we onlyuse the weakly-supervision as mentioned in Sec.
Revised: In the training stage, we only use the weakly-supervision as mentioned in Sec.

Original: 1, i .
Revised: 1, i .

Original: e ., the only supervision for SF-VidSGG is anunlocalized scene graph for a middle frame of the video V , denoted as (cid:101)
Revised: e ., the only supervision for SF-VidSGG is an unlocalized scene graph for a middle frame of the video V , denoted as (cid:101)

Original: G m =
Revised: G m =

Original: ( (cid:101) N m , (cid:101) E m ) , and I mis the only frame in the video with annotation.
Revised: ( (cid:101) N m , (cid:101) E m ) , and I m is the only frame in the video with annotation.

Original: 3. M ODEL O VERVIEW
Revised: 3. M ODEL O VERVIEW

Original: To tackle the SF-VidSGG task, we propose a novel framework: Pseudo Label Assignment ( PLA ).
Revised: To tackle the SF-VidSGG task, we propose a novel framework: Pseudo Label Assignment ( PLA ).

Original: The whole pipeline of PLA is illustrated in Figure 2.
Revised: The whole pipeline of PLA is illustrated in Figure 2.

Original: Specifically, it consists of three components:object pseudo label assignment (Obj-PLA) module, relation pseudo label assignment (Rel-PLA)module, and future predicate prediction (FPP) module.
Revised: Specifically, it consists of three components: object pseudo label assignment (Obj-PLA) module, relation pseudo label assignment (Rel-PLA) module, and future predicate prediction (FPP) module.

Original: In the Obj-PLA, we detect object regionproposals for all the frames.
Revised: In the Obj-PLA, we detect object region proposals for all the frames.

Original: For the frame with weak annotation, we further complete its unlocalized scene graph (cid:101) G to obtain a pseudo localized scene graph G pseudo . In the Rel-PLA, we proposetwo relation pseudo label assignment teachers: model-based and model-free. The model-basedteacher assigns “soft” predicate pseudo labels P soft by an ImgSGG model, while the model-freeteacher assigns “hard” predicate pseudo labels P hard by some heuristic rules. After P soft and P hard
Revised: For the frame with weak annotation, we further complete its unlocal-

Original: [Figure]
Revised: [Figure]

Original:  O BJECT P ESUDO L ABEL A SSIGNMENT (O BJ -PLA)
Revised: 3. O BJECT P ESUDO L ABEL A SSIGNMENT (O BJ -PLA)

Original: In this module, we aim to generate pseudo object category annotations for all frames.
Revised: In this module we aim to generate pseudo object category annotations for all frames.

Original: For eachframe I t in input video, we first use an off-the-shelf detector to generate a set of proposal O t .
Revised: For each frame I t in input video, we first use an off-the-shelf detector to generate a set of proposal O t .

Original: Eachproposal o ti ∈
Revised: Each proposal o ti ∈

Original: O t has a corresponding bounding box position b ti and an initial object category ˆ c ti ,i .
Revised: O t has a corresponding bounding box position b ti and an initial object category ˆ c ti , i .

Original: e ., o ti =
Revised: e ., o ti =

Original: ( b ti , ˆ c ti ) .
Revised: ( b ti , ˆ c ti ) .

Original: Since the differences in object category ontology between the pretrained detectorand VidSGG benchmarks, we then map the detected object categories to the object categories in the
Revised: Since the differences in object category ontology between the pretrained detector and VidSGG benchmarks, we then map the detected object categories to the object categories in the

Original: VidSGG dataset if there is overlapping between their synonyms in WordNet (Miller, 1995), e .
Revised: VidSGG dataset if there is overlapping between their synonyms in WordNet (Miller, 1995), e .

Original: g .,“ woman ” is mapped to “ person ”, “ pizza box ” is mapped to “ box ”.
Revised: g ., “ woman ” is mapped to “ person ”, “ pizza box ” is mapped to “ box ”.

Original:  For the annotated frame I m , Obj-PLA has an additional step: matching these detected boundingboxes with entities in the unlocalized scene graph to obtain a pseudo localized scene graph.
Revised: [Figure] For the annotated frame I m , Obj-PLA has an additional step: matching these detected bounding boxes with entities in the unlocalized scene graph to obtain a pseudo localized scene graph.

Original: Herewe use a category-based strategy: a bounding box and a entity will be matched if they have the samecategory.
Revised: Here we use a category-based strategy: a bounding box and a entity will be matched if they have the same category.

Original: This strategy has a weakness that it cannot distinguish bounding boxes with the sameobject category.
Revised: This strategy has a weakness that it cannot distinguish bounding boxes with the same object category.

Original: When the detector finds multiple bounding boxes with the same object category,we just randomly choose one of them to match.
Revised: When the detector finds multiple bounding boxes with the same object category, we just randomly choose one of them to match.

Original: To solve this problem, Li et al. (2022c) utilizespretrained VL models to calculate relevant scores between entities with bounding boxes.
Revised: To solve this problem, Li et al. (2022c) utilizes pretrained VL models to calculate relevant scores between entities with bounding boxes.

Original: The finalmatches are decided by the object category labels and the relevant scores.
Revised: The final matches are decided by the object category labels and the relevant scores.

Original: We leave these advancedmethods for the future work.
Revised: We leave these advanced methods for the future work.

Original: 3. R ELATION P ESUDO L ABEL A SSIGNMENT (R EL -PLA)
Revised: 3. R ELATION P ESUDO L ABEL A SSIGNMENT (R EL -PLA)

Original: [Figure] After Obj-PLA, we obtain a pseudo localize scene graphfor the annotated Frame and object bboxes for all otherunannotated frames.
Revised: After Obj-PLA, we obtain a pseudo localize scene graph for the annotated Frame and object bboxes for all other unannotated frames.

Original: In Rel-PLA, we design two strategies (teachers) to generate pseudo predicate labels for theunannotated frames.
Revised: In Rel-PLA, we design two strategies (teachers) to generate pseudo predicate labels for the unannotated frames.

Original: Model-based Teacher.
Revised: Model-based Teacher.

Original: Model-based Teacher assigns“soft” predicate pseudo labels by an ImgSGG model.
Revised: Model-based Teacher assigns “soft” predicate pseudo labels by an ImgSGG model.

Original: Itfirst obtains pseudo localized scene graphs G pseudo fromthe annotated frame set by Obj-PLA, then train a ImgSGGmodel by these scene graphs.
Revised: It first obtains pseudo localized scene graphs G pseudo from the annotated frame set by ObjPLA, then train a ImgSGG model by these scene graphs.

Original: After training, each unannotated frame I t is fed to the network to obtain G t soft =
Revised: After training, each un-annotated frame I t is fed to the network to obtain G t soft =

Original: M img ( I t ) , the scene graph for I t .
Revised: M img ( I t ) , the scene graph for I t .

Original: Here we represent thepredicate in G t soft by the category distribution of predicate, besides one-hot labels.
Revised: Here we represent the predicate in G t soft by the category distribution of predicate, besides one-hot labels.

Original: Inspired by knowledge dis- tillation, we believe that the soft pseudo label can preserve more information than the hard pseudo label (Hinton et al., 2015).
Revised: Inspired by knowledge distillation, we believe that the soft pseudo label can preserve more information than the hard pseudo label (Hinton et al., 2015).

Original: Model-free Teacher.
Revised: Model-free Teacher.

Original: Model-free teacher assigns “hard” pseudo prediacte labels by some heuristicrules.
Revised: Model-free teacher assigns “hard” pseudo prediacte labels by some heuristic rules.

Original: The first step is object sequence matching .
Revised: The first step is object sequence matching .

Original: We adopt the object category and the IoU tomatch the objects detected in the different frames.
Revised: We adopt the object category and the IoU to match the objects detected in the different frames.

Original: Two object o ti and o t +1 j in adjacent frames can bematched if they have same object categories and IoU ( b ti , b t +1 j ) >
Revised: Two object o ti and o t +1 j in adjacent frames can be matched if they have same object categories and IoU ( b ti , b t +1 j ) >

Original: η , where η is the IoU matchingthreshold to filter objects.
Revised: η , where η is the IoU matching threshold to filter objects.

Original: We frame-by-framely match objects from the middle frame to both ends.
Revised: We frame-by-framely match objects from the middle frame to both ends.

Original: As shown in Figure 3, “ book ” and “ towel ” in I m +2 are not matched with any object in I m +1because there is no object detected in I m +1 with category “ book ” and IoU of bounding boxes withcategory “ towel ” in I m {++
+}
Revised: As shown in Figure 3, “ book ” and “ towel ” in I m +2 are not matched with any object in I m + because there is no object detected in I m +1 with category “ book ” and IoU of bounding boxes with category “ towel ” in I m {++
+}

Original: [-+
-] and I m +2 is lower than η .
Revised: [-+
-] and I m +2 is lower than η .

Original: The second step is relation propagation .
Revised: The second step is relation propagation .

Original: Wesimply propagate the predicate of a subject-object pair in the annotated frame to the same subjectobject pair in the un-annotated frames.
Revised: We simply propagate the predicate of a subject-object pair in the annotated frame to the same subjectobject pair in the un-annotated frames.

Original: 3. F UTURE P REDICATE P REDICTION (FPP) M ODULE
Revised: 3. F UTURE P REDICATE P REDICTION (FPP) M ODULE

Original: Multi-Teacher Fusion Strategy.
Revised: Multi-Teacher Fusion Strategy.

Original: After obtaining P soft and P hard by two teachers, the simplest fusingstrategy is treating each teacher equally and averaging the two pseudo labels.
Revised: After obtaining P soft and P hard by two teachers, the simplest fusing strategy is treating each teacher equally and averaging the two pseudo labels.

Original: However, this strategyoverlooks the noise in the model-based teacher.
Revised: However, this strategy overlooks the noise in the model-based teacher.

Original: The model-based teacher sometimes assign wrongpseudo labels due to the incorrect predictions in model M img .
Revised: The model-based teacher sometimes assign wrong pseudo labels due to the incorrect predictions in model M img .

Original: On the other hand, pseudo labelsassigned by the model-free teacher can be seen as the default predictions because these labels havealready appeared in the video.
Revised: On the other hand, pseudo labels assigned by the model-free teacher can be seen as the default predictions because these labels have already appeared in the video.

Original: To some extent, the model-free teacher can be seen as a complementwhen the model-based teacher assign a wrong pseudo label.
Revised: To some extent, the model-free teacher can be seen as a complement when the model-based teacher assign a wrong pseudo label.

Original: Therefore, a better strategy is assigningadaptive weights for P soft and P hard and then calculating the final pseudo labels by the weighted sum:
Revised: Therefore, a better strategy is assigning adaptive weights for P soft and P hard and then calculating the final pseudo labels by the weighted sum:

Original: [Equation]
Revised: [Equation]

Original: where w i soft and w i hard denotes the weights of the soft and hard predicate labels for video i , respectively.
Revised: where w i soft and w i hard denotes the weights of the soft and hard predicate labels for video i , respectively.

Original: In the FPP module, we calculate w i soft and w i hard by regularity of relation transition.
Revised: In the FPP module, we calculate w i soft and w i hard by regularity of relation transition.

Original: Relation Transition.
Revised: Relation Transition.

Original: Relation transition means how the predicates change between the samesubject-object pairs in different frames.
Revised: Relation transition means how the predicates change between the same subject-object pairs in different frames.

Original: Mi et al.
Revised: Mi et al.

Original: (2021) has proved that relation transition hasthe temporal tendency, which means we can infer their relation in the next frame according totheir relation in the current frame for the same subject-object pair.
Revised: (2021) has proved that relation transition has the temporal tendency, which means we can infer their relation in the next frame according to their relation in the current frame for the same subject-object pair.

Original: As the example shown in
Revised: As the example shown in

Original: Figure 4, we can find two main conclusions: 1) Relation transitions have some regular patterns,e .
Revised: Figure 4, we can find two main conclusions: 1) Relation transitions have some regular patterns, e .

Original: g ., for subject-object pair “ people-chair ”, relation “ leaning on ” is mainly transferred to“ sitting on ”, but hardly transferred to “ lying on ” and “ standing on ”.
Revised: g ., for subject-object pair “ people-chair ”, relation “ leaning on ” is mainly transferred to “ sitting on ”, but hardly transferred to “ lying on ” and “ standing on ”.

Original: 2) Relation transitions with different subject-object pairs have different regular patterns, e .
Revised: 2) Relation transitions with different subject-object pairs have different regular patterns, e .

Original: g ., the statistical distributions for “ people-chair ” and “ people-food ” are different.
Revised: g ., the statistical distributions for “ people-chair ” and “ people-food ” are different.

Original: Therefore, we can evaluate thequality of predicate pseudo labels by regularity of relation transition in video.
Revised: Therefore, we can evaluate the quality of predicate pseudo labels by regularity of relation transition in video.

Original: Specifically, irregular relation transitions ( e . g .
Revised: Specifically, irregular relation transitions ( e . g .

Original: , “ leaning on ” → “ lying on ” for “ people-chair ”) are of lowquality, then we assign low weights to them.
Revised: , “ leaning on ” → “ lying on ” for “ people-chair ”) are of low quality, then we assign low weights to them.

Original: Proposed FPP.
Revised: Proposed FPP.

Original: As shown in Figure 5, We take two steps to calculate weights w soft and w hard . 1)
Revised: As shown in Figure 5, We take two steps to calculate weights w soft and w hard . 1)

Original: Predicate prediction : For each predicate p t in I t assigned by the model-based teacher, we predictˆ p t +1 as the predicate that p t will transition to in I t
Revised: Predicate prediction : For each predicate p t in I t assigned by the model-based teacher, we predict ˆ p t +1 as the predicate that p t will transition to in I t

Original: +1 .
Revised: +1 .

Original: For each ˆ p t +1 , we find its correspondingpredicate p t +1 in I t
Revised: For each ˆ p t +1 , we find its corresponding predicate p t +1 in I t

Original: +1 that actually assigned by the model-based teacher.
Revised: +1 that actually assigned by the model-based teacher.

Original: 2) Adapted weight assignment :
Revised: 2) Adapted weight assignment :

Original: We first calculate KL-divergence between ˆ p t +1 and p t +1 as the inconsistency score.
Revised: We first calculate KL-divergence between ˆ p t +1 and p t +1 as the inconsistency score.

Original: Then, we assign low weights to pseudo labels with high inconsistency scores and high weights topseudo labels with low inconsistency scores.
Revised: Then, we assign low weights to pseudo labels with high inconsistency scores and high weights to pseudo labels with low inconsistency scores.

Original: In the following, we detailed introduce these two steps.
Revised: In the following, we detailed introduce these two steps.

Original: Predicate Prediction.
Revised: Predicate Prediction.

Original: To predict the future predicates, we construct a relation transition matrix
Revised: To predict the future predicates, we construct a relation transition matrix

Original: T ∈ R N o ∗ N r ∗ N r , where N o , N r are the number of objects and relations in the dataset, respectively.
Revised: T ∈ R N o ∗ N r ∗ N r , where N o , N r are the number of objects and relations in the dataset, respectively.

Original: The relation transition matrix indicates the statistical distribution of relation transition in the VidSGGdataset.
Revised: The relation transition matrix indicates the statistical distribution of relation transition in the VidSGG dataset.

Original: Given a predicate p t in frame I t with object category c
Revised: Given a predicate p t in frame I t with object category c

Original: o , we predict ˆ p t +1 as:
Revised: o , we predict ˆ p t +1 as:

Original: [Equation]
Revised: [Equation]

Revised: [Figure]

Revised: [Figure]

Original: where id ( c o ) is the id of c o in the vocabulary C go .
Revised: where id ( c o ) is the id of c o in the vocabulary C go .

Original: Then, for each ˆ p t +1 , we try to find its corresponding predicate p t +1 in I
Revised: Then, for each ˆ p t +1 , we try to find its corresponding predicate p t +1 in I

Original: t +1 that actually assigned by the model-based teacher.
Revised: t +1 that actually assigned by the model-based teacher.

Original: Sometimes p t +1 is notexisted because the detector can not detect the subject-object pair of ˆ p t +1 in frame I t +1 .
Revised: Sometimes p t +1 is not existed because the detector can not detect the subject-object pair of ˆ p t +1 in frame I t +1 .

Original: To handle[Figure] [Figure] this problem, we adopt the object category and the IoU to filter predicates without corresponding subject-object pair in the next frame, as we did in object sequence matching of the model-freeteacher.
Revised: To handle this problem, we adopt the object category and the IoU to filter predicates without corresponding subject-object pair in the next frame, as we did in object sequence matching of the model-free teacher.

Original: The set of ˆ p t +1 and p t +1 after filtering in the whole video are denoted as ˆ P filter and P filter ,respectively.
Revised: The set of ˆ p t +1 and p t +1 after filtering in the whole video are denoted as ˆ P filter and P filter , respectively.

Original: Adapted Weight Assignment.
Revised: Adapted Weight Assignment.

Original: We first calculate the inconsistency score by KL-divergence as below:
Revised: We first calculate the inconsistency score by KL-divergence as below:

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Then, we calculate the inconsistency score for P soft , denoted as S softincon .
Revised: Then, we calculate the inconsistency score for P soft , denoted as S softincon .

Original: A high inconsistency scoreindicates that these predicate pseudo labels do not satisfy the regularity of relation transition, so weshould assign low weights to them.
Revised: A high inconsistency score indicates that these predicate pseudo labels do not satisfy the regularity of relation transition, so we should assign low weights to them.

Original: To the end, we define the function as:
Revised: To the end, we define the function as:

Original: [Equation]
Revised: [Equation]

Original:  T RAINING O BJECTIVES
Revised: 3. T RAINING O BJECTIVES

Original: FPP module.
Revised: FPP module.

Original: We assume that the relation transition matrix T in FPP module indicates the statisticaldistribution of relation transition in the VidSGG dataset.
Revised: We assume that the relation transition matrix T in FPP module indicates the statistical distribution of relation transition in the VidSGG dataset.

Original: However, in the SF-VidSGG task, we cannot count the distribution of relation transition directly because we only have single-frame weaksupervision for each video.
Revised: However, in the SF-VidSGG task, we can not count the distribution of relation transition directly because we only have single-frame weak supervision for each video.

Original: An alternative approach is generating pseudo labels by the model-basedteacher and then learning the distribution of these pseudo labels.
Revised: An alternative approach is generating pseudo labels by the model-based teacher and then learning the distribution of these pseudo labels.

Original: To accomplish this, we formulatethe loss function as below to train the FPP module:
Revised: To accomplish this, we formulate the loss function as below to train the FPP module:

Original: [Equation]
Revised: [Equation]

Original: Student Model.
Revised: Student Model.

Original: We adopt the KL-divergence loss for training like most knowledge distillationmethods, which can be formulated as follows:
Revised: We adopt the KL-divergence loss for training like most knowledge distillation methods, which can be formulated as follows:

Original: [Equation]
Revised: [Equation]

Original: ) where P pred denotes the output of the student model, and P final denotes the predicate pseudo labels.
Revised: where P pred denotes the output of the student model, and P final denotes the predicate pseudo labels.

Original: Finally, the total training loss of PLA is:
Revised: Finally, the total training loss of PLA is:

Original: [Equation]
Revised: [Equation]

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Due to the limited space, the details about experimental settings (including the evaluation datasetand metrics) and implementation details are left in the appendix.
Revised: Due to the limited space, the details about experimental settings (including the evaluation dataset and metrics) and implementation details are left in the appendix.

Original: 4. A BLATION S TUDIES
Revised: 4. A BLATION S TUDIES

Original: In this section, we validated the effectiveness of our PLA by answering the following questions:
Revised: In this section, we validated the effectiveness of our PLA by answering the following questions:

Original: Q1 :
Revised: Q1 :

Original: How does each component in PLA contribute to the performance on the SF-VidSGG task?
Revised: How does each component in PLA contribute to the performance on the SF-VidSGG task?

Original: Q2 :
Revised: Q2 :

Revised: [Table]

Revised: [Table]

Original: How far are the gaps between PLA and fully-supervised VidSGG approaches?
Revised: How far are the gaps between PLA and fully-supervised VidSGG approaches?

Original: Q3 :
Revised: Q3 :

Original: Can a strongerstudent model help to improve the performance of PLA?
Revised: Can a stronger student model help to improve the performance of PLA?

Original: Q4 :
Revised: Q4 :

Original: What is the upper-bound under ourpseudo label assignment training paradigm?
Revised: What is the upper-bound under our pseudo label assignment training paradigm?

Original: Q5 :
Revised: Q5 :

Original: What is the influence of different annotation framechoosing strategies?[Table] [Table] [Table]
Revised: What is the influence of different annotation frame choosing strategies?

Original:  E FFECTIVENESS OF E ACH C OMPONENT (Q1)
Revised: 4.1. E FFECTIVENESS OF E ACH C OMPONENT (Q1)

Original: Settings.
Revised: Settings.

Original: To analyze the importance of each component in PLA, we implemented the straightforward model mentioned in Sec.
Revised: To analyze the importance of each component in PLA, we implemented the straightforward model mentioned in Sec.

Original: 1 as the baseline model.
Revised: 1 as the baseline model.

Original: It first obtained pseudo localized scenegraphs G pseudo from the annotated frame set by Obj-PLA, then trained a STTran † model by thesescene graphs.
Revised: It first obtained pseudo localized scene graphs G pseudo from the annotated frame set by Obj-PLA, then trained a STTran † model by these scene graphs.

Original: The gray lines in Table 1 show the results of the baseline model (model A ).
Revised: The gray lines in Table 1 show the results of the baseline model (model A ).

Original: We alsotrained the models with different components of PLA, which is shown in Table 1.
Revised: We also trained the models with different components of PLA, which is shown in Table 1.

Original: Effectiveness of Rel-PLA.
Revised: Effectiveness of Rel-PLA.

Original: Compared to the baseline model A , models with model-based teacher ormodel-free teacher (model B and model C ) has a significant improvement.
Revised: Compared to the baseline model A , models with model-based teacher or model-free teacher (model B and model C ) has a significant improvement.

Original: The experimental resultsdemonstrate that by utilizing the unannotated frames in the video, model B and model C are ableto outperform the baseline model
Revised: The experimental results demonstrate that by utilizing the unannotated frames in the video, model B and model C are able to outperform the baseline model

Original: A across almost all the metrics, only except for no constrained
Revised: A across almost all the metrics, only except for no constrained

Original: PredCls-R@50 of model B .
Revised: PredCls-R@50 of model B .

Original: Particularly, model B and model C have outperformed the baseline by5.24% (15.07 v .
Revised: Particularly, model B and model C have outperformed the baseline by 5.24% (15.07 v .

Original: s .14.32) and 2.58% (14.69 v .
Revised: s .14.32) and 2.58% (14.69 v .

Original: s .14.32)
Revised: s .14.32)

Original: relatively in terms of R@10 with constraintcriteria for SGDet, respectively.
Revised: relatively in terms of R@10 with constraint criteria for SGDet, respectively.

Original: This is owing to that the two teachers have the ability to assignhigh-quality pseudo labels to the unannotated frames.
Revised: This is owing to that the two teachers have the ability to assign high-quality pseudo labels to the unannotated frames.

Original: Effectiveness of FPP.
Revised: Effectiveness of FPP.

Original: As shown in Table 1, model D and model E are all use both teachers, butmodel E uses the FPP module to set adapted weights.
Revised: As shown in Table 1, model D and model E are all use both teachers, but model E uses the FPP module to set adapted weights.

Original: We can observe that model E outperformsmodel D across all the metrics.
Revised: We can observe that model E outperforms model D across all the metrics.

Original: Especially, model E has achieved about 2% relative improvementover model D on R@10 metric under the no constraint criteria for SGDet.
Revised: Especially, model E has achieved about 2% relative improvement over model D on R@10 metric under the no constraint criteria for SGDet.

Original: The improvement provesthat FFP has the ability to assign more accurate pseudo labels than the simple fixed weight strategy.
Revised: The improvement proves that FFP has the ability to assign more accurate pseudo labels than the simple fixed weight strategy.

Original: [Table]
Revised: [Table]

Original: [Table]

Original: [Table]
Revised: [Table]

Original: G AP BETWEEN PLA AND FULLY S UPERVISED A PPROACHS (Q2)
Revised: G AP BETWEEN PLA AND FULLY S UPERVISED A PPROACHS (Q2)

Original: Settings.
Revised: Settings.

Original: We compared our PLA with three state-of-the-art fully-supervised VidSGG methods:
Revised: We compared our PLA with three state-of-the-art fully-supervised VidSGG methods:

Original: STTran (Cong et al., 2021), APT (Li et al., 2022d), and DSG-DETR (Feng et al., 2021).
Revised: STTran (Cong et al., 2021), APT (Li et al., 2022e) [Table] and DSG-DETR (Feng et al., 2021).

Original: Results.
Revised: Results.

Original: As shown in Table 2, we can observe that there is still a gap between the performanceof PLA and other fully-supervised approaches ( e . g ., 15.4 v .
Revised: As shown in Table [Table] we can observe that there is still a gap between the performance of PLA and other fully-supervised approaches ( e . g ., 15.4 v .

Original: s .25.

Revised: s .25.

Original: for STTran in terms of R@10with constraint criteria for SGDet).
Revised: for STTran in terms of R@ with constraint criteria for SGDet).

Original: However, considering that the annotation cost of our methodis extremely lower ( < 5%) than that of full supervision.
Revised: However, considering that the annotation cost of our method is extremely lower ( < 5%) than that of full supervision.

Original: we think it is still a good start for weaklysupervised frame-level VidSGG.
Revised: we think it is still a good start for weakly supervised frame-level VidSGG.

Original: We leave the task to reduce the gap as the future work.
Revised: We leave the task to reduce the gap as the future work.

Original: 4.1. E FFECTS OF D IFFERENT S TUDENT M ODELS 
Revised: 4.1. E FFECTS OF D IFFERENT S TUDENT M ODELS (Q3)

Original: (Q3)

Original: Settings.
Revised: Settings.

Original: We used DSG-DETR as the student model of PLA.
Revised: We used DSG-DETR as the student model of PLA.

Original: Since PLA is agnostic to the specific
Revised: Since PLA is agnostic to the specific

Original: VidSGG architecture, it can be easily incorporated into any advanced VidSGG models.
Revised: VidSGG architecture, it can be easily incorporated into any advanced VidSGG models.

Original: We usedthe same teachers as for STTran to generate pseudo labels for DSG-DETR, then trained a fullysupervised VidSGG model by these pseudo labels.
Revised: We used the same teachers as for STTran to generate pseudo labels for DSG-DETR, then trained a fullysupervised VidSGG model by these pseudo labels.

Original: Results.
Revised: Results.

Original: As shown in Table 2, PLA with DSG-DETR also achieves similar performance with
Revised: As shown in Table 2, PLA with DSG-DETR also achieves similar performance with

Original: STTran across all the metrics.
Revised: STTran across all the metrics.

Original: These results show that PLA is a general framework and can easily extend to advanced fully supervised VidSGG methods.
Revised: These results show that PLA is a general framework and can easily extend to advanced fully supervised VidSGG methods.

Original: 4.1. B EST C ASE : G ROUND - TRUTH U NLOCALIZED S CENE G RAPHS (Q4)
Revised: 4.1. B EST C ASE : G ROUND - TRUTH U NLOCALIZED S CENE G RAPHS (Q4)

Original: Settings.
Revised: Settings.

Original: Like most existing SGG methods, PLA has two main steps: the first step is to detect objectsin each frame, and the second step is to predict relations between these objects.
Revised: Like most existing SGG methods, PLA has two main steps: the first step is to detect objects in each frame, and the second step is to predict relations between these objects.

Original: The performanceof PLA is mainly affected by the first step due to the poor performance of the off-shelf detectorin Action Genome.
Revised: The performance of PLA is mainly affected by the first step due to the poor performance of the off-shelf detector in Action Genome.

Original: Under this object detection strategy, the upper-bound performance is not high.
Revised: Under this object detection strategy, the upper-bound performance is not high.

Original: We can train an upper-bound model as follows: first detect objects by Obj-PLA, then assign groundtruth relation labels between them, finally use these labels to train the student model.
Revised: We can train an upper-bound model as follows: first detect objects by Obj-PLA, then assign ground truth relation labels between them, finally use these labels to train the student model.

Original: We named thisupper-bound model as GTRel.
Revised: We named this upper-bound model as GTRel.

Original: Results.
Revised: Results.

Original: As shown in Table 3, we observe that GTRel significantly improves the performanceover the baseline.
Revised: As shown in Table 3, we observe that GTRel significantly improves the performance over the baseline.

Original: Specifically, GTRel has outperformed the baseline by 13.19% (16.73 v .
Revised: Specifically, GTRel has outperformed the baseline by 13.19% (16.73 v .

Original: s .14.72)relatively in terms of R@10 no constraint criteria for SGDet.
Revised: s .14.72) relatively in terms of R@10 no constraint criteria for SGDet.

Original: We also demonstrate that PLA hasability to reduce the gap between the baseline model and GTRel.
Revised: We also demonstrate that PLA has ability to reduce the gap between the baseline model and GTRel.

Original: For example, GTRel improvesby 12.43% (16.10 v .
Revised: For example, GTRel improves by 12.43% (16.10 v .

Original: s .14.32)
Revised: s .14.32)

Original: relatively over baseline in terms of R@10 with constraint criteria for
Revised: relatively over baseline in terms of R@10 with constraint criteria for

Original: SGDet, but only improves by 4.61% (16.10 v .
Revised: SGDet, but only improves by 4.61% (16.10 v .

Original: s .15.39) relatively over PLA on same metric.
Revised: s .15.39) relatively over PLA on same metric.

Original: Theseresults show that PLA is able to reduce about half performance gap between the baseline modeland the upper-bound model GTRel for the SF-VidSGG task.
Revised: These results show that PLA is able to reduce about half performance gap between the baseline model and the upper-bound model GTRel for the SF-VidSGG task.

Original: Besides, the method to detect objectsunder unlocalized scene graph setting is not our main contribution.
Revised: Besides, the method to detect objects under unlocalized scene graph setting is not our main contribution.

Original: As a general framework, if bettermethods to detect objects are proposed, PLA can easily extend to them.
Revised: As a general framework, if better methods to detect objects are proposed, PLA can easily extend to them.

Original: 4.1. E FFECTS OF D IFFERENT A NNOTATION F RAME -C HOOSING S TRATEGIES . (Q5)
Revised: 4.1. E FFECTS OF D IFFERENT A NNOTATION F RAME -C HOOSING S TRATEGIES . (Q5)

Original: Settings.
Revised: Settings.

Original: We compared four simple annotation frame choosing strategies.
Revised: We compared four simple annotation frame choosing strategies.

Original: Specifically, “First”,“Last”, and “Middle” means annotating the “first”, “last”, and “middle” frame with an unlocalizedscene graph in each video, respectively.
Revised: Specifically, “First”, “Last”, and “Middle” means annotating the “first”, “last”, and “middle” frame with an unlocalized scene graph in each video, respectively.

Original: And “Random” means choosing one frame randomly.
Revised: And “Random” means choosing one frame randomly.

Original: Results.
Revised: [Figure]

Original: As shown in Table 4, We can observe that model trained by the middle annotation frame hasthe best performance, eg., in terms of R@10 with constraint criteria for SGDet, strategy “Middle”
Revised: Weakly

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original:  outperforms “First”, “Last”, and “Random” by 1.01, 1.14, and 0.48, respectively.
Revised: Results. As shown in Table 4, We can observe that model trained by the middle annotation frame has the best performance, eg., in terms of R@10 with constraint criteria for SGDet, strategy “Middle” outperforms “First”, “Last”, and “Random” by 1.01, 1.14, and 0.48, respectively.

Original: It also proves thatthe middle frame usually contains more information.
Revised: It also proves that the middle frame usually contains more information.

Original: 4. Q UALITATIVE R ESULTS
Revised: 4. Q UALITATIVE R ESULTS

Original: Pseudo Label Visualization.
Revised: Pseudo Label Visualization.

Original: Figure 6 shows a qualitative comparison of the pseudo labels generated by different variants of PLA.
Revised: Figure 6 shows a qualitative comparison of the pseudo labels generated by different variants of PLA.

Original: Results indicates that the adapted weights given by FPP modulecan generate more accurate pseudo labels than fixed weights.
Revised: Results indicates that the adapted weights given by FPP module can generate more accurate pseudo labels than fixed weights.

Original: Specifically, relation in T m +1 (“ notcontacting ”) is different from that in T m (“ touching ” and “ sitting on ”), so the modelfree teacher give totally wrong relations (“ touching ” and “ sitting on ”) and the model-basedteacher predict an almost accurate relation (98% confidence for “ not contacting ”).
Revised: Specifically, relation in T m +1 (“ not contacting ”) is different from that in T m (“ touching ” and “ sitting on ”), so the modelfree teacher give totally wrong relations (“ touching ” and “ sitting on ”) and the model-based teacher predict an almost accurate relation (98% confidence for “ not contacting ”).

Original: Compared to fixed weight strategy, the FPP module gives more weight to the model-based teacher (79%v .
Revised: Compared to fixed weight strategy, the FPP module gives more weight to the model-based teacher (79% v .

Original: s .49% confidence for “ not contacting ”) and generates more accurate pseudo labels.
Revised: s .49% confidence for “ not contacting ”) and generates more accurate pseudo labels.

Original: Relation Transition Distribution Visualization.
Revised: Relation Transition Distribution Visualization.

Original: Figure 7 demonstrates the statistical distributionsof relation transition counted by ground truth and learned by PLA.
Revised: Figure 7 demonstrates the statistical distributions of relation transition counted by ground truth and learned by PLA.

Original: From Figure 7, we can observethat PLA can learn a distribution similar to the ground truth.
Revised: From Figure 7, we can observe that PLA can learn a distribution similar to the ground truth.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: In this paper, we presented the first weakly-supervised VidSGG task with only single-frame weak supervision: SF-VidSGG.
Revised: In this paper, we presented the first weakly-supervised VidSGG task with only single-frame weak supervision: SF-VidSGG.

Original: Unlike VidSGG, SF-VidSGG only require a weak annotaion without bounding box for one frame in each video.
Revised: Unlike VidSGG, SF-VidSGG only require a weak annotaion without bounding box for one frame in each video.

Original: To the end, we proposed a novel and efficient method named
Revised: To the end, we proposed a novel and efficient method named

Original: PLA for SF-VidSGG, which based on pseudo label assignment for the video.
Revised: PLA for SF-VidSGG, which based on pseudo label assignment for the video.

Original: We validated the effectiveness of each component of PLA through extensive experiments.
Revised: We validated the effectiveness of each component of PLA through extensive experiments.

Original: In future work, we wouldlike to explore weaker supervision in VidSGG or more accurate pseudo label assignment.
Revised: In future work, we would like to explore weaker supervision in VidSGG or more accurate pseudo label assignment.

Original: Ethics Statement
Revised: Ethics Statement

Original: Video scene generation models may be utilized in unauthorized human monitoring and surveillance,which leads to some ethical and privacy issues.
Revised: Video scene generation models may be utilized in unauthorized human monitoring and surveillance, which leads to some ethical and privacy issues.

Original: Apart from these general issues that already exist in the video scene graph generation task, our paper has no additional ethical issues
Revised: Apart from these general issues that already exist in the video scene graph generation task, our paper has no additional ethical issues.

Original: .Reproducibility Statement
Revised: Reproducibility Statement

Original: PLA is mainly implemented based on the released code of STTran (Cong et al., 2021) and evaluatedon the Action Genome (AG)
Revised: PLA is mainly implemented based on the released code of STTran (Cong et al., 2021) and evaluated on the Action Genome (AG)

Original: (Ji et al., 2020) dataset, which is publicly available with a license thatallows free usage for research purposes.
Revised: (Ji et al., 2020) dataset, which is publicly available with a license that allows free usage for research purposes.

Original: We will also release our code of PLA.
Revised: We will also release our code of PLA.

Revised: Acknowledgment

Revised: This work was supported by the National Key Research & Development Project of China (2021ZD0110700), the National Natural Science Foundation of China (U19B2043, 61976185) and the Fundamental Research Funds for the Central Universities (226-2022-00051)

Original: A A PPENDIX
Revised: A A PPENDIX

Original: This Appendix has the following contents:
Revised: This Appendix has the following contents:

Original: • The details about the valuation dataset are given in Sec.
Revised: • The details about the valuation dataset are given in Sec.

Original: A.1• The evaluation metrics are given in Sec.
Revised: A. • The evaluation metrics are given in Sec.

Original: A.2.• The implementation details are given in Sec.
Revised: A.2. • The implementation details are given in Sec.

Original: A.3.• The detailed hyperparameter setting are given in Sec.
Revised: A.3. • The detailed hyperparameter setting are given in Sec.

Original: A.4.• Ablation studies on imperfect object detector in Sec.
Revised: A.4. • Ablation studies on imperfect object detector in Sec.

Original: A.5.• Ablation studies on the different proportions of the weakly supervised labels in Sec.
Revised: A.5. • Ablation studies on the different proportions of the weakly supervised labels in Sec.

Original: A.6.•
Revised: A.6.

Original:  More qualitative results are in Sec.
Revised: • More qualitative results are in Sec.

Original: A.7.
Revised: A.7.

Original: A D ATASET
Revised: A. D ATASET

Original: We evaluated our method PLA on the challenging VidSGG benchmark: Action Genome (AG) (Jiet al., 2020).
Revised: We evaluated our method PLA on the challenging VidSGG benchmark: Action Genome (AG) (Ji et al., 2020).

Original: AG is a prevalent VidSGG dataset providing human-object relationships with framelevel annotations.
Revised: AG is a prevalent VidSGG dataset providing human-object relationships with framelevel annotations.

Original: Specifically, AG consists of 9,201 videos, which annotated for 234,253 frames by476,299 bounding boxes of 36 object categories and 1,715,568 instances of 25 relation categories.
Revised: Specifically, AG consists of 9,201 videos, which annotated for 234,253 frames by 476,299 bounding boxes of 36 object categories and 1,715,568 instances of 25 relation categories.

Original: All relation categories in AG are split into three types: 3 attention relations, 6 spatial relations and16 contact relations.
Revised: All relation categories in AG are split into three types: 3 attention relations, 6 spatial relations and 16 contact relations.

Original: Each subject-object pair has one attention relation but may has multiple spatialor contact relations.
Revised: Each subject-object pair has one attention relation but may has multiple spatial or contact relations.

Original: In our SF-VidSGG setting, we used the official splits as fully-supervised work,i .
Revised: In our SF-VidSGG setting, we used the official splits as fully-supervised work, i .

Original: e ., 7,464 videos for training, 1,737 videos for testing.
Revised: e ., 7,464 videos for training, 1,737 videos for testing.

Original: For each video in the training set, we onlyused the unlocalized scene graph of the middle frame as supervision.
Revised: For each video in the training set, we only used the unlocalized scene graph of the middle frame as supervision.

Original: A. E VALUATION M ETRICS
Revised: A. E VALUATION M ETRICS

Original: Following the fully-supervised setting, we evaluated PLA on three different settings: 1) Scene Graph
Revised: Following the fully-supervised setting, we evaluated PLA on three different settings: 1) Scene Graph

Original: Detection ( SGDet ): Given a video, the model needs to detect the objects and predict predicatecategories of object pairs for all frame.
Revised: Detection ( SGDet ): Given a video, the model needs to detect the objects and predict predicate categories of object pairs for all frame.

Original: In SGDet, an object is considered to be successfully detectedif the IoU between the predicted bbox and its ground-truth bbox is larger than 0.5.
Revised: In SGDet, an object is considered to be successfully detected if the IoU between the predicted bbox and its ground-truth bbox is larger than 0.5.

Original: 2) Scene Graph
Revised: 2) Scene Graph

Original: Classification ( SGCls ): Given a video and object bboxes, the model needs to predict object andpredicate categories for all frames.
Revised: Classification ( SGCls ): Given a video and object bboxes, the model needs to predict object and predicate categories for all frames.

Original: Predicate Classification ( PredCls ): Given a video, objectbboxes and object labels, the model needs to predict predicate categories for all frames.
Revised: Predicate Classification ( PredCls ): Given a video, object bboxes and object labels, the model needs to predict predicate categories for all frames.

Original: Since the ground-truth bboxes are not provided in SF-VidSGG, we only trained models under SGDet settings,then evaluated them under these three settings.
Revised: Since the ground-truth bboxes are not provided in SF-VidSGG, we only trained models under SGDet settings, then evaluated them under these three settings.

Original: We used Recall@K ( R@K , K = [10,20,50]) as ourevaluation metric, which measures the ratio of the ground-truth relation triplets among the top-Kpredicted relation triplets.
Revised: We used Recall@K ( R@K , K = [10,20,50]) as our evaluation metric, which measures the ratio of the ground-truth relation triplets among the top-K predicted relation triplets.

Original: Furthermore, we adopt two typical strategies to generate dynamic scenegraphs
Revised: Furthermore, we adopt two typical strategies to generate dynamic scene graphs

Original: : 1) With Constraint : It only allows each subject-object pair to have at most one predicate.
Revised: : 1) With Constraint : It only allows each subject-object pair to have at most one predicate

Original: No Constraint : It allows each subject-object pair to have multiple predicates.
Revised: No Constraint : It allows each subject-object pair to have multiple predicates.

Original: A. I MPLEMENTATION D ETAILS
Revised: A. I MPLEMENTATION D ETAILS

Original: In the Obj-PLA module, we used the pretrained VinVL (Li et al., 2020; Zhang et al., 2021) withbackbone ResNeXt-152 C4 as the off-the-shelf detector.
Revised: In the Obj-PLA module, we used the pretrained VinVL (Li et al., 2020; Zhang et al., 2021) with backbone ResNeXt-152 C4 as the off-the-shelf detector.

Original: Specifically, it is pre-trained on multipledetection benchmarks, including COCO (Lin et al., 2014), OpenImages (Kuznetsova et al., 2020),
Revised: Specifically, it is pre-trained on multiple detection benchmarks, including COCO (Lin et al., 2014), OpenImages (Kuznetsova et al., 2020),

Original: Objects365 (Shao et al., 2019), and Visual Genome (Krishna et al., 2017).
Revised: Objects365 (Shao et al., 2019), and Visual Genome (Krishna et al., 2017).

Original: This detector is capableof detecting 1,594 general object categories.
Revised: This detector is capable of detecting 1,594 general object categories.

Original: We kept objects which confidence is higher than 0.2and extracted the 2048-D region features from the detector.
Revised: We kept objects which confidence is higher than 0. and extracted the 2048-D region features from the detector.

Original: In the Rel-PLA module, we modified
Revised: In the Rel-PLA module, we modified

Original: STTran (Cong et al., 2021) to an ImgSGG model that process one frame at a time, denoted as
Revised: STTran (Cong et al., 2021) to an ImgSGG model that process one frame at a time, denoted as

Original: STTran † .
Revised: STTran † .

Original: We used STTran † as the teacher model to assign pseudo soft labels in Rel-PLA.
Revised: We used STTran † as the teacher model to assign pseudo soft labels in Rel-PLA.

Original: For themodel-free teacher, we set the IOU matching threshold η = 0 .
Revised: For the model-free teacher, we set the IOU matching threshold η = 0 .

Original: In FPP module, we set the initiallearning rate 1 e − 3 .
Revised: In FPP module, we set the initial learning rate 1 e − 3 .

Original: We used STTran as the student model, and followed the same training settings( e .
Revised: We used STTran as the student model, and followed the same training settings ( e .

Original: g ., learning rate and batch size) of the original paper.
Revised: g ., learning rate and batch size) of the original paper.

Original: A. H YPERPARAMETER S ETTINGS
Revised: A. H YPERPARAMETER S ETTINGS

Original: We analyzed the influence of the hyperparameter η , which denote the IoU matching threshold in
Revised: We analyzed the influence of the hyperparameter η , which denote the IoU matching threshold in

Original: Rel-PLA.
Revised: Rel-PLA.

Original: We set different η for the model only uses model-free teacher (model C ).
Revised: We set different η for the model only uses model-free teacher (model C ).

Original: All results are in
Revised: All results are in

Original: Table 5.
Revised: Table 5.

Original: Particularly, when the threshold η is set to 0, we match the objects in different frames onlyby the object category without the IoU of the bounding boxes in adjacent frames.
Revised: Particularly, when the threshold η is set to 0, we match the objects in different frames only by the object category without the IoU of the bounding boxes in adjacent frames.

Original: [Table]
Revised: [Table]

Original: Results.
Revised: Results.

Original: As shown in Table
Revised: As shown in Table

Original: [Table]
Revised: [Table]

Original: we can observe that the model with η = 0 has a weak performancedue to more wrong objects matching without IoU information.
Revised: we can observe that the model with η = 0 has a weak performance due to more wrong objects matching without IoU information.

Original: And the model with η = 0 .
Revised: And the model with η = 0 .

Original: 5 slightlyoutperforms the model with η = 0 .
Revised: 5 slightly outperforms the model with η = 0 .

Original: Therefore, we set η = 0.5 for all experiments.
Revised: Therefore, we set η = 0.5 for all experiments.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: A. E FFECTS OF I MPERFECT O BJECT D ETECTORS
Revised: A. E FFECTS OF I MPERFECT O BJECT D ETECTORS

Original: Settings.
Revised: Settings.

Original: To simulate an imperfect object detector, we randomly dropped a certain proportion ofbounding boxes given by the object detector.
Revised: To simulate an imperfect object detector, we randomly dropped a certain proportion of bounding boxes given by the object detector.

Original: Without loss of generality, we set five dropped proportion as 0%, 10%, 20%, 40% and 80%.
Revised: Without loss of generality, we set five dropped proportion as 0%, 10%, 20%, 40% and 80%.

Original: Results.
Revised: Results.

Original: As shown in Table 6, we can observe that the bigger dropped proportion, the worse theperformance of PLA, eg.
Revised: As shown in Table 6, we can observe that the bigger dropped proportion, the worse the performance of PLA, eg.

Original: the setting with 0% dropped proportion outperforms settings with 10%,20%, 40% and 80% dropped proportion by 1.23, 2.21, 3.91 and 10.55, respectively.
Revised: the setting with 0% dropped proportion outperforms settings with 10%, 20%, 40% and 80% dropped proportion by 1.23, 2.21, 3.91 and 10.55, respectively.

Original: When dropping10%, 20% and 40% bboxes, the performance has not decreased much, but the performance decreasedsignificantly if dropping 80% bboxes.
Revised: When dropping 10%, 20% and 40% bboxes, the performance has not decreased much, but the performance decreased significantly if dropping 80% bboxes.

Original: Results show that PLA is affected by the preformance of theobject detector, but it is the weakness for all two-stage (first detect objects, then predict relationsbetween them) SGG method.
Revised: Results show that PLA is affected by the preformance of the object detector, but it is the weakness for all two-stage (first detect objects, then predict relations between them) SGG method.

Original: A. E FFECTS OF THE S IZE OF W EAKLY S UPERVISED L ABELS
Revised: A. E FFECTS OF THE S IZE OF W EAKLY S UPERVISED L ABELS

Original: Settings.
Revised: Settings.

Original: Although PLA is designed for single-frame weak supervision, it can be easily extendedto multi-frame weak supervision.
Revised: Although PLA is designed for single-frame weak supervision, it can be easily extended to multi-frame weak supervision.

Original: For the model-based teacher, we just train the model with moreframes.
Revised: For the model-based teacher, we just train the model with more frames.

Original: For the model-free teacher, we take each annotated frame as a starting point to propagaterelations.
Revised: For the model-free teacher, we take each annotated frame as a starting point to propagate relations.

Original: Without loss of generality, we set four annotated proportion as 10%, 20%, 40% and 80%.
Revised: Without loss of generality, we set four annotated proportion as 10%, 20%, 40% and 80%.

Original: We annotated frame evenly in temporal axis, for example, if a 20-frame video with 20% annotatedproportion, we will annotate its 1st, 7th, 14th and 20th frame.
Revised: We annotated frame evenly in temporal axis, for example, if a 20-frame video with 20% annotated proportion, we will annotate its 1st, 7th, 14th and 20th frame.

Original: Results.
Revised: Results.

Original: As shown in 7, we can observe that the more annotated frames, the better the performanceof PLA, e.g., model with 80% annotation frames outperforms models with 10%, 20% and 40%annotation frames 1.16, 0.26 and 0.03.
Revised: As shown in 7, we can observe that the more annotated frames, the better the performance of PLA, e.g., model with 80% annotation frames outperforms models with 10%, 20% and 40% annotation frames 1.16, 0.26 and 0.03.

Original: It proves that PLA can easily adapt to multi-frame weaksupervision setting.
Revised: It proves that PLA can easily adapt to multi-frame weak supervision setting.

Original: A. M ORE QUALITATIVE RESULTS ON A CTION G ENOME DATASET
Revised: A. M ORE QUALITATIVE RESULTS ON A CTION G ENOME DATASET

Original: More qualitative results of PLA on Action Genome are shown in Figure 8.
Revised: More qualitative results of PLA on Action Genome are shown in Figure 8.

Original: From Figure 8, we canobserve that PLA has the  person 
Revised: From Figure 8, we can observe that PLA has the ability to detect dynamic relations, e . g ., in the third video, the relation between person and table from “ not contacting ” to “ touching ”. [Figure]

Original: Standing on
Revised: Standing on

Original: Holding shoe floorperson
Revised: Holding shoe floor

Revised: [Figure]

Original: Standing on
Revised: Standing on

Original: Not contanting shoe floor person
Revised: Not contanting shoe [Figure] person

Original: Standing on
Revised: Standing on

Original: Not contanting shoe floor shoe
Revised: Not contanting shoe [Figure] shoe

Original: Not contanting person
Revised: Not contanting person

Original: Sitting on
Revised: Sitting on

Original: Not contacting table chair person
Revised: Not contacting table chair person

Original: Sitting on
Revised: Sitting on

Original: Not contacting table chair bag
Revised: Not contacting table chair bag

Original: Touching bag
Revised: Touching bag

Original: Holding person person
Revised: Holding person person

Original: Not contacting
Revised: Not contacting

Original: Touching laptop table person
Revised: Touching laptop table person

Original: Not contacting
Revised: Not contacting

Original: Touching laptop table chair
Revised: Touching laptop table chair

Original: Sitting on person
Revised: Sitting on person

Original: Touching
Revised: Touching

Original: Sitting on chair table table
Revised: Sitting on chair table table

Original: Touching
Revised: Touching