Predicate Debiasing in Vision-Language Models Integration for Scene Graph Generation Enhancement (2024)

Abstract: Scene Graph Generation (SGG) provides basic language representation of visual scenes, requiring models to grasp complex and diverse semantics between objects. This complexity and diversity in SGG leads to underrepresentation, where parts of triplet labels are rare or even unseen during training, resulting in imprecise predictions. To tackle this, we propose integrating the pretrained Vision-language Models to enhance representation. However, due to the gap between pretraining and SGG, direct inference of pretrained VLMs on SGG leads to severe bias, which stems from the imbalanced predicates distribution in the pretraining language set. To alleviate the bias, we introduce a novel LM Estimation to approximate the unattainable predicates distribution. Finally, we ensemble the debiased VLMs with SGG models to enhance the representation, where we design a certainty-aware indicator to score each sample and dynamically adjust the ensemble weights. Our training-free method effectively addresses the predicates bias in pretrained VLMs, enhances SGG’s representation, and significantly improve the performance.

## introduction scene graph generation (sgg) is a fundamental vision-language task that has attracted much effort. it bridges natural languages with scene representations and serves various applications, from robotic contextual awareness to helping visually impaired people. the key challenge in sgg is to grasp complex semantics to understand inter-object relationships in a scene. existing researches in sgg focus primarily on refining model architectures that are trained from scratch with datasets like visual genome @xcite or open images @xcite . however, sgg tasks inherently face another challenge of underrepresentation. due to the inherent complexities of sgg, there exists exponential variability of triplets combined by the subject, object, and relation (predicate). it is extremely challenging for a training set to cover such diversity. as a result, a part of the test distribution is underrepresented in training, leading to poor prediction quality. in a severe case, some triplet labels that appear in the test set are unseen in training. in figure 1 , we highlight the relation class "carrying" from visual genome, showing samples and their confidence scores of the ground truth class from a baseline model's predictions. while wellrepresented samples score higher, the samples labeled with unseen triplets like "woman carrying towel" score fairly low. furthermore, one "woman carrying umbrella" scores only 0.15 due to the umbrella being closed, while its counterpart with an open umbrella scores markedly higher (0.65). although the triplet is seen in training set, the closed "umbrella" is still short of representation. a straightforward solution to this issue is to expand the model's knowledge by integrating advanced vision-language models (vlms) pretrained on extensive datasets @xcite @xcite @xcite , using their comprehensive knowledge to compensate for underrepresented samples. employing the masked language modeling (mlm) prompt format, such as "woman is [mask] towel," allows for direct extraction of relation predictions from the fill-in answers provided by zero-shot vlms, which fully preserve the pretraining knowledge. nonetheless, this direct inference of zeroshot models on sgg introduces significant predicate bias due to disparities in data distribution and objectives between pretraining and sgg tasks. this predicate bias originates from the imbalanced frequency of predicates in the pretraining language set, causing the vlms to favor the predicates that are prevalent in the pretraining data. unfortunately, existing debiasing methods rely on explicit training distribution, which is often unattainable for pretrained vlms: (1) the pretraining data are often confidential. (2) since the pretraining objectives are different with sgg, there is no direct label correspondence from pretraining to sgg. to alleviate the predicate bias, we introduce a novel approach named lagrange-multiplier estimation (lm estimation) based on constrained optimization. since there is no explicit distribution of relation labels in the pretraining data, lm estimation seeks to estimate a surrogate distribution of sgg predicates within vlms. upon obtaining the estimated distribution, we proceed with predicates debiasing via post-hoc logits adjustment. our lm estimation, as demonstrated by comprehensive experiments, is proved to be exceedingly effective in mitigating the bias for zero-shot vlms. finally, we ensemble the debiased vlms with the sgg models to address their underrepresentation issue. we observe that some samples are better represented by the zero-shot vlm, while others align better with the sgg model. therefore, we propose to dynamically ensemble the two models. for each sample, we employ a certainty-aware indicator to score its representation level in the pretrained vlm and the sgg model, which subsequently determines the ensemble weights. our contributions can be summarized as follows: pre-trained vision-language models (vlms) have been widely applied in diverse visionlanguage tasks @xcite @xcite and have achieved substantial performance improvements with the vast knowledge base obtained during pre-training. recently works start to adapt the comprehensive pre-trained knowledge in vlms to relation recognition and scene graph generation @xcite @xcite ; text prompt (mlm): [cls] woman is [mask] bench. text prompt (vqa): [cls] what is the rela2onship between the woman and the bench? text prompt (mlm): [cls] man is [mask] bench. text prompt (vqa): [cls] what is the rela2onship between the man and the bench? fine-tuned vlm (𝑓 !" ) 🔥 vl transformer visual tokens … textual tokens (vqa) … classifier training set frequency zero-shot vlm (𝑓 #! ) ❄ vl transformer mlm head … … visual tokens … textual tokens (mlm) … lagrange @xmath1: [cls] woman is [mask] man. text prompt (vqa): [cls] what is the rela2onship between the woman and the man? 𝐱 @xmath0 , 𝑧 & ) 𝐨 !" 𝒌 log 𝛑 $% [𝐨 "& ' ; 𝐨 "& 𝒌 ] [𝐨 "& ' ; 0 𝐨 "& 𝒌 ] rela5on label debias zhang et al., 2023; zhao et al., 2023). through prompt-tuning, (he et al., 2022) is the first employing vlms to open-vocabulary scene graph generation. then more approaches (zhang et al., 2023; yu et al., 2023; gao et al., 2023) are designed towards this task. these works demonstrate the capability of vlms on recognizing relation, inspiring us to utilize vlms to improve the sgg representation. the objective is to learn a model f that, given the predicted objects z i and z j for each pair with their cropped image region x i,@xmath2, produces logits o for all relations @xmath3, i.e., @xmath4, z j , x i,j ). ## conclusion in conclusion, our study has made significant strides in efficiently and effectively integrate pretrained vlms to sgg. by introducing the novel lm estimation, we effectively mitigate the predicate bias inside pre-trained vlms, allowing their comprehensive knowledge to be employed in sgg. besides, our certainty-aware ensemble strategy, which ensembles the zero-shot vlms with sgg model, effectively addresses the underrepresentation issue and demonstrates a significant improvement in sgg performance. our work contributes to the field of sgg, suggesting potential pathways for reducing language bias of pretraining and leverage them in more complex language tasks.