  Traditional scene graph generation methods are trained using cross-entropy
losses that treat objects and relationships as independent entities. Such a
formulation, however, ignores the structure in the output space, in an
inherently structured prediction problem. In this work, we introduce a novel
energy-based learning framework for generating scene graphs. The proposed
formulation allows for efficiently incorporating the structure of scene graphs
in the output space. This additional constraint in the learning framework acts
as an inductive bias and allows models to learn efficiently from a small number
of labels. We use the proposed energy-based framework to train existing
state-of-the-art models and obtain a significant performance improvement, of up
to 21% and 27%, on the Visual Genome and GQA benchmark datasets, respectively.
Furthermore, we showcase the learning efficiency of the proposed framework by
demonstrating superior performance in the zero- and few-shot settings where
data is scarce.
