We propose a knowledge-enhanced approach, ERNIE-ViL,  to learn joint representations of vision and language . ERNIE-ViL tries to construct the detailed semantic connections (objects, attributes of objects and relationships between objects in visual scenes ) across vision and language, which are essential to vision-language cross-modal tasks. Incorporating knowledge from scene graphs , ERNIE-ViL constructs Scene Graph Prediction tasks, i.e., Object Prediction, Attribute Prediction and Relationship Prediction  in the pre-training phase. More specifically , these prediction tasks are implemented by predicting nodes of different types in the scene graph parsed from the sentence. Thus, ERNIE-ViL can model the joint representation characterizing the alignments of the detailed semantics across vision and language. Pre-trained on two large image-text alignment datasets(Conceptual Captions and SBU), ERNIE-ViL learns better and more robust joint representations. It achieves state-of-the-art performance on 5 vision-language downstream tasksafter fine-tuning ERNIE-ViL. Furthermore, it ranked the 1st place on the VCR leader-board with an absolute improvement of 3.7\%.
We propose a knowledge-enhanced approach, ERNIE-ViL, which incorporates structured knowledge obtained from scene graphs to learn joint representations of vision-language . ERNIE-ViL tries to build the detailed semantic connections (objects, attributes of objects and relationships between objects ) across vision and language, which are essential to vision-language cross-modal tasks. Utilizing scene graphs of visual scenes , ERNIE-ViL constructs Scene Graph Prediction tasks, i.e., Object Prediction, Attribute Prediction and Relationship Prediction tasks in the pre-training phase. Specifically , these prediction tasks are implemented by predicting nodes of different types in the scene graph parsed from the sentence. Thus, ERNIE-ViL can learn the joint representations characterizing the alignments of the detailed semantics across vision and language. After pre-training on large scale image-text aligned datasets, we validate the effectiveness of ERNIE-ViL on 5 cross-modal downstream tasks. ERNIE-ViL achieves state-of-the-art performances on all these tasks and ranks the first place on the VCR leaderboard with an absolute improvement of 3.7\%.
A
which incorporates structured knowledge obtained from scene graphs
R
vision and language
vision-language
R
construct
build
D
in visual scenes
R
Incorporating knowledge from scene graphs
Utilizing scene graphs of visual scenes
A
tasks
R
More specifically
Specifically
R
model the joint representation
learn the joint representations
R
Pre-trained on two large
After pre-training on large scale
R
alignment datasets(Conceptual Captions and SBU),
aligned datasets, we validate the effectiveness of
D
learns better and more robust joint representations. It achieves state-of-the-art performance
R
vision-language downstream tasksafter fine-tuning ERNIE-ViL. Furthermore, it ranked the 1st
cross-modal downstream tasks. ERNIE-ViL achieves state-of-the-art performances on all these tasks and ranks the first
R
leader-board
leaderboard