This paper presents a framework for localization or grounding of phrases in images using a large collection of linguistic and visual cues. 1 We model the appearance, size, and position of entity bounding boxes, adjectives that contain attribute information, and spatial relationships between pairs of entities connected by verbs or prepositions. We pay special attention to relationships between people and clothing or body part mentions, as they are useful for distinguishing individuals. We automatically learn weights for combining these cues and at test time, perform joint inference over all phrases in a caption. The resulting system produces a 5% improvement in accuracy over the state of the art on phrase localization on the Flickr30k Entities dataset [25] and a 4-10% improvement for visual relationship detection on the Stanford VRD dataset [20] .
Plummer et al. REF grounded phrases in images by fusing several visual features like appearance, size, bounding boxes, and linguistic cues (like adjectives that describe attribute information).
Phrase Localization and Visual Relationship Detection with Comprehensive Linguistic Cues.
ArXiv
ArXiv
Computer Science