Existing image retrieval systems use text queries to provide a natural and practical way for users to express what they are looking for. However, fine-grained image retrieval often requires the ability to also express the where in the image the content they are looking for is. The textual modality can only cumbersomely express such localization preferences, whereas pointing would be a natural fit. In this paper, we describe an image retrieval setup  where the user simultaneously describes an image using both spoken natural language (the "what" ) and mouse traces over an empty canvas (the "where" ) to express the characteristics of the desired target image. To this end, we learn an image retrieval model using the Localized Narratives dataset, which is capable of performing early fusion between text descriptions and synchronized mouse traces . Qualitative and quantitative experiments show that our model is capable of taking this spatial guidance into account, and provides  more accurate retrieval results compared to text-only equivalent systems.
Most existing image retrieval systems use text queries as a way for the user to express what they are looking for. However, fine-grained image retrieval often requires the ability to also express the where in the image the content they are looking for is. The text modality can only cumbersomely express such localization preferences, whereas pointing is a more natural fit. In this paper, we propose an image retrieval setup with a new form of multimodal queries, where the user simultaneously uses both spoken natural language (the what ) and mouse traces over an empty canvas (the where ) to express the characteristics of the desired target image. We then describe simple modifications to an existing image retrieval model , enabling it to operate in this setup . Qualitative and quantitative experiments show that our model effectively takes this spatial guidance into account, and provides significantly more accurate retrieval results compared to text-only equivalent systems.
R
Existing
Most existing
R
to provide a natural and practical way for users
as a way for the user
R
textual
text
R
would be a
is a more
R
describe
propose
A
with a new form of multimodal queries,
R
describes an image using
uses
R
"what"
what
R
"where"
where
R
To this end, we learn an
We then describe simple modifications to an existing
R
using the Localized Narratives dataset, which is capable of performing early fusion between text descriptions and synchronized mouse traces
, enabling it to operate in this setup
R
is capable of taking
effectively takes
A
significantly