Language- C onditioned F eature P yramids for V isual S election T asks (2020)

Abstract: Referring expression comprehension, which is the ability to locate language to an object in an image, plays an important role in creating common ground. Many models that fuse visual and linguistic features have been proposed. However, few models consider the fusion of linguistic features with multiple visual features with different sizes of receptive fields, though the proper size of the receptive field of visual features intuitively varies depending on expressions. In this paper, we introduce a neural network architecture that modulates visual features with varying sizes of receptive field by linguistic features. We evaluate our architecture on tasks related to referring expression comprehension in two visual dialogue games. The results show the advantages and broad applicability of our architecture. Source code is available at https://github.com/Alab-NII/lcfp .

## introduction referring expressions are a ubiquitous part of human communication @xcite that must be studied in order to create machines that work smoothly with humans. much effort has been taken to improve methods of creating visual common ground between machines, which have limited means of expression and knowledge about the real world, and humans, from the perspectives of both referring expression comprehension and generation @xcite @xcite @xcite . even now, researchers are exploring possible methods of designing more realistic scenarios for applications, such as in visual dialogue games @xcite @xcite . many models have been proposed for referring expression comprehension so far. as image recognition matured, @xcite object retrieval methods based on category labels predicted by the recognition models. @xcite extended this approach to broader natural language expression including categories of objects, their attributes, positional configurations, and interactions. in recent years, models that fuse linguistic features with visual features using deep learning have been studied @xcite @xcite @xcite @xcite . when fusing the linguistic features of a spatial referring expression with visual features, the size of the receptive field of visual features foot_0 is important. let us take figure 1 as an example. we can refer to the gray dot in the figure in various ways: as shown in the figure, there is an optimum size of receptive field when fusing the features of these expressions with the visual features. although the small receptive field (in the second panel to the left) matches the expression a gray dot, it does not capture information about the triangle consisting of three dots to the upper left. conversely, the largest receptive field (in the panel to the right) includes the triangle, but contains too much information to determine the color of the gray dot. thus, linguistic and visual features have an optimum size of receptive field for fusion. few existing models, however, use fusion of linguistic features with visual features with different receptive field sizes. this is possibly because major datasets for referring expression comprehension, for example, @xcite ; @xcite ; @xcite , use photographs and weigh expressions related to object category more often than positional relationships. @xcite ; @xcite ; @xcite reveal that people often use group-based expressions (relative positional relationships of multiple objects) when there is no clear difference between objects; therefore, these expressions are not so unusual. further investigation should be done on methods that handle referring expressions based on positional relationships. for this reason, we focus on the onecommon corpus @xcite , a recently proposed corpus on a visual dialogue game using composite images of simple figures. it captures various expressions based on positional relationships, such as group-based expressions, as shown in figure 2 . in this paper, we introduce a neural network architecture for referring expression comprehension considering visual features with different sizes of the receptive fields, and evaluate it on the onecommon task. our structure combines feature pyramid networks (fpn) @xcite and feature-wise linear modulation (film) @xcite and modulates visual features with different sizes of the receptive fields with linguistic features of referring expressions. fpn is an architecture that uses each layer of the hierarchical convolutional neural network (cnn) feature extractor for object detection; in the onecommon framework, two players observe slightly different views due to parallel shift. the game requires them to create common ground about the views through free conversation and identify the same dot. we show part of an utterance and underline some expressions that refer to an object or a group. to confirm the broad applicability of our architecture, we further evaluate it on another task, which is expected to require the ability of object category recognition more than onecommon does because it uses photographs. we find that our architecture achieves better accuracy in these tasks than some existing models, suggesting the advantage of fusion of linguistic features with multiple visual features that have different receptive fields. the contributions of this paper are as follows: ## dialogue history object retrieval the main focus of this paper is the task of predicting the final object selected by the speaker given a dialogue history, a scene image, and candidate objects in the image. a dialogue history consists of a list of speaker and utterance pairs. we consider dialogues where speakers switch every turn. candidate objects are indicated by bounding boxes in the image. some task instances provide additional information, such as object categories. here, we call this task dialogue history object retrieval. onecommon target selection task onecommon is a dialogue corpus for common grounding. it contains 6,760 dialogues from a collaborative referring game where two players are given a view that contains 7 dots, as shown in figure 2 . dots have four attributes: x/y coordinates on a plane, size, and color. only some dots are seen in common because the centers of the players' views are different. the goal of the game is to select the same dot after talking. target selection is a subtask of the game, requiring prediction of the dot that a player chose based on a given player's view and dialogue history. guesswhat?! guessor subtask guesswhat?! @xcite is a game related to multimodal dialogue. two players play the roles of oracle and questioner. they are given a photo and the oracle mentally selects an object. then, the questioner asks the oracle yes-or-no questions to guess the object. the goal of the game is to select the object at the end of a question sequence. a published collection of game records consists of 150,000 games with human players, with a total of 800,000 visual question-answer pairs on 66,000 images extracted from the ms coco dataset @xcite . the guesser subtask is to predict the correct object from 3-20 candidate objects based on a given photo and set of question-answer pairs. candidate information includes bounding boxes and object category. in addition to dialogue history object retrieval, there is an increasing amount of research on task design for visual dialogue games that require unique common understanding. for example, in the photo-book dataset @xcite , two participants are presented with multiple images, and they predict whether an image is presented only to them or also to the other person through conversation. ## related work this section first describes an overview of the models for referring expression comprehension and then gives some details about models related to the onecommon corpus and guesswhat?!. ## preliminary we introduce two prerequisite architectures to describe our proposal. ## proposed method our architecture consists of language-conditioned feature pyramids (lcfp) for general feature extraction and a feature extractor for specific tasks, as shown in figure 3 . in this section, we describe lcfp and the following structure for dialogue history object retrieval. ## experiments we first validate the advantage of our architecture on two tasks in dialogue history object retrieval described in section 2. we then investigate the cause of the advantage through ablation studies. common text processing we consider dialogue history as a text that starts with task name followed by @xmath0, with a sequence of utterances and @xmath1. each utterance is interposed between a speaker token, <@xmath2, and an end-of-sequence @xmath3. tokenization of utterances is different for each task. ## discussion finally, this section focuses on linguistic expressions. we discuss the effect of our architecture on group-based referring expressions and our first intuition regarding the relationship between expression and receptive fields using onecommon. ## conclusion to improve referring expression comprehension, this paper proposes a neural network architecture that modulates visual features; the visual features have different sizes of receptive fields in each hierarchy extracted by cnns with linguistic features. as our architecture affine transforms visual features with linguistic features, it requires a lower calculation cost than methods that generate convolution kernels. our evaluation of referring expression comprehension tasks on two visual dialogue games demonstrates the model's advantage in the understanding of referring expressions and the broad applicability of our architecture. ablation studies support the importance of multiple fusion. we expect that hierarchical visual information is also important to generation. however, our architecture is difficult to directly apply to referring expression generation because it outputs modulated feature maps. therefore, the future direction is to extend our architecture to language generation.