L IFT - THE - FLAP : WHAT , WHERE AND WHEN FOR CONTEXT REASONING
Anonymous authors
Paper under double-blind review
A BSTRACT
Context reasoning is critical in a wide variety of applications where current inputs need to be interpreted in the light of previous experience and knowledge.
Both spatial and temporal contextual information play a critical role in the domain of visual recognition.
Here we investigate spatial constraints (what image features provide contextual information and where they are located), and temporal constraints (when different contextual cues matter) for visual recognition.
The task is to reason about the scene context and infer what a target object hidden behind a ﬂap is in a natural image.
To tackle this problem, we ﬁrst describe an online human psychophysics experiment recording active sampling via mouse clicks in lift-the-ﬂap games and identify clicking patterns and features which are diagnostic for high contextual reasoning accuracy.
As a proof of the usefulness of these clicking patterns and visual features, we extend a state-of-the-art recurrent model capable of attending to salient context regions, dynamically integrating useful information, making inferences, and predicting class label for the target object over multiple clicks.
The proposed model achieves human-level contextual reasoning accuracy, shares human-like sampling behavior and learns interpretable features for contextual reasoning.
I NTRODUCTION
The tiny object on the table is probably a spoon, not an elephant.
Objects do not appear in isolation.
Instead, they co-vary with other objects, their sizes and colors usually respect regularities with respect to nearby elements, and objects tend to appear at speciﬁc locations within a scene.
Humans exploit these contextual associations.
Contextual analyses based on the statistical summary of object relationships, provide an effective source of information for perceptual inference tasks, such as object detection (Torralba (2003); Park et al.
; Hoiem et al.
; Torralba et al.
; Liu et al.
(2018b)), scene classiﬁcation (Gonfaus et al.
; Torralba et al.
; Yao et al.
(2012)), semantic segmentation (Yao et al.
(2012)), and visual question answering (Teney et al.
An example of how contextual information is incorporated during object recognition is lift-the-ﬂap books, where a ﬂap covers part of the page.
Children make guesses about what is behind the ﬂap based on the context and check their answers by lifting the ﬂap (Figure 1a).
Here we investigate what image features matter for contextual reasoning and where those features are with respect to the target object of interest.
Furthermore, scene interpretation in humans involves a sequence of eye movements (Zhang et al.
(2018)), each one of these image samples providing additional context to inform interpretation of the contents of the next location.
Therefore, we also investigate whether integration of scene information over time ( ”when” information) matters for context reasoning.
To tackle the problem of contextual reasoning, we introduce the lift-the-ﬂap task and conduct online psychophysics experiments where subjects make mouse clicks while they explore important contextual cues to identify a hidden target (Figure 1b).
We investigate the contextual reasoning strategies observed from human active clicking patterns.
As a proof of concept, we propose a recurrent attention model (ClickNet), to automatically learn these contextual reasoning strategies.
The model guides attention towards regions with informative context, decides where to click on the image, and makes inferences about the target behind the ﬂap.
The learnt clicking patterns and predicted class labels by ClickNet share remarkable similarities with human behavior.
[Figure]
subjects typed a noun describing the hidden target object
The were conducted online using Amazon Mechanical Turk Turk ( on 100 subjects, 50 trials subject.
Figure 3 also shows results for a variation of the experiment conducted in the lab while tracking movements.
R ELATED W ORKS 2. R OLE OF C ONTEXT IN H UMAN V ISION
Contextual information affects the efﬁciency of several visual processes (Auckland et al., 2007;
Biederman et al., 1982; Hollingworth, 1998; Bar & Aminoff, 2003; Goh et al., 2004; Aminoff et al., 2006), such as object recognition (Auckland et al., 2007), object detection (Biederman et al., 1982; Hollingworth, 1998), visual working memory (Friedman, 1979; Aminoff et al., 2006) and visual search (Henderson et al., 1999).
Objects appearing in a familiar background can be detected more accurately and processed more quickly than objects appearing in an incongruent scene.
Here we focus on what visual features contribute to contextual reasoning, which parts of image regions attract human attention to make inferences, and the dynamic sequence of directed sampling needed to idenﬁty a hidden target.
2. R OLE OF C ONTEXT IN C OMPUTER V ISION
Contextual reasoning about objects and relations is critical to machine vision.
In fact, many object recognition studies using natural image datasets such as ImageNet, rely implicitly but strongly on contextual feature regularities Geirhos et al.
; Brendel & Bethge (2019).
Some works also show that these models can fail when objects are placed in an incongruent context (Beery et al.
Dvornik et al. ; Choi et al.
These studies motivate examining the role of pure context without any object information in situations of complete occlusion (lift-the-ﬂap); here we provide human and computational benchmarks for object inference using exclusively context information.
Several studies use context to improve object detection (Park et al., 2010; Hoiem et al., 2005; Torralba et al., 2010; Liu et al., 2018b; Chen et al., 2018b). Context can take the form of global scene context (Torralba et al., 2010), ground plane estimation (Park et al., 2010), geometric context (Hoiem et al., 2005), relative location Desai et al.
(2011), 3D layout (Lin et al., 2013), and spatial support and geographic information (Divvala et al., 2009).
Researchers proposed Conditional Random Field (CRF) models that reason jointly across multiple computer vision tasks in image labeling and scene classiﬁcation Gonfaus et al.
; Yao et al.
; Ladicky et al. (2010). Context can lead to improved performance in both object detection and semantic segmentation tasks
Mottaghi et al.
Neural network architectures incorporating contextual information have been used in object priming (Torralba, 2003), place and object recognition
(Wu et al., 2018; Torralba et al., 2005), object detection (Liu et al., 2018b; Chen et al., 2018b), and visual question answering (Teney et al., 2017).
Here we focus on developing a biologically inspired computational model for contextual reasoning that can automatically and dynamically sample image regions of interest, integrating information in space and time to make inferences about a hidden object.
Additionally, we compare the model’s performance against human behavior in the same task.
Several interesting approaches have combined graphical models with deep neural networks for structural inference, primarily in structured prediction tasks (Marino et al., 2016; Choi & Savarese, 2012;
Chen et al., 2018a; Teney et al., 2017;
Hu et al., 2016; Battaglia et al., 2016;
Xu et al., 2017; Chen et al., 2018b; Choi et al., 2012).
Hu et al.
(2016) designed a structured model to improve classiﬁcation performance by leveraging relations among scenes, objects, and their attributes.
A structured inference model is also used in (Choi & Savarese, 2012; Deng et al., 2016) to analyze relations in group activity recognition.
Several works, like Structural-RNN Jain et al. (2016) and
Interaction Net Battaglia et al.
(2016), combine spatiotemporal graphs and sequence learning.
These works assume full contextual information is available, while in our experiment we consider only partial contextual information that is sequentially revealed after each mouse click.
DeepLab sends the response at the ﬁnal layer of a CNN to a CRF model for semantic image segmentation Chen et al. (2018a).
Subsequently, Schwing & Urtasun (2015); Zheng et al.
(2015) transformed the CRF model into a Recurrent Neural Network.
Breaking away from these studies where graph optimization is performed globally, our proposed model selects important visual features using an attention mechanism and integrates partial information over multiple steps, which is computationally more efﬁcient and accurate in the current task (Section 5).
L IFT - THE - FLAP TASK 3. H UMAN B EHAVIORAL E XPERIMENTS
Subjects were presented with a natural image where one object was hidden by a rectangular black box and everything else was blurred.
They were allowed a ﬁxed number of mouse clicks betweenand 8, each click revealed part of the image in high resolution.
To minimize overlap between clicks, we enforced subjects to click at places at least 10 pixels apart from all previous clicks.
After the target number of clicks, they had to provide a single word to describe the object hidden behind the black box (Figure 1b).
The clicking experiments were run on Amazon Mechanical Turk Turk (2012).
The stimulus set consisted of 573 images from the test set of the MSCOCO Dataset (Lin et al., 2014), spanning 80 object categories.
This dataset has been widely used for object recognition and detection studies (Lin et al., 2014).
We constrained the stimulus set to have a uniform distribution of 6 - 8 target objects per category.
To avoid any potential memory effects, subjects were only exposed to each image once.
The trial presentation order was randomized.
3. G ROUND T RUTH R ESPONSES
In contrast to experiments where subjects are forced to perform N-way categorization (e.g., Tang et al.
, here there were no constraints on how subjects describe the hidden object.
This probing mechanism was implemented for two reasons: (i) it is difﬁcult for humans to memorizeobject classes and there could be non-uniform memory effects impacting the results; (ii) presenting humans with an 80-choice question could introduce biases in their inference processes.
We could not simply use the 80 category labels to evaluate performance because subjects could use other words or synonyms and we are interested in context reasoning rather than language abilities.
Therefore, to evaluate humans’ performance, we separately collected a distribution of ground truth answers for each hidden target by presenting to 10 other subjects, who did not participate in the main task, the same set of images with the target objects highlighted by a bounding box but not hidden.
During the lift-the-ﬂap task, a response was considered to be correct if it matched any of the ground truth labels, allowing for plurals and misspellings.
3. E VALUATION M ETRICS
We introduce several evaluation metrics to measure contextual reasoning accuracy and to compare the consistency of mouse clicking patterns between humans and computational models.
We evaluated ClickNet on the MSCOCO Dataset using the typical classiﬁcation accuracy measure.
In Fig 5b, we report top-1 classiﬁcation accuracy as a function of context-object ratio .
The context-object ratio is deﬁned as the total area of the image excluding the hidden target divided by the hidden target object size.
For example, a context-object ratio of 1 implies that the size of the black box and the size of the contextual information is the same (see Figure 5b for example images with different context-object ratios).
[Figure]
To measure the degree of spatiotemporal consistency between two mouse clicking patterns (human versus human; or human versus computational models), we introduced two measures. First, we computed the minimum Euclidean distance between the sequence of clicks in each trial, regardless of order.
The smaller the median in the distribution of distances, the more similar the two mouse clicking patterns are.
Second, we introduced a click sequence score, as originally deﬁned in evaluating eye ﬁxation sequence consistency Madsen et al.
; Zhang et al.
Here, we use the same metrics to compare the spatial-temporal consistency between click sequences.
The larger the click sequence score, the more similar the sequences are.
C LICK N ET A RCHITECTURE
We propose a recurrent neural network for context reasoning (ClickNet), extending previous work on image captioning
(Xu et al., 2015).
ClickNet integrates attention-modulated context information over multiple clicks, makes a decision about the next click location based on the attention map, and infers the class label of the hidden target after every click (Figure 2).
As in the human psychophysics experiment (Figure 1b), ClickNet is ﬁrst presented with a blurred image I 0 , which is the original image I with uniform gaussian blur and where the target object is covered by a black box.
ClickNet makes the ﬁrst attempt to predict a class label y
out of a pre-deﬁned set of C object classes and decides its ﬁrst click location m 1 .
In every trial, over a series of T clicks, the input image I t to ClickNet gets updated with circular regions of constant radius R centered at all previous click locations M = { m 1 , ...m T } , revealed in its original resolution in I .
The black box is constant and none of its content is ever revealed, even if the model opts to click within the box or if the circle centered on the click encompasses part of the box.
4. C ONVOLUTIONAL F EATURE E XTRACTION
At each time t where t ∈ { 0 , ..., T } , ClickNet takes I t as input and uses a feed-forward convolutional neural network to extract feature maps a t .
We use the VGG16 network (Simonyan & Zisserman, 2014), pre-trained on ImageNet (Deng et al., 2009).
Consistent with previous works showing that these feed-forward convolutional neural networks for visual recognition tasks are vulnerable to domain shift such as image blur and noise when those transformations do not exist in the images used for training (Liu et al.
(2018a)
; Dodge & Karam (2016)), we observe that it is necessary to
(see details in Section A). To focus on speciﬁc parts of the image and select features at those locations, we have to preserve the spatial organization of features; thus, ClickNet uses the output feature maps at the last convolution layer of VGG16.
A feature vector a ti of dimension D represents the part of the image I t at location i , where i = 1 , .., L and L =
W × H and W and H are the width and height, respectively, of the feature map:
[Equation]
4. A TTENTIONAL M ODULATION
We use a “soft-attention” mechanism as introduced by Ba et al.
(2014) to compute “the context gist” (cid:98)
z t on I t
(Figure S1).
For each location i in a t , the attention mechanism generates a positive scalar α ti , representing the relative importance of the feature vector a ti for context reasoning.
This relative importance α ti depends on the feature vectors a ti , combined with the hidden state at the previous step h t − 1 of a recurrent network described below.
[Equation]
where A h ∈ R 1 × n and A a ∈ R 1 × D are weight matrices initialized randomly and to be learnt.
Instead of addition, an alternative is to use element-wise multiplication: A h h t − 1 ◦
A a a ti .
Empirically, we did not observe any performance difference with such a multiplicative term (see Section A). Because not all attended regions might be useful for context reasoning, the soft attention module also predicts a gating vector β t from the previous hidden state h t − 1 , such that β t determines how much the current observation contributes to the context vector at each location: β t = σ ( W β h t − 1 ) , where W β ∈ R L × n is a weight matrix and each element β ti in β t is a gating scalar at location i .
As also noted by Xu et al. (2015), β t helps put more emphasis on the salient objects in the images.
Once the attention map α t and the gating scale β t are computed, the model applies the “soft-attention” mechanism to compute (cid:98) z t by summing over all the L regions in the image:
[Equation]
The next click location m t +1 corresponded to the maximum on the attention map:
[Equation]
The attentional module is smooth and differentiable and ClickNet can learn all the weight matrices in an end-to-end fashion via back-propagation.
4. R ECURRENT C ONNECTIONS USING LONG SHORT - TERM MEMORY (LSTM)
We use a long short-term memory (LSTM) network to output a predicted class label y t based on the previous hidden state h t − 1 and the context gist vector (cid:98) z t for I t
(Figure S2).
Our implementation of
LSTM closely follows Zaremba et al.
(2014) where T s,t : R s → R t deﬁnes a linear transformation with learnable parameters.
The variables i t , f t , c t , o t , h t represent the input, forget, memory, output and hidden state of the LSTM respectively:
[Equation]
[Equation] [Equation] where n is the dimensionality of LSTM, σ is the logistic sigmoid activation, and (cid:74) indicates element-wise multiplication.
To cue ClickNet about the location of the hidden target, we initialize the memory state c 0 and hidden state h 0 of the LSTM based on a binary mask that contains zeros everywhere and ones in the hidden target location.
Speciﬁcally, c 0 and h 0 are predicted by an average of all feature vectors a 0 over all
L locations with two separate linear transformations W c 0 ∈ R n × D and W h 0 ∈ R n × D :
[Equation]
To predict the class label y t of the hidden target, the LSTM computes a classiﬁcation vector where each entry denotes a class probability given the hidden state:
[Equation]
where L h ∈ R C × n is a matrix of learnt parameters initialized randomly.
4. T RAINING AND I MPLEMENTATION D ETAILS
We trained ClickNet end-to-end by minimizing the cross entropy loss between the predicted label y t at each time step t and the ground truth label x .
In contrast with previous work where there is a regularization term (cid:80) t α ti = 1 in the loss function to encourage the model to acquire as much information all over the image by exploration (Xu et al.
, we did not ﬁnd a signiﬁcant performance increase in the lift-the-ﬂap task when adding such a term (see Section A).
Hence, let ClickNet freely explore the image without any such constraints.
Since we do not impose any constraints on the next click locations, ClickNet might make decisions to click at the previously clicked locations..
The loss function is deﬁnef by:
[Equation]
We used all images from the MSCOCO training set for training and validating all models.
On every training image, each object can be blocked out as the hidden target.
The input image size to ClickNet was 400 × 400 pixels.
We used a Gaussian ﬁlter of size 51 × 51 with variance 64 pixels to blur the images.
The radius R of the circular region revealed by each click was 55 pixels.
As in the human psychophysics experiments (Fig 1b), in each trial, we set the total number of time steps T = 8 for training ClickNet (ClickNet predicts the label after the 1st click at T = 1 ).
The dimension of the
LSTM module was n = 512 .
The feature maps extracted from the last convolution layer was of size 2048 × 28 × 28 , and the total number of locations was L = 28 × 28 = 784 .
The Adam optimizer (Kingma & Ba, 2014) was used with a learning rate of 10 − 4 to ﬁne-tune the VGG16 network, and a learning rate of 4 × 10 − 4 to train the attentional module and the LSTM module.
The network was developed in Pytorch, based on (Xu et al., 2015).
All source code for our proposed architecture, and the data from the psychophysics experiments will be released publicly upon publication.
4. V ARIATIONS OF P ROPOSED N ETWORK A RCHITECTURE AND C OMPARATIVE M ETHODS
Previous work has shown that it is possible to augment vision systems with human perceptual supervision on many difﬁcult computer vision tasks, such as Vondrick et al.
; Kovashka et al.
One central goal in our study is to investigate what, where, and when matter for human contextual reasoning in the lift-the-ﬂap game and whether these factors could help improve current machine learning algorithms.
We now introduce two variations of ClickNet with human inputs at the testing stage:
ClickNet-humanclick.
Instead of clicking at the location with highest activation value on the attention map predicted by ClickNet, we substitute the input with human clicking images.
ClickNet-RandPrior.
We observe strong spatial bias in the human clicking patterns where most of the clicks tend to be nearby the hidden target (see Sec 5.4).
To test whether this is sufﬁcient to account for human behavior, we generated random clicks surrounding the black bounding box and used the resulting images with a strong spatial clicking prior as inputs to ClickNet.
We set the same criterion to consider two clicks as overlapping as in the human behavioral experiments; the probability that two clicks overlap in this model is 0.0065. A cc u r a cy ( % ) C li ck N e t C li ck N e t − hu
m an c li ck
C li ck N e t − R
and
P r i o r
V G G − A tt en t i on
V G G − A tt en t i on − hu m an c li ck
V G G − hu m an c li ck
V G G − R ando m (
[Figure]
HumanClick−1HumanFixation−200 ms    80 (b)
A cc u r a cy ( % )
C li ck N e t
C li ck N e t − hu
m an c li ck
C li ck N e t − R
and
P r i o r
V G G − A tt en t i on
V G G − A tt en t i on − hu m an c li ck
V G G − hu m an c li ck
V G G − R ando m
HumanClick−8HumanFixation−1600 ms
Column 4 and 8 show the attentional map predicted by ClickNet.
Brighter regions denote higher attentional values.
To gauge how much contextual information helps object recognition, it is interesting to isolate the effect of context and object, analyze them separately and then study both combined.
The lift-the-ﬂap task provides a benchmark to study the contextual reasoning problem alone.
In addition, we evaluate
ClickNet when both the object region and the context are revealed to ClickNet (an upper bound) and when only the object region (the tightest bounding box encompassing the object) is revealed to
ClickNet (Section D).
To study the role of attention and recurrent connections, we introduced two ablated models.
Variations of VGG16.
One intuitive way of solving the context reasoning problem is to use a feed-forward object recognition network pre-trained on ImageNet, e.g. VGG16 (Simonyan &
Zisserman, 2014), and ﬁne-tune it to classify the hidden target on MSCOCO dataset.
During training, the input to the network was an image where one object on the image was randomly covered with a black bounding box.
We tested the performance of this alternative model on the 573 images selected for human psychophysics experiments with different input variations: human clicking images ( VGG-humanclick ), the blurred images ( VGG-Blur ), the full-resolution images ( VGG-Fullres ) and images with random clicks ( VGG-Random ). [Figure] A c t u a l L a b e l s b a s e b a ll b a t b a s e b a ll g l o v e s k a t
e
b o a r d s u r f b o a r d t e n n i s r a c k e t b o tt l e w i n e g l a s s c u p f
o r k k n i f e
[Figure] [Figure] VGG-Attention. Previous work has demonstrated the efﬁciency of attention in computer vision tasks (Nguyen et al., 2018), such as question answering and image captioning Xu et al.
To study the effect of attention in contextual reasoning, we added an attention module to the end of
VGG16.
To make the complexity of the architecture comparable with ClickNet, we added the same number of fully connected layers as in the LSTM module.
As in ClickNet, we used the location with the highest activation value on the attention map to predict the next click.
VGG-Attention takes the updated image as input and iteratively predicts the hidden target label.
In contrast to ClickNet, the network is feed-forward and there is no incorporation of past information integrated over clicks.
We also test VGG-Attention with human clicks ( VGG-Attention-humanclick ) and randomly generated click locations with strong spatial priors ( VGG-Attention-RandPrior ).
We also considered several competitive baselines and existing methods of modeling temporal dynamics.
These include Human-ﬁxations , SVM-category , SVM-category-instances , Hidden
Markov Model (HMM) and DeepLab-Conditional Random Field (CRF) .
See Section B for these comparison methods.
R ESULTS 5. W HAT : REGION SELECTION VIA ATTENTION AND PRIOR INFORMATION
Subjects inferred the identity of the hidden target object with 36 . 7 ± 0 .
9% accuracy afterclick (Fig 3, gray horizontal line).
Performance showed a small, but signiﬁcant improvement when allowing subjects 8 clicks, reaching 48 .
4 ± 0 .
9% (Fig 3, black horizontal line, p < 10 − 9 , two-tailed t-test, t=-6, df=2593).
In-lab experiments corroborated these results showing accuracies of 48 .
1 ± 0 .
9% after 200 ms exposure and 56 .
3 ± 0 .
9% after 1,600 ms exposure to the images.
The same images that subjects saw were used to evaluate ClickNet (Figure 3).
The ClickNet model showed a close approximation to human performance in the mturk experiments, reaching a topclassiﬁcation performance of 34 .
9 ± 0 . 9% for 1 click and 48 . 87 ± 0 .
9% for 8 clicks.
In both cases, performance was only slightly lower than human performance in 1 click ( p = 0 .
47 , two-tailed t-test, t=0.73, df=1828) and 8 clicks ( p = 0 .
86 , two-tailed t-test, t=0.17, df=1909).
Performance for intermediate numbers of clicks is shown in (Figure S4).
For all the computational models, random guessing would yield accuracy = 1 .
The worst performing model, VGG-Random, yielded performance above chance levels, emphasizing that even small amounts of high-resolution contextual data at arbitrary locations can help solve the problem.
Yet, VGG-Random was well below ClickNet’s performance ( p < 10 − 15 , two-tailed t-test, t=21, df=3028).
Adding attention to the model (VGG-Attention) yielded only minimal improvement.
An important ingredient missing in VGG-Random and VGG-Attention is the informed location of the clicks.
Humans and ClickNet do not sample the image randomly, but rather explore informative locations.
Figure 4 shows qualitative examples of clicking patterns from humans and ClickNet.
Both humans and ClickNet attend to salient regions on the images.
For example, clicks often occur near birds in the ﬁrst example and near trafﬁc lights in the second example.
Accordingly, substituting the random clicks for the human clicks into the VGG models (VGG-humanclick and VGG-Attention-humanclick) yielded a large performance boost.
Conversely, substituting the ClickNet clicks with random clicks leads to large drop in performance when there is only 1 click, even when we artiﬁcially try to boost performance by constraining the clicks to be near the hidden target object (ClickNet-RandPrior).
This effect is also evident with 2 clicks and 4 clicks (Fig. S4), but disappears with 8 clicks because there is already a lot of high resolution information in the image surrounding the hidden target, and ClickNet can integrate information over time to capitalize on it.
Interestingly, the ClickNet sampling clicks are sufﬁciently close to human clicks that substituting the ClickNet clicks with human clicks does not improve performance (ClickNet-humanclick).
We considered several other comparative models (Fig. S4).
Interestingly, using just a few clicks, ClickNet reaches performance that is essentially equivalent to that of VGG using a full resolution version of the entire image.
Other comparative models (VGG-Blur, SVM-category,
SVM-category-instances, HMM, DeepLab-CRF) showed above chance performance but their accuracies were well below ClickNet.
5. W HAT : THE MORE , THE MERRIER
To investigate how much context information is needed to enhance recognition, we evaluated accuracy as a function of context-object ratio (Fig 5b, Sec 3.3).
Images with higher context-object ratio contained more context information for inference, and yielded higher accuracy both for humans and models.
Similarly, accuracy improved with increasing numbers of clicks (Fig 3 and Fig 5b).
It is not just the quantity of context, but also the speciﬁc quality of contextual information that matters.
In the real world, objects do not tend to appear in isolation but rather they co-vary with other objects.
As ClickNet explores more regions on the image, it integrates information at previous clicked locations and learns associations of objects.
The pattern of mistakes made by ClickNet is indicative of those associations (Fig 5a and Fig S5).
ClickNet often makes “reasonable” wrong guesses when there is ambiguity in context reasoning, as humans do.
For example, knife tends to be associated and therefore confused with spoon, fork, and wine glass, but knife seldom co-occurs with baseball bat or skateboard in these images.
5. W HERE : CONSISTENCY OF HUMAN AND MODEL CLICKS
We hinted at the similarity in the clicking patterns between humans and ClickNet based on the accuracy of the ClickNet-humanclick and ClickNet-RandPrior models in Fig 3.
To more directly assess whether ClickNet learned to sample the image to gather information about areas of contextual relevance, we directly quantiﬁed the similarity in clicking patterns (Figure 6 and S6).
To interpret the distance between human and model clicks, we computed the degree of human-human consistency in the clicking patterns.
The clicking patterns of ClickNet were overall similar to those made by humans.
The model clicks were still different from the consistency between two humans (8 clicks: p < 10 − 15 , two-tailed t-test, t=36, df=29542); yet, the model clicks were more similar to human clicks than random clicks (8 clicks: p < 10 − 15 , two-tailed t-test, t=44, df=35310).
5. W HERE : TENDENCY OF CLICKING NEARBY THE TARGET
There was a strong spatial bias towards clicking near the target for both humans and ClickNet (e.g., Figure 4).
To quantify this spatial bias, we computed the Euclidean distance between the clicked locations and the center of the bounding box, normalized by the diagonal of the bounding box (Figure 6c).
Humans tended to click within one diagonal distance of the target box.
Interestingly, although ClickNet does not take any human supervision during training, ClickNet still learned to capture the tendency of clicking near the target.
We asked whether this spatial bias in sampling behavior is sufﬁcient to explain performance in this task in a modiﬁed version of ClickNet.
We removed the clicks dictated by the attention module and instead forced the clicks to be randomly distributed while still respecting the spatial distribution in Fig 6c (ClickNet-RandPrior).
Both ClickNet and ClickNet-humanclick surpassed ClickNet-RandPrior by 28.3% and 7% over all click conditions respectively (Figure 3).
Similar results were obtained when using only the VGG [Figure] architecture: VGG-Attention-humanclick was 16% better than VGG-Attention-RandPrior (Fig S4).
Therefore, the spatial bias in clicking behavior cannot account for performance: sampling for context reasoning involves more than clicking near the target.
5. W HEN : TEMPORAL INTEGRATION HELPS RECOGNITION
Several lines of evidence support the importance of the recurrent network in the LSTM module in ClickNet.
ClickNet outperformed the competitive baselines and state-of-the-art comparative methods to make inferences (Figure 3 and Fig S4).
We tested whether the co-occurrence of object categories or the number of objects per category present in the image would be sufﬁcient for context reasoning (SVM-category and SVM-category-instances).
Even though these alternative models were exposed to full contextual information on the image and assumed perfect labeling of all objects in the image (except for the hidden target object) there was still a large overall performance drop in performance with respect to ClickNet.
Moreover, graphical models for inference, such as Hidden Markov Model and DeepLab with Conditional Random Field (Sec 4.5) failed to reach ClickNet’s accuracy in this task (Fig S4).
The ablation studies eliminating the LSTM module further support the role of integrating information over multiple clicks in this task, as evidenced by the observation that ClickNet outperformed the VGG-Attention model.
D ISCUSSION
Here we quantitatively studied the role of contextual information in visual recognition in human observers and computational models in a task that involved inferring the identity of a hidden target object.
Context inﬂuenced recognition based on the amount of context, the speciﬁc location of contextual cues, and the dynamic integration of salient visual features.
We introduced a recurrent neural network model that combines a feed-forward visual stream module that extracts image features in a dynamic fashion, combined with an attention module to prioritize different image locations and select the next click location, and a recurrent LSTM module that integrates information over time and produces a label for the hidden object.
Surprisingly, even though the model lacks the expertise that humans have in interacting with objects in their context, the model approximates human sampling behavior (Click consistency in spatial domains (Euclidean distance distribution and spatial bias in Figure 6) and temporal domains (click sequence similarity score in Figure S6)) and reaches almost human-level performance in this contextual reasoning task (contextual reasoning accuracy in Figure 3 and Figure 5b) and reaches almost human-level performance in this contextual reasoning task.
The model opens the doors to examine more complex form of reasoning about scenes and how to integrate sequential sampling with prior knowledge.
A PPENDIX
We provide supplementary ﬁgures and materials here.
All labels in supplementary ﬁgures and tables are pre-ﬁxed with letter S in front. A I NTRODUCTION TO ADDITIONAL ABLATED MODELS AND CONTROLS
ClickNet-noFineTune.
As explained in Section 4.1, there is a domain shift problem in feed-forward convolutional neural networks for visual recognition tasks when trained using one type of images (e.g., without blurring) and tested using other types of images (blurred).
To quantify this effect, here we introduce ClickNet-noFineTune where the weights of the feature extractor module were loaded from the VGG16 network pre-trained on ImageNet and were ﬁxed during training.
The results are reported in Figure S4. There was a performance drop of around 13% with 1 click and 15.1% withclicks, compared with ClickNet. ClickNet-WithAlphaLoss. Previous work included an additional alpha loss term in equation 9 (Xu et al.
To study the effect of this alpha loss term as explained in Section 4.4, we introduced
ClickNet-WithAlphaLoss .
The updated loss function becomes:
[Equation]
We chose the hyperparameter λ = 2 in this equation.
The other training speciﬁcations remain the same.
The result is reported in Figure S4.
There was a performance drop of around 2% with 1 click and 4% with 8 clicks compared with ClickNet.
Results presented throughout the main text do not include this alpha term.
ClickNet-AttenMulti.
In Section 4.2, we additively combined h t − 1 and a ti .
An alternative way of computing attention is to combine h t − 1 and a ti via multiplication.
As mentioned in Section 4.4, we introduced ClickNet-AttenMulti and empirically test this possibility.
The attention computation in
Equation 2 was updated as follows:
[Equation]
where ◦ denotes element-wise multiplication.
The results are reported in Figure S4.
There was a performance drop of around 1% with 1 click and 4% with 8 clicks compared with ClickNet.
B I NTRODUCTION TO COMPETITIVE BASELINES AND EXISTING METHODS
Human-ﬁxations.
We were concerned about the variable viewing conditions in the MTurk experiments.
Therefore, we conducted in-lab psychophysics measurements as a benchmark.
In the in-lab experiment, after 500 ms ﬁxation, a bounding box with a ﬁxation cross in the center presented for 1,000 ms indicated the target position and cued subjects to attend to the hidden target location.
To ensure that in-lab subjects paid attention to the hidden target location, we recorded their eye movements using an EyeLink D1000 system (SR Research, Canada).
The image with the black box was shown for 200, 400, 800, or
1600 ms. Subjects freely moved their eyes; after stimulus offset, subjects said a single noun describing what the hidden target was.
We recruited 4 naive subjects ( to 24 years old, 2 female), each one participating in 573 trials.
SVM-category.
To study the effect of object co-occurrences, we used a binary vector of size 1 × C as input to a classiﬁer, where the i th entry is 1 if there was an object from category i th in the image and 0 otherwise.
We assume that the model had perfect information about all the object labels (all objects were visible except for the hidden target).
A multi-class support vector machine (SVM) classiﬁer was used to predict the hidden target based on this vector.
SVM-category-instances.
Extending SVM-category, we constructed a vector of size 1 × C where the i th entry contained the number n of instances of the i th category in the image.
A multi-class
SVM classiﬁer was used to predict the hidden target based on this vector.
Hidden Markov Model (HMM).
To study the temporal dynamics over multiple clicks, we considered a Hidden Markov Model where we used all the training images in MSCOCO dataset to calculate the co-occurence matrix of size C × C as the transition probability matrix.
We use normalized uniform vector of size 1 × C as the initial probability.
We ﬁne-tuned VGG16 on the
MSCOCO dataset and used it for classifying the cropped region at the human clicked locations where the classiﬁcation vector contributes to emission matrix.
The Viterbi decoding algorithm
Blunsom (2004) was used for making inferences about the hidden target.
DeepLab-Conditional Random Field (CRF)
One interesting solution to reason about the hidden target is to run state-of-the-art semantic segmentation algorithms and use majority voting on the predicted labels over all pixels in the bounding box.
We used the instantiation in DeepLab-CRF (Chen et al., 2017).
C U PPER BOUND : PERFORMANCE INCLUDING OBJECT INFORMATION
Our paper focuses on contextual reasoning with the target object completely occluded.
It is interesting to provide an upper bound for performance where the object regions are revealed to
ClickNet using the same images ( ClickNet-ObjRevealed .
The setup is as in the lift-the-ﬂap problem but the initial condition is different:
ClickNet-ObjRevealed is ﬁrst presented with the tightest bounding box of the target object with the target revealed in high resolution and context blurred, and then the model has to click on the image in a sequential manner and all the previous clicked regions are deblurred.
We ﬁne-tuned the original ClickNet on this task and tested it on the same set of images but with the target objects revealed.
The results are reported in Figure S4.
Withclick, ClickNet-ObjRevealed showed an accuracy of 66 .
1 ± 0 .
As more context is revealed (8 clicks), the accuracy of ClickNet-ObjRevealed increased to 71 .
7 ± 0 .
In addition, we include another baseline ClickNet-ObjOnly where only the object is presented to
ClickNet-ObjOnly in high resolution from the beginning and there is no context at all.
In this case, the clicks have no relevance.
We report the performance in Figure S4.
With one click,
ClickNet-ObjRevealed showed an accuracy of 47 .
3 ± 0 .
93 % and with 8 clicks the accuracy was 47 .
2 ± 0 .
As an additional upper bound comparison point we considered state-of-the-art object detection algorithms.
In the literature, there are many works relying on context for object detection Chen et al.
(2018b)
and Redmon & Farhadi (2018).
Here, we tested YOLO3 Redmon & Farhadi (2018) on these two cases (object revealed and object only).
Since there are no recurrent connections in
YOLO3, the reported results below assume YOLO3 can see full-resolution, i.e. no-blurring.
In the object-revealed case, the accuracy was 65 ± 2 % and in the object-only case, the accuracy was 44 ± 2 %.
There are several interesting observations from these upper bound measurements.
(i) As expected, revealing the object leads to better performance than the lift-the-ﬂap condition with a hidden object.
Yet, remarkably, contextual information above was essentially equivalent to object information alone (compare Figure 3 with the numbers above).
Furthermore, revealing the object only increased performance by about 10-15 % with respect to the context only condition.
(ii)
The accuracy of ClickNet-ObjectRevealed increases over clicks, which further conﬁrms the important role of context in object recognition even when the object is shown.
(iii) ClickNet-ObjectRevealed and ClickNet-ObjectOnly slightly outperform YOLO3 in the object-revealed and object-only cases.
(iv) The summation of accuracies of ClickNet-ObjectOnly and ClickNet is not equal to ClickNet-ObjectRevealed which shows that the combination of context and object recognition is not linear.
(v) ClickNet-ObjectOnly is robust to time changes: although more clicks do not reveal any more information in this case, the recurrent connections did a good job in maintaining its accuracy (instead of forgetting what ClickNet has seen at the zero click).
D I NCONGRUENT CONTEXT
The observation that contextual information can help infer what a completely occluded object suggests that placing objects in a ”wrong” context could impair recognition.
Indeed, several behavioral and computational tests have shown that objects are harder to recognize when they are out of context (Beery et al.
Dvornik et al.
; Choi et al.
We veriﬁed these previous observations in a separate experiments where we constructed images where the objects were placed in incongruent contexts. To be consistent with the lift-the-ﬂap case, we used the same test set from MSCOCO, cropped the objects and pasted them in either a congruent context (Figure S7(a)) or in an incongruent context (Figure S7(b)), as done in previous studies Choi et al. We tested these congruent and incongruent images both on ClickNet and humans.
As expected, and consistent with previous work, we observed that performance in congruent images was higher than performance in incongruent images both for ClickNet (13.8% difference) and humans (18.4% difference).
𝑎 𝑡𝑖 𝑖 ∈ {1,2,…,𝐿} ℎ 𝑡 1×𝑛 1×𝐷 1× 𝛼 𝑡𝑖 𝑖 ∈ {1,2,…,𝐿} argmax 𝑖 𝛼 𝑡𝑖 𝑚 𝑡 Feature vector at each location 𝐴 ℎ1×𝑛 𝐴 𝑎1×𝐷 1× 𝛽 𝑡𝑖 𝑖 ∈ {1,2,…,𝐿} 𝑊 𝛽1×𝑛
Attention value at each location 1×
Gating value at each location
Click location
Attention modulated feature vector at each location 𝛽 𝑡𝑖 𝛼 𝑡𝑖 𝑎 𝑡𝑖 𝑖 ∈ {1,2,…,𝐿} 1×𝐷 𝛼 𝑡1 𝛼 𝑡 𝛼 𝑡𝐿 𝛼 𝑡𝑖 𝛼 𝑡
Attention Map Click Location
[Figure]
multiply sigmoid
[Figure]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
and
[Table]
b j R e v ea l ed
V G G − A tt en t i on
V G G − A tt en t i on − hu m an c li ck
V G G − A tt en t i on − R and P r
i
o r
V G G − F u ll r e s
V G G − hu m an c li ck
V G G − R ando m
V G G − B l u r
SV M − c a t ego r y
SV M − c a t ego r y − i n s t an c e s
H i dden
M a r k o v M ode l
D eepLab − C R F
Mouseclick− (b) ClickA cc u r a cy ( % )
H u m an C li ck
H u m an F
i x a t i on C li ck N e t
C li ck N e t − hu
m an c li ck
C li ck N e t − R and P r i o r
C li ck N e t − N o F i ne T une C li ck N e t − A tt en t i on M u l t i C li ck N e t − W i t h
A l phaLo ss
C li ck N e t − O b j O n l y C li ck N e t − O b
j R e v ea l ed V G G − A tt en t i on V G G − A tt en t i on − hu m an c li ck V G G − A tt en t i on − R and P r
i
o r
V G G − F u ll r e s
V G G − hu m an c li ck
V G G − R ando m
V G G − B l u r
SV M − c a t ego r y
SV M − c a t ego r y − i n s t an c e s
H i dden
M a r k o v M ode l
D eepLab − C R F
Mouseclick− [Figure]
A cc u r a cy ( % )
H u m an C li ck
H u m an F
i x a t i on C li ck N e t
C li ck N e t − hu
m an c li ck
C li ck N e t − R and P r i o r
C li ck N e t − N o F i ne T une C li ck N e t − A tt en t i on M u l t i C li ck N e t − W i t h
A l phaLo ss
C li ck N e t − O b j O n l y C li ck N e t − O b
j R e v ea l ed V G G − A tt en t i on V G G − A tt en t i on − hu m an c li ck V G G − A tt en t i on − R and P r
i
o r
V G G − F u ll r e s
V G G − hu m an c li ck
V G G − R ando m
V G G − B l u r
SV M − c a t ego r y
SV M − c a t ego r y − i n s t an c e s
H i dden
M a r k o v M ode l
D eepLab − C R F
Mouseclick− [Figure] [Figure] C li ck
S equen c e
S i m il a r i t
y S c
o r e
B e t w een − H u m an C li
c
k
H u m an C li ck − C li c k N e t
B e t w een − H u m an C li
c
k
H u m an C li ck − C li ck N e t
B e t w een − H u m an C li
c
k
H u m an C li ck − C li ck N e t
[Figure]
4−click
[Figure]
[Figure]