Original: L EARNING TO C OUNT O BJECTS IN N ATURAL I MAGES FOR V ISUAL Q UESTION A NSWERING
Revised: L EARNING TO C OUNT O BJECTS IN N ATURAL I MAGES FOR V ISUAL Q UESTION A NSWERING

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Visual Question Answering (VQA) models have struggled with counting objects in natural images so far.
Revised: Visual Question Answering (VQA) models have struggled with counting objects in natural images so far.

Original: We identify a fundamental problem due to soft attention in these models as a cause.
Revised: We identify a fundamental problem due to soft attention in these models as a cause.

Original: To circumvent this problem, we propose a neural network component that allows robust counting from object proposals.
Revised: To circumvent this problem, we propose a neural network component that allows robust counting from object proposals.

Original: Experiments on a toy task show the effectiveness of this component and we obtain state-of-theart accuracy on the number category of the VQA v2 dataset without negatively affecting other categories, even outperforming ensemble models with our single model.
Revised: Experiments on a toy task show the effectiveness of this component and we obtain state-of-theart accuracy on the number category of the VQA v2 dataset without negatively affecting other categories, even outperforming ensemble models with our single model.

Original: On a difﬁcult balanced pair metric, the component gives a substantial improvement in counting over a strong baseline by 6.6%.
Revised: On a difﬁcult balanced pair metric, the component gives a substantial improvement in counting over a strong baseline by 6.6%.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Consider the problem of counting how many cats there are in Figure 1.
Revised: Consider the problem of counting how many cats there are in Figure 1.

Original: Solving this involves several rough steps: understanding what instances of that type can look like, ﬁnding them in the image, and adding them up.
Revised: Solving this involves several rough steps: understanding what instances of that type can look like, ﬁnding them in the image, and adding them up.

Original: This is a common task in Visual Question Answering (VQA) – answering questions about images – and is rated as among the tasks requiring the lowest human age to be able to answer (Antol et al., 2015).
Revised: This is a common task in Visual Question Answering (VQA) – answering questions about images – and is rated as among the tasks requiring the lowest human age to be able to answer (Antol et al., 2015).

Original: However, current models for VQA on natural images struggle to answer any counting questions successfully outside of dataset biases (Jabri et al., 2016).
Revised: However, current models for VQA on natural images struggle to answer any counting questions successfully outside of dataset biases (Jabri et al., 2016).

Original: One reason for this is the presence of a fundamental problem with counting in the widely-used soft attention mechanisms (section 3).
Revised: One reason for this is the presence of a fundamental problem with counting in the widely-used soft attention mechanisms (section 3).

Original: Another reason is that unlike standard counting tasks, there is no ground truth labeling of where the objects to count are.
Revised: Another reason is that unlike standard counting tasks, there is no ground truth labeling of where the objects to count are.

Original: Coupled with the fact that models need to be able to count a large variety of objects and that, ideally, performance on non-counting questions should not be compromised, the task of counting in VQA seems very challenging.
Revised: Coupled with the fact that models need to be able to count a large variety of objects and that, ideally, performance on non-counting questions should not be compromised, the task of counting in VQA seems very challenging.

Original: To make this task easier, we can use object proposals – pairs of a bounding box and object features – from object detection networks as input instead of learning from pixels directly.
Revised: To make this task easier, we can use object proposals – pairs of a bounding box and object features – from object detection networks as input instead of learning from pixels directly.

Original: In any moderately complex scene, this runs into the issue of double-counting overlapping object proposals.
Revised: In any moderately complex scene, this runs into the issue of double-counting overlapping object proposals.

Original: This is a problem present in many natural images, which leads to inaccurate counting in real-world scenarios.
Revised: This is a problem present in many natural images, which leads to inaccurate counting in real-world scenarios.

Original: Our main contribution is a differentiable neural network component that tackles this problem and consequently can learn to count (section 4).
Revised: Our main contribution is a differentiable neural network component that tackles this problem and consequently can learn to count (section 4).

Original: Used alongside an attention mechanism, this component avoids a fundamental limitation of soft attention while producing strong counting features.
Revised: Used alongside an attention mechanism, this component avoids a fundamental limitation of soft attention while producing strong counting features.

Original: We provide experimental evidence of the effectiveness of this component (section 5).
Revised: We provide experimental evidence of the effectiveness of this component (section 5).

Original: On a toy dataset, we demonstrate that this component enables robust counting in a variety of scenarios.
Revised: On a toy dataset, we demonstrate that this component enables robust counting in a variety of scenarios.

Original: On the number category of the VQA v2 Open-Ended dataset (Goyal et al., 2017), a relatively simple baseline model using the counting component outperforms all previous models – including large ensembles of state-of-the-art methods – without degrading performance on other categories.
Revised: On the number category of the VQA v2 Open-Ended dataset (Goyal et al., 2017), a relatively simple baseline model using the counting component outperforms all previous models – including large ensembles of state-of-the-art methods – without degrading performance on other categories.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Usually, greedy non-maximum suppression (NMS) is used to eliminate duplicate bounding boxes.
Revised: Usually, greedy non-maximum suppression (NMS) is used to eliminate duplicate bounding boxes.

Original: The main problem with using it as part of a model is that its gradient is piecewise constant.
Revised: The main problem with using it as part of a model is that its gradient is piecewise constant.

Original: Various differentiable variants such as by Azadi al
Revised: Various differentiable variants such as by Azadi et al.

Original: ), Hosang et al.
Revised: (2017), Hosang et al.

Original: (2017), and Henderson & Ferrari (2017) exist.
Revised: (2017), and Henderson & Ferrari (2017) exist.

Original: The main difference is that, since we are interested in counting, our component does not need to make binary decisions about which bounding boxes to keep; it outputs counting features, not a smaller set of bounding boxes.
Revised: The main difference is that, since we are interested in counting, our component does not need to make discrete decisions about which bounding boxes to keep; it outputs counting features, not a smaller set of bounding boxes.

Original: Our component is also easily integrated into standard VQA models that utilize soft attention without any need for other network architecture changes and can be used without using true bounding boxes for supervision.
Revised: Our component is also easily integrated into standard VQA models that utilize soft attention without any need for other network architecture changes and can be used without using true bounding boxes for supervision.

Original: On the VQA v2 dataset (Goyal et al., 2017) that we apply our method on, only few advances on counting questions have been made, none explicitly targeting counting questions.
Revised: On the VQA v2 dataset (Goyal et al., 2017) that we apply our method on, only few advances on counting questions have been made.

Original: The main improvement in accuracy is due to the use of object proposals in the visual processing pipeline, proposed by Anderson et al.
Revised: The main improvement in accuracy is due to the use of object proposals in the visual processing pipeline, proposed by Anderson et al.

Original: Their object proposal network is trained with classes in singular and plural forms, for example “tree” versus “trees”, which only allows primitive counting information to be present in the object features after region-of-interest pooling.
Revised: Their object proposal network is trained with classes in singular and plural forms, for example “tree” versus “trees”, which only allows primitive counting information to be present in the object features after region-of-interest pooling.

Original: Our approach differs in the way that instead of relying on counting features being present in the input, we create counting features using information present in the attention map over object proposals.
Revised: Our approach differs in the way that instead of relying on counting features being present in the input, we create counting features using information present in the attention map over object proposals.

Original: This has the beneﬁt of being able to count anything that the attention mechanism can discriminate instead of only objects that belong to the predetermined set of classes that had plural forms.
Revised: This has the beneﬁt of being able to count anything that the attention mechanism can discriminate instead of only objects that belong to the predetermined set of classes that had plural forms.

Revised: Using these object proposals, Trott et al.

Revised: (2018) train a sequential counting mechanism with a reinforcement learning loss on the counting question subsets of VQA v2 and Visual Genome.

Revised: They achieve a small increase in accuracy and an interpretable set of objects that their model counted, but it is unclear whether their method can be integrated into traditional VQA models due to their loss not applying to non-counting questions.

Revised: Since they evaluate on their own dataset, their results can not be easily compared to traditional VQA models capable of answering non-counting questions.

Original: Methods such as in Santoro et al.
Revised: Methods such as in Santoro et al.

Original: (2017) and Perez et al.
Revised: (2017) and Perez et al.

Original: (2017) can count on the synthetic CLEVR
Revised: (2017) can count on the synthetic CLEVR

Original: VQA dataset (Johnson et al., 2017) successfully without bounding boxes and supervision of where the objects to count are.
Revised: VQA dataset (Johnson et al., 2017) successfully without bounding boxes and supervision of where the objects to count are.

Original: They also use more training data ( ∼ 250,000 counting questions in the CLEVR training set versus ∼ 50,000 counting questions in the VQA v2 training set), much simpler objects, and synthetic question structures.
Revised: They also use more training data ( ∼ 250,000 counting questions in the CLEVR training set versus ∼ 50,000 counting questions in the VQA v2 training set), much simpler objects, and synthetic question structures.

Original: More traditional approaches based on Lempitsky & Zisserman (2010) learn to produce a target density map, from which a count is computed by integrating over it.
Revised: More traditional approaches based on Lempitsky & Zisserman (2010) learn to produce a target density map, from which a count is computed by integrating over it.

Original: In this setting, Cohen et al.
Revised: In this setting, Cohen et al.

Original: (2017) make use of overlaps of convolutional receptive ﬁelds to improve counting performance.
Revised: (2017) make use of overlaps of convolutional receptive ﬁelds to improve counting performance.

Original: Chattopadhyay et al.
Revised: Chattopadhyay et al.

Original: (2017) use an approach that divides the image into smaller non-overlapping chunks, each of which is counted individually and combined together at the end.
Revised: (2017) use an approach that divides the image into smaller non-overlapping chunks, each of which is counted individually and combined together at the end.

Original: In both of these contexts, the convolutional receptive ﬁelds or chunks can be seen as sets of bounding boxes with a ﬁxed structure in their positioning.
Revised: In both of these contexts, the convolutional receptive ﬁelds or chunks can be seen as sets of bounding boxes with a ﬁxed structure in their positioning.

Original: Note that while Chattopadhyay et al.
Revised: Note that while Chattopadhyay et al.

Original: (2017) evaluate their models on a small subset of counting questions in VQA, major differences in training setup make their results not comparable to our work.
Revised: (2017) evaluate their models on a small subset of counting questions in VQA, major differences in training setup make their results not comparable to our work.

Original:  P ROBLEMS WITH SOFT ATTENTION
Revised:  P ROBLEMS WITH SOFT ATTENTION

Original: The main message in this section is that using the feature vectors obtained after the attention mechanism is not enough to be able to count; the attention maps themselves should be used, which is what we do in our counting component.
Revised: The main message in this section is that using the feature vectors obtained after the attention mechanism is not enough to be able to count; the attention maps themselves should be used, which is what we do in our counting component.

Original: Models in VQA have consistently beneﬁted from the use of soft attention (Mnih et al., 2014; Bahdanau et al., 2015) on the image, commonly implemented with a shallow convolutional network.
Revised: Models in VQA have consistently beneﬁted from the use of soft attention (Mnih et al., 2014; Bahdanau et al., 2015) on the image, commonly implemented with a shallow convolutional network.

Original: It learns to output a weight for the feature vector at each spatial position in the feature map, which is ﬁrst normalized and then used for performing a weighted sum over the spatial positions to produce a single feature vector.
Revised: It learns to output a weight for the feature vector at each spatial position in the feature map, which is ﬁrst normalized and then used for performing a weighted sum over the spatial positions to produce a single feature vector.

Original: However, soft spatial attention severely limits the ability for a model to count.
Revised: However, soft spatial attention severely limits the ability for a model to count.

Original: Consider the task of counting the number of cats for two images: an image showing a single cat on a clean background and an image that consists of two side-by-side copies of the ﬁrst image.
Revised: Consider the task of counting the number of cats for two images: an image showing a single cat on a clean background and an image that consists of two side-by-side copies of the ﬁrst image.

Original: What we will describe applies to both spatial feature maps and sets of object proposals as input, but we focus on the latter case for simplicity.
Revised: What we will describe applies to both spatial feature maps and sets of object proposals as input, but we focus on the latter case for simplicity.

Original: With an object detection network, we detect one cat in the ﬁrst image and two cats in the second image, producing the same feature vector for all three detections.
Revised: With an object detection network, we detect one cat in the ﬁrst image and two cats in the second image, producing the same feature vector for all three detections.

Original: The attention mechanism then assigns all three instances of the same cat the same weight.
Revised: The attention mechanism then assigns all three instances of the same cat the same weight.

Original: The usual normalization used for the attention weights is the softmax function, which normalizes the weights to sum to 1.
Revised: The usual normalization used for the attention weights is the softmax function, which normalizes the weights to sum to 1.

Original: Herein lies the problem: the cat in the ﬁrst image receives a normalized weight of 1, but the two cats in the second image now each receive a weight of 0.5.
Revised: Herein lies the problem: the cat in the ﬁrst image receives a normalized weight of 1, but the two cats in the second image now each receive a weight of 0.5.

Original: After the weighted sum,we are effectively averaging the two cats in the second image back to a single cat. As a consequence, the feature vector obtained after the weighted sum is exactly the same between the two images and we have lost all information about a possible count from the attention map. Any method that normalizes the weights to sum to 1 suffers from this issue.
Revised: After the weighted sum,

Original: [Figure]
Revised: [Figure]

Original: Multiple glimpses (Larochelle & Hinton, 2010) – sets of attention weights that the attention mechanism outputs – or several steps of attention (Yang et al., 2016; Lu et al., 2016) do not circumvent this problem.
Revised: Multiple glimpses (Larochelle & Hinton, 2010) – sets of attention weights that the attention mechanism outputs – or several steps of attention (Yang et al., 2016; Lu et al., 2016) do not circumvent this problem.

Original: Each glimpse or step can not separate out an object each, since the attention weight given to one feature vector does not depend on the other feature vectors to be attended over.
Revised: Each glimpse or step can not separate out an object each, since the attention weight given to one feature vector does not depend on the other feature vectors to be attended over.

Original: Hard attention (Ba et al., 2015; Mnih et al., 2014) and structured attention (Kim et al., 2017) may be possible solutions to this, though no signiﬁcant improvement in counting ability has been found for the latter so far (Zhu et al., 2017).
Revised: Hard attention (Ba et al., 2015; Mnih et al., 2014) and structured attention (Kim et al., 2017) may be possible solutions to this, though no signiﬁcant improvement in counting ability has been found for the latter so far (Zhu et al., 2017).

Original: Ren & Zemel (2017) circumvent the problem by limiting attention to only work within one generated bounding box at a time, remotely similar to our approach of using object proposal features.
Revised: Ren & Zemel (2017) circumvent the problem by limiting attention to only work within one generated bounding box at a time, remotely similar to our approach of using object proposal features.

Original: Without normalization of weights to sum to one, the scale of the output features depends on the number of objects detected.
Revised: Without normalization of weights to sum to one, the scale of the output features depends on the number of objects detected.

Original: In an image with 10 cats, the output feature vector is scaled up by 10.
Revised: In an image with 10 cats, the output feature vector is scaled up by 10.

Original: Since deep neural networks are typically very scale-sensitive – the scale of weight initializations and activations is generally considered quite important (Mishkin & Matas, 2016) – and the classiﬁer would have to learn that joint scaling of all features is somehow related to count, this approach is not reasonable for counting objects.
Revised: Since deep neural networks are typically very scale-sensitive – the scale of weight initializations and activations is generally considered quite important (Mishkin & Matas, 2016) – and the classiﬁer would have to learn that joint scaling of all features is somehow related to count, this approach is not reasonable for counting objects.

Original: This is evidenced in Teney et al.
Revised: This is evidenced in Teney et al.

Original: (2017) where they provide evidence that sigmoid normalization not only degrades accuracy on non-number questions slightly, but also does not help with counting.
Revised: (2017) where they provide evidence that sigmoid normalization not only degrades accuracy on non-number questions slightly, but also does not help with counting.

Original:  C OUNTING COMPONENT
Revised:  C OUNTING COMPONENT

Original: In this section, we describe a differentiable mechanism for counting from attention weights, while also dealing with the problem of overlapping object proposals to reduce double-counting of objects.
Revised: In this section, we describe a differentiable mechanism for counting from attention weights, while also dealing with the problem of overlapping object proposals to reduce double-counting of objects.

Original: This involves some nontrivial details to produce counts that are as accurate as possible.
Revised: This involves some nontrivial details to produce counts that are as accurate as possible.

Original: The main idea is illustrated in Figure 1 with the two main steps shown in Figure 2 and Figure 3.
Revised: The main idea is illustrated in Figure 1 with the two main steps shown in Figure 2 and Figure 3.

Original: The use of this component allows a model to count while still being able to exploit the beneﬁts of soft attention.
Revised: The use of this component allows a model to count while still being able to exploit the beneﬁts of soft attention.

Original: Our key idea for dealing with overlapping object proposals is to turn these object proposals into a graph that is based on how they overlap.
Revised: Our key idea for dealing with overlapping object proposals is to turn these object proposals into a graph that is based on how they overlap.

Original: We then remove and scale edges in a speciﬁc way such that an estimate of the number of underlying objects is recovered.
Revised: We then remove and scale edges in a speciﬁc way such that an estimate of the number of underlying objects is recovered.

Original: Our general strategy is to primarily design the component for the unrealistic extreme cases of perfect attention maps and bounding boxes that are either fully overlapping or fully distinct.
Revised: Our general strategy is to primarily design the component for the unrealistic extreme cases of perfect attention maps and bounding boxes that are either fully overlapping or fully distinct.

Original: By introducing some parameters and only using differentiable operations, we give the ability for the module to interpolate between the correct behaviours for these extreme cases to handle the more realistic cases.
Revised: By introducing some parameters and only using differentiable operations, we give the ability for the module to interpolate between the correct behaviours for these extreme cases to handle the more realistic cases.

Original: These parameters are responsible for handling variations in attention weights and partial bounding box overlaps in a manner suitable for a given dataset.
Revised: These parameters are responsible for handling variations in attention weights and partial bounding box overlaps in a manner suitable for a given dataset.

Original: To achieve this, we use several piecewise linear functions f 1 , . . .
Revised: To achieve this, we use several piecewise linear functions f 1 , . . .

Original: , f 8 as activation functions (deﬁned in Appendix A), approximating arbitrary functions with domain and range [0, 1].
Revised: , f 8 as activation functions (deﬁned in Appendix A), approximating arbitrary functions with domain and range [0, 1].

Original: The shapes of these functions are learned to handle the speciﬁc nonlinear interactions necessary for dealing with overlapping proposals.
Revised: The shapes of these functions are learned to handle the speciﬁc nonlinear interactions necessary for dealing with overlapping proposals.

Original: Through their parametrization we enforce that f k (0) = 0 , f k (1) = 1 , and that they are monotonically increasing.
Revised: Through their parametrization we enforce that f k (0) = 0 , f k (1) = 1 , and that they are monotonically increasing.

Original: The ﬁrst two properties are required so that the extreme cases that we explicitly handle are left unchanged.
Revised: The ﬁrst two properties are required so that the extreme cases that we explicitly handle are left unchanged.

Original: In those cases, f k is only applied to values of 0 or 1, so the activation functions can be safely ignored for understanding how the component handles them.
Revised: In those cases, f k is only applied to values of 0 or 1, so the activation functions can be safely ignored for understanding how the component handles them.

Original: By enforcing monotonicity, we can make sure that, for example, an increased value in an attention map should never result in the prediction of the count to decrease.
Revised: By enforcing monotonicity, we can make sure that, for example, an increased value in an attention map should never result in the prediction of the count to decrease.

Original: 4. I NPUT
Revised: 4. I NPUT

Original: Given a set of features from object proposals, an attention mechanism produces a weight for each proposal based on the question.
Revised: Given a set of features from object proposals, an attention mechanism produces a weight for each proposal based on the question.

Original: The counting component takes as input the n largest attention weights a =
Revised: The counting component takes as input the n largest attention weights a =

Original: [ a 1 , . .
Revised: [ a 1 , . .

Original: . , a n ] T and their corresponding bounding boxes b =
Revised: . , a n ] T and their corresponding bounding boxes b =

Original: [ b 1 , . . .
Revised: [ b 1 , . . .

Original: , b n ] T .
Revised: , b n ] T .

Original: We assume that the weights lie in the interval [0 , 1] , which can easily be achieved by applying a logistic function to them.
Revised: We assume that the weights lie in the interval [0 , 1] , which can easily be achieved by applying a logistic function to them.

Original: In the extreme cases that we explicitly handle, we assume that the attention mechanism assigns a value of 1 to a i whenever the i th proposal contains a relevant object and a value of 0 whenever it does not.
Revised: In the extreme cases that we explicitly handle, we assume that the attention mechanism assigns a value of 1 to a i whenever the i th proposal contains a relevant object and a value of 0 whenever it does not.

Original: This is in line with what usual soft attention mechanisms learn, as they produce higher weights for relevant inputs.
Revised: This is in line with what usual soft attention mechanisms learn, as they produce higher weights for relevant inputs.

Original: We also assume that either two object proposals fully overlap (in which case they must be showing the same object and thus receive the same attention weight) or that they are fully distinct (in which case they show different objects).
Revised: We also assume that either two object proposals fully overlap (in which case they must be showing the same object and thus receive the same attention weight) or that they are fully distinct (in which case they show different objects).

Original: Keep in mind that while we make these assumptions to make reasoning about the behaviour easier, the learned parameters in the activation functions are intended to handle the more realistic scenarios when the assumptions do not apply.
Revised: Keep in mind that while we make these assumptions to make reasoning about the behaviour easier, the learned parameters in the activation functions are intended to handle the more realistic scenarios when the assumptions do not apply.

Original: Instead of partially overlapping proposals, the problem now becomes the handling of exact duplicate proposals of underlying objects in a differentiable manner.
Revised: Instead of partially overlapping proposals, the problem now becomes the handling of exact duplicate proposals of underlying objects in a differentiable manner.

Original: 4. D EDUPLICATION
Revised: 4. D EDUPLICATION

Original: We start by changing the vector of attention weights a into a graph representation in which bounding boxes can be utilized more easily.
Revised: We start by changing the vector of attention weights a into a graph representation in which bounding boxes can be utilized more easily.

Original: Hence, we compute the outer product of the attention weights to obtain an attention matrix.
Revised: Hence, we compute the outer product of the attention weights to obtain an attention matrix.

Original: [Equation]
Revised: [Equation]

Original: A ∈ R n × n can be interpreted as an adjacency matrix for a weighted directed graph.
Revised: A ∈ R n × n can be interpreted as an adjacency matrix for a weighted directed graph.

Original: In this graph, the i th vertex represents the object proposal associated with a i and the edge between any pair of vertices ( i, j ) has weight a i a j .
Revised: In this graph, the i th vertex represents the object proposal associated with a i and the edge between any pair of vertices ( i, j ) has weight a i a j .

Original: In the extreme case where a i is virtually 0 or 1 , products are equivalent to logical AND operators.
Revised: In the extreme case where a i is virtually 0 or 1 , products are equivalent to logical AND operators.

Original: It follows that the subgraph containing only the vertices satisfying a i = 1 is a complete digraph with self-loops.
Revised: It follows that the subgraph containing only the vertices satisfying a i = 1 is a complete digraph with self-loops.

Original: In this representation, our objective is to eliminate edges in such a way that, conceptually, the underlying true objects – instead of proposals thereof – are the vertices of that complete subgraph.
Revised: In this representation, our objective is to eliminate edges in such a way that, conceptually, the underlying true objects – instead of proposals thereof – are the vertices of that complete subgraph.

Original: In order to then turn that graph into a count, recall that the number of edges | E | in a complete digraph with self-loops relates to the number of vertices | V | through | E | = | V | 2 .
Revised: In order to then turn that graph into a count, recall that the number of edges | E | in a complete digraph with self-loops relates to the number of vertices | V | through | E | = | V | 2 .

Original: | E | can be computed by summing over the entries in an adjacency matrix and | V | is then the count.
Revised: | E | can be computed by summing over the entries in an adjacency matrix and | V | is then the count.

Original: Notice how when | E | is set to the sum over A , (cid:112) | E | = (cid:80) i a i holds.
Revised: Notice how when | E | is set to the sum over A , (cid:112) | E | = (cid:80) i a i holds.

Original: This convenient property implies that when all proposals are fully distinct, the component can output the same as simply summing over the original attention weights by default.
Revised: This convenient property implies that when all proposals are fully distinct, the component can output the same as simply summing over the original attention weights by default.

Original: There are two types of duplicate edges to eliminate to achieve our objective: intra -object edges and inter -object edges.
Revised: There are two types of duplicate edges to eliminate to achieve our objective: intra -object edges and inter -object edges.

Original: 4.2. I NTRA - OBJECT EDGES
Revised: 4.2. I NTRA - OBJECT EDGES

Original: First, we eliminate intra-object edges between duplicate proposals of a single underlying object. 
Revised: First, we eliminate intra-object edges between duplicate proposals of a single underlying object. [Figure] [Figure] =

Revised: [Equation]

Original: To compare two bounding boxes, we use the usual intersection-over-union (IoU) metric.
Revised: To compare two bounding boxes, we use the usual intersection-over-union (IoU) metric.

Original: We deﬁne the distance matrix D ∈ R n × n to be
Revised: We deﬁne the distance matrix D ∈ R n × n to be

Original: [Equation]
Revised: [Equation]

Original: [Figure]

Original: [Figure]

Original: = ≡

Original: D can also be interpreted as an adjacency matrix.
Revised: D can also be interpreted as an adjacency matrix.

Original: It represents a graph that has edges everywhere except when the two bounding boxes that an edge connects would overlap
Revised: It represents a graph that has edges everywhere except when the two bounding boxes that an edge connects would overlap.

Original: Intra-object edges are removed by elementwise multiplying ( (cid:12) ) the distance matrix with the attention matrix (Figure 2).
Revised: Intra-object edges are removed by elementwise multiplying ( (cid:12) ) the distance matrix with the attention matrix (Figure 2).

Original: [Equation]
Revised: [Equation]

Original: ˜ A no longer has self-loops, so we need to add them back in at a later point to still satisfy | E | = | V | 2 .
Revised: ˜ A no longer has self-loops, so we need to add them back in at a later point to still satisfy | E | = | V | 2 .

Original: Notice that we start making use of the activation functions mentioned earlier to handle intermediate values in the interval (0 , 1) for both A and D .
Revised: Notice that we start making use of the activation functions mentioned earlier to handle intermediate values in the interval (0 , 1) for both A and D .

Original: They regulate the inﬂuence of attention weights that are not close to 0 or 1 and the inﬂuence of partial overlaps.
Revised: They regulate the inﬂuence of attention weights that are not close to 0 or 1 and the inﬂuence of partial overlaps.

Original: 4.2. I NTER - OBJECT EDGES
Revised: 4.2. I NTER - OBJECT EDGES

Original: Second, we eliminate inter-object edges between duplicate proposals of different underlying objects.
Revised: Second, we eliminate inter-object edges between duplicate proposals of different underlying objects.

Original: The main idea (depicted in Figure 3) is to count the number of proposals associated to each invidual object, then scale down the weight of their associated edges by that number.
Revised: The main idea (depicted in Figure 3) is to count the number of proposals associated to each invidual object, then scale down the weight of their associated edges by that number.

Original: If there are two proposals of a single object, the edges involving those proposals should be scaled by 0.5.
Revised: If there are two proposals of a single object, the edges involving those proposals should be scaled by 0.5.

Original: In essence, this averages over the proposals within each underlying object because we only use the sum over the edge weights to compute the count at the end.
Revised: In essence, this averages over the proposals within each underlying object because we only use the sum over the edge weights to compute the count at the end.

Original: Conceptually, this reduces multiple proposals of an object down to one as desired.
Revised: Conceptually, this reduces multiple proposals of an object down to one as desired.

Original: Since we do not know how many proposals belong to an object, we have to estimate this.
Revised: Since we do not know how many proposals belong to an object, we have to estimate this.

Original: We do this by using the fact that proposals of the same object are similar.
Revised: We do this by using the fact that proposals of the same object are similar.

Original: Keep in mind that ˜
Revised: Keep in mind that ˜

Original: A has no self-loops nor edges between proposals of the same object.
Revised: A has no self-loops nor edges between proposals of the same object.

Original: As a consequence, two nonzero rows in ˜ A are the same if and only if the proposals are the same.
Revised: As a consequence, two nonzero rows in ˜ A are the same if and only if the proposals are the same.

Original: If the two rows differ in at least one entry, then one proposal overlaps a proposal that the other proposal does not overlap, so they must be different proposals.
Revised: If the two rows differ in at least one entry, then one proposal overlaps a proposal that the other proposal does not overlap, so they must be different proposals.

Original: This means for comparing rows, we need a similarity function that satisﬁes the criteria of taking the value 1 when they differ in no places andif they differ in at least one place.
Revised: This means for comparing rows, we need a similarity function that satisﬁes the criteria of taking the value 1 when they differ in no places andif they differ in at least one place.

Original: We deﬁne a differentiable similarity between proposals i and j as
Revised: We deﬁne a differentiable similarity between proposals i and j as

Original: [Equation]
Revised: [Equation]

Original: where X = f 4 ( A ) (cid:12) f 5 ( D ) is the same as ˜ A except with different activation functions.
Revised: where X = f 4 ( A ) (cid:12) f 5 ( D ) is the same as ˜ A except with different activation functions.

Original: The (cid:81) term compares the rows of proposals i and j .
Revised: The (cid:81) term compares the rows of proposals i and j .

Original: Using this term instead of f 4 (1 − D ij ) was more robust to inaccurate bounding boxes in initial experiments.
Revised: Using this term instead of f 4 (1 − D ij ) was more robust to inaccurate bounding boxes in initial experiments.

Original: Note that the f 3 (1 − | a i − a j | ) term handles the edge case when there is only one proposal to count.
Revised: Note that the f 3 (1 − | a i − a j | ) term handles the edge case when there is only one proposal to count.

Original: Since X does not have self-loops, X contains only zeros in that case, which causes the row corresponding to a i = 1 to be incorrectly similar to the rows where a j (cid:54)
Revised: Since X does not have self-loops, X contains only zeros in that case, which causes the row corresponding to a i = 1 to be incorrectly similar to the rows where a j (cid:54)

Original: =
Revised: =

Original: i = 0 .
Revised: i = 0 .

Original: By comparing the attention weights through that term as well, this issue is avoided.
Revised: By comparing the attention weights through that term as well, this issue is avoided.

Original: Now that we can check how similar two proposals are, we count the number of times any row is the same as any other row and compute a scaling factor
Revised: Now that we can check how similar two proposals are, we count the number of times any row is the same as any other row and compute a scaling factor

Original: s i for each vertex i .
Revised: s i for each vertex i .

Original: [Equation]
Revised: [Equation]

Original: The time complexity of computing s =
Revised: The time complexity of computing s =

Original: [ s 1 , . . .
Revised: [ s 1 , . . .

Original: , s n ] T is Θ( n 3 ) as there are n 2 pairs of rows and Θ( n ) operations to compute the similarity of any pair of rows.
Revised: , s n ] T is Θ( n 3 ) as there are n 2 pairs of rows and Θ( n ) operations to compute the similarity of any pair of rows.

Original: Since these scaling factors apply to each vertex, we have to expand s into a matrix using the outer product in order to scale both incoming and outgoing edges of each vertex.
Revised: Since these scaling factors apply to each vertex, we have to expand s into a matrix using the outer product in order to scale both incoming and outgoing edges of each vertex.

Original: We can also add self-loops back in, which need to be scaled by s as well.
Revised: We can also add self-loops back in, which need to be scaled by s as well.

Original: Then, the count matrix C is
Revised: Then, the count matrix C is

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: ) where diag ( · ) expands a vector into a diagonal matrix with the vector on the diagonal.
Revised: (6) where diag ( · ) expands a vector into a diagonal matrix with the vector on the diagonal.

Original: The scaling of self-loops involves a non-obvious detail.
Revised: The scaling of self-loops involves a non-obvious detail.

Original: Recall that the diagonal that was removed when going from A to ˜ A contains the entries f 1 ( a (cid:12) a ) .
Revised: Recall that the diagonal that was removed when going from A to ˜ A contains the entries f 1 ( a (cid:12) a ) .

Original: Notice however that we are scaling this diagonal by s and not s (cid:12) s .
Revised: Notice however that we are scaling this diagonal by s and not s (cid:12) s .

Original: This is because the number of inter-object edges scales quadratically with respect to the number of proposals per object, but the number of self-loops only scales linearly.
Revised: This is because the number of inter-object edges scales quadratically with respect to the number of proposals per object, but the number of self-loops only scales linearly.

Original: 4. O UTPUT
Revised: 4. O UTPUT

Original: Under a sum, C is now equivalent to a complete graph with self-loops that involves all relevant objects instead of relevant proposals as originally desired.
Revised: Under a sum, C is now equivalent to a complete graph with self-loops that involves all relevant objects instead of relevant proposals as originally desired.

Original: To turn C into a count c , we set | E | =
Revised: To turn C into a count c , we set | E | =

Original: (cid:80) i,j C ij as mentioned and
Revised: (cid:80) i,j C ij as mentioned and

Original: [Equation]
Revised: [Equation]

Original: We veriﬁed experimentally that when our extreme case assumptions hold, c is always an integer and equal to the correct count, regardless of the number of duplicate object proposals.
Revised: We veriﬁed experimentally that when our extreme case assumptions hold, c is always an integer and equal to the correct count, regardless of the number of duplicate object proposals.

Original: To avoid issues with scale when the number of objects is large, we turn this single feature into several classes, one for each possible number.
Revised: To avoid issues with scale when the number of objects is large, we turn this single feature into several classes, one for each possible number.

Original: Since we only used the object proposals with the largest n weights, the predicted count c can be at most n .
Revised: Since we only used the object proposals with the largest n weights, the predicted count c can be at most n .

Original: We deﬁne the output o =
Revised: We deﬁne the output o =

Original: [ o 0 , o 1 , . . .
Revised: [ o 0 , o 1 , . . .

Original: , o n ] T to be
Revised: , o n ] T to be

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: This results in a vector that is 1 at the index of the count and 0 everywhere else when c is exactly an integer, and a linear interpolation between the two corresponding one-hot vectors when the count falls inbetween two integers.
Revised: This results in a vector that is 1 at the index of the count and 0 everywhere else when c is exactly an integer, and a linear interpolation between the two corresponding one-hot vectors when the count falls inbetween two integers.

Original: 4.3. O UTPUT CONFIDENCE
Revised: 4.3. O UTPUT CONFIDENCE

Original: Finally, we might consider a prediction made from values of a and D that are either close to 0 or close to 1 to be more reliable – we explicitly handle these after all – than when many values are close to 0 . 5 .
Revised: Finally, we might consider a prediction made from values of a and D that are either close to 0 or close to 1 to be more reliable – we explicitly handle these after all – than when many values are close to 0 . 5 .

Original: To incorporate this idea, we scale o by a conﬁdence value in the interval [0 , 1] .
Revised: To incorporate this idea, we scale o by a conﬁdence value in the interval [0 , 1] .

Original: [Figure]

Original: We deﬁne p a and p D to be the average distances to 0 .
Revised: We deﬁne p a and p D to be the average distances to 0 .

Original: The choice of 0 .
Revised: The choice of 0 .

Original: 5 is not important, because the module can learn to change it by changing where f 6 ( x ) = 0 .
Revised: 5 is not important, because the module can learn to change it by changing where f 6 ( x ) = 0 .

Original: 5 and f 7 ( x ) = 0 .
Revised: 5 and f 7 ( x ) = 0 .

Original: [Equation]
Revised: [Equation]

Original: Then, the output of the component with conﬁdence scaling is
Revised: Then, the output of the component with conﬁdence scaling is

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: In summary, we only used diffentiable operations to deduplicate object proposals and obtain a feature vector that represents the predicted count.
Revised: In summary, we only used diffentiable operations to deduplicate object proposals and obtain a feature vector that represents the predicted count.

Original: This allows easy integration into any model with soft attention, enabling a model to count from an attention map.
Revised: This allows easy integration into any model with soft attention, enabling a model to count from an attention map.

Original:  E XPERIMENTS 5. T OY TASK
Revised:  E XPERIMENTS 5. T OY TASK

Original: First, we design a simple toy task to evaluate counting ability.
Revised: First, we design a simple toy task to evaluate counting ability.

Original: This dataset is intended to only evaluate the performance of counting; thus, we skip any processing steps that are not directly related such as the processing of an input image.
Revised: This dataset is intended to only evaluate the performance of counting; thus, we skip any processing steps that are not directly related such as the processing of an input image.

Original: Samples from this dataset are given in Appendix D
Revised: Samples from this dataset are given in Appendix D

Original: The classiﬁcation task is to predict an integer count ˆ c of true objects, uniformly drawn from 0 toinclusive, from a set of bounding boxes and the associated attention weights.
Revised: The classiﬁcation task is to predict an integer count ˆ c of true objects, uniformly drawn from 0 toinclusive, from a set of bounding boxes and the associated attention weights.

Original: 10 square bounding boxes with side length l ∈ (0 , 1] are placed in a square image with unit side length.
Revised: 10 square bounding boxes with side length l ∈ (0 , 1] are placed in a square image with unit side length.

Original: The x and y coordinates of their top left corners are uniformly drawn from U (0 , 1 − l ) so that the boxes do not extend beyond the image border.
Revised: The x and y coordinates of their top left corners are uniformly drawn from U (0 , 1 − l ) so that the boxes do not extend beyond the image border.

Original: l is used to control the overlapping of bounding boxes: a larger l leads to the ﬁxed number of objects to be more tightly packed, increasing the chance of overlaps.
Revised: l is used to control the overlapping of bounding boxes: a larger l leads to the ﬁxed number of objects to be more tightly packed, increasing the chance of overlaps.

Original: ˆ c number of these boxes are randomly chosen to be true bounding boxes.
Revised: ˆ c number of these boxes are randomly chosen to be true bounding boxes.

Original: The score of a bounding box is the maximum IoU overlap of it with any true bounding box.
Revised: The score of a bounding box is the maximum IoU overlap of it with any true bounding box.

Original: Then, the attention weight is a linear interpolation between the score and a noise value drawn from U (0 , 1) , with q ∈ [0 , 1] controlling this trade-off.
Revised: Then, the attention weight is a linear interpolation between the score and a noise value drawn from U (0 , 1) , with q ∈ [0 , 1] controlling this trade-off.

Original: q is the attention noise parameter: when q is 0, there is no noise and when q is 1, there is no signal.
Revised: q is the attention noise parameter: when q is 0, there is no noise and when q is 1, there is no signal.

Original: Increasing q also indirectly simulates imprecise placements of bounding boxes.
Revised: Increasing q also indirectly simulates imprecise placements of bounding boxes.

Original: We compare the counting component against a simple baseline that simply sums the attention weights and turns the sum into a feature vector with Equation 8.
Revised: We compare the counting component against a simple baseline that simply sums the attention weights and turns the sum into a feature vector with Equation 8.

Original: Both models are followed by a linear projection to the classes 0 to 10 inclusive and a softmax activation.
Revised: Both models are followed by a linear projection to the classes 0 to 10 inclusive and a softmax activation.

Original: They are trained with crossentropy loss for 1000 iterations using Adam (Kingma & Ba, 2015) with a learning rate of 0.01 and a batch size of 1024.
Revised: They are trained with crossentropy loss for 1000 iterations using Adam (Kingma & Ba, 2015) with a learning rate of 0.01 and a batch size of 1024.

Original: 5.1. R ESULTS
Revised: 5.1. R ESULTS

Original: The results of varying l while keeping q ﬁxed at various values and vice versa are shown in Figure 4.
Revised: The results of varying l while keeping q ﬁxed at various values and vice versa are shown in Figure 4.

Original: Regardless of l and q , the counting component performs better than the baseline in most cases, often[Figure] signiﬁcantly so.
Revised: Regardless of l and q , the counting component performs better than the baseline in most cases, often signiﬁcantly so.

Original: Particularly when the noise is low, the component can deal with high values for l very successfully, showing that it accomplishes the goal of increased robustness to overlapping proposals.
Revised: Particularly when the noise is low, the component can deal with high values for l very successfully, showing that it accomplishes the goal of increased robustness to overlapping proposals.

Original: The component also handles moderate noise levels decently as long as the overlaps are limited.
Revised: The component also handles moderate noise levels decently as long as the overlaps are limited.

Original: The performance when both l and q are high is closely matched by the baseline, likely due to the high difﬁculty of those parametrizations leaving little information to extract in the ﬁrst place.
Revised: The performance when both l and q are high is closely matched by the baseline, likely due to the high difﬁculty of those parametrizations leaving little information to extract in the ﬁrst place.

Original: We can also look at the shape of the activation functions themselves, shown in Figure 5 and Appendix C, to understand how the behaviour changes with varying dataset parameters.
Revised: We can also look at the shape of the activation functions themselves, shown in Figure 5 and Appendix C, to understand how the behaviour changes with varying dataset parameters.

Original: For simplicity,  we limit our description to the two easiest-to-interpret functions: f 1 for the attention weights and ffor the bounding box distances.
Revised: For simplicity, [Figure] [Figure] we limit our description to the two easiest-to-interpret functions: f 1 for the attention weights and ffor the bounding box distances

Revised: [Figure]

Original: When increasing the side length, the height of the “step” in f 1 decreases to compensate for the generally greater degree of overlapping bounding boxes.
Revised: When increasing the side length, the height of the “step” in f 1 decreases to compensate for the generally greater degree of overlapping bounding boxes.

Original: A similar effect is seen with f 2 : it varies over requiring a high pairwise distance when l is low – when partial overlaps are most likely spurious – and considering small distances enough for proposals to be considered different when l is high.
Revised: A similar effect is seen with f 2 : it varies over requiring a high pairwise distance when l is low – when partial overlaps are most likely spurious – and considering small distances enough for proposals to be considered different when l is high.

Original: At the highest values for l , there is little signal in the overlaps left since everything overlaps with everything, which explains why f 2 returns to its default linear initialization for those parameters.
Revised: At the highest values for l , there is little signal in the overlaps left since everything overlaps with everything, which explains why f 2 returns to its default linear initialization for those parameters.

Original: When varying the amount of noise, without noise f 1 resembles a step function where the step starts close to x = 1 and takes a value of close to 1 after the step.
Revised: When varying the amount of noise, without noise f 1 resembles a step function where the step starts close to x = 1 and takes a value of close to 1 after the step.

Original: Since a true proposal will always have a weight of 1 when there is no noise, anything below this can be safely zeroed out.
Revised: Since a true proposal will always have a weight of 1 when there is no noise, anything below this can be safely zeroed out.

Original: With increasing noise, this step moves away from 1 for both x and f 1 ( x ) , capturing the uncertainty when a bounding box belongs to a true object.
Revised: With increasing noise, this step moves away from 1 for both x and f 1 ( x ) , capturing the uncertainty when a bounding box belongs to a true object.

Original: With lower q , f 2 considers a pair of proposals to be distinct for lower distances, whereas with higher q , f 2 follows a more sigmoidal shape.
Revised: With lower q , f 2 considers a pair of proposals to be distinct for lower distances, whereas with higher q , f 2 follows a more sigmoidal shape.

Original: This can be explained by the model taking the increased uncertainty of the precise bounding box placements into account by requiring higher distances for proposals to be considered completely different.
Revised: This can be explained by the model taking the increased uncertainty of the precise bounding box placements into account by requiring higher distances for proposals to be considered completely different.

Original: 5. VQA
Revised: 5. VQA

Original: VQA v2 (Goyal et al., 2017) is the updated version of the VQA v1 dataset (Antol et al., 2015) where greater care has been taken to reduce dataset biases through balanced pairs: for each question, a pair of images is identiﬁed where the answer to that question differs.
Revised: VQA v2 (Goyal et al., 2017) is the updated version of the VQA v1 dataset (Antol et al., 2015) where greater care has been taken to reduce dataset biases through balanced pairs: for each question, a pair of images is identiﬁed where the answer to that question differs.

Original: The standard accuracy metric on this dataset accounts for disagreements in human answers by averaging min( 13 agreeing , 1) over all 10-choose-9 subsets of human answers, where agreeing is the number of human answers that agree with the given answer.
Revised: The standard accuracy metric on this dataset accounts for disagreements in human answers by averaging min( 13 agreeing , 1) over all 10-choose-9 subsets of human answers, where agreeing is the number of human answers that agree with the given answer.

Original: This can be shown to be equal to min(0 .
Revised: This can be shown to be equal to min(0 .

Original: 3 agreeing , 1) without averaging.
Revised: 3 agreeing , 1) without averaging.

Original: We use an improved version of the strong VQA baseline by Kazemi & Elqursh (2017) as baseline model (details in Appendix B).
Revised: We use an improved version of the strong VQA baseline by Kazemi & Elqursh (2017) as baseline model (details in Appendix B).

Original: We have not performed any tuning of this baseline to maximize the performance difference between it and the baseline with counting module.
Revised: We have not performed any tuning of this baseline to maximize the performance difference between it and the baseline with counting module.

Original: To augment this model with the counting component, we extract the attention weights of the ﬁrst attention glimpse (there are two in the baseline) before softmax normalization, and feed them into the counting component after  applying a logistic function.
Revised: To augment this model with the counting component, we extract the attention weights of the ﬁrst attention glimpse (there are two in the baseline) before softmax normalization, and feed them into the counting component after [Table] [Table] applying a logistic function.

Original: Since object proposal features from Anderson et al.
Revised: Since object proposal features from Anderson et al.

Original: (2017) vary fromto 100 per image, a natural choice for the number of top- n proposals to use is 10.
Revised: (2017) vary fromto 100 per image, a natural choice for the number of top- n proposals to use is 10.

Original: The output of the component is linearly projected into the same space as the hidden layer of the classiﬁer, followed by
Revised: The output of the component is linearly projected into the same space as the hidden layer of the classiﬁer, followed by

Original: ReLU activation, batch normalization, and addition with the features in the hidden layer.
Revised: ReLU activation, batch normalization, and addition with the features in the hidden layer.

Original: [Table]

Original: [Table]

Original: 5.2. R ESULTS
Revised: 5.2. R ESULTS

Original: Table 1 shows the results on the ofﬁcial VQA v2 leaderboard.
Revised: Table 1 shows the results on the ofﬁcial VQA v2 leaderboard.

Original: The baseline with our component has a signiﬁcantly higher accuracy on number questions without compromising accuracy on other categories compared to the baseline result.
Revised: The baseline with our component has a signiﬁcantly higher accuracy on number questions without compromising accuracy on other categories compared to the baseline result.

Original: Despite our single-model baseline being substantially worse than the state-of-the-art, by simply adding the counting component we outperform even the 8-model ensemble in Zhou et al.
Revised: Despite our single-model baseline being substantially worse than the state-of-the-art, by simply adding the counting component we outperform even the 8-model ensemble in Zhou et al.

Original: (2017) on the number category.
Revised: (2017) on the number category.

Original: We expect further improvements in number accuracy when incorporating their techniques to improve the quality of attention weights, especially since the current state-of-the-art models suffer from the problems with counting that we mention in section 3.
Revised: We expect further improvements in number accuracy when incorporating their techniques to improve the quality of attention weights, especially since the current state-of-the-art models suffer from the problems with counting that we mention in section 3.

Original: Some qualitative examples of inputs and activations within the counting component are shown in Appendix E.
Revised: Some qualitative examples of inputs and activations within the counting component are shown in Appendix E.

Original: We also evaluate our models on the validation set of VQA v2, shown in Table 2.
Revised: We also evaluate our models on the validation set of VQA v2, shown in Table 2.

Original: This allows us to consider only the counting questions within number questions, since number questions include questions such as ”what time is it?”
Revised: This allows us to consider only the counting questions within number questions, since number questions include questions such as ”what time is it?”

Original: as well.
Revised: as well.

Original: We treat any question starting with the words ”how many” as a counting question.
Revised: We treat any question starting with the words ”how many” as a counting question.

Original: As we expect, the beneﬁt of using the counting module on the counting question subset is higher than on number questions in general.
Revised: As we expect, the beneﬁt of using the counting module on the counting question subset is higher than on number questions in general.

Original: Additionally, we try an approach where we simply replace the counting module with NMS, using the average of the attention glimpses as scoring, and one-hot encoding the number of proposals left.
Revised: Additionally, we try an approach where we simply replace the counting module with NMS, using the average of the attention glimpses as scoring, and one-hot encoding the number of proposals left.

Original: Also, the NMS-based approach, using an IoU threshold of 0.5 and no score thresholding based on validation set performance, does not improve on the baseline, which suggests that the piecewise gradient of NMS is a major problem for learning to count in VQA and that conversely, there is a substantial beneﬁt of being able to differentiate through the counting module.
Revised: Also, the NMS-based approach, using an IoU threshold of 0.5 and no score thresholding based on validation set performance, does not improve on the baseline, which suggests that the piecewise gradient of NMS is a major problem for learning to count in VQA and that conversely, there is a substantial beneﬁt of being able to differentiate through the counting module.

Original: Additionally, we can evaluate the accuracy over balanced pairs as proposed by Teney et al.
Revised: Additionally, we can evaluate the accuracy over balanced pairs as proposed by Teney et al.

Original: (2017): the ratio of balanced pairs on which the VQA accuracy for both questions is 1.0.
Revised: (2017): the ratio of balanced pairs on which the VQA accuracy for both questions is 1.0.

Original: This is a much more difﬁcult metric, since it requires the model to ﬁnd the subtle details between images instead of being able to rely on question biases in the dataset.
Revised: This is a much more difﬁcult metric, since it requires the model to ﬁnd the subtle details between images instead of being able to rely on question biases in the dataset.

Original: First, notice how all binary pair accuracies are greatly reduced compared to their respective VQA accuracy.
Revised: First, notice how all binary pair accuracies are greatly reduced compared to their respective VQA accuracy.

Original: More importantly, the absolute accuracy improvement of the counting module is still fully present with the more challenging metric, which is further evidence that the component can properly count rather than simply ﬁtting better to dataset biases.
Revised: More importantly, the absolute accuracy improvement of the counting module is still fully present with the more challenging metric, which is further evidence that the component can properly count rather than simply ﬁtting better to dataset biases.

Original: When looking at the activation functions of the trained model, shown in Figure 9, we ﬁnd that some characteristics of them are shared with high-noise parametrizations of the toy dataset.
Revised: When looking at the activation functions of the trained model, shown in Figure 9, we ﬁnd that some characteristics of them are shared with high-noise parametrizations of the toy dataset.

Original: This suggests that the current attention mechanisms and object proposal network are still very inaccurate, which explains the perhaps small-seeming increase in counting performance.
Revised: This suggests that the current attention mechanisms and object proposal network are still very inaccurate, which explains the perhaps small-seeming increase in counting performance.

Original: This provides further evidence that the balanced pair accuracy is maybe a more reﬂective measure of how well current VQA models perform than the overall VQA accuracies of over 70% of the current top models.
Revised: This provides further evidence that the balanced pair accuracy is maybe a more reﬂective measure of how well current VQA models perform than the overall VQA accuracies of over 70% of the current top models.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: After understanding why VQA models struggle to count, we designed a counting component that alleviates this problem through differentiable bounding box deduplication.
Revised: After understanding why VQA models struggle to count, we designed a counting component that alleviates this problem through differentiable bounding box deduplication.

Original: The component can readily be used alongside any future improvements in VQA models, as long as they still use soft attention as all current top models on VQA v2 do.
Revised: The component can readily be used alongside any future improvements in VQA models, as long as they still use soft attention as all current top models on VQA v2 do.

Original: It has uses outside of VQA as well: for many counting tasks, it can allow an object-proposal-based approach to work without ground-truth objects available as long as there is a – possibly learned – per-proposal scoring (for example using a classiﬁcation score) and a notion of how dissimilar a pair of proposals are.
Revised: It has uses outside of VQA as well: for many counting tasks, it can allow an object-proposal-based approach to work without ground-truth objects available as long as there is a – possibly learned – per-proposal scoring (for example using a classiﬁcation score) and a notion of how dissimilar a pair of proposals are.

Original: Since each step in the component has a clear purpose and interpretation, the learned weights of the activation functions are also interpretable.
Revised: Since each step in the component has a clear purpose and interpretation, the learned weights of the activation functions are also interpretable.

Original: The design of the counting component is an example showing how by encoding inductive biases into a deep learning model, challenging problems such as counting of arbitrary objects can be approached when only relatively little supervisory information is available.
Revised: The design of the counting component is an example showing how by encoding inductive biases into a deep learning model, challenging problems such as counting of arbitrary objects can be approached when only relatively little supervisory information is available.

Original: For future research, it should be kept in mind that VQA v2 requires a versatile skill set that current models do not have.
Revised: For future research, it should be kept in mind that VQA v2 requires a versatile skill set that current models do not have.

Original: To make progress on this dataset, we advocate focusing on understanding of what the current shortcomings of models are and ﬁnding ways to mitigate them.
Revised: To make progress on this dataset, we advocate focusing on understanding of what the current shortcomings of models are and ﬁnding ways to mitigate them.

Original: A P IECEWISE LINEAR ACTIVATION FUNCTION
Revised: A P IECEWISE LINEAR ACTIVATION FUNCTION

Original: Intuitively, the interval [0 , 1] is split into d equal size intervals.
Revised: Intuitively, the interval [0 , 1] is split into d equal size intervals.

Original: Each contains a line segment that is connected to the neighboring line segments at the boundaries of the intervals.
Revised: Each contains a line segment that is connected to the neighboring line segments at the boundaries of the intervals.

Original: These line segments form the shape of the activation function.
Revised: These line segments form the shape of the activation function.

Original: For each function f k , there are d weights w k 1 , . . .
Revised: For each function f k , there are d weights w k 1 , . . .

Original: , w kd , where the weight w ki is the gradient for the interval [ i − 1 d , id ) .
Revised: , w kd , where the weight w ki is the gradient for the interval [ i − 1 d , id ) .

Original: We arbitrarily ﬁx d to be 16 in this paper, observing no signiﬁcant difference when changing it to 8 and 32 in preliminary experiments.
Revised: We arbitrarily ﬁx d to be 16 in this paper, observing no signiﬁcant difference when changing it to 8 and 32 in preliminary experiments.

Original: All w ki are enforced to be non-negative by always using the absolute value of them, which yields the monotonicity property.
Revised: All w ki are enforced to be non-negative by always using the absolute value of them, which yields the monotonicity property.

Original: Dividing the weights by (cid:80) dm | w km | yields the property that f (1)
Revised: Dividing the weights by (cid:80) dm | w km | yields the property that f (1)

Original: = 1 .
Revised: = 1 .

Original: The function can be written as
Revised: The function can be written as

Original: [Equation]
Revised: [Equation]

Original: In essence, the max term selects the two nearest boundary values of an interval, which are normalized cumulative sums over the w k weights, and linearly interpolates between the two.
Revised: In essence, the max term selects the two nearest boundary values of an interval, which are normalized cumulative sums over the w k weights, and linearly interpolates between the two.

Original: This approach is similar to the subgradient approach by Jaderberg et al.
Revised: This approach is similar to the subgradient approach by Jaderberg et al.

Original: (2015) to make sampling from indices differentiable.
Revised: (2015) to make sampling from indices differentiable.

Original: All w ki are initialized to 1, which makes the functions linear on initialization.
Revised: All w ki are initialized to 1, which makes the functions linear on initialization.

Original: When applying f k ( x ) to a vector-valued input x , it is assumed to be applied elementwise.
Revised: When applying f k ( x ) to a vector-valued input x , it is assumed to be applied elementwise.

Original: By caching the normalized cumulative sum (cid:80) ij | w kj | / (cid:80) dm | w km | , this function has linear time complexity with respect to d and is efﬁciently implementable on GPUs.
Revised: By caching the normalized cumulative sum (cid:80) ij | w kj | / (cid:80) dm | w km | , this function has linear time complexity with respect to d and is efﬁciently implementable on GPUs.

Original: Extensions to this are possible through Deep Lattice Networks (You et al., 2017), which preserve monotonicity across several nonlinear neural network layers.
Revised: Extensions to this are possible through Deep Lattice Networks (You et al., 2017), which preserve monotonicity across several nonlinear neural network layers.

Original: They would allow A and D to be combined in more sophisticated ways beyond an elementwise product, possibly improving counting performance as long as the property of the range lying within [0 , 1] is still enforced in some way.
Revised: They would allow A and D to be combined in more sophisticated ways beyond an elementwise product, possibly improving counting performance as long as the property of the range lying within [0 , 1] is still enforced in some way.

Original: B B ASELINE ARCHITECTURE
Revised: B B ASELINE ARCHITECTURE

Original: This model is based on the work of Kazemi & Elqursh (2017), who outperformed most previous
Revised: This model is based on the work of Kazemi & Elqursh (2017), who outperformed most previous

Original: VQA models on the VQA v1 dataset with a simple baseline architecture.
Revised: VQA models on the VQA v1 dataset with a simple baseline architecture.

Original: We adapt the model to the
Revised: We adapt the model to the

Original: VQA v2 dataset and make various tweaks that improve validation accuracy slightly.
Revised: VQA v2 dataset and make various tweaks that improve validation accuracy slightly.

Original: The architecture is illustrated in Figure 6.
Revised: The architecture is illustrated in Figure 6.

Original: Details not mentioned here can be assumed to be the same as in their paper.
Revised: Details not mentioned here can be assumed to be the same as in their paper.

Original: The most signiﬁcant change that we make is the use of object proposal features by Anderson et al.
Revised: The most signiﬁcant change that we make is the use of object proposal features by Anderson et al.

Original: (2017) as previously mentioned.
Revised: (2017) as previously mentioned.

Original: The following tweaks were made without considering the performance impact on the counting component; only the validation accuracy of the baseline was optimized.
Revised: The following tweaks were made without considering the performance impact on the counting component; only the validation accuracy of the baseline was optimized.

Original: To fuse vision features x and question features y , the baseline concatenates and linearly projects them, followed by a ReLU activation.
Revised: To fuse vision features x and question features y , the baseline concatenates and linearly projects them, followed by a ReLU activation.

Original: This is equivalent to ReLU ( W x x + W y y ) .
Revised: This is equivalent to ReLU ( W x x + W y y ) .

Original: We include an additional term that measures how different the projected x is from the projected y , changing the fusion mechanism to x (cid:5)
Revised: We include an additional term that measures how different the projected x is from the projected y , changing the fusion mechanism to x (cid:5)

Original: y = ReLU ( W x x + W y y ) − ( W x x − W y y ) 2 .
Revised: y = ReLU ( W x x + W y y ) − ( W x x − W y y ) 2 .

Original: The LSTM (Hochreiter & Schmidhuber, 1997) for question encoding is replaced with a GRU (Cho et al., 2014) with the same hidden size with dynamic per-example unrolling instead of a ﬁxedwords per question.
Revised: The LSTM (Hochreiter & Schmidhuber, 1997) for question encoding is replaced with a GRU (Cho et al., 2014) with the same hidden size with dynamic per-example unrolling instead of a ﬁxedwords per question.

Original: We apply batch normalization (Ioffe & Szegedy, 2015) before the last linear projection in the classiﬁer to the 3000 classes.
Revised: We apply batch normalization (Ioffe & Szegedy, 2015) before the last linear projection in the classiﬁer to the 3000 classes.

Original: The learning rate is increased from 0.001 to 0.
Revised: The learning rate is increased from 0.001 to 0.

Original: and the batch size is doubled to 256.
Revised: and the batch size is doubled to 256.

Original: The model is trained for 100 epochs (1697 iterations per epoch to train on the training set, 2517 iterations per epoch to train on both training and validation sets) instead of 100,000 iterations, roughly in line with the doubling of dataset size when going from VQA v1 to VQA v2.
Revised: The model is trained for 100 epochs (1697 iterations per epoch to train on the training set, 2517 iterations per epoch to train on both training and validation sets) instead of 100,000 iterations, roughly in line with the doubling of dataset size when going from VQA v1 to VQA v2.

Original: Note that this single-model baseline is regularized with dropout (Srivastava et al., 2014), while the other current top models skip this and rely on ensembling to reduce overﬁtting.
Revised: Note that this single-model baseline is regularized with dropout (Srivastava et al., 2014), while the other current top models skip this and rely on ensembling to reduce overﬁtting.

Original: This explains why our single-model baseline outperforms most single-model results of the state-of-the-art models.
Revised: This explains why our single-model baseline outperforms most single-model results of the state-of-the-art models.

Original: We found ensembling of the regularized baseline to provide a much smaller beneﬁt in preliminary experiments compared to the results of ensembling unregularized networks reported in Teney et al. 
Revised: We found ensembling of the regularized baseline to provide a much smaller beneﬁt in preliminary experiments compared to the results of ensembling unregularized networks reported in Teney et al. [Figure] Image [Figure]

Revised: -CNN

Original: [Figure]
Revised: [Figure]

Original: Question
Revised: Question

Original: Embedding
Revised: Embedding

Original: GRU [Figure]
Revised: GRU (cid:5) w softmax (cid:80) (cid:5) +

Original:  w [Figure]
Revised: BN w softmax

Original: Count
Revised: Count

Original: [Figure]
Revised: BN

Original:  w w w w w
Revised: ReLU w w w w w

Original: C F ULL PLOTS OF ACTIVATION FUNCTIONS
Revised: C F ULL PLOTS OF ACTIVATION FUNCTIONS

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: D E XAMPLE TOY DATASET DATA
Revised: D E XAMPLE TOY DATASET DATA

Original: [Figure]
Revised: [Figure]

Original: E Q UALITATIVE EXAMPLES OF INTERMEDIATE ACTIVATIONS
Revised: E Q UALITATIVE EXAMPLES OF INTERMEDIATE ACTIVATIONS

Original: How many birds?
Revised: How many birds?

Original: c = 1 .
Revised: c = 1 .

Original: Predicted =
Revised: Predicted =

Original: Ground truth =
Revised: Ground truth =

Original: A
Revised: A

Original: D
Revised: D

Original: C
Revised: C

Original: How many athletes are on the ﬁeld?
Revised: How many athletes are on the ﬁeld?

Original: c = 5 .
Revised: c = 5 .

Original: Predicted =
Revised: Predicted =

Original: Ground truth =
Revised: Ground truth =

Original: A
Revised: A

Original: D
Revised: D

Original: C
Revised: C

Original: How many people are visible in this picture?
Revised: How many people are visible in this picture?

Original: c = 0 .
Revised: c = 0 .

Original: Predicted =
Revised: Predicted =

Original: Ground truth =
Revised: Ground truth =

Original: A
Revised: A

Original: D
Revised: D

Original: C
Revised: C

Original: How many cows?
Revised: How many cows?

Original: c = 4 .
Revised: c = 4 .

Original: Predicted =
Revised: Predicted =

Original: Ground truth =
Revised: Ground truth =

Original: A
Revised: A

Original: D
Revised: D

Original: C
Revised: C