Original: M EASURING CLEVR NESS : B LACK - BOX T ESTING OF V ISUAL R EASONING M ODELS
Revised: M EASURING CLEVR NESS : B LACK - BOX T ESTING OF V ISUAL R EASONING M ODELS

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: How to measure the reasoning capabilities of intelligence systems?
Revised: How can we measure the reasoning capabilities of intelligence systems?

Original: Visual question answering provides a convenient framework for testing the model’s abilities by interrogating the model through questions about the scene.
Revised: Visual question answering provides a convenient framework for testing the model’s abilities by interrogating the model through questions about the scene.

Original: However, despite the score of various visual QA datasets and architectures, which sometimes yield even a super-human performance, the question of whether those architectures can do reasoning remains open.
Revised: However, despite scores of various visual QA datasets and architectures, which sometimes yield even a super-human performance, the question of whether those architectures can actually reason remains open to debate.

Original: To answer such a question, we extend the visual question answering framework and propose the following behavioural test in the form of a two-player game.
Revised: To answer this, we extend the visual question answering framework and propose the following behavioral test in the form of a two-player game.

Original: We consider black-box neural models of CLEVR.
Revised: We consider black-box neural models of CLEVR.

Original: Those models are trained on a diagnostic dataset benchmarking reasoning.
Revised: These models are trained on a diagnostic dataset benchmarking reasoning.

Original: Next, we train an adversarial player that re-conﬁgures the scene to fool the CLEVR model.
Revised: Next, we train an adversarial player that re-configures the scene to fool the CLEVR model.

Original: We show that CLEVR models, which otherwise may perform at a human level, can easily be fooled by our agent.
Revised: We show that CLEVR models, which otherwise could perform at a “human level”, can easily be fooled by our agent.

Original: Our results question one more time whether datadriven approaches can do reasoning without exploiting  numerous biases that are often present in those datasets.
Revised: Our results put in doubt whether data-driven approaches can do reasoning without exploiting the numerous biases that are often present in those datasets.

Original: Finally, we also propose a controlled experiment measuring the efﬁciency of such models to learn and perform reasoning.
Revised: Finally, we also propose a controlled experiment measuring the efficiency of such models to learn and perform reasoning

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Are our artiﬁcial intelligence systems capable of reasoning?
Revised: Are our artificial intelligence systems capable of reasoning?

Original: Or like Clever Hans , they use various cues only tangentially related to the task and rely on rote memorization with poor generalization?
Revised: Or like Clever Hans , they use various cues only tangentially related to the task and rely on rote memorization with poor generalization?

Original: (Pfungst, 1911; Johnson et al., 2017a)
Revised: (Pfungst, 1911; Johnson et al., 2017a)

Original: This work revisits such a question and proposes an interactive framework with the communication channel between two players.
Revised: This work revisits such a question and proposes an interactive framework with the communication channel between two players.

Original: The ﬁrst player, which reasoning capabilities we are about to test, performs visual reasoning tasks.
Revised: The first player, which reasoning capabilities we are about to test, performs visual reasoning tasks, we call it VisualQA Player .

Original: The second player, which we call the Adversarial Player , is manipulating the scene so that it fools the ﬁrst player even though those changes still lead to correct reasoning steps among humans.
Revised: The second player, which we call the Adversarial Player , is manipulating the scene so that it fools the first player even though those changes still lead to correct reasoning steps among humans.

Original: Both players interact with each other only through questions, answers and the visual scene as shown in Figure 1.
Revised: Both players interact with each other only through questions, answers and the visual scene as shown in Figure 1.

Original: Since we use the visual question answering framework as the interrogation medium, assuming that both players can deal with textual questions and answers, we also call the ﬁrst player the Visual-QA Player .

Original: If the Adversarial Player manipulating the scene causes the Visual-QA Player to change its answer even though the new scene is still valid for the same question and answer, it is then the reasoning failure.
Revised: If the Adversarial Player manipulating the scene causes the Visual-QA Player to change its answer even though the new scene is still valid for the same question and answer, it is then the reasoning failure.

Original: It is similar to the following situation.
Revised: It is similar to the following situation.

Original: Imagine a box is placed between two spheres.
Revised: Imagine a box is placed between two spheres.

Original: If you ask a question, is there a box between two spheres?
Revised: If you ask a question, is there a box between two spheres?

Original: , the answer should be positive.
Revised: , the answer should be positive.

Original: Now, if we move the box anywhere so it does not cross any of the spheres, and ask the same question, the response should remain unchanged.
Revised: Now, if we move the box anywhere so it does not cross any of the spheres, and ask the same question, the response should remain unchanged.

Original: In other words, we postulate that reasoning outputs of agents need to be invariant under scene conﬁgurations that are consistent with the questions-answer pairs.
Revised: In other words, we postulate that reasoning outputs of agents need to be invariant under scene configurations that are consistent with the questions-answer pairs.

Original: Moreover, in the spirit of generic adversarial attacks (Szegedy et al., 2014; Goodfellow et al., 2015), we seek conﬁgurations that also pose little if any reasoning challenges for humans.
Revised: Moreover, in the spirit of generic adversarial attacks, we seek configurations that also pose little if any reasoning challenges for humans.

Original: In this work, we propose an automatic and agnostic pipeline to benchmark the reasoning capabilities of various models. The only assumption that we make about those models is that they canperceive the scene and communicate by answering questions about the scene.
Revised: We propose an automatic and agnostic pipeline to benchmark the reasoning capabilities of various models, only assuming they can communicate by answering questions about the scene.

Original: Due to the recent stream of research in vision-and-language (Zhang et al., 2021; Jiang et al., 2020; Guo et al., 2019b; Su et al., 2020; Wang et al., 2020; Kamath et al., 2021; Tan & Bansal, 2019; Chen et al., 2020), we believe there will be an increasing number of vision models thatcan operate through language.
Revised: Due to the recent stream of research in vision-and-language (Zhang et al., 2021; Jiang et al., 2020; Guo et al., 2019b; Su et al., 2020; Wang et al., 2020; Kamath et al., 2021; Tan & Bansal, 2019; Chen et al., 2020), we believe there will be an increasing number of vision models that operate through language.

Original: Moreover, we also consider the visual question answering framework set-up as a two-player system as an excellent benchmarking pipeline.
Revised: Moreover, we also consider the visual question answering framework set-up as a two-player system as an excellent benchmarking pipeline.

Original: We perform all tests by scene manipulations and observing how a tested model behaves under such changes.
Revised: We perform all tests by scene manipulations and observing how a tested model behaves under such changes.

Original: The pipeline does not require any knowledge of the internals of the tested model.
Revised: The pipeline does not require any knowledge of the internals of the tested model.

Original: It also does not manipulate the sensory information of such a model, i.e., pixels in the images, and all the manipulations are physically meaningful.
Revised: It also does not manipulate the sensory information of such a model, e.g., pixels in the images, and all the manipulations are physically meaningful.

Original: Even though our current pipeline uses synthetic scenes as only those can easily be automatically manipulated, our results have also real-world ramiﬁcations.
Revised: Even though our current pipeline uses synthetic scenes as only those can easily be automatically manipulated, our results have also real-world ramifications.

Original: If models are susceptible to semantically meaningless changes 1 in scene conﬁgurations, in a synthetic setting, there are valid concerns that real-world robots could also be prone to manipulation of objects in a room.
Revised: If models are susceptible to semantically meaningless changes 1 in scene configurations, in a synthetic setting, there are valid concerns that real-world robots could also be prone to manipulation of objects in a room.

Original: Finally, our work also questions the possibility of training and benchmarking networks in a purely data-driven and ofﬂine, static manner.
Revised: Finally, our work also questions the possibility of training and benchmarking networks in a purely data-driven and offline, static manner.

Original: Although most of our CLEVR models can reach a ‘superhuman’ performance on CLEVR, they are much less-CLEVR if another agent can manipulate the scene.

Original: Our results show that those models fail on various reasoning tasks, even though a relatively simple interpolation between conﬁgurations is possible; where even no new features are introduced.

Original: We also show that the existing models may be inefﬁcient in learning to perform reasoning.

Original: Contributions.
Revised: Contributions.

Original: The main contributions of our work could be summarized in three points.
Revised: The main contributions of our work could be summarized in three points.

Original: First , we propose a strong black-box adversarial test, which makes no assumptions about  underlying mechanics of a tested model, formulated as a game between two players.
Revised: First , we propose a strong black-box adversarial test, which makes no assumptions about the underlying mechanics of a tested model, formulated as a game between two players.

Original: Our test does not require any direct access to the tested model, even through its sensory information.
Revised: Our test does not require any direct access to the tested model, even through its sensory information.

Original: In particular, it does not require gradients, output probabilities, or any access to the perceived image.
Revised: In particular, it does not require gradients, output probabilities, or any access to the perceived image.

Original: Our work also deviates from bounded perturbations and instead focuses on global scene manipulations that are still consistent with the task constraints, and can change the behaviour of a tested model.
Revised: Our work also deviates from bounded perturbations and instead focuses on global scene manipulations that are still consistent with the task constraints, and can change the behavior of a tested model.

Original: Second , we reformulate visual reasoning by integrating visual question answering with zero-sum two-player game frameworks.
Revised: Second , we reformulate visual reasoning by integrating visual question answering with zero-sum two-player game frameworks.

Original: Under our novel formulation, a visual and adversary agents compete against each other through content manipulation.
Revised: Under our novel formulation, a visual and adversary agents compete against each other through content manipulation.

Original: We believe that this is an initial step towards more sophisticated frameworks that integrate computer vision with multi-agent systems.
Revised: We believe that this is an initial step towards more sophisticated frameworks that integrate computer vision with multi-agent systems.

Original: Third , we explore the limits of the data-driven approaches in synthetic visual scenarios, and demonstrate that current CLEVR models are lacking the efﬁciency to learn robust reasoning steps.
Revised: Third , we explore the limits of the data-driven approaches in synthetic visual scenarios, and demonstrate that current CLEVR models are lacking the efficiency to learn robust reasoning steps.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Our work touches upon various research directions, which we brieﬂy describe here.
Revised: Our work touches upon various research directions, which we briefly describe here.

Original: Visual QA.
Revised: Visual QA.

Original: Introduced as a visual counter-part of the Turing Test (Malinowski & Fritz, 2014; Geman et al., 2015), it became a computer vision task that requires a holistic visual understanding.
Revised: Introduced as a visual counter-part of the Turing Test (Malinowski & Fritz, 2014; Geman et al., 2015), it became a computer vision task that requires a holistic visual understanding.

Original: Many other works have extended the task to larger datasets or videos or adversarial train-test splits (Antol et al., 2015;
Revised: Many other works have extended the task to larger datasets or videos or adversarial train-test splits (Antol et al., 2015;

Original: Agrawal et al., 2018; Tapaswi et al., 2016; Yu et al.,2019; Girdhar & Ramanan, 2019).
Revised: Agrawal et al., 2018; Tapaswi et al., 2016; Yu et al., 2019).

Original: More recently, we also observe the generalization of the task to become a part of the vision-pluslanguage suite of benchmarks (Lu et al., 2019; Chen et al., 2019; Wang et al., 2021).
Revised: More recently, we also observe the generalization of the task to become a part of the vision-plus-language suite of benchmarks (Lu et al., 2019; Chen et al., 2019; Wang et al., 2021).

Original: Johnson et al.
Revised: Johnson et al.

Original: (2017a) have introduced a synthetic variant of the Visual QA problem that is more focused on reasoning and the control of the experimentation.
Revised: (2017a) have introduced a synthetic variant of the Visual QA problem that is more focused on reasoning and the control of the experimentation.

Original: Although it was shown that traditional methods (Yang et al., 2016) are not enough to solve that dataset, newer methods can pass the human baseline on this task.
Revised: Although it was shown that traditional methods (Yang et al., 2016) are not enough to solve that dataset, newer methods can pass the human baseline on this task.

Original: Those methods are trained to operate on pixels and text directly (Santoro et al., 2017; Perez et al., 2018; Hudson & Manning, 2018; Malinowski et al., 2018; Kamath et al., 2021) or they induce programs as an intermediate representation (Johnson et al., 2017b; Mascharka et al., 2018;
Revised: These methods are trained to operate on pixels and text directly (Santoro et al., 2017; Perez et al., 2018; Hudson & Manning, 2018; Malinowski et al., 2018; Kamath et al., 2021) or they induce programs as an intermediate representation (Johnson et al., 2017b; Mascharka et al., 2018;

Original: Yi et al., 2018).
Revised: Yi et al., 2018).

Revised: Hudson & Manning (2019) have shown that Visual QA models lack some reasoning robustness but their approach is mostly linguistic and still static.

Revised: Some other works also observe issues with static evaluation and proposed more dynamic benchmarks, e.g., with a human-in-the-loop (Khashabi et al., 2021; Nie et al., 2020; Li et al., 2021; Sheng et al., 2021).

Original: We extend this line of research by reformulating the Visual QA task as an interactive twoagents system, where we automatically test the reasoning capabilities of some methods above in the controlled setting. We also show that CLEVR under such a new setting is still an unsolved problem. 
Revised: We extend the line of research on Visual QA by reformulating it as an interactive two-agents system, and show that CLEVR under such a new setting is still an unsolved problem. Our work also differs from previous works in that it focuses on visual reasoning, is interactive and fully automatic.

Original: Adversarial perturbations.
Revised: Adversarial perturbations.

Original: Szegedy et al.
Revised: Szegedy et al.

Original: ; Goodfellow et al.
Revised: ; Goodfellow et al.

Original: ; Kurakin et al.
Revised: ; Kurakin et al.

Original: Moosavi-Dezfooli et al.

Original: ; Carlini & Wagner (2016) have introduced early methods that can ‘fool’ networks by performing tiny, visually imperceptible, perturbations of the input image.
Revised: Carlini & Wagner (2016) have introduced early methods that can ‘fool’ networks by performing tiny, visually imperceptible, perturbations of the input image.

Original: Moosavi-Dezfooli et al.
Revised: Moosavi-Dezfooli et al.

Original: (2017) have also shown universal and untargeted adversarial perturbations that are more transferable than previous approaches.
Revised: (2017) have also shown universal and untargeted adversarial perturbations that are more transferable than previous approaches.

Original: Most methods perform white-box attacks, where an adversarial model has access to the target model’s parameters or gradients.
Revised: Most methods perform white-box attacks, where an adversarial model has access to the target model’s parameters or gradients.

Original: In black-box attacks such access is removed and, e.g., only output probabilities are available (Guo et al., 2019a).
Revised: In black-box attacks such access is removed and, e.g., only output probabilities are available (Guo et al., 2019a).

Original: Other black-box attacks involve the use of surrogates to estimate gradients of the real target (Papernot et al., 2017; Cheng et al., 2019) or gradient-free methods (Alzantot et al., 2018).
Revised: Other black-box attacks involve the use of surrogates to estimate gradients of the real target (Papernot et al., 2017; Cheng et al., 2019) or gradient-free methods (Alzantot et al., 2018).

Original: More related to our work, Cheng et al.
Revised: More related to our work, Cheng et al.

Original: (2018) describe a black-box system, which only assumesthe access to outputs, so called hard-labels, of the network for a given query.
Revised: (2018) describe a black-box system, which only assumes access to outputs of the network.

Original: However, all the perturbations described above are continuous-based where individual pixels are changed independently.
Revised: However, all the perturbations described above are continuous-based where individual pixels are changed independently.

Original: This leads to improbable images and assumesthe access to the sensory inputs of the neural network.
Revised: This leads to improbable images and assumes access to the sensory inputs of the neural network.

Original: In contrary, semantic perturbations (Joshi et al., 2019; Zeng et al., 2019), operate on semantically meaningful chunks of the input, leading to more plausible and physically interpretable perturbations.
Revised: In contrary, semantic perturbations (Joshi et al., 2019; Zeng et al., 2019), operate on semantically meaningful chunks of the input, leading to more plausible and physically interpretable perturbations.

Original: However, they still assumea direct access to either images or ideally differentiable renderers.
Revised: However, they still assume direct access to either images or ideally differentiable renderers.

Original: We can interpret our work within the adversarial perturbations framework, where the adversarial model can change the original scene by its manipulations, with zero access to the target model, including its sensory information and not limited to tiny changes but instead to semantically and globally meaningless modiﬁcations.
Revised: We can interpret our work within the adversarial perturbations framework, where the adversarial model can change the original scene by its manipulations, with zero access to the target model, including its sensory information and not limited to tiny changes but instead to semantically and globally meaningless modifications.

Original: Reinforcement learning.
Revised: Reinforcement learning.

Original: The game between the Visual-QA Player and Adversarial Player is a zero-sum two-player game.
Revised: The game between the Visual-QA Player and Adversarial Player is a zero-sum two-player game.

Original: Conneau et al.
Revised: Conneau et al.

Original: (2017) use a conceptually similar two-player game to ﬁnd a better word alignment between two languages without parallel corpus.
Revised: (2017) use a conceptually similar two-player game to find a better word alignment between two languages without parallel corpus.

Original: Inspired by such a problem formulation, we consider our two-player game as a tool to achieve better reasoning models.
Revised: Inspired by such a problem formulation, we consider our two-player game as a tool to achieve better reasoning models.

Original: We optimize the Adversarial Player with Advantage Actor-Critic (A2C) algorithm (Sutton & Barto, 2018; Degris et al., 2012).
Revised: We optimize the Adversarial Player with Advantage Actor-Critic (A2C) algorithm (Sutton & Barto, 2018; Degris et al., 2012).

Original: Though there are two players in the game between Visual-QA Player and

Original: Adversarial Player , in experiments presented in this work, the Visual-QA Player is frozen.

Original: Thus, technically, simpler optimization methods used for single-player games are applicable.

Original: Probing and measuring intelligence.
Revised: Probing and measuring intelligence.

Original: Turing Test (Turing, 2009) is considered to be among the earliest works on measuring the intelligence of artiﬁcial systems.
Revised: Turing Test (Turing, 2009) is considered to be among the earliest works on measuring the intelligence of artificial systems.

Original: Crucially to us, it is also formulated as a two-player game with an interrogation protocol.
Revised: Crucially to us, it is also formulated as a two-player game with an interrogation protocol.

Original: Legg (2008) systematizes and relates  notion of intelligence from different ﬁelds with the main focus on ‘universality’.
Revised: Legg (2008) systematizes and relates the notion of intelligence from different fields with the main focus on ‘universality’.

Original: Similarly, Chollet (2019) investigates various attempts at deﬁning or measuring intelligence and highlights the role of skill-acquisition efﬁciency.

Original: Johnson et al.
Revised: Johnson et al.

Original: (2017a) have transferred the notion of intelligence onto the visual ground and emphasize reasoning rather than universality.
Revised: (2017a) have transferred the notion of intelligence onto the visual ground and emphasized reasoning rather than universality.

Original: Nonetheless, it seems there is noa single widely acceptable dataset or a deﬁnition that encompasses our intuitions about intelligence.
Revised: It seems there is no widely accepted dataset or definition that encompasses our intuitions about intelligence.

Original: However, there are a few recent directions showing the lack thereof, mainly highlighting bias ampliﬁcation or absence of mathematical capabilities (Hendricks et al., 2018; Bhardwaj et al., 2021; Piekos et al., 2021).
Revised: However, there are a few recent directions showing the lack thereof, mainly highlighting bias amplification or absence of mathematical capabilities (Hendricks et al., 2018; Bhardwaj et al., 2021; Piekos et al., 2021).

Original: Here, we are mostly interested in visual reasoning and use the visual question answering framework as a form of communication with the otherwise black-box system.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: In this section, we explain brieﬂy how the CLEVR dataset (Johnson et al., 2017a) is constructed and introduce our notation and deﬁnitions.
Revised: In this section, we explain briefly how the CLEVR dataset (Johnson et al., 2017a) is constructed and introduce our notation and definitions.

Original: CLEVR is a synthetic visual question answering dataset introduced by Johnson et al.
Revised: CLEVR is a synthetic visual question answering dataset introduced by Johnson et al.

Original: (2017a), which consists of about 700k training and 150k validation image-question-answer triplets.
Revised: (2017a), which consists of about 700k training and 150k validation image-question-answer triplets.

Original: Images are artiﬁcially constructed and rendered from scene graphs – a special structure containing information about object attributes such as position or color.
Revised: Images are artificially constructed and rendered from scene graphs – a special structure containing information about object attributes such as position or color.

Original: Such a scene graph is also used to synthesize the ground-truth question-answer pairs by expanding templates according to the depth-ﬁrst-search ordering.
Revised: Such a scene graph is also used to synthesize the ground-truth question-answer pairs by expanding templates according to the depth-first-search ordering.

Original: Ambiguous scenes are rejected.
Revised: Ambiguous scenes are rejected.

Original: Each image represents an isometric view on the scene containing from two to ten objects.
Revised: Each image represents an isometric view of the scene containing from two to ten objects.

Original: There are three classes of objects: spheres, cubes and cylinders.
Revised: There are three classes of objects, spheres , cubes and cylinders .

Original: Each object can also be either large or small and has one colour out of four (brown, purple, cyan, yellow).
Revised: Each object can also be either large or small and has one color out of four (brown, purple, cyan, yellow).

Original: It can also be either metallic or rubber-made.
Revised: It can also be either metallic or rubber-made.

Original: Every object has x and y coordinates that are conﬁned within the ( − 3 , +3) range.
Revised: Every object has x and y coordinates that are confined within the ( − 3 , +3) range.

Original: We use the same generation process to render modiﬁed scenes.
Revised: We use the same generation process to render modified scenes.

Original: CLEVR models.
Revised: CLEVR models.

Original: Various models have been introduced to work with the CLEVR dataset, some even ‘solving’ the dataset by achieving near perfect performance.
Revised: Various models have been introduced to work with the CLEVR dataset, some even ‘solving’ the dataset by achieving near perfect performance.

Original: Despite of strong ofﬂine performance, we test if those models performance perpetuates in the more interactive setting where conﬁgurations of the scene could be changed.
Revised: Despite the strong offline performance, we test if those models’ performance perpetuates in the more interactive setting where configurations of the scene could be changed.

Original: Whenever possible, we use pre-trained CLEVR models.
Revised: Whenever possible, we use pre-trained CLEVR models.

Original: Otherwise, we train the remaining models from scratch by making sure we achieve results similar to published accuracy numbers on the validation set.
Revised: Otherwise, we train the remaining models from scratch by making sure we achieve results similar to published accuracy numbers on the validation set.

Original: We summarize all the models in Table 1.
Revised: We summarize all the models in Table 1.

Original: We show the accuracy on the CLEVR dataset ( Accuracy ), indicate if an architecture is trained from scratch ( Re-trained ), brieﬂy describe how multimodal fusion and reasoning is conducted ( Reasoning Mechanism ), and indicate any extra privileged information required during the training process ( Extra ).
Revised: We show the accuracy on the CLEVR dataset ( Accuracy ), indicate if an architecture is trained from scratch ( Re-trained ), briefly describe how multi-modal fusion and reasoning is conducted ( Reasoning Mechanism ), and indicate any extra privileged information required during the training process ( Extra ).

Original: For instance, some models require extra access to functional programs used during the dataset generation, use scene graphs as a supervisory signal (states), or always operate on scene graphs (input-states).
Revised: For instance, some models require extra access to functional programs used during the dataset generation, use scene graphs as a supervisory signal (states), or always operate on scene graphs (input-states).

Original: Otherwise the models were trained only from image-question-answer triples.
Revised: Otherwise, the models were trained only from image-question-answer triples.

Original: Mini-games .
Revised: Mini-games .

Original: We formulate our problem as a Game between two players: Visual-QA Player and
Revised: We formulate our problem as a Game between two players, Visual-QA Player and

Original: Adversarial Player .
Revised: Adversarial Player .

Original: The Visual-QA Player takes as input question-image pairs and provide answers to such questions.
Revised: The Visual-QA Player takes as input question-image pairs and provide answers to such questions.

Original: Some models use states (scene-graphs) that replace images or require programs (Johnson et al., 2017a).
Revised: Some models use states (scene-graphs) that replace images or require programs (Johnson et al., 2017a).

Original: The whole game consists of all CLEVR data points.
Revised: The whole game consists of all CLEVR data points.

Original: For our purpose, we extend the notion of the Game into Mini-games .
Revised: For our purpose, we extend the notion of the Game into Mini-games .

Original: The rules of Mini-games are identical to the whole Game .
Revised: The rules of Mini-games are identical to the whole Game .

Original: The only difference is that each Mini-game operates on a subset of the CLEVR dataset.
Revised: The only difference is that each Mini-game operates on a subset of the CLEVR dataset.

Original: We deﬁne the size of a Mini-game by the number of datapoints that are attached to that Minigame .
Revised: We define the size of a Mini-game by the number of datapoints that are attached to that Minigame .

Original: We sample data points for each Mini-game randomly and mutually exclusively.
Revised: We sample data points for each Mini-game randomly and mutually exclusively.

Original: Mini-games have analogies in the adversarial perturbations literature.
Revised: Mini-games have analogies in the adversarial perturbations literature.

Original: Mini-games of size one resemble perimage adversarial perturbations (Goodfellow et al., 2015; Moosavi-Dezfooli et al., 2016) whereas a
Revised: Mini-games of size one resemble perimage adversarial perturbations (Goodfellow et al., 2015; Moosavi-Dezfooli et al., 2016) whereas a

Original: Mini-game that has all data pointsis similar to universal adversarial perturbations (Moosavi-Dezfooli et al., 2017).
Revised: Mini-game that has all data pointsis similar to universal adversarial perturbations (Moosavi-Dezfooli et al., 2017).

Original: In this work, we investigate various Mini-game sizes but due to the sheer scale we were unable to use the whole game as the Mini-game .
Revised: In this work, we investigate various Mini-game sizes but due to the sheer scale we were unable to use the whole game as the Mini-game .

Original: Larger Mini-games make the optimization process more difﬁcult as the domain where the Adversarial Player needs to operate increases.
Revised: Larger Mini-games make the optimization process more difficult as the domain where the Adversarial Player needs to operate increases.

Original: The training is also much more time-consuming and a sequential process.
Revised: The training is also much more time-consuming and a sequential process.

Original: Instead, we can train multiple players on different Mini-games independently and thus massively.
Revised: Instead, we can train multiple players on different Mini-games independently and thus massively.

Original: We leave the arduous training of the universal Adversarial Player on the whole Game as a possible future direction.
Revised: We leave the arduous training of the universal Adversarial Player on the whole Game as a possible future direction.

Original: [Table]
Revised: [Table]

Original:  E NVIRONMENT
Revised:  E NVIRONMENT

Original: We need to ensure that Adversarial Players create valid scenes that are consistent and in-distribution .
Revised: We need to ensure that Adversarial Players create valid scenes that are consistent and in-distribution .

Original: Both properties are guaranteed by our environment enforcers.
Revised: Both properties are guaranteed by our environment enforcers.

Original: Consistency.
Revised: Consistency.

Original: Since scene manipulation may change the answer for a given question, we need to ensure this does not happen.
Revised: Since scene manipulation may change the answer for a given question, we need to ensure this does not happen.

Original: That is, the new scene is still consistent with the question-answer pair.
Revised: That is, the new scene is still consistent with the question-answer pair.

Original: The question-relevance enforcer achieves that by running functional programs associated with each question (Johnson et al., 2017a) on the modiﬁed scene-graph.
Revised: The question-relevance enforcer achieves that by running functional programs associated with each question (Johnson et al., 2017a) on the modified scene-graph.

Original: Hence, it gets the new ground-truth answer.
Revised: Hence, it gets the new ground-truth answer.

Original: The enforcer rejects the new scene if that new answer differs from the previous one.
Revised: The enforcer rejects the new scene if that new answer differs from the previous one.

Original: In this way, it guarantees the newly generated scenes give the same answers as the original scenes on the same question.
Revised: In this way, it guarantees the newly generated scenes give the same answers as the original scenes on the same question.

Original: Thus, we can generate equivalent scenes containing the same objects that have identical answers for the same questions.
Revised: Thus, we can generate equivalent scenes containing the same objects that have identical answers for the same questions.

Original: Using that enforcer, we can test if the Visual-QA Player ’s answers are invariant under such an equivalent class of scenes.
Revised: Using that enforcer, we can test if the Visual-QA Player ’s answers are invariant under such an equivalent class of scenes.

Original: In-distribution.
Revised: In-distribution.

Original: Even with the question-relevance enforcer, the Adversarial Player may still produce undesired outputs.
Revised: Even with the question-relevance enforcer, the Adversarial Player may still produce undesired outputs.

Original: For instance, it can stretch the whole scene thus violating the scene boundaries from the original CLEVR dataset, making, e.g., everything to look very small (Section A.9 in the appendix).
Revised: For instance, it can stretch the whole scene thus violating the scene boundaries from the original CLEVR dataset, making, e.g., everything to look very small (Section A.9 in the appendix).

Original: Although it is still an interesting form ofthe adversarial scene manipulation, we focus rather on the in-distribution scene manipulations that respect the original boundaries.
Revised: Although it is still an interesting form of adversarial scene manipulation, we focus rather on the in-distribution scene manipulations that respect the original boundaries.

Original: To enforce that property, we use  scene-constraint enforcer that checks the boundaries of the scene.
Revised: To enforce that property, we use a scene-constraint enforcer that checks the boundaries of the scene.

Original: Without that enforcer, the Adversarial Player would quickly resort to stretching the whole scenes, achieving a form of adversarial attack that uses distribution shifts rather than content manipulation.
Revised: Without that enforcer, the Adversarial Player would quickly resort to stretching the whole scenes, achieving a form of adversarial attack that uses distribution shifts rather than content manipulation.

Original: It does so, e.g., by moving the camera away until objects are barely visible.
Revised: It does so, e.g., by moving the camera away until objects are barely visible.

Original: We give a few such examples in the appendix (Section A.9).
Revised: We give a few such examples in the appendix (Section A.9).

Original:  Adversarial Player
Revised:  Adversarial Player

Original: In this section, we describe our Adversarial Player in details.

Original: Meaningful scene manipulations require not only generic scene understanding, but also the ability to distinguish which objects to displace and how.
Revised: Meaningful scene manipulations require not only generic scene understanding, but also the ability to distinguish which objects to displace and how.

Original: Hence, the player is a composition of a multi-modal module, which creates input representation, and a decision maker, which decide how to manipulate the scene.
Revised: Hence, the player is a composition of a multi-modal module, which creates input representation, and a decision maker, which decides how to control the scene.

Original: In the following, we will describe that composition of modules, the game of scene manipulations, and the training protocol.

Original: Figure 1 illustrates the Adversarial Player and the game between both players.
Revised: Figure 1 illustrates the Adversarial Player and the game between both players.

Original: Multi-modal module.
Revised: Multi-modal module.

Original: We have experimented with the same multi-modal modules as those in Table 1, but found out we have a better performance and the convergence rate if the Adversarial Player operates on the scene-graphs (states) instead of pixels.
Revised: We have experimented with the same multi-modal modules as those in Table 1, but found out we have a better performance and the convergence rate if the Adversarial Player operates on the scene-graphs (states) instead of pixels.

Original: For that, we use state-input variant of Relation Networks (Santoro et al., 2017).
Revised: For that, we use state-input variant of Relation Networks (Santoro et al., 2017).

Original: The model receives as the input 10 ∗ 6 object tokens, and question tokens.
Revised: The model receives as the input 10 ∗ 6 object tokens, and question tokens.

Original: Every object token represents one-out-of-ten possible objects in the scene by its attributes such as position, color, shape, material and size.
Revised: Every object token represents one-out-of-ten possible objects in the scene by its attributes such as position, color, shape, material and size.

Original: If the scene has fewer than ten objects, we use ∅ token to indicate that, which also acts as a padding.
Revised: If the scene has fewer than ten objects, we use ∅ token to indicate that, which also acts as a padding.

Original: We also have special tokens that separate questions from the objects which we add as a latent embedding, e.g., emb ( material ) + emb ( object ) .
Revised: We also have special tokens that separate questions from the objects which we add as a latent embedding, e.g., emb ( material ) + emb ( object ) .

Original: Such an input encoding is similar to our State-Input Transformer and described in Section A.3.
Revised: Such an input encoding is similar to our State-Input Transformer and described in Section A.3.

Original: The embedded vectors are given to the Relation Network (RN).
Revised: The embedded vectors are given to the Relation Network (RN).

Original: Finally, we train that network on the
Revised: Finally, we train that network on the

Original: CLEVR visual question answering task, where we achieve 97 .
Revised: CLEVR visual question answering task, where we achieve 97 .

Original: 6% on the validation set, and use the representation just after the last relational layer for the decision maker.
Revised: 6% on the validation set, and use the representation just after the last relational layer for the decision maker.

Original: Decision maker.
Revised: Decision maker.

Original: Inspired by the work on reinforcement learning (Mnih et al., 2016; Wu et al., 2017;
Revised: Inspired by the work on reinforcement learning (Mnih et al., 2016; Wu et al., 2017;

Original: Lillicrap et al., 2016; Wang et al., 2016; Schulman et al., 2015), we use an actor-critic module that acts on scenes.
Revised: Lillicrap et al., 2016; Wang et al., 2016; Schulman et al., 2015), we use an actor-critic module that acts on scenes.

Original: The actor is a general-purpose fully connected layer with ten object-speciﬁc heads.
Revised: The actor is a general-purpose fully connected layer with ten object-specific heads.

Original: Each head is randomly assigned to a unique object in the scene for its manipulation.
Revised: Each head is randomly assigned to a unique object in the scene for its manipulation.

Original: Every head produces a displacement in x and y coordinates of the corresponding object.
Revised: Every head produces a displacement in x and y coordinates of the corresponding object.

Original: Although we have initially experimented with the continuous output space we have found out the following simple strategy is more effective.
Revised: Although we have initially experimented with the continuous output space, we have found out the following simple strategy is more effective.

Original: First, we discretise all the x and y coordinates into N bins each.
Revised: First, we discretize all the x and y coordinates into N bins each.

Original: Now, each head produces two N -dimensional vectors that are next projected into a probabilistic space via softmax.
Revised: Now, each head produces two N -dimensional vectors that are next projected into a probabilistic space via softmax.

Original: Next, we sample displacements in x and y axis independently from both softmax distributions.
Revised: Next, we sample displacements in x and y axis independently from both softmax distributions.

Original: Note that, even though we do not model the joint distribution explicitly due to computational reasons, both samples condition on the common head and thus are only conditionally independent of each other.
Revised: Note that, even though we do not model the joint distribution explicitly due to computational reasons, both samples condition on the common head and thus are only conditionally independent of each other.

Original: We discretise the scene where each axis has values in [ − 3 , 3] onto N = 7 bins per axis.
Revised: We discretize the scene where each axis has values in [ − 3 , 3] onto N = 7 bins per axis.

Original: Our critic is a simple three layer feed-forward network (with relu as activations) that predicts a reward score between − 1 and +1 via tanh activation ( 1 . 2 ∗ tanh for better numerical properties).
Revised: Our critic is a simple three layer feed-forward network (with relu as activations) that predicts a reward score between − 1 and +1 via tanh activation ( 1 . 2 ∗ tanh for better numerical properties).

Original: The game of scene manipulations.
Revised: The game of scene manipulations.

Original: Due to our formulation of Adversarial Player and the environment, we can benchmark various reasoning models purely in the black-box setting via a series of questions about the scene.
Revised: Due to our formulation of Adversarial Player and the environment, we can benchmark various reasoning models purely in the black-box setting via a series of questions about the scene.

Original: Adversarial Player manipulates the scene so that it is still consistent with the question-answer pair.
Revised: Adversarial Player manipulates the scene so that it is still consistent with the question-answer pair.

Original: The manipulations are applied to scene-graphs, and the resulting scene-graph is evaluated by the environment enforcers described in Section 4.
Revised: The manipulations are applied to scene-graphs, and the resulting scene-graph is evaluated by the environment enforcers described in Section 4.

Original: Invalid scenes are thus discarded.
Revised: Invalid scenes are thus discarded.

Original: In this way, we ensure the in-distribution and consistency in the scene generation.
Revised: In this way, we ensure the in-distribution and consistency in the scene generation.

Original: Original image-question pairs are fed to a Visual-QA Player that produces corresponding answers.
Revised: Original image-question pairs are fed to a Visual-QA Player that produces corresponding answers.

Original: We refer to that answers as old answers .
Revised: We refer to that answers as old answers .

Original: After the scene manipulation, new images paired with the same questions are also given to the Visual-QA Player that produces new answers .
Revised: After the scene manipulation, new images paired with the same questions are also given to the Visual-QA Player that produces new answers .

Original: We construct rewards based on old answers , new answers and ground-truth answers .
Revised: We construct rewards based on old answers , new answers and ground-truth answers .

Original: If the Adversarial Player forces the Visual-QA Player to change the answer, i.e., an old answer is different than a new answer , it gets Consistency Drop Reward ( cr ).
Revised: If the Adversarial Player forces the Visual-QA Player to change the answer, i.e., an old answer is different than a new answer , it gets Consistency Drop Reward ( cr ).

Original: If an old answer is the groundtruth answer, it gets instead Accuracy Drop Reward ( dr ).
Revised: If an old answer is the groundtruth answer, it gets instead Accuracy Drop Reward ( dr ).

Original: Both rewards differentiate between simply confusing the model and causing a drop in its performance.
Revised: Both rewards differentiate between simply confusing the model and causing a drop in its performance.

Original: If Adversarial Player produces an invalid scene it gets Invalid Scene Reward ( isr ).
Revised: If Adversarial Player produces an invalid scene it gets Invalid Scene Reward ( isr ).

Original: This reward encourages producing scenes that pass the environment enforcers tests.
Revised: This reward encourages producing scenes that pass the environment enforcers tests.

Original: Finally, if Adversarial Player does not manage to fool the model, it gets Fail Reward ( fr ).
Revised: Finally, if Adversarial Player does not manage to fool the model, it gets Fail Reward ( fr ).

Original: We use the following values: dr = 1 , cr = 0 .
Revised: We use the following values: dr = 1 , cr = 0 .

Original: 1 , fr = − 0 .
Revised: 1 , fr = − 0 .

Original: 1 , isr = − 0 .
Revised: 1 , isr = − 0 .

Original: Training algorithm.
Revised: Training algorithm.

Original: To train Adversarial Player we use the A2C algorithm with the episode length set to one as we do not need to model long-range consequences of the decision-making mechanism.
Revised: To train Adversarial Player we use the A2C algorithm with the episode length set to one as we do not need to model long-range consequences of the decision-making mechanism.

Original: Batches contain images, question, answers, programs and scene-graphs.
Revised: Batches contain images, question, answers, programs and scene-graphs.

Original: We train the Adversarial
Revised: We train the Adversarial

Original: Player for each Mini-game independently using the same architecture.
Revised: Player for each Mini-game independently using the same architecture.

Original: We experiment with the following Mini-game sizes 10 , 100 , 1000 .
Revised: We experiment with the following Mini-game sizes 10 , 100 , 1000 .

Original: All Mini-games are constructed randomly.
Revised: All Mini-games are constructed randomly.

Original: Under our discretisation scheme, the action space is N k where N is the number of bins and k is the number of objects in the scene.
Revised: Under our discretization scheme, the action space is N k where N is the number of bins and k is the number of objects in the scene.

Original: In practice, it is up to 49 10 .
Revised: In practice, it is up to 49 10 .

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Main experimental results are presented in Table 2.

Original: First, we explain the terminology and this is followed by analysis of quantitative and qualitative results, additional experiments and limitations.

Original: In our experiments, we consider Adversarial Player with a multi-modal module pre-trained either on states ( state-input ) or directly from pixels ( pixel-input ).
Revised: We consider Adversarial Player with a multi-modal module pre-trained either on states ( state-input ) or directly from pixels ( pixel-input ).

Original: As explained in Section 5, we train

Original:  Adversarial Player for each Mini-game independently.
Revised: We train Adversarial Player for each Mini-game independently.

Original: We use different Mini-game sizes in our experiments: 10 , 100 and 1000 .
Revised: We use different Mini-game sizes in our experiments: 10 , 100 and 1000 .

Original: To obtain statistically signiﬁcant results, we run each Mini-game thirty times with randomly initialized Adversarial Players , each with different seed per trial.
Revised: To obtain statistically significant results, we run each Mini-game thirty times with randomly initialized Adversarial Players , each with different seed per trial.

Original: We report the results that are averaged over all those runs.
Revised: We report the results that are averaged over all those runs.

Original: We compute two metrics for each Mini-game .
Revised: We compute two metrics for each Mini-game .

Original: Consistency refers to the fraction of times Adversarial Player has changed the Visual-QA Player ’s answer, regardless if that answer was correct.
Revised: Consistency refers to the fraction of times Adversarial Player has changed the Visual-QA Player ’s answer, regardless if that answer was correct.

Original: Drop refers to the fraction of times Adversarial Player has changed the correct Visual-QA Player ’s answer.
Revised: Drop refers to the fraction of times Adversarial Player has changed the correct Visual-QA Player ’s answer.

Original: We aggregate the results using the following statistics.
Revised: We aggregate the results using two statistics.

Original: Average Accuracy averages accuracies over all Mini-games .
Revised: Average Accuracy averages accuracies over all Mini-games .

Original: Maximal Accuracy refers to the maximal accuracy over all Mini-games (so it refers to accuracy of a single Mini-game ).
Revised: Maximal Accuracy refers to the worst (best) case of a single Mini-game from the

Original: It is the worst (best) case from the Visual-QA Player  ( Adversarial Player ) perspective.
Revised: Visual-QA Player ’s ( Adversarial Player ’s) perspective.

Original: We also computed p -values of T-Test, with the null hypothesis that a manipulation is unsuccessful.
Revised: We also computed p -values of T-Test, with the null hypothesis that a manipulation is unsuccessful.

Original: In almost all cases we reject the null hypothesis with small p -values.
Revised: In almost all cases we reject the null hypothesis with small p -values.

Original: We only did not ‘fool’ TbD and MDetr, and only if pixel-input Adversarial Player is considered (Section A.11, in the appendix).
Revised: The p -values are available in the appendix (Section A.11).

Original: Quantitative results.
Revised: Quantitative results.

Original: We show our results in Table 2 for the state-input and pixel-input Adversarial Players , and for each Visual-QA Player presented in Table 1.
Revised: We show our results in Table 2 for the state-input and pixel-input Adversarial Players , and for each Visual-QA Player presented in Table 1.

Original: All the Visual-QA Players in the comparison are trained from pixels, apart form our custom State-Input Transformer architecture (Section A.3 in the appendix).
Revised: All the Visual-QA Players in the comparison are trained from pixels, apart form our custom State-Input Transformer architecture (Section A.3 in the appendix).

Original: When comparing both Adversarial Players , we can observe that the state-input Adversarial Player is signiﬁcantly more successful than the pixel-input one.
Revised: When comparing both Adversarial Players , we can observe that the state-input Adversarial Player is significantly more successful than the pixel-input one.

Original: We hypothesize that this is the effect of having more structured and less ambiguous input information.
Revised: We hypothesize that this is the effect of having more structured and less ambiguous input information.

Original: It could also be the case that multi-modal modules trained from pixels ignore features in the same way and hence smaller discrepancy between all models operating on pixels.
Revised: It could also be the case that multi-modal modules trained from pixels ignore features in the same way and hence smaller discrepancy between all models operating on pixels.

Original: Another observation is that in the case of TbD and MDetr, pixel-input Adversarial Players are unable to solve the smallest-size

Original: Mini-games . Moreover, as the size of Mini-games increases, it is becoming more difﬁcult for the Adversarial Player to successfully manipulate the model on all examples from the Mini-game .
Revised: Moreover, as the size of Minigames increases, it is becoming more difficult for the Adversarial Player to successfully manipulate the model on all examples from the Mini-game .

Original: It is an expected behavior resembling standard adversarial perturbations (Goodfellow et al., 2015) and their universal variants (Moosavi-Dezfooli et al., 2017).
Revised: This is an expected behavior resembling standard adversarial perturbations (Goodfellow et al., 2015) and their universal variants (Moosavi-Dezfooli et al., 2017).

Original: The highest performing CLEVR models, TbD and MDetr, are the most robust under the manipulation.
Revised: The highest performing CLEVR models, TbD and MDetr, are the most robust under the manipulation.

Original: However, they are still susceptible to the scene manipulations in some Mini-games as the Maximal Consistency metric indicates, especially if our Adversarial Player uses states as the input.
Revised: However, they are still susceptible to the scene manipulations in some Mini-games as the Maximal Consistency metric indicates, especially if our Adversarial Player uses states as the input.

Original: Section A.8 (appendix) also shows richer performance statistics.
Revised: Section A.8 (appendix) also shows richer performance statistics.

Original: Our detailed investigation shows that the models above are particularly sensitive to manipulations of scenes associated with counting or existence questions.
Revised: Our detailed investigation shows that the models above are particularly sensitive to manipulations of scenes associated with counting or existence questions.

Original: Questions of those types are often more complex, hence, increasing the likelihood of the reasoning failure at some stage (Section A.6 in the appendix).
Revised: Such question are often more complex, hence, increasing the likelihood of the reasoning failure at some stage (Section A.6 in the appendix).

Original: Qualitative results.
Revised: Qualitative results.

Original: Figure 4 shows qualitative results of the scene manipulations for the three highest performing and distinctive models from Table 1; e.g., simple feed-forward or program induction models.
Revised: Figure 3 shows qualitative results of the scene manipulations for the three highest performing and distinctive models from Table 1; e.g., simple feed-forward or program induction models.

Original: For the reader’s convenience, we provide in Section A.2 (appendix) examples of CLEVR object attributes.
Revised: For the reader’s convenience, we provide in Section A.2 (appendix) examples of CLEVR object attributes.

Original: We can observe that the scene manipulations are surprisingly minimal and are semantically meaningless , i.e., they should not ‘fool’ human players.
Revised: We can observe that the scene manipulations are surprisingly minimal and are semantically meaningless , i.e., they should not ‘fool’ human players.

Original: Saturation of Adversarial Player .
Revised: Reasoning steps and visual cues. To better understand the source of errors (reasoning steps vs visual cues), we have conducted the following experiments. In the first experiment, shown in Section B.4 (appendix), we visualize the attention mechanisms of Visual-QA Players before and after the scene manipulations. We observe changes in the attention map of all models but MDetr.

Original: Training an Adversarial Player on larger Mini-games become an increasingly more difﬁcult task as the player needs to operate successfully on more diverse examples.
Revised: The results suggest, most models’s perception is impacted by the manipulations. However, all objects are still correctly identified by MDetr, thus suggesting some issues are also stemming from the reasoning steps.

Original: Figure 2 shows the performance of the Adversarial Player against Mini-game sizes, where FiLM, TbD, MDetr and Input-State Transformer (Trans.) are chosen as Visual-QA Players .
Revised: In the second experiment, we deployed Slot-attention object detector (Locatello, 2020), and show that the detector identifies the same properties before and after manipulations.

Original: We can see thatthe performance of Adversarial Player saturates around Mini-games ofsize 10k, which we call the Critical mini-game size .
Revised: This result (Section B.6; appendix) shows that Adversarial Player does not produce ‘corner cases’ of the [Table] perceptual system.

Original: We use two trials for our experiments, and the Accuracy Drop as a metric.
Revised: We also have similar conclusions based on our small-scale human experiment (Section B.5; appendix)

Original: It measures the drop in accuracy of a Visual-QA Player .
Revised: [Table]

Original: Adversarial training.
Revised: Adversarial training.

Original: In initial experiments, we have adapted a widely used adversarial training  (Goodfellow et al., 2018), where we include manipulated scenes in the training protocol of a Visual-QA Player .
Revised: We have adapted a widely used adversarial training schema (Goodfellow et al., 2018) [Table] and included manipulated scenes in the training protocol of a Visual-QA Player .

Original: We use pixel-input RN (Santoro et al., 2017) as the Visual-QA Player .
Revised: We use pixel-input RN (Santoro et al., 2017) as the Visual-QA Player .

Original: Using this method, we have obtained only marginal improvements in robustness at the cost of a slight performance degradation on the original CLEVR dataset.
Revised: Using this method, we have obtained only marginal improvements in robustness at the cost of a slight performance degradation on the original CLEVR dataset.

Original: Limitations of data-driven CLEVR models.
Revised: Limitations of data-driven reasoning models.

Original: If we can’t achieve robustness under scene manipulations with the adversarial training, what fraction of all data is sufﬁcient then?
Revised: Due to the data-driven nature of our Visual-QA Players , we pose the following question, How many examples of Adversarial Player manipulations would be enough for a visual reasoning model to train on, in order for it to be robust against any unseen ones?

Original: Intuitively, if the network has enough capacity, and was trained on all possible data points or its manipulations, it should rely on a look-up strategy to solve the problem accurately.
Revised: Intuitively, if the network has enough capacity, and was trained on all possible data points (or its manipulations), it could rely on a look-up strategy to solve the problem accurately.

Original: We propose the following experiment to address such a question.
Revised: We propose the following experiment to address such a question.

Original: We create a dataset containing three objects in six different starting positions and seven templated questions.
Revised: We created a series of datasets as follows.

Original: We consider only object displacements on the grid 7 × 7  ,and thus 49 3 = 117649 combinations are possible.
Revised: We treat a scene as a discrete 7 × 7 grid (manipulations are restricted only to this grid). Thus each dataset contains 49 2 = 2401 , 49 3 = 117649 , 49 4 = 5764801 scene combinations, covering all possible scene manipulations with two, three, and four objects respectively. In the case of four objects, due to computational reasons, we keep a single object stationary (thus it also has 49scene manipulations). The set of questions associated with each image dataset requires the model to perform either one reasoning step (Onehop), two reasoning steps (Twohop) or are a mixture of both (Mixhop).

Original: We use a few architectures from Table 1 asa pixel-input Visual-QA Player .
Revised: We use RN, FiLM and TbD as Visual-QA Players .

Original: We train them on X % data and next evaluate on (100 − X )% remaining, unseen data.
Revised: We train them on X % data and next evaluate on (100 − X )% remaining, unseen data.

Original: We have conducted ten trials by forming X % training data randomly.
Revised: We have conducted ten trials by forming X % training data randomly.

Original: Figureshows that when trained on 10% of data, the model becomes resistant to 40% of unseen manipulations.

Original: The robustness of the model increases linearly, saturating at roughly 97% accuracy after observing almost all data ( 90% ). The behaviour of all tested models is similar with slightly better overall performance of TbD. The results indicate that a lot of training examplesare neededto cover most conﬁgurations of our task in the absence of stronger inductive biases.
Revised: We notice that the behavior of all tested models is similar so we report a joint average. Results are presented in Figure 2. As the number of objects and diversity of question increases, so does the amount of training examples needed in order to achieve robustness.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: We present a novel approach to benchmarking the reasoning capabilities ofvarious visual models by formulating a game between two players.
Revised: We present a novel approach to isolate and benchmark the reasoning capabilities of visual models by formulating a game between two players.

Original: Both players are independent and can only communicate through a common interface: an environment and question-answer pairs.
Revised: Both players are independent and can only communicate through a common interface, an environment and question-answer pairs.

Original: One player, named Adversarial Player , is trying to manipulate the environment so that the second player, named Visual-QA
Revised: One player, named Adversarial Player , is trying to manipulate the environment so that the second player, named Visual-QA

Original: Player , can’t answer the given question anymore.
Revised: Player , can’t answer the given question anymore.

Original: To test our framework, we run our game against a few popularCLEVR models that achieve solid results onthe CLEVRdataset; often surpassing the human baseline. We show that such models are susceptible to scene manipulations, and their performance may degrade signiﬁcantly.
Revised: We show that popular models with solid results on CLEVR are susceptible to scene manipulations, and their performance may degrade significantly.

Original: Moreover, we also conducted a controlled study on the generalization capabilities of such models.
Revised: Moreover, we also conducted a controlled study on the generalization capabilities of such models.

Original: We did so by creatinga synthetic and simpliﬁed dataset of all possible scene manipulations.
Revised: We did so by creating synthetic and simplified datasets of all possible scene manipulations.

Original: We show that the existing models are not particularly efﬁcient learners and require signiﬁcantly more than 50% of all data to generalize to all possible scene conﬁgurations.
Revised: We show that the existing models are susceptible to reasoning gaps and require extra data, proportional to their task complexity in order to generalize to all possible configurations.

Original: One possible direction to increase efﬁciency could be the use of stronger inductive biases.
Revised: One possible direction to increase efficiency could be the use of stronger inductive biases.

Original: Another to change a training paradigm that is more interactive.
Revised: Another to change the training paradigm into a more interactive one.

Original: However, the environment is synthetic, and it could be that learning from real-world and multi-modal data is somewhat more efﬁcient.
Revised: However, the environment is synthetic, and it could be that learning from real-world and multi-modal data is somewhat more efficient.

Original: Finally, we believe that melding two-player games with the visual question answering framework is both natural and beneﬁcial for testing the reasoning capabilitiesof various models; and potentially can also be generalized to other settings. 
Revised: Finally, we believe that melding two-player games with the visual question answering framework is both natural and beneficial for testing the reasoning capabilities and potentially can also be generalized to other settings. A CKNOWLEDGMENTS

Original: In the long term, we plan to test such models through web interfaces.
Revised: All experiments were performed using the Entropy cluster funded by NVIDIA, Intel, the Polish

Original: [Table]
Revised: National Science Center grant UMO-2017/26/E/ST6/00622 and ERC Starting Grant TOTAL. The work of Spyridon Mouselinos and Henryk Michalewski was supported by the Polish National Science Center grant UMO-2018/

Original: What is the color
Revised: What is the color

Original: How many other objects
Revised: How many other objects

Original: There is a tiny block How many other things are of the block that is are the same shape as thebehind the purple cylinder; the same shape as the big left of the big biggray matte object? what number of red matte metallic thing?
Revised: How many other things are of the block that is are the same shape as the the same shape as the big left of the big big

Original: blue matte ball?
Revised: [Figure] object?

Original: spheres are in front of it? F i L M - B e f o r e
Revised: [Figure] thing?

Original: Purple
Revised: blue

Original: F i L M - A f t e r
Revised: [Figure]

Original: Brown
Revised: [Figure]

Original: What number of tiny
Revised: What number of tiny

Original: Are there any other things
Revised: Are there any other things

Original: Isthe shape of the brown rubber Is there a bigbrown object matte cylinders that have the same material as object the same as the thing that of the same shape as the are there? the big yellow ball? is left of the big object?
Revised: Is there a big

Original: green thing? T b D - B e f
Revised: [Figure]

Original: o r e
Revised: object

Original: Yes Yes Yes
Revised: [Figure]

Original: T b D - A f t e r
Revised: cylinders

Original: No No No
Revised: [Figure]

Original: The red thing that is Is there anything else Do the object behind the 
Revised: have the same material as of the same shape [Figure]

Original: What number of things are large the same material as the that has the same material as big matte sphere and the metallic blocks on the brown cube is what size?
Revised: the [Figure] there? [Figure]

Original: the large green cube?
Revised: What number of things are large [Figure] same [Figure]

Original: thing on the right side of left side of the brown metallic the red thing have the same material?
Revised: the [Figure] the same material as [Figure]

Revised: blocks on the

Revised: [Figure]

Original: thing or big cyan shiny objects?
Revised: cube [Figure] what size? the large

Original: M D e t r - B e f o r e
Revised: [Figure] cube? [Figure] of the

Original: Large
Revised: [Figure] thing

Original: No Yes
Revised: [Figure]

Original: M D e t r - A f t e r Small
Revised: big cyan shiny objects?

Original: Yes No
Revised: [Figure]

Original: A A PPENDIX A. URL S OF CLEVR M ODELS
Revised: A A PPENDIX A. URL S OF CLEVR M ODELS

Original: Table 3 shows the URLs to models used in our investigations (also Table 1 in the main paper).
Revised: Table 3 shows the URLs to models used in our investigations (also Table 1 in the main paper).

Original: We also report if we re-trained a model from scratch (type Architecture ) or used already trained models (type Model ).
Revised: We also report if we re-trained a model from scratch (type Architecture ) or used already trained models (type Model ).

Original: Please note that the latter type proves that our testing procedure is fully black-box.
Revised: Please note that the latter type proves that our testing procedure is fully black-box.

Original: [Table]
Revised: [Table]

Original: A. O BJECT C ATEGORIES
Revised: A. O BJECT C ATEGORIES

Original: Figure 5 shows a made-up image that contains all available CLEVR object categories: shapes, sizes and materials.
Revised: Figure 4 shows a made-up image that contains all available CLEVR object categories: shapes, sizes, and materials.

Original: Small objects are rendered at approximately 40% of the size of their large counterparts.
Revised: Small objects are rendered at approximately 40% of the size of their large counterparts.

Original: Metalic materials can be identiﬁed by the high albedo property and ‘shininess’ of the object.
Revised: Metallic materials can be identified by the high albedo property and ‘shininess’ of the object.

Original: We provide it as the reference point for a reader to make it easier to compare our qualitative results.
Revised: We provide it as the reference point for a reader to make it easier to compare our qualitative results.

Original: A. S TATE -I NPUT T RANSFORMER
Revised: A. S TATE -I NPUT T RANSFORMER

Original: To test our two-player game against a state-input Visual-QA Player , we have designed a transformerbased architecture Vaswani et al. (2017) that receives six types of input features – object sizes, object shapes, object materials, object colors, object positions, and question tokens – and uses a cross-modal attention mechanism.
Revised: To test our two-player game against a state-input Visual-QA Player , we have designed a transformerbased architecture (Vaswani et al., 2017) that receives six types of input features – object sizes, object shapes, object materials, object colors, object positions, and question tokens – and uses a cross-modal attention mechanism.

Original: In that mechanism, queries from one modality attend to keys and values from the another modality.
Revised: In that mechanism, queries from one modality attend to keys and values from the other modality.

Original: Since each data point containsa variable-length inputs describingthe all objects in the scene as well as question tokens that compose the question itself, we do padding with a special token ∅ to the maximal input length.
Revised: Each data point contains variable-length inputs, describing all objects in the scene. It also contains question tokens that compose the question itself. Hence, we do padding with a special token ∅ to the maximal input length.

Original: We set the maximal length for the objects tokens to be 10 ∗ 6 = 60 and 50 for the question tokens.
Revised: We set the maximal length for the objects tokens to be 10 ∗ 6 = 60 and 50 for the question tokens.

Original: All in all, we have 110 input tokens.
Revised: All in all, we have 110 input tokens.

Original: Each token type is projected into a different embedding space and a learnable type embedding is added to it, separately for object and question tokens.
Revised: Each token type is projected into a different embedding space and a learnable type embedding is added to it, separately for object and question tokens.

Original: For instance, emb ( material )+ emb ( object ) .
Revised: For instance, emb ( material ) + emb ( object ) .

Original: Furthermore, three special learnable embeddings are used as an additional input.
Revised: Furthermore, three special learnable embeddings are used as an additional input.

Original: We use them as queries to reduce the overall computational costs of the cross-attention mechanism.
Revised: We use them as queries to reduce the overall computational costs of the cross-attention mechanism.

Original: The same mechanism is also used in Perceiver (Jaegle et al., 2021).
Revised: The same mechanism is also used in Perceiver (Jaegle et al., 2021).

Original: The input tokens (concatenated object and question tokens) form keys and values.
Revised: The input tokens (concatenated object and question tokens) form keys and values.

Original: In every transformer block, we apply cross-attention between all input embeddings and those three special tokens.
Revised: In every transformer block, we apply cross-attention between all input embeddings and those three special tokens.

Original: This is repeated ﬁve times.
Revised: This is repeated five times.

Original: As the ﬁnal block, a feed-forward network (classiﬁer) receives as inputs the three latent token and outputs answer probabilities.
Revised: As the final block, a feed-forward network (classifier) receives as inputs the three latent tokens and outputs answer probabilities.

Original: We show that architecture in Figure 6.
Revised: We show that architecture in Figure 5.

Original: A. M ANIPULATIONS OF State-Input Visual-QA Player
Revised: A. M ANIPULATIONS OF State-Input Visual-QA Player

Original: Figures 7 and 8 show manipulations of Visual-QA Player that is trained to take states instead of pixels as the input.
Revised: Figures 6 and 7 show manipulations of Visual-QA Player that is trained to take states instead of pixels as the input.

Original: Such a state-input Visual-QA Player receives a direct overview of the scene, bypassing any need for any image renderings.
Revised: Such a state-input Visual-QA Player receives a direct overview of the scene, bypassing any need for any image renderings.

Original: We use State-Input Transformer (Section A.3) as the multi-modal component of that player.
Revised: We use State-Input Transformer (Section A.3) as the multi-modal component of that player.

Original: Since such a model gets the perfect visual information as the input, it makes it more robust under scene manipulations.
Revised: Since such a model gets the perfect visual information as the input, it makes it more robust under scene manipulations.

Original: As a consequence, our Adversarial Player tend to manipulate the scene so that objects are placed closely together.
Revised: As a consequence, our Adversarial Player tend to manipulate the scene so that objects are placed closely together.

Original: Often, simpler questions, which require fewer stages of reasoning, are safer to Visual-QA Player .
Revised: Simple questions, which require few stages of reasoning, are usually less effective against Visual-QA Players .

Original: Let us consider the last example  in Figure 8.
Revised: Let us consider the last example (rows 3 and 4) in Figure 7.

Original: If the question was What is the material of the cyan cube? , the model would answer rightfully metal .
Revised: If the question was What is the material of the cyan cube? , the Visual-QA Player would answer rightfully metal .

Revised: That question required no reasoning steps, but rather the classification of an object’s attribute.

Revised: However, the original question of the example, What is the material of the cube that is behind the rubber cube?

Revised: , requires one reasoning hop.

Revised: The Visual-QA Player needs to locate the rubber cube first, and then identify the material of another cube behind it.

Revised: More reasoning steps allow for more ways of error in the Visual-QA

Original: Here, Adversarial Player ‘breaks’ the spatial reasoning step by creating ambiguity between the cubes and thus a wrong answer is derived. Are there an equal number of small things that are left of the gray object and large cyan matte balls?
Revised: Player ’s decision process. In this example, the Adversarial Player ‘breaks’ the spatial reasoning step by creating ambiguity between the cubes, and thus a wrong answer is derived. [Figure]

Original: S t a t e - B e f o r e
Revised: S t a t eB e f o r e

Original: False
Revised: [Figure]

Original: S t a t e - A f t e r
Revised: a t eA f t e r

Original: True Is the number of big blocks that are in front of the brown metallic cylinder the same as the number of tiny cylinders on the right side of the rubber cube?
Revised: [Figure]

Original: S t a t e - B e f o r e
Revised: S t a t eB e f o r e

Original: False
Revised: False

Original: S t a t e - A f t e r
Revised: S t a t eA f t e r

Original: True
Revised: True

Original: How many other things are both behind the purple rubber sphere and to the right of the gray cylinder?
Revised: How many other things are both behind the purple rubber sphere and to the right of the gray cylinder?

Original: S t a t e - B e f o r e

Original: S t a t e - A f t e r What is the material of the cube that is behind the rubber cube? S t a t e - B e f o r e Metal
Revised: [Figure]

Original: S t a t e - A f t e r

Original: Rubber

Original: A. A LGORITHMS
Revised: A. A LGORITHMS

Original: We show pseudo-algorithms that we use to (Algorithm 1) calculate rewards, (Algorithm 2) train
Revised: We show pseudo-algorithms that we use to (Algorithm 1) calculate rewards, (Algorithm 2) train

Original: Adversarial Player , (Algorithm 3)  play a game, (Algorithm ?? )
Revised: Adversarial Player , (Algorithm 3) and play a game.

Original: create a defence mechanism.

Original:  dr  Drop Reward 5: cr : Consistency Reward 6: fr : Fail Reward 7: reward ←8: if new answer (cid:54) = old answer then 9: if old answer (cid:54) = gt answer then 10: reward = reward + cr 11: else 12: reward = reward + dr 13: end if 14: else 15: reward = reward + fr 16: end if 17: return reward
Revised: 3: gt answer : Ground Truth Answer 4: dr : Drop Reward 5: cr : Consistency Reward 6: fr : Fail Reward 7: reward ←8: if new answer ̸ = old answer then 9: if old answer ̸ = gt answer then 10: reward = reward + cr 11: else 12: reward = reward + dr 13: end if 14: else 15: reward = reward + fr 16: end if 17: return reward

Original: 1: M : Minigames 2: V a : Visual Agent, Fe : Feature Extractor 3: A : Actor, C : Critic 4:
Revised: 1: M : Minigames 2: V a : Visual Agent, Fe : Feature Extractor 3: A : Actor, C : Critic 4:

Original: Qre : Question Relevance Enforcer, Sce : Scene Constraint Enforcer 5: for batch in M do 6: rewards, state values, logprobs ← DataGame ( M, V a, Fe, A, C, Qre, Sce ) 7: n ← | rewards | 8: advantage ← rewards − state values 9: ploss ← − logprobs × advantage 10: vloss ← ( state values − stop grad ( rewards )) 2 /n 11: loss ← ploss + vloss 12: backprop ( loss, A, C ) 13: end for 14: return
Revised: Qre : Question Relevance Enforcer, Sce : Scene Constraint Enforcer 5: for batch in M do 6: rewards, state values, logprobs ← DataGame ( M, V a, Fe, A, C, Qre, Sce ) 7: n ← | rewards | 8: advantage ← rewards − state values 9: ploss ← − logprobs × advantage 10: vloss ← ( state values − stop grad ( rewards )) 2 /n 11: loss ← ploss + vloss 12: backprop ( loss, A, C ) 13: end for 14: return

Original: 1: M : Mini-game 2: P VQA : Visual Agent, S k : Multi-modal Backbone 3: A : Actor, C : Critic 4:
Revised: 1: M : Mini-game 2: P VQA : Visual Agent, S k : Multi-modal Backbone 3: A : Actor, C : Critic 4:

Original: Qre : Question Relevance Enforcer, Sce : Scene Constraint Enforcer 5: isr : Invalid Scene Reward 6: for batch in M do 7: rewards ← {} 8: state values ← {} 9: logprobs ← {} 10: for ( image, scene, question, program, gt answer ) in batch do 11: old answer ← P VQA ( image, question ) 12: extracted features ← S k
Revised: Qre : Question Relevance Enforcer, Sce : Scene Constraint Enforcer 5: isr : Invalid Scene Reward 6: for batch in M do 7: rewards ← {} 8: state values ← {} 9: logprobs ← {} 10: for ( image, scene, question, program, gt answer ) in batch do 11: old answer ← P VQA ( image, question ) 12: extracted features ← S k

Original: ( scene, question ) 13: dx probs, dy probs ← A ( extracted features ) 14:
Revised: ( scene, question ) 13: dx probs, dy probs ← A ( extracted features ) 14:

Original: CatX ← Categorical ( dx probs ) 15:
Revised: CatX ← Categorical ( dx probs ) 15:

Original: CatY ← Categorical ( dy probs ) 16: state value ← C ( extracted features ) 17: dx ← sample
Revised: CatY ← Categorical ( dy probs ) 16: state value ← C ( extracted features ) 17: dx ← sample

Original: CatX 18: dy ← sample CatY 19: new scene ← Perturbate ( scene, dx, dy ) 20: check scene ← Sce ( new scene ) 21: check question ← Qre ( new scene, program, gt answer ) 22: if check scene ∧ check question =
Revised: CatX 18: dy ← sample CatY 19: new scene ← Perturbate ( scene, dx, dy ) 20: check scene ← Sce ( new scene ) 21: check question ← Qre ( new scene, program, gt answer ) 22: if check scene ∧ check question =

Original: True then 23: new image ← Render ( new scene ) 24: new answer ← P VQA ( new image, question ) 25: rewards ← rewards || CalcRewards ( new answer, old answer, gt answer ) 26: else 27: rewards ← rewards || isr 28: end if 29: state values ← state values
Revised: True then 23: new image ← Render ( new scene ) 24: new answer ← P VQA ( new image, question ) 25: rewards ← rewards || CalcRewards ( new answer, old answer, gt answer ) 26: else 27: rewards ← rewards || isr 28: end if 29: state values ← state values

Original: || state value 30: logprob ← logprob || log ( p ( dx | CatX )))
Revised: || state value 30: logprob ← logprob || log ( p ( dx | CatX )))

Original: + log ( p ( dy | CatY ))) 31: end for 32: end for 33: return rewards, state values, logprobs
Revised: + log ( p ( dy | CatY ))) 31: end for 32: end for 33: return rewards, state values, logprobs

Original: A. S ENSITIVITY TO Q UESTION T YPES
Revised: A. S ENSITIVITY TO Q UESTION T YPES

Original: Figure 9 shows the susceptibility of CLEVR models to scene manipulations of Adversarial Player .
Revised: Figure 8 shows the susceptibility of CLEVR models to scene manipulations of Adversarial Player .

Original: We observe that especially counting and existence questions are the ‘back-doors‘ for our scene manipulations.
Revised: We observe that especially counting and existence questions are the ‘back-doors‘ for our scene manipulations.

Original: Questions of those two types typically involves multiple steps of reasoning.
Revised: Questions of those two types typically involve multiple steps of reasoning.

Original: Adversarial
Revised: Adversarial

Original: Player ’s scene manipulations make them more likely that reasoning will fail at some stage.
Revised: Player ’s scene manipulations make them more likely that reasoning will fail at some stage.

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: A. C ONVERGENCE

Original: Figure 10 shows convergence plots of Adversarial Player during training against FiLM, TbD and
Revised: Figure 9 shows convergence plots of Adversarial Player during training against FiLM, TbD and

Original: MDetr.
Revised: MDetr.

Original: We use different initialization seeds and 30 trials.
Revised: We use different initialization seeds and 30 trials.

Original: Adversarial Player is trained either from states or pixels .
Revised: Adversarial Player is trained either from states or pixels .

Original: The former yields higher performance and sometimes better convergence.
Revised: The former yields higher performance and sometimes better convergence.

Original: We use the Drop metric that measures the accuracy drop after the Visual-QA Player is manipulated.
Revised: We use the Drop metric that measures the accuracy drop after the Visual-QA Player is manipulated.

Original: The higher Drop , the more successful manipulations are.
Revised: The higher Drop , the more successful manipulations are.

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: Figures 11 and 12 show the distribution of the Adversarial Player performance, where we show how often (y-axis)
Revised: Figures 10 and 11 show the distribution of the Adversarial Player performance, where we show how often (y-axis)

Original: the given Drop score is achieved (x-axis).
Revised: the given Drop score is achieved (x-axis).

Original: The Drop metric measures the accuracy drop after the Visual-QA Player is manipulated.
Revised: The Drop metric measures the accuracy drop after the Visual-QA Player is manipulated.

Original: The higher Drop , the more successful manipulations are.
Revised: The higher Drop , the more successful manipulations are.

Original: Larger Mini-games lead to more consistent (narrower distributions) but lower performance.
Revised: Larger Mini-games lead to more consistent (narrower distributions) but lower performance.

Original: This behaviour suggests that scene diversity present in larger Mini-games lead to more stable scene manipulations.
Revised: This behavior suggests that scene diversity present in larger Mini-games leads to more stable scene manipulations.

Original: However, they do so at the cost of increasing the complexity of the optimization problem.
Revised: However, they do so at the cost of increasing the complexity of the optimization problem.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original:  t r
Revised: M D e t r

Original: A. O UT - OF -D ISTRIBUTION M ANIPULATIONS
Revised: A. O UT - OF -D ISTRIBUTION M ANIPULATIONS

Original: If we do not provide extra constraints on the scene generation process with our environment enforcers, Adversarial Player may ﬁnd an easy manipulation that changes the camera pose or location, effectively ‘zooming out’ the whole scene.
Revised: If we do not provide extra constraints on the scene generation process with our environment enforcers, Adversarial Player may find an easy manipulation that changes the camera pose or location, effectively ‘zooming out’ the whole scene.

Original: It does so by stretching all the object coordinates in the scene.
Revised: It does so by stretching all the object coordinates in the scene.

Original: Figure 13 illustrates that.
Revised: Figure 12 illustrates that.

Original: For instance, large objects look much smaller than typical examples.
Revised: For instance, large objects look much smaller than typical examples.

Original: As such constructed scenes are quite different from the ones that Visual-QA Player has observed during training, we categorize such manipulations to be out-of-distribution .
Revised: As such constructed scenes are quite different from the ones that Visual-QA Player has observed during training, we categorize such manipulations to be out-of-distribution .

Original: Note that, due to our in-distribution environment enforcers, such scenes are prohibited in our pipeline.
Revised: Note that, due to our in-distribution environment enforcers, such scenes are prohibited in our pipeline.

Original: [Figure]
Revised: [Figure]

Original: A.
Revised: A.

Original: A DDITIONAL Q UALITATIVE R ESULTS
Revised: A DDITIONAL Q UALITATIVE R ESULTS

Original: In Figures 14-23, we provide more qualitative results.
Revised: In Figures 13-21, we provide more qualitative results.

Original: Is the number of small cubes that
Revised: Is the number of small cubes that

Original: What is the color of the metallic thing are right of the blue rubber block greater that is on the right side of than the number of small balls that the shiny thing behind are in front of the cyan object?
Revised: What is the color of the metallic thing are right of the blue rubber block greater that is on the right side of than the number of small balls that the shiny thing behind are in front of the cyan object?

Original: the large gray object?
Revised: the large gray object?

Original: S
Revised: S

Original: ANB e f o r e
Revised: ANB e f o r e

Original: Yes
Revised: Yes

Original: Gray
Revised: Gray

Original: S A NA f t e r
Revised: [Figure] N [Figure]

Original: No
Revised: No

Original: Green
Revised: Green

Original: There is a cube on the leftWhat
Revised: There is a cube on the left

Original: shape

Original: is

Original:  the thing that side of the brown object is in front of the small yellow that is behind the red ball; cylinder and to the right what size is it?
Revised: What shape is the thing that side of the brown object is in front of the small yellow that is behind the red ball; cylinder and to the right what size is it?

Original: of the rubber thing?
Revised: of the rubber thing?

Original: S
Revised: S

Original: ANB e f o r e
Revised: ANB e f o r e

Original: Large
Revised: Large

Original: Cylinder
Revised: Cylinder

Original: S
Revised: S

Original: ANA f t e r
Revised: ANA f t e r

Original: Small
Revised: Small

Original: Sphere
Revised: Sphere

Original: How big is the shiny cylinder There is a thing that is the that is behind the same color as the cube; rubber cylinder?
Revised: How big is the shiny cylinder There is a thing that is the that is behind the same color as the cube; rubber cylinder?

Original: what is it made of?
Revised: what is it made of?

Original: I E
Revised: I E

Original: PB e f o r e
Revised: PB e f o r e

Original: Large
Revised: Large

Original: Rubber
Revised: Rubber

Original: I E
Revised: I E

Original: PA f t e r
Revised: PA f t e r

Original: Small
Revised: Small

Original: Metal
Revised: Metal

Original: Does the cyan metal object that
Revised: Does the cyan metal object that

Original: What is the big thing that is behind is behind the tiny gray block have the large red shiny sphere the same size as the block behind made of?
Revised: What is the big thing that is behind is behind the tiny gray block have the large red shiny sphere the same size as the block behind made of?

Original: the small cyan metallic objec?
Revised: the small cyan metallic objec?

Original: I E
Revised: I E

Original: PB e f o r e
Revised: PB e f o r e

Original: No
Revised: No

Original: Rubber
Revised: Rubber

Original: I E
Revised: I E

Original: PA f t e r
Revised: PA f t e r

Original: Yes
Revised: Yes

Original: Metal
Revised: Metal

Original: There is a metallic object; How many other small shiny objects does it have the same size as are the same shape as the the rubber block right of the red object?
Revised: There is a metallic object; How many other small shiny objects does it have the same size as are the same shape as the the rubber block right of the red object?

Original: tiny shiny cube?
Revised: tiny shiny cube?

Original: R
Revised: R

Original: NB e f o r e
Revised: NB e f o r e

Original: No
Revised: No

Original: R
Revised: [Figure]

Original: NA f t e r
Revised: N [Figure]

Original: Yes
Revised: Yes

Original: Are there any other things Are there any other things that are the same shape as the that are the same size big metallic object?
Revised: Are there any other things Are there any other things that are the same shape as the that are the same size big metallic object?

Original: as the ball?
Revised: as the ball?

Original: R
Revised: R

Original: NB e f o r eNo
Revised: NB e f o r e

Original: [Figure]
Revised: [Figure]

Original: R
Revised: R

Original: NA f t e r
Revised: NA f t e r

Original: [Figure]
Revised: [Figure]

Original: How many objects are there
Revised: How many objects are there

Original: There is another object that is the of the same size as same size as the blue the brown object?
Revised: There is another object that is the of the same size as same size as the blue the brown object?

Original: rubber object; What is it made of?
Revised: rubber object What is it made of?

Original: M
Revised: M

Original: D e t rB e f o r e
Revised: D e t rB e f o r e

Original: Rubber
Revised: Rubber

Original: M
Revised: M

Original: D e t rA f t e r
Revised: D e t rA f t e r

Original: Metal
Revised: Metal

Original: What number of other objects What number of metal objects are the same color are small objects or as the block ?
Revised: What number of other objects What number of metal objects are the same color are small objects or as the block ?

Original: big cyan objects?
Revised: big cyan objects?

Original: M
Revised: M

Original: D e t rB e f o r e
Revised: D e t rB e f o r e

Original: M
Revised: M

Original: D e t rA f t e r
Revised: D e t rA f t e r

Original: Is there an equal number of cubes
Revised: Is there an equal number of cubes

Original: Are there less matte objects in front of the large shiny thing on the left of the green block behind the cyan block than metal things?
Revised: Are there less matte objects in front of the large shiny thing on the left of the green block behind the cyan block than metal things?

Original: and behind the yellow cube?
Revised: and behind the yellow cube?

Original: T b
Revised: T b

Original: DB e f o r e
Revised: DB e f o r e

Original: No
Revised: No

Original: Yes
Revised: Yes

Original: T b
Revised: T b

Original: DA f t e rYes No
Revised: DA f t e r

Original: There is an object on the left Are there an equal number of tiny matte blocks of the yellow sphere in front of that are on the right side of the cyan metal thing; the matte cylinder and tiny What shape is it?

Original: cyan rubber cylinders?

Original: T b

Original: DB e f o r e

Original: Cube

Original: Yes
Revised: Yes

Original: T b

Original: DA f t e r

Original: Sphere

Original: No
Revised: No

Original: A. Q UANTITATIVE R ESULTS WITH p - VALUES
Revised: A. Q UANTITATIVE R ESULTS WITH P - VALUES

Original: We extend the quantitative results of the main paper by also reporting p -values of the one-sample hypothesis testing.
Revised: We extend the quantitative results of the main paper by also reporting p -values of the one-sample hypothesis testing.

Original: Here, we also use the Accuracy Drop metric that measures the accuracy drop after the Visual-QA Player is manipulated, and the Consistency Drop that measures how many times the manipulated Visual-QA Player changes its answer, independently if that is a correct or wrong answer.
Revised: We use the Accuracy Drop metric that measures the accuracy drop after the Visual-QA Player is manipulated, and the Consistency Drop that measures how many times the manipulated Visual-QA Player changes its answer, independently if that is a correct or wrong answer.

Original: The higher Drop Accuracy or Consistency Drop , the more successful manipulations are.
Revised: The higher Drop Accuracy or Consistency Drop , the more successful manipulations are.

Original: Note that we use the same metrics as rewards to our Adversarial Player .

Original: We use T-Test (Helmert, 1876; L¨uroth, 1876) for each metric all the runs.
Revised: We use T-Test (Helmert, 1876; L¨uroth, 1876) for each metric all the runs.

Original: Our null hypothesis is that the population mean of all the games is zero, indicating that a manipulation is unsuccessful, and the results of our Adversarial 
Revised: Our null hypothesis is that the population mean of all the games is zero, indicating that a manipulation is unsuccessful, and the results of our Adversarial Players are extreme cases of a good performance, i.e,

Original: Players are extreme cases of a good performance, i.e, H 0 : µ l = 0 , l ∈ [ Drop , Consistency ] .
Revised: H 0 : µ l = 0 , l ∈ [ Drop , Consistency ] .

Original: As an alternative hypothesis, we assume that the population mean is greater than zero, i.e., H A : µ l > 0 , l ∈ [ Drop , Consistency ] .
Revised: As an alternative hypothesis, we assume that the population mean is greater than zero, i.e., H A : µ l > 0 , l ∈ [ Drop , Consistency ] .

Original: The same setup stands for both the Consistency and the Drop metric.
Revised: The same setup stands for both the Consistency and the Drop metric.

Original: Tables 4 and 5 show the full table with p -values computed.
Revised: Tables 4 and 5 show the full table with p -values computed.

Original: Note that only in two cases, Adversarial Player does not manipulate the scene convincingly (TbD and Mdetr for the pixel-input Adversarial 
Revised: Note that only in two cases, Adversarial Player does not manipulate the scene convincingly (TbD and Mdetr for the pixel-input Adversarial Player ). [Table] [Table]

Original: Player ).
Revised: [Figure] [Table] [Table] IEP

Original: [Table] 
Revised: [Table] [Table] [Table]

Revised: TbD

Original: [Table]
Revised: [Table]

Revised: Mdetr

Original: [Table]
Revised: [Table]

Revised: State

Original: [Table]
Revised: [Table]

Revised: Input

Original: [Table]
Revised: [Table]

Revised: Model

Revised: [Table]

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: B A PPENDIX B B. M ODEL PERFORMANCE ON MAXIMAL DROP MINIGAMES .
Revised: B A PPENDIX B B. M ODEL PERFORMANCE ON MAXIMAL DROP MINIGAMES .

Original: In the table below, we present model accuracy on each of the minigames where the maximal  drop was observed.
Revised: In the table below [Table] we present model accuracy on each of the Mini-games where the maximal performance drop was observed.

Original: Thesecond and third column refer to  unmanipulated image-question pair accuracy in the minigame. The fourth and ﬁfth columns  refer to themeasured model accuracy in those minigames after the Adversarial Player manipulation.This indicates that minigames of maximal drop do not necessarily consist of ”hard” CLEVR examples, but rather exploit model and instancespeciﬁc vulnerabilities of the architecture under test.
Revised: The third and fourth columns (Original Data section) refer to the model accuracy on unmanipulated image-question pairs of the respective Mini-game . The fifth and sixth columns (Manipulated Data section) refer to the model accuracy in those Mini-game examples after the Adversarial Player manipulation.

Original: [Table]
Revised: [Table]

Original: B.  S CENE GENERATION METHOD AND UNINTENDED ISSUES .
Revised: B. G ENERATING ‘V ISUALLY F AIR ’ S CENES .

Original: In this section we will elaborate more on how we enforce that generated scenes are visually fair , in the sense that unintended issues such as object occlusion, or object placement out of ﬁeld of view are not existent, as well as answerable meaning that they can still be answered and  the  ground truth answer of the posed question before and after the scenemanipulations remains the same.
Revised: We define a scene as visually fair if there are no occlusions and all objects are within the field-ofview. The ground truth answer of a posed question on a visually fair scene remains the same before and after object manipulation.

Original: For the generation of new images/scenes we use the open-source Blender Graphics Engine(v2.79b), and the original 3D models of the CLEVR dataset.
Revised: For the generation of new images/scenes we use the open-source Blender Graphics Engine(v2.79b), and the original 3D models of the CLEVR dataset.

Original: Scenes suggested by our Adversarial
Revised: Scenes suggested by our Adversarial

Original: Player , are encoded in the form of arguments for Blender, which renders the new images according to those directives.
Revised: Player , are encoded in the form of arguments for Blender, which renders the new images according to those directives.

Original: In order to guarantee visual fairness, and question validity, we employ two modules (scene-constraint enforcer / question relevance enforcer), presented brieﬂy - in Section 4 in the
Revised: In order to guarantee visual fairness, and question validity, we employ two modules (scene-constraint enforcer / question relevance enforcer), presented briefly - in Section 4 in the

Original: Consistency and In-Distribution paragraphs.
Revised: Consistency and In-Distribution paragraphs.

Original: The scene-constraint enforcer is responsible for ensuringthat the creation of a scene/image that respects all constraints and statistical properties of the original dataset.
Revised: The scene-constraint enforcer is responsible for ensuring the creation of a scene/image that respects all constraints and statistical properties of the original dataset.

Original: This piece of code is borrowed from the original CLEVR codebase 3 and as in the original CLEVR paper the following constraints are checked (Lines 66-83 in the python code): 
Revised: This piece of code is borrowed from the original CLEVR codebase 3 and as in the original CLEVR paper the following constraints are checked (Lines 66-83 in the python code): Blender - a 3D modeling and rendering package. https://www. .org

Original: Minimum / Maximum Number of Objects: 3 / 10.
Revised: Minimum / Maximum Number of Objects: 3 / 10.

Original: Minimum allowed distance between object centers: 0. 3.
Revised: Minimum allowed distance between object centers: 0. 3.

Original: Margin along with cardinal directions: 0.4 (This ensures lack of ambiguity by keeping objects at least 0.4 points of distance apart.
Revised: Margin along with cardinal directions: 0.4 (This ensures lack of ambiguity by keeping objects at least 0.4 points of distance apart.

Original: This distance is a Blender-speciﬁc measure that spans from -3 to +3.) 4.
Revised: This distance is a Blender-specific measure that spans from -3 to +3.) 4.

Original: Min pixels per object: 200 ( This ensures that an object will not be occluded, by requiring at least 200 pixels of it being visible.)
Revised: Min pixels per object: 200 ( This ensures that an object will not be occluded, by requiring at least 200 pixels of it being visible.)

Original: The question-relevance enforcer checks if the newly generated scene is indeed answerable and has the same “ground truth” answer as the old scene.
Revised: The question-relevance enforcer checks if the newly generated scene is indeed answerable and has the same “ground truth” answer as the old scene.

Original: By not enforcing this, every manipulation will be falsely “successful” as the question (which is kept the same before and after the attack) will have a different answer in the newly generated scene.
Revised: By not enforcing this, every manipulation will be falsely “successful” as the question (which is kept the same before and after the attack) will have a different answer in the newly generated scene.

Original: This is also resolved by using the same question engine module provided by the CLEVR authors in their codebase 4 .
Revised: This is also resolved by using the same question engine module provided by the CLEVR authors in their codebase 4 .

Original: This module receives a question and a scene and calculates the ground truth answer based on a set of functional programs that are apriori known and used to generate the questions.
Revised: The module receives a question and a scene and calculates the ground truth answer based on a set of functional programs that are apriori known and used to generate the questions.

Original: (Supplementary Material A in the CLEVR paper (Johnson et al., 2017a))
Revised: (Supplementary Material A in the CLEVR paper (Johnson et al., 2017a))

Original: The full training pipeline should be the following: Our Adversarial Player initially receives an input image-question pair.
Revised: We use the following pipeline. Our Adversarial Player initially receives an input image-question pair.

Original: Then, it suggests a new scene, that passes through the scene-constraint enforcer.
Revised: Then, it suggests a new scene, that passes through the scene-constraint enforcer.

Original: If the new scene respects all boundaries and constraints it is then evaluated alongside the question by the question-relevance enforcer.
Revised: If the new scene respects all boundaries and constraints it is then evaluated alongside the question by the question-relevance enforcer.

Original: If the new ground truth answer to the suggested scene/image is the same as the original answer, the image-question pair is marked as in-distribution and valid.
Revised: If the new ground truth answer to the suggested scene/image is the same as the original answer, the image-question pair is marked as in-distribution and valid.

Original: In any other case, it is removed from the training pipeline.
Revised: Otherwise, it is removed from the training pipeline.

Original: /main question_generation/question_ py

Original: B. C OMPARISON WITH RANDOMLY SAMPLED EQUIVALENT SCENES .
Revised: B. C OMPARISON WITH RANDOMLY SAMPLED EQUIVALENT SCENES .

Original: Arguably, out of the semantically equivalent space of images given a question and an accompanying answer, the Adversarial Playerplayer effectively discovers cases for which the model under  test can be fooled.
Revised: Arguably, out of the semantically equivalent space of images given a question and an accompanying answer, the Adversarial Player effectively discovers cases for which the model under the test can be fooled.

Original: This however, does not indicate how rare those examples are, neither the degree of robustness of the modelunder test against other examples in this space.
Revised: This, however, does not indicate how rare those examples are, neither the degree of robustness of the model against other examples in this space.

Original: Given a speciﬁc initial image-question pair, a tractable estimate  would be the number of manipulated scenes ( F )discovered that are able to fool the model, divided by all the possible permutations of available objects ( N ) in our discrete 7x7 grid.
Revised: Let us limit ourselves on a specific image-question pair. Then, a tractable estimate of how rare ”fooling examples” are would be the number of manipulated scenes ( F ) that are able to fool the model, divided by all the possible permutations of available objects ( N ) in our discrete 7x7 grid.

Original: Rarity = F 49 N , N ∈ [3 , 10] .
Revised: Rarity = F 49 N , N ∈ [3 , 10] .

Original: A scene of only 3 objects and 1 discovered successful manipulation would make it 0.00085% possible to ﬁnd it by chance and would indicate 99.9915% robustness against that sphere of equivalent image-question pairs.
Revised: A scene of only 3 objects and 1 discovered successful manipulation would make it 0.00085% possible to find it by chance and would indicate 99.9915% robustness against that sphere of equivalent image-question pairs.

Original: Of course not all of those conﬁgurations are valid due to scene constraint violations, as well as objects can theoretically move freely on the scene.
Revised: Of course not all of those configurations are valid due to scene constraint violations, as well as objects can theoretically move freely on the scene.

Original: As a way to approach this question, we compare our Adversarial Playerplayer against a random scene generator (RSG).
Revised: As a way to approach this question, we compare our Adversarial Player against a random scene generator (RSG).

Original: The random scene generator is able to manipulate any object freely, on the constraint that the resulting scene is valid under the scene constraint enforcer as well as the question relevance enforcer.
Revised: The random scene generator is able to manipulate any object freely, under the constraint that the resulting scene is valid under the scene constraint enforcer as well as the question relevance enforcer.

Original: Manipulations of lighting or camera angles, as well as addition or removal of objects is not permitted.
Revised: Manipulations of lighting or camera angles, as well as addition or removal of objects is not permitted.

Original: In order to reduce the huge rendering times needed for each random search, we allow the RSG to operate on a 10 length Minigame where scene manipulations that cause accuracy drop have been discovered by the Adversarial
Revised: In order to reduce the huge rendering times needed for each random search, we allow the RSG to operate on a 10 length Minigame where scene manipulations that cause accuracy drop have been discovered by the Adversarial

Original: Player .
Revised: Player .

Original: For each image-question pair the RSG has a budget of 5000 queries, and if not successful it proceedes to the next Mini-game entry.
Revised: For each image-question pair the RSG has a budget of 5000 queries, and if not successful it proceeds to the next Mini-game entry.

Original: [Table]
Revised: [Table]

Original: As it can be seen, randomly question-uninformed suggested object placements do not seem to be effective into fooling the models under test.
Revised: As it can be seen, randomly question-uninformed suggested object placements do not seem to be effective into fooling the models under test.

Original: This further supports the argument that models under test theoretically and statistically seem robust, but nevertheless have vulnerabilities that can be effectively exploited by our Adversarial Player .
Revised: This further supports the argument that models under test theoretically and statistically seem robust, but nevertheless have vulnerabilities that can be effectively exploited by our Adversarial Player .

Original: B. B EHIND THE SCENES VISUALIZATION OF FAILURE MODES .
Revised: B. V ISUALIZATIONS .

Revised: Here, we try to isolate reasoning gaps from visual causes after the scene manipulations; with the following setup. Each Visual-QA Player is introduced with two image-question pairs. The first consists of the original image and the second of a scene created by our Adversarial Player that successfully fools the Visual-QA Player .

Original: Since the models under test are operating under different assumptions and architectural designs, weset up a visualization experiment to analyse visually their inner working mechanisms under a normal and a manipulated scene scenario. Speciﬁcally, we choose to visualize:
Revised: We then inspect the visualization of the Visual-QA Player ’s inner-workings in both pairs. We are especially interested in patterns that suggest the root of failure. As each Visual-QA Player operates under different assumptions and architectural designs, we choose to visualize :

Original: The attention maps of Stack-Attention Networks.
Revised: The attention maps of Stack-Attention Networks.

Original: The gradients of the pre-ultimate and ultimate convolutional blocks of FiLM.
Revised: The gradient-weighted activations of the pre-ultimate and ultimate convolutional blocks of FiLM.

Original: The programs that are synthesized alongside their soft-attention maps of each module block in TbD network.
Revised: The programs that are synthesized alongside their soft-attention maps of each module block in TbD network.

Original: The objects that were detected during the computation steps of a MDetr model.
Revised: The objects that were detected during the computation steps of a MDetr model.

Original: For each experiment, we choose two image-question pairs. The ﬁrst consists of the original image and the second ofa scene created by our Adversarial Player that succesfully fools themodel under test.
Revised: In Figure 22, we visualize the pre-ultimate (column 2) and ultimate (column 3) attention maps of the

Original: As we can observe, in Stack-Attention Network the model seems to concentrate all its focus on a single ﬁnal object in manipulated examples, while in the original ones, the relational jumps required for the answer are still visible on the attention map.
Revised: Stack-Attention Network. We observe that the model focuses on a single final object in manipulated examples. In the original ones, the relational jumps required for the answer are still visible on the attention map.

Original: The ﬁnal attention map that is presented  isultimately passed to a fully-connected classiﬁer in order for theﬁnal answer to be created.
Revised: The final attention map that is presented (column 3) is passed to a fully-connected classifier in order for the answer to be created.

Original: Hence, limited focus to all the necessary objects / locations may create lack of necessary features for the correct answer derivation.
Revised: We observe limited focus to all the necessary objects / locations that creates a lack of necessary features for the correct answer derivation.

Original: In the case of FiLM model  we observe a similar scenario.
Revised: In the case of FiLM model in Figure 23, we observe a similar scenario.

Original: In the ﬁrst example,  the gradient-weighted activations of the model seem to be focusing on multiple irrelevant objects regarding the ﬁnal answer, while in the second example  the model seems to include the purple cube mistakenly as one of the possible spheres of the image.
Revised: In the first example, (rows 1 and 2) the gradient-weighted activations of the model seem to be focusing on multiple irrelevant objects regarding the final answer, while in the second example (rows 3 and 4) the model seems to include the purple cube mistakenly as one of the possible spheres of the image.

Revised: In both cases above it is difficult to exactly pin down the root of (reasoning or visual) failures.

Original: In the case of the MDetr model, a speciﬁc pattern could not be identiﬁed.
Revised: MDetr model can be seen in Figure 24 and Figure 25. Here, a specific pattern could not be identified.

Original: All objects were correctly classiﬁed and the reasoning steps towards the derivation of  the answer are presented.
Revised: All objects were correctly classified. It suggests that the cause of failures stem rather from the reasoning steps.

Original: This objectdetection experiment, strengthens our claim of visually fair scene manipulations that respect the original dataset properties and do not promote object or attribute hindrance.

Original: Finally, in the case of the TbD, in  both the original and manipulated scenes the programs that the model generates are exactly the same.
Revised: Tbd is presented in Figure 26. In both the original and manipulated scenes the programs that the model generates are exactly the same.

Original: This is to no surprise since the program generator module is using exclusively the question as input.
Revised: This is to no surprise since the program generator module is using exclusively the question as input.

Original: Nevertheless, a similar pattern of scattered model focus is presented here as well.
Revised: Nevertheless, a similar pattern of scattered model focus is presented here as well.

Original: The attention maps in both examples are the same between the original and manipulated scenes, in all but the ﬁnal reasoning step.
Revised: The attention maps in both examples are the same between the original and manipulated scenes, in all but the final reasoning step.

Original: In the ﬁrst example  the attention is leaked towards the gray cube that has nothing to do with the correct answer, while in the second  the attention is spread among many matte objects but peaks near the big yellow ball.
Revised: In the first example (rows 1 and 2) the attention is leaked towards the gray cube that has nothing to do with the correct answer, while in the second (rows 3 and 4) the attention is spread among many matte objects but peaks near the big yellow ball.

Original: This is wrong since the word ”other” that exists in the question should drive the model focus exclusively to other objects apart from the starting point of the reasoning chain.[Figure]
Revised: This is wrong since the word ”other” that exists in the question should drive the model focus exclusively to other objects apart from the starting point of the reasoning chain.

Original: [Figure]
Revised: [Figure]

Original: How many other objects are the same shape as the big gray matte object?

Original: 1 −→

Original: O r i g i n a l S ce n e
Revised: O r i g i n a l S ce n e

Original: M a n i pu l a t e dS ce n e
Revised: M a n i pu l a t e d

Original: How many other things are the same shape as the big metallic thing?

Original: 1 −→

Original: O r i g i n a l

Original: S ce n e
Revised: S ce n e

Original: M a n i pu l a t e d S ce n e

Original: [Figure]
Revised: [Figure]

Original: what

Original: [Figure]

Original: −→

Original: [Figure]

Original:  g i n a l S ce n e
Revised: O r i g i n a l S ce n e

Original: M a n i pu l a t e d S ce n e
Revised: M a n i pu l a t e d S ce n e

Original: Do the object behind the big matte sphere and the thing on the right

Original: [Figure]
Revised: [Figure]

Original: of the

Original: [Figure]
Revised: [Figure]

Original: O r i g i n a l S ce n e M a n i pu l a t e d S c e n e
Revised: [Figure]

Original: Is there a big brown object of the same shape as the green thing?
Revised: Is there a big brown object of the same shape as the green thing?

Original: Yes −→
Revised: Yes −→

Original: No
Revised: No

Original: O r i g i n a l
Revised: O r i g i n a l

Original: S ce n e
Revised: S ce n e

Original: M a n i pu l a t e d S ce n e
Revised: M a n i pu l a t e d S ce n e

Original: Are there any other things that have the same material as the big yellow ball?
Revised: Are there any other things that have the same material as the big yellow ball?

Original: Yes −→
Revised: Yes −→

Original: No
Revised: No

Original: O r i g i n a l
Revised: O r i g i n a l

Original: S ce n e
Revised: S ce n e

Original: M a n i p u l a t e d
Revised: M a n i p u l a t e d

Original: S ce n e
Revised: S ce n e

Original: B. H UMAN - IN - THE - LOOP E XPERIMENT
Revised: B. H UMAN - IN - THE - LOOP E XPERIMENT

Original: Would a human be fooled from scene manipulations of this kind?
Revised: Can our Adversarial Player ‘fool’ a human-agent?

Original: The perceptual abilities of human observers have been previously measured (92.6%) on this dataset, however human robustness is atotally different question.
Revised: The perceptual abilities of human observers have been previously measured (92.6%) on this dataset, however, human robustness is a different question.

Revised: We, hence, decided to assess human robustness by a simple experiment.

Original: In our experiment, a human observer is presented with a set of 2 images and a question.
Revised: In our setting, a human observer is presented with a set of two images and a question.

Original: Then the observer is required to answer to each imageimage question pair: (Image Before - Question / Image After - Question), and report whether the answer has changed or remained the same.
Revised: Then the observer is required to answer to each image question pair, (image before - question / image after - question), and report whether the answer has changed or remained the same.

Original: Participants were ﬁrst made familiar with the concepts of the CLEVR dataset (colors/shapes/materials/sizes) and then were left unsupervised to answer a set of 20 such examples.
Revised: Participants were first made familiar with the concepts of the CLEVR dataset (colors / shapes / materials / sizes) and then were left unsupervised to answer a set of 20 such examples.

Original: Each contained 1 CLEVR question and 2 CLEVR scenes, before and after the Adversarial Player manipulations.
Revised: Each contained one CLEVR question and two CLEVR scenes, before and after the Adversarial Player manipulations.

Original: Half of those scenes successfully fooled models and half of those did not.
Revised: Half of those scenes successfully fooled models and half of those did not.

Original: An example of what a survey question looks like is given below: Here is the confusion matrix from the survey results:
Revised: An example of what a survey question looks like is given in Figure 27, together with the confusion matrix Figure 28 (results of our survey).

Original: As we can see, our human observers exhibit 91% precision in identifying cases where scene manipulations are not causing change to the questions answer, and 95% precision in identifying cases where the manipulations do cause a change.
Revised: As we can see in Figure 28, human observers exhibit 91% precision in identifying cases where scene manipulations are not causing changes to the questions answer, and 95% precision in identifying cases where the manipulations do cause a change.

Original: While this small scale experiment may not reﬂect a universal truth about human robustness in VQA scenarios, it is a suggestion that human observation is still reliable.
Revised: While this small scale experiment may not reflect a universal truth about human robustness in VQA scenarios, it is a suggestion that human observation is still reliable after our agent’s manipulations.

Original: B. O BJECT D ETECTION E XPERIMENT
Revised: B. O BJECT D ETECTION E XPERIMENT

Original: As an extra step towards quantitatively testing if the adversarial manipulations make the image indeed visually ambiguous, we decided to used a pre-trained CLEVR object detector to measure the difference in performance between original and manipulated images. Theoretically if the objects before and after the manipulations remain detected by the model, as well as their attributes remain correctly classiﬁed, that would signify that our Adversarial Player manipulationsdo not cause any situation that would hinder the identiﬁcation of an object. For this experiment, we used Slot Attention Locatello (2020) with pre-trained weights.
Revised: As an extra test to confirm the visual fairness (see section B.2) of Adversarial Player manipulations we run the following experiment.

Original: It is an unsupervisedmethod for object discovery that achieves almost perfect performance on CLEVR. We were not able to ﬁnd a pair of images in which an object that was identiﬁed initially was not after the manipulations. The same ﬁndings stood for the object identiﬁcation part, since not a pair of initially classiﬁed object was misclassiﬁed after the manipulations. That concludes that our Adversarial
Revised: We employed an unsupervised object discovery model, called

Original: Player manipulations do not cause any speciﬁc visual obstacle for an object to be correctly identiﬁed. O r i g i n a l S ce n e [Figure]
Revised: Slot Attention (Locatello, 2020), pre-trained on CLEVR, together with a classifier trained to identify object attributes.

Original: a n i [Figure] l a t [Figure]
Revised: Next, we used that model to identify objects and attributes on original and manipulated scenes.

Original: S ce n e [Figure] [Figure] i
Revised: Ultimately, we could not find any pairs in which the number of objects or their attributes were different before and after the manipulations.

Original: [Figure] i n a l [Figure] n e
Revised: The results for two pairs can be seen in Figure 29 and Figure 30.

Original: [Figure] l a t
Revised: We thus conclude that our Adversarial Player manipulations create ‘visually fair’ scenes, without producing any corner cases of perceptual system.

Original: [Figure]
Revised: [Figure]

Original: n e

Original: [Figure]
Revised: [Figure]