Original: M EASURING CLEVR NESS : B LACK - BOX T ESTING OF V ISUAL R EASONING M ODELS
Revised: M EASURING CLEVR NESS : B LACK - BOX T ESTING OF V ISUAL R EASONING M ODELS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: How to measure the reasoning capabilities of intelligence systems?
Revised: How to measure the reasoning capabilities of intelligence systems?

Original: Visual question answering provides a convenient framework for testing the model’s abilities by interrogating the model through questions about the scene.
Revised: Visual question answering provides a convenient framework for testing the model’s abilities by interrogating the model through questions about the scene.

Original: However, despite the score of various visual QA datasets and architectures, which sometimes yield even a super-human performance, the question of whether those architectures can do reasoning remains open.
Revised: However, despite the score of various visual QA datasets and architectures, which sometimes yield even a super-human performance, the question of whether those architectures can do reasoning remains open.

Original: To answer such a question, we extend the visual question answering framework and propose the following behavioural test in the form of a two-player game.
Revised: To answer such a question, we extend the visual question answering framework and propose the following behavioural test in the form of a two-player game.

Original: We consider black-box neural models of CLEVR.
Revised: We consider black-box neural models of CLEVR.

Original: Those models are trained on a diagnostic dataset benchmarking reasoning.
Revised: Those models are trained on a diagnostic dataset benchmarking reasoning.

Original: Next, we train an adversarial player that re-conﬁgures the scene to fool the CLEVR model.
Revised: Next, we train an adversarial player that re-conﬁgures the scene to fool the CLEVR model.

Original: We show that CLEVR models, which otherwise may perform at a human level, can easily be fooled by our agent.
Revised: We show that CLEVR models, which otherwise may perform at a human level, can easily be fooled by our agent.

Original: Our results question one more time whether datadriven approaches can do reasoning without exploiting numerous biases that are often present in those datasets.
Revised: Our results question one more time whether datadriven approaches can do reasoning without exploiting numerous biases that are often present in those datasets.

Original: Finally, we also propose a controlled experiment measuring the efﬁciency of such models to learn and perform reasoning.
Revised: Finally, we also propose a controlled experiment measuring the efﬁciency of such models to learn and perform reasoning.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Are our artiﬁcial intelligence systems capable of reasoning?
Revised: Are our artiﬁcial intelligence systems capable of reasoning?

Original: Or like Clever Hans , they use various cues only tangentially related to the task and rely on rote memorization with poor generalization?
Revised: Or like Clever Hans , they use various cues only tangentially related to the task and rely on rote memorization with poor generalization?

Original: (Pfungst, 1911; Johnson et al., 2017a)
Revised: (Pfungst, 1911; Johnson et al., 2017a)

Original: This work revisits such a question and proposes an interactive framework with the communication channel between two players.
Revised: This work revisits such a question and proposes an interactive framework with the communication channel between two players.

Original: The ﬁrst player, which reasoning capabilities we are about to test, performs visual reasoning tasks.
Revised: The ﬁrst player, which reasoning capabilities we are about to test, performs visual reasoning tasks.

Original: The second player, which we call the Adversarial Player , is manipulating the scene so that it fools the ﬁrst player even though those changes still lead to correct reasoning steps among humans.
Revised: The second player, which we call the Adversarial Player , is manipulating the scene so that it fools the ﬁrst player even though those changes still lead to correct reasoning steps among humans.

Original: Both players interact with each other only through questions, answers and the visual scene as shown in Figure 1.
Revised: Both players interact with each other only through questions, answers and the visual scene as shown in Figure 1.

Original: Since we use the visual question answering framework as the interrogation medium, assuming that both players can deal with textual questions and answers, we also call the ﬁrst player the Visual-QA Player .
Revised: Since we use the visual question answering framework as the interrogation medium, assuming that both players can deal with textual questions and answers, we also call the ﬁrst player the Visual-QA Player .

Original: If the Adversarial Player manipulating the scene causes the Visual-QA Player to change its answer even though the new scene is still valid for the same question and answer, it is then the reasoning failure.
Revised: If the Adversarial Player manipulating the scene causes the Visual-QA Player to change its answer even though the new scene is still valid for the same question and answer, it is then the reasoning failure.

Original: It is similar to the following situation.
Revised: It is similar to the following situation.

Original: Imagine a box is placed between two spheres.
Revised: Imagine a box is placed between two spheres.

Original: If you ask a question, is there a box between two spheres?
Revised: If you ask a question, is there a box between two spheres?

Original: , the answer should be positive.
Revised: , the answer should be positive.

Original: Now, if we move the box anywhere so it does not cross any of the spheres, and ask the same question, the response should remain unchanged.
Revised: Now, if we move the box anywhere so it does not cross any of the spheres, and ask the same question, the response should remain unchanged.

Original: In other words, we postulate that reasoning outputs of agents need to be invariant under scene conﬁgurations that are consistent with the questions-answer pairs.
Revised: In other words, we postulate that reasoning outputs of agents need to be invariant under scene conﬁgurations that are consistent with the questions-answer pairs.

Original: Moreover, in the spirit of generic adversarial attacks (Szegedy et al., 2014; Goodfellow et al., 2015), we seek conﬁgurations that also pose little if any reasoning challenges for humans.
Revised: Moreover, in the spirit of generic adversarial attacks (Szegedy et al., 2014; Goodfellow et al., 2015), we seek conﬁgurations that also pose little if any reasoning challenges for humans.

Original: In this work, we propose an automatic and agnostic pipeline to benchmark the reasoning capabilities of various models.
Revised: In this work, we propose an automatic and agnostic pipeline to benchmark the reasoning capabilities of various models.

Original: The only assumption that we make about those models is that they can perceive the scene and communicate by answering questions about the scene.
Revised: The only assumption that we make about those models is that they can perceive the scene and communicate by answering questions about the scene.

Original: Due to the recent stream of research in vision-and-language (Zhang et al., 2021; Jiang et al., 2020; Guo et al., 2019b; Su et al., 2020; Wang et al., 2020; Kamath et al., 2021; Tan & Bansal, 2019; Chen et al., 2020), we believe there will be an increasing number of vision models that can operate through language.
Revised: Due to the recent stream of research in vision-and-language (Zhang et al., 2021; Jiang et al., 2020; Guo et al., 2019b; Su et al., 2020; Wang et al., 2020; Kamath et al., 2021; Tan & Bansal, 2019; Chen et al., 2020), we believe there will be an increasing number of vision models that can operate through language.

Original: Moreover, we also consider the visual question answering framework set-up as a two-player system as an excellent benchmarking pipeline.
Revised: Moreover, we also consider the visual question answering framework set-up as a two-player system as an excellent benchmarking pipeline.

Original: We perform all tests by scene manipulations and observing how a tested model behaves under such changes.
Revised: We perform all tests by scene manipulations and observing how a tested model behaves under such changes.

Original: The pipeline does not require any knowledge of the internals of the tested model.
Revised: The pipeline does not require any knowledge of the internals of the tested model.

Original: It also does not manipulate the sensory information of such a model, i.e., pixels in the images, and all the manipulations are physically meaningful.
Revised: It also does not manipulate the sensory information of such a model, i.e., pixels in the images, and all the manipulations are physically meaningful.

Original: Even though our current pipeline uses synthetic scenes as only those can easily be automatically manipulated, our results have also real-world ramiﬁcations.
Revised: Even though our current pipeline uses synthetic scenes as only those can easily be automatically manipulated, our results have also real-world ramiﬁcations.

Original: If models are susceptible to semantically meaningless changes 1 in scene conﬁgurations, in a synthetic setting, there are valid concerns that real-world robots could also be prone to manipulation of objects in a room.
Revised: If models are susceptible to semantically meaningless changes 1 in scene conﬁgurations, in a synthetic setting, there are valid concerns that real-world robots could also be prone to manipulation of objects in a room.

Original: Finally, our work also questions the possibility of training and benchmarking networks in a purely data-driven and ofﬂine, static manner.
Revised: Finally, our work also questions the possibility of training and benchmarking networks in a purely data-driven and ofﬂine, static manner.

Original: Although most of our CLEVR models can reach a ‘superhuman’ performance on CLEVR, they are much less-CLEVR if another agent can manipulate the scene.
Revised: Although most of our CLEVR models can reach a ‘superhuman’ performance on CLEVR, they are much less-CLEVR if another agent can manipulate the scene.

Original: Our results show that those models fail on various reasoning tasks, even though a relatively simple interpolation between conﬁgurations is possible; where even no new features are introduced.
Revised: Our results show that those models fail on various reasoning tasks, even though a relatively simple interpolation between conﬁgurations is possible; where even no new features are introduced.

Original: We also show that the existing models may be inefﬁcient in learning to perform reasoning.
Revised: We also show that the existing models may be inefﬁcient in learning to perform reasoning.

Original: Contributions.
Revised: Contributions.

Original: The main contributions of our work could be summarized in three points.
Revised: The main contributions of our work could be summarized in three points.

Original: First , we propose a strong black-box adversarial test, which makes no assumptions about underlying mechanics of a tested model, formulated as a game between two players.
Revised: First , we propose a strong black-box adversarial test, which makes no assumptions about underlying mechanics of a tested model, formulated as a game between two players.

Original: Our test does not require any direct access to the tested model, even through its sensory information.
Revised: Our test does not require any direct access to the tested model, even through its sensory information.

Original: In particular, it does not require gradients, output probabilities, or any access to the perceived image.
Revised: In particular, it does not require gradients, output probabilities, or any access to the perceived image.

Original: Our work also deviates from bounded perturbations and instead focuses on global scene manipulations that are still consistent with the task constraints, and can change the behaviour of a tested model.
Revised: Our work also deviates from bounded perturbations and instead focuses on global scene manipulations that are still consistent with the task constraints, and can change the behaviour of a tested model.

Original: Second , we reformulate visual reasoning by integrating visual question answering with zero-sum two-player game frameworks.
Revised: Second , we reformulate visual reasoning by integrating visual question answering with zero-sum two-player game frameworks.

Original: Under our novel formulation, a visual and adversary agents compete against each other through content manipulation.
Revised: Under our novel formulation, a visual and adversary agents compete against each other through content manipulation.

Original: We believe that this is an initial step towards more sophisticated frameworks that integrate computer vision with multi-agent systems.
Revised: We believe that this is an initial step towards more sophisticated frameworks that integrate computer vision with multi-agent systems.

Original: Third , we explore the limits of the data-driven approaches in synthetic visual scenarios, and demonstrate that current CLEVR models are lacking the efﬁciency to learn robust reasoning steps.
Revised: Third , we explore the limits of the data-driven approaches in synthetic visual scenarios, and demonstrate that current CLEVR models are lacking the efﬁciency to learn robust reasoning steps.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Our work touches upon various research directions, which we brieﬂy describe here.
Revised: Our work touches upon various research directions, which we brieﬂy describe here.

Original: Visual QA.
Revised: Visual QA.

Original: Introduced as a visual counter-part of the Turing Test (Malinowski & Fritz, 2014; Geman et al., 2015), it became a computer vision task that requires a holistic visual understanding.
Revised: Introduced as a visual counter-part of the Turing Test (Malinowski & Fritz, 2014; Geman et al., 2015), it became a computer vision task that requires a holistic visual understanding.

Original: Many other works have extended the task to larger datasets or videos or adversarial train-test splits (Antol et al., 2015;
Revised: Many other works have extended the task to larger datasets or videos or adversarial train-test splits (Antol et al., 2015;

Original: Agrawal et al., 2018; Tapaswi et al., 2016; Yu et al., 2019; Girdhar & Ramanan, 2019).
Revised: Agrawal et al., 2018; Tapaswi et al., 2016; Yu et al., 2019; Girdhar & Ramanan, 2019).

Original: More recently, we also observe the generalization of the task to become a part of the vision-pluslanguage suite of benchmarks (Lu et al., 2019; Chen et al., 2019; Wang et al., 2021).
Revised: More recently, we also observe the generalization of the task to become a part of the vision-pluslanguage suite of benchmarks (Lu et al., 2019; Chen et al., 2019; Wang et al., 2021).

Original: Johnson et al.
Revised: Johnson et al.

Original: (2017a) have introduced a synthetic variant of the Visual QA problem that is more focused on reasoning and the control of the experimentation.
Revised: (2017a) have introduced a synthetic variant of the Visual QA problem that is more focused on reasoning and the control of the experimentation.

Original: Although it was shown that traditional methods (Yang et al., 2016) are not enough to solve that dataset, newer methods can pass the human baseline on this task.
Revised: Although it was shown that traditional methods (Yang et al., 2016) are not enough to solve that dataset, newer methods can pass the human baseline on this task.

Original: Those methods are trained to operate on pixels and text directly (Santoro et al., 2017; Perez et al., 2018; Hudson & Manning, 2018; Malinowski et al., 2018; Kamath et al., 2021) or they induce programs as an intermediate representation (Johnson et al., 2017b;
Revised: Those methods are trained to operate on pixels and text directly (Santoro et al., 2017; Perez et al., 2018; Hudson & Manning, 2018; Malinowski et al., 2018; Kamath et al., 2021) or they induce programs as an intermediate representation (Johnson et al., 2017b;

Original: Mascharka et al., 2018;
Revised: Mascharka et al., 2018;

Original: Yi et al., 2018).
Revised: Yi et al., 2018).

Original: We extend this line of research by reformulating the Visual QA task as an interactive twoagents system, where we automatically test the reasoning capabilities of some methods above in the controlled setting.
Revised: We extend this line of research by reformulating the Visual QA task as an interactive twoagents system, where we automatically test the reasoning capabilities of some methods above in the controlled setting.

Original: We also show that CLEVR under such a new setting is still an unsolved problem.
Revised: We also show that CLEVR under such a new setting is still an unsolved problem.

Original: Adversarial perturbations.
Revised: Adversarial perturbations.

Original: Szegedy et al.
Revised: Szegedy et al.

Original: ; Goodfellow et al.
Revised: ; Goodfellow et al.

Original: ; Kurakin et al.
Revised: ; Kurakin et al.

Original: Moosavi-Dezfooli et al.
Revised: Moosavi-Dezfooli et al.

Original: ; Carlini & Wagner (2016) have introduced early methods that can ‘fool’ networks by performing tiny, visually imperceptible, perturbations of the input image.
Revised: ; Carlini & Wagner (2016) have introduced early methods that can ‘fool’ networks by performing tiny, visually imperceptible, perturbations of the input image.

Original: Moosavi-Dezfooli et al.
Revised: Moosavi-Dezfooli et al.

Original: (2017) have also shown universal and untargeted adversarial perturbations that are more transferable than previous approaches.
Revised: (2017) have also shown universal and untargeted adversarial perturbations that are more transferable than previous approaches.

Original: Most methods perform white-box attacks, where an adversarial model has access to the target model’s parameters or gradients.
Revised: Most methods perform white-box attacks, where an adversarial model has access to the target model’s parameters or gradients.

Original: In black-box attacks such access is removed and, e.g., only output probabilities are available (Guo et al., 2019a).
Revised: In black-box attacks such access is removed and, e.g., only output probabilities are available (Guo et al., 2019a).

Original: Other black-box attacks involve the use of surrogates to estimate gradients of the real target (Papernot et al., 2017; Cheng et al., 2019) or gradient-free methods (Alzantot et al., 2018).
Revised: Other black-box attacks involve the use of surrogates to estimate gradients of the real target (Papernot et al., 2017; Cheng et al., 2019) or gradient-free methods (Alzantot et al., 2018).

Original: More related to our work, Cheng et al.
Revised: More related to our work, Cheng et al.

Original: (2018) describe a black-box system, which only assumes the access to outputs, so called hard-labels, of the network for a given query.
Revised: (2018) describe a black-box system, which only assumes the access to outputs, so called hard-labels, of the network for a given query.

Original: However, all the perturbations described above are continuous-based where individual pixels are changed independently.
Revised: However, all the perturbations described above are continuous-based where individual pixels are changed independently.

Original: This leads to improbable images and assumes the access to the sensory inputs of the neural network.
Revised: This leads to improbable images and assumes the access to the sensory inputs of the neural network.

Original: In contrary, semantic perturbations (Joshi et al., 2019; Zeng et al., 2019), operate on semantically meaningful chunks of the input, leading to more plausible and physically interpretable perturbations.
Revised: In contrary, semantic perturbations (Joshi et al., 2019; Zeng et al., 2019), operate on semantically meaningful chunks of the input, leading to more plausible and physically interpretable perturbations.

Original: However, they still assume a direct access to either images or ideally differentiable renderers.
Revised: However, they still assume a direct access to either images or ideally differentiable renderers.

Original: We can interpret our work within the adversarial perturbations framework, where the adversarial model can change the original scene by its manipulations, with zero access to the target model, including its sensory information and not limited to tiny changes but instead to semantically and globally meaningless modiﬁcations.
Revised: We can interpret our work within the adversarial perturbations framework, where the adversarial model can change the original scene by its manipulations, with zero access to the target model, including its sensory information and not limited to tiny changes but instead to semantically and globally meaningless modiﬁcations.

Original: Reinforcement learning.
Revised: Reinforcement learning.

Original: The game between the Visual-QA Player and Adversarial Player is a zero-sum two-player game.
Revised: The game between the Visual-QA Player and Adversarial Player is a zero-sum two-player game.

Original: Conneau et al.
Revised: Conneau et al.

Original: (2017) use a conceptually similar two-player game to ﬁnd a better word alignment between two languages without parallel corpus.
Revised: (2017) use a conceptually similar two-player game to ﬁnd a better word alignment between two languages without parallel corpus.

Original: Inspired by such a problem formulation, we consider our two-player game as a tool to achieve better reasoning models.
Revised: Inspired by such a problem formulation, we consider our two-player game as a tool to achieve better reasoning models.

Original: We optimize the Adversarial Player with Advantage Actor-Critic (A2C) algorithm (Sutton & Barto, 2018; Degris et al., 2012).
Revised: We optimize the Adversarial Player with Advantage Actor-Critic (A2C) algorithm (Sutton & Barto, 2018; Degris et al., 2012).

Original: Though there are two players in the game between Visual-QA Player and
Revised: Though there are two players in the game between Visual-QA Player and

Original: Adversarial Player , in experiments presented in this work, the Visual-QA Player is frozen.
Revised: Adversarial Player , in experiments presented in this work, the Visual-QA Player is frozen.

Original: Thus, technically, simpler optimization methods used for single-player games are applicable.
Revised: Thus, technically, simpler optimization methods used for single-player games are applicable.

Original: Probing and measuring intelligence.
Revised: Probing and measuring intelligence.

Original: Turing Test (Turing, 2009) is considered to be among the earliest works on measuring the intelligence of artiﬁcial systems.
Revised: Turing Test (Turing, 2009) is considered to be among the earliest works on measuring the intelligence of artiﬁcial systems.

Original: Crucially to us, it is also formulated as a two-player game with an interrogation protocol.
Revised: Crucially to us, it is also formulated as a two-player game with an interrogation protocol.

Original: Legg (2008) systematizes and relates notion of intelligence from different ﬁelds with the main focus on ‘universality’.
Revised: Legg (2008) systematizes and relates notion of intelligence from different ﬁelds with the main focus on ‘universality’.

Original: Similarly, Chollet (2019) investigates various attempts at deﬁning or measuring intelligence and highlights the role of skill-acquisition efﬁciency.
Revised: Similarly, Chollet (2019) investigates various attempts at deﬁning or measuring intelligence and highlights the role of skill-acquisition efﬁciency.

Original: Johnson et al.
Revised: Johnson et al.

Original: (2017a) have transferred the notion of intelligence onto the visual ground and emphasize reasoning rather than universality.
Revised: (2017a) have transferred the notion of intelligence onto the visual ground and emphasize reasoning rather than universality.

Original: Nonetheless, it seems there is no a single widely acceptable dataset or a deﬁnition that encompasses our intuitions about intelligence.
Revised: Nonetheless, it seems there is no a single widely acceptable dataset or a deﬁnition that encompasses our intuitions about intelligence.

Original: However, there are a few recent directions showing the lack thereof, mainly highlighting bias ampliﬁcation or absence of mathematical capabilities (Hendricks et al., 2018; Bhardwaj et al., 2021;
Revised: However, there are a few recent directions showing the lack thereof, mainly highlighting bias ampliﬁcation or absence of mathematical capabilities (Hendricks et al., 2018; Bhardwaj et al., 2021;

Original: Piekos et al., 2021).
Revised: Piekos et al., 2021).

Original: Here, we are mostly interested in visual reasoning and use the visual question answering framework as a form of communication with the otherwise black-box system.
Revised: Here, we are mostly interested in visual reasoning and use the visual question answering framework as a form of communication with the otherwise black-box system.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: In this section, we explain brieﬂy how the CLEVR dataset (Johnson et al., 2017a) is constructed and introduce our notation and deﬁnitions.
Revised: In this section, we explain brieﬂy how the CLEVR dataset (Johnson et al., 2017a) is constructed and introduce our notation and deﬁnitions.

Original: CLEVR is a synthetic visual question answering dataset introduced by Johnson et al.
Revised: CLEVR is a synthetic visual question answering dataset introduced by Johnson et al.

Original: (2017a), which consists of about 700k training and 150k validation image-question-answer triplets.
Revised: (2017a), which consists of about 700k training and 150k validation image-question-answer triplets.

Original: Images are artiﬁcially constructed and rendered from scene graphs – a special structure containing information about object attributes such as position or color.
Revised: Images are artiﬁcially constructed and rendered from scene graphs – a special structure containing information about object attributes such as position or color.

Original: Such a scene graph is also used to synthesize the ground-truth question-answer pairs by expanding templates according to the depth-ﬁrst-search ordering.
Revised: Such a scene graph is also used to synthesize the ground-truth question-answer pairs by expanding templates according to the depth-ﬁrst-search ordering.

Original: Ambiguous scenes are rejected.
Revised: Ambiguous scenes are rejected.

Original: Each image represents an isometric view on the scene containing from two to ten objects.
Revised: Each image represents an isometric view on the scene containing from two to ten objects.

Original: There are three classes of objects: spheres, cubes and cylinders.
Revised: There are three classes of objects: spheres, cubes and cylinders.

Original: Each object can also be either large or small and has one colour out of four (brown, purple, cyan, yellow).
Revised: Each object can also be either large or small and has one colour out of four (brown, purple, cyan, yellow).

Original: It can also be either metallic or rubber-made.
Revised: It can also be either metallic or rubber-made.

Original: Every object has x and y coordinates that are conﬁned within the ( − 3 , +3) range.
Revised: Every object has x and y coordinates that are conﬁned within the ( − 3 , +3) range.

Original: We use the same generation process to render modiﬁed scenes.
Revised: We use the same generation process to render modiﬁed scenes.

Original: CLEVR models.
Revised: CLEVR models.

Original: Various models have been introduced to work with the CLEVR dataset, some even ‘solving’ the dataset by achieving near perfect performance.
Revised: Various models have been introduced to work with the CLEVR dataset, some even ‘solving’ the dataset by achieving near perfect performance.

Original: Despite of strong ofﬂine performance, we test if those models performance perpetuates in the more interactive setting where conﬁgurations of the scene could be changed.
Revised: Despite of strong ofﬂine performance, we test if those models performance perpetuates in the more interactive setting where conﬁgurations of the scene could be changed.

Original: Whenever possible, we use pre-trained CLEVR models.
Revised: Whenever possible, we use pre-trained CLEVR models.

Original: Otherwise, we train the remaining models from scratch by making sure we achieve results similar to published accuracy numbers on the validation set.
Revised: Otherwise, we train the remaining models from scratch by making sure we achieve results similar to published accuracy numbers on the validation set.

Original: We summarize all the models in Table 1.
Revised: We summarize all the models in Table 1.

Original: We show the accuracy on the CLEVR dataset ( Accuracy ), indicate if an architecture is trained from scratch ( Re-trained ), brieﬂy describe how multimodal fusion and reasoning is conducted ( Reasoning Mechanism ), and indicate any extra privileged information required during the training process ( Extra ).
Revised: We show the accuracy on the CLEVR dataset ( Accuracy ), indicate if an architecture is trained from scratch ( Re-trained ), brieﬂy describe how multimodal fusion and reasoning is conducted ( Reasoning Mechanism ), and indicate any extra privileged information required during the training process ( Extra ).

Original: For instance, some models require extra access to functional programs used during the dataset generation, use scene graphs as a supervisory signal (states), or always operate on scene graphs (input-states).
Revised: For instance, some models require extra access to functional programs used during the dataset generation, use scene graphs as a supervisory signal (states), or always operate on scene graphs (input-states).

Original: Otherwise the models were trained only from image-question-answer triples.
Revised: Otherwise the models were trained only from image-question-answer triples.

Original: Mini-games .
Revised: Mini-games .

Original: We formulate our problem as a Game between two players: Visual-QA Player and
Revised: We formulate our problem as a Game between two players: Visual-QA Player and

Original: Adversarial Player .
Revised: Adversarial Player .

Original: The Visual-QA Player takes as input question-image pairs and provide answers to such questions.
Revised: The Visual-QA Player takes as input question-image pairs and provide answers to such questions.

Original: Some models use states (scene-graphs) that replace images or require programs (Johnson et al., 2017a).
Revised: Some models use states (scene-graphs) that replace images or require programs (Johnson et al., 2017a).

Original: The whole game consists of all CLEVR data points.
Revised: The whole game consists of all CLEVR data points.

Original: For our purpose, we extend the notion of the Game into Mini-games .
Revised: For our purpose, we extend the notion of the Game into Mini-games .

Original: The rules of Mini-games are identical to the whole Game .
Revised: The rules of Mini-games are identical to the whole Game .

Original: The only difference is that each Mini-game operates on a subset of the CLEVR dataset.
Revised: The only difference is that each Mini-game operates on a subset of the CLEVR dataset.

Original: We deﬁne the size of a Mini-game by the number of datapoints that are attached to that Minigame .
Revised: We deﬁne the size of a Mini-game by the number of datapoints that are attached to that Minigame .

Original: We sample data points for each Mini-game randomly and mutually exclusively.
Revised: We sample data points for each Mini-game randomly and mutually exclusively.

Original: Mini-games have analogies in the adversarial perturbations literature.
Revised: Mini-games have analogies in the adversarial perturbations literature.

Original: Mini-games of size one resemble perimage adversarial perturbations (Goodfellow et al., 2015; Moosavi-Dezfooli et al., 2016) whereas a
Revised: Mini-games of size one resemble perimage adversarial perturbations (Goodfellow et al., 2015; Moosavi-Dezfooli et al., 2016) whereas a

Original: Mini-game that has all data pointsis similar to universal adversarial perturbations (Moosavi-Dezfooli et al., 2017).
Revised: Mini-game that has all data pointsis similar to universal adversarial perturbations (Moosavi-Dezfooli et al., 2017).

Original: In this work, we investigate various Mini-game sizes but due to the sheer scale we were unable to use the whole game as the Mini-game .
Revised: In this work, we investigate various Mini-game sizes but due to the sheer scale we were unable to use the whole game as the Mini-game .

Original: Larger Mini-games make the optimization process more difﬁcult as the domain where the Adversarial Player needs to operate increases.
Revised: Larger Mini-games make the optimization process more difﬁcult as the domain where the Adversarial Player needs to operate increases.

Original: The training is also much more time-consuming and a sequential process.
Revised: The training is also much more time-consuming and a sequential process.

Original: Instead, we can train multiple players on different Mini-games independently and thus massively.
Revised: Instead, we can train multiple players on different Mini-games independently and thus massively.

Original: We leave the arduous training of the universal Adversarial Player on the whole Game as a possible future direction.
Revised: We leave the arduous training of the universal Adversarial Player on the whole Game as a possible future direction.

Original: [Table]
Revised: [Table]

Original:  E NVIRONMENT
Revised:  E NVIRONMENT

Original: We need to ensure that Adversarial Players create valid scenes that are consistent and in-distribution .
Revised: We need to ensure that Adversarial Players create valid scenes that are consistent and in-distribution .

Original: Both properties are guaranteed by our environment enforcers.
Revised: Both properties are guaranteed by our environment enforcers.

Original: Consistency.
Revised: Consistency.

Original: Since scene manipulation may change the answer for a given question, we need to ensure this does not happen.
Revised: Since scene manipulation may change the answer for a given question, we need to ensure this does not happen.

Original: That is, the new scene is still consistent with the question-answer pair.
Revised: That is, the new scene is still consistent with the question-answer pair.

Original: The question-relevance enforcer achieves that by running functional programs associated with each question (Johnson et al., 2017a) on the modiﬁed scene-graph.
Revised: The question-relevance enforcer achieves that by running functional programs associated with each question (Johnson et al., 2017a) on the modiﬁed scene-graph.

Original: Hence, it gets the new ground-truth answer.
Revised: Hence, it gets the new ground-truth answer.

Original: The enforcer rejects the new scene if that new answer differs from the previous one.
Revised: The enforcer rejects the new scene if that new answer differs from the previous one.

Original: In this way, it guarantees the newly generated scenes give the same answers as the original scenes on the same question.
Revised: In this way, it guarantees the newly generated scenes give the same answers as the original scenes on the same question.

Original: Thus, we can generate equivalent scenes containing the same objects that have identical answers for the same questions.
Revised: Thus, we can generate equivalent scenes containing the same objects that have identical answers for the same questions.

Original: Using that enforcer, we can test if the Visual-QA Player ’s answers are invariant under such an equivalent class of scenes.
Revised: Using that enforcer, we can test if the Visual-QA Player ’s answers are invariant under such an equivalent class of scenes.

Original: In-distribution.
Revised: In-distribution.

Original: Even with the question-relevance enforcer, the Adversarial Player may still produce undesired outputs.
Revised: Even with the question-relevance enforcer, the Adversarial Player may still produce undesired outputs.

Original: For instance, it can stretch the whole scene thus violating the scene boundaries from the original CLEVR dataset, making, e.g., everything to look very small (Section A.9 in the appendix).
Revised: For instance, it can stretch the whole scene thus violating the scene boundaries from the original CLEVR dataset, making, e.g., everything to look very small (Section A.9 in the appendix).

Original: Although it is still an interesting form of the adversarial scene manipulation, we focus rather on the in-distribution scene manipulations that respect the original boundaries.
Revised: Although it is still an interesting form of the adversarial scene manipulation, we focus rather on the in-distribution scene manipulations that respect the original boundaries.

Original: To enforce that property, we use scene-constraint enforcer that checks the boundaries of the scene.
Revised: To enforce that property, we use scene-constraint enforcer that checks the boundaries of the scene.

Original: Without that enforcer, the Adversarial Player would quickly resort to stretching the whole scenes, achieving a form of adversarial attack that uses distribution shifts rather than content manipulation.
Revised: Without that enforcer, the Adversarial Player would quickly resort to stretching the whole scenes, achieving a form of adversarial attack that uses distribution shifts rather than content manipulation.

Original: It does so, e.g., by moving the camera away until objects are barely visible.
Revised: It does so, e.g., by moving the camera away until objects are barely visible.

Original: We give a few such examples in the appendix (Section A.9).
Revised: We give a few such examples in the appendix (Section A.9).

Original:  Adversarial Player
Revised:  Adversarial Player

Original: In this section, we describe our Adversarial Player in details.
Revised: In this section, we describe our Adversarial Player in details.

Original: Meaningful scene manipulations require not only generic scene understanding, but also the ability to distinguish which objects to displace and how.
Revised: Meaningful scene manipulations require not only generic scene understanding, but also the ability to distinguish which objects to displace and how.

Original: Hence, the player is a composition of a multi-modal module, which creates input representation, and a decision maker, which decide how to manipulate the scene.
Revised: Hence, the player is a composition of a multi-modal module, which creates input representation, and a decision maker, which decide how to manipulate the scene.

Original: In the following, we will describe that composition of modules, the game of scene manipulations, and the training protocol.
Revised: In the following, we will describe that composition of modules, the game of scene manipulations, and the training protocol.

Original: Figure 1 illustrates the Adversarial Player and the game between both players.
Revised: Figure 1 illustrates the Adversarial Player and the game between both players.

Original: Multi-modal module.
Revised: Multi-modal module.

Original: We have experimented with the same multi-modal modules as those in Table 1, but found out we have a better performance and the convergence rate if the Adversarial Player operates on the scene-graphs (states) instead of pixels.
Revised: We have experimented with the same multi-modal modules as those in Table 1, but found out we have a better performance and the convergence rate if the Adversarial Player operates on the scene-graphs (states) instead of pixels.

Original: For that, we use state-input variant of Relation Networks (Santoro et al., 2017).
Revised: For that, we use state-input variant of Relation Networks (Santoro et al., 2017).

Original: The model receives as the input 10 ∗ 6 object tokens, and question tokens.
Revised: The model receives as the input 10 ∗ 6 object tokens, and question tokens.

Original: Every object token represents one-out-of-ten possible objects in the scene by its attributes such as position, color, shape, material and size.
Revised: Every object token represents one-out-of-ten possible objects in the scene by its attributes such as position, color, shape, material and size.

Original: If the scene has fewer than ten objects, we use ∅ token to indicate that, which also acts as a padding.
Revised: If the scene has fewer than ten objects, we use ∅ token to indicate that, which also acts as a padding.

Original: We also have special tokens that separate questions from the objects which we add as a latent embedding, e.g., emb ( material ) + emb ( object ) .
Revised: We also have special tokens that separate questions from the objects which we add as a latent embedding, e.g., emb ( material ) + emb ( object ) .

Original: Such an input encoding is similar to our State-Input Transformer and described in Section A.3.
Revised: Such an input encoding is similar to our State-Input Transformer and described in Section A.3.

Original: The embedded vectors are given to the Relation Network (RN).
Revised: The embedded vectors are given to the Relation Network (RN).

Original: Finally, we train that network on the
Revised: Finally, we train that network on the

Original: CLEVR visual question answering task, where we achieve 97 .
Revised: CLEVR visual question answering task, where we achieve 97 .

Original: 6% on the validation set, and use the representation just after the last relational layer for the decision maker.
Revised: 6% on the validation set, and use the representation just after the last relational layer for the decision maker.

Original: Decision maker.
Revised: Decision maker.

Original: Inspired by the work on reinforcement learning (Mnih et al., 2016; Wu et al., 2017;
Revised: Inspired by the work on reinforcement learning (Mnih et al., 2016; Wu et al., 2017;

Original: Lillicrap et al., 2016; Wang et al., 2016; Schulman et al., 2015), we use an actor-critic module that acts on scenes.
Revised: Lillicrap et al., 2016; Wang et al., 2016; Schulman et al., 2015), we use an actor-critic module that acts on scenes.

Original: The actor is a general-purpose fully connected layer with ten object-speciﬁc heads.
Revised: The actor is a general-purpose fully connected layer with ten object-speciﬁc heads.

Original: Each head is randomly assigned to a unique object in the scene for its manipulation.
Revised: Each head is randomly assigned to a unique object in the scene for its manipulation.

Original: Every head produces a displacement in x and y coordinates of the corresponding object.
Revised: Every head produces a displacement in x and y coordinates of the corresponding object.

Original: Although we have initially experimented with the continuous output space we have found out the following simple strategy is more effective.
Revised: Although we have initially experimented with the continuous output space we have found out the following simple strategy is more effective.

Original: First, we discretise all the x and y coordinates into N bins each.
Revised: First, we discretise all the x and y coordinates into N bins each.

Original: Now, each head produces two N -dimensional vectors that are next projected into a probabilistic space via softmax.
Revised: Now, each head produces two N -dimensional vectors that are next projected into a probabilistic space via softmax.

Original: Next, we sample displacements in x and y axis independently from both softmax distributions.
Revised: Next, we sample displacements in x and y axis independently from both softmax distributions.

Original: Note that, even though we do not model the joint distribution explicitly due to computational reasons, both samples condition on the common head and thus are only conditionally independent of each other.
Revised: Note that, even though we do not model the joint distribution explicitly due to computational reasons, both samples condition on the common head and thus are only conditionally independent of each other.

Original: We discretise the scene where each axis has values in [ − 3 , 3] onto N = 7 bins per axis.
Revised: We discretise the scene where each axis has values in [ − 3 , 3] onto N = 7 bins per axis.

Original: Our critic is a simple three layer feed-forward network (with relu as activations) that predicts a reward score between − 1 and +1 via tanh activation ( 1 . 2 ∗ tanh for better numerical properties).
Revised: Our critic is a simple three layer feed-forward network (with relu as activations) that predicts a reward score between − 1 and +1 via tanh activation ( 1 . 2 ∗ tanh for better numerical properties).

Original: The game of scene manipulations.
Revised: The game of scene manipulations.

Original: Due to our formulation of Adversarial Player and the environment, we can benchmark various reasoning models purely in the black-box setting via a series of questions about the scene.
Revised: Due to our formulation of Adversarial Player and the environment, we can benchmark various reasoning models purely in the black-box setting via a series of questions about the scene.

Original: Adversarial Player manipulates the scene so that it is still consistent with the question-answer pair.
Revised: Adversarial Player manipulates the scene so that it is still consistent with the question-answer pair.

Original: The manipulations are applied to scene-graphs, and the resulting scene-graph is evaluated by the environment enforcers described in Section 4.
Revised: The manipulations are applied to scene-graphs, and the resulting scene-graph is evaluated by the environment enforcers described in Section 4.

Original: Invalid scenes are thus discarded.
Revised: Invalid scenes are thus discarded.

Original: In this way, we ensure the in-distribution and consistency in the scene generation.
Revised: In this way, we ensure the in-distribution and consistency in the scene generation.

Original: Original image-question pairs are fed to a Visual-QA Player that produces corresponding answers.
Revised: Original image-question pairs are fed to a Visual-QA Player that produces corresponding answers.

Original: We refer to that answers as old answers .
Revised: We refer to that answers as old answers .

Original: After the scene manipulation, new images paired with the same questions are also given to the Visual-QA Player that produces new answers .
Revised: After the scene manipulation, new images paired with the same questions are also given to the Visual-QA Player that produces new answers .

Original: We construct rewards based on old answers , new answers and ground-truth answers .
Revised: We construct rewards based on old answers , new answers and ground-truth answers .

Original: If the Adversarial Player forces the Visual-QA Player to change the answer, i.e., an old answer is different than a new answer , it gets Consistency Drop Reward ( cr ).
Revised: If the Adversarial Player forces the Visual-QA Player to change the answer, i.e., an old answer is different than a new answer , it gets Consistency Drop Reward ( cr ).

Original: If an old answer is the groundtruth answer, it gets instead Accuracy Drop Reward ( dr ).
Revised: If an old answer is the groundtruth answer, it gets instead Accuracy Drop Reward ( dr ).

Original: Both rewards differentiate between simply confusing the model and causing a drop in its performance.
Revised: Both rewards differentiate between simply confusing the model and causing a drop in its performance.

Original: If Adversarial Player produces an invalid scene it gets Invalid Scene Reward ( isr ).
Revised: If Adversarial Player produces an invalid scene it gets Invalid Scene Reward ( isr ).

Original: This reward encourages producing scenes that pass the environment enforcers tests.
Revised: This reward encourages producing scenes that pass the environment enforcers tests.

Original: Finally, if Adversarial Player does not manage to fool the model, it gets Fail Reward ( fr ).
Revised: Finally, if Adversarial Player does not manage to fool the model, it gets Fail Reward ( fr ).

Original: We use the following values: dr = 1 , cr = 0 .
Revised: We use the following values: dr = 1 , cr = 0 .

Original: 1 , fr = − 0 .
Revised: 1 , fr = − 0 .

Original: 1 , isr = − 0 .
Revised: 1 , isr = − 0 .

Original: Training algorithm.
Revised: Training algorithm.

Original: To train Adversarial Player we use the A2C algorithm with the episode length set to one as we do not need to model long-range consequences of the decision-making mechanism.
Revised: To train Adversarial Player we use the A2C algorithm with the episode length set to one as we do not need to model long-range consequences of the decision-making mechanism.

Original: Batches contain images, question, answers, programs and scene-graphs.
Revised: Batches contain images, question, answers, programs and scene-graphs.

Original: We train the Adversarial
Revised: We train the Adversarial

Original: Player for each Mini-game independently using the same architecture.
Revised: Player for each Mini-game independently using the same architecture.

Original: We experiment with the following Mini-game sizes 10 , 100 , 1000 .
Revised: We experiment with the following Mini-game sizes 10 , 100 , 1000 .

Original: All Mini-games are constructed randomly.
Revised: All Mini-games are constructed randomly.

Original: Under our discretisation scheme, the action space is N k where N is the number of bins and k is the number of objects in the scene.
Revised: Under our discretisation scheme, the action space is N k where N is the number of bins and k is the number of objects in the scene.

Original: In practice, it is up to 49 10 .
Revised: In practice, it is up to 49 10 .

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Main experimental results are presented in Table 2.
Revised: Main experimental results are presented in Table 2.

Original: First, we explain the terminology and this is followed by analysis of quantitative and qualitative results, additional experiments and limitations.
Revised: First, we explain the terminology and this is followed by analysis of quantitative and qualitative results, additional experiments and limitations.

Original: In our experiments, we consider Adversarial Player with a multi-modal module pre-trained either on states ( state-input ) or directly from pixels ( pixel-input ).
Revised: In our experiments, we consider Adversarial Player with a multi-modal module pre-trained either on states ( state-input ) or directly from pixels ( pixel-input ).

Original: As explained in Section 5, we train
Revised: As explained in Section 5, we train

Original: Adversarial Player for each Mini-game independently.
Revised: Adversarial Player for each Mini-game independently.

Original: We use different Mini-game sizes in our experiments: 10 , 100 and 1000 .
Revised: We use different Mini-game sizes in our experiments: 10 , 100 and 1000 .

Original: To obtain statistically signiﬁcant results, we run each Mini-game thirty times with randomly initialized Adversarial Players , each with different seed per trial.
Revised: To obtain statistically signiﬁcant results, we run each Mini-game thirty times with randomly initialized Adversarial Players , each with different seed per trial.

Original: We report the results that are averaged over all those runs.
Revised: We report the results that are averaged over all those runs.

Original: We compute two metrics for each Mini-game .
Revised: We compute two metrics for each Mini-game .

Original: Consistency refers to the fraction of times Adversarial Player has changed the Visual-QA Player ’s answer, regardless if that answer was correct.
Revised: Consistency refers to the fraction of times Adversarial Player has changed the Visual-QA Player ’s answer, regardless if that answer was correct.

Original: Drop refers to the fraction of times Adversarial Player has changed the correct Visual-QA Player ’s answer.
Revised: Drop refers to the fraction of times Adversarial Player has changed the correct Visual-QA Player ’s answer.

Original: We aggregate the results using the following statistics.
Revised: We aggregate the results using the following statistics.

Original: Average Accuracy averages accuracies over all Mini-games .
Revised: Average Accuracy averages accuracies over all Mini-games .

Original: Maximal Accuracy refers to the maximal accuracy over all Mini-games (so it refers to accuracy of a single Mini-game ).
Revised: Maximal Accuracy refers to the maximal accuracy over all Mini-games (so it refers to accuracy of a single Mini-game ).

Original: It is the worst (best) case from the Visual-QA Player ( Adversarial Player ) perspective.
Revised: It is the worst (best) case from the Visual-QA Player ( Adversarial Player ) perspective.

Original: We also computed p -values of T-Test, with the null hypothesis that a manipulation is unsuccessful.
Revised: We also computed p -values of T-Test, with the null hypothesis that a manipulation is unsuccessful.

Original: In almost all cases we reject the null hypothesis with small p -values.
Revised: In almost all cases we reject the null hypothesis with small p -values.

Original: We only did not ‘fool’ TbD and MDetr, and only if pixel-input Adversarial Player is considered (Section A.11, in the appendix).
Revised: We only did not ‘fool’ TbD and MDetr, and only if pixel-input Adversarial Player is considered (Section A.11, in the appendix).

Original: Quantitative results.
Revised: Quantitative results.

Original: We show our results in Table 2 for the state-input and pixel-input Adversarial Players , and for each Visual-QA Player presented in Table 1.
Revised: We show our results in Table 2 for the state-input and pixel-input Adversarial Players , and for each Visual-QA Player presented in Table 1.

Original: All the Visual-QA Players in the comparison are trained from pixels, apart form our custom State-Input Transformer architecture (Section A.3 in the appendix).
Revised: All the Visual-QA Players in the comparison are trained from pixels, apart form our custom State-Input Transformer architecture (Section A.3 in the appendix).

Original: When comparing both Adversarial Players , we can observe that the state-input Adversarial Player is signiﬁcantly more successful than the pixel-input one.
Revised: When comparing both Adversarial Players , we can observe that the state-input Adversarial Player is signiﬁcantly more successful than the pixel-input one.

Original: We hypothesize that this is the effect of having more structured and less ambiguous input information.
Revised: We hypothesize that this is the effect of having more structured and less ambiguous input information.

Original: It could also be the case that multi-modal modules trained from pixels ignore features in the same way and hence smaller discrepancy between all models operating on pixels.
Revised: It could also be the case that multi-modal modules trained from pixels ignore features in the same way and hence smaller discrepancy between all models operating on pixels.

Original: Another observation is that in the case of TbD and MDetr, pixel-input Adversarial Players are unable to solve the smallest-size
Revised: Another observation is that in the case of TbD and MDetr, pixel-input Adversarial Players are unable to solve the smallest-size

Original: Mini-games .
Revised: Mini-games .

Original: Moreover, as the size of Mini-games increases, it is becoming more difﬁcult for the
Revised: Moreover, as the size of Mini-games increases, it is becoming more difﬁcult for the

Original: Adversarial Player to successfully manipulate the model on all examples from the Mini-game .
Revised: Adversarial Player to successfully manipulate the model on all examples from the Mini-game .

Original: It is an expected behavior resembling standard adversarial perturbations (Goodfellow et al., 2015) and their universal variants (Moosavi-Dezfooli et al., 2017).
Revised: It is an expected behavior resembling standard adversarial perturbations (Goodfellow et al., 2015) and their universal variants (Moosavi-Dezfooli et al., 2017).

Original: The highest performing CLEVR models,
Revised: The highest performing CLEVR models,

Original: TbD and MDetr, are the most robust under the manipulation.
Revised: TbD and MDetr, are the most robust under the manipulation.

Original: However, they are still susceptible to the scene manipulations in some Mini-games as the Maximal Consistency metric indicates, especially if our Adversarial Player uses states as the input.
Revised: However, they are still susceptible to the scene manipulations in some Mini-games as the Maximal Consistency metric indicates, especially if our Adversarial Player uses states as the input.

Original: Section A.8 (appendix) also shows richer performance statistics.
Revised: Section A.8 (appendix) also shows richer performance statistics.

Original: Our detailed investigation shows that the models above are particularly sensitive to manipulations of scenes associated with counting or existence questions.
Revised: Our detailed investigation shows that the models above are particularly sensitive to manipulations of scenes associated with counting or existence questions.

Original: Questions of those types are often more complex, hence, increasing the likelihood of the reasoning failure at some stage (Section A.6 in the appendix).
Revised: Questions of those types are often more complex, hence, increasing the likelihood of the reasoning failure at some stage (Section A.6 in the appendix).

Original: Qualitative results.
Revised: Qualitative results.

Original: Figure 4 shows qualitative results of the scene manipulations for the three highest performing and distinctive models from Table 1; e.g., simple feed-forward or program induction models.
Revised: Figure 4 shows qualitative results of the scene manipulations for the three highest performing and distinctive models from Table 1; e.g., simple feed-forward or program induction models.

Original: For the reader’s convenience, we provide in Section A.2 (appendix) examples of CLEVR object attributes.
Revised: For the reader’s convenience, we provide in Section A.2 (appendix) examples of CLEVR object attributes.

Original: We can observe that the scene manipulations are surprisingly minimal and are semantically meaningless , i.e., they should not ‘fool’ human players.
Revised: We can observe that the scene manipulations are surprisingly minimal and are semantically meaningless , i.e., they should not ‘fool’ human players.

Original: Saturation of Adversarial Player .
Revised: Saturation of Adversarial Player .

Original: Training an Adversarial Player on larger Mini-games become an increasingly more difﬁcult task as the player needs to operate successfully on more diverse examples.
Revised: Training an Adversarial Player on larger Mini-games become an increasingly more difﬁcult task as the player needs to operate successfully on more diverse examples.

Original: Figure 2 shows the performance of the Adversarial Player against Mini-game sizes, where
Revised: Figure 2 shows the performance of the Adversarial Player against Mini-game sizes, where

Original: FiLM, TbD, MDetr and Input-State Transformer (Trans.) are chosen as Visual-QA Players .
Revised: FiLM, TbD, MDetr and Input-State Transformer (Trans.) are chosen as Visual-QA Players .

Original: We can see that the performance of Adversarial Player saturates around Mini-games of size 10k, which we call the Critical mini-game size .
Revised: We can see that the performance of Adversarial Player saturates around Mini-games of size 10k, which we call the Critical mini-game size .

Original: We use two trials for our experiments, and the Accuracy Drop as a metric.
Revised: We use two trials for our experiments, and the Accuracy Drop as a metric.

Original: It measures the drop in accuracy of a Visual-QA Player .
Revised: It measures the drop in accuracy of a Visual-QA Player .

Original: Adversarial training.
Revised: Adversarial training.

Original: In initial experiments, we have adapted a widely used adversarial training (Goodfellow et al., 2018), where we include manipulated scenes in the training protocol of a
Revised: In initial experiments, we have adapted a widely used adversarial training (Goodfellow et al., 2018), where we include manipulated scenes in the training protocol of a

Original: Visual-QA Player .
Revised: Visual-QA Player .

Original: We use pixel-input RN (Santoro et al., 2017) as the Visual-QA Player .
Revised: We use pixel-input RN (Santoro et al., 2017) as the Visual-QA Player .

Original: Using this method, we have obtained only marginal improvements in robustness at the cost of a slight performance degradation on the original CLEVR dataset.
Revised: Using this method, we have obtained only marginal improvements in robustness at the cost of a slight performance degradation on the original CLEVR dataset.

Original: Limitations of data-driven CLEVR models.
Revised: Limitations of data-driven CLEVR models.

Original: If we can’t achieve robustness under scene manipulations with the adversarial training, what fraction of all data is sufﬁcient then?
Revised: If we can’t achieve robustness under scene manipulations with the adversarial training, what fraction of all data is sufﬁcient then?

Original: Intuitively, if the network has enough capacity, and was trained on all possible data points or its manipulations, it should rely on a look-up strategy to solve the problem accurately.
Revised: Intuitively, if the network has enough capacity, and was trained on all possible data points or its manipulations, it should rely on a look-up strategy to solve the problem accurately.

Original: We propose the following experiment to address such a question.
Revised: We propose the following experiment to address such a question.

Original: We create a dataset containing three objects in six different starting positions and seven templated questions.
Revised: We create a dataset containing three objects in six different starting positions and seven templated questions.

Original: We consider only object displacements on the grid 7 × 7 , and thus 49 3 = 117649 combinations are possible.
Revised: We consider only object displacements on the grid 7 × 7 , and thus 49 3 = 117649 combinations are possible.

Original: We use RN (Santoro et al., 2017) as a pixel-input Visual-QA Player .
Revised: We use a few architectures from Table 1 as a pixel-input Visual-QA Player .

Original: We train such a model on X % data and next evaluate on (100 − X )% remaining, unseen data.
Revised: We train them on X % data and next evaluate on (100 − X )% remaining, unseen data.

Original: We have conducted ten trials by forming X % training data randomly.
Revised: We have conducted ten trials by forming X % training data randomly.

Original: Figure 3 shows that when trained on 10% of data, the model becomes resistant to 40% of unseen manipulations.
Revised: Figureshows that when trained on 10% of data, the model becomes resistant to 40% of unseen manipulations.

Original: The robustness of the model increases linearly, saturating at roughly 97% accuracy after observing almost all data ( 90% ).
Revised: The robustness of the model increases linearly, saturating at roughly 97% accuracy after observing almost all data ( 90% ).

Original: The behaviour of all tested models is similar with slightly better overall performance of TbD. The results indicate that a lot of training examples are needed to cover most conﬁgurations of our task in the absence of stronger inductive biases.
Revised: The behaviour of all tested models is similar with slightly better overall performance of TbD. The results indicate that a lot of training examples are needed to cover most conﬁgurations of our task in the absence of stronger inductive biases.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: We present a novel approach to benchmarking the reasoning capabilities of various visual models by formulating a game between two players.
Revised: We present a novel approach to benchmarking the reasoning capabilities of various visual models by formulating a game between two players.

Original: Both players are independent and can only communicate through a common interface: an environment and question-answer pairs.
Revised: Both players are independent and can only communicate through a common interface: an environment and question-answer pairs.

Original: One player, named Adversarial Player , is trying to manipulate the environment so that the second player, named Visual-QA
Revised: One player, named Adversarial Player , is trying to manipulate the environment so that the second player, named Visual-QA

Original: Player , can’t answer the given question anymore.
Revised: Player , can’t answer the given question anymore.

Original: To test our framework, we run our game against a few popular CLEVR models that achieve solid results on the CLEVR dataset; often surpassing the human baseline.
Revised: To test our framework, we run our game against a few popular CLEVR models that achieve solid results on the CLEVR dataset; often surpassing the human baseline.

Original: We show that such models are susceptible to scene manipulations, and their performance may degrade signiﬁcantly.
Revised: We show that such models are susceptible to scene manipulations, and their performance may degrade signiﬁcantly.

Original: Moreover, we also conducted a controlled study on the generalization capabilities of such models.
Revised: Moreover, we also conducted a controlled study on the generalization capabilities of such models.

Original: We did so by creating a synthetic and simpliﬁed dataset of all possible scene manipulations.
Revised: We did so by creating a synthetic and simpliﬁed dataset of all possible scene manipulations.

Original: We show that the existing models are not particularly efﬁcient learners and require signiﬁcantly more than 50% of all data to generalize to all possible scene conﬁgurations.
Revised: We show that the existing models are not particularly efﬁcient learners and require signiﬁcantly more than 50% of all data to generalize to all possible scene conﬁgurations.

Original: One possible direction to increase efﬁciency could be the use of stronger inductive biases.
Revised: One possible direction to increase efﬁciency could be the use of stronger inductive biases.

Original: Another to change a training paradigm that is more interactive.
Revised: Another to change a training paradigm that is more interactive.

Original: However, the environment is synthetic, and it could be that learning from real-world and multi-modal data is somewhat more efﬁcient.
Revised: However, the environment is synthetic, and it could be that learning from real-world and multi-modal data is somewhat more efﬁcient.

Original: Finally, we believe that melding two-player games with the visual question answering framework is both natural and beneﬁcial for testing the reasoning capabilities of various models; and potentially can also be generalized to other settings.
Revised: Finally, we believe that melding two-player games with the visual question answering framework is both natural and beneﬁcial for testing the reasoning capabilities of various models; and potentially can also be generalized to other settings.

Original: In the long term, we plan to test such models through web interfaces.
Revised: In the long term, we plan to test such models through web interfaces.

Original: [Table]
Revised: [Table]

Original: What is the color
Revised: What is the color

Original: How many other objects
Revised: How many other objects

Original: There is a tiny block
Revised: There is a tiny block

Original: How many other things are of the block that is are the same shape as the behind the purple cylinder; the same shape as the big left of the big big gray matte object?
Revised: How many other things are of the block that is are the same shape as the behind the purple cylinder; the same shape as the big left of the big big gray matte object?

Original: what number of red matte metallic thing?
Revised: what number of red matte metallic thing?

Original: blue matte ball?
Revised: blue matte ball?

Original: spheres are in front of it?
Revised: spheres are in front of it?

Original: F i L M - B e f o r e
Revised: F i L M - B e f o r e

Original: Purple
Revised: Purple

Original: F i L M - A f t e r
Revised: F i L M - A f t e r

Original: Brown
Revised: Brown

Original: What number of tiny
Revised: What number of tiny

Original: Are there any other things
Revised: Are there any other things

Original: Is the shape of the brown rubber
Revised: Is the shape of the brown rubber

Original: Is there a big brown object matte cylinders that have the same material as object the same as the thing that of the same shape as the are there?
Revised: Is there a big brown object matte cylinders that have the same material as object the same as the thing that of the same shape as the are there?

Original: the big yellow ball?
Revised: the big yellow ball?

Original: is left of the big object?
Revised: is left of the big object?

Original: green thing?
Revised: green thing?

Original: T b D - B
Revised: T b D - B

Original: e
Revised: e

Original: f
Revised: f

Original: o r e
Revised: o r e

Original: Yes
Revised: Yes

Original: Yes No
Revised: Yes Yes

Original: T b D - A f t e r
Revised: T b D - A f t e r

Original: No
Revised: No

Original: No Yes
Revised: No No

Original: The red thing that is
Revised: The red thing that is

Original: Is there anything else
Revised: Is there anything else

Original: Do the object behind the
Revised: Do the object behind the

Original: What number of things are large the same material as the that has the same material as big matte sphere and the metallic blocks on the brown cube is what size?
Revised: What number of things are large the same material as the that has the same material as big matte sphere and the metallic blocks on the brown cube is what size?

Original: the large green cube?
Revised: the large green cube?

Original: thing on the right side of left side of the brown metallic the red thing have the same material?
Revised: thing on the right side of left side of the brown metallic the red thing have the same material?

Original: thing or big cyan shiny objects?
Revised: thing or big cyan shiny objects?

Original: M D e t r - B e f o r e
Revised: M D e t r - B e f o r e

Original: Large
Revised: Large

Original: No
Revised: No

Original: Yes
Revised: Yes

Original: M D e t r - A f t e r
Revised: M D e t r - A f t e r

Original: Small
Revised: Small

Original: Yes
Revised: Yes

Original: No
Revised: No

Original: A A PPENDIX A. URL S OF CLEVR M ODELS
Revised: A A PPENDIX A. URL S OF CLEVR M ODELS

Original: Table 3 shows the URLs to models used in our investigations (also Table 1 in the main paper).
Revised: Table 3 shows the URLs to models used in our investigations (also Table 1 in the main paper).

Original: We also report if we re-trained a model from scratch (type Architecture ) or used already trained models (type Model ).
Revised: We also report if we re-trained a model from scratch (type Architecture ) or used already trained models (type Model ).

Original: Please note that the latter type proves that our testing procedure is fully black-box.
Revised: Please note that the latter type proves that our testing procedure is fully black-box.

Original: [Table]
Revised: [Table]

Original: A. O BJECT C ATEGORIES
Revised: A. O BJECT C ATEGORIES

Original: Figure 5 shows a made-up image that contains all available CLEVR object categories: shapes, sizes and materials.
Revised: Figure 5 shows a made-up image that contains all available CLEVR object categories: shapes, sizes and materials.

Original: Small objects are rendered at approximately 40% of the size of their large counterparts.
Revised: Small objects are rendered at approximately 40% of the size of their large counterparts.

Original: Metalic materials can be identiﬁed by the high albedo property and ‘shininess’ of the object.
Revised: Metalic materials can be identiﬁed by the high albedo property and ‘shininess’ of the object.

Original: We provide it as the reference point for a reader to make it easier to compare our qualitative results.
Revised: We provide it as the reference point for a reader to make it easier to compare our qualitative results.

Original: A. S TATE -I NPUT T RANSFORMER
Revised: A. S TATE -I NPUT T RANSFORMER

Original: To test our two-player game against a state-input Visual-QA Player , we have designed a transformerbased architecture Vaswani et al.
Revised: To test our two-player game against a state-input Visual-QA Player , we have designed a transformerbased architecture Vaswani et al.

Original: (2017) that receives six types of input features – object sizes, object shapes, object materials, object colors, object positions, and question tokens – and uses a cross-modal attention mechanism.
Revised: (2017) that receives six types of input features – object sizes, object shapes, object materials, object colors, object positions, and question tokens – and uses a cross-modal attention mechanism.

Original: In that mechanism, queries from one modality attend to keys and values from the another modality.
Revised: In that mechanism, queries from one modality attend to keys and values from the another modality.

Original: Since each data point contains a variable-length inputs describing the all objects in the scene as well as question tokens that compose the question itself, we do padding with a special token ∅ to the maximal input length.
Revised: Since each data point contains a variable-length inputs describing the all objects in the scene as well as question tokens that compose the question itself, we do padding with a special token ∅ to the maximal input length.

Original: We set the maximal length for the objects tokens to be 10 ∗ 6 = 60 and 50 for the question tokens.
Revised: We set the maximal length for the objects tokens to be 10 ∗ 6 = 60 and 50 for the question tokens.

Original: All in all, we have 110 input tokens.
Revised: All in all, we have 110 input tokens.

Original: Each token type is projected into a different embedding space and a learnable type embedding is added to it, separately for object and question tokens.
Revised: Each token type is projected into a different embedding space and a learnable type embedding is added to it, separately for object and question tokens.

Original: For instance, emb ( material )+ emb ( object ) .
Revised: For instance, emb ( material )+ emb ( object ) .

Original: Furthermore, three special learnable embeddings are used as an additional input.
Revised: Furthermore, three special learnable embeddings are used as an additional input.

Original: We use them as queries to reduce the overall computational costs of the cross-attention mechanism.
Revised: We use them as queries to reduce the overall computational costs of the cross-attention mechanism.

Original: The same mechanism is also used in Perceiver (Jaegle et al., 2021).
Revised: The same mechanism is also used in Perceiver (Jaegle et al., 2021).

Original: The input tokens (concatenated object and question tokens) form keys and values.
Revised: The input tokens (concatenated object and question tokens) form keys and values.

Original: In every transformer block, we apply cross-attention between all input embeddings and those three special tokens.
Revised: In every transformer block, we apply cross-attention between all input embeddings and those three special tokens.

Original: This is repeated ﬁve times.
Revised: This is repeated ﬁve times.

Original: As the ﬁnal block, a feed-forward network (classiﬁer) receives as inputs the three latent token and outputs answer probabilities.
Revised: As the ﬁnal block, a feed-forward network (classiﬁer) receives as inputs the three latent token and outputs answer probabilities.

Original: We show that architecture in Figure 6.
Revised: We show that architecture in Figure 6.

Original: A. M ANIPULATIONS OF State-Input Visual-QA Player
Revised: A. M ANIPULATIONS OF State-Input Visual-QA Player

Original: Figures 7 and 8 show manipulations of Visual-QA Player that is trained to take states instead of pixels as the input.
Revised: Figures 7 and 8 show manipulations of Visual-QA Player that is trained to take states instead of pixels as the input.

Original: Such a state-input Visual-QA Player receives a direct overview of the scene, bypassing any need for any image renderings.
Revised: Such a state-input Visual-QA Player receives a direct overview of the scene, bypassing any need for any image renderings.

Original: We use State-Input Transformer (Section A.3) as the multi-modal component of that player.
Revised: We use State-Input Transformer (Section A.3) as the multi-modal component of that player.

Original: Since such a model gets the perfect visual information as the input, it makes it more robust under scene manipulations.
Revised: Since such a model gets the perfect visual information as the input, it makes it more robust under scene manipulations.

Original: As a consequence, our Adversarial Player tend to manipulate the scene so that objects are placed closely together.
Revised: As a consequence, our Adversarial Player tend to manipulate the scene so that objects are placed closely together.

Original: Often, simpler questions, which require fewer stages of reasoning, are safer to Visual-QA Player .
Revised: Often, simpler questions, which require fewer stages of reasoning, are safer to Visual-QA Player .

Original: Let us consider the last example in Figure 8.
Revised: Let us consider the last example in Figure 8.

Original: If the question was What is the material of the cyan cube? , the model would answer rightfully metal .
Revised: If the question was What is the material of the cyan cube? , the model would answer rightfully metal .

Original: Here, Adversarial Player ‘breaks’ the spatial reasoning step by creating ambiguity between the cubes and thus a wrong answer is derived.
Revised: Here, Adversarial Player ‘breaks’ the spatial reasoning step by creating ambiguity between the cubes and thus a wrong answer is derived.

Original: Are there an equal number of small things that are left of the gray object and large cyan matte balls?
Revised: Are there an equal number of small things that are left of the gray object and large cyan matte balls?

Original: S t a t e - B e f o r e
Revised: S t a t e - B e f o r e

Original: False
Revised: False

Original: S t a t e - A f t e r
Revised: S t a t e - A f t e r

Original: True
Revised: True

Original: Is the number of big blocks that are in front of the brown metallic cylinder the same as the number of tiny cylinders on the right side of the rubber cube?
Revised: Is the number of big blocks that are in front of the brown metallic cylinder the same as the number of tiny cylinders on the right side of the rubber cube?

Original: S t a t e - B e f o r e
Revised: S t a t e - B e f o r e

Original: False
Revised: False

Original: S t a t e - A f t e r
Revised: S t a t e - A f t e r

Original: True
Revised: True

Original: How many other things are both behind the purple rubber sphere and to the right of the gray cylinder?
Revised: How many other things are both behind the purple rubber sphere and to the right of the gray cylinder?

Original: S t a t e - B e f o r e
Revised: S t a t e - B e f o r e

Original: S t a t e - A f t e r
Revised: S t a t e - A f t e r

Original: What is the material of the cube that is behind the rubber cube?
Revised: What is the material of the cube that is behind the rubber cube?

Original: S t a t e - B e f o r e
Revised: S t a t e - B e f o r e

Original: Metal
Revised: Metal

Original: S t a t e - A f t e r
Revised: S t a t e - A f t e r

Original: Rubber
Revised: Rubber

Original: A. A LGORITHMS
Revised: A. A LGORITHMS

Original: We show pseudo-algorithms that we use to (Algorithm 1) calculate rewards, (Algorithm 2) train
Revised: We show pseudo-algorithms that we use to (Algorithm 1) calculate rewards, (Algorithm 2) train

Original: Adversarial Player , (Algorithm 3) play a game, (Algorithm 4) create a defence mechanism.
Revised: Adversarial Player , (Algorithm 3) play a game, (Algorithm ?? ) create a defence mechanism.

Original: dr
Revised: dr

Original: Drop Reward 5: cr : Consistency Reward 6: fr : Fail Reward 7: reward ←8: if new answer (cid:54) = old answer then 9: if old answer (cid:54) =
Revised: Drop Reward 5: cr : Consistency Reward 6: fr : Fail Reward 7: reward ←8: if new answer (cid:54) = old answer then 9: if old answer (cid:54) =

Original: gt answer then 10: reward = reward
Revised: gt answer then 10: reward = reward

Original: + cr 11: else 12: reward = reward + dr 13: end if 14: else 15: reward = reward + fr 16: end if 17: return reward
Revised: + cr 11: else 12: reward = reward + dr 13: end if 14: else 15: reward = reward + fr 16: end if 17: return reward

Original: 1: M : Minigames 2: V a : Visual Agent, Fe : Feature Extractor 3: A : Actor, C : Critic 4:
Revised: 1: M : Minigames 2: V a : Visual Agent, Fe : Feature Extractor 3: A : Actor, C : Critic 4:

Original: Qre : Question Relevance Enforcer, Sce : Scene Constraint Enforcer 5: for batch in M do 6: rewards, state values, logprobs ← DataGame ( M, V a, Fe, A, C, Qre, Sce ) 7: n ← | rewards | 8: advantage ← rewards − state values 9: ploss ← − logprobs × advantage 10: vloss ← ( state values − stop grad ( rewards )) 2 /n 11: loss ← ploss + vloss 12: backprop ( loss, A, C ) 13: end for 14: return
Revised: Qre : Question Relevance Enforcer, Sce : Scene Constraint Enforcer 5: for batch in M do 6: rewards, state values, logprobs ← DataGame ( M, V a, Fe, A, C, Qre, Sce ) 7: n ← | rewards | 8: advantage ← rewards − state values 9: ploss ← − logprobs × advantage 10: vloss ← ( state values − stop grad ( rewards )) 2 /n 11: loss ← ploss + vloss 12: backprop ( loss, A, C ) 13: end for 14: return

Original: 1: M : Mini-game 2: P VQA : Visual Agent, S k : Multi-modal Backbone 3: A : Actor, C : Critic 4:
Revised: 1: M : Mini-game 2: P VQA : Visual Agent, S k : Multi-modal Backbone 3: A : Actor, C : Critic 4:

Original: Qre : Question Relevance Enforcer, Sce : Scene Constraint Enforcer 5: isr : Invalid Scene Reward 6: for batch in M do 7: rewards ← {} 8: state values ← {} 9: logprobs ← {} 10: for ( image, scene, question, program, gt answer ) in batch do 11: old answer ← P VQA ( image, question ) 12: extracted features ← S k
Revised: Qre : Question Relevance Enforcer, Sce : Scene Constraint Enforcer 5: isr : Invalid Scene Reward 6: for batch in M do 7: rewards ← {} 8: state values ← {} 9: logprobs ← {} 10: for ( image, scene, question, program, gt answer ) in batch do 11: old answer ← P VQA ( image, question ) 12: extracted features ← S k

Original: ( scene, question ) 13: dx probs, dy probs ← A ( extracted features ) 14:
Revised: ( scene, question ) 13: dx probs, dy probs ← A ( extracted features ) 14:

Original: CatX ← Categorical ( dx probs ) 15:
Revised: CatX ← Categorical ( dx probs ) 15:

Original: CatY ← Categorical ( dy probs ) 16: state value ← C ( extracted features ) 17: dx ← sample
Revised: CatY ← Categorical ( dy probs ) 16: state value ← C ( extracted features ) 17: dx ← sample

Original: CatX 18: dy ← sample CatY 19: new scene ← Perturbate ( scene, dx, dy ) 20: check scene ← Sce ( new scene ) 21: check question ← Qre ( new scene, program, gt answer ) 22: if check scene ∧ check question =
Revised: CatX 18: dy ← sample CatY 19: new scene ← Perturbate ( scene, dx, dy ) 20: check scene ← Sce ( new scene ) 21: check question ← Qre ( new scene, program, gt answer ) 22: if check scene ∧ check question =

Original: True then 23: new image ← Render ( new scene ) 24: new answer ← P VQA ( new image, question ) 25: rewards ← rewards || CalcRewards ( new answer, old answer, gt answer ) 26: else 27: rewards ← rewards || isr 28: end if 29: state values ← state values
Revised: True then 23: new image ← Render ( new scene ) 24: new answer ← P VQA ( new image, question ) 25: rewards ← rewards || CalcRewards ( new answer, old answer, gt answer ) 26: else 27: rewards ← rewards || isr 28: end if 29: state values ← state values

Original: || state value 30: logprob ← logprob || log ( p ( dx | CatX )))
Revised: || state value 30: logprob ← logprob || log ( p ( dx | CatX )))

Original: + log ( p ( dy | CatY ))) 31: end for 32: end for 33: return rewards, state values, logprobs1: N Defense Rounds M
Revised: + log ( p ( dy | CatY ))) 31: end for 32: end for 33: return rewards, state values, logprobs

Original: 8: for epoch do 9: batch solved =10: for batch in M finetune do 11: for ( image, scene, question, program, gt answer ) in batch do 12: predictions ← {} 13: correct ← {} 14: new i mage ← Agent finetune ( scene, question ) 15: if MixExamples =

Original: True then 16: predictions ← predictions ||

Original: V a ( image, question ) 17: correct ← correct

Original: || gt answer 18: end if 19: predictions ← predictions

Original: ||

Original: V a ( new i mage, question ) 20: correct ← correct

Original: || gt answer 21: loss =

Original: CrossEntropy ( predictions, correct ) 22: batch solved + = Accuracy ( predictions, correct ) 23: backprop ( loss, V a ) 24: end for 25: end for 26: if batch solved/len ( M finetune )

Original: = 100 then 27: break 28: end if 29: end for 30: return V a

Original: A. S ENSITIVITY TO Q UESTION T YPES
Revised: A. S ENSITIVITY TO Q UESTION T YPES

Original: Figure 9 shows the susceptibility of CLEVR models to scene manipulations of Adversarial Player .
Revised: Figure 9 shows the susceptibility of CLEVR models to scene manipulations of Adversarial Player .

Original: We observe that especially counting and existence questions are the ‘back-doors‘ for our scene manipulations.
Revised: We observe that especially counting and existence questions are the ‘back-doors‘ for our scene manipulations.

Original: Questions of those two types typically involves multiple steps of reasoning.
Revised: Questions of those two types typically involves multiple steps of reasoning.

Original: Adversarial
Revised: Adversarial

Original: Player ’s scene manipulations make them more likely that reasoning will fail at some stage.
Revised: Player ’s scene manipulations make them more likely that reasoning will fail at some stage.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Figure 10 shows convergence plots of Adversarial Player during training against FiLM, TbD and
Revised: Figure 10 shows convergence plots of Adversarial Player during training against FiLM, TbD and

Original: MDetr.
Revised: MDetr.

Original: We use different initialization seeds and 30 trials.
Revised: We use different initialization seeds and 30 trials.

Original: Adversarial Player is trained either from states or pixels .
Revised: Adversarial Player is trained either from states or pixels .

Original: The former yields higher performance and sometimes better convergence.
Revised: The former yields higher performance and sometimes better convergence.

Original: We use the Drop metric that measures the accuracy drop after the Visual-QA Player is manipulated.
Revised: We use the Drop metric that measures the accuracy drop after the Visual-QA Player is manipulated.

Original: The higher Drop , the more successful manipulations are.
Revised: The higher Drop , the more successful manipulations are.

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: Figures 11 and 12 show the distribution of the Adversarial Player performance, where we show how often (y-axis)
Revised: Figures 11 and 12 show the distribution of the Adversarial Player performance, where we show how often (y-axis)

Original: the given Drop score is achieved (x-axis).
Revised: the given Drop score is achieved (x-axis).

Original: The Drop metric measures the accuracy drop after the Visual-QA Player is manipulated.
Revised: The Drop metric measures the accuracy drop after the Visual-QA Player is manipulated.

Original: The higher Drop , the more successful manipulations are.
Revised: The higher Drop , the more successful manipulations are.

Original: Larger Mini-games lead to more consistent (narrower distributions) but lower performance.
Revised: Larger Mini-games lead to more consistent (narrower distributions) but lower performance.

Original: This behaviour suggests that scene diversity present in larger Mini-games lead to more stable scene manipulations.
Revised: This behaviour suggests that scene diversity present in larger Mini-games lead to more stable scene manipulations.

Original: However, they do so at the cost of increasing the complexity of the optimization problem.
Revised: However, they do so at the cost of increasing the complexity of the optimization problem.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: t r
Revised: t r

Original: A. O UT - OF -D ISTRIBUTION M ANIPULATIONS
Revised: A. O UT - OF -D ISTRIBUTION M ANIPULATIONS

Original: If we do not provide extra constraints on the scene generation process with our environment enforcers, Adversarial Player may ﬁnd an easy manipulation that changes the camera pose or location, effectively ‘zooming out’ the whole scene.
Revised: If we do not provide extra constraints on the scene generation process with our environment enforcers, Adversarial Player may ﬁnd an easy manipulation that changes the camera pose or location, effectively ‘zooming out’ the whole scene.

Original: It does so by stretching all the object coordinates in the scene.
Revised: It does so by stretching all the object coordinates in the scene.

Original: Figure 13 illustrates that.
Revised: Figure 13 illustrates that.

Original: For instance, large objects look much smaller than typical examples.
Revised: For instance, large objects look much smaller than typical examples.

Original: As such constructed scenes are quite different from the ones that Visual-QA Player has observed during training, we categorize such manipulations to be out-of-distribution .
Revised: As such constructed scenes are quite different from the ones that Visual-QA Player has observed during training, we categorize such manipulations to be out-of-distribution .

Original: Note that, due to our in-distribution environment enforcers, such scenes are prohibited in our pipeline.
Revised: Note that, due to our in-distribution environment enforcers, such scenes are prohibited in our pipeline.

Original: [Figure]
Revised: [Figure]

Original: A.
Revised: A.

Original: A DDITIONAL Q UALITATIVE R ESULTS
Revised: A DDITIONAL Q UALITATIVE R ESULTS

Original: In Figures 14-23, we provide more qualitative results.
Revised: In Figures 14-23, we provide more qualitative results.

Original: Is the number of small cubes that
Revised: Is the number of small cubes that

Original: What is the color of the metallic thing are right of the blue rubber block greater that is on the right side of than the number of small balls that the shiny thing behind are in front of the cyan object?
Revised: What is the color of the metallic thing are right of the blue rubber block greater that is on the right side of than the number of small balls that the shiny thing behind are in front of the cyan object?

Original: the large gray object?
Revised: the large gray object?

Original: S
Revised: S

Original: ANB e f o r e
Revised: ANB e f o r e

Original: Yes
Revised: Yes

Original: Gray
Revised: Gray

Original: S A
Revised: S A

Original: NA f t e r
Revised: NA f t e r

Original: No
Revised: No

Original: Green
Revised: Green

Original: There is a cube on the left What
Revised: There is a cube on the left What

Original: shape
Revised: shape

Original: is
Revised: is

Original: the thing that
Revised: the thing that

Original: side of the brown object is in front of the small yellow that is behind the red ball;
Revised: side of the brown object is in front of the small yellow that is behind the red ball;

Original: cylinder
Revised: cylinder

Original: and to the right what size is it?
Revised: and to the right what size is it?

Original: of the rubber thing?
Revised: of the rubber thing?

Original: S
Revised: S

Original: ANB e f o r e
Revised: ANB e f o r e

Original: Large
Revised: Large

Original: Cylinder
Revised: Cylinder

Original: S
Revised: S

Original: ANA f t e r
Revised: ANA f t e r

Original: Small
Revised: Small

Original: Sphere
Revised: Sphere

Original: How big is the shiny cylinder There is a thing that is the that is behind the same color as the cube; rubber cylinder?
Revised: How big is the shiny cylinder There is a thing that is the that is behind the same color as the cube; rubber cylinder?

Original: what is it
Revised: what is it

Original: made
Revised: made

Original: of?
Revised: of?

Original: I E
Revised: I E

Original: PB e f o r e
Revised: PB e f o r e

Original: Large
Revised: Large

Original: Rubber
Revised: Rubber

Original: I E
Revised: I E

Original: PA f t e r
Revised: PA f t e r

Original: Small
Revised: Small

Original: Metal
Revised: Metal

Original: Does the cyan metal object that
Revised: Does the cyan metal object that

Original: What is the big thing that is behind is behind the tiny gray block have the large red shiny sphere the same size as the block behind made of?
Revised: What is the big thing that is behind is behind the tiny gray block have the large red shiny sphere the same size as the block behind made of?

Original: the small cyan metallic objec?
Revised: the small cyan metallic objec?

Original: I E
Revised: I E

Original: PB e f o r e
Revised: PB e f o r e

Original: No
Revised: No

Original: Rubber
Revised: Rubber

Original: I E
Revised: I E

Original: PA f t e r
Revised: PA f t e r

Original: Yes
Revised: Yes

Original: Metal
Revised: Metal

Original: There is a metallic object; How many other small shiny objects does it have the same size as are the same shape as the the rubber block right of the red object?
Revised: There is a metallic object; How many other small shiny objects does it have the same size as are the same shape as the the rubber block right of the red object?

Original: tiny shiny cube?
Revised: tiny shiny cube?

Original: R
Revised: R

Original: NB e f o r e
Revised: NB e f o r e

Original: No
Revised: No

Original: R
Revised: R

Original: NA f t e r
Revised: NA f t e r

Original: Yes
Revised: Yes

Original: Are there any other things Are there any other things that are the same shape as the that are the same size big metallic object?
Revised: Are there any other things Are there any other things that are the same shape as the that are the same size big metallic object?

Original: as the ball?
Revised: as the ball?

Original: R
Revised: R

Original: NB e f o r e
Revised: NB e f o r e

Original: No
Revised: No

Original: No
Revised: [Figure]

Original: R
Revised: R

Original: NA f t e r
Revised: NA f t e r

Original: [Figure]
Revised: [Figure]

Original: How many objects are there
Revised: How many objects are there

Original: There is another object that is the of the same size as same size as the blue the brown object?
Revised: There is another object that is the of the same size as same size as the blue the brown object?

Original: rubber object; What is it made of?
Revised: rubber object; What is it made of?

Original: M
Revised: M

Original: D e t rB e f o r e
Revised: D e t rB e f o r e

Original: Rubber
Revised: Rubber

Original: M
Revised: M

Original: D e t rA f t e r
Revised: D e t rA f t e r

Original: Metal
Revised: Metal

Original: What number of other objects What number of metal objects are the same color are small objects or as the block ?
Revised: What number of other objects What number of metal objects are the same color are small objects or as the block ?

Original: big cyan objects?
Revised: big cyan objects?

Original: M
Revised: M

Original: D e t rB e f o r e
Revised: D e t rB e f o r e

Original: M
Revised: M

Original: D e t rA f t e r
Revised: D e t rA f t e r

Original: Is there an equal number of cubes
Revised: Is there an equal number of cubes

Original: Are there less matte objects in front of the large shiny thing on the left of the green block behind the cyan block than metal things?
Revised: Are there less matte objects in front of the large shiny thing on the left of the green block behind the cyan block than metal things?

Original: and behind the yellow cube?
Revised: and behind the yellow cube?

Original: T b
Revised: T b

Original: DB e f o r e
Revised: DB e f o r e

Original: No
Revised: No

Original: Yes
Revised: Yes

Original: T b
Revised: T b

Original: DA f t e r
Revised: DA f t e r

Original: Yes
Revised: Yes

Original: No
Revised: No

Original: There is an object on the left Are there an equal number of tiny matte blocks of the yellow sphere in front of that are on the right side of the cyan metal thing; the matte cylinder and tiny What shape is it?
Revised: There is an object on the left Are there an equal number of tiny matte blocks of the yellow sphere in front of that are on the right side of the cyan metal thing; the matte cylinder and tiny What shape is it?

Original: cyan rubber cylinders?
Revised: cyan rubber cylinders?

Original: T b
Revised: T b

Original: DB e f o r e
Revised: DB e f o r e

Original: Cube
Revised: Cube

Original: Yes
Revised: Yes

Original: T b
Revised: T b

Original: DA f t e r
Revised: DA f t e r

Original: Sphere
Revised: Sphere

Original: No
Revised: No

Original: A. Q UANTITATIVE R ESULTS WITH p - VALUES
Revised: A. Q UANTITATIVE R ESULTS WITH p - VALUES

Original: We extend the quantitative results of the main paper by also reporting p -values of the one-sample hypothesis testing.
Revised: We extend the quantitative results of the main paper by also reporting p -values of the one-sample hypothesis testing.

Original: Here, we also use the Accuracy Drop metric that measures the accuracy drop after the Visual-QA Player is manipulated, and the Consistency Drop that measures how many times the manipulated Visual-QA Player changes its answer, independently if that is a correct or wrong answer.
Revised: Here, we also use the Accuracy Drop metric that measures the accuracy drop after the Visual-QA Player is manipulated, and the Consistency Drop that measures how many times the manipulated Visual-QA Player changes its answer, independently if that is a correct or wrong answer.

Original: The higher Drop Accuracy or Consistency Drop , the more successful manipulations are.
Revised: The higher Drop Accuracy or Consistency Drop , the more successful manipulations are.

Original: Note that we use the same metrics as rewards to our Adversarial Player .
Revised: Note that we use the same metrics as rewards to our Adversarial Player .

Original: We use T-Test (Helmert, 1876;
Revised: We use T-Test (Helmert, 1876;

Original: L¨uroth, 1876) for each metric all the runs.
Revised: L¨uroth, 1876) for each metric all the runs.

Original: Our null hypothesis is that the population mean of all the games is zero, indicating that a manipulation is unsuccessful, and the results of our Adversarial
Revised: Our null hypothesis is that the population mean of all the games is zero, indicating that a manipulation is unsuccessful, and the results of our Adversarial

Original: Players are extreme cases of a good performance, i.e, H 0 : µ l = 0 , l ∈ [ Drop , Consistency ] .
Revised: Players are extreme cases of a good performance, i.e, H 0 : µ l = 0 , l ∈ [ Drop , Consistency ] .

Original: As an alternative hypothesis, we assume that the population mean is greater than zero, i.e., H A : µ l > 0 , l ∈ [ Drop , Consistency ] .
Revised: As an alternative hypothesis, we assume that the population mean is greater than zero, i.e., H A : µ l > 0 , l ∈ [ Drop , Consistency ] .

Original: The same setup stands for both the Consistency and the Drop metric.
Revised: The same setup stands for both the Consistency and the Drop metric.

Original: Tables 4 and 5 show the full table with p -values computed.
Revised: Tables 4 and 5 show the full table with p -values computed.

Original: Note that only in two cases, Adversarial
Revised: Note that only in two cases, Adversarial

Original: Player does not manipulate the scene convincingly (TbD and Mdetr for the pixel-input Adversarial
Revised: Player does not manipulate the scene convincingly (TbD and Mdetr for the pixel-input Adversarial

Original: Player ).
Revised: Player ).

Revised: [Table]

Revised: [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Revised: B A PPENDIX B B. M ODEL PERFORMANCE ON MAXIMAL DROP MINIGAMES .

Revised: In the table below, we present model accuracy on each of the minigames where the maximal drop was observed.

Revised: The second and third column refer to unmanipulated image-question pair accuracy in the minigame.

Original:  [Table] 
Revised: The fourth and ﬁfth columns refer to the measured model accuracy in those minigames after the Adversarial Player manipulation. This indicates that minigames of maximal drop do not necessarily consist of ”hard” CLEVR examples, but rather exploit model and instancespeciﬁc vulnerabilities of the architecture under test. [Table] B. S CENE GENERATION METHOD AND UNINTENDED ISSUES .

Revised: In this section we will elaborate more on how we enforce that generated scenes are visually fair , in the sense that unintended issues such as object occlusion, or object placement out of ﬁeld of view are not existent, as well as answerable meaning that they can still be answered and the ground truth answer of the posed question before and after the scene manipulations remains the same.

Revised: For the generation of new images/scenes we use the open-source Blender Graphics Engine(v2.79b), and the original 3D models of the CLEVR dataset.

Revised: Scenes suggested by our Adversarial

Revised: Player , are encoded in the form of arguments for Blender, which renders the new images according to those directives.

Revised: In order to guarantee visual fairness, and question validity, we employ two modules (scene-constraint enforcer / question relevance enforcer), presented brieﬂy - in Section 4 in the

Revised: Consistency and In-Distribution paragraphs.

Revised: The scene-constraint enforcer is responsible for ensuring that the creation of a scene/image that respects all constraints and statistical properties of the original dataset.

Revised: This piece of code is borrowed from the original CLEVR codebase 3 and as in the original CLEVR paper the following constraints are checked (Lines 66-83 in the python code):

Revised: Minimum / Maximum Number of Objects: 3 / 10.

Revised: Minimum allowed distance between object centers: 0. 3.

Revised: Margin along with cardinal directions: 0.4 (This ensures lack of ambiguity by keeping objects at least 0.4 points of distance apart.

Revised: This distance is a Blender-speciﬁc measure that spans from -3 to +3.) 4.

Revised: Min pixels per object: 200 ( This ensures that an object will not be occluded, by requiring at least 200 pixels of it being visible.)

Revised: The question-relevance enforcer checks if the newly generated scene is indeed answerable and has the same “ground truth” answer as the old scene.

Revised: By not enforcing this, every manipulation will be falsely “successful” as the question (which is kept the same before and after the attack) will have a different answer in the newly generated scene.

Revised: This is also resolved by using the same question engine module provided by the CLEVR authors in their codebase 4 .

Revised: This module receives a question and a scene and calculates the ground truth answer based on a set of functional programs that are apriori known and used to generate the questions.

Revised: (Supplementary

Revised: Material A in the CLEVR paper (Johnson et al., 2017a))

Revised: The full training pipeline should be the following: Our Adversarial Player initially receives an input image-question pair.

Revised: Then, it suggests a new scene, that passes through the scene-constraint enforcer.

Revised: If the new scene respects all boundaries and constraints it is then evaluated alongside the question by the question-relevance enforcer.

Revised: If the new ground truth answer to the suggested scene/image is the same as the original answer, the image-question pair is marked as in-distribution and valid.

Revised: In any other case, it is removed from the training pipeline.

Revised: /main question_generation/question_ py

Revised: B. C OMPARISON WITH RANDOMLY SAMPLED EQUIVALENT SCENES .

Revised: Arguably, out of the semantically equivalent space of images given a question and an accompanying answer, the Adversarial Player player effectively discovers cases for which the model under test can be fooled.

Revised: This however, does not indicate how rare those examples are, neither the degree of robustness of the model under test against other examples in this space.

Revised: Given a speciﬁc initial image-question pair, a tractable estimate would be the number of manipulated scenes ( F ) discovered that are able to fool the model, divided by all the possible permutations of available objects ( N ) in our discrete 7x7 grid.

Revised: Rarity = F 49 N , N ∈ [3 , 10] .

Revised: A scene of only 3 objects and 1 discovered successful manipulation would make it 0.00085% possible to ﬁnd it by chance and would indicate 99.9915% robustness against that sphere of equivalent image-question pairs.

Revised: Of course not all of those conﬁgurations are valid due to scene constraint violations, as well as objects can theoretically move freely on the scene.

Revised: As a way to approach this question, we compare our Adversarial Player player against a random scene generator (RSG).

Revised: The random scene generator is able to manipulate any object freely, on the constraint that the resulting scene is valid under the scene constraint enforcer as well as the question relevance enforcer.

Revised: Manipulations of lighting or camera angles, as well as addition or removal of objects is not permitted.

Revised: In order to reduce the huge rendering times needed for each random search, we allow the RSG to operate on a 10 length Minigame where scene manipulations that cause accuracy drop have been discovered by the Adversarial

Revised: Player .

Revised: For each image-question pair the RSG has a budget of 5000 queries, and if not successful it proceedes to the next Mini-game entry.

Revised: [Table]

Revised: As it can be seen, randomly question-uninformed suggested object placements do not seem to be effective into fooling the models under test.

Revised: This further supports the argument that models under test theoretically and statistically seem robust, but nevertheless have vulnerabilities that can be effectively exploited by our Adversarial Player .

Revised: B. B EHIND THE SCENES VISUALIZATION OF FAILURE MODES .

Revised: Since the models under test are operating under different assumptions and architectural designs, we set up a visualization experiment to analyse visually their inner working mechanisms under a normal and a manipulated scene scenario.

Revised: Speciﬁcally, we choose to visualize:

Revised: The attention maps of Stack-Attention Networks.

Revised: The gradients of the pre-ultimate and ultimate convolutional blocks of FiLM.

Revised: The programs that are synthesized alongside their soft-attention maps of each module block in TbD network.

Revised: The objects that were detected during the computation steps of a MDetr model.

Revised: For each experiment, we choose two image-question pairs.

Revised: The ﬁrst consists of the original image and the second of a scene created by our Adversarial Player that succesfully fools the model under test.

Revised: As we can observe, in Stack-Attention Network the model seems to concentrate all its focus on a single ﬁnal object in manipulated examples, while in the original ones, the relational jumps required for the answer are still visible on the attention map.

Revised: The ﬁnal attention map that is presented is ultimately passed to a fully-connected classiﬁer in order for the ﬁnal answer to be created.

Revised: Hence, limited focus to all the necessary objects / locations may create lack of necessary features for the correct answer derivation.

Revised: In the case of FiLM model we observe a similar scenario.

Revised: In the ﬁrst example, the gradient-weighted activations of the model seem to be focusing on multiple irrelevant objects regarding the ﬁnal answer, while in the second example the model seems to include the purple cube mistakenly as one of the possible spheres of the image.

Revised: In the case of the MDetr model, a speciﬁc pattern could not be identiﬁed.

Revised: All objects were correctly classiﬁed and the reasoning steps towards the derivation of the answer are presented.

Revised: This objectdetection experiment, strengthens our claim of visually fair scene manipulations that respect the original dataset properties and do not promote object or attribute hindrance.

Revised: Finally, in the case of the TbD, in both the original and manipulated scenes the programs that the model generates are exactly the same.

Revised: This is to no surprise since the program generator module is using exclusively the question as input.

Revised: Nevertheless, a similar pattern of scattered model focus is presented here as well.

Revised: The attention maps in both examples are the same between the original and manipulated scenes, in all but the ﬁnal reasoning step.

Revised: In the ﬁrst example the attention is leaked towards the gray cube that has nothing to do with the correct answer, while in the second the attention is spread among many matte objects but peaks near the big yellow ball.

Revised: This is wrong since the word ”other” that exists in the question should drive the model focus exclusively to other objects apart from the starting point of the reasoning chain.

Revised: [Figure]

Revised: [Figure]

Revised: How many other objects are the same shape as the big gray matte object?

Revised: 1 −→

Revised: O r i g i n a l

Revised: S ce n e

Revised: M a n i pu l a t e d S ce n e

Revised: How many other things are the same shape as the big metallic thing?

Revised: 1 −→

Revised: O r i g i n a l

Revised: S ce n e

Revised: M a n i pu l a t e d S ce n e

Revised: [Figure]

Revised: what

Revised: [Figure]

Revised: −→

Revised: [Figure]

Revised: g i n a l S ce n e

Revised: M a n i pu l a t e d S ce n e

Revised: Do the object behind the big matte sphere and the thing on the right

Revised: [Figure]

Revised: of the

Revised: [Figure]

Revised: O r i g i n a l S ce n e

Revised: M a n i pu l a t e d S c e n e

Revised: Is there a big brown object of the same shape as the green thing?

Revised: Yes −→

Revised: No

Revised: O r i g i n a l

Revised: S ce n e

Revised: M a n i pu l a t e d S ce n e

Revised: Are there any other things that have the same material as the big yellow ball?

Revised: Yes −→

Revised: No

Revised: O r i g i n a l

Revised: S ce n e

Revised: M a n i p u l a t e d

Revised: S ce n e

Revised: B. H UMAN - IN - THE - LOOP E XPERIMENT

Revised: Would a human be fooled from scene manipulations of this kind?

Revised: The perceptual abilities of human observers have been previously measured (92.6%) on this dataset, however human robustness is a totally different question.

Revised: In our experiment, a human observer is presented with a set of 2 images and a question.

Revised: Then the observer is required to answer to each image image question pair: (Image Before - Question / Image After - Question), and report whether the answer has changed or remained the same.

Revised: Participants were ﬁrst made familiar with the concepts of the CLEVR dataset (colors/shapes/materials/sizes) and then were left unsupervised to answer a set of 20 such examples.

Revised: Each contained 1 CLEVR question and 2 CLEVR scenes, before and after the Adversarial Player manipulations.

Revised: Half of those scenes successfully fooled models and half of those did not.

Revised: An example of what a survey question looks like is given below: Here is the confusion matrix from the survey results:

Revised: As we can see, our human observers exhibit 91% precision in identifying cases where scene manipulations are not causing change to the questions answer, and 95% precision in identifying cases where the manipulations do cause a change.

Revised: While this small scale experiment may not reﬂect a universal truth about human robustness in VQA scenarios, it is a suggestion that human observation is still reliable.