DDR PROG : A CLEVR D IFFERENTIABLE D YNAMIC R EASONING P ROGRAMMER
Anonymous authors
Paper under double-blind review
A BSTRACT
We present a generic dynamic architecture that employs a problem speciﬁc differentiable forking mechanism to encode complex assumptions about the problem data structure.
We adapt and apply our model to CLEVR Visual Question
Answering, giving rise to the DDRprog architecture; compared to previous approaches, our model achieves higher accuracy in half as many epochs with ﬁve times fewer learnable parameters.
Our model directly models underlying question logic using a recurrent controller that jointly predicts and executes functional neural modules; it explicitly forks subprocesses to handle logical branching.
While FiLM and other competitive models are static architectures with less supervision, we argue that inclusion of program labels enables learning of higher level logical operations–our architecture achieves particularly high performance on questions requiring counting and integer comparison.
We further demonstrate the generality of our approach though DDRstack – an application of our method to reverse Polish notation expression evaluation in which the inclusion of a stack assumption allows our approach to generalize to long expressions, signiﬁcantly outperforming an LSTM with ten times as many learnable parameters.
I NTRODUCTION AND R ELATED W ORKS
Deep learning is inherently data driven – visual question answering, scene recognition, language modeling, speech recognition, translation, and all other supervised tasks can by deﬁnition be expressed as: given relevant information x, predict desired quality y. While the ﬁeld has attempted to model an immense set of underlying data structures with neural architectures, core convolutional and recurrent building blocks were designed with only the most general notions of spatial and temporal locality in mind.
In some cases, additional information about the problem can be expressed simply as an additional loss.
When hard logical assumptions are present, it is far from obvious how to do so in a manner compatible with backpropagation.
In this work, we examine two problems: visual question answering (VQA) on the CLEVR (Johnson et al.
dataset and expression evaluation in reverse Polish notation (RPN).
In the former, questions seen at training time contain direct programmatic representations well modeled by a set of discrete logical operations and a stack requiring at most one recursive call.
The latter is an extreme case with deep recursion requiring a full stack representation, but supervision is also available at test time.
We present a generic architecture that uses a problem speciﬁc forking behavior to leverage structural information about the input data.
Our approach resolves common differentibility issues and is effective on both problems: we achieve a moderate improvement over previous state-of-the-art on CLEVR and outperform a much larger baseline model on RPN.
Prior work on CLEVR VQA is largely categorized by dynamic and static approaches.
IEP (Johnson et al.
and NMN (Hu et al.
both generalized the original neural module networks architecture (Andreas et al.
and used the functional annotations in CLEVR to predict a static program which is then assembled into a tree of discrete modules and executed. IEP further demonstrated success when program annotations are available for only a few percent of questions.
These are most similar to our approach; we focus largely upon comparison to IEP, which performs signiﬁcantly better.
RN (Santoro et al.
and FiLM (Perez et al. (2017b)), the latter being the direct successor of CBN (Perez et al. (2017a)) are both static architectures which incorporate some form of implicit reasoning module in order to achieve high performance without program annotations.
In contrast, our architecture uses program annotations to explicitly model the underlying question structure and jointly executes the corresponding functional representation.
As a result, our architecture performs comparably on questions requiring linear application of visual transformations, but it performs signiﬁcantly better on questions requiring higher level operations such as counting and numerical comparison.
The RPN task is, to our knowledge, novel.
The most comparable work we are aware of is StackRNN
Joulin & Mikolov (2015)), which endows a recurrent architecture with the ability to push and pop from a stack.
However, StackRNN neither learns nor assumes an explicit stack structure from the labels.
While this approach is more general, StackRNN is limited to learning simpler data structures and is applied only to memorization, binary addition, and language modeling (it underperforms an
LSTM in this case).
In contrast, our model makes stronger structural assumptions that allow it to perform well on expression evaluation in base ten with four discrete operations.
D ATASETS 2. CLEVR
CLEVR is a synthetic VQA dataset consisting of 100k images and 1 million question/answer pairs.
Over 850k of these questions are unique.
Images are high quality 3D Blender (Blender (2017)) renders of scenes containing geometric objects of various shapes, sizes, colors, textures, and materials.
Thus the dataset is quite realistic despite being synthetic.
Furthermore, the authors ran comprehensive tests to avoid exploitable biases in the data.
As the objects are geometric ﬁgures, no external knowledge of natural images is required, as in earlier VQA datasets.
Most importantly, CLEVR provides an additional representation of each question as a functional program.
For example, ”How many red spheres are there?”
is represented as [ ﬁlter red, ﬁlter sphere, count ].
Some questions require nonlinear program structure, such as ”How many objects are red or spheres”, which is represented by a tree with two branches [ ﬁlter red ] and [ ﬁlter sphere ] followed by a binary [ union ] operation and a ﬁnal [ count ].
We are unaware of any dataset with comparable properties of this size and complexity.
This makes the dataset uniquely useful for evaluating dynamic architectures, which are able to directly learn and execute discrete functional program cells.
2. RPN
The speciﬁc expression presentation we consider is the reverse Polish notation (RPN) form [NUM]*( n +1)-[OP] * n , that is, n +1 numbers followed by n operations. This simpliﬁes the problem by eliminating consideration for order of operations.
Thus the task is: given a sequence of tokens corresponding to a valid expression in reverse Polish notation, evaluate the expression and produce a single real valued answer. As currently posed, this problem is incredibly complex for large n . Long expressions are highly nonlinear; without additional supervision, the problem is essentially to learn a hash function, which is intractable in general.
We therefore modify the problem as such: instead of producing only the ﬁnal expression evaluation, produce the sequence of answers to all n intermediate expressions in the answer labels.
For example, 2 3 4 + *, the expected output would be [7, 14] because 3+4=7 and 2*7=14. We further assume the stack structure of the problem is available to the architecture should it be capable of taking advantage of such information. The problem is still sufﬁciently complex–note that to the model, { 1 , 3 , 4 , + , ∗} would all be meaningless tokens: it must learn both the NUM and the OP tokens.
The dataset consists of 100k train, 5k validation, and 20k test expression with n = 10 – that is, 11 numbers followed by 10 operations.
We also provide a 20k expression generalization set with n = 30 .
The label for each question contains the n solutions to each intermediate operation.
During data generation, we sample NUM and OP tokens uniformly, reject expressions including division by zero, and also omit expressions that evaluate to over 100 in magnitude.
The NUM tokens correspond to 0, 0.1, ..., 0.9 and the OP tokens correspond to +, -, *, /; however, architectures are not privy to this information.
DDR A RCHITECTURE
Our architecture is conceptually problem independent, but it requires adaptation to the speciﬁc data structure of the task.
The core is a recurrent controller similar to NPI (Reed & de Freitas (2015)).
In both CLEVR VQA and RPN, our model maintains a discrete set of neural modules, as in IEP and NMN.
However, unlike NPI, our model responds ﬂexibly to the problem supervision.
In VQA, modules are only known at training time: our CLEVR architecture, DDRprog, therefore learns to predict them jointly during inference.
In the RPN task, modules correspond to operations and are also known at test time: our RPN architecture, DDRstack, therefore executes them directly.
Details of the DDR variants DDRprog and DDRstack are provided below.
3. CLEVR V ISUAL Q UESTION A NSWERING : DDR PROG
Our architecture consists of the core recurrent controller and a set of neural modules, which correspond to program annotations in CLEVR.
Some standard encoders are required to adapt DDR to the mixed language and visual data, as well as a small classiﬁer.
We provide pseudocode in Algorithm 1, a visual representation in Figure 1, and subnetwork details in Table 2 (see Appendix).
Algorithm 1 DDRprog.
Note that CNN produces a ﬂattened output and Controller also performs a projection and argmax over program scores to produce programPrediction img, question ← x stack ← Stack() img, imgCopy ← ResNetFeaturizer( img ) langState ← LSTM( question ) for i = 1 ...
MaxProgramLength do visualState ← CNN( img ) programPrediction ← Controller( visualState, langState ) cell ← Cells[ programPrediction ] if cell is Fork then stack .push ( cell ( img, imgCopy ) ) else if cell is Binary then img ← cell ( stack .pop () img ) else img end end for return
The input data x for each sample consists of an (image, question, program) triple and an answer label y .
As the program data is not available at test time, our model must learn to predict it.
The network ﬁrst encodes the question with an LSTM and the image with the same ResNet (He et al. feature extractor used in IEP and FiLM.
This produces a language state and a visual state.
Both of these are passed to the controller–we use a recurrent highway network (RHN) (Zilly et al.
(2016)), though we could have used a more familiar LSTM(Hochreiter & Schmidhuber (1997)).
Note that the RHN accepts ﬂat inputs whereas the visual state contains convolutional maps.
We resolve this disparity with an additional encoding module similar in design to the IEP classiﬁer.
At each time step, the controller predicts an index over the learnable neural modules, which use the same structure as in IEP with smaller hidden dimension.
The corresponding module is applied to the visual state, which is then updated.
The ﬁnal module prediction is followed by a small classiﬁer network, which uses the IEP classiﬁer.
This architecture is sufﬁcient on the subset of CLEVR programs that are linear.
However, some
CLEVR questions contain a logical branching operation (e.g. are there more of ... than ... ? ) and cannot be answered by purely linear programs.
In this case, our architecture forks a subprocess that shares the main process’s language and visual states.
The subprocess proceeds normally until it encounters a binary cell.
As shown in Figure 1, the binary cell merges the visual states of the main process and subprocess and applies the given function.
The subprocess then rejoins the main program.
Note that while CLEVR programs contain at most one branch operation, our architecture is likely to generalize to more difﬁcult questions, as the fork module could push to a deeper stack (as in our DDRstack architecture, below) or in general interface with trees, priority queues, or any other data structure to handle arbitrarily complex program traces.
The fork module must differ from a standard unary module, as it is necessary to pass the original
ResNet features to the subprocess in addition to the current visual state.
Consider the question: “Is the red thing larger than the blue thing?”
In this case, the left program branch ﬁlters by red; it is impossible to recover the blue objects in the right branch given a red ﬁltered image.
We found that it is insufﬁcient to pass only the original images to the subprocess, as the controller is small and has difﬁculty tracking the current branch.
We therefore use a variant of the binary module architecture that merges the original ResNet features with the current visual state (see Algorithm 1).
As the fork module is shared across all branch patterns, it is larger than the other binary modules and also one layer deeper.
3. E XPRESSIONS IN R EVERSE P OLISH N OTATION : DDR STACK
Our architecture consists of the core recurrent controller (in this case an LSTM) and a set of 4 (one per OP) learnable binary modules, as well as an explicit stack.
DDRstack can be viewed as a dynamic wrapper that augments the LSTM to include a stack assumption; it is also a direct adaptation of the analytical RPN expression evaluation algorithm to a neural architecture.
We provide high level pseudocode for the model in Algorithm 2 and a visual representation in Figure 2.
Algorithm 2 DDRstack tokens ← x stack ← Stack() state ← RandomLSTMInitialization for all tok in tokens do if tok is a NUM then out ← Embed( tok ) else if tok is a OP then arg 2 ← stack .pop
() arg 1 ← stack .pop
() out ← Cells[ tok ]( arg 1 , arg 2 ) end if out, state ← LSTMCell( out, state ) stack .push( out ) end for return
Projection( out )
We train a baseline vanilla LSTM supervised with the intermediate solutions in the last n timesteps.
DDRstack uses the same baseline LSTM structure as its core controller, but it processes NUM and
OP tokens differently, as well as interacting with the internal stack, as outlined below.
Predictions are made in the last n timesteps, as in the vanilla model (Algorithm 2 shows only the ﬁnal return).
NUM: Our model embeds the token and passes it to the LSTM.
It then pushes the result to the stack.
OP:
Our model pops twice, calls the OP speciﬁc binary cell, and then passes the results to the LSTM.
It then pushes the result to the stack.
The binary cell is a simple concatenation of the arguments followed by a single fully connected layer with no nonlinearity.
DDR PROG : CLEVR VQA E XPERIMENTS AND D ISCUSSION 4. E XPERIMENTS
We train our model with Adam (Kingma & Ba (2014)) on the full CLEVR dataset, including all program annotations.
The hyperparameters detailed in Table 3 (see Appendix) are minimally conﬁgured from defaults, excepting learning rate and rough network size.
The network overall has 9M parameters.
We exclude the ResNet feature extractor from all calculations because it is also present in the best FiLM model.
Their work further demonstrated it is fairly straightforward to replace it with a from-scratch feature extractor with minimal loss in accuracy.
We pass the ground truth program labels to the model during training.
Critically, the program labels are only used on the validation set for the purpose of model selection, and our ﬁnal accuracy is obtained by rerunning the validation set without the ground truth programs.
We train on a single
GTX 1080 TI and, after 35 epochs, our model matches the previous state-of-the-art accuracy of 97.
percent.
We continue training until the 52nd epoch, dropping the learning rate to 1e-5 for the last few epochs to ensure convergence, and obtain 98.3 percent accuracy.
The model predicts program cells with 99.98 percent accuracy.
D ISCUSSION
RN, FiLM, IEP, and our architecture are all fairly close in overall accuracy, but the accuracies in each category vary.
From Table 1, no architecture has particular difﬁculty with Exist, Query, or Compare questions; the main differentiating factors are Count and Compare Integer.
Though Compare Integer is the smallest class of questions and is therefore assigned less importance by the cross entropy loss, the IEP result suggests that this does not cause models to ignore this question type.
We therefore consider Count and Compare Integer to be the hardest unary and binary tasks, respectively, and we assign most important to these question subsets in our analysis. [Table] [Table]
We ﬁrst compare to IEP.
While our model is less than 1/4 the size of IEP (see Table 1), it closely matches binary performance (+0.2 percent on Compare, -0.3 percent on Compare Integer) and perform at least 2x better across all unary tasks (+1.7 percent on Exist, +3.8 percent on Count, + 1. percent on Query).
We believe that our model’s lack of similar gains on binary task performance can be attributed to the use of a singular fork module shared across all binary modules.
The fork module is responsible for informing the right program branch of progress made in the left program branch.
We have observed that this module is essential to obtaining competitive performance on binary tasks; it is likely suboptimal to use a large shared fork module as opposed to a separate smaller cell for each binary cell.
However, this still requires experimental veriﬁcation.
Our model surpasses RN in all categories of reasoning, achieving a 2.6x reduction in overall error.
RN achieves especially impressive results for its size and lack of program labels.
However, it is questionable whether the all-to-all comparison model will generalize to more logically complex questions.
In particular, Count operations do not have a natural formulation as a comparison between pairs of objects, in which case our model achieves a 6.4 percent improvement.
RN also struggles on the challenging Compare Integer subtask, where we achieve a 4.8 percent improvement.
Furthermore, it is unclear how essential high epoch counts are to the model’s performance.
As detailed in Table 1, RN was trained in a distributed setting for 1000 epochs.
Both our result and FiLM were obtained on single graphics cards and were only limited in number of epochs for practicality–FiLM had not fully converged, and our model was unregularized.
FiLM also surpasses RN in all categories and is the only static, no-programs architecture to surpass
IEP in raw accuracy, and it does so with only slightly larger model size.
FiLM achieves 1.5x relative improvement over our architecture on Exist questions, but this is offset by our 1.5x relative improvement on Count questions.
FiLM does not perform particularly poorly on Count questions, but the contrast between its Compare Integer and its Exist/Query/Compare performance is quite striking– While FiLM surpasses IEP in all other categories, both IEP and our model achieve a roughly 4x improvement on Compare Integer questions (4.9 and 4.6 percent, respectively).
We believe this is because it is difﬁcult to model the more complex binary structures using only implicit branching through batch normalization parameters–many Compare (attribute) questions can be more easily resolved through a sequence of purely visual manipulations.
DDR STACK : RPN E XPERIMENTS AND D ISCUSSION 5. E XPERIMENTS
For our architecture, we use hidden dimension 32 throughout the model, resulting in only 17k parameters overall. We train with Adam using learning rate 1e-3 and obtain a test L1 error of 0.17 after 63 epochs.
Using the same hidden dimension in the pure LSTM baseline (9k parameters) results in test error 0.28.
We overcompensate for the difference in model size by increasing the hidden dimension of the LSTM to 128 (255k parameters), resulting in an only slightly lower test error of 0. after nearly 3000 epochs.
Training curves for the larger LSTM baseline and DDRstack are shown in Figure 3.
After training both models on problems of length n = 10 , we test both models on sequences of length n = 10 and n = 30 .
For a sequence of length n both models predict values not only for the entire expression but also for all n subproblems where index n corresponds to evaluating the entire sequence.
For sequences of both lengths, we evaluate accuracy for the predicted answers on all subproblems. Results are shown in Figure 4.
5. D ISCUSSION
With a sufﬁcient number of epochs and a large parameter budget, it is probable that an LSTM would surpass our model.
We could increase our model size to match, but discounting this line of reasoning, we argue that the LSTM would still fail to learn the underlying stack structure and would therefore fail to generalize to longer expressions.
From Figure 4, both the small and large LSTM baselines approximately match our model’s performance on the ﬁrst 5 subproblems of the n = 10 dataset.
From n = 6 to n = 10 , the performance gap grows between our models–in particular, the small LSTM is unable to learn deep stack behavior, and performance decays sharply.
Both the small and large LSTM perform comparably on all subproblems of the n = 30 dataset.
Somewhat confusingly, performance is far worse on the ﬁrst few subproblems of this dataset than on the test set of the original task.
This is not an error: recall the question formatting [NUM]*( n + 1 )- [OP]* n .
The leading subproblems do not correspond to the leading tokens of the question, but rather to a central crop.
The rapid increase in error on the LSTM implies that it could not even learn this property, let alone the stack structure.
Instead, it memorized all possible subproblems of length n ∈ { 1 , 2 , 3 } expressions preceding the ﬁrst few OP tokens.
Performance quickly decays to L1 error greater than 2.0.
Considering that the standard deviation of answers (discounting the ﬁrst few subproblems) is approximately 6.0, this corresponds to mostly noise.
In contrast, our model’s explicit incorporation of the stack assumption results in a smooth generalization curve with a gradual decay in performance as problem length increases.
We brieﬂy address a few likely concerns surrounding the above line of reasoning.
First, one might argue that the comparison is invalid, as the LSTM does not incorporate any explicit knowledge of the problem structure.
While this evaluation is correct, it is antithetical to the purpose of our architecture.
The LSTM baseline does not incorporate this additional information because there is no obvious way to include it–the prevailing approach would be to ignore it and then argue the model’s superiority on the basis that it performs well with less supervision.
In contrast, our dynamic approach is a minimal augmentation that allows our model to achieve a dramatic increase in performance and generalization precisely by incorporating additional supervision.
This motivation also separates
DDRstack from previous architectures that include some variant of a stack, such as StackRNN.
The former makes a hard assumption about the data structure in order to solve difﬁcult problems, whereas the latter learns to make use of a stack from scratch at the cost of constraint to simpler problems.
The hardest task on which StackRNN was evaluated is binary addition; in comparison, DDRstack effectively evaluates long expressions in base 10 with 4 different operations.
C ONCLUSION
We have introduced a novel architecture that differentiably forks subprocesses in order to encode complex assumptions about the data structure into the model.
We have demonstrated that our architecture is viable when certain structural details are omitted at test time: in the case of CLEVR
VQA, our architecture learns to predict the missing program annotations at test time and achieves state-of-the-art performance.
When structural details are also available at test time, as in the RPN stack structure, our architecture is able to smoothly generalize whereas baseline LSTM performance immediately decays.
It is our intent to continue reﬁning the versatility of our architecture, including more accurate modeling of the fork module, as mentioned in our CLEVR VQA discussion, as well as to generalize to the semisupervised setting where some datapoints in a single problem contain more structural information than others, as in transfer learning to CLEVR Humans.
It is our hope that our architecture and its design principles enable modeling of complex data structure assumptions across a wide class of problems where standard monolithic approaches would ignore such useful properties.
A PPENDIX
Subnetwork Details
ResNetFeaturizer
Features from ResNet101 pretrained on ImageNet, as in IEP and FiLM
LSTM 2-Layer LSTM that encodes the question CNN IEP classiﬁer variant; produces a ﬂat visual state. [Table]
Recurrent Highway Network for language and CNN Encoded visual states
[Table]
IEP set of unary and binary modules, plus our fork module and pads details for DDRprog.
Only the learning rate and model size were coarsely cross validated due to hardware limitations: hyperparameter are not optimal.
[Table]