Compositional Generalization and Neuro-Symbolic Architectures
Abstract
Compositional generalization is the ability to understand novel combinations of known concepts.
Although it is considered as an innate skill for humans, recent studies have shown that neural networks lack this characteristic.
In this paper, we focus on compositional generalization with respect to the two speciﬁc tasks of word problem solving and visual relation recognition and propose a neuro-symbolic solution, using DeepProbLog, that addresses the problem of compositionality in state-of-the-art neural systems for these tasks.
Introduction
Compositional generalization is an instinctive ability of humans to decipher novel combinations of primitive concepts, once they are familiar with the primitive concepts and have observed a few instances of the interactions between these concepts.
For example, if a person knows the meaning of ‘jump’, ‘run’ and ‘run twice’, she can immediately understand the meaning of ‘jump twice’.
Similarly, if a person can identify a cube, sphere, a red sphere and a metallic sphere, it is straightforward for her to recognize a red metallic cube even though she has not seen one before.
Recent research (Lake and Baroni 2018), (Klinger et al. 2020) has demonstrated through extensive experimental studies that purely neural models lack the characteristic of compositional generalization.
While (Lake and Baroni 2018) showed that sequence to sequence models were not effective in learning compositional rules for generalization in the domain of natural language processing (NLP), (Klinger et al. 2020) demonstrated how neural models failed to generalize compositionally in relation recognition in the domain of computer vision.
Word problems or story problems in the mathematics domain are short narratives describing a real-world problem scenario in natural language.
The narrative is accompanied by a question about the problem scenario.
Answering this question requires one to ﬁrst extract pertinent numerical facts from the problem, identify the unknown quantity, form an equation and ﬁnally solve it.
The following are examples of two word problems, mapped to their equations.
These are single-equation single-operation/step problems as they can be solved with a single equation that involves a single operation (addition in Problem 1, subtraction in Problem 2).
Problem 1 : John had 10 apples.
He bought 11 more.
How many apples does he have now?
Equation : x = 10 +
Solution : x =
Problem 2 : John had 10 apples.
He ate 2.
How many apples does he have now?
Equation : x = 10Solution : x =
Compositional generalization in the context of word problem solving is the ability to solve new problems involving equations with multiple operations having observed only single-equation single-operation word problems.
Consider the following problem:
Problem 3 : John had 10 apples.
He bought 11 more.
He ate 2.
How many apples does he have now?
Equation : x = 10 + 11 −
Solution : x =
Although the above problem is a new type of problem as the equation it maps to involves multiple operations (unlike Problems 1 and 2), it is composed of sentences that are similar to the ones in Problems 1 and 2.
If a student knows how to solve Problems 1 and 2, solving Problemis straightforward.
But, as we will show in this paper, the state-of-the-art word problem solvers fail to solve a multioperation problem like Problem 3 at test time, if these are just trained on single-operation problems like 1 and 2.
That is, these solvers fail to generalize compositionally.
In order to evaluate the ability of automatic word problem solvers to generalize compositionally, we create a dataset by collecting (from existing datasets) a speciﬁc type of word problems called Change Problems (Powell and Fuchs 2018).
Change problems involve scenarios where there is an entity of interest with an initial value, that gets updated (either increased or decreased) to a new value as a result of an action.
Problems 1, 2 (single update change) and 3 (two update change) are all examples of change problems.
Compositionality is also the focus of recent research in computer vision due to multi-modality in visual features.
Figure 1 shows an example of visual-spatial intelligence tests, rather easy for humans to address, but problematic for neural-based methods due to compositionality in the color/position/shape of objects.
Although existing deep learning methods have shown promising results in visual relation recognition, they tend to fail with compositional generalization (Klinger et al. 2020).
This is largely due to distributionshift where a model is tested with new combinations of features never observed during training.
In this paper, we study compositionality w.r.t color, shape and position of the objects in the image (as shown in Figure 1).
In order to evaluate compositionality in visual relation recognition, we have devised our own dataset, rather than using available benchmarks (Johnson et al. 2016;
Santoro et al. 2017).
One reason behind such decision is that the aforementioned benchmarks are associated with language input which requires a parallel line of processing separated from the visual recognition task.
Furthermore, by generating our own dataset, we are in control of generating different types of compositionality related to color, shape, position and number of objects.
One can address inability of neural models to generalize compositionally by providing it with more data that has sufﬁcient number of instances of every combination of concepts we are interested in.
There are tasks where providing more data may not be practical, but it may be instead more convenient and efﬁcient to provide knowledge about the tasks that can aid the models in generalizing better.
We, in this paper, show that incorporating task speciﬁc knowledge into purely neural models help address its inability to generalize compositionally.
For this, we propose a neuro-symbolic approach using DeepProbLog (Manhaeve et al.
for both word problem solving and visual (spatial) relation recognition, and, show how by modelling task speciﬁc knowledge in logic, such models can improve the performance of neural models w.r.t compositional generalization.
Our contributions are the following - for the two tasks of word problem solving and visual relation recognition, we
create datasets to test compositional generalization.
analyze compositional generalization ability of state-ofthe-art neural models using the above datasets.
develop neuro-symbolic models based on DeepProbLog to improve compositional generalization ability through inclusion of task speciﬁc knowledge.
Related Work
In this section we brieﬂy survey the following areas: automatic word problem solving, visual relation recognition, and ﬁnally, neuro-symbolic modelling.
Automatic Math Word Problem Solving
There has been a surge of research in automatic math word problem solving in the past six years.
(Zhang et al.
2019) gives an extensive survey on automatic word problem solvers.
While the earlier systems used a purely symbolic approach to solving, the more recent ones are data-driven based systems.
Some consider it as a structure prediction problem, where the structure to be predicted is the expression tree corresponding to the equation that the word problem is mapped onto ((Roy and Roth 2016), (Kushman et al.
Since these required manual feature engineering, the community has transitioned to deep learning approaches.
Most of the existing neural models for math word problem solving utilize Seq2Seq models (Luong, Kayser, and Manning 2015), considering the source sequence as the word problem and the target sequence as the equation.
More recently (Zhang et al. 2020) and (Xie and Sun 2019) introduced graph-based encoders to encode the graph-based representation of the word problem and tree-based decoders to generate expression trees for the word problem.
The research on math word problem solvers started off with datasets like, AddSub (Kushman et al. 2014) and SingleOp (Roy, Vieira, and Roth 2015), containing simple single-equation word problems.
Although some now consider this as a solved research problem, (Patel, Bhattamishra, and Goyal 2021) illustrated through experiments that the existing state-of-the-art solvers relied on superﬁcial heuristics to map the word problems to equations.
They created a challenge dataset named SVAMP, that contained problems created by slightly varying the problems from existing datasets like AddSub and SingleOp.
These variations were created by adding relevant or irrelevant information to the problem or changing the order of phrases in the word problem.
It was shown that the state-of-the-art solvers trained on existing datasets showed poor performance when tested on SVAMP dataset.
In this paper, we introduce another challenge - that of compositional generalization and show that while stateof-the-art neural solvers fail to generalize compositionally, the proposed DeepProbLog solver exhibits promising results.
To test the ability of solvers to generalize compositionally, we train both the state-of-the-art solvers and our
DeepProbLog-based model on simple single-step, singleequation change problems from AddSub and SingleOp (like,
Problem 1 and Problem 2) and then test the trained models on multi-step, single-equation problems (like Problem 3).
Visual Relation Recognition
In recent years, visual relation recognition has also been at the centre of deep learning research in computer vision under different names such as visual question answering (VQA).
Most of the deep learning approaches for VQA are based on convolutional neural networks (CNNs) together with relation networks (RNs)
(Santoro et al. 2017).
For instance, a two-stage Relation Network (2S-RN) (Messina et al. 2018) has achieved 95% accuracy in answering visual relational questions in CLEVR dataset (Johnson et al.
(Ding et al. 2021) also proposed a neural object-based attention model for solving spatio-temporal reasoning task and was tested on CLEVRER (Yi et al. 2020) and CATER (Girdhar and Ramanan 2019) datasets.
The RN-augmented architecture (Santoro et al. 2017) has outperformed the stateof-the-art in tackling relational inferences about complex scenes in CLEVR.
Moreover, (Dai, Zhang, and Lin 2017) proposed an integrated framework called Deep Relational
Network (DR-Net), that could exploit statistical dependencies between objects, their spatial conﬁgurations and appearances.
DR-Net is composed of several modules such as
RCNN-based object detection (He et al. 2017), pair ﬁltering, and joint recognition used for exploring all the possible spatial relations between any two pairs of objects.
A large number of contributions in visual relation recognition has been evaluated on the datasets involving both visual scenes and text annotations, i.e., the proposed models include modules to learn features from the language input in parallel.
Towards more general purpose architectures for relation recognition, (Shanahan et al. 2020) proposed PrediNet, an end-to-end network inspired by symbolic AI based on ﬁrstorder predicate calculus, that learns and maps vectorial representations from raw pixels to propositions with explicit relational structure.
PrediNet (upon a customized dataset) has shown its capacity to address compositionality to a degree, thanks to the core of the network that allows for parallel processing of information organized into small chunks that contribute to learning reusable representations.
However, most proposed deep relation recognition methods perform poorly w.r.t compositional generalization (Klinger et al. 2020).
We show in this paper that a neuro-symbolic model for relation recognition can offer solutions to the problem of compositionality observed in deep neural models.
Neuro-symbolic Models
The area of Neuro-Symbolic Artiﬁcial Intelligence (NeSy
AI) focuses on integrating the neural and symbolic AI in such a way that the desirable characteristics of both are inherited efﬁciently and effectively (De Raedt et al.
The research in the area has resulted in the development of multiple neuro-symbolic frameworks like DeepProbLog (Manhaeve et al. 2018), Concept Learner (Mao et al. 2019), Neural-Grammar-Symbolic Model (NGS) (Li et al. 2020), Neuro Symbolic Forward Reasoner (NSFR) (Shindo, Dhami, and Kersting 2021).
These frameworks differ in the way the symbolic part is made differentiable and incorporated into the training of the neural network part of the framework.
In this paper, we use DeepProbLog to implement neuro-symbolic solutions for the two tasks.
DeepProbLog integrates neural networks with the very expressive ProbLog paradigm (De Raedt, Kimmig, and Toivonen 2007), and allows explicit representation of task-speciﬁc knowledge in ProbLog (unlike Concept Learner, which is based on quasi-symbolic program execution) and is based on a gradient-based learning algorithm (unlike NGS - in which the module that propagates loss back to the neural network is task-dependent and has to be redeﬁned for every task).
DeepProbLog for Compositional Generalization
As mentioned earlier, DeepProbLog combines neural networks with the probabilistic programming language,
ProbLog (De Raedt, Kimmig, and Toivonen 2007), by extending ProbLog with neural predicates.
In Problog, a fact f can be associated with a probability p , as p :: f , indicating f is true with probability p and false with probability 1 − p .
An annotated disjunction in Problog is represented as:
[Equation]
m where, Σ p i < = 1 .
The above translates to ‘if all b i hold, then either one of the h j is true with probability p j , or none is true with probability 1 − Σ p
i ’.
DeepProbLog, through neural predicates, enables us to use neural networks to provide probabilities associated with a fact.
A neural annotated disjunction is of the following form:
[Equation]
where m r represents a neural network that generates a probability distribution
p m r
( O | X = x ) over the domain [ y 1 , ..., y n ] , given a speciﬁc input x = [ x 1 , ..., x k ] (raw data - e.g., sequence of words, tensor representing an image).
The above translates to ‘ r ( X 1 , ..., X k , y i ) is true with probability
p m r
( O = y i |
X 1 = x 1 , ..., X k = x k ) ’.
The neural network is a discriminative classiﬁer, that may use an encoder of choice (e.g., convolutional, recurrent neural networks), but the output layer is expected to normalize the inputs it receives into a probability distribution (like softmax layer).
Thus, a DeepProbLog program is comprised of a set of neural annotated disjunctions and neural facts in addition to a set of probabilistic facts, and rules as in ProbLog program.
The weights of the neural networks represented by the neural predicates in a DeepProbLog program is learned using a gradient-based learning approach.
DeepProbLog relies on gradient semiring in Algebraic ProbLog (Kimmig,
Van den Broeck, and De Raedt 2011) to compute the gradient of the loss with respect to the outputs of the neural networks (which are the probabilities associated with the neural facts).
The gradient of the neural network outputs with respect to the weights of the neural network is computed as in the standard backpropagation algorithm (using chain rule).
We will next describe the details of the proposed
DeepProbLog-based word problem solver and visual relation recognizer, and the solving pipeline as depicted in the grey box in Figures 3 and 4.
DeepProbLog for Word Problem Solving
We break down the task of word problem solving into two components - the ﬁrst is that of perception or understanding and extracting relevant facts from the natural language description and the second that of performing mathematical reasoning on these facts.
Perception In this section, we describe the neural networks associated with the two neural predicates in the program.
In order to address the problem of compositionality, we process the word problem sentence-wise, representing each sentence in logical form.
We use a representation simi- lar to the one described in (Kushman et al. 2014).
The set of predicates used are shown below:
holds(C, E, N) : represents the fact that a container ‘ C ’ holds ‘ N ’ number of the entity ‘ E ’.
increase(C, E, N i ) : represents the fact that the number of items of entity ‘ E ’ held by the container ‘ C ’ is increased by an amount, ‘ N i ’.
decrease(C, E, N d ) : represents the fact that the number of items of entity ‘ E ’ held by the container ‘ C ’ is decreased by an amount, ‘ N d ’.
The logical representation for Problem 1 would be:
• ‘John had 10 apples’ holds(John, apples, 10) • ‘He bought 11 more’ increase(John, apples, 11) • ‘How many does he have now?’
holds(John, apples, ?
X)
Hence, given a sentence, the goals of perception are to identify the predicate associated with the sentence ( holds or increase or decrease ) and the arguments associated with this predicate (the container and the entity).
Figure 2 shows the two neural networks, both discriminative classiﬁers, involved - predicate recognizer and entity/container recognizer .
A predicate recognizer takes the speciﬁc sentence - S , and its context, which is the entire word problem - P , as its two inputs and generates a probability distribution over the domain { holds, increase, decrease } .
The context is in some cases important to decide whether the verb in the sentence represents an action that results in an increase or decrease of the quantity of interest.
Consider the following example:
Problem 4 : There were 6 roses in the vase.
Mary cut some roses from her ﬂower garden.
There are nowroses in the vase.
How many did she cut?
Equation : 6 + x =
Solution : x =
The predicate associated with the second sentence is ‘increase’ as it results in an increase in the number of roses in the vase.
If the sentence is considered in isolation, there are possibilities of associating it with ‘decrease’, as it is describing an action (‘cut’) that caused a reduction in the number of roses in the ﬂower garden.
The neural network uses a
Bidirectional Long Short Term Memory (BiLSTM) to generate contextual representations of the sentence and the word problem, which are then consumed by a MultiLayer Perceptron (MLP) to generate the required probability distribution.
A container/entity recognizer takes three inputs - the sentence - S , the word problem - P and the index - W i (some number between 0 and 13 in case of Problem 1 as shown in
Figure 2) of a word in the word problem.
The network predicts the probability that the corresponding word is a container or entity or neither for the speciﬁc sentence.
The sentence cannot be processed in isolation, as it might not have a container or entity mentioned in it directly.
For example, in the sentence, ‘ He bought 11 more ’, in Figure 2, the pronoun ‘he’ is used to refer to the actual container ‘John’ which is in the previous sentence.
Similarly, the entity, ‘apples’ is also absent in the sentence.
As in the case of predicate recognizer, we use a BiLSTM to generate contextual representations of the sentence and the word problem.
The representation of the speciﬁc word, W i , is obtained from the word problem’s representation which is then concatenated with the sentence before passing it on to MLP.
MLP generates a probability distribution, over the domain { container, entity, none } , for the word.
We invoke the network once for each word in the word problem - since containers and entities are nouns, we invoke the network only for words that are nouns in the word problem.
Thus, the recognizer gives a probability distribution for each noun in the word problem.
Reasoning Figure 3 shows the different phases in solving, starting from perception.
The knowledge provided enables it to derive the equation associated with a word problem.
In the case of change problems, the query is regarding an unknown initial value or update or ﬁnal value.
The query predicate posed to the program is ?
wps(Tok Sentences, POS Sentences,
Quantities, Solution) , where Tok Sentences is the list of sentences/constituents in the word problem- each constituent represented as a list of words/tokens in it, POS Sentences is the list of part of speech tags of words in the problem 2 , Quantities is the list of numerical quantities mentioned in the word problem and ﬁnally, Solution - represents the solution of the word problem.
While training, the actual solution is provided in
Solution and the goal is to ﬁnd a proof for the posed query.
While testing, Solution will be assigned the solution predicted by the program.
The perception phase, as mentioned earlier, employs neural networks for predicate recognition and entity/container recognition.
These networks are represented by the neural predicates, nn vc (vc for verb categorizer as the verb in the sentence determines the predicate - holds/increase/decrease) taking as inputs the Sentence and Context and predicting Y over the domain [holds, increase, decrease] and nn wc (wc for word categorizer identifying whether the word is a container/entity/none) taking as inputs Sentence , Context and a speciﬁc Word and predicting Y over the domain [container, entity, none] .
The knowledge provided to the solver guides it to arrive at an equation and a solution.
We assume that the temporal order of events in the word problem is same as the order in which it is narrated .
This allows sequential processing of the sentences in the word problem.
The pseudocode of the knowledge provided is shown in Figure 3.
getRepresentation(Sentence) invokes the neural networks responsible for perception, through the neural predicates nn vc and nn wc , to obtain the intermediate representation of a sentence.
The solving process starts from the last sentence in the word problem, which is assumed to be the sentence posing the query ( query sentence in pseudocode).
The solver ﬁrst gets the representation of the query sentence ( holds(John, apples, ?
X) in case of Problem 1).
This gives information about the predicate associated with the query sentence ( pred query in pseudocode) and the container and entity of interest.
If the predicate is holds, then the unknown value is either the initial/ﬁnal value (ﬁnal value is queried for in case of Problem 1).
If the predicate is increase/decrease, then the unknown value queried for is the change or the amount by which the value got updated.
The for loop in the pseudocode indicates that the solver processes each sentence in the word problem, starting from the ﬁrst sentence - getting a representation for each sentence (a sentence is relevant if it is about the container and entity mentioned in the query).
Based on the predicate associated with each sentence ( pred = holds/increase/decrease), the numerical quantity associated with the sentence is appended into an accumulator ( Acc in pseudocode) directly or the value in the accumulator is increased or decreased by the quantity.
The processing for Problemis shown below:
Sentence 1: holds(John, apples, 10) − > Acc =
Sentence 2: increase(John, apples, 11) − > Acc =
[10+11]
Sentence 3 (Query): holds(John, apples, ?
X) − > Result =
Acc[0] =
DeepProbLog for Relation Recognition
To show the role of neuro-symbolic frameworks in addressing compositional generalization in vision, we have deﬁned a relation recognition task as the task of identifying images that involve at least a triangle intersecting with any other geometrical shape .
As in the case of word problem solving, we break down the task of visual relation recognition into two parts - perception and reasoning.
Perception
Given that the state-of-the-art deep learning methods in (multi) object detection, like MASK R-CNN (He et al.
, show good performance, we have simpliﬁed visual relation recognition into a classiﬁcation task that accepts as input visual scenes together with the bounding boxes of objects in the scene provided by a pre-trained object detection model like MASK R-CNN.
Since the type of shapes of the objects in the scene is important, we dedicated the perception part of our DeepProbLog model to shape recognition (see Figure 1).
More speciﬁcally, we have implemented a CNN-MLP neural model to classify input images intoclasses - circle , triangle and rectangle .
Reasoning
Figure 4 illustrates the same solving pipeline depicted in
Figure 3, however modiﬁed for relation recognition.
The query predicate is ?
scene(Images, BBoxes,
Solution) , where Images is the list of images, one for each object in the given input image, BBoxes is the list of bounding boxes of the objects and Solution represents the actual (during training) or the predicted class label (during testing) which is either True/False.
Given the input data, the perception phase employs a neural network for shape recognition.
The network is represented by the neural predicate, nn shape , taking as input one element of Images (an object) and predicting Y over the domain [circle, triangle, rectangle] .
The knowledge associated with the solving process identiﬁes the spatial relations between each pair of shapes and infers a suitable class label for the input image accordingly.
The pseudocode of the knowledge is given in Figure 4.
getPair(Images, BBoxes) gets the list of pairs of objects and their equivalent bounding boxes.
Within an iterative process over the list of pairs, getRepresentation(Image) invokes the shape recognition neural network for each object in a pair ( PI[0] and PI[1] ).
If there is at least one triangle predicted in a pair, the solver checks whether the pairs intersects or not using the information related to their bounding boxes ( PB[0] and PB[1] ).
It returns True if an intersect with a triangle is identiﬁed, and returns False otherwise.
Experiments
Here, we compare the state-of-the-art neural models for word problem solving and visual relation recognition with our DeepProbLog models.
Although it is an unfair comparison considering the additional knowledge support that DeepProbLog models possess, our intention is to show how the ability to explicitly represent and reason upon such knowledge helps in addressing generalization problems.
Word problem solving
Datasets As mentioned earlier, we collected 330 Change problems from the datasets, AddSub (Kushman et al. 2014) and SingleOp (Roy, Vieira, and Roth 2015).
Out of theproblems collected, 300 were single-update problems and 30 were two-update problems.
We created two test sets - the ﬁrst, Test similar had 30 single-update problems , the second, Test compositional had 30 two-update problems.
All the remaining ( 270 ) problems were used as the training set.
Baselines We compared the performance of
DeepProbLog-based solver against the state-of-the-art neural solvers.
The ﬁrst baseline, Graph2Tree (Zhang et al. 2020), uses a graph based encoder to encode the word problem and a tree-based decoder to generate the expression tree associated with the word problem.
The second baseline, is the G oal driven T ree S tructured neural solver - GTS (Xie and Sun 2019) that uses Long Short
Term Memory (LSTM) to encode the word problem, and a tree-based decoder to generate the equation.
The third baseline is a Sequence-to-Sequence - Seq2Seq network based on BiLSTM (Luong, Kayser, and Manning 2015), that translates the word problem into equation 3 .
Experimental Setup All the baseline models are provided with RoBERTa pretrained embeddings ( 768 d ), while DeepProbLog starts with random 256 d embeddings.
The embedding layer is trained along with other layers in the network.
It should be noted that, while all the baselines are trained with ( word problem, equation ) pairs, DeepProbLog is trained with ( word problem, final solution ) pairs and hence is weakly supervised.
But, the knowledge provided enables it to derive equations.
Results Table 1 shows performance of the solvers on the two test sets - Test similar and Test compositional .
As can be observed, all the baselines fail to solve any of the problems in Test compositional .
As mentioned earlier, all the solvers are trained with problems that are mapped to equations of the form ( X = N 1 + / − N 2) unlike the problems in Test compositional , which are mapped to equations of the form ( X = N 1 + / − N 2 + / − N 3) .
The baseline models fail to generalize to unseen equation forms/templates.
Since the perception module of DeepProbLog processes the word problem sentence-wise, learning a representation for each sentence and then composing these sentence representations based on the knowledge provided to derive the equation, DeepProbLog is able to perform better on Test compositional , solving 70 % of the problems correctly.
DeepProbLog could not beat the performance of
Graph2Tree or GTS on Test similar .
We believe this is because both Graph2Tree and GTS are trained on the equations directly (fully supervised), rather than on just the ﬁnal solution as done in DeepProbLog.
Visual Relation Detection
Datasets To evaluate state-of-the-art methods more accurately, we built our own dataset composed of images containing geometrical shapes for a simple classiﬁcation task.
Each image is labeled with True or False depending on whether there is a triangle (regardless of color, shape, size) intersecting with another shape in the image or not, respectively.
This allows us to form different set of data w.r.t different compositional conﬁgurations.
[Table] [Table] The dataset contains 5000 RGB images (100 × 100) composed of 3 different types of shapes including circle , triangle , rectangle , in 3 different colors - red , blue and green .
As an individual instance of data, each RGB image in our dataset is associated with a list of bounding boxes related to its shapes and a class label.
The training set which involves 80% of data (= 4000) is generated w.r.t the following constraints: (I)
It is an even 5050 split across the binary True/False class labels.
(II)
There are exactly two shapes in each image.
(III) 90% of the positive instances (i.e., a triangle intersects with another shape) in the training data comply with the following shape/position/color constraints: a red triangle is located at the left side of the second shape that is blue.
The remaining 10% of the positive instances do not follow the above constraint.
(IV)
We apply no shape/position/color constraints on negative (label =
False ) instances in the training set.
Given the above set of constraints for the training set, we have deﬁned 3 test sets, each of size 250, as follows:
• Test similar is the test set with the same distribution as the training set; • Test compositional cps is the test set where the number of shapes per image is as in the training set, however, instead of 90%, only 10% of the positive cases follow the color/position/shape (cps) constraint mentioned in (III).
• Test compositional total is the test set where the constraints (II, III) in the training set are violated (each image is composed of 3 shapes).
Baselines We compare the performance of DeepProbLog with 3 baselines.
The ﬁrst two are the standard neural-based networks - CNN-MLP and ResNet-50 (He et al. 2016).
As the third baseline we applied PrediNet (Shanahan et al.
2020) which, as mentioned earlier, is an end-to-end network that learns to directly map an image to propositions representing relations between objects in the image.
Experimental Setup
The convolutional network used in our CNN-MLP is composed of 2 parts: (1) a 3-layer CNN network as an encoder with batch normalization and ReLU activation function trained by stochastic gradient descent and ADAM optimizer, and (2) a two-layer fully connected
MLP with softmax activation function with output sizerepresenting the binary classes True and False in the dataset.
The perception part in our DeepProbLog implementation also uses the same conﬁguration as mentioned above.
We conﬁgured PrediNet with a single task setting where the task refers to our relation of interest: a triangle intersecting with another shape .
The structure of the PrediNet network evaluated in this work is equipped with a single convolutional input layer and an output multi-layer perceptron (MLP) resulting in a one-hot label denoting True or False.
The three aforementioned neural-based methods accept as input only images.
In order to have a fair comparison between these methods and DeepProblog model which also relies on information about the bounding-boxes, we ﬁrst preprocessed the data by augmenting each image with the bounding boxes of its shapes drawn on the image.
This can assist the neural based models to recognize shapes even when same-color shapes intersect.
Results Table 2 shows performance of the models on the 3 test sets - Test similar and Test compositional cps and
Test compositional total .
As can be observed, all the baselines show signiﬁcant drop in performance when tested on the compositional test sets.
It is not surprising to see this drop speciﬁcally in case of Test compositional total dataset as the baselines were trained on examples with images with just two objects.
Our intention is to show that DeepProbLog is able to generalize better to a variable number of objects without requiring to see such examples during training.
This is because DeepProbLog processes the input image objectwise, learning the shape of each object and then infers the label for the input image based on both the knowledge it has about the pattern of interest ( ‘Is there a triangle intersecting with another shape in the image?’ ) and the geometrical knowledge regarding intersection of bounding boxes.
Discussion & Future work
We focused on compositional generalization with respect to the two speciﬁc tasks of word problem solving and visual relation recognition.
Through an empirical evaluation, we showed that neuro-symbolic models are able to adapt better to distribution shifts as a result of compositionality by leveraging task speciﬁc knowledge.
It is worth mentioning that a neuro-symbolic approach can offer solutions when it is difﬁcult to collect more data but straightforward to represent knowledge useful for tackling generalization problems.
Our next step is to implement models based on other NeSy frameworks and compare them with each other.
Acknowledgments
The authors would like to thank Prof. Luc De Raedt for his insightful comments and Robin Manhaeve for his help with the DeepProbLog framework. This work was partially supported by the Wallenberg AI, Autonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation.