Learning Robust Rule Representations for Abstract Reasoning via Internal Inferences
Abstract
Abstract reasoning, as one of the hallmarks of human intelligence, involves collecting information, identifying abstract rules, and applying the rules to solve newproblems.
Although the neural networks have achieved human-level performancesin several tasks, the abstract reasoning techniques still far lag behind due to thecomplexity of learning and applying the logic rules, especially in an unsupervisedmanner.
In this work, we propose a novel framework, ARII, that learns rule representations for Abstract Reasoning via Internal Inferences .
The key idea is torepeatedly apply a rule to different instances in hope of having a comprehensiveunderstanding (i.e., representations) of the rule.
Specifically, ARII consists of arule encoder, a reasoner, and an internal referrer.
Based on the representationsproduced by the rule encoder, the reasoner draws the conclusion while the referrerperforms internal inferences to regularize rule representations to be robust andgeneralizable.
We evaluate ARII on two benchmark datasets, including PGM and
I-RAVEN.
We observe that ARII achieves the new state-of-the-art records on themajority of the reasoning tasks, including most of the generalization tests in PGM.
Introduction
Abstract reasoning, the ability to extract patterns and rules from concrete instances and apply themto solve new problems, is one of the hallmarks of human intelligence.
It is a critical topic to endowneural networks with the capacity of abstract reasoning on the road from artificial intelligence (AI) tohuman-like intelligence.
As “thinking in pictures” is one of the most effortless and natural ways forhumans to perform inference, Raven’s Progressive Matrices (RPM) test [1] is a widely accepted taskto evaluate AI’s reasoning ability.
As shown in Fig 1, a RPM problem contains an incomplete 3 × 3matrix where the bottom-right entry is missing.
The subjects are aware of that the rows in the matriximplicitly share identical rules and are asked to pick the suitable answer from the choices that wouldbest complete the missing entry.
Considering that the RPM test is shown to be highly correlated withreal intelligence [2] in cognitive and psychological science, AI community recently presents a hugeinterest in this task to explore the model’s capacity of abstract reasoning [3, 4, 5, 6, 7].
In the early years, computational models for RPM usually assume the access to the symbolicrepresentations of images and thus solve RPM with heuristics.
Recently, various neural networksare proposed to accomplish abstract reasoning with the help of abundant reasoning data (such as
Procedurally Generated Matrices dataset (PGM) [5], Relational and Analogical Visual rEasoNingdataset (RAVEN) [7], and Impartial-RAVEN dataset (I-RAVEN) [8]).
These models mainly treat the RPM problem as a classification task based on the rules hidden in rows or columns.
For example, Contrastive Perceptual Inference Network (CoPINet) [9] utilizes contrast effect to improve the featureextraction capacity [7].
Scattering Composition Learner (SCL) designs compositions of neuralnetworks according to the specific characteristics in RPM (e.g., the number of the attributes)
[Figure]
Rule
Although existing models have achieved impressive performance on abstract reasoning, the criticalcapacity, i.e. generalization to different environments remains unsolved [5, 11, 12, 13, 14].
Inparticular, Małki´nski [15] report that, the performances of the current models would significantlydecrease even if the test images changes slightly [5, 11, 12, 13].
For example, a generalization regime(i.e. , an evaluation task) in the PGM dataset is extrapolation, where the model is trained on the imageswith color or size restricted to the lower half of the value set and tested on images taking these valuesfrom the upper half.
Multi-Layer Relation Network (MLRN) [13] achieves nearly perfect accuracy inthe normal test of the PGM dataset, but fails completely in the extrapolation regime, yielding onlyslightly above chance.
This reveals that these models are just overfitted to the specific data ratherthan mastering the reasoning essence, reflected by low generalization to distracting features [16].
This paper provides a novel framework that ameliorates the reasoning generalizability by learninga robust rule representation.
To achieve it, we make two assumptions of the rule representation forabstract reasoning.
On the one hand, we assume the space of the abstract rules is finite and usuallynot large.
Thus, we propose a rule encoder that produces discrete rule representations by vectorquantisation [17].
On the other hand, we assume the rule representation should be invariant to specificinstances or distracting features.
In human reasoning, people usually apply an abstract rule to differentinstances for several times to deepen the understanding of the rule.
We mimic this cognitive processby introducing an internal inference process based on the learned rule representations.
Specifically,we randomly mask one panel of the first two rows and ask the referrer to infer the masked panelbased on the rule representation.
We perform the internal inference for multiple times to make therule representation invariant to specific instances.
Based on these instance-invariant representations,we apply another neural network to make conclusions according to the specific context images.
Therefore, we call our model ARII, the abbreviation of “abstract reasoning via internal inferences.”
We first conduct comprehensive empirical experiments on the I-RAVEN [8] datasets.
Compared withthe current state-of-the-art models (e.g., SCL [10] and CoPINet [9]), experiments show that ARIIachieves new records on four of the seven single training tasks in the I-RAVEN datasets.
As thegeneralization tests can better reflect the real reasoning ability of a model, we also evaluate ARII onthe seven generalization regimes of the PGM dataset and observe that ARII outperforms the othercompeting method in the five regimes.
To summarize, our contributions include:
• We propose a novel framework that learns rule representations for abstract reasoning by
internal inferences.
In particular, we introduce a rule encoder to produce discrete rule representations by vector quantisation.
We design an internal inference process to learn an
instance-invariant and robust rule representation.
• Our model achieves superior results to state-of-the-art methods on the majority of tasks (four out of seven) on I-RAVEN and better generalization performance on most PGM regimes.• Visualization results on the rule representations indicate that the learned representations
can be automatically clustered with respect to the rule categories.
Deep analysis shows that
we can identify some features in the discrete representations for a specific reasoning rule,
validating that the representations are interpretable and instance-invariant.
Related Work
RPM problems are firstly investigated in the cognitive science community to better understandintelligence, and many computational models are proposed to address this test since 1990s.
Forexample, Carpenter et al.
[2] develop a production system that takes hand-coded textual descriptionsof problems as input and predicts the answer.
Overall, these cognitive models mainly solve specificproblem sets 1 with only a limited number of hand-craft instances in cognitive science [19, 20, 21].
Recently, RPM problems have gained a lot of attention in the AI field to explore abstract reasoningcapability.
The PGM [5] together with the RAVEN dataset [7] expands the size of RPM instancesthrough automatic generation algorithms, serving as the benchmark datasets for deep learning networkto study abstract reasoning.
However, Hu et al.
[8] found severe defects (a short cut for predictions)existing in the RAVEN dataset and created an unbiased version called I-RAVEN dataset to solve it.
Therefore, we evaluate our method on the PGM and I-RAVEN rather than RAVEN.
Based on the above datasets, various end-to-end approaches have been proposed to study the abstractreasoning ability.
The typical architectures in computer vision, that simply use CNN as visual featureextractor followed by MLP [3] or LSTM [5, 22] to process the features from all the image panels,are shown to lack the reasoning ability and struggle in the RPM test.
To improve the reasoningcapacity, Relation Network [23] is equipped and extended in many models, including Wild Relation
Network [5] and Multi-scale Relation Network [24].
Although the above models obtain elegant performance on RAVEN and I-RAVEN, they still fail toachieve satisfactory generalization results on PGM, reflecting the overfitting issue of these reasoningmodels.
Auxiliary labels [25, 11, 16] are therefore utilized to develop representations that areamenable to generalization by informing the meta-targets (i.e., labels which encode relevant relationsand attributes).
However, it requires additional prior information, which is not a general way toaddress the reasoning problem.
Our work is closely related to several advanced abstract reasoners, such as the SCL [10] and
MXGNet [11], a multi-layer multiplex graph neural net. In particular, ARII uses the splitting andgrouping embeddings and row-wise embeddings in SCL and MXGNet, respectively. Our workis also connected with neuro-symbolic methods for logical reasoning [26, 27, 28, 29, 30]. Forinstance, Logic Tensor Networks integrates neural networks with first-order fuzzy logic to performreasoning with logical formulas.
Different from these studies, ARII explicitly maintains a discreterule representation for abstract reasoning and introduces a novel internal inference process to enforcethe rule representation to be robust.
Methodology
3.1 Problem formulation
The RPM test is conducted on a 3 × 3 matrix where the rows or columns comply with an identicalrule.
Subjects are presented with an incomplete matrix where an entry on the third row is missing,and are asked to select a candidate answer from the choice set to complete the matrix.
Formally speaking, a RPM problem contains 16 image panels
X
= { x i } 16 i =1 divided into 8 contextimages X c
= { x i } 8 i =1 and 8 candidate answer images
X a = { x i } 16 i =9 .
The 8 context images areplaced in a 3 × 3 matrix with the last element blank.
For each particular data point, the three rows inthe matrix implicitly share the identical abstract rule.
Given the X , the machine is asked to select acandidate answer image from X a to best complete the matrix (i.e. adhering to the underlying abstractrule in context images).
The correct answer is denoted by x ∗ , where x ∗ ∈
X a .
3.2 The ARII model
ARII consists of three modules: the rule encoder, the reasoner, and the internal inferrer.
[-The
 Standard Progressive Matrices [18] contains 60 problems rule encoder-]The ruleencoder induces the rule representation from the context matrix.
The inferrer performs internalinferences to regularize the rule representations to be robust and instance invariant.
The reasoner 60 problems draws the reasoning conclusion based on the rule representations.
The rest of this section elaborateson these components and the training process.
3.3 Rule encoder
To tackle this reasoning problem, an ideal reasoning process is to figure out the underlying rule basedon the given context images.
To achieve this, we aim to learn the rule representations r using aneffective encoder.
In principle, the rule can be extracted from particular row of images.
But the aboveextraction has the potential to make mistakes since a single row of images may satisfy multiple rules.
Therefore, to guarantee the adequate information for rule extraction, we propose to encode the firsttwo rows of images to represent the underlying rule, given by
[Equation]
where r 1&2 stands for the representation of the rule underlying the first two rows in the given matrix.
Rule-Enc stands for the rule encoder in the ARII model.
In the rest of this section, we first obtain theimage encodings and then build rule representations based on the encodings of the involved images.
Image encodings.
As each row contains three separated images, we adopt convolutional neuralnetwork (CNN) blocks to get the feature map ( f ) for each image separately.
[Equation]
Once we get the feature map, we further fuse the features of different positions in the feature map.
Intuitively, the rule of RPM is highly correlated with visuospatial ability [31] and usually lies in thesame position of different panels.
Therefore, we first divide the feature map into multiple groups;each group can be considered to contain information from a certain position.
Then we apply a fullyconnected layer (FC) to each group to further extract group features, and concatenate the groupfeatures into output features, computed by
[Equation]
where || stands for the concatenation operation of the vectors.
f ki is k -th group features after splittingoriginal
f i into K z groups.
Similar to the scattering transformation in SCL [10], the group features{ f ki } are divided evenly from the feature map f i .
Finally, the K z new groups of the features areconcatenated (denoted by || K z k ) to obtain the i -th image encodings z i .
Continuous rule representations.
Next, the rule encoder fuses the image encodings and extracts therule representations.
Similar to the processing of images, we extract the relation corresponding to thedifferent positions of the six image encodings as Fig 2.
Specifically, the encodings of six images arefirst divided into K r groups independently.
The groups at the same location in six image encodingsare combined together and fed into a fully connected layer to further extract the relation feature.
Finally, the multiple relation features are merged to form rule features as follows:
[Equation]
where z ki is the k -th group of the i -th image encodings z i .
||
6 i =
z ki means six k -th group featuresfrom embeddings are concatenated, and fed into a fully connected layer (FC) to get the k -th relationfeatures.
We further merge the K r relation features and obtain the continuous rule representations r c .
Discrete rule representations.
Note that the above rule representations r c are continuous anddependent on the specific instances.
In this work, we suppose that the underlying rules are independentof specific instances and can be reused in different problems that share the same rules.
Based on thisassumption, the discrete and finite representations are more suitable to describe abstract rules.
Therefore, we propose to discretize the latent representations of the rules via vector quantisation (VQ)[17].
The general idea of VQ is to use a finite number of vectors (denoted as codebook) to build therule space.
It outputs the closest vector from the codebook for the given input (Appendix Fig 1).
Inparticular, the rule encoder maintains a discrete codebook E ∈ R K
e
× D .
K e is the number of thediscrete code, and D is the dimensionality of each discrete code E k .
We measure the L2 distancebetween the continuous representation r c and code vectors in the codebook.
For each vector r l c inr c ∈ R K r × D , where K r is the number of the continuous vector and D is the dimensionality of eachcontinuous vector r lc , the code vector that yields the minimum distance is taken to obtain the discreterule representation q l . The detailed computations are [Equation] where
[Equation]
Rule Encoder 𝑥𝑥 5 𝑥 6 𝑥𝑥 2 𝑥
FC 𝑬 𝟏 𝑬𝑬 𝑲𝒆 ••• 𝒓 𝑐 𝑬 𝒓 1& 𝑥𝑥 5 𝑥 6 𝑥𝑥 2 𝑥𝑥 8 ?
𝑥𝒓 1& 𝒓 2& 𝒓 1& 𝒒
Sim(:, :)
Sim(:, :) × 1/ × 1/ ⊕ Similarity Score
Prediction 𝑥𝑥 5 𝑥 6 𝑥𝑥
Image Encoder
Mask
Flatten
Concat
Inferrer ෤𝑥𝑠 9 𝑠 10 𝑠
𝑠𝑠 13 𝑠 14 𝑠 15 𝑠𝑥 9 𝑥 10 𝑥 11 𝑥𝑥 13 𝑥 14 𝑥 15 𝑥
Image Encoder
Rule Encoder Rule Encoder
Internal inference
Rule encoder
[Figure]
where q l is the closest quantized vector for the continuous vector r lc .
Via vector quantisation, we canfind the closest code for the given continuous rule representations, i.e.,
[Equation]
To make the above vector quantisation work, we need to train both the neural networks and thecodebook in the rule encoder towards the minimum of the distance between the continuous anddiscrete representations.
The objective function of vector quantisation for a particular data point is
[Equation]
(7) where tru( · ) performs the gradient truncation during forward computation.
In this way, we can derivethe discrete rule representations, which is expected to be interpretable and instance-invariant.
3.4 Reasoner
The reasoner is to draw conclusions (i.e., the selection of answer images) based on the extracted ruleand the specific context (i.e., the images on the third row).
That is, the reasoner applies the extractedrule to the third row and figures out which candidate answer could best complete the missing entry.
According to this standard, we therefore extract the rule that the third row implicates.
Since we have8 candidate answer images (i.e., X a ), there are 8 rules for the third row with the different answerimages in the missing entry.
However, the rule encoder is designed to take two rows of images asinputs and yield the rule representations.
Therefore, we combine the third row with the first row toderive the rule that these two rows implicitly share, computed by
[Equation]
where r j 1&3 denotes the rule representations of the rule underlying the first and the third row when thethird row is completed with the j -th answer image.
Similarly, we can obtain the rule representationsof the rule underlying the second and third row, given by
[Equation]
Based on the derived rule representations for the third row, the reasoner next measures the similaritybetween these rule representations and the reference rule (i.e., r 1&2 ) and selects the most similar oneas the prediction.
In particular, the reasoner measures the similarity of two rules by calculating theinner product of the corresponding representations.
For example, the similarity between the rule r 1&3and the reference rule is calculated by
[Equation]
where sim( · ) is the similarity function of two rule representations.
Similarly, the reasoner derives thesimilarity between the rule r 2&3 and the reference rule by
[Equation]
Based on the similarities between individual rules, the reasoner then predicts the candidate answer y .
[Equation]
[Equation]
For the optimization of the reasoner, we adopt the cross entropy loss to guide the reasoner makeselect the correct answer images, given by
[Equation]
3.5 Internal inferences
Human usually applies an abstract rule to different instances for several times to deepen the understanding of the rule.
We mimic this cognitive process by performing internal inferences based on therule representations.
In the internal inference, we introduce another reasoner to address the reasoningtask on the two rows of images when the rule representations are available.
In this way, our method
ARII could also better “understand” the rule and have more robust rule representations.
A robust rulerepresentation is insensitive to the sample noises and thus benefits the overall reasoning ability[32].
In particular, the first and second rows are used in internal inference process.
We first extract the rulerepresentations r 1&2 of these rows and then randomly mask one of the six images { x i } 6 i =1 by a whiteblank image.
We coin the blank image as ˆ x m , where m is the index of the masked images.
Thus, themasked two rows can be expressed by
[Equation]
) where { 1 , 2 , 3 , 4 , 5 , 6 }\ m stands for a set excluding the index m and we insert blank image ˆ x m inthe original m -th position.
Next, we introduce another neural network named inferrer that takes therule representations and the encodings of the images in the masked two rows as inputs.
[Equation]
) where Inferrer is another CNN model.
˜ x m stands for the predicted image for the incomplete rows,and ImageEnc ( I m ) derives the encoding of I m by Equation 2 and 3.
To avoid using additional datain our internal inference, we formulate the above prediction as to the generation process.
That is tosay, the inferrer aims to generate the masked image x m at the pixel level.
In addition, we repeat the above internal inferences for 6 times by masking each image to learn morerobust rule representations.
We adopt mean squared error (MSE) as the objective function to optimizethe inferrer as well as the rule encoder.
In this way, we obtain the overall objective function J if bycollecting the predictions of these internal inference processes, given by
[Equation]
3.6 Optimization of ARII
As introduced in the section on problem formulation, the major goal of our method ARII is to select acandidate answer image for the incomplete matrix as accurately as possible (as shown in Equation 13).
In the meanwhile, we assume the rule representations are discrete and can be reused in internalinferences.
That is, apart from the major goal, we have two additional objectives, i.e., the objective ofthe vector quantisation process and the objective of the internal inferences.
Therefore, we combinethese three objectives together for the optimization of the individual components of ARII, given by
[Equation]
17) where the context images X c is parts of the given data point X
and D is the dataset.
λ
re , λ vq and λ ifare the three hyper-parameters that coordinates the importance of the components in ARII.
[Table]
Experiment
4.1 Datasets
PGM.
Each matrix in PGM is governed by abstract rules, which are sampled from a rule set
R = { [ s, o, a ] : s ∈ S , o ∈
O , a ∈ A} , where S , O , A are primitive sets of relations, objects andattributes, respectively.
S =
{ progression, XOR, OR, AND, consistent union } , O = { shape, line } ,
A = { size, type, color, position, number } .
There are eight regimes in PGM.
The easiest one iscalled neutral where the training and test set are sampled from the same distribution
[Table]
and theothers are generalization regimes where the training and test data differ in the pre-defined ways.
The generalization regimes consists of seven variants: interpolation, extrapolation, held-out triples(H.O.Triples), held-out attribute pairs (H.O.A.P.), held-out pairs of triples (H.O.T.P.), held-out shapecolour (H.O.S-C) and held-out line-type (H.O.L-T).
More details can be found in Barrett et al.
RAVEN and I-RAVEN.
The RAVEN and I-RAVEN datasets are extensions to PGM.
They sharean identical set of attributes but differ in the following attributes: progression, constant, union andarithmetic.
I-RAVEN is adapted from RAVEN dataset for its biased answer sets which may lead to ashortcut solution that only candidate answers can yield accurate answers [8].
In our experiments, wefirst evaluate the models’ reasoning ability I-RAVEN since it is easier than PGM.
4.2 Competing methods and implementation details
We compare our model with several state-of-the-art models, including CNN + LSTM [5], ResNetbased [33] image classifier, Wild ResNet [5], Wild Relation Network (WReN) [5], Contrastive
Perceptual Inference network (CoPINet) [9], Logic Embedding Network (LEN) [16], Scattering
Compositional Learner (SCL) [10], Stratified Rule-Aware Network (SRAN) [8], Multi-scale Relation
Network (MRNet) [24], and Multi-Layer Relation Network (MLRN) [13].
Note that our model istrained end to end and needs solely the ground truth answer label without auxiliary information.
Wereport the result of the baseline models without auxiliary loss for fair comparison.
Please refer to
Appendix A for more implementation details of ARII.
4.3 Results
Table 1 shows the test performance of individual models on I-RAVEN dataset.
We observe that thetypical architectures in computer vision, that simply uses CNN as visual feature extractor followedby MLP or LSTM, fail to yield satisfactory reasoning results.
More advanced models for abstractreasoning such as LEN and SCL significantly improve the performance.
In particular, the previouslystate-of-the-art model SCL achieves nearly 100% accuracy on the configurations of Center, L-R, U-D,and O-IC, but SCL could not make similar success on the other configurations.
We conjecture thatthe reasoning complexity of these four configurations are relative low and easy.
As for our method,
ARII performs similarly to SCL on the above four configurations but largely outperforms it on theother three more difficult configurations, leading to a notably higher average score on I-RAVEN.
In addition, Table 2 presents the performance on the PGM dataset.
For the neutral regime, our modelachieves decent result that is better than majority of baseline models, although it does not outperformstate-of-the-art models such as MRNet and MLRN.
Note that MLRN achieves near perfect accuracyin the neutral regime due to the sophisticated design which could not applied to other problems.
More importantly, neutral regime is not our main focus of metric, since it could hardly indicate thereasoning ability and generalization towards real intelligence. [Table] [Table]
PGM also allows us to investigate the generalization capacity of the models under various regimes.
We observe from Table 2 that although MLRN achieves superior performance on neutral, it isprone to overfitting and performs poorly on the interpolation and extrapolation regimes, clearlyindicating MLRN is overfitted to the neutral setting.
On the contrary, the proposed internal inferencein ARII severs as the regularities of the rule representations, making the rule representations robustand instance-invariant.
As a result, our model outperforms all other methods on five of the sevengeneralization regimes in PGM dataset.
In particular, the accuracy of ARII increases by a sizablemargin over the baselines in the held-out attribute pairs and extrapolation regimes (above 20% andnearly 10% respectively), further validating the superior general ability of abstract reasoning.
4.4 Ablation study
We analyze ARII in more detail to validate the effectiveness of the proposed components on tworegimes in the PGM dataset (Table 3).
In the interpolation regime, the performance of ARII withoutdiscretization decreases largely by around 10%, which shows the discrete representation plays itsrole in generalization of abstract reasoning.
The performance is also largely declined by around 30%when the internal inference is removed, suggesting that the internal inference is vital to learn a morerobust rule representation for abstract reasoning.
Similar results can be found in H.O.A.P. regime.
We further conduct ablation studies to investigate the impact of generation in internal inference,influences of transpose matrices, the combinatorial benefit of r 1&3 and r 2&3 in the reasoner, andeffectiveness of the internal inference to other models. Please see Appendix B-E for more discussion.
4.5 Visualization of rule representations
Then, we try to validate whether ARII has learned the instance-invariant representations.
We firstobtain all the rule representations in the test set of the interpolation regime of PGM, and use therepresentations derived from the problems which contain only a single rule (i.e., a triple { [ s, o, a ] } ).
The rule representations are projected to 2D space by t-distributed stochastic neighbor embedding(t-SNE) [36] in Fig 3.
We observe that the rule representations are clustered according to individualrule triples.
Note that these clusters for each rule are formed without supervision of the rule labels.
And the representations are extracted from the generalization test set which has a different distributionfrom the training set.
The clustered representations show that these rule representations are universaland independent to the specific instances, even to the out-of-distribution brand new instances.
Taking one step further, we investigate whether the discrete rule representation is interpretable.
Inparticular, we want to see the composition of the representations for each specific rule.
Here, we takethe rules related to the line object and the color attribute (totally five rules, i.e., [ s ∈ S , line, color], [Figure] denoted by the [ · ,line, color] rules) as an example.
We first collect the rule representations from allthe instances governed by these rules, which are sampled from the codebook E .
Then we calculatethe selected frequencies of the codes and visualize their distributions.
Fig 4 (a-e) shows that all thefive rules share the same codes E 358 and E 181 in their top-3 codes, indicating that the codes E 358 and
E 181 are very relevant to the [ · ,line, color] rules.
In addition, these codes are also specific to theserules, since they rarely appear in the dissimilar rules such as the rule [progression, shape, size] (Fig 4(f)).
These results reveal that the rule representations learned by ARII are moderately interpretable.
Limitations
Reasoning is the ability that consciously applies logic from premises to conclusion [37, 38].
Forneural networks, they use different layers of the distributed neurons to implicitly represent thepremises, the logic and the temporal results, and formulate the final conclusion-making process asthe classification or generation task.
Therefore, neural networks lack in explaining how they performinduction and reasoning.
Although our method takes one step further to build interpretable rulerepresentations, it is still far from the goal of a fully interpretable, robust, and accurate reasoner forgeneral reasoning.
Another limitation of this work is the image encoder, which relies on the rigidspatial structure and is difficult for learning new rules in other abstract reasoning problems.
Thegeneral object detector pretrained on natural images is a potential solution.
Conclusion
In this work, we propose a novel architecture ARII that learns robust rule representations for abstractreasoning via internal inferences.
Experiments on benchmark datasets (I-RAVEN and PGM) showthat ARII outperforms previous state-of-the-art methods without using auxiliary annotations on themajority of the reasoning tasks.
Moreover, the rule representations of ARII present meaningful andinterpretable characteristics, facilitating the exploration of the composition of the rules in future work.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] Our model has improved thegeneralization performance by a notable margin on majority of tests in PGM, butfurther research is needed to reach the human level of generalization.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A](d)
Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[N/A]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] We haveuploaded the code in the supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] See Appendix A.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[No]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] The datasets used in
this work are cited in [5] and [8].(b)
Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]