Learning Robust Rule Representations for Abstract Reasoning via Internal Inferences
Abstract
Abstract reasoning, as one of the hallmarks of human intelligence, involves collecting information, identifying abstract rules, and applying the rules to solve new problems.
Although neural networks have achieved human-level performances in several tasks, the abstract reasoning techniques still far lag behind due to the complexity of learning and applying the logic rules, especially in an unsupervised manner.
In this work, we propose a novel framework, ARII, that learns rule representations for Abstract Reasoning via Internal Inferences .
The key idea is to repeatedly apply a rule to different instances in hope of having a comprehensive understanding (i.e., representations) of the rule.
Specifically, ARII consists of a rule encoder, a reasoner, and an internal referrer.
Based on the representations produced by the rule encoder, the reasoner draws the conclusion while the referrer performs internal inferences to regularize rule representations to be robust and generalizable.
We evaluate ARII on two benchmark datasets, including PGM and I-RAVEN.
We observe that ARII achieves new state-of-the-art records on the majority of the reasoning tasks, including most of the generalization tests in PGM.
Our codes are available at https://github.com/Zhangwenbo0324/ARII .
Introduction
Abstract reasoning, the ability to extract patterns and rules from concrete instances and apply them to solve new problems, is one of the hallmarks of human intelligence.
It is a critical topic to endow neural networks with the capacity of abstract reasoning on the road from artificial intelligence (AI) to human-like intelligence.
As “thinking in pictures” is one of the most effortless and natural ways for humans to perform inference, Raven’s Progressive Matrices (RPM) test [1] is a widely accepted task to evaluate AI’s reasoning ability.
As shown in Fig 1, an RPM problem contains an incomplete 3 ×matrix where the bottom-right entry is missing.
The subjects are aware that the rows in the matrix implicitly share identical rules and are asked to pick the suitable answer from the choices that would best complete the missing entry.
Considering that the RPM test is shown to be highly correlated with real intelligence [2] in cognitive and psychological science, AI community recently presents a huge interest in this task to explore the model’s capacity of abstract reasoning [3, 4, 5, 6, 7].
In the early years, computational models for RPM usually assume access to the symbolic representations of images and thus solve RPM with heuristics.
Recently, various neural networks are proposed
[Figure]
x t
[Figure]
a t
[Figure]
i
[Figure]
Rule to accomplish abstract reasoning with the help of abundant reasoning data (such as Procedurally Generated Matrices dataset (PGM) [5], Relational and Analogical Visual rEasoNing dataset (RAVEN) [7], and Impartial-RAVEN dataset (I-RAVEN) [8]).
These models mainly treat the RPM problem as a classification task based on the rules hidden in rows or columns.
For example, Contrastive Perceptual
Inference Network (CoPINet) [9] utilizes contrast effect to improve the feature extraction capacity [7].
Scattering Composition Learner (SCL) designs compositions of neural networks according to the specific characteristics in RPM (e.g., the number of the attributes)
Although existing models have achieved impressive performance on abstract reasoning, the critical capacity, i.e. generalization to different environments remains unsolved [5, 11, 12, 13, 14].
In particular, Małki´nski [15] reports that, the performances of the current models would significantly decrease even if the test images changes slightly [5, 11, 12, 13].
For example, one of the generalization regimes (i.e., an evaluation task) in the PGM dataset is extrapolation, where the model is trained on the images with color or size restricted to the lower half of the value set and tested on images taking these values from the upper half.
Multi-Layer Relation Network (MLRN)
[13] achieves nearly perfect accuracy in the normal test of the PGM dataset, but fails completely in the extrapolation regime, yielding only slightly above chance.
This reveals that these models are just overfitted to the specific data rather than mastering the reasoning essence, also reflected by low generalization to distracting features [16].
This paper provides a novel framework that ameliorates the reasoning generalizability by learning robust rule representations.
To achieve it, we make two assumptions of the rule representation for abstract reasoning.
On the one hand, we assume the space of the abstract rules is finite and usually not large.
Thus, we propose a rule encoder that produces discrete rule representations by vector quantisation [17].
On the other hand, we assume the rule representations should be invariant to specific instances or distracting features.
In human reasoning, people usually apply an abstract rule to different instances several times to deepen the understanding of the rule.
We mimic this cognitive process by introducing an internal inference process based on the learned rule representations.
Specifically, we randomly mask one panel of the first two rows and ask the referrer to infer the masked panel based on the rule representation.
We perform the internal inference multiple times to make the rule representation invariant to specific instances.
Based on these instance-invariant representations, conclusions can be made according to the specific context images.
Therefore, we call our model ARII, the abbreviation of “abstract reasoning via internal inferences.”
We first conduct comprehensive empirical experiments on the I-RAVEN [8] datasets.
Compared with the current state-of-the-art models (e.g., SCL [10] and CoPINet [9]), experiments show that ARII achieves new records on four of the seven single training tasks in the I-RAVEN datasets.
As the generalization tests can better reflect the real reasoning ability of a model, we also evaluate ARII on the seven generalization regimes of the PGM dataset and observe that ARII outperforms the other competing method in the five regimes.
To summarize, our contributions include:
• We propose a novel framework that learns rule representations for abstract reasoning by internal inferences.
In particular, we introduce a rule encoder to produce discrete rule representations by vector quantisation.
We design an internal inference process to learn an instance-invariant and robust rule representation.
• Our model achieves superior results to state-of-the-art methods on the majority of tasks (four out of seven) on I-RAVEN and better generalization performance on most PGM regimes.
• Visualization results on the rule representations indicate that the learned representations can be automatically clustered with respect to the rule categories.
Deep analysis shows that we can identify some features in the discrete representations for a specific reasoning rule, validating that the representations are interpretable and instance-invariant.
Related Work
RPM problems are firstly investigated in the cognitive science community to better understand intelligence, and many computational models are proposed to address this test since 1990s.
For example, Carpenter et al.
[2] develop a production system that takes hand-coded textual descriptions of problems as input and predicts the answer.
Overall, these cognitive models mainly solve specific problem sets 2 with only a limited number of hand-craft instances in cognitive science [19, 20, 21].
Recently, RPM problems have gained a lot of attention in the AI field to explore abstract reasoning capability.
The PGM [5] together with the RAVEN dataset [7] expands the size of RPM instances through automatic generation algorithms, serving as the benchmark datasets for deep learning network to study abstract reasoning.
However, Hu et al.
[8] found severe defects (a short cut for predictions) existing in the RAVEN dataset and created an unbiased version called I-RAVEN dataset to solve it.
Therefore, we evaluate our method on the PGM and I-RAVEN rather than RAVEN.
Based on the above datasets, various end-to-end approaches have been proposed to study the abstract reasoning ability.
The typical architectures in computer vision, that simply use CNN as visual feature extractor followed by MLP [3] or LSTM [5, 22] to process the features from all the image panels, are shown to lack the reasoning ability and struggle in the RPM test.
To improve the reasoning capacity, Relation Network [23] is equipped and extended in many models, including Wild Relation
Network [5] and Multi-scale Relation Network [24].
Although the above models obtain elegant performance on RAVEN and I-RAVEN, they still fail to achieve satisfactory generalization results on PGM, reflecting the overfitting issue of these reasoning models.
Auxiliary labels [25, 11, 16] are therefore utilized to develop representations that are amenable to generalization by informing the meta-targets (i.e., labels which encode relevant relations and attributes).
However, it requires additional prior information, which is not a general way to address the reasoning problem.
Our work is closely related to several advanced abstract reasoners, such as the SCL [10] and
MXGNet [11], a multi-layer multiplex graph neural net.
In particular, ARII uses the splitting and grouping embeddings and row-wise embeddings in SCL and MXGNet, respectively.
Our work is also connected with neuro-symbolic methods for logical reasoning [26, 27, 28, 29, 30, 31, 32].
For instance, Logic Tensor Networks integrates neural networks with first-order fuzzy logic to perform reasoning with logical formulas.
Different from these studies, ARII explicitly maintains a discrete rule representation for abstract reasoning and introduces a novel internal inference process to enforce the rule representation to be robust.
The internal inference can be viewed as one type of self-supervised learning algorithm for the rule representation learning, where we mask parts of the context matrix and require the model to reconstruct it.
But the traditional self-supervised learning models are usually trained first and finetuned for the downstream tasks afterwards [33, 34, 35].
Different from them, we explore an interesting strategy that performs the internal inference and the supervised learning simultaneously.
Methodology 3.1 Problem formulation
The RPM test is conducted on a 3 × 3 matrix where the rows comply with an identical rule.
Subjects are presented with an incomplete matrix where an entry on the third row is missing, and are asked to select a candidate answer from the choice set to complete the matrix.
Formally speaking, a RPM problem contains 16 image panels
X
= { x i } 16 i =1 divided into 8 context images X c
= { x i } 8 i =1 and 8 candidate answer images
X a = { x i } 16 i =9 .
The 8 context images are placed in a 3 × 3 matrix with the last element blank.
For each particular data point, the three rows in the matrix implicitly share the identical abstract rule.
Given the X , the machine is asked to select a candidate answer image from X a to best complete the matrix (i.e. adhering to the underlying abstract rule in context images).
The correct answer is denoted by x ∗ , where x ∗ ∈
X a .
3.2 The ARII model
ARII consists of three modules: the rule encoder, the reasoner, and the internal inferrer.
The rule encoder induces rule representations from the context matrix.
The inferrer performs internal inferences to regularize the rule representations to be robust and instance invariant.
The reasoner draws the reasoning conclusion based on the rule representations.
The rest of this section elaborates on these components and the training process.
3.3 Rule encoder
To tackle this reasoning problem, an ideal reasoning process is to figure out the underlying rule based on the given context images.
To achieve this, we aim to learn the rule representations r using an effective encoder.
In principle, the rule can be extracted from particular row of images.
But the above extraction has the potential to make mistakes since a single row of images may satisfy multiple rules.
Therefore, to guarantee the adequate information for rule extraction, we propose to encode the first two rows of images to represent the underlying rule, given by
[Equation]
where r 1&2 stands for the representation of the rule underlying the first two rows in the given matrix.
Rule-Enc stands for the rule encoder in the ARII model.
In the rest of this section, we first obtain the image encodings and then build rule representations based on the encodings of the involved images.
Image encodings.
As each row contains three separated images, we adopt convolutional neural network (CNN) blocks to get the feature map ( f ) for each image separately.
f
[Equation]
Once we get the feature map, we further fuse the features of different positions in the feature map.
Intuitively, the rule of RPM is highly correlated with visuospatial ability [36] and usually lies in the same position of different panels.
Therefore, we first divide the feature map into multiple groups; each group can be considered to contain information from a certain position.
Then we apply a fully connected layer (FC) to each group to further extract group features, and concatenate the group features into output features, computed by
[Equation]
where || stands for the concatenation operation of the vectors.
f ki is k -th group features after splitting original f
i into K z groups.
Similar to the scattering transformation in SCL [10], the group features { f ki } are divided evenly from the feature map f i .
Finally, the K z new groups of the features are concatenated (denoted by || K z k ) to obtain the i -th image encodings z i .
Continuous rule representations.
Next, the rule encoder fuses the image encodings and extracts the rule representations.
Similar to the processing of images, we extract the relation corresponding to the different positions of the six image encodings as Fig 2.
Specifically, the encodings of six images are first divided into K r groups independently.
The groups at the same location in six image encodings are combined together and fed into a fully connected layer to further extract the relation feature.
Finally, the multiple relation features are merged to form rule features as follows:
[Equation]
where z ki is the k -th group of the i -th image encodings z i .
||
6 i =
z ki means six k -th group features from encodings are concatenated, and fed into a fully connected layer (FC) to get the k -th relation features.
We further merge the K r relation features and obtain the continuous rule representations r c .
Discrete rule representations.
Note that the above rule representations r c are continuous and dependent on the specific instances.
In this work, we assume the space of the abstract rules is finite
Rule Encoder 𝑥𝑥 5 𝑥 6 𝑥𝑥 2 𝑥
FC 𝑬 𝟏 𝑬𝑬 𝑲𝒆 ••• 𝒓 𝑐 𝑬 𝒓 1& 𝑥𝑥 5 𝑥 6 𝑥𝑥 2 𝑥𝑥 8 ?
𝑥𝒓 1& 𝒓 2& 𝒓 1& 𝒒
Sim(:, :)
Sim(:, :) × 1/ × 1/ ⊕
Similarity Score
Prediction 𝑥𝑥 5 𝑥 6 𝑥𝑥
Image Encoder
Mask
Flatten
Concat
Inferrer ෤𝑥𝑠 9 𝑠 10 𝑠
𝑠𝑠 13 𝑠 14 𝑠 15 𝑠𝑥 9 𝑥 10 𝑥 11 𝑥𝑥 13 𝑥 14 𝑥 15 𝑥
Image Encoder
Rule Encoder Rule Encoder
Internal inference
Rule encoder
[Figure]
and usually not large.
Based on this assumption, the discrete and finite representations are more suitable to describe abstract rules.
Therefore, we propose to discretize the latent representations of the rules via vector quantisation (VQ) [17].
The general idea of VQ is to use a finite number of vectors (denoted as codebook) to build the rule space.
It outputs the closest vector from the codebook for the given input (Appendix Fig 1).
In particular, the rule encoder maintains a discrete codebook E ∈ R K
e
× D .
K e is the number of the discrete code, and D is the dimensionality of each discrete code E k .
We measure the L2 distance between the continuous representation r c and code vectors in the codebook.
For each vector r lc in r c ∈ R K r × D , where K r is the number of the continuous vector and D is the dimensionality of each continuous vector r lc , the code vector that yields the minimum distance is taken to obtain the discrete rule representation q l .
The detailed computations are
[Equation]
where q l is the closest quantized vector for the continuous vector r lc .
Via vector quantisation, we can find the closest code for the given continuous rule representations, i.e.,
[Equation]
To make the above vector quantisation work, we need to train both the neural networks and the codebook in the rule encoder towards the minimum of the distance between the continuous and discrete representations.
The objective function of vector quantisation for a particular data point is
[Equation]
where sg( · ) is stop gradient operator during computation.
In this way, we can derive the discrete rule representations, which is expected to be interpretable and instance-invariant.
3.4 Reasoner
The reasoner is to draw conclusions (i.e., the selection of answer images) based on the extracted rule and the specific context (i.e., the images on the third row).
That is, the reasoner applies the extracted rule to the third row and figures out which candidate answer could best complete the missing entry.
According to this standard, we therefore extract the rule that the third row implicates.
Since we have 8 candidate answer images (i.e., X a ), there are 8 rules for the third row with the different answer images in the missing entry.
However, the rule encoder is designed to take two rows of images as inputs and yield the rule representations.
Therefore, we combine the third row with the first row to derive the rule that these two rows implicitly share, computed by
[Equation]
where r j 1&3 denotes the rule representations of the rule underlying the first and the third row when the third row is completed with the j -th answer image.
Similarly, we can obtain the rule representations of the rule underlying the second and third row, given by
[Equation]
Based on the derived rule representations for the third row, the reasoner next measures the similarity between these rule representations and the reference rule (i.e., r 1&2 ) and selects the most similar one as the prediction.
In particular, the reasoner measures the similarity of two rules by calculating the inner product of the corresponding representations.
For example, the similarity between the rule r 1& and the reference rule is calculated by
[Equation]
where sim( · ) is the similarity function of two rule representations.
Similarly, the reasoner derives the similarity between the rule r 2&3 and the reference rule by
[Equation]
Based on the similarities between individual rules, the reasoner then predicts the candidate answer y .
[Equation]
For the optimization of the reasoner, we adopt the cross entropy loss to guide the reasoner to select the correct answer, given by
[Equation]
3.5 Internal inferences
Human usually applies an abstract rule to different instances for several times to deepen the understanding of the rule.
We suppose that the underlying rules are independent of specific instances and can be reused in different problems that share the same rules.
Therefore we mimic this cognitive process by performing internal inferences based on the rule representations.
In the internal inference, we introduce another reasoner to address the reasoning task on the two rows of images when the rule representations are available.
In this way, our method ARII could also better “understand” the rule and have more robust rule representations.
A robust rule representation is insensitive to the sample noises and thus benefits the overall reasoning ability[37].
In particular, the first and second rows are used in internal inference process.
We first extract the rule representations r 1&2 of these rows and then randomly mask one of the six images { x i } 6 i =1 by a white blank image.
We coin the blank image as ˆ x m , where m is the index of the masked images.
Thus, the masked two rows can be expressed by
[Equation]
[Equation]
where { 1 , 2 , 3 , 4 , 5 , 6 }\ m stands for a set excluding the index m
and we insert blank image ˆ x m in the original m -th position.
Next, we introduce another neural network named inferrer that takes the rule representations and the encodings of the images in the masked two rows as inputs. [Equation]
[Equation]
where Inferrer is another CNN model.
˜ x m stands for the predicted image for the incomplete rows, and ImageEnc ( I m ) derives the encoding of I m by Equation 2 and 3.
To avoid using additional data in our internal inference, we formulate the above prediction as to the generation process.
That is to say, the inferrer aims to generate the masked image x m at the pixel level.
In addition, we repeat the above internal inferences for 6 times by masking each image to learn more robust rule representations.
We adopt mean squared error (MSE) as the objective function to optimize [Table] the inferrer as well as the rule encoder.
In this way
[Table]
we obtain the overall objective function J if by collecting the predictions of these internal inference processes, given by
[Equation]
3.6 Optimization of ARII
As introduced in the section on problem formulation
[Table]
the major goal of our method ARII is to select a candidate answer image for the incomplete matrix as accurately as possible (as shown in Equation 13).
In the meanwhile
[Table]
we assume the rule representations are discrete and can be reused in internal inferences.
That is, apart from the major goal, we have two additional objectives, i [Table] e., the objective of the vector quantisation process and the objective of the internal inferences.
Therefore, we combine these three objectives together for the optimization of the individual components of ARII, given by
J = (cid:88)
[Equation]
17) where the context images X c is parts of the given data point X
and D is the dataset.
λ re , λ vq and λ if are the three hyper-parameters that coordinates the importance of the components in ARII.
Experiment 4.1 Datasets
PGM.
Each matrix in PGM is governed by abstract rules, which are sampled from a rule set
R = { [ s, o, a ] : s ∈ S , o ∈
O , a ∈ A} , where S , O , A are primitive sets of relations, objects and attributes, respectively.
S =
{ progression, XOR, OR, AND, consistent union } , O = { shape, line } ,
A = { size, type, color, position, number } .
There are eight regimes in PGM.
The easiest one is called neutral where the training and test set are sampled from the same distribution, and the others are generalization regimes where the training and test data differ in the pre-defined ways.
The generalization regimes consists of seven variants: interpolation, extrapolation, held-out triples (H.O.Triples), held-out attribute pairs (H.O.A.P.), held-out pairs of triples (H.O.T.P.), held-out shapecolour (H.O.S-C) and held-out line-type (H.O.L-T).
More details can be found in Barrett et al.
RAVEN and I-RAVEN.
The RAVEN and I-RAVEN datasets are extensions to PGM.
They share an identical set of attributes but differ in the following attributes: progression, constant, union and arithmetic.
I-RAVEN is adapted from RAVEN dataset for its biased answer sets which may lead to a shortcut solution that only candidate answers can yield accurate answers [8].
In our experiments, we first evaluate the models’ reasoning ability on I-RAVEN since it is easier than PGM.
4.2 Competing methods and implementation details
We compare our model with several state-of-the-art models, including CNN + LSTM [5], ResNetbased [38] image classifier, Wild ResNet [5], Wild Relation Network (WReN) [5], Contrastive
Perceptual Inference network (CoPINet) [9], Logic Embedding Network (LEN) [16], Scattering
Compositional Learner (SCL) [10], Stratified Rule-Aware Network (SRAN) [8], Multi-scale Relation [Table]
[Table] Network (MRNet) [24], and Multi-Layer Relation Network (MLRN) [13].
Note that our model is trained end to end and needs solely the ground truth answer label without auxiliary information (i.e., rule labels).
We report the result of the baseline models trained without auxiliary loss for fair comparison.
Please refer to Appendix A for more implementation details of ARII.
4.3 Results
Table 1 shows the test performance of individual models on I-RAVEN dataset.
We observe that the typical architectures in computer vision, that simply use CNN as visual feature extractor followed by MLP or LSTM, fail to yield satisfactory reasoning results.
More advanced models for abstract reasoning such as LEN and SCL significantly improve the performance.
In particular, the previous state-of-the-art model SCL achieves nearly 100% accuracy on the configurations of Center, L-R, U-D, and O-IC, but SCL could not make similar success on the other configurations.
We conjecture that the reasoning complexity of these four configurations is relatively low and easy.
As for our method,
ARII performs similarly to SCL on the above four configurations but largely outperforms it on the other three more difficult configurations, leading to a notably higher average score on I-RAVEN.
In addition, Table 2 presents the performance on the PGM dataset.
For the neutral regime, our model achieves a decent result that is better than majority of baseline models, although it does not outperform state-of-the-art models such as MRNet and MLRN.
Note that MLRN achieves near perfect accuracy in the neutral regime due to the sophisticated design which could not be applied to other problems.
More importantly, the neutral regime is not our main focus of metric, since it could hardly indicate the reasoning ability and generalization towards real intelligence.
PGM also allows us to investigate the generalization capacity of the models under various regimes.
We observe from Table 2 that although MLRN achieves superior performance on neutral, it is prone to overfitting and performs poorly on the interpolation and extrapolation regimes, clearly indicating MLRN is overfitted to the neutral setting.
On the contrary, the proposed internal inference in ARII serves as the regularities of the rule representations, making the rule representations robust and instance-invariant.
As a result, our model outperforms all other methods on five of the seven generalization regimes in PGM dataset.
In particular, the accuracy of ARII increases by a sizable margin over the baselines in the H.O.A.P. and H.O.T.P. regimes (around 10%), further validating the superior generalization ability of abstract reasoning.
[Figure]
4.4 Ablation study
We analyze ARII in more detail to validate the effectiveness of the proposed components on two regimes in the PGM dataset (Table 3).
In the H.O.A.P. regime, the performance of ARII without discretization decreases by 2.5%, which shows that discrete representation plays its role in generalization of abstract reasoning.
The performance is also largely declined by 18% when the internal inference is removed, suggesting that the internal inference is vital to learn a more robust rule representation for abstract reasoning.
Similar results can be found in the interpolation regime.
We further conduct ablation studies to investigate the impact of generation in internal inference, the combinatorial benefit of r 1&3 and r 2&3 in the reasoner
[Figure]
and effectiveness of the internal inference to other models.
Please see Appendix B-D for more discussion.
4.5 Visualization of rule representations
Then, we try to validate whether ARII has learned the instance-invariant representations.
We first obtain all the rule representations in the test set of the interpolation regime of PGM, and use the representations derived from the problems which contain only a single rule (i.e., a triple { [ s, o, a ] } ).
The rule representations are projected to 2D space by t-distributed stochastic neighbor embedding (t-SNE) [41] in Fig 3.
We observe that the rule representations are clustered according to individual rule triples.
Note that these clusters for each rule are formed without supervision of the rule labels.
And the representations are extracted from the generalization test set which has a different distribution from the training set.
The clustered representations show that these rule representations are universal and independent to the specific instances, even to the out-of-distribution brand new instances.
Taking one step further, we investigate whether the discrete rule representation is interpretable.
In particular, we want to see the composition of the representations for each specific rule.
Here, we take the rules related to the line object and the color attribute (totally five rules, i.e., [ s ∈ S , line, color], denoted by the [ · ,line, color] rules) as an example.
We first collect the rule representations from all the instances governed by these rules, which are sampled from the codebook E .
Then we calculate the selected frequencies of the codes at the 30-th position in the rule representations and visualize their distributions.
Fig 4 (a-e) shows that all the five rules share the same codes E 358 and E 181 in their top-3 codes, indicating that the codes E 358 and E 181 are very relevant to the [ · ,line, color] rules.
In addition, these codes are also specific to these rules, since they rarely appear in the dissimilar rules such as the rule [progression, shape, size] (Fig 4 (f)).
These results reveal that the rule representations learned by ARII are moderately interpretable.
Besides, we conduct a classification experiment to see whether the rules can be classified from the rule representations.
The classification accuracy based on the rule representations from ARII is 72.1%, largely higher than a baseline model SRAN [8] that yields 59.4%.
This indicates the superiority of ARII which can capture the underlying rules and learn more robust rule representations.
Limitations
Abstract reasoning is the ability that consciously applies logic from premises to conclusion [42, 43].
For neural networks, they use different layers of the distributed neurons to implicitly represent the premises, the logic and the temporal results, and formulate the final conclusion-making process as the classification or generation task.
Therefore, neural networks lack in explaining how they perform induction and reasoning.
Although our method takes one step further to build interpretable rule representations, it is still far from the goal of a fully interpretable, robust, and accurate reasoner for general reasoning.
Another limitation of this work is the image encoder, which relies on the rigid spatial structure and is difficult for learning new rules in other abstract reasoning problems.
The general object detector pre-trained on natural images is a potential solution.
Conclusion
In this work, we propose a novel architecture ARII that learns robust rule representations for abstract reasoning via internal inferences.
Experiments on benchmark datasets (I-RAVEN and PGM) show that ARII outperforms previous state-of-the-art methods without using auxiliary annotations on the majority of the reasoning tasks.
Moreover, the rule representations of ARII present meaningful and interpretable characteristics, facilitating the exploration of the composition of the rules in future work
Acknowledgements
This work was supported by the National Natural Science Foundation of China under Grantand and a grant from Institute Guo Qiang, Tsinghua University.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] Our model has improved the generalization performance by a notable margin on majority of tests in PGM, but further research is needed to reach the human level of generalization.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[N/A] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] We have uploaded the code in the supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Appendix.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[No] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] The datasets used in this work are cited in [5] and [8].
(b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]