Neural Matching Fields: Implicit Representation of Matching Fields for Visual Correspondence
Abstract
Existing pipelines of semantic correspondence commonly include extracting highlevel semantic features for the invariance against intra-class variations and background clutters.
This architecture, however, inevitably results in a low-resolution matching field that additionally requires an ad-hoc interpolation process as a postprocessing for converting it into a high-resolution one, certainly limiting the overall performance of matching results.
To overcome this, inspired by recent success of implicit neural representation, we present a novel method for semantic correspondence, called Neural Matching Field (NeMF).
However, complicacy and highdimensionality of a 4D matching field are the major hindrances, which we propose a cost embedding network to process a coarse cost volume to use as a guidance for establishing high-precision matching field through the following fully-connected network.
Nevertheless, learning a high-dimensional matching field remains challenging mainly due to computational complexity, since a naïve exhaustive inference would require querying from all pixels in the 4D space to infer pixel-wise correspondences.
To overcome this, we propose adequate training and inference procedures, which in the training phase, we randomly sample matching candidates and in the inference phase, we iteratively performs PatchMatch-based inference and coordinate optimization at test time.
With these combined, competitive results are attained on several standard benchmarks for semantic correspondence.
Code and pre-trained weights are available at https://ku-cvlab.github.io/NeMF/ .
Introduction
Establishing visual correspondence across semantically similar images is a fundamental problem in computer vision, which has been facilitating many applications including visual localization [68, 38], structure-from-motion [69], image editing [1] and autonomous driving [33].
Unlike traditional dense correspondence tasks [20, 23], where visually similar images of the same scene are used as inputs semantic correspondence problem poses additional challenges due to intra-class appearance and severe geometry variations among object instances [15, 16].
Much research [65, 48, 29, 25, 32, 50, 64, 26, 47, 31, 8, 35, 77, 9] in semantic correspondence literature attempt to address above challenges by leveraging Convolutional Neural Networks (CNNs)- based features thanks to their greater semantic invariance than traditional hand-crafted descriptors [37, 10, 66, 3] that only capture low-level local structure.
As shown in Fig. 1(a), they typically perform matching with deeper features that contain high-level semantics to obtain a low-resolution correspondence map, and they are enforced to use hand-crafted interpolation techniques, e.g., bilinear
[Figure]
interpolation [65, 64, 26, 8, 77, 9] or TPS warping with sparse keypoints [48, 50, 47], significantly reducing localization precision in matching details.
Instead of these hand-crafted designs, several works [25, 76, 19] attempted to formulate a coarse-to-fine approach by utilizing multi-level features, but they often suffer from the propagation of initial error from the early coarse level.
Inspired by recent success of Implicit Neural Representation (INR) [44, 57, 73, 45, 5, 51] where a coordinate-based neural network allow to model a continuous field, we propose a novel learnable framework, dubbed Neural Matching Field (NeMF), that aims to establish high-precision correspondence at arbitrary original image resolution.
However, typically, matching field between a pair of images is complicated and high-dimensional, where a simple fully-connected network, which is commonly used in INR, may fail to implicitly represent such a high-dimensional matching field.
To better structure the intricate matching field, we propose a cost embedding network that takes a coarse cost volume to learn cost feature representation and use it as a guidance for generating high-precision matching field through the following fully-connected network.
This is accomplished by designing the cost embedding network with convolutions [18] and self-attention layers [78] to encapsulate local contexts and impart to all pixels with global receptive fields of self-attention, which also helps to compensate for the lack of inductive bias of Transformer by injecting convolutional inductive bias.
The intuition of the proposed method is illustrated in Fig. 1(b).
Although leveraging a cost representation may alleviate the issues for learning the matching field with details preserved, naïvely performing feed-forward for all pixels of matching field to find pixel-wise correspondences that are used for providing supervisory signals or inference would be computationally intractable.
To this end, we learn a neural matching field by enforcing the network to predict the correctness of a correspondence given a set consisting of randomly sampled points and the ground-truth point.
Furthermore, as the intractability issue applies similarly at inference phase, we propose a novel test-time optimization method that not only adopts a PatchMatch [1]-based search space sampling strategy in the learned neural matching, but also optimizes the coordinates for a means of correction that lead to find better correspondences as the iteration progresses.
We alternatively perform both PatchMatch-like inference and coordinate optimization, which works as an exploration and exploitation solution.
In the experiments, we evaluate the effectiveness of the proposed method using the standard benchmarks for semantic correspondence [49, 15, 16].
We demonstrate that the proposed implicit matching field effectively finds high precision correspondences, reporting the dramatically boosted performances in comparison to that of hand-crafted interpolation techniques.
We also conduct extensive ablation study to validate our design choices and explore the effectiveness of each components.
Related Work
Semantic Correspondence.
The earliest works [10, 66, 3, 37] focused on feature extraction stage by proposing the hand-crafted feature descriptors.
Although these works are probably based on the most well-known traditional hand-crafted feature descriptors, they exhibit limited capability to capture high-level semantics.
Resolving such an issue, convolutional neural networks (CNNs) [72, 18] made a paradigm shift thanks to their robust representations to deformations, at first replacing the hand-crafted features with deep features, rapidly converging towards end-to-end learning. Since [Figure]
then, leveraging deep features has become the de facto standard.
Rocco et al.
[62] first proposed an end-to-end geometric matching networks based on the deep feature maps extracted using CNNs and correlation maps computed between the extracted feature maps.
Since then, using correlation map which contains all pairs of similarities between descriptors has become popular by numerous matching networks [65, 63, 43, 76, 19, 31, 48, 50, 47, 64, 26, 34, 82, 32, 40, 8].
However, not only exhaustively computing and storing all pairwise similarities require quadratic memory and computation complexity with respect to the input spatial size, which is a major downside, but also it is infeasible to compute them as the resolution increases.
This inevitably caused existing methods to utilize correlation map defined at low resolutions.
On the other hand, notable methods include NC-Net [65] which first proposes to employ 4D convolutions to identify spatially consistent matches by exploring neighbourhood consensus.
DHPF [50] applied probabilistic Hough matching (PHM) [7] to find the matching points.
CHM [47] extends the PHM by employing high-dimensional convolutional kernels to aggregate 6D correlation maps.
CATs [8] and its extension [9] use transformers [78, 12] to explore global consensus from correlation maps thanks to transformers’ ability to consider long-range interactions.
All these works exploit rich semantics present at high-level features for robust matching across semantically similar images.
However, this inevitably necessitates up-sampling the predicted correspondence field to original image-level resolutions, which may result in losing precision.
Unlike them, we implicitly represent a matching field at arbitrary image-level resolution, eliminating such a loss and ensure high-precision correspondence field to be found, as shown in Fig.
Implicit Neural Representation.
Implicit neural representation (INR), also known as coordinatebased representations, is continuous, differentiable signal representation parameterized by neural networks [45].
INR recently received huge attention, and substantial progress has been made in this direction.
INR is not coupled to spatial resolution, making the memory requirements to parameterize the input signals orthogonal to spatial resolution.
Notable contributions to INR include COIN [13] that proposes a compression method with INR and
LIIF [5] learns a continuous representation for images that can be presented in arbitrary resolution.
DeepSDF [53] was a pioneering work that enables high quality representation from 3D input data by leveraging a learned continuous signed distance function.
Occupancy networks [44] implicitly represent the 3D surface as the continuous decision boundary.
IM-Net [6] also takes a similar approach, learning a mapping from coordinates conditioned by shape feature vectors to determine whether a point is outside or inside the 3D shape.
Since then, INR-based works consistently have attained state-of-the-art performance in 3D computer vision.
As a pioneering work, NeRF [45] represents 3D scenes as neural radiance fields for novel view synthesis.
Inspired by NeRF, a large number of works [81, 54, 70, 28, 55, 59, 14, 79, 42, 58, 2, 73, 4, 51, 80, 61, 5] made a progress in this direction.
Although flourished in 3D computer vision tasks, INR has never been properly studied or explored in visual correspondence tasks, which we address in this work.
𝐼 𝑠 𝐼 𝑡
F e a t u r e
E x t r a c t i o n
C o s t C o m pu t a t i o n
C o s t
E m b e dd i n g
M L P
Matching Field
F e a t u r e
E x t r a c t i o n 𝑤 𝑠 𝑤 𝑡 ℎ 𝑡 ℎ 𝑠 Source dim.
Target dim.
(ℎ s ,𝑤 s ,ℎ t ,𝑤 t ) (ℎ s ,𝑤 s ,ℎ t ,𝑤 t ,𝐾) 𝑤 𝑠 𝑤 𝑡 ℎ 𝑡 ℎ 𝑠 Source dim.
Target dim.
𝜙 𝐶,𝐩 𝛾 𝐩 𝑀 𝐩 𝐩
Neural Radiance Field (NeRF) is a continuous function f ω with parameters ω that computes a volume density σ and RGB color value c =
( r, g, b ) taking as an input a 3D location o =
( x, y, z ) and 2D viewing direction d
= ( θ, ϕ ) , such that f ω : ( o , d ) → ( σ, c ) .
In practice, as shown in [45, 74], mapping low dimensional inputs x and d to higher dimensional features before passing them through the neural networks enables better representing high frequency variations.
Specifically, denoting γ ( · ) as an encoding function and L as the number of frequency octaves as
[Equation]
[Equation]
[Equation]
[Equation]
the overall process f ω : R L
o × R L d → R + × R 3 is defined as
[Equation]
where L o and L d denote output dimension of the encoded coordinate o and viewing direction d , respectively.
The function f ω is formulated as a fully-connected deep network.
This implicit neural representations are not coupled to spatial resolution for using continuous functions, making the memory consumption required to parameterize the signal independent of spatial resolution [45, 74].
Neural Matching Fields (NeMF) 4.1 Problem Statement and Overview
The overview of NeMF is shown in Fig.
Given a pair of source and target images as I s ∈ R H s × W s
and I t ∈ R H t × W t , our objective is to find a dense correspondence field F ( x ) that is defined for each pixel x in original image resolution, which warps I s towards I t
so as to satisfy I t ( x )
≈
I s ( x + F ( x )) .
Following traditional matching pipeline, we first extract dense features D s and D t from the input source and target images, and then compute full pair-wise similarity scores between them using cosine distance such that:
[Equation]
where x ∈ [0 , h s ) × [0 , w s ) and y ∈ [0 , h t ) × [0 , w t ) , and ∥ · ∥ denotes l -2 normalization.
Previous approaches [50, 47, 8] extracted features from the deep layers of CNN to have high-level semantic invariance, resulting the spatial resolution of D s and D t to be reduced, i.e., h < H and w < W .
Consequently, utilizing the coarse similarity scores C ( x , y ) to infer correspondences inevitably yields a low-resolution correspondence map, which additionally requires post-processing to be interpolated into a high-resolution map [8, 47].
To alleviate this issue, we propose an INR-based learnable framework, called neural matching field (NeMF), that implicitly represents a high-dimensional 4D matching field to infer high-precision correspondences at arbitrary scales without any post-processing procedure.
Specifically, we formulate a continuous function f θ as a multi layer perceptron (MLP) with parameters θ in which encoded position and its corresponding cost feature vector are taken as an input.
Formally, denoting 4D coordinates defined in original image resolution as p =
[ x , y ] where x ∈ [0 , H s ) × [0 , W s ) and y ∈ [0 , H t ) ×
[0 , W t ) , our neural matching field M ∈ R H s × W s × H t × W t is computed as
[Equation]
where γ ( p ) is an encoded point of p , and ϕ ( C, p ) denotes the cost feature vector at p extracted from coarse cost volume C .
A possible way for the architecture design of function f θ is first concatenating the two inputs γ ( p ) and ϕ ( C, p ) , and then passing them through the fully-connected network.
This approach, however, may impose memory intensive batch normalization operation [44].
To address this, we adopt the architecture of [52], where ϕ ( C, p ) is added to the input features of each fullyconnected block.
In addition, to guarantee the value of 4D matching cost fields to be lied within the range from 0 to 1 , we use a sigmoid function [17] at the end of the networks.
4.2 Cost Embedding Network
Our assumption is that formulating the function f θ ( · ) without any condition may be challenged when representing complicated and high-dimensional continuous field.
We address this by introducing cost embedding network where the raw cost volume is further embedded into a cost feature volume ϕ ( C, p ) , which is used as a guidance for establishing high-precision matching field through the following fully-connected network f θ ( · ) .
Motivated by recent works [8, 22] that aggregate matching costs for better correspondence hypothesis, we further embed the raw cost volume through the global receptive fields of self-attention layer [78, 12].
Although these representations are explicitly encoded from all pixels of a cost volume, the absence of operations that impart inductive bias, i.e., translation equivariance by convolutions or relative positioning bias, may yield representations with errors.
To this end, we combine Transformer architecture [78] with convolution operator to compensate the lack of inductive bias, allowing local and global integration of matching cues by encapsulating the local contexts and imparting them to all pixels via self-attention.
Concretely, before providing matching cost to the function f θ , the 4D raw cost volume C is embedded into 5D cost feature volume C ′
∈
R H s × W s × H t × W t × K with K channels which is still defined at low resolution.
We then use a quadlinear interpolation on C ′ for a query point p to a cost feature vector ϕ ( C, p ) ∈
R K × 1 .
4.3 Training
As shown in Fig.
4, to train the networks, we use a ground-truth keypoint pair { x , x ′ } between an input image pair in a manner that if a query point p =
[ x , y ] is classified as the ground-truth correspondence, i.e., y = x ′ , the network output should be encouraged to be 1 , and 0 otherwise.
We formulate this as a classification problem, and thus we apply cross-entropy loss to learn to predict the correctness of a correspondence for a query x in the source image among sampled negative keypoints y in the target image (where y ̸ = x ′ ) and that of ground-truth x ′ .
Even though the better negative sampling techniques, e.g., hard negative mining [27, 24], can be used, in experiments, we simply adopt random sampling from uniform distribution as negative samples, as background clutters or extreme geometric variations inherently present across semantic correspondence datasets [15, 16, 49] would contribute to robust representation learning.
Formally, the loss function is defined as follows:
[Equation]
[Equation] ) where p k denotes k -th query samples for k ∈ { 1 , ..., K } , and M ∗ k is a ground-truth matching score, e.g., M ∗ k = 1 if p k is a ground-truth keypoint pair, and 0 otherwise.
Although this would provide sufficient supervisory signal, we found it is beneficial to provide an additional explicit supervisory signal for learning better cost features, which positively affect our
PatchMatch [1]-based inference strategy as will be further detailed in Sec. 4.4.
To this end, we use end-point-error [8] between the predicted keypoints using the cost feature representations ϕ ( C, p ) directly and the ground-truth keypoints.
Concretely, we obtain a channel-wise average pooled cost feature volume V = avgpool( ϕ ( C, p ))
∈
R H s × W s × H t × W t and compute F pred by applying soft argmax function to V .
We then calculate the Euclidean distance between the ground-truth flow map computed using the ground-truth keypoints and the predicted flow map as
[Equation]
Combining with Eq.
6, we define the final objective function L total with balancing weights λ f and λ c : L total =
λ f L f + λ c L c .
4.4 Inference
PatchMatch-based Sampling.
At the inference stage, we aim to find a dense correspondence field F ( x ) by leveraging the trained network to determine the correct correspondence for each query p =
[ x , y ] .
However, searching the best match for each coordinate x over all possible matching candidates y in exhaustive manner results in H s × W s × H t × W t number of feed-forward per sample, which is an extremely time consuming and computationally intensive.
To alleviate the issue, we propose PatchMatch [1]-based sampling.
PatchMatch [1] runs a sequence of propagation and update steps to reduce a search space.
We use the learned NeMF f θ as a scoring function to determine the correspondence.
To overcome its time-consuming process induced by serial processing inherited from PatchMatch [1], we propose a GPU-friendly PatchMatch optimization that performs propagation and update in a parallel manner.
Specifically, for initialization, we utilize an average pooled cost feature volume V introduced in
Section 4.3 .
Note that the objective of Eq.
6 directly connects to the initialization step of this approach.
This implies that the better cost feature representations would help to obtain a better initialization for PatchMatch-based inference.
Then for a query x , we sample a set of candidate correspondences by considering adjacent pixels such that Z l − 1 = { F l − 1 ( z ) } z for adjacent pixels z at ( l − 1) -th iteration.
In addition, a few random points sampled from a uniform distribution are used to augment Z l − 1 such that Y l − 1 =
(cid:83) (cid:0) Z l − 1 , { y } (cid:1) for randomly sampled pixels y where (cid:83) denotes an union of the sets.
Then the correspondence fields are undated by considering the set of matching candidate Y l − 1 such that
[Equation]
This process is iterated until the convergence.
In practice, this candidate selection and scoring run in parallel for every target pixel.
This makes the inference process efficient and GPU-friendly in the original resolution, compared to serial propagation and update in [30].
Coordinate Optimization.
Although the proposed PatchMatch-based inference strategy could prevent exhaustive searching by effectively sampling the search range for determining the correct correspondence for each pixel, this may degrade the performance due to several reasons including insufficient number of iterations that may result in a sub-optimal solution and a limited search range that provides relatively fewer candidates for consideration.
To address this issue, we provide a means
[Figure]
to reduce the potential erroneous inference by adopting test-time optimization strategy that directly optimize coordinates y that maximizes the correctness of the correspondence using the learned the network f θ .
Concretely, because the network f θ is naturally differentiable, as shown in Fig.
5, we use a gradient descent to optimize the target coordinate y in the direction of decreasing the negative log likelihood of the matching score, with respect to the corresponding source coordinate x .
Formally, the coordinate optimization is performed by iterative updates which can be formulated as:
[Equation]
where α denotes a step size.
Any advanced optimizer can also be used for improved optimization [60, 39].
Note that the source coordinate is not updated during this optimization.
With the proposed coordinate optimization, we combine with PatchMatch-based sampling to establish a final correspondence field as shown in Fig.
Each iteration number is defined as N .
As exemplified in Fig.
6, NeMF predicts more precise matching fields by PatchMatch-based sampling and coordinate optimization as evolving iterations.
Note that the key difference of this test-time optimization to that of DMP [19] is that we optimize the coordinates to correct themselves to find a better correspondence with by leveraging the already learned network, while DMP optimizes the parameters of the networks.
Experiments 5.1 Implementation Details
For backbone feature extractor, we use ResNet-101 [18] pre-trained on ImageNet [11].
We use the feature maps resized to 16 × 16 for constructing a coarse cost volume.
For the cost embedding network, we build upon [8] and its implementations.
We implemented our network using PyTorch [56], and
AdamW [41] optimizer with an initial learning rate of 3e − 5.
We set N = 10 for both PatchMatch and coordinate optimizations, and learning rate of 3e − 4 is used for coordinate optimization.
Additional details are provided in the supplemenatry material.
5.2 Experimental Settings
Datasets.
We use three benchmarks, which include SPair-71k [49], PF-PASCAL [16] and PFWILLOW [15], to evaluate the effectiveness of the proposed method.
SPair-71k [49] provides total 70,958 image pairs, PF-PASCAL [16] contains 1,351 image pairs from 20 categories, and
PF-WILLOW [15] contains 900 image pairs from 4 categories.
Each dataset contains ground-truth annotations, which we use them for evaluation and training.
Evaluation Metric.
For the evaluation metric, we employ a percentage of correct keypoints (PCK), which is computed as the ratio of estimated keypoints within the threshold from ground-truths to the total number of keypoints.
Assume a predicted keypoint k pred and a ground-truth keypoint k GT , the number of correctly predicted keypoints are counted, and the condition for deciding the
[Table]
[Table]
correctness is defined as follows:
d ( k pred , k GT ) ≤ α · max( H, W ) , where d ( · ) and α denote
Euclidean distance and a threshold.
When we evaluate on PF-PASCAL, we use α img following other works [16, 48, 50, 8], SPair-71k and PF-WILLOW with α bbox ; H and W denote height and width of the object bounding box or entire image, respectively.
5.3 Matching Results
To ensure a fair comparison, the model evaluated on SPair-71k [49] is trained on training split of
SPair-71k [49] and the model evaluated on PF-PASCAL [16] and PF-WILLOW [15] is trained on training split of PF-PASCAL [16].
The results are summarized in Table 1 and the qualitative results are shown in Fig.
We note the resolution which the method is evaluated, since [9, 77] observe that the resolution of images affect the PCK performance, and the resolution of which the method outputs the correspondence field.
It is shown that NeMF achieves competitive performance or even attains state-of-the-art performance for several alpha thresholds.
More concretely, for lower alpha thresholds, we tend to achieve higher
PCK compared to other works.
This implies that the existing works, which rely on interpolation techniques that prevent from fine-grained matching due to the use of matching field defined at low resolution, may have suffered from the large resolution gap between the predicted flow and that of ground-truth.
For example, CATs [8] processes the cost volume at 16 4 and infers a flow map at this resolution.
On the contrary, NeMF avoids this by implicitly representing a matching field at higher resolution, demonstrating its advantageous approach.
5.4 Ablation Study
In this section, we conduct ablation study to investigate the effectiveness of different configurations for cost embedding network and effectiveness of the proposed inference strategies.
We train the networks on the training split of SPair-71k [49] and evaluated on the test split. [Figure]
Note that correspondences are estimated at the original resolutions of iamges.
Different Cost Feature Representation.
Table 3 summarizes the comparison between the effectiveness of cost feature representations learned from different configurations of cost embedding network.
In this ablation study, we compare four configurations.
( I ) shows the results for exploiting raw cost volume defined at coarse level.
From ( II ) to ( IV ), we show the effectiveness of extracting cost features via convolutions, self-attention layers and integration of both, respectively
[Table]
[Table]
We observe that simply leveraging a raw cost volume struggles to learn a complicated matching field, as it does not provide a sufficient structural or detailed information among pixel-wise similarities.
As the cost embedding network is introduced to learn feature representations, the performance is dramatically boosted [Table] and our approach clearly helps to attain the best performance by learning more powerful representations than ( II ) and ( III ).
[Table]
Inference Strategies.
In this ablation study, we aim to show a quantitative comparison between different strategies at the inference phase.
Table 4 summarizes the results.
Note that we included ( I ) to highlight that at the inference phase, the evaluation on a pair of images with original resolution, for example, would take approximately more than 300k seconds.
This clearly shows the infeasibility of adopting naïve inference strategy.
From ( II ) to ( III ), we observe an apparent performance boost, which demonstrates that the proposed test-time coordinate optimization helps to correct the coordinates for finding better correspondences.
However, this approach has a downside.
Applying coordinate optimization inevitably increases the time taken for the inference, which is a typical limitation of test-time optimization. However, 0.
second is a minor sacrifice for a better performance.
Note that further improvement could be made by adopting better optimizing strategy, i.e., learning rate, search range or optimizer.
Computational Complexity.
Although the proposed inference strategy enables significantly reduced time for establishing correspondence field between a pair of images, in practice, we observe that assuming we set N = 10, the time taken at the inference phase for a single sample is approximately 8-9 seconds on a single GPU Geforce RTX 3090, which prevents from a real-time inference. This is an apparent limitation of the proposed approach, but we refer the readers to supplementary material where we show that without affecting the performance, the memory consumption and run-time can be controlled. [Table] In addition, we also provide experimental results that demonstrates the efficiency of the proposed approach in comparison to existing works, which is summarized in Table 5.
Let us assume that we are representing cost volumes of four different resolutions, e.g., 16, 32, 64 and 128.
At the training phase, unlike other works that inevitably consume more memory as the resolution increases, the proposed approach successfully deviates from it thanks to the proposed training strategy.
Furthermore, at the inference phase, we observe that the proposed approach has an advantage over other methods. Although NeMF may suffer from relatively larger computation and memory consumption than CATs [8] and CHM [47] when the resolution is low, it has an advantage when the resolution is high, allowing the network to exploit highly accurate cost volume with relatively less memory consumption.
Conclusion
In this paper, we proposed a novel INR-based architecture, called neural matching fields (NeMF), that implicitly represent a 4D matching field to find high-precision correspondences.
This method proposed an architecture and training and inference procedures targeted to handle complicacy and high-dimensionality of a matching field that acts as major hindrances.
Specifically, we embed the raw cost volume with convolutions and Transformer to obtain local and global integration of matching cues to handle the complicacy, and sampling-based training and inference procedure to handle the high-dimensionality.
We have shown that the proposed method attains state-of-the-art performance on several benchmarks for semantic correspondence.
We also conducted an extensive ablation study to validate our choices.
Broader Impact
Our implicit representation of cost volume may be beneficial for other domains that utilize a correlation map, which include semantic segmentation [67, 75, 46], object detection [36], and image editing [37].
It can help to boost the performance by preserving the fine-detailed information within the cost volume.
However, as the proposed approach aims to implicitly represent the cost volume, on its own, it is not feasible to use for a malicious purpose.
Acknowledgements.
This research was supported by the MSIT, Korea (IITP-2022-2020-0-01819,
ICT Creative Consilience program, No. 2020-0-00368, A Neural-Symbolic Model for Knowledge Acquisition and Inference Techniques), and National Research Foundation of Korea (NRF2021R1C1C1006897).