Simultaneous Missing Value Imputation and Structure Learning with Groups
Abstract
Learning structures between groups of variables from data with missing values is an important task in the real world, yet difficult to solve.
One typical scenario is discovering the structure among topics in the education domain to identify learning pathways.
Here, the observations are student performances for questions under each topic which contain missing values.
However, most existing methods focus on learning structures between a few individual variables from the complete data.
In this work, we propose VISL, a novel scalable structure learning approach that can simultaneously infer structures between groups of variables under missing data and perform missing value imputations with deep learning.
Particularly, we propose a generative model with a structured latent space and a graph neural network-based architecture, scaling to a large number of variables.
Empirically, we conduct extensive experiments on synthetic, semi-synthetic, and real-world education data sets.
We show improved performances on both imputation and structure learning accuracy compared to popular and recent approaches.
Introduction
Understanding the structural relationships among different variables provides critical insights in many real-world applications, such as medicine, economics and education [42, 62].
Thus, learning graphs from observed data, known as structure learning, has recently made remarkable progress [10, 61, 63, 64].
For many applications, variables in the data can be gathered into semantically meaningful groups, where useful insights are at group level.
For example, in finance, one may be interested in how a financial situation influences different industries (i e. groups) instead of individual companies (i e. variables).
Similarly, in education, the data can contain student responses to thousands of individual questions (i e. variables), where each question belongs to a broader topic (i.e. groups).
Again, it is insightful to find relationships between topics instead of individual questions.
Moreover, real-world data such as educational data is inherently sparse since it is not feasible to ask every question to every student; the dimensions of the data in terms of the number of variables and the number of observations are very high, posing a scalability challenge.
Despite the progress in structure learning, no existing method can discover group-wise relationships given large-scale partially observed data.
In this work, we present VISL (missing value imputation with structural learning), a novel approach to simultaneously tackle group-wise structure learning and missing value imputations driven by the
[Figure]
}}} x n
Z n
G n = 1 ,...,N
[Figure]
real-world topic relationship discovery in an education setting.
This is accomplished by combining variational inference with a generative model that leverages a structured latent space and a decoder based on message-passing Graph Neural Networks (GNN) [13].
Namely, the structured latent space endows each group of variables with its latent subspace, and the interactions between the subspaces are regulated by a GNN whose behavior depends on the inferred graph from variational inference, see Fig. 1(a).
VISL satisfies all the desired properties: it leverages continuous optimization of the structure learning to achieve scalability [63, 64]; the VISL formulation naturally handles missing values, and it can discover relations at different levels of granularity with pre-defined groups.
Empirically, we evaluate VISL on one synthetic and two real-world problems including the aforementioned education scenario.
VISL shows improved performance in both missing data imputation and structure learning accuracy compared to popular and recent approaches for each task.
We worked closely with an education domain expert to evaluate the learned topic relationships, and our model has provided insightful results as recognized by the domain experts.
Model Description
In the following, we present the formulation of VISL for scalable group-wise structure learning with partial observations using a novel deep generative model based framework.
2.1 Problem setting
Assume a training data set X
= { x n } Nn =1 with x n ∈ R D .
The observed and missing values are denoted as X O and X U , respectively, where we assume the data are missing completely at random (MCAR) or missing at random (MAR).
In Appx.
A, we explain how to handle MAR.
In particular, variables can be gathered into M groups, where each can be denoted as χ n,m =
[ x n,i ] i ∈I m .
I m containing the variable indices belonging to group m
(e.g., I 2 =
[4 , 5 , 6] indicates group 2 includes the 4 th , 5 th and 6 th variables).
One should note that each I m may have varying sizes for different m (i.e. varying group sizes).
Throughout the paper, we assume the group information is provided a priori.
If this information is unavailable, casual representation learning can be leveraged to represent low-level signals into groups.
The goal of VISL is to (i) perform missing value imputation for test samples and (ii) infer structures between groups of variables.
We use the adjacency matrix
G ∈ [0 , 1] M × M to represent a graph, where G ij = 1 or 0 indicates whether there is a directed edge from i − th to j − th group or not.
In the context of the education domain, the above formulation can be rephrased as follows: variable x n containing the student’s responses to a set of questions.
x i,j =represents student i has answered question j correctly.
Groups can be defined as the topic associated with each question.
I m contains the question IDs that belong to the same topic, and χ m represents a group of responses related to that topic.
Clearly, not all students can answer every question.
Thus,
X O , X U represent the existing responses and un-answered questions, respectively.
The goal of
VISL is to (i) predict students’ responses to un-answered questions, which by itself is important in the education domain [56, 57], and (ii) discover the relationships between topics, which can help education experts optimize the learning experience and the curriculum.
For structure learning, we adopt a Bayesian approach for graphs [18].
Namely, we seek to maximize the posterior probability of
G given partially observed training data X O within the space of all DAGs:
[Equation]
To optimize over the structure with the DAG constraint in Eq.
1, we resort to recent continuous optimization techniques [25, 63, 64], where a differentiable measure of ’DAG-ness’, R ( G ) = tr( e G ⊙ G ) − D − 1 , was proposed and is zero if and only if G is a DAG.
To leverage this DAG-ness characterisation, we follow Kyono et al.
[25], Yu et al.
[61] and introduce a regulariser based on
R ( G ) to favour the DAG-ness of the solution, i.e.
[Equation]
In the following two sections, we present our detailed formulation, training and imputation algorithms of VISL, that allows the model to infer the latent structure G and impute missing values ˜ x U in a test sample ˜ x ∈ R D based on the observed ˜ x O .
2.2 Generative model and variational inference
G ij ∼ Bernoulli ( p ij ) for n ∈ { 1 , 2 , · · · , N } do
Z n ∼ N ( 0 , σ 2 z I ) x n ∼ N ( f θ ( Z n , G ) , σ 2 x I ) end for
For the generation of observation X , we adopt the latent variable model of Fig.
Particularly, given an inferred graph G and latent Z , the generative path from Z to X is provided in
Fig.
1, where we use a graph neural network (GNN) decoder that respects the learned graph structure G and the provided grouping structure.
Then the joint model likelihood is
[Equation]
Using this marginal likelihood p ( X ) is consistent with Bayesian score based causal discovery [18].
As Kaiser and Sipos [22] and Reisach et al.
[39] pointed out that other commonly used objectives, e.g. L2 loss in NOTEARS [63], are sensitive to data scaling and result in learning directions towards the high variance nodes.
Loh and Bühlmann [28] and Ng et al.
[35] showed that using a proper likelihood function can address these problems.
Next, we leverage amortized variational inference to sidestep the intractable marginalization of the joint likelihood.
Amortized variational inference.
The true posterior distribution over Z and G in Eq.
3 is intractable since we use a complex deep learning architecture.
Therefore, we resort to an efficient amortized variational inference as in Kingma and Welling [24], Kingma et al.
Here, we consider a fully factorized variational distribution q
( Z , G | X ) =
q ϕ ( G ) (cid:81)
Nn =1 q ϕ
( Z n | x n ) , where q ϕ ( Z n | x n ) is a
Gaussian whose mean and (diagonal) covariance matrix are given by an encoder .
For q( G ) , we consider the product of independent Bernoulli distributions over the edges; that is, the presence of each edge from i to j is associated with a probability p ij to be estimated.
With the above formulation, the evidence lower bound (ELBO) is
[Equation]
Next, we explain our choice of the generator (decoder), which uses a GNN over a learned graph G to model the interactions between latent variables, representing the information about each group.
Then, we focus on the inference network (encoder), representing the mapping from the group of observed variables to its corresponding latent representation.
Generator .
The generator (i.e., decoder) takes Z n and G as inputs and outputs the reconstructed ˆ x n = f θ ( Z n , G ) , where θ are the decoder parameters.
In order to respect the pre-defined group structure, as shown in Fig.
1, Z n is partitioned into M parts, where z n,m represents the latent variable for the group of observations χ n,m .
This defines a group-wise structured latent space.
We adopt a two-step process for the generative path Z n to X n : (i) GNN message passing with respect to the learned graph G between latent z n,m ; (ii) final read-out layer to generate X n .
GNN message passing in the generator .
In message passing, the information flows between nodes in T consecutive node-to-edge (n2e) and edge-to-node (e2n) operations [13].
At the t -th step, we compute an embedding h fi → j for each edge i → j , called forward embedding, which summarizes the information sent from node i to j .
Specifically, the n2e/e2n operations in VISL are
[Equation]
Here, t refers to the t -th iteration of message passing (that is, Z (0) =
Z n , notice that we omit subindex n for clarity).
Finally, MLP f , and MLP e 2 n are MLPs to be trained.
Interestingly, the message passing updates indicate that the information flows between latent nodes if a directed edge is specified in graph G .
Hence, the inferred structure G directly defines relations for latent space Z which contains the information of pre-defined groups.
We show that under certain conditions, the inferred graph G also represents the group-wise structure in observational space, and the corresponding model can be reformulated to a general structural equation model (SEM) [37] (see
Appx.
B).
Read-out layer in the generator .
After T iterations of GNN message passing, we have Z ( T ) .
Due to we allow Z ( T ) and x to have different dimensions, we apply a final function that maps
Z ( T ) to the reconstructed ˆ x , which also respects the pre-defined group structure.
Since the observation x =
[ χ 1 , . . .
, χ M ] may contain χ m with different dimensions, we adopt M different
MLPs, one for each group as the final read-out layer, to respect the group structure.
Namely, ˆ x = ( g 1 ( z T 1 ) , . .
g M ( z TM )) ⊤ , where g m represents the MLP for group m .
Thus, the decoder parameters θ include the parameters of the following neural networks: MLP f , MLP e 2 n and g m for m = 1 , . . .
, M .
Inference network .
As in standard VAEs, the encoder maps a sample x n to its latent representation
Z n .
As discussed before, Z n is partitioned into M parts, where each z n,m contains the information of the observation in group m .
Similar to the read-out layer, we utilize the M MLPs to map groups of observations to the mean/variance of the latent variables:
[Equation]
Here, µ mϕ µm and σ mϕ σm are neural networks for group m .
When missing values are present, we replace them with a constant as in [34].
Under this formulation, VISL can infer latent variables Z n from incomplete x n for both training and test data.
A graphic representation of how the encoder respects the structure of the latent space is shown in the appendix, Fig. 6(b).
2.3 Training VISL
Given the model described above, we propose the training objective to minimize w.r.t.
θ , ϕ and G :
[Equation]
where ELBO is given by Eq.
4 and the DAG regulariser R ( G ) was introduced in Eq.
2 to favor the
DAG-ness of learned graph G .
Evaluating the training loss L VISL .
VISL can work with any type of data.
The log-likelihood term ( log p θ ( x n | Z n , G ) in Eq. 4) is defined according to the data type.
We use a Gaussian likelihood for continuous variables and a Bernoulli likelihood for binary ones.
For the inference of Z and G , the standard reparametrization trick is used to sample Z n from the Gaussian q ϕ ( Z n | x n ) [23, 24].
To backpropagate the gradients through the discrete variable G , we resort to the Gumbel-softmax trick to sample from q( G ) [21, 31].
The KL [q ϕ ( Z n | x n ) || p( Z n )] and KL [q( G ) || p( G ))] terms can be obtained in closed-form since they are Gaussian distributions and independent Bernoulli distributions over the edges, respectively.
This formulation brings additional advantages in real-life applications since one can easily incorporate domain knowledge and prior information into the VISL framework.
For example, if the existence of a specific edge is known a priori, the edge probability can be set to 0/1 in the prior distribution.
Finally, the DAG-loss regulariser in Eq.
8 can be computed by evaluating the function R on a Gumbel-softmax sample from q( G ) .
To adapt the model to different missing levels in the training data X , we adopt the masking strategy [15, 30], which drops a random percent of the observed values during training.
The entire training procedure for VISL is summarised in
Algorithm 2.
Two-step training .
After training, we obtain the posterior of the graph G , which respects the underlying structure of the groups as shown in Appx.
B. With the trained network, we can impute missing values in the groups where their ancestors contain some observations but if a group has no ancestors no information can be propagated during imputation.
After learning the graph structure and to facilitate the imputation task, we introduce a backwards edge: for an edge from j to i we denote the backwards edge information as h bi → j which codifies the information that the i → j edge lets flow from the j -th to the i -th node.
It is defined in the same way as Eq.
5, i.e.,: h ( t ) ,b i → j =
MLP b (cid:16)(cid:104) z
( t − 1) i , z ( t − 1) j
(cid:105)(cid:17) , where MLP b is the backward MLP; and the e2n update (Eq.6) is modified to z ( t ) i =
MLP e 2 n (cid:16)(cid:80)
k ̸
= i G ki · (cid:110) h ( t ) ,f k → i + h ( t ) ,b i → k (cid:111)(cid:17) .
In summary, we propose a two-stage training process, where the first stage — described in previous sections — focuses on discovering the edge directions between nodes without the MLP b (i.e., we do not train the MLP b ).
In the second stage, we fix the graph structure G and continue to train the model with the backward MLP.
This two-stage training process allows VISL to leverage the backward MLP for the imputation task without updating the graph structure.
Revisiting the learning objectives .
The optimal graph of relationships, denoted as G ⋆ in Eq. 2, is given by the estimated posterior probabilities of graph G .
In addition, the regularizer R ( G ) provides a way to evaluate if the resulting graph is a DAG.
By tuning the regularizer strength λ , one can ensure that the resulting G ∗ represents a proper DAG.
For imputation, similar to Ma et al.
[30], Nazabal et al.
[34], the trained model can impute missing values for a test instance (cid:101) x as
[Equation]
Therefore, the distribution over (cid:101) x
U (missing values) is obtained by applying the encoder and decoder with (cid:101) x as input.
Previous work [30, 34] also uses variational autoencoders for missing value imputation, which can have problems with overfitting due to spurious correlations [25] even with sufficient training data.
One important distinction of VISL is the incorporation of the learned structure
G , which helps the model to be robust to spurious correlations.
Special case: variable-wise relations .
In the above formulation, we have defined VISL for groupwise structure learning.
Variable-wise relations can be regarded as a special case.
In particular, we can set M =
D
and I m = { m } (see Fig.
5 (a) in the appendix), i.e. each group only contains a single variable.
Through this modification, we can further simplify the encoder and read-out layer.
Instead of using M different MLPs, a single MLP can be shared across all variables since each group has dimension of 1.
The mean function for the encoder is then defined as
[Equation]
One can define encoder variance σ (Fig. 6 (a) in the appendix) and the read-out layer g analogously.
Computational cost.
The main computational bottleneck is the training of VISL, where we require a different inference network and read-out layer for each group m ∈ M .
However, some weight sharing schemes can be used to reduce the computational cost.
When each group has the same dimensionality, we share the weights of the inference networks.
In the case of different dimensionalities, one can infer the latent variables by parallelizing inference network forward passes to reduce computational cost.
Table 11 in appendix provides the wall-clock time comparison for each of the methods for up to 512 variables.
We can see that all deep learning based methods (i.e. NOTEARS, DAG-GNN, VISL, etc.) have similar time complexity.
Related Work
Since VISL simultaneously tackles missing value imputation and structure learning, we review both fields.
Moreover, we review recent works regarding improving the deep learning performance with structure learning.
Finally, as one of the focused applications of this work is education, we review recent advances of AI in education.
Structure learning .
Structure learning aims to infer the underlying structures associated with some observations.
There are mainly three types of methods: constrained-based, score-based, and hybrid.
Constraint-based ones exploit (conditional) independence tests to find the underlying structure, such as PC [47] and Fast Causal Inference (FCI) [48].
They have recently been extended to handle partially observed data through test-wise deletion and adjustments [50, 52].
Score-based methods find the structure by optimizing a proper scoring function.
The core difficulty lies in the number of possible graphs growing super-exponentially with the number of nodes [4].
Thus, explicitly solving the optimization can only be done up to a few nodes [7, 36, 46].
Therefore, approximation have been proposed to ease the computational burden, including searching over topological ordering [43, 44, 51], greedy search [3, 38], coordinate descent [1, 11, 17].
Recently, continuous optimization, called Notears , has become very popular [63].
Notears proposed a differentiable characterization of the DAG to learn the model parameters and graph structures jointly.
Notears has inspired the development of other methods, Notears-MLP and Notears-Sob [64],
Grandag [26], and DAG-GNN [61], which extends to model nonlinear relationships between variables.
However, their formulations cannot handle missing values and have been observed to be sensitive to data scaling [22].
DAG-GNN also adopts a specially-designed GNN to perform structure learning [61].
There are three key distinctions compared to VISL: (i) our model handles the group-wise relationship, while DAG-GNN focuses on variables level; (ii) our model is capable of missing value imputation and group-wise structure learning simultaneously, whereas the original formulation of
DAG-GNN and related work require complete data; (iii) VISL adopts Bayesian view for the graphs, compared to a point estimation.
VISL is also related to Bayesian DAG learning.
Viinikka et al.
[55] proposed to sample the graph posterior using MCMC, but it suffers from high computation complexity.
VISL, in contrary, can easily be scaled to high dimensional datasets.
BCD [6] and
DiBS [29] have recently leveraged the approximate inference for scalable Bayesian DAG learning.
BCD nets focused on linear Gaussian SEM (v.s. nonlinear relationship by VISL).
DiBS adopted a full Bayesian treatment and used a particle sampler to draw from a joint posterior.
Compared to
VISL, it cannot handle missing values nor variable groups.
Geffner et al.
[12] recently proposed an end-to-end causal inference framework (DECI) combining causal discovery and inference with missing data.
Compared to VISL, DECI is built upon nonlinear additive noise models [19], whereas
VISL uses an autoencoder structure with a GNN decoder.
When the GNN reaches equilibrium state, Appendix B shows that VISL represents a general SEM, which includes additive noise models as a special case.
Apart from the likelihood-based causal discovery, Rolland et al.
[40] used score matching to extract causal relationships for non-linear additive noise models.
Our work differs in four aspects: (1) structure learning (VISL) v.s. causal discovery; (2) VISL is based on the auto-encoder framework; (3) VISL can handle MAR missing data and (4) VISL uses a Bayesian view of graphs.
Structure deep learning .
Continuous optimization for learning structures has been used to boost performance in classification.
In CASTLE [25], structure learning is introduced as a regulariser for a classification model.
This regulariser reconstructs only the most relevant features, leading to improved out-of-sample predictions.
In SLAPS [10], the classification objective is supplemented with a self-supervised task that learns a graph of interactions between variables through a GNN.
These works focused on leveraging the structure learning instead of advancing its performance.
Missing values imputation .
The relevance of missing data in real-world problems has motivated a long history of research [9, 41].
A popular approach is to estimate the missing values based on the observed ones through different techniques [45], e.g. Random Forest [49], Bayesian Ridge
Regression [2].
Wu et al.
[59] explored the use of generative model for missing value imputation, although fully observed training data is required.
This limitation is addressed in both Nazabal et al.
[34] with zero-imputing strategy, and Ma et al.
[30] with a permutation invariant set encoder.
Mattei and Frellsen [32] proposed to use the importance weighted autoencoder, which enables a tighter lower bound than ELBO and leads to improved performance.
Ivanov et al.
[20] parameterized the imputation as sampling from a conditional distribution, and proposed a method for arbitrary conditioning with VAEs.
AI in education .
There has been tremendous progress in AI for educational applications, e.g. knowledge tracing [27, 33, 54]; grading students’ performance [58]; generating feedback for students working on coding challenges [60].
In particular, most related to VISL is imputing missing values in students’ responses.
Wang et al.
[56] adopts a partial VAE [30] to perform missing value imputation and personalization.
However, partial VAE does not consider the structural relations between questions/topics.
With the additional insights from structure learning, VISL can provide more information to teachers to help curriculum design than just imputations.
[Figure]
[Figure]
.5 threshold, we get the predicted graph in Fig
[Table] [Table]
Experiments
We evaluate the performance of VISL in three different problems: a synthetic experiment where the data generation process is controlled, a semi-synthetic problem (simulated data from a real-world problem) (Neuropathic Pain), and the real-world problem that motivated the group-level structure learning (Eedi).
The first two datasets are on the variable level.
The last one focuses on the group level and real-world usage, and have worked closely with the domain expert to evaluate the results.
Additional experiments are presented in the appendix.
Baselines .
We consider five baselines for the structure discovery task at the variable level.
PC [48] and GES [3] are popular constraint-based and score-based approaches, respectively.
NOTEARS (NOT.)
[63], the non-linear (NL) extension of NOTEARS [64], and DAG-GNN [61] are the other three closely related baselines.
Unlike VISL, these baselines cannot deal with missing values.
Therefore, we work with complete training data in the first two sections.
The Eedi real-world data is only partially observed, where these baselines are not applicable.
For the missing data imputation, we also consider five baselines.
Mean Imputing (Mean), Majority Vote (Major) (refer to Appx.
D for short descriptions), Missforest [49] and MICE [2] are four widely-used imputation algorithms, and
PVAE [30] is a recent algorithm based on amortized inference.
[Table] [Table]
Metrics .
Imputation performance is evaluated with standard metrics such as RMSE (continuous variables) and accuracy (binary variables).
For binary variables , we also provide the area under the
ROC and the Precision-Recall curves (AUROC and AUPR, respectively), which are especially useful for imbalanced data (such as Neuropathic Pain).
We follow common practice [14, 52] regarding structure discovery performance, and consider metrics on the adjacency and orientation .
While the former does not take into account the direction of the edges, the latter does.
We report recall, precision and F 1 -score.
We also provide causal accuracy , a discovery metric that considers orientation [5].
4.1 Synthetic experiment
We simulate fifteen synthetic datasets.
For each simulated dataset, we first sample the true structure
G ; see Fig. 2(a) for an example.
The appendix provides detailed generation mechanism, including a visualisation of the data in Fig.
For each dataset, we simulate 5000 training and 1000 test samples.
Imputation performance .
VISL outperforms the baselines in terms of imputation across all synthetic datasets ( Table 1).The results grouped by the number of variables are presented by Table 8 in the appendix.
Structure discovery performance .
VISL obtains better performance than the baselines, see Table 2.
The results split by the number of variables are shown in the appendix, Table 10.
Specifically, we observe VISL consistently outperforms the baseline method in all metrics considered with all datasets (see Table 10).
In this small synthetic experiment, it is possible to visualize the predicted graph.
Fig.
3 shows the posterior probability of each edge (i.e. the estimated matrix G ) for the simulated dataset in Fig. 2(a).
Using a threshold of 0.5, we obtain the predicted graph in Fig. 2(b).
We observed that VISL recovers the ground truth graphs with one additional edge.
The sources of its advanced performances are twofold: 1) VISL uses the ELBO (Eq. 4) as training objective, a surrogate for marginal likelihood.
Kaiser and Sipos [22], Reisach et al.
[39] reported that L2 loss used in
NOTEARS is sensitive to data scaling and directs the learning of directions towards high variance nodes.
This can be resolved by using a proper likelihood training objective [28, 35]; 2) instead of using observed variables to construct the structural equation model (NOTEARS) or designing a specialized GNN structure (DAG-GNN), VISL is more flexible in terms of transforming the observed variables into a latent distribution via the encoder, and adopting a general message-passing GNN decoder.
Finally, VISL can scale to large data in terms of data points and dimensionality.
We demonstrate the computational efficiency with synthetic data ranging from 4 to 512 nodes in the appendix, Table 11.
4.2 Neuropathic pain dataset
We evaluate our method using a benchmark in healthcare applications [53].
The dataset contains records of patients regarding the symptoms associated with neuropathic pain.
There are 222 variables
[Table] [Table]
in this dataset.
Unlike the previous experiment with continuous data
[Table]
this dataset has binary variables indicating the symptoms.
The train and test sets have 1000 and 500 patients, respectively.
Imputation performance .
VISL shows competitive or superior performance when compared to the baselines, see Table 3.
Notice that AUROC and AUPR allow for an appropriate threshold-free assessment in this imbalanced scenario.
Indeed, as expected from medical data, the minority of values are 1 (symptoms); here, the prevalence of symptoms is around 8% in the test set.
Interestingly, it is precisely in AUPR where the differences between VISL and the rest of the baselines are larger except MICE, whose performance is very similar to VISL in this dataset.
Structure discovery results .
As in the synthetic experiment, VISL outperforms the causality-based baselines; see Table 4.
Notice that NOTEARS (NL) is slightly better in terms of adjacency-precision, i.e. the edges that it predicts are slightly more reliable.
However, this is at the expense of a significantly lower capacity to detect true edges, see the recall and the trade-off between both (F 1 -score).
4.3 Eedi topics dataset
Finally, we evaluate our method on an even more challenging real-world dataset in education to discover the group-wise structure between topics while the observations are question-answer pairs under these topics.
This is an important real-world problem in the field of AI-powered educational systems [56, 57].
This dataset is very sparse, with 74.1% of the values missing.
The dataset contains the responses from 6147 students to 948 mathematics questions.
The 948 variables are binary (1: correct and 0 otherwise).
These questions target specific mathematical concepts and are grouped into a meaningful hierarchy of topics ; see Fig. 4 in Appx.
E.
Here we apply VISL to find the structures among the topics using the third level hierarchy (Fig. 4), resulting in 57 group-level nodes.
Imputation results.
VISL achieves competitive or superior performance when compared to the baselines (Table 5).
Although the dataset is more balanced (54% of the values are 1), we provide AUROC and AUPR for completeness.
Notice that this setting is more challenging since the information flows at less granular level (i.e. group).
Interestingly, even in this case, VISL obtains similar or improved imputation results compared to the baselines.
Structure discovery results between groups.
Most of the baselines used so far cannot be applied here because i) they cannot deal with missing data or ii) they cannot learn group-level relationships.
DAG-GNN is the only one that can be adapted to satisfy both properties.
For missing data, we adapt
DAG-GNN following the same strategy as in VISL, i.e. replacing missing entries with a constant value.
For the second one, we further adapt it by using group-specific mappings like VISL to cope with arbitrary groups.
We also include another baseline, Random , where the structures between topics are randomly sampled from a Bernoulli distribution.
Due to the lack of ground truth relationships, we ask two experts (teachers) to assess the validity of the relationships found by VISL, DAG-GNN, and
Random .
For each relationship, they are asked to rate a value from 1 (strongly disagree) to 5 (strongly agree) for the adjacency (whether it is sensible to connect the two topics) and the orientation (whether the first one is a prerequisite for the second one).
We release the the complete list of relationships and expert evaluations for VISL, DAG-GNN, and Random ; see Table 15, Table 16, and Table 17, respectively.
Thus, the results is reproducible and allow the community to build on this data as a structure discovery benchmark.
In summary, Table 6 shows here the average evaluations: we see that the relationships discovered by VISL score much more highly across both metrics than the baseline models.
Another interesting aspect is how the relationships between level-3 topics are distributed in higherlevel topics.
Intuitively, it is expected that most of the relationships happen inside higher-level topics (e.g. Number-related concepts are more probably related to each other than to Geometry-related ones).
Table 7 in appendix shows the distribution for the compared methods.
Indeed, notice that the percentage of inside-topic relationships is higher for VISL (82%) and
DAG-GNN (42%) than for
Random (34%).
An analogous analysis for the 25 level-2 topics is provided in the appendix; see
Table 12 (VISL), Table 13 (DAG-GNN), and Table 14 ( Random ).
6% of the connections happen inside level 2 topics for Random , it is 14% for DAG-GNN and 36% for VISL.
Education Impact.
Lastly, to make a real-world impact, we have been provided with an additional education dataset in the same format as Eedi by an education organization to help provide insight for math curriculum building.
The final structure among all topics found by VISL is presented by figure
Fig.
8 in the appendix.
This figure provides insights into which topics are foundational and need to be covered earlier (topics with many originating edges), or later (topics with many incoming edges).
This allowed us to re-evaluate the order of topics in a nationwide used secondary curriculum.
Specifically, topics such as “arithmetic” or “properties of shapes” were moved earlier in the curriculum, while topics such as “negative numbers” or “proportion and similarity” were moved to a later stage in the curriculum.
Another interesting example found by the domain expert are “Venn diagrams”, which were originally taught in year 9/10 and are now suggested to be moved to year 7.
Experts found that the topic “Venn diagram” has been a useful tool in teaching other topics which are currently taught before year 10.
This emphasises the real-world impact our model can have in planning curricula.
Conclusions
We introduced VISL, a novel approach that simultaneously performs group-wise structure discovery and learns to impute missing values.
Both tasks are performed jointly: imputation is informed by the discovered relationships and vice-versa, leading to improved performance for both tasks.
Moreover, motivated by a real-world problem, VISLshows its impact in the real-world education domain to aid domain experts in setting up curriculum.
Despite of the improved performance on structure learning with missing data, VISL can be further extended in several directions.
First, one potential limitation is that the inferred structure from VISL is not causal.
Appendix B shows that VISL satisfies the cauasl Markov assumption under equilibrium, which opens a door for potential causal claims.
Another direction for future work is to extend the M(C)AR assumption to missing not at random (MNAR), which can be more practical for real world usage.
In the end, the performance can further be boosted by designing better graph posteriors beyond an independent Bernoulli distribution.
Acknowledgments and Disclosure of Funding
PMA acknowledges project PY20_00286 funded by Junta de Andalucía.