A DAPTIVE P SEUDO - LABELING FOR Q UANTUM C ALCULATIONS
Anonymous authors
Paper under double-blind review
A BSTRACT
Machine learning models have recently shown promise in predicting molecular quantum chemical properties.
However, the path to real-life adoption requires (1) learning under low-resource constraints and (2) out-of-distribution generalization to unseen, structurally diverse molecules.
We observe that these two challenges can be alleviated via abundant labels, often not the case in quantum chemistry.
We hypothesize that pseudo-labeling on a vast array of unlabeled molecules can serve as gold-label proxies to expand the training labeled dataset signiﬁcantly.
The challenge in pseudo-labeling is to prevent the bad pseudo-labels from biasing the model.
Motivated by entropy minimization framework, we develop a simple and effective strategy P SEUD σ that can assign pseudo-labels, detect bad pseudo-labels through evidential uncertainty, and prevent them from biasing the model using adaptive weighting.
Empirically, P SEUD σ improves quantum calculations accuracy across full data, low data, and out-of-distribution settings.
I NTRODUCTION
Accurate quantum mechanical (QM) calculations of drug-like molecules at CCSDT (coupled cluster single-double-triple) or MP2 (second order Møller-Plesset) (Watts et al., 1992) level of theory, which is essential to characterize biomolecular interactions, continue to be prohibitively expensive, despite recent advances in hardware capabilities.
Machine learning (ML) models have astonishing performance in approximating these calculations at a fraction of the computational cost (von Lilienfeld & Burke, 2020).
Such speedups could accelerate the discovery of new therapeutics.
In the absence of large-scale benchmark data sets reporting CCSDT or MP2 level calculations, most publications on this topic have relied on QM9, a standard benchmark of Density Functional Theory (DFT)-level energy and properties, for training and evaluating QM/ML models.
This data set contains molecular geometries and energies that were generated using DFT, which provides a faster (compared to CCSDT, for example), but less accurate description of molecular energetics and properties.
Previous publications have utilized computed geometries and properties from in-distribution molecules in
QM9 to test and validate model performance, showing excellent performance of QM/ML models as demonstrated by low mean absolute error (MAE) of predicted energies of other properties relative to the gold-standard DFT calculations (i.e., gold label).
However, it is unclear how the reported architectures (e.g., SchNet (Sch¨utt et al., 2017)) would perform in the regime of low but accurate data (CCSDT or MP2).
Two challenges remain in the way of the realistic adoption of ML-aided QM calculations.
Firstly, training molecules only cover part of the distribution, and real-world adoption requires out-ofdistribution generalization.
Secondly, computing QM properties for datasets of the size of QM9 is costly.
Widespread applicability of ML for QM (e.g., such as to CCSDT/MP2) thus requires models to generalize well given low-data constraints.
Both challenges listed above can be attributed to the difﬁculty and cost of generating experimental data in chemistry and materials science.
If we have a large and diverse set of labeled molecules, the ML model can generalize to larger chemical spaces and achieve better predictive performance.
Present work.
Our key observation is that state-of-the-art ML models can obtain a low-precision estimation of the properties even given a small set of labeled molecules.
This suggests that the
ML predicted label for any unlabelled molecule can be used as a low-precision estimation of the true label.
We hypothesize that by considering these low-precision pseudo-labels as the training targets in the training process, we can largely increase the training dataset size and diversity and alleviate the fundamental label scarcity issue, along with the associated challenges in low-data and out-of-distribution generalization.
In our study, we develop a simple, effective, and model-agnostic pseudo-labeling strategy called P SEUD σ .
Notably, we solicit an extensive array of unlabeled molecules from the PC dataset (Glavatskikh et al., 2019) where an ML model assigns pseudo-labels for them.
This generates a large set of ”labeled” training data points, even when only a relatively small number of reference
QM properties are available for training.
As the unlabeled dataset is diverse and unseen in QM9, it also helps generalization to unseen data.
One crucial issue in pseudo-labeling is the introduced bias from low-quality pseudo-labels.
To resolve this, we rely on a key observation that a data point with less evidence/higher model uncertainty is more likely to be of low-quality pseudo-label (Section 6).
Thus, we use model-generated evidential uncertainty to quantify each unlabeled data and use it to adaptively lower the weight of bad pseudo-labels in the training loss to reduce the bias effect.
In summary, we have made the following contributions: (1) Previous QM focus on in-distribution and label abundant setting while we point out that the realistic ML-aided QM challenges lie in low-data and out-of-distribution settings; (2) Pivoting away from the status quo in improving the physics-based representation, we propose to look at data-centric approaches on learning from the vast array of unlabeled molecules; (3) We propose an episodic pseudo-labeling scheduling strategy designed speciﬁcally for QM since many classic pseudo-labeling tricks do not work well in QM (e.g., data augmentation, model noise, student-training, re-initialization); (4) To prevent bad pseudo-label in biasing the training, we devise an adaptive weighting scheme where the weights are generated using evidential uncertainty such that bad pseudo-labels are automatically ﬁltered out; (5) We derive a theoretical connection between the evidential loss and entropy minimization framework; (6)
Empirically, we show P SEUD σ can improve QM accuracy for any atomistic model across full-data, low-data, and out-of-distribution settings.
R ELATED W ORKS
ML-aided quantum calculations.
Recently, many ML models have been proposed to improve quantum calculations.
They mainly focus on improving the physics-based representation in the full
QM9 dataset setting (Sch ¨ utt et al., 2017; Unke & Meuwly, 2019; Anderson et al., 2019; Lu et al., 2019; Klicpera et al., 2020; Liu et al., 2021; Qiao et al., 2021).
In contrast, our work proposes to shift the focus on improving training strategy instead of the model architecture.
In addition, P SEUD σ is model-agnostic, and it can improve on any atomistic model.
Additionally, we focus on realistic quantum calculations such as learning in the low-data regime and out-of-distribution inference.
Pseudo-labeling.
Pseudo-labeling/self-training generates pseudo-labels for unlabeled data.
Numerous works exist for how to assign pseudo-labels, notably through trained ML model prediction (Lee et al., 2013), label propagation(Shi et al., 2018; Iscen et al., 2019), and history cache (Likhomanenko et al., 2021; Higuchi et al., 2021).
P SEUD σ is different as it focuses on detecting and preventing bad pseudo-label from affecting the model.
Also, P SEUD σ adopts a novel episodic pseudo-label strategy with a re-initialized learning rate.
(Xie et al., 2020) re-initialize the network as a student when a new pseudo-label set is generated along with noise per epoch.
In contrast, P SEUD σ has no student, and no noise as both are shown to be ineffective for QM in Section 6.
In addition, small perturbational noise in 3D molecular geometry could easily lead to a drastic energy difference.
Thus, the noise strategy does not work for QM tasks.
More related is a concurrent work (Rizve et al., 2021) that develops an uncertainty-aware pseudo-labeling strategy, but they introduce additional hyperparameters to remove pseudo-labels at some uncertainties.
In contrast, P SEUD σ uses an effective adaptive weighting scheme, along with an episodic pseudo-labeling training schedule.
Additionally, P SEUD σ is the ﬁrst method that studies pseudo-label in quantum calculations, which present unique challenges and motivations.
Uncertainty.
Model uncertainty is a well-studied subject (Kendall & Gal, 2017; Lakshminarayanan et al., 2017; Blundell et al., 2015).
(Amini et al., 2020) use evidential uncertainty to add a prior over the gaussian parameters to search for higher-order patterns for regression tasks.
P SEUD σ leverages evidential uncertainty as the uncertainty measure.
Note that P SEUD σ is uncertainty measure-agnostic.
We can easily switch to alternative uncertainty measures.
Recently, (Soleimany et al., 2021) adapt evidential uncertainty and has shown it can successfully help guide property prediction. In contrast, [Table] =</latexit
[Table]
</ [Table] [Equation] [Table]
[Equation]
[Table]
[Figure]
is used as weight to adaptively adjust the loss in this episode’s neural network f ( k ) ’s training to reduce the effect of bad pseudo-labels in an inner-loop training with N epochs.
we leverage evidential uncertainty as a proxy for pseudo-label quality to tackle low-data and out-ofdistribution challenges in realistic quantum calculations setup.
P ROBLEM F ORMULATION
Let X = { x 1 , . . .
, x N } denote N molecules, where each molecule x i is uniquely deﬁned by 3D coordinates { ( a ij , b ij , c ij ) } N i j =1 for N i atoms with atom types { t j } N i j =1 in the corresponding molecule.
We then denote Y =
{ y 1 , . . .
y N } a set of quantum mechanical properties for each molecule.
The labeled dataset thus consists of a set of pairs of 3D coordinates and scalar labels
[Equation]
In addition to the labeled data, we solicit a large quantity of unlabeled data to generate pseudo-labels.
We denote an unlabeled dataset U = { x 1 , . . .
, x M } , where M is the size of the unlabeled dataset.
Given an atomistic model f ( · ) , we can generate pseudo-labels ˆ Y =
{ ˆ y 1 , . . .
, ˆ y M } , where ˆ y i
=
f
( x i ) for x i ∈ U .
The problem is to ﬁnd a machine learning-based atomistic model f : x (cid:55)→ y that can establish an accurate map from 3D coordinates to the quantum mechanical properties of the molecules, with the help of pseudo-labeled dataset U .
P SEUD σ : A DAPTIVE P SEUDO - LABELING F OR Q UANTUM C ALCULATIONS
P SEUD σ (Figure 1) is an approach for quantum chemical property prediction.
Building on theoretical motivation from Section 5, P SEUD σ solicits pseudo-labels on a vast array of an unlabeled dataset to increase the diversity of the training space via an episodic labeling strategy.
Then, it adaptively weights the pseudo-labels using evidential uncertainty to allow a positive transfer.
The overview is in
Algorithm 1.
Episodic Pseudo-labeling.
We devise a pseudo-labeling strategy that can ensure learning from the pseudo-labels to the fullest extent for QM.
We have made two distinct modiﬁcations compared to existing works. First is the pseudo-label scheduling. In the standard pseudo-labeling (Lee et al., 2013), pseudo-labels are updated in every update and the model is continuously trained.
In contrast, we devise an episodic training strategy, where each episode consists of K epochs, and pseudo-labels are regenerated in every episode, while the model is continuously trained.
This is important because we observe that fast updates on pseudo-labeling prevents the model to fully learn from all useful information in pseudo-labels.
In contrast, our episodic approach gives the model more time to absorb useful information from a given set of pseudo-labels. Second modiﬁcation is the model update.
In self-training (Rizve et al., 2021; Xie et al., 2020), a set of pseudo-labels are regenerated after K epochs and the model is reinitialized.
Instead, we train the same model throughout episodes.
This new strategy allows the model to be exposed to a larger number of labels or training data points given the same time frame.
For each episode, we also reinitialize the learning rate with a small step-wise decay strategy to allow the model a chance to jump out of the local optimum from the previous set of pseudo-labels.
Formally, P SEUD σ mainly consists of three stages: in the ﬁrst stage, regular training is conducted on labeled data D , and the output model is the initialized model f (1) .
In the second stage, the updated model at episode k then conducts inference on the entire unlabeled data ˆ Y = f
( k ) ( U ) to generate the pseudo-label set.
The per-episode pseudo-label set is then combined with the gold-labeled data to form the training data for the next episode.
In the third stage, the model is further trained using the combined dataset to get a new model f
( k +1) after N epochs.
The second and third stages are then reiterated till the loss stops decreasing.
Evidential uncertainty quantiﬁcation.
Pseudo-labels are noisy.
Many are incorrect and can potentially lead to negative transfer.
Thus, it is instrumental in detecting the quality of pseudo-labels.
However, there is no auxiliary information in the dataset about the pseudo-labels.
Thus, we need to quantify it through some proxies that can be assigned without auxiliary information.
Our key observation is that low-quality pseudo-labels have high model uncertainty, and high-quality pseudo-labels have low model uncertainty.
Another advantage of model uncertainty is that it can be estimated solely from x , if we make it model uncertainty-aware.
Building on the theoretical motivation about the connection between evidential uncertainty and entropy minimization in Section 5, we use evidential uncertainty as to the proxy for label quality.
The evidential modeling of molecular property allows us to derive analytical solution of uncertainty, which could be directly used to weight the pseudo-labels.
Formally, we can model the label probabilistically as it is drawn from ( y 1 , · · · , y i ) ∼ N ( µ, σ 2 ) , where the mean µ are variance σ 2 are unknown.
To estimate them, we pose a prior
[Equation]
where the parameters θ = ( µ, σ ) is an instantiation of the posterior p ( µ, σ 2 | γ, v, α, β ) .
The choice of prior allows the factorization p ( µ, σ 2 ) = p ( µ ) p ( σ 2 ) (Jordan, 2009).
The posterior then becomes a NormalInvGamma( γ, v, α, β ) where the maximum likelihood estimation of θ can be analytically found as
[Equation]
Here, E [ σ 2 ] plays the role of the aleatoric (data) uncertainty.
The uncertainty of the model prediction can also be calculated, i.e. epistemic uncertainty:
[Equation]
As the MLE is deterministic, the model can output the four prior parameters { γ, v, α, β } directly where the prediction and uncertainty can be derived from them analytically.
The prior is optimized by evidential loss L evi (Amini et al., 2020):
[Equation]
where the ﬁrst term is to maximize the log-likelihood of the posterior predictive, which is derived as the Student’s t-distribution. The second term is a regularizer that encourages lower total evidence, which correlates positively to higher epistemic uncertainty, when the gap between ground truth and prediction is high (i.e. when model prediction is with high error). Similarly, it encourages lower uncertainty when the model prediction is errorneous.
This encourages the model to generate an accurate estimate of uncertainty or the degree of errors for the pseudo-labeled data points.
The regularization is controlled by a hyperparameter λ .
Adaptive weighting.
The evidential uncertainty detects the low-quality pseudo-labels.
The next step is to remove the noisy effect from the model training.
Naive methods often use removal based on a threshold (Rizve et al., 2021).
However, it has two disadvantages: (1) it introduces a new hyperparameter - the threshold; (2) it removes a portion of unlabeled noisy data, which can contain useful information.
Instead, we propose an adaptive weighting mechanism that adapts the evidential loss given the inverse epistemic uncertainty.
Intuitively, a higher uncertainty data point requires a lower effect in the loss function because it is more likely that the sample has a low pseudo-label quality,
Input : Labeled data D = { ( x 1 , y 1 ) , · · · , ( x N , y N ) } , unlabled data U = { x 1 , · · · , x N U } ˆ
U ← {} , ˆ W ← {} // Initialize with empty unlabeled data for k ∈ { 1 , · · · , K } //
Outer-loop with K episodes do
T ← D ∪ ˆ U // Join updated pseudo-labels for ( x i , y i ) ∈ T // Inner-loop with N epochs do θ i =
( γ i , v i , α i , β i ) =
f ( k − 1) ( x i ) //
Evidental parameters ˆ y i
=
E [ µ ] = γ
i //
Posterior prediction
L = L( ˆ y i , y
i , θ i , ˆ W i ) //
Adaptive evidential loss via Eq.f ( k − 1) =
Update( f ( k − 1) , L ) //
Inner-loop update end f ( k ) ← f ( k − 1) // Update teacher model for pseudo-labels for x i ∈ U do ˆ θ i =
(ˆ γ i , ˆ v i , ˆ α i , ˆ β i ) =
f ( k ) ( x i ) ˆ y i
= ˆ γ i // Infer a new set of pseudo-labels ˆ U i ← ( x i , ˆ y i ) // Update pseudo-labels ˆ W i ← Var[ µ ] − 1 i
= ˆ v i ∗ (ˆ α i − 1) / ˆ β
i // Update adaptive weights end end
and we want to reduce its effect on the model.
Conversely, if a pseudo-label has low uncertainty, the label quality is high enough to be used as a proxy of a gold-label.
Thus, we should assign a higher score in the loss for it. The uncertainty is from the teacher model in the previous episode and is ﬁxed throughout the current episode. Thus, the adaptive weight for each pseudo data point i becomes ˆ W i =
Var[ µ ] − 1 i .
The ﬁnal loss then becomes
[Equation]
where the ﬁrst term corresponds to the labeled dataset D does not have weights unlike the second term corresponding to the unlabeled data.
This adaptive loss solves two disadvantages since it has zero hyperparameters, and it removes the effect of bad pseudo-labels while retaining all training examples including the noisy ones to maximize the diversity of the training space.
P SEUD σ M OTIVATION : C ONNECTION TO E NTROPY M INIMIZATION
We derive motivation about why evidential uncertainty and the weighting mechanism could be beneﬁcial to pseudo-labeling based on the entropy minimization framework for semi-supervised learning from Grandvalet & Bengio (2004; 2006); Lee et al.
Notably, our use of Bayesian modeling enables us to analytically derive a conditional entropy for pseudo-labeled data.
We ﬁnd that evidential loss strongly relates to conditional entropy, and minimizing evidential loss directly minimizes entropy.
Secondly, we ﬁnd the conditional entropy could be decomposed into the inverse epistemic uncertainty and the log-likelihood, which motivates our weighting mechanism.
In the standard regression setting, one seeks to maximize the likelihood of the model p θ ( Y|X ) on the labeled data set D .
To utilize the unlabeled data set, we need to extract some useful information on how the model behaves on the unlabeled dataset and inject this information to improve the model.
To measure the utility, entropy H ( Y | U ) is introduced (Grandvalet & Bengio, 2006) as a proxy to measure the amount of information in unlabeled data: [Equation]
[Equation]
Throughout the text, we are referring to entropy as Shannon entropy.
Intuitively, high entropy is indicative of the poor model conﬁdence on the unlabeled data point, while the low entropy can be interpreted as high conﬁdence of the model predictions.
High entropy, associated with random predictions, while low entropy is associated with non-random behavior.
Hence force, we hypothesize that small entropy may be indication of a signal that our model can beneﬁt from. Small entropy, as seen below, corresponds to high model conﬁdence. and vice versa. Large entropy corresponds to high model uncertainty.
Entropy minimization framework casts the regression as the following optimization problem:
[Equation]
where c is the proportionality constant.
Intuitively, here, the objective tends to maximize the loglikelihood on the labeled dataset while minimizing the entropy on the unlabeled data set at the same time to transfer knowledge from unlabeled data.
In previous works (Sch¨utt et al., 2017; Liu et al., 2021), molecule properties are not modeled probabilistically such that entropy calculation is infeasible.
In contrast, P SEUD σ uses Bayesian modeling approaches that allow us to analytically calculate the entropy.
For every molecule x i the machine learning model outputs four parameters f
( x i ) =
( α i , β i , γ i , ν i ) .
Based on these parameters, the likelihood of label y given the input molecule x
i is given by the Student’s t-distribution in the context of evidential regression
[Equation]
evaluated at location parameter γ i , Student’s t-distribution scale parameter σ 2 st,i = β
[Equation]
and 2 α i degrees of freedom.
The entropy of the Student’s t-distribution given in terms of evidential parameters is readily available (Appendix A):
[Equation]
where Ψ is a digamma function and B( · , · ) is a beta function.
If we take model (epistemic uncertainty) (Eq. 3) as our evidential uncertainty, we can show that minimizing entropy directly relates to minimizing epistemic uncertainty.
We plot the relation between the entropy and epistemic uncertainty in the Figure 2.
As the next steps, we aim to uncover the dependence of the entropy on the model uncertainty of our pseudo-labeling approach. This can be done if we make two simpliﬁcations in entropy evaluation.
Firstly, to introduce iterations as in pseudo-labeling, we replace entropy with the crossentropy between two probability distributions: the predictions y are generated from the probability distribution p θ ( t − 1) ( y | x i ) at iteration step t − 1 and log-likelihood are evaluated with respect to probability distribution p θ ( t ) ( y | x i ) at iterative step t:
[Equation]
Upon convergence, t → + ∞ , the probability distributions at every iterative step p θ ( t − 1) ( y | x i ) ≈ p θ ( t ) ( y | x i ) and are approximately the same and one can view introduced cross-entropy with respect to time step t as entropy.
At the earlier stages of training cross-entropy acts as a regularizer encouraging network parameters θ ( t ) to match θ ( t − 1) . As a second approximation, to uncover model uncertainty in mathematical formulas, we approximate the probability distribution at time step t − 1 .
We resort to empirical estimate of the entropy, as done in (Grandvalet & Bengio, 2004). We select labels y at the highest mode of probability distribution p θ ( t − 1) ( y | x i ) , which corresponds to y =
γ t − 1 i . We obtain the following approximate for the entropy:
[Equation]
where the log probabilities are weighted by empirical probabilities as weights E t − 1 i evaluated at iterative step t − 1 when plugged into Eq. 8 (also see Appendix. Eq. 18 for exact formula of Student’s t-distribution)
[Equation]
To establish a relationship between empirical weights E t − 1 i and aleatoric E [ σ 2 i ] / epistemic Var[ µ i ] uncertainties we rewrite
[Equation] Empirical coefﬁcients depend on aleatoric, epistemic uncertainties and α i parameter, which can be interpreted as virtual observations in support of the variance estimation (Jordan, 2009). In the limiting case α i (cid:29) 1 one can approximate beta function via Stirling formula B( 12 , α i ) ≈ √ π α − 12 i and empirical weights become
[Equation] We can express the empirical coefﬁcients depend both on aleatoric and epistemic uncertainties in a symmetric fashion. We selected adaptive pseudo-labeling coefﬁcients W i in our pseudo-labeling approach Eq. 5 to be inverse epistemic/model uncertainties.
We can see, that those coefﬁcients directly relate to empirical coefﬁcients derived from entropy minimization approach Eq. 14, as empirical coefﬁcients also depend on model uncertainty in the inverse fashion. As the model uncertainty increases, the empirical coefﬁcients E i tend to decrease to minimize the entropy.
[Table]
[Table]
E XPERIMENTS 6. D ATASET AND E XPERIMENTAL S ETUPS
We evaluate P SEUD σ using QM9 dataset (Wu et al., 2018) under two settings.
(A) Full-data : We follow the previous works (Liu et al., 2021; Klicpera et al., 2020) where a 110,000/10,000/10, training/validation/testing set is obtained.
For the unlabeled data, we solicit to PC9, a dataset of 99,234 molecules that consists of the same elements as QM9, curated by (Glavatskikh et al., 2019).
(B) Low-data : we set k % of QM9 full training set as the training set (i.e. k × 110,000) and we remove the label of (1- k %)
QM9 full training set and make it as the unlabeled set.
We evaluate in two k values, 1 and 10, which means only 1,100/11,000 data points is trained respectively.
Dataset statistics summary is in Table 1.
Note that PC9 has a wider chemical diversity than QM9, demonstrated by wider distribution of distances of chemical bonds and more functional groups (Glavatskikh et al., 2019).
P SEUD σ is model-agnostic.
We evaluate it with two model backbones SchNet (Sch ¨ utt et al., 2017)
(P SEUD σ -S) and DimeNet++ (Klicpera et al., 2020) (P SEUD σ -D).
We do not experiment with the
SOTA atomistic model SphereNet (Liu et al., 2021) because it is highly computationally expensive.
Our result is conducted on two targets σ HOMO , σ LUMO , because the PC9 dataset only has these two targets.
We use mean absolute error as the evaluation metric.
For baselines, we compare with 6 state-of-the-art baselines, including SchNet (Sch ¨ utt et al., 2017),
PhysNet (Unke & Meuwly, 2019), Cormorant (Anderson et al., 2019), MGCN (Lu et al., 2019),
DimeNet++ (Klicpera et al., 2020), and SphereNet (Liu et al., 2021).
We report the best results taken from the original authors’ paper while using the same fraction of data split in the full data setting.
For P SEUD σ , we conduct two hyperparameter tunings on σ HOMO with SchNet backbone on the validation MAE with full data/low-data setting, respectively.
The optimal hyperparameter is then used for both targets.
Note that the atomistic model itself has the same hyperparameter as the original authors.
Code will be released upon anonymous review period.
6. R ESULTS
Overview of results.
We report performances of P SEUD σ in full data (Table 2), low-data (Table 3), out-of-distribution (Table 4) settings and ﬁnd P SEUD σ achieves the best performance across all settings, suggesting the robustness of the pseudo-labeling strategy.
Systematic ablation study (Table 5) also show the importance of each module in P SEUD σ .
P SEUD σ improves on fully supervised QM calculations.
We report P SEUD σ against 6 state-of-theart models in Table 2.
P SEUD σ -D surpasses all baselines in both targets σ HOMO , σ LUMO .
Notably,
P SEUD σ -D improves the SOTA by 3.2 meV, a signiﬁcant margin.
Particularly, comparing P SEUD σ -S with SchNet and P SEUD σ -D with DimeNet++, we ﬁnd P SEUD σ can consistently improve even on the fully supervised setting by a large margin (8.1 meV for SchNet and 4.2 meV for DimeNet++), highlighting the utility of P SEUD σ and the high quality of PC9 as unlabeled data. It also shows that [Table] [Table]
this direction of improving learning strategy instead of improving physics-based representation has potentials
[Table]
P SEUD σ signiﬁcantly improves on low-data QM calculations.
In Table 3, we investigate how
P SEUD σ can improve on the low-data regime with only 1%, 10% training data point.
That is to do
QM using only 1,100 and 11,000 known data.
This is to simulate the more expensive QM levels such as CCSD(T)/MP2.
We observe P SEUD σ can consistently and signiﬁcantly improve the prediction in σ HOMO , σ LUMO across both low-data settings and both model backbones, suggesting P SEUD σ can help prediction in realistic low-data quantum calculations.
Notably, in σ LUMO with 1% of QM9 data,
P SEUD σ improves upon SchNet by 57.8 meV, a considerable margin.
We also observe that the gain margin is much more signiﬁcant when the number of training data is smaller.
This showcases the utility of P SEUD σ in extremely low-resource settings, such as QM.
P SEUD σ improves out-of-distribution QM calculations.
Another realistic challenge is to infer accurately on unseen data distribution away from QM9.
We conduct inference on the PC9 dataset where it has calculated σ HOMO , σ LUMO .
We also ﬁnd P SEUD σ can again signiﬁcantly improve OOD accuracy over DimeNet++, a SOTA method, with over 16.0 meV improvement on σ HOMO and 8.
meV improvement on σ LUMO , highlighting the robustness of P SEUD σ .
[Figure]
Evidential uncertainty highly correlates to label quality.
[Figure] σ utilizes uncertainty as a proxy of label quality because they are highly correlated for unseen molecules.
In this experiment, we want to validate this hypothesis.
We train on the complete QM training set with evidential uncertainty and then infer on the QM testing set.
We ﬁnd that the non-parametric Spearman correlation between MAE and epistemic uncertainty is 0.42 with a p-value < 1e-16.
Additionally, we evaluate on PC9 out-of-distribution set, and the Spearman correlation is 0.35 with p-value <
1e-16, suggesting our uncertainty is a robust measure of label quality.
Ablations.
In Table 5, we conduct a systematic ablation study using
SchNet as the backbone architecture on the fully supervised QM9 setting.
We show that each component in P SEUD σ is indispensable for P SEUD σ .
In Table 2, we have reported original authors best performance following standard practices Klicpera et al. ; Liu et al. To further clearly demonstrate the utility of pseudo-labeling, in -pseudo-label, we keep all hyperparameters the same but remove the pseudo-labeling part. We show our pseudo-labeling strategy improve by a large margin.
Next, in the -uncertainty ablation, we use a vanilla per-epoch pseudo-labeling strategy with no uncertainty, corresponding to a vanilla pseudo-labeling strategy. We demonstrate it does not work for QM as it even deteriorates compared to no pseudo-labeling (-pseudo-label), calling for specialized strategy design such as ours.
Then, to compared with self-training strategy, -student ablation retrains a model in every episode as in (Xie et al., 2020) and the decreased performance shows that it is not ideal in QM calculations.
Lastly, the -uniform ablation uses the same weight for all pseudo-labels with no uncertainty reweighting. The decreased performance shows the importance of detection and [Table] adaptive removal of bad pseudo-labels, achieved by our evidential characterization of the molecular property.
C ONCLUSION
We introduce P SEUD σ , a simple, effective, model-agnostic pseudo-labeling strategy that can improve quantum calculations accuracy in abundant data, low data and out-of-distribution settings.
P SEUD σ learns from vast unlabeled data by assigning uncertainty-aware pseudo-labels.
These pseudo-labels are adaptively selected to be absorbed into the model via an episodic schedule.
Unlike eariler methods in QM that focuses on physics-based representation, we show the potential of data-centric approach A E NTROPY OF S TUDENT ’ S T - DISTRIBUTION
While the entropy of the Student’s t-distribution is well known, we derive it for completeness.
Student’s probability distribution deﬁned in terms of location γ , scale factor σ 2 st and ν st degrees of freedom is
[Equation]
where Γ i a gamma function.
Student’s t-distribution can be written in terms of beta function
B = Γ( x )Γ( y ) Γ( x + y ) if we take advantage of the fact that Γ( 12 ) = √ π
[Equation]
In the main text of the manuscript, we used empirical estimate of Student’s t-distribution, which corresponds to evaluation at the highest mode y = γ .
Empirical estimation of probability becomes:
[Equation]
If we introduce a new variable t = y − γ σ st , Student’s t-distribution converts into the standard form with probability density
[Equation]
A. P ROPOSITION
Proposition : Entropy of the generalized and standard Student’s t-distributions are related via the formula
[Equation]
Proof: The transformation t = g( y ) =
y − γ σ st is bijective and invertible with the inverse transformation y =
g − 1 ( t ) =
σ st t + γ .
The Jacobin of the transformation g is dd y g
( y )
= 1 σ
st .
According to the change of variables probability density formula
[Equation]
The equation for the entropy transformation (equation 20) follows directly from the deﬁnition of the entropy.
To ﬁnd the generalized entropy, we just need to calculate the entropy of the standard Student’s t-distribution
[Equation]
[Equation]
To ﬁnd the second integral
[Equation]
we make a substitution x = t 2 ν st and obtain ν st +
[Equation]
where digamma function is deﬁned as Ψ( x )
=
Γ (cid:48)
( x ) Γ( x ) .
Putting all the terms together
[Equation]
the entropy of the standard Student’s t-distribution becomes
[Equation]
The ﬁnal formula for the entropy of the labels y is given by
[Equation]
B P SEUDO - LABELING , E NTROPY MINIMIZATION AND A LEATORIC UNCERTAINTY
In the main section of the text, we have considered the case where observed targets ( y 1 , · · · , y i ) ∼
N ( µ, σ 2 ) are drawn from the Normal distribution with unknown mean and variance µ and σ 2 and we have imposed a prior on them.
The problem is signiﬁcantly simpliﬁed if we treat µ and σ 2 in a deterministic way, such that our model f outputs two parameters µ and σ 2 .
The model here is able to estimate aleatoric (data) uncertainty σ 2 but unable to model epistemic (model) uncertainty.
By minimizing negative log-likelihood, the loss is signiﬁcantly simpler than in Eq.
[Equation]
Empirical estimate of the entropy on the unlabeled data set becomes
[Equation]
(33) with log probability weights E empi
=
{+1 √ 
+}
πσ 2 i .
One can notice that the weights are inversely related to aleatoric uncertainties E i ∼ ( σ 2 i ) − 12 .