Given such a sparsiﬁer, by setting S = { u } and T = { v } , one can recover whether or not edge ( u, v ) is present in G for any u, v ∈ V . discount in its parent’s contribution to the cost, which after cascading gives a third view of Eq.

Given such a sparsiﬁer, by setting S = { u } and T = { v } , one can recover whether or not edge ( u, v ) is present in G for any u, v ∈ V . is the second observation: the negative term w G ( S ∪ T, S ∪ T ) that internal node S contributes to the cost also appears as a positive term in its parent’s contribution to the cost. We can pass this term as a discount in its parent’s contribution to the cost, which after cascading gives a third view of Eq.


Line-Entry System) or GNN-based (Graph Neural Networks) MRL methods either take SMILES strings as input that have difﬁculty in encoding molecule structure information, or over-emphasize the importance of GNN architectures but neglect their generalization ability. Here we propose using chemical reactions to assist learning molecule representation. The key idea of our approach is to preserve the equivalence of molecules with respect to chemical reactions in the embedding space, i.e., forcing the sum of reactant embeddings and the sum of product embeddings to be equal for each chemical equation. This constraint is proven effective to 1) keep the embedding space well-organized and 2) improve the generalization ability of molecule embeddings. Moreover, our model can use any GNN as the molecule encoder and is thus agnostic to GNN architectures. Experimental results demonstrate that our method achieves state-of-the-art performance in a variety of downstream tasks, e.g., 17.4% absolute Hit@1 gain in chemical reaction  prediction, 2.3% absolute AUC gain in molecule property prediction,  and 18.5% relative RMSE gain in graph-edit-distance prediction, respectively, over the best baseline method. All experimental code is provided in the supplementary material.

Line-Entry System) or GNN-based (Graph Neural Networks) MRL methods either take SMILES strings as input that have difﬁculty in encoding molecule structure information, or over-emphasize the importance of GNN architectures but neglect their generalization ability. Here we propose using chemical reactions to assist learning molecule representation. The key idea of our approach is to preserve the equivalence of molecules with respect to chemical reactions in the embedding space, i.e., forcing the sum of reactant embeddings and the sum of product embeddings to be equal for each chemical equation. This constraint is proven effective to 1) keep the embedding space well-organized and 2) improve the generalization ability of molecule embeddings. Moreover, our model can use any GNN as the molecule encoder and is thus agnostic to GNN architectures. Experimental results demonstrate that our method achieves state-of-the-art performance in a variety of downstream tasks, e.g., reaction product prediction, molecule property prediction, reaction classiﬁcation, and graph-edit-distance prediction. The code is available at https://github.com/hwwang55/MolR .

Instruction: Remove unnecessary details on specific numerical performance of the model. Link to https://github.com/hwwang55/MolR instead of supplementary material.

Instruction: Make the second last sentence from the end of this paragraph more concise by removing too precise details.  For the last sentence, the code is now provided on github.


We have proposed FROB which uses the generated support boundary of the normal data distribution for few-shot OoD detection. FROB tackles the few-shot problem using classiﬁcation with OoD detection. In real-world applications, in the wild, it is a challenge to robustly perform classiﬁcation and few-shot OoD detection with high levels of reliability. The contribution of FROB is the combination of the generated boundary in a self-supervised learning manner and the imposition of low conﬁdence at this learned boundary. To improve robustness, FROB generates strong adversarial samples on the boundary, and forces samples from OoD and on the boundary to be less conﬁdent. By including the  boundary, FROB reduces the threshold linked to the model’s few-shot robustness. FROB redesigns, restructures, and streamlines OE to work even for zero-shots. FROB maintains the OoD performance approximately constant, independent of the few-shot number. The performance of FROB with the self-supervised learning boundary is robust and effective as the performance is approximately stable as the few-shot outliers decrease in number, while the performance of FROB without O ( z ) decreases as the few-shots decrease. The evaluation of FROB, on many sets, shows that it is effective, achieves competitive state-of-the-art performance, and outperforms benchmarks in the few-shot OoD detection setting in AUC-type metrics. In the future, in addition to conﬁdence and  class, FROB will also output important regions and bounding boxes around abnormal objects.

We have proposed FROB which uses the generated support boundary of the normal data distribution for few-shot OoD detection. FROB tackles the few-shot problem using classiﬁcation with OoD detection. The contribution of FROB is the combination of the generated boundary in a self-supervised learning manner and the imposition of low conﬁdence at this learned boundary. To improve robustness, FROB generates strong adversarial samples on the boundary, and forces samples from OoD and on the boundary to be less conﬁdent. By including the self-produced boundary, FROB reduces the threshold linked to the model’s few-shot robustness. FROB redesigns, restructures, and streamlines OE to work even for zero-shots. It robustly performs classiﬁcation and few-shot OoD detection with a high level of reliability in real-world applications, in the wild. FROB maintains the OoD performance approximately constant, independent of the few-shot number. The performance of FROB with the self-supervised learning boundary is robust and effective, as the performance is approximately stable as the few-shot outliers decrease in number, while the performance of FROB without O ( z ) decreases as the few-shots decrease. The evaluation of FROB, on many sets, shows that it is effective, achieves competitive state-of-the-art performance, and outperforms benchmarks in the few-shot OoD detection setting in AUC-type metrics. In the future, in addition to conﬁdence and the class, FROB will also output important regions and bounding boxes around abnormal objects.


Several other variations of this basic setup have been considered. For example, [12] have considered this problem in the presence of structural constraints. [11, 31, 34] considered a setting where vertices are embedded in a metric space and the similarity/dissimilarity between two vertices is given by their distances. The most relevant to our work amongst these is [34] which considered this metric embedded hierarchical clustering problem in a streaming setting. However, the stream in their setting is composed of vertices while edge weights can be directly inferred using distances between vertices; whereas the stream in our streaming setting is composed of edges while vertices are already known. Moreover, their study is only limited to the streaming setting. There has also been work on designing faster/parallel agglomerative algorithms such as single-linkage, average-linkage etc.[40, 17]. However, these  algorithms  are not known to achieve a good approximation factor for Dasgupta’s objective, which is  the main focus of our paper. [27] studied the hierarchical clustering problem in an MPC setting. However, their work only considered the maximization objectives[32, 15], while our work is primarily focussed on the minimization objective of [16].

Several other variations of this basic setup have been considered. For example, [12] have considered this problem in the presence of structural constraints. [11, 34, 37] considered a setting where vertices are embedded in a metric space and the similarity/dissimilarity between two vertices is given by their distances. The most relevant to our work amongst these is [37] which considered this metric embedded hierarchical clustering problem in a streaming setting. However, the stream in their setting is composed of vertices while edge weights can be directly inferred using distances between vertices; whereas the stream in our streaming setting is composed of edges while vertices are already known. Moreover, their study is only limited to the streaming setting. There has also been work on designing faster/parallel agglomerative algorithms such as single-linkage, average-linkage etc. While these works share the same motivation as ours, namely, scaling HC algorithms to massive datasets, these results are largely orthogonal to ours. The primary philosophical difference is that these aforementioned works are aimed at speeding up/parallelizing very speciﬁc kinds of linkage based algorithms, while recovering the same or similar cluster trees (under very different notions of similarity) that would have been computed by the slower/sequential algorithm. Moreover, the speciﬁc algorithms considered in these works have no known approximation guarantees for Dasgupta’s objective. Our work on the other hand approaches this problem from an optimization perspective. Through data sparsiﬁcation, we aim to recover a cluster tree with marginal loss in objective function value as compared to one computed over the entire (dense) input data by any given HC algorithm as a blackbox, achieving a speedup in runtime or reducing its memory requirement due to sparsity. [29] studied the hierarchical clustering problem in an MPC setting. However, their work only considered the maximization objectives [35, 17], while our work is primarily focussed on the minimization objective of [18].


Modern deep neural networks have achieved unprecedented success in known contexts for which they are trained, yet they do not necessarily know what they don’t know (Nguyen et al., 2015). In particular, neural networks have been shown to produce high posterior probability for out-of-distribution (OOD) test inputs, which  should not be predicted by the model. Taking self-driving car as an example, an object detection model trained to recognize in-distribution objects ( e.g., cars, stop signs) can produce a high-conﬁdence prediction for an unseen object of a moose (see Figure 1(a)). Such a failure case raises concerns in model reliability, and worse, may lead to a catastrophic effect when deployed in safety-critical applications.

Modern deep neural networks have achieved unprecedented success in known contexts for which they are trained, yet they often struggle to handle the unknowns. In particular, neural networks have been shown to produce high posterior probability for out-of-distribution (OOD) test inputs (Nguyen et al., 2015), which arise from unknown categories and should not be predicted by the model. Taking self-driving car as an example, an object detection model trained to recognize in-distribution objects ( e.g., cars, stop signs) can produce a high-confidence prediction for an unseen object of a moose; see Figure 1(a). Such a failure case raises concerns in model reliability, and worse, may lead to catastrophe when deployed in safety-critical applications.

Instruction: Improve the English of this paragraph.

Instruction: Make this paragraph more formal and fitting to academic style.


We proposed a new inference lgorithm for distributions parametrized by normalizing ﬂow models. The need for approximate inference is motivated by our theoretical hardness result for exact inference,which is surprising given that it applies to invertible models. We also presented a detailed empirical evaluation of our method with both quantitative and qualitative results on a wide range of tasks and datasets. Overall, we believe that the idea of a pre-generator creating structured noise is a useful and general method for leveraging pre-trained generators to solve new generative problems.

We proposed a new inference algorithm for distributions parametrized by a ﬂow. The need for approximate inference is motivated by the hardness of exact inference. We also presented a detailed empirical evaluation of our method with both quantitative and qualitative results on a wide range of tasks and datasets. Overall, we believe that the idea of a pre-generator creating structured noise is a useful and general method for leveraging pre-trained generators to solve new generative problems.

Instruction: Remove details which are unnecessary for the overall paragraph. Fix any spelling mistakes.

Instruction: Correct and concise the two first sentences.


We use the following toy example to further illustrate the trade-offs achieved by EVL. Consider a random generated MDP. When the operator can be applied exactly, the Bellman optimality operator is sufﬁcient to learn the optimal value V ∗ . However, applying operators with an ofﬂine dataset raises a noise on the actual operator due to the estimation error with ﬁnite and biased data. We simulate this effect by adding random Gaussian noise to the operator. Applying the optimality operator on ofﬂine datasets can lead to severe overestimation due to the maximization bias and bootstrapping. The value estimation learned by EVL, on the contrary, achieves a trade-off between learning optimal policy and behavior cloning and can be close to the optimal value with proper chosen τ , as depicted in Figure 2. The estimation error can be signiﬁcant when the dataset is small, and EVL needs a smaller τ to be more conservative and closer to behavior cloning. When the dataset is large, the estimation error becomes small, and we  can use a larger τ to recover the optimal policy. However, the expectile operator in Equation 2 does not have a closed-form solution. In practice, we consider the one-step gradient expectile operator

We use the following toy example to further illustrate the trade-offs achieved by EVL. Consider a random generated MDP. When the operator can be applied exactly, the Bellman optimality operator is sufﬁcient to learn the optimal value V ∗ . However, applying operators with an ofﬂine dataset raises a noise on the actual operator due to the estimation error with ﬁnite and biased data. We simulate this effect by adding random Gaussian noise to the operator. Applying the optimality operator on ofﬂine datasets can lead to severe overestimation due to the maximization bias and bootstrapping. The value estimation learned by EVL, on the contrary, achieves a trade-off between learning optimal policy and behavior cloning and can be close to the optimal value with proper chosen τ , as depicted in Figure 2. The noise upon the operator largely depends on the size of the dataset. Estimation error can be signiﬁcant with insufﬁcent data. In this case, we need a small τ to be conservative and be close to behavior cloning. When the dataset is large and we are able to have an accurate estimation for the operator, we can use a larger τ to recover the optimal policy. By adjusting τ , the expectile operator can accommodate variant types of datasets. However, the expectile operator in Equation 4 does not have a closed-form solution. In practice, we consider the one-step gradient expectile operator


Xie et al., 2020; Sohn et al., 2020) (also known as input consistency regularization) or ﬁt the unlabeled data on its predictions generated by a previously learned model (Lee, 2013; Chen et al., 2020b). It is interesting that UDA (Xie et al., 2020) reveals the crucial role of noise produced by advanced data augmentation methods. FixMatch (Sohn et al., 2020) usestwo versions of augmentation (weak augmentation and strong augmentation) and argues that predictions from weakly-augmented imagescan be used to supervise the output of strongly augmented data. SimCLRv2 (Chen et al., 2020b) ﬁrst ﬁne-tunes the pre-trained model from the labeled data and then distills on the unlabeled data. Self-Tuning (Wang et al., 2021) further improves data efﬁciency by a pseudo group contrast (PGC) mechanism but  limited on classiﬁcation setup. Moreover, various recent methods (van denOord et al., 2018; He et al., 2020; Wu et al., 2018; Hadsell et al., 2006; Tian et al., 2019; Chen et al., 2020a) improve data efﬁciency by self-supervised learning. However, most existing data-efﬁcient methods focus on classiﬁcation setup while rare attention has been paid to deep regression .

Xie et al., 2020; Sohn et al., 2020) (a.k.a. input consistency regularization) or ﬁt the unlabeled data on its predictions generated by a previously learned model (Lee, 2013; Chen et al., 2020b). Further, Co-Training (Blum & Mitchell, 1998b), Deep Co-Training Qiao et al. and Tri-Training (Zhou & Li, 2005a) improve data efﬁciency from an interesting perspective of different views of classiﬁers. MixMatch (Berthelot et al., 2019), ReMixMatch (Berthelot et al., 2020) and UDA (Xie et al., 2020) reveal the crucial role of noise produced by advanced data augmentation methods. FixMatch (Sohn et al., 2020) uses predictions from weakly-augmented images to supervise the output of strongly augmented data. Meta Pseudo Labels (Pham et al., 2021) further improves data efﬁciency by making the teacher constantly adapted by the feedback of the student’s performance on the labeled dataset. SimCLRv2 (Chen et al., 2020b) ﬁrst ﬁne-tunes the pre-trained model from the labeled data and then distills on the unlabeled data. Self-Tuning (Wang et al., 2021) introduces a pseudo group contrast (PGC) mechanism but is limited on classiﬁcation setup. Besides of involving unlabeled data from the same distribution, another promising direction for improving data efﬁciency is introducing a complementary perspective to further improve data efﬁciency by introducing a related but different domain (Long et al., 2015; Ganin & Lempitsky, 2015; Long et al., 2017; Saito et al., 2018b; Lee et al., 2019; Zhang et al., 2019; Saito et al., 2018a; 2019). Moreover, various recent methods (van den Oord et al., 2018; He et al., 2020; Wu et al., 2018; Hadsell et al., 2006; Tian et al., 2019; Chen et al., 2020a) improve data efﬁciency by self-supervised learning. However, most existing data-efﬁcient methods focus on classiﬁcation setup while rare attention has been paid to deep regression .


CNN-based Architectures. Since AlexNet (Krizhevsky et al., 2012) won the ImageNet competition in 2012, the CNN-based architectures have gradually been utilized to automatically extract image features instead of hand-crafted features. Subsequently, the VGG network (Simonyan & Zisserman, 2015) is proposed, which purely uses a series of 3 × 3 convolution and fully connected layers, and obtains outstanding performance in image classiﬁcation. Furthermore, ResNet (He et al., 2016) is proposed, which utilizes the residual connection to transfer features in different layers, thereby alleviating the problem of gradient vanishing and obtaining superior performance. After that, the residual module becomes an important component of the network design and is also employed in subsequent transformer-based architectures and MLP-based architectures. Some papers have made further improvements to the convolution operation in CNN-based architecture, such as dilated convolution (Yu & Koltun, 2016) and deformable convolution (Dai et al., 2017). EfﬁcientNet (Tan & Le, 2019; 2021) introduces neural architecture search into CNN to search for a suitable  structure.

CNN-based Architectures. Since AlexNet (Krizhevsky et al., 2012) won the ImageNet competition in 2012, the CNN-based architectures have gradually been utilized to automatically extract image features instead of hand-crafted features. Subsequently, the VGG network (Simonyan & Zisserman, 2015) is proposed, which purely uses a series of 3 × 3 convolution and fully connected layers. ResNet (He et al., 2016) utilizes the residual connection to transfer features in different layers, which alleviates the gradient vanishing and obtains superior performance. Some papers make further improvements to the convolution operation in CNN-based architecture, such as dilated convolution (Yu & Koltun, 2016) and deformable convolution (Dai et al., 2017). EfﬁcientNet (Tan & Le, 2019; 2021) introduces neural architecture search into CNN to search for a suitable network structure.

Instruction: Make this paragraph more concise.

Instruction: Remove the sentence about the residual module. Make the paragraph more concise.


In this section, we evaluate three different aspects of our contributions: the usefulness of exclusive multiset-equivariance (Subsection 4.1), the differences between  our  implicit and automatic differentiation (Subsection 4.2), and the applicability of iDSPN to a larger-scale dataset (Subsection 4.3). We provide detailed descriptions of the experimental procedure in Appendix D, show example inputs and outputs in Appendix E, and open-source the code to reproduce all experiments at https://github.com/<redacted>/<redacted> .

In this section, we evaluate three different aspects of our contributions: the usefulness of exclusive multiset-equivariance (Section 4.1), the differences between automatic and our approximate implicit differentiation (Section 4.2), and the applicability of iDSPN to a larger-scale dataset (Section 4.3). We provide detailed descriptions of the experimental procedure in Appendix D, show example inputs and outputs in Appendix E, and open-source the code to reproduce all experiments at https: //github.com/<redacted>/<redacted> and in the supplementary material.

Instruction: Be clear about references.

Instruction: Lightly clarify the text. Add a reference to appendix at the end.


Our main example willinstead be online job scheduling via minimizing the fractional makespan, following Lattanzi et al. They consider the problem of assigning each in a sequence of variablesized jobs to one of m machines [30, Section 3]. The authors provide an algorithm that uses predictions ˆw ∈ R m> 0 of “good" machine weights w ∈ R m> 0 to assign jobs based on how well ˆw corresponds to machine demand; the algorithm has a runtime guarantee of O (cid:0) log min { max i ˆw [ i ] / w [ i ] , m } (cid:1) . They also discus learning linear and more complicated predictors, but without guarantees. In this section we provide guarantees for the linear prediction setting in which we target the logarithm of the machine weights, which makes the problem convex. Note we assume features lie in the f -dimensional simplex, and for simplicity we only consider learning the linear transform from features to predictors and not the intercept, as the latter is subsumed by the former. For the online result, we use the parameter-free algorithm of Orabona and Pal [38], an OGD-type method that allows us to not assume any bound on the machine weights and thus  compete with the optimal linear predictor in all of R m × f .

Our main example will be online job scheduling via minimizing the fractional makespan [30], where we must assign each in a sequence of variable-sized jobs to one of m machines. Lattanzi et al. [30] provide an algorithm that uses predictions ˆw ∈ R m> 0 of “good” machine weights w ∈ R m> 0 to assign jobs based on how well ˆw corresponds to machine demand; the method has a performance guarantee of O (log min { max i ˆw [ i ] w [ i ] , m } ) . They also discuss learning linear and other predictors, but without guarantees. We study linear prediction of the logarithm of the machine weights, which makes the problem convex, and assume features lie in the f -dimensional simplex. For simplicity we only consider learning the linear transform from features to predictors and not the intercept, as the former subsumes the latter. For the online result, we use KT-OCO [38, Algorithm 1], a parameter-free subgradient method with update x t +1 ← 1+ (cid:80) ts =1 (cid:104) g s , x s (cid:105) t + (cid:80) ts =1 g s for g s = ∇ U s ( x s ) ; it allows us to not assume any bound on the machine weights and thus to compete with the optimal linear predictor in all of R m × f .


Generally, those statistical notions can be expressed in terms of different (conditional) independence statements between the involved random variables (Barocas et al., 2019): ¯ y ⊥ s (eq. 5), ¯ y ⊥ s | y (eq. 6–7), and y ⊥ s | ¯ y (eq. 8–9). If our training set has no positive outcome for the demographic s = 0 , i.e. M y =1 ,s =0 = ∅ , the true positive rate for this group will suffer, and therefore we will likely not be able to satisfy, among others, equality of true positive rate (eq. 6).

Generally, those statistical notions can be expressed in terms of different (conditional) independence statements between the involved random variables (Barocas et al., 2019): ¯ y ⊥ s (equation 5), ¯ y ⊥ s | y (equation 6 – equation 7), and y ⊥ s | ¯ y (equation 8 – equation 9). If our training set has no positive outcome for the demographic s = 0 , i.e. M y =1 ,s =0 = ∅ , the true positive rate for this group will suffer, and therefore we will likely not be able to satisfy, among others, equality of true positive rate.

Instruction: Prefer extended forms over abbreviations of words.

Instruction: Write the abbreviation in their full form.


We highlight an adaptation of BN named as static Batch Normaliztion (sBN) for optimizing privacy constrained heterogeneous models. During the training phase, sBN does not track running estimates and simply normalize batch data. We do not track the local running statistics as the size of local models may also vary dynamically. This method is suitable for HeteroFL as every communication round is independent. After the training process ﬁnishes, the server sequentially query local clients and cumulatively update global BN statistics. Thus, this method greatly reduces the risk of leaking private data because the calculation of BN statistics  and  the optimization of parameters are isolated. We also empirically found this trick significantly outperforms other forms of normalization methods including the InstanceNorm (Ulyanov et al., 2016), GroupNorm (Wu & He, 2018) , and LayerNorm (Ba et al., 2016) as shown in Table 4 and Table 5.

We highlight an adaptation of BN named as static Batch Normaliztion (sBN) for optimizing privacy constrained heterogeneous models. During the training phase, sBN does not track running estimates and simply normalize batch data. We do not track the local running statistics as the size of local models may also vary dynamically. This method is suitable for HeteroFL as every communication round is independent. After the training process ﬁnishes, the server sequentially query local clients and cumulatively update global BN statistics. There exist privacy concerns about calculating global statistics cumulatively and we hope to address those issues in the future work. We also empirically found this trick significantly outperforms other forms of normalization methods including the InstanceNorm (Ulyanov et al., 2016), GroupNorm (Wu & He, 2018) , and LayerNorm (Ba et al., 2016) as shown in Table 4 and Table 5.


Importantly, it is equipped with a simple normalization of the layer’s activations, and an optional temperature-scaling mechanism (Hinton et al., 2015), producing a soft WTA instead of selecting a single "hard" winner neuron. This allows us to prove formally that a SoftHebb layer is a generative mixture model that objectively minimizes its Kullback-Leibler (KL) divergence from the input distribution through Bayesian inference, thus providing a new formal ML-theoretic perspective of these networks. We complement our main results, which are theoretical, with experiments that are small-scale but produce intriguing results. As a generative model, SoftHebb has a broader scope than classiﬁcation, but we test it in simulations on the tasks of recognizing MNIST handwritten digits and Fashion-MNIST fashion products. First, we conﬁrm that SoftHebb is more accurate than a hard-WTA model. Second, we validate that it minimizes a loss function (cross-entropy) even though it has no access to it or to labels during learning. In addition, likely owing to its Bayesian and generative properties, the unsupervised WTA model  outperforms a supervised two-layer perceptron in several aspects: learning speed and accuracy in the ﬁrst presentation of the training dataset, robustness to noisy data, and increased robustness to one of the strongest white-box adversarial attacks, i.e. projected gradient descent (PGD) (Madry et al., 2017), and without any explicit defence. Interestingly, the SoftHebb model also exhibits inherent properties of deﬂection (Qin et al., 2020) of the adversarial attacks, and generates object interpolations.

Importantly, it is equipped with a simple normalization of the layer’s activations, and an optional temperature-scaling mechanism (Hinton et al., 2015), producing a soft WTA instead of selecting a single "hard" winner neuron. This allows us to prove formally that a SoftHebb layer is a generative mixture model that objectively minimizes its Kullback-Leibler (KL) divergence from the input distribution through Bayesian inference, thus providing a new formal ML-theoretic perspective of these networks. We complement our main results, which are theoretical, with experiments that are small-scale but produce intriguing results. As a generative model, SoftHebb has a broader scope than classiﬁcation, but we test it on image classiﬁcation tasks. Surprisingly, in addition to overcoming several inefﬁciencies of backpropagation, the unsupervised WTA model also outperforms a supervised two-layer perceptron in several aspects: learning speed and accuracy in the ﬁrst presentation of the training dataset, robustness to noisy data and to one of the strongest white-box adversarial attacks, i.e. projected gradient descent (PGD) (Madry et al., 2017), and without any explicit defence. Interestingly, the SoftHebb model also exhibits inherent properties of deﬂection (Qin et al., 2020) of the adversarial attacks, and generates object interpolations.

Instruction: Make this paragraph shorter by removing details.

Instruction: Summarize the middle of the paragraph to make it shorter and more concise. Remove unnecessary details.


Early works understand the InfoNCE loss based on maximizing the mutual information (MI) between positive samples (Oord et al., 2018; Bachman et al., 2019; Hjelm et al., 2018; Tian et al., 2019; 2020). However, a rigorous relationship between mutual information and the downstream classification error has not been established. Tschannen et al. (2019) also find that optimizing tighter bounds of MI does not imply better representations. Thus, MI may not fully explain the success of InfoNCE. Besides, Arora et al. (2019) directly analyze the generalization of InfoNCE loss based on the assumption that positive samples are drawn from the same latent classes, which is different from practical contrastive algorithms. Ash et al. (2021) study the role of negative samples in contrastive SSL, and show an interesting collision-coverage trade-off theoretically. Furthermore, HaoChen et al. (2021) study contrastive SSL from a matrix decomposition perspective, but it is only applicable to their spectral contrastive loss. The behavior of InfoNCE is also studied from the perspective of alignment and uniformity (Wang & Isola, 2020), sparse coding model (Wen & Li, 2021), and the “expansion” assumption (Wei et al., 2020).

Early works understand the InfoNCE loss based on maximizing the mutual information (MI) between positive samples (Oord et al., 2018; Bachman et al., 2019; Hjelm et al., 2018; Tian et al., 2019; 2020; Tschannen et al., 2019). However, a rigorous relationship between mutual information and downstream performance has not been established. Besides, Arora et al. (2019) directly analyze the generalization of InfoNCE loss based on the assumption that positive samples are drawn from the same latent classes, which is different from practical algorithms. Ash et al. (2021) study the role of negative samples and show an interesting collision-coverage trade-off theoretically. HaoChen et al. (2021) study contrastive SSL from a matrix decomposition perspective, but it is only applicable to their spectral contrastive loss. The behavior of InfoNCE is also studied from the perspective of alignment and uniformity (Wang & Isola, 2020), sparse coding model (Wen & Li, 2021), the expansion assumption (Wei et al., 2020), stochastic neighbor embedding (Hu et al., 2022), and augmentation robustness (Zhao et al., 2023).


Set prediction modelsmake use of set-to-set functions that are permutation-equivariant (Lee et al., 2019; Locatello et al., 2020; Carion et al., 2020; Kosiorek et al., 2020). This is desirable when processing sets because it prevents a function from relying on the arbitrary order of the set in its matrix representation. Permutation-equivariant functions can be easily composed to build larger models that remain equivariant, which ﬁts well into deep learning architectures as building blocks.

Recent set prediction models (Lee et al., 2019; Locatello et al., 2020; Carion et al., 2020; Kosiorek et al., 2020) make use of set-to-set (permutation-equivariant list-to-list) functions to reﬁne an initial set Y 0 , which is usually a randomly generated or learnable matrix. Permutation-equivariance is desirable when processing sets because it prevents a function from relying on the arbitrary order of the set in its matrix representation. Such functions can be easily composed to build larger models that remain equivariant, which ﬁts well into deep learning architectures as building blocks.


Hollinworth et al. found that senior citizens generally lose the cursor due to poor eyesight and sustained concentration [15]. Therefore, the implemented a Field Mouse (a mouse with a touch sensor attached) and proposed a technique wherein the cursor moves to the center of the screen when the user hold the mouse. This technique reduced the time required to search for the cursor, which in turn reduced the movement time. Stephane et al. focused on screen torus settings [16]. With this setting, for example, when the cursor reaches the right edge, it appears from the left edge. As a result, users can easily lose sight of the cursor when it warps. Therefore, they proposed a TorusDesktop technique that adds appropriate visual feedback between the time the cursor warping, and making it difficult to lose sight of the cursor.

Hollinworth et al. found that senior citizens lose the cursor be- cause of poor eyesight and sustained concentration, and therefore, they implemented a Field Mouse (a mouse with a touch sensor at- tached) and proposed a technique wherein the cursor moves to the center of the screen when the user holds the mouse [15]. This tech- nique help reduce the time required to search for the cursor, which in turn reduces the movement time. Stephane et al. focused on screen torus settings [16]. With this setting, when the cursor reaches the screen edge, it appears from the opposite end. For example, when the cursor reaches the right edge, it appears from the left edge. However, users can easily lose sight of the cursor when it warps around the edges. To overcome this issue, they proposed a TorusDesktop technique that adds appropriate visual feedback between the time the cursor warps. These studies focused on the user losing sight of the cursor, but these did not focus on a scenario where the cursor is hidden.


To warm-up the model, we perform  regular steps in the ﬁrst  epoch. We switch from regular steps to re-balancing steps if | d speed ( t ) | > α , where α is  the imbalance  parameter . The training takes Q re-balancing steps before returning to regular mode. We refer to  Q as the re-balancing window size .

To warm-up the model, we perform only regular steps in the ﬁrst training epoch. Then we switch from regular steps to re-balancing steps if | d speed ( t ) | > α , where α is a hyperparameter, referred to as the imbalance tolerance parameter . The training takes Q re-balancing steps before returning to regular mode. We refer to the hyperparameter Q as the re-balancing window size .

Instruction: Change the descriptions so that the hyperparameters can be easily referred to later


Unlike other numerical methods such as ﬁnite differences and ﬁnite elements, the derivatives of the desired function can be analytically calculated to any order. This framework therefore, enables the solution of high order non-linear PDEs. The proposed algorithm is a uniﬁed formulation of both forward and inverse problems where the optimized loss function consists of few elements: ﬁdelity terms of Land L ∞ norms, boundary conditions constraints and additional regularizers. This setting is ﬂexible in the sense that regularizers can be tailored to speciﬁc problems. We demonstrate our method on several free shape 2D second order systems with application to Electrical Impedance Tomography (EIT), diffusion and wave equations.

Unlike other numerical methods such as ﬁnite differences and ﬁnite elements, the derivatives of the desired function can be analytically calculated to any order. This framework therefore, enables the solution of high order non-linear PDEs. The proposed algorithm is a uniﬁed formulation of both forward and inverse problems where the optimized loss function consists of few elements: ﬁdelity terms of Land L ∞ norms that unlike previous methods promote a strong solution. Robust boundary conditions constraints and additional regularizers are included as well. This setting is ﬂexible in the sense that regularizers can be tailored to speciﬁc problems. We demonstrate our method on several free shape 2D second order systems with application to Electrical Impedance Tomography (EIT), diffusion and wave equations.


The design of the calendar should avoid design elements that in- troduce clutter to the calendar ( DG3 ) . Design elements such as colors, sliders, labels, and markers should be carefully employed to avoid overwhelming the calendar. One of the reasons why Design B was preferred is because it is less cluttered: medication entries can be rendered effectively using position, shape, and size. The size of a medication entry should be as small as possible so as not to occupy too much space. Size should not be used to indicate either allowed or preferred administration time of a medication entry, and the size of the entry should also be uniform regardless of the length of the allowed period of administration. Using shapes with a colored outline and transparent fill was associated with less noise and hence preferred by the users. While the slider design was effective in com- municating both the allowed and preferred administration period, it made the entry occupy a lot of calendar space and was also misread by some participants as an indicator of delayed release for certain medications. Sliders should thus be avoided. Familiar icons such as tablets can be used to indicate medication entries.

The design of the calendar should avoid design elements that in- troduce clutter to the calendar ( DG3 ) . One of the reasons why Design B was preferred is because it is less cluttered: medication entries can be rendered effectively using position, shape, and size. The size of a medication entry should be as small as possible so as not to occupy too much space. Size should not be used to indicate either allowed or preferred administration time of a medication entry, and the size of the entry should also be uniform regardless of the length of the allowed period of administration. Using shapes with a colored outline and transparent fill was associated with less noise by participants. While the slider design was effective in communicating both the allowed and preferred administration period, it made the entry occupy a lot of calendar space and was also misread by some participants. Familiar icons such as tablets can be used to indicate medication entries.

Instruction: Remove unnecessary details and explanations.


S t +1 = f st ( S t , A t , U st ) . This is always possible using auto-regressive uniformization. The DAG G of the resulting SCM is shown in ﬁg. 1. This procedure is closely related to the ‘reparameterization trick’ for models with lotion-scale distributions (Kingma & Welling, 2013; Rezende et al., 2014).

S t +1 = f st ( S t , A t , U st ) . This is always possible using auto-regressive uniformization, see Lemma 2 in the appendix. The DAG G of the resulting SCM is shown in ﬁg. 1. This procedure is closely related to the ‘reparameterization trick’ for models with location-scale distributions (Kingma & Welling, 2013; Rezende et al., 2014).


The prior rotamers ˜ χ j are  inaccurate or unknown in many cases. For example, if we mutate some amino acids in the protein complex, the rotamers of the mutated amino acids are unknown, and the rotamers of amino acids nearby the mutated ones are inaccurate because they are affected by the mutation. The probability density is defined over the d -dimensional torus T D = ( S 1 ) D , and we show below our proposed flow-based architecture to model the density.

The prior rotamers ˜ χ j are often inaccurate or unknown. For example, if we mutate some residues, the rotamers of the mutated residues are unknown, and the rotamers of residues nearby the mutated ones are inaccurate because they are affected by the mutation. The probability density is defined over the d -dimensional torus T D = ( S 1 ) D , and we describe below the flow-based architecture to model the density.

Instruction: Replace every apparition of "amino acids" or "amino acids in the protein complex" by "residues"

Instruction: Replace occurrences of amino acids by residues. Make this paragraph a lit bit more concise.


Margin-Density (Nguyen & Smeulders, 2004). Scores candidates by the product of their margin and their density estimates, so as to increase diversity. The density is computed by ﬁrst clustering the penultimate layer activations of  all |Z| candidate points via K -means. Then, the density score of candidate x i is computed as: | C ( x i ) | / |Z| , where C ( x i ) is the cluster containing x i . We useclusters.

Margin-Density (Nguyen & Smeulders, 2004). Scores candidates by the product of their margin and their density estimates, so as to increase diversity. The density is computed by ﬁrst clustering the penultimate layer activations of the current model on all |Z| candidate points via K -means. Then, the density score of candidate x i is computed as: | C ( x i ) | / |Z| , where C ( x i ) is the cluster containing x i . We use min { 20 , |Z|} clusters.


Parsing signal from noise is critical to learning from weak and rule-based supervision. Accordingly, we compare ReGAL’s ability to that of our baselines in accurately classifying instances based on a set of seed rules, which are shown in Table 1. For each dataset, we provide exactly one seed LF foreach class. Each seed LF contains exactly six single-token keywords. If any of these keywords is found in document d i , the LF assigns its label; otherwise, it abstains from labeling d i .

Parsing signal from noise is critical to learning from weak and rule-based supervision. Accordingly, we compare ReGAL’s ability to that of our baselines in accurately classifying instances based on a set of seed rules, which are shown in Table 1. We provided exactly each class with exactly one labeling function consisting of six keywords or phrases adapted from [16]. If any of these keywords is found in document d i , the LF assigns its label; otherwise, it abstains from labeling d i .


In this paper, we introduce ZeroC, a new framework for zero-shot concept recognition and acquisition at inference time. Our experiments show that in a challenging grid-world domain, ZeroC is able to recognize complex, hierarchical concepts composed of English characters in a grid-world in a zero-shot manner, being given a high-level, symbolic speciﬁcation of their structures, and after being trained with simpler concepts. In addition, we demonstrate that an independently trained ZeroC is able to transfer hierarchical concepts across different domains at inference. Although this work is evaluated only in grid-world  domain, we are the ﬁrst to address this difﬁcult challenge, and  hope that this work will make a useful step in the development of composable neural systems, capable of zero-shot concept recognition and acquisition and hence suitable for more diverse tasks.

In this paper, we introduce ZeroC, a new framework for zero-shot concept recognition and acquisition at inference time. Our experiments show that in a challenging grid-world domain, ZeroC is able to recognize complex, hierarchical concepts composed of English characters in a grid-world in a zero-shot manner, being given a high-level, symbolic speciﬁcation of their structures, and after being trained with simpler concepts. In addition, we demonstrate that an independently trained ZeroC is able to transfer hierarchical concepts across different domains at inference. Although this work is evaluated only in grid-world visual domain, we are the ﬁrst to address this difﬁcult challenge. We are also excited to see its potential application in broader domains, e.g. in AI for scientiﬁc discovery, where it may infer novel patterns and concepts from data in a zero-shot manner. We hope that this work will make a useful step in the development of composable neural systems, capable of zero-shot concept recognition and acquisition and hence suitable for more diverse tasks.


Attention on all AP metrics. It is better at the attribute classiﬁcation  ignoring the 3d coordinates (96. → 98.8) and improves especially for the  stricter AP thresholds like AP 0 . 125 (7.9 → 76.9). Note that a stricter AP threshold is always upper bounded by the looser AP threshold, so iDSPN is guaranteed to be better than Slot Attention † on AP 0 . We observed some overﬁtting when using 128x image inputs, which did not show up in preliminary experiments when training an autoencoder with the ground-truth set as input. We reduced this overﬁtting signiﬁcantly by increasing the image size to 256x256 while keeping the latent vector size the same, which results in further performance improvements. We thus believe that the overﬁtting is due to the ResNet18 image encoder rather than iDSPN.

Attention on all AP metrics. It is better at attribute classiﬁcation when ignoring the 3d coordinates (AP ∞ , 96.4% → 98.8%) and improves especially for the metrics with stricter 3d coordinate thresholds (AP 0 . 125 , 7.9% → 76.9%). This is despite Slot Attention † using a three times higher weight on the loss for the coordinates than iDSPN. Note that a stricter AP threshold is always upper bounded by a looser AP threshold, so iDSPN is guaranteed to be better than Slot Attention † on AP 0 . We observe some overﬁtting with 128x128 image inputs (1.6e-4 train loss, 5.4e-4 validation loss), which did not appear in preliminary experiments when training an autoencoder with the ground-truth set as input. We reduce this generalization gap by increasing the image size to 256x256 while keeping the latent vector size the same, which results in further performance improvements (1.1e-4 train loss, 2.5e-4 validation loss). We thus believe that the overﬁtting is due to the ResNet18 image encoder rather than iDSPN.


Less Constraints: During encoding, the explicit word order information used inRNN will help the vector representation capture more of the temporally-speciﬁc relationships among words, but this same constraint (if using RNN  as the decoder) could be an inappropriate constraint in  the decoding process.

The results are presented in the Table 1. Generally, the three different decoding settings didn’t make much of a difference in terms of the performance on selected downstream tasks, with RNN or CNN as the decoder. The results tell us that, in terms of learning good sentence representations, the autoregressive decoder doesn’t require the correct ground-truth words as the inputs.

Instruction: Can you reformulate my entire paragraph?


Unfortunately, the distortion in w G ( S, T ) can be very large depending on the quantities on the right, and the cumulative error in cost G ( T ) blows up with the depth of the tree which is even worse. Hereis the second observation: the negative term w G ( S ∪ T, S ∪ T ) that internal node S contributes to thecost also appears as a positive term in its parent’s contribution to the cost. We can pass this term as a since there always exists an optimal hierarchy that is binary.

Unfortunately, the distortion in w G ( S, T ) can be very large depending on the quantities on the right, and the cumulative error in cost G ( T ) blows up with the depth of the tree which is even worse. Here optimal hierarchy that is binary.


Results. Fig. 4 shows that certain alterations—such as completely removing articles from the evaluated text—have almost no impact on  the divergence between our reference and test corpora for various ∆ . In fact, text without any articles is judged as better than GPT-2 XL ’s by most of the cluster-based divergences. Further, while this perturbation undoubtedly affects the text’s ﬂuency, it has less of an effect on this divergence than, e.g., truncating texts. This is arguably undesirable: A metric of text quality should place more emphasis on ﬂuency than surface statistics, such as length.

Results. Fig. 4 shows that certain alterations to the evaluated text—such as completely removing articles—have almost no impact on its divergences from the reference corpora for various ∆ . In fact, text without any articles is judged as better than GPT-2 XL ’s by all of the cluster-based divergences (see Fig. 9 for a zoomed in version). Further, while this perturbation undoubtedly affects the text’s ﬂuency, it has less of an effect on ∆ than, e.g., truncating texts. This is arguably undesirable: A metric of text quality should place more emphasis on ﬂuency than surface statistics, such as length.

Instruction: Make the concepts a bit more specific, such that some vague ideas are more clear.

Instruction: Revise the writing for better readability.


We follow several rules when selecting victim nodes. First, the attack must be successful on the victim node to fool the model. Next, we try our best to ﬁnd successful attacks on victim nodes with different node degree to evaluate diverse victim nodes’ properties. Finally, we choose victim nodes among those with the same degree uniformly at random to perform the detection. We observe that without considering detection, the Multi-edges direct attack is the most successful attacking model, followed by Single-edge attack and ﬁnally Multi-edges indirect attack. Therefore, we selected 20, 10, 6 victim nodes respectively for these three attack methods. The selected victim node degrees are shown in the appendix.

We follow several rules when selecting victim nodes. First, the attack must be successful on the victim node to fool the model. Next, we try our best to ﬁnd successful attacks on victim nodes with different node degree to evaluate diverse victim nodes’ properties. Finally, we choose victim nodes among those with the same degree uniformly at random to perform the detection. We observe that without considering detection, the Multi-edges direct attack is the most successful attacking model, followed by Single-edge attack and ﬁnally Multi-edges indirect attack. Therefore, we selected 20, 10, 6 victim nodes respectively for these three attack methods on real-world data. For synthetic data, we simply pick two victim nodes, one with the smallest degree and the other with the largest degree. The selected victim node degrees are shown in the appendix.


In this paper, we prove new generalization bounds for convolutional networks that take account of this effect. As in earlier analyses for the fully connected case, our bounds are in terms of the distance from the initial weights, and the number of parameters. Additionally, our bounds are “size-free”, in the sense that they are independent of the number of pixels in the input, or the height and width of the hidden feature maps.

In this paper, we prove new generalization bounds for convolutional networks that take account of this effect. As in earlier analyses for the fully connected case, our bounds are in terms of the distance from the initial weights, and the number of parameters. Additionally, our bounds independent of the number of pixels in the input, or the height and width of the hidden feature maps.

Instruction: Make the ideas more concise.

Instruction: Remove unnecessary details.


We compared RECOO with Algorithm 1 in [30], where learning rates are summarized in Table 4in Appendix H. Figure 2 includes the cumulative losses and violations. In particular, The (mean,variance) pair of RECOO for loss and violation are p 42510 . 05 q and p 713 . 45 , 1 . 60 q at the end of learning horizon, respectively while that of Algorithm 1 in [30] are p 43011 . 17 q and p 1684 . 17 , 2 . 85 q . It is justiﬁed RECOO performs better, especially in terms of the cumulative violation,which further shows the “rectiﬁed” design reduces the cumulative hard constraint violation.

We compared RECOO with Algorithm 1 in [27], with α t “ η t “ ? t , γ t “ t 1 { 2 ` 0 . 01 in RECOO; andα t “ 0 . 8 {? t, β t “ 5 {? t and γ t “ 0 . 5 {? t for Algorithm 1 in [27] (these are the optimized learningrates). Figure 2 includes the cumulative losses and violations. In particular, The (mean, variance)pair of RECOO for loss and violation are p 42510 . 05 q and p 713 . 45 , 1 . 60 q at the end of learning horizon, respectively while that of Algorithm 1 in [27] are p 43011 . 17 q and p 1684 . 17 , 2 . 85 q . It is justiﬁed RECOO performs better, especially in terms of the cumulative violation, which furthershows the “rectiﬁed” design reduces the cumulative hard constraint violation.


Intelligent agents can solve tasks in a variety of ways depending on the action set at their disposal. For instance, while using a toolkit for repair, the choice of tool (the action) closely depends on what other tools are available. Yet, such dependence on other available actions is ignored in conventional reinforcement learning (RL)since it assumes a ﬁxed action set. In this work, we posit that learning the interdependence between actions is crucial for RL agents acting under a varying action set. To this end, we propose a novel policy architecture that consists ofan input graph composed of available actions and a graph attention network to learn the action interdependence. We demonstrate that our architecture makes  action decisions by correctly attending to the relevant actions in both value-based and policy-based RL. Consequently, it consistently outperforms non-relational architectures on applications where the action space can vary, such as recommender systems and physical reasoning with tools and skills.

Intelligent agents can solve tasks in various ways depending on their available set of actions. However, conventional reinforcement learning (RL) assumes a ﬁxed action set. This work asserts that tasks with varying action sets require reasoning of the relations between the available actions. For instance, taking a nail-action in a repair task is meaningful only if a hammer-action is also available. To learn and utilize such action relations, we propose a novel policy architecture consisting of a graph attention network over the available actions. We show that our model makes informed action decisions by correctly attending to other related actions in both value-based and policy-based RL. Consequently, it outperforms non-relational architectures on applications where the action space often varies, such as recommender systems and physical reasoning with tools and skills.

Instruction: Make reasoning understandable, use accurate words.


Ermon, 2019). Song et al. (2020b) shows that diffusion models are trained using denoising score matching (Vincent, 2011), a conditional objective that provides unbiased gradients with respect to the score matching objective. Conditional Flow Matching draws inspiration from this result, but generalizes to matching vector fields directly. Due to the ease of scalability, diffusion models have received increased attention, producing a variety of improvements such as loss-rescaling (Song et al., 2021), adding classifier guidance along with architectural improvements (Dhariwal & Nichol, 2021), and learning the noise schedule (Nichol & Dhariwal, 2021; Kingma et al., 2021). However, (Nichol & Dhariwal, 2021) and (Kingma et al., 2021) only consider a restricted setting of Gaussian conditional paths defined by simple diffusion processes with a single parameter—in particular, it does not include our conditional OT path. While existing works make use of a connection between diffusion processes and continuous normalizing flows with the same probability path (Maoutsa et al., 2020b; Song et al., 2020b; 2021), our work allows us to generalize beyond the class of probability paths modeled by simple diffusion. With our work, it is possible to completely sidestep the diffusion process construction and reason directly with probability paths, while still retaining efficient training and log-likelihood evaluations.

Ermon, 2019). Song et al. (2020b) shows that diffusion models are trained using denoising score matching (Vincent, 2011), a conditional objective that provides unbiased gradients with respect to the score matching objective. Conditional Flow Matching draws inspiration from this result, but generalizes to matching vector fields directly. Due to the ease of scalability, diffusion models have received increased attention, producing a variety of improvements such as loss-rescaling (Song et al., 2021), adding classifier guidance along with architectural improvements (Dhariwal & Nichol, 2021), and learning the noise schedule (Nichol & Dhariwal, 2021; Kingma et al., 2021). However, (Nichol & Dhariwal, 2021) and (Kingma et al., 2021) only consider a restricted setting of Gaussian conditional paths defined by simple diffusion processes with a single parameter—in particular, it does not include our conditional OT path. In an another line of works, (De Bortoli et al., 2021; Wang et al., 2021; Peluchetti, 2021) proposed finite time diffusion constructions via diffusion bridges theory resolving the approximation error incurred by infinite time denoising constructions. While existing works make use of a connection between diffusion processes and continuous normalizing flows with the same probability path (Maoutsa et al., 2020b; Song et al., 2020b; 2021), our work allows us to generalize beyond the class of probability paths modeled by simple diffusion. With our work, it is possible to completely sidestep the diffusion process construction and reason directly with probability paths, while still retaining efficient training and log-likelihood evaluations.


BALD (Houlsby et al., 2011) estimates the mutual information (MI) between the datapoints and the model weights, the idea being that points with large MI between the predicted label and weights have a larger impact on the trained model’s performance. The measure, denoted I , is the conditional entropy over predictions given weights less the earlier entropy term.

BALD (Houlsby et al., 2011) estimates the mutual information (MI) between the datapoints and the model weights, the idea being that points with large MI between the predicted label and weights have a larger impact on the trained model’s performance. The measure, denoted I , is approximated as:


On the other hand, our metrics deem text with stopwords removed as utterly different from the reference. Permuting words within texts has a similar effect, demonstrating that, at least to some extent, the embedding space captures notions of syntax and grammaticality, rather than pure unigram These results inspire us to investigate which surface features of text are encoded in embedding clusters. Following our setup in §6.1, we look at whether clusters encode the percentage of stopwords or punctuation  in texts. We use solely the WebText dataset to train our clustering functions in this setting. We then compute the average percentage of stopwords or punctuation per cluster in half of our strings.

On the other hand, our metrics deem text with stopwords removed as utterly different from the reference. Permuting words within texts has a similar effect, demonstrating that, at least to some extent, the embedding space captures notions of syntax and grammaticality, rather than pure unigram statistics. The increase in ∆ shown when performing sentence-level permutations likewise suggests that the clusters delineate different levels of coherence to some extent. In Fig. 10 (in App. E), we perform an additional experiment where we again probe the clusters (as in §6.1), but for surface features of text this time, such as the percentage of stopwords and punctuation symbols in a text. There we see evidence that such features of text are not strongly encoded in the clustering scheme.


Optimization algorithms can provide insight and guidance in the design of deep network architectures (Vogel & Pock, 2017;  Yang et al., 2016; Zhang & Ghanem, 2018). For example, Yang et al. (2016) have proposed a deep network architecture for compressed sensing. Their network, dubbed ADMM-Net, is inspired by ADMM updates (Boyd et al., 2011) on the compressed sensing objective. Similarly, Zhang & Ghanem (2018) demonstrated that unrolling a proximal gradient descent solver (Beck & Teboulle, 2009) on the same problem can further improve performance.

Optimization algorithms can provide insight and guidance in the design of deep network architectures (Vogel & Pock, 2017; Kobler et al., 2017; Yang et al., 2016; Zhang & Ghanem, 2018). For example, Yang et al. (2016) have proposed a deep network architecture for compressed sensing. Their network, dubbed ADMM-Net, is inspired by ADMM updates (Boyd et al., 2011) on the compressed sensing objective. Similarly, Zhang & Ghanem (2018) demonstrated that unrolling a proximal gradient descent solver (Beck & Teboulle, 2009) on the same problem can further improve performance. Moreover, the work of Kobler et al. (2017) demonstrated a relation between incremental proximal methods and ResNet blocks.


Optimization Dilemma in OOD Algorithms. Along with the developments of OOD methods, the optimization dilemma in OOD generalization is gradually perceived in the literature, and raises new puzzles to the community. In fact, several recent works also notice the optimization dilemma in OOD algorithms, specifically, the trade-off between discovering the statistical correlations (i.e., ERM) and preventing the usage of spurious correlations (e.g., IRM). Empirically, Gulrajani & Lopez-Paz (2021) observe that, with careful hyperparameter tuning and evaluation setting, many OOD algorithms cannot outperform ERM in domain generalization, demonstrating the difficulties of properly mitigating the trade-offs between OOD and ERM objectives in practice. Moreover, Sagawa* et al. (2020); Zhai et al. (2022) find that, regularization on ERM, or sacrificing ERM performance, is usually needed for achieving satisfactory OOD performance, which aligns with our findings through Pareto front as shown in Fig. 6(a) and Fig. 7(a). Besides, Lin et al. (2022a) find that IRM can easily overfit and learns unexpected features when applying IRM on large neural networks. Zhou et al. (2022) propose to alleviate this problem by imposing sparsity constraints. Orthogonal to Lin et al. (2022a) ; Zhou et al. (2022) that focuses on the optimization consequences, we focus on the optimization process of OOD objectives. In addition, Zhang et al. (2022a) find that, the performance of OOD algorithms largely relies on choosing proper pretraining epochs which aligns with our findings in Fig. 1(d), hence propose to construct a ready-to-use features for stable OOD generalization performance. Orthogonal to Zhang et al. (2022a), we focus on developing better optimization scheme for OOD algorithms, including choosing the proper objectives and the achievability of the invariant predictors. Besides, Lv et al. (2021) propose ParetoDA to leverage MOO to resolve the gradient conflicts amon the objectives in Domain Adaption. ParetoDA uses the guidance of validation loss based on the data that has the identical distribution to test distribution, to trade-off the conflicts in domain adaption objectives.

Optimization Dilemma in OOD Algorithms. Along with the developments of OOD methods, the optimization dilemma in OOD generalization is gradually perceived in the literature, and raises new puzzles to the community. In fact, several recent works also notice the optimization dilemma in OOD algorithms, specifically, the trade-off between discovering the statistical correlations (i.e., ERM) and preventing the usage of spurious correlations (e.g., IRM). Empirically, Gulrajani & Lopez-Paz (2021) observe that, with careful hyperparameter tuning and evaluation setting, many OOD algorithms cannot outperform ERM in domain generalization, demonstrating the difficulties of properly mitigating the trade-offs between OOD and ERM objectives in practice. Moreover, Sagawa* et al. (2020); Zhai et al. (2022) find that, regularization on ERM, or sacrificing ERM performance, is usually needed for achieving satisfactory OOD performance. A similar phenomenon has also been observed by Zhao et al. Xie et al. ; Sadeghi et al. ; Sener & Koltun (2022); Teney et al. (2022), which aligns with our findings through Pareto front as shown in Fig. 6(a) and Fig. 7(a). Besides, Lin et al. (2022a) find that IRM can easily overfit and learns unexpected features when applying IRM on large neural networks. Zhou et al. (2022) propose to alleviate this problem by imposing sparsity constraints. Orthogonal to Lin et al. (2022a) ; Zhou et al. (2022) that focuses on the optimization consequences, we focus on the optimization process of OOD objectives. In addition, Zhang et al. (2022a) find that, the performance of OOD algorithms largely relies on choosing proper pretraining epochs which aligns with our findings in Fig. 1(d), hence propose to construct a ready-to-use features for stable OOD generalization performance. Orthogonal to Zhang et al. (2022a), we focus on developing better optimization scheme for OOD algorithms, including choosing the proper objectives and the achievability of the invariant predictors. Besides, Lv et al. (2021) propose ParetoDA to leverage MOO to resolve the gradient conflicts amon the objectives in Domain Adaption. ParetoDA uses the guidance of validation loss based on the data that has the identical distribution to test distribution, to trade-off the conflicts in domain adaption objectives.


We thank Huan Zhang for helpful discussions and Alex Wang for helpful comments on a draft of this work. CW was supported by a NSF Graduate Research Fellowship. Toyota Research Institute provided funds to support this work.

We thank Huan Zhang for helpful discussions and Alex Wang for helpful comments on a draft of this work. CW was supported by a NSF Graduate Research Fellowship. Portions of this work were supported by funds from Toyota Research Institute and the Bosch Center for AI.


Discussion. In this paper, we show that ID-calibrated ensembles, a simple method of calibrating a standard and robust model only on ID data and then ensembling them, can eliminate the tradeoff between in-distribution (ID) and out-of-distribution (OOD) accuracy on a wide range of natural shifts. We hope that this leads to more widespread use and deployment of robustness interventions.

Conclusion and Future Work. In this paper, we show that ID-calibrated ensembles, a simple method of calibrating a standard and robust model only on ID data and then ensembling them, can eliminate the tradeoff between in-distribution (ID) and out-of-distribution (OOD) accuracy on a wide range of natural shifts. We hope that this leads to more widespread use and deployment of robustness interventions.

Instruction: Rename this section to a more appropriate title.

Instruction: Rename the section "Conclusion and Future Work"


Overview Our method consists of three parts. The core component is the rotamer density estimator (RDE),  a conditional normalizing flow that models the probability density of sidechain conformations (rotamers) given the amino acid type and environments (Section 3.2). Next is the algorithm for estimating the entropy of the distribution parameterized by the normalizing flow (Section 3.3). Finally, we show how to use the entropy of the  mutated and wild-type protein-protein interfaces at both bound and unbound states to estimate the change in binding free energy ( ∆∆ G ) upon mutation, and how to use neural networks to predict ∆∆ G more accurately using the unsupervised representations from the RDE (Section 3.4). A protein-protein complex is a multi-chain protein structure that can be divided into two groups. Each group contains at least one protein chain and each chain consists of multiple amino acids. For a protein complex containing n amino acids, we number them from 1 to n . The two groups of the complex can be represented by two disjoint sets of indices A, B ⊂ { 1 . . . n } An amino acid is characterised by its type, position, orientation, and sidechain conformation. We denote the type, position, and orientation of the i -th ( i ∈ { 1 . . . n } ) amino acid as a i ∈ { 1 . . . 20 } , p i ∈ R 3 , and O i ∈ SO (3) respectively. The sidechain conformation of the amino acid is called rotamer . As the conformational degree of freedom of the sidechain is defined by rotatable bonds, a rotamer can be sufficiently parameterized by torsional angles w.r.t. the rotatable bonds. The number of torsional angles vary between 0 to 4 depending on the amino acid type. For an amino acid with d torsional angles, we denote the k -th ( k ∈ { 1 . . . 4 } ) torsional angle by χ ( k ) i ∈ [0 , 2 π ) . Collectively, all the torsional angles are denoted by a vector χ i = ( χ ( k ) i ) dk =1 . Using the language of geometry, an angle can  represented by a point on the unit circle S 1 . A vector consisting of d angular values reside on the product of d unit circle, known as the d -dimensional torus T D = ( S 1 ) D .

Overview Our method comprises three main components. The first is the Rotamer Density Estimator (RDE), which is a conditional normalizing flow that models the probability density of sidechain conformations (rotamers) based on the amino acid type and backbone structures (Section 3.2). The second component is an algorithm that estimates the entropy of the distribution parameterized by the normalizing flow (Section 3.3). Lastly, we describe how we use the entropy of the protein-protein interfaces in both the mutated and wild-type states, both bound and unbound, to estimate the change in binding free energy ( ∆∆ G ). We also detail how we use neural networks to achieve more accurate predictions of ∆∆ G using the unsupervised representations from the RDE (Section 3.4). Definitions and Notations A protein-protein complex is a multi-chain protein structure that can be divided into two groups. Each group contains at least one protein chain and each chain consists of multiple (amino acid) residues. For a protein complex containing n residues, we number them from 1 to n . The two groups of the complex can be represented by two disjoint sets of indices A, B ⊂ { 1 . . . n } . A residue is characterized by its type, position, orientation, and sidechain conformation. We denote the type, position, and orientation of the i -th ( i ∈ { 1 . . . n } ) residue as a i ∈ { 1 . . . 20 } , p i ∈ R 3 , and O i ∈ SO (3) respectively. The sidechain conformation of the residue is called rotamer . As the conformational degree of freedom of the sidechain is defined by rotatable bonds, a rotamer can be parameterized by torsional angles w.r.t. the rotatable bonds. The number of torsional angles varies between 0 to 4 depending on the residue type. For a residue with d torsional angles, we denote the k -th ( k ∈ { 1 . . . 4 } ) torsional angle by χ ( k ) i ∈ [0 , 2 π ) . Collectively, all the torsional angles are denoted by a vector χ i = ( χ ( k ) i ) dk =1 . Using the language of geometry, an angle can be represented by a point on the unit circle S 1 . A vector consisting of d angular values resides on the product of d unit circle, known as the d -dimensional torus T D = ( S 1 ) D .

Instruction: Generate a more formal version of this paragraph

Instruction: Replace all mentions of amino acid by 'residue'. Revise this paragraph for clarity.


Lightweight image super-resolution (SR) networks have obtained promising results with moderate model size. However, they are impractical or neglected to be extended to larger networks. At the same time, model compression techniques, like neural architecture search and knowledge distillation, typically consume considerable computation resources. In contrast, network pruning is a cheap and effective model compression technique. However, it is hard to be applied to SR networks directly, because ﬁlter pruning for residual blocks is well-known tricky. To address the above issues, we propose structure-regularized pruning (SRP), which imposes regularization on the pruned structure to make sure the locations of pruned ﬁlters are aligned across different layers. Speciﬁcally, for the layers connected by the same residual, we select the ﬁlters of the same indices as unimportant ﬁlters. To transfer the expressive power in the unimportant ﬁlters to the rest of the network, we employ L 2 regularization to drive the weights towards zero so that eventually their absence will cause minimal performance degradation. We apply SRP to train efﬁcient image SR networks, resulting in a lightweight network SRPN-L and a very deep one SRPN. We conduct extensive comparisons with both lightweight and larger image SR networks. SRPN-L and SRPN achieve superior performance gains over recent methods quantitatively and visually.

Several image super-resolution (SR) networks have been proposed of late for efﬁcient SR, achieving promising results. However, they are still not lightweight enough and neglect to be extended to larger networks. At the same time, model compression techniques, like neural architecture search and knowledge distillation, typically consume considerable computation resources. In contrast, network pruning is a cheap and effective model compression technique. However, it is hard to be applied to SR networks directly because ﬁlter pruning for residual blocks is well-known tricky. To address the above issues, we propose structure-regularized pruning (SRP), which imposes regularization on the pruned structure to ensure the locations of pruned ﬁlters are aligned across different layers. Speciﬁcally, for the layers connected by the same residual, we select the ﬁlters of the same indices as unimportant ﬁlters. To transfer the expressive power in the unimportant ﬁlters to the rest of the network, we employ L 2 regularization to drive the weights towards zero so that eventually, their absence will cause minimal performance degradation. We apply SRP to train efﬁcient image SR networks, resulting in a lightweight network SRPN-Lite and a very deep one SRPN. We conduct extensive comparisons with both lightweight and larger networks. SRPN-Lite and SRPN perform favorably against other recent efﬁcient SR approaches quantitatively and visually.

Instruction: Replace all occurrences of SRPN-L with SRPN-Lite. Improve the english of this paragraph.

Instruction: Replace SRPN-L by SPRN-Lite. Make the first and last sentence more fitting to the academic style.


For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope? [Yes] See Section (b) Did you describe the limitations of your work? [Yes] See Section (c) Did you discuss any potential negative societal impacts of your work? [No] (d) Have you read the ethics review guidelines and ensured that your paper conforms tothem? [Yes]2. If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results? [Yes] See Section 2(b) Did you include complete proofs of all theoretical results? [Yes] See the supplementalmaterial (Appendix.pdf)3. If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material or as a URL)? [Yes] See the supplemental material. Our codes contains README.md to reproduce the our experi- mental results. (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section (c) Did you report error bars (e g., with respect to the random seed after running experiments multiple times)? [Yes] See Section (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)? [Yes] See Section 5. If you are using existing assets (e g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators? [N/A] (b) Did you mention the license of the assets? [Yes] See the supplemental materials. Our codes contains LICENSE.txt. (c) Did you include any new assets either in the supplemental material or as a URL? [Yes] See the supplemental materials containing our source codes (d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating? [N/A] Our dataset was reproduced by ourselves based on the literature (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content? [Yes] If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable? [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]

For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? [Yes] See Section(b) Did you describe the limitations of your work? [Yes] See Section(c) Did you discuss any potential negative societal impacts of your work? [No] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] The paper conforms with the provided guidelines. If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results? [Yes] See Section(b) Did you include complete proofs of all theoretical results? [Yes] See the supplemental material (Appendix.pdf) 3. If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [Yes] See the supplemental material. Our codes contains README.md to reproduce the our experimental results. (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [Yes] See Section(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Section 5. 4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators? [N/A] (b) Did you mention the license of the assets? [Yes] See the supplemental materials. Our codes contains LICENSE.txt. (c) Did you include any new assets either in the supplemental material or as a URL? [Yes] See the supplemental materials containing our source codes (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating? [N/A] Our dataset was reproduced by ourselves based on the literature (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] 5. If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]


We conduct a preliminary investigation of this issue by studying the generalizability of KD, BU and LID for detecting previously unseen attack strategies on the CIFAR-10 dataset. The KD, BU and LID detectors are trained on samples of the simplest attack strategy, FGM, and then tested on samples of the more complex attacks BIM-a, BIM-b, JSMA and Opt. The training and test datasets are generated in the same way as in our previous experiments, and the  test attack data is standardized by scaling so as to ﬁt the training data. The results are shown in Table 2, from which we see that the LID detector trained on FGM can accurately detect the much more complex attacks of the other strategies. The KD and BU characteristics can also achieve good performance on this transfer learning task, but are less consistent than our proposed LID characteristic. The results appear to indicate that the adversarial regions generated by different attack strategies possess similar dimensional properties.

We conduct a preliminary investigation of this issue by studying the generalizability of KD, BU and LID for detecting previously unseen attack strategies on the CIFAR-10 dataset. The KD, BU and LID detectors are trained on samples of the simplest attack strategy, FGM, and then tested on samples of the more complex attacks BIM-a, BIM-b, JSMA and Opt. The training and test datasets are generated in the same way as in our previous experiments with only the FGM attack applied on the train set while the other attacks applied separately on the test set. The test attack data is standardized by scaling so as to ﬁt the training data. The results are shown in Table 2, from which we see that the LID detector trained on FGM can accurately detect the much more complex attacks of the other strategies. The KD and BU characteristics can also achieve good performance on this transfer learning task, but are less consistent than our proposed LID characteristic. The results appear to indicate that the adversarial regions generated by different attack strategies possess similar dimensional properties.


A , three with Design B, and 10 with Design C . To complete this task, participants could rely on the bars that indicate allowed medication intake times with Design A and Design C . With Design B , the marker on the medication entry indicated the allowed time. As most participants were not successful in completing this task with Design B , they provided ample feedback on this design. Five participants said the design does not support the task, for example, P1  “it doesn’t show any other time of the day that you can take it”. Three participants (P5, P11, and P12) said they can reschedule in any free slot, for example, P9 said “It seems that 7am is a possibility because there is no other and there is no indication of conflicts.” . P10 commented they would move it to a slot and observe if a conflict was flagged, saying “I don’t know. I think I would just move it and see if a conflict came up.” . P8 participants indicated that use use of the bar to indicate allowed schedule times could also be read as extended release time, saying “That would mean that it’s something that it’s an extended release. Warfarin is not an extended release.” P10 and P6 reasoned that a medication that is supposed to be taken at a specific time point should not occupy a full hour on the calendar. For example, P10 said “I really don’t like this fact that it says 6am on the side and then it makes it a block of time.”

B . To complete this task, participants could rely on the bars that indicate allowed medication intake times with Design A and Design C . With Design B , the marker on the medication entry indi- cated the allowed time and five participants said Design B did not support that task. For example, P1 said “it doesn’t show any other time of the day that you can take it”. Three participants (P5, P11, and P12) said they could reschedule in any free slot, for example, P9 said “It seems that 7am is a possibility because there is no other and there is no indication of conflicts.” . P10 commented they would move it to a slot and observe if a conflict was flagged, saying “I don’t know. I think I would just move it and see if a conflict came up.” . P indicated the use of the bar to indicate allowed schedule times could also be read as extended release time, saying “That would mean that it’s something that it’s an extended release. Warfarin is not an extended release.” P10 and P6 reasoned that a medication that is supposed to be taken at a specific time point should not occupy a full hour on the calendar. For example, P10 said “I really don’t like this fact that it says 6am on the side and then it makes it a block of time.”

Instruction: Make this paragraph much more concise.

Instruction: Please write more concisely about design B.


Semantic co-occurrence. Semantic context characterizes the co-occurrences of different objects and can be used  to group and separate pixels. We deﬁne semantic context as the union of object classes in each image. Even without the location of  labels, we can leverage semantic context to impose global regularization in the latent  feature space, i.e., the pixel-wise feature embedding should be separated from all the other pixels from images without  overlapping object categories.

Semantic co-occurrence. Semantic context characterizes the co-occurrences of different objects, which can be used as a prior to group and separate pixels. We deﬁne semantic context as the union of object classes in each image. Even without the pixel-wise localization of semantic labels, we can leverage semantic context to impose global regularization on the latent feature: The feature should separate images without any overlapping object categories.

Instruction: Rewrite this paragraph to be more concise.

Instruction: Split the last sentence and make it slightly shorter. Improve the english.


Results Table 2 shows our results. iDSPN is close to solving the problem at any training set size by being exclusively multiset-equivariant like DSPN. As expected, the set-equivariant models are unable to solve this task because they cannot map the equal elements in the input to different elements in the output. This applies even to transformers with random position embeddings, which similarly to TSPN (Kosiorek et al., 2020) and Slot Attention (Locatello et al., 2020) use noise to make elements not exactly equal. Meanwhile,  transformers with position encoding and BiLSTM require at least 100 × more training samples to come close to the performance of iDSPN. This is because they lack the correct structural bias of not relying on the order of the elements, which makes them less sample-efﬁcient. Note that the non-equivariant models are unlikely to beneﬁt signiﬁcantly from even more training data because they overﬁt on 1 × and 10 × , but no longer on 100 × .

Results. Table 2 shows our results. The two exclusively multiset-equivariant models DSPN and iDSPN perform well at any training set size. As expected, the set-equivariant models are unable to solve this task because they cannot map the equal elements in the input to different elements in the output. This applies even to transformers with random “position” embeddings, which similarly to TSPN (Kosiorek et al., 2020) and Slot Attention (Locatello et al., 2020) use noise to make elements not exactly equal. Meanwhile, BiLSTM and transformers with normal positional encoding require at least 100 × more training samples to come close to the performance of iDSPN. This is because they lack the correct structural bias of not relying on the order of the elements, which makes them less sample-efﬁcient. Note that the non-equivariant models are unlikely to beneﬁt signiﬁcantly from even more training data because they overﬁt on 1 × and 10 × , but no longer on 100 × .

Instruction: Simplify the second sentence.

Instruction: Rewrite the 2nd sentence to make it easier to read and less confusing.


Colored-and-gray-MNIST (Kim et al., 2019) is a synthetic dataset based on MNIST (LeCun et al., 1998). In the training set of 60,000 examples, each example has two images, a gray-scale image and a monochromatic image, with color strongly correlated with its digit label. For the validation set of 10,000 examples, each example also has a gray-scale  and a corresponding monochromatic image however with a low correlation between the color and its label. We consider the monochromatic image as the ﬁrst modality m 0 and the gray-scale one as the second modality m 1 . We use a neural network with four convolutional layers as the uni-modal branch and employ three MMTMs (Joze et al., 2020) to connect them. The corresponding uni-modal DNNs trained on the monochromatic images and We use this synthetic dataset mainly to show that the proposed greedy learner hypothesis happens in practice.

Colored-and-gray-MNIST (Kim et al., 2019) is a synthetic dataset based on MNIST (LeCun et al., 1998). In the training set of 60,000 examples, each example has two images, a gray-scale image and a monochromatic image, with color strongly correlated with its digit label. For the validation set of 10,000 examples, each example also has a gray-scale image and a corresponding monochromatic image however with a low correlation between the color and its label. We consider the monochromatic image as the ﬁrst modality m 0 and the gray-scale one as the second modality m 1 . We use a neural network with four convolutional layers as the uni-modal branch and employ three MMTMs to connect them. The corresponding uni-modal DNNs trained on the monochromatic images and the gray-scale images achieve the validation accuracies of 43% and 98%, respectively. We use this synthetic dataset mainly to demonstrate the proposed greedy learner hypothesis.


In this work, we develop a novel third family of attacks, recursive gradient attack on privacy (RGAP), that is based on a recursive, depth-wise algorithm for recovering training data from gradient information. Different from the analytical attack using the bias term, R-GAP utilizes much more information and is the ﬁrst closed-form algorithm that works on both convolutional networks and fully connected networks with or without bias term. Compared to optimization-based attacks, it is not susceptible to local optima, and is orders of magnitude faster to run with a deterministic running time. Furthermore, the insights gained from the closed form of our recursive attack have lead to a reﬁned rank analysis that predicts which network architectures enable full recovery, and which lead to provable security or noisy recovery due to rank-deﬁciency. This explains well the performance of both closed-form and optimization-based attacks.

In this work, we develop a novel third family of attacks, recursive gradient attack on privacy (RGAP), that is based on a recursive, depth-wise algorithm for recovering training data from gradient information. Different from the analytical attack using the bias term, R-GAP utilizes much more information and is the ﬁrst closed-form algorithm that works on both convolutional networks and fully connected networks with or without bias term. Compared to optimization-based attacks, it is not susceptible to local optima, and is orders of magnitude faster to run with a deterministic running time. Furthermore, we show that under certain conditions our recursive attack can fully recover training data in cases where optimization attacks fail. Additionally, the insights gained from the closed form of our recursive attack have lead to a reﬁned rank analysis that predicts which network architectures enable full recovery, and which lead to provable noisy recovery due to rankdeﬁciency. This explains well the performance of both closed-form and optimization-based attacks.


Finally, we evaluate iDSPN on the CLEVR (Johnson et al., 2017) object property prediction task that was used in DSPN (Zhang et al., 2019) and Slot Attention (Locatello et al., 2020). Given an image of a synthetic 3d scene containing several objects, the goal is to predict the set of their properties: 3d coordinate, size, color, and material (18 dimensions in total). An additional dimension is added as a binary indicator for whether an element is present or not to account for the different set sizes in the dataset. The main comparison we intend to make here is against Slot Attention, the state-of-the-art model on this dataset. Setup We  follow the experimental setup of DSPN where a ResNet (He et al., 2016) encodes the image into the input vector z , which is decoded into the set by iDSPN. We  include a few modiﬁcations to improve our results and also simplify the model slightly, which we elaborate on in Appendix D. Most notably, we use Nesterov’s Accelerated Gradient as iDSPN optimizer.

Finally, we evaluate iDSPN on the CLEVR (Johnson et al., 2017) object property prediction task that was used in DSPN (Zhang et al., 2019) and Slot Attention (Locatello et al., 2020). Given an image of a synthetic 3d scene containing up to ten objects, the goal is to predict the set of their properties: 3d coordinate, size, color, and material (18 dimensions in total). To account for different set sizes in the dataset, a 19th dimension is added as a binary indicator for whether an element is present or not. Setup. We mostly follow the experimental setup of DSPN where a ResNet (He et al., 2016) encodes the image into the input vector z , which is decoded into a set by iDSPN. We also include a few modiﬁcations, which we elaborate on in Appendix D. Most notably, we use Nesterov’s Accelerated Gradient as iDSPN optimizer and increase the number of optimization steps, both changes that would make DSPN even slower and use more memory.


Pratt, 1998) and (multi-source) domain adaptation. In fact, there are a few works in domain generalization that are inspired by the meta-learning principles, such as Li et al. (2018a) ; Balaji et al. ; Li et al. Dou et al. In Ren et al. (2018), we also see the leveraging of gradient inner product in meta-learning, where it is used to determine the importance weight of training examples. We discuss the connection between our proposed algorithm to meta-learning in more details in Appendix A.1.

Pratt, 1998) and (multi-source) domain adaptation. In fact, there are a few works in domain generalization that are inspired by the meta-learning principles, such as Li et al. (2018a); Balaji et al. ; Li et al. Dou et al. Speciﬁcally, Li et al. (2020) also proposes to adapt Reptile for domain generalization tasks, however they study their method under the sequential learning setting, whereas our method can be trained on all domains and therefore learns faster, especially when the number of domains is large. In Ren et al. (2018), we also see the leveraging of gradient inner product in meta-learning, where it is used to determine the importance weight of training examples. We discuss the connection between our proposed algorithm to meta-learning in more details in Appendix A.1.


Twelve local university students from a different participants group from experiment 1 participated in this experiment. The average age was 22.3 years ( SD = 1 . 67). All participants were skillful in mouse operation and used their dominant right hand.

A total of 12 local university students from a different participants group from that in Experiment 1 participated in this experiment. The average age was 22.3 years ( SD = 1 . 67). All participants were skilled in mouse operation and used their dominant hand (right hand).

Instruction: Rephrase the paragraph

Instruction: Revise this text to make it more clear.


Reweighting methods weight individuals with balanced score to obtain globally balanced distributions, represented by the inverse propensity score (IPS) approach (Rosenbaum & Rubin, 1983a) and its doubly robust variant (Robins et al., 1994). Imai & Ratkovic (2014) and Fong et al. (2018) propose to calculate the balancing score via an optimization problem. Kuang et al. (2017b) and Kuang et al. (2017a) further consider the non-confounding factors in covariates. However, these methods  suffer from  high variance and are vulnerable to non-overlapped units.

Reweighting-based methods weight individuals with balanced scores to achieve globally balanced distributions, represented by the inverse propensity score (IPS) approach (Rosenbaum & Rubin, 1983a) and its doubly robust variant (Robins et al., 1994). Imai & Ratkovic (2014) and Fong et al. (2018) propose calculating the balancing score by solving an optimization problem. Kuang et al. (2017b) and Kuang et al. (2017a) consider additional non-confounding factors in covariates. However, these methods are susceptible to non-overlapping units and suffer from a high variance issue.

Instruction: Use formal words in the last sentence.

Instruction: Reorder the last sentence arguments. Make this paragraph a bit more precise.


MVC problem respectively. In both problems and across the five datasets, Meta-EGN siginificantly outperforms EGN and RUN-CSP, both before and after the fine-tuning step. In comparison with the traditional CO solvers, Meta-EGN narrows the gap from Gurobi9.5 on those real small graphs. For RB graphs, Meta-EGN outperforms Gurobi9.5  for  the MC problem. For the MVC problem, Meta-EGN outperforms Gurobi9.5 on RB500.

MVC problem respectively. In both problems and across the five datasets, Meta-EGN significantly outperforms EGN and RUN-CSP, both before and after the fine-tuning step. In comparison with the traditional CO solvers, Meta-EGN narrows the gap from Gurobi9.5 on those real small graphs. For RB graphs, Meta-EGN outperforms Gurobi9.5 on RB500 for both the MC and MVC problems.

Instruction: Fuse the last two sentences for conciseness.

Instruction: Merge the two last sentences to make it shorter.


A curious feature of our model is that during training one has to back-propagate over the gradient of  target distribution multiple times to optimize R . In (Titsias & Dellaportas, 2019), the authors avoid multiple back-propagation by stopping the derivative calculation at the density gradient term. In our experiment, we ﬁnd it is necessary for good performance.

A curious feature of our model is that during training one has to back-propagate over the gradient of the target distribution multiple times to optimize R . In (Titsias & Dellaportas, 2019) the authors avoid multiple back-propagation by stopping the derivative calculation at the density gradient term. In our experiment we do not use this trick and perform full back-propagation without encountering any issue. We found that stopping the derivative computation instead harms performance.