Results According to Table 1, our RDE-Network outperforms all the baselines. Notably,RDENetwork improves per-structure correlations by a large margin, which implies that it is significantly more reliable for practical applications. The advantage of RDE-Network over MIF-Network shows that representations obtained by fitting rotamer densities are more effective than those from masked inverse folding because protein binding is driven by atomic interactions which RDE captures  well by modeling the conformation of sidechain atoms.

Results According to Table 1, our RDE-Network outperforms all the baselines. Notably, it demonstrates a significant improvement in per-structure correlations, indicating its greater reliability for practical applications. The superior performance of RDE-Network over MIF-Network suggests that representations derived from fitting rotamer densities are more effective than those from masked inverse folding, as RDE captures atomic interactions well by modeling the conformation of sidechain atoms.

Instruction: Paraphrase this paragraph using formal language

Instruction: Revise this paragraph in a more academic style.


In this section, we introduce the greedy learner hypothesis to explain challenges observed in training multi-modal DNNs. Before describing our hypothesis, we start by discussing some assumptions on the multi-modal data and preliminary observations made in the literature on multi-modal learning.

In this section, we introduce the greedy learner hypothesis to explain challenges observed in training multi-modal DNNs. Before describing our hypothesis, we discuss some assumptions on the multi-modal data and preliminary observations made in the multi-modal learning literature.

Instruction: Make expression concise.

Instruction: Improve the English of this paragraph.


Compared with ESTR in [17], (cid:15) -FALB in [15] and LowESTR in [ ], our algorithms are designedfor nonlinear reward framework. Compared with LowGLOC in [26], our algorithms  achieve a betterregret bound, can work with varying action sets and are computationally feasible. For G-ESTT,we extend the GLM-UCB algorithms [11] via a regularization technique along with some noveltechniques. Our proposed G-ESTS is simple and could be easily implemented based on anystate-of-the-art generalized linear bandit algorithms. In particular, when we combine G-ESTS withsome efﬁcient algorithms (e.g. SGD-TS [9]), the total time complexity after a warm-up stage scalesas O ( Tr ( d 1 + d 2 )) . We verify that G-ESTT and G-ESTS are the ﬁrst two algorithms to attain the˜ O (( d 1 + d 2 ) r √ T ) optimal regret bound of low-rank matrix bandit problems up to logarithmic terms.

Compared with ESTR in [17], (cid:15) -FALB in [15] and LowESTR in [26], our algorithms are proposed forthe nonlinear reward framework with arbitrary action matrices. Compared with LowGLOC in [26], our algorithms not only achieve a better regret bound in theory, but also are computationally feasiblein practice. For G-ESTT, we extend the GLM-UCB algorithms [11] via a novel regularizationtechnique. Our proposed G-ESTS is simple and could be easily implemented based on anystate-of-the-art generalized linear bandit algorithms. In particular, when we combine G-ESTS withsome efﬁcient algorithms (e.g. SGD-TS [9]), the total time complexity after a warm-up stage scalesas O ( Tr ( d 1 + d 2 )) . We verify that G-ESTT and G-ESTS are the ﬁrst two algorithms to attain the˜ O (( d 1 + d 2 ) r √ T ) optimal regret bound of low-rank matrix bandit problems up to logarithmic terms.

Instruction: Improve the English of this paragraph

Instruction: Rewrite the first part of the paragraph to make it more convincing.


A and Design B , and 7.0 seconds with Design C . To complete this task with Design A , participants had tovertically scroll to the end of the day to see all entries. Three participants (P3, P5, and P7) complained about this. For example, P3 said “I find it’s a lot of scrolling down. It would be helpful if there was a way to condense it or to make it possible to see the entire calendar available in terms of morning, afternoon, and evening.” , and P8 said “The time frames are a bit big. So it makes like I said, it really makes it scroll off that you can’t see it all in one consolidated view” . With Design B , participants were expected to use the daily medica- tion summaries provided at the top. Three participants (P1, P6, and P9) found the daily summaries helpful in performing this task. For example, P9 said “Yeah, I like the idea of having the first row on the calendar dedicated only for the medications that needs to be taken. I think it brings an overall idea [of] what should be taken during that day.” . Three participants (P3, P4, and P5) also complained about the lines demarcating days not being clear. For example, P4 said “I have a harder time differentiating the calendar component the days, because there’s not a strong border between the days of the week.” .

Three participants (P3, P5, and P7) complained about the need to scroll to the end of the day with Design A . For example, P3 said “I find it’s a lot of scrolling down. It would be helpful if there was a way to condense it or to make it possible to see the entire calendar available in terms of morning, afternoon, and evening.” , and P8 said “The time frames are a bit big. So it makes like I said, it really makes it scroll off that you can’t see it all in one consolidated view” . With Design B , participants were expected to use the daily medication summaries provided at the top. Three participants (P1, P6, and P9) found the daily summaries helpful in performing this task. For example, P9 said “Yeah, I like the idea of having the first row on the calendar dedicated only for the medica- tions that needs to be taken. I think it brings an overall idea [of] what should be taken during that day.” . Three participants (P3, P4, and P5) also complained about the lines demarcating days not being clear. For example, P4 said “I have a harder time differentiating the calendar component the days, because there’s not a strong border between the days of the week.” .

Instruction: Remove unnecessary details for the paragraph.

Instruction: The first sentence is a bit unclear.


However, these “black box" policies lack interpretability, and reliability and, moreimportantly, cannot operate under the  TCP datapath’s ultra-contingent latencyand computational constraints. This paper proposes a novel two-stage solutionto achieve the best of both worlds: ﬁrst to train a deep RL agent, then distill its(over-)parameterized NN policy into white-box, light-weight rules in the formof symbolic expressions that are much easier to understand and to implementin constrained environments. At the core of our proposal is a novel symbolicbranching algorithm that allows the rule to be “context-aware” of various networkconditions, eventually converting the NN policy into a symbolic tree. The distilledsymbolic rules preserve and often improve performance over state-of-the-art NNpolicies while being faster and simpler than a standard neural network. We validatethe performance of our distilled symbolic rules on both simulation and emulationnetwork systems. Our code will be released upon acceptance.

However, such “black-box” policies lack interpretability and reliability, and often, they need to operate outside the traditional TCP datapath due to the use of complex NNs. This paper proposes a novel two-stage solution to achieve the best of both worlds: first to train a deep RL agent, then distill its (over-)parameterized NN policy into white-box, light-weight rules in the form of symbolic expressions that are much easier to understand and to implement in constrained environments. At the core of our proposal is a novel symbolic branching algorithm that enables the rule to be aware of the context in terms of various network conditions, eventually converting the NN policy into a symbolic tree. The distilled symbolic rules preserve and often improve performance over state-of-the-art NN policies while being faster and simpler than a standard neural network. We validate the performance of our distilled symbolic rules on both simulation and emulation environments. Our code is available at https://github.com/VITA-Group/SymbolicPCC .

Instruction: Review the following paragraph, update if possible, delete unnecessary details.


We choose f to be a simple model: a single linear layer that maps from dimensionalityto 75. We use triplet loss (Schroff et al., 2015) to move the representation of the anchor vector V A closer to the representation of the positive vector V P and farther apart from the representation of the negative vector V N . Following Hoffer & Ailon (2015), we calculate the softmax version of the triplet loss:

We choose f to be a simple model: a single linear layer that maps from dimensionalityto 75. The dimensional of the transformation was chosen according to development set performance. We use triplet loss (Schroff et al., 2015) to move the representation of the anchor vector V A closer to the representation of the positive vector V P and farther apart from the representation of the negative vector V N . Following Hoffer & Ailon (2015), we calculate the softmax version of the triplet loss:


KG, D RAGON consists of a cross-modal encoder (GreaseLM) that fuses the input text-KG pair bidirectionally (§2.2), and a pretraining objective that performs bidirectional self-supervision on the text-KG input (§2.3). Our pretraining objective unifies masked language modeling (MLM) and KG link prediction (LinkPred) to make text and KG mutually inform each other and learn joint reasoning over them. Finally, we describe how we finetune the pretrained D RAGON model for downstream tasks (§2.4). While the individual piece of our approach (GreaseLM, MLM, LinkPred) is not new in itself, our contribution is that we are the first to bring these pieces together, present how to unify them  effectively and show that this produces a significantly performant pretrained model (§3, §4).

KG, D RAGON consists of a cross-modal encoder (GreaseLM) that fuses the input text-KG pair bidirectionally (§2.2), and a pretraining objective that performs bidirectional self-supervision on the text-KG input (§2.3). Our pretraining objective unifies masked language modeling (MLM) and KG link prediction (LinkPred) to make text and KG mutually inform each other and learn joint reasoning over them. Finally, we describe how we finetune the pretrained D RAGON model for downstream tasks (§2.4). While each individual piece of our approach (GreaseLM, MLM, LinkPred) is not new in itself, we are the first to bring them together effectively and demonstrate that the resulting model has strong empirical results.

Instruction: Make this paragraph more concise.

Instruction: Make the last sentence more concise.


Pruned Index Constraint . Filter pruning in residual networks is well-known tricky because the add operators in residual blocks demand the pruned ﬁlter indices  must be aligned. Filter pruning within a residual block is shown in Fig. 2(b). A typical residual block (e.g., in EDSR (Lim et al., 2017), RCAN (Zhang et al., 2018b)) is made up of two convolutional layers. All the convolutional layers can be categorized into two groups based on their connection relationship among one another. One group comprises the layers that can be pruned without any constraint , dubbed free Conv layers in this work; the other comprises  layers in which the ﬁlters must be pruned at the same indices , dubbed constrained Conv layers . For a concrete example, in Fig. 2(b), the layer W ( i )  is a free Conv layer and layer W ( i +1) is a constrained Conv layer.

Pruned Index Constraint . Pruning ﬁlters in residual networks is well-known non-trivial as the Add operators in residual blocks require the pruned ﬁlter indices across different residual blocks must be aligned. A ﬁgurative illustration of ﬁlter pruning within a residual block is shown in Fig. 2(b). A typical residual block (e.g., in EDSR (Lim et al., 2017), RCAN (Zhang et al., 2018b)) consists of two convolutional layers. According to the mutual connection relationship, the convolutional layers can be categorized into two groups. One group is made up with the layers that can be pruned without any constraint , dubbed free Conv layers in this work; the other comprises Conv layers whose ﬁlters must be pruned at the same indices , dubbed constrained Conv layers . Concretely, the layer W ( i ) in Fig. 2(b) is a free Conv layer, while the layer W ( i +1) is a constrained one.

Instruction: Improve the language to make it more fitting to the academic style.


Another interesting future direction is to improve upper bounds on the pseudo-dimension by restricting heuristic functions to some classes, as mentioned in Section 2. Further study of this direction will be important, particularly when applying GBFS/A* with learned heuristics to path-finding instances with extremely many vertices. In Appendix D, we present an illustrative example where we can achieve polylog( n ) upper bounds on the pseudo-dimension by assuming that heuristic functions with much fewer tunable parameters than n can be designed in an instance-specific manner.

Another interesting future direction is to improve upper bounds on the pseudo-dimension by restricting heuristic functions to some classes. Appendix D will present an illustrative example where we can achieve polylog( n ) upper bounds on the pseudo-dimension by assuming that heuristic functions with much fewer tunable parameters than n can be designed in an instance-specific manner.

Instruction: Make this paragraph shorter by eliminating details about further work.

Instruction: Delete the sentence about further study and the reference to section 2.


Statistical estimators exhibit competitive performance on the PEHE metric. In particular, neuralnetwork estimators outperformthe linear and random forest methods because they can depict the nonlinearity in data. TARNet obtains better overall performance than other statistic estimators by absorbing the advantages (R et al., 2019) ofboth T-learner and S-learner. However, the  treatment selection bias makes these estimators biased, leading to sub-optimal performance.

Statistical estimators exhibit competitive performance on the PEHE metric. Due to the superiority to depict non-linearity, neural estimators outperform linear and random forest methods. In particular, TARNet that absorbs the advantage of T-learner and S-learner achieves the best overall performance in statistic estimators. However, the circumvention to treatment selection bias leads to inferior performance.

Instruction: The second sentence is too complicated. Make it more understandable. Also brush up the rest.

Instruction: Reorganize the ideas in the sentences to improve the logical flow of the text.


We then propose to divide states into several groups  and incorporate them with an attention mechanism to select an appropriate  state  as a goal to encourage further exploration. In order tohelp RL agents learn efﬁciently, we update goal generation hindsight and value estimation with related trajectories.

Speciﬁcally, we ﬁrst divide states into several groups according to their uncertainty and locations in the graph. We then adopt an attention mechanism to select an appropriate group and assign the state with the highest value in the graph as a goal to encourage further exploration. We also propose to update goal generation hindsightly and value estimation with related trajectories, to help RL agents learn efﬁciently.

Instruction: I need more detailed explanations.


• AGILE-Tuned without sync-freq-change: In Mnih et al. (2015), the authors used the periodic syncing between the target and the main networks to alleviate the issue of the frequently moving target in updating the Q-network. In this work, we compared two extreme cases of the frequency period in syncing the networks; 10 ( Sync-freq=10 in Fig. 13 (a)) and 500 ( AGILE-Tuned ). • AGILE-Tuned without graph-dim-change: In order to understand the difﬁculty in expressing the action relation through the compact representation, we compared the big and the small representations in the action graph, i e., the node-features are encoded in 32 ( Graph-dim=32 ) or 64( AGILE-Tuned ) dimensions.

• AGILE-Tuned without sync-freq-change: In Mnih et al. (2015), the authors used the periodic syncing between the target and the main networks to alleviate the issue of frequently moving Qvalue targets. In this work, we compare two extreme cases of the sync frequency: 10 depicted by Sync-freq=10 in Fig. 13 (a) and 500 depicted by AGILE-Tuned . • AGILE-Tuned without graph-dim-change: To understand the difﬁculty in expressing the action relations through a compact representation, we compare two hidden dimension sizes. The node-features are encoded in 32 ( Graph-dim=32 ) or 64( AGILE-Tuned ) dimensions.

Instruction: Improve the English of this paragraph and make it shorter.

Instruction: Make the wording of this paragraph much more straight forward, to be more concise.


Cross-Modal Alignment Objective Functions Most previous methods adopt triplet loss as a major objective function for video-language modeling. CGMSCD [13] points out that the triplet loss sometimes leads to a wrong learning direction and thus devises an adaptive margin triplet loss for representation learning. More recent works [40, 17, 18] propose to apply the InfoNCE contrastiveloss [46, 37, 6] to enhance representation learning. Particularly, COTS [31] introduces a momentummechanism [14] to maintain more negative samples for image-text contrastive learning. Following COTS, we propose momentum video-level contrastive learning for video-text global alignment. Note that MIL-NCE [34] enhances the InfoNCE loss with multiple-instance learning (MIL) to cope with the misaligned narration descriptions in HowTo100M [35]. In this paper, we thus propose momentum frame-level MSL-contrastive learning to assist in addressing the misaligned frame problem.

Functions Most previous methods adopt triplet loss as a major objective function for video-language modeling. CGMSCD [14] points out that the triplet loss sometimes leads to a wrong learning direction and thus devises an adaptive margin triplet loss for representation learning. More recent works [41, 18, 19] propose to apply the InfoNCE contrastive loss [47, 38, 6] to enhance representation learning. Particularly, COTS [32] and BriVL [12] introduce a momentum mechanism [15] to maintain more negative samples for image-text contrastive learning. Following these two state-of-the-art models, we propose momentum video-level contrastive learning for video-text global alignment in this paper. Note that MIL-NCE [35] enhances the InfoNCE loss with multiple-instance learning (MIL) to cope with the misaligned narration descriptions in HowTo100M [36]. In this work, we thus propose momentum frame-level MSL-contrastive learning to assist in addressing the misaligned frame problem.


Problems in DSPN Increasing the number of optimization steps in DSPN generally results in a better solution, but requires signiﬁcantly more memory and computation time. To be able to backpropagate through the inner optimization, the activations of every intermediate  step have to be kept in memory. Furthermore, each additional optimization step  requires backpropagating that step in the backward pass as well. These issues limit the number of iterations (10 in DSPN) that are computationally feasible, which can have a negative effect on the modeling capacity. We aim to address these problems in the following.

Problems in DSPN. Increasing the number of optimization steps for solving Equation 7 generally results in a better solution (Zhang et al., 2019), but requires signiﬁcantly more memory and computation time and can lead to training problems (Belanger et al., 2017). To be able to backpropagate through Equation 7, the activations of every intermediate gradient descent step have to be kept in memory. Each additional optimization step in the forward pass also requires backpropagating that step in the backward pass. These issues limit the number of iterations that are computationally feasible (DSPN uses only 10 steps), which can have a negative effect on the modeling capacity due to insufﬁcient minimization of Equation 7. We aim to address these problems in the following.


AdaptFormer is only employed in recognition tasks in this work, it’s unclear whether it can workwell in tasks beyond recognition, e.g. , object detection and semantic segmentation. We leave it forthe future exploration. Since our method is specially designed for efﬁcient ﬁne-tuning, we do notforesee obvious undesirable ethical/social impacts at this moment.

AdaptFormer is only employed in recognition tasks in this work, it’s unclear whether it can workwell in tasks beyond recognition, e.g. , object detection and semantic segmentation. We leave it forthe future exploration. Since our method is specially designed for efficient fine-tuning, we do notforesee obvious undesirable ethical/social impacts at this moment. Checklist For all authors...


Reproducibility Statement. For theory, we provide proof and additional results in the Appendix. For empirical results, we provide implementation and environment details and hyperparameters in the Appendix. We also submit anonymous code in the supplemental materials.

R EPRODUCIBILITY S TATEMENT For theory, we provide proof and additional results in the Appendix. For empirical results, we provide implementation and environment details and hyperparameters in the Appendix. We also submit anonymous code in the supplemental materials.


Functional Causal Model In functional causal model (FCM), the relationships between variables are expressed through deterministic, functional equations: x i = f i ( pa i , u i ) , i = 1 , ..., N . The uncertainty in FCM is introduced via the assumption that variables u i , i = 1 , ..., N , are not observed (Pearl et al., 2000). If each function in FCM represents an autonomous mechanism, such FCM is called a structural model. Moreover, if each mechanism determines the value of one and only one variable, then the model is called a structural causal model (SCM). Taking the view from the SCM’s perspective, we want to learn a mixture of causal models whose inputs are pure latent variables and whose output is a single high-dimensional variable that describes complex data such as images.

Functional Causal Model In functional causal model (FCM), the relationships between variables are expressed through deterministic, functional equations: x i = f i ( pa i , u i ) , i = 1 , ..., N . The uncertainty in FCM is introduced via the assumption that variables u i , i = 1 , ..., N , are not observed (Pearl et al., 2000). If each function in FCM represents an autonomous mechanism, such FCM is called a structural model. Moreover, if each mechanism determines the value of one and only one variable, then the model is called a structural causal model (SCM). The SCMs form the basis for many statistical methods (Mooij & Heskes, 2013; Mooij et al., 2016) that aim at inferring knowledge of the underlying causal structure from data (Bongers et al., 2016). Taking the view from the SCM’s perspective, we want to learn a mixture of causal models whose inputs are pure latent variables and whose output is a single high-dimensional variable that describes complex data such as images.


Hierarchical Modules. As reported in Table 7, on top of our final solution of Point-M2AE in the first row, we respectively experiment with removing the hierarchical encoder, hierarchical decoder,skip connections, and local spatial self-attention layers from our framework. Specifically, we replace our encoder and decoder with 1-stage plain architectures similar to MAE, which contains 15 and 2blocks of vanilla self-attention layers, respectively. We observe the absence of multi-stage structures either in encoder or decoder would hurt the performance, and the hierarchical encoder plays a better role than the decoder. Also, the skip connectionsand local spatial attention can well benefit thenetwork by providing complementary information and local inductive bias.

Hierarchical Modules. As reported in Table 7, on top of our final solution, Point-M2AE, in the first row, we respectively experiment with removing the hierarchical encoder, hierarchical decoder, and skip connections from our framework. Specifically, we replace our encoder and decoder with 1-stage plain architectures similar to MAE, which contains 15 and 2 vanilla transformer blocks, respectively. We observe the absence of multi-stage structures either in encoder or decoder hurts the performance, and the hierarchical encoder plays a better role than the decoder. Also, the skip connections well benefits the accuracy by providing complementary information for the decoder.

Instruction: Rewrite the last sentence to make it more concise.

Instruction: Make the paragraph shorter.


Conventional TCP CC adopts a heuristic-based approach where the heuristic functions are manually crafted to adjust the trafﬁc rate in a deterministic manner. Some proposals use packet loss as a signal for network congestion, e.g., Cubic [4], Reno [20],and NewReno [6], and  others rely onthe variation of delay, e.g., Vegas [5]. Other CC designs combine packet lossand delay [21, 22]. Recently, different CC techniques specialized for data-center networks are alsoproposed [2, 3, 23].

Conventional TCP CC adopts a heuristic-based approach where the heuristic functions are manually crafted to adjust the traffic rate in a deterministic manner. Some proposals use packet loss as a signal for network congestion, e.g., Cubic [4], Reno [24], and NewReno [6]; while others rely on the variation of delay, e.g., Vegas [5], or combine packet loss and delay [25, 26]. Different CC techniques specialized for datacenter networks are also proposed [3, 27].

Instruction: Make the last sentence slightly shorter.

Instruction: Make the paragraph slightly shorter.


In this work we try to address that gap, and study the suitability of different state space topological properties as completeness thresholds for cost optimal planning with actions with 0-cost. We identify the sublist diameter as a completeness threshold, which has the advantage of being practically bounded. We also identify a new topological property, the subset diameter , as a completeness threshold  and show that no tighter completeness threshold can be computed for a given problem without exploiting cost information, the ini- tial state, or the goal. To test the practical utility of the completeness thresholds we found, we devise a SAT compilation for cost optimal planning, and use that in an any-time planning as satisﬁability algorithm, where the horizon is ﬁxed from the beginning to the completeness threshold. This algorithm starts with an upper bound on the total cost and improves that cost upper bound every iteration. Experiments show that the algorithm is able to compute plans with costs better than the initial costs, and in many cases it can compute plans whose cost matches the optimal cost. Furthermore, the algorithm is able to prove the optimality of certain costs for a number of instances, some of which could not be proven optimal by the widely used LM-cut (Pommerening and Helmert 2012) planning heuristic.

In this work we try to address that gap, and study the suitability of different state space topological properties for being completeness thresholds for cost optimal planning with actions with 0-cost. We identify a completeness threshold that can be practically bounded, and show that no tighter completeness threshold can be computed for a given problem without exploiting cost information, the initial state, or the goal. To test the practical utility of this completeness threshold, we devise a SAT compilation for cost optimal planning, and use that in an any-time planning as satisﬁabil- ity algorithm, where the horizon is ﬁxed from the beginning to the completeness threshold. This algorithm starts with an upper bound on the total cost and improves that cost upper bound every iteration. Experiments show that the algorithm is able to compute plans with costs better than the initial costs, and in many cases it can compute plans whose cost matches the optimal cost. Furthermore, the algorithm is able to prove the optimality of certain costs for a number of instances, some of which could not be proven optimal by the widely used LM-cut planning heuristic.

Instruction: Shorten this paragraph.

Instruction: Make the beginning of this paragraph shorter.


Discussion . In this work, different kinds of objects are handled without being distinguished. We havenot used the category labels that may help the model better fit multi-class data. How to incorporatethe unified model with category labels should be further studied. In practical applications, the normalsamples are not as consistent as those in MVTec-AD. Therefore, the ability to deal with the scenarioswhere the normal samples share some diversity is important. Our UniAD is capable of handling all15 categories in MVTec-AD, hence would be more suitable for real scenes.

Discussion . In this work, different kinds of objects are handled without being distinguished. We havenot used the category labels that may help the model better fit multi-class data. How to incorporatethe unified model with category labels should be further studied. In practical uses, normal samples arenot as consistent as those in MVTec-AD, often manifest themselves in some diversity. Our UniADcould handle all 15 categories in MVTec-AD, hence would be more suitable for real scenes. However,anomaly detection may be used for video surveillance, which may infringe personal privacy.


We show how induced matchings guarantee the spatially correct matching between barcodes in a segmentation setting. Furthermore, we propose an efficient algorithm to compute TopoMatch for images. We show that TopoMatch is an interpretable metric to evaluate the topological correctness of segmentations. Moreover,we demonstrate how induced matchings can be used to train segmentation networks and improve the topological correctness of the segmentations across all 6 baseline datasets while preserving  volumetricsegmentation performance.

We show how induced matchings guarantee the spatially correct matching between barcodes in a segmentation setting. Furthermore, we propose an efficient algorithm to compute TopoMatch for images. We show that TopoMatch is an interpretable metric to evaluate the topological correctness of segmentations, which is more sensitive than the well-established Betti number error. Moreover, the differentiability of the TopoMatch loss enables its use as a loss function. It improves the topological performance of segmentation networks across six diverse datasets while preserving the volumetric performance.


In summary, our contributions include: 1) proposing a novel ( σ, δ ) -measure to quantify the data augmentation; 2) proposing a theoretical framework for contrastive SSL, which suggests that  alignment, divergence, and concentration are key factors of generalization ability; 3) provably verifying that not only the InfoNCE loss but also the cross-correlation loss satisfy the alignment and divergence; 4) empirically showing that the concentration w.r.t. the proposed augmented distance is highly related to the downstream performance.

In summary, our contributions include: 1) proposing a novel ( σ, δ ) -measure to quantify data augmentation; 2) presenting a theoretical framework for contrastive SSL that highlights alignment, divergence, and concentration as key factors for generalization ability; provably verifying that not only the InfoNCE loss but also the cross-correlation loss satisfy alignment and divergence; 4) showing a strong correlation between downstream performance and concentration of augmented data.

Instruction: Make the sentence precise.

Instruction: Improve english in this text.


We observed the main effect of W ( F 2 , 22 = 25 . 3, p < 0 . 001, η 2 p = 0 . 967) (Figure 4 (i)). Pair-wise comparisons showed that the error rates increased as W decreased. The other parameters did not show the main effects . No significant interaction was observed.

We observed the main effect of W ( F 2 , 22 = 25 . 3, p < 0 . 001, η 2 p = 0 . 967) (Figure 4 (i)). The pair-wise comparisons showed that error rates increased with a decrease in W . The other parameters did not show the main effects. No significant interaction was observed.

Instruction: Replace some words in the paragraph

Instruction: Slightly revise for readability.


As all ﬁgures in the FAUST data set are similarly meshed and oriented, breaking the gauge equivariance in higher layers can actually be beneﬁcial. As shown in Weiler & Cesa (2019), symmetry can be broken by treating non-invariant features as invariant features as input to the ﬁnal 1 × 1 convolution. Such architectures are equivariant on lower levels, while allowing orientation sensitivity at higher layers.

As all meshes in the FAUST dataset share the same topology, breaking the gauge equivariance in higher layers can actually be beneﬁcial. As shown in (Weiler & Cesa, 2019), symmetry can be broken by treating non-invariant features as invariant features as input to the ﬁnal 1 × 1 convolution. Such architectures are equivariant on lower levels, while allowing orientation sensitivity at higher layers.

Instruction: Rephrase the paragraph

Instruction: Rephrase the first sentence.


In order to evaluate the feasibility of our suggested approach for deriving diverse sets of plans according to various existing metrics, we have implemented our approach on top of the Fast Downward planning system (Helmert 2006). The code can be made available upon request. Further, we implemented an external component, that given a set of plans and a metric returns the score of the set under that metric (Katz and Sohrabi 2019).

In order to evaluate the feasibility of our suggested approach for deriving diverse sets of plans according to various existing metrics, we have implemented our approach on top of the Fast Downward planning system (Helmert 2006). Our planners, ForbidIterative (FI) diverse planners are available as part of the collection of ForbidIterative planners (Katz, Sohrabi, and Udrea 2019a). Further, we implemented an external component, that given a set of plans and a metric returns the score of the set under that metric (Katz and Sohrabi 2019).


In practice, we do not have access to the true distribution p w . Rather, we are typically given a corpus { w p w n } N n =1 , whose instances we assume to be sampled i.i.d. from p w . The common approach to address this shortcoming is (when possible) to derive a statistical estimator (cid:98) ∆ that uses this corpus to approximate ∆ . There are two common strategies for building such estimators: Monte Carlo estimation and plug-in estimation. 3.2. M ONTE C ARLO E STIMATION Our i.i.d. assumption w.r.t. samples  w p w ∼ p w allows us to derive a Monte Carlo estimator for certain divergences. We start with the forward KL divergence—present in both ∆ → and ∆ exp :

In practice, we do not have access to the true distribution p w . Rather, we are typically given a corpus { w p w n } N n =1 , whose instances we assume to be sampled i.i.d. from p w . The common approach to address this issue is thus to derive a statistical estimator (cid:98) ∆ that uses this corpus to approximate ∆ . There are two common strategies for building such estimators: Monte Carlo and plug-in estimation. Monte Carlo Estimation. Our i.i.d. assumption w.r.t. samples in { w p w n } Nn =1 allows us to derive a Monte Carlo estimator for certain divergences. We start with the forward KL divergence:

Instruction: Fix formatting issues and simplify the wording of the paragraph.

Instruction: Fix the caplocks problem. Slightly shorten the paragraph.


Q: What is the extra computation time cost of PLOT over CoOp baseline? A: Around 10 %inference speed and 5 % training time . Despite the performance improvement, the extra computationcost is still a limitation of PLOT. Please see the detailed  analysis in the supplementary materials.

Q: What is the extra computation time cost of PLOT over CoOp baseline? A: Around 10 %inference speed and 5 % training time . Please see the detailed comparisons and analysis in thesupplementary materials.

Instruction: Remove any redundant information that is not essential for the research question answered.


Surprisingly, the baselines used to assess the performance of AutoML tools are typically only other AutoML tools but no “simple” baselines. For example, a very simple  baseline  would be to imitate the steps a human data scientist would take, so that such an approach should be at least considered as a baseline. Without such baselines, we do not learn how AutoML tools improve upon ad-hoc techniques but only how they compare relatively to each other. To our knowledge, the only work accounting for such baselines is Thornton et al. (2013) using the Exhaustive-Default (“Ex-def”) baseline, which is to take the default parametrized model that is best in a cross-validation. They also discuss a grid search, which is however not applicable in practice.

The baselines used to assess the performance of AutoML tools are often other AutoML tools or random search. A simple but perhaps more sensible baseline than random search would be to imitate the steps a human data scientist would take. Without such baselines, we do not learn how AutoML tools improve upon ad-hoc techniques but only how they compare relatively to each other. To our knowledge, the only work accounting for such baselines is (Thornton et al., 2013), using the Exhaustive-Default (“Ex-def”) baseline which is to take the default parametrized model that is best in a cross-validation. They also discuss a grid search, which is however not applicable in practice.

Instruction: Edit some formulations to sound more neutral.

Instruction: Make the beginning of the paragraph shorter.


Quantization effect In Appendix 4.6, we also study how the performance of DAT is robust to gradient quantization. We ﬁnd that when the number of bits is reduced from 32 to 8 , the resulting TA and RA becomes slightly worse than the best 32 -bit case. For example, in the worst case  of CIFAR-10, TA drops 0 . 91% and 6 . 33% for DAT-PGD and DAT-FGSM, respectively. And RA drops 4 . 73% and 5 . 22% , respectively. However, the use of quantization reduces the amount of data transmission per iteration. We also show that if a high performance computing cluster of nodes (with NVLink high-speed GPU interconnect (Foley & Danskin, 2017)) is used, the communication cost can be further reduced.

Quantization effect In Appendix 4.6, we also study how the performance of DAT is affected by gradient quantization. We ﬁnd that when the number of bits is reduced from 32 to 8 , the resulting TA and RA becomes worse than the best 32 -bit case. For example, in the worst case (8-bit 2-sided quantization) of CIFAR-10, TA drops 1 . 52% and 6 . 32% for DAT-PGD and DAT-FGSM, respectively. And RA drops 4 . 74% and 5 . 58% , respectively. Note that our main communication conﬁguration is given by Ring-AllReduce that calls for 1-sided (rather than 2-sided) quantization. We also observe that DAT-FGSM is more sensitive to effect of gradient quantization than DAT-PGD. Even in the centralized setting, the use of 8-bit quantization can lead to a non-trivial drop in TA (see Table A5). However, the use of quantization reduces the amount of data transmission per iteration. We also show that if a high performance computing cluster of nodes (with NVLink high-speed GPU interconnect (Foley & Danskin, 2017)) is used, the communication cost can be further reduced.


There was a main effect  in  Position, and Position = Inside has a longer movement time than  Position = Outside . We observed a significant interaction of I × Position . At I = 0 , the movement time increased compared to  the condition  no notch by approxi- mately 11.8% in Position = Inside , and  approximately 4.93% in Position = Outside .

Another effect was observed, in that Position = Inside had a longer movement time than that for Position = Outside . We observed a significant interaction of I × Position . At I = 0 , the movement time increased compared to that for the condition of no notch by approximately 11.8% in Position = Inside , and by approximately 4.93% in Position = Outside .

Instruction: Modify the first sentence

Instruction: Revise this text to make it more readable and direct.


• The shape of the loss function after spontaneous symmetry breaking has the same shape observed by Goodfellow et al. (2014) towards the end of training, see Figure 1. • A cyclical learning rate (Smith & Topin, 2017) helps to get to the new minimum faster, see Section 2.5. • Stochasticity in gradient descent juggles the loss function such that the weights are no longer at the local maximum of Figure 1. A gradient descent step is taken to further take the weights towards the local minimum. Stochasticity helps the network to generalize better. • When the learning rate is too small to move away from A in Figure 1. Non-linearities move the weight away from A , this corresponds to breaking the symmetry explicitly in Theorem 1. PReLU’s (He et al., 2015b)performance could be related to the optimization of this process. • Results from Shwartz-Ziv & Tishby (2017) are due to spontaneous symmetry breaking, see Section 4. • Identity mapping outperforms other skip connections (He et al., 2016) is a result of the residual unit’s output being small. Then the residual units can be decoupled leading to a small λ and so it is easier for spontaneous symmetry breaking to occur, from m 2 = − µ 2 + 14 λη 2 . • Skip connection across residual units breaks additional symmetry. Suppose now an identity skip connection connects x 1 and the output of F 2 . Now perform a symmetry transformation on x 1 and x 2 , Q 1 and Q 2 ∈ G , respectively. Then the output after two residual untis is Q x = Q 1 x 1 + Q 2 x 2 + Q 2 F 2 . Neither Q = Q 1 nor Q = Q 2 can satisﬁes the covariance under G . This is observed by Orhan & Pitkow (2017). • The shattered gradient problem (Balduzzi et al., 2017). It is observed that the gradient in deep (non-residual) networks is very close to white noise. This is reﬂected in the exponential in Equation (6). This effect on ResNet is reduced because of the decoupling limit λ → 0 . This leads to the weight eigenvalues m 2 being larger in non-residual networks owing to m 2 = − µ 2 + 14 λη 2 . And so a higher oscillation frequency in the correlation function. • In recurrent neural networks, multiplicative gating (Yuhuai et al., 2016) combines the input x and the hidden state h by an element-wise product. Their method outperforms the method with an addition x + h because the multiplication gating breaks the covariance of the output. A transformation Q x ∗ Q h (cid:54) = Q ( x ∗ h ) , whereas for addition the output remains covariant Q x + Q h = Q ( x + h ) .

• The shape of the loss function after spontaneous symmetry breaking has the same shape observed by Goodfellow et al. (2014) towards the end of training, see Figure 1. • The training error typically drops drastically when learning rate is decreased. This occurs when the learning rate drops below η c , forcing a phase transition so that new minima develop. See Figure 1. • A cyclical learning rate (Smith & Topin, 2017) helps to get to the new minimum faster, see Section 2.5. • Stochasticity in gradient descent juggles the loss function such that the weights are no longer at the local maximum of Figure 1. A gradient descent step is taken to further take the weights towards the local minimum. Stochasticity helps the network to generalize better. • When the learning rate is too small to move away from A in Figure 1. PReLU’s (He et al., 2015b) could move the weight away from A through the training of the non-linearity. This corresponds to breaking the symmetry explicitly in Theorem 1. • Results from Shwartz-Ziv & Tishby (2017) are due to spontaneous symmetry breaking, see Section 4. • Identity mapping outperforms other skip connections (He et al., 2016) is a result of the residual unit’s output being small. Then the residual units can be decoupled leading to a small λ and so it is easier for spontaneous symmetry breaking to occur, from m 2 = − µ 2 + 14 λη 2 . • Skip connection across residual units breaks additional symmetry. Suppose now an identity skip connection connects x 1 and the output of F 2 . Now perform a symmetry transformation on x 1 and x 2 , Q 1 and Q 2 ∈ G , respectively. Then the output after two residual untis is Q x = Q 1 x 1 + Q 2 x 2 + Q 2 F 2 . Neither Q = Q 1 nor Q = Q 2 can satisﬁes the covariance under G . This is observed by Orhan & Pitkow (2017). • The shattered gradient problem (Balduzzi et al., 2017). It is observed that the gradient in deep (non-residual) networks is very close to white noise. This is reﬂected in the exponential in Equation (6). This effect on ResNet is reduced because of the decoupling limit λ → 0 . This leads to the weight eigenvalues m 2 being larger in non-residual networks owing to m 2 = − µ 2 + 14 λη 2 . And so a higher oscillation frequency in the correlation function. • In recurrent neural networks, multiplicative gating (Yuhuai et al., 2016) combines the input x and the hidden state h by an element-wise product. Their method outperforms the method with an addition x + h because the multiplication gating breaks the covariance of the output. A transformation Q x ∗ Q h (cid:54) = Q ( x ∗ h ) , whereas for addition the output remains covariant Q x + Q h = Q ( x + h ) .


Practically, the existing methods to compute the recurrence diameter have a doubly exponential worst case running time (Kroening and Strichman 2003; Abdulaziz and Berger 2021), and they are only useful when applied to small abstractions in the context of compositionally computing upper bounds on other topological properties. Furthermore, there is not a compositional algorithm that can compute upper bounds on the recurrence diameter using abstractions’ recurrence diameters (Abdulaziz 2017)[Chapter 3, Theorem 2]. Accordingly,  due to this absence of a practical way to compute it or tightly bound it, the recurrence diameter cannot be practically used as a completeness threshold.

Practically, the existing methods to compute the recurrence diameter have a doubly exponential worst case running time (Kroening and Strichman 2003; Abdulaziz and Berger 2021), and they are only useful when applied to small abstractions in the context of compositionally computing upper bounds on other topological properties. Furthermore, there is not a compositional algorithm that can compute upper bounds on the recurrence diameter using abstractions recurrence diameter. Accordingly, the recurrence diameter cannot be practically used as a completeness threshold due to the absence of a practical way to compute it or tightly bound it.

Instruction: I do not need references in the in second sentence. Rephrase the last sentence.

Instruction: Remove the references in the second half of the paragraph. Reorder the last sentence to improve readability.


To remedy this, some previous work has investigated the addition of diﬀerentiable stack data structures to RNNs (Sun et al., 1995; Grefenstette et al., 2015; Joulin & Mikolov, 2015; DuSell & Chiang, 2020). Just as adding a stack to a ﬁnite state machine, which makes it a pushdown automaton (PDA), enables it to recognize context-free languages (CFLs), the hope is that adding stacks to RNNs will increase the range of problems on which they can be used eﬀectively. We also expect stacks to aid training by introducing an inductive bias for learning hierarchical patterns, and to increase generalization power by structuring the model’s memory in a way that better predicts held-out hierarchical data.

To remedy this, some previous work has investigated the addition of diﬀerentiable stack data structures to RNNs (Sun et al., 1995; Grefenstette et al., 2015; Joulin & Mikolov, 2015; DuSell & Chiang, 2020), which is closely related to work on neural networks that model shift-reduce parsers (Bowman et al., 2016; Dyer et al., 2016; Shen et al., 2019a). Just as adding a stack to a ﬁnite state machine, which makes it a pushdown automaton (PDA), enables it to recognize context-free languages (CFLs), the hope is that adding stacks to RNNs will increase the range of problems on which they can be used eﬀectively. We also expect stacks to aid training by introducing an inductive bias for learning hierarchical patterns, and to increase generalization power by structuring the model’s memory in a way that better predicts held-out hierarchical data.


Delayed augmentation . We experiment on the generalization when we start to use augmentationlately as 10M, 20M. As shown in Figure 2(d) and Figure 2(e), the generalization rapidly increases after using augmentation at 10M and 20M. Although we use augmentation lately, the augmentation helps the generalization regardless of the usage timing. Golatkar et al . [9] shows that delayed augmentation cannot achieve as much as using augmentation during whole training in supervised-learning. However, (10, 25) improves the generalization  comparable with  (0, 25), which use augmentation throughouttraining, unlike supervised learning. However, when augmentation noticeably helps the training, suchas Figure 2(e), delayed augmentation struggles to follow earlier one in Figure 2(f), because the RLgradually improves the policy and trajectory by Markov property. Furthermore, RL has a limitednumber of samples unlike supervised learning, so using augmentation from the initial time is morecritical than supervised learning if augmentation helps the training.

Delayed augmentation . To determine when we start to use augmentation, we delayed its use until after 10M or 20M steps. The generalization rapidly increases after using augmentation at 10M and 20M (Figre 2(d), 2(e)). Although we impose augmentation late, the augmentation helps the generalization regardless of the start timing. In SL, delayed augmentation cannot achieve as much as using augmentation during whole training [9]. However, (10, 25) improves the generalization to be comparable with that of (0, 25), which use augmentation throughout training; this result differs from the case of supervised learning. However, when augmentation noticeably helps the training, the performance achieved using delayed augmentation may not catch up (Figure 2(e)) to the performance achieved using early augmentation (Figure 2(f)), because the RL gradually improves the policy and trajectory, as a result of its Markov property. Furthermore, the number of samples is limited for RL, but not for supervised learning, so using augmentation from the initial time is more critical than supervised learning if augmentation helps the training.

Instruction: Use clearer expression, use concise words.

Instruction: Revise this paragraph to use clearer and more precise words.