sadat2307/MSciNLI huggingface dataset:
Domain: NeurIPS
Sentence 1: We make three contributions towards this goal: (i) we propose a novel recurrent neural network architecture that can simultaneously predict the responses of thousands of cortical neurons while accounting for neural variability caused by eye movements or brain state changes related to measurable factors such as running state or pupil dilations; (ii) we demonstrate that training our model on natural movies allows some extent of domain transfer and recovers neurons' tuning properties, such as orientation tuning, direction tuning, and receptive field structure, under artificial stimuli; and (iii) we analyze the limits of domain transfer to show that models trained directly on the target domain always outperform those trained on other domains in terms of predictive performance.
Sentence 2: we also demonstrate that the nonlinear feature representation learned on natural movies can transfer well to the noise domain by fine-tuning the last layer's weights, while the converse is not true (i. e. generalization from noise to movies).
Relationship: contrasting
Domain: NeurIPS
Sentence 1: This can be paired with a loss that maximises the entropy of the average distribution across each batch: L diversity = −H(q) = i qi log qi .
Sentence 2: we found that using a ReLU nonlinearity [33] followed by divisive normalisation 3 , paired with an L 2 diversity loss, L diversity = (k • q − 1) 2 , tends to converge more robustly.
Relationship: contrasting
Domain: NeurIPS
Sentence 1: Consider a formula ¬R(x, y) ∨ S(y, z) where each variable has domain-size equal to d. The data-complexity of computing the satisfied groundings in this formula is clearly d 3 .
Sentence 2: for any specific value of y, say y = A, the satisfied groundings in this formula can be computed in closed form as, n 1 d + n 2 d − n 1 n 2 , where n 1 is the number of false groundings of R(x, A) and n 2 is the number of true groundings in S(A, z).
Relationship: contrasting
Domain: NeurIPS
Sentence 1: Choosing a MAP state as the starting point has two advantages: 1) we are guaranteed that x M AP is an interior point and 2) it is the point with the highest probability density and therefore highest probability mass in a small local neighborhood.
Sentence 2: starting from a MAP state elevates the importance of the second question, since the MAP state often lies exactly on the boundary of the polytope and therefore we are likely to start the sampling algorithm from a vertex of the polytope.
Relationship: contrasting
Domain: NeurIPS
Sentence 1: The two key findings in this work are that 1) the SGLD algorithm converges weakly to the target distribution π, 2) the optimal rate of convergence to equilibrium scales as n −1/3 where n is the number of iterations, see [31,Section 5].
Sentence 2: in most of the applications, constant rather than decreasing step sizes are used, see [1,8,18,22,30,32].
Relationship: contrasting
Domain: NeurIPS
Sentence 1: In particular, we have DM = L 2 max ξ(κ(F ) √ rRQ max ), ξ(z) = z 2 + z, where L max denotes the maximal value of all elements in L, κ(F ) is the condition number of the matrix F and r in an upper bound on the rank of F. The constants R and Q max come from the kernel ASGD setup and, importantly, depend only on the data distribution, but not on the loss L or score matrix F . Note that for a given subspace F, the choice of matrix F is arbitrary and it can always be chosen as an orthonormal basis of F giving a κ(F ) of one.
Sentence 2: such F can lead to inefficient prediction (1), which makes the whole framework less appealing.
Relationship: contrasting
Domain: NeurIPS
Sentence 1: In such cases, we can still apply the Gaussian copula (and other copulas derived from elliptically symmetric distributions), since it is parametrized by the same covariance matrix as a n-dimensional Gaussian.
Sentence 2: the Gaussian copula becomes prohibitively expensive to fit in high dimensions, since evaluating the likelihood requires an exponential number of evaluations of the multivariate Gaussian cdf, which itself must be computed numerically.
Relationship: contrasting
Domain: NeurIPS
Sentence 1: for x ∈ R and z ∈ R, the variational distribution q θ (z) = N (z|θ, 1) for θ ∈ R, and its reparameterization f θ ( ) = + θ and q( ) = N ( |0, 1) for ∈ R. For an observed value x 0 = 0, the joint density 1), where c 1 = N (0|5, 1) and c 2 = N (0|−2, 1).
Sentence 2: notice that r is non-differentiable only at z = 0 and {0} is a null set in R. For any θ, ∇ θ ELBO θ is computed as follows: Since log(r(z)/q θ (z RepGrad θ is computed as follows: After reparameterizing z into , we have and we obtain RepGrad θ = −θ.
Relationship: contrasting
Domain: NeurIPS
Sentence 1: The main culprit here is that interchanging differentiation and integration is sometimes invalid: for The RepGrad θ term in Theorem 1 can be easily estimated by the standard Monte Carlo: We write RepGrad θ for this estimate.
Sentence 2: estimating the other BouContr θ term is not that easy, because of the difficulties in estimating surface integrals in the term.
Relationship: contrasting
Domain: NeurIPS
Sentence 1: • Add a value: Choose a rule r m ∈ R [t] , a condition c k ∈ r m and then a candidate value • Remove a condition: Choose a rule r m ∈ R [t] and a condition c k ∈ r m , then where we use ν(•) to access the feature in a condition.
Sentence 2: if the misclassified example is negative, it means R [t] covers more than it should and therefore needs to reduce the coverage by randomly choosing one of the following actions.
Relationship: contrasting
Domain: NeurIPS
Sentence 1: Given the prominence of Nash equilibrium as a solution concept in game theory (compared to coarser notions such as correlated equilibria or the Hannan set), this problem lies at the heart of multi-agent learning [4].
Sentence 2: convergence to a Nash equilibrium is, in the words of [4], "considerably more difficult" than attaining a no-regret state for all players (which leads to weaker notion of coarse correlated equilibrium in finite games).
Relationship: contrasting
Domain: NeurIPS
Sentence 1: At these higher values, nearly all discriminative information in the question encoding is lost -with the VQA model sacrificing its own performance to lower that of the question-only model.
Sentence 2: we observe that for reasonable values of λ H , the strength of the question-only adversary can be varied over a much wider range with less dramatic losses (blue curve in Figure 2).
Relationship: contrasting
Domain: NeurIPS
Sentence 1: As in [LS05], noise tolerance is facilitated by the fact that the path through the network is not affected by altering the label of an example.
Sentence 2: balancing the distribution before passing it to the weak learner, which was needed to use a standard weak learner, may disturb the independence between the event that an example is noisy, and the random draw of x.
Relationship: contrasting
Domain: NeurIPS
Sentence 1: It is worth pointing out that the proof will hold for a larger class of priors on K, as long as the prior shrinks at least exponentially fast (meaning that it is bounded from above by ae −bK for constants a, b > 0).
Sentence 2: a prior at this exponential limit will require tuning, because the optimal a and b will depend on K(f 0 , Ω EB ).
Relationship: contrasting
Domain: NeurIPS
Sentence 1: A straightforward idea is using the optimized decisions under certain penalty to supervise the decision network.
Sentence 2: for a backbone CNN with m layers, the time complexity of collecting the supervised signal is O( m i=1 n m ), which is NP-hard and unacceptable for prevalent very deep architecture such as VGG [37] and ResNet [3].
Relationship: contrasting
Domain: NeurIPS
Sentence 1: For one specific dataset D, a natural way to quantify support would be the posterior probability of a model given the data, Pr(θ|D).
Sentence 2: randomly generating a large number of datasets so as to approximate the expected value of Pr(θ|D) over all datasets, in the spirit of previous work on model identifiability [23], becomes intractable for complex models such as ours.
Relationship: contrasting
Domain: NeurIPS
Sentence 1: Alternating optimization algorithms have been used in both of the early methods [2,14] to solve Problem (1) which alternate between solving two subproblems: solve for β t with fixed c; solve for c with fixed β t . The convergence property of such an alternating algorithm has been analyzed in [2] that it converges to a local minimizer.
Sentence 2: both subproblems in the existing methods can only be solved using iterative algorithms such as gradient descent, linear or quadratic program solvers.
Relationship: contrasting
Domain: NeurIPS
Sentence 1: During dropout, the error of any classifier h is characterized by two measures.
Sentence 2: in the end, we are interested in the usual generalization error (expected risk) of h where x is drawn from the underlying data-generating measure: Err (h) since dropout training works on the corrupted data x (see (3)), in the limit of infinite data, the dropout estimator will converge to the minimizer of the generalization error with respect to the dropout measure over x: Err (h) The main difficulty in analyzing the generalization of dropout is that classical theory tells us that the generalization error with respect to the dropout measure will decrease as n !
Relationship: contrasting
Domain: NeurIPS
Sentence 1: Recent years have seen significant progress in using machine learning to reason with relational data [16], especially within the context of knowledge graph embeddings [6,23,18,28,29,45], probabilistic soft logic [3], and differentiable tensor-based logic [11,33].
Sentence 2: existing work in this area primarily focuses on using logical reasoning to improve edge prediction in knowledge graphs [14,13,27], for example, by using logical rules as regularization [15,20,35,37].
Relationship: contrasting
Domain: NeurIPS
Sentence 1: The memory layers are stacked in the following way: • The input to layers above the first is the sum of the output o k and the input u k from layer k (different ways to combine o k and u k are proposed later): • Each layer has its own embedding matrices A k , C k , used to embed the inputs {x i }.
Sentence 2: as discussed below, they are constrained to ease training and reduce the number of parameters.
Relationship: contrasting
Domain: NeurIPS
Sentence 1: While NMT with beam search has been proved to be successful, it has several obvious issues, including exposure bias [9], loss-evaluation mismatch [9] and label bias [16], which have been studied.
Sentence 2: we observe that there is still an important issue associated with beam search of NMT, the myopic bias, which unfortunately is largely ignored, to the best of our knowledge.
Relationship: contrasting
Domain: NeurIPS
Sentence 1: The DDM and DEPC variants, further, replicate the finding that the buildup of neural firing depends approximately linear on coherence (normalised mean square error of a corresponding linear model was 0.04 and 0.03, respectively).
Sentence 2: the CONST variant exhibited an inverse relation between coherence and buildup of predicted neural response, i.e., buildup was larger for small coherences.
Relationship: contrasting
Domain: NeurIPS
Sentence 1: They are related to Hopfield networks, which have been used in the context of models of hallucinations before [11].
Sentence 2: whereas the latter model some abstract memory system, DBMs (and the related Deep Belief Nets) learn hierarchical representations of data [7], thus capturing aspects of the visual cortex [12], the locus where visual hallucinations are ultimately realized [1,13].
Relationship: contrasting
Domain: NeurIPS
Sentence 1: Until recently, neural-symbolic systems were not able to fully represent, reason and learn expressive languages other than propositional and fragments of first-order logic (Cloete & Zurada, 2000).
Sentence 2: in (d'Avila Garcez et al., 2002b;d'Avila Garcez et al., 2002c;d'Avila Garcez et al., 2003), a new approach to knowledge representation and reasoning in neural-symbolic systems based on neural networks ensembles has been introduced.
Relationship: contrasting
Domain: NeurIPS
Sentence 1: So long as we use a gift value g σ (I 1 , a ) ≤ CBV σ2 (I 1 ) − CBV σ2 (I 1 , a ), the resulting strategy will be safe.
Sentence 2: using a lower bound further guarantees a reduction to exploitability when a P 1 best response reaches with positive probability an infoset I 1 ∈ S top that has positive margin, as proven in Theorem 1.
Relationship: contrasting
Domain: NeurIPS
Sentence 1: The empirical distribution provides a maximum likelihood estimate of the context distribution and the EALP and UCB-EALP algorithms achieve similar performance as ALP and UCB-ALP, respectively, as observed in numerical simulations.
Sentence 2: a rigorous analysis for EALP and UCB-EALP is much more challenging due to the dependency introduced by the empirical distribution.
Relationship: contrasting
Domain: NeurIPS
Sentence 1: Overall, this analysis yields a dynamical equivalence between mAMP and bAMP as long as at each iteration time t, the optimal loss and regularizer for mAMP are chosen through the smoothing operation in ( 18), but using time-dependent smoothing parameters λ t η and λ t h whose evolution is governed by (7).
Sentence 2: 4 Determining optimal smoothing parameters via state evolution of AMP In the previous section, we have shown that mAMP and bAMP have the same dynamics, as long as, at each iteration t of mAMP, we choose a time dependent optimal loss L opt t and regularizer σ opt t through (18), where the time dependence is inherited from the time dependent smoothing parameters λ t η and λ t h . mAMP was motivated as an algorithmic solution to the M-estimation problem in (5) for a fixed loss and regularizer, while bAMP was motivated as a method of performing the Bayesian integral in (2).
Relationship: contrasting
Domain: NeurIPS
Sentence 1: Previous works [6,13,11] generally first develop a kind of approximation (or relaxation) for QP matching problem by ignoring the doubly stochastic constraint, and then aim to find the optimum of the relaxation problem by developing an algorithm.
Sentence 2: our work focus on the general and challengeable QP problem with doubly stochastic constraint (Eq.
Relationship: contrasting
Domain: NeurIPS
Sentence 1: The visual question answering (VQA) problem is an excellent way to test such reasoning capabilities of an AI model and its multimodal representation learning.
Sentence 2: the current VQA models are oversimplified deep neural networks, comprised of a long short-term memory (LSTM) unit for question comprehension and a convolutional neural network (CNN) for learning single image representation.
Relationship: contrasting
Domain: NeurIPS
Sentence 1: A standard Algorithm 2: ADA-BARRONS 2 Run BARRONS with parameter β and η, where after each round t, if the following holds: with α t defined in Theorem 1 and then set β ← β 2 , and rerun BARRONS from Line 2 with time index reset to 1. doubling trick would suggest halving β whenever it is larger than α t (u * t ) and then restart the algorithm.
Sentence 2: since α t (u * t ) is not monotone in t, standard analysis of doubling trick does not work.
Relationship: contrasting
Domain: NeurIPS
Sentence 1: Our work can be viewed as an instance of meta learning [Andrychowicz et al., 2016, Santoro et al., 2016, Nichol and Schulman, 2018 in the sense that the intrinsic reward function module acts as a meta-learner that learns to improve the agent's objective (i.e., mixture of extrinsic and intrinsic reward) by taking into account how each gradient step of the agent affects the true objective (i.e., extrinsic reward) through the meta-gradient.
Sentence 2: a key distinction from the prior work on meta learning for RL [Finn et al., 2017, Wang et al., 2016, Duan et al., 2016b is that our method aims to meta-learn intrinsic rewards within a single task, whereas much of the prior work is designed to quickly adapt to new tasks in a few-shot learning scenario.
Relationship: contrasting
Domain: NeurIPS
Sentence 1: Previously, we assumed opp i = 0 and therefore pur i = 1.
Sentence 2: if we imagine that there is an η fraction of examples on which the two views disagree, and that positive and negative regions expand uniformly at the same rate, then even if initially opp 0 = 0, it is natural to assume the following form of increase in cov and opp: (2) We show here how Definition 2 implies Definition 1.
Relationship: contrasting
Domain: NeurIPS
Sentence 1: The sublinear convergence rate obtained with this approach is up to constants identical to that in Theorem 2 for our MP variants, see [21].
Sentence 2: as the correct scaling is unknown, one has to either take the risk of choosing τ too small and hence failing to recover an optimal solution of ( 4), or to rely on too large τ which can result in slow convergence.
Relationship: contrasting
Domain: NeurIPS
Sentence 1: Our Propagated Semantic Transfer, using the raw image descriptors to build a neighborhood structure, achieves 81.2% AUC and 40.5% Acc.
Sentence 2: when propagating on the 85-dimensional attribute space, we improve over [11] and [9] to 83.7% AUC and 42.7% Acc.
Relationship: contrasting
Domain: NeurIPS
Sentence 1: But in dimension d > 1, following the Brownian sensing approach while sampling over the whole space would require generating M Brownian sheets (extension of Brownian motions to d > 1 dimensions) over X , and then building Assuming that the covariance matrix V X is invertible, this Brownian sensing matrix is also Gaussian and enjoys the same recovery properties as in the one-dimensional case.
Sentence 2: in this case, estimating the stochastic integrals b m = � X f dB m using sensing points along a (d-dimensional) grid would provide an estimation error ε = b− � b that scales poorly with d since we integrate over a d dimensional space.
Relationship: contrasting
Domain: NeurIPS
Sentence 1: In the context of large margin classifiers, many techniques have been proposed for incorporating unlabeled data in an SVM, most of which are intuitively based on ensuring that large margins are also preserved on the unlabeled training data [8,2], just as in our case.
Sentence 2: none of these previous proposals have formulated a convex optimization procedure that was guaranteed to directly maximize the margin, as we propose in Section 3.
Relationship: contrasting
Domain: NeurIPS
Sentence 1: We know that 0 ≤ H(Π) ≤ log(|Π|) for any set of nodes Π, hence the result of Corollary 1 could be further manipulated to achieve a bound for the difference between BIC and BIC * of at most N log(min{|X|, |Π 1 |, |Π 2 |}).
Sentence 2: corollary 1 is stronger and can still be computed efficiently as follows.
Relationship: contrasting
Domain: NeurIPS
Sentence 1: That is, we maximize the data likelihood of q(x) (Eq.3) w.r.t φ, yielding the gradient: The second term involves estimating the expectation w.r.t an energy-based distribution E q(x) [•], which is in general very challenging.
Sentence 2: we can exploit the special structure of q ∝ p θ exp{αf φ } for efficient approximation.
Relationship: contrasting
Domain: NeurIPS
Sentence 1: Note that a linear convergence rate for the training cost does not translate into a similar rate for the testing cost, and an appealing propertly of SG methods is that they achieve the optimal O(1/k) rate for the testing cost as long as every datapoint is seen only once.
Sentence 2: as is common in machine learning, we assume that we are only given a finite training data set and thus that datapoints are revisited multiple times.
Relationship: contrasting
Domain: NeurIPS
Sentence 1: That work showed that the likelihood of finding a causal relationship rose with the base rate peE) in non-contingent cases, suggesting that structural decisions are a relevant part of the present data.
Sentence 2: the rating scale of the current task seems to encourage strength judgments rather, than purely structural decisions, because it is anchored at the endpoints by two qualitatively different causal strengths (strong generative, strong preventive).
Relationship: contrasting
Domain: NeurIPS
Sentence 1: First, we will require that the objective is strongly concave in the discriminator parameter space at equilibrium (note that it is concave by default).
Sentence 2: on the other hand, we cannot require the objective to be strongly convex in the generator parameter space as we saw that the objective is not convex-concave even in the nicest scenario, even arbitrarily close to equilbrium.
Relationship: contrasting
Domain: NeurIPS
Sentence 1: Let x t , 1 ≤ t ≤ N 0 be generated using a mixture of Gaussians (1) At a high level our analysis shows that at each step of the streaming updates, the error decreases on average.
Sentence 2: due to the non-convexity of the objective function we can show such a decrease only if the current estimates of our centers lie in a small ball around the true centers of the gaussians.
Relationship: contrasting
Domain: NeurIPS
Sentence 1: It is also notable that the worst-case time complexity of S-AdaNCG is worse than that of a recent algorithm called Natasha2 proposed in [2], which has a state-of-the-art time complexity of O d/ 3.5 1 for finding an ( 1 , √ 1 ) second-order stationary point.
Sentence 2: s-AdaNCG is much simpler than Natasha2, which involves many parameters and switches between several procedures.
Relationship: contrasting
Domain: NeurIPS
Sentence 1: This result guarantees that the sequence p k produced by ADMM globally converges under a mild assumption.
Sentence 2: to establish the linear convergence rate, the key lies in estimating the other side inequality, Then one can combine these two results together to prove that sequence p k converges globally and linearly with , which further can be used to show the R−linear convergence of iterates x k , y k and z k . To estimate the constant γ, we first describe the geometry formed by the optimal primal solutions x * , y * and dual solutions z * of the LP (3).
Relationship: contrasting
Domain: NeurIPS
Sentence 1: Let H be the RKHS embedded with the dot product ⟨•, •⟩ H (we abbreviate ⟨•, •⟩ H as ⟨•, •⟩ for simplicity) and a positive definite kernel k. Additionally, let ϕ : M → H. Then, we define the Koopman operator on the feature map ϕ by ) Thus, the Koopman operator K H is a linear operator in H. Note that almost of the theoretical claims in this and the next sections do not necessarily require ϕ to be in RKHS (it is sufficient that ϕ stays in a Hilbert space).
Sentence 2: this assumption should perform the calculation in practice (as described in the last parts of this and the next sections).
Relationship: contrasting
Domain: NeurIPS
Sentence 1: [15], that the bound would be 3 O(( If an upper bound U on u is known, we can use it to tune η to obtain an upper bound of the order of O(U √ T ).
Sentence 2: we obtain for DFEG a bound of O( u log( u T + 1) T ), that is optimal bound, up to logarithmic terms, without knowing U .
Relationship: contrasting
Domain: NeurIPS
Sentence 1: This is similar to the trend observed for FedAvg algorithm while reducing the Client fraction (C) which makes the (stochastic) agent interaction matrix sparser.
Sentence 2: from the plot of the variance of accuracy values over agents (a smooth version using moving average filter), it can be observed that the level of consensus is more stable for denser networks compared to that for sparser networks.
Relationship: contrasting
Domain: NeurIPS
Sentence 1: Efficient inference algorithms with provable guarantees have been proposed by applying recent advances in tensor decomposition methods [14,15], directly clustering the users [16,17], or using sampling methods [18].
Sentence 2: this mixture PL is still restrictive, and both clustering and tensor based approaches rely heavily on the fact that the distribution is a "mixture" and require additional incoherence assumptions on Θ * .
Relationship: contrasting