In this task, you are given a part of an article. Your task is to generate headline (title) for this text. Preferred headlines are under fifteen words.

Input: Consider Input: Dropout and other feature noising schemes control overfitting by artificially cor-<lb>rupting the training data. For generalized linear models, dropout performs a form<lb>of adaptive regularization. Using this viewpoint, we show that the dropout regular-<lb>izer is first-order equivalent to an L2 regularizer applied after scaling the features<lb>by an estimate of the inverse diagonal Fisher information matrix. We also establish<lb>a connection to AdaGrad, an online learning algorithm, and find that a close rel-<lb>ative of AdaGrad operates by repeatedly solving linear dropout-regularized prob-<lb>lems. By casting dropout as regularization, we develop a natural semi-supervised<lb>algorithm that uses unlabeled data to create a better adaptive regularizer. We ap-<lb>ply this idea to document classification tasks, and show that it consistently boosts<lb>the performance of dropout training, improving on state-of-the-art results on the<lb>IMDB reviews dataset.

Output: Dropout Training as Adaptive Regularization

Input: Consider Input: Sentences are important semantic units of natural language. A generic, distributional representation of sentences that can capture the latent semantics is beneficial to multiple downstream applications. We observe a simple geometry of sentences the word representations of a given sentence (on average 10.23 words in all SemEval datasets with a standard deviation 4.84) roughly lie in a low-rank subspace (roughly, rank 4). Motivated by this observation, we represent a sentence by the low-rank subspace spanned by its word vectors. Such an unsupervised representation is empirically validated via semantic textual similarity tasks on 19 different datasets, where it outperforms the sophisticated neural network models, including skip-thought vectors, by 15% on average.

Output: Representing Sentences as Low-Rank Subspaces

Input: Consider Input: Stochastic variational inference (SVI) lets us scale up Bayesian computation to massive data. It uses stochastic optimization to fit a variational distribution, following easy-to-compute noisy natural gradients. As with most traditional stochastic optimization methods, SVI takes precautions to use unbiased stochastic gradients whose expectations are equal to the true gradients. In this paper, we explore the idea of following biased stochastic gradients in SVI. Our method replaces the natural gradient with a similarly constructed vector that uses a fixed-window moving average of some of its previous terms. We will demonstrate the many advantages of this technique. First, its computational cost is the same as for SVI and storage requirements only multiply by a constant factor. Second, it enjoys significant variance reduction over the unbiased estimates, smaller bias than averaged gradients, and leads to smaller mean-squared error against the full gradient. We test our method on latent Dirichlet allocation with three large corpora.
Output: Smoothed Gradients for Stochastic Variational Inference