In this task, you are given a part of an article. Your task is to generate headline (title) for this text. Preferred headlines are under fifteen words.

Q: Common activation functions used in neural networks can yield to training difficulties due to the saturation behavior of the activation function, which may hide dependencies which are not visible to first order (using only gradients). Gating mechanisms that use softly saturating activation functions to emulate the discrete switching of digital logic circuits are good examples of this. We propose to exploit the injection of appropriate noise so that some gradients may sometimes flow, even if the noiseless application of the activation function would yield zero gradient. Large noise will dominate the noise-free gradient and allow stochastic gradient descent to be more exploratory. By adding noise only to the problematic parts of the activation function we allow the optimization procedure to explore the boundary between the degenerate (saturating) and the well-behaved parts of the activation function. We also establish connections to simulated annealing, when the amount of noise is annealed down, making it easier to optimize hard objective functions. We find experimentally that replacing such saturating activation functions by by noisy variants helps training in many contexts, yielding state-of-the-art results on several datasets, especially when training seems to be the most difficult, e.g., when curriculum learning is necessary to obtain good results.

A: Noisy Activation Functions
****
Q: Fixed-vocabulary language models fail to account for one of the most characteristic statistical facts of natural language: the frequent creation and reuse of new word types. Although character-level language models offer a partial solution in that they can create word types not attested in the training corpus, they do not capture the "bursty" distribution of such words. In this paper, we augment a hierarchical LSTM language model that generates sequences of word tokens character by character with a caching mechanism that learns to reuse previously generated words. To validate our model we construct a new open-vocabulary language modeling corpus (the Multilingual Wikipedia Corpus; MWC) from comparable Wikipedia articles in 7 typologically diverse languages and demonstrate the effectiveness of our model across this range of languages.

A: Learning to Create and Reuse Words in Open-Vocabulary Neural Language Modeling
****
Q: <lb>The minimization of the logistic loss is a popular approach to batch supervised learning. Our<lb>paper starts from the surprising observation that, when fitting linear (or kernelized) classifiers,<lb>the minimization of the logistic loss is equivalent to the minimization of an exponential rado-loss<lb>computed (i) over transformed data that we call Rademacher observations (rados), and (ii) over<lb>the same classifier as the one of the logistic loss. Thus, a classifier learnt from rados can be<lb>directly used to classify observations. We provide a learning algorithm over rados with boosting-<lb>compliant convergence rates on the logistic loss (computed over examples). Experiments on<lb>domains with up to millions of examples, backed up by theoretical arguments, display that<lb>learning over a small set of random rados can challenge the state of the art that learns over<lb>the complete set of examples. We show that rados comply with various privacy requirements<lb>that make them good candidates for machine learning in a privacy framework. We give several<lb>algebraic, geometric and computational hardness results on reconstructing examples from rados.<lb>We also show how it is possible to craft, and efficiently learn from, rados in a differential privacy<lb>framework. Tests reveal that learning from differentially private rados can compete with learning<lb>from random rados, and hence with batch learning from examples, achieving non-trivial privacy<lb>vs accuracy tradeoffs.

A:
Rademacher Observations, Private Data, and Boosting
****