In this task, you are given the abstract of a research paper. Your task is to generate a summary of this abstract. Your summary should not be very short, but it's better if it's not more than 30 words.
--------
Question: Asynchronous distributed gradient descent algorithms for training of deep neural
networks are usually considered as inefficient, mainly because of the Gradient delay
problem. In this paper, we propose a novel asynchronous distributed algorithm
that tackles this limitation by well-thought-out averaging of model updates, computed
by workers. The algorithm allows computing gradients along the process
of gradient merge, thus, reducing or even completely eliminating worker idle time
due to communication overhead, which is a pitfall of existing asynchronous methods.
We provide theoretical analysis of the proposed asynchronous algorithm,
and show its regret bounds. According to our analysis, the crucial parameter for
keeping high convergence rate is the maximal discrepancy between local parameter
vectors of any pair of workers. As long as it is kept relatively small, the
convergence rate of the algorithm is shown to be the same as the one of a sequential
online learning. Furthermore, in our algorithm, this discrepancy is bounded
by an expression that involves the staleness parameter of the algorithm, and is
independent on the number of workers. This is the main differentiator between
our approach and other solutions, such as Elastic Asynchronous SGD or Downpour
SGD, in which that maximal discrepancy is bounded by an expression that
depends on the number of workers, due to gradient delay problem. To demonstrate
effectiveness of our approach, we conduct a series of experiments on image
classification task on a cluster with 4 machines, equipped with a commodity communication
switch and with a single GPU card per machine. Our experiments
show a linear scaling on 4-machine cluster without sacrificing the test accuracy,
while eliminating almost completely worker idle time. Since our method allows
using commodity communication switch, it paves a way for large scale distributed
training performed on commodity clusters.

Answer: A method for an efficient asynchronous distributed training of deep learning models along with theoretical regret bounds.

Question: We present Cross-View Training (CVT), a simple but effective method for deep semi-supervised learning. On labeled examples, the model is trained with standard cross-entropy loss. On an unlabeled example, the model first performs inference (acting as a "teacher") to produce soft targets. The model then learns from these soft targets (acting as a ''"student"). We deviate from prior work by adding multiple auxiliary student prediction layers to the model. The input to each student layer is a sub-network of the full model that has a restricted view of the input (e.g., only seeing one region of an image). The students can learn from the teacher (the full model) because the teacher sees more of each example. Concurrently, the students improve the quality of the representations used by the teacher as they learn to make predictions with limited data. When combined with Virtual Adversarial Training, CVT improves upon the current state-of-the-art on semi-supervised CIFAR-10 and semi-supervised SVHN. We also apply CVT to train models on five natural language processing tasks using hundreds of millions of sentences of unlabeled data. On all tasks CVT substantially outperforms supervised learning alone, resulting in models that improve upon or are competitive with the current state-of-the-art.

Answer: Self-training with different views of the input gives excellent results for semi-supervised image recognition, sequence tagging, and dependency parsing.

Question: Modern neural networks often require deep compositions of high-dimensional nonlinear functions (wide architecture) to achieve high test accuracy, and thus can have overwhelming number of parameters. Repeated high cost in prediction at test-time makes neural networks ill-suited for devices with constrained memory or computational power. We introduce an efficient mechanism, reshaped tensor decomposition, to compress neural networks by exploiting three types of invariant structures: periodicity, modulation and low rank. Our reshaped tensor decomposition method exploits such invariance structures using a technique called tensorization (reshaping the layers into higher-order tensors) combined with higher order tensor decompositions on top of the tensorized layers. Our compression method improves low rank approximation methods and can be incorporated to (is complementary to) most of the existing compression methods for neural networks to achieve better compression. Experiments on LeNet-5 (MNIST), ResNet-32 (CI- FAR10) and ResNet-50 (ImageNet) demonstrate that our reshaped tensor decomposition outperforms (5% test accuracy improvement universally on CIFAR10) the state-of-the-art low-rank approximation techniques under same compression rate, besides achieving orders of magnitude faster convergence rates.

Answer:
Compression of neural networks which improves the state-of-the-art low rank approximation techniques and is complementary to most of other compression techniques.