Query: Train longer, generalize better: closing the generalization gap in large batch training of neural networks

Relevant passages:
Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour <=SEP=> Deep learning thrives with large neural networks and large datasets. However, larger networks and larger datasets result in longer training times that impede research and development progress. Distributed synchronous SGD offers a potential solution to this problem by dividing SGD minibatches over a pool of parallel workers. Yet to make this scheme efficient, the per-worker workload must be large, which implies nontrivial growth in the SGD minibatch size. In this paper, we empirically show that on the ImageNet dataset large minibatches cause optimization difficulties, but when these are addressed the trained networks exhibit good generalization. Specifically, we show no loss of accuracy when training with large minibatch sizes up to 8192 images. To achieve this result, we adopt a linear scaling rule for adjusting learning rates as a function of minibatch size and develop a new warmup scheme that overcomes optimization challenges early in training. With these simple techniques, our Caffe2-based system trains ResNet50 with a minibatch size of 8192 on 256 GPUs in one hour, while matching small minibatch accuracy. Using commodity hardware, our implementation achieves ∼90% scaling efficiency when moving from 8 to 256 GPUs. This system enables us to train visual recognition models on internetscale data with high efficiency.

RMSProp and equilibrated adaptive learning rates for non-convex optimization <=SEP=> Parameter-specific adaptive learning rate methods are computationally efficient ways to reduce the ill-conditioning problems encountered when training large deep networks. Following recent work that strongly suggests that most of the critical points encountered when training such networks are saddle points, we find how considering the presence of negative eigenvalues of the Hessian could help us design better suited adaptive learning rate schemes. We show that the popular Jacobi preconditioner has undesirable behavior in the presence of both positive and negative curvature, and present theoretical and empirical evidence that the socalled equilibration preconditioner is comparatively better suited to non-convex problems. We introduce a novel adaptive learning rate scheme, called ESGD, based on the equilibration preconditioner. Our experiments show that ESGD performs as well or better than RMSProp in terms of convergence speed, always clearly improving over plain stochastic gradient descent.

Understanding deep learning requires rethinking generalization <=SEP=> Despite their massivesize, successful deep artificial neural networkscan exhibit a remarkably small differencebetween training and test performance. Conventional wisdom attributessmall generalization error either to propertiesof themodel family, or to the regularization techniquesused during training. Through extensive systematic experiments, we show how these traditional approaches fail to explain why large neural networks generalize well in practice. Specifically, our experimentsestablish that state-of-the-art convolutional networks for image classification trained with stochastic gradient methods easily fit a random labeling of the training data. This phenomenon is qualitatively unaffected by explicit regularization, and occurs even if we replace the true images by completely unstructured random noise. We corroborate these experimental findings with a theoretical construction showing that simpledepth two neural networksalready haveperfect finitesampleexpressivity assoon as thenumber of parameters exceeds thenumber of datapointsas it usually does in practice. We interpret our experimental findingsby comparison with traditional models.

Deep Speech 2 : End-to-End Speech Recognition in English and Mandarin <=SEP=> We show that an end-to-end deep learning approach can be used to recognize either English or Mandarin Chinese speech—two vastly different languages. Because it replaces entire pipelines of hand-engineered components with neural networks, end-to-end learning allows us to handle a diverse variety of speech including noisy environments, accents and different languages. Key to our approach is our application of HPC techniques, resulting in a 7x speedup over our previous system [26]. Because of this efficiency, experiments that previously took weeks now run in days. This enables us to iterate more quickly to identify superior architectures and algorithms. As a result, in several cases, our system is competitive with the transcription of human workers when benchmarked on standard datasets. Finally, using a technique called Batch Dispatch with GPUs in the data center, we show that our system can be inexpensively deployed in an online setting, delivering low latency when serving users at scale.

An overview of gradient descent optimization algorithms <=SEP=> Gradient descent optimization algorithms, while increasingly popular, are often used as black-box optimizers, as practical explanations of their strengths and weaknesses are hard to come by. This article aims to provide the reader with intuitions with regard to the behaviour of different algorithms that will allow her to put them to use. In the course of this overview, we look at different variants of gradient descent, summarize challenges, introduce the most common optimization algorithms, review architectures in a parallel and distributed setting, and investigate additional strategies for optimizing gradient descent.

Expressing emotion through posture and gesture Introduction <=SEP=> Introduction Emotion and its physical expression are an integral part of social interaction, informing others about how we are feeling and affecting social outcomes (Vosk, Forehand, and Figueroa 1983). Studies on the physical expression of emotion can be traced back to the 19th century with Darwin’s seminal book “The Expression of the Emotions in Man and Animals” that reveals the key role of facial expressions and body movement in communicating status and emotion (Darwin 1872).

Robust and Computationally Lightweight Autonomous Tracking of Vehicle Taillights and Signal Detection by Embedded Smart Cameras <=SEP=> An important aspect of collision avoidance and driver assistance systems, as well as autonomous vehicles, is the tracking of vehicle taillights and the detection of alert signals (turns and brakes). In this paper, we present the design and implementation of a robust and computationally lightweight algorithm for a real-time vision system, capable of detecting and tracking vehicle taillights, recognizing common alert signals using a vehicle-mounted embedded smart camera, and counting the cars passing on both sides of the vehicle. The system is low-power and processes scenes entirely on the microprocessor of an embedded smart camera. In contrast to most existing work that addresses either daytime or nighttime detection, the presented system provides the ability to track vehicle taillights and detect alert signals regardless of lighting conditions. The mobile vision system has been tested in actual traffic scenes and the results obtained demonstrate the performance and the lightweight nature of the algorithm.

Role of the Immune system in chronic pain <=SEP=> During the past two decades, an important focus of pain research has been the study of chronic pain mechanisms, particularly the processes that lead to the abnormal sensitivity — spontaneous pain and hyperalgesia — that is associated with these states. For some time it has been recognized that inflammatory mediators released from immune cells can contribute to these persistent pain states. However, it has only recently become clear that immune cell products might have a crucial role not just in inflammatory pain, but also in neuropathic pain caused by damage to peripheral nerves or to the CNS.

An Energy-Efficient Architecture for Binary Weight Convolutional Neural Networks <=SEP=> Binary weight convolutional neural networks (BCNNs) can achieve near state-of-the-art classification accuracy and have far less computation complexity compared with traditional CNNs using high-precision weights. Due to their binary weights, BCNNs are well suited for vision-based Internet-of-Things systems being sensitive to power consumption. BCNNs make it possible to achieve very high throughput with moderate power dissipation. In this paper, an energy-efficient architecture for BCNNs is proposed. It fully exploits the binary weights and other hardware-friendly characteristics of BCNNs. A judicious processing schedule is proposed so that off-chip I/O access is minimized and activations are maximally reused. To significantly reduce the critical path delay, we introduce optimized compressor trees and approximate binary multipliers with two novel compensation schemes. The latter is able to save significant hardware resource, and almost no computation accuracy is compromised. Taking advantage of error resiliency of BCNNs, an innovative approximate adder is developed, which significantly reduces the silicon area and data path delay. Thorough error analysis and extensive experimental results on several data sets show that the approximate adders in the data path cause negligible accuracy loss. Moreover, algorithmic transformations for certain layers of BCNNs and a memory-efficient quantization scheme are incorporated to further reduce the energy cost and on-chip storage requirement. Finally, the proposed BCNN hardware architecture is implemented with the SMIC 130-nm technology. The postlayout results demonstrate that our design can achieve an energy efficiency over 2.0TOp/s/W when scaled to 65 nm, which is more than two times better than the prior art.

Application of ordinal logistic regression analysis in determining risk factors of child malnutrition in Bangladesh <=SEP=> BACKGROUND
The study attempts to develop an ordinal logistic regression (OLR) model to identify the determinants of child malnutrition instead of developing traditional binary logistic regression (BLR) model using the data of Bangladesh Demographic and Health Survey 2004.


METHODS
Based on weight-for-age anthropometric index (Z-score) child nutrition status is categorized into three groups-severely undernourished (< -3.0), moderately undernourished (-3.0 to -2.01) and nourished (≥-2.0). Since nutrition status is ordinal, an OLR model-proportional odds model (POM) can be developed instead of two separate BLR models to find predictors of both malnutrition and severe malnutrition if the proportional odds assumption satisfies. The assumption is satisfied with low p-value (0.144) due to violation of the assumption for one co-variate. So partial proportional odds model (PPOM) and two BLR models have also been developed to check the applicability of the OLR model. Graphical test has also been adopted for checking the proportional odds assumption.


RESULTS
All the models determine that age of child, birth interval, mothers' education, maternal nutrition, household wealth status, child feeding index, and incidence of fever, ARI & diarrhoea were the significant predictors of child malnutrition; however, results of PPOM were more precise than those of other models.


CONCLUSION
These findings clearly justify that OLR models (POM and PPOM) are appropriate to find predictors of malnutrition instead of BLR models.

Why Sleep Matters-The Economic Costs of Insufficient Sleep: A Cross-Country Comparative Analysis. <=SEP=> The Centers for Disease Control and Prevention (CDC) in the United States has declared insufficient sleep a "public health problem." Indeed, according to a recent CDC study, more than a third of American adults are not getting enough sleep on a regular basis. However, insufficient sleep is not exclusively a US problem, and equally concerns other industrialised countries such as the United Kingdom, Japan, Germany, or Canada. According to some evidence, the proportion of people sleeping less than the recommended hours of sleep is rising and associated with lifestyle factors related to a modern 24/7 society, such as psychosocial stress, alcohol consumption, smoking, lack of physical activity and excessive electronic media use, among others. This is alarming as insufficient sleep has been found to be associated with a range of negative health and social outcomes, including success at school and in the labour market. Over the last few decades, for example, there has been growing evidence suggesting a strong association between short sleep duration and elevated mortality risks. Given the potential adverse effects of insufficient sleep on health, well-being and productivity, the consequences of sleep-deprivation have far-reaching economic consequences. Hence, in order to raise awareness of the scale of insufficient sleep as a public-health issue, comparative quantitative figures need to be provided for policy- and decision-makers, as well as recommendations and potential solutions that can help tackling the problem.

User-defined Interface Gestures: Dataset and Analysis <=SEP=> We present a video-based gesture dataset and a methodology for annotating video-based gesture datasets. Our dataset consists of user-defined gestures generated by 18 participants from a previous investigation of gesture memorability. We design and use a crowd-sourced classification task to annotate the videos. The results are made available through a web-based visualization that allows researchers and designers to explore the dataset. Finally, we perform an additional descriptive analysis and quantitative modeling exercise that provide additional insights into the results of the original study. To facilitate the use of the presented methodology by other researchers we share the data, the source of the human intelligence tasks for crowdsourcing, a new taxonomy that integrates previous work, and the source code of the visualization tool.

Decision Support for Handling Mismatches between COTS Products and System Requirements <=SEP=> In the process of selecting commercial off-the-shelf (COTS) products, it is inevitable to encounter mismatches between COTS products and system requirements. Mismatches occur when COTS attributes do not exactly match our requirements. Many of these mismatches are resolved after selecting a COTS product in order to improve its fitness with the requirements. This paper proposes a decision support approach that aims at addressing COTS mismatches during and after the selection process. Our approach can be integrated with existing COTS selection methods at two stages: (I) When evaluating COTS candidates: our approach is used to estimate the anticipated fitness of the candidates if their mismatches are resolved. This helps to base our COTS selection decisions on the fitness that the COTS candidates will eventually have if selected. (2) After selecting a COTS product: the approach suggests alternative plans for resolving the most appropriate mismatches using suitable actions, such that the most important risk, technical, and resource constraints are met. A case study from the e-services domain is used to illustrate the method and to discuss its added value

Multimodal Classification of Remote Sensing Images: A Review and Future Directions <=SEP=> Earth observation through remote sensing images allows the accurate characterization and identification of materials on the surface from space and airborne platforms. Multiple and heterogeneous image sources can be available for the same geographical region: multispectral, hyperspectral, radar, multitemporal, and multiangular images can today be acquired over a given scene. These sources can be combined/fused to improve classification of the materials on the surface. Even if this type of systems is generally accurate, the field is about to face new challenges: the upcoming constellations of satellite sensors will acquire large amounts of images of different spatial, spectral, angular, and temporal resolutions. In this scenario, multimodal image fusion stands out as the appropriate framework to address these problems. In this paper, we provide a taxonomical view of the field and review the current methodologies for multimodal classification of remote sensing images. We also highlight the most recent advances, which exploit synergies with machine learning and signal processing: sparse methods, kernel-based fusion, Markov modeling, and manifold alignment. Then, we illustrate the different approaches in seven challenging remote sensing applications: 1) multiresolution fusion for multispectral image classification; 2) image downscaling as a form of multitemporal image fusion and multidimensional interpolation among sensors of different spatial, spectral, and temporal resolutions; 3) multiangular image classification; 4) multisensor image fusion exploiting physically-based feature extractions; 5) multitemporal image classification of land covers in incomplete, inconsistent, and vague image sources; 6) spatiospectral multisensor fusion of optical and radar images for change detection; and 7) cross-sensor adaptation of classifiers. The adoption of these techniques in operational settings will help to monitor our planet from space in the very near future.

Interface engineering of highly efficient perovskite solar cells <=SEP=> Advancing perovskite solar cell technologies toward their theoretical power conversion efficiency (PCE) requires delicate control over the carrier dynamics throughout the entire device. By controlling the formation of the perovskite layer and careful choices of other materials, we suppressed carrier recombination in the absorber, facilitated carrier injection into the carrier transport layers, and maintained good carrier extraction at the electrodes. When measured via reverse bias scan, cell PCE is typically boosted to 16.6% on average, with the highest efficiency of ~19.3% in a planar geometry without antireflective coating. The fabrication of our perovskite solar cells was conducted in air and from solution at low temperatures, which should simplify manufacturing of large-area perovskite devices that are inexpensive and perform at high levels.

Concatenate text embeddings for text classification <=SEP=> Text embedding has gained a lot of interests in text classification area. This paper investigates the popular neural document embedding method Paragraph Vector as a source of evidence in document ranking. We focus on the effects of combining knowledge-based with knowledge-free document embeddings for text classification task. We concatenate these two representations so that the classification can be done more accurately. The results of our experiments show that this approach achieves better performances on a popular dataset.

Limits of predictability in human mobility. <=SEP=> A range of applications, from predicting the spread of human and electronic viruses to city planning and resource management in mobile communications, depend on our ability to foresee the whereabouts and mobility of individuals, raising a fundamental question: To what degree is human behavior predictable? Here we explore the limits of predictability in human dynamics by studying the mobility patterns of anonymized mobile phone users. By measuring the entropy of each individual's trajectory, we find a 93% potential predictability in user mobility across the whole user base. Despite the significant differences in the travel patterns, we find a remarkable lack of variability in predictability, which is largely independent of the distance users cover on a regular basis.

Reusing Hardware Performance Counters to Detect and Identify Kernel Control-Flow Modifying Rootkits <=SEP=> Kernel rootkits are formidable threats to computer systems. They are stealthy and can have unrestricted access to system resources. This paper presents NumChecker, a new virtual machine (VM) monitor based framework to detect and identify control-flow modifying kernel rootkits in a guest VM. NumChecker detects and identifies malicious modifications to a system call in the guest VM by measuring the number of certain hardware events that occur during the system call's execution. To automatically count these events, NumChecker leverages the hardware performance counters (HPCs), which exist in modern processors. By using HPCs, the checking cost is significantly reduced and the tamper-resistance is enhanced. We implement a prototype of NumChecker on Linux with the kernel-based VM. An HPC-based two-phase kernel rootkit detection and identification technique is presented and evaluated on a number of real-world kernel rootkits. The results demonstrate its practicality and effectiveness.

Development and investigation of efficient artificial bee colony algorithm for numerical function optimization <=SEP=> Artificial bee colony algorithm (ABC), which is inspired by the foraging behavior of honey bee swarm, is a biological-inspired optimization. It shows more effective than genetic algorithm (GA), particle swarm optimization (PSO) and ant colony optimization (ACO). However, ABC is good at exploration but poor at exploitation, and its convergence speed is also an issue in some cases. For these insufficiencies, we propose an improved ABC algorithm called I-ABC. In I-ABC, the best-so-far solution, inertia weight and acceleration coefficients are introduced to modify the search process. Inertia weight and acceleration coefficients are defined as functions of the fitness. In addition, to further balance search processes, the modification forms of the employed bees and the onlooker ones are different in the second acceleration coefficient. Experiments show that, for most functions, the I-ABC has a faster convergence speed and ptimization better performances than each of ABC and the gbest-guided ABC (GABC). But I-ABC could not still substantially achieve the best solution for all optimization problems. In a few cases, it could not find better results than ABC or GABC. In order to inherit the bright sides of ABC, GABC and I-ABC, a high-efficiency hybrid ABC algorithm, which is called PS-ABC, is proposed. PS-ABC owns the abilities of prediction and selection. Results show that PS-ABC has a faster convergence speed like I-ABC and better search ability ods f than other relevant meth

Comparative Evaluation of Anomaly Detection Techniques for Sequence Data <=SEP=> We present a comparative evaluation of a large number of anomaly detection techniques on a variety of publicly available as well as artificially generated data sets. Many of these are existing techniques while some are slight variants and/or adaptations of traditional anomaly detection techniques to sequence data.

Flexible and Stretchable Electronics Paving the Way for Soft Robotics <=SEP=> Planar and rigid wafer-based electronics are intrinsically incompatible with curvilinear and deformable organisms. Recent development of organic and inorganic flexible and stretchable electronics enabled sensing, stimulation, and actuation of/for soft biological and artificial entities. This review summarizes the enabling technologies of soft sensors and actuators, as well as power sources based on flexible and stretchable electronics. Examples include artificial electronic skins, wearable biosensors and stimulators, electronics-enabled programmable soft actuators, and mechanically compliant power sources. Their potential applications in soft robotics are illustrated in the framework of a five-step human–robot interaction loop. Outlooks of future directions and challenges are provided at the end.

CycleGAN, a Master of Steganography <=SEP=> CycleGAN [Zhu et al., 2017] is one recent successful approach to learn a transformation between two image distributions. In a series of experiments, we demonstrate an intriguing property of the model: CycleGAN learns to “hide” information about a source image into the images it generates in a nearly imperceptible, highfrequency signal. This trick ensures that the generator can recover the original sample and thus satisfy the cyclic consistency requirement, while the generated image remains realistic. We connect this phenomenon with adversarial attacks by viewing CycleGAN’s training procedure as training a generator of adversarial examples and demonstrate that the cyclic consistency loss causes CycleGAN to be especially vulnerable to adversarial attacks.

A Novel Variable Reluctance Resolver for HEV/EV Applications <=SEP=> In order to simplify the manufacturing process of variable reluctance (VR) resolvers for hybrid electric vehicle/electric vehicle (HEV/EV) applications, a novel VR resolver with nonoverlapping tooth-coil windings is proposed in this paper. A comparison of the winding configurations is first carried out between the existing and the proposed designs, followed by the description of the operating principle. Furthermore, the influence of actual application conditions is investigated by finite-element (FE) analyses, including operating speed and assembling eccentricity. In addition, identical stator and windings of the novel design can be employed in three resolvers of different rotor saliencies. The voltage difference among the three rotor combinations, as well as the detecting accuracy, is further investigated. Finally, prototypes are fabricated and tested to verify the analyses.

A content analysis of depression-related tweets <=SEP=> This study examines depression-related chatter on Twitter to glean insight into social networking about mental health. We assessed themes of a random sample (n=2,000) of depression-related tweets (sent 4-11 to 5-4-14). Tweets were coded for expression of DSM-5 symptoms for Major Depressive Disorder (MDD). Supportive or helpful tweets about depression was the most common theme (n=787, 40%), closely followed by disclosing feelings of depression (n=625; 32%). Two-thirds of tweets revealed one or more symptoms for the diagnosis of MDD and/or communicated thoughts or ideas that were consistent with struggles with depression after accounting for tweets that mentioned depression trivially. Health professionals can use our findings to tailor and target prevention and awareness messages to those Twitter users in need.

A Meta-Analysis of Methodologies for Research in Knowledge Management, Organizational Learning and Organizational Memory: Five Years at HICSS <=SEP=> The Task Force on Organizational Memory presented a report at the Hawaii International Conference for System Sciences in January 1998. The report included perspectives on knowledge-oriented research, conceptual models for organizational memory, and research methodologies for researchers considering work in organizational memory. This paper builds on the ideas originally presented in the 1998 report by examining research presented at HICSS in the general areas of knowledge management, organizational memory and organizational learning in the five years since the original task force report.

The Satisfaction With Life Scale. <=SEP=> 

Organizational Learning and Management Information Systems <=SEP=> T he articles in this issue ofDATA BASE were chosen b y Anthony G . Hopwood, who is a professor of accounting and financial reporting at the London Graduate Schoo l of Business Studies . The articles contain important ideas , Professor Hopwood wrote, of significance to all intereste d in information systems, be they practitioners or academics . The authors, with their professional affiliations at th e time, were Chris Argyris, Graduate School of Education , Harvard University; Bo Hedberg and Sten Jonsson, Department of Business Administration, University o f Gothenburg; J . Frisco den Hertog, N . V. Philips' Gloeilampenfabrieken, The Netherlands, and Michael J . Earl, Oxford Centre for Management Studies . The articles appeared originally in Accounting, Organizations and Society, a publication of which Professor Hopwood is editor-in-chief. AOS exists to monitor emergin g developments and to actively encourage new approaches and perspectives .

Missing Data Estimation in High-Dimensional Datasets: A Swarm Intelligence-Deep Neural Network Approach <=SEP=> In this paper, we examine the problem of missing data in high-dimensional datasets by taking into consideration the Missing Completely at Random and Missing at Random mechanisms, as well as the Arbitrary missing pattern. Additionally, this paper employs a methodology based on Deep Learning and Swarm Intelligence algorithms in order to provide reliable estimates for missing data. The deep learning technique is used to extract features from the input data via an unsupervised learning approach by modeling the data distribution based on the input. This deep learning technique is then used as part of the objective function for the swarm intelligence technique in order to estimate the missing data after a supervised fine-tuning phase by minimizing an error function based on the interrelationship and correlation between features in the dataset. The investigated methodology in this paper therefore has longer running times, however, the promising potential outcomes justify the trade-off. Also, basic knowledge of statistics is presumed.

Spatio-Temporal Join on Apache Spark <=SEP=> Effective processing of extremely large volumes of spatial data has led to many organizations employing distributed processing frameworks. Apache Spark is one such open-source framework that is enjoying widespread adoption. Within this data space, it is important to note that most of the observational data (i.e., data collected by sensors, either moving or stationary) has a temporal component, or timestamp. In order to perform advanced analytics and gain insights, the temporal component becomes equally important as the spatial and attribute components. In this paper, we detail several variants of a spatial join operation that addresses both spatial, temporal, and attribute-based joins. Our spatial join technique differs from other approaches in that it combines spatial, temporal, and attribute predicates in the join operator.
 In addition, our spatio-temporal join algorithm and implementation differs from others in that it runs in commercial off-the-shelf (COTS) application. The users of this functionality are assumed to be GIS analysts with little if any knowledge of the implementation details of spatio-temporal joins or distributed processing. They are comfortable using simple tools that do not provide the ability to tweak the configuration of the

Class segmentation and object localization with superpixel neighborhoods <=SEP=> We propose a method to identify and localize object classes in images. Instead of operating at the pixel level, we advocate the use of superpixels as the basic unit of a class segmentation or pixel localization scheme. To this end, we construct a classifier on the histogram of local features found in each superpixel. We regularize this classifier by aggregating histograms in the neighborhood of each superpixel and then refine our results further by using the classifier in a conditional random field operating on the superpixel graph. Our proposed method exceeds the previously published state-of-the-art on two challenging datasets: Graz-02 and the PASCAL VOC 2007 Segmentation Challenge.