Query: On rectified linear units for speech processing

Relevant passages:
Large Scale Distributed Deep Networks <=SEP=> Recent work in unsupervised feature learning and deep learning has shown that being able to train large models can dramatically improve performance. In this paper, we consider the problem of training a deep network with billions of parameters using tens of thousands of CPU cores. We have developed a software framework called DistBelief that can utilize computing clusters with thousands of machines to train large models. Within this framework, we have developed two algorithms for large-scale distributed training: (i) Downpour SGD, an asynchronous stochastic gradient descent procedure supporting a large number of model replicas, and (ii) Sandblaster, a framework that supports a variety of distributed batch optimization procedures, including a distributed implementation of L-BFGS. Downpour SGD and Sandblaster L-BFGS both increase the scale and speed of deep network training. We have successfully used our system to train a deep network 30x larger than previously reported in the literature, and achieves state-of-the-art performance on ImageNet, a visual object recognition task with 16 million images and 21k categories. We show that these same techniques dramatically accelerate the training of a more modestlysized deep network for a commercial speech recognition service. Although we focus on and report performance of these methods as applied to training large neural networks, the underlying algorithms are applicable to any gradient-based machine learning algorithm.

Adaptive Subgradient Methods for Online Learning and Stochastic Optimization <=SEP=> We present a new family of subgradient methods that dynamica lly incorporate knowledge of the geometry of the data observed in earlier iterations to perfo rm more informative gradient-based learning. Metaphorically, the adaptation allows us to find n eedles in haystacks in the form of very predictive but rarely seen features. Our paradigm stems fro m recent advances in stochastic optimization and online learning which employ proximal funct ions to control the gradient steps of the algorithm. We describe and analyze an apparatus for adap tively modifying the proximal function, which significantly simplifies setting a learning rate nd results in regret guarantees that are provably as good as the best proximal function that can be cho sen in hindsight. We give several efficient algorithms for empirical risk minimization probl ems with common and important regularization functions and domain constraints. We experimen tally study our theoretical analysis and show that adaptive subgradient methods outperform state-o f-the-art, yet non-adaptive, subgradient algorithms.

Training Products of Experts by Minimizing Contrastive Divergence <=SEP=> It is possible to combine multiple latent-variable models of the same data by multiplying their probability distributions together and then renormalizing. This way of combining individual expert models makes it hard to generate samples from the combined model but easy to infer the values of the latent variables of each expert, because the combination rule ensures that the latent variables of different experts are conditionally independent when given the data. A product of experts (PoE) is therefore an interesting candidate for a perceptual system in which rapid inference is vital and generation is unnecessary. Training a PoE by maximizing the likelihood of the data is difficult because it is hard even to approximate the derivatives of the renormalization term in the combination rule. Fortunately, a PoE can be trained using a different objective function called contrastive divergence whose derivatives with regard to the parameters can be approximated accurately and efficiently. Examples are presented of contrastive divergence learning using several types of expert on several types of data.

Auto-encoder bottleneck features using deep belief networks <=SEP=> Neural network (NN) bottleneck (BN) features are typically created by training a NN with a middle bottleneck layer. Recently, an alternative structure was proposed which trains a NN with a constant number of hidden units to predict output targets, and then reduces the dimensionality of these output probabilities through an auto-encoder, to create auto-encoder bottleneck (AE-BN) features. The benefit of placing the BN after the posterior estimation network is that it avoids the loss in frame classification accuracy incurred by networks that place the BN before the softmax. In this work, we investigate the use of pre-training when creating AE-BN features. Our experiments indicate that with the AE-BN architecture, pre-trained and deeper NNs produce better AE-BN features. On a 50-hour English Broadcast News task, the AE-BN features provide over a 1% absolute improvement compared to a state-of-the-art GMM/HMM with a WER of 18.8% and pre-trained NN hybrid system with a WER of 18.4%. In addition, on a larger 430-hour Broadcast News task, AE-BN features provide a 0.5% absolute improvement over a strong GMM/HMM baseline with a WER of 16.0%. Finally, system combination with the GMM/HMM baseline and AE-BN systems provides an additional 0.5% absolute on 430 hours over the AE-BN system alone, yielding a final WER of 15.0%.

Unsupervised learning of feature hierarchies <=SEP=> The applicability of machine learning methods is often limi ted by the amount of available labeled data, and by the ability (or inability) of the de signer to produce good internal representations and good similarity measures for the input data vectors. The aim of this thesis is to alleviate these two limitations by proposing al gorithms tolearngood internal representations, and invariant feature hierarchies from u nlabeled data. These methods go beyond traditional supervised learning algorithms, and rely on unsupervised, and semi-supervised learning. In particular, this work focuses on “deep learning” methods , a et of techniques and principles to train hierarchical models. Hierarchical mod els produce feature hierarchies that can capture complex non-linear dependencies among the observed data variables in a concise and efficient manner. After training, these mode ls can be employed in real-time systems because they compute the representation by a very fast forward propagation of the input through a sequence of non-linear transf ormations. When the paucity of labeled data does not allow the use of traditional supervi s d algorithms, each layer of the hierarchy can be trained in sequence starting at the bott om by using unsupervised or semi-supervised algorithms. Once each layer has been train ed, the whole system can be fine-tuned in an end-to-end fashion. We propose several unsu pervised algorithms that can be used as building block to train such feature hierarchi es. We investigate algorithms that produce sparse overcomplete representations a nd fe tures that are invariant to known and learned transformations. These algorithms are designed using the Energy-

A Mixed Fragmentation Methodology For Initial Distributed Database Design <=SEP=> We deene mixed fragmentation as a process of simultaneously applying the horizontal and vertical fragmentation on a relation. It can be achieved in one of two ways: by performing horizontal fragmentation followed by vertical fragmentation or by performing vertical fragmentation followed by horizontal fragmentation. The need for mixed fragmentation arises in distributed databases because database users usually access subsets of data which are vertical and horizontal fragments of global relations and there is a need to process queries or transactions that would access these fragments optimally. We present algorithms for generating candidate vertical and horizontal fragmentation schemes and propose a methodology for distributed database design using these fragmentation schemes. When applied together these schemes form a grid. This grid consisting of cells is then merged to form mixed fragments so as to minimize the number of disk accesses required to process the distributed transactions. We have implemented the vertical and horizontal fragmentation algorithms and are developing a Distributed Database Design Tool to support the proposed methodology.

Large scale predictive process mining and analytics of university degree course data <=SEP=> For students, in particular freshmen, the degree pathway from semester to semester is not that transparent, although students have a reasonable idea what courses are expected to be taken each semester. An often-pondered question by students is: "what can I expect in the next semester?" More precisely, given the commitment and engagement I presented in this particular course and the respective performance I achieved, can I expect a similar outcome in the next semester in the particular course I selected? Are the demands and expectations in this course much higher so that I need to adjust my commitment and engagement and overall workload if I expect a similar outcome? Is it better to drop a course to manage expectations rather than to (predictably) fail, and perhaps have to leave the degree altogether? Degree and course advisors and student support units find it challenging to provide evidence based advise to students. This paper presents research into educational process mining and student data analytics in a whole university scale approach with the aim of providing insight into the degree pathway questions raised above. The beta-version of our course level degree pathway tool has been used to shed light for university staff and students alike into our university's 1,300 degrees and associated 6 million course enrolments over the past 20 years.

Getting plugged in: an overview of internet addiction. <=SEP=> Internet addiction is not formally recognised as a clinical disorder by the WHO despite increasing evidence that excessive internet use can interfere with daily life and work. There is increasing pressure from Psychologists for Internet addiction to be recognised. This article explores the prevalence, symptoms and management of Internet addiction and the consequences of ignoring the ever growing concerns from public figures and institutions.

A 96-GHz Ortho-Mode Transducer for the Polatron <=SEP=> We describe the design, simulation, fabrication, and performance of a 96-GHz ortho-mode transducer (OMT) to be used for the Polatron—a bolometric receiver with polarization capability. The OMT has low loss, good isolation, and moderately broad bandwidth, and its performance closely resembles simulation results.

A11y Attacks: Exploiting Accessibility in Operating Systems <=SEP=> Driven in part by federal law, accessibility (a11y) support for disabled users is becoming ubiquitous in commodity OSs. Some assistive technologies such as natural language user interfaces in mobile devices are welcomed by the general user population. Unfortunately, adding new features in modern, complex OSs usually introduces new security vulnerabilities. Accessibility support is no exception. Assistive technologies can be defined as computing subsystems that either transform user input into interaction requests for other applications and the underlying OS, or transform application and OS output for display on alternative devices. Inadequate security checks on these new I/O paths make it possible to launch attacks from accessibility interfaces. In this paper, we present the first security evaluation of accessibility support for four of the most popular computing platforms: Microsoft Windows, Ubuntu Linux, iOS, and Android. We identify twelve attacks that can bypass state-of-the-art defense mechanisms deployed on these OSs, including UAC, the Yama security module, the iOS sandbox, and the Android sandbox. Further analysis of the identified vulnerabilities shows that their root cause is that the design and implementation of accessibility support involves inevitable trade-offs among compatibility, usability, security, and (economic) cost. These trade-offs make it difficult to secure a system against misuse of accessibility support. Based on our findings, we propose a number of recommendations to either make the implementation of all necessary security checks easier and more intuitive, or to alleviate the impact of missing/incorrect checks. We also point out open problems and challenges in automatically analyzing accessibility support and identifying security vulnerabilities.

Low-cost traffic analysis of Tor <=SEP=> Tor is the second generation onion router supporting the anonymous transport of TCP streams over the Internet. Its low latency makes it very suitable for common tasks, such as Web browsing, but insecure against traffic-analysis attacks by a global passive adversary. We present new traffic-analysis techniques that allow adversaries with only a partial view of the network to infer which nodes are being used to relay the anonymous streams and therefore greatly reduce the anonymity provided by Tor. Furthermore, we show that otherwise unrelated streams can be linked back to the same initiator Our attack is feasible for the adversary anticipated by the Tor designers. Our theoretical attacks are backed up by experiments performed on the deployed, albeit experimental, Tor network. Our techniques should also be applicable to any low latency anonymous network. These attacks highlight the relationship between the field of traffic-analysis and more traditional computer security issues, such as covert channel analysis. Our research also highlights that the inability to directly observe network links does not prevent an attacker from performing traffic-analysis: the adversary can use the anonymising network as an oracle to infer the traffic load on remote nodes in order to perform traffic-analysis.

DeltaIoT: A Self-Adaptive Internet of Things Exemplar <=SEP=> Internet of Things (IoT) consists of networked tiny embedded computers (motes) that are capable of monitoring and controlling the physical world. Examples range from building security monitoring to smart factories. A central problem of IoT is minimising the energy consumption of the motes, while guaranteeing high packet delivery performance, regardless of uncertainties such as sudden changes in traffic load and communication interference. Traditionally, to deal with uncertainties the network settings are either hand-tuned or over-provisioned, resulting in continuous network maintenance or inefficiencies. Enhancing the IoT network with self-adaptation can automate these tasks. This paper presents DeltaIoT, an exemplar that enables researchers to evaluate and compare new methods, techniques and tools for self-adaptation in IoT. DeltaIoT is the first exemplar for research on self-adaptation that provides both a simulator for offline experimentation and a physical setup that can be accessed remotely for real-world experimentation.

Connecting ” and “ Disconnecting ” With Civic Life : Patterns of Internet Use and the Production of Social Capital <=SEP=> This article explores the relationship between Internet use and the individual-level production of social capital. To do so, the authors adopt a motivational perspective to distinguish among types of Internet use when examining the factors predicting civic engagement, interpersonal trust, and life contentment. The predictive power of new media use is then analyzed relative to key demographic, contextual, and traditional media use variables using the 1999 DDB Life Style Study. Although the size of associations is generally small, the data suggest that informational uses of the Internet are positively related to individual differences in the production of social capital, whereas social-recreational uses are negatively related to these civic indicators. Analyses within subsamples defined by generational age breaks further suggest that social capital production is related to Internet use among Generation X, while it is tied to television use among Baby Boomers and newspaper use among members of the Civic Generation. The possibility of life cycle and cohort effects is discussed.

The spatio-temporal generalized additive model for criminal incidents <=SEP=> Law enforcement agencies need to model spatio-temporal patterns of criminal incidents. With well developed models, they can study the causality of crimes and predict future criminal incidents, and they can use the results to help prevent crimes. In this paper, we described our newly developed spatio-temporal generalized additive model (S-T GAM) to discover underlying factors related to crimes and predict future incidents. The model can fully utilize many different types of data, such as spatial, temporal, geographic, and demographic data, to make predictions. We efficiently estimated the parameters for S-T GAM using iteratively re-weighted least squares and maximum likelihood and the resulting estimates provided for model interpretability. In this paper we showed the evaluation of S-T GAM with the actual criminal incident data from Charlottesville, Virginia. The evaluation results showed that S-T GAM outperformed the previous spatial prediction models in predicting future criminal incidents.

Characterizing and detecting performance bugs for smartphone applications <=SEP=> Smartphone applications’ performance has a vital impact on user experience. However, many smartphone applications suffer from bugs that cause significant performance degradation, thereby losing their competitive edge. Unfortunately, people have little understanding of these performance bugs. They also lack effective techniques to fight with such bugs. To bridge this gap, we conducted a study of 70 real-world performance bugs collected from eight large-scale and popular Android applications. We studied the characteristics (e.g., bug types and how they manifested) of these bugs and identified their common patterns. These findings can support follow-up research on performance bug avoidance, testing, debugging and analysis for smartphone applications. To demonstrate the usefulness of our findings, we implemented a static code analyzer, PerfChecker, to detect our identified performance bug patterns. We experimentally evaluated PerfChecker by applying it to 29 popular Android applications, which comprise 1.1 million lines of Java code. PerfChecker successfully detected 126 matching instances of our performance bug patterns. Among them, 68 were quickly confirmed by developers as previously-unknown issues that affect application performance, and 20 were fixed soon afterwards by following our optimization suggestions.

Person re-identification in multi-camera system by signature based on interest point descriptors collected on short video sequences <=SEP=> We present and evaluate a person re-identification scheme for multi-camera surveillance system. Our approach uses matching of signatures based on interest-points descriptors collected on short video sequences. One of the originalities of our method is to accumulate interest points on several sufficiently time-spaced images during person tracking within each camera, in order to capture appearance variability. A first experimental evaluation conducted on a publicly available set of low-resolution videos in a commercial mall shows very promising inter-camera person re-identification performances (a precision of 82% for a recall of 78%). It should also be noted that our matching method is very fast: ~ 1/8s for re-identification of one target person among 10 previously seen persons, and a logarithmic dependence with the number of stored person models, making re- identification among hundreds of persons computationally feasible in less than ~ 1/5 second.

ThingTalk : A Distributed Language for a Social Internet of Things <=SEP=> The Internet of Things has increased the number of connected devices by an order of magnitude. This increase comes with a corresponding increase in complexity, which has led to centralized architectures that do not respect the privacy of the users and incur high development and deployment costs. We propose the Open Thing Platform as a new distributed architecture which is built around ThingTalk, a new high level declarative programming language that can express the interaction between multiple physical devices and online services. The architecture abstracts away the details of the communication without sacrificing privacy. We experiment with ThingTalk in multiple domains and we show that several applications can be written in a few lines of code and with no need for any server infrastructure.

Using Mobile LiDAR Data for Rapidly Updating Road Markings <=SEP=> Updating road markings is one of the routine tasks of transportation agencies. Compared with traditional road inventory mapping techniques, vehicle-borne mobile light detection and ranging (LiDAR) systems can undertake the job safely and efficiently. However, current hurdles include software and computing challenges when handling huge volumes of highly dense and irregularly distributed 3-D mobile LiDAR point clouds. This paper presents the development and implementation aspects of an automated object extraction strategy for rapid and accurate road marking inventory. The proposed road marking extraction method is based on 2-D georeferenced feature (GRF) images, which are interpolated from 3-D road surface points through a modified inverse distance weighted (IDW) interpolation. Weighted neighboring difference histogram (WNDH)-based dynamic thresholding and multiscale tensor voting (MSTV) are proposed to segment and extract road markings from the noisy corrupted GRF images. The results obtained using 3-D point clouds acquired by a RIEGL VMX-450 mobile LiDAR system in a subtropical urban environment are encouraging.

A review on the platform design, dynamic modeling and control of hybrid UAVs <=SEP=> This article presents a review on the platform design, dynamic modeling and control of hybrid Unmanned Aerial Vehicles (UAVs). For now, miniature UAVs which have experienced a tremendous development are dominated by two main types, i.e., fixed-wing UAV and Vertical Take-Off and Landing (VTOL) UAV, each of which, however, has its own inherent limitations on such as flexibility, payload, axnd endurance. Enhanced popularity and interest are recently gained by a newer type of UAVs, named hybrid UAV that integrates the beneficial features of both conventional ones. In this paper, a technical overview of the recent advances of the hybrid UAV is presented. More specifically, the hybrid UAV's platform design together with the associated technical details and features are introduced first. Next, the work on hybrid UAV's flight dynamics modeling is then categorized and explained. As for the flight control system design for the hybrid UAV, several flight control strategies implemented are discussed and compared in terms of theory, linearity and implementation.

Neural Joint Model for Transition-based Chinese Syntactic Analysis <=SEP=> We present neural network-based joint models for Chinese word segmentation, POS tagging and dependency parsing. Our models are the first neural approaches for fully joint Chinese analysis that is known to prevent the error propagation problem of pipeline models. Although word embeddings play a key role in dependency parsing, they cannot be applied directly to the joint task in the previous work. To address this problem, we propose embeddings of character strings, in addition to words. Experiments show that our models outperform existing systems in Chinese word segmentation and POS tagging, and perform preferable accuracies in dependency parsing. We also explore bi-LSTM models with fewer features.

Power divider in ridge gap waveguide technology <=SEP=> A power divider in ridge gap waveguide technology has been designed to work at 15GHz. Measurements and simulations are provided. Besides, a broad numerical study of the characteristic impedance of a ridge gap waveguide is presented.

Mapping the antigenic and genetic evolution of influenza virus. <=SEP=> The antigenic evolution of influenza A (H3N2) virus was quantified and visualized from its introduction into humans in 1968 to 2003. Although there was remarkable correspondence between antigenic and genetic evolution, significant differences were observed: Antigenic evolution was more punctuated than genetic evolution, and genetic change sometimes had a disproportionately large antigenic effect. The method readily allows monitoring of antigenic differences among vaccine and circulating strains and thus estimation of the effects of vaccination. Further, this approach offers a route to predicting the relative success of emerging strains, which could be achieved by quantifying the combined effects of population level immune escape and viral fitness on strain evolution.

Some Comments on Cp <=SEP=> 

Remarkable archaeal diversity detected in a Yellowstone National Park hot spring environment. <=SEP=> Of the three primary phylogenetic domains--Archaea (archaebacteria), Bacteria (eubacteria), and Eucarya (eukaryotes)--Archaea is the least understood in terms of its diversity, physiologies, and ecological panorama. Although many species of Crenarchaeota (one of the two recognized archaeal kingdoms sensu Woese [Woese, C. R., Kandler, O. & Wheelis, M. L. (1990) Proc. Natl. Acad. Sci. USA 87, 4576-4579]) have been isolated, they constitute a relatively tight-knit cluster of lineages in phylogenetic analyses of rRNA sequences. It seemed possible that this limited diversity is merely apparent and reflects only a failure to culture organisms, not their absence. We report here phylogenetic characterization of many archaeal small subunit rRNA gene sequences obtained by polymerase chain reaction amplification of mixed population DNA extracted directly from sediment of a hot spring in Yellowstone National Park. This approach obviates the need for cultivation to identify organisms. The analyses document the existence not only of species belonging to well-characterized crenarchaeal genera or families but also of crenarchaeal species for which no close relatives have so far been found. The large number of distinct archaeal sequence types retrieved from this single hot spring was unexpected and demonstrates that Crenarchaeota is a much more diverse group than was previously suspected. The results have impact on our concepts of the phylogenetic organization of Archaea.

Color image processing [basics and special issue overview] <=SEP=> umans have always seen the world in color but only recently have we been able to generate vast quantities of color images with such ease. In the last three decades, we have seen a rapid and enormous transition from grayscale images to color ones. Today, we are exposed to color images on a daily basis in print, photographs, television, computer displays, and cinema movies, where color now plays a vital role in the advertising and dissemination of information throughout the world. Color monitors, printers, and copiers now dominate the office and home environments, with color becoming increasingly cheaper and easier to generate and reproduce. Color demands have soared in the marketplace and are projected to do so for years to come. With this rapid progression, color and multispectral properties of images are becoming increasingly crucial to the field of image processing, often extending and/or replacing previously known grayscale techniques. We have seen the birth of color algorithms that range from direct extensions of grayscale ones, where images are treated as three monochrome separations, to more sophisticated approaches that exploit the correlations among the color bands, yielding more accurate results. Hence, it is becoming increasingly necessary for the signal processing community to understand the fundamental differences between color and grayscale imaging. There are more than a few extensions of concepts

Relational active learning for link-based classification <=SEP=> Many information tasks involve objects that are explicitly or implicitly connected in a network (or graph), such as webpages connected by hyperlinks or people linked by “friendships” in a social network. Research on link-based classification (LBC) has shown how to leverage these connections to improve classification accuracy. Unfortunately, acquiring a sufficient number of labeled examples to enable accurate learning for LBC can often be expensive or impractical. In response, some recent work has proposed the use of active learning, where the LBC method can intelligently select a limited set of additional labels to acquire, so as to reduce the overall cost of learning a model with sufficient accuracy. This work, however, has produced conflicting results and has not considered recent progress for LBC inference and semi-supervised learning. In this paper, we evaluate multiple prior methods for active learning and demonstrate that none consistently improve upon random guessing. We then introduce two new methods that both seek to improve active learning by leveraging the link structure to identify nodes to acquire that are more representative of the underlying data. We show that both approaches have some merit, but that one method, by proactively acquiring nodes so as to produce a more representative distribution of known labels, often leads to significant accuracy increases with minimal computational cost.

MRJS : A JavaScript MapReduce Framework for Web Browsers <=SEP=> This paper describes MRJS, a JavaScript based MapReduce framework. MRJS runs on the open internet over HTTP rather than in a data center. Instead of using clusters of machines for processing power, it relies on volunteer clients in the form of JavaScript engines in web browsers. Users can specify, execute, and fetch data for arbitrary MapReduce jobs with HTTP requests. Our centralized job server acts as a coordinator. We have implemented MRJS and results show that it performs well for certain types of MapReduce computations.

In Search of the Uncanny Valley <=SEP=> Recent advances in computer animation and robotics have lead to greater and greater realism of human appearance to be obtained both on screen and in physical devices. A particular issue that has arisen in this pursuit is whether increases in realism necessarily lead to increases in acceptance. The concept of the uncanny valley suggests that high, though not perfect, levels of realism will result in poor acceptance. We review this concept and its psychological basis.

Risk Management for Service-Oriented Systems <=SEP=> Web service technology can be used for integrating heterogeneous and autonomous applications into cross-organizational systems. A key problem is to support a high quality of service-oriented systems despite vulnerabilities caused by the use of external web services. One important aspect that has received little attention so far is risk management for such systems. This paper discusses risks peculiar for servicebased systems, their impact and ways of mitigation. In the context of service-oriented design, risks can be reduced by selection of appropriate business partners, web service discovery, service composition and Quality of Service (QoS) management. Advisors. Vincenzo D’Andrea.

Product Classification in E-Commerce using Distributional Semantics <=SEP=> Product classification is the task of automatically predicting a taxonomy path for a product in a predefined taxonomy hierarchy given a textual product description or title. For efficient product classification we require a suitable representation for a document (the textual description of a product) feature vector and efficient and fast algorithms for prediction. To address the above challenges, we propose a new distributional semantics representation for document vector formation. We also develop a new two-level ensemble approach utilizing (with respect to the taxonomy tree) a path-wise, node-wise and depth-wise classifiers for error reduction in the final product classification. Our experiments show the effectiveness of the distributional representation and the ensemble approach on data sets from a leading e-commerce platform and achieve better results on various evaluation metrics compared to earlier approaches.