Query: Long short-term memory recurrent neural network architectures for large scale acoustic modeling

Relevant passages:
Large Scale Distributed Deep Networks <=SEP=> Recent work in unsupervised feature learning and deep learning has shown that being able to train large models can dramatically improve performance. In this paper, we consider the problem of training a deep network with billions of parameters using tens of thousands of CPU cores. We have developed a software framework called DistBelief that can utilize computing clusters with thousands of machines to train large models. Within this framework, we have developed two algorithms for large-scale distributed training: (i) Downpour SGD, an asynchronous stochastic gradient descent procedure supporting a large number of model replicas, and (ii) Sandblaster, a framework that supports a variety of distributed batch optimization procedures, including a distributed implementation of L-BFGS. Downpour SGD and Sandblaster L-BFGS both increase the scale and speed of deep network training. We have successfully used our system to train a deep network 30x larger than previously reported in the literature, and achieves state-of-the-art performance on ImageNet, a visual object recognition task with 16 million images and 21k categories. We show that these same techniques dramatically accelerate the training of a more modestlysized deep network for a commercial speech recognition service. Although we focus on and report performance of these methods as applied to training large neural networks, the underlying algorithms are applicable to any gradient-based machine learning algorithm.

Speech recognition with deep recurrent neural networks <=SEP=> Recurrent neural networks (RNNs) are a powerful model for sequential data. End-to-end training methods such as Connectionist Temporal Classification make it possible to train RNNs for sequence labelling problems where the input-output alignment is unknown. The combination of these methods with the Long Short-term Memory RNN architecture has proved particularly fruitful, delivering state-of-the-art results in cursive handwriting recognition. However RNN performance in speech recognition has so far been disappointing, with better results returned by deep feedforward networks. This paper investigates deep recurrent neural networks, which combine the multiple levels of representation that have proved so effective in deep networks with the flexible use of long range context that empowers RNNs. When trained end-to-end with suitable regularisation, we find that deep Long Short-term Memory RNNs achieve a test set error of 17.7% on the TIMIT phoneme recognition benchmark, which to our knowledge is the best recorded score.

A Novel Connectionist System for Unconstrained Handwriting Recognition <=SEP=> Recognizing lines of unconstrained handwritten text is a challenging task. The difficulty of segmenting cursive or overlapping characters, combined with the need to exploit surrounding context, has led to low recognition rates for even the best current recognizers. Most recent progress in the field has been made either through improved preprocessing or through advances in language modeling. Relatively little work has been done on the basic recognition algorithms. Indeed, most systems rely on the same hidden Markov models that have been used for decades in speech and handwriting recognition, despite their well-known shortcomings. This paper proposes an alternative approach based on a novel type of recurrent neural network, specifically designed for sequence labeling tasks where the data is hard to segment and contains long-range bidirectional interdependencies. In experiments on two large unconstrained handwriting databases, our approach achieves word recognition accuracies of 79.7 percent on online data and 74.1 percent on offline data, significantly outperforming a state-of-the-art HMM-based system. In addition, we demonstrate the network's robustness to lexicon size, measure the individual influence of its hidden layers, and analyze its use of context. Last, we provide an in-depth discussion of the differences between the network and HMMs, suggesting reasons for the network's superior performance.

Connectionist speech recognition of Broadcast News <=SEP=> This paper describes connectionist techniques for recogni ti n of Broadcast News. The fundamental difference between connectionist systems and more convent ional mixture-of-Gaussian systems is that connectionist models directly estimate posterior probabi lities as opposed to likelihoods. Access to posterior probabilities has enabled us to develop a number of no vel approaches to confidence estimation, pronunciation modelling and search. In addition we have inv stigated a new feature extraction technique based on the modulation-filtered spectrogram, and methods f or combining multiple information sources. We have incorporated all of these techniques into a system fo r the transcription of Broadcast News, and we present results on the 1998 DARPA Hub-4E Broadcast News ev aluation data.

Application of Pretrained Deep Neural Networks to Large Vocabulary Speech Recognition <=SEP=> The use of Deep Belief Networks (DBN) to pretrain Neural Networks has recently led to a resurgence in the use of Artificial Neural Network Hidden Markov Model (ANN/HMM) hybrid systems for Automatic Speech Recognition (ASR). In this paper we report results of a DBN-pretrained context-dependent ANN/HMM system trained on two datasets that are much larger than any reported previously with DBN-pretrained ANN/HMM systems 5870 hours of Voice Search and 1400 hours of YouTube data. On the first dataset, the pretrained ANN/HMM system outperforms the best Gaussian Mixture Model Hidden Markov Model (GMM/HMM) baseline, built with a much larger dataset by 3.7% absolute WER, while on the second dataset, it outperforms the GMM/HMM baseline by 4.7% absolute. Maximum Mutual Information (MMI) fine tuning and model combination using Segmental Conditional Random Fields (SCARF) give additional gains of 0.1% and 0.4% on the first dataset and 0.5% and 0.9% absolute on the second dataset.

Ml.lib: robust, cross-platform, open-source machine learning for max and pure data <=SEP=> This paper documents the development of ml.lib: a set of opensource tools designed for employing a wide range of machine learning techniques within two popular real-time programming environments, namely Max and Pure Data. ml.lib is a crossplatform, lightweight wrapper around Nick Gillian’s Gesture Recognition Toolkit, a C++ library that includes a wide range of data processing and machine learning techniques. ml.lib adapts these techniques for real-time use within popular dataflow IDEs, allowing instrument designers and performers to integrate robust learning, classification and mapping approaches within their existing workflows. ml.lib has been carefully designed to allow users to experiment with and incorporate machine learning techniques within an interactive arts context with minimal prior knowledge. A simple, logical and consistent, scalable interface has been provided across over sixteen externals in order to maximize learnability and discoverability. A focus on portability and maintainability has enabled ml.lib to support a range of computing architectures—including ARM— and operating systems such as Mac OS, GNU/Linux and Windows, making it the most comprehensive machine learning implementation available for Max and Pure Data.

CrowdProbe: Non-invasive Crowd Monitoring with Wi-Fi Probe <=SEP=> Devices with integrated Wi-Fi chips broadcast beacons for network connection management purposes. Such information can be captured with inexpensive monitors and used to extract user behavior. To understand the behavior of visitors, we deployed our passive monitoring system---CrowdProbe, in a multi-floor museum for six months. We used a Hidden Markov Models (HMM) based trajectory inference algorithm to infer crowd movement using more than 1.7 million opportunistically obtained probe request frames.
 However, as more devices adopt schemes to randomize their MAC addresses in the passive probe session to protect user privacy, it becomes more difficult to track crowd and understand their behavior. In this paper, we try to make use of historical transition probability to reason about the movement of those randomized devices with spatial and temporal constraints. With CrowdProbe, we are able to achieve sufficient accuracy to understand the movement of visitors carrying devices with randomized MAC addresses.

Boosting Named Entity Recognition with Neural Character Embeddings <=SEP=> Most state-of-the-art named entity recognition (NER) systems rely on handcrafted features and on the output of other NLP tasks such as part-of-speech (POS) tagging and text chunking. In this work we propose a language-independent NER system that uses automatically learned features only. Our approach is based on the CharWNN deep neural network, which uses word-level and character-level representations (embeddings) to perform sequential classification. We perform an extensive number of experiments using two annotated corpora in two different languages: HAREM I corpus, which contains texts in Portuguese; and the SPA CoNLL2002 corpus, which contains texts in Spanish. Our experimental results shade light on the contribution of neural character embeddings for NER. Moreover, we demonstrate that the same neural network which has been successfully applied to POS tagging can also achieve state-of-the-art results for language-independet NER, using the same hyperparameters, and without any handcrafted features. For the HAREM I corpus, CharWNN outperforms the state-of-the-art system by 7.9 points in the F1-score for the total scenario (ten NE classes), and by 7.2 points in the F1 for the selective scenario (five NE classes).

Treatment for an endosseous implant migrated into the maxillary sinus not causing maxillary sinusitis: case report. <=SEP=> Placement of endosseous implants in the maxilla has been proven to be a reliable treatment modality. If there is lack of supporting bone, the placed implant may not have enough primary stability and may migrate into the maxillary sinus. Displaced implants must be removed. If there are no signs of maxillary sinusitis, augmentation of the resulting alveolar defect can be performed during the same procedure.

The SLAM project: debugging system software via static analysis <=SEP=> The goal of the SLAM project is to check whether or not a program obeys "API usage rules" that specify what it means to be a good client of an API. The SLAM toolkit statically analyzes a C program to determine whether or not it violates given usage rules. The toolkit has two unique aspects: it does not require the programmer to annotate the source program (invariants are inferred); it minimizes noise (false error messages) through a process known as "counterexample-driven refinement". SLAM exploits and extends results from program analysis, model checking and automated deduction. We have successfully applied the SLAM toolkit to Windows XP device drivers, to both validate behavior and find defects in their usage of kernel APIs.

ACC radar sensor technology, test requirements, and test solutions <=SEP=> This paper presents an overview of the radar sensor technology used in adaptive cruise control (ACC) applications. The performance tradeoffs of the different modulation schemes and scanning antenna types used by these radar sensors are discussed. The test requirements of these millimeter wave radar sensors during production, installation on a vehicle, and during after-market service are also discussed. Different test methods used for characterizing and aligning these radar sensors are presented.

IntEx: A Syntactic Role Driven Protein-Protein Interaction Extractor for Bio-Medical Text <=SEP=> In this paper, we present a fully automated extraction system, named IntEx, to identify gene and protein interactions in biomedical text. Our approach is based on first splitting complex sentences into simple clausal structures made up of syntactic roles. Then, tagging biological entities with the help of biomedical and linguistic ontologies. Finally, extracting complete interactions by analyzing the matching contents of syntactic roles and their linguistically significant combinations. Our extraction system handles complex sentences and extracts multiple and nested interactions specified in a sentence. Experimental evaluations with two other state of the art extraction systems indicate that the IntEx system achieves better performance without the labor intensive pattern engineering requirement. ∗

Estimating the Tensor of Curvature of a Surface from a Polyhedral Approximation <=SEP=> Estimating principal curvatures and principal directions of a surface from a polyhedral approximation with a large number of small faces, such as those produced by iso-surface construction algorithms, has become a basic step in many computer vision algorithms. Particularly in those targeted at medical applications. In this paper we describe a method to estimate the tensor of curvature of a surface at the vertices of a polyhedral approximation. Principal curvatures and principal directions are obtained by computing in closed form the eigenvalues and eigenvectors of certain 3 3 symmetric matrices defined by integral formulas, and closely related to the matrix representation of the tensor of curvature. The resulting algorithm is linear, both in time and in space, as a function of the number of vertices and faces of the polyhedral surface.

Omnidirectional Circularly Polarized Antenna Combining Monopole and Loop Radiators <=SEP=> In this letter, a new omnidirectional circularly polarized (CP) antenna is proposed. By adopting a coaxial probe, in which a metal sleeve acts as a monopole, and printed spoke-like metal strips fabricated on two substrates, which act as a loop, this antenna exhibits an omnidirectional circularly polarized radiation in the azimuthal plane. Theory, design, simulation, and experiment of the antenna are presented. Measurement results show that the proposed antenna can achieve a -10-dB impedance bandwidth of 13.8% (ranges from 1.55 to 1.78 GHz) and a 3-dB axial-ratio (AR) bandwidth of 17.5% (ranges from 1.46 to 1.74 GHz) in the azimuthal plane. At the frequency of 1.575 GHz, the measured average axial ratio and right-hand CP (RHCP) gain are 1.91 dB and 0.86 dBi, respectively, in the azimuthal plane.

A task description language for robot control <=SEP=> Robot systems must achieve high level goals while remaining reactive to contingencies and new opportunities. This typically requires robot systems to coordinate concurrent activities, monitor the environment, and deal with exceptions. We have developed a new language to support such task-level control. The language, TDL, is an extension of C++ that provides syntactic support for task decomposition, synchronization, execution monitoring, and exception handling. A compiler transforms TDL into pure C++ code that utilizes a platform-independent task management library. This paper introduces TDL, describes the task treerepresentation that underlies the language, and presents some aspects of its implementation and use in an autonomous mobile robot.

Topology and Prediction Focused Research on Graph Convolutional Neural Networks <=SEP=> Important advances have been made using convolutional neural network (CNN) approaches to solve complicated problems in areas that rely on grid structured data such as image processing and object classification. Recently, research on graph convolutional neural networks (GCNN) has increased dramatically as researchers try to replicate the success of CNN for graph structured data. Unfortunately, traditional CNN methods are not readily transferable to GCNN, given the irregularity and geometric complexity of graphs. The emerging field of GCNN is further complicated by research papers that differ greatly in their scope, detail, and level of academic sophistication needed by the reader. The present paper provides a review of some basic properties of GCNN. As a guide to the interested reader, recent examples of GCNN research are then grouped according to techniques that attempt to uncover the underlying topology of the graph model and those that seek to generalize traditional CNN methods on graph data to improve prediction of class membership. Discrete Signal Processing on Graphs (DSPG) is used as a theoretical framework to better understand some of the performance gains and limitations of these recent GCNN approaches. A brief discussion of Topology Adaptive Graph Convolutional Networks (TAGCN) is presented as an approach motivated by DSPG and future research directions using this approach are briefly discussed.

Block-Matching Convolutional Neural Network for Image Denoising <=SEP=> There are two main streams in up-to-date image denoising algorithms: non-local self similarity (NSS) prior based methods and convolutional neural network (CNN) based methods. The NSS based methods are favorable on images with regular and repetitive patterns while the CNN based methods perform better on irregular structures. In this paper, we propose a blockmatching convolutional neural network (BMCNN) method that combines NSS prior and CNN. Initially, similar local patches in the input image are integrated into a 3D block. In order to prevent the noise from messing up the block matching, we first apply an existing denoising algorithm on the noisy image. The denoised image is employed as a pilot signal for the block matching, and then denoising function for the block is learned by a CNN structure. Experimental results show that the proposed BMCNN algorithm achieves state-of-the-art performance. In detail, BMCNN can restore both repetitive and irregular structures.

Towards a Machine-Learning Approach for Sickness Prediction in 360° Stereoscopic Videos <=SEP=> Virtual reality systems are widely believed to be the next major computing platform. There are, however, some barriers to adoption that must be addressed, such as that of motion sickness — which can lead to undesirable symptoms including postural instability, headaches, and nausea. Motion sickness in virtual reality occurs as a result of moving visual stimuli that cause users to perceive self-motion while they remain stationary in the real world. There are several contributing factors to both this perception of motion and the subsequent onset of sickness, including field of view, motion velocity, and stimulus depth. We verify first that differences in vection due to relative stimulus depth remain correlated with sickness. Then, we build a dataset of stereoscopic 3D videos and their corresponding sickness ratings in order to quantify their nauseogenicity, which we make available for future use. Using this dataset, we train a machine learning algorithm on hand-crafted features (quantifying speed, direction, and depth as functions of time) from each video, learning the contributions of these various features to the sickness ratings. Our predictor generally outperforms a naïve estimate, but is ultimately limited by the size of the dataset. However, our result is promising and opens the door to future work with more extensive datasets. This and further advances in this space have the potential to alleviate developer and end user concerns about motion sickness in the increasingly commonplace virtual world.

Towards Security-Aware Virtual Environments for Digital Twins <=SEP=> Digital twins open up new possibilities in terms of monitoring, simulating, optimizing and predicting the state of cyber-physical systems (CPSs). Furthermore, we argue that a fully functional, virtual replica of a CPS can also play an important role in securing the system. In this work, we present a framework that allows users to create and execute digital twins, closely matching their physical counterparts. We focus on a novel approach to automatically generate the virtual environment from specification, taking advantage of engineering data exchange formats. From a security perspective, an identical (in terms of the system's specification), simulated environment can be freely explored and tested by security professionals, without risking negative impacts on live systems. Going a step further, security modules on top of the framework support security analysts in monitoring the current state of CPSs. We demonstrate the viability of the framework in a proof of concept, including the automated generation of digital twins and the monitoring of security and safety rules.

Joint learning of ontology and semantic parser from text <=SEP=> Semantic parsing methods are used for capturing and representing semantic meaning of text. Meaning representation capturing all the concepts in the text may not always be available or may not be sufficiently complete. Ontologies provide a structured and reasoning-capable way to model the content of a collection of texts. In this work, we present a novel approach to joint learning of ontology and semantic parser from text. The method is based on semi-automatic induction of a context-free grammar from semantically annotated text. The grammar parses the text into semantic trees. Both, the grammar and the semantic trees are used to learn the ontology on several levels – classes, instances, taxonomic and non-taxonomic relations. The approach was evaluated on the first sentences of Wikipedia pages describing people.

Understanding Online Consumer Stickiness in E-commerce Environment : A Relationship Formation Model <=SEP=> Consumers with online shopping experience often stick to some special websites. Does it mean that long-term relationships between consumers and the websites form? To solve the problem, this paper analyzed the belief and attitude factors influencing online consumer stickiness intention. Based on the relationships between belief-attitude-intention implied in TPB (Theory of Planned Behavior), according to Expectation-Confirmation theory and Commitment-trust theory, we developed a concept model on online consumer stickiness. Six research hypotheses derived from this model were empirically validated using a survey of online shoppers. SEM (Structure Equation Modeling) was as a data analysis method. The results suggest that online consumer stickiness intention is influenced by consumers’ commitment to the website (e-vendors) and their overall satisfaction to the transaction process, and commitment’s effect is stronger; in turn, both consumers’ commitment and overall satisfaction are influenced by consumers’ ongoing trust significantly; confirmation has a significant effect on overall satisfaction but has not the same effect on commitment. The findings show that the long-term relationships between consumers and special websites exactly exist during online repeat consumption.

Differential effects of attentional focus strategies during long-term resistance training. <=SEP=> The purpose of this study was to investigate the effects of using an internal versus external focus of attention during resistance training on muscular adaptations. Thirty untrained college-aged men were randomly assigned to an internal focus group (INTERNAL) that focused on contracting the target muscle during training (n = 15) or an external focus group (EXTERNAL) that focused on the outcome of the lift (n = 15). Training for both routines consisted of 3 weekly sessions performed on non-consecutive days for 8 weeks. Subjects performed 4 sets of 8-12 repetitions per exercise. Changes in strength were assessed by six repetition maximum in the biceps curl and isometric maximal voluntary contraction in knee extension and elbow flexion. Changes in muscle thickness for the elbow flexors and quadriceps were assessed by ultrasound. Results show significantly greater increases in elbow flexor thickness in INTERNAL versus EXTERNAL (12.4% vs. 6.9%, respectively); similar changes were noted in quadriceps thickness. Isometric elbow flexion strength was greater for INTERNAL while isometric knee extension strength was greater for EXTERNAL, although neither reached statistical significance. The findings lend support to the use of a mind-muscle connection to enhance muscle hypertrophy.

Design of an Optical Character Recognition System for Camera-based Handheld Devices <=SEP=> This paper presents a complete Optical Character Recognition (OCR) system for camera captured image/graphics embedded textual documents for handheld devices. At first, text regions are extracted and skew corrected. Then, these regions are binarized and segmented into lines and characters. Characters are passed into the recognition module. Experimenting with a set of 100 business card images, captured by cell phone camera, we have achieved a maximum recognition accuracy of 92.74%. Compared to Tesseract, an open source desktop-based powerful OCR engine, present recognition accuracy is worth contributing. Moreover, the developed technique is computationally efficient and consumes low memory so as to be applicable on handheld devices.

"It Won't Happen To Me!": Self-Disclosure in Online Social Networks <=SEP=> Despite the considerable amount of self-disclosure in Online Social Networks (OSN), the motivation behind this phenomenon is still little understood. Building on the Privacy Calculus theory, this study fills this gap by taking a closer look at the factors behind individual self-disclosure decisions. In a Structural Equation Model with 237 subjects we find Perceived Enjoyment and Privacy Concerns to be significant determinants of information revelation. We confirm that the privacy concerns of OSN users are primarily determined by the perceived likelihood of a privacy violation and much less by the expected damage. These insights provide a solid basis for OSN providers and policy-makers in their effort to ensure healthy disclosure levels that are based on objective rationale rather than subjective misconceptions.

Super-Resolution with Deep Convolutional Sufficient Statistics <=SEP=> Inverse problems in image and audio, and super-resolution in particular, can be seen as high-dimensional structured prediction problems, where the goal is to characterize the conditional distribution of a high-resolution output given its lowresolution corrupted observation. When the scaling ratio is small, point estimates achieve impressive performance, but soon they suffer from the regression-to-themean problem, result of their inability to capture the multi-modality of this conditional distribution. Modeling high-dimensional image and audio distributions is a hard task, requiring both the ability to model complex geometrical structures and textured regions. In this paper, we propose to use as conditional model a Gibbs distribution, where its sufficient statistics are given by deep convolutional neural networks. The features computed by the network are stable to local deformation, and have reduced variance when the input is a stationary texture. These properties imply that the resulting sufficient statistics minimize the uncertainty of the target signals given the degraded observations, while being highly informative. The filters of the CNN are initialized by multiscale complex wavelets, and then we propose an algorithm to fine-tune them by estimating the gradient of the conditional log-likelihood, which bears some similarities with Generative Adversarial Networks. We evaluate experimentally the proposed approach in the image superresolution task, but the approach is general and could be used in other challenging ill-posed problems such as audio bandwidth extension.

Development of the Conformity to Masculine Norms Inventory <=SEP=> This article describes the construction of the Conformity to Masculine Norms Inventory (CMNI), and 5 studies that examined its psychometric properties. Factor analysis indicated 11 distinct factors: Winning, Emotional Control, Risk-Taking, Violence, Dominance, Playboy, SelfReliance, Primacy of Work, Power Over Women, Disdain for Homosexuals, and Pursuit of Status. Results from Studies 2–5 indicated that the CMNI had strong internal consistency estimates and good differential validity comparing men with women and groups of men on healthrelated questions; all of the CMNI subscales were significantly and positively related to other masculinity-related measures, with several subscales being related significantly and positively to psychological distress, social dominance, aggression, and the desire to be more muscular, and significantly and negatively to attitudes toward psychological help seeking and social desirability; and CMNI scores had high test–retest estimates for a 2–3 week period.

Visual Transfer between Atari Games using Competitive Reinforcement Learning <=SEP=> This paper explores the use of deep reinforcement learning agents to transfer knowledge from one environment to another. More specifically, the method takes advantage of asynchronous advantage actor critic (A3C) architecture to generalize a target game using an agent trained on a source game in Atari. Instead of fine-tuning a pre-trained model for the target game, we propose a learning approach to update the model using multiple agents trained in parallel with different representations of the target game. Visual mapping between video sequences of transfer pairs is used to derive new representations of the target game; training on these visual representations of the target game improves model updates in terms of performance, data efficiency and stability. In order to demonstrate the functionality of the architecture, Atari games Pong-v0 and Breakout-v0 are being used from the OpenAI gym environment; as the source and target environment.

Approximate Gaussian Elimination <=SEP=> Approximate Gaussian Elimination

A two-view based multilayer feature graph for robot navigation <=SEP=> To facilitate scene understanding and robot navigation in a modern urban area, we design a multilayer feature graph (MFG) based on two views from an on-board camera. The nodes of an MFG are features such as scale invariant feature transformation (SIFT) feature points, line segments, lines, and planes while edges of the MFG represent different geometric relationships such as adjacency, parallelism, collinearity, and coplanarity. MFG also connects the features in two views and the corresponding 3D coordinate system. Building on SIFT feature points and line segments, MFG is constructed using feature fusion which incrementally, iteratively, and extensively verifies the aforementioned geometric relationships using random sample consensus (RANSAC) framework. Physical experiments show that MFG can be successfully constructed in urban area and the construction method is demonstrated to be very robust in identifying feature correspondence.

Multi-label convolutional neural network based pedestrian attribute classification <=SEP=> Recently, pedestrian attributes like gender, age, clothing etc., have been used as soft biometric traits for recognizing people. Unlike existing methods that assume the independence of attributes during their prediction, we propose a multi-label convolutional neural network (MLCNN) to predict multiple attributes together in a unified framework. Firstly, a pedestrian image is roughly divided into multiple overlapping body parts, which are simultaneously integrated in the multi-label convolutional neural network. Secondly, these parts are filtered independently and aggregated in the cost layer. The cost function is a combination of multiple binary attribute classification cost functions. Experiments show that the proposed method significantly outperforms the SVM based method on the PETA database.