Query: Tradeoffs in Neural Variational Inference Thesis

Relevant passages:
Representation Learning: A Review and New Perspectives <=SEP=> The success of machine learning algorithms generally depends on data representation, and we hypothesize that this is because different representations can entangle and hide more or less the different explanatory factors of variation behind the data. Although specific domain knowledge can be used to help design representations, learning with generic priors can also be used, and the quest for AI is motivating the design of more powerful representation-learning algorithms implementing such priors. This paper reviews recent work in the area of unsupervised feature learning and deep learning, covering advances in probabilistic models, autoencoders, manifold learning, and deep networks. This motivates longer term unanswered questions about the appropriate objectives for learning good representations, for computing representations (i.e., inference), and the geometrical connections between representation learning, density estimation, and manifold learning.

How to Train Deep Variational Autoencoders and Probabilistic Ladder Networks <=SEP=> Variational autoencoders are a powerful framework for unsupervised learning. However, previous work has been restricted to shallow models with one or two layers of fully factorized stochastic latent variables, limiting the flexibility of the latent representation. We propose three advances in training algorithms of variational autoencoders, for the first time allowing to train deep models of up to five stochastic layers, (1) using a structure similar to the Ladder network as the inference model, (2) warm-up period to support stochastic units staying active in early training, and (3) use of batch normalization. Using these improvements we show state-of-the-art log-likelihood results for generative modeling on several benchmark datasets.

Wide Residual Networks <=SEP=> Deep residual networks were shown to be able to scale up to thousands of layers and still have improving performance. However, each fraction of a percent of improved accuracy costs nearly doubling the number of layers, and so training very deep residual networks has a problem of diminishing feature reuse, which makes these networks very slow to train. To tackle these problems, in this paper we conduct a detailed experimental study on the architecture of ResNet blocks, based on which we propose a novel architecture where we decrease depth and increase width of residual networks. We call the resulting network structures wide residual networks (WRNs) and show that these are far superior over their commonly used thin and very deep counterparts. For example, we demonstrate that even a simple 16-layer-deep wide residual network outperforms in accuracy and efficiency all previous deep residual networks, including thousand-layerdeep networks, achieving new state-of-the-art results on CIFAR, SVHN, COCO, and significant improvements on ImageNet. Our code and models are available at https: //github.com/szagoruyko/wide-residual-networks.

Adaptive Subgradient Methods for Online Learning and Stochastic Optimization <=SEP=> We present a new family of subgradient methods that dynamica lly incorporate knowledge of the geometry of the data observed in earlier iterations to perfo rm more informative gradient-based learning. Metaphorically, the adaptation allows us to find n eedles in haystacks in the form of very predictive but rarely seen features. Our paradigm stems fro m recent advances in stochastic optimization and online learning which employ proximal funct ions to control the gradient steps of the algorithm. We describe and analyze an apparatus for adap tively modifying the proximal function, which significantly simplifies setting a learning rate nd results in regret guarantees that are provably as good as the best proximal function that can be cho sen in hindsight. We give several efficient algorithms for empirical risk minimization probl ems with common and important regularization functions and domain constraints. We experimen tally study our theoretical analysis and show that adaptive subgradient methods outperform state-o f-the-art, yet non-adaptive, subgradient algorithms.

Deep learning in neural networks: An overview <=SEP=> In recent years, deep artificial neural networks (including recurrent ones) have won numerous contests in pattern recognition and machine learning. This historical survey compactly summarizes relevant work, much of it from the previous millennium. Shallow and Deep Learners are distinguished by the depth of their credit assignment paths, which are chains of possibly learnable, causal links between actions and effects. I review deep supervised learning (also recapitulating the history of backpropagation), unsupervised learning, reinforcement learning & evolutionary computation, and indirect search for short programs encoding deep and large networks.

Hardware resource estimation for heterogeneous FPGA-based SoCs <=SEP=> The increasing complexity of recent System-on-Chip (SoC) designs introduces new challenges for design space exploration tools. In addition to the time-to-market challenge, designers need to estimate rapidly and accurately both performance and area occupation of complex and diverse applications. High-Level Synthesis (HLS) has been emerged as an attractive solution for designers to address these challenges in order to explore a large number of SoC configurations. In this paper, we target hybrid CPU-FPGA based SoCs. We propose a high-level area estimation tool based on an analytic model without requiring register-transfer level (RTL) implementations. This technique allows to estimate the required FPGA resources at the source code level to map an application to a hybrid CPU-FPGA system. The proposed model also enables a fast design exploration with different trade-offs through HLS optimization pragmas. Experimental results show that the proposed area analytic model provides an accurate estimation with a negligible error (less than 5+) compared to RTL implementations.

Video games do affect social outcomes: a meta-analytic review of the effects of violent and prosocial video game play. <=SEP=> Whether video game play affects social behavior is a topic of debate. Many argue that aggression and helping are affected by video game play, whereas this stance is disputed by others. The present research provides a meta-analytical test of the idea that depending on their content, video games do affect social outcomes. Data from 98 independent studies with 36,965 participants revealed that for both violent video games and prosocial video games, there was a significant association with social outcomes. Whereas violent video games increase aggression and aggression-related variables and decrease prosocial outcomes, prosocial video games have the opposite effects. These effects were reliable across experimental, correlational, and longitudinal studies, indicating that video game exposure causally affects social outcomes and that there are both short- and long-term effects.

A Privacy Preservation Model for Facebook-Style Social Network Systems <=SEP=> Recent years have seen unprecedented growth in the popularity of social network systems, with Facebook being an archetypical example. The access control paradigm behind the privacy preservation mechanism of Facebook is distinctly different from such existing access control paradigms as Discretionary Access Control, Role-Based Access Control, Capability Systems, and Trust Management Systems. This work takes a first step in deepening the understanding of this access control paradigm, by proposing an access control model that formalizes and generalizes the privacy preservation mechanism of Facebook. The model can be instantiated into a family of Facebook-style social network systems, each with a recognizably different access control mechanism, so that Facebook is but one instantiation of the model. We also demonstrate that the model can be instantiated to express policies that are not currently supported by Facebook but possess rich and natural social significance. This work thus delineates the design space of privacy preservation mechanisms for Facebook-style social network systems, and lays out a formal framework for policy analysis in these systems.

Adversarial machine learning <=SEP=> In this paper (expanded from an invited talk at AISEC 2010), we discuss an emerging field of study: adversarial machine learning---the study of effective machine learning techniques against an adversarial opponent. In this paper, we: give a taxonomy for classifying attacks against online machine learning algorithms; discuss application-specific factors that limit an adversary's capabilities; introduce two models for modeling an adversary's capabilities; explore the limits of an adversary's knowledge about the algorithm, feature space, training, and input data; explore vulnerabilities in machine learning algorithms; discuss countermeasures against attacks; introduce the evasion challenge; and discuss privacy-preserving learning techniques.

Voxel-Based Morphometry—The Methods <=SEP=> At its simplest, voxel-based morphometry (VBM) involves a voxel-wise comparison of the local concentration of gray matter between two groups of subjects. The procedure is relatively straightforward and involves spatially normalizing high-resolution images from all the subjects in the study into the same stereotactic space. This is followed by segmenting the gray matter from the spatially normalized images and smoothing the gray-matter segments. Voxel-wise parametric statistical tests which compare the smoothed gray-matter images from the two groups are performed. Corrections for multiple comparisons are made using the theory of Gaussian random fields. This paper describes the steps involved in VBM, with particular emphasis on segmenting gray matter from MR images with nonuniformity artifact. We provide evaluations of the assumptions that underpin the method, including the accuracy of the segmentation and the assumptions made about the statistical distribution of the data.

CRITICAL SUCCESS FACTORS FOR CUSTOMER RELATIONSHIP MANAGEMENT IMPLEMENTATIONS <=SEP=> The growing forces of increasing global competition, continuing customer demands, and the significant revolution in Commercial Off The Shelf (COTS) solutions, especially Customer Relationship Management (CRM) applications, have together put pressure upon many organisations to implement CRM solutions and to switch their organisational processes from being product-centric to being customer-centric. A CRM initiative is not only technology; it is a business strategy supported by technology which automates and enhances the processes associated with managing customer relationships. By the end of 2010, it is predicted that companies will be spending almost $11 billion yearly on CRM solutions. However, studies have found that 70% of CRM projects have failed. Understanding the factors that enable success of CRM is vital. There is very few existing specific research into Critical Success Factors (CSFs) of CRM implementations, and there is no comprehensive view that captures all the aspects for successful CRM implementation and their inter-relationships. Therefore, the aim of this paper is to explore the current literature base of CSFs for CRM implementations and proposes a taxonomy for them. Future research work will continue to investigate in depth these factors by exploring the complex system links between CSFs using systems thinking techniques such as causal maps to investigate the complex, systemic networks of CSFs in organisations which result in emergent effects which themselves influence the failure or success of a CRM.

Efficient Algorithms for Convolutional Sparse Representations <=SEP=> When applying sparse representation techniques to images, the standard approach is to independently compute the representations for a set of overlapping image patches. This method performs very well in a variety of applications, but results in a representation that is multi-valued and not optimized with respect to the entire image. An alternative representation structure is provided by a convolutional sparse representation, in which a sparse representation of an entire image is computed by replacing the linear combination of a set of dictionary vectors by the sum of a set of convolutions with dictionary filters. The resulting representation is both single-valued and jointly optimized over the entire image. While this form of a sparse representation has been applied to a variety of problems in signal and image processing and computer vision, the computational expense of the corresponding optimization problems has restricted application to relatively small signals and images. This paper presents new, efficient algorithms that substantially improve on the performance of other recent methods, contributing to the development of this type of representation as a practical tool for a wider range of problems.

Blind Trust Online : Experimental Evidence from Baseball Cards <=SEP=> Before reaping transaction cost savings from the Internet, online sellers must credibly communicate the quality of their goods or their reliability of delivery. We use baseball cards as an example to address (1) whether consumers fully understand the risk of online trading, (2) the extent to which consumers employ seller reputation and professional grading to alleviate the risk they are exposed to, and (3) the impact of online trading on the retail market and supporting industries. An experiment is carried out to obtain actual baseball cards from both online and retail markets whose quality is then professionally graded and compared to the prices paid by online buyers for goods with similar claims. Our findings indicate that some uninformed online buyers are misled by non-credible claims of quality; they pay higher prices but do not receive better quality and in fact are defrauded more often. Although a combination of professional grading and reputation mechanisms appear to have the potential to solve the problem, they have only had limited success so far. Data on structural changes in the baseball card industry suggest the problems stemming from buyer misconception may have far-reaching consequences, extending beyond the immediate losses of defrauded consumers to influencing producer and supply decisions. Our findings also suggest that empirical studies using online price data should be careful of what assumptions they are making of online users and what behaviors of buyers and sellers are likely to drive the price data. Department of Economics, University of Maryland, 3115 G Tydings, College Park, MD 20742. Contact Email: jin@econ.umd.edu. We thank Seth Sanders, John List, John Shea, Dan Vincent, David Lucking-Reiley, Larry Ausubel, Peter Cramton, V. Joseph Hotz, Jimmy Chan, Vincent Crawford, attendants in Johns Hopkins UMD conference and participants of UCLA,UCSD and FTC seminars for constructive advice and suggestions. We are grateful to 8 friends who acted as our agents in purchasing baseball cards in retail markets, and numerous sports card store owners for their time and insights regarding the sports card industry. Excellent research assistance from Randy Alexander Moore and Krzysztof Fizyta is gratefully acknowledged. Any remaining errors are ours.

Tail call elimination on the Java Virtual Machine <=SEP=> A problem that often has to be solved by compilers for functional languages targeting the Java Virtual Machine is the elimination of tail calls. This paper explains how we solved it in our Funnel compiler and presents some experimental results about the impact our technique has on both performance and size of the compiled programs.

Vehicle location finder using Global position system and Global System for Mobile <=SEP=> Each year, the number of stolen vehicle is on the rise. Usually, to prevent theft, a physical type countermeasure is used such as padlock, disk break lock and other more which is a preventive action but it is not enough safe. The objective of this study is to create a controllable system that can display the location of a vehicle using Global position system (GPS) to pin point the location and Global System for Mobile (GSM) as a mean for communicating with the vehicle for ease of finding after a theft attempt. The system is made to test the accuracy of the location that is send to the user when the vehicle is in motion and stationary in the city and suburb. The system is made by combining a micro controller with GPS and GSM, then comparing it with other similar device available in the market like Garmin and a reference website to find the radius of error. The study of proposed device begins by studying IEEE journal about alternative product and the vehicle itself. The hardware and program development is done by research and trial and error as the controller do not interact with both module at the same time, after successfully programming both module, it is combined into a single program with addition of interrupt program. The experiment is done in three set of tests so that the system accuracy can be determine when stationary and in motion on vehicle, output controlling is the test to determine if the controller can be made into anti-theft system. The result of the test concludes that the system can provide standard GPS coordinate when requested via Short Message Service (SMS). The system can also be used to control an actuator.

Studying weak central coherence at low levels: children with autism do not succumb to visual illusions. A research note. <=SEP=> While anecdotal reports of abnormal perceptual experiences in autism abound, there have been to date no experimental studies showing fundamental perceptual peculiarities. The present paper reports results from a first study of low-level visual integration in autism. Twenty-five subjects with autism, 21 normal 7- and 8-year-olds, and 26 children with learning difficulties were asked to make simple judgements about six well-known visual illusions. Two conditions were used, in an attempt to explore group differences; standard two-dimensional black and white line drawings, and the same figures augmented with raised coloured lines. The subjects with autism were less likely to succumb to the two-dimensional illusions than were the other groups, and were less aided by the three-dimensional 'disembedded' condition. These striking results are discussed with reference to the 'central coherence' account of autism.

A 12.77-MHz 31 ppm/°C On-Chip RC Relaxation Oscillator With Digital Compensation Technique <=SEP=> The design of a 12.77-MHz on-chip RC relaxation oscillator with digital compensation technique is presented. To maintain the frequency stability versus temperature and supply voltage variations, loop delay tuning by a digital feedback loop is developed in this system. In order to generate an on-chip reference for digital calibration, a replica comparator is added. The on-chip relaxation oscillator is fabricated in 0.18-μm CMOS process. The measured output frequency variation is 31 ppm/°C across -30 to 120 °C temperature range after compensation. The frequency variation over the supply voltage from 0.6 V to 1.1 V is ±0.5%/V. The measured total power consumption is 56.2 μW at 0.9-V supply voltage when the digital compensation blocks are enabled. After digital compensation, the compensation blocks can be shutdown for power saving, and the main oscillator consumes only 12.8 μW.

Time is of the essence: improving recency ranking using Twitter data <=SEP=> Realtime web search refers to the retrieval of very fresh content which is in high demand. An effective portal web search engine must support a variety of search needs, including realtime web search. However, supporting realtime web search introduces two challenges not encountered in non-realtime web search: quickly crawling relevant content and ranking documents with impoverished link and click information. In this paper, we advocate the use of realtime micro-blogging data for addressing both of these problems. We propose a method to use the micro-blogging data stream to detect fresh URLs. We also use micro-blogging data to compute novel and effective features for ranking fresh URLs. We demonstrate these methods improve effective of the portal web search engine for realtime web search.

Characterizing in-text citations in scientific articles: A large-scale analysis <=SEP=> We report characteristics of in-text citations in over five million full text articles from two large databases – the PubMed Central Open Access subset and Elsevier journals – as functions of time, textual progression, and scientific field. The purpose of this study is to understand the characteristics of in-text citations in a detailed way prior to pursuing other studies focused on answering more substantive research questions. As such, we have analyzed in-text citations in several ways and report many findings here. Perhaps most significantly, we find that there are large field-level differences that are reflected in position within the text, citation interval (or reference age), and citation counts of references. In general, the fields of Biomedical and Health Sciences, Life and Earth Sciences, and Physical Sciences and Engineering have similar reference distributions, although they vary in their specifics. The two remaining fields, Mathematics and Computer Science and Social Science and Humanities, have different reference distributions from the other three fields and between themselves. We also show that in all fields the numbers of sentences, references, and in-text mentions per article have increased over time, and that there are field-level and temporal differences in the numbers of in-text mentions per reference. A final finding is that references mentioned only once tend to be much more highly cited than those mentioned multiple times.

Low-Power High-Efficiency Class D Audio Power Amplifiers <=SEP=> The architecture, design, and implementation of two clock-free analog class D audio power amplifiers using 0.5 ¿m CMOS standard technology are introduced. The amplifiers are designed to consume significantly less power than former implementations. Both designs operate with a 2.7 V single voltage supply and deliver a maximum output power of 250 mW into an 8 ¿ speaker. The two class-D amplifiers are based on a hysteretic sliding mode controller, which avoids the complex task of generating the highly linear triangle carrier signal used in conventional architectures. The first design generates a two-level modulated signal, known as binary modulated amplifier (BMA); the second topology implements a three-level modulated signal, also called ternary modulated amplifier (TMA). The architectures and implementations are simple and compact, providing very low quiescent power consumption. Experimental results of the BMA/TMA yield an efficiency of 89/90% and a total harmonic distortion plus noise (THD+N) of 0.02/0.03%, respectively. The efficiency and linearity is comparable to state-of-the-art amplifiers but the static power consumption is less than one tenth of previous proposed architectures. Both class D amplifiers achieve a power supply rejection ratio greater than 75 dB at 217 Hz, and a signal-to-noise ratio (SNR) higher than 90 dB within the whole audio band. Each amplifier occupies less than 1.5 mm2.

Mining asynchronous periodic patterns in time series data <=SEP=> Periodicy detection in time series data is a challenging pro blem of great importance in many applications. Most previous work focused on mining synchronous p eriodic patterns and did not recognize misaligned presence of a pattern due to the intervention of r andom noise. In this paper, we propose a more flexible model of asynchronous periodic pattern that ma y be present only within a subsequence and whose occurrences may be shifted due to disturbance. Two parametersmin rep andmax dis are employed to specify the minimum number of repetitions th at is required within each segment of non-disrupted pattern occurrences and the maximum allowed disturbance between any two successive valid segments. Upon satisfying these two requirements, th e longest valid subsequence of a pattern is returned. A two phase algorithm is devised to first generate p o ntial periods by distance-based pruning followed by an iterative procedure to derive and validat e candidate patterns and locate the longest valid subsequence. We also show that this algorithm can not o nly provide linear time complexity with respect to the length of the sequence but also achieve space e ffici ncy.

Incremental Fuzzy Clustering With Multiple Medoids for Large Data <=SEP=> As an important technique of data analysis, clustering plays an important role in finding the underlying pattern structure embedded in unlabeled data. Clustering algorithms that need to store all the data into the memory for analysis become infeasible when the dataset is too large to be stored. To handle such large data, incremental clustering approaches are proposed. The key idea behind these approaches is to find representatives (centroids or medoids) to represent each cluster in each data chunk, which is a packet of the data, and final data analysis is carried out based on those identified representatives from all the chunks. In this paper, we propose a new incremental clustering approach called incremental multiple medoids-based fuzzy clustering (IMMFC) to handle complex patterns that are not compact and well separated. We would like to investigate whether IMMFC is a good alternative to capturing the underlying data structure more accurately. IMMFC not only facilitates the selection of multiple medoids for each cluster in a data chunk, but also has the mechanism to make use of relationships among those identified medoids as side information to help the final data clustering process. The detailed problem formulation, updating rules derivation, and the in-depth analysis of the proposed IMMFC are provided. Experimental studies on several large datasets that include real world malware datasets have been conducted. IMMFC outperforms existing incremental fuzzy clustering approaches in terms of clustering accuracy and robustness to the order of data. These results demonstrate the great potential of IMMFC for large-data analysis.

M-learning in Nigerian higher education: an experimental study with Edmodo <=SEP=> Social media technologies have recently gained remarkable popularity in the education sector. Recent research indicated that students rely on social media for educational purposes like social networking, chatting, and knowledge sharing. With social media aiding learning experiences, m-learning is anticipated to improve the application of social media. In the paper, we investigated the preference of social media tools and mobile devices for learning, their benefits and effectiveness, and how they can possibly improve learning process in Nigeria. Furthermore, we evaluated learning experiences of students using Edmodo social media-based learning environment in a Nigerian university. We used a mixed method research approach, where the data was triangulated from two questionnaires and students’ interviews. Data gathered after the course shows that students learning response was improved. Students were eager to use mobile devices for accessing social media sites, thus providing experimental evidence of the place of social media in m-learning.

Scientific Workflow: A Survey and Research Directions <=SEP=> Workflow technologies are emerging as the dominant approach to coordinate groups of distributed services. However with a space filled with competing specifications, standards and frameworks from multiple domains, choosing the right tool for the job is not always a straightforward task. Researchers are often unaware of the range of technology that already exists and focus on implementing yet another proprietary workflow system. As an antidote to this common problem, this paper presents a concise survey of existing workflow technology from the business and scientific domain and makes a number of key suggestions towards the future development of scientific workflow systems.

Deliberation Networks: Sequence Generation Beyond One-Pass Decoding <=SEP=> The encoder-decoder framework has achieved promising progress for many sequence generation tasks, including machine translation, text summarization, dialog system, image captioning, etc. Such a framework adopts an one-pass forward process while decoding and generating a sequence, but lacks the deliberation process: A generated sequence is directly used as final output without further polishing. However, deliberation is a common behavior in human’s daily life like reading news and writing papers/articles/books. In this work, we introduce the deliberation process into the encoder-decoder framework and propose deliberation networks for sequence generation. A deliberation network has two levels of decoders, where the first-pass decoder generates a raw sequence and the second-pass decoder polishes and refines the raw sentence with deliberation. Since the second-pass deliberation decoder has global information about what the sequence to be generated might be, it has the potential to generate a better sequence by looking into future words in the raw sentence. Experiments on neural machine translation and text summarization demonstrate the effectiveness of the proposed deliberation networks. On the WMT 2014 English-to-French translation task, our model establishes a new state-of-the-art BLEU score of 41.5.

Real-Time Semantic Segmentation with Label Propagation <=SEP=> Despite of the success of convolutional neural networks for semantic image segmentation, CNNs cannot be used for many applications due to limited computational resources. Even efficient approaches based on random forests are not efficient enough for real-time performance in some cases. In this work, we propose an approach based on superpixels and label propagation that reduces the runtime of a random forest approach by factor 192 while increasing the segmentation accuracy.

Mitigation of Denial of Service Attack with Hardware Trojans in NoC Architectures <=SEP=> As Multiprocessor System-on-Chips (MPSoCs) continue to scale, security for Network-on-Chips (NoCs) is a growing concern as rogue agents threaten to infringe on the hardware's trust and maliciously implant Hardware Trojans (HTs) to undermine their reliability. The trustworthiness of MPSoCs will rely on our ability to detect Denial-of-Service (DoS) threats posed by the HTs and mitigate HTs in a compromised NoC to permit graceful network degradation. In this paper, we propose a new light-weight target-activated sequential payload (TASP) HT model that performs packet inspection and injects faults to create a new type of DoS attack. Faults injected are used to trigger a response from error correction code (ECC) schemes and cause repeated retransmission to starve network resources and create deadlocks capable of rendering single-application to full chip failures. To circumvent the threat of HTs, we propose a heuristic threat detection model to classify faults and discover HTs within compromised links. To prevent further disruption, we propose several switch-to-switch link obfuscation methods to avoid triggering of HTs in an effort to continue using links instead of rerouting packets with minimal overhead (1-3 cycles). Our proposed modifications complement existing fault detection and obfuscation methods and only adds 2% in area overhead and 6% in excess power consumption in the NoC micro-architecture.

Quadrifilar helical antennas : Wideband and multiband behavior for GPS applications <=SEP=> A novel multi-frequency printed quadrifilar helix antenna based on Multiple arm technique is presented in this paper. Double frequency and satisfactory antenna characteristics are achieved. The antenna has relatively compact size and hemispherical pattern with excellent circularly polarized coverage. The antenna is designed and simulated with the application of HFSS software. The simulation results and analyses are presented.

A Broadband Passive Monopulse Comparator MMIC <=SEP=> A broadband monopulse comparator MMIC (Monolithic Microwave Integrated Circuit) based on GaAs process is presented in this Letter. The comparator network constructed by three magic tees and one lumped power divider is proposed for one sum channel and two delta channels. The measurement results show that very wide frequency band from 15 to 30 GHz (66.7% relatively frequency bandwidth) with less than 2.5-dB loss can be achieved for the sum channel. And the null depth is more than 22 dB in 15–27GHz and 17 dB in 27–30 GHz for two delta channels. The total chip size is 3.4 mm 3.4 mm (<inline-formula> <tex-math notation="LaTeX">$0.26\lambda _{0}~0.26\lambda _{0}$ </tex-math></inline-formula> at the center frequency of 22.5 GHz).

A Framework for the Automatic Extraction of Rules from Online Text <=SEP=>