Query: Natural actor-critic algorithms

Relevant passages:
Infinite-Horizon Policy-Gradient Estimation <=SEP=> Gradient-based approaches to direct policy search in reinf orcement learning have received much recent attention as a means to solve problems of partial observability and to avoid some of the problems associated with policy degradation in value-f unction methods. In this paper we introduceGPOMDP, a simulation-based algorithm for generating a bi sedestimate of the gradient of theaverage rewardin Partially Observable Markov Decision Processes ( POMDPs) controlled by parameterized stochastic policies. A similar algorithm wa s proposed by Kimura, Yamamura, and Kobayashi (1995). The algorithm’s chief advantages are tha t i requires storage of only twice the number of policy parameters, uses one free parameter 2 [0; 1) (which has a natural interpretation in terms of bias-variance trade-off), and requires no knowl edge of the underlying state. We prove convergence of GPOMDP, and show how the correct choice of the parameter is related to the mixing timeof the controlledPOMDP. We briefly describe extensions of GPOMDP to controlled Markov chains, continuous state, observation and control s paces, multiple-agents, higher-order derivatives, and a version for training stochastic policie s with internal states. In a companion paper (Baxter, Bartlett, & Weaver, 2001) we show how the gradient e stimates generated by GPOMDP can be used in both a traditional stochastic gradient algori thm and a conjugate-gradient procedure to find local optima of the average reward.

Autonomous Inverted Helicopter Flight via Reinforcement Learning <=SEP=> Helicopters have highly stochastic, nonlinear, dynamics, and autonomous helicopter flight is widely regarded to be a challenging control problem. As helicopters are highly unstable at low speeds, it is particularly difficult to design controllers for low speed aerobatic maneuvers. In this paper, we describe a successful application of reinforcement learning to designing a controller for sustained inverted flight on an autonomous helicopter. Using data collected from the helicopter in flight, we began by learning a stochastic, nonlinear model of the helicopter’s dynamics. Then, a reinforcement learning algorithm was applied to automatically learn a controller for autonomous inverted hovering. Finally, the resulting controller was successfully tested on our autonomous helicopter platform.

Dynamic Programming and Optimal Control <=SEP=> This is an updated version of Chapter 4 of the author’s Dynamic Programming and Optimal Control, Vol. II, 4th Edition, Athena Scientific, 2012. It includes new material, and it is substantially revised and expanded (it has more than doubled in size). The new material aims to provide a unified treatment of several models, all of which lack the contractive structure that is characteristic of the discounted problems of Chapters 1 and 2: positive and negative cost models, deterministic optimal control (including adaptive DP), stochastic shortest path models, and risk-sensitive models. Here is a summary of the new material:

On-line Q-learning using connectionist systems <=SEP=> Reinforcement learning algorithms are a powerful machine learning technique. However, much of the work on these algorithms has been developed with regard to discrete nite-state Markovian problems, which is too restrictive for many real-world environments. Therefore, it is desirable to extend these methods to high dimensional continuous state-spaces, which requires the use of function approximation to generalise the information learnt by the system. In this report, the use of back-propagation neural networks (Rumelhart, Hinton and Williams 1986) is considered in this context. We consider a number of di erent algorithms based around Q-Learning (Watkins 1989) combined with the Temporal Di erence algorithm (Sutton 1988), including a new algorithm (Modi ed Connectionist Q-Learning), and Q( ) (Peng and Williams 1994). In addition, we present algorithms for applying these updates on-line during trials, unlike backward replay used by Lin (1993) that requires waiting until the end of each trial before updating can occur. On-line updating is found to be more robust to the choice of training parameters than backward replay, and also enables the algorithms to be used in continuously operating systems where no end of trial conditions occur. We compare the performance of these algorithms on a realistic robot navigation problem, where a simulated mobile robot is trained to guide itself to a goal position in the presence of obstacles. The robot must rely on limited sensory feedback from its surroundings, and make decisions that can be generalised to arbitrary layouts of obstacles. These simulations show that on-line learning algorithms are less sensitive to the choice of training parameters than backward replay, and that the alternative update rules of MCQ-L and Q( ) are more robust than standard Q-learning updates. 1

Autonomous Helicopter Flight via Reinforcement Learning <=SEP=> Autonomous helicopter flight represents a challenging cont rol problem, with complex, noisy, dynamics. In this paper, we describe a s uccessful application of reinforcement learning to autonomous helic opter flight. We first fit a stochastic, nonlinear model of the helicopter dy namics. We then use the model to learn to hover in place, and to fly a number of maneuvers taken from an RC helicopter competition.

Latent syntactic structure-based sentiment analysis <=SEP=> People share their opinions about things like products, movies and services using social media channels. The analysis of these textual contents for sentiments is a gold mine for marketing experts, thus automatic sentiment analysis is a popular area of applied artificial intelligence. We propose a latent syntactic structure-based approach for sentiment analysis which requires only sentence-level polarity labels for training. Our experiments on three domains (movie, IT products, restaurant) show that a sentiment analyzer that exploits syntactic parses and has access only to sentence-level polarity annotation for in-domain sentences can outperform state-of-the-art models that were trained on out-domain parse trees with sentiment annotation for each node of the trees. In practice, millions of sentence-level polarity annotations are usually available for a particular domain thus our approach is applicable for training a sentiment analyzer for a new domain while it can exploit the syntactic structure of sentences as well.

Boosting the Quality of Approximate String Matching by Synonyms <=SEP=> A string-similarity measure quantifies the similarity between two text strings for approximate string matching or comparison. For example, the strings “Sam” and “Samuel” can be considered to be similar. Most existing work that computes the similarity of two strings only considers syntactic similarities, for example, number of common words or q-grams. While this is indeed an indicator of similarity, there are many important cases where syntactically-different strings can represent the same real-world object. For example, “Bill” is a short form of “William,” and “Database Management Systems” can be abbreviated as “DBMS.” Given a collection of predefined synonyms, the purpose of this article is to explore such existing knowledge to effectively evaluate the similarity between two strings and efficiently perform similarity searches and joins, thereby boosting the quality of approximate string matching.
 In particular, we first present an expansion-based framework to measure string similarities efficiently while considering synonyms. We then study efficient algorithms for similarity searches and joins by proposing two novel indexes, called SI-trees and QP-trees, which combine signature-filtering and length-filtering strategies. In order to improve the efficiency of our algorithms, we develop an estimator to estimate the size of candidates to enable an online selection of signature filters. This estimator provides strong low-error, high-confidence guarantees while requiring only logarithmic space and time costs, thus making our method attractive both in theory and in practice. Finally, the experimental results from a comprehensive study of the algorithms with three real datasets verify the effectiveness and efficiency of our approaches.

Augmented Reality with Hololens: Experiential Architectures Embedded in the Real World <=SEP=> Early hands-on experiences with the Microsoft Hololens augmented/mixed reality device are reported and discussed, with a general aim of exploring basic 3D visualization. A range of usage cases are tested, including data visualization and immersive data spaces, in-situ visualization of 3D models and full scale architectural form visualization. Ultimately, the Hololens is found to provide a remarkable tool for moving from traditional visualization of 3D objects on a 2D screen, to fully experiential 3D visualizations embedded in the real world.

Aspect based sentiment analysis <=SEP=> In this fast paced and social media frenzy world, decision making has been revolutionized as there are lots of opinions floating on the internet in the form of blogs, social media updates, forums etc. This paper focuses on how banks can utilize these reviews to improve their services. The reviews are scraped from the internet and the sentiment analysis is done on an aspect level to have a much clear idea of where the banks are performing bad according to the customers or users.

Real time object detection and tracking using the Kalman Filter embedded in single board in a robot <=SEP=> This paper presents an algorithm implemented in real time in a laboratory robot, designed to trace and detect the color of an object in an indoor environment using the Kalman filter. The image capture is done by a kinect camera installed in the robot. The algorithm is programmed into a single board computer, BeagleBone Black that uses the ROS — OpenCV system in image processing.

A Case for Malleable Thread-Level Linear Algebra Libraries: The LU Factorization With Partial Pivoting <=SEP=> We propose two novel techniques for overcoming load-imbalance encountered when implementing so-called look-ahead mechanisms in relevant dense matrix factorizations for the solution of linear systems. Both techniques target the scenario where two thread teams are created/activated during the factorization, with each team in charge of performing an independent task/branch of execution. The first technique promotes worker sharing (WS) between the two tasks, allowing the threads of the task that completes first to be reallocated for use by the costlier task. The second technique allows a fast task to alert the slower task of completion, enforcing the early termination (ET) of the second task, and a smooth transition of the factorization procedure into the next iteration. The two mechanisms are instantiated via a new malleable thread-level implementation of the basic linear algebra subprograms, and their benefits are illustrated via an implementation of the LU factorization with partial pivoting enhanced with look-ahead. Concretely, our experimental results on an Intel-Xeon system with 12 cores show the benefits of combining WS+ET, reporting competitive performance in comparison with a task-parallel runtime-based solution.

A generic statistical approach for spam detection in Online Social Networks <=SEP=> 25 26 27 28 29 30 31 32 33 34 35 36 Article history: Received 13 February 2012 Received in revised form 18 March 2013 Accepted 4 April 2013 Available online xxxx

Fast 3D mapping in highly dynamic environments using normal distributions transform occupancy maps <=SEP=> Autonomous vehicles operating in real-world industrial environments have to overcome numerous challenges, chief among which is the creation and maintenance of consistent 3D world models. This paper focuses on a particularly important challenge: mapping in dynamic environments. We introduce several improvements to the recently proposed Normal Distributions Transform Occupancy Map (NDT-OM) aimed for efficient mapping in dynamic environments. A careful consistency analysis is given based on convergence and similarity metrics specifically designed for evaluation of NDT maps in dynamic environments. We show that in the context of mapping with known poses the proposed method results in improved consistency and in superior runtime performance, when compared against 3D occupancy grids at the same size and resolution. Additionally, we demonstrate that NDT-OM features real-time performance in a highly dynamic 3D mapping and tracking scenario with centimeter accuracy over a 1.5km trajectory.

CHAPTER TWO Self-Distancing : Theory , Research , and Current Directions <=SEP=> When people experience negative events, they often try to understand their feelings to improve the way they feel. Although engaging in this meaning-making process leads people to feel better at times, it frequently breaks down leading people to ruminate and feel worse. This raises the question: What factors determine whether people’s attempts to “work-through” their negative feelings succeed or fail? In this article, we describe an integrative program of research that has addressed this issue by focusing on the role that self-distancing plays in facilitating adaptive self-reflection. We begin by describing the “self-reflection puzzle” that initially motivated this line of work. Next, we introduce the concept of self-distancing and describe the conceptual framework we developed to explain how this process should facilitate adaptive self-reflection. After describing the early studies that evaluated this framework, we discuss how these findings have been extended to broaden and deepen our understanding of the role that this process plays in self-regulation. We conclude by offering several parting thoughts that integrate the ideas discussed in this chapter. 1. THE SELF-REFLECTION PUZZLE Many people try to understand their feelings when they are upset, under the assumption that doing so will lead them to feel better. Indeed, it would seem that many of us reflexively heed Socrates’ advice to “know thyself ” when we experience emotional pain. But are people’s attempts to work-through their feelings productive? Do they actually lead people to feel better? A great deal of research has addressed these questions over the past 40 years, and the results reveal a puzzle. On the one hand, several studies suggest that it is indeed helpful for people to reflect on their emotions when they experience distress 82 E. Kross and O. Ayduk

An Embedded Support Vector Machine <=SEP=> In this paper we work on the balance between hardware and software implementation of a machine learning algorithm, which belongs to the area of statistical learning theory. We use system-on-chip technology to demonstrate the potential usefulness of moving the critical sections of an algorithm into HW: the so-called hardware/software balance. Our experiments show that the approach can achieve speedups using a complex machine learning algorithm called a support vector machine. The experiments are conducted on a real-time Java virtual machine named Java optimized processor

Toward an E-orientation platform: Using hybrid recommendation systems <=SEP=> Choosing the right career can be difficult for students, because they have to take into considerations many elements, in order to be on the best path. Several studies in many disciplines have cooperate to help students making their career decision. The purpose of this work is to invest the subject of school orientation, identifying its background and studying some related works. This paper aims also to define the student's profile, and conceiving basis of a recommendation system that will play the advisor role in students helping to choose their career path.

Personalized News Recommendation: A Review and an Experimental Investigation <=SEP=> Online news articles, as a new format of press releases, have sprung up on the Internet. With its convenience and recency, more and more people prefer to read news online instead of reading the paper-format press releases. However, a gigantic amount of news events might be released at a rate of hundreds, even thousands per hour. A challenging problem is how to effciently select specific news articles from a large corpus of newly-published press releases to recommend to individual readers, where the selected news items should match the reader's reading preference as much as possible. This issue refers to personalized news recommendation. Recently, personalized news recommendation has become a promising research direction as the Internet provides fast access to real-time information from multiple sources around the world. Existing personalized news recommendation systems strive to adapt their services to individual users by virtue of both user and news content information. A variety of techniques have been proposed to tackle personalized news recommendation, including content-based, collaborative filtering systems and hybrid versions of these two. In this paper, we provide a comprehensive investigation of existing personalized news recommenders. We discuss several essential issues underlying the problem of personalized news recommendation, and explore possible solutions for performance improvement. Further, we provide an empirical study on a collection of news articles obtained from various news websites, and evaluate the effect of different factors for personalized news recommendation. We hope our discussion and exploration would provide insights for researchers who are interested in personalized news recommendation.

KNN Model-Based Approach in Classification <=SEP=> The k-Nearest-Neighbours (kNN) is a simple but effective method for classification. The major drawbacks with respect to kNN are (1) its low efficiency being a lazy learning method prohibits it in many applications such as dynamic web mining for a large repository, and (2) its dependency on the selection of a “good value” for k. In this paper, we propose a novel kNN type method for classification that is aimed at overcoming these shortcomings. Our method constructs a kNN model for the data, which replaces the data to serve as the basis of classification. The value of k is automatically determined, is varied for different data, and is optimal in terms of classification accuracy. The construction of the model reduces the dependency on k and makes classification faster. Experiments were carried out on some public datasets collected from the UCI machine learning repository in order to test our method. The experimental results show that the kNN based model compares well with C5.0 and kNN in terms of classification accuracy, but is more efficient than the standard kNN.

Paying for performance: Performance incentives increase desire for the reward object. <=SEP=> The current research examines how exposure to performance incentives affects one's desire for the reward object. We hypothesized that the flexible nature of performance incentives creates an attentional fixation on the reward object (e.g., money), which leads people to become more desirous of the rewards. Results from 5 laboratory experiments and 1 large-scale field study provide support for this prediction. When performance was incentivized with monetary rewards, participants reported being more desirous of money (Study 1), put in more effort to earn additional money in an ensuing task (Study 2), and were less willing to donate money to charity (Study 4). We replicated the result with nonmonetary rewards (Study 5). We also found that performance incentives increased attention to the reward object during the task, which in part explains the observed effects (Study 6). A large-scale field study replicated these findings in a real-world setting (Study 7). One laboratory experiment failed to replicate (Study 3). (PsycINFO Database Record

Childhood sexual abuse and psychiatric disorder in young adulthood: II. Psychiatric outcomes of childhood sexual abuse. <=SEP=> OBJECTIVE
This is the second in a series of articles that describe the prevalence, correlates, and consequences of childhood sexual abuse (CSA) in a birth cohort of more than 1,000 New Zealand children studied to the age of 18 years. This article examines the associations between reports of CSA at age 18 and DSM-IV diagnostic classifications at age 18.


METHOD
A birth cohort of New Zealand children was studied at annual intervals from birth to age 16 years. At age 18 years retrospective reports of CSA prior to age 16 and concurrently measured psychiatric symptoms were obtained.


RESULTS
Those reporting CSA had higher rates of major depression, anxiety disorder, conduct disorder, substance use disorder, and suicidal behaviors than those not reporting CSA (p < .002). There were consistent relationships between the extent of CSA and risk of disorder, with those reporting CSA involving intercourse having the highest risk of disorder. These results persisted when findings were adjusted for prospectively measured childhood family and related factors. Similar but less marked relationships between CSA and nonconcurrently measured disorders were found.


CONCLUSIONS
The findings suggest that CSA, and particularly severe CSA, was associated with increased risk of psychiatric disorder in young adults even when due allowance was made for prospectively measured confounding factors.

A stochastic data discrimination based autoencoder approach for network anomaly detection <=SEP=> Machine learning based network anomaly detection methods, which are already effective defense mechanisms against known network intrusion attacks, have also proven themselves to be more successful on the detection of zero-day attacks compared to other types of detection methods. Therefore, research on network anomaly detection using deep learning is getting more attention constantly. In this study we created an anomaly detection model based on a deterministic autoencoder, which discriminates normal and abnormal data by using our proposed stochastic threshold determination approach. We tested our proposed anomaly detection model on the NSL-KDD's test dataset KDDTest+ and obtained an accuracy of 88.28%. The experimental results show that our proposed anomaly detection model can perform almost the same as the most successful and up-to-date machine learning based anomaly detection models in the literature.

Complementary Skyrmion Racetrack Memory With Voltage Manipulation <=SEP=> Magnetic skyrmion holds promise as information carriers in the next-generation memory and logic devices, owing to the topological stability, small size, and extremely low current needed to drive it. One of the most potential applications of skyrmion is to design racetrack memory (RM), named Sk-RM, instead of utilizing domain wall. However, current studies face some key design challenges, e.g., skyrmion manipulation, data representation, and synchronization. To address these challenges, we propose here a complementary Sk-RM structure with voltage manipulation. Functionality and performance of the proposed design are investigated with micromagnetic simulations.

Power Attack on Small RSA Public Exponent <=SEP=> In this paper, we present a new attack on RSA when the public exponent is short, for instance 3 or 2 +1, and when the classical exponent randomization is used. This attack works even if blinding is used on the messages. From a Simple Power Analysis (SPA) we study the problem of recovering the RSA private key when non consecutive bits of it leak from the implementation. We also show that such information can be gained from sliding window implementations not protected against SPA.

Understanding and Supporting Cross-Device Web Search for Exploratory Tasks with Mobile Touch Interactions <=SEP=> Mobile devices enable people to look for information at the moment when their information needs are triggered. While experiencing complex information needs that require multiple search sessions, users may utilize desktop computers to fulfill information needs started on mobile devices. Under the context of mobile-to-desktop web search, this article analyzes users’ behavioral patterns and compares them to the patterns in desktop-to-desktop web search. Then, we examine several approaches of using Mobile Touch Interactions (MTIs) to infer relevant content so that such content can be used for supporting subsequent search queries on desktop computers. The experimental data used in this article was collected through a user study involving 24 participants and six properly designed cross-device web search tasks. Our experimental results show that (1) users’ mobile-to-desktop search behaviors do significantly differ from desktop-to-desktop search behaviors in terms of information exploration, sense-making and repeated behaviors. (2) MTIs can be employed to predict the relevance of click-through documents, but applying document-level relevant content based on the predicted relevance does not improve search performance. (3) MTIs can also be used to identify the relevant text chunks at a fine-grained subdocument level. Such relevant information can achieve better search performance than the document-level relevant content. In addition, such subdocument relevant information can be combined with document-level relevance to further improve the search performance. However, the effectiveness of these methods relies on the sufficiency of click-through documents. (4) MTIs can also be obtained from the Search Engine Results Pages (SERPs). The subdocument feedbacks inferred from this set of MTIs even outperform the MTI-based subdocument feedback from the click-through documents.

Ethical Considerations in Artificial Intelligence Courses <=SEP=> The recent surge in interest in ethics in artificial intelligence may leave many educators wondering how to address moral, ethical, and philosophical issues in their AI courses. As instructors we want to develop curriculum that not only prepares students to be artificial intelligence practitioners, but also to understand the moral, ethical, and philosophical impacts that artificial intelligence will have on society. In this article we provide practical case studies and links to resources for use by AI educators. We also provide concrete suggestions on how to integrate AI ethics into a general artificial intelligence course and how to teach a stand-alone artificial intelligence ethics course.

Monocular Visual–Inertial State Estimation With Online Initialization and Camera–IMU Extrinsic Calibration <=SEP=> There have been increasing demands for developing microaerial vehicles with vision-based autonomy for search and rescue missions in complex environments. In particular, the monocular visual–inertial system (VINS), which consists of only an inertial measurement unit (IMU) and a camera, forms a great lightweight sensor suite due to its low weight and small footprint. In this paper, we address two challenges for rapid deployment of monocular VINS: 1) the initialization problem and 2) the calibration problem. We propose a methodology that is able to initialize velocity, gravity, visual scale, and camera–IMU extrinsic calibration on the fly. Our approach operates in natural environments and does not use any artificial markers. It also does not require any prior knowledge about the mechanical configuration of the system. It is a significant step toward plug-and-play and highly customizable visual navigation for mobile robots. We show through online experiments that our method leads to accurate calibration of camera–IMU transformation, with errors less than 0.02 m in translation and 1° in rotation. We compare out method with a state-of-the-art marker-based offline calibration method and show superior results. We also demonstrate the performance of the proposed approach in large-scale indoor and outdoor experiments.

Quasi-online reinforcement learning for robots <=SEP=> This paper describes quasi-online reinforcement learning: while a robot is exploring its environment, in the background a probabilistic model of the environment is built on the fly as new experiences arrive; the policy is trained concurrently based on this model using an anytime algorithm. Prioritized sweeping, directed exploration, and transformed reward functions provide additional speed-ups. The robot quickly learns goal-directed policies from scratch, requiring few interactions with the environment and making efficient use of available computation time. From an outside perspective it learns the behavior online and in real time. We describe comparisons with standard methods and show the individual utility of each of the proposed techniques

High speed network traffic analysis with commodity multi-core systems <=SEP=> Multi-core systems are the current dominant trend in computer processors. However, kernel network layers often do not fully exploit multi-core architectures. This is due to issues such as legacy code, resource competition of the RX-queues in network interfaces, as well as unnecessary memory copies between the OS layers. The result is that packet capture, the core operation in every network monitoring application, may even experience performance penalties when adapted to multi-core architectures. This work presents common pitfalls of network monitoring applications when used with multi-core systems, and presents solutions to these issues. We describe the design and implementation of a novel multi-core aware packet capture kernel module that enables monitoring applications to scale with the number of cores. We showcase that we can achieve high packet capture performance on modern commodity hardware.

RF Localization in Indoor Environment <=SEP=> In this paper indoor localization system based on the RF power measurements of the Received Signal Strength (RSS) in WLAN environment is presented. Today, the most viable solution for localization is the RSS fingerprinting based approach, where in order to establish a relationship between RSS values and location, different machine learning approaches are used. The advantage of this approach based on WLAN technology is that it does not need new infrastructure (it reuses already and widely deployed equipment), and the RSS measurement is part of the normal operating mode of wireless equipment. We derive the Cramér-Rao Lower Bound (CRLB) of localization accuracy for RSS measurements. In analysis of the bound we give insight in localization performance and deployment issues of a localization system, which could help designing an efficient localization system. To compare different machine learning approaches we developed a localization system based on an artificial neural network, k-nearest neighbors, probabilistic method based on the Gaussian kernel and the histogram method. We tested the developed system in real world WLAN indoor environment, where realistic RSS measurements were collected. Experimental comparison of the results has been investigated and average location estimation error of around 2 meters was obtained.

Lateral prefrontal cortex and self-control in intertemporal choice <=SEP=> Disruption of function of left, but not right, lateral prefrontal cortex (LPFC) with low-frequency repetitive transcranial magnetic stimulation (rTMS) increased choices of immediate rewards over larger delayed rewards. rTMS did not change choices involving only delayed rewards or valuation judgments of immediate and delayed rewards, providing causal evidence for a neural lateral-prefrontal cortex–based self-control mechanism in intertemporal choice.