Query: DroidDet: Effective and robust detection of android malware using static analysis along with rotation forest model

Relevant passages:
OPEM: A Static-Dynamic Approach for Machine-Learning-Based Malware Detection <=SEP=> Malware is any computer software potentially harmful to both computers and networks. The amount of malware is growing every year and poses a serious global security threat. Signature-based detection is the most extended method in commercial antivirus software, however, it consistently fails to detect new malware. Supervised machine learning has been adopted to solve this issue. There are two types of features that supervised malware detectors use: (i) static features and (ii) dynamic features. Static features are extracted without executing the sample whereas dynamic ones requires an execution. Both approaches have their advantages and disadvantages. In this paper, we propose for the first time, OPEM, an hybrid unknown malware detector which combines the frequency of occurrence of operational codes (statically obtained) with the information of the execution trace of an executable (dynamically obtained). We show that this hybrid approach enhances the performance of both approaches when run separately.

An Efficient Non-Negative Matrix-Factorization-Based Approach to Collaborative Filtering for Recommender Systems <=SEP=> Matrix-factorization (MF)-based approaches prove to be highly accurate and scalable in addressing collaborative filtering (CF) problems. During the MF process, the non-negativity, which ensures good representativeness of the learnt model, is critically important. However, current non-negative MF (NMF) models are mostly designed for problems in computer vision, while CF problems differ from them due to their extreme sparsity of the target rating-matrix. Currently available NMF-based CF models are based on matrix manipulation and lack practicability for industrial use. In this work, we focus on developing an NMF-based CF model with a single-element-based approach. The idea is to investigate the non-negative update process depending on each involved feature rather than on the whole feature matrices. With the non-negative single-element-based update rules, we subsequently integrate the Tikhonov regularizing terms, and propose the regularized single-element-based NMF (RSNMF) model. RSNMF is especially suitable for solving CF problems subject to the constraint of non-negativity. The experiments on large industrial datasets show high accuracy and low-computational complexity achieved by RSNMF.

Toward Automated Dynamic Malware Analysis Using CWSandbox <=SEP=> Malware is notoriously difficult to combat because it appears and spreads so quickly. In this article, we describe the design and implementation of CWSandbox, a malware analysis tool that fulfills our three design criteria of automation, effectiveness, and correctness for the Win32 family of operating systems

Exploring Multiple Execution Paths for Malware Analysis <=SEP=> Malicious code (or Malware) is defined as software that fulfills the deliberately harmful intent of an attacker. Malware analysis is the process of determining the behavior and purpose of a given Malware sample (such as a virus, worm, or Trojan horse). This process is a necessary step to be able to develop effective detection techniques and removal tools. Currently, Malware analysis is mostly a manual process that is tedious and time-intensive. To mitigate this problem, a number of analysis tools have been proposed that automatically extract the behavior of an unknown program by executing it in a restricted environment and recording the operating system calls that are invoked. The problem of dynamic analysis tools is that only a single program execution is observed. Unfortunately, however, it is possible that certain malicious actions are only triggered under specific circumstances (e.g., on a particular day, when a certain file is present, or when a certain command is received). In this paper, we propose a system that allows us to explore multiple execution paths and identify malicious actions that are executed only when certain conditions are met. This enables us to automatically extract a more complete view of the program under analysis and identify under which circumstances suspicious actions are carried out. Our experimental results demonstrate that many Malware samples show different behavior depending on input read from the environment. Thus, by exploring multiple execution paths, we can obtain a more complete picture of their actions.

Random Forests <=SEP=> Random forests are a combination of tree predictors such that each tree depends on the values of a random vector sampled independently and with the same distribution for all trees in the forest. The generalization error for forests converges a.s. to a limit as the number of trees in the forest becomes large. The generalization error of a forest of tree classifiers depends on the strength of the individual trees in the forest and the correlation between them. Using a random selection of features to split each node yields error rates that compare favorably to Adaboost (Y. Freund & R. Schapire, Machine Learning: Proceedings of the Thirteenth International conference, ***, 148–156), but are more robust with respect to noise. Internal estimates monitor error, strength, and correlation and these are used to show the response to increasing the number of features used in the splitting. Internal estimates are also used to measure variable importance. These ideas are also applicable to regression.

Optimizing number of hidden neurons in neural networks <=SEP=> In this paper, a novel and effective criterion based on the estimation of the signal-to-noise-ratio figure (SNRF) is proposed to optimize the number of hidden neurons in neural networks to avoid overfitting in the function approximation. SNRF can quantitatively measure the useful information left unlearned so that overfitting can be automatically detected from the training error only without use of a separate validation set. It is illustrated by optimizing the number of hidden neurons in a multi-layer perceptron (MLP) using benchmark datasets. The criterion can be further utilized in the optimization of other parameters of neural networks when overfitting needs to be considered.

Neural Relation Extraction via Inner-Sentence Noise Reduction and Transfer Learning <=SEP=> Extracting relations is critical for knowledge base completion and construction in which distant supervised methods are widely used to extract relational facts automatically with the existing knowledge bases. However, the automatically constructed datasets comprise amounts of low-quality sentences containing noisy words, which is neglected by current distant supervised methods resulting in unacceptable precisions. To mitigate this problem, we propose a novel word-level distant supervised approach for relation extraction. We first build Sub-Tree Parse (STP) to remove noisy words that are irrelevant to relations. Then we construct a neural network inputting the subtree while applying the entity-wise attention to identify the important semantic features of relational words in each instance. To make our model more robust against noisy words, we initialize our network with a priori knowledge learned from the relevant task of entity classification by transfer learning. We conduct extensive experiments using the corpora of New York Times (NYT) and Freebase. Experiments show that our approach is effective and improves the area of Precision/Recall (PR) from 0.35 to 0.39 over the state-of-the-art work.

An Evaluation of Aggregation Techniques in Crowdsourcing <=SEP=> As the volumes of AI problems involving human knowledge are likely to soar, crowdsourcing has become essential in a wide range of world-wide-web applications. One of the biggest challenges of crowdsourcing is aggregating the answers collected from the crowd since the workers might have wide-ranging levels of expertise. In order to tackle this challenge, many aggregation techniques have been proposed. These techniques, however, have never been compared and analyzed under the same setting, rendering a ‘right’ choice for a particular application very difficult. Addressing this problem, this paper presents a benchmark that offers a comprehensive empirical study on the performance comparison of the aggregation techniques. Specifically, we integrated several stateof-the-art methods in a comparable manner, and measured various performance metrics with our benchmark, including computation time, accuracy, robustness to spammers, and adaptivity to multi-labeling. We then provide in-depth analysis of benchmarking results, obtained by simulating the crowdsourcing process with different types of workers. We believe that the findings from the benchmark will be able to serve as a practical guideline for crowdsourcing applications.

ONTOLOGY ALIGNMENT USING MACHINE LEARNING TECHNIQUES <=SEP=> In semantic web, ontology plays an important role to provide formal definitions of concepts and relationships. Therefore, communicating similar ontologies becomes essential to provide ontologies interpretability and extendibility. Thus, it is inevitable to have similar but not the same ontologies in a particular domain, since there might be several definitions for a given concept. This paper presents a method to combine similarity measures of different categories without having ontology instances or any user feedbacks towards aligning two given ontologies. To align different ontologies efficiently, K Nearest Neighbor (KNN), Support Vector Machine (SVM), Decision Tree (DT) and AdaBoost classifiers are investigated. Each classifier is optimized based on the lower cost and better classification rate. Experimental results demonstrate that the F-measure criterion improves to 99% using feature selection and combination of AdaBoost and DT classifiers, which is highly comparable, and outperforms the previous reported F-measures. Computer Engineering Department, Shahid Chamran University, Ahvaz, Iran.

Pictogrammar: an AAC device based on a semantic grammar <=SEP=> As many as two-thirds of individuals with an Autism Spectrum Disorder (ASD) also have language impairments, which can range from mild limitations to complete non-verbal behavior. For such cases, there are several Augmentative and Alternative Communication (AAC) devices available. These are computer-designed tools in order to help people with ASD to palliate or overcome such limitations, at least partially. Some of the most popular AAC devices are based on pictograms, so that a pictogram is the graphical representation of a simple concept and sentences are composed by concatenating a number of such pictograms. Usually, these tools have to manage a vocabulary made up of hundreds of pictograms/concepts, with no or very poor knowledge of the language at semantic and pragmatic level. In this paper we present Pictogrammar, an AAC system which takes advantage of SUpO and PictOntology. SUpO (Simple Upper Ontology) is a formal semantic ontology which is made up of detailed knowledge of facts of everyday life such as simple words, with special interest in linguistic issues, allowing automated grammatical supervision. PictOntology is an ontology developed to manage sets of pictograms, linked to SUpO. Both ontologies make possible the development of tools which are able to take advantage of a formal semantics.

Multi-modal Mean-Fields via Cardinality-Based Clamping <=SEP=> Mean Field inference is central to statistical physics. It has attracted much interest in the Computer Vision community to efficiently solve problems expressible in terms of large Conditional Random Fields. However, since it models the posterior probability distribution as a product of marginal probabilities, it may fail to properly account for important dependencies between variables. We therefore replace the fully factorized distribution of Mean Field by a weighted mixture of such distributions, that similarly minimizes the KL-Divergence to the true posterior. By introducing two new ideas, namely, conditioning on groups of variables instead of single ones and using a parameter of the conditional random field potentials, that we identify to the temperature in the sense of statistical physics to select such groups, we can perform this minimization efficiently. Our extension of the clamping method proposed in previous works allows us to both produce a more descriptive approximation of the true posterior and, inspired by the diverse MAP paradigms, fit a mixture of Mean Field approximations. We demonstrate that this positively impacts real-world algorithms that initially relied on mean fields.

Design and Implementation of a Reliable Wireless Real-Time Home Automation System Based on Arduino Uno Single-Board Microcontroller <=SEP=> —This paper presents design and implementation concepts for a wireless real-time home automation system based on Arduino Uno microcontroller as central controllers. The proposed system has two operational modes. The first one is denoted as a manually–automated mode in which the user can monitor and control the home appliances from anywhere over the world using the cellular phone through Wi-Fi communication technology. The second one is referred to a self-automated mode that makes the controllers to be capable of monitoring and controlling different appliances in the home automatically in response to the signals comes from the related sensors. To support the usefulness of the proposed technique, a hardware implementation with Matlab-GUI platform for the proposed system is carried out and the reliability of the system is introduced. The proposed system is shown to be a simple, cost effective and flexible that making it a suitable and a good candidate for the smart home future. I. INTRODUCTION ecently, man's work and life are increasingly tight with the rapid growth in communications and information technology. The informationized society has changed human being's way of life as well as challenged the traditional residence. Followed by the rapid economic expansion, living standard keeps raising up day by day that people have a higher requirement for dwelling functions. The intellectualized society brings diversified information where safe, economic, comfortable and convenient life has become the ideal for every modern family [1].

Security Keys: Practical Cryptographic Second Factors for the Modern Web <=SEP=> “Security Keys” are second-factor devices that protect users against phishing and man-in-the-middle attacks. Users carry a single device and can self-register it with any online service that supports the protocol. The devices are simple to implement and deploy, simple to use, privacy preserving, and secure against strong attackers. We have shipped support for Security Keys in the Chrome web browser and in Google’s online services. We show that Security Keys lead to both an increased level of security and user satisfaction by analyzing a two year deployment which began within Google and has extended to our consumer-facing web applications. The Security Key design has been standardized by the FIDO Alliance, an organization with more than 250 member companies spanning the industry. Currently, Security Keys have been deployed by Google, Dropbox, and GitHub. An updated and extended tech report is available at https://github.com/google/u2fref-code/docs/SecurityKeys_TechReport.pdf.

A web-based bayesian intelligent tutoring system for computer programming <=SEP=> In this paper, we present a Web-based intelligent tutoring system, called BITS. The decision making process conducted in our intelligent system is guided by a Bayesian network approach to support students in learning computer programming. Our system takes full advantage of Bayesian networks, which are a formal framework for uncertainty management in Artificial Intelligence based on probability theory. We discuss how to employ Bayesian networks as an inference engine to guide the students’ learning processes . In addition, we describe the architecture of BITS and the role of each module in the system. Whereas many tutoring systems are static HTML Web pages of a class textbook or lecture notes, our intelligent system can help a student nav igate through the online course materials, recommend learning goals, and generate appropriate reading sequences.

Adjustable Autonomy : From Theory to Implementation <=SEP=> Recent exciting, ambitious applications in agent technology involve agents acting individually or in teams in support of critical activities of individual humans or entire human organizations. Applications range from intelligent homes [13], to "routine" organizational coordination[16], to electronic commerce[4] to long-term space missions[12, 6]. These new applications have brought forth an increasing interest in agents’ adjustable autonomy (AA), i.e., in agents’ dynamically adjusting their own level of autonomy based on the situation[8]. In fact, many of these applications will not be deployed, unless reliable AA reasoning is a central component. At the heart of AA is the question of whether and when agents should make autonomous decisions and when they should transfer decision-making control to other entities (e.g., human users). Unfortunately, previous work in adjustable autonomy has focused on individual agent-human interactions and tile techniques developed fail to scale-up to complex heterogeneous organizations. Indeed, as a first step, we focused on a smallscale, but real-world agent-human organization called Electric Elves, where an individual agent and human worked together within a larger multiagent context. Although’the application limits the interactions among entities, key weaknesses of previous approaches to adjustable autonomy are readily apparent. In particular, previous approaches to transferof-control are seen to be too rigid, employing one-shot transfersof-control that can result in unacceptable coordination failures. Furthermore, the previous approaches ignore potential costs (e.g., from delays) to an agent’s team due to such transfers of control. To remedy such problems, we propose a novel approach to AA, based on the notion of a transfer-of-control strategy. A transfer-of-control strategy consists of a conditional sequence of two types of actions: (i) actions to transfer decision-making control (e.g., from the agent to the user or vice versa) and (ii) actions to change an agent’s pre-specified coordination constraints with team members, aimed at minimizing miscoordination costs. The goal is for high quality individual decisions to be made with minimal disruption to the coordination of the team. We operationalize such strategies via Markov decision processes (MDPs) which select the optimal strategy given an uncertain environment and costs to individuals and teams. We have developed a general reward function and state representation for such an MDP, to facilitate application of the approach to different domains. We present results from a careful evaluation of this approach, including via its use in our real-world, deployed Electric Elves system.

The Granular Tabu Search and Its Application to the Vehicle-Routing Problem <=SEP=> 

An Item-based Collaborative Filtering Recommendation Algorithm Using Slope One Scheme Smoothing <=SEP=> Collaborative filtering is one of the most important technologies in electronic commerce. With the development of recommender systems, the magnitudes of users and items grow rapidly, resulted in the extreme sparsity of user rating data set. Traditional similarity measure methods work poor in this situation, make the quality of recommendation system decreased dramatically. Poor quality is one major challenge in collaborative filtering recommender systems. Sparsity of users’ ratings is the major reason causing the poor quality. To address this issue, an item-based collaborative filtering recommendation algorithm using slope one scheme smoothing is presented. This approach predicts item ratings that users have not rated by the employ of slope one scheme, and then uses Pearson correlation similarity measurement to find the target items’ neighbors, lastly produces the recommendations. The experiments are made on a common data set using different recommender algorithms. The results show that the proposed approach can improve the accuracy of the collaborative filtering recommender system.

Design methodology using inversion coefficient for low-voltage low-power CMOS voltage reference <=SEP=> This paper presents an analog design methodology, using the selection of inversion coefficient of MOS devices, to design low voltage and low-power (LVLP) CMOS voltage references. These circuits often work under subthreshold operation. Hence, there is a demand for analog design methods that optimize the sizing process of transistors working in weak and moderate inversion. The advantage of the presented method -- compared with the traditional approach to design circuits -- is the reduction of design cycle time and minimization of trial-and-error simulations, if the proposed equations are used. As a case study, a LVLP voltage reference based on subthreshold MOSFETs with supply voltage of 0.7 V was designed for 0.18-¼m CMOS technology.

Exploring commonalities across participants in the neural representation of objects. <=SEP=> The question of whether the neural encodings of objects are similar across different people is one of the key questions in cognitive neuroscience. This article examines the commonalities in the internal representation of objects, as measured with fMRI, across individuals in two complementary ways. First, we examine the commonalities in the internal representation of objects across people at the level of interobject distances, derived from whole brain fMRI data, and second, at the level of spatially localized anatomical brain regions that contain sufficient information for identification of object categories, without making the assumption that their voxel patterns are spatially matched in a common space. We examine the commonalities in internal representation of objects on 3T fMRI data collected while participants viewed line drawings depicting various tools and dwellings. This exploratory study revealed the extent to which the representation of individual concepts, and their mutual similarity, is shared across participants.

Hierarchy of fibrillar organization levels in the polytene interphase chromosomes of Chironomus. <=SEP=> 

Video Paragraph Captioning Using Hierarchical Recurrent Neural Networks <=SEP=> We present an approach that exploits hierarchical Recurrent Neural Networks (RNNs) to tackle the video captioning problem, i.e., generating one or multiple sentences to describe a realistic video. Our hierarchical framework contains a sentence generator and a paragraph generator. The sentence generator produces one simple short sentence that describes a specific short video interval. It exploits both temporal-and spatial-attention mechanisms to selectively focus on visual elements during generation. The paragraph generator captures the inter-sentence dependency by taking as input the sentential embedding produced by the sentence generator, combining it with the paragraph history, and outputting the new initial state for the sentence generator. We evaluate our approach on two large-scale benchmark datasets: YouTubeClips and TACoS-MultiLevel. The experiments demonstrate that our approach significantly outperforms the current state-of-the-art methods with BLEU@4 scores 0.499 and 0.305 respectively.

Autonomous Self-Assembly in Swarm-Bots <=SEP=> In this paper, we discuss the self-assembling capabilities of the swarm-bot, a distributed robotics concept that lies at the intersection between collective and self-reconfigurable robotics. A swarm-bot is comprised of autonomous mobile robots called s-bots. S-bots can either act independently or self-assemble into a swarm-bot by using their grippers. We report on experiments in which we study the process that leads a group of s-bots to self-assemble. In particular, we present results of experiments in which we vary the number of s-bots (up to 16 physical robots), their starting configurations, and the properties of the terrain on which self-assembly takes place. In view of the very successful experimental results, swarm-bot qualifies as the current state of the art in autonomous self-assembly

Facial Expression Recognition Using Enhanced Deep 3D Convolutional Neural Networks <=SEP=> Deep Neural Networks (DNNs) have shown to outperform traditional methods in various visual recognition tasks including Facial Expression Recognition (FER). In spite of efforts made to improve the accuracy of FER systems using DNN, existing methods still are not generalizable enough in practical applications. This paper proposes a 3D Convolutional Neural Network method for FER in videos. This new network architecture consists of 3D Inception-ResNet layers followed by an LSTM unit that together extracts the spatial relations within facial images as well as the temporal relations between different frames in the video. Facial landmark points are also used as inputs to our network which emphasize on the importance of facial components rather than the facial regions that may not contribute significantly to generating facial expressions. Our proposed method is evaluated using four publicly available databases in subject-independent and cross-database tasks and outperforms state-of-the-art methods.

A Java API and Web Service Gateway for wireless M-Bus <=SEP=> M-Bus Systems are used to support the remote data exchange with meter units through a network. They provide an easy extend able and cost effective method to connect many meter devices to one coherent network. Master applications operate as concentrated data collectors and are used to process measured values further. As modern approaches facilitate the possibility of a Smart Grid, M-Bus can be seen as the foundation of this technology. With the current focus on a more effective power grid, Smart Meters and Smart Grids are an important research topic. This bachelor thesis first gives an overview of the M-Bus standard and then presents a Java library and API to access M-Bus devices remotely in a standardized way through Web Services. Integration into common IT applications requires interoperable interfaces which also facilitate automated machine-to-machine communication. The oBIX (Open Building Information Exchange) standard provides such standardized objects and thus is used to create a Web Service gateway for the Java API.

AWEsome: An open-source test platform for airborne wind energy systems <=SEP=> In this paper we present AWEsome (Airborne Wind Energy Standardized Open-source Model Environment), a test platform for airborne wind energy systems that consists of low-cost hardware and is entirely based on open-source software. It can hence be used without the need of large financial investments, in particular by research groups and startups to acquire first experiences in their flight operations, to test novel control strategies or technical designs, or for usage in public relations. Our system consists of a modified off-the-shelf model aircraft that is controlled by the pixhawk autopilot hardware and the ardupilot software for fixed wing aircraft. The aircraft is attached to the ground by a tether. We have implemented new flight modes for the autonomous tethered flight of the aircraft along periodic patterns. We present the principal functionality of our algorithms. We report on first successful tests of these modes in real flights. ∗Author names are in alphabetical order. a Universität Bonn, bechtle@physik.uni-bonn.de b Universität Bonn, thomasgehrmann@gmx.net c Humboldt-Universität zu Berlin, csieg@physik.hu-berlin.de d Daidalos Capital, zillmann@daidalos-capital.com awesome.physik.uni-bonn.de 1 ar X iv :1 70 4. 08 69 5v 1 [ cs .S Y ] 2 7 A pr 2 01 7 Section

Optimization of Belt-Type Electrostatic Separation of Triboaerodynamically Charged Granular Plastic Mixtures <=SEP=> Electrostatic separation is frequently employed for the selective sorting of conducting and insulating materials for waste electric and electronic equipment. In a series of recent papers, the authors have presented several novel triboelectrostatic separation installations for the recycling of the main categories of plastics contained in information technology wastes. The aim of the present work is to optimize the design and the operation of such an installation, composed of a belt-type electrostatic separator associated to a propeller-type tribocharging device. The experimental design method is employed for modeling and optimizing the separation of a mixture of high-impact polystyrene and acrylonitrile butadiene styrene granules originating from shredded out-of-use computer cases. A distinct experiment is carried out on three synthetic mixtures of virgin plastic granules: 75% polyamide (PA)+ 25% polycarbonate (PC), 50% PA+ 50% PC, and 25% PA + 75% PC. The best results are obtained for the mixtures containing equal quantities of the two constituents.

Transformerless micro-inverter for grid-connected photovoltaic systems <=SEP=> The leakage currents caused by high-frequency common-mode (CM) voltage have become a major concern in transformerless photovoltaic (PV) inverters. This paper addresses to a review on dc-ac converters applied to PV systems that can avoid the circulation of leakage currents. Looking for a lower cost and higher reliability solution, a 250 W PV transformerless micro-inverter prototype based on the bipolar full-bridge topology was built and tested. As it is confirmed by experimental results, this topology is able to maintain the CM voltage constant and prevent the circulation of CM currents through the circuit.

GQ ( λ ) : A general gradient algorithm for temporal-difference prediction learning with eligibility traces <=SEP=> A new family of gradient temporal-difference learning algorithms have recently been introduced by Sutton, Maei and others in which function approximation is much more straightforward. In this paper, we introduce the GQ(λ) algorithm which can be seen as extension of that work to a more general setting including eligibility traces and off-policy learning of temporally abstract predictions. These extensions bring us closer to the ultimate goal of this work—the development of a universal prediction learning algorithm suitable for learning experientially grounded knowledge of the world. Eligibility traces are essential to this goal because they bridge the temporal gaps in cause and effect when experience is processed at a temporally fine resolution. Temporally abstract predictions are also essential as the means for representing abstract, higher-level knowledge about courses of action, or options. GQ(λ) can be thought of as an extension of Q-learning. We extend existing convergence results for policy evaluation to this setting and carry out a forward-view/backwardview analysis to derive and prove the validity of the new algorithm. Introduction One of the main challenges in artificial intelligence (AI) is to connect the low-level experience to high-level representations (grounded world knowledge). Low-level experience refers to rich signals received back and forth between the agent and the world. Recent theoretical developments in temporal-difference learning combined with mathematical ideas developed for temporally abstract options, known as intra-option learning, can be used to address this challenge (Sutton, 2009). Intra-option learning (Sutton, Precup, and Singh, 1998) is seen as a potential method for temporalabstraction in reinforcement learning. Intra-option learning is a type of off-policy learning. Off-policy learning refers to learning about a target policy while following another policy, known as behavior policy. Offpolicy learning arises in Q-learning where the target policy is a greedy optimal policy while the behavior policy is exploratory. It is also needed for intra-option learning. Intra-option methods look inside options and allow AI agent to learn about multiple different options simultaneously from a single stream of received data. Option refers to a temporally course of actions with a termination condition. Options are ubiquitous in our everyday life. For example, to go for hiking, we need to consider and evaluate multiple options such as transportation options to the hiking trail. Each option includes a course of primitive actions and only is excited in particular states. The main feature of intra-option learning is its ability to predict the consequences of each option policy without executing it while data is received from a different policy. Temporal difference (TD) methods in reinforcement learning are considered as powerful techniques for prediction problems. In this paper, we consider predictions always in the form of answers to the questions. Questions are like “If of follow this trail, would I see a creek?” The answers to such questions are in the form of a single scalar (value function) that tells us about the expected future consequences given the current state. In general, due to the large number of states, it is not feasible to compute the exact value of each state entry. One of the key features of TD methods is their ability to generalize predictions to states that may not have visited; this is known as function approximation. Recently, Sutton et al. (2009b) and Maei et al. (2009) introduced a new family of gradient TD methods in which function approximation is much more straightforward than conventional methods. Prior to their work, the existing classical TD algorithms (e.g.; TD(λ) and Q-learning) with function approximation could become unstable and diverge (Baird, 1995; Tsitsiklis and Van Roy, 1997). In this paper, we extend their work to a more general setting that includes off-policy learning of temporally abstract predictions and eligibility traces. Temporally abstract predictions are essential for representing higher-level knowledge about the course of actions, or options (Sutton et al., 1998). Eligibility traces bridge between the temporal gaps when experience is processes at a temporally fine resolution. In this paper, we introduce the GQ(λ) algorithm that can be thought of as an extension to Q-learning (Watkins and Dayan, 1989); one of the most popular off-policy learning algorithms in reinforcement learning. Our algorithm incorporates gradient-descent ideas originally developed by Sutton et al. (2009a,b), for option conditional predictions with varying eligibility traces. We extend existing convergence results for policy evaluation to this setting and carry forward-view/backwardview analysis and prove the validity of the new algorithm. The organization of the paper is as follows: First, we describe the problem setting and define our notations. Then we introduce the GQ(λ) algorithm and describe how to use it. In the next sections we provide derivation of the algorithm and carry out analytical analysis on the equivalence of TD forward-view/backward-view. We finish the paper with convergence proof and conclusion section. Notation and background We consider the problem of policy evaluation in finite state-action Markov Decision Process (MDP). Under standard conditions, however, our results can be extended to MDPs with infinite state–action pairs. We use a standard reinforcement learning (RL) framework. In this setting, data is obtained from a continually evolving MDP with states st ∈ S, actions at ∈ A, and rewards rt ∈ <, for t = 1, 2, . . ., with each state and reward as a function of the preceding state and action. Actions are chosen according to the behavior policy b, which is assumed fixed and exciting, b(s, a) > 0,∀s, a. We consider the transition probabilities between state– action pairs, and for simplicity we assume there is a finite number N of state–action pairs. Suppose the agent find itself at time t in a state– action pair st, at. The agent likes its answer at that time to tell something about the future sequence st+1, at+1, . . . , st+k if actions from t + 1 on were taken according to the option until it terminated at time t+k. The option policy is denoted π : S × A → [0, 1] and whose termination condition is denoted β : S → [0, 1]. The answer is always in the form of a single number, and of course we have to be more specific about what we are trying to predict. There are two common cases: 1) we are trying to predict the outcome of the option; we want to know about the expected value of some function of the state at the time the option terminates. We call this function the outcome target function, and denote it z : S → <, 2) we are trying to predict the transient; that is, what happens during the option rather than its end. The most common thing to predict about the transient is the total or discounted reward during the option. We denote the reward function r : S ×A → <. Finally, the answer could conceivably be a mixture of both a transient and an outcome. Here we will present the algorithm for answering questions with both an outcome part z and a transient part r, with the two added together. In the common place where one wants only one of the two, the other is set to zero. Now we can start to state the goal of learning more precisely. In particular, we would like our answer to be equal to the expected value of the outcome target function at termination plus the cumulative sum of the transient reward function along the way: Q(st, at) (1) ≡ E [ rt+1 + γrt+2 + · · ·+ γrt+k + zt+k | π, β ] , where γ ∈ (0, 1] is discount factor and Q(s, a) denotes action value function that evaluates policy π given state-action pair s, a. To simplify the notation, from now on, we drop the superscript π on action values. In many problems the number of state-action pairs is large and therefore it is not feasible to compute the action values for each state-action entry. Therefore, we need to approximate the action values through generalization techniques. Here, we use linear function approximation; that is, the answer to a question is always formed linearly as Qθ(s, a) = θ>φ(s, a) ≈ Q(s, a) for all s ∈ S and a ∈ A, where θ ∈ < is a learned weight vector and φ(s, a) ∈ < indicates a state–action feature vector. The goal is to learn parameter vector θ through a learning method such as TD learning. The above (1) describes the target in a Monte Carlo sense, but of course we want to include the possibility of temporal-difference learning; one of the widely used techniques in reinforcement learning. To do this, we provide an eligibility-trace function λ : S → [0, 1] as described in Sutton and Barto (1998). We let eligibilitytrace function, λ, to vary over different states. In the next section, first we introduce GQ(λ); a general temporal-difference learning algorithm that is stable under off-policy training, and show how to use it. Then in later sections we provide the derivation of the algorithm and convergence proof. The GQ(λ) algorithm In this section we introduce the GQ(λ) algorithm for off-policy learning about the outcomes and transients of options, in other words, intra-option GQ(λ) for learning the answer to a question chosen from a wide (possibly universal) class of option-conditional predictive questions. To specify the question one provides four functions: π and β, for the option, and z and r, for the target functions. To specify how the answers will be formed one provides their functional form (here in linear form), the feature vectors φ(s, a) for all state–action pairs, and the eligibility-trace function λ. The discount factor γ can be taken to be 1, and thus ignored; the same effect as discounting can be achieved through the choice of β. Now, we specify the GQ(λ) algorithm as follows: The weight vector θ ∈ < is initialized arbitrarily. The secondary weight vector w ∈ < is init

Improved Shortest Path Maps with GPU Shaders <=SEP=> We present in this paper several improvements for computing shortest path maps using OpenGL shaders [1]. The approach explores GPU rasterization as a way to propagate optimal costs on a polygonal 2D environment, producing shortest path maps which can efficiently be queried at run-time. Our improved method relies on Compute Shaders for improved performance, does not require any CPU pre-computation, and handles shortest path maps both with source points and with line segment sources. The produced path maps partition the input environment into regions sharing a same parent point along the shortest path to the closest source point or segment source. Our method produces paths with global optimality, a characteristic which has been mostly neglected in animated virtual environments. The proposed approach is particularly suitable for the animation of multiple agents moving toward the entrances or exits of a virtual environment, a situation which is efficiently represented with the proposed path maps.

Efficient Program Analyses Using Deductive and Semantic Methodologies <=SEP=> Program analysis is the process of gathering deeper insights about a source code and analysing them to resolve software problems of arbitrary complexity. The key challenge in program analysis is to keep it fast, precise and straightforward. This research focuses on three key objectives to achieve an efficient program analysis: (i) expressive data representation, (ii) optimised data structure and (iii) fast data processing mechanisms. State of the art technologies such as Resource Description Framework (RDF) as data representation format, triplestores as the storage &amp; processing layer, and datalog to represent program analysis rules are considered in our research. diagram(BDD) to be embedded in the triplestore. Additionally, an ontology is being designed to standardise the definitions of concepts and representation of the knowledge in the program analysis domain.