pwc-archive/papers-with-abstracts huggingface dataset:
Policy Gradient for items Recommendation on Virtual Taobao
Authors: Wei Fang, Fanyuan Zeng
Date: 2020-12-14 00:00:00
Proceeding: CUHK Course IERG5350 2020 12

Recent years have witnessed digital content appear with plenty of forms (including online courses, online shopping and e-news) in daily life of people, which has provided with opportunities as well as challenges for systems to provide users with personalized services and information. The goal of our project is to design a recommender algorithm that can return a good list such that the consumers might have high chance of clicking the items on a simulated environment named Virtual Taobao, a simulator trained from the real-data from Taobao. Firstly, We tried some state-of-art deep-reinforcement algorithms, such as deep deterministic policy gradient (DDPG) method and Twin Delayed DDPG (TD3), what's more, we also used the Proximal Policy Optimisation (PPO) algorithm and tried to improve the PPO algorithm with the features of the attributes of the consumers.
Video Link: https://drive.google.com/file/d/1WVoAjKcJ-4t5o6n_U5KaoDn7BymksYqr/view?usp=sharing
Learning Mobile Robot Navigation in the Dense Crowd with Deep Reinforcement Learning
Authors: Keyu Li, Ye Lu
Date: 2020-12-14 00:00:00
Proceeding: CUHK Course IERG5350 2020 12
Tasks: Decision Making, Deep Reinforcement Learning, reinforcement-learning, Reinforcement Learning (RL), Robot Navigation, Sequential Decision Making

In recent years, the growing demand for more intelligent service robots is pushing the development of mobile robot navigation algorithms. Moving in a dense crowd safely and efficiently is an important yet challenging task for the operation of the mobile robots. Reinforcement learning (RL) approaches have shown superior ability in solving sequential decision making problems, and recent work has explored its potential to learn navigation polices in a socially compliant manner. In our project, we propose to apply an RL framework to develop a smart system for human-aware navigation. We propose to use value function approximation based methods to learn human-aware navigation policies and implement the reward shaping, hindsight experience replay, and curriculum learning techniques for our task. The effectiveness of our methods is validated in a simulated human-aware navigation environment. The video demonstration is at \url{https://mycuhk-my.sharepoint.com/:v:/g/personal/1155131468_link_cuhk_edu_hk/ERJ6SdHXiVZNrT2FiyapPaEBYg0RxWedc2h6_tM-X1q0iQ?e=q2recC}.
IPM Move Planner: AN EFFICIENT EXPLOITING DEEP REINFORCEMENT LEARNING WITH MONTE CARLO TREE SEARCH
Authors: Fan Bai, Fei Meng
Date: 2020-12-14 00:00:00
Proceeding: CUHK Course IERG5350 2020 12
Tasks: Blocking, Deep Reinforcement Learning, Reinforcement Learning (RL)

The literature about scene rearrangement focus on developing a move planner totransform a pair of layouts on a limited plane.  Poorly considered moving planscan cause blocking each other and waste of manpower.  Currently, the planningperformance by using a Deep Q-network (DQN) based Monte Carlo Tree Search(MCTS) algorithm catches researchers’ attention due to its powerful sequencedecision-making ability.  However, the deterministic value-based policy of thereinforcement learning (RL) agent cannot represent multiple optimal actions andDQN network increases computation cost significantly, especially when beingcombined with MCTS. To enhance the ability of MCTS to select the optimalaction sequence efficiently, we develop a novel IPM Move Planner1, which isthe MCTS embedded with Proximal Policy Optimization (PPO) supervised byimitation learning (IL). To be precise, we let PPO explore optimal action sequencethrough exploiting the good previous trajectories efficiently; then the policy andvalue neural network plays a crucial part in the expansion and simulation processes.The primary policy network with residual structure is pre-trained by imitatingthe teacher policy through supervised learning, and finally is put into the overallframework to learn an advanced policy, contributing to a complex policy only with alightweight network. Experiments on synthetic layouts demonstrate that IPM MovePlanner generates an outstanding moving performance with less transportationlength and higher success rate than the state of the art.
Increasing Data Efficiency of Driving Agent By World Model
Authors: Yicheng Liu, CAO Qianqian
Date: 2020-12-14 00:00:00
Proceeding: CUHK Course IERG5350 2020 12
Tasks: Autonomous Driving, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

Reinforcement learning algorithms for real-world autonomous driving must be able to handle complex, unknown dynamical systems. This requirement is han- dled well by model-free algorithm such as PPO. However, model-free approach tend to be substantially less sample-efficient. In this work, we aim to retain the advantages of model-free method and increase the stability and data-efficiency of PPO. To this end we proposed a world model to model popular reinforcement learning environments through compressed spatio-temporal representations, which allow model-free method learning behaviors from imagined outcomes to increase sample-efficiency. The experimental results indicate that our approach mitigating the inefficiency of PPO, increasing the stability, and largely reducing the train- ing time. code is available at www.github.com/Mrmoore98/World-Model.git. The video can be found here.
Optimization of Multi-Factor Model in Quantitative Trading Based On Reinforcement Learning
Authors: Dylan Zhang, Xiaotong LIN
Date: 2020-12-14 00:00:00
Proceeding: CUHK Course IERG5350 2020 12
Tasks: Decision Making, reinforcement-learning, Reinforcement Learning (RL)

Quantitative trading strategies play an important role in stock trading, and reinforcement learning (RL) has been increasingly applied to trading activities in recent years. In this paper, we mainly study the optimization of multi-factor model in quantitative trading by using the method of RL, that we train an agent with a series of historical trading data of the stock market. From the experiment results, we can see that RL is feasible in solving and optimizing similar investment decision-making problems in financial field, which can help us to obtain more stable returns.Eventually, we hope that our simple work can make more and more people notice the application of RL in investment.
Reinforcement Learning Based Adaptive WalkingAssistance Control of a Lower Limb Exoskeleton
Authors: Yingying Wang, Yutian SHEN
Date: 2020-12-14 00:00:00
Proceeding: CUHK Course IERG5350 2020 12
Tasks: reinforcement-learning, Reinforcement Learning (RL)

Lower limb exoskeleton for people suffering from unilateral weakness or even hemiparesis has attracted considerable interest in recent years. For walking assistance in such scenarios, the exoskeleton is expected to help generate symmetrically coordinated motion with the unaffected side. However, control strategies remains a challenge as the exoskeleton system is an under-actuated system with some degree of freedoms remains passive and it includes human-machine interaction making it more complex than fully actuated robotic systems where classical PID controller can achieve satisfying performance. In this project, we propose a reinforcement learning based controller to provide walking assistance based on a modeled leader-follower system where the task can be reformulated as a motion trajectory tracking problem. The proposed control method is validated in simulation platform with data obtained from a healthy subject imitating hemiplegia patients with a pre-developed set of single-side lower limb exoskeleton.
To Rip or not to Rip: A Reinforcement Learning-based Rip-up and Reroute Algorithm for Global Routing
Authors: Fangzhou Wang, Wanying ZHENG
Date: 2020-12-14 00:00:00
Proceeding: CUHK Course IERG5350 2020 12

Routing, including global routing and detailed routing, has been a critical step in
the design of integrated circuits. Most of the existing global routers will firstly use
techniques like pattern routing and layer assignment to quickly generate a routing
solution and optimize total wirelength and via usage. Then rip-up and reroute
(RRR) scheme will be applied to iteratively reduce the number of overflows in the
whole design. However, compared with initial routing stage, RRR will be much
more time consuming. It will rip up all the nets that pass through overflowed area
and reroute them sequentially. Even if the routing resources in one routing cell
is overused by 1, the router will rip up all the nets that are routed on the routing
cell, as it does not know which net will be the best choice to rip up. In this way,
RRR may be doing a lot of redundant work. Besides, some initial routing solutions
that are optimal in terms of wirelength will also be wasted when they are ripped
up, causing a loss of routing quality. Therefore, in this project, we propose to use
reinforcement learning to help decide which nets to rip up in each RRR iterations.
An actor-critic based Proximal Policy Optimization (PPO) agent is trained for this
task. Experimental results show that the proposed approach can successfully reduce
the number of rerouted nets with little loss of routing quality on the ICCAD’19
global routing contest benchmarks, which demonstrate the effectiveness of our
model.
Automatic Source Code Summarization via Reinforcement Learning
Authors: Zhuangbin Chen
Date: 2020-12-14 00:00:00
Proceeding: CUHK Course IERG5350 2020 12
Tasks: Cloud Computing, Code Summarization, Management, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL), Source Code Summarization

Due to the fast development of computer and software, the volume of today's code has reached an unprecedented level. For large-scale systems (e.g., cloud computing systems) with billions lines of codes, the majority of its maintenance effort is code management. And much of this effort is spent on understanding related source codes. With high-quality code summaries, one can quickly understand what a function does (even without reading the code). However, it’s nontrivial for a programmer to write good comments for source codes. If the code summary can be automatically generated, then we can greatly accelerate the whole pipeline of software development. To this end, in this project we develop a reinforcement learning framework to enhance the automatic generation of code summarization. By properly defining the key modules, we employ an actor-critic network to solve this problem. Experimental results demonstrate that our model outperforms vallina sequence to sequence model by a noticeable margin.
Mobile Robots Autonomous Exploration with Reinforcement Learning
Authors: Haojie Shi, LI ANG
Date: 2020-12-14 00:00:00
Proceeding: CUHK Course IERG5350 2020 12
Tasks: reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

Reinforcement Learning, a computational approach to learning whereby an agent
tries to maximize the total amount of reward it receives while interacting with
a complex and uncertain environment. And it has quite a few applications in
playing game or controlling a robot after exploration and exploitation. Our project
is aimed at designing an efficient robot autonomous exploration algorithm based
on reinforcement learning. Our work is based on the HouseExpo dataset and
we developed reinforcement learning algorithm on it, showing that the robot can
efficiently explore uncertain environments with a smart behavior. Our work
is based on Double DQN with proportional prioritization[7], and we improve
the algorithm by encoding history observation, adding global explored map and
designing external reward via Artificial Potential Field.
Mobile Robots Exploration via Deep Reinforcement Learning
Authors: Han Ma, Peng Xu
Date: 2020-12-14 00:00:00
Proceeding: CUHK Course IERG5350 2020 12
Tasks: Deep Reinforcement Learning, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

n this paper,  we try to solve the mobile robot exploration problem in a 2D indoor office environment by deep reinforcement learning. Finally, we find that PPO with Convolutional Neural Network (CNN) and Dense layers converge well in our problem. The video link of our current work is here.
Learn to Play Tetris with Deep Reinforcement Learning
Authors: Hanyuan Liu, Lixin Liu
Date: 2020-12-14 00:00:00
Proceeding: CUHK Course IERG5350 2020 12
Tasks: Deep Reinforcement Learning, Imitation Learning, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

Tetris is one of the most popular video games ever created, perhaps in part because its difficulty makes it addictive. 
In this course project, we successfully trained a DQN agent in a simplified Tetris environment with state-action pruning. 
This simple agent is able to deal with the Tetris Problem with reasonable performance. 
We also applied several state of the art reinforcement learning algorithms such as Dreamer, DrQ, and Plan2Explore in the real-world Tetris game environment. 
We augment the Dreamer algorithm with imitation learning as Lucid Dreamer algorithm. 
Our experiments demonstrate that the mentioned state of art methods and their variants fail to play the original Tetris game. 
The complex state-action space make original Tetris a quite difficult game for non-population based reinforcement learning agents. 
Video link and related resources: https://drive.google.com/drive/folders/14aounKtRyg28azhtPcRsnymKdlhAgfDM?usp=sharing
A Coarse-to-Fine Auto-Sampler For Long-tailed Image Recognition
Authors: Tong Wu, Hao Li
Date: 2020-12-14 00:00:00
Proceeding: CUHK Course IERG5350 2020 12
Tasks: Data Augmentation, Representation Learning

The long-tail distributed data in the real world has always been a great challenge for deep learning. Current approaches to mitigate the long-tail issue include re-sampling and data augmentation. However, the hand-crafted re-sampling and augmentation strategies are sub-optimal. In this paper, we propose a coarse-to-fine auto-sampler that re-samples and augments imbalanced data automatically using reinforcement learning. The sampler consists of two parts: a patch sampler for augmenting data during representation learning, and a class-wise weighted sampler for classifier learning. Experiments on standard long-tailed datasets including CIFAR-10/100-LT and ImageNet-LTT shows the effectiveness of our method. We also conduct visualization to better demonstrate the policy of our agents. Our code will be made public before the final revision. Our video is available at https://drive.google.com/file/d/ 1oamWNap3rC3Ulyd5jSs1ZrZIQiBB-iR1/view?usp=sharing.
Learn To Manage Portfolio With Reinforcement Learning
Authors: Liu Yuan, Zhou Qianyu
Date: 2020-12-14 00:00:00
Proceeding: CUHK Course IERG5350 2020 12
Tasks: Management, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

Abstract：
With the increase of high net worth population, wealth management is getting more and more attention. Traditional portfolio management requires a high professional threshold. Now that reinforcement learning is popular, due to the dynamic and continuity of portfolio management, it is very suitable to use continuous task reinforcement learning methods to solve this problem, so this article attempts to learn existing algorithms(such as PG and PPO) and apply them to this wealth management task.
Video Address：https://drive.google.com/file/d/1zW4NTEbD7TZfLtGMdoJtELqD3dqpW287/view?usp=sharing
Towards Understanding Deep Policy Gradients: A Case Study on PPO
Authors: Buhua Liu, CHONG YIN
Date: 2020-12-14 00:00:00
Proceeding: CUHK Course IERG5350 2020 12
Tasks: Decision Making, Deep Reinforcement Learning, reinforcement-learning, Reinforcement Learning (RL)

Deep reinforcement learning has shown impressive performance on many decision-making problems, where deep policy gradient algorithms prevail in continuous action space tasks. Although many algorithm-level improvements on policy gradient algorithms have been proposed, recent studies have found that code-level optimizations also play a critical role in the claimed enhancement. In this paper, we further investigate several code-level optimizations for the popular Proximal Policy Optimization (PPO) algorithm, aiming to provide insights into the importance of different components in the practical implementations.\footnote{Video presentation is available at \url{https://youtu.be/M0uTLoEUwGQ}}
Reinforcement Learning for Contact-Rich Tasks: Robotic Peg Insertion Strategies
Authors: Jianbang Liu, Ang ZHANG
Date: 2020-12-14 00:00:00
Proceeding: CUHK Course IERG5350 2020 12
Tasks: Contact-rich Manipulation, Deep Reinforcement Learning, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

Vision and touch are especially important when doing contact-rich manipulation
tasks in unstructured environments. It is non-trivial to manually design a robot con-
troller that combines these modalities which have very different characteristics. In
this project, to connect vision and touch, we first equip robots with both visual and
tactile sensors and collect a large-scale dataset of corresponding vision and tactile
sequences. We use self-supervision to learn a compact and multimodal representa-
tion of our sensory inputs, which can then be used to improve the sample efficiency
of our policy learning. We will train a policy in a simulation environment using
deep reinforcement learning algorithms. The learned policy is also transferable to
handle real-world tasks. The peg insertion is chosen as the task for demonstration
in this project. A preliminary version of our python implementation is avail-
able at: https://github.com/Henry1iu/ierg5350_rl_course_project.
A video introducing our project is available at: https://mycuhk-my.
sharepoint.com/:v:/g/personal/1155071948_link_cuhk_edu_hk/
EaKiGmkUvjJOoSqdWxrqjXYBpz3dCSAfOD9Co8krttyqUQ?e=RXsHD2
Reinforcement Learning Based Character Controlling
Authors: Jingbo Wang, Zijing YIN
Date: 2020-12-14 00:00:00
Proceeding: CUHK Course IERG5350 2020 12
Tasks: motion prediction, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)


Character controlling is a longstanding problem in understanding the behavior of human. This task aims to generate various and high quality human motion in the simulated environment as in the real world controlled by human. Therefore, how to use the captured real human motions is the crucial component to solve this problem. Rather than regressing the human motion directly in previous motion prediction methods, in this project, a reinforcement learning method is adopted in to our framework to learn robust control policies capable of imitating a broad range of example motion clips. Besides, we also explore the new reward functions to encourage the motion similarity between the real human and the virtual character. With these new rewards, the algorithm will convergence faster than recent advances.
R-SAC: Reinforcement Sample Consensus
Authors: Zhaoyang Huang, Yan Xu
Date: 2020-12-14 00:00:00
Proceeding: CUHK Course IERG5350 2020 12

The rejection of outliers in observed data is the foundation for accurate model estimation.
Random sample consensus (RANSAC) is a classical algorithm aiming to find the inliers for robust model estimation. 
After sampling a series of minimal sets that can support the hypothesis estimation and generating the respective hypotheses, the best hypothesis that earns the maximum consensus is chosen for the final model estimation. 
However,
this strategy may face exponentially computational growth as the outlier ratio increases.
Besides, fitting a model from a minimal set may hinder the accurate model estimation especially when the inliers are extremely rare.  
In contrast, a model estimated from more observations may be better than from a minimum set.
To approach such problem, we propose reinforcement sample consensus (R-SAC) to train a neural network to classify the inliers and outliers among all the correspondences with reinforcement learning.   
During training, we regard the number of inliers as a reward and encourage the agent to find the optimal subset supporting the final model estimation in a unsupervised manner.  
During inference, the R-SAC network is able to directly generate the inlier 
set, which could significantly reduce the computational resources in sampling and is able to select a more robust model hypothesis fitted from more correspondences.
Empirical results show that our method achieves comparable performance compared with the previous supervised counterparts and remarkable efficiency especially when the outlier ratio is large.
Demystify Painting with RL
Authors: Linning Xu, Yunxiang Zhang
Date: 2020-12-14 00:00:00
Proceeding: CUHK Course IERG5350 2020 12
Tasks: Decision Making, Reinforcement Learning (RL), Sequential Decision Making

Given an image of an arbitrary scene, experienced artists are skillful at accurately perceiving the visual contents within the scene, such as objects, lighting, and tint, and presenting them in different painting styles. Essentially, this artistic creation procedure starts from a blank canvas and proceeds in a stroke-by-stroke manner, which could be modeled as a sequence of carefully-chosen stroke actions. Given the fact that reinforcement learning (RL) offers a principled approach to tackling sequential decision-making tasks, it is natural to consider applying appropriate RL techniques to training machines to mimic the painting procedure accomplished by human artists. In this project, we aim to learn an agent capable of automatically planning a sequence of strokes that result in a painting with desired visual contents and artistic styles, just as what human painters would do during artistic creation. We perform extensive experiments under different algorithmic designs as an attempt to demystify the learning mechanism and capability of current RL-based painting agents.
Ranking Items in Large-Scale Item Search Engines with Reinforcement Learning
Authors: Chang Gao
Date: 2020-12-14 00:00:00
Proceeding: CUHK Course IERG5350 2020 12
Tasks: Decision Making, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

Ranking items in large-scale item search engines such as Amazon and Taobao is a typical multi-step decision-making problem. Due to the interactive nature between the human user and the search engine, reinforcement learning is a natural solution to this problem. In this project, we use Virtual-Taobao as the environment and some effective methods to solve this problem. Experimental results show that TD3 performs best on this problem.
Train a snake with reinforcement learning algorithms
Authors: Ruikai Cai, Condi Zhang
Date: 2020-12-14 00:00:00
Proceeding: CUHK Course IERG5350 2020 12
Tasks: reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

Snake is a typical video game where the player maneuvers a line which grows in length, with the line itself being a primary obstacle.Our goal is to train a more efficient agent than human for Snake.To approoach this goal, firstly we use pygame achieve a simple Snake game as the environment.Because Snake game have so many states that it is impossible to use tabular method represent them and human play Snake and learn it by geting the image of it. We believe that Double-DQN which is a classical algorithm of RL is suitable for our agent training.Besides,Snake is also similar with a walking or climbing process.We note that PPO has good performance for resolving such problem by restricting the variance during training.That is why we want to compare Double-DQN with PPO in our environment.Then we use Double-DQN and PPO to train our agent.At present, we get the results in current environment.We find that Double-DQN is more stable but the peak of PPO is better.