From: ~Yuna_Inamori1
title: Reply to AnonReviewer2
comment: We appreciate the reviewer for the comments, which helps us to improve the quality of the paper.


(1) You mention that some simulated cars drive in straight lines, and others copy "the past trained agents behaviors.” Are these incomplete training behaviors? Are these are incomplete training behaviors? Why not just use human models of varying capability?

“The past trained agents behaviors” means incomplete training agents. In a realistic environment, drivers have different driving skills such as beginner and/or expert. To represent those cars, we introduced the incomplete training agents as drivers having various skills. As you said, introducing human models is reasonable way to construct realistic environment, which is one of our future work on our simulator settings.


(2) I'm a little bit confused about your human baseline: Were the humans who served as a comparison seeing the task for the first time, or did they have time to practice? That being said was it really that in 50% of driving scenarios, humans failed to avoid the other vehicles? And if that’s the case, does it really make sense to use humans as a baseline for building from?

We obtained results of human baseline for the first time but we initially explained the details of the task and vehicle control method on a simulator.
Because, in our experiment, we set an agent and other cars randomly, difficult cases that an agent cannot avoid collisions might be generated. In such cases, it would be difficult for human baseline to avoid collision with other cars. Moreover, we aim to further update the model parameters to acquire better controls than human. Therefore, we set more complex task for human baseline and we compare scores with human baseline and our methods.


(3) Was an episode that was considered a ‘goal score’ always necessarily a survival score?

“Goal score” indicates the number of episodes that an agent changed into the target lane. This score increases even if an agent does not consider collisions in the future and forcibly changes the lane. Therefore, we use a couple of scores to consider lane-changing and collision avoidance.


(4) In training, you’re fixing the number of cars to be 5, but during the experimentation, you’re varying the number of other cars. This seems counterproductive, as your expecting the network to have learned a solution for which it wasn’t trained.

As shown in Fig. 4, our network inputs trimmed images centering at an agent, which captures the neighboring environment. Therefore, if we use the fewer number of other cars in training, we would train much easier cases that other cars do not exist around an agent. Using 5 other cars during training, we can acquire controls to cope with various situations including more difficult and easier cases. Therefore, we fixed the number of other cars as 5.


(5) Seeing as the cars were only able to avoid a collision less than 70% of the time, maybe the case in which all the cars are driving straight isn’t too easy, as you claim. I’d definitely like to see results on that to prove that indeed this method is robust.

As with (2), our lane changing simulator randomly initializes the positions of an agent and other cars, in some cases and difficult situations to avoid collision with other cars are generated. Therefore, survival score remains at about 70%. Improvement of our simulator is included in our future work.


(6) It’s very fascinating that you were able to attempt this on a real vehicle. I’m seriously interested as to how you set up the scenario in which the vehicle was simulated in that environment. I'd love to have seen more details about how the car was configured in an environment with the simulator.

* Experimental set up on a real vehicle operation
This real-world experiment does not use real vehicles of other cars for the sake of safety. We carefully select a scenario that a stable control can be obtained. Tests on the other scenarios is one of our future work.

* Configuration of a real vehicle control from an action of a model
We used “TOYOTA AUTO BODY COMS”, which has been modified by a vender to control steering and throttle (i.e., accelerator and brake pedals) from a laptop and dedicated program. The vehicle controls (steering and throttle) values in a real environment is manually set to reproduce the same behaviors as a simulator. To consider vehicular inertia force and smooth control, we further adopted a PID control approach. As a result, we set the parameters as described in Section 4.D (P=2000, I=100, D=0).


(7) Good on you for using the same network architecture for training the DQN, as well as clearly dictating your reward, training episodes, and RL parameters.

Thank you for the positive remark. We hope that the detailed descriptions about the experimental settings would be helpful for readers to do the same or similar experiments.

From: ~Yuna_Inamori1
title: Reply to AnonReviewer1
comment: We thank the reviewer for the constructive suggestion:

(1)	How do the reward and mean survival steps in Fig. 5 differ for the evaluation table?

As we described in Section 4.C, “survival step” in Fig. 5 is a different metric to “survival score” in Table 3. Survival step in Fig. 5 means the number of action steps until a collision at each episode, which is used for training steps. Meanwhile, survival score in Table 3 means the number of episodes in which an agent did not collide with other cars or obstacles during 300 action steps, which is used for evaluation. Because each method was tested for 100 episodes, survival score takes a value from 0 (worst score) to 100 (best score).

(2)	How does method B differ from imitation learning followed by transfer learning/fine tuning?

Our method can acquire better control than human. In an imitation learning approach, we aim to acquire the same controls as human. Our method also acquires human-like controls at the first training step, which is corresponding to an imitation learning approach. However, we further improve the obtained controls at the second training step. Experimental results demonstrate that our method (prior knowledge B) out performs human baseline.