Paper: Dispatching Ambulances using Deep Reinforcement Learning (ICLR-2024)

Reviewer comment (Experiments):
Since your metric is waiting time, why not compare it to an agent that sends the ambulance that takes the shortest time to get to the location? It is obvious that today's maps have a time of arrival (TOA) estimation function. The method of using the shortest distance compared in the paper is not persuasive enough, because the closest distance does not necessarily represent the fastest arrival speed. At the same time, the authors did not compare other reinforcement learning methods mentioned in background section and Table 1.

Author response:
This was the original plan for this paper. However reproducibility has been prioritised in this case. Most TOA apis have rate limits and costs, which is why openstreetmap is used in this case. Furthermore this paper compares itself to how the EMS today dispatches ambulances, which is by the shortest euclidean distance. A discussion among other STOA is added to the revision.

Reviewer comment (Evaluation):
How fast does the proposed method perform? Because dispatch of an ambulance is an urgent matter, it is also important to be able to make the decision in the shortest time.

Author response:
I dont have specific numbers at this time. However i can assure you it is fast.

Reviewer comment (Experiments):
As related to the above weakness, given that the paper mentions results from the random agent being likely due to the small environment considered, how would the model perform in larger urban areas with more complex terrains and road networks?

Author response:
We appreciate the encouraging comments from the reviewer, and refer to the updated paper for futher improvements.  We agree that a study of other and larger urban areas would be of great interest.  However, real-world EMS data is extremely hard to come back and we had not access to such data at the time of doing this research and writing the paper.  We have included this topic as a topic for future research.

Reviewer comment (Experiments):
The experiment section is highly under-developed: The comparisons made to baselines are neither rigorous, nor complete. I would've expected to see how the proposed PPO adoption in this particular setting compares to other RL-based baselines. Why was PPO the right choice?

Author response:
# The experiment section is highly under-developed ...
Added a extra figure and paragraph to experiment 2. Implementing multiple STOA is outside of the scope of this paper, implementing all the logic in the simulation and dataset preparation is difficult as is. The simulation in this paper is as close to a real world scenario as any other paper on ambulance dispatching that i've seen.

Reviewer comment (Novelty):
The work lacks novelty: To me the work appears to just be an implementation of PPO algorithm in a particular use-case (ambulance dispatching). I feel the work lacks novelty both in problem and solution. Even if the solution isn't novel, I'd like to understand what makes this problem technically hard and why PPO is the right choice as a solution. Hence, I don't find the paper to meet the bar.

Author response:
# The work lacks novelty ...
As written in the revision: 

"Primarily the novelty in this work lies in the integration of the many complex real world aspects of the
ambulance dispatching problem. Especially the integration of incident priority, incident queue, am-
bulance shifts and ambulance availability in unison. Integration of these aspects in a RL aspect is
hard, due to the many dimensions such as time, location, availablity and priority." - introduction - contributions

"Nasrollahzadeh et al. (2018) and Bandara et al. (2012) were the only
found papers found which considers RL and incident priority to ambulance dispatching."

There is also a "choice of model" which explains why PPO were chosen as a model, and a discussion among other STOA.
This is not a review paper, but an overview of previous research is also given.

Reviewer comment (Evaluation):
Emergency response systems typically follow hard constraints on response time (e.g., maximize number of patients served within D minutes). I failed to understand how the proposed reward function considers such threshold values. Even in experimental results, only mean response times are shown. Having percentage of requests from different severity levels served within threshold times would have been an ideal metric.

Author response:
This would have been good for future work. I understand the concerns on the hard constrains, however the makeshift survival function incorporates how much more important a acute incident is compared to an urgent, as a function of time. Which is more usable as a RL reward function. The contraints provided by the EMS is also somewhat unclear, as they do not give a clear distinction between "sparse" and "densley" populated areas.

Reviewer comment (Experiments):
There is a high chance that the simulator is biased towards historical demand distributions. As the spatio-temporal demand patterns typically change over time, some risk analysis is required on whether the assignment policy is overfitted. Moreover, it is not clear how the policy will adapt to dynamic nature of demand patterns.

Author response:
Yes it can be overfitted to historical demand. This is one of the reasons a method for synthetic incident generation is outlined. Furthermore it is possible to train the model while in use, in a online fashion (e.g train the model with the previous week).

Reviewer comment (Experiments):
Why is the proposed method not compared with other deep RL based dispatching methods?

Author response:
With more time disposal this coud have been done. But since Euclidean distance is currently in use by the ems, this is the standard this paper uses. Furthermore implementation of RL dispatching methods takes time, which was limited in this scenario. This is a paper written out from my master thesis. Futhermore ive yet to see papers on ambulance dispatching comparing different RL implementaions.

Reviewer comment (Evaluation):
Have you done any analysis on what percentage of requests are served within threshold time for different priorities of incidents?

Author response:
This would have been good for future work. I understand the concerns on the hard constrains, however the makeshift survival function incorporates how much more important a acute incident is compared to an urgent, as a function of time. Which is more usable as a RL reward function. The contraints provided by the EMS is also somewhat unclear, as they do not give a clear distinction between "sparse" and "densley" populated areas.

Reviewer comment (Experiments):
Can the simulator or learned policy be overfitted to historical demand? How to continuously adapt the policy to changes in demand patterns?

Author response:
Yes it can be overfitted to historical demand. This is one of the reasons a method for synthetic incident generation is outlined. Furthermore it is possible to train the model while in use, in a online fashion (e.g train the model with the previous week).

Reviewer comment (Reproducibility):
Usually, the action space in RL is fixed, but here the action space can change depending upon the number of available ambulances and patient requests. How are you dealing with this dynamic action space?

Author response:
This was delt with by using action space masking. Which is not optimal since retraining is needed if more ambulances than during training is introduced. This is why this problem is not novel, there are many dimensions which change in size. These are dimensions such as availability, time, space and incident priority.