Towards Trustworthy Automatic Diagnosis Systems by Emulating Doctors’ Reasoning with Deep Reinforcement Learning
Abstract
The automation of the medical evidence acquisition and diagnosis process has recently attracted increasing attention in order to reduce the workload of doctors and democratize access to medical care.
However, most works proposed in the machine learning literature focus solely on improving the prediction accuracy of a patient’s pathology.
We argue that this objective is insuf ﬁ cient to ensure doctors’ acceptability of such systems.
In their initial interaction with patients, doctors do not only focus on identifying the pathology a patient is suffering from; they instead generate a differential diagnosis (in the form of a short list of plausible diseases) because the medical evidence collected from patients is often insuf ﬁ cient to establish a ﬁ nal diagnosis.
Moreover, doctors explicitly explore severe pathologies before potentially ruling them out from the differential, especially in acute care settings.
Finally, for doctors to trust a system’s recommendations, they need to understand how the gathered evidences led to the predicted diseases.
In particular, interactions between a system and a patient need to emulate the reasoning of doctors.
We therefore propose to model the evidence acquisition and automatic diagnosis tasks using a deep reinforcement learning framework that considers three essential aspects of a doctor’s reasoning, namely generating a differential diagnosis using an exploration-con ﬁ rmation approach while prioritizing severe pathologies .
We propose metrics for evaluating interaction quality based on these three aspects.
We show that our approach performs better than existing models while maintaining competitive pathology prediction accuracy.
Introduction
In recent years, the digital healthcare industry has grown rapidly, bene ﬁ ting from advances in machine learning (Esteva et al., 2019; Xiao et al., 2018).
In particular, telemedicine, i.e., healthcare services provided via digital means, has received much attention (Kichloo et al., 2020).
Aiming to reduce the workload of human doctors and thereby broaden access to care by automating parts of the interaction with patients, telemedicine applications typically involve two signi ﬁ cant components, among others: evidence acquisition and automatic diagnosis .
In a typical interaction, a patient ﬁ rst presents his/her chief complaint 1 to the telemedicine system, then the system asks further questions to gather more evidences (i.e., information about additional symptoms the patient might be experiencing and antecedents / risk factors the patient might have), and ﬁ nally makes a prediction regarding the underlying diseases based on all collected evidences.
Importantly, a human doctor would typically review the entire interaction, including the collected evidences and the predicted diseases, before establishing a diagnosis and deciding on next steps (e.g., ordering additional tests, preparing a prescription, etc.).
There are many existing works in the machine learning literature that aim to improve the automated evidence acquisition and disease diagnosis steps, using Reinforcement Learning (RL) (Kao et al., 2018; Wei et al., 2018; Yuan and Yu, 2021), Bayesian networks (Guan and Baral, 2021; Liu et al., 2022), or supervised approaches (Luo et al., 2020; Chen et al., 2022).
These methods are generally trained to collect relevant evidences and to predict the pathology the patient is suffering from, while minimizing the number of questions asked to the patient.
However, these works overlook several crucial aspects of medical history taking, which is the process used by a doctor to interact with a patient with the goal of building the patient’s medical history.
Medical history is a broad concept encompassing various kinds of information relevant to a patient’s current concerns (Nichol et al., 2018).
It includes past illnesses, current symptoms, demographics, etc.
In taking medical history, a doctor organizes the questions asked to a patient under an explorationcon ﬁ rmation framework.
At any point during the interaction, the doctor has in their mind a differential diagnosis , corresponding to a list of diseases worth considering, which might be modi ﬁ ed throughout the interaction, based on the answers provided by the patient.
In forming the differential diagnosis, the doctor treats severe pathologies differently by prioritizing questions that could help ruling them out even if they are less likely.
In a usual in-person clinical setting, a physical examination will then be conducted to seek speci ﬁ c physical signs and improve the sensitivity and speci ﬁ city of the medical history.
A visual examination and/or supervised self-examination can also be conducted in a virtual care setting.
After the medical history and physical examination are completed, the physician will decide if additional tests are necessary to establish the ﬁ nal diagnosis.
Finally, a care plan will be established with the patient and proper follow-up, if needed.
We next provide more details about the main concepts.
Differential diagnosis During the interaction with a patient, a medical doctor considers a set of plausible diseases, known as the differential diagnosis or simply differential , which is re ﬁ ned throughout the interaction based on the information provided by the patient (Henderson et al., 2012;
Guyatt et al., 2002; Rhoads et al., 2017).
The ﬁ nal differential often contains several diseases because the patient’s symptoms and antecedents are insuf ﬁ cient to pinpoint a single pathology. As a result, the doctor might order follow up exams such as imagery and blood works to collect additional information to re ﬁ ne the differential and identify the pathology the patient is suffering from. Prior work on automatic diagnosis primarily focused on predicting the ground truth pathology, i.e., the disease causing the patient’s symptoms (Chen et al., 2022; Zhao et al., 2021; Yuan and Yu, 2021;
Guan and Baral, 2021; Wei et al., 2018; Xu et al., 2019; Kao et al., 2018).
Richens et al.
(2020) considers the differential in their approach; however it is unclear whether all evidences are provided to the model at the same time or if the model is part of the acquisition process.
Our approach considers the differential diagnosis as an essential part of the model.
Considering differentials instead of a single pathology has the added bene ﬁ t of accounting for uncertainty and errors inherent in the diagnosis, especially when decisions are made solely based on interactions with patients and without medical exams or tests.
For this reason, being able to predict the differential rather than the ground truth pathology is an important part of gaining the trust of doctors in a model.
Exploration-con ﬁ rmation
In addition to considering differentials, inquiring about evidences in a manner similar to the way a doctor would engage with patients is another important factor in gaining the trust of doctors in automated systems, and this is largely overlooked in prior works as well.
Doctors generally proceed according to a diagnostic framework that provides an organized and logical approach for building a differential.
More speci ﬁ cally, the interactions conducted by a doctor generally consist of two distinct phases: exploration followed by con ﬁ rmation .
During exploration, the doctor mainly asks questions that keep multiple pathologies under evaluation.
In that phase, pathologies can be removed from or added to the differential depending on the patient’s answers.
During the con ﬁ rmation phase, the doctor inquires about evidences that strengthen the actual differential they are considering (Mansoor, 2018; Richardson and Wilson, 2015; Rhoads et al., 2017).
This two-phase property of the interaction of doctors with patients is a separate dimension of the acquisition process that differs from the objective of predicting differentials, and should therefore be considered and measured separately from the common paradigm of optimizing prediction accuracy.
Severe pathologies Among all possible diseases, severe pathologies receive more attention than others.
Generally, a doctor does not want to miss out on them and would therefore explicitly gather evidences to rule them out from the differential as soon as they become plausible, even if they are less unlikely than other diseases (Rhoads et al., 2017; Ramanayake and Basnayake, 2018).
Emulating this behavior within automated systems would contribute to increasing the trust of doctors in these systems.
In this work, we make a case for explicitly modelling the reasoning of doctors in designing evidence acquisition and automated diagnosis models using RL.
More speci
ﬁ cally, we focus on (1) using the differential diagnosis , rather than the ground truth pathology, as the training target of models, (2) modulating the evidence acquisition process to mimic a doctor’ exploration-con ﬁ rmation approach, and (3) prioritizing severe pathologies .
In the remaining sections, we review existing works and their shortcomings (Section 2); we describe our method that improves upon prior works especially in mimicking the reasoning of doctors (Section 3); we provide empirical results demonstrating the bene ﬁ ts of the proposed approach (Sectionsand 5); ﬁ nally, we discuss the limitations (Section 6) before concluding this work (Section 7).
Our main contributions are: (1) we reformulate the task of evidence acquisition and automated diagnosis by introducing doctors’ trust as a desideratum, and argue for explicitly designing models towards this goal; (2) we propose a RL agent, CASANDE, that promotes the desired behavior in addition to accurate predictions, by means of predicting differentials and reward shaping; (3) we empirically show that existing strong models on this task, even with modi ﬁ cations, are not suf ﬁ cient for the proposed goal of mimicking the reasoning of human doctors, and that CASANDE improves over existing models while being competitive on conventional metrics.
Related work
There have been a variety of prior works on evidence acquisition and automatic diagnosis.
For example, Wei et al.
(2018) proposes to use DQN (Mnih et al., 2015) to tackle this problem.
Kao et al.
(2018) additionally proposes to create a hierarchy of symptoms and diseases based on anatomical parts and train an agent using Hierarchical Reinforcement Learning (Sutton et al., 1999).
Xu et al.
(2019), Zhao et al.
(2021), and Liu et al.
(2022) propose to encode relations among different evidences and evidence-disease pairs to enhance the ef ﬁ ciency of an agent trained using DQN.
Peng et al.
(2018) uses a policy gradient algorithm (Williams, 1992) and relies on reward shaping functions (Ng et al., 1999; Wiewiora et al., 2003; Devlin and Kudenko, 2012) de ﬁ ned in the state space to favour positive evidence acquisition.
Our proposed approach also makes use of reward shaping functions, but unlike the work done in (Peng et al., 2018), those functions are de ﬁ ned in the action space and are used to induce a bias in our agent re ﬂ ecting the reasoning of doctors.
In the aforementioned approaches, the space of the evidences to be collected and the one of the diseases to be predicted are merged together.
Janisch et al.
(2019) considers to deal with those spaces separately and proposes an agent which has two branches, one that decides which evidence to inquire about, trained using RL, and one that predicts the disease, trained using supervised learning.
Other approaches enhance the training of the evidence inquiry branch by using information from the classi ﬁ er branch.
For example, Kachuee et al.
(2019) relies on Monte-Carlo dropout (Gal and
Ghahramani, 2016) to estimate the certainty improvement from the classi ﬁ er branch output and uses that information as an evidence acquisition reward to train the acquisition branch.
Yuan and Yu (2021) proposes an adaptive method to align the tasks performed by the two branches.
More speci ﬁ cally, the acquisition branch receives an extra reward of the change in entropy of the disease prediction, and the acquisition process ends when the entropy of the prediction is below a dynamically learned, disease -speci ﬁ c threshold.
Our approach also uses the two-branches setting and designs a mechanism to inform the acquisition process when to stop based on information from the classi ﬁ er branch.
Unlike
Yuan and Yu (2021), we explicitly introduce a stop action whose Q-value is trained to replicate a reward derived from the classi ﬁ cation branch.
The evidence acquisition branch can be trained with learning paradigms different from RL.
Chen et al. (2022) proposes to model the evidence acquisition process as a sequential generation task and trains the agent in a similar manner to BERT’s Masked Language Modelling (MLM) objective (Devlin et al., 2019).
Similarly, Luo et al.
(2020) relies on randomly generated trajectories to train a system to collect evidences from patients in a supervised way.
Finally, Guan and Baral (2021) makes use of the
Quick Medical Reference belief network (Miller et al., 1986), and applies a Bayesian experimental design (Chaloner and Verdinelli, 1995) in the inquiry phase while relying on Bayes rule to infer the corresponding disease distribution.
However, most of these approaches are primarily focused on predicting the single ground truth pathology, as opposed to the differential diagnosis in our work.
Also, they have rarely made speci ﬁ c efforts to shape or guide the interaction to resemble how doctors would interact with patients.
As a result, it is debatable how much doctors could trust such systems, even though some may perform well on benchmark datasets, therefore casting doubt on their applicability in real-world applications.
Method
Let E and D be the number of all evidences (i.e., symptoms and antecedents) and diseases under consideration.
Evidences can be binary (e.g., are you coughing? ), categorical (e.g., what is the pain intensity on a scale of 1 to 10? ), or multi-choice (e.g., where is your pain located? ).
The task of automated evidence collection and diagnosis can be viewed as a sequential decision process where each interaction with a patient is formalized using a ﬁ nite-horizon Markov Decision Process
M = ( S , A , P, R, γ , T ) with a state space S = S ′ ∪ { s ⊥ } (with s ⊥ being the terminal state), an action space A , a dynamics P , a reward function R , a discount factor γ , and a maximum episode length T .
A state s ∈ S ′ encodes socio-demographic data regarding the patient (e.g., age, sex) as well as the evidences provided by the patient so far.
A is de ﬁ ned as { 1 , · · · , E, E + 1 } where the ﬁ rst E elements, referred to as acquisition actions , are used to inquire about corresponding evidences, and the last element is the exit action that is used to explicitly terminate the interaction.
At any point in time, only actions not yet selected in the episode are available for future selection.
P is deterministic and updates the current state based on the current action and the patient response if the current turn is less than T , otherwise, the next state is set to s ⊥ .
Finally, regarding R which is de ﬁ ned on the space S × A × S like P , each acquisition action is characterized by an inquiry cost r i , and depending on whether the underlying patient is experiencing the corresponding evidence, it may additionally incur a retrieval reward r p or a missing penalty r n .
Upon the termination of the interaction, a prediction regarding the patient’s differential is made based on the collected evidences.
An overview of the proposed approach is depicted in Figure 1.
Our model is made of two branches: an evidence acquisition branch in charge of the policy for interacting with patients and a classi ﬁ er branch in charge of predicting the differential at each turn.
Both the classi ﬁ er and the policy networks rely on a latent representation computed by an encoder from the evidences collected so far from the patient.
Importantly, we introduce a reward shaping module which leverages the output of the classi ﬁ er to de ﬁ ne auxiliary rewards that help induce in the agent, during training, the desired properties of doctor reasoning identi ﬁ ed in Section 1 .
More precisely, the exploration and con ﬁ rmation rewards are designed to encourage the two-phase based interactions, whereas the severity reward is used to explicitly handle severe pathologies.
Finally, the classi ﬁ cation reward is used to train the agent to recognize when to stop the interaction process.
The proposed agent predicts at each turn t the next action a t as well as the belief bel t regarding the patient’s differential from the current state s t .
We rely on a DQN variant algorithm, Rainbow (Hessel et al., 2018) (without noisy networks Fortunato et al.
(2018)), to search for the optimal policy and the objective of the proposed auxiliary rewards is to encourage trajectories similar to those of experienced medical practitioners.
In the next sections, we thoroughly describe each of these auxiliary rewards.
3.1 Exploration reward: encouraging the evaluation of multiple pathologies
We want our agent to undertake actions that favour considering several pathologies in the differential diagnosis during the ﬁ rst phase of the interaction.
In practice, this means that the agent’s belief is constantly changing after inquiring about a new evidence. As more evidences are collected, this I have fever
[Figure] strep [Figure] ﬂ uctuation should gradually reduce until it becomes insigni ﬁ cant.
Based on this intuition, we derive the exploration reward R Ex as
[Equation]
(1) where JSD is the Jensen-Shannon divergence that computes the dissimilarity between consecutive beliefs bel t and bel t +1 , and w Ex ( t ) ∈ [0 , 1] is a dynamic weight which aims at controlling the importance of this shaping component as a function of time.
Inspired from the shape of the sigmoid function, we design w Ex ( t ) as a translated version in [0 , T ] of the “ ﬂ ipped” sigmoid function within the interval [¯ x min , ¯ x max ] .
In other words, using trans ( t ) = t ∗ (¯ x [Equation] , we have
[Equation]
where ∆ Ex is a hyper-parameter that controls how fast the function saturates at 0 (cf.
Appendix A.1).
The beliefs bel t and bel t +1 are generated by the classi ﬁ er branch based respectively on states s t and s t +1 , with the latter state resulting from the execution of action a t when in state s t .
3.2 Con ﬁ rmation reward: Strengthening the agent’s belief in the pathology
In the second phase of the interaction with the patient, the agent should inquire about evidences that help strengthen its belief regarding the differential diagnosis.
This means that the more the agent collects evidences, the closer its belief should be to the ground truth distribution y .
The importance of this shaping component should gradually increase as the agent moves towards the end of the interaction.
From this intuition and inspired from potential-based reward shaping Ng et al.
(1999), we derive the con ﬁ rmation reward R Co as
[Equation]
where CE is the cross-entropy function, and w Co ( t ) ∈ [0 , 1] controls the importance of this shaping component as a function of time.
Like w Ex ( t ) , w Co ( t ) is a translated version of the sigmoid function with the difference that it increases over time.
Thus, we have:
[Equation]
where ∆ Co is a hyper-parameter controlling how fast the function saturates at 1 (cf.
Appendix A.1).
3.3 Severity reward: evidence gathering for ruling out severe pathologies
We want our agent to collect evidences that help rule out severe pathologies that are not part of the ground truth differential as soon as they become plausible.
One proxy to achieve this for the agent is to behave in such a way to monotonically increase the number SevOut of severe pathologies that are not in both the ground truth differential and its predictions through time.
As such, we de ﬁ ne the severity reward R Sev as
[Equation]
SevOut t and SevOut t +1 correspond to the number of severe pathologies which are not in both the ground truth differential and the differentials respectively predicted at time t and t + 1 , i.e., the beliefs bel t and bel t +1 .
Here, a pathology is not part of a predicted distribution if its probability is below a given threshold.
3.4 Classi ﬁ cation reward
This reward is designed to provide feedback to the agent describing how good its ﬁ nal predicted differential is with respect to the ground truth differential y when the interaction process is over.
Let SevIn t be the number of severe pathologies that are part of both the ground truth differential y and the predicted belief bel t , and let Sev y be the number of severe pathologies in y .
As noted previously, a pathology is not part of a predicted distribution if its probability is below a given threshold.
We de ﬁ ne V ( s t , y )
= − CE ( bel t , y )
+
w si SevIn t
Sev y as a measure of the quality of the belief prediction from state s t where CE is the cross-entropy function, and the second term, controlled by the hyper-parameter w si , measures the inclusion rate of the relevant severe pathologies in bel t .
We thus de ﬁ ne the classi ﬁ cation reward R Cl as
[Equation]
The above mentioned auxiliary rewards are combined together as
[Equation]
where the α weights are hyper-parameters, and the ( s t , a t , s t +1 ) arguments for F and all reward functions were dropped for clarity.
The resulting function F is then combined with the environment reward R , leading to a new reward function R ′ =
R + F that is used to train the policy network.
For simplicity, this section relies on DQN to present the loss for training each part of our agent, with the extension to Rainbow easy to do.
Let θ and ϕ be the agent and target network parameters.
Policy network loss function: The Q-value of the exit action does not depend on any following state.
Also, the reward received when explicitly exiting is the classi ﬁ cation reward.
Therefore, to synchronize the agent branches, the exit action Q-value needs to be reconciled with the classi ﬁ cation reward.
This should be done at each interaction time step during the training, given that the agent needs to have a good estimate of the expected Q-value of the exit action to select this action.
Thus,
[Equation]
where B is a batch from the replay buffer, r t = R ( s t , a t , s t +1 ) is the R reward at time t , Q θ ( s t , a exit ) is the predicted Q-value for the pair ( s t , a exit ) , Q t = r ′ t + 1 s t +1 ̸
= s ⊥ γ max a ∈ A Q ϕ ( s t +1 , a ) is the target Q-value for the pair ( s t , a t ) with r ′ t = R ′ ( s t , a t , s t +1 ) =
r t + F ( s t , a t , s t +1 ) .
Classi ﬁ er network loss function: The classi ﬁ er is updated using the following loss at the end of an interaction:
[Equation]
Training process:
Using the loss functions de ﬁ ned in Equations 8 and 9, the two branches are alternatively updated until the stopping criteria are satis ﬁ ed.
Both updates share the same batch of data B which is sampled from the replay buffer.
The training process is described in Appendix A.2.
Experiments
Datasets The datasets used in prior works such as DX (Wei et al., 2018), Muzhi (Xu et al., 2019),
SymCAT (Peng et al., 2018), HPO (Guan and Baral, 2021), and MedlinePlus (Yuan and Yu, 2021) do not provide differential diagnosis information, and are therefore ineligible for validating the approach proposed in this work.
Besides, as supported by Yuan and Yu (2021), the patients simulated using SymCAT are not suf ﬁ ciently realistic for testing automatic diagnosis systems.
We instead use the DDXPlus dataset (Fansi Tchango et al., 2022) for that purpose.
In addition to providing differential diagnosis and pathology severity information, the DDXPlus dataset, unlike the above mentioned datasets, is not restricted to binary evidences, but also has categorical and multi-choice evidences, allowing for more ef ﬁ cient interactions with patients.
Furthermore, the dataset containspathologies and 223 evidences (corresponding to 110 symptoms and 113 antecedents).
Finally, the dataset is split into 3 subsets: a training subset with more than 10 6 synthetic patients, and validation and test subsets containing roughly 1 . 4 × 10 5 synthetic patients each.
Baselines We consider 4 baselines, AARLC (short for Adaptive Alignment of Reinforcement
Learning and Classi ﬁ cation) (Yuan and Yu, 2021), Diaformer (Chen et al., 2022), BED (Guan and
Baral, 2021), and BASD which is an extension on Luo et al.
AARLC demonstrates SOTA results on the SymCAT dataset while Diaformer shows competitive results on the Muzhi and DX datasets.
BED, an approach that does not require training, has impressive results on the HPO dataset.
All these methods are designed to only handle binary evidences and had to be modi ﬁ ed to deal with the different evidence types in the DDXPlus dataset.
Moreover, those methods were designed to predict the ground truth pathology, and had to be modi ﬁ ed to handle differentials.
Additional details are provided in Appendix B.
Experimental settings Each patient in the DDXPlus dataset is characterized by a chief complaint, which is presented to the models at the beginning of the interaction, a set of additional evidences that the models need to discover through inquiry, a differential diagnosis, and a ground truth pathology.
We allow interactions to have a maximum of T = 30 turns.
We tune the hyper-parameters for each model, including the baselines, separately on the validation set and use the resulting optimal set of parameters to report the performances.
For more details, see Appendix C.
Evaluation metrics We report on the interaction length (IL), the recall of the inquired evidences (PER), the ground truth pathology accuracy when only considering the top entry of the differential (GTPA@1) and when considering all entries (GTPA), the F1 score of the predicted differential diagnosis (DDF1), and the harmonic mean of the rule-in and rule-out rates of severe pathologies (DSHM).
See Appendix D for the de ﬁ nition of these metrics.
The GTPA@1 metric is only relevant for models trained to predict the patient’s ground truth pathology.
It is not relevant for models trained to predict the differential as the ground truth pathology is not always the top pathology in the differential (Fansi Tchango et al., 2022).
We do not report the precision of the inquired evidences as it can be useful and sometimes necessary to ask questions that lead to negative answers from patients.
For example, such questions need to be asked in order to safely rule out severe pathologies.
Measuring the quality of the collected negative evidence is left as future work.
Results
Disease prediction and evidence acquisition Table 1 depicts the performance of each model at the end of the interaction process.
We ﬁ rst focus on metrics measuring the quality of the predicted diseases. All models are on par with regard to the recovery of the ground truth pathology (GTPA), even when they are trained to predict the differential and are not given any indication about the pathology in the ground truth differential that corresponds to the disease the patient is suffering from. As for recovering the right differential (DDF1), models perform poorly when trained to predict the ground truth pathology even if some of those models generate a posterior pathology distribution.
Performance greatly improves when models are trained to predict the differential, with CASANDE doing signi ﬁ cantly better than all other models, outperforming the second-best model by more than 10% absolute on average.
Likewise, CASANDE outperforms the baselines in handling the severe pathologies (DSHM) with an absolute margin of more than 4% when trained with the differential.
[Table] Those results indicate that training on the differential is essential and that CASANDE signi ﬁ cantly outperforms the baselines.
CASANDE achieves the best performance on the PER metric demonstrating its ability to collect relevant evidences. We observe that the PER score also improves for AARLC when trained on the differential, suggesting that differentials are bene ﬁ cial for collecting evidences. This improvement is not observed for BASD because its differential classi ﬁ er branch only operates at the end of the interaction process, after the evidence collection.
Similarly, Diaformer generates the differential at the end of the evidence collection and doesn’t bene ﬁ t from it to improve the collection process.
Finally, CASANDE has the longest interaction length.
This is unsurprising, as CASANDE is trained to emulate the reasoning of doctors.
In particular, the exploration and con ﬁ rmation phases may lead to the model asking more questions.
Additionally, CASANDE’s interaction length is still considerably lower than the maximal value (30), which indicates that CASANDE is capable of terminating an interaction rather than always continuing to the end.
Together, this shows that the slight increase in
CASANDE’s interaction length (less than 2 turns compared to the model with the shortest length) may be due to the different priorities of CASANDE and the baselines.
A qualitative evaluation of CASANDE was performed by a doctor who analyzed CASANDE’s predictions for 20 patients randomly selected from the test set.
The pro ﬁ les of those patients, the predictions of CASANDE as well as the doctor’s evaluation are presented in Appendix F. The doctor concluded that the evidences collected by CASANDE are indeed helpful for establishing a differential. Meanwhile, he cannot de ﬁ nitively assess CASANDE’s disease predictions since, unlike CASANDE, the scope of diseases he considers is not limited to the 49 pathologies in DDXPlus. Trajectory quality evaluation We plot in Figure 2 the con ﬁ rmation score versus the exploration score throughout the trajectories for the different models.
The exploration score captures how distant two consecutive agent predictions are, while the con ﬁ rmation score captures how close the agent prediction is with respect to the target distribution (see Appendix D.2 for the de ﬁ nitions).
Ideally, an agent would start with a high exploration score and a low con ﬁ rmation score (upper-left corner), gradually decreases the former and increases the latter, until the exploration score reaches a low value and the con ﬁ rmation score reaches a high value at the end of the interaction (lower-right corner).
As can be observed, this trend is highly followed by CASANDE which exhibits the highest exploration score at the beginning of the interaction and the highest con ﬁ rmation score at the end of the interaction while, unlike other models, it is consistently moving towards the lower-right corner of the chart.
Appendix G presents the differentials predicted by CASANDE at each interaction turn for 3 patients from the test set.
We clearly observe CASANDE exploring various differentials at the beginning of the interaction and then focusing on a differential towards the end of the interaction.
Handling of severe pathologies We are interested in analysing the pace at which severe pathologies are ruled out from or ruled in within the predicted differentials throughout the trajectories.
We show in Figure 3 the curves corresponding to the average values of the harmonic mean scores of the rule-in and rule-out rates (i.e., DSHM), the rule-out rate, and the rule-in rate of severe pathologies throughout the trajectories (see Appendix D.1 for the de ﬁ nitions).
As observed, CASANDE improves signi ﬁ cantly on the harmonic mean throughout a trajectory, and eventually performs the best among all models.
This shows that the capability of CASANDE for handling severe pathologies is heavily grounded on the evidence it gradually collects.
This evidence-based behavior is further con ﬁ rmed by the rule-out and rule-in rates as CASANDE is the only method that signi ﬁ cantly improves on both rates throughout the trajectories.
��� ��� ��� ��� ��� ��� ������������������ ��� ��� ��� ��� ��� ��� � � � � � � � � � � � � � � � � � ��������������������������� ����� ����� ������������ ���� ������������ ��� ������� ��������� ����������������� �������������� � �� �� �� �� ��� � throughout trajectories on average for each model on the test set.
For each trajectory, the scores are computed for each turn.
We then select a ﬁ xed number of 21 equally spaced turns, which represent 0%, 5%, · · · , 100% of the trajectory, and visualize the scores at these turns.
Finally, we average these 21 turns across trajectories for each method. Interestingly, when focusing on the ruleout curve, it is noticeable that CASANDE performs the second-worst at the beginning, trailing the best performing model by more than 20% , but it improves to the secondbest at the end, with the gap smaller than 3% .
This suggests that at the beginning,
CASANDE is more lenient on including severe diseases in its differentials, resulting in more false positives than others.
However, this is a desired behavior of CASANDE, as during the exploration phase, it is expected to consider unlikely but severe diseases before gathering enough evidences to rule them out.
In other words, at the beginning of a trajectory where few evidences are gathered, a model should err on the side of caution by keeping unlikely but severe diseases into consideration.
Ablation studies We perform an ablation analysis to illustrate the contribution of each auxiliary reward component of CASANDE to the overall performance.
Table 2 shows the results when some reward components are deactivated.
We observe a drop in DDF1 and DSHM when most rewards are disabled.
This indicates that each reward function helps the agent capture complementary information that is useful for predicting the differential.
Also, as expected, the exploration and con ﬁ rmation scores contribute in increasing the interaction length.
Finally, we observe a decrease of the DSHM metric when the severity reward is not used.
Additional studies are provided in Appendix E.
Limitations and potential negative social impact
Limitations In this work, we set out to include doctors’ trust as part of the desiderata in building evidence acquisition and automated diagnosis systems.
In doing so, we focus our efforts on three essential aspects of the reasoning of doctors, namely the generation of a differential diagnosis, the exploration-con ﬁ rmation approach, and the prioritized handling of severe pathologies.
We chose to focus on them as advised by our collaborating physician.
However, to most accurately evaluate doctors’ trust, the proposed method has to be deployed so as to allow doctors different from the one � �� �� �� �� ��� �������������� ���� ���� ���� ������������� � �� �� �� �� ��� �������������� ��� ��� ��� ������������� ������������������ ������� �������������� ��� ������������� � �� �� �� �� ��� �������������� ���� ���� ���� ������������ [Table] we collaborated with to judge our approach.
Additionally, it is reasonable to assume that doctors in other locations and/or specialties may have different ways of engaging with patients.
Therefore, while we provide evidence in the medical literature (Section 1) to show the identi ﬁ ed desiderata are widely applicable, we recognize that there may be cases where this work does not apply.
In this work, we conduct experiments on synthetic patients, mainly due to the lack of real-patient datasets that contain differential diagnoses.
We recognize that synthetic patients can be different from real patients in various and important ways, and that therefore results reported on synthetic patients may not extend to real patients.
Potential negative social impact As we have made clear from the beginning, evidence acquisition and automated diagnosis systems are not substitutes for human doctors, but rather they are supportive tools for doctors, who should make the ﬁ nal decisions.
However, the predictions of such systems potentially might be provided to patients as the ﬁ nal medical advice, without the intervention of human doctors.
In such cases, the instructions that patients receive may be misleading or erroneous, and thus can do more harm than good to patients’ health.
Conclusion
We re ﬂ ected on the task formulation of evidence acquisition and automatic diagnosis that are essential for telemedicine services, and introduced doctors’ trust as an additional desideratum.
Concretely, we argued that emulating the reasoning of doctors is critical for gaining their trust, and we identi ﬁ ed three essential doctor reasoning features that models can mimic.
We proposed a novel RL agent with these features built in.
We showed empirically existing models are insuf ﬁ cient for imitating the reasoning of doctors.
We then demonstrated the importance of the explicit modelling of the differential diagnosis, and the ef ﬁ cacy of our model in emulating doctors, while being competitive on conventional metrics.
This work is a ﬁ rst step towards reshaping the research in automatic diagnosis systems, and there is abundant potential for future work to explore.
First, it is important to continue working with doctors to determine whether additional medical elements need to be considered when building machine learning approaches.
Second, we need to build datasets that cover a wide spectrum of pathologies and train agents using this extensive diagnostic space along with all the corresponding evidences to uncover if the learned strategies are similar to expert doctors and how well machine learning approaches scale when the action space and the pathology space become much larger.
Third, it is important to ﬁ nd a way of measuring the quality of collected negative evidences.
Finally, it would be useful to consider online learning methods where doctors could identify missing evidences and give their overall feedback on the collected medical history.
Acknowledgments and Disclosure of Funding
We would like to thank Dialogue Health Technologies Inc. for providing us access to the physician who supported us throughout this work and to some of the computational resources used to run the experiments.
We would also like to thank Quebec’s Ministry of Economy and Innovation and Invest
AI for their ﬁ nancial support.