Original: Improving Neural Models for Radiology Report Retrieval with Lexicon-based Automated Annotation
Revised: Improving Neural Models for Radiology Report Retrieval with Lexicon-based Automated Annotation

Original: Anonymous ACL submission

Original: Abstract
Revised: Abstract

Original: Many clinical informatics tasks that are basedon electronic health records  need relevant patient cohorts to be selected based on ﬁndings,symptoms, and diseases.
Revised: Many clinical informatics tasks that are based on electronic health records (EHR) need relevant patient cohorts to be selected based on ﬁndings, symptoms and diseases.

Original: Frequently, theseconditions are described in radiology reportswhich can be retrieved using information retrieval (IR) methods.
Revised: Frequently, these conditions are described in radiology reports which can be retrieved using information retrieval (IR) methods.

Original: The latest of these techniques utilize neural IR models such as BERTtrained on clinical text.
Revised: The latest of these techniques utilize neural IR models such as BERT trained on clinical text.

Original: However, these methods still lack semantic understanding of the underlying clinical conditions as well as ruledout ﬁndings, resulting in poor precision during retrieval.
Revised: However, these methods still lack semantic understanding of the underlying clinical conditions as well as ruled out ﬁndings, resulting in poor precision during retrieval.

Original: In this paper we combine clinicalﬁnding detection with supervised query matchlearning.
Revised: In this paper we combine clinical ﬁnding detection with supervised query match learning.

Original: Speciﬁcally, we use lexicon-drivenconcept detection to detect relevant ﬁndings insentences.
Revised: Speciﬁcally, we use lexicon-driven concept detection to detect relevant ﬁndings in sentences.

Original: These ﬁndings are used as queriesto train a Sentence-BERT (SBERT) model using triplet loss on matched and unmatchedquery-sentence pairs.
Revised: These ﬁndings are used as queries to train a Sentence-BERT (SBERT) model using triplet loss on matched and unmatched query-sentence pairs.

Original: We show that the proposed supervised training task remarkably improves the retrieval performance of SBERT.
Revised: We show that the proposed supervised training task remarkably improves the retrieval performance of SBERT.

Original: The trained model generalizes well to unseenqueries and reports from different collections.
Revised: The trained model generalizes well to unseen queries and reports from different collections.

Original:  Introduction
Revised:  Introduction

Original: Electronic health record (EHR) retrieval is important for clinicians, staff and researchers.
Revised: Electronic health record (EHR) retrieval is important for clinicians, staff and researchers.

Original: The toolsfor performing clinically relevant searches couldaid in many use cases such as clinical decision support (Syeda-Mahmood, 2010), auditing, revenuecycle management, and cohort selection for clinical studies.
Revised: The tools for performing clinically relevant searches could aid in many use cases such as clinical decision support (Syeda-Mahmood, 2010), auditing, revenue cycle management, and cohort selection for clinical studies.

Original: Frequently, these searches involveretrieval of patients based on clinical ﬁndings thatare often captured in unstructured textual reportssuch as radiology reports, encounter notes, etc.
Revised: Frequently, these searches involve retrieval of patients based on clinical ﬁndings that are often captured in unstructured textual reports such as radiology reports, encounter notes, etc.

Original: Unlike structured query-based lookup of EHR, retrieval of unstructured (free-text) EHRs is muchmore challenging, requiring a semantic understanding of the underlying clinical conditions present orabsent.
Revised: Unlike structured query-based lookup of EHR, retrieval of unstructured (free-text) EHRs is much more challenging, requiring a semantic understanding of the underlying clinical conditions present or absent.

Original: Conventional exact or approximate termbased retrieval methods such as BM25 (Robertsonand Zaragoza, 2009) often perform poorly in response to ad-hoc queries (Chamberlin et al., 2020), as these methods lack the ability of semantic understanding of the clinical as well as language context.
Revised: Conventional exact or approximate termbased retrieval methods such as BM25 (Robertson and Zaragoza, 2009) often perform poorly in response to ad-hoc queries (Chamberlin et al., 2020), as these methods lack the ability of semantic understanding of the clinical as well as language context.

Original: With the emergence of deep learning encoding models, new retrieval methods have emerged with studies showing BERT-based neural methods outperforming BM25 on multiple retrieval benchmarks (Yilmaz et al., 2019a;
Revised: With the emergence of deep learning encoding models, new retrieval methods have emerged with studies showing BERT-based neural methods outperforming BM25 on multiple retrieval benchmarks (Yilmaz et al., 2019a;

Original: Chang et al., 2020; Nogueira and Cho, 2019; Yilmaz et al., 2019b; Qiao et al., 2019).
Revised: Chang et al., 2020; Nogueira and Cho, 2019; Yilmaz et al., 2019b; Qiao et al., 2019).

Original: The BERT-based retrieval methods can be classiﬁed into two categories: the cross-attention (or interaction-based) models (Yilmaz et al., 2019a;
Revised: The BERT-based retrieval methods can be classiﬁed into two categories: the cross-attention (or interaction-based) models (Yilmaz et al., 2019a;

Original: Nogueira and Cho, 2019; Yilmaz et al., 2019b) and the embedding-based (or representation-based) models (Chang et al., 2020; Reimers and Gurevych, 2019).
Revised: Nogueira and Cho, 2019; Yilmaz et al., 2019b) and the embedding-based (or representation-based) models (Chang et al., 2020; Reimers and Gurevych, 2019).

Original: While the BERT-style cross-attention models are very successful, they cannot be directly applied to large-scale retrieval problems because computing the similarity score for every possible query-document pair during inference can be prohibitively expensive.
Revised: While the BERT-style cross-attention models are very successful, they cannot be directly applied to large-scale retrieval problems because computing the similarity score for every possible query-document pair during inference can be prohibitively expensive.

Original: Therefore, they were often used as a re-ranker after a initial candidate retrieval round using BM25.
Revised: Therefore, they were often used as a re-ranker after a initial candidate retrieval round using BM25.

Original: The embedding-based methods can pre-encode the documents, and only the queries need to be encoded upon retrieval.
Revised: The embedding-based methods can pre-encode the documents, and only the queries need to be encoded upon retrieval.

Original: Retrieval can be achieved via approximate nearestneighbor search in the embedding space very efﬁciently (Johnson et al., 2021).
Revised: Retrieval can be achieved via approximate nearestneighbor search in the embedding space very efﬁciently (Johnson et al., 2021).

Original: In this study, we focus on the embedding-based retrieval BERT models.
Revised: In this study, we focus on the embedding-based retrieval BERT models.

Original: Speciﬁcally, we adopted the sentence-level retrieval setting, as studies suggested that the "best" sentence in a document provides a good proxy for document relevance (Yilmaz et al., 2019a).
Revised: Speciﬁcally, we adopted the sentence-level retrieval setting, as studies suggested that the "best" sentence in a document provides a good proxy for document relevance (Yilmaz et al., 2019a).

Original: Different pre-training tasks were used to train the BERT-based models for retrieval.
Revised: Different pre-training tasks were used to train the BERT-based models for retrieval.

Original: The pretraining tasks range from masked language modelling (MLM) over unlabeled free-text to supervised training on labeled datasets such as STS (Cer et al., 2017),
Revised: The pretraining tasks range from masked language modelling (MLM) over unlabeled free-text to supervised training on labeled datasets such as STS (Cer et al., 2017),

Original: MS MARCO (Nguyen et al., 2016) or TREC Microblog track (Lin et al., 2014).
Revised: MS MARCO (Nguyen et al., 2016) or TREC Microblog track (Lin et al., 2014).

Original: How- ever, MLM is not tailored for the purpose of information retrieval (IR), and labeled datasets areusually small and not easily accessible.
Revised: How- ever, MLM is not tailored for the purpose of information retrieval (IR), and labeled datasets are usually small and not easily accessible.

Original: Recently,pre-trained models on biomedical corpora suchas BioClinicalBERT (Alsentzer et al., 2019) and
Revised: Recently, pre-trained models on biomedical corpora such as BioClinicalBERT (Alsentzer et al., 2019) and

Original: BioBERT (Lee et al., 2020) can obtain embeddingswith medical-domain-speciﬁc knowledge, but theywere still trained with MLM.
Revised: BioBERT (Lee et al., 2020) can obtain embeddings with medical-domain-speciﬁc knowledge, but they were still trained with MLM.

Original: Early studies (Natarajan et al., 2010) showedthat most clinical queries are actually short queries(e.g. a disease or a syndrome).
Revised: Early studies (Natarajan et al., 2010) showed that most clinical queries are actually short queries (e.g. a disease or a syndrome).

Original: We found that theexisting BERT models pre-trained with MLM performed poorly on short queries as well as negativequeries (i.e. queries asking for lack of a ﬁnding).
Revised: We found that the existing BERT models pre-trained with MLM performed poorly on short queries as well as negative queries (i.e. queries asking for lack of a ﬁnding).

Original: Ideally, if  retrieval systems could be trained bymatched and unmatched query-sentence pairs, inboth positive and negated instances, we can expecta higher precision and recall in retrieval.
Revised: Ideally, if a retrieval system could be trained by matched and unmatched query-sentence pairs, in both positive and negated instances, we can expect a higher precision and recall in retrieval.

Original: However,manually labeling a large dataset is impractical, particularly for the medical domain where the numberof clinical ﬁndings is very large.
Revised: However, manually labeling a large dataset is impractical, particularly for the medical domain where the number of clinical ﬁndings is very large.

Original: Training neural IRmodels using weak supervision has been previouslyinvestigated (Dehghani et al., 2017; MacAvaneyet al., 2019), which use unsupervised methods (e.g.
Revised: Training neural IR models using weak supervision has been previously investigated (Dehghani et al., 2017; MacAvaney et al., 2019).

Original:  BM25) or article headings to provide pseudo labels.
Revised: These methods use unsupervised methods (e.g. BM25) or article headings to provide pseudo labels.

Original: However, these pseudo labelsare usually impreciseand the article headings are not always available.
Revised: However, these pseudo labels usually are imprecise and do not consider negative queries. Moreover, the article headings are not always available.

Original: Motivated by these challenges, we present ahybrid approach where we combine automatedclinical ﬁnding detection with supervised querysentence pair learning.
Revised: Motivated by these challenges, we present a hybrid approach where we combine automated clinical ﬁnding detection with supervised querysentence pair learning.

Original: Speciﬁcally, we use anautomatic lexicon-driven concept detection methodto detect relevant  chest X-ray (CXR) ﬁndings insentences.
Revised: Speciﬁcally, we use an automatic lexicon-driven concept detection method to detect relevant positive or negative chest X-ray (CXR) ﬁndings in sentences.

Original: These ﬁndings paired with the sentencescontaining them serve as weakly labeled trainingdata for Sentence-BERT (SBERT)(Reimers and
Revised: These ﬁndings paired with the sentences containing them serve as weakly labeled training data for Sentence-BERT (SBERT)

Original:  Gurevych, 2019).
Revised: (Reimers and Gurevych, 2019).

Original: The resulting approach avoidsmanual annotation and can be scaled for trainingon a large number of query-sentence pairs.
Revised: The resulting approach avoids manual annotation and can be scaled for training on a large number of query-sentence pairs.

Original: Weshow that the proposed training task remarkablyimproves the retrieval performance of SBERT ondatasets with automatic annotations and human annotations.
Revised: We show that the proposed training task remarkably improves the retrieval performance of SBERT on datasets with automatic annotations and human annotations.

Original:  Methods 
Revised:  Methods 2. Fine-grained concept extraction

Original: 2. Fine-grained concept extraction

Original: The algorithm for extracting ﬁndings from sentences in reports uses a vocabulary-driven approach.
Revised: The algorithm for extracting ﬁndings from sentences in reports uses a vocabulary-driven approach.

Original: Speciﬁcally, a domain-speciﬁc CXR ﬁnding lexicon was used.
Revised: Speciﬁcally, a domain-speciﬁc CXR ﬁnding lexicon was used.

Original: This lexicon captures the name ofﬁnding along with its potential variants and synonyms mined from over 200,000 chest radiology reports.
Revised: This lexicon captures the name of ﬁnding along with its potential variants and synonyms mined from over 200,000 chest radiology reports.

Original: To spot the occurrence of a ﬁnding lexicon phrase within reports, a string matching algorithm called the longest common subﬁx (LCF) algorithm was used.
Revised: To spot the occurrence of a ﬁnding lexicon phrase within reports, a string matching algorithm called the longest common subﬁx (LCF) algorithm was used.

Original: To determine if a core ﬁnding is positive or negative (e.g. ”no pneumothorax”), a two-step approach that combines language structuring and vocabulary-based negation detection is used.
Revised: To determine if a core ﬁnding is positive or negative (e.g. "no pneumothorax"), a two-step approach that combines language structuring and vocabulary-based negation detection is used.

Original: The method is reported to be highly accurate ( < 3% errors) compared with human labels.
Revised: The method is reported to be highly accurate ( < 3% errors) compared with human labels.

Original: More details are described in (Syeda-Mahmood et al., 2020).
Revised: More details are described in (Syeda-Mahmood et al., 2020).

Original: 2. Labeled data generation
Revised: 2. Labeled data generation

Original: In this paper, we focus on "anatomical ﬁndings" as well as "disease concepts" as those are the most commonly searched in EHR (Natarajan et al., 2010).
Revised: In this paper, we focus on "anatomical ﬁndings" as well as "disease concepts" as those are the most commonly searched in EHR (Natarajan et al., 2010).

Original: We use these ﬁnding modiﬁers as surrogates for queries.
Revised: We use these ﬁnding modiﬁers as surrogates for queries.

Original: For each sentence S j in our data collection, we have a set with K j labeled data entries
Revised: For each sentence S j in our data collection, we have a set with K j labeled data entries

Original: I j = { ( S j , N j,i , M j,i ) } 1 ≤ i ≤ K j .
Revised: I j = { ( S j , N j,i , M j,i ) } 1 ≤ i ≤ K j .

Original: For each labeled entry ( S j , N j,i , M j,i ) , M j,i is the i -th ﬁnding for
Revised: For each labeled entry ( S j , N j,i , M j,i ) , M j,i is the i -th ﬁnding for

Original: S j , and N j,i =
Revised: S j , and N j,i =

Original: yes | no indicates a positive or ruled out ﬁnding.
Revised: yes | no indicates a positive or ruled out ﬁnding.

Original: By using the ﬁndings as query surrogates, we can designate a query Q j,i =
Revised: By using the ﬁndings as query surrogates, we can designate a query Q j,i =

Original: ( N j,i , M j,i ) paired with S j : if N j,i equals to yes , Q j,i is a positive query, otherwise Q j,i is a negative query.
Revised: ( N j,i , M j,i ) paired with S j : if N j,i equals to yes , Q j,i is a positive query, otherwise Q j,i is a negative query.

Original: For example, ( yes, vascular congestion ) and ( no, pulmonary edema ) are two queries for the sentence "lungs: central vascular congestion without overt edema.
Revised: For example, ( yes, vascular congestion ) and ( no, pulmonary edema ) are two queries for the sentence "lungs: central vascular congestion without overt edema.

Original: " The actual queries may be more properly phrased  such as "presence/absence of X".
Revised: " The actual queries can be more properly phrased for data augmentation in training, such as "presence of M " or " M is observed" for a positive query and "no evidence of M " or "absence of M " for a negative query, where M is a ﬁnding. In this study, however, we only consider the simple form of " M " and "no M " as positive and negative queries, in both training and evaluation.

Original: Since we labeled all the sentences in our training dataset extensively with all the ﬁnding types we summarized, we can create a dictionary using each unique query Q =
Revised: Since we labeled all the sentences in our training dataset extensively with all the ﬁnding types we summarized, we can create a dictionary using each unique query Q =

Original: ( N, M ) as the key and the list of all the sentences that contain that query as the dictionary value.
Revised: ( N, M ) as the key and the list of all the sentences that contain that query as the dictionary value.

Original: Any sentence in the list is considered as a matched sentence for that query, whereas other sentences are considered as unmatched sentences.
Revised: Any sentence in the list is considered as a matched sentence for that query, whereas other sentences are considered as unmatched sentences.

Original: 2. Model
Revised: 2. Model

Original: We used SBERT as our retrieval model.
Revised: We used SBERT as our retrieval model.

Original: MEANpooling was used to derive a ﬁxed size sentence embedding (for either queries or  EHR sentences).
Revised: MEANpooling was used to derive a ﬁxed size sentence embedding (for either a query or an EHR sentence).

Original: We used the triplet objective function (Reimers and Gurevych, 2019) to train our model.
Revised: We used the triplet objective function (Reimers and Gurevych, 2019) to train our model.

Revised: A diagram of the training objective function is shown in Figure 1.

Original: Given a query q , a matched sentence m and an unmatched sentence u , the triplet loss tunes the network such that the distance between  q and m is smaller than the distance between  q and u by a margin (cid:15) :[Equation]
Revised: Given a query q , a matched sentence m and an unmatched sentence u , the triplet loss tunes the network such that the distance between the embeddings of q and m is smaller than the distance between the embeddings of q and u by a margin (cid:15) :

Original: [Equation]
Revised: [Figure]

Original: where e q , e m and e u are the sentence embeddings for q , m and u , respectively.
Revised: where e q , e m and e u are the sentence embeddings for q , m and u , respectively.

Original: (cid:107)·(cid:107) is a distancemetric.
Revised: (cid:107)·(cid:107) is a distance metric.

Original: We used the cosine distance and (cid:15) = 0 .
Revised: We used the cosine distance and (cid:15) = 0 .

Original: To improve training, we further used hardsampling (HS) to mine the hardest unmatched sentence for the triplet loss within a training batch.
Revised: To improve training, we further used hardsampling (HS) to mine the hardest unmatched sentence for the triplet loss within a training batch.

Original: Tobe speciﬁc, we performed inference within a batchbeforehand to ﬁnd the unmatched sentence with thehighest cosine similarity score (the most confusingunmatched sentence) for each query.
Revised: To be speciﬁc, we performed inference within a batch beforehand to ﬁnd the unmatched sentence with the highest cosine similarity score (the most confusing unmatched sentence) for each query.

Original: We furtherapplied mega-batching (MB)
Revised: We further applied mega-batching (MB)

Original: (Wieting and Gimpel,2018) to encourage the model to learn to distinguish "harder" unmatched sentences by increasingthe batch size.
Revised: (Wieting and Gimpel, 2018) to encourage the model to learn to distinguish "harder" unmatched sentences by increasing the batch size.

Revised: At inference, the cosine similarity between the query embedding and the report sentence embedding is used to determine the level of relevance.

Original:  Experiments and Results 
Revised:  Experiments and Results 3. Datasets

Original: 3. Datasets

Original: The experiments in Section 3.1-3.4 were carriedout on two public collections of radiology reportsprovided by Indiana University (Demner-Fushmanet al., 2016) and NIH (Wang et al., 2017).
Revised: The experiments in Section 3.1-3.4 were carried out on two public collections of radiology reports provided by Indiana University (Demner-Fushman et al., 2016) and NIH (Wang et al., 2017).

Original: Afterpruning for duplicates and applying our labeleddata generation algorithm described in Section 2.2,a total of 21,612 labeled entries were generated forthe Indiana dataset, which include 10,363 uniquesentences, 200 positive queries and 75 negativequeries.
Revised: After pruning for duplicates and applying our labeled data generation algorithm described in Section 2.2, a total of 21,612 labeled entries were generated for the Indiana dataset, which include 10,363 unique sentences, 200 positive queries and 75 negative queries.

Original: For the NIH dataset, 17,047 labeled entrieswere generated, including 9,091 unique sentences,250 positive queries and 30 negative queries.
Revised: For the NIH dataset, 17,047 labeled entries were generated, including 9,091 unique sentences, 250 positive queries and 30 negative queries.

Original: 3. Ablation study and parameter tuning
Revised: 3. Sensitivity analysis and parameter tuning

Original: We ﬁrst run an ablation study on the Indiana dataset(IND) to investigate if hard-sampling (HS) andmega-batching (MB) can bring improvement overrandom-sampling (RS, randomly select unmatchedsentence within a batch) and normal-batching (NB,size 32).
Revised: We ﬁrst run a sensitivity analysis on the Indiana dataset (IND) to investigate how much improvement hard-sampling (HS) and mega-batching (MB) can bring over random-sampling (RS, randomly select unmatched sentence within a batch) and normal-batching (NB, size 32).

Original: We randomly split the IND dataset intotwo halves with non-overlapping ﬁndings with theconstraint that they should roughly have equal number of labeled entries.
Revised: We randomly split the IND dataset into two halves with nonoverlapping ﬁndings with the constraint that they should roughly have equal number of labeled entries.

Original: After the split, the two setshave 117/44 and 83/31 positive/negative queries,respectively.
Revised: After the split, the two sets have 117/44 and 83/31 positive/negative queries, respectively.

Original: We performed  2-fold cross-validationand reported the average of the two test resultsregarding mean Average Precision (mAP).
Revised: We performed a 2-fold cross-validation and reported the average of the two test results regarding mean Average Precision (mAP).

Original: Thisallows us to evaluate the model performance on unseen queries.
Revised: This allows us to evaluate the model performance on unseen queries.

Original: The evaluation was performed overpositive queries (Pos. Q.), negative queries (Neg.
Revised: The evaluation was performed over positive queries (Pos. Q.), negative queries (Neg.

Original: Q.) and all queries (All Q.) separately.
Revised: Q.) and all queries (All Q.) separately.

Original: The results in Table 1 shows that the combinationof HS and MB achieved the best results.
Revised: The results in Table 1 shows that the combination of HS and MB achieved the best results.

Original: Increasing the mega-batching size to 128 resulted the bestperformance, but further increasing the batch size slightly degraded the performance.
Revised: Increasing the mega-batching size to 128 resulted the best performance, but further increasing the batch size slightly degraded the performance.

Original: The remarkable improvent of SBERT over the baseline BioClinicalBERT also suggests that the proposed model can 
Revised: The remarkable improvent of SBERT over the baseline BioClinicalBERT also suggests that the proposed model can generalize well to unseen queries.

Original: [Table]
Revised: [Table]

Original: generalize well to unseen queries.
Revised: The analysis was performed for positive queries (Pos. Q.), negative queries (Neg. Q.) and all queries (All Q.) separately.

Original: [Table]
Revised: [Table]

Original: 3. Cross-dataset study
Revised: 3. Cross-dataset study

Original: We also trained on the IND dataset and tested on the unique sentences in the NIH dataset and vice versa to investigate whether a trained model can generalize well to a different dataset.
Revised: We also trained on the IND dataset and tested on the unique sentences in the NIH dataset and vice versa to investigate whether a trained model can generalize well to a different dataset.

Original: The best
Revised: The best

Original: SBERT model from Table 1 was used here.
Revised: SBERT model from Table 1 was used here.

Original: We further included Okapi BM25 ( k 1 =1.5, b =0.75), the pre-trained BERT (Huggingface "BERT-baseuncased"), the ﬁne-tuned BERT (trained on the
Revised: We further included Okapi BM25 ( k 1 =1.5, b =0.75), the pre-trained BERT (Huggingface "BERT-baseuncased"), the ﬁne-tuned BERT (trained on the

Original: EHR sentences using MLM, without using our generated annotations), the BioClinicalBERT and
Revised: EHR sentences using MLM, without using our generated annotations), the BioClinicalBERT and

Original: SBERT pre-trained on MS MARCO dataset for comparison.
Revised: SBERT pre-trained on MS MARCO dataset for comparison.

Original: More details about these models are given in the appendix.
Revised: More details about these models are given in the appendix.

Original: In addition to mAP, mean
Revised: In addition to mAP, mean

Original: Recall (over all the queries) was also reported, where Recall was deﬁned as the ratio of the number of correctly retrieved sentences to the size of the query’s ground truth list.
Revised: Recall (over all the queries) was also reported, where Recall was deﬁned as the ratio of the number of correctly retrieved sentences to the size of the query’s ground truth list.

Original: Table 2 shows that our ﬁne-tuned SBERT performs very well on the dataset from another collection regarding both mAP and mR, and outperformed the other BERT/SBERT models by large margins.
Revised: Table 2 shows that our ﬁne-tuned SBERT performs very well on the dataset from another collection regarding both mAP and mR, and outperformed the other BERT/SBERT models by large margins.

Original: The baseline BERT without pretraining over medical texts obtained the worst results.
Revised: The baseline BERT without pretraining over medical texts obtained the worst results.

Original: The results for BERT (ﬁne-tuned) and BioClinicalBERT suggest that MLM training over the texts from the same domain can lead to some improvements but is still not ideal for direct use of retrieval.
Revised: The results for BERT (ﬁne-tuned) and BioClinicalBERT suggest that MLM training over the texts from the same domain can lead to some improvements but is still not ideal for direct use of retrieval.

Original: SBERT pre-trained on MS
Revised: SBERT pre-trained on MS

Original: MARCO dataset showed signiﬁcant improvements over BERT trained with MLM, but lacks domainspeciﬁc knowledge and shows performance drop on negative queries.
Revised: MARCO dataset showed signiﬁcant improvements over BERT trained with MLM, but lacks domainspeciﬁc knowledge and shows performance drop on negative queries.

Original: BM25 performs well on positive queries with performance degradation on negative queries as well, because negation is not always[Table] explicitly expressed in EHR.
Revised: BM25 performs well on positive queries with performance degradation on negative queries as well, because negation is not always explicitly expressed in EHR.

Original: 3. Embedding separation analysis
Revised: 3. Embedding separation analysis

Original: [Table]

Original: Because we have the negation labels, we canalso create opposite-negation queries.
Revised: Because we have the negation labels, we can also create opposite-negation queries.

Original: For example, the  opposite-negation query for "no opacity"would be "opacity".
Revised: For example, the [Table] opposite-negation query for "no opacity" would be "opacity".

Original: Ideally,  for a given sentence,the similarity score between the matched queryand sentence should be larger than that betweenthe opposite-negation query and the sentence.
Revised: Ideally, with a high-precision retrieval system, for a given sentence, the similarity score between the matched query and sentence should be higher than that between the opposite-negation query and the sentence.

Original: Wereported (Table 3) the differences (mean ± std) between these two scores for all the entries in eachdataset with all the BERT embedding-based methods.
Revised: We reported (Table 3) the differences (mean ± std) between these two scores for all the entries in each dataset with all the BERT embedding-based methods.

Original: Our trained SBERT showed a clear separationin the embedding space.
Revised: Our trained SBERT showed a clear separation in the embedding space.

Original: The distances for the other BERT models are all around zero with even negative distances, suggesting poor negation awareness.
Revised: The distances for the other BERT models are all around zero with even negative distances, suggesting poor negation awareness.

Original: 3. Evaluation on human-annotated data
Revised: 3. Evaluation on human-annotated data

Original: We also evaluated our model on a separate humanannotated dataset.
Revised: We also evaluated our model on a separate humanannotated dataset.

Revised: The radiology reports used in this section are private anonymized data obtained from our collaborative partners.

Revised: HIPPA was fully enforced and all data were handled according to the

Original:  206 CT reports and 120 CXR reports were annotated  by 3 radiologists on sentencelevel with majority voting (more details in appendix).
Revised: Declaration of Helsinki. All reports were written in the English language. 206 CT reports and 120 chest X-ray (CXR) reports were annotated for various disease ﬁndings on the sentence level by 3 radiologists using the brat rapid annotation tool (available at https://brat.nlplab.org/ ).

Revised: Majority voting was used to handle disagreements.

Original: This resulted in 2,990 unique sentences/8queries for CT reports and 1,810 unique sentences/18 queries for CXR reports.
Revised: This resulted in 2,990 unique sentences/8 queries for CT reports and 1,810 unique sentences/18 queries for CXR reports.

Revised: Note that the candidate sentences for retrieval also include those sentences without any our interested disease ﬁndings.

Original: Table 4 showsthat our SBERT ﬁne-tuned oneither IND or NIHdataset outperforms the other compared methodsby large margins.
Revised: For [Table] the CT reports, the annotation was based on the presence or absence of 4 diseases (resulting inqueries): thoracic aneurysm, abdominal aneurysm, lung nodule and pulmonary embolism. The average number of matched sentences for each query is 42 ± 33.

Revised: For the CXR reports, the annotation was based on 10 diseases (resulting in 18 queries, as 2 negative queries do not have the corresponding matched sentences): pulmonary embolism, airspace opacity, lung nodule, emphysema, pneumothorax, abdominal aortic aneurysm, thoracic aortic aneurysm, rib fracture, scapula fracture and spine fracture.

Revised: The average number of matched sentences for each query is 18 ± 16.

Revised: It it worth noting that some of the diseases are not even used as queries in the IND/NIH training data, including thoracic aneurysm, abdominal aneurysm and spine fracture.

Revised: Table 4 shows that our SBERT ﬁne-tuned on either IND or NIH dataset outperforms the other compared methods by large margins.

Original:  Discussion
Revised:  Discussion

Original: In this paper we demonstrated that the proposedsupervised pre-training tasks with automated annotation can greatly improve the IR performance[Table] of SBERT on short and negative queries.
Revised: In this paper we demonstrated that the proposed supervised pre-training tasks with automated annotation can greatly improve the IR performance of SBERT on short and negative queries.

Original: The proposed labeled data generation method can also be used to train the cross-attention BERT models for further improvement when computation speed is not the bottleneck.
Revised: The proposed labeled data generation method can also be used to train the cross-attention BERT models for further improvement when computation speed is not the bottleneck.

Original: We focused on short queries in this study, and
Revised: We focused on short queries in this study, and

Original: BM25 still performs well on positive queries.
Revised: BM25 still performs well on positive queries.

Original: The embedding-based BERT models are expected to show more advantages over BM25 on complicated queries that require semantic understanding.
Revised: The embedding-based BERT models are expected to show more advantages over BM25 on complicated queries that require semantic understanding.

Original: Having the comprehensive negation and ﬁnding labels for each sentence also allows us to assemble more complicated queries that include more than one ﬁnding, such as “A and B” or “A without C” where A, B and C represent three different ﬁndings.
Revised: Having the comprehensive negation and ﬁnding labels for each sentence also allows us to assemble more complicated queries that include more than one ﬁnding, such as " A and B " or " A without C " where A , B and C represent three different ﬁndings.

Original: These more challenging tasks can be explored in the future work.
Revised: These more challenging tasks can be explored in the future work.

Original: The label generation tool can also be extended to training IR models in domains other than medical domain, such as ﬁnance, law, or retail, provided with the corresponding lexicons.
Revised: The label generation tool can also be extended to training IR models in domains other than medical domain, such as ﬁnance, law, or retail, provided with the corresponding lexicons.

Revised: In this study we did not evaluate retrieval on the report-level because we have the sentence-level annotations, which enable ﬁne-grained evaluation.

Revised: The report-level evaluation can be included in the future work.

Original:  Conclusion
Revised:  Conclusion

Original: In this work we proposed to generate querysentence pairs automatically using a CXR lexicon for training embedding-based BERT models on the EHR retrieval problem.
Revised: In this work we proposed to generate querysentence pairs automatically using a CXR lexicon for training embedding-based BERT models on the EHR retrieval problem.

Original: We showed that the ﬁne-tuned SBERT obtained a substantial performance gain over the other pre-trained models.
Revised: We showed that the ﬁne-tuned SBERT obtained a substantial performance gain over the other pre-trained models.

Original: The trained model can also generalize well to unseen queries and data from another source.
Revised: The trained model can also generalize well to unseen queries and data from another source.

Original: The proposed method can be especially helpful in training and evaluating neural IR models in domains with limited human-labeled data.
Revised: The proposed method can be especially helpful in training and evaluating neural IR models in domains with limited human-labeled data.

Original: A Appendix: Model training details
Revised: A Appendix: Model training details

Original: Here we provide more details on the models usedin Section 3.
Revised: Here we provide more details on the models used in Section 3.

Original: We used the Huggingface "BERTbase-uncased" model (pre-trained on BookCorpusand English Wikipedia, availabel at: https://huggingface.co/bert-base-uncased )as our BERT model for comparison.
Revised: We used the Huggingface "BERTbase-uncased" model (pre-trained on BookCorpus and English Wikipedia, availabel at: https:// huggingface.co/bert-base-uncased ) as our BERT model for comparison.

Original: The BERT(ﬁne-tuned) model was ﬁne-tuned on the EHR text(Indiana or NIH dataset) using MLM for 5 epochsbased on the "BERT-base-uncased" model.
Revised: The BERT (ﬁne-tuned) model was ﬁne-tuned on the EHR text (Indiana or NIH dataset) using MLM for 5 epochs based on the "BERT-base-uncased" model.

Original: Thepre-trained BioClinicalBERT (Alsentzer et al. (availabel at: https://github.com/
Revised: The pre-trained BioClinicalBERT (Alsentzer et al., 2019) (availabel at: https://github.com/

Original: EmilyAlsentzer/clinicalBERT ) wasinitialized with BioBERT (Lee et al., 2020) andﬁne-tuned on clinical notes.
Revised: EmilyAlsentzer/clinicalBERT ) was initialized with BioBERT (Lee et al., 2020) and ﬁne-tuned on clinical notes.

Original: Our SBERT model was initialized with the BioClinicalBERT.
Revised: Our SBERT model was initialized with the BioClinicalBERT.

Original: We ﬁne-tuned SBERT using tripletloss for 10 epochs for all datasets in this study.
Revised: We ﬁne-tuned SBERT using the triplet loss for 10 epochs for all datasets in this study.

Original: Weused AdamW optimizer with  learning rate 2e-5,weight decay 0.01 and a linear learning rate warmup of 100 steps.
Revised: We used AdamW optimizer with the learning rate 2e-5, weight decay 0.01 and a linear learning rate warm-up of 100 steps.

Original: The SBERT model used as comparison waspre-trained on 500K (query, answer) pairs fromthe MS MARCO dataset.
Revised: The SBERT model used as comparison was pre-trained on 500K (query, answer) pairs from the MS MARCO dataset.

Original: This pre-trained model(msmarco-bert-base-dot-v5) was one of the recommended sentence embedding models from the ofﬁcial SBERT webpage ( https://www.sbert.net/docs/pretrained_models.html ).
Revised: This pre-trained model (msmarco-bert-base-dot-v5) was one of the recommended sentence embedding models from the ofﬁcial SBERT webpage ( https://www.sbert. net/docs/pretrained_models.html ).

Original: Among all the pre-trained models, we picked thisone because it is the only pre-trained model basedon "BERT-base" model, to be consistent with all the other models (all based on "BERT-base") in our experiments.
Revised: Among all the pre-trained models, we picked this one because it is the only pre-trained model based on "BERT-base" model, to be consistent with all the other models (all based on "BERT-base") in our experiments.

Original: Since this model was tuned to be used with dot-product, we used dot-product to calculate similarity scores only for this model in the retrieval experiments in Table 2.
Revised: Since this model was tuned to be used with dot-product, we used dot-product to calculate similarity scores only for this model in the retrieval experiments in Table 2.

Original: For all the other models, cosine-similarity was used to calculate scores.
Revised: For all the other models, cosine-similarity was used to calculate scores.

Original: However, for the embedding separation analysis in Table 3, cosine-similarity was used for SBERT (MS MARCO) as well so that the scale of the similarity scores is comparable to the others.B Appendix: Human-annotated data The radiology reports used in Section 3.5 are private data obtained from our collaborative partners. All private data used were anonymized.
Revised: However, for the embedding separation analysis in Table 3, cosine-similarity was used for SBERT (MS MARCO) as well so that the scale of the similarity scores is comparable to the others.

Original: HIPPA was fully enforced and all data were handled according to the Declaration of Helsinki.

Original: 206 CT reports and 120 chest X-ray (CXR) reports were annotated by 3 radiologists on sentence-level using the brat rapid annotation tool (available at https: //brat.nlplab.org/ ).

Original: Majority voting was used to handle disagreements.

Original: This resulted in 2,990 unique sentences/8 queries for CT reports and 1,810 unique sentences/18 queries for CXR reports.

Original: Note that the candidate sentences for retrieval also include those sentences without any our interested disease ﬁndings.

Original: For the CT reports, the annotation was based on the presence or absence of 4 diseases (resulting in 8 queries): thoracic aneurysm, abdominal aneurysm, lung nodule and pulmonary embolism.

Original: The average number of matched sentences for each query is 42 ± 33.

Original: For the CXR reports, the annotation was based on 10 diseases (resulting in 18 queries, as 2 negative queries do not have the corresponding matched sentences): pulmonary embolism, airspace opacity, lung nodule, emphysema, pneumothorax, abdominal aortic aneurysm, thoracic aortic aneurysm, rib fracture, scapula fracture and spine fracture.

Original: The average number of matched sentences for each query is 18 ± 16.

Original: It it worth noting that some of the diseases are not even used as queries in the IND/NIH training data, including thoracic aneurysm, abdominal aneurysm and spine fracture.