Boosting Transformers and Language Models for Clinical Prediction in Immunotherapy (2023)

Abstract: Clinical prediction is an essential task in the healthcare industry. However, the recent success of transformers, on which large language models are built, has not been extended to this domain. In this research, we explore the use of transformers and language models in prognostic prediction for immunotherapy using real-world patients’ clinical data and molecular profiles. This paper investigates the potential of transformers to improve clinical prediction compared to conventional machine learning approaches and addresses the challenge of few-shot learning in predicting rare disease areas. The study benchmarks the efficacy of baselines and language models on prognostic prediction across multiple cancer types and investigates the impact of different pretrained language models under few-shot regimes. The results demonstrate significant improvements in accuracy and highlight the potential of NLP in clinical research to improve early detection and intervention for different diseases.

## introduction predicting and measuring treatment response is among the most fundamental tasks in clinical medicine. particularly, in cancer immunotherapy @xcite , antibodies against programmed death-1/programmed death ligand 1 (pd-1/pd-l1) have led to us fda approval of several pd-1/pd-l1 treatment strategies for patients with metastatic cancer. however, not all patients derive clinical benefits @xcite , emphasizing the need to identify who will respond to immunotherapy @xcite . thus, accurate treatment response and disease progress forecast based on the patient's clinical features and molecular profile will effectively improve the treatment efficiency and spur the development of precise medication. in order to facilitate medical decisionmaking and health outcomes, clinical prediction models @xcite we evaluate the prediction performance (auc) of a patient's probability of immunotherapy response across multiple cancer types under settings with a small number of training samples on a public clinical dataset from @xcite . given the fact that most clinical data is stored in tabular form, current mainstream machine learning approaches @xcite to cancer prognosis @xcite are still tree-based ensemble models such as boosting @xcite and bagging @xcite . in contrast, transformers @xcite have revolutionized enormous fields including natural language processing (nlp) @xcite and computer vision @xcite . many attempts to apply transformers on tabular modeling (e.g., tabtransformer @xcite have also achieved success. considering that the disparity between clinical data and other natural tabular data is not large, it is appealing that we can also translate this success from other domains to clinical prediction. as such, we seek to answer the first question in this cancer type age albumin drug class os months bladder 74 4.1 pd1/pdl1 9.1 nsclc 72 4.2 combo 27.3 input: a patient has been diagnosed with bladder cancer. the age is 74. the albumin is 4.1. the drug class is pd1/pdl1, etc. template: the {attribute} is {value}. ## llms for few-shot clinical prediction figure 2 is an overview of applying llms for clinical prediction. as discussed in section 1, purely supervised learning via transformer encoders is often hampered when training samples are limited. llms provide a viable direction with astonishing in-context learning capability that exploits knowledge from other resources to downstream tasks with minimal tuning. serialization. to leverage llms on clinical tabular data, the feature columns must be serialized into sequences of natural language tokens that llms can comprehend and encode. recently, there have been a few trials @xcite investigating various serialization techniques and exploring the corresponding performance across different tasks, which turns out that llms for tabular modeling rely more on the correct values than the structure of the features @xcite . to avoid repetitive work, in this work, we focus more on how different pretrained llms using different knowledge sources may affect the prediction performance by simply following a manual serialization template, the {attribute} is {value}., which has been proven to generate competitive results compared to other llms prompting-based regeneration methods by @xcite . knowledge sources. the pretraining corpus is also known as the knowledge source for llms. clinical language is notably different from the standard nlp text in terms of vocabulary and syntax @xcite . as a result, following advancements in language modeling from the larger nlp community, the clinical nlp sub-community frequently trains domain-specific models on clinical corpora. following bert @xcite , various clinical and biomedical versions appeared quickly, including biobert @xcite , clinicalbert @xcite , scib-ert @xcite , pubmedbert @xcite , etc. however, domain-agnostic llms like gpt-3 have so far been unable to achieve competitive results on biomedical nlp tasks @xcite , revealing the fact that the relevance and the knowledge reservation of pretraining sources have a significant impact to the knowledge migration in downstream tasks (e.g., finetuning or prompting). thus, we aim to evaluate the downstream performance in few-shot settings with a few different llms pretrained on different resources and benchmark the gaps. omnivorous loss objective. compared to conventional machine learning approaches, deep learning allows efficient end-to-end learning of image/text encoders in the presence of multi-modality along with tabular data benefiting from the modularized design. more importantly, the customized loss objectives corresponding to different tasks can often be combined for joint training, also known as multi-task learning @xcite . the inductive transfer across related tasks can help improve a model by introducing an inductive bias, which causes a model to prefer some hypotheses over others, that generally leads to solutions that generalize better. in cancer prognostic prediction, we usually have multiple endpoints to predict. for example, overall survival (os), progression-free survival (pfs), and best overall response (bor), etc. as such, in this work, we consistently adopt a join learning paradigm that merges multiple endpoints into one unified loss objective l f for all studies using the following term: where i is the total number of tasks and α i represents the soft weight for any task i. more specifically, in our experiments, we adopt crossentropy loss for bor and coxph loss for os and pfs prediction following deepsurv @xcite . ## experiments and results data. this dataset is acquired by memorial sloan kettering cancer center (mskcc) from a comprehensively curated cohort (msk-impact) with 1,479 patients treated with immune checkpoint blockade (icb) across 16 different cancer types @xcite , where patients are either responder (r) or non-responders (nr) to the treatment (pd-1/pd-l1 inhibitors, ctla-4 blockade or a combination) based on response evaluation criteria in solid tumors (recist) v1.1 @xcite or best overall response on imaging. each patient was collected up to 16 biological features, including genomic, molecular, clinical, and demographic variables. the train set contains 1,184 patients, and the test set contains 295 patients. the evaluation target is to predict clinical response to immunotherapy (binary classification) and both overall survival and progression-free survival (regression) in the test data across different cancer. training settings. for fair comparison, we adopt a hidden dimensionality of 768 for both clintat and berts (base versions). specifically, clintat is a stack of 6 transformer encoder layers with 8 heads. to prevent overfitting, we set the attention dropout rate to 0.3 and feedforward dropout rate to 0.1. for berts, all layers are frozen while we add one independent encoder on top of it to finetune. in the main figures and tables, we utilize a single linear layer to demonstrate the feasibility of llms for few-shot regimes. in ablation studies, we also investigate other encoder types such as another small transformer encoder. the optimizer of adamw is adopted consistently for all trainings, and the basic learning rates for clintat and berts are 1.25e -4 and 1.25e -5 with a weight decay of 0.01, correspondingly. a linear warmup (up to 5 epochs with a total training of 200 epochs) with cosine annealing strategy (warmup learning rate is set to 2.5e -7 ) is also applied. for other machine learning baselines, we utilize the grid search to find the optimal hyper-parameters and report the best results. more details can be found in the appedix. ## limitations this study is based on a single clinical cohort consisted of 1479 patients, which may limit the generalizability of the results to other clinical cohorts. this specific cohort of patients may not be representative enough of the general population, which may inject certain level of bias brought by the dissimilar distributions of gender, age, race, etc. while we envision the generalization capability of the language models is applicable to other clinical prediction tasks, the focus of this work is majorly about prognostic prediction of cancer immunotherapy, and we hereby have not provided solid evidence to prove that the success can also be extended to other relevant trials. additionally, we have yet only compared a limited set of transformers and language models, and it is possible that other models may perform better on the tasks evaluated in this study. finally, it is important to note that while the models in this study achieve high accuracy in clinical prediction, the ultimate value of these models in improving patient outcomes will depend on how well they are integrated into clinical decision-making processes and the impact they have on patient care. ## ethical considerations as this work uses real-world patients' clinical data and molecular profiles, which may raise concerns about data privacy and confidentiality. we ensure that all the patients' data is de-identified and protected from unauthorized access and use. the public patient data foot_2 was approved by the memorial sloan kettering cancer center (mskcc) foot_3 institutional review board for scientific use. researchers have ensured that they obtain proper ethical approval and informed consent from patients before using their data. even though this is a dataset that has been carefully curated to prevent the negative impact brought by human bias, there maybe existing a risk of introducing bias into the clinical cohort of data we analyze, particularly in the selection of patients and the choice of clinical features and molecular profiles. additionally, the use of predictive models to guide clinical decision-making might raise concerns about fair access to healthcare. we hereby ensure that the use of predictive models does not result in the inequitable distribution of healthcare resources and that patients from all socioeconomic backgrounds have equal access to the best possible care. this study uses natural language processing and machine learning algorithms to predict disease prognosis, which may raise broader ethical considerations related to the responsible use of technology in healthcare. we ensure that the use of all approaches discussed in this work is guided by general ethical principles, such as transparency, accountability, and patient-centered care. even though we focus on relatively large scale language models in this work, our finetuning strategy only requires a considerably small amount of computation as only the encoder part needs to be finetuned. in practice, the single linear layer finetuning can be obtained in about 2 hours on a machine with single nvidia a10 gpu; training completes within 5 hours on a machine with one nvidia a10 gpu for another transformer encoder with a depth of 6 and dimensionality of 768. all the pretrained language model weights are publicly available (e.g., huggingface).