A deep-learning model to assist thyroid nodule diagnosis and management – Authors' reply
Authors: Yihao Liu, Jinyu Liang, Sui Peng, Wei Wang, Haipeng Xiao
Journal: The Lancet Digital Health (2021)

We thank Joon-Hyop Lee and Young Jun Chai for their insightful comments about our work1Peng S Liu Y Lv W et al.Deep learning-based artificial intelligence model to assist thyroid nodule diagnosis and management: a multicentre diagnostic study.Lancet Digit Health. 2021; 3: e250-e259Summary Full Text Full Text PDF PubMed Scopus (7) Google Scholar and important suggestions for improvements to our analyses. We have revised the appendix to report the number of malignant and benign nodules in test set A, B, and C. The number of correctly and incorrectly diagnosed benign and malignant nodules by the 12 radiologists without ThyNet, ThyNet-only, and radiologists with ThyNet are reported in the appendix. Sensitivity, specificity, negative predictive value, positive predictive value, and accuracy are included in the original Article. Images were collected from the same nodule in different scanning planes. In clinical practice, radiologists will store one or more ultrasound images as needed for diagnosis.2Li X Zhang S Zhang Q et al.Diagnosis of thyroid cancer using deep convolutional neural network models applied to sonographic images: a retrospective, multicohort, diagnostic study.Lancet Oncol. 2019; 20: 193-201Summary Full Text Full Text PDF PubMed Scopus (119) Google Scholar All images were initially included, but low-quality ultrasound images—such as those with severe artifacts or low image resolution—were excluded after quality control. Images from different scanning planes of thyroid nodules enhanced the robustness of our model. Different pathological subtypes might have an effect on the performance of artificial intelligence (AI) models. For example, ThyNet more accurately identified papillary thyroid cancer compared with follicular thyroid cancer, but there was no significant difference between them; this might be because papillary thyroid cancer has more typical ultrasound features. The performance of ThyNet in follicular thyroid cancer might be improved by increasing the number of follicular thyroid cancer images in the training set. Medullary thyroid cancer and anaplastic thyroid cancer had poor prognosis. Misdiagnosis of these cancers might be crucial to future clinical decisions; however, ThyNet correctly identified all instances of medullary thyroid cancer and anaplastic thyroid cancer. Despite the accuracy of ThyNet, the datasets were not representative of the real clinical setting. We also thank Bin Zhang and colleagues for their concerns and future directions for our study. First, papillary thyroid microcarcinoma is defined as a papillary tumour 1 cm or less in size. Despite their small size, papillary thyroid microcarcinoma are not low-risk or clinically non-significant tumours, and according to American Thyroid Association guidelines,4Haugen BR Sawka AM Alexander EK et al.2015 American thyroid association management guidelines for adult patients with thyroid nodules and differentiated thyroid cancer: the American Thyroid Association Guidelines Task Force on thyroid nodules and differentiated thyroid cancer.Thyroid. 2016; 26: 1-133Crossref PubMed Scopus (5904) Google Scholar active surveillance is not recommended for papillary thyroid microcarcinoma with clinically evident metastases, local invasion with convincing cytological, or molecular evidence of aggressive disease. In addition, nodules of 0·3–1 cm are difficult to biopsy and are not recommended for fine-needle aspiration; diagnosis of these small tumours is dependent on ultrasound examination. Second, to ensure the accuracy of pathological labels, all patients with thyroid nodules in the training set and test sets were pathologically examined in a hospital setting. This selection bias would make the malignancy rate higher. The hospital setting of our investigation was discussed as a limitation of our study. Third, ResNet, ResNext, and DenseNet were selected as the backbone for the deep-learning neural network. The area under the receiver operating characteristic curve (AUC ROC) of ResNet was 0·9376, ResNext was 0·9348, and DenseNet was 0·9401 in the internal validation set (appendix of the Article). To improve the performance of ThyNet, we used a technique called ensemble modelling which is widely applied to improve the performance of the classifier in deep learning. The AUC ROC of the ensemble model was 0·9504 (appendix of the Article), which outperformed ResNet, ResNext, and DenseNet. Fourth, American College of Radiology Thyroid Imaging, Reporting and Data System (ACR TI-RADS) is a guideline for fine-needle aspiration, which does not allow for direct differentiation of thyroid nodules. In our study, radiologists diagnosed nodules as either benign or malignant after reviewing images on the basis of their experience; the radiologists then made risk stratification and fine-needle aspiration recommendations based on ACR-TIRADS. Nonetheless, the diagnostic performance of radiologists might be underestimated if they only reviewed static images alone. Therefore, we designed the third test to validate our model in a clinical setting, by using images and videos for thyroid review (using test set C). We agree that developing a deep-learning model based on dynamic videos is more clinically significant, and this is the aim of ongoing work. Fifth, there are many types of ultrasound equipment in clinical practice. Using images from the same ultrasound machine for training and testing might limit the generalisability of the AI model. Sixth, in the review process, both junior and senior radiologists reviewed standardised ultrasound images instead of collecting images themselves, which might narrow their diagnostic performance. In addition, the standardised training of junior radiologists at the included institutions is systematic and led by experienced staff; junior radiologists can have good diagnostic performance after 1–3 years of training. Finally, the combination of the AI model and clinical indices is key to create a more reliable diagnostic model, and we agree that testing in different geographic settings or populations is crucial for the improvement of AI. We have reached cooperation agreements with several hospitals in the north and east of China for a new AI model, which is based on dynamic videos and a combined model. We declare no competing interests. Download .pdf (.14 MB) Help with pdf files Supplementary appendix Deep learning-based artificial intelligence model to assist thyroid nodule diagnosis and management: a multicentre diagnostic studyThe ThyNet-assisted strategy can significantly improve the diagnostic performance of radiologists and help reduce unnecessary fine needle aspirations for thyroid nodules. Full-Text PDF Open AccessA deep-learning model to assist thyroid nodule diagnosis and managementIn The Lancet Digital Health, Sui Peng and colleagues,1 showed the ability of a deep-learning model (ThyNet) to improve radiologists' diagnostic performance when determining the nature of a thyroid nodule. Unlike previous studies that compared the accuracy of humans and artificial intelligence (AI) performance,2 this study is important because it shows that the future of diagnostics is cooperation and synergy between humans and AI, rather than exclusivity and usurpation. Radiological diagnosis of thyroid nodules can be difficult in many situations, even with the American College of Rheumatology Thyroid Imaging Reporting and Data System,3 and if ThyNet can augment radiologists' diagnostic capabilities in general settings, it would be a breakthrough in managing thyroid nodules. Full-Text PDF Open AccessA deep-learning model to assist thyroid nodule diagnosis and managementThe study by Sui Peng and colleagues1 published in The Lancet Digital Health adds to the growing body of evidence that the application of deep-learning (or ensemble transfer learning) models can significantly improve the diagnostic performance of both junior and senior radiologists and reduce the number of unnecessary biopsies. Full-Text PDF Open Access