Spinal metastasis, a metastatic cancer of the spine, is the most common malignant disease in the spine. In this study, we investigate the feasibility of automated spinal metastasis detection in magnetic resonance imaging (MRI) by using deep learning methods. To accommodate the large variability in metastatic lesion sizes, we develop a Siamese deep neural network approach comprising three identical subnetworks for multi-resolution analysis and detection of spinal metastasis. At each location of interest, three image patches at three different resolutions are extracted and used as the input to the networks. To further reduce the false positives (FPs), we leverage the similarity between neighboring MRI slices, and adopt a weighted averaging strategy to aggregate the results obtained by the Siamese neural networks. The detection performance is evaluated on a set of 26 cases using a free-response receiver operating characteristic (FROC) analysis. The results show that the proposed approach correctly detects all the spinal metastatic lesions while producing only 0.40 FPs per case. At a true positive (TP) rate of 90%, the use of the aggregation reduces the FPs from 0.375 FPs per case to 0.207 FPs per case, a nearly 44.8% reduction. The results indicate that the proposed Siamese neural network method, combined with the aggregation strategy, provide a viable strategy for the automated detection of spinal metastasis in MRI images. Keywords: deep learning, Siamese neural network, multi-resolution analysis, spinal metastasis, magnetic resonance imaging 23 disease (e.g. involving single or multiple segments). However, despite the 24 advantages mentioned above, the manual detection of spinal metastasis in 25 3 MRI is time-consuming and tedious considering the large number of slices in 26 each MRI sequence, as well as the large number of MRI sequences usually 27 acquired for each patient. Therefore, it is now becoming essential to develop 28 computerized algorithms for automated detection of spinal metastases in 29 MRI sequences. 30 Especially, as image acquisition speed improves, many more images with a 31 higher spatial resolution can be acquired during an examination, and as such 32 developing computer-aided analysis methods is essential to assist radiologists 33 in making a thorough evaluation of the entire image set within a reasonable 34 reading time. This is a trend for all imaging modalities (and all patholo-35 gies), and particularly true for MRI because of the multiple sets of images 36 acquired using different sequences. Although at the present time computer-37 aided analysis cannot yet replace visual inspection by trained radiologists, 38 nonetheless it can already provide an important tool for displaying the most 39 critical information from hundreds of images, in a convenient way, to assist 40 radiologists during diagnosis. In time, computerized methods may become as 41 good, or even better than human experts and lead to significant economies of 42 scale and accuracy. Thus, in short, it is important to develop computerized 43 methods to analyze MRI and other imaging modalities in medicine. 44 Given the importance of automated spinal metastasis detection, a few 45 approaches have been developed in the literature. For example, Roth et al. 46 [10] used a deep convolutional neural network (CNN) as the 2nd tier of a two-47 tiered, coarse-to-fine, cascade framework to refine the candidate lesions from 48 the first tier for sclerotic spine metastasis detection in computer tomography 49 (CT) images. Wiese et al. [11] developed an automatic method based on a 50 4 watershed algorithm and graph cut for detecting sclerotic spine metastases in 51 CT images. And Yao et al. [12] applied a support vector machine to refine 52 the initial detections produced with a watershed algorithm for lytic bone 53 metastasis detection in CT images. However, as can be seen, these studies 54 are based on CT images and do not use spinal MRI sequences. 55 On the MRI side, efforts for analyzing MRI sequences have focused on 56 different problems. For example, Carballido-Gamio et al. [13] developed a 57 normalized cut method for vertebra segmentation in spinal MRI. Huang et al. 58 [14] proposed an AdaBoost method for vertebra detection and an iterative 59 normalized cut algorithm for vertebra segmentation in spinal MRI. Neubert 60 et al. [15] designed an automatic method using statistical shape analysis 61 and registration of gray level intensity profiles for 3D intervertebral disc and 62 vertebral body segmentation in MRI. However, in spite of these efforts and to 63 the best of our knowledge, there is no study in the literature that is focused 64 on detecting spinal metastases in MRI sequences. 65 Automated and accurate spinal metastasis detection in MRI is a difficult 66 task, in large part because of the considerable variability in the size of ver-67 tebrae. Spinal metastases usually grow in the vertebrae, which are divided 68 into five regions: cervical, thoracic, lumbar, sacrum, and coccyx. The size of 69 the vertebrae varies considerably both within an individual, as well as across 70 individuals. For example, Zhou et al. [16] investigated the lumbar vertebrae 71 from 126 CT images and found that the upper vertebral width is 40.9 ± 3.6 72 mm in females and 46.1 ± 3.2 mm in males at L3, 46.7 ± 4.7 mm in females 73 and 50.8±3.7 mm in males at L4, and 50.4±4.4 mm in females and 54.5±4.9 74 mm in males at L5. 75 5 In recent years [17], neural networks and deep learning have been used to 76 successfully tackle a variety of problems in engineering, ranging from com-77 puter vision [18, 19, 20] to speech recognition [21], as well as in the natural 78 sciences, in areas ranging from high energy physics [22, 23], to chemistry 79 [24, 25], and to biology [26, 27]. Thus it is natural to consider applying 80 deep learning methods also to biomedical images. For example, Ciresan et 81 al. applied a deep learning architecture to each pixel for addressing prob-82 lems of membrane segmentation in electron microscopy images [28]; Shen et 83 al. developed multi-scale convolutional neural network for lung nodule de-84 tection in CT images [29]; Wang et al. adopted a GoogLeNet-based method 85 for automated detection and diagnosis of metastatic breast cancer in whole 86 slide images of sentinel lymph node biopsies [30]; and Wang et al. devised 87 a 12-layer convolutional neural network for cardiovascular disease detection 88 in mammograms [31]. Therefore it is natural to hypothesize that neural net-89 works and deep learning methods can be harnessed for the effective detection 90 of spinal metastases in MRI sequences. 91 Thus, in short, the purpose of this study is to develop an accurate comput-92 erized method to locate metastatic cancer in the spine using deep learning 93 methods. Specifically, a multi-resolution analysis is proposed to deal with 94 the large variability of vertebral sizes, and a Siamese neural network [32, 33] 95 is developed to incorporate the multi-resolution representation of each MRI 96 slice. More precisely, for each location under consideration, a set of image 97 patches centered at this location and at different resolutions are extracted 98 from the MRI slice, resulting in a multi-resolution representation of the in-99 put. Then the multi-resolution image patches are fed into a Siamese neural 100 6 network to predict the probability that the corresponding location corre-101 sponds to a metastatic lesion. The Siamese neural network is comprised of 102 several identical twin networks, each dealing with patches of a different reso-103 lution. To further remove false positives (FPs) in spinal metastasis detection, 104 we consider the structural similarity of neighboring slices in MRI sequences 105 and aggregate the outputs of the Siamese neural networks using a weighted 106 averaging approach. 107 The rest of paper is organized as follows. The proposed Siamese neural 108 network and slice-based aggregation methods for spinal metastasis detection 109 are descried in Section 2 together with other methodological aspects. The 110 data collection and experiments are described in in Section 3. Finally, the 111 results are presented in Section 4 and discussed in Section 5. 112 2. Methods 113 126 putationally efficient. However, in our case, if the local image patch is too 127 small and the vertebra is large, then the detector sees only a fraction of the 128 vertebra and cannot accurately distinguish between lesion and normal. In 129 contrast, if the local image patch is big and the vertebra is small, then most of 130 the information contained in the patch is not relevant for the discrimination. 131 Therefore, to accommodate for large variations in vertebral size, we propose 132 a multi-resolution approach for spinal metastasis detection. Specifically, for 133 each location of interest, we extract three image patches with same size in 134 pixels from each MRI slice, corresponding to three different resolutions, yield-135 ing three different local representations of the location under consideration. 136 8
In the literature of REF , Juan et al. proposed a method that automatically detect the spinal metastasis in the magnetic resonance imaging (MRI).