Deep Learning in Medical Image Analysis

Deep Learning for Computer-Aided Detection

The goal of CADe is to find or localize abnormal or suspicious regions in structural images, and thus to alert clinicians. CADe aims to increase the detection rate of diseased regions while
⁶ D( A, B) = 2( A ∩ B)/( A + B), where ∩ is the intersection.
1

64

₃ 32
_{3 3} 3

3
³ 3

32 3

64

2
16 ₁₆ ³²

3
^{3 32} 64

64 ^{3 3}

64

64
₆₄ 32
32 ₃₂
32
¹ 32
16 ³



64
64
16 _4
4 4
64 ⁴

64

Figure 8

Convolution

First layer group

Convolution

Second layer group
Convolution


Third layer group
Deconvolution


Fourth layer group
4
Deconvolution


Fifth layer group

Annu. Rev. Biomed. Eng. 2017.19:221-248. Downloaded from www.annualreviews.org Access provided by 82.215.98.77 on 06/08/22. For personal use only.

The architecture of the fully convolutional network used for tissue segmentation in Reference 48.

reducing the false-negative rate, which may be due to error or fatigue on the part of the observers. Although CADe is well established in medical imaging, deep learning methods have improved its performance in different clinical applications.

Typically, CADe occurs as follows: (a) The candidate regions are detected by means of image processing techniques; (b) the candidate regions are represented by a set of features, such as morphological or statistical information; and (c) the features are fed into a classifier, such as a support vector machine (SVM), to output a probability or make a decision as to whether disease is present. As explained in Section 1, human-designed feature representations can be incorporated into deep learning. Many groups have successfully used their own deep models in applications such as detection of pulmonary nodules, detection of lymph nodes, classification of interstitial lung disease in CT images, detection of cerebral microbleeds, and detection of multiple sclerosis lesions in MR images. Notably, most of the methods described in the literature exploited deep convolutional models to maximally utilize structural information in two, two-and-a-half, or three dimensions.
Ciompi et al. (43) used a pretrained OverFeat (111) out of the box as a feature extractor and empirically showed that a CNN learned from a completely different domain of natural images can provide useful feature descriptions for classification of pulmonary perifissural nodules. Roth et al. (40) focused on training deep models from scratch. To confront the problem of data insuffi- ciency in training deep CNNs, they expanded their data set by scaling, translation, and rotation in random overtraining samples. They augmented the test samples in a similar way; obtained CNN outputs for every augmented test sample; and took the average of the outputs of the randomly transformed, scaled, and rotated patches for detection of lymph nodes and colonic polyps. To bet- ter utilize volumetric information in images, both Ciompi et al. (43) and Roth et al. (40) considered two-and-a-half-dimensional (2.5D) information with 2D patches of three orthogonal views (axial, sagittal, and coronal). Setio et al. (42) considered three sets of orthogonal views for a total of nine views from a 3D patch and used ensemble methods to fuse information from different views for detection of pulmonary nodules.
Gao et al. (112) focused on the holistic classification of CT patterns for interstitial lung disease by using a deep CNN. They borrowed the network architecture from Reference 113, with six units

Annu. Rev. Biomed. Eng. 2017.19:221-248. Downloaded from www.annualreviews.org Access provided by 82.215.98.77 on 06/08/22. For personal use only.
at the output layer, to classify patches into normal, emphysema, ground glass, fibrosis, micronod- ules, and consolidation. To overcome the overfitting problem, they utilized a data augmentation strategy by generating images by randomly jittering and cropping 10 subimages per original CT slice. At the testing stage, they generated 10 jittered images and fed them into the trained CNN. Finally, they predicted the input slice by aggregation, similar to the research by Roth et al. (40).
Shin et al. (45) conducted experiments on data sets of thoraco-abdominal lymph node detec- tion and interstitial lung disease classification to explore how the performance of a CNN changes according to architecture, data set characteristics, and transfer learning. They considered five deep CNNs, namely CifarNet (114), AlexNet (113), OverFeat (111), VGG-16 (115), and GoogLeNet (116), which achieved state-of-the-art performance in various computer vision applications. From their extensive experiments, these authors drew some interesting conclusions: (a) It was consis- tently beneficial for CADe problems to transfer features learned from the large-scale annotated natural image data sets (ImageNet), and (b) applications of off-the-shelf deep CNN features to CADe problems could be improved by exploring the performance-complementary properties of human-designed features.
Unlike the studies above, which used deterministic deep architectures, van Tulder & de Bruijne
(35) exploited a deep generative model with a convolutional RBM as the basic building block for classification of interstitial lung disease. Specifically, they used a discriminative RBM with an additional label layer along with input and hidden layers to improve the discriminative power of learned feature representations. These experiments demonstrated the advantages of combining generative and discriminative learning objectives by achieving higher performance than that of purely generative or discriminative learning methods.
Pereira et al. (34) investigated brain tumor segmentation by using CNNs in MR images. They explored small-sized kernels in order to have fewer parameters but deeper architectures. They trained different CNN architectures for low- and high-grade tumors and validated their method in the 2013 Brain Tumor Segmentation (BRATS) Challenge,⁷ where their technique ranked at the top for the complete, core, and enhancing regions for the challenge data set. Brosch et al. (49) applied deep learning for multiple sclerosis lesion segmentation on MR images. Their model was a 3D CNN composed of two interconnected pathways, namely a convolutional pathway that learned hierarchical feature representations similar to those of other CNNs and a deconvolutional pathway consisting of deconvolutional and unpooling layers with shortcut connections to the cor- responding convolutional layers. The deconvolutional layers were designed to calculate abstract segmentation features from the features represented by each convolutional layer and the activa- tion of the previous deconvolutional layer, if applicable. In comparison to five publicly available methods for multiple sclerosis lesion segmentation, this method achieved the best performance in terms of Dice similarity coefficient, absolution volume difference, and lesion false-positive rate.
An important limitation of typical deep CNNs arises from the fixed architecture of the models themselves. When an input observation is larger than the unit in the input layer, the straightforward solution is to apply a sliding-window strategy. However, it is computationally very expensive and time/memory consuming to do so. Because of this scalability issue in CNNs, Dou et al. (36) devised a 3D fully connected network by transforming units in the fully connected layers into a 3D (1×1×1)
convolutionable kernel that enabled an arbitrary-sized input to be processed efficiently (101). The
output of this 3D fully connected network could be remapped back onto the original input, making it possible to interpret the network output more intuitively. For detection of cerebral microbleeds in MR images, these authors designed a cascade framework. They first screened the input with
⁷ For details, refer to http://martinos.org/qtim/miccai2013/.

Annu. Rev. Biomed. Eng. 2017.19:221-248. Downloaded from www.annualreviews.org Access provided by 82.215.98.77 on 06/08/22. For personal use only.
the proposed 3D fully connected network to retrieve candidates with high probabilities of being cerebral microbleeds, and then applied a 3D CNN discrimination model for final detection. These experiments validated the effectiveness of the method by removing massive redundant computations and dramatically speeding up the detection process.

5.