White matter hyperintensity segmentation of multiple sclerosis and neuromyelitis optical spectrum disorders using 2.5D FrC-ResUnet

Introduction

Multiple sclerosis (MS) is common neuroinflammatory and neurodegenerative disease of the central nervous system (CNS), characterized by demyelinating white matter (WM) lesions in the brain and spinal cord (1-3). It is relatively uncommon in the Asia-Pacific region (4). Meanwhile, neuromyelitis optical spectrum disorder (NMOSD) is a rare inflammatory demyelinating disease of the CNS associated with aquaporin-4 immunoglobin G antibody (AQP4-IgG) (5). The worldwide prevalence is low with 0.3–4.4/100,000 (6).

Magnetic resonance imaging (MRI) is suitable for visualizing white matter hyperintensities (WMHs), which indicate demyelination lesions and are the vital radiological features in MS and NMOSD (1,2,4). The assessment of brain WMH is essential for the diagnosis, treatment, and prognosis of MS and NMOSD (6,7). However, lesion burden assessment by radiologists is a time-consuming task and is prone to significant intra- and inter-observer variability (7-10), especially when delineating the lesions on three-dimensional (3D) T2-weighted fluid-attenuated inversion recovery (T2-FLAIR) images. Therefore, a fast and fully automatic quantitative segmentation approach for MS and NMOSD brain lesions is urgently needed to assist clinical work (11).

With the emergence of convolutional neural networks (CNNs) (12) in recent years, deep learning methods have been applied to various medical image segmentation tasks with outstanding performance (13), such as MS lesion segmentation (14), brain tumor segmentation (15), and infant brain segmentation (16), among others.

The best-known model for medical image segmentation is the U-Net (17), an encoder-decoder network with skip-connection to combine low-level and high-level information. Based on U-Net, variants of U-shaped network structures have been proposed to improve the segmentation performance. In 2018, Zhang et al. proposed ResUNet (18), which leveraged the advantages of ResNet (19) and U-net. ResUNet can facilitate information propagation and enhance segmentation accuracy. Over the years, several modifications have been made to these U-shaped structures to achieve precise and accurate segmentation results. For example, Farshad et al. proposed an architecture based on U-net, which combined frequency domain features and spatial domain features to improve the segmentation performance of retinal optical coherence tomography (OCT) images (20). Yin et al. developed a medical image segmentation network model utilizing atrous and multi-scale convolution, aiming to achieve both lightweight design and performance enhancement. Additionally, they integrated residual attention mechanism modules into skip connections (21).

Recently, research in the field of MS lesion segmentation using multiple MRI modalities and deep learning architectures has been advanced, with several methods based on CNNs being applied (22). These methods can automatically generate representative features from various types of input data, such as two-dimensional (2D) slices (23), the so-called 2.5-dimensional (2.5D) image generated by combining slices (24), or 3D volumes (25). Optimal performance can be achieved by processing intricate representations within a high-dimensional feature space. In 2019, Aslani et al. proposed a multi-branch network, which includes a multi-scale feature fusion block and a multi-scale feature up-sampling block, to combine and up-sample the features from different modalities and different resolutions (14). In 2019, Zhang et al. proposed a 2.5D method with a FC-DenseNet model and achieved the best performance in the IEEE International Symposium on Biomedical Imaging (ISBI) 2015 Longitudinal MS Lesion Segmentation Challenge (24). Rather than using regular convolutions, Rondinella et al. proposed an augmented U-Net architecture with a convolutional long short-term memory layer and an attention mechanism, enabling more precise segmentation of MS lesions (26). Kang et al. resented a 3D attention context U-Net (ACU-Net) for MS lesion segmentation, which contains a 3D spatial attention block and a context-guided module. It addresses the problem of insufficient utilization of context information and feature representation (27).

Despite significant advancements, the segmentation of MS lesions remains challenging for several reasons. Firstly, the spatial distribution and appearance of MS lesions are heterogeneous (28). Secondly, the lesions’ edges are blurry, complicating their identification. Thirdly, MS lesions constitute only a tiny fraction of the total brain volume, resulting in a highly imbalanced dataset that presents additional challenges in training models (25).

Additionally, public datasets for scientific research on NMOSD are scarce due to their rarity. Furthermore, it is often challenging to segment and quantify WMH automatically in NMOSD (as opposed to in MS or stroke), because the lesions are usually located very close to the lateral ventricle as well as the entire ventricular system (6,7). Thus, robust automatic segmentation methods for detecting NMOSD brain lesions still need to be developed.

In this work, we propose a fully automated 2.5D FrC-ResUnet deep learning approach that excels in capturing discrete MS brain WMH and accurately delineating lesion boundaries. Moreover, the model demonstrates promising results in segmenting NMOSD brain WMH. We present this article in accordance with the CLEAR reporting checklist (available at https://qims.amegroups.com/article/view/10.21037/qims-24-2384/rc).

Methods

Datasets and preprocessing

Datasets

Three datasets were used in this study. One public dataset was sourced from the MICCAI 2016 MS Lesion Segmentation Challenge (29). This study designated it as the MICCAI 2016 MS dataset for simplicity. The two local datasets include the local MS and NMOSD data, which were retrospectively enrolled from the First Hospital of Jilin University, China. The three datasets used in this study are illustrated in Figure S1.

The study was conducted in accordance with the Declaration of Helsinki and its subsequent amendments. The study was approved by the local ethics committee of the First Hospital of Jilin University. The informed written consent was provided by each participant before data collection and analysis. Brain 3D T2-FLAIR images were collected in this study. Figure 1 shows patients’ brain MR images and the matched ground-truth in these datasets separately. Table S1 presents the detailed information of all enrolled patients. The MRI acquisition parameters are detailed in Table S2.

Figure 1 Examples of image data from the three datasets. Manually segmented lesions are in red and overlaid on the T2-FLAIR images. (A) Examples of image data from the MICCAI 2016 MS dataset. (B) Examples of image data from the MS local dataset. (C) Examples of image data from the NMOSD local dataset. MS, multiple sclerosis; NMOSD, neuromyelitis optical spectrum disorder; T2-FLAIR, T2-weighted fluid-attenuated inversion recovery.

MICCAI 2016 MS dataset

The MICCAI 2016 MS dataset includes 53 MS patients. These MR images were acquired from different MRI scanners (Siemens 3T Verio scanner in Rennes, GE Discovery 3T scanner in Bordeaux, Siemens Aera 1.5T scanner in Lyon, Philips Ingenia 3T scanner in Lyon) in three centers following the Observatoire français de la sclérose eplaques (OFSEP) protocol recommendations (8). The images provided for each patient include 3D T2-FLAIR, 3D T1-weighted, and 3D T1-weighted contrast-enhanced images, etc. The dataset provided raw images and preprocessed images. All the 3D T2-FLAIR images were preprocessed using the nonlocal mean algorithm and the N4 bias field correction algorithm. Seven experts manually delineated all MS lesions, and a consensus was computed from LOP STAPLE. To comprehensively incorporate all manual annotations, our study used the consensus for each subject as the final ground-truth. One patient was excluded because there were no brain lesions.

Local MS and NMOSD dataset

In the local datasets, 36 MS subjects and 33 NMOSD subjects were recruited separately for the MS and NMOSD cohorts, respectively. All the 3D T2-FLAIR images were acquired on a 3.0T scanner (Philips Ingenia, Philips Healthcare, Best, Netherlands). The MS patients fulfilled the revised diagnostic criteria (2,28), and the NMOSD patients received the revised 2015 NMOSD diagnostic criteria (30). All the subjects had at least one lesion larger than three voxels on their brains’ 3D T2-FLAIR images. The exclusion criteria were: (I) patients with CNS comorbidities, a history of head trauma, or surgery; (II) low-quality images or images with noticeable artifacts caused by motion.

MS and NMOSD lesions were manually segmented on T2-FLAIR images using ITK-SNAP software (an open-source software: www.itk-snap.org). WM lesions were manually labeled by a radiologist (L.A.) with more than six years of diagnostic experience, and validated by another radiologist (L.Z.) with over ten years of diagnostic expertise to ensure high accuracy in the segmentation. The validated manual segmentation labels were used as the final ground-truth WMH lesion labels.

Image preprocessing

In the MICCAI 2016 MS dataset, a preprocessed version of the images was used. For the MS and NMOSD local datasets, we first used FSL-BET (Brain Extraction Tool) (31) to extract the brain mask from the T1-weighted images and applied them to the T2-FLAIR images to achieve skull stripping. Next, bias correction was performed using N4ITK (32) to account for the image contrast changes due to magnetic field inhomogeneities. Due to extensive background areas with values approximating zero in the images, which do not contain any information relevant to lesion segmentation, we finally cropped the raw images to remove the background. Precise cropping was then applied to the ground-truth. Specifically, the minimal bounding box containing all brain tissues was determined for each image. Then, the areas outside this bounding box were cropped, resulting in more explicit images of brain tissues. Intensity normalization was performed using kernel density estimation (24). Moreover, during the training process, the images underwent random flipping (along one of the three axes) and random rotation, with a fixed probability of P=0.5.

Algorithm development

2.5D CNN MR images segmentation workflow

In the clinical setting, radiologists typically interpret MR images as follows: (I) Radiologists commonly review a minimum of three consecutive slices to assess whether the hyperintensity observed on T2-FLAIR images constitutes a lesion. (II) When faced with an unclear boundary of a lesion, radiologists reconstruct the axial images to the coronal and sagittal views to facilitate decision-making processes. Thus, the 2.5D CNN segmentation workflow is adopted in this work, mirroring the clinical approach (see Figure 2) (24). This workflow decomposes 3D volumes into 2D slices from three dimensions and uses adjacent slices around a slice as multi-channel input to supplement spatial information. Therefore, the input size of FrC-ResUnet is (B, 3, W, H), where B represents the batch-size, and W and H denote the slice dimensions.

Figure 2 The overall workflow of the 2.5D CNN MRI image segmentation method. 3D, three-dimensional; CNN, convolutional neural network; MRI, magnetic resonance imaging.

FrC-ResUnet structure

Different 2D CNN structures may significantly impact the final segmentation results. Therefore, the Fourier Convolutional ResUnet (FrC-ResUnet) structure is proposed to achieve better segmentation performance. The overall structure of the proposed model is depicted in Figure 3A. In FrC-ResUnet, the ResUnet (18) serves as the backbone. A Spectral Encoder is introduced to facilitate the extraction and processing of global frequency domain features (20), which may be overlooked by spatial convolutions. Additionally, the selective feature module (SFM) (33) and the convolutional block attention module (CBAM) (34) are incorporated into the Spatial Decoder to suppress noise and enhance relevant features.

Figure 3 The structure of Frc-ResUnet, FC_down Block, and the selective feature module. (A) The structure of our proposed FrC-ResUnet. ResUnet serves as the backbone, and we incorporate the Spectral Encoder in yellow, the SFM in orange, and the CBAM in pink. (B) The structure of FC_down Block. (C) The structure of selective feature module. BN, batch normalization; CBAM, convolutional block attention module; FC, Fourier Convolutional; FFT, Fast Fourier Transform; IFFT, Inverse Fast Fourier Transform; MP, max-pooling; ReLU, rectified linear unit; SFM, selective feature module.

These designs assist the model in identifying discretely distributed small lesions and delineating the contours of clustered lesions, thereby achieving precise lesion segmentation.

Spectral encoder

Figure 3B illustrates the specific structure of the Spectral Encoder. It consists of four Fourier Convolutional down blocks (FC_down block), and each block has two inputs and two outputs of global information x_g and local information x_l. Image pixels are regarded as local information, and the original input image lacks global information. Therefore, the inputs of the first FC_down block, x_g and x_l, are zero and the original image, respectively. Each block uses Fast Fourier Transform (FFT), Inverse Fast Fourier Transform (IFFT), and convolutions to extract frequency domain features from x_g. Simultaneously, three convolutions are used to extract global and local spatial features from x_g and x_l. Local spatial features are separately element-wise added with frequency domain features and global spatial features. Following this, batch normalization (BN), an activation function (rectified linear unit, ReLU), and a max-pooling (MP) are applied to generate the input for the subsequent FC_down block.

SFM

Merely element-wise adding features from the Spatial and Spectral Encoders is not a practical approach to extracting image information. Therefore, a nonlinear method dynamically aggregates these features, allowing FrC-ResUnet to adjust feature weights adaptively to learn the most credible and beneficial information for accurate segmentation. The specific structure is shown in Figure 3C.

Firstly, the Spatial Encoder extracts spatial features S∈RH×W×C, and the Spectral Encoder extracts frequency domain features F∈RH×W×C. The size of the feature maps is H × W, with C channels. These extracted features, S and F, are fused by element-wise addition and embedded by simple global average pooling to obtain the global information vector g∈RC×1. The elements g_c in g are calculated from channel c of S and F.

gc=1H×W∑i=1H∑j=1W(Sc+Fc)[1]

Secondly, g is fed into a fully connected layer to create a more compact feature a∈Rd×1. This dimensionality reduction operation can reduce computational costs. The output dimension d is controlled with a reduction ratio, which allows for a trade-off between performance and computational cost:

d=max(C/r,L)[2]

where L denotes the minimal value of d (γ=2 and L=32 are typical settings in our experiments).

Thirdly, soft attention across channels is used for precise and adaptive selection. Specifically, a softmax operator is applied on the channel-wise digits, where a_S, a_F denote the soft attention vector for S and F, respectively.

Finally, the final feature map V is obtained by assigning different attention weights to the features extracted from different domains. The elements V_c in V are computed by the following formula:

Vc=aSc⋅Sc+aFc⋅FcaSc+aFc=1[3]

CBAM

Convolutions can mix cross-channel and cross-space information to extract features. So the CBAM is utilized to emphasize relevant features and suppress irrelevant features. It sequentially includes channel and spatial attention modules (see Figure 4).

Figure 4 The structure of CBAM. (A) The structure of channel attention module. (B) The structure of spatial attention module. CBAM, convolutional block attention module; MLP, multiple layer perception.

The input feature map x∈RC×H×W, α∈RC×1×1 represents the channel attention map and β∈R1×H×W represents the spatial attention map.

The entire process is described as follows:

y′=α(x)⊙x[4]

y″=β(y′)⊙y′[5]

where ⊙ denotes element-wise multiplication, y' and y'' represents the output of channel and spatial attention, respectively.

The structure of the channel attention module is shown in Figure 4A. For the input feature map x, global average and maximal poolings are first used to obtain global information along the channel dimension. Then, they are fed to a shared multiple layer perception (MLP) to compute the channel attention matrix. The structure of the spatial attention module is shown in Figure 4B. Average and maximum pooling are applied along the channel dimension first. Then, they are concatenated and encoded.

Loss function

Since the lesion area is minimal compared to the whole brain image, there is a data imbalance problem. In this paper, Focal loss is used as the loss function to overcome this problem effectively (35).

The formula for the Focal Loss function is:

FL(p,y)=−αy(1−p)rlog(p)−(1−α)(1−y)prlog(1−p)[6]

Where y∈{0,1} represents the ground truth, p∈[0,1] represents the corresponding prediction probability, α is the weighting factor to balance positive and negative examples, γ is the modulating factor to down-weight easy-classified examples and focus on hard-classified examples.

Experiments

Evaluation metrics

Dice similarity coefficient (DSC)

Dice=2|X∩Y||X|+|Y|[7]

Let X denote the predicted segmented, Y represent the ground-truth, ∩ denote the intersection of X and Y. The higher DSC, the higher the similarity between the expected result and the ground-truth. A higher DSC reflects an increased similarity between the predicted outcome and the ground-truth.

Positive prediction value (PPV)

PPV=TPTP+FP[8]

TP and FP represent true positive and the false positive voxels, respectively. A higher PPV indicates the improved precision.

True positive rate (TPR)

TPR=TPTP+FN[9]

FN designates false negative voxels. A heightened TPR is indicative of superior segmentation performance.

Volume difference (VD)

VD=|TPs−TPgt|TPgt[10]

TP_s represent the total number of voxels predicted by the lesion, while TP_gt denotes the ground-truth total number of voxels for the lesion. A reduced VD signifies that the predicted lesion volume is closer to the volume of the ground-truth lesion.

Implementation details

To limit extremely unbalanced data and omit samples with insufficient information, the training subset was determined by selecting only the slices containing at least one pixel marked as the lesion. The proposed method was implemented by PyTorch (36) on a 20-core CPU and 128G memory server with one NVIDIA 1080ti GPU. The maximum training epochs was 300, with an early stop of 160. The training and testing sets were split in a 4:1 ratio across the three datasets: 42 and 10 images for the MICCAI 2016 MS dataset, 29 and 7 images for the MS local dataset, and 26 and 7 images for the NMOSD local dataset, respectively. The slices used for training were 1,1023, 7,935, and 7,840, respectively. The mini-batch size was 64, each including randomly stacked slices from three dimensions. The optimizer used in training was Adam (7). The initial learning rate is 2e-4 for 100 epochs and then linearly decayed to 0 within the set epoch number. We set γ=1 and α=0.4 for the Focal Loss function to train 2D CNN models.

Results

Quantitative results

Our proposed model (FrC-ResUnet) is compared with UNet (17), ResUNet (18), FC-DenseNet (37), and AttentionUNet (38) on three datasets: the MICCAI 2016 MS dataset, local MS and NMOSD datasets, respectively. The experimental results demonstrate that the overall performance of FrC-ResUnet surpasses other advanced CNN models. To further validate the effectiveness of our model, we conducted comparisons with both traditional machine learning and state-of-the-art segmentation methods. Specifically, we compared our results with the lesion prediction algorithm (LPA) implemented in the Lesion Segmentation Toolbox version 3.0.0 (LST, http://www.applied-statistics.de/lst.html) (39), a widely used open-source tool for lesion segmentation (40). Additionally, we evaluated our model against a more recent and advanced method: Sequence Adaptive Multimodal SEGmentation (SAMSEG), a cutting-edge segmentation algorithm from the FreeSurfer suite (39).

The experimental results on the MICCAI 2016 MS dataset are presented in Table 1. We conducted a comparative analysis of FrC-ResUnet with UNet, ResUNet, FC-DenseNet, AttentionUNet, LPA and SAMSEG. Table 1 also provides the parameters and segmentation times for each method. Our model achieved the best DSC and VD. The DSC was 2.3% higher than the second-best result, indicating that the results predicted by our method are more consistent with ground-truth. The VD exceeded the second-best result by 27.9%, indicating a diminished relative volume difference between the prediction from our model and the corresponding ground-truth.

The experimental results on the MS local dataset are shown in Table 2, and our model achieved the best DSC and TPR. The DSC was 0.3% higher than the second-best result. The TPR was 2.3% higher than the second-best result, demonstrating that our model achieves the best overlap with ground-truth. The experimental results on the NMOSD local dataset are shown in Table 3. Our model obtained the best DSC and VD, and PPV was the second-best result. The DSC exceeded the second-best method by 1.4%. The VD surpassed the second-best method by 8.6%.

To further assess the effectiveness of the proposed model, we conducted paired t-tests comparing the segmentation results of the 2.5D FrC-ResUNet with those of the LPA model across the three datasets. The resulting p-values were 0.038, 0.043, and 0.022, indicating statistically significant differences in performance between the two models. Similarly, comparisons with the SAMSEG model yielded P values of 0.040, 0.031, and 0.010, further supporting the statistical significance of the performance difference in favor of the proposed model.

Additionally, as the FrC-ResUNet is an extension of the ResUNet architecture, we performed t-tests between these two models as well. The corresponding p-values were 0.062, 0.056, and 0.053, none of which reached statistical significance. We also performed Wilcoxon signed-rank tests to compare the segmentation results of FrC-ResUnet with the baseline ResUNet across the three datasets. The corresponding P values were 0.071, 0.063 and 0.068. These results suggest that, although the FrC-ResUNet demonstrates performance improvements over the baseline ResUNet, the differences are not statistically significant.

This outcome can be attributed to the inherent characteristics of supervised deep learning. In contrast to unsupervised methods such as LPA and SAMSEG, the FrC-ResUNet adopts a fundamentally different architecture, reflecting the distinct nature of supervised learning approaches. Supervised models like ResUnet are generally more stable in performance, particularly those based on U-Net architectures, which are known for their robustness and consistency. Although the proposed model incorporates architectural enhancements and achieves improved performance metrics, the differences compared to the baseline ResUnet did not reach statistical significance, as indicated by the t-test results.

To provide a more comprehensive understanding of the model performance, we calculated the 95% confidence intervals (CIs) for the DSC and VD achieved by the proposed FrC-ResUnet model across the three datasets. For MICCAI 2016 MS dataset, the average DSC was 0.710 (95% CI: 0.676–0.744) and the VD was 0.406 (95% CI: 0.305–0.507). For local MS dataset, the average DSC was 0.667 (95% CI: 0.620–0.714) and the VD was 0.265 (95% CI: 0.177–0.353). For local NMOSD dataset, the average DSC reached 0.822 (95% CI: 0.779–0.865), while the VD was 0.186 (95% CI: 0.087–0.285).

To further validate the model’s ability to detect both small and large lesions, we conducted a statistical analysis (Table 4). We consider lesions less than 20 voxels to be small lesions and those more than 50 voxels to be large lesions. A statistical analysis was conducted on three datasets: the MICCAI 2016 MS dataset, the local MS dataset, and the local NMOSD dataset, revealing 693, 56, 908 small lesions and 906, 761, 262 large lesions, respectively. A lesion is considered detected if the predicted result overlaps with the ground truth by more than 5 voxels. Our model successfully identified 435, 32, 581 small lesions, as well as 903, 743, and 258 large lesions, respectively.

Qualitative results

Figure 5A presents the 2.5D FrC-ResUnet segmentation outputs for two patients of the MICCAI 2016 MS dataset compared with ground-truth. Among these patients, patient 3 has a low lesion load, and patient 25 has a high one. Similarly, Figure 5B illustrates the 2.5D FrC-ResUnet segmentation results for patient 42 (with low lesion load) and patient 8 (with high lesion load) from the MS local dataset. Observing these two figures reveals that our model consistently yields commendable segmentation results across both datasets, regardless of whether the lesion load is high or low.

Figure 5 Examples of segmentation using ResUnet and Frc-ResUnet. The automatically segmented lesions are colored in red. (A) MS segmentation results of 2.5D ResUnet and FrC-ResUnet on two patients from the MICCAI 2016 MS dataset compared with ground-truth annotations. Manual annotations marked in blue rectangles present small punctate lesions. (B) MS segmentation results of 2.5D ResUnet and FrC-ResUnet on two patients from the MS local dataset compared with ground-truth annotations. Manual annotations marked in blue rectangles indicate the regions between two adjacent lesions. MS, multiple sclerosis; NMOSD, neuromyelitis optical spectrum disorder; T2-FLAIR, T2-weighted fluid-attenuated inversion recovery.

T2-FLAIR images of 4 NMOSD patients were randomly selected in the NMOSD local dataset with lesion loads ranging from sparse to clustered. Figure 6 presents the segmentation visualization of these 4 patients using 2.5D FrC-ResUnet. True positive, false positive, and false negative lesions are shown in different colors. The lesion loads of patient 17, patient 3, patient 25, and patient 20 were 1.572, 7.637, 10.595 mL, and 39.017 mL, respectively. The results demonstrate the capability of 2.5D FrC-ResUnet to delineate the contours of lesions in variant appearances and lesion loads. To provide a comprehensive evaluation, Figure 7 presents selected cases with slightly suboptimal segmentation results. To complement these examples and avoid focusing solely on isolated instances, Figure 8 illustrates the overall segmentation performance of our model compared to the ground truth across all three datasets.

Figure 6 NMOSD segmentation results of T2-FLAIR images using 2.5D FrC-ResUnet for four patients are presented. In all images, true positive, false positive, and false negative lesions are colored red, yellow, and green, respectively. NMOSD, neuromyelitis optical spectrum disorder; T2-FLAIR, T2-weighted fluid-attenuated inversion recovery.

Figure 7 Examples of segmentation using 2.5D Frc-ResUnet. The lesions are colored in red. (A-C) The segmentation results using 2.5D Frc-ResUnet on the MICCAI 2016 MS dataset, MS local dataset, and NMOSD local dataset, respectively, along with comparisons to the ground-truth annotations. The poorly segmented areas are marked with yellow rectangles. MS, multiple sclerosis; NMOSD, neuromyelitis optical spectrum disorder; T2-FLAIR, T2-weighted fluid-attenuated inversion recovery.

Figure 8 The overall segmentation results of the ground-truth and our model are shown across three datasets and five different thresholds. (A) Examples from the MICCAI 2016 MS dataset. (B) Examples from the MS local dataset. (C) Examples from the NMOSD local dataset. MS, multiple sclerosis; NMOSD, neuromyelitis optical spectrum disorder.

Ablation experiments

Comparative experiments were conducted using the NMOSD dataset to validate the efficacy of the Spectral Encoder, SFM, and CBAM modules. Table 5 presents the training results for various combinations of these three modules, while maintaining the other parameter settings constant. The results indicate that our designed architecture achieves optimal performance on DSC, TPR, and VD.

Discussion

Accurate and robust WMH segmentation is a crucial post-processing related task in clinical studies and research (41-43). Research on the automatic segmentation of MS lesions using deep learning models has been ongoing for many years and has yielded numerous reliable outcomes. CNN architectures remain the most widely used due to their low resource requirements and high segmentation performance (22). However, owing to the irregular shapes and dispersed distribution of MS lesions, there is still significant research value in segmenting these lesions. Meanwhile, as a rare disease, research on NMOSD lesion segmentation remains relatively scarce.

Although there have been validation studies on larger sample sizes for MS lesion segmentation using deep learning segmentation methods (44), most of these methods have been trained on 2D T2-FLAIR sequences with 20–40 slices per subject. As 3D MRI techniques have become more routinely available on clinical scanners, the recently published guideline on the use of MRI in MS patients preferably recommended 3D T2-FLAIR sequences for MS diagnosis, assessment of disease activity, and monitoring the effectiveness of the disease-modifying treatments (45). Thus, labeled 3D T2-FLAIR images comprising 30,000 slices of MS and NMOSD were included in our study, although our research recruited a small sample size of MS and NMOSD subjects.

Through the analysis of three datasets, as shown in Table S1, the total lesion volume (TLV) of the MICCAI 2016 MS dataset is notably lower than the TLVs of the other two local datasets. Specific images within the MICCAI 2016 MS dataset exhibit only a limited number of small and dispersed lesions, leading to misclassifications by the other comparative models. Notably, areas such as the lateral ventricular choroid plexus, which appear hyperintense on T2-FLAIR images, are erroneously identified as lesions. Consequently, as shown in Table 1, while other models demonstrate a high TPR, their PPV is conspicuously low. Nevertheless, our model excels in segmenting these dot lesions and adeptly suppressing noise, yielding fewer false positive voxels and consequently achieving enhanced segmentation metrics, particularly in terms of VD. Despite the absence of such images in the MS and NMOSD local datasets, our model consistently achieves the highest DSC across all datasets, emphasizing its unwaveringly superior performance.

Figure 5 illustrates our model’s segmentation results on the two MS datasets, highlighting its strong performance, notably in challenging areas. The visualization results of patient 3 (Figure 5A) demonstrate that our model can accurately localize each discrete lesion site. Patient 25’s visualization results (Figure 5A) show precise segmentation even for tiny punctate lesions within blue rectangles. Patient 8’s visualization results (Figure 5B) reveal that for the prominent clustered flaky lesions, the predicted results of our method are relatively consistent with the manual segmentation results of experts, with some differences only at the lesion edges. Patient 42’s visualization results (Figure 5B) highlight our model’s ability to distinguish interrupted and connected parts between adjacent lesions. (The lesions within the upper right blue rectangle are interrupted, and those within the lower left rectangle are connected).

Unlike other brain diseases, such as tumors and cerebral hemorrhage, which feature concentrated lesions, NMOSD lesions are relatively dispersed. Most NMOSD brain lesions exhibit highly irregular shapes and lack distinct contours. Even professional radiologists struggle to delineate these blurred lesions accurately during remission. As depicted in Figure 6, our method accurately locates these lesions and is only prone to false positives or false negatives at the margins, which may affect the overall performance.

Figures 5,6 illustrate examples where our model achieves strong segmentation performance. To provide a more comprehensive evaluation and reflect the method’s overall capabilities, Figure 7 presents several cases with comparatively poorer segmentation results. Regions with suboptimal performance are highlighted using yellow rectangles. These errors are predominantly observed in small lesions and along lesion boundaries—areas that are inherently more challenging to segment. Nonetheless, the model continues to perform well in segmenting larger and more distinct lesions. When compared to the high-quality results shown in Figures 5,6, these findings suggest that while the model demonstrates improved segmentation in difficult regions, there remains room for further improvement.

LPA is a widely used open-source toolbox for brain WMH segmentation. A comparison in Tables 1-3 reveals that, across both MS datasets, LPA exhibits relatively high PPV but significantly lower TPR compared to our method. Conversely, in segmenting NMOSD lesions, LPA achieves the highest TPR but the lowest PPV. Our methods also show slightly different performances between the MS and NMOSD datasets. The differences in the susceptible parts and morphological characteristics of MS and NMOSD may contribute to variations in segmentation performance for these two diseases. Firstly, MS patients have more WMH than NMOSD patients, mainly located above the posterior horn of the lateral ventricle. Secondly, the brain lesions in NMOSD are usually located in areas with high AQP4-IgG expression, often adjacent to the ventricular system (46), thus making accurate segmentation challenging. However, our model demonstrates competitive results on the datasets of these two diseases, indicating excellent stability and scalability. Furthermore, as shown in Table 1, the deep learning methods used in the experiments outperform LPA in specific evaluation metrics and require shorter image segmentation times, highlighting the robust capability of deep learning in medical image segmentation.

Additionally, we conducted ablation experiments on the network structure, demonstrating the effectiveness of the individual modules in our proposed model. The results of the ablation experiments using the NMOSD dataset are presented in Table 4. Only adding CBAM to ResUnet improves PPV and VD, but the effect is not pronounced. Adding only the Spectral Encoder increases the DSC by 0.5%, indicating that the Spectral Encoder indeed extracts features helpful for lesion segmentation. When SFM and CBAM are added individually, DSC is slightly improved. However, when augmented with these three modules, our proposed model achieved the highest DSC, TPR, and VD, manifesting improvements of 0.7%, 2.1%, and 9.3%, respectively, over the second-highest combination. This result indicates that combining SFM and CBAM is the most effective method to extract essential features and suppress noise.

To provide a more comprehensive evaluation of the model’s performance and avoid overemphasis on isolated cases, we analyzed segmentation results across the entire patient cohort. Specifically, we aggregated the segmentation outputs for all patients within each dataset and visualized the overall lesion distribution and segmentation performance across a range of thresholds, progressing from general to more focused representations. Figure 8 illustrates the aggregated segmentation results of the ground truth and our model across the three datasets at five different threshold levels. As shown in Figure 8, the model demonstrates consistent lesion segmentation overall. With increasing threshold levels, the visualized lesions become more representative of typical cases, and it is evident that these typical lesions are segmented with high accuracy.

Our study only included single-center data, and we did not conduct multi-center validation on real-world clinical MRI data. Our future work will focus on advancing deep learning algorithms to achieve more precise segmentation of WMH across multi-centers.

Conclusions

The recently developed 2.5D FrC-ResUnet model, integrating ResUnet as its backbone network and incorporating the SFM and CBAM modules, has been demonstrated to improve MS WMH segmentation. Furthermore, it is particularly suitable for the rare NMOSD WMH segmentation.

Acknowledgments

We would like to thank the participants themselves, all of whom contributed greatly to the successful completion of this study.

Footnote

Reporting Checklist: The authors have completed the CLEAR reporting checklist. Available at https://qims.amegroups.com/article/view/10.21037/qims-24-2384/rc

Data Sharing Statement: Available at https://qims.amegroups.com/article/view/10.21037/qims-24-2384/dss

Funding: This work was supported by the Natural Science Foundation of Jilin Province (No. 20240304034SF), the Science and Technology Achievement Transformation Fund of the First Hospital of Jilin University (No. JDYY2021-A0010), and the Development Project of Jilin Province of China (No. 20240101364JC).

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at https://qims.amegroups.com/article/view/10.21037/qims-24-2384/coif). Z.S. is a current employee of Philips Healthcare. The other authors have no conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. The study was conducted in accordance with the Declaration of Helsinki and its subsequent amendments. The study was approved by the local ethics committee of the First Hospital of Jilin University. The informed written consent was provided by each participant before data collection and analysis.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. Steinman L. Multiple sclerosis: a coordinated immunological attack against myelin in the central nervous system. Cell 1996;85:299-302. [Crossref] [PubMed]
2. Thompson AJ, Banwell BL, Barkhof F, Carroll WM, Coetzee T, Comi G, et al. Diagnosis of multiple sclerosis: 2017 revisions of the McDonald criteria. Lancet Neurol 2018;17:162-73. [Crossref] [PubMed]
3. Tamanini JVG, Sabino JV, Cordeiro RA, Mizubuti V, Villarinho LL, Duarte JÁ, Pereira FV, Appenzeller S, Damasceno A, Reis F. The Role of MRI in Differentiating Demyelinating and Inflammatory (not Infectious) Myelopathies. Semin Ultrasound CT MR 2023;44:469-88. [Crossref] [PubMed]
4. Cheong WL, Mohan D, Warren N, Reidpath DD. Multiple Sclerosis in the Asia Pacific Region: A Systematic Review of a Neglected Neurological Disease. Front Neurol 2018;9:432. [Crossref] [PubMed]
5. Liu C, Shi M, Zhu M, Chu F, Jin T, Zhu J. Comparisons of clinical phenotype, radiological and laboratory features, and therapy of neuromyelitis optica spectrum disorder by regions: update and challenges. Autoimmun Rev 2022;21:102921. [Crossref] [PubMed]
6. Filippi M, Rocca MA, Ciccarelli O, De Stefano N, Evangelou N, Kappos L, Rovira A, Sastre-Garriga J, Tintorè M, Frederiksen JL, Gasperini C, Palace J, Reich DS, Banwell B, Montalban X, Barkhof FMAGNIMS Study Group. MRI criteria for the diagnosis of multiple sclerosis: MAGNIMS consensus guidelines. Lancet Neurol 2016;15:292-303. [Crossref] [PubMed]
7. Kim HJ, Paul F, Lana-Peixoto MA, Tenembaum S, Asgari N, Palace J, Klawiter EC, Sato DK, de Seze J, Wuerfel J, Banwell BL, Villoslada P, Saiz A, Fujihara K, Kim SH. Guthy-Jackson Charitable Foundation NMO International Clinical Consortium & Biorepository. MRI characteristics of neuromyelitis optica spectrum disorder: an international update. Neurology 2015;84:1165-73. [Crossref] [PubMed]
8. Commowick O, Istace A, Kain M, Laurent B, Leray F, Simon M, et al. Objective Evaluation of Multiple Sclerosis Lesion Segmentation using a Data Management and Processing Infrastructure. Sci Rep 2018;8:13650. [Crossref] [PubMed]
9. Lesjak Ž, Galimzianova A, Koren A, Lukin M, Pernuš F, Likar B, Špiclin Ž. A Novel Public MR Image Dataset of Multiple Sclerosis Patients With Lesion Segmentations Based on Multi-rater Consensus. Neuroinformatics 2018;16:51-63. [Crossref] [PubMed]
10. Sweeney EM, Shinohara RT, Shiee N, Mateen FJ, Chudgar AA, Cuzzocreo JL, Calabresi PA, Pham DL, Reich DS, Crainiceanu CM. OASIS is Automated Statistical Inference for Segmentation, with applications to multiple sclerosis lesion segmentation in MRI. Neuroimage Clin 2013;2:402-13. [Crossref] [PubMed]
11. Zheng G, Fei B, Ge A, Liu Y, Liu Y, Yang Z, Chen Z, Wang X, Wang H, Ding J. U-fiber analysis: a toolbox for automated quantification of U-fibers and white matter hyperintensities. Quant Imaging Med Surg 2024;14:662-83. [Crossref] [PubMed]
12. LeCun Y, Bottou L, Bengio Y, Haffner P. Gradient-based learning applied to document recognition. Proceedings of the IEEE 1998;86:2278-324.
13. Litjens G, Kooi T, Bejnordi BE, Setio AAA, Ciompi F, Ghafoorian M, van der Laak JAWM, van Ginneken B, Sánchez CI. A survey on deep learning in medical image analysis. Med Image Anal 2017;42:60-88. [Crossref] [PubMed]
14. Aslani S, Dayan M, Storelli L, Filippi M, Murino V, Rocca MA, Sona D. Multi-branch convolutional neural network for multiple sclerosis lesion segmentation. Neuroimage 2019;196:1-15. [Crossref] [PubMed]
15. Havaei M, Davy A, Warde-Farley D, Biard A, Courville A, Bengio Y, Pal C, Jodoin PM, Larochelle H. Brain tumor segmentation with Deep Neural Networks. Med Image Anal 2017;35:18-31. [Crossref] [PubMed]
16. Wang L, Nie D, Li G, Puybareau E, Dolz J, Zhang Q. Benchmark on automatic six-month-old infant brain segmentation algorithms: the iSeg-2017 challenge. IEEE Transactions on Medical Imaging 2019;38:2219-30. [Crossref] [PubMed]
17. Ronneberger O, Fischer P, Brox T. U-net: Convolutional networks for biomedical image segmentation. In: Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18: 2015. Springer: 234-241.
18. Zhang Z, Liu Q, Wang Y. Road extraction by deep residual u-net. IEEE Geoscience and Remote Sensing Letters 2018;15:749-53.
19. He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016; 770-8.
20. Farshad A, Yeganeh Y, Gehlbach P, Navab N. Y-Net: A spatiospectral dual-encoder network for medical image segmentation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer; 2022:582-92.
21. Yin Y, Han Z, Jian M, Wang GG, Chen L, Wang R. AMSUnet: A neural network using atrous multi-scale convolution for medical image segmentation. Comput Biol Med 2023;162:107120. [Crossref] [PubMed]
22. Shoeibi A, Khodatars M, Jafari M, Moridian P, Rezaei M, Alizadehsani R, Khozeimeh F, Gorriz JM, Heras J, Panahiazar M, Nahavandi S, Acharya UR. Applications of deep learning techniques for automated multiple sclerosis detection using magnetic resonance imaging: A review. Comput Biol Med 2021;136:104697. [Crossref] [PubMed]
23. Aslani S, Dayan M, Murino V, Sona D. Deep 2D encoder-decoder convolutional neural network for multiple sclerosis lesion segmentation in brain MRI. In: Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries: 4th International Workshop, BrainLes 2018, Held in Conjunction with MICCAI 2018, Granada, Spain, September 16, 2018, Revised Selected Papers, Part I 4: 2019. Springer: 132-141.
24. Zhang H, Valcarcel AM, Bakshi R, Chu R, Bagnato F, Shinohara RT, Hett K, Oguz I. Multiple Sclerosis Lesion Segmentation with Tiramisu and 2.5D Stacked Slices. Med Image Comput Comput Assist Interv 2019;11766:338-46. [Crossref] [PubMed]
25. Valverde S, Cabezas M, Roura E, González-Villà S, Pareto D, Vilanova JC, Ramió-Torrentà L, Rovira À, Oliver A, Lladó X. Improving automated multiple sclerosis lesion segmentation with a cascaded 3D convolutional neural network approach. Neuroimage 2017;155:159-68. [Crossref] [PubMed]
26. Rondinella A, Crispino E, Guarnera F, Giudice O, Ortis A, Russo G, Di Lorenzo C, Maimone D, Pappalardo F, Battiato S. Boosting multiple sclerosis lesion segmentation through attention mechanism. Comput Biol Med 2023;161:107021. [Crossref] [PubMed]
27. Hu C, Kang G, Hou B, Ma Y, Labeau F, Su Z. Acu-net: A 3D attention context u-net for multiple sclerosis lesion segmentation. In: ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP): IEEE; 2020:1384-8.
28. Polman CH, Reingold SC, Banwell B, Clanet M, Cohen JA, Filippi M, Fujihara K, Havrdova E, Hutchinson M, Kappos L, Lublin FD, Montalban X, O’Connor P, Sandberg-Wollheim M, Thompson AJ, Waubant E, Weinshenker B, Wolinsky JS. Diagnostic criteria for multiple sclerosis: 2010 revisions to the McDonald criteria. Ann Neurol 2011;69:292-302. [Crossref] [PubMed]
29. Commowick O, Kain M, Casey R, Ameli R, Ferré JC, Kerbrat A, Tourdias T, Cervenansky F, Camarasu-Pop S, Glatard T, Vukusic S, Edan G, Barillot C, Dojat M, Cotton F. Multiple sclerosis lesions segmentation from multiple experts: The MICCAI 2016 challenge dataset. Neuroimage 2021;244:118589. [Crossref] [PubMed]
30. Wingerchuk DM, Banwell B, Bennett JL, Cabre P, Carroll W, Chitnis T, de Seze J, Fujihara K, Greenberg B, Jacob A, Jarius S, Lana-Peixoto M, Levy M, Simon JH, Tenembaum S, Traboulsee AL, Waters P, Wellik KE, Weinshenker BGInternational Panel for NMO Diagnosis. International consensus diagnostic criteria for neuromyelitis optica spectrum disorders. Neurology 2015;85:177-89. [Crossref] [PubMed]
31. Smith SM. Fast robust automated brain extraction. Hum Brain Mapp 2002;17:143-55. [Crossref] [PubMed]
32. Tustison NJ, Avants BB, Cook PA, Zheng Y, Egan A, Yushkevich PA, Gee JC. N4ITK: improved N3 bias correction. IEEE Trans Med Imaging 2010;29:1310-20. [Crossref] [PubMed]
33. Li X, Wang W, Hu X, Yang J. Selective kernel networks. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019:510-519.
34. Woo S, Park J, Lee JY, Kweon JS. Cbam: Convolutional block attention module. In: Proceedings of the European conference on computer vision (ECCV). 2018:3-19.
35. Lin TY, Goyal P, Girshick R, He K, Dollar P. Focal loss for dense object detection. In: Proceedings of the IEEE international conference on computer vision. 2017:2980-8.
36. Paszke A, Gross S, Massa F, Lerer A, Bradbury J, Chanan G, Killeen T, Lin Z, Gimelshein N, Antiga L. Pytorch: An imperative style, high-performance deep learning library. In: Advances in neural information processing systems. 2019;32.
37. Jégou S, Drozdzal M, Vazquez D, Romero A, Bengio Y. The one hundred layers tiramisu: Fully convolutional densenets for semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops. 2017:11-9.
38. Oktay O, Schlemper J, Folgoc LL, Lee M, Heinrich M, Misawa K, Mori K, McDonagh S, Hammerla NY, Kainz B, Glocker B, Rueckert D. Attention U-Net: Learning Where to Look for the Pancreas. MIDL 2018. Available online: https://doi.org/10.48550/arXiv.1804.03999.
39. Klauschen F, Goldman A, Barra V, Meyer-Lindenberg A, Lundervold A. Evaluation of automated brain MR image segmentation and volumetry methods. Hum Brain Mapp 2009;30:1310-27. [Crossref] [PubMed]
40. Schmidt P, Wink L: LST: A lesion segmentation tool for SPM. Manual/Documentation for version 2017;2:15. Available online: https://www.applied-statistics.de/lst.html#:~:text=The%20toolbox%20%22LST%3A%20Lesion%20Segmentation%20Tool%22%20is%20an,to%20segment%20T2%20hyperintense%20lesions%20in%20FLAIR%20images.
41. Guo C, Ferreira D, Fink K, Westman E, Granberg T. Repeatability and reproducibility of FreeSurfer, FSL-SIENAX and SPM brain volumetric measurements and the effect of lesion filling in multiple sclerosis. Eur Radiol 2019;29:1355-64. [Crossref] [PubMed]
42. Guo C, Niu K, Luo Y, Shi L, Wang Z, Zhao M, Wang D, Zhu W, Zhang H, Sun L. Intra-Scanner and Inter-Scanner Reproducibility of Automatic White Matter Hyperintensities Quantification. Front Neurosci 2019;13:679. [Crossref] [PubMed]
43. Guo X, Ye C, Yang Y, Zhang L, Liang L, Lu S, Lv H, Guo C, Ma T. Ensemble learning via supervision augmentation for white matter hyperintensity segmentation. Front Neurosci 2022;16:946343. [Crossref] [PubMed]
44. Gabr RE, Coronado I, Robinson M, Sujit SJ, Datta S, Sun X, Allen WJ, Lublin FD, Wolinsky JS, Narayana PA. Brain and lesion segmentation in multiple sclerosis using fully convolutional neural networks: A large-scale study. Mult Scler 2020;26:1217-26. [Crossref] [PubMed]
45. Wattjes MP, Ciccarelli O, Reich DS, Banwell B, de Stefano N, Enzinger C, et al. 2021 MAGNIMS-CMSC-NAIMS consensus recommendations on the use of MRI in patients with multiple sclerosis. Lancet Neurol 2021;20:653-70. [Crossref] [PubMed]
46. Pittock SJ, Weinshenker BG, Lucchinetti CF, Wingerchuk DM, Corboy JR, Lennon VA. Neuromyelitis optica brain lesions localized at sites of high aquaporin 4 expression. Arch Neurol 2006;63:964-8. [Crossref] [PubMed]