Mamba-based brain tumor segmentation of incomplete multi-modal MR images

Introduction

Background

Accurate evaluation of brain tumors is not only essential for understanding the progression of various types of encephalopathy and tumor-related systemic diseases but also plays a critical role in clinical diagnosis, prognosis, and treatment planning. Among available imaging modalities, magnetic resonance imaging (MRI) stands out as a widely used, non-invasive structural imaging technique capable of providing detailed and high-resolution visualizations of intracranial anatomy. In particular, multi-modal MRI—which includes sequences such as fluid-attenuated inversion recovery (FLAIR), T1-weighted (T1), contrast-enhanced T1 (T1c), and T2-weighted (T2)—offers complementary information by capturing different tissue characteristics, contrast levels, and temporal dynamics (1). However, in real-world clinical settings, complete multi-modal magnetic resonance (MR) images are often not available for every patient case. This limitation poses significant challenges for automated medical image analysis and diagnostic algorithms that rely on full multi-modal input. These practical challenges underscore the need for robust and adaptable methods capable of handling incomplete multi-modal data while still maintaining reliable diagnostic performance (2). In addition, brain tumors in MRI scans often visualize heterogeneous structures, meaning different sub-regions of the tumor show distinct biological and radiological characteristics. MRI sequences (e.g., T1, T1c, T2, FLAIR) capture different biological properties for brain tumor evaluation (3).

Existing approaches based on convolutional neural networks (CNNs) have played a central role in medical image segmentation, since automating the tasks of detecting, segmenting, and classifying tissue or lesions can free up the doctors’ time for higher value tasks and reduce errors due to fatigue and subjectivity (4,5). U-Net architectures are often developed, such as a Sharp U-Net architecture, which is designed with new connections between the encoder and decoder subnetworks using a depthwise convolution of the encoder feature maps with a sharpening spatial filter to address the semantic gap issue between the encoder and decoder features (4,5). However, due to the inherent locality of convolution operations, CNN-based models often struggle to capture global contextual features, which limits their ability to fully exploit the complementary information across multiple modalities. While CNNs excel at capturing local, hierarchical features (edges, textures, small patterns), their fixed and often limited receptive fields struggle to model long-range dependencies. In medical imaging, understanding the global context of an organ or lesion (e.g., the relationship between different parts of a large tumor, or how a kidney relates to the spine) is crucial for accurate segmentation. Without sufficient global context, CNNs can produce fragmented segmentation or miss large, diffuse structures. CNNs focus heavily on local neighborhoods, which can lead to misclassifications when global information is required to disambiguate similar local patterns.

More recently, vision transformers (ViTs) have been successfully applied to medical image segmentation, thanks to their self-attention mechanism, which is particularly effective at modeling long-range dependencies. By dividing input images into patches or tokens, transformers are able to compute pairwise interactions between all patches, enabling a more holistic understanding of the image. This ability has led to the integration of transformer modules in multi-modal MRI segmentation frameworks (6-9), improving the capture of global features and mitigating the effects of absent or incomplete modalities. However, this comes at a significant computational cost, scaling quadratically with the input sequence length. For high-resolution two-dimensional (2D) medical images or even three-dimensional (3D) volumetric data, this becomes computationally prohibitive and memory-intensive, limiting their practical application.

State-space models (SSMs) like Mamba have demonstrated potential on effectively reducing computational complexity to a linear level while capturing global features. This motivates us to develop a new framework for multi-modal medical image segmentation task. Mamba’s selective SSM design allows it to model long-range dependencies with computational complexity that scales linearly with the sequence length. This enables it to order the magnitude more efficient than transformers for high-resolution images, unlocking the possibility of processing entire medical scans (even 3D volumes) without downsampling them excessively or resorting to patch-based inference that compromises global context. Despite the linear complexity, Mamba has demonstrated the ability to effectively capture global contextual information, similar to transformers. It does this through its selective scan mechanism, which allows it to dynamically filter and propagate information along the sequence, effectively focusing on relevant long-range dependencies.

In this work, a new framework is designed to seamlessly integrate Mamba’s global context modeling with CNNs’ robust local feature extraction capabilities. This allows the model to simultaneously extract the fine details (e.g., tumor boundaries, subtle texture changes) and the broader anatomical context (e.g., the tumor’s relationship to surrounding organs, its overall shape and extent). Different MRI modalities (T1, T1c, T2, FLAIR) provide distinct, complementary information. A new framework can incorporate separate CNN encoders for each modality to extract modality-specific features. These features can then be fused before or within Mamba blocks to allow the SSM to learn global correlations across modalities. This is a powerful way to leverage all available modality features. Cross-level fusion Mamba (CFM) blocks are designed to capture global features of the low-level feature through its contextual learning mechanism. Cross-level feature fusion can be extended to fuse features not only across different levels within a single modality’s processing stream but also across different modalities at various levels of abstraction. This allows the model to leverage the complementary information from all available MRI sequences. By integrating information from diverse modalities, the model can create a richer representation that better distinguishes tumor subregions [e.g., enhancing tumor (ET), necrosis, edema] from healthy tissue, even in cases where one modality alone might be ambiguous, and therefore, cross-level fusion can potentially improve boundaries of tumor subregions. In addition, a cross-level uncertainty (CU) constraint is imposed on each class of the final predicted tumor. This strategy integrates modal information via lesion uncertainty and employs a weight mechanism, guiding the model to learn more discriminative regional features. Cross-level features imply that uncertainty is considered not just at the final output layer but potentially at different feature levels within the network. High-level features might capture the overall presence of a tumor but lack precise boundary information, leading to uncertainty at fine scales. Low-level features might have sharp edges but lack global context, leading to uncertainty in segmentation. By imposing a constraint across these levels, the model is forced to ensure that its uncertainty estimations are consistent and informed by both coarse and fine details. For example, if a high-level feature map is uncertain about a tumor region, the low-level features contributing to that region should also reflect a corresponding level of uncertainty regarding their local details. Uncertainty is often highest at object boundaries or inconsistent regions among different modalities. By specifically constraining uncertainty at these critical regions across different feature levels, the model is compelled to learn more precise and confident delineations. It forces the network to refine the features that contribute to distinguishing the tumor from its surroundings.

Related work

Brain tumor segmentation in multi-modal MR images is a critical task for diagnosis, treatment planning, and prognosis. Recent research has heavily focused on leveraging deep learning techniques, particularly CNNs and their variants, due to their remarkable performance in image analysis. Multi-modal MRI (e.g., T1, T1c, T2, FLAIR sequences) provides complementary information about the tumor, its subregions (ET, necrotic/cystic core, and peritumoral edema), and surrounding healthy tissue, which is crucial for accurate delineation.

Multi-modality MR image for brain disease diagnosis

Deep learning is increasingly utilized in medical imaging for various applications, including cancer detection, neuroimaging for cognitive impairment such as Alzheimer’s disease (AD), and real-time imaging for surgical guidance (10). Deep learning models, such as CNNs, have been widely applied to MRI data for AD detection. The Dual-3DM3AD approach, which uses a mixed transformer-based semantic segmentation and triplet pre-processing method, represents a significant advancement in the early multi-class diagnosis of AD (11).

Deep learning models can classify brain tumors into different categories, such as glioma, meningioma, and pituitary tumors, based on MRI features (12). Ensemble deep learning techniques have been employed to improve the accuracy and robustness of brain tumor classification (13). Deep learning models can differentiate between high- and low-grade glioblastoma multiforme while varying feature correlation limits (14).

Deep learning techniques can predict survival in patients with brain tumors, aiding in treatment planning and patient management (15). Predicting the survival of patients with brain tumors is a critical task that can inform treatment decisions (16). Deep learning models can integrate clinical factors, imaging features, and genomic data to provide personalized survival predictions (17). Dynamic segmentation networks integrate adversarial learning, dynamic CNNs, and attention mechanisms to enhance performance (18). Radiomics models, based on multi-view multiscale analysis, have been used to predict overall survival (19). Context-aware deep learning models can also predict survival by considering the uncertainty of tumor location in radiological images (20).

Multi-modality MR image segmentation

The U-Net architecture remains a cornerstone for medical image segmentation due to its ability to capture both local and global contextual information. Integrating attention gates or modules (e.g., convolutional block attention module, coordinate attention) (21,22) helps the model focus on salient features, particularly in the tumor regions, and suppress irrelevant information, leading to better segmentation accuracy and boundary delineation. Researchers are exploring multi-encoder U-Nets (23) that can process different MRI modalities separately and then fuse the extracted features at various levels. This allows for better leveraging of the distinct information from each modality. For volumetric MRI data, 3D U-Nets or extensions like V-Net (24) are commonly used to capture 3D spatial relationships and hierarchical features, providing better volumetric segmentation. However, they often come with increased computational complexity. Transformers, originally popular in natural language processing, are gaining traction in medical imaging due to their ability to model long-range dependencies. UNETR (Transformers for 3D Medical Image Segmentation) (25) is an example of this trend. While promising, integrating them effectively with CNNs (hybrid models) is a current area of research. While primarily an object detection model, improved versions of YOLO are being adapted for segmentation tasks in brain MRI, incorporating modules like Atrous Spatial Pyramid Pooling (ASPP) and attention mechanisms to enhance multi-scale feature representation and focus on critical tumor features (26). Models are designed to dynamically focus on salient features across modalities by jointly learning interdependencies between imaging sequences, leading to more precise boundary delineations, especially for challenging cases like low-grade astrocytomas (27). Quantifying the uncertainty of segmentation predictions is becoming increasingly important for clinical adoption (28). This helps clinicians identify areas where the model might be less confident, guiding manual review and building trust in CNNs. Approaches include Monte Carlo dropout and test-time augmentation. While complex models often achieve higher accuracy, computational intensity is a concern. Brain tumors exhibit high variability in shape, size, location, and intensity characteristics, making segmentation challenging. Models need to be robust to this variability (29). Ensemble networks that incorporate multi-view attention mechanisms are gaining prominence in brain tumor segmentation (30). These networks leverage information from multiple MRI modalities, such as T1, T2, and T1c images, to improve segmentation accuracy (31). Multi-view attention mechanisms allow the model to focus on the most relevant features in each modality, while the ensemble approach combines the predictions of multiple models to enhance robustness and reduce errors (32). Integration of multi-modal images across multiple scales, and effectively eliminating noise interference in brain tumor images, remains a major limitation in segmentation tasks (33).

Absent modality synthesis

The field of incomplete multi-modal MR image segmentation has seen the emergence of generative methods in addressing the issues of absent modalities by synthesizing the absent data from available modalities. U-HVED (34) was a hetero-modal variational encoder-decoder for tumor segmentation of absent modalities. Multi-modal variational auto-encoders were introduced to create the common latent variables, and a mixture sampling procedure was introduced to evaluate the reconstruction error on all the modalities. Sharma et al. (35) proposed a variant of generative adversarial network (GAN) capable of leveraging redundant information contained within multiple available sequences in order to generate one or more absent sequences for a patient scan. Hyper-GAE (36) was a unified and adaptive multi-modal MR image synthesis network for tumor segmentation with absent modalities. The feature-level and image-level completion was achieved by a shared hyper-encoder for embedding each available modality into the feature space, a graph-attention-based fusion block to aggregate the features of available modalities to the fused features, and a shared hyper-decoder for image reconstruction. In addition, a hypernet-based modulation module to adaptively utilize the real and synthetic modalities. Meng et al. (37) proposed a unified multi-modal modality masked diffusion network to synthesize arbitrary absent modalities of brain MRI within a single network. Based on the learned common latent space, absent modalities can be drawn from Gaussian noise via a diffusion model. Zhang et al. (38) proposed a unified multi-modal image synthesis method for absent modality imputation. A commonality and discrepancy-sensitive encoder for the generator was designed to exploit both modality-invariant and specific information contained in input modalities. A dynamic feature unification module was designed to hardly and softly integrate information from a varying number of available modalities. A multi-input, multi-output network was proposed to combine information from all the available pulse sequences and synthesize the absent ones in a single forward pass.

Incomplete modality fusion and interaction

HeMIS (39) was a respective convolutional pipeline to independently process each modality for computing mapwise statistics such as the mean and the variance. The mean and variance feature maps of available modalities were concatenated and fed into a final set of convolutional stages. Chen et al. (40) used feature disentanglement to decompose the input modalities into the modality-specific appearance code for each modality and the modality-invariant content code for the segmentation task. The disentangled content code from each modality was fused into a shared representation for incomplete modalities. RFNet (41) was designed with a region-aware fusion module to fuse modal feature from available image modalities according to different regions. The region-aware fusion was conducted on the divided features in each region, and modal-wise attention weights were learned individually in different regions to aggregate the corresponding features. SMU-Net (42) was a style matching U-Net (SMU-Net) for brain tumor segmentation on MR images. A content and style-matching mechanism was proposed to distill the informative features from the full-modality network into an absent modality network. mmFormer (6) consisted of the hybrid modality-specific encoders, an inter-modal transformer, and a decoder. The hybrid modality-specific encoder was used to extract both local and global context information within a specific modality by bridging a convolutional encoder and an intramodal transformer. The auxiliary regularizer in the convolutional decoder was used to force the decoder to generate segmentation even when certain modalities were absent. Qiu et al. (43) froze the model with incomplete modalities and incorporate a lightweight modal-aware adapter during finetuning. The features generated by the modality state classifier was used as prompts that were conscious of the absence of modalities. M2FTrans (7) introduced learnable fusion tokens and masked self-attention to stably build long range dependency across modalities, while being more flexible to learn from incomplete modalities and modality-specific features were further re-weighted through spatial weight attention and channel-wise fusion transformers for feature redundancy reduction and modality rebalancing. IMS2Trans (8) was a Swin Transformer network by utilizing a single encoder to extract latent feature maps from all available modalities. A lightweight shifted multi-layer perceptron coupled with a masking bottleneck was designed. A feature distillation strategy between individual modalities and the entire set of modalities was used to by comparing the features of each modality with the averaged features computed across all modalities. MIFPN (9) was designed to introduce modality invariant feature prompts in modal information interaction, guiding the model to learn the incomplete modality information and facilitating integration. Modality-aware masks and modality selection strategies were used to merge shallow features of the encoder.

Methods

To mitigate reduced model performance from absent modalities, our novel network is elaborated, specifically engineered for the segmentation of brain tumors in MRI scans with arbitrary MRI modalities absent, as shown in Figure 1A. During the interaction stage, introducing SSMs allows for capturing deep knowledge of absent modalities from cross-modal correlations, thereby effectively learning inter-modal information and guiding the model to learn information about the absent modality. As shown in Figure 1B, the Mamba block is proposed for each modality encoded feature with single-axis scan along width for efficiency. As shown in Figure 1C, the Mamba fusion (MF) block is proposed to foster the learning of absent modality features, leading to a more comprehensive representation of multi-modal MR images for tumor segmentation. CFM blocks are designed to capture global features of the low-level feature through its contextual learning mechanism, as shown in Figure 1D. A CU constraint is imposed on each class of the final predicted tumor. This strategy integrates modal information via lesion uncertainty and employs a weight mechanism, guiding the model to learn more discriminative regional features. Through these components, the model significantly improves brain tumor image segmentation accuracy, especially in scenarios with absent modalities.

Figure 1 Our method for the segmentation of brain tumors in MRI scans with arbitrary MRI modalities absent. (A) Our framework. (B) Mamba block for each modality encoded feature with single-axis scan along width for efficiency. (C) MF block to efficiently capture contextual relations. (D) CFM block as the blocks of U-Net decoder. 3D, three-dimensional; CFM, cross-level fusion Mamba; CIL, a 3D convolutional layer with a 3×3 kernel size, instance normalization, and a LeakyReLU activation function; LN, layer normalization; MF, Mamba fusion; MRI, magnetic resonance imaging; SSM, state-space model.

Datasets

Two brain tumor segmentation datasets are used to evaluate the proposed method. The two datasets were distributed in the multi-modal Brain Tumor Segmentation Challenge (BraTS) (1): BraTS2018 and BraTS2020. These datasets comprise four MRI modalities: FLAIR, T1, T1c, and T2. Following previous works (6-9,40), we have implemented several preprocessing steps, including the removal of non-brain black background and normalization of each modality to zero mean and unit variance. For consistency and fair comparison (6-9,40), the same grouping strategy as the above previous works is strictly utilized. The BraTS2018 dataset is divided into 199 training samples, 29 validation samples, and 57 test samples, with evaluation conducted using a three-fold cross-validation approach. Similarly, the BraTS2020 dataset is divided into 219 training, 50 validation, and 100 test samples. The study was conducted in accordance with the Declaration of Helsinki and its subsequent amendments.

MF block

An MF block is proposed to efficiently capture global contextual relationships across multi-modal MR images. Its core strength lies in using SSMs to effectively learn and integrate features from absent modalities during interactions, which significantly improves the completeness of the fused features. Given that fused features inherently possess extensive cross-modal correlations, vision Mamba (ViM) plays a crucial role in fostering the learning of these absent modality features. This leads to a more comprehensive representation of multi-modal MR images, ultimately ET segmentation. As illustrated in Figure 1B, our Mamba-based fusion block is specifically designed to fuse modality-specific features through these cross-modal interactions.

In our task, one of the multi-modal MR images can be defined as Im∈{IFlair,IT1,IT1ce,IT2}, m∈{Flair,T1,T1ce,T2} and y represents ground truth of the tumor annotated by experienced doctors. To better extract modality-specific feature, the modality features are independently extracted. For each modality, a modality-specific encoder is constructed to obtain the corresponding high-level feature fm∈Rc×d×h×w as

fm=Em(Im,θm)[1]

where c,d,h,w respectively denotes the channel, depth, height, and width of f_m, E_m denotes the modality-specific encoder of the m^th modality MR image I_m and θ_m denotes the learnable parameter of E_m. In our framework, E_m is constructed as the encoder of 3D-U-Net, which comprises five feature extraction blocks in succession. Each block comprises a 3D convolutional layer with a 3×3×3 kernel size, instance normalization, and a LeakyReLU activation function. A convolutional layer with a stride of 2 between consecutive levels is used to downsample the feature maps. The number of channels is sequentially set to 16, 32, 64, 128, and 256 at each level.

To efficiently capture intra-modality contextual relations, we propose a mamba block for each modality-specific feature f_m. Since Mamba can only encode sequential data, the vanilla 2D ViM block in (44) has four cross-scans horizontally and vertically across spatial dimensions to generate four spatial vision sequences, which are then processed by four separate S6 models to extract spatial features. However, such a scanning strategy is inefficient for extracting spatial features from 3D volume images. Adding scans in each additional direction increases the number of sequences in parallel and actually leads to an increase in the memory consumption, the computational complexity, and the number of parameters. Therefore, we propose the spatial unidirectional scan strategy to efficiently extract global spatial features. As shown in Figure 1B, for the input feature fin∈Rc×d×h×w of our 3D ViM block, the channel is permuted to the last dimension and layer normalization is performed to produce the feature f′in∈Rc×d×h×w. Two linear transformations are parallel applied to f′in to change the same channel as c′. The sigmoid linear unit (SiLU) activation is applied to the resulted feature in the bottom path as

fina=SiLU(LTc→c′(fin))∈Rd×h×w×c′[2]

where fina is the output feature of the bottom path, and LTc→c′ denotes a linear transformation for changing the channel from c to c′. Similarly, a linear transformation is applied to f′in to change the same channel as c′ in the top path and the channel is permuted. CIL is used to extract local features, where CIL denotes a 3D convolutional layer with a 3×3×3 kernel size, instance normalization, and a LeakyReLU activation function. A voxel-by-voxel scan along the width of feature is performed to generate a spatial vision sequence. The voxel-by-voxel scan is implemented by voxel flattening and dimension permuting and the spatial vision sequence can be obtained as f_t. This sequence f_t is then fed into an S6 model for state-space evolution and SSM within Mamba can be defined as follows:

ht=A¯ht−1+B¯ftyt=Cht[3]

where A¯,B¯∈Rdhw×c'×cssm, and C∈Rdhw×cssm are all the training parameters of the S6 model and cssm denotes the Mamba hidden state dimension. After efficient spatial Mamba processing, a layer normalization and a reshaping operation are performed to obtain a feature with d×h×w×c′, and a linear projection are performed to original embedded dimension c. Finally, the Mamba feature fssm∈Rc×d×h×w is obtained by dimension permuting as the original input as follows

fssm=Permute(LTc′→c(fina⊗reshape(LN(yt))))[4]

where ⊗ denotes the Hadamard product operation.

To simulate modality unavailability in real scenarios, a Bernoulli indicator δ={δFlair,δT1,δT1ce,δT2} is introduced for modality-specific feature zeroing, i.e., modality-specific feature of a absent modality is replaced by a zero tensor. As shown in Figure 1C, for the input zeroed and concatenated feature fδ,cc=concat(δFlairfFlair,δT1fT1,δT1cfT1c,δT2fT2)∈R4c×d×h×w of our 3D ViM block, the channel is permuted to the last dimension and layer normalization is performed to produce the feature f′δ,cc∈Rd×h×w×4c. Two linear transformations are parallel applied to f′δ,cc to change the same channel as c′. The SiLU activation is applied to the resulted feature in the bottom path as

fδ,cca,b=SiLU(LTc→c′(f′δ,cc))∈Rd×h×w×c′[5]

where fδ,cca,b is the output feature of the bottom path. Similarly, a linear transformation is applied to f′δ,cc to change the same channel as c′ in the top path and the channel is permuted to obtain fδ,cct. CIL and voxel-by-voxel scan along the width of feature are performed to generate a spatial vision sequence. The spatial vision sequence can be obtained as fδ,cct. This sequence fδ,cct is then fed into an S6 model within Mamba can be defined as follows

hδ,cct=A¯cchδ,cct−1+B¯ccfδ,cctyδ,cct=Ccchδ,cct[6]

where A¯cc, B¯cc, and Ccc are all the training parameters of the S6 model. After efficient spatial Mamba processing, a layer normalization and a reshaping operation are performed to obtain a feature with d×h×w×c′, and a linear projection are performed to original embedded dimension c. Finally, the Mamba feature fδ,ccssm is obtained by dimension permuting as the original input as

fδ,ccssm=Permute(LTc'→c(fδ,cca⊗reshape(LN(yδ,cct))))[7]

The fused Mamba feature is further added to the input zeroed and concatenated feature with a CIL operation as

ffuse=CIL(fδ,cc)+fδ,ccssm[8]

For the input concatenated feature fcc=concat(fFlair,fT1,fT1c,fT2)∈R4c×d×h×w of our 3D ViM block, the channel is permuted to the last dimension and layer normalization is performed to produce the feature f′cc∈Rd×h×w×4c. Two linear transformations are parallel applied to f′cc to change the same channel as c′. CIL and voxel-by-voxel scan along the width of feature are performed to generate a spatial vision sequence. The spatial vision sequence can be obtained as fcct. This sequence fcct is then fed into an S6 model within Mamba can be defined as

hcct=A¯cchcct−1+B¯ccfcctycct=Ccchcct[9]

The fused feature with the proposed Mamba block of available modalities is aligned with that of full modalities, such that the proposed Mamba block can extract the similar feature of full modalities. Mahalanobis distance is used to measure the difference between fused features of available modalities and full modalities as

LM=(ycct−yδ,cct)TS−1(ycct−yδ,cct)[10]

where S is the covariance matrix of the features. Unlike Euclidean distance, Mahalanobis distance accounts for the correlations between fused features according to their variance. It measures distance in terms of standard deviations from the mean of a distribution.

CFM block

Global context is critical: tumors are not isolated; their shape, location, and relationship to surrounding structures (e.g., ventricles, white matter) determine their segmentation. To fuse the fine details and the macro features, CFM blocks are designed as the blocks of U-Net decoder, as shown in Figure 1C. In this architecture, the high-level feature is upsampled and fed into the Mamba branch. The Mamba layer captures global features of the low-level feature through its contextual learning mechanism. Mamba’s contextual learning mechanism, through selective state updates and long-range propagation, efficiently models these relationships. By combining upsampled high-level features (semantic guidance) with low-level features (fine details), Mamba ensures that the global context it captures is both semantically meaningful high-level knowledge and low-level details—ultimately improving the model’s ability to segment complex, heterogeneous structures like brain tumors. This mechanism distinguishes Mamba from attention-based or purely local methods, making it well-suited for our architecture’s goal of fusing multi-scale features while preserving global contextual coherence.

To be similar to Figure 1B, the first branch upsamples and projects the input features linearly (linear), and processes the sequence data through 3D convolution operations to enhance feature representation. These processed features are then fed into the SSM. The SSM enables global modeling of the sequence, allowing effective information propagation across different voxels and capturing the sequence’s global characteristics of the high-level feature. In addition, the SSM focuses on the most important of the sequence while ignoring less relevant voxels to improve both the efficiency and effectiveness of high-level feature extraction. For the input feature ffusel−1∈Rc×d×h×w of our CF-Mamba block, upsampling is performed and the channel is permuted to the last dimension and layer normalization is performed to produce the feature f′fusel−1∈R2d×2h×2w×c. Two linear transformations are parallel applied to f′fusel−1 to change the same channel as c/2. The SiLU activation is applied to the resulted feature in the bottom path as

ffusea,l−1=SiLU(LTc→c/2(f′fusel−1))∈R2d×2h×2w×c/2[11]

where ffusea,l−1 is the output feature of the bottom path of the l−1^th level fused feature, and LTc→c/2 denotes a linear transformation for changing the channel from c to c/2. Similarly, a linear transformation is applied to ffuse'l−1 to change the same channel as c/2 in the top path and the channel is permuted of the l−1^th level fused feature. CIL is used to extract local features. The voxel-by-voxel scan is implemented by voxel flattening and dimension permuting and the spatial vision sequence can be obtained as f′t. This sequencef′t is then fed into an S6 model for state-space evolution and SSM within Mamba as

h′t=A¯′h′t−1+B¯′f′ty′t=C′h′t[12]

where A¯′, B¯′ and C′ are all the training parameters of the S6 model. After efficient spatial Mamba processing, a layer normalization and a reshaping operation are performed to obtain a feature with a size of 2d×2h×2w×c/2. The output features of the l−1 level features undergo the Hadamard product operation as

ffuse,ssml−1=ffusea,l−1⊗reshape(LN(y′t))[13]

The available and zeroed modality-specific features δFlairfFlairl−1,δT1fT1cel−1,δT1cefT1cel−1,δT2fT2l−1 are concatenated and fed into a CIL block to fuse the skip connections. Similarly, the output feature is then processed as the top path to capture long-range dependencies of the fine details in the lower-level feature, and the Hadamard product operation is used to fuse the two-level features as

ffuse,ssml=ffusea,l−1⊗reshape(LN(y″t))[14]

where y″t denotes the Mamba feature of the skip connections in the l^th level as Eq. [12]. ffuse,ssml−1 and ffuse,ssml are then concatenated and fed into a linear transformation layer and a CIL block to further fuse the two-level features. The decoder comprises four CFM blocks and a 3D convolutional layer to predict the tumor.

CU

Due to the limitation of the model, multiple-level features do not have a fixed contour or shape of a lesion or an organ, and even the neighboring-level features may vary largely from each other for the same image. The shape or size of a lesion or an organ may vary largely from each other for different patients or different time. This probably leads to the degradation of deep neural network models for variable shape or size of a lesion or an organ in multiple-level features. Therefore, it is necessary to measure the uncertain lesion area sizes.

To achieve more accurate segmentation of brain tumors in the presence of missing modalities, we adopt a deep supervision segmentation strategy. For the l^th level fused feature ffusel, a 2^l−1 upsampling operation is applied to restore the fused feature to the original resolution, and then the output feature is fed into a 3D convolutional layer to predict the tumor. The loss functions utilized in the model training process consist of Dice loss and cross-entropy (CE) loss. These two loss functions are widely used in medical image segmentation tasks due to their effectiveness in handling class imbalance and capturing fine details in segmentation boundaries. The Dice loss is particularly advantageous for improving the overlap between predicted and annotated tumors, and the CE loss provides a measure for pixel-wise classification. Compared with the ground truth to compute the loss can be defined by

y^fusel=softmax(Conv3D(Up2l−1(ffusel)))Ll=−1NvNcls∑v=1Nv∑cls=1Nclsyvclslogy^v,fusecls,l−1Ncls∑cls=1Ncls2∑v=1Nvyvclsy^v,fusecls,l∑v=1Nvyvcls+∑v=1Nvy^v,fusecls,l[15]

where y^fusel denote the l^th level predicted tumor, Up2l−1 denotes a 2^l−1 upsampling operation, Conv3D denotes a 3D convolutional layer. yvcls denotes the v^th voxel of the cls^th class of the label, and y^v,fusecls,l denotes the v^th voxel of the cls^th class of the l^th level predicted tumor. N_v denotes the number of voxels and N_cls the number of classes.

Considering that different levels of features contribute differently to the final segmentation result, we design a level consistency mechanism to measure the relationships between different levels of features and make more effective use of different levels of features. This strategy maximizes the utilization of different scales of features and semantic information at each level, enhances our understanding of brain tumors, and enables more accurate segmentation. The CU of the cls^th class can be defined as

Ucls=1L−1∑l=1L(y^fusecls,l−1L∑l=1Ly^fusecls,l)2[16]

where L denotes the number of levels. The CU can be used to constrain the final segmentation prediction as

Lf=−1NvNcls∑v=1Nv∑cls=1Ncls(1+Uvcls)yvclslogy^vcls−1Ncls∑cls=1Ncls2∑v=1Nv(1+Uvcls)yvclsy^vcls∑v=1Nvyvcls+∑v=1Nvy^vcls[17]

where y^vcls denotes the v^th voxel of the cls^th class of the final predicted tumor.

The overall loss includes the deep supervision loss L^l and the final segmentation loss L^f, written as

Ltotal=Lf+λLM+1L∑l=1LLl[18]

where λ is the coefficient of the Mahalanobis distance L_M.

Results

Evaluation metrics

To comprehensively analyze the segmentation performance of the proposed method, the dice similarity coefficient (DSC) and Hausdorff distance (HD) are used as evaluation metrics. The DSC measures the volume overlap between the predicted tumors and the manually annotated labels by experienced doctors, with higher DSC scores indicating better segmentation accuracy. The 95% HD (HD95) is used to measure the surface distances between the predicted tumors and the manually annotated labels, with lower values indicating more approximate surface as that of ground truth.

Implementation details

Our method is implemented in PyTorch. Our method and baselines are trained and tested by utilizing a Nvidia GeForce RTX 4090. During the training process, the batch size is set to 2, and the AdamW optimizer is used to optimize model parameters, with an initial learning rate of 0.0002 and a weight decay of 0.0001. Considering memory consumption, each modality is randomly cropped to a subvolume with 80×80×80 voxels, and various data augmentation techniques such as random rotations, intensity variations, and mirror flipping are used. In addition, a warm-up learning rate adjustment strategy and a polynomial decay strategy is also used. To simulate absent modality scenarios, 15 different cases of modality absent are constructed as FLAIR, T1, T1c, T2, T1c + T2, T1 + T1c, FLAIR + T1, T1 + T2, FLAIR + T2, FLAIR + T1c, FLAIR + T1 + T1c, FLAIR + T1 + T2, FLAIR + T1c + T2, T1 + T1c + T2, FLAIR + T1 + T1c + T2. The 15 patterns are sampled patterns uniformly at training time and the test set is evaluated all 15 patterns evenly. The folds of the BraTS2018 dataset and the BraTS2020 dataset are followed the previous works. In the inference stage, sliding-window inference with overlap covers the entire volume and no bias patches toward lesion voxels during training. As previous works (6-9,40,41), random factor with uniform distribution is generated to randomly select one of these scenarios to simulate absent modalities.

Comparison with state-of-the-art method

Based on the availability of source codes and data splits, six state-of-the-art methods, adopting the publicly available data splits for incomplete multi-modal brain tumor segmentation, were included for comparison, including RobustSeg (41), RFNet (41), mmFormer (6), M2FTrans (7), IMS2Trans (8), and MIFPN (9).

Comparison on BraTS2018

Table 1 summarized the quantitative comparison results across all 15 multi-modal combinations on the BraTS2018 dataset. RobustSeg achieved mean DSC scores of 83.25, 72.72, and 49.95 for whole tumor (WT), tumor core (TC), and ET, respectively. Among the comparison methods, MIFPN demonstrated the best overall segmentation performance for WT, TC, and ET, surpassing RobustSeg with a mean increase of 3.11, 5.64, and 14.53 in DSC scores, respectively. Our proposed method, however, achieved the overall best segmentation performance for WT, TC, and ET, outperforming MIFPN with an increase of 2.05, 2.86, and 7.05 in the mean DSC scores, respectively.

To validate that our method offers improvements over others, we used a Wilcoxon matched-pairs signed-ranks two-tailed test for the mean DSC and HD95 for WT, TC, and ET to calculate the P values. The P values for the scores to verify whether our method significantly improved segmentation results under these conditions. As shown in Table 2, P<0.05 of Wilcoxon matched-pairs signed-ranks two-tailed test for the mean DSC scores for WT, TC, and ET confirmed that our method statistically significantly outperformed previous methods.

In the three missing-modality patterns on the BraTS2018, the proposed method improved the mean DSC scores for WT, TC, and ET of 1.5, 1.1, and 10.3 over MIFPN and the mean HD95 for WT, TC, and ET of 0.9, 1.76, and 1.41 over MIFPN. In the two missing-modality patterns on the BraTS2018, the proposed method improved the mean DSC scores for WT, TC, and ET of 1.96, 3.2, and 5.5 over MIFPN and the mean HD95 for WT, TC, and ET of 1.16, 1.26, and 1.01 over MIFPN. In the one missing-modality patterns on the BraTS2018, the proposed method improved the mean DSC scores for WT, TC, and ET of 2.35, 3.72, and 6.79 over MIFPN and the mean HD95 for WT, TC, and ET of 0.92, 0.26, and 0.17 over MIFPN. In clinical reality, certain modality absences (e.g., missing T1c due to contrast allergy) were more common and clinically consequential than others. For the pattern of missing only T1c on the BraTS2018 dataset, the proposed method surpassed MIFPN with mean DSC score increases of 2.23, 2.09, and 13.36 for WT, TC, and ET. For the pattern of missing both T1 and T1c on the BraTS2018 dataset, the proposed method surpassed MIFPN with mean DSC score increases of 2.2, 2.32, and 1.7 for WT, TC, and ET. These results showed that less missing-modality patterns and larger gains in the mean DSC scores of the BraTS2018.

Table 3 summarized the HD95 for all fifteen multi-modal combinations on the BraTS2018 dataset. RFNet recorded the highest mean HD95 scores across all combinations, with values of 12.52, 15.3, and 10.36 for WT, TC, and ET, respectively. Among the comparison methods, MIFPN achieved the lowest mean HD95 scores at 6.84 (WT), 6.41 (TC), and 5.34 (ET). Our proposed method, however, attained the overall lowest mean HD95 for WT, TC, and ET, surpassing MIFPN with a decrease of 1.06, 1.2, and 0.98 in the mean HD95 scores, respectively. P<0.05 of Wilcoxon matched-pairs signed-ranks two-tailed test for the mean HD95 for WT, TC, and ET confirmed that our method statistically significantly outperformed previous methods.

Comparison on BraTS2020

To further analyze the impact of the number of absent modalities on our model’s segmentation performance, we conducted comparative experiments against state-of-the-art methods using the BraTS2020 dataset.

Similar observations regarding performance trends are found on the BraTS2020 dataset, as summarized in Table 4. RobustSeg recorded the lowest mean DSC scores across all fifteen multi-modal combinations for WT and TC, at 82.72 and 70.38, respectively. For ET, RobustSeg’s mean DSC score was slightly higher (0.47) than RFNet. Among the comparison methods, MIFPN achieved the highest mean DSC scores for WT, TC, and ET, outperforming RobustSeg with mean increases of 5.28, 9.77, and 17.57, respectively. Our proposed method, however, attained the best overall segmentation performance for WT, TC, and ET, surpassing MIFPN with mean DSC score increases of 1.32, 2.57, and 1.86, respectively. P<0.05 of Wilcoxon matched-pairs signed-ranks two-tailed test for the mean DSC scores for WT, TC, and ET confirmed that our method statistically significantly outperformed previous methods.

In the three missing-modality patterns on the BraTS2020, the proposed method improved the mean DSC scores for WT, TC, and ET of 0.94, 7.04, and 1.48 over MIFPN and the mean HD95 for WT, TC, and ET of 0.49, 0.97, and 0.06 over MIFPN. In the two missing-modality patterns on the BraTS2020, the proposed method improved the mean DSC scores for WT, TC, and ET of 1.19, 0.15, and 4.7 over MIFPN and the mean HD95 for WT, TC, and ET of 0.67, 0.97, and 0.55 over MIFPN. In the one missing-modality patterns on the BraTS2020, the proposed method improved the mean DSC scores for WT, TC, and ET of 1.69, 2.15, and 2.37 over MIFPN and the mean HD95 for WT, TC, and ET of 0.68, 0.41, and 1.21 over MIFPN. These results showed that less missing-modality patterns and larger gains in the mean DSC scores of the BraTS2020. For the pattern of missing only T1c on the BraTS2020 dataset, the proposed method surpassed MIFPN with mean DSC score increases of 1.74 for WT. For the pattern of missing both T1 and T1c on the BraTS2020 dataset, the proposed method surpassed MIFPN with mean DSC score increases of 0.88, 0.51, and 2.28 for WT, TC, and ET. These results also showed that less missing-modality patterns and larger gains in the mean DSC scores of the BraTS2020.

Taking into account the issue of multiple testing, we applied the Holm-Bonferroni correction, with the corrected significance threshold set at 0.05. Tables 1-5 demonstrated that our method showed statistically significant performance improvements on two datasets, with most P values being below 0.05. This confirmed the stability of our approach under various modality settings.

Computational efficiency

We have provided a comparison of computational metrics such as inference time, training time, graphics processing unit (GPU) memory footprint, or number of parameters against previous methods on the BraTS2020 dataset. The details were shown in Table 6. RFNet had the smallest number of trainable parameters, but MIFPN has the largest number of trainable parameters. Compared to the best transformer-based method MIFPN, our achieved better performance with fewer model parameters and peak GPU memory, due to the inherent property of transformers involving more model parameters. In our implementation, selective SSM/Mamba scales linearly with sequence length. This aligned with ViM’s resource consumption with progressively increasing resolutions as reported in previous work (45). IMS2Trans was Swin Transformer network which consumes the most peak GPU memory. However, the Mamba-based design offered a superior accuracy with moderate training time and least inference time, which was critical for clinical deployment.

Visualization

As illustrated in Figure 2, we tested our method to visually demonstrate its performance in scenarios with absent modalities, showcasing segmentation results across 15 configurations. Our results were optimal when all modalities were present and remained competitive even with some modalities absent. The visualizations were presented in axial, sagittal, and coronal views, as shown in Figure 2, where it clearly demonstrated that our method outperformed MIFPN in the majority of the 15 combinations with a superior segmentation performance.

Figure 2 Visualization of segmentation results under 15 different absent modality conditions for MIFPN and our method compared to ground truth. The results are shown in axial, sagittal, and coronal views. The colors represent different tumor classes: red for non-enhancing TC, green for peritumoral edema, and blue for ET. ET, enhancing tumor; FLAIR, fluid-attenuated inversion recovery; T1, T1-weighted; T1c, contrast-enhanced T1; T2, T2-weighted; TC, tumor core.

Ablation study

In this study, we have proposed a method consisting of four components: Mamba block, MF block, CFM block, and CU. In this section, we analyzed the effects of each component on the BraTS2020 dataset using the mean DSC score as the evaluation metric to demonstrate their effectiveness, as in Table 7. 3D-CNN with convolution fusion of the input zeroed and concatenated feature was used as the baseline.

Mamba block

To assess the contribution of the proposed Mamba block for feature extraction, we conducted an ablation study. As shown in Table 7, our baseline model achieved a mean Dice score of 74.05 for brain tumor segmentation. When integrating the last encoder blocks with our Mamba block, the mean Dice score increased to 76.03. Gains on ET are larger than TC and WT because edema benefits from improved discrimination by Mamba. This improvement demonstrates the effectiveness of the Mamba block in capturing multi-modality dependencies crucial for accurate tumor segmentation.

To evaluate the effectiveness of our model architecture, we have tried single unidirectional scan along height, depth, depth, width and inversed width, width and height, width, height and depth on the BraTS2020 dataset in Table 8. Contrary to expectations, the single unidirectional scan along width did not exhibit a notable decline in performance on the BraTS2020 dataset and multi-axis or bidirectional scans did not yield significant improvements. In fact, its slower convergence rate resulted in inferior performance metrics compared to the baseline under identical training settings.

MF block

The efficacy of the MF block was rigorously evaluated, focusing exclusively on its contribution to feature integration. Our findings confirmed that the MF block facilitates the effective fusion and interaction of potentially incomplete modal features.

To clearly show the contribution of this specific loss term to the model’s ability to align features from incomplete and complete modalities, we have included an ablation within Table 7 that specifically evaluated the contribution of the L_M loss term by comparing the MF module with and without it, to demonstrate its necessity for robust performance with missing modalities. Quantitative analysis, presented in Table 7, revealed improvements in segmentation performance: the mean DSC scores for the ET, TC, and WT regions increased by 2.87, 0.82, and 0.52, respectively, when incorporating the L_M loss term over the baseline + Mamba block + MF block without L_M. The mean DSC scores for the ET, TC, and WT regions increased by 4.27, 2.48, and 1.17, respectively, when incorporating the MF block over the baseline + Mamba block + L_M. This culminated in an average increase of 2.64 in the mean DSC across all three regions. These results strongly suggested that the MF block offered a superior mechanism for efficiently learning and consolidating multi-modal features during the fusion and interaction phases, particularly when contrasted with more simplistic fusion methodologies.

CFM block

We evaluated the effectiveness of the CFM block, a critical component designed to enhance multi-modal medical image segmentation. This module harnessed the global modeling strengths of SSM to ensure efficient information propagation across all voxels. This dual benefit allowed the CFM block to not only improve the representation quality of individual modalities but also to facilitate the contextual inference of absent information and the robust fusion of diverse multi-modal features. Quantitative results, presented in Table 7, demonstrated the CFM block’s individual contribution: mean DSC scores surpassed the Baseline by 2.87, 0.82, and 0.52 in the ET, TC, and WT regions, respectively, yielding an average regional improvement of 1.41.

More strikingly, the synergistic combination of the Mamba block, the MF block, and the CFM block led to even greater performance enhancements. Table 5 further showed that this combined approach increased mean DSC scores by 10.37, 4.89, and 2.83 for ET, TC, and WT, respectively, culminating in an overall average increase of 6.03 across the three regions. This compelling evidence highlighted how the incorporation of SSM modules enables our method to effectively synthesize relatively complete modal features, even in scenarios with absent input modalities.

CU

Quantitative results in Table 7 revealed that the integration of CU led to respective increases of 1.39, 0.45, and 0.38 in segmentation DSC for the ET, TC, and WT regions when compared against the ‘Mamba + MF’. These gains validated the CU strategy’s contribution to integrating multi-level features and achieving more comprehensive and robust modal representations.

To more intuitively demonstrate the effectiveness of our CU strategy, we have visualized the uncertainty maps, as shown in Figure 3. As shown in Figure 3C, the uncertainty map can highlight non-ET, and its boundary is also consistent with the contour of non-ET, indicating that our approach can focus on this region. As shown in Figure 3D,3E, the uncertainty maps also can highlight peritumoral edema and ET.

Figure 3 Visualization of the uncertainty maps. (A) The axial view of a FLAIR image. (B) The label with the FLAIR image. (C) The uncertainty map and contour of non-ET in the FLAIR image. (D) The uncertainty map and contour of peritumoral edema in the FLAIR image. (E) The uncertainty map and contour of ET in the FLAIR image. The red region denotes non-ET, the green region denotes peritumoral edema, and the yellow region denotes ET. ET, enhancing tumor; FLAIR, fluid-attenuated inversion recovery.

Hyperparameter selection

Deep learning models learn complex, often correlated, features from different modalities. For example, a feature indicating “contrast enhancement” from T1c might be correlated with a feature indicating “edema” from FLAIR in tumor regions. Euclidean distance treats all dimensions equally and independently. Two feature vectors might look “far” in Euclidean space due to high variance in a single feature, but Mahalanobis distance would correctly show them as “close” if their variation is consistent with the learned correlations within a class. This leads to more precise boundary delineation. Mahalanobis distance, by incorporating the inverse covariance matrix, inherently decorrelates the features and scales them according to their variance. This means it can accurately measure the true statistical distance in a high-dimensional fused feature space, even if individual features are highly correlated. This is crucial because it allows the model to better align fused feature of incomplete modalities to that of complete modalities.

The Mahalanobis distance weight (denoted as λ) is a critical hyperparameter in our framework, as it balances the loss term measuring the difference between fused features of available modalities and full modalities. To evaluate its impact on model performance, we conducted a systematic sensitivity analysis on the BraTS2020 dataset. We tested λ across a reasonable interval: {2.0, 4.0, 8.0, 12.0, 14.0}. This range was chosen to cover scenarios where the Mahalanobis distance term has minimal (e.g., λ=2.0) to dominant (e.g., λ=12.0) influence on the total loss. All other hyperparameters (e.g., learning rate, network depth) were fixed across experiments. For each λ, we performed 5 independent repeated experiments to account for randomness in training initialization, reporting mean ± standard deviation of key metrics in Table 9. The low standard deviation (<0.5) across 5 repetitions confirmed the consistency of these trends, reinforcing that the optimal λ range was robust to random initialization. This structure highlighted the rigor of our repeated experiments, connected the hyperparameter’s impact to model behavior, and tied the findings back to the broader utility of our method.

Discussion

Although the proposed method demonstrates superior performance compared to existing approaches on both the BraTS2018 and BraTS2020 datasets, it still suffers from the following two limitations: (I) the missingness patterns in the data of this study were artificially simulated and may not accurately represent clinically realistic, not-missing-at-random (NMAR) scenarios. In research on Mamba-based SSMs, scenarios involving missing data in incomplete multi-modal MRI are typically constructed through simulation methods (e.g., randomly removing a specific modality). The design logic of such simulations often assumes that the missing process is independent of the pathological information in MR images and the patient’s condition. However, missing data in clinical multi-modal MRI generally exhibits NMAR characteristics. Simulated missingness fails to replicate the complex relationship—where the cause of missingness is deeply tied to the clinical information in images. Consequently, when Mamba-based SSMs process real-world incomplete clinical data, they tend to suffer from biases in modal feature fusion (e.g., over-reliance on complete modalities and neglect of potential associated information from missing modalities). This ultimately leads to issues such as blurred boundary segmentation and missed detection of small lesions, limiting the model’s clinical generalization ability. As shown in Figure 4A,4B, multiple modalities without T1c where ET does not exist; however, as shown in Figure 4C, the proposed method incorrectly segments non-enhancing TC as ET and surrounding normal tissues incorrectly segments as edema. (II) The single-axis scanning mechanism of Mamba blocks processes data along one dimension, which can lead to anisotropic feature representations and reduced spatial coherence. When Mamba-based SSMs extract features from multi-modal MR images, Mamba blocks typically adopt a single-axis scanning strategy (e.g., processing 3D MR images through sequential slicing only along the z-axis, i.e., the body’s long axis). This scanning method directly induces anisotropy—an imbalance in resolution and information density across different spatial dimensions of the MR image: for example, under single-axis scanning, the slice interval along the z-axis may reach 2 mm (low resolution), while the pixel spacing in the x-/y-axis (axial plane) is only 0.5 mm (high resolution). The negative impact of anisotropy on image segmentation manifests in two key ways: first, critical anatomical details in low-resolution dimensions (e.g., z-axis) are blurred, preventing Mamba blocks from effectively capturing these weak features. Second, resolution differences across dimensions cause the Mamba-based SSM to overemphasize information from high-resolution dimensions during feature fusion, resulting in imbalanced feature weighting. This further reduces the spatial consistency of segmentation results (e.g., “discontinuities” in segmented lesion regions along the z-axis). As shown in Figure 4D,4E, multiple modalities without T1c where edema is large and small ETs scatter in the non-enhancing TC; however, as shown in Figure 4C,4F, the green arrows point to surrounding normal tissues, which are incorrectly segmented as edema by the proposed method; and the blue arrows point to small ETs, which are incorrectly segmented by the proposed method.

Figure 4 Two failure cases. (A,D) T1c image, (B,E) ground truth, and (C,F) segmented result by our method. The green arrows point to surrounding normal tissues, which are incorrectly segmented as edema by the proposed method; and the blue arrows point to small ETs, which are incorrectly segmented by the proposed method. ET, enhancing tumor; T1, T1-weighted; T1c, contrast-enhanced T1.

To address the aforementioned issues, future research plans to optimize the feature extraction process of Mamba blocks using a multi-axis scanning strategy: By scanning multi-modal MR images along the three spatial axes (x, y, and z) respectively, feature data with balanced resolution across all dimensions will be considered. This approach not only supplements critical anatomical details in low-resolution dimensions (a shortcoming of single-axis scanning) and mitigates the interference of anisotropy on feature extraction but also provides a reference for real-world clinical missing scenarios through the complete data acquired via multi-axis scanning. This will support the development of a missing data generation mechanism that aligns with NMAR patterns, thereby enhancing the clinical applicability of the model in incomplete multi-modal MR image segmentation.

The integration of deep learning into clinical radiology workflows holds significant promise for improving brain disease diagnostic accuracy and efficiency (46-48). Integrating deep learning algorithms for brain tumor segmentation into existing clinical workflows typically requires incorporating them seamlessly into the Picture Archiving and Communication Systems (PACS) (46,47,49). Future research plans to integrate the proposed method for efficient radiology operations of brain tumors. Crucially, this integration must also support the display of uncertainty maps alongside the final tumor segmentation. These maps highlight regions where the deep learning model is less confident in its prediction of the tumor boundary, providing the radiologist with vital information to identify potential segmentation errors or areas requiring further human scrutiny before finalizing the diagnostic report.

Data privacy is a significant concern when dealing with sensitive medical data such as brain MR images. Federated learning (FL) offers a promising approach to train models on decentralized data sources without directly sharing the data, thus preserving privacy (45,50,51). In a FL framework, local models are trained on individual client devices or institutions, and only the model updates are shared with a central server for aggregation (52). Future research plans to combine FL with privacy-enhancing technologies like homomorphic encryption to further protect the sensitive data of patients with brain tumors (53).

Conclusions

In this study, we have introduced an approach for interacting with incomplete multi-modal information with state-space models. During this process, the MF block is proposed to foster the learning of absent modality features, leading to a more comprehensive representation of multi-modal MR images for tumor segmentation, mitigating the challenges associated with feature incompleteness due to absent modalities and enhancing the model’s capability to navigate these complex situations. CFM blocks are designed to capture global features of the low-level feature through its contextual learning mechanism. A CU constraint is imposed on each class of the final predicted tumor. This strategy integrates modal feature via lesion uncertainty and employs a weight mechanism, guiding the model to learn more discriminative regional features. Extensive experiments on BraTS2018 and BraTS2020 show that our method exhibits superior performance under various incomplete multi-modal settings compared to existing state-of-the-art methods.

Acknowledgments

None.

Footnote

Funding: This work was supported by the Fujian Provincial Natural Science Foundation (No. 2022J011202); the Hailing District Social Development Project, Jiangsu Medical Vocational College 2024 Campus Local Collaborative Innovation Research Project (No. 202491803); the Traditional Chinese Medicine Technology Project of Taizhou 2024, and Enhancing the Innovation Capability of the Greater Health Industry (Social Development) (No. TS202305); and the Social Development Project of Hailing District, Taizhou (No. HLKF-2024-3).

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at https://qims.amegroups.com/article/view/10.21037/qims-2025-1913/coif). The authors have no conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. This study used publicly available datasets sourced from BraTS. The study was conducted in accordance with the Declaration of Helsinki and its subsequent amendments.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. Menze BH, Jakab A, Bauer S, Kalpathy-Cramer J, Farahani K, Kirby J, et al. The Multimodal Brain Tumor Image Segmentation Benchmark (BRATS). IEEE Trans Med Imaging 2015;34:1993-2024. [Crossref] [PubMed]
2. Ghaffari M, Sowmya A, Oliver R. Automated Brain Tumor Segmentation Using Multimodal Brain Scans: A Survey Based on Models Submitted to the BraTS 2012-2018 Challenges. IEEE Rev Biomed Eng 2020;13:156-68. [Crossref] [PubMed]
3. Aggarwal M, Tiwari AK, Sarathi MP. Comparative analysis of deep learning models on brain tumor segmentation datasets: BraTS 2015-2020 datasets. Rev Intell Artif 2022;36:863-71.
4. Zunair H, Ben Hamza A. Sharp U-Net: Depthwise convolutional network for biomedical image segmentation. Comput Biol Med 2021;136:104699. [Crossref] [PubMed]
5. Verma R, Kumar N, Patil A, Kurian NC, Rane S, Graham S, et al. MoNuSAC2020: A Multi-Organ Nuclei Segmentation and Classification Challenge. IEEE Trans Med Imaging 2021;40:3413-23. [Crossref] [PubMed]
6. Zhang Y, He N, Yang J, Li Y, Wei D, Huang Y, Zhang Y, He Z, Zheng Y. mmFormer: Multimodal medical transformer for incomplete multimodal learning of brain tumor segmentation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer;2022:107-117.
7. Shi J, Yu L, Cheng Q, Yang X, Cheng KT, Yan Z. Mftrans: Modality-masked fusion transformer for incomplete multi-modality brain tumor segmentation. IEEE J Biomed Health Inform 2023;28:379-90. [Crossref] [PubMed]
8. Zhang D, Wang C, Chen T, Chen W, Shen Y. Scalable Swin Transformer network for brain tumor segmentation from incomplete MRI modalities. Artif Intell Med 2024;149:102788. [Crossref] [PubMed]
9. Diao YQ, Fang HH, Yu HY, Li F, Xu YW. Multimodal invariant feature prompt network for brain tumor segmentation with missing modalities. Neurocomputing 2025;616:128847.
10. Chen XL, Xie HR, Tao XH, Wang FL, Leng MM, Lei BY. Artificial intelligence and multimodal data fusion for smart healthcare: topic modeling and bibliometrics. Artif Intell Rev 2024;57:91.
11. Khan AA, Mahendran RK, Perumal K, Faheem M. Dual-3DM3AD: Mixed Transformer Based Semantic Segmentation and Triplet Pre-Processing for Early Multi-Class Alzheimer’s Diagnosis. IEEE Trans Neural Syst Rehabil Eng 2024;32:696-707. [Crossref] [PubMed]
12. Jader RF, Kareem S, Awla H. Ensemble deep learning technique for detecting MRI brain tumors. Appl Comput Intell Soft Comput 2024;2024:6615468.
13. Islam N, Azam S, Islam S, Kanchan MH, Parvez AS, Islam M. An improved deep learning-based hybrid model with ensemble techniques for brain tumor detection from MR image. Informatics Med Unlocked 2024;47:101483.
14. Wankhede DS, Selvarani R. Dynamic architecture-based deep learning approach for glioblastoma brain tumor survival prediction. Neurosci Inform 2022;2:100062.
15. Hashmi A, Osman AH. Brain tumor classification using conditional segmentation with residual network and attention approach by extreme gradient boost. Appl Sci 2022;12:10791.
16. Yogananda CGB, Wagner B, Nalawade SS, Murugesan GK, Pinho MC, Fei B, Madhuranthakam AJ, Maldjian JA. Fully automated brain tumor segmentation and survival prediction of gliomas using deep learning and MRI. In: International MICCAI Brainlesion Workshop. Cham: Springer; 2020:99-112.
17. Khalighi S, Reddy K, Midya A, Pandav KB, Madabhushi A, Abedalthagafi M. Artificial intelligence in neuro-oncology: advances and challenges in brain tumor diagnosis, prognosis, and precision treatment. NPJ Precis Oncol 2024;8:80. [Crossref] [PubMed]
18. Yeafi A, Islam M, Yusuf SU. A deep learning framework for 3D brain tumor segmentation and survival prediction. Healthc Anal 2025;8:100418.
19. Fiaz K, Madni TM, Anwar F, Janjua UI, Rafi A, Abid MMN, Sultana N. Brain tumor segmentation and multiview multiscale-based radiomic model for patient’s overall survival prediction. Int J Imaging Sys Tech 2022;32:982-99.
20. Pei L, Vidyaratne L, Rahman MM, Iftekharuddin KM. Context aware deep learning for brain tumor segmentation, subtype classification, and survival prediction using radiology images. Sci Rep 2020;10:19726. [Crossref] [PubMed]
21. Xun SY, Zhang Y, Duan SX, Wang MW, Chen JG, Tong T, Gao QQ. LAM CT, Hu MH, Tan T. ARGA-UNet: Advanced U-Net segmentation model using residual grouped convolution and attention mechanism for brain tumor MR image segmentation. Virtual Reality Intell Hardware 2024;6:203-16.
22. Yang T, Lu X, Yang L, Yang M, Chen J, Zhao H. Application of MRI image segmentation algorithm for brain tumors based on improved YOLO. Front Neurosci 2025;18:1510175. [Crossref] [PubMed]
23. Xue RZ, Zhang ZF, Zhao YX, Zhang Q, Liang MH. MMEFU-Net: A Mamba-guided multi-encoder fusion U-Net for tumor segmentation in CT images. IEEE Access 2025;13:76257-70.
24. Prathipati SC, Satpathy SK. Transforming 3D brain tumour image segmentation: an enhanced V-Net approach for precise diagnosis and treatment planning. In: 2024 International Conference on Advances in Computing, Communication and Applied Informatics (ACCAI). IEEE; 2024:1-6.
25. Hatamizadeh A, Tang Y, Nath V, Yang D, Myronenko A, Landman B, Roth HR, Xu D. Unetr: Transformers for 3d medical image segmentation. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. IEEE; 2022:574-84.
26. Almufareh MF, Imran M, Khan A, Humayun M, Asim M. Automated brain tumor segmentation and classification in MRI using YOLO-based deep learning. IEEE Access 2024;12:16189-207.
27. Seshimo H, Rashed EA. Segmentation of Low-Grade Brain Tumors Using Mutual Attention Multimodal MRI. Sensors (Basel) 2024;24:7576. [Crossref] [PubMed]
28. Zhao J, Xing Z, Chen Z, Wan L, Han T, Fu H, Zhu L. Uncertainty-Aware Multi-Dimensional Mutual Learning for Brain and Brain Tumor Segmentation. IEEE J Biomed Health Inform 2023;27:4362-72. [Crossref] [PubMed]
29. Xue H, Yao Y, Teng Y. Multi-modal tumor segmentation methods based on deep learning: a narrative review. Quant Imaging Med Surg 2024;14:1122-40. [Crossref] [PubMed]
30. Manjunatha B, Manohar G, Anand LV, Manikandan M, Kavitha B, David DB. Automated brain tumor segmentation with deep learning. Latin Am Appl Res Int J 2025;55:309-14.
31. Thenmoezhi N, Perumal B, Lakshmi A. Multi-view image fusion using ensemble deep learning algorithm for MRI and CT images. ACM Trans Asian Low-Resour Lang Inf Process 2024;23:1-24.
32. Lu JC, Huang YH, Zhang YW, Ding H. MSA-net: An efficient attention-aware 3D network for brain tumor segmentation in MRI. In: Proceedings of the 3rd International Conference on Computer, Artificial Intelligence and Control Engineering; 2024:131-6.
33. Uppal D, Prakash S. Ms2adm-bts: Multi-scale dual attention-guided diffusion model for volumetric brain tumor segmentation. Pattern Recognit Lett 2025;198:115-22.
34. Dorent R, Joutard S, Modat M, Ourselin S, Vercauteren T. Hetero-modal variational encoder-decoder for joint modality completion and segmentation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. Cham: Springer International Publishing; 2019:74-82.
35. Sharma A, Hamarneh G. Missing MRI Pulse Sequence Synthesis Using Multi-Modal Generative Adversarial Network. IEEE Trans Med Imaging 2020;39:1170-83. [Crossref] [PubMed]
36. Yang H, Sun J, Xu Z. Learning Unified Hyper-Network for Multi-Modal MR Image Synthesis and Tumor Segmentation With Missing Modalities. IEEE Trans Med Imaging 2023;42:3678-89. [Crossref] [PubMed]
37. Meng X, Sun K, Xu J, He X, Shen D. Multi-Modal Modality-Masked Diffusion Network for Brain MRI Synthesis With Random Modality Missing. IEEE Trans Med Imaging 2024;43:2587-98. [Crossref] [PubMed]
38. Zhang Y, Peng C, Wang Q, Song D, Li K, Kevin Zhou S. Unified Multi-Modal Image Synthesis for Missing Modality Imputation. IEEE Trans Med Imaging 2025;44:4-18. [Crossref] [PubMed]
39. Havaei M, Guizard N, Chapados N, Bengio Y. Hemis: Hetero-modal image segmentation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. Cham: Springer International Publishing; 2016:469-77.
40. Chen C, Dou Q, Jin Y, Chen H, Qin J, Heng PA. Robust multimodal brain tumor segmentation via feature disentanglement and gated fusion. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. Cham: Springer International Publishing; 2019:447-56.
41. Ding Y, Yu X, Yang Y. RFNet: Region-aware fusion network for incomplete multi-modal brain tumor segmentation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision; 2021:3975-84.
42. Azad R, Khosravi N, Merhof D. SMU-net: Style matching U-net for brain tumor segmentation with missing modalities. In: International Conference on Medical Imaging with Deep Learning. PMLR; 2022:48-62.
43. Qiu YS, Zhao ZY, Yao HD, Chen DL, Wang Z. Modal-aware visual prompting for incomplete multi-modal brain tumor segmentation. In: Proceedings of the 31st ACM International Conference on Multimedia; 2023:3228-39.
44. Liu Y, Tian Y, Zhao Y, Yu H, Xie L, Wang Y, Ye Q, Jiao J, Liu Y. VMamba: Visual state space model. Adv Neural Inf Process Syst 2024;37:103031-63.
45. Sohn JH, Chillakuru YR, Lee S, Lee AY, Kelil T, Hess CP, Seo Y, Vu T, Joe BN. An Open-Source, Vender Agnostic Hardware and Software Pipeline for Integration of Artificial Intelligence in Radiology Workflow. J Digit Imaging 2020;33:1041-6. [Crossref] [PubMed]
46. Blezek DJ, Olson-Williams L, Missert A, Korfiatis P. AI Integration in the Clinical Workflow. J Digit Imaging 2021;34:1435-46. [Crossref] [PubMed]
47. Canguit JA, Sison M. Diagnostic efficiency, challenges and job satisfaction of radiologic technologists in picture archiving and communication system (PACS). Int J Acad Res Prog Educ Dev 2025;14:2393-408.
48. Zhang L, LaBelle W, Unberath M, Chen H, Hu J, Li G, Dreizin D. A vendor-agnostic, PACS integrated, and DICOM-compatible software-server pipeline for testing segmentation algorithms within the clinical radiology workflow. Front Med (Lausanne) 2023;10:1241570. [Crossref] [PubMed]
49. Albalawi E, Mahesh TR, Thakur A, Kumar VV, Gupta M, Khan SB, Almusharraf A. Integrated approach of federated learning with transfer learning for classification and diagnosis of brain tumor. BMC Med Imaging 2024;24:110. Erratum in: BMC Med Imaging 2024;24:161. [Crossref] [PubMed]
50. Zhang W, Jin W, Rho S, Jiang F, Yang C. A federated learning framework for brain tumor segmentation without sharing patient data. Int J Imaging Syst Technol 2024;34:e23147.
51. Joshi M, Pal A, Sankarasubbu M. Federated learning for healthcare domain-pipeline, applications and challenges. ACM Trans Comput Healthc 2022;3:1-36.
52. Mazher M, Razzak I, Qayyum A, Tanveer M, Beier S, Khan T, Niederer SA. Self-supervised spatial-temporal transformer fusion-based federated framework for 4D cardiovascular image segmentation. Inf Fusion 2024;106:102256.
53. Wang B, Li H, Guo Y, Wang J. PPFLHE: A privacy-preserving federated learning scheme with homomorphic encryption for healthcare data. Appl Soft Comput 2023;146:110677.