Parotid gland tumor computed tomography image segmentation and benign-malignant differentiation diagnosis

Introduction

The parotid gland is the largest salivary gland in the human body, located superficially anterior and inferior to the bilateral external auditory canals. Parotid gland tumors are the most common type of salivary gland neoplasms, accounting for 80% of all salivary gland tumors (1,2), and predominantly occur in individuals aged 30–50 years (1,2). Approximately 80% of parotid tumors arise in the superficial lobe of the gland (3). The primary clinical manifestations include swelling and abnormal masses anterior to the tragus or inferior to the earlobe. Benign tumors account for approximately 75% of cases, significantly outnumbering malignant lesions. Although most patients have a favorable prognosis after standardized treatment, postoperative recurrence remains a concern (1,4). Currently, the clinical diagnosis and treatment strategy for parotid gland tumors relies on a comprehensive assessment integrating the patient’s medical history, physical examination findings, and imaging results (5). Early and accurate determination of tumor nature is crucial for improving treatment efficacy and prognosis (6). Among various imaging modalities, non-contrast computed tomography (CT) has become an essential tool for clinically differentiating the nature of parotid gland tumors due to its advantages of simplicity, rapidity, and non-invasiveness (7,8).

In recent years, radiomics research based on cross-sectional imaging [including enhanced CT and magnetic resonance imaging (MRI)] has focused on the differential diagnosis of parotid gland tumors, which has yielded some results. For example, relevant studies have shown that MRI radiomics analysis can provide valuable auxiliary information for preoperative benign and malignant differentiation of parotid gland tumors, and its diagnostic efficiency is better than conventional visual evaluation (9). However, a common limitation of current radiological methods is the substantial overlap of imaging features between various malignant subtypes and some benign and malignant tumors, which usually hinders diagnostic specificity and limits clinical generalizability. These challenges highlight the need for more advanced feature extraction and fusion strategies, such as those based on deep learning (DL) and multimodal integration.

DL, a significant branch of machine learning (ML) (10), with core technologies such as convolutional neural networks (CNN) (11) and the Transformer architecture (12), has achieved notable success in medical image analysis for conditions such as thyroid cancer and breast cancer (13,14). Although existing research has demonstrated the potential of multimodal DL systems in the identification of various tumors (15), studies specifically focusing on differentiating benign from malignant parotid gland tumors using non-contrast CT images combined with patients’ clinical text information remain relatively scarce.

This study aimed to propose a method that leverages DL technology for precise segmentation of parotid gland tumors on non-contrast CT images and employed cross-modal fusion of textual information for the differential diagnosis of benign and malignant tumors, thereby providing new technical support for clinical decision-making. We present this article in accordance with the STARD-AI reporting checklist (available at https://qims.amegroups.com/article/view/10.21037/qims-2026-0691/rc).

Methods

Design and implementation of PT-TransUNet for image segmentation

Baseline architecture: TransUNet

On CT imaging, parotid gland tumors typically display relatively high contrast against the surrounding normal parotid tissue (16). The inherently low density of the parotid gland makes precise regional localization essential for accurate tumor identification (17). However, the density of parotid tumors is similar to that of muscle tissue. Consequently, when a tumor abuts adjacent muscles such as the masseter or sternocleidomastoid, or when a malignant tumor invades surrounding structures, the boundaries become particularly difficult to delineate (18). In addition, if a parotid tumor is complicated by secondary infection, the interface between the tumor and normal tissue may also become indistinct (19), further complicating image interpretation.

Given these practical challenges, we selected TransUNet (20) as the foundational model to achieve precise segmentation of parotid tumors in non-contrast-enhanced CT. TransUNet, a hybrid architecture combining Transformer and U-Net, introduced in 2021, is specifically designed for medical image segmentation. By integrating Transformer with the conventional U-Net, it achieves excellent performance in medical image segmentation tasks (21).

Overview of the proposed PT-TransUNet

We propose the PT-TransUNet model. Its implementation incorporates a learnable Difference of Gaussians (DoG) edge enhancement module (22,23) at the input stage of the standard TransUNet and introduces a PSA-CBAM attention module—which integrates channel and spatial attention via a Pyramid Split Attention mechanism—into the decoder.

DoG edge enhancement module

The specific operational procedure is as follows: CT images are input into the model at a size of 224×224 pixels. After undergoing simple data augmentation, they are fed into a learnable DoG edge enhancement module. The principle of this module is to highlight edge information in the image based on the convolution difference between two Gaussian kernels of different scales. The mathematical formulation of the Gaussian function and the DoG response are provided in Appendix 1. We improved the traditional DoG by replacing the fixed parameters with learnable parameters that can be optimized via backpropagation. This enables the model to adaptively optimize the scale for edge detection, thereby automatically adapting to the specific edge characteristics of parotid gland tumors and the varying noise levels across different images during the training process. The architecture of the learnable DoG edge filter module is shown in Figure 1.

Figure 1 Working principle of the DoG module. DoG, Difference of Gaussians.

The DoG edge filter is effective for image enhancement of cystic lesions. However, parotid gland tumors are mostly solid or mixed cystic–solid. Visual inspection reveals that although the filter enhances the edge features of the tumor, it homogenizes the internal content of the tumor, reducing contrast with the parotid gland and surrounding tissues. As a result, it fails to highlight differences in tissue density within the tumor. With respect to the parotid tissue, the glandular capsule is enhanced, but small structures such as blood vessels and lymph nodes within the parotid tissue are weakened, leading to a partial loss of characteristic features of the parotid tissue. The comparison of CT images processed by the DoG module is shown in Figure 2.

Figure 2 Visualization results of the DoG module. It is evident that while the edge features of various tissues are enhanced, some characteristics of the parotid gland tissue and parotid gland tumors are also obscured. Red areas indicate parotid gland tumor regions. Green areas indicate normal parotid gland tissue regions. DoG, Difference of Gaussians.

Therefore, a weighted combination of the DoG-processed features and the original image was designed to preserve the characteristics of both the parotid gland tissue and parotid tumors as much as possible. The final output is the weighted sum of the original image and the DoG response (see Appendix 1 for the formula), where the fusion weight is learnable.

The image processed by the DoG edge filter module is replicated into three channels, divided into patches, and supplemented with learnable positional embeddings before being fed into the Transformer encoder. The output is a feature sequence processed through multiple blocks. Subsequently, the decoder performs progressive upsampling and fuses features from skip connections, while applying the PSA-CBAM module to capture richer contextual information and highlight important features.

PSA-CBAM attention module

The PSA-CBAM attention module is a hybrid attention mechanism that integrates multi-scale convolution (Pyramid Split) (24) and the Convolutional Block Attention Module (CBAM) (25). This module first splits the input feature map along the channel dimension into three subsets. Feature extraction is then performed using convolutional layers with kernel sizes of 3×3, 5×5, and 7×7. Subsequently, the extracted features are concatenated. This multi-scale fused feature is then sequentially passed through the channel attention sub-module of CBAM (global average pooling, two fully connected layers, and an activation function) to generate channel weight vectors, followed by the spatial attention sub-module (average pooling and max pooling along the channel dimension, concatenation, and convolution). Finally, the processed features are fused with the original multi-scale features via a residual connection. The detailed formulas for these operations are provided in Appendix 1. This design enables the decoder to simultaneously focus on multi-scale detailed information and critical spatial regions, effectively improving the accuracy of segmentation boundaries. The working principle of the PSA-CBAM module is illustrated in Figure 3.

Figure 3 Working principle of the PSA-CBAM module. CBAM, convolutional block attention module; PSA, pyramid split attention.

The output of the final decoder block is mapped to the number of classes via a segmentation head, and the segmentation map is generated.

Design and implementation of multimodal diagnostic model

Branches of multimodal diagnosis model

Based on the previously described PT-TransUNet, we propose a multimodal diagnostic scheme that extracts and fuses features from both CT images of parotid tumors and clinical text data—including the patient’s chief complaint, history of present illness, personal history, and physical examination findings—to determine whether the tumor is benign or malignant. For the image branch, we use the complete PT-TransUNet model as an image encoder. For the text branch, we introduce the Chinese BERT Whole Word Masking (WWM) model (26), a pre-trained language model optimized for Chinese natural language processing tasks. Proposed in 2019 by a research team from Harbin Institute of Technology and iFLYTEK, this model incorporates a whole-word masking strategy into the original BERT framework, significantly enhancing its performance in Chinese text comprehension (27).

Image and text features are extracted independently by their respective dedicated branches. In the image branch, PT-TransUNet outputs a scalar logit as a compact representation of the CT image. This design preserves the architectural improvements of PT-TransUNet and enables the discriminative cues learned from segmentation to contribute to multimodal diagnosis. The scalar is then projected into a 768-dimensional common semantic space via a learnable linear layer. The text modality features are obtained via the pre-trained BERT model, which outputs a 768-dimensional representation derived from the [CLS] token. In the BERT architecture, [CLS] (classification token) is a special token prepended to the input sequence, and its final hidden state serves as a sentence-level representation for classification tasks. These two modality features are then fused using a cross-modal attention mechanism and finally fed into a classifier to yield a benign or malignant diagnosis.

Fusion strategy

In the cross-modal attention mechanism, a bidirectional cross-attention design is employed (28). Image features serve as the Query, with text features as the Key-Value; conversely, text features serve as the Query, with image features as the Key-Value, and cross-attention is computed separately for each direction. The outputs from both attention directions are concatenated to form a 1,536-dimensional fused feature. This feature is subsequently processed by a classification layer to perform binary classification for benign or malignant tumors, thereby achieving deep semantic interaction and fusion between image and text information. The principle of the cross-modal attention mechanism is illustrated in Figure 4.

Figure 4 Schematic diagram of the cross-modal attention mechanism.

Data acquisition and preprocessing

CT image data

CT imaging data and medical text information were collected from 110 patients with parotid gland tumors at a tertiary hospital in Harbin, China. These data comprised 89 cases of benign tumors and 21 cases of malignant tumors. All CT examinations were performed on a Philips iCT256 scanner (Philips Healthcare, Best, Netherlands) in non-contrast, plain-scan mode with automatic tube current modulation (DoseRight, Philips Healthcare). The tube current was modulated in real time based on the patient’s anatomy to maintain a predefined image quality level. As a result, the tube current varied automatically from 90 to 300 mA, and the corresponding tube current–time product (exposure) ranged from 90 to 155 mAs. The tube voltage was fixed at 120 kVp. Slice thickness ranged from 0.5 to 2 mm, with reconstruction intervals matching the slice thickness. All images were reconstructed with a standard soft-tissue kernel (Filter B) combined with iDose iterative reconstruction (level 3). The resulting Digital Imaging and Communications in Medicine (DICOM) images had a matrix size of 512×512 pixels. For visualization and preprocessing, the window level was set to 50 Hounsfield units (HU) and the window width to 350 HU. Segmentation labels for parotid gland tissue and parotid gland tumors were manually delineated by two senior radiologists using the ITK-Snap software (https://www.itksnap.org/pmwiki/pmwiki.php). To assess inter-observer variability, the two radiologists independently segmented a random subset of 10 cases, yielding mean Dice coefficients of 0.875±0.042 for tumors and 0.832±0.058 for the parotid gland, indicating good agreement.

Public dataset

Additionally, the Public Domain Database for Computational Anatomy (PDDCA) dataset (29) was utilized. This is a publicly available head and neck CT imaging dataset primarily used for research on automatic segmentation of organs at risk in radiotherapy planning. It contains head CT scan data from 48 patients, which includes segmentation labels for the parotid glands. This dataset was incorporated into the training and testing of the PT-TransUNet model solely as tumor-free cases to enhance the model’s ability to recognize parotid gland tissue.

Clinical text data

All medical text information was collected from patient medical records, including the patient’s chief complaint, present illness history, personal history, smoking and alcohol history, and clinical examination findings. All content was written in simplified Chinese. The narrative sequence, wording, and character count of the text content were standardized to improve training efficiency.

All the aforementioned data underwent de-identification to protect patient privacy.

Data splitting for segmentation and diagnosis tasks

Prior to the experiments, a total of 158 CT cases—comprising 89 cases of benign parotid tumors, 21 cases of malignant tumors, and 48 cases without tumors—were stratified and split into a training set and a test set in an 8:2 ratio. Specifically, the test set consisted of 18 benign tumor cases, 4 malignant tumor cases, and 10 non-tumor cases, whereas the remaining data formed the training set. This split was used for evaluating the image segmentation performance of PT-TransUNet.

For the experiments involving the multimodal diagnostic model, only the CT image data and corresponding medical text information from 110 parotid tumor patients were utilized. These data were also stratified and split into a training set and a test set in an 8:2 ratio. Here, the test set consisted of 18 benign tumor cases and 4 malignant tumor cases, with the remaining data forming the training set. The test set was used solely for final evaluation and did not participate in any training or validation steps. The dataset was split before any model training. The test set labels were not revealed to the model developers until the final evaluation.

Inclusion and exclusion criteria

In September 2025, we randomly selected 148 patients with pathologically confirmed parotid gland tumors from the database of a tertiary hospital in Harbin, spanning the previous three years. After applying the inclusion and exclusion criteria, 110 patients were included in the final multimodal analysis. Additionally, 48 tumor-free cases from the PDDCA public dataset were included for the segmentation task, as shown in Figure 5.

Figure 5 Flowchart of patient enrollment, exclusion, and group assignment. CT, computed tomography; PDDCA, Public Domain Database for Computational Anatomy.

The inclusion criteria for this study were as follows: (I) patients with histopathologically confirmed parotid gland tumors (benign or malignant) based on postoperative or biopsy pathology; (II) patients who underwent non-contrast CT examination of the parotid region prior to any treatment; (III) availability of complete clinical text records, including chief complaint, history of present illness, personal history, and physical examination findings; and (IV) CT images with clear visualization of the parotid region and absence of severe motion or metal artifacts.

The exclusion criteria were as follows: (I) patients with recurrent parotid tumors who had received prior surgical or radiotherapy treatment; (II) patients with a history of other head and neck malignancies; (III) CT images with significant artifacts that prevented reliable manual segmentation; (IV) incomplete or missing clinical text data; and (V) patients with parotid region lesions that were not pathologically confirmed as primary parotid gland tumors.

The sample size was determined by the availability of eligible cases meeting the inclusion criteria during the study period. No formal power calculation was performed, as this is an exploratory study.

Ethical considerations

This study was retrospective in nature. The study was conducted from September 2025 to March 2026. All data were collected from existing clinical records and imaging databases, without any prospective intervention or patient contact. The study was conducted in accordance with the Declaration of Helsinki and its subsequent amendments. The study was approved by the Ethics Committee of The First Affiliated Hospital of Harbin Medical University (No. 2025365). The requirement for written informed consent was waived by the Ethics Committee due to the retrospective design and the use of de-identified data.

Experimental design and evaluation methods for image segmentation

Ablation and comparative study setup

To evaluate the segmentation performance of PT-TransUNet, an ablation study was conducted to compare the performance of the traditional TransUNet model, TransUNet + DoG, TransUNet + CBAM, and TransUNet + CBAM + DoG in segmenting benign parotid tumors, malignant parotid tumors, and non-tumor parotid tissue. Concurrently, a comparative experiment was performed using five well-established and high-performing models in medical image segmentation, including 3DUNet (30), ResUNet (31), nnUNet (32), SwinUNet (33), and AttentionUNet (34).

Training and validation strategy

In all experimental groups, a five-fold cross-validation approach was adopted to ensure model accuracy and generalizability. Specifically, the 126 cases of data, excluding the test set, were evenly divided into five folds. Four folds were used as the training set, and one fold served as the validation set, with the process repeated in rotation. The final test results were obtained by averaging the outcomes from the five model tests. All imaging data underwent augmentation via random flipping and random rotation to enhance model robustness. For the training strategy, the SGD optimizer, mixed precision, and gradient accumulation techniques were employed. Training was conducted based on models pre-trained on ImageNet (35). An early stopping strategy (patience =5) was applied to all model groups.

Evaluation metrics

We evaluated performance using the Dice similarity coefficient (DSC), 95% Hausdorff distance (HD95), recall, and precision. The DSC quantifies the spatial overlap between the predicted segmentation and the ground truth, where a higher value indicates greater volumetric agreement. HD95 assesses boundary accuracy by computing the 95th percentile of distances between the predicted and true contours, with a smaller value denoting superior boundary alignment. Recall measures the model’s sensitivity in detecting the target region, whereas precision indicates the specificity of its predictions. The specific calculation formulas for these metrics are provided in Appendix 2.

Experimental design and evaluation methods for multimodal diagnosis

Ablation and comparative study setup

To assess the performance of the multimodal diagnostic model for distinguishing benign from malignant parotid gland tumors—constructed using PT-TransUNet and a Chinese BERT model—we performed an ablation study. The model was evaluated under three conditions: using medical text alone, using CT imaging alone, and using fused multimodal data to predict tumor malignancy. We also compared four alternative fusion methods to analyze the effect of the bidirectional cross-attention mechanism on diagnostic performance. These comprised two feature-level strategies, Concatenation Linear (36) and Gated Fusion (37), and two decision-level strategies, Learned Weight (38) and Weighted Average (39).

Training and validation strategies

All five models were tested under identical experimental conditions: the AdamW optimizer and a cosine annealing scheduler were used for training. To balance efficiency and stability, we initialized the model with weights pre-trained on ImageNet and the Chinese BERT model, respectively. Five-fold cross-validation and an early stopping strategy (patience =3) were also employed to ensure an accurate evaluation of model performance. A default probability cutoff of 0.5 was used to classify tumors as malignant (≥0.5) or benign (<0.5).

Evaluation metrics

We evaluated classification performance using accuracy, precision, recall (sensitivity), the F1 score and specificity. Accuracy represents the overall proportion of correct classifications, calculated as the ratio of the sum of true positives and true negatives to the total number of cases. Precision quantifies the purity of positive predictions, defined as the proportion of true positives among all samples predicted as positive. Recall measures the model’s ability to identify positive instances, expressed as the proportion of actual positives that are correctly detected. The F1 score, the harmonic mean of precision and recall, provides a balanced assessment that mitigates the bias of either metric alone. Specificity evaluates the model’s capacity to identify negative instances, calculated as the proportion of true negatives among all actual negatives. The specific calculation formulas for accuracy, F1 score, and specificity, as well as the definitions of true positives, false positives, true negatives, and false negatives, are provided in Appendix 2.

All experiments were conducted on a single NVIDIA RTX 4060 GPU. All models were implemented using PyTorch (version 1.13.1).

Statistical analysis

Paired two-tailed t-tests were used to compare model performances across the five cross-validation folds. Significance levels are indicated as *, P<0.05; **, P<0.01; and ***, P<0.001 in all tables. To control the family-wise error rate due to multiple comparisons, we further performed one-way analysis of variance (ANOVA) followed by Tukey’s HSD post-hoc test for each set of comparisons. The results confirmed that the significant differences reported in the tables (based on t-tests) remained statistically significant after correction for multiple comparisons, except where noted. The paired design minimizes the influence of fold-to-fold variability, and the consistently small standard deviations ensure adequate statistical power even for modest mean differences.

Results

Impact of key components in PT-TransUNet on segmentation performance

A series of ablation experiments were conducted to analyze the roles of specific model components. In these studies, only the key modules—the learnable DoG edge filter and the PSA-CBAM attention mechanism in the decoder—were modified, whereas all other settings remained fixed. The results are presented in Tables 1-3.

Given the local invasion and infiltration characteristic of malignant tumors, the evaluation was conducted across three categories: benign tumors, malignant tumors, and non-tumor cases.

The improved models incorporating the PSA-CBAM module, the DoG module, or both achieved superior Dice coefficients across all three test datasets compared to the baseline, with all improvements being statistically significant (P<0.05 or P<0.01). The model integrating both modules achieved the best overall performance. For benign tumor segmentation, its Dice scores for the parotid gland and tumor were 0.8212±0.0269 and 0.8075±0.0245, respectively. For malignant tumor targets, the corresponding Dice scores were 0.8020±0.0203 and 0.7676±0.0291. Even on the PDDCA dataset, where all methods exhibited performance degradation, this model achieved a Dice coefficient of 0.6434±0.0232. Both the TransUNet + PSA-CBAM and TransUNet + DoG combinations also outperformed the baseline TransUNet. The PSA-CBAM module contributed a greater improvement in the Dice coefficient than the DoG module, likely due to its enhanced capability for image feature perception.

Other metrics further substantiated the performance gains from the PSA-CBAM module. For benign tumor segmentation, the mean recall increased from 0.6819 to 0.7398, whereas for malignant tumors, it improved from 0.6775 to 0.735. These increases indicate a significant reduction in missed detections and more complete tumor coverage. In contrast, the DoG module primarily refined segmentation boundaries, as evidenced by a substantial reduction in the HD95. For the challenging malignant tumor task, it reduced the mean HD95 from 10.95 mm to 6.92 mm, demonstrating superior boundary precision.

Performance comparison of different image segmentation methods

To evaluate the effectiveness of the proposed model, its performance was compared with that of several other models. The experimental results are presented in Tables 4-6.

Systematic comparisons with five mainstream segmentation models fully validated the superiority of the proposed method. Across three distinct segmentation tasks—benign tumors, malignant tumors, and non-tumor cases—PT-TransUNet achieved favorable Dice coefficients, with differences that were statistically significant (P<0.05 or P<0.01) compared to all benchmark models (3DUNet, ResUNet, nnUNet, SwinUNet, AttentionUNet). On the benign tumor dataset, PT-TransUNet achieved Dice scores of 0.8212±0.0269 for the parotid gland and 0.8075±0.0245 for benign tumors, significantly outperforming the other methods, including nnUNet. On the more challenging malignant tumor and PDDCA dataset, it attained Dice scores of 0.8020±0.0203 for the parotid gland and 0.7676±0.0291 for malignant tumors, whereas for tumor-free parotid gland segmentation, it achieved a Dice score of 0.6434±0.0232—all reflecting excellent performance. Moreover, PT-TransUNet consistently exhibited small standard deviations, indicating low result variability and strong robustness. Although advanced methods such as nnUNet performed well on certain tasks, PT-TransUNet demonstrated overall superiority across comprehensive metrics, confirming the effectiveness and advancement of the proposed approach and highlighting its clinical potential.

Examination of additional metrics further confirmed the segmentation advantages of PT-TransUNet. Although the traditional 3DUNet captures spatial context through 3D convolutions, its sensitivity to fine tumor boundaries is limited. This is particularly evident in malignant parotid tumor segmentation, where its mean HD95 reached 9.88 mm—substantially higher than that of nnUNet (8.96 mm) and the proposed method (6.28 mm). ResUNet, which employs a relatively simple skip-connection mechanism, exhibits inefficient multi-scale feature fusion, resulting in mid-to-lower-level performance across all three test sets. Among the six models, nnUNet demonstrated the most robust performance overall, benefiting from its automated configuration and powerful data preprocessing. In benign tumor segmentation, its mean precision (0.8162) was second only to the proposed method, and its HD95 (16.75 mm) was also favorable. However, for the more challenging malignant tumor segmentation task, the proposed method significantly outperformed nnUNet in both Dice (0.7676 vs. 0.7333) and HD95 (6.28 vs. 8.96 mm), indicating stronger segmentation and boundary-fitting capabilities when handling irregular, boundary-ambiguous malignant tumors. SwinUNet, based on the Swin Transformer, achieved a recall of only 0.6963±0.0584 in malignant tumor segmentation, suggesting inadequate capture of infiltrative tumor boundaries. AttentionUNet lagged behind across all metrics on the three test sets, likely because its single-scale attention mechanism struggles to simultaneously address the large-scale tissue distribution of the parotid gland and the fine structural details of tumors.

Additionally, we conducted a visual analysis of the prediction results from the aforementioned models (Figure 6). The first three rows correspond to cases of benign tumors, whereas the last two rows show malignant tumor cases. In the first row of comparative images, artifacts can be observed in the tumor tissue within the deep lobe of the parotid gland, indicating suboptimal imaging quality. With the exception of PT-TransUNet, 3DUNet, and nnUNet, the remaining models experienced varying degrees of interference. In the second row, all models demonstrated limitations in predicting the nodular protrusions on the tumor surface in parotid tumors exhibiting a “nodular” growth pattern. However, PT-TransUNet generally achieved greater accuracy in delineating tumor boundaries. As shown in the third to fifth rows, regardless of whether the tumor was benign or malignant, when the tumor was located at the posterior pole of the parotid gland and adjacent to the anterior margin of the sternocleidomastoid muscle, predictions of the posterior aspect near the muscle tissue were compromised due to the similar density of surrounding soft tissues. Overall, PT-TransUNet, 3DUNet, and nnUNet exhibited greater precision in handling edge details, with PT-TransUNet’s predictions most closely resembling the standard segmentation results.

Figure 6 Visual analysis of comparative experiments for the segmentation model. Rows 1 to 3 correspond to benign tumor cases, while rows 4 and 5 correspond to malignant tumor cases. CT, computed tomography.

Comparison of diagnostic performance for benign and malignant lesions under different modality conditions

We conducted ablation experiments to analyze the differences in diagnostic accuracy of the model under single-modality versus multi-modality conditions. In this study, only the modality of the input information was modified while keeping all other settings fixed. The results of the ablation study are shown in Table 7.

A comparison of accuracy revealed the limited performance of unimodal approaches: the model using only clinical text achieved an accuracy of 0.7273±0.0321, whereas the model relying solely on CT images yielded a significantly lower accuracy of 0.6091±0.0249. Both were markedly outperformed by the multimodal model that integrated text and image information (0.8909±0.0249, P<0.001). These findings underscore the critical role of complementary multimodal information in enhancing tumor diagnostic performance and highlight the considerable clinical potential of the proposed model.

Evaluation of additional metrics further confirmed the substantial superiority of the multimodal model over its unimodal counterparts. Compared with using text alone, the multimodal model improved precision to 0.6510±0.0693 (an increase of approximately 0.3477), raised recall to 0.9000±0.1369 (an increase of 0.5), achieved an F1-score of 0.7488±0.0558 (an increase of 0.4055), and improved specificity to 0.8889±0.0393 (an increase of 0.0889). In contrast, the CT-only model produced notably poor results, with precision of 0.0905±0.0832, recall of 0.1500±0.1369, F1-score of 0.1127±0.1032, and specificity of 0.7111±0.0465, indicating its inadequacy for accurate diagnosis when used independently. Relative to the CT-only model, the multimodal model delivered substantial improvements across all metrics, raising precision to 0.6510±0.0693 (an improvement of approximately 0.5605), recall to 0.9 (an improvement of 0.75), F1-score to 0.7488 (an improvement of 0.6361), and specificity to 0.8889 (an improvement of 0.1778).

The differential diagnostic results under the three different modality conditions were visualized using confusion matrix heatmaps, as shown in Figure 7.

Figure 7 The confusion matrix heatmap of the five-fold ablation experiment for different modality diagnosis models. CT, computed tomography.

The confusion matrix heatmaps reveal that prediction accuracy for both benign and malignant tumors is lower under the single-modality conditions—medical text alone or CT images alone—than under the multimodal condition. Relying solely on CT images also results in a relatively high rate of underdiagnosis and misdiagnosis for malignant tumors. However, due to subjective bias in textual descriptions, the limited precision of CT for parotid gland tumors, and the subtle clinical presentation of early malignancies, the multimodal model’s accuracy for malignant tumors still requires improvement.

Impact of different multimodal fusion strategies on the performance of the multimodal diagnostic system

To evaluate the effects of different fusion methods on model performance, we compared our proposed fusion approach with two feature-level fusion strategies and two decision-level fusion strategies. The experimental results are presented in Table 8.

Comparative experiments revealed significant performance differences among the five fusion strategies, reflecting how their distinct mechanisms interact features and integrate decisions. The feature-level fusion method employing a bidirectional cross-modal attention mechanism (Cross Attention) achieved the highest accuracy (0.8909±0.0249) and demonstrated superior stability and precision. Another feature-level approach, which used concatenation followed by a linear layer (Concat Linear), achieved an accuracy of 0.8455±0.0518. Although this difference was not statistically significant (P>0.05), the larger standard deviation indicated greater variability in the results. In contrast, Gated Fusion produced a significantly lower accuracy of 0.6818±0.0321 (P<0.001), suggesting that its mechanism was ineffective for integrating these multimodal features. Among the decision-level methods, both Learned Weight (0.7727±0.0321) and Weighted Average (0.7545±0.0249) achieved accuracies significantly below that of Cross Attention (P<0.001), indicating that simple combinations at the decision level cannot fully exploit the deep complementary information between modalities.

Regarding other metrics, the Concat Linear method achieved an F1-score of 0.6691±0.1216, the closest among the four models to the Cross Attention score of 0.7488±0.0558. However, its recall (0.8500±0.1369) was slightly lower than that of Cross Attention (0.9000±0.1369), implying that simple concatenation may inadequately fuse highly heterogeneous multimodal data. The performance of Gated Fusion was weak across all metrics, with an F1-score of only 0.3649±0.0236. Although its gating mechanism was designed to adaptively modulate the contribution of each modality, the model likely failed to effectively balance the interaction between image and text features, possibly due to unstable parameter learning or sensitivity to data distribution.

Among the decision-level methods, Learned Weight outperformed the fixed-weight Weighted Average, as reflected by its higher F1-score (0.5473±0.0354 vs. 0.5013±0.0967). This result indicates that learning dynamic weights can partially mitigate inter-modality prediction inconsistencies. Nevertheless, both methods underperformed relative to the feature-level fusion strategies, underscoring a key limitation of decision-level fusion: weighted integration at the prediction level cannot fully leverage complementary feature-level information. Consequently, the model’s capacity to capture fine-grained semantic associations is compromised, leading to suboptimal performance when complex correspondences exist between textual descriptions and image regions.

Overall, the experiments indicate that for fusing multimodal medical images and text, feature-level fusion methods generally exceed the performance of decision-level fusion. The cross-modal attention mechanism, owing to its capacity to model interactions between modalities, represents the most effective approach for predicting parotid gland tumor malignancy.

Discussion

Clinical value of PT-TransUNet for parotid segmentation

In clinical practice, non-contrast CT of the parotid gland is inferior to contrast-enhanced CT and MRI for soft tissue imaging. However, its advantages—fast imaging speed, absence of contrast agent-related risks, and low cost—make it the most commonly used imaging modality in oral and maxillofacial surgery, rendering it of considerable research value.

In this study, we applied PT-TransUNet to segment parotid gland tissue on non-contrast CT images, achieving a Dice coefficient of 0.8212±0.0269. This result demonstrates that our approach effectively mitigates the difficulty of accurately localizing the parotid region caused by the low density of the parotid gland, thereby laying a foundation for further imaging segmentation of parotid tumors. For benign parotid tumors, the model achieved a Dice coefficient of 0.8075±0.0245 and an HD95 of 16.14±0.58 mm; for malignant parotid tumors, it achieved a Dice coefficient of 0.7676±0.0291 and an HD95 of 6.28±0.58 mm. These results indicate that the proposed method attains high segmentation accuracy and edge sensitivity for target tumors. Visualization of the segmented targets enables clinicians to gain a more intuitive understanding of the anatomical relationship between the tumor and adjacent soft tissues, thereby assisting in the design of better surgical approaches and reducing injury to surrounding tissues.

Advantage of multimodal fusion and comparison with human experts

Furthermore, due to the deep-seated anatomical location of parotid tumors, preoperative biopsy is generally not performed to determine tumor nature, in order to prevent seeding and spread of malignant tumors. Consequently, the basis for preoperative differentiation between benign and malignant tumors is relatively limited, which adversely affects surgical planning. By fusing non-contrast CT images with clinical text information, this study achieved an accuracy of 0.8909±0.0249 and a recall of 0.9000±0.1369, demonstrating the profound application potential of this strategy for the differential diagnosis of benign and malignant tumors. We hope that this study can assist clinicians in preoperative diagnosis and clinical decision-making, particularly for primary medical institutions lacking experienced radiologists or oral and maxillofacial surgeons, by enabling more rational allocation of medical resources and improving diagnostic and treatment efficiency.

In addition to the quantitative results presented above, it is essential to evaluate the potential clinical utility of the proposed model by comparing its performance with that of human experts in a real-world hospital setting. To this end, we retrospectively analyzed the initial diagnostic records of the 110 patients with parotid gland tumors included in this study. Notably, among the 21 pathologically confirmed malignant cases, only 5 (23.8%) were unequivocally diagnosed as malignant at the initial clinical presentation. The majority of cases were conservatively recorded as “parotid tumor” or “parotid region tumor”, reflecting the inherent difficulty of preoperative differentiation based on non-contrast CT and routine clinical information alone. Such conservative diagnostic labels lack sufficient statistical significance for direct comparison with model outputs.

We therefore invited the two senior radiologists who participated in the manual segmentation of the dataset—each with more than 10 years of experience in head and neck imaging—to independently re-evaluate the 110 cases using the same non-contrast CT images and clinical text information available to the model. Their diagnostic accuracies for benign-malignant differentiation were 79.09% and 82.73%, respectively. In comparison, our proposed multimodal model achieved a significantly higher accuracy of 89.09%. Furthermore, previous studies have shown that radiologists using contrast-enhanced CT achieve an average diagnostic accuracy of approximately 81.71% for parotid gland tumors (40). Given that non-contrast CT provides inferior soft-tissue contrast compared with contrast-enhanced CT, the real-world accuracy of radiologists in our specific clinical setting is likely to be lower than 81.71%. Thus, although a direct head-to-head comparison with a human expert remains challenging due to the retrospective nature of this study and the class imbalance, our results suggest that the proposed model may offer a level of diagnostic performance that is at least comparable to, and potentially better than, that of experienced radiologists working with non-contrast CT. This finding, together with the segmentation and multimodal fusion results discussed above, supports the potential of the proposed system as a valuable auxiliary tool for preoperative diagnosis of parotid gland tumors, especially in settings where access to experienced radiologists or contrast-enhanced imaging is limited.

Limitations

Several limitations of this study warrant consideration. First, although data augmentation and five-fold cross-validation were used to improve training reliability, the dataset remained limited to 158 CT cases, which may result in insufficient training or overfitting. The small number of malignant tumors (n=21) results in a class imbalance that could compromise the model’s generalizability. Furthermore, all tumor data originated from a single center, so their representativeness must be validated with larger, multi-center datasets. Second, although the reliability of the tumor-free PDDCA dataset we used is widely recognized, we observed significantly lower performance on this dataset during our experiments. After conducting visualization analysis and consulting with imaging professionals, we attribute this issue to the fact that the field of view (FOV) in the PDDCA dataset is larger than that in the CT images we collected. As a result, the volume proportion of parotid tissue within the images is smaller, which leads to poorer recognition and segmentation performance. A comparison of the FOV between our parotid tumor CT images and the PDDCA dataset is shown in Figure 8.

Figure 8 Comparison of the FOV between the parotid gland tumor CT images and the tumor-free (PDDCA) dataset. Red areas indicate parotid gland tumor regions. Green areas indicate normal parotid gland tissue regions. CT, computed tomography; FOV, field of view; PDDCA, Public Domain Database for Computational Anatomy.

To maintain consistency in input image size, we did not apply additional scaling to the images. Although the final experimental results still demonstrate the performance advantages of PT-TransUNet, we will pay closer attention to the uniformity of CT scanning FOV in future experiments to ensure effective model training. Third, the medical text information within the multimodal framework was extracted from clinical records; despite standardization, such text inherently contains subjective descriptions and diagnostic biases from physicians, which could affect dataset objectivity and model judgments. Future studies should investigate more standardized text feature extraction or quantitative diagnostic criteria as supplementary information. Finally, the inherent imaging limitations of non-contrast CT, particularly relative to contrast-enhanced CT and MRI, remain a constraint, though its clinical applicability and non-invasive nature continue to motivate exploration of its diagnostic utility.

Future directions

Future work will focus on three directions. First, due to the current limitation of sample size, the proposed PT-TransUNet still requires further learning and optimization. We need to build a training dataset with a more balanced number of benign and malignant cases. We plan to expand the dataset with multi-center, large-scale samples to improve model robustness and stability before clinical deployment. Second, we will conduct prospective validation studies in real clinical settings. Specifically, the preoperative benign-malignant predictions generated by the multimodal model will be systematically compared with postoperative pathological results to quantitatively verify the reliability and clinical utility of the proposed method. Only when the model demonstrates stable and satisfactory performance in these validation studies will it be considered for routine clinical practice as an auxiliary tool for preoperative diagnosis of parotid gland tumors. Third, we will continue to optimize the multimodal fusion strategy to reduce subjectivity in text feature extraction and further improve diagnostic accuracy. Interpretability technology will be implemented and verified on the basis of improving model performance and expanding data samples.

Conclusions

The proposed PT-TransUNet model, which incorporates a learnable DoG edge enhancement module and a PSA-CBAM attention mechanism, achieved Dice coefficients of 0.8212±0.0269 for parotid gland segmentation and 0.8075±0.0245 for benign tumor segmentation, outperforming traditional methods such as 3DUNet and nnUNet. These results indicate that the architecture is effective for medical image segmentation of parotid structures. Furthermore, a multimodal diagnostic model, constructed by integrating PT-TransUNet with a Chinese BERT model via a cross-modal attention mechanism, achieved a diagnostic accuracy of 0.8909±0.0249 and a sensitivity of 0.9000±0.1369. These results demonstrate that combining radiological characteristics with clinical textual information provides a more comprehensive basis for discriminating between benign and malignant parotid gland tumors than using either modality alone.

Acknowledgments

During the preparation of this work, the authors used DeepSeek and Mita AI in order to conduct searches and read relevant literature, and refine certain parts of the language in the articles. After using these tools, the authors reviewed and edited the content as needed and take full responsibility for the content of the published article.

Footnote

Reporting Checklist: The authors have completed the STARD-AI reporting checklist. Available at https://qims.amegroups.com/article/view/10.21037/qims-2026-0691/rc

Data Sharing Statement: Available at https://qims.amegroups.com/article/view/10.21037/qims-2026-0691/dss

Funding: This study was supported by the Horizontal Scientific Research Project of Harbin Medical University (No. 2023HX033), Innovation Training Project for Students of Harbin Medical University (No. 202510226058) and Innovation Training Project for Students of Harbin Medical University (No. 202410226069).

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at https://qims.amegroups.com/article/view/10.21037/qims-2026-0691/coif). All authors report that this study received support from the following projects: Horizontal Scientific Research Project of Harbin Medical University (No. 2023HX033, principal investigator: S.P.), and Innovation Training Projects for Students of Harbin Medical University (No. 202510226058, principal investigator: S.P. and T.F.; and No. 202410226069, principal investigator: S.P. and K.C.). All authors report that the core algorithmic framework was provided by a pending patent (application No. 202511776096.5, inventors: N.S. and S.P.). The authors have no other conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. The study was conducted in accordance with the Declaration of Helsinki and its subsequent amendments. The study was approved by the Ethics Committee of The First Affiliated Hospital of Harbin Medical University (No. 2025365). The requirement for written informed consent was waived by the ethics committee due to the retrospective design and the use of de-identified data.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. Kavali RBH, Nelluri N, Kishan M, Prasad K, Syamji A, Kumar KP. A Clinico-Pathological Study and Management of Parotid Tumours. SAS Journal of Surgery 2021;7:719-22.
2. Baghel A, Kushwaha M, Sagar A, Saxena P. Cytomorphological Features of Salivary Gland Lesions: Correlation with Histopathological Findings. International Journal of Laboratory and Biomedical Pathology Research 2024;13:343-8.
3. Stoev L, Dokova K, Yankov Y, Halil M, Stoeva M, Manev R. Ten Years of Parotid Gland Tumors: Regional Data from Bulgaria. Galician Medical Journal 2025;32:e-GMJ2025-A29.
4. Kornevs E, Tars J, Bigestans A, Lauskis G. Treatment of parotid gland tumors in Latvian Oncological Center. Stomatologija 2005;7:110-4.
5. Vitorino M, Tinoco J, Chaves AF. Basal Cell Adenocarcinoma Arising from the Parotid Gland. Biomed Hub 2022;7:173-8. [Crossref] [PubMed]
6. Mahadevan S, Pradeepsankar S, Arunachalam M, Satish MJ, Karthika P, Chandramohan K. Numerous diagnostic methods for salivary gland tumors. International Journal of Health Sciences 2022;6:3221-30.
7. Zuo H. The Clinical Characteristics and CT Findings of Parotid and Submandibular Gland Tumours. Journal of Oncology 2021;2021:8874100.
8. Shen XM, Mao L, Yang ZY, Chai ZK, Sun TG, Xu Y, Sun ZJ. Deep learning-assisted diagnosis of parotid gland tumors by using contrast-enhanced CT imaging. Oral Dis 2023;29:3325-36. [Crossref] [PubMed]
9. Gabelloni M, Faggioni L, Attanasio S, Vani V, Goddi A, Colantonio S, Germanese D, Caudai C, Bruschini L, Scarano M, Seccia V, Neri E. Can Magnetic Resonance Radiomics Analysis Discriminate Parotid Gland Tumors? A Pilot Study. Diagnostics (Basel) 2020;10:900.
10. Zhang Z, Kong L, Zhang L, Pan X, Das T, Wang B, Liu B, Wang F, Nape I, Shen Y, Forbes A. Structured light meets machine intelligence. eLight 2025;5:1-30.
11. He K, Zhang X, Ren S, Sun J. Deep Residual Learning for Image Recognition. arXiv 2015. arXiv:1512.03385.
12. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser L, Polosukhin I. Attention Is All You Need. arXiv 2017. arXiv:1706.03762.
13. Tian R, Yu M, Liao L, Zhang C, Zhao J, Sang L, Qian W, Wang Z, Huang L, Ma H. An effective convolutional neural network for classification of benign and malignant breast and thyroid tumors from ultrasound images. Phys Eng Sci Med 2023;46:995-1013. [Crossref] [PubMed]
14. Habchi Y, Kheddar H, Himeur Y, Boukabou A, Chouchane A, Ouamane A, Atalla S, Mansoor W. Machine Learning and Vision Transformers for Thyroid Carcinoma Diagnosis: A review. arXiv 2024. arXiv:2403.13843.
15. Salvi M, Loh HW, Seoni S, Barua PD, García S, Molinari F, Acharya UR. Multi-modality approaches for medical support systems: A systematic review of the last decade. Information Fusion 2023;103:1-22.
16. Yuan Y, Hong Y, Lv X, Peng J, Li M, Guo D, Huang P, Chen C, Yan Z, Chen C, Li H, Ma H, Wang Y. Differentiating benign and malignant parotid gland tumors using CT images and machine learning algorithms. International Journal of Clinical and Experimental Medicine 2021;14:119-25.
17. Guntinas-Lichius O, Burmeister HP. The Diagnostic Imaging of Extended Parotid Tumors. In: Management of Extended Parotid Tumors. Cham: Springer; 2016.
18. Guntinas-Lichius O. Parotid Tumors with Muscle Extension. Springer International Publishing; 2016.
19. Lim SW, Te BC, Mohamad WEW. Infected Warthin Tumor: Complicated with Acute Facial Nerve Paralysis. International Journal of Health Sciences & Research 2016;6:422-5.
20. Chen J, Lu Y, Yu Q, Luo X, Adeli E, Wang Y, Lu L, Yuille AL, Zhou Y. TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation. arXiv 2021. arXiv:2102.04306.
21. Chen J, Mei J, Li X, Lu Y, Yu Q, Wei Q, Luo X, Xie Y, Adeli E, Wang Y, Lungren MP, Zhang S, Xing L, Lu L, Yuille A, Zhou Y. TransUNet: Rethinking the U-Net architecture design for medical image segmentation through the lens of transformers. Med Image Anal 2024;97:103280. [Crossref] [PubMed]
22. Marr D, Hildreth E. Theory of edge detection. Proc R Soc Lond B Biol Sci 1980;207:187-217. [Crossref] [PubMed]
23. Assirati L. Silva NuRd, Berton L, Bruno AdAL, Odemir M. Performing edge detection by difference of Gaussians using q-Gaussian kernels. Journal of Physics: Conference Series 2013;490:012020.
24. Zhang H, Zu K, Lu J, Zou Y, Meng D. EPSANet: An Efficient Pyramid Split Attention Block on Convolutional Neural Network. arXiv 2021. arXiv:2105.14447.
25. Woo S, Park J, Lee JY, Kweon IS. CBAM: Convolutional Block Attention Module. arXiv 2018. arXiv:1807.06521.
26. Cui Y, Che W, Liu T, Qin B, Yang Z. Pre-Training With Whole Word Masking for Chinese BERT. IEEE/ACM Transactions on Audio, Speech, and Language Processing 2021;29:3504-14.
27. Cui Y, Che W, Liu T, Qin B, Wang S, Hu G. Revisiting Pre-Trained Models for Chinese Natural Language Processing. arXiv 2020. arXiv:2004.13922.
28. Liu Y, Peng Y, Zhou H, Liu H, Lu X, Lou J, Wu C, Zhao W, Li J. LOVO: Efficient Complex Object Query in Large-Scale Video Datasets. arXiv 2025. arXiv:2507.14301.
29. Qiu B, van der Wel H, Kraeima J, Glas HH, Guo J, Borra RJH, Witjes MJH, van Ooijen PMA. Automatic Segmentation of Mandible from Conventional Methods to Deep Learning—A Review. Journal of Personalized Medicine 2021;11:629.
30. Çiçek Ö, Abdulkadir A, Lienkamp SS, Brox T, Ronneberger O. 3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation. arXiv 2016. arXiv:1606.06650.
31. Brox OR, Philipp F, Thomas B. U-Net: Convolutional Networks for Biomedical Image Segmentation. arXiv 2015. arXiv:1505.04597.
32. Isensee F, Jaeger PF, Kohl SAA, Petersen J, Maier-Hein KH. nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nat Methods 2021;18:203-11. [Crossref] [PubMed]
33. Cao H, Wang Y, Chen J, Jiang D, Zhang X, Tian Q, Wang M. Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation. arXiv 2021. arXiv:2105.05537.
34. Oktay O, Schlemper J, Folgoc LL, Lee M, Heinrich M, Misawa K, Mori K, McDonagh S, Hammerla NY, Kainz B, Glocker B, Rueckert D. Attention U-Net: Learning Where to Look for the Pancreas. arXiv 2018. arXiv:1804.03999.
35. Li FF, Deng J, Li K. ImageNet: Constructing a large-scale image database. Journal of Vision 2010;9:1037.
36. Lan T, Zheng Y, Yin J. Diffusion-Based Cross-Modal Feature Extraction for Multi-Label Classification. arXiv 2025.
37. Li X, Zhao H, Han L, Tong Y, Tan S, Yang K. Gated Fully Fusion for Semantic Segmentation. Proceedings of the AAAI Conference on Artificial Intelligence 2020;34:11418-25.
38. Lai KT, Liu D, Chang SF, Chen MS. Learning Sample Specific Weights for Late Fusion. IEEE Trans Image Process 2015;24:2772-83.
39. Tao R, Tang S, Qin H, Wang W, Wei Y, Zhao Y. LEDNet: a multimodal foundation model for robust deepfake detection. Science China Information Sciences 2025;68:1-15.
40. Deng D, Dong H. Advantages of contrast-enhanced CT combined with DCE-MRI in identifying malignant parotid tumor. Am J Transl Res 2022;14:9047-56.