End-to-end deep learning radiomics: development and validation of a novel attention-based aggregate convolutional neural network to distinguish breast diffuse large B-cell lymphoma from breast invasive ductal carcinoma

Introduction

Breast cancer is the most common cancer and the leading cause of death in women worldwide, of which invasive ductal carcinoma (IDC) is the most common pathological type. In China, newly diagnosed breast cancer cases accounts for 16.72% (n=306,000) of all new cancer cases and the number of deaths accounts for 8.12% (n=71,700) of the total cancer-related deaths in women (1,2). The current treatment for breast cancer includes surgery, chemotherapy, hormonal therapy, radiotherapy, targeted therapy, and immunotherapy, among which surgery is the first choice for early-stage breast cancers (3). Lymphoma represents a diverse group of diseases caused by clonal proliferation of lymphocytes (4). Lymphomas usually involve lymphoid organs and tissues. However, about 40% of lymphomas can also be located outside the lymph nodes (5). Breast lymphoma is a rare disease, the incidence of which is less than 0.5% of all breast malignancies, and diffuse large B-cell lymphoma (DLBCL) is the most common type of breast lymphoma (6). Breast lymphoma usually presents as a painless palpable mass, which is indistinguishable from breast cancer, and the common imaging examinations include mammographic, ultrasound, computed tomography (CT), and breast magnetic resonance imaging (MRI), which are unable to distinguish lymphoma from breast cancer (7,8). The most common diagnostic methods used to differentiate breast lymphoma from breast cancer are physical examination or image-guided needle biopsy (9). However, needle biopsy is an invasive technique which can damage normal breast tissue; misdiagnosis can occur if necrotic tissue is pierced when puncturing a large breast nodule containing necrotic tissue. Furthermore, as breast lymphoma is a rare disease with a low incidence, many breast nodules are misdiagnosed as breast cancer and are surgically removed directly (10). Unlike for breast cancer, which is primarily treated with surgery, radiation and chemotherapy are the main treatment modalities for breast lymphoma (9,11). Therefore, it is clinically important and necessary to explore a non-invasive technique that can differentially diagnose breast DLBCL from breast IDC in advance.

¹⁸F-fluorodeoxyglucose (¹⁸F-FDG) positron emission tomography/computed tomography (PET/CT) has the dual advantages of anatomical and functional imaging, which plays an increasingly important role in the diagnosis and treatment of lymphoma (12,13). ¹⁸F-FDG PET/CT imaging has been used for the initial staging, restaging, early treatment response, efficacy evaluation, prognosis prediction, and follow-up of lymphoma patients (14,15). Radiomics refers to the high-throughput extraction of a large number of image features that describe the characteristics of tumors, which is non-invasive and has been widely used for auxiliary diagnosis and classification for tumors (16). Ou et al. showed that radiomics features extracted from PET/CT and PET parameters can differentiate breast lymphoma from breast carcinoma using machine learning (17,18). However, their study was conducted within a single institution and the sample was very small, with only 44 patients recruited. Moreover, machine learning is a manual and labor-intensive technique with poor feature stability and repeatability (19,20). Despite these insufficiencies, no other non-invasive methods for distinguishing the 2 diseases have been reported so far. Deep learning is one of the hottest branches of machine learning, the biggest advantage of which is that it is deep enough and the network capacity is large enough. The most obvious benefit of a deep enough network is that it can accommodate richer semantic information (21,22).

Hence, this study was conducted to develop and validate an end-to-end deep learning radiomics method through ¹⁸F-FDG PET/CT images that could accurately and noninvasively distinguish breast DLBCL from breast IDC. We present this article in accordance with the TRIPOD reporting checklist (available at https://qims.amegroups.com/article/view/10.21037/qims-22-1333/rc).

Methods

The outline of the workflow from radiomic analysis with deep learning is illustrated in Figure 1, which mainly included the following steps: (I) data enrollment and ¹⁸F-FDG PET/CT acquisition; (II) patient division and grouping with IDC and DLBCL; (III) tumor annotation and PET/CT image preprocessing; (IV) attention-based aggregate convolutional neural network (AACNN) model architectures building; (V) model training and hyperparameter optimization; (VI) model evaluation based on prediction results and statistical analysis.

Figure 1 Outline of the workflow from radiomic analysis with deep learning. IDC, invasive ductal carcinoma; DLBCL, diffuse large B-cell lymphoma; FUSCC, Fudan University Shanghai Cancer Center; SPHIC, Shanghai Proton and Heavy Ion Center; AACNN, attention-based aggregate convolutional neural network; ROC, receiver operating characteristic; AUC, area under the ROC curve.

Study population

In our study, we analyzed 324 breast nodules from 236 patients (160 breast IDC and 76 breast DLBCL patients) who underwent baseline ¹⁸F-FDG PET/CT from January 2009 to December 2021 at Fudan University Shanghai Cancer Center (FUSCC) or Shanghai Proton and Heavy Ion Center (SPHIC). All patients aged over 18 years with a pathological diagnosis of breast IDC or DLBCL were included. The exclusion criteria were as follows: (I) other pathological types apart from IDC or DLBCL; (II) surgery, chemotherapy, radiotherapy, or other treatments had been administered before ¹⁸F-FDG PET/CT imaging; (III) other types of cancers; (IV) incomplete clinical data.

A total of 249 breast nodules (182 patients) from FUSCC were selected as the internal dataset and were used to train and build the AACNN model. At a ratio of 4:1, these participants were randomly divided into the training and testing datasets based on stratified sampling (23-25). A further 75 nodules (54 patients) from SPHIC were selected as an independent external dataset to further evaluate the robustness and generalization ability of the AACNN model. Details of patient selection and grouping are shown in Figure 2. We reviewed the hospital medical records of all 236 patients, and the baseline characteristics including gender, age, height, weight, and clinical stage of the patients, and the size, volume, and PET parameters of nodules were summarized. The study was conducted in accordance with the Declaration of Helsinki (as revised in 2013). The study was approved by the Ethics Committees of FUSCC (No. 1909207-14-1910) and SPHIC (No. 200217EXP-01) and the requirement for individual consent for this retrospective analysis was waived.

Figure 2 Flow chart of patient selection and study design. IDC, invasive ductal carcinoma; DLBCL, diffuse large B-cell lymphoma; ¹⁸F-FDG, ¹⁸F-fluorodeoxyglucose; PET/CT, positron emission tomography/computed tomography; FUSCC, Fudan University Shanghai Cancer Center; SPHIC, Shanghai Proton and Heavy Ion Center.

¹⁸F-FDG PET/CT image acquisition

The baseline ¹⁸F-FDG PET/CT imaging were performed on the Biograph 16HR and mCT Flow Siemens PET/CT scanners (Siemens, Erlangen, Germany) in FUSCC and a Biograph 16 Siemens PET/CT scanner (Siemens, Germany) in SPHIC. Before the scan, all patients were fasted for at least 6 hours with the blood glucose levels maintained below 8 mmol/L. After receiving an intravenous injection with 3.7 MBq/kg of ¹⁸F-FDG, the patients rested for approximately 1 hour, and then a whole-body scan from the head to the mid-thigh was performed for each patient. For PET/CT scans, the CT scan was performed firstly using a low-dose technique (120 kV and 140 mA for Biograph 16HR and mCT Flow scanners, 120 kV and 150 mA for Biograph 16 scanner). PET scans were acquired with 2 minutes allocated for each table position. The PET imaging dataset was reconstructed using the ordered subsets expectation maximum (OSEM) iterative reconstruction algorithm with CT data attenuated.

Image preprocessing and preparation

As described above, PET/CT images were acquired by 3 different machines in 2 different centers. On the one hand, the principles of CT and PET imaging are different, and on the other hand, the image information contained in these 2 image modalities is different, so we adopted different image preprocessing methods accordingly.

A diagram displaying the specifics of PET/CT image preprocessing is provided in Figure 3. The slice thickness and in-plane resolution of CT images are different. Firstly, all CT images were resampled into isotropic voxels of unit dimension through the bilinear interpolation algorithm to ensure comparability, and the voxel volume was set to 1 mm³ (26). Secondly, we normalized the heat unit (HU) maximum and minimum values and adjusted the window range to (−200 to 300). Similarly, the slice thickness and in-plane resolution of PET images were not consistent. Firstly, we converted PET scanning intensity from count units (CNTS) with absolute activity concentration (Bq/mL) to standard uptake value (SUV) values according to patient weight (27). Secondly, the PET image was resampled to 1 mm³ voxel using the bilinear interpolation algorithm to ensure comparability and reduce the impact on model classification (28).

Figure 3 Flowchart of image preprocessing and preparation. SUV, standardized uptake value; CT, computed tomography; PET, positron emission tomography; HU, heat unit; AACNN, attention-based aggregate convolutional neural network.

The ITK-SNAP software (http://www.itksnap.org/) was used to segment all breast nodules on PET images with anatomical information provided by CT images. The semi-automatic segmentation was interpreted and delineated by 2 experienced nuclear medicine radiologists who were blinded to the patients’ pathological information. If the radiologists had different opinions, a consensus was reached through discussion. According to the segmentation results confirmed by nuclear medicine radiologists, we cut each tumor region of interest (ROI) into 2-dimensional (2D) patches of 128×128 (PET images) and 64×64 (CT images). Finally, a series of data balance and enhancement techniques (rotation, translation, horizontal and vertical flip, etc.) were applied to increase the data size and data diversity of the training image, so as to prepare for the data input of the AACNN model (29).

For quantitative analysis, the 2D and 3-dimensional (3D) nodule size, the volume, and PET parameters including the minimum, mean, and maximum SUV (SUV_min, SUV_mean, and SUV_max, respectively) were calculated using python programs by delineating the ROI of the lesions. Metabolic tumor volume (MTV) was recorded at the threshold of 41% SUV_max in hypermetabolic regions, and total lesion glucose (TLG) was calculated following the equation below:

TLG=SUVmean×MTV[1]

AACNN model architectures

Figure 4 shows the network structure of our proposed AACNN model for discriminating breast lymphoma from breast cancer. The AACNN model takes PET tumor region patches and CT tumor region patches as input, respectively. It trains different independent models to identify disease classes based on the above images. The network architecture contains and extends bottleneck modules for dimensionality or feature reduction, residual modules that facilitate gradient propagation, and includes element-wise addition and channel-wise concatenation operation. Besides, it uses hierarchical and iterative skip connections to merge class-feature pyramid networks for improved resolution (30,31).

Figure 4 Schematic diagram of the network structure of the AACNN model. SN, switchable normalization; AACNN, attention-based aggregate convolutional neural network.

The first block of the network consists of a simple 3×3 convolutional layer (32), a switchable normalization layer (33), and the Mish activation function (34). The convolution calculation principle is given in Eq. [2], the adaptive normalization principle is given in Eq. [3], and the Mish activation function principle is given in Eq. [4].

N=(W−K+2×P)S+1[2]

N represents the output size, W represents the input size, K represents the convolution kernel size, P represents the size of the padding value, and S represents the step size.

h^ncij=γhncij−∑k∈Ωwkμk∑k∈Ωw′kδk2+ε+β[3]

Each pixel of the image is denoted as hncij, and the normalized pixel is denoted as. h∧ncijn,c,i,j are the minibatch size, the number of channels, the width of channels, and the height of channels, respectively. The statistics of switchable normalization (mean µ, variance σ2) are weighted and averaged by a suitable normalization method in the set Ω=[bn,in,ln⁡]. w_kand w^'_k are the weight coefficients of the corresponding statistics.

f(x)=x×tanh⁡(ln⁡(1+ex))[4]

x represents the input feature and f (x) represents the activated feature.

The second block of the network includes 2 branches. The main branch contains a bottleneck module and a pooling layer to reduce the feature dimension. 1×1 convolutions were inserted to learn more non-sparse features. The feature information of the 2 branches is superimposed. Inspired by the Inception V3 network structure (35), the third and fourth blocks of the network used asymmetric convolutions with 1×n and n×1 filters to replace the traditional n×n symmetric convolution on the basis of the second part, where channel∈ [8, 32]. The purpose of this structural design is to reduce the parameter size and computational cost. In order to better highlight the main features and suppress irrelevant features, we also added a convolutional block attention module (CBAM) between the third and fourth blocks (36). In the fourth block, channel-wise concatenation was added to further enhance deep features. The equation for calculating the CBAM is as follows:

F'=Mc(F)⊗F,F''=Ms(F')⊗F',Mc(F)=σ(MLP(AvgPool(F))+MLP(MaxPool(F))),Ms(F)=σ(f7×7([AvgPool(F);MaxPool(F)]))[5]

Where F represents intermediate features, M_c represents channel attention features, M_s represents spatial attention features,⊗ represents element-wise multiplication, F^' represents features computed by the channel attention mechanism, F^'' is the output of final refinement, MLP is multi-layer perceptron, σ is sigmoid function, AvgPool stands for average pooling, MaxPool stands for maximum pooling, and f^7×7 is a 7×7 convolution kernel.

Model training and hyperparameters

The input size of the AACNN model was set to 64×64×1. During model training, the convolution weights were initialized using the ‘He’ uniform variance scaling initialization method (37). The L2 regularization technique (weight decay coefficient =0.0001) was used to limit the squared magnitude of the kernel weights to avoid overfitting. The Dropout technique was applied to the fully connected layer to make the fully connected layer randomly select some nodes to not play a role during the learning process (38). The Adam optimizer was used to update the network gradient by minimizing the loss function to train the model. In order to ensure the robustness of the prediction model and improve the computational cost efficiency, an early termination condition was introduced, and the training was automatically stopped when the accuracy of the validation set had still not improved after 20 repetitions. Other hyperparameter settings included a dynamic learning rate of 0.001, a number of iterations of 1,000, and a batch size of 64.

In addition, to alleviate the impact of erroneous labels, based on the idea of label smoothing (39), we improved the binary cross-entropy loss function, which is formulated as follows:

Loss={(1−ε)×(−∑i=12pilog⁡qi),ifi=yε×(−∑i=120.2×p′ilog⁡qi),otherwise[6]

Where i represents 1 of the 2 classes, y represents the actual class, p_irepresents the actual label, ε represents constant 0.1, q_i represents the predicted label, and p′i represents an array of 1 (which has the same shape and type as the predicted label array).

Performance evaluation

In this study, multiple repeated random subsampling validation was used to evaluate the model, which has the same purpose as the fixed split in k-fold cross validation, and can better control the number of model training and validation, as well as the ratio of training set to testing set, and obtain more accurate results (37). The performance of the AACNN model was investigated by 7 evaluation indexes, namely, accuracy (ACC), sensitivity (SEN), specificity (SPE), positive predictive value (PPV), negative predictive value (NPV), harmonic mean of precision and sensitivity (F1), and the area under the receiver operating characteristic (ROC) curve (AUC). The equations are expressed as follows:

ACC=TP+TNTP+FP+TN+FN[7]

SEN=TPTP+FN[8]

SPE=TNTN+FP[9]

PPV=TPTP+FP[10]

NPV=TNTN+FN[11]

F1=2×TP2×TP+FP+FN[12]

TP represents the number of true positives, FP represents the number of false positives, TN represents the number of true negatives, and FN represents the number of false negatives. SEN represents the proportion of all DLBCL lesions that have been correctly predicted by the AACNN model, which measures AACNN’s ability to recognize breast lymphoma. SPE represents the proportion of all IDC lesions that have been correctly predicted by the AACNN model, which measures AACNN’s ability to recognize breast cancer. PPV indicates the proportion of DLBCL lesions predicted by the AACNN model that are indeed DLBCL. NPV indicates the proportion of IDC lesions predicted by that AACNN model that are indeed IDC.

Platform and software

A computer running Windows 10 with an Intel Core I7-8750H @ 2.2 GHz × 2.2 GHz CPU, 8 GB memory, and an Nvidia GeForce GTX 1060 graphics processing unit (Nvidia, San Jose, CA, USA) was used for the analysis. The software used in this paper included PyCharm (version 2018.2, http://www.jetbrains.com), ITK-SNAP (version 3.8.0), 3D Slicer (version 4.11; http://www.slicer.org), MATLAB (version R2018a; http://www.mathworks.com), and SPSS (version 26.0; http://www.spss.com.cn). To build the AACNN model and process PET/CT images, we utilized several publicly available software packages, including Keras, SimpleITK, NumPy, skimage, os, pandas, SciPy, Scikit-Learn, and Pydicom.

Statistical analysis

The SPSS software (IBM Corp., Armonk, NY, USA) was used for the statistical analysis. Statistical methods included independent samples t-test for the comparison between 2 groups, Mann-Whiney U test for continuous characteristics, and χ² test or Fishers exact test for categorical characteristics. DeLong’s method using PyCharm was used to test the statistical significance between AUCs. A P value of below 0.05 was considered statistically significant.

Results

Study population

A total of 236 patients (235 females and 1 male; average age, 51.31 years; range 24 to 86 years) were included in our multi-center study (Table 1). Among the patients, the average height, weight, and body mass index (BMI) was 1.59±0.05 m, 58.90±8.69 kg, and 23.30±3.24 kg/m², respectively. The clinical features of the patients were not significantly different between the groups. Of the 236 patients, 160 (67.80%) had breast IDC and 76 (32.20%) had breast DLBCL (Table S1). Among the breast IDC patients, 12 (7.50%), 66 (41.25%), 31 (19.38%), and 51 (31.88%) patients were with tumor-node-metastasis (TNM) stage I, II, III, and IV, and among the breast DLBCL patients, 24 (31.58%), 30 (39.47%), 1 (1.32%), and 21 (27.63%) patients were with Ann Arbor stage I, II, III, and IV, respectively. The clinical stage between breast IDC and DLBCL was statistically significant (P<0.001).

As shown in Table 2, a total of 249 (135 IDC and 114 DLBCL) and 75 (39 IDC and 36 DLBCL) breast nodules were included in the internal and external datasets, respectively. The clinical characteristics of the nodules had no differences between the internal and external datasets. The nodule size (containing 2D and 3D size and nodule volume) and PET parameters (containing SUV_mean, SUV_max, MTV, and TLG) were significantly different between the IDC and DLBCL groups in the internal dataset testing according to the univariate analysis (P<0.001) (Table 3). Meanwhile, in the external dataset testing, only PET parameters (SUV_min, SUV_mean, and SUV_max) were statistically significant between IDC and DLBCL (Table S2). The representative 2D and 3D schematic diagrams of breast nodules are shown in Figure 5.

Figure 5 2D and 3D schematic diagram of breast nodules. 2D (A) and 3D (B) schematic diagram of a breast cancer lesion. 2D (C) and 3D (D) schematic diagram of a breast lymphoma lesion. 2D schematic diagram of a breast lymphoma lesion. 2D, 2-dimensional; 3D, 3-dimensional.

AACNN model prediction results

The AACNN model was trained based on CT (AACNN_CT) and PET (AACNN_PET) patches respectively, and the external dataset was adopted to further test the robustness of the model. As shown in Table 4, AACNN_CT was tested on the internal testing dataset, the highest AUC was 0.886, the highest prediction ACC was 82.2%, the highest SEN was 82.4%, the highest SPE was 82.0%, the highest PPV was 82.6%, the highest NPV was 81.8%, and the highest F1 value was 0.825, respectively. The highest predictions of AACNN_CT on the external testing dataset achieved AUC, ACC, SEN, SPE, PPV, NPV, and F1 of 0.784, 71.6%, 66.3%, 78.5%, 80.2%, 64.0%, and 0.726, respectively. When PET patches were used as the input of the AACNN model, the best internal testing performance showed that the AUC, ACC, SEN, SPE, PPV, NPV, and F1 were 0.831, 76.5%, 78.1%, 74.7%, 76.2%, 76.8%, and 0.771, respectively. Similarly, on the external testing dataset, the best AUC, ACC, SEN, SPE, PPV, NPV, and F1 were 0.726, 65.5%, 57.5%, 76.1%, 75.9%, 57.7%, and 0.654, respectively. In general, the prediction performance of AACNN_CT model was better than AACNN_PET.

As shown in Table 4, the AACNN_ensemble (AACNN_E) model prediction results (AUC, ACC, SEN, SPE, PPV, NPV, and F1) were 0.886, 83.0%, 80.9%, 85.0%, 84.8%, 81.2%, and 0.828 (internal testing) and 0.788, 71.6%, 61.4%, 84.7%, 84.0%, 62.6%, and 0.709 (external testing), respectively, which was improved in 5 performance metrics (AUC, ACC, SPE, PPV, and F1) compared to all models, and the remaining 2 metrics (SEN, NPV) was also close to the highest (82.4%, 81.8%). The model results of the AACNN_E was a back-end fusion based on distance weight voting between the model prediction probability values of AACNN_CT and AACNN_PET (40), the Eq. [13] for the back-end fusion based on distance weights voting is displayed below.

Pfusion={max⁡(PAACNN_CT,PAACNN_PET),ifPAACNN_CT≥0.5andPAACNN_PET≥0.5min⁡(PAACNN_CT,PAACNN_PET),ifPAACNN_CT<0.5andPAACNN_PET<0.5max⁡(|PAACNN_CT−0.5|,|PAACNN_PET−0.5|),otherwise[13]

P_fusion represents fusion prediction results, P_{AACNN_CT} represents prediction probability values of AACNN_CT, and P_{AACNN_PET}represents prediction probability values of AACNN_PET.

Further, the AUCs between AACNN_CT and AACNN_PET, AACNN_CT and AACNN_E, AACNN_PET, and AACNN_E were statistically significant (P<0.05, DeLong’s test). The ROC curves of all models are shown in Figure 6.

Figure 6 ROC curves of the best AACNN applied to the internal/external testing. ROC curve for internal (A) and external (B) testing on AACNN_CT model. ROC curve for internal (C) and external (D) testing on AACNN_PET model. ROC curve for internal (E) and external (F) testing on AACNN_E model. ROC, receiver operating characteristic; AUC, area under the ROC curve; AACNN, attention-based aggregate convolutional neural network; CT, computed tomography; PET, positron emission tomography; E, ensemble.

The proposed AACNN models can extract deep features with different locations, sizes, contours, and texture abstractions in convolutional layers. To more clearly demonstrate the ability of different layers of the model to extract features for lesions, the class activation maps from shallow to deep is represented in Figure 7.

Figure 7 Class activation maps visualization of AACNN layer by layer. AACNN, attention-based aggregate convolutional neural network.

Finally, as shown in Table 5, the prediction probability values of the 3 models AACNN_CT, AACNN_PET, and AACNN_E were analyzed by univariate analysis and found to be statistically significant (P<0.001). The effectiveness of the AACNN models proposed in this paper was further validated statistically.

Discussion

Breast nodules are overwhelmingly seen in women, of which breast IDC is the most common malignancy. Although breast DLBCL is a rare disease, its incidence rate is increasing with the changes of people’s living environment and other factors. The ineffectiveness of conventional imaging in distinguishing these 2 diseases leads to unnecessary surgery in patients with breast lymphoma (8). Artificial intelligence technology represented by deep learning is a widely used intelligent analysis technology in medical data or image analysis, which can be used in various image analysis tasks involved in clinical diagnosis, treatment, and prognosis prediction. Deep learning radiomics does not require precise segmentation of tumor regions and pre-definition of high-throughput feature sets based on prior knowledge. Key features relevant to clinical questions can be automatically extracted and optimized through different deep learning network architectures. Automatic feature extraction does not involve human interaction, and the extracted features are the most implicit and advanced. In our study, we proposed an end-to-end AACNN model which can effectively differentiate breast DLBCL from breast IDC, and the radiomic features learned from the model had significant differences between the 2 diseases. Meanwhile, we used the multi-center data for preliminary verification. As most breast DLBCL patients show multiple lesions, most breast IDC patients show single lesion, and the AACNN model we established was to make accurate differentiation of the lesions, 76 breast DLBCL patients (150 lesions) and 160 breast IDC patients (174 lesions) were recruited finally, and the number of lesion samples was balanced at almost 1:1. To the best of our knowledge, this is the first deep learning model to identify these 2 diseases.

In previous studies, researchers usually only considered the 2D diameter of the lesion. Contrastingly, in our study, we added the 3D diameter and volume of the lesion, and the results showed that the spatial data of the lesions had statistical significance in differentiating the 2 diseases in the internal dataset testing, which can provide more comprehensive spatial distribution information for the lesion and may provide more effective information for the selection of treatment and prognosis evaluation of patients (41). Representative 2D and 3D images of the lesion are shown in Figure 5. PET parameters (including SUV, MTV, TLG, etc.) derived from PET/CT are closely related to tumor metabolism. Ou et al.’s study revealed that the SUV metrics from PET/CT images had potential utility in differentiating breast lymphoma from carcinoma. In our study, the PET parameters (including SUV_mean, SUV_max, MTV, and TLG) after the above image preprocessing were significant to differentiate the 2 diseases on the internal dataset testing, which was consistent with a previous finding (17). However, only 3 PET parameters (SUV_min, SUV_mean, and SUV_max) showed significance in the external dataset testing; the other parameters showed no statistical differences. Firstly, the reason may be due to the small sample size with limited diversity included in the external testing. Secondly, the clinical parameters may not be specific indicators of breast disease, which may lead to difficulty in discriminating between breast IDC and DLBCL. Similarly, the sensitivity of the external dataset for our model was low, with 0.663 for AACNN_CT, 0.575 for AACNN_PET, and 0.614 for AACNN_E, which further verified our conjecture (Table 4). Thirdly, because of the instability and non-specificity of these clinical factors, the potential of radiomic analysis in this field is urgently needed.

In order to effectively and noninvasively distinguish breast DLBCL and breast IDC patients, we designed a new AACNN model structure (Figure 4). The structure combines the Mish activation function, the improved loss function based on label smoothing, and the convolutional attention mechanism module. We also proposed a back-end voting strategy basing on distance weight voting, and built an ensemble model named AACNN_E. The experimental results (AUC =0.886; ACC =83.0%) indicated that the ensemble model could improve the discrimination ability of breast DLBCL from breast IDC to a certain extent.

In addition, compared to existing studies, data from 3 PET/CT machines of different types in 2 centers with longer time spans were included in our study, which increased the diversity and reliability of the samples (Table S3). Independent AACNN models based on PET and CT were trained and tested by internal cohorts, and independent external cohorts were used to validate the generalization and robustness of the model (Table 4 and Figure 6) (42). Meanwhile, we do not need to manually extract information such as lesion size, dimension, texture, and so on. Our model can automatically learn these features and make predictions directly. Further, we have demonstrated that the prediction result of the AACNN models can be directly used as a significant factor for discriminating breast IDC from DLBCL. The AUC of our model was improved by nearly 5% compared to existing models.

Last but not least, the class activation map visualization (from shallow to deep) (Figure 7) realized the interpretability of AACNN prediction results, revealed a corner of the mystery of the “black box” problem in deep learning, and further reflected the effectiveness in extracting disease radiomics features of the AACNN model proposed in this paper. Class activation map visualization refers to the generation of a heat map of class activations on the input image, indicating the importance of each position to the category. It helps to understand which part of a picture causes the AACNN to make the final decision. We have presented the class activation map of the largest tumor area patch of a patient with breast lymphoma. The brighter (redder) pixel regions in the maps indicate the greater contribution of the classification. On the contrary, the bluer and darker pixel regions account for a smaller proportion of the contribution to the classification. It is worth noting that the obtained class activation maps are consistent with the lesion regions that clinicians pay attention to, indicating that the AACNN classification model trained in this paper can well locate image features with clinical diagnostic value.

Our study still had some limitations. Our study was based on a relatively small sample of breast DLBCL, and participants with incomplete clinical data were omitted, which may affect the accuracy and reliability of the research results. Although the data enhancement method was used to alleviate the problem of insufficient sample diversity, there are still uncertainties and inauthenticity compared with the real data. Combined with the actual clinical application and promotion, PET/CT fusion images may have better effects as the input of the model. This study did not combine deep learning with traditional radiomics methods, and the method of fusing deep features and traditional omics features may further improve the performance of the current AACNN model.

Conclusions

This study developed a deep learning radiomics strategy based on AACNN model and ensemble prediction which may automatically achieve remarkable prediction performance at the classification task of distinguishing breast DLBCL from breast IDC. The proposed model relied on data from 2 centers and was validated through an external dataset, demonstrating that deep features learned through convolutional layers could distinguish between the 2 diseases. The experimental results showed that our proposed method could learn more lesion information, significantly improving the ability to differentiate between breast IDC and breast DLBCL. Our proposed model is expected to become a non-invasive auxiliary tool for precision diagnosis in the future.

Acknowledgments

Funding: This study was supported by the National Natural Science Foundation of China (Nos. 82102097 and 81971648) and the Talent Introduction Fund of Fudan University Shanghai Cancer Center (No. YJRC202104).

Footnote

Reporting Checklist: The authors have completed the TRIPOD reporting checklist. Available at https://qims.amegroups.com/article/view/10.21037/qims-22-1333/rc

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at https://qims.amegroups.com/article/view/10.21037/qims-22-1333/coif). The authors have no conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. The study was conducted in accordance with the Declaration of Helsinki (as revised in 2013). The study was approved by the Ethics Committees of FUSCC (No. 1909207-14-1910) and SPHIC (No. 200217EXP-01) and the requirement for individual consent for this retrospective analysis was waived.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. Zheng R, Zhang S, Zeng H, Wang S, Sun K, Chen R, Li L, Wei W, He J. Cancer incidence and mortality in China, 2016. J Natl Cancer Cent 2022;2:1-9.
2. Sung H, Ferlay J, Siegel RL, Laversanne M, Soerjomataram I, Jemal A, Bray F. Global Cancer Statistics 2020: GLOBOCAN Estimates of Incidence and Mortality Worldwide for 36 Cancers in 185 Countries. CA Cancer J Clin 2021;71:209-49. [Crossref] [PubMed]
3. Rodrigues-Ferreira S, Nahmias C. Predictive biomarkers for personalized medicine in breast cancer. Cancer Lett 2022;545:215828. [Crossref] [PubMed]
4. Matasar MJ, Zelenetz AD. Overview of lymphoma diagnosis and management. Radiol Clin North Am 2008;46:175-98. vii. [Crossref] [PubMed]
5. Gurney KA, Cartwright RA. Increasing incidence and descriptive epidemiology of extranodal non-Hodgkin lymphoma in parts of England and Wales. Hematol J 2002;3:95-104. [Crossref] [PubMed]
6. Sousaris N, Barr RG. Sonoelastography of Breast Lymphoma. Ultrasound Q 2016;32:208-11. [Crossref] [PubMed]
7. Surov A, Holzhausen HJ, Wienke A, Schmidt J, Thomssen C, Arnold D, Ruschke K, Spielmann RP. Primary and secondary breast lymphoma: prevalence, clinical signs and radiological features. Br J Radiol 2012;85:e195-205. [Crossref] [PubMed]
8. Picasso R, Tagliafico A, Calabrese M, Martinoli C, Pistoia F, Rossi A, Zaottini F, Derchi L. Primary and Secondary Breast Lymphoma: Focus on Epidemiology and Imaging Features. Pathol Oncol Res 2020;26:1483-8. [Crossref] [PubMed]
9. Nicholson BT, Bhatti RM, Glassman L. Extranodal Lymphoma of the Breast. Radiol Clin North Am 2016;54:711-26. [Crossref] [PubMed]
10. Breese RO, Friend K. Thirteen-Centimeter Breast Lymphoma. Am Surg 2022;88:1891-2. [Crossref] [PubMed]
11. Gluskin J, D'Alessio D, Kim AC, Morris EA, Chiu A, Noy A. Primary lymphoma of the breast: A report of two cases. Clin Imaging 2020;68:295-9. [Crossref] [PubMed]
12. Cheson BD. Staging and response assessment in lymphomas: the new Lugano classification. Chin Clin Oncol 2015;4:5.
13. Meignan M, Itti E, Gallamini A, Younes A. FDG PET/CT imaging as a biomarker in lymphoma. Eur J Nucl Med Mol Imaging 2015;42:623-33. [Crossref] [PubMed]
14. Thanarajasingam G, Bennani-Baiti N, Thompson CA. PET-CT in Staging, Response Evaluation, and Surveillance of Lymphoma. Curr Treat Options Oncol 2016;17:24. [Crossref] [PubMed]
15. Eertink JJ, Burggraaff CN, Heymans MW, Dührsen U, Hüttmann A, Schmitz C, et al. Optimal timing and criteria of interim PET in DLBCL: a comparative study of 1692 patients. Blood Adv 2021;5:2375-84. [Crossref] [PubMed]
16. Lee JW, Lee SM. Radiomics in Oncological PET/CT: Clinical Applications. Nucl Med Mol Imaging 2018;52:170-89. [Crossref] [PubMed]
17. Ou X, Wang J, Zhou R, Zhu S, Pang F, Zhou Y, Tian R, Ma X. Ability of (18)F-FDG PET/CT Radiomic Features to Distinguish Breast Carcinoma from Breast Lymphoma. Contrast Media Mol Imaging 2019;2019:4507694. [Crossref] [PubMed]
18. Ou X, Zhang J, Wang J, Pang F, Wang Y, Wei X, Ma X. Radiomics based on (18) F-FDG PET/CT could differentiate breast carcinoma from breast lymphoma using machine-learning approach: A preliminary study. Cancer Med 2020;9:496-506. [Crossref] [PubMed]
19. Jarrett D, Stride E, Vallis K, Gooding MJ. Applications and limitations of machine learning in radiation oncology. Br J Radiol 2019;92:20190001. [Crossref] [PubMed]
20. Le NQK, Kha QH, Nguyen VH, Chen YC, Cheng SJ, Chen CY. Machine Learning-Based Radiomics Signatures for EGFR and KRAS Mutations Prediction in Non-Small-Cell Lung Cancer. Int J Mol Sci 2021;22:9254. [Crossref] [PubMed]
21. McBee MP, Awan OA, Colucci AT, Ghobadi CW, Kadom N, Kansagra AP, Tridandapani S, Auffermann WF. Deep Learning in Radiology. Acad Radiol 2018;25:1472-80. [Crossref] [PubMed]
22. Saba L, Biswas M, Kuppili V, Cuadrado Godia E, Suri HS, Edla DR, Omerzu T, Laird JR, Khanna NN, Mavrogeni S, Protogerou A, Sfikakis PP, Viswanathan V, Kitas GD, Nicolaides A, Gupta A, Suri JS. The present and future of deep learning in radiology. Eur J Radiol 2019;114:14-24. [Crossref] [PubMed]
23. Chen W, Hou X, Hu Y, Huang G, Ye X, Nie S. A deep learning- and CT image-based prognostic model for the prediction of survival in non-small cell lung cancer. Med Phys 2021;48:7946-58. [Crossref] [PubMed]
24. Lu Z, Xu S, Shao W, Wu Y, Zhang J, Han Z, Feng Q, Huang K. Deep-Learning-Based Characterization of Tumor-Infiltrating Lymphocytes in Breast Cancers From Histopathology Images and Multiomics Data. JCO Clin Cancer Inform 2020;4:480-90. [Crossref] [PubMed]
25. Mohanty SP, Hughes DP, Salathé M. Using Deep Learning for Image-Based Plant Disease Detection. Front Plant Sci 2016;7:1419. [Crossref] [PubMed]
26. Gong J, Liu J, Li H, Zhu H, Wang T, Hu T, Li M, Xia X, Hu X, Peng W, Wang S, Tong T, Gu Y. Deep Learning-Based Stage-Wise Risk Stratification for Early Lung Adenocarcinoma in CT Images: A Multi-Center Study. Cancers (Basel) 2021.
27. Liu Q, Sun D, Li N, Kim J, Feng D, Huang G, Wang L, Song S. Predicting EGFR mutation subtypes in lung adenocarcinoma using (18)F-FDG PET/CT radiomic features. Transl Lung Cancer Res 2020;9:549-62. [Crossref] [PubMed]
28. Eertink JJ, Brug T, Wiegers SE, Zwezerijnen G, Zijlstra JM. 18F-FDG PET/CT baseline rdiomics features improve the prediction of treatment outcome in diffuse large B-cell lymphoma patients. Blood 2020;136:27-8.
29. Zhang J, Xia Y, Zeng H, Zhang Y. NODULe: Combining constrained multi-scale LoG filters with densely dilated 3D deep convolutional neural network for pulmonary nodule detection. Neurocomputing 2018;317:159-67.
30. He KM, Zhang XY, Ren SQ, Sun J. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016 June 27-30; Las Vegas, NV, USA. Piscataway: Proceedings of the IEEE; 2016:770-8.
31. Lin TY, Dollar P, Girshick R, He KM, Hariharan B, Belongie S. Feature pyramid networks for object detection. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017 July 21-26; Honolulu, HI, USA. Piscataway: Proceedings of the IEEE; 2017:936-44.
32. Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition. International Conference on Learning Representations (ICLR), 2015 May 7-9; San Diego, CA. Boston: OpenReview.net; 2015:1-14.
33. Luo P, Zhang R, Ren J, Peng Z, Li J. Switchable Normalization for Learning-to-Normalize Deep Representation. IEEE Trans Pattern Anal Mach Intell 2021;43:712-28. [Crossref] [PubMed]
34. Misra D. Mish: A self-regularized non-monotonic neural activation function. arXiv 2020, August:1-13. doi: 10.48550/arXiv.1908.08681.
35. Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z. Rethinking the inception architecture for computer vision. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR); 2016 June 27-30; Las Vegas, NV, USA. Piscataway: Proceedings of the IEEE; 2016:2818-26.
36. Wei W, Wong Y, Du Y, Hu Y, Kankanhalli M, Geng W. A multi-stream convolutional neural network for sEMG-based gesture recognition in muscle-computer interface. Pattern Recognit Lett 2019;109:131-8.
37. Christopher AR, Timothy AW, Aaron EM. Evaluation of sampling and cross-validation tuning strategies for regional-scale machine learning classification. Remote Sensing 2019;11:185.
38. Srivastava N, Hinton G, Krizhevsky A, Sutskever I, Salakhutdinov R. Dropout: a simple way to prevent neural networks from overfitting. J Mach Learn Res 2014;15:1929-58.
39. Müller R, Kornblith S, Hinton G. When does label smoothing help? NeurIPS 2019;422:4694-703.
40. Liao T, Lei Z, Zhu T, Zeng S, Li Y, Yuan C. Deep Metric Learning for K Nearest Neighbor Classification. IEEE Trans Knowl Data Eng 2023;35:264-75.
41. Wang X, Bao N, Xin X, Tan J, Li H, Zhou S, Liu H. Automatic evaluation of endometrial receptivity in three-dimensional transvaginal ultrasound images based on 3D U-Net segmentation. Quant Imaging Med Surg 2022;12:4095-108. [Crossref] [PubMed]
42. Lam NFD, Sun H, Song L, Yang D, Zhi S, Ren G, Chou PH, Wan SBN, Wong MFE, Chan KK, Tsang HCH, Kong FS, Wáng YXJ, Qin J, Chan LWC, Ying M, Cai J. Development and validation of bone-suppressed deep learning classification of COVID-19 presentation in chest radiographs. Quant Imaging Med Surg 2022;12:3917-31. [Crossref] [PubMed]