Classification of superficial suspected lymph nodes: non-invasive radiomic model based on multiphase contrast-enhanced ultrasound for therapeutic options of lymphadenopathy

Introduction

Lymph nodes (LNs) are important peripheral immune organs and participate in the body’s immune responses. Benign LNs, lymphoma, and metastasis LNs are the three most commonly confused types of lymphadenopathy (1). Accurate determination of the LN properties is of great importance in disease diagnosis and treatment (2). Currently, the gold standard for the classification of LNs remains the pathological results obtained by biopsies or surgeries, but these procedures are invasive. The differential diagnosis of lymphadenopathy is challenging, especially when there is a lack of reliable medical histories and typical clinical symptoms (3).

Several imaging modalities can be used to monitor the superficial suspected LNs, such as ultrasound, computed tomography (CT), magnetic resonance imaging (MRI), and positron emission tomography (PET) (4). The role of CT is limited for the nodal staging (5). Although contrast-enhanced MRI can increase the diagnostic efficiency compared with standard MRI and CT, it requires additional scanning time and is relatively expensive (6). PET is useful in identifying metastatic nodal spread outside the primary sites, but it has low spatial resolution and lacks anatomic details (7,8). Among these imaging modalities, ultrasound is more convenient, economical, and radiation-free. Contrast-enhanced ultrasound (CEUS) may provide real-time visualization of blood flow within LNs, allowing for immediate assessment. It is non-nephrotoxic and is generally considered safe and well-tolerated. Besides, CEUS can be performed with most ultrasound facilities, making it widely available. Although ultrasound is operator dependent and has limited ability to assess tissue characteristics or morphology compared to CT and MRI, it is recommended as the preferred diagnostic method for LN evaluation (9). Notably, the accurate classification of superficial suspected lymphadenopathy by ultrasound remains challenging (3). A more accurate noninvasive imaging diagnostic strategy is needed.

As a kind of artificial intelligence technology, radiomics can extract high-throughput quantitative information from medical images and help in the clinical diagnosis and treatment (10,11). To date, some studies have applied radiomics methods in the diagnosis of LN lesions. Zhu et al. reported that radiomics of dualmodality ultrasound images can help the diagnosis of unexplained cervical lymphadenopathy (3). With the help of the hierarchical diagnostic model, the diagnostic accuracies of the radiologists are generally improved. Chen et al. derived intrinsic imaging phenotypes from the ultrasound images and analysed the role of imaging phenotypes in the LN discrimination (12). Additionally, Liu et al. conducted a multicentered study to explore the ultrasound-based radiomics for differentiating cervical lymphadenopathy (13). However, most of these studies extracted the radiomics features from traditional grayscale ultrasound images, with few studies focusing on the CEUS images, which contain more information about the blood perfusion of the lesion that can improve the diagnostic accuracy (14). The feasibility of radiomics tools to classify superficial lymphadenopathy based on CEUS remains to be verified.

Therefore, the purpose of this study was to develop a noninvasive imaging method for the classification of lymphadenopathy as a possible alternative to the traditional pathological biopsy. Specifically, a noninvasive radiomic model based on multiphase CEUS images was applied for the classification of different types of lymphadenopathy in this study. We present this article in accordance with the TRIPOD reporting checklist (available at https://qims.amegroups.com/article/view/10.21037/qims-23-1182/rc).

Methods

The study was conducted in accordance with the Declaration of Helsinki (as revised in 2013). Approval for this retrospective cohort study was granted by the Ethics Committee of Shaoxing People’s Hospital (No. 2022-097-Y01). Informed written consent to participate was waived given the retrospective nature of the work.

Patient selection and demographic characteristics

From February 2019 to March 2023, the patients with suspected superficial LNs who received conventional ultrasound examination were retrospectively analysed in this study. The suspected ultrasound characteristics of the LNs included long/short (L/S) diameter ratio <2, round shape, loss of the fatty hilum, presence of cystic content, presence of calcification, and peripheral vascularity (15). The inclusion criteria were as follows: (I) the presence of superficial suspected LNs; (II) LNs with complete CEUS images; and (III) LNs with pathological classification confirmed by biopsy or surgery. The exclusion criteria were as follows: (I) patients with ambiguous pathological results; (II) patients without complete clinical information; and (III) patients with poor imaging data. The data were randomly dichotomized into a training cohort (60%) and an internal testing cohort (40%). Briefly, all the participants in the study were assigned a unique identifier. A randomization table was used to generate a random sequence of numbers. Starting from the first participant, the random sequence of numbers was used to assign each participant to either the training cohort or the internal testing cohort based on the 6:4 ratio. This process continued until all participants had been assigned to one of the two cohorts, ensuring that the desired ratio was met. Figure 1 presents the flowchart of the patient selection.

Figure 1 Patient selection flowchart. Hospital 1, Shaoxing People’s Hospital; hospital 2, Xiamen Hospital, Beijing University of Chinese Medicine; hospital 3, the First Affiliated Hospital of Ningbo University. LNs, lymph nodes; CEUS, contrast-enhanced ultrasound.

Pathological data

All the LNs analysed in our study had pathological results made by pathologist. The pathologist evaluates the histological findings and determines the diagnosis (16). The accurate pathologic classification of the lymphoma may require further immunohistochemistry and molecular testing (17). Details of the pathological results are presented in Table S1.

Conventional ultrasound evaluation

All ultrasound examinations were performed by trained and experienced sonographers using ultrasound instruments equipped with linear array transducers. Details of the ultrasound instruments used in the study are presented in Tables S2,S3. The morphology and size of the selected LNs were evaluated by grayscale ultrasound, while the blood flow of the LNs was evaluated by color Doppler imaging or superb microvascular imaging (SMI). The recorded ultrasound features included: the size, the longitudinal diameter (L), the short diameter (S), L/S ratio, boundary, hilum of LN, calcification (present or absent), liquefaction (present or absent), and peripheral vascularity (present or absent). After the collection of all the ultrasound examinations, two experienced radiologists (with 8 and 10 years of experience respectively) independently reviewed the images blindly. Any discrepancies were resolved by consensus.

CEUS evaluation

CEUS was performed immediately after conventional ultrasound evaluation. Dynamic CEUS videos of the LNs were taken simultaneously with the injection of SonoVue (Bracco, Italy) and lasted for at least 60 s (18). The videos were then digitally stored for offline analyses. The echoes of the LNs were compared with those of adjacent tissues after CEUS, and the perfusion and enhancement features of the selected LNs were described from the following aspects: enhancement intensity (hypo/isoenhancement, or hyperenhancement), enhancement pattern (centrifugal, centripetal, or overall), enhancement homogeneity (homogeneous or heterogeneous), ring enhancement (present or absent), and perfusion defect (present or absent). The wash-in time represented the time when the echo intensity of the contrast agent changes from arrival to peak, while the wash-out time represented the time when the echo intensity of the contrast agent changes from peak to baseline. The CEUS videos were independently and blindly reviewed by two experienced sonographers. Any discrepancies were resolved by consensus. The diagnostic performances of the radiologists were compared with the pathological findings.

Image segmentation and radiomics feature extraction

Each region of the LNs was manually segmented by two independent experienced radiologists who were blinded to the pathological results. First, the radiologists reviewed the CEUS videos to identify the boundaries of the LNs. Then, multiphase images of each LN were derived from the dynamic videos. Multiphase images included the images from the following three time points: the contrast agent arrived at the LN (prevascular phases, usually within 10–20 s), the echo of contrast agent reached peak intensity (vascular phases, usually around 30–45 s), and the contrast agent washed out (postvascular phases, usually more than 45 s after the injection). The time points in the multiphase images were identified by the sonographer by observing the enhancement pattern within the LNs after the administration of an ultrasound contrast agent according to the European Federation of Societies for Ultrasound in Medicine and Biology guidelines (19). Finally, all the images were converted to black and white and imported into ITK-SNAP software (www.itksnap.org) for region of interest (ROI) delineation. After segmentation, high/low-pass wavelet filters and Laplacian-Gaussian filters were used to preprocess the original image. Besides, z score was applied to normalize the images and B-spline interpolation was applied to resample the images to the same resolution. The radiomics features were extracted with the PyRadiomic platform. There were 1,317 quantitative radiomics features, including first-order features, shape features, greyscale co-occurrence matrix (GLCM), grey-level size zone matrix (GLSZM), grey-level run length matrix (GLRLM), grey-level distance-zone matrix (GLDM), and neighbourhood grey-tone difference matrix (NGTDM). Intraobserver and interobserver consistency were confirmed by using intraclass and interclass correlation coefficients (ICCs) to ensure the reproducibility of image segmentations. Feature selection was based on the analysis of variance (ANOVA).

Model establishment and evaluation

Since the dependent variables in our study were disordered multiclassification data (different types of lymphadenopathy), multiclassification logistic regression (LR) analysis was conducted to explore the relationship between the conventional features and the lymphadenopathies. The features associated with lymphadenopathies in uni- and multivariable analysis (P<0.05) were used to build model 1.

Several types of classifiers, including decision tree (DT), adaptive boosting (AdaBoost), linear support vector classification (SVC), random forest (RF) and LR, were assessed. The classifier that balanced complexity and accuracy in both the training cohort and the internal testing cohort was applied to establish model 2. Finally, by integrating the output of model 1 and model 2 via uni- and multivariable LR, the combined model (model 3) was established.

The receiver operating characteristic (ROC) curves of the three models were plotted to quantify the clinical utilities of the three models. The performances of the models were compared by the confusion matrix. The usefulness of the models was assessed in different threshold probabilities by decision curve analysis (DCA). The flowchart of the study is presented in Figure 2.

Figure 2 Schematic of contrast-enhanced ultrasound-based multiphase radiomics for the classification of superficial suspected lymph nodes. ROI, region of interest; GLCM, greyscale co-occurrence matrix; GLSZM, grey-level size zone matrix; GLRLM, grey-level run length matrix; GLDM, grey-level distance-zone matrix; NGTDM, neighbourhood grey-tone difference matrix; ANOVA, analysis of variance; LR, logistic regression; LNs, lymph nodes.

Statistical analysis

Statistical analysis was performed with SPSS (version 26.0), MedCalc (vision 19.5.6), and R software (version 4.0.2). The Kolmogorov-Smirnov test was used to evaluate the normality of the data. Continuous variables that were nonnormally distributed were expressed as medians (interquartile ranges). Categorical variables were expressed as frequencies and percentages. Group differences among the three categories were evaluated by the Kruskal-Wallis H test for continuous variables and by the Chi-squared test for categorical variables. To identify statistically significant features, both univariate and multivariable analyses were performed. The odds ratio (OR) of the included features were calculated. The DeLong test was applied to compare the areas under the curves (AUCs) of different models among the three lymphadenopathies. F1 score was used to quantify the accuracy of lymphadenopathy classification, calculated by the formulation: Score_F1 = 2 × (precision × recall)/(precision + recall). Two-tailed P values ≤0.05 represented that the difference was statistically significant.

Results

Patient characteristics

A total of 426 patients with superficial suspected LNs were finally included in this study. Of these, 317 cases from hospital 1 (Shaoxing People’s Hospital) were assigned to the training cohort (n=190) and the internal testing cohort (n=127) using a random sampling method, and 109 cases from hospitals 2 (Xiamen Hospital, Beijing University of Chinese Medicine) and 3 (The First Affiliated Hospital of Ningbo University) were used as the external testing cohort. There were 139 patients (32.6%) with benign LNs, 110 patients (25.8%) with lymphoma, and 177 patients (41.5%) with metastatic LNs in our population. In the training cohort and internal testing cohort, the age of the patients was slightly older than that in the external testing cohort [61.00 (54.00–71.00) and 65.00 (57.00–75.00) vs. 50.00 (32.00–65.00) years, P<0.01]. In the training cohort, there were 63 patients (33.2%) with benign LNs, 49 patients (25.8%) with lymphoma, and 78 patients (41.1%) with metastatic LNs. Eighty patients (42.1%) were female, and 109 (57.4%) were in the left. In the internal testing cohort, there were 42 patients (33.1%) with benign LNs, 32 patients (25.2%) with lymphoma, and 53 patients (41.7%) with metastatic LNs. Sixty-two patients (48.8%) were female, and 71 (55.9%) exhibited a left location of the suspected nodes. In the external testing cohort, there were 34 patients (31.2%) with benign LNs, 29 patients (26.6%) with lymphoma, and 46 patients (42.2%) with metastatic LNs. Fifty-seven patients (52.3%) were female, and 53 (48.6%) had a left location of the suspected nodes. There were no significant differences in gender (P=0.201), location (P=0.326), or proportions of the three classifications (P=0.997) between the three groups (Table 1).

Conventional features: model 1

The ultrasound features of different lymphadenopathies in our study are summarized in Table 2. The signs of unclear boundary and unclear hilum were more obvious in lymphoma and metastatic LNs than in benign LNs [14 (12.7%) and 28 (15.8%) vs. 4 (2.9%) in unclear boundary; 87 (79.1%) and 124 (70.1%) vs. 71 (51.1%) in unclear hilum, all P<0.01]. Additionally, the presence of peripheral vascularity was more commonly seen in lymphoma than in benign LNs and metastatic LNs [58 (52.7%) vs. 30 (21.6%) and 80 (45.2%), P<0.01]. Concerning CEUS features, hyperechoic enhancement was more significant in lymphoma and metastatic LNs than in benign LNs [89 (80.9%) and 127 (71.8%) vs. 76 (54.7%), P<0.01]. More metastatic LNs were enhanced heterogeneously [134 (75.7%)], with the presence of ring enhancement [21 (11.9%)] and perfusion defects [45 (25.4%)]. In addition, the contrast enhanced features of fast-in and slow-out could be found in metastatic LNs. The typical ultrasonographic performance of benign LNs, lymphoma, and metastatic LNs is presented in Figure 3.

Figure 3 Typical ultrasonographic performance of benign LNs, lymphoma, and metastatic LNs. In the first column are the blood flows of the LNs that evaluated by SMI. In the second column is the prevascular phase of the contrast-enhance ultrasound images of the LNs. In the third column is the vascular phase of the contrast-enhance ultrasound images of the LNs. In the fourth column is the postvascular phase of the contrast-enhance ultrasound images of the LNs. In the fifth column are the histological findings of the LNs (hematoxylin-eosin staining, magnification 200×). LNs, lymph nodes; SMI, superb microvascular imaging.

Multiclassification LR was performed by including the statistically significant variables (P<0.05) from the analysis. The results indicated that unclear hila, peripheral vascularity present, and hyperechoic enhancement intensity were independent features related to lymphoma for differentiating them from benign LNs [β=0.95, OR =2.585, 95% confidence interval (CI): 1.139–5.868, P=0.023; β=1.115, OR =3.050, 95% CI: 1.456–6.388, P=0.003; and β=0.908, OR =2.478, 95% CI: 1.154–5.322, P=0.020, respectively]. Additionally, age, calcification present, heterogeneous enhancement, faster wash-in, and slower wash-out were independent features related to metastatic LNs for differentiating them from benign LNs (β=0.041, OR =1.042, 95% CI: 1.014–1.072, P=0.003; β=2.110, OR =8.246, 95% CI: 1.465–46.421, P=0.017; β=1.189, OR =3.284, 95% CI: 1.562–6.906, P=0.002; β=−0.072; OR =0.931, 95% CI: 0.888–0.976, P=0.003; and β=0.041, OR =1.042, 95% CI: 1.022–1.062, P<0.01, respectively) (Table 3).

The constructed model 1 showed that in the training cohort, the mean AUC for differentiating lymphadenopathy was 0.818, the mean accuracy rate (ACC) was 64.74%, the mean sensitivity was 62.70%, and the mean specificity was 82.23%. In the internal testing cohort, the mean AUC for differentiating the lymphadenopathy was 0.799, the mean ACC was 61.42%, the mean sensitivity was 58.05%, and the mean specificity was 79.97%. In the external testing cohort, the mean AUC for differentiating the lymphadenopathy was 0.765, the mean ACC was 60.55%, the mean sensitivity was 60.42%, and the mean specificity was 80.26% (Table 4).

Multiphase radiomics features: model 2

The primary aims of our study is to investigate the quantitative, algorithm-derived radiomic model, so ICCs were applied to confirm the intra/interobserver consistency of the multiphase radiomics features. Features with ICCs lower than 0.80 were deleted in the model construction. After applying the ANOVA F value algorithm in the training cohort, a total of twenty radiomics features were selected as candidates for the LN classification, including the values of the first three-order wavelet and the variations in texture features between the three phases of the CEUS images. The radiomics features included (I) the first-order features: features that described the pixel intensity and distribution, such as first-order and first-order after wavelet decomposition; (II) the shape features: features that described the two-dimensional size and shape of the ROIs, such as original_shape _maximum; and (III) the texture features: features that described tiny differences in the images, such as GLRLM, GLSZM, GLDM and their features after wavelet decomposition, which can be seen in Table S4.

After evaluating the performance of DT, AdaBoost, linear SVC, RF and LR, we identified LR as the most appropriate approach to build the radiomic model (model 2). As shown in Table S5, RF performed better in the training cohort but not as well in the testing cohort, which indicated overfitting. After comprehensively evaluating the performance of the classifiers by metrics like AUC, ACC, sensitivity, specificity, negative predictive value (NPV), positive predictive value (PPV), and F1-score, we identified LR as the most appropriate approach for differentially diagnosing lymphadenopathies.

The diagnostic efficiency of multiphase radiomics features based on CEUS for different lymphadenopathies is compared in Table 5. The diagnostic efficiency of the combined phase was superior to that of the prevascular phase, vascular phase, and postvascular phase (AUC: 0.838, ACC: 70%, sensitivity: 67.39%, specificity: 84.58% in the training cohort; AUC: 0.739, ACC: 64.57%, sensitivity: 62.54%, specificity: 82.05% in the internal testing cohort; and AUC: 0.733, ACC: 58.72%, sensitivity: 57.45%, specificity: 79.20% in the external testing cohort). The performance comparison of different lymphadenopathies in multiphase radiomics features in the three cohorts is also provided in Table S6.

Combined features: model 3

The formulas of model 3 that assess the risk probability of each type of lymphadenopathy are available in Appendix 1. Combined with both conventional features and multiphase radiomic features, the diagnostic efficiency of model 3 was improved (AUC improved from 0.818 to 0.943 in the training cohort, from 0.799 to 0.823 in the internal testing cohort, and from 0.765 to 0.785 in the external testing cohort, respectively; ACC improved from 64.74% to 81.05% in the training cohort, from 61.42% to 66.93% in the internal testing cohort, and from 60.55% to 65.14% in the external testing cohort, respectively; sensitivity improved from 62.70% to 80% in the training cohort, from 58.05% to 64.18% in the internal testing cohort, and from 60.42% to 64.46% in the external testing cohort, respectively; and specificity improved from 82.23% to 90.43% in the training cohort, from 79.97% to 82.80% in the internal testing cohort, and from 80.26% to 82.57% in the external testing cohort, respectively). The addition of conventional ultrasound features to the radiomic models could improve the performance.

Performance of model classification

The ROC curves of models 1, 2, and 3 are shown in Figure 4. As presented in Table S7, in the training cohort, model 3 outperformed model 1 and model 2 in diagnosing the three types of lymphadenopathy. In the internal testing cohorts, model 3 showcased enhanced efficacy in diagnosing lymphoma and metastatic LNs compared to model 1 and model 2, while there were no significant differences among the three models in diagnosing benign LNs. In the external testing cohort, model 3 demonstrated improved indicative efficacy in diagnosing metastatic LNs, while no significant differences were observed among the three models in diagnosing benign LNs and lymphomas. Confusion matrices of the three models, plotted by using R software, are presented in Figure 5. The false-positive and false-negative rates in model 3 were lower than those in model 1 and model 2 in the training cohort, internal testing cohort, and external testing cohort. DCA showed that model 3 had better performance than both model 1 and model 2, as shown in Figure 6. These results indicated that the combined model was more valuable to doctors in clinical practice.

Figure 4 Classification performances of the different models in the training cohort, internal testing cohort, and external testing cohort. Plots show the receiver operating characteristic curves of model 1, model 2, and model 3, in benign LNs (A-C), lymphoma (D-F) and metastatic LNs (G-I) in the training cohort, internal testing cohort, and external testing cohort, respectively. Model 1, the conventional feature model; model 2, the multi-phase radiomics model; model 3, the combined feature model. LNs, lymph nodes; AUC, area under curve.

Figure 5 Confusion matrices of different models. (A-C) Confusion matrices of model 1 in training cohort, internal testing cohort, and external testing cohort. (D-F) Confusion matrices of model 2 in training cohort, internal testing cohort, and external testing cohort. (G-I) Confusion matrices of model 3 in training cohort, internal testing cohort, and external testing cohort. The x-axes represent the predicted labels, and the y-axes represent the true labels. The false-positive and false-negative rates in model 3 were lower than model 1 in all the cohorts. Model 1, the conventional feature model; model 2, the multiphase radiomics model; model 3, the combined feature model; LNs, lymph nodes.

Figure 6 Decision curves analysis of different lymphadenopathies. (A-C) Decision curves of benign LNs in training cohort, internal testing cohort, and external testing cohort. (D-F) Decision curves analysis of lymphoma in training cohort, internal testing cohort, and external testing cohort. (G-I) Decision curves analysis of metastatic LNs in training cohort, internal testing cohort, and external testing cohort. The x-axes represent the threshold probability, and the y-axes represent the net benefit. Model 1, the conventional feature model; model 2, the multi-phase radiomics model; model 3, the combined feature model. LNs, lymph nodes.

Discussion

In our multicentered study, we used CEUS-based multiphase radiomics as a noninvasive imaging method for the classification of lymphadenopathy. We found that the diagnostic efficiency of the combined phase was superior to that of the prevascular phase, vascular phase, and postvascular phase, which implied that compared with single-phase images, multiphase images contained more information about intra LN heterogeneity. In additional, the combination of conventional features and radiomic features could classify lymphadenopathy well, and it performed better than conventional feature model and radiomic model alone.

Accurate differential diagnosis of lymphadenopathy is pivotal for disease diagnosis and further treatment. Pathological biopsy is the gold standard but may be associated with some complications as it is invasive. For LNs with a rich blood supply, the procedure of biopsy may be accompanied by a risk of hematoma. Besides, biopsy of metastatic LNs may induce the risk of needle tract metastases. Traditional visual assessment is sometimes confounded because there may be overlap in the imaging features among different types of lymphadenopathy (3). Benign LNs often maintain a well-defined, rounded or ovoid shape with smooth contours. The echogenicity of benign LNs are homogeneous and isoechoic or of slightly hypoechoic appearance compared to adjacent fatty tissue, reflecting the normal structure without infiltration or disruption. Correspondingly, suspicious ultrasound features of malignant LNs include irregular shape, indistinct margins, microcalcifications, cystic areas, or peripheral vascularity. CEUS is an imaging modality that provides qualitative and quantitative information about the blood flow perfusion of the organs (20,21). To better identify the independent factors used to differentiate lymphoma and metastatic LNs from benign LNs, a multiclassification LR model (model 1) that based on clinical information and conventional ultrasound features was established in our study. It was revealed that unclear hila, peripheral vascularity present, and hyperechoic enhancement were independent features related to lymphoma for differentiating them from benign LNs. Lymphoma described in the literature has the characteristics of lymphatic hila structure destruction, dense fibrous envelope, and peripheral blood vessels (22), which are consistent with our ultrasound features. In addition, the presence of many immature microvessels within lymphoma may induce blizzard-like enhancements in the CEUS images (23). Age, calcification present, heterogeneous enhancement, faster wash-in, and slower wash-out were found to be independent features related to metastatic LNs for differentiating them from benign LNs. As presented in our results, most metastatic LNs (75.7%) revealed a heterogeneous enhancement pattern on CEUS images, which may be because immature neovascularization and necrotic areas impede the distribution of the contrast agent and lead to perfusion defects. This was in concordance with previous findings (24,25). In addition, wash-in time and wash-out time support quantitative evaluation of the time period during which the contrast agent fills the lesions and the time period during which the contrast agent fades. We found that the metastatic LNs showed a fast-in and slow-out enhancement pattern, which could be explained by the fact that neovascularization spread within the metastatic LNs leads to rapid perfusion and poor drainage (26).

Although there were some typical characteristic imaging findings within different types of lymphadenopathy, no sonographic feature was sensitive enough for the accurate classification (26). The combination with the clinical information and multimodal ultrasound features at the same time may help in the classification of lymphadenopathy (27). Nevertheless, comprehensive analysis of clinical information and multimodal image features poses a challenge to clinicians (28).

Different from the limitations of the conventional features in the classification, radiomics and machine learning may offer the potential for the noninvasive classification of lymphadenopathy (29,30). Our study had several strengths compared with previous studies. First, our study developed a CEUS-based radiomics model to classify different types of lymphadenopathy, which contained not only conventional ultrasound features, but also blood flow perfusion information of the LNs provided by CEUS (2,12,13,31). Second, we integrated phase information from longitudinal ultrasound images, which showed superior performance in the classification compared with single-phase images (32,33). Although researchers in previous studies reported that morphological features were associated with the disease progression (34), only shape features were associated with LN classification in our study. Images that relied on anatomy were inadequate to demonstrate the internal characteristics of LNs, the incorporated shape features could enable better decision-making for the classification (35). We also found that texture features and wavelet features gained by decomposing the original CEUS images could reflect the tiny differences presented in the images (10,36). The performance of the combined phase was superior to that of the single-phase (with AUCs of 0.838, 0.739, and 0.733; ACCs of 70%, 64.57%, and 58.72%; sensitivities of 67.39%, 62.54%, and 57.45%; specificities of 84.58%, 82.05%, and 79.20% in the training cohort, internal testing cohort, and external testing cohort respectively), which implied that multiphase images may contain more information about intra LN heterogeneity (33). Third, compared with conventional ultrasound images, the contrast-enhanced agent makes the LNs more distinct from the surrounding tissue, which makes the delineation of ROIs and image segmentation more accurate. As presented in our study, model 3 of the aggregated approach presents better performance than model 1 and model 2. This improvement can be attributed to the integration of advanced machine learning algorithms and novel imaging techniques, which enhance the robustness and reliability of the classification results (37). In addition, the aggregation approach has better generalization performance in diagnosing metastatic LNs, which could help in distinguishing metastatic LNs from benign or lymphomas in the real world. Confusion matrices revealed that the false-positive and false-negative rates in model 3 were lower than those in model 1 and model 2 in the training cohort, internal testing cohort, and external testing cohort. These phenomena indicated that more dramatic changes in the LNs were reflected on multiphase ultrasound images, which enabled LNs to be powerfully classified by the combined radiomics model.

Based on the purpose and content of our research, we used holdout validation rather than K-fold cross-validation. Although K-fold validation can improve the robustness of the models, but during the procedure of cross validation, the clinical parameters will be regrouped and redistributed, resulting in difficulties in combining the radiomic model with the clinical model. Meanwhile, with regard to data split, the reason to the choice of train/test split as 60/40 is for the provision of sufficient training data and adequate internal testing data. In our study, the LNs in both the training cohort and the testing cohort needed to be made up by three kinds of lymphadenopathy (benign LNs, lymphoma, and metastasis LNs). If the split of 80/20 is applied, the number of each lymphadenopathy in the internal testing cohort may be insufficient. Using a larger testing set (40% in this study) could lead to more reliable performance evaluation metrics (13). What’s more, to ensure that the distribution of classes was maintained across the splits, a stratified sampling approach was used to create the splits.

In our study, ANOVA was used for the feature selection. The reason to use ANOVA for the feature selection is that ANOVA is useful when dealing with categorical or group-based data. It allows for comparisons between multiple groups simultaneously, assessing whether there are significant differences in the means of the outcome variable across these groups. About the establishment of model 3, combining clinical model and radiomic model to create a combined model is a common approach in medical research and can often lead to improved predictive performance. The combined model can leverage the complementary information from both clinical variables and imaging data (38-40). In our study, clinical and radiomic models were trained separately, and then their outputs were combined by LR. As shown in our result, this integration allows for a more comprehensive and accurate prediction of the outcomes.

Moreover, we considered that the instrument differences in ultrasonography might influence the model performance, so we collected ultrasound images acquired with different ultrasound instruments in both the training and testing cohorts. As shown in Tables S2,S3, although data collected by the different instruments were not represented equally in our study, the data of misclassified LNs were indeed not collected more often with particular instruments, which imply that the methods may be system-independent. Besides, validation of the models on external datasets acquired from different scanners was performed to assess their generalizability and performance in real-world scenarios. After preprocessing and postprocessing the images, radiomics features were extracted and put into the models. The combined model presented good classification performance in the external testing cohort, with AUCs of 0.696, 0.805 and 0.834 in benign LNs, lymphoma and metastatic LNs, respectively. It was also found that the combined phase was much better for the external testing cohort compared to the other three phases whereas all four phases were similar for the training and internal testing cohorts. The reason may be explained by the fact that the external testing cohort has a greater variability compared to the training and internal testing cohorts, which can provide a more comprehensive representation of the target population and allow the combined phase to capture additional informative features that were not as prominent or discernible in the less diverse training and internal testing cohorts. Good specificities [88.00% (benign LNs), 78.75% (lymphoma) and 80.95% (metastatic LNs)] were observed in the external testing cohort.

There are some limitations in our study. First, the radiomics model was established using retrospective data. Prospective data would provide better evidence for clinical application. Second, imbalanced ratios of different types of lymphadenopathy may influence the clinical utility of the radiomics model, especially given the relatively small proportion of lymphoma and the lack of tuberculosis patients in our study population. Third, the radiomics features were only extracted from images in which contrast agent arrived at the LN, reached peak intensity and washed out. The feature analysis from the CEUS videos may supply more complete LN blood flow perfusion information.

Conclusions

To conclude, combined feature model was constructed with CEUS images, enabling data from the prevascular phase, vascular phase, and postvascular phase of CEUS to be used to classify different types of lymphadenopathy. For patients with superficial suspected LNs, this model can help clinicians to make a decision on the LN type noninvasively and choose appropriate treatments. Studies with external validation could provide better clinical evidence for our diagnostic models and help determine its clinical application value in wider populations.

Acknowledgments

Funding: This work was supported by Shaoxing Medical Key Discipline (No. 2019SZD05), Shaoxing People’s Hospital Youth Research Fund (No. 2022YB06), and School management project of Fujian University of Chinese Medicine (No. XB2021109).

Footnote

Reporting Checklist: The authors have completed the TRIPOD reporting checklist. Available at https://qims.amegroups.com/article/view/10.21037/qims-23-1182/rc

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at https://qims.amegroups.com/article/view/10.21037/qims-23-1182/coif). All authors report that this work was supported by Shaoxing Medical Key Discipline (No. 2019SZD05), Shaoxing People’s Hospital Youth Research Fund (No. 2022YB06), and School management project of Fujian University of Chinese Medicine (No. XB2021109). The authors have no other conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions relating to the accuracy or integrity of any part of the work are appropriately investigated and resolved. The study was conducted in accordance with the Declaration of Helsinki (as revised in 2013). Approval for this retrospective cohort study was granted by the Ethics Committee of Shaoxing People’s Hospital (No. 2022-097-Y01). Informed written consent to participate was waived given the retrospective nature of the work.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. Bandoh N, Goto T, Akahane T, Ohnuki N, Yamaguchi T, Kamada H, Harabuchi Y, Tanaka S, Nishihara H. Diagnostic value of liquid-based cytology with fine needle aspiration specimens for cervical lymphadenopathy. Diagn Cytopathol 2016;44:169-76. [Crossref] [PubMed]
2. Hoerig C, Wallace K, Wu M, Mamou J. Classification of Metastatic Lymph Nodes In Vivo Using Quantitative Ultrasound at Clinical Frequencies. Ultrasound Med Biol 2023;49:787-801. [Crossref] [PubMed]
3. Zhu Y, Meng Z, Fan X, Duan Y, Jia Y, Dong T, Wang Y, Song J, Tian J, Wang K, Nie F. Deep learning radiomics of dual-modality ultrasound images for hierarchical diagnosis of unexplained cervical lymphadenopathy. BMC Med 2022;20:269. [Crossref] [PubMed]
4. Mahieu R, Krijger GC, Ververs FFT, de Roos R, de Bree R, de Keizer B. [68Ga]Ga-tilmanocept PET/CT lymphoscintigraphy: a novel technique for sentinel lymph node imaging. Eur J Nucl Med Mol Imaging 2021;48:963-5.
5. Cheung YC, Chen SC, Hsieh IC, Lo YF, Tsai HP, Hsueh S, Yen TC. Multidetector computed tomography assessment on tumor size and nodal status in patients with locally advanced breast cancer before and after neoadjuvant chemotherapy. Eur J Surg Oncol 2006;32:1186-90. [Crossref] [PubMed]
6. Kuijs VJ, Moossdorff M, Schipper RJ, Beets-Tan RG, Heuts EM, Keymeulen KB, Smidt ML, Lobbes MB. The role of MRI in axillary lymph node imaging in breast cancer patients: a systematic review. Insights Imaging 2015;6:203-15. [Crossref] [PubMed]
7. Marino MA, Avendano D, Zapata P, Riedl CC, Pinker K. Lymph Node Imaging in Patients with Primary Breast Cancer: Concurrent Diagnostic Tools. Oncologist 2020;25:e231-42. [Crossref] [PubMed]
8. Cui Y, Jiang Y, Deng X, Long W, Liu B, Fan W, Li Y, Zhang X. (18)F-FDG PET-Based Combined Baseline and End-Of-Treatment Radiomics Model Improves the Prognosis Prediction in Diffuse Large B Cell Lymphoma After First-Line Therapy. Acad Radiol 2023;30:1408-18. [Crossref] [PubMed]
9. Loh Z, Hawkes EA, Chionh F, Azad A, Chong G. Use of Ultrasonography Facilitates Noninvasive Evaluation of Lymphadenopathy in a Lymph Node Diagnostic Clinic. Clin Lymphoma Myeloma Leuk 2021;21:e179-84. [Crossref] [PubMed]
10. Huang Y, Zhu T, Zhang X, Li W, Zheng X, Cheng M, Ji F, Zhang L, Yang C, Wu Z, Ye G, Lin Y, Wang K. Longitudinal MRI-based fusion novel model predicts pathological complete response in breast cancer treated with neoadjuvant chemotherapy: a multicenter, retrospective study. EClinicalMedicine 2023;58:101899. [Crossref] [PubMed]
11. Zheng X, Yao Z, Huang Y, Yu Y, Wang Y, Liu Y, Mao R, Li F, Xiao Y, Wang Y, Hu Y, Yu J, Zhou J. Deep learning radiomics can predict axillary lymph node status in early-stage breast cancer. Nat Commun 2020;11:1236. [Crossref] [PubMed]
12. Chen Y, Jiang J, Shi J, Chang W, Shi J, Chen M, Zhang Q. Dual-mode ultrasound radiomics and intrinsic imaging phenotypes for diagnosis of lymph node lesions. Ann Transl Med 2020;8:742. [Crossref] [PubMed]
13. Liu Y, Chen J, Zhang C, Li Q, Zhou H, Zeng Y, Zhang Y, Li J, Xv W, Li W, Zhu J, Zhao Y, Chen Q, Huang Y, Li H, Huang Y, Yang G, Huang P. Ultrasound-Based Radiomics Can Classify the Etiology of Cervical Lymphadenopathy: A Multi-Center Retrospective Study. Front Oncol 2022;12:856605. [Crossref] [PubMed]
14. Yu H, Liang X, Zhang M, Fan Y, Wang G, Wang S, Sun J, Zhang J. LN-Net: Perfusion Pattern-Guided Deep Learning for Lymph Node Metastasis Diagnosis Based on Contrast-Enhanced Ultrasound Videos. Ultrasound Med Biol 2023;49:1248-58. [Crossref] [PubMed]
15. Gao L, Li X, Xia Y, Liu R, Liu C, Shi X, Wu Y, Ma L, Jiang Y. Large-Volume Lateral Lymph Node Metastasis Predicts Worse Prognosis in Papillary Thyroid Carcinoma Patients With N1b. Front Endocrinol (Lausanne) 2021;12:815207. [Crossref] [PubMed]
16. Swerdlow SH, Campo E, Pileri SA, Harris NL, Stein H, Siebert R, Advani R, Ghielmini M, Salles GA, Zelenetz AD, Jaffe ES. The 2016 revision of the World Health Organization classification of lymphoid neoplasms. Blood 2016;127:2375-90. [Crossref] [PubMed]
17. Ramsay AD. 30 years of lymph node pathology: biomarkers and other advances. Appl Immunohistochem Mol Morphol 2013;21:103-9. [Crossref] [PubMed]
18. Xiao L, Zhou J, Tan W, Liu Y, Zheng H, Wang G, Zheng W, Pei X, Yang A, Liu L. Contrast-enhanced US with Perfluorobutane to Diagnose Small Lateral Cervical Lymph Node Metastases of Papillary Thyroid Carcinoma. Radiology 2023;307:e221465. [Crossref] [PubMed]
19. Sidhu PS, Cantisani V, Dietrich CF, Gilja OH, Saftoiu A, Bartels E, et al. The EFSUMB Guidelines and Recommendations for the Clinical Practice of Contrast-Enhanced Ultrasound (CEUS) in Non-Hepatic Applications: Update 2017 (Short Version). Ultraschall Med 2018;39:154-80. [Crossref] [PubMed]
20. Peng Q, Niu C, Zhang M, Peng Q, Chen S. Sonographic Characteristics of Papillary Thyroid Carcinoma with Coexistent Hashimoto's Thyroiditis: Conventional Ultrasound, Acoustic Radiation Force Impulse Imaging and Contrast-Enhanced Ultrasound. Ultrasound Med Biol 2019;45:471-80. [Crossref] [PubMed]
21. Zhu Y, Jia Y, Pang W, Duan Y, Chen K, Nie F. Ultrasound contrast-enhanced patterns of sentinel lymph nodes: predictive value for nodal status and metastatic burden in early breast cancer. Quant Imaging Med Surg 2023;13:160-70. [Crossref] [PubMed]
22. Khanna R, Sharma AD, Khanna S, Kumar M, Shukla RC. Usefulness of ultrasonography for the evaluation of cervical lymphadenopathy. World J Surg Oncol 2011;9:29. [Crossref] [PubMed]
23. Cui NY, Gong XT, Tian YT, Wang Y, Zhang R, Liu MJ, Han J, Wang B, Yang D. Contrast-enhanced ultrasound imaging for intestinal lymphoma. World J Gastroenterol 2021;27:5438-47. [Crossref] [PubMed]
24. Wei Y, Yu MA, Niu Y, Hao Y, Di JX, Zhao ZL, Cao XJ, Peng LL, Li Y. Combination of Lymphatic and Intravenous Contrast-Enhanced Ultrasound for Evaluation of Cervical Lymph Node Metastasis from Papillary Thyroid Carcinoma: A Preliminary Study. Ultrasound Med Biol 2021;47:252-60. [Crossref] [PubMed]
25. Chen L, Chen L, Liu J, Wang B, Zhang H. Value of Qualitative and Quantitative Contrast-Enhanced Ultrasound Analysis in Preoperative Diagnosis of Cervical Lymph Node Metastasis From Papillary Thyroid Carcinoma. J Ultrasound Med 2020;39:73-81. [Crossref] [PubMed]
26. Fang F, Gong Y, Liao L, Ye F, Zuo Z, Li X, Zhang Q, Tang K, Xu Y, Zhang R, Chen S, Niu C. Value of Contrast-Enhanced Ultrasound for Evaluation of Cervical Lymph Node Metastasis in Papillary Thyroid Carcinoma. Front Endocrinol (Lausanne) 2022;13:812475. [Crossref] [PubMed]
27. Franzen A, Günzel T, Buchali A, Coordes A. Etiologic and differential diagnostic significance of tumor location in the supraclavicular fossa. Laryngoscope 2018;128:646-50. [Crossref] [PubMed]
28. Meng Z, Zhu Y, Pang W, Tian J, Nie F, Wang K. MSMFN: An Ultrasound Based Multi-Step Modality Fusion Network for Identifying the Histologic Subtypes of Metastatic Cervical Lymphadenopathy. IEEE Trans Med Imaging 2023;42:996-1008. [Crossref] [PubMed]
29. Lin M, Tang X, Cao L, Liao Y, Zhang Y, Zhou J. Using ultrasound radiomics analysis to diagnose cervical lymph node metastasis in patients with nasopharyngeal carcinoma. Eur Radiol 2023;33:774-83. [Crossref] [PubMed]
30. Li X, Yang L, Jiao X. Comparison of Traditional Radiomics, Deep Learning Radiomics and Fusion Methods for Axillary Lymph Node Metastasis Prediction in Breast Cancer. Acad Radiol 2023;30:1281-7. [Crossref] [PubMed]
31. Zhang W, Peng J, Zhao S, Wu W, Yang J, Ye J, Xu S. Deep learning combined with radiomics for the classification of enlarged cervical lymph nodes. J Cancer Res Clin Oncol 2022;148:2773-80. [Crossref] [PubMed]
32. Zhou H, Mao H, Dong D, Fang M, Gu D, Liu X, Xu M, Yang S, Zou J, Yin R, Zheng H, Tian J, Pan C, Fang X. Development and External Validation of Radiomics Approach for Nuclear Grading in Clear Cell Renal Cell Carcinoma. Ann Surg Oncol 2020;27:4057-65. [Crossref] [PubMed]
33. Uhm KH, Jung SW, Choi MH, Shin HK, Yoo JI, Oh SW, Kim JY, Kim HG, Lee YJ, Youn SY, Hong SH, Ko SJ. Deep learning for end-to-end kidney cancer diagnosis on multi-phase abdominal computed tomography. NPJ Precis Oncol 2021;5:54. [Crossref] [PubMed]
34. Liu Q, Jiang P, Jiang Y, Ge H, Li S, Jin H, Li Y. Prediction of Aneurysm Stability Using a Machine Learning Model Based on PyRadiomics-Derived Morphological Features. Stroke 2019;50:2314-21. [Crossref] [PubMed]
35. Xie C, Hu Y, Han L, Fu J, Vardhanabhuti V, Yang H. Prediction of Individual Lymph Node Metastatic Status in Esophageal Squamous Cell Carcinoma Using Routine Computed Tomography Imaging: Comparison of Size-Based Measurements and Radiomics-Based Models. Ann Surg Oncol 2022;29:8117-26. [Crossref] [PubMed]
36. He C, Xie D, Fu LF, Yu JN, Wu FY, Qiu YG, Xu HW. A nomogram based on radiomics intermuscular adipose analysis to indicate arteriosclerosis in patients with newly diagnosed type 2 diabetes. Front Endocrinol (Lausanne) 2023;14:1201110. [Crossref] [PubMed]
37. Wang X, Wan Q, Chen H, Li Y, Li X. Classification of pulmonary lesion based on multiparametric MRI: utility of radiomics and comparison of machine learning methods. Eur Radiol 2020;30:4595-605. [Crossref] [PubMed]
38. Xia TY, Zhou ZH, Meng XP, Zha JH, Yu Q, Wang WL, Song Y, Wang YC, Tang TY, Xu J, Zhang T, Long XY, Liang Y, Xiao WB, Ju SH. Predicting Microvascular Invasion in Hepatocellular Carcinoma Using CT-based Radiomics Model. Radiology 2023;307:e222729. [Crossref] [PubMed]
39. Wang T, Gong J, Li Q, Chu C, Shen W, Peng W, Gu Y, Li W. A combined radiomics and clinical variables model for prediction of malignancy in T2 hyperintense uterine mesenchymal tumors on MRI. Eur Radiol 2021;31:6125-35. [Crossref] [PubMed]
40. Chee CG, Yoon MA, Kim KW, Ko Y, Ham SJ, Cho YC, Park B, Chung HW. Combined radiomics-clinical model to predict malignancy of vertebral compression fractures on CT. Eur Radiol 2021;31:6825-34. [Crossref] [PubMed]