A quantitative heterogeneity analysis approach to molecular subtype recognition of breast cancer in dynamic contrast-enhanced magnetic imaging images from radiomics data

Introduction

Breast cancer refers to malignant tumors of the breast duct epithelium, which is a common malignant cancer in women between the ages of 45–55 years old and is a rising trend in China. Over the past decade, medical clinical consensus has been confirmed to divide breast cancer into four different subtypes based on histopathology, patient-driven genes, hormone receptor expression and cellular molecular status. The four molecular subtypes discussed in this paper are luminal A, luminal B, human epidermal growth factor receptor-2 overexpressing (HER-2) and basal-like. It is important that there are differences in the treatment of different types of breast cancer in clinical practice. Dynamic contrast-enhanced breast magnetic resonance imaging (DCE-MRI) is a good imaging technique that can show temporal information about the kinetics of the contrast agent in suspicious breast lesions as well as acceptable spatial resolution. However, the increasing number of medical images has led to greater difficulties for radiologists that can become fatigued, which leads to incorrect diagnoses. Computer-assisted diagnostic detection systems without human intervention can assist radiologists in the diagnosis of breast cancer and guide radiologists in subsequent treatment plans.

Individualized and systematic treatment programs have been developed for different molecular subtypes in clinical practice. However, it is important that there are differences in the treatment of different types of breast cancer in clinical practice, and there are obvious differences in the treatment response of different treatment options to individual patients. Cancer lesion heterogeneity has been verified by histological and genetic methods, and intratumor genetic heterogeneity has been studied and associated with adverse clinical outcomes (1). Breast cancer has a high degree of heterogeneity. The heterogeneity of the lesion area has the characteristics of synchronous dynamic change with time and lesion growth, and it is an important factor affecting the staging of breast cancer and poor prognosis (2).

This paper proposes an innovative method for breast cancer lesion detection and molecular subtype recognition that gives clinicians more knowledge for decisions regarding consequential treatment plans. The workflow of the presented method is shown in Figure 1. The method proposed in this paper consists of four steps: cancer lesion segmentation, heterogeneous region extraction, creation of a temporal bag of visual words and molecular subtype recognition model training and validation. Cancer lesion segmentation is used to obtain lesion locations by a deep learning model on the basis of lesions labeled by radiologists. The heterogeneous region extraction is carried out, and the precise boundary of the lesion is found by the method of regional growth. Clustering is conducted on the lesions to extract the heterogeneous subregions. A variety of visual and dynamic features are extracted based on the data of these heterogeneous regions. The feature dictionaries of heterogeneous regions with different time phases are trained according to time evolution. Finally, temporal bag of visual word (TBOVW) features are obtained, and a variety of classification models are used for classification and cross-validation to verify the effective role of the TBOVW model in molecular subtype recognition.

Figure 1 Workflow of presented computer aided detection system.

This work is subsequent to our previous research, which treated the lesion as a whole by radiomics analysis without considering the heterogeneous characteristics of breast cancer (3). This paper explores the impacts of heterogeneous subregions on molecular subtypes. Meanwhile, the lesion detection algorithm is newly applied in the automatic workflow. The main contributions of this article are as follows.

• A lesion heterogeneous subregion extraction algorithm based on clustering is proposed, and visual and dynamic radiomics features are extracted for each heterogeneous region to express different region characteristics.
• An improved bag of visual word (BOVW) model called TBOVW is proposed, which takes into account the temporal characteristics of each visual word, and each lesion is expressed as a bag of visual word sequences.
• Extensive experiments on clinical real-world datasets demonstrate the effectiveness of the proposed radiomics analysis method with quantitative heterogeneity by the TBOVW model.

Radiomics is proposed to refer to the extraction and analysis of large numbers of advanced quantitative imaging features with high throughput from medical images (3,4). Wang et al. presented a radiomic analysis method with 60 features to build a prediction model for recognizing malignant and benign tumors combined with medical characteristics (5). Many visual features are extracted to quantify the image intensity, shape and texture of lesions and tumor surroundings, which are associated with underlying gene expression patterns (6). It is also feasible to use radiomics methods to exploit normal liver features and make predictions for treatment-associated liver injury (7), as well as to distinguish malignant nodules from benign nodules (8).

Tumors exhibit genomic and phenotypic heterogeneity, which has prognostic significance and may influence the response to therapy (9). Genetic, epigenetic and phenomenological data support the existence of intratumor genetic heterogeneity in breast cancers (10,11). Region-based features were extracted by the proposed regionalization method to quantify intratumor heterogeneity on breast DCE-MRI (12). Banaie et al. proposed a method to help physicians determine the likelihood of malignancy in breast cancer using DCE-MRI images without biopsy (13). Quantitative heterogeneity analysis of breast cancer may enable precision medicine to differentiate luminal A and luminal B breast cancer molecular subtypes (14,15). Computer-extracted image phenotypes as well as dynamic features from tumor and background parenchymal enhancement were used to determine DCE-MRI characteristics discriminating among four molecular subtypes of breast cancer.

Precise lesion location in the image is an important step in molecular subtype recognition. Recently, most researchers have performed pixel-level segmentation of breast cancer lesions in DCE-MRI to determine the location of breast cancer lesions. The research is mainly divided into two categories based on traditional machine learning or deep learning (16,17). The first method extracts manual vision features with a priori knowledge, such as the hemodynamic characteristics of the lesion and the size of the lesion, and then uses a classifier to decide whether each pixel is a lesion or non-lesion (18,19). The second method is based on deep learning approaches. Fan proposed a 3D-Mask regional convolutional neural network for cancer lesion detection, with an improved accuracy of 0.93 compared to 2D-Mask. Molecular subtypes can also be classified by a specific convolutional neural networks (CNN) architecture that heuristically explores possible parameter combinations. This shows that computer-aided diagnosis systems provide substantial help in classifying molecular subtypes, improving overall typing accuracy. Molecular subtypes can also be classified by a specific CNN architecture that heuristically explores possible parameter combinations. The U-Net model is applied to obtain the lesion area (20). However, both methods require radiologists to mark the precise edges of breast cancer lesions, which is a time-consuming task.

Therefore, the original regions with lesion locations in DCE-MRI data are identified by two radiologists with more than ten years of clinical experience and are manually marked using a rectangle without the edges of the lesions. It is difficult to use traditional machine learning methods and U-Net-like deep learning methods to detect lesion edges on such image data. The fast region-based convolutional network (Faster R-CNN) algorithm has achieved quite good results in similar target detection tasks and does not require precise lesion edges and is used in this paper to obtain the precise area of cancer lesions (21,22). This work goes a step further on lesion heterogeneity and lesion temporal characteristics rather than learning from scratch, where only tumor patches were reported in (23,24). Computer-extracted image phenotypes show promise for high-throughput identification of breast cancer subtypes and may yield quantitative predictive features for advancing precision medicine (25,26). The differences between this paper and these existing research works are that the heterogeneity and time characteristics of the lesion are considered in this paper. The features of the heterogeneous subregions are segmented by the clustering method, and these features in time evolution are also considered.

There are several approaches to molecular typing, among which for glioma subtypes, a three-level machine learning model of multimodal MR reflectometry is proposed for their classification. There is also breast image classification by fine-tuning Inception-v3 convolutional neural networks. Based on this, a bootstrapped soft attention network is proposed that uses CNNs to perform additional supervision on the classification of images by first localizing the region of interest and then guiding the classification network. The molecular subtype diagnosis of breast cancer is imbalanced in distribution. Generally, luminal A and luminal B are the majority, and other data sizes are relatively small. Therefore, the existing work does not consider this issue as reported in (27). Holli-Helenius et al. applied MRI texture analysis in differentiating luminal A and luminal B breast cancer molecular subtypes (15). Zhu et al. applied neural networks trained on natural images with GoogLeNet, VGG, and CIFAR and fine-tuned them using tumor patches (28). A multisequence MRI-based radiomic model was used for preoperative assessment of MUC4 status in pancreatic ductal adenocarcinoma (29). In view of this unbalanced problem, this paper adopts the visual word bag method and balances the dictionary data of each kind of subtype training. However, the traditional bag of visual word model does not consider the features of word evolution. This paper adds the time feature to the bag of visual words extraction process, which forms a new temporal bag of visual words model.

Methods

Automatic computer-aided detection of breast cancer needs to perform two tasks, namely, detecting lesions and identifying molecular subtypes of breast cancer. The first task is divided into several steps, including breast region data preparation, Faster R-CNN training, and breast cancer lesion detection. The second task of molecular subtype recognition is achieved by three steps: heterogeneity region extraction, temporal bag of words creation and molecular subtype recognition.

The data are all cases with one or more malignant lesions. Malignant lesions mostly manifest as an internal interval enhancement mode represented in DCE-MRI images, which is called internal unevenness enhancement. The edge of the lesion is not clear. Therefore, the approximate location of each lesion in this dataset is labeled by experienced radiologists. The radiologists only mark the lesion locations in the images. Then, an automatic regional growth algorithm is applied to obtain the edge of the tumors. Different heterogeneity regions in the lesion are extracted based on the cluster analysis algorithm. The radiomics features are extracted for each heterogeneous region, including texture features, morphological features, statistical features and dynamic enhancement characteristics, and then the feature extraction process is applied to sublesion data of different time phases for the temporal bag of visual word model. Finally, classification methods, including logistic regression, support vector machine, random forest and gradient boosting decision tree classifiers, are used to conduct cross-validation and verify the performance of molecular subtype recognition.

Precise breast region extraction in DCE-MRI

First, we use threshold-based presegmentation for low-intensity areas, as shown in Figure 2. Then, the marked points of A, B and C are found to determine the horizontal and diagonal lines. Point A is on the middle and vertical line on the DCE-MRI image. Points B and C according to A determine a horizontal line that is generated between points B and C. Two lines are generated with a 25-degree angle along with the horizontal lines starting from points B and C. Therefore, the unrelated regions, such as the heart, which has a great interference effect on the lesion locating process, are removed, which significantly reduces the occurrence of false positives, and the breast regions are finally obtained.

Figure 2 The breast detection regions are below the green line. The dotted vertical line is used to find point A and a 1*5 mask under point A is added, and then the left and right breasts bottom lines are extended, respectively. If there are more than or equal to two pixels with a pixel value greater than 0, two vertical lines perpendicular to the horizontal line are found. The intersection point of the horizontal line and the vertical line is the final marking points B and C. The green horizontal lines are generated based on the two points B and C.

Faster R-CNN model training in DCE-MRI

Fast R-CNN is proposed to solve the RCNN shortcomings, solve the duplicate feature calculation problem of the target candidate region, and avoid the limitations of scaling images to a fixed resolution. To solve the problem of image fixed resolution input, an region of interest (ROI) pooling layer is proposed, the softmax classifier is used to classify multiple targets simultaneously, and the error function of the class judgment and target border return to the two peer output layers is fused to facilitate network training.

Faster R-CNN abandons the traditional sliding window and search methods and directly uses a region proposal network (RPN) to generate the detection frame, which greatly improves object detection speed. This model has two tasks: detecting the position of the target to obtain the bounding box and classifying the target to obtain the target’s category. The basic framework and network of Faster R-CNN used in this paper are shown in Figure 3 and Figure 4, respectively.

Figure 3 Faster R-CNN architecture for breast cancer detection in DCE-MRI. Faster R-CNN, fast region-based convolutional network; DCE-MRI, dynamic contrast-enhanced breast magnetic resonance imaging; ROI, region of interest.

Figure 4 Faster R-CNN network structure diagram. Faster R-CNN, fast region-based convolutional network; RPN, region proposal network; ROI, region of interest.

There are eight-time phases in DCE images, the first three of which are selected in this paper. The patient’s motion may cause artifacts in the latter phases. The input images of the Faster R-CNN framework include 3 channels, and their relative images are in the three-time phases of S0, S1 and S3. S0 is the image without contrast injection, and S1 and S3 are the images with contrast injection. The S3 lesion area is obvious. We tried the images at different time periods during the experiment and proved that the best results were obtained at S0, S1 and S3. Its output is a bounding box with cancer likelihood scores.

The shared convolution layers use VGG16 as a classification network to extract the input image features and obtain the feature map (30). The role of the RPN network is to find a predefined number of region proposals that may contain objects. Here, we use a 3×3 sliding window. The anchor box corresponds to 3 scales applied in this model, 16×16, 32×32, 64×64, and 3 aspects of 1:1, 1:2, 1:0.5. Finally, 9 anchors are obtained. Through the RPN layer, we obtain 300 regions with scores greater than 0.7 as the proposal lesions. The ROI pooling layer maps the properties to the corresponding position of the feature map according to the input image and resizes to a uniform size. Finally, the fixed-size feature map obtained by the ROI pooling layer is fully connected to perform classification and regression on breast cancer lesions. The Faster R-CNN used in this paper is a multitask loss function for end-to-end training. The loss function L is defined as follows.

L(p,u*,t,t*)=Lcls(p,u*)+λ[u*≥1]Lloc(t,t*)[1]

where L_cls is the classification loss function using a cross-entropy loss function. p is the probability of each proposal predicting a breast cancer lesion, and u* is the category label. L_loc represents a regression loss function using smooth L1 loss, where t is a vector representing the 4 parameterized coordinates of the predicted bounding box, and t* shows the coordinate vector of the ground-truth bounding box corresponding to the positive anchor. λ denotes balancing the weights of Lcls and Lloc, which are equal when they are equal to 10.

To avoid overfitting problems and improve the generalization ability of the network during the experiment, we enhanced the data of the sample by using the method of horizontally reversing the training data to expand the data volume to twice the size. The shared convolutional layer is initialized with a pretrained VGG16 model trained on the ImageNet benchmark. The other layers are initialized randomly. The paper uses the cross-validation method to find the best hyperparameters and then trains the model with the best hyperparameters. We divide the original dataset into 5 similarly sized subsets. Each time, the remaining 4 subsets are used as the training set and another subset is used as the test set. All slices from the same DCE-MRI are assigned to the same subset, which prevents the training and test sets from having the same data. The training set is used to train the model, and the validation set is used to evaluate the performance metrics of the trained model. The function of cross-validation is to try to use different training/validation sets to perform multiple sets of different training/validation of the model to deal with the problem that the individual test results are too one-sided and the training data are insufficient. For our data, there are relatively more 2D features, and the quantity of data decreases after extracting 3D features, which is not conducive to training. The parameters are set as follows: min-batch is equal to 128, an epoch is equal to 100, and the learning rate at the beginning of the experiment is set to 0.001 and then decreased by 0.1 times every 20 epochs. Faster R-CNN trains the RPN network and the Fast R-CNN network alternately in 4 steps during the training process to achieve end-to-end training (31). The first step is to separately train the RPN network with the initialized model, thereby generating a proposal bounding box. The second step is to use the initialized model to separately train and detect the Fast R-CNN network. The bounding box used in training comes from the RPN network in the first step. It is worth noting that at this time, the RPN network and the Fast R-CNN network have not yet shared the convolutional layer. In the third step, the parameters generated by the Fast R-CNN in the second step are used to initialize the RPN network. The shared convolution layer is fixed, and only the parameters of the RPN unique layer are fine-tuned. In the final step, the shared convolution layer is kept fixed, and the proposal output from the adjusted RPN in the third step is used as the input remaining parameters of the fine-tuning detection network Fast R-CNN. The preliminary results are obtained by the trained Faster R-CNN model to detect breast cancer lesions on different slices, which are shown in Figure 5.

Figure 5 Results of lesion detection on different image slices in DCE-MRI. The three points A-C represent the lesion area of the image. DCE-MRI, dynamic contrast-enhanced breast magnetic resonance imaging.

Breast cancer lesion detection in DCE-MRI

The true breast cancer lesion is detected on multiple image slices using the characteristics of three-dimensional DCE-MRI. However, those who do not show positive symptoms may be detected as positive for various reasons during the detection process and called false positives. In this way, the false positive lesions are removed if the candidate lesions are only detected on one or discontinuous slices. A fusion method is used to merge and delete the bounding box to determine the final breast cancer lesion.

First, we calculate the overlap ratio of the bounding box detected on any two slices. The overlap ratio is defined as the ratio of the intersection and union of the two bounding boxes. If the value of the overlap ratio is greater than the preset threshold, they are combined into one bounding box, and the bounding box with the largest score is taken. Second, the bounding box detected on only one slice is deleted. Finally, the merged ROI is compared with the standard set labeled by the doctor. If the overlap ratio between the bounding box of the candidate lesion and the bounding box of the true label is greater than or equal to 0.5, then the lesion is considered a true positive. Otherwise, this lesion was treated as a false positive.

The red box in Figure 4 represents the results of preliminary breast cancer tests. The red box represented by area A can be merged into a bounding box, which obtains the bounding box with the highest cancer likelihood score. Then, the bounding boxes represented by the separate B and C are removed. Finally, the A locations of breast cancer detected in DCE-MRI images should be the true lesion regions.

Heterogeneity region segmentation in breast cancer lesions

The breast cancer lesions are marked by previous lesion detection algorithms using a rectangle, as shown in Figure 6 (yellow rectangle), which is defined as the symbol M_s for the lesion data matrix. Since the lesions are annotated only with the approximate location, not the edge of the lesion, this paper first obtains the region with an accurate lesion boundary through the method of regional growth, and the lesion with a precise edge is defined as the symbol M_m with the same size as M_s. The edge of the lesion and the internal positions are set as 1, and the other background is 0 in M_m. The precise lesion data matrix of the lesion is defined as M_l=M_s&M_m.

Figure 6 Lesion region marked by radiologists and multicycle variation of DCE-MRI lesions. DCE-MRI, dynamic contrast enhanced breast magnetic resonance imaging.

Because of the different degrees of enhancement inside the lesion, the subregion is divided according to the similarity of each pixel. It is not expected that there are many heterogeneous regions in the lesion, so the clustering algorithm is suitable for this problem.

First, we need to determine the similarity metric function sim among pixels, which requires consideration not only of static features (grayscale values) but also of dynamic characteristics (multiple phase variations). There are 8 phases in this paper and each pixel is represented as x=(x0,x1,⋯⋯,x7). The similarity metric function sim is defined as:

sim(xi,yj)=∑k=07(xi−yi)2[2]

The k-means clustering method is used first in this paper to extract all the initial heterogeneous regions in the lesion, and the number of clusters is automatically determined by elbow rules. Then, the final clustering results are judged according to the connectivity and number of pixels of the cluster on the image. However, these pixels in the same areas are treated as the same cluster element according to the similarity space, but a cluster may not be connective in planar space. Therefore, new clusters split from the initial cluster are determined by connectivity. An old cluster region is divided into several connected subregions. The connective region is removed if the number of pixels in the connective region is too small (e.g., it is set to 10), and finally, we obtain the regions that are defined as the heterogeneous regions of the lesion in this paper.

The above idea is the heterogeneous region extraction method proposed in this paper, as shown in Algorithm 1. The algorithm is applied to all lesions in the dataset to obtain the heterogeneous regions of each lesion. The sample heterogeneous regions of the lesion are shown in Figure 7.

Figure 7 Heterogeneous regions of a lesion with colors segmented by the above algorithm.

Visual word extraction for molecular subtypes of cancer

According to the above extracted heterogeneous region dataset Φ, the visual and dynamic parameters of each heterogeneous region Φ are extracted, where ϕ∈Φ. The radiomics feature data extracted in this paper include texture parameters, kinetic parameters, statistical parameters and morphological parameters. There are 8 different time-phase data for each heterogeneous region (including the original phase without agent injection). Each time-phase data matrix is represented by S0,S1,⋯,S7, respectively. The texture parameters are calculated based on the gray-level cooccurrence matrix (GLCM) method, and the kinetic parameters are calculated in the three time phases of S0, S1 and S3.

According to the breast imaging reporting and data system (BI-RADS) standard, the internal enhancement of malignant lesions is mostly characterized by internal interval enhancement. Internal enhancement is more common in high-level ductal carcinoma or rich vascular tumor lesions than in the surrounding significant central enhancement. The different enhancement methods are reflected in the difference in the corresponding texture feature parameters.

Energy: F=∑i=1k∑j=1kp(i,j)2[3]

Contrast: F=∑n=0k−1n2{∑|i−j|=np(i,j)}[4]

Correlation: F=∑i=1k∑j=1k(ij)p(i,j)−μiμjσiσj[5]

Entropy: F=∑i=1k∑j=1kp(i,j)lgp(i,j)[6]

DeficitMatrix: F=∑i=1k∑j=1kp(i,j)1+(i−j)2[7]

where p is the gray-level cooccurrence matrix with size m × n, and p(i, j) represents the number of two pixels with grayscale i and j simultaneous occurrence.

The dynamic information of the lesions shows the agent signal change in the lesion or normal tissue. In this paper, the kinetic parameters are extracted as the dynamic enhancement rate, which is defined as follows.

RTtS={r|r=IT(i)−It(i)It(i),i=1,2,⋯,N},where T=1,2t=0,1T>t[8]

where S indicates the lesion and T and t represent the time phases, e.g., R10 is the lesion’s dynamic enhancement rate from time T0 to T1. We can also calculate the dynamic enhancement ratio characteristic, e.g., R20/R10. Finally, the standard deviation, mean value, maximum value and minimum value are calculated according to the above values as the final dynamic parameters.

The calculations of the grayscale values of each pixel point in the lesion are as follows.

Skewness=∑i=1n(xi−μ)3(n−1)σ3[9]

Kurtosis=∑i=1n(xi−μ)4(n−1)σ4−3[10]

where n represents the number of pixel values, x represents the pixel value, µ represents the mean, and σ represents the standard deviation.

The radial length of the heterogeneous region of the lesion is defined as follows.

d(i)=[x(i)−x0]2+[y(i)−y0]2MaxD,i=1,2,⋯,N[11]

where (x0,y0) is the center point, (xi,yi) is the boundary point, MaxD is the max-imum radial length. The standardized radial length mean and standard deviation based on radial length are calculated. The degree of tightness and roughness are defined as follows.

Comp=A4πP2[12]

Roughness=1N∑i=1N(d(i)−μd)44−1N∑i=1N(d(i)−μd)2μd[13]

where A represents the lesion area, P is the contour circumference, N represents the number of nonzero pixels, d(i) is computed using the above formula, and µd represents the radial length mean.

A total of 62 dimensional feature parameters are obtained based on the above parameters calculation, and this size of parameters is marked as D. Assuming that the dataset extracts the number of heterogeneous regions as H, the extracted parameter data are a matrix F with size D × H. We can obtain eight matrices with the time axis, that are F0, F1, · · ·, F7 at time phase T1, · · ·, T7, respectively.

The clustering method is used on the heterogeneous region data Ft at each time phase t, and then the clusters are obtained. The central vectors for each cluster are considered a visual word. Then, we obtain k centers for every time phase. The word dictionary can be represented as follows.

yt=(C1t,C2t,⋯,Ckt)T=(ct11⋯ctD1⋮⋱⋮ct1k⋯ctDk)[14]

Each of the central vectors in this paper is based on the center of gravity calculated by all heterogeneous vectors in the cluster. A new lesion is converted into N heterogeneous subregions as ϕ1,ϕ2,⋯⋯,ϕN; then the feature data are extracted as follows in 8 time phases.

(ϕ1⋮ϕN)=(F10⋯F17⋮⋱⋮FN0⋯FN7)=(a011⋯a0D1⋮⋱⋮a71k⋯a7Dk⋮a01N⋯a0DN⋮⋱⋮a71N⋯a7DN)[15]

The bag of word vector Yt(ϕi) for each heterogeneous region of the lesion is calculated by following formula.

Yt(ϕi)=Sign(sim(ϕi,yt)max(sim(ϕi,yt))), wheresim(ϕi,yt)=(yt×Fit)12=[(ct11⋯ctD1⋮⋱⋮ct1k⋯ctDk)×(at1i⋮atDi)]12Sign(x)={1if x≥10if x<1[16]

Y(ϕ)=(∑i=1NY0ϕi,⋯,∑i=1NY7ϕi)[17]

Recognition of molecular subtypes by the classification model

At the genetic level, the lesion is divided into four molecular subtypes, namely, luminal A (luminal epithelial type A), luminal B (luminal epithelial type B), human epidermal growth factor receptor 2 (HER-2) and basal-like. The different molecular subtypes are shown in Figure 8 and Figure 9.

Figure 8 Different molecular subtypes, where (A) represents luminal A, (B) represents luminal B, and (C) represents HER-2 and basal-like, and (D) represents basal-like. HER-2, human epidermal growth factor receptor 2.

Figure 9 Map of lesions with different molecular typing, where (A) represents luminal A, (B) represents luminal B, (C) represents HER-2 and (D) represents basal-like.

The temporal bag of visual word feature data obtained by the above method not only accounts for the heterogeneity distribution of each lesion but also considers the change trend of this heterogeneity in different time phases. However, there are relatively large differences between the data sample categories due to the situation of breast cancer data itself in the pathogenesis, such as basal-like subtype is only a third of luminal B with maximum number in the dataset. Therefore, the imbalance problem of the dataset in this paper will affect the performance of the learning model.

In this paper, a cascade processing method of oversampling is used to preprocess the unbalanced problem of samples, which is called SMOTE (synthetic minority oversampling technique). For each sample x of the few classes, the distance of the point from other sample points in a few classes is calculated to obtain the nearest k neighbors. The sampling ratio is set as the sample unbalance ratio. Several samples are selected randomly from the k neighbor for each minority sample x. Assuming that the selected neighbor is x', a new sample xn is constructed with the original sample according to the following formula.

Xn=X+random(0,1)×(X′−X−K)[18]

In the classification stage, four classic classifiers are used, and the results show different performances on different molecular subtypes for each classifier. Therefore, a combination classification model is designed for better performance for different molecular subtypes based on stacking.

Stacking consists of two phases, training and combining, which first train and generate several base-learning models, followed by combining the outputs of these base-learning models as the final output results using some strategy. There are two methods for dividing the training subsets of the base-learning models, which are the training subsets of individual learning models obtained by random sampling through the self-help method and the results generated by the previous training model as the input of the latter base-learning model. The basic framework of the stacking model is shown in Figure 10. The stacking method has a stronger nonlinear expression ability than a single classification model and can reduce generalization error and prevent overfitting. In this paper, three classifiers, support vector machine (SVM), random forest (RF), and gradient boosting decision tree (GBDT), are used in the primary model, and the second layer adopts the LR (logistic regression) classifier. The process is described as follows. The schematic of the stacking algorithm is shown in Figure 11.

Figure 10 The basic framework of the stacking model, where the four classifiers are Support Vector Machine (SVM), Random Forest (RF), and Gradient Boosting Decision Tree (GBDT), Logistic Regression (LR).

Figure 11 The schematic of stacking algorithm.

The dataset is divided into two parts: one part is the training set, and the other part is the testing set. The training set is then divided into three parts, each of which is used to train classification models such as SVM, RF and GBDT. The number of cross-validation folds is related to the number of classification models. The testing set is predicted simultaneously. The prediction results of all training sets are obtained after 5 rounds of training and prediction for the three models, and the average of the testing results for 5 prediction sets is calculated as the final prediction results for the input testing data of the next classification process by repeating step I. Steps I and II are repeated until all three primary models complete the above process. The training results and test results obtained from the three models are used as training sets and test sets for the level 1 model. The LR classifier is selected as a level 1 classifier, and the data obtained in step 4 are used for final training and prediction.

Results

To verify the method proposed in this paper, DCE-MRI and pathology data of breast cancer are collected simultaneously, and the locations of the lesions in the images are labeled by two breast radiologists with more than 10 years of clinical experience. The suspected lesion area is marked manually by a rectangular box. In this paper, our experiment is to use the Faster R-CNN framework to train a target detection model on our dataset, using the detection model to verify its performance on the test set. In the task of identifying molecular subtypes of breast cancer, we performed comparative experiments. First, the features are extracted directly at the location of the lesion without using the time series word bag model and dividing the heterogeneous region, and then the classification model is trained to recognize molecular subtypes. Second, the temporal bag of visual word features is extracted from the heterogeneous regions proposed in this paper. The same classification model is then used to validate the recognition performance.

Patient cohort and data acquisition

In this paper, the dataset consists of 322 cases of patients in total. All cases were malignant cases of breast cancer in women that were confirmed by histopathology examination after the patient received DCE-MRI examination. In this study, DCE-MRI was performed on a 3.0T scanner using a dedicated 16-channel dual breast phase-control coil. Patients were positioned prone, and each patient was scanned using the same sequence of scans, starting with a conventional MRI scan with triplane localization, followed by a spectrally selective attenuated inversion recovery, and finally, a sequence of breast-enhanced T1 high-resolution isotropic volume excitation with fat suppression. A total of 8 temporal sampling points (1st contrast phase before contrast injection and 7th contrast phases after contrast injection) were included. The scanning time for each time phase was approximately 55 s, and the total time was approximately 8 min. The pathological data of 322 patients are shown in Table 1, as well as statistics of molecular subtypes, where P1–P10 represent pathology of intracatheter cancer, invasive ductal carcinoma, invasive micropapillary carcinoma, mucinous carcinoma, invasive lobular carcinoma, medullary carcinoma, solid papillary carcinoma, ductal carcinoma in situ, extensive ductal carcinoma, and extensive ductal carcinoma in situ, respectively. It is easy to see that the dataset has an imbalance problem in molecular subtypes.

Performance of cancer lesion detection

In the task of detecting breast cancer lesions, the Faster R-CNN algorithm we used was performed on an NVIDIA GTX 1080Ti 11G GPU.

Using our method, the best breast cancer lesion detection result is a sensitivity of 1 at 0.20 FPs/case. The proposed method is compared with the results of previous studies on breast cancer localization in DCE-MRI, as shown in Table 2. The sensitivity is 0.94 at 7 FPs/case by Albert et al. (18), 1 at 6.30 FPs/case by Chang et al. (19) 0.98 at 0.16 FPs/case by Renz et al. (32).

The experimental results show that the neural network structure in this paper can learn the effective features of breast cancer lesions. To our knowledge, this may be the first study to detect breast cancer in DCE-MRI based on the Faster R-CNN framework.

Performance of lesion molecular subtype classification

In this paper, the algorithms of heterogeneous region extraction and feature extraction were all implemented based on the Python platform. All the machine learning algorithms applied in this paper were from the open source scikit-learn software library. First, the original patient image data were randomly divided into two parts as training and testing sets, which ensured that the proportion of each subtype of molecule in the two datasets was the same as that in the whole dataset.

The experimental process was divided into two stages. First, the four kinds of features of texture, morphology, statistics and kinetics were extracted under the condition that each breast cancer lesion was not divided into heterogeneous regions. Then, the four classifiers of LR, SVM, RF and GBDT were used in this paper for training and testing verification. A set of results were obtained, which are listed as lines that are marked as normal in Table 3. Then, the method proposed in this paper was used. The heterogeneous regions were extracted, and 817 subregions were obtained (N=817). The above four features were extracted from these heterogeneous regions, and the bag of visual word dictionary with a length of 100 was trained (k=100). Afterward, we obtained the heterogeneous temporal bag of visual word features for each lesion. Based on the feature data by the temporal bag of visual word model, the verification tasks were carried out by a single classifier, and then a set of experimental results were obtained as the lines labeled TBOVW tag in Table 3. Finally, the above four classifiers were integrated into the stacking model that is trained and tested on the feature data of the temporal bag of visual words in this paper, and a set of experimental results were obtained, as shown in Table 3.

The results of the experiment are evaluated using four metrics, namely, accuracy (Acc.), precision (Pre.), recall rate (Rec.) and F1-score value (F1). The results are shown in Table 3.

Based on the TBOVW model, the performances of these classifiers significantly improved, and the accuracy of each classifier reached more than 80%. This finding shows that the existence of breast heterogeneity has an effect on image feature extraction. In addition, the TBOVW method also makes the classification results more balanced. However, there are no great differences in classification performance between different classifiers, which is unlike the conventional methods. The TBOVW model can effectively reduce the effect of classification performance caused by the data imbalance problem.

The stacking classification model achieves optimal results in these comparisons, which is slightly better than using classifiers alone. That means that each classifier has differences in the recognition performance of different molecular subtypes. Each subtype of classifier can perform the best by the stacking model. Thus, the recognition of each molecular subtype is optimal. The results of identifying each subtype of molecules by the stacking model are shown in Table 4.

Table 4 shows that the stacking model is good for the classification performance of luminal A, luminal B, HER-2 expression and basal-like. The classification effect of the logistic regression stacking model improved compared with the single model of logic regression, support vector machine, random forest and gradient boosting decision tree, which proves the validity of the model combination. The optimization of the experimental results can be realized through stacking the model. After the model combination in this experiment, the ensemble classifier is better than a single model.

Discussion

A computer-aided diagnosis method is proposed to automatically locate breast cancer lesions and identify molecular subtypes of breast cancer with heterogeneity analysis from radiomics data. A Faster R-CNN framework is first applied to images to detect breast cancer lesions. Then, the heterogeneous regions of every breast cancer lesion are extracted. Based on the multiple visual and kinetic radiomics features extracted from the heterogeneous regions, a temporal bag of visual word model is proposed, which takes into account the dynamic characteristics of both lesion and heterogeneous regions in images over time. The recognition task of molecular subtypes of breast lesions is realized based on a stacking classification model. According to the above testing results, the image molecular classification recognition reaches the highest 90% accuracy, and the causes of the model error may come from two aspects. First, there are errors in the classification model, and second, some molecular subtypes are not accurate in pathology determination. For the former, we can solve the method by studying more feature extraction methods and more classification models. The latter problem depends on subtype testing by local biopsy tissue, and it is known that each lesion has heterogeneity. However, this is characteristic of the previous image heterogeneity extraction description. However, pathological biopsy only extracts part of the biopsy tissue, which is not a precise and comprehensive evaluation. Therefore, the second type of error cause does not indicate that this method has shortcomings for the problem of subtype classification. It may be wrong to determine the molecular subtype itself in routine clinical pathology. The problem has to be further confirmed by medical clinics, which is beyond the scope of this study.

Conclusions

In this paper, our computer-aided approach can detect breast cancer lesions and identify molecular subtypes of breast cancer. First, a Faster R-CNN framework method for detecting breast cancer lesions is proposed to locate the approximate location of breast cancer in DCE-RMI images. Then, a molecular subtype recognition method with quantitative heterogeneity analysis is applied to breast cancer lesions. From the experimental results, the method has a high performance on our dataset. Breast cancer detection verifies the effectiveness of the Faster R-CNN algorithm for breast cancer lesion detection tasks in DCE-MRI. The identification method of breast cancer molecular subtypes shows the influence of heterogeneous regions on the recognition process.

Acknowledgments

The data used in this article were obtained from Liaoning Cancer Hospital.

Funding: This work was supported by the National Key R&D Program of China (No. 2021YFC2701003), the Fundamental Research Funds for the Central Universities (No. N2016006), and the Natural Science Foundation of Liaoning Province (No. 2021-MS-085).

Footnote

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at https://qims.amegroups.com/article/view/10.21037/qims-22-1230/coif). The other authors have no conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. This article does not contain any studies with human participants or animals performed by any of the authors, ethical approval and written informed consent are not needed.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. Sala E, Mema E, Himoto Y, Veeraraghavan H, Brenton JD, Snyder A, Weigelt B, Vargas HA. Unravelling tumour heterogeneity using next-generation imaging: radiomics, radioge-nomics, and habitat imaging. Clin Radiol 2017;72:3-10. [Crossref] [PubMed]
2. O'Connor JP, Rose CJ, Waterton JC, Carano RA, Parker GJ, Jackson A. Imaging intra-tumor heterogeneity: role in therapy response, resistance, and clinical outcome. Clin Cancer Res 2015;21:249-57. [Crossref] [PubMed]
3. Li W, Yu K, Feng C, Zhao D. Molecular Subtypes Recognition of Breast Cancer in Dynamic Contrast-Enhanced Breast Magnetic Resonance Imaging Phenotypes from Radiomics Data. Comput Math Methods Med 2019;2019:6978650. [Crossref] [PubMed]
4. Lambin P, Rios-Velazquez E, Leijenaar R, Carvalho S, van Stiphout RG, Granton P, Zegers CM, Gillies R, Boellard R, Dekker A, Aerts HJ. Radiomics: extracting more information from medical images using advanced feature analysis. Eur J Cancer 2012;48:441-6. [Crossref] [PubMed]
5. Wang Jun, Liu Xia. Di Dong, Jiangdian Song, Min Xu, Yali Zang, Jie Tian. Prediction of malignant and benign of lung tumor using a quantitative radiomic method. Annu Int Conf IEEE Eng Med Biol Soc 2016;2016:1272-5.
6. Aerts HJ, Velazquez ER, Leijenaar RT, Parmar C, Grossmann P, Carvalho S, Bussink J, Monshouwer R, Haibe-Kains B, Rietveld D, Hoebers F, Rietbergen MM, Leemans CR, Dekker A, Quackenbush J, Gillies RJ, Lambin P. Decoding tumour phenotype by non-invasive imaging using a quantitative radiomics approach. Nat Commun 2014;5:4006. [Crossref] [PubMed]
7. Xia F, Hu P, Wang J, Hu W, Li Guichao, Zhang Z. Application of radiomics approach for decoding normal liver features and predicting chemotherapy-associated liver injury: A preliminary study. China Oncology 2016;26:521-6.
8. Jia T, Yu W, Fu X. Progress on application of radiomics in precise treatment of non-small cell lung cancer. Chinese Journal of Radiological Medicine and Protection 2016;36:947-50.
9. Mahmood T, Arsalan M, Owais M, Lee MB, Park KR. Artificial Intelligence-Based Mitosis Detection in Breast Cancer Histopathology Images Using Faster R-CNN and Deep CNNs. J Clin Med 2020;9:749. [Crossref] [PubMed]
10. Martelotto LG, Ng CK, Piscuoglio S, Weigelt B, Reis-Filho JS. Breast cancer intra-tumor heterogeneity. Breast Cancer Res 2014;16:210. [Crossref] [PubMed]
11. Adoui M, Drisis S, Larhmam MA, Benjelloun ML. Breast cancer heterogeneity analysis as index of response to treatment using mri images: A review. Imaging in Medicine 2017;9:109-19.
12. Chang RF, Chen HH, Chang YC, Huang CS, Chen JH, Lo CM. Quantification of breast tumor heterogeneity for ER status, HER2 status, and TN molecular subtype evaluation on DCE-MRI. Magn Reson Imaging 2016;34:809-19. [Crossref] [PubMed]
13. Banaie M, Soltanian-Zadeh H, Saligheh-Rad HR, Gity M. Spatiotemporal features of DCE-MRI for breast cancer diagnosis. Comput Methods Programs Biomed 2018;155:153-64. [Crossref] [PubMed]
14. Sutton EJ, Huang EP, Drukker K, Burnside ES, Li H, Net JM, Rao A, Whitman GJ, Zuley M, Ganott M, Bonaccio E, Giger ML, Morris EATCGA group. Breast MRI radiomics: com-parison of computer- and human-extracted imaging phenotypes. Eur Radiol Exp 2017;1:22. [Crossref] [PubMed]
15. Holli-Helenius K, Salminen A, Rinta-Kiikka I, Koskivuo I, Brück N, Boström P, Parkkola R. MRI texture analysis in differentiating luminal A and luminal B breast cancer molecular subtypes - a feasibility study. BMC Med Imaging 2017;17:69. [Crossref] [PubMed]
16. ZhangYNockWWyseMWeberZAdamsEAsadSStockardSTallmanDWinerEPLinNUCherianMLustbergMBRamaswamyBSardesaiSVanDeusenJWilliamsNWesolowski W, Stover DG. Machine learning predicts rapid relapse of triple negative breast cancer.bioRxiv, 613604 (2019). doi: .10.1101/613604
17. Dhahri H, Rahmany I, Mahmood A, Al Maghayreh E, Elkilani W. Tabu Search and Ma-chine-Learning Classification of Benign and Malignant Proliferative Breast Lesions. Biomed Res Int 2020;2020:4671349. Erratum in: Biomed Res Int 2020;2020:7638969. [PubMed]
18. Gubern-Mérida A, Martí R, Melendez J, Hauth JL, Mann RM, Karssemeijer N, Platel B. Automated localization of breast cancer in DCE-MRI. Med Image Anal 2015;20:265-74. [Crossref] [PubMed]
19. Chang YC, Huang YH, Huang CS, Chen JH, Chang RF. Computerized breast lesions detection using kinetic and morphologic analysis for dynamic contrast-enhanced MRI. Magn Reson Imaging 2014;32:514-22. [Crossref] [PubMed]
20. Ronneberger O, Fischer P, Brox T. U-net: Convolutional networks for biomedical image segmentation. In: Navab N, Hornegger J, Wells W, Frangi A. editors. International Con-ference on Medical Image Computing and Computer-Assisted Intervention, Springer, Cham, 234-241.
21. Ren S, He K, Girshick R, Sun J. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. IEEE Trans Pattern Anal Mach Intell 2017;39:1137-49. [Crossref] [PubMed]
22. Zhang J, Saha A, Zhu Z, Mazurowski MA. Hierarchical Convolutional Neural Networks for Segmentation of Breast Tumors in MRI With Application to Radiogenomics. IEEE Trans Med Imaging 2019;38:435-47. [Crossref] [PubMed]
23. Wu J, Sun X, Wang J, Cui Y, Kato F, Shirato H, Ikeda DM, Li R. Identifying relations be-tween imaging phenotypes and molecular subtypes of breast cancer: Model discovery and external validation. J Magn Reson Imaging 2017;46:1017-27. [Crossref] [PubMed]
24. Fan M, Li H, Wang S, Zheng B, Zhang J, Li L. Radiomic analysis reveals DCE-MRI features for prediction of molecular subtypes of breast cancer. PLoS One 2017;12:e0171683. [Crossref] [PubMed]
25. Li H, Zhu Y, Burnside ES, Huang E, Drukker K, Hoadley KA, Fan C, Conzen SD, Zuley M, Net JM, Sutton E, Whitman GJ, Morris E, Perou CM, Ji Y, Giger ML. Quantitative MRI ra-diomics in the prediction of molecular classifications of breast cancer subtypes in the TCGA/TCIA data set. NPJ Breast Cancer 2016;2:16012. [Crossref] [PubMed]
26. Cho N. Molecular subtypes and imaging phenotypes of breast cancer. Ultrasonography 2016;35:281-8. [Crossref] [PubMed]
27. Zhu Z, Albadawy E, Saha A, Zhang J, Harowicz MR, Mazurowski MA. Deep learning for identifying radiogenomic associations in breast cancer. Comput Biol Med 2019;109:85-90. [Crossref] [PubMed]
28. Zhu JY, Krähenbühl P, Shechtman E, Efros AA. Generative Visual Manipulation on the Natural Image Manifold. In: Leibe B, Matas J, Sebe N, Welling M. editors. Computer Vision – ECCV 2016. Lecture Notes in Computer Science, vol 9909. Springer, Cham.
29. Deng Y, Li Y, Wu JL, Zhou T, Tang MY, Chen Y, Zuo HD, Tang W, Chen TW, Zhang XM. Radiomics models based on multi-sequence MRI for preoperative evaluation of MUC4 status in pancreatic ductal adenocarcinoma: a preliminary study. Quant Imaging Med Surg 2022;12:5129-39. [Crossref] [PubMed]
30. Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recongnition. San Diego: International Conference of Learning Representation, 2015.
31. Girshick R. Fast r-cnn. Proceedings of the IEEE International Conference on Computer Vision. 2015: 1440-8.
32. Renz DM, Böttcher J, Diekmann F, Poellinger A, Maurer MH, Pfeil A, Streitparth F, Col-lettini F, Bick U, Hamm B, Fallenberg EM. Detection and classification of con-trast-enhancing masses by a fully automatic computer-assisted diagnosis system for breast MRI. J Magn Reson Imaging 2012;35:1077-88. [Crossref] [PubMed]