Using 2-dimensional hand photographs to predict postoperative biochemical remission in acromegaly patients: a transfer learning approach
Original Article

Using 2-dimensional hand photographs to predict postoperative biochemical remission in acromegaly patients: a transfer learning approach

Mengqi Wang1#^, Chengbin Duan1#, Shun Yao1, Jinping Chen1, Shaolin Zhang1, Zongming Wang1, Bin Hu1, Zhigang Mao1, Haijun Wang1, Yonghong Zhu2, Wenli Chen1

1Center for Pituitary Tumor Surgery, Department of Neurosurgery, The First Affiliated Hospital, Sun Yat-sen University, Guangzhou, China; 2Department of Histology and Embryology, Zhongshan School of Medicine, Sun Yat-sen University, Guangzhou, China

Contributions: (I) Conception and design: M Wang, H Wang, Y Zhu, W Chen; (II) Administrative support: S Yao, W Chen; (III) Provision of study materials or patients: M Wang, C Duan, J Chen, S Zhang; (IV) Collection and assembly of data: J Chen, Z Wang, B Hu, Z Mao; (V) Data analysis and interpretation: M Wang, C Duan, S Yao; (VI) Manuscript writing: All authors; (VII) Final approval of manuscript: All authors.

#These authors contributed equally to this work.

^ORCID: 0000-0003-2458-6192.

Correspondence to: Haijun Wang. Center for Pituitary Tumor Surgery, Department of Neurosurgery, The First Affiliated Hospital, Sun Yat-sen University, Guangzhou 510080, China. Email: wanghaij@mail.sysu.edu.cn; Yonghong Zhu. Department of Histology and Embryology, Zhongshan School of Medicine, Sun Yat-sen University, Guangzhou 510080, China. Email: zhuyongh@mail.sysu.edu.cn; Wenli Chen. Center for Pituitary Tumor Surgery, Department of Neurosurgery, The First Affiliated Hospital, Sun Yat-sen University, Guangzhou 510080, China. Email: chenwenli@mail.sysu.edu.cn.

Background: The primary treatment goals in acromegaly patients are complete surgical removal of underlying pituitary tumors and biochemical remission. One of the challenges in developing countries is the difficulty in monitoring postoperative biochemical levels in acromegaly patients, particularly those who live in remote areas or regions with limited medical resources.

Methods: In an attempt to overcome the abovementioned challenges, we conducted a retrospective study and established a mobile and low-cost method to predict biochemical remission in acromegaly patients after surgery, the efficacy of which was assessed retrospectively using the China Acromegaly Patient Association (CAPA) database. A total of 368 surgical patients from the CAPA database were successfully followed up to obtain their hand photographs. Demographics, baseline clinical characteristics, pituitary tumor features, and treatment details were collated. Postoperative outcome, defined as biochemical remission at the last follow-up timepoint, was assessed. Transfer learning with a new mobile tailored neurocomputing architecture, MobileNetv2, was used to explore the identical features that could be used as predictors of long-term biochemical remission after surgery.

Results: As expected, the MobileNetv2-based transfer learning algorithm was shown to predict biochemical remission with statistical accuracies of 0.96 and 0.76 in the training cohort (n=803) and validation cohort (n=200), respectively, and the loss function value was 0.82.

Conclusions: Our findings demonstrate the potential of the MobileNetv2-based transfer learning algorithm in predicting biochemical remission for postoperative patients who are at home or live far away from a pituitary or neuroendocrinological treatment center.

Keywords: Acromegaly; transfer learning; MobileNetv2; biochemical remission


Submitted Oct 11, 2022. Accepted for publication Mar 27, 2023. Published online Apr 06, 2023.

doi: 10.21037/qims-22-1101


Introduction

Acromegaly is a rare neuroendocrinological disorder characterized by over-secretion of growth hormone (GH) and insulin-like growth factor 1 (IGF-1) levels (1,2). The majority of GH-secreting pituitary tumors are benign, although some tumors may present with aggressive features and have a high risk of recurrence after surgical treatment (3). Persistently high levels of GH and/or IGF-1 can lead to comorbidities of the cardiovascular, respiratory, bone, and endocrine systems, and are associated with increased mortality rates (2,4,5). Therefore, normalizing GH and/or IGF-1 levels is a critical treatment goal for acromegaly patients.

According to the most recent consensus statement on acromegaly management (6), surgical resection of the pituitary tumor is recommended as the first-line medical therapy. Surgical treatment can effectively lead to biochemical control. Furthermore, baseline characteristics that can impact long-term outcomes include age at diagnosis (7), gender (7), preoperative serum levels of GH and IGF-1 (8-13), preoperative medication (12), tumor size (8,9,12), invasiveness (12,14), Knosp grading scores (8,9,12), and histological pathology [Ki-67, somatostatin receptor subtype 2 expression, arylhydrocarbon receptor inter acting protein (AIP) expression, and granularity] (14). Therefore, identifying independent factors that predict long-term outcomes, such as biochemical remission or recurrence, may be crucial in facilitating clinical management.

As a rare disease, patients are usually diagnosed with acromegaly in a special pituitary surgery center or neuroendocrinological department, and many of them, particularly those in the developing countries, usually come from remote areas, resulting in a high rate of loss to follow-up (12,15). With the emerging development of artificial intelligence (AI), some algorithms have demonstrated a potential application in auto-segmentation (16,17), early preoperative diagnosis (18,19), subtype classification (20-22), personalized treatment (23), assessment of response to treatments (24-26), outcome prediction (27-29), and other decision-making roles in the clinical management of patients with pituitary or other brain tumors (30). Theoretically, an ideal AI model should be trained and generated using a wealth of data, however, this is a major challenge in real-world applications. Although it is possible to obtain enough data to establish a classification task in 1 domain of interest, it is a challenge to apply this model to another dataset wherein the domain of interest is different to that in the parent dataset. To overcome the data shortage in many medical disciplines, the promising machine learning technique, transfer learning, has shown its advantage in classification, regression, and clustering problems (31). Therefore, clinical translational researchers may apply AI in the management of postoperative follow-up in acromegaly patients, with the aim of improving their quality of life (32-34).

The China Acromegaly Patient Association (CAPA) database captures data from 112 hospitals over an extended period of time in China; however, to date, the hand photograph dataset on acromegaly has not been fully reported. Large population series such as the CAPA database enable us to effectively identify and report predictors of long-term biochemical recurrence in acromegaly patients after surgery. Taking advantage of AI techniques and the CAPA database, this study used the transfer learning approach with an updated mobile tailored neurocomputing architecture, MobileNetv2, to predict the postoperative long-term biochemical remission of patients diagnosed with acromegaly.


Methods

Study design and database

The CAPA database [July 1998–December 2018; (https://capa.wohenok.com/)] is a patient-advocacy and non-profit organization, aiming to facilitate the recognition of acromegaly for patients and their families in mainland China. Demographics, clinical characteristics, radiographic features, hands photographs, and treatment approaches and outcomes have been described and reported by members in the Chinese Pituitary Adenoma Collaborative Group (12,19,32,35). Photographs of hands were taken at 3 months postoperatively by members of our research team (Mengqi Wang, Chengbin Duan, and Wenli Chen) at the outpatient department. All patients in the CAPA database were clinically diagnosed with acromegaly according to the established guidelines for acromegaly management [2021] (36). Diagnostic measurements included clinical findings, radiological identification of a pituitary tumor, and biochemical analyses (fasting GH >2.5 µg/L and lowest GH >1 µg/L in glucose GH inhibition tests, and IGF-1 levels higher than the normal range of age- and sex-matched healthy cases where there is available information of IGF-1 in some hospitals). The disease duration was calculated from the time of the report of the first alteration. In this study, we only enrolled acromegaly patients with long-term (over 6 months) follow-up after surgery. In total, 368 surgical patients from the CAPA database were successfully followed up and their hand photographs were obtained.

Outcomes of interest

The outcomes were assessed using the biochemical measurements at last follow-up after surgery (over 6 months). The dataset was divided into two groups according to the last follow-up biochemical status: a remission group, and a non-remission group. The criteria of postoperative biochemical remission/recurrence were based on the Chinese Guideline of Acromegaly Diagnosis and Management [2021] (36). Patients were determined to be in biochemical remission if random serum GH levels were below 1.0 µg/L or GH nadir levels were <1 µg/L after an oral glucose tolerance test (OGTT), and/or IGF-1 levels were decreased to normal levels.

Removal of background noise in hand photographs

To reduce the computing burden and improve the predicting accuracy, background noise in the hand photographs was removed for image analysis. There are several different color spaces, each with its own significance, including RGB (red, green, blue), HIS (hue, intensity, saturation), HSL (hue, saturation, lightness), HSV (hue, saturation, value; also known as HSB), YCrCb, CIE XYZ, CIE Lab, and so on. To process the Cr component in YCrCb, we converted RGB to YCrCb, and applied the Otsu’s thresholding method in OpenCV (Open-Source Computer Vision) to remove the background noise (37). Finally, a total of 420 hand photographs were available for further analysis (Figure 1).

Figure 1 Illustration of background removal in hand photographs using Otsu’s thresholding method.

Transfer learning

The traditional deep learning method usually assigns randomized initialization parameters to the neural network which will learn each task from scratch. Under certain circumstances, the neural network can use the previously learned knowledge to solve the new problems it faces. This kind of knowledge transfer is called transfer learning. In the field of image classification, transfer learning is a technique that utilizes a model already trained on a specific data set (31). The model is reused to solve other classification problems, and the model is used as the starting point of training to update the parameters so that it is adapted to the target data set. ImageNet is an excellent image classification data set with 3.2 million images, and is widely used in the field of object detection and image classification. The experiment in this study transferred the parameters obtained from the training of MobileNetv2 (38) on ImageNet to our model, aiming to build an optimal model suitable for the identification of postoperative biochemical remission in patients with acromegaly. Compared with learning from scratch, transfer learning can effectively speed up network convergence and guarantee the efficiency and accuracy of model learning in the case of insufficient samples.

Network architecture

In this investigation, the MobileNetv2 algorithm was used for the identification of postoperative biochemical remission in patients with acromegaly through transfer learning. The MobileNetv2 algorithm is an improved algorithm based on the MobileNetv1 algorithm and it still deploys the depthwise separable convolutions (DSCs) implemented in the MobileNetv1 algorithm, and introduces the bottleneck block inverted residual module, which effectively improves the accuracy of image classification and inspection tasks that are used on the mobile application (38).

DSC consists of a depth wise convolution (DConv) operation and a pointwise convolution (PConv) operation (39). The former runs a convolution operation with a single convolution kernel size of 3×3 in each channel dimension of the input, and the latter combines all input channel dimensions by performing a standard convolution operation with a convolution kernel size of 1×1 features to build the improved features. The calculation amount of the standard convolution is k2 times different from the depth separable convolution (k is the size of the convolution kernel). The MobileNetv2 network uses k=3 (namely 3×3), so without reducing the accuracy, the amount of calculation is only 1/9 to 1/8 of the standard convolution.

Performing the rectified linear unit (ReLU) activation function operation on the low-dimensional convolutional layer can easily cause the loss of feature information, whereas performing the ReLU activation function operation on the high-dimensional convolutional layer can effectively reduce the loss of feature information (40). Therefore, the MobileNetv2 algorithm does not use the activation function in the low-dimensional convolutional layer of the bottleneck block, that is, the linear bottleneck layer is used, and the ReLU6 activation function proposed in the MobileNetV1 algorithm is more suitable for quantizing the network in other layers to prevent the nonlinear layer from destroying too much feature information.

The residual bottleneck block structure of the MobileNetv2 algorithm is shown in Figure 2. The inverted residual block is mainly used to promote the effective transmission of multi-layer feature information and enhance the feature extraction capability of the network. The input of the inverted residual module is first subjected to convolution with a convolution kernel size of 1×1 for channel expansion, and then a deep convolution with a convolution kernel size of 3×3 is used to extract features, and finally through the convolution kernel size in the linear bottleneck layer. It is a point-by-point convolution of 1×1, which reduces the channel dimension. When the number of bottleneck blocks used in the MobileNetv2 algorithm is more than 1 block, the bottleneck block contains the inverted residual module, otherwise it does not contain the inverted residual module.

Figure 2 Illustration of the residual bottleneck block structure used in the MobileNetv2 algorithm.

The overall structure of MobileNetv2 network is shown in Figure 3. The input to the network is the hand photographs of acromegaly patients, the standardized image size is 224×224, and the output is the patient’s long-term outcomes consisting of biochemical remission and non-remission.

Figure 3 Illustrative framework of our proposed MobileNetv2-based transfer learning algorithm to automatically predict postoperative remission for acromegaly patients using hand photographs. s, stride; p, padding; inc, input channel; t, expansion factor.

Loss function

The loss function categorical cross-entropy was used for binary classification of hand photographs (see the formula below). The model output layer used the softmax activation function for two classifications, and calculates the classification probability of the pixels.

loss=i=1ny^i1log2pi1+y^i2log2pi2++y^imlog2pim

In the formula, n is the number of sample size, m is the number of categories, y^i1 refers to the first category of the first sample in the first category, and Pi1 1 refers to the probability of the first sample in the first category, and finally, the loss value of the n samples is obtained.

Experiments

The hardware configurations of the workstation used in this experiment are as follows: 64-bit Window10 operating system, Intel(R) Core (TM) i9-10900K CPU @ 3.70 GHz processor, and 1 graphics processing unit, namely, the NVIDIA GeForce RTX 3080 (NVIDIA Corp., Santa Clara, CA, USA). The software environment was Python3.6 (Python Software Foundation, Wilmington, DE, USA) and TensorFlow 2.41 (Google Brain, Mountain View, CA, USA). Model training was completed on the JupyterLab platform (https://jupyter.org/). Adam (41) was used as the model optimizer with 15 iterations, 4 batches, hand photograph size of 224×224×3, and the ReLU function was used as the activation function. The different sizes of hand photographs were adjusted to a standardized size of 224×224 in the preprocessing stage. All hand images were finally classified as remission or non-remission.

Statistical analyses

According to variable types, baseline characteristics are presented as mean ± standard deviation (SD), medians [interquartile range (IQR)], or frequencies (proportion). Statistical analyses included the two-sample Student’s t-test for normally distributed values, the Mann-Whitney U-test for continuous variables for nonparametric values, and Fisher’s exact and Pearson’s Chi-Squared test for categorical variables. All statistical analyses were performed using the R software, version 3.6.3 (R Foundation for Statistical Computing, https://www.R-project.org/).

Ethical statement

This study was conducted in accordance with the Declaration of Helsinki (as revised in 2013) and was approved by the Ethics Committee of the First Affiliated Hospital of Sun Yat-sen University. The requirement for individual consent was waived due to the retrospective nature of the study.


Results

Baseline characteristics

A total of 368 acromegaly patients were selected from the CAPA database. The detailed baseline characteristics and the long-term outcome of recurrence are shown in Table 1. The mean patient age was 36.6±9.8 years old. There were more females than males, but there was no significant difference in postoperative remission between the sexes. The median history of disease course and follow-up was 4.5 and 2.6 years, respectively. The most common clinical symptoms and signs were enlarged nose and prognathism (97.8%), overgrowth of extremities (94.0%), menstrual disturbance in females (69.4%), snoring or sleep apnea (62.5%), headaches (61.1%), hyperhidrosis (57.1%), and visual field abnormalities (52.4%). Other comorbidities were also noted in clinics, including cardiovascular diseases, impaired glucose tolerance, osteoporosis, arthritis, galactorrhea, and colonic polyps.

Table 1

Demographics and clinical characteristics of the patients (N=368)

Category Distribution
Mean age (years), mean ± SD 36.6±9.8
Male/Female 144/224
Disease duration (years), median (IQR) 5.0 (3.0–8.0)
Follow-up after initial surgery (months), median [IQR] 31 [13–60]
Clinical symptoms and signs, n (%)
   Enlarged nose and prognathism 368 (97.8)
   Overgrowth of extremities 346 (94.0)
   Menstrual disturbance (female) 156 (69.4)
   Snoring or sleep apnea 230 (62.5)
   Headache 225 (61.1)
   Hyperhidrosis 210 (57.1)
   Visual field abnormalities 193 (52.4)
   Hypertrophy of frontal bones 180 (48.9)
   Cardiovascular disease 136 (37.0)
   Hypertrichosis 133 (36.1)
   Impaired glucose tolerance 116 (31.5)
   Jaw malocclusion 93 (25.2)
   Osteoporosis 70 (19.0)
   Arthritis 60 (16.3)
   Galactorrhea 51 (13.9)
   Colonic polyps 50 (13.6)
   Colonic cancer 3 (0.8)
Chinese Pituitary Adenoma Cooperative Group, n (%) 275 (74.7)
Resident area, n (%)
   Large city 164 (44.6)
   Medium city 127 (34.5)
   Small city/town 77 (20.9)

SD, standard deviation; IQR, interquartile range.

In our study dataset (Table 2), the mean levels of initial and postoperative GH were 26.0 and 6.5 µg/L, respectively, whereas the mean level of sex-/age-normalized initial and postoperative IGF-1 were 1.4% upper limit of normal (ULN) and 0.5% ULN, respectively. As for imaging features, 54 cases (14.7%) and 314 cases (85.3%) were diagnosed with microadenomas and macroadenomas, respectively.

Table 2

Tumor characteristics and treatment (N=368)

Categories Distribution
Initial biochemical levels (mean ± SD)
   GH (μg/L) 26.0±20.5
   IGF-1 (% ULN) 1.4±0.8
Last follow-up biochemical levels (mean ± SD)
   GH (μg/L) 6.5±16.6
   IGF-1 (% ULN) 0.5±0.6
Tumor size, n (%)
   Microadenomas (<1 cm) 54 (14.7)
   Macroadenomas (≥1 cm) 314 (85.3)
Knosp score, n (%)
   Score 0 31 (8.4)
   Score 1 64 (17.4)
   Score 2 31 (8.4)
   Score 3 184 (50.0)
   Score 4 58 (15.8)
Apoplexy, n (%) 35 (9.5)
Preoperative medication, n (%) 82 (22.3)
Pathology, n (%)
   GH 301 (81.8)
   GH + PRL 49 (13.3)
   GH + TSH 3 (0.8)
   GH + ACTH 2 (0.5)
   GH + (>2 hormones) 13 (3.5)
Postoperative complications, n (%)
   Transient hyponatremia 107 (29.1)
   Transient diabetes insipidus 63 (17.1)
   Cerebrospinal fluid leak 25 (6.8)
   Meningitis 2 (0.5)
   ICA injury (intraoperative) 1 (0.3)

SD, standard deviation; GH, growth hormone; IGF-1, insulin-like growth factor 1; ULN, upper limit of normal; PRL, prolactin; TSH, thyroid stimulating hormone; ACTH, adrenocorticotropic hormone; ICA, intracranial artery.

Patients with microadenomas yielded higher remission rates (35.2%) compared to those with macroadenomas (21.7%) (Table 3), suggesting that tumor size may be associated with long-term remission. Apoplexy, different surgical approach, pathology, and place of residence did not differ significantly between the remission and non-remission subgroups. Preoperatively, 26 patients (22.9%) were on medication and showed a slightly favorable outcome compared to patients who did not receive preoperative medication (29.9% vs. 19.9%, P=0.051). Most pituitary patients with tumor invasion to the cavernous sinus (Knosp 3–4) showed lower surgical remission versus patients who did not have evidence of invasiveness (Knosp 0–2) (16.5% vs. 37.3%). Interestingly, patients who received treatment in hospitals within the Chinese Pituitary Adenoma Cooperative Group achieved a higher remission rate that those who received treatment in other hospitals (26.5% vs. 15.1%).

Table 3

A comparison of the demographics and clinical presentation between non-remission and remission cases at 3 months

Categories Surgical remission P value*
Yes (n=87) No (n=281)
Age (years), mean ± SD 38.3±9.4 36.1±9.9 0.072
Sex, n (%) 0.607
   Male 32 (36.8) 112 (39.9)
   Female 55 (63.2) 169 (60.1)
Disease duration (years), median [IQR] 5 [2–7] 5 [3–8] 0.662
Initial GH levels (μg/L), mean ± SD 25.8±20.0 26.1±20.6 0.897
Tumor size, n (%) 0.031
   Microadenomas 19 (21.8) 35 (12.5)
   Macroadenomas 68 (78.2) 246 (87.5)
Knosp score, n (%) <0.001
   0–2 47 (54.0) 79 (28.1)
   3–4 40 (46.0) 202 (71.9)
Apoplexy, n (%) 31 (9.0) 19 (9.5) 0.471
Preoperative medication, n (%) 26 (29.9) 56 (19.9) 0.051
Surgical approach, n (%) 0.326
   Transcranial surgery 5 (5.7) 12 (4.3)
   Microscopic TSS 28 (32.2) 115 (40.9)
   Endoscopic TSS 54 (62.1) 154 (54.8)
Pathology, n (%) 0.582
   GH 73 (83.9) 228 (81.1)
   GH + ACTH 1 (1.1) 1 (0.4)
   GH + PRL 11 (12.6) 38 (13.5)
   GH + TSH 1 (1.1) 2 (0.7)
   GH + (>2 hormones) 1 (1.1) 12 (4.3)

*, FDR correction. SD, standard deviation; IQR, interquartile range; GH, growth hormone; TSS, endonasal transsphenoidal surgery; ACTH, adrenocorticotropic hormone; PRL, prolactin; TSH, thyroid stimulating hormone; FDR, false discovery rate.

Classification performance of transfer learning algorithm

The hand photograph dataset was divided randomly into a training set (584 images) and a validation set (134 images). Approximately 20% of the hand photographs were used as the validation set. Epoch plots were created to depict the training procedure of the transfer learning algorithm (Figure 4). The MobileNetv2-based transfer learning algorithm was shown to predict biochemical remission with statistical accuracy in the training cohort (n=803) and the validation cohort (n=200) of 0.96 and 0.76, respectively, and a loss function value of 0.82. Receiver operating characteristic (ROC) curve analyses showed that the classification model derived from the transfer learning algorithm yielded modest discrimination power for an unfavorable outcome with an area under the curve (AUC) value of 0.605, a sensitivity of 85.4%, a specificity/recall of 35.5%, and a precision of 73.9% (Figure 5).

Figure 4 A diagram showing the transfer learning algorithm.
Figure 5 Epoch plots of the transfer learning algorithm-based predicting model performance over each epoch round.

Discussion

The primary goals of surgical therapy in acromegaly are total resection of pituitary tumors and normalization of the hypersecreting pattern of circulating GH and/or IGF-1 levels. The current study demonstrated that hand photographs can be used to predict whether patients with acromegaly will reach biochemical remission after surgery. The outcome prediction could be conducted on mobile devices such as a personal computer or mobile phone with an accuracy of 81%.

Acromegaly is the main clinical syndrome of GH pituitary adenomas. More than 95% of acromegaly patients present with abnormalities in their pituitary glands, with other causes being ectopic GH adenomas or pituitary hyperplasia. About 70% of acromegaly patients are formally diagnosed with aggressive macroadenomas, with coexisting conditions such as headaches and invisible acral and soft tissue changes (12). Some patients may initially consult a dentist, orthopedist, rheumatologist, or cardiologist, resulting in delayed diagnosis of acromegaly for an average of about 10 years for men and 8 years for women after the onset of symptoms (42). Therefore, there is an urgent need for early diagnosis in this population. Regular follow-up over a long period after medical treatment is essential for the overall management of patients with acromegaly (2). The ultimate goal of surgery is to achieve long-term biochemical remission. However, most acromegaly patients do not achieve remission after initial surgery. A recent large cohort study of 546 surgical patients in China revealed that the short-term remission rate at 3 months after their initial surgery was only 40% (12). Many of these patients required further medical treatment, but due to the coronavirus disease of 2019 (COVID-19) pandemic or other unpredicted reasons, their follow-up schedules were often not fulfilled. Therefore, there is a need to develop a method that can monitor a patient’s biochemical status through mobile devices.

Notably, almost all patients with acromegaly have typical appearance changes, including enlarged nose and prognathism, overgrowth of extremities, hypertrophy of frontal bones, and jaw malocclusion (5,12). These features provide us with rich resources for intuitive diagnosis in addition to medical images and laboratory test results. In our previous study, we applied the deep convolution neural network (DCNN) on hand photographs to establish a model that could effectively differentiate patients with acromegaly and normal individuals with an accuracy of 98.3% (19). Other studies have applied facial recognition techniques to diagnose acromegaly with a sensitivity as high as 92.8–96% (43-46). The above findings suggest that it is effective and feasible to apply AI in the classification of acromegaly.

In this study, the AI-based algorithm successfully classified patients into a remission group and a non-remission group with high accuracy, by identifying the different areas of interest in the hand photographs of surgical patients with acromegaly. This demonstrated the powerful ability of the MobileNetv2-based transfer learning algorithm to identify subtle differences in hand photographs of acromegaly patients. In 2011, Miller et al. developed a computer program that could classify facial photographs of acromegaly patients from those of normal cases, which was much more accurate than the classification by 10 generalist physicians, suggesting it is theoretically possible to diagnose acromegaly at an early stage by extracting features from facial photographs (47). In 2018, Kong et al. trained several machine learning models on a large sample size of facial photographs from 527 acromegaly patients and 596 normal cases. The models were then tested on a different dataset consisting of 114 acromegaly patients and 128 controls (44). The best results of their proposed model showed a sensitivity of 96%, a specificity of 96%, a positive predictive value (PPV) of 96%, and a negative predictive value (NPV) of 95%, which overperformed 9 board-certified physicians specializing in acromegaly. In 2020, Meng et al. used 3D facial imaging techniques and a machine learning approach to identify facial features to diagnose patients with acromegaly, suggesting that 3D imaging can enable accurate quantification of facial characteristics in patients with acromegaly (43).

However, due to privacy concerns, facial recognition technology may be not practical in the real world. To avoid this limitation, our team used hand photographs to assist the clinical diagnosis of acromegaly and showed a sensitivity of 0.983, a specificity of 0.920, a PPV of 0.966, an NPV of 0.958, and an F1-score of 0.974, thereby demonstrating the potential application of machine learning techniques to detect acromegaly using hand photographs (19). We herein used a new machine learning technique, the MobileNetv2-based transfer learning algorithm, to extract identical features from hand photographs to predict biochemical remission in surgical acromegaly patients with high accuracy. With the wide-spread use of smart mobile phones, it may be possible to develop an application that integrates all possible de-identified data including hand photographs and other patient self-reported symptoms and signs to predict biochemical remission after surgery. This information may be useful to determine whether patients need to consult an endocrinologist or a neurosurgeon who specializes in pituitary diseases.

Although there were some interesting findings, our study had several limitations that should be addressed. First, this was a retrospective analysis. Due to the incompleteness of data from earlier years, most cases included in this study were newer cases. Therefore the median follow-up time was relatively short. Also, the lack of preoperative photographs hindered us from monitoring the changes in the hands after surgery. With the gradual development of the database, related studies in the future could involve longer follow-up time and more complete follow-up information. Second, assays used for hormone measurements might have changed over the extended duration of this study and IGF-1 was not routinely measured in some hospitals, which may have led to difficulties in the differentiation of biochemical remission. Third, the missing data of IGF-1 levels may have led to selection bias which may affect our model performance. However, the CAPA database allows us to investigate the predictive value of hand photographs on biochemical remission for surgical outcomes using state-of-the-art AI techniques and is the largest available hand photograph dataset of acromegaly in China.


Conclusions

This investigation demonstrated that analysis of hand photographs from acromegaly patients can predict biochemical remission in such patients postoperatively. The findings herein show that the MobileNetv2-based transfer learning algorithm is efficient and feasible in the earlier recognition of biochemical remission in acromegaly patients after surgery. Future prospective, multicenter, large cohort acromegaly studies are warranted across various under-represented populations or regions in China to further validate the efficacy of this model.


Acknowledgments

Funding: This research was supported by Clinical Research Project of The East Division (the First Affiliated Hospital, Sun Yat-sen University, No. 2019004 to Wenli Chen); Guangdong Basic and Applied Basic Research Foundation (No. 2023A1515011644 to Shun Yao); Chinese Postdoctoral Science Foundation (No. 2019M663271 to Shun Yao); and the National Natural Science Foundation of China (No. 82203179 to Wenli Chen).


Footnote

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at https://qims.amegroups.com/article/view/10.21037/qims-22-1101/coif). SY was supported by the Guangdong Basic and Applied Basic Research Foundation (No. 2023A1515011644) and the Chinese Postdoctoral Science Foundation (No. 2019M663271). WC was supported by the Clinical Research Project of The East Division (the First Affiliated Hospital, Sun Yat-sen University, No. 2019004) and the National Natural Science Foundation of China (No. 82203179). The other authors have no conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. This study was conducted in accordance with the Declaration of Helsinki (as revised in 2013). The study was approved by the Ethics Committee of the First Affiliated Hospital of Sun Yat-sen University and the requirement for individual consent for this retrospective analysis was waived.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.


References

  1. Kamusheva M, Vandeva S, Mitov K, Parvanova A, Pesheva M, Ganov N, Rusenova Y, Marinov L, Getova V, Elenkova A, Petrova G. Adherence to Acromegaly Treatment and Analysis of the Related Factors-A Real-World Study in Bulgaria. Pharmaceutics 2023; [Crossref] [PubMed]
  2. Colao A, Grasso LFS, Giustina A, Melmed S, Chanson P, Pereira AM, Pivonello R. Acromegaly. Nat Rev Dis Primers 2019;5:20. [Crossref] [PubMed]
  3. Donoho DA, Bose N, Zada G, Carmichael JD. Management of aggressive growth hormone secreting pituitary adenomas. Pituitary 2017;20:169-78. [Crossref] [PubMed]
  4. Störmann S, Schilbach K. Delving into Acromegaly. J Clin Med 2023; [Crossref] [PubMed]
  5. Melmed S. Medical progress: Acromegaly. N Engl J Med 2006;355:2558-73. [Crossref] [PubMed]
  6. Melmed S, Bronstein MD, Chanson P, Klibanski A, Casanueva FF, Wass JAH, Strasburger CJ, Luger A, Clemmons DR, Giustina A. A Consensus Statement on acromegaly therapeutic outcomes. Nat Rev Endocrinol 2018;14:552-61. [Crossref] [PubMed]
  7. Cardinal T, Rutkowski MJ, Micko A, Shiroishi M, Jason Liu CS, Wrobel B, Carmichael J, Zada G. Impact of tumor characteristics and pre- and postoperative hormone levels on hormonal remission following endoscopic transsphenoidal surgery in patients with acromegaly. Neurosurg Focus 2020;48:E10. [Crossref] [PubMed]
  8. Guo X, Zhang R, Zhang D, Wang Z, Gao L, Yao Y, Deng K, Bao X, Feng M, Xu Z, Yang Y, Lian W, Wang R, Ma W, Xing B. Determinants of immediate and long-term remission after initial transsphenoidal surgery for acromegaly and outcome patterns during follow-up: a longitudinal study on 659 patients. J Neurosurg 2022; Epub ahead of print. [Crossref] [PubMed]
  9. Tomasik A, Stelmachowska-Banaś M, Maksymowicz M, Czajka-Oraniec I, Raczkiewicz D, Zieliński G, Kunicki J, Zgliczyński W. Clinical, hormonal and pathomorphological markers of somatotroph pituitary neuroendocrine tumors predicting the treatment outcome in acromegaly. Front Endocrinol (Lausanne) 2022;13:957301. [Crossref] [PubMed]
  10. Ferrés A, Reyes L, Di Somma A, Topczewski T, Mosteiro A, Guizzardi G, De Rosa A, Halperin I, Hanzu F, Mora M, Alobid I, Aldecoa I, Bargalló N, Enseñat J. The Prognostic-Based Approach in Growth Hormone-Secreting Pituitary Neuroendocrine Tumors (PitNET): Tertiary Reference Center, Single Senior Surgeon, and Long-Term Follow-Up. Cancers (Basel) 2022; [Crossref] [PubMed]
  11. Koylu B, Firlatan B, Sendur SN, Oguz SH, Dagdelen S, Erbas T. Giant growth hormone-secreting pituitary adenomas from the endocrinologist's perspective. Endocrine 2023;79:545-53. [Crossref] [PubMed]
  12. Yao S, Chen WL, Tavakol S, Akter F, Catalino MP, Guo X, Luo J, Zeng AL, Zekelman L, Mao ZG, Zhu YH, Wu QZ, Laws ER Jr, Bi WL, Wang HJ. Predictors of postoperative biochemical remission in acromegaly. J Neurooncol 2021;151:313-24. [Crossref] [PubMed]
  13. Bona C, Prencipe N, Berton AM, Bioletto F, Parasiliti-Caprino M, Gasco V, Ghigo E, Grottoli S. Mean GH profile is more accurate than single fasting GH in the evaluation of acromegaly disease control during somatostatin receptor ligands therapy. J Endocrinol Invest 2022;45:1955-65. [Crossref] [PubMed]
  14. Nie D, Fang Q, Wong W, Gui S, Zhao P, Li C, Zhang Y. The effect of endoscopic transsphenoidal somatotroph tumors resection on pituitary hormones: systematic review and meta-analysis. World J Surg Oncol 2023;21:71. [Crossref] [PubMed]
  15. Delemer B, Chanson P, Foubert L, Borson-Chazot F, Chabre O, Tabarin A, Weryha G, Cortet-Rudelli C, Raingeard I, Reznik Y, Reines C, Bisot-Locard S, Castinetti F. Patients lost to follow-up in acromegaly: results of the ACROSPECT study. Eur J Endocrinol 2014;170:791-7. [Crossref] [PubMed]
  16. Harkey T, Baker D, Hagen J, Scott H, Palys V. Practical methods for segmentation and calculation of brain volume and intracranial volume: a guide and comparison. Quant Imaging Med Surg 2022;12:3748-61. [Crossref] [PubMed]
  17. Liang J, Yang C, Zeng M, Wang X. TransConver: transformer and convolution parallel network for developing automatic brain tumor segmentation in MRI images. Quant Imaging Med Surg 2022;12:2397-415. [Crossref] [PubMed]
  18. Qian Y, Qiu Y, Li CC, Wang ZY, Cao BW, Huang HX, Ni YH, Chen LL, Sun JY. A novel diagnostic method for pituitary adenoma based on magnetic resonance imaging using a convolutional neural network. Pituitary 2020;23:246-52. [Crossref] [PubMed]
  19. Duan C, Chen W, Yao S, Mao Z, Wang Z, Hu B, Jiao H, Zhu D, Zhu Y. Automatic Detection for Acromegaly Using Hand Photographs: A Deep-Learning Approach. IEEE Access 2021;9:2846-53. doi: 10.1109/ACCESS.2020.3046878.10.1109/ACCESS.2020.3046878
  20. Li H, Zhao Q, Zhang Y, Sai K, Xu L, Mou Y, Xie Y, Ren J, Jiang X. Image-driven classification of functioning and nonfunctioning pituitary adenoma by deep convolutional neural networks. Comput Struct Biotechnol J 2021;19:3077-86. [Crossref] [PubMed]
  21. Peng A, Dai H, Duan H, Chen Y, Huang J, Zhou L, Chen L. A machine learning model to precisely immunohistochemically classify pituitary adenoma subtypes with radiomics based on preoperative magnetic resonance imaging. Eur J Radiol 2020;125:108892. [Crossref] [PubMed]
  22. Rui W, Qiao N, Wu Y, Zhang Y, Aili A, Zhang Z, Ye H, Wang Y, Zhao Y, Yao Z. Radiomics analysis allows for precise prediction of silent corticotroph adenoma among non-functioning pituitary adenomas. Eur Radiol 2022;32:1570-8. [Crossref] [PubMed]
  23. Shu XJ, Chang H, Wang Q, Chen WG, Zhao K, Li BY, Sun GC, Chen SB, Xu BN. Deep Learning model-based approach for preoperative prediction of Ki67 labeling index status in a noninvasive way using magnetic resonance images: A single-center study. Clin Neurol Neurosurg 2022;219:107301. [Crossref] [PubMed]
  24. Park YW, Eom J, Kim S, Kim H, Ahn SS, Ku CR, Kim EH, Lee EJ, Kim SH, Lee SK. Radiomics With Ensemble Machine Learning Predicts Dopamine Agonist Response in Patients With Prolactinoma. J Clin Endocrinol Metab 2021;106:e3069-77. [Crossref] [PubMed]
  25. Galm BP, Buckless C, Swearingen B, Torriani M, Klibanski A, Bredella MA, Tritos NA. MRI texture analysis in acromegaly and its role in predicting response to somatostatin receptor ligands. Pituitary 2020;23:212-22. [Crossref] [PubMed]
  26. Kocak B, Durmaz ES, Kadioglu P, Polat Korkmaz O, Comunoglu N, Tanriover N, Kocer N, Islak C, Kizilkilic O. Predicting response to somatostatin analogues in acromegaly: machine learning-based high-dimensional quantitative texture analysis on T2-weighted MRI. Eur Radiol 2019;29:2731-9. [Crossref] [PubMed]
  27. Machado LF, Elias PCL, Moreira AC, Dos Santos AC, Murta LO Junior. MRI radiomics for the prediction of recurrence in patients with clinically non-functioning pituitary macroadenomas. Comput Biol Med 2020;124:103966. [Crossref] [PubMed]
  28. Galm BP, Martinez-Salazar EL, Swearingen B, Torriani M, Klibanski A, Bredella MA, Tritos NA. MRI texture analysis as a predictor of tumor recurrence or progression in patients with clinically non-functioning pituitary adenomas. Eur J Endocrinol 2018;179:191-8. [Crossref] [PubMed]
  29. Zhang Y, Ko CC, Chen JH, Chang KT, Chen TY, Lim SW, Tsui YK, Su MY. Radiomics Approach for Prediction of Recurrence in Non-Functioning Pituitary Macroadenomas. Front Oncol 2020;10:590083. [Crossref] [PubMed]
  30. Dai C, Sun B, Wang R, Kang J. The Application of Artificial Intelligence and Machine Learning in Pituitary Adenomas. Front Oncol 2021;11:784819. [Crossref] [PubMed]
  31. Thombare M, Chandwadkar R. A Survey on Transfer Learning for Brain Computer Interfaces using Data Mining Techniques. International Journal of Innovative Research in Science Engineering and Technology 2020;9:3810-7.
  32. Guo X, Wang K, Yu S, Gao L, Wang Z, Zhu H, Xing B, Zhang S, Dong D. Quality of Life and its Determinants in Patients With Treated Acromegaly: A Cross-Sectional Nationwide Study in China. J Clin Endocrinol Metab 2021;106:211-25. [Crossref] [PubMed]
  33. Solomon E, Brănișteanu D, Dumbravă A, Solomon RG, Kiss L, Glod M, Preda C. Executive functioning and quality of life in acromegaly. Psychol Res Behav Manag 2019;12:39-44. [Crossref] [PubMed]
  34. Pertichetti M, Serioli S, Belotti F, Mattavelli D, Schreiber A, Cappelli C, Padovani A, Gasparotti R, Nicolai P, Fontanella MM, Doglietto F. Pituitary adenomas and neuropsychological status: a systematic literature review. Neurosurg Rev 2020;43:1065-78. [Crossref] [PubMed]
  35. Guo X, Wang K, Yu S, Gao L, Wang Z, Zhu H, Xing B, Zhang S, Dong D. Patient Characteristics, Diagnostic Delays, Treatment Patterns, Treatment Outcomes, Comorbidities, and Treatment Costs of Acromegaly in China: A Nationwide Study. Front Endocrinol (Lausanne) 2020;11:610519. [Crossref] [PubMed]
  36. Duan L, Wang SR, Zhu HJ, Wang RZ. Updated key points of Chinese Consensus for the Diagnosis and Treatment of Acromegaly (2021 edition). Zhonghua Yi Xue Za Zhi 2021;101:2111-4. [PubMed]
  37. Otsu N. A Threshold Selection Method from Gray-Level Histograms. IEEE Transactions on Systems, Man, and Cybernetics 1979;9:62-6. [Crossref]
  38. Sandler M, Howard A, Zhu M, Zhmoginov A, Chen LC. MobileNetV2: Inverted Residuals and Linear Bottlenecks. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018:4510-20.
  39. Chollet F. Xception: Deep Learning with Depthwise Separable Convolutions. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017:1800-7.
  40. Saha M, Chakraborty C, Racoceanu D. Efficient deep learning model for mitosis detection using breast histopathology images. Comput Med Imaging Graph 2018;64:29-40. [Crossref] [PubMed]
  41. Kingma DP, Ba J. Adam: A Method for Stochastic Optimization. Available online: https://doi.org/10.48550/arXiv.1412.6980
  42. Petrossians P, Daly AF, Natchev E, Maione L, Blijdorp K, Sahnoun-Fathallah M, et al. Acromegaly at diagnosis in 3173 patients from the Liège Acromegaly Survey (LAS) Database. Endocr Relat Cancer 2017;24:505-18. [Crossref] [PubMed]
  43. Meng T, Guo X, Lian W, Deng K, Gao L, Wang Z, Huang J, Wang X, Long X, Xing B. Identifying Facial Features and Predicting Patients of Acromegaly Using Three-Dimensional Imaging Techniques and Machine Learning. Front Endocrinol (Lausanne) 2020;11:492. [Crossref] [PubMed]
  44. Kong X, Gong S, Su L, Howard N, Kong Y. Automatic Detection of Acromegaly From Facial Photographs Using Machine Learning Methods. EBioMedicine 2018;27:94-102. [Crossref] [PubMed]
  45. Wei R, Jiang C, Gao J, Xu P, Zhang D, Sun Z, Liu X, Deng K, Bao X, Sun G, Yao Y, Lu L, Zhu H, Wang R, Feng M. Deep-Learning Approach to Automatic Identification of Facial Anomalies in Endocrine Disorders. Neuroendocrinology 2020;110:328-37. [Crossref] [PubMed]
  46. Schneider HJ, Kosilek RP, Günther M, Roemmler J, Stalla GK, Sievers C, Reincke M, Schopohl J, Würtz RP. A novel approach to the detection of acromegaly: accuracy of diagnosis by automatic face classification. J Clin Endocrinol Metab 2011;96:2074-80. [Crossref] [PubMed]
  47. Miller RE, Learned-Miller EG, Trainer P, Paisley A, Blanz V. Early diagnosis of acromegaly: computers vs clinicians. Clin Endocrinol (Oxf) 2011;75:226-31. [Crossref] [PubMed]
Cite this article as: Wang M, Duan C, Yao S, Chen J, Zhang S, Wang Z, Hu B, Mao Z, Wang H, Zhu Y, Chen W. Using 2-dimensional hand photographs to predict postoperative biochemical remission in acromegaly patients: a transfer learning approach. Quant Imaging Med Surg 2023;13(6):3747-3759. doi: 10.21037/qims-22-1101

Download Citation