Deep learning-enhanced Colmap for 3D reconstruction and segmentation of facial port-wine stains for comprehensive evaluation

Introduction

Port-wine stains (PWS) are a congenital microvascular disorder primarily characterized by the abnormal dilation of capillaries in the upper dermis of the skin (1,2). PWS often occurs on the face and neck, typically appearing in shades of bright red, dark red, or purple-red, and it may worsen over time without intervention (3-5). The challenge in treating PWS is removing the dilated and abnormal capillary networks in the superficial dermis while preserving the overlying epidermis and the underlying deep dermal tissue, to achieve scar-free treatment. In 1991, Gu et al. pioneered vascular-targeted photodynamic therapy (V-PDT) (6). This method involves the intravenous injection of photosensitizers preferentially absorbed by vascular dermal cells. Subsequently, a specific wavelength laser irradiates the affected area, causing the photosensitive destruction of the malformed capillary network, thereby achieving therapeutic outcomes. The normal epidermis remains intact because it does not absorb the photosensitizers, and the deeper dermis is unaffected due to the laser’s limited penetration depth (7-9).

Currently, V-PDT is the preferred treatment for PWS in China due to its favorable therapeutic outcomes (10-13). However, the structure of PWS lesions is complex and varied, and some patients may experience limited therapeutic response (14-16). To ensure efficient V-PDT treatment, it is crucial to maintain a uniform light dose across the entire treatment area. Given that PWS lesions are not flat, a single treatment light spot cannot uniformly cover the entire lesion. Thus, treatment often involves manually dividing the lesion into several flat regions for sequential light exposure, which is time-consuming and laborious. Moreover, later-treated areas may experience reduced efficacy due to photosensitizer metabolism. Additionally, treatment assessment relies on subjective visual observation, lacking precision. To overcome these challenges, obtaining three-dimensional (3D) data on PWS lesions is essential. Objective evaluation can be made by quantifying the 3D surface area and lesion color changes pre- and post-V-PDT. Furthermore, combining this with optical field modulation allows for a customized light field tailored to the lesion 3D topography, eliminating the need for manual segmentation and sequential light exposure, thereby achieving optimized outcomes. Therefore, 3D reconstruction and lesion segmentation of PWS patients are highly significant.

Frigerio et al. employed a high-resolution 3D six-camera system for 3D facial reconstructions in PWS patients (17). Their experimental results confirmed the feasibility of using 3D reconstruction technology to evaluate PWS quantitatively. However, the use of multiple cameras in this scheme adds system complexity and necessitates manual delineation of lesion boundaries to calculate the lesion surface area. Zhang et al. utilized 3D scanning technology to obtain patients’ 3D point cloud data and projected the point cloud onto two-dimensional (2D) planes. By integrating color features, they segmented the lesion area using simple linear iterative clustering (SLIC) superpixel segmentation. Subsequently, they applied the greedy projection triangulation algorithm to reconstruct the 3D surface of the lesion area and calculated the irregular surface area of the lesion. The results indicated that this method is capable of segmenting the lesion area and can evaluate the recovery trend of the lesion area based on changes in chromaticity distribution within the color space (18). However, the segmentation algorithm lacks robustness, and the color evaluation method is cumbersome. Additionally, due to limitations in the accuracy and color fidelity of the 3D scanner, the quality of the reconstructed and segmented point clouds is suboptimal. Ke et al. proposed a study combining 3D scanning and deep learning to segment PWS lesion areas (19). In their study, a 3D scanner was used to obtain 3D images and create texture maps. The improved DeepLabV3+ network was then used to extract PWS lesions from the texture map. The experimental results demonstrated that this method enabled accurate segmentation and area quantification of PWS lesions. However, the majority of the network’s training data originated from a mannequin head with an artificial PWS lesion, with limited data from actual patients. The PWS designed for the model has a simplistic structure and a single color, posing potential challenges for clinical application.

The limitations of existing schemes include: (I) low-quality 3D reconstructions and suboptimal color fidelity; (II) absence of real patient data to validate the effectiveness and limitations of 3D reconstruction and segmentation methodologies; (III) to streamline computations, the 3D point cloud is projected onto a 2D surface before segmentation, unavoidably introducing projection errors. In addition to the limitations of existing schemes, the 3D reconstruction and segmentation of PWS are challenged by the diverse shapes of lesions, color heterogeneity, and indistinct boundaries.

Considering the limitations above and the goal of achieving optimal 3D lesion reconstruction, performing 2D lesion segmentation first and then 3D lesion reconstruction is a good choice. First, it is easier to obtain high-quality, high-color fidelity 2D images, which provides a good guarantee for the subsequent reconstruction of high-quality 3D lesions. Second, it avoids the error of the 3D to 2D projection. Finally, deep learning-based image segmentation methods have exhibited exceptional performance (20-22) and can accurately segment 2D lesion areas and reduce 3D reconstruction errors. However, most PWS 2D segmentation models neglect the independence of color space (23-25), making it challenging to achieve precise segmentation in areas where lesions blend with normal skin. Therefore, our primary focus was on improving the accuracy of 2D lesion segmentations. In 2023, our research group proposed a multi-color space adaptive fusion network (M-CSAFN), which integrates various color models. The M-CSAFN model minimizes interference among color spaces and enhances their synergistic integration in PWS segmentation, enabling superior 2D segmentation outcomes. It attains Dice and Jaccard scores of 92.29% and 86.14% on clinical images, respectively (26).

In this paper, we integrated the M-CSAFN segmentation algorithm with the Colmap 3D reconstruction algorithm to develop a strategy that involves segmenting PWS in 2D and subsequently reconstructing its 3D morphology. To validate the robustness and clinical applicability of our approach, we collected data and performed 3D reconstruction on 17 PWS patients. We established a standard image acquisition system with uniform and constant illumination to capture high-quality, color-accurate 2D sequence images. This process laid a solid foundation for the Colmap algorithm to generate high-quality, color-accurate 3D point clouds and record prior 3D information. De-noising and smoothing algorithms were used to further refine the 3D point clouds (27). By registering the 3D point cloud with the one generated by the structured light camera, we found that the average root mean square error (RMSE) of point cloud registration was 0.9611 mm, indicating the high accuracy of our 3D point cloud reconstruction. Next, the M-CSAFN model was used to segment the 2D lesions from the 2D images accurately. The 2D lesions and their prior information on the corresponding 3D point cloud models were then input into the Colmap algorithm again to facilitate rapid reconstruction of the 3D lesion point cloud, reducing the computational complexity. Finally, the 3D lesion point cloud was further converted into a 3D mesh model using the Poisson surface reconstruction algorithm, and the contrastive language image pretraining (CLIP) similarity alongside the surface area of the lesion was computed for quantitative evaluation. The results obtained from patients and facial models indicated that the average relative error between the calculated 3D surface area of the lesion and the ground truth area was only 4.59%.

Methods

Figure 1 provides an overview of our proposed method for segmenting PWS in 2D and subsequently reconstructing its 3D morphology. The method is mainly divided into two parts: (I) the M-CSAFN model performs segmentation of lesions in 2D images; (II) the Colmap algorithm reconstructs the segmented 2D lesion images from different angles into a 3D lesion model and converts it into a mesh model suitable for calculating the 3D surface area using the Poisson surface reconstruction algorithm. Notably, the 2D images come from a standard image acquisition system that we built with uniform and constant illumination to ensure high-quality, color-accurate images. 3D lesion reconstruction also requires prior 3D information about the patient. The prior 3D information about the patient refers to the geometric information and camera parameters obtained by performing 3D reconstruction on the patient’s multi-angle 2D images using the Colmap algorithm. The geometric information is represented by the 3D point cloud of the patient’s entire face, and the camera parameters comprise the camera pose for each input image (i.e., the rotation matrix R and translation vector t) and the camera’s intrinsic parameter matrix K. Therefore, our overall workflow consists of computing this prior 3D information from the original 2D image sequence using Colmap. Subsequently, we apply the M-CSAFN model to generate 2D lesion segmentation results and reuse the prior 3D information to perform efficient 3D reconstruction of only the lesion region, thereby avoiding repeated and computationally expensive Structure from Motion (SfM) operations.

Figure 1 Overview of 2D segmentation, 3D reconstruction, and 3D mesh model conversion of port-wine stains.

Standard image acquisition system

To accurately reconstruct the 3D morphology of the PWS and faithfully restore its color and facial features, a standardized image acquisition system with uniform and constant illumination was built, and the schematic diagram along, with the actual shooting setup, is presented in Figure 2A,2B, respectively. Two standard fill lights (NANLITE Forza 300B; Nanlite, Shantou, China) with soft umbrellas were positioned symmetrically on each side of the head. The two fill lights were angled at 45° from the central horizontal axis of the head to illuminate the face, with a color temperature set to 6500K. A high-definition camera (Canon 5D Mark IV; Tokyo, Japan) was positioned between the two fill lights, aligned with the height of the facial area, and angled about 10° from the central horizontal axis. This setup ensures more uniform illumination when fill lights are angled at 45°. During data collection, the camera was stabilized on a tripod, while patients rotated at a constant speed on a fixed seat, capturing images from various angles to ensure stability during shooting. To ensure color accuracy in captured images, we placed a monochromatic background wall and a 48-color standard colorimetric chart in the background. Each captured image includes the 48-color standard colorimetric chart. After capturing the patient image, each image is entered into the DataColor (Lawrenceville, NJ, USA) Spyder Checkr color calibration program. By comparing each image to the standard colorimetric chart, the program calculates color deviations and applies color correction to each image individually. To minimize interference, the standard colorimetric chart area is cropped from the image after color correction.

Figure 2 Schematic diagram of a standard image acquisition system, actual scene, and Colmap 3D reconstruction workflow. (A) Schematic diagram of a standard image acquisition system, where α is about 45° and β is about 10°. (B) Actual scene of a standard image acquisition system. (C) Colmap 3D reconstruction workflow. MVS, Multi-View Stereo; SfM, Structure from Motion.

Dataset

A total of 5,965 2D images were collected from 17 patients (Table S1) with PWS to generate their 3D point cloud data. Each image underwent color correction to ensure the generation of high-quality, color-accurate 3D point clouds. Since the color, shape, and location complexity of lesions vary from patient to patient, as well as factors such as patient movement and degree of cooperation during the acquisition process, the number of images finally acquired for each patient is slightly different. However, the number of valid images collected for each patient is sufficient for 3D reconstruction. The patients’ ages ranged from 10 to 54 years, comprising 9 males and 8 females: 7 with red type, 8 with purple-red type, and 2 with thickened type. Additionally, three patients (cases 9, 16, 17) were scanned from different angles using a structured light camera (Revopoint Surface 120; Shenzhen, China), and the resulting point clouds were merged using Revo Scan 5 software to obtain complete facial point clouds. The 3D point cloud reconstructed using our method was registered with the 3D point cloud generated by structured light reconstruction to validate the accuracy of our proposed reconstruction approach.

The M-CSAFN segmentation model was trained on a dataset comprising 1,193 patients with PWS, totaling 1,413 images, which is the largest PWS segmentation dataset known to us. The data annotation was conducted under the expert guidance of clinicians. The collected images were randomly split into training and test sets in a 6:1 ratio.

In addition, color data of purple-red type lesions was extracted, and 9 graphic files of varying shapes and sizes were created using Microsoft Visio (Redmond, Washington, USA), then printed on white paper to produce lesion patches that simulate the appearance of PWS lesions on the facial surface. Combined with the paper dimensions, determine the true surface area of the lesion patch. To validate the accuracy of our proposed 3D lesion area calculation method, we compared the 3D surface areas of different lesions on the facial model, as calculated using our method, with the true surface area values.

This study was conducted in accordance with the Declaration of Helsinki and its subsequent amendments. This study was approved by the Ethics Committee of Chinese PLA General Hospital (No. S2022-143-01). The patient data used in this study were sourced from the Department of Laser Medicine, Chinese PLA General Hospital. Informed consent was obtained from all patients or their legal guardians. All images containing identifiable facial features included in this article have been published with the informed consent of the patients/participants.

Colmap 3D reconstruction

Colmap 3D reconstruction utilizes the SfM algorithm and the Multi-View Stereo (MVS) algorithm to generate a 3D model from a 2D image sequence (28,29). The 3D model generated by the Colmap system is a point cloud with color and positional data, which simplifies subsequent lesion area calculation and evaluation. The reconstruction process is shown in Figure 2C. In the SfM algorithm, feature extraction and matching are conducted on the acquired image sequence, and matching points satisfying geometric constraints are retained. The geometrically verified matching image pairs are selected as the initialization model and used to calculate the camera’s posture and position. Then, 3D coordinates are derived for the matching point pairs through triangulation. The formula for triangulation is as follows:

P=argminP(‖p1−π(M1,P)‖2+‖p2−π(M2,P)‖2)[1]

where, P is the optimal 3D point obtained by iterative optimization on the right side of the formula, and p₁ and p₂ are the corresponding 2D projection points in the two cameras. π(M, P) is the projection function of the camera, which is used to map a 3D point P in the world coordinate system to a 2D image plane. The projection matrix M=K[R|T], so M₁ and M₂ are the projection matrices of the two cameras. K and [R|T] are the intrinsic and extrinsic matrices of the camera, respectively, and their formulas are as follows:

K=[fx0cx0fycy001][2]

[R|T]=[Rt01][3]

where, f_x, f_y are the focal lengths of the camera, c_x, c_y are the coordinates of the center of the optical axis in the image coordinate system, R is the rotation matrix from the world coordinate system to the camera coordinate system, and t is the translation vector.

After initialization, new images continue to be added to the model. Uses the Perspective-n-Point algorithm for each newly added image to compute the camera’s extrinsic parameters based on the known 3D points and their 2D projections in the new image. Once new images are added and more 3D points are generated, local bundle adjustment is applied to optimize the pose of the most recently added camera and the corresponding 3D point positions. In addition, global bundle adjustment is periodically applied to optimize the poses of all cameras and the positions of all 3D points, minimizing the reprojection error across the system. Finally, sparse reconstruction is completed, and the poses of all matching image sequences are output, enabling the generation of sparse point clouds that serve as the foundational model for 3D reconstruction.

Based on sparse reconstruction, uses the MVS algorithm to estimate the depth of matched images and generate depth maps, which are then fused with the sparse point cloud to create a dense point cloud through an interpolation method:

P(x,y,z)=(1−u)(1−v)P1+u(1−v)P2+(1−u)vP3+uvP4[4]

where, P(x,y,z) is the interpolated point, P₁, P₂, P₃, P₄ are the sparse points of the 4 nearest neighbors in the neighborhood, and the neighborhood coordinate range is [x₁, x₂], [y₁, y₂]. u=x−x1x2−x1, v=y−y1y2−y1, u and v represent the normalized offsets in the horizontal and vertical directions in the local neighborhood, respectively, reflecting the position ratio of the target point in the vertical interval of the neighborhood. For the generated dense point cloud, the optimization is still carried out by minimizing the reprojection error. Since the camera’s intrinsic and extrinsic parameters have been calculated by the SfM algorithm, the dense point cloud is optimized by adjusting the position of the point. It is worth noting that the point cloud’s color information is derived from the input 2D images. A standard image acquisition system is used to capture color-accurate 2D images, ensuring that the generated point cloud retains accurate color information. However, the quality of dense point clouds is generally poor, necessitating further quality optimization of the point cloud.

M-CSAFN image segmentation network model

The overall architecture of the M-CSAFN model is shown in the upper half of Figure 1, comprising two main components: multi-color space detection and adaptive fusion. Multi-color space detection utilizes a six-branch structure based on U-Net (30), with each branch representing a distinct color space. The input Red-Green-Blue (RGB) image is mapped to six different color spaces through the color conversion layer to enhance the representation of color information in the image. Due to the use of six parallel U-Net-based encoder-decoder networks, the model parameter count is sixfold that of a single-branch network, requiring substantial video memory and making deployment on low-configuration hardware challenging. An excessive number of parameters also increases the risk of model overfitting. To address this, the dense dilated residual (DDR) block is introduced to substitute the stacked convolutional layers at the base of the traditional U-Net architecture, to reduce model parameters and mitigate overfitting (DDR contains 32 basic dilated residual units). Adaptive fusion is a key step in the M-CSAFN framework that combines prediction results from different color-space branches, enhancing segmentation performance through complementary information across multiple color spaces. This adaptively weighted combination strategy first assigns an initial weight value to each branch, with weight values continuously updated during network training until convergence. The calculation method of the model output is:

Ypred=sigmod(∑Cωc⋅XC+b)[5]

where, ωc is used to control the weight of each color branch, and its initial value is set to 1/6. b is a bias term used to balance the system error, and its initial value is set to 0. The weight value is updated as the network is trained. When training, M-CSAFN trains the network by constructing a combination of binary cross entropy (BCE) loss function and structure similarity index measure (SSIM) loss function. Among them, the BCE loss function is defined as:

LBCE=−1N∑i=1N(yi⋅log(pi)+(1−yi)⋅log(1−pi))[6]

where, yi∈[0,1] represents the label of pixel i in y_true; pi∈[0,1] represents the predicted probability value of pixel i in y_pred; N is the total number of pixels in y_true. The structural similarity loss function is defined as:

LSSIM(X,Ytrue,Ypred)=1−mean(SSIM(X⊙Ytrue,X⊙Ypred))+std(SSIM(X⊙Ytrue,X⊙Ypred))[7]

where, the ⊙ operation is a dot product operation. The segmented image of the lesion can be obtained by operating the input image X with Y_true. The final loss function is defined as:

Loss(X,Ytrue,Ypred)=λLBCE+(1−λ)⋅LSSIM(X,Ytrue,Ypred)[8]

where, λ is the balance coefficient, which is set to 0.5 based on experience.

The deep learning model implemented was based on the TensorFlow 2.5 framework, utilizing Python version 3.8.10. The training environment comprised Linux Ubuntu 18.04, with an NVIDIA GeForce RTX 4090 graphics card featuring 24GB of video memory (Santa Clara, CA, USA). The model parameters were configured with a batch size of 2 and an epoch count of 300. The Adam optimizer was employed for stochastic gradient descent; the initial learning rate was set to 0.001, the termination learning rate was established at 10⁻⁸, and the hyperbolic tangent function facilitated dynamic learning rate adjustment. The dynamic learning rate enables the model to converge quickly in the early stages of training and allows for finer adjustments and refinement of model weights in the later stages of training.

Results

3D point cloud reconstruction, optimization, and mesh model conversion

The Colmap algorithm is used to perform a 3D reconstruction of the collected patient 2D data. The reconstruction parameters are shown in Table 1, and the reconstruction example is shown in Figure 3. The reconstructed 3D point cloud encompasses the patient’s facial information; however, it also contains abnormal points and outliers. Consequently, optimizing the quality of the dense point cloud is essential for eliminating abnormal points and outliers. Upon analyzing the reconstructed point cloud, it was observed that the RGB values (175, 145, 120) of the facial point cloud for most patients differed significantly from the RGB values (110, 114, 114) representing gray. As a result, the gray abnormal points in the reconstructed point cloud were eliminated based on the RGB numerical differences between the background gray point cloud and the reconstructed facial point cloud. Statistical filters and bilateral filters were utilized to eliminate outliers and smooth the reconstructed point cloud. The effectiveness of the quality optimization applied to the 3D point cloud is illustrated in Figure 3A. However, the point cloud remains discrete and requires Poisson surface reconstruction to generate a closed 3D surface. This method effectively preserves local details, allowing the reconstructed surface to more accurately reflect the original point cloud’s characteristics. The Poisson surface reconstruction algorithm generates a continuous triangular mesh surface from the discrete point cloud for subsequent surface area calculation, as shown in Figure 3B.

Figure 3 Colmap point cloud reconstruction and optimization (Case 8). (A) Comparison of the original point cloud and the optimized point cloud. (B) Poisson surface reconstruction of the discrete point cloud into a continuous mesh model.

3D point cloud accuracy verification

Next, we use the point cloud obtained by the structured light camera (31) as the standard, and perform point cloud registration between the point cloud generated by our scheme (point cloud reconstructed using Colmap) and the point cloud generated by the structured light. The RMSE after point cloud registration is used to verify the effectiveness of this solution in reconstructing the 3D point cloud. The accuracy of the point cloud surface measured by the structured light camera is reliable and widely recognized (32,33). The RMSE can be used to measure the error between two point cloud registration points. The smaller the value, the higher the registration accuracy, and vice versa. Figure 4 compares the point clouds generated by the structured light camera and our method at similar angles, displaying the results of facial reconstruction at five different angles. In Figure 4A-4C, the first row displays the point cloud scanned and compiled by the structured light camera, while the second row presents our method’s 3D reconstruction results. At similar angles, our reconstructed point cloud surface closely resembles that of the structured light camera. However, the point cloud generated by the structured light camera exhibits uneven coloration and low fidelity, making it hard to distinguish the lesion area. In contrast, the point cloud produced by our method demonstrates more accurate coloration, facilitating the differentiation between normal skin and lesions. Color is a critical parameter for assessing PWS treatment efficacy, with correct coloration enabling more precise evaluation. Furthermore, the three sets of data in Figure 4 underwent point cloud registration, with the statistical results detailed in Table 2. Our method generated significantly more point clouds than the structured light camera, approximately 10 to 20 times greater, indicating a substantial advantage in quantity. The RMSEs observed after point cloud registration were 0.8107, 1.0020, and 1.0706 mm, respectively, yielding an average value of 0.9611 mm. These minimal errors, relative to facial dimensions, indicate high morphological similarity between the two point clouds, validating our method’s effectiveness.

Figure 4 Comparison of the point cloud reconstructed by the structured light camera and the point cloud reconstructed by our method (Cases 9, 16, 17). The first line of A-C is the point cloud reconstructed by a structured light camera, and the second line is the point cloud reconstructed by our method.

To further validate our mesh model’s accuracy, the point cloud model in Figure 4 was reconstructed using Poisson surface reconstruction to create a mesh model, and then the CLIP similarity between this mesh model and the corresponding 2D images at similar angles was calculated across different patient datasets. CLIP is a multimodal model developed by OpenAI capable of understanding both images and text simultaneously (34). Two primary reasons motivated our choice of CLIP over other evaluation metrics. (I) Traditional image similarity metrics, such as the structural similarity index measure (SSIM) or peak signal-to-noise ratio, rely on pixel-level point-by-point comparisons. These metrics are highly sensitive to image resolution, illumination, and viewpoint changes. CLIP can extract high-level semantic features of the image, which makes CLIP insensitive to pixel-level changes and focuses on evaluating the similarity of the two in terms of visual content (shape, structure, spatial relationship), which is more in line with our verification goal. (II) Our comparison task essentially involves comparing a real clinical photograph with a rendered image of the 3D model at a specific viewpoint. These images originate from distinct domains, exhibiting inherent differences in texture, noise, and illumination details. The strength of CLIP lies in its ability to capture and compare image semantics across domains, thus providing a fair and robust metric for cross-domain comparisons. For two content-related images, the similarity score is typically distributed in the range [0, 1]. Scores closer to 1 indicate that the two images are highly similar within the feature space learned by CLIP. Scores >0.85 typically indicate high similarity, with the images nearly identical in subject matter, structure, and layout. Scores between 0.7 and 0.85 indicate good similarity, though minor differences in details may exist. Scores <0.7 suggest that the subject matter is loosely related. Figure 5 compares our mesh model and the 2D images at similar angles, along with the corresponding CLIP similarity scores. The first row in Figure 5A-5C displays the 2D images, the second row shows the corresponding 3D mesh model results, and the third row indicates the CLIP similarity scores. Comparing the 2D images and the reconstructed mesh model reveals that the 3D reconstruction has clear results and effectively captures the lesion areas. However, the robustness of the results regarding hair remains inadequate, leading to sparse reconstruction points and artifacts. Furthermore, across the three data groups, the CLIP similarity ranged from 0.8994 to 0.9436, with average scores of 0.9209, 0.9210, and 0.9249 for each group, respectively, indicating high similarity between all reconstructed 3D mesh models and their corresponding 2D images at similar angles.

Figure 5 Comparison between the 3D mesh reconstructed model and the approximate angle 2D image. CLIP, contrastive language-image pre-training.

Lesion segmentation, 3D reconstruction, and quantitative evaluation

After establishing a foundation for an accurate 3D model, the original images were segmented using the M-CSAFN model. The segmentation results were then masked onto the original data, resulting in images from various angles that contained only the lesions. Subsequently, using the prior camera pose and internal parameters, the 2D lesion image is converted into an accurate 3D lesion point cloud through sparse reconstruction and dense reconstruction. This approach helped reduce matching errors caused by the loss of effective image areas and significantly lowered computational complexity. Patient data representing red, purple-red, and thickening types were selected for lesion segmentation and 3D reconstruction. The results of this process are illustrated in Figure 6. The results indicate that the 3D mesh models of the lesions can be effectively reconstructed across different types. When comparing the 2D images taken from similar angles, the shapes of the lesions in both formats align closely. The CLIP similarity data further supports this, showing that the average CLIP similarity across the three classifications exceeds 0.93. This suggests that the 3D reconstruction results of the lesions exhibit a high similarity to their corresponding 2D images.

Figure 6 The results of first performing 2D segmentation on red, purple-red, and thickened lesions, followed by the reconstruction of their 3D mesh models (Cases 6, 12, 13). CLIP, contrastive language-image pre-training.

The surface area of the lesion serves as a crucial indicator for the quantitative evaluation of PWS. The 3D mesh model of the lesion comprises three vertices that form a triangle, with several smaller triangles utilized to represent the surface of the 3D object. Consequently, the surface area of the 3D lesion can be determined by calculating the area of each triangle and aggregating these using Heron’s formula. The surface area values for the red type, purple-red type, and thickened type, as calculated using Heron’s formula for the 3D lesion mesh model depicted in Figure 6, are 9.8432, 26.8481, and 166.8408 cm², respectively.

To validate the accuracy of the proposed area calculation method, nine PWS patches with known surface areas of varying shapes and sizes were created, as illustrated in Figure 7A. Among them, 5 regular shapes are simple geometric shapes (such as triangles, squares, and circles) that are easy to calculate the area. The remaining four patches featured irregular shapes with complex boundaries, derived from segmented lesion regions in 2D images of PWS patients. Figure 7B shows an example of a PWS lesion patch attached to a facial model. The proposed method was then used to calculate the 3D surface area of different lesion patches on the facial model. At the same time, the lesion patches in the 2D image were segmented in two dimensions, and the area of the 2D lesion patches was calculated using ImageJ software and a ruler. The relative error between the results measured by the 2D method and the 3D method and the true value of the surface area of the lesion patch was calculated, and the formula for the relative error is as follows:

Error=|Sgt−S|Sgt∗100%[9]

Figure 7 Validation of PWS 3D lesion surface area measurement accuracy using a mannequin head with artificial PWS lesions. (A) Lesion patches with known surface area truth value. (B) Facial model with a lesion patch attached. (C) Comparison of 3D and 2D measured surface areas of lesions with the truth value. PWS, port-wine stains.

where Error is the relative error value. S_gt is the true value of the lesion patch, derived by flattening the patch and correlating its pixel area with a known physical reference (e.g., A4 paper dimensions). S is the area value calculated by the method in this paper. The calculated comparison results of the lesion surface areas of the nine groups of facial models are shown in Figure 7C and Table 3. A comparison of the 2D measurement values with the 3D measurements reveals that the latter are closer to the true value. Furthermore, the overall average relative error for the 2D measurements is 9.22%, while that for the 3D measurements is 4.59%. This further demonstrates that the error in the 3D measurement of the lesion surface area is smaller than that in the 2D image measurement, indicating that our proposed method achieves high-precision 3D reconstruction of the lesion.

Discussion

Convenient and efficient acquisition of the 3D shape of PWS lesions is crucial for clinical applications. On the other hand, by observing the lesion from alternative angles beyond those provided by the input image, the 3D model allows for examining surfaces and details not visible in the 2D image. This indicates that 3D reconstruction technology enables the acquisition of richer information beyond the constraints of the viewing angle present in the input image. We demonstrate how deep learning can be integrated with the Colmap 3D reconstruction algorithm to achieve 2D segmentation, 3D reconstruction, and quantitative assessment of lesions in PWS patients. This solution enables direct and efficient 3D reconstruction. It requires only a camera to capture a patient’s 2D image, and subsequent steps are automatically processed by a computer. Although studies have shown that 3D point clouds can be converted to color space and segmented using thresholding or clustering, actual lesion boundaries may pose challenges for accurate segmentation due to low boundary contrast. However, lesion segmentation technology in 2D imaging is relatively advanced, and the M-CSAFN model we proposed further improves 2D segmentation performance. The Jaccard index results of M-CSAFN surpass those achieved by the conventional U-Net and Deeplabv3 networks. Additionally, its accuracy, sensitivity, precision, specificity, and Dice coefficient also outperform those of the U-Net and Deeplabv3 networks. It demonstrates superior performance in handling PWS segmentation tasks and can effectively segment lesions with poor boundary contrast. Based on 2D segmentation, converting a series of segmented 2D lesion images into a 3D model yields an accurate 3D lesion model while also significantly reducing computational complexity.

Fortunately, the Colmap 3D reconstruction algorithm can achieve this goal while facilitating the post-processing of the generated 3D model. The Colmap reconstruction algorithm can generate 3D point clouds from 2D image sequences taken at different angles. Thanks to a standard image acquisition system with uniform and constant illumination, we employed a high-definition camera to capture image sequences, ensuring sufficient resolution in each image. In the SfM algorithm, more feature points can be extracted for matching between images, resulting in lower errors in the sparse point cloud. In the dense point cloud reconstruction stage of the MVS algorithm, the point cloud color is computed through a weighted calculation based on the pixel colors in the 2D image sequence. When using the standard image acquisition system to acquire the 2D image sequence, we placed a 48-color standard colorimetric card and employed the Datacolor Spyder Checkr software to ensure consistent color correction across all images. Therefore, the color of the point cloud generated in this work is more realistic and accurate, enhancing its applicability for PWS. In addition, the quality of the reconstructed 3D point cloud was further improved through the application of RGB value differences, statistical filters, and bilateral filters. Comparing the point clouds before and after optimization, our optimization scheme effectively eliminates abnormal points and outliers from the point cloud, reduces roughness, and improves smoothness. Subsequent comparison with the 3D point cloud reconstructed by the structured light camera demonstrates this method’s capability to reconstruct accurate 3D information of the patient. Compared with the reconstruction results of the structured light camera, the point cloud reconstructed by our method achieves greater color uniformity. Because the structured light camera requires the splicing of multiple point clouds, it is constrained by the RGB camera, leading to inconsistent coloration in the reconstructed point cloud when capturing at different angles. Equipment that supports both depth detection and accurate color, such as Artec scanners (Senningerberg, Luxembourg), is costly and hard to implement widely. This work utilizes a standard image acquisition system with uniform and constant illumination for 3D reconstruction, achieving both precise 3D shape reconstruction and accurate color. This approach employs simple equipment, facilitating clinical adoption.

Lastly, the 3D point cloud can be transformed into a mesh model using the Poisson surface reconstruction algorithm. The CLIP similarity value between the mesh model and the 2D image at similar angles further demonstrates the accuracy and robustness of our reconstructed 3D model. Moreover, the mesh model allows for surface area calculation via the Heron formula, fulfilling the quantitative evaluation task. Existing studies have shown that in clinical settings, the relative error range in 3D surface area calculations using scanners or stereo measurement systems is between 1.7% and 5% (35,36). For 3D surface area calculations using the SfM reconstruction method, the relative error range is between 3% and 5% (37-39). Lesion segmentation in these studies relied solely on manual segmentation. However, in this study, the relative error range was between 2.46% and 7.15%, with an average of 4.59%. Compared with errors in manually segmented lesions, the 3D area calculation method proposed here not only automates segmentation but also achieves error margins close to those in manual segmentation, indicating that the approach of first segmenting and then reconstructing the 3D lesion for surface area calculation is effective.

One limitation of this study is the potential incompleteness or inaccuracy of the information extracted by the feature extraction algorithm during the 3D reconstruction process, which can lead to abnormal points, outliers, and information loss in the reconstructed original point cloud. To address this, employing an improved feature extraction algorithm that incorporates semantic information may enhance the quality of the original point cloud and potentially obviate the need for secondary optimization. Secondly, while our proposed M-CSAFN model exhibits excellent segmentation performance, all training data were obtained from a single institution. This can lead the model to learn data-specific biases, which may limit its generalizability to other datasets. To mitigate this limitation, we applied several strategies, including the core multi-color space fusion architecture of M-CSAFN, data augmentation to increase sample diversity, and an auxiliary SSIM loss function that imposes structural similarity constraints on brightness and contrast. Through these strategies, M-CSAFN demonstrated promising performance on four additional public skin lesion datasets. Nevertheless, in data-driven deep learning scenarios, constructing large, multi-institutional datasets remains the most robust and effective approach for mitigating data bias. Lastly, although the calculated 3D lesion surface area had an average relative error of only 4.59% compared to the actual area, there is still room for further improvement in calculation accuracy. Exploring deep learning algorithms that combine point cloud semantic and color information for direct segmentation of 3D models could refine the accuracy of 3D lesion delineation, reduce the relative error in lesion surface area calculation, and boost the precision of quantitative evaluations.

In future applications, flexible free-form Organic Light-Emitting Diode (OLED) light sources will be utilized. Leveraging 3D segmentation results of the treatment area, the OLED’s deformation will be meticulously customized to ensure optimal physical alignment between the light source and the skin’s contour. Furthermore, curvature and other surface parameters will be modulated to control the irradiation angles of the therapeutic light. This advanced system guarantees uniform light delivery to each targeted lesion area, enabling simultaneous treatment of the entire affected region. Such technological integration markedly improves the efficacy of V-PDT in treating PWS.

Conclusions

In summary, a novel 3D reconstruction method for PWS was proposed to enhance clinical treatment and evaluation of PWS. A standard image acquisition system with uniform and constant illumination was established, allowing for the collection of high-quality, color-accurate 2D images of patients with PWS, thereby providing a solid foundation for the precise reconstruction of the 3D point cloud model. The Colmap 3D reconstruction technology and point cloud optimization scheme were employed to generate a high-quality 3D point cloud model of PWS, incorporating color and facial information. The high registration of the Colmap-optimized point cloud with the 3D point cloud generated by the structured light camera (average RMSE of 0.9611 mm) and the average CLIP value of the 3D mesh model compared to the 2D image at similar angles (with an average CLIP value exceeding 0.92) collectively demonstrated the accuracy of the 3D model reconstruction achieved by this method. The M-CSAFN network model was utilized to segment the 2D images, enabling accurate 3D reconstruction of the lesions through the Colmap algorithm and the prior information from the 3D point cloud. The lesion’s surface area in 3D space was calculated using the Heron formula, facilitating an efficient 3D quantitative evaluation. Experimental validation indicated that the average error between the surface area measured by this method and the actual surface area was merely 4.59%, significantly lower than the 9.22% associated with the 2D image measurement method. Our method enables high-quality and accurate 3D reconstruction of PWS lesions, offering a promising tool for PWS assessment and treatment planning.

Acknowledgments

None.

Footnote

Data Sharing Statement: Available at https://qims.amegroups.com/article/view/10.21037/qims-2025-112/dss

Funding: This work was supported by the National Key Research and Development Program of China (No. 2023YFB3609100), the National Natural Science Foundation of China (Grant Nos. 62205025, 62227823 and T2293753) and the Beijing Institute of Technology Research Fund Program for Young Scholars (No. XSQD-202123001).

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at https://qims.amegroups.com/article/view/10.21037/qims-2025-112/coif). The authors have no conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. This study was conducted in accordance with the Declaration of Helsinki and its subsequent amendments. This study was approved by the Ethics Committee of Chinese PLA General Hospital (No. S2022-143-01). Informed consent was obtained from all patients or their legal guardians.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. Ortiz AE, Nelson JS. Port-wine stain laser treatments and novel approaches. Facial Plast Surg 2012;28:611-20. [Crossref] [PubMed]
2. Kelly KM, Choi B, McFarlane S, Motosue A, Jung B, Khan MH, Ramirez-San-Juan JC, Nelson JS. Description and analysis of treatments for port-wine stain birthmarks. Arch Facial Plast Surg 2005;7:287-94. [Crossref] [PubMed]
3. Savas JA, Ledon JA, Franca K, Chacon A, Nouri K. Pulsed dye laser-resistant port-wine stains: mechanisms of resistance and implications for treatment. Br J Dermatol 2013;168:941-53. [Crossref] [PubMed]
4. Liu Y, Chen D, Zeng J, Qiu H, Gu Y. Progress in Port-Wine Stains Lesion Structure and Its Optical Imaging Technique in Vascular-Targeted Photodynamic Therapy. Chinese Journal of Lasers 2022;49:1507102.
5. Ren J, Chen DF, Wang Y, Gu Y. Recent Advances in optical imaging techniques for clinical assessment of port wine stains. Chinese Journal of Laser Medicine & Surgery 2013;22:224-31.
6. Gu Y, Huang NY, Liang J, Pan YM, Liu FG. Clinical study of 1949 cases of port wine stains treated with vascular photodynamic therapy (Gu's PDT). Ann Dermatol Venereol 2007;134:241-4. [Crossref] [PubMed]
7. Yuan KH, Li Q, Yu WL, Zeng D, Zhang C, Huang Z. Comparison of photodynamic therapy and pulsed dye laser in patients with port wine stain birthmarks: a retrospective analysis. Photodiagnosis Photodyn Ther 2008;5:50-7. [Crossref] [PubMed]
8. Rajadhyaksha M, González S, Zavislan JM, Anderson RR, Webb RH. In vivo confocal scanning laser microscopy of human skin II: advances in instrumentation and comparison with histology. J Invest Dermatol 1999;113:293-303. [Crossref] [PubMed]
9. Chang CJ, Yu JS, Nelson JS. Confocal microscopy study of neurovascular distribution in facial port wine stains (capillary malformation). J Formos Med Assoc 2008;107:559-66. Retracted Publication. [Crossref] [PubMed]
10. Gu Y, Li J, Jiang Y, Liang J, Zhou P, Cui X, Pan Y, Wang K. A clinical study of photodynamic therapy for port wine stain: a report of 40 cases. Chinese Journal of Laser Medicine & Surgery 1992;6-10.
11. Gu Y, Li J, Shan H, Wang K, Jiang Y, Zhang Y, Liang J, Pan Y, Yu Z, Zhang J, Liu G, Tu Y, Ren L. Clinical application of copper vapor laser in PDT for fifty cases of PWS. Chin J Laser Med Surg 1994;3:215-7.
12. Chen D, Yuan W, Park HC, Li X. In vivo assessment of vascular-targeted photodynamic therapy effects on tumor microvasculature using ultrahigh-resolution functional optical coherence tomography. Biomed Opt Express 2020;11:4316-25. [Crossref] [PubMed]
13. Qiu H, Gu Y, Wang Y, Huang N. Twenty years of clinical experience with a new modality of vascular-targeted photodynamic therapy for port wine stains. Dermatol Surg 2011;37:1603-10. [Crossref] [PubMed]
14. Chen D, Ren J, Wang Y, Li B, Gu Y. Intraoperative monitoring of blood perfusion in port wine stains by laser Doppler imaging during vascular targeted photodynamic therapy: A preliminary study. Photodiagnosis Photodyn Ther 2016;14:142-51. [Crossref] [PubMed]
15. Chen D, Wang Y, Zhao H, Qiu H, Wang Y, Yang J, Gu Y. Monitoring perfusion and oxygen saturation in port-wine stains during vascular targeted photodynamic therapy. Ann Transl Med 2021;9:214. [Crossref] [PubMed]
16. Qin ZP, Li KL, Ren L, Liu XJ. Photodynamic therapy of port wine stains-a report of 238 cases. Photodiagnosis Photodyn Ther 2007;4:53-9. [Crossref] [PubMed]
17. Frigerio A, Bhama PK, Tan OT. Quantitative three-dimensional assessment of port-wine stain clearance after laser treatments. Lasers Surg Med 2014;46:180-5. [Crossref] [PubMed]
18. Zhang J, Hu X, Mo Y, Li Q Sr. The approach for quantification assessment for port-wine stains in three-dimensional and color space. Optics in Health Care and Biomedical Optics VIII. SPIE 2018;10820:290-9.
19. Ke C, Huang Y, Yang J, Zhang Y, Zhan H, Wu C, Bi M, Huang Z. Lesion segmentation using 3D scan and deep learning for the evaluation of facial portwine stain birthmarks. Photodiagnosis Photodyn Ther 2024;46:104030. [Crossref] [PubMed]
20. Minaee S, Boykov Y, Porikli F, Plaza A, Kehtarnavaz N, Terzopoulos D. Image Segmentation Using Deep Learning: A Survey. IEEE Trans Pattern Anal Mach Intell 2022;44:3523-42. [Crossref] [PubMed]
21. Asgari Taghanaki S, Abhishek K, Cohen JP, Cohen-Adad J, Hamarneh G. Deep semantic segmentation of natural and medical images: a review. Artificial Intelligence Review 2021;54:137-78.
22. LeCun Y, Bengio Y, Hinton G. Deep learning. Nature 2015;521:436-44. [Crossref] [PubMed]
23. Yuan Y, Lo YC. Improving Dermoscopic Image Segmentation with Enhanced Convolutional-Deconvolutional Networks. IEEE J Biomed Health Inform 2019;23:519-26. [Crossref] [PubMed]
24. Tang P, Liang Q, Yan X, Xiang S, Sun W, Zhang D, Coppola G. Efficient skin lesion segmentation using separable-Unet with stochastic weight averaging. Comput Methods Programs Biomed 2019;178:289-301. [Crossref] [PubMed]
25. Hajabdollahi M, Esfandiarpoor R, Khadivi P, Soroushmehr SMR, Karimi N, Samavi S. Simplification of neural networks for skin lesion image segmentation using color channel pruning. Comput Med Imaging Graph 2020;82:101729. [Crossref] [PubMed]
26. Mu J, Lin Y, Meng X, Fan J, Ai D, Chen D, Qiu H, Yang J, Gu Y. M-CSAFN: Multi-Color Space Adaptive Fusion Network for Automated Port-Wine Stains Segmentation. IEEE J Biomed Health Inform 2023;27:3924-35. [Crossref] [PubMed]
27. Digne J, De Franchis C. The bilateral filter for point clouds. Image Processing On Line 2017;7:278-87.
28. Schonberger JL, Frahm JM. Structure-from-motion revisited. Proceedings of the IEEE conference on computer vision and pattern recognition 2016;4104-13.
29. Schönberger JL, Zheng E, Frahm JM, Pollefeys M. Pixelwise view selection for unstructured multi-view stereo. European conference on computer vision. Cham: Springer International Publishing 2016;501-18.
30. Ronneberger O, Fischer P, Brox T. U-net: Convolutional networks for biomedical image segmentation. International Conference on Medical image computing and computer-assisted intervention. Cham: Springer International Publishing; 2015;234-41.
31. Geng J. Structured-light 3D surface imaging: a tutorial. Advances in optics and photonics 2011;3:128-60.
32. Boyer KL, Kak AC. Color-encoded structured light for rapid active ranging. IEEE Trans Pattern Anal Mach Intell 1987;9:14-28. [Crossref] [PubMed]
33. Zhang X, Zhu L, Chu L. Evaluation of coded structured light methods using ground truth. 2011 IEEE 5th International Conference on Cybernetics and Intelligent Systems (CIS). IEEE 2011;117-23.
34. Radford A, Kim JW, Hallacy C, Ramesh A, Goh G, Agarwal S, Sastry G, Askell A, Mishkin P, Clark J, Krueger G, Sutskever I. Learning transferable visual models from natural language supervision. International conference on machine learning. PMLR 2021;8748-63.
35. Pavlovčič U, Jezeršek M. Handheld 3-dimensional wound measuring system. Skin Res Technol 2018;24:326-33. [Crossref] [PubMed]
36. Mamone V, Fonzo MD, Esposito N, Ferrari M, Ferrari V. Monitoring Wound Healing With Contactless Measurements and Augmented Reality. IEEE J Transl Eng Health Med 2020;8:2700412. [Crossref] [PubMed]
37. Liu C, Fan X, Guo Z, Mo Z, Chang EI, Xu Y. Wound area measurement with 3D transformation and smartphone images. BMC Bioinformatics 2019;20:724. [Crossref] [PubMed]
38. Mirzaalian Dastjerdi H, Töpfer D, Rupitsch SJ, Maier A. Measuring Surface Area of Skin Lesions with 2D and 3D Algorithms. Int J Biomed Imaging 2019;2019:4035148. [Crossref] [PubMed]
39. Barbosa FM, Carvalho BM, Gomes RB. Accurate chronic wound area measurement using structure from motion. 2020 IEEE 33rd International Symposium on Computer-Based Medical Systems (CBMS). IEEE 2020;208-13.