THE POINT CLOUD SEMANTIC SEGMENTATION METHOD FOR THE DOUGONG OF MING AND QING DYNASTY OFFICIAL-STYLE ARCHITECTURES

: To solve the problem that Dougong has various shapes and complex structures, the corresponding solutions are proposed in this paper. Our proposed method mainly consist of two parts. At first, the surface primitives were segmented using the machine learning (Random Forest). In this stage, the features including the curvature, normas and other features based on covariance matrix. Then, the knowledge from the construction rules were applied to label the segmented surface primitives into correct categories. The corresponding height constraint, concave-convex constraint, and symmetry constraint are proposed as the judgment conditions to mark the geometric elements belonging to the same dougong component and complete the point cloud segmentation of the dougong component. To verify the performance of our proposed method, the point cloud of a Qing-style single-arch flat-bodied Dougong was tested. The experimental results show that the classification accuracy of point cloud is 96.0%.


INTRODUCTION
Dougong was considered as the most important components of the Ming and Qing Dynasty official-style architectures (MQDOAs) and played an important role in the load-bearing. Suffering from weathering, fires and rotting, a mass of MQDOAs with wooden structural frame disappeared. To preserve historical buildings, using the 3D point cloud which provided the precise geometric coordinates (X, Y, Z) in the form of millions of points to record the shape of the culture heritage has become one of the most efficient methods. The captured original point cloud lacked structured information such as semantics and hierarchy between parts, which disturbed the usage of point cloud in other application fields. Hence, how to segment the point cloud into the sub dataset with semantic meaning has become a research hotspot (Huo, 2020). Nowadays, automated point cloud segmentation methods have been proposed by researcher and achieved good performance (Dong, 2018). However, it was still a challenge to segment the Dougong point cloud into the correct categories. One hand, the types of the components composed of the Dougong were large and the shape of components varied greatly. Only relying on the geometric features which have been used in other architectural heritage was difficult to meet the segmentation requirements. On the other hand, the cross combination between components caused a whole component was broken in 3D space. To further support the scan to BIM process, the 3D point cloud of Dougong was segmented into the sub-class with semantic information in this paper. Considering that the construction of the Ming and the Qing Dynasty official-style architecture followed certain rules which can be searched for in YingzaoFashi (Building Standards) of the Song Dynasty or Gongchengzuofazeli (Structural Regulations) published by Qing, we proposed a point cloud segmentation method for Dougong.

RELATED WORK
This text discusses various methods for point cloud segmentation, which is the process of partitioning a point cloud into clusters to facilitate subsequent processing such as fitting and recognition.

* Corresponding author
The current methods for point cloud segmentation can be divided into conventional point cloud segmentation methods and machine learning-based point cloud segmentation methods. Conventional point cloud segmentation methods include edgebased segmentation methods, region growing, model fitting, and clustering-based segmentation methods. Each method has its own advantages, disadvantages, and applicability (Cheng, 2021). Edge-based segmentation methods identify the edge features of point clouds (such as gradients, curvature, etc.) to recognize the edge features of the segmented object and background point clouds, and remove the background point clouds to complete point cloud segmentation. This algorithm is fast, but not accurate enough when faced with complex and irregular edges and may have large gaps. Region growing is a method that detects seed points and their neighbourhoods. If they have similar features in geometric features, colour, etc., each collection is merged into a region, and then surrounding adjacent points are detected (Shi-Min, 2018). The key to this algorithm is the selection of seed points and the selection of termination thresholds, which require multiple experiments to select the best threshold. Model fitting is mainly used to detect basic geometric shapes such as planes, cylinders, and spheres through algorithms such as least-squares plane fitting and random sample consensus (RANSAC). RANSAC was first proposed in the paper and is currently the most widely used model fitting method with strong robustness and reduced noise interference. Clustering-based methods are based on unsupervised classification to cluster points with similar attributes, such as commonly used Euclidean clustering algorithms, K-means algorithms, etc., which are commonly used in segmentation. Gaussian mapping and improved fuzzy Cmeans clustering algorithms are used to enhance the robustness of segmentation. The clustering algorithm is generally effective, but in complex scenes, it is difficult to segment the image because the distance between multiple points is too short. Conventional point cloud segmentation algorithms (such as region growing and model fitting) usually follow a more rigorous mathematical model, usually used for segmentation and extraction of regular components of ancient architecture (Teruggi, 2020). However, due to the complex characteristics of dougong structures, existing point cloud segmentation and classification ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume X-M-1-2023 29th CIPA Symposium "Documenting, Understanding, Preserving Cultural Heritage: Humanities and Digital Technologies for Shaping the Future", 25-30 June 2023, Florence, Italy methods cannot be directly utilized. The main difficulty is that, on the one hand, dougong components have both planes and curved surfaces, and existing point cloud segmentation methods alone cannot properly segment point clouds into curved surfaces. On the other hand, dougong components overlap and intersperse, making it difficult to determine which points belong to a component, making it difficult to extract complete components. Machine learning-based point cloud classification techniques are mainly divided into two types: point-based point cloud classification and object-based point cloud classification. Pointbased classification is based on the local features of a single point for classification and usually includes three steps: neighborhood selection, local feature extraction, and feature learning and classification using a classifier. Single points only contain coordinate values and other features, so selecting their neighborhoods around single points can indirectly obtain the normal vectors and curvature of the region in which the point is located, as well as other features. Object-based point cloud classification uses machine learning algorithms to classify and recognize objects in point clouds. It is more accurate than point-based classification, but it requires more computing resources and time. Among the DL approaches, PointNet and its later improvement PointNet++ (Haoyu, 2019) were considered as pioneer works. Compared with the regular supervise machine learning, the DL methods do not need to design the features. A good review related on cloud segmentation based on DL can be seen in Ref. (Barazzetti, 2016). In the field of the architecture heritage, P. R. et al. (Shen, 2020) proposed a DL framework for cultural heritage based on the DGCNN. Y. Ji et al. (Zhang, 2014)modified the DGCNN for the segmentation of the MQDOAs roof. The segmentation accuracy of the modified DGCNN performed better than PointNet, DGCNN and LDGCNN, and reached 87.14%. Although the architectural heritage classification methods based on deep learning performed efficiently, the DL approaches significantly rely on the training datasets. Nowadays, the published datasets, such as ModelNet 40 , KITTI (Suter, 2013), Sydney Urban Objects (Borrmann, 2009), Semantic3D (Jin, 2018), S3DIS and ArCH , were mainly collected from urban environments. There are still no published datasets focusing on MQDOAs with an adequate level of detail. This limited the usage of deep learning in the point cloud segmentation of the MQDOAs roof.

Characteristics of Dougong
Dougong was called "Douke" in the Qing Dynasty and "Pave" in the Song Dynasty. It is a unique structure in Chinese wooden architecture. Dougong, as a transitional structure connecting columns and beam frames, is composed of dougong, dougong, rising and rising components, which are supported layer by layer and stretched outward. Dougong has played a great role in the structure, including expanding the bearing area of the column head, transferring the load, and reducing the shear force at the junction to prevent breaking. The outer eave dougong also has the function of overhanging the eaves to protect the wall from rain erosion. Dougong is not only an indispensable component in the structure, but also has a strong decorative effect in the appearance. According to different positions, dougongs can be divided into the flat body family, the stigma family and the horn family.

The proposed method for the segmentation of Dougong
As is shown in Figure 3, our proposed method mainly consist of two parts. At first, the surface primitives were segmented using the machine learning (Random Forest); then, the knowledge from the construction rules were applied to label the segmented surface primitives into correct categories.
• In first stage, the features including curvature, normal, roughness, planarity, linearity, and verticality were extracted; subsequently, the random forest classifier is used to describe the probability distribution of According to the principle of PCA, the divergence matrix is singular value decomposed to obtain eigenvectors and eigenvalues, in which the largest eigenvalue corresponds to the main direction of the building. Calculate the included angle θ between the main direction and the original coordinate axis, and rotate the divided bucket about Z axis θ.

Surface primitives classification of the Dougong based on RF:
The feature extraction of point cloud has a great influence on the classification effect of point cloud, and the geometric information of point cloud can reflect the spatial characteristics of point cloud. In this paper, aiming at the scene where the density distribution of laser point cloud is relatively uniform, KNN neighbourhood is used to obtain the local neighbourhood of point cloud, and then the features are extracted. Based on the spatial arrangement of point clouds in the selected local neighbourhood, the classification characteristics of point X are calculated. Let's assume that there are n neighboring points of point P, which are = ( , , ), = (1,2, … , ) . The covariance matrix of P and the selected neighborhood is: Where ̅ = 1 ∑ =1 is the center of gravity of the neighbourhood of point P. The covariance matrix of point cloud data is a symmetric matrix with three rows and columns, so solving the covariance matrix will get three eigenvalues, which are 1 > 2 > 3 > 0. Among the features related to eigenvalues and eigenvectors, curvature and normal are most commonly used, which are important properties of geometric surfaces. The eigenvector corresponding to the minimum eigenvalue is taken as the normal of the segment, and the ratio of the minimum eigenvalue to the sum of eigenvalues is taken as the curvature of the segment. Because both eigenvalues and eigenvectors carry some effective information, this paper not only uses the curvature and normals of the segment as the features of the segment, but also takes all eigenvalues obtained by solving the covariance matrix as the features of the segment. Curvature:

Sum of eigenvalue:
= 1 + 2 + 3 (5) Total variance: The above features such as curvature, normal, flatness, linearity and verticality are selected to comprehensively reflect the geometric shape, patch direction and other information of the bucket point cloud. Here, a set of feature vectors will be formed by comprehensively utilizing these selected direct features of point clouds and their derived features to realize the classification of point clouds in feature space. From this, the feature vector of point cloud including feature values is calculated, as shown in formula (9): = [ , , , 3 , , , , ] ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume X-M-1-2023 29th CIPA Symposium "Documenting, Understanding, Preserving Cultural Heritage: Humanities and Digital Technologies for Shaping the Future", 25-30 June 2023, Florence, Italy In this study, the classic random forest algorithm was used to train the classification model. Random Forests is an ensemble classifier, which has been widely applied in point cloud classification (Chevrier, 2010). By integrating multiple weak classifiers (i.e., decision tree), the random forest algorithm is not prone to over fitting when classifying a large data set with high dimensions, so that the overall classification result has high accuracy and good generalization ability . A decision tree can avoid over fitting by pruning, but it will inevitably learn some noise information when training the classification model if there is only one tree. The random forest uses Bagging's sampling method to combine multiple decision trees, and finally determines the target category by voting (the minority is subordinate to the majority). In machine learning, some parameters such as accuracy, precision, and recall are usually calculated to measure the pros and cons of the classification model .

Surface primitives classification based on the multiple Constraints:
To classify these primitives into the correct categorises, we build the adjacency relationship of the surface primitives firstly; subsequently, the multiple constrains are used for classify the surface primitives into the correct categories. Adjacency refers to the fact that objects or shapes close to each other in space tend to form the same group and are described by Euclidean geometric distance. In this paper, the adjacency relationship of the surface primitives is judged. The method based on grid is used to quickly find the adjacent surface primitives. If a grid contains points of multiple surface primitives, they are considered to be adjacent. After the adjacency relationship of the surface primitives is determined, the constrains include height, Concave-convex relationship of adjacent primitives and symmetry are used to classify the surface primitives.
As we can see, the components of Dougong are convex polygons. According to the convex-concave angle between two adjacent surface primitives, it is judged whether they belong to the same component. Convex falls into one category.

•
Height. For a surface primitive containing multiple components, divide it by the height of the components that cross it.
Where: the normal vector N represents the direction of the normal parallel to, or at the same angle as the plane. The area represents the area represented by the point cloud quantity.
represents the height position. Since the point cloud has been aligned, the surface elements of the same component should be in the same Z value range. Assuming that two component objects A and B, Z_min and Z_max are the minimum and maximum values of the Z value in their point cloud coordinates, respectively, it should be satisfied that � − � < && � − � < ; the distance threshold D is used to exclude bucket-like components, which are far from the ends of the arch, and can be excluded by this threshold, only targeting the transverse components that have been broken. The distance between at least two points in each group of surface elements is lower than the critical value, and the distance threshold D is set to the distance of one bucket mouth. It is worth noting that the symmetric transformation of S constitutes a group structure, called the symmetry group of S. Members in the symmetry group are not necessarily symmetric to each other, but can be determined as belonging to the same component.

Experimental Data
In this paper, we take the Qing-style single-arch flat-bodied Dougong as an example, and segment the surface primitives of Dougong point cloud based on the above segmentation method. The method of photogrammetric modelling is used to collect the image data of the dougong integral model. The overlap of the photos is not less than 80%. The collected images need to be loaded into the Agisoft PhotoScan software platform to complete the processing of space three solution, image matching, etc. After processing, the reprojection error is 1.65 pixel.

Evaluation criteria
In order to test the applicability of the classification model, the above parameters were calculated by constructing a mixed matrix of the classification results of the training samples. The calculation formulas are shown in Equations (11)-(13), where N represents the total number of samples, TP represents the number of positive samples that are predicted as the positive class, FN represents the number of positive samples that are predicted as the negative class, FP represents the number of negative samples that are predicted as the positive class, and TN represents the number of negative samples that are predicted as the negative class. The accuracy rate (Equation (11)) is an intuitive indicator to measure the classification accuracy of the overall sample, and the accuracy rate (Equation (12)) and recall rate (Equation (13)) are indicators to measure the classification accuracy of a single category. In addition, the classification result of F1_score was also calculated (Equation (14)) in this study in order to consider the comprehensive evaluation indicators of precision and recall rate. F1_Score is the harmonic mean of precision and recall rates, which indicates the credibility of the prediction results. The value ranges of the above indicators are (0, 1). If the value of F1_Score is closer to 1, then the classification effect will be better.

Experimental Results
First, select training samples, and select 14 different kinds of surface primitives according to different curvature and normal direction. These surface primitives include vertical plane, horizontal plane, inclined plane and non-planar plane with different normal vector directions. Then, the random forest program is called, and the decision tree generated by sample training is selected to learn and classify the primitives. The experimental results are shown in the following figure, and different types of face primitives in the segmentation results are displayed in different colors. At the same time, use formulas (11) to (14)   The experimental results show that this method can effectively classify the surface primitive in the Dougong, and the total classification accuracy on train set is 0.999, accuracy on test set is 0.969. Since the vertical elevations of the single head and the grasshopper head are connected with the hemp leaf head and the brace and the valonia brace, they are classified into the same geometric primitive. It needs to be divided by height constraint. After dividing into discrete panel primitives, the panel primitives of the unified component are assigned by symmetry constraints and concave-convex constraints. Finally, the dougong model is restored. The results show that the algorithm can not only accurately segment the geometric primitives of dougong members, but also effectively suppress the interference of adjacent faces.

CONCLUSIONS
In this paper, we put forward a corresponding solution to segment component level point clouds of Dougong due to its diverse shapes and complex structures. The overall idea is to segment geometric primitives first, and then restore the model structure according to semantic knowledge. Because of the curves, broken lines and various irregular surfaces contained in the ancient building components, it is difficult to extract the feature parameters of these components from the point cloud at the same time. First, point clouds are classified according to different curvature, normals and other features based on machine learning to obtain geometric primitives that have no connection with each other. Aiming at the situation that different components are coplanar and the same component is broken and separated, the corresponding height constraint, concave-convex constraint, and symmetry constraint are proposed as the judgment conditions to mark the geometric elements belonging to the same dougong component and complete the point cloud segmentation of the dougong component.