Reconstruction of architectural heritage with symmetrical components

: Data capturing through either Lidar or photogrammetry, often results in incomplete and partial information related to a surface due to occlusion or inaccessibility of the clear object vision. In case of asymmetrical objects yet the reconstruction is unattainable by any means, meanwhile the approach for the development of the missing information could be done in cases of symmetrical objects. In this paper we have advised a semi-automatic approach for recreating missing or incomplete information from the partially captured data using space sub-division and 3D transformation. The study has been done on a 175 year-old building whose scanned information is available for only one side and captures a façade with four columns. The idea is to first extract the symmetrical parts through segmentation of different building parts. Then the columns with partial information have been oriented as per a reference plane based on the pose and centre computed from the horizontal parts. The instance is then used to fill in the lost information through duplication and transformation. This approach can be used to recreate structures with symmetrical elements, which are partially destroyed from withering, disaster, or any human intervention.


INTRODUCTION 1.1 Background
Repairing models are an active area of research in computer vision and 3D modelling domain.In architectural heritage conservation also there has been extensive research related to virtual modelling of an artefact or recreating the digital twin of the existing building features (Donato and Giuffrida, 2019;Zaragoza et al., 2021).Photogrammetry and LiDAR are two of the main sources of data acquisition which can derive sufficient information for any surveyed object under consideration (Dzwierzynska, and Prokop, 2022;Prokop et al., 2021).In case of monitoring and preserving the historical monuments, these remote sensing methods are considered most safe and effective as they prevent interaction with all kinds of physical objects and can be sensed without any contact.Due to this contactless mode of surveying, there are multiple issues that occur during data acquisition, one of such issue is called occlusion and information loss due to obstruction created by other objects in the line of sight.As Photogrammetry and LiDAR acquire data on the principle of line of sight, both suffer by data loss.Numerous literatures shows that the obstructions create overlay spaces due to dispersiveness of data while modelling detailed architectural masterpieces (Remondino, 2011;Caterina et al., 2017;Caciora et al., 2021, Nikoohemat et al, 2020;Kushwaha et al. 2020).
Mathematically speaking, the concept of symmetry can be used to describe a large portion of the "regularity" in 3-D space.A human face is reflectively symmetric, two parallel lines in space are translationally symmetric, and a cube is both reflectively and rotationally symmetric.Several types of symmetry can be seen of as high-level primitives that can be used to describe a range of regular objects and scenes, especially in man-made environments, if points and lines are taken as the fundamental primitives that one employs to model an arbitrary scene.Symmetry is a powerful psychological signal for human visual perception, and the computer vision field has also discussed the computational benefits of symmetry (Sinha et al., 2012).
To identify and recover structures under perspective, orthogonal, and affine projections, many instances of symmetry have been investigated.Completing damaged or un-acquired spaces plays a very prominent role in filling the missing information in architectural heritage context.In this paper we have developed a semi-automatic approach for recreating information which were missing or were not acquired due to occlusion form single scan of Lidar data.We specifically focus our approach on building objects which have rotational symmetry such as columns.
This study by no means claims to be the first to recognise the possibility of retrieving 3D data via symmetry, particularly reflected (or bilateral) symmetry.There are mostly two approaches which have been found in the literature: 1) utilising a reference model of an existing object for adding up the missing information, and 2) other one using incomplete scanned objects itself.It is further divided into correspondence based or symmetry-based algorithms, where the first one uses the corresponding regions to fill the missing information, in a symmetry-based method exploit the symmetries to complete the object (Loy and Eklundh, 2006;Wenzel et al., 2008;Zhou et al 2016, Nagar andRaman, 2019).

Related work
The classical theory of symmetry groups describes the structure of transformations, which map objects to themselves exactly.Such exact, global symmetry leads to a group structure because after applying a transformation, we end up with the same situation as before, creating a closed algebraic structure (Mitra et al., 2013).One of the earliest studies on the topic of reconstructing a 3-D object using mirror image-based planar symmetry was done by Mitsumoto et al., 1992.The generation of projective invariants can also be done under the assumption of reflecting symmetry, which as Rothwell et al., 1993 noted can remove some constraints on the corresponding points.One nonaccidental 2-D model view is sufficient for recognition for any reflective symmetric 3-D object, according to (Vetter and Poggio, 1994) research.A thorough overview of research of reflecting symmetry and rotational symmetry in computer vision was given by (Zabrodsky et al. 1995) recognising 3-D objects and poses.A more thorough analysis of the repetitions in architectural scenes based on symmetry was presented by (Wu et al. 2010).
A bilaterally symmetric 3D object has a plane of symmetry, where corresponding points are opposite to each other with respect to this plane.The normal associated with this plane can be considered as the direction of symmetry (Sinha et al., 2012).
The properties of symmetry have been understood as one of intrinsic way of mapping a similar object with a relation of allowable transformation.Geometric data is typically represented as a collection of low-level primitives, such as point clouds, polygon meshes, NURBS patches, etc., without the explicit encoding of any underlying high-level structure, whether it was obtained through scanning or was created from scratch.Therefore, finding symmetries in such geometric data is a crucial topic in geometry processing.Extracted symmetry information has been used for many applications, such as shape matching (Sharma and Ovsjanikov, 2021), retrieval (Zhao et al., 2021), geometry completeness (Schiebener et al., 2016), structural and façade accuracy (Harshit et al., 2021, Zhou et al., 2016), and procedural modelling (Rumezhak et al., 2021).

Reconstruction Approach
As mentioned above, to recreate any existing building structure or heritage item, the main approaches are Photogrammetry and LiDAR.In the former images acquired from a distance with some common overlap is used which is also known as multi-view stereo in some of the literature (Hernández and Vogiatzis, 2010;Yang et al., 2005).The process of reconstructing a building in 3D by modelling it from uncalibrated photographs is achievable through the principles of photogrammetry and computer vision (Bourke, 2014).Photogrammetry based reconstruction is passively sensed data-based reconstruction.Utilising image based textural information and feature correspondences in image pairs to perceive depth is main principle used in such type of surface reconstruction.
Point clouds are the basic output derived through these types of processing, which are again used to generate reconstructed surface model.Photographic images are typically created using perspective projection, which follows the rules of linear perspective (Leopold, 2015).Because perspective projection is the projective representation of the object from a particular viewing position, it is possible to reconstruct the shape and location of the object based on its perspective image.This process of reconstruction was initiated by J.H. Lambert in the eighteenth century, and it has been a subject of research ever since.In recent years, reconstructing objects based on perspective images (i.e., photographs) has become increasingly popular, and various methods for achieving such reconstruction have been proposed.By using multiple photographs of an architectural object and knowing the position of the camera lens while shooting, it is possible to reconstruct the object accurately (Kulawiak, 2022;Ozimek et al., 2021;Harshit et al., 2022]. Laser scanning is getting prominence not only for documenting historical artefacts but also for modelling sophisticated outdoor/indoor building forms that are difficult to measure using other techniques.This mainly applies to intricate building facades or challenging-to-measure interior spaces (Zhou et al., 2016;Staats et al., 2018).Consequently, it appears that laser scanning is a quick and economical approach to collect the data of complex shapes (Kurdi et al., 2020).The geometric complexity of the building components that must be surveyed determines the optimal scanner type and scan settings (Borodinecs et al., 2018).Several commercial laser scanning devices are available now that can be categorised as semiautomatic (Thomson and Boehn, 2015).
Data for our study were gathered using terrestrial laser scanning.A tripod was used to support the scanner during the measurement.It generated laser pulses, which were subsequently reflected by the object.A point cloud was created using the measurements obtained from the laser scanning, and it served as a source of data for our object's measurements, geometry, and colours.The number of measured points in space with the coordinates (x, y, z) that make up a point cloud relies mostly on the settings used, the scanning quality, as well as the measurement time.Photogrammetry being based on image-based properties sometimes misses a lot of information in the building object and due to lack of feature points it might generate a lot of artefacts which are not present in the architecture.However, we have demonstrated photogrammetric data just to show a basic quality comparison between these two output point clouds.

Pre-processing
Columns are prominent features in ancient architectural marvels, for example in famous buildings such as Pantheon, Parthenon, and in Greek and Roman architectural masterpieces with their revival in colonial architecture in 18th century.In colonial architecture era, buildings have been mostly influenced by Greek and Roman way to design a space.This is the reason colonial architecture is also known as a revival of Roman architecture.The concept of order of columns which has been initially used in Greek period is adapted with few modifications in colonial architecture.The building under our study consists of an example of Tuscan order.Column architecture consists of three main components known as capital, shaft, and base.The data used here is one of those examples of colonial era in Indian architecture: the building is mere 175-year-old yet provides a marvellous grandeur in the city of Roorkee or its golden past.This heritage building is also known as James Thomason Building and is functioning as an administrative block for Indian Institute of technology Roorkee, India.
Point cloud for the columns were acquired using FARO terrestrial laser scanner.A single scan was taken for this study to develop the idea of recreating the missing information using rotational symmetry of the object with non-rigid alignment for minimising the geometric errors and distortion.Initial processing of the point cloud requires a manual edit to clip out only the portion of point cloud with partial column data.The information can be seen in the Figure 2. The point cloud is cleaned, and noise have been removed using Noise filter in cloud compare.Then point cloud has been exported to the python based Jupyter environment for further processing.

Detecting symmetrical axis
After pre-processing of point cloud of column segment, data further sub-divided into small horizontal subsets in form slices.These slices are used to detect the centre at multi-levels of subsets.By averaging the values of the points in all three dimensions, the centre of the object (cx, cy, and cz) can be calculated.It is assumed that all things perceived have a z-axis upward orientation since for fully unknown items, the ground truth orientation is similarly unknown.Finding only the χ that regulate the rotation in the z-axis is now the only challenge associated with getting posture information (ω, φ, χ).Then the incomplete point cloud vertical subsets are used for duplication and further 3D transformation.A Pseudo algorithm used for generating a symmetrical point cloud is shown in detail below.Find pose (cx, cy, cz, ω, φ, χ) 6: Generate local frame at cx, cy, cz 7: for each point p in i do 8: Calculate Euclidean distance (c, p) 9: Negate distance and find point m 10: Add m to i 11: end for 12: end for 13: end procedure Figure 3. Column point cloud and horizontal subsets

3D transformation and Reconstruction
3D Transformation parameters mostly requires the rotational elements as no translational movement is allowed.Using rigid transformation case with stable rotational axis in Z direction, the degree of freedom here is restricted.The centroid of the item is assumed to be a local reference frame, and the Euclidean distance between each point and the centroid is determined.By scaling each point by a negative value of one unit in relation to the distance, all the points are reflected on the occluded side.Hence, a mirrored point Mi (px, py, pz) is formed for each and every point Pi (px, py, pz), where px, py, and pz are the distances in the x, y, and z directions from the object's centre.

RESULTS
The images shown below are the output of the above-mentioned algorithm.These column objects after completion are refurbished in their location through point-to-point registration.To obtain a complete approximated model of the perceived object, the points perceived are mirrored based on the computed orientation (cx, cy, cz, 0, 0, χ).Though the orientation does not contain any ω and φ values, this mirroring method can work on any arbitrary orientation of the object.Reconstruction always begins with establishing the underlying principles of perspective, whereas the chosen technique for perspective representation depends on where the object is positioned in relation to the observation plane.The orientation of the scanner affects how accurate the reconstruction approach is used.Yet, the survey's findings have demonstrated that it is enough for reconstructing historical events.The available data is used as a starting point for the reconstruction techniques for architectural objects.To create a 3D model using the information and the method, geometrical knowledge is needed.When it comes to historically significant buildings which have artifacts that are non-existent, geometric reconstruction from a point cloud is both feasible and necessary.
In cases of reconstruction for tourism and historical study as well as for social and cultural objectives, such as to reinforce a region's sense of identity, the results may be helpful.(Schnabel et al., 2007), as shown above in the figure 6, these segments comprise of capital, shaft and base of the column which has been clearly identified based on the neighbourhood principle used in segmentation algorithm.

Accuracy and Experimentation
Reconstructed point cloud has been compared with a modelled mesh of column generated through physical observation (figure 7).The overall error is found to be normally distributed.Near top portion of the reconstructed column shows a positive deviation from reference surface and the base and lower portion of shaft shows negative deviation from the reference model.This might be caused by the noise points in those areas which has been generated during the partial acquisition of the original data, thus filtering for those points before mirroring is suggested.
Utilising recreated columns for an extended application has been experimented to create a series of columns.Extremities of the other partial column objects with defined centres from the scan has been used as a reference to register the duplicated column objects.The approach for repetitive element completion is acquired here in semi-automatic fashion for an architectural heritage with symmetrical component.The registration and duplication can be automated as an extension of this research in future development.

CONCLUSIONS
This research provides a semi-automatic approach of solving problem of incomplete data due to occlusion and insufficient coverage in surveying symmetrical architectural objects.The object that has been used for reconstruction plays an important role, as the final output depends on the initial acquired data.
Symmetry has been existed for a very long time in the object that has been built historically.In most of ancient architectural philosophy, symmetry can be described as structural invariances through transformation.The approach demonstrated here is exploitation of that same concept to recreate the geometry of symmetrical component of those architecture.In heritage modelling using geospatial techniques this approach could help in filling the gaps in the observed data by utilising the symmetrical aspect of any structure.Our workflow has been deployed in CloudCompare which is a open source software for point cloud processing and algorithm development is done in python based environment using jupyter notebook.
The limitation of this study is to acquire partial data very accurately.The assessment of this generated output is done by inspecting its dimension with respect to the real-world object.
The future scope of this research could be to create a hybrid approach that will allow for the use of reconstruction from photographs, when they exist, and when the object cannot be scanned because it has lost its architectural components.Alternatively, it will be possible to scan existing building components when they are not visible in the photographs, while still maintaining the architectural integrity of the selected heritage building.

Figure 2 .
Figure 2. LiDar data for the front facade.

Algorithm 1
Generate Symmetrical Point cloud input (point cloud cloud) 1: procedure CompletepointCloud 2: Segment out the Column 3: Segment subset Objects 4: for each i in Objects do 5:

Figure 4 .
Figure 4. Workflow for Point cloud Completion

Figure 5 .
Figure 5. Subset data for the column object with rotational symmetry highlighted in figure 2 (left), point cloud with completed geometry (right)

Figure 8 .
Figure 8. a) Set of columns with initial point cloud, b) Processed output