PERFORMANCE TESTING OF 3 D POINT CLOUD SOFTWARE

LiDAR systems are being used widely in recent years for many applications in the engineering field: civil engineering, cultural heritage, mining, industry and environmental engineering. One of the most important limitations of this technology is the large computational requirements involved in data processing, especially for large mobile LiDAR datasets. Several software solutions for data managing are available in the market, including open source suites, however, users often unknown methodologies to verify their performance properly. In this work a methodology for LiDAR software performance testing is presented and four different suites are studied: QT Modeler, VR Mesh, AutoCAD 3D Civil and the Point Cloud Library running in software developed at the University of Vigo (SITEGI). The software based on the Point Cloud Library shows better results in the loading time of the point clouds and CPU usage. However, it is not as strong as commercial suites in working set and commit size tests.

The different systems can be divided in aerial and terrestrial LiDAR.Terrestrial LiDAR can be further divided in stationary and mobile LiDAR.Stationary LiDAR uses base stations to the data acquisition and registration process.Mobile LiDAR uses navigation systems to determine the position and orientation of the mobile platform, based on Global Navigation Satellite Systems and Inertial Measurement Units (Kavanagh, 2007).They are combined with the laser measurements to obtain the whole dataset (Petri 2010;Puente, 2013).One common characteristic between mobile and static systems is the huge datasets typically obtained.Over tens of millions of points is common.This involves the use of high performance computers and software suites to process the datasets.
The aim of this work is to design and perform a software performance test to understand the behaviour of different possibilities available for LiDAR users.The software under study are Quick Terrain Modeler from Applied Imagery, AutoCAD 3D Civil from Autodesk, VRMesh from Virtual Grid and the open source Point Cloud Library integrated in a software developed at the University of Vigo (SITEGI).There are other possibilities in the market (i.e.Cloud Compare, Mehslab, LASTools), but it is difficult to treat them all in a single work.

Area of study
The A-52 highway (Figure 1) joins the cities of Benavente and O Porriño, both in North West Spain.It has 306 km long and crosses three provinces.It was built between 1994 and 1999.
The length of the highway section used for this study is about 1 km, located around the village of Ponteareas.

Surveying
The data acquisition was done with the Optech Lynx mobile LiDAR (Figure 2).The Lynx system combines the Applanix POS LV520 for positioning and navigation with two LiDAR sensors.The accuracy of the range data is 6 mm, the scan frequency is 200 Hz, the field of view is 360º and the pulse repetition rate is 500 kHz.The point density on the road surface (2 m length from LiDAR sensors) is about 3500 points/m 2 .The navigation system provides an accuracy of 0.015º in heading, 0.005º in roll and pitch, 0.02 m in horizontal and 0.04 m in vertical positioning (values calculated using differential GPS positioning).The system also includes four digital cameras for the colorization of the point cloud.They present a resolution of 5 Mpx.All the sensors (LiDAR and cameras) are boresighted to the navigation system.

Data processing
Point cloud was obtained using the Applanix POSPac and Dashmap software.The first one generated the SBET -eStimated BEst Trajectory from the positioning and navigation data of the POS LV520.Positioning data were corrected using a RINEX file from a static GNSS -global navigation satellite ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume II-5/W2, 2013ISPRS Workshop Laser Scanning 2013, 11 -13 November 2013, Antalya, Turkey system.Navigation data were taken from the ring laser gyros integrated in the inertial measurement unit.The data were combined using a Kalman filter by the POSPac to obtain the SBET.Dashmap software uses the time stamp of the GNSS and the boresighting to combine the range data from the LiDAR with the SBET data.The output of the software gives a point cloud (LAS format) with UTM coordinates (Figure 3).

LiDAR management software
Four different software suites were tested for this work: QT Modeler, AutoCAD 3D Civil, VRMesh and Point Cloud Library with SITEGI software.Next, a brief description of the main characteristics is provided.QT Modeler allows the fusion between LiDAR and imagery data to enhance the realism of the scene.

AutoCAD 3D Civil
AutoCAD Civil 3D software is a Building Information Modeling solution for design and documentation in the civil engineering field from Autodesk.AutoCAD Civil 3D extends the value of the 3D modeling with new workflows and productivity tools for civil infrastructure.Some of the features of the software include surveying tools, hydraulics and hydrology, parceling, profiling, pipe modeling, corridor modeling and point cloud utilities.In addition, it allows the extraction of road features, automatic point cloud classification, creation of ground surfaces and segments, ground filtering and segmentation and simple classification.

VRMesh
VRMesh is the point cloud and triangle mesh processing solution from Virtual Grid.VRMesh is an advanced solution covering the complete workflow from automatic point cloud classification, feature extraction, point cloud decimation, denoising, global registration, point cloud triangulation, mesh repair and editing.

PCL -Point Cloud Library (SITEGI software)
PCL is an open source library with specific modules for the management of point cloud data (e.g. visualization, filtering, segmentation).The library was integrated in a software suite specifically developed at the University of Vigo.The basic requirements of this software were: running under different operating systems, powerful language, object orientation, free license and technical support.
To meet the above premises, the framework Qt was selected which uses the C++ language.MinGW was used as compiler (64 bits).Qt was also connected with the VTK visualizer through the QVTKWidget.The complete system was called SITEGI software (Figure 4).PCL does not support the LAS format, so the dataset was converted to specific binary PCD files (PCL format).

Software testing methodology
Software testing is an investigation conducted in order to provide information about the quality of the product under test (Singh, 2012;Shahrokni, 2013).The software testing can be started as the process of validating and verifying that a computer program meets the requirements that guided its design and development, works as expected, can be implemented with the same characteristics and satisfies the needs of stakeholders.
The testing methodology includes the following aspects: loading time, CPU usage, working set and commit size.The CPU usage is the amount of time for which a CPU -central processing unit was used for processing instructions of a computer program.Typically is mentioned as the percentage of the CPU's capacity at any multi-tasking environment.
Working set consists of the amount of memory in the private working set of the process and amount of memory shared with other process (Denning, 1968).The working set of information of a process time is defined as the collection of information referenced by the process during the process time interval.
Typically the units of information in question are considered to be memory pages.This is suggested to be an approximation of the set of pages that the process will access in the future during the next ∆t time units, and more specifically is suggested to be an indication of what pages ought to be kept in main memory to allow most progress to be made in the execution process.It is one of the main aspects when a program must manage large datasets.
Peak working set is defined as the maximum amount of working set memory used by the process since its inception.
Private working set is the amount of memory in the private working set, non-shared with other processes.
Commit size consists of the virtual memory reserved for the process.This term, used mainly in Microsoft operating systems, describes the total amount of pageable virtual address space for which no backing store is assigned other that the pagefile.On systems with a pagefile, it may be thought of as the maximum potential pagefile usage.On systems with no pagefile, it is still counted, but all such virtual address space must remain in physical RAM memory at all times.

Computer
The computer used for the tests is a Dell Precision M6600 with a control processing unit Intel Core i7 -2920 XM CPU 2.5 GHz, 8 GB of installed RAM memory and Windows 7 x64.Loading time is one of the key factors in the software evaluation because it directly affects the productivity of the work.For example, AutoCAD 3D Civil forces to stop the activities of the technician around 5 minutes during the loading time of a dataset of 25 million points.This loading time, in a road of 100 km segmented in strips of 1 km for easy processing, results in around 500 minutes spent in loading (around one day's work), so the negative impact on productivity is clearly shown.Figure 6 exhibits the CPU usage.The usage of the CPU of SITEGI is the smallest one.This fact results very important because allows free resources for other tasks in the computer.It should be remarked that it is the only software package that combines a quick loading time with a small CPU usage.For example AutoCAD 3D Civil also depicts a low CPU usage, however the loading time is high.On the contrary, the loading time of QT Modeler or VR Mesh is quite fast with a high CPU usage.

MATERIALS AND METHODS
Figure 6.Comparison of CPU usage.
Figure 7 shows the behaviour of the commit size.SITEGI software shows poorer results than the other commercial options under study.The commit size reaches around 4 GB, while the commercial solutions are always under 1 GB.These results are in agreement with those obtained for the working set.
Figure 8 exhibits the working set results.SITEGI software shows a working set higher than the other software under study (e.g. 3.2 GB for 25 million points, while the other software suites require less than 1 GB).This characteristic affects to all the software running in parallel with these one and limits its capability to manage large datasets with large memory requirements.This behaviour seems to come from the Visualization Toolkit Library (VTK 2013).PCL uses VTK for the processing of the point clouds and its visualization.VTK apparently loads an important number of modules on memory for the operations.
ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume II-5/W2, 2013 ISPRS Workshop Laser Scanning 2013, 11 -13 November 2013, Antalya, Turkey   On the other hand, SITEGI software requires important resources related with the different memory requirements (working set and commit size), being the other systems more efficient.

Figure 4 .
Figure 4. Screenshot of SITEGI software.Imagery on the top and point cloud on the bottom.
Loading time is the time it takes to load point clouds of different sizes.Starting from a full dataset consisting of 25 ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume II-5/W2, 2013 ISPRS Workshop Laser Scanning 2013, 11 -13 November 2013, Antalya, Turkey million points, 25 subsets with a size running from 1 to 25 million points were created.

Figure 5
Figure 5 shows the loading time in the software under study for different sizes of point clouds.It ranges from 2.2 s for 1 million points to 22.0 s for 25 million points in the SITEGI software, which shows the best results of the software suites under study.The results obtained from QT Modeler are close to those obtained from SITEGI (2.5 s for 1 million points and 33.5 s for 25 million points).VRMesh shows intermediate behaviour, while AutoCAD 3D Civil depicts the worst results (40.7 s for 1 million points and 353.8 s for 25 million points).

Figure 5 .
Figure 5.Comparison of point cloud loading time.

Figure 8 .
Figure 8.Comparison of working set.
the evaluation of different LiDAR software was developed and tested for the commercial suites QTModeler, AutoCAD 3D Civil, and VRMesh and the open source SITEGI software.SITEGI software uses the Point Cloud Library for the management of the LiDAR datasets.The methodology was tested using only mobile LiDAR datasets.Results provided by the SITEGI software show the combination of PCL and the developments of the University of Vigo for the software.The methodology does not introduce operations that could affect the accuracy of the data.SITEGI software depicts the best results in terms of loading time and CPU usage.The loading times range from 2 s for 1 million points to around 20 s for 25 million points, while other software as AutoCAD 3D Civil depicts values ranging between 40 s and 354 s for the same sizes.A small loading time improves the productivity of the work and contributes to increase the human -computer interactivity.QT Modeler shows results similar to SITEGI, and VRMesh intermediate results between AutoCAD 3D Civil and the other suites.