SITEGI PROJECT: APPLYING GEOTECHNOLOGIES TO ROAD INSPECTION. SENSOR INTEGRATION AND SOFTWARE PROCESSING

Infrastructure management represents a critical economic milestone. The current decision-making process in infrastructure rehabilitation is essentially based on qualitative parameters obtained from visual inspections and subject to the ability of technicians. In order to increase both efficiency and productivity in infrastructure management, this work addresses the integration of different instrumentation and sensors in a mobile mapping vehicle. This vehicle allows the continuous recording of quantitative data suitable for roadside inspection. The geometric integration and synchronization of these sensors is achieved through hardware and/or software strategies that permit the georeferencing of the data obtained with each sensor. In addition, a visualization software for simpler data management was implemented using Qt framework, PCL library and C++. As a result, the developed system supports the decision-making in road inspection, providing quantitative information suitable for sophisticated analysis systems.


INTRODUCTION
Good quality infrastructure and their preservation play a key role in any country's economy.Transportation systems are the most demanded infrastructure.Therefore, investments in the road network often suppose a substantial share in the total transport investment.Moreover, due to the cost of implementation of new transportation systems, roads and infrastructure maintenance needs must be addressed in order to save costs and optimize financial resources.Issues such as road exploitation and road traffic have become topics of great relevance in the current Spanish society, where a road accident is one of the main problems that public administrations have to face (DGT, 2013).
Existing inspection systems are mainly based on technical reports provided by visual inspections, with a minimal technological background and, limited in most cases, to the digital imaging and on site georeferenced data gathered by experienced practitioners.Data obtained from such inspections, must feed official databases for road and associated infrastructure.These databases basically contain information regarding geometric inventories and damage inspection.The lack of technical capacity and productivity is one of the cons detected in these procedures.Productivity is estimated by means of the number of road kilometres analysed per time unit and human resources used for the acquisition and evaluation.Consequently, the quality and quantity of information used for decision-making is very low.
These traditional procedures contrast with the need for objective and quantifiable data that can support decision-making and priority interventions.The project named "Technological Systems for Infrastructure Management" (SITEGI) project is set within this context.Given that the geometric data may provide an accurate knowledge of the current state of infrastructure, the main objective of this project is to apply geomatics for infrastructure inspection.These geometric data are meant to be used as an input for calculation functions based on mathematical methods (finiteelement based modelling for structural analysis) and computational methods (neural networks, Bayesian networks, decision trees, etc.).The obtained information predicts the future infrastructure behaviour, thus making decision-making more effective (Solla et al., 2012).
Besides, recent advances in Mobile Mapping and Surveying systems allow for productivity during data collecting .There are a number of novel examples in the literature that apply these techniques to road mapping and inventory.The Finnish Geodetic Institute has presented a mobile mapping system, named ROAMER, that consists of navigation systems, and a 3D data acquisition system.This system has been applied to road environment applications (Kukko et al., 2007).A Mobile Mapping System named Road-Scanner by SITECO has been applied to highly detailed surveys of dense urban areas.This system consists of 2GPS and IMU sensors for navigation (Applanix POSLV), four cameras for close range photogrammetric surveys and a Laser scanner FARO LS880 (Gandolfi et al., 2008).The FGI Sensei is a low-cost mini-UAV-based system consisting of a number of measurement sensors, which mounted on a car is suitable for road mapping.This system is capable of not only obtaining object geometry with laser scanners, but also collecting image data, laser intensity, as well as hyperspectral and thermographic images (Jaakkola et al., 2010).A LIDAR based road edge extraction algorithm is presented in (Mc Elhinney et al., 2010).In order to obtain the datasets, a new mobile mapping platform XP-1 was designed and implemented.The system comprises an IXSEA LandINS GPS/INS, a Riegl 3D LIDAR sensor, and an imaging system compound of both progressive and thermographic cameras and a 5-CCD multispectral camera for visible,infrared and thermal spectral signatures recording.Other Mobile Mapping approaches are discussed in (Puente et al., 2013a).
The context of the present research covers the application and adaptation of mobile geotechnologies to road infrastructure inspection as well as the application of collected data for undertak-ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume II-5/W2, 2013ISPRS Workshop Laser Scanning 2013, 11 -13 November 2013, Antalya, Turkey ing the structural evaluation.
This paper presents two essential milestones in the SITEGI project.Firstly, the development of a laboratory vehicle that integrates a number of sensors for the geometric and radiometric characterization of roads and other infrastructure.This vehicle ensures quality data collection.And secondly, the development of a software application that supports the interpretation of the data collected by the sensors.This software also allows feeding the official databases for geometric road inventory and damage inspection guides.

MATERIALS AND METHODS
The need for a high productivity vehicle arises from the huge amount of data required for road routine inspections, also permitting the automated data acquisition from a number of sensors.This mobile-mapping vehicle is defined as a Mobile Inspection Unit (MIU) and must integrate and synchronize the measurements of the different sensors that is composed of.(view Figure 1).The MIU is based on an Optech Lynx system, which supports the synchronization of external inputs by its inertial measurement unit (IMU).In the next subsections, the different external sensors and their synchronizing strategies are described whereas the final subsection deals with the requirements and properties of the developed software.

Optech Lynx
The Lynx mobile mapping system by Optech allows for massive auscultation of road elements (paving, tunnel and slope) and operates synchronized with the movement of the vehicle.Thanks to its GNSS system, its measurements are automatically georeferenced.
The Lynx consists of different elements, namely: • Applanix POS LV-520 GNSS/INS system, where long-term accuracy of the GNSS system is complemented with shortterm measurements of the INS .This system provides georeferencing, positioning and timing information of the captured data, aided by attitude sensors and odometry.• Two laser scanning units that generate 3D point clouds, which allow the construction of digital models of infrastructure and their surrounding areas.Having two scanning heads minimizes the generation of shadows in the point cloud.Technical characteristics of these laser scanners are shown in Table 2.
• Multicamera computer vision system.The system has installed a set of four geometrically calibrated cameras that permits colorizing the point cloud as well as a georeferenced image inspection.Camera parameters are shown in

Synchronization capabilities of the Optech Lynx system
The synchronization of data from different sensors with the smooth best estimated trajectory (SBET) of the mobile unit can be performed based on three variables that are unequivocally related: 1. IMU Timestamp.The timescale used is known as GPS time.GPS time is set by atomic clocks and represents an accurate time measurement suitable for synchronization.In addition to the registration of this timestamp, the navigation system provides a synchronizing signal called 1PPS (1 Pulse Per Second) that consists of a Transistor-Transistor Logic (TTL) flank signal which is coincident with the GPS time.Registration of 1PPS signal enables a third-party sensor to synchronize data with GPS time.
2. Absolute position.In case of a sensor registers its data to GNSS absolute coordinates, these coordinates could be used for integration, considering the restriction posed by the achieved precision.
3. Distance travelled from the SBET origin.Some kinds of sensors provide data that are tagged with an odometer distance measurement.In Figure 2   clock whose frequency depends on the speed of the MIU; this clock can be used for sensor synchronization.
In order to assign one of these three variables to data gathered from an external sensor, the IMU provides two main synchronization methods: 1. Event tagging.Two different external signals can be introduced as events to the POS system.These events consist of a TTL flank signal and are tagged with position, attitude values and a timestamp with a frequency up to 300 Hz.
2. NMEA messaging.The NMEA is a standard protocol used to exchange text messages, namely NMEA sentences, with GPS receivers.The protocol supports a number of different messages including position and time communication.
Applanix POS equipment provides five ports which can be used to send NMEA messages for synchronization with a maximum frequency of 50 Hz.

Thermographic Camera
Infrared thermography is a technique for producing an image of the infrared light, invisible to the human eye, emitted by bodies due to thermal conditions.This radiation is generated by the characteristic vibration and rotation of the atoms and molecules that comprise a material, thus, the higher the temperature, the greater the movement of these particles and accordingly, higher energy radiation.Given that a body radiates energy as a function directly proportional to its temperature, a thermographic camera is able to measure the radiation from an object, and to perform the necessary calculations to generate an image with related temperature data.
The radiant behaviour of a body is defined by two physical laws (Kaplan, 2007): Equation 1 is the Stephan-Boltzmann law that relates the radiant energy (W )emitted from the target and its absolute temperature (T ), where is the emissivity and δ is a constant.Equation 2 is the Wien's displacement law, where λm is the wavelength of maximum radiation, b = 2897 and T is the target temperature.
Applications of thermography to road inspection are numerous.For example, during road construction, monitoring pavement and soil temperatures can be useful to detect blistering in bridge decks (Stimolo, 2002).A variation of surface temperatures is a good indication of possible future slippery areas to appear on the roads.A thermographic study would provide useful information for drivers to prevent accidents.The use of thermography inspection is especially suitable in tunnels, where inspections can be carried out in the absence of solar radiation.
The thermographic camera is available for the MIU, is the GOBI 384 by Xenics.This camera is an affordable passive thermography unit, with a flexible TCP / IP interface and compact design.Table 4 summarizes the technical specifications thereof.

Synchronization
The synchronization strategy for the thermographic camera is based on NMEA messaging and event tagging.This HW strategy is implemented with an Arduino board (view Figure 3).This Arduino board accepts serial port NMEA messages from the POS LV and generates a HW trigger signal for the camera and for the POS LV event tagging input simultaneously.The system interface to practitioners consists of two buttons for start and stop acquisition.With a start button press, the system stores the timestamp from the POS LV and enables camera triggering.With the stop button, the system stops camera triggering and sends the stored timestamp to the practitioner's laptop through the serial port.

Ground Penetrating Radar (GPR)
The GPR is a geophysics instrumentation that permits the inspection of terrain sub-superficial properties.It is well-known that the velocity of an electromagnetic wave depends on the properties of the medium through which it is traveling.Taking into account that most of the materials in GPR applications have a magnetic permeability µ = µ0, and the GPR working frequencies, the velocity is defined by Equation 3, where r is the dielectric permittivity of the medium.
ISPRS Annals the Photogrammetry, Remote Sensing and Information Sciences, Volume II-5/W2, 2013 ISPRS Workshop Laser Scanning 2013, 11 -13 November 2013, Antalya, Turkey GPR emits high-frequency electromagnetic signals that travel through the subsoil, where changes in the electrical properties of a medium cause reflected waves that return to the surface; these waves are stored by the GPR for interpretation.These stored waves are usually named as traces.
Given that GPR provides subsoil imaging, it is able to detect buried facilities, such as pipelines or metal rods in reinforced concrete.In road inspection, the application of GPR is a keystone for measuring pavement layers thicknesses (Puente et al., 2013b); this information is very useful to compute necessary reinforcements after ongoing traffic and weather impacts over time.
GPR can be used to discover gaps and water between pavement layers as well.For the structural evaluation, and combined with laser scanner and photogrammetry modelling, GPR can gather information about the inner materials in the soil that allows for finite element analysis of damaged structures (Lubowiecka et al., 2009).
GPR surveys are performed using a fully nonmetallic trolley that has been specifically designed considering the requirements of using a multipurpose GPR system and ground-coupled antennas to road inspection (Lorenzo et al., 2011).

Synchronization
The GPR measurement equipment used in this work is the ProEx control unit by MALA Geoscience and two ground-coupled antennas with frequencies of 1GHz and 2.3GHz, which are capable of measurements up to 1 meter and 0,6 meter respectively.The control unit of the GPR can be triggered based on a time interval (SW) or a spatial interval (HW) measured with an odometer.Accordingly, synchronization strategies 1 and 3 in Section 2.1.1 are possible.Given that the GPR acquisition software can connect to an external GPS for trace tagging, NMEA synchronization was used in this case.If the connexion is set properly, the GPR acquisition software creates a number of files in order to get the GPR traces paired with a GGA sentence.After post-processing, the traces are referenced to the SBET.

Laser profilometer
The International Roughness Index (IRI) parameter is used for determining regularity of the pavement and, thus, traveling comfortable on the road taking into account a model of the vehicle.The extended model is the so-called quarter-car because it represents only a quarter of the vehicle.A sketch of this model is shown in Figure 4. Using a simplified system comprising a suspended mass (the mass of a quarter of a car) connected to a nonsuspended mass (the shaft and pneumatic) through a spring and a damper line (suspension) and finally a linear spring modelling the tyre (Sayers and Karamihas, 1998).
The dynamic equations of these elements form the system of Equations 4 that uses the road profile y as an input and where z1 and z2 are the vertical displacements of unsprung mass (m1)and sprung mass (m2) respectively.k1,2 are the rigidity constants and c1,2 are the shock constants of the suspensions, where subscript 1 denotes the primary suspension between pavement and unsprung mass (the tyre) and subscript 2 denotes the secondary suspension between sprung and unsprung masses (the car suspension) .In order to obtain a non-contact measurement of the road, a laser profilometer was added to the system.This profilometer consists of two laser distancemeters and a couple of accelerometers per distancemeter whose measurements permit to obtain the car model.
Figure 4: Mathematical model of the quarter-car

Synchronization
The laser profilometer makes use of an incremental encoder as an odometer in order to trigger the measurement of the distance and orientation sensors at a fixed spatial interval.Thus, this sensor can be synchronized using the odometer data as a clock signal because the distance recorded by the profilometer is exactly the same as reflected by the Applanix system.After post processing, every measurement of the profilometer is tagged with the actual value of position referenced to the SBET.In Figure 5 an adapting cable for sharing the POS LV encoder with the profilometer control unit is shown.The profilometer can be synchronized using NMEA messaging as well: the profilometer control unit provides a user input for external communication that is compatible with the NMEA GGA sentence, which contains time and position information from the POS LV system.The frequency acquisition of the communication is limited to 1 Hz,therefore, the control unit stores absolute coordinates every second and more importantly, the GPS time received.Accurate coordinates in the SBET file that belong to that exact GPS time are used as a reference after post processing.Accurate timing for the GPS time is achieved through the 1PPS signal.
Figure 5: Adapting cable for sharing the same encoder sensor among POS LV and profilometer control unit.

Software methodology
A software application arises from the need to display the different data gathered from different sensors.After data collection with the mobile system, the georeferencing and synchronization of the different elements is achieved by comparing the fields of spatial coordinates and timestamp (GPS time).In order to make ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume II-5/W2, 2013 ISPRS Workshop Laser Scanning 2013, 11 -13 November 2013, Antalya, Turkey the data importation from the sensors easier, this piece of software implements the file conversion from data-acquisition output to the objects structure that handles the application.This reduces the number of resources needed for reading files by removing unnecessary information.The vewhicle trajectory is the nexus of all data that the application software has to manage, allowing synchronization of all input elements by using the spatial synchronization between the coordinates of the elements.

Used Libraries
The application was implemented in C++ language, and for the graphical user interface (GUI), Qt framework was selected.Qt is a set of libraries distributed under the license GNU Lesser General Public License, that is designed to develop cross-platform GUI applications and uses the C++ language natively.One of the requisites of the software is loading large point cloud files and to permit a fluid navigation of the data.
In order to achieve these requirements as much as possible, point cloud library (PCL) was selected for data management (Rusu and Cousins, 2011).PCL is an open source library for point cloud management with specific modules for visualization, filtering or segmentation, among others.This library uses its own point cloud file format, PCD, which was created in order to complement existing file formats.We selected the PCD format because of the flexibility and speed of its files, the ability to store and process organized data sets in a binary format that results in a fastest way of storing and retrieving data on disk.
2.5.2Data Analysis Some sensors provide an output that is not easy to understand by the operators responsible for the inspection of roads.This is the case for both the GPR and the profilometer.
For the profilometer, the most popular measurement needed in road inspection is the IRI.In our case, we use the commercial software supplied with the sensor that allows the user to mark the start and the end of each measurement.Standard IRI measurements are obtained for road intervals of 100 m.
The case of GPR is different because results depend on the degree of expertise of the user.The georeferenced radargrams obtained in the system should be analyzed by expert users in order to achieve a correct interpretation of the road condition.In our case, the value of interest obtained from radargrams is the thickness of the different layers of pavement.This value is dependent on the type of antenna used and its configuration, requiring the auscultation by expert users: the interfaces between pavement layers are reflected in the traces obtained by the GPR, enabling the measurement of layer thickness based on delay-time or amplitude of the echoes for ground-coupled and air-launched antennas, respectively.

RESULTS
In this section the principal results of the integration of the sensors are presented for both the inspection vehicle and the software application developed.

Mobile Inspection Unit
As a result of the synchronization strategies described in the Section 2 of this document, a text file is generated for each of the integrated sensors.This text file is divided in columns that contain the absolute coordinates of the sensor, the GPS timestamp and an information column depending on the sensor.This column shows the value of the IRI for the profilometer, the pavement layer thickness for the GPR and the filename of the thermogram for the thermographic camera.Optionally, values of the attitude Frame rate 10 fps (16 bit) Frame Distance @ 60km/h 1.67m.Pavement Field of view H: 9.6 m.V: 3.2 m.
Table 5: Technical specification of the GOBI 384 sensors of the POS LV could be addressed for the thermographic camera.
The thermographic camera has some particular properties.The necessary integration time of the camera sensor for a good contrast in the pavement, limits the maximum speed of the van to 60km/h.Given the position of the camera in the roof rack of the van and the field of view of the camera, the field of view in the pavement is of 9.6x3.2meter.Considering the maximum speed of the van, an acquisition frequency of 10 fps will be enough for obtaining a continuous recording of the lane , resulting in savings in bandwidth regarding the maximum frame rate of the camera (25 fps @16bits).In Table 5 some properties of the integration are shown.

Visualization Software application
The application consists of a tree of elements where imported items are loaded.From this tree of elements, configuration, preview and access to certain properties are possible through a context menu that appears when right clicking on the item.
The right area of the screen is the display area, in such a way that when the user clicks on an item of the element tree, it will be displayed in this area.As mentioned, given that all the elements are referred to the SBET, it is possible to navigate through them synchronously with the trajectory.
The point of view configuration of the point cloud permits to set the point cloud visualization from top or front.The latter is useful to have the view from vehicle's position.During navigation, the trajectory shows the actual point where we are located on the SBET by a red dot, also indicated with an arrow on the point cloud and a marker in IRI and GPR profiles.
For continuous visualization mode, advances on the trajectory every 1, 5, and 20 meters are possible.Moreover, the time-lapse for visualization is configurable.In addition, it is possible to set a proper point on the trajectory manually.Figure 6 shows a view of the SITEGI application interface where an example of typical data set is presented; in this picture, are displayed (from left to right, top to bottom): a window with the trajectory of the vehicle, a point cloud colored with intensity data return the laser beam, a panoramic RGB image, a query window with firm thicknesses obtained using GPR, and a window with the IRI data of the section obtained using the profilometer data.
ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume II-5/W2, 2013ISPRS Workshop Laser Scanning 2013, 11 -13 November 2013, Antalya, Turkey Visualization of thermographic images of the roadside is possible as well, although thermographic information is not enabled in this example in order to improve the readability of the picture.

CONCLUSIONS AND FUTURE WORK
As a result of the process of integration and synchronization of the instrumentation implemented in the MIU, every sensor data is georeferenced.The position of the data is known with the SBET as a reference and taking into account the lever-arm of the sensor, defined by its position in the van.Orientation is also known for those sensors whose events are tagged by the POS LV system.
The development of the integrated measurement system presented in this work allows the collection of a large amount of heterogeneous data in an automated and efficient manner.The improvement in productivity in infrastructure inspection is one of the main advantages over previous inspection methodologies.Although the speed limit of the vehicle with all integrated equipment is fixed to the slowest sensor, another notable feature of the system is its modularity.Practitioners can easily adapt the number and type of sensors equipped depending on the infrastructure or the specific application.Generating the right amount of data is a key to make the inspection process more efficient.
The successive introduction of the generated data in a visualization software allows the infrastructure assessment and management more efficiently.This results in reduced management and maintenance costs.
Analysis and evaluation of the structural condition of the infrastructure remain as future work.This will result in reducing the risk of deterioration and minimize the occurrence of accidents because of structural failures.

Figure 1 :
Figure 1: Sketch of the Mobile Mapping System and external sensors for roadside inspection.

Figure 2 :
Figure2: Output signal of an odometer sensor.This output signal may be considered as a clock for integration purposes.

Figure 3 :
Figure 3: Schematic of the Arduino board to be connected to the thermographic camera, user PC and POS LV.

Figure 6 :
Figure 6: GUI of the software application for data management.Different sensor data referenced to the trajectory are shown.

Table 2 :
Properties of the laser scanning heads available in the Optech Lynx.

Table 3 :
a typical output of an odometer is presented.This output signal may be considered as a Camera System Technical Specification.ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, VolumeII-5/W2, 2013  ISPRS Workshop Laser Scanning 2013, 11 -13 November 2013, Antalya, Turkey