An Interactive Tool for Analysis and Optimization of Texture Parameters in Photorealistic Virtual 3d Models

Texture mapping is a common method for combining surface geometry with image data, with the resulting photorealistic 3D models being suitable not only for visualization purposes but also for interpretation and spatial measurement, in many application fields, such as cultural heritage and the earth sciences. When acquiring images for creation of photorealistic models, it is usual to collect more data than is finally necessary for the texturing process. Images may be collected from multiple locations, sometimes with different cameras or lens configurations and large amounts of overlap may exist. Consequently, much redundancy may be present, requiring sorting to choose the most suitable images to texture the model triangles. This paper presents a framework for visualization and analysis of the geometric relations between triangles of the terrain model and covering image sets. The application provides decision support for selection of an image subset optimized for 3D model texturing purposes, for non-specialists. It aims to improve the communication of geometrical dependencies between model triangles and the available digital images, through the use of static and interactive information visualization methods. The tool was used for computer-aided selection of image subsets optimized for texturing of 3D geological outcrop models. The resulting textured models were of high quality and with a minimum of missing texture, and the time spent in time-consuming reprocessing was reduced. Anecdotal evidence indicated that an increased user confidence in the final textured model quality and completeness makes the framework highly beneficial.


INTRODUCTION
The advent of communication and visualization technology has created an increased demand for photorealistic 3D models of real-world scenes.Such models are very useful in many application fields ranging from the earth sciences and archaeology to urban planning, virtual tourism and heritage documentation.Approaches to model creation and data acquisition differ depending on the domain requirements.Where high resolution and high accuracy 3D models are essential, spatial data can be acquired using laser scanners (lidar -Light Detection and Ranging) and images with calibrated metric or semi-metric digital cameras (Vosselman and Maas, 2010).In applications where high resolution and accuracy are not crucial, automatic reconstruction of the 3D structure from multiple images taken from different views (structure from motion or from stereo vision) is regarded as an attractive lowcost approach in 3D model creation (e.g.Snavely et al., 2008).In this case images are mostly collected with semi-metric or non-metric digital cameras and the intrinsic and exterior camera orientation parameters are recovered by the software generating the model.The accuracy of these parameters depends on the software and the input data quality, influencing the quality of the final textured model (Remondino and Ressl, 2006).
Regardless of the 3D surface creation approach, texture mapping is a critical step for enhancing the visual appearance of 3D models.When capturing images for texturing a model, it is usual to collect data from multiple locations, mostly with large overlap, and sometimes with different camera and lens configurations.Consequently, much redundancy may be present, which for larger datasets can equate to increased processing times, requiring sorting to choose the most appropriate images to texture the model triangles.Although automated texture mapping measures are available, they do not necessarily provide the best textured model when using all available photos.With an increasing number of images used for texturing, visual artefacts on seam lines between image borders, caused by small registration errors between images and the model, may occur.Therefore it is usual to limit the number of images in the texture mapping process and keep only the most suitable texture candidates (Haala, 2004).So far, this has been realized by manual image selection that can be particularly challenging and time-consuming for large models covering complex topography.In such cases the user must account for occlusions and ensure that both the horizontal and vertical image overlap is sufficient to cover all model triangles.Often, most of the data processing, including image selection, is performed by specialists from outside of the geomatics domain, who often lack the technical background to make the required decisions.A missed image, or poor orientation with respect to the 3D model, will require a new sorting and texture mapping to be carried out.For large and complicated models, consisting of millions of triangles and covered by hundreds of images, the cost in processing time of rerunning the procedure may be high.This paper presents a framework for visualization and analysis of the geometric relations between triangles of the terrain model and covering image sets (Figure 1).The application was developed with three main aims: 1) to provide decision support for the selection of an image subset optimized for 3D model texturing, for non-specialists; 2) to provide the user with predicted texture quality and completeness measures before running time-consuming texture mapping algorithms; 3) to improve the communication of geometrical dependencies between model triangles and the available digital images, through the use of static and interactive information visualization methods.

RELATED WORK
Texture mapping is a method for combining surface geometry with colour/intensity or image data.It was introduced in 1974 (Catmull, 1974) and quickly became an important research area in computer graphics (Heckbert, 1986).In 1991 Busch automatically textured surface models from multiple images (Busch, 1991).He considered the highest texture resolution as the control parameter in texture mapping.Niem and Brioszio (1995) presented an extension of this technique, based on grouping adjacent triangles so that they could be textured with a common image.Further texture mapping algorithms exist, such as view-dependant texturing (Debevec et al., 1996) or texture colour blending (Poulin, 1998).
Optimization of the texture quality is mostly addressed in papers presenting advances in texturing of 3D city models, and is restricted to refinement of the texture mapping algorithms.Recently Iwaszczuk and Stilla (2010) developed a quality measure for texture extracted from airborne thermal infrared image sequences, for rapid identification of buildings with high heat loss.In order to increase reliability of the results the authors introduced a weighted quality measure based on the polygon occlusion, texture resolution and viewing angle between the polygon normal and the camera viewing ray to the polygon centroid, in a similar manner to Frueh et al. (2004).
Selecting images relevant to texturing in highly redundant datasets has been addressed in several studies.Bénitez and Baillard (2009) described a procedure for automatically selecting façade images for texturing 3D city models, from sequences captured using a mobile mapping system.The reported result of data capture is thousands of images, of which every image is potentially relevant to texturing.In order to limit the number of candidate images, three methods for determining occlusions were employed: 3D Z-Buffering, 2D and 3D ray tracing.The authors focused on the method comparison rather than the texture quality criteria.Haala (2004) reported the need for image sorting algorithms for optimized city model texturing.In order to reduce the workload related to image registration and colour balancing, that author proposed the use of panoramic images that cover a much wider field of view than conventional frame photography.
Most related work has focused on the optimization of texture mapping procedures rather than on the quality assessment of textured models.An approach to estimate and visualize façade texture quality is presented in Lorenz and Doellner (2006).In their study, aerial imagery was projected onto a 3D city model.The authors reported that texturing the city model with quality maps facilitated determination of façade coverage from different images.The quality map encoded the actual effective spatial texture resolution.However, to date, the use of visualization methods to help users to assess texture quality is rarely reported.

AN INTERACTIVE TOOL FOR ANALYSIS AND OPTIMIZATON OF TEXTURE PARAMETERS
In the proposed framework for analysis and optimization of texture parameters, combined analytical and visualization methods were employed in order to provide the human operator with assistance in the image selection process.The new decision support tool provided objective analytical indicators about the predicted resulting texture quality, as well as visual aids in the form of diverse quality maps, tables and plots.In the spatial domain, analytical quality indicators, expressed purely numerically, may be insufficient to convey the necessary decision making information to users (MacEachren and Ganter, 1990).However, numerical information can be supplemented with visualizations of quality measures, such as colour-coded error maps, in order to facilitate understanding of complex spatial problems (MacEachren and Ganter, 1990).
Consequently, the user's decision making is additionally supported in the developed software using several forms of visual data quality representation, in linked views, and by using static and interactive visualization methods.
The application was implemented in C++ and makes use of the OpenSceneGraph (Wang and Qian, 2010) library for 3D scene handling, and Qt4 (Qt4, 2011) for graphical user interface control.In the first step of the analysis, a triangulated 3D model together with a list of images and their interior and exterior orientation are supplied.A visibility (line-plane intersection) test, based on geometrical dependencies between the model triangles and images, is performed directly after data loading.
For each model triangle, the images are identified that contain all three triangle vertices using perspective projection and image orientation parameters.Only those images with coverage of the 3D model are kept for further analysis.Additionally, the user is notified (Figure 1, log window) about the number of triangles that are not covered by any image.

Quality measures
In order to help the user decide which images are most suitable for texturing, objective image quality indicators, often used by texture mapping algorithms, were defined: camera-triangle viewing angle, and effective resolution of the triangle in the image, expressed by a scale parameter.These two quality measures are computed for all triangles in every covering image.
The camera-triangle viewing angle is the angle between the viewing ray from the camera centre to the triangle centroid and the normal vector of the triangle.The closer to orthogonal the texture image, the more suitable the image is for texturing the triangle.Therefore smaller viewing angle values are favourable.
The scale control parameter represents the effective resolution of the triangle in the image, taking into account image resolution, camera lens and the distance from the camera to the outcrop.Because the value range of the scale differs significantly between triangles, the parameter is normalized per triangle so that the image with the highest effective resolution (best) gets a value equal to one.A decrease in the effective triangle resolution in the image is proportional to an increase in scale: i.e. a scale equal to two denotes a twice worse effective triangle resolution than in the best available image.Parameter normalisation eases comparison and makes the scale parameter more comprehensive for the user.To facilitate understanding of the spatial quality distribution, the 3D model is textured with a map of predicted viewing angle and scale values (Figure 3).This visualization enables the user to locate spots in the 3D model with insufficient texture quality prior to texture mapping.These areas can then be refined by extending the image subset so that re-texturing is avoided.An alternative way of assessing the theoretical quality of the selected image subset is provided to the user in histogram form.

Texture completeness control.
Texture completeness control is a very important part of the quality assurance, as missing texture is one of the most visually disturbing factors in textured The known image exterior orientation is used to assess texture completeness and visualise image set coverage in the application.The user can check how many triangles will not be textured with the current image subset (areas that will not be textured are highlighted), and the percentage of the total area they account for is reported.The user can then verify if the size and location of the areas without texture are significant and acceptable, prior to running the texture mapping algorithm.
Allowing for a relatively small part of the model without texture may significantly decrease the number of images needed for texturing (Figure 4).A secondary, purely visual assessment of texture completeness can be performed by displaying image ground coverage for one or multiple images, highlighting the relevant triangles of the 3D model (see Figure 1, top left).

Computer assisted image selection
In order to quickly provide the user with image subsets potentially suitable to texture a model, automated image selection procedures have been implemented.The resulting image set may be used as the first approximation of the final solution.The user can control the coverage of the automatically selected images by defining the maximum allowed percentage of the model without texture.However, the spatial distribution of untextured triangles is not controlled, and it can be undesirable if triangles with no texture are concentrated in one large area.Furthermore, the selection algorithms are based on a purely analytical approach, using average quality indicators, and thus they do not control the quality of the texture locally.Therefore the image subset resulting from automated selection procedures should always be verified using the visual assessment tools, and, if necessary, adjusted manually.
The implemented image selection algorithms are based on a greedy algorithm for the set-cover problem (Chvatal, 1979).
The basic algorithm, minimizing the number of used images, searches for the highest quality image that covers the largest model area.The part of the model seen from the identified image is discarded and the next image is selected using the same criteria.Two variations of the above described algorithm were additionally implemented: maximizing quality (MQ) and optimizing quality (OQ).In the MQ algorithm, images with lowest (best) image score are prioritized in selection.The OQ algorithm combines the two previous approaches to deliver a relatively small set of images with lower (better) image score.
The right balance should be kept between the number of images and the potential loss of texture quality when selecting image subsets for 3D model texturing.The user can run a series of automated image selection simulations with different definitions of the maximum allowable untextured area.A plot of the predicted quality measures resulting from the simulation can provide an insight into the possible texturing scenarios.Figure 4 shows results of 11 runs of automatic image selection with allowable untextured area changing from 0 to 10%, plotted in the application against the number of images in the set.

Background and datasets
One area where photorealistic 3D models are widely used is the 3D modelling of geological outcrops.Outcrops (Figure 5) are exposed cliff sections or quarries, which are used as analogues for subsurface hydrocarbon reservoirs.Analysis, measurement and interpretation of high-accuracy and high resolution 3D models of geological outcrops provides statistical data that helps improve the understanding of geometrical relationships between geological features, and are used, for example, in the oil and gas industry for populating stochastic models of subsurface reservoirs (e.g.Bellian et al., 2005;Pringle et al., 2006;Buckley et al., 2008a).
Figure 5.A photorealistic 3D model of a geological outcrop.
Model textured using images selected by the automatic OQ algorithm with 3% of allowable untextured area.
The workflow for collecting and using digital outcrop data has been developed and successfully applied for geological outcrops in recent years (Enge et al., 2007;Buckley et al., 2008a;Buckley et al., 2008b).Data can be captured using terrestrial or airborne laser scanning, though the principles of 3D model creation are similar.Both dataset types were successfully used for testing the texture optimization framework, and airborne data were used in order to obtain anecdotal evidence in a user assessment exercise.A Riegl LMS-Z420i scanner with a calibrated Nikon D200 (10 megapixel) camera mounted on top was employed to collect terrestrial data (Figure 3) in a quarry in Cantabria, northern Spain.A dense point cloud and 828 images were collected from 14 scanner positions.
The dataset used by domain specialists in geology (see Section 4.2) was obtained using the Helimap system (Vallet, 2007) mounted obliquely on a helicopter platform.The system combined a Riegl LMS Q240i-60 airborne laser scanner and a Hasselblad H1 22 megapixel camera.Direct sensor orientation was used to reconstruct the 3D model (estimated mapping accuracy <0.15m; Vallet, 2007) and retrieve image orientation parameters.The dataset covered an outcrop section of the Book Cliffs, near to Green River, Utah, USA and consisted of 826 images, their orientation parameters and three relatively small model sections, for which only 50, 70 and 71 images were relevant.
In order to facilitate handling of the data, it is common to divide larger models into smaller sections (<2 million triangles) before texture mapping.As a result, images relevant to each model section need to be identified from all the photos collected in the project.In addition to the previously mentioned reasons to decrease the number of images used for texturing, in the case of geological outcrop models manual image colour adjustment may be also needed prior to texturing.This process can become very time-consuming considering the large size of digital images (10MB to 150MB for uncompressed files).
As the developed framework was capable to analyse texture quality of 3D models with up to 250 000 triangles (32-bit machine with 4 GB of RAM), a lower level of model detail was used instead of the full resolution models.

Computer-aided image selection
To evaluate the usefulness of the presented framework to domain specialists in geology, three users experienced with the manual image selection workflow were asked to use the newly developed tool and give their feedback.Considering the limited availability of experts working with processing of digital outcrop models, conducting a full user study was not possible at this stage.The users were supplied with a seven-minute introductory movie explaining the basic knowledge of how to perform a computer-aided image selection and the meaning of image viewing angle and scale as quality indicators.At first they were asked to perform the image selection using their usual manual workflow.Once this task was accomplished the three model sections were swapped between the users and the computer-aided image selection was conducted.As previously mentioned, it was required to identify all the images relevant to the provided model section in the first step.Among these images a subset of the images optimal for 3D model texturing was selected.

Anecdotal evidence and discussion
Both procedures were accomplished in shorter time when using the computer-aided image selection method.Pre-selection of the images relevant for the analysed model section (50, 70 and 71) was performed automatically by the presented framework (see Section 3) and therefore was less time-consuming (20-35s) than the manual procedures (3-6min).Furthermore, several images were missed (3 out of 50, 8 out of 70 and 7 out of 71) in the manual image selection process.The overall time spent on the two tasks was decreased from 6-16 minutes in the case of manual image selection to 1-2 minutes when using the presented application.The values of several quality measures (also from internal testing performed by the authors) indicate that the automated image selection procedure may be an alternative to manual image selection.In fact the users were very keen on using the automated selection algorithms, but did not prioritise manual image set refinement, even if they were instructed to do so in the introductory video.Therefore, the algorithm performance should be tested with further datasets and compared with more manually selected image subsets.The geologists were also asked for their comments on usability, functionality and performance of the software.The general feedback was very positive and indicated that the users felt more comfortable and secure in taking decisions using the computer-assisted method.They underlined the time gain and declared that they would prefer to use the software next time they need to texture geological 3D models.

CONCLUSIONS AND FUTURE WORK
Visualization methods successfully facilitate analysis in many science fields and have played a central role in communication of dependencies in spatial datasets.In this paper a framework for visualization and analysis of predicted texture quality measures in 3D photorealistic models was presented.The new method was used to provide computer assistance in selection of image subsets optimized for texturing of geological outcrop models.Employing different visualization methods during analysis offered enhanced understanding of geometrical relationships between the triangular model and covering image sets.Supplementing analytical predicted texture quality parameters with spatial quality maps made understanding of local texture quality changes easier for users.Anecdotal evidence obtained from the user's feedback indicated that an increased confidence in the final textured model quality and completeness, and shorter time spent on image selection make the framework highly beneficial.Enabling the user to identify regions with low predicted quality allowed the image set to be refined prior to starting time-consuming texture mapping procedures.Automated image selection algorithms, complemented with a manual subset refinement offer a promising alternative to the manual methods.More work is required to determine the most reliable and effective selection algorithm.Obtaining more feedback from users is desirable in order to assess and improve the visualization effectiveness.

Figure 1 .
Figure 1.Overview of framework for analysis and optimization of texture parameters.3D scene display (top left): an image footprint (orange outline) with the corresponding image ground coverage (light blue) and a 3D model coloured with texture viewing angle quality map.Log window (bottom left) provides numerical information.2D image display (bottom right): map of the scale parameter values overlaid on the image thumbnail.Histograms of texture quality measures (top right).
suitability for texturing.Suitability of an image for texturing is communicated to the user numerically by average viewing angle and scale parameters.Additionally an image score is computed as the average product of the viewing angle and scale, in order to provide the user with one single quality indicator.The smaller the image score, the more suitable the image is for texturing purposes.These quality measures express global image quality, because they are based on average values of parameters.An image with high score might still be necessary in the final image subset, because part of it is of high quality, and the user can choose to keep it in the final set.Therefore, to supplement the average quality indicators for an image with indicators of local data quality, a colour coded map of the viewing angle and scale computed for all triangles seen by the image can be displayed.This colour map can be overlaid on the original image thumbnail (Figure2), aiding spatial interpretation and enabling the user to quickly understand which parts of the image are most suitable for texturing.

Figure 2 .
Figure 2. Image overlaid with maps of quality measures classified into three categories: high/good (green), medium (orange) and low/bad (red).3.1.2.Predicted quality indicators of textured model.In order to help the user estimate the quality of the model textured with an image subset, several indicators are computed for the solution, using the same defined geometric criteria that are applied during the later texture mapping.The average viewing angle and average scale of the textured model are complemented by two additional indicators, angle error and solution error.The latter two are computed with respect to the hypothetically best achievable texture quality.Theoretically the best possible solution is realized when a 3D model is textured with all the images (solution error and angle error equal to zero).Angle error expresses discrepancies of viewing angle compared to the best possible solution.Solution error is computed similarly but using viewing angle and scale.These indicators are useful for comparing image subsets with the same number of images; however, they do not address problems Distributions of viewing angle and scale are plotted and overlaid with reference values representing the best achievable geometrical solution, where all relevant images are used for texturing (see Figure 1, top right).As for the angle error and solution error numerical indicators, these reference plots do not account for registration errors, and should be used as purely theoretical quality indicators to compare the quality of different, similarly sized, image subsets.

Figure 4 .
Figure 4. Predicted analytical texture quality measures depending on the number of selected images.