Web-based Visualization and Query of semantically segmented multiresolution 3D Models in the Field of Cultural Heritage

Commission for Archaeology of NonEuropean Cultures (KAAK), Bonn German Archaeological Institute (DAI) Podbielskiallee 69-71 14195 Berlin Affiliated Partner: 3D Optical Metrology 3DOM Bruno Kessler Foundation (FBK) Trento, Italy The MayaArch3D project aims to realize a web-based archaeological research platform bringing spatial and non-spatial databases together and providing visualization and analysis tools. Especially the 3D components of the platform use hierarchical segmentation concepts to structure the data and to perform queries on semantic entities.


INTRODUCTION
In the realm of Cultural Heritage documentation, visualization and analysis important sites and monuments are studied over time by different means of technical equipment, methods and intentions by different researchers.This generally results in heterogeneous datasets spread around different research institutions, stored and archived in different media types and formats with heterogeneous data models and quality.These general characteristics also apply for the UNESCO World Heritage Site of Copán (Honduras), which already has been studied for over 100 years.It is an important historical place where rich written sources have been found along with impressive temple structures and stone carvings.This archaeological site provides unique insights into its history of construction which can be linked over several centuries to the local ruling dynasty of at least 16 rulers.Since the 19th century a lot of archaeological finds like sculptures, inscriptions and ceramics, but also scientific data have been found and recorded at different locations and archives all over the world.In recent years the creation of 3D models in cultural heritage has become more and more popular.This is due to several reasons: a) increasing availability of capturing technology, specifically designed for objects of different scales, complexity in shape and surface and different materials b) increasing capabilities of hardware and software to process, store and visualize models, c) increasing demand for different use cases, e.g.conservation, restoration, education, 3D GIS analysis (Pavlidis et al. 2007;Manferdini and Remondino 2012).In addition to the creation of reality-based 3D models of existing objects, there is often the need to create reconstructed versions of objects using CAD programs or other 3D modelling software to test archaeological or historical hypothesis and to present the results to the public.The diverse possibilities of 3D model creation in cultural heritage results in collections of very heterogeneous types of 3D data for a single cultural site with different resolutions, accuracy, surface properties, coordinate systems etc.For the resource management, research and public-educational outreach of large heritage sites it is crucial to organize and facilitate the access to 3D models and the corresponding meta-, media-and attribute data.As the amount of data grows and the data are distributed at different archives and databases, the usual way is to tie all information sources together and grant access to them through web-based solutions.In this scenario 3D models can be used as an intuitive graphical interface to access the data behind.A segmentation of the 3D models, no matter being reality-based or reconstructed, into semantic entities and subentities according to different aims is an important step in preparing and organizing the data to connect geometry with semantic information using virtual reality as an intuitive communication vehicle.The following introductive sections describe the frames determining the conditions in which our approach applies semantically segmented models, the greater thematic research frame and the technical circumstances that have to be considered.Section 2 will describe in detail the process of visualization and query and all the preconditions for storing, transmitting, interacting and querying such kind of data.The paper ends with a short conclusion and outlook of the upcoming fields of research.

MayaArch3D research platform
The MayaArch3D project -a collaboration between the German Archaeological Institute (DAI), the Chair of GIScience Heidelberg University and the 3D Optical Metrology Research Group of the Bruno Kessler Foundation (FBK) Trento, funded by the German Ministry of Education and Research (BMBF)is currently working on a web-based research platform for the Maya archaeology of Copán (Billen et al., 2013).It will combine 2D and 3D georeferenced datasets together with ungeoreferenced datasets and tie these to further textual and graphical data stored in the non-spatial iDAI.Field database of the DAI, which contains mainly archaeological records and associated media like images and drawings based on FileMakerPro.It will further combine airborne and terrestrial laser scanner data with photogrammetrically derived and CADreconstructed 3D models of different resolutions.All 2D and 3D geometric information will be stored in a spatial database (based on PostgreSQL/PostGIS), while all semantic information will be kept inside the yet existing archaeological database iDAI.Field.The platform will connect both databases, spatial and non-spatial, to a group of mostly standardized web services to form a spatial data infrastructure (SDI) for archaeological research.These services include 2D/3D portrayal services (OGC WMS, W3DS), 2D/3D feature services (OGC WFS, 3D-PostGIS-Geometry-Service, FileMakerPro-Attribute-Service), a raster service (OGC WCS), a Time-Service to convert Gregorian Calendar dates to Maya Long Count dates and vice versa and an Lightweight Directory Access Protocol Service (LDAP) as security service to provide different levels of access to the data.Having all these web services at hand, a web-based Geographic Information System (GIS) is built to access, visualize and analyze the collected data in a spatio-temporal way over the World Wide Web.The client application therefore builds on top of the open-source geomajas2 2D web GIS framework and is complemented by a spatially enabled 3D framework GIScene developed by GIScience Heidelberg University using WebGL technology (Khronos Group, 2013) and the open-source Three.js3JavaScript library.For the 3D part of the application two components will be integrated.First a georeferenced 3D scene is provided to explore landscape, settlement, building and other structural features.Because of the georeferenced coordinate system in this scene, it becomes possible to combine it with other GIS related data e.g.overlays from raster files or map services or user defined vector data, thus providing individual analysis possibilities for researchers from different domains with different research questions and approaches.Besides the scene viewer for georeferenced 3D data, a single object viewer is provided to study objects in higher resolutions, providing different modes for georeferenced and nongeoreferenced models.Both 3D components support the user's interaction with models segmented into a hierarchy of semantic entities and provide the necessary user interface to access attribute information from databases on all semantic segmentation levels and groupings.Segmented models can be used to differentiate parts of objects with different meanings and as interface to either access models of higher resolutions or other information related to that specific semantic entity (Manferdini and Remondino, 2012).
Three.js is an open-source JavaScript API for 3D visualization using WebGL, http://threejs.org/With this infrastructure, the extent of data that can be visualized and analyzed reaches from the Mesoamerican distribution of archaeological Maya sites to small 3-dimensional laser-scanned stone objects of a couple of centimetres found during excavations of temples and other structures.

Constraints and potential of web-based 3D visualization using WebGL
Using WebGL for the visualization of 3D models has several implications.One the one hand, it offers the integration of 3D content into web applications without any dependencies of third parties.No plugins or Java applets have to be installed by the end-user and all major browsers4 have implemented at least a partial support.Further in the scope of 3D GIS analysis the usage of a WebGL based technology has the advantage over plugin-based solutions, in that there is a direct access to the rendering pipeline of the graphics card, the so called programmable pipeline of OpenGL ES 2.0 (Khronos Group, 2010).Instead of simply using predefined rendering shaders, e.g.Gouraud or Phong shading, developers can define their own functions to be executed very fast in parallel on the graphics processing unit (GPU) (Auer, 2012).This offers a great potential for new web-based 3D GIS analysis.First results have been achieved in the project visualizing orientation or alignment of surfaces towards compass directions or point locations.
On the other hand, a WebGL application completely relies on the JavaScript engine the browsers uses to interpret the application code.In contrast to desktop applications, browser applications written in JavaScript have no direct control over the memory management.Dealing with large 3D models of heritage sites implies that a careful selected strategy to load and unload 3D model data has to be applied.This can only be reached when the model data can be loaded incrementally as parts and in different resolutions depending on the visible field, observer distance and viewing angle.The goal here is to visualize only those parts in high resolutions that are near and visible to the user and to reduce the amount of data for the rest of the scene.With the multi-resolution and segmentation approach presented in this paper such a strategy can be realized.
A test about memory consumption using Three.jsrevealed that it is suitable to show models or scenes with up to 1.000.000triangles at a consumption rate of about 1 GB of RAM, whereas point cloud representations are far less memory intensive (Fig. 1).
Figure 1.Memory consumption visualizing large 3D models using Three.js

RELATED WORK
In terms of a 3D platform to query semantic information with the help of segmented models, a previous project developed a tool named QueryArch3D (Agugiaro et al. 2011a,b;von Schwerin et al. 2013).In contrast to the current project, the predecessor was based on a plugin solution from unity3d 5 and exclusively dealt with 3D data.
Combining 2D and 3D georeferenced data in a single web GIS application has also been realized by a project called 3D Sutras but without using segmented 3D models as user interface for data queries.On the other hand, that project used the W3DS as well and other OGC standardized web services to deliver 2D and 3D data and similar to this project integrated a non-spatial database already existing with the newly created spatial database (Auer et al. 2011 , 2013) or CityGML, an OGC standard for modelling 3D city models (Gröger, 2012).The approach presented in this paper is reflecting those principles, but specifically adopt the necessary object entities to the concepts of Maya architecture and arts.Manferdini and Remondino (2012) give an overview of segmentation methods, purposes of semantic segmentation and existing hierarchical modelling concepts from other domains which relate geometry parts to semantic information.Unfortunately, the existing concepts are not always suitable for an application in cultural heritage because of the variety of structures found in ancient cultures like in the Maya architecture and art, which often is mixed with textual information in form of relief glyphs.

VISUALIZATION AND QUERY OF SEMANTICALLY SEGMENTED 3D MODELS
The following sections describe our approach to store, transmit and interactively query semantically segmented 3D models in a web application.In the case of the MayaArch3D project the segmentation process is done manually by Maya researchers.The entities to which the different kinds of objects are segmented have been defined beforehand by agreeing on a hierarchical system of semantic object classes and several levels of subclasses.These well-defined ontologies ensure that the segmentation process is transparent, replicable and reusable for future acquisitions and incorporation of Maya structures into the research platform presented in this paper.

Storing semantically segmented models
For the MayaArch3D project an approach to store and integrate both the geometric and the thematic information has been studied and implemented, given the several constraints and requirements to be taken into account during the development of the database structure.The goal was to overcome interoperability and performance issues between PostgreSQL and FileMakerPro (the latter already containing all attributes of 5 https://unity3d.com/unity the Copan features).As a consequence, following design decisions were taken: 1. PostgreSQL (together with its spatial extension PostGIS 2.0) stores only geometries, textures and the very minimum amount of data to allow the connection to the corresponding FileMakerPro records (essentially: foreign IDs).This split design corresponds to an initial requirement to keep the FileMakerPro in use as source of the attribute data.Each object stored in PostgreSQL is linked to the corresponding FileMakerPro record containing the attributes and metadata.The spatial extension PostGIS 2.0 has been chosen as, besides the well-proven capabilities in 2D geometry, new 3D object types with corresponding 3D operations have been added.They can directly be used during the SQL queries.
For a better reading, in the following the PostgreSQL/PostGIS-based database will be called 3dmaya-db, while the FileMakerPro-based database will be referred to iDAI.Field.The former is hosted at the Heidelberg University, the latter by the German Archaeological Institute (DAI) in Cologne.iDAI.Field applies a thoroughly developed database schema, which was tested and enhanced in several archaeological projects.2. Particular attention was paid to create an ontology for the objects to be stored in 3dmaya-db (temples, palaces, etc.), as well as the accompanying hierarchies for all "part-of"-relations.The goal was to ensure a spatiosemantic coherency which allows to define what exactly an object is and how it relates to the other objects (e.g. a wall to a room).3.As a consequence, it is possible to store models both as a whole, or their segments.Moreover, for the same object, multiple representations are possible.For example, one can store a model as a 2D feature, its extruded 3D model, or other fully 3D models obtained by means of laser scanning or photogrammetry (either as mesh or point cloud).4. Despite this initial "split" configuration, particular care has been put into the design of the database structure to obey to the rule of simplicity, avoiding unnecessary data redundancy, and allowing for enough flexibility to be extended in future in case of special or new needs.The separate storing of geometry and attributes make it easy to adopt the system to use other attribute databases in case of changes due to technology advancements or new project requirements.For example, should one day iDAI.Field be dropped completely, it will not be hard to integrate the existing attributes into the existing 3dmaya-db structures or connect to another database.
In order to cope with the complexity of multiple geometric models, which are required to reflect independent data collection processes, multiple levels of details (LoDs) were defined.LoDs facilitate efficient visualization and data analysis.
In analogy to the CityGML approach (Kolbe, 2009), and drawing from previous experiences with Copán data (Agugiaro, 2011b), five levels of detail were defined for the structures of Copán: the higher the LoD rank is, the more detailed and accurate the model is.
• LoD0 contains simplified 2D polygon (or multipolygon) entities.This LoD is thought to allow for a classical 2D GIS-based representation.Original data were provided as a shapefile (Richards-Rissetto, 2010).These data are not used in the 3D visualization tools.• LoD1 contains simplified 3D prismatic entities with flat roofs.More specifically, all LoD1 models were obtained starting from the LoD0 data by means of extrusion and triangulation of the features.• LoD2 contains 3D structures at a higher level of detail, however only the exteriors of the structures.The substructures (e.g.walls, roofs or external stairs) can be identified.For the LoD2, hypothetical reconstructions models obtained for example in 3D Studio Max can be imported and stored.The upper limit in terms of triangles is set to be 150.000 in an area of 1024 m².• LoD3 adds the interior elements (rooms, corridors, etc.) to the structures.Some simplified, reality-based model parts can be optionally added, both to the interior and to the exterior of the structures.The reality-based models can be obtained from the more detailed ones used in LoD4 by applying mesh simplification algorithms.For each feature, the upper limit in terms of triangles is set to be 100.000.• LoD4 contains structures (or parts of them) as highresolution (e.g.laser-scanner-acquired) models.These models can be further segmented into subparts.For each feature, the upper limit in terms of triangles is set to be 300.000.
The limits of triangles chosen in the above LoD concept reflect several constraints, such as the amount of data to guarantee a high rendering performance, the memory management in JavaScript (see section 1.2) but also the application design.The application design allows users to open several windows with 3D content, so the limits for each LoD is not what theoretically could be displayed but rather a practical estimation to keep the web application stable in different usage scenarios.The adoption of a LoD-dependent hierarchical schema required the contextual definition of a geometric and semantic hierarchical schema.This was achieved by an accurate identification and description of the so-called "part-of"relations, in order to guarantee a spatio-semantic coherence (Stadler and Kolbe, 2007).
Such a database structure that reflects all the considerations mentioned above, allows at the same time the development of a user interface which enables jumping forth and back between different LoDs and other representation types, and also traversing up and down through the semantic segmentation levels using the same database structure for temples, houses, stelas, altars and other object types.
The key point of the database structure here is, on the one hand, the use of consistent IDs for all LoDs and further representations of the same object and, at the same time, to store parent relationships for all objects to reflect the position of each object inside the semantic hierarchy tree, as shown in Figure 2.

Transmission of semantically segmented models
For the transmission of the segmented models and their semantic content several web services have been deployed.The Web3DService (W3DS) is currently in the process of standardization and its specification is available as a discussion paper at the Open Geospatial Consortium (OGC)6 (Schilling and Kolbe, 2010).Though very useful for building 3D web GIS because of its capability to retrieve a collection of georeferenced 3D models by specifying a spatial bounding box and a LoD, in its current version it lacks the possibility of requesting single objects by their ID.Therefore a second web service -the Geometry Service -has been implemented to specifically demand models or model segments in a given LoD by specifying its ID (Fig. 3).It traverses the graph structure in the 3dmaya-db and allows to either request for a specific node or all children of a certain node.If a "children" request is received, all child-nodes of the given node-ID will be returned.If a specific "node" is requested, the service returns this node and its geometry, if available.The service is implemented as RESTful interface as shown below (Fig. 3).
Figure 3. RESTful URL schema of the Geometry Service Both services, the W3DS and the Geometry Service, are able to retrieve data from the 3dmaya-db and convert them to a suitable format for delivery.Generally, for the exchange of hierarchically segmented models all formats that are capable to represent a scenegraph could be used.In this specific case, the native scene format of Three.js is used.It is a JSON7 based text format, which specifies the parent-children relationships of the scene, its materials and it can either contain inline geometry definitions or references to other sources also stored in different formats.This approach grants a great flexibility for the future in case other formats for single objects in the scenegraph would prove more successful.
A third service can be used to gather attribute information on the geometries fetched earlier from one of the two 3D services.This Attribute Service is defining a RESTful interface to retrieve data by IDs which are coherently attached to the geometries.To achieve short download and response times in the web application it is of great importance to reduce the amount of data to be transferred.A compression of data to be transferred over the web always is a trade-off between the time needed for the compression and the time for the transfer without compression.Large polygonal 3D models with 300.000 triangles have a size of about 30MB as JSON text.Depending on the network speed, it can take up to several minutes to load.
A very fast and on-the-fly applicable compression algorithm is the deflate-algorithm (Network Working Group, 1996).In several tests with different models in the initial project phase it achieved a compression rate of about 25%.One big advantage of this compression method is the capability of all major browsers to automatically decompress files that are indicated as compressed by their http headers.This means no further processing in JavaScript is necessary to receive the original uncompressed content.

Interaction and query user interface for semantically segmented models
After the automatic decompression of the incoming scene file from the services by the web browser, the resulting JSON text can directly be converted into JavaScript objects.Nevertheless it is necessary to process this information to: 1. Generate an internal JavaScript representation of the delivered scenegraph 2. Generate all the necessary data structures (geometry, textures, texture coordinates, colors, normals etc.) and send them via the WebGL API to the graphics card for visualization 3. When dealing with georeferenced 3D models whose coordinates contain big numbers apply a shifting transformation to obtain short coordinates.This is necessary to avoid jitter effects in the display because of limited precision of number representations on the GPU 4. Retrieve all leaf nodes of the scenegraph (nodes which contain geometry) to make them selectable for interactive information retrieval Once the data structures inside JavaScript and the GPU have been created, it is necessary to provide a user interface that allows the user to easily explore the data and query the respective attributes from the database via the Attribute Service.The user interface consists of two components: 1) the canvas which depicts the 3D scene 2) an additional dialog on the sidebar to choose the segmentation level.
In a first step the user selects a model segment in the 3D canvas which will be highlighted, e.g. a glyph.Directly after this selection, the second dialog will be generated.Therefore it is necessary to determine all parent nodes of the selected segment and their IDs.A process traverses up the internal scenegraph representation of the segmented model from the clicked segment up to the root node and collects a list of parent entities.This list can then be used to generically create the dialog.
Hovering over the resulting list with the segmentation branch the user can select the semantic segmentation level he is interested in, e.g. the stela side, to perform a database query via the Attribute Service and the given ID. Figure 4 illustrates the process of clicking one of the objects of the highest segmentation level and the interactive highlighting of all possible parent groupings which then can be selected to retrieve further information.For an effective browsing of the data only a few characteristic attributes are shown directly inside the sidebar dialog.Several of such short attribute lists can be retrieved and compared.Then in a second step the user can decide to display a complete list of attributes for a segment in an additional floating window.This way an effective and rapid browsing of the data can be achieved to facilitate the creation of new hypothesis about relations in the data or getting quick access to the data of a specific semantic piece.

CONCLUSION AND OUTLOOK
The work presented in this article is an approach for the visualization and query of semantically segmented models of multiple resolutions.It outlines the thematic and technological frames in which this approach is successfully applied.It describes in detail the conceptual considerations and technical challenges that led to the presented solutions of storage, transmission, interaction and query of the specific data structures of semantically segmented models in the field cultural heritage.
The proposed infrastructure is highly flexible and compatible with other cultural heritage sites as it separates between the two-and three-dimensional geometry data and attribute data connected to the 3D models.Two 3D components have been developed: a) a georeferenced 3D scene allowing the exploration of landscape, settlement, building and additional structural features and b) a single object viewer allowing the study of georeferenced or non-georeferenced objects in high resolutions.These tools give researchers of various domains in the field of cultural heritage the possibility to visualize and interact with semantically segmented 3D models and provide the necessary user interface to access attribute information from databases on all parts of a semantically segmented model.The use of WebGL technology proves to be a feasible approach.Despite its constraints in memory management, it offers a great new field of possibilities to develop 3D GIS visualization and analysis components for the World Wide Web while improving the accessibility to the growing amount of detailed and complex cultural heritage data.Future work will concentrate on the study and development of 3D-GIS analysis functionality in the area of 3D visibility and accessibility of ancient settlement structures in the scope of Maya archaeology.Further interest exists in the comparison of semantic segments of different objects located at different places to study patterns of similarity, e.g. of stone carved glyphs and motifs.

Figure 2 .
Figure 2. Database structure to store semantically segmented models in the MayaArch3D project

Figure 4 .
Figure 4. Compressed transmission of segmented models from PostGIS 2.0 database via web services to the 3D components of MayaArch3D