COLOR AND TEXTURE INTERPOLATION BETWEEN ORTHOIMAGERY AND VECTOR DATA

Graphic interfaces of geoportals allow visualizing, sometimes overlaying often by image blending, various data and representations of geographical spaces: vector data, maps, aerial imagery, Digital Terrain Model (DTM), etc. After some previous works it appears that image blending is insufficient to allow efficient co-visualization of orthoimagery and vector data. Our purpose is to be able to manage an hybrid visualization of orthoimagery and vector data, efficient and useful. The diversity of hybridization levels requires to be able to control a continuum between such data. We thus have to propose rendering methods to mix heterogeneous data, in order to propose homogeneous and continuous intermediary representations. This paper proposes a methodology to interpolate graphic parameters between an orthoimage and related vector data, to control the level of orthophotorealism, all along the continuum. We detail the methodology based on several interpolation components, to control abstraction and realism levels, by manipulating colors and textures (natural, procedural or mixed).


INTRODUCTION
Geoportals and geovisualization tools provide various geographical data, i.e. vector data, maps, aerial imagery, Digital Terrain Model (DTM), immersive views, etc., and various ways to covisualize them.We define the co-visualization of data as the simultaneous visualization of two or more various data or graphic representations, together, in a same visualization interface.Figure 1 presents three examples of co-visualizations of various data based on various types of interface tools: integration of historical photographs in Google Street View 1 , swap between satellite image and historical map in a lens 2 and finally, swipe between a satellite image and a related raster representing temperature estimation 3 .
Figure 1: Co-visualizations of geographical data Nevertheless, parallel visualization is a current trend, but implies some difficulties to maintain the visual attention.Our purpose is thus to overlay and blend those data in a same view to facilitate the users' tasks.Therefore it requires to be able to manage the heterogeneity of data in order to visually integrate data together.Heterogeneity concerns color contrasts, visualization scales, generalization levels, geographical content or temporality, amongst others.In parallel, we consider that visualizing heterogeneous data together enhances also the specificity and the relevancy of data, by complementing each other.Users may benefit from these hybrid visualizations of heterogeneous data, while better visualizing and understanding geographical phenomena in a geographical space.
Our long-term research issue is to find the balance between the need for preprocessing data steps and the call for rendering techniques adaptation.These challenges both aim at improving map efficiency, and thus map quality.We also aim at exploring photorealistic and non-photorealistic rendering techniques in order to increase the realism of maps and to interrogate the effect of this realism on map quality.Orthoimagery provides an idea of realism through a direct visualization of objects of the real world, in their completeness, related to the time of capture: users actually take more and more advantage of orthoimagery in mashups, websites, cartographic and augmented reality applications (Hoarau, 2012).Vector data provide a symbolized geometric representation of objects of the real world, associated to attributes, as a result of many abstraction processes.Hybrid visualizations of orthoimages and vector data would be an efficient way to add (ortho)photorealism to abstract data.Plus, the diversity of possibilities to manage this level of realism encourages us to provide an interactive continuum of intermediary visualizations between an orthoimage and related vector data.Our purpose is to preserve a homogeneous visualization all along the continuum, regarding the level of realism (Hoarau, 2015).
In such a context, we face the difficulty to find relevant graphics parameters and rendering methods to propose a continuous transition between an orthoimage and vector data, while controlling the level of (ortho)photorealism.Therefore, we suggest to interpolate color and texture between an orthoimage and related vector data.
After reviewing related work in Section 2, we detail the components of our methodology to interpolate colors and textures be-tween orthoimagery and vector data in Section 3. We present and discuss the result of the application of our methodology on a dataset to design a continuum between an orthoimage and its related vector data in Section 4. Further work is addressed in Section 5. Medley and Haddad (2011) present the notion of continuum as 'a series of pictures, iteratively reduced in representation from its referent', based on photographs, drawings, sketches, etc. Authors notice that 'images of greater realism help to solve the homogeneity problem: distinguishing objects in the same class (, whereas) images of reduced or distilled detail facilitate object hypotheses: distinguishing between classes of objects.'.Providing a style continuum between an orthoimage and vector data, based on photorealism, is a research problem related to four types of research works: first, the design of methods to adapt color contrasts, second, the user perception of realism in various types of maps based on heterogeneous data, third, the way to add photo or non-photorealism in visualizations, and fourth, the interaction methods to control realism/abstraction levels.

Adapting the color contrasts
Hoarau (2012) demonstrates that using image blending to overlay orthoimagery and vector layers is insufficient to allow efficient co-visualization, when layers are both transparent.Overlaying orthoimagery and vector data is mainly addressed as an issue of by-default colors mixing, requiring to handle the resulting color contrasts.Raposo and Brewer (2013) propose a global symbolization method for vector data, based on colored outlines and dotted lines, adapted to any backgrounds map or imagery.This solution is likely to make the concerned vector data, e.g.roads in general, very salient in order to be always legible regarding to the background.Hoarau et al. (2013) propose an automated adaptive method to symbolize each edge of roads, with the complementary color of the dominant color of the orthophotography below: the variation of colors is not salient and help to preserve a suitable color contrasts all along roads edges.This method will be used at every intermediary step of the continuum to manage color contrasts well in intermediary hybrid visualizations.Nevertheless, in order to handle continuous transitions all along the continuum, colors should be also interpolated between each end of this continuum.The issue of colors of orthoimagery and vector data is thus at stake here: do they convey orthophoto-realism or not?

Perceiving realism in visualizations
The efficiency of co-visualized data, and in particular the perception of realism in maps, has been evaluated by visual experimentation.Wilkening and Fabrikant (2011) evaluate how different map types can influence people's visuo-spatial decision making, specifically for a complex slope detection task involving three spatial dimensions: as a result, the authors highlight that while people might prefer more realistic looking maps, they do not necessarily perform better with them.Bernabé-Poveda and C ¸öltekin (2014) assess that there is a correlation between a cartographic task and the realism level of the representation.They highlight a limit of the understanding of geographical phenomenon between several representations is the case of the relief perception: cartographic shading and natural shading are incompatible.Bernabé-Poveda and C ¸öltekin (2014) quantify empirically this relief inversion phenomenon on orthophotographies, according to user profiles and location on Earth.According to these results, it would be relevant to add realism, and in particular orthophoto-realism, in abstract visualizations, to benefit from its advantages while combining it with abstraction advantages.

Adding realism in visualizations
In order to take advantage of heterogeneous data, some authors exploit the user perception of realism to propose more realist maps based on photorealism or non-photorealism techniques.For instance, natural color maps are investigated by (Patterson and Kelso, 2004), relief realism is based on its enhancement by illumination in (Patterson, 2002), by watercolorization on oblique views in (Jenny et al., 2015) or by synthetic vectorial textures in (Loi et al., 2013) ; water surfaces are rendered by realist textures (Patterson, 2002), animated textures in (Yu et al., 2011), by expressive renderings in (Semmo et al., 2013).The use of textures, procedural or coming from the orthoimage should be thus relevant to manage realims in our continuum.
2.4 Interacting with data and realism/abstraction levels Semmo et al. (2012), Semmo et al. (2013) and Semmo and Döllner (2014) propose to use parameterization of rendering methods to make progressive transitions between various levels of abstraction: various strategies to distribute the level of abstraction in the representation according to the distance from the image center or the saliency of rendered objects (Semmo et al., 2012), river rendering according to more or less cartographic styles (Semmo et al., 2013), more or less complex textures according to scene depth and expected abstraction level (Semmo and Döllner, 2014).Biljecki et al. (2014) propose metrics to describe the level of detail that are used to make discrete scales of level of detail.Bektas and C ¸öltekin (2012) and Bektas et al. (2015) aim at reducing the level of detail in imagery with a technique called 'geofoveation'.Authors identify the area of interest of the user in acquiring gaze points with an eyetracker, e.g. the focus zone, and remove perceptually irrelevant detail in the visual field, e.g. the context zone: details are removed from the periphery based on a model of the foveal vision.Another way to manage the visual attention of the user is to generate masks according to relevant geographical objects and scene depth and highlight specific objects as soon as the user perceives them (Trapp et al., 2011) .
In such a context, we aim at reaching continuity and homogeneity between orthoimagery and vector data, while managing the level of orthophoto-realism.The interaction mode is proposed to be a slider to handle the continuum between an orthoimage and vector data.The problem addressed in the paper is related to the visualization design and rendering aspects.Our purpose is to interpolate graphic parameters between one end (an orthoimage) to another (related vector data), in order to control the level of photorealism, by manipulating homogeneously colors and textures (natural, procedural or mixed).

METHODOLOGY
Our methodology is based on three steps.
First, we identify relevant visual characteristics in the orthoimage, as a graphic inspiration source.We consider it conveys realism, that could be mixed with abstraction all along the continuum, based on its natural color and texture.Second, after being extracted, these characteristics are used as pivot parameters between ends' value parameters in our global interpolation method.These characteristics can be modified with the help of their related graphic parameters described in the style, for instance the color or the opacity of object fills, but also by using procedural rendering methods to add texture effects, or by blending imagery extracts.The Styled Layer Descriptor (SLD) and Symbology Encoding (SE) norms are used to manipulate the style.Third, we provide a set of interpolation methods.Each interpolation method generate interpolated values for one of the graphic characteristics of in-between representations.Those methods can be used independently and combined to create a graphic path from one representation to another, for instance orthoimagery and vector data.

Relevant visual characteristics inspired by orthoimagery
Relevant visual characteristics to be interpolated are those which have an important visual impact within cartographic visual variables: the color and the texture of the cartographic objects.We first suggest to define and extract natural color and texture from orthoimagery, and then to generate various types of textures, inspired by orthoimagery, to handle realism levels.

Color
Our color extraction method aims at extracting the main natural color for each cartographic theme coming from orthoimagery.We assume that objects of a theme have similar colors in the reality and in consequence in orthoimages.Using cartographic data allows to identify object footprints: these objects have been grouped regarding their nature and their properties.Binary masks are created from vector layers and used to cut the orthoimagery out.The color is then extracted for each mask thanks to the color classification method described in (Christophe et al., 2013).The figure 2 provides an example of the extraction of the natural color of the buildings: the natural color extracted here reflects the color of the building roofs.Bertin (1967).This visual variable consists in the repetition of a small pattern.Regarding hybrid visualization and covisualization methods, we propose an extension of this concept in order to take into account orthoimagery background layers and photorealistic rendering textures.A texture can be neutral if object are symbolized by plain colors, procedural if the texture comes from a procedural rendering algorithm, natural if the texture comes from a picture or an orthoimage of the objects, or mixed if the texture is both procedural and neutral.These four categories are illustrated in figure 3).In consequence, we can modify several graphic parameters to manipulate all types of texture.

Natural textures
The natural texture is a visual characteristic that allows increasing the realism of the representation.In order to obtain the natural texture of an orthoimage, we can modify the opacity level of the imagery, convert it in greyscale or render it with different blending modes.
Considering that a vector layer is overlaid to an orthoimage, modifying the opacity of the vector layer will make appear the natural texture of the orthoimage.Another way to convey the natural texture of the orthoimagery is to convert the orthoimagery in greyscale.This rendering method allows to manipulate independently the natural texture and colors of the orthoimagery.Finally, blending modes can be used to blend colors of overlaid vector data with the natural ones of the orthoimagery.We use the Overlay mode described by (Porter and Duff, 1984) because it preserves highlights and shadows of the background by reflecting the lightness or darkness of the backdrop color when mixing them with the overlaid color.The figure 4 illustrates these different methods to extract the natural texture of the imagery and reinject it in the map design process.Procedural textures In order to find controllable pivot parameters, we use realistic rendering methods to introduce progressively realistic characteristics of the imagery texture.Such rendering methods provide a realistic appearance to the user and enable the cartographers to manipulate independently the color, the scale, the amount, the angle of the texture.A Perlin Noise texture provided by (Perlin, 1985) has been used in this paper.In order to define two colors for the Perlin texture, we decide to choose natural color from orthoimage and color from the related vector data.The figure 5 presents the final texture obtained by mixing the plain light green of the map and the dark and complex texture of the imagery.It can now be used as a perfect controllable pivot symbolization during interpolation.The scale of the natural texture have been chosen to reflect the natural scale of the vegetation.Our relevant visual characteristics, color and texture, have thus been described in this paragraph.In order to manage them, we have now to consider their elementary design parameters.

Graphic parameters matching
The style of each end of a continuum is described as symbolizers in the SLD standard (Lupp, 2007).For vector data, each symbolizer contains several graphic parameters such as the color and the opacity of the fill and the stroke.The symbolization of raster data are described by a specific raster symbolizer.

SLD expressivity extended for texture manipulation
In order to be able to manage our different types of textures, we propose an extension of the fill description, in adding graphic parameters describing procedural texture characteristics: amount, scale, stretch, angle and colors required to describe a Perlin noise texture (Cf.figure 6).

SLD intermediate media to match continuum end styles
An interpolation method requires an intermediate media, aiming at storing all the graphic parameters required for the description of both continuum end styles.We first match graphic parameters of our two ends of the continuum, e.g. the color of their fill.All of these common graphic parameters are then added to the SLD intermediate media.Second, we add the graphic parameters that only appear in one continuum end style.These new graphic parameters are associated with an opacity parameter, in order to be able to manage their visibility along the continuum in case of non bijection between continuum end styles.The SLD intermediate media parameters have first empty values.These values are filled in by the style interpolation methods detailed in next section.These values thus change continuously from one intermediate representation to another.

Styles interpolation methods
In order to provide a style continuum evolving regularly, a linear interpolation function given by equation 1 has been used for all interpolation of graphic parameters.
(1) where f (x) is the interpolated value at the interpolation step given by x, and (xa, ya), (xa, ya) the coordinates of the end parameters.

Color interpolation
The color interpolation is done in the CIELab color space.This color space provides a perceptually uniform color space (CIE, 1976).In this color space every color can be defined by three coordinates: its lightness L and two chromatic coordinates, a between red and green, and between yellow and blue.We apply the linear interpolation function (Cf.equation 1) to these coordinates of the color in order to create a continuous color range.The perceptual consistency allows us to ensure the regularity of the perception regarding color evolution along the color range: the distance between two colors is consistent regarding the perception thanks to our color space choice and all the colors are equidistant in the range thanks to our interpolation function choice.The figure 7 shows a color range interpolating purple and green.Eighteen equidistant colors have been extracted from the range and are presented above and under the range in order to illustrate the color evolution regularity.Between two maps based on colors The first example consists in interpolating every color theme of a vector data or map.Continuum ends come from (Christophe and Hoarau, 2012).The color interpolation is done independently for each vector theme.So far, there is no design constraint aiming at guaranteeing that intermediate representations convey the same semantic relationships than ends' representations.Between two procedural textures The second example is the interpolation of colors of procedural textures.Indeed, the color does not impact the structure of noise based textures, so it can be modified without destroying the texture.Between two textures with a common color, the second color is interpolated in order to create a texture range: in the range of figure 9, the green is fixed and the second color is interpolated between white and yellow.If the two colors of the procedural textures to be interpolated are different, the interpolation is done by two steps, using a pivot symbolization created with a color of both ends' textures: in the figure 10, the first color is interpolated between light green and dark green (on the right) and the second color is interpolated between white and yellow (on the left).Between two baselayer rasters The third example aims at creating a graphic path between two possible base layers: a white baselayer and an orthoimage.The interpolation is made in two steps in order to manipulate separately the color and the natural texture of the imagery.In consequence, the imagery is first converted in greyscale and then the color of every pixel is interpolated first from its natural color to the greyscaled one and secondly from the greyscaled color to white.The figure 11 shows interpolation result samples.

Texture interpolation
As we said in paragraph 3.1.2,several graphic parameters allows to modify a texture of a symbolized object.We describe here how the opacity of a natural texture and the amount of a procedural texture can be interpolated to create graphic paths between texture and non textured representations.

By opacity
The first parameter that allows to control a natural texture is the opacity.This parameter is widely defined between 0 and 100.So, we just interpolate the value regarding the position along the continuum to create intermediate transparent styles.The figure 12 illustrates four intermediate opacity interpolations.As the representations have an imagery background, its natural texture appears more or less depending on the opacity of the overlaid vector data.
The opacity parameter can also be used to manage the appearance or disappearance of the related symbolizer, when only one end of the continuum contains it.For example, in order to interpolate the style between buildings symbolized with outlines and buildings symbolized without outlines, the opacity of the outlines is interpolated in the matched intermediate symbolization.By procedural texturing The amount parameter of a procedural texture is interpolated in order to make appear or disappear the rendered texture.Figure 13 shows a texture range where the amount is interpolated between a plain light green style and a Perlin noise texture style.
Figure 13: Amount interpolation of a Perlin noise texture

RESULTS AND DISCUSSION
This section presents a continuum of styles between orthoimagery and vector data.

Dataset
The dataset is a cartographic vector database and related orthoimages.The visualization scale of the cartographic vector database is from 1:15000 and 1/25000.We use six themes from this database: buildings, hydrography, vegetation, roads, contour lines and toponyms.The background layer is also manipulated.Orthoimages have a sufficient resolution to be visualized at similar scales as the cartographic database.

Style interpolation for one cartographic theme
We illustrate here how we can combine several graphic parameters of styles, in order to create a graphic path between the light green plain style of the vegetation on the map and the representation of the vegetation in the imagery.We use five pivot symbolizations and six symbolization interpolation components.This combination comes from a user test described in Hoarau (2015).The resulting graphic path presented in figure 14 introduces color and texture evolutions.A Perlin noise texture is used between plain color symbolizations and natural textures, as a semi-realistic symbolization of the vegetation.The transparency is also used to browse all the categories of our texture typology and make evolve the realism perception along the continuum of vegetation symbolizations.

Style interpolation for a geovisualization, theme by theme
In order to create a complete continuum between a map and an imagery, every cartographic theme is interpolated using several interpolation components.The figure 15 indicates the symbolization interpolation components and pivot symbolizations used to create our style continuum.
Roads' symbolization interpolation Three different components are used to interpolate roads' symbolizations.Each of these three components interpolates the color between the bright colors of the map and the natural grey extracted from the imagery.This combination allows controlling the abstraction coming from the administrative categorization of roads in the map.
Buildings' symbolization interpolation The abstract categorization is controlled with a similar combination of interpolation components than for the roads.Three color interpolation components are used simultaneously to interpolate between abstract colors of the map and the natural red extracted from the roofs in the imagery.Moreover, the natural texture of the roofs is manipulated independently by an opacity interpolation components.

Hydrography symbolization interpolation
The natural color and texture of the hydrography are manipulated independently, by two different interpolation components.A color interpolation component makes the hydrography evolve between the light blue of the map and the dark blue extracted from the imagery.Moreover, the natural texture of the see is manipulated independently by an opacity interpolation components.

Vegetation symbolization interpolation
The complex combination described in the previous paragraph is used to make the vegetation symbolization evolve from the map abstract one and the imagery realistic one.
Background layer interpolation Finally, the symbolization of the background layer evolves according to the process described in figure 15.
The figure 16 introduces the ends' representations of our continuum and four intermediate representations created by using the interpolation components of figure 15.

Discussion
Our method provides design control to the cartographer and enables him to create a continuous graphic path between two heterogeneous representations.Interpolating graphic parameters independently allows us to control the evolution of every theme of the representation.Indeed, the cartographer is able to choose one or several pivot symbolizations and the corresponding interpolation components.These components can be used simultaneously or consecutively.Thus, the method provides a wide range of symbolization designs to the cartographer.Our proposition between a map and an orthoimagery already used seventeen interpolation components and eight pivot symbolizations to define only five graphic paths.This complex combination shows the flexibility offered by our interpolation component-based method.
Nevertheless, the interpolation methods should be constrained to ensure the quality of each intermediate representation.Indeed, our proposition intends to allow the final user to choose the intermediate representation to be displayed.We aim at making the users able to adapt the visualization to their need and their cartographic task.We also aim at providing them a continuous transition in order to help to identify relevant objects of geographical structures from an end to the other.In consequence, the interpolation could be constrained for each intermediate representation in order to preserve the semantic relationships existing between the themes of the abstract end of our continuum.The interpolation methods could also be constrained along the continuum by previously fixing given intermediate representations, depending on the users needs and tasks.Moreover, our combination highlights the mutual impact of the different interpolation components.For example, in the half part of the continuum close to the imagery side, two opacity interpolation components are used in order to make appear the natural texture of the see and buildings' roofs.This opacity interpolation makes the background layer more or less visible.At the same time, a saturation interpolation component is used between a greyscaled imagery to a colored one.Consequently, the natural color is coming from the pivot color of the sea in the middle of the continuum, from the imagery at the imagery end of the continuum and blended from those two layers in-between.
Finally, some cartographic themes do not appear in the configuration schema of figure 15.Indeed, toponyms and contour lines symbolizations have not been interpolated or modified along the continuum.These themes convey abstract information that does not appear in the orthoimagery.As a consequence, it was not possible to extract graphic characteristics about them from the orthoimage in order to define pivot symbolizations.Moreover, these themes are relevant for the understanding of the landscape, as well as roads.Ory et al. (2015) identify these themes as salient characteristics of the topographic style.That is why it is not conceivable for us to make them disappear in the imagery by using an opacity interpolation component.We assume that such relevant themes (roads, contour lines, toponyms, etc.) can not be transparent.They have to be visible and readable or not to be.Therefore, we symbolize them with an opaque color and remove them on the imagery side of the continuum without using progressive opacity.Roads are so important for the understanding of the territory that we suggest in (Hoarau, 2015) to use a method of local adaptation.The symbolization choice should enhance the contrast between these roads and their graphic context around coming from the orthoimage.This symbolization method can be used as a complementary component to the interpolation components proposed in this paper.

CONCLUSION
This paper addresses the problem of the interpolation between two styles of graphic representations of a geographical space, i.e. orthoimagery and vector data, making the control of the levels of realism and abstraction possible.The approach is based on relevant graphic parameters of both representations which are useful to manage realism and abstraction levels, i.e. color and texture.We propose a method to create a set of hybrid visualizations by mixing regularly heterogeneous geographical data.Our work contribute to improve image-based representation design by using imagery as a graphic inspiration source.We assume that transferring graphic characteristics of orthoimagery in map design enables cartographers to create better hybrid visualizations.Finally, style continua created using our interpolation method are conceived as support for the navigation between heterogeneous visualizations in order to allow the user to browse several realism levels and representation styles.Therefore, our style continua should improve focus+context interactive tools such as visualization lenses presented by Karnik et al. (2009); Pindat et al. (2012), by providing locally parameterised transitions.Moreover, it could be interesting to increase this interaction allowing the user to customize the continuum by choosing pivot symbolizations and interpolation components.
Finally, we would like to apply those style interpolation methods to other types of heterogeneous data: for instance, an interpolation between an old map an its contemporary one would be interesting to highlight some design choices.The interpolation between two orthoimages, taken at two moments of time would be interesting also in order to make continuous temporal series for multiplexing interfaces.Interpolation methods will be improved to handle non-linear interpolation and to include constraints to make graphic parameters interpolating while taking into account visual or semantical relationships between geographical themes.
Mixed textures are natural and procedural textures that have been overlaid, with high transparency for one of the two textures.ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume II-3/W5, 2015 ISPRS Geospatial Week 2015, 28 Sep -03 Oct 2015, La Grande Motte, France This contribution has been peer-reviewed.The double-blind peer-review was conducted on the basis of the full paper.Editors: S. Christophe and A. Cöltekin doi:10.5194/isprsannals-II-3-W5-507-2015

Figure 5 :
Figure 5: An example of procedural texture.

Figure 6 :
Figure 6: SLD extension for Perlin noise Texture description

Figure 7 :
Figure 7: Color ramp generation by interpolation

Figure 9 :
Figure 9: Simple color interpolation of Perlin noise texture

Figure 10 :
Figure 10: Double color interpolation of Perlin noise texture

Figure 12 :
Figure 12: Opacity interpolation of overlaid vector data

Figure 15 :
Figure 15: Symbolization interpolation components and pivot symbolizations used to create our style continuum.

Figure 16 :
Figure 16: Final geovisualizations forming a continuum between a map and an imagery.