THE POTENTIAL OF SPECULAR REFLECTIONS FOR FAÇADE IMAGE ANALYSIS

Several approaches for window detection have been proposed in recent years, which all use the window size, shape or position within the façade as characteristics for the detection. In this paper another property of windows is proposed for their detection: Windows are made of glass, so they tend to specularly reflect sun light, while the surrounding wall diffusely reflects it. To this end, colour values in multiple views are analysed to detect specular reflections. In the experiments, walls often show a high homogeneity of small colour differences, and many windows are clearly visible in the form of colour differences. Nevertheless, it is difficult to segment windows from the façade if there is no prior knowledge about the intensity of the scene’s illumination. Promising results are obtained if the façade lies in the shadow, e.g., of another building or a tree.


INTRODUCTION
Window detection and extraction from images or point clouds is a very active research field in photogrammetry and computer vision.When inspecting only images, windows often show a big contrast to their surrounding, the masonry.Thus, edges and corners of windows can sufficiently be detected.E.g., (Lee and Nevatia, 2004) detect windows based on cumulative gradients in rectified images, and in (Reznik and Mayer, 2008) candidates for windows are found using an implicit shape model.While the first approach relies on horizontally and vertically aligned windows, the second approach uses the window alignment to determine consistent window parameters for a fac ¸ade.
Another way of describing windows namely as aggregate of substructures is employed in ( Čech and Šára, 2009) considering two characteristics: the darkness of window panes in comparison of the surrounding window frames as well as the compositional structure of fac ¸ades.Again, rectified views and the grid structure of windows are essential for the success of the approach.Perspective views have been used as input in (Ali et al., 2007), where windows are detected on the basis of their texture.Although, such image-based approaches produce good results on (rectified) down-town fac ¸ades, they obviously have problems with occlusions, strongly perspective views or strongly textured or decorated fac ¸ades.Moreover, many windows to rooms in the cellar or attic are often missed by the approaches above, because they are usually very small.When working with point clouds from laser scanning, windows often reflect or absorb the light ray, so point clouds typically contain no points at window positions.This fact is used for extracting windows in, e.g., (Pu and Vosselman, 2007)or (Wang et al., 2011).If a point cloud has points at windows, e.g., because it has been reconstructed by dense image matching, then these points lie behind the fac ¸ade plane.Consequently, the relative position of windows with respect to the fac ¸ade's surface is often used for window detection, e.g., (Tuttas and Stilla, 2013), (Aijazi et al., 2014) or (Nguatem et al., 2014).Here again, the alignment of * Corresponding author.
windows in a regular grid is often used to obtain consistent parameters.
In contrast to these approaches which detect windows locally, there also exist various methods for deriving a description of the whole fac ¸ade.Particularly, the presence of windows is verified by analysing colour and geometrical features, e.g., (Ripperda and Brenner, 2006) or (Becker and Haala, 2008).Analogue to ( Čech and Šára, 2009), colour features and context information from the spatial arrangement of fac ¸ade parts are important for interpreting fac ¸ades, as shown by recently proposed approaches for deriving fac ¸ade descriptions from orthographic projections, e.g., (Martinovic et al., 2012) or (Teboul et al., 2013).Here, the constructed rules for deriving a fac ¸ade description highly depend on the building styles and perform very well as long as the fac ¸ades have a homogeneous appearance and structure.
Various methods have been proposed in the cited work above using several types of data such as images as well as point clouds from matching or laser scanning.They all have in common that they consider properties of windows that are described in architectural guides, e.g., (Fletcher, 1905).For architects, windows are described as openings of the masonry, which are used for air and light supply of the building's rooms, and their size, shape, position and the grouping within the wall are very important for the visual appearance of the entire fac ¸ade.
Yet, one important characteristics is missing in such architectural descriptions, or it is only implicitly mentioned: Window panes have almost always a planar surface, which consists of translucent and mirroring glass.Consequently, windows may show an other reflection behaviour than the surrounding masonry, cf.Fig. 1.Windows may reflect light like a mirror, i.e., specular reflections may occur, and the masonry normally reflects light diffusely, i.e., its colour changes less.
In this paper, multiple views of fac ¸ades are used to reconstruct the scene in 3D.A reflection analysis can be performed when using image information together with their 3D pose and the reconstructed scene.For this it is assumed that the illumination is constant during the time of image acquisition.Regarding urban building scenes, there is one dominant light source, namely the sun, yet there is also ambient light due to diffusion processes in the atmosphere and due to interobject reflections.Because of its complexity, the illumination of the scene is not explicitly modelled in this work.This also refers to the light intensity and its absorbed portion at different surfaces.
A further assumption concerns the smoothness of the building surfaces.The visual appearance of buildings and their parts is assumed to change as less than the appearance of vegetation.In aerial images vegetation may show the illuminated surfaces in one view and the shadow side in another.Due to the smoothness of building surfaces, this effect does usually not occur strongly in fac ¸ade images.This paper is organised as follows.In the next section, the related work on analysing reflections is summarised and discussed with a focus on the detection of specular reflections.In Sec. 3, the derived approach for reflection analysis is presented.Experiments are described in Sec. 4, which also includes the presentation and discussion of the results.In the last section, the potential of analysing reflections is summarised and an outlook on further work concludes the paper.

RELATED WORK ON REFLECTION MODELLING AND DETECTION
In optics and computer graphics, the appearance of matte surfaces only depends on the pose of the surface and the direction to the illuminating light and it is modelled using the Lambertian or cosine law.Hence, the intensity of the reflected light called diffuse reflection is proportional to the cosine of the angle between the surface's normal vector and the vector to the light source.I.e., a diffusely reflecting surface appears equally bright from different view points.In contrast to diffuse reflections, specular reflections do not spread the reflected light in all directions, but perfect mirrors keep all light together, reflecting it in only one direction.
In reality, most surfaces are neither a perfect mirror nor a perfect diffuse reflector, they show a combination of both types.Consequently, both reflection types are mixed in computer graphics, where the modelling of surface reflections, interobject reflections and the illumination are the task for rendering synthetic images.
Several methods for modelling reflections as combinations of diffuse and specular reflections have been proposed, e.g., (Phong, 1975), (Cook and Torrance, 1982) or (Ward, 1992).Additionally, detailed models for bidirectional reflectance distribution functions (BRDF) have been proposed.I.e., the radiance of a synthetic pixel depends at least on the direction of the illuminating light because its intensity gets partially reflected and absorbed, also the viewing direction on the surface point.
Recently, (Schwartz et al., 2014) presented their experimental setup for determining BRDFs of small objects from images, simultaneously acquired in a half-spherical dome of cameras.Unfortunately, this approach is not applicable to fac ¸ade analysis, and it is not convincing that the BRDFs of fac ¸ades and window panes can be modelled by adopting BRDFs of bricks, plaster and flinders of glass observed in a laboratory environment.
Considering computer vision, real images show the effects of illumination and reflections, e.g., by specular highlights or shadows.Specular reflections are clearly visible in images, because they lead to bright image pixels in certain views, even if the surface colour is dark.Sometimes images also show the effect of blooming, if the radiosity of the reflected light is too large and also neighbouring pixels show large brightness.(Healey and Binford, 1988) reconstruct cylindrical object surfaces from resulting elongated specular reflections.It is one of the first works, where geometrical and optical analysis were combined.
Videos of building scenes from static webcams have been analysed, e.g., by (Sunkavalli et al., 2007).In such multi-temporal image sequences, the illumination changes only depend on the position and brightness of the sun light.In their experiments, (Sunkavalli et al., 2007) present nearly constant intensity values for pixels in the shadow and significantly higher values with a larger variance for pixels showing directly illuminated building surfaces.All intensity profiles of an image are used to segment shadow areas and then the contributions of ambient illumination from the sky and direct illumination from the sun are determined.
In (Klinker et al., 1990) the colour distribution is analysed and the distribution of pixels showing diffuse reflections is estimated as well as the distribution of pixels showing specular reflections.
Both distributions can be bounded by cylinders in colour space, forming a skewed "T".In further analysis the colour distribution of diffuse reflections has been in focus, and the potential of the approach is demonstrated for imagery acquired in a controlled environment, i.e., in a laboratory.Considering outdoor images, especially fac ¸ade images, this approach seems not applicable, because the colours of pixels showing diffuse reflections cannot be bounded by a slim cylinder, but they have a more complex distribution.Consequently, the detection of specular highlights seems to be more promising.
If the view point changes slightly, the position of the specular highlights also moves slightly in the acquired images.The trajectories of highlights has been analysed in (Oren and Nayar, 1996), observing a shiny, metallic sphere from different directions.Recently, specular reflections of more complex metallic surfaces have been analysed in (Herbort and Wöhler, 2012).In the proposed algorithm, active range scanner data and image-based information are combined to obtain reconstructions also fulfilling the constraints of the reflection model.
( McHenry et al., 2005) and (Netz and Osadchy, 2013) are two recently published works, where specular highlights are detected in single views.Both approaches use a simple threshold segmentation to perform the separation of bright image parts from others.
In (McHenry et al., 2005), the threshold is set by 0.9 times the largest intensity value and in (Netz and Osadchy, 2013), the 10% brightest pixels are segmented.In both cases, all important specular reflections are detected, but also many other bright pixels.These false detections are eliminated by further inspections, e.g., a classifier on the segmented edges close to the detected highlights (McHenry et al., 2005) or a comparison with rendered visualisations of 3D object models from a data base (Netz and Osadchy, 2013).Both segmentations cannot be adopted to fac ¸ade image analysis, because too many bright objects such as clouds and white fac ¸ades are also segmented as specular highlight.
( Lee and Bajcsy, 1992) analysed the spatial expansion in colour space for diffuse and specular reflections in two views.Since diffuse reflections appear nearly equal from different view points, all pixels showing diffuse reflections have the same colour distribution over both images.In contrast, pixels with specular reflections have a colour which only appears in one image, and thus, these pixels can be separated from the others.Yet if both images show specular reflections, this approach fails.It has been adopted to multiple views in (Lin et al., 2002).There, the comparison of colours is not performed over all pixels of other views, but along the epipolar lines.
In this paper, the idea of integrating the relative orientation between the multiple views is extended by additionally considering the reconstructed object surface.Image sets are first relatively oriented (Mayer et al., 2012).Secondly, dense point clouds are obtained from semi-global matching (SGM) (Hirschmüller, 2008) and multi-resolution range data fusion (Kuhn et al., 2013).Last, surface models are derived employing (Bodenmüller, 2009).Thus, the 3D geometry of the approximated surface can be considered for analysis of reflections as well as the corresponding image intensities.

DETECTING SPECULAR REFLECTIONS IN MULTI-VIEW COLOUR IMAGES
Let In with n = 1, . . ., N be the N fac ¸ade images, Pn the corresponding projection matrices, and M the reconstructed mesh model, consisting of K triangles T k with k = 1, . . ., K. The mesh does not need to be water-tight, so it may and will have holes caused, e.g., by occlusions or opened windows or doors.In Fig. 2, the relation between the images and the reconstructed scene represented by a single triangle of the mesh is sketched.Furthermore, let X k1 , X k2 and X k3 be the three 3D-points which span triangle T k .The three points are visible in image In at pixels x nk1 , x nk2 and x nk3 with if there is no occluding triangle in between.The z-buffer algorithm as presented in (Foley et al., 1996) is used to enforce this visibility constraint.Almost always, the pixels x nki are disjunct for different i, and often they are also not adjacent.Hence, triangle T k is visible in all pixels intersecting the convex hull H nk of the three pixels x nk1 , x nk2 and x nk3 .All image pixels which do not relate to any triangle of the mesh are not further considered for the reflection analysis.Thus, reflections from moving objects such as cars and objects out of the depth range such as clouds are not analysed.
All images were taken with a Ricoh Caplio 500SE camera with a fixed focal length of 5.8 mm.The image sequences were originally acquired for 3D reconstruction (Mayer et al., 2012) and functional modelling (Nguatem et al., 2014).Since a normalization of intensity values is not necessary for both approaches, the exposure time had not been fixed.So, small intensity corrections might have been performed automatically.
Let c nk be the average RGB colour vector of all pixels x nki in H nk .I.e., triangle T k has colour c nk when viewed in image In.If T k is occluded in image In as determined by the visibility analysis above, then there is no colour c nk available.The colour difference for triangle T k can be determined by comparing the colour over all images where it is visible obtaining where ||.||2 represents the Euclidean distance between both colour vectors.Alternative colour spaces can also be considered, e.g., the CIE LUV colour space, in which two equally distant colours are perceived as equally distant by human viewers (Foley et al., 1996).
It is proposed to segment specular reflections by thresholding the triangle's colour differences.I.e., if d k > θ, then specular reflections occur in at least one image.For the RGB colour space and 8 bit imagery is employed.It should get analysed if θ can be chosen independently from the illumination of the scene.
Triangles with d k ≈ 0 show a perfect diffusely reflecting surface, and triangles with d k near the upper boundary (≈ 441) appear nearly black in at least one image and white in at least another.Thus, these triangles show a specularly reflecting surface.If only windows have a mirroring surface within a fac ¸ade, they can be detected using this reflectance analysis using multiple views.For visualisation purposes, the colour differences can be transferred to corresponding image pixels, rounding all distance values and setting the maximum intensity to 255, i.e., all larger distances with d k > 255 are set to white.The time of acquisition is short enough, so that all images of one sequence can be assumed to be taken under the same illumination condition.The first nine image sequences clearly show specular reflections, visible by blooming in some images.The tenth sequence contains one image with a direct view into the sunlight.By image analysis only, it would be very difficult to distinguish this direct view into the light source from specular reflections.The other five data sets do not show any image with blooming, so all those images probably show a mixture of diffuse and specular reflections.
For this is a preliminary study all colour images were downsized to 1208 × 906 pixels to have short processing time, although the geometrical accuracy of the reconstructed scene decreased simultaneously.The number of triangles per mesh varies between 60 000 and 560 000, cf.Tab. 1, i.e., the mesh still consists of many small surfaces.The size of triangles is well recognisable, cf.Fig. 3 where the colour differences between the triangles make them visible.The method for reflection analysis as proposed in section 3. only contains one parameter that has to be determined: θ.It is used for segmenting the triangles into two parts: triangles with a specular reflection in at least one view and triangles with only diffuse reflections.It can be shown that a sufficient global threshold for any image cannot be found.In another experiment, different image-specific values for θ are derived from the histogram of the colour difference image, and also gradients are determined and discussed with respect to window detection.

Manually Labelled Specular Reflections
Eight of the fac ¸ade images show specular reflections in the form of blooming.These eight image parts were selected to determine colour differences d k for each triangle of the mesh are significantly larger than their surroundings.Fig. 3 shows the image parts with these specular reflections.The upper rows present the original colour image containing the blooming, and the corresponding image below shows the visualisation of the colour distances.
Each specular reflection has been labeled by the pixel position for the center of circle and its radius.The radius was selected in a way that the brightness of the pixels only decreases outside the circle.Some specular reflections cover a few hundred pixels and others several thousands.
In each circularly labelled specular reflection, the colour differences d k were analysed by determining their minimum value (min d k ), mean value (mean d k ) and maximum value (max d k ), cf.Tab. 2. The wall parts around the selected specular reflections shows dark and medium gray values, but the specular reflections at the window panes are often very bright, i.e., the colour differences in these parts are mostly close to 200 or more.Sometimes, the shape of the window is well recognisable, and the image area showing blooming is also often good visible.No The minimum values do neither seem to be appropriate for detecting specular reflections nor for detecting windows, because almost each selected area contains at least one triangle with small colour differences.This may be due to holes in the mesh, bright reflections also in other directions, the visibility of the triangle in only a few views, or a false reconstruction, e.g., by a shift of points, so that the triangle appears in wrong image parts.
The maximum values are very large.The smallest maximum value of example c, is 186.4,all others are close to 255 or significantly larger.The mean values are also in a very high range, except for the eighth example.Visual inspection shows that the window causing the specular reflection often shows brighter values than its surrounding.Other large colour differences might exist at geometrical edges and corners of a window's opening or of a cornice.The second example b shows a bright part below the window, there the shadow edge has disturbed the 3D scene reconstruction and the surface of the wall is relatively rough in that area.

Image Segmentation
Due to different light intensities, e.g., caused by shadows, there is no global threshold for segmenting windows from a fac ¸ade wall.Consequently, the image specific histogram is used for deriving appropriate thresholds and gradient magnitudes are determined for a qualitative analysis.Regarding the threshold based segmentation, four different values to segment the colour difference images are derived from the histogram as the 30%, 50%, 70% and 80% quantiles with colour differences over 0 (meaning that they show a reconstructed surface).The first two values always lead to unsatisfying segmentations, but the higher two values often distinguish well between wall and window pixels.After determining gradient magnitudes, small values below 1.5 times the average magnitude are suppressed and the other magnitudes are visualised in a binary image.

Results
In this paper, only a qualitative colour difference analysis is presented.No experiments on actual window detection have been performed so far.There are still two problems to be solved for a reliable window detection.The first problem is shown in Fig. 4. At the right border of the colour difference image the two letters "So" are clearly visible, although there are no letters visible in the fac ¸ade image.In this case, another image of the sequence shows the first two letters of the street sign.Because, no other image contains the street sign, there does not exist any triangle in the mesh representing the surface of the street sign.Consequently, the colour of the street sign is considered as the surface colour of the fac ¸ade.wall.Nevertheless, the two regions of the image with the highest colour differences are the window showing specular reflection, and the fac ¸ade part in the upper left corner.The homogeneity of the colour differences at the wall is also visible, in the upper row of Fig. 8. There, the colour differences of windows are only slightly higher, but mostly caused by window openings.The example in the bottom row of fig.8 shows a view on a building corner.Again, higher colour difference values can be found at the sunny fac ¸ade, but window detection seems to be very difficult in such a strong perspective view.
Further results are presented in Figs. 9, 10 and 11.They all demonstrate the potential of analysing reflections for fac ¸ade image interpretation based on homogeneity of the colour difference values at walls and the clear visibility of windows due to mostly higher colour difference values.
Figure 6.From top left to bottom right: fac ¸ade image, visualization of colour differences to neighbouring images only, visualization of colour differences to complete sequence, segmentation of colour difference image using the 0.7 and 0.8 quantile of pixels unequal 0, image with gradient magnitudes.

CONCLUSION AND OUTLOOK
In this paper, a new view on window detection is proposed by analysing reflections based on colour intensities in multiple views.Instead of only focusing on window size, shape or position in fac ¸ades, also the differences in reflection behaviour can be analysed.This seems particularly useful when verifying other window detection results.I.e., if image-based window detection, e.g., (Reznik and Mayer, 2008) returns different results than approaches analysing the 3D point cloud, such as (Nguatem et al., 2014), the presented reflection analysis could lead to a disambiguation.Contradictory results for window detection occur if windows are drawn on fac ¸ades, or the window panes lie in the same plane as the fac ¸ade.
In the presented approach, the colour differences of a reconstructed surface in multiple views are determined and analysed.
The experiments show promising results regarding the separation of specular highlights from image parts with diffuse reflection.Since the images have been acquired on a sunny day, most colour differences are large at positions around windows, thus the approach can be improved and extended for window detection.
Since the results also show several problems, the improvements proposed in the main part of the paper should be considered for further work.At least, the experiments should be repeated under different illumination conditions, especially on cloudy days, for which the ambient illumination of the sky prevents significant specular reflections.
Illumination conditions may strongly affect several areas of image and scene analysis, e.g., image matching, 3D reconstruction, and scene interpretation.It certainly would improve the results if all these steps were integrated into a big approach combining them all.This is a very big goal and in the moment it does not seem to be possible in a efficient way.
When matching images for image orientation, approaches for images with a small baseline, e.g., (Snavely et al., 2006), may ignore reflection analysis, because there are only big changes if Figure 7. From top left to bottom right: fac ¸ade image, visualization of colour differences to neighbouring images only, visualization of colour differences to complete sequence, segmentation of colour difference image using the 0.7 quantile of pixels unequal 0.
specular reflections occur.Contrarily, approaches for matching images with a wide baseline, e.g., (Mayer et al., 2012), should consider reflection analysis, because the brightness of many surfaces changes under these circumstances.A first major improvement could be the integration of reflectance analysis as part of 3D reconstruction similar to (Herbort and Wöhler, 2012).
This paper has not considered scene interpretation in general, i.e., the recognition of buildings, streets, sky or vegetation.Although knowledge about semantics can help to decide on the existence of an object surface and its smoothness or roughness, a second major improvement would be to consider approximated geometric primitives as smooth object surfaces, e.g., by detecting planar structures, cylinders, spheres, or cones as proposed in (Schnabel et al., 2007).Geometric approximation could easily be integrated directly after 3D scene reconstruction.
A third suggestion concerns the length of an image sequence or the maximum angle towards a surface from different view points, respectively.If changes of brightness of object surfaces are studied, it is likely that two corresponding colours in views with a small base line are more similar than the colours of two views with a large base line.In general, such changes depend on direction of the light source(s), the pose of the surface and its smoothness as well as the range of the different view points.
As fourth suggestion, the colour differences should be analysed in more detail with imagery acquired under controlled conditions, e.g, in a laboratory, so that the knowledge about the illumination can also be considered, and interobject reflections can mostly be excluded.Then it should be possible to detect specular reflections from colour differences, especially if the luminous intensity of the incoming light is known.If the scene is geo-located and the time of image acquisition is known, the approximate position of the sun can be determined using (Reda and Andreas, 2008).Then, the position of the light source can be considered for reflection analysis.

Figure 1 .
Figure 1.Eight different views of four windows of one fac ¸ade.While the colour of pixels showing the surrounding masonry only changes slightly, the brightness of pixels of the window panes can vary a lot, especially when specular reflections occur.

Figure 2 .
Figure 2. Sketch of a scene with a reconstructed mesh represented by triangle T k ∈ M and three views.The brightness of the projected triangle varies in all three images from white (I1) to black (IN ).
15 image sequences are used to evaluate the proposed method.The images were acquired on a sunny day in August and show typical fac ¸ades from a Gründerzeit quarter in Berlin, Germany, showing (often refurbished) historical and contemporary fac ¸ades.

Figure 3 .
Figure 3. Eight manually labelled specular reflections in fac ¸ade images and the visualisation of the colour differences d k for the visible triangles in this scene.The gray values are equal to the colour difference, and all differences above 255 are set to 255 (white).

Figure 4 .
Figure 4. Fac ¸ade image and visualization of colour differences to the complete sequence.The street sign showing the two letters "So" are visible in one image, thus there are big colour differences at that position.

Figure 5 .
Figure5.From top to bottom: fac ¸ade image, visualization of colour differences to neighbouring images only, and visualization of colour differences to complete sequence.The fac ¸ade in the center of the image reflects with a very bright intensity.If compared to the neighbouring images only, the differences are very small.Consequently, the wall diffusely reflects the light.If the colour differences are calculated over the complete sequence, the borders between wall and windows do not stay so clear.This might be caused by a specular fraction of the reflection or by incorrect reconstructed object surfaces in the mesh close to the window openings.

Figure 8 .
Figure 8. Two fac ¸ade images and the corresponding colour difference images over the complete sequence.

Figure 10 .
Figure10.From top left to bottom right: fac ¸ade image, visualization of colour differences to complete sequence, segmentation of colour difference image using the 0.3, 0.5 and 0.7 quantiles of pixels unequal 0, image with gradient magnitudes.

Table 1
. Data sets with number of images and reconstructed mesh size.The last column gives if there exists at least one image with a specular reflection (including blooming) in the data set."sun" represents a data set, where a direct view into the sun occurs.

Table 2 .
Values of d k for pixels at manually labelled specular reflections.The last row below the line does not refer to a specular reflection, but to a direct view into the sun.
. # pixel min d k mean d k max d k