ROAD MARKING EXTRACTION USING A MODEL&DATA-DRIVEN RJ-MCMC

: We propose an integrated bottom-up/top-down approach to road-marking extraction from image space. It is based on energy minimization using marked point processes. A generic road marking object model enable us to deﬁne universal energy functions that handle various types of road-marking objects (dashed-lines, arrows, characters, etc.). A RJ-MCMC sampler coupled with a simulated annealing is applied to ﬁnd the conﬁguration corresponding to the minimum of the proposed energy. We used input data measurements to guide the sampler process (data driven RJ-MCMC). The approach is enhanced with a model-driven kernel using preprocessed auto-correlation and inter-correlation of road-marking templates, in order to resolve type and transformation ambiguities. The method is generic and can be applied to detect road-markings in any orthogonal view produced from optical sensors or laser scanners from aerial or terrestrial platforms. We show the results an ortho-image computed from ground-based laser scanning.


INTRODUCTION
Road markings are painted on the road surface to provide traffic rules to drivers and pedestrians.Since the 90s, Inteligent Trasportation Systems (ITS) applied on-board lane marking detection to drive within a portion of the road (Bertozzi and Broggi, 1998).In the last decade the dedicated methods became more and more elaborated and some benchmarks were proposed (Veit et al., 2008;Fritsch et al., 2013).Most of the methods developed by the ITS community are devoted to 2D road-marking detection in image space.Nowadays, there is a growing need for georeferenced road-marking geographic databases.Such databases are required in applications such as road modelling for traffic simulation (Despine and Baillard, 2011) and precise absolute localization of vehicles (Ziegler et al., 2014).In this paper we aim at developing a generic approach for extraction of road-marking from an extensible road-marking library.

State of the art
Road-marking mapping was investigated by photogrammetric community using ground-based stereo-pairs acquired by MMSs (Mobile Mapping Systems) since the 90s.The proposed methods in (He and Novak, 1992;Tao et al., 1998;Habib, 2000) studied detection and 3D reconstruction of center-lines.Soheilian et al. (2010) presented an approach for 3D reconstruction of zebracrossings and different types of dashed-lines.Some other systems used high resolution georeferenced aerial images (Steger et al., 1997;Tournaire et al., 2006;Kim et al., 2006).
Since development of ground-based mobile Laser scanners, some authors used intensity of 3D point clouds for road-marking segmentation.Most of the authors compute a road ortho-image using intensity of points.Segmentation step is then performed in the image space by chaining a set of thresholding and morphological filters (Yang et al., 2012;Kumar et al., 2014;Guan et al., 2014).Yu et al. (2014a) used the same strategy of segmentation followed by clustering the pixels into individual markings that are classified to semantic markings.
Most of the aforementioned methods are based on bottom-up strategy.They begin with low level feature extraction.Then grouping algorithms provide higher level objects that are classified in semantic categories by classification algorithms.Bottomup pipelines are fast and geometrically precise but very sensible to imperfections of input data.
In contrast to bottom-up approaches, top-down methods take benefit from the known context (prior) to propose plausible solutions and verify their coherence with image data.Generally, top-down approaches are more robust to imperfections in the input data, but in return they are slower.This strategy was also applied to the problem of road feature extraction.Lacoste et al. (2005) detects road network with stochastic point processes in remote sensing images.Similar methodology was applied by Tournaire and Paparoditis (2009) for rectangular road-marking and by Yu et al. (2014b) for manhole detection.
To conclude, many authors investigated bottom-up or top-down approaches to road-marking detection.Some of them deal with road-marking type recognition with a set of ad-hoc methods adapted to different road-marking types.We aim at developing an integrated bottom-up/top-down approach which deals with any type of road-marking.

Proposed approach
We propose a top-down approach, based on a Marked Point Process of roadmarking instances.It combines the advantages of a top-down approach which enforces global priors (eg : object interactions) and the increased performance of a data-driven exploration of the search space.A fist contribution is the proposed modelling of the roadmarkings from an extensible library of binary templates.Secondly, this approach is supplemented by a model-driven kernel, which uses a preprocessing of the roadmarking templates to resolve type and transformation ambiguities.Namely, a roadmarking template is correlated with all possible type and transformation modifications in a generic preprocessing phase which is data-independent.This analysis is used to escape from local minima by proposing roadmarking configuration modifications that tend to explore neighboring local minima.
After formulating the problem as an energy minimization over the set of roadmarking configurations (section 2.), we will present in section 3. how this optimization is tackled using a Reversible Jump Markov Chain Monte Carlo (RJ-MCMC) approach.Finally, sections 4. will analysis the results, before concluding in section 5..

Input data and pattern library
In this study we use georeferenced point clouds acquired by a MMS in urban areas (cf.Fig. 1(a)).An ortho-image is computed by vertical projection of points.The pixel size is set according to the resolution of points.To every pixel is associated the intensity value of the lowest projected point.A set of morphological filters is subsequently applied to fill in the holes, following similar approaches to (Yang et al., 2012;Kumar et al., 2014;Guan et al., 2014).Fig. 1(b) shows the ortho-image I corresponding to our running example.
In most countries, road-markings are painted following strict geometric specifications.We propose to use such a specification as a pattern library L of road-markings.This specification is for instance available in France as the Instruction interministérielle sur la signalisation routière 1 (Fig. 2).

Model definition
Road-marking detection is formulated as positioning an unknown number (n) of patterns (I i ∈ L) in image I. Positioning consists here in determining conformal parameters (x, y, θ, λ) allowing 1 Instruction interministérielle sur la signalisation routière.Septième partie: Marques sur chaussée.Ministère de l' Écologie, de l' Énergie, du Développement durable et de l'Amenagement du territoire.Dec. 2011.Paris, France Figure 2. A sample of our road-marking pattern library L with (GSD = 2cm).The white (resp.black) pixels correspond to the road-marking object (resp.background) and the grey pixels indicate mask pixels inside which the correlation (data term) is computed.
to re-sample a pattern to image space (cf.Fig. 3).The whole configuration of roadmarkings is parameterized by a vector X: where : Xi = ( i, xi, yi, θi, λi) roadmarking parameters ( 2) We further denote T x,y,θ,λ the change of coordinates that maps the pixels of the road-marking pattern I to its reconstruction Irec(Xi) = T x i ,y i ,θ i ,λ i (I i ) onto ortho-image I.

Energy formulation
We apply an energetic formulation to the problem.The energy should reflect the coherency of the road-marking objects, parameterized by X, with the ortho-image I and prior knowledge concerning the objects.We define an energy function that is composed of two terms called external (data attachment) and internal (prior) as follows: In the following sections we explain these energies.

External energy term
External energy called also data attachment expresses the likelihood of the objects X in relation to the image I.We have : We define function f that measures the coherency of an object Xi with image I using zero-mean normalized correlation (ZMNC): I is the image template of pattern in the road-marking library L (Fig. 2).T −1 x,y,θ,λ is used as it is the input ortho-image I which is re-sampled to the pattern geometry.In order to reduce the effect of neighboring road-markings (where the bounding box of two road-marking objects have an intersection for instance) the ZMNC is computed using only the pixels situated in a mask around the pattern (grey regions in Fig. 2).We used ZMNC because it is the simplest measure of similarity that is quite invariant to changes in local contrasts.This measure can be substituted by more robust measurements such as mutual information (Maes et al., 1997) without any change in the algorithm.
As shown in Fig. 4, the external energy associated with the object Xi is computed using the following equation: In this equation f 0 ∈ (0, 1) is a similarity threshold below which the external energy will be positive and penalize the object.This parameter was set manually by trial and test to f 0 = 0.35.Thus, if the similarity score is higher than f 0 , the associated energy is negative and leads to reduction of total energy.Lower scores increase the energy.
f (X i , I) Overlap interaction This energy enables us to penalize the overlapping objects.It is computed for pairs of nearby objects.
First the overlap of bounding boxes are tested.In case bounding boxes overlaps, a pixel-wise overlapping test, measuring the proportion of the overlap, is performed (cf.The overlapping energy is defined as follows: where S( , x, y, θ, λ) = T x,y,θ,λ (I ) is the resampled image of the pattern and | • | and ∩ denote respectively the area and intersection of white pixels.The coefficient β expresses the influence of overlapping energy in the total energy.This energy is a penalizing energy and is equal to 0 for non-overlapping objects.

Marked point processes, RJ-MCMC and Simulated annealing
The optimization problem at hand is facing a search space with varying dimensionality, as the number of road-markings itself is to be determined.Similar to (Tournaire and Paparoditis, 2009), we adopt a global strategy that is based on stochastic geometry, namely Marked Point Processes.Each road-marking object may indeed be defined as a point (x, y) with continuous marks (θ, λ) and a discrete mark l.Thanks to the homogeneous parameterization of objects across road-marking types, there is no need to consider a Multi-Marked Point Process (Lafarge et al., 2010;Mallet et al., 2010), simplifying the approach.
The solution of the optimization problem is sought as a realization of a Marked Point Process.Thus the search space must be probabilized using a so-called reference process, which is generally taken as a Poisson distribution over the number of objects coupled with an independent sampling of each object parameters uniformly over the search space.We propose here a datadependent reference process that replaces this uniform object parameter distribution with a distribution biased by a likelyhood map computed from the input data I.This distribution will be presented in section 3.2.1.This has the advantage of not even proposing roadmarkings that fall in the image but not on the road where lidar points have been acquired.
Reversible Jump Markov Chain Monte Carlo (RJ-MCMC) enables the sampling of a Marked Point Process that is sampled from an unnormalized target distribution.By embedding the RJ-MCMC sampler inside a simulated annealing framework, Monte Carlo sampling is turned into an optimization procedure : the target distribution evolves continuously from the reference process at temperature t = ∞ to a discrete sampler of the global energy minima at t = 0, provided that the temperature decrease is sufficiently slow (logarithmic).In practice, the initial temperature is commnly chosen sufficiently high and a geometric temperature schedule is preferred for faster processing at the cost of loosing the global optimality property (Descombes, 2012).
The RJ-MCMC algorithm maintains a Markov Chain of configurations.At each step, a configuration modification is proposed by one of the available kernels.This configuration is either accepted according to its Green acceptance ratio (Green, 1995) or rejected, leaving the configuration unmodified.Unformally speaking, the Green ratio ensures that whatever which kernel is used, the resulting Markov Chain of configurations converges to the desired target distribution, as long as the whole support of the target distribution is reachable by composing kernel modifications.This enables the modular use of kernels which come in pairs to propose reversible configuration modification (jumps).Whereas Birth and Death kernels are sufficient for the algorithm to be correct, the following kernels enable a smarter exploration of the search space by tunneling between parameter subspaces that yield similar energy levels, aiming at the connecting regions containing distinct function minima.

Data-driven Birth and death kernels
Birth and death kernels allow to add an object to the configuration or remove it.These kernels correspond to jumps in higher or lower dimensions and ensure to explore the whole configuration space.These kernels allow to resolve for unknown number of objects in marked point processes.In most of the RJ-MCMC samplers, points and marks are sampled from uniform distributions supposing that no information about the distribution of parameters is available.Similar to (Zhu et al., 2000) we propose to apply bottom-up pattern recognition techniques in order to guide the top-down approach.This is done by estimating a coarse prediction for the point distribution of the marked point process.Thus, in the birth kernel the points are sampled from the estimated distribution allowing to visit more frequently the locations predicted as potential road-markings by the bottom-up method.This is done by a set of rough filtering on the input image (cf.

Estimating the road-marking probability density
As road-markings generally exhibit much higher reflectance properties than their surroundings, we are looking for bright pixels on a dark background in the input image.In order to amplify the contrast and to deal with local variations of intensity a top-hat filter is applied on the image.A sufficiently small threshold enables to remove a large number of background pixels without removing any bright pixel.Finally a smoothing filter provide an image corresponding to the road-marking probability density (pdfx,y) shown in Fig. 6.As one can see, the values of pixels on road-markings are usually higher than other objects (ex.curbs).The background usually yields very small values.This probability map being used in the reference process effectively restricts the search space to its support.
In the birth kernel the points are sampled from the estimated probability density and the marks are sampled from uniform densities.
xi, yi ∼ pdfx,y Even though the ortho-image's pixel size is known, λi is allowed to vary slightly around 1 since in practice the road marking objects are not set-up exactly according to the specifications and the objects can be painted on the road slightly larger or smaller than their specified size.

Geometric perturbation kernel
This kernel modifies one of the transformation parameters (xi, yi, θi, λi) of an object (Xi).An object is selected and a parameter is modified following a uniform distribution.This kernel is useful for adjusting the objects.

Model-driven perturbation kernel
Perturbation kernels such that the one described in the previous subsection are usually used to modify the position and the geometric form of objects, and applied perturbations are most of the time sampled following uniform distributions.In the following, we present a perturbation kernel that can modify all dimensions of an object including type of the mark (ex.Bike→ BUS).Also, as it is described in the following section, we aim at conducting the proposed perturbation by estimating their distributions a priori.

Estimating probability distribution of five dimensional perturbation
In this section, we describe a generic way to determine automatically type and transformation perturbations that may be applied to an object, given a generic collection of roadmarking templates, to yield a new object that yields a similar reconstructed image.To obtain a prior distribution on the perturbation to apply, we compute ZMNC maps between objects, considering both intra-type and inter-type correlations.In details, we consider a basic objects i (the original pattern) and compute its correlation with a modified object j around i. Let us consider one of the untransformed original pattern of the road-marking library Xi = ( i,0, 0, 0, 1) and another object Xj = ( j ,xj, yj, θj, λj).
The correlation between the reconstructed images Irec(Xi) and Irec(Xj) is computed and stored in a correlation map Mi.This correlation map contains such correlations for any j ∈ {1 . . .|L|} and for relative transformation parameters (xj, yj, θj, λj) discretized as follows: Here [v1, δ, v2] is the set containing the values v = v1 + k.δ where k ≥ 0 and v < v2.For any original pattern i of the roadmarking library, the perturbation space for the computation of the with ∆x = ∆y = 50 pixels δx = δy = 3 pixels In order to reduce the noise and thus the number of local maxima, the patterns were blurred before correlation.The parameters (∆x, ∆y, δx , δy, δ θ ) were empirically tuned in a way to avoid over-sampling regarding to the significance of correlation scores.
While varying λj could yield better results, we only considered fixed scaling perturbations in our experiments (∆ λ = 0).These correlation maps are turned into probability maps by thresholding negative values and normalization.The perturbations that are the most likely to occur are then sampled using the correlation maps {Mi}i=1,...,n.This is described in figure 8 that presents two interesting correlation results.On the left of the figure, we present a part of the correlation maps associated to the BIG ARROW mark ( = 4).This is the 2D correlation map, where θj = 0, λj = 1, i = BIG ARROW and j = TRIANGLE.The colors of the correlation map goes from blue (lower correlation values) to red (strong correlation values).Hence, as it is depicted in this part of the figure, applying a y * j shift in the y dimension to a TRIANGLE leads to a strong correlation with the original BIG ARROW (the basic big arrow in the road-marking library).Hence, if a BIG ARROW mark object X k = (4, x k , y k , θ k , λ k ) is selected by the process, the model-driven perturbation kernel will be likely propose its modification to a TRIANGLE ( = 3) mark object close to The right part of the figure describes the intra-object correlation map between two objects Xi, Xj of type BIKE, where xi = xj = 0, yi = θi = 0 and λi = λj = 1.It highlights the fact that applying a transformation such that yj = y * j and θj = θ * j = π leads to a strong correlation (the wheels are superimposed).Therefore, a BIKE looks similar to a y * j -shifted π-rotated BIKE, and the modeldriven perturbation kernel will encourage such data changes to be tested.Please not that the discovery of these correlations is done exhaustively on all pairs of type rather than resulting from ad-hoc expert rules.
Hence, model-driven perturbation kernel is sampled following a priori computed correlation distributions that depicts the sim-ilarities between objects of identical or different type.So, the model-driven perturbation guides the process to look for objects that look similar to the ones already chosen.

RESULTS
We present some results of the model and data-driven RJMCMC algorithm with the ortho-image extracted from a point cloud acquired by a MMS (see Fig. 1).Such an image is interesting since it contains an important number of marks (39) among several types of marks (4).The proposed method has been implemented using the librjmcmc open-source library (Brédif and Tournaire, 2012).
Figure 9(a) presents the final result obtained after more than nine millions of iterations.Such a result is quite satisfying since the algorithm correctly detects 27 of the 39 marks in the image.To obtain such results, a simulated annealing with 9 millions of iterations have been computed, with a geometric decreasing rate of 0.999999 and an initial temperature of 10.The process is made offline and may be used for roadmarking database production.With such parameters, it take 8 hours of computation (each of the 10 millions of iterations is composed of several resampling operations).Parameters f 0 and β have been set to 0.35 and 100 (a high β value will prevent from overlapping marks).
False detections As depicted in Fig. 9(b) among the 35 detected marks, 2 detected objects do not correspond to roadmarkings (n • 1 and 3 in Fig. 9(b)) and two triangles were detected on a roadmarking that the method should have been able to detect as a crosswalk strip (n • 2 in Fig. 9(b)).
Wrong identifications 4 crosswalk strips are wrongly detected as triangles (Fig. 9(c)).Those strips are situated in the sides of crosswalks where the strips are broken to follow the road curbs.These broken strips can not be modelled by our road marking parametrization.Indeed, the method deals with mark scale variations but is not designed to deal with anisotropic scaling.It would be interesting to deal with anisotropic scaling, to take into account the independent changes of length and width, at the cost of adding 1 dimension to the object's parameter set.However, detecting such parts of walking area marks as triangles is quite acceptable since the shapes look similar.As an extension, the lack of plausibility of such configurations could be embedded in the prior energy, to penalize these configurations.
False negatives Among the 39 ground truth marks, 7 marks have not been detected at all (Fig. 9(d)).The object n • 4 is a part of a lane marking and hence was not expected to be detected with the presented method since its not included in the pattern library (our method is not designed to deal with anisotropic scaling).shows that the method tends to converge.After a first step of important variations (during 2 millions of iterations), the energy evolution stagnates (the flat part between 2 and 5 millions of iterations).Finally, the energy tends to its final minimum in the last part of the process, when it detects marks that fit well on the image.
Figure 11 shows the evolution of the process every 600000 iterations.In the beginning, the algorithm explores various solutions in the image (Fig. 11 At the last stage, energy increase is almost prohibited by the RJ-MCMC acceptance ratio and good object proposals have already been accepted, yielding an overall poor birth./deathacceptation rate.In contrast, most of the small perturbations proposed by perturbation kernels allowing to fit the objects were accepted.

CONCLUSIONS AND TRENDS
We have presented in this paper an optimization algorithm based on stochastic geometry to detect punctual road-marking objects in ortho-view images.A generic model enables us to define any kind of punctual road-marking object homogeneously with one template binary image and four parameters of a conformal transformation.Thanks to homogeneous parametrization, the configurations of road-marking objects can be modelled by marked point processes and it is needles to use multi-marked point processes.
Another interest is in enabling the method to deal with an extensible pattern library for detecting still more punctual road-marking objects.An external (data attachment) together with an internal (regularization) energy functions enable to associate an energy to every configuration.The problem of road-marking detection was formulated as an energy minimization.An RJ-MCMC sampler coupled with a simulated annealing was applied to find the optimum configuration.We applied a data-driven birth kernel in order to improve the control of the RJ-MCMC sampler.Another important contribution of the paper is in proposing a modeldriven perturbation kernel improving the efficiency of the process in resolution of type and transformation ambiguities.
This work is in progress and some topics will be investigated in future work.Firstly, we aim at integrating higher level semantical priors in the internal energy such as local pairwise orientation relations, and more rules related to road grammar.Secondly, datadriven kernel can still be improved using national road graphs for better control of orientation sampling.Finally, better management of the parameters related to simulated annealing (White, 1984) and associated weights to energy terms (Chatelain et al., 2009) allows to find a trade-off between quality of results and computation time.
In the proposed method, model-driven perturbation is performed using correlation between two vector templates and the data energy uses a vector template as well.Instead of relying on vector templates, roadmarkings could be learned by example from a sufficiently representative bank of annotated real datasets to better match the variability of real-world roadmarkings.Finally, this approach has currently only be applied to datasets of limited sizes.To scale this approach, an interesting data-driven approach we would like to extend is proposed by Verdié and Lafarge (2012) which parallelizes the RJ-MCMC proposals and acceptations over a quad-tree decomposition of the search space of object centers.

Figure 4 .
Figure 4. External energy value as a function of ZMNC Fig 5).

Figure 5 .
Figure 5.An example of pair-wise overlap.(a) Non-overlapping objects.(b) Overlapping objects with overlap region in reds.

Figure 6 .
Figure 6.An estimation of road-marking probability density (pdfx,y) for the running example of Fig. 1(b). ∆x

Figure 8 .
Figure 8. Two interesting examples showing how the correlation maps control the model-driven perturbation kernel (cf. the text for explanations).correlation map Mi is :
(a)(b)), and then concentrates on the parts of the image where the road-marking probability density is higher.It shows that using such a density map accelerates the process since it quickly focuses on interesting parts of the image.The flat part in figure 10(b) corresponds to images 11(c) to (h), and the last decreasing energy part goes from (i) to (n).The birth/death kernel tend to propose large (positive or negative) energy variations.