CHANGE DETECTION VIA MORPHOLOGICAL COMPARATIVE FILTERS

In this paper we propose the new change detection technique based on morphological comparative filtering. This technique generalizes the morphological image analysis scheme proposed by Pytiev. A new class of comparative filters based on guided contrasting is developed. Comparative filtering based on diffusion morphology is implemented too. The change detection pipeline contains: comparative filtering on image pyramid, calculation of morphological difference map, binarization, extraction of change proposals and testing change proposals using local morphological correlation coefficient. Experimental results demonstrate the applicability of proposed approach. * Corresponding author


INTRODUCTION
This paper addresses some theoretical and practical aspects of the of the change detection problem.It means detecting new or disappeared objects on images registered at different moments of time and possibly in various lighting, weather and season conditions.The most popular and challenging change detection problem appears in analysis of bi-temporal or multi-temporal spaceborne or airborne remote sensing data.
A lot of change detection techniques are developed for remote sensing applications (Singh et al., 1989;Lu et al., 2004;Hussain et al., 2013).There are two main categories of change detection techniques: pixel-level and object-level.Pixel-based methods usually provide the attractive computational efficiency, but relatively low detection characteristics.In contrast, the object-based techniques usually provide the high detection quality, but require much more computational efforts.
In this paper we present a new change detection technique based on generalized ideas of Morphological Image Analysis (MIA) proposed by Pyt'ev (Pyt'ev, 1993).The overview of modern MIA results is given in (Vizilter et al., 2015).The original morphological change detection approach is based on the analysis of difference between the test image and its projection to the shape of reference image.In our generalized approach the morphological filter-projector is substituted by the comparative morphological filter with weaker properties, which transforms the test image guided by the local shape of reference image.So, such approach implements some important properties of object-level image comparison immediately in the pixel-level image filtering.Due to this, we can talk about the morphological mid-level change detection.It should provide the desired compromise between the computational efficiency of pixel-based methods and detection quality of object-based techniques.
The theoretical contributions of this paper are:  new change detection scheme based on morphological comparative filtering;  new morphological filters based on guided contrasting.
We need to note that comparative filtering scheme is close in some sense to the guided filtering scheme (He et al., 2010;Zhang et al., 2014), but proposed morphological filters additionally satisfy the special mathematical properties that allows constructing the morphological theory and utilizing the traditional morphological tools for image comparison.
The practical contributions of this paper are:  new algorithmic pipeline for change detection in remote sensing data;  implementation of change detection scheme based on both guided contrasting and diffusion morphology previously proposed in (Vizilter et al., 2014).
Qualitative experiments with proposed comparative filters based on guided contrasting are performed on a wide set of real images in different change detection tasks.Quantitative experiments with proposed change detection pipeline are performed on the public benchmark containing simulated aerial images.
This paper is structured as follows.In the second section the related works are briefly described.The third section contains the theoretical basics, new ideas, schemes and algorithms.In the fourth section the experimental results are reported.

RELATED WORKS
A lot of change detection techniques are developed for remote sensing applications (Singh et al., 1989;Lu et al. 2004;Hussain et al. 2013).In (Hussain et al. 2013) two main categories of methods are pointed: pixel-based change detection (PBCD) and object-based change detection (OBCD).The PBCD category contains the direct, transform-based, classification-based and learning-based comparison of images at the pixel level.The OBCD category contains direct, classified and composite change detection at the object level.We start our brief overview from PBCD techniques and then go to OBCD.
The simplest direct comparison techniques are the image difference (Lu et al., 2005) and image rationing (Howarth, Wickware, 1981).Image regression represents second image as a linear function of first (Lunetta, 1999).
Change vector analysis (CVA) was developed for change detection in multiple image bands (Bayarjargal, 2006).Change vectors are calculated by subtracting pixel vectors of coregistered different-time dates.Principal component analysis (PCA) is applied for change detection in two main ways: applying PCA to images separately and then compare them using differencing or rationing (Richards, 1984) or merging the compared images into one set and then applying the PCA transform (Deng et al, 2008).Tasseled cap transformation (Kauth and Thomas, 1976) produces stable spectral components for long-term studies of forest and vegetation (Jin, Sader, 2005;Rogan et al., 2002).Some other texture-based transforms are developed in (Tomowski et al., 2011).
Classification-based change detection contains the postclassification and composite classification.Post-classification comparison presumes that images are first rectified and classified, and then the classified images are compared to measure changes (Bouziani et al., 2010;Im and Jensen, 2005).The supervised (Ji et al., 2006) or unsupervised classification (Ghosh et al., 2011) can be of use.Unfortunately, the errors from classification are propagated into the final change map (Lillesand et al., 2008).In the composite or direct multidate classification (Lunetta et al., 2006) the rectified multispectral images are stacked together and PCA technique is applied to reduce the number of spectral components.
Machine Learning algorithms are extensively utilized in change detection.Artificial Neural Networks (ANN) are usually trained for generating the complex non-linear regression between input pair of images and output change map (Pijanowski et al., 2005).
The Support Vector Machine (SVM) approach based on (Vapnik, 2000) considers the finding change and no-change regions as a problem of binary classification in a space of spectral features (Huang et al., 2008;Bovolo et al., 2008).
Object-based techniques operate with extracted objects.The Direct Object change detection (DOCD) approach is based on the comparison of object geometrical properties (Lefebvre et al., 2008), spectral information (Miller et al., 2005;Hall and Hay, 2003) or texture features (Lefebvre et al., 2008;Tomowski et al., 2011).In Classified Objects change detection (COCD) approach the extracted objects are compared based on the geometry and class labels (Chant, Kelly, 2009).The framework based on post-classification (Blaschke, 2005) presumes extracting objects and independently classifying them (Im and Jensen, 2005;Hansen and Loveland, 2012).Multitemporalobject change detection presumes that the joint segmentation is performed once for stacked (composite) images (Conchedda et al., 2008).
In this paper we present a new change detection technique based on generalized ideas of Morphological Image Analysis (MIA) proposed by Pyt'ev (Pyt'ev, 1993) and further developed in (Evsegneev, Pyt'ev, 2006;Vizilter, Zheltov, 2012;Pyt'ev, 2013;Vizilter et al., 2014).Let's note that terms "morphology", "morphological filter" and "morphological analysis" refer to Mathematical Morphology (MM) proposed by Serra (Serra,1982) as well as to MIA.These theories of shape have a common algebraic basis (lattice theory), but different tasks and tools.The overview of MIA and its relation to MM is given in (Vizilter et al., 2015).Morphological change detection approach is based on the analysis of morphological difference map formed as a difference between test image and its morphological projection to the shape of sample image (see section 3.1 for details and formulas).

METHODOLOGY
This section contains the description of proposed mid-level change detection methodology based on morphological comparative filtering.

Morphological image analysis and comparative filtering scheme
Let's consider the mosaic image model utilized in MIA where nnumber of connected regions of tessellation F of the image frame R 2 , F={F1,…,Fn}; f=(f1,…,fn)intensity values; Fi(x,y){0,1}support function of i-th region.The tessellation should be obtained by some image segmentation procedure.The mosaic shape is a set of images with the same frame tessellation: For any image g(x,y)L 2 () the projection onto the shape F is defining as averaging pixel intensities of image g over each area of mosaic shape F: The similarity of images f(x,y) and g(x,y) is estimated by the normalized morphological correlation coefficients (MCC) where O(x,y)  constany constant-valued (flat) image, PO f = fo  mean(f(x,y)) and PO g = go  mean(g(x,y))mean values of projected images.Morphological extraction of differences on image g(x,y) relative to shape F is performed via comparison of g(x,y) with its projection to F, and vice versa: Let's note that this MIA scheme is basically asymmetrical due to different roles of input images: test image is projected to the shape; sample (or reference) image determines the shape.In this paper we propose the weaker scheme of morphological image analysis that excludes the ideas of shape and projection, but preserves the idea of asymmetrical comparative filtering for robust similarity estimation and change detection.
Let two images are given: sample (or reference) image f(x,y) and test image g(x,y).The mapping that takes these two images as input data and forms the filtered version gf(x,y) of test image depending on its relation with sample image we refer as а comparative filter: If sample image f is fixed, then comparative filter takes the usual form with one input and one output image: We call such filter the morphological comparative filter if it satisfies the following conditions: 3)   It is easy to see that if sample image f(x,y) satisfies the mosaic model (1), then the mapping (f, g) = PF g satisfies to (7)-( 9).Thus morphological projector is a particular case of morphological comparative filter, but morphological comparative filter should not be a projector in general case.Consequently, the described comparative filtering scheme is a correct generalization of MIA including the formulas for MCC (10) and MDM (11) generalizing the MIA formulas ( 5) and ( 6) correspondingly.

Comparative filters based on guided contrasting
Let's consider the comparative filter based on the local linear correlation.Let w(x,y) is a sliding window at position (x,y).The guided local contrasting filter is defined as: ( where g w(x,y) (u,v) is g(x,y) localized in a window w(x,y), go w(x,y)mean value of g(x,y) in a window w(x,y), K(f w(x,y) ,g w(x,y) )local linear correlation coefficient in a sliding window w(x,y).Such filter (12) satisfies conditions (9), preserves the similar details and smooths the non-similar details on a test image g guided by sample image f.So, the difference map (11) based on filter (12) can be applied for change detection.The size of details is determined by the size of window w(x,y).If we need to detect details of different size, one can use the image pyramid.In practice we proceed to guided contrasting filter with local search (in a zone p(x,y)) that provides the robustness relative to weak geometrical discrepancy of images: , The scheme of comparative filtering based on guided contrasting (Fig. 1) demonstrates the main idea of this approach.At first we perform the local smoothing of test image.Then we estimate the local similarity of extracted details and recover or not recover them depending on this local similarity.In result, similar details should be recovered, but non-similar details should be extremely smoothed.This simple and general idea can be expressed in the following form of generalized guided contrasting filter where a(f,g w(x,y) ) is a local similarity coefficient (LSC) of test image fragment g w(x,y) with sample f.Any LSC with such properties generates the filter satisfying (9).Different variants of LSC a(f,g w(x,y) ) can be considered for change detection task: -local MCC (5) (Pyt'ev, 1993); -LSC based on mutual information (Maes, 1997); -local mean square MCC (Vizilter, Zheltov, 2012); -geometrical correlation coefficients (Vizilter, Zheltov, 2012), etc.
In this paper we implement and explore just the guided contrasting filters ( 12)-( 13) based on local linear correlation.But in the future all of these variants can be implemented and tested both in change detection and in image matching tasks.
The main advantage of proposed guided contrasting filters relative to MIA projectors is the elimination of sample image segmentation step that allows obtaining the more precise and computationally efficient solutions.Additionally, this approach provides the robustness relative to weak geometrical discrepancy of images.

Comparative filters based on diffusion operators
The theory of diffusion maps was proposed in (Lafon, 2004), (Coifman, Lafon 2006) and (Coifman et al, 2007).It is based on notions of "heat kernels" and "heat dissipation" introduced in (Belkin, Niyogi, 2001) in the context of Laplacian Eigenmap technique for nonlinear dimensionality reduction.In (Vizilter et al, 2014) the generalized MIA formalism called Diffusion Morphology (DM) was proposed based on heat kernels and diffusion maps.It was developed as an image-to-shape matching technique, but it looks reasonable to try this approach in the change detection task too.Moreover, the connection between DM and MIA allows considering morphological diffusion filters as comparative filters in the sense stated above.The diffusion shape model for image f is introduced as a heat kernel hf (x,y,u,v): Corresponding diffusion filter Pf is formed as a linear convolution with normalized diffusion kernel pf(x,y,u,v), such that If f(x,y) satisfies the mosaic model (1) and heat kernel is formed as a relation "(x,y) and (u,v) are in the same flat region" Then diffusion filter Pf (16) turns into morphological projector PF (4).So, MIA is a particular case of DM.
In general case diffusion model ( 15) is formed as a heat kernel proposed in (Coifman et al, 2007): where vf(x,y) is a feature vector describing the sample image f(x,y) in some neighborhood (local window) w(x,y) around the point (x,y); >0 is a tuning parameter controlling the sensibility to feature vectors similarity.But the diffusion filtering with such heat kernel ( 18) is a time-consuming procedure.So, we prefer the diffusion filters proposed in (Vizilter et al, 2014) and based on point feature descriptor iLBP (intensity + LBP): where m(x,y)mean value of image in a window w(x,y); LBP(x,y)threshold local binary pattern LBP (Ahonen, 2004) calculated as a binary vector for central pixel (x,y) based on a comparison of its value and values of its neighbors in a window w(x,y).If the value of neighbor pixel is less than the value of central pixel and the difference between them is greater than threshold, then the corresponding bit is set to 1, otherwiseto 0. Correspondingly the heat kernel is where dham -Hamming distance, βtuning parameter balancing the importance of intensity and LBP parts in iLBP.Local binary patterns are stored as bit fields, and the computation of Hamming distance is performed via bitwise XOR operation.The exponent is calculated using table values.Mean value in a sliding window is computed by a fast algorithm with sliding sum recalculation.Due to this, the usage of iLBP allows both increasing the computational speed and obtaining heat kernels very similar to (18).
As stated above, the projective mapping (f, g) = PF g (4) satisfies to definition of morphological comparative filter and properties (9).From the functional point of view the generalized diffusion mapping (f, g) = Pf g (16) should be a kind of comparative filtering too, because it is a smoothing filter, and it transforms the test image in accordance with diffusion shape of sample image.Unfortunately, it does not formally match the exact matching property (9).But in this case we can talk about the soft matching property: diffusion filters preserve images of sample shape essentially better than images of other shapes.So, we may refer diffusion filters as soft comparative filters and use the corresponding diffusion difference map of form (11) for solution of change detection task.

Change detection pipeline based on comparative filtering
For the task of change detection in long-range (spaceborne or airborne) remote sensing data we propose the new change detection pipeline based on comparative filtering.It contains the following steps: 1. Comparative filtering (7) using the image pyramid; 2. Calculation of corresponding morphological difference map (11); 3. Binarization and filtering of morphological difference map; 4. Forming change proposals; 5. Testing change proposals using local morphological correlation coefficient (5); 6. Forming the output binary map of changes.
At the first stage of pipeline we apply the morphological comparative filters described above.The pipeline is the same both for guided contrasting ( 12)-( 13) and for diffusion filtering ( 16)-( 20).The use of image pyramid allows detecting details of different size.
Morphological difference map obtained at second stage is binarized using graph cut technique (Boykov and Kolmogorov, 2004).Then this binary image is filtered sequentially by morphological closing and opening filters (Serra, 1982) with small disk-shaped structured element.Such filtering allows deleting noisy regions of binarized difference map.The list of change proposals is formed via calculation of minimal bounding rectangles for all connected regions of filtered binarized difference map.Then the each change proposal is checked by the value of local MCC (5) compared with a pre-learned threshold.Local MCC is calculated in a weakly expanded proposal rectangle.The expansion of rectangle is performed in order to add some small neighbourhood of the proposed object.If the value of MCC is greater than threshold, corresponding connected region of filtered binarized difference map is painted on the output binary map of relative changes.
Let's note that this scheme is asymmetricalit provides the detection of changes in (new) test image g relative to (old) reference image f.If we need to find all differences in both images, it is required to repeat the procedure two times: at first using g as test and f as reference, and then using g as reference and f as a test.We also need to comment the presence of traditional MCC in our pipeline.We have stated above that MCC is not stable and/or fast due to problems with image segmentation.But this problem exists at the global image scale.Coarse or unstable segmentation leads to false object proposals or lost objects.But the mosaic shape of small local areas containing the change proposals is simple enough.The simplest histogram-based segmentation small number of levels provides local segmentation that looks fine for proposals testing.From the other hand, the properties of morphological projector are essentially different from the properties of comparative filters.So, testing proposals created by comparative filters via MCC means the combination of evidences from independent information sources.Such combination is preferable from the statistical point of view.It provides the more reliable detection results.

EXPERIMENTS
The results of experimental exploration of both comparative filtering and proposed change detection pipeline are reported in this section.In the first part of section some examples of guided contrasting and corresponding morphological difference map forming are demonstrated applying to real images for different scene types and change detection cases.In the second part the results of change detection experiments on the public benchmark containing simulated aerial images are described.Such qualitative experiments allow concluding that in all considered cases the proposed approach for difference map forming based on guided contrasting filtering provides reasonable scene change proposals and demonstrates the enough robustness relative to changes in lighting and other image capturing conditions.From the other hand, some true image shape changes are extracted which are not required to be detected as scene changes from the semantical point of view.For example, clouds and flowers in a building construction case really appear in a scene, but they should not be of interest regarding the building construction stage comparison.So, some additional analysis of formed morphological difference map is needed for final testing of the formed change proposals based on other type of task-specific information.It means that guided contrasting filtering and corresponding morphological difference maps can be useful as parts of different task-oriented change detection pipelines.Such pipeline for change detection in remote sensing data is described above.The results of its testing are reported in next subsection.

Quantitative change detection experiments
In our experiments with proposed change detection pipeline for long-range remote sensing we use the public Change Detection dataset introduced in (Bourdis et al., 2011).This dataset contains 1000 pairs of 800x600 simulated aerial images and 1000 corresponding 800x600 ground truth masks.Each pair consists of one reference and one test image.Some of image pairs contain scene changes and illumination differences.The dataset consists of 100 different scenes with moderate surface relief and several objects (trees, buildings etc.).Each scene is rendered with various viewpoints.The cameras are distributed at steps of 10 degrees on a circle of radius 100 meters at approximately 250 meters high, and with a fixed tilt of about 70 degrees.All images are modelled with a ground resolution of about 50cm pixel.
The methodology of our experiments is the following.We select a subset of 100 reference and test image pairs for 50 different scenes with 0 degrees relative camera angle.As proposed in (Bourdis et al., 2011) we compare the detection results with respect to the ground truth at pixel level, but calculate the precision and recall values at the object (region) level.In order to do this, we from the list of ground truth objects and list of detected objects (accepted regions of filtered binarized morphological difference map).Then we perform the object-toobject comparison via computing of object intersection area.If the intersection area is more than 50% then we decide that objects match each other.The numbers of true and false object detections determine the corresponding precision and recall values.
We implement and test our pipeline with following parameters: guided contrasting window size is 77 pixels; number of pyramid levels is 3; the size of disk structuring element in MM opening and closing is 5 pixels, the threshold value for morphological correlation coefficient at the final testing step is 0. The reported result of approach (Bourdis et al., 2011) on this database is about Precision=0.51 and Recall=0.52(by the graph).It is not the totally correct comparison, because this paper utilizes a little bit different testing methodology, but it seems reasonable to state that our results are at least not worse than competing ones.So, we can conclude that the proposed pipeline is useful for change detection in long-range remote sensing data.

CONCLUSION
This paper presents the new scheme of morphological comparison filtering that generalizes the morphological image analysis (MIA) scheme proposed by Pyt'ev (Pyt'ev, 1993).Our morphological scheme excludes the MIA ideas of shape and projection, but preserves the idea of asymmetrical comparative filtering for robust similarity estimation and change detection.The mapping that takes two images (reference and test) as input data and forms the filtered version of test image depending on its relation with reference image is called а comparative filter.Such filter is called the morphological filter, if it is a smoothing mapping that preserves any constant (flat) image and preserves the test image equal to the reference image.For implementation of this scheme we propose a new class of morphological filters based on guided contrasting.The idea of guided contrasting is that the local contrast (energy) of filtered test image is controlled by its local similarity with reference image.If we use the similarity estimation with local search, then such filters demonstrate the robustness relative to weak geometrical discrepancy of compared images.Another essential advantage of guided contrasting filters relative to traditional morphological projectors is the elimination of unstable image segmentation step.
The new change detection pipeline based on comparative filtering is proposed for analysis of bi-temporal long-range (spaceborne or airborne) remote sensing data.This pipeline contains comparative filtering on image pyramid, calculation of morphological difference map, binarization, extraction of change proposals and testing change proposals using local morphological correlation coefficient.We implement this pipeline based on both guided contrasting filters and morphological diffusion filters previously proposed in (Vizilter et al, 2014) for shape-based matching.Qualitative experiments with guided contrasting filtering in different change detection tasks demonstrate that they provide the reasonable scene change proposals and demonstrate the enough robustness relative to changes in lighting and other image acquisition conditions.Quantitative experiments on the public benchmark containing simulated aerial images demonstrate that the proposed pipeline is useful for change detection in long-range remote sensing data.The future work on proposed morphological schemes will be connected with development and implementation of new comparative filters and new change detection pipelines for different types of tasks.In particular, new guided contrasting filters can be obtained via different combinations of several smoothing procedures and local similarity estimators.The main experimental work will consist in massive testing of implemented comparative filters and corresponding change detection pipelines on the large datasets containing both simulated and real images.
x,y)  constany constant-valued (flat) image.The first and third conditions describe the smoothing properties.Second condition describes the exact matching property: filter should preserve the test image if it is equal to sample one.Due to the properties (9) the similarity of test g(x,y) and sample f(x,y) can be estimated like in MIA by the morphological correlation coefficient (MCC) of the form

Figure 1 .
Figure 1.The scheme of comparative filtering based on guided local contrasting.
Stages of change detection pipeline: a) reference image; b) test image; c) morphological difference map (MDM); d) binarized MDM; e) change proposals (regions of filtered binarized MDM); f) accepted change proposals.

A
lot of qualitative experiments with comparative filters based on guided contrasting are performed on a wide set of real images.Different types of scenes and image acquisition conditions are considered.Fig.3 and Fig.4 demonstrate some examples of morphological difference map forming based on comparative guided contrasting filtering.The first row contains reference images, second rowtest images; third rowmorphological difference maps.Different columns illustrate scenes and changes of different types.In the first column of Fig.3 the example of outdoor video surveillance change detection case is shown.Such case presumes the short-term changes in illumination conditions combined with presence of moving or appeared/disappeared objects.Second column of Fig.4 demonstrates the building construction case that requires comparison of buildings at different stages of construction based on images captured in different weather and season conditions from the close but not exactly the same viewpoint.In the first column of Fig.4 the example of changes in indoor (inoffice) close-range scene is shown.Some new objects appear on a table in a test image.The second column of Fig.4 demonstrates the most popular case of change detection task in remote sensing imagery (spaceborne or airborne).

Figure 3 .
Figure 3. Examples of morphological difference maps based on guided contrasting: first rowreference images; second rowtest images; third rowmorphological difference maps.

Figure 4 .
Figure 4. Examples of morphological difference map based on guided contrasting: first rowreference images; second rowtest images; third rowmorphological difference maps.
5. The obtained results are:  for change detection based on diffusion morphological filtering -Precision=0.61and Recall=0.6; for change detection based on guided contrasting -Example of simulated data from benchmark: a) reference image; b) test image; c) ground truth mask.