An IMPROVED IMAGE REGISTRATION ALGORITHM FOR THERMAL INFRARED AND PANCHROMATIC IMAGE BASED ON GEOMETRIC STRUCTURAL PROPERTIES

: In order to improve the image registration accuracy of heterogeneous remote sensing images with large radiation differences, an improved image registration algorithm for thermal infrared and panchromatic images is proposed. This method uses the phase consistency of intensity and direction to construct a geometric structure feature descriptor called the Histogram of Oriented Phase Congruency (HOPC) for image registration. It employs the Curvature Scale Space (CSS) corner detection to concentrate and extract feature points in contour areas, and defines a similarity metric (called HOPCn) based on the Euclidean distance between the descriptors for high-precision heterogeneous images registration. Five sets of panchromatic and thermal infrared images were selected for verification. The results show that the improved algorithm can match the homonymy points around the image contour, and the Correct Matching Rate (CMR) is improved by 2.7% and the Root Mean Square Error (RMSE) is improved by 3.7% compared to the original algorithm. This proves that the proposed method has good robustness under conditions of large radiation differences.


INTRODUCTION
With the rapid development of sensor technology, remote sensing imagery has become an important way to observe the Earth's surface (Foody et al., 2002).Remote sensing imagery can be classified into satellite imagery (Hu et al., 2021), aerial imagery (Rahaghi et al., 2019), and so on, which depends on the different platform carrying the sensor.According to the classification of sensors, remote sensing imagery can be classified into Multispectral remote sensing (Bai et al., 2023), Thermal remote sensing (Masouleh et al., 2019), LiDAR (Light Detection and Ranging) (Dematteis et al., 2018), and so on.Different types of remote sensing imagery are widely applied in various fields such as environment monitoring (Zhang et al., 2023, Zhang et al., 2023), resource exploration (Wang et al., 2020, Joshi et al., 2016) and urban planning (Li et al., 2013, Gong et al., 2020), agriculture application (Valero et al., 2013, Deng et al., 2018), etc.For example, Landsat data, an multispectral remote sensing image, can be used to draw forest cover change maps (Song et al., 2015); optical and synthetic aperture radar (SAR) data can be used to monitor vegetation and soil moisture conditions (Ramat et al., 2023); ICESat-2's single photon laser altimetry data, combined with Google satellite imagery data, can obtain building heights (Zhao et al., 2023); and thermal infrared remote sensing imagery data can be used to evaluate urban heat islands (Weng et al., 2009).
Different types of remote sensing imagery provide different information.Different dimensions information of target can be obtained from thermal infrared and visible spectral remote sensing due to their different imaging principles.Thermal remote * Corresponding author sensing provides surface temperature information (Sobrino et al., 2004), while visible spectral remote sensing has higher resolution and can get object location and shape information (Blaschke et al., 2010).Stereo imaging can significantly improve the positioning accuracy of object (Gao et al., 2023).The fusion of optical and thermal infrared images can more effectively observe the earth's surface, further expanding the scope of remote sensing applications (Han et al., 2014).However, Prior to image fusion, it is necessary to perform image registration.
Image registration refers to aligning images of the same scene that are captured at different times, viewpoints, or by different sensors.(Zitova et al., 2003).It is a crucial step in remote sensing image processing and plays a vital role to improve the quality and reliability of remote sensing data (Ye et al., 2017).Scale invariant feature transform (SIFT) is a classic registration algorithm that was firstly proposed in the 1990s (Lowe D G., 2004) and has been widely used for registration of various images (Fischler et al., 1981).The algorithm can get good matching results for cross modality remote sensing images with consistent geographical locations and strong similarity in grayscale and structure (Yu et al., 2021).However, due to the different imaging methods and spectral characteristics of thermal remote sensing images and visible spectral remote sensing images, there is a large radiometric difference between thermal remote sensing images and visible spectral remote sensing images, and the registration effect of SIFT on cross modality remote sensing images is not satisfactory (Wang et al., 2022).Xi Shaoli et al. combined the SIFT operator with the Oriented Fast and Rotated Brief (ORB) operator, and by extracting a large number of feature points, they greatly improved the number and accuracy of matching points (Xi et al., 2020).Duan Linfeng proposed an image registration method based on Nonsubsampled contourlet transform (NSCT) contour extraction and consistency matching of the main direction, which combined the 128-dimensional SIFT descriptor with the CSS algorithm to improve the accuracy of image registration.Yu Zhi et al. invented an image registration method based on feature contour quadrilaterals, and the experime.ntalresults showed that this method can achieve more accurate contour positioning and efficient registration (Yu et al., 2021).Li Hongwei found contour feature points in the image based on the CSS algorithm, assigned a feature vector to each feature point for bilateral matching, and finally fused multiple feature information in the image (Li et al., 2022).Jiang Qian et al. improved the CSS algorithm to detect the edge of the image, then extracted the descriptor on the main direction of the feature point, performed image registration based on line consistency, and obtained the affine transformation relationship between the registered points to achieve image fusion.This method can achieve significant accuracy improvement compared with other registration algorithms (Jiang et al., 2020).Ye Yuanxin proposed an automatic registration algorithm based on geometric structure similarity, which can achieve high registration accuracy based on the structure of the image and can effectively solve the difficult registration problem of multimodal remote sensing images.
Due to the differences in sensors type, viewing angles, exposure times, and orbit heights between different satellites, such as visible spectral remote sensing satellites and thermal infrared remote sensing satellites, thermal infrared remote sensing and visible spectral remote sensing images captured in the same area often exhibit significant differences in radiation.Therefore, compared to image registration within the same satellite, thermal infrared remote sensing and visible spectral remote sensing images.matching from different satellites is more difficult, and traditional algorithms cannot achieve good matching results under these circumstances.To address these issues, this paper proposes a matching algorithm (Histogram of Orientated Phase Congreucy, HOPCSS) that combines the CSS detection operator with geometric structural feature descriptors.The algorithm first constructs the HOPC feature vectors of two images using phase congruency intensity and direction information, then extracts main image feature points using the CSS operator.After that, an affine transformation model is used to determine a search space of a certain size in the auxiliary image.The algorithm utilizes HOPCSS as a similarity metric for matching corresponding image points between the two images and finally employs the Random Sample Consensus (RANSAC) algorithm to eliminate mis-matching points, achieving the registration of thermal infrared remote sensing and visible spectral remote sensing images.

Improved Infrared and visual Image Registration with HOPCSS
The HOPCSS algorithm process is briefly introduced by an explanation of the geometric structure descriptors, CSS corners, Harris (Harris Corner Detector) corners, and similarity metric .
The principle and process of the HOPCSS algorithm is shown in the figure 1.Firstly, geometric descriptors are constructed for the primary and secondary images.Then, the CSS feature points are extracted from the primary image, and the corresponding image point are detected using HOPCn.After that, an affine transformation model is used for registration, and the registration points with large errors are removed.

Construction of Geometric Descriptors
Firstly, HOPC descriptors are constructed by using phase congruency intensity and orientation information, and HOPCn is used to establish a similarity metric for image registration.

A. Strength of Phase Congruency.
The strength of phase congruency of the image is calculated using the Log Gabor wavelet with multiple orientations and scales (Kovesi ., 1999), and its mathematical expression is as follows: Where ( , ) PC x y represents the strength of phase congruency value of the pixel in the image,  is a nonzero constant,   remains unchanged when   >0 and is 0 when   <0.T is the noise threshold, ( , ) no A x y is the amplitude of the pixel ( , ) xy on the log Gabor filter scale n and orientation o, which is multiplied by  The orientation of phase congruency represents the direction of the most drastic change in image features (similar to the gradient direction).It can be calculated using the odd-symmetric filter ( , ) no o x y of the Log Gabor wavelet.As shown in Figure 2.6, the odd-symmetric filter of the Log Gabor wavelet is similar to a derivative checking template, and its convolution result can represent the energy change of the image in a certain direction.When calculating phase congruency, multiple directions of Log Gabor odd-symmetric filters are used.Therefore, by projecting the odd-symmetric filter results onto the horizontal direction (Xaxis) and vertical direction (Y-axis), the energy in the horizontal and vertical directions can be obtained respectively.Then, by calculating the value of inverse tangent function, the phase congruency feature direction  can be obtained, and the calculation formula is as follows: ( )

D. similarity metric
Considering the significant difference in brightness between the thermal infrared remote sensing and visible spectral remote sensing images , while their geometric structure features are relatively similar, the Euclidean distance between the HOPC descriptors is used as the similarity metric (HOPCn) for corresponding image point matching.The expression is as follows: In the equation,

Improvements in Corner Detection
Due to the fact that the matching algorithm based on geometric structural similarity proposed by Yuanxin Ye (HOPC algorithm) uses the Harris corner detection operator , which has good repeatability under rotation and various lighting conditions and is commonly used for stereo matching and image database retrieval, but is susceptible to noise and loss of localization accuracy, and is also sensitive to scale changes (Harris et al., 1988).However, the Harris algorithm tends to detect unnecessary feature point in blank areas of the image due to uniform sampling of the image.
To address these shortcomings of the Harris algorithm, this paper adopts a new CSS corner detection algorithm, which includes the following main steps: After the above steps are completed, traverse all contours and detect the corners where the absolute value of curvature has a local maximum.These corners are regarded as the initial candidate corners.The contour curve is denoted as A, and its parameter form is given as: ( ) ( ( ), ( )) t x t y t = (6) For the curvature at point A on the edge, it can be obtained using the following equation: ( )

B. Removing Rounded Corners
The initial list of candidate corners is is the set of pixels on the contour.
N is the number of pixels on the contour, and ( , ) The initial list of candidate corners includes all local maxima, although the maximum curvature of a rounded corner may not be significantly different from that of its neighborhood, as shown in (a) and (c) of the figure 6. Figure (b) has a curvature that is similar to a rounded corner but has a lower absolute maximum.Its size is usually significantly larger than that of its neighborhood, and the overall or global curvature features of its neighbors typically change more abruptly, as shown in (d) of the figure 6.To eliminate rounded corners rather than obtuse corners using the global curvature features of the neighborhood, the region of support(ROS) is defined.
The ROS of a corner is defined by the contour segment delimited by the two nearest minimum curvatures of the corner.A local threshold is adaptively computed using the ROS of each corner, where u is the position of the candidate corner on the contour, L1+L2 is the size of the ROS centered at u, and R is a coefficient.
In equation ( 9), K is the average curvature of the ROS.If the curvature of a corner is greater than () Tu , then it is considered a true corner; otherwise, it is removed from the list.This eliminates rounded corners because the rate of curvature decrease in L1+L2 is faster for obtuse angles than in the ROS region.Thus, the average curvature of an obtuse angle is smaller than that of a rounded corner, and the absolute curvature of a rounded corner is often less than () Tu , while that of an obtuse angle is often greater than () Tu , even if their absolute curvatures are similar.According to the experience in this paper, the coefficient R should be set to 1.5.

C. Removal of False Corners
In general, a clearly defined corner should have a relatively sharp angle.If the opening angle of each corner on the contour is known, a clear distinction can be made between true corners and false corners by using the following formula to calculate the corner opening angle.
u is the position of the corner to be judged, 1 L is the number of points between u and

EXPERIMENTS
To demonstrate the feasibility of the matching algorithm HOPCSS proposed in this paper, five sets of thermal infrared and visible images with different time periods, sensors, and experimental loads were selected as experimental samples.The experiments were implemented using Matlab software in a PC environment with a memory of 16 GB under the Win10 operating system.The selected images involve multiple test areas, including satellite images and unmanned aerial vehicle (UAV) images.In response to issues such as the sensitivity of the Harris operator to scale changes, this paper compared and analyzed the accuracy and precision of the matching algorithm HOPCSS proposed in this paper and the HOPC algorithm.Due to the different resolution of thermal infrared and visible images, the resolution of the two types of images will be different in the same area.The thermal infrared images need to be processed by Gaussian filtering to ensure that the two images are registered at the resolution size.This paper selected the correct match rate (CMR) and root-mean-square error (RMSE) as the evaluation indicators for image registration accuracy.CMR is an effective evaluation indicator.When the registration results of the matching algorithm are better, CMR will increase accordingly.RMSE is another precision evaluation indicator, which quantifies the matching accuracy by calculating the root mean square error of the corresponding image point in the matching algorithm results, making the experimental results more credible.Figure 7 shows the registration results of the proposed improved algorithm and the original algorithm.Table 1 summarizes the registration results obtained by the two matching algorithms for the five sets of thermal infrared and visible images.To verify the feasibility of the proposed algorithm HOPCSS, five groups of cross modality images with different sensor were applied for experiments.In Figure 7, image pairs (1/2) consist of GF-7/ZY3-03 panchromatic images and thermal infrared images, while image pairs (3/4/5) consist of high-resolution images taken by unmanned aerial vehicles(UAV).
In this paper, the GF-7 satellite, a domestically developed submeter optical stereoscopic mapping satellite, was successfully launched and put into operation in Taiyuan, China on November 3, 2019.the rear-view images from the panchromatic band (0.45~0.90 m  ) of the GF-7 satellite are used, with a resolution of 0.65 m .the ZY3-03 satellite, launched and put into operation in Taiyuan, China on July 15th, is a mapping satellite.the panchromatic images from the ZY3-03 satellite are used, with a resolution of 2.072 m .the thermal infrared remote sensing images have a resolution of 3 m , while the infrared resolution of the unmanned aerial vehicle (UAV) is 0.1 m .
In this paper, the improved algorithm HOPCSS, which combines CSS corner detection with phase congruency, was compared with the HOPC algorithm.By the matching results of multiple image groups, it was found that when the number of feature points detected by Harris was more than that detected by CSS, HOPCSS algorithm had higher matching accuracy than HOPC.In the registration results of HOPCSS, it can be seen that the corresponding image point are concentrated on the edge of the contour, while the feature points of HOPC are relatively evenly distributed.This is because the Harris operator used in HOPC algorithm can detect feature points even in blank areas of the image, which can lead to registration errors.In contrast, CSS corner detection detects the maximum curvature point of the contour and removes rounded corners and false corners, making corner detection more concentrated on the edge of the contour and avoiding corner extraction in the blank areas of the image, thereby effectively reducing noise interference in the image.In summary, the improved algorithm enhances registration accuracy and reduces a large number of invalid corresponding image point compared to the original algorithm.
In image group (1), the original algorithm and the improved algorithm extracted 1400 and 1373 feature points, respectively.However, the HOPCSS algorithm had more accurate matching points than the HOPC algorithm.It is worth noting that the matching points detected by the HOPCSS algorithm are distributed mainly at the edges and areas with significant curvature changes, with few in blank areas.Compared with image pairs (3/4/5), the improvement in satellite image registration was little, while the improvement in UAV image registration was more significant.This is likely due to the resolution of the thermal infrared and panchromatic images by the UAV was higher and without Gaussian blur, resulting in better registration effects.Compared with the HOPC algorithm, the proposed method performed better in matching results.
Combining the table 1 and the figure 7, it can be seen that the proposed method has certain robustness.In summary, this paper studied the registration of thermal infrared and visible images, and the results showed that whether in satellite or UAV images, the HOPCSS algorithm had a significant increasement in the number of corresponding image point compared with the HOPC algorithm, and had significant advantages in matching accuracy.

CONCLUSION
This paper proposes an improved image registration algorithm based on geometric structure attributes for thermal infrared and visible images with large radiometric differences.Firstly, the HOPC feature vector is constructed based on the phase congruency principle, and then the main image CSS feature points are extracted.The CSS corner detection algorithm is combined with geometric structure descriptors to propose the HOPCSS algorithm for the registration of images.The advantages of CSS corner detection are expressed, and the following conclusions are drawn: (1) Through analysis of selected satellite images, unmanned aerial vehicle (UAV) images, and professional camera images, it was found that when detecting a smaller number of feature points, the proposed algorithm in this paper had more accurately matched points and overall higher accuracy than the original algorithm.
(2) The registration results show that many of the corresponding image points matched by the original algorithm are distributed in the blank areas of the images, resulting in the generation of invalid corresponding image points.In contrast, the corresponding image points by the proposed algorithm in this paper are more concentrated around the contours of the images, reducing the number of invalid corresponding image points.From this perspective, the improved algorithm proposed in this paper can to some extent resist noise interference.
(3) Through the analysis of the principle of matching algorithms, it is found that in order to automatically register images with large radiation differences, it is necessary to start from the structural similarity of the two images and then perform matching.
The algorithm proposed in this paper still has room for improvement and areas that require further refinement, providing potential research directions and opportunities for improvement.
(1) When using the HOPCSS algorithm proposed in this paper for the registration of heterogeneous remote sensing images, it is necessary to apply Gaussian blurring to the images to ensure that both images are at the same resolution.The matching performance may be affected when the resolutions of the two images are different.Future research can explore matching algorithms for heterogeneous remote sensing images at different resolutions.
(2) The algorithm proposed in this paper cannot handle heterogeneous remote sensing images with significant rotation angles.Future research can explore matching algorithms for heterogeneous remote sensing images with large rotation angles.

Figure 1 .
Figure 1.Registration Process of HOPCSS Algorithm E x y is a local energy function defined as follows: are the even- symmetric and odd-symmetric filter signal response values ofLogGabor , respectively.B. Orientation of Phase Congruency.The Log Gabor filters can only obtain the strength of phase congruency values of the image, and geometric structure descriptors cannot be constructed solely based on phase congruency intensity values.It is also necessary to provide the orientation of phase congruency .

V
represent the HOPC descriptors of the template windows A and B, respectively.

Figure 4 .Figure 5 .
Figure 4. CSS corner detection process diagram.A. Canny edge detection for edge extraction.The Canny algorithm is an excellent image edge detection algorithm(Canny., 1986), but there is a problem of missing pixels in the process of edge extraction, which leads to the the contour.If the starting point and ending point of the contour are close, it is a closed contour; otherwise, it is an open contourthreshold T is used to determine whether two endpoints are close enough.In this paper, the value of T is set to 4. (a) Rounded corner (b) Obtuse angle (c) Curvature map of rounded corners in (a) (d) Curvature map of obtuse angles in (b) Figure 6.Difference between rounded corners and obtuse angles.

Figure 7 .
Registration of Experimental Image Data.

Table 1 .
Evaluation metrics for different matching methods.