DETECTION AND CORRECTION OF CHANGES IN EXTERIOR AND INTERIOR ORIENTATIONS WHILE ESTIMATING 3-D OBJECT DEFORMATIONS FROM MULTIPLE IMAGES WITH WEAK OR STRONG IMAGING GEOMETRY

The paper deals with estimation of 3-D object deformation from multiple images initially in fixed positions with weak or strong imaging geometry. A new method is proposed to detect automatically if the exterior or interior orientations (rotations, translations, focal length, principal point) of one or several images have changed and which image or images contain the error, when the object deforms at the same time. The method is based on comparing novel feature vectors computed for each image from changes in the image coordinates of the object points and from residuals derived from the collinearity equations. Bundle adjustment is performed to simultaneously estimate the deformation of the object and to correct the changed orientations of the images. The rigidity needed in the weak case is obtained by approximating the deformation by a novel shape function containing parameters the values of which are estimated during adjustment. Test results with synthetic data show that even rather small changes in one orientation parameter of one image can be detected with high confidence. Weak imaging geometry allows to detect smaller changes than the strong one. The closer an initial approximation of deformation is available, the higher is the probability of correct detection. Subsequent correction of changed orientations and estimation of deformation may provide a high accuracy of 1:140000 of the object dimensions for both weak and strong imaging geometries, when the noise level in the image measurements is 0.1 pixel. Experiments with real data illustrate the good performance of the methods.


INTRODUCTION
We consider the case where multiple cameras mounted in fixed positions and having a weak or strong imaging geometry monitor an object with special target points attached on the surface of the object.Long-term monitoring aims to detect changes in the image coordinates of the target points and to interpret the changes observed.Possible reasons for changes in the image coordinates include that the object has experienced a deformation or something has changed in the measurement system such as the exterior or interior orientation of one or several images has changed.The main question addressed in this paper is to develop an automated method to distinguish between object deformation and changes in image orientations when they occur at the same time.Moreover, the objective is to correct the changed orientations simultaneously with estimation of the object deformation when the imaging geometry may be either weak or strong.Weak imaging geometry (cameras close to each others with respect to depth to the object) may be the only choice when, e.g., restrictions set by the measurement environment hinder obtaining strong imaging geometry.
It is well-known that errors in the exterior or interior orientation of some image result in large residuals in the image coordinates when 3-D object points are reconstructed from the image measurements.It is obvious, however, that the large residuals spread also into the other images having correct orientations.It may be thus difficult to identify automatically which image contains the error.In fact, we have found no published papers where automatic detection of changed orientations would have been investigated.
For strong imaging geometry, it is common to perform selfcalibration during 3-D reconstruction (Maas, 1998).For weak imaging geometry, additional information is typically incorporated to strengthen the geometry.These include adding stable control points behind the deforming object and utilizing all the measurements of moving target points during deformation (Fraser and Riedel, 2000), using a constraint that object deformation is restricted to a particular direction (Albert et al., 2002, Jiang andJauregui, 2010), or considering isometric deformations of a paper-like surface with a constraint that the rulings of the paper remain parallel (Taddei and Bartoli, 2008).Fraser and Riedel (2000) adjust also the exterior orientations during 3-D reconstruction while in the other works mentioned, the orientations are kept fixed although Jiang and Jauregui (2010) perform self-calibration before the actual deformation measurement.Similar ideas to strengthen the geometry appear also in 3-D trajectory reconstruction with constraints on the way the points can move, see e.g. Park et al. (2010).
In this paper, we present a new method to detect automatically if the exterior or interior orientations (rotations, translations, focal length, principal point) of one or several images have changed and which image or images contain the error, when the object deforms at the same time.The method is based on comparing novel feature vectors computed for each image from the changes in the image coordinates of the target points and from the residuals derived from the collinearity equations.We perform bundle adjustment to estimate the deformation of the object, with the help of a novel shape function, simultaneously with the correction of the changed orientations of the images.This latter method extends our previous work in (Jokinen and Haggrén, 2011), where we assumed that the orientations stay fixed and deformation is only estimated.The proposed method is suitable for both weak and strong imaging geometries.The rigidity needed in the weak case is obtained by approximating the deformation by a suitable shape function containing parameters the values of which are estimated during adjustment.Consequently, it is assumed that some knowledge of the functional form of deformation that is expected to occur is available.Such information may be obtained from analytical calculations, numerical simulations, previous experience on similar situations, or experience gathered during long-term monitoring of the same object.The assumptions include also that the 3-D coordinates of the target points on the object surface before deformation and the 2-D image coordinates of the target points before and after deformation are available, and that all the cameras are synchronized when capturing the images.Possible application areas include quality control during building construction, where the deformations are often smooth and can be described by continuously differentiable shape functions and moreover, where the cameras may be difficult to place optimally due to the ongoing construction work.
The paper is organized as follows.The methods for detection and correction of changed orientations and estimation of object deformation are presented in Section 2. The methods are tested with two synthetically generated data in Section 3, including finding out the smallest detectable change in orientation, verifying the performance of the methods for different number of changed orientations and for weak and strong imaging geometries, and evaluating the accuracy of the estimated deformation.Test results with real data are also presented in Section 3. The paper ends with conclusions in Section 4.

Detection of changed orientations
Assume that we have a set of target points r i ,i=1, , N , given on the surface of the object in an object coordinate system.When the object experiences a deformation, the points move to r i ' =r i  r i .The object is monitored by K≥3 cameras with exterior orientations given by rotation matrices R k and translation vectors t k and interior orientations given by focal lengths c k and locations of principal points  x 0 k , y 0 k  for k =1,, K .The lens distortions are assumed corrected.The image coordinates of the target points before deformation are given by the collinearity equations for i=1,, N , k =1, , K , where R 1 °k , R 2°k , R 3°k denote the first, second, and third row of the rotation matrix, respectively.After deformation, we observe the changed image coordinates  x i k  ' , y i k  ' and may also have approximate values  r i  appr for the deformations.Inserting these in Eq. 1, multiplying by the denominator, and rearranging the terms, we then define In general, these quantities values measure essentially the discrepancy between the changed image coordinates  x i k  ' , y i k  ' and the image coordinates which were obtained if the deformation were  r i  appr and the orientations unchanged.Other distinctive features we consider are the changes in the rectified image coordinates which measure the change to the situation before any deformation and any changes in orientations.The images are rectified to a common plane according to the current estimates of the exterior and interior orientation parameters.The coordinate systems in the rectified images are oriented so that the image x axes of all the rectified images are parallel pointing to the same direction and the same is holding also for the image y axes.Let  x i kr , y i kr denote the changes in the image coordinates of point i in a rectified image k.A feature vector of 4N elements, where N is the number of target points, is then computed for each rectified image.The first two elements for each point are the magnitude and direction of the change in the rectified image coordinates given by where atan2 denotes the four quadrant arc tangent of  y i kr /  x i kr with a range of (-π, π].We prefer here polar coordinates to Cartesian ones since it allows to separate small changes in absolute value which differ in the direction of the change.The other two quantities for each point are u i k ,v i k .These four quantities are scaled by dividing each quantity by its maximum absolute value over i=1, , N and k =1, , K and the feature vector of image k is thus given by The difference d q n ,q k  of two feature vectors is a 4N vector, the elements of which are obtained as a simple subtraction of the corresponding elements of the original vectors for all other elements except for the angles where the overlined arc cosine denotes its principal branch with a range of [0, π].The distances between the feature vectors of different images are evaluated and a discrepancy measure D k is computed for each image, as a sum of distances to the feature vectors of other images divided by the maximum sum over all the images as follows The images having a discrepancy larger than an adaptive threshold are classified as changed orientation.The threshold is given by the median of discrepancies plus the standard deviation of discrepancies if the mean of discrepancies is not larger than 0.8.Otherwise, all image orientations are considered unchanged.Note that D k obtains values between zero and one.

Correction of changed orientations and estimation of object deformation
We consider the case where the object deformation can be expressed with the help of a shape function f : ℝ 3 ×ℝ m ℝ 3 depending on m parameters a as follows  r i = f r i ;a for i=1,, N .Instead of estimating the deformations at each point, it is enough to estimate the values of the parameters a and then evaluate the point wise deformations using the shape function.This makes the problem solvable not only for strong but also for weak imaging geometry.The merit function F minimized in the bundle adjustment constrained by the shape function is given by where vector b includes the exterior and interior orientation parameters of images classified as changed orientation (9 parameters for each image of changed orientation), u i k , v i k are given in Eq. 2 with  r i '  appr replaced by r i  f  r i ; a , and w i k are non-negative weights which can be set inversely proportional to the variance of image measurements at each point.The minimization problem is solved using the Levenberg-Marquardt algorithm.The algorithm requires initial values for the estimated parameters a and b, which may be obtained at the same time as the functional form of the shape function is determined as concerns parameters a while for parameters b, the current values are used as initial estimates for the changed ones.

Imaging configuration and object deformation
The methods were tested first with synthetic data involving eight convergent cameras located equidistantly on the circumference of a circle of radius s and center at (0,0,10) meters on a plane Z=10 meters in an object coordinate system at a distance of about 10 m from the object, which in turn was located near the ground plane Z=0 meters.The focal lengths of the cameras were 10 mm, the principal points were located in the middles of the images, the images were assumed to be free of lens distortions, and the pixel sizes of the images were 10×10 μm 2 .Normally distributed noise of zero mean and standard deviation of 0.1 or one pixel was added to all the image measurements made.Originally a sinusoidal object surface covering an area of 10×10 m 2 with 441 target points (forming a grid of 21×21 points) was deformed so that the X and Y coordinates were deformed by a sine function and the Z coordinates by a combination of multivariate exponential and sine functions in the first data and by a multivariate polynomial of fourth degree in the second data.More specifically, the deformations T were given by for the first data and for the second data with a parameter vector a=[ a 1  a 8 ] T in both of the cases.The related shape function f depended thus non-linearly on parameters a in the first data and linearly in the second data.

Smallest detectable change in orientation
It was first studied how small the change in the exterior and interior orientations can be that it is still detectable, when the object deforms at the same time according to Eq. 7. Hundred randomized trials were performed by having for each trial, different true values of parameters a, different initial values of parameters a given by perturbing the true values by plus or minus five percent, one randomly selected image (out of eight), the orientation parameters of which were changed by a given partly randomized amount (see below), and different randomized noise added to the image observations of the target points.The results presented in Table 1 should be interpreted as follows.For example, for weak imaging geometry (s = 0.1 m) and a noise level of  =0.1 pixel in the image measurements, 90 percent or better correct detection rate was achieved if one of the following changes in one image k was made: focal length ±0.13 mm, position of the principal point 9 pixels in a randomly selected direction in the image plane, rotation angle  k by ±0.4 degree around the X k axis of a camera centered coordinate system, rotation angle  k by ±0.4 degree around the Y k axis, rotation angle  k by ±0.7 degree around the Z k axis (optical axis of the camera), or pose of the projection center 0.09 m in a randomly selected direction in the 3-D space.The results indicate that smaller changes in orientation can be detected when the imaging geometry is weak than when it is strong while there are not much differences between the results for the noise levels of 0.1 and one pixel.
Table 1.Smallest detectable changes in the orientation of one image out of eight for weak ( s = 0.1 m) and strong ( s = 5 m) imaging geometry and for different noise levels  in the image observations with a rate of correct detection of at least 90 percent and perturbation of the shape function parameters by 5 percent.imaging geometry weak strong The rate of correct detection depends on the closeness of the initial approximation of the deformation to the true deformation.This was investigated by perturbing the shape function parameters by 0 to 15 percent from the true values and running the detection algorithm for the smallest detectable changes in the orientation of one image out of eight according to Table 1 for weak and strong imaging geometries with the noise level of 0.1 pixel.The results in Fig. 1 show that the percentages of correct detections decrease as the difference between the initial approximation and true deformation increases.The decrease is faster for weak than for strong imaging geometry.

Detection performance for different number of changed image orientations
It was studied how well the detection algorithm works when the orientations of more than one image have changed.The results for deformations according to Eq. 7 (exp.+ sin.data) and Eq. 8 (polynomial data) with weak (s = 0.1 m) and strong (s = 5 m) imaging geometries and various numbers of changed orientations are shown in Table 2. Initial approximations for the deformations were given by changing the shape function parameter values by five percent from the true ones.The results prove that for weak imaging geometry, a high detection rate is achieved up to three images of changed orientations while for strong imaging geometry, the percentages of successful detections are high up to two images and somewhat lower for three images of changed orientations.When there are no changes in orientations, the algorithm gives a false positive detection in 5 and 18 percent of trials out of the total of 100 trials performed for the first and second data, respectively.

Accuracy of estimated deformation
The algorithm for correction of changed orientations and estimation of object deformation was tested for the various cases listed in Table 2, with all the weights equal to one in Eq. 6.The RMS distances between the estimated and true deformation at the target points in correctly detected and successfully converged trials on the average are low for all cases.The convergence of the iterative algorithm is good as 99 percent for the first data and 100 percent for the second data, of correctly detected trials converge successfully when the initial values for the shape function parameters are within 5 percent from the true ones.For the second data, the algorithm convergences to a correct solution even from an initial approximation for the deformation of no deformation at all, but then the percentages of correctly detected changed orientations are lower.For both data, the accuracies in the weak cases are almost as high as in the strong ones.Figures 2 and 3 further illustrate the estimated deformation and its difference to the true one for typical cases of both data with weak imaging geometries and with two images having changed orientations out of eight.
For comparison to a standard approach without any detection of changed orientations, the bundle adjustment was also carried out with the interior and exterior orientation parameters of all the images and the shape function parameters as unknowns.The results indicated that the algorithm did not converge properly for the weak cases.For the strong cases of the first data, only 65 percent of trials converged but with a large RMSE of 120 mm.For the strong cases of the second data, 100 percent convergence was achieved with a RMSE of 0.21 mm.The detection of changed orientations is thus important for accurate deformation estimation.

Testing with real data
Real data testing was carried out with four calibrated digital cameras (two 10 Mpix Nikon D200, one 6 Mpix Nikon D100 and one 6 Mpix Nikon D50) having a strong imaging geometry.All the cameras were located side by side at a distance of 1.4 ... 1.5 m from an object and the distance between the cameras located farthermost from each other was about 1.0 m.The object being monitored was a plate with 24 circular targets attached on it as shown in Fig. 4. The image coordinates of the centers of the circular targets were measured from all the images using iWitness software, which thereafter provided also the exterior orientations of the cameras and the 3-D coordinates of the target points before deformation.A deformation was then caused on the object by turning two screws fixed to the middle of the plate and to the upper edge of the plate, respectively.Besides deformation, the exterior orientation of one of the cameras was perturbed by turning the camera into different orientations on a flexible and rather unstable tripod.After these changes, the image coordinates of the targets were measured again and the detection algorithm was performed with an initial estimate for the deformation being no deformation at all.The discrepancy measures for the images are shown in Fig. 5 together with the adaptive threshold as a dashed red line.The figure correctly points out the camera number two, the exterior orientation of which had been perturbed.
The changed values of the exterior orientation parameters were then estimated together with object deformation using bivariate polynomials of second degree in X and Z to model the deformations in the X and Z directions and a bivariate polynomial of fourth degree in X and Z to model the deformation in the Y direction with all the 27 coefficients of the polynomial terms as parameters of the shape function in an object centered coordinate system, where the origin was in the middle of the plate and the Y axis was parallel to the normal of a plane fitted to the four stable points next to the plate.The result is illustrated in Fig. 6.For comparison, the deformed surface was also reconstructed using a traditional bundle adjustment method with the exterior orientation parameters of the changed camera and all the object point coordinates as unknowns.The RMS distance between the target points after deformation given by the proposed and traditional methods was 0.08 mm.
Figure 6.In red, the surface after deformation and correction of the exterior orientation of one image using the proposed method based on a shape function; In dark blue, the deformed surface reconstructed using the traditional method; In cyan, the surface before deformation given by iWitness software.
Further tests were carried out with the four cameras moved closer to each other (distance between the farthermost cameras was 44 cm).Besides deforming the plate, the interior orientation of one camera was perturbed by changing slightly the focal length of the camera into three different positions.When the camera was touched, its exterior orientation also changed slightly.All the three changes in the focal length including minor changes in the exterior orientation could be successfully detected.The exterior orientation of the changed camera was then perturbed more roughly into three different orientations and all the perturbations could be again correctly detected.The RMS distances between the target points after deformation estimated by the shape function based method and by the traditional one with simultaneous correction of the changed focal length and exterior orientation were 0.12 ... 0.13 mm in all the cases.

CONCLUSIONS
The paper proposed a new method for automatic detection of changed exterior and interior orientations of images simultaneously when the object, being monitored with a multiple camera system, experienced a deformation.Bundle adjustment was performed for correction of the changed orientations and estimation of object deformation using a novel idea of a shape function.All the methods were intended for both weak and strong imaging geometries.
Testing with synthetically generated data of eight images showed that even rather small changes in one orientation parameter of one image could be detected with high confidence.
With weak imaging geometry, smaller changes were detectable than with a strong one, when the initial approximation for the deformation was rather close to the true one.The closer the initial approximation was, the higher probability of correct detection was achieved.Up to three images with changed orientations could be detected with high confidence for weak imaging geometry while for strong imaging geometry, the percentages of successful detections were high up to two images and somewhat lower for three images of changed orientations.Subsequent correction of changed orientations and estimation of deformation showed that a high accuracy of about 1:140000 of the object dimensions could be achieved for both imaging geometries with a noise level of 0.1 pixel in the image measurements, but rather close initial values for the shape function parameters were needed in the first data involving nonlinear dependencies.Experiments with real data illustrated the good performance of the methods with strong imaging geometry and changed exterior orientation and/or focal length of one camera out of four.
Current limitation of the proposed detection method is that it only detects which image contains an error in orientation, but it does not infer which of the nine orientation parameters have changed.The method for correction of changed orientations and estimation of deformation is limited to cases where the deformation can be expressed with the help of a suitable shape function.Note that the detection method does not need a shape function although it is used to generate an initial approximation for the deformation in the tests.

Figure 1 .
Figure 1.Percentages of trials where the image of changed orientation was correctly detected as a function of perturbation in the shape function parameters for various changes in the orientation with a) weak and b) strong imaging geometry.
Figure 2. a) Estimated and true deformation in red and blue, respectively, for the deformation in Eq. 7 with weak imaging geometry.b) Difference between the estimated and true Z coordinates of the deformation in millimeters.a)

Figure 4 .
Figure 4.The plate being monitored with four cameras.

Figure 5 .
Figure 5. Discrepancy measures for the four images and the adaptive threshold as a dashed red line.

Table 2
. Percentages of trials where all images of changed orientations were correctly detected and RMSE of estimated deformation at the target points in successfully converged correct detections for a noise level of  =0.1 pixel and for a moderate change in the orientations: focal length 0.2 mm, principal point 10 pixels, rotation angles 2 degrees each, and pose of projection center 0.1 m.