Occlusion handling in spatio-temporal object-based image sequence matching

Abstract. Dynamic photogrammetry is an established method for acquiring 3D information of deforming objects or dynamic scenes in various close-range applications. A crucial impact has occlusions caused by object deformations, obstacles or camera movements. Temporal occlusions are highly application-specific and sometimes difficult to predict, resulting in a significant reduction of reconstruction quality or the aborting of image sequence processing. Previous approaches usually model such occlusions as semantic information and consider them using image masks. However, generating these image masks requires complex methods and extensive training data. Due to the unpredictability of the complexity and movements of dynamic scenes, generating training data is challenging in many applications. Therefore, this paper proposes an alternative modelling approach, which can be part of a spatio-temporal matching process. Based on the characteristic high redundancy, occlusions can be detected using robust estimation methods and considered in the optimisation. Therefore, no information about the occlusions and further processing steps are necessary. We evaluate our approach with synthetic and real data of an industrial application regarding the accuracy and ability to detect occlusion simultaneously. The evaluation of the proposed approach shows that the impact of occlusion can be eliminated, and the quality of the results is comparable to conventional methods.