Integrating Viewing Direction and Image Features for Robust Multi-View Multi-Object 3D Pedestrian Tracking

Abstract. Recently, there has been growing interest in the development of 3D multi-view, multi-object detection and tracking models (MV-MOD and MV-MOT), resulting in significant methodological advances. However, many of these developments do not address the critical challenge of generalization across different camera constellations, i.e., having camera constellations that differ between training and testing, limiting their effectiveness in real-world applications. A key factor often overlooked is the influence of the direction of the optical axis during image capture, which is not adequately propagated in the model. In this work, we propose a novel convolutional neural network-based method for 3D MV-MOD and MV-MOT that enhances generalization by incorporating the direction from which the images were captured as an additional input to this network. For each image, this directional information is combined with the 2D features extracted from that image, before 3D features are computed, using the 2D features from all images. We empirically evaluate the performance of the proposed method on the real-world Wildtrack dataset, demonstrating the effectiveness of the proposed approach.