Squatter structures have been a serious threat to human safety and health for a long time. And monitoring their changes is important to facilitate government management of squatters. However, existing methods are still not automatic, accurate and fast enough to meet the actual needs of practical applications. In this paper, we propose a novel deep learning-based method to detect squatter structure changes from bi-temporal remotely sensed (RS) images and digital surface models (DSMs). The proposed convolutional neural network (CNN) takes the advantages of the spectral information from high resolution image and the height information from the DSM, so as to detect changes more accurately in type and height of squatter structures. Moreover, we create a data set for deep learning model training, covering a variety of squatter structures in Hong Kong. Compared with three existing representative methods, Our model performs the best, with Kappa of 0.6786 and 0.6458 in the detection results of the two test regions, respectively, which indicates that it has application potential.