Remote sensing change detection (RSCD) aims to identify pixel-wise surface changes from co-registered bi-temporal images. However, many deep learning–based RSCD methods rely solely on change-map annotations and underuse the semantic information in non-changing regions, which limits robustness under illumination variation, off-nadir views, and scarce labels. This paper presents ChangeDINO, an end-to-end multiscale Siamese framework for optical building change detection. The model fuses a lightweight backbone stream with features transferred from a frozen DINOv3, yielding semantic- and context-rich pyramids even on small datasets. A spatial–spectral differential transformer decoder then exploits multi-scale absolute differences as change priors to highlight true building changes and suppress irrelevant responses. Finally, a learnable morphology module refines the upsampled logits to recover clean boundaries. Experiments on four public benchmarks demonstrate that ChangeDINO achieves strong accuracy and robustness under cross-temporal appearance variations, yielding cleaner building boundaries with improved data efficiency.