Sky-NeRF: Learning 4D Cloud Topography in a Dynamic Neural Radiance Field

Abstract. We present Sky-NeRF, a novel method for cloud topography estimation based on Dynamic Neural Radiance Fields. Similar to NeRF, we propose to model the 3D structure of clouds as a radiance field, encoded in the parameters of a neural representation. Our goal is to reconstruct the 3D geometry, appearance, and motion of the cloud using a stereo-video of high-resolution top of the atmosphere radiance images. In this paper, we evaluate a novel way of modeling the dynamic behavior of clouds, with the goal of extracting added-value physical information regarding the cloud such as advection speed and direction, velocity field and cloud trajectories. We investigate how to include a simple physical prior, advection, into the learning system and evaluate its impact. Our results show that Sky-NeRF is able to provide a more complete 4D reconstruction than traditional stereo-matching-based algorithms. Moreover, thanks to a physics-based interpolation, Sky-NeRF is able to generate coherent new images from unseen viewing angles, and at any time between the observed frames.