With the rapid development of aerial imaging technology, aerial target detection has become a research hotspot with broad applications in intelligent transportation, agricultural monitoring, and military surveillance. However, the performance of aerial detection models is often degraded under adverse weather conditions such as fog, sandstorms, and low illumination. In such environments, aerial images typically suffer from reduced contrast and color distortion, which significantly affects the model’s ability to accurately identify targets. To this end, a Lightweight All-in-One Aerial Detection Enhancer Under Adverse Weather (LAD-Enhancer) has been proposed. The designed enhancer processes and restores degraded aerial images, thereby enhancing the detection model’s ability to perceive potential targets. Unlike conventional image restoration models, LAD-Enhancer integrates detection labels as additional supervision during training to ensure that enhancement is detection-oriented rather than purely visual. Furthermore, a three-stage training strategy and a Mixture of Experts (MoE) framework are employed to adaptively classify and process images captured under different degradation conditions. Experimental results demonstrate that, with an increase of fewer than 3K parameters, the proposed model significantly improves detection performance under adverse weather conditions while maintaining almost unchanged performance on clear-weather images.