Cultural heritage sites face growing threats from environmental factors and human activities, highlighting the need for efficient techniques to monitor and preserve their structural integrity. While advanced machine learning models, such as Segment Anything Model (SAM), have shown success in areas such as healthcare, their potential for cultural heritage conservation remains largely unexplored. In this research, we propose an automatic decay detection and visualization framework by combining advanced segmentation techniques with 3D reconstruction methods. We fine-tune SAM and integrate it with You Only Look Once (YOLO) to create a fully automatic, real-time segmentation framework that offers strong generalization for identifying unseen decay types. By incorporating Structure from motion (SfM) and multi-view stereo (MVS), the framework produces 3D models that highlight decay regions, providing a robust tool for structural assessment and visualization. Through both quantitative and qualitative evaluations, we show that our approach outperforms several state-of-the-art models, demonstrating its effectiveness in identifying and visualizing stone decay. Our results contributes to heritage preservation by providing a novel, scalable solution for real-time monitoring of cultural heritage sites.