Point clouds acquired by light detection and ranging (lidar) and photogrammetry technology (e.g., structure from motion/multi-view stereo-SfM/MVS) are widely used for various applications such topographic mapping due to their high resolution and accuracy. To generate a digital elevation model (DEM) or extract other features in the data, the ground points and non-ground points usually need to be separated first. This process, called ground filtering, can be tedious and time consuming as it requires substantial manual effort for high quality results. Although many have developed automated ground filtering algorithms, very few have the versatility to process data acquired from different scenes and systems. In this paper, we propose a versatile ground filter based on multi-scale voxelization and smooth segments, named Vo-SmoG. The proposed method introduces a novel voxelization approach, followed by isolated voxel filtering, lowest point filtering, local smooth filtering, and ground clustering. The result of the Vo-SmoG ground filtering is a classified point cloud. The effectiveness and efficiency of our method are demonstrated qualitatively and quantitatively. The quantitative evaluation consists of both point-wise and grid-wise comparisons. The recall, precision, and F1-score are over 97% in terms of classification while the root mean squared error (RMSE) of the DEM is within 0.1 m, which is on par with the reported vertical accuracy of the tested data. We further demonstrate the versatility of the Vo-SmoG via large-scale, real-world datasets collected from different environments with mobile laser scanning, airborne laser scanning, terrestrial laser scanning, uncrewed aircraft system (UAS)-SfM, and UAS-lidar.