In this work, we present a surface-based method to extract the contours of planar building elements in the urban scene. A bottom-up segmentation method that utilizes global graph-based optimization and supervoxel structure is developed, enabling an automatic and unsupervised segmentation of point clouds. Then, a planarity-based extraction is conducted to segments, and only the planar segments, as well as their neighborhoods, are selected as candidates for the plane fitting. The points of the plane can be identified by the parametric model given by the planarity calculation. Afterward, the boundary points of the extracted plane are extracted by the alpha-shape. Optionally, line segments can be fitted and optimized by the energy minimization with the local graphical model. The experimental results using different datasets reveal that our proposed segmentation methods can be effective and comparable with other method, and the contours of planar building elements can be well extracted from the complex urban scene.