Understanding pedestrian behaviors is the foundation of simulation for space planning. However, conventional behavior modeling methods are insufficient for learning detailed interactions, and deep learning methods often lack interpretability. This study aims to develop a pedestrian trajectory modeling approach based on discovering causal relationships among pedestrians. The proposed method consists of two parts: analyzing causal relationships among pedestrians using statistical causal discovery methods and predicting trajectories using attention-based deep learning methods. The first part employs a semi-parametric method to identify the causal relationships underlying observed pedestrian behavior and construct a spatial-temporal graph based on these causal relationships. The second part primarily uses the graph attention network to learn interactions among pedestrians. The experimental results demonstrate that the proposed method achieves a good balance between prediction accuracy and interpretability, while also identifying limitations, including at low-density scenes and due to causal model assumptions.