To address the spatial limitations and inefficiency of seismic damage assessment methods for traditional buildings, this study proposes a convolutional neural network (CNN)-based method for seismic damage feature extraction and recognition. The proposed method integrates border regression, mask generation, and feature classification to achieve the effective extraction and identification of the building damage characteristics. First, a comprehensive seismic damage dataset of buildings damaged by historical earthquakes in Yunnan Province was constructed using the post-earthquake unmanned aerial vehicle imagery of areas affected by the 2014 Ludian M6.5, 2014 Jinggu M6.6, and 2021 Yangbi M6.4 earthquakes, augmented with data enhancement techniques. Second, a CNN model (Mask R-CNN) was trained and optimized using this dataset to enable damage feature extraction and classification. Finally, validation with real earthquake cases demonstrated the method’s effectiveness, achieving an average recognition accuracy of 87.28%, a mean intersection over union (IoU) of 83%, and all image IoU values exceeding 0.5. This approach significantly enhances the efficiency and spatial resolution of post-earthquake building damage assessments.