A two-dimensional model of a buildings-alluvial valley system is established using the finite-element method, and the viscoelastic artificial boundary is applied on the truncated boundary of the soil body. By comparing and analyzing the seismic responses of the buildings-alluvial valley system and the single alluvial valley both in the frequency domain and the time domain, the dynamic interaction between alluvial valley and buildings under earthquake is studied. The results show that because of the resonance effect under low-frequency incident waves, the surface displacement response amplitude of the buildings-alluvial valley system is larger than that of the single alluvial valley in some areas. As the frequency of the incident wave increases, the buildings will have a significant weakening effect on the seismic response. In addition, the impact of the buildings on the valley is also related to the building height and building spacing, and the responses at different locations are significantly different. The numerical results can provide a theoretical basis for the setting of fortification intensity in alluvial valleys and the seismic design of projects.