In this paper, we use two-dimensional numerical analysis to study the importance of establishing a reasonable damping matrix in the seismic response analysis of high-gravity dams in the time domain. First, we selected four high-gravity dams of different heights as examples and determined their seismic responses under horizontal excitation by three seismic waves using six different damping matrices. Then, the reasonability of the different damping matrices was assessed by comparing their solutions in the frequency domain. The results indicate that the seismic response in the time domain of a gravity dam with a height greater than 250 m is a dynamic analysis problem of a long-period system, and attention should be paid to the approach used to model the damping matrix. In this case, we suggest using a Rayleigh damping matrix with two frequency parameters rather than a mass-proportional damping matrix or a stiffness-proportional damping matrix with one frequency parameter. To accurately analyze the seismic responses of a dam, both the fundamental frequency of the dam and the frequency characteristics of the input seismic wave should be considered.