This paper deals with the effect of the Rayleigh damping matrix on site response analysis in the time domain. We introduce the fundamental theory of time domain analysis and summarize several methods for constructing a damping matrix. In our study, we to developed a series of numerical models to simulate wave propagation through an elastic medium after using the finite difference method to consider the intensity and spectral characteristics of the input motions. We adopted a full Rayleigh damping and selected five methods to determine the high frequency. The results show that damping models have a significant influence on acceleration and spectral acceleration, and their degree of influence is related to the frequency characteristics of the input motions. When the fundamental site frequency is close to the predominant frequency of the input motions, it is feasible to determine the damping matrix based on this consideration only. However, the peak ground acceleration and spectral value are underestimated when the predominant frequency of the input motion is high or far higher than the fundamental site frequency. In time domain analysis, damping considerations should include the site frequency and frequency characteristics of the input motions.