Special bridges, like skewed bridges, are increasingly used at highway intersections and interchanges mainly to overcome space constraints. Skewed bridges require special considerations in their analysis, design, and construction. Moreover, these bridges exhibit complex behavior in which the vertical, lateral, and torsional motions are often strongly coupled, raising many concerns regarding their behavior under dynamic loads. Research on the effect of bi-directional seismic excitation is essential for the analysis and design of seismically isolated bridges because the bi-directional motion is coupled and two independent unidirectional models cannot accurately describe the bi-directional behavior. The seismic response of an interchange skewed bridge model under bi-directional earthquake ground motions is investigated in this paper. A 3D finite element model has been developed, in which the pounding of decks at cap-beams, the friction of beams at bearings, and the interaction relation of pile and earth have been accounted for. The beam, piers, and piles are simulated by the three-dimensional beam element; the contact element is applied to simulate the pounding effect between the abutment and the adjacent beam end. Rayleigh damping was used to model the damping in the system, and the damping ratio of the concrete is assumed to be equal to 5%. In this study, the piles were regarded as elastic laterally loaded beams, and the soil surrounding them were idealized as a series of independent springs with constant stiffness, where the lateral stiffness at one point does not affect the lateral stiffness at other points along the depth of the pile. The numerical analysis showed that the direction of ground motion greatly influenced the forces and displacements of piers and girders of skewed bridges; therefore, these factors must be accurately taken into account for the future design and vulnerability assessment of skewed bridges. We conclude that the pounding effect resulted in an increasing possibility of structure failure of the beam decks and changed the regularity of stress and strain on the bridge structures. From the numerical results, the deformation with consideration of the pounding effect was about twice that without the consideration of the pounding effect, which also has an obvious impact on the bending moment of the beam.