Historical earthquake damage shows that the connection stiffness of infilled walls and frames exerts an important influence on the seismic performance of frame structures. Code for Seismic Design of Buildings (2010) suggests that infilled walls should be separate from or connected flexibly to columns but does not provide the appropriate connection stiffness between these structures. In this work, pseudo-static tests on frame structures considering the effect of connection stiffness between infilled walls and frames were numerically simulated by using ABAQUS. Then, based on a typical frame structure, finite element models that do not consider infilled walls but consider different levels of connection stiffness between infilled walls and frames were established. The failure features and hysteretic curves of frame structures with infilled walls were compared with those without, and the reliability of the finite element models was verified. Modal analysis indicated that connection stiffness exerts a small influence on the frequency of the structure and that the structures showed the same dynamic properties. By analyzing the elastic-plastic time histories of the structures under frequent and rare earthquakes, the appropriate connection stiffness was confirmed to improve the seismic performance of the structures of interest. This paper provides a reference for the seismic design of frame structures.