Compared to ordinary reinforced concrete (RC), a certain height of steel fiber reinforced concrete (SFRC) applied in the plastic hinge region at the bottom of piers can be used to enhance the seismic performance of the piers. To systematically study the mechanical behavior of SFRC piers under compression-bending-shear-torsion combined loadings, three pier specimens with a clear height of 1200 mm, a cross-sectional diameter of 300 mm, and a SFRC height of 300 mm in the plastic hinge region were designed and fabricated for quasi-static tests. The damage patterns and failure modes of the specimens were observed. On this basis, multiple numerical analysis models were established using ABAQUS software to study the effects of the torsion-bending ratio and SFRC height on the seismic performance of the piers. The results show that compression-bending-shear-torsion coupling effect reduced the bending and torsional capacities of the piers. The increasing in torsional effect causes the position of the plastic hinge to shift upward, resulting in more significant damage to the specimens. SFRC can effectively improve the displacement ductility factor of the piers, delay stiffness degradation, control concrete crack propagation, and enhance the damage tolerance, bending capacity, and torsional capacity under earthquake action. When the height of SFRC is about 600 mm, it can not only improve the seismic performance, but also reduce the construction cost and avoid material waste.