Recent post-earthquake investigations have revealed that instability failure of dampers can cause severe engineering losses. Conventional viscoelastic dampers (VEDs), fabricated by vulcanizing steel plates and viscoelastic material in alternating layers, dissipate energy through shear deformation of the viscoelastic material caused by the relative movement of the steel plates. However, their high height-to-thickness ratio renders them susceptible to out-of-plane instability under complex practical loading conditions. To overcome this limitation, a novel VED with enhanced out-of-plane stiffness is proposed. Full-scale tests were conducted on both the proposed and conventional VEDs, encompassing basic mechanical performance tests, stability tests, and out-of-plane static pushover tests. The mechanical properties and out-of-plane stability of the two damper types were systematically compared. The results demonstrate that the critical load, critical frequency, and instability mode of the conventional VED agree well with numerical simulations. Moreover, the proposed VED exhibits an out-of-plane stiffness 11-15 times greater than that of the conventional VED, confirming its excellent stability. This study provides an experimental foundation for the engineering application of the proposed VEDs.