To address the limitations of traditional CamShift algorithm in seismic isolation engineering applications—specifically, its over-reliance on color information and susceptibility to environmental interference—this study proposes a vision-based method for measuring seismic isolation bearing displacements. First, the captured video was subjected to image preprocessing, after which an enhanced tracking stability was achieved by dynamically adjusting the weights of target edge information (extracted via the Canny operator) and color information (derived from the CamShift algorithm), thereby generating a fused information histogram. When the target was unobstructed, the improved CamShift algorithm was directly used to determine its position. In cases of partial or full occlusion, the occlusion severity was quantified by the obscured area ratio, and a Kalman gain was introduced to predict the target position. The final state estimation of target was obtained by fusing predicted and observed positions. Real displacement values were then calculated through coordinate transformation. The method’s accuracy was validated by shaking table tests on a three-story steel frame model, demonstrating that the maximum displacement error between the results of vision-based method and wire-drawn displacement sensors remains below 6.84%, with correlation coefficients exceeding 0.91. Finally, the vision-based method was applied to an actual project, and the results revealed a measurement error of merely 0.15 mm(98.56% accuracy) between the proposed method and wire-drawn displacement sensors. These results confirm the method’s accuracy and robustness in complex isolation layer environments with variable lighting, dust, and occlusions.