With the default setting RotationAction->"Fit", 3D graphics are rescaled to fit in their image region at the end of every interactive rotation action. This will typically make the graphics appear to jump slightly when the mouse is released.
With RotationAction->"Clip", 3D graphics are not rescaled at the end of a rotation action, so they may be clipped or leave extra space, but will not appear to jump.
RotationAction can be set globally in the Preferences dialog.