OpenGL/SFML Context Recreation Issue

Описание: When the game initially runs and the OpenGL context is created there are no issues, rotations are smooth and everything works fine. After I change the settings at runtime, recreate the window/context and reload my shaders, VAOs and textures, there seems to be an issue where models move weirdly when I rotate my camera. Seems like the model transforms are lagging behind a little?

EDIT: I narrowed it down to being somehow related to the framerate. The lower the framerate the bigger the issue. But even at huge framerates it's still noticeable, although a lot less