It's actually not because of any mods (probably). It's a common bug with Skyrim actually.. only I got a mild version so far. Some people claim they get 30-60s delays while I only got 1-2s.
Anyway, I managed to fix it more or less. It made the game a little bit choppy when walking around big cities but it's because I have like 100 people walking around there so I'm getting a fps drop to 50. Anyway, it was caused by vsync. I raised the fps cap to 100 as well as enabled all sorts of multicore supports and 4gb memory in the ini following a guide. Havent noticed anything funny so far and the lag is gone too.