Well, I did some more testing, and it seems my CPU temperatures actually go to 86°C (after only couple minutes of playing) which is pretty bad. I'll disassemble my whole PC and dust it off today, and see if that will help.
I'm running 4GB 1333MHz, and though I'm open to the possibility that is the problem, I'm not too sure, because even when I set particles to low same thing happens. Obviously, the problem is loading assets and I had the very same thing in Diablo 3 and someone found a fix - installing O&O CleverCache and changing some setting in NVIDIA profile - which would suggest it's probably a HDD thing. Now, I tried the exact same fix here to no avail.
Anyhow, thanks for suggestions, I'll clean the PC up and test some more so we'll see