You have a trinket, you have an enemy cast bar. If you see him casting chaos bolt you should probably use your trinket and silence him. It's the only thing you have to avoid when fighting a destro lock, if you allowed them to get a chaos bolt off it was your fault, you can feign, silence, deterrence, los etc.