The impact will depend on the type of agreement, and the manner in which the withdrawal is conducted.
Doing it in a planned and orderly and diplomatic way that will control the impact will certainly limit the consequences.
But if you do it Trump's way, and announce on twitter that you're leaving the Paris Climate accord, or an FTA, that leads to a real loss of credibility and soft power.
Add to this calls to the Taiwan president, or plans to move the US embassies to Jerusalem, and one man single-handedly affects decades of international policies.
Same goes with Brexit. Obviously after the referendum your partners can prepare for what's coming next, but if you're not doing your homework and being constructive and proactive in defining what comes next your partners will eventually lose patience.
States are not people, and the agreements they enter into can't be left in a rushed and messy way. There is a reason changes are done in small, calculated steps. Your partners have stuff to deal with at home first, and don't take erratic, unilateral behaviour on the international stage kindly or patiently for very long.
You can legally do it, but you will also piss your partners off, lose their respect, some of your influence and your credibility when you sit at the table to negotiate the next agreement.
And that takes years, possibly decades to restore.