Sidequests give a pittance of EXP and rewards, and very few are worth doing for the story. Unless you are a completionist, or you really want to get the tiny scraps of lore or atmosphere about a people or a zone, then I'd recommend skipping them.
The only sidequests (that don't have blue icons) I'd recommend doing for the story would be:
ARR
- Whitebrim Front in Central Coerthas Highlands (gets you a feel for Ishgard, a nation worn down by war against Dragons)
Heavensward
- The Convictory in the Western Coerthas Highlands (a camp in the snowy wastes where poor men and the younger sons of nobles with no hope of a decent inheritance go to try to kill dragons in order to win prestige and try to return to Ishgard with higher social standing. Most die, however).
- Tailfeather in the Dravanian Forelands (an independent village that lives off of hunting big game). Also, Anyx Trine in the same zone.
- After you beat the 3.3 MSQ, go back to Fortemps Manor and do the short questchain you see in there.
Stormblood
- I actually enjoyed almost all of the sidequests here. You really got a feel for life under Garlean occupation in Ala Mhigo and Yanxia. The sidequests in the Ruby Sea explore a pirate nation in the East called the Confederacy. And the quests on the Azim Steppe get you acquainted with several different Xaela tribes.
Shadowbringers
- Doing sidequests in Kholusia, Fanow in Raktika, and Amh Araeng gives you a feel for how hopeless life is there. There were several good quests in Ill Mheg, though there are a lot of boring fetch quests around the starting area.