I only get frustrated with it in terms of efficiency. I'm like that with any game, as an altoholic I make efficient questing an art. :P It is really difficult to do in Caledon Forest though, especially so if you're going through it your first time on a new character. Once you're guaranteed to be downleveled it doesn't matter as much.
I have an overall bias towards Caledon Forest too because of the abundance of Krait combined with a significant amount of underwater combat. I hate Krait and when few to none of your underwater weapon skills have been unlocked they are an absolute nightmare. I consider Caledon Forest to be the most difficult starting zone.