I tried gliding from Dal and hit an invisible wall. Going to the Ironhorn enclave FP and gliding down to Felsoul was how I got mine.
As far as Downfall himself, just make sure you glide back into his circle after he punts you away to avoid falling and resetting. If you end up standing on the invisible platforms on the pillar, DO NOT use your whistle there. It will go on CD and not take you to the FP.