Why bother with all of that? If you think they're good enough and you need their class/role, give them a trial in guild. If they're not good enough, they don't pass the trial. They wasted their own money because they overhyped themselves and couldn't perform in an actual raid - that's not your problem.
I'm pretty sure most of the valid ideas have been mentioned already. There really aren't that many options to test out someone's performance x-realm other than logs, fraps, or some x-realm grouping in trivial content.
In the end, the whole point of a trial is to actually test someone. Some guilds forget that not every trial has to make it to member: It's their choice to apply to your guild, so if they don't meet your standards during the trial you are free to remove them from the guild afterwards. While some prescreening should be done to protect people from themselves, in the end it is not your responsibility how well those trials perform: If it's not up to your standards, they simply overestimated themselves.
Part of the question probably depends on what level of guild you are. I imagine someone in the top 50 guilds is looking at a cross realm recruit for different things than I would be. I'm more concerned with intangibles, once we get past the basic 'are you gemming right and do you stand in the fire' questions. Are they annoying on vent? Do they blow up if things go wrong? Can they identify problems and point them out?
I'd think that challenge modes would be ideal for something like this. You'll find out if they're the sort to prepare and read up on the fights, you'll find out whether they're willing to talk about alternate strategies before the pull, you'll find out if they can react on the fly and move quickly.
If you're willing to be a bit mean, you could even have one of your guys sandbag the run close to the end, and then ask the recruit if he can identify who the culprit was and what they did wrong, and suggest a way to work around it. Then you can find out if they can give feedback without being a dick about it.
Honestly, LFR/challenge modes are probably what i would use the most. Watch how they deal with mechanics (not just stand in it because you can in LFR) have them take it seriously. Honestly the PTR would probably be the best, but i'd fail at that due to lacking my addons. most namely power auras. It isnt a situation of my addons playing the game for me, its moreso ive set it up over the past few years how i want it. in highschool i could pass 100 WPM on my keyboard but in keyboarding class in HS i was closer to 65-70. It wasnt -my- keyboard, i wasnt at my best due to that.
Watch them one-shot all the rank 8 Brawler's guild fights if PTR isnt possible.
Doesn't show anything. Even the best players will struggle on Disruptron without levitate. You can get fucked on Zen'Shar pretty easily depending on class and laser movement, and millhouse can just RNG you.