I'm getting insane fps tankage when using Mumble and it happened with Ventrilo also. It lags really bad, like going from 60+ FPS down to 15 FPS.
When I change the setting to Realtek Digital Input, the FPS drop goes away 100% but I can't be heard on Mumble. I'm absolutely LOST trying to figure this out.
When they're open and set to default device (my headset, Razer Carcharias) my FPS tanks from 60+ to 15-20 FPS. If I change the audio device to "Realtek Digital Input (Realtek High Definition Audio)" (is what it is labeled) my FPS jumps back to normal but I can't be heard by anyone.
My computer isn't terrible by any means really. It's not a dust issue as I spent an hour and a half religiously cleaning every nook and cranny of it.
I really doubt Ventrilo and/or Mumble would be affecting your graphics performance.
But they can affect CPU performance - which would tank the game. I'd guess something is wrong with the Audio Drivers. It would be a good idea to keep watch on the AUDIODG.EXE resource usage when the problem is happening.
Also make sure that that recording device is set to 16bit 48khz mode, or whatever format Mumble/Ventrilo uses - as otherwise they will need to resample the audio using up additional CPU resources, and having a negative effect on quality.
Do you have a discrete soundcard? If so this is an issue where Audiodg.exe starts becoming a huge memory leak and consuming up to 50% of the CPU any time there is mic involvement. There is a hotfix for this problem: http://support.microsoft.com/kb/981013
If you are using a Creative Sound Blaster X-fi card, you need to ditch the factory drivers and utilities and get the XFI_Support_Pack_2.0.