It would most likely have something to do with the OpenGL Display mode. As a standard WVM uses a "failsafe" setting which result in high pixelation. If you recently changed/updated your VMW, chances are that would be the reason. As for what setting you need, bruteforce is really your only option. Test your way through it and see if the program produces errors. I highly doubt that it would, but also be careful about the settings since the render can become too smooth. By too smooth, you will have a hard time cutting the render out by using Select Color Range or using magic wand. This will result in you getting a tiny (though still very visual) blue border (or whatever background colour you are using) around the render since Photoshop can't work with such "smooth" pixels.
When changing the OpenGL Display Mode, what you need to keep an eye out for is the FSAA value.
FSAA: 16 tends to produce the so called "smooth effect". While the renders become dazzling and ever so beautiful sparkly, you will still have to either feather your selection (producing a very ugly cut out) or somehow delete the blue border which takes time.
FSAA: 8 tends to be what I use. Good quality render and Photoshop can still work with the pixels of it.
I personally use Colour:32 Depth:24 Alpha:8 FSAA:8 DoubleBuffer Hardware mode.
I can't really tell you what would be the best setting for your rig (cards and drivers), so you'll have to just test on until you either grow bored and annoyed, or find a good setting. To limit your test feel, try the FSAA: 8 and stay in that range. But for good measure try the FSAA: 16 and see if your Photoshop can work with it. If it can, you'll get top quality renders, which may result in happiness.
(Actual result of happiness may vary)
Hope my babble helps.