Leaving aside my above post... because we want to do things that are real, not virtual? Even if they feel realistic, the knowledge that they are not would be sufficient for many people.
Course, then there are the would-be Evil Overlords who view 3/4 of humanity plugging themselves into the Matrix as the perfect opportunity to take over the world :P .