Here's some real problems I've had with it after using the Rift for over a year. Many of these things can ultimately be addressed in software (e.g. no use of your phone? mirror it or just use something like google voice):

It is more immersive than the movies. This is awesome, but also terrifying. When you sit down on your couch to play a game, do you want to be as restricted as you are when you go to the movies? Well, you are more restricted. You can't look at your phone and keep up with friends. Fine in a two hour movie you've been looking forward to, but in the game you are going to log 200 hours in this year?

You can't take a sip of your drink without learning what it means to be blind.

The movies are a bit more unforgiving in other areas--you have to block out time and get in your car and travel. But with Oculus what you end up doing is things like taking off the googles to give your long neglected and probably creeped-out-at-no-eye-contact dog some attention. Then when you put them back on, they fog up. You take them off and breath on them to get them up to the ambient temperature near your face so there won't be more condensation and wipe them off. Put them back on and opps, grease from your forehead just got on the lenses; back off again. Wipe them down, and finally back on. Sort of it's own car trip.

If you have a young kid or baby you will be seriously alienating to them and probably scar them by neglecting an internal drive for eye contact and attention. You might be pretty neglectful due to normal TV and web browsing, but this is going to bring it to a whole new level. So, many people will have to choose between making limited use of the tech or being seriously abusive towards and retarding development of their young children.

CastAR seems to solve almost all of the problems. Something like it, or pseudo-holographic displays (just a normal 3d display, with headtracking, perspective correct 3d, etc. It is a far superior experience to 3DTV viewing or gaming, which is usually viewed at the wrong field of view for the content, and goes wonky and immediately breaks the illusion with any head movement, especially tilting).

Something like a 60" 4K curved LCD with lightboost allowing shutter glasses to be "open" more than 90% of the time, and thus not darken your vision or obscure eye contact is where I think the sweet spot will be for a while for having a serious workspace/playspace, when combined with accurate low latency headtracking. Panels fast enough for lightboost currently have bad viewing angles, but if you are tracking the users head you can just correct a lot of it with a simple shader. If systems like CastAR can get the resolution up they are even better, because two people can use the same surface at once and get the correct view. If the above setup sounds underwhelming, remember that Oculus is initially targeting a seated experience.

Full-VR approaches will start to overcome a lot of the issues once they can track your eyes facial expressions, give you an optional overlaid view of the real world from high field-of-view cameras on the headset. At that point you still can't make extended use of it if you have a baby or toddler or pet (or staunch non-participant) that you care about, but plenty of people could make it work and have very social lives. I completely disagree with the author's cynicism about something like a shared virtual world being a terrible thing.

If for one reason or another some business strategy involving heavy network-effect lock-in does arise and all development efforts, content, and friends are drawn into one fascist dystopian system, yeah we're fucked. But I don't see the path there (even assuming Facebook currently has John Carmack unknowningly working towards some kind of bizarro-world metaverse), and this guy didn't really lay it out for me.

(one extra thought after having left a job a few months back... the downsides of VR I outlined sound suspiciously like the downsides of a typical desk job...)