I've worked on teams that do precisely this and, and it's worked remarkably well - sometimes. It varies a lot by team; some combinations of people just absolutely fly with this process, they're happy and extremely productive, while others flounder around unsure of what to do. Some of my observations from it that hopefully answer some of your concerns:

1.) This only works when you have a team full of self-motivated engineers that are each able to see the big picture (and they're all on-board with that big picture). Most successful startups have this team structure; occasionally big companies do (my positive experiences with this process were at Google), but once growth rates slow you typically start attracting people who just like to focus on one niche and do it well rather than worrying about the overall success of the project.

2.) It really helps to have the whole team colocated in one area. That gives everyone the tacit context that makes it all work. Yes, I know this goes against many of the other aims of the async manifesto.

3.) This works best on large projects where the deliverable is something that can be easily communicated. Visual redesigns and large refactorings are two good examples: you can say "You're done when the software looks like this interface mockup, with all the corner cases handled" or "You're done when all code conforms to the new API." It works worst on exploratory projects or things that require complicated algorithm design. I couldn't imagine using this to design, say, a new OS scheduler, and I tried using it (and eventually backed out) to develop Google's Authorship program but it was poorly suited for an exploratory project with vague success criteria. Oftentimes those projects are better off handed to a single developer, though.

4.) Priorities need to be rebalanced regularly - it was usually weekly or biweekly when I've seen this done successfully. And it has to be okay to just drop deprioritized work on the floor and assume it will never get done.

5.) Granularity isn't really a big issue. When you have a task whose granularity is too high, it becomes a cover bug: the person who takes it on treats their task as "Figure out all the subtasks that need to get done to achieve this", and pushes them onto the queue as dependencies with the same priority. Once this happens, the product owner may choose to deprioritize the whole lot of them if it looks like a lot of work for little benefit.

6.) The product owner is the point person, usually a tech lead or product manager. This does not preclude all the other developers from also taking ownership and responsibility for shipping the product; in a well-functioning team, that will happen. But there has to be someone responsible for interfacing with the rest of the organization or with the outside world, and responsible for ultimate delivery of the product. That's what they mean by "product owner".

7.) I've seen this work with both shared workqueues and personal workqueues, but it seems to work better with a shared workqueue, where pending tasks belong to the team as a whole rather than any one developer and people assign them to themselves only when they're actively working on them. The reason is because it load balances better and makes the team gel: instead of one person inevitably feeling overworked while another slacks off, you can get this amazing dynamic of people helping out and stretching themselves in any way they can. It makes it feel like the project belongs to the team as a whole rather than having someone tell you what to do. It also lessens the stress caused by having too much work to do or having personal issues get in the way.

8.) I'm a little surprised to see this in a manifesto on async development that explicitly discourages tribal knowledge. I've only seen this work in colocated teams with large amounts of tribal knowledge. I'm trying to think of what it would look like in a distributed team; the closest analogy I can imagine would be an open-source project with a public bugtracker, where product priorities aren't assigned by a "product owner" but rather by the number of people who say "When will this software package support feature X? I really need it to consider using this. Here's a patch."

9.) I've felt more engaged with projects using this style of task assignment than any other system I've encountered. It gives you rapid, immediate feedback on how fast you are progressing, a clear picture of what's left to do, and a sense of team camaraderie as you make shared progress together.