I have no trouble with the concept of spec work as a way of judging ability, but I do have a problem with it being approached the wrong way.

I recently applied to join a YC-funded startup as their UX designer. We had a lot of discussion back and forth, and went through several (very fun) interviews. Then, after we'd all met and gotten along very well, and they liked my background, they asked for a spec design.

The spec was both the UX and design for an upcoming feature on the site. I asked as many questions as I could, although being an interview process rather than a client, I found that I wasn't able to have the kind of back and forth, collaborative discussions that I would normally. Presumably they wanted me to just show them my chops, and didn't want to invest the time in longer discussions, or felt a successful candidate wouldn't need that kind of involvement from them.

My first step was to provide a set of detailed user scenarios, laying out the rationale behind my approach, and showing how a typical user interaction would play out through a series of steps. I included some wireframes of the screens I felt would be needed for the scenario. The feedback they gave was positive, and I moved on to the design.

A couple days later, I delivered my design. I asked if I could present it to them, explain the reasons behind it, and get any feedback, maybe even make some edits if it wasn't where they wanted to go. Instead, they asked me to just email them the designs without any discussion. I sent them off, and waited.

Two or three days after that, I got an email saying simply that they had shown the designs to a creative director that was helping them out, and decided not to proceed with me. They cited as negatives several things about the UX that were clearly shown in the earlier deliverable, which surprised me, since I had kept them in the design based on the positive feedback I'd heard earlier.
The startup didn't have any established look and feel or branding yet, and they told me to just do whatever I felt like doing. With nothing to go on, I had to take a shot in the dark. When the feedback came that it wasn't the kind of look they wanted, I felt frustrated because there was no opportunity to revise it to go in a direction they would like.
There was no opportunity to talk about the work, try to get on the same page, or see how I actually performed in a startup environment. It just felt like I'd been participating in a "spray & pray" design competition.

I'm not upset that I didn't get the job, although they were great guys. I don't think they owed me any "do-overs" or anything like that. What I did feel was that my time and effort was taken for granted, and having my work dismissed by someone I'd never met did feel disrespectful and a bit ill-advised.

When you're running a startup with limited resources and tight schedules, you have to be very careful about the hiring decisions you make. Asking someone to demonstrate their abilities and work style by doing some work is a great idea. If you are going to do that, though, you should put that person in a real work environment, not an unpaid design competition. Treat them like a freelancer, bring them on for a test, and give them a task. Pay them something out of respect for their time. It doesn't have to be a corporate market rate, but it should be enough to show that you're serious. Work just like you would with any team member, even though it's a quick task. Large agencies do it all the time by bringing in freelancers and hiring the ones that do the best job. It's a test drive.

If you approach hiring this way as a startup, you'll have the best opportunity to see who will actually make great employees, and you'll get a great reputation to boot.