A work-sample challenge is a practical problem that mimics the actual work done on a job (the term comes from W. Edwards Deming back in the 1950s). Doing a Github-PR-style code review is a perfect example of a work-sample challenge; you can come up with any number of work-sample challenges that involve actually coding; a friend used them to recruit for his sales team by having them write outbound mails or handle simulated inbound requests.

Our work sample challenges at Latacora involve as assessment of a Django web application, an assessment of an AWS deployment environment hosting that application, and the construction of a simple scanning tool to spot the vulnerabilities discovered. At my previous firm, the challenges were similar – there was no programming challenge, but there was an assessment of a custom binary trading protocol.

I'd balk at robot automated interviews too. I'd be concerned that my responses wouldn't fit whatever weird template they were using; also, I'd be almost certain that any robo-tronic assessment thingy they were doing almost certainly wouldn't be determinative of the outcome of the qualification process, but rather just an arbitrary hurdle I'd have to jump over before the "real" interview happened. Obviously, if my options include equally attractive companies that don't present that same extra hurdle, I'm not going to bother applying!

I also think there is certainly a class of senior developer that won't do practical at-home challenges at all, no matter how carefully designed, even if they're entirely determinative of outcome and even if they're calibrated carefully to make the interview process less logistically arduous (for instance, by reducing the time commitment for actual interviews from 4-6 hours down to a single hour, and assuring before the interview happens that you're going in already tech-qualified). Certainly, work-sample qualification isn't the norm in our industry, and senior developers also tend to move from job to job through networking and very warm introductions.

What I'll contend, though, is that there's a pretty significant adverse selection problem in that pool of senior developers. I don't doubt that the bulk of them have earned their experience and reputation, but there're also a lot of developers swimming in that pool that have failed upward into it, and move from job to job largely through interviewing skill. What I've learned (or think I have) is that there's virtually no correlation between someone's ability to talk about programming and their ability to effectively deliver code on a commercial project.

What makes the work-sample process so attractive to me is that the opposite phenomenon also seems to exist: in the pool of "not labeled senior" developers, there seems to be a usefully large number of exceptionally talented developers who have, owing to circumstance, simply not acquired the resume labels to navigate themselves into good jobs. Hiring from that pool is very risky if you rely on interviews! But what we learned over the last 10 years is that you can build practical challenges and get to a level of confidence where you can rely on them over interviews and resumes, and when you do, you can hire a lot of talented people that other firms can't, because their interview processes won't let them.