This post describes an unusually good technical interview program, in which most of the tech-out is apparently done at home, the interview is prefaced with a short but useful work-sample code review test, standardized questions are used, and candidates are allowed to use their preferred tools and languages.

That's commendable, but I think the argument is still faulty.

As I understand it, the premise of the argument for interviews and against work-sample testing is that candidates won't perform work-sample challenges. I've interviewed almost as many candidates as this person has (and screened out far more based on challenges) and that simply hasn't been my experience. More importantly, the argument is incomplete. Stipulate for a moment that the "best" developers won't take work-sample tests (I'll come back to that)... and?

The point of work-sample tests is twofold: to stop being beholden to subjective impressions of how well candidates can perform the work you need them to do, and to provide a framework for tech qualification that's straightforward to iterate on. If your work-sample tests are sound and you're getting enough candidates to saturate your headcount requirements after setting a bar you're comfortable with, why obsess over who exactly you're hiring? The industry is full of developers with glittering resumes and lists of past achievements who are dead weight on teams†. Why assume the developers with the best marketing are the ones who will perform best on your team? Why not instead just spend the time to figure out what exactly a good performer is, and then directly test (and iterate on testing) for that?

Regardless, the belief that strong candidates won't do work-sample tests is pervasive. And for good reason: work-sample testing at most companies seems cargo-culted: "take-home tests" (that often follow tech-out interviews) that can only rule candidates out and never rule them in. I agree: that's bad. But the solution is straightforward:

* (1) Allot a budget to the entire technical qualification process, including all interviews and challenges, (2) try to make as much of the process objectively scored based on a rubric your team has committed to writing, and (3) ideally move as much of the process "offline" (so the candidate can perform it at home and without you watching) as possible.

* Make the work-sample challenges as determinative as possible. Candidates who do well on our work-sample challenges are presumed technically qualified; 1:1 interviews that follow it are largely pro-forma. At Matasano / NCC Group (while I was there) the bar for a technical interviewer to overrule the output of the work sample challenges was quite high.

Regardless of whether you use work-sample challenges or scripted, structured interviews, a bit of low-hanging fruit that nobody seems to pluck is: relentlessly keep candidates informed about what to expect in your process. Since about 2010 we've been buying books (expensive books!) for candidates and providing study guides to go with them; we also provide fairly detailed descriptions of what our challenges will be. We're even moving to a system now where we offer candidates a practice version of some of our challenges (if I could fake up the entire AWS API, I'd do practice challenges for all our challenges).

But most employers seem determined to make their tech-out process as much of a gotcha game as possible, and tell candidates almost nothing about the actual hard problems they'll be expected to deal with.

This post uses a framing that says whatever tech companies do to qualify candidates must be rational; after all, if they're wrong, they're raising their own costs. I respectfully disagree: engineering teams do not, as a general rule, optimize their processes for efficiency or shareholder value. Many teams literally use interviews as a hazing process (one they often exempt "in-network" hires from). Still more delegate interviewing out to the entire engineering team and exhibit little or no interest in establishing rigor or rolling back bias. But engineers are as biased as the rest of humans, and most of hiring today is not in fact rational.

Also: sorting algorithms make dumb interview questions.

† This is in part due to a dynamics in our industry that make it easy to "gradually fail upwards": most companies can't fire fast enough after making a bad hiring decision to prevent the bad hire from accruing another positive line-item on their resume, so smart people can bounce from team to team for years without ever making a serious contribution and end out ahead.