First a note on how to read the relevant study. The link to the study in the WSJ article is paywalled. This is a non-paywalled link [1] is the results [2]. 22 applicants solved a problem alone, 26 with a proctor present. Without the proctor present about 2/3rds passed, with the proctor present 1/2 passed. Using score >= 2 as "passing", 12 out of 16 men passed in private and 4 out of 4 women. 11 our of 20 men passed with the proctor, and 0 out of 6 women. The methodology looks robust, but especially with the claims with respect to gender I'd want to see a sample size larger than the single digits before making any generalizations.

With that aside, my broader thoughts on tech interview processes: Companies want an interview process that are,

- Successfully distinguishes between people that have the
knowledge and abilities required to perform the job.

- Has systems on accountability, consistency.

- Is relatively easy to train employees to administer the interview.

- Has a relatively low time-commitment for everyone involved, both interviewer and candidate.

In reality, though, there are tradeoffs between each of these points. For instance, using a set question bank improves consistency and accountability. Rubrics can be more explicitly defined, and bias limited. But it means candidates can google for questions beforehand. This was a salient issue when I worked at Dropbox, there were only about a dozen technical. interview questions for a 2,500+ person company. Having developers come up with their own unique question helps mitigate this, but reduces accountability.

Likewise, I've had some novel interview processes that more closely approximate real working conditions. One company's interview was conducted over git-hub. It was asynchronous, with tasks spread out over a week. After building the first solution, the interview came back with further feature requests and comments on the first iteration. This tested the candidate's ability to refactor existing solutions to meet changing tasks, and the ability to integrate feedback. These are things that are rarely captured by whiteboard interviews, but are arguably some of the more important skills in software development. But on the flip side, it was much more time-consuming in aggregate than 4 hour-long interviews.

No one interview process has all the advantages. I think a lot of companies settle into a pattern of 1 or 2 remote technical screens followed by a circuit 3-4 hour-long interviews because it's logistically robust. It's a format candidates are familiar with. It's easy to train new employees to conduct these interviews, and there's broad enough set of people participating that one biased signal isn't going to be decisive.

> Companies could also drop problem-solving tests as currently offered and instead ask candidates to spend five minutes explaining how they would perform a particular job-related task, Dr. Parnin says. Focusing on communication skills in this way, Dr. Parnin says, can reveal how a candidate thinks.

But do we want to hire the candidate that can talk about high level ideas for five minutes in a convincing manner, but can't code Fizz Buzz? Or a binary search? I get that some people might stumble due to pressure, but at the end of the day if you need some mechanism to determine if the candidate has the required skills or not. And administering a test is an effective way of doing this.

This isn't the first time articles like these have been written. X group is disadvantaged by problem-solving tests, so don't use said tests. But how then do you determine whether or not the candidate has the required skills? Usually whatever supplants the technical interviews are also subject to unfairness: referrals, recruiting alumni from specific universities or companies. While far from perfect, I still have trouble seeing what could replace problem-based tasks as a means of demonstrating skills.

1. http://chrisparnin.me/pdf/stress_FSE_20.pdf

2. https://imgur.com/EIiofkQ