Also lots of questions emerge when one is trying to be constructive.

Hiring is meant to have fidelity. Fidelity to what? To the extent there are company values, nobody explicitly “tests” for them in hiring. Certainly when I worked at US Engineering, which had an explicit values statement, I don’t recall being interviewed for how I “exhibit integrity in everything we do” or “ensure a safe working environment.” I doubt Amazon is like “so let’s talk customer obsession. when is the last time that you stalked a customer and got obsessed with making their life easy?” (I mean you can question the value as well, but to the extent that Amazon tells you its core values, are they a part of hiring?)

How much fidelity are we talking? Use the “nines” metaphor. Do you want five nines, one bad hire in a hundred thousand? Probably you need a month-long “interview” process then, you’re basically at the question of “let us contract some work out to you for a month and if we like you we will hire you.” But, consider: are more nines even better? If we have BS metrics that do not properly reflect our interests then we only find out about them by hiring people we would not have rationally hired. Maybe you only want one nine, or more radically even half a nine (that is 68%, if you have never heard of it), of fidelity. What if you baked that fidelity in? What if a unilateral consensus to not hire still generated a 10% chance of hiring and a unilateral consensus to hire still generated only a 90% chance of hiring, so that interviewers had to know getting into this that absolute sureness was not a goal?

That brings up an even better question, which is: can you have a good hiring process without a good firing process? How do you give yourself the one-nine process and then make sure that people who were retroactively not a good fit get a decent severance but are done away with?

I have mentioned elsewhere that Bayes’s theorem suggests that evidence should be measured more or less in decibels; this is covered explicitly in Jaynes’ Probability Theory ch. 4. But consider [1] where researchers found that they did not see substantially more correlation between different interviewers evaluating the same candidate for 20 minutes, and other interviewers watching videos of those interviews over only the first 20 seconds (the initial greetings of the video). So can we actually trust interviewers to give us, say, +10dB of evidence that this person is a good fit? Or if we get 5 interviewers do we get a range from “meh, pass” to “hire immediately for twice what they’re asking” as much as if we had done job interview “speed dating”?

Constructive advice is hard. One wants to instead force the hiring committee to watch talk after talk by Alan Kay until the roughly-every-other-talk discourse Alan gives about how “we live in a pink world and then we have a deviant thought but then we kersplat it until one day in the shower we have a kerpow that launches us into a new blue plane of existence—but that idea like most is more likely to be mediocre or terrible, so we have to then come back to it with something like science so that we can discard terrible ideas” becomes somehow ingrained in the hiring committee and they start trading their “kerpows” and evaluating them and coming to a better process than this “pink world” we see with our rose-colored glasses.

Because without that sort of systematic re-invention, aren’t we just guessing?

[1] https://www.researchgate.net/publication/313878823_The_impor... . Side note: in tracking this study back down I got to see a lot of interesting takes on it but the most common consensus is that “we make our first impressions in that first 20s and then do not budge from them” which I do not think is warranted given the data. That is, while a significant correlation coefficient is observed and it is comparable to the correlation between different long-form interviewers, at no point is that correlation coefficient, like, 0.9 or 0.99 or anything like that, suppressing interpretations that these first 20 seconds determine the interview. For that matter if that was your goal you would want as a control any other 20 seconds in the interview—you would want to prove that it was the first 20 seconds that determined the overall interview impression, not that people were judging based on some je-ne-sais-quoi demeanor that is visible throughout the interview.