The HN reaction to this paper is disappointing, most notably because so few of the people with strident opinions (against or for its conclusions) appear to have read it. I feel qualified to judge the prevailing sentiment on HN about criminal justice, and based on that: I feel like HN should be all over this paper; it should confirm all their biases except for one.

Here are some Cliff's Notes:

First, the author: John Pfaff is an internationally recognized legal scholar specializing in the dynamics of mass incarceration. This paper isn't a drive-by by a random economist; he's cited widely on the subject, including in lots of very readable blogs. I lost an hour today reading things about his work. It's interesting.

Next, on the drug war: the author appears to be against the drug war and against mass incarceration. Some commenters are responding to this paper as if it was an appeal to continue the drug war. That's the wrong way to read it. The author sets out a simple problem statement: incarceration is increasing even as the crime rate falls. That can't be right! The point of the paper is simple: "end the drug war" is a bromide for mass incarceration. If we're really to make a dent in our prison population, more radical interventions are needed.

Third, and the reason I'm so irritated at HN's response: Pfaff agrees with HN about criminal justice in the US! The problem, as he sees it, is prosecutorial discretion. As crime has fallen, the likelihood than a US arrest will convert to a felony charge has drastically increased. That's so counterintuitive that the author is moved to hypothesize about why it's happening. Perhaps crime is falling, but neighborhoods are continuing to decline, and frustrated prosecutors are overcharging accused people as a sort of "quality of life" tactic. Or perhaps the incentives are all screwy: maybe we have too many prosecutors with too few budget constraints, and, as crime falls, they have to scrape the bottom of the criminal barrel to justify their paychecks.

This paper is fascinating, carefully laid out, and full of counterintuitive observations. Most of the places you'd look for simple answers to US prison population aren't actually in the evidence. You think it's revolving-door sentences, or "three strikes laws"? Nope! Most offenders aren't reincarcerated at all, and of those that are, there's a sharp drop-off after 2 prison stays. Think it's long sentences? Nope! Most offenders serve short sentences (the author notes the impact that nominal sentences have on prosecutor incentives, though). Think it's parole and drug-related parole violations? Nope, that's not in the evidence either; in fact, parole strictness doesn't seem to be a major driver at all.

Again: the point of this paper is not that prison is full of violent offenders and so mass incarceration isn't a problem. Even among the cohort of violent offenders, the author is concerned about prison population; as he points out, many violent offenders don't belong in prison either. The author is discomfited by the fact that prison population and the budget drain of our criminal justice process are increasing even as crime falls. Mass incarceration is a problem. If you're serious about wanting to end it, this paper is good news: it strongly suggests that letting college kids buy weed at Walgreens isn't nearly enough to allow us to write that problem off.