No, I haven’t, thanks for mentioning it. Would Goodhart’s Law actually apply here though, since right now the data (as far as I can tell) isn’t being measured at all?

To your counter-example, maybe the metrics I described aren’t good enough and should integrate some disease severity criteria or site-weighting or comorbidity score (although as one continues to subdivide the population this way eventually you end up with n=1 and the results are useless again) but like, surely we should be trying to measure something other than the good feels and word-of-mouth of people who have to work with each other?

It physically hurts my brain when I think about how we measure the dumbest shit in software engineering, like which shade of blue to use to improve clickthroughs[0], but when it comes to even attempting quantification of activities which are literally life or death, sorry, too hard, can’t do it. Surgeons will refuse to do hard procedures, insurers will destroy careers, EMRs are full of bad data (so what is the point of the bloody records if they have become that useless‽), surveying patients would cost too much…

I will ultimately defer to the experience of people in the field—I am not a physician or statistician—but sometimes I feel like I’m just being fed arguments repurposed from the bad cops playbook. Oh, we can’t ever possibly start quantifying individual officers’ use of force, because some parts of the city have more crime, and if we do that then those officers will look worse, so they will stop responding to violent calls in those areas, and there’ll be even more crime, so get off our backs man and stop trying to create more objective metrics for accountability.

To be clear I don’t think you are arguing in bad faith and I don’t intend my statement about accountability to suggest that you personally are trying to avoid it or shield bad actors or anything. What you are saying is probably true and I may be wrong to challenge it at all since I have no personal insight into what is going on behind the scenes, and I genuinely appreciate you answering my questions from your perspective and giving me additional perspectives and things to think about. It just feels so, so frustrating as a patient. All I want is some ability to measure risk that’s better than looking up studies on procedure X on pubmed that I’m unqualified to interpret (and which don’t apply anyway because the lead author of the research won’t be doing my procedure), or shaking the magic eight ball.

If I were a physician, I would absolutely want to track the shit out of my own patient outcomes so I could improve, and the amount of resistance that seems to exist (this is not the first time I’ve talked to docs about this and received similar fatalistic answers) is just baffling to me.

We’re not talking about Frogger here, metrics aren’t some high score, if you have an 80% complication rate for some procedure that isn’t necessarily a reflection on you as a practitioner but it would suggest that there is a problem that needs to be identified (bad procedure, bad training, bad support, bad patient, bad luck). Right now, it seems like no one really knows.

This isn’t bullshit alternative medicine, so why, when I scratch beneath the surface, does it so often feel like it is anyway?

[0] https://www.zeldman.com/2009/03/20/41-shades-of-blue/