> > by definition it hasn't looked at non-randomised trials, or pragmatic trials
>
> They looked at quality studies, not bad studies that fail to control for all the confounding factors, yes.

That's quite the straw-man argument. You've equated RCTs with being "(good) quality" and anything else with being "bad". So, to address those assumptions:

Not all trials are good, even if they are double-masked randomised placebo-controlled. (I'm not saying they're all bad, either.) To be fair, the Cochrane review process and other similar processes do assess quality of the papers they review, and the authors in the cited paper do this (see their Figures 1 and 2). They point out several weaknesses in the studies they reviewed: "All included trials were randomised, but in only 28 trials (12%) was it clear that patient allocation had been adequately concealed. [...] We regarded the risk of bias as low in 16 trials (8%) [...] In 61 trials the comparison between placebo and an experimental active treatment was described as ‘double blind’, whereas in the remaining 141 such trials comparisons were not double blind (or not reported). Observer‐reported outcomes were clearly assessed by a blinded observer in 22 trials, but this was unclear in 41 trials."

In other words, the raw materials they have reviewed were of low-to-moderate quality in a lot of cases. This is not to say the conclusions of the Cochrane review are wrong, it is just that the quality of their raw data (and their assessment thereof) must be born in mind.

The authors also make a somewhat sweeping judgement, which is tucked away here: "In 29 out of the 234 trials (12%), outcome data had not been reported in a way that was suited for meta‐analysis [...] Based on a qualitative assessment, there was no clear tendency for the findings in the 29 trials without outcome data to be different from the findings in the 202 trials we meta‐analysed." This seems to mean that they decided that 88% of the trials that could not be statistically assessed should be considered similar to the 12% they could statistically assess, based on some "qualitative assessment" that is not explained in the review. It would be very useful to have the details of this qualitative assessment made public - how was it done? Again, this isn't to denigrate their findings, but it does demonstrate that even in a strongly quantitative Cochrane review, there is a component of subjectivity.

The above brings up another point: not all reviews are good quality. This one probably is (the Cochrane process is rigorous) although as I've suggested there can still be subjective judgement in even the most quantitative approach. That's perfectly normal and can be fine, iff subjective processes are described in full.

Next point: not all non-RCTs are bad. There is nothing wrong with, for instance, a well-conducted pragmatic trial since it will expose what actually happens in the real world. Patients forget to take their pills, or refuse to do so. Doctors are 'just human', they rely on their instincts (and that's neither "good" nor "bad", it just is). There is absolutely no point in wasting years and millions of $CURRENCY in developing an intervention in a lab if it fails in the clinic for reasons that could have been uncovered beforehand - and research waste is a major problem. Non-RCTs are a part of this process. I'm not saying that RCTs are not part of this process; they have their place alongside other forms of investigation. RCTs are simply not a complete scientific healthcare methodology in themselves.

Next point: RCTs are not always appropriate. It's a real shame that the myth of the RCT as the One True Way persists. Yes, the RCT is the gold standard in some disciplines, and rightly so. But it isn't The Only Way, and other approaches can be appropriate. A huge amount has been written on this in recent years. For example in [0]: "[Evidence-based medicine] can and should shift from evidence-based individual decisions (in which the evidence is generally simple, with a linear chain of causation and derived from randomised controlled trials) to evidence-based public health (in which evidence is complex, with non-linear chains of causation and derived from a wider range of research designs including natural experiments and community-based participatory research)." Frankly, this seems obvious to me. People live and exist in highly non-linear, stochastic situations.

> You claim that you believe that modern medicine works, but you also claim it "often" works for reasons that we don't understand - are you implying most modern medicine is placebo?

Nope. I'd point out that healthcare is broader than medicine, and medicine is broader than pharmacology. But let's start with pharmacology since RCTs are well-suited here. We certainly understand pharmacokinetics very well, and the procedures that take drugs to market are robust, necessary, and rightly include placebo-controlled RCTs. But most guidelines for drug prescription are based on single drugs for single diseases [1] that have been trialled in non-representative patient groups [2]. This falls apart when the patient is in their 80s and taking multiple medications [3]. And that's just a pharmacology example where RCT is very well suited.

Finally, I reassert that the placebo effect is omnipresent. It makes no sense to compare "real medicine" with placebo since "real medicine" includes placebo whether we like it or not. Removing confounding factors under controled conditions is necessary to gain a particular understanding e.g. risk ratios, number needed to treat, etc etc. These are all good and necessary, although difficult to do well and prone to misunderstanding e.g. [4], [5]. But healthcare doesn't stop there. Even the most robust, rigorous, large-scale placebo-controlled RCT of drug efficacy won't stop the real-world reality of a red pill selling better than a yellow one, and won't reduce the value of humane nursing during chemotherapy, for example.

[0] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4556220/

[1] https://www.bmj.com/content/350/bmj.h1059.full

[2] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4632358/

[3] https://bmjopen.bmj.com/content/9/8/e031601

[4] https://journals.plos.org/plosone/article?id=10.1371/journal...

[5] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5455127/

Further reading:

https://bit.ly/3ifxE3I particularly from page 92.

85% of health research is wasted: https://bit.ly/3ik1LHh

Placebo surgery for knee arthritis just as good as real surgery: https://pubmed.ncbi.nlm.nih.gov/12110735/