This is via Alexandros Marinos on Twitter, streamlined via the Thread Reader app:
I’m realizing that the insitence on Randomized Controlled Trials (RCTs) as the only evidence that matters when deciding if a medicine/supplement should be used, structurally biases against generics, over-the-counter meds/supplements, and those with few side-effects. Here’s why: The first class of problems has to do with wide availability when the subject of effectiveness on a new disease is raised.
1. Cheap OTC generics with few side-effects get used a lot in an emergency, where word of mouth spreads, making it much harder to form a control group.
2. These substances, when there’s a suspicion they can be effective in an important disease, will spark many studies all over the world. This means there will be many small trials, of varying protocol/dosage and study quality.
This is a big problem for two reasons: 2a. Varying dosage/protocol means that the results will necessarily be suboptimal. Even if some studies use the perfect protocol, others won’t.
2b. Some studies will be poor quality and/or fake. Critics can focus on these to invalidate every other study by diverting attention.
3. However, that’s why we have meta-analyses right? Well, for something this controversial, there will likely be many meta-analyses, allowing motivated actors to pick and choose which one they prefer.
3a. Meta-analyses can focus on only “statistically significant” trials, or only on RCTs, and in general play with inclusion-exclusion criteria. Setting the criteria too tight, allows some analyses to remove most of the data, concluding “no evidence exists to suggest…”.
3b. As some trials get shown to be of bad quality, calls for retraction of other meta-analyses are made, delaying conclusions. For a meta-analysis to be updated, it would need resubmission that can take months, even though the update takes a few minutes.
3c. Meta-analyses can water down the results of the strongest trials by averaging them out with the weakest trials. When the appropriate dosing is not obvious, this can significantly dull the effects seen.
4. Of course we shouldn’t forget the obvious: RCTs are arduous, expensive, and doing one that’s big enough to be “convincing” is prohibitively expensive.
A napkin calculation of what it would take to do a study for a 70% effective early treatment for COVID turned up a $10m cost. Naturally, a new medication that is not available to anyone before the trial has none of these problems, and since it’s going to be patented, budget is not a problem. The sponsor can do small unpublished studies to optimize the protocol, and then do a big one to showcase effect. And if there’s only one study to speak of, none of the meta-analysis problems apply.
This is why Obama’s former head of the CDC, Thomas Frieden, wrote [in 2017] about moving beyond the RCT.
He then cites another Twitter account, Simon Vallee (@sival84), that goes into further detail:
It’s even worse than that. RCTs are extremely time-consuming and labor-intensive. Plus, they’re inflexible, they test one product at one given dosage and that’s it.
They’re designed to find novel drugs customized for a disease with major institutional backing only…
Repurposed generics don’t have the institutional backing for RCTs to be made for them. Repurposed treatments often require many drugs and supplements to maximize benefits, whereas RCTs test them one at a time only. So if you have a protocol of 4 drugs/supplements that reduce mortality by 50%, but individually even the most efficacious of the lot used alone yields just 20% mortality, then RCTs will usually fail to obtain a significant result, which some dishonest people will conclude to mean “no effect is found” when in fact the proper conclusion should be “the trial is underpowered to verify the effect size”.
So RCTs struggle to provide useful information in due time for multidrug protocols, especially for early treatment when risks of hospitalization/death are already low, it’s bad practice and bad science.
For early treatment protocols, which require massive numbers of patients to provide statistical significance, observational studies, which are a lot less time-consuming and labor-intensive, are more appropriate and studies found…good observational studies provide the SAME results as good RCTs. The obsession with RCTs and rejection of clinical experience of physicians is making it nearly impossible to develop early treatment protocols from repurposed generics
Back to Marinos:
Fascinating point about the steep, stepwise shift in cost and complexity. For most things they will either be too weak, or overkill:
Meditating on this, I suspect what we need is a meta-analytical framework that doesn’t penalize a hypothesis for having been studied *too much*, that takes dose dependence into account. I don’t know if it exists, but if it does, I bet the word “bayesian” is somewhere in the name.
Marinos brings up RCTs because of the controversy over Ivermectin. The small number of elites/media types that aren’t snarking about “horse medicine” will, when pressed, say that Ivermectin cannot be used against Covid because we don’t have large RCTs out there that support its use.
1 Comment