Statistical significance versus clinically meaningful in interventional trials

In clinical trials, researchers use statistical significance to determine whether an intervention is effective. Typically, if the p-value is less than 0.05, the intervention is considered effective. Statistical significance is often criticised because while an intervention may have statistical significance, it may not lead to ‘meaningful’ improvements. For example, the intervention may improve symptoms by 5 to 10%, but this may not be enough change for people to notice.

Therefore, some researchers argue that we should consider whether the intervention has a clinically meaningful/ beneficial effect. A 50% improvement in symptoms is often referred to as clinically meaningful. If it is less than 50%, then it is argued that the treatment effect size is small, not clinically meaningful, and therefore ineffective.

A major problem with this logic is that it does not consider the intensity of the intervention. A 12-week intervention, requiring 30+ hours of professional input and costing thousands of dollars, may lead to clinically meaningful change. For this treatment intensity, I would expect nothing less. But what about taking a single natural supplement for 12 weeks that costs a total of $100? Should this be expected to have the same treatment outcomes as the former intervention? It would be nice, but this is unrealistic.

We need to be careful about utilising a medical model in clinical trials that relies on the ‘magic pill’ mentality. One interesting observation from the placebo-controlled clinical trials I have conducted relates to participant expectations. At the end of the trial, we typically ask participants to predict which intervention they were allocated to (placebo or active treatment). Many people will have experienced an improvement of 20% or more in their symptoms (based on questionnaires) but still say they were on the placebo because they were not cured of their disease. They expected that the supplement would rid them of their depression, anxiety, sleep problems, memory problems, etc. This would be nice, but it is unrealistic. We humans are way too complex, and treatments are typically multifaceted.

Based on my years of experience as a clinical psychologist, I would be content with an intervention that reduces my symptoms by 10 or 20%, costs $30 a month, is well tolerated, and only requires 1 minute of my time a day to implement (i.e., the time to swallow the pill). Imagine if I incorporated 5 or 6 of these low-cost, low-intensity treatments at the same time!

So when you review a clinical trial on an intervention, think about how much change the intervention resulted in, how much the intervention cost, how much time was required to implement the intervention, and how safe the intervention was. You might find that the intervention that resulted in the slight improvement in symptoms may have been associated with a ‘larger treatment effect’ after you consider the resources required to implement the intervention