How to find the cure for what ails you

How do we know that the treatment for something is responsible for the result we are hoping for? Personally, after years of mental health struggles, I was diagnosed with Depression and Generalized Anxiety Disorder. After continued difficulties, I was diagnosed with OCD and then ADHD.

My wife had immense abdominal pain that became chronic, and was eventually diagnosed with Small Intestinal Bacterial Overgrowth (SIBO). She later had chronic foot pain and was eventually diagnosed with Os Trigonum Syndrome.

My mother, after managing a variety of symptoms, was diagnosed with Chiari Malformation, and after COVID-19, contracted long COVID. These struggles (plus even more among close family) have had us in and out of doctors' offices, hospitals, clinics, etc., for decades. The journey of treatments, medications, supplements, therapies, etc., can be arduous and costly. I have been wary of “snake-oil salesmen”¹.

Thankfully, we have (mostly) avoided those, but sometimes we come across a treatment that someone genuinely believes works, but really the results are a coincidence or placebo effect. So how do we know that something truly works or is just a coincidence? That sunscreen reduces the risk of skin cancer, that a cast helps heal a broken bone, and that Exposure and Response Prevention Therapy helps reduce symptoms of OCD? We can look towards the above-mentioned pilots, margarine, and pigeons to help understand.

I’ll explain.

Regression to the mean

This comes from Daniel Kahneman, the Nobel-winning author of Thinking Fast and Slow². He was brought in to look at the Air Force training program. After a training exercise, the pilots who did poorly were yelled at and criticized by superiors. The pilots then tended to do better on their next training exercise. The pilots who did well received praise and then tended to do worse.

The conclusion from leadership was that criticism worked better than praise for performance. What Kahneman figured out was that this is just an example of the regression to the mean. If someone does something much better than average, they will tend to come back to average in subsequent attempts. The same is true if someone is well below average. With time, they will regress to the mean.

This tendency to approach average or “normal” is critically important when looking at wellness remedies, treatments, and clinical research. It is possible someone “got better” just with time.

Correlation vs. causation

The other key point is correlation vs. causation. Causation says that trend A caused trend B. Correlation can say that trend A and trend B are following similar trend lines, but we don’t know if A is causing B, if B is causing A, if C is causing both A and B, or if it is just a plain coincidence. These are called Spurious Correlations, and there are some interesting ones, like per capita margarine consumption and divorce³ or the number of Nicolas Cage films per year and swimming pool drownings.

Paying attention to the regression to the mean and correlation vs. causation is crucial to determining if a treatment is actually the reason for improvement or just a coincidence.

Superstitious pigeons

Possibly my favorite metaphor from a psychology experiment is the idea of “superstitious pigeons.” This is an experiment led by B.F. Skinner⁴, where they fed different pigeons at random time intervals. Each one started to think that certain behavior led to it being fed (though again, it was a randomized feeding schedule). So some bobbed their heads, some spun in circles, some swung side to side, etc.

They believed that the behavior led to the feeding because, well, sometimes it did! The times they were not fed with the behavior didn’t seem to affect the belief. This is similar in some wellness circles. People might take some treatment or eat some food/supplement and believe that the behavior led to them feeling better; however, it could be a correlation and/or regression to the mean.

Research methods

Looking across industries of health and wellness, how do we KNOW that the treatment is working and not just a coincidence? The answer for much of the research is Randomized Controlled Trials (RCTs).

Randomized control trials

Why are RCTs considered the best way to see if a treatment is effective? They are set up in a way to reduce correlation. You first get a large sample size, then randomly assign them to groups that either receive the treatment or the control group (often a placebo).

This reduces bias in selecting who will get treatment, as well as the risk that the people who sign up for the study are a specific type of person, which could influence the results.

These groups are then compared. The group with the treatment should have a significant improvement over the control group. By setting up experiments in this way, the chance of correlation can be greatly reduced, and we can have more confidence that the treatment actually is effective.

Confidence intervals

The trust in the results is then called the confidence level (or interval). A 95% confidence level is pretty good. That means if you ran the experiment 100 times, the likelihood of getting these results just by chance (and not because of the treatment) is 5%.

But actually, that means 1 out of 20 experiments with 95% confidence will actually be due to chance and not the treatment.

This is highlighted in an excellent XKCD comic about jelly beans⁵.

That’s why it is good to have over 99% confidence, large sample sizes, and to run multiple RCTs by multiple organizations with different populations.

Meta-analysis

Sometimes all the RCTs are reviewed in a meta-analysis, which can tell us if the treatment is effective across the board.

The reason that ERP for OCD is considered the “gold standard” treatment is that it has many RCTs across different populations and meta-analyses that show that it makes a significant difference with a large effect size⁶.

A brief note on the word “significant” in a scientific context vs. colloquially. In science, significance has to do with the confidence that the difference that people experience is due to the treatment and not due to chance. You can have significant findings that the effect is there, but only minor. This is sometimes called the effect size.

Conclusion

Putting it all together—when someone on TikTok or Instagram promises a new treatment that will cure your headaches, depression, pain, etc., there are ways to check the likelihood that it is true. First, we can check if it is likely that there is a correlation or regression to the mean just with time. Then we can dig into the research.

Have there been RCTs? Have there been meta-analyses of the RCTs? Is there a strong confidence level, and is the effect size high? If so, great! You might have just found a cure for what ails you. If not, it could be snake oil or a well-meaning superstitious pigeon.