"There is a 5% or less chance that we would see this result happen if there was really nothing going on"
This standard makes sense when you are running laboratory experiments, but it can be very misleading when applied to correlations you observe in the world. If you take a hundred surveys on things that have nothing to do with each other, then five of them will produce results that are 'statistically significant'. This is for exactly the same reason that if a hundred people go to Vegas and play the slot machines, five of them might get lucky and come home with more money then they started. The world is full of randomness. This comic illustrates the effect well.
It appears that the writers and editors at The Economist do not understand this fact:
According to his "Guide to Complementary and Alternative Medicine", around 95% of the treatments he and his colleagues examined—in fields as diverse as acupuncture, herbal medicine, homeopathy and reflexology—are statistically indistinguishable from placebo treatments. In only 5% of cases was there either a clear benefit above and beyond a placebo ..., or even just a hint that something interesting was happening to suggest that further research might be warranted.
Finding a 'benefit' in 5% of cases is exactly what you would expect to find if all of the treatments were just placebos. The article is pretty good otherwise, but they realy should have pointed this out. As printed, people might think that 5% of these things are actually doing something, and naturally they will assume that whatever they believe in is one of the things that has been 'proven' to work.
No comments:
Post a Comment