« Top Ten | Main | A Likely Story »

Our Biases

Over the past few days, I've been reading this book...

...which is quite entertaining and eye-opening. I highly recommend it to anyone who doesn't mind seriously messing with their perceptions of how much we truly understand of what's happening in the world around us. It turns out we have this tendency to put patterns around random events. I mean, sure, we all knew that, but I had no idea how pervasive the problem is.

While reading, it has occured to me that the author would probably not have much use either for the content of this site or L2si. We suffer from some pretty severe biases on these blogs. First off, we consistently engage in...

Selection Bias

Selection bias, sometimes referred to as the selection effect, is the error of distorting a statistical analysis due to the methodology of how the samples are collected. For example the sample selection may involve pre- or post-selecting the samples that may preferentially include or exclude certain kinds of results. Typically this causes measures of statistical significance to appear much stronger than they are, but it is also possible to cause completely illusory artifacts. Selection bias can be the result of scientific fraud which manipulate data directly, but more often is either unconscious or due to biases in the instruments used for observation. For example, astronomical observations will typically find more blue galaxies than red ones simply because most instruments are more sensitive to blue light than red light. If the selection bias is not taken into account then any conclusions drawn may be invalid.

So for example, when I report that nearly 50% of Speculist readers believe that the first permanent moon colony will be established by private developers, I am failing to take into consideration the fact that readers who aren't interested in space -- or who hate online surveys -- might significantly skew the data in a different direction if their perspectives were included. Or put another way, space enthusiasts might might be more highly motivated to respond to the survey than others, with private-space-development-enthusiasts the most highly motivated of all. In that case, all the poll has shown is which group is the most motivated to respond.

Of course, what mitigates this bias is the fact that we never claim anything remotely like statistical significance to our surveys. They are primarily intended for entertainment and to generate discussion. But then there's the interesting matter of...

Survivor Bias

Survivorship bias (or "Survivor bias") is a statistical artifact in applications outside of finance, where studies on the remaining population are fallaciously compared with the historic average despite the survivors having unusual properties.

Mostly, the unusual property in question is a track record of success (like the successful funds). For example, the scrupulous parapsychology researcher Joseph Banks Rhine believed he had identified the few individuals from hundreds of potential subjects who had powers of ESP. His calculations were based on the improbability of these few subjects guessing the Zener cards shown to a partner by chance.

In the book, Taleb talks about the Survivor bias displayed in a book like The Millionaire Next Door, which purports to identify the characteristics of regular folks who make good -- frugality, optimism, and enthusiasm -- without taking into account how many of us non-millionaires next door possess precisley the same qualities. Or to put it in Speculist terms, a very strong rebuttal to our Better All the Time thesis is that it ignores all those for whom the world is not getting better or that it is merely a gussied up "survival of the fittest" tautology. Take those penguins, for example. Sure life is pretty good for the ones who got cleaned up. But what about the countless other waterfowl over the years who haven't been so lucky where oil spills are concerned?

Bon voyage, liitle birdies indeed.

On the other hand, unlike the Millionaire Next Door argument, we need the failure cases to be there in order for our argument to make any sense. The fact that lots of birds have died from oil spills in the past helps make the case that now, when even a handful of penguins receive the care and attention they need to make it through the experience okay, we've turned some kind of corner. The Better All the Time principle might be restated in survivor bias terms. The universe itself seems to have a strong survivor bias (or is it a selection bias?) in favor of us, as reflected by concepts like the anthropic principle and the law of accelerating returns.

The numbers don't seem to add up on their own. Am I being fooled by randomness? Very likely I am. But then again, maybe some patterns really are real, after all.

Post a comment