Why does Wisdom of Crowds work in the first Place?
When it doesn't work, what went wrong?

In 2004 James Suroweiki wrote Wisdom of Crowds which contained some wonderful examples of how amazingly accurate answers could be gained to questions by asking lots of complete strangers things like "where should we look for a sunken submarine when we have very sketchy available info". After reading the book, I was convinced, but was bothered that there was no "why this works" chapter..and then thought about all the times it didn't seem to work. For instance, we've all heard the saw that a camel is a horse designed by a comittee. So, here's my theory on how and why it works and when it doesn't, why it doesn't. So, here's my theory. Wisdom of Crowds generates accurate answers as Multiple Mirror Telescopes focus and magnify light from distant and obscure objects.

Each individual's knowledge overlaps another's.
The first observation is that we all collect facts, which leads to knowledge and its derivative wisdom. Now some people collect lots of facts but don't convert that to knowledge and others have fewer facts but by linking these points together in network fashion are able to have surprising wisdom. If you think of each person's knowledge as if it were the circle of light created by a flashlight in a dark room, then more people's knowledge should equal more light...but this light would be all over the place and overlapping - not sharp and "point bright".

The question (and its quality) serves as the aiming and focusing agent.
So how do we "get to the point"? The question is the focusing and aiming device. By means of a well phrased question the various knowledge sets of the individuals will be focused to draw up the particular info desired from their well of knowledge. This knowledge will then be constrained to the realm of the question, creating focused answers, where the wrong answers are dramatically overcome by more correct answers - so that the average of the answers generates that point in the focus where the collective knowledge yields an answer that is usually better than a best individual answer. So that,

Addressing the question to multiple uncorrellated parties creates the equivalent of a multi mirror telescope where each respondent's knowledge and judgement are collectively brought to bear on answering the question adding greater and greater light/knowledge.
The better the quality of the crowd's wisdom, the better the results. If you want really great results about the sea floor, ask oceanographers. If you want to know the behaviour of the rich and famous, ask butlers and concierges. But...

Correllation among those questioned (market gaming/corruption, politicised debate, ideological posturing, peer pressure driven responses) generate "optical" distortion and destroy or disfigure the results.
Once the parties being asked to answer the question know earlier answers, or who among them might be asked and worst of all, they all are required to answer in public and on the record, then the collimation and focus are lost and you are then getting answers to unspoke questions that nobody asked - introducing uncorrectable distortion into the responses. What are these unasked questions? "Will I still have a job tomorrow if I say what I think?", "If I say what I think, I'll be the outlier and can lose my reputation"...so, reputation, ego, peer pressure - all militate against accuracy and truth - leading to camel/committee like results. In regard to markets, one of the great things they can provide is "price discovery" - which means that a company's shares can be prices in real time by the actions of an uncorrellated - wisdom of crowds based market where all the information known, knowable and anticipated can be "boiled down" into a current share price. The problem with this model is that money creates the motivation to bring the price discovery activity to the market...and as we all know, people will try to game the market in order to maximize their money by corrupt behavior and other activities which distort the price discovery mechanism. So, the paradox is that price discovery only happens because people "care" about the money - the incentive...and without the incentive, you can't get the discovery.

This says to me that new kinds of incentives are needed that minimize distorting influences but still provide strong incentives to participate in the knowledge discovery process. What sorts of incentives? FUN - as in serious games (see:Luis Von Ahn of CMU and others), Ego, Performanc prizes and such.

Like any good theory it needs to be falsifiable. So, if the above theory is true then it predicts that:

peer pressure driven answers will be suboptimal/inaccurate and misleading as the knowledge will be distorted by these effects
large numbers of participants who know little to nothing regarding the question will not generate remarkable results
small numbers of uncorrellated experts on the topic will beat large numbers of pressured or naive persons no matter their expertise
large numbers of uncorrellated experts on a topic will beat small numbers of same
markets provide an incentive mechanism for larger numbers of experts to participate
markets attract participants that "steal the signal" for uses other than the simple answer - gaming away much of its predictive power
wikipedia and Nasdaq are excellent examples of both the wisdom and corruption of crowds

In a future entry we'll talk a bit about the focusing apparatus - the question