On the Wisdom of Crowds

The Wisdom of Crowds is an interesting phenomenon. Here are some articles on the topic:

The last of these articles is of particular interest. It explicates my earlier hunch that the Wisdom of Crowds phenomenon has to do with something like the law of large numbers:

Surowiecki’s archetypal example comes from a 1906 county fair where 800 people participated in a contest to guess what the weight of an ox would be after it was butchered. The average guess was 1,197 pounds. The actual weight turned out to be 1,198 pounds. On its face, this seems like a dramatic testament to the ideals of democracy, but the accuracy of the average guess has much more to do with the nature of the problem than with the wisdom of the crowd.

Their task was clearly-defined and required no special information. Each person was free to guess any weight they wanted, but the higher or lower their guess, the more obviously wrong it would be. Random variation ensured that every high guess was counter-balanced by a low guess that was equally off the mark. After 800 such guesses, the average would stick right in the middle. In this case, the average happened to be the truth.

You can tease the same kind of wisdom out of a handful of dice. Say you hold a contest to guess the number you’re thinking of: 3.5. Only six-sided dice can enter this contest and, therefore, all guesses will range from 1-6. (Note that each die is physically incapable of guessing correctly, as dice can only express whole numbers.) Each die can enter the contest as many times as it wants and, eventually, you gather several hundred entries. Miraculously, the average “guess” is exactly 3.5! Again, the average just happens to be the truth.

The trick is that truly diverse (i.e. random) opinions will always vary around the mean. When you aggregate a whole lot of random opinions, you get a deceptively precise average, but this is not “wisdom” in any real sense. It’s a statistical artifact called the Law of Large Numbers and it has nothing to do with intelligence.

There are two frustratingly common factors that throw this trick right off the rails. The first is communication, as discussed above. It leads to the primacy effects and power law distributions that plague news aggregator sites. The second is bias that arises from common wisdom… or lack thereof.

What if you asked a crowd to answer the following well-defined question: “What is the distance to Alpha Centauri?” Because astronomical distances are so much larger than anything in a normal person’s experience, their guesses would probably fall short of 25 trillion miles. (An astronomer, on the other hand, would be right on the money.) In this case, the average just isn’t the truth.

Leave a Reply