The Folly of the Crowd

There is much excitement about the potential for Web 2.0, in particular what is known as the Wisdom of the Crowd. Wikipedia becomes the repository of all knowledge, Google search statistics are the zeitgeist of the times and MySpace is the face of the world. Page rank is a measure of authority. Corporations appeal to the public for solutions to problems. The ivory tower is replaced by the democracy of the commons; the proclamations of the cathedral displaced by the hubbub of the bazaar. Not so fast.

The Crowd listens to the wisdom of Oprah and reads USA Today. The Crowd doesn’t understand how their status-defining mobile phone works. The Crowd believes in Roswell and psychics and crystals. The Crowd litters the Web with blogs nobody reads; inane twitterings bereft of insight, the daily drivel that was once mercifully hidden away in a diary. The Crowd looks to rock stars and movie stars for leadership.

Majority does not equal truth. Voting is good for social decision making but not for advancing knowledge. The majority once wanted to burn witches. The majority seems to think alternative medicines are superior to pharmaceuticals. The majority thinks the outcome of Survivor is important. The majority mocked Darwin and were stumped by Einstein.

It isn’t even a majority anyway that fills the forums and blogs and wikis. It is the voice of a fanatical few. Either they hold views so strongly they are motivated to work on the content, or they have no life and spend all their time on a computer. Either way they do not represent the views of the majority. They represent the views of extremists and social outsiders. This explains why some of the most successful blogs are blogs about blogging.

The Wisdom of the Crowd is the final triumph of post-modernism: the belief that any position is as valid as another; that it is all relative to the people involved; that there is no absolute truth; that shamanism has as much to tell us as science.

As a result, the Crowd equates fame with wisdom. The views of someone who has spent a decade absorbing the accumulated wisdom of all civilisation (and passed stringent tests to prove they have succeeded) are no more important than the views of a high-school dropout who can play an electric guitar, or a drug-crazed tart who can act.

The Crowd also fails to discriminate between sources of information. Google pulls up webpages from learned institutions intermingled with the ravings of the lunatic fringes. Creationists, conspiracy theorists, new agers, nutters and racists thrive on the web, and the Crowd laps it up.
Wikipedia may be more extensive than Encyclopaedia Britannica but it will never be as authorative. Google gives us access to information but not to knowledge, let alone wisdom. MySpace and FaceBook and Wordpress do precisely nothing to advance the human race. Hailing the Wisdom of the Crowd only cheapens the Wisdom of Civilisation. Wisdom belongs among the guardians who have preserved and built it from generation to generation, the academics; not spattered across a billion websites like textbook pages thrown to the wind.

The internet has shredded knowledge. It has torn it up into digestible little pages and blended an amorphous mass that Google dips into almost at random. We don’t need to learn or remember or think because we can pull up an answer on demand. Nobody studies textbooks any more, or even reads journals. Information rains down in the patter of RSS feeds or flicks past in email headlines or babbles in sound-bite videos or scrolls by as news-pages and blog headings: so much information from so many people who know so little.

Just like me and this post.

This is not to say that Web 2.0 is a Bad Thing, nor does it argue that community involvement is counter-productive. The message here is to avoid putting the Crowd on a pedestal as some font of wisdom. It is an unreliable unpredictable superstitious mosh-pit of ideas and data: potentially fruitful if managed; potentially dangerous if idealised and idolised.

Scale

The key is to manage the Crowd, and the main thing to watch is scale. On a large scale like Wikipedia, the Crowd does average out to something approaching a stable depiction of accurate information. An excellent example is the entry on homeopathy which had a wild ride before settling down to an objective rational entry, though debate remains brisk and may never end, because the Crowd is an incubator for ideas good and bad, scientific and superstitious.

The idea of community knowledge becomes more problematic as the Crowd gets smaller. If we look at a niche community such as ITIL practitioners - closer to home for readers of this article - then even the Wikipedia entry becomes the work of a handful of people, and specialist wikis are either the work of their original founders or die from apathy.

Bring the community down to the even smaller population of the staff of an organisation and the whole idea of the Wisdom of the Crowd starts to fall apart.

Wiki-style help or documentation systems work only with the energy of a responsible owner and/or an enthusiastic few. They tend to be patchy, difficult to navigate, rapidly outdated and eventually fall into dusty disuse (like just about every knowledge repository within organisations).

Discussion mechanisms are mildly useful in a geographically-dispersed company, but email serves just as well. More importantly, email doesn’t get displaced by the forums, so discussion tends to get fragmented between the mechanisms. In theory a forum is an open discussion and email a closed one, but in all but the largest and most dispersed organisations the practical difference is moot. The forum mechanism seems to provide a useful long-term record of what was discussed, but who will find it? Only those who remember what was said, or a lucky few who have the time and tools (e.g. Google Desktop) to be searching it. Open archiving of the emails would serve just as well.

Moving back up in scale to the internet, forums do provide value, and the most extraordinarily useful titbits turn up on Google. Unfortunately once again these need to be treated with extreme caution and treated only as suggestions because they are subject to even less review and debate than wiki entries.

The forum equivalent of Wikipedia is probably Yahoo! Answers, and even with the largest possible Crowd this provides information of mixed quality. The factor at work is again scale, and more so with forums. The forum world is highly fragmented and each Crowd is small. Taking ITIL as an example again, most ITIL-oriented forums are moribund. Of the few active ones, most are full of newbies asking inane questions and self-appointed experts providing questionable answers. Only the forum owned by the itSMF has middling activity level and attracts authorative participants, precisely because it is not owned or moderated by the Crowd.

Filtering

Even in the best of cases, the useful information is still polluted by misinformation. So the second consideration after scale is filtering. On the internet there is sometimes filtering in the form of moderation. There is occasionally filtering in the form of an expert review panel. The vast majority of websites have no filtering other than the discriminating ability of the reader, and sadly in the post-modernist world critical faculties are being destroyed not developed.

Within an organisation, there can be much better filtering. Staff are accountable for what they say, and anonymity may not be permitted. Resources can be assigned to review and quality-assure the information. In some cases an open forum is desired – to stimulate debate and creativity – but mostly the accuracy and reliability of the information is paramount. The wisdom of the organisational Crowd is only as good as the people who own and manage it.

In summary, the Crowd does gather data. That data gets turned into information if it can be searched adequately. That information tends towards useful reliable knowledge if it comes from a large enough community and is subject to sufficiently robust filtering. In rare cases the Crowd may accidentally produce wisdom, but only if the reader has the capacity to separate it from the dross.