Making the most of the numbers – 1st August 2008
November 4, 2008

Information is now easier to capture because it is in digital form – rather than a person having to tabulate figures by hand (which is what I was doing in the War Office in London in the 1960s as a junior clerk). The progress in developing computers has enabled this revolution to progress rapidly. Information is now not only easy to acquire and store – it is virtually costless to copy.

Business and government professionals are relying more and more on databases to guide their decisions. It is a story about new breakthroughs in advances in technology – rather than any dramatic breakthrough in the statistical art of prediction itself.

The size of the computer databases is amazing. One of the world’s largest libraries is the US Library of Congress. The Wal-Mart chain of stores has an equivalent computer data base 28 times larger than the Library.

Ian Ayres has written “Super Crunchers: How Anything Can be Predicted” (London: John Murray, 2007). Increasingly the people who operate off their gut instinct – “intuitivist’s” – are being beaten by the Super Crunchers who can provide the figures to refute the so–called commonsense claims of the people who rely on intuition and skill.

Before Hurricane Ivan hit Florida in 2004 Wal-Mart already had started rushing strawberry Pop-Tarts to stores in the hurricane’s wake. Wal-Mart knew from its Super-Crunching research that Americans would be yearning for the gooey comfort of Pop-Tarts, finger food that does not require cooking or refrigeration.

Harrah’s casinos know their clients. They know the “pain point” of each regular client and when a person is getting close to losing too much. Each person’s gambling transaction is being fed back to a giant computer that issues the warning signal to management. As the pain point is being reached, a casino staff person will approach the person and give them a voucher to get a free meal. It is no longer a pain – it has become a good experience.

Ayres also writes about the Super Crunchers who are able to predict the outcome of cases before the US Supreme Court. They have been able to identify six key factors. In a 2002 project, the Super Crunchers correctly predicted the outcome of 75 per cent of the cases, while a group of legal experts only scored 59.1 per cent relying on their intuition and skill.

We are living, then, in an era where statistics are increasingly important. I have had 11 years on the Australian Government’s National Community Services Data Committee (indeed I was one of the original members). Australia is an international pioneer in health and community services data dictionaries. The work is co-ordinated through the Australian Institute of Health and Welfare, Canberra.

Good statistics require good definitions – you need to be sure of what you are measuring. Australia’s two dictionaries – now freely available online – are helping the smooth running in both health and community services. The Prime Minister Kevin Rudd talks about “evidence-based” government – the data dictionaries are a very important part of that process.

I have just retired as Wesley Mission’s representative on the National Community Services Data Committee. It has been a memorable 11 years because of all the progress that has been made.

Sure, the business community has made great use of its own customer information – but there is also a great scope for health and community service providers to also make great use of the information revolution.