One book that I am really looking forward to is Haruki Murakami’s IQ84. It has been out in Japan for a while but I am still waiting for the English translation to arrive. A little while ago it turned up on Amazon and I was very close to buying it but I couldn’t see anywhere on the Amazon page specifically stating that it was the English version. A few days later a one star review appeared complaining that the book was in fact in German.
The Amazon five star review system has always seemed to me to be of dubious value and this review highlighted some the inadequacies with it. But is it more good than bad?
On the face of it the Amazon review system is nothing but a good thing, giving a critical voice to anyone that wants to use it and a wealth of information for those looking for it, but there are a number of things that would be good to keep in mind when using it.
The five Amazon stars are defined as;
One star – I hate it
Two stars – I don’t like it
Three stars – It’s OK
Four stars – I like it
Five stars – I love it
But what do these ratings really mean and how do people really use them? The definitions are far from clear. The words OK, hate and love are hugely prone to personal interpretation. This is one of the reasons why I don’t write Amazon reviews. It’s not like an Ebay review where all you’re really doing is stating whether or not the shipper managed to get the item to you in a timely manner. An Amazon review encapsulates a wide variety of different aspects of the item in question and somehow all these different aspects, the quality of the book, the quality of the writing, the quality of the printing, the quality of the service needs to be summed up in a simple rating out of five. Is it fair to give a novel a one star review because the Amazon staff failed to state that it is in German?
Possibly. It is after all of huge importance. But that review is now forever linked to the book and will be taken into account in the statistical average of all reviews that go to make up a books score. This is another aspect of the Amazon review system – and not just Amazon, also all of the meta-critic reviews of films and video-games – that is of questionable value.
Without wanting to get too into statistics, partly because it is dull and mostly because I would probably get it wrong, these overall scores seem worth thinking about. The overall score is simply the mean average of all the reviews. This means that if someone else comes along and gives it five stars because they can read German and also loved the book the new overall score of the book is three. Now, this probably isn’t going to be an issue with IQ84 because in time it will get a lot of reviews, seeing as Murakami is a very popular author. The significance of a mean is dependant on the amount of information that is going into calculating it. A less popular author or a debut novelist is going to attract a lot less attention and so a one star review based on something out of the authors hands could drag the overall score down significantly.
But do people really judge a book by its overall score? I have no idea but I am going to assume that a lot do based on how ubiquitous scores like that have become. From what I understand Metacritic has become quite feared and respected in its associated industries. But one of the problems of a mean average is that the final number doesn’t necessarily have any relation to the numbers used to calculate it.
The old example of this that I remember is the town planner who has to figure out how many bedrooms the new houses he wants to build should have. If the mean number of children per family is 2.4 do you build a lot of houses with three bedrooms and slightly fewer with four? What about the weird outlier families with no children or ten children? What if this town is solely populated with families of one and five children and no one has any use for a three bedroom house? A mean average doesn’t really help.
Here’s an example from Amazon. At the time of writing Dan Brown’s The Da Vinci Code has the following ratings.
One star – 724
Two stars – 417
Three stars – 546
Four stars – 683
Five stars – 1644
The average score it gets off of these numbers is 3.5
I chose The Da Vinci Code because I know it is both popular and also much derided and that is represented in the hard numbers. A lot of people ‘loved it’ and the next highest group of people ‘hated it’. The mean score however would suggest that the book is somewhere between ‘OK’ and ‘likeable’. The average tells us nothing. A more telling fact of the book, rather than whether or not it is any good, is that it can provoke quite strong reactions from its readers.
Now this doesn’t mean that the 3.5 average is entirely useless. In fact 1646 reviewers did not give it either a one or five star review and 1229 people rated it according to the final average. On the other hand 2368 – the number of people that either ‘loved’ or ‘hated’ the book would say that The Da Vinci Code is anything but ‘OK’. Statistics is a famously malleable way of analysing numbers.
But what are the alternatives? If I were interested to discover whether or not The Da Vinci Code is a good read am I really going to read all 4014 reviews? Probably not. Without a star system to categorise the reviews neatly would any of those individual reviews ever stand much of a chance of being read at all? Not that The Da Vinci Code is a particularly good example, being extremely famous to begin with. But a lesser known novel with a fraction of these reviews is a different concern.
Having spent a lot of time ploughing through Amazon reviews it is astonishing how many people feel justified in giving a one star review for a single negative criticism or a five star review without any real explanation of why. Should the old adage be changed to ‘don’t judge a book by the mean average of a statistically insignificant number of short reviews’? Probably.
toby wallis 