Game ratings don't matter
by Vladimir Cole 
{ Dec 8th 2005 at 11:59AM }
Research analysts Jason Kraft and Chris Kwak examined a sample of 275 video games released over the last five years and analyzed the ratings data for those games. They determined that the correlation between ratings given to games by reviewers and the subsequent sales of those games is statistically insignificant. The chart above demonstrates this. On the Y-axis are unit sales. On the X-axis is the average rating of various titles.
In their own words: "After going through multiple scenarios, we believe a game rating, in most cases, is not a reliable tool for predicting game sales…. The next time someone suggests that a game will shine or fade based on a rating, we encourage the reader to take it with a grain of salt."
So what does affect sales? The best indicators of sales strength appear to be the strength of the publisher behind the game, the performance of prior iterations of the game (if it is a sequel) and the performance of the box-office sales of the movie version of the game.
Why? We're guessing it's because the hardcore gamers who tend to pay attention to ratings have very little impact overall on the financial performance of a game. Consumers who don't read gaming publications dominate as a percent of the gaming population.
Send an email to kraftandkwak [at symbol] sig.com for a copy of the report.
[Image credit: Game Ratings correlation chart from Susquehanna Financial Group analysts Jason Kraft and Chris Kwak]
Reader Comments (Page 1 of 1)
bd @ Dec 18th 2005 9:06PM
Videogames are a business too, and without the right marketing, a great game can sell poorly.
On the other hand, shitty games with great licences sell a lot. Just look at games from movies or games from EA Sports.
We also forget that most of the game raters are hardcore gamer, so some "great games" maybe don't appeal to the mainstream crowd.
Jayman16 @ Dec 18th 2005 9:06PM
Where the hell is Halo 2 on this? Or any Xbox game for that matter?
GlitchCog @ Dec 18th 2005 9:06PM
Maybe it has something to do with the fact that lots of game ratings are crap? There's too much bias and too much marketing hype. If a scale goes from 1 to 10, then all those numbers need to be used. If you just reduce the scale to 6 to 10, game review ratings start to make more sense.
Based on game reviews, you might think that 96% of all games were awesome. This is obviously not true.
Justin Tischler @ Dec 18th 2005 9:06PM
Given the shape of the data, it's not surprising that they concluded the results were insignificant - they used a standard linear regression. Had they used a log-log regression (examine the natural log of the dependent variable versus the natural log of the independent variable) and then run the same regression it likely would have had statistically significant results. These new results would be interpreted as an elasticity - a 1% increase in average review score will lead to an x% increase in sales.
Notice the nonlinear shape (sharp upward trend) toward the right hand side of the data. Very high scores generally lead to higher sales - the changes I have suggested to their very basic model would no doubt show that.
Tighty Whities @ Dec 18th 2005 9:06PM
"the hardcore gamers who tend to pay attention to ratings have very little impact overall on the financial performance of a game"
This fact that hardcore gamers don't make or break a games success could come as a shock to the hardcore out there but its true. The casual gamer is the majority and thats where the money is at. Success with the casual gamer = overall success. The hardcore gamers out there are just too few and far between. I would say maybe 5% of gamers are actually hardcore. Maybe Nintendo aren't so insane afterall?
Toast @ Dec 18th 2005 9:06PM
Looks like we need the average gamer to be a little more discriminating on which games that they buy. (i.e. good, innovative games, instead of just titles with familiar names and sequels.) Orwell was right indeed. "If there is hope... it lies in the proles"
AK @ Dec 18th 2005 9:06PM
This is why there are so many craptacular titles out there. Because people buy them!!!
MumbleyJoe @ Dec 18th 2005 9:06PM
Thank you Justin for pointing out some of the many over-simplifications of the analysis. Do you see any poorly scored games selling as much as GTA? No. Does it factor in the installed base of the consoles upon which games are made available? No. Is the scoring of each game done using the same method? No.
Man, it's visually apparent that there is some correlation, but its obviously confounded with a number of other factors.
I don't even care about the discussion, I'm just irritated by the oversimplification and analysis.
SickNic @ Dec 18th 2005 9:06PM
This is exactly why we get games like "Mary Kate and Ashley hourse adventures", and builder bob. Because idiots keep buying them...
recklessgemini @ Dec 18th 2005 9:06PM
meaning... the majority of you guys that post in here don't mean shit to the industry
Stretchpants @ Dec 18th 2005 9:06PM
It's just like movies, man. Critcal acclaim does not translate instantly into $$$. It's all based on marketing, demographics and star-power. In the case of video games, replace star-power with "character"-power and "developer"-power.
Easy as that.
CornPuff @ Dec 18th 2005 9:06PM
Can we get a link to the actual data or report? I take it you have to be a paying customer of the research firm, but I'm really curious. We have that one graph, I want to see more!
JoeAverage2 @ Dec 18th 2005 9:06PM
Actually, if you read the report, it's clear that the analysis is far from oversimplified. Email the analysts kraftandkwak@sig.com.
Most people oversimplify when they say there is a correlation.
Also, a log-log regression doesn't seem like it would change the results. Frankly, there are only a handful of games where it would be germane.
They say "A cursory look at video game reviews suggests that titles with poor ratings generally sell poorly, and blockbuster
titles generally have superb ratings. Often, this leads readers, investors, and analysts to conclude there is a correlation between game ratings and game sales. But if this is the upshot of game ratings that blockbusters
have high ratings and poorly rated games sell poorly then the thesis that ratings are a predictor of game sales is uninteresting."
Read the report.
BenR @ Dec 18th 2005 9:06PM
If score had no effect, wouldn't there be as many games in the top left corner of the graph as in the top-right?
vc @ Dec 18th 2005 9:06PM
Folks -- there's a lot more to it than I can summarize in 5 paragraphs. Request a copy of the report and read it for yourself.
The ZeroCorpse @ Dec 18th 2005 9:06PM
BenR - I guess statisticians don't "count" a few exceptions like the Maddens or GTA series on the upper right. There would have to be a larger set number up there in order for a number-brained dork to believe it means anything.
Which is why I hate mathematicians. `Buncha number-crunching jackasses...
Jeff @ Dec 18th 2005 9:06PM
"If score had no effect, wouldn't there be as many games in the top left corner of the graph as in the top-right?"
Not if there are also more games on the *bottom* right than on the bottom left. Which it appears there are.
I'm no math wiz but it seems to me that the main story is not that poorly rated games sell so well, but that highly rated games often sell so poorly. That's what's skewing the results.
But if you take that as an assumption (which I think you can assume, from the graph), then you have to look at what games and genres are getting high ratings, vs. the games and genres getting low ratings. It could just be that a lot of highly-rated games appeal to a particular niche. They may appeal to that niche extremely well, but if it is not a mainstream niche, the games won't sell.
So, the upshot of that would be that analysts aren't wrong after all, because an analyst would be looking at a game like that and already know what the built-in audience is. If they know a game only has a potential market of 200,000 interested people, and it scores a 90%, they're not going to suddenly think it's going to sell 5 million copies. They're going to think it might sell 180,000, because they've taken genre into account.
Poorly rated games, on the other hand, never seem to break out above that 0.5 million barrier, regardless of genre. So while the correlation may be "statistically insignificant" based on raw numbers, it probably is not when you take other factors into account.
Joseph @ Dec 18th 2005 9:06PM
As gamers, she we care about sales in making our gaming decisions? Nope. It just means that there are a lot of uninformed gamers who don't read reviews, but only buy games based on name brand recognition, or the movie it's been licensed off of.
bd @ Dec 18th 2005 9:06PM
I want the report, because I thought that Resident Evil sold like 5 million copies in two months.
Art Guy @ Dec 18th 2005 9:06PM
Look at box office receipts compared to reviews. It's all about marketing and perception. A good game/movie with great reviews and marketing will sell great. But marketing and perception can make people pay money for crap while good entertainment goes broke.
These aren't the best examples, but from the current box office:
Chicken Little got awful reviews (36%) but made over a hundred million, while Wallace and Gromit got great reviews (95%) and only made half as much.
Kiss Kiss Bang Bang, got great reviews (81%) and only made 3 million.
Aeon Flux only got 11% but has already pulled in 12 million.
Scott Vieth @ Dec 18th 2005 9:06PM
I'm just going to make an assumption here ... I think it's a PS2-only list.
Considering I don't see any XBox titles and Resident Evil 4 did sell like hotcakes ... on Gamecube!
Adam @ Dec 18th 2005 9:06PM
Another reason is because bad reviews are simply a symptom of the disease: bad quality. Meaning lower reviewed games are usually bad quality games. These games sell poorly because of a large number of factors - bad marketing, no word of mouth, etc.
Also, I'm glad someone pointed out the logarithmic trend of the data - as it would seem quite obvious. However, what you see is a few of the games reach the upper right portion of the graph, while most are still clustered near the bottom. You don't really see a general logarithmic trend. For this reason, the R^2 would likely not increase too much if the curve is fit to a log curve.
This would be extremely easy to check if someone was willing to enter their data into Excel.
SuicideNinja @ Dec 18th 2005 9:06PM
The first comment is probably spot on.
There are a lot of great games gamers of all types have never heard of. Reviews, commercials, word of mouth, name familiarity, game store recommendations and system popularity all play in how well a game sells.
I had never seen an ad/commercial for Resident Evil 4 until it came out for the PS2.
Sean DL @ Dec 18th 2005 9:06PM
But you can't compare Kiss Kiss Bang Bang to Aeon Flux. Aeon Flux is getting played in 90% of the theaters, while Kiss Kiss is a indy film and is in about less then half of the theaters Flux is being played in.
Chicken Little and Wallace and Gromit was the better example. Both had an equal playing ground except for one had the Disney(EA) hype and the other had credit hype. Not to say Wallace and Gromit flopped, it made more money then it would of if it wasn't for such glowing reviews.
Thing is Chicken Little 2 won't do asd good as say another Wallace and Gromit movie will. Why? because the money it made was because people liked it.
That's how they judge stuff like that in Hollywood. We can't really do that in gaming....yet. But one day it will happen, people will say "Hey this was made by EA, it'll suck, don't buy it" even if the game doesn't and this didn't come from a "Hardcore gamer". It's a future the gaming industry will have to get ready for.
Todd @ Dec 18th 2005 9:06PM
I tend to believe that out of all gamers everywhere the percentage of gamers that constantly keep up with news, previews, & reviews is around 25%, but definately no more than half. Out of everyone that I know that has a game system of some kind very few care about keeping up with what the industry is doing. Most only find out about a game they want through newspaper ads, tv commercials, or a major magazine. They might see it in a store and look at the back of the case & get it. If it's good then they'll keep it, if not they'll trade it in on something else.
Hmm, if Nintendo can dazzle the average gamer enough to get them to buy a Revolution & keep them interested enough to not trade it in, then this data would suggest that Nintendo could do really well even if the hardcore gamer thought the control was crap. That'd be kinda ironic wouldn't it?
LaughingTarget @ Dec 18th 2005 9:06PM
Ratings probably do have an effect on sales. Not the only effect, but farily significant depending on pre-existing popularity.
Look at the titles that are "high" on the list. Star Wars, Matrix, Dragonball Z, Madden, GTA. All these titles have in common is pre-existing popularity. You'll also notice that the sales curve upward from Star Wars to GTA.
It would appear when dealing with a popular title, ratings become a significant factor over when dealing with an unknown title. The casual gamer is likely going to read the reviews of games that the casual gamer is aware of. Specifically search for them. They are not likely to notice good marks for Resident Evil, Beyond Good and Evil, or Shadow of the Colossus because they aren't looking for them.
Take a look at Madden 2004-06. The sales have been fairly constant between the three games. While the scores have declined each year, the constant sales can be attributed to a greater market presence. Madden 04 was scored higher than 05, but was not as well known as 05. Same from 05 to 06.
So, it is most likely that reviews DO have an effect on game sales, but only among those who actually get around to reading them.
Adam @ Dec 18th 2005 9:06PM
Just remember that correlation does not necessarily imply causation. Just because Resident Evil and GTA scored high AND sold well doesn't mean they sold well BECAUSE they scored well.
Slashbunny @ Dec 18th 2005 9:06PM
Ah yes, more proof that noone cares what hardcore gamers think. So people wetting their panties on here and on other game forums online going over every little detail about every little system means nothing. The people with the buying power, aka non-hardcore gamers, don't care about these things. It's word of mouth, marketing and image. Has Xbox360 done this? Is it "cool" or "essential" to have one? How will PS3 compare? Couldn't the PS3 do well just on the fact that it's a sequel to a good system and a very recognizable name? Certainly technical specifications don't matter, that is clear. Not like I had to point that out anyways. The system with dominance is the least powerful of the 4 systems released this gen. But of course we forget that every generation that technical superiority means nothing. But everyone still bickers about it.
White Rose Duelist @ Dec 18th 2005 9:06PM
Todd - not ironic, really. That's Nintendo's strategy for the Revolution. Well, minus the hardcore gamer thinking the control is crap. Who cares if a few tens of thousands of hardcore gamers get the Revolution if 50 million casual gamers pick it up and another 20 million people decide to make it their first gaming console?
/hardcore gamer who doesn't think the control is crap
Clay @ Dec 18th 2005 9:06PM
You cannot just pick and choose which games to put on the graph! If you want to show that "Game ratings don't matter" you must take a random sampling of games and then put their scores vs price on there.
The only thing you have shown is that the games you have selected do not correlate their sales to their ratings.
dre @ Dec 18th 2005 9:06PM
if that graphic fully reflects the results and the study, then the whole survey is crap. how can they possibly rate the sale of games equally when the games are on different platforms. at the minimum they need to adjust for install base. RE4 may have sold few, but may have ALSO reached a larger percent of gamecube owners than GTA did of ps2 owners. How can you possibly rank games sold on a system with 20 million systems sold against games on systems with 80 million sold against games on all three systems? further, what criteria were used to pick a sampling of games? this is clearly not the population of games, and given the ignorance displayed in the use of data, i am suspect that their sampling is normal and unbiased. and finally, as has been mentionned, linear correlation does not make sense. There are plenty of consumers for whom the likelyhood of a purchase of a game rated below, say, 50% is the same- it's ZERO (or near it).
matt @ Dec 18th 2005 9:06PM
Whoa!!! look at the R^2 score. For anyone who knows statisitics, that is an extremely low score. An R^2 of .17 means the cirteria is only accouting for approcimately 17% of the variance in that data.
Where are the T abd b scores? Give us that and we can tell how well of a correlation there is.
There was also no indication if the researchers ran a multiple regression which not only would have controlled for other variables but also measured their impact. Man this is basic stats here. I wish we had access to this data, we could run it through SPSS in about 5 minutes and get some better results
Mazinkaiser @ Dec 18th 2005 9:06PM
I feel like saying DUH! but people will just get mad at me. So instead I'll say, have you ever enjoyed a movie that was reviewed badly? Have you ever hated a movie that was supposedly the hit of the year? So then what's the difference? A rating is biased to the reviewer's taste and doesn't nessecarily reflet those of the gaming community as a whole. Some reviewers are clearly not even gamers, thats why rental stores do so well!
dre @ Dec 18th 2005 9:06PM
"Have you ever hated a movie that was supposedly the hit of the year? So then what's the difference?"
The idea, I'm guessing, is that using a meta score creates an average that weeds out abnormal reviews. Since the score is based on many reviews, it should be an unbiased estimator of the consensus on a game's quality. Of course, this is less meaningful for games reviewed very differently and for which no satisfactory consensus exists (see Killler 7), but should be satisfactory for most games.
KilgoreTrout XL @ Dec 18th 2005 9:06PM
I want F tests and T tests on this data immediately even though I can't for the life of me remember how either one works or what either test will reveal!
dre @ Dec 18th 2005 9:06PM
"Also the DS itself is selling, although it's a piece of big silver crap."
way to be a ps fanboy. the link is a nice touch.
Mazinkaiser @ Dec 18th 2005 9:06PM
dre,
I get the idea as you explained it, but as I said, in the end it means very little to me. I feel comfortable making my own judgements and I don't care what mr x said about it. I think marketing makes a bigger impact then reviews. When I see a trailer, I decide if i'm interested in seeing the movie; its that simple for some people. I also think who made the game plays a bit of a role; you know which companies make games you typically like. Reviews are helpful when you read some of the details from the person who played the game, like frame rates dropped, but i don't bother giving much credit to the score!
Melanieh @ Dec 18th 2005 9:06PM
I think the study focuses on PS2. It seems like they also look at attach rates too.
Craig @ Dec 18th 2005 9:06PM
Presumably someone already said this, but it appears that the number of ratings is a more important factor than the ratings numbers.
Justin Tischler @ Dec 18th 2005 9:06PM
Since I don't have access to their data, I input estimates into Excel based on the detailed scatterplot. I ran a semi-log regression (since review score was already in percentage form) and my results showed a 4.5% increase in sales for each 1% increase in review score. However my R^2 of 0.21 is still low, and the model would likely be better served with some other relevant explanatory variables, such as publicity (as Craig alludes to through number of reviews), marketing expenditures, time since release, and dummy variables for season of release, genre, and others. Controlling for these factors would more appropriately isolate the relationship between review score and sales figures.
Tanner @ Dec 18th 2005 9:06PM
Okay look people a logarithmic graph wouldn't curve up that much at all. This is because like adam said there's a massive cluster of low selling high rated games that would prevent the line from curving up. Those few(about 6 or 7) really high selling high rated games are pretty much just outliers, and remember none of them were new franchises,(although gta3 was a huge step forward from 1 & 2 it was still highly publicized). Because of this it would just go to further prove their hypothesis rather than void it. I'm sure they tried all the possible equations, graphed differences, and compared r^2 you know all that statistics jazz. I thnk what we all need to do ourselves is make sure that all of our friends are in the educated(i don't think you have to be a hardcore gamer to not buy crap) gamer category.
JRT @ Dec 18th 2005 9:06PM
To # 41: You sound like an educated person, but I think you have too much faith in the statisticians. I am not a professional statistical analyst, but I do have common sense. The bottom line is that there is a significant correlation between extremely high ratings and the ability for the game to be a HUGE seller. By any sensible definition, this is a "significant" correlation. Now, there also may be a correlation between really high ratings and significantly low sales for some reason. But discounting the positive correlation I just mentioned by mentioning the additional correlation between high ratings and low sales is a fallacy of causation reasoning because you have to be assuming that the high scores caused the low sales as well as the high sales. This is ridiculous. Statistics aside, noone chooses NOT to buy a game because it was highly rated. This is the piece that the statisticians are missing. You have to look at statistics with common sense in mind. Read "Freakonomics" if you don't already know this. It's a great book. Or, you can download the torrent of the audiobook as I did. Some great factoids in there.
JRT @ Dec 18th 2005 9:06PM
Another thought: I mirror the thought already expressed by a few: the bleeting masses really get on my farking nerves. As it is with games, so it is with politics and everything else. If you want to blame someone about our society, don't start with the individuals that people place in power (whether it's with their votes or their dollars (same same)), start with your idiot parents or idiot siblings or idiot friends.
The interesting question is what this conclusion means going forward. The most interesting conclusion is that the vast majority of people will never think for themselves, so there is little difference in teh end between political setups. The only difference is the path that needs to be taken to gain control over the masses. In communist states, it was struggling through the beauracracy to be head of the Party. In republics it's about appealing to the vote of the common idiot denominator. In fascist societies it's about forcing yourself into absolute power or gaining influence with the few in power. My conclusion: Plato was right: the ideal society is an oligarchy ruled by benevolent philosopher kings. This isn't possible now, but perhaps later when we learn more about DNA we can start classifying people at birth and get the ball rolling towards Utopia!