IGN, Gamespot review score inflation revealed
Metafuture is a new game review watchdog blog that inflates or deflates a review score based upon the reviewer's history of awarding that score multiple times. The analysis of both IGN and Gamespot reviews (pictured) revealed a large rightward shift of the good 'ole bell curve, possibly confirming what many have believed for years (cough, cough; "get'n pay'd to review games").
IGN's average review score was a whopping 8.0, and the site heavily favors evenly rounded scores. Gamespot, while a little more balanced and even in their curve, has an average review of 7.0. So either there are a butt load of great games we're overlooking, or something fishy is going on.
[via Kotaku]
Reader Comments (Page 1 of 2)
Chasmodai @ Aug 7th 2006 1:22PM
Why lump gamespot in on that lame complaint about too many high scores? If their review is a 7.0, that doesn't sound too positive to me. Last I checked, a 7/10 was only 70% -- which would have hardly gotten you a C- in school...
KawF @ Aug 7th 2006 1:25PM
As with printed magazines, I'd doubt that even the bigger online counterparts really give a rats ass about the 2-man sharewaregames with stink written all over them. The same for games that have actually been developed for a a bit bigger publisher.
And then, it just might be something as simple at publishers just don't release the really bad games simply because... well who'd buy them.
But conspiracies are much more fun to dream up.
Vazel @ Aug 7th 2006 1:25PM
Is it too much to believe that there just aren't that many horrid games out there? Even many less than stellar games out there still have some enjoyable redeeming aspects to them.
I also wouldn't say 7.0 or 8.0 signifies a great game. Great games get 9+ scores.
As far as i'm concerned this is yet another dumbass post by Joystiq. Oh well, you people are good most of the time which is what matters.
Grant @ Aug 7th 2006 1:28PM
Yeah, 70% is supposed to be average in school, and big game sites probably just plain ignore most/many of the bad games out there. These curves make much sense to me.
Mephy @ Aug 7th 2006 1:30PM
Yeah, this is no big shock. Reviewers use the 4.0 scale of reviewing, which essentially means that anything under a 65 is an absolute F. The problem arises from the fact that they're using a number scale without any reference to said 4.0 grading structure.
m3mnoch @ Aug 7th 2006 1:33PM
that explains to me exactly why rotten tomatoes pushes up the threshold for a "fresh" game.
heh. maybe it just shows reviewers are mostly children playing at critics. they are so used to 70% being the cutoff in school that it's pushed into their reviews.
side note to non-math majors: 70 is not the median of 0-100. an average game on a bell curve should sit around 50 -- not 70.
so, maybe it's not so much payola, but more like "crayola."
m3mnoch.
Marty @ Aug 7th 2006 1:34PM
The truth is... With so many more games coming out these days (as compared to the 16 bit days)... reviewers are actually just skipping the crappy games and reviewing only big name and good titles.
I really don't think there is a consipiracy with reviewers getting paid.. Why??? Because XBOX did not get that many great games, and if anyone had the money to pay, it would be M$. They do need to boost sales of their systems, and it would have been a lot cheaper to pay off reviewers than the 5-10 billion they lost in XBOX!!!
Don @ Aug 7th 2006 1:36PM
My one point would be that games are reviewed once. Every new game "in theory" should be better than its predecessors. Games are usually always improving on graphics and gameplay. When a game comes out it makes a certain impression but when a new game comes out it can only be compared to what already exists. I guess the easiest way to look at it would be a game like madden. Madden 2006 scored an 8.0 on IGN. Lets say that Madden 2007 also scores an 8.0. The 2006 probably would not still be rated 8.0 given that a new version comes out.
Basically all this really proves is that new games are continuing to improve on the older games.
LordAlu @ Aug 7th 2006 1:39PM
Most universities in the UK seem to use a different scale - below 40% is considered a fail (F), 41%-48% is an E, 49%-55% a D, 56%-61% a C, 62%-69% a B, and 70% above an A... so an average game should be between 5.2 and 6.1... if the average review score is 7 or above, that means they're all A-grade games!
I don't know how they're looking at it, but to me that means there's a bucket load of great games out there I may be missing!
Thomas Crymes @ Aug 7th 2006 1:40PM
If they are using an (A-F) grading scale (associated with percentages) in their heads they are doing it wrong. Otherwise half of their scale goes completely unused. I'm going to assume they are smarter than that.
On a percent scale and average game should score a 50. 70 and 80 are scores of pretty good games.
acurmudgeon @ Aug 7th 2006 1:46PM
I have noticed this trend as well. I remember a time when I believed that anything over a 7 might be good, conceding that a lot of the time the difference between a 7 and higher was usually reviewer preference, and you had to take into account the reviewer's feelings about the genre, the story, multiplayer, etc. Now reviews mean nothing to me. I mean really, I don't feel I can trust them at all. But for me the biggest indicator that the entire professional review system is bought out is the delay in the release of the title and the review. Remember the good old days when the reviewers got an advance copy of the game around when it went gold and the review was out for a couple of weeks before release? It really did happen that way, if you don't remember. Those days are long gone. Why? I firmly believe that the companies producing the games don't want reviews before it hits the shelves, so the hype machine they have built up prior to the release can sucker in as many people to buy before the negative press hits the proverbial street. Previews are a straight up, old school bait-and-switch con. When was the last time you read one? Think about it the next time you tuck into one of the professinal previews. Look for anything that might make you think twice about purchasing it.
Oh, and for the record many of those games reviewed with a 7.0 or and 8.0 were horrid.
JRM @ Aug 7th 2006 1:48PM
No. Fanboys have already deciphered that Gamespot (and IGN)code to reviews:
AAA:90-99
AA:80-89
A:70-79
every other score below 70 is irrelevant, meaning that you shouldn't touch it with a ten-foot pole. Therefore, the data should only be measured for scores of 70 and above.
Darth Pixel @ Aug 7th 2006 1:50PM
That must be the dumbest argument ever made.
Obviously, the curve is skewed.
That's only because developers put more efforts into producing good games than bad games.
Most reviewers use a relative scale when scoring games. Basically, you rate games in a certain time period, as compared to games released in that same time period.
GlitchCog @ Aug 7th 2006 1:54PM
It's a question of scale. Why should video game ranking scales be like high school geometry marks? If 7.5 is an "okay" or "average" game, it implies that 0 through 5.0 are all reserved for ranking nuances of crappy games. The same system that’s designed to make failures feel good about their school grades shouldn’t be used to judge video games. Why don't these sites just take their 5.0 to 10.0 scale and translate it to 0 to 10...
5.0 = 0
5.5 = 1
6.0 = 2
...
9.5 = 9
10 = 10
A scale that doesn't uch more logical and less misleading.
This score inflation is a symptom of the overall poor quality of video game journalism. The issue in my eyes is that video game journalism caters to the stereotypical video game player: some kid who gushes at big-titted fighting game heroines, giant guns on overly masculine first person shooter heroes, subtly homoerotic professional wrestling games and giant space explosions with cool graphics… oh yeah, and gets 70% in history class. These types of fans are easily swayed by 7.0s to buy unintelligent and poorly crafted video games, and so these massive video game media empires can be paid off to shift game ranking scales up.
Jeff @ Aug 7th 2006 1:56PM
"On a percent scale and average game should score a 50. 70 and 80 are scores of pretty good games."
Works out for GameSpot, then, because they don't use a percent scale. And they have explained their system countless times; "7" is average. This is not just me guessing as to their system; this is their system. I can't seem to get into their site right now but I've read this there and you can too just as easily. They do use the 4.0 grading system and they do reference it. So all this graph is doing is confirming that they're doing their jobs properly.
I make a point of not reading IGN's 54 page reviews that contain about 53 pages of fluff and 1/2 a page that's actually relevant, but last I remembered, they don't use a percentage scale either.
Bottom line being this graph is pretty worthless because its premise is completely flawed, i.e. that all sites should be using a center-weighted percentage system. There's no rule that says that's the case and in fact, at least one of these sites (and probably both) spells that out pretty clearly.
Ian @ Aug 7th 2006 1:59PM
This is why I hate all video game reviewers. Their "average" is usually above 7. I mean you can say "well thats a C" but mathematically 5 is the average from 1-10. Its right in the fucking middle. It pisses me off, GI even explained in one of their magazines that 7 is their average, wtf! It shouldn't be. When you look at the "how we review games" thing, it says 4 stands for this and 6 stands for this, etc. Well almost always they say 5 is an average score but their average is actually 7. It makes no sense to me. I mean I used to read PSM for 2 months till I realized no game got lower then 7 except 1. I looked at all their reviews for the last few months in the back of their mag and I saw like 1 5 and 2 sixes and everything else was 7 and 8. How the hell did 7 become the new 5! /rant.
White Rose Duelist @ Aug 7th 2006 2:03PM
Everyone needs a quick refresher on the word "average".
adj.
Being intermediate between extremes, as on a scale: a player of average ability.
Usual or ordinary in kind or character: a poll of average people; average eyesight.
If publishers are not releasing the truly bad games, then the scale should shift to put the average game near 50%. Why would reviewers withhold the bottom half of the scale to rate games that have never been released?
Note that no other media reviews (movies, books, etc.) use 70 or 80 as a middle ground.
The school analogy makes no sense, because grades are not (usually) arbitrary. A 70% means that you got 7 of 10 questions right. 50% is failure because knowing half the material on a subject is not sufficient to demonstrate competence.
Hugh Jeffner @ Aug 7th 2006 2:13PM
I have not been to one of these review sites in quite a while but maybe this 'adjustment' is for advertising purposes. Perhaps if they had the balls to post scores that actually looked bad instead of an arbitrary average, some of thier advertising clients would get upset. Same deal goes for magazines, I think I read that EGM has been grilled in the past for thier somewhat more brutally honest reviews.
Probot @ Aug 7th 2006 2:14PM
Here's what Jeff was looking for:
"The average rating on GameSpot lies between a high 6 and a low 7, which is fully in line with what we believe is the fairly good quality of the average game on store shelves. Because we do not strictly grade on a curve, we have not set 5.0 as our average rating. We believe the high end of our rating scale (the 8 and 9 range) works suitably well to distinguish truly outstanding games from all the others. However, most games really aren't bad."
http://www.gamespot.com/misc/reviewguidelines.html
The only argument I can think of to counter their review process is that giving an average game an above average score (in a mathematical sense) may be misleading. The obvious proof of that flaw is appearent when reading some comments here. Some people expect 5 to be the average for a review score.
However, to counter my own counter-argument, Gamespot clearly states how their reviews work (and there is a link to the page on every review.) Not to mention that it is based on the common grading in school, which is also not based on mathematical average.
To me, the flaw in review systems is not its use of the word average. The main flaw it its use of numbers.
Tiago @ Aug 7th 2006 2:15PM
Most of the game ratings (on the web or on magazines) tell you that a game below 70% isn't worth your time, a game that's 70% - 79% is worth a shot and all the games above that are great.
Game reviewers have two major points they must consider:
a) How crappy is the game
b) Who will like it
Usually the games below 70% are the crappy games, the 70% to 79% are the quirky games not everyone will like and the rest are the "enjoyable by all" rating.
Scott @ Aug 7th 2006 2:16PM
M3mnoch, you're confusing "median" and "average" (also known as "mean"). It's nonsensical to say talk about the "median of 0-100".
The median is the middle number in a range. So, if for a given video game title, five reviewers gave it scores of 65, 72, 78, 85, and 94, the median would be 78. The average, or mean, would be 78.8.
The median is often a preferred measure because just one wild number in the range can completely skew an average. Imagine in the above range that the first number is zero. Suddenly the average drops to 65.8.
Concerning the wider issue of score inflation, there are probably a number of factors at play. It seems to resemble ratings deployed by Wall Street analysts on the companies they cover. I bet if you graphed those, they would look similar to the curve above -- lots of "buy", "outperform" and "hold" ratings, and only a few "sell" ones. Maybe this is because they only cover companies/games they like. Why bother reviewing a crap game or a company that's about to go bankrupt? Look at reviews for books and games on Amazon.com. Most people review what they like, and tend to over-reach when describing it. Look how many ho-hum or moderately good albums are declared to be "the best ever!" It's easy for people to skew their own perceptions.
Then again, a professional reviewer should be able to rise above such sentiments and keep perspective on the material in front of them. Here, I would turn to someone like Roger Ebert. How many movies does he give 4-5 stars? Are all of those movies of the caliber of "The Godfather" or "Citizen Cain" or what have you? No, but he rates them not only on their technical achievements but also just on how much he enjoyed them.
Anyway, there are a lot of reasons why game reviews are curved the way they are without resorting to accusing the authors of being paid by the game companies to slant their reviews.
Chris @ Aug 7th 2006 2:20PM
I used to freelance for Gamespot. The truth is, the numbers are completely arbitrary. At the end of a review, after several back and forths with an editor, you take about 30 seconds to craft some numbers. There's no weighting system, no scales that says "pixel shaders = +0.5 graphics score". You just think up some numbers of what sounds reasonable and plop them on the page.
It always amazes me how much weight people put on the numbers, when the meat of the review is what they should be looking at. We don't spend time on the numbers, guys -- don't dwell on them.
32_Footsteps @ Aug 7th 2006 2:22PM
Of course, for those who argue in favor on making 70 the average score for a game, I have one question for you - what use are the numbers below 70? You're practically ignoring them.
With the average game being a 50 on a scale of 100, it makes sense. You're just as likely to find a game better (51-100) as you are to find one worse (0-49).
Though I have to admit some amusement from this quote from GameSpot: "However, most games really aren't bad."
Really? Have you even played most games, in that case? There's some really bad ones out there. Heart-stopping bad. "I'd rather kiss Uwe Boll" bad. Despite what GameSpot and others think, video games are not somehow immune to Sturgeon's Law.
m3mnoch @ Aug 7th 2006 2:22PM
"Bottom line being this graph is pretty worthless because its premise is completely flawed"
i am not so sure about that. maybe that's the case with games (see my post about "crayola") but not with other review systems. take movies and their 4 star scale. it's not "percentage-based" either, right?
yet, it works out to average movies are 2.5 stars.
any given time there are a few 1 and 1.5 star movies but the majority of them are 2.5ish. (damn me that i still watch them!) so, if some movie gets 2 or 2.5 stars, they're pretty damn average.
another note to non-math majors: the median between 1 and 4 stars is ... drumroll ... 2.5.
yeah. you're whole "wrong scale" thing is totally and irrevocably inaccurate.
m3mnoch.
Gary @ Aug 7th 2006 2:23PM
As with other reviews of media, be it movies, books, music etc. the audience must take the whole review article and its quality into account.
I've read many reviews that explain exactly why I would like to play a game...and go on to give it a relatively low numeric score. I've read other glowing reviews that prove to me that the type of gameplay involved isn't for me.
A site's grading system has to just be typical for that specific site in order for it to serve it's relevant purpose.
The numerical score is one of the least helpful tools of the review in helping an individual to decide a game's quality.
Leshrac @ Aug 7th 2006 2:28PM
I have to second Darth Pixel's comment.
Just because reviews don't conform to a bell curve does not mean that the reviews are either paid for or even inflated.
In fact, it'd be almost ridiculous for them to conform to a bell curve, because every game isn't released at the same time. In school, everyone takes the same test and then every test is ranked, which can then be translated into a perfect bell curve.
If reviewers wanted to do this, I suppose they could pick some game as their "average" reference point, and then rank every game around that. But, given how subjective game reviews are, this would be kind of ridiculous.
I also think that this is standard for just about all kinds of reviews and not just game reviews. Think about it. Movies typically get rated on a 4 star scale, where anything above 3 stars is considered good and below that is basically considered bad. I see book reviews based on the ABC scale all the time, where anything below a B is basically considered bad. Why are games any different? For the most part, I won't buy any games that don't receive at least an 80% average review, with maybe the occassional one dipping into the 70s. Most people who check game reviews probably don't buy anything below that anyway.
Besides, I've known for a long time that IGN typically slightly overrates and Gamespot typically slightly underrates (most of the time). Was that really so difficult to figure out?
m3mnoch @ Aug 7th 2006 2:28PM
thanks for the statistics lesson, scott.
i know the difference between mean and median. if you read it, i was talking about the middle number and not the average -- therefore, the median and not the mean.
meaning, 50 is the middle of the percentage scale, not 70.
m3mnoch.
The ZeroCorpse @ Aug 7th 2006 2:29PM
I think they should just drop the numbers, and go with "Thumbs Up" and "Thumbs Down" and leave it at that.
Have three members of a review crew give the up or down for a game. Best of three gives you a good idea what you're getting... Providing that they're not just saying nice things because EA gives them swag.
MusashiX2 @ Aug 7th 2006 2:30PM
to hell with both sites. i am tired of reviews. just rent it and make a decision for yourself. you can also find yourself a rich friend or a friend who pirates so you can go over their house and try the games out.
m3mnoch @ Aug 7th 2006 2:50PM
"Just because reviews don't conform to a bell curve does not mean that the reviews are either paid for or even inflated."
nope. it just means it's misleading. i mean, c'mon. are you grading games against their peers or against games 10 years ago?
if the "average" game is getting an 8.0 score, sounds to me like the scale needs to be sliding. because an average game of today would kick the shit out of an average game from 10 years ago -- yet, in the context of today's market, it's still freakin' average. meaning, it needs an average number to represent it.
and, you know what? if you keep grading games against their peers, you can accurately look at any games from any time period and relate them all to each other.
that'd make the fanboys happy. some validation that, yes, zelda was that much better than halo.
i'm all about zerocorpse and the thumbs up or thumbs down. maybe even a thumb sideways. game are good, bad or average. not pis-poor, good, great or fantastic!
m3mnoch.
phantom42 @ Aug 7th 2006 2:53PM
in and of themselves, the scores hold little meaning. the same goes for any type of rating system without any sort of context.
there used to be a gaming mag (its been a long time, and i forget which one it was) who would have multiple reviewers give their take and ratings on the game. but it would also give a bit of background on the reviewer. if a reviewer who really only liked sports games gave an RPG a low score, i knew not to pay attention to his scoring. conversely, when a RPG player gave an RPG a low score, i knew i should probably take heed.
i'd really like to see stuff like that more often, or something else similar to give you some sort of gauge on how much the particular reviewer's scores can really be trusted in this case. keeping with RPG's as an example, maybe show some past scores given by that particular reviewer to other RPG's of varying quality so you can get a line on what they think is a high end or low end game.
Derbeste @ Aug 7th 2006 2:55PM
Perhaps the problem isn't with the game reviews but using the bell curve to analyze review trends at all.
I detest the bell curve. It is a relativist system that is does not look at it's subject based upon its merrits...but by comparing its merrits to the merrits of another.
In other words....no matter how enjoyable a game may be, If 10 other games were SUBJECTIVELY "better" than this great game must be given a "bad" review.
That's the problem with the bell curve defining 50% as average. Some game MUST fail regardless of its merrits. In school it means that even if you got 90% of your questions correct, you can still "fail" because everyone else got 95%.
It's misguided in school and it's misguided in game review. Is it really fair that 1/2 of all games review MUST be "bad" and 1/2 MUST be good? Think about how skewed that is! That means the game you made is only good depending on which other game is released that year. Can you just imagine the review process?
"Wow! I really enjoyed this game! So well put together! I give it a 9/10! What? Zelda, Metal Gear, and WoW were just released? Oh....this game sucks! What? Truck Drivers, Craptastic XIII, and Health Inspector the game just came out too? Oh...this game is average."
Traditional bell curves do NOT work for review trends!
Luckilly, most schools and reviewers DON'T USE THE BELL CURVE! They use the "mode" (The number that appears most frequently). In school, it's been found through long term testing that an "average student" SHOULD be able to get 70% of his tasks completed correctly. That where they have defined "average".
Every reviewer has their own bar and it's up to the reader to find which site is more consistent with their tastes.
m3mnoch @ Aug 7th 2006 3:04PM
"It is a relativist system that is does not look at it's subject based upon its merrits...but by comparing its merrits to the merrits of another."
last i checked, that's how "the best" is chosen. wouldn't that be a nice un-relativistic world to live in where all athletes win because they all tried so hard! "good job! gold for everyone! we're all winners!"
so you're saying that video game reviewers are more like the tee-ball baseball league. yay! everyone gets a trophy!
again -- see my "crayola" remark above.
it's a relativistic world we live in, man.
m3mnoch.
Derbeste @ Aug 7th 2006 3:12PM
/sigh.....
You choose to ignore the whole picture of relativism.
First off, who said game reviewers were looking for "the best"? Who says we ONLY buy the "best" game? I've got news for you. If you only buy the "best" game, you'll NEVER buy a game because you can't decide that till every game is made.
I, on the other hand, buy GOOD games that are ENJOYABLE to play. And whether that game is ENJOYABLE or not does NOT depend on how ENJOYABLE another game is.
Secondly, you single out that I mentioned 50% of games MUST fail and accused me of thinking that we should give awards to everyone.......
Did you ever stop to think that the logic goes the other way?
If 50% of games MUST fail on the bell curve, than 50% of games MUST be "good" even though some might not deserve it!!!!!
Lastly, I am a revativist too. But the Bell curve is an extremely misguided approach to measuring it....especially in this case.
There is nothing crayola about my comments. And I would certainly NOT give you a trophy for yours.
onetrueping @ Aug 7th 2006 3:32PM
It's kinda funny how a simple graph, completely open to interpretation, is garnering so much response. Don't you think it'd be better to wait for a more thorough analysis of the data before arguing over it? Or has Joystiq suddenly become like Popular Science?
Hunting_Phoenix @ Aug 7th 2006 3:33PM
Here's my thought, a long tail is visible on the Gamespot chart, on the low side of the curve. The high side looks pretty Gaussian to me. Now could it be that the low scoring games are hard to rate, leading to a spreading of their scores? How do you rate a crappy Powerpuff girls game? Is a 3.4 really any different than a 4.0? Not really. They're both in the realm of "nearly unplayable."
Now think about pushing up the low ends of the curves. If the low scores wern't so spread out, it would be much close to a Gaussian distribution. Then it all depends on your definition of 'average.' typically these sites call 7.0 average. so i think they're not too far off really.
it's not like the most common scores are 9.X or anything. They do a pretty fair job, IMHO.
m3mnoch @ Aug 7th 2006 3:36PM
and here's another thing that bothers me:
"In school, it's been found through long term testing that an "average student" SHOULD be able to get 70% of his tasks completed correctly. That where they have defined "average"."
say what? who cares? that's a system that tests readiness for the entire school population. it's not a test to see what student is the best fit for some finite position. are we testing game readiness or which ones are the best?
with marketing and selling games, you're better making comparisons to valedictorian and salutatorian. again, it shows that the reviewers are populated with children.
in the real world, it's about what game (notice singular) offers you the most reward for a finite resource -- your money -- not a "churn them through the system" type of thing.
where on earth does it make sense to say "i know you can only afford 2 or 3 games so here's 3000 to choose from! and, rather than spread out the results over 10 increments, we're gonna institute decimal places so we can squeeze them all into a third the space! they're all winners!"
no, no, no. how long until we reach that point where 9.5 sucks. 9.6 is average. 9.9 is stellar?
this whole mess works totally different in a non-finite resource system. say, like books or movies. the difference? movies are 1/5th the price of games. (not to mention "free" on tv) books are 1/12th.
as the percentage of your budget required to consume something rises, the need to only spend money on the best also rises.
enter piracy and rentals. people can't tell what the best games are, so they steal or rent them all.
hmmmm... i wonder if game devs can actually blame piracy on stupid review scores. heh. someone better tell cloud and hollenshead!
m3mnoch.
Jason @ Aug 7th 2006 3:44PM
This is not even remotely fishy. The Bell Curve is what is fishy. The fact is, the word "average" is a misnomer when applied to reviews (see DPReview.com, where "average" for a digital camera really means "below average" in terms of the cameras they review). Giving a game a 5/10 score should in no way be expected to be the average. Heck even in grade school (U.S.) the average test score is supposed to be 70-80%.
This is a total non-story. Come on Joystiq, you're better than this.
AoE @ Aug 7th 2006 3:49PM
This debate is hilarious, how many of you REALLY care what reviewers have to say anyway? Whether a game is "enjoyable" or not is so subjective that reviews inherently can't have much meaning (to whit, how did you feel about chromehounds? I think it's a pretty enjoyable title myself), unless your tastes happen to lie along the exact same lines as the reviewer in question. Otherwise the best one can hope for (imo at leasT) is a very complete review, one with enough detail regarding gameplay etc that you can effectively remove the reviewers bias and make an educated guess based on said reviewers' description of gameplay and controls.
As for average titles getting a 70%, there are a few factors at work here. First of all we all get, at minimum, 12 years of training (in the US at least) in this kind of scoring/grading system; it's ingrained in us. When you hear something has a score of 50% most likely your snap judgment is "failing" not "average." I realize math majors probably don't think this way... but think about joe-bob gamer who still reads game magazines; I'll be he thinks of 50% as failing. It's probably also political... as you may or may not have noticed, game publishers like spending advertising money on game review websites, especially the high-profile ones... do you think their advertising budget would grow for "X" review site if said site started rating average games @ 50% instead of 70%? Probably not. I can't think of a single major publisher that hasn't released at least a few stinkers. Finally, isn't it possible that there are games out there that YOU personally may have not played (how many of you played Pulse Racer?) that are sooooo bad that by comparison average titles actually do deserve the 70%? If you don't think that's possible, get ahold of Pluse Racer and Blackstone: Magic & Steel for xbox, and then get back to me...
m3mnoch @ Aug 7th 2006 3:51PM
"You choose to ignore the whole picture of relativism."
/sigh
you're a student aren't you?
if the average gamer could consume a relatively infinite number of games ($20 is the accepted economic price point to define such general consumables) then, the review scores would make sense.
no. i'm not ignoring relativism. that's my point. merit-based ratings in a finite system is stupid "confuse the marketplace" thinking.
welcome to captialism, crayola-man. it's all about being better than your competitor -- not being better than some "standard."
m3mnoch.
Warbleflit @ Aug 7th 2006 3:52PM
This is why I prefer a 0-4 scale over a 0-100% scale. Even when the 0-4 is divided into .5 it ends up being a more realistic rating that makes sense to me. I mean look at this this way, I've seen MANY games on the 4 point scale get a 1.5 or a 2 or a 2.5, or a 3, and on that scale a 2.5 game is not a "never buy" game but a "you might like it a little, maybe" game. If we say that the 70% is average, that is almost a 3.5, and on the 4 point scale a 3.5 is a GREAT game, almost perfect. It just seems easier to tell what the score really means, and whether the game is good or not, when there are only 4 (or 8) points to worry about.
Derbeste @ Aug 7th 2006 4:03PM
I see the point you're trying to make and I appreciate a more docile tone.
I think we are argueing about extremes while the "sweet spot" is in the middle.
Game reviews do tend to gravitate toward rediculous extremes and decimal sillyness.
But the traditional Guassian bell curve is the other extreme. Especially considering it puts indie teams and taking risks on new ideas at an EXTREME disadvantage.
My only point is that a games score should not be penalized because gaming had a "good year" or that a mediocre game should get a better score because the industry is in a bit of a drought.
There is flexibility to be had without becoming a 9.x joke.
Derbeste @ Aug 7th 2006 4:18PM
p.s. m3mnoch
No...I'm not a student. Nor will I spout off a bunch of credentials that I can't prove on the internet.
Nor will I name call in an attempt to distract from the point.
The point is that NO reviewer seems to think the that the traditional bell curve is an accurate way to express to gamers how many games out there are worth buying.
If you think it is, set up your own review site and do better....see what happens. If it succeeds, I will GLADLY concede and probably frequent your site.
Until then maybe the fact that no one uses the traditional bell curve should tell you something. It doesn't work!!
I don't know of many relativistic industries in capitolism that don't use a weighted curve anyway. Because in capitolism "Average" varies by industry! Sure you always want to be better than your competitor, but there is NO rule (other than traditional, archaic bell) that states "failures MUST equal successes" (ie average of 50%).
m3mnoch @ Aug 7th 2006 4:59PM
so. here's a question then:
what's the difference between what they are doing now and a tradional bell curve?
answer?
there's not one. that's exactly what it is, but pushed over 2 points.
except that it is misleading and inaccurate and unconforming to established methods of measurement. it (from a traditional point of view) paints the picture that almost all games are above average.
if all games are above average, how do you define average? and, yes, you need to define them as average in realation to the group -- the group being the current year's games. on aggregate, games are generally getting better, year over year. that, and comparing quality of games directly over the years just doesn't hold up under scrutiny.
i mean, i'm as big a fan of citizen kane as the next guy. do i think it's a better movie than say... syriana? nope. cut out the nostalgic factor and release it today and it's crap. for its time, tho? sheer brilliance.
there needs to be a credible measure of a game against its peers. grouped by genres (rpgs or sports for example) or grouped by reviewers (roger ebert or michael wilmington) -- sure! that makes sense. find a reviewer you agree with and go with their bell curve.
and again, if they were priced so that everyone could enjoy all the games they "might" be interested in, it'd be a moot point and a bell curve would be inappropriate as to judging the veracity of game quality. however, until then, we'll still need a measure to compare games to their peers.
there still needs to be a way to skim the cream from the top.
current game sites are silly in that everything is cream when compared to traditional measurements.
all i'm asking for is "stop the inflation and shift all of your scores by 2 points." just slide the damn scale back down so places like rotten tomatoes don't have to compensate for you. it does nothing but confuse your audience.
well. that and give everyone a trophy....
oh. and write my own review site? sure. whatever. i'll just add that to the rest of the stuff i'm working on. me? i do more than just monday morning quarterback. i actually participate in the gaming community.
m3mnoch.
m3mnoch @ Aug 7th 2006 5:02PM
p.s.
"Until then maybe the fact that no one uses the traditional bell curve should tell you something. It doesn't work!!"
tell that to movie and book reviewers. works like a charm.
m3mnoch.
Derbeste @ Aug 7th 2006 5:16PM
One more shorty, M3M...
First off, Standard, Guassion bell curve shifted over 2 points = a weighted curve. And it's VERY common.
Instead of saying 1/2 must fail and 1/2 must succeed, it can say well...in our industry 2/3 are enjoyable and 1/3 is not worth anyone's time. Or in computers manufacturing where 3% hardware failure rate (after shipping) and 97% flawless hardware is "average". The weighted curve reflects the different quality standards of various industries. Where do you think the term "Industry Standard" came from?
Secondly, I disagree that reviews must be half and half. In fact, I ALMOST disagree that games should be given a number at all. If you really want a accurate analysis, read the ENTIRE review (or many reviews) and research its pros and cons. If you think about it, why give it an overall rating number at all since numbered reviews are only an attempt to quantitate qualitative data.
As for your work load. If you don't have time to do something because you are too busy in the gaming community....well....let's just say I call that a "high end" problem. Some of us only have time to participate in a game in between two jobs. I still mean it though. Show us how the concept should work, and I garantee hits from me.
P.S. ok...not so short. Hopefully I'm done though.
cringer8 @ Aug 7th 2006 5:34PM
What's so bad about the "Five Star" rating system?
Anyhow, I only read reviews that actually have substance to them. If someone gives a game a "65" but in the review he/she is constantly complaining about aspects of the game that I don't care about, then I can take that "65" and move it up accordingly. I can't do anything with a number alone.
Numbers mean shit, especially when they're based on subjective opinion. These websites are grading on a scale of 50-100. A 75 would be average in this case. If they give a game anything less than 50, it's because they're being spiteful. You just have to look at a “30” the same as “50”. You have to be smart enough to cut through the BS.
m3mnoch @ Aug 7th 2006 5:43PM
"Where do you think the term "Industry Standard" came from?"
again. see my spam on merit-based reviews for a finite system. reviews are subjective and in relation to each other. hence, average is average. not a weighted average. a weighted average for reviews is just bowing-to-pressure-fluff. no other medium does it -- just games.
games. where a bunch of pimply-faced, fresh out of school (or still in it) mediocres are rating games arbitrarily. (see chris's confirmation above) trying to align to some industry standard they think is "neat." it smacks of the old days of "sound: 8.5" breakdowns. that's where your rating a game based on its merits comes from. it's school kids running amok and confusing ratings with reviews.
and we sit here and wonder why the scales are whack?
however, not using numbers to quantify a subjective thing? you and i totally agree on that. bring on the popcorn bags!
but, seriously, the game review industry needs to break out into personalities instead of media conglomerates. that way, just like movies, music, books and every other artform, you can find a critic who likes the same stuff you do. that way, you can just blindly play what they think is good and be confident that you will at least enjoy it. no worrying about whether some random reviewer is a comedy fan, a sci-fi fan or a racing game fan.
m3mnoch.
32_footsteps @ Aug 7th 2006 6:34PM
"so. here's a question then:
what's the difference between what they are doing now and a tradional bell curve?
answer?
there's not one. that's exactly what it is, but pushed over 2 points."
M3mnoch's point here is so good, that it needs repeating in another comment.
It seems like everyone is pushing for a bell curve. It's just a question of whether its middle point is 70 or 50.
Schpyder @ Aug 7th 2006 7:03PM
Personally, I think a lot of the non-problem that most people have with the score distribution is with the 10-point/percentage rubric. It really just offers too much granularity to be genuinely useful. Furthermore, it carries some cultural nuances with the school grading system in the US, where 70% is considered average. All the whining in the world won't change the fact that if these reviewers readjusted their median score to 5/50%, the vast majority of the American public would consider those games well below average. Like it or not, that's the way it is, and these places have to review with their audience in mind, lest they fall by the wayside to more sensible competition. Remember, pedantry never wins over the common man.
Personally, I think a 4-star (or even 5-star) system works much better, even with half stars. It doesn't carry the same cultural baggage, and every place I've seen that's used it (like, say, Gamespy's 5-star system) has actually had average games land right in the middle of the scoring range. Everyone wins!