Reviews of MMO Games are Fundamentally Flawed

by Vladimir Cole { Jun 1st 2005 at 10:55AM }

Blizzard's World of Warcraft game has received high marks from all of the gaming publications for its ease of use and for its newbie-friendly game design, but there's a fatal flaw in these reviews. They invariably focus on the early experience because it takes several months before players are able to reach the level at which they are ready for the so-called "end-game content."

A quick primer on end-game content. In WoW (and most other large MMOs) players band together into groups so that they can access content that no single player could do on his own. Some of the content in WoW requires raids—40 top-level players all working together in a coordinated, organized fashion.

Large-scale raiding typically emerges months after the release of an MMO product, long after most reviews are written and published. As a result, reviews of MMOs are worthless at describing content that typically occupies 80% of the long-term player's time. It's like a car review that only writes about the experience of parking the car; a movie review that describes only the opening credits; or a restaurant review that only describes the appetizers.

This probably won't change until intelligent games consumers (like you who read this blog!) start asking reviewers to give us the goods on end-game content in MMOs.

Reader Comments (Page 1 of 1)

-g- @ Dec 18th 2005 9:06PM

Yeah, I'd also like to point out, that that is the main reason Runescape suck. Once you are about level 10. IT just isnt rewarding anymore.

TrackZero @ Dec 18th 2005 9:06PM

Yeah, because you know that reviews always play non-MMORPG games right to the end before reviewing them right? *cough* BS *cough*

I can't count the number of reviews where it's apparent that no more than a few hours were put into the game before the review was written, MMORPGs are no different. Period.

vlad @ Dec 18th 2005 9:06PM

MMOs are different. No other game genre has not just hundreds of hours, but MONTHS of pure play time.

I can write a decent review of Half Life with just 5-10 hours of gameplay. The last level of the game is more or less the same as the first level. The play mechanics do not significantly change.

It takes most players 360 hours of gameplay to even reach level 60 of WoW, and the play mechanics at level 60 are VASTLY different than the play mechanics at level 1. It is a completely different game.

And while it takes 360-ish hours to reach level 60, most players will spend another 1,000 or 2,000 hours playing the game once they've reached level 60.

Some Everquest players spend years playing the game at max level -- upwards of 7,000 or 10,000 hours.

In this way, MMOs are quite different from typical games, and deserve to be reviewed differently.

Tyler Smith @ Dec 18th 2005 9:06PM

Yes, lets all jump on the bandwagon and play the entire game the whole way through *shock*. Its called a game, not a lifestyle. Lets have all reviews play every single character, do every single thing in any given MMORPG. 4 or 5 years down the road, we'll finally have our review.

*chuckle* Could you imagine an editor telling one of the writers "Look, your deadline for the peice is due in 5 years. Better get busy."

Give me a break.

JDH @ Dec 18th 2005 9:06PM

The review flaw should be noted. However, an even greater flaw in the MMO genre is playing the game to "level".

Guild Wars lets you start out as a max level character. Why can't other games?

I would be more inclined to play City of Heroes with a 30 level character than to have to build my character to level 30.

The way Star Wars Galaxies works it would be easy. Plus most MMO's allow for a respec periodically.

Chris Norman @ Dec 18th 2005 9:06PM

But has the core of game itself significantly changed?

A review of an MMOG can be trated like most other games - you can get enough of a feel from early play to gauge things like general difficulty, atmosphere, graphics, and generally (particularly) how successful the developer is in keeping you wanting to play more.

If that last bit is a success, then the game is worth the money and should get a positive review, no?

At that point I think that players will be able to judge for themselves whether or not the game is worth their investment in terms of time, and if you play enough to achieve end-game results, then you have invested enough in a game to realize that there are better ways to judge the value of the game than reviews - i.e. message boards.

But the game itself has not changed, only the way that you play it. These end-game challenges are basically the same kind of quests found early on in the game, just on a larger scale.

Generally, by the time a significant number of players have reached the "endgame", the developer looks to release new content, which might prompt a new review taking this experience into account. But this review will obviously appeal to the players it is targeted at - where as a general review of the game addresses everybody.

All that said, I think 1up has the right idea with their Game Diaries (i.e. http://www.1up.com/do/feature?cId=3138465 ) - weekly articles that give readers a better idea of the whole experience of the game.

WizarDru @ Dec 18th 2005 9:06PM

You also forget that not everyone plays an MMORPG the same way. Some folks don't power-level to 40th after playing straight 6 hours a day for a solid week. Some people take months or years to get that high. How do you review for them?

Plus, I've yet to meet an MMORPG player who doesn't have multiple characters. So, the low-level content is actually more important as you'll see it more often. Add to that fact that many players drop out fairly quickly, and you'll see why basing a review on ultra-high content (which is the content most likely to be added after launch, and thus after the initial release review) isn't as helpful as it could be. It's good to note, but it's hardly the make-or-break of a review.

WoWFan @ Dec 18th 2005 9:06PM

This might be the single stupidest thing I've read in a month or more.

End game content is 80% of a players time?

Most players don't REACH the end game in MMOs, only a very select few do. Even with WoW, the fastest game to the end game, look at some server pop numbers, at most 10% of the populace is level 60.

The reviews are for the majority of consumers and deal with what the majority of consumers will see.

I've played WoW to the "end-game" content and it's nowhere near 80% of my time. I'd say less than 5% of WoW players will ever see MC or Onyxia.

Chris @ Dec 18th 2005 9:06PM

I agree with #8. Where's the mod button, because this article isn't worth the blog-estate? I'd expect to find drivel such as this in the WoW forums along with all the other whiners.

Havok @ Dec 18th 2005 9:06PM

Wow, people actually read reviews? I couldn't care less what a publication is saying about a game. I'll play it, if I like it I'll play it more, if I don't, I'll stop playing it.

I like to make up my own mind about things.

Knight37 @ Dec 18th 2005 9:06PM

Havok, you buy all the MMORPGs when they are released so you can make up you own mind? A lot of people don't. They rely on reviewers to give them an idea of what the game is like so they can decide if they want it or not.

vlad @ Dec 18th 2005 9:06PM

You fellas are so kind.

All game reviews are flawed to an extent, but MMO reviews in particular are extremely poor representations of the experience of the product.

I'm not advocating that reviews come out months after a game's release. Merely that it be stated up front that the review only represents the newbie game and does not represent the "end game."

Also, I'm not sure where "WoWFan" got his figures, but far more than 5% of players play the end game. WoW is an incredibly easy game to play and level. Most players achieve top level within the game in a relatively short period of time.

What they do once their there really hasn't been the subject of any of the early reviews, and that's a flaw both in the review structure and how we as game consumers approach our evaluations of these products.

The cynical marketer rubs his hands gleefully: "I can have the development team focus solely on the newbie game. Make it slick! Make it fun! Make it sing! The end-game is not a feature we have to sell or spend any money on for product launch."

Reviewing an MMO should not be like reviewing a typical game product. They're very different animals.

Jeff Freeman @ Dec 18th 2005 9:06PM

Quibbles over the precise statics aside, it's a valid criticism of game reviews in general: in that they tend to be opinions of the game based on a 'glimpse' of gameplay.

"This is fun." Fine. But is it fun for three months? Six? Or does the second half of the game turn into jumping-puzzle HELL? 'Cause that'd sure be nice to know.

Did ANY reviewer of Black and White play past the tutorial part?

tevetorbes @ Dec 18th 2005 9:06PM

Agree 100% with #11.

Anybody who played EQ when PoP came out knows exactly what this article is about. Since it takes a bit of time for people to actually reach the end game, it was SOE's scheme to release broken and unfinished content, but have plenty of pretty glossy screenshots of "content-to-be". People read reviews, saw what was _possible_ (although not even finished!) in the game, and bought the expansion. And wouldn't you know it, Uber_Guild_001 reaches the Plane of Time and realizes "Oh snap! The damn thing is all broken!"

I don't doubt that the review-ability of MMORPG's is seriously flawed. No, beginner (or mid-level) content is _not_ as important, more important, or anywhere _near_ as important as end-game content. Period. The only reason that you are ever level 5 or 10 or 30 or 55 is because those are all numbers lower than the highest level that you can attain.

Cacophony @ Dec 18th 2005 9:06PM

I seriously doubt that any game publisher wouldn't create a temporary endgame level character for a reviewer from a popular mag or online source.

I guess this doesn't address the question of beta or early-release dearth of other endgame level players with whom to play.

Leo Kasatonov @ Dec 18th 2005 9:06PM

ok a review is someones opinion

if you like tomatoes and i dont, then your review of a dish would be MUCH different than mine.

Im saying who cares about reviews all there good for is pros/cons and screenshots (sometimes)

thats really what matters, not what bob had to say about it.

Coyote @ Dec 18th 2005 9:06PM

My favorite will always be L.O.R.D. or Legend Of the Red Dragon the old OLD school door game I played back in the late 80's early 90's man that game was fun while not really "massively" multi player it could support all your friends and had expandability for more areas to visit more fields to battle on and a brothel that was just wonderful. And the RIPTerm graphics were just awesome.. even if it did take 3 minutes for the dragon on the title screen to load on my ol'2400 modem.

steve @ Dec 18th 2005 9:06PM

A lot of MMO companies will buff a character so a reviewer can experience late-level content, or for when expansions appear.

As for the main topic, this is a non-issue, especially when you consider that no one would care about the high-level content if the low-level stuff was crap. So if a review only focuses on that aspect of the game, it's served its purpose for most readers.

And honestly, doesn't everyone know that reviews are a snapshot of a game at that specific time? Games get patches, but no one "patches" their review. (I don't think they should, but you can make an argument otherwise.)

Knight37 @ Dec 18th 2005 9:06PM

Reviewers have no choice but to base their review on a subset of the overall gaming experience. Lets face it, a review that comes out 3 months after the game is out is a review that very few people bother to read. It's irrelevant. The only reviews that really get any attention are the first ones or the ones close to being first. And since there's a mad rush for reviewers to finish things ASAP, they tend to only cover a limited portion of any game, especially a MMORPG. There is no way to change this. Gamers demand information about a game before or very soon after the game is out, and that's what kind of time table reviewers are working under.