Microsoft has disabled Xbox Live "Free" members' ability to rate Marketplace content on Xbox.com in an apparent effort to curb ratings manipulation. The announcement was made on the XNA Game Studio Team Blog and was specifically addressed to the Indie Games Community.
"By implementing this change, we believe our customers will experience more consistent ratings and a significantly reduced potential for abuse across the entire Xbox catalog," said the statement. "We have also investigated rolling back suspect votes, however, we determined it will not be possible to do this."
Several weeks ago, developers of some of the top-rated Indie Games suspected foul play when their games began to slide down the Top Rated listing after receiving, by one developer's count, "hundreds of 1 star ratings" in a single week. While the XNA Team conceded that it would be unable to take away such seemingly unwarranted ratings, it did add that "we are investigating users who may have violated their user agreement during this series of events."
"Violating user agreements may result in actions up to and including banning from Xbox Live, removal of currently posted games, and loss of ability to post games in the future," the statement continued.
A few Indie Games developers have expressed gratitude, as reported by Armless Octopus, for Microsoft listening to their concerns and taking action. (Historically, it's been a somewhat rocky relationship between the two sides.) While limiting rating powers on Xbox.com to paying Gold members would seem to thwart the most egregious attacks and manipulation of content, it does not solve some of the other issues with the system; most notably that a user need not play, let alone purchase, content to rate it.
Reader Comments (39)
Posted: Apr 19th 2011 9:05PM Revengez said
They could've just made it so you have to purchase it to rate it... I mean that is a logical choice, right?
Posted: Apr 19th 2011 9:11PM TraceurRyuk Part III said
@Revengez
Yeah, that's what I would have thought.
Reply
Yeah, that's what I would have thought.
Posted: Apr 19th 2011 9:11PM Ezio Auditore da Firenze said
@Revengez
Yeah, but that would make sense.
So, why do it?
Reply
Yeah, but that would make sense.
So, why do it?
Posted: Apr 19th 2011 9:21PM Tachyonic Cargo said
@Revengez
Purchased it, or at the very least, played it with your LIVE account.
For that matter, that rule should have applied from the very beginning to EVERY game on Xbox 360. You have no idea how many times it pisses me off, to see a game that is not even going to be released for another year, have literally thousands and thousands of user ratings for the game itself, when you go look it up.
This kind of behavior is childish, and screws with the overall average of any game's score. And was glaringly apparent way back in 2007, that like with Metacritic user reviews, was only a matter of time before someone decided to exploit Microsoft's laziness in implementing the system, to their advantage. It's a damned shame that so many other Indie devs had to suffer as a result of screw up. Which in my opinion, still is not fixed, and will not be fixed until said time as Microsoft requires everyone posting a rating to have at least played the games they are rating.
Reply
Purchased it, or at the very least, played it with your LIVE account.
For that matter, that rule should have applied from the very beginning to EVERY game on Xbox 360. You have no idea how many times it pisses me off, to see a game that is not even going to be released for another year, have literally thousands and thousands of user ratings for the game itself, when you go look it up.
This kind of behavior is childish, and screws with the overall average of any game's score. And was glaringly apparent way back in 2007, that like with Metacritic user reviews, was only a matter of time before someone decided to exploit Microsoft's laziness in implementing the system, to their advantage. It's a damned shame that so many other Indie devs had to suffer as a result of screw up. Which in my opinion, still is not fixed, and will not be fixed until said time as Microsoft requires everyone posting a rating to have at least played the games they are rating.
Posted: Apr 19th 2011 9:28PM WeRequireMOARMinerals said
@Revengez Why would you take the logical way out?
Reply
Posted: Apr 19th 2011 9:38PM DVMGames Richard said
@Revengez
An interesting thing about a potential scenario where ratings are only restricted to people who've purchased the game - it'd make it *even easier* to abuse the ratings system than it already is, given the way ratings work.
Imagine that I've got a really terrible game - an absolutely rubbish one. And the trial's so bad that no-one buys it, so no-one can rate it. Because of this, it has 0 ratings.
It takes just 10 ratings on a game for it to be ranked on the Top Rated list, which appears to be based on the mean rating of each game and nothing else - so a game with a mean rating of 4.5 will rank above a game with a mean rating of 4.3 regardless of how many votes each game has.
So with as little as 800 Microsoft Points, I could purchase my game 10 times, rate it 5 stars 10 times and now my terrible game is the #1 best rated of all time! And here's the best bit, to rate my game down, other people have to buy it meaning more money for me!
This is easily repeated for any game and pretty quickly, the whole ratings system becomes worthless.
To avoid this scenario, there'd have to be a pretty big overhaul of the way ratings are calculated across the system. This is one of the reasons why I'm opposed to restricting ratings to purchasers only - I'd much rather have it be restricted to people who've at least played the trial. But the current fix should be sufficient to prevent the suspected automated bot abuse via Xbox.com.
Reply
An interesting thing about a potential scenario where ratings are only restricted to people who've purchased the game - it'd make it *even easier* to abuse the ratings system than it already is, given the way ratings work.
Imagine that I've got a really terrible game - an absolutely rubbish one. And the trial's so bad that no-one buys it, so no-one can rate it. Because of this, it has 0 ratings.
It takes just 10 ratings on a game for it to be ranked on the Top Rated list, which appears to be based on the mean rating of each game and nothing else - so a game with a mean rating of 4.5 will rank above a game with a mean rating of 4.3 regardless of how many votes each game has.
So with as little as 800 Microsoft Points, I could purchase my game 10 times, rate it 5 stars 10 times and now my terrible game is the #1 best rated of all time! And here's the best bit, to rate my game down, other people have to buy it meaning more money for me!
This is easily repeated for any game and pretty quickly, the whole ratings system becomes worthless.
To avoid this scenario, there'd have to be a pretty big overhaul of the way ratings are calculated across the system. This is one of the reasons why I'm opposed to restricting ratings to purchasers only - I'd much rather have it be restricted to people who've at least played the trial. But the current fix should be sufficient to prevent the suspected automated bot abuse via Xbox.com.
Posted: Apr 19th 2011 10:16PM Startingline13 ArmlessOctopus said
@Revengez The reason they don't force you to purchase it before you rate it is because by that point most people who purchase it, do so because they enjoy the game. Therefore few (if any) individuals would rate titles down, and you would only see highly rated titles. That's the fear which many XBLIG developers share.
What I propose is that gamers play at least the entire trial (8 minutes) then be allowed to vote. Thereby it curbs ninja rating.
Reply
What I propose is that gamers play at least the entire trial (8 minutes) then be allowed to vote. Thereby it curbs ninja rating.
Posted: Apr 19th 2011 10:40PM Grubasaurus Rex said
@Ezio Auditore da Firenze That's right. If it makes sense, it cannot be done.
Reply
Posted: Apr 19th 2011 11:46PM Jack Kevorkian said
@Tachyonic Cargo which game being released in the future has ratings? Are you talking about Amazon or XBL? Because on xbl you can rate demos and game vids for games coming out in the future. Which should not be changed.
Reply
Posted: Apr 20th 2011 9:23AM xreadmore said
@DVMGames Richard
Just so you know, you can't purchase an item more than once, so to purchase an item 10 times you'd need 10 different accounts. The minimum amount of points you can purchase on an account is 500, so you'd end up buying 5000 points to do what you're claiming. That I think is more silly than what is currently happening.
Ratings should be restricted to people who bought the game. End of story. Only allowing Gold members to rate something is just stupid because I can have a "free" account and purchase games so why should I not be allowed to rate the game I bought??
This is a non-fix and is more a political move so it looks like they're doing something when they really haven't.
Why should they only allow purchasers to rate? Hard to believe I have to address this, but unless you've played a game (for more than the trial) you can not give a proper rating!!!!
If Joystiq game reviewers started to rate games and said "Well we played the demo and here is our verdict" then everyone would call them out for not giving a proper review. Same goes for these ratings. Unless you played the game you can't really give a fair rating! Personally I'd trust three reviews from people I knew purchased the game, than 300 reviews from random who-knows-what.
So one of two things should happen. MS should just remove the rating system because it's totally unreliable and thus pointless, or the Devs should just stop complaining because the rating system is totally unreliable and thus pointless.
In the end, the rating system is just unreliable, and thus pointless.
Reply
Just so you know, you can't purchase an item more than once, so to purchase an item 10 times you'd need 10 different accounts. The minimum amount of points you can purchase on an account is 500, so you'd end up buying 5000 points to do what you're claiming. That I think is more silly than what is currently happening.
Ratings should be restricted to people who bought the game. End of story. Only allowing Gold members to rate something is just stupid because I can have a "free" account and purchase games so why should I not be allowed to rate the game I bought??
This is a non-fix and is more a political move so it looks like they're doing something when they really haven't.
Why should they only allow purchasers to rate? Hard to believe I have to address this, but unless you've played a game (for more than the trial) you can not give a proper rating!!!!
If Joystiq game reviewers started to rate games and said "Well we played the demo and here is our verdict" then everyone would call them out for not giving a proper review. Same goes for these ratings. Unless you played the game you can't really give a fair rating! Personally I'd trust three reviews from people I knew purchased the game, than 300 reviews from random who-knows-what.
So one of two things should happen. MS should just remove the rating system because it's totally unreliable and thus pointless, or the Devs should just stop complaining because the rating system is totally unreliable and thus pointless.
In the end, the rating system is just unreliable, and thus pointless.
Posted: Apr 19th 2011 9:06PM SMTDante said
So wait, can free members not rate anything anymore or just indie games? From the way it sounds, free members can't rate anything anymore.
Posted: Apr 19th 2011 9:20PM Colonel Falafel said
@SMTDante
Free members can rate anything and everything, just not from Xbox.com. This only affects the website so that people can't make dummy accounts and fix ratings through marketplace.xbox.com
Reply
Free members can rate anything and everything, just not from Xbox.com. This only affects the website so that people can't make dummy accounts and fix ratings through marketplace.xbox.com
Posted: Apr 19th 2011 10:30PM This Little Man Says His Name Is said
@kyte420
You can go around voting games at 1 star with your gold account all you like.
Reply
You can go around voting games at 1 star with your gold account all you like.
Posted: Apr 19th 2011 9:13PM Ruinkind said
Although I have yet to witness the fruits of Microsoft's action, I give it 1 out of 5 stars.
Posted: Apr 19th 2011 9:21PM Ellimem said
That Armless Octopus is one hell of a site. They have all the biggest news on Indie games. I guess I should keep going there.
Posted: Apr 19th 2011 9:26PM Colonel Falafel said
Just to clear it up (as there seems to be some confusion in the comments), this seems to ONLY affect Xbox.com.
So, free members can still rate any and every game on their Xbox 360. This is just in place to stop people from creating free accounts with the sole purpose of boosting, or lowering, ratings on marketplace.xbox.com
So, free members can still rate any and every game on their Xbox 360. This is just in place to stop people from creating free accounts with the sole purpose of boosting, or lowering, ratings on marketplace.xbox.com
Posted: Apr 19th 2011 10:26PM Startingline13 ArmlessOctopus said
@Colonel Falafel Correct, as many were getting concerned about programming scripts which could validate falsely validate 5000 Xbox Live silver accounts, as seen here: http://www.armlessoctopus.com/2011/04/13/graphic-design-firm-contracts-script-for-5000-xbox-live-silver-accounts/
Reply
Posted: Apr 19th 2011 9:49PM kyte420 said
The only thing being manipulated is MS by indie devs with big egos.
Posted: Apr 19th 2011 9:49PM DVMGames Richard said
I do welcome Microsoft taking action to prevent further abuse in the future, but I find it disappointing that they won't be attempting to remove the abusive votes. The most recent suspicion of the source of the abuse is that there were automated bot attacks on the ratings via Xbox.com.
I'm not familiar with the inner workings of the ratings system, but I'd have thought that searching for ratings made by Silver accounts created and submitted on Xbox.com with 0 GamerScore and created from March 1-Now would yield few false positives.
The thing is, these ratings will likely never recover on their own - many games hold their Top Rated position for a matter of several months or years, but the shift in rankings for some highly-rated games has been unprecedented recently. For example, one game had consistently held a Top 30 position since before the start of the year. In the last month it's dropped outside the Top 150. And it's by no means an isolated case.
These games are already showing reduced visibility and earnings, so unless Microsoft reassesses the feasability of removing the ratings, the affected developers will continue to lose sales (which very few can afford to have happen) through no fault of their own.
I'm not familiar with the inner workings of the ratings system, but I'd have thought that searching for ratings made by Silver accounts created and submitted on Xbox.com with 0 GamerScore and created from March 1-Now would yield few false positives.
The thing is, these ratings will likely never recover on their own - many games hold their Top Rated position for a matter of several months or years, but the shift in rankings for some highly-rated games has been unprecedented recently. For example, one game had consistently held a Top 30 position since before the start of the year. In the last month it's dropped outside the Top 150. And it's by no means an isolated case.
These games are already showing reduced visibility and earnings, so unless Microsoft reassesses the feasability of removing the ratings, the affected developers will continue to lose sales (which very few can afford to have happen) through no fault of their own.
Posted: Apr 19th 2011 10:04PM onlysublime said
I like this idea. Rating systems are whack every whether it's Yelp or Amazon or whatever when there's no way to weed out the fake reviews. You can go read a "review" on some product on Amazon and it says things that anyone that owns the product knows is blatantly false. The only thing Amazon does is it tags the review as purchased so at least the review has some credibility.
and by restricting it to gold members, at least that fake guy only counts once unless he's stupid enough to pay for multiple subscriptions just to make fake reviews. the live free accounts are almost as bad as PSN where people have tons of accounts each.
and by restricting it to gold members, at least that fake guy only counts once unless he's stupid enough to pay for multiple subscriptions just to make fake reviews. the live free accounts are almost as bad as PSN where people have tons of accounts each.
Posted: Apr 19th 2011 11:07PM zoron007 said
How do they know free members are doing this and not gold members?
Posted: Apr 19th 2011 11:52PM DVMGames Richard said
@zoron007
Because of this - http://www.armlessoctopus.com/2011/04/13/graphic-design-firm-contracts-script-for-5000-xbox-live-silver-accounts/
Reply
Because of this - http://www.armlessoctopus.com/2011/04/13/graphic-design-firm-contracts-script-for-5000-xbox-live-silver-accounts/
Posted: Apr 20th 2011 2:49AM i77ogical said
Of course Joystiq has to say "alleged" manipulation. We wouldn't want to tarnish the reputations of the fine people who troll indie game ratings. Imagine how hard it would be for them to get jobs, housing, and educational opportunities, if Joystiq were to accuse them falsely. Great save, 'Stiq!
Posted: Apr 20th 2011 9:41AM Mike Wall said
@i77ogical Of course its alleged manipulation. Microsoft hasn't come out and officially said what happened, other than the fact that they were looking into things and this is their solution.
So, anything else is merely speculation and interpretation. It is irresponsible journalism to state things as fact without having any confirmation of it truly being fact, thus it is alleged.
It's possible that it's the work of 1 person with script, it's possible that it is the result of numerous fanbases attacking and counterattacking each other, or it's possible that it's the law of averages, and as games with niche appeals receive exposure to wider audiences, those audiences don't embrace them.
Reply
So, anything else is merely speculation and interpretation. It is irresponsible journalism to state things as fact without having any confirmation of it truly being fact, thus it is alleged.
It's possible that it's the work of 1 person with script, it's possible that it is the result of numerous fanbases attacking and counterattacking each other, or it's possible that it's the law of averages, and as games with niche appeals receive exposure to wider audiences, those audiences don't embrace them.
Posted: Apr 20th 2011 3:24AM dblue said
if the game isn't on your gamertag you shouldn't be able to rate a game
Posted: Apr 20th 2011 8:30AM Negatron said
@dblue
Thats a little extreme if you ask me, since that would require actually owning the game...
Better off with a granting permission at a minimum of having the demo downloaded. If not and purchase is necessary then at least give out the Dev's email so the appropriate hate mail* can be sent.
Some games on there aren't worthy floating around the abyss of a smart phones app. market...for free*
Reply
Thats a little extreme if you ask me, since that would require actually owning the game...
Better off with a granting permission at a minimum of having the demo downloaded. If not and purchase is necessary then at least give out the Dev's email so the appropriate hate mail* can be sent.
Some games on there aren't worthy floating around the abyss of a smart phones app. market...for free*
Posted: Apr 20th 2011 3:33AM gofish1985 said
To rate items you must purchase it first. It only makes sense why can someone rate something without a purchase??? People will just butcher games for the hell of it.
Posted: Apr 20th 2011 8:05AM Negatron said
Like EVERYONE is saying it makes sense to at least download the demo in order to have the option to vote. Now with that out of the way, I must say either I'm getting old or jaded but, many of the games available are remarkably bad.
I've tried about 2, on a whim, no research, no outside reference, just browsing and reading descriptions.
Haven't been in that (indie) tab since. *HORK!*
I've tried about 2, on a whim, no research, no outside reference, just browsing and reading descriptions.
Haven't been in that (indie) tab since. *HORK!*
Posted: Apr 20th 2011 4:08PM rbtroj said
They shouldn't do star-based ratings. A game is either good or bad. Sure there's a range, but as far as making a purchase recommendation don't you think it's enough to have the option to say "yes" or "no"? Have thumbs-up/thumbs-down buttons and allow one per paid user. Better yet, avoid downvote storms and allow thumbs-up only. If a person recommends a game, they click thumbs-up. If not, they click nothing. That's probably the fairest way to do it.







