Heavy Rain in the "Uncanny Valley" [update 1]
by Conrad Quilty-Harper 
{ May 18th 2006 at 9:25AM }
Clive Thompson over at collision detection has spotted a very bad example of the "Uncanny Valley" rearing its ugly head (literally) in the trailer for the upcoming PS3 game, Heavy Rain, the sequel to Fahrenheit: Indigo Prophecy. The theory of the "Uncanny Valley", realized by Japanese robotics engineer Masahiro Mori, states that people's attitudes towards robots (or in-game characters) will become negative at the point at which the model is regarded as "almost human". Characters that lie within the valley are technically realistic, but their subtle differences compared to real humans freak us out.This trailer well and truly scrapes the bottom of the valley: the poor lip animation, the glazed eyes and the lifeless skin combine to make the model look like a mash-up of Sofia Coppola and a deformed porcelain doll (ouch!). Now that console hardware is capable of near-CGI quality graphics, game developers and artists better watch their backs if they don't want to unintentionally scare their customers. As for designers of horror games, they can quite happily add another tool to their belt.
[Via collision detection]
Update: added an extra word that restored sense to the sentence (and to life, the universe and everything).
Get a WordPress.com Blog
Reader Comments (Page 1 of 2)
eletodd @ May 18th 2006 9:33AM
wow...that was bad. Holy teeth, did she get a free pair of shoes with those chompers!....and whats with the weird unfinished hands the corners of the mouths?
CakeOrDeath @ May 18th 2006 9:34AM
She looks so much like Micheal Jackson!
Justin @ May 18th 2006 9:38AM
Wow, that is really dreadful. The body animation is pretty great, but everything else is frighteningly bad. One again, HL2's facial animation engine FTW!
elmer @ May 18th 2006 9:41AM
Kojima and the Square guys figured out the solution for this:
Make everyone an over the top cool characature of reality. Barely open character's mouths, ensure all girls have high squeaky voices, all males have extremely deep voices. The funny thing about the lip-synching in this video is that Valve have clearly done a better job with HL2, and their system is almost fully automated.
Most people play video games for escape. If they wanted to watch talking heads all day long the soaps are on at noon. I say the industry should stick with the caracatures for another 3 or 4 years until they've perfected the artificial human. Then they can spring it on us and scare the Bejesus out of us.
boots (former bd (former b)) @ May 18th 2006 9:42AM
OK, I wouldn't classify it as "ugly", but anyway... it looks great. Sure, they have to *heavily* work on the lip synch, but it's still in development, so they've got time.
Other than that, the movement on the "chick", the facial gestures, the lightning effects, etcetera. It all looks very very realistic. In fact, the first 5 seconds of the video, I didn't know the trailer had started already! I thought it was a commercial or something until I realized that the lip synch was shit (ok, not shit, but again, needs improvement).
Like Killzone, it shows promise, except this looks a lot more genuine (i.e. not pre-rendered).
boots (former bd (former b)) @ May 18th 2006 9:43AM
Oh, and by the way, you have to remember E3 2005 when they showed Alfred Molina being actually rendered by the dev-kit. That was more impressive, and I look forward to games like that.
Let's not forget about gameplay though.
Chris @ May 18th 2006 9:44AM
I think one of the basic problems with humans in computer animation and computer games is that an accurate graphical representation is really only a small part of the equation. It's our subtle behaviors that convey our humanity. Compare The Polar Express, with its 'walking dead' children to The Incredibles. The Incredibles are clearly less human looking, but the way the character's eyes dart around and the subtle expressions they make convince us that they are human.
As it is now, most ultra-realistic human characters in computer games come off as robotic puppets. The entire face needs to be 'alive,' not just the mouth and the eyelids. It's like to think that better facial motion capture could fix this, but it really didn't seem to help with any of the characters in The Polar Express.
Zero_ @ May 18th 2006 9:45AM
Having seen the trailer, yes; the mouth synchronisation is somewhat exaggerated and the addition of the large eyes gives her the disease known as 'MaikeruuJackuson-itis', which isn't a good thing.
However, the movement of that character is extremely fluid and life like. Not only that, but when she pulled out that fantastic looking gun, you can really see the tension in her face.
This 'uncanney valley' is certainly an interesting theory, but who cares? It looks good, and I was intruiged by a trailer that was -just a woman talking-!
apoc06 @ May 18th 2006 9:45AM
The x360 and the PS3 will both encounter this problem. I predict that this will especially with games that are produced abroad; if you think that the subtle inconsistencies of her lip movement were odd, imagine watching her lips move in synch to a totally different language.
jaemz @ May 18th 2006 9:52AM
I appreciate the work that must have went into this clip, but there were a few issues that really turned me off.
First, they try too hard in areas that didn't make sense... why did they add the camera wobble and bad framing/focusing? and when the 'character' spoke, her lips barely moved, and when they did it looked like the lips might fall right off her face. A little more attention to detail here would have really made a difference.
I understand the difficulties of making realistic eyes and skin. The light absorption, reflection/refraction of real skin/eyes must be very hard to recreate in a computer environment. Maybe because the lip-syncing was so bad in this video I didn't notice the skin textures... but I sure noticed those teeth (not even going there, yikes).
Maybe it's just my preference, but I would rather see a Buzz Lightyear/Woody type CGI character than something that "truly scrapes the bottom of the valley".
apoc06 @ May 18th 2006 9:54AM
Also, I noticed something about the Wii that seemed remotely similar. Did anyone else notice how Super Smash Brothers Brawl contained a Mario character model that had [relatively] finely textured clothing, but the model for his face remained cartoonish? [...yes, I know they are still working to make the texturing more uniform]
As technology advances, is it always necessary to push the unrealistic towards realism? At some point you have to decide to stay in the realm of fantasy or asymptote towards realism.
Think about it. How much more "real" can you make Mario without losing the intrinsic qualities of his face that we identify Mario with?
Being that it is not a goal attainable within the confines of modern console technology as it stands, I believe thats why Nintendo for so long took the "cartoonish" stance. For example, why it made Wind Waker cell shaded instead of more realistic. Maybe they have the right idea? You can't make Link more "real"; you can make a more realistic model and dress it the same and call it Link, but you still have to hope that consumers can make that mental association and transition between models. With Wind Waker, they did the opposite and still failed. ["Link isn't blonde!!!"]
TravistyOJ @ May 18th 2006 9:55AM
Yikes!!!!
That was disturbing to say the least.
Jules @ May 18th 2006 9:56AM
Agree with previous posters, yes some aspects of the animation still need heavy tuning, but you can clearly see a lot of talent and really good artistic direction on this.
So why comment so negatively on a project that's already really under-publicized?
For those who haven't played it, indigo prophecy was ace.
blopa @ May 18th 2006 9:57AM
People stop it.
Thats not the problem here. We need fresh ideas
Please not another boring Cry-ad stupid namehere- generic shooter.
Give me those distorted grapichs any day.
If I was a dev, I would made this game ?Awful plastic surgery 2? this misshapen art style ? kinda like the Soundgardens Black hole sun video- would be perfect. ( soory 4 the poor english)
David D @ May 18th 2006 9:59AM
I do think Square is aware of this when they make their cut scenes for the newer games. The exaggerated hair, anime style faces, and crazy clothing aren't real life, even though you can stare at the scenes amazed at how real they look. They conform to anime, not real life, and that removes much of the Uncanny Valley effect.
As for that video...the subtle gestures of her hands and the crossing of her legs was nice, but that girl showed waaaay too much teeth when she talked. She talked less like a human and more like Mr. Ed. I think they laced her gums with digital peanut butter.
polly @ May 18th 2006 10:00AM
"However, the movement of that character is extremely fluid and life like."
That's because David Cage used motion capture. They didn't actually animate her movements. The facial expressions are very ugly though. Personally, I thought Cage's Indigo Prophecy was interesting, but it wasn't really a game. It was more of an interactive movie. And I don't think that's where we should be heading in games.
djork @ May 18th 2006 10:03AM
OMFGCREEPY. For real, that video is freaking me out. I can't look at that woman talk without getting chills.
Shaggy @ May 18th 2006 10:05AM
I think the negative reaction most of us had were due to the things that just weren't done correctly, which most have pointed out as the lip synch and the eyes. However, after reading the wikipedia article explaining the Uncanny Valley Theory, I'm a little confused as to where this video falls. According to the article, things that fall in the Uncanny Valley relate to corpses, zombies, and prosthetics to us; all these things have no real life or emotion. This "virtual actor" on the other hand, shows "signs of life" and emotion; watch her cry when she puts the gun to her head. And watch when she walks out of the room...she knows she didn't get the part, and even looks a little hurt by it. So what is it that makes this strange to us? Is it just the downfalls of the design (the lip thing and the eyes, especially), or is the Uncanny Valley?
Amir @ May 18th 2006 10:08AM
AAAAAAAAAAAAAAAHHHHHHHH! Man, that pic was freaky. I remember thinking when I rented and completed Fahrenheit, "Man, if only the character models were better". At least those ones didn't make me crap me pants.
Fan @ May 18th 2006 10:13AM
I was rather bored with Farenheit too(European name for Indigo prophecy). But i really admire his work with the camera. The whole picture in picture 24 series style was a huge hit with me. Also the acting was top noch. As it was in this clip.
MetaHuman @ May 18th 2006 10:14AM
Of course the best way to animate is manually and with qualified animators/artists. Motion capture, all these animation AIs implemented but UNTOUCHED in the final game will look like crap, regardless. When Euphoria was shown off, I thought it was really cool, but during E3 I saw gameplay for Indiana Jones 2007 (a game using this AI), it looked horrendous. It's still early, as the framerate was crap too and Indy's whipping action was doublecrap. We'll see when it comes out.
Ryan @ May 18th 2006 10:16AM
Don't do it unless you can make it look real. I liken this to when CGI was introduced into movies. It completely removes you from the experience to the point to where you're commenting on the graphical nature rather than the quality of the storytelling.
Remember Star Wars Episode I (and II and III)? Ick. Some of those scenes and characters were so poorly done (and yet they were meticulously created). You shouldn't be able to tell which characters are real and which are computer generated. If you can, don't do it. That's not "realisim."
Which brings us right back to video games. If you want realism, this is not the generation you're going to get it. True realism is actually capturing the complete factual nature of things as they are (or should be). I think it's a waste of money and time to try to achieve this with sub-par means.
OddGod @ May 18th 2006 10:20AM
Personally, I enjoyed it just to see where things are developing to. It's not all that bad, it's kind of cool what they've set up there.
I remember totally turned off by Final Fantasy X by them adding ridiculous voice acting and I got through that. Oh wait, no I didn't I never finished it. All the Final Fantasys before were better. I don't know where I am going with this anymore. Um, it was neat.
While I am semi-off-topic... (THREE HYPHENS! WOOOAH!)
Post #2: I really enjoyed the capitalization of "Bejesus." This is not sarcasm, I just thought it was funny. Like it's Jesus' not as well-known brother. That reminds me of the Life of Brian...
*trailing off into indiscernable non-relational thoughts... please continue*
Platinum_Skeet @ May 18th 2006 10:23AM
#1 I said the same thing when I saw the picture. Either that or the girl from planet of the apes but without the wierd chin...
Momus @ May 18th 2006 10:30AM
If you build this game around having your face frozen with Botox, I'd say "well done."
Enycma @ May 18th 2006 10:31AM
I don't know about you guys. Seeing this gave me the chills. But it got worse when I imagined games like GTF for example. Imagine this kind of reality perfected, and people still going on killing others just for fun. Imagine that they go all the way through to perfect character animations were the people shout for help or beg for morcy to the way were it's chilling. Would you be able to really pull the trigger in the game? It still is game, but it looks so real. Would you be able to do it? I know I wouldn't. I think there's a line we must no cross in gaming that is making thinks as real as possible. The "Uncanny Valley" sounds interesting. Reminds of what happened in "The Animatrix" where they were beating up a woman, but it turns out to be a robot. But the way it begged, made you feel un comfy.
Justin @ May 18th 2006 10:31AM
The problem lies in the fact that they didn't use the facial muscles when she talked and smiled. When someone goes to talk or make a facial expression then almost all the muscles in the face are used. In this demo, that just wasn't the case.
Jon @ May 18th 2006 10:34AM
"a mash-up of Sofia Coppola and a deformed porcelain doll"
This line just made my day. :)
robert @ May 18th 2006 10:37AM
You guys have probably all seen these before, but for those who haven't:
http://ndeaa.jpl.nasa.gov/nasa-nde/biomimetics/Biomimetic-robot-Hanson.mov
http://androidworld.com/Hanson-Head.wmv
http://www.akihabaranews.com/en/news-11711-SDTV%3A+EveR-1+l+Actroid+Korean%2C+the+video.html
There is also a really cool japanese female android that I can't remember the link to - similar to Ever-1, but IMO better - if anyone has this I would like it.
samson120 @ May 18th 2006 10:45AM
"The problem lies in the fact that they didn't use the facial muscles when she talked and smiled. When someone goes to talk or make a facial expression then almost all the muscles in the face are used. In this demo, that just wasn't the case."
To add to this, you see a flicker of light (that is supposed to be the tongue) but you never see the tongue when talking, it looks like a deep black hole. This also made the teeth stand out too much. Some items in the scene (frig magnets) apparently didn't run through the same lighting engine as the rest of the scene (notice how they glow). Her skin was way too smooth looking (light reflected and shined rather than being absorbed).
A few other things didn't work for me, like lake of breathing, some of the hair, etc....
And to answer one of the other questions, why did they use out of focus shots? It shows a state of mastery. Shooting something out of focus (esp. in an artistic manner) isn't simple in a digital age. Even more difficult is switching focus. When you see this in movies (esp animation Lion King pops into mind) the tech is saying "Look what I can do..."....
anyhow my two scents.
jaemz @ May 18th 2006 10:51AM
Thanks robert. That 'hanson-head' android is freaky... I would take 3P0 as a sidekick anyday over these (and the Philip K. Dick android).
e. @ May 18th 2006 10:57AM
What the HELL was that?
I'm freaking terrified.
I don't exactly agree with the "mash-up of Sofia Coppola and a deformed porcelain doll", I'd say she looks more like Penelope Cruz mixed with a blow-up doll with the mannerism of Jack Sparrow and a fistful of marbles in her mouth.
I won't be able to sleep for a year now. Thanks a bunch.
QuanticDream is the anti-christ of game developers, but it's nice of them to let us know that we should avoid the game even before a demo comes out.
They should do something like this for Omikron and Fahrenheit to spare me some of the pain.
Ahms @ May 18th 2006 10:59AM
I hope they stay away from sytlized characters, such as Link, Sonic, etc. Characters like that should stay in their original stylized concepts; seeing them as "realistic" makes me lose my appetite. I remember when they fully rendered Mario and Luigi for the Super Mario Brothers Super Show, right down to the pizza stains. That was disturbing
Jonn @ May 18th 2006 11:02AM
I'm proud to say I was the guy who sent him the link. My first thought was "Clive Thompson has got to see this."
Tad @ May 18th 2006 11:04AM
As a huge fan of Indigo Prophecy, this has me very excited. Although it had it's issues, I liked what this trailer (or tech demo) was doing with both character and camera motion. The acting was also decent, for a video game at least.
It's weird how such a small thing, such as her upper lip never moving, can have such a profound effect on the viewing. Again, maybe it's because I want to love the game that I can forgive some of the other "Uncanny Valley" effects such as the skin and eyes.
But I will say that the fluid and mostly realistic movement goes a long way to making the character believable.
epobirs @ May 18th 2006 11:05AM
It seemed like she was recovering from a stroke or Bell's Palsy and still suffering some facial paralysis. Add to that too much effort to give her little bodily gestures and she came off as partially paralyzed above the neck and hyperactive below. This makes the viewer feel guiltily suspicious of the character. It is instinctive to avoid defective people and animals since they may not behave predictably or may have cooties of some sort. It isn't rational but it is a real drive to avoid what may seen at some mental level as a danger even if we know there is no real threat.
So what is the target platform here? Is it going to be real-time rendering with much happening in the background simultaneously? On older platforms (Xbox/PS2/GC) the limited resources could understandably cut down on the complexity of the facial animation model. If this is for PC/Xbox 360/PS3, there is far less excuse. Creating an engaging rather than subtly repulsive character is one of those improvements we expect from the next gen investment beyond prettier pictures.
Gonzo @ May 18th 2006 11:11AM
And my family thought I was crazy.
Ever since I was a kid I was always more scared of the Carrousel of Progress and Hall of Presidents than the Haunted Mansion.
It's all fake talking dead guys anyway.
Paolo @ May 18th 2006 11:11AM
I liked this. Sure, the character is no beauty queen, but there is good in that, because most people are not models. The music was excellent, that's what I liked about Indigo prophecy, and the tear detail was awesome.
I agree with the lips problems, but other than that I found it very well made.
Trachalio @ May 18th 2006 11:13AM
Wow that is totally creepy. It's got great dialogue, the body language is good, but it's just off enough to creep me out completely.
It's like watching Freddy Mercury in drag trying to audition for a role in serial killer flick.
jV @ May 18th 2006 11:14AM
Sarah Silverman?
epobirs @ May 18th 2006 11:27AM
#10
True. Upgrading the visuals on a character like Mario is a tricky business. Especially since his proportions would be grotesque in real life. 'Better' and 'real' and two separate values. The visual style of Wind Waker let the artists express everything they needed in a cartoonish way that was nicely efficient on the resource side.
On the issue in #8, that is actually less of a problem than you may think. Lipsyncing engines can be tuned for different languages phonemes and voice tracks. Not an issue at all for realtime rendering. For pre-rendered stuff you just run the render again with the localized voice track to produce a separate final animation for that region. One you've got every scene rendered once to satisfaction it isn't a big deal beyond time on the render farm to do another run through for different voice tracks.
Pixars has been doing that kind of customization for a long time. One of the first examples was on 'A Bug's Life' where they produced a separate version of the movie for non-widescreen TV presentation. The cost was fairly small, especially since an editor didn't have to produce a pan and scan cut of the movie. As a result the viewers of the TV version lost something from the theatrical presentation but nowhere near as much as is normal for a movie 'shot' in widescreen being altered for 4:3 TV.
Pixar and other CGI productions have since gone on to do the same for foreign voice releases where the market was big enough to merit it. As machine time for high quality CG drops in cost it becomes a normal part of doing business. But it only has that merit if the facial animation was good in the first place.
kevin @ May 18th 2006 11:28AM
...personally, i don't even want ultra-hyper-photorealistic characters.
...i think it's kind of a gyp. i see people all day every day. if i want to see ultraphotorealism in maximumm definition, i'll turn to my right and look at the soccer mom in the cubicle across from me.
...i look at games for different environments, different art styles.
...also... the incredibles was infinitely more believable than this 'heavy rain' tripe. she looks like an animated MANNEQUIN. kinda cool to watch, but c'mon: the lip animation is terrible, and the inside of her mouth is horrifying. her skin looks hard and, if you look closely, you see she moves like the tin man. her fingers are stiff and don't "rest" on anything.
...if developers would stop trynna wow us with ultrahypersuperduperphotorealism, maybe we'd bypass the "uncanny valley" and have actually interesting artistic styles instead of the hollow, soft-plastic skinned mannequins that populate games nowadays.
Staticneuron @ May 18th 2006 11:31AM
"Characters that lie within the valley are technically realistic, but their subtle differences compared to real humans freak us out."
I think the theory is meant for Hyper-realistic characters such as Aki from FF:spirits within. I think that this females facial animation is actually pretty horrible. (not saying that these guys cant do better) But her mouth movements, way to subtle face movements added to her "teeth all the time" dialog doesn't scream Realistic to me.
martin @ May 18th 2006 11:53AM
http://comoedia.fr/afficheActeur/7008
Here is the real actresses page with an image on it. They used huge eyes because she really does have huge eyes. I think later on when they are actually doing the scene some of it can look really good. Still a long way to go though. Facial expressions would be helpful. If you notice though, any profile shot actually does seem to pull it across a little better.
Eric @ May 18th 2006 12:01PM
The only thing that will fix this "Uncanny Valley" is Playstation 4's "emotion engine 2!"
Jules @ May 18th 2006 12:01PM
As an addition, E3 commenters (GS) told this was not pre-rendered, so it can't be compared to any cut scene from Square or Konami, say.
Btw, I think it's meant to creep you out, even a little bit, à la David Lynch.
Pete S @ May 18th 2006 12:04PM
Anyone remember when Joystiq was about all the fun in videogames, instead of a bashing site?
I like how they grabbed probably the ugliest frame of that video for the post.
Can anyone suggest a replacement site? A place that's about gaming as a source of fun rather than a topic for hating on?
Darth Pixel @ May 18th 2006 12:06PM
Wow!
This looks great!
If you compare that to Halo 2's in-game cinematics rendering, it's like night and day.
This whole thing is progressing much faster that expected.
jV @ May 18th 2006 12:15PM
#44: Yep, try www.sensitivebitches.com
"I like how they grabbed probably the ugliest frame of that video for the post."
Are you frikkin' kidding me? You're reeeeeeaally reaching now, aren't you?
Eric @ May 18th 2006 12:15PM
Sure Pete,
Go to PS3fanboy.com
I'm sure you'll feel more welcome there.