Contour captures faces, avoids Uncanny Valley
by Christopher Grant 
{ Aug 1st 2006 at 11:30PM }
Contour is the newest project from Steve Perlman -- an alum from tech companies like Apple, WebTV, and Moxi -- and it aims to not only greatly increase the resolution of facial capture techniques, but to bypass the entire Uncanny Valley thing entirely. What black magic are they employing to accomplish their task? They've dropped the white dots and traded up for some classy phosphorescent powder. The New York Times explains the process:
"The Contour system requires actors to cover their faces and clothes with makeup containing phosphorescent powder that is not visible under normal lighting. In a light-sealed room, the actors face two arrays of inexpensive video cameras that are synchronized to simultaneously record their appearance and shape. Scenes are lit by rapidly flashing fluorescent lights, and the cameras capture light from the glowing powder during intervals of darkness that are too short for humans to perceive."
The Wall Street Journal links to an incredible video demonstration (.rm) of the technology, pictured above. Naturally, there's promise for the film industry, but gaming journalist Dean Takahashi hits the gaming angle in his Mercury News piece, "The problem of animating human faces is getting harder because the latest video game consoles -- the Xbox 360 today and the PlayStation 3 coming this fall -- are being hyped for their ability to render graphics with high-definition details." Reading about Contour, I can't decide what's more impressive: the results or the relatively modest technology used to achieve them? Incredible stuff.
[Via Grand Text Auto]
Read - Camera System Creates Sophisticated 3-D Effects - NYTimes
Read - Digital Replicas May Change Face of Films - WSJ
Read - Q&A With Steve Perlman, founder of Mova - Mercury News
Read - Steve Perlman's Mova: Capturing Faces For Video Games - Mercury News
Reader Comments (Page 1 of 1)
Goober @ Aug 1st 2006 11:50PM
Uncanny valley isn't just the look, a large part of it is the movement. If you look 100% real but still move like a ragdoll, its still creepy. This technology just does the look, far as I can tell. We still need good performance/motion cap, and good animators.
BklynKid @ Aug 1st 2006 11:55PM
Is that that one girl from that one movie?
Goober @ Aug 2nd 2006 12:00AM
After reading more, it looks like it DOES do animation? Too bad I'll never watch their stupid .rm file. Anyone have a link to a real file format?
Scott @ Aug 2nd 2006 12:05AM
This Uncanny Valley concept is, in my opinion, gonna wreck havoc on the ps3. Cartoony cell-shaded games will become more popular i think, not by much, but a little. Also censorship is gonna have a field day with the ps3/xbox360. Rockstar got away with the content in the GTA series cause the graphics were abysmal. With near perfect graphics on the way (Crysis anyone?), the censors are gonna come down hard on photo-realistic violent games, which will see more violent games reach the Wii cause the censors wont see it as a bigger impact since the graphics are only twice as good as the xbox. This console war is going to be the most interesting by far.
squashedyoshi @ Aug 2nd 2006 12:15AM
Capcom and other horror game creators must see this opportunity. If done correctly the uncanny valley can creep gamers the hell out for all to enjoy.
?att @ Aug 2nd 2006 12:17AM
@ Goober
Looks like you owned your self. That file is in a "real" format. ;)
Anyways, if the section on the right is really a 3-d model, thats fricking amazing. Wonder how much power it requires to render though.
radiantthought @ Aug 2nd 2006 12:20AM
As for the real media comments we've got people slinging, don't ever download the ridiculous software real puts out... get real alternative http://www.free-codecs.com/download/Real_Alternative.htm it's small, unintrusive and just plays the videos and lets you get on with life.
BklynKid @ Aug 2nd 2006 12:22AM
#5, it is much easier to compress cartoons than it is to compress live-action video due to color and animation complexities.
Do you know anything?
Scott @ Aug 2nd 2006 12:28AM
#6
The Uncanny Valley is a lot more than realisitc looking in-game character being creepy. When a game is saturated in the uncanny valley, the interaction between the gamer and in-game character becomes more intense, and wen it does, the enjoyment of the game takes a back seat as the subconsious mind begins to scruitinize the flaws in the game.
You'd be playing a totaly realistic looking survival horror game, but you'd feel weird running up against a perfect looking table with chairs stacked on it, and the chairs dont fall down totally realistically. Locked doors will become even more annoying as the photo realism would make you think "Why can't i kick down the door".
As games become more realistic, tiny flaws become more and more obvious, making the game seem inomplete and rushed, even diminishing the enjoyment of the game. It is a big problem, which developers are going to have to circumvent.
Vincent @ Aug 2nd 2006 12:33AM
Matt,
One of the articles linked above describes the creator's computer that models, and I'm assuming, renders the scenes as a "graphics supercomputer". That may just be ignorance, or hyperbole; either way, he built it himself, in his garage, and the technology isn't going to cost all that much to implement compared to motion-capture studios, suggesting that the power needed for this kind of work is pretty great, but not otherworldly. In a year or so (when he's planning on selling the tech) the computers used to model and render will undoubtedly be much cheaper.
Ultimately, though, crossing the uncanny valley is up to the skills of the animator. Although technology like this will make animating a face a far less time-consuming process, not all animators are good with the tools they're given--just like any other artist, really. Photorealistic faces are attainable here using strictly the technology this guy created; however, what will happen when you get into more compositing stuff, like Davy Jones of the latest Pirates of the Caribbean? It'll still be up to the animator (and the actor) to breathe life into a bunch of splines. And the animators may still end up with plastic imitations of life. The valley hasn't been crossed yet.
darryl @ Aug 2nd 2006 12:52AM
"It is a big problem, which developers are going to have to circumvent."
Isn't that what this post is about? More processing power and new capture technology sounds like an attempt to circumvent to me.
Psaakyrn @ Aug 2nd 2006 1:36AM
More processing power by itself won't work. The amount of memory required to store the entire body composition is rediculously large, not to mention the processing power and data bandwidth to transfer and manipulate that information (note that the information also includes texture, which is usually seperate from the object shape). Basically we can no longer rely on polygons to store and manipulate 3D image, and will have to look for something more efficient.
As a minor note, I think this technology would still come out short in the realism department, because I doubt it's capabilities to actually store realistic textures and fine details (Didn't watch the vid, but from the exampleimage above the hair wasn't captured). It would probably help to make cel-shaded objects to look less polygony though.
Peter Davis @ Aug 2nd 2006 2:23AM
i dont get this new technology at all. How is this any more useful than simply green screening? does it create a full 3d model from the output, a projection, or motion capping-like data?
im really, honestly trying to figure out how this applies and can be useful to the games industry...so if someone wisers wants to chime in...
ill trooper @ Aug 2nd 2006 3:06AM
It looks great! Peter, I really don't know the technology so I'm just guessing, but it looks like it uses the 'grey-scale' values ('Green-scale?') created by the make up to create almost what appears to be an inverse of a texture-map on a model - it seems to take the information from the 'texture map' (the green-scale') and apply it to a complex surface/fabric of polygons, actually creating the model from the capture, rather than build a model and apply motion to it - her face changes shape entirely, not just on weird points... I'm impressed with the tank-top fabric folding and moving.
In other imaging news, did you see this on slashdot yesterday? It's about a software called PhotoSynth...
http://slashdot.org/article.pl?sid=06/08/01/1339253
ill trooper @ Aug 2nd 2006 3:10AM
Sorry, back again... Just wanted to add that I wish they would have had the 3D version from a different or moving camera angle, rather than the exact same angle as the capture.
THAT would be interesting to me as well.
Scott @ Aug 2nd 2006 5:25AM
#12
More processing power isnt enough, and it will be quite a while till it is enough. There is going to be a large gap between graphics looking like real-life, and graphics moving and interacting like real life, and it is in this gap that games trying to look perfect will seem wierd to play, with the subconscious part of your mind picking out the flaws in it's realism. It has been proven that this can detract from the gameplay experiance. In the future, perhaps the ps5 could handle it, but a game that behaves just like real life and looks the same?....even the simplest games would take years to make. Like i have said before, this coming generation is going to be the most interesting by far, with the two biggest heavyweights, microsoft and sony, battling insane development costs, hardware costs and the Uncanny Valley. Nintendo has never been in a better position to reclaim number 1, but Nintendo has a nack of not pulling things off (barring the handheld market) so they had better get down the local library and pick up the latest issue of "Successful Marketing", cause they need it.
Jdoki @ Aug 2nd 2006 7:04AM
A lot of this isn't just 'uncanny valley', it's about suspension of disbelief in the audience.
Movies use it all the time. What we see on the screen often walks a fine line between 'whoah cool', and laugh out loud ridiculous.
Video games have a harder time with this because what we see is grounded outside of what we can interpret as real. Even if we have photo realistic graphics that are perfectly animated our brains still 'know' that it is computer generated, so is harder to 'fool' it into to believeing it.
When we see a movie, such as Poltergeist - where we see chairs stacked up on a table - our brains know that they are real chairs in a real room, and subconciously ignores the fact that it is a special effect - this is suspension of disbelief.
In a game if we see a load of chairs stacked up we know they are computer generate chairs in a computer generated enviroment dictated by the computer generated physics. So our brains have more trouble suspending disbelief even if the chairs react in the same way in a movie.
Creating games will become as much about psychology as incorporating realistic graphics, animation and physics in games.
The industry is obviously getting there - the Assassins Creed demo, where the main charcater moved through a crowd realistically will certainly go a long way to helping us gamers become even more immersed in the environments.
MR @ Aug 2nd 2006 9:19AM
So wait.. instead of just filming the actors, we are filming the actors, then doing a lot of costly computer graphics effects to them, and then they look just like real actors?
Magus @ Aug 2nd 2006 9:26AM
All tho this does seem like an interesting technology I have a hard time believeing this can be used outside cinematics and or the film industry.
Game graphics in general suffer from 2 flaws. First is the polygon count. Doesnt matter how nice your motion is comming in if your face model doesnt have enough flexible points to corelate the motion.
The second is a bit more complicated. In movies you have the ability to motion capture an entire aperance of the cg character. In games you dont always have that luxury, a character may have near 10 hours of dialogue which will likly be auto generated.
As far as the limits of uncanney valley it would also depend on the game and the enviorment. No one said anything about the uncanneyvaley in relation to rezident evil, indego prophecy or the many other games that use humans without significant stylazation.
teknotom @ Aug 2nd 2006 9:32AM
#18: Yes because then they can be manipulated as we/game designers/see fit. Which is both cool and a bit scary at the same time.
darryl @ Aug 2nd 2006 9:54AM
"So wait.. instead of just filming the actors, we are filming the actors,..."
If you read the article, you'd have answered your own question. They're not just filming the actors. This technology allows them to selectively scan an actor as a 3D model, motion included - not just a static model. It's ingenious yet simple.
rasbill @ Aug 2nd 2006 10:21AM
jesus those are some lopsided titays
WizarDru @ Aug 2nd 2006 11:44AM
It should be noted that the Uncanny Valley is a theory, not a fact. Studies have been inconclusive, either way. The movie version of Final Fantasy is often trotted out as the poster child to this concept, while the huge success of The Polar Express is often ignored, just like the fact that FF:SW had a ponderous and unsatisfying story to blame for it's failure much moreso than it's visuals.
This technology is like motion-capture mixed with morphing. The idea is that you map out an actor and then can 'gollum' like stuff on a much more sophisticated level, such as the Meryl Streep example in the article. I forsee lawsuits from this the first time they try to use someone's likeness, such as Steve McQueen, with this process.
TC @ Aug 2nd 2006 1:45PM
I hope that make-up comes off easy, else her boyfriend is in for a shock tonite.
Steve Perlman @ Aug 4th 2006 10:49PM
I may be joining the discussion thread a little late, but I just got back from Siggraph. We designed Contour.
You've brought up some good issues in this discussion thread about the Uncanny Valley and the future of gaming once photoreal faces are readily available. Also, there are some questions about Contour and what it can and can't do. If people are interested in carrying the conversation further, please post questions and I'll try to answer them.
Contour captures a scene volumetrically (i.e. in voxels) at roughly the same resolution as a conventional camera captures a scene as a flat image from a single viewpoint (i.e. in pixels). It is, effetively, a volumetric camera system.
The output of Contour is pretty cool to look at, but bear in mind that Contour is just another tool in the creative toolbox. It was designed to give video game or motion picture production teams the ability to START with capture data that is photoreal, but then it is up to that team how they choose to use that capture data. Of course, one obvious use is for characters that look just like the performer. But an equally important use is to for non-real characters driven by very subtle performances (e.g. lip motion or crinkles around the eyes).
Anyway, if anyone is still following this thread and has questions or comments, I'll be checking in on it.
-- Steve Perlman, president, Mova