Demonstrated: Controlling facial expressions in-game with camera

by Christopher Grant on Oct 15th 2009 10:45AM

There are two things we can say about Torben Sko. First: He's probably never had a problem getting 'torbensko' usernames. We'd like to applaud his forward-thinking parents who clearly saw the benefit of a unique name in the distant, internet-enabled future. Second: Sko's technology demos (remember this head-tracking demo?) have helped us conceptualize what a Natal-enabled future will be like, more so than jury-rigged versions of Burnout Paradise or Katamari Damacy.

In his latest video (found after the break), Sko demonstrates how to control the facial expressions of a virtual character using nothing but a plain ol' web cam (oh, and the same FaceAPI software he used for his first demo). Much like Hideo Kojima's visions of a Natal-enabled future, Sko's never involve replacing the controller (or keyboard and mouse) but rather supplementing it with, in this case, your face. No word on support for particularly ugly faces, but we suspect Microsoft's got its best and most beautiful people on it.

Source

Tags: faceAPI, Microsoft, Project-Natal, Source-engine, Torben-Sko, Xbox-360

Reader Comments (61)

You haven't Quincy'ed... until you've Quincy'ed with Natal.

I can pretend to be Sean Connery! Fucking Awesome!

More importantly, Sean Connery can pretend to be you.

"I'm waiting to be impressed"

reminds me of the japanese PSeye tech demo from a couple of months ago

can we get Jenna Haze to do this please?

YOU can actually BE Jenna Haze...

*shudder*

I can't tell if that would suck more or less than the game Haze.

I've seen her work, she sucks pretty hard.

I think we can all agree that no matter what Haze we are talking about, they all blow.

*slow clap*

well said gentlemen

It would be pretty hilarious to be playing Gears 3 and literally see the look on your enemy's face when you jump around the corner and chainsaw him in half!

whoa whoa whoa whoa....theres a halo 26?

yeah that's exactly what keeps innovation away from most games. Geez

I CAN'T WAIT FOR QUADRUPLE WIELDING!
Hey.
Guess where he holds one of his guns! :D

Chief: "Sarge, cover my rear... I'm goin' in!"
Johnson: "But you're covering your own rear! Aw hell... don't go dyin' on my now soldier! Show those alien scum we ain't going down without a fight!"

Now they need to build a Star Trek game around this, where you control Spock and must lift your eye brow up at the correct moment during various encounters.

You could also make statements about how illogical things are by using a mic.

The graphics in Half Life, and that of any game running on the Steam engine is, and always will be, substandard in todays market.
The engine is simply not powerful enough to perform at todays standards and the graphics serve only for hardcore fanboys who can't give it up and nostalgia

...

HAAAAAAAAAX!!!!!

Shut up Dr. Hax.

(In case you missed the reference, here's a link: http://www.youtube.com/watch?v=t2eUUHkpvQA )

Faceposer made easy. Maybe this will actually work on systems other than XP.

I'm with Spyder.

At least Source games have style and don't look like they were shat out by the vaseline monster, except for TF2, but that was intentional!

Kind of cool, but not that much, really. Maybe this would be most useful for motion capture, having the voice actors themselves guide the facial expressions on characters?

I would like to see a practical, real-time application in a game.

Yeah, I don't really see how this could be something that would enhance actual gameplay.

Maybe Milo can tell see how you're reacting to things he says or situations and then respond accordingly or something.

This could be the best enhancement to multiplayer gaming since XBL. Would do a lot to make it feel more personal, more like actually hanging out with friends.

Or you could ... you know... actually hang out with friends?

I'm hard pressed to find an application for this that I can care about, but it might slightly enhance online communication, in say "Home" or something. I don't know, it doesn't really get me too amped though.

I don't get it either. how often do you see your characters face? Unless it's a game based around making stupid faces...

Confirmed: Stupid Faces - Q4, 2010.

Oh, the porn industry will figure something out. They always do. You'll see.

Confirmed: 'Oh' Faces - Q4, 2010.

@Enzemo.. Steam isn't a game engine. Steam is a service to purchase and download PC games.

I think Half Life 2 looks dated, but Episode 2 more than keeps the world pretty. As for style, I don't really think Half Life has ever had much of a "style" beyond realism.

But, the Source updates keep the engine looking fresh. What it lacks in specular maps it makes up for in lighting and animation, as well as performance.

Besides, TF2 is a great looking game, especially with it's style.

I remember Sony doing something similar. Sony's had problems with the face spazzing, and this seems less evident with this implementation. However, these guys didn't turn their heads to the side, where there were the most problems.

Im happy that my lifeless zombie stare can be captured and conveyed on screen while I play games.

Looks laggy :(

Everyone has done more with existing tech on game console than the console manufacturers themselves. Why is it always some guy who pulls this stuff of yet nobody in authority ever implements it into a game? I mean I'm so glad I bought the Live vision camera for R6Vegas and NOTHING ELSE!

I find the picture a bit disturbing with the guy having such a long face...

Wow...that looked absolutely terrible. I guess if you want to be able to track whether or not the lips are moving (nevermind accurately), and track your eyebrows poorly, this is useful. Otherwise, file away as, "OK. Next."

There's really no application for this.

Is it possible I can mouth "Fuck you" without actually saying it?

I'm not really sure why people say there's no application for this. If the tech was refined a bit to get some better / more accurate expressions, then this could be a great tool to help improve facial animation in games... which is something that is currently severely lacking (with a handful of games as an exception). This is NOT to say that it would replace animators in the task, but that it would help easily create more base animations quickly and easily for animators to work with.

"If the tech was refined a bit to get some better / more accurate expressions"

You see, that's the problem.

Ashtika makes my point exactly. Perhaps I shouldn't say there's no application for this, but I should say there is no application for this product specifically. It looks terrible. There are already facial mocap applications and software out there that do a far more accurate job. All facial movements in games are not hand animated. Heck, a quick google search will get you to OptiTrack which is doing facial mocap for about $5-6,000. That's well within a game developer's budget, and they'll actually get exportable data and far more accurate motions.

This quality of tracking was a starting point...about 10 years ago.

Even if you don't like the tracking job it does, this thing still has some interesting potential. Ok, so you're complaining about the quality compared to some more expensive mocap facial software. Keep in mind that this is happening more or less in real time using a single webcam. The stuff you're describing (Optitrack in your case) isn't even set up for the same purpose.

Ok, let's say that it wouldn't specifically be a tool for animators per se. Just imagine being able to create facial expressions on your own character while playing a multiplayer game that reflects how you're ACTUALLY reacting at that time? Optitrack won't do that. If think tracking is improved for this kind of program, we could have even more immersive community games. The fact that this can be done with a simple webcam rather than a more costly rig is the thing I find impressive.

It's the source 2 engine. It was plain source for halflife 1, and became source 2 for halflife 2. Even though they refer to it as only source. And halflife 2 still looks awesome, even on 360, even though it's just a port.

famous titties for 300, trebek!

lmao, that screen shot looks like shit.

The douchebag making the face, and connery look nothing alike in facial gestures.

There always needs to be a starting point though. It's a little silly to think this wouldn't be added to. I still think this application has a lot of potential.

aaaaand this was supposed to be a reply to Ashitaka

| 1 | 2 | Most Recent | Next 50 Comments

Sorry, you must be logged in to leave a comment.

Breaking News

The most popular posts
in the last 7 days

Vita 'UMD Passport' won't be offered in US 221 comments
Kingdoms of Amalur: Reckoning review: A tempting fate 153 comments
David Jaffe leaves Eat Sleep Play, layoffs hit developer [Update] 107 comments
Blizzard taking Valve to court over 'DOTA' trademark 96 comments
Don't call it a remake: Final Fantasy X is a 'remaster,' to be clear 95 comments

Demonstrated: Controlling facial expressions in-game with camera

Reader Comments (61)

Posted: Oct 15th 2009 10:49AM copa said

Posted: Oct 15th 2009 10:55AM Courtney said

Posted: Oct 15th 2009 11:00AM (Unverified) said

Posted: Oct 15th 2009 11:46AM eat it said

Posted: Oct 15th 2009 10:56AM MaliceMajorE15 said

Posted: Oct 15th 2009 11:04AM ChomskyKnows said

Posted: Oct 15th 2009 11:31AM Uncle Jesse said

Posted: Oct 15th 2009 11:43AM Chaz Winterbottom said

Posted: Oct 15th 2009 11:48AM eat it said

Posted: Oct 15th 2009 11:53AM Uncle Jesse said

Posted: Oct 15th 2009 12:45PM Xemplar said

Posted: Oct 15th 2009 1:20PM Jjjjames said

Posted: Oct 15th 2009 11:04AM WiNGSPANTT from TopTierTacticsco said

Posted: Oct 15th 2009 11:22AM PedoJokerBear said

Posted: Oct 15th 2009 11:29AM TheDarkWayne said

Posted: Oct 15th 2009 12:37PM ZaxCG2 said

Posted: Oct 15th 2009 2:19PM sonicspike41 said

Posted: Oct 15th 2009 11:21AM sonofman said

Posted: Oct 15th 2009 11:23AM (Unverified) said

Posted: Oct 15th 2009 11:31AM PoisonedAl said

Posted: Oct 15th 2009 4:06PM Dr Acula said

Posted: Oct 15th 2009 11:32AM Timerider said

Posted: Oct 15th 2009 11:32AM Assmar said

Posted: Oct 15th 2009 11:34AM mr nimblewick said

Posted: Oct 15th 2009 11:58AM Uncle Jesse said

Posted: Oct 15th 2009 11:42AM ThatFuzzyBastard said

Posted: Oct 15th 2009 12:39PM ZaxCG2 said

Posted: Oct 15th 2009 11:46AM JeezWhiz87 said

Posted: Oct 15th 2009 11:51AM eat it said

Posted: Oct 15th 2009 11:54AM Uncle Jesse said

Posted: Oct 15th 2009 12:36PM Xemplar said

Posted: Oct 15th 2009 12:43PM Uncle Jesse said

Posted: Oct 15th 2009 12:19PM abelpc said

Posted: Oct 15th 2009 12:24PM RageOverdose said

Posted: Oct 15th 2009 12:35PM (Unverified) said

Posted: Oct 15th 2009 12:38PM Dirty said

Posted: Oct 15th 2009 12:46PM Vcize said

Posted: Oct 15th 2009 1:24PM horngreen said

Posted: Oct 15th 2009 1:31PM fourblades said

Posted: Oct 15th 2009 1:36PM berzirk said

Posted: Oct 15th 2009 1:43PM (Unverified) said

Posted: Oct 15th 2009 1:50PM Gaddes said

Posted: Oct 15th 2009 2:53PM Ashitaka said

Posted: Oct 15th 2009 5:28PM berzirk said

Posted: Oct 15th 2009 5:44PM Gaddes said

Posted: Oct 15th 2009 2:40PM (Unverified) said

Posted: Oct 15th 2009 2:51PM nadomars said

Posted: Oct 15th 2009 2:54PM 1251 said

Posted: Oct 15th 2009 2:59PM Gaddes said

Posted: Oct 15th 2009 2:59PM Gaddes said

Info

Description

MSRP

Release Date

Genre

Developer

Publisher

Rating

Breaking News

Double Fine Kickstarter passes $1 million in under 24 hours, breaking yet another record

Featured Stories

The Joystiq Show - 024: Kingdoms of Double Fine

Shank 2 review: Refined brutality

Ghost Recon: Future Soldier brings information warfare to the front

The most popular postsin the last 7 days

Engadget

TUAW

Massively

WoW

The most popular posts
in the last 7 days