A full Minority Report interface is one of the endgame dreams of Kinect hackers. We've seen plenty of brilliant Kinect hacks so far, but this latest one by members of the MIT Computer Science and Artificial Intelligence Laboratory is the closest thing we've seen to the dream.
The hack uses several components to distinguish fingers, allowing hand detection of 60,000 points at 30 frames a second. If you want to go down the rabbit hole of techno-babble gobbledygook (yes, it's a scientific term), you can go here. For those who just want to see something cool, go past the break.
@vidguy The reason we still use the mouse is because it's so undeniably efficient. Moving the mouse a quarter inch can move the cursor across half the screen. Scrolling is fast. At the same time, it's highly accurate and precise because it easily switches between wider movement gestures smaller pointing ones.
Touch-screen is nice on small devices, but wholly impractical on larger displays or for extended use, as you would only ever want to touch downwards, not forwards, so mounted displays suffer. Touchscreens also have a 1:1 efficiency. Each inch of movement across them is an inch of movement you must do, which adds up.
These gestures suffer from the same problem. They are neat to watch but too wide for extended use. Why would you want to wave your arms like that when you can move your wrist a quarter of an inch, or your index finger across a wheel?
The only solution to this is extremely advanced finger tracking. If a sensor could be mounted beneath a laptop screen, for instance, and merely raising your index finger gave you mouse control, that would be impressive and useful. The tiny degrees of difference between different gestures would be absurdly complicated to map - I don't think impossible, but absurdly complicated, and Kinect isn't precise enough to make those sorts of differentiations.
Also, FPS gamers are never giving up that mouse. You'll have to pry it from their cold, arthritic, Alzheimers-ridden hands.
Oh I definitely agree about the mouse's efficiency. But you don't think it could be improved? Hell, there has already been proof of concept of input directly from the user's mind using electrodes. In 2054 we might be interacting with computers without making /any/ physical movements.
The mouse has already been significantly changed (more buttons, optics instead of trackballs, scroll wheels) and I just think it silly to assume we won't see further advances in input devices over the next 40 years.
Have you heard of Brain Fingers technology? It allows you to manipulate things on a PC screen using your mind. The future is nothing at all but thought to manipulate our external devices.
You're right, with today's technology motion tracking is not the way to go. I think given 30 years though the mouse might not be around anymore. I think the motion tracking will refine to the point of advanced, complex finger tracking. Most likely just mimicking a mouse movement by pointing and clicking in the air. Even further in the future though we'll have retina tracking and brainwave pattern recognition. All of this is around today, just far to primitive to be useful in daily situations.
But screw Minority Report you know what the real goal is? Air Guitar Hero.
@vidguy Oh, we'll certainly see advances. Here's the thing about brain wave tracking - it's not as efficient as you might think. Now, you might wonder how that could be the case.
Take a professional guitarist. Part of learning to play guitar is committing patterns to muscle memory. These movements become instinctual and no longer involve the brain. If you gave a professional guitarist a mind-guitar that instantly played any chord or note he thought of, he'd be significantly slowed down by having to actively think about them.
As long as brain tracking requires focused concentration on a task - New Tab, Scroll Down, etc - it won't be nearly as quick as physical input methods. The human body already has built-in shortcuts; pulling your hand out of fire, firing off a sick riff, and touch-typing all involve doing things before your brain realizes that the action is taking place. The human nervous system is FAST and reacts to unconscious thoughts. Truly responsive mental interpretation would have to interpret your desires _before you consciously knew you had them_; that is how fast the human brain works. And as long as it's thinking about raising the cursor, it won't be pleasant, natural, or quick. It would have to be so discrete that we wouldn't even notice it functioning, and that technology is a long, long way off. Even longer when you consider that it would have to be SO PERFECT that institutions and businesses would adopt it, and that it would be have to be competitive in price with simple mice.
Furthermore, the mouse-keyboard combo is remarkably quick. I've written so much in Word that I have Ctrl-S literally down to instinct. I never have to think about saving my work because I do it after almost typing burst. The keyboard is an incredibly efficient input device because it's multi-touch, utilizes human muscle memory, and has a vast array of input options. A user who learns to utilize keyboard controls and macros will fly around their operating system at the speed of light - no speculative futuretech needed. There are (or were) even Firefox extensions that allowed full mouseless browsing experiences. And I've argued the mouse is efficient earlier - but nothing is more efficient than the keyboard.
I'm not so naive as to think that the mouse will absolutely last for the next 30 years, and certainly it will experience more innovations of its own. Still, it's lasted many, many years and has literally never been topped in effectiveness as a pointing-based input method (except drawing pads, as they can detect pressure, but they're cumbersome for daily navigation.) Nothing I've seen from gesturing or brain controlling has surpassed it, nor do I expect it to for a long time. Multi-touch is useful IN CIRCUMSTANCES, but cannot function as a replacement for everyday computing.
Brainwave analysis will be useful, though, if it can work both ways. If computers could speak directly to our minds, Shadowrun deck style, then we could eliminate monitors for some uses.
That is the best synopsis I've read yet of why Kinect and other such devices will remain in the experimental stages, for a while at least. It deserves to be quoted widely.
exactly... plus imagine if you even wanted to listen to music while navigating your PC... on comes some hard dubstep and your brainwaves are immediately fried, rendering any type of brainwave activity null.
I'm not so convinced the mouse is going to be around that long. Likewise though, motion control is probably also a fad at best.
Ultimately, I think that what's really going to define the future of user interfaces are processing power, AI (near sentient) and display technology. For example, will we still be dependent upon projected images in 40 years, or will we figure out some trick that will allow us to create or bend light at any point in 3d space without having some specially prepared area to use it in? Will computers e both fast enough and intelligent enough to fully understand theee wide range of human speech patterns and body gestures on an individual basis without having to calibrate them first?
Without these points being met, we can assume the following will probably still be true in 40 years:
- interfaces will primarily still be used on "flat" surfaces - these interfaces will probably still largely resemble multitouch of today - input devices will likely move away from tactile surfaces in favor of efficiency - users will physically point at items rather than move a cursor to them - all devices will be voice/gesture assisted for error correction
There may be some element of Minority Report to this stuff, but if you have to stand and move the entire arm above shoulder height, it won't make the cut except for exercise, gaming or immersion.
* possible sidenote - augmented reality *
Rather than using projected interfaces, all interfaces are simply projected over the reality of the individual user via LED embedded contact lenses. This introduces some unusual properties, such as how close to an object one has to be to interact with it and whether the said object can reciproocate that interaction. Also, how do you let a user interact with an interface only they can see without looking like a lunatic flapping their arms in the middle of nowhere or mumbling to themselves... while at the same time keeping the solution from coming off as being too invasive (like a implants in the brain, neck, spine, ear or eye).
Exactly. I'm almost glad someone can finally make this input method... so people can use it... get annoyed by how tedious and slow it is... and stop talking about it.
@Bones3D I've long pondered the supra-reality display myself, brought about through some sort of more direct interfacing. Wireless glasses or contacts with some sort of yet-undiscovered display type? Speculating is a fun mental exercise, but the future is probably about something so far from our minds that we can't even conceive it.
If there's one lesson to learn from history, it's that we can't really imagine the future. We always conceive it like the present but slightly altered or so wildly, radically and ridiculously different that it's impractical. Furthermore, we both overestimate and underestimate how quickly it changes.
Look at the early nineties. Everyone said "Twenty years from now we'll just be strapping on VR helmets!" That never even came CLOSE to being the case. By the time technology actually evolved to the point that something like that become viable, it had taken so many other routes and twists that the future was quite different. Come to find out, it's all about vast data stores and wireless, always-on communication. Whoops.
And yet for all that, the personal PC looks and functions much like it did back then (though more efficiently), video games still function in largely the same way, etc.
@Kuipo That's the perfect analysis. I think you could say that almost word for word about the Virtual Boy (just replace tedious and slow with two better adjectives - isolating and uncomfortable?). Sometimes the future isn't so futuristic.
@MasterYogurt Yes, the mouse is efficient. But it is a dried up husk of efficiency next to the power of the keyboard! Mice are precise, but keys never tease.
This sorta exists, except it is refined and quite pleasant to work with. Apple makes it, and its called the magic trackpad. It tracks up to 4 fingers, actually "clicks" when you "click" it (as opposed to touch pad "tapping," which just about everyone hates), is nice and big, its accurate, and just, well, pleasant to use.
When I use my macbook (we were assigned them at work, too pricey for my taste), I actually don't bother hooking up a usb mouse. Which is saying something, because I usually hate trackpads.
It sucks for gaming, though. Mouse still wins there.
A real mouse, not Apple's "magic mouse" (no me gusta).
@Enosoma Okay, so, maybe that guy was wrong, and they're not impractical for the reason he listed. However, imagine OTHER PEOPLE driving them. There you go. Proof that they must never be made.
I'm very impressed. I wonder how much programming this took. I doubt that the games for the Kinect use the hardware very well, but put me down for the Kinect2.
@Noksukow And my keyboard/mouse will trump your stick and A button any day. And if you don't like keyboard or mouse then you can still use a controller pad. So there is no valid rebuttal.
@bigwhiteyeti somehow i don't think they give a deuce deuce about hacked fingers when they have so many GAMES coming out in the near future and beyond (and not just another video about what some dude did in his basement).
@swooded The future of slow? It's extremely cool, but I hope I never have to use this to do anything that needs me to be, like, productive. Minority Report might work for Tivo's, but I don't think it'll be too practical for real computer use.
In terms of genuinely useful interfaces, I am afraid we're going to have to skip Minority Report and go straight to Lawnmower Man.
@sigma8 Try lightening up a bit. "Welcome to the future" is a phrase commonly used for comedic effect. Beyond that, whether or not this is the most efficient user interface or not, the tech is undeniably interesting at the very least.
@sigma8 You tryed the game and noticed the lag, one second or less, but you noticed it. This video is the first kinect related video I see, that has no lag.
And that is exactly what i found extrange... obviously that "fake" part needs to be taken as joke or as i said, with "joystiq mode" on.
The thing about Kinect is the hardware is a pretty amazing piece of technology, of the 3 big companies MS did come up with something that could be the future of not just gaming but other forms of entertainment. The problem is MS isn't doing anything remotely cool with the technology it's "hackers, Proffesors, everyday people" who are coming up with better ideas for the Kinect. Could you imagine if MS had actually marketed this as something other than "you are the controller," oh MS you silly people.
About mouse vs Motion, you do have to realize the mouse is only good for your desktop PC and even then it can be greatly improved. The thing is that doing things on the PC will be something that is not regulated to just your PC and the mouse doesn't really work that well in those environments.
Case in point would be if I was doing the picture thing in my living room. Having a surface to perform those functions with a mouse is not always the best. Being able to easily do this from the sofa sitting or even laying down sounds like a plan. To bad you have to be so close to the sensor in order for Kinect to detect your fingers.
People keep saying that the hack community is the one doing all the cool stuff but not MS. What you have to realize is that the stuff you see from the community is proof of concept that most companies including MS have done plenty of work but you do not see it. Hell most hacked Kinect vids have been done already with a standard web cam including the eyetoy.
Proof of concept type ideals are easy, making a product that will sell is something totally different especially if its a game because it has to be consistant and get entertainment for a longer period of time besides 10 minutes.
I believe the real Kinect games will take just as long as any standard game of 18 months or longer because of the learning curve. People need to be patients I am sure there are a lot of concepts out there that we will not see until they surface as retail games.
