I was working on some Data Structure programs on my computer that night, and I was fed up because the code was having some kind of logical error. I took a break from work and started looking at a DVD which came with a technology magazine and started browsing some videos on it. I came across a TED talk by Pranav Mistry about the SixthSense technology and I was very impressed. I started imagining about the ways with which we will interact with computers in the future. I got deeper into it and started exploring the domain and I was very happy to know that a lot of work is being done to make this dream into a reality. In my final year I was supposed to do a project, and I thought of controlling computers with gestures contributing to the work done in this domain. I began with a simple webcam and image processing algorithms and with a yearlong effort with my team; we got this dream into reality. Yes, we were doing hand gestures in front of the camera and the computer was recognizing it. The computer changed slides when we did that, and we all were fascinated by the fact that we all are controlling a computer with our hands. It was like teaching a new language to someone and communicating with a computer in a very different way. All of us were proud that we did it. We got a lot of appreciations, but the truth is that only us, the people who made that program, were able to use it and nobody else. It was a great piece of technology to show off, but it was just for the purpose of showing off. In reality, it was nor useful, neither usable in the Wild.

By this time I was familiar with a lot of technologies related to interface with the computers by using gestures, motion control, and thought control. My curiosity led me to explore a lot in this domain. I interacted with different types of technologies. I did a lot of development on various pieces of technology to get closer to my dream of interacting with a computer in natural ways in reality. Now, what do I mean by natural? Natural means to me something which is already in our instincts. The way in which we talk to people in our surroundings is natural.  Whenever we talk to people, we do gestures, gestures with suggest something. These are signs which make our intentions more clear and precise. It gives meaning to what we say and gives a realization that what we say makes more sense because of these actions. There is an interesting experiment which you can try out. For a day, to whoever you meet, interact with them without moving your hands. Just keep them suspended. It will feel weird, isn’t it?  Now imagine interacting with computers as if you are interacting with humans. Sounds interesting, right?

Now is the time when I will introduce you to Natural User Interfaces. Assuming that you know what user interfaces are, Natural User Interfaces or NUIs are the interfaces which allow you to interact with computers in a natural way. A natural way is something which is in your instincts. You don’t really have to learn a lot to user them. Some examples of NUIs are touch or multi touch based interfaces, gesture based interfaces, speech based interfaces, and brain or thought control interfaces.  Steve Ballmer, the CEO of Microsoft said, “I believe we will look back on 2010 as the year we expanded beyond the mouse and keyboard and started incorporating more natural forms of interaction such as touch, speech, gestures, handwriting, and vision— what computer scientists call the ‘NUI’ or natural user interface.” [Interactions-Norman] According to the book Brave NUI World, NUIs promise to reduce the barriers of computing. They mention that it is not Natural User Interface, but it is Natural User Interface. This shows that the focus is not on making the interface natural, but on the natural actions of the user.  A natural user interface is not one that magically responds to any action, somehow correctly guessing the user’s intent. Instead, a NUI fosters the development of skilled behavior and engenders a feeling of mastery by eliciting appropriate actions from users and shaping them into skilled behavior smoothly, efficiently, and enjoyably. [Brave NUI World]

This focus of this paper is on gesture based Natural User Interfaces. They will be addressed by different names throughout the paper, but whenever I use the work NUI or Natural User Interface, I mean by gesture based interfaces. These interfaces are addressed by different names. For example, Dan Saffer in his book Designing Gestural Interfaces calls these interfaces as Free-form gestural interfaces because they don’t require the user to touch or handle them directly [Designing Gestural Interfaces]. Doug Bowman in his article in the Encyclopedia of Human-Computer interaction calls it as 3D User Interfaces, because these interfaces are based on 3D user interaction and requires 3D spatial input.[Encyclopedia of HCI Bowman] [3D UI]

When I started to interact with the computer with gestures using the webcam, it was not natural at all. There was a lot of work to be done. We see the movies like Minority Report, Iron Man, Avatar, and Prometheus and the technology in these movies are our expectations for Natural User Interfaces to be. Today, it has been almost 10 years after the introduction of gesture based technologies, but still we are way far away from what we see in the movies like minority report. Why is it happening? Why don’t we have NUIs like the ones we see in the movies?  Why are these Natural User Interfaces not successful? Why do they fail in the wild? Are they only meant for labs?

This paper is important because in the near future this will be one of the way with which we will be interacting with computers and day to day technology. Now is the right time to find and attack the problems and create usable, useful and meaningful interfaces.

I will start the paper by talking about history of gesture based NUIs, and then I will give some examples of existing technologies and critique them. To put some light on our expectations, I will give some examples of the movies and interfaces in them, then I will discuss about the issues with the technology and issues with the application of user centered design principles to development of such interfaces. Later on, I will also talk about some issues with the user acceptance and future directions and expectations. Throughout the paper I will also share my experience with the gesture based technologies I worked with and what I learned from them, which is also one of the reasons why I am writing this paper.