# Interaction Gestures

# Main interaction gestures

# Pinch


The pinch gesture is performed by touching the tip of the thumb to the tip of the index finger while extending the remaining fingers. The pinch is best recognized if the hand is viewed from the side.

The pinch is used for selection when the fingers touch, or for manipulation by holding the gesture and moving.

Pinch gesture is an efficient immersive gesture for validation, because it leads to auto-haptic feedback.

To release the selection, the required gesture is Open-Hand.

# Grab


The grab gesture is performed by placing the hand in front of the camera and closing the fist.

This gesture is used to grab and manipulate proximal voluminous objects.

Grab gesture also allows to have an auto-haptic feedback.

To release the grabbing, the required gesture is Open-Hand.

# Point


The point gesture is performed by extending the index finger while keeping the rest of the fingers closed.

This gesture is usually used when interacting with a GUI (e.g. pressing a button) in a proximal interaction.

# Open-Hand


The Open-Hand gesture is a neutral position. It is performed by extending the hand with the fingers opened and the palm away from the camera.

Open-Hand is generally used to attach the ray cast in order to interact with distal element.

This gesture is used to display raycast or as a release gesture.

# Hand representation

# Augmented Reality

In an AR context, it is not recommended to display a virtual hand on top of the real one. It is recommended to focus on feedback on the virtual elements, rather than on the hand avatar.

# Virtual Reality

3D models leveraging inverse kinematics are best used for VR applications. A 3D representation of the hand is overlaid over the hand in digital space, creating a more immersive experience for users. It is important to adapt the hand’s representation according to the context of the demo.

In addition, the 3D hand must return visual effects to warn the user that he/she is interacting.

Here are 2 examples of 3D hand avatar:

Alpha hand

Ghost Open-Hand Ghost Pinch

Harlequin hand

Arlequin Open-Hand Arlequin Pinch

# Feedback, cues and affordances

Because hands don’t provide tactile feedback the way other input devices do, it’s essential to compensate for it, through visual and audio feedback when users interact with 3D objects. It’s important to design distinct sounds and visual alterations that confirm when a user interacts with a component.

To improve the user experience, consider including the equivalent of real world feedback into the interaction design. This will help confirm, via a visual or audio cue, that an interaction with an object or a gesture is being, or has been, successfully executed.

Generally, the main states of a virtual element are the following:

State Visual Feedback Audio Feedback
Idle None None
Hover Yes Yes
Selected Yes Yes

# Visual feedback


The behavior of an object can change, or the object can be highlighted when successfully engaged. The object could also change shape or size in reaction to an interaction or gesture.

Button Visual Feedback

Object states from left to right: idle, hover, selected.


Depending on the context, it is recommended to give visual feedback on the 3D hand avatar, in addition to the interactive objects.

Generally, the hand goes through three states: no interaction, collision with the interactive element, interacting. Here is a representation of the type of effect the 3D hand can have:

Hand Visual Feedback

Hand feedback: idle, hover, selected


To specify the element one wants to interact with, especially in a distal interaction context, it is important to return different types of effects on the pointer. Classically, in HMI interfaces, the pointers' aspect vary according to their interaction state. These behaviors are therefore quite intuitive for the user. It is crucial to think about the different types of behaviors that the ray cast and its reticle can have during the design phase.

Raycast Visual Feedback

Reticle feedback: idle, hover, selected

# Audio feedback

It is preferable that audio feedback be implemented on virtual objects only because an interaction sound is context dependent. If a user interacts with a 2D menu, the audio feedback will be different than if he/she interacts in a video game.

Audio can also be used for environmental sounds, still context dependent.

# Interaction cue

Anim Pinch

The interaction cue is a hand animation which is triggered when the user’s hand is not detected. This component guides the user when he/she doesn't know how to interact with the virtual elements.

If the user has not interacted for x seconds (to be defined when designing the demo), the component is animated in a loop until the system detects an interaction.

# Cheat Sheet

Gestures and feedback best adapted to interaction states:

Proximal Distal Feedback
Target Collision Open-Hand (raycast) Visual, Audio
Select Pinch, Grab Pinch Visual, Audio
Manipulate Pinch, Grab Pinch Visual, depending on context
UI Target Collision Open-Hand (raycast) Visual, Audio
UI Select Pinch,Point,Open-Hand Pinch Visual, Audio
UI Manipulate Pinch, Point, Open-Hand Pinch Visual, Audio