Skip to main content

Interaction Gestures

Main interaction gestures

Pinch

Pinch

The pinch gesture is performed by touching the tip of the thumb to the tip of the index finger while extending the remaining fingers. The pinch is best recognized if the hand is viewed from the side.

The pinch is used for selection when the fingers touch, or for manipulation by holding the gesture and moving.

Pinch gesture is an efficient immersive gesture for validation, because it leads to auto-haptic feedback.

Grab

Grab

The grab gesture is performed by placing the hand in front of the camera and closing the fist.

This gesture is used to grab and manipulate proximal voluminous objects.

Grab gesture also allows to have an auto-haptic feedback.

Open-Hand

Open-Hand

The Open-Hand gesture is a neutral position. It is performed by extending the hand with the fingers opened and the palm away from the camera.

Open-Hand is generally used to attach the ray cast in order to interact with distal element.

This gesture is used to display raycast or as a release gesture.

Point (deprecated)

Point

As of Snapdragon Spaces version 0.12.1, the point gesture is deprecated. Only use this gesture when developing with Snapdragon Spaces versions 0.11.1 and below.

The point gesture is performed by extending the index finger while keeping the rest of the fingers closed.

This gesture is usually used when interacting with a GUI (e.g. pressing a button) in a proximal interaction.

Hand representation

Augmented Reality

In an AR context, it is not recommended to display a virtual hand on top of the real one. It is recommended to focus on feedback on the virtual elements, rather than on the hand avatar.

Virtual Reality

3D models leveraging inverse kinematics are best used for VR applications. A 3D representation of the hand is overlaid over the hand in digital space, creating a more immersive experience for users. It is important to adapt the hand’s representation according to the context of the demo.

In addition, the 3D hand must return visual effects to warn the user that he/she is interacting.

Here are 2 examples of 3D hand avatar:

Alpha hand

Ghost Open-HandGhost Pinch

Harlequin hand

Arlequin Open-HandArlequin Pinch

Feedback, cues and affordances

Because hands don’t provide tactile feedback the way other input devices do, it’s essential to compensate for it, through visual and audio feedback when users interact with 3D objects. It’s important to design distinct sounds and visual alterations that confirm when a user interacts with a component.

To improve the user experience, consider including the equivalent of real world feedback into the interaction design. This will help confirm, via a visual or audio cue, that an interaction with an object or a gesture is being, or has been, successfully executed.

Generally, the main states of a virtual element are the following:

StateVisual FeedbackAudio Feedback
IdleNoneNone
HoverYesYes
SelectedYesYes

Visual feedback

Object

The behavior of an object can change, or the object can be highlighted when successfully engaged. The object could also change shape or size in reaction to an interaction or gesture.

Button Visual Feedback

Object states from left to right: idle, hover, selected.

Hand

Depending on the context, it is recommended to give visual feedback on the 3D hand avatar, in addition to the interactive objects.

Generally, the hand goes through three states: no interaction, collision with the interactive element, interacting. Here is a representation of the type of effect the 3D hand can have:

Hand Visual Feedback

Hand feedback: idle, hover, selected

Reticle

To specify the element one wants to interact with, especially in a distal interaction context, it is important to return different types of effects on the pointer. Classically, in HMI interfaces, the pointers' aspect vary according to their interaction state. These behaviors are therefore quite intuitive for the user. It is crucial to think about the different types of behaviors that the ray cast and its reticle can have during the design phase.

Raycast Visual Feedback

Reticle feedback: idle, hover, selected

Audio feedback

It is preferable that audio feedback be implemented on virtual objects only because an interaction sound is context dependent. If a user interacts with a 2D menu, the audio feedback will be different than if he/she interacts in a video game.

Audio can also be used for environmental sounds, still context dependent.

Interaction cue

Anim Pinch

The interaction cue is a hand animation which is triggered when the user’s hand is not detected. This component guides the user when he/she doesn't know how to interact with the virtual elements.

If the user has not interacted for x seconds (to be defined when designing the demo), the component is animated in a loop until the system detects an interaction.

Cheat Sheet

Gestures and feedback best adapted to interaction states:

ProximalDistalFeedback
TargetCollisionOpen-Hand (raycast)Visual, Audio
SelectPinch, GrabPinchVisual, Audio
ManipulatePinch, GrabPinchVisual, depending on context
UI TargetCollisionOpen-Hand (raycast)Visual, Audio
UI SelectPinch,Point,Open-HandPinchVisual, Audio
UI ManipulatePinch, Point, Open-HandPinchVisual, Audio