# Hand Tracking Sample

This sample demonstrates how to track the user's hands and visualize their joints. Please refer to the best practices section, to learn more about how to use hand tracking. In order to use this feature it has to be enabled in the OpenXR plugin settings located under Project Settings > XR Plug-in Management > OpenXR.

TIP

To leverage the full Hand Tracking capabilites of the Spaces Services, refer to the Extended Hand Tracking documentation.

WARNING

The Spaces Services application's camera permissions have to be enabled as described here for hand tracking to work properly.

# How the sample works

The Snapdragon Spaces Unity package provides custom hand tracking subsystems and managers since there are none currently provided by Unity. The provided subsystem enables the developer to work in a known AR Foundation-like environment.

There is a manager component attached to the AR Session Origin similar to all the other sample scenes.

Spaces Hand Manager

A UI panel in the scene will show the current gestures detected per hand. Next to the gesture type, the gesture ratio and the flip ratio will also be shown. The gesture ratio is indicating, how much the gesture is applied (in percent) and the flip ratio is indicating, if the hand gesture is detected from the back (-1), from the front (1) or in between.

Hand gestures UI

Furthermore, a mirror in the scene is used to reflect the player position, the head rotation as well as the hand joints. The player object is rendered with the StencilEffectShader, and the mirror planes of the mirror object with the MirrorMaskShader. This creates a portal-like illusion where the player object is only visible through the masking planes.

Mirror Example

# Spaces Hand Manager

The Spaces Hand Manager component is of type ARTrackableManager and is programmed in a way that is similar to all the other managers from AR Foundation - by providing a callback function to retrieve changes in form of added, updated and removed items


public void Start() {
    spacesHandManager.handsChanged += OnHandsChanged;
}

...

private void OnHandsChanged(SpacesHandsChangedEventArgs args) {
    foreach (var hand in args.added) {
        ...
    }

    foreach (var hand in args.updated) {
        ...
    }

    foreach (var hand in args.removed) {
        ...
    }
}

It also provides an inspector field to define the default prefab that should be spawned upon detecting hands. The Default Spaces Hand prefab from the samples consists of two additional components, as seen in the image below. They are described in detail in the following sections.

Default Spaces Hand Prefab

# Spaces Hand component

This component is the common interface to get all hand related data. It is of type ARTrackable and therefore has common properties like a TrackableID, TrackingState and a Pose that will be the defined by the tracked hand's wrist joint.

It additionally provides three more properties:

  • IsLeft is a boolean returning true if the tracked hand is the left one. Otherwise, the returned value is false.
  • Joints is an array of Qualcomm.Snapdragon.Spaces.SpacesHand.Joint types. This type has the following properties:
    • Pose is a Unity pose type that returns the pose of the hand joint.
    • Type is of type Qualcomm.Snapdragon.Spaces.SpacesHand.JointType and returns an enum value, which hand joint is addressed.
  • Gesture is of type Qualcomm.Snapdragon.Spaces.SpacesHand.Gesture and has the following properties:
    • Type is of type Qualcomm.Snapdragon.Spaces.SpacesHand.GestureType and returns an enum value, which gesture was detected for the hand.
    • GestureRatio is a float value between 0 and 1, indicating how much the gesture is applied.
    • FlipRatio is a float value between -1 and 1, indicating if the hand gesture is detected from the back (-1), from the front (1) or in between.

For more information about gestures, please refer to the interaction gestures documentation.

namespace Qualcomm.Snapdragon.Spaces.SpacesHand
{
    public enum JointType
    {
        PALM = 0,
        WRIST = 1,
        THUMB_METACARPAL = 2,
        THUMB_PROXIMAL = 3,
        THUMB_DISTAL = 4,
        THUMB_TIP = 5,
        INDEX_METACARPAL = 6,
        INDEX_PROXIMAL = 7,
        INDEX_INTERMEDIATE = 8,
        INDEX_DISTAL = 9,
        INDEX_TIP = 10,
        MIDDLE_METACARPAL = 11,
        MIDDLE_PROXIMAL = 12,
        MIDDLE_INTERMEDIATE = 13,
        MIDDLE_DISTAL = 14,
        MIDDLE_TIP = 15,
        RING_METACARPAL = 16,
        RING_PROXIMAL = 17,
        RING_INTERMEDIATE = 18,
        RING_DISTAL = 19,
        RING_TIP = 20,
        LITTLE_METACARPAL = 21,
        LITTLE_PROXIMAL = 22,
        LITTLE_INTERMEDIATE = 23,
        LITTLE_DISTAL = 24,
        LITTLE_TIP = 25
    }
}
namespace Qualcomm.Snapdragon.Spaces.SpacesHand
{
    public enum GestureType
    {
        UNKNOWN = -1,
        OPEN_HAND = 0,
        //FLIP = 1,
        GRAB = 2,
        //UP = 3,
        //DOWN = 4,
        //SWIPE = 5,
        //SWIPE_OUT = 6,
        PINCH = 7,
        POINT = 8,
        VICTORY = 9,
        //CALL = 10,
        METAL = 11
    }
}

# Spaces Hand Joint Visualizer component

This component provides some properties to change the appearance of the joint visualization, such as:

  • JointMesh is the mesh that should be instantiated for every joint.
  • JointMaterial is the material that should be applied on the mesh.
  • JointMeshScale is a float value between 0.005 and 0.05 and defines the scaling that should be applied to the mesh.
  • UseNormalizedColors is a boolean value. If set to true, the _Color property of the applied material's shader, will be colored by the component.

In the samples, a simple sphere mesh included in the UnityEngine is set as the JointMesh as well as the Default-Material as the JointMaterial.