Camera Frame Access Sample

danger

The Camera Frame Access feature is marked as experimental because optimizations on the plugin and Snapdragon Spaces Services side are breaking backwards compatibility from release to release at the moment.

This sample demonstrates how to access relevant camera information from a supported device, in this case the camera image and the intrinsics. Currently this feature is restricted to the RGB camera.

How the sample works

By default, when the sample is running, the UI shows the RGB image captured by the device and the intrinsic values related to it. The user can pause and resume frame capturing with the corresponding buttons.

If the device doesn't grant permission for Camera Access, the image, buttons, and camera information will be replaced by a warning message advising the user to enable it.

Camera Frame Access AR Manager

The sample uses the BP_CameraFrameAccessARManager blueprint asset (located under SnapdragonSpacesSamples Content > SnapdragonSpaces > Samples > CameraFrameAccess > Placeable) to start and stop the camera capture using the Toggle AR Capture, it has to be set to ON, if capture should start, and to OFF, if it should stop capturing. Additionally, Camera must be set as the capture type of that node. Any SpacesSessionConfig asset's configuration is supported with this feature. It can also be enabled or disabled using the Toggle Spaces Feature method.

Camera capture library

The Unreal AR interface provides us the functions to get information from the camera.

• Get AR Texture
  • Returns the camera frames. The Snapdragon Spaces plugin expands the information to provide the RGB frame as a 2D Texture. Cast to Spaces AR Camera Image Texture class to get Frame RGB Texture. Additionally, Camera Image must be set as the texture type of that node.
• Get Camera Intrinsics
  • Returns the Image Resolution, Focal Length and the Principal Point of the camera.

The Snapdragon Spaces plugin provides additional functions to help manage the camera capture from blueprints:

• Get Camera Extrinsic Pose
  • Returns a FTransform that represents the physical pose of the camera in world coordinates.
• Set Camera Frame Access State
  • To pause capturing with the camera, set Active to FALSE. To resume capturing with the camera, set Active to TRUE. Deprecated in version 0.15.0, use Pause AR Session or Pause Spaces Feature instead.
• Is Camera Frame Access Supported
  • Returns TRUE if the capture is available to use, and FALSE otherwise. It is recommended to check this result during Tick if the application uses capturing. Deprecated in version 0.15.0, use Is Feature Available instead.

• Set Camera Frame Resolution
  • Used to select the quality of the frame. This function takes in an ECameraFrameResolution enum, which allows the developer to choose between Full, Half, Quarter, and Eighth resolutions.

tip

The behaviour of the sample is implemented in the WBP_CameraFrameAccess blueprint asset (located under SnapdragonSpacesSamples Content > SnapdragonSpaces > Samples > CameraFrameAccess > UI).

Advanced

This section describes how to access the raw YUV camera frame data without having to extract it from the generated texture. It also explains how to obtain extra data that is not accessible in Blueprints. All the data types and structs involved in this process are included, including the available functions which are part of USpacesRuntimeBlueprintLibrary and included in SpacesRuntimeBlueprintLibrary.h.

tip

These data and functions are only accessible via C++.

Data Types

The ESpacesPlaneCameraFrameType enum describes the type of plane of a frame.

UENUM()
enum class ESpacesPlaneCameraFrameType : uint8
{
    Y = 0,
    U = 1,
    V = 2,
    UV = 3
};

The ESpacesDistortionCameraFrameModel enum describes the different lens distortion models used for camera calibration.

UENUM()
enum class ESpacesDistortionCameraFrameModel : uint8
{
    Linear = 0,
    Radial_2 = 1,
    Radial_3 = 2,
    Radial_6 = 3,
    FishEye_1 = 4,
    FishEye_4 = 5
};

The ESpacesCameraFrameFormat enum describes the different formats of camera frames.

UENUM()
enum class ESpacesCameraFrameFormat : uint8
{
    Unknown = 0,
    Yuv420_NV12 = 1,
    Yuv420_NV21 = 2,
    Mjpeg = 3,
    Size = 4,
};

The FFrameDataOffset structure describes the offset to the sensor image data in the frame buffer data.

USTRUCT()
struct FFrameDataOffset
{
    GENERATED_BODY()
    int32 X;
    int32 Y;
};

The FSpacesPlaneCameraFrameData structure describes a plane in a frame buffer.

USTRUCT()
struct FSpacesPlaneCameraFrameData
{
    GENERATED_BODY()
    uint32 PlaneOffset;
    uint32 PlaneStride;
    ESpacesPlaneCameraFrameType PlaneType;
};

• PlaneOffset contains the offset to the beginning of the plane data from the beginning of the buffer.
• PlaneStride indicates the byte distance from one row to the next row.
• PlaneType describes the type of the frame data.

The FSpacesSensorCameraFrameData structure contains expanded camera intrinsics data.

USTRUCT()
struct FSpacesSensorCameraFrameData
{
    GENERATED_BODY()
    FARCameraIntrinsics SensorCameraIntrinsics;
    FFrameDataOffset SensorImageOffset;
    TArray<float> SensorRadialDistortion;
    TArray<float> SensorTangentialDistortion;
    ESpacesDistortionCameraFrameModel DistortionCameraFrameModel;
};

• SensorCameraIntrinsics contains the image resolution, principal point and focal length of the camera.
• SensorImageOffset is the offset to the sensor image data in the frame buffer data.
• SensorRadialDistortion array of float describing the radial distortion coefficients.
• SensorTangentialDistortion array of float describing the tangential distortion coefficients.
• DistortionCameraFrameModel describes the lens distortion model used for the camera calibration.

The FSpacesCameraFrameData structure contains the frame data and camera data.

USTRUCT()
struct FSpacesCameraFrameData
{
    GENERATED_BODY()
    uint32 BufferSize;
    uint8* Buffer;
    ESpacesCameraFrameFormat FrameFormat;
    TArray<FSpacesPlaneCameraFrameData> Planes;
    FSpacesSensorCameraFrameData SensorData;
};

• BufferSize is the size of the buffer that contains the data.
• Buffer is a pointer to the frame data.
• FrameFormat is the format of the camera frame.
• Planes is an array that contains the frame planes.
• SensorData contains the expanded intrinsics data of the camera that captured the frame.

Functions

• static FSpacesCameraFrameData GetCameraYUVFrameData(): Accesses and returns the latest camera frame data in FSpacesCameraFrameData format.
• static bool ReleaseCameraFrameData(): Releases the previously accessed frame. It has to be used after using the previous camera frame data in order to access another frame.