Camera Frame Access Sample
The Camera Frame Access feature is marked as experimental because optimizations on the plugin and Snapdragon Spaces Services side are breaking backwards compatibility from release to release at the moment.
This sample demonstrates how to access relevant camera information from a supported device, in this case the camera image and the intrinsics. Currently this feature is restricted to the RGB camera.
How the sample works
By default, when the sample is running, the UI shows the RGB image captured by the device and the intrinsic values related to it. The user can pause and resume frame capturing with the corresponding buttons.
If the device doesn't grant permission for Camera Access, the image, buttons, and camera information will be replaced by a warning message advising the user to enable it.
Camera Frame Access AR Manager
The sample uses the BP_CameraFrameAccessARManager
blueprint asset (located under SnapdragonSpacesSamples Content > SnapdragonSpaces > Samples > CameraFrameAccess > Placeable) to start and stop the camera capture using the Toggle AR Capture
, it has to be set to ON, if capture should start, and to OFF, if it should stop capturing. Additionally, Camera must be set as the capture type of that node. Any SpacesSessionConfig
asset's configuration is supported with this feature.
It can also be enabled or disabled using the Toggle Spaces Feature
method.
Camera capture library
The Unreal AR interface provides us the functions to get information from the camera.
Get AR Texture
- Returns the camera frames. The Snapdragon Spaces plugin expands the information to provide the RGB frame as a 2D Texture. Cast to
Spaces AR Camera Image Texture
class to get Frame RGB Texture. Additionally, Camera Image must be set as the texture type of that node.
- Returns the camera frames. The Snapdragon Spaces plugin expands the information to provide the RGB frame as a 2D Texture. Cast to
Get Camera Intrinsics
- Returns the Image Resolution, Focal Length and the Principal Point of the camera.
The Snapdragon Spaces plugin provides additional functions to help manage the camera capture from blueprints:
Get Camera Extrinsic Pose
- Returns a
FTransform
that represents the physical pose of the camera in world coordinates.
- Returns a
Set Camera Frame Access State
- To pause capturing with the camera, set
Active
to FALSE. To resume capturing with the camera, setActive
to TRUE. Deprecated in version 0.15.0, use Pause AR Session or Pause Spaces Feature instead.
- To pause capturing with the camera, set
Is Camera Frame Access Supported
- Returns TRUE if the capture is available to use, and FALSE otherwise. It is recommended to check this result during Tick if the application uses capturing. Deprecated in version 0.15.0, use Is Feature Available instead.
Set Camera Frame Resolution
- Used to select the quality of the frame. This function takes in an
ECameraFrameResolution
enum, which allows the developer to choose betweenFull, Half, Quarter, and Eighth
resolutions.
- Used to select the quality of the frame. This function takes in an
The behaviour of the sample is implemented in the WBP_CameraFrameAccess
blueprint asset (located under SnapdragonSpacesSamples Content > SnapdragonSpaces > Samples > CameraFrameAccess > UI).
Advanced
This section describes how to access the raw YUV camera frame data without having to extract it from the generated texture. It also explains how to obtain extra data that is not accessible in Blueprints. All the data types and structs involved in this process are included, including the available functions which are part of USpacesRuntimeBlueprintLibrary
and included in SpacesRuntimeBlueprintLibrary.h
.
These data and functions are only accessible via C++.
Data Types
The ESpacesPlaneCameraFrameType
enum describes the type of plane of a frame.
UENUM()
enum class ESpacesPlaneCameraFrameType : uint8
{
Y = 0,
U = 1,
V = 2,
UV = 3
};
The ESpacesDistortionCameraFrameModel
enum describes the different lens distortion models used for camera calibration.
UENUM()
enum class ESpacesDistortionCameraFrameModel : uint8
{
Linear = 0,
Radial_2 = 1,
Radial_3 = 2,
Radial_6 = 3,
FishEye_1 = 4,
FishEye_4 = 5
};
The ESpacesCameraFrameFormat
enum describes the different formats of camera frames.
UENUM()
enum class ESpacesCameraFrameFormat : uint8
{
Unknown = 0,
Yuv420_NV12 = 1,
Yuv420_NV21 = 2,
Mjpeg = 3,
Size = 4,
};
The FFrameDataOffset
structure describes the offset to the sensor image data in the frame buffer data.
USTRUCT()
struct FFrameDataOffset
{
GENERATED_BODY()
int32 X;
int32 Y;
};
The FSpacesPlaneCameraFrameData
structure describes a plane in a frame buffer.
USTRUCT()
struct FSpacesPlaneCameraFrameData
{
GENERATED_BODY()
uint32 PlaneOffset;
uint32 PlaneStride;
ESpacesPlaneCameraFrameType PlaneType;
};
PlaneOffset
contains the offset to the beginning of the plane data from the beginning of the buffer.PlaneStride
indicates the byte distance from one row to the next row.PlaneType
describes the type of the frame data.
The FSpacesSensorCameraFrameData
structure contains expanded camera intrinsics data.
USTRUCT()
struct FSpacesSensorCameraFrameData
{
GENERATED_BODY()
FARCameraIntrinsics SensorCameraIntrinsics;
FFrameDataOffset SensorImageOffset;
TArray<float> SensorRadialDistortion;
TArray<float> SensorTangentialDistortion;
ESpacesDistortionCameraFrameModel DistortionCameraFrameModel;
};
SensorCameraIntrinsics
contains the image resolution, principal point and focal length of the camera.SensorImageOffset
is the offset to the sensor image data in the frame buffer data.SensorRadialDistortion
array offloat
describing the radial distortion coefficients.SensorTangentialDistortion
array offloat
describing the tangential distortion coefficients.DistortionCameraFrameModel
describes the lens distortion model used for the camera calibration.
The FSpacesCameraFrameData
structure contains the frame data and camera data.
USTRUCT()
struct FSpacesCameraFrameData
{
GENERATED_BODY()
uint32 BufferSize;
uint8* Buffer;
ESpacesCameraFrameFormat FrameFormat;
TArray<FSpacesPlaneCameraFrameData> Planes;
FSpacesSensorCameraFrameData SensorData;
};
BufferSize
is the size of the buffer that contains the data.Buffer
is a pointer to the frame data.FrameFormat
is the format of the camera frame.Planes
is an array that contains the frame planes.SensorData
contains the expanded intrinsics data of the camera that captured the frame.
Functions
static FSpacesCameraFrameData GetCameraYUVFrameData()
: Accesses and returns the latest camera frame data inFSpacesCameraFrameData
format.static bool ReleaseCameraFrameData()
: Releases the previously accessed frame. It has to be used after using the previous camera frame data in order to access another frame.