Create a simple persistent AR experience

This tutorial goes through how to use ConjureKit and the Manna module to create a persistent AR experience from scratch in Unity. AR assets are anchored in the environment thanks to posemesh domains which are essentially virtual real estate on top of physical spaces.

Before you begin

There are a couple of things you need to do before getting started with this project:

1. Set up your development environment by following the steps in the Quickstart.
2. Create and set up a domain in a physical location.

Unity & package versions

• Unity 2022.3.16f1
• Auki Labs ARFoundation Integration v0.6.42
• Auki Labs Ceres Module v0.0.4
• Auki Labs ConjureKit v0.6.82
• Auki Labs ConjureKit Ark module v0.6.14
• Auki Labs ConjureKit Manna module v0.6.98
• AR Foundation v5.1.4
• Apple ARKit XR Plugin v5.1.4 (for iOS)

Project setup

1. Configure the project for XR development. Go to Edit -> Project Settings -> XR Plug-In Management
   • -> iOS -> enable the Apple ARKit checkbox for iOS devices.
   • -> Android -> enable the ARCore checkbox for Android devices.
2. Configure build parameters. Go to Project Settings -> Player
   • -> iOS -> Other Settings -> Camera Usage Description and write a camera description, e.g. "Required by AR."
     • On Android, this permission is automatically added to manifest on build. Android will ask for Camera permissions when the app will first run.
   • -> Android -> Other Settings and have
     • Auto Graphics API disabled and OpenGLES3 at the top of the list
     • Scripting backend set to IL2CPP and both ARMv7 and ARM64 selected in Target Architectures
3. Go to Window -> Package Manager and install the packages listed above in Unity & package versions.

tip

We suggest to check which ARFoundation and ARKit/ARCore package versions are installed by Package Manager, as they might not be equal. This is not automatically handled when installing a "Preview" version and mismatch can happen.

note

ARFoundation versions below 4.2.6 do not support iOS 16.

4. In the sample scene, delete the Main Camera.
5. Create a new AR Session by selecting GameObject -> XR -> AR Session.
6. Create a new XR Origin by selecting GameObject -> XR -> XR Origin (Mobile AR).
7. In XR Origin, add two components: AR Raycast Manager and AR Plane Manager.

Basic UI

1. Create a Canvas by selecting GameObject -> UI -> Canvas.
2. Inside the canvas, create a Button element by selecting GameObject -> UI -> Legacy -> Button. Adjust button parameters and text.
3. Also inside the canvas, create an empty GameObject by selecting GameObject -> Create Empty. This will be used to hold the Calibration UI elements, so we'll call it CalibrateUI.
4. Inside CalibrateUI, create an Image element by selecting GameObject -> UI -> Image. Adjust the image parameters and duplicate it so that they cover the top and bottom of the mobile screen, as well as the previously created button.
5. Create a Text element by selecting GameObject -> UI -> Legacy -> Text. Add some brief instructions about calibrating using lighthouses.
6. Create a cube by selecting GameObject -> 3D Object -> Cube. Change the scale of the cube to 0.2 so it appears as a 20cm cube in AR. This should be set to inactive so we don't see it until after calibrating into the domain.

ConjureKit and domain implementation

1. Create a new MonoBehaviour script named PersistentARinDomain and attach it to an empty GameObject in the scene.
2. Import ConjureKit and Manna, as well as the following namespaces: UnityEngine.UI, UnityEngine.XR.ARFoundation, and UnityEngine.XR.ARSubsystems.

using Auki.ConjureKit;
using Auki.ConjureKit.Manna;
using Auki.Integration.ARFoundation;
using UnityEngine.UI;
using UnityEngine.XR.ARFoundation;
using UnityEngine.XR.ARSubsystems;

3. Declare a serialized field for the AR Camera:

[SerializeField] private Camera arCamera;

4. Use strings to store your App Key and App Secret from the Posemesh console. If you don't have these yet, please go back to the Quickstart.

private const string AppKey = "YOUR_APP_KEY";
private const string AppSecret = "YOUR_APP_SECRET";

caution

Never share your app secret with anyone.

5. Create private IConjureKit and Manna variables.

private IConjureKit _conjureKit;
private Manna _manna;

6. In the Start() method, initialize ConjureKit and Manna with the app key and secret:

private void Start()
{
    _conjureKit = new ConjureKit(
        arCamera.transform,
        AppKey,
        AppSecret);
    _manna = new Manna(_conjureKit);
    
    var textureProviderComp = CameraFrameProvider.GetOrCreateComponent();
    textureProviderComp.OnNewFrameReady += frame => _manna.ProcessVideoFrameTexture(frame.Texture, frame.ARProjectionMatrix, frame.ARWorldToCameraMatrix);
    _manna.OnLighthouseTracked += OnLighthouseTracked;
    
    _conjureKit.Connect();
}

7. Declare more serialized fields for the cube and the calibration UI:

[SerializeField] private GameObject cube;
[SerializeField] private GameObject calibrateUI;

8. Create a private bool variable to store the state of calibration into a posemesh domain:

private bool _calibrated = false;

9. Define the OnLighthouseTracked() method used in Start() to handle the lighthouse tracking event. This method will be called when a lighthouse QR code is tracked by the camera. It will calibrate into the posemesh domain and hide the calibration UI, showing the cube marker instead.

private void OnLighthouseTracked(Lighthouse lighthouse, Pose qrPose, bool isCalibrationGood)
{
    // If the QR detection was good enough and the QR code is static (generated from the posemesh console),
    // hide the calibration view and show the cube marker 
    if (isCalibrationGood && lighthouse.Type == Lighthouse.LighthouseType.Static)
    {
        if(!_calibrated)
        {
            _calibrated = true;
            calibrateUI.SetActive(false);
            cube.SetActive(true);
        }
    }
}

Object spawning

1. Create serialized fields for the Raycast Manager and Create Cube button:

[SerializeField] private ARRaycastManager raycastManager;
[SerializeField] private Button createCubeButton;

2. Declare a list to store AR raycast hits:

private List<ARRaycastHit> _arRaycastHits = new List<ARRaycastHit>();

3. In the Update() method, raycast from the center of the screen to an AR plane and place the cube marker where the raycast hits a plane.

private void Update()
{
    // Make a raycast from the center of the screen to an AR plane (floor, wall, or any other surface detected by ARFoundation)
    var ray = arCamera.ViewportPointToRay(Vector3.one * 0.5f);
    if (raycastManager.Raycast(ray, _arRaycastHits, TrackableType.PlaneWithinPolygon))
    {
        // Place the cube where the raycast hits a plane. Move it half the cube size along the hit normal (up if on the ground, forward if on the wall)
        cube.transform.position = _arRaycastHits[0].pose.position + _arRaycastHits[0].pose.up * cube.transform.localScale.x / 2f;
        // Rotate the cube only around y axis to always face the camera
        cube.transform.rotation = Quaternion.Euler(Vector3.Scale(arCamera.transform.rotation.eulerAngles, Vector3.up));
    }
}

4. Define a PlaceCube() method to be called when the Create Cube button is clicked. This method will instantiate a cube with the specified position, rotation, and color parameters.

private void PlaceCube(Vector3 position, Quaternion rotation, Color color)
{
    var placedCube = Instantiate(cube, position, rotation);
    placedCube.GetComponent<Renderer>().material.color = color;
    placedCube.gameObject.SetActive(true);
}

5. Now create the method OnCubeButtonClick() that gets a random color and places the cube where the cube marker is.

private void OnCubeButtonClick()
{
    var color = Random.ColorHSV();
    // Place the cube where the cube marker is 
    PlaceCube(cube.transform.position, cube.transform.rotation, color);
}

6. Import the Random function used above:

using Random = UnityEngine.Random;

7. In the Start() method, attach an event listener to createCubeButton that triggers the OnCubeButtonClick() method when the button is clicked:

private void Start()
{
    ...
    _manna.OnLighthouseTracked += OnLighthouseTracked;
    
    createCubeButton.onClick.AddListener(OnCubeButtonClick);
    
    _conjureKit.Connect();
}

Data persistence

In this sample project we'll save data about the spawned cubes in the device's local storage in JSON format. Because Unity-specific data types like Vector3, Quaternion, and Color are not inherently serializable into JSON format, we'll introduce the custom serializable classes SerializableVector3, SerializableQuaternion, and SerializableColor, to be used when converting to and from JSON format.

1. Create a new C# script named SaveData.cs:

using System;
using System.Collections.Generic;
using UnityEngine;

[Serializable]
public class SaveData
{
    public List<CubeData> cubes = new List<CubeData>();
}

// Because Unity's Vector3, Quaternion and Color structs are not marked as [Serializable] they can't be serialized into JSON.
// For that we create serializable versions of each one. There can be other approaches depending on how you serialize/deserialize the data.
[Serializable]
public class CubeData
{
    public SerializableVector3 position;
    public SerializableQuaternion rotation;
    public SerializableColor color;

    public CubeData() {}

    public CubeData(Vector3 position, Quaternion rotation, Color color)
    {
        this.position = new SerializableVector3(position);
        this.rotation = new SerializableQuaternion(rotation);
        this.color = new SerializableColor(color);
    }
}

[Serializable]
public class SerializableVector3
{
    public float x, y, z;
    
    public SerializableVector3() {}

    public SerializableVector3(Vector3 sourceVector)
    {
        x = sourceVector.x;
        y = sourceVector.y;
        z = sourceVector.z;
    }

    public Vector3 ToVector3() => new Vector3(x, y, z);
}

[Serializable]
public class SerializableQuaternion
{
    public float x, y, z, w;
    
    public SerializableQuaternion() {}

    public SerializableQuaternion(Quaternion sourceQuaternion)
    {
        x = sourceQuaternion.x;
        y = sourceQuaternion.y;
        z = sourceQuaternion.z;
        w = sourceQuaternion.w;
    }
    
    public Quaternion ToQuaternion() => new Quaternion(x, y, z, w);
}

[Serializable]
public class SerializableColor
{
    public float r, g, b, a;
    
    public SerializableColor() {}

    public SerializableColor(Color sourceColor)
    {
        r = sourceColor.r;
        g = sourceColor.g;
        b = sourceColor.b;
        a = sourceColor.a;
    }
    
    public Color ToColor() => new Color(r, g, b, a);
}

2. Back in PersistentARinDomain.cs, create a field for the SaveData object:

private SaveData _saveData = new SaveData();

3. Define a SaveLocally() method to serialize _saveData to JSON save it to the device's local storage using PlayerPrefs:

private void SaveLocally()
{
    var json = JsonUtility.ToJson(_saveData);
    PlayerPrefs.SetString("_saveData", json);
    PlayerPrefs.Save();
}

4. Define another method LoadLocally() to load the JSON data from the device's local storage, deserialize it into _saveData, and place the cubes in the scene:

private void LoadLocally()
{
    if(!PlayerPrefs.HasKey("_saveData"))
        return;
    
    var json = PlayerPrefs.GetString("_saveData");
    _saveData = JsonUtility.FromJson<SaveData>(json);

    foreach (var savedCube in _saveData.cubes)
    {
        PlaceCube(savedCube.position.ToVector3(), savedCube.rotation.ToQuaternion(), savedCube.color.ToColor());
    }
}

5. Save cube data when a new cube is placed by the OnCubeButtonClick() method:

private void OnCubeButtonClick()
{
    ...
    PlaceCube(cube.transform.position, cube.transform.rotation, color);
    // Save the position and rotation information locally
    _saveData.cubes.Add(new CubeData(cube.transform.position, cube.transform.rotation, color));
    SaveLocally();
}

6. Finally, call LoadLocally() in the OnLighthouseTracked() method after the user calibrates into a domain:

private void OnLighthouseTracked(Lighthouse lighthouse, Pose qrPose, bool isCalibrationGood)
{
    ...
        if(!_calibrated)
        {
            _calibrated = true;
            calibrateUI.SetActive(false);
            cube.SetActive(true);
            LoadLocally();
        }
    }
}

Assign references in Unity

The last step is to assign the references to the AR Camera, Cube, Calibrate UI, Raycast Manager, and Create Cube button in the Unity Editor by dragging and dropping the GameObjects into the script's corresponding serialized fields.

Now when you build and run the project, you should be able to calibrate into a domain, place cubes in it, and have them persist across sessions.

Complete code

PersistentARinDomain.cs:

using System.Collections;
using System.Collections.Generic;
using Auki.ConjureKit;
using Auki.ConjureKit.Manna;
using Auki.Integration.ARFoundation;
using UnityEngine;
using UnityEngine.UI;
using UnityEngine.XR.ARFoundation;
using UnityEngine.XR.ARSubsystems;
using Random = UnityEngine.Random;

public class PersistentARinDomain : MonoBehaviour
{
    [SerializeField] private Camera arCamera;
    [SerializeField] private GameObject cube;
    [SerializeField] private GameObject calibrateUI;
    [SerializeField] private ARRaycastManager raycastManager;
    [SerializeField] private Button createCubeButton;

    private const string AppKey = "YOUR_APP_KEY";
    private const string AppSecret = "YOUR_APP_SECRET";

    private IConjureKit _conjureKit;
    private Manna _manna;
    private bool _calibrated = false;
    private List<ARRaycastHit> _arRaycastHits = new List<ARRaycastHit>();
    private SaveData _saveData = new SaveData();
    
    private void Start()
    {
        _conjureKit = new ConjureKit(
            arCamera.transform,
            AppKey,
            AppSecret);
        _manna = new Manna(_conjureKit);
    
        var textureProviderComp = CameraFrameProvider.GetOrCreateComponent();
        textureProviderComp.OnNewFrameReady += frame => _manna.ProcessVideoFrameTexture(frame.Texture, frame.ARProjectionMatrix, frame.ARWorldToCameraMatrix);
        _manna.OnLighthouseTracked += OnLighthouseTracked;
        
        createCubeButton.onClick.AddListener(OnCubeButtonClick);
    
        _conjureKit.Connect();
    }
    
    private void OnLighthouseTracked(Lighthouse lighthouse, Pose qrPose, bool isCalibrationGood)
    {
        // If the QR detection was good enough and the QR code is static (generated from the posemesh console),
        // hide the calibration view and show the cube marker 
        if (isCalibrationGood && lighthouse.Type == Lighthouse.LighthouseType.Static)
        {
            if(!_calibrated)
            {
                _calibrated = true;
                calibrateUI.SetActive(false);
                cube.SetActive(true);
                LoadLocally();
            }
        }
    }

    private void Update()
    {
        // Make a raycast from the center of the screen to an AR plane (floor, wall, or any other surface detected by ARFoundation)
        var ray = arCamera.ViewportPointToRay(Vector3.one * 0.5f);
        if (raycastManager.Raycast(ray, _arRaycastHits, TrackableType.PlaneWithinPolygon))
        {
            // Place the cube where the raycast hits a plane. Move it half the cube size along the hit normal (up if on the ground, forward if on the wall)
            cube.transform.position = _arRaycastHits[0].pose.position + _arRaycastHits[0].pose.up * cube.transform.localScale.x / 2f;
            // Rotate the cube only around y axis to always face the camera
            cube.transform.rotation = Quaternion.Euler(Vector3.Scale(arCamera.transform.rotation.eulerAngles, Vector3.up));
        }
    }
    
    private void PlaceCube(Vector3 position, Quaternion rotation, Color color)
    {
        var placedCube = Instantiate(cube, position, rotation);
        placedCube.GetComponent<Renderer>().material.color = color;
        placedCube.gameObject.SetActive(true);
    }
    
    private void OnCubeButtonClick()
    {
        var color = Random.ColorHSV();
        // Place the cube where the cube marker is 
        PlaceCube(cube.transform.position, cube.transform.rotation, color);
        // Save the position and rotation information locally
        _saveData.cubes.Add(new CubeData(cube.transform.position, cube.transform.rotation, color));
        SaveLocally();
    }
    
    private void SaveLocally()
    {
        var json = JsonUtility.ToJson(_saveData);
        PlayerPrefs.SetString("_saveData", json);
        PlayerPrefs.Save();
    }
    
    private void LoadLocally()
    {
        if(!PlayerPrefs.HasKey("_saveData"))
            return;
    
        var json = PlayerPrefs.GetString("_saveData");
        _saveData = JsonUtility.FromJson<SaveData>(json);

        foreach (var savedCube in _saveData.cubes)
        {
            PlaceCube(savedCube.position.ToVector3(), savedCube.rotation.ToQuaternion(), savedCube.color.ToColor());
        }
    }
}

SaveData.cs: Data persistence

The full project can be found on GitHub.