Using Custom Devices in the Unity Input System to create gestures for the Nreal Light

The “new” Unity input system has been available for some time now, allowing developers to more easily assign input to multiple platforms and devices. While the system is ofcourse great, and build-in support for common devices such as gamepads, touchpads, and keyboard/mouse is already implemented within the engine, new devices such as the NREAL Light are not yet supported. To provide support for the developers using these unsupported devices, Unity does allow for adding custom devices to this system. In this blog post I will go over a simple Custom Device setup for the NREAL Light, explaining how I set it up.

I must admit that it got a bit carried away, and wrote more than intended. No blaim if you just download the scripts, dump them in Unity, find out they dont work, rant to your dev friends, and never look at it again.

NREAL SDK: 1.2.1

Unity: 2019.3.3f1

Video Explanation

This video contains the same information as you find on the rest of this page, but then comprised in a 25 minute visual walkthrough.

Links to the C# files

The bottom links will download the code I have written for the  custom device. 

NREALDevices.cs is the custom device for the Input System

Controls.cs contains the action settings for the Input System

Controls.inputactions is the object generated from the Controls.cs that allows you to edit the input actions

InputSystemTest.cs is a testscript for testing whether the input works as intended

Nreal Light

For those not familiar with the Nreal Light, they are mixed-reality glasses that you connect to your smartphone/pc via USB-C. I am personally interested in the device, and hope to get my hands on a developer kit, hence why I started trying out their SDK. As support is not yet part of the Unity Engine / Input System, I thought it was a great opportunity for trying out the option for adding Custom Devices. 

WARNING: I do NOT have an NREAL yet, and am therefore not able to test this code on the device itself. It runs well in editor, but I have no clue whether it works well on the device. If you have an NREAL device, feel free to download and use this code (just please let me know if it worked well).

For this project, I used the NREAL SDK 1.2.1

Input System Setup

Although the new Input System is not too complex to get into, I would advice watching a good example video on how to set it up. In case you have never used the new input system, I would recommand the following video by Dapper Dino to get up to speed: https://www.youtube.com/watch?v=Ikt5T-v2ZrM

If you want an empty & fully documented example on Custom Devices, then adding the Custom Device option via the Input Systems Package Manager page provides some great additional information.

 

I setup the Input Actions so they reflect the basic input provided by the Nreal Controller (when using a smartphone).

Control Schemes: Did not set anything over here
Action Maps: Player
Actions: Trigger, Home, App, Swipe, Stick, Pad

2 Sections down, the full settings of the Input System are explained.

Custom Device Settings (NREALDevice.cs)

Lets start with adding a new custom device to the project. I will go over all major elements of the script for the custom device, and provide some feedback on the items I changed from the unity provided example. The example provided by unity provides some great documentation on the exact function off all elements in general.

If you want to read along with the full script on the side, then you can download it over here:

Libraries

The custom device has a number of libraries included. Add the NRKernal to allow the script to use the NREAL sdk.InputSystemSettings

using System.Linq;
using System.Runtime.InteropServices;
using UnityEngine;
using UnityEngine.InputSystem;
using UnityEngine.InputSystem.Controls;
using UnityEngine.InputSystem.Layouts;
using UnityEngine.InputSystem.LowLevel;
using UnityEngine.InputSystem.Utilities;
using NRKernal;

#if UNITY_EDITOR
using UnityEditor;
#endif

NREALDeviceState struct

In this section you setup InputControls for the major parts of the NREAL sdk.

Trigger, App, and Home, are the 3 main buttons on the inputdevice, I decided to continue treating these like buttons

I setup the Swipe directions as Buttons as well. This was largely done so that no further checks would be required (such as vector2 comparisons). Instead, with the new input system, unity events can be triggered as if a button was pressed.

For Stick & Touchpad support, I decided to go with the in the Custom Device example provided Stick settings. These provided the exact options I needed.

public struct NREALDeviceState : IInputStateTypeInfo
{
    public FourCC format => new FourCC('I', 'N', 'P', 'T');

    //Inputcontrol for the basic NREAl buttons
    [InputControl(name = "triggerButton", layout = "Button", bit = 0, displayName = "Trigger Button")]
    [InputControl(name = "appButton", layout = "Button", bit = 1, displayName = "App Button")]
    [InputControl(name = "homeButton", layout = "Button", bit = 2, displayName = "Home Button")]

    //Inputcontrol for swipe directions
    [InputControl(name = "swipeUp", layout = "Button", bit = 3, displayName = "Swipe Up")]
    [InputControl(name = "swipeDown", layout = "Button", bit = 4, displayName = "Swipe Down")]
    [InputControl(name = "swipeLeft", layout = "Button", bit = 5, displayName = "Swipe Left")]
    [InputControl(name = "swipeRight", layout = "Button", bit = 6, displayName = "Swipe Right")]

    public ushort buttons;

    //Inputcontrol for stick movement (up/down/left/right) and touchpad (stick/x, stick/y)
    [InputControl(name = "stick", format = "VC2B", layout = "Stick", displayName = "Main Stick")]

    [InputControl(name = "stick/x", defaultState = 127, format = "BYTE",
        offset = 0,
        parameters = "normalize,normalizeMin=0,normalizeMax=1,normalizeZero=0.5")]
    public byte x;
    [InputControl(name = "stick/y", defaultState = 127, format = "BYTE",
        offset = 1,
        parameters = "normalize,normalizeMin=0,normalizeMax=1,normalizeZero=0.5")]

    [InputControl(name = "stick/up", parameters = "normalize,normalizeMin=0,normalizeMax=1,normalizeZero=0.5,clamp=2,clampMin=0,clampMax=1")]
    [InputControl(name = "stick/down", parameters = "normalize,normalizeMin=0,normalizeMax=1,normalizeZero=0.5,clamp=2,clampMin=-1,clampMax=0,invert")]
    [InputControl(name = "stick/left", parameters = "normalize,normalizeMin=0,normalizeMax=1,normalizeZero=0.5,clamp=2,clampMin=-1,clampMax=0,invert")]
    [InputControl(name = "stick/right", parameters = "normalize,normalizeMin=0,normalizeMax=1,normalizeZero=0.5,clamp=2,clampMin=0,clampMax=1")]
    public byte y;
}

NREALDevice class

Within this section, a couple of changes were needed, at first, dont forget to change the InputControlLayout to the one you setup in the first section, and change the classname to something sensible, I made it “NREALDevice”, because, it kinda made sense to me.

Then you need to make sure that the same Class is called in initialize.

Make sure that all of your buttons and sticks are correctly setup, this needs to happen as variables, as well as in the FinishSetup section.

#if UNITY_EDITOR
[InitializeOnLoad] // Call static class constructor in editor.
#endif
[InputControlLayout(stateType = typeof(NREALDeviceState))]
public class NREALDevice : InputDevice, IInputUpdateCallbackReceiver
{
#if UNITY_EDITOR
    static NREALDevice()
    {
        Initialize();
    }

#endif

    [RuntimeInitializeOnLoadMethod]
    private static void Initialize()
    {
        InputSystem.RegisterLayout<NREALDevice>(
            matches: new InputDeviceMatcher()
                .WithInterface("NREALInput"));
    }

    public ButtonControl triggerButton { get; private set; }
    public ButtonControl appButton { get; private set; }
    public ButtonControl homeButton { get; private set; }
    public ButtonControl swipeUp { get; private set; }
    public ButtonControl swipeDown { get; private set; }
    public ButtonControl swipeLeft { get; private set; }
    public ButtonControl swipeRight { get; private set; }

    public StickControl stick { get; private set; }

    protected override void FinishSetup()
    {
        base.FinishSetup();

        triggerButton = GetChildControl<ButtonControl>("triggerButton");
        appButton = GetChildControl<ButtonControl>("appButton");
        homeButton = GetChildControl<ButtonControl>("homeButton");

        swipeUp = GetChildControl<ButtonControl>("swipeUp");
        swipeDown = GetChildControl<ButtonControl>("swipeDown");
        swipeLeft = GetChildControl<ButtonControl>("swipeLeft");
        swipeRight = GetChildControl<ButtonControl>("swipeRight");

        stick = GetChildControl<StickControl>("stick");
    }

    public static NREALDevice current { get; private set; }
    public override void MakeCurrent()
    {
        base.MakeCurrent();
        current = this;
    }

    protected override void OnRemoved()
    {
        base.OnRemoved();
        if (current == this)
            current = null;
    }

Editor Menu Items

There  is a chunk that adds some MenuItems. This section allows you to assign the custom device inside the editor, something further explained later on.

This piece is essential for testing the Custom Device in editor. What must mostly happen over here, is that the interfaceName must be altered to the correct name, and that the correct class must be assigned. 


#if UNITY_EDITOR
    [MenuItem("Tools/Custom Device Sample/Create NREAL Device")]
    private static void CreateDevice()
    {
        InputSystem.AddDevice(new InputDeviceDescription
        {
            interfaceName = "NREALInput",
            product = "Sample Product"
        });
    }

    [MenuItem("Tools/Custom Device Sample/Remove NREAL Device")]
    private static void RemoveDevice()
    {
        var nrealDevice = InputSystem.devices.FirstOrDefault(x => x is NREALDevice);
        if (nrealDevice != null)
            InputSystem.RemoveDevice(nrealDevice);
    }

#endif

OnUpdate

During the update, all buttons and stick settings are provided.
The first section focuses on the Touchpad and Stick settings. Here the X, Y coordinates of NRInput.GetTouch are provided.
The second section assigns the APP and HOME button on Button Up.

After this, the calculations for the TRIGGER and Swipe are set.
StartTouch is called when ButtonDown on the TRIGGER is detected, this will store the starting position and time of the touch interaction.
EndTouch is called When the ButtonUp is detected on the trigger, this stores the ending position and time of the touch interaction.

The swipedirection is then calculated by CalculateSwipe, providing a vector 2 where the Up,Down,Left,Right is determined based on whether the x y directions are set to -1 or 1.
If this vector2 is {0,0}, then a regular tap / Button press for the TRIGGER is detected. This makes sure that it is never possible to have both a swipe and button press at the same time. 

   
        public void OnUpdate()
    {
        
        var state = new NREALDeviceState();
        
        //Set TouchPad X & Y Coordinates, 127 = 0
        state.x = 127;
        state.y = 127;
        if (NRInput.GetButton(ControllerButton.TRIGGER))
        {
            Vector2 touchPostion = NRInput.GetTouch();
            state.x = (byte)(state.x - (127 * touchPostion.x));
            state.y = (byte)(state.y - (127 * touchPostion.y));
        }

        //Pass input for App and Home button when button is released
        if (NRInput.GetButtonUp(ControllerButton.APP))
        {
            state.buttons |= 1 << 1;
        }
        if (NRInput.GetButtonUp(ControllerButton.HOME))
        {
            state.buttons |= 1 << 2;
        }

        //Store the start position of the trigger button / touchpad
        if (NRInput.GetButtonDown(ControllerButton.TRIGGER))
        {
            StartTouch();
        }

        //Store the start position of the trigger button / touchpad
        //Run the swipe script to detect whether the interaction was a tap, or a swipe
        //Set the touchpad buttons (swipe or touch interactions) to the correct value
        if (NRInput.GetButtonUp(ControllerButton.TRIGGER))
        {
            EndTouch();

            Vector2 swipeDirection = CalculateSwipe();
            if (swipeDirection.y == 1) state.buttons |= 1 << 3;
            else if (swipeDirection.y == -1) state.buttons |= 1 << 4;
            else if (swipeDirection.x == -1) state.buttons |= 1 << 5;
            else if (swipeDirection.x == 1) state.buttons |= 1 << 6;
            else state.buttons |= 1 << 0;
        }

        InputSystem.QueueStateEvent(this, state);
    }

CalculateSwipe

CalculateSwipe is the part that determines wheter the user swiped, or tapped on the Trigger Area.
First the duration and distance of the touchpad interaction are calculated, if these are within the specified boundaries (in my example < 1 second, and more that 0.3 on Touchpad Distance), then it is a swipe, if not, then it will automatically be considered a regular button press.

Great, so, we have a swipe, what now? First we calculate the Horizontal, and the Vertical swipe distance, this will be used to determine whether the swipe was horizontal or vertical, no fancy diagonal swipes for now.
After determining whether it was horizontal or vertical, the start and end position of that axis will decide the final swipe direction.

After calculating the swipe direction, the start and endPost are reset.

    //Values used for tracking touchpad interactions
    Vector2 startPos;
    Vector2 endPos;
    float startTime;
    float endTime;
    bool swiping = false;

    //Values for separating a swipe from a tap
    public float swipeDistance = 0.3f; //touchpad distance (touchpad ranges from -1 to 1 on the x and y axis)
    public float swipeDuration = 1; //seconds

    //Store starting postion of touchpad interaction
    void StartTouch()
    {
        swiping = true;
        startPos = NRInput.GetTouch();
        startTime = Time.time;
    }

    //Store end postion of touchpad interaction
    void EndTouch()
    {
        endPos = NRInput.GetTouch();
        endTime = Time.time;
        swiping = false;
    }

    //Calculate whether the interaction was a swipe or a tap. 
    //The interaction is a swipe if it is within the desired swipe time, and the minimum amount of distance is traveled over the touchpad
    //The interaction is a tap if it took longer than a swipe would, or if the distance traveled was lower than required

    Vector2 CalculateSwipe()
    {
        //Assure the user is no longer swiping
        if (swiping == false)
        {
            //Calculate touchpad interaction duration and distance
            float totalDuration = endTime - startTime;
            float totalDistance = Vector2.Distance(startPos, endPos);
            Vector2 swipeMultiplier = new Vector2 { x = 0, y = 0 };

            //If the touchpad interaction is within the desired time, and larger than the minimum required distance
            if ((totalDistance > swipeDistance) && (totalDuration < swipeDuration))
            {
                //Calculate horizontal and vertical swipe distance
                float hDistance = Mathf.Abs(startPos.x - endPos.x);
                float vDistance = Mathf.Abs(startPos.y - endPos.y);

                //If horizontal is larger than vertical, then determine whetherit was to the left, or the right
                if (hDistance > vDistance)
                {
                    if (startPos.x > endPos.x)
                        swipeMultiplier.x = -1;
                    else
                        swipeMultiplier.x = 1;
                }
                //If vertical is larger than horizontal, then determine whetherit was up or down
                else
                {
                    if (startPos.y > endPos.y)
                        swipeMultiplier.y = -1;
                    else
                        swipeMultiplier.y = 1;
                }
            }
            //Reset touchpad interaction values
            startPos = new Vector2 { x = 0, y = 0 };
            endPos = new Vector2 { x = 0, y = 0 };
            startTime = 0;
            endTime = 0;
            return swipeMultiplier;
        }
        else
            return new Vector2 { x = 0, y = 0 };
    }
}

And this was everything setup to create a custom device using the NREAL SDK, it was quite a read I imagine. Although it is handy to know what I changed, I can imagine right now you just want to copy and paste the file into your project, to do so, I have provided the full C# file over here:

Input System Settings (Controls.cs)

Now that the custom device is added, we can assign it to the earlier setup Input System.
I will go over the settings that I used for the individual pieces of input. 

Luckily, the settings for the input system are stored in .cs files, allowing you to download the file below, and paste the settings that I used (or, alter them yourself using the info below, whatever suits you best:

 

You can find the NREALDevice by going into the Input Actions -> Select the Binding you want to change -> Select Path in the Binding Properties -> Other -> NREAl Device

APP

Action Type: Button
Binding 1: App Button [NrealDevice]
Binding 2: Enter [Keyboard]

 

HOME

Action Type: Button
Binding 1: Home Button [NrealDevice]
Binding 2: Escape [Keyboard]

TRIGGER

Action Type: Button
Binding 1: Trigger Button [NrealDevice]
Binding 2: Space [Keyboard]

SWIPE

Action Type: Button

Binding 1: NREAL

Composite Type: 2D Vector
Mode: Digital Normalize

Up: Swipe Up [NREALDevice]
Down: Swipe Down [NREALDevice]
Left: Swipe Left [NREALDevice]
Right: Swipe Right [NREALDevice]

Binding 2: Keyboard
Composite Type: 2D Vector
Mode: Digital Normalize
Up: Up Arrow [Keyboard]
Down: Down Arrow [Keyboard]
Left: Left Arrow [Keyboard]
Right: Right Arrow [Keyboard]

STICK

Action Type: Value

Control Type: Vector2

Binding 1: NREAL

Composite Type: 2D Vector
Mode: Digital Normalize
Processors: Invert Vector 2 (Invert X = True, Invert Y = True)

Up: Main Stick/Up [NREALDevice]
Down: Main Stick/Down [NREALDevice]
Left: Main Stick/Left [NREALDevice]
Right: Main Stick/Right [NREALDevice]

Binding 2: Keyboard
Composite Type: 2D Vector
Mode: Analog
Up: Up Arrow [Keyboard]
Down: Down Arrow [Keyboard]
Left: Left Arrow [Keyboard]
Right: Right Arrow [Keyboard]

PAD

Action Type: Value
Control Type: Vector2

Binding 1: NREAL
Composite Type: 2D Vector
Mode: Analog
Down: Main Stick/Y [NREALDevice]
Left: Main Stick/X [NREALDevice]

Binding 2: Keyboard
Composite Type: 2D Vector
Mode: Digital Normalize
Up: Up Arrow [Keyboard]
Down: Down Arrow [Keyboard]
Left: Left Arrow [Keyboard]
Right: Right Arrow [Keyboard]

Adding Device to Input Debug

Although the controls are now fully setup, I have encountered the issue that custom devices are not by default added to the available device list. This can be fixed however (and the solution has already been setup in our Custom Device Script). 

Open the Input Debugger by going to: Window -> Analysis -> Input Debugger

This shows the Input Debug window, where you should see the connected devices added to the devices list. By default (for me at least) it shows the keyboard and mouse.

Add the custom NREAL Device by going to: Tools -> Custom Device Sample -> Create NREAL Device

Now the NREALDevice should be added to the list of Devices in the Input Debug. If you do not do this step, the  the NREALDevice will not be added to the list, and you are unable to test with the created inputscripts.

THIS MUST BE DONE EVERYTIME UNITY IS RESTARTED!!!!!

 

Testing (InputSystemTest.cs)

So lets test the input now. I provided a handy script (inputsystemtest.cs) to help you check whether the custom input is working.

I will go over the script, and explain what it does, as well as the information you are given when running this script. You can download the full script over here:

 

Library

Make sure you include the UnityEngine.InputSystem library to make use of the new Unity InputSystem.

using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using UnityEngine.InputSystem;

Enabling the Controls

To use the desired controls, you’ll need to enable and disable it on runtime. 

In the Awake section you can specify which functions are called when specific buttons are pressed.

public class InputSystemTest : MonoBehaviour
{
    private Controls controls = null;

    private void Awake()
    {
        controls = new Controls();
        controls.Player.Trigger.performed += ctx => TriggerTest();
        controls.Player.App.performed += ctx => AppTest();
        controls.Player.Home.performed += ctx => HomeTest();
        controls.Player.Swipe.performed += ctx => SwipeTest();
    }

    private void OnEnable()
    {
        controls.Enable();
    }
    private void OnDisable()
    {
        controls.Disable();
    }

Touchpad and Stick

Stick and Touchpad input are non-event based/ continuous functions. These can be called from Update, and provide a constant value.

Stick shows the positive or negative offset relative to the starting position of the touch input, this would function like a virtual joystick.

Pad shows the current position of the touchposition on the trigger pad, where 0,0 is the exact center, -1,-1 is the bottom-left position, and 1,1 is the top-left position.

    void Update()
    {
        //PadTest();
        //StickTest();
    }
    public void StickTest()
    {
        Vector2 stickInput = controls.Player.Stick.ReadValue();
        if (stickInput != new Vector2 { x = 0, y = 0 })
        {
            Vector3 stick = new Vector3
            {
                x = stickInput.x,
                z = stickInput.y
            }.normalized;

            Debug.Log(stick);
        }
    }
    public void PadTest()
    {
        Vector2 padInput = controls.Player.Pad.ReadValue();

        Debug.Log(padInput);
    }

Swipe

The swipetest acts like a button, however, the direction is still set as a vector 2.
The direction can be determined by checking the x,y input, where only one option is possible at any given time (only horizontal, or vertical, no diagonal input at the moment).

 

    void SwipeTest()
    {
        Vector2 swipeInput = controls.Player.Swipe.ReadValue();
        if (swipeInput != new Vector2 { x = 0, y = 0 })
        {
            if (swipeInput.x == 1) Debug.Log("Swipe Right");
            if (swipeInput.x == -1) Debug.Log("Swipe Left");
            if (swipeInput.y == 1) Debug.Log("Swipe Up"); ;
            if (swipeInput.y == -1) Debug.Log("Swipe Down");
        }
    }

Buttons

The following debug logs are shown when button input on the 3 main buttons are detected.

    public void TriggerTest()
    {
        Debug.Log("Trigger");
    }
    public void HomeTest()
    {
        Debug.Log("Home");
    }
    public void AppTest()
    {
        Debug.Log("App");
    }
}

These are some of the basic input options you can retrieve from the Input System, and the ones I am currently using in my own project.

There is no raycasting option in this example, this merely handles touch input.

Current Issues

So, everything is now working, hoooray, however, there are a couple of issues that are either part of Unity, the NREAL sdk, or that I should probably try to fix in the future. 

Enable Controls: The current way of adding the device to the Input Debug every time you open the project is less than ideal. This issue was reported by more people using Custom Controls.

NREAL Controller: The current sdk had the issue that the controller would regularly stop working, forcing me to restart the project.

RayCasting: This project has no included raycasting, as it only focuses on touch input, this means that the raycast must either be done via the NREAL default method, or that a raycast must be added on TRIGGER input. 

Stick: The stick as I currently set it up only works well when started at the center of the touchpad. I need to check what is wrong, so for now dont trust that one too much please 😀

Testing on NREAL: I have NOT been able to test on NREAL, only inside of the Editor on a Windows 10 Machine, this entire thing might just not run in the end, i’ll be sad, but i’ll get over it. Please dont be mad.

 

Conclusion

That’s it folks. 

All by all, I found the new Input System to be highly usable, and the option of adding custom devices could be incredibly usefull for cases such as this one.

I would love to say Like and Subscribe, but this is not Youtube, and I cannot promise to make more of these, soooo, thank you for reading this piece, and I hope it helped you out a bit.