Some of the most popular experiences of Augmented or Mixed Reality are currently in a gaming or an immersive 3D form. But what makes headsets like the HoloLens so different and special is the ability to still see into the physical world. It is this link that opens up so many new opportunities and future possibilities of spatial computing. An interesting scenario is using HoloLens to interact with IoT devices in the real world. Just by gazing at devices around a room imagine you could identify a specific IoT device to review its real-time telemetry and control it over the air!
To test out this scenario the first thing we need here is access to some IoT devices to identify. In our case and for sake of simplicity we use some simulated IoT devices using the Azure IoT Solution Accelerators Remote Monitoring sample which you can try out.
Once the Azure IoT Remote Monitoring solution is provisioned and ready you can select it to review a list of devices. This provides us with the list of Device Name Ids we will use to generate the VuForia VuMarks in the next steps.
The second thing we need is to be able to identify each IoT device in HoloLens. One approach we tried during a HoloLens hack was to use Vuforia VuMarks to identify each device. A VuMark template contains a particular type of encoded data; numeric, string or raw bytes. Initially I tried out the default numeric type VuMarks from the VuForia Samples to check everything was working before trying anything more complex. Bear in mind there will also be a number of physical and environmental factors including VuMark placement, size and lighting conditions in the area to test and consider.
Tip: I found it useful to test the VuMarks by saving all the generated images on my iPhone and testing them in the Unity Editor using the built-in web cam.
I used the VuForia VuMark Illustrator template to create a custom VuMark. In my case I wanted to a support a 32 character length string to contain a GUID so I created a string type VuMark with 280 data elements. To save time designing your own VuMark you can download my finished custom GUID VuMark SVG. If you want to create your own VuMark I’ve included a list of VuMark element requirements below so you can get an idea of how complex the design would need to be and compare how many elements are required for each data type:
|Id length||String elements required||Byte elements required|
|Maximum numeric Id||Numeric elements required|
For more info on designing VuMarks you can download the VuMark design guide or view design guide docs. I also found the video explaining the VuMark design process to be most helpful. NB: To design your own custom VuMarks you will need Adobe Illustrator to run the VuMark template scripts.
Illustrator / VuMark Scripts troubleshooting notes:
C:\Program Files\Adobe\Adobe Illustrator CC 2018\Presets\en_US\Scriptsdirectory.
Once you’ve designed your custom VuMark in Illustrator and it passes all the tests you will be ready to export your VuMark Template artwork. If you don’t have your own design ready you can download my GUID VuMark SVG artwork.
Note: If you’re starting a new design it’s preferable to avoid rotational symmetry in your VuMark’s border or contour otherwise you will have some additional work to do, as well as this the validation scripts don’t seem to provide a clear indication if this is completed correctly. You might also notice the Border and Clear Space width is only shown as “VERIFY” status - this check is left to the designer to manually check that the magenta overlay around the VuMark contour falls within the border and clear space boundary.
For my sample IoT devices I generated the following device Ids; “chiller-01.0”, “chiller-02.0”, “elevator-01.0”, “elevator-02.0”, “furnace-01.0”.
Tip: Save the generated VuMark images to iPhone / Android to test with. (I just saved the generated VuMark PNG images to my OneDrive images folder to sync onto my iPhone.)
To run the Mixed Reality IoT Monitoring Unity project you will also need to setup the Azure Function APIs to get the device data from the Azure IoT Remote Monitoring sample.
Add the following environment variables using your Azure Function app settings :
Using VuForia VuMarks we are able to identify an IoT device using a HoloLens. Then using the recognized device Id as a param we can poll an Azure Functions endpoint to return the device’s telemetry. The next steps in this scenario would be to add buttons to call methods listed in the device’s payload.