Analytics for Mixed Reality

Behind every good user experience is great analytics

If you ever designed or developed client side applications or websites you’ve probably integrated with an analytics service to provide telemetry data to help make informed design choices and development decisions to improve user experience and business outcomes.
One of the key performance indicators is when you track steps or funnel operations as conversions to calculate a conversion rate for each session. You would want to know how the conversion rate can be improved upon and a good idea would be to watch out for the steps with a high bounce rate where users are dropping off. If the bounce rate is very high then there might even be a blocker or flaw in regards to the user. Either way we can understand the benefit for the collection and study of analytics which is essential for providing the insights that will help designers and developers craft better user experiences and advance product development.

Application Insights for Unity

If you’re a Unity developer or you develop in VR, AR, MR or XR you might have stuck the issue of gathering analytics onto the backlog but it should be one of the first items done so you can use it to help plan and prioritize the other features. To help you get started I’ve made an Application Insights for Unity sample so you can start logging telemetry in just a few minutes! All you have to do to add this to your existing Unity app or game is drop the Unity Application Insights script onto a Game Object, add your Application Insights Instrumentation key and you will be all set to record valuable user session telemetry automatically. After that all you have to do it wait around 5 mins for the telemetry to display in the Application Insights Usage section in Azure. You can also extend this in you own app or game to record any custom events or metrics you want to know about. But right out of the box (without any additional effort on your part) you will be able to visualize telemetry for users, sessions and user flow and their journey across the scenes of your Unity app or game. Here are just some of the visualizations already built-in to Application Insights in the Azure portal:

User Flows

Chart user flow across Unity scene changes and split by custom or interaction events.

Unity App Insights User Flow

Sessions

View users and events during sessions.

Unity App Insights User sessions

Funnels

Create funnels by creating step by step conditions to get conversion rates.

Unity App Insights Conversions

Retention

Review returning users over a period of time.

Unity App Insights Retention

Analytics for Mixed Reality interactions

In the Unity project there is also a MR sample to show how to setup Application Insights for recording custom interaction events and metrics in a scene. To use the sample please fork or clone the Unity Application Insights sample project, import the plugins and setup Application Insights in Azure portal if you haven’t already. Once you’ve got that you can get up and running on HoloLens straight from the Unity Editor:

  1. To view the Mixed Reality sample in HoloLens open the scene named “Scene-MR”. (Make sure you’ve pasted in your Instrumentation key into the Application Insights game object script)
  2. Connect to remote HoloLens device using Window > XR > Holographic Emulation window. Note: Requires the Holographic Remoting Player installed and open on HoloLens to get the Remote Machine IP address.
  3. Hit Play and you will start recording interaction telemetry with the holograms.

The Application Insights MR scripts will record taps, gaze time and object proximity – when users physically “visit” a hologram by moving closer to it.
You can also create your own custom dashboard templates using Ibex Dashboard (which is another project I helped with) and is designed for visualizing data from Application Insights using Kusto queries.

You can add the dashboard template for MR shown above to visualize the telemetry for MR custom events and metrics. Check out the readme on github for more info about installing custom Ibex Dashboard templates.

Querying Application Insights for data visualisation

Ibex dashboard is an open source web app for displaying telemetry data from Application Insights. It comes with a number of sample templates including analytics dashboards for Bots. If you’re developing a bot and you want to see how your bot is performing over time then you can select the Bot Instrumentation template which requires you to enter your Application Insights App Id and App Key. Also depending on your bot you will need to add Node.js instrumentation or C# instrumentation in order to enable logging to Application Insights. Then after a couple of minutes you will start to see the data come through! The dashboard can be completely customised using generic components including charts, tables, score cards and drill-down dialogs. These elements can be used to review how your bot performs over time, monitor usage stats, message sentiment, user retention and inspect user intents.

If you are new to Application Insights one of the useful features of the Ibex dashboard is the ability to inspect an element’s Application Insights query and the formatted JSON data side by side. 

This query can be copied and played back inside your Application Insights live code editor. This is a good way to learn how the Application Insights queries work as you can step through the query by commenting out various lines with double slashes ‘//’.

Writing Azure Log Analytics queries for Ibex dashboard

The Ibex dashboard schema is composed of meta data, data sources, filters, elements and dialogs. Each data source allows you to define a query and a ‘calculated’ javascript function to process the query’s results for display purposes. Before learning to write Application Insights queries I was used to writing javascript map / reduce functions to aggregate data and so it’s all too easy to rely on previous javascript knowledge to process the data from a basic query. But often this javascript ‘reduce’ aggregation logic can done in an Application Insights query with a lot less effort. So invest some time up front to learn the key Application Insights query concepts and it will pay off in the long run!

To help to illustrate this we can look at the Application Insights query for tracking a bot to human hand-off during a user’s conversation session. For this scenario we built a QnA bot with the hand-off module installed. If a customer asks the QnA bot a question and no answer was found in the knowledge base we trigger an automatic hand-off to human. We want to show the fastest, longest and average times for customer waiting for an human agent to respond in the dashboard.

We can start by writing a basic query in Application Insights to get all the transcripts from the ‘customEvents’ table and ‘project’ only the information we need.

But in this example we are not using Application Insights to aggregate the results so we end up with a lot of results to process. Given the query above the following code snippet is the amount of Javascript required.

The first ‘reduce’ block is required to group the transcripts per user Id. Then for every user we track the state change from waiting and talking to human agent and calculate the time difference in seconds. Where ‘state’ is an integer value that marks the current status of the conversion.

0 = Bot
1 = Waiting
2 = Human agent

But we can optimise the code by doing the aggregation within the Application Insights query by using the ‘summarize’ operator and ‘count’ function.

Notice how you can apply aggregations in multiple passes, in this case the ‘summarize’ operator and ‘count’ function is used to aggregate results twice in conjunction with multiple ‘where’ statements that are used to filter the results. Now the javascript ‘calculated’ function code can be greatly simplified:

The only thing we do is a ‘reduce’ function to convert the time format ‘hh:mm:ss’ returned from the Application Insights query into a number of seconds for the various calculations for displaying in a score card element.

The final Application Insights query is available in the hand-off to human dashboard template and is included with Ibex dashboard.

Further reading and resources: