We already have access to MR (aka. Magic Reality) components in Canvas Apps. Implementation is straight forward, but as expected they come with a limited set of features. The components are based on Babylon.js, and make for a quick and easy way to place and view a 3D model in an augmented reality context.

For our solution, we wanted the user to also be able to interact with the virtual objects in front of them, which is not OOB features, so by expressing our power user love, we decided to explore the possibilities around custom XR enabled PCF components.

Being ACDC craftsmen, knowing the potential issues of going too far down the wrong path, we decided to do some proof of concepts, creating custom PCF components with third party XR libraries, acting like proper thieving bastards on the way.

First off, we had a look at AR.js, which is built on ARToolkit, a relatively old library. This library could provide us with wide device support, which really didn’t have that much value, considering the component would be running inside the Power Apps mobile app. We would also be forced to use either image target or marker tracking, with no modern AR spatial tracking.

Looking closer at the OOB components, we tried to find a way to leverage the OOB Babylon.js logic, hopefully being able to hook into the React Native part of the implementation, which would give great benefits in terms of access to device specific XR features (ARCore for Android and ARKit for iOS). We did, however, decide to leave this path, and focus elsewhere.