Hermione’s Beaded Bag

Canvas App and AI Builder
Hermione’s beaded bag can hold unlimited amounts of ingredients, spells and potions, as well as a myriad of other handy things. With the Hermiones Beaded Bag Canvas App the students can take a picture of an ingredient and it will automatically detect what kind of ingredient it is, identify it from the ingredient list, and add it to the student ingredient inventory.

3D Map of Hogwarts
Students at Hogwarts walk around the school to collect ingredients to make potions throughout the year. With a school this size it’s hard to keep track of where they have been and where ingredients pop up, but luckily for them, they can navigate virtually through the 3D model of the building straight from the app.

Potion Matching Agent
Once they have collected an ingredient, its added to their ingredient inventory in the Beaded Bag, and they can start making potions. To explore the craft of potion making they can enlist the help from the magical potion match making agent who can tell them what potions they can make with the ingredients they already have, but also what ingredients they are missing to make potions they need.

The agent help the students to manage their inventory for them, and once they have all the ingredients they need to make potion the agent adds that to their potion inventory. This agent is available to the students through the Hermione’s Beaded Bag app, but also through the Diagon Alley website. There are rumors that Dumbledore himself has even added it to his own personal teams chat.

Collect, get points, graduate and get a diploma
As the students evolve, collect ingredients and make potions, they get points based on the level of difficulty. These points are a crusial part of the points neede to level up in their Owls levels as well! This is a neat way for the professors and headmaster Dumbledore himself to keep track of the students progress. When students reach a certain level, a diploma is automatically created and will be signed by Dumbledore himself. Pure magic. The students will receive an SMS that they have graduated to a new level, and they will get an email with the diploma, signed by Dumbledore and their professor.

Tracking progress
The professors keep an eye on the students progress, inventory and levels all the time, and they build reports in top of the data which they use to report to Dumbledore. This way they can help students who struggle, sort and filter students based on the type of potions they find, and maybe intersect before the students who are up to no good do any harm.

Business Value

The real world impact of our solution might not be obvious at first glance, but we work a lot with non-profit and volunteer driven companies. We know that there is a constant awareness with these companies to lower the efforts on administrative tasks and optimize the impact of every donated amount.

One of the administrative tasks that take a lot of time and effort is matching volunteers with activities. This is because finding the right activity fit for the right volunteer rely on a myriad of aspects; Certificates, Certifications, Courses, Competencies, Languages they speak and interest they have. If we can create an AI driven solution that is smart enough to look at all of these aspects and perspectives and automatically match volunteers with activities, we can get an incredible upturn in volunteer activities.

This solution can also be used by search and rescue. The app allow you to scan objects and move through buildings, and also have a way to track and view where you have been and what areas of a space that has been searched. This could potentially be life saving in a critical situation.

Low Code

We have used an object detection model to scan ingredients and add them to the bag.

The model is custom-built and trained on images we have captured ourselves of various objects around our premises.

We have mapped the images to different labels so the model can recognize the objects. These objects are matched with our ingredients in Dataverse.

We have created a canvas app where you can navigate to collect ingredients. Here you have the ability to use your phone camera to take whatever picture you like of an object that is recognized as an ingredient. The ingredient will be listed in a table just below your image.

We use the code

Filter(ObjectDetector1.GroupedResults, ObjectCount = 1)

to show which ingredient the AI model recognized. Now you can add the scanned ingredient to our inventory by adding a button with the code

With({
    firstIngredient:First(Filter(ObjectDetector1.GroupedResults, ObjectCount = 1)),
    firstIngredientName:First(Filter(ObjectDetector1.GroupedResults, ObjectCount = 1)).TagName
    },
Patch(
     'Ingredient Inventories',
     Defaults('Ingredient Inventories'),
     {
        'Ingredient Inventory (pp_ingredientinventory)':firstIngredientName,
        Ingredient:LookUp(Ingredients,'Ingredient (pp_ingredient)'=firstIngredientName),
        Quantity:firstIngredient.ObjectCount
        }
)
)

In the inventory you can see all of your collected ingredients using the custom objective detection model.

Canvas app and AI builder for adding ingredients to inventory

Pro Code

3D geolocation mapping

By leveraging the web browser geolocation API inside a Canvas App PCF component, we were able to provide a 3D representation of the active users’ current location inside the venue.

We were able to find a simple volume model of the buildings on the map service Kommunekart 3D, but these data seem to be provided by Norkart, which is not freely available.

We decided to scrape the 3D model off of the site, by fetching all the resources that looked like binary 3D data. We found the data was in B3DM format and we found the buildings in one of these. We used Blender to clean up the model, by removing surrounding buildings and exporting it to glTF 3D file format, for use in a WebGL 3D context.

The representation of the 3D model, we decided to do with Three.js, which let us create an HTML canvas element inside the PCF component and using its WebGL context to render out the model in 3D. The canvas is continuously rendered using requestAnimationFrame under the hood, making it efficient in a browser context. The glTF model was loaded using a data URI, as a workaround for the web resource file format restrictions.

The coordinates from the user’s mobile device comes in as geographical coordinates, with longitude, latitude and altitude. The next step was to map these values relative to a known coordinate in the building, which we chose to be the main entrance. By using the main entrance geographical coordinates, we could then convert that to cartesian coordinates, with X, Y and Z, do the same to the realtime coordinates from the user, and subtract the origin, to get the offset in meters. The conversion from geographic to geocentric coordinates were done like so:

// eslint-disable-next-line @typescript-eslint/consistent-type-definitions
export type CartesianCoordinates = { x: number; y: number; z: number };

// eslint-disable-next-line @typescript-eslint/consistent-type-definitions
export type GeographicCoordinates = { lat: number; lon: number; alt: number };

// Conversion factor from degrees to radians
const DEG_TO_RAD = Math.PI / 180;

// Constants for WGS84 Ellipsoid
const WGS84_A = 6378137.0; // Semi-major axis in meters
const WGS84_E2 = 0.00669437999014; // Square of eccentricity

// Function to convert geographic coordinates (lat, lon, alt) to ECEF (x, y, z)
export function geographicToECEF(coords: GeographicCoordinates): { x: number; y: number; z: number } {
	// Convert degrees to radians
	const latRad = coords.lat * DEG_TO_RAD;
	const lonRad = coords.lon * DEG_TO_RAD;

	// Calculate the radius of curvature in the prime vertical
	const N = WGS84_A / Math.sqrt(1 - WGS84_E2 * Math.sin(latRad) * Math.sin(latRad));

	// ECEF coordinates
	const x = (N + coords.alt) * Math.cos(latRad) * Math.cos(lonRad);
	const y = (N + coords.alt) * Math.cos(latRad) * Math.sin(lonRad);
	const z = (N * (1 - WGS84_E2) + coords.alt) * Math.sin(latRad);

	return { x, y, z };
}

This gave us fairly good precision, but not without the expected inaccuracy caused by being indoors.

In our solution the current position is then represented by an icon moving around the 3D model based on the current GPS data from the device.

To connect this representation to realtime data from all the currently active users, we decided to set up an Azure SignalR Service, with an accompanying Azure Storage and Azure Function App for the backend. With this setup, we could use the @microsoft/azure package inside the PCF component, receiving connection, disconnection and location update message broadcast from all other users.

Canvas App XR

To further enhance the connection between the app and the real world, we also explored the possibilities of implementing XR as part of the user experience. Using augmented reality, we could let users interactively gather ingredients, but the OOB MR components in the Canvas App does provide the necessary interactivity, so we compared Babylon.js and AR.js (with Three.js/A-FRAME) to see if any of these third party libraries would be suitable for custom PCF components. Based on early POC’s we concluded that without access to the device’s low level XR features (ARCore for Android and ARKit for iOS), we decided these

Magic Matrix

Every time you add an ingredient to your inventory you will automatically get more point that count towards the Owls level you are at. Showing the professors you are progressing. When you have reach a specified threshold you will get to a new level, and you will automatically receive a diploma and be notified by email and SMS.

Flow 1:

• A trigger is set on the inventory row – if a row is added or if the quantity column is updated.
• Then we compose a calculation of which level the wizard has achieved based on the potions it has made. And the quantity of the potions.
• We update the wizards level – and make a HTTP Patch Request to the Graph Api and update the user in Entra with the level.

Flow 2

• When the level row for the wizard/user gets modified – we check which level it is, and then start the process for sending the diploma – which is described under:

OneFlow has allowed us to create diplomas for wizards who graduate from our potions and spells program. Whenever a wizard or witch has reached the highest level, they will get a diploma signed by Dumbledore himself.

The diploma is created in Word and exported as a PDF.

We uploaded it to oneflow and created it as a contract to be signed.

We enabled the API for Power Automate Flow

Setting up the Power Automate flow to create a contact with oneflow, add a participant and then publish the contact. Then we create a new row in the Diploma custom table in Dataverse where we connect the contract to the wizard (contact).

We receive an email with the confirmation that we have received the diploma.

And the link takes us to the document

Link Mobility

Setting up the connection reference is easy

Then we add the action to send an SMS to the wizard to notify them that they have graduated!

Triggering the flow will send an SMS with the link to view the diploma!

Digital Transformation

Asking copilot agent what potions you can make with the ingredients your find, and add it to your potion inventory.

You can use your potions – or you can choose to sell it to the highest bidder. Business Central is keeping track of your inventory and allows you to sell your items through a Power Pages site.

All this magic is delivered by Copilot Studio Agents.

Creating the Agent

The magical wizard who knows more about potions and spells and ingredients than Hermione Granger herself is actually a Copilot Agent.

Adding a public knowledge source

We created it with the Harry Potter Wiki as a knowledge source: https://harrypotter.fandom.com/wiki/List_of_potions

Connecting the agent to Dataverse as a knowledge source

In the setup we choose the tables that we want to make available as sources for the agent to draw data from.

Allow the Potions Matcher to create potions

The agent will be able to create new potions in your inventory in the beaded bag based on the ingredients you have. To allow for this the agent is granted permissions to create new rows in the Potion Inventory table on the users behalf

The end result being that the copilot agent is capable of doing magic on behalf of the students.

Business Central

The potential when connecting Business Central with Dataverse is incredible. We know that well fed students are happy students, and by allowing them to sell their potions on a website they will be able to earn some money while in school.

Fabric Fera Verto

Collecting all of this data is all good and well, and the different professors responsible for this solution needs to report on the students different levels to headmaster Dumbledore. To achieve this we send data to Fabric and create Power BI reports

We built a lakehouse to store all the data from Dataverse and the Harry Potter dataset. So that we could clean the data and also share the data with other professors that need the insight.

We use Dataflow Gen 2/Direct Query to always have the latest data for Dumbledore. And cleaned up the data in the process to easier make reports.

Used Power BI desktop to form relationships and to make the reports

Business Value
Reporting and keeping track of progress, activity levels, geographical differences as well as demographic differences within the volunteer group is incredibly important, and we know that the evolution of volunteer efforts in the global scope is changing. As we as humans change our ways and habits, our activities, loyalties (or lack there of) is also changing. Like all other enterprises, the volunteer programs around the world has to evolve with it. But what to do? What are the trends? What works and what doesn’t? It’s all there in the data, and with the right type of reporting and analytics we can get the crustal insights to evolve the programs with the rest of the world.

ALM Magic

We have set up power platform pipelines and have used solutions for all our components. Developing the datamodel first, using the new Plan Designer first to give us inspiration and doublechecking our way of thinking, and then building the data model from scratch to make sure we get the naming convention just right and follow best practices.

Business Value
The other aspects of our solution is not so exciting maybe, but we have made sure to follow development best practices by having security roles, environments and pipelines set up. We developed a naming convention and a solution strategy on day one and we have followed it. We have made sure that we create a solution that we are proud of to the point where we can show it to customers as-is – because that is exactly what we will do on a tech day event at Itera on February 6th. Wish us good luck 😀

Let the Power Potters harness this method to charm their way to flawless, fast-paced development! 🧙‍♂️

The pipeline:

Naming conventions

What	Convention	Example
Columns	Pascal case	pp_PowerPotters
Solutions	Capital letter for each word, short and descriptive name	Power Potter Potion Solution
Forms
Views

Adding copilot to agent and moving across environments

Of course, because we developed the datamodel in DEV, we could also easily develop and train the Copilot Agent in DEV and move it to production using solutions and Power Platform Pipelines, as well as the AI Models, and the Power Pages site

Entra ID Security Groups

We use Entra ID / Azure AD Security groups to control who has access to the different environments.

The security group is linked to the environment.