When we’re creating an app for everyone one to use for their everyday Pirate activites coming from F&O, we might as well include som external data as well to make the experience for the Pirate even better. We have never, and never will, argue that you will get all the information you need every day from F&O, so we have given every Pirate a bit more data they can use every day in their canvas app:
When you first log in to the app in the mornig, it’s handy to get a little greeting just “welcomming” you to a new day, right? We have created a PowerAutomate that uses OpenAI to find a new Pirate greeting just for you (if using AI to cheer up your Pirates isn’t an innovative use of AI, then what is?):
2. If you feel down, and just want some text to make you laugh a little, you don’t need to go to the internet, we have created an Insult of tha day feature in your app. Just click it and start feeling better, at least if you think that this insult is regarding you bunk-buddy 🙂
3. And then, if you want to know what kind of weather it is, look no further, it’s already right there in you Pirate app:
Why reinvent the wheel when you can have a flying start from the tech that others already have created.
SharePoint: The amazing community behind PNP created an awesome template that we have decided to use for the intranet.
Power Automate:
Working with Power Automate connected to Dataverse is almost impossible without the great tools of XRMToolbox and Jonas Rapps FetchXML Builder!
This tool has saved the “lives” of som many Dataverse/Dynamcis Consultants.
We also have used a community connector for the OpenAI that returns pirate name from contacts that are entered.
PCF: Even though we are not going to use our Model driven app for the end result, we imported a PCF controll from pcf.gallery just for the purpose of making it look better:)
Ahoy there! It’s finally the first day of ACDC 2023 and we are starting off strong with a full team of five pirates.
Our team har kept an extremely good team spirit throughout the first day, and we have already come far with our work. We are working with technology that we are both familiar and unfamiliar with. We are making everything from making a boat simulator to face recognition. We support each other, share ideas and fool around to boost morale and increase motivation.
With hats, hoodies and lots of tech stuff we consider us happy campers
We startet by setting up an Azure Synapse workspace and a Data Lake Storage Gen2. Then we gave ourself the required access roles which is “Owner” and “Storage Blob Data Contributor”. After this was done, we started making the Synapse Link from Power Apps and picked the tables that we needed for our solution. We created a new SQL database and created views with the tables that we imported through Synapse Link. We chose to use Azure Synapse Analytics because we wanted to have near real-time data which is detecting whenever new data comes in. Our next step was to connect Power BI with “Azure Synapse Analytics SQL” using Direct Query. We started working on the report when all the data was imported and the relationship between the tables was working properly.
Any people who want to be recruited to a pirate organization must go through our recruitment process. It starts by being recruited by a parrot which is used by face recognition, and it is going to put a pirate head on your head automatically. After this, everyone will have the chance of trying out our ship simulator called “Pirate 365 Enhance”. Your performance will be recorded, and the data will be sent to our Power BI record which uses direct query in order to get real-time data.
We will have multiple reports, “Pirate 365 Enhance”, “Pirate 365 Enhance Evaluation” and “Parrot Obervation AI”.
The data from our participants will firstly be used to create a leaderboard in order to see who perform the best. We display the number of games played, total score, participants and how many levels have been completed. Our next page is “Pirate 365 Enhance Evaluation” (which is not created yet) will be used by the people in HR who are going to evaluate each participant by their performance. The total score will have a huge part in which ship you will be recruited to, and which role you will get. If your score is too low, then you must walk the plank!
The next site “Pirate 365 Evaluation” have an embedded canvas app from Power Apps. Here you will see how many points each participant from the ship simulator got and the canvas app will suggest which role suits the participants the best based on a points-criteria, but you don’t have to pick the suggested role. The person evaluating this process can choose which ship the participant has to go to.
In this case. We gave “Jan-Agger Jack” a role as a “Surgeon” in “The Black Pearl”. This next step is to either onboard or offboard this participant.
Then a video will be shown where the captain of the ship will either welcome you to the ship or to walk the plank based on the option that you pick.
The reports are embedded in the recruitment Teams channel so HR easily can follow up
Our PRT is doing real time object detection to spot pirates and bottles, but the data doesn’t leave the terminal. As the captain wants to know how many recruits we are attracting we needed to expose it to his “weapon of choice”; Power BI.
Luckily as the app is using a modern data store🍍 this is easily achievable by inserting a listener and sending the data to power bi on change 👍.
The way we connected to Power BI was to go to “My workspace” and create a new dashboard which has a streaming dataset which connects to an API. Then we gave inserted the right values from stream in order to get the right data.
We are now getting the real-time data into Power BI, we just need to add visuals into the dashboard. We go to “Edit”, and add a tile by using the real-time dataset that we created. We added two gauges and two line charts.
These visuals will now give real-time metrics from the parrot when there are either faces or bottles detected from the camera.
Shiny, glossy user interfaces would earn this badge, it won’t break on small screens right?
So, at the request of the judges, we will shed some more light on our app, its UI and super glossy pixels.
The app is made in vue.js and contains three main components:
The ThreeD component, responsible for rendering our 3d parrot
The FormUI component, responsible for rendering text and input
The Camera component, responsible for gathering camera input, object detecting and debug interface
The ThreeD Component
In this component we use troisjs and three to setup a scene containing camera, lights, our parrot model, and positioning this to match the rest of the UI.
As the model we are using contains all the different animations inside on animation sequence, we have to split it up into different actions, like idle, scream, takeoff etc.
The FormUI Component
Adhering to the shadow first(tm) principle
In this component we use vue template logic in combination with CSS and stolen borrowed texture to create the interface for providing instructions and gather inputs. We also managed to sneak in the actual font borrowed from The Secret of Monkey Island, after battling with getting ttfs converted to woff🐶.
The Camera component
The camera component is not seen on the screen and does not directly contribute to the UI. However when for debugging we’ve build a little UI to show the detected objects.
Tensorflow can run in the web browser, and is able to both detect persons and other objects like bottles.
Hooking up the webcamera with the frame analyzerDetecting bottles and pirates
ChatGPT to create PirateNames
We have already blogged about how we are usting OpenAI’s DaVinci model to generate Pirate Names here
Here we use the following prompt
Create a creative and cool pirate name from the name "{firstname lastname}". Do not use the word "captain". Answer only with the name - max 3 words. Only output text characters a-z, no special characters!
Hipster-ish technology
Making this we leveraged vue.js, pinia, three, troisjs, tensorflow, ChatGPT. The list goes on 😉. All pretty hip, (almost) non-ms technology.
Badges
We hope this sholud be sufficient for getting the AI- and Hipster badge 🏴☠️✌️🦜
An other team needed helå styling their canvas app to make it look a bit nicer than what they had managed so far. And, in addition to this we also could help them with an html element. We are happy to do whatever we can to make others sucseed, so of corse one of our team mates spent a good 20 minutes with them 🙂
To best explain how much we love the Power Platform, we can have a quick look into the different components of the Power Platform we are using in our solution
Captain The captain is using a Power Page to register new incoming raids.
Harbor Master: The harbor master is notified by email and the speaker system at the dock by using a Power Automate flow connected with an external HTTP request to SONOS
This flow is only one of MANY flows controlling the application.
Crew: Crew members update their personal information via Canvas APP
Harbor Master: Has full control over all financial details on raids via the Power BI report that he has set up.
Model driven apps:
Of course, everything is possible to navigate via Model-Driven apps, but we have decided not to include this app in our end-user setup.
When we need new pirates on our crew we have an app where we can take their picture, and the app send the picture to a server that uses AI to piratefy your face.
The app is a Canvas App with a camera control on the first screen
When user click the camera control they navigate to the next screen where they will see their picture.
They can go back and take a new picture if they are not happy, or move on to “Piratefy me”.
If user click the “Pratefy me” button a power automate flow is triggered. We pass the image base64 value as a value and store the reply value in a variable.
The flow is “Me as a Pirate”
The trigger is a power apps trigger.
First we initiate reusable variables
We grab the image value from the Power Automate
Passing the image over to the CLIP service powered by Stable Diffusion at the Frogner/Oslo datacenter we annotate images to mostly correct the gender or non-gender of the person in the photo. This is our responsible way to ensure the pirate representation of the person is not turning the most beautiful guy and girl into something they definitely don’t identify with.
We parse the response we get
Next we pass in the captured photo, and overlay it with some serious pirate’y prompting to ensure our new crew mates meats the bar (and also the real bar when port-side).
We need to modify the image string to make it the right format for the Ai to handle.
We trigger a child flow (documented further down)
We gather the reponse and send it back to the Power App.
The flow triggered as a Child flow is called “get Crew Member Data – PirateGDP”
The core API in out solution is PirateGPT, a large language model fully capable assigning awesome pirate names and corresponding ships, with the identifiable traits of the pirate. Which we as a responsible HR ship keep fully secure and up to date as needed. When you have seven John’s on one ship, it’s quite important to distinguish them by traits such as red beard or cock farmer.
We PARSE the results we get for get the right format
Respond with the values to the “parent” flow
The Power App
Back in the Canvas App the user will see the newly created name and their picture Piratified:
It also navigate to the next screen where they will see the pirate name they are given and their picture Piratified.
If this isnt glossy and extreme business value, I don’t know what is?
Imagine this being a manager taking pictures of new employees. We all know that Gen Z are used to looking at themselves with prettifying filters. This way the old middle manager can take pictures of their new employees and let Open AI add the familiar filters so that when the Gen Z employee see their own new profile picture they will actually recognize themselves.
Badges
Glossy Pixels – because pirates are beautiful.
Go with the Flow – because it’s all flowing left and right
The Existential Risk – Who knew LLM could be used to generate schematized JSON data structures, and who knew all those crazy internet pics would be the basis for pirate faces of 2023?
ACDC Craftman – All API’s are tested and documented in a shared Postman collection, all is ran over https (not from the beginning) to avoid photo snooping, and API keys are used as the extra layer. No one wants to be the next TMZ victim!
