With this blog post we aim to show case that our solution is developed using best practices in terms of environment strategy and solution strategy.
Created a new solution:
Created a publisher:
Ready to use:
Function calling to open AI
In the spirit of collaboration and efficiency, our project seamlessly integrates several third-party solutions to enhance its capabilities. At the forefront, we employ Hololenses, a #hip and mixed reality device that brings magic into our everyday lives.
#Thieving the great knowledge of OpenAI plays a vital role in our project, helping us define the materials needed to construct a table – and potentially bigger projects in the future. By tapping into OpenAI’s advanced AI capabilities, we empower our users to receive real-time insights and instructions, making the crafting process both intuitive and engaging.
Power Automate serves as the orchestrator behind the scenes, effortlessly connecting different components of our solution. From updating FinOps with material quantities to generating purchase orders and sales orders, Power Automate ensures a seamless flow of information.
As soon as the Hololens have detected the necessary materials and quantity, it triggers a flow in Power Automate:
The flow ships the information into FinOps to automatically generate purchase and sales orders to improve efficiency and to automate critical business processes.
Further on, the sales order information travels back to the Power Automate and sends a response to OpenAI:
Using Function calling to OpenAI in combination with Power Automate we intend to claim the #Power User Love badge along with #Hipster for extensial use of hip technology and #Thieving Bastards for stealing the awesomeness of OpenAI!
We claim badges:
Hipster: We protect privacy by not saving any user information except the name in our database. Also, provide cartoon avatars instead of real face of the employees.
The Existential Risk: Our AI generates interesting background stories where humans may fail to do so.
Mario badge: Gamification of the onboarding process. Instead of boring onboarding we provide users a game character creation experience when joining the company
Thieving Bastards: We use paid ChatGPT api to do api requests.
Do you want a fast onboarding process for your agents? We have the right solution.
In our company, we use AI to generate images that look like your agents. We have customizable prompts to generate different characters. We can create a quick backstory for an immersive customer experience.
You can choose different appearance options. For Image generation, we will need a prompt and an image of the person. Based on the selected character AI will make your avatar a lookalike.
Take your picture and generate an image. We are using chatgpt API for image generation. It is a subscription model for text generation and pay as you go model for image generation. Using existing api instead of building a stable diffusion solution save from time. We use power automate cloud flows to call API requests and retrieve responses. We let the ai combine user picture and prompt to create a cartoony avatar.We do not save any user information in our dataverse to keep up with privacy requirements
Choose your theme for the background. AI will create a nice-looking story for you. Based on what is said in your story it will set a skill level too. This time we are using text generation apis of chatgpt with flows. Based on selected background option ai will generate a background story and skill level. Skill level considers generated story too, so it is meaningful.
.
We claim badges:
High Level Diagram
Implementation
All realtime communication is implemented via the soket.io that hosted on Azure.
Project Update: Real-Time Communication and Enhanced User Interaction!
We’re excited to share the latest update on our project, featuring a significant enhancement in real-time communication capabilities. Now, administrators and backend integrations can seamlessly send messages to users who have opened specific records. This means that when there’s a change in the record status (e.g., case completion, modification), users will receive instant notifications prompting them to update their page for the latest changes.
This feature not only streamlines communication but also serves as a handy tool for various onboarding processes. Additionally, it facilitates users in understanding what has changed post the deployment of a new release.
Behind the scenes, we’ve implemented Clippy PCF, built upon the ClippyJs project, to empower this real-time communication. Leveraging the power of socket.io hosted on Azure, our solution ensures swift and efficient message delivery.
Moreover, the backbone of the real-time communication lies in Azure Functions, written in Node.js. These functions diligently send out notifications to the socket.io instance hosted on Azure, creating a seamless and responsive user experience.
Exciting times lie ahead as we continue to innovate and refine our project to meet the evolving needs of our users. Stay tuned for more updates, and thank you for being a part of our journey!
Business scenario
It supports real-time communication. So, Administrator or backend integration can send the message to the user that has opened the record. For instance: backend integration changed the status of the record (case has been completed, changed, etc..), all user that has opened record will receive the notification that they need to update the page to receive the lates changes.
We built a web-based platform that lets you chat with a custom Copilot that knows everything about plumbing – PlumbBot. The Copilot is created with Copilot Studio and integrated in our Power Pages app as a part of our Plumber as a Service (PLaaS), and it rocks two of the hackathon badges: Crawler and Hipster.
The Copilot has a huge knowledge base on plumbing. By providing domain knowledge of plumbing and practical plumbing solutions through files like PDFs or other text documents (as well as authenticated, private resources). By embedding the documents, Copilot uses vector search to find relevant paragraphs for grounding the Copilot, as well as providing helpful references for more in-depth details.
Dataminer: strategic use of external data to enrich our solution’s business value
By enhancing our knowledge base with specialized PDF documents on plumbing, we’ve not just enriched our repository—we’ve transformed it. This approach not only solves practical business problems by making specialized knowledge readily accessible but also adds significant value to our existing data.
Moreover, the trainee can iteratively ask for details for each step for more detailed explanations and specifics regarding plumbing tools or plumbing terminology, directly on the Member’s homepage.
To make sure we could trust the advice given by our PlumbBot, we only allow it to use Generative AI when triggered with highly specific plumbing terminology or words related to plumbing issues. Moreover dynamic chaining is disabled to limit any misleading information from other, unverified connections:
But most importantly, by default, Copilot locks high content moderation to uploaded files as shown above. This makes sure the Copilot is grounded when prompted and will not answer to request out of its scope and knowledge domain.
Moreover, when attempting to force the PlumbBot to give misleading answers or create unrealistic scenarios (e.g. not having a main water valve to the building) for plumbing issues, it continues to only use the grounding, domain specific documents provided for the PlumbBot. While iteratively attempting to mislead it, it remains true to the knowledge base due to the scoping of the bot.
As this is super hip tech, the MS Doc is slightly misleading.
It says you can create add a Custom Copilot in the new Power Pages studio, while the link refers to the legacy studio, as the new studio doesn’t properly support it:
But that doesn’t stop us from using bleeding edge technology for providing customer and business value, with user privacy!
Our solution is using Power Automate and HTTP trigger to accept new pirates
And put into a workflow used by our Pirate onboarding app.
Badges
Dataminer – The great database of LLM allows us to mine for awesome pirate names and visual traits for our crew.
Hipster – LLM are so hot, it will be a while until they are not. A multi-billion tech, via Azure Open AI studio cannot be beaten!
Existential Risk – It’s mining, it’s a huge learned model, living on the edge! Sure, we didn’t spend the billions training the model, but that’s hard to do on a pirate’s salary in 2023.
The Orange Bandits’ operation center is need for a mechanism to track vehicles to get more insights about ongoing looting operations. A raspberry pi with a tracking device is the first solution that comes in mind!
Though, this is not a straightforward solution. Some Python code is due at this point. Unfortunately, The Orange Bandits owns an old Raspberry pi with an old SD card that doesn’t tolerate new updates; not retro enough! Well, we had to use an old version of Python to get this to work!
Embedding a device that flip bytes is not an easy ride. We need to come with innovative ways to to keep it online and to keep the GPS tracker in the “Chinese” satellite signal range!
Pirates 365 uses several AI components
We are using Tensorflow to detect passing pirates
Tensorflow can run in the web browser, and is able to both detect persons and other objects like bottles.
We have already blogged about how we are usting OpenAI’s DaVinci model to generate Pirate Names here
Here we use the following prompt
Create a creative and cool pirate name from the name "{firstname lastname}". Do not use the word "captain". Answer only with the name - max 3 words. Only output text characters a-z, no special characters!Making this we leveraged vue.js, pinia, three, troisjs, tensorflow, ChatGPT. The list goes on 😉. All pretty hip, (almost) non-ms technology.
We hope this sholud be sufficient for getting the AI- and Hipster badge 🏴☠️✌️🦜
Recruitment is an important, and integrated part of Pirates 365, to acquire the best talent for our crew. To help us with this we have developed The pirate recruitment terminal (PRT). This solution provides us with a visually appealing and an engaging experience to gather new recruits.
PRT User flow
Since pirate recruits normally aren’t that familiar with modern interfaces, we have chosen a living parrot to help and guide the user and provide a smooth user experience.
When a pirate is detected, our engaging parrot approaches the pirate and presents the sign-up option. We collect personal details and complete the process by taking a picture. The collected data is then sent into the platform for future processing.
The Pirate recruitment terminal technology stack
PRT The app is running on the vue.js progressive javaScript framework with Pinia🍍 for state management. For the 3d animated parrot🦜 we are using three.js and TroisJS with WebGL rendering. For AI vision👁️ and object detection🤠 we are running tensorflow under the hood. If available the model is running on GPU, if not we fall back to CPU.
Images, 3d model, fonts is stolen captured on the internetsea.
Everything is running in the local browser🤓
Figma, the hippest UI kid on the block, has a setting for components called “properties”. They are intended to mimic how we use properties in frameworks like React. However when working with multiple components and when swapping components they grant some extra value to the design process.
Having properties for the values makes it easier to update multiple values though-out your design.
But more importantly, when using variations of a component the component remembers its override-values even when switching to a variant that does not have the same values in the component, making updating your design that much easier.
Just trust me on this, it’s the best practise
– Every Developer ever, and now also a designer