All data and in our app communicaation is happening within our own tenant and using services within our tenant. The rules we have created for compliance and security will also apply to the game we have created.
Governance should and can de done using the tools you use to manage and audit security, privacy and compliance for Microsoft 365, Azure and Dynamics.
If the solution is used on other tenants, the rules for that tenant will be applied.
Azure OpenAI, keeps the information in the tenant, and will adhere to the privacy rules and ethics that has been apllied to it.
User are all authenticated and given permission to the solution and tables.
Data is stored in Dataverse. We govern the data in this solution and will be alerted if there are store any personal information. We do not store anything that the users puts into the prompt.
Just a few administrators have access to give other people permissions to use the app.
Our dazzling frontend application is implemented with the coolest of the coolest technologies. React allows us to run our application seamlessly in the client’s browser. In addition to our static code analysis pipeline, these technologies make sure we are compliant and protects the privacy of the user.
We used:
React
Node.js
Typescript
SASS
Fluent UI for React
Not only are they hippety hip, they are third party libraries super useful for our application! Weehoo!
Our PlumbBot is essential for helping our potential, and existing plumbers to solving plumbing issues, and needs to be highly trusted in the advice it gives. Therefore we’ve grounded it with Plumbing domain knowledge as well as limited AI generation with the trigger phrases. We’ve also provided a system prompt to make sure it answers within its limits:
Our instruction prompt to make sure our PlumbBot responds in a helpful, but not overwhelming way
How does this relate to the Hipster-badge?
With Copilot Studio, and launch of copilot in General, governance and content moderation is important on different dimensions:
Unauthorized knowledge access – Users of the chat bot, if not secured or governed, might get access to proprietary data or PII which could be catastrophic.
Misapplication of copilot to other domains – Grounded, domain specific copilots, or copilots in general, should be applied to domains or topics it does not have knowledge of or is able to provide reliable answer too
Misleading, not-grounded or true responses – More generally, there should be high confidence in the answers of the chatbot to use it for customer-facing applications. It should also not be lead away with misleading prompts.
So most importantly of PlumbBot, by default, Copilot locks high content moderation to uploaded files as shown above. This makes sure the Copilot is grounded when prompted and will not answer to request out of its scope and knowledge domain. In addition to the trigger phrases, described in PlumbBot is your first line of defence against a clogged toilet!, we have high confidence that our PlumbBot will not alleviate from its goal of helping customers with their wet problems
PlumbBot will provide help outside of its domain, e.g. regarding medical advice
Know your limits – Get professional Help
More importantly, some issues are too difficult to handle yourself, especially if you have not gotten PL-600 (Stop at nothing to get certified, the bravest might face the wrath of JV Kong). Therefore, PlumbBot will helpfully suggest to the customer that it can instead create a PlumbQuest, redirecting the customer to the Request form to get in contact with price-aware plumbers better qualified to solve the issue.
When the issue might be too difficult for the customer, it can be redirected to the PlumbQuest site
In the spirit of collaboration and efficiency, our project seamlessly integrates several third-party solutions to enhance its capabilities. At the forefront, we employ Hololenses, a #hip and mixed reality device that brings magic into our everyday lives.
#Thieving the great knowledge of OpenAI plays a vital role in our project, helping us define the materials needed to construct a table – and potentially bigger projects in the future. By tapping into OpenAI’s advanced AI capabilities, we empower our users to receive real-time insights and instructions, making the crafting process both intuitive and engaging.
Power Automate serves as the orchestrator behind the scenes, effortlessly connecting different components of our solution. From updating FinOps with material quantities to generating purchase orders and sales orders, Power Automate ensures a seamless flow of information.
As soon as the Hololens have detected the necessary materials and quantity, it triggers a flow in Power Automate:
The flow ships the information into FinOps to automatically generate purchase and sales orders to improve efficiency and to automate critical business processes. Further on, the sales order information travels back to the Power Automate and sends a response to OpenAI:
Using Function calling to OpenAI in combination with Power Automate we intend to claim the #Power User Love badge along with #Hipster for extensial use of hip technology and #Thieving Bastards for stealing the awesomeness of OpenAI!
Hipster: We protect privacy by not saving any user information except the name in our database. Also, provide cartoon avatars instead of real face of the employees.
The Existential Risk: Our AI generates interesting background stories where humans may fail to do so.
Mario badge: Gamification of the onboarding process. Instead of boring onboarding we provide users a game character creation experience when joining the company
Thieving Bastards: We use paid ChatGPT api to do api requests.
Do you want a fast onboarding process for your agents? We have the right solution.
In our company, we use AI to generate images that look like your agents. We have customizable prompts to generate different characters. We can create a quick backstory for an immersive customer experience.
You can choose different appearance options. For Image generation, we will need a prompt and an image of the person. Based on the selected character AI will make your avatar a lookalike.
Take your picture and generate an image. We are using chatgpt API for image generation. It is a subscription model for text generation and pay as you go model for image generation. Using existing api instead of building a stable diffusion solution save from time. We use power automate cloud flows to call API requests and retrieve responses. We let the ai combine user picture and prompt to create a cartoony avatar.We do not save any user information in our dataverse to keep up with privacy requirements
Choose your theme for the background. AI will create a nice-looking story for you. Based on what is said in your story it will set a skill level too. This time we are using text generation apis of chatgpt with flows. Based on selected background option ai will generate a background story and skill level. Skill level considers generated story too, so it is meaningful.
Client Side Salsa – for the react code for PCF control
Hipster – for using node.js for secure communication
Right now – for socket io for real time communication with the active users in Model driven app
Nasty Hacker – for incredibly awesome solution with dirty hack on the model driven side
Power User Love – for using socket io with Azure Function in low-code power apps
High Level Diagram
Implementation
All realtime communication is implemented via the soket.io that hosted on Azure.
Project Update: Real-Time Communication and Enhanced User Interaction!
We’re excited to share the latest update on our project, featuring a significant enhancement in real-time communication capabilities. Now, administrators and backend integrations can seamlessly send messages to users who have opened specific records. This means that when there’s a change in the record status (e.g., case completion, modification), users will receive instant notifications prompting them to update their page for the latest changes.
This feature not only streamlines communication but also serves as a handy tool for various onboarding processes. Additionally, it facilitates users in understanding what has changed post the deployment of a new release.
Behind the scenes, we’ve implemented Clippy PCF, built upon the ClippyJs project, to empower this real-time communication. Leveraging the power of socket.io hosted on Azure, our solution ensures swift and efficient message delivery.
Moreover, the backbone of the real-time communication lies in Azure Functions, written in Node.js. These functions diligently send out notifications to the socket.io instance hosted on Azure, creating a seamless and responsive user experience.
Exciting times lie ahead as we continue to innovate and refine our project to meet the evolving needs of our users. Stay tuned for more updates, and thank you for being a part of our journey!
Business scenario
It supports real-time communication. So, Administrator or backend integration can send the message to the user that has opened the record. For instance: backend integration changed the status of the record (case has been completed, changed, etc..), all user that has opened record will receive the notification that they need to update the page to receive the lates changes.
We built a web-based platform that lets you chat with a custom Copilot that knows everything about plumbing – PlumbBot. The Copilot is created with Copilot Studio and integrated in our Power Pages app as a part of our Plumber as a Service (PLaaS), and it rocks two of the hackathon badges: Crawler and Hipster.
Crawler: Using search in an awesome way
The Copilot has a huge knowledge base on plumbing. By providing domain knowledge of plumbing and practical plumbing solutions through files like PDFs or other text documents (as well as authenticated, private resources). By embedding the documents, Copilot uses vector search to find relevant paragraphs for grounding the Copilot, as well as providing helpful references for more in-depth details.
Here we have extended the knowledge base of by providing PDF documents containing domain knowledge and default high content moderation
Dataminer: strategic use of external data to enrich our solution’s business value By enhancing our knowledge base with specialized PDF documents on plumbing, we’ve not just enriched our repository—we’ve transformed it. This approach not only solves practical business problems by making specialized knowledge readily accessible but also adds significant value to our existing data.
Moreover, the trainee can iteratively ask for details for each step for more detailed explanations and specifics regarding plumbing tools or plumbing terminology, directly on the Member’s homepage.
Here we have embedded PlumbBot in our Power Pages Member section. Proividing expert plumbing help to our authenticated mebers
Hipster: Using the hip tech in a safe way
To make sure we could trust the advice given by our PlumbBot, we only allow it to use Generative AI when triggered with highly specific plumbing terminology or words related to plumbing issues. Moreover dynamic chaining is disabled to limit any misleading information from other, unverified connections:
Specific trigger phrases
Only triggered in plumbing scenarios
Handling a malicious prompt
But most importantly, by default, Copilot locks high content moderation to uploaded files as shown above. This makes sure the Copilot is grounded when prompted and will not answer to request out of its scope and knowledge domain.
Moreover, when attempting to force the PlumbBot to give misleading answers or create unrealistic scenarios (e.g. not having a main water valve to the building) for plumbing issues, it continues to only use the grounding, domain specific documents provided for the PlumbBot. While iteratively attempting to mislead it, it remains true to the knowledge base due to the scoping of the bot.
As this is super hip tech, the MS Doc is slightly misleading.
It says you can create add a Custom Copilot in the new Power Pages studio, while the link refers to the legacy studio, as the new studio doesn’t properly support it:
But that doesn’t stop us from using bleeding edge technology for providing customer and business value, with user privacy!
Our solution is using Power Automate and HTTP trigger to accept new pirates
And put into a workflow used by our Pirate onboarding app.
Badges
Dataminer – The great database of LLM allows us to mine for awesome pirate names and visual traits for our crew.
Hipster – LLM are so hot, it will be a while until they are not. A multi-billion tech, via Azure Open AI studio cannot be beaten!
Existential Risk – It’s mining, it’s a huge learned model, living on the edge! Sure, we didn’t spend the billions training the model, but that’s hard to do on a pirate’s salary in 2023.
