Tensorflow can run in the web browser, and is able to both detect persons and other objects like bottles.
Hooking up the webcamera with the frame analyzerDetecting bottles and pirates
ChatGPT to create PirateNames
We have already blogged about how we are usting OpenAI’s DaVinci model to generate Pirate Names here
Here we use the following prompt
Create a creative and cool pirate name from the name "{firstname lastname}". Do not use the word "captain". Answer only with the name - max 3 words. Only output text characters a-z, no special characters!
Hipster-ish technology
Making this we leveraged vue.js, pinia, three, troisjs, tensorflow, ChatGPT. The list goes on 😉. All pretty hip, (almost) non-ms technology.
Badges
We hope this sholud be sufficient for getting the AI- and Hipster badge 🏴☠️✌️🦜
When we need new pirates on our crew we have an app where we can take their picture, and the app send the picture to a server that uses AI to piratefy your face.
The app is a Canvas App with a camera control on the first screen
When user click the camera control they navigate to the next screen where they will see their picture.
They can go back and take a new picture if they are not happy, or move on to “Piratefy me”.
If user click the “Pratefy me” button a power automate flow is triggered. We pass the image base64 value as a value and store the reply value in a variable.
The flow is “Me as a Pirate”
The trigger is a power apps trigger.
First we initiate reusable variables
We grab the image value from the Power Automate
Passing the image over to the CLIP service powered by Stable Diffusion at the Frogner/Oslo datacenter we annotate images to mostly correct the gender or non-gender of the person in the photo. This is our responsible way to ensure the pirate representation of the person is not turning the most beautiful guy and girl into something they definitely don’t identify with.
We parse the response we get
Next we pass in the captured photo, and overlay it with some serious pirate’y prompting to ensure our new crew mates meats the bar (and also the real bar when port-side).
We need to modify the image string to make it the right format for the Ai to handle.
We trigger a child flow (documented further down)
We gather the reponse and send it back to the Power App.
The flow triggered as a Child flow is called “get Crew Member Data – PirateGDP”
The core API in out solution is PirateGPT, a large language model fully capable assigning awesome pirate names and corresponding ships, with the identifiable traits of the pirate. Which we as a responsible HR ship keep fully secure and up to date as needed. When you have seven John’s on one ship, it’s quite important to distinguish them by traits such as red beard or cock farmer.
We PARSE the results we get for get the right format
Respond with the values to the “parent” flow
The Power App
Back in the Canvas App the user will see the newly created name and their picture Piratified:
It also navigate to the next screen where they will see the pirate name they are given and their picture Piratified.
If this isnt glossy and extreme business value, I don’t know what is?
Imagine this being a manager taking pictures of new employees. We all know that Gen Z are used to looking at themselves with prettifying filters. This way the old middle manager can take pictures of their new employees and let Open AI add the familiar filters so that when the Gen Z employee see their own new profile picture they will actually recognize themselves.
Badges
Glossy Pixels – because pirates are beautiful.
Go with the Flow – because it’s all flowing left and right
The Existential Risk – Who knew LLM could be used to generate schematized JSON data structures, and who knew all those crazy internet pics would be the basis for pirate faces of 2023?
ACDC Craftman – All API’s are tested and documented in a shared Postman collection, all is ran over https (not from the beginning) to avoid photo snooping, and API keys are used as the extra layer. No one wants to be the next TMZ victim!
This is every pirate’s best friend and a must have!! Loaded with awesome features this killer app should be installed on the mobile phone of all pirates in your organization.
Targets
Get a quick overview of your targets. Once a ship has been become a target in Dynamics 365, you will be able to see it in your mission overview. This is perfect when you have a long day of plundering i front of you and need to know which ones to do next.
Rumglasses
Have you ever been too deep in the rum bottle and close to do something stupid with what you believed to be a mermaid out on the seven seas? Or maybe you actually did and are now in for a hard time facing your mates with humiliation and regret?
Not anymore! Rumglasses let you take a picture of your believe-to-be mermaid and tells you if you are right or not. By training an AI to recognize and distinguish manatees from mermaids you can now relax.
My Ship
A map showing your location when out on the seven seas. You will se both your ship on the map and all other ships in the area. This is how you know you are close to your target when out on a plunder. The best part is that military or law enforcement ships will occur on the map as well so you can wait until the right moment before attacking the target.
Bottle Mail
Communicate with your mates by using bottle mail – a feature for sending email when you are on the fly. It suggests recipients in you address book or you can write to a pirate outside your organization. This way, you don’t have to switch to another app (email client) to send messages to your crew.
Pollybot
You are not a real pirate unless you have a parrot. Unfortunately (or luckily) there are laws about how to treat your animals today that did not exist in the old glory days. It might not be appropriate to bring it with you on a raid :/ This is where Pollybot comes to the rescue! Your own digital parrot who you can talk to, ask questions and keep you with company. As a real parrot, it sometimes answer your questions, but most of the time you will get an insult or just nonsense in return. But that doesn’t matter! Where a real parrot will fly away to save itself, Pollybot will always stay close.
Mayday, Mayday
When Jack Sparrow was stranded on a desert island, he managed to escape by tying his back hair together and catch a couple of sea turtles who gave him a ride back home. As fun and comfortably that sounds, not all pirates are equipped with back hair that long. However, most of us got their phone in their pockets at all times!
Whether you are stranded at a desert island, your ship is about to sink or you are stuck in a safe full of treasure, this feature will save you.
When using the mayday function your coordinates are then sent to a email address monitored by you crew so they can come and pick you up. If your mates prioritize your request is however another matter.
Because of the fancy design, the use of AI and the huge amount of useful features in this app, we hereby claim the Glossy pixels, Existential Risk and Feature Bombing badges. We also claim the Power user Love because the app is created in Power Platform 😀 #ProCodeNoCodeUnite
Recruitment is an important, and integrated part of Pirates 365, to acquire the best talent for our crew. To help us with this we have developed The pirate recruitment terminal(PRT). This solution provides us with a visually appealing and an engaging experience to gather new recruits.
Illustration of the Pirate recruitment terminal’s place within the Pirate 365 Recruitment module, along with the other modules.
PRT User flow
Since pirate recruits normally aren’t that familiar with modern interfaces, we have chosen a living parrot to help and guide the user and provide a smooth user experience.
When a pirate is detected, our engaging parrot approaches the pirate and presents the sign-up option. We collect personal details and complete the process by taking a picture. The collected data is then sent into the platform for future processing.
Sign-up formPicture capture
The Pirate recruitment terminal technology stack
PRT The app is running on the vue.js progressive javaScript framework with Pinia🍍 for state management. For the 3d animated parrot🦜 we are using three.js and TroisJS with WebGL rendering. For AI vision👁️ and object detection🤠 we are running tensorflow under the hood. If available the model is running on GPU, if not we fall back to CPU.
Images, 3d model, fonts is stolen captured on the internetsea.
Captain Black Bart, is planning a big raid, Oslo is the target and according to ChatGPT the loot potential is #awsome.
Invitation is sent to all chosen crewmates using top of the art “Power Automate flows”. Crewmates are free to Accept or Decline the raid invitation.
When it’s time for the raid, we requires all crew members who have accepted the raid to check in, using their RFID card or phone.
If the crewmate have accepted more than one raid, he/she can choose to check in using our Raid Planning App
With the help of Midjourney AI, https://docs.midjourney.com/, we generated the app design component by asking the AI to provide us witht the latest & greates piracy themes.
The RFID data for all potential crewmates is stored in Azure AD using the “Postal Code” field, pretty “Nasty Hack” according to our “bosun”.
This data is imported to Dataverse “AAD User” virtual table for our app.
The waters run deep and dark when you are at sea. This is why intel like maps, a spyglass and knowing the tides help. To give our pirates an advantage we crawl barenswatch.no for other ships location. But, just as in dogeball, it is not enough to know where something is, you need to know where it is going and when you need to be there to intercept. So how do we calculate the best opportunities and how to intercept them?
Step 1: Crawl and Hijack Valuable Data 🦀
From the system diagram we see our Functions App communicates with the Braenswatch API and stors all boats data including current location. We poll this information several times giving us multiple data points not only for ballast and weight, but also for speed, location and direction (this will be important later).
Screen grab of system model for the Pirate Pillaging app
For ease of use while creating the service we compiled all the data using Power BI Dashboards to see what we were working with. The dashboard can also be used to help captains understand the state of the seven seas, although we appreciate that more data is not necessarily better data. It depends on how you visualize, precent and timing of when to precent data to you user.
Video of Solveig ☀️ explaining how we gather, structure, compile and visualize ship data in a way highlighting opportune targets.
Step 2: Knowing which Opportunities to Strike 🏆
From the ships length, type and destination we can estimate its value. Slow moving ships heading for freight docs are ripe for looting, fast moving ships heading for a dry dock are empty and better left alone. For the sharp eye you have also spotted we have a Threat flag in or model, this is for marking known pirates, intercepters and the fuzz of the seven seas. Once again referencing to Sun Tzu: You always win the wars you never fight (dont search this quote).
Image displaying famous quote from Sun Tzu about fighting good
If we decide that this opportunity is ripe for plundering, we look at its course and if it is feasible to intercept based on our current course. Good opportunities are heavy, slow moving ships. Bad opportunities are war ships of any kind or any opportunity in the vicinity of a war ship.
There is also a certain Goldilocks of opportunities: Based on your own vessel, you do not want to attack another ship that is too large, since it will be a harder fight, or too small, since it carries less booty. We can use the dashboard to visually find the opportune boats and use this investigation to write code that targets these boats automatically.
Captain Mats looking at data The Power BI Dashboard populated with crawled and manipulated data from Barenswatch.no, structured by ship type, weight and destination, highlighting the most opportune ships in the collection
When we flag ships that are both opportune and not adjacent to warships we plot paths to figure out where they are going in relations to our course.
Step 3: Using Advanced Trigonometry to Pinpoint your Enemy 🎯
Youtube can teach you the basics about how to calculate a ships heading based on longitude, latitude, direction, true north and the course deviation.
Youtube is telling us what we need to know about complex
True north will change depending on where on the globe you are positioned, so for intercept courses it is paramount to get this stuff right. You want to be at the place where your target ship will be, not where they have been. And you want to be there before they get there, not to ride up after them and trailing them forever. This is where we use the latitude, longitude, direction and speed from earlier and _try_ to plot the path of the ships. I stress try because Solveig just looked at Sebastian with the “do you mind not being an idiot” when he asked how se was doing it and if it was done. Then just sort of sigh’ed and continued clicking keys. So it works, trust us.
Solveig ☀️ working on triangulating and plotting ships path to calculate intercept courses.
For our challenge we also need to estimate if it is a viable option to change our current corse to intercept the opportunity. For all business cases there is a cost/effect probability calculation. Fo us this depends on added time to destination with deviation and added length of trip, both of which adds to the total cost of provisions and consumables like petrol.
Step 4: Plot Optimal Course for Interception 🏴☠️
Finally when we have a list of opportune targets, we know which targets will be adjacent to our planned course form A to B we plot intercept maneuvers. Which, is a topic for another blog post – because this part is most certainly done… Trust us… 🤞🤞🤞
Okey, we do agree that solution that won is a bit better than this, but that’s why we think this is a dirty hack! Instead of doing what you should do with the challenge, we ended up creating a canvas app that could run offline and add that to the Power Page. It still solves the challenge, doesn’t it?
And, you might as how we got this brilliant idea.. What is more innovative use of AI if it’s not asking these questions you don’t have the answer for yourself? So we asked ChatGPT how we could solve this challenge, and this is the answer we got:
We get satellite images from the Sentinel Hub API where we can get images from the coordinates we want. This makes it easy to compare images with AIS data from the same time period and coordinates.
Sattelite image of shipsCounting ships
To identify ships from the satellite images, we need more eyes than we have available. So what’s better than letting the machines do the work. We have developed our own ML code in Python to recognize ships. This will make it much more efficient than manually reviewing all the images. Currently, the machine has 99% confidence in recognizing ships. It then creates a heat map that shows where it thinks (with 99% probability) there are ships. Then it counts the number of ships and sends it on.
Training the model to gain confidence
When we know how many ships are within a given area at a given time, it is easy to compare this with data from AIS. Here we search through the API of the Norwegian Coastal Administration, where we can count the number of ships within the same time and area.
JSON from AIS data
Using external data in this way provides great value for our customers as the threat at sea is significantly lower 💾🏴☠️
The reason we are calling this the jackpot app because we believe that it’s hitting multiple score-points!
The idea of this app is to enlist pirates’ fleet without a single character punch!!! Aided and infused by AI Builder, external API’s and Power Automate Flows; this Power Apps canvas app is state of the art geekiness!
Although, registered vehicles are the last of our worries (as pirates), but hey we thought of doing it anyways 🙂 The app starts by taking a picture of the registration plate of the vehicle. the app will send the picture to the AI Builder through a power automate flow that process the image and read the registration plate and show it on the app for confirmation.
Once the you are satisfied with the result, you can check the vehicle against the national registry and search for more registration information. This indeed does check the car against the Norwegian car registry (Statens vegvesen) show if the vehicle is approved for going on the European roads (EU-godskjenning). Also least of our concerns!
Actual data from Statens VegvesenCrawling and datamining data from external sources
Now, after confirming and inserting car data into our system. We need to identify the vehicle type and compare with our collected data in our backend system. However, identify the care is a simple process that doesn’t need any text input from the user. One picture will do while the AI Builder will take care of identifying the type of the vehicle and insert it into our backend system.
Vehicle plate identification flow (calls another flow through HTTP)AI Builder made as an endpoint (all AI Builder models and flow are on separate environment, as AI Builder can’t be included in a solution for deployment)
We are using a mix of built-in and custom AI models to achieve efficiency and productivity in an innovative way.
Bacground: When recruiting potential pirates we want to give them proper pirate names. Pirates 365 is a service oriented platform, and the recruitment frontend posts potential new pirates through a Pirate
Our solutions is using Power Automate and the HTTP Trigger to accept new pirates
We are using the OpenAI (independent publisher) step found in Power Automate (https://learn.microsoft.com/en-us/connectors/openaiip/) since it is yet not so easy to get access to Azure Open AI which probably would be better.
We had to do some tuning to get a little bit different answers – at first it just returned “Captain firstname lastname”, but after some tuning by setting down the temperature from 1 to 0,7 and explicity saying not to use the word “captain” we got the desired result.
We are creating the new pirate recruits as Leads in Dataverse so the recruitment team can contact them for possible empolyment
Here is the lead created in Dataverse shown in the “Pirate 365 Manage”, ready to be qualified as a proper Pirate!
We aim for the badges “go with the flow” for using Power Automate Cloud Flows and “The Extstential Risk” for innovative use of AI
