Mates Upcoming Raid…

Eyyrr mates!!! 

Captain Black Bart, is planning a big raid, Oslo is the target and according to ChatGPT the loot potential is #awsome.

Invitation is sent to all chosen crewmates using top of the art “Power Automate flows”. 
Crewmates are free to Accept or Decline the raid invitation.   

When it’s time for the raid, we requires all crew members who have accepted the raid to check in, using their RFID card or phone. 

If the crewmate have accepted more than one raid, he/she can choose to check in using our Raid Planning App 

With the help of Midjourney AI, https://docs.midjourney.com/,  we generated the app design component by asking the AI to provide us witht the latest & greates piracy themes.  
 

The RFID data for all potential crewmates is stored in Azure AD using the “Postal Code” field, pretty “Nasty Hack” according to our “bosun”. 

This data is imported to Dataverse “AAD User” virtual table for our app. 

Let’s raid mates. 

🦀 Crawling the Seven Seas for Hidden Treasure 🏆

The waters run deep and dark when you are at sea. This is why intel like maps, a spyglass and knowing the tides help. To give our pirates an advantage we crawl barenswatch.no for other ships location. But, just as in dogeball, it is not enough to know where something is, you need to know where it is going and when you need to be there to intercept. So how do we calculate the best opportunities and how to intercept them?

Step 1: Crawl and Hijack Valuable Data 🦀

From the system diagram we see our Functions App communicates with the Braenswatch API and stors all boats data including current location. We poll this information several times giving us multiple data points not only for ballast and weight, but also for speed, location and direction (this will be important later).

Screen grab of system model for the Pirate Pillaging app

For ease of use while creating the service we compiled all the data using Power BI Dashboards to see what we were working with. The dashboard can also be used to help captains understand the state of the seven seas, although we appreciate that more data is not necessarily better data. It depends on how you visualize, precent and timing of when to precent data to you user.

Video of Solveig ☀️ explaining how we gather, structure, compile and visualize ship data in a way highlighting opportune targets.

Step 2: Knowing which Opportunities to Strike 🏆

From the ships length, type and destination we can estimate its value. Slow moving ships heading for freight docs are ripe for looting, fast moving ships heading for a dry dock are empty and better left alone. For the sharp eye you have also spotted we have a Threat flag in or model, this is for marking known pirates, intercepters and the fuzz of the seven seas. Once again referencing to Sun Tzu: You always win the wars you never fight (dont search this quote).

Image displaying famous quote from Sun Tzu about fighting good

If we decide that this opportunity is ripe for plundering, we look at its course and if it is feasible to intercept based on our current course. Good opportunities are heavy, slow moving ships. Bad opportunities are war ships of any kind or any opportunity in the vicinity of a war ship.

There is also a certain Goldilocks of opportunities: Based on your own vessel, you do not want to attack another ship that is too large, since it will be a harder fight, or too small, since it carries less booty. We can use the dashboard to visually find the opportune boats and use this investigation to write code that targets these boats automatically.

Captain Mats looking at data The Power BI Dashboard populated with crawled and manipulated data from Barenswatch.no, structured by ship type, weight and destination, highlighting the most opportune ships in the collection

When we flag ships that are both opportune and not adjacent to warships we plot paths to figure out where they are going in relations to our course.

Step 3: Using Advanced Trigonometry to Pinpoint your Enemy 🎯

Youtube can teach you the basics about how to calculate a ships heading based on longitude, latitude, direction, true north and the course deviation.

Youtube is telling us what we need to know about complex

True north will change depending on where on the globe you are positioned, so for intercept courses it is paramount to get this stuff right. You want to be at the place where your target ship will be, not where they have been. And you want to be there before they get there, not to ride up after them and trailing them forever. This is where we use the latitude, longitude, direction and speed from earlier and _try_ to plot the path of the ships. I stress try because Solveig just looked at Sebastian with the “do you mind not being an idiot” when he asked how se was doing it and if it was done. Then just sort of sigh’ed and continued clicking keys. So it works, trust us.

Solveig ☀️ working on triangulating and plotting ships path to calculate intercept courses.

For our challenge we also need to estimate if it is a viable option to change our current corse to intercept the opportunity. For all business cases there is a cost/effect probability calculation. Fo us this depends on added time to destination with deviation and added length of trip, both of which adds to the total cost of provisions and consumables like petrol.

Step 4: Plot Optimal Course for Interception 🏴‍☠️

Finally when we have a list of opportune targets, we know which targets will be adjacent to our planned course form A to B we plot intercept maneuvers. Which, is a topic for another blog post – because this part is most certainly done… Trust us… 🤞🤞🤞

A dirty hack to the first Head2Head challenge?

Okey, we do agree that solution that won is a bit better than this, but that’s why we think this is a dirty hack! Instead of doing what you should do with the challenge, we ended up creating a canvas app that could run offline and add that to the Power Page. It still solves the challenge, doesn’t it?

And, you might as how we got this brilliant idea.. What is more innovative use of AI if it’s not asking these questions you don’t have the answer for yourself? So we asked ChatGPT how we could solve this challenge, and this is the answer we got:

Let the machines work 🤖

We get satellite images from the Sentinel Hub API where we can get images from the coordinates we want. This makes it easy to compare images with AIS data from the same time period and coordinates.

Sattelite image of ships
Counting ships

To identify ships from the satellite images, we need more eyes than we have available. So what’s better than letting the machines do the work.
We have developed our own ML code in Python to recognize ships. This will make it much more efficient than manually reviewing all the images.
Currently, the machine has 99% confidence in recognizing ships. It then creates a heat map that shows where it thinks (with 99% probability) there are ships.
Then it counts the number of ships and sends it on.

Training the model to gain confidence

When we know how many ships are within a given area at a given time, it is easy to compare this with data from AIS.
Here we search through the API of the Norwegian Coastal Administration, where we can count the number of ships within the same time and area.

JSON from AIS data

Using external data in this way provides great value for our customers as the threat at sea is significantly lower 💾🏴‍☠️

The Jackpot App :: Pirates fleet enlisting

The reason we are calling this the jackpot app because we believe that it’s hitting multiple score-points!

The idea of this app is to enlist pirates’ fleet without a single character punch!!! Aided and infused by AI Builder, external API’s and Power Automate Flows; this Power Apps canvas app is state of the art geekiness!

Although, registered vehicles are the last of our worries (as pirates), but hey we thought of doing it anyways 🙂 The app starts by taking a picture of the registration plate of the vehicle. the app will send the picture to the AI Builder through a power automate flow that process the image and read the registration plate and show it on the app for confirmation.

Once the you are satisfied with the result, you can check the vehicle against the national registry and search for more registration information. This indeed does check the car against the Norwegian car registry (Statens vegvesen) show if the vehicle is approved for going on the European roads (EU-godskjenning). Also least of our concerns!

Actual data from Statens Vegvesen
Crawling and datamining data from external sources

Now, after confirming and inserting car data into our system. We need to identify the vehicle type and compare with our collected data in our backend system. However, identify the care is a simple process that doesn’t need any text input from the user. One picture will do while the AI Builder will take care of identifying the type of the vehicle and insert it into our backend system.

Vehicle plate identification flow (calls another flow through HTTP)
AI Builder made as an endpoint (all AI Builder models and flow are on separate environment, as AI Builder
can’t be included in a solution for deployment)

We are using a mix of built-in and custom AI models to achieve efficiency and productivity in an innovative way.

AI Builder model

Using Power Automate to create Pirate Names using OpenAPIs Davinci model

Bacground: When recruiting potential pirates we want to give them proper pirate names. Pirates 365 is a service oriented platform, and the recruitment frontend posts potential new pirates through a Pirate

Our solutions is using Power Automate and the HTTP Trigger to accept new pirates

We are using the OpenAI (independent publisher) step found in Power Automate (https://learn.microsoft.com/en-us/connectors/openaiip/) since it is yet not so easy to get access to Azure Open AI which probably would be better.

We had to do some tuning to get a little bit different answers – at first it just returned “Captain firstname lastname”, but after some tuning by setting down the temperature from 1 to 0,7 and explicity saying not to use the word “captain” we got the desired result.

We are creating the new pirate recruits as Leads in Dataverse so the recruitment team can contact them for possible empolyment

Here is the lead created in Dataverse shown in the “Pirate 365 Manage”, ready to be qualified as a proper Pirate!

We aim for the badges “go with the flow” for using Power Automate Cloud Flows and “The Extstential Risk” for innovative use of AI

Bombshells face an eksistensial risk

Som del av prosjektet med å kartlegge trusler i Oslofjorden er det også viktig å kartlegge artsmangfoldet i fjorden for å vurdere hvilke arter som potensielt er truet og på den måten kartlegge risiko. I sensor-hubene har Bombshells satt inn kamera som trigges av sonarsensoren for å ta bilder av dyrelivet i området. Ved å bruke AI gjenkjenner vi ulike arter av skillpadder som potensielt vil være sårbare for vannforurensning og heve alvorlighetsgraden ved en hendelse.

Vi har trent opp en AI model i Power Apps til og identifisere tre ulike typer skilpadder, landskilpadder, havskilpadder og sumpskilpadder. Vi har matet AI med bilder av tre typer skillpadder i sine respektive miljø.

Vi bruker bilder for og trene modellen minimum 256 x 256 piksler som er kravet for og trene modellen. Bildet nedenfor viser hvordan bildene med ulike typer markeres og kategoriseres.

Treningsgrensesnitt
Landskilpadde
Sumpskillpadde
Havskilpadde

Etter vi har markert alle bildene vi ønsker og bruke som treningsgrunnlag, trener vi modellen og får en ytelse score. Her er den publiserte modellen etter den er ferdig trent.

Til slutt for å teste systemet har vi bygget en PowerApp som holder på en enkel komponent som laster inn et bilde og sender det til AI for identifisering. Resulatet vises i form av at et gjenkjent objekt markeres og betegnes med riktig kategori i grensesnittet, sammen med en score på hvor sikker AI er på at resultatet er riktig.

Kilde til risikoanalyse

Med disse dataene knyttet opp mot lokasjonsdata vil man kunne hente inn og sammenstille beregninger som brukes i risikovurdering av en hendelse.

The True Risk to our Existence

Who can pick up the mantel after we retire? Who is truly a ninja!? We need to recruit the future of TMNT and the best to do it is through AI. We will use AI Builder and a custom model to identify our future ninjas, in return we offer cheap pizza, a full 5% discount on your purchase!

Only the customers masked as a TMNT will be offered the precious discount!

Kjøleskapet kan se pt. 2

Nå har vi bygget videre og kommet opp med et mye mer robust kjøleskap. Flytteesker kan brukes til så mangt 😎 Fortsatt en del testing med forskjellige frukter (man tager hva man haver). Responsen er så som så, men vi er på god vei. Klementin ble tidligere eple (ref. https://acdc.blog/in2022/kjoleskapet-kan-se/), nå er appelsin appelsin, og eple er eple 🎉

Her har vi en appelsin innendørs i et kjøleskap med både vegger og gulv. Imponerende? Yes!!!
Og her er hva som ble tatt bilde av. 1-0 til AI

Oppsettet er som følger:
Webkamera montert i kjøleskapsdør, koblet til Raspberry PI.
Arduino med knapp (les mye frustrasjon med KNAPP!!!!) som trigger kamerabilde ved åpning av kjøleskapsdør.
Arduino styrer i tillegg en trinnmotor som spinner en kul 3D-printet bling på toppen av kjøleskapet (bling kommer, men se bilde).
Alt på Arduino styres av den samme Raspberry PI’en ved hjelp av Python.
Når bilde er tatt, sendes det til Azure Cognitive services for analyse, og responsen kommer som oppdagede objekter. Deretter matcher vi dette med vår egen ingrediensdatabase i Dataverse via en Power Automate flow (kjøres via en HTTP request trigger i flowen) som oppdaterer ingredienser med attributtet “In fridge” = True/False.

Bildet under viser flowen som trigges fra kjøleskaps-raspberryen. Deretter oppdateres status på hva vi har i kjøleskapet. Dette er deilig å slippe og gjøre manuelt.

Her har vi integrert rubbel og bit, og det fungerer!

Det vi frykter nå er at maskinene skal vende seg mot oss, og si at kjøleskapet er tomt til en hver tid. Da kan vi risikere å sulte i hjel foran kjøleskapsdøren.
Men inntil videre velger vi å stole på vår kjære Fridgitoid 9000