Category Archives: Pirates of the CaribIN

Final delivery 🏴‍☠️🦜Protector of the Seas

Posted on by

As the captain of a ship, I want to be sure at all times that I will not encounter pirate ships.
This makes me and my crew safer in the work we have to do, which in turn leads to higher efficiency in our deliveries.

This is the user story we had as a starting point.

When we had to build a solution for this, we needed to be creative.
At first we thought we could use satellite images to recognize pirate ships from above. But it’s not that easy. Because flags that hang vertically or worn sails are not necessarily so easy to see from above.
Then we came up with a good idea. Pirates probably do not send information on AIS. But all law-abiding ships do.
If we then compare the number of ships we find in satellite images with the number of ships we find through AIS, within the same geographical area, we find out if any of the ships are pirates 🥳

Satellite image of ships

What we do.
We download satellite images from the Sentinel Hub API, which is an open API for satellite images. We then process the images with our own machine learning model, built from scratch in Python. This is trained to recognize ships at sea.
We get back the number of ships within the selected sector.
Then we check the Norwegian Coast Guard’s API for AIS data within the same coordinates. We get the number of ships from there as well.

We then compare the number of ships from satellite and AIS and store this in an Azure SQL table.

This is our backend solution in a nutshell.

Pirate tracker 3000 – web front end

In the front end, we have built a web app in Reajc.js that allows the users, authorities or shipping companies, to check sectors for pirate ships.
All ships in the sector appear on the map, where you can also click on them to read more information about the ship. If pirate ships are discovered, they appear with their own pirate flag symbols. Green needles are police or military vessels.

Ships in the sector

It is possible to send an alarm to all ships in the area if pirates are detected. A notification is then sent out to the ships that subscribe to our service, which sends a notification via a Teams bot. It tells what the threat is and where it is.
The light on board the bridge will also flash red if you have purchased this additional service.
In Teams, it will be possible to click to read more about the alert and see on the map where it is.
When the coast is clear, the alarm can be removed from the Teams client with a click.

Alarm light

From Teams on board the ship, it will be possible to check if there are pirates at coordinates you provide by sending a message to our bot. It then politely answers yes or no.
If there is something wrong with the message you send, you will also be informed about it in a polite way.

Notification on alarm
Check sector
Clear alarm
Intranet

This solution has two potential customer groups – authorities and shipping companies. The authorities want to have safe coastal routes for ships, shipping companies want their ships to be safe.

We have learned a lot through this assignment. Much of what we have created has been built from scratch, precisely to learn as much as possible. But we have also used a lot of off-the-shelf products.
Microsoft Teams and Sharepoint are used to have familiar, simple surfaces for end users. Power Automate is used to be able to easily flash alarm lights.
Otherwise, the rest of the solution is hosted in Azure.

Overall architecture
Starting plan

End users – The reason for why we are doing this!💰😀

Posted on by

We have mentioned little about our front end web solution so far. But here is some info.
The interface is made in React.js built with Vite. It is faster than Webpack.⏩

This is a page to be used by authorities, the coast guard or separate centers for shipping companies.
The website is used to scan areas on the map for ships. If a pirate ship is detected, an alarm can be sent to all other ships in the area.
In order to be able to be used on all surfaces, it is of course responsive 😉

Big Screen
Small screen

Sending an alarm from the web interface will trigger an alarm in Teams clients over on the ships, as well as cause the light on the bridge to flash red.

Best practice as pirate hunters

Posted on by

Since our team is on the law-abiding side of the game, we want to be orderly at all stages.
When new code is written and pushed, we make sure to create pull requests so that a colleague can review the code and ensure that we are on the right track.

For build and deploy we use Github Actions

Transparency is also important to us, which is why our repo is open. We have nothing to hide. Feel free to run a code review on our code 😉

Remarkable team spirit

Posted on by

Since we first set foot on board, th’ atmosphere at Pirates o’ th’ CaribIN be nothin’ short of grand. We arrived on Thursday mornin’ and got everything set up, and since then we’ve been havin’ a blast! We’ve printed everything from pirate ships to gold doubloons, and we’ve lent a hand to our neighbors and gotten help in return. Here be two snaps that show off our mighty fine team spirit. First day, and mornin’ of th’ third day. We were up late last night makin’ improvements to our PoS system, ye scallywags!

A jolly group of pirate trackers
We were having a blast last night. Our work space shows it well

When a matey be in dire straits 🆘🏴‍☠️

Posted on by

Arrr matey! Listen up ye landlubbers, th’ teams round us be workin’ hard n’ fast to come up with grand solutions. Even though we scallywags at Pirates o’ th’ CaribIN be battlin’ against pirates, we must lend a hand to our neighbors when they be in need. Cap’n Hack Sparrow @in2 needed a figurehead to rally his crew, and since our 3D printer be swift and sharp, we took on th’ task with glee.

3D model

They sent us a 3D model of a pirate they fancied, which we then sliced and printed with ease. Now they be all smiles n’ joyful, hoisting their mugs high! Yarrr!

Yarr – CaptaIN in the makin’
Soo joyful

Let the machines work 🤖

Posted on by

We get satellite images from the Sentinel Hub API where we can get images from the coordinates we want. This makes it easy to compare images with AIS data from the same time period and coordinates.

Sattelite image of ships
Counting ships

To identify ships from the satellite images, we need more eyes than we have available. So what’s better than letting the machines do the work.
We have developed our own ML code in Python to recognize ships. This will make it much more efficient than manually reviewing all the images.
Currently, the machine has 99% confidence in recognizing ships. It then creates a heat map that shows where it thinks (with 99% probability) there are ships.
Then it counts the number of ships and sends it on.

Training the model to gain confidence

When we know how many ships are within a given area at a given time, it is easy to compare this with data from AIS.
Here we search through the API of the Norwegian Coastal Administration, where we can count the number of ships within the same time and area.

JSON from AIS data

Using external data in this way provides great value for our customers as the threat at sea is significantly lower 💾🏴‍☠️

They see me flowin’ 🏴‍☠️🦜

Posted on by

When we need to merge things together, what could be easier than using Power Automate.
We have seen that there is a need for visual warning on board ships to be ready if pirates are coming.

Power Automate flow for Hue lights
Triggered 🤔


Therefore, we have connected a Zigbee light chain to a Philips Hue base, on which we use a Power Automate flow to activate a red alarm light.
This is done by calling an endpoint with ALERT in the body.
When the coast is clear again, we can send a “no danger” message and the light will return to green.
It is solved by calling the same endpoint with CLEAR.

Some code for sending alerts to Teams and lights

To make this work, we have connected a Raspberry Pi to the same local network as Philips Hue. We have used a Philips Hue connector from an independent developer.
Fine with everything that is ready-made from 3rd parties 😃

Raspberry PI connecting it all together

What do we do? 🤔

Posted on by

To find out if there are pirates in the waters where commercial ships operate, we will compare data from AIS (Automatic Identification System) with satellite images from the same area.
If there are more ships in the satellite images than the AIS data shows, we will trigger an alarm. This alarm alerts via flashing lights (Zigbee), Teams and an app we build ourselves.

High level architecture

The image analysis itself is carried out using the ResNet architecture, which Herman has written himself with Python. We get the satellite images from open APIs, the same with AIS data.

Example of sattelite image with ships
Training the machine to identify ships in images
Response from AIS API

The comparison takes place by counting the number of ships on the satellite image within given coordinates, then we compare that number with the total number of ships we find through AIS. If the number is unequal, we assume that there are pirates in the area.
This finding is posted in a table in Dataverse, where we have a Power Automate flow that picks up new rows and sends them to our Teams app in addition to triggering a red warning light in Philips Hue.

We also have a React web app that presents data to the emergency response team that keeps track and notifies the right authorities and ships.