A new dawn rises, and bananas are still about. No rest for the wicked, Team Super INtendo is up and more ready than ever to continue its journey to make roads worldwide banana free.
Early bird(o) gets the worm
Author Archives: Super INtendo
Not reinventing the (kart)wheel
Time is of the essence when there is undetected bananas about. We will probably spend some time training our algorithm to detect all sorts of potential road impediments, but in order to rush to market and deliver value as early as possible, we instead grab a mushroom and cut some corners. Much like Boo in Mario Party, we can steal other people’s hard work!
We started by forking the Tensorflow repo to make the basis of our app. Tensorflow is an open-source end-to-end platform for machine learning, and a great starting point for our app. We can use it to create models for machine learning. And while speaking of thieving, Tensorflow is also built on other public packages – also a form of stealing.
Speaking of models, why make your own when there are several detection models we can “borrow”
Mwa ha ha! Surprise, Luigi! It’s-a me, King Boo!
King boo – Luigi’s mansion 3
Others have also created several models for mobile use that we can reuse while we try to train our own. MobileNet model is one of these and can be used straight out of the box. These models are in turn also trained on public data that we can “steal” and modify as we please.
We also “borrow” more tangible things. Like non-licensed photos of bananas for training our detection model and map and geo-services for location data.
TL;DR Thieving is good – at least in programming
Banana detected from working solution!
Yellow alert! Banana detected!
Ever driven down a road, cruising at a stable regular pace, when your car suddenly starts to spin and you lose valuable time going to important places (question mark boxes). Chances are some pesky co-drivers are trying to sabotage your life by throwing banana peel in the road. Mama mia! as Mario himself would have put it.
Mama mia!
Super mario – various games
You are not alone! But fear not, a solution to this coming to your preferred application store. The app will run in the background with your phone mounted in a custom 3d-printed stand easily placed in your car. There will be a live video feed running (the user will not need to look at it for obvious safety reasons) automatically detecting incoming bananas. alerting the driver with a friendy alert urging you to avoid the detected banana peel. Early alert sketch:
Also alerts will be sent to the next of kin, conveying a message that your loved one have been in a banana accident.
System sketch
The project will use a Android app to access camera and GPS functionalities. The app features a Tensorflow model that processes the video feed and labels bananas. Once a banana is detected, multiple events occur. First, the user is alerted through the app by a blinking light and the image of a peeled banana. This helps the user avoid the banana peel. Second, the GPS position is extracted and communicated to Dataverse. Dataverse stores the position of known bananas and they are displayed in a map that is available through Teams. Lastly, multiple alerts are made. In addition to the in-car alert, the system features a next-of-kin alert in case the user hits the banana peel. Moreover, Statens Vegvesen is alerted of the banana position so that it can be cleaned up.
Final delivery 🏴☠️🦜Protector of the Seas
As the captain of a ship, I want to be sure at all times that I will not encounter pirate ships.
This makes me and my crew safer in the work we have to do, which in turn leads to higher efficiency in our deliveries.
This is the user story we had as a starting point.
When we had to build a solution for this, we needed to be creative.
At first we thought we could use satellite images to recognize pirate ships from above. But it’s not that easy. Because flags that hang vertically or worn sails are not necessarily so easy to see from above.
Then we came up with a good idea. Pirates probably do not send information on AIS. But all law-abiding ships do.
If we then compare the number of ships we find in satellite images with the number of ships we find through AIS, within the same geographical area, we find out if any of the ships are pirates 🥳
What we do.
We download satellite images from the Sentinel Hub API, which is an open API for satellite images. We then process the images with our own machine learning model, built from scratch in Python. This is trained to recognize ships at sea.
We get back the number of ships within the selected sector.
Then we check the Norwegian Coast Guard’s API for AIS data within the same coordinates. We get the number of ships from there as well.
We then compare the number of ships from satellite and AIS and store this in an Azure SQL table.
This is our backend solution in a nutshell.
In the front end, we have built a web app in Reajc.js that allows the users, authorities or shipping companies, to check sectors for pirate ships.
All ships in the sector appear on the map, where you can also click on them to read more information about the ship. If pirate ships are discovered, they appear with their own pirate flag symbols. Green needles are police or military vessels.
It is possible to send an alarm to all ships in the area if pirates are detected. A notification is then sent out to the ships that subscribe to our service, which sends a notification via a Teams bot. It tells what the threat is and where it is.
The light on board the bridge will also flash red if you have purchased this additional service.
In Teams, it will be possible to click to read more about the alert and see on the map where it is.
When the coast is clear, the alarm can be removed from the Teams client with a click.
From Teams on board the ship, it will be possible to check if there are pirates at coordinates you provide by sending a message to our bot. It then politely answers yes or no.
If there is something wrong with the message you send, you will also be informed about it in a polite way.
This solution has two potential customer groups – authorities and shipping companies. The authorities want to have safe coastal routes for ships, shipping companies want their ships to be safe.
We have learned a lot through this assignment. Much of what we have created has been built from scratch, precisely to learn as much as possible. But we have also used a lot of off-the-shelf products.
Microsoft Teams and Sharepoint are used to have familiar, simple surfaces for end users. Power Automate is used to be able to easily flash alarm lights.
Otherwise, the rest of the solution is hosted in Azure.
End users – The reason for why we are doing this!💰😀
We have mentioned little about our front end web solution so far. But here is some info.
The interface is made in React.js built with Vite. It is faster than Webpack.⏩
This is a page to be used by authorities, the coast guard or separate centers for shipping companies.
The website is used to scan areas on the map for ships. If a pirate ship is detected, an alarm can be sent to all other ships in the area.
In order to be able to be used on all surfaces, it is of course responsive 😉
Sending an alarm from the web interface will trigger an alarm in Teams clients over on the ships, as well as cause the light on the bridge to flash red.
Best practice as pirate hunters
Since our team is on the law-abiding side of the game, we want to be orderly at all stages.
When new code is written and pushed, we make sure to create pull requests so that a colleague can review the code and ensure that we are on the right track.
For build and deploy we use Github Actions
Transparency is also important to us, which is why our repo is open. We have nothing to hide. Feel free to run a code review on our code 😉
Remarkable team spirit
Since we first set foot on board, th’ atmosphere at Pirates o’ th’ CaribIN be nothin’ short of grand. We arrived on Thursday mornin’ and got everything set up, and since then we’ve been havin’ a blast! We’ve printed everything from pirate ships to gold doubloons, and we’ve lent a hand to our neighbors and gotten help in return. Here be two snaps that show off our mighty fine team spirit. First day, and mornin’ of th’ third day. We were up late last night makin’ improvements to our PoS system, ye scallywags!
When a matey be in dire straits 🆘🏴☠️
Arrr matey! Listen up ye landlubbers, th’ teams round us be workin’ hard n’ fast to come up with grand solutions. Even though we scallywags at Pirates o’ th’ CaribIN be battlin’ against pirates, we must lend a hand to our neighbors when they be in need. Cap’n Hack Sparrow @in2 needed a figurehead to rally his crew, and since our 3D printer be swift and sharp, we took on th’ task with glee.
They sent us a 3D model of a pirate they fancied, which we then sliced and printed with ease. Now they be all smiles n’ joyful, hoisting their mugs high! Yarrr!
Lots of sea lots of shells
Rapid development = rapid testing = a lot of mess in our test environment. We are using PNP PowerShell to set up our demo team to keep it fresh and clean.
