Dude, where’s my dog?

Ever wondered where your dog are when there’s no food around? How about asking Cortana, Siri, Alexa, or even better Boten Anna?

We have created an integrated solution, with CI/CD in Azure Devops, Cognitive Services, Azure Functions and Power Virtual Agent.

The solution starts with an Azure Function written in Visual Studio.

When code is completed, committed and Pull Requested into the main branch, our CICD pipeline (YML) fires and pushes the solution to Azure Functions

The Azure function is running with a set of pre-captured surveillance images (Due to GDPR we are not using live video). Each folder consists of three photos, one from the kitchen, one from the hallway and one from the livingroom.

Azure Blob storage

Each of these photos will be evaluated with the cognitive service vision functionality. Each of the evaulated photos will be returned with an url for the user to check, and with a textual answer to where the dog is actually located.


Boten Anna

The evaluated images:

https://acdc2021storage.blob.core.windows.net/dogimages/DogOrder/03/gang.jpeg
https://acdc2021storage.blob.core.windows.net/dogimages/DogOrder/03/kjokken.jpeg
https://acdc2021storage.blob.core.windows.net/dogimages/DogOrder/03/stue.jpeg

With this solution we claim the following badges:

  1. Power User Love – for usage of the Power Virtual Agent
  2. Stairway to Heaven – For the usage of Azure Devops, Azure Functions, Cognitive Service, Azure Storage, Power Virtual Agent
  3. Power of the shell – for the CI/CD YML publishing
  4. The Existential Risk – for usage of AI through Cognitive Service Vision
Power User Love
Stairway to heaven
power of the shell
the existential risk

Family Engagement API

Enabling external developers to utilize the “Family Engagement Platform”

Technologies:

  • Azure API Managment
    • Api
    • Products
    • Subscriptions
  • Azure Function Apps, with httptriggers
    • Key vault
    • ADFS Application with Auth and setting of access in the Contoso AD
  • GIT
  • Web application
  • Azure DevOps
    • Piplines
    • Deploys

Repo in Azure DevOps
Connected to a Pipeline and Release
Deployes to a Azur Function App, With HTTP Triggers
Client Id and Secret is stored in Key Vault
Howto referance Key Valut value from Configuration
Created an AD Identity to talk to KeyVault
Key Vault
The function App is given access to the Key Vault
And the 2 values in the key vault

Contoso Domain

App registert for the function App to access the Contoso Domain

Permissions given for the app in App Registrations
You need to give Admin Consent to the permissions under Enterprise Applications

API Managment

Base information about the backend this api is calling. For now there is no authorization
The API is defined in API Managment (APIM)
Setting a parameter for all endpoints, the value is stored under Named Values.
The code value can be obtained from the Function App, using the Master key you can use the same Code for all functions in that app.
Define the products that should be available in the API
Create a user that is given a way to access your API
Her we can see that the user Kai has been granted access to the product Family

Then we wil try to call the API

This is about it, now we have an API exposed through APIM backed by a Function App reacting on HTTPTriggers.

The function app uses the Graph API to read Groups and Members in the Contoso domain from the Slevik Vel dominan.

We are wrapping Groups and Users as Family and Family Members.

External Apps

The Family Engagement API is available for remote applications and apps that want to develop and use the family engagement platform. As an example: The family engagement web application is intend to use the API to fetch family members and calendar events from the Microsoft Graph API and statistics from The Family Graph API.

CI/CD

Family engagement APP inside Microsoft Teams

Family engagement APP from browser

We also created a React app based on Fluent UI (microsoft.com)

Power Chess one step further

API og Azure Cognitive Services

Gjennom å integrere oss mot SharePoint, function app og ACS (Azure Cognitive Services)

I den tekniske løsningen vår benytter vi en integrasjon mot SharePoint, for å detektere at en ny fil har blitt lagt til, samt uthenting av denne filens innhold.

Videre har vi en integrasjon mot Function App som vi kjører et POST-kall mot tjenesten som gir oss 64 objekter tilbake.

Disse 64 objektene sendes til Azure Cognitive Services for analysering av innholdet.

Deretter blir dataen(Sjakkbrettet) prosessert og sendt til DataVerse ved en create.

Flow:

Ved å bruke Power Automate sin Cloud Flow har vi opprettet en Flow som trigges ved opprettelse av en ny fil i SharePoint i en mappe vi har kalt “Moves”. Ved å bruke en standard funksjon i Flow som henter innholdet i filen, får vi en string som vi kan sende til function app.

CRM / Dataverse

CRM gir oss en visuell fremstilling av sjakktrekkene og gir oss et digitalt sjakkbrett for å verifisere at AI har plukket opp riktig sjakktrekk.

Visuell fremstilling av sjakktrekkene
Digital fremstilling av sjakkbrettet som vil vise siste gjennomført sjakktrekk

Function-app

Function appen mottar en string gjennom HTTP protokollen og deretter kutter opp bildet i 64 objekter, samt legger metadata på hver enkelt objekt for å identifisere hvor plasseringen er. Deretter blir disse sendt tilbake til flowen gjennom HTTP.

Power BI

Vi har påbegynt Power BI rapporten som henter ut data fra hvert sjakkparti. Fra rapporten under ser dere hvem som spiller mot hverandre, vinnersjanse, sjakktrekk og lokasjonen til Fredrik og Mathias.

Rapporten er under utvikling

Lego-armen

Vi har begynt å bygge armen som skal flytte legobrikkene / Sjakkbrikkene. Armen er enda ikke komplett som dere ser, men vi har fått armen til å bevege seg.

Badges vi claimer: