After the student is allocated to the new Faculty, the Wayfinder Academy provides a hyper care by allowing students to verbally communicate with the voice digital twin of their mentor.
From the technology part, we are using:
1. LiveKit – to handle real-time audio communication. Students join room via the LiveKit SDK embedded in the Next.js frontend.
2. ChatGPT Voice Assistant – to process voice inputs using a pipeline:
– ASR (speech-to-text)
- LLM (GPT-4) to generate intelligent responses
- TTS text-to-speech
- STT to process audio streams and convert them to text
3.Next.js application – serves as the frontend:
- SSR ensures fast loading and API integration
- connects students to LiveKit rooms and the assistant, displaying responses or playing them as audio
Here are more details on the backend part:
- Transcription Handler:
an entrypoint function to connect to LiveKit Room, track subscriptions and initialise the assistant:
