Next-Gen AI Interview: Multilingual and Real-Time Analysis
Tags: devchallenge, twiliochallenge, ai, twilio
Published: 2024-06-23 16:22:30


This is a submission for the [Twilio Challenge ](https://dev.to/challenges/twilio)

## What I Built
The motivation behind this project stems from a deep desire to democratize the interview process and make it more accessible, personalized, and fair. Job interviews can be incredibly stressful, and traditional methods often fail to account for individual differences in language proficiency, communication style, and comfort levels. By harnessing the power of AI and advanced communication technologies, we can create an interview experience that not only assesses a candidate's qualifications but also provides real-time support and feedback. This platform aims to empower job seekers, helping them present their best selves and gain valuable insights to improve their performance. In a world where talent is everywhere but opportunities are not, this project seeks to bridge that gap and bring us closer to a more equitable job market.

So with this motivation, I have created an innovative AI-powered platform that personalizes the interview experience for job candidates. Users can fill out a survey detailing their job position, experience, skills, goals, and preferred language for the interview. Based on their responses, they can choose to conduct the interview via web browser or WhatsApp Business. 

If a user opts for WhatsApp, they will receive interview questions generated by the Gemini API directly on WhatsApp, with the ability to respond and receive follow-up questions in their chosen language. Upon completion, they receive a voice call and a WhatsApp message with AI-generated feedback.

For those who prefer a web browser interview, the platform provides a video interface that opens the user's camera and microphone. Questions appear onscreen, and the AI analyzes body language using TensorFlow and PoseNet. Users receive real-time feedback on their posture and eye contact, can replay their responses, and view live transcripts of their speech. After the interview, they exit the video room and receive detailed feedback.


## Demo
{% embed https://www.youtube.com/watch?v=7nrJ0u7j3jA %}

**Multi Language Support->** Spanish
![Demo Screenshot](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/wjv92oj6fplx29m1vmf9.png)
<!-- Share a link to your app and include some screenshots here. -->

## Source Code
{% embed https://github.com/bilal1718/Twilio_Competition %}
## Twilio and AI
Twilio APIs form the backbone of my project, enabling seamless communication across different channels. Twilio WhatsApp Business API ensures smooth delivery of interview questions and responses, while Twilio Voice API handles voice feedback calls. Twilio Video API powers the browser-based video interviews, integrating with TensorFlow and PoseNet for real-time body language analysis. The Gemini API generates intelligent, context-specific questions and feedback, while Microsoft Text Translator ensures language accessibility. Together, these technologies create a cohesive and intuitive interview experience that is accessible, multilingual, and highly interactive.

## Features

- Personalized Interview Setup
- Multilingual Support
- Dual Interview Modes (WhatsApp Business Mode, Web Browser Mode)
- AI-Generated Questions and Feedback
- Real-Time Body Language Analysis
- Replay and Transcript Features
  - Users can replay their spoken answers to review performance.
  - Live transcription of spoken responses displayed on screen.
- Seamless Integration with Twilio APIs
- Comprehensive Feedback

## Tech Stack

- Frontend: **React JS**
- Backend: **Express JS**
- Styling: **Tailwind CSS**
- AI and ML: **TensorFlow, PoseNet**
- APIs:
    - _Twilio APIs:_
      - **WhatsApp Business API**
      - **Voice API**
      - **Video API**
    - _Gemini API (for intelligent question generation and feedback)_    
    - _Microsoft Text Translator API (for multilingual support)_


## Additional Prize Categories

**Twilio Times Two**: The project uses _Twilio Whatsapp Business API_, _Twilio Voice API_ and _Twilio Video API_.

**Impactful Innovators**: This project is beneficial for job seekers that can use AI-driven, multilingual support and real-time feedback, empowers them to shine and bridging the gap between global talent and opportunities.

**Entertaining Endeavors**: The platform enhances the interview experience by incorporating interactive features like _real-time body language_ analysis and _replayable responses_, making the process both informative and engaging for candidates.


<!-- Don't forget to add a cover image (if you want). -->

<!-- Thanks for participating! →