🎙️ ESP32 Real-Time Audio Transcriber

A real-time speech-to-text system that captures audio from an ESP32 + INMP441 microphone, streams it to a cloud-hosted FastAPI server, transcribes it via Deepgram Nova-3, and generates conversational AI responses using Google Gemini 2.5 Flash — all viewable in a sleek live web dashboard.

📸 Product Images

Product Image 1 Product Image 2

ESP32 + INMP441 assembly housed in a custom 3D printed case

🎥 Demo Video

▶ Watch the product demo video

✨ Features

Real-time transcription — Live interim + final transcripts powered by Deepgram Nova-3
AI conversation — Press Q or click Ask Gemini to get a conversational response from Gemini 2.5 Flash with multi-turn context
WiFi captive portal — No hardcoded credentials; configure WiFi via phone/laptop on first boot
One-click deploy — Server deploys to Railway with zero configuration
Live status dashboard — Monitor WebSocket, Deepgram, ESP32, and Gemini connection states in real time
Debug panel — Built-in debug log overlay for troubleshooting

🏗️ Architecture

┌─────────────┐        WSS (binary audio)        ┌──────────────────┐       WSS        ┌───────────┐
│   ESP32 +   │ ──────────────────────────────▶   │  FastAPI Server  │ ──────────────▶  │ Deepgram  │
│   INMP441   │                                   │  (Railway)       │ ◀──────────────  │ Nova-3    │
└─────────────┘                                   │                  │   transcripts    └───────────┘
                                                  │                  │
                                                  │                  │       REST        ┌───────────┐
┌─────────────┐        WSS (text events)          │                  │ ──────────────▶  │  Gemini   │
│   Browser   │ ◀──────────────────────────────   │                  │ ◀──────────────  │  2.5 Flash│
│   Dashboard │ ──────────────────────────────▶   └──────────────────┘   AI response    └───────────┘
└─────────────┘        "PROCESS" / "CLEAR"

📁 Project Structure

audio_transcriber/
├── server.py                        # FastAPI server (transcription + LLM + web UI)
├── esp32_firmware/
│   └── esp32_firmware.ino           # Arduino sketch for ESP32 + INMP441
├── requirements.txt                 # Python dependencies
├── Procfile                         # Railway process definition
├── runtime.txt                      # Python version for Railway
├── .gitignore                       # Ignored files (venv, cache, etc.)
└── README.md                        # This file

🔧 Hardware Requirements

Component	Description
ESP32 Dev Board	Any ESP32-WROOM-32 based board
INMP441	I2S MEMS microphone module
Jumper wires	5 connections (see wiring below)
USB cable	For flashing and serial monitor

Wiring Diagram

ESP32 Pin	INMP441 Pin	Function
`GPIO 26`	`SCK`	Bit Clock
`GPIO 32`	`WS`	Word Select (LRCLK)
`GPIO 33`	`SD`	Serial Data Out
`3.3V`	`VDD`	Power
`GND`	`GND` + `L/R`	Ground (L/R → GND for left channel)

🚀 Getting Started

1. Deploy the Server to Railway

Push this repository to GitHub
Go to railway.app → New Project → Deploy from GitHub Repo
Railway auto-detects the Procfile and runtime.txt — no extra config needed
Note your deployed URL (e.g. audio-transcriber.up.railway.app)

2. Flash the ESP32 Firmware

Open esp32_firmware/esp32_firmware.ino in Arduino IDE
Install the required libraries via Library Manager:
- WiFiManager by tzapu (≥ 2.0)
- WebSockets by Markus Sattler (≥ 2.4)

Update SERVER_HOST in the sketch if your Railway URL differs:

const char* SERVER_HOST = "your-app-name.up.railway.app";

Select board: ESP32 Dev Module
Flash and open the Serial Monitor at 115200 baud

3. Connect to WiFi

On first boot, the ESP32 creates a WiFi AP named Transcriber-Setup
Connect to it from your phone or laptop
Select your home WiFi network and enter the password
The ESP32 saves the credentials and auto-connects on subsequent boots

💡 Reset WiFi: Hold the BOOT button (GPIO 0) while powering on to clear saved credentials.

4. View the Dashboard

Open your Railway URL in a browser:

https://audio-transcriber.up.railway.app

You'll see live transcription appear as the ESP32 streams audio.

🧠 Using Gemini AI

Once transcription lines appear:

Press Q on your keyboard or click the ⬡ Ask Gemini button
Gemini reads the buffered transcript and responds in a casual, conversational tone
The conversation supports multi-turn context — keep talking and asking
Click Reset Chat to clear the conversation history

⚙️ Configuration

Server Environment

Variable	Default	Description
`PORT`	`8000`	HTTP port (auto-set by Railway)

API Keys

API keys are currently embedded in server.py. For production, move them to environment variables:

DEEPGRAM_API_KEY = os.environ.get("DEEPGRAM_API_KEY")
GOOGLE_API_KEY   = os.environ.get("GOOGLE_API_KEY")

ESP32 Firmware

Constant	Default	Description
`SERVER_HOST`	`audio-transcriber.up.railway.app`	Railway domain
`SERVER_PORT`	`443`	HTTPS/WSS port
`WS_PATH`	`/ws/audio`	WebSocket endpoint
`SAMPLE_RATE`	`16000`	Audio sample rate (Hz)
`I2S_READ_LEN`	`256`	Samples per I2S read cycle

🛠️ Local Development

# Clone the repo
git clone https://github.com/<your-username>/audio_transcriber.git
cd audio_transcriber

# Create a virtual environment
python -m venv venv
source venv/bin/activate        # Linux/Mac
# venv\Scripts\activate         # Windows

# Install dependencies
pip install -r requirements.txt

# Run the server locally
python server.py
# → Server starts at http://localhost:8000

Update SERVER_HOST in the ESP32 sketch to your local IP for local testing.

📦 Dependencies

Python (Server)

Package	Purpose
`fastapi`	Async web framework & WebSocket handling
`uvicorn`	ASGI server
`websockets`	Async WebSocket client for Deepgram
`langchain-google-genai`	LangChain integration for Gemini

Arduino (ESP32)

Library	Purpose
`WiFiManager`	Captive portal for WiFi provisioning
`WebSockets`	WebSocket client with SSL support

📝 API Endpoints

Endpoint	Type	Description
`GET /`	HTTP	Serves the live transcription dashboard
`WS /ws`	WebSocket	Browser ↔ Server (transcripts + LLM events)
`WS /ws/audio`	WebSocket	ESP32 → Server (raw 16-bit PCM audio)

Browser WebSocket Messages

Direction	Message	Description
Browser → Server	`PROCESS`	Trigger Gemini on buffered transcript
Browser → Server	`CLEAR_HISTORY`	Reset conversation history
Server → Browser	`__FINAL__:<text>`	Final transcript line
Server → Browser	`__INTERIM__:<text>`	Interim (partial) transcript
Server → Browser	`__LLM_START__`	Gemini processing started
Server → Browser	`__LLM_TOKEN__:<text>`	Gemini response text
Server → Browser	`__LLM_DONE__`	Gemini processing complete
Server → Browser	`__LLM_ERROR__:<msg>`	Gemini error message
Server → Browser	`__STATUS__:<state>`	Connection state change

🤝 Contributing

Fork the repository
Create your feature branch (git checkout -b feature/amazing-feature)
Commit your changes (git commit -m 'Add amazing feature')
Push to the branch (git push origin feature/amazing-feature)
Open a Pull Request

📄 License

This project is open source and available under the MIT License.

Built with ❤️ using ESP32 · FastAPI · Deepgram · Gemini

Downloads last month: -; Downloads are not tracked for this model. How to track

Inference Providers NEW

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support