ranveerphakade/ISL-Sign-Lang-Detection

Indian Sign Language Detection with IoT Integration

A real-time Indian Sign Language (ISL) detection system using TensorFlow, MediaPipe, and ESP32-CAM. This project can recognize hand gestures for alphabets (A-Z) and numbers (0-9) in real-time using either a webcam or ESP32-CAM, with results displayed on an OLED screen.

Features

• Real-time hand gesture recognition
• Support for 36 different signs (A-Z, 0-9)
• Word suggestions based on detected gestures
• IoT integration with ESP32-CAM
• OLED display output
• Support for both single and double-handed gestures
• ~98% accuracy on test data

System Architecture

1. Image Capture:

   • ESP32-CAM captures images
   • Sends via HTTP POST to Flask server

2. Processing Server:

   • Flask server receives images
   • Processes using MediaPipe for hand landmark detection
   • Uses TensorFlow model for gesture recognition

3. Display:

   • Results sent back to ESP32
   • Displayed on SSD1306 OLED screen
   • Shows gesture and confidence score

Requirements

Software

• Python 3.8 or higher
• Arduino IDE for ESP32
• Required Python packages (see requirements.txt)

Hardware

• ESP32-CAM module
• SSD1306 OLED Display (128x64)
• USB-TTL converter for ESP32 programming
• Connecting wires
• 5V power supply

Installation

1. Clone the repository:

git clone https://github.com/ranveerphakade/ISL-sign-lang-detection.git
cd ISL-sign-lang-detection

2. Create and activate a virtual environment:

# On Windows
python -m venv venv
venv\Scripts\activate

# On Linux/Mac
python3 -m venv venv
source venv/bin/activate

3. Install required packages:

pip install -r requirements.txt

4. Configure ESP32:
   • Open esp32_code/esp32_cam_oled.ino in Arduino IDE
   • Install required libraries:
     • ESP32 board support
     • Adafruit SSD1306
     • Adafruit GFX
   • Update WiFi credentials and server IP
   • Upload to ESP32-CAM

Project Structure

ISL-sign-lang-detection/
├── flask_server/
│   ├── app.py              # Flask server
│   ├── static/             # Static files
│   ├── templates/          # HTML templates
│   └── test_client.py      # Test script
├── esp32_code/
│   └── esp32_cam_oled.ino  # ESP32 code
├── dummy_input/            # Test images
├── gesture_model.h5        # Trained model
├── train.py               # Training script
├── detect.py              # Webcam detection script
├── process.py             # Data processing
└── README.md

Usage

Server Setup

1. Start the Flask server:

cd flask_server
python app.py

Testing Without Hardware

1. Add test images to dummy_input/ directory
2. Run the test client:

cd flask_server
python test_client.py

Hardware Setup

1. Connect OLED display to ESP32-CAM:

   • VCC → 5V
   • GND → GND
   • SCL → GPIO22
   • SDA → GPIO21

2. Power up the ESP32-CAM

3. The system will automatically:

   • Connect to WiFi
   • Start capturing images
   • Display results on OLED

Model Architecture

The system uses a deep neural network with:

• Input layer: Hand landmark features
• Hidden layers with dropout and batch normalization
• Output layer: 36 classes (A-Z, 0-9)

Performance

• Training accuracy: ~98%
• Real-time detection with confidence scores
• Gesture stability checking to prevent false detections

Contributing

Feel free to contribute to this project by:

1. Fork the repository
2. Create your feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

License

This project is licensed under the MIT License - see the LICENSE file for details.

Acknowledgments

• TensorFlow team for the deep learning framework
• MediaPipe team for the hand landmark detection system
• OpenCV team for the computer vision tools
• ESP32 community for IoT support