Delete README.md
Browse files
README.md
DELETED
@@ -1,371 +0,0 @@
|
|
1 |
-
---
|
2 |
-
title: Visual Search System
|
3 |
-
emoji: π
|
4 |
-
colorFrom: blue
|
5 |
-
colorTo: green
|
6 |
-
sdk: streamlit
|
7 |
-
sdk_version: "1.37.0"
|
8 |
-
app_file: app.py
|
9 |
-
pinned: false
|
10 |
-
license: mit
|
11 |
-
---
|
12 |
-
|
13 |
-
|
14 |
-
|
15 |
-
|
16 |
-
# Modern Visual Search System (CLIP-based)
|
17 |
-
|
18 |
-
## Overview
|
19 |
-
|
20 |
-
This project implements a modern, scalable visual search system using OpenAI's CLIP model. It provides both a REST API (FastAPI) and a web interface (Streamlit) for searching large image datasets using natural language queries and image queries, returning the top 5 most visually relevant images.
|
21 |
-
|
22 |
-
## Features
|
23 |
-
|
24 |
-
- **REST API**: FastAPI backend with comprehensive endpoints
|
25 |
-
- **Web Interface**: Streamlit frontend with intuitive UI
|
26 |
-
- **Text Search**: Search images using natural language descriptions
|
27 |
-
- **Image Search**: Find similar images by providing an image as query
|
28 |
-
- **Image Upload**: Upload images to find similar images in the dataset
|
29 |
-
- **Top 5 Results**: Always returns the 5 most similar images
|
30 |
-
- **Fast Search**: Uses precomputed embeddings for optimal performance
|
31 |
-
- **GPU Acceleration**: Automatic GPU detection and utilization
|
32 |
-
- **API Documentation**: Auto-generated Swagger/OpenAPI docs
|
33 |
-
|
34 |
-
## Architecture
|
35 |
-
|
36 |
-
```
|
37 |
-
βββββββββββββββββββ βββββββββββββββββββ βββββββββββββββββββ
|
38 |
-
β Streamlit β β FastAPI β β CLIP Model β
|
39 |
-
β Frontend βββββΊβ Backend βββββΊβ (Encoder) β
|
40 |
-
β (Port 8501) β β (Port 8000) β β β
|
41 |
-
βββββββββββββββββββ βββββββββββββββββββ βββββββββββββββββββ
|
42 |
-
β
|
43 |
-
βΌ
|
44 |
-
βββββββββββββββββββ
|
45 |
-
β Image β
|
46 |
-
β Database β
|
47 |
-
β (images/) β
|
48 |
-
βββββββββββββββββββ
|
49 |
-
```
|
50 |
-
|
51 |
-
## Technical Details
|
52 |
-
|
53 |
-
- **Image & Text Features**: CLIP (ViT-B-32, openai weights)
|
54 |
-
- **Search Algorithm**: Cosine similarity in CLIP embedding space
|
55 |
-
- **Embedding Dimension**: 512 (default for ViT-B-32)
|
56 |
-
- **Performance**: Fast, GPU-accelerated if available
|
57 |
-
- **Dependencies**: `open-clip-torch`, `torch`, `fastapi`, `streamlit`, `Pillow`, `numpy`
|
58 |
-
|
59 |
-
## Quick Start
|
60 |
-
|
61 |
-
### 1. Install Dependencies
|
62 |
-
|
63 |
-
```bash
|
64 |
-
pip install -r requirements.txt
|
65 |
-
```
|
66 |
-
|
67 |
-
### 2. Prepare Your Images
|
68 |
-
|
69 |
-
The application will automatically check and download images for you:
|
70 |
-
|
71 |
-
```bash
|
72 |
-
# Check image status
|
73 |
-
python start_app.py --check-only
|
74 |
-
|
75 |
-
# Download missing images (if any)
|
76 |
-
python start_app.py --download-images
|
77 |
-
|
78 |
-
# Or let the app handle everything automatically
|
79 |
-
python start_app.py
|
80 |
-
```
|
81 |
-
|
82 |
-
**Automatic Features:**
|
83 |
-
- β
Checks if images are already downloaded
|
84 |
-
- β
Downloads missing images in parallel (up to 20 workers)
|
85 |
-
- β
Skips existing images to save time
|
86 |
-
- β
Shows progress and statistics
|
87 |
-
- β
Handles errors gracefully
|
88 |
-
- β
**Smart startup**: Allows application to start with sufficient images (default: 1000+)
|
89 |
-
- β
**Flexible requirements**: Can adjust minimum image requirements
|
90 |
-
- β
**Graceful degradation**: Works even with partial image database
|
91 |
-
|
92 |
-
### 3. Start the Application
|
93 |
-
|
94 |
-
#### Option A: Start Both Services (Recommended)
|
95 |
-
```bash
|
96 |
-
python start_app.py
|
97 |
-
```
|
98 |
-
|
99 |
-
This will:
|
100 |
-
1. Check dependencies
|
101 |
-
2. Verify/download images automatically
|
102 |
-
3. Start both the FastAPI backend and Streamlit frontend
|
103 |
-
|
104 |
-
#### Command Line Options
|
105 |
-
|
106 |
-
```bash
|
107 |
-
# Basic startup (recommended)
|
108 |
-
python start_app.py
|
109 |
-
|
110 |
-
# Skip automatic image download
|
111 |
-
python start_app.py --skip-download
|
112 |
-
|
113 |
-
# Force download images even if some exist
|
114 |
-
python start_app.py --download-images
|
115 |
-
|
116 |
-
# Use custom number of parallel workers
|
117 |
-
python start_app.py --max-workers 10
|
118 |
-
|
119 |
-
# Use custom images directory
|
120 |
-
python start_app.py --images-dir my_images
|
121 |
-
|
122 |
-
# Set custom minimum image requirement
|
123 |
-
python start_app.py --min-images 500
|
124 |
-
|
125 |
-
# Only check image status, don't start services
|
126 |
-
python start_app.py --check-only
|
127 |
-
```
|
128 |
-
|
129 |
-
#### Option B: Start Services Separately
|
130 |
-
|
131 |
-
**Start FastAPI Backend:**
|
132 |
-
```bash
|
133 |
-
cd api && python main.py
|
134 |
-
```
|
135 |
-
|
136 |
-
**Start Streamlit Frontend (in another terminal):**
|
137 |
-
```bash
|
138 |
-
streamlit run streamlit_app.py
|
139 |
-
```
|
140 |
-
|
141 |
-
#### Option C: Download Images Only
|
142 |
-
|
143 |
-
```bash
|
144 |
-
# Download all images
|
145 |
-
python download_images.py
|
146 |
-
|
147 |
-
# Download with custom settings
|
148 |
-
python download_images.py --max-workers 15 --output-dir images
|
149 |
-
|
150 |
-
# Check status only
|
151 |
-
python download_images.py --check-only
|
152 |
-
```
|
153 |
-
|
154 |
-
### 4. Access the Application
|
155 |
-
|
156 |
-
- **Web Interface**: http://localhost:8501
|
157 |
-
- **API Documentation**: http://localhost:8000/docs
|
158 |
-
- **API Health Check**: http://localhost:8000/api/health
|
159 |
-
|
160 |
-
## API Endpoints
|
161 |
-
|
162 |
-
### Core Endpoints
|
163 |
-
|
164 |
-
| Endpoint | Method | Description |
|
165 |
-
|----------|--------|-------------|
|
166 |
-
| `/` | GET | Root endpoint with API info |
|
167 |
-
| `/api/health` | GET | Health check |
|
168 |
-
| `/api/info` | GET | System information |
|
169 |
-
| `/api/images` | GET | List available images |
|
170 |
-
|
171 |
-
### Search Endpoints
|
172 |
-
|
173 |
-
| Endpoint | Method | Description |
|
174 |
-
|----------|--------|-------------|
|
175 |
-
| `/api/search/text` | POST | Text-based search |
|
176 |
-
| `/api/search/image` | POST | Image-based search |
|
177 |
-
| `/api/search/image/upload` | POST | Upload image for search |
|
178 |
-
|
179 |
-
### Example API Usage
|
180 |
-
|
181 |
-
#### Text Search
|
182 |
-
```bash
|
183 |
-
curl -X POST "http://localhost:8000/api/search/text" \
|
184 |
-
-H "Content-Type: application/json" \
|
185 |
-
-d '{"query": "red car", "top_k": 5}'
|
186 |
-
```
|
187 |
-
|
188 |
-
#### Image Search
|
189 |
-
```bash
|
190 |
-
curl -X POST "http://localhost:8000/api/search/image" \
|
191 |
-
-H "Content-Type: application/json" \
|
192 |
-
-d '{"image_path": "images/0001.jpg", "top_k": 5}'
|
193 |
-
```
|
194 |
-
|
195 |
-
#### Image Upload Search
|
196 |
-
```bash
|
197 |
-
curl -X POST "http://localhost:8000/api/search/image/upload" \
|
198 |
-
-F "file=@your_image.jpg" \
|
199 |
-
-F "top_k=5"
|
200 |
-
```
|
201 |
-
|
202 |
-
## Web Interface Features
|
203 |
-
|
204 |
-
### Text Search
|
205 |
-
- Enter natural language descriptions
|
206 |
-
- Examples: "a cat sitting", "red car", "person walking", "building with windows"
|
207 |
-
- Returns images that best match the text description
|
208 |
-
|
209 |
-
### Image Search
|
210 |
-
- Select an image from your dataset
|
211 |
-
- Finds images visually similar to the selected image
|
212 |
-
- Excludes the query image from results
|
213 |
-
|
214 |
-
### Image Upload Search
|
215 |
-
- Upload any image file (JPG, PNG, BMP, TIFF)
|
216 |
-
- Find similar images in your dataset
|
217 |
-
- Real-time processing and results
|
218 |
-
|
219 |
-
### Results Display
|
220 |
-
- **Ranking**: 1-5 (top 5 results)
|
221 |
-
- **Filename**: Name of the image file
|
222 |
-
- **Path**: Full path to the image
|
223 |
-
- **Similarity Score**: Percentage indicating how well the image matches the query
|
224 |
-
- **Image Preview**: Visual display of results
|
225 |
-
|
226 |
-
## Testing
|
227 |
-
|
228 |
-
### Test the API
|
229 |
-
```bash
|
230 |
-
python test_api.py
|
231 |
-
```
|
232 |
-
|
233 |
-
### Test the CLI (Legacy)
|
234 |
-
```bash
|
235 |
-
python test_search.py
|
236 |
-
```
|
237 |
-
|
238 |
-
### Run Demo
|
239 |
-
```bash
|
240 |
-
python demo_search.py
|
241 |
-
```
|
242 |
-
|
243 |
-
## Development
|
244 |
-
|
245 |
-
### Project Structure
|
246 |
-
```
|
247 |
-
visual-search-system/
|
248 |
-
βββ api/
|
249 |
-
β βββ main.py # FastAPI backend
|
250 |
-
βββ src/
|
251 |
-
β βββ models/
|
252 |
-
β βββ multimodal_encoder.py # CLIP encoder
|
253 |
-
βββ images/ # Image database
|
254 |
-
βββ models/
|
255 |
-
β βββ clip_index/ # Precomputed embeddings
|
256 |
-
βββ streamlit_app.py # Streamlit frontend
|
257 |
-
βββ start_app.py # Startup script
|
258 |
-
βββ test_api.py # API tests
|
259 |
-
βββ run_search.py # CLI interface (legacy)
|
260 |
-
βββ requirements.txt # Dependencies
|
261 |
-
```
|
262 |
-
|
263 |
-
### Adding New Features
|
264 |
-
|
265 |
-
1. **New API Endpoints**: Add to `api/main.py`
|
266 |
-
2. **New UI Components**: Add to `streamlit_app.py`
|
267 |
-
3. **New Search Methods**: Extend `src/models/multimodal_encoder.py`
|
268 |
-
|
269 |
-
## Performance
|
270 |
-
|
271 |
-
- **Precomputed Embeddings**: Uses existing embeddings in `models/clip_index/` for faster search
|
272 |
-
- **Real-time Fallback**: Falls back to real-time encoding if precomputed embeddings aren't available
|
273 |
-
- **GPU Acceleration**: Automatically uses GPU if available for faster processing
|
274 |
-
- **Memory Efficient**: Processes images in batches to manage memory usage
|
275 |
-
- **Async Processing**: FastAPI handles concurrent requests efficiently
|
276 |
-
|
277 |
-
## Deployment
|
278 |
-
|
279 |
-
### Hugging Face Spaces (Recommended)
|
280 |
-
|
281 |
-
This application is designed to work seamlessly on Hugging Face Spaces:
|
282 |
-
|
283 |
-
1. **Fork this repository** to your Hugging Face account
|
284 |
-
2. **Create a new Space** on Hugging Face Spaces
|
285 |
-
3. **Connect your forked repository** to the Space
|
286 |
-
4. **The application will automatically**:
|
287 |
-
- Check for downloaded images
|
288 |
-
- Download missing images in parallel
|
289 |
-
- Start the Streamlit interface
|
290 |
-
- Handle all dependencies automatically
|
291 |
-
|
292 |
-
**Features for Hugging Face Spaces:**
|
293 |
-
- β
Automatic image downloading with progress tracking
|
294 |
-
- β
Optimized for Spaces environment (reduced parallel workers)
|
295 |
-
- β
Built-in error handling and recovery
|
296 |
-
- β
No manual setup required
|
297 |
-
- β
Works with the provided `photos_url.csv` dataset
|
298 |
-
|
299 |
-
**Space Configuration:**
|
300 |
-
- **SDK**: Streamlit
|
301 |
-
- **Hardware**: CPU Basic (free tier) or GPU for faster processing
|
302 |
-
- **App File**: `app.py`
|
303 |
-
|
304 |
-
### Docker Deployment
|
305 |
-
|
306 |
-
Create a `Dockerfile`:
|
307 |
-
|
308 |
-
```Dockerfile
|
309 |
-
FROM python:3.9-slim
|
310 |
-
|
311 |
-
WORKDIR /app
|
312 |
-
|
313 |
-
# Install system dependencies
|
314 |
-
RUN apt-get update && apt-get install -y \
|
315 |
-
libgl1-mesa-glx \
|
316 |
-
libglib2.0-0 \
|
317 |
-
&& rm -rf /var/lib/apt/lists/*
|
318 |
-
|
319 |
-
# Copy requirements and install Python dependencies
|
320 |
-
COPY requirements.txt .
|
321 |
-
RUN pip install --no-cache-dir -r requirements.txt
|
322 |
-
|
323 |
-
# Copy application code
|
324 |
-
COPY . .
|
325 |
-
|
326 |
-
# Expose ports
|
327 |
-
EXPOSE 8000 8501
|
328 |
-
|
329 |
-
# Start the application
|
330 |
-
CMD ["python", "start_app.py"]
|
331 |
-
```
|
332 |
-
|
333 |
-
Build and run:
|
334 |
-
```bash
|
335 |
-
docker build -t visual-search .
|
336 |
-
docker run -p 8000:8000 -p 8501:8501 -v $(pwd)/images:/app/images visual-search
|
337 |
-
```
|
338 |
-
|
339 |
-
### Production Deployment
|
340 |
-
|
341 |
-
For production, consider:
|
342 |
-
- Using a production ASGI server like Gunicorn
|
343 |
-
- Setting up proper CORS configuration
|
344 |
-
- Implementing authentication
|
345 |
-
- Using a reverse proxy (nginx)
|
346 |
-
- Setting up monitoring and logging
|
347 |
-
|
348 |
-
## Troubleshooting
|
349 |
-
|
350 |
-
### Common Issues
|
351 |
-
|
352 |
-
- **API Connection Failed**: Make sure the FastAPI server is running on port 8000
|
353 |
-
- **No images found**: Ensure images are in the `images/` directory with `.jpg` extension
|
354 |
-
- **Model loading errors**: Check that all dependencies are installed correctly
|
355 |
-
- **GPU issues**: Install appropriate CUDA version for your GPU
|
356 |
-
- **Memory errors**: Reduce batch size or use CPU-only mode
|
357 |
-
- **Port conflicts**: Change ports in `start_app.py` if needed
|
358 |
-
|
359 |
-
### Debug Mode
|
360 |
-
|
361 |
-
Run with debug information:
|
362 |
-
```bash
|
363 |
-
# API with debug logging
|
364 |
-
cd api && uvicorn main:app --reload --log-level debug
|
365 |
-
|
366 |
-
# Streamlit with debug
|
367 |
-
streamlit run streamlit_app.py --logger.level debug
|
368 |
-
```
|
369 |
-
|
370 |
-
## License
|
371 |
-
MIT
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|