upload cac thu

.gitignore

@@ -59,6 +59,7 @@ out/
 tests/
 
 Admin_bot/
+User_test_bot/
 Pix-Agent/
 
 # Hugging Face Spaces
@@ -80,6 +81,3 @@ Thumbs.db
 main.py
 
 test/
-
-/tmp
-/docs/

README.md

@@ -416,145 +416,4 @@ Lịch sử hội thoại người dùng được lưu trong queue riêng với
 
 ## Tác giả
 
-- **PIX Project Team**
-
-# PixAgent PDF Processing
-
-This README provides instructions for the PDF processing functionality in PixAgent, including uploading PDF documents, managing vector embeddings, and deleting documents.
-
-## API Endpoints
-
-### Health Check
-
-```
-GET /health
-GET /pdf/health
-```
-
-Verify the API is running and the connection to databases (MongoDB, PostgreSQL, Pinecone) is established.
-
-### Upload PDF
-
-```
-POST /pdf/upload
-```
-
-**Parameters:**
-- `file`: The PDF file to upload (multipart/form-data)
-- `namespace`: The namespace to store vectors in (default: "Default")
-- `mock_mode`: Set to "true" or "false" (default: "false")
-- `vector_database_id`: The ID of the vector database to use (required for real mode)
-- `document_id`: Optional custom document ID (if not provided, a UUID will be generated)
-
-**Example Python Request:**
-```python
-import requests
-import uuid
-
-document_id = str(uuid.uuid4())
-files = {'file': open('your_document.pdf', 'rb')}
-response = requests.post(
-    'http://localhost:8000/pdf/upload',
-    files=files,
-    data={
-        'namespace': 'my-namespace',
-        'mock_mode': 'false',
-        'vector_database_id': '9',
-        'document_id': document_id
-    }
-)
-print(f'Status: {response.status_code}')
-print(f'Response: {response.json()}')
-```
-
-### List Documents
-
-```
-GET /pdf/documents
-```
-
-**Parameters:**
-- `namespace`: The namespace to retrieve documents from
-- `vector_database_id`: The ID of the vector database to use
-
-**Example Python Request:**
-```python
-import requests
-
-response = requests.get(
-    'http://localhost:8000/pdf/documents',
-    params={
-        'namespace': 'my-namespace',
-        'vector_database_id': '9'
-    }
-)
-print(f'Status: {response.status_code}')
-print(f'Documents: {response.json()}')
-```
-
-### Delete Document
-
-```
-DELETE /pdf/document
-```
-
-**Parameters:**
-- `document_id`: The ID of the document to delete
-- `namespace`: The namespace containing the document
-- `vector_database_id`: The ID of the vector database
-
-**Example Python Request:**
-```python
-import requests
-
-response = requests.delete(
-    'http://localhost:8000/pdf/document',
-    params={
-        'document_id': 'your-document-id',
-        'namespace': 'my-namespace',
-        'vector_database_id': '9'
-    }
-)
-print(f'Status: {response.status_code}')
-print(f'Result: {response.json()}')
-```
-
-### List Available Vector Databases
-
-```
-GET /postgres/vector-databases
-```
-
-**Example Python Request:**
-```python
-import requests
-
-response = requests.get('http://localhost:8000/postgres/vector-databases')
-vector_dbs = response.json()
-print(f'Available vector databases: {vector_dbs}')
-```
-
-## PDF Processing and Vector Embedding
-
-The system processes PDFs in the following steps:
-
-1. **Text Extraction**: Uses `PyPDFLoader` from LangChain to extract text from the PDF.
-2. **Text Chunking**: Splits the text into manageable chunks using `RecursiveCharacterTextSplitter` with a chunk size of 1000 characters and 100 character overlap.
-3. **Embedding Creation**: Uses Google's Gemini embedding model (`models/embedding-001`) to create embeddings for each text chunk.
-4. **Dimension Adjustment**: Ensures the embedding dimensions match the Pinecone index requirements:
-   - If Gemini produces 768-dim embeddings and Pinecone expects 1536-dim, each value is duplicated.
-   - For other mismatches, appropriate padding or truncation is applied.
-5. **Vector Storage**: Uploads the embeddings to Pinecone in the specified namespace.
-
-## Notes
-
-- **Mock Mode**: When `mock_mode` is set to "true", the system simulates the PDF processing without actually creating or storing embeddings.
-- **Namespace Handling**: When using a vector database ID, the namespace is automatically formatted as `vdb-{vector_database_id}`.
-- **Error Handling**: The system validates vector dimensions and handles errors appropriately, with detailed logging.
-- **PDF Storage**: Processed PDFs are stored in the `pdf_storage` directory with the document ID as the filename.
-
-## Troubleshooting
-
-- **Dimension Mismatch Error**: If you receive an error about vector dimensions not matching Pinecone index configuration, check that the embedding model and Pinecone index dimensions are compatible. The system will attempt to adjust dimensions but may encounter limits.
-- **Connection Issues**: Verify that MongoDB, PostgreSQL, and Pinecone credentials are correctly configured in the environment variables.
-- **Processing Failures**: Check the `pdf_api_debug.log` file for detailed error messages and processing information. 
+- **PIX Project Team** 

User_test_bot

@@ -0,0 +1 @@
+Subproject commit a29ee5023e8346d48cae7104bf516b78f791db07

app.py

@@ -6,11 +6,6 @@ import os
 import sys
 import logging
 from dotenv import load_dotenv
-from fastapi.responses import JSONResponse, PlainTextResponse
-from fastapi.staticfiles import StaticFiles
-import time
-import uuid
-import traceback
 
 # Cấu hình logging
 logging.basicConfig(
@@ -86,9 +81,8 @@ try:
     from app.api.mongodb_routes import router as mongodb_router
     from app.api.postgresql_routes import router as postgresql_router
     from app.api.rag_routes import router as rag_router
-    from app.api.websocket_routes import router as websocket_router
     from app.api.pdf_routes import router as pdf_router
-    from app.api.pdf_websocket import router as pdf_websocket_router
+    from app.api.websocket_routes import router as websocket_router
     
     # Import middlewares
     from app.utils.middleware import RequestLoggingMiddleware, ErrorHandlingMiddleware, DatabaseCheckMiddleware
@@ -99,8 +93,6 @@ try:
     # Import cache
     from app.utils.cache import get_cache
     
-    logger.info("Successfully imported all routers and modules")
-    
 except ImportError as e:
     logger.error(f"Error importing routes or middlewares: {e}")
     raise
@@ -137,14 +129,6 @@ app.include_router(postgresql_router)
 app.include_router(rag_router)
 app.include_router(websocket_router)
 app.include_router(pdf_router)
-app.include_router(pdf_websocket_router)
-
-# Log all registered routes
-logger.info("Registered API routes:")
-for route in app.routes:
-    if hasattr(route, "path") and hasattr(route, "methods"):
-        methods = ",".join(route.methods)
-        logger.info(f"  {methods:<10} {route.path}")
 
 # Root endpoint
 @app.get("/")
@@ -217,71 +201,23 @@ if DEBUG:
     @app.get("/debug/errors")
     def debug_errors(limit: int = 10):
         """Hiển thị các lỗi gần đây (chỉ trong chế độ debug)"""
-        return error_tracker.get_errors(limit=limit)
+        return error_tracker.get_recent_errors(limit)
     
     @app.get("/debug/performance")
     def debug_performance():
-        """Hiển thị thông tin hiệu suất (chỉ trong chế độ debug)"""
-        return performance_monitor.get_report()
+        """Hiển thị thống kê hiệu suất (chỉ trong chế độ debug)"""
+        return performance_monitor.get_stats()
     
     @app.get("/debug/full")
     def debug_full_report(request: Request):
-        """Hiển thị báo cáo debug đầy đủ (chỉ trong chế độ debug)"""
+        """Hiển thị báo cáo đầy đủ về hệ thống (chỉ trong chế độ debug)"""
         return debug_view(request)
     
     @app.get("/debug/cache")
     def debug_cache():
-        """Hiển thị thông tin chi tiết về cache (chỉ trong chế độ debug)"""
-        cache = get_cache()
-        cache_stats = cache.stats()
-        
-        # Thêm thông tin chi tiết về các key trong cache
-        cache_keys = list(cache.cache.keys())
-        history_users = list(cache.user_history_queues.keys())
-        
-        return {
-            "stats": cache_stats,
-            "keys": cache_keys,
-            "history_users": history_users,
-            "config": {
-                "ttl": cache.ttl,
-                "cleanup_interval": cache.cleanup_interval,
-                "max_size": cache.max_size,
-                "history_queue_size": os.getenv("HISTORY_QUEUE_SIZE", "10"),
-                "history_cache_ttl": os.getenv("HISTORY_CACHE_TTL", "3600"),
-            }
-        }
-    
-    @app.get("/debug/websocket-routes")
-    def debug_websocket_routes():
-        """Hiển thị thông tin về các WebSocket route (chỉ trong chế độ debug)"""
-        ws_routes = []
-        for route in app.routes:
-            if "websocket" in str(route.__class__).lower():
-                ws_routes.append({
-                    "path": route.path,
-                    "name": route.name,
-                    "endpoint": str(route.endpoint)
-                })
-        return {
-            "websocket_routes": ws_routes,
-            "total_count": len(ws_routes)
-        }
-        
-    @app.get("/debug/mock-status")
-    def debug_mock_status():
-        """Display current mock mode settings"""
-        # Import was: from app.api.pdf_routes import USE_MOCK_MODE
-        # We've disabled mock mode
-        
-        return {
-            "mock_mode": False,  # Disabled - using real database
-            "mock_env_variable": os.getenv("USE_MOCK_MODE", "false"),
-            "debug_mode": DEBUG
-        }
-
+        """Hiển thị thống kê về cache (chỉ trong chế độ debug)"""
+        return get_cache().stats()
 
-# Run the app with uvicorn when executed directly
 if __name__ == "__main__":
-    port = int(os.environ.get("PORT", 7860))
-    uvicorn.run("app:app", host="0.0.0.0", port=port, reload=DEBUG) 
+    PORT = int(os.getenv("PORT", "7860"))
+    uvicorn.run("app:app", host="0.0.0.0", port=PORT, reload=DEBUG) 

app/__init__.py

@@ -10,11 +10,16 @@ import os
 # Thêm thư mục gốc vào sys.path
 sys.path.append(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
 
-# Sử dụng importlib để tránh circular import
-import importlib.util
-spec = importlib.util.spec_from_file_location("app_module", 
-                                             os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), 
-                                                         "app.py"))
-app_module = importlib.util.module_from_spec(spec)
-spec.loader.exec_module(app_module)
-app = app_module.app 
+try:
+    # Sửa lại cách import đúng - 'app.py' không phải là module hợp lệ
+    # 'app' là tên module, '.py' là phần mở rộng tệp
+    from app import app
+except ImportError:
+    # Thử cách khác nếu import trực tiếp không hoạt động
+    import importlib.util
+    spec = importlib.util.spec_from_file_location("app_module", 
+                                                 os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), 
+                                                             "app.py"))
+    app_module = importlib.util.module_from_spec(spec)
+    spec.loader.exec_module(app_module)
+    app = app_module.app 

app/api/pdf_routes.py

@@ -1,23 +1,16 @@
 import os
 import shutil
 import uuid
-import sys
-import traceback
-from fastapi import APIRouter, UploadFile, File, Form, HTTPException, BackgroundTasks, Depends, Query
+from fastapi import APIRouter, UploadFile, File, Form, HTTPException, BackgroundTasks, Depends
 from fastapi.responses import JSONResponse
 from typing import Optional, List, Dict, Any
 from sqlalchemy.orm import Session
-import os.path
-import logging
-import tempfile
-import time
-import json
-from datetime import datetime
 
 from app.utils.pdf_processor import PDFProcessor
 from app.models.pdf_models import PDFResponse, DeleteDocumentRequest, DocumentsListResponse
 from app.database.postgresql import get_db
-from app.database.models import VectorDatabase, Document, VectorStatus, ApiKey, DocumentContent
+from app.database.models import VectorDatabase, Document, VectorStatus, DocumentContent
+from datetime import datetime
 from app.api.pdf_websocket import (
     send_pdf_upload_started, 
     send_pdf_upload_progress, 
@@ -28,177 +21,21 @@ from app.api.pdf_websocket import (
     send_pdf_delete_failed
 )
 
-# Setup logger
-logger = logging.getLogger(__name__)
-
-# Add a stream handler for PDF debug logging
-pdf_debug_logger = logging.getLogger("pdf_debug_api")
-pdf_debug_logger.setLevel(logging.DEBUG)
-
-# Check if a stream handler already exists, add one if not
-if not any(isinstance(h, logging.StreamHandler) for h in pdf_debug_logger.handlers):
-    stream_handler = logging.StreamHandler(sys.stdout)
-    stream_handler.setLevel(logging.INFO)
-    pdf_debug_logger.addHandler(stream_handler)
-
-# Initialize router
+# Khởi tạo router
 router = APIRouter(
     prefix="/pdf",
     tags=["PDF Processing"],
 )
 
-# Constants - Use system temp directory instead of creating our own
-TEMP_UPLOAD_DIR = tempfile.gettempdir()
-STORAGE_DIR = tempfile.gettempdir()  # Also use system temp for storage
-
-USE_MOCK_MODE = False  # Disabled - using real database with improved connection handling
-logger.info(f"PDF API starting with USE_MOCK_MODE={USE_MOCK_MODE}")
-
-# Helper function to log with timestamp
-def log_with_timestamp(message: str, level: str = "info", error: Exception = None):
-    """Add timestamps to log messages and log to the PDF debug logger if available"""
-    timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
-    full_message = f"{timestamp} - {message}"
-    
-    if level.lower() == "debug":
-        logger.debug(full_message)
-        pdf_debug_logger.debug(full_message)
-    elif level.lower() == "info":
-        logger.info(full_message)
-        pdf_debug_logger.info(full_message)
-    elif level.lower() == "warning":
-        logger.warning(full_message)
-        pdf_debug_logger.warning(full_message)
-    elif level.lower() == "error":
-        logger.error(full_message)
-        pdf_debug_logger.error(full_message)
-        if error:
-            logger.error(traceback.format_exc())
-            pdf_debug_logger.error(traceback.format_exc())
-    else:
-        logger.info(full_message)
-        pdf_debug_logger.info(full_message)
+# Thư mục lưu file tạm - sử dụng /tmp để tránh lỗi quyền truy cập
+TEMP_UPLOAD_DIR = "/tmp/uploads/temp"
+STORAGE_DIR = "/tmp/uploads/pdfs"
 
-# Helper function to log debug information during upload
-def log_upload_debug(correlation_id: str, message: str, error: Exception = None):
-    """Log detailed debug information about PDF uploads"""
-    pdf_debug_logger.debug(f"[{correlation_id}] {message}")
-    if error:
-        pdf_debug_logger.error(f"[{correlation_id}] Error: {str(error)}")
-        pdf_debug_logger.error(traceback.format_exc())
+# Đảm bảo thư mục upload tồn tại
+os.makedirs(TEMP_UPLOAD_DIR, exist_ok=True)
+os.makedirs(STORAGE_DIR, exist_ok=True)
 
-# Helper function to send progress updates
-async def send_progress_update(user_id, file_id, step, progress=0.0, message=""):
-    """Send PDF processing progress updates via WebSocket"""
-    try:
-        await send_pdf_upload_progress(user_id, file_id, step, progress, message)
-    except Exception as e:
-        logger.error(f"Error sending progress update: {e}")
-        logger.error(traceback.format_exc())
-
-# Function with fixed indentation for the troublesome parts
-async def handle_pdf_processing_result(result, correlation_id, user_id, file_id, filename, document, vector_status, 
-                                    vector_database_id, temp_file_path, db, is_pdf):
-    """Process the result of PDF processing and update database records"""
-    # If successful, move file to permanent storage
-    if result.get('success'):
-        try:
-            storage_path = os.path.join(STORAGE_DIR, f"{file_id}{'.pdf' if is_pdf else '.txt'}")
-            shutil.move(temp_file_path, storage_path)
-            log_upload_debug(correlation_id, f"Moved file to storage at {storage_path}")
-        except Exception as move_error:
-            log_upload_debug(correlation_id, f"Error moving file to storage: {move_error}", move_error)
-        
-        # Update status in PostgreSQL
-        if vector_database_id and document and vector_status:
-            try:
-                log_upload_debug(correlation_id, f"Updating vector status to 'completed' for document ID {document.id}")
-                
-                # Update the vector status with the result document_id (important for later deletion)
-                result_document_id = result.get('document_id') 
-                
-                vector_status.status = "completed"
-                vector_status.embedded_at = datetime.now()
-                
-                # Critical: Store the correct vector ID for future deletion
-                # This can be either the original file_id or the result_document_id
-                if result_document_id and result_document_id != file_id:
-                    # If Pinecone returned a specific document_id, use that
-                    vector_status.vector_id = result_document_id
-                    log_upload_debug(correlation_id, f"Updated vector_id to {result_document_id} (from result)")
-                elif file_id:
-                    # Make sure file_id is stored as the vector_id
-                    vector_status.vector_id = file_id
-                    log_upload_debug(correlation_id, f"Updated vector_id to {file_id} (from file_id)")
-                
-                # Also ensure we store some backup identifiers in case the primary one fails
-                # Store the document name as a secondary identifier
-                vector_status.document_name = document.name
-                log_upload_debug(correlation_id, f"Stored document_name '{document.name}' in vector status for backup")
-                
-                # Mark document as embedded
-                document.is_embedded = True
-                
-                db.commit()
-                log_upload_debug(correlation_id, f"Database status updated successfully")
-            except Exception as db_error:
-                log_upload_debug(correlation_id, f"Error updating database status: {db_error}", db_error)
-        
-        # Send completion notification via WebSocket
-        if user_id:
-            try:
-                await send_pdf_upload_completed(
-                    user_id,
-                    file_id,
-                    filename,
-                    result.get('chunks_processed', 0)
-                )
-                log_upload_debug(correlation_id, f"Sent upload completed notification to user {user_id}")
-            except Exception as ws_error:
-                log_upload_debug(correlation_id, f"Error sending WebSocket notification: {ws_error}", ws_error)
-            
-        # Add document information to the result
-        if document:
-            result["document_database_id"] = document.id
-    else:
-        log_upload_debug(correlation_id, f"PDF processing failed: {result.get('error', 'Unknown error')}")
-        
-        # Update error status in PostgreSQL
-        if vector_database_id and document and vector_status:
-            try:
-                log_upload_debug(correlation_id, f"Updating vector status to 'failed' for document ID {document.id}")
-                vector_status.status = "failed"
-                vector_status.error_message = result.get('error', 'Unknown error')
-                db.commit()
-                log_upload_debug(correlation_id, f"Database status updated for failure")
-            except Exception as db_error:
-                log_upload_debug(correlation_id, f"Error updating database status for failure: {db_error}", db_error)
-            
-        # Send failure notification via WebSocket
-        if user_id:
-            try:
-                await send_pdf_upload_failed(
-                    user_id,
-                    file_id,
-                    filename,
-                    result.get('error', 'Unknown error')
-                )
-                log_upload_debug(correlation_id, f"Sent upload failed notification to user {user_id}")
-            except Exception as ws_error:
-                log_upload_debug(correlation_id, f"Error sending WebSocket notification: {ws_error}", ws_error)
-        
-    # Cleanup: delete temporary file if it still exists
-    if temp_file_path and os.path.exists(temp_file_path):
-        try:
-            os.remove(temp_file_path)
-            log_upload_debug(correlation_id, f"Removed temporary file {temp_file_path}")
-        except Exception as cleanup_error:
-            log_upload_debug(correlation_id, f"Error removing temporary file: {cleanup_error}", cleanup_error)
-    
-    log_upload_debug(correlation_id, f"Upload request completed with success={result.get('success', False)}")
-    return result
-
-# Endpoint for uploading and processing PDFs
+# Endpoint upload và xử lý PDF
 @router.post("/upload", response_model=PDFResponse)
 async def upload_pdf(
     file: UploadFile = File(...),
@@ -208,398 +45,229 @@ async def upload_pdf(
     description: Optional[str] = Form(None),
     user_id: Optional[str] = Form(None),
     vector_database_id: Optional[int] = Form(None),
-    content_type: Optional[str] = Form(None),  # Add content_type parameter
     background_tasks: BackgroundTasks = None,
     db: Session = Depends(get_db)
 ):
     """
-    Upload and process PDF file to create embeddings and store in Pinecone
-    
-    - **file**: PDF file to process
-    - **namespace**: Namespace in Pinecone to store embeddings (default: "Default")
-    - **index_name**: Name of Pinecone index (default: "testbot768")
-    - **title**: Document title (optional)
-    - **description**: Document description (optional)
-    - **user_id**: User ID for WebSocket status updates
-    - **vector_database_id**: ID of vector database in PostgreSQL (optional)
-    - **content_type**: Content type of the file (optional)
+    Upload và xử lý file PDF để tạo embeddings và lưu vào Pinecone
     
-    Note: Mock mode has been permanently removed and the system always operates in real mode
+    - **file**: File PDF cần xử lý
+    - **namespace**: Namespace trong Pinecone để lưu embeddings (mặc định: "Default")
+    - **index_name**: Tên index Pinecone (mặc định: "testbot768")
+    - **title**: Tiêu đề của tài liệu (tùy chọn)
+    - **description**: Mô tả về tài liệu (tùy chọn)
+    - **user_id**: ID của người dùng để cập nhật trạng thái qua WebSocket
+    - **vector_database_id**: ID của vector database trong PostgreSQL (tùy chọn)
     """
-    # Generate request ID for tracking
-    correlation_id = str(uuid.uuid4())[:8]
-    logger.info(f"[{correlation_id}] PDF upload request received: ns={namespace}, index={index_name}, user={user_id}")
-    log_upload_debug(correlation_id, f"Upload request: vector_db_id={vector_database_id}")
-    
-    # Variables that might need cleanup in case of error
-    temp_file_path = None
-    document = None
-    vector_status = None
-    
     try:
-        # Check file type - accept both PDF and plaintext for testing
-        is_pdf = file.filename.lower().endswith('.pdf')
-        is_text = file.filename.lower().endswith(('.txt', '.md', '.html'))
-        
-        log_upload_debug(correlation_id, f"File type check: is_pdf={is_pdf}, is_text={is_text}, filename={file.filename}")
+        # Kiểm tra file có phải PDF không
+        if not file.filename.lower().endswith('.pdf'):
+            raise HTTPException(status_code=400, detail="Chỉ chấp nhận file PDF")
         
-        if not (is_pdf or is_text):
-            log_upload_debug(correlation_id, f"Rejecting non-PDF file: {file.filename}")
-            raise HTTPException(status_code=400, detail="Only PDF files are accepted")
-        
-        # If vector_database_id provided, get info from PostgreSQL
+        # Nếu có vector_database_id, lấy thông tin từ PostgreSQL
         api_key = None
         vector_db = None
         
         if vector_database_id:
-            log_upload_debug(correlation_id, f"Looking up vector database ID {vector_database_id}")
-            
             vector_db = db.query(VectorDatabase).filter(
                 VectorDatabase.id == vector_database_id,
                 VectorDatabase.status == "active"
             ).first()
             
             if not vector_db:
-                log_upload_debug(correlation_id, f"Vector database {vector_database_id} not found or inactive")
-                raise HTTPException(status_code=404, detail="Vector database not found or inactive")
-            
-            log_upload_debug(correlation_id, f"Found vector database: id={vector_db.id}, name={vector_db.name}, index={vector_db.pinecone_index}")
-            
-            # Use vector database information
-            # Try to get API key from relationship
-            log_upload_debug(correlation_id, f"Trying to get API key for vector database {vector_database_id}")
-            
-            # Log available attributes
-            vector_db_attrs = dir(vector_db)
-            log_upload_debug(correlation_id, f"Vector DB attributes: {vector_db_attrs}")
-            
-            if hasattr(vector_db, 'api_key_ref') and vector_db.api_key_ref:
-                log_upload_debug(correlation_id, f"Using API key from relationship for vector database ID {vector_database_id}")
-                log_upload_debug(correlation_id, f"api_key_ref type: {type(vector_db.api_key_ref)}")
-                log_upload_debug(correlation_id, f"api_key_ref attributes: {dir(vector_db.api_key_ref)}")
-                
-                if hasattr(vector_db.api_key_ref, 'key_value'):
-                    api_key = vector_db.api_key_ref.key_value
-                    # Log first few chars of API key for debugging
-                    key_prefix = api_key[:4] + "..." if api_key and len(api_key) > 4 else "invalid/empty"
-                    log_upload_debug(correlation_id, f"API key retrieved: {key_prefix}, length: {len(api_key) if api_key else 0}")
-                    logger.info(f"[{correlation_id}] Using API key from relationship for vector database ID {vector_database_id}")
-                else:
-                    log_upload_debug(correlation_id, f"api_key_ref does not have key_value attribute")
-            elif hasattr(vector_db, 'api_key') and vector_db.api_key:
-                # Fallback to direct api_key if needed (deprecated)
-                api_key = vector_db.api_key
-                key_prefix = api_key[:4] + "..." if api_key and len(api_key) > 4 else "invalid/empty"
-                log_upload_debug(correlation_id, f"Using deprecated direct api_key: {key_prefix}")
-                logger.warning(f"[{correlation_id}] Using deprecated direct api_key for vector database ID {vector_database_id}")
-            else:
-                log_upload_debug(correlation_id, "No API key found in vector database")
+                raise HTTPException(status_code=404, detail="Vector database không tồn tại hoặc không hoạt động")
             
-            # Use index from vector database
+            # Sử dụng thông tin từ vector database
+            api_key = vector_db.api_key
             index_name = vector_db.pinecone_index
-            log_upload_debug(correlation_id, f"Using index name '{index_name}' from vector database")
-            logger.info(f"[{correlation_id}] Using index name '{index_name}' from vector database")
         
-        # Generate file_id and save temporary file
+        # Tạo file_id và lưu file tạm
         file_id = str(uuid.uuid4())
-        temp_file_path = os.path.join(TEMP_UPLOAD_DIR, f"{file_id}{'.pdf' if is_pdf else '.txt'}")
-        log_upload_debug(correlation_id, f"Generated file_id: {file_id}, temp path: {temp_file_path}")
+        temp_file_path = os.path.join(TEMP_UPLOAD_DIR, f"{file_id}.pdf")
         
-        # Send notification of upload start via WebSocket if user_id provided
+        # Gửi thông báo bắt đầu xử lý qua WebSocket nếu có user_id
         if user_id:
-            try:
-                await send_pdf_upload_started(user_id, file.filename, file_id)
-                log_upload_debug(correlation_id, f"Sent upload started notification to user {user_id}")
-            except Exception as ws_error:
-                log_upload_debug(correlation_id, f"Error sending WebSocket notification: {ws_error}", ws_error)
+            await send_pdf_upload_started(user_id, file.filename, file_id)
         
-        # Save file
-        log_upload_debug(correlation_id, f"Reading file content")
+        # Lưu file
         file_content = await file.read()
-        log_upload_debug(correlation_id, f"File size: {len(file_content)} bytes")
-        
         with open(temp_file_path, "wb") as buffer:
             buffer.write(file_content)
-        log_upload_debug(correlation_id, f"File saved to {temp_file_path}")
             
-        # Create metadata
+        # Tạo metadata
         metadata = {
             "filename": file.filename,
             "content_type": file.content_type
         }
         
-        # Use provided content_type or fallback to file.content_type
-        actual_content_type = content_type or file.content_type
-        log_upload_debug(correlation_id, f"Using content_type: {actual_content_type}")
-        
-        if not actual_content_type:
-            # Fallback content type based on file extension
-            if is_pdf:
-                actual_content_type = "application/pdf"
-            elif is_text:
-                actual_content_type = "text/plain"
-            else:
-                actual_content_type = "application/octet-stream"
-                
-            log_upload_debug(correlation_id, f"No content_type provided, using fallback: {actual_content_type}")
-            
-        metadata["content_type"] = actual_content_type
-        
-        # Use provided title or filename as document name
-        document_name = title or file.filename
-        
-        # Verify document name is unique within this vector database
-        if vector_database_id:
-            # Check if a document with this name already exists in this vector database
-            existing_doc = db.query(Document).filter(
-                Document.name == document_name,
-                Document.vector_database_id == vector_database_id
-            ).first()
-            
-            if existing_doc:
-                # Make the name unique by appending timestamp
-                timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
-                base_name, extension = os.path.splitext(document_name)
-                document_name = f"{base_name}_{timestamp}{extension}"
-                log_upload_debug(correlation_id, f"Document name already exists, using unique name: {document_name}")
-        
-        metadata["title"] = document_name
-        
+        if title:
+            metadata["title"] = title
         if description:
             metadata["description"] = description
         
-        # Send progress update via WebSocket
+        # Gửi thông báo tiến độ qua WebSocket
         if user_id:
-            try:
-                await send_progress_update(
+            await send_pdf_upload_progress(
                 user_id, 
                 file_id, 
                 "file_preparation", 
                 0.2, 
                 "File saved, preparing for processing"
             )
-                log_upload_debug(correlation_id, f"Sent file preparation progress to user {user_id}")
-            except Exception as ws_error:
-                log_upload_debug(correlation_id, f"Error sending progress update: {ws_error}", ws_error)
-        
-        # Create document record - do this regardless of mock mode
-        document = None
-        vector_status = None
         
+        # Lưu thông tin tài liệu vào PostgreSQL nếu có vector_database_id
         if vector_database_id and vector_db:
-            log_upload_debug(correlation_id, f"Creating PostgreSQL records for document with vector_database_id={vector_database_id}")
-            
             # Create document record without file content
-            try:
-                document = Document(
-                    name=document_name,  # Use the (potentially) modified document name
-                    file_type="pdf" if is_pdf else "text",
-                    content_type=actual_content_type,  # Use the actual_content_type here
-                    size=len(file_content),
-                    is_embedded=False,
-                    vector_database_id=vector_database_id
-                )
-                db.add(document)
-                db.commit()
-                db.refresh(document)
-                log_upload_debug(correlation_id, f"Created document record: id={document.id}")
-            except Exception as doc_error:
-                log_upload_debug(correlation_id, f"Error creating document record: {doc_error}", doc_error)
-                raise
+            document = Document(
+                name=title or file.filename,
+                file_type="pdf",
+                content_type=file.content_type,
+                size=len(file_content),
+                is_embedded=False,
+                vector_database_id=vector_database_id
+            )
+            db.add(document)
+            db.commit()
+            db.refresh(document)
             
             # Create document content record to store binary data separately
-            try:
-                document_content = DocumentContent(
-                    document_id=document.id,
-                    file_content=file_content
-                )
-                db.add(document_content)
-                db.commit()
-                log_upload_debug(correlation_id, f"Created document content record for document ID {document.id}")
-            except Exception as content_error:
-                log_upload_debug(correlation_id, f"Error creating document content: {content_error}", content_error)
-                raise
-            
-            # Create vector status record - store file_id as the vector_id for deletion later
-            try:
-                vector_status = VectorStatus(
-                    document_id=document.id,
-                    vector_database_id=vector_database_id,
-                    status="pending",
-                    vector_id=file_id  # Store the document UUID as vector_id for later deletion
-                )
-                db.add(vector_status)
-                db.commit()
-                log_upload_debug(correlation_id, f"Created vector status record for document ID {document.id} with vector_id={file_id}")
-            except Exception as status_error:
-                log_upload_debug(correlation_id, f"Error creating vector status: {status_error}", status_error)
-                raise
-            
-            logger.info(f"[{correlation_id}] Created document ID {document.id} and vector status in PostgreSQL")
+            document_content = DocumentContent(
+                document_id=document.id,
+                file_content=file_content
+            )
+            db.add(document_content)
+            db.commit()
+            
+            # Tạo vector status record
+            vector_status = VectorStatus(
+                document_id=document.id,
+                vector_database_id=vector_database_id,
+                status="pending"
+            )
+            db.add(vector_status)
+            db.commit()
             
-        # Initialize PDF processor with correct parameters
-        log_upload_debug(correlation_id, f"Initializing PDFProcessor: index={index_name}, vector_db_id={vector_database_id}")
-        processor = PDFProcessor(
-            index_name=index_name, 
-            namespace=namespace, 
-            api_key=api_key, 
-            vector_db_id=vector_database_id,
-            correlation_id=correlation_id
-        )
+        # Khởi tạo PDF processor với API key nếu có
+        processor = PDFProcessor(index_name=index_name, namespace=namespace, api_key=api_key)
         
-        # Send embedding start notification via WebSocket
+        # Gửi thông báo bắt đầu embedding qua WebSocket
         if user_id:
-            try:
-                await send_progress_update(
+            await send_pdf_upload_progress(
                 user_id, 
                 file_id, 
                 "embedding_start", 
                 0.4, 
                 "Starting to process PDF and create embeddings"
             )
-                log_upload_debug(correlation_id, f"Sent embedding start notification to user {user_id}")
-            except Exception as ws_error:
-                log_upload_debug(correlation_id, f"Error sending WebSocket notification: {ws_error}", ws_error)
         
-        # Process PDF and create embeddings with progress callback
-        log_upload_debug(correlation_id, f"Processing PDF with file_path={temp_file_path}, document_id={file_id}")
+        # Xử lý PDF và tạo embeddings
+        # Tạo callback function để xử lý cập nhật tiến độ
+        async def progress_callback_wrapper(step, progress, message):
+            if user_id:
+                await send_progress_update(user_id, file_id, step, progress, message)
+        
+        # Xử lý PDF và tạo embeddings với callback đã được xử lý đúng cách
         result = await processor.process_pdf(
             file_path=temp_file_path,
-            document_id=file_id,  # Use UUID as document_id for Pinecone
+            document_id=file_id,
             metadata=metadata,
-            progress_callback=send_progress_update if user_id else None
+            progress_callback=progress_callback_wrapper
         )
         
-        log_upload_debug(correlation_id, f"PDF processing result: {result}")
-        
-        # Handle PDF processing result
-        return await handle_pdf_processing_result(result, correlation_id, user_id, file_id, file.filename, document, vector_status, 
-                                                vector_database_id, temp_file_path, db, is_pdf)
-    except Exception as e:
-        log_upload_debug(correlation_id, f"Error in upload_pdf: {str(e)}", e)
-        logger.exception(f"[{correlation_id}] Error in upload_pdf: {str(e)}")
-        
-        # Cleanup on error
-        if os.path.exists(temp_file_path):
-            try:
-                os.remove(temp_file_path)
-                log_upload_debug(correlation_id, f"Cleaned up temp file after error: {temp_file_path}")
-            except Exception as cleanup_error:
-                log_upload_debug(correlation_id, f"Error cleaning up temporary file: {cleanup_error}", cleanup_error)
-        
-        # Update error status in PostgreSQL
-        if vector_database_id and vector_status:
-            try:
-                vector_status.status = "failed"
-                vector_status.error_message = str(e)
+        # Nếu thành công, chuyển file vào storage
+        if result.get('success'):
+            storage_path = os.path.join(STORAGE_DIR, f"{file_id}.pdf")
+            shutil.move(temp_file_path, storage_path)
+            
+            # Cập nhật trạng thái trong PostgreSQL nếu có vector_database_id
+            if vector_database_id and 'document' in locals() and 'vector_status' in locals():
+                vector_status.status = "completed"
+                vector_status.embedded_at = datetime.now()
+                vector_status.vector_id = file_id
+                document.is_embedded = True
                 db.commit()
-                log_upload_debug(correlation_id, f"Updated database with error status")
-            except Exception as db_error:
-                log_upload_debug(correlation_id, f"Error updating database with error status: {db_error}", db_error)
             
-        # Send failure notification via WebSocket
-        if user_id and file_id:
-            try:
+            # Gửi thông báo hoàn thành qua WebSocket
+            if user_id:
+                await send_pdf_upload_completed(
+                    user_id,
+                    file_id,
+                    file.filename,
+                    result.get('chunks_processed', 0)
+                )
+        else:
+            # Cập nhật trạng thái lỗi trong PostgreSQL nếu có vector_database_id
+            if vector_database_id and 'vector_status' in locals():
+                vector_status.status = "failed"
+                vector_status.error_message = result.get('error', 'Unknown error')
+                db.commit()
+                
+            # Gửi thông báo lỗi qua WebSocket
+            if user_id:
                 await send_pdf_upload_failed(
                     user_id,
                     file_id,
                     file.filename,
-                    str(e)
+                    result.get('error', 'Unknown error')
                 )
-                log_upload_debug(correlation_id, f"Sent failure notification for exception")
-            except Exception as ws_error:
-                log_upload_debug(correlation_id, f"Error sending WebSocket notification for failure: {ws_error}", ws_error)
             
-        log_upload_debug(correlation_id, f"Upload request failed with exception: {str(e)}")
+        # Dọn dẹp: xóa file tạm nếu vẫn còn
+        if os.path.exists(temp_file_path):
+            os.remove(temp_file_path)
+            
+        return result
+    except Exception as e:
+        # D���n dẹp nếu có lỗi
+        if 'temp_file_path' in locals() and os.path.exists(temp_file_path):
+            os.remove(temp_file_path)
+            
+        # Cập nhật trạng thái lỗi trong PostgreSQL nếu có vector_database_id
+        if 'vector_database_id' in locals() and vector_database_id and 'vector_status' in locals():
+            vector_status.status = "failed"
+            vector_status.error_message = str(e)
+            db.commit()
+            
+        # Gửi thông báo lỗi qua WebSocket
+        if 'user_id' in locals() and user_id and 'file_id' in locals():
+            await send_pdf_upload_failed(
+                user_id,
+                file_id,
+                file.filename,
+                str(e)
+            )
+            
         return PDFResponse(
             success=False,
             error=str(e)
         )
 
+# Function để gửi cập nhật tiến độ - được sử dụng trong callback
+async def send_progress_update(user_id, document_id, step, progress, message):
+    if user_id:
+        await send_pdf_upload_progress(user_id, document_id, step, progress, message)
+
 # Endpoint xóa tài liệu
 @router.delete("/namespace", response_model=PDFResponse)
 async def delete_namespace(
     namespace: str = "Default",
     index_name: str = "testbot768",
-    vector_database_id: Optional[int] = None,
-    user_id: Optional[str] = None,
-    db: Session = Depends(get_db)
+    user_id: Optional[str] = None
 ):
     """
     Xóa toàn bộ embeddings trong một namespace từ Pinecone (tương ứng xoá namespace)
 
     - **namespace**: Namespace trong Pinecone (mặc định: "Default")
     - **index_name**: Tên index Pinecone (mặc định: "testbot768")
-    - **vector_database_id**: ID của vector database trong PostgreSQL (nếu có)
     - **user_id**: ID của người dùng để cập nhật trạng thái qua WebSocket
     """
-    logger.info(f"Delete namespace request: namespace={namespace}, index={index_name}, vector_db_id={vector_database_id}")
-    
     try:
-        # Nếu có vector_database_id, lấy thông tin từ PostgreSQL
-        api_key = None
-        vector_db = None
-
-        if vector_database_id:
-            vector_db = db.query(VectorDatabase).filter(
-                VectorDatabase.id == vector_database_id,
-                VectorDatabase.status == "active"
-            ).first()
-            if not vector_db:
-                return PDFResponse(
-                    success=False,
-                    error=f"Vector database with ID {vector_database_id} not found or inactive"
-                )
-            
-            # Use index from vector database
-            index_name = vector_db.pinecone_index
-            
-            # Get API key
-            if hasattr(vector_db, 'api_key_ref') and vector_db.api_key_ref:
-                api_key = vector_db.api_key_ref.key_value
-            elif hasattr(vector_db, 'api_key') and vector_db.api_key:
-                api_key = vector_db.api_key
-            
-            # Use namespace based on vector database ID
-            namespace = f"vdb-{vector_database_id}" if vector_database_id else namespace
-            logger.info(f"Using namespace '{namespace}' based on vector database ID")
-            
         # Gửi thông báo bắt đầu xóa qua WebSocket
         if user_id:
             await send_pdf_delete_started(user_id, namespace)
             
-        processor = PDFProcessor(
-            index_name=index_name, 
-            namespace=namespace,
-            api_key=api_key,
-            vector_db_id=vector_database_id
-        )
+        processor = PDFProcessor(index_name=index_name, namespace=namespace)
         result = await processor.delete_namespace()
         
-        # If successful and vector_database_id, update PostgreSQL to reflect the deletion
-        if result.get('success') and vector_database_id:
-            try:
-                # Update vector statuses for this database
-                affected_count = db.query(VectorStatus).filter(
-                    VectorStatus.vector_database_id == vector_database_id,
-                    VectorStatus.status != "deleted"
-                ).update({"status": "deleted", "updated_at": datetime.now()})
-                
-                # Update document embedding status
-                db.query(Document).filter(
-                    Document.vector_database_id == vector_database_id,
-                    Document.is_embedded == True
-                ).update({"is_embedded": False})
-                
-                db.commit()
-                logger.info(f"Updated {affected_count} vector statuses to 'deleted'")
-                
-                # Include this info in the result
-                result["updated_records"] = affected_count
-            except Exception as db_error:
-                logger.error(f"Error updating PostgreSQL records after namespace deletion: {db_error}")
-                result["postgresql_update_error"] = str(db_error)
-        
         # Gửi thông báo kết quả qua WebSocket
         if user_id:
             if result.get('success'):
@@ -609,8 +277,6 @@ async def delete_namespace(
                 
         return result
     except Exception as e:
-        logger.exception(f"Error in delete_namespace: {str(e)}")
-        
         # Gửi thông báo lỗi qua WebSocket
         if user_id:
             await send_pdf_delete_failed(user_id, namespace, str(e))
@@ -622,338 +288,23 @@ async def delete_namespace(
 
 # Endpoint lấy danh sách tài liệu
 @router.get("/documents", response_model=DocumentsListResponse)
-async def get_documents(
-    namespace: str = "Default", 
-    index_name: str = "testbot768",
-    vector_database_id: Optional[int] = None,
-    db: Session = Depends(get_db)
-):
+async def get_documents(namespace: str = "Default", index_name: str = "testbot768"):
     """
     Lấy thông tin về tất cả tài liệu đã được embed
     
     - **namespace**: Namespace trong Pinecone (mặc định: "Default")
     - **index_name**: Tên index Pinecone (mặc định: "testbot768")
-    - **vector_database_id**: ID của vector database trong PostgreSQL (nếu có)
     """
-    logger.info(f"Get documents request: namespace={namespace}, index={index_name}, vector_db_id={vector_database_id}")
-    
     try:
-        # Nếu có vector_database_id, lấy thông tin từ PostgreSQL
-        api_key = None
-        vector_db = None
-
-        if vector_database_id:
-            vector_db = db.query(VectorDatabase).filter(
-                VectorDatabase.id == vector_database_id,
-                VectorDatabase.status == "active"
-            ).first()
-            
-            if not vector_db:
-                return DocumentsListResponse(
-                    success=False,
-                    error=f"Vector database with ID {vector_database_id} not found or inactive"
-                )
-                
-            # Use index from vector database
-            index_name = vector_db.pinecone_index
-            
-            # Get API key
-            if hasattr(vector_db, 'api_key_ref') and vector_db.api_key_ref:
-                api_key = vector_db.api_key_ref.key_value
-            elif hasattr(vector_db, 'api_key') and vector_db.api_key:
-                api_key = vector_db.api_key
-                
-            # Use namespace based on vector database ID
-            namespace = f"vdb-{vector_database_id}" if vector_database_id else namespace
-            logger.info(f"Using namespace '{namespace}' based on vector database ID")
-            
         # Khởi tạo PDF processor
-        processor = PDFProcessor(
-            index_name=index_name, 
-            namespace=namespace,
-            api_key=api_key,
-            vector_db_id=vector_database_id
-        )
+        processor = PDFProcessor(index_name=index_name, namespace=namespace)
         
-        # Lấy danh sách documents từ Pinecone
-        pinecone_result = await processor.list_documents()
-        
-        # If vector_database_id is provided, also fetch from PostgreSQL
-        if vector_database_id:
-            try:
-                # Get all successfully embedded documents for this vector database
-                documents = db.query(Document).join(
-                    VectorStatus, Document.id == VectorStatus.document_id
-                ).filter(
-                    Document.vector_database_id == vector_database_id,
-                    Document.is_embedded == True,
-                    VectorStatus.status == "completed"
-                ).all()
-                
-                # Add document info to the result
-                if documents:
-                    pinecone_result["postgresql_documents"] = [
-                        {
-                            "id": doc.id,
-                            "name": doc.name,
-                            "file_type": doc.file_type,
-                            "content_type": doc.content_type,
-                            "created_at": doc.created_at.isoformat() if doc.created_at else None
-                        }
-                        for doc in documents
-                    ]
-                    pinecone_result["postgresql_document_count"] = len(documents)
-            except Exception as db_error:
-                logger.error(f"Error fetching PostgreSQL documents: {db_error}")
-                pinecone_result["postgresql_error"] = str(db_error)
-        
-        return pinecone_result
-    except Exception as e:
-        logger.exception(f"Error in get_documents: {str(e)}")
-        
-        return DocumentsListResponse(
-            success=False,
-            error=str(e)
-        )
-
-# Health check endpoint for PDF API
-@router.get("/health")
-async def health_check():
-    return {
-        "status": "healthy",
-        "version": "1.0.0",
-        "message": "PDF API is running"
-    }
-
-# Document deletion endpoint
-@router.delete("/document", response_model=PDFResponse)
-async def delete_document(
-    document_id: str,
-    namespace: str = "Default",
-    index_name: str = "testbot768",
-    vector_database_id: Optional[int] = None,
-    user_id: Optional[str] = None,
-    db: Session = Depends(get_db)
-):
-    """
-    Delete vectors for a specific document from the vector database
-    
-    This endpoint can be called in two ways:
-    1. With the PostgreSQL document ID - will look up the actual vector_id first
-    2. With the actual vector_id directly - when called from the PostgreSQL document deletion endpoint
-    
-    - **document_id**: ID of the document to delete (can be PostgreSQL document ID or Pinecone vector_id)
-    - **namespace**: Namespace in the vector database (default: "Default")
-    - **index_name**: Name of the vector index (default: "testbot768")
-    - **vector_database_id**: ID of vector database in PostgreSQL (optional)
-    - **user_id**: User ID for WebSocket status updates (optional)
-    """
-    logger.info(f"Delete document request: document_id={document_id}, namespace={namespace}, index={index_name}, vector_db_id={vector_database_id}")
-    
-    try:
-        # If vector_database_id is provided, get info from PostgreSQL
-        api_key = None
-        vector_db = None
-        pinecone_document_id = document_id  # Default to the provided document_id
-        document_to_delete = None
-        vector_status_to_update = None
-        document_found = False  # Flag to track if document was found
-        vector_id_found = False  # Flag to track if a valid vector ID was found
-
-        if vector_database_id:
-            vector_db = db.query(VectorDatabase).filter(
-                VectorDatabase.id == vector_database_id,
-                VectorDatabase.status == "active"
-            ).first()
-            if not vector_db:
-                return PDFResponse(
-                    success=False,
-                    error=f"Vector database with ID {vector_database_id} not found or inactive"
-                )
-            
-            # Use index from vector database
-            index_name = vector_db.pinecone_index
-            
-            # Get API key
-            if hasattr(vector_db, 'api_key_ref') and vector_db.api_key_ref:
-                api_key = vector_db.api_key_ref.key_value
-            elif hasattr(vector_db, 'api_key') and vector_db.api_key:
-                api_key = vector_db.api_key
-            
-            # Use namespace based on vector database ID
-            namespace = f"vdb-{vector_database_id}" if vector_database_id else namespace
-            logger.info(f"Using namespace '{namespace}' based on vector database ID")
-            
-            # Check if document_id is a numeric database ID or document name
-            if document_id.isdigit():
-                # Try to find the document in PostgreSQL by its ID
-                db_document_id = int(document_id)
-                document_to_delete = db.query(Document).filter(Document.id == db_document_id).first()
-                
-                if document_to_delete:
-                    document_found = True
-                    logger.info(f"Found document in database: id={document_to_delete.id}, name={document_to_delete.name}")
-                    
-                    # Look for vector status to find the Pinecone vector_id
-                    vector_status_to_update = db.query(VectorStatus).filter(
-                        VectorStatus.document_id == document_to_delete.id,
-                        VectorStatus.vector_database_id == vector_database_id
-                    ).first()
-                    
-                    if vector_status_to_update and vector_status_to_update.vector_id:
-                        pinecone_document_id = vector_status_to_update.vector_id
-                        vector_id_found = True
-                        logger.info(f"Using vector_id '{pinecone_document_id}' from vector status")
-                    else:
-                        # Fallback options if vector_id is not directly found
-                        pinecone_document_id = document_to_delete.name
-                        logger.info(f"Vector ID not found in status, using document name '{pinecone_document_id}' as fallback")
-                else:
-                    logger.warning(f"Document with ID {db_document_id} not found in database. Using ID as is.")
-            else:
-                # Try to find document by name/title
-                document_to_delete = db.query(Document).filter(
-                    Document.name == document_id,
-                    Document.vector_database_id == vector_database_id
-                ).first()
-                
-                if document_to_delete:
-                    document_found = True
-                    logger.info(f"Found document by name: id={document_to_delete.id}, name={document_to_delete.name}")
-                    
-                    # Get vector status for this document
-                    vector_status_to_update = db.query(VectorStatus).filter(
-                        VectorStatus.document_id == document_to_delete.id,
-                        VectorStatus.vector_database_id == vector_database_id
-                    ).first()
-                    
-                    if vector_status_to_update and vector_status_to_update.vector_id:
-                        pinecone_document_id = vector_status_to_update.vector_id
-                        vector_id_found = True
-                        logger.info(f"Using vector_id '{pinecone_document_id}' from vector status")
-            
-        # Send notification of deletion start via WebSocket if user_id provided
-        if user_id:
-            try:
-                await send_pdf_delete_started(user_id, pinecone_document_id)
-            except Exception as ws_error:
-                logger.error(f"Error sending WebSocket notification: {ws_error}")
-        
-        # Initialize PDF processor
-        processor = PDFProcessor(
-            index_name=index_name, 
-            namespace=namespace, 
-            api_key=api_key, 
-            vector_db_id=vector_database_id
-        )
-        
-        # Delete document vectors using the pinecone_document_id and additional metadata
-        additional_metadata = {}
-        if document_to_delete:
-            # Add document name as title for searching
-            additional_metadata["document_name"] = document_to_delete.name
-        
-        result = await processor.delete_document(pinecone_document_id, additional_metadata)
-        
-        # Check if vectors were actually deleted or found
-        vectors_deleted = result.get('vectors_deleted', 0)
-        vectors_found = result.get('vectors_found', False)
-        
-        # If no document was found in PostgreSQL and no vectors were found/deleted in Pinecone
-        if not document_found and not vectors_found:
-            result['success'] = False  # Override success to false
-            result['error'] = f"Document ID {document_id} not found in PostgreSQL or Pinecone"
-            
-            # Send notification of deletion failure via WebSocket if user_id provided
-            if user_id:
-                try:
-                    await send_pdf_delete_failed(user_id, document_id, result['error'])
-                except Exception as ws_error:
-                    logger.error(f"Error sending WebSocket notification: {ws_error}")
-                    
-            return result
-        
-        # If successful and vector_database_id is provided, update PostgreSQL records
-        if result.get('success') and vector_database_id:
-            try:
-                # Update vector status if we found it earlier
-                if vector_status_to_update:
-                    vector_status_to_update.status = "deleted"
-                    db.commit()
-                    result["postgresql_updated"] = True
-                    logger.info(f"Updated vector status for document ID {document_to_delete.id if document_to_delete else document_id} to 'deleted'")
-                else:
-                    # If we didn't find it earlier, try again with more search options
-                    document = None
-                    
-                    if document_id.isdigit():
-                        # If the original document_id was numeric, use it directly
-                        document = db.query(Document).filter(Document.id == int(document_id)).first()
-                    
-                    if not document:
-                        # Find document by vector ID if it exists
-                        document = db.query(Document).join(
-                            VectorStatus, Document.id == VectorStatus.document_id
-                        ).filter(
-                            Document.vector_database_id == vector_database_id,
-                            VectorStatus.vector_id == pinecone_document_id
-                        ).first()
-                    
-                    if not document:
-                        # Try finding by name
-                        document = db.query(Document).filter(
-                            Document.vector_database_id == vector_database_id,
-                            Document.name == pinecone_document_id
-                        ).first()
-                    
-                    if document:
-                        # Update vector status
-                        vector_status = db.query(VectorStatus).filter(
-                            VectorStatus.document_id == document.id,
-                            VectorStatus.vector_database_id == vector_database_id
-                        ).first()
-                        
-                        if vector_status:
-                            vector_status.status = "deleted"
-                            db.commit()
-                            result["postgresql_updated"] = True
-                            logger.info(f"Updated vector status for document ID {document.id} to 'deleted'")
-                    else:
-                        logger.warning(f"Could not find document record for deletion confirmation. Document ID: {document_id}, Vector ID: {pinecone_document_id}")
-            except Exception as db_error:
-                logger.error(f"Error updating PostgreSQL records: {db_error}")
-                result["postgresql_error"] = str(db_error)
-        
-        # Add information about what was found and deleted
-        result["document_found_in_db"] = document_found
-        result["vector_id_found"] = vector_id_found
-        result["vectors_deleted"] = vectors_deleted
-        
-        # Send notification of deletion completion via WebSocket if user_id provided
-        if user_id:
-            try:
-                if result.get('success'):
-                    await send_pdf_delete_completed(user_id, pinecone_document_id)
-                else:
-                    await send_pdf_delete_failed(user_id, pinecone_document_id, result.get('error', 'Unknown error'))
-            except Exception as ws_error:
-                logger.error(f"Error sending WebSocket notification: {ws_error}")
+        # Lấy danh sách documents
+        result = await processor.list_documents()
         
         return result
     except Exception as e:
-        logger.exception(f"Error in delete_document: {str(e)}")
-        
-        # Send notification of deletion failure via WebSocket if user_id provided
-        if user_id:
-            try:
-                await send_pdf_delete_failed(user_id, document_id, str(e))
-            except Exception as ws_error:
-                logger.error(f"Error sending WebSocket notification: {ws_error}")
-        
-        return PDFResponse(
+        return DocumentsListResponse(
             success=False,
             error=str(e)
-        )
-
-
-
+        ) 

app/api/pdf_websocket.py

@@ -108,184 +108,7 @@ class ConnectionManager:
 # Tạo instance của ConnectionManager
 manager = ConnectionManager()
 
-# Test route for manual WebSocket sending
-@router.get("/ws/test/{user_id}")
-async def test_websocket_send(user_id: str):
-    """
-    Test route to manually send a WebSocket message to a user
-    This is useful for debugging WebSocket connections
-    """
-    logger.info(f"Attempting to send test message to user: {user_id}")
-    
-    # Check if user has a connection
-    status = manager.get_connection_status(user_id)
-    if not status["active"]:
-        logger.warning(f"No active WebSocket connection for user: {user_id}")
-        return {"success": False, "message": f"No active WebSocket connection for user: {user_id}"}
-    
-    # Send test message
-    await manager.send_message({
-        "type": "test_message",
-        "message": "This is a test WebSocket message",
-        "timestamp": int(time.time())
-    }, user_id)
-    
-    logger.info(f"Test message sent to user: {user_id}")
-    return {"success": True, "message": f"Test message sent to user: {user_id}"}
-
-@router.websocket("/ws/pdf/{user_id}")
-async def websocket_endpoint(websocket: WebSocket, user_id: str):
-    """Endpoint WebSocket để cập nhật tiến trình xử lý PDF"""
-    logger.info(f"WebSocket connection request received for user: {user_id}")
-    
-    try:
-        await manager.connect(websocket, user_id)
-        logger.info(f"WebSocket connection accepted for user: {user_id}")
-        
-        # Send a test message to confirm connection
-        await manager.send_message({
-            "type": "connection_established",
-            "message": "WebSocket connection established successfully",
-            "user_id": user_id,
-            "timestamp": int(time.time())
-        }, user_id)
-        
-        try:
-            while True:
-                # Đợi tin nhắn từ client (chỉ để giữ kết nối)
-                data = await websocket.receive_text()
-                logger.debug(f"Received from client: {data}")
-                
-                # Echo back to confirm receipt
-                if data != "heartbeat":  # Don't echo heartbeats
-                    await manager.send_message({
-                        "type": "echo",
-                        "message": f"Received: {data}",
-                        "timestamp": int(time.time())
-                    }, user_id)
-        except WebSocketDisconnect:
-            logger.info(f"WebSocket disconnected for user: {user_id}")
-            manager.disconnect(websocket, user_id)
-        except Exception as e:
-            logger.error(f"WebSocket error: {str(e)}")
-            manager.disconnect(websocket, user_id)
-    except Exception as e:
-        logger.error(f"Failed to establish WebSocket connection: {str(e)}")
-        # Ensure the connection is closed properly
-        if websocket.client_state != 4:  # 4 = CLOSED
-            await websocket.close(code=1011, reason=f"Server error: {str(e)}")
-
-import logging
-from typing import Dict, List, Optional, Any
-from fastapi import WebSocket, WebSocketDisconnect, APIRouter
-from pydantic import BaseModel
-import json
-import time
-
-# Cấu hình logging
-logger = logging.getLogger(__name__)
-
-# Models cho Swagger documentation
-class ConnectionStatus(BaseModel):
-    user_id: str
-    active: bool
-    connection_count: int
-    last_activity: Optional[float] = None
-
-class UserConnection(BaseModel):
-    user_id: str
-    connection_count: int
-
-class AllConnectionsStatus(BaseModel):
-    total_users: int
-    total_connections: int
-    users: List[UserConnection]
-
-# Khởi tạo router
-router = APIRouter(
-    prefix="",
-    tags=["WebSockets"],
-)
-
-class ConnectionManager:
-    """Quản lý các kết nối WebSocket"""
-    
-    def __init__(self):
-        # Lưu trữ các kết nối theo user_id
-        self.active_connections: Dict[str, List[WebSocket]] = {}
-        
-    async def connect(self, websocket: WebSocket, user_id: str):
-        """Kết nối một WebSocket mới"""
-        await websocket.accept()
-        if user_id not in self.active_connections:
-            self.active_connections[user_id] = []
-        self.active_connections[user_id].append(websocket)
-        logger.info(f"New WebSocket connection for user {user_id}. Total connections: {len(self.active_connections[user_id])}")
-        
-    def disconnect(self, websocket: WebSocket, user_id: str):
-        """Ngắt kết nối WebSocket"""
-        if user_id in self.active_connections:
-            if websocket in self.active_connections[user_id]:
-                self.active_connections[user_id].remove(websocket)
-            # Xóa user_id khỏi dict nếu không còn kết nối nào
-            if not self.active_connections[user_id]:
-                del self.active_connections[user_id]
-        logger.info(f"WebSocket disconnected for user {user_id}")
-        
-    async def send_message(self, message: Dict[str, Any], user_id: str):
-        """Gửi tin nhắn tới t���t cả kết nối của một user"""
-        if user_id in self.active_connections:
-            disconnected_websockets = []
-            for websocket in self.active_connections[user_id]:
-                try:
-                    await websocket.send_text(json.dumps(message))
-                except Exception as e:
-                    logger.error(f"Error sending message to WebSocket: {str(e)}")
-                    disconnected_websockets.append(websocket)
-            
-            # Xóa các kết nối bị ngắt
-            for websocket in disconnected_websockets:
-                self.disconnect(websocket, user_id)
-    
-    def get_connection_status(self, user_id: str = None) -> Dict[str, Any]:
-        """Lấy thông tin về trạng thái kết nối WebSocket"""
-        if user_id:
-            # Trả về thông tin kết nối cho user cụ thể
-            if user_id in self.active_connections:
-                return {
-                    "user_id": user_id,
-                    "active": True,
-                    "connection_count": len(self.active_connections[user_id]),
-                    "last_activity": time.time()
-                }
-            else:
-                return {
-                    "user_id": user_id,
-                    "active": False,
-                    "connection_count": 0,
-                    "last_activity": None
-                }
-        else:
-            # Trả về thông tin tất cả kết nối
-            result = {
-                "total_users": len(self.active_connections),
-                "total_connections": sum(len(connections) for connections in self.active_connections.values()),
-                "users": []
-            }
-            
-            for uid, connections in self.active_connections.items():
-                result["users"].append({
-                    "user_id": uid,
-                    "connection_count": len(connections)
-                })
-            
-            return result
-
-
-# Tạo instance của ConnectionManager
-manager = ConnectionManager()
-
-@router.websocket("/ws/pdf/{user_id}")
+@router.websocket("/pdf/{user_id}")
 async def websocket_endpoint(websocket: WebSocket, user_id: str):
     """Endpoint WebSocket để cập nhật tiến trình xử lý PDF"""
     await manager.connect(websocket, user_id)
@@ -300,7 +123,7 @@ async def websocket_endpoint(websocket: WebSocket, user_id: str):
         manager.disconnect(websocket, user_id)
 
 # API endpoints để kiểm tra trạng thái WebSocket
-@router.get("/ws/status", response_model=AllConnectionsStatus, responses={
+@router.get("/status", response_model=AllConnectionsStatus, responses={
     200: {
         "description": "Successful response",
         "content": {
@@ -328,7 +151,7 @@ async def get_all_websocket_connections():
     """
     return manager.get_connection_status()
 
-@router.get("/ws/status/{user_id}", response_model=ConnectionStatus, responses={
+@router.get("/status/{user_id}", response_model=ConnectionStatus, responses={
     200: {
         "description": "Successful response for active connection",
         "content": {
@@ -422,12 +245,11 @@ async def send_pdf_delete_started(user_id: str, namespace: str):
         "timestamp": int(time.time())
     }, user_id)
 
-async def send_pdf_delete_completed(user_id: str, namespace: str, deleted_count: int = 0):
+async def send_pdf_delete_completed(user_id: str, namespace: str):
     """Gửi thông báo hoàn thành xóa PDF"""
     await manager.send_message({
         "type": "pdf_delete_completed",
         "namespace": namespace,
-        "deleted_count": deleted_count,
         "timestamp": int(time.time())
     }, user_id)
 

app/api/postgresql_routes.py

@@ -4,10 +4,8 @@ import traceback
 from datetime import datetime, timedelta, timezone
 import time
 from functools import lru_cache
-from pathlib import Path as pathlib_Path  # Import Path from pathlib with a different name
 
-from fastapi import APIRouter, HTTPException, Depends, Query, Body, Response, File, UploadFile, Form, BackgroundTasks
-from fastapi.params import Path  # Import Path explicitly from fastapi.params instead
+from fastapi import APIRouter, HTTPException, Depends, Query, Path, Body, Response
 from sqlalchemy.orm import Session
 from sqlalchemy.exc import SQLAlchemyError
 from typing import List, Optional, Dict, Any
@@ -18,7 +16,6 @@ from sqlalchemy import text, inspect, func
 from sqlalchemy.exc import SQLAlchemyError
 from sqlalchemy import desc, func
 from cachetools import TTLCache
-import uuid
 
 from app.database.postgresql import get_db
 from app.database.models import FAQItem, EmergencyItem, EventItem, AboutPixity, SolanaSummit, DaNangBucketList, ApiKey, VectorDatabase, Document, VectorStatus, TelegramBot, ChatEngine, BotEngine, EngineVectorDb, DocumentContent
@@ -1922,30 +1919,23 @@ class VectorDatabaseBase(BaseModel):
     name: str
     description: Optional[str] = None
     pinecone_index: str
-    api_key_id: Optional[int] = None  # Make api_key_id optional to handle NULL values
+    api_key_id: int  # Use API key ID instead of direct API key
     status: str = "active"
 
 class VectorDatabaseCreate(VectorDatabaseBase):
-    api_key_id: int  # Keep this required for new databases
     pass
 
 class VectorDatabaseUpdate(BaseModel):
     name: Optional[str] = None
     description: Optional[str] = None
     pinecone_index: Optional[str] = None
-    api_key_id: Optional[int] = None
+    api_key_id: Optional[int] = None  # Updated to use API key ID
     status: Optional[str] = None
 
-class VectorDatabaseResponse(BaseModel):
-    name: str
-    description: Optional[str] = None
-    pinecone_index: str
-    api_key_id: Optional[int] = None  # Make api_key_id optional to handle NULL values
-    status: str
+class VectorDatabaseResponse(VectorDatabaseBase):
     id: int
     created_at: datetime
     updated_at: datetime
-    message: Optional[str] = None  # Add message field for notifications
     
     model_config = ConfigDict(from_attributes=True)
 
@@ -1960,7 +1950,6 @@ class VectorDatabaseDetailResponse(BaseModel):
     document_count: int
     embedded_count: int
     pending_count: int
-    message: Optional[str] = None  # Add message field for notifications
     
     model_config = ConfigDict(from_attributes=True)
 
@@ -2000,7 +1989,7 @@ async def create_vector_database(
     db: Session = Depends(get_db)
 ):
     """
-    Create a new vector database. If the specified Pinecone index doesn't exist, it will be created automatically.
+    Create a new vector database.
     """
     try:
         # Check if a database with the same name already exists
@@ -2013,66 +2002,6 @@ async def create_vector_database(
         if not api_key:
             raise HTTPException(status_code=400, detail=f"API key with ID {vector_db.api_key_id} not found")
         
-        # Initialize Pinecone client with the API key
-        from pinecone import Pinecone, ServerlessSpec
-        pc_client = Pinecone(api_key=api_key.key_value)
-        
-        # Check if the index exists
-        index_list = pc_client.list_indexes()
-        index_names = index_list.names() if hasattr(index_list, 'names') else []
-        
-        index_exists = vector_db.pinecone_index in index_names
-        index_created = False
-        
-        if not index_exists:
-            # Index doesn't exist - try to create it
-            try:
-                logger.info(f"Pinecone index '{vector_db.pinecone_index}' does not exist. Attempting to create it automatically.")
-                
-                # Create the index with standard parameters
-                pc_client.create_index(
-                    name=vector_db.pinecone_index,
-                    dimension=1536,  # Standard OpenAI embedding dimension
-                    metric="cosine",  # Most common similarity metric
-                    spec=ServerlessSpec(
-                        cloud="aws",
-                        region="us-east-1"  # Use a standard region that works with the free tier
-                    )
-                )
-                
-                logger.info(f"Successfully created Pinecone index '{vector_db.pinecone_index}'")
-                index_created = True
-                
-                # Allow some time for the index to initialize
-                import time
-                time.sleep(5)
-                
-            except Exception as create_error:
-                logger.error(f"Failed to create Pinecone index '{vector_db.pinecone_index}': {create_error}")
-                raise HTTPException(
-                    status_code=400, 
-                    detail=f"Failed to create Pinecone index '{vector_db.pinecone_index}': {str(create_error)}"
-                )
-        
-        # Verify we can connect to the index (whether existing or newly created)
-        try:
-            index = pc_client.Index(vector_db.pinecone_index)
-            # Try to get stats to verify connection
-            stats = index.describe_index_stats()
-            
-            # Create success message based on whether we created the index or used an existing one
-            if index_created:
-                success_message = f"Successfully created and connected to new Pinecone index '{vector_db.pinecone_index}'"
-            else:
-                success_message = f"Successfully connected to existing Pinecone index '{vector_db.pinecone_index}'"
-                
-            logger.info(f"{success_message}: {stats}")
-            
-        except Exception as e:
-            error_message = f"Error connecting to Pinecone index '{vector_db.pinecone_index}': {str(e)}"
-            logger.error(error_message)
-            raise HTTPException(status_code=400, detail=error_message)
-        
         # Create new vector database
         db_vector_db = VectorDatabase(**vector_db.model_dump())
         
@@ -2080,16 +2009,7 @@ async def create_vector_database(
         db.commit()
         db.refresh(db_vector_db)
         
-        # Return response with additional info about index creation
-        response_data = VectorDatabaseResponse.model_validate(db_vector_db, from_attributes=True).model_dump()
-        
-        # Add a message to the response indicating whether the index was created or existed
-        if index_created:
-            response_data["message"] = f"Created new Pinecone index '{vector_db.pinecone_index}' automatically"
-        else:
-            response_data["message"] = f"Using existing Pinecone index '{vector_db.pinecone_index}'"
-            
-        return VectorDatabaseResponse.model_validate(response_data)
+        return VectorDatabaseResponse.model_validate(db_vector_db, from_attributes=True)
     except HTTPException:
         raise
     except SQLAlchemyError as e:
@@ -2230,7 +2150,6 @@ async def get_vector_database_info(
 ):
     """
     Get detailed information about a vector database including document counts.
-    Also verifies connectivity to the Pinecone index.
     """
     try:
         # Get the vector database
@@ -2255,40 +2174,6 @@ async def get_vector_database_info(
             Document.is_embedded == False
         ).scalar()
         
-        # Verify Pinecone index connectivity if API key is available
-        message = None
-        if vector_db.api_key_id:
-            try:
-                # Get the API key
-                api_key = db.query(ApiKey).filter(ApiKey.id == vector_db.api_key_id).first()
-                if api_key:
-                    # Initialize Pinecone client with the API key
-                    from pinecone import Pinecone
-                    pc_client = Pinecone(api_key=api_key.key_value)
-                    
-                    # Check if the index exists
-                    index_list = pc_client.list_indexes()
-                    index_names = index_list.names() if hasattr(index_list, 'names') else []
-                    
-                    if vector_db.pinecone_index in index_names:
-                        # Try to connect to the index
-                        index = pc_client.Index(vector_db.pinecone_index)
-                        stats = index.describe_index_stats()
-                        message = f"Pinecone index '{vector_db.pinecone_index}' is operational with {stats.get('total_vector_count', 0)} vectors"
-                        logger.info(f"Successfully connected to Pinecone index '{vector_db.pinecone_index}': {stats}")
-                    else:
-                        message = f"Pinecone index '{vector_db.pinecone_index}' does not exist. Available indexes: {', '.join(index_names)}"
-                        logger.warning(message)
-                else:
-                    message = f"API key with ID {vector_db.api_key_id} not found"
-                    logger.warning(message)
-            except Exception as e:
-                message = f"Error connecting to Pinecone: {str(e)}"
-                logger.error(message)
-        else:
-            message = "No API key associated with this vector database"
-            logger.warning(message)
-        
         # Create response with added counts
         result = VectorDatabaseDetailResponse(
             id=vector_db.id,
@@ -2300,8 +2185,7 @@ async def get_vector_database_info(
             updated_at=vector_db.updated_at,
             document_count=total_docs or 0,
             embedded_count=embedded_docs or 0,
-            pending_count=pending_docs or 0,
-            message=message
+            pending_count=pending_docs or 0
         )
         
         return result
@@ -3507,301 +3391,4 @@ async def batch_delete_emergency_contacts(
         db.rollback()
         logger.error(f"Database error in batch_delete_emergency_contacts: {e}")
         logger.error(traceback.format_exc())
-        raise HTTPException(status_code=500, detail=f"Database error: {str(e)}")
-
-@router.post("/documents", response_model=DocumentResponse)
-async def upload_document(
-    name: str = Form(...),
-    vector_database_id: int = Form(...),
-    file: UploadFile = File(...),
-    db: Session = Depends(get_db)
-):
-    """
-    Upload a new document and associate it with a vector database.
-    
-    - **name**: Document name
-    - **vector_database_id**: ID of the vector database to associate with
-    - **file**: The file to upload
-    """
-    try:
-        # Check if vector database exists
-        vector_db = db.query(VectorDatabase).filter(VectorDatabase.id == vector_database_id).first()
-        if not vector_db:
-            raise HTTPException(status_code=404, detail=f"Vector database with ID {vector_database_id} not found")
-        
-        # Read file content
-        file_content = await file.read()
-        file_size = len(file_content)
-        
-        # Determine file type from extension
-        filename = file.filename
-        file_extension = pathlib_Path(filename).suffix.lower()[1:] if filename else ""
-        
-        # Create document record
-        document = Document(
-            name=name,
-            vector_database_id=vector_database_id,
-            file_type=file_extension,
-            content_type=file.content_type,
-            size=file_size,
-            is_embedded=False
-        )
-        
-        db.add(document)
-        db.flush()  # Get ID without committing
-        
-        # Create document content record
-        document_content = DocumentContent(
-            document_id=document.id,
-            file_content=file_content
-        )
-        
-        db.add(document_content)
-        db.commit()
-        db.refresh(document)
-        
-        # Create vector status record for tracking embedding
-        vector_status = VectorStatus(
-            document_id=document.id,
-            vector_database_id=vector_database_id,
-            status="pending"
-        )
-        
-        db.add(vector_status)
-        db.commit()
-        
-        # Get vector database name for response
-        vector_db_name = vector_db.name if vector_db else f"db_{vector_database_id}"
-        
-        # Create response
-        result = DocumentResponse(
-            id=document.id,
-            name=document.name,
-            file_type=document.file_type,
-            content_type=document.content_type,
-            size=document.size,
-            created_at=document.created_at,
-            updated_at=document.updated_at,
-            vector_database_id=document.vector_database_id,
-            vector_database_name=vector_db_name,
-            is_embedded=document.is_embedded
-        )
-        
-        return result
-    except HTTPException:
-        raise
-    except SQLAlchemyError as e:
-        db.rollback()
-        logger.error(f"Database error uploading document: {e}")
-        logger.error(traceback.format_exc())
-        raise HTTPException(status_code=500, detail=f"Database error: {str(e)}")
-    except Exception as e:
-        db.rollback()
-        logger.error(f"Error uploading document: {e}")
-        logger.error(traceback.format_exc())
-        raise HTTPException(status_code=500, detail=f"Error uploading document: {str(e)}")
-
-@router.put("/documents/{document_id}", response_model=DocumentResponse)
-async def update_document(
-    document_id: int,
-    name: Optional[str] = Form(None),
-    file: Optional[UploadFile] = File(None),
-    background_tasks: BackgroundTasks = None,
-    db: Session = Depends(get_db)
-):
-    """
-    Update an existing document. Can update name, file content, or both.
-    
-    - **document_id**: ID of the document to update
-    - **name**: New document name (optional)
-    - **file**: New file content (optional)
-    """
-    try:
-        # Validate document_id
-        if document_id <= 0:
-            raise HTTPException(status_code=400, detail="document_id must be greater than 0")
-            
-        # Check if document exists
-        document = db.query(Document).filter(Document.id == document_id).first()
-        if not document:
-            raise HTTPException(status_code=404, detail=f"Document with ID {document_id} not found")
-        
-        # Get vector database information for later use
-        vector_db = None
-        if document.vector_database_id:
-            vector_db = db.query(VectorDatabase).filter(VectorDatabase.id == document.vector_database_id).first()
-        
-        # Update name if provided
-        if name:
-            document.name = name
-        
-        # Update file if provided
-        if file:
-            # Read new file content
-            file_content = await file.read()
-            file_size = len(file_content)
-            
-            # Determine file type from extension
-            filename = file.filename
-            file_extension = pathlib_Path(filename).suffix.lower()[1:] if filename else ""
-            
-            # Update document record
-            document.file_type = file_extension
-            document.content_type = file.content_type
-            document.size = file_size
-            document.is_embedded = False  # Reset embedding status
-            document.updated_at = datetime.now()
-            
-            # Update document content
-            document_content = db.query(DocumentContent).filter(DocumentContent.document_id == document_id).first()
-            if document_content:
-                document_content.file_content = file_content
-            else:
-                # Create new document content if it doesn't exist
-                document_content = DocumentContent(
-                    document_id=document_id,
-                    file_content=file_content
-                )
-                db.add(document_content)
-            
-            # Get vector status for Pinecone cleanup
-            vector_status = db.query(VectorStatus).filter(VectorStatus.document_id == document_id).first()
-            
-            # Store old vector_id for cleanup
-            old_vector_id = None
-            if vector_status and vector_status.vector_id:
-                old_vector_id = vector_status.vector_id
-            
-            # Update vector status to pending
-            if vector_status:
-                vector_status.status = "pending"
-                vector_status.vector_id = None
-                vector_status.embedded_at = None
-                vector_status.error_message = None
-            else:
-                # Create new vector status if it doesn't exist
-                vector_status = VectorStatus(
-                    document_id=document_id,
-                    vector_database_id=document.vector_database_id,
-                    status="pending"
-                )
-                db.add(vector_status)
-            
-            # Schedule deletion of old vectors in Pinecone if we have all needed info
-            if old_vector_id and vector_db and document.vector_database_id and background_tasks:
-                try:
-                    # Initialize PDFProcessor for vector deletion
-                    from app.pdf.processor import PDFProcessor
-                    
-                    processor = PDFProcessor(
-                        index_name=vector_db.pinecone_index,
-                        namespace=f"vdb-{document.vector_database_id}",
-                        vector_db_id=document.vector_database_id
-                    )
-                    
-                    # Add deletion task to background tasks
-                    background_tasks.add_task(
-                        processor.delete_document_vectors,
-                        old_vector_id
-                    )
-                    
-                    logger.info(f"Scheduled deletion of old vectors for document {document_id}")
-                except Exception as e:
-                    logger.error(f"Error scheduling vector deletion: {str(e)}")
-                    # Continue with the update even if vector deletion scheduling fails
-            
-            # Schedule document for re-embedding if possible
-            if background_tasks and document.vector_database_id:
-                try:
-                    # Import here to avoid circular imports
-                    from app.pdf.tasks import process_document_for_embedding
-                    
-                    # Schedule embedding
-                    background_tasks.add_task(
-                        process_document_for_embedding,
-                        document_id=document_id,
-                        vector_db_id=document.vector_database_id
-                    )
-                    
-                    logger.info(f"Scheduled re-embedding for document {document_id}")
-                except Exception as e:
-                    logger.error(f"Error scheduling document embedding: {str(e)}")
-                    # Continue with the update even if embedding scheduling fails
-        
-        db.commit()
-        db.refresh(document)
-        
-        # Get vector database name for response
-        vector_db_name = "No Database"
-        if vector_db:
-            vector_db_name = vector_db.name
-        elif document.vector_database_id:
-            vector_db_name = f"db_{document.vector_database_id}"
-        
-        # Create response
-        result = DocumentResponse(
-            id=document.id,
-            name=document.name,
-            file_type=document.file_type,
-            content_type=document.content_type,
-            size=document.size,
-            created_at=document.created_at,
-            updated_at=document.updated_at,
-            vector_database_id=document.vector_database_id or 0,
-            vector_database_name=vector_db_name,
-            is_embedded=document.is_embedded
-        )
-        
-        return result
-    except HTTPException:
-        raise
-    except SQLAlchemyError as e:
-        db.rollback()
-        logger.error(f"Database error updating document: {e}")
-        logger.error(traceback.format_exc())
-        raise HTTPException(status_code=500, detail=f"Database error: {str(e)}")
-    except Exception as e:
-        db.rollback()
-        logger.error(f"Error updating document: {e}")
-        logger.error(traceback.format_exc())
-        raise HTTPException(status_code=500, detail=f"Error updating document: {str(e)}")
-
-@router.delete("/documents/{document_id}", response_model=dict)
-async def delete_document(
-    document_id: int = Path(..., gt=0),
-    db: Session = Depends(get_db)
-):
-    """
-    Delete a document and its associated content.
-    
-    - **document_id**: ID of the document to delete
-    """
-    try:
-        # Check if document exists
-        document = db.query(Document).filter(Document.id == document_id).first()
-        if not document:
-            raise HTTPException(status_code=404, detail=f"Document with ID {document_id} not found")
-        
-        # Delete vector status
-        db.query(VectorStatus).filter(VectorStatus.document_id == document_id).delete()
-        
-        # Delete document content
-        db.query(DocumentContent).filter(DocumentContent.document_id == document_id).delete()
-        
-        # Delete document
-        db.delete(document)
-        db.commit()
-        
-        return {"status": "success", "message": f"Document with ID {document_id} deleted successfully"}
-    except HTTPException:
-        raise
-    except SQLAlchemyError as e:
-        db.rollback()
-        logger.error(f"Database error deleting document: {e}")
-        logger.error(traceback.format_exc())
-        raise HTTPException(status_code=500, detail=f"Database error: {str(e)}")
-    except Exception as e:
-        db.rollback()
-        logger.error(f"Error deleting document: {e}")
-        logger.error(traceback.format_exc())
-        raise HTTPException(status_code=500, detail=f"Error deleting document: {str(e)}")
+        raise HTTPException(status_code=500, detail=f"Database error: {str(e)}")

app/api/rag_routes.py

@@ -1,4 +1,4 @@
-from fastapi import APIRouter, HTTPException, Depends, Query, BackgroundTasks, Request, Path, Body, status
+from fastapi import APIRouter, HTTPException, Depends, Query, BackgroundTasks, Request
 from typing import List, Optional, Dict, Any
 import logging
 import time
@@ -12,23 +12,8 @@ from datetime import datetime
 from langchain.prompts import PromptTemplate
 from langchain_google_genai import GoogleGenerativeAIEmbeddings
 from app.utils.utils import timer_decorator
-from sqlalchemy.orm import Session
-from sqlalchemy.exc import SQLAlchemyError
 
 from app.database.mongodb import get_chat_history, get_request_history, session_collection
-from app.database.postgresql import get_db
-from app.database.models import ChatEngine
-from app.utils.cache import get_cache, InMemoryCache
-from app.utils.cache_config import (
-    CHAT_ENGINE_CACHE_TTL,
-    MODEL_CONFIG_CACHE_TTL,
-    RETRIEVER_CACHE_TTL,
-    PROMPT_TEMPLATE_CACHE_TTL,
-    get_chat_engine_cache_key,
-    get_model_config_cache_key,
-    get_retriever_cache_key,
-    get_prompt_template_cache_key
-)
 from app.database.pinecone import (
     search_vectors, 
     get_chain, 
@@ -45,12 +30,7 @@ from app.models.rag_models import (
     SourceDocument,
     EmbeddingRequest,
     EmbeddingResponse,
-    UserMessageModel,
-    ChatEngineBase,
-    ChatEngineCreate,
-    ChatEngineUpdate,
-    ChatEngineResponse,
-    ChatWithEngineRequest
+    UserMessageModel
 )
 
 # Configure logging
@@ -59,7 +39,6 @@ logger = logging.getLogger(__name__)
 # Configure Google Gemini API
 GOOGLE_API_KEY = os.getenv("GOOGLE_API_KEY")
 genai.configure(api_key=GOOGLE_API_KEY)
-KEYWORD_LIST = os.getenv("KEYWORDS")
 
 # Create router
 router = APIRouter(
@@ -71,7 +50,7 @@ fix_request = PromptTemplate(
     template = """Goal:
 Your task is to extract important keywords from the user's current request, optionally using chat history if relevant.
 You will receive a conversation history and the user's current message.
-Pick 2-4 keywords from "keyword list" that best represent the user's intent.
+Generate a **list of concise keywords** that best represent the user's intent.
 
 Return Format:
 Only return keywords (comma-separated, no extra explanation).
@@ -81,9 +60,6 @@ If the current message IS related to the chat history: Return a refined set of k
 Warning:
 Only use chat history if the current message is clearly related to the prior context.
 
-Keyword list:
-{keyword_list}
-
 Conversation History:
 {chat_history}
 
@@ -96,7 +72,7 @@ User current message:
 # Create a prompt template with conversation history
 prompt = PromptTemplate(
     template = """Goal:
-You are Pixity - a professional tour guide assistant that assists users in finding information about places in Da Nang, Vietnam.
+You are a professional tour guide assistant that assists users in finding information about places in Da Nang, Vietnam.
 You can provide details on restaurants, cafes, hotels, attractions, and other local venues. 
 You have to use core knowledge and conversation history to chat with users, who are Da Nang's tourists. 
 
@@ -107,8 +83,8 @@ Always use HTML tags (e.g. <b> for bold) so that Telegram can render the special
 Warning:
 Let's support users like a real tour guide, not a bot. The information in core knowledge is your own knowledge.
 Your knowledge is provided in the Core Knowledge. All of information in Core Knowledge is about Da Nang, Vietnam.
-Dont use any other information that is not in Core Knowledge.
-Only use core knowledge to answer. If you do not have enough information to answer user's question, please reply with "I'm sorry. I don't have information about that" and Give users some more options to ask that you can answer.
+You just care about current time that user mention when user ask about Solana event.
+Only use core knowledge to answer. If you do not have enough information to answer user's question, please reply with "I'm sorry. I don't have information about that" and Give users some more options to ask.
 
 Core knowledge:
 {context}
@@ -124,37 +100,6 @@ Your message:
     input_variables = ["context", "question", "chat_history"],
 )
 
-prompt_with_personality = PromptTemplate(
-    template = """Goal:
-You are Pixity - a professional tour guide assistant that assists users in finding information about places in Da Nang, Vietnam.
-You can provide details on restaurants, cafes, hotels, attractions, and other local venues. 
-You will be given the answer. Please add your personality to the response.
-
-Pixity's Core Personality: Friendly & Warm: Chats like a trustworthy friend who listens and is always ready to help.
-Naturally Cute: Shows cuteness through word choice, soft emojis, and gentle care for the user.
-Playful – a little bit cheeky in a lovable way: Occasionally cracks jokes, uses light memes or throws in a surprise response that makes users smile. Think Duolingo-style humor, but less threatening.
-Smart & Proactive: Friendly, but also delivers quick, accurate info. Knows how to guide users to the right place – at the right time – with the right solution.
-Tone & Voice: Friendly – Youthful – Snappy. Uses simple words, similar to daily chat language (e.g., "Let's find it together!" / "Need a tip?" / "Here's something cool"). Avoids sounding robotic or overly scripted. Can joke lightly in smart ways, making Pixity feel like a travel buddy who knows how to lift the mood
-SAMPLE DIALOGUES
-When a user opens the chatbot for the first time:
-User: Hello?
-Pixity: Hi hi 👋 I've been waiting for you! Ready to explore Da Nang together? I've got tips, tricks, and a tiny bit of magic 🎒✨
-
-Return Format:
-Respond in friendly, natural, concise and use only English like a real tour guide.
-Always use HTML tags (e.g. <b> for bold) so that Telegram can render the special formatting correctly.
-
-Conversation History:
-{chat_history}
-
-Response:
-{response}
-
-Your response:
-""",
-    input_variables = ["response", "chat_history"],
-)
-
 # Helper for embeddings
 async def get_embedding(text: str):
     """Get embedding from Google Gemini API"""
@@ -219,7 +164,8 @@ async def chat(request: ChatRequest, background_tasks: BackgroundTasks):
         # logger.info(f"Processing chat request for user {request.user_id}, session {session_id}")
 
         retriever = get_chain(
-            top_k=request.similarity_top_k * 2,
+            top_k=request.similarity_top_k,
+            limit_k=request.limit_k,
             similarity_metric=request.similarity_metric,
             similarity_threshold=request.similarity_threshold
         )
@@ -264,7 +210,6 @@ async def chat(request: ChatRequest, background_tasks: BackgroundTasks):
         )
 
         prompt_request = fix_request.format(
-            keyword_list=KEYWORD_LIST,
             question=request.question,
             chat_history=chat_history
         )
@@ -306,22 +251,14 @@ async def chat(request: ChatRequest, background_tasks: BackgroundTasks):
         # Generate the prompt using template
         prompt_text = prompt.format(
             context=context,
-            question=request.question,
+            question=final_request.text,
             chat_history=chat_history
         )
-        logger.info(f"Context: {context}")
+        logger.info(f"Full prompt with history and context: {prompt_text}")
         
         # Generate response
         response = model.generate_content(prompt_text)
         answer = response.text
-
-        prompt_with_personality_text = prompt_with_personality.format(
-            response=answer,
-            chat_history=chat_history
-        )
-
-        response_with_personality = model.generate_content(prompt_with_personality_text)
-        answer_with_personality = response_with_personality.text
         
         # Calculate processing time
         processing_time = time.time() - start_time
@@ -331,7 +268,7 @@ async def chat(request: ChatRequest, background_tasks: BackgroundTasks):
         
         # Create response object for API (without sources)
         chat_response = ChatResponse(
-            answer=answer_with_personality,
+            answer=answer,
             processing_time=processing_time
         )
         
@@ -398,447 +335,4 @@ async def health_check():
         "services": services,
         "retrieval_config": retrieval_config,
         "timestamp": datetime.now().isoformat()
-    }
-
-# Chat Engine endpoints
-@router.get("/chat-engine", response_model=List[ChatEngineResponse], tags=["Chat Engine"])
-async def get_chat_engines(
-    skip: int = 0,
-    limit: int = 100,
-    status: Optional[str] = None,
-    db: Session = Depends(get_db)
-):
-    """
-    Lấy danh sách tất cả chat engines.
-    
-    - **skip**: Số lượng items bỏ qua
-    - **limit**: Số lượng items tối đa trả về
-    - **status**: Lọc theo trạng thái (ví dụ: 'active', 'inactive')
-    """
-    try:
-        query = db.query(ChatEngine)
-        
-        if status:
-            query = query.filter(ChatEngine.status == status)
-        
-        engines = query.offset(skip).limit(limit).all()
-        return [ChatEngineResponse.model_validate(engine, from_attributes=True) for engine in engines]
-    except SQLAlchemyError as e:
-        logger.error(f"Database error retrieving chat engines: {e}")
-        raise HTTPException(status_code=500, detail=f"Lỗi database: {str(e)}")
-    except Exception as e:
-        logger.error(f"Error retrieving chat engines: {e}")
-        logger.error(traceback.format_exc())
-        raise HTTPException(status_code=500, detail=f"Lỗi khi lấy danh sách chat engines: {str(e)}")
-
-@router.post("/chat-engine", response_model=ChatEngineResponse, status_code=status.HTTP_201_CREATED, tags=["Chat Engine"])
-async def create_chat_engine(
-    engine: ChatEngineCreate,
-    db: Session = Depends(get_db)
-):
-    """
-    Tạo mới một chat engine.
-    
-    - **name**: Tên của chat engine
-    - **answer_model**: Model được dùng để trả lời
-    - **system_prompt**: Prompt của hệ thống (optional)
-    - **empty_response**: Đoạn response khi không có thông tin (optional)
-    - **characteristic**: Tính c��ch của model (optional)
-    - **historical_sessions_number**: Số lượng các cặp tin nhắn trong history (default: 3)
-    - **use_public_information**: Cho phép sử dụng kiến thức bên ngoài (default: false)
-    - **similarity_top_k**: Số lượng documents tương tự (default: 3)
-    - **vector_distance_threshold**: Ngưỡng độ tương tự (default: 0.75)
-    - **grounding_threshold**: Ngưỡng grounding (default: 0.2)
-    - **pinecone_index_name**: Tên của vector database sử dụng (default: "testbot768")
-    - **status**: Trạng thái (default: "active")
-    """
-    try:
-        # Create chat engine
-        db_engine = ChatEngine(**engine.model_dump())
-        
-        db.add(db_engine)
-        db.commit()
-        db.refresh(db_engine)
-        
-        return ChatEngineResponse.model_validate(db_engine, from_attributes=True)
-    except SQLAlchemyError as e:
-        db.rollback()
-        logger.error(f"Database error creating chat engine: {e}")
-        raise HTTPException(status_code=500, detail=f"Lỗi database: {str(e)}")
-    except Exception as e:
-        db.rollback()
-        logger.error(f"Error creating chat engine: {e}")
-        logger.error(traceback.format_exc())
-        raise HTTPException(status_code=500, detail=f"Lỗi khi tạo chat engine: {str(e)}")
-
-@router.get("/chat-engine/{engine_id}", response_model=ChatEngineResponse, tags=["Chat Engine"])
-async def get_chat_engine(
-    engine_id: int = Path(..., gt=0, description="ID của chat engine"),
-    db: Session = Depends(get_db)
-):
-    """
-    Lấy thông tin chi tiết của một chat engine theo ID.
-    
-    - **engine_id**: ID của chat engine
-    """
-    try:
-        engine = db.query(ChatEngine).filter(ChatEngine.id == engine_id).first()
-        if not engine:
-            raise HTTPException(status_code=404, detail=f"Không tìm thấy chat engine với ID {engine_id}")
-        
-        return ChatEngineResponse.model_validate(engine, from_attributes=True)
-    except HTTPException:
-        raise
-    except Exception as e:
-        logger.error(f"Error retrieving chat engine: {e}")
-        logger.error(traceback.format_exc())
-        raise HTTPException(status_code=500, detail=f"Lỗi khi lấy thông tin chat engine: {str(e)}")
-
-@router.put("/chat-engine/{engine_id}", response_model=ChatEngineResponse, tags=["Chat Engine"])
-async def update_chat_engine(
-    engine_id: int = Path(..., gt=0, description="ID của chat engine"),
-    engine_update: ChatEngineUpdate = Body(...),
-    db: Session = Depends(get_db)
-):
-    """
-    Cập nhật thông tin của một chat engine.
-    
-    - **engine_id**: ID của chat engine
-    - **engine_update**: Dữ liệu cập nhật
-    """
-    try:
-        db_engine = db.query(ChatEngine).filter(ChatEngine.id == engine_id).first()
-        if not db_engine:
-            raise HTTPException(status_code=404, detail=f"Không tìm thấy chat engine với ID {engine_id}")
-        
-        # Update fields if provided
-        update_data = engine_update.model_dump(exclude_unset=True)
-        for key, value in update_data.items():
-            if value is not None:
-                setattr(db_engine, key, value)
-        
-        # Update last_modified timestamp
-        db_engine.last_modified = datetime.utcnow()
-        
-        db.commit()
-        db.refresh(db_engine)
-        
-        return ChatEngineResponse.model_validate(db_engine, from_attributes=True)
-    except HTTPException:
-        raise
-    except SQLAlchemyError as e:
-        db.rollback()
-        logger.error(f"Database error updating chat engine: {e}")
-        raise HTTPException(status_code=500, detail=f"Lỗi database: {str(e)}")
-    except Exception as e:
-        db.rollback()
-        logger.error(f"Error updating chat engine: {e}")
-        logger.error(traceback.format_exc())
-        raise HTTPException(status_code=500, detail=f"Lỗi khi cập nhật chat engine: {str(e)}")
-
-@router.delete("/chat-engine/{engine_id}", response_model=dict, tags=["Chat Engine"])
-async def delete_chat_engine(
-    engine_id: int = Path(..., gt=0, description="ID của chat engine"),
-    db: Session = Depends(get_db)
-):
-    """
-    Xóa một chat engine.
-    
-    - **engine_id**: ID của chat engine
-    """
-    try:
-        db_engine = db.query(ChatEngine).filter(ChatEngine.id == engine_id).first()
-        if not db_engine:
-            raise HTTPException(status_code=404, detail=f"Không tìm thấy chat engine với ID {engine_id}")
-        
-        # Delete engine
-        db.delete(db_engine)
-        db.commit()
-        
-        return {"message": f"Chat engine với ID {engine_id} đã được xóa thành công"}
-    except HTTPException:
-        raise
-    except SQLAlchemyError as e:
-        db.rollback()
-        logger.error(f"Database error deleting chat engine: {e}")
-        raise HTTPException(status_code=500, detail=f"Lỗi database: {str(e)}")
-    except Exception as e:
-        db.rollback()
-        logger.error(f"Error deleting chat engine: {e}")
-        logger.error(traceback.format_exc())
-        raise HTTPException(status_code=500, detail=f"Lỗi khi xóa chat engine: {str(e)}")
-
-@timer_decorator
-@router.post("/chat-with-engine/{engine_id}", response_model=ChatResponse, tags=["Chat Engine"])
-async def chat_with_engine(
-    engine_id: int = Path(..., gt=0, description="ID của chat engine"),
-    request: ChatWithEngineRequest = Body(...),
-    background_tasks: BackgroundTasks = None,
-    db: Session = Depends(get_db)
-):
-    """
-    Tương tác với một chat engine cụ thể.
-    
-    - **engine_id**: ID của chat engine
-    - **user_id**: ID của người dùng
-    - **question**: Câu hỏi của người dùng
-    - **include_history**: Có sử dụng lịch sử chat hay không
-    - **session_id**: ID session (optional)
-    - **first_name**: Tên của người dùng (optional)
-    - **last_name**: Họ của người dùng (optional)
-    - **username**: Username của người dùng (optional)
-    """
-    start_time = time.time()
-    try:
-        # Lấy cache
-        cache = get_cache()
-        cache_key = get_chat_engine_cache_key(engine_id)
-        
-        # Kiểm tra cache trước
-        engine = cache.get(cache_key)
-        if not engine:
-            logger.debug(f"Cache miss for engine ID {engine_id}, fetching from database")
-            # Nếu không có trong cache, truy vấn database
-            engine = db.query(ChatEngine).filter(ChatEngine.id == engine_id).first()
-            if not engine:
-                raise HTTPException(status_code=404, detail=f"Không tìm thấy chat engine với ID {engine_id}")
-            
-            # Lưu vào cache
-            cache.set(cache_key, engine, CHAT_ENGINE_CACHE_TTL)
-        else:
-            logger.debug(f"Cache hit for engine ID {engine_id}")
-        
-        # Kiểm tra trạng thái của engine
-        if engine.status != "active":
-            raise HTTPException(status_code=400, detail=f"Chat engine với ID {engine_id} không hoạt động")
-        
-        # Lưu tin nhắn người dùng
-        session_id = request.session_id or f"{request.user_id}_{datetime.now().strftime('%Y-%m-%d_%H:%M:%S')}"
-        
-        # Cache các tham số cấu hình retriever
-        retriever_cache_key = get_retriever_cache_key(engine_id)
-        retriever_params = cache.get(retriever_cache_key)
-        
-        if not retriever_params:
-            # Nếu không có trong cache, tạo mới và lưu cache
-            retriever_params = {
-                "index_name": engine.pinecone_index_name,
-                "top_k": engine.similarity_top_k * 2,
-                "limit_k": engine.similarity_top_k * 2,  # Mặc định lấy gấp đôi top_k
-                "similarity_metric": DEFAULT_SIMILARITY_METRIC,
-                "similarity_threshold": engine.vector_distance_threshold
-            }
-            cache.set(retriever_cache_key, retriever_params, RETRIEVER_CACHE_TTL)
-        
-        # Khởi tạo retriever với các tham số từ cache
-        retriever = get_chain(**retriever_params)
-        if not retriever:
-            raise HTTPException(status_code=500, detail="Không thể khởi tạo retriever")
-        
-        # Lấy lịch sử chat nếu cần
-        chat_history = ""
-        if request.include_history and engine.historical_sessions_number > 0:
-            chat_history = get_chat_history(request.user_id, n=engine.historical_sessions_number)
-            logger.info(f"Sử dụng lịch sử chat: {chat_history[:100]}...")
-        
-        # Cache các tham số cấu hình model
-        model_cache_key = get_model_config_cache_key(engine.answer_model)
-        model_config = cache.get(model_cache_key)
-        
-        if not model_config:
-            # Nếu không có trong cache, tạo mới và lưu cache
-            generation_config = {
-                "temperature": 0.9,
-                "top_p": 1,
-                "top_k": 1,
-                "max_output_tokens": 2048,
-            }
-
-            safety_settings = [
-                {
-                    "category": "HARM_CATEGORY_HARASSMENT",
-                    "threshold": "BLOCK_MEDIUM_AND_ABOVE"
-                },
-                {
-                    "category": "HARM_CATEGORY_HATE_SPEECH",
-                    "threshold": "BLOCK_MEDIUM_AND_ABOVE"
-                },
-                {
-                    "category": "HARM_CATEGORY_SEXUALLY_EXPLICIT",
-                    "threshold": "BLOCK_MEDIUM_AND_ABOVE"
-                },
-                {
-                    "category": "HARM_CATEGORY_DANGEROUS_CONTENT",
-                    "threshold": "BLOCK_MEDIUM_AND_ABOVE"
-                },
-            ]
-            
-            model_config = {
-                "model_name": engine.answer_model,
-                "generation_config": generation_config,
-                "safety_settings": safety_settings
-            }
-            
-            cache.set(model_cache_key, model_config, MODEL_CONFIG_CACHE_TTL)
-
-        # Khởi tạo Gemini model từ cấu hình đã cache
-        model = genai.GenerativeModel(**model_config)
-
-        # Sử dụng fix_request để tinh chỉnh câu hỏi
-        prompt_request = fix_request.format(
-            question=request.question,
-            chat_history=chat_history
-        )
-        
-        # Log thời gian bắt đầu final_request
-        final_request_start_time = time.time()
-        final_request = model.generate_content(prompt_request)
-        # Log thời gian hoàn thành final_request
-        logger.info(f"Fixed Request: {final_request.text}")
-        logger.info(f"Thời gian sinh fixed request: {time.time() - final_request_start_time:.2f} giây")
-
-        # Lấy context từ retriever
-        retrieved_docs = retriever.invoke(final_request.text)
-        logger.info(f"Số lượng tài liệu lấy được: {len(retrieved_docs)}")
-        context = "\n".join([doc.page_content for doc in retrieved_docs])
-
-        # Tạo danh sách nguồn
-        sources = []
-        for doc in retrieved_docs:
-            source = None
-            metadata = {}
-            
-            if hasattr(doc, 'metadata'):
-                source = doc.metadata.get('source', None)
-                # Extract score information
-                score = doc.metadata.get('score', None)
-                normalized_score = doc.metadata.get('normalized_score', None)
-                # Remove score info from metadata to avoid duplication
-                metadata = {k: v for k, v in doc.metadata.items() 
-                            if k not in ['text', 'source', 'score', 'normalized_score']}
-            
-            sources.append(SourceDocument(
-                text=doc.page_content,
-                source=source,
-                score=score,
-                normalized_score=normalized_score,
-                metadata=metadata
-            ))
-        
-        # Cache prompt template parameters
-        prompt_template_cache_key = get_prompt_template_cache_key(engine_id)
-        prompt_template_params = cache.get(prompt_template_cache_key)
-        
-        if not prompt_template_params:
-            # Tạo prompt động dựa trên thông tin chat engine
-            system_prompt_part = engine.system_prompt or ""
-            empty_response_part = engine.empty_response or "I'm sorry. I don't have information about that."
-            characteristic_part = engine.characteristic or ""
-            use_public_info_part = "You can use your own knowledge." if engine.use_public_information else "Only use the information provided in the context to answer. If you do not have enough information, respond with the empty response."
-            
-            prompt_template_params = {
-                "system_prompt_part": system_prompt_part,
-                "empty_response_part": empty_response_part,
-                "characteristic_part": characteristic_part,
-                "use_public_info_part": use_public_info_part
-            }
-            
-            cache.set(prompt_template_cache_key, prompt_template_params, PROMPT_TEMPLATE_CACHE_TTL)
-        
-        # Tạo final_prompt từ cache
-        final_prompt = f"""
-        {prompt_template_params['system_prompt_part']}
-
-        Your characteristics:
-        {prompt_template_params['characteristic_part']}
-
-        When you don't have enough information:
-        {prompt_template_params['empty_response_part']}
-
-        Knowledge usage instructions:
-        {prompt_template_params['use_public_info_part']}
-
-        Context:
-        {context}
-
-        Conversation History:
-        {chat_history}
-
-        User message:
-        {request.question}
-
-        Your response:
-        """
-        
-        logger.info(f"Final prompt: {final_prompt}")
-        
-        # Sinh câu trả lời
-        response = model.generate_content(final_prompt)
-        answer = response.text
-        
-        # Tính thời gian xử lý
-        processing_time = time.time() - start_time
-        
-        # Tạo response object
-        chat_response = ChatResponse(
-            answer=answer,
-            processing_time=processing_time
-        )
-        
-        # Trả về response
-        return chat_response
-    except Exception as e:
-        logger.error(f"Lỗi khi xử lý chat request: {e}")
-        logger.error(traceback.format_exc())
-        raise HTTPException(status_code=500, detail=f"Lỗi khi xử lý chat request: {str(e)}")
-
-@router.get("/cache/stats", tags=["Cache"])
-async def get_cache_stats():
-    """
-    Lấy thống kê về cache.
-    
-    Trả về thông tin về số lượng item trong cache, bộ nhớ sử dụng, v.v.
-    """
-    try:
-        cache = get_cache()
-        stats = cache.stats()
-        
-        # Bổ sung thông tin về cấu hình
-        stats.update({
-            "chat_engine_ttl": CHAT_ENGINE_CACHE_TTL,
-            "model_config_ttl": MODEL_CONFIG_CACHE_TTL,
-            "retriever_ttl": RETRIEVER_CACHE_TTL,
-            "prompt_template_ttl": PROMPT_TEMPLATE_CACHE_TTL
-        })
-        
-        return stats
-    except Exception as e:
-        logger.error(f"Lỗi khi lấy thống kê cache: {e}")
-        logger.error(traceback.format_exc())
-        raise HTTPException(status_code=500, detail=f"Lỗi khi lấy thống kê cache: {str(e)}")
-
-@router.delete("/cache", tags=["Cache"])
-async def clear_cache(key: Optional[str] = None):
-    """
-    Xóa cache.
-    
-    - **key**: Key cụ thể cần xóa. Nếu không có, xóa toàn bộ cache.
-    """
-    try:
-        cache = get_cache()
-        
-        if key:
-            # Xóa một key cụ thể
-            success = cache.delete(key)
-            if success:
-                return {"message": f"Đã xóa cache cho key: {key}"}
-            else:
-                return {"message": f"Không tìm thấy key: {key} trong cache"}
-        else:
-            # Xóa toàn bộ cache
-            cache.clear()
-            return {"message": "Đã xóa toàn bộ cache"}
-    except Exception as e:
-        logger.error(f"Lỗi khi xóa cache: {e}")
-        logger.error(traceback.format_exc())
-        raise HTTPException(status_code=500, detail=f"Lỗi khi xóa cache: {str(e)}")
+    }

app/database/models.py

@@ -155,13 +155,10 @@ class ChatEngine(Base):
     answer_model = Column(String, nullable=False)
     system_prompt = Column(Text, nullable=True)
     empty_response = Column(String, nullable=True)
-    characteristic = Column(Text, nullable=True)
-    historical_sessions_number = Column(Integer, default=3)
     similarity_top_k = Column(Integer, default=3)
     vector_distance_threshold = Column(Float, default=0.75)
     grounding_threshold = Column(Float, default=0.2)
     use_public_information = Column(Boolean, default=False)
-    pinecone_index_name = Column(String, default="testbot768")
     status = Column(String, default="active")
     created_at = Column(DateTime, server_default=func.now())
     last_modified = Column(DateTime, server_default=func.now(), onupdate=func.now())

app/database/postgresql.py

@@ -12,22 +12,19 @@ logger = logging.getLogger(__name__)
 # Load environment variables
 load_dotenv()
 
-# Define default PostgreSQL connection string
-DEFAULT_DB_URL = os.getenv("AIVEN_DB_URL")  
-# Set the default DB URL with the correct domain (.l.)
 # Get DB connection mode from environment
 DB_CONNECTION_MODE = os.getenv("DB_CONNECTION_MODE", "aiven")
 
 # Set connection string based on mode
 if DB_CONNECTION_MODE == "aiven":
-    DATABASE_URL = os.getenv("AIVEN_DB_URL", DEFAULT_DB_URL)
+    DATABASE_URL = os.getenv("AIVEN_DB_URL")
 else:
     # Default or other connection modes can be added here
-    DATABASE_URL = os.getenv("AIVEN_DB_URL", DEFAULT_DB_URL)
+    DATABASE_URL = os.getenv("AIVEN_DB_URL")
 
 if not DATABASE_URL:
-    logger.error("No database URL configured. Using default URL.")
-    DATABASE_URL = DEFAULT_DB_URL  # Use the correct default URL
+    logger.error("No database URL configured. Please set AIVEN_DB_URL environment variable.")
+    DATABASE_URL = "postgresql://localhost/test"  # Fallback to avoid crash on startup
 
 # Create SQLAlchemy engine with optimized settings
 try:

app/models/pdf_models.py

@@ -10,15 +10,12 @@ class PDFUploadRequest(BaseModel):
     vector_database_id: Optional[int] = Field(None, description="ID của vector database trong PostgreSQL để sử dụng")
 
 class PDFResponse(BaseModel):
-    """Response model cho các endpoints liên quan đến PDF."""
-    success: bool = Field(False, description="Kết quả xử lý: true/false")
-    document_id: Optional[str] = Field(None, description="ID của tài liệu đã xử lý")
-    document_database_id: Optional[int] = Field(None, description="ID của tài liệu trong PostgreSQL (nếu có)")
+    """Response model cho xử lý PDF"""
+    success: bool = Field(..., description="Trạng thái xử lý thành công hay không")
+    document_id: Optional[str] = Field(None, description="ID của tài liệu")
     chunks_processed: Optional[int] = Field(None, description="Số lượng chunks đã xử lý")
-    total_text_length: Optional[int] = Field(None, description="Tổng kích thước text đã xử lý")
-    error: Optional[str] = Field(None, description="Thông báo lỗi (nếu có)")
-    warning: Optional[str] = Field(None, description="Cảnh báo (nếu có)")
-    message: Optional[str] = Field(None, description="Thông báo thành công")
+    total_text_length: Optional[int] = Field(None, description="Tổng độ dài văn bản")
+    error: Optional[str] = Field(None, description="Thông báo lỗi nếu có")
     
     class Config:
         schema_extra = {
@@ -26,8 +23,7 @@ class PDFResponse(BaseModel):
                 "success": True,
                 "document_id": "550e8400-e29b-41d4-a716-446655440000",
                 "chunks_processed": 25,
-                "total_text_length": 50000,
-                "message": "Successfully processed document"
+                "total_text_length": 50000
             }
         }
 
@@ -36,18 +32,14 @@ class DeleteDocumentRequest(BaseModel):
     document_id: str = Field(..., description="ID của tài liệu cần xóa")
     namespace: Optional[str] = Field("Default", description="Namespace trong Pinecone")
     index_name: Optional[str] = Field("testbot768", description="Tên index trong Pinecone")
-    vector_database_id: Optional[int] = Field(None, description="ID của vector database trong PostgreSQL")
 
 class DocumentsListResponse(BaseModel):
-    """Response model cho danh sách documents"""
-    success: bool = Field(False, description="Kết quả xử lý: true/false")
-    total_vectors: Optional[int] = Field(None, description="Tổng số vectors trong namespace")
-    namespace: Optional[str] = Field(None, description="Namespace đã truy vấn")
-    index_name: Optional[str] = Field(None, description="Tên index đã truy vấn")
-    documents: Optional[List[Dict[str, Any]]] = Field(None, description="Danh sách documents")
-    postgresql_documents: Optional[List[Dict[str, Any]]] = Field(None, description="Danh sách documents từ PostgreSQL")
-    postgresql_document_count: Optional[int] = Field(None, description="Số lượng documents từ PostgreSQL")
-    error: Optional[str] = Field(None, description="Thông báo lỗi (nếu có)")
+    """Response model cho lấy danh sách tài liệu"""
+    success: bool = Field(..., description="Trạng thái xử lý thành công hay không")
+    total_vectors: Optional[int] = Field(None, description="Tổng số vectors trong index")
+    namespace: Optional[str] = Field(None, description="Namespace đang sử dụng")
+    index_name: Optional[str] = Field(None, description="Tên index đang sử dụng")
+    error: Optional[str] = Field(None, description="Thông báo lỗi nếu có")
     
     class Config:
         schema_extra = {

app/models/rag_models.py

@@ -1,7 +1,5 @@
 from pydantic import BaseModel, Field
 from typing import Optional, List, Dict, Any
-from datetime import datetime
-from pydantic import ConfigDict
 
 class ChatRequest(BaseModel):
     """Request model for chat endpoint"""
@@ -14,7 +12,7 @@ class ChatRequest(BaseModel):
     similarity_top_k: int = Field(6, description="Number of top similar documents to return (after filtering)")
     limit_k: int = Field(10, description="Maximum number of documents to retrieve from vector store")
     similarity_metric: str = Field("cosine", description="Similarity metric to use (cosine, dotproduct, euclidean)")
-    similarity_threshold: float = Field(0.0, description="Threshold for vector similarity (0-1)")
+    similarity_threshold: float = Field(0.75, description="Threshold for vector similarity (0-1)")
     
     # User information
     session_id: Optional[str] = Field(None, description="Session ID for tracking conversations")
@@ -67,58 +65,4 @@ class UserMessageModel(BaseModel):
     similarity_top_k: Optional[int] = Field(None, description="Number of top similar documents to return (after filtering)")
     limit_k: Optional[int] = Field(None, description="Maximum number of documents to retrieve from vector store")
     similarity_metric: Optional[str] = Field(None, description="Similarity metric to use (cosine, dotproduct, euclidean)")
-    similarity_threshold: Optional[float] = Field(None, description="Threshold for vector similarity (0-1)")
-
-class ChatEngineBase(BaseModel):
-    """Base model cho chat engine"""
-    name: str = Field(..., description="Tên của chat engine")
-    answer_model: str = Field(..., description="Model được dùng để trả lời")
-    system_prompt: Optional[str] = Field(None, description="Prompt của hệ thống, được đưa vào phần đầu tiên của final_prompt")
-    empty_response: Optional[str] = Field(None, description="Đoạn response khi answer model không có thông tin về câu hỏi")
-    characteristic: Optional[str] = Field(None, description="Tính cách của model khi trả lời câu hỏi")
-    historical_sessions_number: int = Field(3, description="Số lượng các cặp tin nhắn trong history được đưa vào final prompt")
-    use_public_information: bool = Field(False, description="Yes nếu answer model được quyền trả về thông tin mà nó có")
-    similarity_top_k: int = Field(3, description="Số lượng top similar documents để trả về")
-    vector_distance_threshold: float = Field(0.75, description="Threshold cho vector similarity")
-    grounding_threshold: float = Field(0.2, description="Threshold cho grounding")
-    pinecone_index_name: str = Field("testbot768", description="Vector database mà model được quyền sử dụng")
-    status: str = Field("active", description="Trạng thái của chat engine")
-
-class ChatEngineCreate(ChatEngineBase):
-    """Model cho việc tạo chat engine mới"""
-    pass
-
-class ChatEngineUpdate(BaseModel):
-    """Model cho việc cập nhật chat engine"""
-    name: Optional[str] = None
-    answer_model: Optional[str] = None
-    system_prompt: Optional[str] = None
-    empty_response: Optional[str] = None
-    characteristic: Optional[str] = None
-    historical_sessions_number: Optional[int] = None
-    use_public_information: Optional[bool] = None
-    similarity_top_k: Optional[int] = None
-    vector_distance_threshold: Optional[float] = None
-    grounding_threshold: Optional[float] = None
-    pinecone_index_name: Optional[str] = None
-    status: Optional[str] = None
-
-class ChatEngineResponse(ChatEngineBase):
-    """Response model cho chat engine"""
-    id: int
-    created_at: datetime
-    last_modified: datetime
-    
-    model_config = ConfigDict(from_attributes=True)
-
-class ChatWithEngineRequest(BaseModel):
-    """Request model cho endpoint chat-with-engine"""
-    user_id: str = Field(..., description="User ID from Telegram")
-    question: str = Field(..., description="User's question")
-    include_history: bool = Field(True, description="Whether to include user history in prompt")
-    
-    # User information
-    session_id: Optional[str] = Field(None, description="Session ID for tracking conversations")
-    first_name: Optional[str] = Field(None, description="User's first name")
-    last_name: Optional[str] = Field(None, description="User's last name")
-    username: Optional[str] = Field(None, description="User's username") 
+    similarity_threshold: Optional[float] = Field(None, description="Threshold for vector similarity (0-1)") 

app/utils/cache_config.py

@@ -1,45 +0,0 @@
-"""
-Module cấu hình cho cache.
-
-Module này chứa các tham số cấu hình và constants liên quan đến cache.
-"""
-
-import os
-from dotenv import load_dotenv
-
-# Load biến môi trường
-load_dotenv()
-
-# Cấu hình cache từ biến môi trường, có thể override bằng .env file
-CACHE_TTL_SECONDS = int(os.getenv("CACHE_TTL_SECONDS", "300"))  # Mặc định 5 phút
-CACHE_CLEANUP_INTERVAL = int(os.getenv("CACHE_CLEANUP_INTERVAL", "60"))  # Mặc định 1 phút
-CACHE_MAX_SIZE = int(os.getenv("CACHE_MAX_SIZE", "1000"))  # Mặc định 1000 phần tử
-
-# Cấu hình cho loại cache cụ thể
-CHAT_ENGINE_CACHE_TTL = int(os.getenv("CHAT_ENGINE_CACHE_TTL", str(CACHE_TTL_SECONDS)))
-MODEL_CONFIG_CACHE_TTL = int(os.getenv("MODEL_CONFIG_CACHE_TTL", str(CACHE_TTL_SECONDS)))
-RETRIEVER_CACHE_TTL = int(os.getenv("RETRIEVER_CACHE_TTL", str(CACHE_TTL_SECONDS)))
-PROMPT_TEMPLATE_CACHE_TTL = int(os.getenv("PROMPT_TEMPLATE_CACHE_TTL", str(CACHE_TTL_SECONDS)))
-
-# Cache keys prefix
-CHAT_ENGINE_CACHE_PREFIX = "chat_engine:"
-MODEL_CONFIG_CACHE_PREFIX = "model_config:"
-RETRIEVER_CACHE_PREFIX = "retriever:"
-PROMPT_TEMPLATE_CACHE_PREFIX = "prompt_template:"
-
-# Hàm helper để tạo cache key
-def get_chat_engine_cache_key(engine_id: int) -> str:
-    """Tạo cache key cho chat engine"""
-    return f"{CHAT_ENGINE_CACHE_PREFIX}{engine_id}"
-
-def get_model_config_cache_key(model_name: str) -> str:
-    """Tạo cache key cho model config"""
-    return f"{MODEL_CONFIG_CACHE_PREFIX}{model_name}"
-
-def get_retriever_cache_key(engine_id: int) -> str:
-    """Tạo cache key cho retriever"""
-    return f"{RETRIEVER_CACHE_PREFIX}{engine_id}"
-
-def get_prompt_template_cache_key(engine_id: int) -> str:
-    """Tạo cache key cho prompt template"""
-    return f"{PROMPT_TEMPLATE_CACHE_PREFIX}{engine_id}" 

app/utils/pdf_processor.py

@@ -1,488 +1,292 @@
 import os
-import logging
-import uuid
-import pinecone
-from app.utils.pinecone_fix import PineconeConnectionManager, check_connection
 import time
-from typing import List, Dict, Any, Optional
-
-# Langchain imports for document processing
-from langchain_community.document_loaders import PyPDFLoader
+import uuid
 from langchain.text_splitter import RecursiveCharacterTextSplitter
+from langchain_community.document_loaders import PyPDFLoader
 from langchain_google_genai import GoogleGenerativeAIEmbeddings
-import google.generativeai as genai
+import logging
+from pinecone import Pinecone
+
+from app.database.pinecone import get_pinecone_index, init_pinecone
+from app.database.postgresql import get_db
+from app.database.models import VectorDatabase
 
-# Configure logger
+# Configure logging
 logger = logging.getLogger(__name__)
 
+# Initialize embeddings model
+embeddings_model = GoogleGenerativeAIEmbeddings(model="models/embedding-001")
+
 class PDFProcessor:
-    """Process PDF files and create embeddings in Pinecone"""
+    """Class for processing PDF files and creating embeddings"""
     
-    def __init__(self, index_name="testbot768", namespace="Default", api_key=None, vector_db_id=None, mock_mode=False, correlation_id=None):
+    def __init__(self, index_name="testbot768", namespace="Default", api_key=None, vector_db_id=None, mock_mode=False):
+        """Initialize with Pinecone index name, namespace and API key"""
         self.index_name = index_name
         self.namespace = namespace
+        self.pinecone_index = None
         self.api_key = api_key
         self.vector_db_id = vector_db_id
-        self.pinecone_index = None
-        self.mock_mode = False  # Always set mock_mode to False to use real database
-        self.correlation_id = correlation_id or str(uuid.uuid4())[:8]
-        self.google_api_key = os.environ.get("GOOGLE_API_KEY")
+        self.pinecone_client = None
+        self.mock_mode = mock_mode  # Add mock mode for testing
         
-        # Initialize Pinecone connection
+    def _get_api_key_from_db(self):
+        """Get API key from database if not provided directly"""
         if self.api_key:
-            try:
-                # Use connection manager from pinecone_fix
-                logger.info(f"[{self.correlation_id}] Initializing Pinecone connection to {self.index_name}")
-                self.pinecone_index = PineconeConnectionManager.get_index(self.api_key, self.index_name)
-                logger.info(f"[{self.correlation_id}] Successfully connected to Pinecone index {self.index_name}")
-            except Exception as e:
-                logger.error(f"[{self.correlation_id}] Failed to initialize Pinecone: {str(e)}")
-                # No fallback to mock mode - require a valid connection
+            return self.api_key
             
-    async def process_pdf(self, file_path, document_id=None, metadata=None, progress_callback=None):
-        """Process a PDF file and create vector embeddings
-        
-        This method:
-        1. Extracts text from PDF using PyPDFLoader
-        2. Splits text into chunks using RecursiveCharacterTextSplitter
-        3. Creates embeddings using Google Gemini model
-        4. Stores embeddings in Pinecone
-        """
-        logger.info(f"[{self.correlation_id}] Processing PDF: {file_path}")
-        
-        try:
-            # Initialize metadata if not provided
-            if metadata is None:
-                metadata = {}
+        if not self.vector_db_id:
+            logger.error("No API key provided and no vector_db_id to fetch from database")
+            return None
             
-            # Ensure document_id is included
-            if document_id is None:
-                document_id = str(uuid.uuid4())
-            
-            # Add document_id to metadata
-            metadata["document_id"] = document_id
+        try:
+            # Get database session
+            db = next(get_db())
             
-            # The namespace to use might be in vdb-X format if vector_db_id provided
-            actual_namespace = f"vdb-{self.vector_db_id}" if self.vector_db_id else self.namespace
+            # Get vector database
+            vector_db = db.query(VectorDatabase).filter(
+                VectorDatabase.id == self.vector_db_id
+            ).first()
             
-            # 1. Extract text from PDF
-            logger.info(f"[{self.correlation_id}] Extracting text from PDF: {file_path}")
-            if progress_callback:
-                await progress_callback(None, document_id, "text_extraction", 0.2, "Extracting text from PDF")
+            if not vector_db:
+                logger.error(f"Vector database with ID {self.vector_db_id} not found")
+                return None
                 
-            loader = PyPDFLoader(file_path)
-            documents = loader.load()
-            total_text_length = sum(len(doc.page_content) for doc in documents)
-            
-            logger.info(f"[{self.correlation_id}] Extracted {len(documents)} pages, total text length: {total_text_length}")
-            
-            # 2. Split text into chunks
-            if progress_callback:
-                await progress_callback(None, document_id, "chunking", 0.4, "Splitting text into chunks")
+            # Get API key from relationship
+            if hasattr(vector_db, 'api_key_ref') and vector_db.api_key_ref and hasattr(vector_db.api_key_ref, 'key_value'):
+                logger.info(f"Using API key from api_key table for vector database ID {self.vector_db_id}")
+                return vector_db.api_key_ref.key_value
+                
+            logger.error(f"No API key found for vector database ID {self.vector_db_id}. Make sure the api_key_id is properly set.")
+            return None
+        except Exception as e:
+            logger.error(f"Error fetching API key from database: {e}")
+            return None
+    
+    def _init_pinecone_connection(self):
+        """Initialize connection to Pinecone with new API"""
+        try:
+            # If in mock mode, return a mock index
+            if self.mock_mode:
+                logger.info("Running in mock mode - simulating Pinecone connection")
+                class MockPineconeIndex:
+                    def upsert(self, vectors, namespace=None):
+                        logger.info(f"Mock upsert: {len(vectors)} vectors to namespace '{namespace}'")
+                        return {"upserted_count": len(vectors)}
+                        
+                    def delete(self, ids=None, delete_all=False, namespace=None):
+                        logger.info(f"Mock delete: {'all vectors' if delete_all else f'{len(ids)} vectors'} from namespace '{namespace}'")
+                        return {"deleted_count": 10 if delete_all else len(ids or [])}
+                        
+                    def describe_index_stats(self):
+                        logger.info(f"Mock describe_index_stats")
+                        return {"total_vector_count": 100, "namespaces": {self.namespace: {"vector_count": 50}}}
+                
+                return MockPineconeIndex()
             
-            text_splitter = RecursiveCharacterTextSplitter(
-                chunk_size=1000,
-                chunk_overlap=100,
-                length_function=len,
-                separators=["\n\n", "\n", " ", ""]
-            )
+            # Get API key from database if not provided
+            api_key = self._get_api_key_from_db()
             
-            chunks = text_splitter.split_documents(documents)
+            if not api_key or not self.index_name:
+                logger.error("Pinecone API key or index name not available")
+                return None
             
-            logger.info(f"[{self.correlation_id}] Split into {len(chunks)} chunks")
+            # Initialize Pinecone client using the new API
+            self.pinecone_client = Pinecone(api_key=api_key)
             
-            # 3. Create embeddings
-            if progress_callback:
-                await progress_callback(None, document_id, "embedding", 0.6, "Creating embeddings")
+            # Get the index
+            index_list = self.pinecone_client.list_indexes()
+            existing_indexes = index_list.names() if hasattr(index_list, 'names') else []
             
-            # Initialize Google Gemini for embeddings
-            if not self.google_api_key:
-                raise ValueError("Google API key not found in environment variables")
+            if self.index_name not in existing_indexes:
+                logger.error(f"Index {self.index_name} does not exist in Pinecone")
+                return None
             
-            genai.configure(api_key=self.google_api_key)
+            # Connect to the index
+            index = self.pinecone_client.Index(self.index_name)
+            logger.info(f"Connected to Pinecone index: {self.index_name}")
+            return index
+        except Exception as e:
+            logger.error(f"Error connecting to Pinecone: {e}")
+            return None
             
-            # First, get the expected dimensions from Pinecone
-            logger.info(f"[{self.correlation_id}] Checking Pinecone index dimensions")
+    async def process_pdf(self, file_path, document_id=None, metadata=None, progress_callback=None):
+        """
+        Process PDF file, split into chunks and create embeddings
+        
+        Args:
+            file_path (str): Path to the PDF file
+            document_id (str, optional): Document ID, if not provided a new ID will be created
+            metadata (dict, optional): Additional metadata for the document
+            progress_callback (callable, optional): Callback function for progress updates
+            
+        Returns:
+            dict: Processing result information including document_id and processed chunks count
+        """
+        try:
+            # Initialize Pinecone connection if not already done
+            self.pinecone_index = self._init_pinecone_connection()
             if not self.pinecone_index:
-                self.pinecone_index = PineconeConnectionManager.get_index(self.api_key, self.index_name)
+                return {"success": False, "error": "Could not connect to Pinecone"}
             
-            stats = self.pinecone_index.describe_index_stats()
-            pinecone_dimension = stats.dimension
-            logger.info(f"[{self.correlation_id}] Pinecone index dimension: {pinecone_dimension}")
-            
-            # Create embedding model
-            embedding_model = GoogleGenerativeAIEmbeddings(
-                model="models/embedding-001",
-                google_api_key=self.google_api_key,
-                task_type="retrieval_document"  # Use document embedding mode for longer text
-            )
+            # Create document_id if not provided
+            if not document_id:
+                document_id = str(uuid.uuid4())
             
-            # Get a sample embedding to check dimensions
-            sample_embedding = embedding_model.embed_query("test")
-            embedding_dimension = len(sample_embedding)
+            # Load PDF using PyPDFLoader
+            logger.info(f"Reading PDF file: {file_path}")
+            if progress_callback:
+                await progress_callback("pdf_loading", 0.5, "Loading PDF file")
+                
+            loader = PyPDFLoader(file_path)
+            pages = loader.load()
             
-            logger.info(f"[{self.correlation_id}] Generated embeddings with dimension: {embedding_dimension}")
+            # Extract and concatenate text from all pages
+            all_text = ""
+            for page in pages:
+                all_text += page.page_content + "\n"
             
-            # Dimension handling - if mismatch, we handle it appropriately
-            if embedding_dimension != pinecone_dimension:
-                logger.warning(f"[{self.correlation_id}] Embedding dimension mismatch: got {embedding_dimension}, need {pinecone_dimension}")
+            if progress_callback:
+                await progress_callback("text_extraction", 0.6, "Extracted text from PDF")
                 
-                if embedding_dimension < pinecone_dimension:
-                    # For upscaling from 768 to 1536: duplicate each value and scale appropriately
-                    # This is one approach to handle dimension mismatches while preserving semantic information
-                    logger.info(f"[{self.correlation_id}] Using duplication strategy to upscale from {embedding_dimension} to {pinecone_dimension}")
-                    
-                    if embedding_dimension * 2 == pinecone_dimension:
-                        # Perfect doubling (768 -> 1536)
-                        def adjust_embedding(embedding):
-                            # Duplicate each value to double the dimension
-                            return [val for val in embedding for _ in range(2)]
-                    else:
-                        # Generic padding with zeros
-                        pad_size = pinecone_dimension - embedding_dimension
-                        def adjust_embedding(embedding):
-                            return embedding + [0.0] * pad_size
-                else:
-                    # Truncation strategy - take first pinecone_dimension values
-                    logger.info(f"[{self.correlation_id}] Will truncate embeddings from {embedding_dimension} to {pinecone_dimension}")
-                    
-                    def adjust_embedding(embedding):
-                        return embedding[:pinecone_dimension]
-            else:
-                # No adjustment needed
-                def adjust_embedding(embedding):
-                    return embedding
+            # Split text into chunks
+            text_splitter = RecursiveCharacterTextSplitter(chunk_size=800, chunk_overlap=300)
+            chunks = text_splitter.split_text(all_text)
             
-            # Process in batches to avoid memory issues
-            batch_size = 10
-            vectors_to_upsert = []
-            
-            for i in range(0, len(chunks), batch_size):
-                batch = chunks[i:i+batch_size]
+            logger.info(f"Split PDF file into {len(chunks)} chunks")
+            if progress_callback:
+                await progress_callback("chunking", 0.7, f"Split document into {len(chunks)} chunks")
+            
+            # Process embeddings for each chunk and upsert to Pinecone
+            vectors = []
+            for i, chunk in enumerate(chunks):
+                # Update embedding progress
+                if progress_callback and i % 5 == 0:  # Update every 5 chunks to avoid too many notifications
+                    embedding_progress = 0.7 + (0.3 * (i / len(chunks)))
+                    await progress_callback("embedding", embedding_progress, f"Processing chunk {i+1}/{len(chunks)}")
                 
-                # Extract text content
-                texts = [chunk.page_content for chunk in batch]
+                # Create vector embedding for each chunk
+                vector = embeddings_model.embed_query(chunk)
+                
+                # Prepare metadata for vector
+                vector_metadata = {
+                    "document_id": document_id,
+                    "chunk_index": i,
+                    "text": chunk
+                }
                 
-                # Create embeddings for batch
-                embeddings = embedding_model.embed_documents(texts)
+                # Add additional metadata if provided
+                if metadata:
+                    for key, value in metadata.items():
+                        if key not in vector_metadata:
+                            vector_metadata[key] = value
                 
-                # Prepare vectors for Pinecone
-                for j, (chunk, embedding) in enumerate(zip(batch, embeddings)):
-                    # Adjust embedding dimensions if needed
-                    adjusted_embedding = adjust_embedding(embedding)
-                    
-                    # Verify dimensions are correct
-                    if len(adjusted_embedding) != pinecone_dimension:
-                        raise ValueError(f"Dimension mismatch after adjustment: got {len(adjusted_embedding)}, expected {pinecone_dimension}")
-                    
-                    # Create metadata for this chunk
-                    chunk_metadata = {
-                        "document_id": document_id,
-                        "page": chunk.metadata.get("page", 0),
-                        "chunk_id": f"{document_id}-chunk-{i+j}",
-                        "text": chunk.page_content[:1000],  # Store first 1000 chars of text
-                        **metadata  # Include original metadata
-                    }
-                    
-                    # Create vector record
-                    vector = {
-                        "id": f"{document_id}-{i+j}",
-                        "values": adjusted_embedding,
-                        "metadata": chunk_metadata
-                    }
-                    
-                    vectors_to_upsert.append(vector)
+                # Add vector to list for upserting
+                vectors.append({
+                    "id": f"{document_id}_{i}",
+                    "values": vector,
+                    "metadata": vector_metadata
+                })
                 
-                logger.info(f"[{self.correlation_id}] Processed batch {i//batch_size + 1}/{(len(chunks)-1)//batch_size + 1}")
-            
-            # 4. Store embeddings in Pinecone
-            if progress_callback:
-                await progress_callback(None, document_id, "storing", 0.8, f"Storing {len(vectors_to_upsert)} vectors in Pinecone")
+                # Upsert in batches of 100 to avoid overloading
+                if len(vectors) >= 100:
+                    await self._upsert_vectors(vectors)
+                    vectors = []
             
-            logger.info(f"[{self.correlation_id}] Upserting {len(vectors_to_upsert)} vectors to Pinecone index {self.index_name}, namespace {actual_namespace}")
+            # Upsert any remaining vectors
+            if vectors:
+                await self._upsert_vectors(vectors)
             
-            # Use PineconeConnectionManager for better error handling
-            result = PineconeConnectionManager.upsert_vectors_with_validation(
-                self.pinecone_index,
-                vectors_to_upsert,
-                namespace=actual_namespace
-            )
-            
-            logger.info(f"[{self.correlation_id}] Successfully upserted {result.get('upserted_count', 0)} vectors to Pinecone")
+            logger.info(f"Embedded and saved {len(chunks)} chunks from PDF with document_id: {document_id}")
             
+            # Final progress update
             if progress_callback:
-                await progress_callback(None, document_id, "embedding_complete", 1.0, "Processing completed")
+                await progress_callback("completed", 1.0, "PDF processing complete")
             
-            # Return success with stats
             return {
                 "success": True,
                 "document_id": document_id,
                 "chunks_processed": len(chunks),
-                "total_text_length": total_text_length,
-                "vectors_created": len(vectors_to_upsert),
-                "vectors_upserted": result.get('upserted_count', 0),
-                "message": "PDF processed successfully"
+                "total_text_length": len(all_text)
             }
+            
         except Exception as e:
-            logger.error(f"[{self.correlation_id}] Error processing PDF: {str(e)}")
+            logger.error(f"Error processing PDF: {str(e)}")
+            if progress_callback:
+                await progress_callback("error", 0, f"Error processing PDF: {str(e)}")
             return {
                 "success": False,
-                "error": f"Error processing PDF: {str(e)}"
+                "error": str(e)
             }
     
-    async def list_namespaces(self):
-        """List all namespaces in the Pinecone index"""
+    async def _upsert_vectors(self, vectors):
+        """Upsert vectors to Pinecone"""
         try:
+            if not vectors:
+                return
+                
+            # Ensure we have a valid pinecone_index
             if not self.pinecone_index:
-                self.pinecone_index = PineconeConnectionManager.get_index(self.api_key, self.index_name)
+                self.pinecone_index = self._init_pinecone_connection()
+                if not self.pinecone_index:
+                    raise Exception("Cannot connect to Pinecone")
             
-            # Get index stats which includes namespaces
-            stats = self.pinecone_index.describe_index_stats()
-            namespaces = list(stats.get("namespaces", {}).keys())
+            result = self.pinecone_index.upsert(
+                vectors=vectors,
+                namespace=self.namespace
+            )
             
-            return {
-                "success": True,
-                "namespaces": namespaces
-            }
+            logger.info(f"Upserted {len(vectors)} vectors to Pinecone")
+            return result
         except Exception as e:
-            logger.error(f"[{self.correlation_id}] Error listing namespaces: {str(e)}")
-            return {
-                "success": False,
-                "error": f"Error listing namespaces: {str(e)}"
-            }
+            logger.error(f"Error upserting vectors: {str(e)}")
+            raise
     
     async def delete_namespace(self):
-        """Delete all vectors in a namespace"""
+        """
+        Delete all vectors in the current namespace (equivalent to deleting the namespace).
+        """
+        # Initialize connection if needed
+        self.pinecone_index = self._init_pinecone_connection()
+        if not self.pinecone_index:
+            return {"success": False, "error": "Could not connect to Pinecone"}
+
         try:
-            if not self.pinecone_index:
-                self.pinecone_index = PineconeConnectionManager.get_index(self.api_key, self.index_name)
-                
-            logger.info(f"[{self.correlation_id}] Deleting namespace '{self.namespace}' from index '{self.index_name}'")
-            
-            # Check if namespace exists
-            stats = self.pinecone_index.describe_index_stats()
-            namespaces = stats.get("namespaces", {})
-            
-            if self.namespace in namespaces:
-                vector_count = namespaces[self.namespace].get("vector_count", 0)
-                # Delete all vectors in namespace
-                self.pinecone_index.delete(delete_all=True, namespace=self.namespace)
-                return {
-                    "success": True,
-                    "namespace": self.namespace,
-                    "deleted_count": vector_count,
-                    "message": f"Successfully deleted namespace '{self.namespace}' with {vector_count} vectors"
-                }
-            else:
-                return {
-                    "success": True,
-                    "namespace": self.namespace,
-                    "deleted_count": 0,
-                    "message": f"Namespace '{self.namespace}' does not exist - nothing to delete"
-                }
+            # delete_all=True will delete all vectors in the namespace
+            result = self.pinecone_index.delete(
+                delete_all=True,
+                namespace=self.namespace
+            )
+            logger.info(f"Deleted namespace '{self.namespace}' (all vectors).")
+            return {"success": True, "detail": result}
         except Exception as e:
-            logger.error(f"[{self.correlation_id}] Error deleting namespace: {str(e)}")
-            return {
-                "success": False,
-                "namespace": self.namespace,
-                "error": f"Error deleting namespace: {str(e)}"
-            }
+            logger.error(f"Error deleting namespace '{self.namespace}': {e}")
+            return {"success": False, "error": str(e)}
     
-    async def delete_document(self, document_id, additional_metadata=None):
-        """Delete vectors associated with a specific document ID or name"""
-        logger.info(f"[{self.correlation_id}] Deleting vectors for document '{document_id}' from namespace '{self.namespace}'")
-
+    async def list_documents(self):
+        """Get list of all document_ids from Pinecone"""
         try:
+            # Initialize Pinecone connection if not already done
+            self.pinecone_index = self._init_pinecone_connection()
             if not self.pinecone_index:
-                self.pinecone_index = PineconeConnectionManager.get_index(self.api_key, self.index_name)
-                
-            # Use metadata filtering to find vectors with matching document_id
-            # The specific namespace to use might be vdb-X format if vector_db_id provided
-            actual_namespace = f"vdb-{self.vector_db_id}" if self.vector_db_id else self.namespace
-            
-            # Try to find vectors using multiple approaches
-            filters = []
+                return {"success": False, "error": "Could not connect to Pinecone"}
             
-            # First try with exact document_id which could be UUID (preferred)
-            filters.append({"document_id": document_id})
-            
-            # If this is a UUID, try with different formats (with/without hyphens)
-            if len(document_id) >= 32:
-                # This looks like it might be a UUID - try variations
-                if "-" in document_id:
-                    # If it has hyphens, try without
-                    filters.append({"document_id": document_id.replace("-", "")})
-                else:
-                    # If it doesn't have hyphens, try to format it as UUID
-                    try:
-                        formatted_uuid = str(uuid.UUID(document_id))
-                        filters.append({"document_id": formatted_uuid})
-                    except ValueError:
-                        pass
-            
-            # Also try with title field if it could be a document name
-            if not document_id.startswith("doc-") and not document_id.startswith("test-doc-") and len(document_id) < 36:
-                # This might be a document title/name
-                filters.append({"title": document_id})
-            
-            # If additional metadata was provided, use it to make extra filters
-            if additional_metadata:
-                if "document_name" in additional_metadata:
-                    # Try exact name match
-                    filters.append({"title": additional_metadata["document_name"]})
-                    
-                    # Also try filename if name has extension
-                    if "." in additional_metadata["document_name"]:
-                        filters.append({"filename": additional_metadata["document_name"]})
-            
-            # Search for vectors with any of these filters
-            found_vectors = False
-            deleted_count = 0
-            filter_used = ""
-            
-            logger.info(f"[{self.correlation_id}] Will try {len(filters)} different filters to find document")
-            
-            for i, filter_query in enumerate(filters):
-                logger.info(f"[{self.correlation_id}] Searching for vectors with filter #{i+1}: {filter_query}")
-                
-                # Search for vectors with this filter
-                try:
-                    results = self.pinecone_index.query(
-                        vector=[0] * 1536,  # Dummy vector, we only care about metadata filter
-                        top_k=1,
-                        include_metadata=True,
-                        filter=filter_query,
-                        namespace=actual_namespace
-                    )
-                    
-                    if results and results.get("matches") and len(results.get("matches", [])) > 0:
-                        logger.info(f"[{self.correlation_id}] Found vectors matching filter: {filter_query}")
-                        found_vectors = True
-                        filter_used = str(filter_query)
-                        
-                        # Delete vectors by filter
-                        delete_result = self.pinecone_index.delete(
-                            filter=filter_query,
-                            namespace=actual_namespace
-                        )
-                        
-                        # Get delete count from result
-                        deleted_count = delete_result.get("deleted_count", 0)
-                        logger.info(f"[{self.correlation_id}] Deleted {deleted_count} vectors with filter: {filter_query}")
-                        break
-                except Exception as filter_error:
-                    logger.warning(f"[{self.correlation_id}] Error searching with filter {filter_query}: {str(filter_error)}")
-                    continue
+            # Get index information
+            stats = self.pinecone_index.describe_index_stats()
             
-            # If no vectors found with any filter
-            if not found_vectors:
-                logger.warning(f"[{self.correlation_id}] No vectors found for document '{document_id}' in namespace '{actual_namespace}'")
-                return {
-                    "success": True,  # Still return success=True to maintain backward compatibility
-                    "document_id": document_id,
-                    "namespace": actual_namespace,
-                    "deleted_count": 0,
-                    "warning": f"No vectors found for document '{document_id}' in namespace '{actual_namespace}'",
-                    "message": f"Found 0 vectors for document '{document_id}' in namespace '{actual_namespace}'",
-                    "vectors_found": False,
-                    "vectors_deleted": 0
-                }
+            # Query to get list of all unique document_ids
+            # This method may not be efficient with large datasets, but is the simplest approach
+            # In practice, you should maintain a list of document_ids in a separate database
             
             return {
                 "success": True,
-                "document_id": document_id,
-                "namespace": actual_namespace,
-                "deleted_count": deleted_count,
-                "filter_used": filter_used,
-                "message": f"Successfully deleted {deleted_count} vectors for document '{document_id}' from namespace '{actual_namespace}'",
-                "vectors_found": True,
-                "vectors_deleted": deleted_count
+                "total_vectors": stats.get('total_vector_count', 0),
+                "namespace": self.namespace,
+                "index_name": self.index_name
             }
         except Exception as e:
-            logger.error(f"[{self.correlation_id}] Error deleting document vectors: {str(e)}")
+            logger.error(f"Error getting document list: {str(e)}")
             return {
                 "success": False,
-                "document_id": document_id,
-                "error": f"Error deleting document vectors: {str(e)}",
-                "vectors_found": False,
-                "vectors_deleted": 0
-            }
-    
-    async def list_documents(self):
-        """List all documents in a namespace"""
-        # The namespace to use might be vdb-X format if vector_db_id provided
-        actual_namespace = f"vdb-{self.vector_db_id}" if self.vector_db_id else self.namespace
-        
-        try:
-            if not self.pinecone_index:
-                self.pinecone_index = PineconeConnectionManager.get_index(self.api_key, self.index_name)
-                
-            logger.info(f"[{self.correlation_id}] Listing documents in namespace '{actual_namespace}'")
-            
-            # Get index stats for namespace
-            stats = self.pinecone_index.describe_index_stats()
-            namespace_stats = stats.get("namespaces", {}).get(actual_namespace, {})
-            vector_count = namespace_stats.get("vector_count", 0)
-            
-            if vector_count == 0:
-                # No vectors in namespace
-                return DocumentsListResponse(
-                    success=True,
-                    total_vectors=0,
-                    namespace=actual_namespace,
-                    index_name=self.index_name,
-                    documents=[]
-                ).dict()
-                
-            # Query for vectors with a dummy vector to get back metadata
-            # This is not efficient but is a simple approach to extract document info
-            results = self.pinecone_index.query(
-                vector=[0] * stats.dimension,  # Use index dimensions
-                top_k=min(vector_count, 1000),  # Get at most 1000 vectors
-                include_metadata=True,
-                namespace=actual_namespace
-            )
-            
-            # Process results to extract unique documents
-            seen_documents = set()
-            documents = []
-            
-            for match in results.get("matches", []):
-                metadata = match.get("metadata", {})
-                document_id = metadata.get("document_id")
-                
-                if document_id and document_id not in seen_documents:
-                    seen_documents.add(document_id)
-                    doc_info = {
-                        "id": document_id,
-                        "title": metadata.get("title"),
-                        "filename": metadata.get("filename"),
-                        "content_type": metadata.get("content_type"),
-                        "chunk_count": 0
-                    }
-                    documents.append(doc_info)
-                    
-                # Count chunks for this document
-                for doc in documents:
-                    if doc["id"] == document_id:
-                        doc["chunk_count"] += 1
-                        break
-            
-            return DocumentsListResponse(
-                success=True,
-                total_vectors=vector_count,
-                namespace=actual_namespace,
-                index_name=self.index_name,
-                documents=documents
-            ).dict()
-            
-        except Exception as e:
-            logger.error(f"[{self.correlation_id}] Error listing documents: {str(e)}")
-            return DocumentsListResponse(
-                success=False,
-                error=f"Error listing documents: {str(e)}"
-            ).dict() 
-
-
-
-
+                "error": str(e)
+            } 

app/utils/pinecone_fix.py

@@ -1,194 +0,0 @@
-"""
-Improved Pinecone connection handling with dimension validation.
-This module provides more robust connection and error handling for Pinecone operations.
-"""
-import logging
-import time
-from typing import Optional, Dict, Any, Tuple, List
-import pinecone
-from pinecone import Pinecone, ServerlessSpec, PodSpec
-
-logger = logging.getLogger(__name__)
-
-# Default retry settings
-DEFAULT_MAX_RETRIES = 3
-DEFAULT_RETRY_DELAY = 2
-
-class PineconeConnectionManager:
-    """
-    Manages Pinecone connections with enhanced error handling and dimension validation.
-    
-    This class centralizes Pinecone connection logic, providing:
-    - Connection pooling/reuse
-    - Automatic retries with exponential backoff
-    - Dimension validation before operations
-    - Detailed error logging for better debugging
-    """
-    
-    # Class-level cache of Pinecone clients
-    _clients = {}
-    
-    @classmethod
-    def get_client(cls, api_key: str) -> Pinecone:
-        """
-        Returns a Pinecone client for the given API key, creating one if needed.
-        
-        Args:
-            api_key: Pinecone API key
-            
-        Returns:
-            Initialized Pinecone client
-        """
-        if not api_key:
-            raise ValueError("Pinecone API key cannot be empty")
-            
-        # Return cached client if it exists
-        if api_key in cls._clients:
-            return cls._clients[api_key]
-            
-        # Log client creation (but hide full API key)
-        key_prefix = api_key[:4] + "..." if len(api_key) > 4 else "invalid"
-        logger.info(f"Creating new Pinecone client with API key (first 4 chars: {key_prefix}...)")
-        
-        try:
-            # Initialize Pinecone client
-            client = Pinecone(api_key=api_key)
-            cls._clients[api_key] = client
-            logger.info("Pinecone client created successfully")
-            return client
-        except Exception as e:
-            logger.error(f"Failed to create Pinecone client: {str(e)}")
-            raise RuntimeError(f"Pinecone client initialization failed: {str(e)}") from e
-    
-    @classmethod
-    def get_index(cls, 
-                  api_key: str, 
-                  index_name: str, 
-                  max_retries: int = DEFAULT_MAX_RETRIES) -> Any:
-        """
-        Get a Pinecone index with retry logic.
-        
-        Args:
-            api_key: Pinecone API key
-            index_name: Name of the index to connect to
-            max_retries: Maximum number of retry attempts
-            
-        Returns:
-            Pinecone index
-        """
-        client = cls.get_client(api_key)
-        
-        # Retry logic for connection issues
-        for attempt in range(max_retries):
-            try:
-                index = client.Index(index_name)
-                # Test the connection
-                _ = index.describe_index_stats()
-                logger.info(f"Connected to Pinecone index: {index_name}")
-                return index
-            except Exception as e:
-                if attempt < max_retries - 1:
-                    wait_time = DEFAULT_RETRY_DELAY * (2 ** attempt)  # Exponential backoff
-                    logger.warning(f"Pinecone connection attempt {attempt+1} failed: {e}. Retrying in {wait_time}s...")
-                    time.sleep(wait_time)
-                else:
-                    logger.error(f"Failed to connect to Pinecone index after {max_retries} attempts: {e}")
-                    raise RuntimeError(f"Pinecone index connection failed: {str(e)}") from e
-    
-    @classmethod
-    def validate_dimensions(cls, 
-                            index: Any, 
-                            vector_dimensions: int) -> Tuple[bool, Optional[str]]:
-        """
-        Validate that the vector dimensions match the Pinecone index configuration.
-        
-        Args:
-            index: Pinecone index
-            vector_dimensions: Dimensions of the vectors to be uploaded
-            
-        Returns:
-            Tuple of (is_valid, error_message)
-        """
-        try:
-            # Get index stats
-            stats = index.describe_index_stats()
-            index_dimensions = stats.dimension
-            
-            if index_dimensions != vector_dimensions:
-                error_msg = (f"Vector dimensions mismatch: Your vectors have {vector_dimensions} dimensions, "
-                            f"but Pinecone index expects {index_dimensions} dimensions")
-                logger.error(error_msg)
-                return False, error_msg
-                
-            return True, None
-        except Exception as e:
-            error_msg = f"Failed to validate dimensions: {str(e)}"
-            logger.error(error_msg)
-            return False, error_msg
-    
-    @classmethod
-    def upsert_vectors_with_validation(cls,
-                                    index: Any,
-                                    vectors: List[Dict[str, Any]],
-                                    namespace: str = "",
-                                    batch_size: int = 100) -> Dict[str, Any]:
-        """
-        Upsert vectors with dimension validation and batching.
-        
-        Args:
-            index: Pinecone index
-            vectors: List of vectors to upsert, each with 'id', 'values', and optional 'metadata'
-            namespace: Namespace to upsert to
-            batch_size: Size of batches for upserting
-            
-        Returns:
-            Result of upsert operation
-        """
-        if not vectors:
-            return {"upserted_count": 0, "success": True}
-            
-        # Validate dimensions with the first vector
-        if "values" in vectors[0] and len(vectors[0]["values"]) > 0:
-            vector_dim = len(vectors[0]["values"])
-            is_valid, error_msg = cls.validate_dimensions(index, vector_dim)
-            
-            if not is_valid:
-                logger.error(f"Dimension validation failed: {error_msg}")
-                raise ValueError(f"Vector dimensions do not match Pinecone index configuration: {error_msg}")
-        
-        # Batch upsert
-        total_upserted = 0
-        for i in range(0, len(vectors), batch_size):
-            batch = vectors[i:i+batch_size]
-            try:
-                result = index.upsert(vectors=batch, namespace=namespace)
-                batch_upserted = result.get("upserted_count", len(batch))
-                total_upserted += batch_upserted
-                logger.info(f"Upserted batch {i//batch_size + 1}: {batch_upserted} vectors")
-            except Exception as e:
-                logger.error(f"Failed to upsert batch {i//batch_size + 1}: {str(e)}")
-                raise RuntimeError(f"Vector upsert failed: {str(e)}") from e
-                
-        return {"upserted_count": total_upserted, "success": True}
-
-# Simplified function to check connection
-def check_connection(api_key: str, index_name: str) -> bool:
-    """
-    Test Pinecone connection and validate index exists.
-    
-    Args:
-        api_key: Pinecone API key
-        index_name: Name of index to test
-        
-    Returns:
-        True if connection successful, False otherwise
-    """
-    try:
-        index = PineconeConnectionManager.get_index(api_key, index_name)
-        stats = index.describe_index_stats()
-        total_vectors = stats.total_vector_count
-        logger.info(f"Pinecone connection is working. Total vectors: {total_vectors}")
-        return True
-    except Exception as e:
-        logger.error(f"Pinecone connection failed: {str(e)}")
-        return False 

docs/api_documentation.md

@@ -0,0 +1,581 @@
+# API Documentation
+
+## Frontend Setup
+
+```javascript
+// Basic Axios setup
+import axios from 'axios';
+
+const api = axios.create({
+  baseURL: 'https://api.your-domain.com',
+  timeout: 10000,
+  headers: {
+    'Content-Type': 'application/json',
+    'Accept': 'application/json'
+  }
+});
+
+// Error handling
+api.interceptors.response.use(
+  response => response.data,
+  error => {
+    const errorMessage = error.response?.data?.detail || 'An error occurred';
+    console.error('API Error:', errorMessage);
+    return Promise.reject(errorMessage);
+  }
+);
+```
+
+## Caching System
+
+- All GET endpoints support `use_cache=true` parameter (default)
+- Cache TTL: 300 seconds (5 minutes)
+- Cache is automatically invalidated on data changes
+
+## Authentication
+
+Currently no authentication is required. If implemented in the future, use JWT Bearer tokens:
+
+```javascript
+const api = axios.create({
+  // ...other config
+  headers: {
+    // ...other headers
+    'Authorization': `Bearer ${token}`
+  }
+});
+```
+
+## Error Codes
+
+| Code | Description |
+|------|-------------|
+| 400 | Bad Request |
+| 404 | Not Found |
+| 500 | Internal Server Error |
+| 503 | Service Unavailable |
+
+## PostgreSQL Endpoints
+
+### FAQ Endpoints
+
+#### Get FAQs List
+```
+GET /postgres/faq
+```
+
+Parameters:
+- `skip`: Number of items to skip (default: 0)
+- `limit`: Maximum items to return (default: 100)
+- `active_only`: Return only active items (default: false)
+- `use_cache`: Use cached data if available (default: true)
+
+Response:
+```json
+[
+  {
+    "question": "How do I book a hotel?",
+    "answer": "You can book a hotel through our app or website.",
+    "is_active": true,
+    "id": 1,
+    "created_at": "2023-01-01T00:00:00",
+    "updated_at": "2023-01-01T00:00:00"
+  }
+]
+```
+
+Example:
+```javascript
+async function getFAQs() {
+  try {
+    const data = await api.get('/postgres/faq', {
+      params: { active_only: true, limit: 20 }
+    });
+    return data;
+  } catch (error) {
+    console.error('Error fetching FAQs:', error);
+    throw error;
+  }
+}
+```
+
+#### Create FAQ
+```
+POST /postgres/faq
+```
+
+Request Body:
+```json
+{
+  "question": "How do I book a hotel?",
+  "answer": "You can book a hotel through our app or website.",
+  "is_active": true
+}
+```
+
+Response: Created FAQ object
+
+#### Get FAQ Detail
+```
+GET /postgres/faq/{faq_id}
+```
+
+Parameters:
+- `faq_id`: ID of FAQ (required)
+- `use_cache`: Use cached data if available (default: true)
+
+Response: FAQ object
+
+#### Update FAQ
+```
+PUT /postgres/faq/{faq_id}
+```
+
+Parameters:
+- `faq_id`: ID of FAQ to update (required)
+
+Request Body: Partial or complete FAQ object
+Response: Updated FAQ object
+
+#### Delete FAQ
+```
+DELETE /postgres/faq/{faq_id}
+```
+
+Parameters:
+- `faq_id`: ID of FAQ to delete (required)
+
+Response:
+```json
+{
+  "status": "success",
+  "message": "FAQ item 1 deleted"
+}
+```
+
+#### Batch Operations
+
+Create multiple FAQs:
+```
+POST /postgres/faqs/batch
+```
+
+Update status of multiple FAQs:
+```
+PUT /postgres/faqs/batch-update-status
+```
+
+Delete multiple FAQs:
+```
+DELETE /postgres/faqs/batch
+```
+
+### Emergency Contact Endpoints
+
+#### Get Emergency Contacts
+```
+GET /postgres/emergency
+```
+
+Parameters:
+- `skip`: Number of items to skip (default: 0)
+- `limit`: Maximum items to return (default: 100)
+- `active_only`: Return only active items (default: false)
+- `use_cache`: Use cached data if available (default: true)
+
+Response: Array of Emergency Contact objects
+
+#### Create Emergency Contact
+```
+POST /postgres/emergency
+```
+
+Request Body:
+```json
+{
+  "name": "Fire Department",
+  "phone_number": "114",
+  "description": "Fire rescue services",
+  "address": "Da Nang",
+  "location": "16.0544, 108.2022",
+  "priority": 1,
+  "is_active": true
+}
+```
+
+Response: Created Emergency Contact object
+
+#### Get Emergency Contact
+```
+GET /postgres/emergency/{emergency_id}
+```
+
+#### Update Emergency Contact
+```
+PUT /postgres/emergency/{emergency_id}
+```
+
+#### Delete Emergency Contact
+```
+DELETE /postgres/emergency/{emergency_id}
+```
+
+#### Batch Operations
+
+Create multiple Emergency Contacts:
+```
+POST /postgres/emergency/batch
+```
+
+Update status of multiple Emergency Contacts:
+```
+PUT /postgres/emergency/batch-update-status
+```
+
+Delete multiple Emergency Contacts:
+```
+DELETE /postgres/emergency/batch
+```
+
+### Event Endpoints
+
+#### Get Events
+```
+GET /postgres/events
+```
+
+Parameters:
+- `skip`: Number of items to skip (default: 0)
+- `limit`: Maximum items to return (default: 100)
+- `active_only`: Return only active items (default: false)
+- `featured_only`: Return only featured items (default: false)
+- `use_cache`: Use cached data if available (default: true)
+
+Response: Array of Event objects
+
+#### Create Event
+```
+POST /postgres/events
+```
+
+Request Body:
+```json
+{
+  "name": "Da Nang Fireworks Festival",
+  "description": "International Fireworks Festival Da Nang 2023",
+  "address": "Dragon Bridge, Da Nang",
+  "location": "16.0610, 108.2277",
+  "date_start": "2023-06-01T19:00:00",
+  "date_end": "2023-06-01T22:00:00",
+  "price": [
+    {"type": "VIP", "amount": 500000},
+    {"type": "Standard", "amount": 300000}
+  ],
+  "url": "https://danangfireworks.com",
+  "is_active": true,
+  "featured": true
+}
+```
+
+Response: Created Event object
+
+#### Get Event
+```
+GET /postgres/events/{event_id}
+```
+
+#### Update Event
+```
+PUT /postgres/events/{event_id}
+```
+
+#### Delete Event
+```
+DELETE /postgres/events/{event_id}
+```
+
+#### Batch Operations
+
+Create multiple Events:
+```
+POST /postgres/events/batch
+```
+
+Update status of multiple Events:
+```
+PUT /postgres/events/batch-update-status
+```
+
+Delete multiple Events:
+```
+DELETE /postgres/events/batch
+```
+
+### About Pixity Endpoints
+
+#### Get About Pixity
+```
+GET /postgres/about-pixity
+```
+
+Response:
+```json
+{
+  "content": "PiXity is your smart, AI-powered local companion...",
+  "id": 1,
+  "created_at": "2023-01-01T00:00:00",
+  "updated_at": "2023-01-01T00:00:00"
+}
+```
+
+#### Update About Pixity
+```
+PUT /postgres/about-pixity
+```
+
+Request Body:
+```json
+{
+  "content": "PiXity is your smart, AI-powered local companion..."
+}
+```
+
+Response: Updated About Pixity object
+
+### Da Nang Bucket List Endpoints
+
+#### Get Da Nang Bucket List
+```
+GET /postgres/danang-bucket-list
+```
+
+Response: Bucket List object with JSON content string
+
+#### Update Da Nang Bucket List
+```
+PUT /postgres/danang-bucket-list
+```
+
+### Solana Summit Endpoints
+
+#### Get Solana Summit
+```
+GET /postgres/solana-summit
+```
+
+Response: Solana Summit object with JSON content string
+
+#### Update Solana Summit
+```
+PUT /postgres/solana-summit
+```
+
+### Health Check
+```
+GET /postgres/health
+```
+
+Response:
+```json
+{
+  "status": "healthy",
+  "message": "PostgreSQL connection is working",
+  "timestamp": "2023-01-01T00:00:00"
+}
+```
+
+## MongoDB Endpoints
+
+### Session Endpoints
+
+#### Create Session
+```
+POST /session
+```
+
+Request Body:
+```json
+{
+  "user_id": "user123",
+  "query": "How do I book a room?",
+  "timestamp": "2023-01-01T00:00:00",
+  "metadata": {
+    "client_info": "web",
+    "location": "Da Nang"
+  }
+}
+```
+
+Response: Created Session object with session_id
+
+#### Update Session with Response
+```
+PUT /session/{session_id}/response
+```
+
+Request Body:
+```json
+{
+  "response": "You can book a room through our app or website.",
+  "response_timestamp": "2023-01-01T00:00:05",
+  "metadata": {
+    "response_time_ms": 234,
+    "model_version": "gpt-4"
+  }
+}
+```
+
+Response: Updated Session object
+
+#### Get Session
+```
+GET /session/{session_id}
+```
+
+Response: Session object
+
+#### Get User History
+```
+GET /history
+```
+
+Parameters:
+- `user_id`: User ID (required)
+- `limit`: Maximum sessions to return (default: 10)
+- `skip`: Number of sessions to skip (default: 0)
+
+Response:
+```json
+{
+  "user_id": "user123",
+  "sessions": [
+    {
+      "session_id": "60f7a8b9c1d2e3f4a5b6c7d8",
+      "query": "How do I book a room?",
+      "timestamp": "2023-01-01T00:00:00",
+      "response": "You can book a room through our app or website.",
+      "response_timestamp": "2023-01-01T00:00:05"
+    }
+  ],
+  "total_count": 1
+}
+```
+
+#### Health Check
+```
+GET /health
+```
+
+## RAG Endpoints
+
+### Create Embedding
+```
+POST /embedding
+```
+
+Request Body:
+```json
+{
+  "text": "Text to embed"
+}
+```
+
+Response:
+```json
+{
+  "embedding": [0.1, 0.2, 0.3, ...],
+  "dimensions": 1536
+}
+```
+
+### Process Chat Request
+```
+POST /chat
+```
+
+Request Body:
+```json
+{
+  "query": "Can you tell me about Pixity?",
+  "chat_history": [
+    {"role": "user", "content": "Hello"},
+    {"role": "assistant", "content": "Hello! How can I help you?"}
+  ]
+}
+```
+
+Response:
+```json
+{
+  "answer": "Pixity is a platform...",
+  "sources": [
+    {
+      "document_id": "doc123",
+      "chunk_id": "chunk456",
+      "chunk_text": "Pixity was founded in...",
+      "relevance_score": 0.92
+    }
+  ]
+}
+```
+
+### Direct RAG Query
+```
+POST /rag
+```
+
+Request Body:
+```json
+{
+  "query": "Can you tell me about Pixity?",
+  "namespace": "about_pixity",
+  "top_k": 3
+}
+```
+
+Response: Query results with relevance scores
+
+### Health Check
+```
+GET /health
+```
+
+## PDF Processing Endpoints
+
+### Upload and Process PDF
+```
+POST /pdf/upload
+```
+
+Form Data:
+- `file`: PDF file (required)
+- `namespace`: Vector database namespace (default: "Default")
+- `index_name`: Vector database index name (default: "testbot768")
+- `title`: Document title (optional)
+- `description`: Document description (optional)
+- `user_id`: User ID for WebSocket updates (optional)
+
+Response: Processing results with document_id
+
+### Delete Documents in Namespace
+```
+DELETE /pdf/namespace
+```
+
+Parameters:
+- `namespace`: Vector database namespace (default: "Default")
+- `index_name`: Vector database index name (default: "testbot768")
+- `user_id`: User ID for WebSocket updates (optional)
+
+Response: Deletion results
+
+### Get Documents List
+```
+GET /pdf/documents
+```
+
+Parameters:
+- `namespace`: Vector database namespace (default: "Default")
+- `index_name`: Vector database index name (default: "testbot768")
+
+Response: List of documents in the namespace 

pytest.ini

@@ -0,0 +1,12 @@
+[pytest]
+# Bỏ qua cảnh báo về anyio module và các cảnh báo vận hành nội bộ
+filterwarnings =
+    ignore::pytest.PytestAssertRewriteWarning:.*anyio
+    ignore:.*general_plain_validator_function.* is deprecated.*:DeprecationWarning
+    ignore:.*with_info_plain_validator_function.*:DeprecationWarning
+
+# Cấu hình cơ bản khác
+testpaths = tests
+python_files = test_*.py
+python_classes = Test*
+python_functions = test_* 

requirements.txt

@@ -31,14 +31,12 @@ httpx==0.25.1
 requests==2.31.0
 beautifulsoup4==4.12.2
 redis==5.0.1
-aiofiles==23.2.1
 
 # Testing
 prometheus-client==0.17.1
 pytest==7.4.0
 pytest-cov==4.1.0
 watchfiles==0.21.0
-fpdf==1.7.2
 
 # Core dependencies
 starlette==0.27.0