🚀 Final fix v20250913_220639: Comprehensive solution for dependency and configuration issues

README.md

@@ -6,6 +6,7 @@ colorTo: purple
 sdk: gradio
 sdk_version: 4.44.0
 app_file: app.py
+entrypoint: start.sh
 startup_duration_timeout: 600
 pinned: false
 license: mit
@@ -20,26 +21,37 @@ A robust, production-ready AI assistant powered by Microsoft's Phi-3.5-MoE model
 
 ## 🚀 Key Features
 
-- **🧠 Expert Routing**: Automatically routes queries to specialized experts
+- **🧠 Expert Routing**: Automatically routes queries to specialized experts (Code, Math, Reasoning, Multilingual, General)
 - **🔧 Environment Adaptive**: Works seamlessly on both CPU and GPU environments
-- **🛡️ Robust Dependency Management**: Conditional installation of dependencies
-- **📦 Simple Architecture**: Clean, maintainable codebase
-- **⚡ Performance Optimized**: Environment-specific optimizations
+- **🛡️ Robust Dependency Management**: Conditional installation of dependencies based on environment
+- **📦 Fault Tolerance**: Handles missing dependencies with fallback mechanisms
+- **⚡ Performance Optimized**: Environment-specific optimizations for best performance
 
 ## 🔧 Recent Fixes
 
-- ✅ **Missing Dependencies**: Added `einops` to requirements
+- ✅ **Missing Dependencies**: Added `einops` to requirements, conditional `flash_attn` installation
 - ✅ **Deprecated Parameters**: Fixed all `torch_dtype` → `dtype` usage
-- ✅ **CPU Compatibility**: Environment-specific model configuration
+- ✅ **CPU Compatibility**: Automatic CPU-safe model revision selection
 - ✅ **Error Handling**: Comprehensive fallback mechanisms
 - ✅ **Security**: Updated to Gradio 4.44.0+ for security fixes
 
+## 🏗️ Architecture
+
+```
+app.py              # Main application entry point
+preinstall.py       # Pre-installation script for dependencies
+model_patch.py      # Patch for handling missing dependencies
+start.sh            # Startup script
+requirements.txt    # Core dependencies
+```
+
 ## 🎯 How It Works
 
 1. **Environment Detection**: Automatically detects CPU vs GPU environment
-2. **Model Configuration**: Uses optimal settings for each environment
-3. **Response Generation**: Generates contextual responses to user queries
-4. **Graceful Fallbacks**: Works even when model loading fails
+2. **Dependency Management**: Installs required dependencies based on environment
+3. **Model Configuration**: Uses optimal settings for each environment
+4. **Expert Routing**: Classifies queries and routes to appropriate expert
+5. **Graceful Fallbacks**: Works even when dependencies are missing
 
 ## 📊 Performance
 
@@ -48,6 +60,14 @@ A robust, production-ready AI assistant powered by Microsoft's Phi-3.5-MoE model
 | **CPU**     | 3-5 min | 8-12 GB | 2-5 |
 | **GPU**     | 2-3 min | 16-20 GB | 15-30 |
 
+## 🔍 Troubleshooting
+
+If you encounter issues:
+1. Check the logs for dependency installation
+2. Verify the pre-installation script executed successfully
+3. Ensure all required packages are installed
+4. Try the fallback mode if model loading fails
+
 ---
 
 **Built with ❤️ for reliable, production-ready AI applications**

app.py

@@ -1,7 +1,21 @@
+#!/usr/bin/env python3
+"""
+Phi-3.5-MoE Expert Assistant
+Robust application with CPU/GPU environment detection and dependency handling
+"""
+
+import os
+import sys
 import gradio as gr
 import torch
 from transformers import AutoTokenizer, AutoModelForCausalLM
-import os
+
+# Apply the model patch if available
+try:
+    import model_patch
+    print("✅ Applied model patch for handling missing dependencies")
+except ImportError:
+    print("ℹ️ Model patch not found, continuing without it")
 
 # Environment detection
 ON_GPU = torch.cuda.is_available()
@@ -10,20 +24,36 @@ REVISION = os.getenv("HF_REVISION")
 
 # Configuration based on environment
 if ON_GPU:
-    attn_impl = "sdpa"
-    dtype = torch.bfloat16
-    device_map = "auto"
+    attn_impl = "sdpa"  # Fast attention for GPU
+    dtype = torch.bfloat16  # Mixed precision for GPU
+    device_map = "auto"  # Auto device mapping for GPU
+    low_cpu_mem = False  # Don't need low memory usage on GPU
 else:
-    attn_impl = "eager"
-    dtype = torch.float32
-    device_map = "cpu"
+    attn_impl = "eager"  # Standard attention for CPU
+    dtype = torch.float32  # Full precision for CPU
+    device_map = "cpu"  # Force CPU device
+    low_cpu_mem = True  # Enable low memory usage on CPU
 
 print(f"🚀 Loading model: {MODEL_ID}")
 print(f"🔧 Environment: {'GPU' if ON_GPU else 'CPU'}")
-print(f"📊 Attention: {attn_impl}, Dtype: {dtype}")
+print(f"📊 Configuration: attn={attn_impl}, dtype={dtype}, device={device_map}, revision={REVISION}")
+
+# Expert categories for query classification
+EXPERT_CATEGORIES = {
+    "Code": ["programming", "software", "development", "coding", "algorithm", "python", "javascript", "java", "function", "code", "debug", "api", "framework", "library", "class", "method", "variable"],
+    "Math": ["mathematics", "calculation", "equation", "formula", "statistics", "derivative", "integral", "algebra", "calculus", "math", "solve", "calculate", "probability", "geometry", "trigonometry"],
+    "Reasoning": ["logic", "analysis", "reasoning", "problem-solving", "critical", "explain", "why", "how", "because", "analyze", "evaluate", "compare", "contrast", "deduce", "infer"],
+    "Multilingual": ["translation", "language", "multilingual", "localization", "translate", "spanish", "french", "german", "chinese", "japanese", "korean", "arabic", "russian", "portuguese"],
+    "General": ["general", "conversation", "assistance", "help", "hello", "hi", "what", "who", "when", "where", "tell", "describe", "explain"]
+}
+
+# Load model with robust error handling
+model = None
+tokenizer = None
 
 try:
     # Load tokenizer
+    print("📝 Loading tokenizer...")
     tokenizer = AutoTokenizer.from_pretrained(
         MODEL_ID, 
         trust_remote_code=True, 
@@ -31,6 +61,7 @@ try:
     )
     
     # Load model with environment-specific settings
+    print("🧠 Loading model...")
     model = AutoModelForCausalLM.from_pretrained(
         MODEL_ID,
         trust_remote_code=True,
@@ -38,24 +69,54 @@ try:
         attn_implementation=attn_impl,
         dtype=dtype,  # Fixed: Use dtype instead of torch_dtype
         device_map=device_map,
-        low_cpu_mem_usage=not ON_GPU
+        low_cpu_mem_usage=low_cpu_mem
     ).eval()
     
     print("✅ Model loaded successfully!")
     
+    # Verify model works with a simple generation
+    print("🔍 Running quick model test...")
+    test_input = tokenizer("Hello, I am", return_tensors="pt").to(device_map if device_map != "auto" else model.device)
+    with torch.no_grad():
+        test_output = model.generate(**test_input, max_new_tokens=5)
+    print("✅ Model test successful!")
+    
 except Exception as e:
-    print(f"❌ Model loading failed: {e}")
-    model = None
-    tokenizer = None
+    print(f"⚠️ Model loading failed: {e}")
+    print("⚠️ Continuing with limited functionality")
 
-def generate_response(prompt, max_tokens=512, temperature=0.7):
+def classify_expert(query):
+    """Classify query to determine which expert should handle it."""
+    query_lower = query.lower()
+    scores = {}
+    
+    for expert, keywords in EXPERT_CATEGORIES.items():
+        score = sum(1 for keyword in keywords if keyword in query_lower)
+        scores[expert] = score
+    
+    # Get expert with highest score, default to General if tied or no matches
+    max_score = max(scores.values()) if scores else 0
+    if max_score > 0:
+        experts = [expert for expert, score in scores.items() if score == max_score]
+        return experts[0]
+    return "General"
+
+def generate_response(prompt, max_tokens=512, temperature=0.7, expert=None):
     """Generate response from the model."""
     if model is None or tokenizer is None:
-        return "❌ Model not loaded. Please check the logs for errors."
+        return "⚠️ Model not loaded. Please check the logs for errors."
     
     try:
+        # Determine expert if not provided
+        if expert is None:
+            expert = classify_expert(prompt)
+        
+        # Create expert-specific prompt
+        system_prompt = f"You are an AI assistant specialized in {expert}. "
+        full_prompt = f"{system_prompt}\n\nUser: {prompt}\n\nAssistant:"
+        
         # Tokenize input
-        inputs = tokenizer(prompt, return_tensors="pt")
+        inputs = tokenizer(full_prompt, return_tensors="pt")
         if ON_GPU:
             inputs = {k: v.to(model.device) for k, v in inputs.items()}
         
@@ -72,12 +133,12 @@ def generate_response(prompt, max_tokens=512, temperature=0.7):
         # Decode response
         response = tokenizer.decode(outputs[0], skip_special_tokens=True)
         # Remove the input prompt from the response
-        response = response[len(prompt):].strip()
+        response = response[len(full_prompt):].strip()
         
         return response
         
     except Exception as e:
-        return f"❌ Generation failed: {str(e)}"
+        return f"⚠️ Generation failed: {str(e)}"
 
 def create_interface():
     """Create the Gradio interface."""
@@ -86,6 +147,9 @@ def create_interface():
         gr.Markdown("# 🤖 Phi-3.5-MoE Expert Assistant")
         gr.Markdown(f"**Environment:** {'GPU' if ON_GPU else 'CPU'} | **Model:** {MODEL_ID}")
         
+        if model is None:
+            gr.Markdown("⚠️ **Model failed to load. Limited functionality available.**")
+        
         with gr.Row():
             with gr.Column(scale=3):
                 prompt = gr.Textbox(
@@ -103,6 +167,12 @@ def create_interface():
                         minimum=0.1, maximum=2.0, value=0.7, step=0.1,
                         label="Temperature"
                     )
+                    expert = gr.Dropdown(
+                        choices=list(EXPERT_CATEGORIES.keys()),
+                        value=None,
+                        label="Expert (Optional)",
+                        allow_custom_value=False
+                    )
                 
                 generate_btn = gr.Button("Generate Response", variant="primary")
             
@@ -116,25 +186,26 @@ def create_interface():
         # Example prompts
         gr.Examples(
             examples=[
-                "Explain quantum computing in simple terms",
-                "Write a Python function to calculate fibonacci numbers",
-                "What are the benefits of renewable energy?",
-                "How does machine learning work?",
-                "Translate 'Hello, how are you?' to Spanish"
+                ["Explain quantum computing in simple terms", None],
+                ["Write a Python function to calculate fibonacci numbers", "Code"],
+                ["What are the benefits of renewable energy?", "General"],
+                ["How does machine learning work?", "Reasoning"],
+                ["Translate 'Hello, how are you?' to Spanish", "Multilingual"],
+                ["Solve the equation 3x^2 + 5x - 2 = 0", "Math"]
             ],
-            inputs=prompt
+            inputs=[prompt, expert]
         )
         
         # Event handlers
         generate_btn.click(
             fn=generate_response,
-            inputs=[prompt, max_tokens, temperature],
+            inputs=[prompt, max_tokens, temperature, expert],
             outputs=response
         )
         
         prompt.submit(
             fn=generate_response,
-            inputs=[prompt, max_tokens, temperature],
+            inputs=[prompt, max_tokens, temperature, expert],
             outputs=response
         )
     
@@ -146,4 +217,4 @@ if __name__ == "__main__":
         server_name="0.0.0.0",
         server_port=7860,
         share=False
-    )
+    )

deploy_timestamp_20250913_220639.txt

@@ -0,0 +1 @@
+Final fix deployed at 2025-09-13 22:06:39.021771

preinstall.py

@@ -0,0 +1,134 @@
+#!/usr/bin/env python3
+"""
+Pre-installation script for Phi-3.5-MoE Space
+Installs required dependencies and selects CPU-safe model revision if needed
+"""
+
+import os
+import sys
+import subprocess
+import torch
+import re
+from pathlib import Path
+from huggingface_hub import HfApi
+
+def install_dependencies():
+    """Install required dependencies based on environment."""
+    print("🔧 Installing required dependencies...")
+    
+    # Always install einops
+    subprocess.check_call([sys.executable, "-m", "pip", "install", "einops>=0.7.0"])
+    print("✅ Installed einops")
+    
+    # Install flash-attn only if CUDA is available
+    if torch.cuda.is_available():
+        try:
+            subprocess.check_call([sys.executable, "-m", "pip", "install", "flash-attn>=2.6.0", "--no-build-isolation"])
+            print("✅ Installed flash-attn for GPU runtime")
+        except subprocess.CalledProcessError:
+            print("⚠️ Failed to install flash-attn, continuing without it")
+    else:
+        print("ℹ️ CPU runtime detected: skipping flash-attn installation")
+
+def select_cpu_safe_revision():
+    """Select a CPU-safe model revision by checking commit history."""
+    if torch.cuda.is_available() or os.getenv("HF_REVISION"):
+        return
+    
+    MODEL_ID = os.getenv("HF_MODEL_ID", "microsoft/Phi-3.5-MoE-instruct")
+    TARGET_FILE = "modeling_phimoe.py"
+    ENV_FILE = ".env"
+    
+    print(f"🔍 Selecting CPU-safe revision for {MODEL_ID}...")
+    
+    try:
+        api = HfApi()
+        for commit in api.list_repo_commits(MODEL_ID, repo_type="model"):
+            sha = commit.commit_id
+            try:
+                file_path = api.hf_hub_download(MODEL_ID, TARGET_FILE, revision=sha, repo_type="model")
+                with open(file_path, "r", encoding="utf-8") as f:
+                    code = f.read()
+                
+                # Check if this version doesn't have flash_attn as a top-level import
+                if not re.search(r'^\s*import\s+flash_attn|^\s*from\s+flash_attn', code, flags=re.M):
+                    # Write to .env file
+                    with open(ENV_FILE, "a", encoding="utf-8") as env_file:
+                        env_file.write(f"HF_REVISION={sha}\n")
+                    
+                    # Also set it in the current environment
+                    os.environ["HF_REVISION"] = sha
+                    
+                    print(f"✅ Selected CPU-safe revision: {sha}")
+                    return
+            except Exception:
+                continue
+        
+        print("⚠️ No CPU-safe revision found")
+    except Exception as e:
+        print(f"⚠️ Error selecting CPU-safe revision: {e}")
+
+def create_model_patch():
+    """Create a patch file to fix the model loading code."""
+    PATCH_FILE = "model_patch.py"
+    
+    patch_content = """
+# Monkey patch for transformers.dynamic_module_utils
+import sys
+import importlib
+from importlib.abc import Loader
+from importlib.machinery import ModuleSpec
+from transformers.dynamic_module_utils import check_imports
+
+# Create mock modules for missing dependencies
+class MockModule:
+    def __init__(self, name):
+        self.__name__ = name
+        self.__spec__ = ModuleSpec(name, None)
+    
+    def __getattr__(self, key):
+        return MockModule(f"{self.__name__}.{key}")
+
+# Override check_imports to handle missing dependencies
+original_check_imports = check_imports
+def patched_check_imports(resolved_module_file):
+    try:
+        return original_check_imports(resolved_module_file)
+    except ImportError as e:
+        # Extract missing modules
+        import re
+        missing = re.findall(r'packages that were not found in your environment: ([^.]+)', str(e))
+        if missing:
+            missing_modules = [m.strip() for m in missing[0].split(',')]
+            print(f"⚠️ Missing dependencies: {', '.join(missing_modules)}")
+            print("🔧 Creating mock modules to continue loading...")
+            
+            # Create mock modules
+            for module_name in missing_modules:
+                if module_name not in sys.modules:
+                    mock_module = MockModule(module_name)
+                    sys.modules[module_name] = mock_module
+                    print(f"✅ Created mock for {module_name}")
+            
+            # Try again
+            return original_check_imports(resolved_module_file)
+        else:
+            raise
+
+# Apply the patch
+from transformers import dynamic_module_utils
+dynamic_module_utils.check_imports = patched_check_imports
+print("✅ Applied transformers patch for handling missing dependencies")
+"""
+    
+    with open(PATCH_FILE, "w", encoding="utf-8") as f:
+        f.write(patch_content)
+    
+    print(f"✅ Created model patch file: {PATCH_FILE}")
+
+if __name__ == "__main__":
+    print("🚀 Running pre-installation script...")
+    install_dependencies()
+    select_cpu_safe_revision()
+    create_model_patch()
+    print("✅ Pre-installation complete!")

requirements.txt

@@ -2,9 +2,9 @@ gradio>=4.44.0
 torch>=2.0.0
 transformers>=4.46.0
 accelerate>=0.31.0
-einops>=0.8.0
+einops>=0.7.0
 sentencepiece>=0.1.99
 protobuf>=3.20.0
-huggingface-hub>=0.19.0
+huggingface-hub>=0.23.0
 tokenizers>=0.15.0
 safetensors>=0.4.0

start.sh

@@ -4,17 +4,10 @@ set -euo pipefail
 echo "🚀 Starting Phi-3.5-MoE Expert Assistant..."
 echo "📅 $(date)"
 
-# Ensure we're in the right directory
-cd /home/user
-
-# Make prestart script executable
-chmod +x prestart.sh
-
-# Run prestart setup
-echo "🔧 Running prestart setup..."
-./prestart.sh
+# Run pre-installation script
+echo "🔧 Running pre-installation script..."
+python preinstall.py
 
 # Start the application
 echo "🚀 Starting application..."
-cd /home/user
-python app/app.py
+python app.py