Upload README.md with huggingface_hub

README.md

@@ -5,7 +5,6 @@ tags:
 - conversation-analysis
 - pytorch
 - attention
-- lstm
 license: mit
 datasets:
 - leonvanbokhorst/topic-drift-v2
@@ -35,18 +34,16 @@ model-index:
 
 # Topic Drift Detector Model
 
-## Version: v20241225_184257
+## Version: v20241226_105737
 
-This model detects topic drift in conversations using an enhanced hierarchical attention-based architecture. Trained on the [leonvanbokhorst/topic-drift-v2](https://huggingface.co/datasets/leonvanbokhorst/topic-drift-v2) dataset.
+This model detects topic drift in conversations using a streamlined attention-based architecture. Trained on the [leonvanbokhorst/topic-drift-v2](https://huggingface.co/datasets/leonvanbokhorst/topic-drift-v2) dataset.
 
 ## Model Architecture
-- Multi-head attention mechanism (4 heads, head dimension 128)
-- Hierarchical pattern detection with multi-scale analysis
-- Explicit transition point detection with linguistic markers
-- Pattern-aware self-attention mechanism
-- Dynamic window augmentation
-- Contrastive learning with pattern-aware sampling
-- Adversarial training with pattern-aware perturbations
+- Efficient single-layer attention mechanism
+- Direct pattern recognition
+- Streamlined processing pipeline
+- Optimized scaling factor (4.0)
+- PreNorm layers with residual connections
 
 ### Key Components:
 1. **Embedding Processor**:
@@ -55,22 +52,16 @@ This model detects topic drift in conversations using an enhanced hierarchical a
    - Dropout rate: 0.35
    - PreNorm layers with residual connections
 
-2. **Attention Blocks**:
-   - 3 layers of attention
-   - 4 attention heads
-   - Feed-forward dimension: 2048
+2. **Attention Block**:
+   - Single attention layer
+   - Feed-forward dimension: 512
    - Learned position encodings
+   - Residual connections
 
-3. **Pattern Detection**:
-   - Hierarchical LSTM layers
-   - Bidirectional processing
-   - Multi-scale pattern analysis
-   - Pattern classification with 7 types
-
-4. **Transition Detection**:
-   - Linguistic marker attention
-   - Explicit transition scoring
-   - Marker-based context integration
+3. **Pattern Recognition**:
+   - Direct feature extraction
+   - Efficient tensor operations
+   - Optimized memory usage
 
 ## Performance Metrics
 ```txt
@@ -88,28 +79,24 @@ R²: 0.8666
 - Dataset: 6400 conversations (5120 train, 640 val, 640 test)
 - Window size: 8 turns
 - Batch size: 32
-- Learning rate: 0.0001 with cosine decay
-- Warmup steps: 100
+- Learning rate: 0.0001
 - Early stopping patience: 15
-- Max gradient norm: 1.0
-- Mixed precision training (AMP)
+- Distribution regularization weight: 0.1
+- Target standard deviation: 0.2
 - Base embeddings: BAAI/bge-m3
 
-### Training Enhancements:
-1. **Dynamic Window Augmentation**:
-   - Adaptive window sizes
-   - Interpolation-based resizing
-   - Maintains temporal consistency
-
-2. **Contrastive Learning**:
-   - Pattern-aware positive/negative sampling
-   - Temperature-scaled similarities
-   - Weighted combination of embeddings
+## Key Improvements
+1. **Simplified Architecture**:
+   - Reduced complexity
+   - Focused pattern detection
+   - Efficient processing
+   - Optimized memory usage
 
-3. **Adversarial Training**:
-   - Pattern-aware perturbations
-   - Self-distillation loss
-   - Epsilon ball projection
+2. **Performance Benefits**:
+   - Improved RMSE (0.0144)
+   - Strong R² score (0.8666)
+   - Consistent predictions
+   - Wide score range
 
 ## Usage Example
 ```python
@@ -121,7 +108,7 @@ base_model = AutoModel.from_pretrained('BAAI/bge-m3')
 tokenizer = AutoTokenizer.from_pretrained('BAAI/bge-m3')
 
 # Load topic drift detector
-model = torch.load('models/v20241225_184257/topic_drift_model.pt')
+model = torch.load('models/v20241226_105737/topic_drift_model.pt')
 model.eval()
 
 # Prepare conversation window (8 turns)
@@ -151,22 +138,11 @@ print(f"Topic drift score: {drift_scores.item():.4f}")
 # Higher scores indicate more topic drift
 ```
 
-## Pattern Types
-The model detects 7 distinct pattern types:
-1. "maintain" - No significant drift
-2. "gentle_wave" - Subtle topic evolution
-3. "single_peak" - One clear transition
-4. "multi_peak" - Multiple transitions
-5. "ascending" - Gradually increasing drift
-6. "descending" - Gradually decreasing drift
-7. "abrupt" - Sudden topic change
-
 ## Limitations
 - Works best with English conversations
 - Requires exactly 8 turns of conversation
 - Each turn should be between 1-512 tokens
 - Relies on BAAI/bge-m3 embeddings
-- May be sensitive to conversation style variations
 
 ## Training Curves
-![Training Curves](plots/v20241225_184257/training_curves.png)
+![Training Curves](plots/v20241226_105737/training_curves.png)