video-comprehension-score 1.0.0
Video Comprehension Score (VCS) - A comprehensive metric for evaluating narrative similarity between reference and generated text
License: MIT
Requires Python: >=3.8
Keywords: text-similarity, narrative-analysis, nlp, video-comprehension, text-evaluation, semantic-similarity, alignment-metrics

<div align="center">
  <a href="https://github.com/hdubey-debug/vcs">
    <img src=".github/assets/vcs.gif" alt="VCS Process Flow" width="100%"/>
  </a>
  <p align="center">
    <em>A Comprehensive Python Library for Narrative Similarity Evaluation between two very long descriptions </em>
    <br />
  </p>
</div>

<div align="center">

[![PyPI version](https://img.shields.io/pypi/v/video-comprehension-score?color=teal&style=for-the-badge)](https://badge.fury.io/py/video-comprehension-score)
[![Python 3.10+](https://img.shields.io/badge/python-3.10+-teal?style=for-the-badge&logo=python&logoColor=white)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/badge/License-MIT-teal?style=for-the-badge)](https://opensource.org/licenses/MIT)
[![Documentation](https://img.shields.io/badge/docs-github.io-teal?style=for-the-badge&logo=gitbook&logoColor=white)](https://multimodal-intelligence-lab.github.io/Video-Comprehension-Score/)

</div>

<p align="center">
  <a href="https://arxiv.org/abs/placeholder-link">📄 Research Paper</a>
  ·
  <a href="https://colab.research.google.com/drive/1l6GXWNBGFM1UwGohnIu1b071bn8ekJIf?usp=sharing">📓 Interactive Notebook</a>
</p>

---

## 🤔 What is VCS?

Recent advances in Large Video Language Models (LVLMs) have significantly enhanced automated video understanding, enabling detailed, long-form narratives of complex video content. However, accurately evaluating whether these models genuinely comprehend the video's narrative—its events, entities, interactions, and chronological coherence—remains challenging.

**Why Existing Metrics Fall Short:**

- **N-gram Metrics (e.g., BLEU, ROUGE, CIDEr)**: Primarily measure lexical overlap, penalizing valid linguistic variations and inadequately evaluating narrative chronology.

- **Embedding-based Metrics (e.g., BERTScore, SBERT)**: Improve semantic sensitivity but struggle with extended context, detailed content alignment, and narrative sequencing.

- **LLM-based Evaluations**: Often inconsistent, lacking clear criteria for narrative structure and chronology assessments.

Moreover, traditional benchmarks largely rely on question-answering tasks, which only test isolated events or entities rather than holistic video comprehension. A model answering specific questions correctly does not necessarily demonstrate understanding of the overall narrative or the intricate interplay of events.

**Introducing VCS (Video Comprehension Score):**

VCS is a Python library specifically designed to overcome these challenges by evaluating narrative comprehension through direct comparison of extensive, detailed video descriptions generated by LVLMs against human-written references. Unlike traditional metrics, VCS assesses whether models capture the overall narrative structure, event sequencing, and thematic coherence, not just lexical or isolated semantic matches.

**Core Components of VCS:**

- 🌍 **Global Alignment Score (GAS)**: Captures overall thematic alignment, tolerating stylistic variations without penalizing valid linguistic differences.

- 🎯 **Local Alignment Score (LAS)**: Checks detailed semantic correspondence at a chunk-level, allowing minor descriptive variations while penalizing significant inaccuracies or omissions.

- 📖 **Narrative Alignment Score (NAS)**: Evaluates chronological consistency, balancing the need for both strict event sequencing and permissible narrative flexibility.

Initially developed for evaluating video comprehension by comparing generated and human-written video narratives, VCS is versatile enough for broader applications, including document-level narrative comparisons, analysis of extensive narrative content, and various other narrative similarity tasks.

## 🚀 **Key Applications**

<div align="center">

**🎯 Transform Your Narrative Analysis Across Every Domain**

</div>

<table align="center" width="100%" style="border-collapse: collapse;">
<tr>
<td width="33%" align="center" style="padding: 15px;">
<div style="background: linear-gradient(135deg, #667eea 0%, #764ba2 100%); border-radius: 15px; padding: 20px; color: white; box-shadow: 0 8px 25px rgba(102, 126, 234, 0.3);">
<h3 style="margin: 0 0 10px 0; font-size: 1.2em;">🎬 Video Intelligence</h3>
<p style="margin: 0; font-size: 0.9em; opacity: 0.9;">Evaluate video language models' narrative comprehension</p>
</div>
</td>
<td width="33%" align="center" style="padding: 15px;">
<div style="background: linear-gradient(135deg, #f093fb 0%, #f5576c 100%); border-radius: 15px; padding: 20px; color: white; box-shadow: 0 8px 25px rgba(240, 147, 251, 0.3);">
<h3 style="margin: 0 0 10px 0; font-size: 1.2em;">📄 Document Similarity</h3>
<p style="margin: 0; font-size: 0.9em; opacity: 0.9;">Compare semantic alignment between long-form documents</p>
</div>
</td>
<td width="33%" align="center" style="padding: 15px;">
<div style="background: linear-gradient(135deg, #4facfe 0%, #00f2fe 100%); border-radius: 15px; padding: 20px; color: white; box-shadow: 0 8px 25px rgba(79, 172, 254, 0.3);">
<h3 style="margin: 0 0 10px 0; font-size: 1.2em;">📖 Story Analysis</h3>
<p style="margin: 0; font-size: 0.9em; opacity: 0.9;">Measure narrative similarity between different stories</p>
</div>
</td>
</tr>
<tr>
<td width="33%" align="center" style="padding: 15px;">
<div style="background: linear-gradient(135deg, #fa709a 0%, #fee140 100%); border-radius: 15px; padding: 20px; color: white; box-shadow: 0 8px 25px rgba(250, 112, 154, 0.3);">
<h3 style="margin: 0 0 10px 0; font-size: 1.2em;">🎓 Academic Research</h3>
<p style="margin: 0; font-size: 0.9em; opacity: 0.9;">Detect conceptual plagiarism and idea overlap</p>
</div>
</td>
<td width="33%" align="center" style="padding: 15px;">
<div style="background: linear-gradient(135deg, #a8edea 0%, #fed6e3 100%); border-radius: 15px; padding: 20px; color: #2d3748; box-shadow: 0 8px 25px rgba(168, 237, 234, 0.3);">
<h3 style="margin: 0 0 10px 0; font-size: 1.2em;">📝 Paragraph Analysis</h3>
<p style="margin: 0; font-size: 0.9em; opacity: 0.8;">Evaluate narrative coherence within text sections</p>
</div>
</td>
<td width="33%" align="center" style="padding: 15px;">
<div style="background: linear-gradient(135deg, #ffecd2 0%, #fcb69f 100%); border-radius: 15px; padding: 20px; color: #2d3748; box-shadow: 0 8px 25px rgba(255, 236, 210, 0.3);">
<h3 style="margin: 0 0 10px 0; font-size: 1.2em;">🎯 Short Caption Evaluation</h3>
<p style="margin: 0; font-size: 0.9em; opacity: 0.8;">Evaluate short captions and brief descriptions</p>
</div>
</td>
</tr>
</table>


---

## 🌟 Key Features

Explore the comprehensive capabilities that make VCS a powerful narrative evaluation toolkit. **To understand these features in detail, read our [research paper](https://arxiv.org/abs/placeholder-link), then visit our [interactive playground](https://multimodal-intelligence-lab.github.io/Video-Comprehension-Score/) to see them in action.**

<table width="100%" align="center" style="border: none; border-collapse: collapse;">
  <tr style="background-color: transparent;">
    <td style="padding: 10px; border: none; vertical-align: top;">
      <details style="border: 1px solid #14b8a6; border-radius: 12px; padding: 20px; background: linear-gradient(145deg, #1f2937, #111827); color: #e5e7eb; box-shadow: 0 4px 6px rgba(0, 0, 0, 0.1);">
        <summary style="cursor: pointer; font-weight: bold; font-size: 1.2em; color: #6ee7b7;">🧮 Comprehensive Metric Suite</summary>
        <p style="padding-top: 10px;">Computes VCS along with detailed breakdowns: GAS (global thematic similarity), LAS with precision/recall components, and NAS with distance-based and line-based sub-metrics. Access all internal calculations including penalty systems, mapping windows, and alignment paths.</p>
      </details>
    </td>
    <td style="padding: 10px; border: none; vertical-align: top;">
      <details style="border: 1px solid #14b8a6; border-radius: 12px; padding: 20px; background: linear-gradient(145deg, #1f2937, #111827); color: #e5e7eb; box-shadow: 0 4px 6px rgba(0, 0, 0, 0.1);">
        <summary style="cursor: pointer; font-weight: bold; font-size: 1.2em; color: #6ee7b7;">📊 Advanced Visualization Engine</summary>
        <p style="padding-top: 10px;">11 specialized visualization functions including similarity heatmaps, alignment analysis, best-match visualizations, narrative flow diagrams, and precision/recall breakdowns. Each metric component can be visualized with publication-quality plots.</p>
      </details>
    </td>
  </tr>
  <tr style="background-color: transparent;">
    <td style="padding: 10px; border: none; vertical-align: top;">
      <details style="border: 1px solid #14b8a6; border-radius: 12px; padding: 20px; background: linear-gradient(145deg, #1f2937, #111827); color: #e5e7eb; box-shadow: 0 4px 6px rgba(0, 0, 0, 0.1);">
        <summary style="cursor: pointer; font-weight: bold; font-size: 1.2em; color: #6ee7b7;">📋 Professional PDF Reports</summary>
        <p style="padding-top: 10px;">Generate comprehensive multi-page PDF reports with all metrics, visualizations, and analysis details. Supports both complete reports and customizable selective reports. Professional formatting suitable for research publications.</p>
      </details>
    </td>
    <td style="padding: 10px; border: none; vertical-align: top;">
      <details style="border: 1px solid #14b8a6; border-radius: 12px; padding: 20px; background: linear-gradient(145deg, #1f2937, #111827); color: #e5e7eb; box-shadow: 0 4px 6px rgba(0, 0, 0, 0.1);">
        <summary style="cursor: pointer; font-weight: bold; font-size: 1.2em; color: #6ee7b7;">⚙️ Flexible Configuration System</summary>
        <p style="padding-top: 10px;">Fine-tune evaluation with configurable parameters: chunk sizes, similarity thresholds, context windows, and Local Chronology Tolerance (LCT). Supports custom segmentation and embedding functions for domain-specific applications.</p>
      </details>
    </td>
  </tr>
</table>

---

## ⚡ Getting Started

Welcome to VCS Metrics! This guide will walk you through everything you need to start analyzing narrative similarity between texts. We'll cover installation, setup, and your first VCS analysis step by step.


---

### 📦 Step 1: Installation

Choose the installation method that fits your needs:

<table align="center" width="100%">
<tr>
<td width="50%" align="center">

### 🎯 **For Most Users** 
*Recommended if you just want to use VCS*

<details>
<summary><b>🖱️ Click to expand installation steps</b></summary>

<br>

**Terminal Installation:**
```bash
pip install video-comprehension-score
```

**Jupyter/Colab Installation:**
```bash
!pip install video-comprehension-score
```

<div align="center">

✅ **Ready in 30 seconds**  
🔥 **Zero configuration needed**  
⚡ **Instant access to all features**

</div>

</details>

</td>
<td width="50%" align="center">

### 🛠️ **For Developers**
*If you want to contribute or modify VCS*

<details>
<summary><b>🖱️ Click to expand development setup</b></summary>

<br>

**Terminal Installation:**
```bash
git clone https://github.com/hdubey-debug/vcs.git
cd vcs
pip install -e ".[dev]"
pre-commit install
```

**Jupyter/Colab Installation:**
```bash
!git clone https://github.com/hdubey-debug/vcs.git
%cd vcs
!pip install -e ".[dev]"
!pre-commit install
```

<div align="center">

🔧 **Latest features first**  
🧪 **Testing capabilities**  
🤝 **Contribution ready**

</div>

</details>

</td>
</tr>
</table>

---

### 🛠️ System Requirements

Before installing VCS, make sure your system meets these requirements:

<table align="center" width="100%">
<tr>
<td width="50%" align="center">

### 🐍 **Python**

<div style="background: linear-gradient(145deg, #dbeafe, #bfdbfe); padding: 20px; border-radius: 12px; border: 2px solid #3b82f6;">

<img src="https://img.shields.io/badge/Python-3.10+-306998?style=for-the-badge&logo=python&logoColor=white" alt="Python"/>

**Required: Python 3.10 or higher**

VCS Metrics uses modern Python features and requires Python 3.10+. We recommend Python 3.11+ for optimal performance.

</div>

</td>
<td width="50%" align="center">

### 🔥 **PyTorch**

<div style="background: linear-gradient(145deg, #fed7d7, #fca5a5); padding: 20px; border-radius: 12px; border: 2px solid #dc2626;">

<img src="https://img.shields.io/badge/PyTorch-1.9+-ee4c2c?style=for-the-badge&logo=pytorch&logoColor=white" alt="PyTorch"/>

**Required: PyTorch 1.9.0+**

VCS needs PyTorch but doesn't install it automatically to avoid conflicts. Get it from the [official PyTorch website](https://pytorch.org/get-started/locally/).

**💡 Pro Tip:** In Google Colab, PyTorch is pre-installed!

</div>

</td>
</tr>
</table>

<div align="center">
<table style="border: 2px solid #10b981; border-radius: 12px; background: linear-gradient(145deg, #d1fae5, #a7f3d0); padding: 15px; margin: 20px 0;">
<tr>
<td align="center">

**💡 Note:** VCS automatically installs dependencies: numpy≥1.20.0, matplotlib≥3.5.0, seaborn≥0.11.0

</td>
</tr>
</table>
</div>

---

### 🔧 Step 2: Prepare Your Functions

Now that VCS is installed, you need to define two functions before you can use the VCS API. Here's how VCS works:

#### 📋 VCS API Overview

```python
from vcs import compute_vcs_score

result = compute_vcs_score(
    reference_text="Your reference text here",
    generated_text="Your generated text here", 
    segmenter_fn=your_segmenter_function,        # ← You provide this
    embedding_fn_las=your_embedding_function,    # ← You provide this  
    embedding_fn_gas=your_embedding_function,    # ← You provide this
    return_all_metrics=True
)

print(f"VCS Score: {result['VCS']:.4f}")
```

As you can see, VCS requires two custom functions from you. Let's understand what each should do:


<table align="center" width="100%">
<tr>
<td width="50%" align="center">

### 🔪 **Segmenter Function**

<div style="background: linear-gradient(145deg, #fef3c7, #fde68a); padding: 20px; border-radius: 12px; border: 2px solid #f59e0b;">

**What it does:** Splits text into meaningful segments (sentences, paragraphs, etc.)

**Required signature:**
```python
def your_segmenter(text: str) -> List[str]:
    # Your implementation here
    return list_of_text_segments
```

**Arguments:** `text` (str) - Input text to be segmented  
**Returns:** `List[str]` - List of text segments  
**You can use:** Any library or model (NLTK, spaCy, custom logic, etc.)

</div>

</td>
<td width="50%" align="center">

### 🧠 **Embedding Function**

<div style="background: linear-gradient(145deg, #f3e8ff, #e9d5ff); padding: 20px; border-radius: 12px; border: 2px solid #7c3aed;">

**What it does:** Converts text segments into numerical vectors (embeddings)

**Required signature:**
```python
def your_embedder(texts: List[str]) -> torch.Tensor:
    # Your implementation here  
    return tensor_of_embeddings
```

**Arguments:** `texts` (List[str]) - List of text segments to embed  
**Returns:** `torch.Tensor` - Tensor of shape `(len(texts), embedding_dim)`  
**You can use:** Any embedding model (sentence-transformers, OpenAI, etc.)

</div>

</td>
</tr>
</table>

#### 🌟 Author Recommendations (2025)

For best results, we recommend these state-of-the-art models:

<div align="center">
<table style="border: 2px solid #dc2626; border-radius: 12px; background: linear-gradient(145deg, #fecaca, #fca5a5); padding: 15px; margin: 20px 0;">
<tr>
<td align="center">

⚠️ **Note:** These recommendations are current as of 2025. Always research the latest SOTA options.

</td>
</tr>
</table>
</div>

<table align="center" width="100%">
<tr>
<td width="50%" align="center">

### 🔪 **Segmentation Champion**

<div align="center">
<img src="https://img.shields.io/badge/Segment_Any_Text-SAT-ff6b6b?style=for-the-badge&logo=artificial-intelligence&logoColor=white" alt="SAT"/>
</div>

<div style="background: linear-gradient(145deg, #fff5f5, #fed7d7); padding: 20px; border-radius: 12px; border: 2px solid #e53e3e;">

**🏆 Recommended: Segment Any Text (SAT)**

✨ **Why we recommend SAT:**
- 🎯 State-of-the-art segmentation accuracy  
- ⚡ Intelligent boundary detection  
- 🧠 Context-aware text splitting  
- 🔬 Research-grade performance  

🔗 **Repository:** [github.com/segment-any-text/wtpsplit](https://github.com/segment-any-text/wtpsplit)

</div>

</td>
<td width="50%" align="center">

### 🧠 **Embedding Powerhouse**

<div align="center">
<img src="https://img.shields.io/badge/NVIDIA-NV--Embed--v2-76b900?style=for-the-badge&logo=nvidia&logoColor=white" alt="NV-Embed"/>
</div>

<div style="background: linear-gradient(145deg, #f0fff4, #c6f6d5); padding: 20px; border-radius: 12px; border: 2px solid #38a169;">

**🥇 Recommended: nv-embed-v2**

🌟 **Why we recommend nv-embed-v2:**
- 📊 Top performer on [MTEB Leaderboard](https://huggingface.co/spaces/mteb/leaderboard)  
- 🚀 Superior semantic understanding  
- 💪 Robust multilingual support  
- ⚡ Excellent for VCS analysis  

🔗 **Model:** [nvidia/NV-Embed-v2](https://huggingface.co/nvidia/NV-Embed-v2)

</div>

</td>
</tr>
</table>

<div align="center">
<table style="border: 2px solid #3182ce; border-radius: 12px; background: linear-gradient(145deg, #bee3f8, #90cdf4); padding: 15px; margin: 20px 0;">
<tr>
<td align="center">

**💡 Alternative Options:** NLTK, spaCy, sentence-transformers, or build your own custom functions!

</td>
</tr>
</table>
</div>

---

### 💻 Step 3: Run Your First VCS Analysis

Now let's see VCS in action with a complete working example:

<div align="center">
<table style="border: 2px solid #059669; border-radius: 12px; background: linear-gradient(145deg, #d1fae5, #a7f3d0); padding: 15px; margin: 20px 0;">
<tr>
<td align="center">

**⚡ Performance Notes**  
*SOTA models require GPU. For CPU testing, this example uses lightweight alternatives.*

</td>
</tr>
</table>
</div>

<details>
<summary><h3><b>🚀 Quick Example</b> - Click to expand complete tutorial</h3></summary>

<div style="background: linear-gradient(145deg, #1f2937, #111827); padding: 25px; border-radius: 15px; border: 2px solid #6366f1;">

### 🎯 **Complete Working Example**
*Copy, paste, and run this code to see VCS in action*

```python
# Fix import path issue if running from vcs/ root directory
import sys
import os
if os.path.basename(os.getcwd()) == 'vcs' and os.path.exists('src/vcs'):
    sys.path.insert(0, 'src')
    print("🔧 Fixed import path for development directory")

# Test the installation
try:
    import vcs
    print("✅ VCS package imported successfully!")
    
    # Test main function availability
    if hasattr(vcs, 'compute_vcs_score'):
        print("✅ Main function 'compute_vcs_score' is available!")
    else:
        print("⚠️ Main function not found - there might be an installation issue")
        
    # Try to get version
    try:
        print(f"📦 Version: {vcs.__version__}")
    except AttributeError:
        print("📦 Version: Unable to determine (this is normal for development installs)")
        
except ImportError as e:
    print(f"❌ Import failed: {e}")
    print("💡 Make sure you:")
    print("   1. Installed VCS correctly: pip install -e .[dev]")
    print("   2. Restarted your notebook kernel") 
    print("   3. You're NOT in the root vcs/ directory (this causes import conflicts)")

# Import required libraries
import torch
from typing import List

# Define lightweight segmenter function
def simple_segmenter(text: str) -> List[str]:
    """
    Simple sentence segmenter using period splitting.
    
    Args:
        text: Input text to segment
        
    Returns:
        List of text segments
    """
    # Split by periods and clean up
    segments = [s.strip() for s in text.split('.') if s.strip()]
    return segments

# Define lightweight embedding function using sentence-transformers
def lightweight_embedding_function(texts: List[str]) -> torch.Tensor:
    """
    Lightweight embedding function using sentence-transformers.
    
    Args:
        texts: List of text segments to embed
        
    Returns:
        PyTorch tensor of shape (len(texts), embedding_dim)
    """
    try:
        from sentence_transformers import SentenceTransformer
        
        # Use a lightweight model (only downloads ~80MB)
        model = SentenceTransformer('all-MiniLM-L6-v2')
        
        # Generate embeddings
        embeddings = model.encode(texts)
        return torch.tensor(embeddings, dtype=torch.float32)
        
    except ImportError:
        print("⚠️ sentence-transformers not found. Installing...")
        import subprocess
        import sys
        subprocess.check_call([sys.executable, "-m", "pip", "install", "sentence-transformers"])
        
        # Try again after installation
        from sentence_transformers import SentenceTransformer
        model = SentenceTransformer('all-MiniLM-L6-v2')
        embeddings = model.encode(texts)
        return torch.tensor(embeddings, dtype=torch.float32)

# Example texts
reference_text = """
The quick brown fox jumps over the lazy dog.
It was a beautiful sunny day in the forest.
The fox was looking for food for its family.
"""

generated_text = """
A brown fox jumped over a sleeping dog.
The weather was nice and sunny in the woods.
The fox needed to find food for its cubs.
"""

# Compute VCS score
print("🧠 Computing VCS score...")
try:
    result = vcs.compute_vcs_score(
        reference_text=reference_text,
        generated_text=generated_text,
        segmenter_fn=simple_segmenter,
        embedding_fn_las=lightweight_embedding_function,
        embedding_fn_gas=lightweight_embedding_function,
        return_all_metrics=True,
        return_internals=True
    )
    
    print("🎯 VCS Results:")
    print(f"VCS Score: {result['VCS']:.4f}")
    print(f"GAS Score: {result['GAS']:.4f}")
    print(f"LAS Score: {result['LAS']:.4f}")
    print(f"NAS Score: {result['NAS']:.4f}")
    print("✅ VCS is working correctly!")
    
    # Generate visualization (optional)
    if 'internals' in result:
        try:
            fig = vcs.visualize_metrics_summary(result['internals'])
            print("📊 Visualization generated successfully!")
            # fig.show()  # Uncomment to display
        except Exception as viz_error:
            print(f"⚠️ Visualization failed (this is normal in some environments): {viz_error}")
    
except Exception as e:
    print(f"❌ Error running VCS: {e}")
    print("💡 Make sure PyTorch is installed and try restarting your kernel")
```

<div align="center">
<table style="border: 2px solid #3b82f6; border-radius: 12px; background: linear-gradient(145deg, #dbeafe, #bfdbfe); padding: 15px; margin: 20px 0;">
<tr>
<td align="center">

**📝 Scale Note:** This example uses small text for illustration - VCS excels with long-form content!  
**⚠️ Import Tip:** Running from `vcs/` root? The example includes an automatic path fix.

</td>
</tr>
</table>
</div>

</div>

</details>

---

## ⚙️ Advanced Configuration

Once you're comfortable with the basics, you can fine-tune VCS behavior for your specific use case:

<table align="center" width="100%">
<tr>
<td width="33%" align="center">

### 🎯 **Core Parameters**

<div style="background: linear-gradient(145deg, #ede9fe, #ddd6fe); padding: 20px; border-radius: 12px; border: 2px solid #7c3aed;">

**🎛️ Essential Controls:**

| Parameter | Default | Purpose |
|:----------|:-------:|:--------|
| `chunk_size` | 1 | Segment grouping |
| `context_cutoff_value` | 0.6 | Similarity threshold |
| `context_window_control` | 4.0 | Context window size |
| `lct` | 0 | Narrative reordering tolerance |

</div>

</td>
<td width="33%" align="center">

### 📊 **Return All Metrics**

<div style="background: linear-gradient(145deg, #fef3c7, #fde68a); padding: 20px; border-radius: 12px; border: 2px solid #f59e0b;">

**🎛️ Control Parameter:**

| Parameter | Default | Purpose |
|:----------|:-------:|:--------|
| `return_all_metrics` | False | Return detailed metric breakdown |

**When set to `True`, you get:**
- Individual GAS, LAS, NAS scores
- LAS precision and recall components
- Distance-based and line-based NAS sub-metrics
- Complete metric breakdown for analysis

</div>

</td>
<td width="33%" align="center">

### 🔍 **Return Internals**

<div style="background: linear-gradient(145deg, #e0f2fe, #b3e5fc); padding: 20px; border-radius: 12px; border: 2px solid #0288d1;">

**🎛️ Control Parameter:**

| Parameter | Default | Purpose |
|:----------|:-------:|:--------|
| `return_internals` | False | Return internal computation data |

**When set to `True`, you get:**
- Similarity matrices and alignment paths
- Mapping windows and penalty calculations
- Text chunks and segmentation details
- All data needed for visualization

</div>

</td>
</tr>
</table>

<table align="center" width="100%" style="margin-top: 20px;">
<tr>
<td width="100%">
<div align="center">
<h3>🚀 **Example Configuration**</h3>
</div>
<div style="background: linear-gradient(145deg, #ecfdf5, #d1fae5); padding: 20px; border-radius: 12px; border: 2px solid #059669; text-align: left;">

```python
# 🎯 Comprehensive configuration with all features enabled
result = compute_vcs_score(
    reference_text=ref_text,
    generated_text=gen_text,
    segmenter_fn=segmenter,
    embedding_fn_las=embedder,
    embedding_fn_gas=embedder,
    chunk_size=2,                  # Group segments
    context_cutoff_value=0.7,      # Higher threshold
    context_window_control=3.0,    # Tighter windows
    lct=1,                         # Some reordering OK
    return_all_metrics=True,       # Get detailed breakdown
    return_internals=True          # Get visualization data
)
```

</div>

</td>
</tr>
</table>

<div align="center">
<table style="border: 2px solid #3b82f6; border-radius: 12px; background: linear-gradient(145deg, #dbeafe, #bfdbfe); padding: 15px; margin: 20px 0;">
<tr>
<td align="center">

**📚 For complete API documentation and visualization guides, visit our [API Documentation](https://multimodal-intelligence-lab.github.io/Video-Comprehension-Score/)**

</td>
</tr>
</table>
</div>

---

## ❓ Frequently Asked Questions

<details>
<summary><strong>🤔 How does VCS differ from BLEU/ROUGE?</strong></summary>
<p>Unlike BLEU and ROUGE which rely on hard n-gram matching, VCS utilizes latent space matching by comparing embeddings at both global and local chunk levels. VCS also evaluates the chronological order of content chunks and combines these three dimensions to generate a comprehensive final score that better captures semantic similarity and narrative structure.</p>
</details>

<details>
<summary><strong>⚡ What's the minimum text length for VCS?</strong></summary>
<p>VCS works with any text length, but it's optimized for longer texts (100+ words) where narrative structure is important. For very short texts, simpler metrics might be more appropriate.</p>
</details>

<details>
<summary><strong>📏 What's the maximum text length for VCS?</strong></summary>
<p>There is no upper limit on text length for VCS. The framework is designed to handle texts of any size, from short paragraphs to extensive documents, making it suitable for large-scale narrative evaluation tasks.</p>
</details>

<details>
<summary><strong>🧠 Which embedding models work best?</strong></summary>
<p>We recommend checking the <a href="https://huggingface.co/spaces/mteb/leaderboard">MTEB leaderboard</a> for the latest SOTA models. As of 2024, nv-embed-v2 and similar transformer-based models provide excellent results.</p>
</details>

<details>
<summary><strong>🎯 How do I control the granularity of comparison?</strong></summary>
<p>Use the <code>chunk_size</code> parameter to control the granularity of text comparison. A smaller chunk size provides more fine-grained analysis, while a larger chunk size offers broader, more general comparisons. The default value is 1 for maximum granularity.</p>
</details>

<details>
<summary><strong>⏱️ How do I control the strictness of chronological matching?</strong></summary>
<p>Use the <code>lct</code> (Local Chronology Tolerance) parameter to control chronological matching strictness. A higher LCT value means more lenient chronological ordering, allowing for greater flexibility in narrative sequence evaluation. The default value is 0 for strict chronological matching.</p>
</details>

<details>
<summary><strong>🔗 Can I use different embedding functions for GAS and LAS?</strong></summary>
<p>Yes, you can specify different embedding functions for Global Alignment Score (GAS) and Local Alignment Score (LAS) using the <code>embedding_fn_gas</code> and <code>embedding_fn_las</code> parameters respectively. This allows you to optimize each component with models best suited for their specific evaluation tasks.</p>
</details>

---

## 🏗️ Project Structure

```
vcs/
├── 📁 src/vcs/                  # Main package source code
│   ├── 📄 __init__.py           # Package initialization
│   ├── 📄 scorer.py             # Main VCS API entry point
│   ├── 📄 _config.py            # Configuration settings
│   ├── 📁 _metrics/             # Core VCS metrics implementations
│   │   ├── 📁 _gas/             # Global Alignment Score
│   │   ├── 📁 _las/             # Local Alignment Score  
│   │   ├── 📁 _nas/             # Narrative Alignment Score with components
│   │   │   └── 📁 _nas_components/  # Distance NAS, Line NAS, Regularize NAS
│   │   └── 📁 _vcs/             # Combined VCS computation
│   ├── 📁 _visualize_vcs/       # Comprehensive visualization suite
│   │   ├── 📁 _similarity_matrix/  # Similarity matrix visualizations
│   │   ├── 📁 _best_match/      # Best match analysis plots
│   │   ├── 📁 _distance_nas/    # Distance-based NAS visualizations
│   │   ├── 📁 _line_nas/        # Line-based NAS visualizations
│   │   ├── 📁 _mapping_windows/ # Context window visualizations
│   │   ├── 📁 _metrics_summary/ # Overall metrics summary plots
│   │   ├── 📁 _pdf_report/      # PDF report generation
│   │   ├── 📁 _text_chunks/     # Text chunk visualizations
│   │   ├── 📁 _window_regularizer/ # Window regularizer plots
│   │   ├── 📁 _las/             # LAS-specific visualizations
│   │   └── 📁 _config/          # Visualization configuration
│   ├── 📁 _segmenting/          # Text segmentation utilities
│   ├── 📁 _matching/            # Optimal text matching algorithms
│   ├── 📁 _mapping_windows/     # Context window management
│   └── 📁 _utils/               # Helper utilities
├── 📁 docs/                     # Documentation and interactive demos
│   ├── 📄 index.html            # Main documentation website
│   ├── 📁 pages/                # Documentation pages
│   │   ├── 📄 api.html          # API reference
│   │   ├── 📄 playground.html   # Interactive playground
│   │   └── 📄 example.html      # Usage examples
│   ├── 📁 widgets/              # Interactive visualization widgets
│   ├── 📁 sphinx/               # Sphinx documentation source
│   └── 📁 assets/               # Documentation assets (CSS, JS, videos)
├── 📁 .github/                  # GitHub configuration
│   ├── 📁 assets/               # README assets (images, gifs)
│   ├── 📁 scripts/              # GitHub automation scripts
│   └── 📁 workflows/            # CI/CD automation pipelines
│       ├── 📄 test.yml          # Continuous testing
│       ├── 📄 publish.yml       # Package publishing
│       └── 📄 deploy-docs.yml   # Documentation deployment
├── 📄 pyproject.toml           # Package configuration & dependencies
├── 📄 CONTRIBUTING.md          # Development contribution guide
├── 📄 DEPLOYMENT.md            # Release and deployment guide
├── 📄 CHANGELOG.md             # Version history and changes
├── 📄 MANIFEST.in              # Package manifest
├── 📄 tag_version.py           # Version tagging script
├── 📄 LICENSE                  # MIT license
└── 📄 README.md                # This documentation
```

---

## 🚀 Development & Contributing

We welcome contributions to VCS Metrics! Whether you're fixing bugs, adding features, or improving documentation, here's how to get started.

### 🛠️ Quick Development Setup

<details>
<summary><b>🖱️ Click to expand development setup</b></summary>

<br>

```bash
# 1. Clone and setup
git clone https://github.com/hdubey-debug/vcs.git
cd vcs
python -m venv venv
source venv/bin/activate  # Windows: venv\Scripts\activate

# 2. Install development dependencies
pip install -e .[dev]

# 3. Create your feature branch
git checkout -b feature/your-feature-name

# 4. Make your changes
# Edit files in src/vcs/
# Add tests if needed
# Update docs if necessary

# 5. Run quality checks
black src/ && isort src/ && flake8 src/ && mypy src/

# 6. Commit with semantic format
git commit -m "minor: add new awesome feature"

# 7. Push and create PR
git push origin feature/your-feature-name
```

</details>

### 📋 Contribution Workflow

<table align="center" width="100%">
<tr>
<td width="50%" align="center">

### 🔄 **Development Process**

<div style="background: linear-gradient(145deg, #dbeafe, #bfdbfe); padding: 20px; border-radius: 12px; border: 2px solid #3b82f6;">

**1. Fork & Clone**  
**2. Create Feature Branch**  
**3. Make Changes**  
**4. Write Tests**  
**5. Submit PR**  
**6. Code Review**  
**7. Merge to Main**  

✅ **Automated testing on every PR**  
✅ **Fast feedback in ~2-3 minutes**

</div>

</td>
<td width="50%" align="center">

### 📦 **Release Process**

<div style="background: linear-gradient(145deg, #ecfdf5, #d1fae5); padding: 20px; border-radius: 12px; border: 2px solid #059669;">

**1. Semantic Commit Messages**  
**2. GitHub Release Creation**  
**3. Automated Version Calculation**  
**4. Package Building**  
**5. TestPyPI Publishing**  
**6. Production Release**  

🚀 **Industry-standard CI/CD pipeline**  
⚡ **Zero manual version management**

</div>

</td>
</tr>
</table>

### 💡 Semantic Commit Format

We use semantic commits for automatic version bumping:

<div align="center">
<table style="border: 2px solid #7c3aed; border-radius: 12px; background: linear-gradient(145deg, #f3e8ff, #e9d5ff); padding: 15px; margin: 20px 0;">
<tr>
<td align="center">

| **Commit Type** | **Version Bump** | **Example** |
|:---|:---:|:---|
| `minor: description` | New features | `1.0.4 → 1.1.0` |
| `major: description` | Breaking changes | `1.0.4 → 2.0.0` |
| `anything else` | Bug fixes (default) | `1.0.4 → 1.0.5` |

</td>
</tr>
</table>
</div>

### 🔧 Automated Testing & CI/CD

Our comprehensive CI/CD pipeline ensures code quality and reliability on every commit:

<div align="center">
<table style="border: 2px solid #059669; border-radius: 12px; background: linear-gradient(145deg, #ecfdf5, #d1fae5); padding: 20px; margin: 20px 0;">
<tr>
<td align="center">

### 🚀 **What Gets Tested**

**✅ Matrix Testing** - Python 3.11 & 3.12 compatibility  
**✅ Package Validation** - Import testing & API availability  
**✅ Integration Testing** - Full getting-started example  
**✅ Code Quality** - Flake8 linting & complexity checks  
**✅ Build Testing** - Package build verification  

**🔄 Triggers:** Every push and pull request to `main`

</td>
</tr>
</table>
</div>

<div align="center">
<table style="border: 2px solid #059669; border-radius: 12px; background: linear-gradient(145deg, #ecfdf5, #d1fae5); padding: 15px; margin: 20px 0;">
<tr>
<td align="center">

[![Tests](https://img.shields.io/github/actions/workflow/status/hdubey-debug/vcs/test.yml?branch=main&label=Tests&logo=github-actions&logoColor=white&style=for-the-badge)](https://github.com/hdubey-debug/vcs/actions/workflows/test.yml)
[![Build](https://img.shields.io/github/actions/workflow/status/hdubey-debug/vcs/publish.yml?label=Build&logo=github-actions&logoColor=white&style=for-the-badge)](https://github.com/hdubey-debug/vcs/actions/workflows/publish.yml)

**✅ Automated testing ensures every change is production-ready**

</td>
</tr>
</table>
</div>

### 📖 Detailed Guides

For comprehensive information about contributing and development:

<div align="center">

[![Contributing Guide](https://img.shields.io/badge/📖_Full_Contributing_Guide-2563eb?style=for-the-badge&logo=gitbook&logoColor=white)](./CONTRIBUTING.md)
[![Deployment Guide](https://img.shields.io/badge/🚀_Deployment_Guide-059669?style=for-the-badge&logo=rocket&logoColor=white)](./DEPLOYMENT.md)

</div>

### 🤝 Getting Help

<table align="center" width="100%">
<tr>
<td width="33%" align="center">

**🐛 Bug Reports**  
[Create GitHub Issue](https://github.com/hdubey-debug/vcs/issues)

</td>
<td width="33%" align="center">

**💬 Questions**  
[GitHub Discussions](https://github.com/hdubey-debug/vcs/discussions)

</td>
<td width="33%" align="center">

**💡 Feature Requests**  
[Feature Request Issue](https://github.com/hdubey-debug/vcs/issues/new)

</td>
</tr>
</table>

---

## 📚 Citation

If you use VCS Metrics in your research, please cite:

```bibtex
@software{vcs_metrics_2024,
  title = {VCS Metrics: Video Comprehension Score for Text Similarity Evaluation},
  author = {Dubey, Harsh and Ali, Mukhtiar and Mishra, Sugam and Pack, Chulwoo},
  year = {2024},
  institution = {South Dakota State University},
  url = {https://github.com/hdubey-debug/vcs},
  note = {Python package for narrative similarity evaluation}
}
```

---

## 🤖 CLIP-CC Ecosystem Integration

VCS is designed to work seamlessly with [CLIP-CC Dataset](https://github.com/hdubey-debug/CLIP-CC) for comprehensive video understanding evaluation.

<div align="center">
<table style="border: 2px solid #7c3aed; border-radius: 12px; background: linear-gradient(145deg, #f3e8ff, #e9d5ff); padding: 20px; margin: 20px 0;">
<tr>
<td align="center">

[![CLIP-CC Dataset](https://img.shields.io/badge/🤖_Companion_Dataset-CLIP--CC-9333ea?style=for-the-badge&logo=python&logoColor=white)](https://github.com/hdubey-debug/CLIP-CC)

**🔄 Perfect Integration: VCS + CLIP-CC**
- 🎥 **CLIP-CC provides the data** → Rich video dataset with human summaries
- 🔍 **VCS provides the evaluation** → Advanced narrative comprehension metrics  
- 🏆 **Together: Complete research pipeline** → From data loading to evaluation

</td>
</tr>
</table>
</div>

---

## 🏆 Meet Our Contributors

<div align="center">

### 🌟 **The VCS Team - Building the Future of Text Similarity**

</div>

<table>
<tr>
<td align="center">

<a href="https://github.com/hdubey-debug">
  <img src="https://github.com/hdubey-debug.png" width="100" height="100" style="border-radius: 50%;"/>
</a>

**Harsh Dubey**  
*Lead Developer & Research Scientist*  
*South Dakota State University*

| Commits | Lines | Files |
|:---:|:---:|:---:|
| **2** | **49K** | **171** |

**📋 Key Work:**
• VCS Algorithm Architecture  
• Visualization Engine  
• LAS, GAS, and NAS Metrics  

[![GitHub](https://img.shields.io/badge/-GitHub-14b8a6?style=flat&logo=github)](https://github.com/hdubey-debug)

</td>
</tr>
</table>

<div align="center">

### 🤖 **Automated Contributors**

| **Contributor** | **Role** | **Contributions** | **Badge** |
|:---:|:---:|:---:|:---:|
| 🤖 **GitHub Actions** | CI/CD Automation | Clean history setup | [![Bot](https://img.shields.io/badge/Bot-Automated_Testing-6c5ce7?style=flat&logo=github-actions&logoColor=white)](#) |

###