Initial Commit

ReadMe.md

@@ -0,0 +1,179 @@
+# VidInsight AI: AI-Powered YouTube Content Analyzer
+
+## Overview
+VidInsight AI is an AI-powered application designed to analyze YouTube videos for a given subject, extract insights, provide transcriptions, topic, summary, key-points and a new content idea! 
+The application is built to assist:
+- content creators,
+- educators & researchers, and
+- everyday users in understanding video content quickly and effectively.
+
+---
+This ReadMe file documents the current phase of the project and will be updated as new features are implemented.
+
+**Current Features (Asif's Code):**
+
+	1.	YouTube Video Retrieval:
+    	•	Fetches up to 10 YouTube videos based on a user-provided topic.
+    	•	Filters videos based on criteria such as keywords, view counts, and trusted channels.
+    	•	Selects the top 3 videos based on relevance and view counts.
+    
+	2.	Transcription:
+    	•	Transcribes audio from the top 3 selected videos using OpenAI’s Whisper model.
+    	•	Saves the complete transcripts in an `output` folder for further processing.
+    
+	3.	User Interface:
+    	•	Input
+        	•	Provides a user-friendly interface built with Gradio.
+    	•	Output
+        	•	Displays video details (title, channel, views) and a preview of the transcription.
+        	•	Analysis (Topic, Summary & Key Points)
+        	•	Content Idea with comprehensive details
+---
+
+## Project Structure
+
+VidInsight-AI/\
+├── app.py                                 # Gradio web interface for user interaction\
+├── config.py                              # Configuration file for API keys and filters\
+├── fetch_youtube_videos.py                # Fetches and filters YouTube videos\
+├── transcribe_videos.py                   # Transcribes videos and saves transcripts\
+├── summary.py                             # Generates summaries from transcriptions\
+├── YouTubeAgent.py                        # Creates content ideas using Gemini AI\
+├── main.py                                # CLI-based alternative to run the app\
+├── requirements.txt                       # Project dependencies\
+├── keys1.env                              # Environment variables (API keys)\
+└── output/                                # Folder for saved transcripts\
+&nbsp;&nbsp;&nbsp; └── <video_id>.txt     &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; # Transcripts saved as text files\
+
+### Key Components:
+	1.	Interface Files:
+    	•	`app.py`: Web interface using Gradio
+    	•	`main.py`: Command-line interface
+	2.	Core Processing Files:
+    	•	`fetch_youtube_videos.py`: Video retrieval
+    	•	`transcribe_videos.py`: Audio transcription
+    	•	`summary.py`: Content summarization
+    	•	`YouTubeAgent.py`: Content idea generation
+	3.	Configuration Files:
+    	•	`config.py`: Settings and filters
+    	•	`keys1.env`: API keys
+    	•	`requirements.txt`: Dependencies
+	4.	Output Directory:
+    	•	`output/`: Stores generated transcripts
+
+---
+
+## Setup Instructions (need to be completed)
+
+1. Prerequisites\
+	•	Python 3.8 or higher\
+	•	FFmpeg installed on the system (for audio processing)
+	•	A YouTube Data API key (create one via Google Cloud Console)
+	•	A GEMINI API key 
+	•	A TAVILY API key 
+
+3. Installation
+	1.	Clone the repository:
+      ```python
+      git clone <repository_url>
+      ```
+   	2.	Install required dependencies:
+
+   	3.	Set up your API key:
+	•	Create a `.env` file or update `keys1.env` with your YouTube API key:
+    ```python
+    YOUTUBE_API_KEY="your_api_key_here"
+    GEMINI_API_KEY="your_api_key_here"
+    TAVILY_API_KEY="your_api_key_here"
+    ```
+   \
+    
+4. Running the Application\
+	•	Using the Gradio Interface:
+    ```python
+    python app.py
+    ```
+    \
+   •	Using the CLI:
+    ```python
+    python main.py
+    ```
+---
+
+## Usage
+
+#### Gradio App
+	1.	Enter a topic in the “Enter learning topic” field (e.g., “Machine Learning”).
+	2.	Click “Submit” to fetch and analyze videos.
+	3.	View results, including:
+    	•	Video title, channel name, view count.
+    	•	A preview of the transcription.
+    	•	The path to the saved transcript file.
+    	•	Topic, Summary, and Key-Points
+    	•	A New Content Idea with Compreehensive Details    
+#### Output Folder
+	•	Complete transcripts are saved in the `output/` folder as `.txt` files.
+	���	File names are based on unique YouTube video IDs (e.g., `ukzFI9rgwfU.txt`).
+
+---
+
+## Configuration
+
+The `config.py` file allows customization of filtering criteria:
+```python
+FILTER_CONFIG = {
+    "videoDuration": "medium",  # Focus on videos between 4 and 20 minutes
+    "order": "relevance",       # Sort by relevance
+    "trusted_channels": {
+        "Khan Academy": "UC4a-Gbdw7vOaccHmFo40b9g",
+        "edX": "UCEBb1b_L6zDS3xTUrIALZOw",
+        "Coursera": "UC58aowNEXHHnflR_5YTtP4g",
+    },
+    "teaching_keywords": {"tutorial", "lesson", "course", "how-to", "introduction", "basics"},
+    "non_teaching_keywords": {"fun", "experiment", "joke", "prank", "vlog"},
+    "max_results": 10,          # Maximum number of videos fetched from YouTube API
+    "min_view_count": 10000     # Minimum view count for relevance
+}
+```
+
+---
+
+## Known Issues
+	1.	If no results are found or an error occurs during video fetching, the app displays an error message in JSON format.
+	2.	Ensure that valid topics are entered; overly broad or unrelated topics may not yield meaningful results.
+
+---
+
+## Future Features        
+	1.	Multilingual Support (Future):
+    	•	Add support for transcription in other languages (e.g., Spanish, French).
+        
+	2.	Interactive Q&A (Future):
+    	•	Allow users to ask questions about analyzed video content.
+
+---
+
+## 🛠️ Technology Stack
+
+| Task  | Technology |
+| -------- | ------- |
+| Video Retrieval | YouTube Data API, google-api-python-client   |
+| Transcription | yt-dlp, OpenAI Whisper     |
+| Summarization  | Gemini AI, LangChain  |
+| Content Generation | Gemini AI, LangChain   |
+| Vectorizaton | ____  |
+| Vector Database | ____  |
+
+
+---
+## 📌 Contributors
+	•	Asif Khan – Developer and Project Lead
+    •	Kade Thomas – Summarization Specialist
+    •	Amit Gaikwad - Vector Database Specialist
+    •	Simranpreet Saini – AI Agent Specialist
+    •	Jason Brooks – Documentation Specialist
+
+---
+## 🙏 Acknowledgements 
+- Special thanks to Firas Obeid for being an advisor on the project
+- Special thanks to OpenAI, Hugging Face, and YouTube API, Gemini API, and Tavily API for providing the tools that made this project possible. 🚀

YouTubeAgent.py

@@ -0,0 +1,169 @@
+"""
+# YouTube Content Idea Generator Module
+
+This module leverages Google's Gemini AI to generate structured content ideas for YouTube videos
+based on provided summaries and key points.
+
+## Summary
+- Uses Gemini AI model for content generation
+- Creates detailed video proposals including:
+  - Title and hook
+  - Main talking points
+  - Video structure
+  - Thumbnail concepts
+  - Target audience
+  - SEO keywords
+- Formats output with clear section separation
+
+## Dependencies
+
+### System Requirements
+- Python 3.8+
+- Internet connection for API calls
+
+### Package Dependencies
+1. **langchain-google-genai**
+   - Install: `pip install langchain-google-genai`
+   - Purpose: Interface with Gemini AI model
+
+2. **langchain-community**
+   - Install: `pip install langchain-community`
+   - Purpose: Access to Tavily search tools
+
+3. **python-dotenv**
+   - Install: `pip install python-dotenv`
+   - Purpose: Load environment variables
+
+### Project Dependencies
+1. **keys1.env file**
+   - Must contain:
+     - GEMINI_API_KEY
+     - TAVILY_API_KEY
+   - Format:
+     ```
+     GEMINI_API_KEY=your_gemini_api_key
+     TAVILY_API_KEY=your_tavily_api_key
+     ```
+
+2. **Input Requirements**
+   - Dictionary containing:
+     - summary: Text summarizing content
+     - keypoints: List of key points
+
+## Functions
+generateidea(input)
+- Args: Dictionary with 'summary' and 'keypoints'
+- Returns: Formatted string containing structured content idea
+- Error Returns: Error message if generation fails
+
+## Returns
+Structured string containing:
+1. Title
+2. Description/Hook
+3. Main Talking Points
+4. Video Structure
+5. Thumbnail Concepts
+6. Target Audience
+7. Estimated Length
+8. SEO Keywords
+
+## Error Handling
+- Returns error message if:
+  - API keys are missing
+  - API calls fail
+  - Response formatting fails
+"""
+
+
+from langchain_google_genai import ChatGoogleGenerativeAI
+from langchain_community.tools.tavily_search import TavilySearchResults
+from dotenv import load_dotenv, find_dotenv
+import os
+from langchain.agents import initialize_agent
+from langchain_community.agent_toolkits.load_tools import load_tools
+
+# Load environment variables
+load_dotenv(find_dotenv('keys1.env'))
+
+# Set the model name and API keys
+GEMINI_MODEL = "gemini-1.5-flash"
+GEMINI_API_KEY = os.getenv("GEMINI_API_KEY")
+os.environ["TAVILY_API_KEY"] = os.getenv("TAVILY_API_KEY")
+
+def generateidea(input):
+    """Generate content ideas based on summary and key points."""
+    try:
+        # Initialize the model with higher temperature for creativity
+        llm = ChatGoogleGenerativeAI(
+            google_api_key=GEMINI_API_KEY,
+            model=GEMINI_MODEL,
+            temperature=0.7,
+            top_p=0.9,
+            max_output_tokens=2048  # Ensure longer output
+        )
+        
+        # Create a specific prompt template
+        prompt = f"""
+        Based on this content:
+        Summary: {input["summary"]}
+        Key Points: {input["keypoints"]}
+
+        Generate a detailed YouTube video idea using exactly this format:
+
+        1. **Title:**
+        [Create an attention-grabbing, SEO-friendly title]
+
+        2. **Description/Hook:**
+        [Write 2-3 compelling sentences that hook viewers]
+
+        3. **Main Talking Points:**
+        • [Main point 1]
+        • [Main point 2]
+        • [Main point 3]
+        • [Main point 4]
+        • [Main point 5]
+
+        4. **Suggested Video Structure:**
+        • [00:00-02:00] Introduction
+        • [02:00-05:00] First Topic
+        • [05:00-08:00] Second Topic
+        • [08:00-12:00] Third Topic
+        • [12:00-15:00] Examples and Applications
+        • [15:00-17:00] Conclusion
+
+        5. **Potential Thumbnail Concepts:**
+        • [Thumbnail idea 1]
+        • [Thumbnail idea 2]
+        • [Thumbnail idea 3]
+
+        6. **Target Audience:**
+        [Describe ideal viewer demographic and background]
+
+        7. **Estimated Video Length:**
+        [Specify length in minutes]
+
+        8. **Keywords for SEO:**
+        [List 8-10 relevant keywords separated by commas]
+
+        Ensure each section is detailed and properly formatted.
+        """
+
+        # Generate response directly with LLM
+        response = llm.predict(prompt)
+        
+        # Format the response
+        formatted_response = response.replace("1. **", "\n\n1. **")
+        formatted_response = formatted_response.replace("2. **", "\n\n2. **")
+        formatted_response = formatted_response.replace("3. **", "\n\n3. **")
+        formatted_response = formatted_response.replace("4. **", "\n\n4. **")
+        formatted_response = formatted_response.replace("5. **", "\n\n5. **")
+        formatted_response = formatted_response.replace("6. **", "\n\n6. **")
+        formatted_response = formatted_response.replace("7. **", "\n\n7. **")
+        formatted_response = formatted_response.replace("8. **", "\n\n8. **")
+        
+        return formatted_response.strip()
+
+    except Exception as e:
+        return f"Error generating content idea: {str(e)}"
+
+

app.py

@@ -0,0 +1,222 @@
+"""
+# Main Application Module (Gradio Interface)
+
+This module provides the web interface and core functionality for the VidInsight AI application,
+integrating video fetching, transcription, summarization, and content idea generation.
+
+## Summary
+- Creates a Gradio web interface
+- Processes user topic input
+- Coordinates video fetching and transcription
+- Generates summaries and content ideas
+- Displays results in a formatted JSON output
+
+## Dependencies
+
+### System Requirements
+- Python 3.8+
+- Internet connection for API calls
+- FFmpeg for audio processing
+
+### Package Dependencies
+1. **gradio==3.50.2**
+   - Install: `pip install gradio`
+   - Purpose: Web interface creation
+
+2. **Other Project Packages**
+   - fetch_youtube_videos
+   - transcribe_videos
+   - summary
+   - YouTubeAgent
+
+### Project Dependencies
+1. **Local Modules**
+   - fetch_youtube_videos.py: For YouTube video retrieval
+   - transcribe_videos.py: For video transcription
+   - summary.py: For generating summaries
+   - YouTubeAgent.py: For content idea generation
+
+2. **Output Directory**
+   - 'output/' folder for saving transcriptions
+
+## Functions
+
+1. format_results(results)
+   - Formats view counts with commas
+   - Cleans transcript preview text
+   
+2. analyze(topic)
+   - Main processing function
+   - Coordinates all operations:
+     - Video fetching
+     - Transcription
+     - Summary generation
+     - Content idea creation
+
+## Returns
+JSON output containing:
+1. Video Information
+   - Title
+   - Channel
+   - Views
+   - Transcript preview
+   - File paths
+2. Analysis
+   - Topic title
+   - Summary
+   - Key points
+   - Content ideas
+
+## Error Handling
+- Empty topic validation
+- Video fetching errors
+- Transcription failures
+- Analysis generation issues
+
+"""
+
+
+import gradio as gr
+from fetch_youtube_videos import fetch_videos
+from transcribe_videos import transcribe_and_save
+from summary import generate_combined_summary_and_key_points
+from YouTubeAgent import generateidea
+from embeddings import mainApp
+
+def format_results(results):
+    """Format results for better display"""
+    if isinstance(results, list):
+        for result in results:
+            if 'Views' in result:
+                result['Views'] = f"{result['Views']:,}"  # Format numbers with commas
+            if 'Transcript Preview' in result:
+                result['Transcript Preview'] = result['Transcript Preview'].replace('\n', ' ')
+    return results
+
+def analyze(topic):
+    """
+    Fetch videos, transcribe them, and generate analysis including summaries and content ideas.
+    """
+    if not topic.strip():
+        return {"error": "⚠️ Please enter a topic to analyze"}
+    
+    try:
+        # Fetch videos based on topic
+        videos = fetch_videos(topic)
+        
+        if isinstance(videos, str):
+            return {"error": f"⚠️ {videos}"}
+        
+        if not videos:
+            return {"error": "⚠️ No relevant videos found for this topic."}
+        
+        results = []
+        transcriptions = []  # Store transcriptions for summary generation
+        
+        # Process each video
+        for video in videos:
+            transcription_result = transcribe_and_save(video['url'])
+            
+            if "error" in transcription_result:
+                results.append({
+                    'Video': video['title'],
+                    'Channel': video['channel'],
+                    'Views': video['views'],
+                    'Transcript Preview': transcription_result["error"]
+                })
+            else:
+                results.append({
+                    'Video': video['title'],
+                    'Channel': video['channel'],
+                    'Views': video['views'],
+                    'Transcript Preview': transcription_result["transcription"][:500] + "...",
+                    'Transcript File': transcription_result["file_path"]
+                })
+                # Add transcription for summary generation
+                transcriptions.append(transcription_result["transcription"])
+        
+        # Generate summary and content ideas if transcriptions exist
+        if transcriptions:
+            
+            mainApp(topic)
+            
+            topic_title, summary, key_points = generate_combined_summary_and_key_points(transcriptions)
+            
+            # Generate content idea
+            input_for_idea = {
+                "summary": summary,
+                "keypoints": key_points
+            }
+            content_idea = generateidea(input_for_idea)
+            
+            # Add analysis to results
+            results.append({
+                "Analysis": {
+                    "Topic Title": topic_title,
+                    "Summary": summary,
+                    "Key Points": key_points,
+                    "Content Idea": content_idea
+                }
+            })
+        
+        return format_results(results)
+
+    except Exception as e:
+        return {"error": f"⚠️ An unexpected error occurred: {str(e)}"}
+
+# Create Gradio interface with improved styling
+with gr.Blocks(theme=gr.themes.Soft()) as app:
+    gr.Markdown(
+        """
+        # 🎥 VidInsight AI
+        ### AI-Powered YouTube Content Analyzer
+        
+        This tool helps you:
+        - 📝 Get transcriptions of educational videos
+        - 📊 Generate summaries and key points
+        - 💡 Create content ideas
+        """
+    )
+    
+    with gr.Row():
+        with gr.Column(scale=2):
+            topic_input = gr.Textbox(
+                label="Enter Topic",
+                placeholder="e.g., Machine Learning, Data Science, Python Programming",
+                lines=2
+            )
+            
+        with gr.Column(scale=1):
+            submit_btn = gr.Button("🔍 Analyze", variant="primary")
+            clear_btn = gr.Button("🗑️ Clear")
+    
+    with gr.Row():
+        output = gr.JSON(
+            label="Analysis Results",
+            show_label=True
+        )
+    
+    # Add footer
+    gr.Markdown(
+        """
+        ---
+        📌 **Note**: This tool analyzes educational YouTube videos and generates AI-powered insights.
+        
+        Made by VidInsight Team 🤖
+        """
+    )
+    
+    # Set up button actions
+    submit_btn.click(
+        fn=analyze,
+        inputs=topic_input,
+        outputs=output,
+        api_name="analyze"
+    )
+    clear_btn.click(lambda: None, None, topic_input, queue=False)
+
+if __name__ == "__main__":
+    app.launch()
+
+
+

config.py

@@ -0,0 +1,88 @@
+"""
+# Configuration Module for VidInsight AI
+
+This module manages configuration settings and environment variables for the VidInsight AI project.
+
+## Summary
+- Loads API keys from environment file
+- Defines filtering criteria for YouTube video search
+- Configures trusted educational channels
+- Sets up keyword-based content filtering
+- Establishes quality thresholds (views, duration)
+
+## Dependencies
+
+### System Requirements
+- Python 3.8+
+- Read access to environment file location
+
+### Package Dependencies
+1. **python-dotenv**
+   - Install: `pip install python-dotenv`
+   - Purpose: Load environment variables from file
+
+### Project Dependencies
+1. **keys1.env file**
+   - Must contain:
+     - YOUTUBE_API_KEY
+   - Format:
+     ```
+     YOUTUBE_API_KEY=your_youtube_api_key_here
+     ```
+   - Location: Project root directory
+
+## Configuration Parameters
+
+### Video Search Settings
+- videoDuration: "medium" (4-20 minutes)
+- order: "relevance"
+- max_results: 10 videos per search
+- min_view_count: 10,000 views threshold
+
+### Content Filtering
+1. Trusted Channels (Whitelist):
+   - Khan Academy
+   - edX
+   - Coursera
+
+2. Keyword Filters:
+   - Teaching Keywords (Positive):
+     {tutorial, lesson, course, how-to, introduction, basics}
+   - Non-Teaching Keywords (Negative):
+     {fun, experiment, joke, prank, vlog}
+
+## Notes
+- Keep keys1.env secure and never commit to version control
+- Adjust filter criteria as needed for different use cases
+- Channel IDs must be exact matches for trusted channel filtering
+
+"""
+
+
+
+# dependencies
+from dotenv import load_dotenv, find_dotenv
+import os
+
+# Load environment variables from .env file
+load_dotenv(find_dotenv('keys1.env'))
+
+# YouTube API Configuration
+YOUTUBE_API_KEY = os.getenv("YOUTUBE_API_KEY")
+
+# Content Filter Settings
+FILTER_CONFIG = {
+    "videoDuration": "medium",  # Focus on videos between 4 and 20 minutes
+    "order": "relevance",       # Sort by relevance
+    # Trusted Channels: Only videos from these channels will bypass keyword filters
+    "trusted_channels": {
+        "Khan Academy": "UC4a-Gbdw7vOaccHmFo40b9g",
+        "edX": "UCEBb1b_L6zDS3xTUrIALZOw",
+        "Coursera": "UC58aowNEXHHnflR_5YTtP4g",
+    },
+    "teaching_keywords": {"tutorial", "lesson", "course", "how-to", "introduction", "basics"}, # Videos containing these words are prioritized
+    "non_teaching_keywords": {"fun", "experiment", "joke", "prank", "vlog"}, #Videos containing these words are deprioritized or ignored
+    # "blocked_keywords": {"fun", "experiment", "joke", "prank", "vlog"},
+    "max_results": 10,          # Limit search results to 10 videos
+    "min_view_count": 10000     # Minimum view count for relevance
+}

dbcone.py

@@ -0,0 +1,39 @@
+from pinecone import Pinecone, ServerlessSpec
+import time
+import os
+
+pc_database = None
+
+
+def getDatabase():
+    pine_cone_key = os.getenv("PINECONE_API_KEY")
+    
+    global pc_database 
+    
+    if pc_database is None:
+        pc_database = Pinecone(api_key = pine_cone_key)
+
+    return pc_database
+
+def getDatabaseIndex(index_name):
+    
+    local_db = getDatabase()
+    
+    if not local_db.has_index(index_name):
+        local_db.create_index(
+            name=index_name,
+            dimension=384, # Replace with your model dimensions
+            metric="cosine", # Replace with your model metric
+            spec=ServerlessSpec(
+                cloud="aws",
+                region="us-east-1"
+            ) 
+        )
+    
+    while not local_db.describe_index(index_name).status['ready']:
+        time.sleep(1)
+
+    index = local_db.Index(index_name)
+    return index
+
+

embeddings.py

@@ -0,0 +1,173 @@
+from sentence_transformers import SentenceTransformer
+from dotenv import load_dotenv, find_dotenv
+from dbcone import getDatabase
+from dbcone import getDatabaseIndex
+import os
+import uuid
+import pandas as pd
+import numpy as np
+from pathlib import Path
+from summary import generate_combined_summary_and_key_points
+
+sentence_model = None
+inputDir = None
+outputDir = None
+topic = None
+db_index_name = None
+db_namespace_name = None
+
+
+
+def initialize_model():
+    global sentence_model
+    sentence_model = SentenceTransformer('all-MiniLM-L6-v2')
+
+def get_model():
+    if sentence_model is None:
+        initialize_model()
+    return sentence_model
+
+def get_sentence_embedding(sentence):
+    model = get_model()
+    return model.encode(sentence)
+
+def getOutputDir(outputDirectory):
+    
+    outputDir = Path(outputDirectory)
+    
+    if not os.path.exists(outputDir):
+        os.makedirs(outputDir)
+    return outputDir
+    
+def read_files(inputDirectory, outputDirectory, topic=None):
+    
+    inputDir = Path(inputDirectory)
+    
+    embeded_lst = []
+    
+    if ( (not os.path.exists(inputDir)) or (not os.path.isdir(inputDir)) ):
+        return embeded_lst
+    
+    files = os.listdir(inputDir)
+    
+    if topic is None:
+        topic = os.path.basename(inputDir)
+
+    if len(files) <= 0:
+        return embeded_lst
+    
+    outputDir = getOutputDir(outputDirectory)
+    
+    for file in files:
+        if file.endswith(".txt"):
+            file_path = os.path.join(inputDir, file)
+            
+            if os.path.isfile(file_path):
+                
+                with open(file_path, 'r') as f:
+            
+                    text = f.read()
+                    embedding = get_sentence_embedding(text)
+                    f.close()
+
+                    if not os.path.isfile(os.path.join(outputDir, file)):
+                        os.rename(file_path, os.path.join(outputDir, file))
+                    else:
+                        os.remove(file_path)
+                        
+                    (topic_gen, summary, keypoints) = generate_combined_summary_and_key_points(text)
+                    
+                    if (topic_gen is not None): 
+                        topic += " - " + topic_gen
+                    
+                    embeded_lst.append(
+                        {
+                            "id" : str(uuid.uuid4().hex),
+                            "metadata": {
+                                'text':text,
+                                "topic": topic,
+                                "summary": summary,
+                                "keypoints":keypoints
+                                },
+                            "values": embedding.tolist()
+                        }
+                    )
+                        
+    return  embeded_lst
+
+def save_to_database(embeded_lst, index_name = 'test_videos' ,namespace="sample-namespace"):
+    
+    if len(embeded_lst) > 0 :
+        db_index = getDatabaseIndex(index_name)
+        
+        db_index.upsert(
+            vectors=embeded_lst,
+            namespace=namespace
+        )    
+
+
+def embed_text_files(inputDir, outputDir, topic):
+    
+    return read_files(inputDirectory=inputDir, outputDirectory=outputDir, topic=topic)
+
+def configureApp(given_topic):
+    
+    global inputDir, outputDir, topic, db_index_name, db_namespace_name
+    
+    currPath = Path.cwd()
+    
+    inputDir = os.path.join( currPath, 'output')
+    outputDir = os.path.join(currPath, 'processed')
+    
+    topic = given_topic
+    db_index_name = 'samplevideos'
+    db_namespace_name="video-namespace"
+        
+    load_dotenv(find_dotenv('Keys1.env'))
+    initialize_model()
+    getDatabase()
+    
+    return True
+
+def fetch_from_database(search_text, topics =[] ,top_k = 5, index_name = 'test-videos' ,namespace="sample-namespace"):
+    
+    db_index = getDatabaseIndex(index_name)
+    
+    results = db_index.query(namespace=namespace,
+        vector=np.array(get_sentence_embedding(search_text)).tolist(),
+        top_k=top_k,
+        include_values=True,
+        include_metadata=True,    
+        filter={
+            "topic": {"$in": topics},
+        }                    
+    )
+    
+    return results
+
+def captureData():
+    
+    global inputDir, outputDir, topic, db_index_name, db_namespace_name
+
+    embeded_lst = embed_text_files(inputDir, outputDir, topic)
+    
+    save_to_database(embeded_lst, index_name =db_index_name, namespace=db_namespace_name)
+    
+
+def queryRepository(search_text, topic):
+    
+    global db_index_name, db_namespace_name
+    
+    result = fetch_from_database(search_text, topics=[topic], index_name = db_index_name, namespace=db_namespace_name)
+
+    print(f'Results: {result}')
+    
+
+def mainApp(topic):
+    
+    configureApp(topic)
+    captureData()
+    
+
+if __name__ == "__main__":
+    mainApp()

fetch_youtube_videos.py

@@ -0,0 +1,115 @@
+"""
+# YouTube Video Fetcher Module
+
+This module is responsible for searching, filtering, and retrieving educational YouTube videos based on user queries.
+
+## Summary
+- Fetches videos using YouTube Data API v3
+- Filters videos based on:
+  - Duration (medium length: 4-20 minutes)
+  - View count (minimum threshold)
+  - Teaching vs non-teaching keywords
+  - Trusted educational channels
+- Returns top 3 most relevant videos sorted by view count
+
+## Dependencies
+
+### System Requirements
+- Python 3.8+
+- Internet connection for API calls
+
+### Package Dependencies
+- google-api-python-client==2.104.0
+  Install: `pip install google-api-python-client`
+
+### Project Dependencies
+1. config.py
+   - Provides YOUTUBE_API_KEY
+   - Contains FILTER_CONFIG dictionary with:
+     - videoDuration
+     - order
+     - trusted_channels
+     - teaching_keywords
+     - non_teaching_keywords
+     - max_results
+     - min_view_count
+
+2. Environment Setup
+   - keys1.env file with YouTube API key
+   - YouTube Data API access enabled in Google Cloud Console
+
+## Returns
+- List of dictionaries containing:
+  - title: Video title
+  - url: YouTube video URL
+  - channel: Channel name
+  - views: View count
+- Or error message string if fetch fails
+
+## Error Handling
+- Returns error message if:
+  - API key is invalid
+  - API quota is exceeded
+  - Network connection fails
+  - YouTube API request fails
+"""
+
+# Import Dependencies
+from googleapiclient.discovery import build
+from config import YOUTUBE_API_KEY, FILTER_CONFIG
+
+def fetch_videos(topic):
+    """Fetch relevant YouTube videos based on topic and filter criteria."""
+    try:
+        youtube = build('youtube', 'v3', developerKey=YOUTUBE_API_KEY)
+        
+        # Fetch videos from YouTube API
+        search_response = youtube.search().list(
+            q=topic,
+            part="snippet",
+            type="video",
+            maxResults=FILTER_CONFIG["max_results"],  # Limit to max_results directly
+            videoDuration=FILTER_CONFIG["videoDuration"],
+            order=FILTER_CONFIG["order"]
+        ).execute()
+
+        # Process video results
+        videos = []
+        for item in search_response.get('items', []):
+            video_id = item['id']['videoId']
+            title = item['snippet']['title'].lower()
+            description = item['snippet']['description'].lower()
+            channel_id = item['snippet']['channelId']
+            
+            # Fetch video statistics (views)
+            stats_response = youtube.videos().list(
+                part="statistics",
+                id=video_id
+            ).execute()
+            
+            stats = stats_response.get('items', [{}])[0].get('statistics', {})
+            view_count = int(stats.get("viewCount", 0))
+            
+            # Apply filters: minimum views, keywords, trusted channels
+            if view_count < FILTER_CONFIG["min_view_count"]:
+                continue
+            
+            teaching_score = len(set(title.split() + description.split()) & FILTER_CONFIG["teaching_keywords"])
+            noise_score = len(set(title.split() + description.split()) & FILTER_CONFIG["non_teaching_keywords"])
+            # noise_score = len(set(title.split() + description.split()) & FILTER_CONFIG["blocked_keywords"])
+            
+            is_trusted_channel = channel_id in FILTER_CONFIG["trusted_channels"].values()
+            
+            if teaching_score > noise_score or is_trusted_channel:
+                videos.append({
+                    'title': item['snippet']['title'],
+                    'url': f'https://youtu.be/{video_id}',
+                    'channel': item['snippet']['channelTitle'],
+                    'views': view_count,
+                })
+
+        # Sort by views (descending) and return top 3 videos
+        return sorted(videos, key=lambda x: x['views'], reverse=True)[:3]
+
+    except Exception as e:
+        return f"Error fetching videos: {str(e)}"

main.py

@@ -0,0 +1,73 @@
+"""
+# Command Line Interface Module
+
+This module provides a CLI alternative to the Gradio web interface for the VidInsight AI application.
+
+## Summary
+- Offers command-line interaction for video analysis
+- Provides sequential processing of videos
+- Displays results directly in the terminal
+- Serves as a debugging and testing tool
+
+## Dependencies
+
+### System Requirements
+- Python 3.8+
+- Internet connection for API calls
+- FFmpeg for audio processing
+
+### Package Dependencies
+No additional package installations required beyond project dependencies
+
+### Project Dependencies
+1. **Local Modules**
+   - fetch_youtube_videos.py: For YouTube video retrieval
+   - transcribe_videos.py: For video transcription
+
+## Functions
+main()
+- Gets user input for topic
+- Coordinates video fetching and transcription
+- Displays results in terminal format
+
+## Usage Example
+python main.py
+Enter topic to analyze: Machine Learning
+
+## Returns
+Terminal output containing:
+1. Video Information
+   - Title
+   - URL
+2. Transcription Status
+   - Success/failure messages
+   - Transcription text or error
+
+## Error Handling
+- Video fetching errors
+- Transcription failures
+- Invalid input handling
+
+"""
+
+
+from fetch_youtube_videos import fetch_videos
+from transcribe_videos import transcribe
+
+def main():
+    topic = input("Enter topic to analyze: ")
+    print("\nFetching videos...")
+    
+    videos = fetch_videos(topic)
+    if isinstance(videos, str):
+        print(f"Error: {videos}")
+        return
+    
+    for idx, video in enumerate(videos, 1):
+        print(f"\nVideo {idx}: {video['title']}")
+        print(f"URL: {video['url']}")
+        print("Transcribing...")
+        print(transcribe(video['url']))
+
+if __name__ == "__main__":
+    main()

summary.py

@@ -0,0 +1,122 @@
+"""
+# Video Summary Generator Module
+
+This module processes transcribed video content to generate summaries, key points, and topic titles 
+using Google's Gemini AI model.
+
+## Summary
+- Takes multiple video transcriptions as input
+- Concatenates transcriptions for unified analysis
+- Uses Gemini AI to generate:
+  - Relevant topic title
+  - Concise content summary
+  - Key points from the content
+- Handles response parsing and error cases
+
+## Dependencies
+
+### System Requirements
+- Python 3.8+
+- Internet connection for API calls
+
+### Package Dependencies
+1. **langchain-google-genai**
+   - Install: `pip install langchain-google-genai`
+   - Purpose: Interface with Gemini AI model
+
+2. **python-dotenv**
+   - Install: `pip install python-dotenv`
+   - Purpose: Load environment variables
+
+### Project Dependencies
+1. **keys1.env file**
+   - Must contain: GEMINI_API_KEY
+   - Format: GEMINI_API_KEY=your_api_key_here
+
+2. **Input Requirements**
+   - Transcription texts from processed videos
+   - Non-empty transcription content
+
+## Functions
+generate_combined_summary_and_key_points(transcriptions)
+- Args: List of transcription texts
+- Returns: Tuple of (topic_title, summary, key_points)
+- Error Returns: Error messages with empty lists if processing fails
+
+## Returns
+Tuple containing:
+1. topic_title (str): Generated title for the content
+2. summary (str): Concise summary of all transcriptions
+3. key_points (list): List of main points extracted
+
+## Error Handling
+- Returns error messages if:
+  - Transcriptions are empty
+  - Gemini API fails to respond
+  - Response parsing fails
+"""
+
+import os
+import glob
+from dotenv import load_dotenv, find_dotenv
+from langchain_google_genai import ChatGoogleGenerativeAI
+
+def generate_combined_summary_and_key_points(transcriptions):
+    if not all(transcriptions):
+        return "Error: No transcription text provided.", [], ""
+
+    # Concatenate the transcriptions into one single string
+    concatenated_transcriptions = "\n".join(transcriptions)
+
+    prompt = f"""
+    The following are transcriptions of videos:
+    ---
+    {concatenated_transcriptions}
+    ---
+    Based on the content, generate a relevant topic title for the transcriptions. 
+    Then, summarize the key insights and extract the main points from these transcriptions together. 
+    Ignore sponsors and focus more on the details rather than the overall outline.
+    Format your response as:
+    Topic Title: [Generated topic title]
+    
+    Summary:
+    [Concise summary of the transcriptions]
+    
+    Key Points:
+    - [Key point 1]
+    - [Key point 2]
+    - [Key point 3]
+    """
+    # Load environment variables
+    load_dotenv(find_dotenv('keys1.env'))
+
+    # Get API key
+    GEMINI_API_KEY = os.getenv("GEMINI_API_KEY")
+    GEMINI_MODEL = "gemini-1.5-flash"
+
+    # Initialize Gemini API
+    llm = ChatGoogleGenerativeAI(model=GEMINI_MODEL, api_key=GEMINI_API_KEY)
+
+    # Generate the response from the model
+    response = llm.predict(prompt)
+    
+    if not response:
+        return "Error: No response generated.", [], ""
+
+    # Extract topic title, summary, and key points from response
+    topic_title_start = response.find("Topic Title:")
+    summary_start = response.find("Summary:")
+    key_points_start = response.find("Key Points:")
+
+    if topic_title_start != -1 and summary_start != -1 and key_points_start != -1:
+        topic_title = response[topic_title_start + len("Topic Title:"): summary_start].strip()
+        summary = response[summary_start + len("Summary:"): key_points_start].strip()
+        key_points_str = response[key_points_start + len("Key Points:"):].strip()
+        key_points = [point.strip(" -") for point in key_points_str.split("\n")]
+    else:
+        topic_title = "Error: Unable to generate topic title."
+        summary = "Error: Unable to extract summary."
+        key_points = []
+
+    return topic_title, summary, key_points
+

transcribe_videos.py

@@ -0,0 +1,124 @@
+"""
+# Video Transcription Module
+
+This module handles the audio extraction and transcription of YouTube videos using Whisper AI.
+
+## Summary
+- Downloads audio from YouTube videos using yt-dlp
+- Transcribes audio using OpenAI's Whisper model
+- Saves transcriptions as text files
+- Handles various YouTube URL formats
+- Provides error handling for failed downloads/transcriptions
+
+## Dependencies
+
+### System Requirements
+1. **FFmpeg**
+   - Windows: Install via chocolatey `choco install ffmpeg`
+   - Mac: Install via homebrew `brew install ffmpeg`
+   - Linux: `sudo apt-get install ffmpeg`
+2. Python 3.8+
+3. Sufficient disk space for temporary audio files
+
+### Package Dependencies
+1. **openai-whisper==20231106**
+   - Install: `pip install openai-whisper`
+   - Purpose: Audio transcription
+   
+2. **yt-dlp==2023.11.16**
+   - Install: `pip install yt-dlp`
+   - Purpose: YouTube audio downloading
+   
+3. **torch**
+   - Install: `pip install torch`
+   - Purpose: Required by Whisper for model operations
+
+### Project Dependencies
+1. **output/** directory
+   - Must exist or have permissions to create
+   - Stores transcription text files
+
+## Functions
+1. extract_video_id(url)
+   - Extracts YouTube video ID from various URL formats
+   - Handles both youtube.com and youtu.be URLs
+   
+2. transcribe_and_save(url, output_dir="output")
+   - Downloads audio
+   - Performs transcription
+   - Saves result to file
+   - Returns file path and transcription text
+
+## Returns
+Dictionary containing:
+- file_path: Path to saved transcription
+- transcription: Full transcription text
+- error: Error message if transcription fails
+
+## Error Handling
+- Returns error dictionary if:
+  - Video URL is invalid
+  - Audio download fails
+  - Transcription fails
+  - File writing fails
+"""
+
+
+# import dependencies
+import whisper
+import yt_dlp
+import os
+
+# Load Whisper model
+MODEL = whisper.load_model("base")
+# MODEL = whisper.load_model("base", weights_only=True)
+
+def extract_video_id(url):
+    """
+    Extracts the video ID from a YouTube URL.
+    Args:
+        url (str): YouTube video URL.
+    Returns:
+        str: Video ID.
+    """
+    if "v=" in url:
+        return url.split("v=")[-1]
+    elif "youtu.be/" in url:
+        return url.split("youtu.be/")[-1]
+    return "unknown_video_id"
+
+def transcribe_and_save(url, output_dir="output"):
+    """
+    Transcribe audio from a YouTube video and save it to a file.
+    Args:
+        url (str): YouTube video URL.
+        output_dir (str): Directory to save the transcription.
+    Returns:
+        dict: Contains the file path and transcription text.
+    """
+    try:
+        # Download audio with yt-dlp
+        with yt_dlp.YoutubeDL({'format': 'bestaudio'}) as ydl:
+            info = ydl.extract_info(url, download=False)
+            audio_url = info['url']
+        
+        # Transcribe audio
+        result = MODEL.transcribe(audio_url)
+        transcription = result['text']
+
+        # Create output directory if it doesn't exist
+        os.makedirs(output_dir, exist_ok=True)
+
+        # Use video ID as file name
+        video_id = extract_video_id(url)
+        file_path = os.path.join(output_dir, f"{video_id}.txt")
+        
+        # Save transcription to a file
+        with open(file_path, "w", encoding="utf-8") as file:
+            file.write(transcription)
+
+        return {"file_path": file_path, "transcription": transcription}
+
+    except Exception as e:
+        return {"error": f"Transcription failed: {str(e)}"}
+