Upload 19 files

SOURCE_DOCUMENTS/confluence.txt

@@ -0,0 +1 @@
+India's Income Tax Laws are framed by the Government The Government imposes a tax on taxable income of all persons who are individuals, Hindu Undivided Families (HUF's), companies, firms, LLP, association of persons, body of individuals, local authority and any other artificial juridical person. According to these laws, levy of tax on a person depends upon his residential status. Every individual who qualifies as a resident of India is required to pay tax on his or her global income. Every financial year, taxpayers have to follow certain rules while filing their Income Tax Returns (ITRs).Income Tax Return - What is it?An Income tax return (ITR) is a form used to file information about your income and tax to the Income Tax Department. The tax liability of a taxpayer is calculated based on his or her income. In case the return shows that excess tax has been paid during a year, then the individual will be eligible to receive a income tax refund from the Income Tax Department.As per the income tax laws, the return must be filed every year by an individual or business that earns any income during a financial year. The income could be in the form of a salary, business profits, income from house property or earned through dividends, capital gains, interests or other sources.Tax returns have to be filed by an individual or a business before a specified date. If a taxpayer fails to abide by the deadline, he or she has to pay a penalty. Is it mandatory to file Income Tax Return?As per the tax laws laid down in India, it is compulsory to file your income tax returns if your income is more than the basic exemption limit. The income tax rate is pre-decided for taxpayers. A delay in filing returns will not only attract late filing fees but also hamper your chances of getting a loan or a visa for travel purposes.Who should file Income Tax Returns?According to the Income Tax Act, income tax has to be paid only by individuals or businesses who fall within certain income brackets. 

SOURCE_DOCUMENTS/transcript.txt

@@ -0,0 +1 @@
+In this video, I'm going to answer the top 3 questions my students ask me about Python. What is Python? What  can you do with it? And why is it so popular? In other words, what does it do that other programming languages don't? Python is the  world's fastest growing and most popular programming language, not just  amongst software engineers, but also amongst mathematicians,  data analysts, scientists, accountants, networking engineers, and even kids! Because it's a very beginner friendly programming  language. So people from different disciplines use Python for a variety of different tasks, such as data analysis and visualization,  artificial intelligence and machine learning, automation  in fact this is one of the big uses of Python amongst people who are not software developers. If you constantly have to do boring, repetitive  tasks, such as copying files and folders around, renaming them,  uploading them to a server, you can easily write a Python script to automate all that and save your time. And that's just one example, if you continuously have to work with excel spreadsheets, PDF's, CS View files, download websites and parse them, you can automate all that stuff with Python. So you don't have to be a software developer to use Python. You could be an accountant, a mathematician, or a scientist, and use Python  to make your life easier. You can also use Python to build  web, mobile and desktop applications as well as software  testing or even hacking. So Python is a multi purpose language.  Now if you have some programming experience you may say, "But Mosh we can do all this stuff with other programming languages, so what's the big deal  about Python?" Here are a few reasons. With Python you can  solve complex problems in less time with fewer lines of code.  Here's an example. Let's say we want to extract the first three  letters of the text Hello World. This is the code we have to write  in C# this is how we do it in JavaScript and here's how we  do it in Python. See how short and clean the language is? And that's just the beginning. Python makes a lot of trivial things really easy with a simple yet powerful syntax. Here are a few other reasons Python is so popular. It's a high level language so you don't have to worry about complex tasks such as memory management,  like you do in C++. It's cross platform which means  you can build and run Python applications on Windows, Mac,  and Linux. It has a huge community so whenever you get  stuck, there is someone out there to help. It has a large ecosystem  of libraries, frameworks and tools which means whatever you wanna do it is likely that someone else has done it before because Python has been around  for over 20 years. So in a nutshell, Python is a multi-purpose language with a simple, clean, and beginner-friendly  syntax. All of that means Python is awesome. Technically everything you do with Python you can do with other programming languages,  but Python's simplicity and elegance has made it grow way  more than other programming languages. That's why it's the number onne language employers are looking for. So whether you're a programmer or  an absolute beginner, learning Python opens up lots of job opportunities  to you. In fact, the average Python developer earns a whopping 116,000 dollars a year. If you found this video helpful, please support my hard work by liking and sharing it with others.  Also, be sure to subscribe to my channel, because I have a couple of awesome Python tutorials for you, you're going to see them on the screen now.  Here's my Python tutorial for beginners, it's a great starting point if you  have limited or no programming experience. On the other hand, if you  do have some programming experience and want to quickly get up to speed with Python,  I have another tutorial just for you. I'm not going to waste your time  telling you what a variable or a function is. I will talk to you like a programmer. There's never been a better time to master Python programming, so click on the tutorial that is right for you and get started. Thank you for watching!

app.py

@@ -0,0 +1,216 @@
+# Import required libraries
+import streamlit as st
+import youtube
+import confluence
+import modJira
+import time
+import similarity
+import ingest
+
+# global transcript_result
+# transcript_result = ""
+
+# Set page configuration and title for Streamlit
+st.set_page_config(page_title="AI-Seeker", page_icon="📼", layout="wide")
+
+# Add header with title and description
+st.markdown(
+    '<p style="display:inline-block;font-size:40px;font-weight:bold;">AI-Seeker</p> <p style="display:inline-block;font-size:16px;">AI-Seeker is a web-app tool that utilizes APIs to extract text content from YouTube, Confluence and Jira. It incorporates Llama-2-7B-Chat-GGML model with Langchain to provide users with a summary and query-based smart response depending on the content of the media source.<br><br></p>',
+    unsafe_allow_html=True
+)
+
+txtInputBox = "YouTube"
+
+
+with st.sidebar.title("Configuration"):
+    usecase = st.sidebar.selectbox("Select Media Type:",("YouTube", "Confluence", "Jira"))
+    if usecase == "YouTube":
+        txtInputBox = "Enter ID of YouTube Video"
+        default_value = "Y8Tko2YC5hA"
+    elif usecase == "Confluence":
+        txtInputBox = "Enter ID of your Confluence Page"
+        default_value = "393217"
+    elif usecase == "Jira":
+        txtInputBox = "Enter the name of your JIRA Project"
+        default_value = "jira_test"
+
+    video_id = st.sidebar.text_input(txtInputBox,value=default_value)
+
+    strTranscript = ""
+    training_status = "yet_to_start"
+    btnTranscript = st.sidebar.button("Transcript")
+    btnSummary = st.sidebar.button("Summary")
+    btnTrain = st.sidebar.button("Train")
+    if btnTrain:
+        with st.spinner("Training in Progress..."):
+            ingest.main()
+
+    query = st.sidebar.text_input('Enter your question below:', value="What is Python?")
+    btnAsk = st.sidebar.button("Query")
+
+    btnClear = st.sidebar.button("Clear Data")
+    if btnClear:
+        st.session_state.clear()
+
+def fnJira():
+    st.info("Transcription")
+
+    if btnTranscript:
+
+        if 'transcript_result' not in st.session_state:
+            st.session_state['transcript_result'] = modJira.get_details(video_id)
+            transcript_result = st.session_state['transcript_result']
+            st.dataframe(transcript_result)
+    else:
+        if 'transcript_result' in st.session_state:
+            transcript_result = st.session_state['transcript_result']
+            st.dataframe(transcript_result)
+
+    st.info("Query")
+        
+    if btnAsk:
+        with st.spinner(text="Retrieving..."):
+            if 'transcript_answer' not in st.session_state:
+                answer = modJira.ask_question(query)
+                st.session_state['transcript_answer'] = answer
+                #st.success(answer)
+            if 'transcript_answer' in st.session_state:
+                answer = st.session_state['transcript_answer']
+
+            st.success(answer)
+
+    else:
+        if 'transcript_answer' in st.session_state:
+            answer = st.session_state['transcript_answer']
+
+            st.success(answer)
+
+
+def fnConfluence():
+    st.info("Transcription")
+
+    if btnTranscript:
+
+        if 'transcript_result' not in st.session_state:
+            st.session_state['transcript_result'] = confluence.transcript(video_id)
+        transcript_result = st.session_state['transcript_result']
+        st.markdown(f"<div style='height: 100px; overflow-y: scroll;'>{transcript_result}</div>", unsafe_allow_html=True)
+    else:
+        if 'transcript_result' in st.session_state:
+            transcript_result = st.session_state['transcript_result']
+            st.markdown(f"<div style='height: 100px; overflow-y: scroll;'>{transcript_result}</div>", unsafe_allow_html=True)
+
+    col1, col2 = st.columns([1, 1])
+
+    with col1:
+        # with col12:
+        st.info("Summary")
+        if btnSummary:
+            if 'transcript_summary' not in st.session_state:
+                with st.spinner(text="Retrieving..."):
+                    st.session_state['transcript_summary'] = confluence.summarize()
+                    summary = st.session_state['transcript_summary'] 
+                    st.success(summary)
+        else:
+            if 'transcript_summary' in st.session_state:
+                summary = st.session_state['transcript_summary']
+                st.success(summary)
+
+    with col2:
+        st.info("Query")
+        
+        if btnAsk:
+            with st.spinner(text="Retrieving..."):
+                if 'transcript_answer' not in st.session_state:
+                    answer = confluence.ask_question(query)
+                    st.session_state['transcript_answer'] = answer
+                    #st.success(answer)
+                if 'transcript_answer' in st.session_state:
+                    answer = st.session_state['transcript_answer']
+
+                st.success(answer)
+
+        else:
+            if 'transcript_answer' in st.session_state:
+                answer = st.session_state['transcript_answer']
+
+                st.success(answer)
+
+def fnYoutube():
+    st.info("Transcription")
+
+    if btnTranscript:
+
+        if 'transcript_result' not in st.session_state:
+            st.session_state['transcript_result'] = youtube.audio_to_transcript(video_id)
+        transcript_result = st.session_state['transcript_result']
+        st.markdown(f"<div style='height: 100px; overflow-y: scroll;'>{transcript_result}</div>", unsafe_allow_html=True)
+    else:
+        if 'transcript_result' in st.session_state:
+            transcript_result = st.session_state['transcript_result']
+            st.markdown(f"<div style='height: 100px; overflow-y: scroll;'>{transcript_result}</div>", unsafe_allow_html=True)
+
+    col1, col2 = st.columns([1, 1])
+
+    with col1:
+        # with col12:
+        st.info("Summary")
+        if btnSummary:
+            if 'transcript_summary' not in st.session_state:
+                with st.spinner(text="Retrieving..."):
+                    st.session_state['transcript_summary'] = youtube.summarize()
+                    summary = st.session_state['transcript_summary'] 
+                    st.success(summary)
+        else:
+            if 'transcript_summary' in st.session_state:
+                summary = st.session_state['transcript_summary']
+                st.success(summary)
+
+    with col2:
+        st.info("Query")
+        
+        if btnAsk:
+            with st.spinner(text="Retrieving..."):
+                if 'transcript_answer' not in st.session_state:
+                    answer = youtube.ask_question(query)
+                    st.session_state['transcript_answer'] = answer
+                    #st.success(answer)
+                if 'transcript_answer' in st.session_state:
+                    answer = st.session_state['transcript_answer']
+
+                st.success(answer)
+
+                transcript_start_time, transcript_end_time = similarity.similarity(strQuery=answer)
+
+                st.video(f"https://www.youtube.com/embed/{video_id}", format="video/mp4", start_time=int(transcript_start_time))
+
+        else:
+            if 'transcript_answer' in st.session_state:
+                answer = st.session_state['transcript_answer']
+
+                st.success(answer)
+
+                transcript_start_time, transcript_end_time = similarity.similarity(strQuery=answer)
+
+                st.video(f"https://www.youtube.com/embed/{video_id}", format="video/mp4", start_time=int(transcript_start_time))
+
+if usecase == "YouTube":
+    fnYoutube()
+elif usecase == "Confluence":
+    fnConfluence()
+elif usecase == "Jira":
+    fnJira()
+
+# Hide Streamlit header, footer, and menu
+hide_st_style = """
+    <style>
+    #MainMenu {visibility: hidden;}
+    footer {visibility: hidden;}
+    header {visibility: hidden;}
+    </style>
+"""
+#"""footer {visibility: hidden;}
+#    header {visibility: hidden;}"""
+
+# Apply CSS code to hide header, footer, and menu
+st.markdown(hide_st_style, unsafe_allow_html=True)

chunks.json

@@ -0,0 +1 @@
+[{"text": "In this video, I'm going to answer the top 3 questions my students ask me about Python. What is Python? What  can you do with it? And why is it so popular? In other words, what does it do that other programming languages don't? Python is the ", "start": 0.0, "end": 16.0}, {"text": "world's fastest growing and most popular programming language, not just  amongst software engineers, but also amongst mathematicians,  data analysts, scientists, accountants, networking engineers, and even kids! Because it's a very beginner friendly programming ", "start": 16.0, "end": 32.0}, {"text": "language. So people from different disciplines use Python for a variety of different tasks, such as data analysis and visualization,  artificial intelligence and machine learning, automation  in fact this is one of the big uses of Python amongst people who are not software", "start": 32.0, "end": 48.0}, {"text": "developers. If you constantly have to do boring, repetitive  tasks, such as copying files and folders around, renaming them,  uploading them to a server, you can easily write a Python script to automate all that and save your time. And that's just one example, if you", "start": 48.0, "end": 64.0}, {"text": "continuously have to work with excel spreadsheets, PDF's, CS View files, download websites and parse them, you can automate all that stuff with Python. So you don't have to be a software developer to use Python. You could be an accountant, a mathematician, or a scientist, and use Python ", "start": 64.0, "end": 80.0}, {"text": "to make your life easier. You can also use Python to build  web, mobile and desktop applications as well as software  testing or even hacking. So Python is a multi purpose language.  Now if you have some programming experience you may say, \"But Mosh", "start": 80.0, "end": 96.0}, {"text": "we can do all this stuff with other programming languages, so what's the big deal  about Python?\" Here are a few reasons. With Python you can  solve complex problems in less time with fewer lines of code.  Here's an example. Let's say we want to extract the first three ", "start": 96.0, "end": 112.0}, {"text": "letters of the text Hello World. This is the code we have to write  in C# this is how we do it in JavaScript and here's how we  do it in Python. See how short and clean the language is? And that's just the beginning. Python makes a lot of trivial things", "start": 112.0, "end": 128.0}, {"text": "really easy with a simple yet powerful syntax. Here are a few other reasons Python is so popular. It's a high level language so you don't have to worry about complex tasks such as memory management,  like you do in C++. It's cross platform which means ", "start": 128.0, "end": 144.0}, {"text": "you can build and run Python applications on Windows, Mac,  and Linux. It has a huge community so whenever you get  stuck, there is someone out there to help. It has a large ecosystem  of libraries, frameworks and tools which means whatever you wanna do", "start": 144.0, "end": 160.0}, {"text": "it is likely that someone else has done it before because Python has been around  for over 20 years. So in a nutshell, Python is a multi-purpose language with a simple, clean, and beginner-friendly  syntax. All of that means Python is awesome.", "start": 160.0, "end": 176.0}, {"text": "Technically everything you do with Python you can do with other programming languages,  but Python's simplicity and elegance has made it grow way  more than other programming languages. That's why it's the number onne language employers are looking for. So whether you're a programmer or ", "start": 176.0, "end": 192.0}, {"text": "an absolute beginner, learning Python opens up lots of job opportunities  to you. In fact, the average Python developer earns a whopping 116,000 dollars a year. If you found this video helpful, please support my hard work by liking and sharing it with others. ", "start": 192.0, "end": 208.0}, {"text": "Also, be sure to subscribe to my channel, because I have a couple of awesome Python tutorials for you, you're going to see them on the screen now.  Here's my Python tutorial for beginners, it's a great starting point if you  have limited or no programming experience. On the other hand, if you ", "start": 208.0, "end": 224.0}, {"text": "do have some programming experience and want to quickly get up to speed with Python,  I have another tutorial just for you. I'm not going to waste your time  telling you what a variable or a function is. I will talk to you like a programmer. There's never been a better time to master Python programming,", "start": 224.0, "end": 240.0}, {"text": "so click on the tutorial that is right for you and get started. Thank you for watching!", "start": 240.0, "end": 246.63}]

chunks_create.py

@@ -0,0 +1,44 @@
+import json
+
+def combine_and_calculate(input_file_path, output_file_path):
+    with open(input_file_path, 'r') as file:
+        output_data = json.load(file)
+
+    combined_json_list = []
+
+    # Calculate the number of groups to create
+    num_groups = (len(output_data) + 7) // 8
+
+    for group_num in range(num_groups):
+        # Calculate the starting index and ending index for the current group
+        start_index = group_num * 8
+        end_index = min(start_index + 8, len(output_data))
+
+        # Extract the "text" values from the current group of dictionaries
+        combined_text = " ".join([item["text"] for item in output_data[start_index:end_index]])
+
+        # Calculate the "start" and "end" for the current group
+        group_start = output_data[start_index]["start"]
+        group_end = output_data[end_index - 1]["end"]
+
+        # Create the combined JSON for the current group
+        combined_json = {
+            "text": combined_text,
+            "start": group_start,
+            "end": group_end,
+        }
+
+        combined_json_list.append(combined_json)
+
+    # Save the combined JSON list to a new file
+    with open(output_file_path, 'w') as output_file:
+        json.dump(combined_json_list, output_file)
+
+# Replace 'output_file.json' with the path to the output JSON file you created previously
+input_file_path = '/home/bharathi/langchain_experiments/GenAI/transcript_end.json'
+
+# Replace 'combined_output_file.json' with the desired path and filename for the combined JSON file
+output_file_path = '/home/bharathi/langchain_experiments/GenAI/chunks.json'
+
+# Call the function to create the combined JSON and save it to a new file
+combine_and_calculate(input_file_path, output_file_path)

confluence.py

@@ -0,0 +1,75 @@
+from bs4 import BeautifulSoup
+import requests
+from requests.auth import HTTPBasicAuth
+import run_localGPT
+
+def start_training():
+    training_status = ingest.main()
+    return training_status
+
+def replace_substring_and_following(input_string, substring):
+    index = input_string.find(substring)
+    if index != -1:
+        return input_string[:index]
+    else:
+        return input_string
+
+def ask_question(strQuestion):
+    answer = run_localGPT.main(device_type='cpu', strQuery=strQuestion)
+    answer_cleaned = replace_substring_and_following(answer, "Unhelpful Answer")
+    return answer_cleaned
+
+def transcript(page_id):
+
+    url = f"https://srikanthnm.atlassian.net/wiki/rest/api/content/{page_id}?expand=body.storage"  # Replace with the actual URL you want to access
+    username = "srikanth.nm@gmail.com"
+    password = "ATATT3xFfGF09rugcjiT06v8xMayt5ggayMNiwz4b6w07PWQxPvpi4fMDzwwHxKt-v8dGx49uiulIMKHUUYroeS8cXvMKYfi7sQnFsYNfGslPVqSq1BQrzPhTio-xmYOHcit5ijzU9cSGGa7eLXUMxQTsSQjLhtZ-EQPI8h6aki690_-evLFZmU=3910FFD4"
+
+
+    response = requests.get(url, auth=HTTPBasicAuth(username, password))
+
+    # Check if the request was successful (status code 200)
+    if response.status_code == 200:
+        # Process the response data (if applicable)
+        data = response.json()
+    else:
+        data = f"Error: Unable to access the URL. Status code: {response.status_code}"
+
+    soup = BeautifulSoup(data['body']['storage']['value'],"html.parser")
+
+    page_content = soup.get_text()
+    page_content_cleaned = page_content.replace('\xa0',' ')
+    page_content_cleaned
+
+    with open('SOURCE_DOCUMENTS/confluence.txt', 'w') as outfile:
+        outfile.write(page_content_cleaned[:1998])
+
+    return page_content_cleaned[:1998]
+
+def summarize():
+    from langchain import PromptTemplate, LLMChain
+    from langchain.text_splitter import CharacterTextSplitter
+    from langchain.chains.mapreduce import MapReduceChain
+    from langchain.prompts import PromptTemplate
+
+    model_id = "TheBloke/Llama-2-7B-Chat-GGML"
+    model_basename = "llama-2-7b-chat.ggmlv3.q4_0.bin"
+
+    llm = run_localGPT.load_model(device_type='cpu', model_id=model_id, model_basename=model_basename)
+
+    text_splitter = CharacterTextSplitter()
+
+    with open("SOURCE_DOCUMENTS/confluence.txt") as f:
+        file_content = f.read()
+    texts = text_splitter.split_text(file_content)
+
+    from langchain.docstore.document import Document
+
+    docs = [Document(page_content=t) for t in texts]
+
+    from langchain.chains.summarize import load_summarize_chain
+
+    chain = load_summarize_chain(llm, chain_type="map_reduce")
+    summary = chain.run(docs)
+
+    return summary

constants.py

@@ -0,0 +1,42 @@
+import os
+
+# from dotenv import load_dotenv
+from chromadb.config import Settings
+
+# https://python.langchain.com/en/latest/modules/indexes/document_loaders/examples/excel.html?highlight=xlsx#microsoft-excel
+from langchain.document_loaders import CSVLoader, PDFMinerLoader, TextLoader, UnstructuredExcelLoader, Docx2txtLoader
+
+# load_dotenv()
+ROOT_DIRECTORY = os.path.dirname(os.path.realpath(__file__))
+
+# Define the folder for storing database
+SOURCE_DIRECTORY = f"{ROOT_DIRECTORY}/SOURCE_DOCUMENTS"
+
+PERSIST_DIRECTORY = f"{ROOT_DIRECTORY}/DB"
+
+# Can be changed to a specific number
+INGEST_THREADS = os.cpu_count() or 8
+
+# Define the Chroma settings
+CHROMA_SETTINGS = Settings(
+    chroma_db_impl="duckdb+parquet", persist_directory=PERSIST_DIRECTORY, anonymized_telemetry=False
+)
+
+# https://python.langchain.com/en/latest/_modules/langchain/document_loaders/excel.html#UnstructuredExcelLoader
+DOCUMENT_MAP = {
+    ".txt": TextLoader,
+    ".md": TextLoader,
+    ".py": TextLoader,
+    ".pdf": PDFMinerLoader,
+    ".csv": CSVLoader,
+    ".xls": UnstructuredExcelLoader,
+    ".xlsx": UnstructuredExcelLoader,
+    ".docx": Docx2txtLoader,
+    ".doc": Docx2txtLoader,
+}
+
+# Default Instructor Model
+EMBEDDING_MODEL_NAME = "hkunlp/instructor-large"
+# You can also choose a smaller model, don't forget to change HuggingFaceInstructEmbeddings
+# to HuggingFaceEmbeddings in both ingest.py and run_localGPT.py
+# EMBEDDING_MODEL_NAME = "all-MiniLM-L6-v2"

end_calculate.py

@@ -0,0 +1,24 @@
+import json
+
+def calculate_end_from_file(input_file_path, output_file_path):
+    with open(input_file_path, 'r') as file:
+        input_data = json.load(file)
+
+    # Iterate through the list of dictionaries and calculate "end" for each one
+    for item in input_data:
+        item["end"] =round(item["start"] + item["duration"],2)
+        del item["duration"]  # Remove the "duration" key from each dictionary
+
+    # Save the updated data to a new JSON file
+    with open(output_file_path, 'w') as output_file:
+        json.dump(input_data, output_file)
+
+# # Replace 'input_file.json' with the actual path to your input JSON file
+# input_file_path = '/home/bharathi/langchain_experiments/GenAI/transcript.json'
+
+# # Replace 'output_file.json' with the desired path and filename for the output JSON file
+# output_file_path = '/home/bharathi/langchain_experiments/GenAI/transcript_end.json'
+
+# # Call the function to calculate the "end" values and remove "duration" and save the new JSON file
+# calculate_end_from_file(input_file_path, output_file_path)
+

ingest.py

@@ -0,0 +1,131 @@
+import logging
+import os
+from concurrent.futures import ProcessPoolExecutor, ThreadPoolExecutor, as_completed
+
+import click
+import torch
+from langchain.docstore.document import Document
+from langchain.embeddings import HuggingFaceInstructEmbeddings
+from langchain.text_splitter import Language, RecursiveCharacterTextSplitter
+from langchain.vectorstores import Chroma
+
+from constants import (
+    CHROMA_SETTINGS,
+    DOCUMENT_MAP,
+    EMBEDDING_MODEL_NAME,
+    INGEST_THREADS,
+    PERSIST_DIRECTORY,
+    SOURCE_DIRECTORY,
+)
+
+
+def load_single_document(file_path: str) -> Document:
+    # Loads a single document from a file path
+    file_extension = os.path.splitext(file_path)[1]
+    loader_class = DOCUMENT_MAP.get(file_extension)
+    if loader_class:
+        loader = loader_class(file_path)
+    else:
+        raise ValueError("Document type is undefined")
+    return loader.load()[0]
+
+
+def load_document_batch(filepaths):
+    logging.info("Loading document batch")
+    # create a thread pool
+    with ThreadPoolExecutor(len(filepaths)) as exe:
+        # load files
+        futures = [exe.submit(load_single_document, name) for name in filepaths]
+        # collect data
+        data_list = [future.result() for future in futures]
+        # return data and file paths
+        return (data_list, filepaths)
+
+
+def load_documents(source_dir: str) -> list[Document]:
+    # Loads all documents from the source documents directory
+    all_files = os.listdir(source_dir)
+    paths = []
+    for file_path in all_files:
+        file_extension = os.path.splitext(file_path)[1]
+        source_file_path = os.path.join(source_dir, file_path)
+        if file_extension in DOCUMENT_MAP.keys():
+            paths.append(source_file_path)
+
+    # Have at least one worker and at most INGEST_THREADS workers
+    n_workers = min(INGEST_THREADS, max(len(paths), 1))
+    chunksize = round(len(paths) / n_workers)
+    docs = []
+    with ProcessPoolExecutor(n_workers) as executor:
+        futures = []
+        # split the load operations into chunks
+        for i in range(0, len(paths), chunksize):
+            # select a chunk of filenames
+            filepaths = paths[i : (i + chunksize)]
+            # submit the task
+            future = executor.submit(load_document_batch, filepaths)
+            futures.append(future)
+        # process all results
+        for future in as_completed(futures):
+            # open the file and load the data
+            contents, _ = future.result()
+            docs.extend(contents)
+
+    return docs
+
+
+def split_documents(documents: list[Document]) -> tuple[list[Document], list[Document]]:
+    # Splits documents for correct Text Splitter
+    text_docs, python_docs = [], []
+    for doc in documents:
+        file_extension = os.path.splitext(doc.metadata["source"])[1]
+        if file_extension == ".py":
+            python_docs.append(doc)
+        else:
+            text_docs.append(doc)
+
+    return text_docs, python_docs
+
+def main():#device_type):
+    # Load documents and split in chunks
+    logging.info(f"Loading documents from {SOURCE_DIRECTORY}")
+    documents = load_documents(SOURCE_DIRECTORY)
+    text_documents, python_documents = split_documents(documents)
+    text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200)
+    python_splitter = RecursiveCharacterTextSplitter.from_language(
+        language=Language.PYTHON, chunk_size=1000, chunk_overlap=200
+    )
+    texts = text_splitter.split_documents(text_documents)
+    texts.extend(python_splitter.split_documents(python_documents))
+    logging.info(f"Loaded {len(documents)} documents from {SOURCE_DIRECTORY}")
+    logging.info(f"Split into {len(texts)} chunks of text")
+
+    # Create embeddings
+    embeddings = HuggingFaceInstructEmbeddings(
+        model_name=EMBEDDING_MODEL_NAME,
+        model_kwargs={"device": "cpu"},
+    )
+    # change the embedding type here if you are running into issues.
+    # These are much smaller embeddings and will work for most appications
+    # If you use HuggingFaceEmbeddings, make sure to also use the same in the
+    # run_localGPT.py file.
+
+    # embeddings = HuggingFaceEmbeddings(model_name=EMBEDDING_MODEL_NAME)
+
+    db = Chroma.from_documents(
+        texts,
+        embeddings,
+        persist_directory=PERSIST_DIRECTORY,
+        client_settings=CHROMA_SETTINGS,
+    )
+    db.persist()
+    db = None
+
+    return "done"
+
+
+if __name__ == "__main__":
+    logging.basicConfig(
+        format="%(asctime)s - %(levelname)s - %(filename)s:%(lineno)s - %(message)s", level=logging.INFO
+    )
+    main()

jira.csv

@@ -0,0 +1,4 @@
+Key,Summary,Assignee
+JT-3,Youtube querying,Srikanth Murleedharan
+JT-2,Functionality for Teams transcript,Bharathi Sriram
+JT-1,Multiple-page UI,Lakshmi Narayanan

modJira.py

@@ -0,0 +1,49 @@
+from jira.client import JIRA
+import pandas as pd
+
+def get_details(project_id):
+        # Specify a server key. It should be your
+    # domain name link. yourdomainname.atlassian.net
+    jiraOptions = {'server': "https://srikanthnm.atlassian.net"}
+    
+    # Get a JIRA client instance, pass,
+    # Authentication parameters
+    # and the Server name.
+    # emailID = your emailID
+    # token = token you receive after registration
+    jira = JIRA(options=jiraOptions, basic_auth=(
+        "srikanth.nm@gmail.com", "ATATT3xFfGF09rugcjiT06v8xMayt5ggayMNiwz4b6w07PWQxPvpi4fMDzwwHxKt-v8dGx49uiulIMKHUUYroeS8cXvMKYfi7sQnFsYNfGslPVqSq1BQrzPhTio-xmYOHcit5ijzU9cSGGa7eLXUMxQTsSQjLhtZ-EQPI8h6aki690_-evLFZmU=3910FFD4"))
+    
+    # Search all issues mentioned against a project name.
+    lstKeys = []
+    lstSummary = []
+    lstReporter = []
+    for singleIssue in jira.search_issues(jql_str=f'project = {project_id}'):
+        lstKeys.append(singleIssue.key)
+        lstSummary.append(singleIssue.fields.summary)
+        lstReporter.append(singleIssue.fields.assignee.displayName)
+
+    df_output = pd.DataFrame()
+    df_output['Key'] = lstKeys
+    df_output['Summary'] = lstSummary
+    df_output['Assignee'] = lstReporter
+
+    df_output.to_csv('jira.csv', index=False)
+
+    return df_output
+
+def ask_question(strQuery):
+    from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
+    import pandas as pd
+
+    tokenizer = AutoTokenizer.from_pretrained("Yale-LILY/reastap-large")
+    model = AutoModelForSeq2SeqLM.from_pretrained("Yale-LILY/reastap-large")
+
+    table = pd.read_csv("jira.csv")
+
+    query = strQuery
+    encoding = tokenizer(table=table, query=query, return_tensors="pt")
+
+    outputs = model.generate(**encoding)
+
+    return (tokenizer.batch_decode(outputs, skip_special_tokens=True)[0])

requirements.txt

@@ -0,0 +1,34 @@
+# Natural Language Processing
+langchain==0.0.191
+chromadb==0.3.22
+llama-cpp-python==0.1.66
+pdfminer.six==20221105
+InstructorEmbedding
+sentence-transformers
+faiss-cpu
+huggingface_hub
+transformers
+protobuf==3.20.0; sys_platform != 'darwin'
+protobuf==3.20.0; sys_platform == 'darwin' and platform_machine != 'arm64'
+protobuf==3.20.3; sys_platform == 'darwin' and platform_machine == 'arm64'
+auto-gptq==0.2.2
+docx2txt
+
+# Utilities
+urllib3==1.26.6
+accelerate
+bitsandbytes ; sys_platform != 'win32'
+bitsandbytes-windows ; sys_platform == 'win32'
+click
+flask
+requests
+
+# Excel File Manipulation
+openpyxl
+
+#custom
+youtube-transcript-api
+streamlit
+jira
+
+

run_localGPT.py

@@ -0,0 +1,245 @@
+import logging
+
+import click
+import torch
+from auto_gptq import AutoGPTQForCausalLM
+from huggingface_hub import hf_hub_download
+from langchain.chains import RetrievalQA
+from langchain.embeddings import HuggingFaceInstructEmbeddings
+from langchain.llms import HuggingFacePipeline, LlamaCpp
+
+# from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler
+from langchain.vectorstores import Chroma
+from transformers import (
+    AutoModelForCausalLM,
+    AutoTokenizer,
+    GenerationConfig,
+    LlamaForCausalLM,
+    LlamaTokenizer,
+    pipeline,
+)
+
+from constants import CHROMA_SETTINGS, EMBEDDING_MODEL_NAME, PERSIST_DIRECTORY
+
+
+def load_model(device_type, model_id, model_basename=None):
+    """
+    Select a model for text generation using the HuggingFace library.
+    If you are running this for the first time, it will download a model for you.
+    subsequent runs will use the model from the disk.
+
+    Args:
+        device_type (str): Type of device to use, e.g., "cuda" for GPU or "cpu" for CPU.
+        model_id (str): Identifier of the model to load from HuggingFace's model hub.
+        model_basename (str, optional): Basename of the model if using quantized models.
+            Defaults to None.
+
+    Returns:
+        HuggingFacePipeline: A pipeline object for text generation using the loaded model.
+
+    Raises:
+        ValueError: If an unsupported model or device type is provided.
+    """
+    logging.info(f"Loading Model: {model_id}, on: {device_type}")
+    logging.info("This action can take a few minutes!")
+
+    if model_basename is not None:
+        if ".ggml" in model_basename:
+            logging.info("Using Llamacpp for GGML quantized models")
+            model_path = hf_hub_download(repo_id=model_id, filename=model_basename)
+            max_ctx_size = 2048
+            kwargs = {
+                "model_path": model_path,
+                "n_ctx": max_ctx_size,
+                "max_tokens": max_ctx_size,
+            }
+            if device_type.lower() == "mps":
+                kwargs["n_gpu_layers"] = 1000
+            if device_type.lower() == "cuda":
+                kwargs["n_gpu_layers"] = 1000
+                kwargs["n_batch"] = max_ctx_size
+            return LlamaCpp(**kwargs)
+
+        else:
+            # The code supports all huggingface models that ends with GPTQ and have some variation
+            # of .no-act.order or .safetensors in their HF repo.
+            logging.info("Using AutoGPTQForCausalLM for quantized models")
+
+            if ".safetensors" in model_basename:
+                # Remove the ".safetensors" ending if present
+                model_basename = model_basename.replace(".safetensors", "")
+
+            tokenizer = AutoTokenizer.from_pretrained(model_id, use_fast=True)
+            logging.info("Tokenizer loaded")
+
+            model = AutoGPTQForCausalLM.from_quantized(
+                model_id,
+                model_basename=model_basename,
+                use_safetensors=True,
+                trust_remote_code=True,
+                device="cuda:0",
+                use_triton=False,
+                quantize_config=None,
+            )
+    elif (
+        device_type.lower() == "cuda"
+    ):  # The code supports all huggingface models that ends with -HF or which have a .bin
+        # file in their HF repo.
+        logging.info("Using AutoModelForCausalLM for full models")
+        tokenizer = AutoTokenizer.from_pretrained(model_id)
+        logging.info("Tokenizer loaded")
+
+        model = AutoModelForCausalLM.from_pretrained(
+            model_id,
+            device_map="auto",
+            torch_dtype=torch.float16,
+            low_cpu_mem_usage=True,
+            trust_remote_code=True,
+            # max_memory={0: "15GB"} # Uncomment this line with you encounter CUDA out of memory errors
+        )
+        model.tie_weights()
+    else:
+        logging.info("Using LlamaTokenizer")
+        tokenizer = LlamaTokenizer.from_pretrained(model_id)
+        model = LlamaForCausalLM.from_pretrained(model_id)
+
+    # Load configuration from the model to avoid warnings
+    generation_config = GenerationConfig.from_pretrained(model_id)
+    # see here for details:
+    # https://huggingface.co/docs/transformers/
+    # main_classes/text_generation#transformers.GenerationConfig.from_pretrained.returns
+
+    # Create a pipeline for text generation
+    pipe = pipeline(
+        "text-generation",
+        model=model,
+        tokenizer=tokenizer,
+        max_length=2048,
+        temperature=0,
+        top_p=0.95,
+        repetition_penalty=1.15,
+        generation_config=generation_config,
+    )
+
+    local_llm = HuggingFacePipeline(pipeline=pipe)
+    logging.info("Local LLM Loaded")
+
+    return local_llm
+
+
+# # chose device typ to run on as well as to show source documents.
+# @click.command()
+# @click.option(
+#     "--device_type",
+#     default="cuda" if torch.cuda.is_available() else "cpu",
+#     type=click.Choice(
+#         [
+#             "cpu",
+#             "cuda",
+#             "ipu",
+#             "xpu",
+#             "mkldnn",
+#             "opengl",
+#             "opencl",
+#             "ideep",
+#             "hip",
+#             "ve",
+#             "fpga",
+#             "ort",
+#             "xla",
+#             "lazy",
+#             "vulkan",
+#             "mps",
+#             "meta",
+#             "hpu",
+#             "mtia",
+#         ],
+#     ),
+#     help="Device to run on. (Default is cuda)",
+# )
+# @click.option(
+#     "--show_sources",
+#     "-s",
+#     is_flag=True,
+#     help="Show sources along with answers (Default is False)",
+# )
+def main(device_type, strQuery):
+    """
+    This function implements the information retrieval task.
+
+
+    1. Loads an embedding model, can be HuggingFaceInstructEmbeddings or HuggingFaceEmbeddings
+    2. Loads the existing vectorestore that was created by inget.py
+    3. Loads the local LLM using load_model function - You can now set different LLMs.
+    4. Setup the Question Answer retreival chain.
+    5. Question answers.
+    """
+
+    logging.info(f"Running on: {device_type}")
+    #logging.info(f"Display Source Documents set to: {show_sources}")
+
+    embeddings = HuggingFaceInstructEmbeddings(model_name=EMBEDDING_MODEL_NAME, model_kwargs={"device": device_type})
+
+    # uncomment the following line if you used HuggingFaceEmbeddings in the ingest.py
+    # embeddings = HuggingFaceEmbeddings(model_name=EMBEDDING_MODEL_NAME)
+
+    # load the vectorstore
+    db = Chroma(
+        persist_directory=PERSIST_DIRECTORY,
+        embedding_function=embeddings,
+        client_settings=CHROMA_SETTINGS,
+    )
+    retriever = db.as_retriever()
+
+    # load the LLM for generating Natural Language responses
+
+    # for HF models
+    # model_id = "TheBloke/vicuna-7B-1.1-HF"
+    # model_basename = None
+    # model_id = "TheBloke/Wizard-Vicuna-7B-Uncensored-HF"
+    # model_id = "TheBloke/guanaco-7B-HF"
+    # model_id = 'NousResearch/Nous-Hermes-13b' # Requires ~ 23GB VRAM. Using STransformers
+    # alongside will 100% create OOM on 24GB cards.
+    # llm = load_model(device_type, model_id=model_id)
+
+    # for GPTQ (quantized) models
+    # model_id = "TheBloke/Nous-Hermes-13B-GPTQ"
+    # model_basename = "nous-hermes-13b-GPTQ-4bit-128g.no-act.order"
+    # model_id = "TheBloke/WizardLM-30B-Uncensored-GPTQ"
+    # model_basename = "WizardLM-30B-Uncensored-GPTQ-4bit.act-order.safetensors" # Requires
+    # ~21GB VRAM. Using STransformers alongside can potentially create OOM on 24GB cards.
+    # model_id = "TheBloke/wizardLM-7B-GPTQ"
+    # model_basename = "wizardLM-7B-GPTQ-4bit.compat.no-act-order.safetensors"
+    # model_id = "TheBloke/WizardLM-7B-uncensored-GPTQ"
+    # model_basename = "WizardLM-7B-uncensored-GPTQ-4bit-128g.compat.no-act-order.safetensors"
+
+    # for GGML (quantized cpu+gpu+mps) models - check if they support llama.cpp
+    # model_id = "TheBloke/wizard-vicuna-13B-GGML"
+    # model_basename = "wizard-vicuna-13B.ggmlv3.q4_0.bin"
+    # model_basename = "wizard-vicuna-13B.ggmlv3.q6_K.bin"
+    # model_basename = "wizard-vicuna-13B.ggmlv3.q2_K.bin"
+    # model_id = "TheBloke/orca_mini_3B-GGML"
+    # model_basename = "orca-mini-3b.ggmlv3.q4_0.bin"
+
+    model_id = "TheBloke/Llama-2-7B-Chat-GGML"
+    model_basename = "llama-2-7b-chat.ggmlv3.q4_0.bin"
+
+    llm = load_model(device_type, model_id=model_id, model_basename=model_basename)
+
+    qa = RetrievalQA.from_chain_type(llm=llm, chain_type="stuff", retriever=retriever, return_source_documents=True)
+    # Interactive questions and answers
+    
+    query = strQuery
+    # Get the answer from the chain
+    res = qa(query)
+    answer, docs = res["result"], res["source_documents"]
+
+    return(answer)
+
+
+
+if __name__ == "__main__":
+    logging.basicConfig(
+        format="%(asctime)s - %(levelname)s - %(filename)s:%(lineno)s - %(message)s", level=logging.INFO
+    )
+    main()

similarity.py

@@ -0,0 +1,22 @@
+from sentence_transformers import SentenceTransformer, util
+import json
+import numpy as np
+model = SentenceTransformer('sentence-transformers/all-MiniLM-L6-v2')
+
+def similarity(strQuery):
+
+    inputs = json.load(open('chunks.json','r'))
+    lstCorpus = [dct['text'] for dct in inputs]
+
+    strQuery = "How many different document types?"
+    qryEmbedding = model.encode(strQuery, convert_to_tensor=True)
+    corpusEmbedding= model.encode(lstCorpus, convert_to_tensor=True)
+        
+    sim_mat = util.pytorch_cos_sim(qryEmbedding, corpusEmbedding)
+    lstSim = sim_mat[0].tolist()
+    npSim = np.array(lstSim)
+    indexMax = npSim.argmax()
+    scoreMax = npSim.max()
+
+    return(inputs[indexMax]['start'], inputs[indexMax]['end'])
+

transcript.json

@@ -0,0 +1,312 @@
+[
+    {
+        "text": "In this video, I'm going to answer the top 3 questions",
+        "start": 0.0,
+        "duration": 4.0
+    },
+    {
+        "text": "my students ask me about Python. What is Python? What ",
+        "start": 4.0,
+        "duration": 4.0
+    },
+    {
+        "text": "can you do with it? And why is it so popular? In other words, what",
+        "start": 8.0,
+        "duration": 4.0
+    },
+    {
+        "text": "does it do that other programming languages don't? Python is the ",
+        "start": 12.0,
+        "duration": 4.0
+    },
+    {
+        "text": "world's fastest growing and most popular programming language, not just ",
+        "start": 16.0,
+        "duration": 4.0
+    },
+    {
+        "text": "amongst software engineers, but also amongst mathematicians, ",
+        "start": 20.0,
+        "duration": 4.0
+    },
+    {
+        "text": "data analysts, scientists, accountants, networking engineers,",
+        "start": 24.0,
+        "duration": 4.0
+    },
+    {
+        "text": "and even kids! Because it's a very beginner friendly programming ",
+        "start": 28.0,
+        "duration": 4.0
+    },
+    {
+        "text": "language. So people from different disciplines use Python",
+        "start": 32.0,
+        "duration": 4.0
+    },
+    {
+        "text": "for a variety of different tasks, such as data analysis and visualization, ",
+        "start": 36.0,
+        "duration": 4.0
+    },
+    {
+        "text": "artificial intelligence and machine learning, automation ",
+        "start": 40.0,
+        "duration": 4.0
+    },
+    {
+        "text": "in fact this is one of the big uses of Python amongst people who are not software",
+        "start": 44.0,
+        "duration": 4.0
+    },
+    {
+        "text": "developers. If you constantly have to do boring, repetitive ",
+        "start": 48.0,
+        "duration": 4.0
+    },
+    {
+        "text": "tasks, such as copying files and folders around, renaming them, ",
+        "start": 52.0,
+        "duration": 4.0
+    },
+    {
+        "text": "uploading them to a server, you can easily write a Python script to",
+        "start": 56.0,
+        "duration": 4.0
+    },
+    {
+        "text": "automate all that and save your time. And that's just one example, if you",
+        "start": 60.0,
+        "duration": 4.0
+    },
+    {
+        "text": "continuously have to work with excel spreadsheets, PDF's, CS",
+        "start": 64.0,
+        "duration": 4.0
+    },
+    {
+        "text": "View files, download websites and parse them, you can automate all",
+        "start": 68.0,
+        "duration": 4.0
+    },
+    {
+        "text": "that stuff with Python. So you don't have to be a software developer to use Python.",
+        "start": 72.0,
+        "duration": 4.0
+    },
+    {
+        "text": "You could be an accountant, a mathematician, or a scientist, and use Python ",
+        "start": 76.0,
+        "duration": 4.0
+    },
+    {
+        "text": "to make your life easier. You can also use Python to build ",
+        "start": 80.0,
+        "duration": 4.0
+    },
+    {
+        "text": "web, mobile and desktop applications as well as software ",
+        "start": 84.0,
+        "duration": 4.0
+    },
+    {
+        "text": "testing or even hacking. So Python is a multi purpose language. ",
+        "start": 88.0,
+        "duration": 4.0
+    },
+    {
+        "text": "Now if you have some programming experience you may say, \"But Mosh",
+        "start": 92.0,
+        "duration": 4.0
+    },
+    {
+        "text": "we can do all this stuff with other programming languages, so what's the big deal ",
+        "start": 96.0,
+        "duration": 4.0
+    },
+    {
+        "text": "about Python?\" Here are a few reasons. With Python you can ",
+        "start": 100.0,
+        "duration": 4.0
+    },
+    {
+        "text": "solve complex problems in less time with fewer lines of code. ",
+        "start": 104.0,
+        "duration": 4.0
+    },
+    {
+        "text": "Here's an example. Let's say we want to extract the first three ",
+        "start": 108.0,
+        "duration": 4.0
+    },
+    {
+        "text": "letters of the text Hello World. This is the code we have to write ",
+        "start": 112.0,
+        "duration": 4.0
+    },
+    {
+        "text": "in C# this is how we do it in JavaScript and here's how we ",
+        "start": 116.0,
+        "duration": 4.0
+    },
+    {
+        "text": "do it in Python. See how short and clean the language is?",
+        "start": 120.0,
+        "duration": 4.0
+    },
+    {
+        "text": "And that's just the beginning. Python makes a lot of trivial things",
+        "start": 124.0,
+        "duration": 4.0
+    },
+    {
+        "text": "really easy with a simple yet powerful syntax. Here are a few",
+        "start": 128.0,
+        "duration": 4.0
+    },
+    {
+        "text": "other reasons Python is so popular. It's a high level language",
+        "start": 132.0,
+        "duration": 4.0
+    },
+    {
+        "text": "so you don't have to worry about complex tasks such as memory management, ",
+        "start": 136.0,
+        "duration": 4.0
+    },
+    {
+        "text": "like you do in C++. It's cross platform which means ",
+        "start": 140.0,
+        "duration": 4.0
+    },
+    {
+        "text": "you can build and run Python applications on Windows, Mac, ",
+        "start": 144.0,
+        "duration": 4.0
+    },
+    {
+        "text": "and Linux. It has a huge community so whenever you get ",
+        "start": 148.0,
+        "duration": 4.0
+    },
+    {
+        "text": "stuck, there is someone out there to help. It has a large ecosystem ",
+        "start": 152.0,
+        "duration": 4.0
+    },
+    {
+        "text": "of libraries, frameworks and tools which means whatever you wanna do",
+        "start": 156.0,
+        "duration": 4.0
+    },
+    {
+        "text": "it is likely that someone else has done it before because Python has been around ",
+        "start": 160.0,
+        "duration": 4.0
+    },
+    {
+        "text": "for over 20 years. So in a nutshell, Python",
+        "start": 164.0,
+        "duration": 4.0
+    },
+    {
+        "text": "is a multi-purpose language with a simple, clean, and beginner-friendly ",
+        "start": 168.0,
+        "duration": 4.0
+    },
+    {
+        "text": "syntax. All of that means Python is awesome.",
+        "start": 172.0,
+        "duration": 4.0
+    },
+    {
+        "text": "Technically everything you do with Python you can do with other programming languages, ",
+        "start": 176.0,
+        "duration": 4.0
+    },
+    {
+        "text": "but Python's simplicity and elegance has made it grow way ",
+        "start": 180.0,
+        "duration": 4.0
+    },
+    {
+        "text": "more than other programming languages. That's why it's the number onne",
+        "start": 184.0,
+        "duration": 4.0
+    },
+    {
+        "text": "language employers are looking for. So whether you're a programmer or ",
+        "start": 188.0,
+        "duration": 4.0
+    },
+    {
+        "text": "an absolute beginner, learning Python opens up lots of job opportunities ",
+        "start": 192.0,
+        "duration": 4.0
+    },
+    {
+        "text": "to you. In fact, the average Python developer earns a whopping",
+        "start": 196.0,
+        "duration": 4.0
+    },
+    {
+        "text": "116,000 dollars a year. If you",
+        "start": 200.0,
+        "duration": 4.0
+    },
+    {
+        "text": "found this video helpful, please support my hard work by liking and sharing it with others. ",
+        "start": 204.0,
+        "duration": 4.0
+    },
+    {
+        "text": "Also, be sure to subscribe to my channel, because I have a couple of",
+        "start": 208.0,
+        "duration": 4.0
+    },
+    {
+        "text": "awesome Python tutorials for you, you're going to see them on the screen now. ",
+        "start": 212.0,
+        "duration": 4.0
+    },
+    {
+        "text": "Here's my Python tutorial for beginners, it's a great starting point if you ",
+        "start": 216.0,
+        "duration": 4.0
+    },
+    {
+        "text": "have limited or no programming experience. On the other hand, if you ",
+        "start": 220.0,
+        "duration": 4.0
+    },
+    {
+        "text": "do have some programming experience and want to quickly get up to speed with Python, ",
+        "start": 224.0,
+        "duration": 4.0
+    },
+    {
+        "text": "I have another tutorial just for you. I'm not going to waste your time ",
+        "start": 228.0,
+        "duration": 4.0
+    },
+    {
+        "text": "telling you what a variable or a function is. I will talk to you like a programmer.",
+        "start": 232.0,
+        "duration": 4.0
+    },
+    {
+        "text": "There's never been a better time to master Python programming,",
+        "start": 236.0,
+        "duration": 4.0
+    },
+    {
+        "text": "so click on the tutorial that is right for you and get started. Thank you for",
+        "start": 240.0,
+        "duration": 4.0
+    },
+    {
+        "text": "watching!",
+        "start": 244.0,
+        "duration": 2.633
+    }
+]

transcript_end.json

@@ -0,0 +1,312 @@
+[
+    {
+        "text": "In this video, I'm going to answer the top 3 questions",
+        "start": 0.0,
+        "end": 4.0
+    },
+    {
+        "text": "my students ask me about Python. What is Python? What ",
+        "start": 4.0,
+        "end": 8.0
+    },
+    {
+        "text": "can you do with it? And why is it so popular? In other words, what",
+        "start": 8.0,
+        "end": 12.0
+    },
+    {
+        "text": "does it do that other programming languages don't? Python is the ",
+        "start": 12.0,
+        "end": 16.0
+    },
+    {
+        "text": "world's fastest growing and most popular programming language, not just ",
+        "start": 16.0,
+        "end": 20.0
+    },
+    {
+        "text": "amongst software engineers, but also amongst mathematicians, ",
+        "start": 20.0,
+        "end": 24.0
+    },
+    {
+        "text": "data analysts, scientists, accountants, networking engineers,",
+        "start": 24.0,
+        "end": 28.0
+    },
+    {
+        "text": "and even kids! Because it's a very beginner friendly programming ",
+        "start": 28.0,
+        "end": 32.0
+    },
+    {
+        "text": "language. So people from different disciplines use Python",
+        "start": 32.0,
+        "end": 36.0
+    },
+    {
+        "text": "for a variety of different tasks, such as data analysis and visualization, ",
+        "start": 36.0,
+        "end": 40.0
+    },
+    {
+        "text": "artificial intelligence and machine learning, automation ",
+        "start": 40.0,
+        "end": 44.0
+    },
+    {
+        "text": "in fact this is one of the big uses of Python amongst people who are not software",
+        "start": 44.0,
+        "end": 48.0
+    },
+    {
+        "text": "developers. If you constantly have to do boring, repetitive ",
+        "start": 48.0,
+        "end": 52.0
+    },
+    {
+        "text": "tasks, such as copying files and folders around, renaming them, ",
+        "start": 52.0,
+        "end": 56.0
+    },
+    {
+        "text": "uploading them to a server, you can easily write a Python script to",
+        "start": 56.0,
+        "end": 60.0
+    },
+    {
+        "text": "automate all that and save your time. And that's just one example, if you",
+        "start": 60.0,
+        "end": 64.0
+    },
+    {
+        "text": "continuously have to work with excel spreadsheets, PDF's, CS",
+        "start": 64.0,
+        "end": 68.0
+    },
+    {
+        "text": "View files, download websites and parse them, you can automate all",
+        "start": 68.0,
+        "end": 72.0
+    },
+    {
+        "text": "that stuff with Python. So you don't have to be a software developer to use Python.",
+        "start": 72.0,
+        "end": 76.0
+    },
+    {
+        "text": "You could be an accountant, a mathematician, or a scientist, and use Python ",
+        "start": 76.0,
+        "end": 80.0
+    },
+    {
+        "text": "to make your life easier. You can also use Python to build ",
+        "start": 80.0,
+        "end": 84.0
+    },
+    {
+        "text": "web, mobile and desktop applications as well as software ",
+        "start": 84.0,
+        "end": 88.0
+    },
+    {
+        "text": "testing or even hacking. So Python is a multi purpose language. ",
+        "start": 88.0,
+        "end": 92.0
+    },
+    {
+        "text": "Now if you have some programming experience you may say, \"But Mosh",
+        "start": 92.0,
+        "end": 96.0
+    },
+    {
+        "text": "we can do all this stuff with other programming languages, so what's the big deal ",
+        "start": 96.0,
+        "end": 100.0
+    },
+    {
+        "text": "about Python?\" Here are a few reasons. With Python you can ",
+        "start": 100.0,
+        "end": 104.0
+    },
+    {
+        "text": "solve complex problems in less time with fewer lines of code. ",
+        "start": 104.0,
+        "end": 108.0
+    },
+    {
+        "text": "Here's an example. Let's say we want to extract the first three ",
+        "start": 108.0,
+        "end": 112.0
+    },
+    {
+        "text": "letters of the text Hello World. This is the code we have to write ",
+        "start": 112.0,
+        "end": 116.0
+    },
+    {
+        "text": "in C# this is how we do it in JavaScript and here's how we ",
+        "start": 116.0,
+        "end": 120.0
+    },
+    {
+        "text": "do it in Python. See how short and clean the language is?",
+        "start": 120.0,
+        "end": 124.0
+    },
+    {
+        "text": "And that's just the beginning. Python makes a lot of trivial things",
+        "start": 124.0,
+        "end": 128.0
+    },
+    {
+        "text": "really easy with a simple yet powerful syntax. Here are a few",
+        "start": 128.0,
+        "end": 132.0
+    },
+    {
+        "text": "other reasons Python is so popular. It's a high level language",
+        "start": 132.0,
+        "end": 136.0
+    },
+    {
+        "text": "so you don't have to worry about complex tasks such as memory management, ",
+        "start": 136.0,
+        "end": 140.0
+    },
+    {
+        "text": "like you do in C++. It's cross platform which means ",
+        "start": 140.0,
+        "end": 144.0
+    },
+    {
+        "text": "you can build and run Python applications on Windows, Mac, ",
+        "start": 144.0,
+        "end": 148.0
+    },
+    {
+        "text": "and Linux. It has a huge community so whenever you get ",
+        "start": 148.0,
+        "end": 152.0
+    },
+    {
+        "text": "stuck, there is someone out there to help. It has a large ecosystem ",
+        "start": 152.0,
+        "end": 156.0
+    },
+    {
+        "text": "of libraries, frameworks and tools which means whatever you wanna do",
+        "start": 156.0,
+        "end": 160.0
+    },
+    {
+        "text": "it is likely that someone else has done it before because Python has been around ",
+        "start": 160.0,
+        "end": 164.0
+    },
+    {
+        "text": "for over 20 years. So in a nutshell, Python",
+        "start": 164.0,
+        "end": 168.0
+    },
+    {
+        "text": "is a multi-purpose language with a simple, clean, and beginner-friendly ",
+        "start": 168.0,
+        "end": 172.0
+    },
+    {
+        "text": "syntax. All of that means Python is awesome.",
+        "start": 172.0,
+        "end": 176.0
+    },
+    {
+        "text": "Technically everything you do with Python you can do with other programming languages, ",
+        "start": 176.0,
+        "end": 180.0
+    },
+    {
+        "text": "but Python's simplicity and elegance has made it grow way ",
+        "start": 180.0,
+        "end": 184.0
+    },
+    {
+        "text": "more than other programming languages. That's why it's the number onne",
+        "start": 184.0,
+        "end": 188.0
+    },
+    {
+        "text": "language employers are looking for. So whether you're a programmer or ",
+        "start": 188.0,
+        "end": 192.0
+    },
+    {
+        "text": "an absolute beginner, learning Python opens up lots of job opportunities ",
+        "start": 192.0,
+        "end": 196.0
+    },
+    {
+        "text": "to you. In fact, the average Python developer earns a whopping",
+        "start": 196.0,
+        "end": 200.0
+    },
+    {
+        "text": "116,000 dollars a year. If you",
+        "start": 200.0,
+        "end": 204.0
+    },
+    {
+        "text": "found this video helpful, please support my hard work by liking and sharing it with others. ",
+        "start": 204.0,
+        "end": 208.0
+    },
+    {
+        "text": "Also, be sure to subscribe to my channel, because I have a couple of",
+        "start": 208.0,
+        "end": 212.0
+    },
+    {
+        "text": "awesome Python tutorials for you, you're going to see them on the screen now. ",
+        "start": 212.0,
+        "end": 216.0
+    },
+    {
+        "text": "Here's my Python tutorial for beginners, it's a great starting point if you ",
+        "start": 216.0,
+        "end": 220.0
+    },
+    {
+        "text": "have limited or no programming experience. On the other hand, if you ",
+        "start": 220.0,
+        "end": 224.0
+    },
+    {
+        "text": "do have some programming experience and want to quickly get up to speed with Python, ",
+        "start": 224.0,
+        "end": 228.0
+    },
+    {
+        "text": "I have another tutorial just for you. I'm not going to waste your time ",
+        "start": 228.0,
+        "end": 232.0
+    },
+    {
+        "text": "telling you what a variable or a function is. I will talk to you like a programmer.",
+        "start": 232.0,
+        "end": 236.0
+    },
+    {
+        "text": "There's never been a better time to master Python programming,",
+        "start": 236.0,
+        "end": 240.0
+    },
+    {
+        "text": "so click on the tutorial that is right for you and get started. Thank you for",
+        "start": 240.0,
+        "end": 244.0
+    },
+    {
+        "text": "watching!",
+        "start": 244.0,
+        "end": 246.63
+    }
+]

utils.py

@@ -0,0 +1,51 @@
+import json
+
+def calculate_ends(input_file_path, output_file_path):
+    with open(input_file_path, 'r') as file:
+        input_data = json.load(file)
+
+    # Iterate through the list of dictionaries and calculate "end" for each one
+    for item in input_data:
+        item["end"] =round(item["start"] + item["duration"],2)
+        del item["duration"]  # Remove the "duration" key from each dictionary
+
+    # Save the updated data to a new JSON file
+    with open(output_file_path, 'w') as output_file:
+        json.dump(input_data, output_file)
+
+import json
+
+def create_chunks(input_file_path, output_file_path):
+    with open(input_file_path, 'r') as file:
+        output_data = json.load(file)
+
+    combined_json_list = []
+
+    # Calculate the number of groups to create
+    num_groups = (len(output_data) + 3) // 4
+
+    for group_num in range(num_groups):
+        # Calculate the starting index and ending index for the current group
+        start_index = group_num * 4
+        end_index = min(start_index + 4, len(output_data))
+
+        # Extract the "text" values from the current group of dictionaries
+        combined_text = " ".join([item["text"] for item in output_data[start_index:end_index]])
+
+        # Calculate the "start" and "end" for the current group
+        group_start = output_data[start_index]["start"]
+        group_end = output_data[end_index - 1]["end"]
+
+        # Create the combined JSON for the current group
+        combined_json = {
+            "text": combined_text,
+            "start": group_start,
+            "end": group_end,
+        }
+
+        combined_json_list.append(combined_json)
+
+    # Save the combined JSON list to a new file
+    with open(output_file_path, 'w') as output_file:
+        json.dump(combined_json_list, output_file)
+

youtube.py

@@ -0,0 +1,68 @@
+from youtube_transcript_api import YouTubeTranscriptApi
+from youtube_transcript_api.formatters import JSONFormatter
+import json
+import ingest
+import run_localGPT
+import utils
+
+def audio_to_transcript(video_id):
+    sub = YouTubeTranscriptApi.get_transcript(video_id)
+    formatted_subs = JSONFormatter().format_transcript(transcript=sub)
+    with open("transcript.json", "w") as outfile:
+        json.dump(sub, outfile)
+    lstTexts = []
+    for dct in sub:
+        lstTexts.append(dct['text'])
+    strResult = ' '.join(lstTexts)
+    with open('SOURCE_DOCUMENTS/transcript.txt', 'w') as outfile:
+        outfile.write(strResult)
+    transcript = ' '.join(lstTexts)
+
+    utils.calculate_ends('transcript.json','transcript_end.json')
+    utils.create_chunks('transcript_end.json','chunks.json')
+
+    return transcript
+
+def start_training():
+    training_status = ingest.main()
+    return training_status
+
+def replace_substring_and_following(input_string, substring):
+    index = input_string.find(substring)
+    if index != -1:
+        return input_string[:index]
+    else:
+        return input_string
+
+def ask_question(strQuestion):
+    answer = run_localGPT.main(device_type='cpu', strQuery=strQuestion)
+    answer_cleaned = replace_substring_and_following(answer, "Unhelpful Answer")
+    return answer_cleaned
+
+def summarize():
+
+    from langchain.text_splitter import CharacterTextSplitter
+    from langchain.chains.mapreduce import MapReduceChain
+    from langchain.prompts import PromptTemplate
+
+    model_id = "TheBloke/Llama-2-7B-Chat-GGML"
+    model_basename = "llama-2-7b-chat.ggmlv3.q4_0.bin"
+
+    llm = run_localGPT.load_model(device_type='cpu', model_id=model_id, model_basename=model_basename)
+
+    text_splitter = CharacterTextSplitter()
+
+    with open("SOURCE_DOCUMENTS/transcript.txt") as f:
+        file_content = f.read()
+    texts = text_splitter.split_text(file_content)
+
+    from langchain.docstore.document import Document
+
+    docs = [Document(page_content=t) for t in texts]
+
+    from langchain.chains.summarize import load_summarize_chain
+
+    chain = load_summarize_chain(llm, chain_type="map_reduce")
+    summary = chain.run(docs)
+    return summary
+