Repository: Marker-Inc-Korea/AutoRAG
Issue: New feature: existing_query_df
Root cause: Unimplemented Feature Gap

I am planning to send PR with the name of this title. This is for the case where users already have QA datasets, then users can use questions at least so that automatic LLM processing is only depending on answering the question.


sample code to run would be:

```
from autorag.data.qacreation import generate_qa_llama_index, make_single_content_qa
from llama_index.llms.openai import OpenAI
from dotenv import load_dotenv

load_dotenv()

prompt = """
Answer the questions based on the contents.

Questions:
{{query}}

contents:
{{text}}

# This will be set 0 as a zero when 'existing_query_df' is activated'
number of questions by contents: {{num_questions}}

Response:
"""


llm = OpenAI(model='gpt-3.5-turbo', temperature=1.0)
qa_df = make_single_content_qa(corpus_df, content_size=10, qa_creation_func=generate_qa_llama_index,
                               existing_query_df=df, llm=llm, prompt=prompt, question_num_per_content=0, local=False)
qa_df.reset_index(drop=True, inplace=True)
qa_df.to_parquet("qa.parquet", engine='pyarrow')


```
New parameter would be: `existing_query_df` and `local`
__existing_query_df__ requires pandas data frame with the name of 'query' column.
__local__ indicates whether user can choose local or openAI embedding model. default is False(use OpenAI) and only activate when `existing_query_df` is activated. Then, FAISS search will be activated to search the documents.




I am expecting many trials and errors(?) when I send PR. It means ofc, it might require many fixes. 😅

@skmanzg 
It is great feature, but I think it is better to make new function for this.
We are trying to build AutoRAG function do one thing. 
It will be great this feature can be a new function. 
Plus, it is easier to make it on your own when it can be a new function, right?

> @vkehfdl1   it is easier to make it on your own when it can be a new function, right?
 
I am not sure if it was easy but I have changed all structures according to your requirements: independent function, use chromaDB, etc..

so.. new run code will be

```
from autorag.data.qacreation import generate_qa_with_existing_queries, make_qa_with_existing_queries
from llama_index.llms.openai import OpenAI
from dotenv import load_dotenv
import pandas as pd

# Prepare your own dataset! (Eaxamples below)
# corpus_df = pd.read_parquet('corpus.parquet')
# df = pd.read_parquet('q.parquet')

load_dotenv()

prompt = """
Answer the questions based on the contents.

Questions:
{{query}}

contents:
{{text}}

Response:
"""

llm = OpenAI(model='gpt-3.5-turbo')
qa_df = make_qa_with_existing_queries(
    corpus_df, 
    existing_query_df=df, 
    content_size=10, 
    qa_creation_func=generate_qa_with_existing_queries,
    llm=llm, 
    prompt=prompt
)
qa_df.reset_index(drop=True, inplace=True)
qa_df.to_parquet("qa_OpenAI.parquet", engine='pyarrow')

```

For test code, it looks very complicated(?) to me. so.. I would like you to guide about it.. more elaborate.. 
but before that, PR first. but I think it will take a few days.

 

close at #607