zhiminy/EvalEng huggingface dataset:
Repository: Marker-Inc-Korea/AutoRAG
Issue: [Data Creation Refactoring] Re-design QA generation process
Root cause: Unimplemented Feature Gap

Current library is little bit complicated if you want to make a custom QA generation process. 
Making custom QA creation process must be easy. 

"The process will change a lot.”

I think the legacy codes will be deprecated and delete at AutoRAG 0.3.0.


- [x] #661
- [x] #662
- [x] #673
- [x] #674
- [x] #675
- [x] #676
- [x] #677 
- [x] #683

It can be a kind of monad(????? but is it monad???? I don’t know).
There is a dataclass. Let's name it.... `Dataset` 
`Dataset` contains `qa` and `corpus`.
Every function in `Dataset` returns `Dataset` instance, but modified `Dataset` instance. 
Every function must be `pure function`. Its input is `Dataset` instance, and output is modified `Dataset` instance.
We can load and chunk from the class function.
We can make question, make answers. 
After that, you can convert it to qa_df and corpus_df (dataframes for AutoRAG)

For example (concept)

```python
Dataset.from_directory('path/to/dir').recursive_chunk(chunk_size=512, overlap=128).sample_corpus(n=500).create_factoid_question(n=50).create_basic_answer().save_to_parquet(qa_data_path='./data/qa.parquet', corpus_data_path='./data/corpus.parquet')
```

## The basic workflow

1. Parsing the raw document and make `Raw`
2. Initial chunk the corpus from `Raw`
3. Sample some chunks to generate questions
4. Generate questions from the selected chunks
5. Generate answer and finish the initial `QA` set
6. Add more `Corpus` to use different chunking methods
7. Map new retrieval_gt from the new `Corpus` at `QA`

Looks like this below.


```python
openai_client = AsyncOpenAI()
parsing_result = Raw(pd.read_parquet('./parse.parquet'))
initial_corpus = parsing_result.chunk(lambda data: recursive_split(data, chunk_size=128, chunk_overlap=24))
initial_qa = initial_corpus.sample(
    lambda data: random_single_hop(data, n=50)
).batch_apply(
    lambda row: factoid_query_gen(row, openai_client, lang='ko')
).batch_apply(
    lambda row: lambda row: make_concise_gen_gt(row, openai_client, lang='ko')
)

corpus_list: List[Corpus] = parsing_result.chunk_pipeline('./chunk.yaml')
qa_list: List[QA] = list(map(lambda corpus: initial_qa.update_corpus(corpus), corpus_list))
for i, qa in enumerate(qa_list):
    qa.to_parquet(qa_path=f'./qa_{i}.parquet', corpus_path=f'./corpus_{i}.parquet')
```

All task is done
I will do the documentation at this branch for the last time
@bwook00
Repository: Marker-Inc-Korea/AutoRAG
Issue: Failed to send a request to an API server deployed with AutoRAG
Root cause: Resource Mishandling

I got an error when sending request to my api server.
## Reproduce
1. Deploy API Server
```python
from autorag.deploy import Runner

runner = Runner.from_yaml("./pipeline.yaml")
runner.run_api_server()
```
2. Send Request
```bash
curl -X POST "http://localhost:8000/run" -H "accept: application/json" -H "Content-Type: application/json" -d "{\"query\":\"my question\"}"
```
3. Error
```bash
results = loop.run_until_complete(process_batch(tasks, batch_size=batch))
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "uvloop/loop.pyx", line 1511, in uvloop.loop.Loop.run_until_complete
  File "uvloop/loop.pyx", line 1504, in uvloop.loop.Loop.run_until_complete
  File "uvloop/loop.pyx", line 1377, in uvloop.loop.Loop.run_forever
  File "uvloop/loop.pyx", line 518, in uvloop.loop.Loop._run
RuntimeError: this event loop is already running.
```
## Environment
OS: Ubuntu 20.04
python: 3.11.9
AutoRAG: 0.2.13

Even I upgrade my AutoRAG with `pip install AutoRAG --upgrade`, I got an error.  
It seems already fixed this error, but I cannot get fixed version.

Hello, this bug fixed at #611 
But did not released at pypi yet.
You can install current main branch through pip like this.

```bash
pip install git+https://github.com/Marker-Inc-Korea/AutoRAG.git
```

Also, the new version will release soon in this week.

@rjwharry It will fix your error. If you couldn't,  let me know.

I tested it on my local and it worked! Thank you for your answer.

@rjwharry That sounds great! Thanks~
Repository: Marker-Inc-Korea/AutoRAG
Issue: Add Generator Module (Llama Index LLM module)
Root cause: Unimplemented Feature Gap


I am going to build two modules. 
One, the basic llm module that do not return log probs and each tokens. This is #62 issue.
Two, the log prob module that do return log probs and each tokens. This is #63 issue.
The basic llm module will return pseudo log probs, so it is not recommended to use at log prob-dependent modules, which will be implemented soon.
Repository: Marker-Inc-Korea/AutoRAG
Issue: [Feature Request] Add pylint or styling guide for consistent code format (with contributors)
GitHub labels: enhancement

**Is your feature request related to a problem? Please describe.**
The maintainers have to reformat again when there are some contributors because we use Pycharm code reformatting. 

**Describe the solution you'd like**
Consider to use [ruff](https://github.com/astral-sh/ruff).
Super fast rust-based python linter. 

**Describe alternatives you've considered**
We can use `black` as well.
Repository: Marker-Inc-Korea/AutoRAG
Issue: [Feature Request] Refactor CONTRIBUTING guide
GitHub labels: documentation

enhancement

**Is your feature request related to a problem? Please describe.**
There is no contributing guide. Need to make one of it.

**Describe the solution you'd like**
I will write it

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

**Additional context**
Add any other context or screenshots about the feature request here.


close at #625
Repository: Marker-Inc-Korea/AutoRAG
Issue: Add Generator module that supports log probs and tokens. 
Root cause: Unimplemented Feature Gap

Generator Node designed to return log probs and tokens as default, but as we can't get these informations at llama index, we had to make log probs llm module for this feature.


Now `vllm` supports proper logprobs and output tokens. 


We have `vllm` and `openai_llm` shich supports token and log probs output
Repository: Marker-Inc-Korea/AutoRAG
Issue: [trivial issue] typo in the document
GitHub labels: documentation

> > @vkehfdl1 
> > There is another very trivial typo in https://docs.auto-rag.com/data_creation/tutorial.html.
> > 
> > it says `# It have to contain 'query' and 'generation_gt' columns.` but `it has to contain` instead of `it have to contain...` ...
> > Idk. tbh, it is very remarkable typo to me. 😅
> > 
> 
> 
> 
> 
> 
> 
> _Originally posted by @skmanzg in https://github.com/Marker-Inc-Korea/AutoRAG/issues/629#issuecomment-2301198157_

            
I thought it was too trivial so I left the comment in the other's PR but I made a issue by the request from @vkehfdl1 .
Since I have no right to fix the typo in the document (ofc!), I leave the issue here only.
            

@skmanzg Thanks 👍
Repository: Marker-Inc-Korea/AutoRAG
Issue: [Feature Request] Retrieval results in deployment results
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
I have generated an evluation set, run Autorag to find best pipeline, great stuff. I would like to use it, and can run the CLI, API and web app. However, these all only seem to reply with the generated text, in many cases it's good to see the retrieval results also to eyeball how well generation is going, but many RAG systems present these to users.

**Describe the solution you'd like**
It would be great if the CLI and API also returned retrieval results.

**Describe alternatives you've considered**


**Additional context**
Thanks!


Little bit duplicate with #493

Anyway @dividor Thanks for your interest and making an issue at AutoRAG.

I have a question. Do you want `retrieved_contents` at both API server and web???
If so, I think it will be better to change this issue for API server, and for web use #493 issue. Is it okay?

Hi! Sorry for the delay. 

API would be fine, thanks.
Repository: Marker-Inc-Korea/AutoRAG
Issue: [BUG] Fix incorrect usage of TokenTextSplitter at data_creation/tutorial.md docs
Root cause: Documentation Deficiency

**Describe the bug**
Incorrect usage of TokenTextSplitter at data_creation/tutorial.md docs.
```
nodes = TokenTextSplitter().get_nodes_from_documents(documents=documents, chunk_size=512, chunk_overlap=128)
```
The LlamaIndex docs says the chunk_size and chunk_overlap must be in init of instance.


[LlamaIndex Docs](https://docs.llamaindex.ai/en/stable/api_reference/node_parsers/token_text_splitter/)
Repository: Marker-Inc-Korea/AutoRAG
Issue: [Feature Request] Delete nest_asyncio dependency at library (FastAPI server execution)
Root cause: Architectural Constraint

**Is your feature request related to a problem? Please describe.**
Using `nest_asyncio` actually breaks the design of the `asyncio`. But we had to use it because of the FastAPI server.
In this issue, we have to figure out how to use FastAPI server and AutoRAG without using `nest_asyncio` since it is not intended usage of `asyncio` [Relevant Link](https://stackoverflow.com/questions/59740704/correct-use-constraints-of-use-of-nest-asyncio)

**Describe the solution you'd like**
A clear and concise description of what you want to happen.

**Describe alternatives you've considered**
We can just use it. Actually this issue is not that serious yet.

**Additional context**
Add any other context or screenshots about the feature request here.


@vkehfdl1  How about using Flask instead of FastAPI?  
We have to use ASGI(uvicorn) to run FastAPI, but cannot avoid running event loop.  
So my suggestion is using Flask instead. As you know, Flask is WSGI Application and it will not cause conflict of event loop.  
I tested on my local, and it works properly without nest_asyncio. I can make draft PR for you

@rjwharry 
Great idea! It will be great you can make PR for it.
I will assign to you in this issue. 
Thank you very much
Repository: Marker-Inc-Korea/AutoRAG
Issue: [Feature Request] Update LlamaIndex version up to 0.11
Root cause: External Dependency Breakage

**Is your feature request related to a problem? Please describe.**
There was major update of LlamaIndex up to 0.11 [Link](https://www.llamaindex.ai/blog/introducing-llamaindex-0-11)

**Describe the solution you'd like**
First update the LlamaIndex version at `requirements.txt` and fix all errors that occured.

**Describe alternatives you've considered**
Constraint the version less than 0.11

**Additional context**
Add any other context or screenshots about the feature request here.
Repository: Marker-Inc-Korea/AutoRAG
Issue: [BUG] Failed to parse yaml when tag exist
Root cause: Configuration Error

**Describe the bug**
Failed to parse yaml when `!!python/tuple` tag exist

**To Reproduce**
Steps to reproduce the behavior:
1. Prepare pipeline.yaml including `!!python/tuple`
2. Run `Runner.from_yaml()`

**Expected behavior**
Make runner instance successfully

**Full Error log**
If applicable, add full error log to help explain your problem.
```bash
ConstructorError: could not determine a constructor for the tag 'tag:yaml.org,2002:python/tuple'
```
**Code that bug is happened**
If applicable, add the code that bug is happened.
```yaml
node_lines:
- node_line_name: retrieve_node_line
  nodes:
  - modules:
    - module_type: hybrid_cc
      normalize_method: mm
      target_module_params: !!python/tuple
      - embedding_batch: 256
        embedding_model: openai
        top_k: 3
      - bm25_tokenizer: porter_stemmer
        top_k: 3
      target_modules: !!python/tuple
      - vectordb
      - bm25
      top_k: 3
      weight: 0.47000000000000003
      ...
```
```python
runner = Runner.from_yaml("./1/pipeline.yaml")
```

**Desktop (please complete the following information):**
 - OS: Linux (Ubuntu 20.04)
 - Python version: 3.11


According to this [link](https://death.andgravity.com/yaml-unknown-tag), we need to change `yaml.safe_load` to `yaml.full_load`. It works on my local. 
I'll make PR for this.

@rjwharry 
I have one question.
Why do you need use `!!python/tuple` at `target_module_params`?? 

@vkehfdl1 
It was automatically generated when extracting pipeline.yaml, so I thought that AutoRAG was adding it.

@rjwharry Oh really? That can be an issue. We'll check about it.

I check that pyyaml add tag automatically.
```python
import yaml

test = {
    "test": (1, 2)
}
print(yaml.dump(test))
```
```bash
# Result
test: !!python/tuple
- 1
- 2
```
My pyyaml version is 6.0.1

@vkehfdl1 
How about using `yaml.safe_dump()` instead of `yaml.dump()`?
```python
import yaml

test = {
    "test": (1, 2)
}
print(yaml.safe_dump(test))
```
```bash
# Result
test:
- 1
- 2
```
And I also tested modifying dump to safe_dump in `extract_best_config` function and it works fine.
Repository: Marker-Inc-Korea/AutoRAG
Issue: add metric and reform style
Root cause: Unimplemented Feature Gap

**Insufficient metric**
There are various metrics available for RAG.
These include projects such as deepeval, or Galileo.
With the addition of that metric, more diverse metrics should be accommodated in AutoRAG.

**Describe the solution you'd like**
- make rag subject schema for input of metric
- every metric function use this schema as input.

**Describe alternatives you've considered**
- [x] #649
- [x] #655
## gallileo metrics
- [ ] #650
- [ ] add gallileo_sexism
- [ ] add gallileo_tone
- [ ] add gallileo_toxicity
- [ ] add gallileo_private_identifiable_information
- [ ] add gallileo_context_relevance
- [ ] add gallileo_prompt_perplexity
- [ ] add gallileo_uncertainty
- [ ] add gallileo_chunk_utilization
- [ ] add gallileo_chunk_attribution
- [ ] add gallileo_completeness
- [ ] add gallileo_correctness
- [ ] add gallileo_context_adherence
- [ ] add gallileo_instruction_adherence
- [ ] add gallileo_chunk_relevance
## MLflow
- [ ] add mlflow_relevance
- [ ] add mlflow_faithfulness
- [ ] add mlflow_answer_relevance
- [ ] add mlflow_answer_similarity
- [ ] add mlflow_anwer_correctness
## deepeval metrics
- [ ] #654
- [ ] #752
- [x] #753
- [ ] #754
- [ ] #755
- [ ] #756
- [ ] #757
- [ ] #758
- [ ] #759
## trulens
- [ ] add trulens_answer relevancy
- [ ] add trulens_context_relevancy
- [ ] add trulens_groundedness
- [ ] add trulens_coherence
- [ ] add trulens_comprehensiveness
- [ ] add trulens_conciseness
- [ ] add trulens_controversialty
- [ ] add trulens_correctness
- [ ] add trulens_criminality
- [ ] add trulens_generate_score
- [ ] add trulens_groundedness
- [ ] add trulens_harmfulness
- [ ] add trulens_helpfulness
- [ ] add trulens_insensitivity
- [ ] add trulens_maliciousness
- [ ] add trulens_misogyny
- [ ] add trulens_relevance
- [ ] add trulens_sentiment
- [ ] add trulens_strereotypes
- [ ] add trulens_summarization
## openai
- [ ] add openai_hate
- [ ] add openai_harassment
- [ ] add openai_self-harm
- [ ] add openai_sexual
- [ ] add openai_violence



Wow there's a lot of metric 👍
Repository: Marker-Inc-Korea/AutoRAG
Issue: [BUG] HuggingfaceLLM is not loaded
Root cause: Interface Contract Mismatch

## Issue
When using a model from Huggingface, the repo ID is not correctly recognized by AutoRAG, so the model fails to load.

## reproduce
Run `https://github.com/Marker-Inc-Korea/AutoRAG-tutorial-ko` with any Huggingface model.
```
                             │ /opt/titan/conda/envs/autorag/lib/python3.10/site-packages/llama_index/llms/huggingface/base. │               
                             │ py:253 in __init__                                                                            │               
                             │                                                                                               │               
                             │    250 │   │   print("self._model", self._model)                                              │               
                             │    251 │   │                                                                                  │               
                             │    252 │   │   # check context_window                                                         │               
                             │ ❱  253 │   │   config_dict = self._model.config.to_dict()                                     │               
                             │    254 │   │   model_context_window = int(                                                    │               
                             │    255 │   │   │   config_dict.get("max_position_embeddings", context_window)                 │               
                             │    256 │   │   )                                                                              │               
                             ╰───────────────────────────────────────────────────────────────────────────────────────────────╯               
                             AttributeError: 'str' object has no attribute 'config'           
```


## Cause analysis
- `HuggingFaceLLM` class from `llama_index` requires either `model` or `model_name`, and the repo ID (a string) must be passed to HuggingFaceLLM via `model_name`, not `model`.
- The current code fails to load the model because it uses `model` as the repo ID (it works okay for openai as it uses `model` for the same purpose.)


## pr
https://github.com/Marker-Inc-Korea/AutoRAG/pull/652
Repository: Marker-Inc-Korea/AutoRAG
Issue: [Feature Request] Add Dockerfile for running AutoRAG
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
Because of the parsing modules, it is become much harder to install everything right in the environment.
Especially, because of the `pySSL`, `poppler-utils`, and `tesseract-ocr`. 
It looks much better to use different container to work on it smoothly. 

**Describe the solution you'd like**
- Write a Dockerfile for this.
- **MUST RUN WHOLE TEST IN THE DOCKERFILE**
- Use Docker image at github actions also. In this scenario, we have to find out how automatically build, how deploy to github registry, and pull it to test actions. => This better be extra issue.

**Describe alternatives you've considered**
Maybe make .sh file for installation. But we have virtual env actually.


Possible deploy to the ECS

@hongsw will assign this issue

https://github.com/Marker-Inc-Korea/AutoRAG/pull/763
Repository: Marker-Inc-Korea/AutoRAG
Issue: [Bug] Add guide for setup bm25_tokenizer in strategy of query_expansion
Root cause: Documentation Deficiency

**Is your feature request related to a problem? Please describe.**
The user must set the value of bm25_tokenizer in startegy on the query_extend node to be the same as the bm25_tokenizer in bm25 on the retrieval node.    
If the bm25_tokenizer in retrieval node is missing any of the values in bm25_tokenizer in the query_extension node, an error is raised with: `bm25_path {bm25_path} does not exist. Please ingest first.`.    
Here is an example
```yaml
- node_line_name: pre_retrieve_node_line 
  nodes:
    - node_type: query_expansion
      strategy:
        metrics: [retrieval_f1, retrieval_recall, retrieval_precision]
        top_k: 5
        retrieval_modules:
          - module_type: bm25
            bm25_tokenizer: [ porter_stemmer, huggingface/new_model ]

- node_line_name: retrieve_node_line 
  nodes:
    - node_type: retrieval
      modules:
        - module_type: bm25
          bm25_tokenizer: [ porter_stemmer ]
```
There is no guidance or notice about this in the documentation, so I recommend adding a cautionary note about this situation.

**Describe the solution you'd like**
There is no guide or notice about this in document. So it would be better to add guide the caution for this situation.


It seems not intended. 
I will look what is wrong.

If it is not intended, then I think we should ingest even in strategy of query_expansion too.

If we evaluate query expansion based on retriever, then I think it has to be same as modules of retrieval node_type

@rjwharry 
Yes we need to check what tokenizer or embedding model used in query expansion at the start of the trial.
Sorry for late bug fix.
We are developing AutoRAG 0.3 version, so it might be delayed.

Okay, then let me work on it.
Repository: Marker-Inc-Korea/AutoRAG
Issue: [Structure Refactoring] Refactor for efficient deploying
Root cause: Architectural Constraint

**Is your feature request related to a problem? Please describe.**
The current deployment is so inefficient, because it loads each model and delete it at inference. 
It is good at optimization process, but not good at inference stage.

**Describe the solution you'd like**
![image](https://github.com/user-attachments/assets/e5954183-e4d8-4153-9112-3dcc6584e140)


## Refactor Modules - each including run.py
- [x] [Structure Refactoring] Retrieval
- [x] [Structure Refactoring] Query Expansion
- [x] [Structure Refactoring] Passage Reranker
- [x] [Structure Refactoring] Passage Augmenter
- [x] [Structure Refactoring] Passage Filter
- [x] [Structure Refactoring] Generator
- [x] [Structure Refactoring] Prompt Maker
- [x] [Structure Refactoring] Passage Compressor
- [x] #728
- [x] #729

## Extra
- [x] Refactoring Evaluator 
- [x] Refactoring Runner

## Test
- [x] Testing and resolve errors
- [x] Full run and real version test
- [x] Documentation
Repository: Marker-Inc-Korea/AutoRAG
Issue: [Data Creation Refactoring] Merge multiple parsing results to one Raw. Load Raw instance to Chunker.
Root cause: Unimplemented Feature Gap

1. Raw needs some kind of merge with another Raw. Because sometimes there will be a multiple parsed result and wants to use at the same RAG pipeline. 
So make something like `Raw` plus override.
2. Use `Raw` instance at `Chunker` `__init__()` and load from file with new classmethod `from_parquet`
Repository: Marker-Inc-Korea/AutoRAG
Issue: [Data Creation] [Feature Request] Add passage dependency filter
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
Add passage dependency filter for filtering passage-dependent questions

**Describe the solution you'd like**
Use LLM. Both openai and LlamaIndex

**Describe alternatives you've considered**
Actually it is hard to get high accuracy at this task. But we have to do something.
Someday we can make custom classification model for this.

**Additional context**
Add any other context or screenshots about the feature request here.
Repository: Marker-Inc-Korea/AutoRAG
Issue: [BUG] Does it work in a multi-GPU environment?
Root cause: Unimplemented Feature Gap

**Describe the bug**
When I try to infer llm locally, only the 0th GPU is used. Is there a guide to using multi GPU?


@daegonYu 
Hello.
We do not have multi-GPU environment yet, so we couldn't experiment multi-GPU environment yet.

@daegonYu 
Hi
Fortunately we get multi-gpu server and investigating this issue

Okay, thank you.

Not resolve this issue yet