zhiminy/EvalEng huggingface dataset:
Repository: Marker-Inc-Korea/AutoRAG
Issue: [HotFix] Add CNAME file for custom docs domain


To use a custom domain with GitHub Pages and ensure that your custom domain settings are preserved during deployment, you need to include a `CNAME` file in the `docs/build/html` directory. This file should contain your custom domain name.

Here's how you can modify your GitHub Actions workflow to include the `CNAME` file:

1. **Create a `CNAME` file**: Add a `CNAME` file in your `docs/source` directory (or wherever your source files are) with your custom domain name. For example, if your custom domain is `www.example.com`, the `CNAME` file should contain:

    ```
    www.example.com
    ```

2. **Ensure the `CNAME` file is copied to the build directory**: Modify your `sphinx-build` command to ensure the `CNAME` file is included in the `docs/build/html` directory.

3. **Update your GitHub Actions workflow**: Add a step to copy the `CNAME` file to the `docs/build/html` directory before deploying.

Here's the updated YAML file:

```yaml
name: Sphinx build

on:
  push:
    branches:
      - main

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4
    - name: Upgrade pip
      run: | 
        python -m pip install --upgrade pip
    - name: Install gcc
      run: |
        sudo apt-get install gcc
    - name: Install dependencies
      run: |
        pip install -r docs/requirements.txt
    - name: Build HTML
      run: |
        sphinx-build -b html docs/source docs/build/html
    - name: Copy CNAME
      run: |
        cp docs/source/CNAME docs/build/html/CNAME
    - name: Upload artifacts
      uses: actions/upload-artifact@v3
      with:
        name: html-docs
        path: docs/build/html/
    - name: Deploy
      uses: peaceiris/actions-gh-pages@v3
      if: github.ref == 'refs/heads/main'
      with:
        github_token: ${{ secrets.GITHUB_TOKEN }}
        publish_dir: docs/build/html
```

### Explanation:

- **Create a `CNAME` file**: Ensure you have a `CNAME` file in your `docs/source` directory with your custom domain name.
- **Copy the `CNAME` file**: The `Copy CNAME` step ensures that the `CNAME` file is included in the `docs/build/html` directory.
- **Deploy**: The `peaceiris/actions-gh-pages@v3` action will now deploy the contents of `docs/build/html` to GitHub Pages, including the `CNAME` file, which will preserve your custom domain settings.

This should fix the issue of your URL reverting to the `github.io` domain.

Citations:
Repository: Marker-Inc-Korea/AutoRAG
Issue: Auto-test at python 3.9~3.12
Root cause: Unimplemented Feature Gap

Use tool like [tox](https://developer.nvidia.com/ko-kr/blog/how-to-manage-virtual-environments-and-automate-testing-with-tox/)

Add it to workflow.
Maybe run when 'merge' to main??

https://github.com/tox-dev/tox-gh
Repository: Marker-Inc-Korea/AutoRAG
Issue: Circular import on calculate_cosine_similarity
Root cause: Architectural Constraint

![image](https://github.com/Marker-Inc-Korea/AutoRAG/assets/28955029/6cacaa90-c88b-44cc-a1b7-af87ecf25388)


This happens duplicate package name of huggingface evaluate and autorag.evaluate.
It occurs confusion between two packages.
Need to change our name to other.
Repository: Marker-Inc-Korea/AutoRAG
Issue: Add implementation of new pytorch acceleration 
Root cause: Documentation Deficiency

Also, may be a good add to the docs - I believe the new pytorch implementations with M-series chipsets have made the LLM steps wildly faster than originally experienced on these machines. Didn't bother to try it at first given one of the notes I read about CUDA optimized runs (especially for flag passage reranking, etc.), and remoted into a Windows WSL2 instance to run it. Turns out, x3 times faster on my M3 Macbook (no memory pinning to contend with there too).

Might be a nice s/o to that userbase if they're still using workarounds for their eval

It is okay to leave a comment at docs.
Repository: Marker-Inc-Korea/AutoRAG
Issue: Please kindly be noticed that you have to specify the version of `numpy`
Root cause: External Dependency Breakage

### Error

```
AttributeError: `np.float_` was removed in the NumPy 2.0 release. Use `np.float64` instead.                                                                                                                                       
```

### Fix

```
numpy<2.0.0
```

We will make it numpy 2.0.0 compatible soon

At #522
Repository: Marker-Inc-Korea/AutoRAG
Issue: Can't load YAML file contains Korean at Windows environment
Root cause: Environment Incompatibility

 ```C:\Users\김동규\PycharmProjects\AutoRAG\autorag\evaluator.py:189 in _load_node_lines                                                                                                                                                                                                                                                 │ 
                             │                                                                                                                                                                                                                                                                                                                                      │ 
                             │   186 │   │   │   raise ValueError(f"YAML file {yaml_path} does not exist.")                                                                                                                                                                                                                                                         │ 
                             │   187 │   │   with open(yaml_path, 'r') as stream:                                                                                                                                                                                                                                                                                   │ 
                             │   188 │   │   │   try:                                                                                                                                                                                                                                                                                                               │ 
                             │ ❱ 189 │   │   │   │   yaml_dict = yaml.safe_load(stream)                                                                                                                                                                                                                                                                             │ 
                             │   190 │   │   │   except yaml.YAMLError as exc:                                                                                                                                                                                                                                                                                      │ 
                             │   191 │   │   │   │   raise ValueError(f"YAML file {yaml_path} could not be loaded.") from exc 
```

 UnicodeDecodeError: 'cp949' codec can't decode byte 0xeb in position 650: illegal multibyte sequence
Repository: Marker-Inc-Korea/AutoRAG
Issue: Allow for variations of metrics
Root cause: Unimplemented Feature Gap

currently, AutoRAG system not allow variations of each metric in once experiments.
examples:
Rouge - need to choice between Rouge-1, Rouge-2, Rouge-L, Rouge-sum (supported from RougeScorer)
BLEU - each "n"  gram
sem score - if user want to compare between many model embedding?

How should we do?

Let's break down to the smaller issues. 
Since we can specify about metric kwargs at YAML, we can make it all work in AutoRAG

Hey, i would like to contribute to this, how do I get started?

@tenseisoham Hello! 
First, check out our `CONTRIBUTING.md` file. It explains the dev setups. like pre-commit or testing.
Then, you can find the generation metric at `autorag/evaluation/metric/generation.py`.

You can see, the Rouge already supports each Rouge-1,  Rouge-2, Rouge-L, Rouge-sum.
We can give BLEU options from the sacrebleu sentence score. But I am not sure about “n” gram. @Eastsidegunn Please check BLEU.


Plus, `sem score - if user want to compare between many model embedding?` This will be the problem, but have to change other part other than metric. It is bit tricky.

You can always allow to add new metric as well. 

We just need to resolve #264 to resolve this issue.
Repository: Marker-Inc-Korea/AutoRAG
Issue: Redesign hybrid-cc related retrievalmodule
Root cause: Unimplemented Feature Gap

The current hybrid module have some problems. It is all hybrid cc (convex combination) related modules except RRF. 
I think we need to change and re-design a whole module. 
Here is my proposed design.

=> [related paper](https://arxiv.org/pdf/2210.11934)

## 1. Make hybrid_cc module to one and get an 'normalize' parameter.

All hybrid_cc related module have different only at the normalization process. If you select different normalization method, the result will be slightly different. Keep this as parameter. 

We must support below normalization method.

- mm : Min-Max normalization
- tmm : Theoretical min-max normalization
- z : z-score normalization
- dbsf : 3-sigma normalization

## 2. Test **many** weights at once

We get semantic and lexical scores from previous run. So, fusion takes so little time to compute. But, fusion quality differs so much , up to 0.2 in NDCG.
So, it if crucial to do experiment on the many weights. 
It will be great we set the weights quantity and do experiment on that. 
Like, setting it as 100, we will experiment each weights from 0.0 to 1.0 a hundred times. 
![image](https://github.com/Marker-Inc-Korea/AutoRAG/assets/28955029/4dfed8c4-6f92-4693-8a01-7421c159e690)

The actual weight optimization will be in the module execution. As result, you will only see the result of the optimization. Remain as you set the weight hyperparameter. Let me think how to save the great weight parameter at the process.

## 3. Specify which module is lexical and which module is semantic.

We have to specify it since we will implement tmm normalization. 
We have to test the name of the retrieval module, and set the theoretical minimum. 
This way, it will much clearer.

## 4. Set min and max value to the both 


5. Do not let each retrieval have missing document score.
Repository: Marker-Inc-Korea/AutoRAG
Issue: Refactor documentation for better understanding
GitHub labels: documentation

Thanks for Filipp who leave kind comment 

---

Filipp Trigub — 2024.07.05. 오전 1:32
Hi @Jeffrey Kim, thanks for the quick response.

In a nutshell I would like a more complete docu. As a dev, I understand that this is not high priority at all (and annoying). 

As a user however, I am truly perplex, why this project is not much more widely known. Difficulties in setting it up and running it might be one explanation. Never underestimate the laziness of tech people 😄 

Some small examples I've encountered:
the prompt maker node https://docs.auto-rag.com/nodes/prompt_maker/prompt_maker.html lists generation modules as a setting. I believe it is optional, but that is not clear at all. Would wish for clear distinction between optional and mandatory settings.
in https://docs.auto-rag.com/data_creation/data_format.html it is not clearly indicated that the doc_id in corpus must be one of the ids in retriaval_gt. Is that not obvious? On it's own, probably, but when you are already dealing with all the other info, it might get lost. Happened to me.
in https://docs.auto-rag.com/optimization/optimization.html is is not clear, which nodes are mandatory and which ones are not (or maybe they all are and I am mistaken?).

Additionally, but not of any importance:
link at llama_index is outdated here https://docs.auto-rag.com/nodes/generator/llama_index_llm.html and here https://docs.auto-rag.com/local_model.html#supporting-llm-models

Hope that helps. I really think, people should use AutoRAG much more and the good thing is that it is actually easy to use, once you've understood the nuances! 

---

Let's fix this!
Repository: Marker-Inc-Korea/AutoRAG
Issue: [HotFix] Duplicate qid at make_single_content_qa
Root cause: Algorithmic Error

```
corpus_df = pd.read_parquet('./corpus_custom_splitter.parquet')
llm = OpenAI(model='gpt-4o', temperature=1.0)
qa_df = make_single_content_qa(corpus_df, content_size=500, qa_creation_func=generate_qa_llama_index,
                               llm=llm, prompt=prompt, question_num_per_content=2)
qa_df.to_parquet('./qa.parquet’)
```

In here, it makes duplicate qid.
Repository: Marker-Inc-Korea/AutoRAG
Issue: Save as cache file while the data generation
Root cause: Unimplemented Feature Gap

If there is an error, the whole data generation process is gone. It is super annoying, especially you want to make big dataset.

So, the feature that creates cache file while data generation is crucial. To save some data while the generation when there is a crucial error.


---

The initial feature request (by 호호) as Korean

```
그리고 커스텀하게 만들다가 중간에 에러가 나는 경우가 있어서 살펴보니, LLM (gpt-4o를 사용해도)에서 Q&A pair를 생성할 때
[Q]
[A]의 형태가 아니고
[Q1]
[A1]
의 형태로 답하는 경우가 가끔씩 있더라구요. (few-shot으로 프롬프트를 구성하면 경우가 거의 없긴합니다.)
그래서 답변을 파싱할 때 에러가 나서.. 생성 중이던 데이터가 모두 날아가는 경험을 한 적이 있습니다 ㅠ (1만여개 넘는 데이터 생성 중이었는데 다 없어졌습니다)

데이터를 로컬에 계속 저장해가면서 진행할 수 있으면, 어떠한 오류로 실패되더라도 데이터는 좀 살릴 수 있을 것 같아서 의견드립니다..!
```
Repository: Marker-Inc-Korea/AutoRAG
Issue: Get relevance score of all passages when execute hybrid retrieval 
Root cause: Algorithmic Error

When we run hybrid retrieval, the non-duplicated passage that do not have relevance score turns to be zero.
But that is not precise way to calculate hybrid retrieval score.

So, getting relevance score when the passage is not the top-k passage is essential for more precise hybrid retrieval.
Repository: Marker-Inc-Korea/AutoRAG
Issue: bm25_path .\resources\bm25_gpt2.pkl does not exist. Please ingest first.
Root cause: Validation Gap

ran through the tutorial using the simple_openai.yaml and it worked. I am trying to use the full.yaml and I am getting the following error

`bm25_path .\resources\bm25_gpt2.pkl does not exist. Please ingest first.`

I checked the resources folder and it does not in fact have the bm25_gpt2.pkl 
![image](https://github.com/user-attachments/assets/4973735a-4c9c-4c60-8846-feac2536be7d)

![image](https://github.com/user-attachments/assets/98db556d-ae00-4d52-a280-26fbcec8f9d6)


I have looked for resources in the github to manually download, but have not found any.

Any help on this error is greatly appreciated.

Hello @maglore9900 
You must create new project directory and specify it when run the evaluation.

use `--project_dir` parameter at cli. Or use `project_dir` parameter when run the trial(evaluation)

Appreciate the quick feedback. I am using the python script you provided, which you can see below

```
from autorag.evaluator import Evaluator
from dotenv import load_dotenv

load_dotenv()
evaluator = Evaluator(qa_data_path='corpus/qa.parquet', corpus_data_path='corpus/corpus.parquet')
evaluator.start_trial('full.yaml')
```

it failed when running the full.yaml, so I restarted it with 
```
from autorag.evaluator import Evaluator
from dotenv import load_dotenv

load_dotenv()

evaluator = Evaluator(qa_data_path='corpus/qa.parquet', corpus_data_path='corpus/corpus.parquet')
evaluator.restart_trial(trial_path='3')
```

and that results in the error provided above. 

so, where would the --project_dir go?

here? 
```
evaluator = Evaluator(qa_data_path='corpus/qa.parquet', corpus_data_path='corpus/corpus.parquet', project_dir='test_dir')
```



if that is the suggestion, it still failed in the same way.

![image](https://github.com/user-attachments/assets/8ef11153-fac1-4add-b615-ae7fa79ebe17)


![image](https://github.com/user-attachments/assets/687513bd-cad4-4239-bfdc-6d88f3096d6e)


I think you need to make a **new directory** as new project directory. And set it to the `project_dir` parameter. Can you try that again?

The issue is that the current code was not recognizing the bm25_porter_stemmer.pkl, even though its supposed to. The solution on my side was basically a hack where I commented out the detection code in the module and explicitly put in bm25_porter_stemmer.pkl. 

I was running autorag with python 3.10 on windows 11 from within visual studio code.

Hope that information allows you to correct the issue. 

Thanks @maglore9900 
It can be a Windows issue.
We'll look into this issue thanks.

It looks okay at mac.
Repository: Marker-Inc-Korea/AutoRAG
Issue: 안녕하세요 AutoRAG 테스트 중 Connection오류 생겨서 질문드립니다.
Root cause: Configuration Error

안녕하세요 현재 AutoRAG 0.2.10 버전을 사용중입니다.
![image](https://github.com/user-attachments/assets/b987b8b6-e983-4f53-bedf-d6b1a5aaf5fe)

현재 중간에 워닝이 한번 뜨고 이후에 에러가 찍히는데

![image](https://github.com/user-attachments/assets/a78fc89d-4cbd-4fa4-93fc-95479b4c0d79)

header가 잘못되었다고 오류가 뜹니다. 혹시 어떻게 해결해야 할까요??ㅠㅠ

config.yaml 은 compat_openai 사용중이고, 데이터는 load_eli5_dataset.py 로 실행해서 사용중입니다.,

ps. 0.2.10 버전에서 autorag/evaluation/metric/generation.py 에서 belu함수의 타입 힌트가 잘못된 거 같습니다.
![image](https://github.com/user-attachments/assets/0f020db7-2992-42a2-a373-eafa26aff7a5)


해결: .env 말고 export 로 api_key를 지정하면 동작한다!

답을 찾으셨네요! 
.env를 사용하려면 파이썬 파일에서 따로 load_dotenv()를 해주셔야 합니다^^
Repository: Marker-Inc-Korea/AutoRAG
Issue: "Access denied" when opening Colab Tutorial 2
GitHub labels: good first issue

The following link is not public : [Step 2: Create evaluation dataset](https://colab.research.google.com/drive/1HXjVHCLTaX7mkmZp3IKlEPt0B3jVeHvP#scrollTo=cgFUCuaUZvTr)

@baptiste-pasquier Thanks for the report. We'll check

@bwook00 please check this

We added tutorial three and two also
Repository: Marker-Inc-Korea/AutoRAG
Issue: Clear cache after evaluating LM-based generation metric
Root cause: Resource Mishandling

At small VRAM gpu like titan xp or gtx 1080 ti (12GB, 11GB of VRAM). The OOM error occured because of bert_score.

BLEU, ROUGE, and METEOR is n-gram based, so it doesn’t matter.
Repository: Marker-Inc-Korea/AutoRAG
Issue: Upgrade cohere reranker to latest version
Root cause: External Dependency Breakage

Since cohere 5.7.0, it is not compatible with this code.
Upgrade to the 5.7.0 for compatibility...

5.7.0a0 is alpha release.
5.7.0 have major difference between the last version, so it is alpha version I think.
Keep in mind this issue and after the cohere library officially released, we have to patch this.

5.8.0 released.
I’ll check it again
Repository: Marker-Inc-Korea/AutoRAG
Issue: Add test run feature for validating data and YAML file
Root cause: Unimplemented Feature Gap

Until run all nodes and modules, you can't know the YAML file, data, or GPU setup will work fine. 
So, I think we need system to validate these with minimum cost. 

<Features>
- cli, Evaluator function to validate
- temp folder for project directory. Temp qa.parquet and corpus.parquet that have little subset of original qa and corpus dataset.
-
Repository: Marker-Inc-Korea/AutoRAG
Issue: I got semantic retrieval module running log when I don't use it.
Root cause: Algorithmic Error

I used only bm25 at retrieval node, but i got semantic retrieval module running log.

![image](https://github.com/user-attachments/assets/b47e4bef-9533-4fe8-86df-e05d42e19ee8)

used yaml file (upload only retrieve_node_line!)

```
node_lines:
  - node_line_name: retrieve_node_line
    nodes:
      - node_type: retrieval
        strategy:
          metrics: [ retrieval_f1 ]
        top_k: 5
        modules:
          - module_type: bm25
            bm25_tokenizer: [ ko_kiwi ]
      - node_type: passage_reranker
        strategy:
          metrics: [ retrieval_f1 ]
        top_k: 3
        modules:
          - module_type: koreranker
```

I think this is not happening more because we refactored whole structure since v0.3
Repository: Marker-Inc-Korea/AutoRAG
Issue: InterruptedError: Failed to generate output of length 1 after 3 retries.
Root cause: Validation Gap

Dear all 

When i make a qa.parquet. 
I set to "--qa_size" as 5. This condition is properly working. 
But, I changed the size as 10. 
I faced the error:

InterruptedError: Failed to generate output of length 1 after 3 retries.

Please give me your hands. 

Thks. 

Wonjoong Cheon

@wjcheon Hello!

I think you used too large number of qa_size.
The `qa_size` that you mentioned means, "I want to generate {qa_size} questions in the **one** chunk"
So, it is possible that 10 questions can't be generated in only **one chunk**. Even human generated it.  

Thus, you have to lower the size of it. 

Oh, hello Donggyu-Kim,

'qa_df = make_single_content_qa(corpus_df, content_size=50, qa_creation_func=generate_qa_llama_index_by_ratio,
llm=llm, prompts_ratio=ratio_dict, question_num_per_content=1, batch=6)'

Then this example wouldn't be appropriate for the homepage, right?

I'm curious about another one. My corpus data has 308 rows. 
If I set qa_size to 3, it seems like 308*3 QnA data should be created, but only 3 are generated.

 Why is this happening? I'm also using the make_single_content_qa function.

@vkehfdl1  

I tested using the example of newjeans. 
I think the "qa_size" mean the number of question what i want to generate. 
Isnt it? 

@wjcheon 

You mean this file at our AutoRAG tutorial, right?
https://github.com/Marker-Inc-Korea/AutoRAG-tutorial-ko/blob/main/make_qa.py

You are right. In that file, the `qa_size` is the number of question. 
Please let me know which file or which code you are using. Because I can't get it right when you don't specify what code you are using.

A. You need to make issue at tutorial-ko repo, since this is not the `AutoRAG` bug. 
B. I think the LLM couldn't generate the question somehow. 
The first thing you can do is check the corpus chunks. It can be contain empty chunk or too small chunk. If the chunk too small, it is hard to generate question.
The second thing you can try is to increase retries. You can use parameter `max_retries` (int type)

The error occurs when the LLM couldn't generate question properly even three retries.
Since this is not the error at the codebase, I am closing this issue.

Thanks.
Repository: Marker-Inc-Korea/AutoRAG
Issue: ERROR with Event Loop
Root cause: Resource Mishandling

Hello. I wrote a small RAG system for ma project and I decided to push it on localhost

```
host = "localhost"
runner = Runner.from_yaml('RAG_system/data_original/test_1/pipline.yaml', project_dir = "RAG_system/data_original/test_1")
runner.run_api_server(host)

```
My code for posting 

```
import requests 
url = "http://localhost:8000/run"
data = {
    "query": "query",
    "result_column": "Answer"
}
import json```
payload = json.dumps(data)
headers = {
    'accept': "application/json",
    'Content-Type': "application/json"
    }
response = requests.request("POST", url, data=payload, headers = headers)
print(response.text)
```

And every time I get this error  
### RuntimeError: This event loop is already running
How do I solve it?

@warjohn Hi, did you run this code in jupyter?

Try this

```python
import nest_asyncio
nest_asyncio.apply()
```

it doen't work on linux, idk

@warjohn 
I am investigating this problem. 
Thanks for reporting the issue. 

@warjohn 
I made PR to resolve this error. #611 
Thanks for reporting a but for me again.
This fix will be update on AutoRAG v0.2.14

> [@warjohn](https://github.com/warjohn) Hi, did you run this code in jupyter?
> 
> Try this
> 
> import nest_asyncio
> nest_asyncio.apply()

Thanks ![Image](https://github.com/user-attachments/assets/4147137f-01ec-4c8a-9290-4845bb4f40ac)