zhiminy/EvalEng huggingface dataset:
Repository: Marker-Inc-Korea/AutoRAG
Issue: [BUG] Possible bug at retrieval module when top-k 1
Root cause: Algorithmic Error

**Describe the bug**
When benchmarking korean embedding model, there are error that top-k 1 and top-k 3 result is inconsistent.
![image](https://github.com/user-attachments/assets/b4cd1ad3-c8af-4d40-8cc2-98ed70e114c3)


**To Reproduce**
Steps to reproduce the behavior:
https://github.com/Marker-Inc-Korea/AutoRAG-example-korean-embedding-benchmark
Go here and follow the instructions
Used main branch(commit=78966cc198a5b502abacaf1a3ea1e606a0209539)

**Expected behavior**
The top-k 1 result must be same with the top-k 3 results top retrieval_id

**Full Error log**
If applicable, add full error log to help explain your problem.

**Code that bug is happened**
If applicable, add the code that bug is happened.
(Especially, your AutoRAG YAML file or python codes that you wrote)

**Desktop (please complete the following information):**
 - OS: [e.g. Windows, Linux, MacOS]
 - Python version [e.g. 3.10]

**Additional context**
This issue posted in AutoRAG Discord

```
안녕하세요~! 유튜브와 세미나를 참고해서 간단하게 지표를 뽑아볼 수 있었습니다. 감사합니다. 결과 페이지에서 MRR 및 MAP 지표와 같이 순서와 연관된 지표에 대해 여쭈어보고싶은 점이 있는데요.
K=1 인 경우, retrieved_ids 배열의 길이는 1이고,
K=3 인 경우, retrieved_ids 배열의 길이는 3이였습니다.
벡터 공간에서 현재 Query와 매칭되는 vector들 중 가장 유사한 Top K개의 결과를 검색해오는 것으로 이해했습니다.

그런데, K=3일 때, 검색 결과 리스트에 나열 된 순서가 정렬이 되지 않은 것을 확인했습니다.
동일 임베딩 모델에 대해 K=1일 때 가져온 Top 1 문서 'A'를 확인했는데, K=3일 때 가져온 Top 3 문서는 ['C', 'B', 'A'] 와 같이 기존에 가장 최상위에 위치한 Top 1 문서 'A'가 리스트의 가장 처음(index = 0)이 아니라 끝(index = -1)에 위치해있는 것을 확인할 수 있었습니다.

이러한 경우, 순서에 따른 가중치를 부여하는 MRR 및 MAP 지표에서 gt에 해당하는 문서 'A'가 리스트의 가장 맨 끝에 있기 때문에 가중치를 제대로 못받으며 계산이 되는 것 같은데요.
이렇게 리스트 내의 검색 된 문서가 유사도 순으로 정렬이 되지 않은 경우, MRR 및 MAP 처럼 순서를 고려하는 지표는 의미가 없어진다고 이해해도 괜찮을까요?
Jeffrey Kim — 어제 오후 9:53
혹시 어떤 모듈 사용하셨을때 정렬이 안되어 있었을까요? 확인 부탁드려요.
또한 retrieve_scores 컬럼도 결과에서 확인이 가능하신가요? 해당 점수도 정렬이 되지 않았는지 확인 부탁드립니다
김기훈_PC
작성자
 — 어제 오후 10:10
https://github.com/Marker-Inc-Korea/AutoRAG-example-korean-embedding-benchmark
샘플 프로젝트 그대로(config도) 사용하면서 데이터만 제가 가진 데이터로 바꿔서 해봤습니다!

첨부해드린 그림에서 확인하실 수 있듯이, K=3일 때 retrieve_scores 기준으로 정렬은 내림차순으로 잘 되어있습니다.
흠.. 그렇다면 K=1일 때의 retrieve_scores 컬럼의 값이 최대인 문서가 조회 되지 않은 것인가요?
```


I confirm that there was an error.
Non-ranking method does not effected, but rank aware methods and passage filter like threshold passage filter.


close at #713
Repository: Marker-Inc-Korea/AutoRAG
Issue: [Feature Request] Add citation paragraph at README.md 
GitHub labels: documentation

enhancement

**Is your feature request related to a problem? Please describe.**
Some researchers want to add AutoRAG to their own papers.
It will be great we add citation link for the researchers. 

https://arxiv.org/pdf/2406.19251

In this paper there is citation for AutoRAG repo
Repository: Marker-Inc-Korea/AutoRAG
Issue: [Feature Request] Compatibility with latest OpenAI gpt-4 o1 model.
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
We have new gpt-4 o1 model and have to use it.

**Describe the solution you'd like**
We need to check compatability at llama_index_llm and openai_llm modules.
Repository: Marker-Inc-Korea/AutoRAG
Issue: [Feature Request] AutoRAG install takes a long time because of botocore
Root cause: Environment Incompatibility

![image](https://github.com/user-attachments/assets/e1ec3bcd-7c2e-4c7d-a860-d28073e0819a)


Python version == 3.9

There was no error like this up to python 3.10
Repository: Marker-Inc-Korea/AutoRAG
Issue: [Data Creation] [Feature Request] Extract Evidence from initial QA data retrieval gt
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
When mapping new corpus, it is better to extract exact evidence paragraph from the passage.
So, it needs extra extraction from the retrieval gt passage

**Describe the solution you'd like**
We will use LLM to extract exact evidence sentence from the retrieval gt passage and mapping it to the `evidence_start_end_idx`

**Describe alternatives you've considered**
From now, you can use `update_corpus` without `extract_evidence` method
Repository: Marker-Inc-Korea/AutoRAG
Issue: [QA Creation]If QA Data already exists, handle cases where answer(generation_gt) is Unanswerable in Corpus
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
If QA Data already exists, handle cases where answer(generation_gt) is Unanswerable in Corpus

**Describe the solution you'd like**
If QA Data already exists, implement a feature to check if it is answerable with data in the corpus


Since unanswerable questions don't have retrieval_gt, I think we should delete them until we deal with unanswerable questions.

I think we should check if it is answerable based on the "Question", whether it has only Q, Q-A, or both.
Repository: Marker-Inc-Korea/AutoRAG
Issue: [Chunk] Implement a feature to parse on a per-document basis when doing semantic chunking, even if you're using a per-page parsing module
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
Semantic chunking cuts into sentences and then merges those with close semantic scores (cosine similarity) to form passages.
However, when parsing on a page-by-page basis, semantic score comparisons are only made between sentences on a page, which may not perform as well as comparing sentences across the document.

**Describe the solution you'd like**
Implement a feature to parse on a per-document basis when doing semantic chunking, even if you're using a per-page parsing module (e.g. clova, table_hybrid_parse)
Repository: Marker-Inc-Korea/AutoRAG
Issue: [Feature Request] WARNING  sacrebleu:bleu.py:418 It is recommended to enable `effective_order` for sentence-level BLEU.
Root cause: Documentation Deficiency

**Is your feature request related to a problem? Please describe.**
I got this warning message when I use BLEU

```
WARNING  sacrebleu:bleu.py:418 It is recommended to enable `effective_order` for sentence-level BLEU.
```

**Describe the solution you'd like**
I think we can enable effective_order, and investigate what is it

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

**Additional context**
pip freeze | grep -i sacrebleu
sacrebleu==2.4.2
Repository: Marker-Inc-Korea/AutoRAG
Issue: [Structure Refactoring] Change beta data creation to default, and makes legacy version to legacy for v0.3
GitHub labels: enhancement

data creation

**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

**Describe the solution you'd like**
A clear and concise description of what you want to happen.

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

**Additional context**
Add any other context or screenshots about the feature request here.
Repository: Marker-Inc-Korea/AutoRAG
Issue: [BUG] [HotFix] Fix vllm.py bug 
Root cause: Others

**Describe the bug**
A clear and concise description of what the bug is.

**To Reproduce**
Steps to reproduce the behavior:
1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

**Expected behavior**
A clear and concise description of what you expected to happen.

**Full Error log**
If applicable, add full error log to help explain your problem.

**Code that bug is happened**
If applicable, add the code that bug is happened.
(Especially, your AutoRAG YAML file or python codes that you wrote)

**Desktop (please complete the following information):**
 - OS: [e.g. Windows, Linux, MacOS]
 - Python version [e.g. 3.10]

**Additional context**
Add any other context about the problem here.
Repository: Marker-Inc-Korea/AutoRAG
Issue: [BUG] YAML file is wrong at chunk_full.yaml and README.md
Root cause: Documentation Deficiency

**Describe the bug**

At the chunk_full.yaml and README.md, add_file_name is `ko` or `en`
Right? @bwook00 

Plus, the import is different (do not use beta) at README.md in data creation
Repository: Marker-Inc-Korea/AutoRAG
Issue: [Feature Request] Add azure api docs and tutorial (+ sample config files)
Root cause: Documentation Deficiency

**Is your feature request related to a problem? Please describe.**
I see many people using Azure OpenAI, because whole company using Azure only. 

**Describe the solution you'd like**
Test Azure openai system in AutoRAG, and find out what will be needed.

**Describe alternatives you've considered**
I think there are some modules that can only use OpenAI (like g_eval..?)
It can be an issue;;

**Additional context**
Add any other context or screenshots about the feature request here.
Repository: Marker-Inc-Korea/AutoRAG
Issue: [Feature Request] Add Windows Support
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
Some people still using windows. We need to test AutoRAG on the Windows machine.

**Describe the solution you'd like**
Test the pytest first. And then if something’s wrong, we will fix it.

**Describe alternatives you've considered**
I think we need to analyze how many people downloading AutoRAG in Windows.

**Additional context**
Add any other context or screenshots about the feature request here.


https://docs.python.org/ko/3/library/tempfile.html
https://www.scivision.dev/python-tempfile-permission-error-windows/

Most of the error occured because of the tempfile library!!

TART will not supported on the Windows.
(tokenizer byte sequence problem)

Plus, you have to install this.
```python
import nltk
nltk.download('averaged_perceptron_tagger_eng')
```


Plus, I got trouble to install libmagic. Install libmagic at Windows is tricky....

We need to change libmagic to the [puremagic](https://github.com/cdgriffith/puremagic) for removing OS dependency.
Why windows libmagic is not working properly?
And it turns out unstructured using libmagic.

https://docs.unstructured.io/open-source/installation/full-installation

And they are forced me to use docker images for that only.


Also UPR will not supported on the Windows, with the same reason.
MonoT5 as well.

+ You cannot use Flag Embedding Reranker with 'batch one'. You have to use more than batch 2. The default batch is 64.

It looks like it is hard to support konlpy korean tokenizers on Windows. Hard to install it.
Repository: Marker-Inc-Korea/AutoRAG
Issue: [Feature Request] Dockerfile for Korean version
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
To use korean version, we need JVM not only python. Because of the konlpy. 
The JVM version for konlpy is tricky, so we have to set it right.

**Describe the solution you'd like**
Make autorag:latest-korean docker container. 

**Describe alternatives you've considered**


**Additional context**
Add any other context or screenshots about the feature request here.
Repository: Marker-Inc-Korea/AutoRAG
Issue: [Feature Request] Add external chunk information at the front of the chunk (Anthropic method)
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
At the antropic, they extract core information for lexical search from the each chunk and put into the front side of the chunk.

**Describe the solution you'd like**
I think it is super expensive if we generate new information at the each chunk while running the `Chunker` 
It is better to use it on the only ‘one chunk’ or the `Corpus` instance.
Or compare between with metadata and without one.

**Describe alternatives you've considered**
Is there any method to create metadata from the raw document?

**Additional context**
https://www.linkedin.com/feed/update/urn:li:activity:7245896802107826176/
https://lnkd.in/gaHby5ZT


I think this will be useful if a new chunk module for contextual chunking is created. Here is the [supporting blog post](https://www.anthropic.com/news/contextual-retrieval)
Repository: Marker-Inc-Korea/AutoRAG
Issue: [Feature Request] Make short version data of Eli5 dataset for quick testing and evaluation
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
Current version of Eli5 dataset is 400 rows (as test data). This is too many for beginners who want to test AutoRAG.

**Describe the solution you'd like**
Cut them to 100 rows and also corpus as well. Corpus maybe about 500 rows?

**Additional context**
After doing this, we will use this dataset at Docker container tutorial. Or maybe use it in colab tutorial also.
Repository: Marker-Inc-Korea/AutoRAG
Issue: [BUG] Bug at casting of retrieva_gt in retrieval run.py
Root cause: Validation Gap

**Describe the bug**
There are strange code at the retrieval run.py

```python
retrieval_gt = qa_df["retrieval_gt"].tolist()
retrieval_gt = [
		[
			[str(uuid) for uuid in sub_array] if sub_array.size > 0 else []
			for sub_array in inner_array
		]
		for inner_array in retrieval_gt
	]
```

In here, the `sub_array.size` will be problem because of the `sub_array` can be string or something else. Especially, it is not working for list.

**To Reproduce**
Steps to reproduce the behavior:
Maybe use list of list retrieval_gt as retrieval_gt columns.

**Expected behavior**
It will not work when sub_array is string or list.

**Full Error log**
![image](https://github.com/user-attachments/assets/aabfe789-ba3e-447b-ad35-413270b48a8b)


**Code that bug is happened**

```yaml
node_lines:
- node_line_name: retrieve_node_line  # Set Node Line (Arbitrary Name)
  nodes:
    - node_type: retrieval  # Set Retrieval Node
      strategy:
        metrics: [retrieval_f1, retrieval_recall, retrieval_ndcg, retrieval_mrr]  # Set Retrieval Metrics
      top_k: 3
      modules:
        - module_type: vectordb
          embedding_model: openai
        - module_type: bm25
        - module_type: hybrid_rrf
          weight_range: (4,80)
- node_line_name: post_retrieve_node_line  # Set Node Line (Arbitrary Name)
  nodes:
    - node_type: prompt_maker  # Set Prompt Maker Node
      strategy:
        metrics:   # Set Generation Metrics
          - metric_name: meteor
          - metric_name: rouge
          - metric_name: sem_score
            embedding_model: openai
      modules:
        - module_type: fstring
          prompt: "Read the passages and answer the given question. \n Question: {query} \n Passage: {retrieved_contents} \n Answer : "
    - node_type: generator  # Set Generator Node
      strategy:
        metrics:  # Set Generation Metrics
          - metric_name: meteor
          - metric_name: rouge
          - metric_name: sem_score
            embedding_model: openai
      modules:
        - module_type: openai_llm
          llm: gpt-4o-mini
          batch: 16
```

**Desktop (please complete the following information):**
 - OS: [e.g. Windows, Linux, MacOS]
 - Python version [e.g. 3.10]

**Additional context**
Add any other context about the problem here.
Repository: Marker-Inc-Korea/AutoRAG
Issue: [Feature Request] New FlashRank Reranker Module
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
New reranker module!!! Yeah~~~

**Describe the solution you'd like**
Go to FlashRank Github page

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

**Additional context**
Add any other context or screenshots about the feature request here.
Repository: Marker-Inc-Korea/AutoRAG
Issue: [BUG] When there are no table in the page in clova, it occurs KeyError
Root cause: Validation Gap

**Describe the bug**

Go to `autorag/data/parse/clova.py` in line 103

```python
		table_html = json_to_html_table(resp_json["images"][0]["tables"][0]["cells"])
		page_text = extract_text_from_fields(resp_json["images"][0]["fields"])
```
When there are no key “tables”, it occurs key error.
It can be possible to put the pdf page that do not include table at all. Then it returns no table. 

**To Reproduce**
Steps to reproduce the behavior:
1. Use non-table exists page to use clova.py

**Expected behavior**
It must pass the table processing procedure.
Repository: Marker-Inc-Korea/AutoRAG
Issue: [BUG] Certain parsing modules only parse the first page of the whole pdf files, but got page_num -1
Root cause: Algorithmic Error

**Describe the bug**
It have to be contain whole pages of the pdf files, but got only first page on the certain modules. 

<The buggy modules>

1. Llamaparse
2. pymupdf (langchain_parse)
3. pdfplumber (langchain_parse)
4. pypdf (langchain_parse)
5. pypdfium2 (langchain_parse)

**To Reproduce**
Use this file . 
[[현장] 데뷔 23개월 만에 도쿄돔 달군 뉴진스, 월드투어 기대감 키웠다.pdf](https://github.com/user-attachments/files/17238758/23.pdf)

**Expected behavior**
It have to be contain whole pages of the pdf files.
Repository: Marker-Inc-Korea/AutoRAG
Issue: [BUG] Cannot get start_end_idx from the langchain_chunk konlpy splitter
Root cause: Validation Gap

**Describe the bug**
I used chunk.yaml file like below

```yaml
  - module_type: langchain_chunk
    chunk_method: konlpy
    chunk_size: 512
    chunk_overlap: 48
```

And I got no start_end_idx on the result corpus.parquet file.

![image](https://github.com/user-attachments/assets/61573de5-6ae0-4cf3-a927-c7ac700c50bd)


**To Reproduce**
Steps to reproduce the behavior:
Use this 5.parquet. 
I can’t upload parquet file in the github issue OMG

**Expected behavior**
It is missing start_end_idx… It is crucial.

**Full Error log**
If applicable, add full error log to help explain your problem.

**Code that bug is happened**
If applicable, add the code that bug is happened.
(Especially, your AutoRAG YAML file or python codes that you wrote)

**Desktop (please complete the following information):**
 - OS: [e.g. Windows, Linux, MacOS]
 - Python version [e.g. 3.10]

**Additional context**
Add any other context about the problem here.


![image](https://github.com/user-attachments/assets/9bfc4887-b3fe-46e9-beb7-2b0ffdb595ac)
I added Support Parsing Modules page
Repository: Marker-Inc-Korea/AutoRAG
Issue: [Feature Request] New Alibabacloud_Aisearch_Rerank module
GitHub labels: enhancement

New Module

[Alibabacloud_Aisearch_Rerank](https://github.com/run-llama/llama_index/tree/main/llama-index-integrations/postprocessor/llama-index-postprocessor-alibabacloud-aisearch-rerank)
Repository: Marker-Inc-Korea/AutoRAG
Issue: Add reranker test to ignore settings
Root cause: Environment Incompatibility


All reranker needs to run as local model, and it is too hard to run at github actions.
It takes lots of time to download it, and it surpass storage of github action runner.

So, we have to ignore reranker tests at GitHub actions, just test it locally.

Because `modal` looks impossible to run `pytest`. 

Reason why : it deploy seperate image on cloud, and it install `AutoRAG` as well. It is totally different environment. So, it can’t import local import statements or modified Reranker test before deployment. It is useless when it can’t test local changes.
Repository: Marker-Inc-Korea/AutoRAG
Issue: [BUG] Flag Embedding Reranker has error at the batch one.
Root cause: Validation Gap

**Describe the bug**
test_flag_embedding_reranker_batch_one test function do not passaed.

**Full Error log**
```
tests/autorag/nodes/passagereranker/test_flag_embedding.py:30 (test_flag_embedding_reranker_batch_one)
flag_embedding_reranker_instance = <autorag.nodes.passagereranker.flag_embedding.FlagEmbeddingReranker object at 0x138707a90>

    @pytest.mark.skipif(is_github_action(), reason="Skipping this test on GitHub Actions")
    def test_flag_embedding_reranker_batch_one(flag_embedding_reranker_instance):
    	top_k = 3
    	batch = 1
    	contents_result, id_result, score_result = flag_embedding_reranker_instance._pure(
    		queries_example, contents_example, ids_example, top_k, batch
    	)
>   	base_reranker_test(contents_result, id_result, score_result, top_k)

test_flag_embedding.py:38: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

contents = [['Paris is the capital of France.', 'havertz is suck at soccer', 'NomaDamas is Great Team'], ['Newjeans has 5 members.', 'LA is a country in the United States.', 'i am hungry']]
ids = [['6649ae5c-6f98-4e9a-8a2e-d94f2ecd928b', '8ba2dacb-59f4-4142-b00a-c11b205a5f03', '26e27df6-a3ab-4974-ad2c-aa1abe906dc...fdfbf058-390b-4395-a480-318a90b87b4c', '853982fc-37df-4fb6-b909-eada007d8c1a', '2a8ad16b-f14d-47a5-a7de-5d171a64a49b']]
scores = [[[7.978793621063232], [-8.296481132507324], [-9.318552017211914]], [[3.5015883445739746], [-8.99830150604248], [-9.481008529663086]]]
top_k = 3, use_ko = False, descending = True

    def base_reranker_test(contents, ids, scores, top_k, use_ko=False, descending=True):
    	assert len(contents) == len(ids) == len(scores) == 2
    	assert len(contents[0]) == len(ids[0]) == len(scores[0]) == top_k
    	for content_list, id_list, score_list in zip(contents, ids, scores):
    		assert isinstance(content_list, list)
    		assert isinstance(id_list, list)
    		assert isinstance(score_list, list)
    		for content, _id, score in zip(content_list, id_list, score_list):
    			assert isinstance(content, str)
    			assert isinstance(_id, str)
>   			assert isinstance(score, float)
E      assert False
E       +  where False = isinstance([7.978793621063232], float)

test_passage_reranker_base.py:73: AssertionError
```

**Code that bug is happened**
If applicable, add the code that bug is happened.
(Especially, your AutoRAG YAML file or python codes that you wrote)

**Desktop (please complete the following information):**
 - OS: Mac, Windows
 - Python version 3.10.14
-FlagEmbedding==1.2.11
Repository: Marker-Inc-Korea/AutoRAG
Issue: [Feature Request] Implement kotaemon to the AutoRAG run_web
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
kotaemon have a nice UI made with gradio for RAG
Why don't we implement it? 

**Describe the solution you'd like**
Let's make kotaemon Runner or something like that.

https://github.com/Cinnamon/kotaemon/

This will resolve issue #493 and #635 as well

I already implement some of it as prototype at my repo.
https://github.com/vkehfdl1/AutoRAG-web-kotaemon
Repository: Marker-Inc-Korea/AutoRAG
Issue: [Feature Request] Confuse when I create yaml file
Root cause: Documentation Deficiency

**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

I'm newbie of RAG and using LLM. When I use autorag in my Kice slayer leader board project, There is a confusing issue when creating a yaml file and using the huggingface model in the Generator node.

![Image](https://github.com/user-attachments/assets/578084fc-c9d9-401f-8b2c-bcf2780a9775)

I misunderstood that I could use a HuggingFace model by setting `module_type` parameter as HuggingFaceLLM. 

For those who don't know that they can use the huggingface model from llamma index llm, it would be nice to add a usecase in the doc to add the huggingface model to the yaml file.

![Image](https://github.com/user-attachments/assets/6f43ec49-c960-403e-9b65-99726dccea4f)