zhiminy/EvalEng huggingface dataset:
Repository: Marker-Inc-Korea/AutoRAG
Issue: [HotFix] I think the logic at the file type reduction is wrong at the parser
Root cause: Validation Gap

**Describe the bug**
parse/run.py 56:61

```python
# create a list of only those file_types that are in file_types but not in set_file_types
missing_file_types = list(file_types - set_file_types)
if list(set_file_types - file_types):
	raise ValueError(
		f"File types {list(set_file_types - file_types)} are not in the data path.”
	)
```
I don’t know why the hell we raise an error when `set_file_types - file_types`. I bet people wants just ignore the error when this kind of error occured. 


**To Reproduce**
When you put file_type to pdf, csv, html, md…
Only put pdf files in the folder.
You can get an `ValueError: File types ['csv', 'json', 'xml', 'html', 'md'] are not in the data path.`

**Expected behavior**
I think it supposed to act like this.

### 1. If there are no file type, even user write an file_type in the YAML file.

Just ignore it. Don’t load the module that doesn’t have any file is quite good idea.

 ### 2. If there is a new file type, even the user doesn’t write an file_type in the YAML file.

Raise an warning, not error. I bet skip it is okay. But just raise an warning for user will notice it.



@bwook00
Repository: Marker-Inc-Korea/AutoRAG
Issue: [BUG] Unexpected keyword argument 'embedding_model' for passage filter module
Root cause: Validation Gap

**Describe the bug**
unexpected keyword argument 'embedding_model' error occurred in passage filter module such as `SimilarityThresholdCutoff._pure()`

**To Reproduce**
1. set `embedding_model` of `similarity_threshold_cutoff` in config.yaml
2. Run autorag

**Expected behavior**
Succeed

**Full Error log**
TypeError: SimilarityThresholdCutoff._pure() got an unexpected keyword argument 'embedding_model'

**Code that bug is happened**
After init `cls` instance, `embedding_model` key must be popped from `kwargs`. But `kwargs` in `BaseModule` and `SimilarityThresholdCutoff` have different id like deep copy.
```python
class BaseModule(metaclass=ABCMeta):
    @classmethod
        def run_evaluator(
            cls,
            project_dir: Union[str, Path],
            previous_result: pd.DataFrame,
            *args,
            **kwargs,
        ):
            instance = cls(project_dir, *args, **kwargs)
            result = instance.pure(previous_result, *args, **kwargs)
            del instance
            return result
class SimilarityThresholdCutoff(BasePassageFilter):
    def __init__(self, project_dir: str, *args, **kwargs):
        super().__init__(project_dir, *args, **kwargs)
        embedding_model_str = kwargs.pop("embedding_model", "openai")
        self.embedding_model = embedding_models[embedding_model_str]()
```

I'm figuring out how to pass kwargs as shallow copy
Repository: Marker-Inc-Korea/AutoRAG
Issue: [BUG] fix docker repo tag name (api-latest)
Root cause: Configuration Error

**Describe the bug**
A clear and concise description of what the bug is.

**To Reproduce**
Steps to reproduce the behavior:
1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

**Expected behavior**
A clear and concise description of what you expected to happen.

**Full Error log**
If applicable, add full error log to help explain your problem.

**Code that bug is happened**
If applicable, add the code that bug is happened.
(Especially, your AutoRAG YAML file or python codes that you wrote)

**Desktop (please complete the following information):**
 - OS: [e.g. Windows, Linux, MacOS]
 - Python version [e.g. 3.10]

**Additional context**
Add any other context about the problem here.


<!-- 새 문제의 본문을 편집한 다음 편집기의 오른쪽 상단에 있는 ✓ "문제 만들기" 버튼을 클릭합니다. 첫 번째 줄은 문제 제목이 됩니다. 담당자와 레이블은 빈 줄 뒤에 옵니다. 문제의 본문을 시작하기 전에 빈 줄을 남겨 두세요. -->
Repository: Marker-Inc-Korea/AutoRAG
Issue: [BUG] Unable to access the documentation page.
GitHub labels: bug

**Describe the bug**

Hi,

The documentation page is showing a 404 error. Could you please check it?"

![Image](https://github.com/user-attachments/assets/07eabde9-5278-4395-b01e-64fdbdc9393c)
Repository: Marker-Inc-Korea/AutoRAG
Issue: [Feature Request] Proposal to Allow Dynamic Use of Additional Embedding Models
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
Hello!

How about improving the system to allow the use of other embedding models dynamically, in addition to the predefined models?

https://github.com/Marker-Inc-Korea/AutoRAG/blob/499f54a2a012ee398850bd94ea10127a1860caf6/autorag/vectordb/base.py#L19

https://github.com/Marker-Inc-Korea/AutoRAG/blob/499f54a2a012ee398850bd94ea10127a1860caf6/autorag/__init__.py#L65-L103

@e7217 Do you have any idea?
If you suggest any idea to config the embedding model, I think we can talk about how to make it.

@vkehfdl1 Thank you for your reply.
I will also consider a good solution for this.
Repository: Marker-Inc-Korea/AutoRAG
Issue: [BUG] Parsing module exports same parsed_result about all modules.
Root cause: Algorithmic Error

**Describe the bug**

Hello,

Could you help me? It seems that there has been a change in the parsing module.
The parsing module now exports the same `parsed_result` for all modules.
I have set up my config file for the parser as shown below:
```yaml
  - module_type: langchain_parse
    parse_method: upstagedocumentparse
    split: [ page ]
    file_type: pdf
    output_format: [ html, markdown ]
  - module_type: llamaparse
    result_type: markdown
    file_type: pdf
    language: ko
```
The current parsed result is as follows:

![Image](https://github.com/user-attachments/assets/ff7b9242-6a4c-41c4-ba39-c1bdd5108799)

However, the previous results were different. For example, when I set the `output_format` to `html` in `upstagedocumentparse`, the parsed result used to contain HTML tags.

![Image](https://github.com/user-attachments/assets/87fc55d7-eee1-4f32-8893-3837fad6e512)

I would appreciate your assistance. Thank you.
Repository: Marker-Inc-Korea/AutoRAG
Issue: [BUG] When `file_type` is set to `all_files`, only `pdfminer` is used
Root cause: Documentation Deficiency

**Describe the bug**
Hello,

It seems that when the parser module is configured with `file_type: all_files`, only pdfminer is applied. I have tried using `langchain_parser/upstagedocumentparse` and `llamaparser`, and both appear to use pdfminer exclusively. Even when I set the `output_format` to `html`, it seems like pdfminer is still being used. Am I mistaken about something?

Below is the YAML file I configured:
```yaml
- module_type: langchain_parse
  parse_method: upstagedocumentparse
  split: page
  file_type: all_files
  output_format: html
```
or
```yaml
- module_type: llamaparse
  result_type: markdown
  file_type: all_files
  language: ko
```

And here is the result:
![Image](https://github.com/user-attachments/assets/6f2a893b-74ba-484b-b77d-40790c8adf98)

I would appreciate your help. Thank you.

I found the way to set all_files.
To use it, we should set `all_files` in `start_parsing()` to `True`
```python
parser.start_parsing(parse_config, all_files=True)
```

what is correct way to set all files between as follows?
- `start_parsing(... , all_files=True)`
- set `file_type` to `all_files` in parser yaml

Additionally, to apply various parse configurations, should I modify the configuration file each time or create a new configuration for each module?

First, sorry for the late comment (I was on vacation this week🫣)

You're right, you need to do `start_parsing(... , all_files=True)` to use file_type as all_files!
I forgot and didn't write it in Docs,, sorry for the confusion!

I designed it so that if you don't put `all_files=True`, it will parse with `pdfminer`, which is the default parse method for PDF.

I'll add it to Docs right away.

Finally, sorry for the confusion and Thanks for your hard contribution!!!
Repository: Marker-Inc-Korea/AutoRAG
Issue: where is the starter.yaml referred in Tutorial doc?
Root cause: Documentation Deficiency

In the [tutorial document](https://docs.auto-rag.com/tutorial.html):

it said:
> Let’s find an optimal RAG pipeline with AutoRAG! After you prepare your evaluation dataset, you need to have a config YAML file. There are few pre-made config YAML files at our GitHub repo sample_config folder. We highly recommend using pre-made config YAML files for starter. Download starter.yaml file to your local environment, and you are ready to go.

However, i could not find the starter.yaml file in the repo. Do you renamed it? Where i can find such config file to have a quick try on AutoRAG?
A lot of thx.

@txsing 
I think it may have been changed.
If you need a simple example, how about referring to the link below?
The maintainers provide the example repository.

https://github.com/Marker-Inc-Korea/AutoRAG-tutorial

Thank you very much! I’ll take a look. It would still be great if the tutorial document could be updated to help more beginners avoid getting stuck at the very early stages.

> [@txsing](https://github.com/txsing) I think it may have been changed. If you need a simple example, how about referring to the link below? The maintainers provide the example repository.
> 
> https://github.com/Marker-Inc-Korea/AutoRAG-tutorial

Seems like this tutorial repo is again not compatible to latest version of AutoRAG. I get stucked at very beginning step again. Is there a latest tutorial code for starters?

@txsing 
The tutorial may not reflect the most recent version of the code.
If you just want to run the sample, I recommend installing AutoRAG[parse]==0.3.10 as the documentation may not be up-to-date.

If needed, please follow these steps:

1. Install the additional package:
```
pip install pycryptodome
```
2. Run the parser:
```
python run_parse.py
```
with this configuration.
```yaml
modules:
  - module_type: langchain_parse
    parse_method: [ pdfminer, pypdf, pymupdf ]
  # - module_type: llamaparse
  #   result_type: markdown
  #   language: en
  # - module_type: table_hybrid_parse
  #   text_parse_module: langchain_parse
  #   text_params:
  #     parse_method: pdfplumber
  #   table_parse_module: llamaparse
  #   table_params:
  #     result_type: markdown
  #     language: en

```
3. Run the chunker:
```
python run_chunk.py --raw_path ./parsed_raw/0.parquet
```
4. Run the QA maker:
```
python make_qa.py --raw_path ./parsed_raw/0.parquet --corpus_path ./chunked_corpus/0.parquet --qa_size 5
```
remember this:
![Image](https://github.com/user-attachments/assets/e41d75c0-edef-496d-8f24-adc081ed0b6f)
5. Run the evaluator:
```
python main.py --config ./config/tutorial.yaml
```

I hope this helps!

Thanks to the @e7217 we are closing this issue.
Thank you!
Repository: Marker-Inc-Korea/AutoRAG
Issue: [Feature Request] I hope the author adds support for the Chinese language. Thank you very much!
GitHub labels: enhancement

I hope the author adds support for the Chinese language. Thank you very much!

Hello! @yahiko-l 
Since we are not native to Chinese, we need someone to provide Chinese translation for all the prompts. 

> Hello! [@yahiko-l](https://github.com/yahiko-l) Since we are not native to Chinese, we need someone to provide Chinese translation for all the prompts.

@vkehfdl1  Hello! I am Chinese. Possibly I can help with that. Let me know if there is any needs. 

+1

If someone can translate in Chinese, it will be really helpful.
https://github.com/Marker-Inc-Korea/AutoRAG/pull/814
Here is the PR to support Japanese
Repository: Marker-Inc-Korea/AutoRAG
Issue: [Feature Request] Add Ollama Embedding
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
Not supporting ollama for embedding model.
It may related to #985 

**Describe the solution you'd like**
Fix the bug and add ollama for embedding model

**Describe alternatives you've considered**


**Additional context**



If you are willing to apply ollama, then please assign to me.

@rjwharry Thank you! I assigned to you.
One thing, we changed a little bit to use embedding model recently. Not much changed, but please check https://github.com/Marker-Inc-Korea/AutoRAG/pull/1063 #1063
Repository: Marker-Inc-Korea/AutoRAG
Issue: ModuleNotFoundError: No module named 'autorag.data'; 'autorag' is not a package
Root cause: Environment Incompatibility

This strangely happens with autorag.data but not autorag. 

import autorag
import autorag.data
from autorag.data.qa.schema import Raw, Corpus

ModuleNotFoundError: No module named 'autorag.data'; 'autorag' is not a package

autorag/data has the __init__.py file as expected.


am I missing some dependency? I think I downloaded everything in the install instructions.
Repository: Marker-Inc-Korea/AutoRAG
Issue: Delete legacy data creation process

```
FAILED tests/autorag/data/legacy/qacreation/test_base_qacreation.py::test_make_qa_with_existing_qa_persistent_client_with_gen_gt - DeprecationWarning: This function is deprecated.
FAILED tests/autorag/data/legacy/qacreation/test_ragas_qa_creation.py::test_generate_qa_ragas - ragas.exceptions.ExceptionInRunner: The runner thread which was ru...
FAILED tests/autorag/data/legacy/qacreation/test_base_qacreation.py::test_make_qa_with_existing_qa_without_gen_gt - DeprecationWarning: This function is deprecated.
FAILED tests/autorag/data/legacy/qacreation/test_base_qacreation.py::test_make_qa_with_existing_qa_with_gen_gt - DeprecationWarning: This function is deprecated.
FAILED tests/autorag/data/legacy/qacreation/test_base_qacreation.py::test_make_qa_with_existing_qa_persistent_client_without_gen_gt - DeprecationWarning: This function is deprecated.
```

Now legacy data creation process is no longer supported.
Repository: Marker-Inc-Korea/AutoRAG
Issue: [Feature Request] Running locally
Root cause: Documentation Deficiency

Finding no resources on how to run this all locally. 

Getting a timeout error, looks like from open_ai

Guessing it's defaulting to open_ai if it doesn't find the llm? Not really sure


Here is my YAML

```yaml
node_lines:
- node_line_name: retrieve_node_line
  nodes:
  llm_model:
  type: huggingface
  model_name: HuggingFaceH4/zephyr-7b-beta 

  
    - node_type: retrieval
      strategy:
        metrics: [retrieval_f1, retrieval_recall, retrieval_ndcg, retrieval_mrr]  # Include multiple metrics for comparison
      top_k: 5  # Adjust based on desired recall depth
      modules:
        - module_type: vectordb
          vectordb: default  # Default vector database used for retrieval
          embedding_model:
            type: huggingface
            model_name: sentence-transformers/all-MiniLM-L6-v2  # Model 1
```

output

[01/27/25 12:21:13] INFO     [posthog.py:22] >> Anonymized telemetry enabled. See                     https://docs.trychroma.com/telemetry posthog.py:22
                             for more information.                                                                                                      
[01/27/25 12:21:14] INFO     [_base_client.py:1666] >> Retrying request to /embeddings in 0.479268 seconds                          _base_client.py:1666
                    INFO     [_base_client.py:1666] >> Retrying request to /embeddings in 0.892558 seconds                          _base_client.py:1666
[01/27/25 12:21:15] INFO     [_base_client.py:1666] >> Retrying request to /embeddings in 1.674951 seconds                          _base_client.py:1666
[01/27/25 12:21:17] INFO     [_base_client.py:1666] >> Retrying request to /embeddings in 3.469390 seconds                          _base_client.py:1666
Ingesting VectorDB... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% 0/1 0:00:49

Anything? Why is openai being accessed??

node_lines:
  - node_line_name: retrieve_node_line
    nodes:
      - node_type: retrieval
        strategy:
          metrics: [retrieval_f1, retrieval_recall, retrieval_ndcg, retrieval_mrr]  # Multiple metrics for comparison
        top_k: 5  # Adjust based on desired recall depth
        modules:
          - module_type: llama_index_llm
            llm: huggingface
            model_name: HuggingFaceH4/zephyr-7b-beta
          - module_type: vectordb
            vectordb: default  # Default vector database used for retrieval
            embedding_model:
              type: huggingface
              model_name: BAAI/bge-large-en

@AlbertoMQ 

Hello, 
https://docs.auto-rag.com/integration/vectordb/vectordb.html
https://docs.auto-rag.com/nodes/retrieval/vectordb.html

You have to configure your embedding model at vectordb section.
Please read above documentations and follow the instruction. 

Thank you.
Repository: Marker-Inc-Korea/AutoRAG
Issue: [BUG] OpenAILIKE additional params are not supported
Root cause: Validation Gap

**Describe the bug**
OpenAILIKE additional params are not supported like is_chat_model param...

**To Reproduce**
Configure the following llm module and use is_chat_model param like the following: 
```yaml
    - node_type: generator
      modules:
        - batch: 2
          module_type: llama_index_llm
          llm: openailike
          model: qwen-plus
          is_chat_model: True
          api_base: https://dashscope.aliyuncs.com/compatible-mode/v1
          api_key: x
```
It will have error like 404 error with the endpoint '/v1/completions'. The error is because is_chat_model param is not passed into the model. If the model itself does not support /v1/completions endpoint, it will have error as it doesn't call /v1/chat/completions'.

**Additional context**
The root cause may be due to llm_instance params is using llm_class.__init__ instead of the llm_class...

```
autorag/nodes/generator/llama_index_llm.py line 52 __init__ method

self.llm_instance: BaseLLM = llm_class(**pop_params(llm_class.__init__, kwargs))

```
When using openAILike, it is converting the OpenAI params with ignoring OpenAILike params....

**Possible fix**
Stupid but quick fix...
```
		llm_openaiLIke_params = None
		if llm_class.class_name() == "OpenAILike":
			llm_openaiLIke_params = pop_params(llm_class, kwargs)
		original_llm_params = pop_params(llm_class.__init__, kwargs)
		if llm_openaiLIke_params is not None:
			llm_params = {**llm_openaiLIke_params,**original_llm_params}
		else:
			llm_params = original_llm_params
		self.llm_instance: BaseLLM = llm_class(**llm_params)
```





Thank you for registering bug!
It looks like the `pop_params` is not well functioning to the pydantic class. I might change it to support it.
Repository: Marker-Inc-Korea/AutoRAG
Issue: [HotFix] Import fix at cli.py
GitHub labels: bug

* As-is

```
from autorag.deploy import extract_best_config as original_extract_best_config
```

* To-be

```
from autorag.deploy.base import extract_best_config as original_extract_best_config
```

at cli.py

This is not error
Repository: Marker-Inc-Korea/AutoRAG
Issue: [Documentation] Write document for merging Feature/#959

First, release a "Newbie" version first.
We need to develop further to the "Pro" version.

Plus these features in this feature.

- Change the name of the service
- Change each version to the "Newbie" and the "Pro" version.
- Re-design `main.tsx`

![Image](https://github.com/user-attachments/assets/dbe0a49b-ebf2-4c9c-b17d-1be1c2cd1060)

![Image](https://github.com/user-attachments/assets/87289d84-ff65-4810-bc41-3f30b36b7ddf)

![Image](https://github.com/user-attachments/assets/d8a50512-3299-4b68-b48e-e2f49d688f01)
Repository: Marker-Inc-Korea/AutoRAG
Issue: [BUG] Custom embedding models
Root cause: Documentation Deficiency

Thank you for making this and for your help.

I am having a near impossible time setting up custom embedding models. 
Here are things I've tried

```python
from llama_index.embeddings.huggingface import HuggingFaceEmbedding
embed_model_name = Config._default_embed_model_name.replace("/", "-")
embed_model = HuggingFaceEmbedding(
                    model_name= "BAAI/bge-large-en",
                    cache_folder=   Config.MODELS_DIR
                )
autorag.embedding_models[embed_model_name] = LazyInit(Config.default_embed_model, model_name=Config._default_embed_model_name)
autorag.embedding_models[embed_model_name] = Config.default_embed_model
```

```yaml
vectordb:
  - name: bge-large
    db_type: chroma
    client_type: persistent
    embedding_model: BAAI-bge-large-en
    collection_name: BAAI-bge-large-en
    path: ${PROJECT_DIR}/data/chroma
node_lines:
- node_line_name: retrieve_node_line
  nodes:
    - node_type: retrieval
      strategy:
        metrics: [retrieval_f1, retrieval_recall, retrieval_precision,
                  retrieval_map, retrieval_mrr, retrieval_ndcg]
      top_k: [1, 3, 5, 10, 50]
      modules:
        - module_type: vectordb
          vectordb: bge-large
```

///////////////////////////////////

                             TypeError: missing a required argument: 'nodes'          

//////////////////////////////////

Switching embeddings                              
  embedding_model:
  - type: huggingface
    model_name: intfloat/multilingual-e5-large-instruct         

///////////////////////////
This gives an error because it is called as embedding_models[] in the code, and this dict won't do.
//////////////////////////

I've tried this style as well.  (I have no idea which I'm supposed to use)

      modules:
        - module_type: vectordb
          embedding_model:
           - bge-large-en
 
                                                        

1. Upgrade AutoRAG version to latest (up to v0.3.13)
2. Try like this.

```yaml
vectordb:
- name: bge-large
  db_type: chroma
  client_type: persistent
  path: ${PROJECT_DIR}/data/chroma
  embedding_model:
  - type: huggingface
    model_name: BAAI/bge-large-en
```

And you can use the "bge-large" vectordb at the retrieval node.
Repository: Marker-Inc-Korea/AutoRAG
Issue: [BUG]
Root cause: Documentation Deficiency

Sorry for all the bug reports lately. I appreciate your help.

I am confused why this exists in the validator. Sampling from corpus means the QA doc_ids likely won't be found in the corpus data.

		with (
			tempfile.NamedTemporaryFile(suffix=".parquet", delete=False) as qa_path,
			tempfile.NamedTemporaryFile(suffix=".parquet", delete=False) as corpus_path,
			tempfile.TemporaryDirectory(ignore_cleanup_errors=True) as temp_project_dir,
		):
			sample_qa_df.to_parquet(qa_path.name, index=False)
			sample_corpus_df.to_parquet(corpus_path.name, index=False)

			evaluator = Evaluator(
				qa_data_path=qa_path.name,
				corpus_data_path=corpus_path.name,
				project_dir=temp_project_dir,
			)

@AlbertoMQ 
Hello! In the validator, the `sample_corpus_df` is constructed from the `sample_qa_df` retrieval_gt ids. 
So it will be matched perfectly unless you are using `passage augmenter` modules.

Thanks for the question