zhiminy/EvalEng huggingface dataset:
Repository: Marker-Inc-Korea/AutoRAG
Issue: Configure VectorDB at a YAML file.
Root cause: Unimplemented Feature Gap

When you configure VectorDB you can use all vectorstores from LlamaIndex.
But, we need to configure it, at least collection name, vector DB URL.
And maybe, it is really easy to understand when we do not configure all vectorDB one by one inside the nodes & modules. It is easy to configure outside node_lines key.

close at #874
Repository: Marker-Inc-Korea/AutoRAG
Issue: [BUG] pip install -U AutoRAG fails: Cannot uninstall distutils-installed blinker 1.4
Root cause: External Dependency Breakage

![Image](https://github.com/user-attachments/assets/c67e56e6-4886-4d45-aeec-287dfcbd3934)


**Describe the bug**

Environment: Google Colab running AutoRAG Tutorial 2 - Evaluation Dataset Creation.ipynb

`pip` fails to install -Uq AutoRAG[parse]>=0.3.0 datasets. It required to uninstall the `blinker` package (version 1.4) when attempting to upgrade `AutoRAG`, preventing the upgrade process from completing. The error message indicates that `blinker` was installed using `distutils`, making it difficult for `pip` to identify all associated files for removal.

**To Reproduce**

1. Run the following command in a terminal or console:  `pip install -Uq AutoRAG[parse]>=0.3.0 datasets`
2. Observe the error message.


**Expected behavior**

`pip` should successfully uninstall the older version of `blinker` and then install/upgrade `AutoRAG` and its dependencies without error.


**Full Error log**

```
error: uninstall-distutils-installed-package
× Cannot uninstall blinker 1.4
└> It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.
```


**Code that bug is happened**

The bug occurs when executing the following command:

```python
# In a terminal or console
pip install -Uq AutoRAG[parse]>=0.3.0 datasets 
```

**Additional context**

* The root cause appears to be the use of the deprecated `distutils` installation method for the `blinker` package.  `pip` has improved its handling of packages over time, but it still struggles with cleanly uninstalling packages installed via `distutils`.  
* Manually removing `blinker` might be a workaround, but it's not recommended due to the risk of accidentally deleting files belonging to other packages.  A more robust solution within `pip` itself is desirable.
![Image](https://github.com/user-attachments/assets/0f58658d-74cb-480e-a590-ce0c83fb3077)


@abdshomad Hello!
Try this before installing AutoRAG in colab.

```
%pip install blinker==1.4 ipykernel==5.5.6
```


> [@abdshomad](https://github.com/abdshomad) Hello! Try this before installing AutoRAG in colab.
> 
> ```
> %pip install blinker==1.4 ipykernel==5.5.6
> ```

I tried but still failed. Here's the screenshot. NOTE: I am running it in Google Colab link provided in this repo. 

![Image](https://github.com/user-attachments/assets/5d2f5d08-008d-477b-928b-3acc10c3a11a)

Any other option I can try? 

@abdshomad 
We are working on this issue on the #888 PR...

@abdshomad 

```
%%shell
apt-get remove python3-blinker
pip install blinker==1.8.2
%pip install -Uq ipykernel==5.5.6 ipywidgets-bokeh==1.0.2
```

Use this on the colab before installing AutoRAG

We updated colab tutorial!
Repository: Marker-Inc-Korea/AutoRAG
Issue: [Feature Request] Add More Vector DB
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
We can support more Vector DBs like Weaviate, Pinecone, FAISS, Vectara, pgvector, mongo search and more. 

**Describe the solution you'd like**
We want to use Langchain implementation, but it has few limitations so we have to implement it on our own. 

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

**Additional context**
Add any other context or screenshots about the feature request here.
Repository: Marker-Inc-Korea/AutoRAG
Issue: [Feature Request] Add custom_query_gen function
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
Different LLM models result in different prompts, especially for local LLMs. Why not allow users to try their own queries?

**Describe the solution you'd like**
Implement custom_query_gen function receiving user's custom query generation prompt.


If you think it is good, please assign to me

 @rjwharry 
Thanks for making this issue!
I think this feature will make it easier to create synthetic queries.

I just assigned it!
Repository: Marker-Inc-Korea/AutoRAG
Issue: [Feature Request] SGlang, the fast serving framework
GitHub labels: enhancement

New Module

**Is your feature request related to a problem? Please describe.**


This is kind of an 'FYI' for @vkehfdl1 from our previous brief coffee chat. You may simply close this issue if you are already fully aware of it. 






**Additional context**


You may refer the sources below:
1. the official [GitHub repo]( https://github.com/sgl-project/sglang)
2. reactions, reviews from [the Korean community](https://arca.live/b/alpaca/119757762?p=1)


@skmanzg 
Thanks:)
It is weird I cannot find sglang llama index integration.
We will implement it soon.

Thanks:)
Repository: Marker-Inc-Korea/AutoRAG
Issue: [Feature Request] Add API server for running each features like validation, starting trial, chunking or parsing.
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
Using AutoRAG at remote service.

**Describe the solution you'd like**
A clear and concise description of what you want to happen.

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

**Additional context**
Add any other context or screenshots about the feature request here.


Work this as private
Repository: Marker-Inc-Korea/AutoRAG
Issue: [Feature Request]Need supporting Azure OpenAI
Root cause: Documentation Deficiency

**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

**Describe the solution you'd like**
A clear and concise description of what you want to happen.

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

**Additional context**
Add any other context or screenshots about the feature request here.


@hellangleZ Hello!
You can use Azure OpenAI using LlamaIndex LLM.
Duplicate with #714 

https://docs.auto-rag.com/local_model.html
See this docs how to add it.

We support all LLMs in LlamaIndex, so you do not worried about LLM integrations.

Since this is duplicate I am closing this issue.
Repository: Marker-Inc-Korea/AutoRAG
Issue: [Feature Request] GitHub Action Causes Large Docker Image Size
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

**Describe the solution you'd like**
A clear and concise description of what you want to happen.

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

**Additional context**
Add any other context or screenshots about the feature request here.

https://hub.docker.com/layers/autoraghq/autorag/api-0.3.7/images/sha256-d3b8569b1e2e1a2649e9f1461cab159830a23eb287dbf38f1891bf2a80c91562?context=explore

https://hub.docker.com/layers/autoraghq/autorag/api-0.3.8/images/sha256-9f08bfc5f178f6c5e57f77925055ee98f7d80ec5b4712a8a7952bb926c06f2c8?context=explore
Repository: Marker-Inc-Korea/AutoRAG
Issue: [Feature Request] Add CLI Version Check
GitHub labels: enhancement

help wanted

**Is your feature request related to a problem? Please describe.**
It’s difficult to verify the CLI version quickly, leading to compatibility issues.

**Describe the solution you'd like**
Add a version check command (e.g., --version or -v) to display the current CLI version.

**Describe alternatives you've considered**
Manually checking version details in files.
/python3.10/site-packages/AutoRAG-0.3.8.dist-info/

or 
```sh 
$ pip show autorag
Name: AutoRAG
Version: 0.3.8
...
```
**Additional context**
N/A

@hongsw 
You can check `VERSION` file.

```python
with open(‘VERSION’, ‘w’) as f:
    version = f.read()
```

Something like this?

$ autorag -V

AutoRAG v0.3.8

Here's what I expect it to do.
@vkehfdl1
Repository: Marker-Inc-Korea/AutoRAG
Issue: [Guide] AttributeError: '_config' object has no attribute 'initialized
Root cause: External Dependency Breakage

**Describe the bug**
A clear and concise description of what the bug is.

```
AttributeError: '_config' object has no attribute 'initialized
```
![Image](https://github.com/user-attachments/assets/4feda015-14e8-4e50-8d46-a4cd64908f70)


**To Reproduce**
Steps to reproduce the behavior:
1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

**Expected behavior**
A clear and concise description of what you expected to happen.

**Full Error log**
If applicable, add full error log to help explain your problem.

**Code that bug is happened**
If applicable, add the code that bug is happened.
(Especially, your AutoRAG YAML file or python codes that you wrote)

**Desktop (please complete the following information):**
 - OS: MacOS
 - Python version 3.10.15
 - AutoRAG version 0.3.8

**Additional context**
Add any other context about the problem here.


```sh
pip freeze > temp_requirements-main.venv.txt    # backup current versions
pip freeze | xargs pip uninstall -y
pip install autorag
```
or

Make sure the Langchain version is '0.3.6'
![Image](https://github.com/user-attachments/assets/baf62883-d84c-44c7-8d21-be518ce2be05)
Repository: Marker-Inc-Korea/AutoRAG
Issue: llm token management + strategy
Root cause: Unimplemented Feature Gap

Need to manage input token on node using LLM. Will be supported in the next version.

[What to do?]

At first, this is a 'token strategy'. It means, you can set token limitation when optimizing each process. Some process consume many token to llm, and it effects overall pricing and speed.
Plus, it matters because the devs sometimes wants to monitor their token usage. 

[Suggestion]

I think it can be like python "logger" for llm token management. 

1. The python logger using 'singleton instance' at each logger name. 
We can define token logger instance like that.
2. When init each token logger, you can set tokenizer for counting it. The supported tokenizer might be `tiktoken` and `AutoTokenier` from now.
3. When log to instance, it logs automatically. It can save to separate log file, or csv.
Also, it supports sum or average of token usage. 
4. It can be cool we can reset or pause the token save. Like the record feature.

With this, we can easily collect token data at optimization process, without modifying any function input. 

I think it can be a standalone library for everyone wants to logging their token.


+ There are repo like `LLMTelemetry` that supports monitoring LLM calls. It can be a solution? Try to look at it.
Repository: Marker-Inc-Korea/AutoRAG
Issue: [BUG] Upstage Embedding not working
Root cause: Validation Gap

At vectordb.py, llamaindex UpstageEmbedding class is instance of OpenAIEmbedding.
So it tires to truncate token to 8000, but tiktoken cannot tokenize it because solar embedding model is not capable of tokenize in tiktoken. 

Add model_name condition at truncate openai token also.



+ 
Just change colab demo to BAAI
Repository: Marker-Inc-Korea/AutoRAG
Issue: [Feature Request] Automated Integration of LlamaIndex with Langfuse for Enhanced Observability
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**

Integrating LlamaIndex with Langfuse for observability currently requires manual code implementation, which can be time-consuming and prone to errors.

**Describe the solution you'd like**

An automated integration between LlamaIndex and Langfuse that simplifies the setup process, allowing developers to achieve observability with minimal configuration. Utilizing Langfuse's `LlamaIndexInstrumentor` for straightforward initialization would be ideal.

**Describe alternatives you've considered**

Manually coding the integration, which increases the risk of inconsistencies and maintenance challenges. Exploring other observability tools was considered, but Langfuse's features align better with our requirements.

**Additional context**

Langfuse's official documentation provides examples for integrating with LlamaIndex.  Implementing an automated integration based on these guidelines would enhance developer productivity. 

https://langfuse.com/docs/integrations/llama-index/example-python-instrumentation-module
Repository: Marker-Inc-Korea/AutoRAG
Issue: [Feature Request] Add Nvidia Nim LLM
Root cause: Unimplemented Feature Gap


We can use Nvidia Nim LLM to use LlamaIndex LLM's OpenAI LIke !

Here is the example YAML file to use `nvidia/llama-3.1-nemotron-70b-instruct` model.

```
nodes:
  - node_line_name: node_line_1
    nodes:
      - node_type: generator
        modules:
          - module_type: llama_index_llm
            llm: openailike
            model: nvidia/llama-3.1-nemotron-70b-instruct
            api_base: https://integrate.api.nvidia.com/v1
            api_key: your_api_key
```
Repository: Marker-Inc-Korea/AutoRAG
Issue: Implement restart point where the error occurred.
Root cause: Unimplemented Feature Gap


0. Save the restart result value in the `trial_path` where the error occurred.
1. get "trial_path" as an input parameter to the Restart function
2. base the restart on the presence of a `summary.csv` file
    - Does node_line have a summary.csv?
        - Does Node's summary.csv exist?
	=> execute from the one without summary.csv
3. the unit of execution is Node.
Repository: Marker-Inc-Korea/AutoRAG
Issue: [BUG] Kkma raises UnicodeDecodeError when text includes emoji
Root cause: Validation Gap

**Describe the bug**
When emoji is included in text, Kkma raises `UnicodeDecodeError` in `morphs(text)` function.

**To Reproduce**
Steps to reproduce the behavior:
1. Use `ko_kkma` as tokenizer in retrieval node
2. Include text with emoji in corpus.parquet
3. Run validator
4. See error

**Expected behavior**
Pass validation

**Full Error log**
`UnicodeDecodeError: 'utf-8' codec can't decode byte 0xed in position 0: invalid continuation byte`

**Desktop (please complete the following information):**
 - OS: Linux
 - Python version 3.10



I think removing emoji can be good solution. Mostly emoji does not have much information

Please assign to me, I'll figure it out

@rjwharry I assigned to you, but it is possible that this error is not occured because of ko_kkma.
I think it will be possible that `normalize_unicode` function occured the error in the `utils/util.py`




It occurred in kkma. Here is the full error log
![Image](https://github.com/user-attachments/assets/853157dc-9548-4e36-9e54-09b153db2bf4)
Repository: Marker-Inc-Korea/AutoRAG
Issue: [Feature Request] Implement DeasyLabs metadata retrieval and metadata rerankers.
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
DeasyLabs supports great metadata retrieval and metadata reranker system. Plus auto-generating metadata for great search. It can help us to get a better RAG performance. 

**Describe the solution you'd like**
New modules required.

- DeasyLabs Ingestion => At the dataset creation.
- DeasyMetadataRetrieval
- DeasyMetadataReranker


First need to implement #943
Repository: Marker-Inc-Korea/AutoRAG
Issue: [BUG] The vectordb setting is always applied as default when validating
Root cause: Documentation Deficiency

**Describe the bug**
Event I configure vectordb, always set as default when validating

**To Reproduce**
Steps to reproduce the behavior:
1. Configure yaml with custom vectordb configuration
2. Run validation
3. See error

**Expected behavior**
Ingest vectordb with my configuration

**Full Error log**
not error

**Code that bug is happened**
`config_dict` is full dict of config.yaml, so it only has **node_lines** as key. 
vectordb_list always becomes empty list because `config_dict` has to "vectordb" key.  
If that's not what you intended, it need to be fixed it.
```python
def load_all_vectordb_from_yaml(
    yaml_path: str, project_dir: str
) -> List[BaseVectorStore]:
    config_dict = load_yaml_config(yaml_path)
    vectordb_list = config_dict.get("vectordb", [])
    if len(vectordb_list) == 0:
        chroma_path = os.path.join(project_dir, "resources", "chroma")
        return [
            load_vectordb(
                "chroma",
                client_type="persistent",
                embedding_model="openai",
                collection_name="openai",
                path=chroma_path,
            )
        ]
...
```
**Suggestion**
We need to ingest vectordb configured in retrieval node



I'll get vectordb configuration from retrieval node

Or maybe ingest all vectordb in config.yaml like bm25 did

I need to be fixed right now! This is not only for validation but also Evaluator.
I think we should ingest all vectordb in config.yaml like bm25

@rjwharry 
Actually I think it is intended. We changed the VectorDB configuration setting since v0.3.8

https://docs.auto-rag.com/integration/vectordb/vectordb.html
Here is the full documentation how to set different vectorDB, also embedding models.

You must configure the vectorDB at the `vectordb` key and use it on the retrieval `vectordb` module. 



I've figured out how to configure VectorDB. This is my mistake, let me close this issue
Repository: Marker-Inc-Korea/AutoRAG
Issue: [BUG] Having trouble using LiteLLM Embedding
GitHub labels: bug

**Describe the bug**
A clear and concise description of what the bug is.

**To Reproduce**
Steps to reproduce the behavior:
1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

**Expected behavior**
A clear and concise description of what you expected to happen.

**Full Error log**
If applicable, add full error log to help explain your problem.

**Code that bug is happened**
If applicable, add the code that bug is happened.
(Especially, your AutoRAG YAML file or python codes that you wrote)

**Desktop (please complete the following information):**
 - OS: [e.g. Windows, Linux, MacOS]
 - Python version [e.g. 3.10]

**Additional context**
Add any other context about the problem here.


The LiteLLm embedding works well.
Since it is not bug, I am closing this issue.
Repository: Marker-Inc-Korea/AutoRAG
Issue: [BUG] Invalid temperature values in full.yaml.
GitHub labels: bug

# Description of  the Bug
The temperature parameter was set as a list with values [0.5, 1.0, 1.5], which includes an invalid value (1.5). Since valid temperature values must be within the range of 0–1, this caused a validation error when running the code.

# To Reproduce:

# Steps to reproduce the behavior:

- The temperature parameter in (https://github.com/Marker-Inc-Korea/AutoRAG/blob/main/sample_config/rag/english/gpu/full.yaml) is set to [0.5, 1.0, 1.5] in the configuration file.

- Run the code.
   Observe the validation error due to the invalid temperature value.

# Expected Behavior
The temperature parameter should only contain values within the range of 0–1.

# Full Error Log
ValidationError: 1 validation error for OpenAI
                             temperature
                               Input should be less than or equal to 1 [type=less_than_equal, input_value=1.5, input_type=float]

**Code that bug is happened**
- full.yaml where the temperature is set as a list
temperature: [0.5, 1.0, 1.5]  # Change to valid values, e.g., [0.5, 1.0]


# Desktop:
 - OS: Windows
 - Python version: 3.10.9

# Additional Context
To fix the issue, consider limiting the list to values within the 0–1 range, e.g., [0.5, 1.0], or use a single value if a range is not required.

Hi, @ImaadHasan2002 
At OpenAI temperature setting, 0 to 2 is valid settings for temperature.

https://platform.openai.com/docs/api-reference/chat/create#chat-create-temperature
```
temperature
number or null

Optional
Defaults to 1
What sampling temperature to use, between 0 and 2. Higher values like 0.8 will make the output more random, while lower values like 0.2 will make it more focused and deterministic.

We generally recommend altering this or top_p but not both.
```
The official api docs say like this.

Since it is not an bug, I will close this issue.
Thank you.
Repository: Marker-Inc-Korea/AutoRAG
Issue: [BUG] Unexpected Open API Requests Triggered Under Inactive Conditions
Root cause: Configuration Error

**Describe the bug**
API calls to OpenAI occur despite having multiple modules configured to operate locally without OpenAI access.

This occurs in `post_retrieve_node_line` within `prompt_maker` when there are multiple instances of `module_type`, such as `fstrings`, during the Validation phase.

**To Reproduce**
Part of the config.yml
```
(...)
- node_line_name: post_retrieve_node_line  # Arbitrary node line name
  nodes:
    - node_type: prompt_maker
      strategy:
        metrics: [ meteor, rouge, bert_score ]
      modules:
          # - module_type: fstring
          #   prompt: "Read the passages and answer the given question. \n Question: {query} \n Passage: {retrieved_contents} \n Answer : "
         - module_type: fstring
           prompt: ["Tell me something about the question: {query} \n\n {retrieved_contents}",
                    "Question: {query} \n Something to read: {retrieved_contents} \n What's your answer?"]
        
      generator_modules:
          - module_type: llama_index_llm
            llm: ollama
            model: gemma2:27b
(...)
````

I get the logs:
```
INFO     [_client.py:1786] >> HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 429 Too Many Requests"                                _client.py:1786
	                    INFO     [_base_client.py:1661] >> Retrying request to /chat/completions in 0.421052 seconds                                                           _base_client.py:1661
	                    INFO     [_client.py:1786] >> HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 429 Too Many Requests"  
```

Swapping the comments between the module_types allows the validation to pass successfully.

Thanks


My mistake. I need to put the `generator_module` inside the strategy:
```yaml
      strategy:
        metrics: [ meteor, rouge ]
        generator_modules:
          - module_type: llama_index_llm
            llm: ollama
            model: gemma2:27b
            batch: 1
            temperature: 1.0
            request_timeout: 300 
```