zhiminy/EvalEng huggingface dataset:
Repository: Marker-Inc-Korea/AutoRAG
Issue: [Feature Request] ROUGE Score Drops Due to <think> Tags in LLM Outputs (Internal Reasoning)
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
Recent LLMs such as Qwen 3 and DeepSeek R1 use <think> tags to include intermediate reasoning steps in their responses. However, this reasoning content is also included in the generated output used for evaluation, which leads to unfairly low ROUGE scores.

**Describe the solution you'd like**
I’d like the evaluation step to exclude content wrapped in <think>...</think> tags. This would allow evaluation to focus only on the final answer intended for users, improving the fairness and accuracy of ROUGE-based scoring.

**Describe alternatives you've considered**
Right now, I'm using a simple workaround by modifying the generation output via regex filtering. Specifically, I patched the following line in:

.venv/lib/python3.10/site-packages/autorag/nodes/generator/llama_index_llm.py, line 84:

```python
generated_texts = list(map(lambda x: re.sub(r'<think>.*?</think>\s*', '', x.text, flags=re.DOTALL).strip(), results))
```
This strips out the <think> blocks before evaluation. While it works for now, I'm not sure if this is the best long-term solution.

Hello @BaconTomatoDeluxe 
Looks like great feature to add! And thank you for suggesting workaround.
Maybe it can be great option to add in the rouge score metric.

I need <think> token variation via model. Not every model use <think> token isn't it? Some model can use <thinkpad> or etc.
If I can get that kind of list, it will be very helpful.

Thanks again for the quick response.
I also really appreciated your point about how the tag names might differ depending on the model.

Since I wanted to avoid including internal reasoning in the output altogether, I dug into the code to understand why the thinking part might still be showing up. So far, I’ve only looked into this in the context of Ollama.
While inspecting the llama_index integration, I noticed that Ollama’s response includes both a content field (representing the actual answer) and a thinking field (which seems to represent intermediate reasoning steps).

Ideally, only the content should be surfaced as the model’s final answer. I’m still not entirely sure why that happens — it could be due to some normalization step I’m missing on my end.

https://ollama.com/blog/thinking

@BaconTomatoDeluxe I think each model integrations have different strategies to deal with think tokens. (like huggingface, vllm, ollama, etc.)
Let's implement your workaround as a metric option first, and build an better version later if we can find a universal interface for classifying thinkpad and real answer of model.
Repository: Marker-Inc-Korea/AutoRAG
Issue: [Bug] CohereReranker compatibility issue with cohere>=5.8.0 - TypeError: unexpected keyword argument 'return_documents'
Root cause: External Dependency Breakage

**Describe the bug**
CohereReranker fails with TypeError when using cohere>=5.8.0. The error indicates that `return_documents` is an unexpected keyword argument for the AsyncV2Client.rerank() method.

**To Reproduce**
Steps to reproduce the behavior:
1. Install AutoRAG 0.3.17 with cohere>=5.8.0 (as required by AutoRAG 0.3.17)
2. Configure passage_reranker with CohereReranker in YAML config
3. See error during passage_reranker node execution

**Expected behavior**
CohereReranker should successfully rerank passages without errors.

**Full Error log**
`TypeError: AsyncV2Client.rerank() got an unexpected keyword argument 'return_documents'`

Full traceback occurs in `/autorag/nodes/passagereranker/cohere.py:107` where `cohere_client.rerank()` is called with incompatible parameters. The error happens during the `cohere_rerank_pure` function execution.

**Code that bug is happened**
AutoRAG config file using:
```
passage_reranker:
  - module_type: cohere
    model: rerank-v3.5
    top_k: 10
```

**Additional context**
This appears to be a breaking change in the Cohere API. The return_documents parameter may have been removed or renamed in cohere>=5.8.0, but AutoRAG's CohereReranker implementation still uses the old parameter name. This creates a dependency conflict where AutoRAG requires cohere>=5.8.0 but the code is incompatible with the new API interface.

Hi @LimHarvey Thank you for assign the issue:)

Happens when cohere>=5.18.0
Repository: Marker-Inc-Korea/AutoRAG
Issue: [BUG] There is an error when deleting the Vllm instance.
Root cause: External Dependency Breakage

**Describe the bug**
There is no vllm model_executor in the latest version of vLLM.
It has to be deleted in the vllm generator module.

**To Reproduce**
Steps to reproduce the behavior:
After you run the Vllm module, it has an error when deleting it. 

**Expected behavior**
It has to be no error.


**Code that bug is happened**
The YAML file contains the vLLM module.

**Desktop (please complete the following information):**
- OS : linux
 - Python version 3.12 I guess

**Additional context**
Add any other context about the problem here.
Repository: Marker-Inc-Korea/AutoRAG
Issue: ✨ Set up Copilot instructions

Configure instructions for this repository as documented in [Best practices for Copilot coding agent in your repository](https://gh.io/copilot-coding-agent-tips).

<Onboard this repo>

First, read the README.md to understand the higher concept.

Read the `docs` folder to see the documentation.
All test files are in the `tests` folder.
This is a monorepo of the AutoRAG web system, which has api, a frontend, and the AutoRAG. But, recommend focusing only on the `autorag` main Python library since the others are no longer supported.
Repository: Marker-Inc-Korea/AutoRAG
Issue: [Hotfix] GItHub Actions(PostgreSQL tests) does not works at branch v0.4
Root cause: Environment Incompatibility

<img width="2724" height="1260" alt="Image" src="https://github.com/user-attachments/assets/258be8c0-73b2-44d4-9506-85a893952f17" />
Repository: Marker-Inc-Korea/AutoRAG
Issue: [BUG] Typo in huggingface
GitHub labels: bug

**Describe the bug**
There is a typo in the huggingface repository. I want to point out that the naive chatbot works but the other two examples do not.

Link: https://huggingface.co/AutoRAG

<img width="1074" height="288" alt="Image" src="https://github.com/user-attachments/assets/08ec36b4-67f1-4570-b763-cd699f4b2bbb" />
Repository: Marker-Inc-Korea/AutoRAG
Issue: Enabling test data evaluation
Root cause: Unimplemented Feature Gap

You can easily extract pipeline and use it for deployment. 
That yaml file can be use for test data evaluation, too. 

### Things to do
- [x] Skip Evaluation at query expansion, prompt maker when there is only one module in node.
- [x] Enabling using target_module_params at run.py for evaluation of hybrid retrieval.
Repository: Marker-Inc-Korea/AutoRAG
Issue: [HotFix] Fix that hybrid_cc return exactly top_k.
Root cause: Algorithmic Error

If there is no duplication of retrieved_ids, hybrid_cc drop that row as default.
Because of that, we have empty retrieved_contents at hybrid_cc and it occurs error at rerank or others.

To solve this, we have to make methods to hedge this kind of error. Like fill the scores to 0 like hybrid_rrf.
Repository: Marker-Inc-Korea/AutoRAG
Issue: Add env variable for easy-to-use 
Root cause: Unimplemented Feature Gap

When user needs to write their api key or api base directly to yaml, it is much better to use env varaible.
But, yaml doesn't support env variable as default.
So, it will be great we support env variable at config yaml file.

Below is the phind answer for making env varaible in yaml file.

---

To use environment variables inside a YAML file when loading it with PyYAML's `safe_load()` function, you need to add custom logic to find and resolve these environment variables. This involves defining a pattern to identify environment variables, creating a custom constructor to replace placeholders with actual environment variable values, and registering this constructor with PyYAML.

Here's a step-by-step guide based on the provided sources:

1. **Define a Pattern to Identify Environment Variables**: Use a regular expression to match your environment variable placeholders. For example, if your placeholders look like `${ENV_VAR}`, you can use the following pattern:
   ```python
   import re
   env_pattern = re.compile(r".*?\${(.*?)}.*?")
   ```

2. **Create a Custom Constructor**: This function will be called by PyYAML when it encounters a node that matches the pattern defined in the previous step. It should replace the placeholder with the actual value of the environment variable:
   ```python
   import os
   def env_constructor(loader, node):
       value = loader.construct_scalar(node)
       matches = env_pattern.findall(value)
       for match in matches:
           value = value.replace(f"${{{match}}}", os.environ.get(match, ""))
       return value
   ```

3. **Register the Pattern and Constructor with PyYAML**: Before calling `safe_load()`, you need to tell PyYAML about your custom pattern and constructor. You also need to specify a custom tag (e.g., `!env`) to trigger your constructor:
   ```python
   import yaml
   yaml.add_implicit_resolver("!env", env_pattern)
   yaml.add_constructor("!env", env_constructor)
   ```

4. **Load Your YAML File**: Now, you can load your YAML file. Any occurrences of `${ENV_VAR}` will be replaced with the corresponding environment variable's value:
   ```python
   with open('your_file.yaml', 'r') as file:
       config = yaml.safe_load(file)
       print(config)
   ```

Here's an example of what your YAML file might look like:
```yaml
config:
  username: admin
  password: ${SERVICE_PASSWORD}
  service: https://${SERVICE_HOST}/service
```

And here's how you might set the environment variables in a Unix-like shell before running your Python script:
```shell
export SERVICE_PASSWORD=yourpassword
export SERVICE_HOST=yourhost.com
```

This approach allows you to dynamically configure your application based on the environment it's running in, making it easier to manage configurations across different deployment environments without hardcoding sensitive information into your codebase or YAML files.
Repository: Marker-Inc-Korea/AutoRAG
Issue: Support GPU at passage reranker modules
Root cause: Resource Mishandling

I wonder `tart` support CUDA well... 
And other reranker modules has not tested yet.

problem at upr.py
I got this error when I run it at gpu server.

RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument index in method wrapper_CUDA__index_select)

```
attention_mask_view = context_attention_mask[i: i + shard_size]                                                        │
                             │   127 │   │   decoder_tensor_view = question_tensor[i: i + shard_size]                                                               │
                             │   128 │   │   with torch.no_grad():                                                                                                  │
                             │ ❱ 129 │   │   │   logits = model(input_ids=encoder_tensor_view,                                                                      │
                             │   130 │   │   │   │   │   │      attention_mask=attention_mask_view,                                                                 │
                             │   131 │   │   │   │   │   │      labels=decoder_tensor_view).logits
```

Phind Answer:
https://www.phind.com/search?cache=fuqrhd654zolq5sp3m1wps3e
Repository: Marker-Inc-Korea/AutoRAG
Issue: Add new metrics for generator evaluation ‘SEMSCORE'
Root cause: Unimplemented Feature Gap

[paper](https://arxiv.org/pdf/2401.17072.pdf)

### Overview
SemScore is a simple metric. Just embeds generate ground truth and model’s generation, and calculates cosine similarity.

Even it is quite simple, correlation with human evaluation and SemScore evaluation is quite high comparing between other evaluation scores. 
We knew that traditional generation evaluation metrics are quite bad. This can be the great solution for us.  

## To-do list
- [x] add [all-mpnet-base-v2](https://huggingface.co/sentence-transformers/all-mpnet-base-v2) embedding to support embedding models
- [x] Get metrics via dictionary with their options. Enabling this will solve the issue #44 either.
- [x] Make SemScore metrics. Default embedding model will be ‘all-mpnet-base-v2’, but I’m sure other embdding model can be effective.
- [x] Make documentation for this metric. (We don’t have documentation for selecting rigth strategy for each module)
Repository: Marker-Inc-Korea/AutoRAG
Issue: Refactor all test codes for fast testing and auto testing
Root cause: Architectural Constraint

We disabled our test CI because of the slow speed plus storage limitation.
We have to optimize this.
For example, LlamaIndex is large and complex framework with hundreds of unit test, but it takes only 19 seconds to test all things.
However, ours was more than 5 minutes and it is now not working at default ubunutu runner in github.

The suggest workflow is like below.

1. Do not use real LLM. Just use `MockLLM` for mocking the functionality of LLM. 
2. Use MonkeyPatch to using LLM or local Model. Testing and donwloading whole model like TART or MonoT5 tooks long time and large download size.
3. Do not use real embedding model. Use Mock embedding model or monkey patch.

After making test faster and lighter, we will enable auto-testing with CI again.

https://moons08.github.io/programming/pytest_2/

Plus, from LlamaIndex v0.10.0, `MockLLM` turns to deprecated. (Legacy code)
Consider making OpenAI MonkeyPatch function for testing.
Repository: Marker-Inc-Korea/AutoRAG
Issue: LlamaIndex Version Upgrade to v0.10.1
Root cause: External Dependency Breakage

There was a really huge version upgrade at LlamaIndex.
Since we are using lots of utils from LlamaIndex, it is inevitable to migrate to v0.10.1

- [Migration Guide](https://github.com/run-llama/llama_index/pull/10537)

Before this, the new installed user might have problems.
Repository: Marker-Inc-Korea/AutoRAG
Issue: Divide cli options to cli.py
Root cause: Architectural Constraint

There are cli options at `evaluator.py` from now.

- Make cli.py at autorag root folder
- Move all cli to `cli.py` file.

This needs to perform before you start to make easy-to-start CLI form for newbies.
Because it uses cli, and needs to be separate because we have more than three cli commands now.
Repository: Marker-Inc-Korea/AutoRAG
Issue: [Hotfix] fix Tart module error
Root cause: Resource Mishandling

i got this error 
"Expected all tensors to be on the
 same device, but found at least two devices,                     
 cuda:0 and cpu! (when checking argument for                      
 argument index in method                                         
 wrapper_CUDA__index_select)"
Repository: Marker-Inc-Korea/AutoRAG
Issue: Runner with vectordb ingest all again
GitHub labels: bug

wontfix

When I use Runner with vectordb, it seems like vectordb ingest all thing.
Why is that? 
I get numerous warning about add of existing id from hnsw.


https://github.com/chroma-core/chroma/issues/1733

It actually did not ingest anything. 

## How to Fix this?

Go to your lib folder and go to here `chromadb/segment/impl/vector/local_persistent_hnsw.py`

At the line of 271, change to this

```
else:
    pass
    # logger.warning(f"Add of existing embedding ID: {id}")
```

Then the awful logging will vanish.

I think I will submit this at troubleshooting docs.

Closing this issue since this is not a AutoRAG bug.
Repository: Marker-Inc-Korea/AutoRAG
Issue: Multi-turn chat support
Root cause: Unimplemented Feature Gap

It is hard and really vague issue name.
But, the real RAG must have multi-turn chat. 
Have conversation history and use it. 

How can we cover this?
The creation of dataset.
The deployment. 
The optimization.

Everything can be obstacles to implement multi-turn chat.

Think it can be a major update.
Repository: Marker-Inc-Korea/AutoRAG
Issue: change guidance requriements to follow github main branch
Root cause: External Dependency Breakage

Due to the late deployment of guidance team + error about ‘tests’ folder that install guidance, it can be occur error when we use guidance pypi latest version.
The github main branch resolve this issue at their PR, so it will be great we follow guidance github version since they deploy current version to pypi.
Repository: Marker-Inc-Korea/AutoRAG
Issue: unallocate gpu CUDA memory after processing each reranker, metric model, embedding model
Root cause: Resource Mishandling

At intense optimization, lots of models allocate to gpu memory.
This happens in single process, so it is hard to orchestrate each models. 
So it must unallocate gpu cuda memory when each node(or module) is done.

Embedding model must implement #207 first.
So, in this issue, I will not implement unallocating embedding models.
Repository: Marker-Inc-Korea/AutoRAG
Issue: late init embedding model
Root cause: Architectural Constraint

When embedding model initialization, it is likely to be downloaded all embedding models. 
It is super inefficient since someone don't want to use local embedding, or only one embedding.

[What to fix]
- All embedding model instances is initializing at autorag/__init__.py. Late initialization all embedding models.

[How to]
- Suggest to search about `python late init`
Repository: Marker-Inc-Korea/AutoRAG
Issue: Prevent tokenizer warning 
Root cause: Configuration Error

[What is wrong?]
I faced a lot of warning when using tokenizer.
```
Be aware, overflowing tokens are not returned for the setting you have chosen, i.e. sequence pairs with the 'longest_first' truncation strategy. So the returned list will always be empty even if some tokens have been removed.
```

And I could find the issue about it. https://github.com/huggingface/transformers/issues/14285

Prevent this annoying warning!

Face this issue at passage reranker node execution.
Repository: Marker-Inc-Korea/AutoRAG
Issue: Add vllm 'llm module' for faster generation performance
Root cause: Architectural Constraint

LlamaIndex supports `vllm` model. However, it is too slow at batch processing. It literally do not support async inference.
Look at my issue at [here](https://github.com/run-llama/llama_index/issues/10496).
So, it does not support any `async` call and it is super slow. It is basically linear process.

I looked at both `llama_index` and `vllm` code, and I found this at vllm code.
```python
def generate(
        self,
        prompts: Optional[Union[str, List[str]]] = None,
        sampling_params: Optional[SamplingParams] = None,
        prompt_token_ids: Optional[List[List[int]]] = None,
        use_tqdm: bool = True,
        lora_request: Optional[LoRARequest] = None,
    ) -> List[RequestOutput]:
```
You can see that you can put multiple prompts at once. 
But look at `llama_index.complete()` method.
```python
    def complete(
        self, prompt: str, formatted: bool = False, **kwargs: Any
    ) -> CompletionResponse:
        kwargs = kwargs if kwargs else {}
        params = {**self._model_kwargs, **kwargs}

        from vllm import SamplingParams

        # build sampling parameters
        sampling_params = SamplingParams(**params)
        outputs = self._client.generate([prompt], sampling_params)
        return CompletionResponse(text=outputs[0].outputs[0].text)
```

Just using "only first" index of input list.

Think about any ML model. You can multiple inputs as batch, and it process async at "GPU" level, until it is not occur OOM error.
I will set proper batch and put to vllm instance directly, since there is **no way** to support real async operation of Vllm model. 
Because their abstraction methods only supports "single prompt" as input.

In my opinion, the optimization of processing speed itself is crucial to AutoRAG. Because it takes so....long time;; I had to optimize it.
Repository: Marker-Inc-Korea/AutoRAG
Issue: Surpass context length when calculating sem_score.
Root cause: Validation Gap

[What is the problem?]
While using sem_score with default mp_net or other local model, you can face ‘AssertionError’ when running in CUDA.
It is because input sentence surpass its context length limitation. 
We have to prevent this kind of error.

[How to?]
I think there is a few method to prevent this. 
1. Max length option?
2. Auto-retry
3. 1+2
Or any other solution will be welcome. Choose wisely.

It looks like LlamaIndex supports this kind of feature as default in `HuggingFaceEmbedding`.

```python
encoded_input = self._tokenizer(
            sentences,
            padding=True,
            max_length=self.max_length,
            truncation=True,
            return_tensors="pt",
        )
```
And the `self.max_length` comes from here.

```python
        if max_length is None:
            try:
                max_length = int(self._model.config.max_position_embeddings)
            except AttributeError as exc:
                raise ValueError(
                    "Unable to find max_length from model config. Please specify max_length."
                ) from exc
```

Plus it looks like there is no problem at CPU embedding.

It was simple. The mp-net model config.json itself was the problem...
Repository: Marker-Inc-Korea/AutoRAG
Issue: Add "AutoRetrieval" as retrieval module.
Root cause: Unimplemented Feature Gap

https://docs.llamaindex.ai/en/stable/examples/vector_stores/chroma_auto_retriever.html

`AutoRetrieval` is a retrieval module that LLM make 'metadata' search query with given user query and metadata descriptions.
User must prepare detailed metadata at corpus.parquet. 
The core part of llm that selected and make search query, follow llama index code.
For searching it, making custom function for filtering from corpus.parquet needed.
Repository: Marker-Inc-Korea/AutoRAG
Issue: Add openai output token support at generator module
Root cause: Unimplemented Feature Gap

part of the #63 

LlamaIndex openai model does not support logprob and output tokens. 
We might need to support it for using various log-prob based modules.

You can see `g_eval` method how to get logprob from openai API call.

Don't have to get log prob from now.

- First, Token support.
- Second, Input token truncate by model's context length.

Third, no error because of the side-effect.
Dependency only in `opneai-python` library.
Repository: Marker-Inc-Korea/AutoRAG
Issue: Add 'custom multi' query expansion module
Root cause: Unimplemented Feature Gap

There is a [MultiQueryRetriever](https://python.langchain.com/docs/modules/data_connection/retrievers/MultiQueryRetriever) in Langchain.
It is kind of 'query expansion', which can generate several queries from given user query.

There is a default prompt, but user can write their own 'query expansion' prompt. 

From now, we only have 'hyde' and 'query_decompose' module and do not have corresponding module from `MutliQueryRetriever`.

To use this, user can adapt any LLM using query expansion method easily, just change prompt in config YAML file.

[How to Suggestion]
It can be great just slice LLM generation to line by line (using \n)
Because this module must generate more than one question, it is important to parse LLM generation.