zhiminy/EvalEng huggingface dataset:
Repository: Marker-Inc-Korea/AutoRAG
Issue: 'Lost in the Middle' reorder module (or possible new node) (or possible new config option for reranker & retrieval)
Root cause: Unimplemented Feature Gap

[Long-context Reorder](https://python.langchain.com/docs/modules/data_connection/retrievers/long_context_reorder) from Langchain.

In the `Lost in the Middle` paper, it is well-known that reordering high-relevance context to be first and the last order in the context prompt. 

[How to?]
It can be four way.

1. Add as Reranker module. => not suggest
Its feature is simillar with Reranker because it reorder passage order.
But, in the real world scenario, it is so common to use both reranker and this 'reorder module'.

2. Add as new node => maybe? But I don't like it.
It is reordering passages, but it uses only scores from retrieval or reranking. 
It can be whole new node.
However, the cons is that it can be enforce so much complexity to choose node in config YAML file.

3. Add as 'retrieval' and 'rerank' option (or strategy)
It is possible to add as optional parameter at retrieval and reranker nodes.
It can be a new 'node parameter'
But my question is, how to test 'use lost in the middle' or not use? If we want to test this, it can be super complex at `run.py` code, because we have to test it with generator node and its metrics.

4. Add as 'prompt maker' module.
It can resolve above issue about 'evaluation' problem.
However, from now we don't have `scores` as input of prompt maker.
So we have to add `scores` as input parameter of prompt maker for making this module as prompt maker module.


I think the best option is 'prompt maker' module... What do you think?
Need to talk about it more.
Repository: Marker-Inc-Korea/AutoRAG
Issue: OpenAI embedding surpass token limit error
Root cause: Validation Gap

When you try to embed over 8192 tokens, there is an error at openai embedding. 
Huggingface embedding manage itself automatically its token size, but maybe openai embedding does not do that.

It can resolve in our codebase temporaly, but I think we can contribute to llama index. 

![1711161029285.png](https://github.com/Marker-Inc-Korea/AutoRAG/assets/28955029/d23c427a-1558-48dc-a013-4fa7ca4a5d69)



There is no such feature that calculate each token size and do not execute embedding in LlamaIndex.
"Why.....?"
Repository: Marker-Inc-Korea/AutoRAG
Issue: Use generator modules at `hyde` and `question_decompose` module.
Root cause: Unimplemented Feature Gap

Now, it uses LLM to decompose or making hypothetical queries.
It often really slow to process it since we use default LlamaIndex only.
So, it can be great we can use generator modules like `vllm` at query_expansion modules.

We did this!
Repository: Marker-Inc-Korea/AutoRAG
Issue: Can’t calculate retrieval scores on multiple OR ground truths
Root cause: Validation Gap

While benchmarking triviaQA, the weird thing happend. 

![image](https://github.com/Marker-Inc-Korea/AutoRAG/assets/28955029/86112de8-9c0a-4135-83ee-6e86445b9582)

As you see, there are duplicated id with ground truth and the retrieved_ids. But all metrics got 0. This is not just one row, for the all row.
Something wrong at the retrieval metrics. Need to inspect carefully.

At first, the metric itself has no problem. We cover this case in our test code. And nothing matters. Something happened in `run.py`

I find the reason.

Because the `retrieval_gt` on trivia qa is 1-d list. And there is nothing to cast qa retrieval gt to 2-d list at the start of evaluator. 
The `run.py` get retrieval_gt from data folder. It always occur error. 

At conclusion, I must add 'cast_qa' data function to start of data folder parquet files, too.
Repository: Marker-Inc-Korea/AutoRAG
Issue: Autorag run_web cli relative path. Hard to execute it.
Root cause: Validation Gap

```
autorag run_web --trial_path ./project/0 --project_dir ./project
```
This works fine, but.

```
autorag run_web --trial_path ./project/0/ --project_dir ./project
```
This is not working. It can't find bm25 path.

This is awful for usability.
Repository: Marker-Inc-Korea/AutoRAG
Issue: Properly tokenize LlamaIndex generator modules
Root cause: Unimplemented Feature Gap

From now, we tokenize each answers using 'gpt2' tokenizer, and it is not selectable.
For more accuracy, we have to tokenize each generator results using corresponding tokenizer to LLM models.

It can be useful to use tokenlog library.
But have to implement LlamaIndex support at tokenlog library.

Last question is...
Is it worth it?
Repository: Marker-Inc-Korea/AutoRAG
Issue: Implement RAGAS test data generation to AutoRAG (or bridge for convert)
Root cause: Unimplemented Feature Gap

https://docs.ragas.io/en/stable/concepts/testset_generation.html

RAGAS already supports great ‘synthetic evaluation data generation’
Some modification will be great for AutoRAG evaluation data creation. 

Their code is Apache 2.0 license like ours, so modification is totally okay!
Repository: Marker-Inc-Korea/AutoRAG
Issue: Add LLM Reranker
GitHub labels: New Module


This LLM reranker from LlamaIndex not working well.
The LLM return only ‘one’ passage, not full passages. 

It means, if I put three passages it selects only the best one, and ignore the rest ones.

Since it is crucial to reranking, I decided to not implement this module.

Closing this issue.
Repository: Marker-Inc-Korea/AutoRAG
Issue: [Bug] Error at testing flag_embedding_reranker
Root cause: External Dependency Breakage

```python
    @passage_reranker_node
    def flag_embedding_reranker(queries: List[str], contents_list: List[List[str]],
                                scores_list: List[List[float]], ids_list: List[List[str]],
                                top_k: int, batch: int = 64, use_fp16: bool = False,
                                model_name: str = "BAAI/bge-reranker-large",
                                ) -> Tuple[List[List[str]], List[List[str]], List[List[float]]]:
        """
        Rerank a list of contents based on their relevance to a query using BAAI Reranker model.
    
        :param queries: The list of queries to use for reranking
        :param contents_list: The list of lists of contents to rerank
        :param scores_list: The list of lists of scores retrieved from the initial ranking
        :param ids_list: The list of lists of ids retrieved from the initial ranking
        :param top_k: The number of passages to be retrieved
        :param batch: The number of queries to be processed in a batch
        :param use_fp16: Whether to use fp16 for inference
        :param model_name: The name of the BAAI Reranker model name.
            Default is "BAAI/bge-reranker-large"
        :return: tuple of lists containing the reranked contents, ids, and scores
        """
        device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
>       model = FlagReranker(
            model_name, use_fp16=use_fp16, device=device
        )
E       TypeError: __init__() got an unexpected keyword argument 'device'

../../../../autorag/nodes/passagereranker/flag_embedding.py:32: TypeError
```

I am using FlagEmbedding==1.2.8 and m2 pro
Please fix this error ASAP. After that, I can release 0.1.2 version.
Repository: Marker-Inc-Korea/AutoRAG
Issue: [Question]: OpenAILike with custom OPENAI_API_BASE
Root cause: Documentation Deficiency

Hi, 

is it possible to use custom LLMs hosted privately through the `OpenAILike` object of LlamaIndex with a custom `OPENAI_API_BASE` ? 

I saw the possibility of defining `openailike` in [the docs](https://marker-inc-korea.github.io/AutoRAG/local_model.html#supporting-llm-models) but with no further information regarding the private host.

Thanks! :)

@omar-araboghli 
Of course you can. You can put any parameters from LlamaIndex `OpenAILike` LLM class. [LlamaIndex Docs](https://docs.llamaindex.ai/en/v0.9.48/api_reference/llms/openai_like.html)

```
nodes:
  - node_line_name: node_line_1
    nodes:
      - node_type: generator
        modules:
          - module_type: llama_index_llm
            llm: openailike
            model: mistralai/Mistral-7B-Instruct-v0.2
            api_base: your_api_base
            api_key: your_api_key
```

I think we need update docs for clarifying this. @bwook00 

Thanks @vkehfdl1, I'll give it a try!
Repository: Marker-Inc-Korea/AutoRAG
Issue: Add 'Time Reranker' module
Root cause: Unimplemented Feature Gap

For keeping recency information, this `time_reranker` module simply reranks a retrieved passage with datetime. 
It uses a last_modified_datetime that in the corpus.parquet. 
User can specify which metadata column they want to use for reranks by time. 
It have to be an datetime object.
Repository: Marker-Inc-Korea/AutoRAG
Issue: Error when using Colbert Reranker at CUDA enabled device
Root cause: Resource Mishandling

```
RuntimeError: Expected all tensors to be on the same                   
                             device, but found at least two devices, cuda:0 and cpu!                
                             (when checking argument for argument index in method                   
                             wrapper_CUDA__index_select)  
```
Repository: Marker-Inc-Korea/AutoRAG
Issue: Refactor for parallel processing that using CUDA at model-based Rerankers
Root cause: Algorithmic Error

From now, all rerankers put a query to reranker model ‘one by one’ 
It uses ‘async’, but it is not effective since the key to the parallize using model is putting model itself.

So, we have to refactor all rerankers that supports ‘real parralell’ processing. Put enough batch size ‘query and context’ set to model, and gather the results at once.

- [x] #304
- [x] #305
- [x] #306
- [x] #307
- [x] #308
- [x] #309

Close this since we parallel processing all rerankers
Repository: Marker-Inc-Korea/AutoRAG
Issue: Parallel processing at Colbert Reranker
Root cause: Algorithmic Error


It looks like the real problem is not `batch` in here. The parallel code itself looks wrong. 

Look at #326.

You can see the original code, and the new code that you pushed.

The original code is like below.

```python
def get_colbert_score(query: str, content_list: List[str],
                      model, tokenizer) -> List[float]:
    query_encoding = tokenizer(query, return_tensors="pt")
    query_embedding = model(**query_encoding).last_hidden_state
    rerank_score_list = []

    for document_text in content_list:
        document_encoding = tokenizer(
            document_text, return_tensors="pt", truncation=True, max_length=512
        )
        document_embedding = model(**document_encoding).last_hidden_state

        sim_matrix = torch.nn.functional.cosine_similarity(
            query_embedding.unsqueeze(2), document_embedding.unsqueeze(1), dim=-1
        )

        # Take the maximum similarity for each query token (across all document tokens)
        # sim_matrix shape: [batch_size, query_length, doc_length]
        max_sim_scores, _ = torch.max(sim_matrix, dim=2)
        rerank_score_list.append(torch.mean(max_sim_scores, dim=1))

    return list(map(float, rerank_score_list))
```

The colbert model acts like 'embedding_model' in here. The `last_hidden_state` is embedding vector. 
It gets cosine similarity between each query_embedding and each content_embedding. And rerank contents by that score.

Return to your code. It is much easier you look at #326

The first thing you did wrong is tokenize query and contents at once. 

```python
# Tokenize both queries and contents together
        feature = tokenizer(flattened_batch_queries, flattened_batch_contents, padding=True, truncation=True,
                            return_tensors="pt").to(device)
```
If you decode the `feature` you can get the result like below.

```
'[CLS] what is the capital of france? [SEP] nomadamas is great team [SEP]'
```
It means, it encodes the query and content in the one sentence. And you put this feature into model 'at once'. 
I think you try to split the result of embedding at the next code, but it does not work that way at all.
Model creates embedding of the tokenized sentence. It did not create embedding each sentence separately, it embeds a whole input string. So splitting it after embedding seems useless.

In conclusion, the code itself after #326 is wrong and might get the bad results and occurs several errors.
Repository: Marker-Inc-Korea/AutoRAG
Issue: Add ASCII art at the begging of trial
GitHub labels: enhancement

like this!

```
                _        _____            _____ 
     /\        | |      |  __ \     /\   / ____|
    /  \  _   _| |_ ___ | |__) |   /  \ | |  __ 
   / /\ \| | | | __/ _ \|  _  /   / /\ \| | |_ |
  / ____ \ |_| | || (_) | | \ \  / ____ \ |__| |
 /_/    \_\__,_|\__\___/|_|  \_\/_/    \_\_____|
                                                
                                               
```

https://patorjk.com/software/taag/#p=display&f=Big&t=AutoRAG

It will be awesome haha

That's really great. I like it
Repository: Marker-Inc-Korea/AutoRAG
Issue: check api keys when start trial, and if don't detected, get as input.
Root cause: Unimplemented Feature Gap

If the user don't read the docs carefully, they sometimes miss to set up api key for their modules.
It is super common for using openai, and when they include external service like cohere or JinaAI, it is even harder to setup all of things.

So, when starting trial, automatically checks the valid api key was given, and if not, get the api key as input from user directly.

There are two options for using api key.
1. As env variable.
2. As module parameter `api_key` or something other.


This will enhance user's experience.
Repository: Marker-Inc-Korea/AutoRAG
Issue: pypi CD at github actions
Root cause: Unimplemented Feature Gap

When merge release branch to main, automatically deploy package to pypi.

https://www.phind.com/search?cache=e5p8wmuqfgiceiv8oocbdrqr

https://velog.io/@bailando/github-actions-%EC%9C%BC%EB%A1%9C-pypi-%ED%8C%A8%ED%82%A4%EC%A7%80-%EB%B0%B0%ED%8F%AC-%EC%9E%90%EB%8F%99%ED%99%94%ED%95%98%EA%B8%B0

We implemented it
Repository: Marker-Inc-Korea/AutoRAG
Issue: ChromaDB: Batch size exceeds maximum batch size using collection.add function
Root cause: Validation Gap

I cannot submit more than 41,666 embeddings at once with chromadb. All embeddings are lost (made me waste money). It would be useful to cache embeddings somewhere.

chromadb issue was discussed [here](https://github.com/chroma-core/chroma/issues/1049)

@JulianLopezB Thank you for reporting chromadb error. I didn't know about this kind of error.
I will fix ingest code for resolve this. Thanks.
Repository: Marker-Inc-Korea/AutoRAG
Issue: Mock Cohere, JinaAI API key related test files
Root cause: Unimplemented Feature Gap

When PR rely on github secrets, outside collaborators can't merge their commit to our repo easily.

Including #377

In this issue, I will cover only Cohere and JinaAI API key related test files.
Do #377 about OPENAI API KEY
Repository: Marker-Inc-Korea/AutoRAG
Issue: Refactor run_prompt_maker_node
Root cause: Resource Mishandling

Now, the evaluation of prompt maker rely on generator node.
Prompt maker requires super-low resources, generator node can require super high resource for running local LLM.
There can be CUDA OOM error when initializing generator node and deleting it. 

So, it will be great to initialize generator module just once if there is a single module and param input. 
And inference at once to generator module for optimizing memory and resource usage.
Repository: Marker-Inc-Korea/AutoRAG
Issue: Flag Reranker already supports batch_size for parallel processing

In `flag_reranker.py` from `FlagReranker` class library code.

```python3
    @torch.no_grad()
    def compute_score(self, sentence_pairs: Union[List[Tuple[str, str]], Tuple[str, str]], batch_size: int = 256,
                      max_length: int = 512, normalize: bool = False) -> List[float]:
        if self.num_gpus > 0:
            batch_size = batch_size * self.num_gpus

        assert isinstance(sentence_pairs, list)
        if isinstance(sentence_pairs[0], str):
            sentence_pairs = [sentence_pairs]

        all_scores = []
        for start_index in tqdm(range(0, len(sentence_pairs), batch_size), desc="Compute Scores",
                                disable=len(sentence_pairs) < 128):
            sentences_batch = sentence_pairs[start_index:start_index + batch_size]
            inputs = self.tokenizer(
                sentences_batch,
                padding=True,
                truncation=True,
                return_tensors='pt',
                max_length=max_length,
            ).to(self.device)

            scores = self.model(**inputs, return_dict=True).logits.view(-1, ).float()
            all_scores.extend(scores.cpu().numpy().tolist())

        if normalize:
            all_scores = [sigmoid(score) for score in all_scores]

        if len(all_scores) == 1:
            return all_scores[0]
        return all_scores
```

It supports batch_size already. Real model computation happens in here.
Repository: Marker-Inc-Korea/AutoRAG
Issue: Any great method to keep update sample YAML file for the latest modules and nodes?
GitHub labels: help wanted

There are several obstacles to keep on update sample YAML file to the latest. 
Plus, it is quite little bit hard to user find the sample files easily.

@bwook00 @Eastsidegunn  Any idea?

I want below things.

The role model is certainly 'Github Actions' YAML marketplace. It is truly amazing for just sharing YAML file across the platform.

1. Easily keep on update. Do not need complicate edit (or commit&push&review).
2. Always check before version release. 
3. Versioning of each YAML files.

What is the simplest yet effective way?
Repository: Marker-Inc-Korea/AutoRAG
Issue: [Passage Augmenter] Do not need to put top_k less than previous result's retrieved passages

```python
# get top_k
top_k = kwargs.pop("top_k")
assert top_k <= len(ids[0][0]), ("The number of top_k must be same or less than the number of retrieved "
                                 "passages.")
```
In `nodes/passageaugmenter/base.py`
I think this assert is not needed, right?

@bwook00
Repository: Marker-Inc-Korea/AutoRAG
Issue: Decorator that convert pd.Series or np.ndarray to list recursively.
Root cause: Unimplemented Feature Gap

In autorag, we use python default list when data processing.
When we use pandas or numpy, it is really easy that lists convert to np.array, tuple, or pd.Series, and it occurs unexpect error easily.

Develop decorator that converts all Iterable types to list, even the 2-d or 3-d list recursively. 
And using it on the functions for preventing that kind of errors.
Repository: Marker-Inc-Korea/AutoRAG
Issue: [HotFix] [threshold cutoff] Error when the input 'scores_list' is np.ndarray.
Root cause: Validation Gap

In the running of AutoRAG, the input of `threshold_cutoff_pure` function's parameter `scores_list` is np.ndarray, it occurs error.

```python
def threshold_cutoff_pure(scores_list: List[float],
                          threshold: float,
                          reverse: bool = False) -> List[int]:
    if reverse:
        remain_indices = [i for i, score in enumerate(scores_list) if score <= threshold]
        default_index = scores_list.index(min(scores_list)) # error here
    else:
        remain_indices = [i for i, score in enumerate(scores_list) if score >= threshold]
        default_index = scores_list.index(max(scores_list)) # error here

    return remain_indices if remain_indices else [default_index]
```

We will fix this kind of error with #438 later.
Repository: Marker-Inc-Korea/AutoRAG
Issue: Add n-gram metric options
Root cause: Unimplemented Feature Gap

From PR #40 

I think these n-gram based metrics can open some options.
- N : int  minimum unit for n-gram
- tokenizer : select tokenizer. (according to language, method etc include in tokenizer, tokenizer can influence value of each metric)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
- etc


---

- [x] #222
- [x] #223

How can get and write option about metrics at yaml file? It can be tricky, but if it is necessary, we'll find a way.

close this issue because all generation metric has options. 
DONE!
Repository: Marker-Inc-Korea/AutoRAG
Issue: This error happens frequently on auto-testing CI
Root cause: Resource Mishandling

```
FAILED tests/autorag/test_cli.py::test_restart_evaluate - AssertionError: assert False
 +  where False = <function exists at 0x7f4843d71f30>('/tmp/tmpq4xuk7vn/1/summary.csv')
 +    where <function exists at 0x7f4843d71f30> = <module 'posixpath' from '/usr/lib/python3.10/posixpath.py'>.exists
 +      where <module 'posixpath' from '/usr/lib/python3.10/posixpath.py'> = os.path
 +    and   '/tmp/tmpq4xuk7vn/1/summary.csv' = <function join at 0x7f4843d727a0>('/tmp/tmpq4xuk7vn/1', 'summary.csv')
 +      where <function join at 0x7f4843d727a0> = <module 'posixpath' from '/usr/lib/python3.10/posixpath.py'>.join
 +        where <module 'posixpath' from '/usr/lib/python3.10/posixpath.py'> = os.path
===== 1 failed, 182 passed, 41 skipped, 326 warnings in 123.71s (0:02:03) ======
```
Repository: Marker-Inc-Korea/AutoRAG
Issue: Refactor method that save best module and best param
Root cause: Architectural Constraint

Now, we save which module and param is best to the best filename directly.
But, it will occur some unintended errors when param or module name has underscore (_) or hyphen (-).
So, I have to find a way how to notify which module and module params are selected in node run function.
Repository: Marker-Inc-Korea/AutoRAG
Issue: Add tonic validate metric with local model.
Root cause: Unimplemented Feature Gap

no_gt retrieval metrics needs large amount of LLM processing.
So, use local LLM model to compute it.

+ ragas context precision need so much LLM calls. So, try to use tonic validate instead.

Do this on 'aragog' branch.
No plan to integrate this to a main branch.
Repository: Marker-Inc-Korea/AutoRAG
Issue: vllm does not remove VRAM cache at multi-gpu environment
Root cause: Resource Mishandling

[data.zip](https://github.com/user-attachments/files/15604248/data.zip)

This is the zip file that happened this error.


![image](https://github.com/Marker-Inc-Korea/AutoRAG/assets/28955029/7102a234-17c3-4d40-bb9c-f9aae0f32fea)


We don't have multi-gpu environment, so this bug is hard to reproduce.