zhiminy/EvalEng huggingface dataset:
Repository: Marker-Inc-Korea/AutoRAG
Issue: [BUG] Error on building GUI installation
Root cause: Environment Incompatibility

**Describe the bug**
Error encoutering "ENOENT: no such file or directory, ..." on ```npm run build```

**To Reproduce**
Steps to reproduce the behavior:
on step to building and running  GUI application, 

```npm run start```

returns following error,
Error: ENOENT: no such file or directory, open '~/AutoRAG/autorag-frontend/.next/server/app/projects/[project_id]/trials/[trial_id]/parquet_wasm_bg.wasm'

**Expected behavior**
A clear and concise description of what you expected to happen.

**Full Error log**
```
 % npm run start

> next-app-template@0.0.1 start
> next start

   ▲ Next.js 15.1.6
   - Local:        http://localhost:3000
   - Network:      http://172.16.21.27:3000

 ✓ Starting...
 ✓ Ready in 155ms
Error: ENOENT: no such file or directory, open '~/AutoRAG/autorag-frontend/.next/server/app/projects/[project_id]/trials/[trial_id]/parquet_wasm_bg.wasm'
    at 52342 (.next/server/app/projects/[project_id]/trials/[trial_id]/page.js:4:678)
    at t (.next/server/webpack-runtime.js:1:142)
    at 84163 (.next/server/app/projects/[project_id]/trials/[trial_id]/page.js:3:18254)
    at Function.t (.next/server/webpack-runtime.js:1:142) {
  errno: -2,
  code: 'ENOENT',
  syscall: 'open',
  path: '~/AutoRAG/autorag-frontend/.next/server/app/projects/[project_id]/trials/[trial_id]/parquet_wasm_bg.wasm'
}
 ⨯ unhandledRejection:  Error: ENOENT: no such file or directory, open '~/AutoRAG/autorag-frontend/.next/server/app/projects/[project_id]/trials/[trial_id]/parquet_wasm_bg.wasm'
    at 52342 (.next/server/app/projects/[project_id]/trials/[trial_id]/page.js:4:678)
    at t (.next/server/webpack-runtime.js:1:142)
    at 84163 (.next/server/app/projects/[project_id]/trials/[trial_id]/page.js:3:18254)
    at Function.t (.next/server/webpack-runtime.js:1:142) {
  errno: -2,
  code: 'ENOENT',
  syscall: 'open',
  path: '~/AutoRAG/autorag-frontend/.next/server/app/projects/[project_id]/trials/[trial_id]/parquet_wasm_bg.wasm'
}
```

**Desktop (please complete the following information):**
 - OS: MacOS
 - Python version [e.g. 3.12]



Have you tried reinstalling it again? Delete `.next` and `node_module` and try again. 😃 



Still not working ;(

@caisarl76 
Hello!!
Apology to the late reply.

Did you run `npm install` right?

I will investigate the issue.

@caisarl76 
Sorry for confusion.
You can try to re-install modules using yarn.

```
yarn install
```

@caisarl76 
In #1091 I added a GUI application at the docker compose also!
Just run `docker compose up -d` and the GUI will be successfully build.

Thank you.
I will close this issue.
Repository: Marker-Inc-Korea/AutoRAG
Issue: How to use the Custom embedding?
Root cause: Documentation Deficiency

Hello, 
Thank you for taking a trouble. 

I want to use the sentence transformer custom embedding, so I tried to configure the main and yaml file. 
I refer to [documentaion](https://medium.com/@autorag/making-benchmark-of-different-embedding-models-1c327a0dae1f), and write the code below;


`main.py`
```
import autorag
from llama_index.embeddings.huggingface import HuggingFaceEmbedding

if not hasattr(autorag, "embedding_models"):
    autorag.embedding_models = {}

# 🔹 기존 embedding_models에 추가
autorag.embedding_models["math_embedding"] = autorag.LazyInit(
    HuggingFaceEmbedding,
    model_name="sentence-transformers/math-similarity/Bert-MLM_arXiv-MP-class_zbMath",
)


def main(config, qa_data_path, corpus_data_path, project_dir):
    load_dotenv()

    if os.getenv("OPENAI_API_KEY") is None:
        raise ValueError("OPENAI_API_KEY environment variable is not set")
    if not os.path.exists(project_dir):
        os.makedirs(project_dir)
    evaluator = Evaluator(qa_data_path, corpus_data_path, project_dir=project_dir)
    evaluator.start_trial(config)

```

`test.yaml`

```
node_lines:
- node_line_name: retrieve_math_node_line
  nodes:
    - node_type: retrieval
      strategy:
        metrics: [retrieval_f1, retrieval_ndcg, retrieval_map]
      top_k: 3
      modules:
        # - module_type: bm25
        #   bm25_tokenizer: [ ko_kiwi, ko_okt, ko_kkma ]
        - module_type: vectordb
          embedding_model: math_embedding
          # - type: huggingface
          #   model_name: intfloat/multilingual-e5-large-instruct
        # - module_type: vectordb
        #   embedding_model: math_embedding
```


when I check the log, the `openai embedding` was still used.

![Image](https://github.com/user-attachments/assets/332a4d58-2aab-4077-b489-8722eb94bbb5)

when I change the hugging face embedding (bge-m3, etc.), only openai embedding was loaded.
how to change the custom embedding?

and second, Is there any way to utilize already computed embeddings by any chance?


Thank you. 😄 

Hello! @girlsending0 
Apology to the late reply

The blog version is deprecated to the current version of the AutoRAG.
Thus, you can look this [documentation](https://docs.auto-rag.com/local_model.html#configure-the-embedding-model) to configure the embedding model.
Repository: Marker-Inc-Korea/AutoRAG
Issue: [Feature Request] Add gpt-4.5-preview
Root cause: Unimplemented Feature Gap

Recently, OpenAI released the gpt-4.5-preview model. However, the openai_llm.py file in AutoRAG does not yet support this model.

I made the following changes to add support for gpt-4.5-preview:

1. Added the context length for gpt-4.5-preview to MAX_TOKEN_DICT.
![Image](https://github.com/user-attachments/assets/6498f8e4-1764-4b30-8799-9785a855cfa4)

2. Since gpt-4.5-preview does not support tiktoken.encoding_for_model(self.llm), I added a conditional case to handle this.
![Image](https://github.com/user-attachments/assets/b2051438-e7f1-4b9c-a7ba-60db67495ae8)

3.  gpt-4.5-preview also does not support the logprobs parameter. I added a conditional case for this as well, with comments explaining the limitation.
Repository: Marker-Inc-Korea/AutoRAG
Issue: [BUG] TypeError: 'NoneType' object is not callable Exception ignored in: <function BasePromptMaker.__del__ at 0x733030a85bd0>
Root cause: Resource Mishandling

run
`from autorag.deploy import Runner
from dotenv import load_dotenv
load_dotenv()
runner = Runner.from_trial_folder('benchmark1/9'
runner.run('How many teams are there in South Korea\'s 2024 baseball season?')
`
and then 

Exception ignored in: <function LlamaIndexLLM.__del__ at 0x733030a84160>
Traceback (most recent call last):
  File "/data/disk1/anaconda3/envs/env_for_autorag/lib/python3.10/site-packages/autorag/nodes/generator/llama_index_llm.py", line 55, in __del__
  File "/data/disk1/anaconda3/envs/env_for_autorag/lib/python3.10/site-packages/autorag/nodes/generator/base.py", line 23, in __del__
  File "/data/disk1/anaconda3/envs/env_for_autorag/lib/python3.10/logging/__init__.py", line 1477, in info
  File "/data/disk1/anaconda3/envs/env_for_autorag/lib/python3.10/logging/__init__.py", line 1624, in _log
  File "/data/disk1/anaconda3/envs/env_for_autorag/lib/python3.10/logging/__init__.py", line 1634, in handle
  File "/data/disk1/anaconda3/envs/env_for_autorag/lib/python3.10/logging/__init__.py", line 1696, in callHandlers
  File "/data/disk1/anaconda3/envs/env_for_autorag/lib/python3.10/logging/__init__.py", line 968, in handle
  File "/home/ps/.local/lib/python3.10/site-packages/rich/logging.py", line 159, in emit
  File "/home/ps/.local/lib/python3.10/site-packages/rich/logging.py", line 185, in render_message
TypeError: 'NoneType' object is not callable
Exception ignored in: <function BasePromptMaker.__del__ at 0x733030a85bd0>


I don't know why but it looks like the `self.__class__` is not working. 
The line that error occured looks like this.

```
	def __del__(self):
		logger.info(f"Deleting generator module - {self.__class__.__name__}")
```

Just a logging for deletion of the class.

Can you provide some script or colab notebook to reproduce this error. Because I do not get this error when I use Runner like you did.
Repository: Marker-Inc-Korea/AutoRAG
Issue: [BUG]  The temperature hyperparameter is out of range. The valid range for the temperature hyperparameter should be between 0 and 1.
Root cause: Validation Gap

![Image](https://github.com/user-attachments/assets/0f50350c-c728-4be9-9471-8ee8e7a2a996)
I suppose that the temperature parameter 1.5 should be removed 

<img width="783" alt="Image" src="https://github.com/user-attachments/assets/d4af0741-da95-4908-9825-1741ec9b5153" />
https://platform.openai.com/docs/api-reference/chat/create

According to the openai official APi documentation, the range is 0 to 2. 

Plus, in theory, you can increase the temperature up to 2. Just there are no specific reason to do that.

In general we use below 1 temperature, but it always have to be an option who want to generate or take an experiment with temperature more than 1.

Since it is not an bug, I am closing this issue.
Repository: Marker-Inc-Korea/AutoRAG
Issue: Please streamline `pip install` process
Root cause: Architectural Constraint

Hi there,

We are attempting to use AutoRAG in a class setting. It has come to our attention that it takes significant time to install, and many installed packages don't seem relevant. For instance, `AutoRAG[parse]` installs *multiple versions* of langsmith and the transformers libraries: 

<img width="775" alt="Image" src="https://github.com/user-attachments/assets/578352de-30e7-45d9-8b66-b284d180bfc2" />

For context, I am installing via `pip` on Python 3.11. My Macbook is generally pretty powerful, so this isn't a matter of CPU. 

It would be best to double-check all of these dependencies and ensure that you're only requiring users to install those that are absolutely necessary. (It is 20 minutes and I am still stuck on installation.)

@ianarawjo Thanks for the reporting. I will check about the issue
 

Resolve this at #1111
Repository: Marker-Inc-Korea/AutoRAG
Issue: [BUG] docker compose up -d failed
Root cause: Environment Incompatibility

**Describe the bug**
run `docker compose up -d` got:

```
2.945   Preparing metadata (setup.py): started
3.035   Preparing metadata (setup.py): finished with status 'error'
3.037   error: subprocess-exited-with-error
3.037   
3.037   × python setup.py egg_info did not run successfully.
3.037   │ exit code: 1
3.037   ╰─> [6 lines of output]
3.037       Traceback (most recent call last):
3.037         File "<string>", line 2, in <module>
3.037         File "<pip-setuptools-caller>", line 34, in <module>
3.037         File "/tmp/pip-install-ontmg_bf/kiwipiepy_0a72cfa866634e7483c609898f62aa58/setup.py", line 15, in <module>
3.037           import numpy as np
3.037       ModuleNotFoundError: No module named 'numpy'
3.037       [end of output]
3.037   
3.037   note: This error originates from a subprocess, and is likely not a problem with pip.
3.095 error: metadata-generation-failed
3.095 
3.095 × Encountered error while generating package metadata.
3.095 ╰─> See above for output.
3.095 
3.095 note: This is an issue with the package mentioned above, not pip.
3.095 hint: See above for details.
------
```


**Code that bug is happened**

incomplete setup in dockerfile of AutoRAG/api

**Desktop (please complete the following information):**
 - OS: MacOS
 - Python version 3.10




can be fixed by updating dockerfile:

```
FROM python:3.10-slim

WORKDIR /app

# Install system dependencies + parsing dependencies
RUN apt-get update && apt-get install -y \
    python3-pip \
    build-essential \
    libmagic-dev \
    libgl1-mesa-dev \
    libglib2.0-0 \
    poppler-utils \
    tesseract-ocr \
    tesseract-ocr-eng \
    tesseract-ocr-kor \
    cmake \
    && rm -rf /var/lib/apt/lists/*

# Upgrade pip
RUN python -m pip install --upgrade pip

# Install Python dependencies
COPY requirements.txt .
RUN pip install numpy
RUN pip install --no-cache-dir -r requirements.txt
RUN pip install watchfiles pdf2image bert_score
# Install NLTK and download model
RUN pip install nltk && \
    python3 -c "import nltk; nltk.download('punkt_tab')" && \
    python3 -c "import nltk; nltk.download('averaged_perceptron_tagger_eng')"
ENV PYTHONPATH=/app
ENV PYTHONUNBUFFERED=1
# # Copy application code
# COPY . .

# Create directory for celery beat schedule
RUN mkdir -p /app/celerybeat

# Add entrypoint script
COPY entrypoint.sh /entrypoint.sh
RUN chmod +x /entrypoint.sh && \
    sed -i 's/\r$//' /entrypoint.sh  # Remove Windows line endings

ENTRYPOINT ["/entrypoint.sh"]
```


Thanks @MoeBuTa your fix worked for me! 

I think I must streamline and fix the installation process. 
Thanks for reporting the bug.
Repository: Marker-Inc-Korea/AutoRAG
Issue: Cannot use two_hop_incremental
Root cause: Validation Gap

Hello,

I was playing around with the QA dataset generation capabilities and getting an error 
`You have to sample more than two passages for making two-hop questions.`

I am using `random_single_hop`, `n=50` to sample from the corpus. Am I doing something wrong? I don't see double hop sampling functions; the only sampling functions available seem to be single hop.

Full code (adapted from your Evaluation data creation tutorial in the docs):

```python
import pandas as pd
from llama_index.llms.openai import OpenAI
from openai import AsyncOpenAI

from autorag.data.qa.filter.dontknow import dontknow_filter_rule_based
from autorag.data.qa.filter.passage_dependency import passage_dependency_filter_openai
from autorag.data.qa.generation_gt.llama_index_gen_gt import (
    make_basic_gen_gt,
    make_concise_gen_gt,
)
from autorag.data.qa.schema import Raw, Corpus
from autorag.data.qa.query.llama_gen_query import factoid_query_gen, concept_completion_query_gen, two_hop_incremental
from autorag.data.qa.sample import random_single_hop

llm = OpenAI()
raw_df = pd.read_parquet("./all_txt_parsing_output/parsed_result.parquet")
raw_instance = Raw(raw_df)

corpus_df = pd.read_parquet("all_txt_chunking_output/0.parquet")
corpus_instance = Corpus(corpus_df, raw_instance)

initial_qa = (
    corpus_instance.sample(random_single_hop, n=50)
    .map(
        lambda df: df.reset_index(drop=True),
    )
    .make_retrieval_gt_contents()
    .batch_apply(
        two_hop_incremental,
        llm=llm, # query generation
    )
    .batch_apply(
        make_basic_gen_gt,  # answer generation (basic)
        llm=llm,
    )
    .batch_apply(
        make_concise_gen_gt,  # answer generation (concise)
        llm=llm,
    )
    .filter(
        dontknow_filter_rule_based,  # filter don't know
        lang="en",
    )
)

initial_qa.to_parquet('./qa_output/factoid_twohop_basic_concise_dontknow/all_docs_qa.parquet', './all_docs_corpus.parquet')
```
Repository: Marker-Inc-Korea/AutoRAG
Issue: [Feature Request] Ability to create QA dataset that is larger than # of chunks
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
Currently, if you try to sample from a corpus, you cannot specify a larger sample than the number of chunks the corpus contains. You will get a `Cannot take a larger sample than population when 'replace=False'` error. However, there is no such option to specify `replace=False` in the sampling function (`random_single_hop`, for example, takes no such argument).

I can work around this by making smaller chunks so that I have more chunks, but this is not ideal as I may have other reasons for preferring larger chunks.

**Describe the solution you'd like**
A way to sample with replacement from chunks so that you can have more Q/As than chunks.

**Describe alternatives you've considered**
I could also rerun the job multiple times to generate more questions

@boltzmann-brain Hello!
It seems like you want to generate multiple questions in one chunk.
It will be better we have this feature.
Thanks for suggestion!
Repository: Marker-Inc-Korea/AutoRAG
Issue: [BUG] auto-rag.com site can’t be reached.
Root cause: Documentation Deficiency

auto-rag.com site can’t be reached.

안녕하세요.
auto-rag.com 사이트가(도메인이) 잘못된 것 같습니다.
접속이 안되네요. 주소 자체는 맞는 것 같은데 DNS_PROBE_FINISHED_NXDOMAIN 라고 나옵니다.
확인 부탁드리겠습니다.
```
DNS records for auto-rag.com
[Cloudflare](https://www.nslookup.io/domains/auto-rag.com/dns-records/#cloudflare)
[Google DNS](https://www.nslookup.io/domains/auto-rag.com/dns-records/#google)
[Authoritative](https://www.nslookup.io/domains/auto-rag.com/dns-records/#authoritative)
Control D

Local DNS

The Google DNS server responded with these DNS records. Google will serve these records for as long as the time to live (TTL) has not expired. After this period, Google will update its cache by querying one of the authoritative name servers.

A records
No A records found.
AAAA records
No AAAA records found.
CNAME record
No CNAME record found.
TXT records
No TXT records found.
```

![Image](https://github.com/user-attachments/assets/ab712548-17cf-4c86-868d-51f09e4a06d1)

![Image](https://github.com/user-attachments/assets/325cc431-63d6-4b89-9923-bdcedc28d191)

수고하세요.

@thejjw 안녕하세요. 
auto-rag.com 도메인이 만료되어, AutoRAG 문서는 https://marker-inc-korea.github.io/AutoRAG/ 해당 URL로 변경해 두었습니다.
또한 메인페이지는 https://autorag.carrd.co/ 해당 URL에서 보실 수 있습니다.

감사합니다.

빠른 코멘트 감사합니다.
문서에서 해당 주소로 연결된 거라...PR 제출하였으니 참고하여 주시면 감사하겠습니다.
수고하세요.
Repository: Marker-Inc-Korea/AutoRAG
Issue: [HotFix] Can't run AutoRAG when installed.
Root cause: Validation Gap

At __init__.py, we cannot find `VERSION` file because it is not loaded to pypi.

```python
root_path = os.path.dirname(os.path.dirname(os.path.realpath(__file__)))
version_path = os.path.join(root_path, 'VERSION')

with open(version_path, 'r') as f:
    __version__ = f.read().strip()
```

This is the problem of code. We must change the path of VERSION, or find another solution.

Plus, set logger basicConfig to "INFO"
NOTSET is too too verbose....
Repository: Marker-Inc-Korea/AutoRAG
Issue: [BUG] pip install 0.3.16 Dependency issue
Root cause: Environment Incompatibility

**Describe the bug**  
When running `pip install AutoRAG==0.3.16`, the installation fails due to an overly complex dependency graph.  
`pip` aborts the resolution process with the following error message:

```
× Dependency resolution exceeded maximum depth  
╰─> Pip cannot resolve the current dependencies as the dependency graph is too complex for pip to solve efficiently.
```

---

**To Reproduce**  
Steps to reproduce the behavior:
1. Create a clean Python virtual environment (e.g., using pyenv and virtualenv)
2. Run `pip install AutoRAG==0.3.16`
3. Observe the dependency resolution error

---

**Expected behavior**  
AutoRAG and its dependencies should install successfully without requiring manual intervention or workarounds.

---

**Full Error log**
```
× Dependency resolution exceeded maximum depth
╰─> Pip cannot resolve the current dependencies as the dependency graph is too complex for pip to solve efficiently.
```

---

**Code that bug is happened**
```bash
# Environment setup
pyenv virtualenv 3.11.12 autorag-env
pyenv activate autorag-env

# Attempted installation
pip install AutoRAG==0.3.16
```

---

**Desktop (please complete the following information):**
- OS: macOS
- Python version: 3.11.12
---

**Additional context**  
This issue appears to stem from complex or conflicting dependencies involving packages like `llama-index`, `langchain`, and `openai`.  
A temporary workaround is to install AutoRAG with `--no-deps` and then manually install the required dependencies one by one.  
However, the standard installation process should not fail in a clean environment.  
It would be helpful if the project could provide a minimal compatible set of pinned dependencies or a `constraints.txt` file.

@jungchanghae Hello
I installed 0.3.16 version several time but couldn't reproduce the issue.
But, I highly recommend to choose `uv` to install AutoRAG. 
Also, it was okay to use AutoRAG using pip in python 3.10 version, so try to downgrade the python version.
Thank you.

The download worked perfectly after switching to uv. Thank you for the helpful suggestion!
Repository: Marker-Inc-Korea/AutoRAG
Issue: [New Node, Separate] New "HybridRetrieval" Node and make RRF and CC will be the module for it
Root cause: Architectural Constraint

We need to build HybridRetrieval node separate from Retrieval node.
Because the run.py and other structure in retrieval module is way too complicated, so it is nearly impossible to add the extra retrieval method or extra hybrid retrieval method. 

So, I decide to add "HybridRetrieval" node separately, and pass both semantic and lexical results to the next node. All I need to do is just select each best results in semantic and lexical retrieval, and add to the result dataframe. The new "HybridRetrieval" node will receive all informations.

The result yaml file look like this.

```yaml
    - node_type: lexical_retrieval  # represents run_node function
      strategy:  # essential for every node
        metrics: [retrieval_f1, retrieval_recall]
      top_k: 10 # node param, which adapt to every module in this node.
      modules:
        - module_type: ${BM25} # for testing env variable
          bm25_tokenizer: [ facebook/opt-125m, porter_stemmer ]
    - node_type: semantic_retrieval
      strategy: # essential for every node
        metrics: [ retrieval_f1, retrieval_recall ]
      top_k: 10
      modules:
        - module_type: vectordb
          vectordb: [ openai_embed_3_large, openai_embed_3_small ]
    - node_type: hybrid_retrieval
      strategy: # essential for every node
        metrics: [ retrieval_f1, retrieval_recall ]
      top_k: 10
      modules:
        - module_type: hybrid_rrf
          weight_range: (4, 30)
```

Looks more complicated, but the code becomes way more intuitive and can be managed. 
Also way more convenient to add new retrieval modules (SPLADE, BM25-F, DAT, etc...)