zhiminy/EvalEng huggingface dataset:
Repository: Marker-Inc-Korea/AutoRAG
Issue: [Feature Request] Refactor API server for processing whole YAML file.
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
We need this features.

- Post query and get generated texts
- The text can be streamed
- Return used passages and its raw document path and passages

**Describe the solution you'd like**
I want to make an OpenAPI sepcifiction for it

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

**Additional context**
Add any other context or screenshots about the feature request here.
Repository: Marker-Inc-Korea/AutoRAG
Issue: [Feature Request] docker image push CI/CD
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
The current process of manually building and pushing Docker images is time-consuming and prone to errors. It's particularly challenging to maintain consistency when managing multiple environments or versions.

**Describe the solution you'd like**
We need an automated CI/CD pipeline using GitHub Actions to automatically build Docker images and push them to Docker Hub. This pipeline should trigger whenever code changes are merged into the main branch.
Describe alternatives you've considered

**Describe alternatives you've considered**
Setting up a CI/CD pipeline using Jenkins
Using GitLab CI/CD (which would require migrating from GitHub to GitLab)
Creating semi-automated deployment scripts

**Additional context**
https://hub.docker.com/repository/docker/autoraghq/autorag/general

docker pull autoraghq/autorag:0.3.3-all
docker pull autoraghq/autorag:0.3.3-ko
docker pull autoraghq/autorag:0.3.3-parse
docker pull autoraghq/autorag:0.3.3-dev

'all' includes all features, 'ko' adds Korean support, 'parse' is a lightweight version with parsing only, and 'dev' is the latest development version with potential instabilities.



This is awesome!

Could we add something that pushes the `autorag-frontend` image too? Also, it would be nice if the `docker-compose.yml` used the pushed images by default so we don't run into issues like #1102 from being on the bleeding edge. 

Maybe we can add a separate `docker-compose.dev.yml` that builds from local Dockerfiles? Or name the other `docker-compose.prod.yml` that pins to stable Docker Hub images and direct users in the README to use that one.

Also it would be nice to add a README note to run `git submodule update --init --recursive` to get the autorag-frontend repo
Repository: Marker-Inc-Korea/AutoRAG
Issue: [BUG] OpenAIEmbedding did not truncate as openai_truncate_by_token
Root cause: Validation Gap

**Describe the bug**
isinstance(embedding_model, OpenAIEmbedding) => this will be False

You can use this

isinstance(embedding_model._instance, OpenAIEmbedding)

**Code that bug is happened**
If applicable, add the code that bug is happened.
(Especially, your AutoRAG YAML file or python codes that you wrote)

**Desktop (please complete the following information):**
 - OS: [e.g. Windows, Linux, MacOS]
 - Python version [e.g. 3.10]

**Additional context**
Add any other context about the problem here.
Repository: Marker-Inc-Korea/AutoRAG
Issue: [BUG] Chroma distance score bug when do hybrid retrieval.
Root cause: Algorithmic Error

**Describe the bug**
```
id_results: GetResult = collection.get(ids, include=["embeddings"])
	temp_collection = temp_client.create_collection(
		name="temp", metadata={"hnsw:space": "cosine"}
	)
	temp_collection.add(ids=id_results["ids"], embeddings=id_results["embeddings"])

	query_result: QueryResult = temp_collection.query(
		query_embeddings=query_embeddings, n_results=len(ids)
	)
	assert len(query_result["ids"]) == len(query_result["distances"])
	id_scores_dict = {id_: [] for id_ in ids}
	for id_list, score_list in zip(query_result["ids"], query_result["distances"]):
		for id_ in list(id_scores_dict.keys()):
			id_idx = id_list.index(id_)
			id_scores_dict[id_].append(score_list[id_idx])
	id_scores_pd = pd.DataFrame(id_scores_dict)
	temp_client.delete_collection("temp")
	return id_scores_pd.max(axis=0).tolist()
```

Need to fix here because our distance score.
Repository: Marker-Inc-Korea/AutoRAG
Issue: [Feature Request] Add OpenAI compatible Server
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
As new runner, add OpenAI compatible server.

**Describe the solution you'd like**
It will be useful to use it if we have OpenAI compatible server! 


I cannot a API spec that can send retrieved passages as well as LLM generated answers.
I don’t know how it will be effective until there is not standard API that retrieves passages.
Repository: Marker-Inc-Korea/AutoRAG
Issue: [Feature Request] AutoRAG only-API-call version
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
Try to make a lightweight Docker container that only contains API-connected modules. (Without local reranker models, longllmlingua, etc.)

**Describe the solution you'd like**
First, use other requirements file at pyproject.toml. 
And build requirements.txt separately.
Maybe we will have torch issue or others. (import issue at __init__.py especially)
Repository: Marker-Inc-Korea/AutoRAG
Issue: [Feature Request] Always run validation at the start_trial
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
Run a whole trial can be time consuming and cost-intensive.
Small error can lead big irritation. 
So, why don’t we run validation at every start_trial??
It will add a minute, but it will help lots of devs.

**Many people will not read docs super carefully before execution**

**Describe the solution you'd like**
Use `Validator` at the start_trial.
Get the option skipping validation, but set it default.
Repository: Marker-Inc-Korea/AutoRAG
Issue: [Bug] Restrict 'threshold' config to numbers only for module_type
Root cause: Validation Gap

> Restrict 'threshold' config to numbers only for module_type 

 _Originally posted by @hongsw in [#821](https://github.com/Marker-Inc-Korea/AutoRAG/issues/821#issuecomment-2403716359)_


## Is your feature request related to a problem? Please describe.
Currently, the `threshold` configuration for `module_type` accepts any type of value, which can lead to validation errors and potential runtime issues. Specifically, we've observed instances where date strings (e.g., "2015-01-01T00:00:00.000Z") are being used instead of numerical values. This inconsistency can cause confusion for users and may result in unexpected behavior in our application.

For example, we've seen the following error:

```
TYPE must be number

  216 |             {
  217 |               "module_type": "recency_filter",
> 218 |               "threshold": "2015-01-01T00:00:00.000Z"
      |                            ^^^^^^^^^^^^^^^^^^^^^^^^^^ 👈🏽  type must be number
```

This indicates that a string is being used where a number is expected.

## Describe the solution you'd like
We propose to modify the JSON schema to explicitly restrict the `threshold` configuration for `module_type` to accept only numerical values. This change should:

1. Update the JSON schema to specify that `threshold` must be a number.
2. Provide clear error messages when non-numerical values are used for `threshold`.
3. Update any relevant documentation to clearly state that `threshold` should be a number.

## Describe alternatives you've considered
1. Allow both numbers and date strings, and convert date strings to numerical timestamps internally:
   - Pros: More flexible for users who prefer working with dates
   - Cons: Increases complexity and may lead to inconsistent usage

2. Use a separate configuration for date-based thresholds:
   - Pros: Clear separation between numerical and date-based thresholds
   - Cons: Requires more configuration options, potentially complicating the schema

## Additional context
This change will improve the consistency of our configuration schema and reduce errors caused by incorrect data types. It's part of our ongoing effort to make the configuration process more robust and user-friendly.

Example of the desired schema change:

```json
{
  "properties": {
    "module_type": {
      "type": "object",
      "properties": {
        "threshold": {
          "type": "number"
        }
      }
    }
  }
}
```

This change will ensure that only numerical values are accepted for the `threshold` configuration.

@hongsw 
But since it is "recency filter", that filtered old passages, the `threshold` parameter must get the exact datetime.
Repository: Marker-Inc-Korea/AutoRAG
Issue: I have idea about dynamic import function in __init__.py

              I have idea about dynamic import function in __init__.py

```py
embedding_models = {
    "openai": LazyInit("llama_index.embeddings.openai.OpenAIEmbedding"),
    "huggingface_bge_small": LazyInit("llama_index.embeddings.huggingface.HuggingFaceEmbedding", model_name="BAAI/bge-small-en-v1.5"),
    "mock": LazyInit("llama_index.embeddings.mock.MockEmbeddingRandom", embed_dim=768),
}
```

_Originally posted by @hongsw in https://github.com/Marker-Inc-Korea/AutoRAG/issues/823#issuecomment-2404324627_
            

You can use `dynamically_find_function` at `autorag.support` for this. 
Implement dynamic import at `LazyInit` itself.
Repository: Marker-Inc-Korea/AutoRAG
Issue: Implement default web UI launch for AutoRAG CLI
Root cause: Unimplemented Feature Gap

              Implement default web UI launch for AutoRAG CLI
- Running `$ autorag` in the terminal should:
- API routes are prefixed with `/api/`
- All other routes serve the React frontend

_Originally posted by @hongsw in https://github.com/Marker-Inc-Korea/AutoRAG/pull/827#pullrequestreview-2362493765_
            

Already launch `autorag run_web —trial_path` ?
Repository: Marker-Inc-Korea/AutoRAG
Issue: [Feature Request] Connect S3 as file loader
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
I want to connect Amazon S3 as the file loader to parse to the VectorDB.

**Describe the solution you'd like**
Use Langchain or LlamaIndex (or something better) one to connect many document source to parsing. 

**Describe alternatives you've considered**
We can use other library, like liteLLM for getting documents. 


As alternative, we can build example jupyter notebook.

To support AWS well, I think it is better to use fsspec. Unified interface for loading files! 
We are now only support pdf, so loading pdf files from all kinds of file system.

Below is the full fsspec supported protocol.

It contains dropbox, google drive, S3, even jupyter & github!

```
['abfs',
 'adl',
 'arrow_hdfs',
 'asynclocal',
 'az',
 'blockcache',
 'box',
 'cached',
 'dask',
 'data',
 'dbfs',
 'dir',
 'dropbox',
 'dvc',
 'file',
 'filecache',
 'ftp',
 'gcs',
 'gdrive',
 'generic',
 'git',
 'github',
 'gs',
 'hdfs',
 'hf',
 'http',
 'https',
 'jlab',
 'jupyter',
 'lakefs',
 'libarchive',
 'local',
 'memory',
 'oci',
 'ocilake',
 'oss',
 'reference',
 'root',
 's3',
 's3a',
 'sftp',
 'simplecache',
 'smb',
 'ssh',
 'tar',
 'wandb',
 'webdav',
 'webhdfs',
 'zip']
```
Repository: Marker-Inc-Korea/AutoRAG
Issue: [Feature Request] Connect different Vector Stores as vectordb.
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
Connect vector databases via llamaindex. 

**Describe the solution you'd like**
Maybe LlamaIndex is okay? We need integrated interface for this...

**Describe alternatives you've considered**
I think we need to manage #824 first for managing complex dependencies.

**Additional context**
Add any other context or screenshots about the feature request here.


But how connect existing vector DB to AutoRAG?
It doesn't have corpus.parquet and how to make it???? corpus.parquet is crucial to use it.
Consider corpus.parquet as one of the Vector Database is viable? Hmmmm
Repository: Marker-Inc-Korea/AutoRAG
Issue: [BUG] `vllm._C` ModuleNotFoundError
Root cause: Environment Incompatibility

**Describe the bug**

**To Reproduce**
use our `simple_local.yaml` i got this error.
(And I used sample qa & corpus file in our resources folder)

evaluator = Evaluator(qa_data_path=qa_parquet, corpus_data_path=corpus_parquet)
evaluator.start_trial(example_yaml)

**Expected behavior**
A clear and concise description of what you expected to happen.

**Full Error log**
```
INFO     [node.py:55] >> Running node             node.py:55
                             generator...                                       
                    INFO     [base.py:19] >> Initialize generator     base.py:19
                             node - Vllm                                        
WARNING 10-11 22:51:03 _custom_ops.py:18] Failed to import from vllm._C with ModuleNotFoundError("No module named 'vllm._C'")
INFO 10-11 22:51:03 importing.py:10] Triton not installed; certain GPU-related functions will not be available.
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
	- Avoid using `tokenizers` before the fork if possible
	- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
	- Avoid using `tokenizers` before the fork if possible
	- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
Traceback (most recent call last):


...


    raise RuntimeError("Failed to infer device type")
RuntimeError: Failed to infer device type
```
Repository: Marker-Inc-Korea/AutoRAG
Issue: [BUG] Import error when don’t install kiwipiepy
Root cause: Validation Gap

**Describe the bug**

At autorag/data/__init__.py

```python
try:
	from kiwipiepy import Kiwi

	def split_by_sentence_kiwi() -> Callable[[str], List[str]]:
		kiwi = Kiwi()

		def split(text: str) -> List[str]:
			kiwi_result = kiwi.split_into_sents(text)
			sentences = list(map(lambda x: x.text, kiwi_result))

			return sentences

		return split

	sentence_splitter_modules = {"kiwi": LazyInit(split_by_sentence_kiwi)}

except ImportError:
	logger.info(
		"You did not install korean version of AutoRAG."
		"To use korean version, run pip install 'AutoRAG[ko]'"
	)
```

In here, if we do not install ko version of AutoRAG, we have import error at here.

data/chunk/base.py

```python
from autorag.data import chunk_modules, sentence_splitter_modules
```
Repository: Marker-Inc-Korea/AutoRAG
Issue: [Question] How can I retrieve image or files from AutoRAG? / 이미지나 파일 등을 어떻게 RAG에 저장할 수 있을까요?
Root cause: Documentation Deficiency

Hi, I'm building a kind of modular RAG system. But my data includes some images, tables ( I used html to save tables), files.
To embed images and files, they are translated into texts by OCR. And original sources are saved as url.

I found AutoRAG is the system I need, but I can't figure out how I should use those html texts and file links.

Can you suggest any hint or idea how I can utilize image and files in modular RAG?


Another Question, I didn't understand the mechanism of PASS module yet. How do you judge which answer is the best answer?

Thanks for reading.

@TheClevers 
안녕하세요! 먼저 AutoRAG를 사용해주셔서 감사합니다.
차트나 그래프 등과 같은 이미지를 포함한 multi-modal RAG를 구성하고 싶으신 것인가요? 
일단, 이미지 같은 경우에는 아직 AutoRAG에서 지원하지는 않지만 @bwook00 님께서 빠른 시일 안에 추가 feature로 추가할 예정입니다. 
html text와 같은 경우에는 LlamaIndex의 [가이드](https://docs.llamaindex.ai/en/stable/examples/data_connectors/WebPageDemo/)를 참고하셔서 HTML을 잘 파싱한 이후에, `raw.parquet`의 형태로 만들어 data creation 단계부터 사용하시면 됩니다. [관련 문서](https://docs.auto-rag.com/data_creation/tutorial.html)
HTML로 되어있는 표 같은 경우에는, 표만 따로 chunk로 만드는 형식도 괜찮을 것 같습니다. 

두 번째로, "pass module"은 아무것도 안하는 모듈이라고 보면 됩니다.
예를 들어서, passage reranker를 시험할 때에 아예 리랭커를 "사용하지 않는" 옵션에 대해서도 시험해 보고 싶을 수 있습니다. 그럴 때 pass reranker를 활용하면, 리랭커를 사용하지 않는 경우를 가정해 실험해 볼 수 있습니다.

아래는 영어 번역입니다.

---

Hello! First of all, thank you for using AutoRAG.

Are you trying to set up a multi-modal RAG that includes images such as charts and graphs? For now, AutoRAG does not support image data yet, but @bwook00 plans to add this feature soon. 

As for HTML text, you can follow LlamaIndex's [guide](https://docs.llamaindex.ai/en/stable/examples/data_connectors/WebPageDemo/) to parse HTML properly and then convert it into a `raw.parquet` format to use from the data creation stage. You can refer to the [related documentation](https://docs.auto-rag.com/data_creation/tutorial.html). For tables in HTML, it could also work to chunk out just the tables separately.

Secondly, the "pass module" can be considered as a module that does nothing. For example, when testing a passage reranker, you might want to test the option of "not using" a reranker at all. In such cases, the pass reranker allows you to experiment assuming the reranker is not used.

@TheClevers 
안녕하세요, 이미지를 파싱하는 기능이 AutoRAG 0.3.7 버전에 추가되었습니다 :)

자세하게는, AutoRAG의 Llama Parse 모듈에서 Multi Modal Parsing 기능을 지원하게 되었습니다.
Multi Modal을 이용한 Parsing이 가능해지면서 문서 속에 있는 이미지를 함께 파싱할 수 있게 되었습니다.
자세한 정보는 [AutoRAG Llama Parse Documentation](https://docs.auto-rag.com/data_creation/parse/llama_parse.html)을 참고해주세요!

아래는 영어 번역입니다.

---

Hello, the ability to parse images has been added to AutoRAG version 0.3.7 :)

Specifically, we've added support for Multi Modal Parsing in the Llama Parse module of AutoRAG.
With Multi Modal parsing, you can now parse images in a document together.
For more information, please see the [AutoRAG Llama Parse Documentation](https://docs.auto-rag.com/data_creation/parse/llama_parse.html)!


알려주셔서 감사합니다!
Repository: Marker-Inc-Korea/AutoRAG
Issue: [Feature Request] Config YAML schema validator
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Yes, users can edit YAML freely, but they don't know if it's valid before running the program.

**Describe the solution you'd like**
A clear and concise description of what you want to happen.

Using JSON Schema, create our YAML and JSON specific config files.
YAML files should be convertible to JSON.

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

Ajv (Ajv JSON schema validator)


**Additional context**
Add any other context or screenshots about the feature request here.


I think I will be implement this on the library itself.
Repository: Marker-Inc-Korea/AutoRAG
Issue: [Question] override default prompt with custom?
Root cause: Documentation Deficiency

In Data Creation process, is it possible to override default prompts for generating queries (and ground truth answers) with my own customs?

This is just my personal opnion, but generated queries makes me feel a little too descriptive (sometimes 2 topics in 1 question) and unnatural as humans' query (of course it's not by human, but human queries are generally much more short). It would be nice to have options to override.

Thanks!

@kun432 
Hello! 

First thing, I will tell you how to use your custom prompt.
In the each data creation function, the inputs are basically a "function". So all you need to do is to make a function for LLM prompting.

Fox example, you can make a custom function like this. The only essential parameter is row, which is dictionary that contains a one column of the target dataframe. Plus be sure make this as async function.

```python
async def gen_query_custom(row: dict, llm: BaseLLM,) -> dict:
	# The input row looks like this:
	# {
	# 	"qid": "43332-455",
	# 	"retrieval_gt": [["doc_id_1"]],
	# 	"retrieval_gt_contents": [["Sample Passage Content"]],
	# }
	
	context = list(itertools.chain.from_iterable(row["retrieval_gt_contents"]))
	context_str = "Text:\n" + "\n".join(
		[f"{i + 1}. {c}" for i, c in enumerate(context)]
	)
	prompt = f"""Please generate question based on the query. Really natural! BE NATURAL!
	Given Context:
	{context_str}
	
	Generated Question:
	"""
	chat_response: ChatResponse = await llm.achat(messages=[
		ChatMessage(role=MessageRole.USER, content=prompt),
	])
	row["query"] = chat_response.message.content
	return row
```

You can use this function like below.
```python
llm = OpenAI(model="gpt-4o")
qa = QA(qa_df)
qa.make_retrieval_gt_contents().batch_apply(gen_query_custom, llm=llm)
```

---

Second, It can be unnatural because human is not too specific at some question.
Why don't you use `compress_ragas` evolving method. It can be good for simplify the too descriptive questions.

@vkehfdl1 

Thanks! just starting out AutoRAG so I hadn't caught up on evolving method. That's good to know!
Will try both.
Repository: Marker-Inc-Korea/AutoRAG
Issue: [Feature Request] Available Modules List Show content directly on the Docs page
GitHub labels: enhancement

**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

**Describe the solution you'd like**
A clear and concise description of what you want to happen.

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

**Additional context**
Add any other context or screenshots about the feature request here.
Repository: Marker-Inc-Korea/AutoRAG
Issue: [BUG] Error at docker push
Root cause: Environment Incompatibility

**Describe the bug**
Docker CD failed
Because.... hahahaha

ERROR: Could not install packages due to an OSError: [Errno 28] No space left on device

**To Reproduce**
Push to main with VERSION changed.

**Desktop (please complete the following information):**
On Github Action


**Additional context**
Do we have to pay for github action?
Or make it smaller... Ah I have a headache for this damn

https://github.com/Marker-Inc-Korea/AutoRAG/actions/runs/11387904505/job/31683629917#step:7:1612

I recommend using the dynamic pip module installation approach https://github.com/Marker-Inc-Korea/AutoRAG/issues/824 to reduce docker image size.

https://flavono123.oopy.io/posts/resolve-no-space-left-on-device-in-github-action-ubuntu-runner
Repository: Marker-Inc-Korea/AutoRAG
Issue: [BUG] Prev Next Augmeneter is impossible to validate
Root cause: Validation Gap

**Describe the bug**
When you use prev-next augmenter, it is impossible to pass validation stage.

**To Reproduce**
Steps to reproduce the behavior:
1. Make YAML file with prev-next augmenter
2. Run validation (the qa must be larger than 5)

**Expected behavior**
This is happening because we cut some part of corpus.parquet. Because for validation we don’t need full corpus. However, at prev_next_augmenter, we have to retrieve contents from the prev id or next id at the corpus. 
Then, at the corpus, the corresponding id is not existed. So the validation failed. 

**Code that bug is happened**
At the prev next augmenter

**Desktop (please complete the following information):**
AutoRAG 0.3.6
python 3.10

**Additional context**
Add any other context about the problem here.
Repository: Marker-Inc-Korea/AutoRAG
Issue: Replace Chroma DB to Chroma VectorStore for compatibility
Root cause: External Dependency Breakage

It is hard to believe we are building this. But okay I admit.
The January Jeffrey was wrong.
We need other DB, not chroma Persistent Client.

The first step is guarantee that everything works fine with Chroma VectorStore class that LlamaIndex made.
We don’t want to make all implementations and our own abstraction to VectorStore for immediate compatibility.

LlamaIndex will not working becuase they don’t have `fetch` function yet...

close at #871