Repository: Marker-Inc-Korea/AutoRAG
Issue: Are the sample parquets in huggingface working?
Root cause: Validation Gap

When I run it with triviaqa data, I get a chroma batch size error, so I split it up and send it to fetch_contents and get a size of 0. What's the problem? 

my test code:
evaluator = Evaluator(qa_data_path='./sample_dataset/dataset/triviaqa/qa_test.parquet', corpus_data_path='./sample_dataset/dataset/triviaqa/corpus.parquet')
evaluator.start_trial('sample_config/compact_local.yaml')

index 0 is out of bounds for axis 0 with size 0  

Hello @hofe7 
Thank you for reporting your error.

Can you tell me where you get the error? The full error track history will be great, which line you get the error. 
It can help us fix this error.

Since then, I'll try to reproduce this error, too. Thanks.

@hofe7  In my case, it works fine with triviaQA sample parquet file on embedding whole corpus file.

What did you use for running AutoRAG? 
You use 'compact_local.yaml' which use local embedding 'baai-small'. It can be occur OOM error on your CUDA machine possibly. 
If your local machine is bottleneck, suggest to use 'compact_openai.yaml' or find something much lighter local embedding model.

@vkehfdl1 The same error occurs when using compact_openai.yaml. 
On line 39 of utils/util.py file, index 0 is out of bounds for axis 0 with size 0 

Can you give me full error record?  @hofe7 
It is hard to what is the problem with limited information. 
Because `line 39 of utils/util.py file` used by a lot of features in AutoRAG...

@hofe7 Are you using Windows? 

@vkehfdl1 I am using an m2 macbook pro 32gb.
I used openai as previously guided and my test code is as follows.

evaluator = Evaluator(qa_data_path='./sample_dataset/dataset/2024/qa.parquet', corpus_data_path='./sample_dataset/dataset/2024/corpus.parquet')
evaluator.start_trial('sample_config/compact_openai.yaml')


I get an error right after the embedding finishes. I'll attach the full error.

File "/Users/hofe7/Documents/dev/llm/AutoRAG/test.py", line 7, in <module>
evaluator.start_trial('sample_config/compact_openai.yaml')
File "/Users/hofe7/Documents/dev/llm/AutoRAG/autorag/evaluator.py", line 83, in start_trial
previous_result = run_node_line(node_line, node_line_dir, previous_result)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/hofe7/Documents/dev/llm/AutoRAG/autorag/node_line.py", line 45, in run_node_line
previous_result = node.run(previous_result, node_line_dir)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/hofe7/Documents/dev/llm/AutoRAG/autorag/schema/node.py", line 57, in run
return self.run_node(modules=input_modules,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/hofe7/Documents/dev/llm/AutoRAG/autorag/nodes/retrieval/run.py", line 75, in run_retrieval_node
non_hybrid_results, non_hybrid_times, non_hybrid_summary_df = run_and_save(non_hybrid_modules,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/hofe7/Documents/dev/llm/AutoRAG/autorag/nodes/retrieval/run.py", line 42, in run_and_save
result, execution_times = zip(*map(lambda task: measure_speed(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/hofe7/Documents/dev/llm/AutoRAG/autorag/nodes/retrieval/run.py", line 42, in <lambda>
  result, execution_times = zip(*map(lambda task: measure_speed(
  ^^^^^^^^^^^^^^
  File "/Users/hofe7/Documents/dev/llm/AutoRAG/autorag/strategy.py", line 13, in measure_speed
  result = func(*args, **kwargs)
  ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/hofe7/Documents/dev/llm/AutoRAG/autorag/utils/util.py", line 39, in wrapper
  results = func(*args, **kwargs)
  ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/hofe7/Documents/dev/llm/AutoRAG/autorag/nodes/retrieval/base.py", line 94, in wrapper
  contents = fetch_contents(corpus_data, ids)
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/hofe7/Documents/dev/llm/AutoRAG/autorag/utils/util.py", line 20, in fetch_contents
  contents = list(map(lambda x: corpus_data.loc[lambda row: row['doc_id'] == x]['contents'].values[0], flat_ids))
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/hofe7/Documents/dev/llm/AutoRAG/autorag/utils/util.py", line 20, in <lambda>
  contents = list(map(lambda x: corpus_data.loc[lambda row: row['doc_id'] == x]['contents'].values[0], flat_ids))
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^
  IndexError: index 0 is out of bounds for axis 0 with size 0

@hofe7 Yes, it looks the `fetch_contents` file can't fetch anything from `Chroma` database. I don't know why,
but I guess you might run your trial at independent directory. 

0. Suggest re-install ChromaDB.

1. Make new directory. 
```bash
mkdir new_trivia_project
```

2. Specify project directory on your code.
```python
evaluator = Evaluator(qa_data_path='./sample_dataset/dataset/2024/qa.parquet', corpus_data_path='./sample_dataset/dataset/2024/corpus.parquet',
project_dir='./new_trivia_project')
evaluator.start_trial('sample_config/compact_openai.yaml')
```

It will perform embedding again, and I hope it works fine. 

+ At #251 , there is an error at `triviaqa` data right now. I will fix bug asap, but until then, recommend to use [eli5](https://huggingface.co/datasets/MarkrAI/eli5_sample_autorag) dataset alternatively.


Now #251 is fixed. You can use triviaQA dataset if you pull AutoRAG from main branch.

@hofe7 Since triviaQA is working fine. I am closing this issue. Thanks.