Repository: Marker-Inc-Korea/AutoRAG
Issue: [BUG]
Root cause: Validation Gap

**Describe the bug**
An error occurs during validation when trying to sample data from qa_data. The application throws a ValueError: Cannot take a larger sample than population when 'replace=False', indicating an attempt to sample more records than are available.

**To Reproduce**

Steps to reproduce the behavior:

Run the command evaluator.start_trial('/content/config.yaml').
Observe the sampling attempt in validator.validate(yaml_path) within the evaluator.py file.
Error is thrown due to the sampling request exceeding available data.

**Expected behavior**
The validator should handle the sampling request correctly, sampling either the specified number of records or the available number if it’s smaller.

**Full Error log**
```
evaluator.start_trial('/content/config.yaml')
[10/31/24 03:48:58] INFO     [evaluator.py:127] >>                                        evaluator.py:127
                                             _        _____            _____                              
                                  /\        | |      |  __ \     /\   / ____|                             
                                 /  \  _   _| |_ ___ | |__) |   /  \ | |  __                              
                                / /\ \| | | | __/ _ \|  _  /   / /\ \| | |_ |                             
                               / ____ \ |_| | || (_) | | \ \  / ____ \ |__| |                             
                              /_/    \_\__,_|\__\___/|_|  \_\/_/    \_\_____|                             
                                                                                                          
                                                                                                          
                    INFO     [evaluator.py:128] >> Start Validation input data and config evaluator.py:128
                             YAML file first. If you want to skip this, put the                           
                             --skip_validation flag or `skip_validation` at the                           
                             start_trial function.                                                        
[10/31/24 03:48:59] ERROR    [__init__.py:60] >> Unexpected exception                       __init__.py:60
                             ╭──────────── Traceback (most recent call last) ─────────────╮               
                             │ in <module>:1                                              │               
                             │                                                            │               
                             │ /Users/jayanthkumar/Documents/development/autorag/venv/lib │               
                             │ /python3.10/site-packages/autorag/evaluator.py:138 in      │               
                             │ start_trial                                                │               
                             │                                                            │               
                             │   135 │   │   │   validator = Validator(                   │               
                             │   136 │   │   │   │   qa_data_path=self.qa_data_path, corp │               
                             │   137 │   │   │   )                                        │               
                             │ ❱ 138 │   │   │   validator.validate(yaml_path)            │               
                             │   139 │   │                                                │               
                             │   140 │   │   os.environ["PROJECT_DIR"] = self.project_dir │               
                             │   141                                                      │               
                             │                                                            │               
                             │ /Users/jayanthkumar/Documents/development/autorag/venv/lib │               
                             │ /python3.10/site-packages/autorag/validator.py:46 in       │               
                             │ validate                                                   │               
                             │                                                            │               
                             │   43 │                                                     │               
                             │   44 │   def validate(self, yaml_path: str, qa_cnt: int =  │               
                             │   45 │   │   # sample QA data                              │               
                             │ ❱ 46 │   │   sample_qa_df = self.qa_data.sample(qa_cnt, ra │               
                             │   47 │   │   sample_qa_df.reset_index(drop=True, inplace=T │               
                             │   48 │   │                                                 │               
                             │   49 │   │   # get doc_id                                  │               
                             │                                                            │               
                             │ /Users/jayanthkumar/Documents/development/autorag/venv/lib │               
                             │ /python3.10/site-packages/pandas/core/generic.py:6118 in   │               
                             │ sample                                                     │               
                             │                                                            │               
                             │    6115 │   │   if weights is not None:                    │               
                             │    6116 │   │   │   weights = sample.preprocess_weights(se │               
                             │    6117 │   │                                              │               
                             │ ❱  6118 │   │   sampled_indices = sample.sample(obj_len, s │               
                             │    6119 │   │   result = self.take(sampled_indices, axis=a │               
                             │    6120 │   │                                              │               
                             │    6121 │   │   if ignore_index:                           │               
                             │                                                            │               
                             │ /Users/jayanthkumar/Documents/development/autorag/venv/lib │               
                             │ /python3.10/site-packages/pandas/core/sample.py:152 in     │               
                             │ sample                                                     │               
                             │                                                            │               
                             │   149 │   │   else:                                        │               
                             │   150 │   │   │   raise ValueError("Invalid weights: weigh │               
                             │   151 │                                                    │               
                             │ ❱ 152 │   return random_state.choice(obj_len, size=size, r │               
                             │   153 │   │   np.intp, copy=False                          │               
                             │   154 │   )                                                │               
                             │   155                                                      │               
                             │                                                            │               
                             │ in numpy.random.mtrand.RandomState.choice:1001             │               
                             ╰────────────────────────────────────────────────────────────╯               
                             ValueError: Cannot take a larger sample than population when                 
                             'replace=False'                                                              
>>> exit()
(venv) (base) ➜  autorag autorag evaluate --config config.yaml --qa_data_path ./content/initial_qa.parquet --corpus_data_path ./content/initial_corpus.parquet


None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.
[10/31/24 03:53:45] INFO     [__init__.py:100] >> You are using API version of AutoRAG.To  __init__.py:100
                             use local version, run pip install 'AutoRAG[gpu]'                            
                    INFO     [__init__.py:125] >> You are using API version of AutoRAG.To  __init__.py:125
                             use local version, run pip install 'AutoRAG[gpu]'                            
[10/31/24 03:53:53] INFO     [_client.py:1038] >> HTTP Request: GET                        _client.py:1038
                             https://api.gradio.app/gradio-messaging/en "HTTP/1.1 200 OK"                 
                    INFO     [evaluator.py:127] >>                                        evaluator.py:127
                                             _        _____            _____                              
                                  /\        | |      |  __ \     /\   / ____|                             
                                 /  \  _   _| |_ ___ | |__) |   /  \ | |  __                              
                                / /\ \| | | | __/ _ \|  _  /   / /\ \| | |_ |                             
                               / ____ \ |_| | || (_) | | \ \  / ____ \ |__| |                             
                              /_/    \_\__,_|\__\___/|_|  \_\/_/    \_\_____|                             
                                                                                                          
                                                                                                          
                    INFO     [evaluator.py:128] >> Start Validation input data and config evaluator.py:128
                             YAML file first. If you want to skip this, put the                           
                             --skip_validation flag or `skip_validation` at the                           
                             start_trial function.                                                        
[10/31/24 03:53:54] ERROR    [__init__.py:60] >> Unexpected exception                       __init__.py:60
                             ╭──────────── Traceback (most recent call last) ─────────────╮               
                             │ /Users/jayanthkumar/Documents/development/autorag/venv/bin │               
                             │ /autorag:8 in <module>                                     │               
                             │                                                            │               
                             │   5 from autorag.cli import cli                            │               
                             │   6 if __name__ == '__main__':                             │               
                             │   7 │   sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '' │               
                             │ ❱ 8 │   sys.exit(cli())                                    │               
                             │   9                                                        │               
                             │                                                            │               
                             │ /Users/jayanthkumar/Documents/development/autorag/venv/lib │               
                             │ /python3.10/site-packages/click/core.py:1157 in __call__   │               
                             │                                                            │               
                             │   1154 │                                                   │               
                             │   1155 │   def __call__(self, *args: t.Any, **kwargs: t.An │               
                             │   1156 │   │   """Alias for :meth:`main`."""               │               
                             │ ❱ 1157 │   │   return self.main(*args, **kwargs)           │               
                             │   1158                                                     │               
                             │   1159                                                     │               
                             │   1160 class Command(BaseCommand):                         │               
                             │                                                            │               
                             │ /Users/jayanthkumar/Documents/development/autorag/venv/lib │               
                             │ /python3.10/site-packages/click/core.py:1078 in main       │               
                             │                                                            │               
                             │   1075 │   │   try:                                        │               
                             │   1076 │   │   │   try:                                    │               
                             │   1077 │   │   │   │   with self.make_context(prog_name, a │               
                             │ ❱ 1078 │   │   │   │   │   rv = self.invoke(ctx)           │               
                             │   1079 │   │   │   │   │   if not standalone_mode:         │               
                             │   1080 │   │   │   │   │   │   return rv                   │               
                             │   1081 │   │   │   │   │   # it's not safe to `ctx.exit(rv │               
                             │                                                            │               
                             │ /Users/jayanthkumar/Documents/development/autorag/venv/lib │               
                             │ /python3.10/site-packages/click/core.py:1688 in invoke     │               
                             │                                                            │               
                             │   1685 │   │   │   │   super().invoke(ctx)                 │               
                             │   1686 │   │   │   │   sub_ctx = cmd.make_context(cmd_name │               
                             │   1687 │   │   │   │   with sub_ctx:                       │               
                             │ ❱ 1688 │   │   │   │   │   return _process_result(sub_ctx. │               
                             │   1689 │   │                                               │               
                             │   1690 │   │   # In chain mode we create the contexts step │               
                             │   1691 │   │   # base command has been invoked.  Because a │               
                             │                                                            │               
                             │ /Users/jayanthkumar/Documents/development/autorag/venv/lib │               
                             │ /python3.10/site-packages/click/core.py:1434 in invoke     │               
                             │                                                            │               
                             │   1431 │   │   │   echo(style(message, fg="red"), err=True │               
                             │   1432 │   │                                               │               
                             │   1433 │   │   if self.callback is not None:               │               
                             │ ❱ 1434 │   │   │   return ctx.invoke(self.callback, **ctx. │               
                             │   1435 │                                                   │               
                             │   1436 │   def shell_complete(self, ctx: Context, incomple │               
                             │   1437 │   │   """Return a list of completions for the inc │               
                             │                                                            │               
                             │ /Users/jayanthkumar/Documents/development/autorag/venv/lib │               
                             │ /python3.10/site-packages/click/core.py:783 in invoke      │               
                             │                                                            │               
                             │    780 │   │                                               │               
                             │    781 │   │   with augment_usage_errors(__self):          │               
                             │    782 │   │   │   with ctx:                               │               
                             │ ❱  783 │   │   │   │   return __callback(*args, **kwargs)  │               
                             │    784 │                                                   │               
                             │    785 │   def forward(                                    │               
                             │    786 │   │   __self, __cmd: "Command", *args: t.Any, **k │               
                             │                                                            │               
                             │ /Users/jayanthkumar/Documents/development/autorag/venv/lib │               
                             │ /python3.10/site-packages/autorag/cli.py:54 in evaluate    │               
                             │                                                            │               
                             │    51 │   if not os.path.exists(config):                   │               
                             │    52 │   │   raise ValueError(f"Config file {config} does │               
                             │    53 │   evaluator = Evaluator(qa_data_path, corpus_data_ │               
                             │ ❱  54 │   evaluator.start_trial(config, skip_validation=sk │               
                             │    55                                                      │               
                             │    56                                                      │               
                             │    57 @click.command()                                     │               
                             │                                                            │               
                             │ /Users/jayanthkumar/Documents/development/autorag/venv/lib │               
                             │ /python3.10/site-packages/autorag/evaluator.py:138 in      │               
                             │ start_trial                                                │               
                             │                                                            │               
                             │   135 │   │   │   validator = Validator(                   │               
                             │   136 │   │   │   │   qa_data_path=self.qa_data_path, corp │               
                             │   137 │   │   │   )                                        │               
                             │ ❱ 138 │   │   │   validator.validate(yaml_path)            │               
                             │   139 │   │                                                │               
                             │   140 │   │   os.environ["PROJECT_DIR"] = self.project_dir │               
                             │   141                                                      │               
                             │                                                            │               
                             │ /Users/jayanthkumar/Documents/development/autorag/venv/lib │               
                             │ /python3.10/site-packages/autorag/validator.py:46 in       │               
                             │ validate                                                   │               
                             │                                                            │               
                             │   43 │                                                     │               
                             │   44 │   def validate(self, yaml_path: str, qa_cnt: int =  │               
                             │   45 │   │   # sample QA data                              │               
                             │ ❱ 46 │   │   sample_qa_df = self.qa_data.sample(qa_cnt, ra │               
                             │   47 │   │   sample_qa_df.reset_index(drop=True, inplace=T │               
                             │   48 │   │                                                 │               
                             │   49 │   │   # get doc_id                                  │               
                             │                                                            │               
                             │ /Users/jayanthkumar/Documents/development/autorag/venv/lib │               
                             │ /python3.10/site-packages/pandas/core/generic.py:6118 in   │               
                             │ sample                                                     │               
                             │                                                            │               
                             │    6115 │   │   if weights is not None:                    │               
                             │    6116 │   │   │   weights = sample.preprocess_weights(se │               
                             │    6117 │   │                                              │               
                             │ ❱  6118 │   │   sampled_indices = sample.sample(obj_len, s │               
                             │    6119 │   │   result = self.take(sampled_indices, axis=a │               
                             │    6120 │   │                                              │               
                             │    6121 │   │   if ignore_index:                           │               
                             │                                                            │               
                             │ /Users/jayanthkumar/Documents/development/autorag/venv/lib │               
                             │ /python3.10/site-packages/pandas/core/sample.py:152 in     │               
                             │ sample                                                     │               
                             │                                                            │               
                             │   149 │   │   else:                                        │               
                             │   150 │   │   │   raise ValueError("Invalid weights: weigh │               
                             │   151 │                                                    │               
                             │ ❱ 152 │   return random_state.choice(obj_len, size=size, r │               
                             │   153 │   │   np.intp, copy=False                          │               
                             │   154 │   )                                                │               
                             │   155                                                      │               
                             │                                                            │               
                             │ in numpy.random.mtrand.RandomState.choice:1001             │               
                             ╰────────────────────────────────────────────────────────────╯               
                             ValueError: Cannot take a larger sample than population when                 
                             'replace=False'         
```

**Code that bug is happened**
evaluator.start_trial('/content/config.yaml')

**Desktop (please complete the following information):**
- OS: MacOS
- Python version: 3.10

**Additional context**
It appears that the number of records requested in the sampling (qa_cnt) exceeds the available records in qa_data.
Consider adding a check to compare qa_cnt with the actual count of records before sampling, or using min(qa_cnt, len(self.qa_data)) in the sampling function to prevent this error.


@jayanthkmr Hello!
It looks like your `qa.parquet` rows are less than 5. 
For validation, you have to prepare more than 5 QAs.
Plus, for accurate result, we recommend to use more than 30 QAs at least.