Repository: Marker-Inc-Korea/AutoRAG
Issue: [BUG] Unable to Route WatsonxLLM Requests to /ml/v1/text/generation — Defaults to Incorrect Endpoint
Root cause: Configuration Error

**Describe the bug**  
In AutoRAG's QA creation process using `WatsonxLLM`, setting the `url` parameter as `f"{os.getenv('RUNTIME_ENV_APSX_URL')}"` does not correctly route requests to the intended `/ml/v1/text/generation` endpoint. Instead, it defaults to `/m1/v1/text/chat`, resulting in a 404 error.

**To Reproduce**  
Steps to reproduce the behavior:
1. Set up `WatsonxLLM` as shown below, ensuring `url=f"{os.getenv('RUNTIME_ENV_APSX_URL')}"` without appending `/ml/v1/text/generation`.
2. Run the QA creation in AutoRAG.
3. Note the error, as `WatsonxLLM` still attempts to reach `/m1/v1/text/chat` instead of the correct endpoint.

**Expected behavior**  
The `WatsonxLLM` instance should route the request to `/ml/v1/text/generation` as intended when `url=f"{os.getenv('RUNTIME_ENV_APSX_URL')}"` is set.

**Full Error log**  
```
---------------------------------------------------------------------------
ApiRequestFailure                         Traceback (most recent call last)
Cell In[107], line 24
     15 corpus_df = pd.read_parquet("chunk.parquet")
     16 corpus_instance = Corpus(corpus_df, raw_instance)
     18 initial_qa = (
     19     corpus_instance.sample(random_single_hop, n=3)
     20     .map(
     21         lambda df: df.reset_index(drop=True),
     22     )
     23     .make_retrieval_gt_contents()
---> 24     .batch_apply(
     25         factoid_query_gen,  # query generation
     26         llm=llm,
     27     )
     28     .batch_apply(
     29         make_basic_gen_gt,  # answer generation (basic)
     30         llm=llm,
     31     )
     32     .batch_apply(
     33         make_concise_gen_gt,  # answer generation (concise)
     34         llm=llm,
     35     )
     36     .filter(
     37         dontknow_filter_rule_based,  # filter don't know
     38         lang="ko",
     39     )
     40 )
     42 initial_qa.to_parquet('./qa.parquet', './corpus.parquet')

File /opt/conda/envs/Python-RT24.1-CUDA/lib/python3.11/site-packages/autorag/data/qa/schema.py:139, in QA.batch_apply(self, fn, batch_size, **kwargs)
    137 loop = get_event_loop()
    138 tasks = [fn(qa_dict, **kwargs) for qa_dict in qa_dicts]
--> 139 results = loop.run_until_complete(process_batch(tasks, batch_size))
    140 return QA(pd.DataFrame(results), self.linked_corpus)

File /opt/conda/envs/Python-RT24.1-CUDA/lib/python3.11/site-packages/nest_asyncio.py:98, in _patch_loop.<locals>.run_until_complete(self, future)
     95 if not f.done():
     96     raise RuntimeError(
     97         'Event loop stopped before Future completed.')
---> 98 return f.result()

File /opt/conda/envs/Python-RT24.1-CUDA/lib/python3.11/asyncio/futures.py:203, in Future.result(self)
    201 self.__log_traceback = False
    202 if self._exception is not None:
--> 203     raise self._exception.with_traceback(self._exception_tb)
    204 return self._result

File /opt/conda/envs/Python-RT24.1-CUDA/lib/python3.11/asyncio/tasks.py:279, in Task.__step(***failed resolving arguments***)
    277         result = coro.send(None)
    278     else:
--> 279         result = coro.throw(exc)
    280 except StopIteration as exc:
    281     if self._must_cancel:
    282         # Task is cancelled right before coro stops.

File /opt/conda/envs/Python-RT24.1-CUDA/lib/python3.11/site-packages/autorag/utils/util.py:304, in process_batch(tasks, batch_size)
    302 for i in range(0, len(tasks), batch_size):
    303 	batch = tasks[i : i + batch_size]
--> 304 	batch_results = await asyncio.gather(*batch)
    305 	results.extend(batch_results)
    307 return results

File /opt/conda/envs/Python-RT24.1-CUDA/lib/python3.11/asyncio/tasks.py:349, in Task.__wakeup(self, future)
    347 def __wakeup(self, future):
    348     try:
--> 349         future.result()
    350     except BaseException as exc:
    351         # This may also be a cancellation.
    352         self.__step(exc)

File /opt/conda/envs/Python-RT24.1-CUDA/lib/python3.11/asyncio/tasks.py:277, in Task.__step(***failed resolving arguments***)
    273 try:
    274     if exc is None:
    275         # We use the `send` method directly, because coroutines
    276         # don't have `__iter__` and `__next__` methods.
--> 277         result = coro.send(None)
    278     else:
    279         result = coro.throw(exc)

File /opt/conda/envs/Python-RT24.1-CUDA/lib/python3.11/site-packages/autorag/data/qa/query/llama_gen_query.py:30, in factoid_query_gen(row, llm, lang)
     25 async def factoid_query_gen(
     26 	row: Dict,
     27 	llm: BaseLLM,
     28 	lang: str = "en",
     29 ) -> Dict:
---> 30 	return await llama_index_generate_base(
     31 		row, llm, QUERY_GEN_PROMPT["factoid_single_hop"][lang]
     32 	)

File /opt/conda/envs/Python-RT24.1-CUDA/lib/python3.11/site-packages/autorag/data/qa/query/llama_gen_query.py:20, in llama_index_generate_base(row, llm, messages)
     18 user_message = ChatMessage(role=MessageRole.USER, content=user_prompt)
     19 new_messages = [*messages, user_message]
---> 20 chat_response: ChatResponse = await llm.achat(messages=new_messages)
     21 row["query"] = chat_response.message.content
     22 return row

File /opt/conda/envs/Python-RT24.1-CUDA/lib/python3.11/site-packages/llama_index/core/instrumentation/dispatcher.py:357, in Dispatcher.span.<locals>.async_wrapper(func, instance, args, kwargs)
    349 self.span_enter(
    350     id_=id_,
    351     bound_args=bound_args,
   (...)
    354     tags=tags,
    355 )
    356 try:
--> 357     result = await func(*args, **kwargs)
    358 except BaseException as e:
    359     self.event(SpanDropEvent(span_id=id_, err_str=str(e)))

File /opt/conda/envs/Python-RT24.1-CUDA/lib/python3.11/site-packages/llama_index/core/llms/callbacks.py:75, in llm_chat_callback.<locals>.wrap.<locals>.wrapped_async_llm_chat(_self, messages, **kwargs)
     66 event_id = callback_manager.on_event_start(
     67     CBEventType.LLM,
     68     payload={
   (...)
     72     },
     73 )
     74 try:
---> 75     f_return_val = await f(_self, messages, **kwargs)
     76 except BaseException as e:
     77     callback_manager.on_event_end(
     78         CBEventType.LLM,
     79         payload={EventPayload.EXCEPTION: e},
     80         event_id=event_id,
     81     )

File /opt/conda/envs/Python-RT24.1-CUDA/lib/python3.11/site-packages/llama_index/llms/ibm/base.py:450, in WatsonxLLM.achat(self, messages, **kwargs)
    444 @llm_chat_callback()
    445 async def achat(
    446     self,
    447     messages: Sequence[ChatMessage],
    448     **kwargs: Any,
    449 ) -> ChatResponse:
--> 450     return self.chat(messages, **kwargs)

File /opt/conda/envs/Python-RT24.1-CUDA/lib/python3.11/site-packages/llama_index/core/instrumentation/dispatcher.py:311, in Dispatcher.span.<locals>.wrapper(func, instance, args, kwargs)
    308             _logger.debug(f"Failed to reset active_span_id: {e}")
    310 try:
--> 311     result = func(*args, **kwargs)
    312     if isinstance(result, asyncio.Future):
    313         # If the result is a Future, wrap it
    314         new_future = asyncio.ensure_future(result)

File /opt/conda/envs/Python-RT24.1-CUDA/lib/python3.11/site-packages/llama_index/core/llms/callbacks.py:173, in llm_chat_callback.<locals>.wrap.<locals>.wrapped_llm_chat(_self, messages, **kwargs)
    164 event_id = callback_manager.on_event_start(
    165     CBEventType.LLM,
    166     payload={
   (...)
    170     },
    171 )
    172 try:
--> 173     f_return_val = f(_self, messages, **kwargs)
    174 except BaseException as e:
    175     callback_manager.on_event_end(
    176         CBEventType.LLM,
    177         payload={EventPayload.EXCEPTION: e},
    178         event_id=event_id,
    179     )

File /opt/conda/envs/Python-RT24.1-CUDA/lib/python3.11/site-packages/llama_index/llms/ibm/base.py:442, in WatsonxLLM.chat(self, messages, **kwargs)
    439 else:
    440     chat_fn = self._chat
--> 442 return chat_fn(messages, **kwargs)

File /opt/conda/envs/Python-RT24.1-CUDA/lib/python3.11/site-packages/llama_index/llms/ibm/base.py:419, in WatsonxLLM._chat(self, messages, **kwargs)
    416 message_dicts = [to_watsonx_message_dict(message) for message in messages]
    418 params, generation_kwargs = self._split_chat_generation_params(kwargs)
--> 419 response = self._model.chat(
    420     messages=message_dicts,
    421     params=params,
    422     tools=generation_kwargs.get("tools"),
    423     tool_choice=generation_kwargs.get("tool_choice"),
    424     tool_choice_option=generation_kwargs.get("tool_choice_option"),
    425 )
    427 wx_message = response["choices"][0]["message"]
    428 message = from_watsonx_message(wx_message)

File /opt/conda/envs/Python-RT24.1-CUDA/lib/python3.11/site-packages/ibm_watsonx_ai/foundation_models/inference/model_inference.py:271, in ModelInference.chat(self, messages, params, tools, tool_choice, tool_choice_option)
    266 if self.model_id is None:
    267     raise WMLClientError(
    268         Messages.get_message(message_id="chat_deployment_scenario")
    269     )
--> 271 return self._inference.chat(
    272     messages=messages,
    273     params=params,
    274     tools=tools,
    275     tool_choice=tool_choice,
    276     tool_choice_option=tool_choice_option,
    277 )

File /opt/conda/envs/Python-RT24.1-CUDA/lib/python3.11/site-packages/ibm_watsonx_ai/foundation_models/inference/fm_model_inference.py:129, in FMModelInference.chat(self, messages, params, tools, tool_choice, tool_choice_option)
    116 def chat(
    117     self,
    118     messages: list[dict] | None = None,
   (...)
    122     tool_choice_option: Literal["none", "auto"] | None = None,
    123 ) -> dict:
    125     text_chat_url = (
    126         self._client.service_instance._href_definitions.get_fm_chat_href("chat")
    127     )
--> 129     return self._send_chat_payload(
    130         messages=messages,
    131         params=params,
    132         generate_url=text_chat_url,
    133         tools=tools,
    134         tool_choice=tool_choice,
    135         tool_choice_option=tool_choice_option,
    136     )

File /opt/conda/envs/Python-RT24.1-CUDA/lib/python3.11/site-packages/ibm_watsonx_ai/foundation_models/inference/base_model_inference.py:273, in BaseModelInference._send_chat_payload(self, messages, params, generate_url, tools, tool_choice, tool_choice_option)
    269     post_params["_retry_status_codes"] = _RETRY_STATUS_CODES
    271 response_scoring = self._http_client.post(**post_params)
--> 273 return self._handle_response(
    274     200,
    275     "generate",
    276     response_scoring,
    277     _field_to_hide="generated_chat",
    278 )

File /opt/conda/envs/Python-RT24.1-CUDA/lib/python3.11/site-packages/ibm_watsonx_ai/wml_resource.py:155, in WMLResource._handle_response(self, expected_status_code, operationName, response, json_response, _silent_response_logging, _field_to_hide)
    153         return response.text
    154 else:
--> 155     raise ApiRequestFailure(
    156         "Failure during {}.".format(operationName),
    157         response,
    158     )

ApiRequestFailure: Failure during generate. (POST [Internal URL]
Status code: 404, body: {"errors":[{"code":"path_not_found_error","message":"URI path '/ml/v1/text/chat' does not exist","more_info":"https://cloud.ibm.com/apidocs/watsonx-ai-cp"}],"trace":"83e0d568-eec6-4a83-80aa-9bb1c5acec70","status_code":404}
```

**Code that bug is happened**  
```python
from llama_index.llms.ibm import WatsonxLLM
import os

llm = WatsonxLLM(
    model_id="meta-llama/llama-3-1-8b-instruct",
    url=f"{os.getenv('RUNTIME_ENV_APSX_URL')}",  # Only base URL is set here
    username="<username>",
    password="<password>",
    instance_id="openshift",
    version="5.0",
    space_id="<space_id>",
    temperature=0.1,
    max_new_tokens=200,
)
```

```python
import pandas as pd

from autorag.data.qa.filter.dontknow import dontknow_filter_rule_based
from autorag.data.qa.generation_gt.llama_index_gen_gt import (
    make_basic_gen_gt,
    make_concise_gen_gt,
)
from autorag.data.qa.schema import Raw, Corpus
from autorag.data.qa.query.llama_gen_query import factoid_query_gen
from autorag.data.qa.sample import random_single_hop

raw_df = pd.read_parquet("parse_done.parquet")
raw_instance = Raw(raw_df)

corpus_df = pd.read_parquet("chunk.parquet")
corpus_instance = Corpus(corpus_df, raw_instance)

initial_qa = (
    corpus_instance.sample(random_single_hop, n=3)
    .map(
        lambda df: df.reset_index(drop=True),
    )
    .make_retrieval_gt_contents()
    .batch_apply(
        factoid_query_gen,  # query generation
        llm=llm,
    )
    .batch_apply(
        make_basic_gen_gt,  # answer generation (basic)
        llm=llm,
    )
    .batch_apply(
        make_concise_gen_gt,  # answer generation (concise)
        llm=llm,
    )
    .filter(
        dontknow_filter_rule_based,  # filter don't know
        lang="ko",
    )
)

initial_qa.to_parquet('./qa.parquet', './corpus.parquet')
```

**Desktop (please complete the following information):**
- OS: Linux
- Python version: 3.11.9

**Additional Context**  
The packages were installed as follows:
```bash
!pip install "AutoRAG[gpu,parse]"
!pip install -qU llama-index-llms-ibm
```

Even with `url=f"{os.getenv('RUNTIME_ENV_APSX_URL')}"`, `WatsonxLLM` defaults to the `/m1/v1/text/chat` endpoint, which causes the issue. An update is needed to ensure that requests are correctly routed to the intended endpoint.

@jjaegii Hello!
I think it is because of the LlamaIndex. 
We use `llm.achat` function, so maybe it will use chat endpoint. 

One option is you can change the function of the each step. 
For example, you can find the prompt of `factoid_query_gen` and making the similar function that uses `llm.acomplete`. 

Second option is changing the LlamaIndex LLM class itself to use `/ml/v1/text/generation` endpoint at `achat` also.

Hi! Thanks for the suggestions.

It seems that the issue indeed stems from `llm.achat` calling the chat endpoint, which isn’t available in my environment. I’ll try creating custom versions of each function step (like `factoid_query_gen`) to use `llm.acomplete` with a prompt-based approach. If needed, I’ll also consider modifying the `LlamaIndex` LLM class to redirect `achat` to the `/ml/v1/text/generation` endpoint.

I appreciate the guidance—this gives me a good starting point.