Repository: explodinggradients/ragas
Issue: FactualCorrectness metric scoring, PromptValue has no len() -
Root cause: Interface Contract Mismatch

I am following this document https://docs.ragas.io/en/latest/concepts/metrics/available_metrics/factual_correctness/#example

Ragas version:0.2.6
Python version: 3.12.3

I have looked into this issue, but its not resolved:  https://github.com/explodinggradients/ragas/issues/576

```
from langchain_openai.chat_models import AzureChatOpenAI
from langchain_openai.embeddings import AzureOpenAIEmbeddings

azure_model=AzureChatOpenAI(
                api_version=config.pf_api_version,
                azure_endpoint=config.pf_oa_endpoint,
                azure_ad_token=token,
                max_tokens=config.max_tokens,
                model=config.pf_llm_deployment,
    
            )

azure_embeddings = AzureOpenAIEmbeddings(
    openai_api_version=config.pf_api_version,
    azure_endpoint=config.pf_oa_endpoint_embed,
    azure_ad_token=token,
    model=config.pf_embedding_engine,
)
```

from ragas.dataset_schema import SingleTurnSample
from ragas.metrics._factual_correctness import FactualCorrectness


```
sample = SingleTurnSample(
    response="The Eiffel Tower is located in Paris.",
    reference="The Eiffel Tower is located in Paris. I has a height of 1000ft."
)

scorer = FactualCorrectness()

scorer.llm = azure_model
scorer.embeddings=azure_embeddings
await scorer.single_turn_ascore(sample)
```
Error:
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[37], line 18
     16 scorer.llm = azure_model
     17 scorer.embeddings=azure_embeddings
---> 18 await scorer.single_turn_score(sample)

File [~/Library/Python/3.12/lib/python/site-packages/ragas/metrics/base.py:285](http://localhost:8888/lab/tree/Downloads/Ragas/~/Library/Python/3.12/lib/python/site-packages/ragas/metrics/base.py#line=284), in SingleTurnMetric.single_turn_score(self, sample, callbacks)
    283     if not group_cm.ended:
    284         rm.on_chain_error(e)
--> 285     raise e
    286 else:
    287     if not group_cm.ended:

File [~/Library/Python/3.12/lib/python/site-packages/ragas/metrics/base.py:279](http://localhost:8888/lab/tree/Downloads/Ragas/~/Library/Python/3.12/lib/python/site-packages/ragas/metrics/base.py#line=278), in SingleTurnMetric.single_turn_score(self, sample, callbacks)
    275             raise ImportError(
    276                 "It seems like your running this in a jupyter-like environment. Please install nest_asyncio with `pip install nest_asyncio` to make it work."
    277             )
    278     loop = asyncio.get_event_loop()
--> 279     score = loop.run_until_complete(
    280         self._single_turn_ascore(sample=sample, callbacks=group_cm)
    281     )
    282 except Exception as e:
    283     if not group_cm.ended:

File [~/Library/Python/3.12/lib/python/site-packages/nest_asyncio.py:98](http://localhost:8888/lab/tree/Downloads/Ragas/~/Library/Python/3.12/lib/python/site-packages/nest_asyncio.py#line=97), in _patch_loop.<locals>.run_until_complete(self, future)
     95 if not f.done():
     96     raise RuntimeError(
     97         'Event loop stopped before Future completed.')
---> 98 return f.result()

File [/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/asyncio/futures.py:203](http://localhost:8888/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/asyncio/futures.py#line=202), in Future.result(self)
    201 self.__log_traceback = False
    202 if self._exception is not None:
--> 203     raise self._exception.with_traceback(self._exception_tb)
    204 return self._result

File [/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/asyncio/tasks.py:314](http://localhost:8888/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/asyncio/tasks.py#line=313), in Task.__step_run_and_handle_result(***failed resolving arguments***)
    310 try:
    311     if exc is None:
    312         # We use the `send` method directly, because coroutines
    313         # don't have `__iter__` and `__next__` methods.
--> 314         result = coro.send(None)
    315     else:
    316         result = coro.throw(exc)

File [~/Library/Python/3.12/lib/python/site-packages/ragas/metrics/_factual_correctness.py:279](http://localhost:8888/lab/tree/Downloads/Ragas/~/Library/Python/3.12/lib/python/site-packages/ragas/metrics/_factual_correctness.py#line=278), in FactualCorrectness._single_turn_ascore(self, sample, callbacks)
    276 assert reference is not None, "Reference is not set"
    277 assert response is not None, "Response is not set"
--> 279 response_claims = await self.decompose_claims(response, callbacks)
    280 reference_response = await self.verify_claims(
    281     premise=reference, hypothesis_list=response_claims, callbacks=callbacks
    282 )
    284 if self.mode != "precision":

File [~/Library/Python/3.12/lib/python/site-packages/ragas/metrics/_factual_correctness.py:252](http://localhost:8888/lab/tree/Downloads/Ragas/~/Library/Python/3.12/lib/python/site-packages/ragas/metrics/_factual_correctness.py#line=251), in FactualCorrectness.decompose_claims(self, response, callbacks)
    250 assert isinstance(sentences, list), "Segmenter must return a list of sentences"
    251 prompt_input = ClaimDecompositionInput(response=response, sentences=sentences)
--> 252 result = await self.claim_decomposition_prompt.generate(
    253     data=prompt_input, llm=self.llm, callbacks=callbacks
    254 )
    255 claims_list = [
    256     claim for claim_list in result.decomposed_claims for claim in claim_list
    257 ]
    258 return claims_list

File [~/Library/Python/3.12/lib/python/site-packages/ragas/prompt/pydantic_prompt.py:125](http://localhost:8888/lab/tree/Downloads/Ragas/~/Library/Python/3.12/lib/python/site-packages/ragas/prompt/pydantic_prompt.py#line=124), in PydanticPrompt.generate(self, llm, data, temperature, stop, callbacks, retries_left)
    122 callbacks = callbacks or []
    124 # this is just a special case of generate_multiple
--> 125 output_single = await self.generate_multiple(
    126     llm=llm,
    127     data=data,
    128     n=1,
    129     temperature=temperature,
    130     stop=stop,
    131     callbacks=callbacks,
    132     retries_left=retries_left,
    133 )
    134 return output_single[0]

File [~/Library/Python/3.12/lib/python/site-packages/ragas/prompt/pydantic_prompt.py:185](http://localhost:8888/lab/tree/Downloads/Ragas/~/Library/Python/3.12/lib/python/site-packages/ragas/prompt/pydantic_prompt.py#line=184), in PydanticPrompt.generate_multiple(self, llm, data, n, temperature, stop, callbacks, retries_left)
    178 prompt_rm, prompt_cb = new_group(
    179     name=self.name,
    180     inputs={"data": processed_data},
    181     callbacks=callbacks,
    182     metadata={"type": ChainType.RAGAS_PROMPT},
    183 )
    184 prompt_value = PromptValue(text=self.to_string(processed_data))
--> 185 resp = await llm.generate(
    186     prompt_value,
    187     n=n,
    188     temperature=temperature,
    189     stop=stop,
    190     callbacks=prompt_cb,
    191 )
    193 output_models = []
    194 parser = RagasOutputParser(pydantic_object=self.output_model)

File [~/Library/Python/3.12/lib/python/site-packages/langchain_core/language_models/chat_models.py:627](http://localhost:8888/lab/tree/Downloads/Ragas/~/Library/Python/3.12/lib/python/site-packages/langchain_core/language_models/chat_models.py#line=626), in BaseChatModel.generate(self, messages, stop, callbacks, tags, metadata, run_name, run_id, **kwargs)
    606 inheritable_metadata = {
    607     **(metadata or {}),
    608     **self._get_ls_params(stop=stop, **kwargs),
    609 }
    611 callback_manager = CallbackManager.configure(
    612     callbacks,
    613     self.callbacks,
   (...)
    618     self.metadata,
    619 )
    620 run_managers = callback_manager.on_chat_model_start(
    621     self._serialized,
    622     messages,
    623     invocation_params=params,
    624     options=options,
    625     name=run_name,
    626     run_id=run_id,
--> 627     batch_size=len(messages),
    628 )
    629 results = []
    630 for i, m in enumerate(messages):

**TypeError: object of type 'StringPromptValue' has no len()**

Either RAGAS in not formatting SingleTurnSample into format the langchain requires?

Your documentation is very confusing. 

@jjmachan @shahules786 @kpeters




Hey, please wrap the Azure model on Langchainmodelwrapper as shown here https://docs.ragas.io/en/stable/howtos/customizations/customize_models/#azure-openai. Close the issue if this solved it for you.