Repository: truera/trulens
Issue: [BUG] Trulens is working properly with the Query Engine but not with the Chat Engine.
Root cause: Interface Contract Mismatch

**Bug Description**
I'm trying to generate a dashboard using Trulens with the query engine, and it is working properly. However, if I use the Chat engine, I am getting the following error:
```
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃                                            Selector check failed                                            ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛

Source of argument source to Groundedness does not exist in app or expected record:


 __record__.app.query.rets.source_nodes[:].node.text.collect()
 # or equivalently
 Select.RecordCalls.query.rets.source_nodes[:].node.text.collect()
                                                                                                               

The data used to make this check may be incomplete. If you expect records produced by your app to contain the  
selected content, you can ignore this error by setting selectors_nocheck in the TruLlama constructor.
Alternatively, setting selectors_check_warning will print out this message but will not raise an error.        


                                            Additional information:

Feedback function signature:

                                                                                                               
 (source: str, statement: str) -> Tuple[float, dict]
                                                                                                               

The prefix __record__.app selects this data that exists in your app or typical records:

 • Object of type dict starting with:

                                                                                                               
       {
         '_retriever': {'retrieve': [...], '_retrieve': [...], '_aretrieve': [...]},
         '_llm': {
           'wrapped_llm_predict': [...],
           'wrapped_async_llm_predict': [...],
           'wrapped_llm_chat': [...],
           'wrapped_async_llm_chat': [...]
         },
         '_memory': {'put': [...]},
         'chat': [RecordAppCall(...), RecordAppCall(...)],
         'achat': [RecordAppCall(...), RecordAppCall(...)],
         'stream_chat': [RecordAppCall(...), RecordAppCall(...)]
       }
                                                                                                               
Traceback (most recent call last):
  File "C:\Users\dhruvin.koshiya\Documents\AI Hackathons\ithelpdesk-bot-rag\test.py", line 110, in <module>    
    chat.handle_query("user query")
  File "C:\Users\dhruvin.koshiya\Documents\AI Hackathons\ithelpdesk-bot-rag\test.py", line 98, in handle_query 
    tru_query_engine_recorder = TruLlama(self.chat_engine_base,
  File "C:\Users\dhruvin.koshiya\Documents\AI Hackathons\ithelpdesk-bot-rag\.venv\lib\site-packages\trulens_eval\tru_llama.py", line 290, in __init__
    super().__init__(**kwargs)
  File "C:\Users\dhruvin.koshiya\Documents\AI Hackathons\ithelpdesk-bot-rag\.venv\lib\site-packages\trulens_eval\app.py", line 563, in __init__
    self._tru_post_init()
  File "C:\Users\dhruvin.koshiya\Documents\AI Hackathons\ithelpdesk-bot-rag\.venv\lib\site-packages\trulens_eval\app.py", line 710, in _tru_post_init
    feedback.check_selectors(
  File "C:\Users\dhruvin.koshiya\Documents\AI Hackathons\ithelpdesk-bot-rag\.venv\lib\site-packages\trulens_eval\feedback\feedback.py", line 732, in check_selectors
    raise ValueError(
ValueError: Some selectors do not exist in the app or record.
```

**To Reproduce**
```
import os
from dotenv import load_dotenv
from llama_index.llms.openai import OpenAI
from llama_index.core import Settings
from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index.core.indices.vector_store.base import VectorStoreIndex
from llama_index.core import SimpleDirectoryReader, StorageContext
from llama_index.core.node_parser import SentenceSplitter
from llama_index.core.memory import ChatMemoryBuffer
from pinecone import Pinecone
from llama_index.vector_stores.pinecone import PineconeVectorStore
from llama_index.core.memory import ChatMemoryBuffer
from trulens_eval import TruLlama
from trulens_eval import Feedback
from trulens_eval.feedback.provider import OpenAI as AI
from trulens_eval.feedback import Groundedness
from trulens_eval.app import App
load_dotenv()

class OpenAIChat:
    def __init__(self):
        self.openai_api_key = os.getenv("OPENAI_API_KEY")
        self.pinecone_index = self.create_pinecone_index(os.getenv("PINECONE_API_KEY"))
        self.setup_llama_index_settings()
        self.setup_chat_engine()

    def create_pinecone_index(self, pinecone_api_key):
        pc = Pinecone(api_key=pinecone_api_key)
        return pc.Index("enter Your pinecone index")

    def setup_llama_index_settings(self):
        llm = OpenAI(model="gpt-3.5-turbo", temperature=0)
        Settings.llm = llm
        Settings.embed_model = OpenAIEmbedding(embed_batch_size=50)
        Settings.text_splitter = SentenceSplitter()
        Settings.transformations = [SentenceSplitter(chunk_size=1024)]

    def setup_chat_engine(self):
        self.documents = self.load_documents()
        self.base_index = self.load_vector_store()
        self.chat_engine_base = self.create_chat_engine()
        self.base_index = self.index_documents()
            
    def initialize_pinecone_vector_store(self):
        return PineconeVectorStore(pinecone_index=self.pinecone_index)

    def index_documents(self):
        vector_store = self.initialize_pinecone_vector_store()
        base_node_parser = SentenceSplitter()
        base_nodes = base_node_parser.get_nodes_from_documents(self.documents)
        storage_context = StorageContext.from_defaults(vector_store=vector_store)
        base_index = VectorStoreIndex(base_nodes, storage_context=storage_context)
        return base_index

    def load_documents(self):
        documents = SimpleDirectoryReader("Enter your data path").load_data()
        return documents

    def load_vector_store(self):
        vector_store = PineconeVectorStore(pinecone_index=self.pinecone_index)
        index = VectorStoreIndex.from_vector_store(vector_store)
        return index

    def create_chat_engine(self):
        chat_engine = self.base_index.as_chat_engine(
            chat_mode="context",
            memory=ChatMemoryBuffer.from_defaults(token_limit=3000),
            system_prompt="Enter your prompt",
        )
        return chat_engine

    def create_feedback_function(self):
        openai = AI()
        openai.endpoint.rpm = 3
        context = App.select_context(self.chat_engine_base)
        grounded = Groundedness(groundedness_provider=AI())

        f_groundedness = (
            Feedback(grounded.groundedness_measure_with_cot_reasons, name="Groundedness")
            .on(context.collect())
            .on_output()
            .aggregate(grounded.grounded_statements_aggregator)
        )

        return [f_groundedness]

    def handle_query(self, user_query):
        tru_functions = self.create_feedback_function()
        tru_query_engine_recorder = TruLlama(self.chat_engine_base,
            app_id='Experiment-1',
            feedbacks=tru_functions
        )

        with tru_query_engine_recorder as recording:
            response = self.chat_engine_base.chat(user_query)
            print(response)


if __name__ == "__main__":
    chat = OpenAIChat()
    chat.handle_query("user query")
```


**Expected behavior**
There should not be any error.

**Environment:**
 - OS: Windows
 - Python Version: 3.10.1
 - TruLens version: 0.27.0
 - Versions of other relevant installed libraries: llama-index==0.10.25

<!-- Greeting -->
Hey there, @DhruvinKoshiya! Good to see you diving into more adventures with Trulens. How's everything going on your end?

<!-- Answer -->
The "Selector check failed" error you're encountering when generating a dashboard using Trulens with a Chat engine, as opposed to the Query engine, likely stems from the way the `Lens` objects are being handled or interpreted differently between the two engines. Given the provided context, the issue might not be directly visible in the code snippet, but a common cause for such errors in dynamic UI generation scenarios involves the handling of data paths or selectors.

In the provided code, the `Selector` class is responsible for managing the selection of data paths (`jpath`) through the UI. When a `Lens` object or a string representation of a lens is passed to a `Selector`, it attempts to convert it into a `Lens` object if it's not already one. This process involves parsing the string to a `Lens` object:

```py
self.jpath = Lens.of_string(select)
```

If the Chat engine and the Query engine produce or expect different data structures, the string representation used to create a `Lens` object might not be correctly interpreted across both engines. This discrepancy can lead to the "Selector check failed" error if the `Lens.of_string` method fails to parse the string correctly for one of the engines, or if the resulting `Lens` object does not correctly point to a valid path in the data structure used by the Chat engine.

To resolve this issue, consider the following steps:

1. **Verify Data Structures:** Ensure that the data structures produced or expected by the Chat engine are compatible with the `Lens` objects or string paths being generated. There might be differences in the structure or naming conventions between what the Query engine produces and what the Chat engine expects.

2. **Debug Lens Parsing:** Add logging or debugging statements around the `Lens.of_string` calls to see what string is being passed and whether it's correctly interpreted. This can help identify if the issue lies in the parsing logic or the input string.

3. **Error Handling:** Improve error handling in the `on_update_select` method of the `Selector` class to catch and log or display more informative errors when the `Lens.of_string` call fails. This can provide more insights into why the selector check fails.

4. **Engine-Specific Handling:** If the data structures or requirements are significantly different between the two engines, consider implementing engine-specific handling in the UI code. This could involve conditional logic to adjust the `Lens` objects or selectors based on the engine being used.

Without more specific details about the differences between the Chat and Query engines' data structures or how they're expected to be accessed, these steps provide a general approach to troubleshooting and resolving the "Selector check failed" error.

For more information, you can refer to the [trulens_eval/trulens_eval/appui.py](https://github.com/truera/trulens/blob/main/trulens_eval/trulens_eval/appui.py) file in the Trulens repository.

As for the structure of the data returned by the Chat engine and how it differs from the data returned by the Query engine, the data returned by the Query engine is expected to be a string (at least in the context of the response content), while the data returned by the Chat engine is structured as a JSON object, which includes numeric values that are subject to precision considerations. The tests focus on ensuring that the response content is consistent across synchronous and asynchronous calls, with specific fields related to metadata and execution details being skipped in comparisons. You can refer to the [trulens_eval/tests/e2e/test_tru_llama.py](https://github.com/truera/trulens/blob/main/trulens_eval/tests/e2e/test_tru_llama.py) file for more details.

I hope this helps! If you have any more questions or need further clarification, feel free to ask.

## Sources
- [trulens_eval/trulens_eval/appui.py](https://github.com/truera/trulens/blob/main/trulens_eval/trulens_eval/appui.py)
- [trulens_eval/trulens_eval/schema.py](https://github.com/truera/trulens/blob/main/trulens_eval/trulens_eval/schema.py)
- [trulens_eval/tests/e2e/test_tru_llama.py](https://github.com/truera/trulens/blob/main/trulens_eval/tests/e2e/test_tru_llama.py)

<!-- Dosu Comment Footer -->
<details>
<summary>

#### About Dosu
</summary>

This response is meant to be useful and save you time. It is **not meant to be a precise solution**, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.
</details>


Hi @DhruvinKoshiya - the culprit seems to be `chat_mode="context"` rather than the chat engine itself.

I was able to get this working by changing the context in your example as follows:

`context = Select.RecordCalls._retriever._retrieve.rets[:].node.text`

Can you give this a try?

@DhruvinKoshiya - did you get a chance to try this?

Closing. @DhruvinKoshiya let me know if your issue is not solved by changing the context selector as mentioned above.

@joshreini1 - I did try the suggested changes but I got the below error:
```
NameError: name 'Select' is not defined
```
Could you please clarify the source of your usage of 'Select'?

Select can be imported from TruLens-eval, e.g.

from trulens_eval import Select

*-Josh*

On Tue, Apr 16, 2024 at 1:34 AM DhruvinKoshiya ***@***.***>
wrote:

> @joshreini1 <https://github.com/joshreini1> - I did try the suggested
> changes but I got the below error:
>
> NameError: name 'Select' is not defined
>
> Could you please clarify the source of your usage of 'Select'?
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/truera/trulens/issues/1044#issuecomment-2058263856>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AORAKDTG4JANGRRQN43FY2TY5SZ5HAVCNFSM6AAAAABFTMVJ56VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANJYGI3DGOBVGY>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>