Repository: truera/trulens
Issue: Context (document Chunks) are incorrect for RAG
Root cause: Validation Gap

Hello,

Somehow for Bedrock and OpenAI both, I see the context (Statement) for context_relevant seems to be the actual LLM response, not the document chunks retriever retrieved from the vector index. It seems the same statement is carried forward for groundedness score as well.

The code is provided below. However, I noticed, I started getting error message below the code. Could it be due to this error ?

**Instrumentation Code:**

```
from langchain.chains import RetrievalQA
from langchain.prompts import PromptTemplate
from langchain.llms.bedrock import Bedrock
from langchain.chat_models import ChatOpenAI
from langchain_core.vectorstores import VectorStoreRetriever
from trulens_eval.tru_custom_app import instrument

class open_ai_rag:
    @instrument
    def query(self, query: str,retriever: VectorStoreRetriever):
        """
        Retrieve relevant text from vector store.
        """
        llm = ChatOpenAI(model_name="gpt-3.5-turbo", temperature=0)
        prompt = hub.pull("rlm/rag-prompt")
        rag_chain = (
            {"context": retriever | format_docs, "question": RunnablePassthrough()}
            | prompt
            | llm
            | StrOutputParser()
        )
        
        completion = rag_chain.invoke(query)
        return completion#,source_documents


pgvector_openai_rag = TruCustomApp(openai_rag,
    app_id = '1_RAG_PGV_SemanticSearch_OpenAI',
    feedbacks = [f_groundedness, f_qa_relevance, f_context_relevance])

for question in eval_questions:
    with pgvector_openai_rag as recording:
        response = openai_rag.query(question,pgvector_retriever)
```

**Error:**

Function <function open_ai_rag.query at 0x7fab485fe5f0> was not found during instrumentation walk. Make sure it is accessible by traversing app <__main__.bedrock_rag object at 0x7fab4c1b12a0> or provide a bound method for it as TruCustomApp constructor argument `methods_to_instrument`.

Unsure what the main input string is for the call to query with args ['What are the keys to building a career in AI?', VectorStoreRetriever(tags=['PGVector', 'OpenAIEmbeddings'], vectorstore=<langchain_community.vectorstores.pgvector.PGVector object at 0x7fab45fa50f0>, search_type='similarity_score_threshold', search_kwargs={'score_threshold': 0.5})].

**Screenshots:**
I have provided the OpenAI example for the same question and it seems when OpenAI is used as LLM, context seems to 

<img width="1566" alt="image" src="https://github.com/truera/trulens/assets/43816307/ea323da0-a473-4f59-a3e0-0820d6b17cd9">

<img width="1596" alt="image" src="https://github.com/truera/trulens/assets/43816307/82c7639c-b5af-4464-b402-745be601e2d9">



Hey @SanjuMLGeek - the selection of text is done by the user via `.on()` in the feedback definition.

For reference, see: https://www.trulens.org/trulens_eval/feedback_function_guide/

Let me know if you need additional assistance here.

``Hi @joshreini1 - I already doing that in feedback function per TueLens example provided in Deeplearning.AI class.

I somehow started getting the below error recently as below which I suspect causing underlying issues. I tried changing the methods_to_instrument = open_ai_rag.query but it didn't work.
Error:

Function <function open_ai_rag.query at 0x7fab485fe5f0> was not found during instrumentation walk. Make sure it is accessible by traversing app <main.bedrock_rag object at 0x7fab4c1b12a0> or provide a bound method for it as TruCustomApp constructor argument methods_to_instrument.


```
# Initialize provider class
tru = Tru()
tru.reset_database()
fopenai = fOpenAI()

grounded = Groundedness(groundedness_provider=fopenai)

# Define a groundedness feedback function
f_groundedness = (
    Feedback(grounded.groundedness_measure_with_cot_reasons, name = "Groundedness")
    .on(Select.RecordCalls.query.rets.collect())
    .on_output()
    .aggregate(grounded.grounded_statements_aggregator)
)

# Question/answer relevance between overall question and answer.
f_qa_relevance = (
    Feedback(fopenai.relevance_with_cot_reasons, name = "Answer Relevance")
    .on(Select.RecordCalls.query.args.query)
    .on_output()
)

# Question/statement relevance between question and each context chunk.
f_context_relevance = (
    Feedback(fopenai.qs_relevance_with_cot_reasons, name = "Context Relevance")
    .on(Select.RecordCalls.query.args.query)
    .on(Select.RecordCalls.query.rets.collect())
    .aggregate(np.mean)
)
```