Repository: truera/trulens
Issue: Evaluation for Llama-Index Streaming
Root cause: Interface Contract Mismatch

When I customize my `query_engine` in LlamaIndex using the following snippet, as shown below, the `output` column is empty and `groundedness` column is missing from the records Dataframe. 

```
response_synthesizer = get_response_synthesizer(
    service_context=service_context,
    text_qa_template=text_qa_template,
    response_mode=ResponseMode.SIMPLE_SUMMARIZE,
    streaming=True
)

storage_context = StorageContext.from_defaults(persist_dir=s3_index_path, fs=s3)
index = load_index_from_storage(storage_context=storage_context)
retriever = VectorIndexRetriever(
    index=index,
    similarity_top_k=2
)

query_eengine = RetrieverQueryEngine.from_args(
    retriever=retriever,
    response_synthesizer=response_synthesizer,
    response_mode=ResponseMode.SIMPLE_SUMMARIZE,
    text_qa_template=text_qa_template,
    service_context=service_context,
    streaming=True
)

tru_query_engine_recorder = TruLlama(query_eengine,
                                     app_id='LlamaIndex_App2',
                                     feedbacks=[f_groundedness, f_qa_relevance, f_qs_relevance])

with tru_query_engine_recorder as recording:
    for query_text in questions[:15]:
        time.sleep(0.2)
        query_eengine.query(query_text)

```

The same thing when executed with a simple `query_engine` works fine and populates both `output` and `groundedness` columns with the query responses and the metric values respectively. A snippet showcasing usage of a simple `query_engine` is given below.

```
query_engine = index.as_query_engine(similarity_top_k=3)

tru_query_engine_recorder = TruLlama(query_engine,
                                     app_id='LlamaIndex_App2',
                                     feedbacks=[f_groundedness, f_qa_relevance, f_qs_relevance])

with tru_query_engine_recorder as recording:
    for query_text in questions[:15]:
        time.sleep(0.2)
        query_engine.query(query_text)
```


Hey @yasergsi !

I don't have access to your stored index, but I recreated what should be close to your situation (where you're using a custom response synthesizer and streaming with:

```python
documents = SimpleWebPageReader(
    html_to_text=True
).load_data(["http://paulgraham.com/worked.html"])
index = VectorStoreIndex.from_documents(documents)

response_synthesizer = get_response_synthesizer(
    response_mode=ResponseMode.SIMPLE_SUMMARIZE
)

query_engine = index.as_query_engine(
    response_synthesizer=response_synthesizer,
    streaming=True)
   ```

However with this setup, I am not seeing the same issue you are - and my feedback functions are populating properly (shown below):

![Screen Shot 2023-12-05 at 3 37 24 PM](https://github.com/truera/trulens/assets/60949774/9390ce82-967e-4da4-8eae-3ecc85174f6e)

Based on this, I think the issue may be with how your index is setup/used. Can you share a more reproducible example so I can help debug? Customizing these things should not impact the ability to use TruLens (but may change the selectors you specify for the feedback functions).

Hey @joshreini1 I don't think it's the index because the same index gives a proper output when I use a simple `query_engine` as shown in the main issue posted above. Also I see that you haven't fully replicated all of my customizations like using the `VectorIndexRetriever` object and building the `query_engine` using the `RetrieverQueryEngine` Class. 

Anyways here is the censored version of my full code that builds the index as you've shown above and still persists the issues.

```
embed_model = AzureOpenAIEmbedding(
    model=embed_model_name,
    deployment_name=embed_deployment_name,
    api_key=api_key,
    api_base=api_base,
    api_type=api_type,
    api_version=api_version,
)

llm = AzureOpenAI(
    model=model_name,
    engine=deployment_name,
    api_key=api_key,
    api_base=api_base,
    api_type=api_type,
    api_version=api_version,
    streaming=True
)

llm_predictor = LLMPredictor(llm=llm)

service_context = ServiceContext.from_defaults(llm_predictor=llm_predictor, context_window=10240,
                                               embed_model=embed_model)  # num_output=num_output)

try:
    tru = Tru()
    tru.reset_database()

    # Initialize provider class
    fopenai = fOpenAI()

    grounded = Groundedness(groundedness_provider=fOpenAI())

    # Define a groundedness feedback function
    f_groundedness = Feedback(grounded.groundedness_measure_with_cot_reasons).on(
        TruLlama.select_source_nodes().node.text
    ).on_output(
    ).aggregate(grounded.grounded_statements_aggregator)

    # Question/answer relevance between overall question and answer.
    f_qa_relevance = Feedback(fopenai.relevance).on_input_output()

    # Question/statement relevance between question and each context chunk.
    f_qs_relevance = Feedback(fopenai.qs_relevance).on_input().on(
        TruLlama.select_source_nodes().node.text
    ).aggregate(np.mean)

    with open("eval-queries.txt") as f:
        query_list = f.readlines()

    questions = [x.strip() for x in query_list if x.split()]

    text_qa_msgs = [
                       ChatMessage(role=MessageRole.SYSTEM, content=QA_SYSTEM_PROMPT_TEMPLATE),
                       ChatMessage(role=MessageRole.USER, content=CUSTOM_QA_PROMPT_TEMPLATE)
                   ]

    text_qa_template = ChatPromptTemplate(text_qa_msgs)

    response_synthesizer = get_response_synthesizer(
        service_context=service_context,
        text_qa_template=text_qa_template,
        response_mode=ResponseMode.SIMPLE_SUMMARIZE,
        streaming=True
    )

    # s3 = s3fs.S3FileSystem()
    # s3_index_path = "*****"
    # storage_context = StorageContext.from_defaults(persist_dir=s3_index_path, fs=s3)
    # index = load_index_from_storage(storage_context=storage_context)
    
    documents = SimpleWebPageReader(html_to_text=True).load_data(["http://paulgraham.com/worked.html"])
    index = VectorStoreIndex.from_documents(documents)

    retriever = VectorIndexRetriever(
        index=index,
        similarity_top_k=2
    )

    # Assemble the query engine
    query_eengine = RetrieverQueryEngine.from_args(
        retriever=retriever,
        response_synthesizer=response_synthesizer,
        response_mode=ResponseMode.SIMPLE_SUMMARIZE,
        text_qa_template=text_qa_template,
        service_context=service_context,
        streaming=True
    )

    tru_query_engine_recorder = TruLlama(query_eengine,
                                         app_id='LlamaIndex_App2',
                                         feedbacks=[f_groundedness, f_qa_relevance, f_qs_relevance])

    with tru_query_engine_recorder as recording:
        for query_text in questions[:15]:
            time.sleep(0.2)
            query_eengine.query(query_text)

    records, feedback = tru.get_records_and_feedback(app_ids=[])
    df = records[["input", "output", "relevance", "latency", "total_tokens", "total_cost"] + feedback]
    summary_local_file_path = "eval_records_summary.csv"
    df["input"] = df["input"].replace('"""', '')
    df.to_csv(summary_local_file_path, index=False)

except Exception as error:
    print(str(error))
    print(f"Traceback: {traceback.format_exc()}")

```

The Runtime errors I get is as follows:

```
Feedback Function exception caught: Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/trulens_eval/feedback/feedback.py", line 481, in run
    result_and_meta, part_cost = Endpoint.track_all_costs_tally(
  File "/usr/local/lib/python3.10/site-packages/trulens_eval/feedback/provider/endpoint/base.py", line 377, in track_all_costs_tally
    result, cbs = Endpoint.track_all_costs(
  File "/usr/local/lib/python3.10/site-packages/trulens_eval/feedback/provider/endpoint/base.py", line 303, in track_all_costs
    return Endpoint._track_costs(thunk, with_endpoints=endpoints)
  File "/usr/local/lib/python3.10/site-packages/trulens_eval/feedback/provider/endpoint/base.py", line 480, in _track_costs
    result: T = thunk()
  File "/usr/local/lib/python3.10/site-packages/trulens_eval/feedback/feedback.py", line 482, in <lambda>
    lambda: self.imp(**ins)
  File "/usr/local/lib/python3.10/site-packages/trulens_eval/feedback/groundedness.py", line 150, in groundedness_measure_with_cot_reasons
    if len(statement) > plausible_junk_char_min:
TypeError: object of type 'NoneType' has no len()

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/trulens_eval/feedback/feedback.py", line 486, in run
    raise RuntimeError(
RuntimeError: Evaluation of groundedness_measure_with_cot_reasons failed on inputs: 
{'source': 'I also worked on spam\n'
           'filters, and did some more painting. I used to have dinners for a '
           
object of type 'NoneType' has no len().
```
Along with these Runtime errors for each evaluation question asked the resulting dataframe's `output` column is empty and the `groundedness` column is missing too. I hope this information is sufficient to replicate the issue.


The issue was with `streaming`. These evaluation tools don't work if streaming response is enabled. Without the parameter `streaming=True` all the issues and errors disappear. I think there should some restrictions to block using streaming enabled `query_engine` or atleast a warning describing to use a non streaming based `query_engine`

@yasergsi can you try this approach:

```
with tru_query_engine_recorder as recording:
    response = await query_engine.aquery("What did the author do growing up?")

print(response)
```

As used in our llama-index async example here: https://github.com/truera/trulens/blob/main/trulens_eval/examples/expositional/frameworks/llama_index/llama_index_async.ipynb

I did these changes but it results to further errors (as shown below) and missing `groundedness` and `qs_relevance` columns in the feedback object.

```
Run of <lambda> in <Thread(TP.submit with debug timeout_0, started 123145504370688)> failed with: Could not locate app.query.rets.source_nodes[:].node.text in app/record.
exception calling callback for <Future at 0x12ae31150 state=finished raised RuntimeError>
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/trulens_eval/feedback/feedback.py", line 669, in extract_selection
    arg_vals[k] = list(q_within_o.get(o))
  File "/usr/local/lib/python3.10/site-packages/trulens_eval/utils/serial.py", line 851, in get
    for start_selection in start_items:
  File "/usr/local/lib/python3.10/site-packages/trulens_eval/utils/serial.py", line 851, in get
    for start_selection in start_items:
  File "/usr/local/lib/python3.10/site-packages/trulens_eval/utils/serial.py", line 851, in get
    for start_selection in start_items:
  [Previous line repeated 2 more times]
  File "/usr/local/lib/python3.10/site-packages/trulens_eval/utils/serial.py", line 852, in get
    for last_selection in last_step.get(start_selection):
  File "/usr/local/lib/python3.10/site-packages/trulens_eval/utils/serial.py", line 329, in get
    raise KeyError(
KeyError: 'Key not in dictionary: query'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/Cellar/python@3.10/3.10.13/Frameworks/Python.framework/Versions/3.10/lib/python3.10/concurrent/futures/_base.py", line 342, in _invoke_callbacks
    callback(self)
  File "/usr/local/lib/python3.10/site-packages/trulens_eval/app.py", line 917, in _add_future_feedback
    _, res = future.result()
  File "/usr/local/Cellar/python@3.10/3.10.13/Frameworks/Python.framework/Versions/3.10/lib/python3.10/concurrent/futures/_base.py", line 451, in result
    return self.__get_result()
  File "/usr/local/Cellar/python@3.10/3.10.13/Frameworks/Python.framework/Versions/3.10/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/usr/local/Cellar/python@3.10/3.10.13/Frameworks/Python.framework/Versions/3.10/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.10/site-packages/trulens_eval/utils/python.py", line 223, in _future_target_wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/trulens_eval/utils/threading.py", line 97, in _run_with_timeout
    raise e
  File "/usr/local/lib/python3.10/site-packages/trulens_eval/utils/threading.py", line 82, in _run_with_timeout
    res: T = fut.result(timeout=timeout)
  File "/usr/local/Cellar/python@3.10/3.10.13/Frameworks/Python.framework/Versions/3.10/lib/python3.10/concurrent/futures/_base.py", line 451, in result
    return self.__get_result()
  File "/usr/local/Cellar/python@3.10/3.10.13/Frameworks/Python.framework/Versions/3.10/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/usr/local/Cellar/python@3.10/3.10.13/Frameworks/Python.framework/Versions/3.10/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.10/site-packages/trulens_eval/tru.py", line 220, in <lambda>
    tp.submit(lambda f: (f, f.run(app=app, record=record)), ffunc)
  File "/usr/local/lib/python3.10/site-packages/trulens_eval/feedback/feedback.py", line 472, in run
    raise e
  File "/usr/local/lib/python3.10/site-packages/trulens_eval/feedback/feedback.py", line 467, in run
    input_combinations = list(
  File "/usr/local/lib/python3.10/site-packages/trulens_eval/feedback/feedback.py", line 671, in extract_selection
    raise RuntimeError(
RuntimeError: Could not locate app.query.rets.source_nodes[:].node.text in app/record.

```

@piotrm0 can you take a look at this async issue?

Got similar error, let me know if you need more info to reproduce it!

I am able to reproduce and I think the issue is that when we capture the return of method that includes a generator (this is how llama index does streaming), the jsonizer iterates over that generator. The intended recipient of that generator then gets an already exhausted generator. Will work on a fix.

@yasergsi @dm-shr can you try running on the latest 0.24.0 release?

@yasergsi @dm-shr Please let us know if this fix is working for you?

Closing for now, please let us know if this is fixed

@yuvneshtruera is this issue fixed?

@yasergsi could you please help me here

hi @piotrm0 @joshreini1 , below is my code

```
import logging
import pickle
import sys
import os
import warnings
import trulens_eval
from dotenv import load_dotenv
from llama_index.core.query_engine import RetrieverQueryEngine
from llama_index.core import VectorStoreIndex, StorageContext, load_index_from_storage, PromptTemplate
from llama_index.retrievers.bm25 import BM25Retriever
from custom_retriever import HybridRetriever
from utils import create_template
from trulens_eval.feedback.provider import OpenAI
from trulens_eval import Feedback, Tru
import numpy as np
from trulens_eval.app import App
from trulens_eval import TruLlama
from llama_index.llms.openai import OpenAI

logging.basicConfig(stream=sys.stdout, level=logging.INFO)
logger = logging.getLogger(__name__)

warnings.filterwarnings('ignore', category=FutureWarning, message="The default dtype for empty Series will be 'object'")
PERSIST_DIR = "./storage"
load_dotenv()
openai_api_key = os.getenv("OPENAI_API_KEY")
tru = Tru()
llm = OpenAI(model="gpt-3.5-turbo", temperature=0.4)
provider = trulens_eval.feedback.provider.OpenAI(openai_api_key)

# settings = {
#     "model": "gpt-3.5-turbo",
#     "temperature": 0.7,
#     "max_tokens": 500,
#     "top_p": 1,
#     "frequency_penalty": 0,
#     "presence_penalty": 0,
# }



with open("doc.pickle", "rb") as f:
    nodes = pickle.load(f)

if not os.path.exists(PERSIST_DIR):
    # load the documents and create the index
    storage_context = StorageContext.from_defaults()
    storage_context.docstore.add_documents(nodes)

    index = VectorStoreIndex(nodes, storage_context=storage_context)
    # store it for later
    index.storage_context.persist(persist_dir=PERSIST_DIR)
else:
    # load the existing index
    storage_context = StorageContext.from_defaults(persist_dir=PERSIST_DIR)
    index = load_index_from_storage(storage_context)

# retrieve the top 10 most similar nodes using embeddings
vector_retriever = index.as_retriever(similarity_top_k=4)

# retrieve the top 10 most similar nodes using bm25
bm25_retriever = BM25Retriever.from_defaults(nodes=nodes, similarity_top_k=4)

hybrid_retriever = HybridRetriever(vector_retriever, bm25_retriever)

query_engine = RetrieverQueryEngine.from_args(
    retriever=hybrid_retriever,
    llm=llm
)

query_engine.update_prompts(
    {'response_synthesizer:text_qa_template': create_template()}
)

# response = query_engine.query("What is jas?")
# print(response)
#
context = App.select_context(query_engine)

# Define a groundedness feedback function
f_groundedness = (
    Feedback(provider.groundedness_measure_with_cot_reasons)
    .on(context.collect())  # collect context chunks into a list
    .on_output()
)

# Question/answer relevance between overall question and answer.
f_answer_relevance = (
    Feedback(provider.relevance)
    .on_input_output()
)
# Question/statement relevance between question and each context chunk.
f_context_relevance = (
    Feedback(provider.context_relevance_with_cot_reasons)
    .on_input()
    .on(context)
    .aggregate(np.mean)
)

tru_query_engine_recorder = TruLlama(query_engine,
    app_id='LlamaIndex_App1',
    feedbacks=[f_groundedness, f_answer_relevance, f_context_relevance])


with tru_query_engine_recorder as recording:
    response = query_engine.query("jas date")


rec = recording.records
records, feedback = tru.get_records_and_feedback(app_ids=["LlamaIndex_App1"])
tru.get_leaderboard(app_ids=["LlamaIndex_App1"])

records.head()

`tru.run_dashboard()`
```
the output column is empty and groundedness column is missing from the records Dataframe.