Repository: truera/trulens
Issue: [BUG] Error removing trivial statements: invalid syntax (<string>, line 1). Proceeding with all statements.  && UserWarning: No supporting evidence provided. Returning score only.   warnings.warn(
Root cause: Validation Gap

**Bug Description**
When I run the evaluation it gave me both warning messages and error messages.

Warning: UserWarning: No supporting evidence provided. Returning score only.
Error:  Error removing trivial statements: invalid syntax (<string>, line 1). Proceeding with all statements.

**To Reproduce**
I was following the example to set up llama-index query engine using OpenAI API and Azure AI Search

**Relevant Logs/Tracebacks**
from langchain_community.callbacks.openai_info import OpenAICallbackHandler
You can use the langchain cli to **automatically** upgrade many imports. Please see documentation here <https://python.langchain.com/v0.2/docs/versions/v0_2/>
  from langchain.callbacks.openai_info import OpenAICallbackHandler
🦑 TruSession initialized with db url sqlite:////Users/xxx/xxx/trulens.db .
🛑 Secret keys may be written to the database. See the `database_redact_keys` option of `TruSession` to prevent this.
✅ In Groundedness, input source will be set to __record__.app.query.rets.source_nodes[:].node.text.collect() .
✅ In Groundedness, input statement will be set to __record__.main_output or `Select.RecordOutput` .
✅ In Answer Relevance, input prompt will be set to __record__.main_input or `Select.RecordInput` .
✅ In Answer Relevance, input response will be set to __record__.main_output or `Select.RecordOutput` .
✅ In Context Relevance, input question will be set to __record__.main_input or `Select.RecordInput` .
✅ In Context Relevance, input context will be set to __record__.app.query.rets.source_nodes[:].node.text .
Processing questions:   1%|█                                                                                                                                          | 1/128 [00:09<20:41,  9.77s/it]/opt/homebrew/Caskroom/miniforge/base/envs/env_name/lib/python3.11/site-packages/trulens/feedback/llm_provider.py:289: UserWarning: No supporting evidence provided. Returning score only.
  warnings.warn(
Processing questions:   2%|██▏                                                                                                                                        | 2/128 [00:24<26:20, 12.54s/it]Error removing trivial statements: invalid syntax (<string>, line 1). Proceeding with all statements.
Processing questions:   2%|███▎                                                                                                                                       | 3/128 [00:28<18:00,  8.64s/it]Error removing trivial statements: invalid syntax (<string>, line 1). Proceeding with all statements.
Processing questions:   3%|████▎                                                                                                                                      | 4/128 [00:40<20:32,  9.94s/it]Error removing trivial statements: invalid syntax (<string>, line 1). Proceeding with all statements.
Processing questions:   4%|█████▍                                                                                                                                     | 5/128 [00:51<21:44, 10.61s/it]Error removing trivial statements: invalid syntax (<string>, line 1). Proceeding with all statements.
Processing questions:  10%|██████████████                                                                                                                            | 13/128 [02:20<21:02, 10.98s/it]Error removing trivial statements: invalid syntax (<string>, line 1). Proceeding with all statements.
Error removing trivial statements: invalid syntax (<string>, line 1). Proceeding with all statements.
Processing questions:  11%|███████████████                                                                                                                           | 14/128 [02:32<21:55, 11.54s/it]Error removing trivial statements: invalid syntax (<string>, line 1). Proceeding with all statements.
Processing questions:  12%|████████████████▏                                                                                                                         | 15/128 [02:41<19:52, 10.55s/it]Error removing trivial statements: invalid syntax (<string>, line 1). Proceeding with all statements.

**Environment:**

- OS: MacOS
- Python Version 3.11
- TruLens version 1.0.1


Hey @zhongshuai-cao - what LLM are you using for your feedback provider? The failures here are a result of the LLM provider failing to follow instructions

Also can you share the text you're evaluating that leads to this error? This will help us debug

> Hey @zhongshuai-cao - what LLM are you using for your feedback provider? The failures here are a result of the LLM provider failing to follow instructions
> 
> Also can you share the text you're evaluating that leads to this error? This will help us debug

Thanks for following up @sfc-gh-jreini  !

The provider is OpenAI, 
```python
from trulens.providers.openai import OpenAI as OpenAIProvider
provider = OpenAIProvider(
    model_engine=LLM_MODEL,
)
```

The evaluation is directly from the tutorial:
```python
    # Define a groundedness feedback function
    f_groundedness = (
        Feedback(
            provider.groundedness_measure_with_cot_reasons, name="Groundedness"
        )
        .on(context.collect())  # collect context chunks into a list
        .on_output()
    )

    # Question/answer relevance between overall question and answer.
    f_answer_relevance = Feedback(
        provider.relevance_with_cot_reasons, name="Answer Relevance"
    ).on_input_output()
    # Question/statement relevance between question and each context chunk.
    f_context_relevance = (
        Feedback(
            provider.context_relevance_with_cot_reasons, name="Context Relevance"
        )
        .on_input()
        .on(context)
        .aggregate(np.mean)
    )

```



I am not sure which one question or which evaluation part causing the error, but in the dashboard I can see some cells are missing[1]. And for these records I didn't find any feedback results[2]. But I did have input and output showed the query answered.
[1]
<img width="586" alt="image" src="https://github.com/user-attachments/assets/41782ad5-3655-4732-b5ac-b0a9cc436067">

[2]
<img width="353" alt="image" src="https://github.com/user-attachments/assets/41c00264-3e1e-46ea-8f10-56c002cfd3bb">



Hey @zhongshuai-cao this looks like a case where it's taking some time to evaluate the feedback results. There is an expected delay between when you log the app's execution and the feedback computation being completed. We handle the feedback function computation async/in the background by default, so you may not see it if you look at the dashboard before it's ready.

If you want to wait until the feedback is completed, you can use the below:

```
with tru_recorder as recording:
    prompt_input = "What is the typical weather in Ireland?"
    llm_response = llm_chain.invoke(prompt_input)
    
record = recording.get()

# Wait for the result futures to be completed and print them.
for feedback, result in record.wait_for_feedback_results().items():
    print(feedback.name, result.result)
```


> Hey @zhongshuai-cao this looks like a case where it's taking some time to evaluate the feedback results. There is an expected delay between when you log the app's execution and the feedback computation being completed. We handle the feedback function computation async/in the background by default, so you may not see it if you look at the dashboard before it's ready.
> 
> If you want to wait until the feedback is completed, you can use the below:
> 
> ```
> with tru_recorder as recording:
>     prompt_input = "What is the typical weather in Ireland?"
>     llm_response = llm_chain.invoke(prompt_input)
>     
> record = recording.get()
> 
> # Wait for the result futures to be completed and print them.
> for feedback, result in record.wait_for_feedback_results().items():
>     print(feedback.name, result.result)
> ```

@sfc-gh-jreini Thanks for the info! I am doing batch evaluation of a list of questions, and usling the llama_index

```python

context = TruLlama.select_context(query_engine)

tru_query_engine_recorder = TruLlama(
        query_engine,
        app_id=app_name,
        app_version="test",
        feedbacks=[f_groundedness, f_answer_relevance, f_context_relevance],
        tru=session
)

def run_evals(eval_questions, tru_recorder, query_engine):
    with tru_recorder as recording:
        for question in tqdm(eval_questions, desc="Processing questions"):
            query_engine.query(question)

run_evals(eval_questions, tru_query_engine_recorder, query_engine)
```

Would you advice if and how I can add the wait_for_feedback_results in my llama_index workflow? And if there exist a batch async method?

You can add `wait_for_feedback_results` into your workflow like below. This will execute/log the query and response, but wait until the feedback results are computed before moving on to the next record.

```python
context = TruLlama.select_context(query_engine)

tru_query_engine_recorder = TruLlama(
        query_engine,
        app_id=app_name,
        app_version="test",
        feedbacks=[f_groundedness, f_answer_relevance, f_context_relevance]
)

def run_evals(eval_questions, tru_recorder, query_engine):
    for question in tqdm(eval_questions, desc="Processing questions"):
        with tru_recorder as recording:
            query_engine.query(question)
        record = recording.get()
        for feedback, result in record.wait_for_feedback_results().items():
            print(feedback.name, result.result)

run_evals(eval_questions, tru_query_engine_recorder, query_engine)
```

To accomplish a similar effect for your case, you can also set `feedback_mode` for your recorder to be `with_app`. This will block the app from returning the response until the feedback functions finish computing:

```python
from trulens.core import FeedbackMode
tru_query_engine_recorder = TruLlama(
        query_engine,
        app_id=app_name,
        app_version="test",
        feedbacks=[f_groundedness, f_answer_relevance, f_context_relevance],
        feedback_mode=FeedbackMode.WITH_APP,
)
```

This being said, unless there is a particular reason you need to know *when* the evaluations are finished computing and since you are doing batch evaluation, I would not recommend either of these approaches and simply go with the default approach.

We also do support batch ingestion. Rather than ingesting one by one (default), this will buffer the records and ingest them in batches to the database. Depending on the database, this can lead to big benefits in ingestion time. You can use by adding `record_ingest_mode` to the recorder:

```python
from trulens.core import FeedbackMode
from trulens.core.schema.app import RecordIngestMode

tru_query_engine_recorder = TruLlama(
        query_engine,
        app_id=app_name,
        app_version="test",
        feedbacks=[f_groundedness, f_answer_relevance, f_context_relevance],
        feedback_mode=FeedbackMode.WITH_APP,
        record_ingest_mode=RecordIngestMode.BUFFERED,
)
```

Closing this for now, @zhongshuai-cao please comment to reopen this again and if you need more help.