Repository: truera/trulens
Issue: Dashboard shows "No events found for app_name: default-RAG, app_ids: None"
Root cause: Interface Contract Mismatch

Hi, I’m using Trulens with a custom RAG app and LiteLLM provider. When I open the dashboard, it shows:No events found for app_name: SpikingBrain-RAG, app_ids: None
```
  import os
  import json
  import requests
  import numpy as np
  from trulens.core import TruSession, Feedback
  from trulens.apps.app import TruApp
  from trulens.providers.litellm import LiteLLM
  
                       
  os.environ["TRULENS_OTEL_TRACING"] = "1"
  
  provider = LiteLLM(
      model="spikingbrain-7b",
      api_base="http://localhost:7860/v1",
      api_key="jxq-spikingbrain-remote-key"
  )
  
  f_groundedness = (
      Feedback(
          provider.groundedness_measure_with_cot_reasons_consider_answerability,
          name="Groundedness",
      )
      .on_context(collect_list=True)
      .on_output()
      .on_input()
  )
  
  f_answer_relevance = (
      Feedback(provider.relevance_with_cot_reasons, name="Answer Relevance")
      .on_input()
      .on_output()
  )
  
  f_context_relevance = (
      Feedback(provider.context_relevance_with_cot_reasons, name="Context Relevance")
      .on_input()
      .on_context(collect_list=False)
      .aggregate(np.mean)
  )
  
  class RAGApp:
      def __call__(self, main_input):
          question = main_input["question"]
          context = main_input["context"]
  
          headers = {
              "Content-Type": "application/json",
              "Authorization": "Bearer jxq-spikingbrain-remote-key"
          }
          payload = {
              "model": "spikingbrain-7b",
              "messages": [
                  {"role": "system", "content": "请根据上下文准确回答问题，只输出简洁、直接答案。"},
                  {"role": "user", "content": f"上下文：\n{context}\n\n问题：{question}"}
              ],
              "temperature": 0.0
          }
  
          try:
              resp = requests.post(
                  "http://localhost:7860/v1/chat/completions",
                  headers=headers,
                  json=payload,
                  timeout=30
              ).json()
              return resp["choices"][0]["message"]["content"]
          except Exception as e:
              return f"模型错误: {e}"
  
  rag_app = RAGApp()
  
  RETRIEVAL_FILE = "/meta/SpikingBrain-7B/knowledge_rag/retrieval_mmarco_results_1000.json"
  with open(RETRIEVAL_FILE, "r", encoding="utf-8") as f:
      retrieval_data = json.load(f)[:5]
  
  print(f"已加载测试样本 {len(retrieval_data)} 条\n")
  
  session = TruSession()
  
  app = TruApp(
      app=rag_app,
      app_id="SpikingBrain-RAG",
      feedbacks=[f_groundedness, f_answer_relevance, f_context_relevance],
      selectors_nocheck=True
  )
  
  for i, sample in enumerate(retrieval_data, 1):
      question = sample["question"]
      context_list = sample["retrieved_docs"]
      context = "\n\n".join([d["text"] for d in context_list])
  
      print(f"▶ 正在评估 ({i}/{len(retrieval_data)}): {question}")
  
      # 运行 RAG App 并收集追踪数据
      app.run({"question": question, "context": context})
  
  print("\n🎉 批量评测完成！")
  
  leaderboard = session.get_leaderboard(app_ids=[app.app_id])
  print("📊 Leaderboard:\n")
  print(leaderboard)
  
  from trulens.dashboard import run_dashboard
  print("\n🚀 启动 TruLens Dashboard：")
  run_dashboard(session)
```

<!-- Greeting -->
Hi @WiFi2333! I'm [Dosu](https://go.dosu.dev/dosubot) and I’m helping the trulens team.

<!-- Answer -->
The dashboard showing "No events found" for your custom RAG app can be caused by a few common issues:

- If your leaderboard prints records but the dashboard is empty, it's likely a filtering or schema mismatch. Make sure both your notebook and dashboard are running the same version of TruLens to avoid missing attributes like `app_version` in the Record class, which can break dashboard filtering logic ([source](https://github.com/truera/trulens/issues/2235)).
- Try launching both the notebook and dashboard in the same environment, and avoid specifying the `app_id` filter when calling `get_leaderboard()` and `get_records_and_feedback()`—this workaround has resolved similar issues ([source](https://github.com/truera/trulens/issues/1884)).
- Always use the same `TruSession` instance for both running evaluations and launching the dashboard, so events are accessible ([source](https://github.com/truera/trulens/discussions/1927)).
- Reset the session database before running evaluations to avoid stale or missing event data ([source](https://github.com/truera/trulens/discussions/1927)).
- Ensure feedback computation is complete before accessing leaderboard or launching the dashboard; if feedback is still processing, records may not appear ([source](https://github.com/truera/trulens/issues/1884)).
- If you see errors about missing `app_name` in the dashboard, check that your `app_json` includes the `app_name` key or defaults correctly ([source](https://github.com/truera/trulens/discussions/1978)).

If you check these areas and still see no events, try a minimal working example with a built-in provider to isolate whether the issue is with your custom setup or the environment.


<!-- Dosu Comment Footer -->
*To reply, just mention [@dosu](https://go.dosu.dev/dosubot).*

---
<sup>How did I do? [Good](https://app.dosu.dev/response-feedback/6fc163f1-573e-4829-8d5a-e85a09b1349b?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/6fc163f1-573e-4829-8d5a-e85a09b1349b?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/6fc163f1-573e-4829-8d5a-e85a09b1349b?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/6fc163f1-573e-4829-8d5a-e85a09b1349b?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/6fc163f1-573e-4829-8d5a-e85a09b1349b?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/6fc163f1-573e-4829-8d5a-e85a09b1349b?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/6fc163f1-573e-4829-8d5a-e85a09b1349b?feedback_type=other)</sup>&nbsp;&nbsp;[![Chat with Dosu](https://dosu.dev/dosu-chat-badge.svg)](https://app.dosu.dev/e16d1378-a540-46b6-bd0e-816c493d2aee/ask?utm_source=github)&nbsp;[![Join Discord](https://img.shields.io/badge/join-5865F2?logo=discord&logoColor=white&label=)](https://go.dosu.dev/discord-bot)&nbsp;[![Share on X](https://img.shields.io/badge/X-share-black)](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/truera/trulens/issues/2291)

Hi @WiFi2333 - there is a "refresh data" button on the left pane of the dashboard. Once you've opened it, can you wait some time and then press the "refresh data" button and let me know if you are able to then see your app/records?

Hello, thank you for your reply. I can now see the records, but my records do not include metrics such as groundedness, answer_relevance, or context_relevance. Could this be due to an issue with how my feedback functions are defined, or is it possible that certain models cannot be used for feedback evaluation?

<img width="2549" height="1412" alt="Image" src="https://github.com/user-attachments/assets/e9c75396-cf4e-4632-8227-64a8ed5f4706" />

Hi @WiFi2333 - it's very possible the model you are using is not able to successfully complete the evals.

Can you try a direct call of the feedback function to see if this is the issue? Use the code below, filling in your example input/output.

```python
 f_answer_relevance("example input", "example output")
```

Hello, I added the following code, but it keeps getting stuck at the line: result = f_groundedness(contexts, output, input_question)
```
contexts = ["我的母语是德语。"]

output = "德语。"
input_question = "你的母语是什么"

print("准备调用 groundedness")
print("contexts:", contexts)
print("output:", output)
print("input_question:", input_question)
result = f_groundedness(contexts, output, input_question)
print("groundedness 调用结束")
```

<img width="924" height="177" alt="Image" src="https://github.com/user-attachments/assets/8d4a981c-2961-4159-a8c8-e547f223b3c9" />

Thanks. Groundedness is a harder task than answer relevance, can you try a stronger model for that feedback function? Seems like the model is failing for this one

Thanks for your help! Can I use a Chinese-finetuned Llama-2-7B (hosted on my server with an OpenAI-compatible API) as the evaluation model?

It doesn't seem to work—it's now producing the following error.
```
Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM.Info: If you need to debug this error, use `litellm.set_verbose=True'.

LiteLLMEndpoint request failed <class 'litellm.exceptions.AuthenticationError'>=AuthenticationError: OpenAIException - Traceback (most recent call last):
  File "/root/anaconda3/envs/spikingBrain/lib/python3.11/site-packages/litellm/llms/openai.py", line 678, in completion
    raise e
  File "/root/anaconda3/envs/spikingBrain/lib/python3.11/site-packages/litellm/llms/openai.py", line 614, in completion
    openai_client = openai(
                    ^^^^^^^
  File "/root/anaconda3/envs/spikingBrain/lib/python3.11/site-packages/openai/_client.py", line 135, in __init__
    raise openaiError(
openai.openaiError: The api_key client option must be set either by passing api_key to the client or by setting the OPENAI_API_KEY environment variable
. Retries remaining=3.
Traceback (most recent call last):
  File "/meta/SpikingBrain-7B/knowledge_rag/default-rag.py", line 102, in <module>
    result = f_answer_relevance(input_question, output)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/spikingBrain/lib/python3.11/site-packages/trulens/core/feedback/feedback.py", line 521, in __call__
    return self.imp(*args, **valid_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/spikingBrain/lib/python3.11/site-packages/trulens/feedback/llm_provider.py", line 801, in relevance_with_cot_reasons
    return self.generate_score_and_reasons(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/spikingBrain/lib/python3.11/site-packages/trulens/feedback/llm_provider.py", line 313, in generate_score_and_reasons
    response = self.endpoint.run_in_pace(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/spikingBrain/lib/python3.11/site-packages/trulens/core/feedback/endpoint.py", line 340, in run_in_pace
    raise RuntimeError(
RuntimeError: Endpoint LiteLLMEndpoint request failed 1 time(s):
        AuthenticationError: OpenAIException - Traceback (most recent call last):
  File "/root/anaconda3/envs/spikingBrain/lib/python3.11/site-packages/litellm/llms/openai.py", line 678, in completion
    raise e
  File "/root/anaconda3/envs/spikingBrain/lib/python3.11/site-packages/litellm/llms/openai.py", line 614, in completion
    openai_client = openai(
                    ^^^^^^^
  File "/root/anaconda3/envs/spikingBrain/lib/python3.11/site-packages/openai/_client.py", line 135, in __init__
    raise openaiError(
openai.openaiError: The api_key client option must be set either by passing api_key to the client or by setting the OPENAI_API_KEY environment variable

```

I printed the provider and found that the configuration failed, and my server cannot connect to the internet.
`tru_class_info=trulens.providers.litellm.provider.LiteLLM endpoint=Endpoint(LiteLLMEndpoint) model_engine='gpt-3.5-turbo' completion_args={}
`


 Hello @sfc-gh-jreini , I followed your suggestion from another thread（https://github.com/truera/trulens/issues/1268#issuecomment-2220772634） and attempted to deploy Ollama offline on my server. However, when I ran the code again, and I’m still encountering the following error.I'm using TruLens version 2.4.2 and litellm version 1.39.5—could this be related to the version?
```
(spikingBrain) root@ecs-jxq:/meta/SpikingBrain-7B/knowledge_rag# python test.py
/root/anaconda3/envs/spikingBrain/lib/python3.11/site-packages/munch/__init__.py:24: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
  import pkg_resources
{'completion_args': {},
 'endpoint': Endpoint(LiteLLMEndpoint),
 'model_engine': 'ollama/llama3:8b-instruct-fp16',
 'tru_class_info': trulens.providers.litellm.provider.LiteLLM}

Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM.Info: If you need to debug this error, use `litellm.set_verbose=True'.


Provider List: https://docs.litellm.ai/docs/providers

LiteLLMEndpoint request failed <class 'litellm.exceptions.APIConnectionError'>=<class 'trulens.feedback.output_schemas.ChainOfThoughtResponse'> cannot be parametrized because it does not inherit from typing.Generic. Retries remaining=3.

Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM.Info: If you need to debug this error, use `litellm.set_verbose=True'.


Provider List: https://docs.litellm.ai/docs/providers

LiteLLMEndpoint request failed <class 'litellm.exceptions.APIConnectionError'>=<class 'trulens.feedback.output_schemas.ChainOfThoughtResponse'> cannot be parametrized because it does not inherit from typing.Generic. Retries remaining=2.

Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM.Info: If you need to debug this error, use `litellm.set_verbose=True'.


Provider List: https://docs.litellm.ai/docs/providers

LiteLLMEndpoint request failed <class 'litellm.exceptions.APIConnectionError'>=<class 'trulens.feedback.output_schemas.ChainOfThoughtResponse'> cannot be parametrized because it does not inherit from typing.Generic. Retries remaining=1.

Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM.Info: If you need to debug this error, use `litellm.set_verbose=True'.


Provider List: https://docs.litellm.ai/docs/providers

LiteLLMEndpoint request failed <class 'litellm.exceptions.APIConnectionError'>=<class 'trulens.feedback.output_schemas.ChainOfThoughtResponse'> cannot be parametrized because it does not inherit from typing.Generic. Retries remaining=0.
Traceback (most recent call last):
  File "/meta/SpikingBrain-7B/knowledge_rag/test.py", line 14, in <module>
    result = provider.relevance_with_cot_reasons(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/spikingBrain/lib/python3.11/site-packages/trulens/feedback/llm_provider.py", line 801, in relevance_with_cot_reasons
    return self.generate_score_and_reasons(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/spikingBrain/lib/python3.11/site-packages/trulens/feedback/llm_provider.py", line 313, in generate_score_and_reasons
    response = self.endpoint.run_in_pace(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/spikingBrain/lib/python3.11/site-packages/trulens/core/feedback/endpoint.py", line 340, in run_in_pace
    raise RuntimeError(
RuntimeError: Endpoint LiteLLMEndpoint request failed 4 time(s):
        <class 'trulens.feedback.output_schemas.ChainOfThoughtResponse'> cannot be parametrized because it does not inherit from typing.Generic
        <class 'trulens.feedback.output_schemas.ChainOfThoughtResponse'> cannot be parametrized because it does not inherit from typing.Generic
        <class 'trulens.feedback.output_schemas.ChainOfThoughtResponse'> cannot be parametrized because it does not inherit from typing.Generic
        <class 'trulens.feedback.output_schemas.ChainOfThoughtResponse'> cannot be parametrized because it does not inherit from typing.Generic
```

I would not recommend using a llama-2 model for evaluation.

I'm currently using ollama/llama3:8b-instruct-fp16.
```{'completion_args': {},
 'endpoint': Endpoint(LiteLLMEndpoint),
 'model_engine': 'ollama/llama3:8b-instruct-fp16',
 'tru_class_info': trulens.providers.litellm.provider.LiteLLM}
```

 @sfc-gh-jreini  Could you please provide an example that can perform evaluation in an offline environment?

- [ ] Hey @WiFi2333 - in an earlier message you had your offline model successfully working on the direct relevance call. The issue seems to be that while the model is sufficient for the easier relevance task it was not for the harder groundedness task.

While we support connecting any model through the available providers, we cannot guarantee success for arbitrary LLMs.