Repository: truera/trulens
Issue: [BUG]The result of langchain quickstart missed groundedness,answer relevance and context relevancein the dashboard
Root cause: Validation Gap

**Bug Description**
I am referring to the LangChain Starter on the official TruLens website to obtain groundedness, answer relevance, and context relevance.My code is identical except that the vector database and ChatModel are different from those in the manual.But when I run session. get_leaderboard (), the result I get is only latency and count_cost.My rag_chain is running well.

All code runs in Jupyter Notebook.

**To Reproduce**
My vector database is qdrant, and the chatmodel is ChatHunyuan

**Expected behavior**
Groundedness, answer relevance, and context relevance should be displayed on the dashboard

**Relevant Logs/Tracebacks**
My code will be displayed below.
```python
from trulens.apps.langchain import TruChain
from trulens.core import TruSession

session = TruSession()
session.reset_database()

import bs4
from langchain import hub
from langchain.chat_models import ChatHunyuan
from langchain.document_loaders import WebBaseLoader
from langchain.schema import StrOutputParser
from langchain_core.runnables import RunnablePassthrough
import os

loader = WebBaseLoader(
    web_paths=("https://lilianweng.github.io/posts/2023-06-23-agent/",),
    bs_kwargs=dict(
        parse_only=bs4.SoupStrainer(
            class_=("post-content", "post-title", "post-header")
        )
    ),
)
docs = loader.load()

from qdrant_client import QdrantClient
from langchain_community.embeddings import HuggingFaceBgeEmbeddings
from langchain_text_splitters import RecursiveCharacterTextSplitter
from qdrant_client.models import VectorParams, Distance, PointStruct

embeddings = HuggingFaceBgeEmbeddings(model_name=r'D:\work\pythonProject\reinforce_interview\RAG_reinforce\embeddings_model\huggingface\models--BAAI--bge-small-en\snapshots\2275a7bdee235e9b4f01fa73aa60d3311983cfea')

text_splitter = RecursiveCharacterTextSplitter()
documents = text_splitter.split_documents(docs)

vectorstore = QdrantClient('http://x.x.x.x:xxxx')

from langchain_community.vectorstores import Qdrant

qdrant = Qdrant(client=vectorstore, collection_name='db3', embeddings=embeddings, content_payload_key="text")
retriever = qdrant.as_retriever()
prompt = hub.pull("rlm/rag-prompt")
llm = ChatHunyuan(
    hunyuan_app_id=os.environ['HUNYUAN_APP_ID'],
    hunyuan_secret_id=os.environ['HUNYUAN_SECRETID'],
    hunyuan_secret_key=os.environ['HUNYUAN_SECRETKEY'],
    model="hunyuan-lite",
)

def format_docs(docs):
    return "\n\n".join(doc.page_content for doc in docs)

rag_chain = (
    {"context": retriever | format_docs, "question": RunnablePassthrough()}
    | prompt
    | llm
    | StrOutputParser()
)

import numpy as np
from trulens.core import Feedback
from trulens.providers.langchain import Langchain

provider = Langchain(chain=llm)

context = TruChain.select_context(rag_chain)

f_groundedness = (
    Feedback(
        provider.groundedness_measure_with_cot_reasons, name="groundedness"
    )
    .on(context.collect())  # collect context chunks into a list
    .on_output()
)

f_answer_relevance = Feedback(
    provider.relevance_with_cot_reasons, name="Answer Relevance"
).on_input_output()

f_context_relevance = (
    Feedback(
        provider.context_relevance_with_cot_reasons, name="Context Relevance"
    )
    .on_input()
    .on(context)
    .aggregate(np.mean)
)

tru_recorder = TruChain(
    rag_chain,
    app_name="ChatApplication",
    app_version="Chain1",
    feedbacks=[f_answer_relevance, f_context_relevance, f_groundedness]
)

with tru_recorder as recording:
    llm_response = rag_chain.invoke("What is Task Decomposition?")

session.get_leaderboard()
```
After executing all the feedback functions, I obtained this result
```
✅ In groundedness, input source will be set to __record__.app.first.steps__.context.first.invoke.rets[:].page_content.collect() .
✅ In groundedness, input statement will be set to __record__.main_output or `Select.RecordOutput` .
✅ In Answer Relevance, input prompt will be set to __record__.main_input or `Select.RecordInput` .
✅ In Answer Relevance, input response will be set to __record__.main_output or `Select.RecordOutput` .
✅ In Context Relevance, input question will be set to __record__.main_input or `Select.RecordInput` .
✅ In Context Relevance, input context will be set to __record__.app.first.steps__.context.first.invoke.rets[:].page_content .
```
When running rag_chain.invoice ("What is Task Decomposition?"), I got this result
```
calling <function RunnableSequence.invoke at 0x000001F9E06D5B20> with ({
  context: VectorStoreRetriever(tags=['Qdrant', 'HuggingFaceBgeEmbeddings'], vectorstore=<langchain_community.vectorstores.qdrant.Qdrant object at 0x000001F9A53AA030>, search_kwargs={})
           | RunnableLambda(format_docs),
  question: RunnablePassthrough()
}
| ChatPromptTemplate(input_variables=['context', 'question'], input_types={}, partial_variables={}, metadata={'lc_hub_owner': 'rlm', 'lc_hub_repo': 'rag-prompt', 'lc_hub_commit_hash': '50442af133e61576e74536c6556cefe1fac147cad032f4377b60c436e6cdcb6e'}, messages=[HumanMessagePromptTemplate(prompt=PromptTemplate(input_variables=['context', 'question'], input_types={}, partial_variables={}, template="You are an assistant for question-answering tasks. Use the following pieces of retrieved context to answer the question. If you don't know the answer, just say that you don't know. Use three sentences maximum and keep the answer concise.\nQuestion: {question} \nContext: {context} \nAnswer:"), additional_kwargs={})])
| ChatHunyuan(hunyuan_app_id=1305233366, hunyuan_secret_id='AKIDWr4EFA1e3FKMJBE9sNfkbHs2yhCZxABK', hunyuan_secret_key=SecretStr('**********'), model_kwargs={})
| StrOutputParser(), 'What is Task Decomposition?')
calling <function RunnableParallel.invoke at 0x000001F9E06D7740> with ({
  context: VectorStoreRetriever(tags=['Qdrant', 'HuggingFaceBgeEmbeddings'], vectorstore=<langchain_community.vectorstores.qdrant.Qdrant object at 0x000001F9A53AA030>, search_kwargs={})
           | RunnableLambda(format_docs),
  question: RunnablePassthrough()
}, 'What is Task Decomposition?', {'tags': [], 'metadata': {}, 'callbacks': <langchain_core.callbacks.manager.CallbackManager object at 0x000001F9C659EAE0>, 'recursion_limit': 25, 'configurable': {}})
calling <function RunnableSequence.invoke at 0x000001F9E06D5B20> with (VectorStoreRetriever(tags=['Qdrant', 'HuggingFaceBgeEmbeddings'], vectorstore=<langchain_community.vectorstores.qdrant.Qdrant object at 0x000001F9A53AA030>, search_kwargs={})
| RunnableLambda(format_docs), 'What is Task Decomposition?', {'tags': [], 'metadata': {}, 'callbacks': <langchain_core.callbacks.manager.CallbackManager object at 0x000001F9C659EAB0>, 'recursion_limit': 25, 'configurable': {}})
calling <function BaseRetriever.invoke at 0x000001F9E0767380> with (VectorStoreRetriever(tags=['Qdrant', 'HuggingFaceBgeEmbeddings'], vectorstore=<langchain_community.vectorstores.qdrant.Qdrant object at 0x000001F9A53AA030>, search_kwargs={}), 'What is Task Decomposition?', {'tags': [], 'metadata': {}, 'callbacks': <langchain_core.callbacks.manager.CallbackManager object at 0x000001F9C659F4D0>, 'recursion_limit': 25, 'configurable': {}})
calling <function RunnablePassthrough.invoke at 0x000001F9E070D9E0> with (RunnablePassthrough(), 'What is Task Decomposition?', {'tags': [], 'metadata': {}, 'callbacks': <langchain_core.callbacks.manager.CallbackManager object at 0x000001F9C659FA10>, 'recursion_limit': 25, 'configurable': {}})
calling <function VectorStoreRetriever._get_relevant_documents at 0x000001F9E0785120> with (VectorStoreRetriever(tags=['Qdrant', 'HuggingFaceBgeEmbeddings'], vectorstore=<langchain_community.vectorstores.qdrant.Qdrant object at 0x000001F9A53AA030>, search_kwargs={}), 'What is Task Decomposition?')
calling <function BasePromptTemplate.invoke at 0x000001F9FBBE53A0> with (ChatPromptTemplate(input_variables=['context', 'question'], input_types={}, partial_variables={}, metadata={'lc_hub_owner': 'rlm', 'lc_hub_repo': 'rag-prompt', 'lc_hub_commit_hash': '50442af133e61576e74536c6556cefe1fac147cad032f4377b60c436e6cdcb6e'}, messages=[HumanMessagePromptTemplate(prompt=PromptTemplate(input_variables=['context', 'question'], input_types={}, partial_variables={}, template="You are an assistant for question-answering tasks. Use the following pieces of retrieved context to answer the question. If you don't know the answer, just say that you don't know. Use three sentences maximum and keep the answer concise.\nQuestion: {question} \nContext: {context} \nAnswer:"), additional_kwargs={})]), {'context': 'Fig. 1. Overview of a LLM-powered autonomous agent system.\nComponent One: Planning#\nA complicated task usually involves many steps. An agent needs to know what they are and plan ahead.\nTask Decomposition#\nChain of thought (CoT; Wei et al. 2022) has become a standard prompting technique for enhancing model performance on complex tasks. The model is instructed to “think step by step” to utilize more test-time computation to decompose hard tasks into smaller and simpler steps. CoT transforms big tasks into multiple manageable tasks and shed lights into an interpretation of the model’s thinking process.\nTree of Thoughts (Yao et al. 2023) extends CoT by exploring multiple reasoning possibilities at each step. It first decomposes the problem into multiple thought steps and generates multiple thoughts per step, creating a tree structure. The search process can be BFS (breadth-first search) or DFS (depth-first search) with each state evaluated by a classifier (via a prompt) or majority vote.\nTask decomposition can be done (1) by LLM with simple prompting like "Steps for XYZ.\\n1.", "What are the subgoals for achieving XYZ?", (2) by using task-specific instructions; e.g. "Write a story outline." for writing a novel, or (3) with human inputs.\nAnother quite distinct approach, LLM+P (Liu et al. 2023), involves relying on an external classical planner to do long-horizon planning. This approach utilizes the Planning Domain Definition Language (PDDL) as an intermediate interface to describe the planning problem. In this process, LLM (1) translates the problem into “Problem PDDL”, then (2) requests a classical planner to generate a PDDL plan based on an existing “Domain PDDL”, and finally (3) translates the PDDL plan back into natural language. Essentially, the planning step is outsourced to an external tool, assuming the availability of domain-specific PDDL and a suitable planner which is common in certain robotic setups but not in many other domains.\nSelf-Reflection#\nSelf-reflection is a vital aspect that allows autonomous agents to improve iteratively by refining past action decisions and correcting previous mistakes. It plays a crucial role in real-world tasks where trial and error are inevitable.\nReAct (Yao et al. 2023) integrates reasoning and acting within LLM by extending the action space to be a combination of task-specific discrete actions and the language space. The former enables LLM to interact with the environment (e.g. use Wikipedia search API), while the latter prompting LLM to generate reasoning traces in natural language.\nThe ReAct prompt template incorporates explicit steps for LLM to think, roughly formatted as:\nThought: ...\nAction: ...\nObservation: ...\n... (Repeated many times)\n\nFig. 2.  Examples of reasoning trajectories for knowledge-intensive tasks (e.g. HotpotQA, FEVER) and decision-making tasks (e.g. AlfWorld Env, WebShop). (Image source: Yao et al. 2023).\nIn both experiments on knowledge-intensive tasks and decision-making tasks, ReAct works better than the Act-only baseline where Thought: … step is removed.\nReflexion (Shinn & Labash 2023) is a framework to equip agents with dynamic memory and self-reflection capabilities to improve reasoning skills. Reflexion has a standard RL setup, in which the reward model provides a simple binary reward and the action space follows the setup in ReAct where the task-specific action space is augmented with language to enable complex reasoning steps. After each action $a_t$, the agent computes a heuristic $h_t$ and optionally may decide to reset the environment to start a new trial depending on the self-reflection results.\n\nFig. 3. Illustration of the Reflexion framework. (Image source: Shinn & Labash, 2023)\nThe heuristic function determines when the trajectory is inefficient or contains hallucination and should be stopped. Inefficient planning refers to trajectories that take too long without success. Hallucination is defined as encountering a sequence of consecutive identical actions that lead to the same observation in the environment.\nSelf-reflection is created by showing two-shot examples to LLM and each example is a pair of (failed trajectory, ideal reflection for guiding future changes in the plan). Then reflections are added into the agent’s working memory, up to three, to be used as context for querying LLM.\n\nFig. 4. Experiments on AlfWorld Env and HotpotQA. Hallucination is a more common failure than inefficient planning in AlfWorld. (Image source: Shinn & Labash, 2023)\nChain of Hindsight (CoH; Liu et al. 2023) encourages the model to improve on its own outputs by explicitly presenting it with a sequence of past outputs, each annotated with feedback. Human feedback data is a collection of $D_h = \\{(x, y_i , r_i , z_i)\\}_{i=1}^n$, where $x$ is the prompt, each $y_i$ is a model completion, $r_i$ is the human rating of $y_i$, and $z_i$ is the corresponding human-provided hindsight feedback. Assume the feedback tuples are ranked by reward, $r_n \\geq r_{n-1} \\geq \\dots \\geq r_1$ The process is supervised fine-tuning where the data is a sequence in the form of $\\tau_h = (x, z_i, y_i, z_j, y_j, \\dots, z_n, y_n)$, where $\\leq i \\leq j \\leq n$. The model is finetuned to only predict $y_n$ where conditioned on the sequence prefix, such that the model can self-reflect to produce better output based on the feedback sequence. The model can optionally receive multiple rounds of instructions with human annotators at test time.\nTo avoid overfitting, CoH adds a regularization term to maximize the log-likelihood of the pre-training dataset. To avoid shortcutting and copying (because there are many common words in feedback sequences), they randomly mask 0% - 5% of past tokens during training.\nThe training dataset in their experiments is a combination of WebGPT comparisons, summarization from human feedback and human preference dataset.\n\nFig. 5. After fine-tuning with CoH, the model can follow instructions to produce outputs with incremental improvement in a sequence. (Image source: Liu et al. 2023)\nThe idea of CoH is to present a history of sequentially improved outputs  in context and train the model to take on the trend to produce better outputs. Algorithm Distillation (AD; Laskin et al. 2023) applies the same idea to cross-episode trajectories in reinforcement learning tasks, where an algorithm is encapsulated in a long history-conditioned policy. Considering that an agent interacts with the environment many times and in each episode the agent gets a little better, AD concatenates this learning history and feeds that into the model. Hence we should expect the next predicted action to lead to better performance than previous trials. The goal is to learn the process of RL instead of training a task-specific policy itself.\n\n(3) Task execution: Expert models execute on the specific tasks and log results.\nInstruction:\n\nWith the input and the inference results, the AI assistant needs to describe the process and results. The previous stages can be formed as - User Input: {{ User Input }}, Task Planning: {{ Tasks }}, Model Selection: {{ Model Assignment }}, Task Execution: {{ Predictions }}. You must first answer the user\'s request in a straightforward manner. Then describe the task process and show your analysis and model inference results to the user in the first person. If inference results contain a file path, must tell the user the complete file path.\n\n(4) Response generation: LLM receives the execution results and provides summarized results to users.\nTo put HuggingGPT into real world usage, a couple challenges need to solve: (1) Efficiency improvement is needed as both LLM inference rounds and interactions with other models slow down the process; (2) It relies on a long context window to communicate over complicated task content; (3) Stability improvement of LLM outputs and external model services.\nAPI-Bank (Li et al. 2023) is a benchmark for evaluating the performance of tool-augmented LLMs. It contains 53 commonly used API tools, a complete tool-augmented LLM workflow, and 264 annotated dialogues that involve 568 API calls. The selection of APIs is quite diverse, including search engines, calculator, calendar queries, smart home control, schedule management, health data management, account authentication workflow and more. Because there are a large number of APIs, LLM first has access to API search engine to find the right API to call and then uses the corresponding documentation to make a call.\n\nFig. 12. Pseudo code of how LLM makes an API call in API-Bank. (Image source: Li et al. 2023)\nIn the API-Bank workflow, LLMs need to make a couple of decisions and at each step we can evaluate how accurate that decision is. Decisions include:\n\nWhether an API call is needed.\nIdentify the right API to call: if not good enough, LLMs need to iteratively modify the API inputs (e.g. deciding search keywords for Search Engine API).\nResponse based on the API results: the model can choose to refine and call again if results are not satisfied.\n\nThis benchmark evaluates the agent’s tool use capabilities at three levels:\n\nLevel-1 evaluates the ability to call the API. Given an API’s description, the model needs to determine whether to call a given API, call it correctly, and respond properly to API returns.\nLevel-2 examines the ability to retrieve the API. The model needs to search for possible APIs that may solve the user’s requirement and learn how to use them by reading documentation.\nLevel-3 assesses the ability to plan API beyond retrieve and call. Given unclear user requests (e.g. schedule group meetings, book flight/hotel/restaurant for a trip), the model may have to conduct multiple API calls to solve it.\n\nCase Studies#\nScientific Discovery Agent#\nChemCrow (Bran et al. 2023) is a domain-specific example in which LLM is augmented with 13 expert-designed tools to accomplish tasks across organic synthesis, drug discovery, and materials design. The workflow, implemented in LangChain, reflects what was previously described in the ReAct and MRKLs and combines CoT reasoning with tools relevant to the tasks:\n\nThe LLM is provided with a list of tool names, descriptions of their utility, and details about the expected input/output.\nIt is then instructed to answer a user-given prompt using the tools provided when necessary. The instruction suggests the model to follow the ReAct format - Thought, Action, Action Input, Observation.\n\nYou should only respond in JSON format as described below\nResponse Format:\n{\n    "thoughts": {\n        "text": "thought",\n        "reasoning": "reasoning",\n        "plan": "- short bulleted\\n- list that conveys\\n- long-term plan",\n        "criticism": "constructive self-criticism",\n        "speak": "thoughts summary to say to user"\n    },\n    "command": {\n        "name": "command name",\n        "args": {\n            "arg name": "value"\n        }\n    }\n}\nEnsure the response can be parsed by Python json.loads\nGPT-Engineer is another project to create a whole repository of code given a task specified in natural language. The GPT-Engineer is instructed to think over a list of smaller components to build and ask for user input to clarify questions as needed.\nHere are a sample conversation for task clarification sent to OpenAI ChatCompletion endpoint used by GPT-Engineer. The user inputs are wrapped in {{user input text}}.\n[\n  {\n    "role": "system",\n    "content": "You will read instructions and not carry them out, only seek to clarify them.\\nSpecifically you will first summarise a list of super short bullets of areas that need clarification.\\nThen you will pick one clarifying question, and wait for an answer from the user.\\n"\n  },\n  {\n    "role": "user",\n    "content": "We are writing {{a Super Mario game in python. MVC components split in separate files. Keyboard control.}}\\n"\n  },\n  {\n    "role": "assistant",\n    "content": "Summary of areas that need clarification:\\n1. Specifics of the Super Mario game (e.g. level design, characters, gameplay mechanics)\\n2. Details about the MVC components (e.g. which components are in each file)\\n3. Keyboard control implementation (e.g. which keys to use, how to handle input)\\n\\nClarifying question:\\nCan you provide more details about the Super Mario game, such as level design, characters, and gameplay mechanics?"\n  },\n  {\n    "role": "user",\n    "content": "{{There are 10 levels in total. The main character is a plumber named Mario, who can walk and jump. It is a classical platform game just like Super Mario. The main character moves from left to right, trying to get to the destination, where there are many obstacles and attacks from enemies in the process.}}\\n\\nIs anything else unclear? If yes, only answer in the form:\\n{remaining unclear areas} remaining questions.\\n{Next question}\\nIf everything is sufficiently clear, only answer \\"Nothing more to clarify.\\"."\n  },\n  {\n    "role": "assistant",\n    "content": "Remaining unclear areas: 2 remaining questions.\\nCan you provide more information about how the MVC components are split into separate files?"\n  },\n  {\n    "role": "user",\n    "content": "{{Make your own assumptions and state them explicitly before starting}}"\n  }\n]\nThen after these clarification, the agent moved into the code writing mode with a different system message.\nSystem message:', 'question': 'What is Task Decomposition?'}, {'tags': [], 'metadata': {}, 'callbacks': <langchain_core.callbacks.manager.CallbackManager object at 0x000001F9C659F320>, 'recursion_limit': 25, 'configurable': {}})
calling <function BaseChatModel.invoke at 0x000001F9FAADB740> with (ChatHunyuan(hunyuan_app_id=1305233366, hunyuan_secret_id='AKIDWr4EFA1e3FKMJBE9sNfkbHs2yhCZxABK', hunyuan_secret_key=SecretStr('**********'), model_kwargs={}), ChatPromptValue(messages=[HumanMessage(content='You are an assistant for question-answering tasks. Use the following pieces of retrieved context to answer the question. If you don\'t know the answer, just say that you don\'t know. Use three sentences maximum and keep the answer concise.\nQuestion: What is Task Decomposition? \nContext: Fig. 1. Overview of a LLM-powered autonomous agent system.\nComponent One: Planning#\nA complicated task usually involves many steps. An agent needs to know what they are and plan ahead.\nTask Decomposition#\nChain of thought (CoT; Wei et al. 2022) has become a standard prompting technique for enhancing model performance on complex tasks. The model is instructed to “think step by step” to utilize more test-time computation to decompose hard tasks into smaller and simpler steps. CoT transforms big tasks into multiple manageable tasks and shed lights into an interpretation of the model’s thinking process.\nTree of Thoughts (Yao et al. 2023) extends CoT by exploring multiple reasoning possibilities at each step. It first decomposes the problem into multiple thought steps and generates multiple thoughts per step, creating a tree structure. The search process can be BFS (breadth-first search) or DFS (depth-first search) with each state evaluated by a classifier (via a prompt) or majority vote.\nTask decomposition can be done (1) by LLM with simple prompting like "Steps for XYZ.\\n1.", "What are the subgoals for achieving XYZ?", (2) by using task-specific instructions; e.g. "Write a story outline." for writing a novel, or (3) with human inputs.\nAnother quite distinct approach, LLM+P (Liu et al. 2023), involves relying on an external classical planner to do long-horizon planning. This approach utilizes the Planning Domain Definition Language (PDDL) as an intermediate interface to describe the planning problem. In this process, LLM (1) translates the problem into “Problem PDDL”, then (2) requests a classical planner to generate a PDDL plan based on an existing “Domain PDDL”, and finally (3) translates the PDDL plan back into natural language. Essentially, the planning step is outsourced to an external tool, assuming the availability of domain-specific PDDL and a suitable planner which is common in certain robotic setups but not in many other domains.\nSelf-Reflection#\nSelf-reflection is a vital aspect that allows autonomous agents to improve iteratively by refining past action decisions and correcting previous mistakes. It plays a crucial role in real-world tasks where trial and error are inevitable.\nReAct (Yao et al. 2023) integrates reasoning and acting within LLM by extending the action space to be a combination of task-specific discrete actions and the language space. The former enables LLM to interact with the environment (e.g. use Wikipedia search API), while the latter prompting LLM to generate reasoning traces in natural language.\nThe ReAct prompt template incorporates explicit steps for LLM to think, roughly formatted as:\nThought: ...\nAction: ...\nObservation: ...\n... (Repeated many times)\n\nFig. 2.  Examples of reasoning trajectories for knowledge-intensive tasks (e.g. HotpotQA, FEVER) and decision-making tasks (e.g. AlfWorld Env, WebShop). (Image source: Yao et al. 2023).\nIn both experiments on knowledge-intensive tasks and decision-making tasks, ReAct works better than the Act-only baseline where Thought: … step is removed.\nReflexion (Shinn & Labash 2023) is a framework to equip agents with dynamic memory and self-reflection capabilities to improve reasoning skills. Reflexion has a standard RL setup, in which the reward model provides a simple binary reward and the action space follows the setup in ReAct where the task-specific action space is augmented with language to enable complex reasoning steps. After each action $a_t$, the agent computes a heuristic $h_t$ and optionally may decide to reset the environment to start a new trial depending on the self-reflection results.\n\nFig. 3. Illustration of the Reflexion framework. (Image source: Shinn & Labash, 2023)\nThe heuristic function determines when the trajectory is inefficient or contains hallucination and should be stopped. Inefficient planning refers to trajectories that take too long without success. Hallucination is defined as encountering a sequence of consecutive identical actions that lead to the same observation in the environment.\nSelf-reflection is created by showing two-shot examples to LLM and each example is a pair of (failed trajectory, ideal reflection for guiding future changes in the plan). Then reflections are added into the agent’s working memory, up to three, to be used as context for querying LLM.\n\nFig. 4. Experiments on AlfWorld Env and HotpotQA. Hallucination is a more common failure than inefficient planning in AlfWorld. (Image source: Shinn & Labash, 2023)\nChain of Hindsight (CoH; Liu et al. 2023) encourages the model to improve on its own outputs by explicitly presenting it with a sequence of past outputs, each annotated with feedback. Human feedback data is a collection of $D_h = \\{(x, y_i , r_i , z_i)\\}_{i=1}^n$, where $x$ is the prompt, each $y_i$ is a model completion, $r_i$ is the human rating of $y_i$, and $z_i$ is the corresponding human-provided hindsight feedback. Assume the feedback tuples are ranked by reward, $r_n \\geq r_{n-1} \\geq \\dots \\geq r_1$ The process is supervised fine-tuning where the data is a sequence in the form of $\\tau_h = (x, z_i, y_i, z_j, y_j, \\dots, z_n, y_n)$, where $\\leq i \\leq j \\leq n$. The model is finetuned to only predict $y_n$ where conditioned on the sequence prefix, such that the model can self-reflect to produce better output based on the feedback sequence. The model can optionally receive multiple rounds of instructions with human annotators at test time.\nTo avoid overfitting, CoH adds a regularization term to maximize the log-likelihood of the pre-training dataset. To avoid shortcutting and copying (because there are many common words in feedback sequences), they randomly mask 0% - 5% of past tokens during training.\nThe training dataset in their experiments is a combination of WebGPT comparisons, summarization from human feedback and human preference dataset.\n\nFig. 5. After fine-tuning with CoH, the model can follow instructions to produce outputs with incremental improvement in a sequence. (Image source: Liu et al. 2023)\nThe idea of CoH is to present a history of sequentially improved outputs  in context and train the model to take on the trend to produce better outputs. Algorithm Distillation (AD; Laskin et al. 2023) applies the same idea to cross-episode trajectories in reinforcement learning tasks, where an algorithm is encapsulated in a long history-conditioned policy. Considering that an agent interacts with the environment many times and in each episode the agent gets a little better, AD concatenates this learning history and feeds that into the model. Hence we should expect the next predicted action to lead to better performance than previous trials. The goal is to learn the process of RL instead of training a task-specific policy itself.\n\n(3) Task execution: Expert models execute on the specific tasks and log results.\nInstruction:\n\nWith the input and the inference results, the AI assistant needs to describe the process and results. The previous stages can be formed as - User Input: {{ User Input }}, Task Planning: {{ Tasks }}, Model Selection: {{ Model Assignment }}, Task Execution: {{ Predictions }}. You must first answer the user\'s request in a straightforward manner. Then describe the task process and show your analysis and model inference results to the user in the first person. If inference results contain a file path, must tell the user the complete file path.\n\n(4) Response generation: LLM receives the execution results and provides summarized results to users.\nTo put HuggingGPT into real world usage, a couple challenges need to solve: (1) Efficiency improvement is needed as both LLM inference rounds and interactions with other models slow down the process; (2) It relies on a long context window to communicate over complicated task content; (3) Stability improvement of LLM outputs and external model services.\nAPI-Bank (Li et al. 2023) is a benchmark for evaluating the performance of tool-augmented LLMs. It contains 53 commonly used API tools, a complete tool-augmented LLM workflow, and 264 annotated dialogues that involve 568 API calls. The selection of APIs is quite diverse, including search engines, calculator, calendar queries, smart home control, schedule management, health data management, account authentication workflow and more. Because there are a large number of APIs, LLM first has access to API search engine to find the right API to call and then uses the corresponding documentation to make a call.\n\nFig. 12. Pseudo code of how LLM makes an API call in API-Bank. (Image source: Li et al. 2023)\nIn the API-Bank workflow, LLMs need to make a couple of decisions and at each step we can evaluate how accurate that decision is. Decisions include:\n\nWhether an API call is needed.\nIdentify the right API to call: if not good enough, LLMs need to iteratively modify the API inputs (e.g. deciding search keywords for Search Engine API).\nResponse based on the API results: the model can choose to refine and call again if results are not satisfied.\n\nThis benchmark evaluates the agent’s tool use capabilities at three levels:\n\nLevel-1 evaluates the ability to call the API. Given an API’s description, the model needs to determine whether to call a given API, call it correctly, and respond properly to API returns.\nLevel-2 examines the ability to retrieve the API. The model needs to search for possible APIs that may solve the user’s requirement and learn how to use them by reading documentation.\nLevel-3 assesses the ability to plan API beyond retrieve and call. Given unclear user requests (e.g. schedule group meetings, book flight/hotel/restaurant for a trip), the model may have to conduct multiple API calls to solve it.\n\nCase Studies#\nScientific Discovery Agent#\nChemCrow (Bran et al. 2023) is a domain-specific example in which LLM is augmented with 13 expert-designed tools to accomplish tasks across organic synthesis, drug discovery, and materials design. The workflow, implemented in LangChain, reflects what was previously described in the ReAct and MRKLs and combines CoT reasoning with tools relevant to the tasks:\n\nThe LLM is provided with a list of tool names, descriptions of their utility, and details about the expected input/output.\nIt is then instructed to answer a user-given prompt using the tools provided when necessary. The instruction suggests the model to follow the ReAct format - Thought, Action, Action Input, Observation.\n\nYou should only respond in JSON format as described below\nResponse Format:\n{\n    "thoughts": {\n        "text": "thought",\n        "reasoning": "reasoning",\n        "plan": "- short bulleted\\n- list that conveys\\n- long-term plan",\n        "criticism": "constructive self-criticism",\n        "speak": "thoughts summary to say to user"\n    },\n    "command": {\n        "name": "command name",\n        "args": {\n            "arg name": "value"\n        }\n    }\n}\nEnsure the response can be parsed by Python json.loads\nGPT-Engineer is another project to create a whole repository of code given a task specified in natural language. The GPT-Engineer is instructed to think over a list of smaller components to build and ask for user input to clarify questions as needed.\nHere are a sample conversation for task clarification sent to OpenAI ChatCompletion endpoint used by GPT-Engineer. The user inputs are wrapped in {{user input text}}.\n[\n  {\n    "role": "system",\n    "content": "You will read instructions and not carry them out, only seek to clarify them.\\nSpecifically you will first summarise a list of super short bullets of areas that need clarification.\\nThen you will pick one clarifying question, and wait for an answer from the user.\\n"\n  },\n  {\n    "role": "user",\n    "content": "We are writing {{a Super Mario game in python. MVC components split in separate files. Keyboard control.}}\\n"\n  },\n  {\n    "role": "assistant",\n    "content": "Summary of areas that need clarification:\\n1. Specifics of the Super Mario game (e.g. level design, characters, gameplay mechanics)\\n2. Details about the MVC components (e.g. which components are in each file)\\n3. Keyboard control implementation (e.g. which keys to use, how to handle input)\\n\\nClarifying question:\\nCan you provide more details about the Super Mario game, such as level design, characters, and gameplay mechanics?"\n  },\n  {\n    "role": "user",\n    "content": "{{There are 10 levels in total. The main character is a plumber named Mario, who can walk and jump. It is a classical platform game just like Super Mario. The main character moves from left to right, trying to get to the destination, where there are many obstacles and attacks from enemies in the process.}}\\n\\nIs anything else unclear? If yes, only answer in the form:\\n{remaining unclear areas} remaining questions.\\n{Next question}\\nIf everything is sufficiently clear, only answer \\"Nothing more to clarify.\\"."\n  },\n  {\n    "role": "assistant",\n    "content": "Remaining unclear areas: 2 remaining questions.\\nCan you provide more information about how the MVC components are split into separate files?"\n  },\n  {\n    "role": "user",\n    "content": "{{Make your own assumptions and state them explicitly before starting}}"\n  }\n]\nThen after these clarification, the agent moved into the code writing mode with a different system message.\nSystem message: \nAnswer:', additional_kwargs={}, response_metadata={})]), {'tags': [], 'metadata': {}, 'callbacks': <langchain_core.callbacks.manager.CallbackManager object at 0x000001F9C578FBC0>, 'recursion_limit': 25, 'configurable': {}})
calling <function BaseOutputParser.invoke at 0x000001F9FBB8B1A0> with (StrOutputParser(), AIMessage(content='{\n    "thoughts": {\n        "text": "The user is providing more detailed information about the Super Mario game, including the total number of levels, the main character, and the basic gameplay mechanics. However, there are still some aspects that need clarification. The main area of uncertainty is how the MVC components are split into separate files, and the user is asking for more specific details.",\n        "reasoning": "The user is providing detailed context for the Super Mario game, but there are still some unresolved questions about the implementation details. It is crucial to address these不明确的地方 to ensure that the game development can proceed smoothly.",\n        "Plan": "The remaining unclear areas are: 2 remaining questions about the MVC components being split into separate files and making assumptions. Next question is to clarify these points.",\n        "Criticism": "The user should provide more specific details about how the MVC components are split into separate files, as this is critical for implementing the game。",\n        "speak": "Your current description covers most of the details, but I still need more specifics about how the MVC components are split into separate files to ensure proper implementation."\n    },\n    "command": {\n        "name": "ask_user_for_additional_details",\n        "args": {"question": "Can you provide more detail about how the MVC components are split into separate files?"}\n    }\n}', additional_kwargs={}, response_metadata={'token_usage': {'PromptTokens': 3010, 'CompletionTokens': 279, 'TotalTokens': 3289}}, id='f896b425-d578-4da3-9aa5-c2aced625a63'), {'tags': [], 'metadata': {}, 'callbacks': <langchain_core.callbacks.manager.CallbackManager object at 0x000001F9C659EB70>, 'recursion_limit': 25, 'configurable': {}})
```


In the end, I only obtained this result.

![Image](https://github.com/user-attachments/assets/58b16ec4-a776-4416-8d81-83de1791f1f4)


**Environment:**

- OS: Windows
- Python Version==3.12
- TruLens version==1.3.2
- trulens-apps-langchain==1.3.2
- tencentcloud-sdk-python==3.0.1311
- transformers==4.48.1
- qdrant-client==1.13.2
- langchain==0.3.15
- langchain-community==0.3.15



I'm very sorry for not copying the information completely. The complete information is as follows.


calling <function RunnableSequence.invoke at 0x000001F9E06D5B20> with ({
  context: VectorStoreRetriever(tags=['Qdrant', 'HuggingFaceBgeEmbeddings'], vectorstore=<langchain_community.vectorstores.qdrant.Qdrant object at 0x000001F9A53AA030>, search_kwargs={})
           | RunnableLambda(format_docs),
  question: RunnablePassthrough()
}
| ChatPromptTemplate(input_variables=['context', 'question'], input_types={}, partial_variables={}, metadata={'lc_hub_owner': 'rlm', 'lc_hub_repo': 'rag-prompt', 'lc_hub_commit_hash': '50442af133e61576e74536c6556cefe1fac147cad032f4377b60c436e6cdcb6e'}, messages=[HumanMessagePromptTemplate(prompt=PromptTemplate(input_variables=['context', 'question'], input_types={}, partial_variables={}, template="You are an assistant for question-answering tasks. Use the following pieces of retrieved context to answer the question. If you don't know the answer, just say that you don't know. Use three sentences maximum and keep the answer concise.\nQuestion: {question} \nContext: {context} \nAnswer:"), additional_kwargs={})])
| ChatHunyuan(hunyuan_app_id=1305233366, hunyuan_secret_id='AKIDWr4EFA1e3FKMJBE9sNfkbHs2yhCZxABK', hunyuan_secret_key=SecretStr('**********'), model_kwargs={})
| StrOutputParser(), 'What is Task Decomposition?')
calling <function RunnableParallel.invoke at 0x000001F9E06D7740> with ({
  context: VectorStoreRetriever(tags=['Qdrant', 'HuggingFaceBgeEmbeddings'], vectorstore=<langchain_community.vectorstores.qdrant.Qdrant object at 0x000001F9A53AA030>, search_kwargs={})
           | RunnableLambda(format_docs),
  question: RunnablePassthrough()
}, 'What is Task Decomposition?', {'tags': [], 'metadata': {}, 'callbacks': <langchain_core.callbacks.manager.CallbackManager object at 0x000001F9C659EAE0>, 'recursion_limit': 25, 'configurable': {}})
calling <function RunnableSequence.invoke at 0x000001F9E06D5B20> with (VectorStoreRetriever(tags=['Qdrant', 'HuggingFaceBgeEmbeddings'], vectorstore=<langchain_community.vectorstores.qdrant.Qdrant object at 0x000001F9A53AA030>, search_kwargs={})
| RunnableLambda(format_docs), 'What is Task Decomposition?', {'tags': [], 'metadata': {}, 'callbacks': <langchain_core.callbacks.manager.CallbackManager object at 0x000001F9C659EAB0>, 'recursion_limit': 25, 'configurable': {}})
calling <function BaseRetriever.invoke at 0x000001F9E0767380> with (VectorStoreRetriever(tags=['Qdrant', 'HuggingFaceBgeEmbeddings'], vectorstore=<langchain_community.vectorstores.qdrant.Qdrant object at 0x000001F9A53AA030>, search_kwargs={}), 'What is Task Decomposition?', {'tags': [], 'metadata': {}, 'callbacks': <langchain_core.callbacks.manager.CallbackManager object at 0x000001F9C659F4D0>, 'recursion_limit': 25, 'configurable': {}})
calling <function RunnablePassthrough.invoke at 0x000001F9E070D9E0> with (RunnablePassthrough(), 'What is Task Decomposition?', {'tags': [], 'metadata': {}, 'callbacks': <langchain_core.callbacks.manager.CallbackManager object at 0x000001F9C659FA10>, 'recursion_limit': 25, 'configurable': {}})
calling <function VectorStoreRetriever._get_relevant_documents at 0x000001F9E0785120> with (VectorStoreRetriever(tags=['Qdrant', 'HuggingFaceBgeEmbeddings'], vectorstore=<langchain_community.vectorstores.qdrant.Qdrant object at 0x000001F9A53AA030>, search_kwargs={}), 'What is Task Decomposition?')
calling <function BasePromptTemplate.invoke at 0x000001F9FBBE53A0> with (ChatPromptTemplate(input_variables=['context', 'question'], input_types={}, partial_variables={}, metadata={'lc_hub_owner': 'rlm', 'lc_hub_repo': 'rag-prompt', 'lc_hub_commit_hash': '50442af133e61576e74536c6556cefe1fac147cad032f4377b60c436e6cdcb6e'}, messages=[HumanMessagePromptTemplate(prompt=PromptTemplate(input_variables=['context', 'question'], input_types={}, partial_variables={}, template="You are an assistant for question-answering tasks. Use the following pieces of retrieved context to answer the question. If you don't know the answer, just say that you don't know. Use three sentences maximum and keep the answer concise.\nQuestion: {question} \nContext: {context} \nAnswer:"), additional_kwargs={})]), {'context': 'Fig. 1. Overview of a LLM-powered autonomous agent system.\nComponent One: Planning#\nA complicated task usually involves many steps. An agent needs to know what they are and plan ahead.\nTask Decomposition#\nChain of thought (CoT; Wei et al. 2022) has become a standard prompting technique for enhancing model performance on complex tasks. The model is instructed to “think step by step” to utilize more test-time computation to decompose hard tasks into smaller and simpler steps. CoT transforms big tasks into multiple manageable tasks and shed lights into an interpretation of the model’s thinking process.\nTree of Thoughts (Yao et al. 2023) extends CoT by exploring multiple reasoning possibilities at each step. It first decomposes the problem into multiple thought steps and generates multiple thoughts per step, creating a tree structure. The search process can be BFS (breadth-first search) or DFS (depth-first search) with each state evaluated by a classifier (via a prompt) or majority vote.\nTask decomposition can be done (1) by LLM with simple prompting like "Steps for XYZ.\\n1.", "What are the subgoals for achieving XYZ?", (2) by using task-specific instructions; e.g. "Write a story outline." for writing a novel, or (3) with human inputs.\nAnother quite distinct approach, LLM+P (Liu et al. 2023), involves relying on an external classical planner to do long-horizon planning. This approach utilizes the Planning Domain Definition Language (PDDL) as an intermediate interface to describe the planning problem. In this process, LLM (1) translates the problem into “Problem PDDL”, then (2) requests a classical planner to generate a PDDL plan based on an existing “Domain PDDL”, and finally (3) translates the PDDL plan back into natural language. Essentially, the planning step is outsourced to an external tool, assuming the availability of domain-specific PDDL and a suitable planner which is common in certain robotic setups but not in many other domains.\nSelf-Reflection#\nSelf-reflection is a vital aspect that allows autonomous agents to improve iteratively by refining past action decisions and correcting previous mistakes. It plays a crucial role in real-world tasks where trial and error are inevitable.\nReAct (Yao et al. 2023) integrates reasoning and acting within LLM by extending the action space to be a combination of task-specific discrete actions and the language space. The former enables LLM to interact with the environment (e.g. use Wikipedia search API), while the latter prompting LLM to generate reasoning traces in natural language.\nThe ReAct prompt template incorporates explicit steps for LLM to think, roughly formatted as:\nThought: ...\nAction: ...\nObservation: ...\n... (Repeated many times)\n\nFig. 2.  Examples of reasoning trajectories for knowledge-intensive tasks (e.g. HotpotQA, FEVER) and decision-making tasks (e.g. AlfWorld Env, WebShop). (Image source: Yao et al. 2023).\nIn both experiments on knowledge-intensive tasks and decision-making tasks, ReAct works better than the Act-only baseline where Thought: … step is removed.\nReflexion (Shinn & Labash 2023) is a framework to equip agents with dynamic memory and self-reflection capabilities to improve reasoning skills. Reflexion has a standard RL setup, in which the reward model provides a simple binary reward and the action space follows the setup in ReAct where the task-specific action space is augmented with language to enable complex reasoning steps. After each action $a_t$, the agent computes a heuristic $h_t$ and optionally may decide to reset the environment to start a new trial depending on the self-reflection results.\n\nFig. 3. Illustration of the Reflexion framework. (Image source: Shinn & Labash, 2023)\nThe heuristic function determines when the trajectory is inefficient or contains hallucination and should be stopped. Inefficient planning refers to trajectories that take too long without success. Hallucination is defined as encountering a sequence of consecutive identical actions that lead to the same observation in the environment.\nSelf-reflection is created by showing two-shot examples to LLM and each example is a pair of (failed trajectory, ideal reflection for guiding future changes in the plan). Then reflections are added into the agent’s working memory, up to three, to be used as context for querying LLM.\n\nFig. 4. Experiments on AlfWorld Env and HotpotQA. Hallucination is a more common failure than inefficient planning in AlfWorld. (Image source: Shinn & Labash, 2023)\nChain of Hindsight (CoH; Liu et al. 2023) encourages the model to improve on its own outputs by explicitly presenting it with a sequence of past outputs, each annotated with feedback. Human feedback data is a collection of $D_h = \\{(x, y_i , r_i , z_i)\\}_{i=1}^n$, where $x$ is the prompt, each $y_i$ is a model completion, $r_i$ is the human rating of $y_i$, and $z_i$ is the corresponding human-provided hindsight feedback. Assume the feedback tuples are ranked by reward, $r_n \\geq r_{n-1} \\geq \\dots \\geq r_1$ The process is supervised fine-tuning where the data is a sequence in the form of $\\tau_h = (x, z_i, y_i, z_j, y_j, \\dots, z_n, y_n)$, where $\\leq i \\leq j \\leq n$. The model is finetuned to only predict $y_n$ where conditioned on the sequence prefix, such that the model can self-reflect to produce better output based on the feedback sequence. The model can optionally receive multiple rounds of instructions with human annotators at test time.\nTo avoid overfitting, CoH adds a regularization term to maximize the log-likelihood of the pre-training dataset. To avoid shortcutting and copying (because there are many common words in feedback sequences), they randomly mask 0% - 5% of past tokens during training.\nThe training dataset in their experiments is a combination of WebGPT comparisons, summarization from human feedback and human preference dataset.\n\nFig. 5. After fine-tuning with CoH, the model can follow instructions to produce outputs with incremental improvement in a sequence. (Image source: Liu et al. 2023)\nThe idea of CoH is to present a history of sequentially improved outputs  in context and train the model to take on the trend to produce better outputs. Algorithm Distillation (AD; Laskin et al. 2023) applies the same idea to cross-episode trajectories in reinforcement learning tasks, where an algorithm is encapsulated in a long history-conditioned policy. Considering that an agent interacts with the environment many times and in each episode the agent gets a little better, AD concatenates this learning history and feeds that into the model. Hence we should expect the next predicted action to lead to better performance than previous trials. The goal is to learn the process of RL instead of training a task-specific policy itself.\n\n(3) Task execution: Expert models execute on the specific tasks and log results.\nInstruction:\n\nWith the input and the inference results, the AI assistant needs to describe the process and results. The previous stages can be formed as - User Input: {{ User Input }}, Task Planning: {{ Tasks }}, Model Selection: {{ Model Assignment }}, Task Execution: {{ Predictions }}. You must first answer the user\'s request in a straightforward manner. Then describe the task process and show your analysis and model inference results to the user in the first person. If inference results contain a file path, must tell the user the complete file path.\n\n(4) Response generation: LLM receives the execution results and provides summarized results to users.\nTo put HuggingGPT into real world usage, a couple challenges need to solve: (1) Efficiency improvement is needed as both LLM inference rounds and interactions with other models slow down the process; (2) It relies on a long context window to communicate over complicated task content; (3) Stability improvement of LLM outputs and external model services.\nAPI-Bank (Li et al. 2023) is a benchmark for evaluating the performance of tool-augmented LLMs. It contains 53 commonly used API tools, a complete tool-augmented LLM workflow, and 264 annotated dialogues that involve 568 API calls. The selection of APIs is quite diverse, including search engines, calculator, calendar queries, smart home control, schedule management, health data management, account authentication workflow and more. Because there are a large number of APIs, LLM first has access to API search engine to find the right API to call and then uses the corresponding documentation to make a call.\n\nFig. 12. Pseudo code of how LLM makes an API call in API-Bank. (Image source: Li et al. 2023)\nIn the API-Bank workflow, LLMs need to make a couple of decisions and at each step we can evaluate how accurate that decision is. Decisions include:\n\nWhether an API call is needed.\nIdentify the right API to call: if not good enough, LLMs need to iteratively modify the API inputs (e.g. deciding search keywords for Search Engine API).\nResponse based on the API results: the model can choose to refine and call again if results are not satisfied.\n\nThis benchmark evaluates the agent’s tool use capabilities at three levels:\n\nLevel-1 evaluates the ability to call the API. Given an API’s description, the model needs to determine whether to call a given API, call it correctly, and respond properly to API returns.\nLevel-2 examines the ability to retrieve the API. The model needs to search for possible APIs that may solve the user’s requirement and learn how to use them by reading documentation.\nLevel-3 assesses the ability to plan API beyond retrieve and call. Given unclear user requests (e.g. schedule group meetings, book flight/hotel/restaurant for a trip), the model may have to conduct multiple API calls to solve it.\n\nCase Studies#\nScientific Discovery Agent#\nChemCrow (Bran et al. 2023) is a domain-specific example in which LLM is augmented with 13 expert-designed tools to accomplish tasks across organic synthesis, drug discovery, and materials design. The workflow, implemented in LangChain, reflects what was previously described in the ReAct and MRKLs and combines CoT reasoning with tools relevant to the tasks:\n\nThe LLM is provided with a list of tool names, descriptions of their utility, and details about the expected input/output.\nIt is then instructed to answer a user-given prompt using the tools provided when necessary. The instruction suggests the model to follow the ReAct format - Thought, Action, Action Input, Observation.\n\nYou should only respond in JSON format as described below\nResponse Format:\n{\n    "thoughts": {\n        "text": "thought",\n        "reasoning": "reasoning",\n        "plan": "- short bulleted\\n- list that conveys\\n- long-term plan",\n        "criticism": "constructive self-criticism",\n        "speak": "thoughts summary to say to user"\n    },\n    "command": {\n        "name": "command name",\n        "args": {\n            "arg name": "value"\n        }\n    }\n}\nEnsure the response can be parsed by Python json.loads\nGPT-Engineer is another project to create a whole repository of code given a task specified in natural language. The GPT-Engineer is instructed to think over a list of smaller components to build and ask for user input to clarify questions as needed.\nHere are a sample conversation for task clarification sent to OpenAI ChatCompletion endpoint used by GPT-Engineer. The user inputs are wrapped in {{user input text}}.\n[\n  {\n    "role": "system",\n    "content": "You will read instructions and not carry them out, only seek to clarify them.\\nSpecifically you will first summarise a list of super short bullets of areas that need clarification.\\nThen you will pick one clarifying question, and wait for an answer from the user.\\n"\n  },\n  {\n    "role": "user",\n    "content": "We are writing {{a Super Mario game in python. MVC components split in separate files. Keyboard control.}}\\n"\n  },\n  {\n    "role": "assistant",\n    "content": "Summary of areas that need clarification:\\n1. Specifics of the Super Mario game (e.g. level design, characters, gameplay mechanics)\\n2. Details about the MVC components (e.g. which components are in each file)\\n3. Keyboard control implementation (e.g. which keys to use, how to handle input)\\n\\nClarifying question:\\nCan you provide more details about the Super Mario game, such as level design, characters, and gameplay mechanics?"\n  },\n  {\n    "role": "user",\n    "content": "{{There are 10 levels in total. The main character is a plumber named Mario, who can walk and jump. It is a classical platform game just like Super Mario. The main character moves from left to right, trying to get to the destination, where there are many obstacles and attacks from enemies in the process.}}\\n\\nIs anything else unclear? If yes, only answer in the form:\\n{remaining unclear areas} remaining questions.\\n{Next question}\\nIf everything is sufficiently clear, only answer \\"Nothing more to clarify.\\"."\n  },\n  {\n    "role": "assistant",\n    "content": "Remaining unclear areas: 2 remaining questions.\\nCan you provide more information about how the MVC components are split into separate files?"\n  },\n  {\n    "role": "user",\n    "content": "{{Make your own assumptions and state them explicitly before starting}}"\n  }\n]\nThen after these clarification, the agent moved into the code writing mode with a different system message.\nSystem message:', 'question': 'What is Task Decomposition?'}, {'tags': [], 'metadata': {}, 'callbacks': <langchain_core.callbacks.manager.CallbackManager object at 0x000001F9C659F320>, 'recursion_limit': 25, 'configurable': {}})
calling <function BaseChatModel.invoke at 0x000001F9FAADB740> with (ChatHunyuan(hunyuan_app_id=1305233366, hunyuan_secret_id='AKIDWr4EFA1e3FKMJBE9sNfkbHs2yhCZxABK', hunyuan_secret_key=SecretStr('**********'), model_kwargs={}), ChatPromptValue(messages=[HumanMessage(content='You are an assistant for question-answering tasks. Use the following pieces of retrieved context to answer the question. If you don\'t know the answer, just say that you don\'t know. Use three sentences maximum and keep the answer concise.\nQuestion: What is Task Decomposition? \nContext: Fig. 1. Overview of a LLM-powered autonomous agent system.\nComponent One: Planning#\nA complicated task usually involves many steps. An agent needs to know what they are and plan ahead.\nTask Decomposition#\nChain of thought (CoT; Wei et al. 2022) has become a standard prompting technique for enhancing model performance on complex tasks. The model is instructed to “think step by step” to utilize more test-time computation to decompose hard tasks into smaller and simpler steps. CoT transforms big tasks into multiple manageable tasks and shed lights into an interpretation of the model’s thinking process.\nTree of Thoughts (Yao et al. 2023) extends CoT by exploring multiple reasoning possibilities at each step. It first decomposes the problem into multiple thought steps and generates multiple thoughts per step, creating a tree structure. The search process can be BFS (breadth-first search) or DFS (depth-first search) with each state evaluated by a classifier (via a prompt) or majority vote.\nTask decomposition can be done (1) by LLM with simple prompting like "Steps for XYZ.\\n1.", "What are the subgoals for achieving XYZ?", (2) by using task-specific instructions; e.g. "Write a story outline." for writing a novel, or (3) with human inputs.\nAnother quite distinct approach, LLM+P (Liu et al. 2023), involves relying on an external classical planner to do long-horizon planning. This approach utilizes the Planning Domain Definition Language (PDDL) as an intermediate interface to describe the planning problem. In this process, LLM (1) translates the problem into “Problem PDDL”, then (2) requests a classical planner to generate a PDDL plan based on an existing “Domain PDDL”, and finally (3) translates the PDDL plan back into natural language. Essentially, the planning step is outsourced to an external tool, assuming the availability of domain-specific PDDL and a suitable planner which is common in certain robotic setups but not in many other domains.\nSelf-Reflection#\nSelf-reflection is a vital aspect that allows autonomous agents to improve iteratively by refining past action decisions and correcting previous mistakes. It plays a crucial role in real-world tasks where trial and error are inevitable.\nReAct (Yao et al. 2023) integrates reasoning and acting within LLM by extending the action space to be a combination of task-specific discrete actions and the language space. The former enables LLM to interact with the environment (e.g. use Wikipedia search API), while the latter prompting LLM to generate reasoning traces in natural language.\nThe ReAct prompt template incorporates explicit steps for LLM to think, roughly formatted as:\nThought: ...\nAction: ...\nObservation: ...\n... (Repeated many times)\n\nFig. 2.  Examples of reasoning trajectories for knowledge-intensive tasks (e.g. HotpotQA, FEVER) and decision-making tasks (e.g. AlfWorld Env, WebShop). (Image source: Yao et al. 2023).\nIn both experiments on knowledge-intensive tasks and decision-making tasks, ReAct works better than the Act-only baseline where Thought: … step is removed.\nReflexion (Shinn & Labash 2023) is a framework to equip agents with dynamic memory and self-reflection capabilities to improve reasoning skills. Reflexion has a standard RL setup, in which the reward model provides a simple binary reward and the action space follows the setup in ReAct where the task-specific action space is augmented with language to enable complex reasoning steps. After each action $a_t$, the agent computes a heuristic $h_t$ and optionally may decide to reset the environment to start a new trial depending on the self-reflection results.\n\nFig. 3. Illustration of the Reflexion framework. (Image source: Shinn & Labash, 2023)\nThe heuristic function determines when the trajectory is inefficient or contains hallucination and should be stopped. Inefficient planning refers to trajectories that take too long without success. Hallucination is defined as encountering a sequence of consecutive identical actions that lead to the same observation in the environment.\nSelf-reflection is created by showing two-shot examples to LLM and each example is a pair of (failed trajectory, ideal reflection for guiding future changes in the plan). Then reflections are added into the agent’s working memory, up to three, to be used as context for querying LLM.\n\nFig. 4. Experiments on AlfWorld Env and HotpotQA. Hallucination is a more common failure than inefficient planning in AlfWorld. (Image source: Shinn & Labash, 2023)\nChain of Hindsight (CoH; Liu et al. 2023) encourages the model to improve on its own outputs by explicitly presenting it with a sequence of past outputs, each annotated with feedback. Human feedback data is a collection of $D_h = \\{(x, y_i , r_i , z_i)\\}_{i=1}^n$, where $x$ is the prompt, each $y_i$ is a model completion, $r_i$ is the human rating of $y_i$, and $z_i$ is the corresponding human-provided hindsight feedback. Assume the feedback tuples are ranked by reward, $r_n \\geq r_{n-1} \\geq \\dots \\geq r_1$ The process is supervised fine-tuning where the data is a sequence in the form of $\\tau_h = (x, z_i, y_i, z_j, y_j, \\dots, z_n, y_n)$, where $\\leq i \\leq j \\leq n$. The model is finetuned to only predict $y_n$ where conditioned on the sequence prefix, such that the model can self-reflect to produce better output based on the feedback sequence. The model can optionally receive multiple rounds of instructions with human annotators at test time.\nTo avoid overfitting, CoH adds a regularization term to maximize the log-likelihood of the pre-training dataset. To avoid shortcutting and copying (because there are many common words in feedback sequences), they randomly mask 0% - 5% of past tokens during training.\nThe training dataset in their experiments is a combination of WebGPT comparisons, summarization from human feedback and human preference dataset.\n\nFig. 5. After fine-tuning with CoH, the model can follow instructions to produce outputs with incremental improvement in a sequence. (Image source: Liu et al. 2023)\nThe idea of CoH is to present a history of sequentially improved outputs  in context and train the model to take on the trend to produce better outputs. Algorithm Distillation (AD; Laskin et al. 2023) applies the same idea to cross-episode trajectories in reinforcement learning tasks, where an algorithm is encapsulated in a long history-conditioned policy. Considering that an agent interacts with the environment many times and in each episode the agent gets a little better, AD concatenates this learning history and feeds that into the model. Hence we should expect the next predicted action to lead to better performance than previous trials. The goal is to learn the process of RL instead of training a task-specific policy itself.\n\n(3) Task execution: Expert models execute on the specific tasks and log results.\nInstruction:\n\nWith the input and the inference results, the AI assistant needs to describe the process and results. The previous stages can be formed as - User Input: {{ User Input }}, Task Planning: {{ Tasks }}, Model Selection: {{ Model Assignment }}, Task Execution: {{ Predictions }}. You must first answer the user\'s request in a straightforward manner. Then describe the task process and show your analysis and model inference results to the user in the first person. If inference results contain a file path, must tell the user the complete file path.\n\n(4) Response generation: LLM receives the execution results and provides summarized results to users.\nTo put HuggingGPT into real world usage, a couple challenges need to solve: (1) Efficiency improvement is needed as both LLM inference rounds and interactions with other models slow down the process; (2) It relies on a long context window to communicate over complicated task content; (3) Stability improvement of LLM outputs and external model services.\nAPI-Bank (Li et al. 2023) is a benchmark for evaluating the performance of tool-augmented LLMs. It contains 53 commonly used API tools, a complete tool-augmented LLM workflow, and 264 annotated dialogues that involve 568 API calls. The selection of APIs is quite diverse, including search engines, calculator, calendar queries, smart home control, schedule management, health data management, account authentication workflow and more. Because there are a large number of APIs, LLM first has access to API search engine to find the right API to call and then uses the corresponding documentation to make a call.\n\nFig. 12. Pseudo code of how LLM makes an API call in API-Bank. (Image source: Li et al. 2023)\nIn the API-Bank workflow, LLMs need to make a couple of decisions and at each step we can evaluate how accurate that decision is. Decisions include:\n\nWhether an API call is needed.\nIdentify the right API to call: if not good enough, LLMs need to iteratively modify the API inputs (e.g. deciding search keywords for Search Engine API).\nResponse based on the API results: the model can choose to refine and call again if results are not satisfied.\n\nThis benchmark evaluates the agent’s tool use capabilities at three levels:\n\nLevel-1 evaluates the ability to call the API. Given an API’s description, the model needs to determine whether to call a given API, call it correctly, and respond properly to API returns.\nLevel-2 examines the ability to retrieve the API. The model needs to search for possible APIs that may solve the user’s requirement and learn how to use them by reading documentation.\nLevel-3 assesses the ability to plan API beyond retrieve and call. Given unclear user requests (e.g. schedule group meetings, book flight/hotel/restaurant for a trip), the model may have to conduct multiple API calls to solve it.\n\nCase Studies#\nScientific Discovery Agent#\nChemCrow (Bran et al. 2023) is a domain-specific example in which LLM is augmented with 13 expert-designed tools to accomplish tasks across organic synthesis, drug discovery, and materials design. The workflow, implemented in LangChain, reflects what was previously described in the ReAct and MRKLs and combines CoT reasoning with tools relevant to the tasks:\n\nThe LLM is provided with a list of tool names, descriptions of their utility, and details about the expected input/output.\nIt is then instructed to answer a user-given prompt using the tools provided when necessary. The instruction suggests the model to follow the ReAct format - Thought, Action, Action Input, Observation.\n\nYou should only respond in JSON format as described below\nResponse Format:\n{\n    "thoughts": {\n        "text": "thought",\n        "reasoning": "reasoning",\n        "plan": "- short bulleted\\n- list that conveys\\n- long-term plan",\n        "criticism": "constructive self-criticism",\n        "speak": "thoughts summary to say to user"\n    },\n    "command": {\n        "name": "command name",\n        "args": {\n            "arg name": "value"\n        }\n    }\n}\nEnsure the response can be parsed by Python json.loads\nGPT-Engineer is another project to create a whole repository of code given a task specified in natural language. The GPT-Engineer is instructed to think over a list of smaller components to build and ask for user input to clarify questions as needed.\nHere are a sample conversation for task clarification sent to OpenAI ChatCompletion endpoint used by GPT-Engineer. The user inputs are wrapped in {{user input text}}.\n[\n  {\n    "role": "system",\n    "content": "You will read instructions and not carry them out, only seek to clarify them.\\nSpecifically you will first summarise a list of super short bullets of areas that need clarification.\\nThen you will pick one clarifying question, and wait for an answer from the user.\\n"\n  },\n  {\n    "role": "user",\n    "content": "We are writing {{a Super Mario game in python. MVC components split in separate files. Keyboard control.}}\\n"\n  },\n  {\n    "role": "assistant",\n    "content": "Summary of areas that need clarification:\\n1. Specifics of the Super Mario game (e.g. level design, characters, gameplay mechanics)\\n2. Details about the MVC components (e.g. which components are in each file)\\n3. Keyboard control implementation (e.g. which keys to use, how to handle input)\\n\\nClarifying question:\\nCan you provide more details about the Super Mario game, such as level design, characters, and gameplay mechanics?"\n  },\n  {\n    "role": "user",\n    "content": "{{There are 10 levels in total. The main character is a plumber named Mario, who can walk and jump. It is a classical platform game just like Super Mario. The main character moves from left to right, trying to get to the destination, where there are many obstacles and attacks from enemies in the process.}}\\n\\nIs anything else unclear? If yes, only answer in the form:\\n{remaining unclear areas} remaining questions.\\n{Next question}\\nIf everything is sufficiently clear, only answer \\"Nothing more to clarify.\\"."\n  },\n  {\n    "role": "assistant",\n    "content": "Remaining unclear areas: 2 remaining questions.\\nCan you provide more information about how the MVC components are split into separate files?"\n  },\n  {\n    "role": "user",\n    "content": "{{Make your own assumptions and state them explicitly before starting}}"\n  }\n]\nThen after these clarification, the agent moved into the code writing mode with a different system message.\nSystem message: \nAnswer:', additional_kwargs={}, response_metadata={})]), {'tags': [], 'metadata': {}, 'callbacks': <langchain_core.callbacks.manager.CallbackManager object at 0x000001F9C578FBC0>, 'recursion_limit': 25, 'configurable': {}})
calling <function BaseOutputParser.invoke at 0x000001F9FBB8B1A0> with (StrOutputParser(), AIMessage(content='{\n    "thoughts": {\n        "text": "The user is providing more detailed information about the Super Mario game, including the total number of levels, the main character, and the basic gameplay mechanics. However, there are still some aspects that need clarification. The main area of uncertainty is how the MVC components are split into separate files, and the user is asking for more specific details.",\n        "reasoning": "The user is providing detailed context for the Super Mario game, but there are still some unresolved questions about the implementation details. It is crucial to address these不明确的地方 to ensure that the game development can proceed smoothly.",\n        "Plan": "The remaining unclear areas are: 2 remaining questions about the MVC components being split into separate files and making assumptions. Next question is to clarify these points.",\n        "Criticism": "The user should provide more specific details about how the MVC components are split into separate files, as this is critical for implementing the game。",\n        "speak": "Your current description covers most of the details, but I still need more specifics about how the MVC components are split into separate files to ensure proper implementation."\n    },\n    "command": {\n        "name": "ask_user_for_additional_details",\n        "args": {"question": "Can you provide more detail about how the MVC components are split into separate files?"}\n    }\n}', additional_kwargs={}, response_metadata={'token_usage': {'PromptTokens': 3010, 'CompletionTokens': 279, 'TotalTokens': 3289}}, id='f896b425-d578-4da3-9aa5-c2aced625a63'), {'tags': [], 'metadata': {}, 'callbacks': <langchain_core.callbacks.manager.CallbackManager object at 0x000001F9C659EB70>, 'recursion_limit': 25, 'configurable': {}})

Sounds similar to https://github.com/truera/trulens/issues/1758

**Getting the same problem using LlamaIndex as well.** 

Initially, I was getting the feedback perfectly. 

![Image](https://github.com/user-attachments/assets/e9d957b6-fc6e-4b8d-8d09-29443a22494e)

But after running the app several times, It creates the problem.

![Image](https://github.com/user-attachments/assets/3dbc6b68-f8c9-428d-8739-2dde67ff288c)


Hi @wahidur028 - are you resetting the database between your "initial" results and later version?

The evaluation results should not be disappearing

Hi @sfc-gh-jreini, here is my code. I observed that after 100 runs, in 60%-70% of the cases, the system returns all three scores—context relevance, answer relevancy, and groundedness—correctly. In the remaining cases, it returns only two of these values, or in some instances, none. However, in all cases, it consistently returns latency and total cost.
```
def query_model(user_query):
    if query_engine is None:
        return "⚠️ Please upload documents first."
    
    max_attempts = 3
    attempt = 0
    
    while attempt < max_attempts:
        response = query_engine.query(user_query)
        
        # Initialize TruLens session
        session = TruSession()
        session.reset_database()
        provider = OpenAI(api_key=OPENAI_API_KEY)
        context = TruLlama.select_context(query_engine)
        
        f_groundedness = Feedback(provider.groundedness_measure_with_cot_reasons, name="Groundedness").on(context.collect()).on_output()
        f_answer_relevance = Feedback(provider.relevance_with_cot_reasons, name="Answer Relevance").on_input_output()
        f_context_relevance = Feedback(provider.context_relevance_with_cot_reasons, name="Context Relevance").on_input().on(context).aggregate(np.mean)
        
        tru_query_engine_recorder = TruLlama(
            query_engine,
            app_name="LlamaIndex_App",
            app_version="base",
            feedbacks=[f_groundedness, f_answer_relevance, f_context_relevance],
        )
        
        with tru_query_engine_recorder as recording:
            response = query_engine.query(user_query)
            print(response)

        last_record = recording.records[-1]
        get_feedback_result(last_record, "Context Relevance")
        
        leaderboard_df = session.get_leaderboard()
        print(leaderboard_df)``
```

Common causes of feedback failure can be:
-  LLM regressions: gpt-4o-mini is the default model for openai. Can you try a more powerful model, like chatgpt-4o-latest
- Irregularities in the data being evaluated (such as missing context, weird formatting) - can you look at the traces for these records missing feedback results to see if the retrieved context, llm response is what you expect to evaluate?

@wahidur028 - have you had a chance to look into the above?

@sfc-gh-jreini Sorry for the late response. I didn’t check the first one, but for the second case, I observed that the LLM responses are completely fine. However, the TruLens app sometimes fails to return the relevance or groundedness scores, although it consistently returns the latency and total cost.

The returned values are shown below:

`calling <function SentenceTransformerRerank._postprocess_nodes at 0x7f0e15d41620> with (SentenceTransformerRerank(callback_manager=<llama_index.core.callbacks.base.CallbackManager object at 0x7f0df917aff0>, model='BAAI/bge-reranker-large', top_n=3, device='cuda', keep_retrieval_score=False, trust_remote_code=False), [NodeWithScore(node=TextNode(id_='d94a6c49-7b40-410d-b3c6-0380303db4e7', embedding=None, metadata={'page_label': '1', 'file_name': 'Sumaya-Babylon-report_edit1.pdf', 'file_path': '/home/infonet/wahid/projects/LQRAG_App/data/Sumaya-Babylon-report_edit1.pdf', 'file_type': 'application/pdf', 'file_size': 446978, 'creation_date': '2025-03-21', 'last_modified_date': '2025-03-21'}, excluded_embed_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], excluded_llm_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], relationships={<NodeRelationship.SOURCE: '1'>: RelatedNodeInfo(node_id='b0cd32ef-47fb-4f03-9652-546f40e2216e', node_type='4', metadata={'page_label': '1', 'file_name': 'Sumaya-Babylon-report_edit1.pdf', 'file_path': '/home/infonet/wahid/projects/LQRAG_App/data/Sumaya-Babylon-report_edit1.pdf', 'file_type': 'application/pdf', 'file_size': 446978, 'creation_date': '2025-03-21', 'last_modified_date': '2025-03-21'}, hash='b2c623371b081d7ba67f433e0aa0c084d4ede449011619c1911356569bb80da7')}, metadata_template='{key}: {value}', metadata_separator='\n', text='GIST INFONET TECHNICAL REPORT 2025 \n \n \n \nComprehensive Analysis of Babylon \n \nSumaiia \nMentor: Seungmin Kim \nAdvisor: Heung-No Lee \nFeb. 24th 2025 \n \n \n \n \n \n \nObjective of the Report \nThis report aims to provide a comprehensive understanding of Babylon’s business model, \ntechnical innovations, testnet operations, Finality Provider mechanism, and Total Value \nLocked (TVL) metrics. By analyzing these aspects, the study will evaluate Babylon’s potential \napplications in PoS networks and broader Web3 ecosystems.', mimetype='text/plain', start_char_idx=0, end_char_idx=515, metadata_seperator='\n', text_template='{metadata_str}\n\n{content}'), score=1.8070924282073975), NodeWithScore(node=TextNode(id_='fedac83b-67eb-43b1-8fe1-e85aad5b8a37', embedding=None, metadata={'page_label': '26', 'file_name': 'Sumaya-Babylon-report_edit1.pdf', 'file_path': '/home/infonet/wahid/projects/LQRAG_App/data/Sumaya-Babylon-report_edit1.pdf', 'file_type': 'application/pdf', 'file_size': 446978, 'creation_date': '2025-03-21', 'last_modified_date': '2025-03-21'}, excluded_embed_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], excluded_llm_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], relationships={<NodeRelationship.SOURCE: '1'>: RelatedNodeInfo(node_id='a84e96c2-4f38-4579-8dfb-b7ecad4f4468', node_type='4', metadata={'page_label': '26', 'file_name': 'Sumaya-Babylon-report_edit1.pdf', 'file_path': '/home/infonet/wahid/projects/LQRAG_App/data/Sumaya-Babylon-report_edit1.pdf', 'file_type': 'application/pdf', 'file_size': 446978, 'creation_date': '2025-03-21', 'last_modified_date': '2025-03-21'}, hash='4455e075c400ae9d01445541e31ff6f1653108794699f6db12f10b540132392c')}, metadata_template='{key}: {value}', metadata_separator='\n', text='GIST INFONET TECHNICAL REPORT 2025 \n \n24 | Page \n \n \n\uf0fc Extracts the Bitcoin secret key (SK) of the finality provider that allegedly engaged \nin selective slashing. It is important for verifying the finality provider who signed \nan invalid transaction. \n4. Ensure the BTC Stake Was Linked to the Finality Provider: \n \n \n \n \n\uf0fc Checks if the BTC delegation was actually staked to the finality provider in \nquestion. If the stake was never linked to the finality provider, then slashing does \nnot occur.  \n5. Check if the Finality Provider Has Already Been slashed: \n \n \n \n \n \n \n\uf0fc Important for ensuring the ﬁnality provider has not already been slashed. A \nprovider should not be slashed multiple times for the same activity.  \n6. Slash the Finality Provider: \n \n \n \nfpIdx := btcDel.GetFpIdx(fpBTCPK) \nif fpIdx == -1 { \n    return nil, types.ErrFpNotFound.Wrapf("BTC delegation is not \nstaked to the finality provider")} \n \n \n \n \n \n \n \n \n \n \n \n \n \nfp, err := ms.GetFinalityProvider(ctx, fpBTCPK.MustMarshal()) \nif err != nil { \n    panic(types.ErrFpNotFound.Wrapf("failing to find the \nfinality provider with BTC delegations"))} \nif fp.IsSlashed() { \n    return nil, types.ErrFpAlreadySlashed} \n \n \n \n \n \n \n \n \n \n \n \n \nif err := ms.SlashFinalityProvider(ctx, fpBTCPK.MustMarshal()); \nerr != nil { \n    panic(err) // failed to slash the finality provider, must be \nprogramming error}', mimetype='text/plain', start_char_idx=0, end_char_idx=1379, metadata_seperator='\n', text_template='{metadata_str}\n\n{content}'), score=1.0283212661743164), NodeWithScore(node=TextNode(id_='5fa6ebff-f97e-47e7-9fe7-39efea1ef433', embedding=None, metadata={'page_label': '25', 'file_name': 'Sumaya-Babylon-report_edit1.pdf', 'file_path': '/home/infonet/wahid/projects/LQRAG_App/data/Sumaya-Babylon-report_edit1.pdf', 'file_type': 'application/pdf', 'file_size': 446978, 'creation_date': '2025-03-21', 'last_modified_date': '2025-03-21'}, excluded_embed_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], excluded_llm_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], relationships={<NodeRelationship.SOURCE: '1'>: RelatedNodeInfo(node_id='2ad5d8e1-4953-4c54-a901-8a6ea2a6d540', node_type='4', metadata={'page_label': '25', 'file_name': 'Sumaya-Babylon-report_edit1.pdf', 'file_path': '/home/infonet/wahid/projects/LQRAG_App/data/Sumaya-Babylon-report_edit1.pdf', 'file_type': 'application/pdf', 'file_size': 446978, 'creation_date': '2025-03-21', 'last_modified_date': '2025-03-21'}, hash='230aba92c13d51fc60cec7c7d098afa422e2d01d14e89b0c7f9d861b4633379f')}, metadata_template='{key}: {value}', metadata_separator='\n', text='GIST INFONET TECHNICAL REPORT 2025 \n \n23 | Page \n \n \nSlashing in Babylon\'s System \n Function: SelectiveSlashingEvidence (Slashing Malicious Finality Providers) \n \n \n \n\uf0fc The function above penalizes the finality providers who perform selective \nslashing. It verifies if a finality provider has acted maliciously and then slashes \nthem.  \n1. Get the staking record: \n \n \n \n\uf0fc Responsible for retrieving the BTC delegation record using the staking transaction \nhash, necessary for confirming the stake before performing slashing. \n2. Check if the Stake is Still Active: \n \n \n \n \n \n\uf0fc The code above is to check if the stake is still active and was unbounded early. To \nperform a slashing the delegation must be active.  \n3. Decide the Finality Provider\'s BTC Secret Key: \n \n \nfunc (ms msgServer) SelectiveSlashingEvidence(goCtx \ncontext.Context, req *types.MsgSelectiveSlashingEvidence) \n(*types.MsgSelectiveSlashingEvidenceResponse, error)  \n \n \n \n \n \n \n \n \n \n \nbtcDel, bsParams, err := ms.getBTCDelWithParams(ctx, \nreq.StakingTxHash) \nif err != nil {return nil, err} \n \n \n \n \n \n \n \n \n \n \n \nbtcTip := ms.btclcKeeper.GetTipInfo(ctx) \nif btcDel.GetStatus(btcTip.Height, bsParams.CovenantQuorum) != \ntypes.BTCDelegationStatus_ACTIVE && !btcDel.IsUnbondedEarly() { \n    return nil, types.ErrBTCDelegationNotFound.Wrap("a BTC \ndelegation that is not active or unbonding early cannot be \nslashed")} \n \n \n \n \n \n \n \n \n \nfpSK, fpPK := btcec.PrivKeyFromBytes(req.RecoveredFpBtcSk) \nfpBTCPK := bbn.NewBIP340PubKeyFromBTCPK(fpPK)', mimetype='text/plain', start_char_idx=0, end_char_idx=1513, metadata_seperator='\n', text_template='{metadata_str}\n\n{content}'), score=1.0073437690734863), NodeWithScore(node=TextNode(id_='c7463736-dd2a-4fa0-b786-cbc7f791221b', embedding=None, metadata={'page_label': '11', 'file_name': 'Sumaya-Babylon-report_edit1.pdf', 'file_path': '/home/infonet/wahid/projects/LQRAG_App/data/Sumaya-Babylon-report_edit1.pdf', 'file_type': 'application/pdf', 'file_size': 446978, 'creation_date': '2025-03-21', 'last_modified_date': '2025-03-21'}, excluded_embed_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], excluded_llm_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], relationships={<NodeRelationship.SOURCE: '1'>: RelatedNodeInfo(node_id='85e18b7f-6d15-40bd-b39b-3a6039f95545', node_type='4', metadata={'page_label': '11', 'file_name': 'Sumaya-Babylon-report_edit1.pdf', 'file_path': '/home/infonet/wahid/projects/LQRAG_App/data/Sumaya-Babylon-report_edit1.pdf', 'file_type': 'application/pdf', 'file_size': 446978, 'creation_date': '2025-03-21', 'last_modified_date': '2025-03-21'}, hash='2adf6a3e04d5d1e14548b89ebb1ae92da420bfd4a44ae64ff2d33fd8448541da'), <NodeRelationship.NEXT: '3'>: RelatedNodeInfo(node_id='48e885c1-bd60-4dbd-8c40-eb11ea7d3edc', node_type='1', metadata={}, hash='8ade181ab6504f44b4b38fa2e79e1c389b61cd25a731e77483358676e8da3c5a')}, metadata_template='{key}: {value}', metadata_separator='\n', text='GIST INFONET TECHNICAL REPORT 2025 \n \n9 | Page \n \n The EOTS ensure that if the provider is caught in double signing of conflicting blocks, \ntheir private key is exposed, which results in immediate slashing and loss of the voting power \nfor.   \n  Finality mechanism is highly decentralized, not a single entity can claim full control over \nthe voting power. This in turn ensures that finality is fairly based on protocol rules, preventing \ncensorship and manipulation.  \nEconomic Incentives and Risks \nRewards for Finality Providers  \nFinality providers are financially incentivized through staking rewards, transaction fees, \nand delegated staking commissions. They earn Babylon tokens (tBABY) based on the number of \nblocks they finalize, ensuring continuous participation and network stability. Additionally, they \nreceive a portion of transaction fees from PoS networks, along with commissions from Bitcoin \nholders who delegate their stakes, providing an additional revenue stream.  \nPotential Slashing Risks & Network Stability \nThe protocol uses two distinct keys:  \n• EOTS Key: Mainly used for EOTS signature generation and stored in the EOTS \nmanager.  \n• Babylon Key: Used for general activities such as: signing transactions and claiming \nawards, and it’s stored in the finality provider Daemon’s keyring.  \nSlashing conditions: Once double signing is detected or submission of conflicting block \nis proven, their Bitcoin stake is forcibly slashed using EOTS mechanism.  \nStatus Transitions: Providers have different states: Registered, Active, Inactive, Jailed, \nand Slashed solely based on their behavior. However, once a provider is slashed, they cannot \nregain the power of voting.  \nUnbounding Protection: The unbounding period is essential for preventing last-minute-\nmalicious activities. As a result, finality providers should undergo the unbounding period before \nwithdrawing their stake. This prevents the cases of creating a fraudulent block and immediately \nexiting the network to escape the penalties.  \nEconomic Disincentives for Malicious Behavior: The design of the protocol states that \nthe cost of attempting an attack should significantly outweigh any future short-term benefits.', mimetype='text/plain', start_char_idx=0, end_char_idx=2208, metadata_seperator='\n', text_template='{metadata_str}\n\n{content}'), score=0.8130505084991455), NodeWithScore(node=TextNode(id_='e2a873a7-0168-4851-aefc-d5652efd7968', embedding=None, metadata={'page_label': '10', 'file_name': 'Sumaya-Babylon-report_edit1.pdf', 'file_path': '/home/infonet/wahid/projects/LQRAG_App/data/Sumaya-Babylon-report_edit1.pdf', 'file_type': 'application/pdf', 'file_size': 446978, 'creation_date': '2025-03-21', 'last_modified_date': '2025-03-21'}, excluded_embed_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], excluded_llm_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], relationships={<NodeRelationship.SOURCE: '1'>: RelatedNodeInfo(node_id='6c098434-a76d-4df8-a546-96dab1c4a22a', node_type='4', metadata={'page_label': '10', 'file_name': 'Sumaya-Babylon-report_edit1.pdf', 'file_path': '/home/infonet/wahid/projects/LQRAG_App/data/Sumaya-Babylon-report_edit1.pdf', 'file_type': 'application/pdf', 'file_size': 446978, 'creation_date': '2025-03-21', 'last_modified_date': '2025-03-21'}, hash='06a4b48aaf11fa19bfea6315214d94e430d5e6a88bde219283e84626810b024f'), <NodeRelationship.NEXT: '3'>: RelatedNodeInfo(node_id='be50492e-6bac-44e3-85e6-ea0238d9f358', node_type='1', metadata={}, hash='a59a95579755d320831eccfca20b4270562e0312983b5e544d31c1b214864aef')}, metadata_template='{key}: {value}', metadata_separator='\n', text='GIST INFONET TECHNICAL REPORT 2025 \n \n8 | Page \n \n4. Finality Provider Mechanism \nRole of Finality Providers \nFinality providers are crucial participants in the Babylon BTC staking protocol mainly for \nsecuring PoS chains by leveraging Bitcoin staking as collateral. More specifically, they provide \nfinality votes on top of Babylon’s consensus mechanism, CometBFT, and get rewarded from \ncommissions from BTC staking delegations. This process advances security and immutability, \nsince adding extra layer of security makes it harder for the attackers to reorganize blocks or \ncause long-range attacks. According to the latest testnet data from  Babylon’s staking dashboard, \ncurrently there are over 100 active finality providers, which demonstrates the increasing \nparticipation.  \nFinality Provider Architecture and Components  \nFinality providers in Babylon network rely on a three-component system which provides \nthem with efficient finality voting and staking mechanisms:  \n1. Babylon Node: This step is crucial for better security, so running a personal node is \nrecommended. The fully synced Babylon node provides a chain data and transaction \nsubmission capabilities and details.  \n2. Extractable One-Time Signature (EOTS) Manager: This component is recommended \nto run on a separate system for enhanced security. The main role is to handling EOTS \nkey operations, generating one-time signatures and ensuring public randomness.  \n3. Finality Provider Daemon: This daemon is responsible for monitoring Babylon \nblocks, committing public randomness, and submitting finality signatures. It also \ntracks provider status transitions (Active, Inactive, Jailed, Slashed) and ensures secure \ncoordination with the EOTS Manager for key operations. Additionally, it handles \nstaking rewards and their distribution, ensuring providers are properly compensated \nfor their contributions.  \nThe above listed components are essential for maintaining a secure and stable chain, \nwhich takes under control that each finality vote satisfies the protocol standards.  \nEnsuring Finality Using Staked BTC \nFinality providers are required to lock up their Bitcoin on the Babylon staking protocol as \ncollateral, solely for the purpose of slashing if they contributed to some malicious activity.', mimetype='text/plain', start_char_idx=0, end_char_idx=2282, metadata_seperator='\n', text_template='{metadata_str}\n\n{content}'), score=0.0), NodeWithScore(node=IndexNode(id_='e95ae07f-eb12-4897-bd45-2b6d37fba1e2', embedding=None, metadata={'page_label': '19', 'file_name': 'Sumaya-Babylon-report_edit1.pdf', 'file_path': '/home/infonet/wahid/projects/LQRAG_App/data/Sumaya-Babylon-report_edit1.pdf', 'file_type': 'application/pdf', 'file_size': 446978, 'creation_date': '2025-03-21', 'last_modified_date': '2025-03-21'}, excluded_embed_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], excluded_llm_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], relationships={<NodeRelationship.SOURCE: '1'>: RelatedNodeInfo(node_id='56e76161-73f4-434b-a7b5-abf5d1f3e2fe', node_type='4', metadata={'page_label': '19', 'file_name': 'Sumaya-Babylon-report_edit1.pdf', 'file_path': '/home/infonet/wahid/projects/LQRAG_App/data/Sumaya-Babylon-report_edit1.pdf', 'file_type': 'application/pdf', 'file_size': 446978, 'creation_date': '2025-03-21', 'last_modified_date': '2025-03-21'}, hash='4f1691d81999176c7f25acd71717b8f435e78cf704ceb37c8022ed47a3251673'), <NodeRelationship.NEXT: '3'>: RelatedNodeInfo(node_id='07e61d02-d5b8-46d1-bbc8-7f7232dcdad6', node_type='1', metadata={}, hash='c2eafcdb1edd433388b417cf054f20d3d887f4b1be62589fbf8e3368f105e8af')}, metadata_template='{key}: {value}', metadata_separator='\n', text="GIST INFONET TECHNICAL REPORT 2025 \n \n17 | Page \n \n7. Conclusion and Recommendations \nKey Takeaways \nIn this report, I have examined the role of Babylon in bridging Bitcoin's security with the \nscalability of PoS networks, evaluating the claim of enhancing interoperability. Although \nBabylon is still in the early stages of development, with a TVL of $6.1 billion representing \napproximately 0.3% of Bitcoin total market value, Babylon demonstrated its initial progress. By \nensuring data availability and robust finality, it contributes to the foundational infrastructure of \nWeb3. However, its long-term effectiveness in supporting decentralized applications and \nfinancial networks will depend on further adoption and ecosystem development.  \nFuture Applications \nGiven its strengths in security and finality, Babylon’s integration with emerging projects like \nWorldLand and My AI Network presents significant opportunities: \n• WorldLand Integration: WorldLand is an advanced blockchain ecosystem that supports \nverifiable AI computing, decentralized cloud services, and Web3 applications. Integrating \nBabylon’s finality mechanism can enhance security and ensure efficient data availability \nfor AI-driven transactions. This collaboration could help in securing AI-generated data \nand decentralized AI processing. \n• My AI Network Synergy: My AI Network operates within WorldLand, allowing users to \nown and monetize AI agents. Babylon’s Bitcoin-backed security model can provide an \nadditional layer of trust for AI-driven economies, ensuring the integrity of AI agent \ntransactions and protecting against fraud. \nRecommendations \n1. Technical Exploration: Conduct a feasibility study on integrating Babylon’s finality \nproviders with WorldLand’s verifiable AI computing infrastructure. \n2. Staked BTC Optimization: Further analyze the dynamics of staked BTC and finality \nproviders to refine the economic incentives for securing AI-driven smart contracts. \n3. Strategic Partnerships: Establish collaborations with decentralized AI ecosystems, \nensuring that Babylon’s security model is integrated with emerging AI and Web3 \ninfrastructures. \n4. Scalability Enhancements: Explore the use of Babylon’s data availability solutions to \noptimize WorldLand’s AI marketplace, ensuring cost-efficient and secure AI transactions.", mimetype='text/plain', start_char_idx=0, end_char_idx=2327, metadata_seperator='\n', text_template='{metadata_str}\n\n{content}', index_id='e95ae07f-eb12-4897-bd45-2b6d37fba1e2', obj=None), score=1.137040376663208)], QueryBundle(query_str='who is sumaiia?', image_path=None, custom_embedding_strs=None, embedding=[0.0010158951627090573, 0.050695810467004776, -0.04942626878619194, -0.01127035915851593, -0.031875766813755035, -0.024496344849467278, -0.016827639192342758, 0.024582069367170334, 0.004126532468944788, -0.01310395635664463, -0.028837574645876884, -0.0009720892994664609, -0.05221825838088989, 0.05251896753907204, -0.03574901446700096, -0.03200455754995346, -0.006196132395416498, -0.008773144334554672, -0.021997224539518356, 0.01637735776603222, 0.07747914642095566, 0.02589547075331211, 0.00020772978314198554, -0.008921219035983086, -0.013038056902587414, 0.022288678213953972, -0.018899250775575638, 0.036544255912303925, 0.030271973460912704, -0.0015405681915581226, -0.026067038998007774, 0.04575793072581291, 0.010354426689445972, -0.06709032505750656, 0.012408632785081863, -0.016573432832956314, -0.015814118087291718, -0.017791351303458214, -0.015372471883893013, -0.052550673484802246, 0.0037052929401397705, -0.02272639609873295, 0.018855150789022446, -0.01289255078881979, -0.04311837628483772, 0.008909991942346096, 9.079441952053457e-05, -0.03936401754617691, -0.0009293255279771984, -0.011030126363039017, 0.02956998534500599, 0.005747542250901461, -0.010003325529396534, 0.05241486802697182, -0.015123318880796432, 0.007019533775746822, -0.0020339840557426214, -0.014275492168962955, 0.026101842522621155, 0.01267262827605009, -0.026606259867548943, 0.006254978012293577, -0.025451114401221275, -0.05567602440714836, 0.016463907435536385, 0.0005294542061164975, -0.0004981959937140346, -0.05206691101193428, 0.007441424299031496, -0.008425175212323666, 0.030250171199440956, 0.0034275269135832787, 0.01766805909574032, -0.006766508333384991, -0.047168321907520294, -0.014041922986507416, -0.0615304633975029, -0.03156650438904762, 0.04091230407357216, -0.005702315829694271, -0.01636134460568428, 0.02052701823413372, -0.038367439061403275, 0.00408252002671361, -0.018052464351058006, -0.004091203678399324, -0.03704344481229782, 0.009193573147058487, -0.024868933483958244, -0.0009684372926130891, 0.05493045225739479, 0.03333289176225662, -0.010079685598611832, 0.024953221902251244, 0.03542080521583557, 0.02226121537387371, 0.023956215009093285, 0.021077994257211685, 0.03100675716996193, 0.05423474684357643, 0.05961038917303085, -0.01022806204855442, 0.016528278589248657, 0.06557615846395493, -0.07733561843633652, 0.031632885336875916, 0.05377384275197983, 0.011903837323188782, 0.010936586186289787, -0.0020524393767118454, 0.05270271748304367, -0.015519830398261547, 0.047970980405807495, 0.004194184672087431, -0.00031209015287458897, -0.0034277159720659256, -0.009329508058726788, 0.03697458654642105, -0.033595047891139984, 0.009909865446388721, -0.02578691393136978, 0.013985238038003445, -0.0032225740142166615, -0.005812849849462509, 0.02085038088262081, -0.029229111969470978, 0.007261431775987148, -0.02071220614016056, 0.0019189596641808748, 0.05074911564588547, -0.01946687512099743, -0.03035566210746765, 0.009005698375403881, -0.008617447689175606, -0.02626771852374077, 0.01731443591415882, 0.0021879670675843954, 0.03172716125845909, 0.03490534797310829, -0.03136248514056206, 0.04176446050405502, -0.02548074722290039, -0.01352185569703579, 0.05717313662171364, -0.018386907875537872, 0.07015816867351532, -0.04015815630555153, 0.00581551855430007, -0.055141136050224304, -0.01852947659790516, -0.02030855603516102, -0.026661530137062073, -0.037026818841695786, 0.020497998222708702, -0.010370928794145584, -0.026305558159947395, 0.005436974577605724, -0.012453988194465637, 0.0024802968837320805, 0.01381638366729021, -0.04774412512779236, 0.023079801350831985, 0.011442109011113644, 0.022440357133746147, -0.04407496750354767, 0.008391786366701126, -0.003397035412490368, 0.0014655098784714937, 0.029205119237303734, 0.03938482701778412, 0.05730351805686951, -0.031263358891010284, -0.0291777104139328, -0.003345177276059985, -0.009897486306726933, 0.009306144900619984, -0.05882636830210686, -0.05816721171140671, 0.020557314157485962, -0.0038747137878090143, 0.010829580016434193, 0.0065782140009105206, -0.004199356306344271, 0.033031776547431946, 0.010179939679801464, 0.01471042912453413, 0.00920167937874794, -0.0437510684132576, -0.0008793672313913703, -0.0022410491947084665, 0.020063301548361778, 0.014838814735412598, -0.018983323127031326, 0.00520396837964654, 0.00025484178331680596, 0.08026646077632904, -0.023393897339701653, -0.01254157442599535, 0.016022751107811928, -0.020452404394745827, -0.007176164537668228, 0.03640264272689819, -0.011486805975437164, -0.007631285581737757, 0.016710057854652405, -0.02356942556798458, 0.004562001675367355, 0.0532086007297039, -0.06018902733922005, 0.008908336982131004, 0.012293574400246143, -0.048517681658267975, -0.0255111251026392, 0.003757712198421359, -0.01873268559575081, 0.022497817873954773, -0.01940866746008396, 0.05747499316930771, -0.025415915995836258, 0.01975446566939354, 0.026429932564496994, 0.04001341760158539, -0.01443413645029068, 0.05117237567901611, -0.011938944458961487, 0.001357102650217712, 0.009572841227054596, 0.00842228252440691, -0.02639063633978367, -0.06185610219836235, 0.026151278987526894, 0.009546431712806225, 0.006212160456925631, 0.04235202074050903, -0.03002232313156128, 0.012591390870511532, -0.004702830221503973, 0.0015056050615385175, -0.02407158352434635, -0.014373005367815495, 0.022173181176185608, -0.0066941301338374615, 0.0678887888789177, 0.0780976414680481, -0.01185495313256979, 0.03039020299911499, 0.006900050677359104, 0.021965784952044487, -0.030815234407782555, 0.035707611590623856, -0.028419125825166702, 0.04196739196777344, 0.05522225424647331, 0.015263793990015984, -0.030646109953522682, -0.009991191327571869, 0.033945076167583466, 0.01659780740737915, -0.017815371975302696, -0.014307310804724693, -0.08722112327814102, -0.0057045528665184975, -0.004316647071391344, 0.0016649097669869661, -0.04559125751256943, 0.038614965975284576, 0.00684282835572958, 0.016996238380670547, -0.062303848564624786, -0.03874281048774719, -0.06315132230520248, -0.004072663839906454, -0.004154942464083433, -0.014205335639417171, 0.021357407793402672, -0.00021814303181599826, 0.029679154977202415, -0.009494594298303127, 0.022583995014429092, 0.026004262268543243, -0.03208911046385765, 0.011640156619250774, -0.04073026403784752, -0.0420396514236927, -0.012764181941747665, 0.06195475533604622, -0.02839975245296955, 0.024930289015173912, -0.018434040248394012, 0.07506375759840012, -0.021133903414011, -0.025537393987178802, -0.0305881816893816, 0.0018539869925007224, 0.029279740527272224, 0.011141750030219555, -0.036931224167346954, -0.0036594399716705084, -0.040480200201272964, -0.05528574436903, 0.06562107056379318, -0.03964264318346977, 0.007067789789289236, 0.007945328019559383, -0.015210132114589214, 0.041544269770383835, 0.0519513338804245, 0.026417387649416924, -0.027396034449338913, 0.02140316553413868, -0.032563891261816025, 0.03124042972922325, -0.06276829540729523, 0.0016515565803274512, -0.04284842684864998, 0.0017541886772960424, 0.007432299200445414, -0.03478172793984413, 0.005960730370134115, -0.02560817077755928, -0.03736412897706032, -0.027816276997327805, 0.0157720185816288, -0.01587742753326893, 0.029314137995243073, 0.009107508696615696, -0.007922493852674961, -0.04321284964680672, 0.008031712844967842, -0.05473210662603378, -0.03142094612121582, -0.007495452184230089, -0.013486392796039581, -0.04183201864361763, 0.003447727533057332, 0.03878981992602348, -0.050059109926223755, 0.037497811019420624, -0.03241322189569473, -0.02562449313700199, 0.02340557798743248, -0.017877021804451942, 0.024488335475325584, 0.007479390129446983, -0.03088970109820366, 0.014465872198343277, 0.014269208535552025, -0.0071269734762609005, -0.02016874961555004, -0.0005692585837095976, -0.013768327422440052, 0.01475517824292183, -0.018490321934223175, 0.03532152250409126, 0.002513164421543479, -0.004594538826495409, 0.038907382637262344, 0.024259600788354874, -0.005710730329155922, -0.012075944803655148, -0.002457464812323451, 0.024477902799844742, 0.02279796451330185, -0.037627238780260086, 0.003879323834553361, -0.03190550580620766, -0.020818565040826797, 0.026317374780774117, 0.023641161620616913, -0.015330915339291096, 0.009711512364447117, -0.0052116853184998035, 0.0015926174819469452, 0.00042677263263612986, -0.005755786783993244, 0.015861012041568756, 0.05162777379155159, -0.03520089387893677, 0.0020889132283627987, 0.005245026666671038, 0.016158565878868103, 0.022832656279206276, 0.060089580714702606, 0.011188307777047157, -0.041081011295318604, -0.007811440154910088, -0.018832966685295105, 0.02148309536278248, 0.01794341951608658, -0.027617355808615685, -0.011088681407272816, 0.007646209094673395, 0.004123230464756489, -0.06223579868674278, -0.057306163012981415, -0.07331576198339462, 0.0020103827118873596, 0.03401079773902893, 0.05484047159552574, -0.05259459465742111, 0.013841233216226101, 0.020249217748641968, 0.0638844296336174, 0.07899625599384308, -0.017785780131816864, 0.02200862020254135, -0.05255885049700737, 0.04846807196736336, 0.010448489338159561, 0.02122301608324051, 0.0011476930230855942, -0.01805044338107109, -0.004905336536467075, -0.0015484835021197796, 0.0581522099673748, -0.045401789247989655, -0.02971539832651615, 0.010343254543840885, -0.024265214800834656, 0.042142353951931, -0.012983846478164196, 0.008132300339639187, -0.019730284810066223, -0.0030585655476897955, 0.028680114075541496, 0.010389099828898907, -0.0573076531291008, -0.0034994983579963446, 0.07439219951629639, 0.017807792872190475, -0.01781143620610237, 0.001715268474072218, -0.018589068204164505, -0.04277926683425903, -0.037720587104558945, 0.058692753314971924, 0.04057915508747101, 0.03549465909600258, -0.043756261467933655, -0.011523324064910412, 0.05368980020284653, 0.06605987250804901, -0.029570886865258217, -0.00991167314350605, -0.006906087510287762, -0.02167317271232605, -0.005791931878775358, -0.009505297057330608, 0.022338608279824257, -0.06815382093191147, 0.04000234976410866, -0.05200377106666565, 0.025937287136912346, 0.027721386402845383, 0.02082665078341961, -0.054858800023794174, 0.020602919161319733, 0.05747261270880699, 0.023239539936184883, 0.07033885270357132, -0.044893089681863785, 0.0022111875005066395, 0.05053861811757088, 0.004941780120134354, -0.02133258432149887, 0.05172829329967499, 0.027334710583090782, 0.02838473580777645, 0.0015172893181443214, 0.017840823158621788, -0.04254251718521118, 0.004480484873056412, -0.023854682222008705, -0.036536697298288345, 0.030905241146683693, -0.025394346565008163, -0.015680324286222458, -0.004377347882837057, 0.00411057285964489, 0.004065140150487423, -0.0006853308877907693, 0.00690446374937892, 0.032757751643657684, -0.030045637860894203, 0.0007478748448193073, -0.032993849366903305, 0.010811427608132362, -0.015650499612092972, -0.0960087776184082, -0.009378175251185894, -0.032513462007045746, -0.0794551745057106, 0.010192830115556717, -0.03841080144047737, 0.00044917294871993363, -0.02864692732691765, -0.02494695410132408, -0.03330395743250847, 0.011596865952014923, -0.01994466222822666, 0.05632080137729645, 0.024166733026504517, 0.000696566654369235, -0.0277131125330925, -0.050106313079595566, -0.010322239249944687, -0.020067457109689713, 0.007434149272739887, -0.025089800357818604, 0.023259233683347702, 0.007690886035561562, 0.01610151305794716, 0.018133582547307014, 0.017951253801584244, 0.010182800702750683, 0.006541278213262558, 0.051409292966127396, -0.031663257628679276, -0.004748832434415817, 0.025459814816713333, 0.04075164720416069, -0.03422742709517479, -0.0014459722442552447, 0.06749130040407181, 0.027132567018270493, 0.010983161628246307, -0.005993236787617207, -0.0028718088287860155, -0.0361236110329628, -0.022137172520160675, 0.047079894691705704, 0.02029377967119217, -0.020372014492750168, -0.05855177715420723, 0.008144364692270756, -0.03914763033390045, 0.04889407381415367, -0.01415191125124693, 0.02098814956843853, -0.006390338763594627, -0.029807258397340775, -0.005180033389478922, -0.03520110994577408, -0.022507021203637123, -0.024255698546767235, 0.005444124341011047, 0.03690272197127342, 0.02202123962342739, 0.019604338333010674, 0.06579367816448212, 0.04069139435887337, 0.04970843344926834, -0.025930112227797508, -0.039826277643442154, -0.01709134504199028, -0.028642859309911728, -0.007313322741538286, -0.01500711403787136, 0.010367024689912796, -0.009524667635560036, -0.02217923477292061, -0.026430774480104446, -0.018223600462079048, -0.05067070201039314, 0.0012985516805201769, -0.002581867156550288, 0.03022136352956295, 0.014987785369157791, 0.02493900991976261, -0.02601814828813076, -0.0033964072354137897, 0.010329005308449268, 0.011413171887397766, 0.05063026770949364, 0.00842503271996975, -0.02516048029065132, -0.012229353189468384, 0.004094393458217382, -0.0371512807905674, -0.014784548431634903, -0.03195544704794884, 0.0009278380894102156, -0.026201942935585976, -0.03963687643408775, 0.0014937104424461722, -0.034550707787275314, 0.04695131257176399, 0.05237670615315437, 0.02410539984703064, 0.024095119908452034, -0.016636507585644722, -0.00047369819367304444, 0.0020396390464156866, -0.004271149169653654, -0.039642687886953354, -0.027228044345974922, -0.03654726967215538, -0.06980584561824799, -0.03660448268055916, -0.058487970381975174, -0.044411949813365936, -0.0010159430094063282, 0.023822948336601257, -0.031424060463905334, 0.0010535132605582476, -0.008737413212656975, -0.019861606881022453, -0.014199109748005867, 0.08897997438907623, 0.015218189917504787, 0.013290251605212688, 0.015983251854777336, -0.040078263729810715, -0.01627511717379093, -0.015314619056880474, -0.0004237864341121167, -0.010191847570240498, -0.013548622839152813, 0.023207975551486015, -0.00993028748780489, 0.013050256296992302, 0.04976662993431091, 0.04365210607647896, -0.006723940838128328, -0.0307376217097044, 0.017161037772893906, 0.017300307750701904, 0.022096578031778336, -0.008430533111095428, -0.015001574531197548, 0.016882309690117836, -0.06888515502214432, 0.0004971316666342318, 0.04797567427158356, 0.005829789210110903, -0.0005152823287062347, 0.017757946625351906, 0.005861619021743536, -0.03095277026295662, 0.03986486792564392, 0.04608485847711563, -0.00804082490503788, -0.05108854919672012, -0.05353133752942085, -0.013574100099503994, -0.05041429027915001, 0.0171110387891531, 0.051714636385440826, -0.02582469955086708, -0.011906726285815239, 0.0040932511910796165, 0.005998461041599512, -0.016232622787356377, -0.05830131098628044, -0.004709525965154171, 0.026897629722952843, -0.026503097265958786, -0.014531200751662254, -0.046309441328048706, 0.012438238598406315, -0.03645934537053108, 0.011752978898584843, 0.023037198930978775, 0.03115774691104889, 0.032571449875831604, 0.033273253589868546, -0.005661999806761742, -0.022054525092244148, -0.0042083729058504105, -0.029505979269742966, 0.023598965257406235, -0.04998089373111725, -0.00210282439365983, 0.03746270388364792, 0.0019776392728090286, -0.002671752357855439, -0.02582649141550064, -0.008568189106881618, -0.01879553683102131, 0.04420524090528488, -0.0029556637164205313, -0.03212076425552368, 0.015998654067516327, 0.019685331732034683, -0.034360371530056, -0.009255768731236458, -0.02752986177802086, -0.03491877764463425, 0.02494100295007229, -0.024107402190566063, 0.0036419574171304703, 0.006740960292518139, 0.07899319380521774, 0.028793057426810265, 0.01465807668864727, -0.025783013552427292, 0.007846415042877197, -0.031331758946180344, 0.020406873896718025, 0.03866224363446236, -0.03165697678923607, 0.057620562613010406, 0.001082408707588911, -0.006535081192851067, 0.009618835523724556, -0.029336048290133476, 0.0070463865995407104, -0.008085191249847412, 0.005499653983861208, -0.04143008217215538, -0.007131089456379414, -0.012225140817463398, -0.0010603175032883883, -0.012422350235283375, -0.016907526180148125, 0.06440352648496628, -0.0026580472476780415, 0.004786627367138863, 0.03641147539019585, 0.01502197701483965, 0.01220446266233921, -0.036141641438007355, 0.046343572437763214, 0.021264536306262016, -0.004224080126732588, 0.040648192167282104, -0.02486540377140045, -0.00312090408988297, 0.05010238662362099, 0.014213424175977707, -0.05455979332327843, 0.036400213837623596, -0.004358419217169285, 0.044235020875930786, 0.016227474436163902, -0.07385113090276718, -0.04065914452075958, -0.11059644818305969, -0.014074117876589298, 0.01669308915734291, 0.002965302439406514, 0.01733516901731491, -0.01343806553632021, -0.034500155597925186, 0.000306966743664816, -0.021776355803012848, -0.02584131807088852, -0.07688827067613602, -0.011206582188606262, 0.017086021602153778, -0.03716106712818146, -0.0017579671693965793, -0.011807533912360668, 0.013409378938376904, 0.027496758848428726, -0.01615527831017971, 0.02142692357301712, -0.03078388050198555, -0.008584236726164818, -0.008309535682201385, -0.028409257531166077, -0.04442479833960533, -0.04886406287550926, -0.01119893416762352, 0.05997785925865173, 0.019910093396902084, -0.0021727147977799177, 0.006508946418762207, 0.02225763350725174, 0.01935679465532303, -0.01639733463525772, -0.009243232198059559, -0.028984427452087402, 0.01010278519243002, 0.016420962288975716, 0.017764264717698097, 0.014661284163594246, -0.03345590457320213, 0.02515655942261219, -0.007830128073692322, 0.07072772085666656, -0.020034343004226685, -0.010759520344436169, 0.03155606612563133, -0.03250659257173538, -0.010098193772137165, -0.01978687196969986, 0.03654475510120392, 0.04768191650509834, -0.023613380268216133, -0.02065170183777809, 0.056056443601846695, 0.005377693567425013, -0.0065779052674770355, -0.009631626307964325, 0.02748824656009674, 0.011287621222436428, -0.016560139134526253, -0.05512787401676178, 0.019315307959914207, 0.05952930077910423, 0.039792437106370926, 0.014185520820319653, 0.04671886935830116, 0.007770878728479147, -0.01409964356571436, 0.0007548668072558939, -0.04261268675327301, -0.020276006311178207, -0.008197678253054619, 0.003627508645877242, 0.00545722059905529, -0.03519666939973831, -0.032374873757362366, -0.0028425813652575016, -0.008116806857287884, 0.02438443712890148, -0.0047633531503379345, 0.01026850938796997, 0.011835048906505108, -0.055234216153621674, -0.00576699897646904, -0.006853118073195219, 0.04393506795167923, 0.034764379262924194, -0.006659071892499924, -0.0013197202933952212, 0.007891862653195858, -0.008955681696534157, 0.02593548782169819, 0.013342591002583504, -0.028777843341231346, -0.01686769165098667, 0.01902434043586254, 0.01862715743482113, 0.019201651215553284, -0.004672296345233917, 0.006761275231838226, -0.009157097898423672, -0.0213024765253067, 0.021611537784337997, 0.02162206918001175, -0.021483521908521652, -0.02151845581829548, 0.000503410876262933, 0.016272297129034996, 0.003730760421603918, 0.012572646141052246, 0.03956632688641548, 0.012002494186162949, 0.015465831384062767, 0.016307910904288292, 0.02580222859978676, -0.02349156327545643, 0.0057894825004041195, 0.011623773723840714, 0.027969537302851677, 0.013972599059343338, 0.041719917207956314, -0.001028863713145256, -0.005007443018257618, -0.009885509498417377, 0.01967627927660942, -0.03105718083679676, -0.028084784746170044, -0.015152561478316784, -0.036549173295497894, 0.0021495854016393423, 0.02319735661149025, -0.02466752752661705, 0.048158470541238785, 0.011194770224392414, -0.02583220973610878, 0.028267376124858856, -0.008589745499193668, -0.01623486913740635, -0.01760750077664852, 0.014607311226427555, -0.0014505407307296991, -0.024830937385559082, 0.0267300046980381, 0.00853796862065792, -0.02997654117643833, 0.019494062289595604, -0.0780017077922821, 0.00864136591553688, -0.0023615886457264423, -0.008418486453592777, -0.031428270041942596, -0.02703976258635521, -0.02932880073785782, -0.01837117038667202, 0.029077447950839996, -0.070770263671875, -0.034663014113903046, 0.07090966403484344, 0.03397457301616669, -0.010397008620202541, -0.033781997859478, -0.027067774906754494, 0.06973231583833694, 0.06487725675106049, -0.01709202118217945, 0.01790681481361389, -0.008515630848705769, -0.06831308454275131, 0.008008219301700592, -0.0054577551782131195, -0.023105595260858536, 0.020635372027754784, -0.04882827028632164, -0.021380268037319183, 0.013389511965215206, -0.004051781725138426, -0.028248950839042664, 0.0682724118232727, -0.013223521411418915, 0.0027428744360804558, -0.003064975142478943, 0.004350913688540459, -0.011374791152775288, -0.02362634800374508, 0.010476970113813877, -0.02742060273885727, 0.046922191977500916, -0.004328732378780842, 0.05180448293685913, -0.012857680208981037, -0.045887891203165054, 0.2398333102464676, 0.013245777226984501, 0.061198554933071136, 0.013426246121525764, 0.051728714257478714, 0.010896995663642883, 0.01834177039563656, -0.01868411898612976, 0.02043427899479866, -0.010348117910325527, 0.02558956854045391, 0.009088661521673203, 0.035500459372997284, -0.009083283133804798, 0.0058626458048820496, 0.05858949199318886, -0.023816177621483803, 0.030412107706069946, 0.003175305901095271, -0.007221709005534649, -0.07176884263753891, -0.03026735782623291, -0.004543688148260117, -0.0217414703220129, 0.009916695766150951, -0.0012547411024570465, 0.012193386442959309, 0.017852280288934708, -0.017866116017103195, -0.002433058572933078, 0.05861714109778404, 0.044921621680259705, 0.02247857302427292, 0.028890402987599373, 0.030434031039476395, 0.006532593630254269, 0.011308629997074604, -0.059207405894994736, -0.0030797875951975584, 0.022837353870272636, 0.010310466401278973, 0.0188568364828825, -0.021737825125455856, -0.01206745021045208, 0.023303736001253128, 0.0019061914645135403, -0.018272006884217262, 0.04825218394398689, -0.01758745312690735, -0.04045376181602478, 0.030469616875052452, -0.01462066825479269, -0.0010745706968009472, -0.04367183893918991, -0.039897192269563675, -0.030453132465481758, -0.027677420526742935, 0.042246878147125244, -0.021744901314377785, -0.0019813848193734884, -0.0075623332522809505, -0.01222431194037199, -0.04018986597657204, 0.029901428148150444, -0.02695791982114315, 0.052673064172267914, 0.013707988895475864, 0.03989354893565178, -0.06219653785228729, 0.04352793097496033, -0.0010810209205374122, -0.04622858017683029, -0.00687115453183651, 0.004669047426432371, 0.03209609165787697, -0.003885348094627261, -0.013828885741531849, 0.01987038366496563, -0.006520298309624195, -0.011158379726111889, 0.018633244559168816, -0.033500637859106064, -0.04505087435245514, 0.04699641466140747, 0.023260822519659996, -0.005497114732861519, 0.015585044398903847, -0.019836755469441414, -0.038336433470249176, 0.04104968532919884, 0.029470151290297508, 0.008905960246920586, -0.027775850147008896, 0.04527006298303604, -0.03574372082948685]))
calling <function CompactAndRefine.get_response at 0x7f0e15cc2700> with (<llama_index.core.response_synthesizers.compact_and_refine.CompactAndRefine object at 0x7f0df437bce0>,)
calling <function Refine.get_response at 0x7f0e15cc1da0> with (<llama_index.core.response_synthesizers.compact_and_refine.CompactAndRefine object at 0x7f0df437bce0>,)
Sumaiia is the author of the GIST INFONET TECHNICAL REPORT 2025, a comprehensive analysis of Babylon's business model, technical innovations, and potential applications in PoS networks and broader Web3 ecosystems. Thanks for asking!`

Hi folks, a quick update here: our latest fix (https://github.com/truera/trulens/pull/2059) should improve performance and mitigate issues with `groundedness` not working.

@wahidur028, please update this thread if you find that a different model works or if you are able to identify more specific issues!