Repository: confident-ai/deepeval
Issue: RuntimeError: Cannot locate @observe function
Root cause: Validation Gap


Hi everyone, I am new to deepEval, and when i run an example code in https://www.confident-ai.com/blog/llm-agent-evaluation-complete-guide for tool correctness, i got an error: RuntimeError: Cannot locate @observe function. Though using end-to-end mode without tracing works. I am using openAI api_key. The code is below,

from typing import List
from deepeval.tracing import observe, update_current_span
from deepeval.metrics import ToolCorrectnessMetric
from deepeval.test_case.llm_test_case import LLMTestCase, ToolCall, ToolCallParams
from deepeval.evaluate import evaluate
from deepeval.dataset import Golden


@observe(
    type="agent",
    metrics=[
        ToolCorrectnessMetric(
            evaluation_params=[ToolCallParams.OUTPUT, ToolCallParams.INPUT_PARAMETERS],
            should_consider_ordering=True
        )
    ]
)
def agent(input: str):
    tools_called = [
        ToolCall(name="WebSearchTool"),
        ToolCall(name="QueryTool")
    ]
    result = "We offer a 30-day full refund at no extra cost."
    update_current_span(
        test_case=LLMTestCase(
            input=input,
            actual_output=result,
            tools_called=tools_called,
            expected_tools=[ToolCall(name="WebSearchTool")]
        )
    )
    return result

evaluate(
    goldens=[Golden(input="What if these shoes don't fit?")],
    observed_callback=agent
)


And detailed error message is below
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
 Cell In[30], line 33
     23     update_current_span(
     24     test_case=LLMTestCase(
     25     input=input,
   (...)     29     )
     30     )
     31     return result
---> 33 evaluate(
     34 goldens=[Golden(input="What if these shoes don't fit?")],
     35 observed_callback=agent
     36 )

File ~/Projects/raid/code/eval/deepeval2/deepeval/evaluate/evaluate.py:227, in evaluate(test_cases, metrics, hyperparameters, goldens, observed_callback, identifier, async_config, display_config, cache_config, error_config)
    225 if async_config.run_async:
    226     loop = get_or_create_event_loop()
--> 227     test_results = loop.run_until_complete(
    228         a_execute_agentic_test_cases(
    229             goldens=goldens,
    230             observed_callback=observed_callback,
    231             ignore_errors=error_config.ignore_errors,
    232             verbose_mode=display_config.verbose_mode,
    233             show_indicator=display_config.show_indicator,
    234             skip_on_missing_params=error_config.skip_on_missing_params,
    235             throttle_value=async_config.throttle_value,
    236             identifier=identifier,
    237             max_concurrent=async_config.max_concurrent,
    238             save_to_disk=cache_config.write_cache,
    239         )
    240     )
    241 else:
    242     test_results = execute_agentic_test_cases(
    243         goldens=goldens,
    244         observed_callback=observed_callback,
   (...)    250         save_to_disk=cache_config.write_cache,
    251     )

File ~/Projects/raid/code/eval/deepeval2/.venv/lib/python3.12/site-packages/nest_asyncio.py:98, in _patch_loop.<locals>.run_until_complete(self, future)
     95 if not f.done():
     96     raise RuntimeError(
     97         'Event loop stopped before Future completed.')
---> 98 return f.result()

File /Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/asyncio/futures.py:203, in Future.result(self)
    201 self.__log_traceback = False
    202 if self._exception is not None:
--> 203     raise self._exception.with_traceback(self._exception_tb)
    204 return self._result

File /Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/asyncio/tasks.py:314, in Task.__step_run_and_handle_result(***failed resolving arguments***)
    310 try:
    311     if exc is None:
    312         # We use the `send` method directly, because coroutines
    313         # don't have `__iter__` and `__next__` methods.
--> 314         result = coro.send(None)
    315     else:
    316         result = coro.throw(exc)

File ~/Projects/raid/code/eval/deepeval2/deepeval/evaluate/execute.py:1168, in a_execute_agentic_test_cases(goldens, observed_callback, verbose_mode, ignore_errors, skip_on_missing_params, show_indicator, throttle_value, max_concurrent, save_to_disk, identifier, _use_bar_indicator, _is_assert_test)
   1165                 tasks.append(asyncio.create_task(task))
   1166                 await asyncio.sleep(throttle_value)
-> 1168         await asyncio.gather(*tasks)
   1169 else:
   1170     for golden in goldens:

File /Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/asyncio/tasks.py:314, in Task.__step_run_and_handle_result(***failed resolving arguments***)
    310 try:
    311     if exc is None:
    312         # We use the `send` method directly, because coroutines
    313         # don't have `__iter__` and `__next__` methods.
--> 314         result = coro.send(None)
    315     else:
    316         result = coro.throw(exc)

File ~/Projects/raid/code/eval/deepeval2/deepeval/evaluate/execute.py:1121, in a_execute_agentic_test_cases.<locals>.execute_with_semaphore(func, *args, **kwargs)
   1119 async def execute_with_semaphore(func: Callable, *args, **kwargs):
   1120     async with semaphore:
-> 1121         return await func(*args, **kwargs)

File ~/Projects/raid/code/eval/deepeval2/deepeval/evaluate/execute.py:1213, in a_execute_agentic_test_case(golden, test_run_manager, test_results, count, verbose_mode, ignore_errors, skip_on_missing_params, show_indicator, _use_bar_indicator, _is_assert_test, observed_callback, trace, progress, pbar_id)
   1194 async def a_execute_agentic_test_case(
   1195     golden: Golden,
   1196     test_run_manager: TestRunManager,
   (...)   1210     pbar_id: Optional[int] = None,
   1211 ):
   1212     if observed_callback:
-> 1213         total_tags = count_observe_decorators_in_module(observed_callback)
   1214         pbar_tags_id = add_pbar(
   1215             progress,
   1216             f"     ⚡ Invoking observed callback (#{count})",
   1217             total=total_tags,
   1218         )
   1220         # Call callback and extract trace

File ~/Projects/raid/code/eval/deepeval2/deepeval/evaluate/execute.py:1393, in count_observe_decorators_in_module(func)
   1391 mod = inspect.getmodule(func)
   1392 if mod is None or not hasattr(mod, "__file__"):
-> 1393     raise RuntimeError("Cannot locate @observe function.")
   1394 module_source = inspect.getsource(mod)
   1395 tree = ast.parse(module_source)

RuntimeError: Cannot locate @observe function.


**❗BEFORE YOU BEGIN❗**
Are you on discord? 🤗 We'd love to have you asking questions on discord instead: https://discord.com/invite/a3K9c8GRGt

**Describe the bug**
A clear and concise description of what the bug is.

**To Reproduce**
Steps to reproduce the behavior:
1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

**Expected behavior**
A clear and concise description of what you expected to happen.

**Screenshots**
If applicable, add screenshots to help explain your problem.

**Desktop (please complete the following information):**
 - OS: [e.g. iOS]
 - Browser [e.g. chrome, safari]
 - Version [e.g. 22]

**Smartphone (please complete the following information):**
 - Device: [e.g. iPhone6]
 - OS: [e.g. iOS8.1]
 - Browser [e.g. stock browser, safari]
 - Version [e.g. 22]

**Additional context**
Add any other context about the problem here.


Hey @jwu19 what version of deepeval is this? 

@penguine-ip deepeval version is 3.2.6, and llm model is azure-openai, gpt-4o