Repository: truera/trulens
Issue: [BUG] 2.3.0 Snowflake Cortex Analyst -- Error: (1304): Run Object: REDACTED does not exists for Object: 'CORTEX_ANALYST' [analyst_only].
Root cause: Validation Gap

**Bug Description**
I have a simple script based entirely on [this trulens cortex tutorial](https://quickstarts.snowflake.com/guide/getting_started_with_ai_observability/?index=..%2F..index#0). It simply takes the cortex search approach for rag in the article and adapts it to cortex analyst agent evaluation. The script is almost entirely working and it is easily confirmable via my logging that client authentication, database role permissions, dataset configuration and trulens workflows are all being configured, instantiated and executed correctly. Trulens logging, however, displays:

```
2025-09-02 07:02:33,742 [ERROR] Evaluator thread encountered an error: 'SnowflakeConnector' object has no attribute '_db'
2025-09-02 07:02:33,745 [DEBUG] Do not know how to jsonify an object of type '<class '__main__.CortexAnalystApp'>'.
2025-09-02 07:02:33,747 [DEBUG] Executing query: SELECT SYSTEM$AIML_RUN_OPERATION('GET', ?); with parameters {"object_name": "CORTEX_ANALYST", "object_type": "EXTERNAL AGENT", "run_name": "experiment_2025-09-02-110225", "object_version": "analyst_only"}
2025-09-02 07:02:33,966 [ERROR] Error executing query: SELECT SYSTEM$AIML_RUN_OPERATION('GET', ?);
Parameters: ('{"object_name": "CORTEX_ANALYST", "object_type": "EXTERNAL AGENT", "run_name": "experiment_2025-09-02-110225", "object_version": "analyst_only"}',)
Error: (1304): 01bec7b6-0417-2834-0067-63039050858a: 399900 (02000): 01bec7b6-0417-2834-0067-63039050858a: Run Object: 'experiment_2025-09-02-110225' does not exists for Object: 'CORTEX_ANALYST' [analyst_only].
```

Partial and Full debug logs are below.


**To Reproduce**
1. Follow [this tutorial](https://quickstarts.snowflake.com/guide/getting_started_with_ai_observability/?index=..%2F..index#0)
2. execute the script below updating the `username`, `password`, `role` and things of this nature. You will also need a test cortex analyst agent and semantic file at which to point, as well as a mock dataset and database for trulens to run against:

```
# perf_eval_cortex_analyst.py

import os
os.environ["TRULENS_OTEL_TRACING"] = "1"

import time
import json
import logging
import requests
import snowflake.connector
from typing import Any, Dict
from datetime import datetime
from dotenv import load_dotenv

from trulens.core.otel.instrument import instrument
from trulens.otel.semconv.trace import SpanAttributes
from trulens.apps.app import TruApp
from trulens.connectors.snowflake import SnowflakeConnector
from trulens.core.run import RunConfig

# =========================
# Environment & Constants
# =========================

load_dotenv()

ACCOUNT   = "$REDACTED.us-east-1"
USER      = "VARIANCE_USER"
PASSWORD  = os.environ["VARIANCE_USER_PASSWORD"]
ROLE      = "observability_user_role"
WAREHOUSE = "ANALYSIS_RBI"

# DATASET LOCATION (and TruLens store location)
DATA_DB       = "SANDBOX"
DATA_SCHEMA   = "JSTEINBERG"
DATASET_TABLE = "CORTEX_SQL_EVAL"
EVAL_TABLE_FQN = f"{DATA_DB}.{DATA_SCHEMA}.{DATASET_TABLE}"

STAGE     = "VARIANCE_04"
SEMANTIC_FILE = "variance.yaml"

TIMEOUT_SECONDS = 300
STATUS_POLL_SEC = 5

# Logging
ts = datetime.utcnow().strftime("%Y-%m-%d-%H%M%S")
os.makedirs("./logs", exist_ok=True)
LOG_FILE = f"./logs/variance_04_eval_{ts}.log"
logging.basicConfig(filename=LOG_FILE, level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")
logging.getLogger("trulens").setLevel(logging.DEBUG)
logging.getLogger("trulens_eval").setLevel(logging.DEBUG)
logger = logging.getLogger(__name__)

# =========================
# Snowflake Utilities
# =========================

def connect_snowflake() -> snowflake.connector.SnowflakeConnection:
    logger.info(
        "Connecting to Snowflake account=%s user=%s role=%s warehouse=%s dataset=%s.%s",
        ACCOUNT, USER, ROLE, WAREHOUSE, DATA_DB, DATA_SCHEMA
    )
    return snowflake.connector.connect(
        user=USER,
        password=PASSWORD,
        account=ACCOUNT,
        warehouse=WAREHOUSE,
        role=ROLE,
        database=DATA_DB,
        schema=DATA_SCHEMA,
    )

# =========================
# Helpers
# =========================

def extract_text(payload: Dict[str, Any]) -> str:
    try:
        msg = payload.get("message")
        if isinstance(msg, dict):
            content = msg.get("content")
            if isinstance(content, list):
                for part in content:
                    if isinstance(part, dict) and part.get("type") == "text" and "text" in part:
                        return str(part["text"])
        if "text" in payload and isinstance(payload["text"], str):
            return payload["text"]
        if payload.get("error") and isinstance(payload.get("text"), str):
            return f"ERROR: {payload['text']}"
        return json.dumps(payload, ensure_ascii=False, separators=(",", ":"))
    except Exception:
        return str(payload)

# =========================
# Cortex Analyst Client (Instrumented)
# =========================

@instrument(span_type=SpanAttributes.SpanType.GENERATION)
def call_cortex_analyst(conn: snowflake.connector.SnowflakeConnection, prompt: str) -> Dict[str, Any]:
    logger.info("Sending request to Cortex Analyst with prompt: %s", prompt)

    request_body = {
        "messages": [{"role": "user", "content": [{"type": "text", "text": prompt}]}],
        "semantic_model_file": f"@{DATA_DB}.{DATA_SCHEMA}.{STAGE}/{SEMANTIC_FILE}",
    }

    url = f"https://{ACCOUNT}.snowflakecomputing.com/api/v2/cortex/analyst/message"
    headers = {
        "Authorization": f'Snowflake Token="{conn.rest.token}"',
        "Content-Type": "application/json",
    }

    resp = requests.post(url, json=request_body, headers=headers, timeout=TIMEOUT_SECONDS)
    request_id = resp.headers.get("X-Snowflake-Request-Id")
    logger.info("Received response status=%s request_id=%s", resp.status_code, request_id)

    if resp.status_code >= 400:
        payload = {"error": True, "status_code": resp.status_code, "request_id": request_id, "text": resp.text}
        logger.error("Cortex Analyst error: %s", payload)
        return payload

    payload = resp.json()
    payload["request_id"] = request_id
    logger.info("Cortex Analyst success keys: %s", list(payload.keys()))
    return payload

class CortexAnalystApp:
    @instrument(
        span_type=SpanAttributes.SpanType.RECORD_ROOT,
        attributes={
            SpanAttributes.RECORD_ROOT.INPUT: "prompt",
            SpanAttributes.RECORD_ROOT.OUTPUT: "return",
        },
    )
    def query(self, prompt: str) -> str:
        logger.info("CortexAnalystApp.query invoked.")
        conn = connect_snowflake()
        try:
            payload = call_cortex_analyst(conn, prompt)
            try:
                logger.info("Raw payload: %s", json.dumps(payload)[:4000])
            except Exception:
                logger.info("Raw payload (non-JSON-serializable): %s", str(payload)[:4000])
            return extract_text(payload)
        finally:
            conn.close()
            logger.info("Closed Snowflake connection.")

# =========================
# TruLens App + Run
# =========================

app = CortexAnalystApp()

# TruLens Snowflake connector writes/reads its internal tables in SANDBOX.JSTEINBERG
tru_snowflake_connector = SnowflakeConnector(
    account=ACCOUNT,
    user=USER,
    password=PASSWORD,
    warehouse=WAREHOUSE,
    role=ROLE,
    database=DATA_DB,     # SANDBOX
    schema=DATA_SCHEMA,   # JSTEINBERG
)

try:
    tru_snowflake_connector.migrate_database()
    logger.info("TruLens DB migration on %s.%s completed.", DATA_DB, DATA_SCHEMA)
except Exception as e:
    logger.warning("DB migration skipped or not needed: %s", e)

tru_app = TruApp(
    app,
    app_name="cortex_analyst",
    app_version="analyst_only",
    connector=tru_snowflake_connector,
    main_method=app.query,
)

RUN_NAME = f"experiment_{ts}"

run_config = RunConfig(
    run_name=RUN_NAME,
    description="Cortex Analyst eval run",
    label="cortex_analyst_eval",
    source_type="TABLE",
    dataset_name=EVAL_TABLE_FQN,  # SANDBOX.JSTEINBERG.CORTEX_SQL_EVAL
    dataset_spec={
        "input": "PROMPT",
        "ground_truth_output": "EXPECTED_SQL",
    },
)

def wait_for_invocation_completion(run) -> None:
    terminal_ok = {"INVOCATION_COMPLETED", "INVOCATION_PARTIALLY_COMPLETED"}
    while True:
        status = run.get_status()
        logger.info("Run status: %s", status)
        if status in terminal_ok:
            return
        time.sleep(STATUS_POLL_SEC)

def main():
    try:
        old = tru_app.get_run(run_name=RUN_NAME)
        if old:
            logger.info("Existing run %s found; deleting.", RUN_NAME)
            old.delete()
    except Exception:
        pass

    logger.info("Creating TruLens run: %s", RUN_NAME)
    run = tru_app.add_run(run_config=run_config)

    try:
        run.start()
        logger.info("Run started: %s", RUN_NAME)
        wait_for_invocation_completion(run)

        metrics = ["answer_relevance"]  # start simple
        run.compute_metrics(metrics)
        logger.info("Metric computations enqueued: %s", metrics)

    except Exception as e:
        logger.exception("Run failed: %s", e)
        raise

if __name__ == "__main__":
    main()
```

**Expected Behavior**
I expect to see evaluations propagate to the Snowflake evaluation UI, but none ever do. There is simply always the screenshot below.

<img width="651" height="467" alt="Image" src="https://github.com/user-attachments/assets/130b50bb-d29d-4c7d-bdab-a60423787c05" />

**Relevant Logs/Tracebacks**
```
# CRITICAL LOG LINES
2025-09-02 07:02:25,171 [WARNING] Cannot explicitly set 'record_root' span type, setting to 'unknown'.
2025-09-02 07:02:33,263 [WARNING] Singleton instance TruSession already exists for name = None.
2025-09-02 07:02:33,740 [INFO] External Agent CORTEX_ANALYST with version analyst_only already exists.
2025-09-02 07:02:33,740 [DEBUG] app: instrumenting object at 0x119c7a540 of class CortexAnalystApp
2025-09-02 07:02:33,741 [DEBUG] skipping base <class '__main__.CortexAnalystApp'> because of class
2025-09-02 07:02:33,741 [DEBUG] skipping base <class 'object'> because of class
2025-09-02 07:02:33,741 [DEBUG] app: Do not know how to instrument object of type CortexAnalystApp.
2025-09-02 07:02:33,742 [INFO] Processing all events.
2025-09-02 07:02:33,742 [ERROR] Evaluator thread encountered an error: 'SnowflakeConnector' object has no attribute '_db'
2025-09-02 07:02:33,745 [DEBUG] Do not know how to jsonify an object of type '<class '__main__.CortexAnalystApp'>'.
2025-09-02 07:02:33,747 [DEBUG] Executing query: SELECT SYSTEM$AIML_RUN_OPERATION('GET', ?); with parameters {"object_name": "CORTEX_ANALYST", "object_type": "EXTERNAL AGENT", "run_name": "experiment_2025-09-02-110225", "object_version": "analyst_only"}
2025-09-02 07:02:33,966 [ERROR] Error executing query: SELECT SYSTEM$AIML_RUN_OPERATION('GET', ?);
Parameters: ('{"object_name": "CORTEX_ANALYST", "object_type": "EXTERNAL AGENT", "run_name": "experiment_2025-09-02-110225", "object_version": "analyst_only"}',)
Error: (1304): 01bec7b6-0417-2834-0067-63039050858a: 399900 (02000): 01bec7b6-0417-2834-0067-63039050858a: Run Object: 'experiment_2025-09-02-110225' does not exists for Object: 'CORTEX_ANALYST' [analyst_only].
Traceback (most recent call last):
  File "/Users/REDACTED/code/REDACTED/.venv/lib/python3.12/site-packages/trulens/connectors/snowflake/dao/sql_utils.py", line 55, in execute_query
    return session.sql(query, params=parameters).collect()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/REDACTED/code/REDACTED/.venv/lib/python3.12/site-packages/snowflake/snowpark/_internal/telemetry.py", line 295, in wrap
    result = func(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/REDACTED/code/REDACTED/.venv/lib/python3.12/site-packages/snowflake/snowpark/_internal/utils.py", line 1131, in call_wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/REDACTED/code/REDACTED/.venv/lib/python3.12/site-packages/snowflake/snowpark/dataframe.py", line 758, in collect
    return self._internal_collect_with_tag_no_telemetry(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/REDACTED/code/REDACTED/.venv/lib/python3.12/site-packages/snowflake/snowpark/dataframe.py", line 828, in _internal_collect_with_tag_no_telemetry
    return self._session._conn.execute(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/REDACTED/code/REDACTED/.venv/lib/python3.12/site-packages/snowflake/snowpark/_internal/server_connection.py", line 638, in execute
    result_set, result_meta = self.get_result_set(
                              ^^^^^^^^^^^^^^^^^^^^
  File "/Users/REDACTED/code/REDACTED/.venv/lib/python3.12/site-packages/snowflake/snowpark/_internal/analyzer/snowflake_plan.py", line 425, in wrap
    raise ne.with_traceback(tb) from None
  File "/Users/REDACTED/code/REDACTED/.venv/lib/python3.12/site-packages/snowflake/snowpark/_internal/analyzer/snowflake_plan.py", line 178, in wrap
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/REDACTED/code/REDACTED/.venv/lib/python3.12/site-packages/snowflake/snowpark/_internal/server_connection.py", line 760, in get_result_set
    result = self.run_query(
             ^^^^^^^^^^^^^^^
  File "/Users/REDACTED/code/REDACTED/.venv/lib/python3.12/site-packages/snowflake/snowpark/_internal/server_connection.py", line 136, in wrap
    raise ex
  File "/Users/REDACTED/code/REDACTED/.venv/lib/python3.12/site-packages/snowflake/snowpark/_internal/server_connection.py", line 130, in wrap
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/REDACTED/code/REDACTED/.venv/lib/python3.12/site-packages/snowflake/snowpark/_internal/server_connection.py", line 534, in run_query
    raise ex
  File "/Users/REDACTED/code/REDACTED/.venv/lib/python3.12/site-packages/snowflake/snowpark/_internal/server_connection.py", line 519, in run_query
    results_cursor = self.execute_and_notify_query_listener(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/REDACTED/code/REDACTED/.venv/lib/python3.12/site-packages/snowflake/snowpark/_internal/server_connection.py", line 449, in execute_and_notify_query_listener
    raise ex
  File "/Users/REDACTED/code/REDACTED/.venv/lib/python3.12/site-packages/snowflake/snowpark/_internal/server_connection.py", line 440, in execute_and_notify_query_listener
    results_cursor = self._cursor.execute(query, **kwargs)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/REDACTED/code/REDACTED/.venv/lib/python3.12/site-packages/snowflake/connector/cursor.py", line 1142, in execute
    Error.errorhandler_wrapper(self.connection, self, error_class, errvalue)
  File "/Users/REDACTED/code/REDACTED/.venv/lib/python3.12/site-packages/snowflake/connector/errors.py", line 286, in errorhandler_wrapper
    handed_over = Error.hand_to_other_handler(
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/REDACTED/code/REDACTED/.venv/lib/python3.12/site-packages/snowflake/connector/errors.py", line 341, in hand_to_other_handler
    cursor.errorhandler(connection, cursor, error_class, error_value)
  File "/Users/REDACTED/code/REDACTED/.venv/lib/python3.12/site-packages/snowflake/connector/errors.py", line 217, in default_errorhandler
    raise error_class(
snowflake.snowpark.exceptions.SnowparkSQLException: (1304): 01bec7b6-0417-2834-0067-63039050858a: 399900 (02000): 01bec7b6-0417-2834-0067-63039050858a: Run Object: 'experiment_2025-09-02-110225' does not exists for Object: 'CORTEX_ANALYST' [analyst_only].
2025-09-02 07:02:33,968 [INFO] Creating TruLens run: experiment_2025-09-02-110225
2025-09-02 07:02:33,968 [DEBUG] Executing query: SELECT SYSTEM$AIML_RUN_OPERATION('CREATE', ?); with parameters {"object_name": "CORTEX_ANALYST", "object_type": "EXTERNAL AGENT", "run_name": "experiment_2025-09-02-110225", "description": "Cortex Analyst eval run", "run_metadata": {"labels": ["cortex_analyst_eval"], "llm_judge_name": "llama3.1-70b"}, "source_info": {"name": "SANDBOX.JSTEINBERG.CORTEX_SQL_EVAL", "column_spec": {"input": "PROMPT", "ground_truth_output": "EXPECTED_SQL"}, "source_type": "TABLE"}, "object_version": "analyst_only"}
```

```
# FULL DEBUG LOG
2025-09-02 07:02:25,171 [WARNING] Cannot explicitly set 'record_root' span type, setting to 'unknown'.
2025-09-02 07:02:25,171 [INFO] Snowflake Connector for Python Version: 3.17.2, Python Version: 3.12.11, Platform: macOS-15.6-arm64-arm-64bit
2025-09-02 07:02:25,171 [INFO] Connecting to GLOBAL Snowflake domain
2025-09-02 07:02:25,197 [INFO] Found credentials in shared credentials file: ~/.aws/credentials
2025-09-02 07:02:27,672 [INFO] Snowpark Session information:
"version" : 1.37.0,
"python.version" : 3.12.11,
"python.connector.version" : 3.17.2,
"python.connector.session.id" : 29100789248616474,
"os.name" : Darwin

2025-09-02 07:02:28,146 [DEBUG] Creating new TruSession singleton instance for name = None.
2025-09-02 07:02:28,148 [INFO] ✅ OpenTelemetry exporter set: NoneType
2025-09-02 07:02:28,419 [DEBUG] Logging 1 for app:test_app version:None run:None
2025-09-02 07:02:28,420 [DEBUG] Writing spans to the protobuf file: /var/folders/nv/w9y8ntdd7433dgtzq6v_trrr0000gq/T/tmpeqlpmib3.pb
2025-09-02 07:02:28,420 [DEBUG] Spans written to the protobuf file: /var/folders/nv/w9y8ntdd7433dgtzq6v_trrr0000gq/T/tmpeqlpmib3.pb
2025-09-02 07:02:28,420 [DEBUG] Creating Snowflake stage if it does not exist
2025-09-02 07:02:28,649 [DEBUG] Uploading file /var/folders/nv/w9y8ntdd7433dgtzq6v_trrr0000gq/T/tmpeqlpmib3.pb to stage trulens_spans
2025-09-02 07:02:30,834 [DEBUG] Removing the temporary protobuf file
2025-09-02 07:02:30,835 [INFO] ✅ Added new TrulensOtelSpanProcessor
2025-09-02 07:02:30,853 [INFO] checked whether snowflake-snowpark-python is available: True
2025-09-02 07:02:33,257 [DEBUG] Nested optional imports context used. This context will be ignored.
2025-09-02 07:02:33,258 [DEBUG] Nested optional imports context used. This context will be ignored.
2025-09-02 07:02:33,263 [WARNING] Singleton instance TruSession already exists for name = None.
2025-09-02 07:02:33,269 [INFO] Initializing Snowflake External Agent DAO for app cortex_analyst version analyst_only
2025-09-02 07:02:33,269 [INFO] Initialized ExternalAgentDao with a Snowpark session.
2025-09-02 07:02:33,553 [INFO] Retrieved list of External Agents.
2025-09-02 07:02:33,554 [INFO] Checking if External Agent CORTEX_ANALYST exists.
2025-09-02 07:02:33,739 [INFO] Retrieved versions for External Agent CORTEX_ANALYST.
2025-09-02 07:02:33,740 [INFO] External Agent CORTEX_ANALYST with version analyst_only already exists.
2025-09-02 07:02:33,740 [DEBUG] app: instrumenting object at 0x119c7a540 of class CortexAnalystApp
2025-09-02 07:02:33,741 [DEBUG] skipping base <class '__main__.CortexAnalystApp'> because of class
2025-09-02 07:02:33,741 [DEBUG] skipping base <class 'object'> because of class
2025-09-02 07:02:33,741 [DEBUG] app: Do not know how to instrument object of type CortexAnalystApp.
2025-09-02 07:02:33,742 [INFO] Processing all events.
2025-09-02 07:02:33,742 [ERROR] Evaluator thread encountered an error: 'SnowflakeConnector' object has no attribute '_db'
2025-09-02 07:02:33,745 [DEBUG] Do not know how to jsonify an object of type '<class '__main__.CortexAnalystApp'>'.
2025-09-02 07:02:33,747 [DEBUG] Executing query: SELECT SYSTEM$AIML_RUN_OPERATION('GET', ?); with parameters {"object_name": "CORTEX_ANALYST", "object_type": "EXTERNAL AGENT", "run_name": "experiment_2025-09-02-110225", "object_version": "analyst_only"}
2025-09-02 07:02:33,966 [ERROR] Error executing query: SELECT SYSTEM$AIML_RUN_OPERATION('GET', ?);
Parameters: ('{"object_name": "CORTEX_ANALYST", "object_type": "EXTERNAL AGENT", "run_name": "experiment_2025-09-02-110225", "object_version": "analyst_only"}',)
Error: (1304): 01bec7b6-0417-2834-0067-63039050858a: 399900 (02000): 01bec7b6-0417-2834-0067-63039050858a: Run Object: 'experiment_2025-09-02-110225' does not exists for Object: 'CORTEX_ANALYST' [analyst_only].
Traceback (most recent call last):
  File "/Users/RBIJS069/code/rbi-variance-service/.venv/lib/python3.12/site-packages/trulens/connectors/snowflake/dao/sql_utils.py", line 55, in execute_query
    return session.sql(query, params=parameters).collect()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/RBIJS069/code/rbi-variance-service/.venv/lib/python3.12/site-packages/snowflake/snowpark/_internal/telemetry.py", line 295, in wrap
    result = func(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/RBIJS069/code/rbi-variance-service/.venv/lib/python3.12/site-packages/snowflake/snowpark/_internal/utils.py", line 1131, in call_wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/RBIJS069/code/rbi-variance-service/.venv/lib/python3.12/site-packages/snowflake/snowpark/dataframe.py", line 758, in collect
    return self._internal_collect_with_tag_no_telemetry(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/RBIJS069/code/rbi-variance-service/.venv/lib/python3.12/site-packages/snowflake/snowpark/dataframe.py", line 828, in _internal_collect_with_tag_no_telemetry
    return self._session._conn.execute(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/RBIJS069/code/rbi-variance-service/.venv/lib/python3.12/site-packages/snowflake/snowpark/_internal/server_connection.py", line 638, in execute
    result_set, result_meta = self.get_result_set(
                              ^^^^^^^^^^^^^^^^^^^^
  File "/Users/RBIJS069/code/rbi-variance-service/.venv/lib/python3.12/site-packages/snowflake/snowpark/_internal/analyzer/snowflake_plan.py", line 425, in wrap
    raise ne.with_traceback(tb) from None
  File "/Users/RBIJS069/code/rbi-variance-service/.venv/lib/python3.12/site-packages/snowflake/snowpark/_internal/analyzer/snowflake_plan.py", line 178, in wrap
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/RBIJS069/code/rbi-variance-service/.venv/lib/python3.12/site-packages/snowflake/snowpark/_internal/server_connection.py", line 760, in get_result_set
    result = self.run_query(
             ^^^^^^^^^^^^^^^
  File "/Users/RBIJS069/code/rbi-variance-service/.venv/lib/python3.12/site-packages/snowflake/snowpark/_internal/server_connection.py", line 136, in wrap
    raise ex
  File "/Users/RBIJS069/code/rbi-variance-service/.venv/lib/python3.12/site-packages/snowflake/snowpark/_internal/server_connection.py", line 130, in wrap
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/RBIJS069/code/rbi-variance-service/.venv/lib/python3.12/site-packages/snowflake/snowpark/_internal/server_connection.py", line 534, in run_query
    raise ex
  File "/Users/RBIJS069/code/rbi-variance-service/.venv/lib/python3.12/site-packages/snowflake/snowpark/_internal/server_connection.py", line 519, in run_query
    results_cursor = self.execute_and_notify_query_listener(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/RBIJS069/code/rbi-variance-service/.venv/lib/python3.12/site-packages/snowflake/snowpark/_internal/server_connection.py", line 449, in execute_and_notify_query_listener
    raise ex
  File "/Users/RBIJS069/code/rbi-variance-service/.venv/lib/python3.12/site-packages/snowflake/snowpark/_internal/server_connection.py", line 440, in execute_and_notify_query_listener
    results_cursor = self._cursor.execute(query, **kwargs)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/RBIJS069/code/rbi-variance-service/.venv/lib/python3.12/site-packages/snowflake/connector/cursor.py", line 1142, in execute
    Error.errorhandler_wrapper(self.connection, self, error_class, errvalue)
  File "/Users/RBIJS069/code/rbi-variance-service/.venv/lib/python3.12/site-packages/snowflake/connector/errors.py", line 286, in errorhandler_wrapper
    handed_over = Error.hand_to_other_handler(
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/RBIJS069/code/rbi-variance-service/.venv/lib/python3.12/site-packages/snowflake/connector/errors.py", line 341, in hand_to_other_handler
    cursor.errorhandler(connection, cursor, error_class, error_value)
  File "/Users/RBIJS069/code/rbi-variance-service/.venv/lib/python3.12/site-packages/snowflake/connector/errors.py", line 217, in default_errorhandler
    raise error_class(
snowflake.snowpark.exceptions.SnowparkSQLException: (1304): 01bec7b6-0417-2834-0067-63039050858a: 399900 (02000): 01bec7b6-0417-2834-0067-63039050858a: Run Object: 'experiment_2025-09-02-110225' does not exists for Object: 'CORTEX_ANALYST' [analyst_only].
2025-09-02 07:02:33,968 [INFO] Creating TruLens run: experiment_2025-09-02-110225
2025-09-02 07:02:33,968 [DEBUG] Executing query: SELECT SYSTEM$AIML_RUN_OPERATION('CREATE', ?); with parameters {"object_name": "CORTEX_ANALYST", "object_type": "EXTERNAL AGENT", "run_name": "experiment_2025-09-02-110225", "description": "Cortex Analyst eval run", "run_metadata": {"labels": ["cortex_analyst_eval"], "llm_judge_name": "llama3.1-70b"}, "source_info": {"name": "SANDBOX.JSTEINBERG.CORTEX_SQL_EVAL", "column_spec": {"input": "PROMPT", "ground_truth_output": "EXPECTED_SQL"}, "source_type": "TABLE"}, "object_version": "analyst_only"}
2025-09-02 07:02:34,225 [INFO] Created new RunMetadata successfully for run 'experiment_2025-09-02-110225'.
2025-09-02 07:02:34,226 [DEBUG] Executing query: SELECT SYSTEM$AIML_RUN_OPERATION('GET', ?); with parameters {"object_name": "CORTEX_ANALYST", "object_type": "EXTERNAL AGENT", "run_name": "experiment_2025-09-02-110225", "object_version": "analyst_only"}
2025-09-02 07:02:34,461 [DEBUG] Executing query: SELECT SYSTEM$AIML_RUN_OPERATION('GET', ?); with parameters {"object_name": "CORTEX_ANALYST", "object_type": "External Agent", "run_name": "experiment_2025-09-02-110225", "object_version": "analyst_only"}
2025-09-02 07:02:34,690 [INFO] Run is created, no invocation nor computation yet.
2025-09-02 07:02:34,691 [INFO] Current run status: RunStatus.CREATED
2025-09-02 07:02:34,691 [DEBUG] Executing query: SELECT SYSTEM$AIML_RUN_OPERATION('GET', ?); with parameters {"object_name": "CORTEX_ANALYST", "object_type": "External Agent", "run_name": "experiment_2025-09-02-110225", "object_version": "analyst_only"}
2025-09-02 07:02:34,888 [INFO] Run is created, no invocation nor computation yet.
2025-09-02 07:02:34,888 [INFO] No input dataframe provided. Fetching input data from source.
2025-09-02 07:02:35,080 [INFO] Creating or updating invocation metadata with 1 records from input.
2025-09-02 07:02:35,082 [INFO] CortexAnalystApp.query invoked.
2025-09-02 07:02:35,082 [INFO] Connecting to Snowflake account=kwa28481.us-east-1 user=VARIANCE_USER role=observability_user_role warehouse=ANALYSIS_RBI dataset=SANDBOX.JSTEINBERG
2025-09-02 07:02:35,082 [INFO] Snowflake Connector for Python Version: 3.17.2, Python Version: 3.12.11, Platform: macOS-15.6-arm64-arm-64bit
2025-09-02 07:02:35,083 [INFO] Connecting to GLOBAL Snowflake domain
2025-09-02 07:02:37,322 [INFO] Sending request to Cortex Analyst with prompt: Which restaurants/stores (profit centers) under BK Americas had the largest variance to forecast (planning cost center) in February 2025 for royalties (410000)?
2025-09-02 07:02:48,412 [INFO] Received response status=200 request_id=4e8fbd87-ea37-44a6-9f46-f0190367815e
2025-09-02 07:02:48,413 [INFO] Cortex Analyst success keys: ['message', 'request_id', 'warnings', 'semantic_model_selection', 'response_metadata']
2025-09-02 07:02:48,415 [INFO] Raw payload: {"message": {"role": "analyst", "content": [{"type": "text", "text": "This is our interpretation of your question:\n\nWhich REDACTED...REDACTED AS a LEFT JOIN budget AS b ON a.profit_center = b.profit_center AND a.account = b.account AND a.entity = b.entity AND a.time = b.time LEFT JOIN metadata AS m ON a.profit_center = m.profit_center) SELECT profit_center_label, parent_node, account, time, SUM(actual_amount) AS total_actual, SUM(budget_amount) AS total_budget, SUM(COALESCE(actual_amount, 0) - COALESCE(budget_amount, 0)) AS variance FROM joined GROUP BY profit_center_label, parent_node, account, time ORDER BY time, profit_center_label, account", "verified_at": 0, "verified_by": ""}}}]}, "request_id": "4e8fbd87-ea37-44a6-9f46-f0190367815e", "warnings": [{"message": "Verified query 'ud1_actuals_rollup_by_hierarchy' referred to physical tables ['sandbox.jsteinberg.ud1_full_rollup_with_level', 'brand_all.onestream.stores_generi
2025-09-02 07:02:48,660 [INFO] Closed Snowflake connection.
2025-09-02 07:02:48,661 [DEBUG] Exiting the OTEL app context.
2025-09-02 07:02:48,662 [DEBUG] Logging 2 for app:cortex_analyst version:analyst_only run:experiment_2025-09-02-110225
2025-09-02 07:02:48,663 [DEBUG] Writing spans to the protobuf file: /var/folders/nv/w9y8ntdd7433dgtzq6v_trrr0000gq/T/tmpljbc9vxh.pb
2025-09-02 07:02:48,663 [DEBUG] Spans written to the protobuf file: /var/folders/nv/w9y8ntdd7433dgtzq6v_trrr0000gq/T/tmpljbc9vxh.pb
2025-09-02 07:02:48,664 [DEBUG] Creating Snowflake stage if it does not exist
2025-09-02 07:02:48,888 [DEBUG] Uploading file /var/folders/nv/w9y8ntdd7433dgtzq6v_trrr0000gq/T/tmpljbc9vxh.pb to stage trulens_spans
2025-09-02 07:03:02,122 [DEBUG] Removing the temporary protobuf file
2025-09-02 07:03:02,123 [INFO] Flushed all spans for the run experiment_2025-09-02-110225; and exported them to the telemetry pipeline.
2025-09-02 07:03:02,124 [INFO] Executing SQL query to setup ingestion: <snowflake.snowpark.dataframe.DataFrame object at 0x139dd2f30>
2025-09-02 07:03:08,044 [DEBUG] Function executed successfully.
2025-09-02 07:03:08,044 [INFO] Run started, invocation done and ingestion in process.
2025-09-02 07:03:08,044 [INFO] Run started: experiment_2025-09-02-110225
2025-09-02 07:03:08,044 [DEBUG] Executing query: SELECT SYSTEM$AIML_RUN_OPERATION('GET', ?); with parameters {"object_name": "CORTEX_ANALYST", "object_type": "External Agent", "run_name": "experiment_2025-09-02-110225", "object_version": "analyst_only"}
2025-09-02 07:03:08,236 [INFO] Run is created, no invocation nor computation yet.
2025-09-02 07:03:08,236 [INFO] Run status: RunStatus.CREATED
2025-09-02 07:03:13,239 [DEBUG] Executing query: SELECT SYSTEM$AIML_RUN_OPERATION('GET', ?); with parameters {"object_name": "CORTEX_ANALYST", "object_type": "External Agent", "run_name": "experiment_2025-09-02-110225", "object_version": "analyst_only"}
2025-09-02 07:03:13,427 [INFO] Run is created, no invocation nor computation yet.
2025-09-02 07:03:13,427 [INFO] Run status: RunStatus.CREATED
2025-09-02 07:03:18,432 [DEBUG] Executing query: SELECT SYSTEM$AIML_RUN_OPERATION('GET', ?); with parameters {"object_name": "CORTEX_ANALYST", "object_type": "External Agent", "run_name": "experiment_2025-09-02-110225", "object_version": "analyst_only"}
2025-09-02 07:03:18,665 [INFO] Run is created, invocation started.
2025-09-02 07:03:18,666 [INFO] Run status: RunStatus.INVOCATION_IN_PROGRESS
2025-09-02 07:03:23,669 [DEBUG] Executing query: SELECT SYSTEM$AIML_RUN_OPERATION('GET', ?); with parameters {"object_name": "CORTEX_ANALYST", "object_type": "External Agent", "run_name": "experiment_2025-09-02-110225", "object_version": "analyst_only"}
2025-09-02 07:03:24,018 [INFO] Run is created, invocation started.
2025-09-02 07:03:24,018 [INFO] Run status: RunStatus.INVOCATION_COMPLETED
2025-09-02 07:03:24,018 [DEBUG] Executing query: SELECT SYSTEM$AIML_RUN_OPERATION('GET', ?); with parameters {"object_name": "CORTEX_ANALYST", "object_type": "External Agent", "run_name": "experiment_2025-09-02-110225", "object_version": "analyst_only"}
2025-09-02 07:03:24,203 [INFO] Run is created, invocation started.
2025-09-02 07:03:24,204 [INFO] Current run status: RunStatus.INVOCATION_COMPLETED
2025-09-02 07:03:24,204 [DEBUG] Executing query: SELECT SYSTEM$AIML_RUN_OPERATION('GET', ?); with parameters {"object_name": "CORTEX_ANALYST", "object_type": "External Agent", "run_name": "experiment_2025-09-02-110225", "object_version": "analyst_only"}
2025-09-02 07:03:24,549 [INFO] Run is created, invocation started.
2025-09-02 07:03:24,549 [DEBUG] Executing query: SELECT SYSTEM$AIML_RUN_OPERATION('GET', ?); with parameters {"object_name": "CORTEX_ANALYST", "object_type": "External Agent", "run_name": "experiment_2025-09-02-110225", "object_version": "analyst_only"}
2025-09-02 07:03:24,809 [INFO] Metrics to compute: ['answer_relevance'].
2025-09-02 07:03:24,810 [INFO] Executing SQL command for metrics computation: <snowflake.snowpark.dataframe.DataFrame object at 0x11c3ff230>
2025-09-02 07:03:24,967 [INFO] Metrics computation job started
2025-09-02 07:03:24,968 [INFO] Metric computations enqueued: ['answer_relevance']
2025-09-02 07:03:24,970 [INFO] Closing session: 29100789248616474
2025-09-02 07:03:24,970 [INFO] Canceling all running queries
2025-09-02 07:03:25,371 [INFO] Closed session: 29100789248616474
2025-09-02 07:03:25,681 [INFO] Stopped evaluator thread (app_name=cortex_analyst, app_version=analyst_only).
```

```
# PIP FREEZE
# all trulens on 2.3.0

absl-py==1.4.0
aiobotocore==2.24.1
aiohappyeyeballs==2.6.1
aiohttp==3.12.15
aioitertools==0.12.0
aiosignal==1.4.0
alembic==1.16.5
altair==5.5.0
annotated-types==0.7.0
anyio==4.10.0
appnope==0.1.4
argon2-cffi==25.1.0
argon2-cffi-bindings==25.1.0
arrow==1.3.0
asn1crypto==1.5.1
asttokens==3.0.0
async-lru==2.0.5
attrs==25.3.0
babel==2.17.0
beautifulsoup4==4.13.5
bleach==6.2.0
blinker==1.9.0
boto3==1.39.11
botocore==1.39.11
cachetools==5.5.2
certifi==2025.8.3
cffi==1.17.1
charset-normalizer==3.4.3
click==8.2.1
cloudpickle==3.0.0
comm==0.2.3
cryptography==45.0.6
debugpy==1.8.16
decorator==5.2.1
defusedxml==0.7.1
dill==0.3.9
executing==2.2.0
fastjsonschema==2.21.2
filelock==3.19.1
fqdn==1.5.1
frozenlist==1.7.0
fsspec==2025.7.0
gitdb==4.0.12
GitPython==3.1.45
h11==0.16.0
httpcore==1.0.9
httpx==0.28.1
idna==3.10
importlib_metadata==8.7.0
importlib_resources==6.5.2
ipykernel==6.30.1
ipython==9.5.0
ipython_pygments_lexers==1.1.1
ipywidgets==8.1.7
isoduration==20.11.0
jedi==0.19.2
Jinja2==3.1.6
jmespath==1.0.1
joblib==1.5.2
json5==0.12.1
jsonpointer==3.0.0
jsonschema==4.25.1
jsonschema-specifications==2025.4.1
jupyter==1.1.1
jupyter-console==6.6.3
jupyter-events==0.12.0
jupyter-lsp==2.3.0
jupyter_client==8.6.3
jupyter_core==5.8.1
jupyter_server==2.17.0
jupyter_server_terminals==0.5.3
jupyterlab==4.4.6
jupyterlab_pygments==0.3.0
jupyterlab_server==2.27.3
jupyterlab_widgets==3.0.15
lark==1.2.2
llvmlite==0.44.0
Mako==1.3.10
markdown-it-py==4.0.0
MarkupSafe==3.0.2
matplotlib-inline==0.1.7
mdurl==0.1.2
mistune==3.1.4
multidict==6.6.4
munch==2.5.0
narwhals==2.2.0
nbclient==0.10.2
nbconvert==7.16.6
nbformat==5.10.4
nest-asyncio==1.6.0
nltk==3.9.1
notebook==7.4.5
notebook_shim==0.2.4
numba==0.61.2
numpy==2.2.6
opentelemetry-api==1.36.0
opentelemetry-proto==1.36.0
opentelemetry-sdk==1.36.0
opentelemetry-semantic-conventions==0.57b0
packaging==24.2
pandas==2.3.2
pandocfilters==1.5.1
parso==0.8.5
pexpect==4.9.0
pillow==11.3.0
platformdirs==4.4.0
plotly==5.24.1
prometheus_client==0.22.1
prompt_toolkit==3.0.52
propcache==0.3.2
protobuf==6.31.1
psutil==5.9.8
ptyprocess==0.7.0
pure_eval==0.2.3
pyarrow==21.0.0
pycparser==2.22
pydantic==2.11.7
pydantic_core==2.33.2
pydeck==0.9.1
Pygments==2.19.2
PyJWT==2.10.1
pyOpenSSL==25.1.0
python-dateutil==2.9.0.post0
python-dotenv==1.1.1
python-json-logger==3.3.0
pytimeparse==1.1.8
pytz==2025.2
PyYAML==6.0.2
pyzmq==27.0.2
referencing==0.36.2
regex==2025.7.34
requests==2.32.5
retrying==1.4.2
rfc3339-validator==0.1.4
rfc3986-validator==0.1.1
rfc3987-syntax==1.1.0
rich==13.9.4
rpds-py==0.27.1
s3fs==2025.7.0
s3transfer==0.13.1
scikit-learn==1.5.2
scipy==1.16.1
Send2Trash==1.8.3
setuptools==80.9.0
shap==0.48.0
six==1.17.0
slicer==0.0.8
smmap==5.0.2
sniffio==1.3.1
snowflake-connector-python==3.17.2
snowflake-ml-python==1.11.0
snowflake-snowpark-python==1.37.0
snowflake-sqlalchemy==1.7.6
snowflake.core==1.7.0
sortedcontainers==2.4.0
soupsieve==2.8
SQLAlchemy==2.0.43
sqlparse==0.5.3
stack-data==0.6.3
streamlit==1.49.0
tenacity==9.1.2
terminado==0.18.1
threadpoolctl==3.6.0
tinycss2==1.4.0
toml==0.10.2
tomlkit==0.13.3
tornado==6.5.2
tqdm==4.67.1
traitlets==5.14.3
trulens-connectors-snowflake==2.3.0
trulens-core==2.3.0
trulens-dashboard==2.3.0
trulens-feedback==2.3.0
trulens-otel-semconv==2.3.0
trulens-providers-cortex==2.3.0
trulens_eval==2.3.0
types-python-dateutil==2.9.0.20250822
typing-inspection==0.4.1
typing_extensions==4.15.0
tzdata==2025.2
tzlocal==5.3.1
uri-template==1.3.0
urllib3==2.5.0
wcwidth==0.2.13
webcolors==24.11.1
webencodings==0.5.1
websocket-client==1.8.0
wheel==0.45.1
widgetsnbextension==4.0.14
wrapt==1.17.3
xgboost==2.1.4
yarl==1.20.1
zipp==3.23.0
```

```
# UNSURPRESSED STDOUT FROM SCRIPT EXECUTION
/Users/REDACTED/code/REDACTED/.venv/lib/python3.12/site-packages/munch/__init__.py:24: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
  import pkg_resources
/Users/REDACTED/code/REDACTED/eval/main.12.py:46: DeprecationWarning: datetime.datetime.utcnow() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.now(datetime.UTC).
  ts = datetime.utcnow().strftime("%Y-%m-%d-%H%M%S")
✅ experimental Feature.OTEL_TRACING enabled.
🔒 experimental Feature.OTEL_TRACING is enabled and cannot be changed.
```

```
# database created for storing traces and evaluations
create or replace TABLE SANDBOX.JSTEINBERG.CORTEX_SQL_EVAL (
	PROMPT VARCHAR(16777216),
	EXPECTED_SQL VARCHAR(16777216)
);
```

**Environment:**

- OS: MacOS 15.6 Apple M2 Pro
- Python 3.12.11
- TruLens version 2.3.0
- Versions of other relevant installed libraries: pip freeze given above

**Additional Context**
I've attached `OBSERVABILITY_USER_ROLE` perms csv below. Basically it owns almost everything, for testing purposes.

[2025-08-29 1_42am.csv](https://github.com/user-attachments/files/22093366/2025-08-29.1_42am.csv)

I've also attached roles granted to `VARIANCE_USER`.

[2025-08-29 1_42am (1).csv](https://github.com/user-attachments/files/22093377/2025-08-29.1_42am.1.csv)

Authentication and execution to the Cortex Agent does succeed without issue:

```
# this is the role connecting to the cortex agent and although the cortex agent response is heavily redacted for proprietary reasons, all of this completes successfully, obviously: 

2025-09-02 07:02:48,413 [INFO] Cortex Analyst success keys: ['message', 'request_id', 'warnings', 'semantic_model_selection', 'response_metadata']
2025-09-02 07:02:48,415 [INFO] Raw payload: {"message": {"role": "analyst", "content": [{"type": "text", "text": "This is our interpretation of your question:\n\nWhich REDACTED...REDACTED AS a LEFT JOIN budget AS b ON a.profit_center = b.profit_center AND a.account = b.account AND a.entity = b.entity AND a.time = b.time LEFT JOIN metadata AS m ON a.profit_center = m.profit_center) SELECT profit_center_label, parent_node, account, time, SUM(actual_amount) AS total_actual, SUM(budget_amount) AS total_budget, SUM(COALESCE(actual_amount, 0) - COALESCE(budget_amount, 0)) AS variance FROM joined GROUP BY profit_center_label, parent_node, account, time ORDER BY time, profit_center_label, account", "verified_at": 0, "verified_by": ""}}}]}, "request_id": "4e8fbd87-ea37-44a6-9f46-f0190367815e", "warnings": [{"message": "Verified query 'ud1_actuals_rollup_by_hierarchy' referred to physical tables ['sandbox.jsteinberg.ud1_full_rollup_with_level', 'brand_all.onestream.stores_generi
```

@sfc-gh-nvytla any insight here? I'm hoping there is something obvious in all the output I've given.

Hey @jsteinberg-rbi - I tested your repro script with this [analyst quickstart](https://quickstarts.snowflake.com/guide/getting_started_with_cortex_analyst/index.html#3) and was able to see the app and traces pop up in snowflake UI.

<img width="1627" height="384" alt="Image" src="https://github.com/user-attachments/assets/a83373e5-9a74-4b9f-ae5f-e1f9fcff3cff" />

Can you try narrowing the repro script to isolate the problem, so I can better see what your issue might be?

@sfc-gh-jreini I've now run through the quickstart you linked to above. Would you mind sharing your version of my repro script so I can get it working against the quickstart above, as well?

> [@sfc-gh-jreini](https://github.com/sfc-gh-jreini) I've now run through the quickstart you linked to above. Would you mind sharing your version of my repro script so I can get it working against the quickstart above, as well?

Hey @jsteinberg-rbi - it was the same as yours except filling in the keys/credentials where needed

@sfc-gh-jreini Is RunConfig even the correct way to do things still? I don't see it anywhere in the API documentation though it is in core source:

```python
run_config = RunConfig(
    run_name=RUN_NAME,
    description="Cortex Analyst eval run",
    label="cortex_analyst_eval",
    source_type="TABLE",
    dataset_name=EVAL_TABLE_FQN,  # SANDBOX.JSTEINBERG.CORTEX_SQL_EVAL
    dataset_spec={
        "input": "PROMPT",
        "ground_truth_output": "EXPECTED_SQL",
    },
)
```

Same question for `compute_metrics`:


```python
run.compute_metrics(metrics)
``` 

So for example in this [snowflake engineer's quickstart ](https://github.com/sfc-gh-ebotwick/cortex_analyst_with_observability/blob/main/cortex_analyst_with_observability.py) he doesn't use this `RunConfig --> run.start() --> run.compute_metrics()` at all. Why? He uses Feedbacks which isn't necessarily tightly coupled to my question, but I'm very confused about what the right approach is? I need to compute all the evaluation metrics computed in the quickstart I just linked to, but in his case he just runs `relevance` of that generated SQL as it pertains to the _user's input_ whereas I need to run trulens in a loop over rows of a database providing a ground truth of questions and expected sql. But I also need to compute other metrics as well. So what is the right sort of "take off" approach to do this? Should I be using `RunConfig --> run.start() --> run.compute_metrics()` or something else? It's weird to me that there is a `compute_metrics` call which seems to work for "stock" "pre-selected" metrics like `"answer_relevance"` or `"coherence"` [as is demonstrated in this other quickstart](https://quickstarts.snowflake.com/guide/getting_started_with_ai_observability/index.html?index=..%2F..index#6) but then in the quickstart I linked to above this approach isn't used at all and metrics are laid out via `Feedback`s. So I'm confused as to what the right approach is.

> [@sfc-gh-jreini](https://github.com/sfc-gh-jreini) Is RunConfig even the correct way to do things still? I don't see it anywhere in the API documentation though it is in core source:
> 
> run_config = RunConfig(
>     run_name=RUN_NAME,
>     description="Cortex Analyst eval run",
>     label="cortex_analyst_eval",
>     source_type="TABLE",
>     dataset_name=EVAL_TABLE_FQN,  # SANDBOX.JSTEINBERG.CORTEX_SQL_EVAL
>     dataset_spec={
>         "input": "PROMPT",
>         "ground_truth_output": "EXPECTED_SQL",
>     },
> )
> Same question for `compute_metrics`:
> 
> run.compute_metrics(metrics)
> So for example in this [snowflake engineer's quickstart ](https://github.com/sfc-gh-ebotwick/cortex_analyst_with_observability/blob/main/cortex_analyst_with_observability.py) he doesn't use this `RunConfig --> run.start() --> run.compute_metrics()` at all. Why? He uses Feedbacks which isn't necessarily tightly coupled to my question, but I'm very confused about what the right approach is? I need to compute all the evaluation metrics computed in the quickstart I just linked to, but in his case he just runs `relevance` of that generated SQL as it pertains to the _user's input_ whereas I need to run trulens in a loop over rows of a database providing a ground truth of questions and expected sql. But I also need to compute other metrics as well. So what is the right sort of "take off" approach to do this? Should I be using `RunConfig --> run.start() --> run.compute_metrics()` or something else? It's weird to me that there is a `compute_metrics` call which seems to work for "stock" "pre-selected" metrics like `"answer_relevance"` or `"coherence"` [as is demonstrated in this other quickstart](https://quickstarts.snowflake.com/guide/getting_started_with_ai_observability/index.html?index=..%2F..index#6) but then in the quickstart I linked to above this approach isn't used at all and metrics are laid out via `Feedback`s. So I'm confused as to what the right approach is.

This is the correct way to do things working with Snowflake AI Observability, following the `RunConfig --> run.start() --> run.compute_metrics()` flow. This is a batch-centric way of doing things, the traces and evals are managed in Snowflake, and the UI is available in Snowsight.

TruLens OSS has more flexibility in the way you can trace and monitor ai apps, run a larger set of evals/feedback functions (including custom functions) and so on. Using TruLens OSS, the logs can be stored in any sqlalchemy-compatible database of your choice, evals can be computed with any llm of your choice, and the UI is available as a local streamlit dashboard.

Totally your choice if you'd like to use the Snowflake-centric route or the TruLens oss route. Does that make sense?




@sfc-gh-jreini I think so? Are you implying that calling Feedbacks() is the more OSS approach? Right because say for example I want to use `cortex.Cortex.groundedness_measure_with_cot_reasons` in my "Snowflake-managed" approach? My sense is I'm, not able to pass that as a metric to `compute_metrics`?

Also, should I be worried about the below? I ran probably 50 different instantiations of my script and it says that at some point `TruLens` created tables finally in my database.schema on August 29th. This is the first time I'm seeing these. I'm logged in as an `ACCOUNTADMIN` and my role has `ACCOUNTADMIN` and yet when I go to look at these tables' data preview in snowflake I get the error message in the image below. Is there some feature or something I'm missing on my tenant? Again I am an account admin and my granted role assigned to my executing snowflake user in the script has account admin assigned as default so I'm baffled as to why I don't see Trulens tables in my _new test schema path_ fairly soon (I wouldn't think batching should take hours for a single inference on a single row)?

```
2025-09-09 17:29:01,094 [INFO] Run is created, invocation started.
2025-09-09 17:29:01,094 [DEBUG] Executing query: SELECT SYSTEM$AIML_RUN_OPERATION('GET', ?); with parameters {"object_name": "CORTEX_ANALYST", "object_type": "External Agent", "run_name": "experiment_2025-09-09-212814", "object_version": "analyst_only"}
2025-09-09 17:29:01,263 [INFO] Metrics to compute: ['answer_relevance'].
2025-09-09 17:29:01,264 [INFO] Executing SQL command for metrics computation: <snowflake.snowpark.dataframe.DataFrame object at 0x12afd8200>
2025-09-09 17:29:01,378 [INFO] Metrics computation job started
2025-09-09 17:29:01,378 [INFO] Metric computations enqueued: ['answer_relevance']
2025-09-09 17:29:01,380 [INFO] Closing session: 29100789254026314
2025-09-09 17:29:01,381 [INFO] Canceling all running queries
2025-09-09 17:29:01,624 [INFO] Closed session: 29100789254026314
2025-09-09 17:29:01,985 [INFO] Stopped evaluator thread (app_name=cortex_analyst, app_version=analyst_only).
```

<img width="521" height="401" alt="Image" src="https://github.com/user-attachments/assets/8531023a-0411-4dfa-a6f8-38f5fe52c77a" />

1. Yes - using `Feedback` is only available in the OSS approach. In the snowflake-managed approach, you can only use the available 5 metrics via run.compute_metrics(). Good news for your question is that `cortex.Cortex.groundedness_measure_with_cot_reasons` is approximately the same metric run by `run.compute_metrics(['groundedness'])`

2. You will not see tables in your snowflake schema following the Snowflake-managed approach. The traces and evals are stored in event tables. The issue here seems to be you are mixing the OSS and snowflake code.

Since you are connected with the Snowflake team working to resolve this issue, closing it on GitHub here.