Repository: evidentlyai/evidently
Issue: Issue: CustomValueTest Prevents Test Suite from Appearing in Evidently UI
Root cause: Validation Gap

When running CustomValueTest within a test suite in a Jupyter Notebook, everything works as expected—no errors occur, and the test suite executes correctly:

<img width="1652" alt="Image" src="https://github.com/user-attachments/assets/aed786c8-9a59-49ff-9461-a3257cd4e856" />


However, when adding the same test suite to my project workspace, the suite is successfully saved to the workspace directory, but it does not appear in the Evidently UI under the Test Suite tab.

Function for creating test suite:
```
def create_test_suite(df, reference_df, column_mapping, report_date):
    print(f"🔹 Generating test suite for {report_date}")
    
    test = [
        TestValueMAE(lt=1),
        TestValueRMSE(lt=2),
        CustomValueTest(func=accuracy_metric_ideal_day, title="Ideal Day Test", gt=.49),
        CustomValueTest(func=accuracy_metric_ideal_day_or_1early, title="Ideal Day or 1 Early Test", gt=.59),
        CustomValueTest(func=accuracy_metric_ideal_day_or_earlier, title="Ideal Day or Earlier Test", gt=.74),
        CustomValueTest(func=accuracy_metric_all_lates, title="All Lates Test", lt=.10)
    ]

    # Filter data for this specific date
    daily_df = df[df["inference_date_utc"] == report_date]

    if daily_df.empty:
        print(f"⚠️ No data for {report_date}, skipping...")
        return None  # No report generated

    # Create and run the test suite
    data_quality_test = TestSuite(tests=test, timestamp=pd.Timestamp(report_date), tags=['test-suite'], options=[SOLARIZED_COLOR_OPTIONS])
    data_quality_test.run(reference_data=reference_df, current_data=daily_df, column_mapping=column_mapping)

    print(f"✅ Test saved for {report_date}")
    return data_quality_test
```

Function for creating project:
```
def create_or_update_project(workspace, project=None):
    if project is None:
        project = workspace.create_project(PROJECT_NAME)
        project.description = PROJECT_DESCRIPTION
    else:
        print(f"Updating existing project: {project.name} (ID: {project.id})")
        
    # Clear existing panels to avoid duplicates
    project.dashboard.panels.clear()
    
    # Add a Title panel for the overall dashboard
    project.dashboard.add_panel(
        DashboardPanelCounter(
            title="Shipment Arrival Estimate (SAE) Model Performance Monitoring",
            filter=ReportFilter(metadata_values={}, tag_values=[]),
            agg=CounterAgg.NONE,
        ),
    )

    project.dashboard.add_panel(
        DashboardPanelTestSuiteCounter(
            title="Test Suite Successes",
            statuses=[TestStatus.SUCCESS],
            size=1,
        ),
    )
    
    project.dashboard.add_panel(
        DashboardPanelTestSuiteCounter(
            title="Test Suite Failures/Errors",
            statuses=[TestStatus.FAIL, TestStatus.ERROR],
            size=1,
        ),
    )
    
    project.dashboard.add_panel(
        DashboardPanelTestSuite(
            title="Failed/Error Test Over Time",
            filter=ReportFilter(metadata_values={}, tag_values=[], include_test_suites=True),
            size=2,
            time_agg="1D",
            statuses=[TestStatus.ERROR, TestStatus.FAIL],
            panel_type=TestSuitePanelType.DETAILED,
        ),
    )

    project.save()
    return project
```

Adding test suite to workspace and project:
```
if __name__ == "__main__":
    # Initialize the workspace
    ws = Workspace.create(WORKSPACE)
    existing_project = ws.get_project("0194f20b-d78e-70e2-85a0-1ce64d8d928f")

    if existing_project is None:
        print(f"No existing project found. Creating new project: {PROJECT_NAME}")
    else:
        print(f"Existing project found: {existing_project.name} (ID: {existing_project.id})")

    project = create_or_update_project(workspace=ws, project=existing_project)

    # Get unique dates from DataFrame
    unique_dates = production_with_target_df["inference_date_utc"].dropna().unique()
    
    # Get all existing snapshots for the project (reports & test suites)
    existing_snapshots = project.list_snapshots()

    # Separate test suites and reports
    existing_test_suite_dates = {
        snap.timestamp.date() for snap in existing_snapshots if not snap.is_report
    }

    # Loop through each date in your unique_dates list
    for report_date in unique_dates:
        report_date_obj = pd.Timestamp(report_date).date()

        # Check and create the test suite if it does not exist
        if report_date_obj not in existing_test_suite_dates:
            test_suite = create_test_suite(
                df=production_with_target_df,
                reference_df=train_val_df,
                column_mapping=column_mapping,
                report_date=report_date
            )
            if test_suite is not None:
                ws.add_test_suite(project.id, test_suite)
                print(f"✅ Test suite added for {report_date}")
        else:
            print(f"Test suite for {report_date_obj} already exists. Skipping test suite creation.")
```

Expected vs. Actual Behavior

- ✅ Expected: The test suite should be visible in the Test Suite tab of the Evidently UI after being saved to the workspace.
- ❌ Actual: The test suite is saved in the workspace directory but does not appear in the UI when CustomValueTest is included.

Notes

- Removing or commenting out CustomValueTest resolves the issue, suggesting it may be causing the problem.
- The issue does not occur when running the test suite inside a Jupyter Notebook.

Request
Could this be a bug with how CustomValueTest interacts with the workspace? Any insights or potential workarounds would be appreciated.

Hi @John-Sellers, 

We've recently released a new Evidently API (where Reports and Test Suites are joined together), so we'd highly recommend to check out the new implementation instead. You can check how to create a custom metric here: https://docs.evidentlyai.com/metrics/customize_metric