zhiminy/EvalEng huggingface dataset:
Repository: evidentlyai/evidently
Issue: Add a new `ExactMatch` descriptor to Evidently
Root cause: Unimplemented Feature Gap

# Add an `ExactMatch()` Descriptor

About Hacktoberfest contributions: https://github.com/evidentlyai/evidently/wiki/Hacktoberfest-2024

**Description:**

The `ExactMatch()` descriptor will compare text between two columns and return `True` if the text in both columns matches exactly, and `False` otherwise.

It should accept `with_column` parameter similar to `SemanticSimilarity` descriptor.

Hi @elenasamuylova, I'm interested in working on this issue to add an ExactMatch() Descriptor. Could I be assigned to this task? I'm eager to contribute but still learning, so any guidance or resources you could provide would be greatly appreciated. Thank you!

@elenasamuylova please review PR.

Hi @oceanthunder, since we already got a PR for this descriptor, we'd review it first. You are welcome to start working on other issues: just leave a comment for others that you are working on it.   

@elenasamuylova sure, thanks!
Repository: evidentlyai/evidently
Issue: Add a new `ContainsLink` descriptor to Evidently
GitHub labels: enhancement

hacktoberfest

About Hacktoberfest contributions: https://github.com/evidentlyai/evidently/wiki/Hacktoberfest-2024

**Description**:

Add a new `ContainsLink` descriptor that checks if the text contains any valid hyperlinks.  It returns True if at least one valid URL is detected in the text and False if no links are found. 

This can be useful for tasks such as checking that LLM-generated response includes a reference link when it is expected.
Repository: evidentlyai/evidently
Issue: Add new `JSONSchemaMatch()` descriptor to Evidently
Root cause: Unimplemented Feature Gap

About Hacktoberfest contributions: https://github.com/evidentlyai/evidently/wiki/Hacktoberfest-2024

**Description:**

Add the `JSONSchemaMatch()` descriptor that will evaluate whether a returned text is a JSON object with a schema that matches a structure specified by the user.  Implement two different matching options.

Minimal match:
* Checks if the JSON includes the listed keys.
* The values associated with these keys must not be `None`.
* Optionally, the descriptor can validate the types of the values for each key (e.g., name: string), as specified by the user.

Exact match (option):
* Ensures that the JSON contains only the provided keys, with no extra or missing keys.
* Validates both the presence and types of keys if specified by the user.

Returns True or False fo JSON schema match.

If the JSON is invalid, the schema can never match.

@elenasamuylova  I would like to work on this , please assign this to me 

Hi @yashaswikakumanu, we don't formally assign issues, but you are very welcome to work on it!
Repository: evidentlyai/evidently
Issue: Add new `JSONMatch()` descriptor to Evidently
Root cause: Unimplemented Feature Gap

# Add a new `JSONMatch()` descriptor

About Hacktoberfest contributions: https://github.com/evidentlyai/evidently/wiki/Hacktoberfest-2024

**Description:**

The `JSONMatch()` descriptor compares a JSON object in a given column against a reference JSON object in the specified reference column and returns `True` or `False` based on whether the JSONs are identical.

**JSON Matching Criteria:**

- Both JSON objects must have the exact same set of keys.
- The values associated with each key must be identical.
- The order of keys does not matter.
- Whitespace outside of the actual values (e.g., spaces, tabs, or newlines) is ignored.
- If either of the inputs is an invalid JSON, they will not match, and the descriptor will return `False`.

**Reference:**
For a two-column descriptor implementation, use the `CustomPairColumnEval` template and check the `SemanticSimilarity` descriptor as an example (implement the `with_column` parameter the same way).
Repository: evidentlyai/evidently
Issue: Add a new `IsValidJSON()` descriptor to Evidently
GitHub labels: enhancement

hacktoberfest

About Hacktoberfest contributions: https://github.com/evidentlyai/evidently/wiki/Hacktoberfest-2024

**Description**

Add the `IsValidJSON()` descriptor that checks whether the text in a specified column is a valid JSON object. It returns True if the text can be parsed as valid JSON and False if it cannot.

@elenasamuylova Created PR. Please review.
Repository: evidentlyai/evidently
Issue: Add a new `BERTScore` descriptor to Evidently
Root cause: Unimplemented Feature Gap

About Hacktoberfest contributions: https://github.com/evidentlyai/evidently/wiki/Hacktoberfest-2024

**Description**:

Implement BERTScore - a metric that computes a similarity score for each token in the candidate sentence with each token in the reference sentence.

Paper: https://arxiv.org/abs/1904.09675

Check existing `SemanticSimilarity` descriptor as a reference implementation.

Hi, could I take this up?


Hi @csking101, there is already a contribution pending to be merged for this one: https://github.com/evidentlyai/evidently/pull/1335
Repository: evidentlyai/evidently
Issue: Add a new `ROUGE` metric to Evidently 
Root cause: Unimplemented Feature Gap

About Hacktoberfest contributions: https://github.com/evidentlyai/evidently/wiki/Hacktoberfest-2024

**Description**

The ROUGE (Recall-Oriented Understudy for Gisting Evaluation) metric evaluates the quality of a generated text by comparing it to a reference text (typically a summary). It measures how much of the reference text is covered by the generated summary through n-gram overlap. Several common ROUGE variants exist:
* ROUGE-1: Measures unigram (word-level) overlap.
* ROUGE-2: Measures bigram (two-word sequence) overlap.
* ROUGE-N: Measures n-gram overlap between the candidate and reference text.

We can implement a ROUGE metric that takes the parameter `n` and computes both the descriptor values (overlap) for each row and a summary ROUGE metric for the dataset.

Note that this implementation would require creating a new Metric (instead of defaulting to `ColumnSummaryMetric` to aggregate descriptors values) to compute and visualize the summary ROUGE score. You can check other dataset-level metrics (e.g., from classification or ranking) for inspiration.


Hey @elenasamuylova , I am working on it. 

![Screenshot 2024-10-02 at 02 45 59](https://github.com/user-attachments/assets/7fa5defe-6409-4b01-82bd-7146bddfbb4c)

Do we need to show text as well for the comparison or just score per row is enough along with summary score?
Repository: evidentlyai/evidently
Issue: Add a new BLEU metric to Evidently 
Root cause: Unimplemented Feature Gap

About Hacktoberfest contributions: https://github.com/evidentlyai/evidently/wiki/Hacktoberfest-2024

**Descripton**. 

The BLEU (Bilingual Evaluation Understudy) metric is used to evaluate the quality of machine-generated text, typically translations, by comparing it to the reference texts. BLEU measures how closely the generated text matches the reference using n-gram precision, with a penalty for overly short or incomplete translations.

We can implement a BLEU metric that computes scores for each row and a summary BLEU metric for the dataset.

Note that this implementation would require creating a new Metric (instead of defaulting to ColumnSummaryMetric to aggregate descriptors values) to compute and visualize the summary BLEU score. You can check other dataset-level metrics (e.g., from classification or ranking) for inspiration.

**Note:** we can also consider implementing METEOR metric as an option.

Hi @elenasamuylova ,

I would like to work on this issue.

EDIT: Don't have the bandwidth. Sorry for the inconvenience.

I can take it. 

@elenasamuylova, does it still in need? I would like to contribute.
Repository: evidentlyai/evidently
Issue: Add a new `IsValidPython` descriptor to Evidently
GitHub labels: enhancement

hacktoberfest

About Hacktoberfest contributions: https://github.com/evidentlyai/evidently/wiki/Hacktoberfest-2024

**Description**

Add the `IsValidPython()` descriptor that checks whether the text in a specified column contains valid Python code. This descriptor returns `True` for valid Python code that does not contain syntax errors and `False` for invalid code.

Hey! I'm new to LLMs and ML. I'm really interested in being a part of hacktoberfest. Could you pls assign this to me. I'll learn and try to do something with this. If there's something similar of even doc work do let me know and assign it to me. Thanks in advance 

Hi @karthikk1006, we do not formally assign the issues, but you are very welcome to start working on it! Thanks for considering to contribute 🚀 

Hi @elenasamuylova , do we need to extract the python code from text and test the python code has syntax errors or not ?
Repository: evidentlyai/evidently
Issue: Add a new `IsValidSQL()` descriptor to Evidently
GitHub labels: enhancement

hacktoberfest

About Hacktoberfest contributions: https://github.com/evidentlyai/evidently/wiki/Hacktoberfest-2024

**Description**

Add the `IsValidSQL()` descriptor that checks whether the text in a specified column is a valid SQL query, without executing the query. This descriptor returns True for valid SQL queries and False for invalid queries.
Repository: evidentlyai/evidently
Issue: SemanticSimilarity descriptor throws pydantic validation error
Root cause: External Dependency Breakage

I am unable to use the SemanticSimilarty descriptor since I already receive a `pydantic.error_wrappers.ValidationError: 1 validation error for SemanticSimilarity` on initializing the descriptor with `SemanticSimilarity()`.

Full error:
```
File "pydantic/main.py", line 341, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for SemanticSimilarity
with_column
  field required (type=value_error.missing)
```

Since the `SemanticSimilarity()` call doesn't take any parameters and by itself isn't dependent on my data or environment I'm unable to resolve this error.

I get the same error with multiple different python versions and in different evironments.

Issue resolved itself
Repository: evidentlyai/evidently
Issue: Add a parameter to use LLM as a Judge Descriptor with privately hosted endpoint
Root cause: Unimplemented Feature Gap

**Description**

Hi team,

I am exploring Evidently AI for LLM Evaluation and I came across custom LLM as a Judge Descriptor in which I am particularly interested in. The current API only allows openai models using openai or litellm wrapper. Could you please add an additional parameter wherein we can pass the endpoint of our privately hosted LLMs?

I have attached the API screenshot for your reference.

Thanks and regards,
Satendra


https://github.com/evidentlyai/evidently/blob/main/examples/how_to_questions/how_to_use_llm_judge_template.ipynb

Hi @syadavlinklaters! This functionality is not yet available, but it's definitely on the roadmap.  

Pretty sure that you've already found it, but for those who may concern i provide the solution with DeclineLLMEval descriptor (tested with llama.cpp hosted model):

`DeclineLLMEval("answer", alias="Denials", provider='openai', model='ruadapt_qwen')], 
options=OpenAIOptions(api_key=<API_TO_YOUR_LLM>, api_url=<OPENAI COMPATIBLE URL>)
`
Repository: evidentlyai/evidently
Issue: Hardcoding of threshold values for data tests
Root cause: Documentation Deficiency

Hello, I was wondering about setting custom thresholds for data tests and I was a bit surprised that there didn't seem to be a straightforward way to do it for many of the tests. Some threshold values appear to be hardcoded, e.g. here:
https://github.com/evidentlyai/evidently/blob/d3e21fb657118f82e5e388223347e57e313c800e/src/evidently/tests/data_integrity_tests.py#L126C75-L126C87

I'm aware that there's a way to make custom drift methods (https://docs.evidentlyai.com/user-guide/customization/options-for-statistical-tests), but it seems very bizarre to me that these thresholds are hardcoded in certain tests rather than simply being optional parameters. Am I missing something here? Why aren't these thresholds always passed as optional parameters?

Hi @AlexJonesNLP, the hardcoded values are only the "sensible defaults". If you do not pass your own test condition, this default will apply and will be derived from reference. 

You can set custom test conditions using parameters like `gt` (greater than), `lt` (less than), etc.

Check the docs here: https://docs.evidentlyai.com/user-guide/tests-and-reports/run-tests#id-3.-set-test-conditions

Here is a usage example using parameters `eq` for equal, `lte` for less than equal, etc.:

```python
tests = TestSuite(
    tests=[
        TestShareOfMissingValues(lte=0.05),
        TestNumberOfConstantColumns(eq=0),
        TestNumberOfEmptyRows(eq=0),
        TestNumberOfEmptyColumns(eq=0),
        TestNumberOfDuplicatedColumns(eq=0),
        generate_column_tests(
            TestColumnDrift,
            columns="all",
            parameters={"stattest_threshold": 0.3, "stattest": "psi", "is_critical": False}
        ),
        TestShareOfDriftedColumns(lte=0.3),
    ]
)

```

@elenasamuylova Ahh I see thank you, looking at these lines I get it now: https://github.com/evidentlyai/evidently/blob/15bfe24d674dc02e4f9bc7f99fc69bb8e7b1d507/src/evidently/tests/base_test.py#L407-L413

I'm still not quite sure why this implementation was chosen over using optional parameters though?
Repository: evidentlyai/evidently
Issue: Feature Request: Support Configurable Base Path When Self-Hosting Evidently UI
Root cause: Unimplemented Feature Gap

Hello,

We are currently self-hosting Evidently UI on our servers and need to integrate it with our existing infrastructure, which includes single sign-on (SSO) and hosting the application under a specific base path.

It would be great if Evidently UI could support configuring a base path when self-hosting, so that all the URLs and static assets correctly reference the base path. This would allow us to host it behind a reverse proxy or under a specific path in our instance. 

Thank you! 

Hi @robert-inkpen did you find any workaround temporarily 

Evidently is in POC phase at our company so for the time being we provisioned it it's own endpoint sitting behind our SSO. Which doesn't allow it to be in our catalogue of tools but gets some of the power users access. 
One of the problems is that it has to be managed separately compared to everything else. 

Hi @robert-inkpen, I'm afraid we don't currently have the capacity to implement this feature, but we welcome your contributions here if you are up to it!

Note that we also offer a commercial enterprise version with extra features (available for self-hosting) if that's an option for you. 

@robert-inkpen I think I have dirty hack for you request, try to use this image (it may be broken in some corner cases 😃 but I would try it) 

```dockerfile
FROM evidently/evidently-service:latest

# Change `BASE_PATH`, escape `/` by `\/` as in example below

RUN BASE_PATH="evidently\/ui" && \
    cd $(python -c "import evidently as e;print(e.__path__[0] + '/ui/assets')") && \
    sed -i "s/\(src\|href\)=\"\//&$BASE_PATH\//g" index.html && \
    find . -type f -name '*.js' -exec sed -i \
        -e "s/\(to\|path\|baseUrl\):\"\/\"/\1:\"\/$BASE_PATH\"/g" \
        -e "s|static/js/|$BASE_PATH/static/js/|g" {} +
```

Also, have you considered using subdomains for hosting evidently?
Repository: evidentlyai/evidently
Issue: Fix semantic similarity in LLM eval tutorial
Root cause: Documentation Deficiency

Fix the example notebook, section "Semantic similarity":
https://github.com/evidentlyai/evidently/blob/main/examples/sample_notebooks/llm_evaluation_tutorial.ipynb

For new API:

```
text_evals_report = Report(metrics=[
    TextEvals(column_name="response", descriptors=[
        SemanticSimilarity(with_column="question", 
                           display_name="Response-Question Similarity"),
    ])
])

text_evals_report.run(reference_data=None,
                      current_data=assistant_logs[:100],
                      column_mapping=column_mapping)
text_evals_report
```
Repository: evidentlyai/evidently
Issue: ClassificationQualityByClassRenderer
Root cause: Algorithmic Error

I am using the ClassificationQualityByClass metric for my classificaiton ML model. In the reference data I have 7 labels, however, in the current data there are only example for 3 of them. When I run a report with this data and metric, the tables for current and reference only show the metrics for the 3 labels that appear on the "current" data, but not for all 7 labels in the reference data. 

I checked the renderer class for this metric, and I can see that for the reference plot, the names being used for x-axis are the ones from the current_metrics frame, instead of the reference_matrix_frame. Is this done on purpose? If not, I think a quick fix that would return the metrics for all labels in the reference data is using 
```
ref_names = ref_metrics_frame.columns.tolist()
x = list(map(str,ref_names))
```

for the reference plot (line 169 in evidently/metrics/classification_performance/quality_by_class_metrics.py).

Otherwise, is there any way to overwrite original renderer plots?
Repository: evidentlyai/evidently
Issue: Feature Request: Add Azure OpenAI 
Root cause: Unimplemented Feature Gap

Many enterprises are not using OpenAI but Azure OpenAI. It would be great to have the ability to change the endpoint for the openai library, through an ENV or other means.

Otherwise we will need to re-create several of the designed tests and metrics with custom implementations.

Understand that adding additional LLMs, such as Gemini, might be out of scope for a while.
Repository: evidentlyai/evidently
Issue: RegressionErrorBiasTable Not Generating Outputs 
Root cause: Documentation Deficiency

Hi, I'm new to model monitoring and currently exploring Evidently AI. I'm interested in leveraging the [regression performance preset](https://docs.evidentlyai.com/presets/reg-performance) for evaluating my regression models. I was particularly drawn to the `RegressionErrorBiasTable`, as it seems like a valuable tool for understanding error biases in my predictions.

However, when I use the `evidently_metrics.ipynb` notebook, it does not generate the `RegressionErrorBiasTable` or its associated plots. I've followed the steps in the documentation, but this specific table and visualizations seem to be missing.

Could this be a bug, or am I perhaps missing some setup steps? Any guidance would be appreciated!

**Environment Details:**
Evidently AI version: [0.4.40]
Python version: [3.10]

Thank you for your help!

Hi @MaxPolak97, 

Apologies for the delay in response. To enable this Metric you need to have the raw data option:

```
report = Report(
    metrics=[
      RegressionErrorBiasTable(),
    ],
    options={"render": {"raw_data": True}}
  )
report.run(reference_data=ref, current_data=cur)
report
```

Hi, I’d like to work on reproducing this bug in notebooks and fixing the missing table/plots — could you assign this to me?
Repository: evidentlyai/evidently
Issue: Using custom metrics in a Dashboard
Root cause: Unimplemented Feature Gap

I have created custom metrics for my Report following section 2 here[https://docs.evidentlyai.com/user-guide/customization/add-custom-metric-or-test].

I am able to create the reports with my custom function but when I create a dashboard and add these reports, it fails when I run evidently ui. I have noticed the error is when loading the snapshots as it cannot identify the new aliases for my metrics. However, if in a python console I import the functions and load the snapshots, it works.

Is there any way we could add those functions when running evidently ui in the CLI?

I realise it is noted in the link above that we won't be able to view such custom Metrics in the Monitoring user interface, but is there any work being done to fix this? It is a bit useless to be able to create custom metrics but not load the reports in the UI.

Thanks.

Hi @sara-vallejo, 

It is indeed not currently possible to view Reports with custom renders in the Evidently UI due to custom code - as noted in the docs. However, we are working on a major release that will address this together with the new way of implementing metrics.
Repository: evidentlyai/evidently
Issue: DashboardPanelDistribution TypeError
Root cause: External Dependency Breakage

Hello,

I want to use the DashboardPanelDistribution in my dashboard and i have tested the following. In my report I set:
```
report = Report(metrics=`ColumnDistributionMetrics(column_name="type"), timestamp=datetime.datetime.now(), tags=[])
```

Then the project is set as:
```
project = ws.create_project(MY_PROJECT_NAME)
project.dashboard.add_panel(
   DashboardPanelDistribution(
      title="Type distribution",
      filter=ReportFilter(metadata_values={}, tag_values=[]),
      value=PanelValue(
         field_path=ColumnDistribution.Metric.fields.current,
         metric_id="ColumnDistributionMetric",
         metric_args={"column_name.name": "type"}
      ),
      barmode=HistBarMode.STACK
)
```

then I add the reports as:
```
ws.add_report(report)
```

However, I get the following error when running `evidently ui`:
```TypeError: Subscripted generics cannot be used with class and instance checks```

My setup is:
- Python 3.9.2
- evidently==0.4.38
Repository: evidentlyai/evidently
Issue: How much is too big?

According to https://docs.evidentlyai.com/support/f.a.q. its recommended to use sampling for large datasets.
Can you please help me understand what is a "large" dataset that would require sampling?

Like how many (rows * columns) would cause issues?

Hi @EVAUTOAI, there is no fixed answer here since 
1) Evidently can evaluate hundreds of different metrics where each has its computational footprint (e.g., there are metrics like "text content drift" that train a whole machine learning model on your data vs. more straightforward metrics that compute the mean value in the column). You can also combine multiple metrics in the same report. 
2) The computation happens in memory, so the limitation will depend on your infrastructure. 

So the simple answer is: if your computation takes longer than you want or fails to compute otherwise, you may consider sampling. Also, sampling often makes sense for metrics like data distribution drift.

Thank you @elenasamuylova !

Are there any recommendations? or examples or mappings regarding memory size with Dataset size?