Repository: evidentlyai/evidently
Issue: Cosine similarity for Drift Analysis
Root cause: Unimplemented Feature Gap

Cosine similarity: https://en.wikipedia.org/wiki/Cosine_similarity 
How to contribute stattest for Drift Analysis: https://github.com/evidentlyai/evidently/wiki/Hacktoberfest

This add-on can be useful in the following scenario: if the user wants to evaluate drift in non-structured data and perform drift detection over embeddings. 

Embeddings are often used as a representation of text or image data. This makes it possible to treat it as structured tabular data to feed it in Evidently under existing API. Even though it is impossible to interpret each column as a meaningful feature, one can still perform some analysis to compare the two datasets to detect drift. 

Disclaimer: This is a workaround. We expect to address drift detection for unstructured data later, but having this metric in the library can be helpful for the users already using this approach.

@emeli-dral 
**Reference:**
[https://arxiv.org/pdf/1812.01172.pdf](https://arxiv.org/pdf/1812.01172.pdf)

Please see the above paper

We cannot convert a tabular data to embedding using image data because that would require a great degree of normalisation which causes information loss. 

Let us take this approach, The paper above states, if you can get to R(p,p) -> R(m,n) using a function f (mapping) then it is feasible to calculate the cosine distances between two matrices. 

The below paper is that of calculating cosine similarity from Euclidean distances
**Reference:**
[https://ccrma.stanford.edu/~dattorro/EDM.pdf](https://ccrma.stanford.edu/~dattorro/EDM.pdf)

Let is consider this:
```
reference_gram_matrix = np.matmul(reference_data, np.transpose(reference_data))
current_gram_matrix = np.matmul(current_data, np.transpose(current_data))

cosine_distance = reference_gram_matrix * current_gram_matrix / (np.linalg.norm(reference_gram_matrix, norm=2, axis=1) * np.linalg.norm(current_gram_matrix, norm=2, axis=1))
```

Also, please note that the only way to convert R(p,p) -> R(m,n) is through convolution or using DEA (Data Envelopment Analysis) techniques. 

DEA is generally used for statistical efficiency evaluation of let's say fuel. Please see DEA below, which is achieved using Convex Optimization (cvxpy)
**Reference:**
[https://en.wikipedia.org/wiki/Data_envelopment_analysis](https://en.wikipedia.org/wiki/Data_envelopment_analysis)

We will have to decide how we are going to relate to a statistical efficiency of two datasets of random origin using Convex Optimization. 

**Test of Covariance and Correlation Matrices**
In the paper, https://arxiv.org/pdf/1812.01172.pdf, I have used the K-sample Test method. 

The algorithm is in the Pull Request #396 

The New pull request is in #428 

Hi @aswinvk28,

Sorry for the long response. Unfortunately, since it required reviewing the article with the method description, it was impossible to fast-track it.

Let me summarise what the article proposes for anyone reading this:
- One can evaluate the covariance Σ (or correlation R) matrix for the reference and current datasets and then compare the similarity between these matrices. Thus, we test H0 : Σ1 = Σ2 for covariance matrix (or H0 : R1 = R2 for correlation matrix) against  the inequality alternative.
- As a result, we will get a single value to evaluate the similarity range for the matrices.
- This method can be applied if the input is a table (e.g., a vocabulary of words and their frequencies, or embeddings, etc.).  

I agree that it is very interesting approach that is useful to understand drift of the whole dataset! 

I went trough your implementation, and I assumed you expected to get the whole dataset as an input, is that correct?
I guessed it form the expression ```np.var(reference_data.values.flatten())``` , which makes total sense for matrices, but does not change anything for vectors.

It looks like your implementation would fit nicely to calculate dataset drift as a whole, but it does not suit to calculate drift for a single feature. This why, it is unfortunately not possible to implement this as a part of Hacktoberfest call for contributions, as the DataDriftOption feature assumes that all StatTest functions should apply to the individual feature (column) level. They expect a single vector or a single column as an input. 

We cannot change the design of this feature to take the table as an input since there are many other features dependent on it (for example, ColumnDriftTest(), ColumnDriftMetric(), DatasetDriftTable() etc.).

It is possible to add the method you describe to the library as a new Metric that would exist separately. However, implementing a new metric from scratch is much more complex and lengthy (as it requires having its own visualization, passing all the tests). We plan to add an instruction on how to add Custom Metrics to the library soon, but that will require some time. We only released the underlying Metrics functionality a week ago.  

I saw that you submitted a new PR. If you still plan to implement the method, I'd suggest to do one of the following (or both!):
- **Implement the “classic” cosine similarity as a drift method** that would take a single vector (column) as an input. That is something we’ll be able to review and test quite fast. I marked your new PR with hacktoberfest-accepted in order to give you more time, if you choose to implement it. Thus we acknowledge work done and as we answered late you should have the opportunity to finish it if you want.
- **Implement the method you suggest as a custom metric**. However, if you choose to go this way, please consider that we do not have very detailed howto on the topic at this point, so you should be able to study the repository and go through with the implementation independently. It might be better to wait a couple of months as we implement more specific instructions and add additional features to simplify adding custom metrics. 

I also want to thank you again for sharing this method! 
I’d suggest that if you share some new original methods in the future to provide a quick summary of the implementation as well (what it expects as an input, what it returns as an output) and to which use cases it is applicable (e.g. when it is better than alternatives) since it would help us grasp the idea and give some feedback faster.

Thanks for your suggestions! It is very understandable. 

> **Implement the method you suggest as a custom metric**. However, if you choose to go this way, please consider that we do not have very detailed how to on the topic at this point, so you should be able to study the repository and go through with the implementation independently. It might be better to wait a couple of months as we implement more specific instructions and add additional features to simplify adding custom metrics.
> 
> - What additional features are they?
> - How are they relevant to Cosine similarity alone?

We cannot change the **design** of this feature to take the table as an input since there are many other features dependent on it (for example, ColumnDriftTest(), ColumnDriftMetric(), DatasetDriftTable() etc.).

- We do not have to change the design

> This why, it is unfortunately not possible to implement this as a part of Hacktoberfest call for contributions, as the DataDriftOption feature assumes that all StatTest functions should apply to the individual feature (column) level. 
> - No worries on that. This feature is a complicate one as it requires permutations and couple of reshapes to include the 
> [: newaxis] ?

They expect a single vector or a single column as an input.
- That is not a problem, I can implement it. 
- A single vector usually reduces to a single number when sum is taken and does not take an overall picture of the dataset which is also a problem because cosine similarity may be an overkill when you consider a single feature. I have reasons to believe this because cosine similarity are Euclidean distances are highly inter-related through a Gram Matrix. 

> I’d suggest that if you share some new original methods in the future to provide a quick summary of the implementation as well (what it expects as an input, what it returns as an output) and to which use cases it is applicable (e.g. when it is better than alternatives) since it would help us grasp the idea and give some feedback faster.
> - I have shared the paper, as per the first comment!
> - There are no existing implementations I have seen related to this cosine similarity implementation in a paper
> - I was planning to have a new notebook similar to what others have provided once the Hacktoberfest PR is accepted and agreed by the team. 
> - Since this is not implemented, and requires modification based on only a **column-wise evaluation**, I will add that notebook visualization in my next PR

Also I need support regarding code quality in the PRs because most of the PRs have been failing. Could you explain me how I actually test this code, because `pytests` do not work, pylint issues have been fixed per file. Still the issue is persisting. Let me retest the code again using an external notebook to see the outputs. 

**Stattest**
That is nice, per column drift detection implementation works for us just fine!

**Pytest**
That's very interesting, ```pytest``` should work just fine. Is ```pytest``` installed correctly? What do you see if you run the command ```pip show pytest```? It should be somewhat like this:
```
Name: pytest
Version: 6.2.5
Summary: pytest: simple powerful testing with Python
...
```

If ```pytest``` is installed and works, the problem might be in the virtual environment you use. We recommend to create virtual environment with python3.6 (like ```conda create -n yourenvname python=3.6```), because that the earliest version of python we support, and we run all tests under this python version. 

Could you please share as well the output you see when you run the command ```pytest -v tests``` from the repository folder?
It should be somewhat like this:
<img width="1437" alt="Screenshot 2022-11-09 at 22 11 54" src="https://user-images.githubusercontent.com/22995068/200922025-2d939661-241c-4951-866e-0014cdcf4a81.png">

**Linter**
For linters, I believe that  is an issue with```black``` configuration.  
```Black``` works slightly differently with different python versions. I suggest you run ```black``` with the following parameters: ```black --check -l 120 -t py36 .``` to see what black suggest you to change or ```black  -l 120 -t py36 .``` to change files.