Repository: evidentlyai/evidently
Issue: Undocumented backwards incompatibility in 0.1.49.dev0
Root cause: Documentation Deficiency


Hello, I found that the most recent version **0.1.49.dev0** of evidently probably introduces some **breaking changes**, which are not recorded in release notes.

1. Add required parameter `feature_type` in following functions
    1. `evidently.analyzers.stattests.kl_div.kl_div`
    2. `evidently.analyzers.stattests.psi.psi`
    3. `evidently.analyzers.stattests.utils.get_binned_data`
2. Parameter position changing, would cause unexpected argument passing if calling the function with all positional arguments.
    1. `n_bins` from 4th to 5th in function `evidently.analyzers.stattests.kl_div.kl_div` and `evidently.analyzers.stattests.psi.psi`
    2. `threshold` from 3rd to 4th in function `evidently.analyzers.stattests.kl_div.kl_div` and `evidently.analyzers.stattests.psi.psi`
    3. `n` from 3rd to 4th in function `evidently.analyzers.stattests.utils.get_binned_data`
3. Change type of `func` parameter in function `evidently.analyzers.stattests.registry.StatTest.__init__`, from a 3 parameter function to a 4 parameter function: (Series, Series, float) to (Series, Series, str, float)

To make it easier to understand the impact of the breaking changes, I write a short code snippet that reproduces the breaking changes.
The following code runs well in 0.1.48.dev0 but gets crashed in 0.1.49.dev0.

```python
# Change 1
evidently.analyzers.stattests.kl_div.kl_div(reference_data=reference_data, current_data=current_data, threshold=threshold, n_bins=n_bins)
# missing a required argument: 'feature_type'

# Change 2
evidently.analyzers.stattests.kl_div.kl_div(reference_data, current_data, threshold, n_bins)
# parameter `feature_type` accepts argument from `threshold` and parameter `threshold` accepts argument from `n_bins`
```

Maybe recording these changes in release notes helps to avoid user confusion during updating this version?

Kind regrads,

@eXceediDeaL

Hi @eXceediDeaL ,
thank you for bringing this up, it makes a lot of sense! 

Indeed, we never added any notes regarding the changes that lie deeper than the interface for generating Dashboards and Profiles. Initially we decided to limit the amount of the release notes details  to the upper level API, cause generally our users are no so interested in other details and we release new things comparably often. Good to know that things are changing! We will rethink notes format.

We are a bit surprised to know, that you use stattests this way! May be you could tell us a little bit more about how exactly do you use stattsests for your analysis? It would be very helpful for designing the API on deeper levels.  We eager to know more :)

@Liraim @Tapot just want to share this with you.

Thank you for your detailed reply!

I'm not a user of evidently. Actually, I used a tool to check API backwards compatibility on popular packages including evidently and found the above results. I guess the APIs are public according their names, so I report these breaking changes.

Feel free to close this issue. I'm sorry if it bother you.
I know evidently is under active development, and I hope these results could help you develop a better product for users. :)

---

BTW, a small suggestion on parameter position changes. Keyword parameter is useful for unordered parameters. For example, 

```python
def f(a, b, *, count=0): pass
# f(1, 2, 3) is not allowed
# because the parameter name 'count' has to be specified:
# f(1, 2, count=3)
```