Repository: mir-evaluation/mir_eval
Issue: T-measure acceleration
Root cause: Algorithmic Error

[Note: This isn't an immediate issue, but I want to jot down some thoughts here for future reference.]

The `hierarchy.tmeasure` implementation involves a [summation](https://github.com/craffel/mir_eval/blob/master/mir_eval/hierarchy.py#L146) that takes `O(n * w**2)` time, where `n` is the number of frames and `w` is the window length.  This calculation derives from counting the number of disordered triples `(q, i, j) || (q, j, i)` between estimate and reference.  We use a naive algorithm to do this via numpy ufuncs.

For small values of `w`, this is acceptably fast.  (There are other reasons to use small `w` for t-measures, as we describe in the paper, so this doesn't really come up in practice yet.)  However, when `w` is large -- or in the case of a new metric I'm developing for label hierarchies, `w==n` -- this approaches cubic time and is generally bad news.

This calculation can be reduced to `O(n w log(w))` with a divide-and-conquer algorithm for inversion counting.  For large `w`, this becomes `O(n^2 log(n))`, which is within a `log(n)` factor of the other frame-wise segmentation metrics (nce, pairwise-f, etc).

I'm implementing this enhancement in the fork that includes the label hierarchy measures.  Unless there are objections, I'll just include it with that PR, but we could also do it separately if there's sufficient need.


For posterity, here's a bit more detail on how t-measures can be computed more efficiently.

After computing the least-common-ancestor matrices for `ref` and `est`, we iterate over query frames `q`.

For each `q`, we slice `ref_scores = lca_ref[q, window]` and `est_scores = lca_est[q, window]` where `window` is a slice around `q`.

Then, for each `i, j` (not counting `q`), we want to know how often the following occurs:

```
ref_scores[i] < ref_scores[j] => est_scores[i] < est_scores[j]
```

The slow algorithm for this is quadratic in the window size, since it looks at all pairs `i,j`.

This can be done more efficiently by the following method:
1. Sort `ref_scores`; let `idx` be the sorting index.
2. For each unique value `h` in `ref_scores`:
   a. Let `s, t` be the slice such that `ref_scores[s:t] == h`
   b. Find and sort the corresponding estimate scores: `pred[h] = sort(est_scores[idx][s:t])`
3. For each pair of reference score values (ie, levels to be compared), compute the number of inversions between `pred[h1], pred[h2]`: that is, values `a in pred[h1], b in pred[h2]` where `a >= b`.
4. Return the total number of detected inversions.

## Running time
1. Step 1 takes `O(w log w)`
2. Step 2 in aggregate takes `O(w log w)` since each value is touched exactly once
3. Step 3 takes `O(H**2 w)`, where `H` is the number of levels in the reference. 
   - In the reduced measures, this simplifies to `O(H w)`, since each level is compared only to its immediate neighbors in the hierarchy.
   - The inversion detection is fast here because the inputs are in sorted order.  See [these notes](https://www.cs.princeton.edu/~wayne/kleinberg-tardos/pdf/05DivideAndConquerI.pdf) for a simple tutorial on inversion counting in totally ordered lists.

The normalization constant can be readily computed in `O(w)` time using the method described in the initial post of this thread.

## Benchmark

The annotation spans a 30-second clip.  With a frame_size of 0.0125, this is equivalent to 4 minutes at frame_size=0.1.  There's probably still some room for improvement here, as this implementation has not been optimized to eliminate redundant calculations.

``` python
# New version
%timeit tmeasure(ref, est, window=None, frame_size=0.025, transitive=True)
1 loop, best of 3: 15.3 s per loop
```

``` python
# Old version
%timeit mir_eval.hierarchy.tmeasure(ref, est,  window=None, frame_size=0.025, transitive=True)
1 loop, best of 3: 1min 59s per loop
```

... so about an 8x speedup.  Results are numerically equivalent in all cases tested:

``` python
for window in [None, 5, 10, 15]:
    for frame_size in [0.05, 0.1, 0.5]:
        for transitive in [False, True]:
            sfast = tmeasure(ref, est, window=window, frame_size=frame_size, transitive=transitive)
            sold = mir_eval.hierarchy.tmeasure(ref, est, window=window, frame_size=frame_size, transitive=transitive)
            if not np.allclose(sfast, sold):
                print('FAIL: {},{},{}: {} != {}'.format(window, frame_size, transitive,
                                                        sfast, sold))
```

passes silently.


With a little more optimization and using run-length encoding to more efficiently count inversions, this is down to

``` python
%timeit tmeasure(ref, est, window=None, frame_size=0.025, transitive=True)
1 loop, best of 3: 2.24 s per loop
```


Resolved by merging #225