Repository: mir-evaluation/mir_eval
Issue: [Question] Re: Hierarchy metrics accumulation strategy

Gonna keep picking on the hierarchy metrics...

Currently _guac accumulates recall scores over each query frame 'q' when row 'q' of the reference meet matrix has any information (ranking difference) to recall. The number of these inversions to be recalled for each frame 'q' is stored as the 'normalizer', and the per-frame recall score is '1 - inversions/normalizer'.
The current accumulation strategy is average over the per-frame recall scores.

At the same time, frame q may have different amount of ranking info to be recalled (different 'normalizer' returned by count inversions). This implies that under the current setup, each triplet (t, u, v) is weighted differently: when row q of the reference meet matrix has few inversions (low normalizer), getting any of those few inversions right or wrong is going to matter much more than if they were in a row q with many more inversions to recall.

If I'm not mistaken, the current strategy of averaging over frames q means that each triplet (t u v) is not weighted equally when computing l measures; but this is not mentioned in the l-measure paper, I don't think?

To make sure all triplets are weighted equally, the fix is easy too: we just accumulate the normalizer and inversion independently over query frames and do the quotient after accumulation, as opposed to accumulating the quotient on each query frame.

@bmcfee thoughts on why this design choice?


> If I'm not mistaken, the current strategy of averaging over frames q means that each triplet (t u v) is not weighted equally when computing l measures; but this is not mentioned in the l-measure paper, I don't think?

That's correct, but it's not quite the way that I think of it.

The motivating idea here was not to weight all triplets equally, but to weight each time instant equally.  The idea ultimately derives from ranking evaluation: we treat each frame q as a query (hence q): the q row of the reference meet gives us a partial order over the remaining frames (these are the i's and j's).  A triple (q, i, j) means that i is closer to q than it is to j.  We then look at the estimate, which also provides a ranking of the same frames relative to q, and see how many of these orderings are satisfied.

You can treat this as a recall score (correctly recovering ordered triples), but you can also think of this as a generalization of AUC-ROC (hence "gauc").  In the usual ROC area case, the "reference ranking" is bipartite: the point is either relevant (positive) or not (negative), while the estimated relevance can be any continuous value.  The AUC score is calculated by sweeping a threshold and evaluating true positives and false positives (ie recall) at each threshold value on the estimate.  All we're doing here is relaxing the idea of "positive" and "negative" to allow for multiple levels of relevance.  

One of the nice things about this notion of the score is that it is calibrated: random ordering will produce a score of 0.5 on average, regardless of how many true positives you have for the query.  This is helpful when aggregating across multiple queries, as we do here.

I suppose you could normalize over all triples instead of per query normalization, though this might introduce other biases (eg short segments being washed out).  You could also consider doing a per-segment normalization, though that gets very tricky when it comes to multi-level segmentations.

The current GAUC accumulation strategy is reasonable, and I assume mir_eval.hierarchy has been used for most recent works doing hierarchical segmentation, so I'm happy to keep it the way it is.

The reason why I started looking into this is I'm trying to visualize the effect of using strictly monotonic meets vs relaxed meet (see #409), and I think I'll plot the GAUC per query frame q, so I can see how the recall score on each frame q is changing between the two different meet matrix versions.

Though.. Can we do something to document the gauc accumulation detail somewhere as I couldn't really find this info in the L-Measure paper?

Did what I wanted to do.. here it is.
The reference hierarchy is not monotonic, so the two versions give different meet mats. Here's how the l measure (recall) changes per frame.

![Image](https://github.com/user-attachments/assets/1900d06c-3d4f-41bf-a71f-b066dacaf73a)

![Image](https://github.com/user-attachments/assets/125abf68-0197-4aa9-b6a3-4ff544c5f4c8)

original lmeasure (0.33758895212858037, 0.35622028313034354, 0.34665445773157033)
monotonic lmeasure (0.44939179600981527, 0.44798926973012, 0.44868943685845125)