August 02 - August 09 Weekly Status
Published: 2021-08-06 20:46:25

This week I experimented with new metrics, which I implemented. 

I implemented 2 metrics based on color histogram and motion vectors. Since motion vectors already used in current version of the algorithm I focused on hostogram metric. Code for it can be found [here] (https://github.com/alexlqrs/rav1e/blob/master/src/api/lookahead.rs). I used histogram crate for this implementation. I calculate histogram of the first plane of the frame (luma component) and compare it to the previous frame's histogram. I used 4 metrics to calculate differences between these histograms:
1. The difference between mean values 
2. The difference between std values
3. Taxicab distance ![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0pr3036m2c1idv1qw6f4.png)
here p, q - histograms, n - amount of bins in each.
4. The square of the Euclidean distance

After that I substract current value from previous to make peaks more distinctive for the threshold.

Below you can find examples of these 4 distances in the same order on the same video. These pictures have scene changes (gray vertical lines) and final metric (blue line): 
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/j6qfxqjrj1dh7kit87ub.png)

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/zqiv2ojwgrc0p9ey8ck1.png)

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/a5kywh6sb1p99lwy19ut.png)

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/avov19jnrg7sa6zkaqll.png)

It can be seen that the metric with the least distinctive peaks is the one with STD difference.


Below you can find results for the first two distances:
* Mean values of histograms:
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5l3g1n3xy1t641dv8u5e.png)
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/gq4dikaytch2h6dvaipp.png)
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/80s919mif0xa3fqt1c9y.png)
* STD values of histograms:
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/v3f2254v44666hus726q.png)
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1cryb5lhaq583pbqhp83.png)
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/gcpzyvhvn8n869ihmlb6.png)


Below you can see the results for these metrics of BBC Planet Earth dataset and manually marked up open source videos:


| Metric | F score on BBC Planet Earth|F score on open source videos|
| --- | ---:| ---:|
|#1(mean)|0.8502|0.6532|
|#2(std)|0.6543|0.5951|
|#3(Euclidian)|0.7031|0.6002|
|#4(Taxicab)|0.7143|0.6231|

The speed of the algorithm with this metric is ~0.87x speed of current version of the algorithm. 

Summary:
The F score of new metrics is better than the current one. 
New metrics are a bit slower that the current metric.  
But since they use different characteristics of the frame (motion vectors and color histograms) in combination they could enhance each other and increase the final F score.

TO DO:
~~Precision recall curves for these metrics with different thresholds.~~
Correlation with previous metric. Would it be better to combine these metrics or use them separately?