src/theory/metrics.qmd

---

title: Metrics

---

In machine learning, it's essential to evaluate the performance of a model to ensure it's accurate, reliable, and effective. There are various metrics to measure model performance, each with its strengths and limitations. Here's an overview of popular metrics, their pros and cons, and examples of tasks that apply to each.

For a quick [Overview](#overview) of metrics by use cases.


## Accuracy

Accuracy measures the ratio of correctly predicted observation to the total observations.

::: columns
::: {.column width="50%"}
Pros:

* Simplest and most intuitive metric.
::: 
::: {.column width="50%"}
Cons:

* Can be misleading in the presence of imbalanced classes.
::: 
::: 
Example tasks:

* General classification tasks where classes are balanced.
* Entry-level benchmarking



## Mean Squared Error (MSE)

MSE measures the average squared difference between predicted and actual values.


::: columns
::: {.column width="50%"}

Pros:

* Easy to calculate
* Sensitive to outliers
:::
::: {.column width="50%"}
Cons:

* Can be heavily influenced by extreme values
:::
:::
Example tasks:

* Regression tasks, such as predicting house prices or stock prices
* Time series forecasting

## Mean Absolute Error (MAE)

MAE measures the average absolute difference between predicted and actual values.

::: columns
::: {.column width="50%"}
Pros:

* Robust to outliers
* Easy to interpret
:::
::: {.column width="50%"}
Cons:

* Can be sensitive to skewness in the data
:::
:::
Example tasks:

* Regression tasks, such as predicting house prices or stock prices
* Time series forecasting

## Mean Absolute Percentage Error (MAPE)

MAPE measures the average absolute percentage difference between predicted and actual values.

::: columns
::: {.column width="50%"}
Pros:

* Easy to interpret
* Sensitive to relative errors
:::
::: {.column width="50%"}
Cons:

* Can be sensitive to outliers
:::
:::
Example tasks:

* Regression tasks, such as predicting house prices or stock prices
* Time series forecasting


## Binary Crossentropy

Binary Crossentropy measures the error rate between the true label and the predicted probability for binary classification tasks.

::: columns
::: {.column width="50%"}
Pros:

* Effective for measuring performance on binary classification problems.
* Directly optimizes the classification threshold.

::: 
::: {.column width="50%"}
Cons:

* Not suitable for multi-class classification tasks.
* Sensitive to the balance of classes in the dataset.
::: 
:::
Example tasks:

* Predicting customer churn (yes or no).
* Medical diagnostics with binary outcomes.

## Categorical Crossentropy

Categorical Crossentropy is used to measure the error rate between the true label and the predicted probabilities for multi-class classification tasks where each class is mutually exclusive.

::: columns
::: {.column width="50%"}
Pros:

* Well-suited for multi-class classification problems.
* Directly optimizes probability distribution across all classes.

::: 
::: {.column width="50%"}
Cons:

* Requires one-hot encoding of labels.
* Can be computationally intensive with many class labels.
::: 
:::
Example tasks:

* Image classification with multiple classes.
* Text classification into various categories.

## Sparse Categorical Crossentropy

Sparse Categorical Crossentropy is similar to Categorical Crossentropy but used for multi-class classification tasks where the classes are encoded as integers, not one-hot vectors.

::: columns
::: {.column width="50%"}
Pros:

* More memory efficient than Categorical Crossentropy when dealing with many classes.
* Eliminates the need for one-hot encoding of labels.

::: 
::: {.column width="50%"}
Cons:

* Not suitable for problems where multiple categories may be applicable to a single observation.
* Requires careful handling of class indices to ensure they are mapped correctly.
::: 
:::
Example tasks:

* Classifying news articles into a predefined set of topics.
* Recognizing the type of product from its description in a single-label system.


## R-Squared (R²)

R² measures the proportion of variance in the dependent variable that's explained by the independent variables.

::: columns
::: {.column width="50%"}
Pros:

* Easy to interpret
* Sensitive to the strength of the relationship

::: 
::: {.column width="50%"}
Cons:

* Can be sensitive to outliers
* Can be misleading for non-linear relationships
::: 
::: 

Example tasks:

* Regression tasks, such as predicting house prices or stock prices
* Feature selection

## Brier Score

The Brier Score measures the average squared difference between predicted and actual probabilities.

::: columns
::: {.column width="50%"}
Pros:

* Sensitive to the quality of the predictions
* Can handle multi-class classification tasks
::: 
::: {.column width="50%"}
Cons:

* Can be sensitive to the choice of threshold
:::
::: 

Example tasks:

* Multi-class classification tasks, such as image classification
* Multi-label classification tasks

## F1 Score

The F1 Score measures the harmonic mean of precision and recall.

::: columns
::: {.column width="50%"}
Pros:

* Sensitive to the balance between precision and recall
* Can handle imbalanced datasets

::: 
::: {.column width="50%"}
Cons:

* Can be sensitive to the choice of threshold

::: 
::: 

Example tasks:

* Binary classification tasks, such as spam detection
* Multi-class classification tasks

## Matthews Correlation Coefficient (MCC)

MCC measures the correlation between predicted and actual labels.

::: columns
::: {.column width="50%"}
Pros:

* Sensitive to the quality of the predictions
* Can handle imbalanced datasets
::: 
::: {.column width="50%"}
Cons:

* Can be sensitive to the choice of threshold
::: 
::: 

Example tasks:

* Binary classification tasks, such as spam detection
* Multi-class classification tasks

## Log Loss

Log Loss measures the average log loss between predicted and actual probabilities.

::: columns
::: {.column width="50%"}
Pros:

* Sensitive to the quality of the predictions
* Can handle multi-class classification tasks
::: 
::: {.column width="50%"}
Cons:

* Can be sensitive to the choice of threshold
::: 
::: 

Example tasks:

* Multi-class classification tasks, such as image classification
* Multi-label classification tasks

## Area Under the Receiver Operating Characteristic Curve (AUC-ROC)

The AUC-ROC curve is a performance measurement for classification problems at various threshold settings.

::: columns
::: {.column width="50%"}
Pros:

* Measures the ability of the model to discriminate between classes.
* Useful for binary classification problems.
::: 
::: {.column width="50%"}
Cons:

* Less effective when dealing with highly imbalanced datasets.
* Does not differentiate between types of errors.
::: 
::: 

Example tasks:

* Medical diagnosis classification.
* Spam detection in emails.

## Area Under the Precision-Recall Curve (AUC-PR)

The AUC-PR summarizes the precision-recall curve as a single number, which is the weighted average of precisions achieved at each threshold.

::: columns
::: {.column width="50%"}
Pros:

* Focuses on the positive class more than AUC-ROC, useful in imbalanced datasets.
::: 
::: {.column width="50%"}
Cons:

* More complex to interpret and explain than AUC-ROC.
:::
::: 

Example tasks:

* Information retrieval.
* Ranking tasks where positive class prevalence is low.


## Precision and Recall

Precision measures the ratio of correctly predicted positive observations to the total predicted positives. Recall measures the ratio of correctly predicted positive observations to all observations in actual class.

::: columns
::: {.column width="50%"}
Pros:

* Provide a more detailed understanding of model performance than accuracy, especially in imbalanced datasets.

::: 
::: {.column width="50%"}
Cons:

* High precision can sometimes be achieved at the expense of recall, and vice versa (precision-recall trade-off).
:::
:::

Example tasks:

* Document classification.
* Customer churn prediction.


## Cohen’s Kappa

Cohen’s Kappa measures the agreement between two raters who each classify N items into C mutually exclusive categories.

::: columns
::: {.column width="50%"}
Pros:

* Accounts for the possibility of the agreement occurring by chance.
* More robust than simple accuracy.
::: 
::: {.column width="50%"}
Cons:

* Can be difficult to interpret, especially with multiple classes.
::: 
::: 

Example tasks:

* Multi-rater reliability test.
* Medical image classification where multiple doctors provide diagnoses.

Here's the explanation of the additional metrics (BLEU, ROUGE, METEOR, MCC, Precision@k, Recall@k) in the format used previously in your document.


---

## BLEU (Bilingual Evaluation Understudy)

BLEU is a metric used to evaluate the quality of text which has been machine-translated from one natural language to another.

::: columns
::: {.column width="50%"}
Pros:

* Provides a quantitative measure for the quality of machine translation.
* Widely used and easy to compute.

:::
::: {.column width="50%"}
Cons:

* May not capture the fluency and grammaticality of the translation.
* Focuses more on precision than recall.

:::
:::
Example tasks:

* Machine translation evaluation.
* Automated text generation assessment.

## ROUGE (Recall-Oriented Understudy for Gisting Evaluation)

ROUGE is used to evaluate automatic summarization of texts as well as machine translation.

::: columns
::: {.column width="50%"}
Pros:

* Measures both the quality and quantity of match between reference and generated summaries.
* Supports evaluation of multiple types of summaries (extractive, abstractive).

:::
::: {.column width="50%"}
Cons:

* Can be biased towards extractive summarization techniques.
* Depends heavily on the quality of reference summaries.

:::
:::
Example tasks:

* Text summarization.
* Machine translation evaluation.

## METEOR (Metric for Evaluation of Translation with Explicit ORdering)

METEOR is a metric for evaluating machine translation that extends beyond simple overlap of vocabulary.

::: columns
::: {.column width="50%"}
Pros:

* Incorporates synonyms, stemming, and paraphrase matching.
* Aligns closely with human judgment compared to BLEU.

:::
::: {.column width="50%"}
Cons:

* More complex to compute than BLEU.
* Can require extensive linguistic resources.

:::
:::
Example tasks:

* Machine translation.
* Natural language generation.

## Matthews Correlation Coefficient (MCC)

MCC is a measure of the quality of binary classifications.

::: columns
::: {.column width="50%"}
Pros:

* Provides a balanced measure even if classes are of very different sizes.
* Considers true and false positives and negatives.

:::
::: {.column width="50%"}
Cons:

* Not applicable to multi-class classifications.
* Can be difficult to interpret in non-binary classification contexts.

:::
:::
Example tasks:

* Binary classification tasks such as spam detection.
* Medical imaging classification.

## Precision-k

Precision-k measures the proportion of relevant items found in the top-k recommendations of a system.

::: columns
::: {.column width="50%"}
Pros:

* Useful for evaluating ranking systems where the order of items is important.
* Easy to understand and implement.

:::
::: {.column width="50%"}
Cons:

* Does not consider the actual ranking of items beyond the cutoff of k.
* Sensitive to the choice of k.

:::
:::
Example tasks:

* Recommender systems.
* Search engine result ranking.

## Recall-k

Recall-k measures the proportion of relevant items that appear in the top-k recommendations out of all relevant items.

::: columns
::: {.column width="50%"}
Pros:

* Indicates the ability of a system to retrieve highly relevant items.
* Useful for systems where retrieving most relevant items is critical.

:::
::: {.column width="50%"}
Cons:

* Like precision-k, it is sensitive to the choice of k.
* Does not account for the relevance of items outside the top k.

:::
:::
Example tasks:

* Information retrieval.
* Content-based filtering in recommender systems.


## Overview
Below is a table outlining typical example cases in machine learning and the corresponding metrics that could be effectively used for evaluating model performance in each case. This table helps in quickly identifying which metrics are most applicable for a given type of task.

| Use Case                                 | Relevant Metrics                                                                      |
|------------------------------------------|---------------------------------------------------------------------------------------|
| **Predicting House Prices**              | MSE, MAE, R²                                                                          |
| **Stock Price Forecasting**              | MSE, MAE, R²                                                                          |
| **Medical Diagnosis Classification**     | Accuracy, F1 Score, AUC-ROC, Cohen’s Kappa                                            |
| **Spam Detection**                       | Precision, Recall, F1 Score, MCC, AUC-ROC                                             |
| **Image Classification**                 | Accuracy, F1 Score, Brier Score, Log Loss, AUC-ROC                                    |
| **Multi-label Classification**           | F1 Score, Brier Score, Log Loss, AUC-PR                                               |
| **Customer Churn Prediction**            | Accuracy, Precision, Recall, F1 Score                                                 |
| **Document Classification**              | Accuracy, Precision, Recall, F1 Score                                                 |
| **Ranking Tasks**                        | AUC-PR, Log Loss                                                                      |
| **Multi-rater Reliability Tests**        | Cohen’s Kappa                                                                         |
| **Multi-class Text Classification**      | Categorical Crossentropy, Sparse Categorical Crossentropy, Accuracy, F1 Score         |
| **Binary Classification of Customer Data** | Binary Crossentropy, Accuracy, Precision, Recall, F1 Score                          |
| **Sentiment Analysis**                   | Accuracy, Precision, Recall, F1 Score, MAE, Log Loss                                  |
| **Time Series Forecasting**              | MSE, MAE, R², MAPE                                                                    |
| **Anomaly Detection**                    | F1 Score, Precision, Recall, MCC                                                      |
| **Search Engine Ranking**                | Precision-k, Recall-k, AUC-ROC, AUC-PR                                                |
| **Natural Language Generation**          | BLEU, ROUGE, METEOR                                                                   |
| **Recommendation Systems**               | Precision, Recall, F1 Score, AUC-ROC                                                  |
| **Real-time Bidding in Advertising**     | Log Loss, ROC AUC                                                                     |
| **User Engagement Prediction**           | MSE, MAE, Log Loss, Brier Score                                                       |
| **Protein Structure Prediction**         | Accuracy, MCC, F1 Score                                                               |
| **Drug Response Modeling**               | MSE, MAE, R², Log Loss                                                                |
| **Credit Scoring**                       | AUC-ROC, Precision, Recall, F1 Score, Log Loss                                        |
| **Fraud Detection**                      | Accuracy, Precision, Recall, F1 Score, AUC-ROC                                        |

Each task is paired with metrics that are most appropriate based on the nature of the data and the specific requirements of the task.
This approach helps ensure that the evaluation of a model is not only accurate but also relevant to the goals of the analysis.