{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# F1 Scores" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "vscode": { "languageId": "r" } }, "outputs": [], "source": [ "library(\"ggpubr\")\n", "library(readr)\n", "library(ggplot2)\n", "library(tidyverse)\n", "library(ARTool)\n", "library(emmeans)\n", "library(multcomp)\n", "library(car)\n", "library(rstatix)" ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "vscode": { "languageId": "r" } }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "New names:\n", "* `` -> ...1\n", "\n", "\u001b[1mRows: \u001b[22m\u001b[34m59\u001b[39m \u001b[1mColumns: \u001b[22m\u001b[34m5\u001b[39m\n", "\u001b[36m--\u001b[39m \u001b[1mColumn specification\u001b[22m \u001b[36m--------------------------------------------------------\u001b[39m\n", "\u001b[1mDelimiter:\u001b[22m \",\"\n", "\u001b[32mdbl\u001b[39m (5): ...1, faiss_dpr, faiss_longformer, es_dpr, es_longformer\n", "\n", "\u001b[36mi\u001b[39m Use `spec()` to retrieve the full column specification for this data.\n", "\u001b[36mi\u001b[39m Specify the column types or set `show_col_types = FALSE` to quiet this message.\n" ] }, { "data": { "text/html": [ "\n", "\n", "\n", "\t\n", "\t\n", "\n", "\n", "\t\n", "\t\n", "\t\n", "\t\n", "\t\n", "\t\n", "\n", "
A tibble: 6 × 4
questionretrieverreaderf1
<dbl><fct><fct><dbl>
0faissdpr 0.0000000
0faisslongformer0.0000000
0es dpr 0.1300813
0es longformer0.7692308
1faissdpr 0.0000000
1faisslongformer0.0000000
\n" ], "text/latex": [ "A tibble: 6 × 4\n", "\\begin{tabular}{llll}\n", " question & retriever & reader & f1\\\\\n", " & & & \\\\\n", "\\hline\n", "\t 0 & faiss & dpr & 0.0000000\\\\\n", "\t 0 & faiss & longformer & 0.0000000\\\\\n", "\t 0 & es & dpr & 0.1300813\\\\\n", "\t 0 & es & longformer & 0.7692308\\\\\n", "\t 1 & faiss & dpr & 0.0000000\\\\\n", "\t 1 & faiss & longformer & 0.0000000\\\\\n", "\\end{tabular}\n" ], "text/markdown": [ "\n", "A tibble: 6 × 4\n", "\n", "| question <dbl> | retriever <fct> | reader <fct> | f1 <dbl> |\n", "|---|---|---|---|\n", "| 0 | faiss | dpr | 0.0000000 |\n", "| 0 | faiss | longformer | 0.0000000 |\n", "| 0 | es | dpr | 0.1300813 |\n", "| 0 | es | longformer | 0.7692308 |\n", "| 1 | faiss | dpr | 0.0000000 |\n", "| 1 | faiss | longformer | 0.0000000 |\n", "\n" ], "text/plain": [ " question retriever reader f1 \n", "1 0 faiss dpr 0.0000000\n", "2 0 faiss longformer 0.0000000\n", "3 0 es dpr 0.1300813\n", "4 0 es longformer 0.7692308\n", "5 1 faiss dpr 0.0000000\n", "6 1 faiss longformer 0.0000000" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "f1_scores <- read_csv(\"f1_scores.csv\") %>%\n", " rename(question = `...1`) %>%\n", " pivot_longer(!question, names_to=c(\"retriever\", \"reader\"), names_sep=\"_\", values_to=\"f1\")\n", "\n", "f1_scores$retriever = as.factor(f1_scores$retriever)\n", "f1_scores$reader = as.factor(f1_scores$reader)\n", "\n", "head(f1_scores)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To test which tests we can use, we need to check for normality. For this, we use a Shapiro-Wilk test of normality. As you can see in the results below, all $p$-values are lower than 0.001, so we reject the null-hypothesis of normality and now know that none of the f1-scores are normally distributed." ] }, { "cell_type": "code", "execution_count": 21, "metadata": { "vscode": { "languageId": "r" } }, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", "\t\n", "\t\n", "\n", "\n", "\t\n", "\n", "
A tibble: 1 × 3
variablestatisticp
<chr><dbl><dbl>
f10.50867063.999447e-18
\n" ], "text/latex": [ "A tibble: 1 × 3\n", "\\begin{tabular}{lll}\n", " variable & statistic & p\\\\\n", " & & \\\\\n", "\\hline\n", "\t f1 & 0.5086706 & 3.999447e-18\\\\\n", "\\end{tabular}\n" ], "text/markdown": [ "\n", "A tibble: 1 × 3\n", "\n", "| variable <chr> | statistic <dbl> | p <dbl> |\n", "|---|---|---|\n", "| f1 | 0.5086706 | 3.999447e-18 |\n", "\n" ], "text/plain": [ " variable statistic p \n", "1 f1 0.5086706 3.999447e-18" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", "\n", "\n", "\t\n", "\t\n", "\n", "\n", "\t\n", "\n", "
A tibble: 1 × 3
variablestatisticp
<chr><dbl><dbl>
f10.77045672.671656e-12
\n" ], "text/latex": [ "A tibble: 1 × 3\n", "\\begin{tabular}{lll}\n", " variable & statistic & p\\\\\n", " & & \\\\\n", "\\hline\n", "\t f1 & 0.7704567 & 2.671656e-12\\\\\n", "\\end{tabular}\n" ], "text/markdown": [ "\n", "A tibble: 1 × 3\n", "\n", "| variable <chr> | statistic <dbl> | p <dbl> |\n", "|---|---|---|\n", "| f1 | 0.7704567 | 2.671656e-12 |\n", "\n" ], "text/plain": [ " variable statistic p \n", "1 f1 0.7704567 2.671656e-12" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", "\n", "\n", "\t\n", "\t\n", "\n", "\n", "\t\n", "\n", "
A tibble: 1 × 3
variablestatisticp
<chr><dbl><dbl>
f10.67410317.912632e-15
\n" ], "text/latex": [ "A tibble: 1 × 3\n", "\\begin{tabular}{lll}\n", " variable & statistic & p\\\\\n", " & & \\\\\n", "\\hline\n", "\t f1 & 0.6741031 & 7.912632e-15\\\\\n", "\\end{tabular}\n" ], "text/markdown": [ "\n", "A tibble: 1 × 3\n", "\n", "| variable <chr> | statistic <dbl> | p <dbl> |\n", "|---|---|---|\n", "| f1 | 0.6741031 | 7.912632e-15 |\n", "\n" ], "text/plain": [ " variable statistic p \n", "1 f1 0.6741031 7.912632e-15" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", "\n", "\n", "\t\n", "\t\n", "\n", "\n", "\t\n", "\n", "
A tibble: 1 × 3
variablestatisticp
<chr><dbl><dbl>
f10.65589353.037616e-15
\n" ], "text/latex": [ "A tibble: 1 × 3\n", "\\begin{tabular}{lll}\n", " variable & statistic & p\\\\\n", " & & \\\\\n", "\\hline\n", "\t f1 & 0.6558935 & 3.037616e-15\\\\\n", "\\end{tabular}\n" ], "text/markdown": [ "\n", "A tibble: 1 × 3\n", "\n", "| variable <chr> | statistic <dbl> | p <dbl> |\n", "|---|---|---|\n", "| f1 | 0.6558935 | 3.037616e-15 |\n", "\n" ], "text/plain": [ " variable statistic p \n", "1 f1 0.6558935 3.037616e-15" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "f1_scores %>%\n", " filter(retriever == \"faiss\") %>%\n", " shapiro_test(f1)\n", "\n", "f1_scores %>%\n", " filter(retriever == \"es\") %>%\n", " shapiro_test(f1)\n", "\n", "f1_scores %>%\n", " filter(reader == \"dpr\") %>%\n", " shapiro_test(f1)\n", "\n", "f1_scores %>%\n", " filter(reader == \"longformer\") %>%\n", " shapiro_test(f1)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Since our data is not normally distributed, we cannot use an ANOVA to compare our results. Therefore, we use an aligned-rank test, which is a non-parameteric version of a factorial repeated measures ANOVA." ] }, { "cell_type": "code", "execution_count": 22, "metadata": { "vscode": { "languageId": "r" } }, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", "\t\n", "\t\n", "\n", "\n", "\t\n", "\t\n", "\t\n", "\n", "
A anova.art: 3 × 7
TermDfDf.resSum SqSum Sq.resF valuePr(>F)
<chr><dbl><dbl><dbl><dbl><dbl><dbl>
retrieverretriever 1232200452.90793168.058.632065.105423e-13
readerreader 1232 66045.36944311.616.226137.620176e-05
retriever:readerretriever:reader1232158290.44843714.043.525872.804257e-10
\n" ], "text/latex": [ "A anova.art: 3 × 7\n", "\\begin{tabular}{r|lllllll}\n", " & Term & Df & Df.res & Sum Sq & Sum Sq.res & F value & Pr(>F)\\\\\n", " & & & & & & & \\\\\n", "\\hline\n", "\tretriever & retriever & 1 & 232 & 200452.90 & 793168.0 & 58.63206 & 5.105423e-13\\\\\n", "\treader & reader & 1 & 232 & 66045.36 & 944311.6 & 16.22613 & 7.620176e-05\\\\\n", "\tretriever:reader & retriever:reader & 1 & 232 & 158290.44 & 843714.0 & 43.52587 & 2.804257e-10\\\\\n", "\\end{tabular}\n" ], "text/markdown": [ "\n", "A anova.art: 3 × 7\n", "\n", "| | Term <chr> | Df <dbl> | Df.res <dbl> | Sum Sq <dbl> | Sum Sq.res <dbl> | F value <dbl> | Pr(>F) <dbl> |\n", "|---|---|---|---|---|---|---|---|\n", "| retriever | retriever | 1 | 232 | 200452.90 | 793168.0 | 58.63206 | 5.105423e-13 |\n", "| reader | reader | 1 | 232 | 66045.36 | 944311.6 | 16.22613 | 7.620176e-05 |\n", "| retriever:reader | retriever:reader | 1 | 232 | 158290.44 | 843714.0 | 43.52587 | 2.804257e-10 |\n", "\n" ], "text/plain": [ " Term Df Df.res Sum Sq Sum Sq.res F value \n", "retriever retriever 1 232 200452.90 793168.0 58.63206\n", "reader reader 1 232 66045.36 944311.6 16.22613\n", "retriever:reader retriever:reader 1 232 158290.44 843714.0 43.52587\n", " Pr(>F) \n", "retriever 5.105423e-13\n", "reader 7.620176e-05\n", "retriever:reader 2.804257e-10" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stderr", "output_type": "stream", "text": [ "NOTE: Results may be misleading due to involvement in interactions\n", "\n" ] }, { "data": { "text/plain": [ " contrast estimate SE df t.ratio p.value\n", " es - faiss 58.3 7.61 232 7.657 <.0001\n", "\n", "Results are averaged over the levels of: reader " ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stderr", "output_type": "stream", "text": [ "NOTE: Results may be misleading due to involvement in interactions\n", "\n" ] }, { "data": { "text/plain": [ " contrast estimate SE df t.ratio p.value\n", " dpr - longformer -33.5 8.31 232 -4.028 0.0001\n", "\n", "Results are averaged over the levels of: retriever " ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "model.acc <- art(f1 ~ retriever * reader, data = f1_scores)\n", "anova(model.acc)\n", "art.con(model.acc, ~ retriever)\n", "art.con(model.acc, ~ reader)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "From these results, we can see that both the retriever and the reader have a significant effect on the F1 score ($F = 58.63$ and $F = 16.23$ respectively, $p < 0.0001$ for both). However, there is also an interaction between the retriever and reader ($F = 43.53$, $p < 0.0001$). The post-hoc analysis of contrasts shows that ElasticSearch performs better than FAISS ($p < 0.0001$) and Longformer performs better than DPR ($p = 0.0001$)." ] } ], "metadata": { "kernelspec": { "display_name": "R", "language": "R", "name": "ir" }, "language_info": { "codemirror_mode": "r", "file_extension": ".r", "mimetype": "text/x-r-source", "name": "R", "pygments_lexer": "r", "version": "4.1.2" }, "orig_nbformat": 4 }, "nbformat": 4, "nbformat_minor": 2 }