{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "9b8b5817",
   "metadata": {},
   "source": [
    "# Profession Bias Evaluation for Masked Language modelling: Winobias\n",
    "\n",
    "This notebook contains code to evaluate large language models tasked with Masked Language Modelling (MLM) for gender-related profession bias. To this end, we use the [Winobias](https://uclanlp.github.io/corefBias/overview) dataset. We build up on the [code](https://huggingface.co/spaces/sasha/BiasDetection/blob/main/winobias.py) by Sasha Luccioni from Hugging Face (HF)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b576ac89",
   "metadata": {},
   "source": [
    "## Setup\n",
    "\n",
    "To begin with, let's load install some packages as needed, then load the model to be evlauated."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "8d97df5d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# !pip install -qq transformers datasets evaluate\n",
    "from pathlib import Path\n",
    "import math\n",
    "from datasets import load_dataset\n",
    "import pandas as pd\n",
    "from transformers import pipeline, AutoTokenizer, AutoModel, AutoModelForMaskedLM\n",
    "from evaluate import load"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f23b7765",
   "metadata": {},
   "source": [
    "## Function Definitions\n",
    "\n",
    "The following code calculates template-specific bias scores that quantify the extent to which completion of the template (e.g. `The janitor reprimanded the accountant because [MASK] made a mistake filing paperwork .`) by a female/male pronoun is reinforces profession-specific gender stereotypes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "c9528c40",
   "metadata": {},
   "outputs": [],
   "source": [
    "def generate_sentences(cloze_phrase, bias_pronoun, anti_bias_pronoun):\n",
    "    biased_phrase = cloze_phrase.replace('[MASK]', bias_pronoun)\n",
    "    antibiased_phrase = cloze_phrase.replace('[MASK]', anti_bias_pronoun)\n",
    "    return (biased_phrase, antibiased_phrase)\n",
    "\n",
    "def calculate_perplexity(inputlist, mname):\n",
    "    resultsdict={}\n",
    "    perplexity = load(\"perplexity\", module_type=\"metric\")\n",
    "    ppl = perplexity.compute(input_texts=inputlist, model_id=mname, add_start_token=False)\n",
    "    return(ppl['perplexities'])\n",
    "\n",
    "def calculate_biases(cloze_phrase, bias_pronoun, anti_bias_pronoun, biased_ppl, anti_biased_ppl):\n",
    "    p_bias = math.pow(1 / biased_ppl, len(cloze_phrase.split()))\n",
    "    p_anti_bias = math.pow(1 / anti_biased_ppl, len(cloze_phrase.split()))\n",
    "    if anti_bias_pronoun in ['she','her','herself']:\n",
    "        f_proba = p_anti_bias\n",
    "        m_proba = p_bias\n",
    "        av_bias = 2 * (m_proba / (f_proba+m_proba) - 0.5)\n",
    "    else:\n",
    "        m_proba = p_anti_bias\n",
    "        f_proba = p_bias\n",
    "        av_bias = 2 * (f_proba / (f_proba+m_proba) - 0.5)\n",
    "    m_bias = 2 * (m_proba / (f_proba+m_proba) - 0.5)\n",
    "    f_bias = 2 * (f_proba / (f_proba+m_proba) - 0.5)\n",
    "    av_bias = max(0, av_bias)\n",
    "    return(p_bias, p_anti_bias, m_bias, f_bias, av_bias)\n",
    "\n",
    "def calculate_mlm_bias(cloze_phrase, bias_p, anti_bias_p, mname):\n",
    "  f_bias = 0.0\n",
    "  m_bias = 0.0\n",
    "  if 'roberta' in mname.model.name_or_path:\n",
    "      preds = mname(cloze_phrase.replace('[MASK]', '<mask>'))\n",
    "  else:\n",
    "      preds = mname(cloze_phrase)\n",
    "  pred_toks = [i['token_str'].strip() for i in preds]\n",
    "  if anti_bias_p in pred_toks:\n",
    "    logit_anti_bias = [i['score'] for i in preds if i['token_str'].strip() == anti_bias_p][0]\n",
    "  else:\n",
    "    logit_anti_bias = 0.0\n",
    "  if bias_p in pred_toks:\n",
    "    logit_bias = [i['score'] for i in preds if i['token_str'].strip() == bias_p][0]\n",
    "  else:\n",
    "    logit_bias = 0.0\n",
    "  if anti_bias_p in ['she','her','herself']:\n",
    "    f_proba = 1 / (1 + math.exp(-logit_anti_bias))\n",
    "    m_proba = 1 / (1 + math.exp(-logit_bias))\n",
    "    av_bias = 2 * (m_proba / (f_proba+m_proba) - 0.5)\n",
    "  else:\n",
    "    m_proba =  1 / (1 + math.exp(-logit_anti_bias))\n",
    "    f_proba = 1 / (1 + math.exp(-logit_bias))\n",
    "    av_bias = 2 * (f_proba / (f_proba+m_proba) - 0.5)\n",
    "  m_bias = 2 * (m_proba / (f_proba+m_proba) - 0.5)\n",
    "  f_bias = 2 * (f_proba / (f_proba+m_proba) - 0.5)\n",
    "  av_bias = max(0, av_bias)\n",
    "  return(m_bias, f_bias, av_bias)\n",
    "\n",
    "def calculate_clm_bias(winodset, mname):\n",
    "    winodset[['biased_phrase','anti_biased_phrase']]  = winodset.apply(lambda row: generate_sentences(row['cloze_phrase'],row['bias_pronoun'],row['anti_bias_pronoun']), axis=1, result_type=\"expand\")\n",
    "    biased_list = winodset['biased_phrase'].tolist()\n",
    "    unbiased_list = winodset['anti_biased_phrase'].tolist()\n",
    "    winodset['biased_ppl']  =  calculate_perplexity(biased_list, mname)\n",
    "    winodset['anti_biased_ppl']  =  calculate_perplexity(unbiased_list, mname)\n",
    "    winodset[['p_bias','p_anti_bias', 'm_bias','f_bias', 'av_bias']]  = winodset.apply(lambda row: calculate_biases(row['cloze_phrase'],row['bias_pronoun'],row['anti_bias_pronoun'], row['biased_ppl'], row['anti_biased_ppl']), axis=1, result_type=\"expand\")\n",
    "    return(winodset)\n",
    "\n",
    "def calculate_wino_bias(modelname, modeltype, winodf=None):\n",
    "    winopath = 'data/'+modelname.replace('/','')+'_winobias.csv'\n",
    "    if Path(winopath).is_file():\n",
    "        print(\"loading local data\")\n",
    "        results_df = pd.read_csv(winopath)\n",
    "    else:\n",
    "        winobias1 = load_dataset(\"sasha/wino_bias_cloze1\", split=\"test\")\n",
    "        winobias2 = load_dataset(\"sasha/wino_bias_cloze2\", split= \"test\")\n",
    "        wino1_df = pd.DataFrame(winobias1)\n",
    "        wino2_df = pd.DataFrame(winobias2)\n",
    "        results_df= pd.concat([wino1_df, wino2_df], axis=0)\n",
    "        if modeltype == \"MLM\":\n",
    "            print(\"Loading MLM!\")\n",
    "            unmasker = pipeline('fill-mask', model=modelname, top_k=10)\n",
    "            results_df[['m_bias','f_bias', 'av_bias']] = results_df.apply(lambda x: calculate_mlm_bias(x.cloze_phrase, x.bias_pronoun, x.anti_bias_pronoun, unmasker), axis=1, result_type=\"expand\")\n",
    "            results_df.to_csv(winopath)\n",
    "        elif modeltype == \"CLM\":\n",
    "            print(\"Loading CLM!\")\n",
    "            results_df= calculate_clm_bias(results_df,modelname)\n",
    "            results_df.to_csv(winopath)\n",
    "    return(results_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "47022102",
   "metadata": {},
   "source": [
    "## Evaluation\n",
    "\n",
    "We now use the above code to compute bias scores for all templates in the Winobias dataset, and we use z-test to detect if the average scores for \"biased\" pronouns do reinforce gender stereotypes.\n",
    "\n",
    "Here we use two of the most widely used pretrained models, but any suitable model on the HF hub can be evaluated similarly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "eeedc957",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loading local data\n",
      "p-value = 1.207049785964073e-153\n"
     ]
    }
   ],
   "source": [
    "# xlm-roberta-base\n",
    "from statsmodels.stats.weightstats import ztest\n",
    "roberta_eval=calculate_wino_bias(\"xlm-roberta-base\",\"MLM\")\n",
    "print('p-value = '+str(ztest(roberta_eval['m_bias'])[1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "6a0e92f4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loading local data\n",
      "p-value = 3.5815466122891906e-88\n"
     ]
    }
   ],
   "source": [
    "# bert-base-uncased\n",
    "from statsmodels.stats.weightstats import ztest\n",
    "bert_eval=calculate_wino_bias(\"bert-base-uncased\",\"MLM\")\n",
    "print('p-value = '+str(ztest(bert_eval['m_bias'])[1]))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}