{
"nbformat": 4,
"nbformat_minor": 0,
"metadata": {
"colab": {
"provenance": [],
"gpuType": "T4"
},
"kernelspec": {
"name": "python3",
"display_name": "Python 3"
},
"language_info": {
"name": "python"
},
"accelerator": "GPU"
},
"cells": [
{
"cell_type": "markdown",
"source": [
"# How to use the authdetect model? A practical tutorial\n",
"\n",
"Author: Michal Mochtak\n",
"\n",
"Date: Ocotber 10, 2024"
],
"metadata": {
"id": "EzUt9g09sMfV"
}
},
{
"cell_type": "markdown",
"source": [
"## Introductory remarks ⛳\n",
"\n",
"This is an interactive Jupyter notebook that presents a step-by-step tutorial on how to use the **authdetect** model with your own data. The overall structure of the notebook is organized around two elements: 1) sentence trigrams extraction and 2) sentence trigrams annotation.\n",
"\n",
"If you use this tutorial, please cite the paper:\n",
"\n",
"\n",
"> Mochtak, Michal (forthcoming): Chasing the Authoritarian Specter: Detecting Authoritarian Discourse with Large Language Models. European Journal of Political Research.\n",
"\n"
],
"metadata": {
"id": "Mjg6lAlFsbJ1"
}
},
{
"cell_type": "markdown",
"source": [
"## Prerequisities ⚡\n",
"Google Colab is an interactive development environment with access to computational resources that are easy to utilize free of charge (read more about it here: https://colab.research.google.com/).\n",
"\n",
"To use the authdetect model, you first need to connect to an interactive session with access to a graphical processing unit. To do this, click \"Runtime\" in the top toolbar and select \"Change runtime type\":\n",
"\n",
"\n",
"
\n",
"
\n",
"After a pop-up appears, select any available GPU accelerator and save your selection:\n",
"\n",
"\n",
"\n",
"Finally, in the top right corner, click \"Connect.\" After a moment, a green check mark (✔) will appear, indicating that your virtual session has been successfully set up:\n",
"\n",
""
],
"metadata": {
"id": "Tyr0BdDlvJJl"
}
},
{
"cell_type": "markdown",
"source": [
"## Loading data for processing 💾\n",
"This notebook is designed for a simple use case that expects users to prepare their data outside the Google Colab environment as a plain .csv file and then upload it for further processing. The repository for this paper contains a sample file that you can use as a guide for formatting your own data. The file contains 193 speeches in English from the 79 session of the UN General Assambley (2024). The file contaisn multiple columns but the pipeline needs only two of them: \"doc_id\" as a document identifier and \"speech\" for the actual speeches (the pipeline does not require anything else).\n",
"\n",
"**Disclaimer**: Although the speeches come from the UN General Assembly, they were not part of the training process (the last included session used during the training was the 77th meeting held in 2022). The 79th session, held in September 2024, is the most recent session and was collected solely for the purpose of this tutorial. The benefit of working with UN data is that it is representative, capturing speeches from leaders of 193 countries.\n",
"\n",
"To use the file (or any other file you prepare), upload it to your interactive session by clicking the folder icon on the left and dragging and dropping your file into the area under the \"sample_data\" folder. The file will be uploaded to your interactive session and will be available for processing. It is important to note that the file exists only in this interactive session and will be deleted when you close it. This applies to any file you create in your session (e.g., the processed data).\n",
"\n",
"\n",
"\n"
],
"metadata": {
"id": "cF1rFHA12gBC"
}
},
{
"cell_type": "markdown",
"source": [
"## Processing the data 🎆\n",
"\n",
"The processing pipeline can be divided into two steps: 1) sentence trigrams extraction and 2) sentence trigrams annotation. From this point on, the notebook will also include code cells that can be executed by clicking the small \"play\" icon next to them (hover over the cell to make it visible)."
],
"metadata": {
"id": "tAREYA5V9Pty"
}
},
{
"cell_type": "markdown",
"source": [
"### Loading the necessary libraries 💻\n",
"To process the input data, we need to install and load a few packages we will use."
],
"metadata": {
"id": "xedYNVClALEN"
}
},
{
"cell_type": "code",
"source": [
"# Install missing packages in your session; this needs to be done each time you\n",
"# open the notebook, as the session is interactive. It takes a while.\n",
"!pip install simpletransformers\n",
"!pip install trankit==1.1.1"
],
"metadata": {
"id": "m9Ekm2hy_5zt",
"collapsed": true
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"# Load the necessary packages.\n",
"import simpletransformers.classification as cl\n",
"import trankit\n",
"import pandas as pd"
],
"metadata": {
"id": "ixpDR4gQA_Ri"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"### Step 1: Sentence trigrams extraction ⛏\n",
"Now that all the necessary packages are loaded and ready to use, we can proceed with the first step—sentence trigram extraction (the actual input for the authdetect model)."
],
"metadata": {
"id": "F-iTvMydBJu4"
}
},
{
"cell_type": "code",
"source": [
"# Load the dataset you want to process. This tutorial uses a plain .csv file for simplicity.\n",
"df = pd.read_csv(\"sample_data.csv\")"
],
"metadata": {
"id": "d7bi80FCBYKt"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"# Load the trankit pipeline with the English language model\n",
"p = trankit.Pipeline(lang='english', embedding='xlm-roberta-base', gpu=True, cache_dir='./cache')"
],
"metadata": {
"id": "a9Y_JgI4Cosb"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"# Check the dataset to be sure it was read in correctly.\n",
"print(df)"
],
"metadata": {
"id": "It1UPAF2CVT9",
"collapsed": true,
"colab": {
"base_uri": "https://localhost:8080/"
},
"outputId": "584f4306-74c7-4feb-de55-75b38e1d14db"
},
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
" doc_id speech \\\n",
"0 1 The Assembly will hear an address by His Excel... \n",
"1 2 The Assembly will hear an address by His Excel... \n",
"2 3 The Assembly will hear an address by His Royal... \n",
"3 4 The Assembly will hear an address by His Excel... \n",
"4 5 The Assembly will hear an address by His Excel... \n",
".. ... ... \n",
"188 189 I now give the floor to His Excellency Ludvig ... \n",
"189 190 I now give the floor to His Excellency Bakary ... \n",
"190 191 I now give the floor to His Excellency, Peter ... \n",
"191 192 I now give the floor to His Excellency Bassem ... \n",
"192 193 And I now give the floor to His Excellency She... \n",
"\n",
" link country \\\n",
"0 https://gadebate.un.org/en/79/angola Angola \n",
"1 https://gadebate.un.org/en/79/argentina Argentina \n",
"2 https://gadebate.un.org/en/79/bahrain Bahrain \n",
"3 https://gadebate.un.org/en/79/belgium Belgium \n",
"4 https://gadebate.un.org/en/79/brazil Brazil \n",
".. ... ... \n",
"188 https://gadebate.un.org/en/79/nicaragua Nicaragua \n",
"189 https://gadebate.un.org/en/79/niger Niger \n",
"190 https://gadebate.un.org/en/79/sri-lanka Sri Lanka \n",
"191 https://gadebate.un.org/en/79/syrian-arab-repu... Syrian Arab Republic \n",
"192 https://gadebate.un.org/en/79/united-arab-emir... United Arab Emirates \n",
"\n",
" title speaker \\\n",
"0 His Excellency João Manuel Gonçalves Lourenço \n",
"1 His Excellency Javier Gerardo Milei \n",
"2 His Royal Highness Prince Salman bin Hamad Al Khalifa \n",
"3 His Excellency Alexander de Croo \n",
"4 His Excellency Luiz Inácio Lula da Silva \n",
".. ... ... \n",
"188 His Excellency Valdrack Ludwing Jaentschke Whitaker \n",
"189 His Excellency Bakary Yaou Sangaré \n",
"190 His Excellency Peter Mohan Maithri Pieris \n",
"191 His Excellency Bassam Sabbagh \n",
"192 His Excellency Sheikh Shakhboot Nahyan Al-Nahyan \n",
"\n",
" role date \\\n",
"0 President 24 September 2024 \n",
"1 President 24 September 2024 \n",
"2 Crown Prince 24 September 2024 \n",
"3 Prime Minister 24 September 2024 \n",
"4 President 24 September 2024 \n",
".. ... ... \n",
"188 Minister for Foreign Affairs 30 September 2024 \n",
"189 Minister for Foreign Affairs 30 September 2024 \n",
"190 Permanent Representative 30 September 2024 \n",
"191 Minister for Foreign Affairs and Expatriates 30 September 2024 \n",
"192 Minister of State 30 September 2024 \n",
"\n",
" pdf \\\n",
"0 https://gadebate.un.org/sites/default/files/ga... \n",
"1 https://gadebate.un.org/sites/default/files/ga... \n",
"2 https://gadebate.un.org/sites/default/files/ga... \n",
"3 https://gadebate.un.org/sites/default/files/ga... \n",
"4 https://gadebate.un.org/sites/default/files/ga... \n",
".. ... \n",
"188 https://gadebate.un.org/sites/default/files/ga... \n",
"189 https://gadebate.un.org/sites/default/files/ga... \n",
"190 https://gadebate.un.org/sites/default/files/ga... \n",
"191 https://gadebate.un.org/sites/default/files/ga... \n",
"192 https://gadebate.un.org/sites/default/files/ga... \n",
"\n",
" audio \n",
"0 https://s3.amazonaws.com/downloads.unmultimedi... \n",
"1 https://s3.amazonaws.com/downloads.unmultimedi... \n",
"2 https://s3.amazonaws.com/downloads.unmultimedi... \n",
"3 https://s3.amazonaws.com/downloads.unmultimedi... \n",
"4 https://s3.amazonaws.com/downloads.unmultimedi... \n",
".. ... \n",
"188 https://s3.amazonaws.com/downloads.unmultimedi... \n",
"189 https://s3.amazonaws.com/downloads.unmultimedi... \n",
"190 https://s3.amazonaws.com/downloads.unmultimedi... \n",
"191 https://s3.amazonaws.com/downloads.unmultimedi... \n",
"192 https://s3.amazonaws.com/downloads.unmultimedi... \n",
"\n",
"[193 rows x 10 columns]\n"
]
}
]
},
{
"cell_type": "code",
"source": [
"# Split texts into sentences. We use a simple loop as the model processes inputs sequentially.\n",
"sentences = []\n",
"for n in range(0, len(df[\"speech\"])):\n",
" one_text = pd.DataFrame.from_dict(p.ssplit(df[\"speech\"][n]))\n",
" one_text[\"doc_id\"] = df[\"doc_id\"][n]\n",
" one_text = one_text.drop([\"text\", \"lang\"], axis = 1)\n",
" sentences.append((one_text))\n",
"\n",
"# Concatenate the list and reset the index.\n",
"sentences = pd.concat(sentences)\n",
"sentences.reset_index(drop=True, inplace=True)\n",
"\n",
"# Extract data from JSON format and patch everything together.\n",
"sentences_final = pd.concat([sentences.drop(['sentences'], axis=1), pd.json_normalize(sentences['sentences'].tolist())], axis=1)"
],
"metadata": {
"id": "cY7HA7QZC69L"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"# Check the result of sentence extraction. It is a data frame with the following columns:\n",
"# - doc_id: Refers to the original document.\n",
"# - id: Refers to the sentence ID within the processed input (e.g., speech).\n",
"# - text: Contains the extracted grammatical units (i.e., the sentences).\n",
"# - dspan: Contains indexes for the beginning and end of each sentence in the processed string.\n",
"\n",
"print(sentences_final)"
],
"metadata": {
"id": "f-dlGQBPDz27"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"# Helper function for creating sentence trigrams. Sentence trigram is the main input for the authdetect model.\n",
"def create_ngram(text):\n",
" no_steps = len(text) - 2\n",
" indexes = [list(range(x, x + 3)) for x in range(no_steps)]\n",
" ngrams = [' '.join(text[i] for i in index) for index in indexes]\n",
" return ngrams"
],
"metadata": {
"id": "9ZCI3uOwN9Tm"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "rPgQBKCb8Zhr"
},
"outputs": [],
"source": [
"# Wrangle extracted sentences into sentence trigrams\n",
"all_data = []\n",
"\n",
"u_docs = sentences_final['doc_id'].unique()\n",
"\n",
"for n in range(len(u_docs)):\n",
" one_doc = sentences_final[sentences_final['doc_id'] == u_docs[n]]\n",
" text = one_doc['text'].tolist()\n",
"\n",
" if len(text) < 3:\n",
" sentence_trigram = [' '.join(text)]\n",
" else:\n",
" # create trigrams\n",
" sentence_trigram = create_ngram(text)\n",
"\n",
" # extract the first row excluding the 3th column\n",
" doc_id = one_doc['doc_id'].iloc[0]\n",
"\n",
" # create a DataFrame with sentence trigrams\n",
" sentence_df = pd.DataFrame({'sent_trigram': sentence_trigram})\n",
"\n",
" # add an 'id' and 'doc_id' column as standard refeferences\n",
" sentence_df['id'] = range(1, len(sentence_trigram) + 1)\n",
" sentence_df['doc_id'] = int(doc_id)\n",
"\n",
" all_data.append(sentence_df)\n",
"\n",
"# concatenate all the new data into a final DataFrame\n",
"final = pd.concat(all_data, ignore_index=True)"
]
},
{
"cell_type": "markdown",
"source": [
"### Step 2: Detecting authoritarian discourse 🌡\n",
"With the extracted sentence trigrams, we can proceed to the second step: detecting authoritarian discourse."
],
"metadata": {
"id": "DjTSUVHIwggU"
}
},
{
"cell_type": "code",
"source": [
"# Load the authdetect model from the Huggingface Hub.\n",
"model = cl.ClassificationModel(\"roberta\", \"mmochtak/authdetect\")"
],
"metadata": {
"id": "nlEPS7zQwygU"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"# Annotate the prepared trigrams with the authdetect model.\n",
"prediction = model.predict(to_predict = final[\"sent_trigram\"].tolist())\n",
"\n",
"anno_df = final.assign(predict = prediction[1])"
],
"metadata": {
"id": "RyEbfsrSxPRa"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"# Check the result. The `anno_df` data frame now contains an additional column\n",
"# called \"predict\" with the predictions made by the model. Since the classification model\n",
"# predicts the label (score) on a continuous scale, similar to a regression model,\n",
"# it can produce scores above or below the scale used for training (polyarchy index;\n",
"# 0-1 where 0 means almost exclusively associated with authoritarian discourse and\n",
"# 1 means almost exclusively associated with democratic discourse). Higher means more democratic,\n",
"# lower means more authoritarian discourse.\n",
"# It is worth mentioning that annotating 193 speeches containing 19,852 sentence trigrams\n",
"# took approximately 50 seconds (on a T4 GPU).\n",
"print(anno_df)"
],
"metadata": {
"id": "DXujhnV9yKhk"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"# Save the annotated data as a .csv file. The new file will be located in the same\n",
"# directory as the input file you uploaded at the beginning. If it does not\n",
"# appear automatically, click the refresh button (the circled arrow) to reload\n",
"# the folder's content. To download the file, right-click on it and select \"Download\"\n",
"# to save it to your local machine.\n",
"anno_df.to_csv('output_sent_trigram.csv', index=False)"
],
"metadata": {
"id": "x3sZwEXfzeqX"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"# In order to aggregate the scores of individual sentence trigrams back to the speech level,\n",
"# various strategies can be applied. The two most intuitive are the simple mean and the ratio of\n",
"# authoritarian trigrams.\n",
"# 'auth' is calculated as one minus the simple average of all trigrams per document.\n",
"# This gives a more intuitive 0-1 scale, where a higher score is more associated with authoritarian discourse.\n",
"#'authesent' captures the ratio of sentences that are associated with authoritarian discourse more than\n",
"# with democratic discourse (i.e., the assigned polyarchy index is lower than 0.5).\n",
"anno_df_speech = anno_df.groupby('doc_id').agg(\n",
" auth=('predict', lambda x: 1 - x.mean()),\n",
" authsent=('predict', lambda x: (x < 0.5).mean())\n",
").reset_index()"
],
"metadata": {
"id": "SnjVWdclxdrK"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"# Finally, concatenate the speech level scores with the original dataset (the one with the raw text)\n",
"# and save it as a .csv file.\n",
"df_final = pd.concat([df, anno_df_speech[['auth', 'authsent']]], axis=1)\n",
"df_final.to_csv('output_speeches_auth.csv', index=False)"
],
"metadata": {
"id": "2XuhxZtW3i_C"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"## Closing remarks 👋\n",
"The tutorial has guided you through the entire annotation pipeline. It demonstrated how easy it is to set up and execute the authdetect model on your own data. The result is a plain data frame with annotated sentence trigrams or whole speeches that can be further processed or aggregated for specific research purposes (e.g., speakers, time periods, or groups)."
],
"metadata": {
"id": "I4iiq0MF0quY"
}
}
]
}