Add application file

pages/.ipynb_checkpoints/1_📊_Define_Target_Variables-checkpoint.py

@@ -0,0 +1,266 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+"""
+Created on Thu May 25 15:16:50 2023
+
+@author: daliagala
+"""
+
+### LIBRARIES ###
+import streamlit as st
+import pandas as pd 
+from sklearn.model_selection import train_test_split
+from utils import assign_labels_by_probabilities, drop_data, train_and_predict
+
+### PAGE CONFIG ###
+st.set_page_config(page_title='EquiVar', page_icon=':robot_face:', layout='wide')
+
+hide_st_style = """
+            <style>
+            #GithubIcon {visibility: hidden;}
+            #MainMenu {visibility: hidden;}
+            footer {visibility: hidden;}
+            header {visibility: hidden;}
+            </style>
+            """
+st.markdown(hide_st_style, unsafe_allow_html=True)
+
+### IMPORT DATA FILES ###
+dataframe = pd.read_csv('./data/dataframe.csv')
+dataframe = dataframe.drop(["Unnamed: 0"], axis = 1)
+dataframe = dataframe.rename(columns={"education_level": "education level"})
+
+### DICTIONARIES AND CONSTANTS###
+groups = ["attention", "reasoning", "memory", "behavioural restraint", "information processing speed"]
+
+groups_dict = {
+          'Divided Visual Attention' : 'attention',
+          'Forward Memory Span': 'memory',
+          'Arithmetic Reasoning' : 'reasoning',
+          'Grammatical Reasoning' : 'reasoning',
+          'Go/No go' : 'behavioural restraint',
+          'Reverse_Memory_Span' : 'memory',
+          'Verbal List Learning' : 'memory',
+          'Delayed Verbal List Learning' : 'memory',
+          'Digit Symbol Coding' : 'information processing speed',
+          'Trail Making Part A' : 'information processing speed',
+          'Trail Making Part B' : 'information processing speed'
+        }
+
+education_dict = {
+    1: 'Some high school',
+    2: 'High school diploma / GED',
+    3: 'Some college',
+    4: 'College degree',
+    5: 'Professional degree',
+    6: "Master's degree",
+    7: 'Ph.D.',
+    8: "Associate's degree",
+    99: 'Other'
+  }
+
+df_keys_dict = {
+          'Divided Visual Attention' :'divided_visual_attention',
+          'Forward Memory Span' :'forward_memory_span',
+          'Arithmetic Reasoning' : 'arithmetic_problem_solving',
+          'Grammatical Reasoning'  : 'logical_reasoning',
+          'Go/No go': 'adaptive_behaviour_response_inhibition',
+          'Reverse_Memory_Span' : 'reverse_memory_span',
+          'Verbal List Learning': 'episodic_verbal_learning',
+          'Delayed Verbal List Learning': 'delayed_recall',
+          'Digit Symbol Coding': 'abstract_symbol_processing_speed',
+          'Trail Making Part A' :'numerical_info_processing_speed',
+          'Trail Making Part B': 'numerical_and_lexical_info_processing_speed'
+        }
+
+
+### CREATE THE "TARGET VARIABLE DEFINITION" PAGE ###
+st.title('Target variable definition')
+
+st.markdown('''On this page, we invite you to imagine that you are hiring for a certain role. Using the sliders below, you will specify two different notions of a “good employee” for that role—two different target variables. Once you’re done, the simulator will build two models, one for each of your target variable definitions. You can visualize these datasets and models—and their effects on fairness and overall features of the models and data—in the Visualize the Results page.''')
+
+st.markdown('''You specify the notions of different employees by assigning weights of importance to cognitive characteristics that a “good” employee would have for the role. The cognitive test data that you’ll be working with comes from real-world people, mirroring an increasing number of hiring algorithms that are based cognitive tests ([Wilson, et al. 2021](https://dl.acm.org/doi/10.1145/3442188.3445928)).''')
+
+st.markdown('''We have pre-set the weights below to reflect two different conceptions of a “good” employee: one conception that emphasizes attentiveness and numerical skills (A) and another conception that emphasizes interpersonal skills and memory (B). If you want to, you can change the slider values as you see fit.''')
+
+st.markdown('''After you’ve set the slider values, click :red[“Assign labels and train your models”]. The simulator will then label certain individuals as "good employees"—in other words it will assign class labels "1" for successful and "0" for not good, based on your selections. Then, the simulator will build the two models.''')
+
+st.markdown('''To learn more about target variable definition and hiring models based in cognitive tests, :green["See explanation"] below.''')
+
+with st.expander("See explanation"):
+    
+    st.markdown('''The models that the simulator builds are of a kind increasingly used in hiring software: companies will have applicants play games that test for different kinds of cognitive ability (like reasoning or memory). Then hiring software will be built to predict which applicants will be successful based on which cognitive characteristics they have. What cognitive characteristics make for a successful employee? This will depend on what role is being hired. And it will also depend on how one defines “successful employee.”''')
+    
+    st.markdown('''In the real world, “successful employee” is defined for these kinds of hiring models in the following way. Managers select a group of current employees that they consider to be successful; this group of employees plays the cognitive test games. A model is then trained to identify applicants who share cognitive characteristics with the current employees that are considered successful. The target variable of “successful employee” is thus defined in terms of comparison to certain people who are deemed successful. ''')
+
+    st.markdown('''One will get different target variables if one deems different current employees as the successful ones. And, as we discussed in the Home page (and as we explain more in the Putting the Idea into Practice page), there will likely be disagreement between managers about which employees are successful. For instance, a manager who values attentiveness and numerical skills will deem different employees “successful” than a manager who values interpersonal skills and memory. Even when different managers roughly share their sensibilities in what characteristics make for a successful employee, there may still be different, equally good ways to “weight” the importance of the various characteristics.''')
+    
+    st.markdown('''In the real world, the cognitive characteristics shared by those considered successful employees is implicit. Companies do not first identify the cognitive characteristics that make for a successful employee; rather, they identify employees who they consider successful, and then the hiring model works backwards to identify what characteristics these employees share.''')
+    
+    st.markdown('''In our simulator, the cognitive characteristics shared by “good” employees are explicit. You assign different weights—using the sliders—to the cognitive characteristics you think are more or less important in a good employee (for the role you’re considering). To illustrate how different target variables have different effects on fairness and overall model attributes, you’ll define “good” employee in two ways. (We’ve made the cognitive characteristics explicit both so you can see the point of different target variable definitions more clearly, and because of limitations of the data that we’re working with.)''')
+
+    st.markdown('''The cognitive characteristics that the simulator works with are from one of datasets of the [NeuroCognitive Performance Test](https://www.nature.com/articles/s41597-022-01872-8). This dataset has eleven different tests which we have grouped into five categories: ''')
+    
+    st.markdown(
+          """
+          - **Memory**: Forward Memory Span, Reverse Memory Span, Verbal List Learning, Delayed Verbal List Learning
+          - **Information Processing Speed**: Digit Symbol Coding, Trail Making Part A, Trail Making Part B
+          - **Reasoning**: Arithmetic Reasoning, Grammatical Reasoning
+          - **Attention**: Divided Visual Attention
+          - **Behavioral Restraint**: Go/No go
+          """)
+    
+    st.markdown('''After you’ve set the weights to these five characteristics using the sliders, you can see which weights are assigned to each test (e.g. Forward Memory Span or Digit Symbol Coding) by ticking the checkbox beneath the sliders. ''')
+
+col1, col2 = st.columns(2)
+
+#Initialise slider values
+list_values_A = (9, 10, 2, 1, 5)
+list_values_B = (1, 2, 10, 9, 3)
+
+selectionsA = {}
+selectionsB = {}
+results_dict_A = groups_dict
+results_dict_B = groups_dict
+
+with col1:
+    st.subheader("Define target variable for model A ")
+        
+    if "slider_values_A" not in st.session_state:
+        for count, value in enumerate(groups):
+            selectionsA[value] = list_values_A[count]
+        st.session_state["slider_values_A"] = selectionsA
+    else:
+        selectionsA = st.session_state["slider_values_A"]
+            
+    for i in groups:
+        nameA = f"{i} importance, model A"
+        value = selectionsA[i]
+        slider = st.slider(nameA, min_value=0, max_value=10, value = value)
+        selectionsA[i] = slider
+        
+    results_dict_A = {k: selectionsA.get(v, v) for k, v in results_dict_A.items()}
+    total = sum(results_dict_A.values())
+    for (key, u) in results_dict_A.items():
+      if total != 0:
+        w = (u/total)
+        results_dict_A[key] = w
+
+    if st.checkbox("Show target variable A weights per subtest", key="A"):
+      for (key, u) in results_dict_A.items():
+        txt = key.replace("_", " ")
+        st.markdown("- " + txt + " : " + f":green[{str(round((u*100), 2))}]")
+        
+    
+    st.session_state["slider_values_A"] = selectionsA
+
+
+with col2:
+    st.subheader("Define target variable for model B ")
+        
+    if "slider_values_B" not in st.session_state:
+        for count, value in enumerate(groups):
+            selectionsB[value] = list_values_B[count]
+        st.session_state["slider_values_B"] = selectionsB
+    else:
+        selectionsB = st.session_state["slider_values_B"]
+            
+    for i in groups:
+      nameB = f"{i} importance, model B"
+      value = selectionsB[i]
+      slider = st.slider(nameB, min_value=0, max_value=10, value = value)
+      selectionsB[i] = slider
+      
+    results_dict_B = {k: selectionsB.get(v, v) for k, v in results_dict_B.items()}
+    total = sum(results_dict_B.values())
+    for (key, u) in results_dict_B.items():
+      if total != 0:
+        w = ((u/total))
+        results_dict_B[key] = w
+
+    if st.checkbox("Show target variable B weights per subtest", key = "B"):
+      for (key, u) in results_dict_B.items():
+        txt = key.replace("_", " ")
+        st.markdown("- " + txt + " : " + f":green[{str(round((u*100), 2))}]")
+      
+    st.session_state["slider_values_B"] = selectionsB
+          
+if st.button("Assign labels and train your models", type = "primary", use_container_width = True):
+  if 'complete_df' in st.session_state:
+      del st.session_state['complete_df']
+  if 'clean_df' in st.session_state:
+      del st.session_state['clean_df']
+  if 'cm_A' in st.session_state:
+      del st.session_state['cm_A']
+  if 'cm_B' in st.session_state:
+      del st.session_state['cm_B']
+  scoreA = pd.DataFrame()
+  scoreB = pd.DataFrame()
+  test1 = all(value == 0 for value in results_dict_A.values())
+  test2 = all(value == 0 for value in results_dict_B.values())
+  if test1 == True or test2 == True:
+    st.error('Cannot train the models if you do not define the target variables. Make your selections for both models first!', icon="🚨")
+  else:
+    for (key, u) in results_dict_A.items():
+      scoreA[df_keys_dict[key]] = u * dataframe[df_keys_dict[key]]
+      scoresA = scoreA.sum(axis=1)
+      dataframe['model_A_scores'] = scoresA
+    for (key, u) in results_dict_B.items():
+      scoreB[df_keys_dict[key]] = u * dataframe[df_keys_dict[key]]
+      scoresB = scoreB.sum(axis=1)
+      dataframe['model_B_scores'] = scoresB
+    
+    new_annotated = assign_labels_by_probabilities(dataframe, "model_A_scores", "Model_A_label", "Model_A_probabilities", quantile=0.85, num_samples=100)
+    new_annotated = assign_labels_by_probabilities(new_annotated, "model_B_scores", "Model_B_label", "Model_B_probabilities", quantile=0.85, num_samples=100)
+    new_annotated = new_annotated.reset_index()
+
+
+    clean_data = drop_data(new_annotated)
+    # specify the columns of interest
+    selected_cols = ['Model_A_label', 'Model_B_label']
+    
+    # count the number of rows where all three selected columns have a value of 1
+    num_rows_with_all_flags_1 = len(new_annotated[new_annotated[selected_cols].sum(axis=1) == len(selected_cols)])
+    
+    # print the result
+    st.write(f"Shared candidates between your target variables: :green[{num_rows_with_all_flags_1}].")
+    with st.spinner('Please wait... The models will be trained now.'):
+
+      X_data, Y_data_A, Y_data_B = clean_data.iloc[:, :-2], clean_data.iloc[:, [-2]], clean_data.iloc[:, [-1]]
+      X_data = X_data.drop(["index"], axis = 1)
+      Y_data_B = Y_data_B.reset_index()
+      X_train, X_test, y_train_A, y_test_A = train_test_split(X_data, Y_data_A, test_size=0.2)
+      y_train_A = y_train_A.reset_index()
+      y_test_A = y_test_A.reset_index()
+      y_train_B = pd.merge(y_train_A,Y_data_B[['index', 'Model_B_label']],on='index', how='left')
+      y_test_B = pd.merge(y_test_A,Y_data_B[['index', 'Model_B_label']],on='index', how='left')
+      y_train_B = y_train_B.drop(labels='Model_A_label', axis = 1)
+      y_test_B = y_test_B.drop(labels='Model_A_label', axis = 1)
+      y_train_A = y_train_A.set_index("index")
+      y_train_B = y_train_B.set_index("index")
+      y_test_A = y_test_A.set_index("index")
+      y_test_B = y_test_B.set_index("index")
+
+      accuracy_A, precision_A, recall_A, X_full_A, cm_A, baseline_accuracy_A = train_and_predict("A", X_train, X_test, y_train_A, y_test_A)
+      accuracy_B, precision_B, recall_B, X_full_B, cm_B, baseline_accuracy_B = train_and_predict("B", X_train, X_test, y_train_B, y_test_B)
+      full = pd.merge(X_full_A,X_full_B[['index','Predicted_B', 'Prob_0_B', "Prob_1_B"]],on='index', how='left')
+      complete = pd.merge(full,new_annotated[['index', 'age', 'gender', 'education level', 'country', 'Model_A_label', 'Model_B_label', 'model_A_scores', 'model_B_scores']],on='index', how='left')
+      complete=complete.replace({"education level": education_dict})
+      complete = complete.rename(columns={"index": "Candidate ID"})
+      
+      if 'complete_df' not in st.session_state:
+        st.session_state['complete_df'] = complete
+      if 'clean_df' not in st.session_state:
+        st.session_state['clean_df'] = clean_data
+      if 'cm_A' not in st.session_state:
+        st.session_state['cm_A'] = cm_A
+      if 'cm_B' not in st.session_state:
+        st.session_state['cm_B'] = cm_B
+
+    row1_space1, row1_1, row1_space2, row1_2, row1_space3 = st.columns((0.1, 3, 0.1, 3, 0.1))
+    with row1_1:
+      st.write(f"Model A accuracy: :green[{baseline_accuracy_A}].")
+    with row1_2:
+      st.write(f"Model B accuracy: :green[{baseline_accuracy_B}].")
+
+    st.success('''Success! You have defined the target variables and trained your models. Head to "Visualise the Results" in the sidebar.''')

pages/.ipynb_checkpoints/2_📈_Visualize_the_Results-checkpoint.py

@@ -0,0 +1,593 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+"""
+Created on Thu May 25 15:30:56 2023
+
+@author: daliagala
+"""
+
+### LIBRARIES ###
+import streamlit as st
+import numpy as np
+import pandas as pd 
+import matplotlib.pyplot as plt
+import plotly.express as px
+import plotly.graph_objects as go
+from matplotlib_venn import venn2
+from sklearn.metrics import confusion_matrix
+from utils import display_proportional, plot_data, run_PCA, create_confusion_matrix_heatmap, plot_conf_rates
+
+### PAGE CONFIG ###
+st.set_page_config(page_title='EquiVar', page_icon=':robot_face:', layout='wide')
+
+hide_st_style = """
+            <style>
+            #GithubIcon {visibility: hidden;}
+            #MainMenu {visibility: hidden;}
+            footer {visibility: hidden;}
+            header {visibility: hidden;}
+            </style>
+            """
+st.markdown(hide_st_style, unsafe_allow_html=True)
+
+### DICTIONARIES AND CONSTANTS###
+
+colours_education = {
+          'Some high school' : 'indigo',
+          'High school diploma / GED' : '#7ae99e',
+          'Some college' : '#0a68c9',
+          'College degree': '#80c4fa',
+          'Professional degree': '#f48508',
+          "Master's degree" : '#2fa493',
+          'Ph.D.' : '#f2a3a1',
+          "Associate's degree" : '#fbcc66',
+          'Other' : '#fa322f'
+        }
+
+colours_country = {
+          'AU' : '#7ae99e',
+          'US': '#80c4fa',
+          'NZ': '#2fa493',
+          'CA' : '#fbcc66'
+        }
+
+colours_gender = {
+          'f' : '#83C9FF',
+          'm': '#0067C9'
+        }
+
+characteristic_dict = {
+          'gender' : colours_gender,
+          'education level' : colours_education,
+          'country' : colours_country,
+          'age' : 'indigo'
+        }
+
+pred_dict = {
+          'Model A' : 'Predicted_A',
+          'Model B' : 'Predicted_B'
+        }
+
+prob_dict = {
+          'Model A' : 'Prob_1_A',
+          'Model B' : 'Prob_1_B'
+        }
+
+model_dict = {
+          'Model A' : 'Model_A_label',
+          'Model B' : 'Model_B_label'
+        }
+
+df_keys_dict = {
+          'Divided Visual Attention' :'divided_visual_attention',
+          'Forward Memory Span' :'forward_memory_span',
+          'Arithmetic Reasoning' : 'arithmetic_problem_solving',
+          'Grammatical Reasoning'  : 'logical_reasoning',
+          'Go/No go': 'adaptive_behaviour_response_inhibition',
+          'Reverse_Memory_Span' : 'reverse_memory_span',
+          'Verbal List Learning': 'episodic_verbal_learning',
+          'Delayed Verbal List Learning': 'delayed_recall',
+          'Digit Symbol Coding': 'abstract_symbol_processing_speed',
+          'Trail Making Part A' :'numerical_info_processing_speed',
+          'Trail Making Part B': 'numerical_and_lexical_info_processing_speed'
+        }
+
+### DEFINE ALL SUB-PAGES AS FUNCTIONS TO CALL ###
+
+def mod_prop(cmA, cmB):
+    st.markdown('''This section contains model confusion matrices, which give us a lot of information about how good the models which we produced really are. We obtain the confusion matrices by plotting "Actual" labels on one axis, and "Predicted" labels on the other. For both models, in each square of each confusion matrix, we can see which group of candidates it represents, the number of candidates in this group, and what percentage of all candidates assessed they represent.''')
+    st.markdown('''
+    - **True Negative (TN)**: candidates predicted to have label "0" whose actual label was "0".
+    - **False Positive (FP)**: candidates predicted to have label "1" whose actual label was "0".
+    - **False Negative (FN)** candidates predicted to have label "0" whose actual label was "1".
+    - **True Positive (TP)**: candidates predicted to have label "1" whose actual label was "1".
+    ''')
+    
+    row1_space1, row1_1, row1_space2, row1_2, row1_space3 = st.columns((0.1, 3, 0.1, 3, 0.1))
+    with row1_1:
+      create_confusion_matrix_heatmap(cmB, "Model A")
+    
+    with row1_2:
+      model = "B"
+      create_confusion_matrix_heatmap(cmB, "Model B")
+    
+    st.subheader("Model Accuracy Rates")
+    st.markdown('''We can also represent each model in terms of its accuracy rates, calculated using the numbers of candidates in each group shown in the confusion matrix:''')
+    st.markdown('''
+    - True Positive Rate (TPR), also called recall, sensitivity or hit rate, is the probability of a positive test result when the result is indeed positive.
+    - True Negative Rate (TNR), or specificity/selectivity, is the probability of a negative test result when the test is indeed negative.
+    - Positive Predictive Value (PPV) or Precision is the ratio of truly positive results to all positive results.
+    - Negative Predictive Value (NPR) is the ratio of truly negative results to all negative results.
+    - False Positive Rate (FPR) or fall-out is the probability of assigning a falsely positive result when the test is negative.
+    - False Negative Rate (FNR) or miss rate is the probability that a test with a positive result will falsely be assigned a negative result.
+    - False Discovery Rate (FDR) it the ratio of false positive results to total number of positive results.
+    ''')
+    measures_A = plot_conf_rates(cmA)
+    measures_B = plot_conf_rates(cmB)
+    fig = go.Figure()
+    fig.add_trace(go.Bar(
+          x=measures_A["Measure"],
+          y=measures_A["Score"],  
+          name='Model A rates',
+          marker_color='rgb(55, 83, 109)'
+      ))
+    fig.add_trace(go.Bar(
+          x=measures_B["Measure"],
+          y=measures_B["Score"],
+          name='Model B rates',
+          marker_color='rgb(26, 118, 255)'
+      ))
+    fig.update_layout(
+          title='Model measures comparison',
+          xaxis_tickfont_size=14,
+          xaxis=dict(
+              title='Measure'),
+          yaxis=dict(
+              title='Score',
+              titlefont_size=16,
+              tickfont_size=14,
+          ),
+          legend=dict(
+              bgcolor='rgba(255, 255, 255, 0)',
+              bordercolor='rgba(255, 255, 255, 0)'
+          ),
+    barmode='group',
+    bargap=0.15, # gap between bars of adjacent location coordinates.
+    bargroupgap=0.1 # gap between bars of the same location coordinate.
+      )
+    st.plotly_chart(fig, use_container_width = True)
+    if st.checkbox("Show tables"):
+        row1_space1, row1_1, row1_space2, row1_2, row1_space3 = st.columns((0.1, 3, 0.1, 3, 0.1))
+        # CSS to inject contained in a string
+        hide_table_row_index = """
+        <style>
+        thead tr th:first-child {display:none}
+        tbody th {display:none}
+        </style> """
+
+        # Inject CSS with Markdown
+        st.markdown(hide_table_row_index, unsafe_allow_html=True)
+        with row1_1:
+            st.markdown("Accuracy rates for model A")
+            st.table(measures_A)
+        with row1_2:
+            st.markdown("Accuracy rates for model B")
+            st.table(measures_B)
+
+def model_scores(dataframe):
+  st.markdown('''This section displays the distribution of scores assigned to each hypothetical employee, according to the values you set on the sliders, compared with the distribution of protected characteristics. These scores are then used to assign labels to the hypothetical employees, creating two distinct datasets to train the models. In essence, these scores explicitly and numerically mimic the viewpoints of hypothetical hiring managers A and B when deciding who to label as "top employees."''')
+  st.markdown('''Just as in the original NCPT dataset analysis, the scores obtained by participants in the cognitive games decline with age. This observation provides insight into the potential issue of ageism inherent in the use of gamified hiring processes.''')
+  # Create a selectbox to choose a protected characteristic to explore
+  plot_radio = st.selectbox('Characteristic to explore', characteristic_dict.keys())
+  row2_space1, row2_1, row2_space2 = st.columns((0.1, 5, 0.1))
+
+  with row2_1:
+    data = dataframe[["model_A_scores", "model_B_scores", plot_radio]]
+
+    if plot_radio == "age":
+        bins= [18,20,30,40,50,60,70,80,90]
+        labels = ['18-20','21-30','31-40','41-50','51-60','61-70','71-80','81-90']
+        data['age_bins'] = pd.cut(data['age'], bins=bins, labels=labels, right=False)
+        plot_radio = 'age_bins'
+    colours = ['rgba(93, 164, 214, 0.5)', 'rgba(255, 144, 14, 0.5)']
+    c1, c2, c3, c4, c5 = st.columns((0.1, 3, 0.1, 3, 0.1))
+    with c2:
+        fig = px.box(data, x = plot_radio, y="model_A_scores", labels={"model_A_scores":"Dataset A Input Scores", 'age_bins':"Age"}, title="Input scores in dataset A")
+        fig.update_layout(showlegend=False)
+        fig.update_traces(marker_color='rgba(93, 164, 214, 0.5)')
+        st.plotly_chart(fig, use_container_width=True)
+    with c4:
+        fig = px.box(data, x = plot_radio, y="model_B_scores", labels={"model_B_scores":"Dataset B Input Scores", 'age_bins':"Age"}, title="Input scores in dataset B")
+        fig.update_traces(marker_color = 'rgba(255, 144, 14, 0.5)')
+        fig.update_layout(showlegend=False)
+        st.plotly_chart(fig, use_container_width=True)
+
+def PCA_general(full_df, dataframe_PCA):
+  st.markdown('''On this page, you can see the distribution of the dataset labels which were assigned based on the scores calculated from the slider values you selected previously. Principal Components Analysis, or PCA, is a technique often used to analyse and subsequently visualize datasets where there are many features per single example. This is the case with the NCPT dataset used in our simulator. Specifically, the battery which we used has 11 features per single example, the example being the player of cognitive games, and, in our metaphor, a hypothetical employee or job candidate. It is impossible to plot 11 dimensions, and PCA allows for the visualisation of multidimensional data, while also preserving as much information as possible.''')
+  choice = st.radio("What would you like to explore?", ("PCAs", "Components loading"), horizontal = True)
+  pcaA, dfA, labelsA, coeffA, componentsA = run_PCA(dataframe_PCA, 'Model_B_label', 'Model_A_label', 2)
+  pcaB, dfB, labelsB, coeffB, componentsB = run_PCA(dataframe_PCA, 'Model_A_label', 'Model_B_label', 2)
+  loadings = pcaB.components_.T * np.sqrt(pcaB.explained_variance_)
+  total_var = pcaA.explained_variance_ratio_.sum() * 100
+  dfA = dfA.rename(columns={'target': 'Dataset A'}).reset_index()
+  dfB = dfB.rename(columns={'target': 'Dataset B'}).reset_index()
+  df_all = pd.merge(dfA, dfB[['index', 'Dataset B']], on='index', how='left')
+
+  conditions = [
+      (df_all['Dataset A'] == 1) & (df_all['Dataset B'] == 0),
+      (df_all['Dataset B'] == 1) & (df_all['Dataset A'] == 0),
+      (df_all['Dataset A'] == 1) & (df_all['Dataset B'] == 1),
+      (df_all['Dataset A'] == 0) & (df_all['Dataset B'] == 0)]
+      
+  values = ['Selected A', 'Selected B', 'Selected both', 'Not selected']
+  df_all['All'] = np.select(conditions, values)
+
+  df_all = df_all.drop(["index"], axis = 1)
+  df_all.All=pd.Categorical(df_all.All,categories=['Not selected', 'Selected A', 'Selected B', 'Selected both'])
+  df_all=df_all.sort_values('All')
+
+  selections_dict = {0: 'Not selected', 1: 'Selected'}
+  df_all = df_all.replace({"Dataset A": selections_dict, "Dataset B": selections_dict})
+
+  color_dict_sel = {'Not selected': '#3366CC', 'Selected': 'grey'}
+
+  if "pca_df" not in st.session_state:
+    st.session_state.pca_df = df_all
+
+  if choice == "PCAs":
+    c1, c2 = st.columns(2)
+    with c1:
+      fig = px.scatter(st.session_state.pca_df, 
+                      x=st.session_state.pca_df['principal component 1'].astype(str),
+                      y=st.session_state.pca_df['principal component 2'].astype(str),
+                      title='Dataset A PCA',
+                      labels={"x": 'PC 1', "y": 'PC 2'},
+                      color=st.session_state.pca_df['Dataset A'],
+                      color_discrete_map=color_dict_sel)
+      fig.update_traces(marker_size = 8)
+      st.plotly_chart(fig, use_container_width=True)
+    with c2:
+      fig = px.scatter(st.session_state.pca_df, 
+                      x=st.session_state.pca_df['principal component 1'].astype(str),
+                      y=st.session_state.pca_df['principal component 2'].astype(str),
+                      title='Dataset B PCA',
+                      labels={"x": 'PC 1', "y": 'PC 2'},
+                      color=st.session_state.pca_df['Dataset B'],
+                      color_discrete_map=color_dict_sel)
+      fig.update_traces(marker_size = 8)
+      st.plotly_chart(fig, use_container_width=True)
+      
+    st.markdown(f'''These plots show the reduction of 11 dimensions (11 subtest results) to 2 dimensions. Total Variance for the data is {total_var:.2f}%. Both of the datasets have the same features, therefore they both have the same total variance. Total variance value indicates what percentage of information has been preserved when the dimensionality was reduced. Note that for both datasets, A and B, different points are labelled "1" or "0". This shows that the two datasets represent the two different target variable definitions which were created by you previously. The plots are interactive - zoom in to explore in detail.''')
+      
+    pcaA, dfA, labelsA, coeffA, componentsA = run_PCA(dataframe_PCA, 'Model_B_label', 'Model_A_label', 2)
+    pcaB, dfB, labelsB, coeffB, componentsB = run_PCA(dataframe_PCA, 'Model_A_label', 'Model_B_label', 2)
+    loadings = pcaB.components_.T * np.sqrt(pcaB.explained_variance_)
+    total_var = pcaA.explained_variance_ratio_.sum() * 100
+    dfA = dfA.rename(columns={'target': 'Dataset A'}).reset_index()
+    dfB = dfB.rename(columns={'target': 'Dataset B'}).reset_index()
+    df_all = pd.merge(dfA, dfB[['index', 'Dataset B']], on='index', how='left')
+
+    conditions = [
+        (df_all['Dataset A'] == 1) & (df_all['Dataset B'] == 0),
+        (df_all['Dataset B'] == 1) & (df_all['Dataset A'] == 0),
+        (df_all['Dataset A'] == 1) & (df_all['Dataset B'] == 1),
+        (df_all['Dataset A'] == 0) & (df_all['Dataset B'] == 0)]
+        
+    values = ['Selected A', 'Selected B', 'Selected both', 'Not selected']
+    df_all['All'] = np.select(conditions, values)
+
+    df_all = df_all.drop(["index"], axis = 1)
+    df_all.All=pd.Categorical(df_all.All,categories=['Not selected', 'Selected A', 'Selected B', 'Selected both'])
+    df_all=df_all.sort_values('All')
+
+    selections_dict = {0: 'Not selected', 1: 'Selected'}
+    df_all = df_all.replace({"Dataset A": selections_dict, "Dataset B": selections_dict})
+
+    if "pca_df" not in st.session_state:
+      st.session_state.pca_df = df_all
+
+    fig = px.scatter(st.session_state.pca_df, 
+                      x=st.session_state.pca_df['principal component 1'],
+                      y=st.session_state.pca_df['principal component 2'],
+                      title="PCA with labelled groups",
+                      color=st.session_state.pca_df["All"], 
+                      width = 800, height = 800,
+                      color_discrete_sequence=px.colors.qualitative.Safe,
+                      opacity = 0.95)
+      
+    fig.update_yaxes(
+        scaleanchor="x",
+        scaleratio=1,
+      )
+    fig.update_traces(marker_size = 10)
+    st.plotly_chart(fig)
+
+  if choice == "Components loading":
+      c1,c2 = st.columns(2)
+      loadings_df = pd.DataFrame(loadings, columns = ["PC1", "PC2"])
+      labels_A_proper = { v:k for k,v in df_keys_dict.items()}
+      loadings_df["Features"] = labels_A_proper.values()
+      with c1:
+          fig = px.bar(loadings_df, x="PC1", y="Features", orientation = 'h')
+          st.plotly_chart(fig, use_container_width = True)
+      with c2:
+          fig = px.bar(loadings_df, x="PC2", y="Features", orientation = 'h')
+          st.plotly_chart(fig, use_container_width = True)
+
+    #   fig = go.Figure()
+    #   fig.add_trace(go.Bar(
+    #       x=loadings_df["PC1"],
+    #       y=loadings_df["Features"],  
+    #       name='Principal Component 1',
+    #       marker_color='rgb(55, 83, 109)',
+    #       orientation='h'
+    #   ))
+    #   fig.add_trace(go.Bar(
+    #       x=loadings_df["PC2"],
+    #       y=loadings_df["Features"],
+    #       name='Principal Component 2',
+    #       marker_color='rgb(26, 118, 255)',
+    #       orientation='h'
+    #   ))
+    #   fig.update_layout(
+    #       title='Component loadings',
+    #       xaxis_tickfont_size=14,
+    #       xaxis=dict(
+    #           title='Loading value'),
+    #       yaxis=dict(
+    #           title='Feature',
+    #           titlefont_size=16,
+    #           tickfont_size=14,
+    #       ),
+    #       legend=dict(
+    #           bgcolor='rgba(255, 255, 255, 0)',
+    #           bordercolor='rgba(255, 255, 255, 0)'
+    #       ),
+    # barmode='group',
+    # bargap=0.15, # gap between bars of adjacent location coordinates.
+    # bargroupgap=0.1 # gap between bars of the same location coordinate.
+    #   )
+    #   st.plotly_chart(fig, use_container_width = True)
+      
+      st.markdown('''On this plot, PCA component loadings can be explored. These facilitate the understanding of how much each variable (which there are 11 of) contributes to a particular principal component. Here, the 11 variables were reduced to 2 components, which are labelled PC1 and PC2. The magnitude of the loading (here displayed as the size of the bar in the bar chart) indicates how strong the relationship between the variable and the component is. Therefore, the higher the bar, the stronger the relationship between that component and that variable. The loading's sign can be positive or negative. This indicates whether the principal component and that variable are positively or negatively correlated. We can see that multiple variables are positively correlated with PC2. Two variables, episodic verbal learning and delayed recall are negatively correlated with both of the components.''')
+
+
+def model_out(full_df):
+  st.markdown('''This section highlights the discrepancies between your two models when presented with the same pool of new, previously unseen candidates to label. Specifically, you'll be investigating the candidates assigned a "1" label by both models. These individuals would be those considered for a job interview or chosen for the role, according to your defined target variable.''')
+  # Create a selectbox to choose a protected characteristic to explore
+  selectbox = st.selectbox('Characteristic to explore', characteristic_dict.keys())
+  representation = st.selectbox("Representation", ("absolute", "proportional"))
+  row1_space1, row1_1, row1_space2, row1_2, row1_space3 = st.columns((0.1, 3, 0.1, 3, 0.1))
+  with row1_1:
+    st.subheader("Candidates selected by model A")
+
+    if representation == "absolute":
+      # Select predicted data ==1
+      data = full_df.loc[full_df['Predicted_A'] == 1]
+
+      # Use function plot_data to plot selected data
+      plot_data(data, selectbox, characteristic_dict[selectbox])
+    else:
+      display_proportional(full_df, selectbox, 'Predicted_A')
+      
+  with row1_2:
+    st.subheader("Candidates selected by model B")
+    
+    if representation == "absolute":
+      # Select predicted data ==1
+      data = full_df.loc[full_df['Predicted_B'] == 1]
+
+      # Use function plot_data to plot selected data
+      plot_data(data, selectbox, characteristic_dict[selectbox])
+
+    else:
+      display_proportional(full_df, selectbox,'Predicted_B')
+      
+  
+  st.markdown('''In this section, you're comparing the model's selections concerning four protected characteristics: age, gender, education level, and country. You can visualize these differences in two ways: "Absolute" or "Proportional".''')
+  st.markdown('''"Absolute" representation gives you the raw numbers or percentages of each characteristic chosen. For instance, if the model labeled 5 female candidates and 5 male candidates as "1", the "Absolute" outcome will display as 50% for both genders."Proportional" representation, on the other hand, shows the percentage of a group selected by the model relative to the total number of that group in the input data. For example, if the model evaluated 100 male candidates and selected 5, you will see a 5% representation. If it evaluated 200 female candidates and selected 5, it will show a 2.5% representation.''')
+  st.markdown('''If you encounter empty categories in the "Proportional" view, this indicates that while candidates from these categories were evaluated, none were labeled as "1". Hence, their proportional representation amounts to 0%.''')
+
+def dataframe_out(full_df):
+  selectbox_M = st.selectbox('Choose which model output to rank by', pred_dict.keys())
+
+  # Select data
+  data = full_df.loc[full_df[pred_dict[selectbox_M]] == 1]
+  data = data.sort_values(by = prob_dict[selectbox_M], ascending = False)
+  data = data[['Candidate ID','Prob_1_A', 'Prob_1_B', 'Predicted_A', 'Predicted_B']]
+  data = data.rename(columns={"Prob_1_A": "Ranking, model A", "Prob_1_B": "Ranking, model B", "Predicted_A": "Predicted label A", "Predicted_B": "Predicted label B"})
+  data.index = np.arange(1, len(data) + 1)
+
+  st.table(data.style.background_gradient(subset = ["Ranking, model A", "Ranking, model B"], axis=0, vmin=0.40).highlight_max(color = '#FFCD9B', subset = ["Predicted label A", "Predicted label B"], axis=0))
+  
+  st.markdown("""In this section, you can review the data for all candidates labeled "1" by the selected model, found at the top of the page. Simultaneously, you can observe the labels assigned to these same candidates by the other model. It's likely that there will be instances where candidates chosen by one model weren't selected by the other. Candidates labeled "1" are highlighted in orange in the "Predicted label A" and "Predicted label B" columns.""")
+  st.markdown('''In addition to this, you can see the probability with which each candidate was labeled "1". The intensity of the blue color indicates the candidate's ranking position - a darker blue represents a higher ranking (with 1 being the maximum and 0 the minimum). You may notice that some candidates highly ranked by one model may be ranked significantly lower by the other model.''')
+  
+def venn_diagram(full_df):
+  row2_space1, row2_1, row2_space2, row2_2, row2_space3 = st.columns((0.1, 1, 0.1, 1, 0.1))
+  with row2_1:
+    fig, ax = plt.subplots()
+
+    list_A = full_df.loc[full_df['Predicted_A'] == 1, 'Candidate ID'].astype(int)
+    list_B = full_df.loc[full_df['Predicted_B'] == 1, 'Candidate ID'].astype(int)
+    set1 = set(list_A)
+    set2 = set(list_B)
+
+    venn2([set1, set2], ('Model A', 'Model B'), ax=ax)
+    st.pyplot(fig)
+
+  with row2_2:
+    st.markdown('''This Venn Diagram visualizes the number of candidates chosen by both models. It's likely that some candidates will be selected by both models, while others may be chosen by only one model. If we consider Model A as the decision of one hiring manager and Model B as another's, it's easy to see how the selection outcome varies depending on the decision-maker. Some candidates may get the opportunity to be hired, while others might not. This serves as an illustration of the inherent arbitrariness in defining the target variable when dealing with highly subjective outcomes.''')
+    st.markdown('''For instance, it's straightforward to define a target variable in a classification problem like distinguishing dragonflies from butterflies, where there's little room for ambiguity. However, defining what makes a 'good' employee is far more challenging due to its subjective nature.''')
+    
+
+
+def model_vis(full_df):
+  st.markdown('''In this section, you can visualize the demographics of the different subgroups of the data. Firstly, you can see the demographic characteristics of the candidates who have positive labels ("1") and negative labels ("0") which were assigned based on the scores calculated from the slider values you selected previously. Then, you can visualize the demographic distributions of the data which was used for training and evaluation of the models.''')
+  choice = st.radio("**Select desired data:**", ("Positive and negative labels", "Training and evaluation data"), horizontal=True)
+  if choice == "Positive and negative labels":
+    # Create a selectbox to choose a protected characteristic to explore
+    selectbox_Lab = st.selectbox('Label to visualize', ('positive labels', 'negative labels'))
+
+    # Create a selectbox to choose a protected characteristic to explore
+    selectbox_Char = st.selectbox('Protected characteristic', characteristic_dict.keys())
+
+    row2_space1, row2_1, row2_space2, row2_2, row2_space3 = st.columns((0.1, 3, 0.1, 3, 0.1))
+
+    with row2_1:
+      st.subheader("Dataset A")
+
+      # Select test data
+      if selectbox_Lab == 'positive labels':
+        data = full_df.loc[full_df['Model_A_label'] == 1]
+      else:
+        data = full_df.loc[full_df['Model_A_label'] == 0]
+      
+      # Use function plot_data to plot selected data
+      plot_data(data, selectbox_Char, characteristic_dict[selectbox_Char])
+
+
+    with row2_2:
+      st.subheader("Dataset B")
+
+      # Select test data
+      if selectbox_Lab == 'positive labels':
+        data = full_df.loc[full_df['Model_B_label'] == 1]
+      else:
+        data = full_df.loc[full_df['Model_B_label'] == 0]
+
+      # Use function plot_data to plot selected data
+      plot_data(data, selectbox_Char, characteristic_dict[selectbox_Char])
+    st.markdown('''You are visualising the demographic composition of those hypothetical employees who were assigned labels "1" or "0" based on your definitions of the target variables. You might see differences in proportions of genders between the two models for the positive labels, as well as a major difference in the age between the positive and negative labels. Visualising the labels in this manner before training the model can help understand and mitigate differences in demographic representation in the modelling outcomes. Likely, if all candidates labelled "1" were in younger age groups, the candidates selected by the model at the deployment stage will also be in younger age groups. Moreover, target variable definition affects the proportional representation. Having defined two target variables, one can choose the dataset and the model which offers more proportional representation.''')
+
+
+  if choice == "Training and evaluation data":
+    # Create a selectbox to choose a protected characteristic to explore
+    selectbox = st.selectbox('Characteristic to explore', characteristic_dict.keys())
+    row1_space1, row1_1, row1_space2, row1_2, row1_space3 = st.columns((0.1, 1, 0.1, 1, 0.1))
+    # Plot training data
+    with row1_1:
+      st.subheader("Training data")
+
+      # Select train data
+      train = full_df.loc[full_df["Predicted_A"] == "train"]
+
+      # Use function plot_data to plot selected data
+      plot_data(train, selectbox, characteristic_dict[selectbox])
+
+    # Plot test data
+
+    with row1_2:
+      st.subheader("Test data")
+
+      # Select test data
+      test = full_df.loc[full_df["Predicted_A"] != "train"]
+
+      # Use function plot_data to plot selected data
+      plot_data(test, selectbox, characteristic_dict[selectbox])
+  
+    st.markdown('''To train a machine learning model, the data has to be split into two different sets. The first set is the training data, which will be used to teach the model the relationships between the input features (11 subtest results) and the corresponding labels ("0" and "1", assigned based on your definitions of target variables and the values you chose for the sliders). The second set is the test data, or evaluation data. It is used to assess the performance of the model. This is the data which is used to plot the confusion matrices and calculate the model metrics which you saw at the bottom of the "Define the target variable" page. This is also the data whose features you can explore in "Modelling outcomes". It is important that the training and testing data are balanced. Here, you can compare the demographic composition of the training and evaluation data. The training and evaluation datasets compositions were the same and contained the same candidates and same features for both models A and B. However, the labels for each dataset were different and based on what you selected in "Define target variable".''')
+
+def filter_for_protected(data):
+  st.markdown('''Sometimes, the overall model metrics can be deceptive when it comes to predicting the results for different groups under consideration. Ideally, for our models, the varying model metrics would be similar across different groups, which would indicate that the overall model performance is reflected in how this model performs for a given group. It is often not the case, and it is likely that you will see that models A and B perform differently when it comes to those metrics. Even the same model can have different metrics for different subgroups.''')
+  model = st.selectbox('Choose which model outputs to assess', pred_dict.keys())
+  test = data.loc[data[pred_dict[model]] != "train"]
+
+  selectbox_Char = st.selectbox('Protected characteristic', characteristic_dict.keys())
+  if selectbox_Char == 'age':
+    bins= [18,20,30,40,50,60,70,80,91]
+    labels = ['18-20','21-30','31-40','41-50','51-60','61-70','71-80','81-90']
+    test['age_bins'] = pd.cut(test['age'], bins=bins, labels=labels, right=False)
+    selectbox_Char = 'age_bins'
+  # which_group = st.selectbox('Which group?', test[selectbox_Char].unique())
+  df = pd.DataFrame({'Measure': ['True Positive Rate', 'True Negative Rate', 'Positive Predictive Value', 'Negative Predictive Value', 'False Positive Rate', 'False Negative Rate', 'False Discovery Rate']})
+  for group in test[selectbox_Char].unique():
+      rslt_df = test[test[selectbox_Char] == group] 
+      y_true = [int(numeric_string) for numeric_string in rslt_df[model_dict[model]]]
+      y_pred = [int(numeric_string) for numeric_string in rslt_df[pred_dict[model]]]
+      cm = confusion_matrix(y_true, y_pred)
+      if cm.shape == (1,1):
+          cm = np.array([[cm[0, 0], 0], [0, 0]])
+      d = plot_conf_rates(cm)
+      df[f"{group}"] = d["Score"]
+
+  fig = go.Figure()
+  for group in test[selectbox_Char].unique():
+      fig.add_trace(go.Bar(
+          x=df["Measure"],
+          y=df[group],  
+          name=group
+      ))
+      
+  fig.update_layout(
+          title='Model metrics per group',
+          xaxis_tickfont_size=14,
+          xaxis=dict(
+              title='Metric'),
+          yaxis=dict(
+              title='Score',
+              titlefont_size=16,
+              tickfont_size=14,
+          ),
+          legend=dict(
+              bgcolor='rgba(255, 255, 255, 0)',
+              bordercolor='rgba(255, 255, 255, 0)'
+          ),
+    barmode='group',
+    bargap=0.15, # gap between bars of adjacent location coordinates.
+    bargroupgap=0.1 # gap between bars of the same location coordinate.
+      )
+  st.plotly_chart(fig, use_container_width = True)
+  if st.checkbox("Show table of scores"):
+    # CSS to inject contained in a string
+    hide_table_row_index = """
+    <style>
+    thead tr th:first-child {display:none}
+    tbody th {display:none}
+    </style> """
+
+    # Inject CSS with Markdown
+    st.markdown(hide_table_row_index, unsafe_allow_html=True)
+    st.markdown(f"Accuracy rates for {selectbox_Char}")
+    st.table(df)
+
+def data_plot(key1, key2, key3, key4):
+  st.title('''Visualize the Results''')
+  if key1 not in st.session_state:
+    st.error('Cannot train the models if you do not define the target variables. Go to "Define Target Variables"!', icon="🚨")
+  else:
+    tab1, tab2 = st.tabs(["Demographic", "Non-demographic"])
+    with tab1:
+      dataframe = st.session_state[key1]
+      clean_data = st.session_state[key2]
+      st.subheader('''**Select what to explore:**''')
+      data_choice = st.radio('''What to explore''', ("Modelling outcomes", "Input data"), horizontal = True, label_visibility = "collapsed")
+      if data_choice == "Modelling outcomes":
+          st.subheader('''Demographics of the overall modelling outcomes''')
+          model_out(dataframe)
+          st.subheader('''Demographics of the selected protected groups''')
+          filter_for_protected(dataframe)
+      else:
+          st.subheader('''Demographics of the input scores''')
+          model_scores(dataframe)
+          st.subheader('''Demographics of the input labels''')
+          model_vis(dataframe)
+    with tab2:
+      dataframe = st.session_state[key1]
+      clean_data = st.session_state[key2]
+      cmA = st.session_state[key3]
+      cmB = st.session_state[key4]
+      st.subheader('''**Select what to explore:**''')
+      data_choice = st.radio('''Select what to explore:''', ("Modelling outcomes", "Input data"), horizontal = True, label_visibility = "collapsed")
+      if data_choice == "Modelling outcomes":
+          st.subheader('''Labelled dataframe''')
+          dataframe_out(dataframe)
+          st.subheader('''Venn Diagram''')
+          venn_diagram(dataframe)
+          st.subheader('''Model accuracy metrics''')
+          mod_prop(cmA, cmB)
+      else:
+          st.subheader('''Principal Component Analysis''')
+          PCA_general(dataframe, clean_data)
+    
+data_plot('complete_df', 'clean_df', 'cm_A', 'cm_B')

pages/.ipynb_checkpoints/3_💡_Put_the_Idea_into_Practice-checkpoint.py

@@ -0,0 +1,47 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+"""
+Created on Thu May 25 15:42:38 2023
+
+@author: daliagala
+"""
+
+### IMPORT LIBRARIES ###
+import streamlit as st
+
+### PAGE CONFIG ###
+st.set_page_config(page_title='EquiVar', page_icon=':robot_face:', layout='wide')
+
+### IDEA IN PRACTICE PAGE ###
+
+st.title("Different target variable definitions in practice")
+st.markdown('''This dashboard is designed to help you understand how the notion of a “good employee” is translated into machine learning models—by defining target variables—and how target variable definition affects fairness in hiring algorithms (and other aspects of such algorithms too). On this page, we describe how to put the dashboard, and the insights it affords, into practice.''')
+
+st.markdown('''How can this be done? The first step is to make two changes to the dashboard, because you cannot simply take it “off the shelf” and immediately put it into practice. This is because:''')
+
+st.markdown('''- The dashboard is not built using your data or your models.''')
+
+st.markdown('''- How you define a target variable in the dashboard is not how it’s done in practice. (Rather, the way you define it in the dashboard is a straightforward way for you to see the effects of target variable definition.)''')
+st.markdown('''Below we describe how to address these two issues.''')
+
+st.subheader('''Using your data and your models''')
+
+st.markdown('''The dashboard offers a starting point: if you want to build something like the dashboard for your data and your models, you now have a blue-print to work from.''')
+
+st.subheader('''Defining the target variable in practice''')
+
+st.markdown('''In the dashboard, you define the target variable by assigning weights to different cognitive characteristics. These weights determine the “positive label:” people in our dataset who perform best on the tests, given the weights you assign, are those assigned the positive label—that is, those that the model treats as “good.” Then, the model is trained to identify people whose cognitive characteristics match those with the positive label.''')
+
+st.markdown('''As we discussed on the Home page, a growing number of hiring algorithms use cognitive tests to identify promising job applicants. However, the way target variable definition works with these real-world algorithms is different from how it works in the dashboard. For example, consider Pymetrics, a leading developer of hiring software. In some cases, Pymetrics builds bespoke algorithms for a company that is hiring for a given role. Pymetrics will ask the company to identify a group of the client’s current employees in that role that the client considers “good.” Then, these “good” employees play cognitive test games similar to the ones used in our dashboard. It is these employees who are assigned the positive labels. From this point on, Pymetrics’ algorithmic development goes just as it does in our dashboard: a model is trained to identify job applicants whose cognitive characteristics are similar to those with the positive label.''')
+
+st.markdown('''So, for hiring algorithms like Pymetrics’, the target variable is defined not by assigning weights to cognitive attributes, but rather by directly identifying a certain group of current employees as “good.” In the dashboard, you can define different target variables by assigning different weights to the cognitive attributes. If you are in practice building an algorithm like Pymetrics’, you can define different target variables by identifying different groups of current employees as “good.”''')
+
+st.markdown('''How might this work? As we discussed on the Home page of the dashboard, reasonable minds may disagree about what makes for a good employee, and relatedly reasonable minds may disagree about which current employees are good employees. For example, within a company, two different managers—call them Manager A and Manager B—may not be perfectly aligned in who they consider to be good employees for a certain role. The managers may agree in some cases. We might imagine that there are 50 employees whom both Manager A and Manager B deem good. But the two managers might disagree about other employees. Imagine that there are 25 further employees whom Manager A thinks of as good but Manager B does not (this needn’t mean that Manager B thinks that these employees are bad, just that they are not the best). Likewise, there might be 25 further employees whom Manager B thinks of as good but Manager A does not.''')
+
+st.markdown('''In this case, there are two different (overlapping) groups of 75 employees, each corresponding to what Managers A and B think of as good employees. These two different groups of employees—and in turn, two different target variable definitions—could be used to train two different models.''')
+
+st.markdown('''Instead of constructing two groups of “good” employees directly from the judgments of Managers A and B, you could weight their judgments against one another. For example, you could have two groups of employees, X and Y. Both X and Y contain the 50 employees that Managers A and B agree on. But group X contains 20 of Manager A’s preferred employees and 5 of Manager B’s, while group Y contains 20 of Manager B’s preferred employees and 5 of Manager A’s. Here again we have different groups of “good” employees, and so two different target variables.''')
+
+st.markdown('''One could select different groups of good employees in other ways still. An employer might have different metrics to evaluate employee success. Different employees might be better than others according to one metric compared to another. Depending on what importance is assigned to the different metrics—depending on how you weight the different metrics against one another—different groups of employees may emerge as “good.”''')
+
+st.markdown('''Our focus in the dashboard has been on hiring algorithms that are based on cognitive test games. There are other kinds of algorithms used in hiring—for example, algorithms that identify promising job applicants on the basis of their resumés. In designing any such algorithm, the target variable must be defined, and the notion of a “good employee” must be translated into algorithmic terms. And so the insights of this dashboard apply, and can be put into practice, for almost any kind of hiring algorithm you’re working with.''')

pages/1_📊_Define_Target_Variables.py

@@ -0,0 +1,266 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+"""
+Created on Thu May 25 15:16:50 2023
+
+@author: daliagala
+"""
+
+### LIBRARIES ###
+import streamlit as st
+import pandas as pd 
+from sklearn.model_selection import train_test_split
+from utils import assign_labels_by_probabilities, drop_data, train_and_predict
+
+### PAGE CONFIG ###
+st.set_page_config(page_title='FairTargetSim', page_icon=':robot_face:', layout='wide')
+
+hide_st_style = """
+            <style>
+            #GithubIcon {visibility: hidden;}
+            #MainMenu {visibility: hidden;}
+            footer {visibility: hidden;}
+            header {visibility: hidden;}
+            </style>
+            """
+st.markdown(hide_st_style, unsafe_allow_html=True)
+
+### IMPORT DATA FILES ###
+dataframe = pd.read_csv('./data/dataframe.csv')
+dataframe = dataframe.drop(["Unnamed: 0"], axis = 1)
+dataframe = dataframe.rename(columns={"education_level": "education level"})
+
+### DICTIONARIES AND CONSTANTS###
+groups = ["attention", "reasoning", "memory", "behavioural restraint", "information processing speed"]
+
+groups_dict = {
+          'Divided Visual Attention' : 'attention',
+          'Forward Memory Span': 'memory',
+          'Arithmetic Reasoning' : 'reasoning',
+          'Grammatical Reasoning' : 'reasoning',
+          'Go/No go' : 'behavioural restraint',
+          'Reverse_Memory_Span' : 'memory',
+          'Verbal List Learning' : 'memory',
+          'Delayed Verbal List Learning' : 'memory',
+          'Digit Symbol Coding' : 'information processing speed',
+          'Trail Making Part A' : 'information processing speed',
+          'Trail Making Part B' : 'information processing speed'
+        }
+
+education_dict = {
+    1: 'Some high school',
+    2: 'High school diploma / GED',
+    3: 'Some college',
+    4: 'College degree',
+    5: 'Professional degree',
+    6: "Master's degree",
+    7: 'Ph.D.',
+    8: "Associate's degree",
+    99: 'Other'
+  }
+
+df_keys_dict = {
+          'Divided Visual Attention' :'divided_visual_attention',
+          'Forward Memory Span' :'forward_memory_span',
+          'Arithmetic Reasoning' : 'arithmetic_problem_solving',
+          'Grammatical Reasoning'  : 'logical_reasoning',
+          'Go/No go': 'adaptive_behaviour_response_inhibition',
+          'Reverse_Memory_Span' : 'reverse_memory_span',
+          'Verbal List Learning': 'episodic_verbal_learning',
+          'Delayed Verbal List Learning': 'delayed_recall',
+          'Digit Symbol Coding': 'abstract_symbol_processing_speed',
+          'Trail Making Part A' :'numerical_info_processing_speed',
+          'Trail Making Part B': 'numerical_and_lexical_info_processing_speed'
+        }
+
+
+### CREATE THE "TARGET VARIABLE DEFINITION" PAGE ###
+st.title('Define Target Variables')
+
+st.markdown('''On this page, we invite you to imagine that you are hiring for a certain role. Using the sliders below, you will specify two different notions of a “good" employee for that role—two different target variables. Once you’re done, the simulator will build two models, one for each of your target variable definitions (the simulator will also create two datasets on which each model is trained). You can visualize these models and datasets — and their effects on fairness and overall features of the models and data — in the Visualize the Results page.''')
+
+st.markdown('''You specify the notions of different employees by assigning weights of importance to cognitive characteristics that a “good” employee would have for the role. The cognitive test data that you’ll be working with comes from real-world people, mirroring an increasing number of hiring algorithms that are based on cognitive tests ([Wilson, et al. 2021](https://dl.acm.org/doi/10.1145/3442188.3445928)).''')
+
+st.markdown('''We have pre-set the weights below to reflect two different conceptions of a “good” employee: one conception that emphasizes attentiveness and numerical skills (A) and another conception that emphasizes interpersonal skills and memory (B). If you want to, you can change the slider values as you see fit.''')
+
+st.markdown('''After you’ve set the slider values, click :red[“Assign labels and train your models”]. The simulator will then label certain individuals as "good employees"—in other words it will assign class labels "1" for "good" and "0" for "not good", based on your selections. Then, the simulator will build the two models.''')
+
+st.markdown('''To learn more about target variable definition and hiring models based in cognitive tests, :green["See explanation"] below.''')
+
+with st.expander("See explanation"):
+    
+    st.markdown('''The models that the simulator builds are of a kind increasingly used in hiring software: companies will have applicants play games that test for different kinds of cognitive ability (like reasoning or memory). Then hiring software will be built to predict which applicants will be "good" based on which cognitive characteristics they have. What cognitive characteristics make for a "good" employee? This will depend on what role is being hired. And it will also depend on how one defines a “good" employee.''')
+    
+    st.markdown('''In the real world, "good" employee is defined for these kinds of hiring models in the following way. Managers select a group of current employees that they consider to be "good"; this group of employees plays the cognitive test games. A model is then trained to identify applicants who share cognitive characteristics with the current employees that are considered "good". The target variable of "good" employee is thus defined in terms of comparison to certain people who are deemed "good". ''')
+
+    st.markdown('''One will get different target variables if one deems different current employees as the "good" ones. And, as we discussed in the Home page (and as we explain more in the Putting the Idea into Practice page), there will likely be disagreement between managers about which employees are "good". For instance, a manager who values attentiveness and numerical skills will deem different employees “good” than a manager who values interpersonal skills and memory. Even when different managers roughly share their sensibilities in what characteristics make for a "good" employee, there may still be different, equally good ways to “weight” the importance of the various characteristics.''')
+    
+    st.markdown('''In the real world, the cognitive characteristics shared by those considered "good" employees are implicit. Companies do not first identify the cognitive characteristics that make for a "good" employee; rather, they identify employees who they consider "good", and then the hiring model works backwards to identify what characteristics these employees share.''')
+    
+    st.markdown('''In our simulator, the cognitive characteristics shared by “good” employees are explicit. You assign different weights—using the sliders—to the cognitive characteristics you think are more or less important in a "good" employee (for the role you’re considering). To illustrate how different target variables have different effects on fairness and overall model attributes, you’ll define “good” employee in two ways. (We’ve made the cognitive characteristics explicit both so you can see the point of different target variable definitions more clearly, and because of limitations of the data that we’re working with.)''')
+
+    st.markdown('''The cognitive characteristics that the simulator works with are from one of the datasets of the [NeuroCognitive Performance Test](https://www.nature.com/articles/s41597-022-01872-8). This dataset has eleven different tests which we have grouped into five categories: ''')
+    
+    st.markdown(
+          """
+          - **Memory**: Forward Memory Span, Reverse Memory Span, Verbal List Learning, Delayed Verbal List Learning
+          - **Information Processing Speed**: Digit Symbol Coding, Trail Making Part A, Trail Making Part B
+          - **Reasoning**: Arithmetic Reasoning, Grammatical Reasoning
+          - **Attention**: Divided Visual Attention
+          - **Behavioral Restraint**: Go/No go
+          """)
+    
+    st.markdown('''After you’ve set the weights to these five characteristics using the sliders, you can see which weights are assigned to each test (e.g. Forward Memory Span or Digit Symbol Coding) by ticking the checkbox beneath the sliders.''')
+
+col1, col2 = st.columns(2)
+
+#Initialise slider values
+list_values_A = (9, 10, 2, 1, 5)
+list_values_B = (1, 2, 10, 9, 3)
+
+selectionsA = {}
+selectionsB = {}
+results_dict_A = groups_dict
+results_dict_B = groups_dict
+
+with col1:
+    st.subheader("Define target variable for model A ")
+        
+    if "slider_values_A" not in st.session_state:
+        for count, value in enumerate(groups):
+            selectionsA[value] = list_values_A[count]
+        st.session_state["slider_values_A"] = selectionsA
+    else:
+        selectionsA = st.session_state["slider_values_A"]
+            
+    for i in groups:
+        nameA = f"{i} importance, model A"
+        value = selectionsA[i]
+        slider = st.slider(nameA, min_value=0, max_value=10, value = value)
+        selectionsA[i] = slider
+        
+    results_dict_A = {k: selectionsA.get(v, v) for k, v in results_dict_A.items()}
+    total = sum(results_dict_A.values())
+    for (key, u) in results_dict_A.items():
+      if total != 0:
+        w = (u/total)
+        results_dict_A[key] = w
+
+    if st.checkbox("Show target variable A weights per subtest", key="A"):
+      for (key, u) in results_dict_A.items():
+        txt = key.replace("_", " ")
+        st.markdown("- " + txt + " : " + f":green[{str(round((u*100), 2))}]")
+        
+    
+    st.session_state["slider_values_A"] = selectionsA
+
+
+with col2:
+    st.subheader("Define target variable for model B ")
+        
+    if "slider_values_B" not in st.session_state:
+        for count, value in enumerate(groups):
+            selectionsB[value] = list_values_B[count]
+        st.session_state["slider_values_B"] = selectionsB
+    else:
+        selectionsB = st.session_state["slider_values_B"]
+            
+    for i in groups:
+      nameB = f"{i} importance, model B"
+      value = selectionsB[i]
+      slider = st.slider(nameB, min_value=0, max_value=10, value = value)
+      selectionsB[i] = slider
+      
+    results_dict_B = {k: selectionsB.get(v, v) for k, v in results_dict_B.items()}
+    total = sum(results_dict_B.values())
+    for (key, u) in results_dict_B.items():
+      if total != 0:
+        w = ((u/total))
+        results_dict_B[key] = w
+
+    if st.checkbox("Show target variable B weights per subtest", key = "B"):
+      for (key, u) in results_dict_B.items():
+        txt = key.replace("_", " ")
+        st.markdown("- " + txt + " : " + f":green[{str(round((u*100), 2))}]")
+      
+    st.session_state["slider_values_B"] = selectionsB
+          
+if st.button("Assign labels and train your models", type = "primary", use_container_width = True):
+  if 'complete_df' in st.session_state:
+      del st.session_state['complete_df']
+  if 'clean_df' in st.session_state:
+      del st.session_state['clean_df']
+  if 'cm_A' in st.session_state:
+      del st.session_state['cm_A']
+  if 'cm_B' in st.session_state:
+      del st.session_state['cm_B']
+  scoreA = pd.DataFrame()
+  scoreB = pd.DataFrame()
+  test1 = all(value == 0 for value in results_dict_A.values())
+  test2 = all(value == 0 for value in results_dict_B.values())
+  if test1 == True or test2 == True:
+    st.error('Cannot train the models if you do not define the target variables. Make your selections for both models first!', icon="🚨")
+  else:
+    for (key, u) in results_dict_A.items():
+      scoreA[df_keys_dict[key]] = u * dataframe[df_keys_dict[key]]
+      scoresA = scoreA.sum(axis=1)
+      dataframe['model_A_scores'] = scoresA
+    for (key, u) in results_dict_B.items():
+      scoreB[df_keys_dict[key]] = u * dataframe[df_keys_dict[key]]
+      scoresB = scoreB.sum(axis=1)
+      dataframe['model_B_scores'] = scoresB
+    
+    new_annotated = assign_labels_by_probabilities(dataframe, "model_A_scores", "Model_A_label", "Model_A_probabilities", quantile=0.85, num_samples=100)
+    new_annotated = assign_labels_by_probabilities(new_annotated, "model_B_scores", "Model_B_label", "Model_B_probabilities", quantile=0.85, num_samples=100)
+    new_annotated = new_annotated.reset_index()
+
+
+    clean_data = drop_data(new_annotated)
+    # specify the columns of interest
+    selected_cols = ['Model_A_label', 'Model_B_label']
+    
+    # count the number of rows where all three selected columns have a value of 1
+    num_rows_with_all_flags_1 = len(new_annotated[new_annotated[selected_cols].sum(axis=1) == len(selected_cols)])
+    
+    # print the result
+    st.write(f"Shared candidates between your target variables: :green[{num_rows_with_all_flags_1}] (among 100 total candidates per each target variable).")
+    with st.spinner('Please wait... The models will be trained now.'):
+
+      X_data, Y_data_A, Y_data_B = clean_data.iloc[:, :-2], clean_data.iloc[:, [-2]], clean_data.iloc[:, [-1]]
+      X_data = X_data.drop(["index"], axis = 1)
+      Y_data_B = Y_data_B.reset_index()
+      X_train, X_test, y_train_A, y_test_A = train_test_split(X_data, Y_data_A, test_size=0.2)
+      y_train_A = y_train_A.reset_index()
+      y_test_A = y_test_A.reset_index()
+      y_train_B = pd.merge(y_train_A,Y_data_B[['index', 'Model_B_label']],on='index', how='left')
+      y_test_B = pd.merge(y_test_A,Y_data_B[['index', 'Model_B_label']],on='index', how='left')
+      y_train_B = y_train_B.drop(labels='Model_A_label', axis = 1)
+      y_test_B = y_test_B.drop(labels='Model_A_label', axis = 1)
+      y_train_A = y_train_A.set_index("index")
+      y_train_B = y_train_B.set_index("index")
+      y_test_A = y_test_A.set_index("index")
+      y_test_B = y_test_B.set_index("index")
+
+      accuracy_A, precision_A, recall_A, X_full_A, cm_A, baseline_accuracy_A = train_and_predict("A", X_train, X_test, y_train_A, y_test_A)
+      accuracy_B, precision_B, recall_B, X_full_B, cm_B, baseline_accuracy_B = train_and_predict("B", X_train, X_test, y_train_B, y_test_B)
+      full = pd.merge(X_full_A,X_full_B[['index','Predicted_B', 'Prob_0_B', "Prob_1_B"]],on='index', how='left')
+      complete = pd.merge(full,new_annotated[['index', 'age', 'gender', 'education level', 'country', 'Model_A_label', 'Model_B_label', 'model_A_scores', 'model_B_scores']],on='index', how='left')
+      complete=complete.replace({"education level": education_dict})
+      complete = complete.rename(columns={"index": "Candidate ID"})
+      
+      if 'complete_df' not in st.session_state:
+        st.session_state['complete_df'] = complete
+      if 'clean_df' not in st.session_state:
+        st.session_state['clean_df'] = clean_data
+      if 'cm_A' not in st.session_state:
+        st.session_state['cm_A'] = cm_A
+      if 'cm_B' not in st.session_state:
+        st.session_state['cm_B'] = cm_B
+
+    row1_space1, row1_1, row1_space2, row1_2, row1_space3 = st.columns((0.1, 3, 0.1, 3, 0.1))
+    with row1_1:
+      st.write(f"Model A accuracy: :green[{baseline_accuracy_A}].")
+    with row1_2:
+      st.write(f"Model B accuracy: :green[{baseline_accuracy_B}].")
+
+    st.success('''Success! You have defined the target variables and trained your models. Head to "Visualise the Results" in the sidebar.''')

pages/2_📈_Visualize_the_Results.py

@@ -0,0 +1,593 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+"""
+Created on Thu May 25 15:30:56 2023
+
+@author: daliagala
+"""
+
+### LIBRARIES ###
+import streamlit as st
+import numpy as np
+import pandas as pd 
+import matplotlib.pyplot as plt
+import plotly.express as px
+import plotly.graph_objects as go
+from matplotlib_venn import venn2
+from sklearn.metrics import confusion_matrix
+from utils import display_proportional, plot_data, run_PCA, create_confusion_matrix_heatmap, plot_conf_rates
+
+### PAGE CONFIG ###
+st.set_page_config(page_title='FairTargetSim', page_icon=':robot_face:', layout='wide')
+
+hide_st_style = """
+            <style>
+            #GithubIcon {visibility: hidden;}
+            #MainMenu {visibility: hidden;}
+            footer {visibility: hidden;}
+            header {visibility: hidden;}
+            </style>
+            """
+st.markdown(hide_st_style, unsafe_allow_html=True)
+
+### DICTIONARIES AND CONSTANTS###
+
+colours_education = {
+          'Some high school' : 'indigo',
+          'High school diploma / GED' : '#7ae99e',
+          'Some college' : '#0a68c9',
+          'College degree': '#80c4fa',
+          'Professional degree': '#f48508',
+          "Master's degree" : '#2fa493',
+          'Ph.D.' : '#f2a3a1',
+          "Associate's degree" : '#fbcc66',
+          'Other' : '#fa322f'
+        }
+
+colours_country = {
+          'AU' : '#7ae99e',
+          'US': '#80c4fa',
+          'NZ': '#2fa493',
+          'CA' : '#fbcc66'
+        }
+
+colours_gender = {
+          'f' : '#83C9FF',
+          'm': '#0067C9'
+        }
+
+characteristic_dict = {
+          'gender' : colours_gender,
+          'education level' : colours_education,
+          'country' : colours_country,
+          'age' : 'indigo'
+        }
+
+pred_dict = {
+          'Model A' : 'Predicted_A',
+          'Model B' : 'Predicted_B'
+        }
+
+prob_dict = {
+          'Model A' : 'Prob_1_A',
+          'Model B' : 'Prob_1_B'
+        }
+
+model_dict = {
+          'Model A' : 'Model_A_label',
+          'Model B' : 'Model_B_label'
+        }
+
+df_keys_dict = {
+          'Divided Visual Attention' :'divided_visual_attention',
+          'Forward Memory Span' :'forward_memory_span',
+          'Arithmetic Reasoning' : 'arithmetic_problem_solving',
+          'Grammatical Reasoning'  : 'logical_reasoning',
+          'Go/No go': 'adaptive_behaviour_response_inhibition',
+          'Reverse_Memory_Span' : 'reverse_memory_span',
+          'Verbal List Learning': 'episodic_verbal_learning',
+          'Delayed Verbal List Learning': 'delayed_recall',
+          'Digit Symbol Coding': 'abstract_symbol_processing_speed',
+          'Trail Making Part A' :'numerical_info_processing_speed',
+          'Trail Making Part B': 'numerical_and_lexical_info_processing_speed'
+        }
+
+### DEFINE ALL SUB-PAGES AS FUNCTIONS TO CALL ###
+
+def mod_prop(cmA, cmB):
+    st.markdown('''This section contains model confusion matrices, which give us a lot of information about how good the models which we produced really are. We obtain the confusion matrices by plotting "Actual" labels on one axis, and "Predicted" labels on the other. For both models, in each square of each confusion matrix, we can see which group of candidates it represents, the number of candidates in this group, and what percentage of all candidates assessed they represent.''')
+    st.markdown('''
+    - **True Negative (TN)**: candidates predicted to have label "0" whose actual label was "0".
+    - **False Positive (FP)**: candidates predicted to have label "1" whose actual label was "0".
+    - **False Negative (FN)** candidates predicted to have label "0" whose actual label was "1".
+    - **True Positive (TP)**: candidates predicted to have label "1" whose actual label was "1".
+    ''')
+    
+    row1_space1, row1_1, row1_space2, row1_2, row1_space3 = st.columns((0.1, 3, 0.1, 3, 0.1))
+    with row1_1:
+      create_confusion_matrix_heatmap(cmB, "Model A")
+    
+    with row1_2:
+      model = "B"
+      create_confusion_matrix_heatmap(cmB, "Model B")
+    
+    st.subheader("Model Accuracy Rates")
+    st.markdown('''We can also represent each model in terms of its accuracy rates, calculated using the numbers of candidates in each group shown in the confusion matrix:''')
+    st.markdown('''
+    - True Positive Rate (TPR), also called recall, sensitivity or hit rate, is the probability of a positive test result when the result is indeed positive.
+    - True Negative Rate (TNR), or specificity/selectivity, is the probability of a negative test result when the test is indeed negative.
+    - Positive Predictive Value (PPV) or Precision is the ratio of truly positive results to all positive results.
+    - Negative Predictive Value (NPR) is the ratio of truly negative results to all negative results.
+    - False Positive Rate (FPR) or fall-out is the probability of assigning a falsely positive result when the test is negative.
+    - False Negative Rate (FNR) or miss rate is the probability that a test with a positive result will falsely be assigned a negative result.
+    - False Discovery Rate (FDR) it the ratio of false positive results to total number of positive results.
+    ''')
+    measures_A = plot_conf_rates(cmA)
+    measures_B = plot_conf_rates(cmB)
+    fig = go.Figure()
+    fig.add_trace(go.Bar(
+          x=measures_A["Measure"],
+          y=measures_A["Score"],  
+          name='Model A rates',
+          marker_color='rgb(55, 83, 109)'
+      ))
+    fig.add_trace(go.Bar(
+          x=measures_B["Measure"],
+          y=measures_B["Score"],
+          name='Model B rates',
+          marker_color='rgb(26, 118, 255)'
+      ))
+    fig.update_layout(
+          title='Model measures comparison',
+          xaxis_tickfont_size=14,
+          xaxis=dict(
+              title='Measure'),
+          yaxis=dict(
+              title='Score',
+              titlefont_size=16,
+              tickfont_size=14,
+          ),
+          legend=dict(
+              bgcolor='rgba(255, 255, 255, 0)',
+              bordercolor='rgba(255, 255, 255, 0)'
+          ),
+    barmode='group',
+    bargap=0.15, # gap between bars of adjacent location coordinates.
+    bargroupgap=0.1 # gap between bars of the same location coordinate.
+      )
+    st.plotly_chart(fig, use_container_width = True)
+    if st.checkbox("Show tables"):
+        row1_space1, row1_1, row1_space2, row1_2, row1_space3 = st.columns((0.1, 3, 0.1, 3, 0.1))
+        # CSS to inject contained in a string
+        hide_table_row_index = """
+        <style>
+        thead tr th:first-child {display:none}
+        tbody th {display:none}
+        </style> """
+
+        # Inject CSS with Markdown
+        st.markdown(hide_table_row_index, unsafe_allow_html=True)
+        with row1_1:
+            st.markdown("Accuracy rates for model A")
+            st.table(measures_A)
+        with row1_2:
+            st.markdown("Accuracy rates for model B")
+            st.table(measures_B)
+
+def model_scores(dataframe):
+  st.markdown('''This section displays the distribution of scores assigned to each hypothetical employee, according to the values you set on the sliders, compared with the distribution of protected characteristics. These scores are then used to assign labels to the hypothetical employees, creating two distinct datasets to train the models. In essence, these scores explicitly and numerically mimic the viewpoints of hypothetical hiring managers A and B when deciding who to label as "top employees."''')
+  st.markdown('''Just as in the original NCPT dataset analysis, the scores obtained by participants in the cognitive games decline with age. This observation provides insight into the potential issue of ageism inherent in the use of gamified hiring processes.''')
+  # Create a selectbox to choose a protected characteristic to explore
+  plot_radio = st.selectbox('Characteristic to explore', characteristic_dict.keys())
+  row2_space1, row2_1, row2_space2 = st.columns((0.1, 5, 0.1))
+
+  with row2_1:
+    data = dataframe[["model_A_scores", "model_B_scores", plot_radio]]
+
+    if plot_radio == "age":
+        bins= [18,20,30,40,50,60,70,80,90]
+        labels = ['18-20','21-30','31-40','41-50','51-60','61-70','71-80','81-90']
+        data['age_bins'] = pd.cut(data['age'], bins=bins, labels=labels, right=False)
+        plot_radio = 'age_bins'
+    colours = ['rgba(93, 164, 214, 0.5)', 'rgba(255, 144, 14, 0.5)']
+    c1, c2, c3, c4, c5 = st.columns((0.1, 3, 0.1, 3, 0.1))
+    with c2:
+        fig = px.box(data, x = plot_radio, y="model_A_scores", labels={"model_A_scores":"Dataset A Input Scores", 'age_bins':"Age"}, title="Input scores in dataset A")
+        fig.update_layout(showlegend=False)
+        fig.update_traces(marker_color='rgba(93, 164, 214, 0.5)')
+        st.plotly_chart(fig, use_container_width=True)
+    with c4:
+        fig = px.box(data, x = plot_radio, y="model_B_scores", labels={"model_B_scores":"Dataset B Input Scores", 'age_bins':"Age"}, title="Input scores in dataset B")
+        fig.update_traces(marker_color = 'rgba(255, 144, 14, 0.5)')
+        fig.update_layout(showlegend=False)
+        st.plotly_chart(fig, use_container_width=True)
+
+def PCA_general(full_df, dataframe_PCA):
+  st.markdown('''On this page, you can see the distribution of the dataset labels which were assigned based on the scores calculated from the slider values you selected previously. Principal Components Analysis, or PCA, is a technique often used to analyse and subsequently visualize datasets where there are many features per single example. This is the case with the NCPT dataset used in our simulator. Specifically, the battery which we used has 11 features per single example, the example being the player of cognitive games, and, in our metaphor, a hypothetical employee or job candidate. It is impossible to plot 11 dimensions, and PCA allows for the visualisation of multidimensional data, while also preserving as much information as possible.''')
+  choice = st.radio("What would you like to explore?", ("PCAs", "Components loading"), horizontal = True)
+  pcaA, dfA, labelsA, coeffA, componentsA = run_PCA(dataframe_PCA, 'Model_B_label', 'Model_A_label', 2)
+  pcaB, dfB, labelsB, coeffB, componentsB = run_PCA(dataframe_PCA, 'Model_A_label', 'Model_B_label', 2)
+  loadings = pcaB.components_.T * np.sqrt(pcaB.explained_variance_)
+  total_var = pcaA.explained_variance_ratio_.sum() * 100
+  dfA = dfA.rename(columns={'target': 'Dataset A'}).reset_index()
+  dfB = dfB.rename(columns={'target': 'Dataset B'}).reset_index()
+  df_all = pd.merge(dfA, dfB[['index', 'Dataset B']], on='index', how='left')
+
+  conditions = [
+      (df_all['Dataset A'] == 1) & (df_all['Dataset B'] == 0),
+      (df_all['Dataset B'] == 1) & (df_all['Dataset A'] == 0),
+      (df_all['Dataset A'] == 1) & (df_all['Dataset B'] == 1),
+      (df_all['Dataset A'] == 0) & (df_all['Dataset B'] == 0)]
+      
+  values = ['Selected A', 'Selected B', 'Selected both', 'Not selected']
+  df_all['All'] = np.select(conditions, values)
+
+  df_all = df_all.drop(["index"], axis = 1)
+  df_all.All=pd.Categorical(df_all.All,categories=['Not selected', 'Selected A', 'Selected B', 'Selected both'])
+  df_all=df_all.sort_values('All')
+
+  selections_dict = {0: 'Not selected', 1: 'Selected'}
+  df_all = df_all.replace({"Dataset A": selections_dict, "Dataset B": selections_dict})
+
+  color_dict_sel = {'Not selected': '#3366CC', 'Selected': 'grey'}
+
+  if "pca_df" not in st.session_state:
+    st.session_state.pca_df = df_all
+
+  if choice == "PCAs":
+    c1, c2 = st.columns(2)
+    with c1:
+      fig = px.scatter(st.session_state.pca_df, 
+                      x=st.session_state.pca_df['principal component 1'].astype(str),
+                      y=st.session_state.pca_df['principal component 2'].astype(str),
+                      title='Dataset A PCA',
+                      labels={"x": 'PC 1', "y": 'PC 2'},
+                      color=st.session_state.pca_df['Dataset A'],
+                      color_discrete_map=color_dict_sel)
+      fig.update_traces(marker_size = 8)
+      st.plotly_chart(fig, use_container_width=True)
+    with c2:
+      fig = px.scatter(st.session_state.pca_df, 
+                      x=st.session_state.pca_df['principal component 1'].astype(str),
+                      y=st.session_state.pca_df['principal component 2'].astype(str),
+                      title='Dataset B PCA',
+                      labels={"x": 'PC 1', "y": 'PC 2'},
+                      color=st.session_state.pca_df['Dataset B'],
+                      color_discrete_map=color_dict_sel)
+      fig.update_traces(marker_size = 8)
+      st.plotly_chart(fig, use_container_width=True)
+      
+    st.markdown(f'''These plots show the reduction of 11 dimensions (11 subtest results) to 2 dimensions. Total Variance for the data is {total_var:.2f}%. Both of the datasets have the same features, therefore they both have the same total variance. Total variance value indicates what percentage of information has been preserved when the dimensionality was reduced. Note that for both datasets, A and B, different points are labelled "1" or "0". This shows that the two datasets represent the two different target variable definitions which were created by you previously. The plots are interactive - zoom in to explore in detail.''')
+      
+    pcaA, dfA, labelsA, coeffA, componentsA = run_PCA(dataframe_PCA, 'Model_B_label', 'Model_A_label', 2)
+    pcaB, dfB, labelsB, coeffB, componentsB = run_PCA(dataframe_PCA, 'Model_A_label', 'Model_B_label', 2)
+    loadings = pcaB.components_.T * np.sqrt(pcaB.explained_variance_)
+    total_var = pcaA.explained_variance_ratio_.sum() * 100
+    dfA = dfA.rename(columns={'target': 'Dataset A'}).reset_index()
+    dfB = dfB.rename(columns={'target': 'Dataset B'}).reset_index()
+    df_all = pd.merge(dfA, dfB[['index', 'Dataset B']], on='index', how='left')
+
+    conditions = [
+        (df_all['Dataset A'] == 1) & (df_all['Dataset B'] == 0),
+        (df_all['Dataset B'] == 1) & (df_all['Dataset A'] == 0),
+        (df_all['Dataset A'] == 1) & (df_all['Dataset B'] == 1),
+        (df_all['Dataset A'] == 0) & (df_all['Dataset B'] == 0)]
+        
+    values = ['Selected A', 'Selected B', 'Selected both', 'Not selected']
+    df_all['All'] = np.select(conditions, values)
+
+    df_all = df_all.drop(["index"], axis = 1)
+    df_all.All=pd.Categorical(df_all.All,categories=['Not selected', 'Selected A', 'Selected B', 'Selected both'])
+    df_all=df_all.sort_values('All')
+
+    selections_dict = {0: 'Not selected', 1: 'Selected'}
+    df_all = df_all.replace({"Dataset A": selections_dict, "Dataset B": selections_dict})
+
+    if "pca_df" not in st.session_state:
+      st.session_state.pca_df = df_all
+
+    fig = px.scatter(st.session_state.pca_df, 
+                      x=st.session_state.pca_df['principal component 1'],
+                      y=st.session_state.pca_df['principal component 2'],
+                      title="PCA with labelled groups",
+                      color=st.session_state.pca_df["All"], 
+                      width = 800, height = 800,
+                      color_discrete_sequence=px.colors.qualitative.Safe,
+                      opacity = 0.95)
+      
+    fig.update_yaxes(
+        scaleanchor="x",
+        scaleratio=1,
+      )
+    fig.update_traces(marker_size = 10)
+    st.plotly_chart(fig)
+
+  if choice == "Components loading":
+      c1,c2 = st.columns(2)
+      loadings_df = pd.DataFrame(loadings, columns = ["PC1", "PC2"])
+      labels_A_proper = { v:k for k,v in df_keys_dict.items()}
+      loadings_df["Features"] = labels_A_proper.values()
+      with c1:
+          fig = px.bar(loadings_df, x="PC1", y="Features", orientation = 'h')
+          st.plotly_chart(fig, use_container_width = True)
+      with c2:
+          fig = px.bar(loadings_df, x="PC2", y="Features", orientation = 'h')
+          st.plotly_chart(fig, use_container_width = True)
+
+    #   fig = go.Figure()
+    #   fig.add_trace(go.Bar(
+    #       x=loadings_df["PC1"],
+    #       y=loadings_df["Features"],  
+    #       name='Principal Component 1',
+    #       marker_color='rgb(55, 83, 109)',
+    #       orientation='h'
+    #   ))
+    #   fig.add_trace(go.Bar(
+    #       x=loadings_df["PC2"],
+    #       y=loadings_df["Features"],
+    #       name='Principal Component 2',
+    #       marker_color='rgb(26, 118, 255)',
+    #       orientation='h'
+    #   ))
+    #   fig.update_layout(
+    #       title='Component loadings',
+    #       xaxis_tickfont_size=14,
+    #       xaxis=dict(
+    #           title='Loading value'),
+    #       yaxis=dict(
+    #           title='Feature',
+    #           titlefont_size=16,
+    #           tickfont_size=14,
+    #       ),
+    #       legend=dict(
+    #           bgcolor='rgba(255, 255, 255, 0)',
+    #           bordercolor='rgba(255, 255, 255, 0)'
+    #       ),
+    # barmode='group',
+    # bargap=0.15, # gap between bars of adjacent location coordinates.
+    # bargroupgap=0.1 # gap between bars of the same location coordinate.
+    #   )
+    #   st.plotly_chart(fig, use_container_width = True)
+      
+      st.markdown('''On this plot, PCA component loadings can be explored. These facilitate the understanding of how much each variable (which there are 11 of) contributes to a particular principal component. Here, the 11 variables were reduced to 2 components, which are labelled PC1 and PC2. The magnitude of the loading (here displayed as the size of the bar in the bar chart) indicates how strong the relationship between the variable and the component is. Therefore, the higher the bar, the stronger the relationship between that component and that variable. The loading's sign can be positive or negative. This indicates whether the principal component and that variable are positively or negatively correlated. We can see that multiple variables are positively correlated with PC2. Two variables, episodic verbal learning and delayed recall are negatively correlated with both of the components.''')
+
+
+def model_out(full_df):
+  st.markdown('''This section highlights the discrepancies between your two models when presented with the same pool of new, previously unseen candidates to label. Specifically, you'll be investigating the candidates assigned a "1" label by both models. These individuals would be those considered for a job interview or chosen for the role, according to your defined target variable.''')
+  # Create a selectbox to choose a protected characteristic to explore
+  selectbox = st.selectbox('Characteristic to explore', characteristic_dict.keys())
+  representation = st.selectbox("Representation", ("absolute", "proportional"))
+  row1_space1, row1_1, row1_space2, row1_2, row1_space3 = st.columns((0.1, 3, 0.1, 3, 0.1))
+  with row1_1:
+    st.subheader("Candidates selected by model A")
+
+    if representation == "absolute":
+      # Select predicted data ==1
+      data = full_df.loc[full_df['Predicted_A'] == 1]
+
+      # Use function plot_data to plot selected data
+      plot_data(data, selectbox, characteristic_dict[selectbox])
+    else:
+      display_proportional(full_df, selectbox, 'Predicted_A')
+      
+  with row1_2:
+    st.subheader("Candidates selected by model B")
+    
+    if representation == "absolute":
+      # Select predicted data ==1
+      data = full_df.loc[full_df['Predicted_B'] == 1]
+
+      # Use function plot_data to plot selected data
+      plot_data(data, selectbox, characteristic_dict[selectbox])
+
+    else:
+      display_proportional(full_df, selectbox,'Predicted_B')
+      
+  
+  st.markdown('''In this section, you're comparing the model's selections concerning four protected characteristics: age, gender, education level, and country. You can visualize these differences in two ways: "Absolute" or "Proportional".''')
+  st.markdown('''"Absolute" representation gives you the raw numbers or percentages of each characteristic chosen. For instance, if the model labeled 5 female candidates and 5 male candidates as "1", the "Absolute" outcome will display as 50% for both genders."Proportional" representation, on the other hand, shows the percentage of a group selected by the model relative to the total number of that group in the input data. For example, if the model evaluated 100 male candidates and selected 5, you will see a 5% representation. If it evaluated 200 female candidates and selected 5, it will show a 2.5% representation.''')
+  st.markdown('''If you encounter empty categories in the "Proportional" view, this indicates that while candidates from these categories were evaluated, none were labeled as "1". Hence, their proportional representation amounts to 0%.''')
+
+def dataframe_out(full_df):
+  selectbox_M = st.selectbox('Choose which model output to rank by', pred_dict.keys())
+
+  # Select data
+  data = full_df.loc[full_df[pred_dict[selectbox_M]] == 1]
+  data = data.sort_values(by = prob_dict[selectbox_M], ascending = False)
+  data = data[['Candidate ID','Prob_1_A', 'Prob_1_B', 'Predicted_A', 'Predicted_B']]
+  data = data.rename(columns={"Prob_1_A": "Ranking, model A", "Prob_1_B": "Ranking, model B", "Predicted_A": "Predicted label A", "Predicted_B": "Predicted label B"})
+  data.index = np.arange(1, len(data) + 1)
+
+  st.table(data.style.background_gradient(subset = ["Ranking, model A", "Ranking, model B"], axis=0, vmin=0.40).highlight_max(color = '#FFCD9B', subset = ["Predicted label A", "Predicted label B"], axis=0))
+  
+  st.markdown("""In this section, you can review the data for all candidates labeled "1" by the selected model, found at the top of the page. Simultaneously, you can observe the labels assigned to these same candidates by the other model. It's likely that there will be instances where candidates chosen by one model weren't selected by the other. Candidates labeled "1" are highlighted in orange in the "Predicted label A" and "Predicted label B" columns.""")
+  st.markdown('''In addition to this, you can see the probability with which each candidate was labeled "1". The intensity of the blue color indicates the candidate's ranking position - a darker blue represents a higher ranking (with 1 being the maximum and 0 the minimum). You may notice that some candidates highly ranked by one model may be ranked significantly lower by the other model.''')
+  
+def venn_diagram(full_df):
+  row2_space1, row2_1, row2_space2, row2_2, row2_space3 = st.columns((0.1, 1, 0.1, 1, 0.1))
+  with row2_1:
+    fig, ax = plt.subplots()
+
+    list_A = full_df.loc[full_df['Predicted_A'] == 1, 'Candidate ID'].astype(int)
+    list_B = full_df.loc[full_df['Predicted_B'] == 1, 'Candidate ID'].astype(int)
+    set1 = set(list_A)
+    set2 = set(list_B)
+
+    venn2([set1, set2], ('Model A', 'Model B'), ax=ax)
+    st.pyplot(fig)
+
+  with row2_2:
+    st.markdown('''This Venn Diagram visualizes the number of candidates chosen by both models. It's likely that some candidates will be selected by both models, while others may be chosen by only one model. If we consider Model A as the decision of one hiring manager and Model B as another's, it's easy to see how the selection outcome varies depending on the decision-maker. Some candidates may get the opportunity to be hired, while others might not. This serves as an illustration of the inherent arbitrariness in defining the target variable when dealing with highly subjective outcomes.''')
+    st.markdown('''For instance, it's straightforward to define a target variable in a classification problem like distinguishing dragonflies from butterflies, where there's little room for ambiguity. However, defining what makes a 'good' employee is far more challenging due to its subjective nature.''')
+    
+
+
+def model_vis(full_df):
+  st.markdown('''In this section, you can visualize the demographics of the different subgroups of the data. Firstly, you can see the demographic characteristics of the candidates who have positive labels ("1") and negative labels ("0") which were assigned based on the scores calculated from the slider values you selected previously. Then, you can visualize the demographic distributions of the data which was used for training and evaluation of the models.''')
+  choice = st.radio("**Select desired data:**", ("Positive and negative labels", "Training and evaluation data"), horizontal=True)
+  if choice == "Positive and negative labels":
+    # Create a selectbox to choose a protected characteristic to explore
+    selectbox_Lab = st.selectbox('Label to visualize', ('positive labels', 'negative labels'))
+
+    # Create a selectbox to choose a protected characteristic to explore
+    selectbox_Char = st.selectbox('Protected characteristic', characteristic_dict.keys())
+
+    row2_space1, row2_1, row2_space2, row2_2, row2_space3 = st.columns((0.1, 3, 0.1, 3, 0.1))
+
+    with row2_1:
+      st.subheader("Dataset A")
+
+      # Select test data
+      if selectbox_Lab == 'positive labels':
+        data = full_df.loc[full_df['Model_A_label'] == 1]
+      else:
+        data = full_df.loc[full_df['Model_A_label'] == 0]
+      
+      # Use function plot_data to plot selected data
+      plot_data(data, selectbox_Char, characteristic_dict[selectbox_Char])
+
+
+    with row2_2:
+      st.subheader("Dataset B")
+
+      # Select test data
+      if selectbox_Lab == 'positive labels':
+        data = full_df.loc[full_df['Model_B_label'] == 1]
+      else:
+        data = full_df.loc[full_df['Model_B_label'] == 0]
+
+      # Use function plot_data to plot selected data
+      plot_data(data, selectbox_Char, characteristic_dict[selectbox_Char])
+    st.markdown('''You are visualising the demographic composition of those hypothetical employees who were assigned labels "1" or "0" based on your definitions of the target variables. You might see differences in proportions of genders between the two models for the positive labels, as well as a major difference in the age between the positive and negative labels. Visualising the labels in this manner before training the model can help understand and mitigate differences in demographic representation in the modelling outcomes. Likely, if all candidates labelled "1" were in younger age groups, the candidates selected by the model at the deployment stage will also be in younger age groups. Moreover, target variable definition affects the proportional representation. Having defined two target variables, one can choose the dataset and the model which offers more proportional representation.''')
+
+
+  if choice == "Training and evaluation data":
+    # Create a selectbox to choose a protected characteristic to explore
+    selectbox = st.selectbox('Characteristic to explore', characteristic_dict.keys())
+    row1_space1, row1_1, row1_space2, row1_2, row1_space3 = st.columns((0.1, 1, 0.1, 1, 0.1))
+    # Plot training data
+    with row1_1:
+      st.subheader("Training data")
+
+      # Select train data
+      train = full_df.loc[full_df["Predicted_A"] == "train"]
+
+      # Use function plot_data to plot selected data
+      plot_data(train, selectbox, characteristic_dict[selectbox])
+
+    # Plot test data
+
+    with row1_2:
+      st.subheader("Test data")
+
+      # Select test data
+      test = full_df.loc[full_df["Predicted_A"] != "train"]
+
+      # Use function plot_data to plot selected data
+      plot_data(test, selectbox, characteristic_dict[selectbox])
+  
+    st.markdown('''To train a machine learning model, the data has to be split into two different sets. The first set is the training data, which will be used to teach the model the relationships between the input features (11 subtest results) and the corresponding labels ("0" and "1", assigned based on your definitions of target variables and the values you chose for the sliders). The second set is the test data, or evaluation data. It is used to assess the performance of the model. This is the data which is used to plot the confusion matrices and calculate the model metrics which you saw at the bottom of the "Define the target variable" page. This is also the data whose features you can explore in "Modelling outcomes". It is important that the training and testing data are balanced. Here, you can compare the demographic composition of the training and evaluation data. The training and evaluation datasets compositions were the same and contained the same candidates and same features for both models A and B. However, the labels for each dataset were different and based on what you selected in "Define target variable".''')
+
+def filter_for_protected(data):
+  st.markdown('''Sometimes, the overall model metrics can be deceptive when it comes to predicting the results for different groups under consideration. Ideally, for our models, the varying model metrics would be similar across different groups, which would indicate that the overall model performance is reflected in how this model performs for a given group. It is often not the case, and it is likely that you will see that models A and B perform differently when it comes to those metrics. Even the same model can have different metrics for different subgroups.''')
+  model = st.selectbox('Choose which model outputs to assess', pred_dict.keys())
+  test = data.loc[data[pred_dict[model]] != "train"]
+
+  selectbox_Char = st.selectbox('Protected characteristic', characteristic_dict.keys())
+  if selectbox_Char == 'age':
+    bins= [18,20,30,40,50,60,70,80,91]
+    labels = ['18-20','21-30','31-40','41-50','51-60','61-70','71-80','81-90']
+    test['age_bins'] = pd.cut(test['age'], bins=bins, labels=labels, right=False)
+    selectbox_Char = 'age_bins'
+  # which_group = st.selectbox('Which group?', test[selectbox_Char].unique())
+  df = pd.DataFrame({'Measure': ['True Positive Rate', 'True Negative Rate', 'Positive Predictive Value', 'Negative Predictive Value', 'False Positive Rate', 'False Negative Rate', 'False Discovery Rate']})
+  for group in test[selectbox_Char].unique():
+      rslt_df = test[test[selectbox_Char] == group] 
+      y_true = [int(numeric_string) for numeric_string in rslt_df[model_dict[model]]]
+      y_pred = [int(numeric_string) for numeric_string in rslt_df[pred_dict[model]]]
+      cm = confusion_matrix(y_true, y_pred)
+      if cm.shape == (1,1):
+          cm = np.array([[cm[0, 0], 0], [0, 0]])
+      d = plot_conf_rates(cm)
+      df[f"{group}"] = d["Score"]
+
+  fig = go.Figure()
+  for group in test[selectbox_Char].unique():
+      fig.add_trace(go.Bar(
+          x=df["Measure"],
+          y=df[group],  
+          name=group
+      ))
+      
+  fig.update_layout(
+          title='Model metrics per group',
+          xaxis_tickfont_size=14,
+          xaxis=dict(
+              title='Metric'),
+          yaxis=dict(
+              title='Score',
+              titlefont_size=16,
+              tickfont_size=14,
+          ),
+          legend=dict(
+              bgcolor='rgba(255, 255, 255, 0)',
+              bordercolor='rgba(255, 255, 255, 0)'
+          ),
+    barmode='group',
+    bargap=0.15, # gap between bars of adjacent location coordinates.
+    bargroupgap=0.1 # gap between bars of the same location coordinate.
+      )
+  st.plotly_chart(fig, use_container_width = True)
+  if st.checkbox("Show table of scores"):
+    # CSS to inject contained in a string
+    hide_table_row_index = """
+    <style>
+    thead tr th:first-child {display:none}
+    tbody th {display:none}
+    </style> """
+
+    # Inject CSS with Markdown
+    st.markdown(hide_table_row_index, unsafe_allow_html=True)
+    st.markdown(f"Accuracy rates for {selectbox_Char}")
+    st.table(df)
+
+def data_plot(key1, key2, key3, key4):
+  st.title('''Visualize the Results''')
+  if key1 not in st.session_state:
+    st.error('Cannot train the models if you do not define the target variables. Go to "Define Target Variables"!', icon="🚨")
+  else:
+    tab1, tab2 = st.tabs(["Demographic", "Non-demographic"])
+    with tab1:
+      dataframe = st.session_state[key1]
+      clean_data = st.session_state[key2]
+      st.subheader('''**Select what to explore:**''')
+      data_choice = st.radio('''What to explore''', ("Modelling outcomes", "Input data"), horizontal = True, label_visibility = "collapsed")
+      if data_choice == "Modelling outcomes":
+          st.subheader('''Demographics of the overall modelling outcomes''')
+          model_out(dataframe)
+          st.subheader('''Demographics of the selected protected groups''')
+          filter_for_protected(dataframe)
+      else:
+          st.subheader('''Demographics of the input scores''')
+          model_scores(dataframe)
+          st.subheader('''Demographics of the input labels''')
+          model_vis(dataframe)
+    with tab2:
+      dataframe = st.session_state[key1]
+      clean_data = st.session_state[key2]
+      cmA = st.session_state[key3]
+      cmB = st.session_state[key4]
+      st.subheader('''**Select what to explore:**''')
+      data_choice = st.radio('''Select what to explore:''', ("Modelling outcomes", "Input data"), horizontal = True, label_visibility = "collapsed")
+      if data_choice == "Modelling outcomes":
+          st.subheader('''Labelled dataframe''')
+          dataframe_out(dataframe)
+          st.subheader('''Venn Diagram''')
+          venn_diagram(dataframe)
+          st.subheader('''Model accuracy metrics''')
+          mod_prop(cmA, cmB)
+      else:
+          st.subheader('''Principal Component Analysis''')
+          PCA_general(dataframe, clean_data)
+    
+data_plot('complete_df', 'clean_df', 'cm_A', 'cm_B')

pages/3_💡_Put_the_Idea_into_Practice.py

@@ -0,0 +1,57 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+"""
+Created on Thu May 25 15:42:38 2023
+
+@author: daliagala
+"""
+
+### IMPORT LIBRARIES ###
+import streamlit as st
+
+### PAGE CONFIG ###
+st.set_page_config(page_title='FairTargetSim', page_icon=':robot_face:', layout='wide')
+
+hide_st_style = """
+            <style>
+            #GithubIcon {visibility: hidden;}
+            #MainMenu {visibility: hidden;}
+            footer {visibility: hidden;}
+            header {visibility: hidden;}
+            </style>
+            """
+st.markdown(hide_st_style, unsafe_allow_html=True)
+
+### IDEA IN PRACTICE PAGE ###
+
+st.title("Put the Idea into Practice")
+st.markdown('''This simulator is designed to help you understand the impacts—social, ethical, and technical—of target variable definition. On this page, we describe how to put the simulator, and the insights it affords, into practice. As with much of the simulator, we will focus on the case of hiring, but what we say applies just as well to other domains, such as college admissions or news recommendation, as we discussed on the Home page.''')
+
+st.markdown('''How can the simulator be put into practice? The first step is to make two changes to the simulator, because you cannot simply take it “off the shelf” and immediately put it into practice. This is because:''')
+
+st.markdown('''- The simulator is not built using your data or your models.''')
+
+st.markdown('''- How you define a target variable in the simulator is not how it’s done in practice. (Rather, the way you define it in the simulator is a straightforward way for you to see the effects of target variable definition.)''')
+st.markdown('''Below we describe how to address these two issues.''')
+
+st.subheader('''Using your data and your models''')
+
+st.markdown('''The simulator offers a starting point: its code is open-source. If you want to build something like the simulator for your data and your models, you now have a blue-print to work from.''')
+
+st.subheader('''Defining the target variable in practice''')
+
+st.markdown('''In the simulator, you define the target variable by assigning weights to different cognitive characteristics. These weights determine the “positive label:” people in our dataset who perform best on the tests, given the weights you assign, are those assigned the positive label—that is, those that the model treats as “good.” Then, the model is trained to identify people whose cognitive characteristics match those with the positive label.''')
+
+st.markdown('''As we discussed on the Home page, a growing number of hiring algorithms use cognitive tests to identify promising job applicants. However, as we touched on in the Visualize Results page, the way target variable definition works with these real-world algorithms is different from how it works in the simulator. For example, consider Pymetrics, a leading developer of hiring software. In some cases, Pymetrics builds bespoke algorithms for a company that is hiring for a given role. Pymetrics will ask the company to identify a group of the client’s current employees in that role that the client considers “good.” Then, these “good” employees play cognitive test games similar to the ones used in our simulator. It is these employees who are assigned the positive labels. From this point on, Pymetrics’ algorithmic development goes just as it does in our simulator: a model is trained to identify job applicants whose cognitive characteristics are similar to those with the positive label.''')
+
+st.markdown('''So, for hiring algorithms like Pymetrics’, the target variable is defined not by assigning weights to cognitive attributes, but rather by directly identifying a certain group of current employees as “good.” In the simulator, you can define different target variables by assigning different weights to the cognitive attributes. If you are in practice building an algorithm like Pymetrics’, you can define different target variables by identifying different groups of current employees as “good.”''')
+
+st.markdown('''How might this work? As we discussed on the Home page of the simulator, reasonable minds may disagree about what makes for a good employee, and relatedly reasonable minds may disagree about which current employees are good employees. For example, within a company, two different managers—call them Manager A and Manager B—may not be perfectly aligned in who they consider to be good employees for a certain role. The managers may agree in some cases. We might imagine that there are 50 employees whom both Manager A and Manager B deem good. But the two managers might disagree about other employees. Imagine that there are 25 further employees whom Manager A thinks of as good but Manager B does not (this needn’t mean that Manager B thinks that these employees are bad, just that they are not the best). Likewise, there might be 25 further employees whom Manager B thinks of as good but Manager A does not.''')
+
+st.markdown('''In this case, there are two different (overlapping) groups of 75 employees, each corresponding to what Managers A and B think of as good employees. These two different groups of employees—and in turn, two different target variable definitions—could be used to train two different models.''')
+
+st.markdown('''Instead of constructing two groups of “good” employees directly from the judgments of Managers A and B, you could weight their judgments against one another. For example, you could have two groups of employees, X and Y. Both X and Y contain the 50 employees that Managers A and B agree on. But group X contains 20 of Manager A’s preferred employees and 5 of Manager B’s, while group Y contains 20 of Manager B’s preferred employees and 5 of Manager A’s. Here again we have different groups of “good” employees, and so two different target variables.''')
+
+st.markdown('''One could select different groups of good employees in other ways still. An employer might have different metrics to evaluate employee success. Different employees might be better than others according to one metric compared to another. Depending on what importance is assigned to the different metrics—depending on how you weight the different metrics against one another—different groups of employees may emerge as “good.”''')
+
+st.markdown('''Our focus in the simulator has been on hiring algorithms that are based on cognitive test games. There are other kinds of algorithms used in hiring—for example, algorithms that identify promising job applicants on the basis of their resumés. In designing any such algorithm, the target variable must be defined, and the notion of a “good employee” must be translated into algorithmic terms. And so the insights of this simulator apply, and can be put into practice, for almost any kind of hiring algorithm you’re working with.''')

requirements.txt

@@ -0,0 +1,20 @@
+streamlit
+plotly
+seaborn
+python-math
+matplotlib_venn
+adjustText
+statsmodels
+numpy
+scipy
+scikit-learn
+pandas
+ipywidgets
+ipython
+matplotlib
+tqdm
+phik
+matplotlib-inline
+pytest
+Jinja2
+statkit

utils.py

@@ -0,0 +1,255 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+"""
+Created on Thu May 25 15:04:19 2023
+
+@author: daliagala
+"""
+
+### IMPORT LIBRARIES ###
+import streamlit as st
+import numpy as np
+import pandas as pd 
+import matplotlib.pyplot as plt
+import plotly.express as px
+import seaborn as sns
+import math
+from sklearn.decomposition import PCA
+from numpy import random
+from sklearn.metrics import confusion_matrix
+from sklearn import svm
+from sklearn import metrics
+from sklearn.metrics import accuracy_score
+from statkit.non_parametric import bootstrap_score
+import plotly.graph_objects as go
+
+### FUNCTIONS ###
+
+# Function: add labels by probabilities
+def assign_labels_by_probabilities(df, scores_col, label_col, probs_col, quantile=0.85, num_samples=100):
+    # Sort the dataframe by scores column in descending order
+    annotated = df.sort_values(by=scores_col, ascending=False)
+    annotated.reset_index(drop=True, inplace=True)
+
+    # Assign probability of 0 to bottom whatever quantile of scores
+    annotated.loc[annotated[scores_col] < annotated[scores_col].quantile(quantile), probs_col] = 0
+
+    # Count the number of NaN values in the probabilities column - how many scores left
+    num_nans = annotated[probs_col].isna().sum()
+
+    # Write a linear function to assign increasing probabilities
+    function = np.linspace(start=0.99, stop=0.01, num=num_nans)
+    sum_func = np.sum(function)
+    function = function/sum_func
+    function = pd.Series(function)
+
+    # Assign increasing probabilities to all NaNs
+    annotated[probs_col].fillna(value=function, inplace=True)
+
+    # Randomly select users based on assigned probabilities
+    selected = random.choice(annotated["user_id"], size=num_samples,replace=False, p=annotated[probs_col])
+    annotated[label_col] = 0
+    annotated.loc[annotated['user_id'].isin(selected), label_col] = 1
+    
+    return annotated
+
+# A function to remove protected characteristics and useless data
+def drop_data(df):
+  labels_to_drop = ["user_id", "age", "gender", "education level", "country", "test_run_id", "battery_id", "time_of_day", 
+                    "model_A_scores", "model_B_scores", "Model_A_probabilities", "Model_B_probabilities"]
+  clean = df.drop(labels_to_drop, axis = 1)
+  return clean
+
+# A function to train an SVM
+def train_and_predict(name, X_train, X_test, y_train, y_test, kernel='poly'):
+    # Define X and Y data
+    name=name
+
+    # Create a svm Classifier
+    clf = svm.SVC(kernel=kernel, probability = True) # Polynomial Kernel
+
+    # Train the model using the training sets
+    model = clf.fit(X_train, y_train.values.ravel())
+
+    # Predict the response for test dataset
+    y_pred = clf.predict(X_test)
+    cm = confusion_matrix(y_pred, y_test)
+
+    # Predict the probabilities for test dataset
+    y_pred_proba = clf.predict_proba(X_test)
+
+    # Change class probabilities into 2 flat numpy arrays
+    array1 = y_pred_proba[:, 0].reshape(-1, 1).flatten()
+    array2 = y_pred_proba[:, 1].reshape(-1, 1).flatten()
+
+    # Append predictions to X_test dataframe
+    X_eval = X_test.copy(deep=True)
+    X_eval[f"Predicted_%s" % name] = y_pred
+
+    # Append probability predictions to X_test dataframe
+    X_eval[f"Prob_0_%s" % name] = array1
+    X_eval[f"Prob_1_%s" % name] = array2
+
+    # Mark which data was used for training
+    X_tr = X_train.copy(deep = True)
+    X_tr[f"Predicted_%s" % name] = "train"
+
+    # Concatenate training and test data
+    X_full = pd.concat([X_eval, X_tr])
+
+    # Reset index and retain old index to be able to get back to sensitive data
+    X_full = X_full.reset_index()
+
+    # Calculate accuracy
+    accuracy = metrics.accuracy_score(y_test, y_pred)
+
+    # Calculate precision
+    precision = metrics.precision_score(y_test, y_pred)
+
+    # Calculate recall
+    recall = metrics.recall_score(y_test, y_pred)
+
+    baseline_accuracy = bootstrap_score(y_test, y_pred, metric=accuracy_score, random_state=5)
+
+    return accuracy, precision, recall, X_full, cm, baseline_accuracy
+
+# A function to display proportional representation of protected characteristics
+
+def display_proportional(data, protected_characteristic, which_model):
+  if protected_characteristic == 'age':
+    bins= [18,20,30,40,50,60,70,80,90]
+    labels = ['18-20','21-30','31-40','41-50','51-60','61-70','71-80','81-90']
+    data['age_bins'] = pd.cut(data['age'], bins=bins, labels=labels, right=False)
+    data_all = data.loc[data[which_model] != "train"]
+    info_all = data_all["age_bins"].value_counts()
+    data_sel = data.loc[data[which_model] == 1]
+    info_sel = data_sel["age_bins"].value_counts()
+    dict_all = dict(info_all)
+    dict_sel = dict(info_sel)
+    for key in dict_all.keys():
+      if key not in dict_sel.keys():
+        dict_sel[key] = 0
+    dict_percentage = {k: round(((dict_sel[k] / dict_all[k])*100), 2) for k in dict_all if k in dict_sel}
+    values = []
+    for label in labels:
+      values.append(dict_percentage[label])
+    fig = px.bar(x = labels, y = values, text_auto='.2s')
+    fig.update_layout(yaxis_title="percentage value", xaxis_title="category")
+    st.plotly_chart(fig, use_container_width=True)
+  else:
+    data_all = data.loc[data[which_model] != "train"]
+    info_all = data_all[protected_characteristic].value_counts()
+    data_sel = data.loc[data[which_model] == 1]
+    info_sel = data_sel[protected_characteristic].value_counts()
+    dict_all = dict(info_all)
+    dict_sel = dict(info_sel)
+    for key in dict_all.keys():
+      if key not in dict_sel.keys():
+        dict_sel[key] = 0
+    dict_percentage = {k: round(((dict_sel[k] / dict_all[k])*100), 2) for k in dict_all if k in dict_sel}
+    names = list(dict_percentage.keys())
+    values = list(dict_percentage.values())
+    fig = px.bar(x = names, y = values, text_auto='.2s')
+    fig.update_layout(yaxis_title="percentage value", xaxis_title="category")
+    st.plotly_chart(fig, use_container_width=True)
+    
+# A function to plot data depending on data type
+def plot_data(data, protected_characteristic, colour_code):
+
+  if protected_characteristic == 'age':
+    mean = data.loc[:, 'age'].mean().round(2)
+    st.markdown(f':green[The mean age for this group is %s years.]' % mean)
+    bin_width= 1
+    nbins = math.ceil((data["age"].max() - data["age"].min()) / bin_width)
+    fig = px.histogram(data, x='age', nbins=nbins)
+    fig.update_layout(margin=dict(l=20, r=20, t=30, b=0))
+    st.plotly_chart(fig, use_container_width=True)
+
+  elif protected_characteristic == 'education level':
+    data = data[protected_characteristic].value_counts().to_frame().reset_index()
+    fig = px.bar(data, x=data.iloc[:,1], y=data.iloc[:,0], orientation='h',color=data.iloc[:,1])
+    fig.update_layout(margin=dict(l=20, r=20, t=30, b=0))
+    fig.update_coloraxes(showscale=False)
+    fig.update_layout(yaxis_title=None)
+    fig.update_layout(xaxis_title=None)
+    st.plotly_chart(fig, use_container_width=True)
+
+  else:
+    data = data[protected_characteristic].value_counts().to_frame().reset_index()
+    fig = px.pie(data, values=data.iloc[:,1], names=data.iloc[:,0], color = data.iloc[:,0],
+                  height=300, width=200, color_discrete_map=colour_code)
+    fig.update_layout(margin=dict(l=20, r=20, t=30, b=0))
+    st.plotly_chart(fig, use_container_width=True)
+
+# A function to run PCA with custom no of components using sklearn
+def run_PCA(df, drop_1, retain_this, n):
+  df_clean = df.drop(columns = [drop_1, retain_this, "index"])
+  labels = list(df_clean.columns)
+  pca = PCA(n_components=n)
+  principalComponents = pca.fit_transform(df_clean)
+  if n == 2:
+    principalDf = pd.DataFrame(data = principalComponents, columns = ['principal component 1', 'principal component 2'])
+  else:
+    principalDf = pd.DataFrame(data = principalComponents, columns = ['principal component 1', 'principal component 2', 'principal component 3'])
+  finalDf = pd.concat([principalDf, df[[retain_this]]], axis = 1)
+  finalDf2 = finalDf.rename(columns = {retain_this : 'target'})
+  coeff = np.transpose(pca.components_[0:2, :])
+  return pca, finalDf2, labels, coeff, principalComponents
+
+# Plot confusion matrices as heatmaps
+def create_confusion_matrix_heatmap(confusion_matrix, model):
+    group_names = ['True Negative (TN)','False Positive (FP)','False Negative (FN)','True Positive (TP)']
+    group_counts = ["{0:0.0f}".format(value) for value in confusion_matrix.flatten()]
+    group_percentages = ["{0:.2%}".format(value) for value in confusion_matrix.flatten()/np.sum(confusion_matrix)]
+    labels = [f"{v1}<br>{v2}<br>{v3}" for v1, v2, v3 in zip(group_names,group_counts,group_percentages)]
+    labels = np.asarray(labels).reshape(2,2)
+    
+    layout = {
+        "title": f"Confusion Matrix, {model}", 
+        "xaxis": {
+            "title": "Predicted value", 
+            "tickmode" : 'array', 
+            "tickvals" : [0, 1], 
+            "ticktext" : ["0", "1"]}, 
+        "yaxis": {
+            "title": "Actual value", 
+            "tickmode" : 'array', 
+            "tickvals" : [0, 1], 
+            "ticktext" : ["0", "1"]},
+        }
+    fig = go.Figure(data=go.Heatmap(
+                    z=confusion_matrix,
+                    text=labels,
+                    texttemplate="%{text}",
+                    textfont={"size":15}), layout = layout)
+    st.plotly_chart(fig, use_container_width = True)
+
+# Display model metrics as tables
+def plot_conf_rates(confusion_matrix):
+  TN = confusion_matrix[0,0]
+  TP = confusion_matrix[1,1]
+  FP = confusion_matrix[0,1]
+  FN = confusion_matrix[1,0]
+
+
+  # Sensitivity, hit rate, recall, or true positive rate
+  TPR = TP/(TP+FN)
+  # Specificity or true negative rate
+  TNR = TN/(TN+FP) 
+  # Precision or positive predictive value
+  PPV = TP/(TP+FP)
+  # Negative predictive value
+  NPV = TN/(TN+FN)
+  # Fall out or false positive rate
+  FPR = FP/(FP+TN)
+  # False negative rate
+  FNR = FN/(TP+FN)
+  # False discovery rate
+  FDR = FP/(TP+FP)
+
+  # Overall accuracy
+  ACC = (TP+TN)/(TP+FP+FN+TN)
+  d = {'Measure': ['True Positive Rate', 'True Negative Rate', 'Positive Predictive Value', 'Negative Predictive Value', 'False Positive Rate', 'False Negative Rate', 'False Discovery Rate'], 
+  'Equation' : ['TPR = TP/(TP+FN)', 'TNR = TN/(TN+FP)', 'PPV = TP/(TP+FP)', 'NPV = TN/(TN+FN)', 'FPR = FP/(FP+TN)', 'FNR = FN/(TP+FN)', 'FDR = FP/(TP+FP)'], 
+  'Score': [TPR, TNR, PPV, NPV, FPR, FNR, FDR]}
+  return d

🏠Home.py

@@ -0,0 +1,58 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+"""
+Created on Thu May 25 15:13:58 2023
+
+@author: daliagala
+"""
+
+### LIBRARIES ###
+import streamlit as st
+
+### PAGE CONFIG ###
+st.set_page_config(page_title='FairTargetSim', page_icon=':robot_face:', layout='wide')
+
+hide_st_style = """
+            <style>
+            #GithubIcon {visibility: hidden;}
+            #MainMenu {visibility: hidden;}
+            footer {visibility: hidden;}
+            header {visibility: hidden;}
+            </style>
+            """
+st.markdown(hide_st_style, unsafe_allow_html=True)
+
+#Set title
+st.title('FairTargetSim')
+st.subheader('''Interactive Target Variable Definition Simulator''')
+
+#Project description
+st.subheader('Motivation')
+
+st.markdown('''As machine learning techniques advance, algorithmic systems play an increasing role in almost every domain of our lives. Machine learning requires translating real-world problems into mathematical representations. Sometimes, the translation is straightforward—for example, in predicting whether someone will default on a loan. Other times, things aren't so simple. Consider an algorithm that aims to predict which job applicants will be a good employee for a given job. In order for an algorithmic system to identify potentially good employees, the notion of a “good” employee must be defined in terms that an algorithm can work with. In machine learning, this is called “defining the target variable” (in the case of hiring, the algorithm “aims at the target” of finding good employees).''')
+
+st.markdown('''Defining the target variable is difficult. Imagine that you are hiring a salesperson. What makes for a good salesperson? Simply someone who makes the most profitable sales? Or is a good salesperson also a good leader? Does a good salesperson come up with new ideas that can improve how the sales team operates as a whole, and not just their individual sales? (The list could go on.) Perhaps the answer is: some of everything.''')
+
+st.markdown('''But then we ask: how much of each thing? How much more important are individual sales than leadership, for example? Put another way: there may be different ways of understanding which qualities matter for being a good salesperson, and to what degree; reasonable minds may disagree on these issues (as anyone who’s been on a hiring committee has experienced). Even once it’s decided what makes for a good salesperson, there is a further question of how to make the notion precise in algorithmic terms: how do we identify job applicants with sales ability, leadership qualities, or innovative thinking? In order for the algorithm to be able to positively select those applicants, those qualities have to somehow be encoded numerically.''')
+
+st.markdown('''Defining the target variable is not only difficult; it can also have profound effects on fairness—by resulting in hiring disparities for protected groups [(Passi & Barocas 2019)](https://dl.acm.org/doi/10.1145/3287560.3287567). For example, if you define the notion of a “good” employee in one way you might end up hiring more women than if you were to define “good” in another way. Relatedly, machine learning models might behave differently depending on how “good” employee is defined. Defining the notion in one way might lead to your model being less accurate for older applicants than for younger applicants.''')
+
+st.markdown('''These issues are not limited to hiring contexts. They arise in any case where there is no simple mathematical translation of a real-world problem. Take university admissions, for example. One might use an algorithm to predict which applicants will be “good” students.” What makes for a good student, though? A student who performs the best on exams at the end of their degree? A student who improves the most in their time at university? A student who doesn’t drop out, or who wins awards, or who gets a prestigious job after graduating, or contributes to the university in extracurricular activities? As with saying what makes for a good salesperson, the answer may be “some of everything,” and so again the question arises: how much of everything? Or consider another case: a news recommendation algorithm for an online platform. What makes for a “good” recommendation? Is it one that maximizes the user’s time on the platform, or that maximizes ad sales, or that is not “biased” along political lines (and then: which political lines?), or that best satisfies the user’s preferences, or that does not spread misinformation, or that prevents political polarization, or…? How these questions are answered—and how these different considerations are weighed against one another—has profound implications for fairness and other social and ethical concerns.''')
+
+st.markdown('''Target variable definition, then, is not a merely technical matter. The question of what makes for a “good” employee, student, or news recommendation (and so on) is fundamentally value-laden. It calls for close attention and transparency [(Fazelpour & Danks, 2021)](https://compass.onlinelibrary.wiley.com/doi/full/10.1111/phc3.12760). All too often, though, target variables are defined in technical settings without attention to fairness. Further, stakeholders who aren't a part of the technical process—like managers in non-technical roles, or those working in upper management or human resources—either do not understand, or are simply not aware of, the fraught nature of target variable definition.''')
+
+st.markdown('''We have developed FairTargetSim (FTS) to help address this issue. The simulator makes the implications of target variable definition explicit—and transparent—and offers a blue-print for those who want to address these effects in practice. FTS uses the case study of hiring; the lessons one can draw from it extend to any domain in which there are no clear-cut answers to the question of target variable definition.''')
+
+st.subheader('Overview of the simulator')
+st.markdown('''The simulator has three pages, which are best visited in order.''')
+st.markdown('''- **Define Target Variables.** On this page, we invite you to imagine that you are building a hiring algorithm for a certain role. You can define two different target variables—two different ways of understanding what counts as a good employee. The simulator then uses your target variables to generate two datasets and two models. The first model predicts which candidates will be good employees according to your first definition of “good;” the second model predicts which candidates will be good employees according to your second definition.''')
+st.markdown("- **Visualize The Results.** This page contains visualizations that illustrate how your two target variable definitions impact issues of fairness and overall model performance. You can see, for example, which model selects more female applicants, or which model is more accurate for older applicants. You can also see, among other things, how the two models differ in overall performance. In addition, you can see how your target variable definitions affect the data that go into training the model.")
+st.markdown("- **Put the Idea into Practice.** This page contains guidance for putting the simulator, and the ideas behind it, into practice. A practitioner who is building or using their own hiring algorithms cannot take our simulator “off the shelf” and apply it directly to their own data or models. We give guidance for how a practitioner could adapt our simulator to use in their own work.")
+
+st.subheader('Example')
+st.markdown('''Below is an example of the simulator in action. On the Define Target Variables page, you’ll assign the importance of different cognitive characteristics by setting sliders that represent five different cognitive characteristics; you do this twice, and then the simulator builds two models, A and B. On the Visualize the Results page, you’ll see how even very small changes—such as changing one point of importance for “behavioral restraint’’ (highlighted in green)—can result in completely different outcomes for the models.''')
+st.markdown('''**From the Define Target Variables page:**''')
+st.image('./images/tests.png')
+
+st.markdown('''**From the Visualize the Results page:**''')
+st.image('./images/pie_charts.png')