removing cached git files

pages/.ipynb_checkpoints/1_📊_Define_Target_Variables-checkpoint.py

@@ -1,266 +0,0 @@
-#!/usr/bin/env python3
-# -*- coding: utf-8 -*-
-"""
-Created on Thu May 25 15:16:50 2023
-
-@author: daliagala
-"""
-
-### LIBRARIES ###
-import streamlit as st
-import pandas as pd 
-from sklearn.model_selection import train_test_split
-from utils import assign_labels_by_probabilities, drop_data, train_and_predict
-
-### PAGE CONFIG ###
-st.set_page_config(page_title='EquiVar', page_icon=':robot_face:', layout='wide')
-
-hide_st_style = """
-            <style>
-            #GithubIcon {visibility: hidden;}
-            #MainMenu {visibility: hidden;}
-            footer {visibility: hidden;}
-            header {visibility: hidden;}
-            </style>
-            """
-st.markdown(hide_st_style, unsafe_allow_html=True)
-
-### IMPORT DATA FILES ###
-dataframe = pd.read_csv('./data/dataframe.csv')
-dataframe = dataframe.drop(["Unnamed: 0"], axis = 1)
-dataframe = dataframe.rename(columns={"education_level": "education level"})
-
-### DICTIONARIES AND CONSTANTS###
-groups = ["attention", "reasoning", "memory", "behavioural restraint", "information processing speed"]
-
-groups_dict = {
-          'Divided Visual Attention' : 'attention',
-          'Forward Memory Span': 'memory',
-          'Arithmetic Reasoning' : 'reasoning',
-          'Grammatical Reasoning' : 'reasoning',
-          'Go/No go' : 'behavioural restraint',
-          'Reverse_Memory_Span' : 'memory',
-          'Verbal List Learning' : 'memory',
-          'Delayed Verbal List Learning' : 'memory',
-          'Digit Symbol Coding' : 'information processing speed',
-          'Trail Making Part A' : 'information processing speed',
-          'Trail Making Part B' : 'information processing speed'
-        }
-
-education_dict = {
-    1: 'Some high school',
-    2: 'High school diploma / GED',
-    3: 'Some college',
-    4: 'College degree',
-    5: 'Professional degree',
-    6: "Master's degree",
-    7: 'Ph.D.',
-    8: "Associate's degree",
-    99: 'Other'
-  }
-
-df_keys_dict = {
-          'Divided Visual Attention' :'divided_visual_attention',
-          'Forward Memory Span' :'forward_memory_span',
-          'Arithmetic Reasoning' : 'arithmetic_problem_solving',
-          'Grammatical Reasoning'  : 'logical_reasoning',
-          'Go/No go': 'adaptive_behaviour_response_inhibition',
-          'Reverse_Memory_Span' : 'reverse_memory_span',
-          'Verbal List Learning': 'episodic_verbal_learning',
-          'Delayed Verbal List Learning': 'delayed_recall',
-          'Digit Symbol Coding': 'abstract_symbol_processing_speed',
-          'Trail Making Part A' :'numerical_info_processing_speed',
-          'Trail Making Part B': 'numerical_and_lexical_info_processing_speed'
-        }
-
-
-### CREATE THE "TARGET VARIABLE DEFINITION" PAGE ###
-st.title('Target variable definition')
-
-st.markdown('''On this page, we invite you to imagine that you are hiring for a certain role. Using the sliders below, you will specify two different notions of a “good employee” for that role—two different target variables. Once you’re done, the simulator will build two models, one for each of your target variable definitions. You can visualize these datasets and models—and their effects on fairness and overall features of the models and data—in the Visualize the Results page.''')
-
-st.markdown('''You specify the notions of different employees by assigning weights of importance to cognitive characteristics that a “good” employee would have for the role. The cognitive test data that you’ll be working with comes from real-world people, mirroring an increasing number of hiring algorithms that are based cognitive tests ([Wilson, et al. 2021](https://dl.acm.org/doi/10.1145/3442188.3445928)).''')
-
-st.markdown('''We have pre-set the weights below to reflect two different conceptions of a “good” employee: one conception that emphasizes attentiveness and numerical skills (A) and another conception that emphasizes interpersonal skills and memory (B). If you want to, you can change the slider values as you see fit.''')
-
-st.markdown('''After you’ve set the slider values, click :red[“Assign labels and train your models”]. The simulator will then label certain individuals as "good employees"—in other words it will assign class labels "1" for successful and "0" for not good, based on your selections. Then, the simulator will build the two models.''')
-
-st.markdown('''To learn more about target variable definition and hiring models based in cognitive tests, :green["See explanation"] below.''')
-
-with st.expander("See explanation"):
-    
-    st.markdown('''The models that the simulator builds are of a kind increasingly used in hiring software: companies will have applicants play games that test for different kinds of cognitive ability (like reasoning or memory). Then hiring software will be built to predict which applicants will be successful based on which cognitive characteristics they have. What cognitive characteristics make for a successful employee? This will depend on what role is being hired. And it will also depend on how one defines “successful employee.”''')
-    
-    st.markdown('''In the real world, “successful employee” is defined for these kinds of hiring models in the following way. Managers select a group of current employees that they consider to be successful; this group of employees plays the cognitive test games. A model is then trained to identify applicants who share cognitive characteristics with the current employees that are considered successful. The target variable of “successful employee” is thus defined in terms of comparison to certain people who are deemed successful. ''')
-
-    st.markdown('''One will get different target variables if one deems different current employees as the successful ones. And, as we discussed in the Home page (and as we explain more in the Putting the Idea into Practice page), there will likely be disagreement between managers about which employees are successful. For instance, a manager who values attentiveness and numerical skills will deem different employees “successful” than a manager who values interpersonal skills and memory. Even when different managers roughly share their sensibilities in what characteristics make for a successful employee, there may still be different, equally good ways to “weight” the importance of the various characteristics.''')
-    
-    st.markdown('''In the real world, the cognitive characteristics shared by those considered successful employees is implicit. Companies do not first identify the cognitive characteristics that make for a successful employee; rather, they identify employees who they consider successful, and then the hiring model works backwards to identify what characteristics these employees share.''')
-    
-    st.markdown('''In our simulator, the cognitive characteristics shared by “good” employees are explicit. You assign different weights—using the sliders—to the cognitive characteristics you think are more or less important in a good employee (for the role you’re considering). To illustrate how different target variables have different effects on fairness and overall model attributes, you’ll define “good” employee in two ways. (We’ve made the cognitive characteristics explicit both so you can see the point of different target variable definitions more clearly, and because of limitations of the data that we’re working with.)''')
-
-    st.markdown('''The cognitive characteristics that the simulator works with are from one of datasets of the [NeuroCognitive Performance Test](https://www.nature.com/articles/s41597-022-01872-8). This dataset has eleven different tests which we have grouped into five categories: ''')
-    
-    st.markdown(
-          """
-          - **Memory**: Forward Memory Span, Reverse Memory Span, Verbal List Learning, Delayed Verbal List Learning
-          - **Information Processing Speed**: Digit Symbol Coding, Trail Making Part A, Trail Making Part B
-          - **Reasoning**: Arithmetic Reasoning, Grammatical Reasoning
-          - **Attention**: Divided Visual Attention
-          - **Behavioral Restraint**: Go/No go
-          """)
-    
-    st.markdown('''After you’ve set the weights to these five characteristics using the sliders, you can see which weights are assigned to each test (e.g. Forward Memory Span or Digit Symbol Coding) by ticking the checkbox beneath the sliders. ''')
-
-col1, col2 = st.columns(2)
-
-#Initialise slider values
-list_values_A = (9, 10, 2, 1, 5)
-list_values_B = (1, 2, 10, 9, 3)
-
-selectionsA = {}
-selectionsB = {}
-results_dict_A = groups_dict
-results_dict_B = groups_dict
-
-with col1:
-    st.subheader("Define target variable for model A ")
-        
-    if "slider_values_A" not in st.session_state:
-        for count, value in enumerate(groups):
-            selectionsA[value] = list_values_A[count]
-        st.session_state["slider_values_A"] = selectionsA
-    else:
-        selectionsA = st.session_state["slider_values_A"]
-            
-    for i in groups:
-        nameA = f"{i} importance, model A"
-        value = selectionsA[i]
-        slider = st.slider(nameA, min_value=0, max_value=10, value = value)
-        selectionsA[i] = slider
-        
-    results_dict_A = {k: selectionsA.get(v, v) for k, v in results_dict_A.items()}
-    total = sum(results_dict_A.values())
-    for (key, u) in results_dict_A.items():
-      if total != 0:
-        w = (u/total)
-        results_dict_A[key] = w
-
-    if st.checkbox("Show target variable A weights per subtest", key="A"):
-      for (key, u) in results_dict_A.items():
-        txt = key.replace("_", " ")
-        st.markdown("- " + txt + " : " + f":green[{str(round((u*100), 2))}]")
-        
-    
-    st.session_state["slider_values_A"] = selectionsA
-
-
-with col2:
-    st.subheader("Define target variable for model B ")
-        
-    if "slider_values_B" not in st.session_state:
-        for count, value in enumerate(groups):
-            selectionsB[value] = list_values_B[count]
-        st.session_state["slider_values_B"] = selectionsB
-    else:
-        selectionsB = st.session_state["slider_values_B"]
-            
-    for i in groups:
-      nameB = f"{i} importance, model B"
-      value = selectionsB[i]
-      slider = st.slider(nameB, min_value=0, max_value=10, value = value)
-      selectionsB[i] = slider
-      
-    results_dict_B = {k: selectionsB.get(v, v) for k, v in results_dict_B.items()}
-    total = sum(results_dict_B.values())
-    for (key, u) in results_dict_B.items():
-      if total != 0:
-        w = ((u/total))
-        results_dict_B[key] = w
-
-    if st.checkbox("Show target variable B weights per subtest", key = "B"):
-      for (key, u) in results_dict_B.items():
-        txt = key.replace("_", " ")
-        st.markdown("- " + txt + " : " + f":green[{str(round((u*100), 2))}]")
-      
-    st.session_state["slider_values_B"] = selectionsB
-          
-if st.button("Assign labels and train your models", type = "primary", use_container_width = True):
-  if 'complete_df' in st.session_state:
-      del st.session_state['complete_df']
-  if 'clean_df' in st.session_state:
-      del st.session_state['clean_df']
-  if 'cm_A' in st.session_state:
-      del st.session_state['cm_A']
-  if 'cm_B' in st.session_state:
-      del st.session_state['cm_B']
-  scoreA = pd.DataFrame()
-  scoreB = pd.DataFrame()
-  test1 = all(value == 0 for value in results_dict_A.values())
-  test2 = all(value == 0 for value in results_dict_B.values())
-  if test1 == True or test2 == True:
-    st.error('Cannot train the models if you do not define the target variables. Make your selections for both models first!', icon="🚨")
-  else:
-    for (key, u) in results_dict_A.items():
-      scoreA[df_keys_dict[key]] = u * dataframe[df_keys_dict[key]]
-      scoresA = scoreA.sum(axis=1)
-      dataframe['model_A_scores'] = scoresA
-    for (key, u) in results_dict_B.items():
-      scoreB[df_keys_dict[key]] = u * dataframe[df_keys_dict[key]]
-      scoresB = scoreB.sum(axis=1)
-      dataframe['model_B_scores'] = scoresB
-    
-    new_annotated = assign_labels_by_probabilities(dataframe, "model_A_scores", "Model_A_label", "Model_A_probabilities", quantile=0.85, num_samples=100)
-    new_annotated = assign_labels_by_probabilities(new_annotated, "model_B_scores", "Model_B_label", "Model_B_probabilities", quantile=0.85, num_samples=100)
-    new_annotated = new_annotated.reset_index()
-
-
-    clean_data = drop_data(new_annotated)
-    # specify the columns of interest
-    selected_cols = ['Model_A_label', 'Model_B_label']
-    
-    # count the number of rows where all three selected columns have a value of 1
-    num_rows_with_all_flags_1 = len(new_annotated[new_annotated[selected_cols].sum(axis=1) == len(selected_cols)])
-    
-    # print the result
-    st.write(f"Shared candidates between your target variables: :green[{num_rows_with_all_flags_1}].")
-    with st.spinner('Please wait... The models will be trained now.'):
-
-      X_data, Y_data_A, Y_data_B = clean_data.iloc[:, :-2], clean_data.iloc[:, [-2]], clean_data.iloc[:, [-1]]
-      X_data = X_data.drop(["index"], axis = 1)
-      Y_data_B = Y_data_B.reset_index()
-      X_train, X_test, y_train_A, y_test_A = train_test_split(X_data, Y_data_A, test_size=0.2)
-      y_train_A = y_train_A.reset_index()
-      y_test_A = y_test_A.reset_index()
-      y_train_B = pd.merge(y_train_A,Y_data_B[['index', 'Model_B_label']],on='index', how='left')
-      y_test_B = pd.merge(y_test_A,Y_data_B[['index', 'Model_B_label']],on='index', how='left')
-      y_train_B = y_train_B.drop(labels='Model_A_label', axis = 1)
-      y_test_B = y_test_B.drop(labels='Model_A_label', axis = 1)
-      y_train_A = y_train_A.set_index("index")
-      y_train_B = y_train_B.set_index("index")
-      y_test_A = y_test_A.set_index("index")
-      y_test_B = y_test_B.set_index("index")
-
-      accuracy_A, precision_A, recall_A, X_full_A, cm_A, baseline_accuracy_A = train_and_predict("A", X_train, X_test, y_train_A, y_test_A)
-      accuracy_B, precision_B, recall_B, X_full_B, cm_B, baseline_accuracy_B = train_and_predict("B", X_train, X_test, y_train_B, y_test_B)
-      full = pd.merge(X_full_A,X_full_B[['index','Predicted_B', 'Prob_0_B', "Prob_1_B"]],on='index', how='left')
-      complete = pd.merge(full,new_annotated[['index', 'age', 'gender', 'education level', 'country', 'Model_A_label', 'Model_B_label', 'model_A_scores', 'model_B_scores']],on='index', how='left')
-      complete=complete.replace({"education level": education_dict})
-      complete = complete.rename(columns={"index": "Candidate ID"})
-      
-      if 'complete_df' not in st.session_state:
-        st.session_state['complete_df'] = complete
-      if 'clean_df' not in st.session_state:
-        st.session_state['clean_df'] = clean_data
-      if 'cm_A' not in st.session_state:
-        st.session_state['cm_A'] = cm_A
-      if 'cm_B' not in st.session_state:
-        st.session_state['cm_B'] = cm_B
-
-    row1_space1, row1_1, row1_space2, row1_2, row1_space3 = st.columns((0.1, 3, 0.1, 3, 0.1))
-    with row1_1:
-      st.write(f"Model A accuracy: :green[{baseline_accuracy_A}].")
-    with row1_2:
-      st.write(f"Model B accuracy: :green[{baseline_accuracy_B}].")
-
-    st.success('''Success! You have defined the target variables and trained your models. Head to "Visualise the Results" in the sidebar.''')

pages/.ipynb_checkpoints/2_📈_Visualize_the_Results-checkpoint.py

@@ -1,593 +0,0 @@
-#!/usr/bin/env python3
-# -*- coding: utf-8 -*-
-"""
-Created on Thu May 25 15:30:56 2023
-
-@author: daliagala
-"""
-
-### LIBRARIES ###
-import streamlit as st
-import numpy as np
-import pandas as pd 
-import matplotlib.pyplot as plt
-import plotly.express as px
-import plotly.graph_objects as go
-from matplotlib_venn import venn2
-from sklearn.metrics import confusion_matrix
-from utils import display_proportional, plot_data, run_PCA, create_confusion_matrix_heatmap, plot_conf_rates
-
-### PAGE CONFIG ###
-st.set_page_config(page_title='EquiVar', page_icon=':robot_face:', layout='wide')
-
-hide_st_style = """
-            <style>
-            #GithubIcon {visibility: hidden;}
-            #MainMenu {visibility: hidden;}
-            footer {visibility: hidden;}
-            header {visibility: hidden;}
-            </style>
-            """
-st.markdown(hide_st_style, unsafe_allow_html=True)
-
-### DICTIONARIES AND CONSTANTS###
-
-colours_education = {
-          'Some high school' : 'indigo',
-          'High school diploma / GED' : '#7ae99e',
-          'Some college' : '#0a68c9',
-          'College degree': '#80c4fa',
-          'Professional degree': '#f48508',
-          "Master's degree" : '#2fa493',
-          'Ph.D.' : '#f2a3a1',
-          "Associate's degree" : '#fbcc66',
-          'Other' : '#fa322f'
-        }
-
-colours_country = {
-          'AU' : '#7ae99e',
-          'US': '#80c4fa',
-          'NZ': '#2fa493',
-          'CA' : '#fbcc66'
-        }
-
-colours_gender = {
-          'f' : '#83C9FF',
-          'm': '#0067C9'
-        }
-
-characteristic_dict = {
-          'gender' : colours_gender,
-          'education level' : colours_education,
-          'country' : colours_country,
-          'age' : 'indigo'
-        }
-
-pred_dict = {
-          'Model A' : 'Predicted_A',
-          'Model B' : 'Predicted_B'
-        }
-
-prob_dict = {
-          'Model A' : 'Prob_1_A',
-          'Model B' : 'Prob_1_B'
-        }
-
-model_dict = {
-          'Model A' : 'Model_A_label',
-          'Model B' : 'Model_B_label'
-        }
-
-df_keys_dict = {
-          'Divided Visual Attention' :'divided_visual_attention',
-          'Forward Memory Span' :'forward_memory_span',
-          'Arithmetic Reasoning' : 'arithmetic_problem_solving',
-          'Grammatical Reasoning'  : 'logical_reasoning',
-          'Go/No go': 'adaptive_behaviour_response_inhibition',
-          'Reverse_Memory_Span' : 'reverse_memory_span',
-          'Verbal List Learning': 'episodic_verbal_learning',
-          'Delayed Verbal List Learning': 'delayed_recall',
-          'Digit Symbol Coding': 'abstract_symbol_processing_speed',
-          'Trail Making Part A' :'numerical_info_processing_speed',
-          'Trail Making Part B': 'numerical_and_lexical_info_processing_speed'
-        }
-
-### DEFINE ALL SUB-PAGES AS FUNCTIONS TO CALL ###
-
-def mod_prop(cmA, cmB):
-    st.markdown('''This section contains model confusion matrices, which give us a lot of information about how good the models which we produced really are. We obtain the confusion matrices by plotting "Actual" labels on one axis, and "Predicted" labels on the other. For both models, in each square of each confusion matrix, we can see which group of candidates it represents, the number of candidates in this group, and what percentage of all candidates assessed they represent.''')
-    st.markdown('''
-    - **True Negative (TN)**: candidates predicted to have label "0" whose actual label was "0".
-    - **False Positive (FP)**: candidates predicted to have label "1" whose actual label was "0".
-    - **False Negative (FN)** candidates predicted to have label "0" whose actual label was "1".
-    - **True Positive (TP)**: candidates predicted to have label "1" whose actual label was "1".
-    ''')
-    
-    row1_space1, row1_1, row1_space2, row1_2, row1_space3 = st.columns((0.1, 3, 0.1, 3, 0.1))
-    with row1_1:
-      create_confusion_matrix_heatmap(cmB, "Model A")
-    
-    with row1_2:
-      model = "B"
-      create_confusion_matrix_heatmap(cmB, "Model B")
-    
-    st.subheader("Model Accuracy Rates")
-    st.markdown('''We can also represent each model in terms of its accuracy rates, calculated using the numbers of candidates in each group shown in the confusion matrix:''')
-    st.markdown('''
-    - True Positive Rate (TPR), also called recall, sensitivity or hit rate, is the probability of a positive test result when the result is indeed positive.
-    - True Negative Rate (TNR), or specificity/selectivity, is the probability of a negative test result when the test is indeed negative.
-    - Positive Predictive Value (PPV) or Precision is the ratio of truly positive results to all positive results.
-    - Negative Predictive Value (NPR) is the ratio of truly negative results to all negative results.
-    - False Positive Rate (FPR) or fall-out is the probability of assigning a falsely positive result when the test is negative.
-    - False Negative Rate (FNR) or miss rate is the probability that a test with a positive result will falsely be assigned a negative result.
-    - False Discovery Rate (FDR) it the ratio of false positive results to total number of positive results.
-    ''')
-    measures_A = plot_conf_rates(cmA)
-    measures_B = plot_conf_rates(cmB)
-    fig = go.Figure()
-    fig.add_trace(go.Bar(
-          x=measures_A["Measure"],
-          y=measures_A["Score"],  
-          name='Model A rates',
-          marker_color='rgb(55, 83, 109)'
-      ))
-    fig.add_trace(go.Bar(
-          x=measures_B["Measure"],
-          y=measures_B["Score"],
-          name='Model B rates',
-          marker_color='rgb(26, 118, 255)'
-      ))
-    fig.update_layout(
-          title='Model measures comparison',
-          xaxis_tickfont_size=14,
-          xaxis=dict(
-              title='Measure'),
-          yaxis=dict(
-              title='Score',
-              titlefont_size=16,
-              tickfont_size=14,
-          ),
-          legend=dict(
-              bgcolor='rgba(255, 255, 255, 0)',
-              bordercolor='rgba(255, 255, 255, 0)'
-          ),
-    barmode='group',
-    bargap=0.15, # gap between bars of adjacent location coordinates.
-    bargroupgap=0.1 # gap between bars of the same location coordinate.
-      )
-    st.plotly_chart(fig, use_container_width = True)
-    if st.checkbox("Show tables"):
-        row1_space1, row1_1, row1_space2, row1_2, row1_space3 = st.columns((0.1, 3, 0.1, 3, 0.1))
-        # CSS to inject contained in a string
-        hide_table_row_index = """
-        <style>
-        thead tr th:first-child {display:none}
-        tbody th {display:none}
-        </style> """
-
-        # Inject CSS with Markdown
-        st.markdown(hide_table_row_index, unsafe_allow_html=True)
-        with row1_1:
-            st.markdown("Accuracy rates for model A")
-            st.table(measures_A)
-        with row1_2:
-            st.markdown("Accuracy rates for model B")
-            st.table(measures_B)
-
-def model_scores(dataframe):
-  st.markdown('''This section displays the distribution of scores assigned to each hypothetical employee, according to the values you set on the sliders, compared with the distribution of protected characteristics. These scores are then used to assign labels to the hypothetical employees, creating two distinct datasets to train the models. In essence, these scores explicitly and numerically mimic the viewpoints of hypothetical hiring managers A and B when deciding who to label as "top employees."''')
-  st.markdown('''Just as in the original NCPT dataset analysis, the scores obtained by participants in the cognitive games decline with age. This observation provides insight into the potential issue of ageism inherent in the use of gamified hiring processes.''')
-  # Create a selectbox to choose a protected characteristic to explore
-  plot_radio = st.selectbox('Characteristic to explore', characteristic_dict.keys())
-  row2_space1, row2_1, row2_space2 = st.columns((0.1, 5, 0.1))
-
-  with row2_1:
-    data = dataframe[["model_A_scores", "model_B_scores", plot_radio]]
-
-    if plot_radio == "age":
-        bins= [18,20,30,40,50,60,70,80,90]
-        labels = ['18-20','21-30','31-40','41-50','51-60','61-70','71-80','81-90']
-        data['age_bins'] = pd.cut(data['age'], bins=bins, labels=labels, right=False)
-        plot_radio = 'age_bins'
-    colours = ['rgba(93, 164, 214, 0.5)', 'rgba(255, 144, 14, 0.5)']
-    c1, c2, c3, c4, c5 = st.columns((0.1, 3, 0.1, 3, 0.1))
-    with c2:
-        fig = px.box(data, x = plot_radio, y="model_A_scores", labels={"model_A_scores":"Dataset A Input Scores", 'age_bins':"Age"}, title="Input scores in dataset A")
-        fig.update_layout(showlegend=False)
-        fig.update_traces(marker_color='rgba(93, 164, 214, 0.5)')
-        st.plotly_chart(fig, use_container_width=True)
-    with c4:
-        fig = px.box(data, x = plot_radio, y="model_B_scores", labels={"model_B_scores":"Dataset B Input Scores", 'age_bins':"Age"}, title="Input scores in dataset B")
-        fig.update_traces(marker_color = 'rgba(255, 144, 14, 0.5)')
-        fig.update_layout(showlegend=False)
-        st.plotly_chart(fig, use_container_width=True)
-
-def PCA_general(full_df, dataframe_PCA):
-  st.markdown('''On this page, you can see the distribution of the dataset labels which were assigned based on the scores calculated from the slider values you selected previously. Principal Components Analysis, or PCA, is a technique often used to analyse and subsequently visualize datasets where there are many features per single example. This is the case with the NCPT dataset used in our simulator. Specifically, the battery which we used has 11 features per single example, the example being the player of cognitive games, and, in our metaphor, a hypothetical employee or job candidate. It is impossible to plot 11 dimensions, and PCA allows for the visualisation of multidimensional data, while also preserving as much information as possible.''')
-  choice = st.radio("What would you like to explore?", ("PCAs", "Components loading"), horizontal = True)
-  pcaA, dfA, labelsA, coeffA, componentsA = run_PCA(dataframe_PCA, 'Model_B_label', 'Model_A_label', 2)
-  pcaB, dfB, labelsB, coeffB, componentsB = run_PCA(dataframe_PCA, 'Model_A_label', 'Model_B_label', 2)
-  loadings = pcaB.components_.T * np.sqrt(pcaB.explained_variance_)
-  total_var = pcaA.explained_variance_ratio_.sum() * 100
-  dfA = dfA.rename(columns={'target': 'Dataset A'}).reset_index()
-  dfB = dfB.rename(columns={'target': 'Dataset B'}).reset_index()
-  df_all = pd.merge(dfA, dfB[['index', 'Dataset B']], on='index', how='left')
-
-  conditions = [
-      (df_all['Dataset A'] == 1) & (df_all['Dataset B'] == 0),
-      (df_all['Dataset B'] == 1) & (df_all['Dataset A'] == 0),
-      (df_all['Dataset A'] == 1) & (df_all['Dataset B'] == 1),
-      (df_all['Dataset A'] == 0) & (df_all['Dataset B'] == 0)]
-      
-  values = ['Selected A', 'Selected B', 'Selected both', 'Not selected']
-  df_all['All'] = np.select(conditions, values)
-
-  df_all = df_all.drop(["index"], axis = 1)
-  df_all.All=pd.Categorical(df_all.All,categories=['Not selected', 'Selected A', 'Selected B', 'Selected both'])
-  df_all=df_all.sort_values('All')
-
-  selections_dict = {0: 'Not selected', 1: 'Selected'}
-  df_all = df_all.replace({"Dataset A": selections_dict, "Dataset B": selections_dict})
-
-  color_dict_sel = {'Not selected': '#3366CC', 'Selected': 'grey'}
-
-  if "pca_df" not in st.session_state:
-    st.session_state.pca_df = df_all
-
-  if choice == "PCAs":
-    c1, c2 = st.columns(2)
-    with c1:
-      fig = px.scatter(st.session_state.pca_df, 
-                      x=st.session_state.pca_df['principal component 1'].astype(str),
-                      y=st.session_state.pca_df['principal component 2'].astype(str),
-                      title='Dataset A PCA',
-                      labels={"x": 'PC 1', "y": 'PC 2'},
-                      color=st.session_state.pca_df['Dataset A'],
-                      color_discrete_map=color_dict_sel)
-      fig.update_traces(marker_size = 8)
-      st.plotly_chart(fig, use_container_width=True)
-    with c2:
-      fig = px.scatter(st.session_state.pca_df, 
-                      x=st.session_state.pca_df['principal component 1'].astype(str),
-                      y=st.session_state.pca_df['principal component 2'].astype(str),
-                      title='Dataset B PCA',
-                      labels={"x": 'PC 1', "y": 'PC 2'},
-                      color=st.session_state.pca_df['Dataset B'],
-                      color_discrete_map=color_dict_sel)
-      fig.update_traces(marker_size = 8)
-      st.plotly_chart(fig, use_container_width=True)
-      
-    st.markdown(f'''These plots show the reduction of 11 dimensions (11 subtest results) to 2 dimensions. Total Variance for the data is {total_var:.2f}%. Both of the datasets have the same features, therefore they both have the same total variance. Total variance value indicates what percentage of information has been preserved when the dimensionality was reduced. Note that for both datasets, A and B, different points are labelled "1" or "0". This shows that the two datasets represent the two different target variable definitions which were created by you previously. The plots are interactive - zoom in to explore in detail.''')
-      
-    pcaA, dfA, labelsA, coeffA, componentsA = run_PCA(dataframe_PCA, 'Model_B_label', 'Model_A_label', 2)
-    pcaB, dfB, labelsB, coeffB, componentsB = run_PCA(dataframe_PCA, 'Model_A_label', 'Model_B_label', 2)
-    loadings = pcaB.components_.T * np.sqrt(pcaB.explained_variance_)
-    total_var = pcaA.explained_variance_ratio_.sum() * 100
-    dfA = dfA.rename(columns={'target': 'Dataset A'}).reset_index()
-    dfB = dfB.rename(columns={'target': 'Dataset B'}).reset_index()
-    df_all = pd.merge(dfA, dfB[['index', 'Dataset B']], on='index', how='left')
-
-    conditions = [
-        (df_all['Dataset A'] == 1) & (df_all['Dataset B'] == 0),
-        (df_all['Dataset B'] == 1) & (df_all['Dataset A'] == 0),
-        (df_all['Dataset A'] == 1) & (df_all['Dataset B'] == 1),
-        (df_all['Dataset A'] == 0) & (df_all['Dataset B'] == 0)]
-        
-    values = ['Selected A', 'Selected B', 'Selected both', 'Not selected']
-    df_all['All'] = np.select(conditions, values)
-
-    df_all = df_all.drop(["index"], axis = 1)
-    df_all.All=pd.Categorical(df_all.All,categories=['Not selected', 'Selected A', 'Selected B', 'Selected both'])
-    df_all=df_all.sort_values('All')
-
-    selections_dict = {0: 'Not selected', 1: 'Selected'}
-    df_all = df_all.replace({"Dataset A": selections_dict, "Dataset B": selections_dict})
-
-    if "pca_df" not in st.session_state:
-      st.session_state.pca_df = df_all
-
-    fig = px.scatter(st.session_state.pca_df, 
-                      x=st.session_state.pca_df['principal component 1'],
-                      y=st.session_state.pca_df['principal component 2'],
-                      title="PCA with labelled groups",
-                      color=st.session_state.pca_df["All"], 
-                      width = 800, height = 800,
-                      color_discrete_sequence=px.colors.qualitative.Safe,
-                      opacity = 0.95)
-      
-    fig.update_yaxes(
-        scaleanchor="x",
-        scaleratio=1,
-      )
-    fig.update_traces(marker_size = 10)
-    st.plotly_chart(fig)
-
-  if choice == "Components loading":
-      c1,c2 = st.columns(2)
-      loadings_df = pd.DataFrame(loadings, columns = ["PC1", "PC2"])
-      labels_A_proper = { v:k for k,v in df_keys_dict.items()}
-      loadings_df["Features"] = labels_A_proper.values()
-      with c1:
-          fig = px.bar(loadings_df, x="PC1", y="Features", orientation = 'h')
-          st.plotly_chart(fig, use_container_width = True)
-      with c2:
-          fig = px.bar(loadings_df, x="PC2", y="Features", orientation = 'h')
-          st.plotly_chart(fig, use_container_width = True)
-
-    #   fig = go.Figure()
-    #   fig.add_trace(go.Bar(
-    #       x=loadings_df["PC1"],
-    #       y=loadings_df["Features"],  
-    #       name='Principal Component 1',
-    #       marker_color='rgb(55, 83, 109)',
-    #       orientation='h'
-    #   ))
-    #   fig.add_trace(go.Bar(
-    #       x=loadings_df["PC2"],
-    #       y=loadings_df["Features"],
-    #       name='Principal Component 2',
-    #       marker_color='rgb(26, 118, 255)',
-    #       orientation='h'
-    #   ))
-    #   fig.update_layout(
-    #       title='Component loadings',
-    #       xaxis_tickfont_size=14,
-    #       xaxis=dict(
-    #           title='Loading value'),
-    #       yaxis=dict(
-    #           title='Feature',
-    #           titlefont_size=16,
-    #           tickfont_size=14,
-    #       ),
-    #       legend=dict(
-    #           bgcolor='rgba(255, 255, 255, 0)',
-    #           bordercolor='rgba(255, 255, 255, 0)'
-    #       ),
-    # barmode='group',
-    # bargap=0.15, # gap between bars of adjacent location coordinates.
-    # bargroupgap=0.1 # gap between bars of the same location coordinate.
-    #   )
-    #   st.plotly_chart(fig, use_container_width = True)
-      
-      st.markdown('''On this plot, PCA component loadings can be explored. These facilitate the understanding of how much each variable (which there are 11 of) contributes to a particular principal component. Here, the 11 variables were reduced to 2 components, which are labelled PC1 and PC2. The magnitude of the loading (here displayed as the size of the bar in the bar chart) indicates how strong the relationship between the variable and the component is. Therefore, the higher the bar, the stronger the relationship between that component and that variable. The loading's sign can be positive or negative. This indicates whether the principal component and that variable are positively or negatively correlated. We can see that multiple variables are positively correlated with PC2. Two variables, episodic verbal learning and delayed recall are negatively correlated with both of the components.''')
-
-
-def model_out(full_df):
-  st.markdown('''This section highlights the discrepancies between your two models when presented with the same pool of new, previously unseen candidates to label. Specifically, you'll be investigating the candidates assigned a "1" label by both models. These individuals would be those considered for a job interview or chosen for the role, according to your defined target variable.''')
-  # Create a selectbox to choose a protected characteristic to explore
-  selectbox = st.selectbox('Characteristic to explore', characteristic_dict.keys())
-  representation = st.selectbox("Representation", ("absolute", "proportional"))
-  row1_space1, row1_1, row1_space2, row1_2, row1_space3 = st.columns((0.1, 3, 0.1, 3, 0.1))
-  with row1_1:
-    st.subheader("Candidates selected by model A")
-
-    if representation == "absolute":
-      # Select predicted data ==1
-      data = full_df.loc[full_df['Predicted_A'] == 1]
-
-      # Use function plot_data to plot selected data
-      plot_data(data, selectbox, characteristic_dict[selectbox])
-    else:
-      display_proportional(full_df, selectbox, 'Predicted_A')
-      
-  with row1_2:
-    st.subheader("Candidates selected by model B")
-    
-    if representation == "absolute":
-      # Select predicted data ==1
-      data = full_df.loc[full_df['Predicted_B'] == 1]
-
-      # Use function plot_data to plot selected data
-      plot_data(data, selectbox, characteristic_dict[selectbox])
-
-    else:
-      display_proportional(full_df, selectbox,'Predicted_B')
-      
-  
-  st.markdown('''In this section, you're comparing the model's selections concerning four protected characteristics: age, gender, education level, and country. You can visualize these differences in two ways: "Absolute" or "Proportional".''')
-  st.markdown('''"Absolute" representation gives you the raw numbers or percentages of each characteristic chosen. For instance, if the model labeled 5 female candidates and 5 male candidates as "1", the "Absolute" outcome will display as 50% for both genders."Proportional" representation, on the other hand, shows the percentage of a group selected by the model relative to the total number of that group in the input data. For example, if the model evaluated 100 male candidates and selected 5, you will see a 5% representation. If it evaluated 200 female candidates and selected 5, it will show a 2.5% representation.''')
-  st.markdown('''If you encounter empty categories in the "Proportional" view, this indicates that while candidates from these categories were evaluated, none were labeled as "1". Hence, their proportional representation amounts to 0%.''')
-
-def dataframe_out(full_df):
-  selectbox_M = st.selectbox('Choose which model output to rank by', pred_dict.keys())
-
-  # Select data
-  data = full_df.loc[full_df[pred_dict[selectbox_M]] == 1]
-  data = data.sort_values(by = prob_dict[selectbox_M], ascending = False)
-  data = data[['Candidate ID','Prob_1_A', 'Prob_1_B', 'Predicted_A', 'Predicted_B']]
-  data = data.rename(columns={"Prob_1_A": "Ranking, model A", "Prob_1_B": "Ranking, model B", "Predicted_A": "Predicted label A", "Predicted_B": "Predicted label B"})
-  data.index = np.arange(1, len(data) + 1)
-
-  st.table(data.style.background_gradient(subset = ["Ranking, model A", "Ranking, model B"], axis=0, vmin=0.40).highlight_max(color = '#FFCD9B', subset = ["Predicted label A", "Predicted label B"], axis=0))
-  
-  st.markdown("""In this section, you can review the data for all candidates labeled "1" by the selected model, found at the top of the page. Simultaneously, you can observe the labels assigned to these same candidates by the other model. It's likely that there will be instances where candidates chosen by one model weren't selected by the other. Candidates labeled "1" are highlighted in orange in the "Predicted label A" and "Predicted label B" columns.""")
-  st.markdown('''In addition to this, you can see the probability with which each candidate was labeled "1". The intensity of the blue color indicates the candidate's ranking position - a darker blue represents a higher ranking (with 1 being the maximum and 0 the minimum). You may notice that some candidates highly ranked by one model may be ranked significantly lower by the other model.''')
-  
-def venn_diagram(full_df):
-  row2_space1, row2_1, row2_space2, row2_2, row2_space3 = st.columns((0.1, 1, 0.1, 1, 0.1))
-  with row2_1:
-    fig, ax = plt.subplots()
-
-    list_A = full_df.loc[full_df['Predicted_A'] == 1, 'Candidate ID'].astype(int)
-    list_B = full_df.loc[full_df['Predicted_B'] == 1, 'Candidate ID'].astype(int)
-    set1 = set(list_A)
-    set2 = set(list_B)
-
-    venn2([set1, set2], ('Model A', 'Model B'), ax=ax)
-    st.pyplot(fig)
-
-  with row2_2:
-    st.markdown('''This Venn Diagram visualizes the number of candidates chosen by both models. It's likely that some candidates will be selected by both models, while others may be chosen by only one model. If we consider Model A as the decision of one hiring manager and Model B as another's, it's easy to see how the selection outcome varies depending on the decision-maker. Some candidates may get the opportunity to be hired, while others might not. This serves as an illustration of the inherent arbitrariness in defining the target variable when dealing with highly subjective outcomes.''')
-    st.markdown('''For instance, it's straightforward to define a target variable in a classification problem like distinguishing dragonflies from butterflies, where there's little room for ambiguity. However, defining what makes a 'good' employee is far more challenging due to its subjective nature.''')
-    
-
-
-def model_vis(full_df):
-  st.markdown('''In this section, you can visualize the demographics of the different subgroups of the data. Firstly, you can see the demographic characteristics of the candidates who have positive labels ("1") and negative labels ("0") which were assigned based on the scores calculated from the slider values you selected previously. Then, you can visualize the demographic distributions of the data which was used for training and evaluation of the models.''')
-  choice = st.radio("**Select desired data:**", ("Positive and negative labels", "Training and evaluation data"), horizontal=True)
-  if choice == "Positive and negative labels":
-    # Create a selectbox to choose a protected characteristic to explore
-    selectbox_Lab = st.selectbox('Label to visualize', ('positive labels', 'negative labels'))
-
-    # Create a selectbox to choose a protected characteristic to explore
-    selectbox_Char = st.selectbox('Protected characteristic', characteristic_dict.keys())
-
-    row2_space1, row2_1, row2_space2, row2_2, row2_space3 = st.columns((0.1, 3, 0.1, 3, 0.1))
-
-    with row2_1:
-      st.subheader("Dataset A")
-
-      # Select test data
-      if selectbox_Lab == 'positive labels':
-        data = full_df.loc[full_df['Model_A_label'] == 1]
-      else:
-        data = full_df.loc[full_df['Model_A_label'] == 0]
-      
-      # Use function plot_data to plot selected data
-      plot_data(data, selectbox_Char, characteristic_dict[selectbox_Char])
-
-
-    with row2_2:
-      st.subheader("Dataset B")
-
-      # Select test data
-      if selectbox_Lab == 'positive labels':
-        data = full_df.loc[full_df['Model_B_label'] == 1]
-      else:
-        data = full_df.loc[full_df['Model_B_label'] == 0]
-
-      # Use function plot_data to plot selected data
-      plot_data(data, selectbox_Char, characteristic_dict[selectbox_Char])
-    st.markdown('''You are visualising the demographic composition of those hypothetical employees who were assigned labels "1" or "0" based on your definitions of the target variables. You might see differences in proportions of genders between the two models for the positive labels, as well as a major difference in the age between the positive and negative labels. Visualising the labels in this manner before training the model can help understand and mitigate differences in demographic representation in the modelling outcomes. Likely, if all candidates labelled "1" were in younger age groups, the candidates selected by the model at the deployment stage will also be in younger age groups. Moreover, target variable definition affects the proportional representation. Having defined two target variables, one can choose the dataset and the model which offers more proportional representation.''')
-
-
-  if choice == "Training and evaluation data":
-    # Create a selectbox to choose a protected characteristic to explore
-    selectbox = st.selectbox('Characteristic to explore', characteristic_dict.keys())
-    row1_space1, row1_1, row1_space2, row1_2, row1_space3 = st.columns((0.1, 1, 0.1, 1, 0.1))
-    # Plot training data
-    with row1_1:
-      st.subheader("Training data")
-
-      # Select train data
-      train = full_df.loc[full_df["Predicted_A"] == "train"]
-
-      # Use function plot_data to plot selected data
-      plot_data(train, selectbox, characteristic_dict[selectbox])
-
-    # Plot test data
-
-    with row1_2:
-      st.subheader("Test data")
-
-      # Select test data
-      test = full_df.loc[full_df["Predicted_A"] != "train"]
-
-      # Use function plot_data to plot selected data
-      plot_data(test, selectbox, characteristic_dict[selectbox])
-  
-    st.markdown('''To train a machine learning model, the data has to be split into two different sets. The first set is the training data, which will be used to teach the model the relationships between the input features (11 subtest results) and the corresponding labels ("0" and "1", assigned based on your definitions of target variables and the values you chose for the sliders). The second set is the test data, or evaluation data. It is used to assess the performance of the model. This is the data which is used to plot the confusion matrices and calculate the model metrics which you saw at the bottom of the "Define the target variable" page. This is also the data whose features you can explore in "Modelling outcomes". It is important that the training and testing data are balanced. Here, you can compare the demographic composition of the training and evaluation data. The training and evaluation datasets compositions were the same and contained the same candidates and same features for both models A and B. However, the labels for each dataset were different and based on what you selected in "Define target variable".''')
-
-def filter_for_protected(data):
-  st.markdown('''Sometimes, the overall model metrics can be deceptive when it comes to predicting the results for different groups under consideration. Ideally, for our models, the varying model metrics would be similar across different groups, which would indicate that the overall model performance is reflected in how this model performs for a given group. It is often not the case, and it is likely that you will see that models A and B perform differently when it comes to those metrics. Even the same model can have different metrics for different subgroups.''')
-  model = st.selectbox('Choose which model outputs to assess', pred_dict.keys())
-  test = data.loc[data[pred_dict[model]] != "train"]
-
-  selectbox_Char = st.selectbox('Protected characteristic', characteristic_dict.keys())
-  if selectbox_Char == 'age':
-    bins= [18,20,30,40,50,60,70,80,91]
-    labels = ['18-20','21-30','31-40','41-50','51-60','61-70','71-80','81-90']
-    test['age_bins'] = pd.cut(test['age'], bins=bins, labels=labels, right=False)
-    selectbox_Char = 'age_bins'
-  # which_group = st.selectbox('Which group?', test[selectbox_Char].unique())
-  df = pd.DataFrame({'Measure': ['True Positive Rate', 'True Negative Rate', 'Positive Predictive Value', 'Negative Predictive Value', 'False Positive Rate', 'False Negative Rate', 'False Discovery Rate']})
-  for group in test[selectbox_Char].unique():
-      rslt_df = test[test[selectbox_Char] == group] 
-      y_true = [int(numeric_string) for numeric_string in rslt_df[model_dict[model]]]
-      y_pred = [int(numeric_string) for numeric_string in rslt_df[pred_dict[model]]]
-      cm = confusion_matrix(y_true, y_pred)
-      if cm.shape == (1,1):
-          cm = np.array([[cm[0, 0], 0], [0, 0]])
-      d = plot_conf_rates(cm)
-      df[f"{group}"] = d["Score"]
-
-  fig = go.Figure()
-  for group in test[selectbox_Char].unique():
-      fig.add_trace(go.Bar(
-          x=df["Measure"],
-          y=df[group],  
-          name=group
-      ))
-      
-  fig.update_layout(
-          title='Model metrics per group',
-          xaxis_tickfont_size=14,
-          xaxis=dict(
-              title='Metric'),
-          yaxis=dict(
-              title='Score',
-              titlefont_size=16,
-              tickfont_size=14,
-          ),
-          legend=dict(
-              bgcolor='rgba(255, 255, 255, 0)',
-              bordercolor='rgba(255, 255, 255, 0)'
-          ),
-    barmode='group',
-    bargap=0.15, # gap between bars of adjacent location coordinates.
-    bargroupgap=0.1 # gap between bars of the same location coordinate.
-      )
-  st.plotly_chart(fig, use_container_width = True)
-  if st.checkbox("Show table of scores"):
-    # CSS to inject contained in a string
-    hide_table_row_index = """
-    <style>
-    thead tr th:first-child {display:none}
-    tbody th {display:none}
-    </style> """
-
-    # Inject CSS with Markdown
-    st.markdown(hide_table_row_index, unsafe_allow_html=True)
-    st.markdown(f"Accuracy rates for {selectbox_Char}")
-    st.table(df)
-
-def data_plot(key1, key2, key3, key4):
-  st.title('''Visualize the Results''')
-  if key1 not in st.session_state:
-    st.error('Cannot train the models if you do not define the target variables. Go to "Define Target Variables"!', icon="🚨")
-  else:
-    tab1, tab2 = st.tabs(["Demographic", "Non-demographic"])
-    with tab1:
-      dataframe = st.session_state[key1]
-      clean_data = st.session_state[key2]
-      st.subheader('''**Select what to explore:**''')
-      data_choice = st.radio('''What to explore''', ("Modelling outcomes", "Input data"), horizontal = True, label_visibility = "collapsed")
-      if data_choice == "Modelling outcomes":
-          st.subheader('''Demographics of the overall modelling outcomes''')
-          model_out(dataframe)
-          st.subheader('''Demographics of the selected protected groups''')
-          filter_for_protected(dataframe)
-      else:
-          st.subheader('''Demographics of the input scores''')
-          model_scores(dataframe)
-          st.subheader('''Demographics of the input labels''')
-          model_vis(dataframe)
-    with tab2:
-      dataframe = st.session_state[key1]
-      clean_data = st.session_state[key2]
-      cmA = st.session_state[key3]
-      cmB = st.session_state[key4]
-      st.subheader('''**Select what to explore:**''')
-      data_choice = st.radio('''Select what to explore:''', ("Modelling outcomes", "Input data"), horizontal = True, label_visibility = "collapsed")
-      if data_choice == "Modelling outcomes":
-          st.subheader('''Labelled dataframe''')
-          dataframe_out(dataframe)
-          st.subheader('''Venn Diagram''')
-          venn_diagram(dataframe)
-          st.subheader('''Model accuracy metrics''')
-          mod_prop(cmA, cmB)
-      else:
-          st.subheader('''Principal Component Analysis''')
-          PCA_general(dataframe, clean_data)
-    
-data_plot('complete_df', 'clean_df', 'cm_A', 'cm_B')

pages/.ipynb_checkpoints/3_💡_Put_the_Idea_into_Practice-checkpoint.py

@@ -1,47 +0,0 @@
-#!/usr/bin/env python3
-# -*- coding: utf-8 -*-
-"""
-Created on Thu May 25 15:42:38 2023
-
-@author: daliagala
-"""
-
-### IMPORT LIBRARIES ###
-import streamlit as st
-
-### PAGE CONFIG ###
-st.set_page_config(page_title='EquiVar', page_icon=':robot_face:', layout='wide')
-
-### IDEA IN PRACTICE PAGE ###
-
-st.title("Different target variable definitions in practice")
-st.markdown('''This dashboard is designed to help you understand how the notion of a “good employee” is translated into machine learning models—by defining target variables—and how target variable definition affects fairness in hiring algorithms (and other aspects of such algorithms too). On this page, we describe how to put the dashboard, and the insights it affords, into practice.''')
-
-st.markdown('''How can this be done? The first step is to make two changes to the dashboard, because you cannot simply take it “off the shelf” and immediately put it into practice. This is because:''')
-
-st.markdown('''- The dashboard is not built using your data or your models.''')
-
-st.markdown('''- How you define a target variable in the dashboard is not how it’s done in practice. (Rather, the way you define it in the dashboard is a straightforward way for you to see the effects of target variable definition.)''')
-st.markdown('''Below we describe how to address these two issues.''')
-
-st.subheader('''Using your data and your models''')
-
-st.markdown('''The dashboard offers a starting point: if you want to build something like the dashboard for your data and your models, you now have a blue-print to work from.''')
-
-st.subheader('''Defining the target variable in practice''')
-
-st.markdown('''In the dashboard, you define the target variable by assigning weights to different cognitive characteristics. These weights determine the “positive label:” people in our dataset who perform best on the tests, given the weights you assign, are those assigned the positive label—that is, those that the model treats as “good.” Then, the model is trained to identify people whose cognitive characteristics match those with the positive label.''')
-
-st.markdown('''As we discussed on the Home page, a growing number of hiring algorithms use cognitive tests to identify promising job applicants. However, the way target variable definition works with these real-world algorithms is different from how it works in the dashboard. For example, consider Pymetrics, a leading developer of hiring software. In some cases, Pymetrics builds bespoke algorithms for a company that is hiring for a given role. Pymetrics will ask the company to identify a group of the client’s current employees in that role that the client considers “good.” Then, these “good” employees play cognitive test games similar to the ones used in our dashboard. It is these employees who are assigned the positive labels. From this point on, Pymetrics’ algorithmic development goes just as it does in our dashboard: a model is trained to identify job applicants whose cognitive characteristics are similar to those with the positive label.''')
-
-st.markdown('''So, for hiring algorithms like Pymetrics’, the target variable is defined not by assigning weights to cognitive attributes, but rather by directly identifying a certain group of current employees as “good.” In the dashboard, you can define different target variables by assigning different weights to the cognitive attributes. If you are in practice building an algorithm like Pymetrics’, you can define different target variables by identifying different groups of current employees as “good.”''')
-
-st.markdown('''How might this work? As we discussed on the Home page of the dashboard, reasonable minds may disagree about what makes for a good employee, and relatedly reasonable minds may disagree about which current employees are good employees. For example, within a company, two different managers—call them Manager A and Manager B—may not be perfectly aligned in who they consider to be good employees for a certain role. The managers may agree in some cases. We might imagine that there are 50 employees whom both Manager A and Manager B deem good. But the two managers might disagree about other employees. Imagine that there are 25 further employees whom Manager A thinks of as good but Manager B does not (this needn’t mean that Manager B thinks that these employees are bad, just that they are not the best). Likewise, there might be 25 further employees whom Manager B thinks of as good but Manager A does not.''')
-
-st.markdown('''In this case, there are two different (overlapping) groups of 75 employees, each corresponding to what Managers A and B think of as good employees. These two different groups of employees—and in turn, two different target variable definitions—could be used to train two different models.''')
-
-st.markdown('''Instead of constructing two groups of “good” employees directly from the judgments of Managers A and B, you could weight their judgments against one another. For example, you could have two groups of employees, X and Y. Both X and Y contain the 50 employees that Managers A and B agree on. But group X contains 20 of Manager A’s preferred employees and 5 of Manager B’s, while group Y contains 20 of Manager B’s preferred employees and 5 of Manager A’s. Here again we have different groups of “good” employees, and so two different target variables.''')
-
-st.markdown('''One could select different groups of good employees in other ways still. An employer might have different metrics to evaluate employee success. Different employees might be better than others according to one metric compared to another. Depending on what importance is assigned to the different metrics—depending on how you weight the different metrics against one another—different groups of employees may emerge as “good.”''')
-
-st.markdown('''Our focus in the dashboard has been on hiring algorithms that are based on cognitive test games. There are other kinds of algorithms used in hiring—for example, algorithms that identify promising job applicants on the basis of their resumés. In designing any such algorithm, the target variable must be defined, and the notion of a “good employee” must be translated into algorithmic terms. And so the insights of this dashboard apply, and can be put into practice, for almost any kind of hiring algorithm you’re working with.''')