Doing not Being: Concrete Language as a Bridge from Language Technology to Ethnically Inclusive Job Ads (2022)

Abstract: This paper makes the case for studying concreteness in language as a bridge that will allow language technology to support the understanding and improvement of ethnic inclusivity in job advertisements. We propose an annotation scheme that guides the assignment of sentences in job ads to classes that reflect concrete actions, i.e., what the employer needs people to do , and abstract dispositions, i.e., who the employer expects people to be . Using an annotated dataset of Dutch-language job ads, we demonstrate that machine learning technology is effectively able to distinguish these classes.

## introduction ethnic minorities are disadvantaged in the employment market @xcite , despite laws that protect them. if people read a job advertisement, and get the sense that the employer will not consider their applications fairly, they will not apply @xcite . this chilling effect can compound already existing employment disadvantages. for this reason, it is important to create welcoming and inclusive job ads. this paper is motivated by the idea that language technology has potential to help identify job ads that are not inclusive and to suggest changes to make them more welcoming. a conventional machine learning approach would ask human annotators to label a large number of job ads as 'inclusive' and 'not inclusive' and train a classifier. however, ethnic minorities themselves must make the final judgement of the difference between welcoming and unwelcoming ads. given the burden already borne by these groups, we argue that laborious labeling work should be avoided and a higher-level approach to understanding inclusion in job ads is desirable. in this paper, we aim to build a bridge between language technology and inclusive job ads by investigating basic semantic characteristics of predicates. specifically, we identify concrete vs. abstract language to be important. in the context of job ads, this distinction translates into the difference between what the employer needs a candidate to do on the job and who the employer wants the candidate to be in terms of their personal traits. our study is inspired by work on stereotypes in job ads by @xcite who found a difference between behavioral statements, e.g., 'you are expected to keep confidential information to yourself', which are concrete and describe the job, and dispositional statements that express the same requirement abstractly, e.g., 'you are reliable'. dispositional statements could be interpreted as a personal judgement that reflects a stereotype that ethnic minorities must frequently face and @xcite found that they discouraged ethnic minority job applicants from applying. we make the case that language technology that could detect the difference between concrete 'doing' and abstract 'being' would make an important contribution to ethnically inclusive job ads. our work makes the following contributions: this paper summarizes the most important findings of a larger study of ethnic discrimination in dutch job advertisements by @xcite . we also release an annotated dataset as a resource for the research community. foot_0 in this section, we provide information on the psychological literature that connects inclusivity with language concreteness and discuss previous work on discrimination detection in job ads. ## method we first discuss the annotation scheme that converts the class scheme of the lcm to the job advertisement domain and how we applied this to manually label a sample of job advertisement phrases. then, we describe a supervised machine learning approach on a small set of job advertisement phrases in order to demonstrate that the distinction between dispositional and behavioral phrasing can be automatically detected consistently and accurately as a proof of concept of the applicability of language concreteness estimation in job ads. ## dispositional/behaviorial detection we took a three-step approach to automatically detecting concreteness/abstractness classes. first, the predicates were extracted from the sentences using the rule-based method described in sec. 3.2. second, the extracted predicates were classified by their relevance, and discarded if they were not dispositional or behavioral. third, the relevant predicates were classified by a binary classifier as dispositional/behavioral (left/right of fig. 1 ) and by a multi-class classifier into the sub-classes (bottom classes of fig. 1 ). we evaluated four classifiers: tf-idf + naïve bayes tf-idf weighed feature vectors were extracted from the data and dimensionality reduction was applied. (we used variance thresholding at @xmath0.0005 and the chi-square test to reduce the vector size to 500.) naïve bayes was implemented using scikit-learn @xcite . word2vec + lstm pre-trained dutch word embeddings (320-dimensional) from @xcite were used as input to an lstm, using python an keras tensorflow. bert 'bertje' (de vries et al., 2019), a dutch, pre-trained, transformer-based bert model, was fine-tuned using python and keras tensorflow. a dropout layer was added for regularization. roberta 'robbert' @xcite ) was fine-tuned in similar fashion. we also experimented with a (one-step) token classification approach, similar to ner. this resulted in incorrect and spurious predicate detection and was not explored further here. ## results tab. 1 presents results that confirm the ability of a machine learning approach to distinguish dispositional and behavioral predicates. the neural models (word2vec + lstm, bert and roberta) outperform naïve bayes and the transformer-based models give the best over-all performance. fig. 3 presents the confusion matrix of the subclasses, which yields the following insights: error severity recall that the sub-classes (except 'learned quality') are placed along a continuum from concrete to abstract. fig. 3 shows that the incorrectly predicted labels are often close to the ground truth label on this continuum. class confusion 'process' is confused most often with 'act'. during the annotation pilots, it was already observed that it is hard to judge the edge cases between these classes. for example, take the predicate taking care of the project documentation. it is not clear-cut to which class this example belongs. the class 'attitude' is confused with 'attitude + action' or 'innate quality'. phrases of these types are often syntactically similar. the class 'learned quality' shows the least confusion. this observation is not surprising because 'learned quality' is the majority class in the data and is most easily identifiable by specific frequently occuring nouns (e.g., names of certificates, education levels, language skills, or words like ervaring english: 'experience' or kennis english: 'knowledge'). ## conclusion and outlook in this paper, we have proposed that language concreteness is useful as a bridge between language technology and ethnic inclusivity in job ads. the connection between inclusivity and concrete language is supported by research that has shown that focusing on doing rather than being can prevent ethnic minorities from being put off by job ads that they are qualified to apply for. it is also supported by the psychology literature on social distance and social power distance. we presented an annotation scheme that supports stable annotation of classes along a continuum that runs from abstract (dispositional) to concrete (behavioral) and have used it to annotate a dataset of dutch-language job ads. the dataset has allowed us to demonstrate that machine learning classifiers can reliably detect differences in language concreteness. we intend our work to be useful to machine learning researchers, who can apply our annotation scheme and reproduce our experiments for different datasets and languages, but especially to social psychologists, as they continue to investigate ethnic inclusivity in the employment market. it is important to note the difference between our work and other work that has been carried out on ethnic bias in nlp models, e.g., @xcite and @xcite . the concern of these studies is stereotypes that are expressed about members of ethnic minorities. in other words, they focus on the context in which ethnic minorities are mentioned and/or what is said about them. in contrast, our work studies textual phrasing that could trigger members of ethnic minorities to be concerned that the writer may hold stereotypes against them. this contrast is important because whether or not a job ad is perceived as inclusive goes far beyond direct mentions of ethnic minorities. we hope that our work is useful to extend the understanding of how ethnic inclusivity can be promoted in society, and how nlp can contribute to this goal.