A machine-learning based model to identify P h D -level skills in job ads (2020)

Abstract: Around 60% of doctoral graduates worldwide ended up working in industry rather than academia. There have been calls to more closely align the PhD curriculum with the needs of industry, but an evidence base is lacking to inform these changes. We need to find better ways to understand what industry employers really want from doctoral graduates. One good source of data is job advertisements where employers provide a ‘wish list’ of skills and expertise. In this paper, a machine learning-natural language processing (ML-NLP) based approach was used to explore and extract skill requirements from research intensive job advertisements, suitable for PhD graduates. The model developed for detecting skill requirements in job ads was driven by SVM. The experiment results showed that ML-NLP approach had the potential to replicate manual efforts in understanding job requirements of PhD graduates. Our model offers a new perspective to look at PhD-level job skill requirements.

## introduction abundant evidence shows that industry employers are often dissatisfied with key aspects of phd graduates' workplace performance (e.g., @xcite @xcite , particularly in relation to professional skills like communications @xcite . phd graduates themselves also indicate the education they received during their candidature did not address job market needs outside of academia @xcite . reports indicate that employers were often dissatisfied with doctoral employees' demonstration of soft skills at work (e.g., @xcite . such frustration from stakeholders leads us to question the fitness for purpose of doctoral degrees. phd was originally designed to help people into academic careers, but its fitness for purpose has been questioned for over 80 years @xcite . as the number of phd graduates increase tremendously @xcite and the academic job market remains relatively static in scale @xcite , many phd graduates will be unable to secure academic positions. despite the fact that most phd graduates will be working outside academia, universities are still training their candidates based on research competencies desired in academia. one such example of a popular research training framework is the vitae researcher development framework (rdf) based on research by @xcite . to enhance phd employability, @xcite argue it is not enough to understand only academic workforce requirements: non-academic professions may have different needs. some initiatives have been undertaken to understand the so called 'transferrable skills' needed from phd job seekers. consequently, many add-on courses based on the long list of skill terms have been put in place at universities @xcite . however, scholars such as @xcite and @xcite have expressed concerns about the generic quality of these initiatives. take professional skills such as teamwork and empathy: these may mean very different things in different workplaces and industry domains. concerns are reasonable about the ambiguity of previous initiatives suspicious of overgeneralisation of skills and neglect of the context in which skills are deployed. however, little has been done so far to empirically test the difference that context makes. we can ask employers to tell us, but relying on retrospective self-reports has an inherent problem of informant inaccuracy @xcite . other methods such as ethnography are too difficult to scale up considering the large number of industry fields. therefore, a machine learning / natural language (ml/nlp) approach is worthy of exploration. to address the gap described above, we developed a machine-learning-based model to identify employers' expectations of qualified job seekers in job adverts. we first manually labelled 400 job ads across two industries based on move-step analysis, an analytic approach widely adopted by applied linguistics researchers to detect contextual difference in written language discourse @xcite . moves are coarse-grained categories, and steps are finegrained categories associated with a particular move. finer granularity is essential to avoiding abstractness and ambiguity in language use. hence, a data retrieval interface with fine-and coarse-grained information can provide more accurate results than an interface without differentiation in information granularity @xcite . after the manual annotation was done, the labelled data were fed into the machine using the svm algorithm. after the parameters were tuned, the majority of the identified skill categories in the model reached good area under the curve (auc) performance. although the model remains to be optimised for certain skill categories, the results showed that natural language processing of job advertisements has the potential to replicate human efforts to provide rich insights into how phd education can be improved. this paper seeks to contribute to the literature in three key ways: our workshop experience can be valuable for future attempts to use natural language processing to inform higher education policy making. 2. curriculum design tools: the coarse and fine granularity in our annotation generates more accurate definitions of skill items, which in turn enables better curriculum design in phd education. 3. evidence: our model can serve as a useful approach to testify the hypothesis that contextual difference exists across industry domains. in addition, it gives evidence of automation being a feasible approach to contribute to human efforts in understanding phd-level skill requirements in job advertisements by accelerating abilities to analyse text systematically at scale. ## background and related work job ads contain rich information regarding employers' expectation of qualified job seekers @xcite ), yet research on the automated identification of skills from job ads is still in its infancy. most previous studies on skills in job ads were manual content analysis (see for example, @xcite . the majority of studies on job ads only examined a single industry domain. studies with manual efforts examined professions such as librarian positions @xcite and accounting @xcite . nevertheless, studies relying on machine learning techniques seem to exclusively examine computing related job positions (e.g., @xcite @xcite . in recent years, scholars have started analysing and extracting skills in job ads with machine learning techniques. the effort to automatically retrieve skill requirements from job ads should enable a content analysis approach to job ads to be expanded to other industry domains. although there are several ml-based studies on job ads, most authors only took technical skills into consideration. for example, @xcite wingkvist (2014), khaouja et al. (2018), rahhal et al. (2019) identified technical skills such as programming languages but ignored non-technical skills such as teamwork, communication skills, user engagement, workplace aesthetics, ethics, etc. obviously, nontechnical skills in job ads remain underexplored by data mining researchers. in order to have a holistic understanding of employers' expectations, we need to examine both technical and non-technical skills listed in job ads. ## results and discussion overall, 12 skills at move level and their associated steps were identified from our manual analysis (see ## conclusion in this study, we developed a model to automatically identify some phd-level skill requirements in job ads. the move-step analysis we adopted in our manual annotation procedure allows for the model to pinpoint both coarse-grained skill items and their associated contextual interpretation. the moves and steps we identified can be used in curriculum design to enhance phd candidates' awareness of skill requirements in different industry domains. our human coder's workshop experience can be useful for scholars who also intend to conduct ml driven analysis of textual data for enabling better decision making in higher education. in addition, the ability of our model to quantify skills provides evidence that contextual difference exists in the number of skills required of qualified phd job seekers. our finding challenges the problematic view that we can set aside contextual factors in phd training. our model has limitations. in this study, we pinpointed areas for further optimizing the model. before the model can be used to automatically identify all the skill categories, the overfitting and underfitting problems need to be solved. additionally, we only experimented with limited number of data in healthcare and computing domains. therefore, extra effort to manually label more data in these two and other industries is necessary before the model can be used to identify skills across different industry contexts.