Explorative study on verbalizing students’ skills with NLP / AI -tool in Digital Living Lab at Laurea UAS , F inland (2023)

Abstract: This explorative study tested Laurea UAS students’ (N=16) abilities to verbalize their skills, before and after the study unit “Digital Analytics and Consumer Insights”. Before the study unit the students listed their skills unaided and afterwards with help of Careerbot AI -service. The findings indicate that the intervention increased both quantity and quality of the skills verbalized, relevant to the learning objectives and generic, 21st century skills.

## introduction the purpose of this explorative study was to research if the students can verbalize their skills and competences better with the help of careerbot ai -service than without it. laurea university of applied sciences in helsinki region in finland has a learning environment called digital living lab (dll), focusing on real-life project-based studies with partner organisations. the dll aims to support the acquisition of "21 st century skills", working life skills focusing on digital service development. in august 2023, five day design sprint for study unit "digital analytics and consumer insights" (daci) was executed in english. this study unit is part of elective studies. the author was the responsible lecturer and the head facilitator during the whole hybrid event. six other facilitators and subject-matter experts supported partially. learning objectives for this 5-credit point (ects) study unit were the following: "after the study unit, the student is able to: ## sample the participating students (@xmath0. 9 of them participated face-to face, and 7 online. study fields were business management (@xmath1, business information technology (@xmath2, hospitality management (@xmath3. the age groups were in the following categories: 25 or less (@xmath4, 25-34 (@xmath5, 35-44 (@xmath6. ## related work the use of artificial intelligence (ai) tools in helping higher education students to verbalize their skills and competences in a job market language (as defined in job ads) has not been researched much yet. @xcite conceptualized "forecasted self", future-oriented digital twin, where a student can explore several future selves equipped with new, acquired skills for projected future jobs with careerbot ai -service. @xcite discussed the prospects for career coaching, with four ai maturity levels in career guidance: 1. aiaware guidance, 2. ai-informed guidance, 3. aiintegrated guidance and 4. ai-transformed guidance. transversal and transferable skills and competences were defined in transval-eu project (2021) as generic working life skills, soft skills, and employability skills. brown and souto-otero (2020) analyzed 21 million job ads in the uk and found that employers are most likely to focus applicants "job readiness", demonstrating both generic, soft skills and technical requirements. @xcite talked about potential job candidates' fit with organization through "narrative of employability" boosted by non-workrelated skills acquirement ct. qualifications. @xcite defined ict literacy above the mastery of ict applications, to include "higherorder thinking processes", like problem solving of information, communication, and knowledge tasks in an ict context, relevant to learning context in the knowledge society. ## technology and data the tool used for students on verbalizing their skills is called "careerbot". this webservice interface has been developed "for helping 34 000 students to pursue their dream careers with the help of ai" by 3amk. 3amk is a strategic alliance of laurea, haaga-helia, and metropolia universities of applied sciences in helsinki region, finland. (3amk.fi, 2023) the ai behind this webservice is called "graphmind" and built by finnish tech company headai ltd. graphmind is a graph machine learning -based semantic computing framework accessible via rest-api for careerbot -service. @xcite . in the first phase graphmind has been taught with unstructured data (millions of news) and e.g., european skills, competences, qualifications and occupations (esco) classification, and in the second phase with reinforced learning. @xcite the main data source for careerbot is job market data in finland (työmarkkinatori, mol and duunitori/employment services) with over 400 000 job ads on a yearly basis since january 2018. the other data sources are 3amk course data, theseus -theses data from finland and global directory of open access journals (doaj) but they were not used for this study. in the careerbot ai -service the students can create their skills profile, a digital twin for skills, "forecasted self". the skills are defined pragmatically, as the words are stated in the job ads. the user experience flow in careerbot aiservice works in the following way: a) login to the system, b) create a new skills profile, c) start typing in the skills words individually (and get suggested related skills words), optionally d) copy-paste personal cv to text field from which the skills words are retrieved, e) select relevant skills word from suggested skills word list, f) look for jobs in finland based on area, time and your skills profile (selected skills words), and g) further update your skills profile from selected job (finnish job ads "soft-and hard skills" that were still missing from one's skills profile). there were some bugs encountered during the session with students. however, every student managed to perform the given assignment on verbalizing their own skills with the help of careerbot ai -service. ## methods during the sprint week, the students were assigned into five teams of 3-4 people randomly, 2 teams online and 3 face-to-face in the digital living lab. teams could choose their team assignments freely, along the learning objectives. strictly speaking this was a quasi-experiment, since 3 students were allowed to change their assigned study method (face-to-face or online) to another one for personal reasons, after ethical consideration. this sub-study focused on students' ability for skills verbalization, not on the skills development per se. the other sub-study will be reporting the findings for perceptions on the course. the briefing for the students is stated below: "…please answer honestly to this questionnaire, how do you feel about the claims right now. these are your personal views, there are no right or wrong answers, and these answers will not affect your study unit grading…" in the pre-questionnaire, the students were given only written instructions via email and online questionnaire with open text field for answer. before the post-questionnaire, on the last day of study unit, the students were introduced to usage of careerbot ai -service for 10 minutes. the students were instructed on how to create a new skills profile, look for jobs based on it, and further educate their skills profiles based on missing skills found on the job ads. the students used 30 minutes for the assignment and filled in their skills words to post-questionnaire with open text field. ## results the results of verbalizing student's skills with the pre vs. post experiment setup indicate a clear increase of quantity and quality of the skills words for every participated student (@xmath7. the quality of the verbalized skills was evaluated based on the relevance to study unit learning objectives and 21 st century skills by the author. from the students open text field -answers the skills words were manually extracted and copied as a list to https://www.wordclouds.com/ from which the frequencies were extracted, synonyms combined and visualized. after the study unit, with the help of careerbot ai -service, the students listed their skills and their frequency as following (top 20): ## discussion the purpose of this exploration was to research if the students can verbalize their skills better with help of careerbot ai -service than without it. as a conclusion, the quantity verbalized skills increased for all the participated students, on average from 3,56 to 13,94 (median from 3,5 to 12,5). the students learned to elaborate more on the topic of the study unit. the quality of the verbalized skills also increased. the mentioned skills can be seen relevant from employment point of view, since they all appear in job ads (=data source). also, more students mentioned keywords relevant to the learning objectives and 21st century skills (generic, soft skills) after the sprint week. evaluating the extent to which the difference was due to teaching method vs. ai is not clear. the earlier sessions using careerbot ai-service with the students have produced results of similar direction, but they have not been documented systematically. both face-to-face and online groups received the same introduction and guidance for the ai service use simultaneously. face-to-face group improved slightly more during the week (3,89 12,3 vs. online 3,1415,2 skills). sample size is too small for conclusions on the difference. based on the earlier feedback of industry partners of laurea uas in autumn 2022, the graduates have had sometimes difficulties in verbalizing and therefore "selling" their skills to the recruiters. for this, the use of ai in addition to teaching and coaching can be useful. this is in line with @xcite who highlighted "narrative of employability" for job candidates. this experiment was using ai as a personal tool, reaching at most the level 2/4 "ai-informed guidance" in maturity model, @xcite . so, there is still a gap for reaching the higher levels in ai maturity in coaching. however, to confirm these initial findings, more studies are needed with larger sample sizes. also, more studies are needed with other tools in comparison, including traditional human guided career coaching and other tools, like latest versions of generative ai (chatgpt, etc.).