Course Recommender Systems Need to Consider the Job Market

Abstract.

Current course recommender systems primarily leverage learner-course interactions, course content, learner preferences, and supplementary course details like instructor, institution, ratings, and reviews, to make their recommendation. However, these systems often overlook a critical aspect: the evolving skill demand of the job market. This paper focuses on the perspective of academic researchers, working in collaboration with the industry, aiming to develop a course recommender system that incorporates job market skill demands. In light of the job market’s rapid changes and the current state of research in course recommender systems, we outline essential properties for course recommender systems to address these demands effectively, including explainable, sequential, unsupervised, and aligned with the job market and user’s goals. Our discussion extends to the challenges and research questions this objective entails, including unsupervised skill extraction from job listings, course descriptions, and resumes, as well as predicting recommendations that align with learner objectives and the job market and designing metrics to evaluate this alignment. Furthermore, we introduce an initial system that addresses some existing limitations of course recommender systems using large Language Models (LLMs) for skill extraction and Reinforcement Learning (RL) for alignment with the job market. We provide empirical results using open-source data to demonstrate its effectiveness.

1. Introduction

The contemporary job market is dynamic and rapidly evolving (Deming and Noray, 2020), necessitating continuous adaptation of individual skill sets to maintain relevance and competitiveness. This evolution introduces a unique challenge: guiding learners in selecting educational courses that enhance their expertise and align with their career objectives and with job market demands. However, a notable mismatch exists between the skills learners possess, the skills taught, and those in demand in the job market (Palmer, 2017). This mismatch can be explained by various factors such as the lag between demand in the market and the adaptation from course providers, unequal access to training, or the lack of information from training providers on the type of skills needed in the job market (Palmer, 2017). This issue significantly limits the employability and career progression of individuals, impacting both their personal development and, at a more systemic level, economies dependent on a skilled workforce.

However, existing course recommender systems often focus solely on learner-course dynamics (Sanguino Perez et al., 2022; Sakboonyarat and Tantatsanawong, 2019), neglecting the crucial aspect of aligning course recommendations with real-time job market trends. This limitation leads to a mismatch between the skills acquired through recommended courses and those actually in demand in the workforce. While one study uses job postings and resumes in a course recommender system (Bothmer and Schlippe, 2023), their approach does not directly consider market trends or users’ goals, making this domain largely unexplored.

In this paper, we present the perspective of academic researchers working in collaboration with industry practitioners aiming to develop and deploy job-market-oriented course recommender systems. We argue that rethinking course recommender systems to consider the job market has the potential for significant economic and societal impact. We outline the properties these systems must have: (P1) aligned with the latest job market trends to prioritize courses teaching high-demand skills; (P2) unsupervised to avoid the resource-intensive process of collecting and annotating up-to-date data; (P3) sequential to recommend a sequence of courses where each course builds upon the knowledge acquired in the preceding ones; (P4) aligned with users’ goals such as attaining a specific role or increase their marketability; (P5) explainable to ensure user trust and engagement. We also highlight research directions and areas for future research to address the challenges in this field: (RD1) Addressing the scarcity of course recommendation datasets by creating or providing datasets to the community for training and evaluation; (RD2) Designing evaluation metrics to take into account alignment with the job market when evaluating the recommendations. (RD3) Estimating user’s goal progress based on their profile and the job market to tailor the recommendations to their needs; (RD4) Develo** Skill-based explainable models and visualization techniques; (RD5) Develo** Unsupervised Skill Matching models to estimate up-to-date skill demand on the job market; (RD6) Unsupervised Taxonomy Construction to include new emerging skills without human labeling.

In this work, we also develop a skill extraction and matching (SEM) method to identify skills and proficiency levels from learners’ resumes, course content, and job descriptions. We also develop an unsupervised, sequential skill-based Job-Market-Oriented Course Recommender system (JCRec) that uses the skills extracted by SEM to determine a candidate’s course options and to estimate the number of job opportunities available to them. JCRec then uses Reinforcement Learning (RL) to recommend a sequence of courses that maximizes the number of job opportunities available to the user. Our system meets all of the properties we previously outlined and lays out further steps for the research directions we’ve mentioned.

The key contributions of our paper include:

• Identification of the desirable properties a job-market-oriented course recommender system should have.

• Identification of the challenges that develo** such systems will pose along with research direction for the community to address these challenges.

• A few-shot skill extraction method to find skills from resumes, job postings, and course descriptions.

• A first job-market-oriented course recommender system.

2. Perspective

This paper presents the perspective of academic researchers working in collaboration with industry practitioners aiming to develop and deploy course recommender systems for a multilingual user base. This work is motivated by the fact that academic work about course recommender systems seldom considers the job market reality (see section 3). This disconnection is reflected in the features used to perform the recommendation, but also the lack of consideration of the objective behind the users taking courses. In this work, following extensive collaboration between academic researchers and industries from the up-skilling and continuing education domain, we propose methods to devise recommender systems that align with users’ pre-existing skills, career objectives, and the current demands of the job market. Thus, we focus on the challenging and realistic situation of the career development of users, where users have experience and seek to obtain new skills, often to find a new position. With this work, we aim to motivate the importance of this problem, propose research directions for the Information Retrieval and Recommendation communities, and provide a prototype of a job-market-oriented course recommender system.

3. Course Recommender Systems

Course recommender systems have been extensively studied, focusing on various aspects such as learning activities recommendation through open learner models (OLMs) (Abdi et al., 2020), recommendation incorporating users’ skills (Sankhe et al., 2020; Piao and Breslin, 2016), peer learner recommendation (Potts et al., 2018) or target course-oriented recommendation (Jiang et al., 2019). In the recent domain of course recommender systems using neural networks (NN), multiple research directions have been pursued. These include optimizing the accuracy of recommendation (Sanguino Perez et al., 2022; Zhu et al., 2022), ensuring fairness (Marras et al., 2021; Khalid et al., 2021), and improving explainability (Shareef et al., 2022; Tian and Liu, 2021; Frej et al., 2023). The majority of these approaches use a combination of learner-course interactions, course content, learner preferences, and additional course information such as teachers, schools, course ratings, or comments, usually in the form of Knowledge Graphs (KG). While we acknowledge the significance of incorporating learner and course data, we consider that an effective course recommender system must incorporate the job market’s current demands, and avoid recommending courses that teach skills lacking demand on the job market.

To our knowledge, Skill scanner (Bothmer and Schlippe, 2023) is the only work that uses resumes, course descriptions, and job descriptions for skill-based course recommendations. This system’s pipeline involves extracting, vectorizing (using word embeggins techniques such as Word2vec (Mikolov et al., 2013) and Glove (Pennington et al., 2014)), clustering, and matching skill sets. Skill scanner can be used to compare courses, learners, and job postings. These comparisons serve not only for skill-based course recommendations but also to inform job seekers and educational institutions about the market relevance of specific skills, enabling them to adapt accordingly. However since their approach relies solely on encoding skill sets in a common representation space, it does not directly consider the job market trends, skill demand distribution, or the user’s goal. Hence, Skill scanner is a first step in the direction of job market-guided course recommendations but most of the work remains to be done.

4. Rethinking Course Recommender Systems to Consider the Job Market

In this section, we first list the properties that job-market-oriented course recommender systems should have and we propose several Research Directions to address the issues we identified to develop such systems. We voluntarily omitted some properties that all recommender systems should satisfy as they are not specific to our case, such as personalized and real-time recommendations.

4.1. Properties

P1: Job Market Alignment. The recommender system must consider the skill demand in the job market when making recommendations. On the job market, skills differ by their popularity. Learning a new skill, depending on whether that skill is in high or low demand from employers, or if it is rare or frequent among other job applicants, will have a very different impact on the learner’s marketability. Thus, when for example comparing similar courses, the recommender system should give preference to the one teaching the skill with greater market demand. This ensures that learners are equipped with the most relevant and sought-after skills to increase their chances of finding a position.
P2: Minimal Supervision. The recommender system should rely on a limited amount of labeled data because it needs to accommodate the rapid evolution of the job market with new skills appearing regularly. However, existing recommenders usually rely on supervised models that would have to be updated regularly. To avoid the high cost of labeling data manually, most of the components should be based on unsupervised learning techniques. For example, scra** job postings and extracting skills from course descriptions using unsupervised models will allow the system to adapt to market trends without the extensive costs of manual data labeling.
P3: Sequential Recommendations. The system must recommend sequences of courses rather than standalone courses. Indeed, individual goals often require a progression through multiple subjects. Note that the order in which these courses are taken often matters. For instance, a front-end developer aspiring to become an LLM engineer might need courses in Python, machine learning, Natural Language Processing (NLP), and LLMs, with each course building upon the knowledge acquired in the preceding ones.
P4: User’s Goal Alignment. The system must align with the objectives of its users, whether that involves attaining a specific role, learning skills to increase overall profile attractiveness, or specializing in a field. Recommendations should thus be aligned with these goals, ensuring that two users with identical profiles but different objectives receive different course suggestions.
P5: Explainable. The recommender system must provide explainable recommendations. Given the time and resources needed for a user to enroll and finish a course, it is crucial for the system to transparently explain its recommendations – especially sequential recommendations. Explainability is not only essential for building trust but also for ensuring that users feel confident in their decision to invest resources in a recommended course. For example, explaining that a course is suggested because it teaches a skill with current high demand in the job market can significantly enhance user confidence in the system’s guidance.

4.2. Research Directions and Challenges

RD1: Course Recommendation Datasets Beyond consideration of the job market, a significant challenge affecting research in course recommendation is the scarcity of publicly accessible, large-scale datasets for this task. Presently, there are only two of such datasets: Xuetang (Zhang et al., 2019) and COCO (Dessì et al., 2018). Xuetang contains courses from the Massive Open Online Courses (MOOCs) platform XuetangX, primarily in Chinese. COCO contains courses from the MOOCs platform Udemy available in 35 languages. This sparsity can be attributed to universities’ reluctance to share student enrollment data due to privacy and ethical concerns and the desire of online course providers to preserve competitive advantages. In response to these challenges, we urge academics and industry professionals to anonymize and share subsets of their enrollment data with the research community. An alternative method we wish to highlight is the generation of synthetic datasets using generative models. While this method has been proposed in the context of job postings for skill extraction using LLMs (Decorte et al., 2023; Clavié and Soulié, 2023; Magron et al., 2024), its application to creating course recommendation datasets remains unexplored. This presents an innovative research direction with the potential to significantly impact the field. Potential directions could involve generating coherent career paths of individuals using the in-context ability of LLMs, then inferring skills required to switch from one position to the next one using databases of skills associated with job roles. Moreover in this work, instead of course enrollment data, we use only course descriptions along with job postings and resumes. While job postings are regularly explored in the literature (Zhang et al., 2022b, a; Magron et al., 2024), few public datasets exist. User profiles, in the form of resumes, are documents that contain personal information and are usually not public; such databases are even harder to come by than job postings, especially at scale. Overall, these data sources are seldom found in languages other than English. We show that we can use such data to build job-market-guided recommender systems, even though it doesn’t allow for direct evaluation of users’ course enrollment.
RD2: Evaluation Evaluating the effectiveness of a job-market-oriented course recommendation poses significant challenges. Most recommender systems estimate the relevance of a course based on user profiles using ranking metrics (MRR, nDGG, Hit) to compare recommendations to the actual courses taken using a user-course interaction matrix. Other metrics are used to measure Novelty, Fairness, Diversity, and Coverage of the recommendations (Zangerle and Bauer, 2022). However, these approaches are not enough for job-market-oriented course recommendations as they do not take into account market demand and the actual impact on a user’s career trajectory. An ideal evaluation framework would also ascertain whether the recommended course enabled the user to meet their career goals, such as securing a desired job or enhancing their marketability. Because this type of information is rarely directly available, estimating the impact of following a course on a user’s profile and achieving their objective presents a considerable challenge. For these reasons, we believe that designing an evaluation methodology for job-based course recommender systems is both a challenging and impactful research problem. In this work, we propose to evaluate the recommendations by estimating the number of jobs the users can apply to upon completing the recommended courses. Although our approach presents a first step for this research direction, user studies are necessary to evaluate if the system provides meaningful recommendations for users. Another potential research direction involves addressing the following problem: how can historical user enrollment data be leveraged to assess the effectiveness of a recommender system in a dynamic job market where past user choices may become less relevant? For instance, consider a scenario where a user previously enrolled in a PHP course, leading to employment in web development. Recommending this same course in the present day may not hold the same relevance, given shifts in the job market. Using this interaction for evaluation might bias the system toward past trends but ignoring all past interactions is not a satisfying solution since course enrollments include valuable information on user preferences that provides insight into specific flaws of evaluated recommender systems. One possible direction would be to ignore the skills of the courses – which might teach outdated skills– and focus on the user enrollment data, evaluating only the coherence and robustness of recommender systems between users.
RD3: Estimating User’s Goal Progress On top of incorporating the dynamics of the job market in the recommendation, we highlight the importance of taking into account the objectives of learners. We propose an initial strategy involving two steps: 1) identifying the goals learners may hold, and 2) develo** functions to assess how much progress learners have made towards achieving their goals. To tackle this, we identify three primary goals related to the job market that a user might pursue as well as the functions needed to estimate their progress towards their goal:
1) Securing a Specific Position: we need to define a function that can compute the compatibility between a learner’s profile and a job description.
2) Boosting Marketability: we need to define a function to evaluate a learner’s marketability using their profile and a collection of job descriptions.
3) Achieving Specialization in a Field: We also need a function to quantify a learner’s level of specialization based on their profile and the target field.
Defining these functions, along with the constraints and properties they must satisfy (Fang et al., 2004), represents a promising and impactful direction for research. This approach aims to align course recommendations more closely with the realities of the job market and the individual goals of learners, thereby enhancing the relevance of such systems.
RD4: Skill-based Explainability Given the significant time and investment required to enroll in a course, such systems must offer clear explanations. Users need to understand how enrolling will advance them toward their goals. We advocate for a skill-based approach to explainability, highlighting the specific skills a course teaches and how acquiring these skills aligns with the user’s objectives. Possible research directions include: 1) exploring visualization techniques to present the skill-based explanations to the learners and 2) develo** sequential and explainable course recommender systems. To our knowledge, recommender systems that are both sequential and explainable have not been studied, presenting a novel research opportunity for the field of recommender systems.
RD5: Unsupervised Skill Matching Skill matching consists of aligning skills from various sources, such as job descriptions, resumes, and courses, with a skill taxonomy (Gnehm et al., 2022). Because this process is crucial for creating skill-based explanations in job-oriented course recommender systems, we highlight the necessity of unsupervised methods to extract skills from resumes, course descriptions, and job postings. Recent advancements in LLMs have facilitated few-shots skill extraction from job postings through the use of synthetic dataset generation to generate demonstrations (Magron et al., 2024). However, extending this approach to include resumes and course descriptions remains largely unexplored. Possible research directions encompass generating synthetic datasets for resumes and course descriptions to enable unsupervised skill matching. Another direction is to assess skill proficiency levels from resumes, job postings, and course descriptions. Such assessments could give the recommender system’s ability to distinguish between introductory and advanced courses. To our knowledge only one supervised model has been used for estimating skill levels from resumes (Banerjee et al., 2022), leaving a gap in research for unsupervised approaches and other types of sources such as job postings and course descriptions. This area offers significant potential for impactful research, given its unexplored status.
RD6: Unsupervised taxonomy construction A skill-based recommender system that would flexibly adapt to evolutions of the job market requires, on top of an unsupervised skill matching tool, a method to adapt the skill taxonomy to emerging market trends. Constructing and maintaining a skill taxonomy manually is a time-consuming and labor-intensive task. Moreover, it requires state-of-the-art expertise in the domain of emerging skills. To address this challenge, unsupervised methods for automatically constructing and updating skill taxonomies can be implemented. In a practical setting, this would be performed during the skill matching step, allowing the identification of skills that fail to be matched with taxonomy items and automatically adding them to the taxonomy (Xu et al., 2023; Takeoka et al., 2021).

5. Job-Oriented Course Recommender

In this section, we describe a first system that satisfies some of the properties we presented in section 4.1. Our methodology comprises two steps: Unsupervised Skill Extraction and Matching (SEM), followed by Job-Market-Oriented Course Recommender (JCRec).

Figure 1. Illustration of the skill extraction and matching (SEM) pipeline. 1 Given a document, SEM extracts relevant skills and proficiency levels from using LLM prompting. 2 For each extracted skill, three candidates from ESCO taxonomy are selected using string matching and embedding similarities. 3 The extracted skill and taxonomy candidates are prompted to an LLM to find the best match.

5.1. Skill Extraction and Matching

Following (Magron et al., 2024), we use an LLM-based system to extract skills from documents (job postings, course descriptions and resumes) and match these skills to the ESCO taxonomy (le Vrang et al., 2014a). We augment this pipeline with a proficiency levels extraction step, classifying each extracted skill into four categories: “beginner", “intermediate", “expert" and “unknown”. Because SEM is unsupervised, works on job postings, resumes, and course descriptions and estimates the proficiency level of skills, it satisfies property P3 while providing a first step for research direction RD5.

As illustrated in Figure 1, the SEM pipeline, follows three steps:
1 Skill & level Extraction from the document;
2 Candidate Selection from the taxonomy;
3 Skill Matching from the candidates;
This multi-step approach allows us to flexibly handle a taxonomy of any size without facing context window limitations. In the following, we describe in depth each of these steps.
1 Skill & level Extraction. We first break down each document into individual sentences that are then grouped into sets of one or two, which are processed through the pipeline sequentially. This approach provides the LLM with some context while addressing constraints related to document length. Next, we utilize the LLM in a six-shot setting to identify skills and their associated proficiency levels within these sentence groups. To ensure an effective extraction, our few-shot demonstrations are deliberately varied. They include a range of examples: positive and negative instances, answers that are both multiple and single, as well as examples of both hard and soft skills drawn from various contexts (Nguyen et al., 2024).
2 Candidate Selection. We select skill candidates from the ESCO taxonomy based on their names and definitions using two methods:
1) rule-based: We assume that if the extracted skill appears exactly in the taxonomy, it will be a good candidate for a match. When an exact match is not found, we use the token_set_ratio method from TheFuzz, which computes the similarity by comparing shared and unique tokens between the extracted and taxonomy skills.
1) embedding-based: We compute the cosine similarity between the extracted skill and all skills in the taxonomy using JobBERT (Decorte et al., 2021) embeddings.

The candidates are selected by taking the union of the top three ESCO skills returned by each method. We chose a hybrid approach because of the complementary merits and limitations of the two methods. The rule-based method selected viable candidates but missed any synonyms, whereas the embedding-based method risked selecting contextually similar yet factually dissimilar candidates.
3 Skill Matching. Extracted skills are matched to the selected candidate’s skills. Formatted candidate options (e.g., A. B. C.) are presented to the LLM, which identifies the best match if any, or indicates no match.

5.2. Recommendation

Figure 2. Illustration of the JCRec pipeline: In the offline phase, SEM is used to 1 extract skills required to take each course and skills provided by the courses, and to 2 extract skills required by each job posting skills. During the online phase, as a user uploads their CV, \small\textbf{1}⃝ SEM extract their skills. \small\textbf{2}⃝ These skills are used to filter the set of courses they can enroll in. \small\textbf{3}⃝ From these, one course is recommended, aiming to maximize the increase in the number of jobs the user can apply for. \small\textbf{4}⃝ The user’s profile is then updated with the skills acquired from the recommended course. \small\textbf{5}⃝ In the case of sequential recommendation, steps \small\textbf{2}⃝, \small\textbf{3}⃝, and \small\textbf{4}⃝ are repeated until $k$ courses are recommended.

As illustrated in Figure 2, JCRec has two main components: an offline preprocessing phase that uses SEM, and the online recommendation. Our system is fully unsupervised, makes explainable skill-based sequential recommendations, and assumes that the user’s goal is to maximize their marketability. These characteristics make our system satisfy properties P1, P2, P3, P5, and partially P4 since we assume a specific goal for the user. JCRec is also an additional step for research directions RD3 and RD4, although user studies will be required to assess the explainability of our approach in practice. In the following, we describe in depth JCRec.
Offline Preprocessing The preprocessing phase involves extracting skills and proficiency levels from courses and jobs.
1 Course Processing. In this phase, we use SEM on each course. This method identifies all the skills required for enrollment and the skills that will be acquired upon completion. As a result, we generate a set of processed courses denoted as $\mathcal{C}$. Each course $c\in\mathcal{C}$ is represented by two sets: $c_{r}$ the required skills with their associated proficiency levels, and $c_{p}$ the provided skills with their associated proficiency levels. We associate each proficiency level with a positive integer. For instance, the course Mastering Python: from beginner to expert might be represented as $(c_{r}:\{(python,1)\};c_{p}:\{(python,3)\})$, with the beginner level corresponding to integer 1 and the expert level to integer 3.
2 Job Processing. A similar preprocessing is applied to jobs, resulting in a set of processed jobs: $\mathcal{J}$. Each job $j\in\mathcal{J}$ is associated with a set of required skills and proficiency levels. For example, a Data Engineer job posting could be represented as $j:\{(python,2),(SQL,1)\}$..
Online Recommendation Using the processed courses and jobs, we can proceed to the online recommendation.
\small\textbf{1}⃝ Resume Processing. JCRec begins by processing the user’s resume. Using SEM, it extracts the user’s skills and proficiency levels, creating a set $u$ consisting of pairs of skills and their corresponding proficiency levels. For instance, a junior data scientist’s skill set might be represented as $u:\{(python,2),(machinelearning,2)\}$.
\small\textbf{2}⃝ Course Filtering. In the second step, we use a user-course relevance function $\text{ucr}(u,c)$ to determine $\mathcal{C}_{u}$: the set of courses available for enrollment by user $u$. The relevance function $\text{ucr}(u,c)$ is defined in Appendix A.2, along with the desirable properties it should satisfy. A threshold $t_{uc}$ is set to filter courses, kee** those with a relevance score higher than $t_{uc}$. Thus, $\mathcal{C}_{u}$ is defined as $\mathcal{C}_{u}=\{c\in\mathcal{C}|\;\text{ucr}(u,c)\geq t_{uc}\}$.
\small\textbf{3}⃝ Course Recommendation. Next, we recommend a course $c\in\mathcal{C}_{u}$. Given the nature of our problem as a sequential decision-making task, we use Reinforcement Learning (RL), as commonly done for sequential recommender systems (Afsar et al., 2022). We define our problem’s Markov Decision Process (MDP) as follows: the state is $u$ the user’s skill set; the action space is $\mathcal{C}$, the set of courses; and the reward to maximize is the marketability that we define as the number of jobs the user can apply to, denoted $|\mathcal{J}_{u}|$. Moreover, if the agent recommends a course that is not in $\mathcal{C}_{u}$, i.e. that the user cannot or should not follow, the reward is set to -1 and the episode is terminated. This choice of reward is motivated by the fact that we are only considering users whose goal is to increase their marketability. If different goals were considered, additional rewards would be required. Finally, the transition probabilities from state to state are deterministic: completing a course adds skills $c_{p}$ to $u$, resulting in probabilities of either 0 or 1.
\small\textbf{4}⃝ Update After recommending course $c$, the user’s skills $u$ are updated with $c_{p}$: the skills provided by course $c$. The number of jobs user $u$ can apply to $|\mathcal{J}_{u}|$ is computed using the similarity function $\text{ujs}(u,j)$ defined in Appendix A.1. To determine $|\mathcal{J}_{u}|$, we set a threshold $t_{uj}$ and consider that a user $u$ can apply to a job $j$ if and only if $\text{ujs}(u,j)\geq t_{uj}$, making the set of applicable jobs defined as: $\mathcal{J}_{u}=\{j\in\mathcal{J}|\;\text{ujs}(u,j)\geq t_{uj}\}$.
\small\textbf{5}⃝ Repeat To recommend a sequence of courses, we repeat steps \small\textbf{2}⃝, \small\textbf{3}⃝, and \small\textbf{4}⃝ until $k$ courses have been recommended.

6. Experimental Setup

6.1. Datasets

In our experiments and evaluation, we used publicly available datasets. We concentrated specifically on English-language documents within the information technology (IT) and IT management industry, due to the ease of access and relevance to our study’s focus.
Taxonomy. We use the ESCO (le Vrang et al., 2014b) taxonomy that comprises 13 890 skills and competencies. We filter out the competencies that are not relevant to IT and IT management roles, resulting in a subset of 1 794 skills.
Job Postings. We scraped around 3,500 English job postings in the domain of Information Technology from various online platforms.
Course Descriptions. The COCO dataset (Dessì et al., 2018) serves as a repository of 43 113 online Udemy courses, containing relevant information such as course descriptions, course objectives, and prerequisite requirements. The dataset is divided into 133 detailed categories and covers 46 languages. We first filter the dataset for only English courses (34 111 courses). Then, we filter the granular categories to include only the 27 that are related to IT and IT management (leaving 12 291 courses). In our experiments, we took a random sub-sample of 3 000 courses.
Resumes. We used two resume datasets from Kaggle for this study. The first dataset comprises 2 484 resumes spanning 24 industries. We filter the dataset to include only the Information-Technology category. The second dataset contains 166 unique resumes of professionals working in the IT department. We exclude resumes from non-IT personnel (i.e., in HR, Arts, or Health and fitness categories). Together, we keep a subset of 233 anonymized resumes from IT professionals.

6.2. Pre-processing

	Jobs	Courses		CVs
average		Prereqs	Target
Words per sentence	19.4	19.5	32.7	23.7
Sentences per doc	26.9	2.0	2.4	29.7
Words per doc	510.5	38.6	77.8	702.4

Table 1. Statistics of post-processed documents

For job postings and course descriptions, we remove the document if the combined text body contains less than 50 and 20 words respectively, a cutoff driven by examinations of documents below these thresholds. For courses, we split descriptive textual features into two categories: text describing prerequisite skills to take the course (“requirements” field), and text describing skills taught in the course (“course objectives” and “course description”). In addition, we chose to include only short descriptions and not long descriptions of the COCO courses because we found too many irrelevant and repetitive keywords mentioned in the long descriptions.

With each document type processed, we segment the text bodies into individual sentences for analysis by SEM Ṫable 1 shows the varying lengths of these documents, indicating notably longer job postings and resumes compared to courses. This difference arises as we only use short course descriptions.

Furthermore, through qualitative analysis, we find that sentences in resumes tend to be independent while those in jobs and courses are contextual. Based on this observation, we process each resume sentence independently, while job and course sentences are handled in pairs.

6.3. Skill matching setup

We run the job postings, course descriptions, and resumes through the matching pipeline, from which we extract 165K, 30K, and 31K total skills respectively. While we do not have annotated data, we can examine the results with heuristics. Table 2 indicates that resumes contain the most extracted skills but show the lowest percentage of proficiency level identification per skill. This trend aligns with common resume practices in the IT sector, where professionals often list numerous skills without specifying their proficiency levels. Moreover, course prerequisites and descriptions, as shown in Table 2, feature the fewest skills. This aligns with the expectation that courses have a limited scope, making the teaching of a vast number of skills impractical. Conversely, the number of skills extracted from job postings at 43.2 skills per document, or about 1.6 skills per sentence is justifiable. IT sector job postings often list many required skills, reflecting this sector’s diverse skill demands.

	Jobs	Courses		CVs
average per doc		Prereqs	Target
Skills extracted	43.2	3.2	6.8	112.9
Levels extracted	71.1%	72.0%	79.6%	45.4%
Skills matched	16.6	2.2	4.2	22.6

Table 2. Statistics on skills standardized by document type. The skills extracted represent the average number of unique skills extracted per document. Levels extracted represent the average percentage of extracted skills for which a known level was extracted. Skills matched is the average number of unique extracted skills that got matched to an entry in the taxonomy.

We also validated the performance of the proficiency level extraction method. A domain expert manually annotated 212 skills from 10 job postings in the IT sector. We compared the annotations with the levels extracted by our system (in-context with 6 demonstrations) and obtained an accuracy of 0.68.

Furthermore, we explore the results of the extraction on each type of document. Table 3 shows the percentage of the extracted proficiency levels that are "expert", "intermediate", "beginner", "unknown", or none of these options. Levels found in job descriptions are primarily "expert" indicating when employers suggest desired skill levels, nearly 50% of the time, they look for experts in that skill. This is expected and agrees with qualitative observations of job descriptions. More than 54% of levels from resumes are "unknown", which is reasonable as stated previously. However, when IT professionals do indicate their skill levels, it is also expected that skills are either at the "expert" level or "intermediate" level. This also agrees with qualitative observations, since professionals tend not to list a skill unless they are proficient. Looking at courses, levels of skills found in both course prerequisites and objectives are primarily "beginners". This is expected because online courses tend not to be in-depth and are geared toward those with a shallow understanding of the subject. It is also expected that very few courses (less than 10%) have intermediate or expert level prerequisites, whereas over 30% of courses teach intermediate or expert level skills. Finally, we observe that despite indicating in the prompt to output only specific words, all documents output a small percentage (under 1%) of non-conforming levels.

	Jobs	Courses		CVs
% of all skills		Prereqs	Target
"expert"	49.2	0.5	3.0	21.8
"intermediate"	19.8	9.0	29.5	20.6
"beginner"	2.1	51.7	47.1	3.0
"unknown"	28.3	38.0	20.1	54.4
other	0.6	0.8	0.3	0.2

Table 3. Distribution of extracted skill mastery levels for each document type. other indicate cases where the LLM failed to output a level in one of the 4 predefined categories.

Finally, we replace unknown levels following the following heuristic: if a level is unknown for a learner it will be set to beginner and if a level is unknown for a job, it will be set to expert. This heuristic allows us to have a conservative system that will not assume that a beginner would be able to apply to a job that requires expert knowledge. Regarding unknown proficiency levels for courses, we replace them with intermediate and correct the following types of inconsistencies: if a course requires skill $s$ at level $l_{r}$ and also teaches the same skill $s$ at a lower level $l_{p}$ such that $l_{r}\geq l_{p}$ this implies an inconsistency: the course demands a higher skill level than it provides. In such cases, we adjust $l_{r}$ to the level before $l_{p}$ to ensure the removal of any such inconsistencies.

For skill extraction and skill matching, we use GPT-3.5-turbo with temperature set to 0, top-p set to 1.0, frequency penalty set to 0.0, and presence penalty set to 0.0.

6.4. Recommendation Algorithms

Model	$k=0$	$k=1$		$k=2$		$k=3$		$k=4$		$k=5$
	Rwd	Rwd	Time (ms)	Rwd	Time (ms)	Rwd	Time (ms)	Rwd	Time (ms)	Rwd	Time (ms)
Exhaustive	0.1	1.0	16	2.5	$10^{2}$	5.6	$10^{4}$	NA	NA	NA	NA
Greedy	0.1	1.0	16	2.0	41	4.1	73	7.0	$10^{2}$	10.5	$10^{2}$
DQN	0.1	0.9	4.8	1.7	8.7	4.1	11	6.3	17	7.1	24
PPO	0.1	0.5	3.7	1.1	6.4	3.4	11	8.0	20	10.4	21

Table 4. Evaluation of the 4 algorithms for course recommendation. Rwd stands for reward and is equal to the average number of jobs learners can apply to after following the course recommendations. The highest reward among Greedy, DQN, and PPO is highlighted in bold for each $k$. Time stands for the average time in milliseconds taken to make a recommendation for a single learner. The lowest time among Greedy, DQN, and PPO is underlined for each $k$. The case $k=0$ indicates the initial state of the learner before any recommendations are made and is, therefore, the same for all algorithms. We did not run the exhaustive algorithm for the case $k=4$ and $k=5$, as we estimated the time taken by the algorithm would be at least $10^{6}$ secondes per learner.

In this work, we compare four algorithms for sequential recommendations: an exhaustive algorithm, a greedy algorithm, and two RL algorithms.
Exhaustive Recommendation: The exhaustive approach evaluates all possible course sequences that can be recommended to a learner. For every possible sequence, it computes $|\mathcal{J}_{u}|$ the number of jobs the learner can apply to after updating their profile. The sequence that maximizes $|\mathcal{J}_{u}|$ is then recommended. Although this approach guarantees the recommendation of the optimal sequence, its practical feasibility is limited due to the high time complexity that increases exponentially with the number of recommended courses: $O(|\mathcal{J}|\cdot|\mathcal{C}|^{k})$. Nevertheless, we have chosen to present the outcomes of this methodology, despite its impracticality for real-world application, as it provides a theoretical maximum for the value of $|\mathcal{J}_{u}|$.
Greedy Recommendation: At each of the $k$ steps, the greedy algorithm recommends a course that maximizes the number of job opportunities available to the learner. This approach is greedy because it selects the best immediate option at each step without guaranteeing an overall optimal sequence. Despite this sub-optimality, it is more time-efficient than the exhaustive approach. For each course recommendation, the learner’s profile is updated and the similarity with every job is computed to estimate the course leading to the highest increase in job applicability. This results in a total number of operations on the order of $O(k\cdot|\mathcal{J}|\cdot|\mathcal{C}|)$.
Reinforcement Learning Recommendation: We compare 2 RL algorithms: Deep Q Network (DQN) (Mnih et al., 2013) and Proximal Policy Optimization (PPO) (Schulman et al., 2017). For both RL algorithms, we used the implementation provided by Stable-Baselines3 (Raffin et al., 2021) with default parameter values. The input size is the number of skills $|\mathcal{S}|$ and the output dimension is the number of actions, or courses, $|\mathcal{C}|$. This makes the time complexity of these algorithms $O(k\cdot|\mathcal{S}|\cdot|\mathcal{C}|)$, meaning that RL has the potential to be more efficient than the greedy heuristic, especially if the number of skills $|\mathcal{S}|$ is one or more order of magnitude smaller than the number of jobs $|\mathcal{J}|$.
Hyperparameters: Both threshold values $t_{uc}$ and $t_{uj}$ were set to 0.8, the RL agents were trained for 5 000 000 steps.

6.5. Evaluation

We evaluate the algorithms algorithms using different values of $k$: the number of courses recommended in the sequence. Our evaluation employs two metrics to compare the algorithms: 1) The average number of jobs a learner can apply to after receiving $k$ course recommendations assessing the relevance of the courses recommended, and 2) the average time taken to make a recommendation, evaluating the algorithms’ feasibility for real-world deployment. Evaluation is conducted on a subset of the dataset described in section 6.1, including 100 jobs, 100 courses, and all resumes containing less than 15 skills for a total of 52 resumes. We decided to limit the number of skills present in the resume to emulate a learner new to the job market.

7. Results

Table 4 presents a comparison of the Exhaustive, Greedy, DQN, and PPO algorithms across different sequence lengths ($k$). Rwd is the average number of jobs learners can apply to after taking the recommended courses and Time is the average duration for each recommendation in milliseconds.

The Exhaustive algorithm, although delivering the best recommendation quality in terms of reward is not suited for real-world deployment due to its significant time consumption. Even on a small dataset of 100 courses and jobs, it can take up to 10 seconds to recommend a sequence of 3 courses. Its usage becomes unfeasible as the sequence length increases, rendering it unsuitable for practical applications. However, the exhaustive algorithm provides an upper bound on the reward useful for analysis as it outputs the optimal course sequence. Nevertheless, in the remainder of our analysis, we will not compare the exhaustive approach to the greedy and RL algorithms.

For shorter sequences ($k=1,2,3$), the Greedy algorithm achieves the highest Rwd with reasonable recommendation times. Particularly at lower values of $k$, the Greedy heuristic offers a balance between speed and reward, making it the preferred approach when recommending a smaller number of courses.

RL approaches demonstrate their strength in efficiency, particularly for recommending longer sequences of courses ($k=4,5$). PPO, in particular, stands out for maintaining a recommendation quality on par with the Greedy algorithm but with significantly faster execution (one order of magnitude faster than the Greedy approach). This characteristic makes PPO a more suitable option for longer sequence recommendations. Overall, the RL algorithms, DQN and PPO, show potential as the fastest methods, especially in scenarios with larger datasets. When longer sequences of courses are required, RL methods offer a more efficient alternative without compromising the quality of recommendations.

In summary, our results suggest using different approaches to course recommendation based on the sequence length: the Greedy heuristic for shorter sequences where fewer courses are recommended, and RL approaches, particularly PPO, for longer sequences and larger datasets where efficiency becomes paramount.

8. Limitations

While our work contributes valuable insights and research directions to the field of job-oriented course recommender systems, it is important to acknowledge several limitations that may impact the generalizability and effectiveness of our recommender system.

Language Restriction. Our system currently operates exclusively on English language data. This limitation restricts the applicability of our model in multilingual contexts and excludes non-English job postings and courses.

Dataset Size. The datasets used in this study are relatively small, limiting the robustness and scalability of our findings.

Evaluation. The lack of expert annotators to validate the course recommendation means our evaluation may not fully capture the system’s effectiveness in real-world scenarios.

Reliance on Heuristics. Our approach includes several heuristic methods. While they are necessary for handling data complexities, these heuristics introduce an element of subjectivity and may not always reflect real-world learning and job market dynamics.

Assumption of Skill Acquisition. A fundamental assumption in our sequential recommendation approach is that completing a course will automatically result in the acquisition of the associated skills by the learners. This overlooks the variability in individual learning outcomes and the probability of successfully acquiring new skills. A probabilistic approach that estimates the likelihood of skill acquisition would provide a more nuanced and realistic model.

9. Conclusion

In this work, we provided the perspective of academic researchers working in collaboration with industry practitioners to develop and deploy a job-market-oriented course recommender system. We proposed to rethink course recommender systems to consider the job market, proposed several properties that such systems should satisfy, as well as research directions that would help develop this field.

We introduced SEM, a skill extraction and matching method that efficiently aligns skills extracted from resumes, course content, and job descriptions with the ESCO taxonomy. Utilizing in-context learning and Large Language Models (LLMs), SEM is fully unsupervised, enabling generalization to any document type and adaptation to an ever-evolving job market.

Building on this foundation, we developed an unsupervised course recommender system that leverages the matched skills to suggest course sequences aimed at maximizing employment opportunities.

Our investigation of sequential recommendation strategies — including a greedy heuristic, an exhaustive approach, and two Reinforcement Learning (RL) models — revealed insightful findings. Notably, the RL approaches, particularly Proximal Policy Optimization (PPO), stand out in recommending longer sequences efficiently. They offer a promising solution for larger datasets, balancing recommendation quality with computational feasibility.

Appendix A Relevance and Similarity function design

In this section, we motivate our design choices for the user-job similarity function and the user-course relevance function described in section 5.2. Drawing inspiration from axiomatic Information Retrieval (Fang and Zhai, 2005), we have designed these similarity functions based on a set of desirable constraints they should fulfill.

A.1. User-Job Similarity

First, we describe the constraints for the user-job similarity function:
UJC1: Assign a higher score to a user who possesses more skills required for a job.
UJC2: Assign a higher or equal score to a user with higher proficiency levels in the skills required for a job.
IJC3: Assign the maximal score to a user who has all the skills and proficiency levels required for a job.

Based on these constraints, we propose the following user-job similarity function, denoted as uj-sim:

(1) $$\text{uj-sim}(u,j)=\sum\limits_{s\in j}\frac{\text{sim}(\text{sl}_{s,u},\text{sl}_{s,j})}{|j|}$$

(2) $$\text{sim}(l_{i},l_{j})=\frac{\min(l_{i},l_{j})}{l_{j}}$$

Here $\text{sl}_{s,j}$ (respectively $\text{sl}_{s,u}$) represents the skill proficiency level for skill $s$ in job $j$ (respectively for user $u$). If a user does not have skill $s$, then $\text{sl}_{s,u}=0$. The use of $\min$ in the skill-skill similarity function (equation 2), ensures that users matching all requirements will achieve maximum similarity, satisfying constraint UJC3.

A.2. User-Course Relevance

Next, we define the constraints for the user-course relevance:
UCC1: Assign a higher score to a user who possesses more skills required for a course.
UCC2: Assign a higher or equal score to a user with higher proficiency levels in the skills required for a course.
UCC3: Assign the minimal score to a user who already possesses all the skills and proficiency levels provided by a course.
UCC3 is based on the rationale that if a user already knows everything taught by a course, they will gain no new skills making the course irrelevant.

We propose the following user-course relevance function, denoted as uc-rel:

(3) $$\text{uc-rel}(u,c)=\text{uc}_{\text{r}}(u,c_{r})\cdot(1-\text{uc}_{\text{p}}(u,c_{p}))$$

(4) $$\text{uc}_{\text{r}}(u,c_{r})=\sum\limits_{s\in c_{r}}\frac{\text{sim}(\text{sl}_{s,u},\text{sl}_{s,c_{r}})}{|c_{r}|}$$

(5) $$\text{uc}_{\text{p}}(u,c_{p})=\sum\limits_{s\in c_{p}}\frac{\text{sim}(\text{sl}_{s,u},\text{sl}_{s,c_{p}})}{|c_{p}|}$$

In these equations, $\text{sl}_{s,c_{r}}$ (respectively $\text{sl}_{s,c_{p}}$) indicates the required (respectively provided) proficiency level for skill $s$ in course $c$. Because $\text{uc}_{\text{p}}(u,c_{p})$ is bounded between 0 and 1 (from equation 2) and returns 1 if and only if user $u$ already possesses all the skills provided by the course, the overall relevance $\text{uc-rel}(u,c)$ will be equal to 0 (the lowest possible value) ensuring constraint UCC3.