Original: CAREER: Economic Prediction of Labor Sequence Data Under Distribution Shift
Revised: CAREER: Economic Prediction of Labor Sequence Data Under Distribution Shift

Original: Abstract
Revised: Abstract

Original: Labor economists regularly analyze employment data by ﬁtting predictive models to small, carefully constructed longitudinal survey datasets.
Revised: Labor economists regularly analyze employment data by ﬁtting predictive models to small, carefully constructed longitudinal survey datasets.

Original: Although modern machine learning methods oﬀer promise for such problems, these survey datasets are too small to take advantage of them.
Revised: Although modern machine learning methods oﬀer promise for such problems, these survey datasets are too small to take advantage of them.

Original: In recent years large datasets of online resumes have also become available, providing data about the career trajectories of millions of individuals.
Revised: In recent years large datasets of online resumes have also become available, providing data about the career trajectories of millions of individuals.

Original: However, the distribution of these large resume datasets diﬀer in meaningful ways from the survey datasets used for economic estimation; standard econometric models cannot take advantage of their scale or make predictions under distribution shift.
Revised: However, the distribution of these large resume datasets diﬀer in meaningful ways from the survey datasets used for economic estimation; standard econometric models cannot take advantage of their scale or make predictions under distribution shift.

Original: To this end we develop CAREER, a transformer-based model that uses transfer learning to learn representations of job sequences.
Revised: To this end we develop CAREER, a transformer-based model that uses transfer learning to learn representations of job sequences.

Original: CAREER is ﬁrst ﬁt to large, passively-collected resume data and then ﬁne-tuned on samples of the downstream data distribution of interest.
Revised: CAREER is ﬁrst ﬁt to large, passively-collected resume data and then ﬁne-tuned on samples of the downstream data distribution of interest.

Original: We ﬁnd that CAREER forms accurate predictions of job sequences, achieving state-of-the-art predictive performance on three widely-used economics datasets.
Revised: We ﬁnd that CAREER forms accurate predictions of job sequences, achieving state-of-the-art predictive performance on three widely-used economics datasets.

Original: We also ﬁnd that CAREER is adept at making predictions under distribution shifts in time.
Revised: We also ﬁnd that CAREER is adept at making predictions under distribution shifts in time.

Original:  Introduction
Revised:  Introduction

Original: A variety of economic analyses rely on models for predicting an individual’s future occupations.
Revised: A variety of economic analyses rely on models for predicting an individual’s future occupations.

Original: These models are crucial for estimating important economic quantities, such as gender or racial diﬀerences in unemployment (Hall, 1972; Fairlie & Sundstrom, 1999); they underpin causal analyses and decompositions that rely on simulating counterfactual occupations for individuals (Brown et al., 1980; Schubert et al., 2021); and they inform policy, by forecasting occupations with rising or declining market shares.
Revised: These models are crucial for estimating important economic quantities, such as gender or racial diﬀerences in unemployment (Hall, 1972; Fairlie & Sundstrom, 1999); they underpin causal analyses and decompositions that rely on simulating counterfactual occupations for individuals (Brown et al., 1980; Schubert et al., 2021); and they inform policy, by forecasting occupations with rising or declining market shares.

Original: These analyses typically involve ﬁtting predictive models to longitudinal surveys that follow a cohort of individuals during their working career (Panel Study of Income Dynamics, 2021; Bureau of Labor
Revised: These analyses typically involve ﬁtting predictive models to longitudinal surveys that follow a cohort of individuals during their working career (Panel Study of Income Dynamics, 2021; Bureau of Labor

Original: Statistics, 2019a).
Revised: Statistics, 2019a).

Original: Such surveys have been carefully collected to represent national demographics, ensuring that the economic analyses can generalize to larger populations.
Revised: Such surveys have been carefully collected to represent national demographics, ensuring that the economic analyses can generalize to larger populations.

Original: But these datasets are also small, usually containing only thousands of workers, because maintaining them requires regularly interviewing each individual.
Revised: But these datasets are also small, usually containing only thousands of workers, because maintaining them requires regularly interviewing each individual.

Original: Consequently, economists use simple sequential models, where a worker’s next occupation depends on their history only through the most recent occupation (Hall, 1972) or a few summary statistics about the past (Blau & Riphahn, 1999).
Revised: Consequently, economists use simple sequential models, where a worker’s next occupation depends on their history only through the most recent occupation (Hall, 1972) or a few summary statistics about the past (Blau & Riphahn, 1999).

Original: In recent years, however, much larger datasets of online resumes have also become available.
Revised: In recent years, however, much larger datasets of online resumes have also become available.

Original: In contrast to longitudinal surveys, these passively-collected datasets are not used directly for economic inferences because they are not constructed to be nationally representative, and thus diﬀer in distribution from the survey datasets of interest.
Revised: In contrast to longitudinal surveys, these passively-collected datasets are not used directly for economic inferences because they are not constructed to be nationally representative, and thus diﬀer in distribution from the survey datasets of interest.

Original: However, they provide occupation sequences of millions of individuals, potentially expanding the scope of insights that can be obtained from analyses on downstream survey datasets.
Revised: However, they provide occupation sequences of millions of individuals, potentially expanding the scope of insights that can be obtained from analyses on downstream survey datasets.

Original: The simple econometric models currently in use cannot take advantage of the scale of the data or make predictions under distribution shift.
Revised: The simple econometric models currently in use cannot take advantage of the scale of the data or make predictions under distribution shift.

Original: To this end, we develop CAREER, a neural sequence model of occupation trajectories.
Revised: To this end, we develop CAREER, a neural sequence model of occupation trajectories.

Original: CAREER is designed to be pretrained on large-scale resume data and then ﬁne-tuned on samples from smaller and better-curated survey data for economic prediction.
Revised: CAREER is designed to be pretrained on large-scale resume data and then ﬁne-tuned on samples from smaller and better-curated survey data for economic prediction.

Original: Its architecture is based on the transformer language model (Vaswani et al., 2017), for which pretraining and ﬁne-tuning has proven to be an eﬀective paradigm for many NLP tasks (Devlin et al., 2019).
Revised: Its architecture is based on the transformer language model (Vaswani et al., 2017), for which pretraining and ﬁne-tuning has proven to be an eﬀective paradigm for many NLP tasks (Devlin et al., 2019).

Original: CAREER extends this transformer-based transfer learning approach to modeling sequences of occupations, rather than text.
Revised: CAREER extends this transformer-based transfer learning approach to modeling sequences of occupations, rather than text.

Original: In this paper, we consider two kinds of distribution shifts: distribution shifts in the kinds of data (i.e. between large resume datasets and smaller economic survey datasets of interest), and distribution shifts in time (i.e. between job sequences before and after some date).
Revised: In this paper, we consider two kinds of distribution shifts: distribution shifts in the kinds of data (i.e. between large resume datasets and smaller economic survey datasets of interest), and distribution shifts in time (i.e. between job sequences before and after some date).

Original: We will show that CAREER’s representations provide eﬀective predictions of occupations under both kinds of distribution shifts.
Revised: We will show that CAREER’s representations provide eﬀective predictions of occupations under both kinds of distribution shifts.

Original:  CAREER
Revised:  CAREER

Original: Occupation models.
Revised: Occupation models.

Original: Consider an individual worker.
Revised: Consider an individual worker.

Original: This person’s career can be deﬁned as a series of timesteps.
Revised: This person’s career can be deﬁned as a series of timesteps.

Original: Here, we use a timestep of one year.
Revised: Here, we use a timestep of one year.

Original: At each timestep, this individual works in a job: it could be the same job as the previous timestep, or a diﬀerent job.
Revised: At each timestep, this individual works in a job: it could be the same job as the previous timestep, or a diﬀerent job.

Original: (Note we use the terms “occupation” and “job” synonymously.)
Revised: (Note we use the terms “occupation” and “job” synonymously.)

Original: We consider “unemployed” and “out-of-labor-force” to be special job types.
Revised: We consider “unemployed” and “out-of-labor-force” to be special job types.

Original: Deﬁne an occupation model to be a probability distribution over sequences of jobs.
Revised: Deﬁne an occupation model to be a probability distribution over sequences of jobs.

Original: An occupation model predicts a worker’s job at each timestep as a function of all previous jobs and other observed characteristics of the worker.
Revised: An occupation model predicts a worker’s job at each timestep as a function of all previous jobs and other observed characteristics of the worker.

Original: More formally, deﬁne an individual’s career to be a sequence ( y 1 , . .
Revised: More formally, deﬁne an individual’s career to be a sequence ( y 1 , . .

Original: , y T ) , where each y t ∈ { 1 , . . .
Revised: , y T ) , where each y t ∈ { 1 , . . .

Original: , J } indexes one of J occupations at time t .
Revised: , J } indexes one of J occupations at time t .

Original: Occupations are categorical; one example of a sequence could be (“cashier”, “salesperson”, ... , “sales manager”).
Revised: Occupations are categorical; one example of a sequence could be (“cashier”, “salesperson”, ... , “sales manager”).

Original: At each timestep, an individual is also associated with C observed covariates x t = { x tc } Cc =1 .
Revised: At each timestep, an individual is also associated with C observed covariates x t = { x tc } Cc =1 .

Original: Covariates are also categorical, with x tc ∈ { 1 , . . .
Revised: Covariates are also categorical, with x tc ∈ { 1 , . . .

Original: , N c } .
Revised: , N c } .

Original: For example, if c corresponds to the most recent educational degree, x tc could be “high school diploma” or “bachelors”, and N c is the number of types of educational degrees.
Revised: For example, if c corresponds to the most recent educational degree, x tc could be “high school diploma” or “bachelors”, and N c is the number of types of educational degrees.

Original: 2 Deﬁne y t =
Revised: 2 Deﬁne y t =

Original: ( y 1 , . . .
Revised: ( y 1 , . . .

Original: , y t ) to index all jobs that have occurred up to time t , with the analogous deﬁnition for x t .
Revised: , y t ) to index all jobs that have occurred up to time t , with the analogous deﬁnition for x t .

Original: At each timestep, an occupation model predicts an individual’s job in the next timestep, p ( y t | y t − 1 , x t ) .
Revised: At each timestep, an occupation model predicts an individual’s job in the next timestep, p ( y t | y t − 1 , x t ) .

Original: This distribution conditions on covariates from the same timestep because these are “pre-transition.”
Revised: This distribution conditions on covariates from the same timestep because these are “pre-transition.”

Original: For example, an individual’s most recent educational degree is available to the model as it predicts their next job.
Revised: For example, an individual’s most recent educational degree is available to the model as it predicts their next job.

Original: An occupation model’s predictions are governed by an individual’s career history; both whether an individual changes jobs and the speciﬁc job they may transition to depend on current and previous jobs and covariates.
Revised: An occupation model’s predictions are governed by an individual’s career history; both whether an individual changes jobs and the speciﬁc job they may transition to depend on current and previous jobs and covariates.

Original: We consider a class of occupation models that make predictions by conditioning on a low-dimensional representation of work history, h t ( y t − 1 , x t ) ∈
Revised: We consider a class of occupation models that make predictions by conditioning on a low-dimensional representation of work history, h t ( y t − 1 , x t ) ∈

Original: R D .
Revised: R D .

Original: This representation is assumed to be a suﬃcient statistic of the past; h t ( y t − 1 , x t ) should contain the relevant observed information for predicting the next job.
Revised: This representation is assumed to be a suﬃcient statistic of the past; h t ( y t − 1 , x t ) should contain the relevant observed information for predicting the next job.

Original: Since individuals frequently stay in the same job between timesteps, we propose a class of models that make predictions in two stages.
Revised: Since individuals frequently stay in the same job between timesteps, we propose a class of models that make predictions in two stages.

Original: In the ﬁrst stage, the career representation h t ( y t − 1 , x t ) is used to predict whether an individual changes jobs.
Revised: In the ﬁrst stage, the career representation h t ( y t − 1 , x t ) is used to predict whether an individual changes jobs.

Original: Deﬁne the binary variable s t to be 1 if a worker’s job at time t is diﬀerent from that at time t − 1 , and 0 otherwise.
Revised: Deﬁne the binary variable s t to be 1 if a worker’s job at time t is diﬀerent from that at time t − 1 , and 0 otherwise.

Original: The ﬁrst stage is a logistic regression, s
Revised: The ﬁrst stage is a logistic regression, s

Original: [Equation]
Revised: [Equation]

Original: ∼ Bernoulli
Revised: ∼ Bernoulli

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where σ ( · ) is the logistic function and η ∈ R D is a vector of coeﬃcients.
Revised: where σ ( · ) is the logistic function and η ∈ R D is a vector of coeﬃcients.

Original: If the model predicts that an individual will transition jobs, it only considers jobs that are diﬀerent from the individual’s most recent job.
Revised: If the model predicts that an individual will transition jobs, it only considers jobs that are diﬀerent from the individual’s most recent job.

Original: To formulate this prediction, it combines the career representation with a vector of occupation-speciﬁc coeﬃcients β j ∈ R D :
Revised: To formulate this prediction, it combines the career representation with a vector of occupation-speciﬁc coeﬃcients β j ∈ R D :

Original: [Equation]
Revised: [Equation]

Original: Otherwise, the next job is deterministic:
Revised: Otherwise, the next job is deterministic:

Original: [Equation]
Revised: [Equation]

Original: CAREER.
Revised: CAREER.

Original: We develop a two-stage representation-based occupation model called CAREER .
Revised: We develop a two-stage representation-based occupation model called CAREER .

Original: This model uses a transformer to parameterize a representation of an individual’s history.
Revised: This model uses a transformer to parameterize a representation of an individual’s history.

Original: CAREER handles distribution shifts between resume datasets and small survey datasets of interest via transfer learning: it is pretrained on large resume datasets and then ﬁne-tuned to make predictions on samples from downstream data distributions.
Revised: CAREER handles distribution shifts between resume datasets and small survey datasets of interest via transfer learning: it is pretrained on large resume datasets and then ﬁne-tuned to make predictions on samples from downstream data distributions.

Original: CAREER is based on a transformer, a sequence model that uses neural networks to learn representations of discrete tokens (Vaswani et al., 2017).
Revised: CAREER is based on a transformer, a sequence model that uses neural networks to learn representations of discrete tokens (Vaswani et al., 2017).

Original: While transformers were developed to model sequences of words, CAREER uses a transformer to model sequences of jobs.
Revised: While transformers were developed to model sequences of words, CAREER uses a transformer to model sequences of jobs.

Original: CAREER makes two modiﬁcations to the transformer architecture to account for diﬀerences between labor and text data.
Revised: CAREER makes two modiﬁcations to the transformer architecture to account for diﬀerences between labor and text data.

Original: First, as described above, the model makes predictions in two stages, making it better-suited to model workers who stay in the same job through consecutive timesteps.
Revised: First, as described above, the model makes predictions in two stages, making it better-suited to model workers who stay in the same job through consecutive timesteps.

Original: (In contrast, words seldom repeat.)
Revised: (In contrast, words seldom repeat.)

Original: Second, while language models only condition on previous words, each career is also associated with covariates x that may aﬀect transition distributions (see Equation 2).
Revised: Second, while language models only condition on previous words, each career is also associated with covariates x that may aﬀect transition distributions (see Equation 2).

Original: See Appendix A for more details.
Revised: See Appendix A for more details.

Original: Distribution shifts.
Revised: Distribution shifts.

Original: Economists apply occupation models to survey datasets that have been carefully collected to represent national demographics.
Revised: Economists apply occupation models to survey datasets that have been carefully collected to represent national demographics.

Original: In the United States, these datasets contain a small number of individuals.
Revised: In the United States, these datasets contain a small number of individuals.

Original: While transformers have been successfully applied to large NLP datasets, they are prone to overﬁtting on small datasets (Kaplan et al., 2020; Dosovitskiy et al., 2021; Variš & Bojar, 2021).
Revised: While transformers have been successfully applied to large NLP datasets, they are prone to overﬁtting on small datasets (Kaplan et al., 2020; Dosovitskiy et al., 2021; Variš & Bojar, 2021).

Original: As such, CAREER may not learn useful representations solely from small survey datasets.
Revised: As such, CAREER may not learn useful representations solely from small survey datasets.

Original: In recent years, however, much larger datasets of online resumes have also become available.
Revised: In recent years, however, much larger datasets of online resumes have also become available.

Original: Although these passively-collected datasets provide job sequences of many more individuals, they are not used for economic estimation for a few reasons.
Revised: Although these passively-collected datasets provide job sequences of many more individuals, they are not used for economic estimation for a few reasons.

Original: The occupation sequences from resumes are imputed from short textual descriptions, a process that inevitably introduces more noise and errors than collecting data from detailed questionnaires.
Revised: The occupation sequences from resumes are imputed from short textual descriptions, a process that inevitably introduces more noise and errors than collecting data from detailed questionnaires.

Original: Additionally, individuals may not accurately list their work experiences on resumes (Wexler, 2006), and important economic variables relating to demographics and wage are not available.
Revised: Additionally, individuals may not accurately list their work experiences on resumes (Wexler, 2006), and important economic variables relating to demographics and wage are not available.

Original: Finally, these datasets are not constructed to ensure that they are representative of the general population.
Revised: Finally, these datasets are not constructed to ensure that they are representative of the general population.

Original: These two types of data reﬂect job sequences from diﬀerent distributions.
Revised: These two types of data reﬂect job sequences from diﬀerent distributions.

Original: On the one hand, resume data is large-scale and contains valuable information about employment patterns.
Revised: On the one hand, resume data is large-scale and contains valuable information about employment patterns.

Original: On the other hand, survey datasets are carefully collected from data distributions of interest.
Revised: On the other hand, survey datasets are carefully collected from data distributions of interest.

Original: Thus CAREER incorporates the patterns embedded in large-scale resume data into the analysis of survey datasets.
Revised: Thus CAREER incorporates the patterns embedded in large-scale resume data into the analysis of survey datasets.

Original: It does this through transfer learning: CAREER is ﬁrst pretrained on a large dataset of resumes to learn an initial representation of careers.
Revised: It does this through transfer learning: CAREER is ﬁrst pretrained on a large dataset of resumes to learn an initial representation of careers.

Original: When CAREER is then ﬁt to a small survey dataset, parameters are not initialized randomly; instead, they are initialized with the representations learned from resumes.
Revised: When CAREER is then ﬁt to a small survey dataset, parameters are not initialized randomly; instead, they are initialized with the representations learned from resumes.

Original: After initialization, all parameters are ﬁne-tuned on samples of the small dataset of interest by optimizing the likelihood.
Revised: After initialization, all parameters are ﬁne-tuned on samples of the small dataset of interest by optimizing the likelihood.

Original: Because the objective function is non-convex, learned representations depend on their initial values.
Revised: Because the objective function is non-convex, learned representations depend on their initial values.

Original: Initializing with the pretrained representations ensures that the model does not need to re-learn representations on the small dataset.
Revised: Initializing with the pretrained representations ensures that the model does not need to re-learn representations on the small dataset.

Original: Instead, it only adjusts representations to account for dataset diﬀerences.
Revised: Instead, it only adjusts representations to account for dataset diﬀerences.

Original: Another distribution shift occurs when CAREER is used to predict future jobs for individuals: occupational trends change over time.
Revised: Another distribution shift occurs when CAREER is used to predict future jobs for individuals: occupational trends change over time.

Original: In other words, the historic sequences of jobs used for training may not reﬂect the future sequences of jobs used for prediction.
Revised: In other words, the historic sequences of jobs used for training may not reﬂect the future sequences of jobs used for prediction.

Original: CAREER handles this case by learning year-speciﬁc embeddings; see Appendix A.
Revised: CAREER handles this case by learning year-speciﬁc embeddings; see Appendix A.

Original: These embeddings can be imputed for unseen years via interpolation techniques.
Revised: These embeddings can be imputed for unseen years via interpolation techniques.

Original:  Empirical Studies
Revised:  Empirical Studies

Original: We assess CAREER’s ability to predict jobs under two kinds of distribution shifts: shifts in kinds of data types (i.e. between large resume data and smaller economic survey datasets of interest), and distribution shifts in time.
Revised: We assess CAREER’s ability to predict jobs under two kinds of distribution shifts: shifts in kinds of data types (i.e. between large resume data and smaller economic survey datasets of interest), and distribution shifts in time.

Original: [Table]
Revised: [Table]

Original: [Table] 
Revised: [Table] [Figure] [Table] [Figure]

Original: [Figure] 
Revised: [Figure] [Figure] [Figure] [Table]

Original:  [Table]
Revised: [Figure] [Table] [Figure] [Table]

Original: [Figure]  [Figure]
Revised: [Figure] [Table] [Figure]

Original: [Figure]

Original: [Figure] [Table] [Figure] [Table]
Revised: [Table]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure] [Table] [Figure] [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: dictive have
Revised: dictive have

Original: Figure 2a compares the test-set perplexity of each model.
Revised: Figure 2a compares the test-set perplexity of each model.

Original: With the transferred representations, CA- REER makes the best predictions on all survey datasets, achieving state-of-the-art performance.
Revised: With the transferred representations, CA- REER makes the best predictions on all survey datasets, achieving state-of-the-art performance.

Original: NEMO, which was designed to model large resume datasets, struggles to make good predictions on these small survey datasets, performing on par with standard econometric baselines.
Revised: NEMO, which was designed to model large resume datasets, struggles to make good predictions on these small survey datasets, performing on par with standard econometric baselines.

Original: Pretraining is the biggest source of CAREER’s improvements.
Revised: Pretraining is the biggest source of CAREER’s improvements.

Original: Although the resume data is noisy and differs in many ways from the survey datasets used for economic prediction, CAREER learns useful representations of work experiences that aid its predictive performance.
Revised: Although the resume data is noisy and differs in many ways from the survey datasets used for economic prediction, CAREER learns useful representations of work experiences that aid its predictive performance.

Original: In Appendix G we show that modifying the baselines to incorporate two-stage prediction (Equations 1 to 3) improves their performance, although CAREER still makes the best predictions across datasets.
Revised: In Appendix G we show that modifying the baselines to incorporate two-stage prediction (Equations 1 to 3) improves their performance, although CAREER still makes the best predictions across datasets.

Original: We include qualitative analysis of CAREER’s predictions in Appendix D.
Revised: We include qualitative analysis of CAREER’s predictions in Appendix D.

Original: To assess how the volume of resumes used for pretraining affects CAREER’s predictions on survey datasets, we downsample the resume dataset and transfer to survey datasets.
Revised: To assess how the volume of resumes used for pretraining affects CAREER’s predictions on survey datasets, we downsample the resume dataset and transfer to survey datasets.

Original: The scaling law for
Revised: The scaling law for

Original: NLSY79 is depicted in Figure 2b.
Revised: NLSY79 is depicted in Figure 2b.

Original: When there are less than 20,000 examples in the resume dataset, pretraining CAREER does not offer any improvement.
Revised: When there are less than 20,000 examples in the resume dataset, pretraining CAREER does not offer any improvement.

Original: The relationship between pretraining volume and ﬁne-tuned perplexity follows a power law, similar to scaling laws in NLP (Kaplan et al., 2020).
Revised: The relationship between pretraining volume and ﬁne-tuned perplexity follows a power law, similar to scaling laws in NLP (Kaplan et al., 2020).

Original: We also assess CAREER’s ability to forecast future career trajectories.
Revised: We also assess CAREER’s ability to forecast future career trajectories.

Original: In contrast to predicting held-out sequences, forecasting involves training models on all sequences before a speciﬁc year.
Revised: In contrast to predicting held-out sequences, forecasting involves training models on all sequences before a speciﬁc year.

Original: To predict future jobs for an individual, the ﬁtted model is used to estimate job probabilities six years into the future by sampling multi-year trajectories.
Revised: To predict future jobs for an individual, the ﬁtted model is used to estimate job probabilities six years into the future by sampling multi-year trajectories.

Original: This setting is useful for assessing a model’s ability to make long-term predictions, especially as occupational trends change over time.
Revised: This setting is useful for assessing a model’s ability to make long-term predictions, especially as occupational trends change over time.

Original: We evaluate CAREER’s forecasting abilities on NLSY97 and PSID.
Revised: We evaluate CAREER’s forecasting abilities on NLSY97 and PSID.

Original: (These datasets are more valu- able for forecasting than NLSY79, which follows a cohort that is near or past retirement age.)
Revised: (These datasets are more valu- able for forecasting than NLSY79, which follows a cohort that is near or past retirement age.)

Original: We train models on all sequences (holding out 10% as a validation set), without including any obser- vations after 2014.
Revised: We train models on all sequences (holding out 10% as a validation set), without including any obser- vations after 2014.

Original: When pretraining CAREER on resumes, we also make sure to only include examples up to 2014.
Revised: When pretraining CAREER on resumes, we also make sure to only include examples up to 2014.

Original: Table 2 compares the forecasting performance of all models.
Revised: Table 2 compares the forecasting performance of all models.

Original: CAREER makes the best overall forecasts.
Revised: CAREER makes the best overall forecasts.

Original: CAREER has a signiﬁcant advantage over baselines at making long-term forecasts, yielding a 17% advantage over the best baseline for 6-year forecasts on NLSY97.
Revised: CAREER has a signiﬁcant advantage over baselines at making long-term forecasts, yielding a 17% advantage over the best baseline for 6-year forecasts on NLSY97.

Original: Again, NEMO, which had been developed to model much larger resume corpora, struggles to make good predictions on these smaller survey datasets.
Revised: Again, NEMO, which had been developed to model much larger resume corpora, struggles to make good predictions on these smaller survey datasets.

Original: a) Test perplexity on survey datasets.
Revised: a) Test perplexity on survey datasets.

Original: Results are averaged over three random seeds.
Revised: Results are averaged over three random seeds.

Original: CAREER (vanilla) includes covariates but not two-stage prediction
Revised: CAREER (vanilla) includes covariates but not two-stage prediction

Original: or pretraining; CAREER (two-stage) adds two-stage prediction.
Revised: or pretraining; CAREER (two-stage) adds two-stage prediction.

Original: [Table] # Pretraining Examples 11.311.
Revised: 101010# Pretraining Examples 11.311.

Original: P e r p l e x i t y
Revised: P e r p l e x i t y

Original: No Pretraining (b) CAREER’s scaling law on
Revised: No Pretraining ( [Figure] CAREER’s scaling law on

Original: NLSY79 as
Revised: NLSY [Figure] function

Original: [Figure]
Revised: [Figure]

Revised: pre

Original: function of pre- training data volume.
Revised: [Figure] training [Figure] volume.

Original: [Table]
Revised: [Table]

Original: Distribution shifts in data types.
Revised: Distribution shifts in data types.

Revised: For pretraining, we use a dataset of 24 million resumes provided by Zippia, a career planning company.

Original: We consider three widely-used economic survey datasets: two cohorts from the National Longitudinal Survey of Youth (NLSY79 and NLSY97) and the Panel Study of Income Dynamics (PSID).
Revised: We consider three widely-used economic survey datasets: two cohorts from the National Longitudinal Survey of Youth (NLSY79 and NLSY97) and the Panel Study of Income Dynamics (PSID).

Original: These datasets have been carefully constructed to be representative of the general population, and they are widely used by economists for estimating important quantities.
Revised: These datasets have been carefully constructed to be representative of the general population, and they are widely used by economists for estimating important quantities.

Original: These survey datasets are considerably smaller than resumes.
Revised: These survey datasets are considerably smaller than resumes.

Original: Moreover, the distribution of job sequences in resumes diﬀers in meaningful ways from those in the survey datasets; for example, manual laborers are under-represented and college graduates are over-represented in resume data (see
Revised: Moreover, the distribution of job sequences in resumes diﬀers in meaningful ways from those in the survey datasets; for example, manual laborers are under-represented and college graduates are over-represented in resume data (see

Original: Appendix F for more details).
Revised: Appendix F for more details).

Original: We compare CAREER to several baseline models: a second-order linear regression with covariates and hand-constructed summary statistics about future employment; a bag-of-jobs model inspired by SHOPPER (Ruiz et al., 2020); and NEMO (Li et al., 2017), an LSTM-based method developed in the data mining community for modeling resumes.
Revised: We compare CAREER to several baseline models: a second-order linear regression with covariates and hand-constructed summary statistics about future employment; a bag-of-jobs model inspired by SHOPPER (Ruiz et al., 2020); and NEMO (Li et al., 2017), an LSTM-based method developed in the data mining community for modeling resumes.

Original: We evaluate the predictive performance of each model by computing held-out perplexity, a common metric in NLP for evaluating probabilistic sequence models.
Revised: We evaluate the predictive performance of each model by computing held-out perplexity, a common metric in NLP for evaluating probabilistic sequence models.

Original: See Appendix H for more experimental details.
Revised: See Appendix H for more experimental details.

Original: Figure 1a compares the test-set perplexity of each model.
Revised: Figure 1a compares the test-set perplexity of each model.

Original: With the transferred representations,
Revised: With the transferred representations,

Original: CAREER makes the best predictions on all survey datasets.
Revised: CAREER makes the best predictions on all survey datasets.

Original: Although the resume data is noisy and diﬀers in many ways from the survey datasets used for economic prediction, CAREER learns useful representations of work experiences that aid its predictive performance.
Revised: Although the resume data is noisy and diﬀers in many ways from the survey datasets used for economic prediction, CAREER learns useful representations of work experiences that aid its predictive performance.

Original: To assess how the volume of resumes used for pretraining aﬀects CAREER’s predictions on survey datasets, we downsample the resume dataset and transfer to survey datasets.
Revised: To assess how the volume of resumes used for pretraining aﬀects CAREER’s predictions on survey datasets, we downsample the resume dataset and transfer to survey datasets.

Original: The scaling law for NLSY79 is depicted in Figure 1b.
Revised: The scaling law for NLSY79 is depicted in Figure 1b.

Original: When there are less than 20,000 examples in the resume dataset, pretraining CAREER does not oﬀer any improvement.
Revised: When there are less than 20,000 examples in the resume dataset, pretraining CAREER does not oﬀer any improvement.

Original: Distribution shifts in time.
Revised: Distribution shifts in time.

Original: We also assess CAREER’s ability to forecast future career trajectories.
Revised: We also assess CAREER’s ability to forecast future career trajectories.

Original: In contrast to predicting held-out sequences, forecasting involves training models on all sequences before a speciﬁc year.
Revised: In contrast to predicting held-out sequences, forecasting involves training models on all sequences before a speciﬁc year.

Original: To predict future jobs for an individual, the ﬁtted model is used to estimate job probabilities six years into the future by sampling multi-year trajectories.
Revised: To predict future jobs for an individual, the ﬁtted model is used to estimate job probabilities six years into the future by sampling multi-year trajectories.

Original: This setting is useful for assessing a model’s ability to make long-term predictions, as occupational trends change over time.
Revised: This setting is useful for assessing a model’s ability to make long-term predictions, as occupational trends change over time.

Original: We evaluate CAREER’s forecasting abilities on NLSY97 and PSID.
Revised: We evaluate CAREER’s forecasting abilities on NLSY97 and PSID.

Original: (These datasets are more valuable for forecasting than NLSY79, which follows a cohort that is near or past retirement age.)
Revised: (These datasets are more valuable for forecasting than NLSY79, which follows a cohort that is near or past retirement age.)

Original: We train models on all sequences (holding out 10% as a validation set), without including any observations after 2014.
Revised: We train models on all sequences (holding out 10% as a validation set), without including any observations after 2014.

Original: When pretraining CAREER on resumes, we also make sure to only include examples up to 2014.
Revised: When pretraining CAREER on resumes, we also make sure to only include examples up to 2014.

Original: Table 1 compares the forecasting performance of all models.
Revised: Table 1 compares the forecasting performance of all models.

Original: CAREER makes the best overall forecasts.
Revised: CAREER makes the best overall forecasts.

Original: CAREER has a signiﬁcant advantage over baselines at making long-term forecasts, yielding a 17% advantage over the best baseline for 6-year forecasts on NLSY97.
Revised: CAREER has a signiﬁcant advantage over baselines at making long-term forecasts, yielding a 17% advantage over the best baseline for 6-year forecasts on NLSY97.

Original: A CAREER Details
Revised: A CAREER Details

Original: Transformers.
Revised: Transformers.

Original: A transformer is a sequence model that uses neural networks to learn representations of discrete tokens (Vaswani et al., 2017).
Revised: A transformer is a sequence model that uses neural networks to learn representations of discrete tokens (Vaswani et al., 2017).

Original: Transformers were originally developed for natural language processing (NLP), to predict words in a sentence.
Revised: Transformers were originally developed for natural language processing (NLP), to predict words in a sentence.

Original: Transformers are able to model complex dependencies between words, and they are a critical component of modern NLP systems including language modeling (Radford et al., 2019) and machine translation (Ott et al., 2018).
Revised: Transformers are able to model complex dependencies between words, and they are a critical component of modern NLP systems including language modeling (Radford et al., 2019) and machine translation (Ott et al., 2018).

Original: CAREER is an occupation model that uses a transformer to parameterize a low-dimensional representation of careers.
Revised: CAREER is an occupation model that uses a transformer to parameterize a low-dimensional representation of careers.

Original: While transformers were developed to model sequences of words, CAREER uses a transformer to model sequences of jobs.
Revised: While transformers were developed to model sequences of words, CAREER uses a transformer to model sequences of jobs.

Original: The transformer enables the model to represent complex career trajectories.
Revised: The transformer enables the model to represent complex career trajectories.

Original: CAREER is similar to the transformers used in NLP, but with two modiﬁcations.
Revised: CAREER is similar to the transformers used in NLP, but with two modiﬁcations.

Original: First, the model makes predictions in two stages, making it better-suited to model workers who stay in the same job through consecutive timesteps.
Revised: First, the model makes predictions in two stages, making it better-suited to model workers who stay in the same job through consecutive timesteps.

Original: (In contrast, words seldom repeat.)
Revised: (In contrast, words seldom repeat.)

Original: Second, while language models only condition on previous words, each career is also associated with covariates x that may aﬀect transition distributions (see Equation 2).
Revised: Second, while language models only condition on previous words, each career is also associated with covariates x that may aﬀect transition distributions (see Equation 2).

Original: We adapt the transformer to these two changes.
Revised: We adapt the transformer to these two changes.

Original: Parameterization.
Revised: Parameterization.

Original: CAREER’s computation graph is depicted in Figure 2.
Revised: CAREER’s computation graph is depicted in Figure 2.

Original: Note that in this section we provide a simpliﬁed description of the ideas underlying the transformer.
Revised: Note that in this section we provide a simpliﬁed description of the ideas underlying the transformer.

Original: Appendix E contains a full description of the model.
Revised: Appendix E contains a full description of the model.

Original: CAREER iteratively builds a representation of career history, h t ( y t − 1 , x t ) ∈ R D , using a stack of
Revised: CAREER iteratively builds a representation of career history, h t ( y t − 1 , x t ) ∈ R D , using a stack of

Original: L layers.
Revised: L layers.

Original: Each layer applies a series of computations to the previous layer’s output to produce its own layer-speciﬁc representation.
Revised: Each layer applies a series of computations to the previous layer’s output to produce its own layer-speciﬁc representation.

Original: The ﬁrst layer’s representation, h (1) t ( y t − 1 , x t ) , considers only the most recent job and covariates.
Revised: The ﬁrst layer’s representation, h (1) t ( y t − 1 , x t ) , considers only the most recent job and covariates.

Original: At each subsequent layer (cid:96) , the transformer forms a representation h ( (cid:96) )
Revised: At each subsequent layer (cid:96) , the transformer forms a representation h ( (cid:96) )

Original: t ( y t − 1 , x t ) by combining the representation of the most recent job with those of preceding jobs.
Revised: t ( y t − 1 , x t ) by combining the representation of the most recent job with those of preceding jobs.

Original: Representations become increasingly complex at each layer, and the ﬁnal layer’s representation, h ( L ) t ( y t − 1 , x t ) , is used to make predictions following Equations 1 to 3.
Revised: Representations become increasingly complex at each layer, and the ﬁnal layer’s representation, h ( L ) t ( y t − 1 , x t ) , is used to make predictions following Equations 1 to 3.

Original: We drop the explicit dependence on y t − 1 and x t going forward, and instead denote each layer’s representation as h ( (cid:96) ) t .
Revised: We drop the explicit dependence on y t − 1 and x t going forward, and instead denote each layer’s representation as h ( (cid:96) ) t .

Original: The ﬁrst layer’s representation combines the previous job, the most recent covariates, and the position of the job in the career.
Revised: The ﬁrst layer’s representation combines the previous job, the most recent covariates, and the position of the job in the career.

Original: It ﬁrst embeds each of these variables in D -dimensional space.
Revised: It ﬁrst embeds each of these variables in D -dimensional space.

Original: Deﬁne an embedding function for occupations, e y :
Revised: Deﬁne an embedding function for occupations, e y :

Original: [ J ] → R D .
Revised: [ J ] → R D .

Original: Additionally, deﬁne a separate embedding function for each covariate, { e c } Cc =1 , with each e c : [ N c ] → R D .
Revised: Additionally, deﬁne a separate embedding function for each covariate, { e c } Cc =1 , with each e c : [ N c ] → R D .

Original: Finally, deﬁne e t : [ T ] → R D to embed the position of the sequence, where T denotes the number of possible sequence lengths.
Revised: Finally, deﬁne e t : [ T ] → R D to embed the position of the sequence, where T denotes the number of possible sequence lengths.

Original: The ﬁrst-layer representation h
Revised: The ﬁrst-layer representation h

Original: [Equation]
Revised: [Equation]

Original: 1) t sums these embeddings
Revised: 1) t sums these embeddings

Original: [Equation]
Revised: [Equation]

Original: For each subsequent layer (cid:96) , the transformer combines representations of the most recent job with those of the preceding jobs and passes them through a neural network:
Revised: For each subsequent layer (cid:96) , the transformer combines representations of the most recent job with those of the preceding jobs and passes them through a neural network:

Original: [Equation]
Revised: [Equation]

Original: where W ( (cid:96) ) ∈ R D
Revised: where W ( (cid:96) ) ∈ R D

Original: × D is a model parameter and FFN ( (cid:96) ) is a two-layer feedforward neural network speciﬁc to layer (cid:96) , with FFN ( (cid:96) ) :
Revised: × D is a model parameter and FFN ( (cid:96) ) is a two-layer feedforward neural network speciﬁc to layer (cid:96) , with FFN ( (cid:96) ) :

Original: R D → R D .
Revised: R D → R D .

Original: The weights { π ( (cid:96) ) t,t (cid:48) } are referred to as attention weights , and they are determined by the career representations and W ( (cid:96) ) .
Revised: The weights { π ( (cid:96) ) t,t (cid:48) } are referred to as attention weights , and they are determined by the career representations and W ( (cid:96) ) .

Original: The attention weights are non-negative and normalized to sum to 1.
Revised: The attention weights are non-negative and normalized to sum to 1.

Original: The matrix W ( (cid:96) ) can be interpreted as a similarity matrix; if W ( (cid:96) ) is the identity matrix, occupations t and t (cid:48) that have similar representations will have large attention weights, and thus t (cid:48) would contribute more to the weighted average in Equation 6.
Revised: The matrix W ( (cid:96) ) can be interpreted as a similarity matrix; if W ( (cid:96) ) is the identity matrix, occupations t and t (cid:48) that have similar representations will have large attention weights, and thus t (cid:48) would contribute more to the weighted average in Equation 6.

Original: Conversely, if W ( (cid:96) ) is the negative identity matrix, occupations that have diﬀering representations will have large attention weights.
Revised: Conversely, if W ( (cid:96) ) is the negative identity matrix, occupations that have diﬀering representations will have large attention weights.

Original: The ﬁnal computation of each layer involves passing the intermediate representation ˜ h
Revised: The ﬁnal computation of each layer involves passing the intermediate representation ˜ h

Original: ( (cid:96) ) t through a neural network, which ensures that representations capture complex nonlinear interactions.
Revised: ( (cid:96) ) t through a neural network, which ensures that representations capture complex nonlinear interactions.

Original: The computations in Equations 5 to 7 are repeated for each of the L layers.
Revised: The computations in Equations 5 to 7 are repeated for each of the L layers.

Original: The last layer’s representation is used to predict the next job:
Revised: The last layer’s representation is used to predict the next job:

Original: [Equation]
Revised: [Equation]

Original: where “two-stage-softmax” refers to the operation in Equations 1 to 3, parameterized by η and β .
Revised: where “two-stage-softmax” refers to the operation in Equations 1 to 3, parameterized by η and β .

Original: All of CAREER’s parameters – including the embedding functions, similarity matrices, feed-forward neural networks, and regression coeﬃcients η and β – are estimated by maximizing the likelihood in
Revised: All of CAREER’s parameters – including the embedding functions, similarity matrices, feed-forward neural networks, and regression coeﬃcients η and β – are estimated by maximizing the likelihood in

Original: Equation 8 with stochastic gradient descent (SGD), marginalizing out the variable s t .
Revised: Equation 8 with stochastic gradient descent (SGD), marginalizing out the variable s t .

Original: B Baseline Models
Revised: B Baseline Models

Original: In this section, we describe baseline occupation models that economists have used to model jobs and other discrete sequences.
Revised: In this section, we describe baseline occupation models that economists have used to model jobs and other discrete sequences.

Original: Markov models and regression.
Revised: Markov models and regression.

Original: A ﬁrst-order Markov model assumes the job at each timestep depends on only the previous job (Hall, 1972; Poterba & Summers, 1986).
Revised: A ﬁrst-order Markov model assumes the job at each timestep depends on only the previous job (Hall, 1972; Poterba & Summers, 1986).

Original: Without covariates, a
Revised: Without covariates, a

Original: Markov model takes the form p ( y t =
Revised: Markov model takes the form p ( y t =

Original: j | y t − 1 ) =
Revised: j | y t − 1 ) =

Original: p ( y t =
Revised: p ( y t =

Original: j | y t − 1 ) .
Revised: j | y t − 1 ) .

Original: The optimal transition probabilities reﬂect the overall frequencies of individuals transitioning from occupation y t − 1 to occupation j .
Revised: The optimal transition probabilities reﬂect the overall frequencies of individuals transitioning from occupation y t − 1 to occupation j .

Original: In a second-order Markov model, the next job depends on the previous two.
Revised: In a second-order Markov model, the next job depends on the previous two.

Original: A multinomial logistic regression can be used to incorporate covariates:
Revised: A multinomial logistic regression can be used to incorporate covariates:

Original: [Equation]
Revised: [Equation]

Original: where β (0) j is an occupation-speciﬁc intercept and y t − 1 and x tc denote J - and N c -dimensional indicator vectors, respectively.
Revised: where β (0) j is an occupation-speciﬁc intercept and y t − 1 and x tc denote J - and N c -dimensional indicator vectors, respectively.

Original: Equation 9 depends on history only through the most recent job, although the covariates can also include hand-crafted summary statistics about the past, such as the duration of the most recent job (McCall, 1990).
Revised: Equation 9 depends on history only through the most recent job, although the covariates can also include hand-crafted summary statistics about the past, such as the duration of the most recent job (McCall, 1990).

Original: This model is ﬁt by maximizing the likelihood with gradient-based methods.
Revised: This model is ﬁt by maximizing the likelihood with gradient-based methods.

Original: Bag-of-jobs.
Revised: Bag-of-jobs.

Original: A weakness of the ﬁrst-order Markov model is that it only uses the most recent job to make predictions.
Revised: A weakness of the ﬁrst-order Markov model is that it only uses the most recent job to make predictions.

Original: However, one’s working history beyond the last job may inform future transitions (Blau & Riphahn, 1999; Neal, 1999).
Revised: However, one’s working history beyond the last job may inform future transitions (Blau & Riphahn, 1999; Neal, 1999).

Original: Another baseline we consider is a bag-of-jobs model, inspired by SHOPPER, a probabilistic model of consumer choice (Ruiz et al., 2020).
Revised: Another baseline we consider is a bag-of-jobs model, inspired by SHOPPER, a probabilistic model of consumer choice (Ruiz et al., 2020).

Original: Unlike the Markov and regression models, the bag-of-jobs model conditions on every job in an individual’s history.
Revised: Unlike the Markov and regression models, the bag-of-jobs model conditions on every job in an individual’s history.

Original: It does so by learning a low-dimensional representation of an individual’s history.
Revised: It does so by learning a low-dimensional representation of an individual’s history.

Original: This model learns a unique embedding for each occupation, similar to a word embedding (Bengio et al., 2003; Mikolov et al., 2013); unlike CAREER, which learns complicated nonlinear interactions between jobs in a history, the bag-of-jobs model combines jobs into a single representation by averaging their embeddings.
Revised: This model learns a unique embedding for each occupation, similar to a word embedding (Bengio et al., 2003; Mikolov et al., 2013); unlike CAREER, which learns complicated nonlinear interactions between jobs in a history, the bag-of-jobs model combines jobs into a single representation by averaging their embeddings.

Original: The bag-of-jobs model assumes that job transitions depend on two terms: a term that captures the eﬀect of the most recent job, and a term that captures the eﬀect of all prior jobs.
Revised: The bag-of-jobs model assumes that job transitions depend on two terms: a term that captures the eﬀect of the most recent job, and a term that captures the eﬀect of all prior jobs.

Original: Accordingly, the model learns two types of representations: an embedding α j ∈ R D of the most recent job j , and an embedding ρ j (cid:48) ∈ R D for prior jobs j (cid:48) .
Revised: Accordingly, the model learns two types of representations: an embedding α j ∈ R D of the most recent job j , and an embedding ρ j (cid:48) ∈ R D for prior jobs j (cid:48) .

Original: To combine the representations for all prior jobs into a single term, the model averages embeddings:
Revised: To combine the representations for all prior jobs into a single term, the model averages embeddings:

Original: [Equation]
Revised: [Equation]

Original: Covariates can be added to the model analogously; for a single covariate, its most recent value is embedded and summed with the average embeddings for its prior values.
Revised: Covariates can be added to the model analogously; for a single covariate, its most recent value is embedded and summed with the average embeddings for its prior values.

Original: All parameters are estimated by maximizing the likelihood in Equation 10 with SGD.
Revised: All parameters are estimated by maximizing the likelihood in Equation 10 with SGD.

Original: C Related Work
Revised: C Related Work

Original: Many economic analyses use log-linear models to predict jobs in survey datasets (Boskin, 1974;
Revised: Many economic analyses use log-linear models to predict jobs in survey datasets (Boskin, 1974;

Original: Schmidt & Strauss, 1975).
Revised: Schmidt & Strauss, 1975).

Original: These models typically use small state spaces consisting of only a few occupation categories.
Revised: These models typically use small state spaces consisting of only a few occupation categories.

Original: For example, some studies categorize occupations into broad skill groups (Keane & Wolpin, 1997; Cortes, 2016); unemployment analyses only consider employment status (employed, unemployed, and out-of-labor-force)
Revised: For example, some studies categorize occupations into broad skill groups (Keane & Wolpin, 1997; Cortes, 2016); unemployment analyses only consider employment status (employed, unemployed, and out-of-labor-force)

Original: (Hall, 1972; Lauerova & Terrell, 2007); and researchers studying occupational mobility only consider occupational change, a binary variable indicating whether an individual changes jobs (Kambourov & Manovskii, 2008; Guvenen et al., 2020).
Revised: (Hall, 1972; Lauerova & Terrell, 2007); and researchers studying occupational mobility only consider occupational change, a binary variable indicating whether an individual changes jobs (Kambourov & Manovskii, 2008; Guvenen et al., 2020).

Original: Although transitions between occupations may depend richly on history, many of these models condition on only the most recent job and a few manually constructed summary statistics about history to make predictions (Hall, 1972; Blau & Riphahn, 1999).
Revised: Although transitions between occupations may depend richly on history, many of these models condition on only the most recent job and a few manually constructed summary statistics about history to make predictions (Hall, 1972; Blau & Riphahn, 1999).

Original: In contrast to these methods, CAREER is nonlinear and conditions on every job in an individual’s history.
Revised: In contrast to these methods, CAREER is nonlinear and conditions on every job in an individual’s history.

Original: The model learns complex representations of careers without relying on manually constructed features.
Revised: The model learns complex representations of careers without relying on manually constructed features.

Original: Moreover, CAREER can eﬀectively predict from among hundreds of occupations.
Revised: Moreover, CAREER can eﬀectively predict from among hundreds of occupations.

Original: Recently, the proliferation of business networking platforms has resulted in the availability of large resume datasets.
Revised: Recently, the proliferation of business networking platforms has resulted in the availability of large resume datasets.

Original: Schubert et al.
Revised: Schubert et al.

Original: (2021) use a large resume dataset to construct a ﬁrst-order Markov model of job transitions; CAREER, which conditions on all jobs in a history, makes more accurate predictions than a Markov model.
Revised: (2021) use a large resume dataset to construct a ﬁrst-order Markov model of job transitions; CAREER, which conditions on all jobs in a history, makes more accurate predictions than a Markov model.

Original: Models developed in the data mining community rely on resumespeciﬁc features such as stock prices (Xu et al., 2018), worker skill (Ghosh et al., 2020), network information (Meng et al., 2019; Zhang et al., 2021), and textual descriptions (He et al., 2021), and are not applicable to survey datasets, as is our goal in this paper (other models reduce to a ﬁrst-order
Revised: Models developed in the data mining community rely on resumespeciﬁc features such as stock prices (Xu et al., 2018), worker skill (Ghosh et al., 2020), network information (Meng et al., 2019; Zhang et al., 2021), and textual descriptions (He et al., 2021), and are not applicable to survey datasets, as is our goal in this paper (other models reduce to a ﬁrst-order

Original: Markov model without these features (Dave et al., 2018; Zhang et al., 2019)).
Revised: Markov model without these features (Dave et al., 2018; Zhang et al., 2019)).

Original: The most suitable model for survey datasets from this line of work is NEMO, an LSTM-based model that is trained on large resume datasets (Li et al., 2017).
Revised: The most suitable model for survey datasets from this line of work is NEMO, an LSTM-based model that is trained on large resume datasets (Li et al., 2017).

Original: Our experiments demonstrate that CAREER outperforms
Revised: Our experiments demonstrate that CAREER outperforms

Original: NEMO when it is adapted to model survey datasets.
Revised: NEMO when it is adapted to model survey datasets.

Original: Recent works in econometrics have applied machine learning methods to sequences of jobs and other discrete data.
Revised: Recent works in econometrics have applied machine learning methods to sequences of jobs and other discrete data.

Original: Ruiz et al.
Revised: Ruiz et al.

Original: (2020) develop a matrix factorization method called SHOPPER to model supermarket basket data.
Revised: (2020) develop a matrix factorization method called SHOPPER to model supermarket basket data.

Original: We consider a baseline “bag-of-jobs” model similar to SHOPPER.
Revised: We consider a baseline “bag-of-jobs” model similar to SHOPPER.

Original: Like the transformer-based model, the bag-of-jobs model conditions on every job in an individual’s history, but it uses relatively simple representations of careers.
Revised: Like the transformer-based model, the bag-of-jobs model conditions on every job in an individual’s history, but it uses relatively simple representations of careers.

Original: Our empirical studies demonstrate that CAREER learns complex representations that are better at modeling job sequences.
Revised: Our empirical studies demonstrate that CAREER learns complex representations that are better at modeling job sequences.

Original: Rajkumar et al.
Revised: Rajkumar et al.

Original: (2021) build on SHOPPER and propose a Bayesian factorization method for predicting job transitions.
Revised: (2021) build on SHOPPER and propose a Bayesian factorization method for predicting job transitions.

Original: Similar to
Revised: Similar to

Original: CAREER, they predict jobs in two stages.
Revised: CAREER, they predict jobs in two stages.

Original: However, their method is focused on modeling individual transitions, so it only conditions on the most recent job in an individual’s history.
Revised: However, their method is focused on modeling individual transitions, so it only conditions on the most recent job in an individual’s history.

Original: In our empirical studies, we show that models like CAREER that condition on every job in an individual’s history form more accurate predictions than Markov models.
Revised: In our empirical studies, we show that models like CAREER that condition on every job in an individual’s history form more accurate predictions than Markov models.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: CAREER is based on a transformer, a successful model for representing sequences of words in natural language processing (NLP).
Revised: CAREER is based on a transformer, a successful model for representing sequences of words in natural language processing (NLP).

Original: In econometrics, transformers have been applied to the text of job descriptions to predict their salaries (Bana, 2021) or authenticity
Revised: In econometrics, transformers have been applied to the text of job descriptions to predict their salaries (Bana, 2021) or authenticity

Original: (Naudé et al., 2022); rather than modeling text, we use transformers to model sequences of occupations.
Revised: (Naudé et al., 2022); rather than modeling text, we use transformers to model sequences of occupations.

Original: Transformers have also been applied successfully to sequences other than text: images (Dosovitskiy et al., 2021), music (Huang et al., 2019), and molecular chemistry (Schwaller et al., 2019).
Revised: Transformers have also been applied successfully to sequences other than text: images (Dosovitskiy et al., 2021), music (Huang et al., 2019), and molecular chemistry (Schwaller et al., 2019).

Original: Inspired by their success in modeling a variety of complex discrete sequential distributions, this paper adapts transformers to modeling sequences of jobs.
Revised: Inspired by their success in modeling a variety of complex discrete sequential distributions, this paper adapts transformers to modeling sequences of jobs.

Original: Transformers are especially adept at learning transferrable representations of text from large corpora (Radford et al., 2018;
Revised: Transformers are especially adept at learning transferrable representations of text from large corpora (Radford et al., 2018;

Original: Devlin et al., 2019).
Revised: Devlin et al., 2019).

Original: We show that CAREER learns representations of job sequences that can be transferred from noisy resume datasets to smaller, well-curated administrative datasets.
Revised: We show that CAREER learns representations of job sequences that can be transferred from noisy resume datasets to smaller, well-curated administrative datasets.

Original: D Qualitative Analysis
Revised: D Qualitative Analysis

Original: Rationalizing predictions.
Revised: Rationalizing predictions.

Original: Figure 3 shows an example of a held-out career sequence from PSID.
Revised: Figure 3 shows an example of a held-out career sequence from PSID.

Original: CAREER is much likelier than a regression and bag-of-jobs baseline to predict this individual’s next job, biological technician.
Revised: CAREER is much likelier than a regression and bag-of-jobs baseline to predict this individual’s next job, biological technician.

Original: To understand CAREER’s prediction, we show the model’s rationale , or the jobs in this individual’s history that are suﬃcient for explaining the model’s prediction.
Revised: To understand CAREER’s prediction, we show the model’s rationale , or the jobs in this individual’s history that are suﬃcient for explaining the model’s prediction.

Original: (We adapt the greedy rationalization method from Vafa et al.
Revised: (We adapt the greedy rationalization method from Vafa et al.

Original: (2021); refer to Appendix H for more details.)
Revised: (2021); refer to Appendix H for more details.)

Original: In this example, CAREER only needs three previous jobs to predict biological technician: animal caretaker, engineering technician, and student.
Revised: In this example, CAREER only needs three previous jobs to predict biological technician: animal caretaker, engineering technician, and student.

Original: The model can combine latent attributes of each job to predict the individual’s next job.
Revised: The model can combine latent attributes of each job to predict the individual’s next job.

Original: Representation similarity.
Revised: Representation similarity.

Original: To demonstrate the quality of the learned representations, we use
Revised: To demonstrate the quality of the learned representations, we use

Original: CAREER’s ﬁne-tuned representations on NLSY97 to ﬁnd pairs of individuals with the most similar career trajectories.
Revised: CAREER’s ﬁne-tuned representations on NLSY97 to ﬁnd pairs of individuals with the most similar career trajectories.

Original: Speciﬁcally, we compute CAREEER’s representation h t ( y t − 1 , x t ) for each individual in NLSY97 who has worked for four years.
Revised: Speciﬁcally, we compute CAREEER’s representation h t ( y t − 1 , x t ) for each individual in NLSY97 who has worked for four years.

Original: We then measure the similarity between all pairs by computing the cosine similarity between representations.
Revised: We then measure the similarity between all pairs by computing the cosine similarity between representations.

Original: In order to depict meaningful matches, we only consider pairs of individuals with no overlapping jobs in their histories (otherwise the model would ﬁnd individuals with the exact same career trajectories).
Revised: In order to depict meaningful matches, we only consider pairs of individuals with no overlapping jobs in their histories (otherwise the model would ﬁnd individuals with the exact same career trajectories).

Original: Figure 4 depicts the career histories with the most similar CAREER representations.
Revised: Figure 4 depicts the career histories with the most similar CAREER representations.

Original: Although none of these pairs have overlapping jobs, the model learns representations that can identify similar careers.
Revised: Although none of these pairs have overlapping jobs, the model learns representations that can identify similar careers.

Original: E Transformer Details
Revised: E Transformer Details

Original: In this section, we expand on the simpliﬁed description of transformers in Section 2 and describe CAREER in full detail.
Revised: In this section, we expand on the simpliﬁed description of transformers in Section 2 and describe CAREER in full detail.

Original: Recall that the model estimates representations in L layers, h (1) t ( y t − 1 , x t ) , . .
Revised: Recall that the model estimates representations in L layers, h (1) t ( y t − 1 , x t ) , . .

Original: , h ( L ) t ( y t − 1 , x t ) , with each representation h ( (cid:96) )
Revised: , h ( L ) t ( y t − 1 , x t ) , with each representation h ( (cid:96) )

Original: t ∈ R D .
Revised: t ∈ R D .

Original: The ﬁnal representation h ( L ) t ( y t − 1 , x t ) is used to represent careers.
Revised: The ﬁnal representation h ( L ) t ( y t − 1 , x t ) is used to represent careers.

Original: We drop the explicit dependence on y t − 1 and x t , and instead denote each representation as h ( (cid:96) ) t .
Revised: We drop the explicit dependence on y t − 1 and x t , and instead denote each representation as h ( (cid:96) ) t .

Original: The ﬁrst transformer layer combines the previous occupation, the most recent covariates, and the position of the occupation in the career.
Revised: The ﬁrst transformer layer combines the previous occupation, the most recent covariates, and the position of the occupation in the career.

Original: It ﬁrst embeds each of these variables in D -dimensional space.
Revised: It ﬁrst embeds each of these variables in D -dimensional space.

Original: Deﬁne an embedding function for occupations, e y :
Revised: Deﬁne an embedding function for occupations, e y :

Original: [ J ] → R D .
Revised: [ J ] → R D .

Original: Additionally, deﬁne a separate embedding function for each covariate, { e c } Cc =1 , with each e c : [ N c ] → R D .
Revised: Additionally, deﬁne a separate embedding function for each covariate, { e c } Cc =1 , with each e c : [ N c ] → R D .

Original: Finally, deﬁne e t : [ T ] → R D to embed the position of the sequence, where T denotes the number of possible sequence lengths.
Revised: Finally, deﬁne e t : [ T ] → R D to embed the position of the sequence, where T denotes the number of possible sequence lengths.

Original: The ﬁrst-layer representation h (1) t sums these embeddings:
Revised: The ﬁrst-layer representation h (1) t sums these embeddings:

Original: [Equation]
Revised: [Equation]

Original: The occupation- and covariate-speciﬁc embeddings, e y and { e c } , are model parameters; the positional embeddings, e t , are set in advance to follow a sinusoidal pattern (Vaswani et al., 2017).
Revised: The occupation- and covariate-speciﬁc embeddings, e y and { e c } , are model parameters; the positional embeddings, e t , are set in advance to follow a sinusoidal pattern (Vaswani et al., 2017).

Original: While these embeddings could also be parameterized, in practice the performance is similar, and using sinusoidal embeddings allows the model to generalize to career sequence lengths unseen in the training data.
Revised: While these embeddings could also be parameterized, in practice the performance is similar, and using sinusoidal embeddings allows the model to generalize to career sequence lengths unseen in the training data.

Original: At each subsequent layer, the transformer combines the representations of all occupations in a history.
Revised: At each subsequent layer, the transformer combines the representations of all occupations in a history.

Original: It combines representations by performing multi-headed attention , which is similar to the process described in Section 2 albeit with multiple attention weights per layer.
Revised: It combines representations by performing multi-headed attention , which is similar to the process described in Section 2 albeit with multiple attention weights per layer.

Original: Speciﬁcally, it uses A speciﬁc attention weights, or heads , per layer.
Revised: Speciﬁcally, it uses A speciﬁc attention weights, or heads , per layer.

Original: The number of heads A should be less than the representation dimension D .
Revised: The number of heads A should be less than the representation dimension D .

Original: (Using A = 1 attention head reduces to the process described in Equations 5 and 6.)
Revised: (Using A = 1 attention head reduces to the process described in Equations 5 and 6.)

Original: The representation dimension D should be divisible by A ; denote
Revised: The representation dimension D should be divisible by A ; denote

Original: K = D/A .
Revised: K = D/A .

Original: First, A diﬀerent sets of attention weights are computed:
Revised: First, A diﬀerent sets of attention weights are computed:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where W ( (cid:96) )
Revised: where W ( (cid:96) )

Original: a ∈ R D
Revised: a ∈ R D

Original: × D is a model parameter, speciﬁc to attention head a and layer l . 

Revised: × D is a model parameter, speciﬁc to attention head a and layer l .

Original: Each attention head forms a convex combination with all previous representations; to diﬀerentiate between attention heads, each representation is transformed by a linear transformation V ( (cid:96) ) a ∈ R K × D unique to an attention head, forming b ( (cid:96) ) a,t ∈ R K :
Revised: Each attention head forms a convex combination with all previous representations; to diﬀerentiate between attention heads, each representation is transformed by a linear transformation V ( (cid:96) ) a ∈ R K × D unique to an attention head, forming b ( (cid:96) ) a,t ∈ R K :

Original: [Equation]
Revised: [Equation]

Original: All attention heads are combined into a single representation by concatenating them into a single vector g ( (cid:96) )
Revised: All attention heads are combined into a single representation by concatenating them into a single vector g ( (cid:96) )

Original: t
Revised: t

Original: [Equation]
Revised: [Equation]

Original: To complete the multi-head attention step and form the intermediate representation ˜ h ( (cid:96) )
Revised: To complete the multi-head attention step and form the intermediate representation ˜ h ( (cid:96) )

Original: t , the concatenated representations g ( (cid:96) )
Revised: t , the concatenated representations g ( (cid:96) )

Original: t undergo a linear transformation and are summed with the pre-attention representation h ( (cid:96) ) t :
Revised: t undergo a linear transformation and are summed with the pre-attention representation h ( (cid:96) ) t :

Original: [Equation]
Revised: [Equation]

Original: with M
Revised: with M

Original: (cid:96)
Revised: (cid:96)

Original: [Table]
Revised: [Table]

Original: intermediate representations
Revised: intermediate representations

Original: ∈ R D combine the representation at timestep t with those preceding timestep t .
Revised: ∈ R D combine the representation at timestep t with those preceding timestep t .

Original: Each layer of the transformer concludes by taking a non-linear transformation of the intermediate representations.
Revised: Each layer of the transformer concludes by taking a non-linear transformation of the intermediate representations.

Original: This non-linear transformation does not depend on any previous representation; it only transforms ˜ h
Revised: This non-linear transformation does not depend on any previous representation; it only transforms ˜ h

Original: ( (cid:96) ) t .
Revised: ( (cid:96) ) t .

Original: Speciﬁcally, ˜ h
Revised: Speciﬁcally, ˜ h

Original: [Equation]
Revised: [Equation]

Original: ) t is passed through a neural network:
Revised: ) t is passed through a neural network:

Original: [Equation]
Revised: [Equation]

Original: (16) where FFN ( (cid:96) ) denotes a two-layer feedforward neural network with N hidden units, with FFN ( (cid:96) ) :
Revised: (16) where FFN ( (cid:96) ) denotes a two-layer feedforward neural network with N hidden units, with FFN ( (cid:96) ) :

Original: R D → R D .
Revised: R D → R D .

Original: We repeat the multi-head attention and feedforward neural network updates above for L layers, using parameters unique to each layer.
Revised: We repeat the multi-head attention and feedforward neural network updates above for L layers, using parameters unique to each layer.

Original: We represent careers with the last-layer representation, h t ( y t − 1 , x t ) =
Revised: We represent careers with the last-layer representation, h t ( y t − 1 , x t ) =

Original: h ( L ) t ( y t − 1 , x t ) .
Revised: h ( L ) t ( y t − 1 , x t ) .

Original: For our experiments, we use model speciﬁcations similar to the generative pretrained transformer (GPT) architecture (Radford et al., 2018).
Revised: For our experiments, we use model speciﬁcations similar to the generative pretrained transformer (GPT) architecture (Radford et al., 2018).

Original: In particular, we use L = 12 layers, a representation dimension of D = 192 , A = 3 attention heads, and N = 768 hidden units and the GELU nonlinearity (Hendrycks & Gimpel, 2016) for all feedforward neural networks.
Revised: In particular, we use L = 12 layers, a representation dimension of D = 192 , A = 3 attention heads, and N = 768 hidden units and the GELU nonlinearity (Hendrycks & Gimpel, 2016) for all feedforward neural networks.

Original: In total, this results in 5.6 million parameters.
Revised: In total, this results in 5.6 million parameters.

Original: This model includes a few extra modiﬁcations to improve training: we use 0 .
Revised: This model includes a few extra modiﬁcations to improve training: we use 0 .

Original: 1 dropout (Srivastava et al., 2014) for the feedforward neural network weights, and 0 .
Revised: 1 dropout (Srivastava et al., 2014) for the feedforward neural network weights, and 0 .

Original: 1 dropout for the attention weights.
Revised: 1 dropout for the attention weights.

Original: Finally, we use layer normalization (Ba et al., 2016) before the updates in Equation 12, after the update in Equation 15, and after the ﬁnal layer’s neural network update in Equation 16.
Revised: Finally, we use layer normalization (Ba et al., 2016) before the updates in Equation 12, after the update in Equation 15, and after the ﬁnal layer’s neural network update in Equation 16.

Original: F Exploratory Data Analysis
Revised: F Exploratory Data Analysis

Original: Table 2 depicts summary statistics of the resume dataset provided by Zippia that is used for pretraining
Revised: Table 2 depicts summary statistics of the resume dataset provided by Zippia that is used for pretraining

Original: CAREER.
Revised: CAREER.

Original: Table 3 compares this resume dataset with the longitudinal survey datasets of interest.
Revised: Table 3 compares this resume dataset with the longitudinal survey datasets of interest.

Original: G Data Preprocessing
Revised: G Data Preprocessing

Original: In this section, we go over the data preprocessing steps we took for each dataset.
Revised: In this section, we go over the data preprocessing steps we took for each dataset.

Original: Resumes.
Revised: Resumes.

Original: We were given access to a large dataset of resumes of American workers by Zippia, a career planning company.
Revised: We were given access to a large dataset of resumes of American workers by Zippia, a career planning company.

Original: This dataset coded each occupation into one of 1,073 O*NET 2010 Standard
Revised: This dataset coded each occupation into one of 1,073 O*NET 2010 Standard

Original: Occupational Classiﬁcation (SOC) categories based on the provided job titles and descriptions in resumes.
Revised: Occupational Classiﬁcation (SOC) categories based on the provided job titles and descriptions in resumes.

Original: We dropped all examples with missing SOC codes.
Revised: We dropped all examples with missing SOC codes.

Original: [Table]
Revised: [Table]

Original: Each resume in the dataset we were given contained covariates that had been imputed based oﬀ other data in the resume.
Revised: Each resume in the dataset we were given contained covariates that had been imputed based oﬀ other data in the resume.

Original: We considered three covariates: year, most recent educational degree, and location.
Revised: We considered three covariates: year, most recent educational degree, and location.

Original: Education degrees had been encoded into one of eight categories: high school diploma, associate, bachelors, masters, doctorate, certiﬁcate, license, and diploma.
Revised: Education degrees had been encoded into one of eight categories: high school diploma, associate, bachelors, masters, doctorate, certiﬁcate, license, and diploma.

Original: Location had been encoded into one of 50 states plus Puerto Rico, Washington D.C., and unknown, for when location could not be imputed.
Revised: Location had been encoded into one of 50 states plus Puerto Rico, Washington D.C., and unknown, for when location could not be imputed.

Original: Some covariates also had missing entries.
Revised: Some covariates also had missing entries.

Original: When an occupation’s year was missing, we had to drop it from the dataset, because we could not position it in an individual’s career.
Revised: When an occupation’s year was missing, we had to drop it from the dataset, because we could not position it in an individual’s career.

Original: Whenever another covariate was missing, we replaced it with a special “missing” token.
Revised: Whenever another covariate was missing, we replaced it with a special “missing” token.

Original: All personally identiﬁable information had been removed from the dataset.
Revised: All personally identiﬁable information had been removed from the dataset.

Original: We transformed each resume in the dataset into a sequence of occupations.
Revised: We transformed each resume in the dataset into a sequence of occupations.

Original: We included an entry for each year starting from the ﬁrst year an individual worked to their last year.
Revised: We included an entry for each year starting from the ﬁrst year an individual worked to their last year.

Original: We included a special “beginning of sequence” token to indicate when each individual’s sequence started.
Revised: We included a special “beginning of sequence” token to indicate when each individual’s sequence started.

Original: For each year between an individual’s ﬁrst and last year, we added the occupation they worked in during that year.
Revised: For each year between an individual’s ﬁrst and last year, we added the occupation they worked in during that year.

Original: If an individual worked in multiple occupations in a year, we took the one where the individual spent more time in that year; if they were both the same amount of time in the particular year, we broke ties by adding the occupation that had started earlier in the career.
Revised: If an individual worked in multiple occupations in a year, we took the one where the individual spent more time in that year; if they were both the same amount of time in the particular year, we broke ties by adding the occupation that had started earlier in the career.

Original: For the experiments predicting future jobs directly on resumes, we added a “no-observed-occupation” token for years where the resume did not list any occupations (we dropped this token when pretraining).
Revised: For the experiments predicting future jobs directly on resumes, we added a “no-observed-occupation” token for years where the resume did not list any occupations (we dropped this token when pretraining).

Original: Each occupation was associated with the individual’s most recent educational degree, which we treated as a dynamic covariate.
Revised: Each occupation was associated with the individual’s most recent educational degree, which we treated as a dynamic covariate.

Original: The year an occupation took place was also considered a dynamic categorical covariate.
Revised: The year an occupation took place was also considered a dynamic categorical covariate.

Original: We treated location as static.
Revised: We treated location as static.

Original: In total, this preprocessing left us with a dataset of 23.7 million resumes, and 245 million individual occupations.
Revised: In total, this preprocessing left us with a dataset of 23.7 million resumes, and 245 million individual occupations.

Original: In order to transfer representations, we had to slightly modify the resumes dataset for pretraining to encode occupations and covariates into a format compatible with the survey datasets.
Revised: In order to transfer representations, we had to slightly modify the resumes dataset for pretraining to encode occupations and covariates into a format compatible with the survey datasets.

Original: The survey datasets we used were encoded with the “occ1990dd” occupation code (Autor & Dorn, 2013) rather than with O*NET’s SOC codes, so we converted the SOC codes to occ1990dd codes using a crosswalk posted online by Destin Royer.
Revised: The survey datasets we used were encoded with the “occ1990dd” occupation code (Autor & Dorn, 2013) rather than with O*NET’s SOC codes, so we converted the SOC codes to occ1990dd codes using a crosswalk posted online by Destin Royer.

Original: Even after we manually added a few missing entries to the crosswalks, there were some SOC codes that did not have corresponding occ1990dd’s.
Revised: Even after we manually added a few missing entries to the crosswalks, there were some SOC codes that did not have corresponding occ1990dd’s.

Original: We gave these tokens special codes that were not used when ﬁne-tuning on the survey datasets (because they did not correspond to occ1990dd occupations).
Revised: We gave these tokens special codes that were not used when ﬁne-tuning on the survey datasets (because they did not correspond to occ1990dd occupations).

Original: When an individual did not work for a given year, the survey datasets diﬀerentiated between three possible states: unemployed, out-of-labor-force, and in-school.
Revised: When an individual did not work for a given year, the survey datasets diﬀerentiated between three possible states: unemployed, out-of-labor-force, and in-school.

Original: The resumes dataset did not have these categories.
Revised: The resumes dataset did not have these categories.

Original: Thus, we initialized parameters for these three new occupational states randomly.
Revised: Thus, we initialized parameters for these three new occupational states randomly.

Original: Additionally, we did not include the “no-observed-occupation” token when pretraining, and instead dropped missing years from the sequence.
Revised: Additionally, we did not include the “no-observed-occupation” token when pretraining, and instead dropped missing years from the sequence.

Original: Since we did not use gender and race/ethnicity covariates when pretraining, we also initialized these covariate-speciﬁc parameters randomly for ﬁne-tuning.
Revised: Since we did not use gender and race/ethnicity covariates when pretraining, we also initialized these covariate-speciﬁc parameters randomly for ﬁne-tuning.

Original: Because we used a version of the survey datasets that encoded each individual’s location as a geographic region rather than as a state, we converted each state in the resumes data to be in one of four regions for pretraining: northeast, northcentral, south, or west.
Revised: Because we used a version of the survey datasets that encoded each individual’s location as a geographic region rather than as a state, we converted each state in the resumes data to be in one of four regions for pretraining: northeast, northcentral, south, or west.

Original: We also added a ﬁfth “other” region for Puerto Rico and for when a state was missing in the original dataset.
Revised: We also added a ﬁfth “other” region for Puerto Rico and for when a state was missing in the original dataset.

Original: We also converted educational degrees to levels of experience: we converted associate’s degree to represent some college experience and bachelor’s degree to represent four-year college experience; we combined masters and doctorate to represent a single “graduate degree” category; and we left the other categories as they were.
Revised: We also converted educational degrees to levels of experience: we converted associate’s degree to represent some college experience and bachelor’s degree to represent four-year college experience; we combined masters and doctorate to represent a single “graduate degree” category; and we left the other categories as they were.

Original: NLSY79.
Revised: NLSY79.

Original: The National Longitudinal Survey of Youth 1979 (NLSY79) is a survey following individuals born in the United States between 1957-1964.
Revised: The National Longitudinal Survey of Youth 1979 (NLSY79) is a survey following individuals born in the United States between 1957-1964.

Original: The survey included individuals who were between 14 and 22 years old when they began collecting data in 1979; they interviewed individuals annually until 1994, and biennially thereafter.
Revised: The survey included individuals who were between 14 and 22 years old when they began collecting data in 1979; they interviewed individuals annually until 1994, and biennially thereafter.

Original: Each individual in the survey is associated with an ID, allowing us to track their careers over time.
Revised: Each individual in the survey is associated with an ID, allowing us to track their careers over time.

Original: We converted occupations, which were initially encoded as OCC codes, into “occ1990dd” codes using a crosswalk (Autor & Dorn, 2013).
Revised: We converted occupations, which were initially encoded as OCC codes, into “occ1990dd” codes using a crosswalk (Autor & Dorn, 2013).

Original: We use a version of the survey that has entries up to 2014.
Revised: We use a version of the survey that has entries up to 2014.

Original: Unlike the resumes dataset, NLSY79 includes three states corresponding to individuals who are not currently employed: unemployed, out-of-labor-force, and in-school.
Revised: Unlike the resumes dataset, NLSY79 includes three states corresponding to individuals who are not currently employed: unemployed, out-of-labor-force, and in-school.

Original: We include special tokens for these states in our sequences.
Revised: We include special tokens for these states in our sequences.

Original: We drop examples with missing occupation states.
Revised: We drop examples with missing occupation states.

Original: We also drop sequences for which the individual is out of the labor force for their whole careers.
Revised: We also drop sequences for which the individual is out of the labor force for their whole careers.

Original: We use the following covariates: years, educational experience, location, race/ethnicity, and gender.
Revised: We use the following covariates: years, educational experience, location, race/ethnicity, and gender.

Original: We drop individuals with less than 9 years of education experience.
Revised: We drop individuals with less than 9 years of education experience.

Original: We convert years of educational experience into discrete categories: no high school degree, high school degree, some college, college, and graduate degree.
Revised: We convert years of educational experience into discrete categories: no high school degree, high school degree, some college, college, and graduate degree.

Original: We convert geographic location to one of four regions: northeast, northcentral, south, and west.
Revised: We convert geographic location to one of four regions: northeast, northcentral, south, and west.

Original: We treat location as a static variable, using each individual’s ﬁrst location.
Revised: We treat location as a static variable, using each individual’s ﬁrst location.

Original: We use the following race/ethnicities: white, African American, Asian, Latino, Native American, and other.
Revised: We use the following race/ethnicities: white, African American, Asian, Latino, Native American, and other.

Original: We treat year and education as dynamic covariates whose values can change over time, and we consider the other covariates as static.
Revised: We treat year and education as dynamic covariates whose values can change over time, and we consider the other covariates as static.

Original: This preprocessing leaves us with a dataset consisting of 12,270 individuals and 239,545 total observations.
Revised: This preprocessing leaves us with a dataset consisting of 12,270 individuals and 239,545 total observations.

Original: NLSY97.
Revised: NLSY97.

Original: The National Longitudinal Survey of Youth 1997 (NLSY97) is a survey following individuals who were between 12 and 17 when the survey began in 1997.
Revised: The National Longitudinal Survey of Youth 1997 (NLSY97) is a survey following individuals who were between 12 and 17 when the survey began in 1997.

Original: Individuals were interviewed annually until 2011, and biennially thereafter.
Revised: Individuals were interviewed annually until 2011, and biennially thereafter.

Original: Our preprocessing of this dataset is similar to that of NLSY79.
Revised: Our preprocessing of this dataset is similar to that of NLSY79.

Original: We convert occupations from
Revised: We convert occupations from

Original: OCC codes into “occ1990dd” codes.
Revised: OCC codes into “occ1990dd” codes.

Original: We use a version of the survey that follows individuals up to 2019.
Revised: We use a version of the survey that follows individuals up to 2019.

Original: We include tokens for unemployed, out-of-labor-force, and in-school occupational states.
Revised: We include tokens for unemployed, out-of-labor-force, and in-school occupational states.

Original: We only consider individuals who are over 18 and drop military-related occupations.
Revised: We only consider individuals who are over 18 and drop military-related occupations.

Original: We use the same covariates as NLSY79.
Revised: We use the same covariates as NLSY79.

Original: We use the following race/ethnicities: white, African-aAmerican, Latino, and other/unknown.
Revised: We use the following race/ethnicities: white, African-aAmerican, Latino, and other/unknown.

Original: We convert years of educational experience into discrete categories: no high school degree, high school degree, some college degree, college degree, graduate degree, and a special token when the education status isn’t known.
Revised: We convert years of educational experience into discrete categories: no high school degree, high school degree, some college degree, college degree, graduate degree, and a special token when the education status isn’t known.

Original: We use the same regions as NLSY79.
Revised: We use the same regions as NLSY79.

Original: We drop sequences for which the individual is out of the labor force for their whole careers.
Revised: We drop sequences for which the individual is out of the labor force for their whole careers.

Original: This preprocessing leaves us with a dataset consisting of 8,770 individuals and 114,141 total observations.
Revised: This preprocessing leaves us with a dataset consisting of 8,770 individuals and 114,141 total observations.

Original: PSID.
Revised: PSID.

Original: The Panel Study of Income Dynamics (PSID) is a longitudinal panel survey following a sample of American families.
Revised: The Panel Study of Income Dynamics (PSID) is a longitudinal panel survey following a sample of American families.

Original: It was collected annually between 1968 and 1997, and biennially afterwards.
Revised: It was collected annually between 1968 and 1997, and biennially afterwards.

Original: The dataset tracks families over time, but it only includes occupation information for the household head and their spouse, so we only include these observations.
Revised: The dataset tracks families over time, but it only includes occupation information for the household head and their spouse, so we only include these observations.

Original: Occupations are encoded with OCC codes, which we convert to “occ1990dd” using a crosswalk (Autor & Dorn, 2013).
Revised: Occupations are encoded with OCC codes, which we convert to “occ1990dd” using a crosswalk (Autor & Dorn, 2013).

Original: Like the NLSY surveys, PSID also includes three states corresponding to individuals who are not currently employed: unemployed, out-of-labor-force, and in-school.
Revised: Like the NLSY surveys, PSID also includes three states corresponding to individuals who are not currently employed: unemployed, out-of-labor-force, and in-school.

Original: We include special tokens for these states in our sequences.
Revised: We include special tokens for these states in our sequences.

Original: We drop other examples with missing or invalid occupation codes.
Revised: We drop other examples with missing or invalid occupation codes.

Original: We also drop sequences for which the individual is out of the labor force for their whole careers.
Revised: We also drop sequences for which the individual is out of the labor force for their whole careers.

Original: We consider ﬁve covariates: year, education, location, gender, and race.
Revised: We consider ﬁve covariates: year, education, location, gender, and race.

Original: We include observations for individuals who were added to the dataset after 1995 and include observations up to 2019.
Revised: We include observations for individuals who were added to the dataset after 1995 and include observations up to 2019.

Original: We exclude observations for individuals with less than 9 years of education experience.
Revised: We exclude observations for individuals with less than 9 years of education experience.

Original: We convert years of education to discrete states: no high school, high school diploma, some college, college, and graduate degree.
Revised: We convert years of education to discrete states: no high school, high school diploma, some college, college, and graduate degree.

Original: We convert geographic location to one of four regions: northeast, northcentral, south, and west.
Revised: We convert geographic location to one of four regions: northeast, northcentral, south, and west.

Original: We treat location as a static variable, using each individual’s ﬁrst location.
Revised: We treat location as a static variable, using each individual’s ﬁrst location.

Original: We use the following races: white, Black, and other.
Revised: We use the following races: white, Black, and other.

Original: We treat year and education as dynamic covariates whose values can change over time, and we consider the other covariates as static.
Revised: We treat year and education as dynamic covariates whose values can change over time, and we consider the other covariates as static.

Original: This preprocessing leaves us with a dataset consisting of 12,338 individuals and 62,665 total observations.
Revised: This preprocessing leaves us with a dataset consisting of 12,338 individuals and 62,665 total observations.

Original: H Experimental Details
Revised: H Experimental Details

Original: Resume pretraining.
Revised: Resume pretraining.

Original: We pretrain CAREER on a large dataset of resumes provided by Zippia Inc., a career planning company.
Revised: We pretrain CAREER on a large dataset of resumes provided by Zippia Inc., a career planning company.

Original: This dataset contains resumes from 23.7 million working Americans.
Revised: This dataset contains resumes from 23.7 million working Americans.

Original: Each job is encoded into one of 330 occupational codes, using the coding scheme of Autor & Dorn (2013).
Revised: Each job is encoded into one of 330 occupational codes, using the coding scheme of Autor & Dorn (2013).

Original: We transform resumes into sequences of jobs by including an occupation’s code for each year in the resume.
Revised: We transform resumes into sequences of jobs by including an occupation’s code for each year in the resume.

Original: For years with multiple jobs, we take the job the individual spent the most time in.
Revised: For years with multiple jobs, we take the job the individual spent the most time in.

Original: We include three covariates: the year each job in an individual’s career took place, along with the individual’s state of residence and most recent educational degree.
Revised: We include three covariates: the year each job in an individual’s career took place, along with the individual’s state of residence and most recent educational degree.

Original: We denote missing covariates with a special token.
Revised: We denote missing covariates with a special token.

Original: See Appendix F for an exploratory data analysis of the resume data.
Revised: See Appendix F for an exploratory data analysis of the resume data.

Original: CAREER uses a 12-layer transformer with 5.6 million parameters.
Revised: CAREER uses a 12-layer transformer with 5.6 million parameters.

Original: Pretraining CAREER on the resumes data takes 18 hours on a single GPU.
Revised: Pretraining CAREER on the resumes data takes 18 hours on a single GPU.

Original: Although our focus is on ﬁne-tuning CAREER to model survey datasets rather than resumes, CAREER also outperforms standard econometric baselines for modeling resumes; see ??
Revised: Although our focus is on ﬁne-tuning CAREER to model survey datasets rather than resumes, CAREER also outperforms standard econometric baselines for modeling resumes; see ??

Original: for more details.
Revised: for more details.

Original: Survey datasets.
Revised: Survey datasets.

Original: We transfer CAREER to three widely-used survey datasets: two cohorts from the National Longitudinal Survey of Youth (NLSY79 and NLSY97) and the Panel Study of Income
Revised: We transfer CAREER to three widely-used survey datasets: two cohorts from the National Longitudinal Survey of Youth (NLSY79 and NLSY97) and the Panel Study of Income

Original: Dynamics (PSID).
Revised: Dynamics (PSID).

Original: These datasets have been carefully constructed to be representative of the general population, and they are widely used by economists for estimating important quantities.
Revised: These datasets have been carefully constructed to be representative of the general population, and they are widely used by economists for estimating important quantities.

Original: NLSY79 is a longitudinal panel survey following a cohort of Americans who were between 14 and 22 when the survey began in 1979, while NLSY97 follows a diﬀerent cohort of individuals who were betweenand 17 when the survey began in 1997.
Revised: NLSY79 is a longitudinal panel survey following a cohort of Americans who were between 14 and 22 when the survey began in 1979, while NLSY97 follows a diﬀerent cohort of individuals who were betweenand 17 when the survey began in 1997.

Original: PSID is a longitudinal survey following a sample of American families, with individuals added over the years.
Revised: PSID is a longitudinal survey following a sample of American families, with individuals added over the years.

Original: Compared to the resumes dataset, these survey datasets are small: we use slices of NLSY79, NLSY97, and PSID that contain 12 thousand, 9 thousand, and 12 thousand individuals, respectively.
Revised: Compared to the resumes dataset, these survey datasets are small: we use slices of NLSY79, NLSY97, and PSID that contain 12 thousand, 9 thousand, and 12 thousand individuals, respectively.

Original: The distribution of job sequences in resumes diﬀers in meaningful ways from those in the survey datasets; for example, manual laborers are under-represented and college graduates are over-represented in resume data (see Appendix F for more details).
Revised: The distribution of job sequences in resumes diﬀers in meaningful ways from those in the survey datasets; for example, manual laborers are under-represented and college graduates are over-represented in resume data (see Appendix F for more details).

Original: We pretrain CAREER on the large resumes dataset and ﬁne-tune on the smaller survey datasets.
Revised: We pretrain CAREER on the large resumes dataset and ﬁne-tune on the smaller survey datasets.

Original: The ﬁne-tuning process is eﬃcient; although CAREER has 5.6 million parameters, ﬁne-tuning on one GPU takes 13 minutes on NLSY79, 7 minutes on
Revised: The ﬁne-tuning process is eﬃcient; although CAREER has 5.6 million parameters, ﬁne-tuning on one GPU takes 13 minutes on NLSY79, 7 minutes on

Original: NLSY97, and 23 minutes on PSID.
Revised: NLSY97, and 23 minutes on PSID.

Original: Baselines.
Revised: Baselines.

Original: We consider a ﬁrst-order Markov model and a second-order Markov model (both without covariates) as baselines.
Revised: We consider a ﬁrst-order Markov model and a second-order Markov model (both without covariates) as baselines.

Original: These models are estimated by averaging observed transition counts.
Revised: These models are estimated by averaging observed transition counts.

Original: We smooth the ﬁrst-order Markov model by taking a weighted average between the empirical transitions in the training set and the empirical distribution of individual jobs.
Revised: We smooth the ﬁrst-order Markov model by taking a weighted average between the empirical transitions in the training set and the empirical distribution of individual jobs.

Original: We perform this smoothing to account for the fact that some feasible transitions may never occur in the training set due to the high-dimensionality of feasible transitions.
Revised: We perform this smoothing to account for the fact that some feasible transitions may never occur in the training set due to the high-dimensionality of feasible transitions.

Original: We assign 0.99 weight to the empirical distributions of transitions and 0.01 to the empirical distribution of individual jobs.
Revised: We assign 0.99 weight to the empirical distributions of transitions and 0.01 to the empirical distribution of individual jobs.

Original: We smooth the second-order model by assigning 0.5 weight to the empirical second-order transitions and 0.5 weight to the smoothed ﬁrst-order Markov model.
Revised: We smooth the second-order model by assigning 0.5 weight to the empirical second-order transitions and 0.5 weight to the smoothed ﬁrst-order Markov model.

Original: When we add covariates to the Markov linear baseline, we also include manually constructed features about history to improve its performance.
Revised: When we add covariates to the Markov linear baseline, we also include manually constructed features about history to improve its performance.

Original: In total, we include the following categorical variables: the most recent job, the prior job, the year, a dummy indicating whether there has been more than one year since the most recent observed job, the education status, a dummy indicating whether the education status has changed, and state (for the experiments on NLSY79 and PSID, we also include an individual’s gender and race/ethnicity).
Revised: In total, we include the following categorical variables: the most recent job, the prior job, the year, a dummy indicating whether there has been more than one year since the most recent observed job, the education status, a dummy indicating whether the education status has changed, and state (for the experiments on NLSY79 and PSID, we also include an individual’s gender and race/ethnicity).

Original: We also add additive eﬀects for the following continuous variables: the number of years an individual has been in the current job and the total number of years for which an individual has been in the dataset.
Revised: We also add additive eﬀects for the following continuous variables: the number of years an individual has been in the current job and the total number of years for which an individual has been in the dataset.

Original: In addition, we include an intercept term.
Revised: In addition, we include an intercept term.

Original: For the bag-of-jobs model, we vary the representation dimension D between 256-2048, and ﬁnd that the predictive performance is not sensitive to the representation dimension, so we use D = 1024 for all experiments.
Revised: For the bag-of-jobs model, we vary the representation dimension D between 256-2048, and ﬁnd that the predictive performance is not sensitive to the representation dimension, so we use D = 1024 for all experiments.

Original: For the LSTM model, we use 3 layers with 436 embedding dimensions so that the model size is comparable to the transformer baseline: the LSTM has 5.8 million parameters, the same number as the transformer.
Revised: For the LSTM model, we use 3 layers with 436 embedding dimensions so that the model size is comparable to the transformer baseline: the LSTM has 5.8 million parameters, the same number as the transformer.

Original: We also compare to NEMO (Li et al., 2017), an LSTM-based method developed for modeling job sequences in resumes.
Revised: We also compare to NEMO (Li et al., 2017), an LSTM-based method developed for modeling job sequences in resumes.

Original: We adapted NEMO to model survey data.
Revised: We adapted NEMO to model survey data.

Original: In its original setting, NEMO took as input static covariates (such as individual skill) and used these to predict both an individual’s next job title and their company.
Revised: In its original setting, NEMO took as input static covariates (such as individual skill) and used these to predict both an individual’s next job title and their company.

Original: Survey datasets diﬀer from this original setting in a few ways: covariates are time-varying, important covariates for predicting jobs on resumes (like skill) are missing, and an individual’s company name is unavailable.
Revised: Survey datasets diﬀer from this original setting in a few ways: covariates are time-varying, important covariates for predicting jobs on resumes (like skill) are missing, and an individual’s company name is unavailable.

Original: Therefore, we made several modiﬁcations to NEMO.
Revised: Therefore, we made several modiﬁcations to NEMO.

Original: We incorporated the available covariates from survey datasets by embedding them and adding them to the job embeddings passed into the LSTM, similar to the method CAREER uses to incorporate covariates.
Revised: We incorporated the available covariates from survey datasets by embedding them and adding them to the job embeddings passed into the LSTM, similar to the method CAREER uses to incorporate covariates.

Original: We removed the company-prediction objective, and instead only used the model to predict an individual’s job in the next timestep.
Revised: We removed the company-prediction objective, and instead only used the model to predict an individual’s job in the next timestep.

Original: We considered two sizes of NEMO: an architecture using the same number of parameters as CAREER, and the smaller architecture proposed in the original paper.
Revised: We considered two sizes of NEMO: an architecture using the same number of parameters as CAREER, and the smaller architecture proposed in the original paper.

Original: We found the smaller architecture performed better on the survey datasets, so we used this for the experiments.
Revised: We found the smaller architecture performed better on the survey datasets, so we used this for the experiments.

Original: This model contains 2 decoder layers and a hidden dimension of 200.
Revised: This model contains 2 decoder layers and a hidden dimension of 200.

Original: When we compared the transferred version of CAREER to a version of CAREER without pretrained representations, we tried various architectures for the non-pretrained version of CAREER.
Revised: When we compared the transferred version of CAREER to a version of CAREER without pretrained representations, we tried various architectures for the non-pretrained version of CAREER.

Original: We found that, without pretraining, the large architecture we used for CAREER was prone to overﬁtting on the smaller survey datasets.
Revised: We found that, without pretraining, the large architecture we used for CAREER was prone to overﬁtting on the smaller survey datasets.

Original: So we performed an ablation of the non-pretrained CAREER with various architectures: we considered 4 and 12 layers, 64 and 192 embedding dimensions, 256 and 768 hidden units for the feedforward neural networks, and 2 or 3 attention heads (using 2 heads for D = 64 andheads for D = 192 so that D was divisible by the number of heads).
Revised: So we performed an ablation of the non-pretrained CAREER with various architectures: we considered 4 and 12 layers, 64 and 192 embedding dimensions, 256 and 768 hidden units for the feedforward neural networks, and 2 or 3 attention heads (using 2 heads for D = 64 andheads for D = 192 so that D was divisible by the number of heads).

Original: We tried all 8 combinations of these parameters on NLSY79, and found that the model with the best validation performance had 4 layers, D = 64 embedding dimensions, 256 hidden units, and 2 attention heads.
Revised: We tried all 8 combinations of these parameters on NLSY79, and found that the model with the best validation performance had 4 layers, D = 64 embedding dimensions, 256 hidden units, and 2 attention heads.

Original: We used this architecture for the non-pretrained version of CAREER on all survey datasets.
Revised: We used this architecture for the non-pretrained version of CAREER on all survey datasets.

Original: Training.
Revised: Training.

Original: We randomly divide the resumes dataset into a training set of 23.6 million sequences, and a validation and test set of 23 thousand sequences each.
Revised: We randomly divide the resumes dataset into a training set of 23.6 million sequences, and a validation and test set of 23 thousand sequences each.

Original: We randomly divide the survey datasets into 70/10/20 train/test/validation splits.
Revised: We randomly divide the survey datasets into 70/10/20 train/test/validation splits.

Original: The ﬁrst- and second-order Markov models without covariates are estimated from empirical transitions counts.
Revised: The ﬁrst- and second-order Markov models without covariates are estimated from empirical transitions counts.

Original: We optimize all other models with stochastic gradient descent with minibatches.
Revised: We optimize all other models with stochastic gradient descent with minibatches.

Original: In total, we use 16,000 total tokens per minibatch, varying the batch size depending on the largest sequence length in the batch.
Revised: In total, we use 16,000 total tokens per minibatch, varying the batch size depending on the largest sequence length in the batch.

Original: We use the Adam learning rate scheduler (Kingma & Ba, 2015).
Revised: We use the Adam learning rate scheduler (Kingma & Ba, 2015).

Original: All experiments on the resumes data warm up the learning rate from 10 − 7 to 0 .
Revised: All experiments on the resumes data warm up the learning rate from 10 − 7 to 0 .

Original: 0005 over 4,000 steps, after which the inverse square root schedule is used (Vaswani et al., 2017).
Revised: 0005 over 4,000 steps, after which the inverse square root schedule is used (Vaswani et al., 2017).

Original: For the survey datasets, we also used the inverse square root scheduler, but experimented with various learning rates and warmup updates, using the one we found to work best for each model.
Revised: For the survey datasets, we also used the inverse square root scheduler, but experimented with various learning rates and warmup updates, using the one we found to work best for each model.

Original: For CAREER with pretrained representations, we used a learning rate of 0 . 0001 and 500 warmup updates; for CAREER without pretraining, we used a learning rate of 0 . 0005 and 500 warmup updates; for the bag of jobs model, we used a learning rate of 0 . 0005 and 5,000 warmup updates; for the regression model, we used a learning rate of 0 .and 4,000 warmup updates.
Revised: For CAREER with pretrained representations, we used a learning rate of 0 . 0001 and 500 warmup updates; for CAREER without pretraining, we used a learning rate of 0 . 0005 and 500 warmup updates; for the bag of jobs model, we used a learning rate of 0 . 0005 and 5,000 warmup updates; for the regression model, we used a learning rate of 0 .and 4,000 warmup updates.

Original: All models besides were also trained with 0.01 weight decay.
Revised: All models besides were also trained with 0.01 weight decay.

Original: All models were trained using Fairseq (Ott et al., 2019).
Revised: All models were trained using Fairseq (Ott et al., 2019).

Original: When training on resumes, we trained for 85,000 steps, using the checkpoint with the best validation performance.
Revised: When training on resumes, we trained for 85,000 steps, using the checkpoint with the best validation performance.

Original: When ﬁne-tuning on the survey datasets, we trained all models until they overﬁt to the validation set, again using the checkpoint with the best validation performance.
Revised: When ﬁne-tuning on the survey datasets, we trained all models until they overﬁt to the validation set, again using the checkpoint with the best validation performance.

Original: We used half precision for training all models, with the exception of the following models (which were only stable with full precision): the bag of jobs model with covariates on the resumes data, and the regression models for all survey dataset experiments.
Revised: We used half precision for training all models, with the exception of the following models (which were only stable with full precision): the bag of jobs model with covariates on the resumes data, and the regression models for all survey dataset experiments.

Original: The tables in Section 3 report results averaged over multiple random seeds.
Revised: The tables in Section 3 report results averaged over multiple random seeds.

Original: For the results in Figure 1a, the randomness includes parameter initialization and minibatch ordering.
Revised: For the results in Figure 1a, the randomness includes parameter initialization and minibatch ordering.

Original: For CAREER, we use the same pretrained model for all settings.
Revised: For CAREER, we use the same pretrained model for all settings.

Original: For the forecasting results in Table 1, the randomness is with respect to the Monte-Carlo sampling used to sample multi-year trajectories for individuals.
Revised: For the forecasting results in Table 1, the randomness is with respect to the Monte-Carlo sampling used to sample multi-year trajectories for individuals.

Original: For the wage prediction experiment in ?? , the randomness is with respect to train/test splits.
Revised: For the wage prediction experiment in ?? , the randomness is with respect to train/test splits.

Original: Forecasting.
Revised: Forecasting.

Original: For the forecasting experiments, occupations that took place after a certain year are dropped from the train and validation sets.
Revised: For the forecasting experiments, occupations that took place after a certain year are dropped from the train and validation sets.

Original: When we forecast on the resumes dataset, we use the same train/test/validation split but drop examples that took place after 2014.
Revised: When we forecast on the resumes dataset, we use the same train/test/validation split but drop examples that took place after 2014.

Original: When we pretrain CAREER on the resumes dataset to make forecasts for PSID and NLSY97, we use a cutoﬀ year of 2014 as well.
Revised: When we pretrain CAREER on the resumes dataset to make forecasts for PSID and NLSY97, we use a cutoﬀ year of 2014 as well.

Original: We incorporate two-stage prediction into the baseline models because we ﬁnd that this improves their predictions.
Revised: We incorporate two-stage prediction into the baseline models because we ﬁnd that this improves their predictions.

Original: Although we do not include any examples after the cutoﬀ during training, all models require estimating year-speciﬁc terms.
Revised: Although we do not include any examples after the cutoﬀ during training, all models require estimating year-speciﬁc terms.

Original: We use the ﬁtted values from the last observed year to estimate these terms.
Revised: We use the ﬁtted values from the last observed year to estimate these terms.

Original: For example, CAREER requires embedding each year.
Revised: For example, CAREER requires embedding each year.

Original: When the cutoﬀ year is 2014, there do not exist embeddings for years after 2014, so we substitute the 2014 embedding.
Revised: When the cutoﬀ year is 2014, there do not exist embeddings for years after 2014, so we substitute the 2014 embedding.

Original: We report forecasting results on a split of the dataset containing examples before and after the cutoﬀ year.
Revised: We report forecasting results on a split of the dataset containing examples before and after the cutoﬀ year.

Original: To make predictions for an individual, we condition on all observations before the cutoﬀ year, and sample 1,000 trajectories through the last forecasting year.
Revised: To make predictions for an individual, we condition on all observations before the cutoﬀ year, and sample 1,000 trajectories through the last forecasting year.

Original: We never condition on any occupations after the cutoﬀ year, although we include updated values of dynamic covariates like education.
Revised: We never condition on any occupations after the cutoﬀ year, although we include updated values of dynamic covariates like education.

Original: For forecasting on the resumes dataset, we set the cutoﬀ for 2014 and forecast occupations for 2015, 2016, and 2017.
Revised: For forecasting on the resumes dataset, we set the cutoﬀ for 2014 and forecast occupations for 2015, 2016, and 2017.

Original: We restrict our test set to individuals in the original test set whose ﬁrst observed occupation was before 2015 and who were observed to have worked until 2017.
Revised: We restrict our test set to individuals in the original test set whose ﬁrst observed occupation was before 2015 and who were observed to have worked until 2017.

Original: PSID and NLSY are biennial, so we forecast for 2015, 2017, and 2019.
Revised: PSID and NLSY are biennial, so we forecast for 2015, 2017, and 2019.

Original: We only make forecasts for individuals who have observations before the cutoﬀ year and through the last year of forecasting, resulting in a total of 16,430 observations for PSID and 18,743 for NLSY97.
Revised: We only make forecasts for individuals who have observations before the cutoﬀ year and through the last year of forecasting, resulting in a total of 16,430 observations for PSID and 18,743 for NLSY97.

Original: Rationalization.
Revised: Rationalization.

Original: The example in Figure 3 shows an example of CAREER’s rationale on PSID.
Revised: The example in Figure 3 shows an example of CAREER’s rationale on PSID.

Original: To simplify the example, this is the rationale for a model trained on no covariates except year.
Revised: To simplify the example, this is the rationale for a model trained on no covariates except year.

Original: In order to conceal individual behavior patterns, the example in Figure 3 is a slightly altered version of a real sequence.
Revised: In order to conceal individual behavior patterns, the example in Figure 3 is a slightly altered version of a real sequence.

Original: For this example, the transformer used for CAREER follows the architecture described in
Revised: For this example, the transformer used for CAREER follows the architecture described in

Original: Radford et al.
Revised: Radford et al.

Original: We ﬁnd the rationale using the greedy rationalization method described in Vafa et al.
Revised: We ﬁnd the rationale using the greedy rationalization method described in Vafa et al.

Original: Greedy rationalization requires ﬁne-tuning the model for compatibility; we do this by ﬁne-tuning with “job dropout”, where with 50% probability, we drop out a uniformly random amount of observations in the history.
Revised: Greedy rationalization requires ﬁne-tuning the model for compatibility; we do this by ﬁne-tuning with “job dropout”, where with 50% probability, we drop out a uniformly random amount of observations in the history.

Original: When making predictions, the model has to implicitly marginalize over the missing observations.
Revised: When making predictions, the model has to implicitly marginalize over the missing observations.

Original: (We pretrain on the resumes dataset without any word dropout).
Revised: (We pretrain on the resumes dataset without any word dropout).

Original: We ﬁnd that training converges quickly when ﬁne-tuning with word dropout, and the model’s performance when conditioning on the full history is similar.
Revised: We ﬁnd that training converges quickly when ﬁne-tuning with word dropout, and the model’s performance when conditioning on the full history is similar.

Original: Greedy rationalization typically adds observations to a history one at a time in the order that will maximize the model’s likelihood of its top prediction.
Revised: Greedy rationalization typically adds observations to a history one at a time in the order that will maximize the model’s likelihood of its top prediction.

Original: For occupations, the model’s top prediction is almost always identical to the previous year’s occupation, so we modify greedy rationalization to add the occupation that will maximize the likelihood of its second-largest prediction.
Revised: For occupations, the model’s top prediction is almost always identical to the previous year’s occupation, so we modify greedy rationalization to add the occupation that will maximize the likelihood of its second-largest prediction.

Original: This can be interpreted as equivalent to greedy rationalization, albeit conditioning on switching occupations.
Revised: This can be interpreted as equivalent to greedy rationalization, albeit conditioning on switching occupations.

Original: Thus, the greedy rationalization procedure stops when the model’s second-largest prediction from the target rationale is equivalent to the model’s second-largest prediction when conditioning on the full history.
Revised: Thus, the greedy rationalization procedure stops when the model’s second-largest prediction from the target rationale is equivalent to the model’s second-largest prediction when conditioning on the full history.