Academics Can Contribute to Domain-Specialized Language Models (2024)

Abstract: Commercially available models dominate academic leaderboards. While impressive, this has concentrated research on creating and adapting general-purpose models to improve NLP leaderboard standings for large language models. However, leaderboards collect many individual tasks and general-purpose models often underperform in specialized domains; domain-specific or adapted models yield superior results. This focus on large general-purpose models excludes many academics and draws attention away from areas where they can make important contributions. We advocate for a renewed focus on developing and evaluating domain- and task-specific models, and highlight the unique role of academics in this endeavor.

## introduction natural language processing (nlp) research has historically produced domain-and task-specific supervised models. the field has shifted course in the past few years, with a singular focus on generalpurpose generative large language models (llms) that, rather than focusing on a single task or domain, do well across many tasks @xcite @xcite @xcite . by training on massive amounts of data from many sources, these models can do well on extremely broad professional and linguistic examinations @xcite , college-level knowledge questions @xcite , and collections of reasoning tasks @xcite . while the trend to develop a single, generalpurpose generative model is a net positive change that has resulted in impressive results, it has also slowed down progress in other areas of nlp. first, we are less focused on problems that cannot be * the project was completed during work at bloomberg. solved with a chat-like interface. second, the bestperforming llms are often commercial systems, which are sometimes opaque about training data, system architecture, and training details. third, frequent model updates hinder reproducibility. the resources required to train large general language models naturally constrain research to large organizations, and researchers (or academics) outside of these organizations have become dependent on closed commercial systems, or open systems with limited transparency regarding their training data. this is partly reflected in broader ai trends @xcite found that roughly 30% of papers at ai conferences (including *cl) have a fortune 500 tech affiliation. increased resources contribute to the success of transformerbased llms @xcite , with available hardware @xcite and benchmarks @xcite both playing a deciding role in what models end up being developed. by optimizing the average score across hundreds of shallow tasks, we are smoothing out any signal that would be gained from deeply engaging with individual tasks. developing domain-specific models can help identify model and training choices that yield improvements on tasks within those domains. in this paper, we argue for renewed attention to domain-specific models with rigorous and domainexpert informed evaluations. because many academics are excluded from llm development due to resource constraints, attention has been drawn away from research areas where academics can make the greatest contributions: deep dives on specific challenging problems. thus, we propose several research questions to reorient the research community towards developing domain-specific models and applications, where academics are uniquely suited to lead. ## llms: a brief history while modern lms date back to @xcite , we summarize very recent history to describe the current environment. in the wake of the popularization of neural word embeddings by word2vec @xcite , contextualized representations of language as features for supervised systems were realized by elmo @xcite followed by bert @xcite . bert and subsequent models became the base models for supervised systems utilizing task-specific finetuning and continued pre-training for new domains @xcite , e.g., for clinical tasks elmo @xcite and clini-calbert @xcite . parallel work utilized transformers for autoregressive llms, resulting in gpt @xcite , gpt-2 @xcite , bart @xcite , ctrl @xcite , t5 @xcite , and xglm @xcite . these models had some few-shot capabilities, but they could each be adapted (fine-tuned) for a specific task of interest. some models were available to academics, though training a new model was beyond reach for many. gpt-3 @xcite greatly increased model size and changed our understanding of llms. impressive in-context (few-shot) learning pushed the idea that a single large model could solve a wide range of tasks. while the cost of resources meant training was restricted to a few groups, work focused on training bigger models @xcite @xcite @xcite . while only a few could train large models, many studied how best to use them: prompt engineering @xcite , prompt tuning @xcite , evaluation @xcite , among many other topics. commercial llm apis, and eventually open source models @xcite @xcite , facilitated this work. @xcite noted the massive research shift to llms reflected in google scholar citations. subsequent work in instruction tuning @xcite and fine-tuning @xcite @xcite have further centralized research around generalpurpose models. many consider fine-tuning for specific applications to be obsolete: why would you tune a model for a specific task when you can tune a single model to do well on all tasks? 1 despite this view, multiple domain-specific llms have demonstrated that domain-specific data leads to models that outperform much larger models @xcite . med-palm has shown that adapting even giant llms to a specific domain leads to vastly increased performance @xcite . 2 furthermore, the release of llama @xcite ) led quickly to alpaca @xcite and a wave of new fine-tuned versions of llama for specific tasks. this trend strongly indicates that domainspecific models, especially for constrained sizes, are still highly relevant. to be clear, our concern is not with closed models, which play an important role in the model ecosystem. models range from full to limited to no access, with some closed models providing incredibly detailed information @xcite @xcite and others providing none @xcite . our lament over this focus on general models, either open or closed, is that it draws attention away from work on task-and domain-specific models and evaluations. academics have become product testers, instead of focusing on tasks where they can play a unique role. moreover, existing academic benchmarks increasingly serve a reduced purpose for commercial models; we are hill-climbing on benchmarks without a way to ensure existing llms have not been trained to excel on these benchmarks @xcite . furthermore, we rely on benchmarks in place of deep engagement with an application and its stakeholders. ## the need for domain-specific llms in general, web data does not reflect the needs of all nlp systems. historically, the community has developed systems for specialized domains such as finance, law, bio-medicine, and science. accordingly, there have been efforts to build llms for these domains @xcite @xcite @xcite @xcite . we need a deep investment in how best to develop and evaluate these models in partnership with domain experts. how should we best integrate insights gained from the development of generalpurpose models with these efforts? we propose several research directions. how can general-purpose models inform domain-specific models? building domainspecific models should benefit from insights and investments into general-purpose models. there are several strategies: training domain-specific models from scratch @xcite , mixing general and domain-specific data @xcite , and fine-tuning existing models @xcite . focusing on domain-specific needs, applications, and knowledge with guidance from topic experts will benefit us in acquiring a better model for specific nlp tasks. which approach yields the best results for task performance and overall cost? what is the role of in-context learning and finetuning? both lima @xcite and med-palm @xcite use a small number of examples to tune a model. with expanding context size, we may soon rely entirely on in-context learning @xcite . this blurs the lines between changing model parameters and conditioning during inference. beyond inference speed tradeoffs between the two, there may be value in tuning on tens of thousands (or more) of examples. which domain-specific examples are the most effective to include and in what manner? how can llms be integrated with domainspecific knowledge? specialized knowledge is key in many domains. rag @xcite and kilt-derived works @xcite focus on knowledge-intensive tasks by including retrieval steps. work on attributed qa @xcite takes a similar approach, as do search llms that require interaction with retrieved data @xcite . rich updated knowledge sources will always exist beyond the model, especially in environments like medicine, finance, and many academic disciplines. ## evaluation of domain-specific models the evaluation of nlp systems is at a crossroads, and the downstream usage of llms and evaluation approaches have diverged. benchmarks assume that their results translate to insights into similar tasks and usefulness for commercial applications. but benchmarks have become increasingly narrow in scope, oftentimes assessing one metric on a single, often flawed, dataset @xcite @xcite . the primary evaluation approach for llms has been to evaluate on a broad set of these narrow benchmarks @xcite . high average performance argues for a broad range of capabilities; however, one size may not fit all. since specific uses of llms are typically much more narrow, we identify three major issues and associated research opportunities with this approach. depth-first evaluation current approaches focus on a single model doing everything well on average instead of being useful in a single domain. however, it is widely acknowledged that the standard benchmarks for most tasks are insufficient (e.g., for summarization, @xcite . task-specific evaluations have thus adopted additional protocols that measure how well models transfer to different domains, how robust they are, and whether they stand up to concept drift @xcite . these details disappear when benchmarking on 100+ tasks. yet, a model's usefulness is not solely defined by doing okay on everything but rather by how well it performs in specific and narrow tasks that provide value. this value is only realized if the model does not suffer from catastrophic failures. exemplar studies that perform deep dives on llms for specific tasks exist in healthcare @xcite @xcite @xcite , law @xcite , and physics @xcite , among other areas. we encourage more work on evaluation practices for specific tasks that can handle various model setups and yield informative insights @xcite . sound metrics for convenience, most benchmark tasks are formulated as multiple choice question answering or classification. this is not how llms are often used. for much more common generation tasks, researchers have been ringing alarms about broken evaluations @xcite . it is dubious whether we gain insights into nontask-specific generation through nlu benchmarks. if we are performing the depth-first evaluation of a generation task, a remaining hurdle -and why researchers fall back to nlu tasks -is the lack of robust metrics. while there is much recent work on better metrics @xcite , a troubling trend is the use of llms as evaluators (e.g., @xcite . this approach poses many risks, including the implicit assumption that the evaluating model has access to the ground truth judgment. while there are some promising results, using an llm out of the box should be avoided (e.g., @xcite . moreover, it is unclear how to evaluate the evaluator when it is a non-deterministic api, or how to scale the development of learned metrics and quantify the strength of a metric. products are not baselines if we really do want to evaluate 100+ tasks, there are many issues with the soundness of evaluation setups. at this scope, it is impossible to run careful ablation studies or to assess the effect of changes to methodology in a causal manner. moreover, different llms respond differently to prompts. the bloom evaluation averaged over multiple prompts and found significant variance @xcite . this variance leads to a lack of reproducibility: llama @xcite claimed high mmlu @xcite performance but didn't release the prompts that led to them. 3 similarly, the evaluation scheme makes a difference @xcite . high evaluation costs mean benchmarks pick a small number of setups (sometimes only one) for each task, which introduces further bias, making it hard to construct fair benchmarks on many tasks. an additional issue with the current benchmarking approach is that the best-performing models are often commercial apis. with limited transparency regarding data and training, we cannot fairly evaluate these models (e.g., data leakage). furthermore, task-specific tuning may have been selected based on these specific benchmarks. moreover, the underlying models change frequently, so it is unclear whether a result will hold for long. these evaluation issues prompt significant open questions: 1) how do we develop consistent evaluation setups across models that give true measures of performance? 2) how do we develop evaluation setups and metrics more closely aligned with downstream usage? 3) how do we develop evaluation suites that support depth-first evaluation and not breadth-first benchmarking? ## the role of academics a focus on general-purpose llms has forced academics to work with large base models and perhaps, shifted the focus to solve problems of immediate industrial interest. many academics feel excluded from current research trends @xcite and the academic and industry relationship is changing @xcite . shifting attention back to domain-specific applications emphasizes areas where academics hold an advantage: partnerships with domain experts to invest in specific tasks, and consideration of broader societal needs. developing domain-specific models requires domain expertise and universities are diverse academic environments that house experts in many domains. collaborations with these experts can identify data sources, tasks, and challenges important within each domain. furthermore, these collaborations are the best avenues for better alignment of evaluations with use cases @xcite , and can support the development of proper metrics. these collaborations are necessary to explore wide open interdisciplinary topics, such as models for protein structure prediction @xcite and games as proxies for reasoning @xcite @xcite . this includes developing domain-specific resources, which require domain experts to properly design and construct the datasets. further, areas where industry underinvests are those where academics could focus attention. for example, low-resource languages are not served by a general-purpose multilingual llm, nor will we reasonably have enough data to support current llm training methods. dialects and variations in languages are still wide open topics @xcite @xcite . general-purpose llms are unlikely to solve problems in many important domains, with many open research problems that can only be solved by domain-specific approaches. focusing on domainspecific knowledge will benefit us in acquiring a better model and developing application strategies more aligned with how humans learn domainspecific knowledge @xcite . for many interdisciplinary areas, subject matter experts are essential, and the problems must be defined clearly. the first pass from an llm is often impressive, but it hides the trenches and areas where things are most interesting. we need a renewed fo-cus on developing and evaluating domain-specific models and applications, an area where academics can play a leading role. let us not be distracted by claims that a single model solves all tasks, and instead deeply explore and understand the needs and challenges of specific domains.