S pecia L ex: A Benchmark for In-Context Specialized Lexicon Learning (2024)

Abstract: Specialized lexicons are collections of words with associated constraints such as special definitions, specific roles, and intended target audiences. These constraints are necessary for content generation and documentation tasks (e.g., writing technical manuals or children’s reading materials), where the goal is to reduce the ambiguity of text content and increase its overall readability for a specific group of audience. Understanding how large language models can capture these constraints can help researchers build better, more impactful tools for wider use beyond the NLP community. Towards this end, we introduce SpeciaLex, a benchmark for evaluating a language model’s ability to follow specialized lexicon-based constraints across 18 diverse subtasks with 1,785 test instances covering core tasks of Checking, Identification, Rewriting, and Open Generation. We present an empirical evaluation of 15 open and closed-source LLMs and discuss insights on how factors such as model scale, openness, setup, and recency affect performance upon evaluating with the benchmark.

## introduction the adoption of large language models (llms) for domains beyond computing and ai has been more evident in recent years, particularly with the release of publicly accessible chat interfaces such as chat-gpt. this widespread use from various multidisciplinary communities can be primarily attributed to modern llms' capabilities to learn patterns from just a few examples during inference-in-context learning (icl)-combined with the use of modern architectures and massive and diverse datasets to train them to follow complex instructions @xcite @xcite . with in-context learning, llms can be treated as task-agnostic systems and can do virtually any text-related task, including open-ended generation and structured prediction, just by being conditioned to provide completions for prompts given task-specific demonstrations @xcite @xcite . one particular point of interest in the wider adoption of llms is evaluating how they can capture lexicon-based constraints for generating text content across different domains. for example, in education, a teacher who knows how to masterfully use an llm (e.g., chatgpt) to generate classroomready reading materials on the fly can accommodate students' various interests in reading @xcite , such as prompting the llm with preferred topics for stories and custom character roles. however, if used this way, the llm should learn constraints such as knowing what specific words are readable by a target audience (e.g., ages 10-11). these special words are often found on specially curated lexicons such as the oxford 5000 wordlist foot_1 . in technical writing, on the other hand, an llm should learn to capture customized word definition constraints as mandated by existing guidelines and standards to avoid producing ambiguous texts. for example, as per simplified technical english (ste) foot_2 guidelines, the word glue cannot be used as a verb to mean stick together; the appropriate word for this is bond or attach. understanding how current llms capture finegrained constraints from specialized lexicons across domains opens a number of opportunities for improving their ability to follow instructions at a very fine level, particularly through in-context learning. however, the main gap here is that there are currently no comprehensive evaluation studies or benchmarks to guide researchers in learning more about the performance and limitations of modern llms on content generation tasks requiring compliance with said constraints. in this study, we fill the gap by introducing spe-cialex, a comprehensive benchmark suite composed of 18 diverse tasks to evaluate the capabilities of llms in capturing lexicon-based constraints such as special roles or part-of-speech, special word definitions, and target audiences. we provide an in-depth comparison of 15 state-of-the-art llms as baselines and release extendable specialex subtask data comprising 1, 785 test instances. we devised four core task variations spanning check-ing, identification, rewriting, and open generation. implementation-wise, we structured specialex to focus on using in-context learning for all tasks as this emulates the most common way for lay people and users to interact with llms through carefully structured prompts with examples or demonstrations. by evaluating a diverse set of commercial and open llms in terms of task performance, scale, and openness, specialex serves as a valuable reference and guide for interdisciplinary researchers who require the use of capable llms but are on a limited computing budget or are concerned only with performance on specific constraints. moreover, by following design principles from estab-lished open llm benchmarks such as legal-bench @xcite , the research community can extend and build upon specialex by contributing new tasks and specialized lexicons from other domains to expand the evaluation of llms in this direction. ## specialex: a benchmark for in-context specialized lexicon learning we build specialex as a general benchmark and reference for evaluating llms to capture lexiconbased constraints through in-context learning. we discuss the task typology and recognized lexiconbased constraints of specialex as seen in figure 1 . ## specialex task construction process this section provides an overview of the construction process we followed for building and evaluating tasks for specialex with resources provided by experts. ## a specialex guide in this section, we outline a number of important points for consideration to guide researchers in using specialex as a reference or an evaluation tool for specific domain data and constraints. do bigger models have better performance? it depends on the task. it is a common observation from empirical experiments with llms that the larger the scale, the higher the generalization and performance across diverse tasks @xcite @xcite . however, the choice of larger models may be expensive and impractical for domain adaptation, where performance on a limited set of tasks (or even a singular task) is often prioritized. upon aggregating the mean results from tables 2 and 3 of models with increasing scale in figure 2 , we see only favorable performance for larger models on ste-based constraints focused on specific pos and special definitions. in the case of using target audience constraint, we observe that even the 8b version of llama3 is better than all other models tested. thus, we recommend researchers consider the nature of the task first, as smaller models have empirically shown to be able to achieve comparable performance on select constraints. are open models good enough? yes. while it is also a common notion that commercial models such as gpt-4 by openai are popularly known and advertised as the go-to standard for general nlp tasks, we provide empirical evidence in this study that open models are equally as performant and can serve as a practical alternative for the research community. revisiting our findings from section 5, open models such as llama3-8b and 70b are able to achieve comparable-if not higher in some cases-performances across the four core tasks based on mean scores. do high-quality training data and model recency matter? yes. model scale may not be the only signal of effectiveness for capturing lexicon-based constraints. we recommend weighing the quality of data used for training the llms and using the most recent model versions released by their research developers. we see this particular advantage in the llama family models with 15t token count used for pre-training data as well as using high-quality data filters foot_6 powered by llama2. with this advantage, llama3 was able to achieve generally higher task performances in specialex than llama2. likewise, we posit that llama3's recency among all the other models may have given certain advantages in terms of data quality through scoping more and larger published open-source datasets used for pre-training. how many demonstrations do i need for icl? five is a good start. specialex benchmarks models via in-context learning since prompting and providing additional information and target output is the most common way of interacting and delegating tasks to llms. as such, we recommend starting with around five or more diverse demonstrations rather than a zero-shot method for lexicon-based constraints to maximize the effectiveness of in-context learning. we support this recommendation by exploring various few-shot techniques from the best-performing models for ste and cefr-based constraints, as seen in figure 3 . from the experiment, we report that using the standard 5-shot setup done in the major experiments in is in-context learning better than domainspecific finetuning? icl allows flexibility and preserves general model performance. the benefit of specialex by benchmarking via in-context learning is that it avoids re-training llms to one specific task only and preserves the user's ability to use the llm (or llm interfaces such as chatgpt) to perform other downstream tasks such question answering, summarizing content, and solving problems to name a few. moreover, in case llms perform poorly on tasks via in-context learning, the results would serve as a helpful direction for domain users to then explore finetuning or other optimization methods. nonetheless, this recommendation is not prescriptive if domain users' priority is to develop a model that performs well in capturing constraints from only one source of reference via the specialized lexicon. ## conclusion in this work, we introduced specialex, a benchmark for evaluating state-of-the-art llms in capturing specialized lexicon-based constraints for content generation tasks commonly prevalent across interdisciplinary areas such as education, technical writing, and engineering. we provided an indepth and empirical exploration of model performance, including looking at the effects of model scale, openness, few-shot setup, and recency. our findings support the use of open models such as llama8-3b as good, competitive starting resources for the benchmark. in hindsight, specialex will serve as a reference guide for researchers within and outside of the nlp community, where they can check what specific models are good for certain task types (checking, identification, rewrit-ing, generation) or handle specific constraints (specific roles, special definitions, target audience) and springboard further research based on their own domain specifications.