Bridging the LLM Accessibility Divide? Performance, Fairness, and Cost of Closed versus Open LLM s for Automated Essay Scoring (2025)

Abstract: Closed large language models (LLMs) such as GPT-4 have set state-of-the-art results across a number of NLP tasks and have become central to NLP and machine learning (ML)-driven solutions. Closed LLMs’ performance and wide adoption has sparked considerable debate about their accessibility in terms of availability, cost, and transparency. In this study, we perform a rigorous comparative analysis of eleven leading LLMs, spanning closed, open, and open-source LLM ecosystems, across text assessment and generation within automated essay scoring, as well as a separate evaluation on abstractive text summarization to examine generalization. Our findings reveal that for few-shot learning-based assessment of human generated essays, open LLMs such as Llama 3 and Qwen 2.5 perform comparably to GPT-4 in terms of predictive performance, with no significant differences in disparate impact scores when considering age- or race-related fairness. For summarization, we find that open models also match GPT-4 in ROUGE and METEOR scores on the CNN/DailyMail benchmark, both in zero- and few-shot settings. Moreover, Llama 3 offers a substantial cost advantage, being up to 37 times more cost-efficient than GPT-4. For generative tasks, we find that essays generated by top open LLMs are comparable to closed LLMs in terms of their semantic composition/embeddings and ML assessed scores. Our findings challenge the dominance of closed LLMs and highlight the democratizing potential of open LLMs, suggesting they can effectively bridge accessibility divides while maintaining competitive performance and fairness.

## introduction the rapid development of machine learning (ml) technologies, particularly large language models (llms), has led to major advancements in natural language processing (nlp, @xcite . while much of this advancement happened under the umbrella of the common task framework which espouses transparency and openness @xcite , in recent years, closed llms such as gpt-3 and gpt-4 have set new performance standards in tasks ranging from text generation to question answering, demonstrating unprecedented capabilities in zero-shot and few-shot learning scenarios @xcite . given the strong performance of closed llms such as gpt-4, many studies within the llm-as-a-judge paradigm rely on their scores as ground truth benchmarks for evaluating both open and closed llms @xcite , further entrenching the dominance of sota closed llms @xcite . along with closed llms, there are also llms where the pre-trained models (i.e., training weights) and inference code are publicly available ("open llms") such as llama @xcite as well as llms where the full training data and training code are also available ("open-source llms") such as olmo @xcite and prometheus @xcite . open and open-source llms provide varying levels of transparency for developers and researchers @xcite . access to model weights, training data, and inference code enables several benefits for the userdeveloper-researcher community, including lower costs per input/output token through third-party api services, support for local/offline pre-training and fine-tuning, and deeper analysis of model biases and debiasing strategies. however, the dominance of closed llms raises a number of concerns, including accessibility and fairness @xcite @xcite . the accessibility divide in this context can be understood in three dimensions: uneven availability due to geographic and economic barriers, prohibitive costs that limit adoption, and a lack of transparency that hinders research and innovation. in the llm space, corporate-driven commod-ification through monopolized apis and exclusive licensing is exacerbating the accessibility divide @xcite . these challenges are both technical and ethical, impacting who can access and benefit from the opportunities afforded by sota llms; those affected include researchers and practitioners residing in less affluent regions and/or complex sociopolitical environments. open and open-source llms such as llama 3, qwen 2.5, and olmo 2 provide greater transparency and customization potential @xcite @xcite . as these models improve in general benchmarking tasks, there is a need to systematically compare open and open-source llms with their closed sota counterparts on different assessment/scoring and generation tasks across various dimensions including performance and fairness. we aim to address this gap by conducting a comprehensive comparative analysis of eleven llms, encompassing closed, open, and open-source llms, across multiple text generation and evaluation tasks. the research questions (rqs) guiding this study are: rq1: how do different generations of open, open-source and closed llms compare in their assessment capabilities? rq2: when performing assessments/scoring, to what extent do closed and open llms exhibit biases? rq3: how comparable are open and opensource llms to their closed counterparts in terms of text generation capabilities? to answer these questions, we use automated essay scoring (aes) as our focal context. aes is well-suited for our research questions; it has been studied extensively by the nlp community @xcite , entails prompt-guided text generation, has readily available large-scale human testbeds with demographic information, and includes welldefined evaluation rubrics. our contributions are three-fold: (1) we provide empirical evidence of the trade-offs between accuracy, cost, and fairness for llms when performing assessment/scoring tasks; (2) we statistically and visually demonstrate the text generation capabilities of leading open, open-source, and closed llms; ## data, models, and experiments to answer our three research questions, we developed a robust analysis framework (figure 1 ). in the remainder of the section, we describe the data, models, and experiments in detail. ## discussion and conclusion this study contributes to the growing body of research exploring llm accessibility divides. while the emerging literature has made some strides in evaluating the performance, bias, and costs assofigure 6: input and output token cost of various llms across asap and fce. the y-axis is log-scaled for readability. costs calculated as of january 2025 additionally, the analysis of fairness of the models showed that top models maintained consistent ∆ scores across race and age, indicating a low propensity for demographic bias when provided with context (i.e., few-shot learning). open llms such as llama 3 offer substantial cost savings, being up to 37 times more costefficient than gpt-4. this cost advantage, combined with relatively comparable performance and fairness, positions newer open llms as attractive