ELECTRA and GPT -4o: Cost-Effective Partners for Sentiment Analysis (2025)

Abstract: Bidirectional transformers excel at sentiment analysis, and Large Language Models (LLM) are effective zero-shot learners. Might they perform better as a team? This paper explores collaborative approaches between ELECTRA and GPT-4o for three-way sentiment classification. We fine-tuned (FT) four models (ELECTRA Base/Large, GPT-4o/4o-mini) using a mix of reviews from Stanford Sentiment Treebank (SST) and DynaSent. We provided input from ELECTRA to GPT as: predicted label, probabilities, and retrieved examples. Sharing ELECTRA Base FT predictions with GPT-4o-mini significantly improved performance over either model alone (82.50 macro F1 vs. 79.14 ELECTRA Base FT, 79.41 GPT-4o-mini) and yielded the lowest cost/performance ratio ($0.12/F1 point). However, when GPT models were fine-tuned, including predictions decreased performance. GPT-4o FT-M was the top performer (86.99), with GPT-4o-mini FT close behind (86.70) at much less cost ($0.38 vs. $1.59/F1 point). Our results show that augmenting prompts with predictions from fine-tuned encoders is an efficient way to boost performance, and a fine-tuned GPT-4o-mini is nearly as good as GPT-4o FT at 76% less cost. Both are affordable options for projects with limited resources.

## introduction sentiment analysis-the computational study of opinions, attitudes, and emotions in text @xcite -has seen major advances from transformer architectures @xcite . bidirectional encoders like bert @xcite , roberta @xcite , and electra @xcite excel at sentiment analysis when finetuned, and large language models (llm) like gpt @xcite are strong zero-shot and few-shot learners @xcite . recent work has explored collaboration between these models, such as using gpt to augment data of minority classes before fine-tuning with roberta @xcite , using gpt for aspect extraction and roberta for sentiment scoring @xcite , and escalating to llms when roberta classification confidence was low @xcite . however, leveraging external knowledge of sentiment from fine-tuned encoders to enhance llms remains under-explored. this research investigates collaborative approaches between electra and gpt-4o models @xcite for three-way sentiment classification (negative, neutral, positive) of reviews. our research focused on the following hypotheses: providing predictions from a fine-tuned electra as context to a gpt model will improve classification performance (h1). the improvement in performance will be less for a fine-tuned gpt (h2). the format of predictions in the prompt will affect performance (h3). including similar examples in the prompt will improve performance (h4). these hypotheses build on electra's strength in capturing nuanced sentiment patterns when finetuned @xcite @xcite , and gpt's versatility through incontext learning @xcite @xcite -they can perform well across diverse tasks when given the appropriate context through prompting @xcite . although they may struggle with emotion and nuance @xcite , retrieved examples can improve performance @xcite . to test these hypotheses, we established four baselines and conducted 28 experiments across three sentiment classification datasets: stanford sentiment treebank (sst), and dynasent rounds 1 and 2. we used electra base/large and gpt-4o/4o-mini, each of which were fine-tuned (ft) on a merge of sst and dynasent reviews. we investigated the effects of different prompt augmentation scenarios using declarative selfimproving python (dspy) @xcite , a framework for programming language models. we started with a prompt to classify a review, and augmented it with knowledge from electra in the form of: the predicted class label, the probabilities of each class, similar reviews with their class labels, and combinations. we evaluated classification performance with the macro average f1 score, and cost-effectiveness by dividing total fine-tuning costs by the f1. our key insights are the following. sharing predictions boosted performance. augmenting gpt-4o-mini (not fine-tuned) with predictions from electra base ft significantly improved performance over either model alone. it also yielded the lowest cost/performance ratio. adding probabilities or examples did not help. using probabilities, or including few-shot examples, did not improve performance more than the predicted label alone for both gpt models. fine-tuned gpts performed best. gpt-4o ft-m alone achieved the highest overall performance on the merged test set, with gpt-4o-mini ft closely following at significantly lower cost. sharing predictions hurt fine-tuned gpts. when gpt models were fine-tuned, including electra predictions decreased performanceeven when fine-tuned with the same inference-time prompt that included the electra prediction. fine-tuning with the prediction for more epochs allowed gpt to discriminate better. performance improved, but the cost grew significantly. fine-tuned electra large outperformed base gpts. electra large fine-tuned was the best performing encoder model, and was better than both gpt-4o and gpt-4o-mini base models. these findings offer affordable options for projects with limited resources. if fine-tuning via api is an option, a fine-tuned gpt-4o-mini is nearly as good as gpt-4o ft at 76% less cost. alternatively, augmenting llm prompts with predictions from fine-tuned encoder models is an efficient way to boost performance. for projects that want to stay local, a fine-tuned electra large model is quite capable, and better than default gpts. the key contributions of this research are: ## models four models were fine-tuned and evaluated in this research, both individually and in collaboration with each other: electra base and large, and gpt-4o and 4o-mini. see electra @xcite , released with an apache 2.0 license, was chosen as the bidirectional transformer because its pre-training architecture gives it an advantage over mlms. it also outperformed roberta in early trials. we evaluated both the base (110m parameters) and large (335m parameters) variants. to function as a classifier, electra's output is sent through a mean pooling layer. a classifier head is appended with 2 hidden layers of dimension 1024, and a final output dimension of 3. swish glu @xcite was used as the hidden activation function, and dropout layers were added with a rate of 0.3. see appendix b for more details on the model architecture and hyper-parameters. for comparison and collaboration, two gpt models were used via openai's api: gpt-4o (ope-nai, 2024b) and gpt-4o-mini (openai, 2024c). although the full specifications are not public, they are state-of-the-art autoregressive language models with strong zero-shot capabilities. gpt-4o is described as a "high-intelligence flagship model for complex, multi-step tasks." gpt-4o-mini is described as an "affordable and intelligent small model for fast, lightweight tasks." ## methods our research progressed through the following stages. code and datasets are available at: https: //github.com/jbeno/sentiment . ## results our experiments revealed significant differences in performance across baseline, fine-tuning, and collaborative scenarios. see fine-tuning. fine-tuning improved performance across all models. electra base's mean macro f1 increased from 69.51 to 79.14, while electra large showed greater gains, improving from 67.94 to 82.76. this improvement is the result of fine-tuning all layers-the baselines had the same classifier head. fine-tuned gpt models had the highest scores (see figure 2 ), with gpt-4omini ft rising from 79.41 to 86.70, and gpt-4o ft-m achieving 86.99 with the minimal template. ## analysis h1. sharing predictions would boost performance. the significant improvement in gpt-4omini's performance when augmented with elec-tra base ft or large ft predictions strongly supports h1. we also saw a similar boost for gpt-4o with electra large ft predictions. however, following electra's predictions had mixed results. when gpt-4o-mini changed its decision and followed electra base ft in round one, it was correct 548 times and wrong 412 times (+136 net improvement, 57.08% success rate). when gpt-4o changed its decision and followed electra large ft, it was correct 521 times and wrong 481 times (+40 net improvement, 52% success rate). most of the improvement was in the neutral and positive classes (see figure 4 ). there was barely any improvement in the negative class, but importantly-it didn't worsen. electra base ft had more than double the incorrect negative predictions, but gpt-4o-mini did not follow them. the negative class was 21.46% of the merged dataset, so electra may not have learned it well. conversely, gpt-4o followed more of the negative predictions, and performance suffered. dynasent r1 was the dominant source of the merged dataset (80,488 samples, or 78.83%), and saw the most improvement. it could be that elec-tra learned this dataset the most, but it also represented less challenging reviews. h2. improvement would be less for fine-tuned gpts. h2 was supported more strongly than anticipated. for a fine-tuned gpt model, including the electra prediction actually decreased performance (see figure 1 ). initially, we thought this was because the fine-tuning context did not include the electra prediction in the prompt. but we still saw a decrease in performance (although less) when it was included. gpt was trained to be a helpful chat assistant that thinks through problems, evaluates information critically, and synthesizes knowledge. when presented with an electra prediction, it can decide when to follow the prediction and when to rely on its own judgment. in contrast, when finetuned specifically for sentiment classification, the model is optimized to map directly from input text to sentiment labels, potentially bypassing the critical thinking that made the base model effective at filtering electra's suggestions. figure 3: impact of context on mean f1 score it may also be that it takes more time to learn the nuances of when electra is right vs. wrong. when we extended fine-tuning from 1 to 5 epochs, gpt was better able to discriminate electra's predictions. the follow rate decreased from 96.34% to 89.66%, and the discrimination gapthe difference between following correct and incorrect predictions-went from 16.89% to 35.95%. although additional epochs improved performance, it comes with a significant increase to the cost: @xmath0251.17 vs. only @xmath10.38 vs @xmath20.12/f1 point). projects that need to stay completely local can fine-tune electra large, which outperformed both base gpt models. future work could explore optimization of inference-time prompts through dspy, and alternate system role instructions during fine-tuning. in addition, this collaborative approach could be extended to different datasets/domains, classification tasks, and model pairings. there may also be potential for including multiple predictions from an ensemble of models. a new collaborative scenario would be fine-tuning gpts on the electra output representations.