jth2/fin_model

Introduction:

Presently, it is difficult for people to educate themselves on financial concepts without facing significant hurdles. Tools & resources are often locked behind some form of paywall, and other forms of gatekeeping make it difficult for people to develop their knowledge. As financial literacy is something that has a large impact on the ability of an individual to participate in their communities, it seems crucial that there be free tools to enable education as well as those already offered by businesses. For this project, the focus was to develop a model which could assist in answering and explaining the solution behind a financial question so that people could develop their understanding. To do so, a base model was pulled from DragonLLM/Llama-Open-Finance-8B (the name of the repository was recently updated to OVHaiLLM/Llama-Open-Finance-8B) and tuned to attempt to develop a model better at these tasks.

Data:

For this project, the model was finetuned on a FinRAG based dataset, which can be found in the references below. For the project, the columns were recombined to form the full prompt in the following order (pre_text, table, post_text, question, choices). The model was then prompted to attach a step-by-step explanation to each problem to provide the logic necessary to achieve the correct solution before the training process. A randomized set of 1000 of these questions were pulled from the dataset using a random seed of 42. These questions were then split into the train, validate, and test datasets in an 80%/10%/10% split respectively.

Methodology:

For this project, LoRA was used to train the model. As there was the potential for resource constraints given the size of the project, it seemed sensible to utilize a method that took a more targeted approach to improving the model. In addition, there were issues given the large size of the data, which often contained multiple pieces of necessary information to answer the questions and thus, could not be shorted for training purposes. After examining different combinations of hyperparameters, the following were used for training:

LoRA – R: 64 LoRA – Alpha: 64 LoRA – Dropout: 0.5 Learning Rate: 0.00001 Epochs: 2 Evaluation Steps: 10

Evaluation

Model	Flare CFA	ConvFinQA	FinanceBench	Test Set
Llama-Open-Finance-8B	60.66%	0%	1.33%	1%
Fine-Tuned	59.33%	2%	2%	0%
Qwen3-8B	71.33%	0.66%	0%	2%
Gemma-4-E2B-it	35.33%	0%	0.66%	1%

As a point of comparison, the above Qwen and Gemma models were selected due to their similar parameter counts as well as reasoning capabilities to the baseline model. The baseline model, the trained model, and the comparison models were evaluated against 150 randomly selected rows from the above benchmarks. The random seed used was also 42. The Flare CFA multiple choice benchmark was used to estimate the capability to answer multiple choice questions. The general ConvFinQA and FinanceBench benchmarks, as well as the test set, were assessed using an outside judge. The model used for the judge was Qwen/Qwen3-0.6B. It was asked to evaluate the response and provide a score of 1 if the answer was correct, 0.5 if somewhat correct, or a score of 0 if wrong. Relative to the other models, the finetuned model performed slightly better than the baseline when it came to the general benchmarks, but slightly underperformed when it came to the multiple-choice benchmark. Although Qwen performed better on the multiple-choice portion, it could be a result of the questions pulled from the benchmark datasets.

Usage and Intended Uses:

This model should be used to help answer financial practice questions and explain how to arrive at the correct answer. This model was trained with long-context multiple choice questions. Please refer to the below for an example of how to load the model.

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_model = "DragonLLM/Llama-Open-Finance-8B"

adapter = "jth2/fin_model"

tokenizer = AutoTokenizer.from_pretrained(base_model)

model = AutoModelForCausalLM.from_pretrained(base_model)

model = PeftModel.from_pretrained(model, adapter)

Prompt Format:

For the utilization of this model, the prompt should be formatted with the context before the question, and the question should be followed by potential answer choices. For an example of how to format a prompt, please refer to the below.

context = "Three apples currently costs 75 cents at the supermarket. Sally has two dollars."
question = "What is the maximum amount of apples Sally can buy?"
choices = "A. 8 B. 6 C. 4 D. 2"
prompt = context + question + choices

Expected Output Format:

The expected output format is the selection of an answer from the multiple choice bank, as well as any necessary explanation. For an example, please refer to the below.

Answer: 8. As three apples cost 75 cents, that means each apple is 25 cents by itself. There is 100 cents in a dollar, and Sally has two, which means she has 200 cents. 200 / 25 = 8 apples.

Limitations:

As with any model, there are potential limitations that users should be aware of. As models are susceptible to hallucinations, users should independently verify outputs when using the tool. In addition, it should be used in conjunction with other resources and not alone as there may be information the model was not trained on. Also, as there was not a material increase on the benchmark tasks relative to the baseline model, users should make sure this model is relevant for their own task before using it in place of another.

References:

Baseline Model: https://huggingface.co/OVHaiLLM/Llama-Open-Finance-8B

Benchmark Models - Qwen: https://huggingface.co/Qwen/Qwen3-8B

Benchmark Model - Gemma: https://huggingface.co/google/gemma-4-E2B-it

Judge Model: https://huggingface.co/Qwen/Qwen3-0.6B

Training Dataset - FinRAG: https://huggingface.co/datasets/trismik/FinRAG

Benchmark Dataset - Flare CFA: https://huggingface.co/datasets/TheFinAI/flare-cfa

Benchmark Dataset - ConvFinQA: https://huggingface.co/datasets/FinGPT/fingpt-convfinqa

Benchmark Dataset - FinanceBench: https://huggingface.co/datasets/PatronusAI/financebench