valory/Olas-Predict-R1-14B

Olas Predict R1 14B — forecasting LoRA adapter

Developed by Valory, a core contributor to Olas, this LoRA adapter specialises DeepSeek-R1-Distill-Qwen-14B in forecasting binary prediction-market outcomes. It requires the base model and a prompt containing the market question, background information and retrieved evidence. It generates reasoning followed by a JSON answer containing the estimated probability of a YES outcome. The model does not retrieve evidence itself.

In a separate evaluation of 2,628 markets that closed after the training period, this model reduced Brier score by 20.3% relative to the base model, from 0.2249 to 0.1792, and increased accuracy from 71.4% to 75.8%. GPT-4.1, given the same evidence, achieved a Brier score of 0.1914 and accuracy of 75.4%. These results support comparable forecasting performance under the evaluation conditions. See Evaluation on later markets.

Model details

Base model	unsloth/DeepSeek-R1-Distill-Qwen-14B (mirror of deepseek-ai/DeepSeek-R1-Distill-Qwen-14B)
Adapter type	LoRA, rank 16, alpha 16, dropout 0
Target modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Trainable parameters	68,812,800 (approximately 69 million)
Adapter download size	275 MB (adapter_model.safetensors)
Stored adapter precision	float32 (all 672 tensors)
Training precision	Base loaded in 4-bit (bitsandbytes) via Unsloth, adapter trained in bf16 with float32 master weights
Context	8192 tokens total, prompts capped at 2048 tokens
Language	English

The adapter was trained against the 4-bit base, so adapter_config.json lists unsloth/deepseek-r1-distill-qwen-14b-unsloth-bnb-4bit as base_model_name_or_path. It applies cleanly to the bf16 base as well, and that is how it was evaluated. Load the base explicitly rather than relying on the config value.

How it was trained

Training data: historical prediction-market questions served by the Olas mech, each stored with the exact prompt the production tool sent to its LLM and the market's final resolution. Two stages:

1. Supervised fine-tuning (SFT). An initial reinforcement-learning checkpoint (GRPO from the base model, same recipe as stage 2) generated reasoning traces for a separate set of markets carved out of the training data by market id, sampling four candidates per prompt at temperature 0.7. We retained traces with a readable probability that outperformed a constant base-rate prediction (Brier skill above zero against a base rate of 0.25). Outcomes were used to select traces, but were not supplied to the model generating them. All passing traces were kept rather than only the most confident one per market, to avoid teaching over-confidence. We then trained an adapter on 5,000 retained examples for one epoch, using a learning rate of 2e-4, before the final reinforcement-learning stage.
2. Reinforcement learning (GRPO). Starting from the SFT adapter, we trained for one epoch on 20% of the training split, using a learning rate of 2e-5 and four sampled completions per prompt. The reward is the Brier skill score against a constant base-rate prediction: (base_rate − y)² − (p_yes − y)², where y is the resolved outcome and base_rate is the YES rate of the training split. Answers without a parseable probability receive a fixed penalty of −2.0, below the worst achievable valid reward. A secondary format reward (weight 0.1) rewards a closed <think> block with a parseable answer.

Frameworks: Unsloth, TRL, PEFT, on a single A100 80GB.

Evaluation

Internal held-out evaluation

The development dataset contains 214,529 prediction examples across 5,116 resolved markets, split by market to prevent the same market appearing in training and testing. The test split contains 21,583 examples across 513 markets. Excluding 7,239 examples that exceed the 1,900-token prompt limit leaves 14,344 evaluated examples across 467 markets. Every example carries equal weight.

Decoding is greedy (temperature 0) with a 1,024-token output budget. The classification threshold for accuracy is the YES rate of the evaluated examples (0.244). Its balanced accuracy and calibration results are separate from the later-market evaluation reported below.

System	Brier (lower is better)	ECE (lower is better)	Balanced accuracy (higher is better)
Base model, no adapter	0.248	0.249	0.509
Production tool (GPT-4.1-based)	0.308	0.284	0.612
This model	0.252	0.204	0.614

• Balanced accuracy averages the fraction of YES outcomes and the fraction of NO outcomes correctly classified at the stated threshold. A score of 0.5 represents chance-level balanced accuracy; higher is better.
• Brier score measures the mean squared difference between predicted YES probabilities and actual outcomes. Lower is better. In this internal test, the adapter improves Brier score relative to the production tool, but not relative to the base model. A constant prediction at the training-split YES rate (0.216) would score 0.185 on this test set, below all three systems, because the base model's malformed answers and the models' over-confidence both cost more than a flat prior on this metric.
• Expected calibration error (ECE) measures discrepancies between predicted probabilities and observed outcome frequencies within probability bins. We calculated ECE using 10 equal-width bins. Lower is better.

Formatting reliability. On the internal test, the released adapter produced 11 malformed answers out of 14,344 examples, compared with 11,138 for the base model, which mostly ignores the answer format. An answer was considered invalid if no probability between 0 and 1 could be parsed from the text after the last </think> tag, in either the JSON or the XML-tag answer format. Invalid answers were scored as a prediction of 0.5 and included in all reported metrics.

Evaluation on later markets

A separate evaluation covered 1,247 Omen markets and 1,381 Polymarket markets that closed between July 16 and August 27, 2026, after the training data ended.

All models received the same frozen evidence collected before market resolution, with no web access or later news. The evaluation contained 4,347 evidence contexts. Scores were averaged within each market so that every market had equal weight. Answers without a usable probability received a Brier score of 1. Open models had a 1,024-token output allowance; GPT-4.1 had 4,096 tokens.

System	Brier score (lower is better)	Accuracy (higher is better)
Base model	0.2249	71.4%
GRPO without SFT warm start	0.1947	72.9%
SFT warm start followed by GRPO (this model)	0.1792	75.8%
GPT-4.1, same evidence	0.1914	75.4%
Market price at evidence capture	0.1264	83.0%

This model's Brier-score difference relative to the base model was −0.0457, with a 95% confidence interval of [−0.0539, −0.0375]. Relative to GPT-4.1, the difference was −0.0122 [−0.0241, −0.0005]. Confidence intervals used 10,000 bootstrap resamples, grouping markets linked to the same real-world event.

The improvement over the base model was clear on both platforms. The advantage over GPT-4.1 was smaller and varied by platform and metric. Market prices achieved better aggregate forecasting scores than every model.

These results use uncalibrated model predictions. Platt scaling improved the internal test results but did not carry over to the separate evaluation, so it was not retained.

Input and output format

Send one user message with no system message, using the complete prompt template at https://github.com/valory-xyz/mech-predict/blob/v0.21.32/packages/valory/customs/finetuned_prediction/finetuned_prediction.py. Replace the market question, background and retrieved-evidence fields while preserving the answer instructions. The PROMPT variable in the examples below refers to this completed template.

Two production prompt variants appear in the training data. One wraps the evidence in a <background> block. The other wraps it in an <additional_information> block, followed by a first-stage reasoning section, and ends with the answer instructions. The skeleton of the second variant:

Here is the user's question: <question>
Here is some additional information that may be relevant to answering the question: <additional_information> ARTICLE 0, URL: ..., CONTENT: ...

ARTICLE 1, URL: ..., CONTENT: ... </additional_information>

Please carefully read the user's question and the additional information provided. Think through the problem step-by-step ...
<reasoning></reasoning>
////
You will be evaluating the likelihood of an event based on a user's question and reasoning provided by another AI.
The user's question is: <user_input> <question> </user_input>

The reasoning from the other AI is: <first-stage reasoning>

Carefully consider the user's question and the provided reasoning. Then, think through the following:
 - The probability that the event specified in the user's question will happen (p_yes)
 - The probability that the event will not happen (p_no)
 - Your confidence level in your prediction
 - How useful the reasoning was in helping you make your prediction (info_utility)
...

The internal evaluation used this prompt format. Performance with other formats should be assessed separately.

The model replies with a <think>...</think> block followed by a JSON answer:

{"p_yes": 0.27, "p_no": 0.73, "confidence": 0.8, "info_utility": 0.6}

Parse the JSON answer after the closing </think> tag and use p_yes as the YES probability. Validate that p_yes and p_no fall between 0 and 1 and sum to 1 within numerical tolerance. Treat missing, incomplete or invalid JSON as a failed forecast.

The additional fields follow the prompt's own definitions: confidence is the model's self-reported confidence in its prediction, and info_utility is its self-reported rating of how useful the provided evidence and reasoning were. Neither field is used by the training reward or the evaluation, and their quality has not been assessed.

The examples allow up to 1,024 generated tokens. This limit does not guarantee a complete answer; detect truncation (no </think> tag or no JSON after it) and handle it explicitly.

How to run

vLLM with runtime LoRA (recommended)

pip install vllm
vllm serve unsloth/DeepSeek-R1-Distill-Qwen-14B \
  --enable-lora \
  --lora-modules forecaster=valory/Olas-Predict-R1-14B \
  --max-lora-rank 16 \
  --dtype bfloat16 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.90

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-used")
resp = client.chat.completions.create(
    model="forecaster",
    messages=[{"role": "user", "content": PROMPT}],
    temperature=0,
    max_tokens=1024,
)
print(resp.choices[0].message.content)

Use bfloat16 to reproduce the reported evaluation. Results for float16 serving were not evaluated.

Transformers + PEFT

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = "unsloth/DeepSeek-R1-Distill-Qwen-14B"
tok = AutoTokenizer.from_pretrained("valory/Olas-Predict-R1-14B")
model = AutoModelForCausalLM.from_pretrained(base, torch_dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(model, "valory/Olas-Predict-R1-14B")

inputs = tok.apply_chat_template(
    [{"role": "user", "content": PROMPT}],
    add_generation_prompt=True, return_tensors="pt",
).to(model.device)
out = model.generate(inputs, max_new_tokens=1024, do_sample=False)
print(tok.decode(out[0][inputs.shape[1]:], skip_special_tokens=True))

To ship a single checkpoint, call model.merge_and_unload() and save, then serve the merged directory with vllm serve without the LoRA flags.

Hardware

Setup	GPU memory	Example GPUs
bf16 base + adapter	~28 GB weights, 40-50 GB with KV cache at 8k context	A100 40/80GB, H100, L40S 48GB
4-bit base (--quantization bitsandbytes --load-format bitsandbytes)	~14 GB	RTX 4090, A10, L4

Four-bit serving reduces weight memory requirements. Its effects on forecast quality, latency and throughput depend on hardware and serving configuration and were not measured. The model-card evaluation used the bf16 base with the adapter.

Memory and concurrency requirements depend on input length, generated tokens and batch size. The hardware figures above are approximate planning estimates, not measured benchmarks.

Limitations

This adapter is intended for English-language binary forecasting questions supplied with background information and retrieved evidence. Performance depends on the relevance, quality and timing of that evidence.

The results do not establish general equivalence to GPT-4.1 or performance across all question types and domains. The later-market evaluation selected researchable binary questions and excluded several categories.

Trading results in the accompanying study are simulations, not live performance. Forecasting scores alone do not establish profitability, which also depends on market prices, fees, execution and strategy. Market prices achieved better aggregate forecasting scores than every evaluated model.

The model remains over-confident on the internal test (ECE 0.20). No calibration layer is shipped.

Lower serving costs relative to the production tool have not yet been demonstrated in the supplied evidence.

License

This adapter is released under the MIT License.

It is built on DeepSeek-R1-Distill-Qwen-14B, released by DeepSeek under the MIT License (Copyright (c) 2023 DeepSeek). That model is itself a fine-tune of Qwen2.5-14B, Copyright (c) Alibaba Cloud, licensed under the Apache License 2.0. This adapter modifies the Qwen2.5-derived weights by fine-tuning; the original license and copyright notices are retained as required by both licenses.