dean22029/pr_fm_llama3_8b_adapter

Conjoint choice adapter — meta-llama/Llama-3.1-8B-Instruct

A QLoRA adapter that predicts forced-choice conjoint outcomes: given a study context, a respondent's characteristics, and two profiles, it scores which profile that respondent chose.

It is a research baseline, trained on 115 published conjoint experiments from the preference_fm data-share bundle. It is not a general-purpose chat model — fine-tuned on a single answer token, it only does this one task.

The base model is gated. You need a HuggingFace account that has accepted the terms on meta-llama/Llama-3.1-8B-Instruct, and huggingface-cli login, before the example below will run.

Results

Evaluated with one forward pass per pair (no generation), 4-bit NF4 base. The label base rate is ~50% by construction, so null log loss is 0.693 and accuracy is pair-winner accuracy. "zeroshot" is the same base model with no adapter, on the same prompts.

Split	Run	n pairs	Accuracy	AUC	Log loss	Δ vs null	Brier
test_chrono	finetuned	58,910	0.577	0.612	0.6800	0.0131	0.2431
test_chrono	zeroshot	58,910	0.502	0.521	1.0682	-0.3751	0.3687
test_within	finetuned	61,120	0.654	0.715	0.6173	0.0758	0.2146
test_within	zeroshot	61,120	0.499	0.498	1.1652	-0.4721	0.3885

Two things to read carefully before quoting these numbers:

• Fine-tuning mostly buys calibration, not discrimination. The zero-shot base models already rank pairs nearly as well; their log loss is far worse than the base-rate null because they are stuck near one letter. The adapter moves mean P(A) back to ~0.5.
• Per-experiment results are highly heterogeneous. The overall test_chrono number is an average over 31 studies whose individual AUCs range from below chance to ~0.94. Rebuilding the evaluation (see Reproducing the data below) writes a by_experiment breakdown.

Training

Base	meta-llama/Llama-3.1-8B-Instruct
Method	QLoRA, 4-bit NF4 + double quant, bf16 compute
LoRA	r=16, alpha=32, dropout 0.05, on q,k,v,o,gate,up,down_proj
Optimiser	paged_adamw_8bit, lr 1e-4, cosine, warmup 3%
Batch	per-device 4, effective ~60 (A100 40GB, DDP)
Steps	6,000
Epochs completed	0.51
Max seq len	1024
Loss	masked to the assistant answer token only

The training set is 711,617 pairs. This run saw roughly 51% of it — training was stopped by a step limit, not by convergence on a full epoch.

Validation choice accuracy over training: 0.624 @ step 2,000, 0.637 @ step 4,000, 0.643 @ step 6,000. It is flat over the last few evaluations, so the step limit is not obviously leaving accuracy on the table.

log_history.json in this repo has the step-by-step loss and eval metrics.

Input format

The adapter was trained on exactly this surface. Paraphrasing the system prompt or reordering the user block puts you out of distribution.

[system]
You simulate survey respondents in conjoint experiments. Given the study context, the respondent's characteristics, and two options, predict which option the respondent chose. Answer with exactly one letter: A or B.

[user]
Conjoint study: <description>
Country: <country>
Year: <year>

Respondent characteristics: <name: value; name: value; ...>

The respondent chose between the following two options.

Option A:
- <factor>: <level>
  ...

Option B:
- <factor>: <level>
  ...

Which option did the respondent choose?

The assistant target is a single letter, A or B.

Scoring convention

eval_example.py in this repo is a runnable version of all of this.

• Do not generate. One forward pass; take logits[:, -1, :], index the single-token ids for "A" and "B", softmax over those two -> P(A).
• Apply the chat template with add_generation_prompt=True, then tokenize with add_special_tokens=False — the template already added them.
• Use padding_side="left" for batched scoring.
• Randomise A/B orientation per pair, or average both orderings. Training data is orientation-randomised; a fixed order reintroduces position bias.
• Load the base model first, then attach the adapter. adapter_config.json here points at meta-llama/Llama-3.1-8B-Instruct, but keep the 4-bit config above to reproduce the numbers — loading the base in bf16 will shift them.

Splits and comparability

SPLITS.md and splits_summary.csv in this repo record exactly which experiments were trained on. The short version: training used every experiment with experiment_year <= 2021, so a test set built with a lower cutoff overlaps our training data. Check SPLITS.md before comparing these numbers against a model split a different way.

Reproducing the data

These adapters were trained on prompts built from the preference_fm conjoint data-share bundle (127 experiments; 115 usable). That bundle is not redistributed here and is not ours to redistribute — 113 of its 127 experiments carry no declared licence, and the underlying replication archives come from Dataverse and similar sources under their own individual terms. You need your own copy of the bundle to rebuild the data.

The scripts are bundled in pipeline/ in this repo.

Steps 1-3 are CPU-only. Step 1 needs R with jsonlite; steps 2-3 need tqdm and nothing else; step 4 needs the GPU stack from the example above.

# 1. experiment.rds -> JSON
Rscript pipeline/export_experiments.R --zips <bundle>/experiment_zips --out exported

# 2. reviewed covariate codebooks (optional; improves how X is verbalised)
python pipeline/make_codebooks.py --exported exported --out codebooks

# 3. prompts + splits
python pipeline/build_dataset.py --exported exported --out llm_dataset \
    --cutoff-year 2021 --seed 42

# 4. score this adapter
python pipeline/evaluate_qlora.py --model <short-name> --adapter <this repo> \
    --data llm_dataset/test_chrono.jsonl --out-prefix results/test_chrono

Step 3 must use --cutoff-year 2021 --seed 42 to reproduce our splits. Compare the splits_summary.csv it writes against the one in this repo: if they differ, the numbers on this card do not apply to your build. SPLITS.md explains what to check.

Note that the evaluated results on this card were produced without the codebooks of step 2 — every published number uses the raw column-dummy rendering of respondent covariates.

Licence

Built with Llama. Use of this adapter is additionally governed by the Llama 3.1 Community License.

The adapter weights are a derivative of meta-llama/Llama-3.1-8B-Instruct and inherit its terms (llama3.1). The training data comes from third-party replication archives with their own individual licences and is not redistributed here.