RAIC — FTT (Feature Tokenizer Transformer) for UK Biobank CVD risk classification

A DistilBERT-style transformer adapted to tabular clinical data. Each row (participant) is treated as a "sentence" whose "tokens" are its features:

• Continuous features are embedded via a per-feature learned weight vector, scaled by the (min-max normalized) value (num_input_val * variable_weights_embeddings[num_input_ids]).
• Categorical features are embedded via a per-value lookup table (variable_weights_embeddings[cat_input_ids]).
• A separate variable/bias embedding table encodes which feature slot ("variable") each token corresponds to, acting like a position embedding (variable_bias_embeddings[num_variable_ids | cat_variable_ids]).

The model was trained to predict 5-year incident cardiovascular disease (CVD) from UK Biobank baseline assessment data (YEAR_5_CVD, binary), using 69 continuous and 76 categorical features.

This checkpoint corresponds to the local experiment 20240603_1_UKB_CVD_cls_hp_search.

Data/label note: No participant-level data is included in this repository — only model weights, config, tokenizers, and aggregate (variable-level) min/max normalization statistics used at training time. UK Biobank data governance means this model should only be applied to appropriately approved data/use cases.

Model details

• Architecture: FTTForSequenceClassification (DistilBERT encoder, 8 layers, 16 heads, hidden dim 128, feed-forward dim 1024)
• Task: binary sequence classification (num_labels=2)
  • 0: no 5-year CVD event
  • 1: 5-year CVD event
• Custom modeling code lives in modeling_ftt.py and is loaded automatically via trust_remote_code=True (see auto_map in config.json).

Files

File	Purpose
config.json	Model hyperparameters + auto_map for custom-code loading
modeling_ftt.py	FTTConfig, FTTModel, FTTForSequenceClassification, FTTForMaskedLM
model.safetensors	Fine-tuned classification weights
ftt_variable_weights_tokenizer_20240530/	Maps each continuous-variable name and each categorical (variable, value) pair to a token id used for the weight embedding table
ftt_variable_bias_tokenizer_20240530/	Maps each variable name to a token id used for the bias/position embedding table
train_continuous_variables_min_max_values.json	Per-continuous-variable [min, max] from the training set, used to min-max normalize raw values into [1, 3] before scoring (see test_upload.py)

Usage

from transformers import AutoConfig, AutoModelForSequenceClassification, AutoTokenizer

repo_id = "thinkhong/RAIC"

config = AutoConfig.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModelForSequenceClassification.from_pretrained(repo_id, trust_remote_code=True)
model.eval()

weights_tok = AutoTokenizer.from_pretrained(repo_id, subfolder="ftt_variable_weights_tokenizer_20240530")
bias_tok = AutoTokenizer.from_pretrained(repo_id, subfolder="ftt_variable_bias_tokenizer_20240530")

The model's forward() does not take input_ids/text. Instead it expects:

• num_input_val (float): normalized continuous values, [CLS] slot = 1.0
• num_input_ids (long): weight-tokenizer ids for each continuous variable's name, [CLS]-prefixed
• num_variable_ids (long): bias-tokenizer ids for each continuous variable's name, [CLS]-prefixed
• cat_input_ids (long): weight-tokenizer ids for each categorical variable_value pair present
• cat_variable_ids (long): bias-tokenizer ids for each categorical variable's name
• attention_mask (long): 1 for real tokens (1 CLS + n_continuous + n_categorical), 0 for padding

See test_upload.py for a full worked (dummy-value) example, including how to normalize a raw continuous value with train_continuous_variables_min_max_values.json.

Testing

Two test scripts are included:

• test_upload.py — a runnable demo/smoke test. Downloads the model from the Hub with trust_remote_code=True, builds a full synthetic example, and prints the logits/predicted class.

  pip install transformers huggingface_hub torch
  python test_upload.py --repo-id thinkhong/RAIC
  

• test_ftt_hub.py — a pytest suite with pass/fail assertions (architecture, param count, tokenizer vocab sizes, forward-pass shape, determinism, and an exact-match check against the reference logits recorded at publish time).

  pip install pytest transformers huggingface_hub torch
  pytest test_ftt_hub.py -v
  

Verification

This checkpoint was uploaded from the original training output directory and verified to produce bit-identical logits to the source checkpoint on an identical input before publishing (see test_upload.py).