thurstonpowers/xll-citation-verifier

XLL Citation Verifier

An experimental Laya decision model that grades an existing selected citation occurrence. It answers three fixed questions:

1. Citation recognition: is this a source reference, even if incomplete?
2. Usable span: does the selected span preserve the target and explicit qualifiers?
3. Source type: statute, regulation, Act, internal reference, manual, etc.

Supply an existing candidate occurrence. The model returns recognition, completeness and source-type decisions. Use the included questions.json unchanged alongside the weights.

Loading and inference

Use laya==0.4.2, transformers==4.57.1, PyTorch and safetensors. The evaluated Spark runtime used NVIDIA PyTorch/CUDA; native training/evaluation used FP16 autocast, whereas the public SDK used eager CUDA BF16. Predictions can vary across devices and precisions. A separate CPU release smoke check tests package loading and inference.

Load this package with the Laya SDK, using its native encoder/tokenizer and decision-head layout. The included helper provides source formatting, checksum verification and complete-input checks.

import json
from huggingface_hub import snapshot_download
import laya

# Pin the full immutable release commit from the repository's Files and versions.
folder = snapshot_download("thurstonpowers/xll-citation-verifier", revision="<release-commit>")
# inference.py is included in that folder; add folder to sys.path or copy/import it.
import sys
sys.path.insert(0, folder)
from inference import build_state, verify_package, predict

questions = verify_package(folder)
agent = laya.load(folder, device="cpu", backend="eager", compile=False, fast=False)
raw = "The applicant shall comply with section 4-1 of the Connecticut General Statutes."
quote = "section 4-1 of the Connecticut General Statutes"
start = raw.index(quote)
state = build_state(raw, start, start + len(quote), section="example")
response = predict(agent, questions, state)
print(json.dumps(response["answers"], indent=2))

The same helper supports device="cuda"; ensure CUDA exists rather than silently accepting SDK fallback. The CLI rejects missing/invalid immutable Hub revisions and verifies all release file hashes. Example for a downloaded local release:

python inference.py --model ./downloaded-model --input source.txt \
  --start 32 --end 79 --section example --device cpu

Choose the actual occurrence's offsets; the CLI numbers above correspond to the example string only. Offsets use Python Unicode code-point indexing. Decode original UTF-8 bytes while preserving newlines. Keep the original text/quote unchanged and select the specific occurrence being graded.

Input contract

The trained state includes containing jurisdiction/type and section identity, the selected quote and original offsets, 48 immediate boundary characters on each side, and an explicitly partial source passage (220 characters before, 260 after) with selected-span markers. Federal inputs use jurisdiction="United States" and the original containing-track expression. Current evaluation covers Connecticut and federal material; other-state evaluation remains open.

The saved native budgets are 8,192 complete sequence tokens / 2,048 head tokens, with a native 48-token option-text cap. The helper checks every question's complete head/options, native fit and SDK truncation statistics. Overflow must be held explicitly. The verifier reads the selected span and bounded surrounding evidence passage.

Outputs

Recognition/span labels are a, b, u (yes, no, uncertain); source-type labels are defined in questions.json. Incomplete references may be recognition-positive but span-negative. Numeric cuts, omitted governing qualifiers and range endpoints are incomplete; complete coordinated list members can be usable.

The SDK returns a probability vector plus two different confidence measures: answer_confidence is calibrated top-answer confidence; confidence is entropy-derived. Use the calibrated answer confidence for confidence diagnostics and thresholds. Applications must preserve uncertainties, upstream extraction failures and known review holds independently.

Training

Dataset: thurstonpowers/xll-citation-verifier, with the frozen optimization, calibration and evaluation splits and annotation provenance.

• Base: convaiinnovations/laya, revision 7b928d828b7b0e022f929d9bd2e44165aa270148, ModernBERT-large encoder.
• Full encoder/head fine-tuning, two epochs / 3,217 actual optimizer updates.
• 38,390 admitted examples: 28,390 CT regulation candidate occurrences plus 10,000 federal examples, balanced at 2,000 per containing source category.
• Native hard-label supervision from user-accepted provisional CLEF labels and provenance-separated assistant/rule repairs. Training targets are hard labels.
• Optimization 34,307, calibration 1,511, evaluation 2,572 examples. Source families/chapter groups and exact source hashes are disjoint across splits.
• Encoder LR 2.5e-5, head LR 1e-4, effective question-item batch 64, seed 102301.
• Exported FP16 weights were checked against the native FP16-rounded parameters and reloaded. Evaluation used the full exported/reloaded model.

CT was excluded from the upstream generative extractor's training, but is included in this verifier's training. The verifier's original unseen CT reserve is identified separately below.

Evaluation

Internal source-disjoint provisional-reference agreement

Question	Agree / items	Agreement
Recognition	2,565 / 2,572	99.73%
Usable span	2,540 / 2,572	98.76%
Source type	2,549 / 2,572	99.11%
All question items	7,654 / 7,716	99.20%

All three labels agreed for 2,525/2,572 cases. Span mistakes included 18 of 304 reference-negative spans accepted and 14 of 2,268 positives rejected. These scores measure agreement with provisional supervision. Expert-gold legal accuracy remains an open evaluation task.

Original unseen CT reserve

The original 3,644-section reserve has zero document/chapter-group/exact-text overlap with optimization, calibration, internal evaluation and teacher-selected sources. All 8,929 prepared exact occurrences received valid SDK responses, with no missing outputs or truncation. Failed/empty source extractions remain in the original section denominator; 1,759 sections had zero candidates, 16 had failed or incomplete extraction, and 141 quote/input holds were preserved.

A uniform 100-candidate-occurrence sample, seed 2026110901, was individually assistant-reviewed and frozen before model answers were inspected:

Question	Agree / scorable judgments
Recognition	100 / 100
Usable span	98 / 100
Source type	98 / 99

One source-type taxonomy judgment was unresolved and excluded before scoring; the original sample remains 100. There were two false span approvals and no false span rejections. The type sample was mostly state regulations (43), state statutes (37), internal references (7) and Acts (6), with only two CFR references and one manual. Broader source-type coverage remains an open evaluation task.

Known mistakes: approving Public Law 90-321 while omitting explicit Title I (confidence 0.9956); approving United States Code inside the publication title United States Code Service; inferring state_statute for bare Chapter 60 without evidence establishing its source class (confidence 0.9929).

These are assistant-reference agreements from an occurrence-uniform sample. Some cases share source documents. The remaining 8,829 predictions are unreferenced. Whole-document citation recall, expert legal accuracy and nationwide generalization remain open evaluation tasks.

Limitations, provenance and license

Treat this as experimental legal-data tooling. Rare source types, other-state generalization, ambiguous publication titles and evidence-sufficiency abstention need further evaluation. High confidence can accompany real errors. Official publications and qualified human legal review remain authoritative.

The release includes FP16 weights, tokenizer/encoder configuration, exact questions, inference contract/helper, aggregate evaluation, provenance and SHA-256 checksums.

Weights SHA-256: efe5338fe2ec926ad6bf214dba0d03e3b5e584ec7767eacf3f44a93380e8847b. Questions SHA-256: bda0ecb8cda7f7502dce129de2fc8d6e92c3a28d924f15c6902b2483649db552.

Apache 2.0; derived from Convai Innovations' Laya and Answer.AI's ModernBERT. See LICENSE, NOTICE and provenance/BASE_MODEL_CARD.md. Fine-tuning and release documentation by Thurston Powers.

Replication dataset

Training and evaluation data: thurstonpowers/xll-citation-verifier. Use the model-specific contract and splits described in the dataset card.