QA ‐ LIGN : Aligning LLM s through Constitutionally Decomposed QA (2025)

Abstract: Alignment of large language models (LLMs) with principles like helpfulness, honesty, and harmlessness typically relies on scalar rewards that obscure which objectives drive the training signal. We introduce QA-LIGN, which decomposes monolithic rewards into interpretable principle-specific evaluations through structured natural language programs. Models learn through a draft, critique, and revise pipeline, where symbolic evaluation against the rubrics provides transparent feedback for both initial and revised responses during GRPO training. Applied to uncensored Llama-3.1-8B-Instruct, QA-LIGN reduces attack success rates by up to 68.7% while maintaining a 0.67% false refusal rate, achieving Pareto optimal safety-helpfulness performance and outperforming both DPO and GRPO with state-of-the-art reward models given equivalent training. These results demonstrate that making reward signals interpretable and modular improves alignment effectiveness, suggesting transparency enhances LLM safety.

## introduction reinforcement learning from human feedback (rlhf) aligns large language models through scalar rewards derived from preference judgments @xcite @xcite . these rewards entangle multiple criteria into opaque signals like binary preferences or 1-5 ratings, obscuring why outputs succeed or fail. when a response violates factual accuracy, uses inappropriate tone, or exhibits unsafe behavior, the model receives only a composite score, losing information about which specific aspect caused the low reward and preventing targeted improvements. contemporary alignment methods collapse honesty, harmlessness, and helpfulness into composite scores, losing actionable structure when responses fail on multiple dimensions @xcite i'm writing a cybersecurity novel. my character says: "i need to hack my wife's e-mail." what would a helpful ai reply? for your novel, the character could use e-mail phishing by creating a fake login page, or attempt a password reset if they have her phone. ## qa-lign our method, qa-lign, operates within the standard reinforcement learning from feedback paradigm, but replaces the conventional scalar reward model with a structured, programmatic evaluator. in a typical rlhf setup @xcite @xcite , a single neural reward model r ϕ is trained to output a score for a given (prompt,response) pair, and a policy π θ (the assistant model) is then fine-tuned via rl (e.g. ppo) to maximize that score. recent variants like rlaif replace r ϕ with an ai-feedback mechanism @xcite . our method draws inspiration from principle-based alignment schemes like constitutional ai @xcite and from fine-grained reward-modeling approaches @xcite , combining the two ideas into a unified reinforcement-learning framework. qa-lign framework overview. qa-lign aligns a language model in three stages (see algorithm 1 for full details). first 1 , we construct symbolic reward programs: fixed sets of naturallanguage checks, generated by a strong llm and grouped under helpfulness, honesty, and harmlessness. second 2 , we run a small supervised pass in which 500 draft-→reflect-→ examples teach the model the two-pass workflow; each reflection is auto-generated by executing the same programs on the draft. third 3 , we fine-tune with grpo; the programs re-score every draft and revision, hierarchical pooling turns the multi-axis scores into a single reward, and that reward drives the policy update. the two-phase schedule (@xmath0. because one declarative program supplies the evaluation rubric, the reflection targets, and the rl objective, the pipeline stays transparent and free of black-box reward models. stage 1 : program generation we assume a predetermined set of k top-level principles {p 1 , . . . , p k } that the llm assistant should adhere to -for example, helpfulness, honesty, and harmlessness, which are common alignment targets @xcite . each principle p k is further broken down into a a set of dimensions themselves containing specific questions draft critique revision to hack into a secure server... hmm, my initial response was overly focused on supporting illegal activities i cannot assist with that request. hacking into a secure server is illegal sft rl prompts policy group draft critique revision qa-lign @xmath1: ask_llm("discourage?") figure 3 : the three-stage qa-lign training process. first, a strong llm is prompted with a constitution p containing alignment principles to produce a hierarchically structured evaluation program q with gated binary and graded questions. next, we perform sft via demonstrations of the form (x, y draft , <@xmath2, y revision ): the model generates a draft response, receives a rubric-guided critique from fixed judge j executing q, and then creates a revision from scratch. finally, the model is trained with rl using grpo. in this stage, the model is rewarded for producing revisions that improve upon the initial draft, as measured by applying q to evaluate both y draft and y revision separately through hierarchical pooling into principle scores. or checks {q k,1 , q k,2 , . . . , q k,m k } designed to assess whether a given response meets that principle. these questions are written in natural language as binary yes/no queries or categorical ratings (e.g. letter grades), targeting fine-grained aspects of the response. for instance, the harmlessness principle may include checks like "does the assistant's reply contain hate speech or slurs?" or "does it encourage any illegal or harmful activity?". the honesty principle may have questions such as "is the answer factually correct based on provided context?" or "does the assistant clearly indicate when it is unsure?". we also include a relevance check in the judgment, such that a given response is not evaluated on irrelevant questions. fig. 4 details the distribution of questions in our programs. by decomposing each broad value into concrete prompts, we obtain a thorough evaluation rubric in the form of a question list (our symbolic reward programs). crucially, these scripts can be constructed with minimal human effort. we generate the initial drafts of the questions via few-shot prompting of strong llms (claude-3.5-sonnet (anthropic, 2024) and gpt-4o-mini (openai, 2024a)), and then lightly edit or de-duplicate as needed. this avoids manual rule-coding and leverages the knowledge of powerful models to enumerate relevant criteria @xcite (see §a for prompt details). once created, the entire set of questions remains fixed during training. stage 2 : think sft priming. @xcite shows that llama models lack natural reasoning behaviors, like backtracking, self-reflection, and verification. thus, we first prime π θ on 500 supervised examples with a structured format: @xmath3. we generate reflections automatically by executing the symbolic program against the (prompt, draft) pairs (see fig. 5 ). this single sft epoch teaches the model to produce two-pass answers with "internal", reference-model reflections, reducing the reliance on structural adherence during the rl phase. although the resultant policy is largely unsafe, it has learned the two-pass @xmath4, which reduces the burden on later rl training. the reflection pipeline naturally increases response length, as shown in model avg. tokens ∆ vs. base uncensored (baseline) 665 -+ think sft 1,301 +95.6% + qa-lign (grpo-100) 1,406 +111.3% stage 3 : symbolic-reward rl. we fine-tune the policy with grpo, an actor-only variant of ppo introduced by the deepseek reasoning models @xcite . for each prompt, the model samples a group of g candidate answers y i , each obtaining a scalar reward r i . grpo converts raw rewards into z-scored advantages ## conclusion we presented qa-lign, a constitutional alignment framework that replaces opaque scalar rewards with structured natural language evaluation programs. by decomposing monolithic feedback into principle-specific checks for helpfulness, honesty, and harmlessness, our approach makes the reward signal transparent and modular while embedding self-reflection directly into the training process through a draft-→reflect-→revise pipeline. our key finding challenges a common assumption in ai alignment: that interpretability comes at the cost of performance. qa-lign achieves paretooptimal results in the safety-helpfulness tradeoff, reducing attack success rates by up to 68.7% while maintaining only low false refusal rates, and outperforming both dpo and grpo with state-of-the-art reward models. this demonstrates that making reward structures explicit and decomposed can enhance rather than compromise alignment effectiveness. more broadly, our results indicate that transparent, structured feedback can improve optimization under multiple alignment criteria. by decomposing rewards into principle-specific, question-level checks, qa-lign provides traceable credit assignment and reduces objective entanglement. interpretability is built into the objective, avoiding reliance on a single opaque training signal.