schonsense/humanen_70B_v1

What Is Different About This Model?

• No human-continuation imitation: The model was not trained with ordinary SFT/NLL against paired human stories. Human writing is used to define a broader stylistic distribution rather than as an exact continuation target.

• Style is learned as a distributional signal: Human, GPT-3.5, Claude Opus, and base-model outputs were analyzed in StyleDistance space to identify stylistic structure that generalizes across different machine-writing populations.

• The training signal operates on predictive hidden states: A linear critic maps the model's final hidden representation immediately before the LM head to a continuous human-associated style score.

• Training is reference-relative: For each base-policy (π0) response, the adapted model's hidden trajectory is compared against the cached hidden trajectory of the unmodified model on the exact same text.

• The objective uses a finite target margin: Rather than maximizing the style critic indefinitely, each response is assigned a bounded desired shift based on its original score. Lower-scoring outputs receive more correction, while already-high-scoring outputs receive reduced or zero pressure.

• Unrelated representation movement is penalized: Hidden-state displacement orthogonal to the validated style direction is explicitly discouraged, favoring comparatively small targeted changes over broad representational drift.

• No token-level likelihood loss is used: The LoRA is optimized directly from the hidden-state style objective; no chosen/rejected log-probability loss and no next-token NLL term are included.

• Conceptually IPO-like, but in representation space: IPO regresses toward a finite reference-relative preference margin in log-probability space. This method instead regresses toward a finite reference-relative style margin in predictive hidden-state space.

Methodology

This model was trained using a reference-relative hidden-state style objective designed to move the model's creative-writing distribution toward stylistic regions associated with human writing without directly maximizing the likelihood of specific human-written continuations.

The method is loosely analogous to IPO in that it optimizes toward a finite target margin rather than an unbounded reward, but the margin is measured in a learned hidden-state style direction rather than in chosen/rejected log-probability ratios.

Pipeline Overview

Human + machine creative-writing responses
                  │
                  ▼
        Prompt-grouped dataset
     1,443 unique prompt groups
                  │
                  ├──────── Human responses
                  ├──────── GPT-3.5 responses
                  ├──────── Claude Opus responses
                  └──────── Base-policy (π0) responses
                  │
                  ▼
        5,733 complete texts
                  │
                  ▼
       StyleDistance embeddings
                  │
                  ▼
 Learn continuous human↔machine
        stylistic scalar field
                  │
                  ▼
Teacher-force the same texts through
       the target 70B base model
                  │
                  ▼
Pool final predictive hidden states
       over response positions
                  │
                  ▼
Fit linear hidden-state surrogate
   to the external style scalar
                  │
                  ▼
Source-center GPT / Opus / π0 targets
to isolate shared within-machine variation
                  │
                  ▼
       Frozen 8,192-D critic
        s(h) = wᵀh + b
                  │
                  ▼
Cache π0 reference hidden trajectories
                  │
                  ▼
        LoRA policy training
                  │
                  ▼
Compare adapted hidden trajectory
to its frozen π0 reference trajectory
                  │
                  ▼
Move a bounded distance along the
validated human-associated style direction
while penalizing orthogonal movement

1. Source Dataset Construction

The starting dataset consists of human-written creative-writing responses paired with machine-generated responses to the same prompts.

Repeated human responses that had separately been paired with GPT-3.5 and Claude Opus generations were collapsed into shared prompt groups. Each group therefore represents one underlying writing task and its available alternative continuations.

The resulting corpus contains:

• 1,443 unique prompt groups
• 1,443 human responses
• 1,421 GPT-3.5 responses
• 1,426 Claude Opus responses
• 1,443 base-policy (π0) responses
• 5,733 total texts

One additional response was generated from the unmodified target model for every prompt. These π0 generations provide a complete snapshot of the model distribution immediately before the intervention.

Responses belonging to the same prompt share a common group_id, allowing all validation procedures to remain prompt-disjoint.

2. External Style Representation

Each complete response was embedded using StyleDistance, providing a representation intended to emphasize stylistic rather than primarily semantic differences.

Long responses were processed in chunks and pooled into a single full-response representation.

Human and machine populations were then used to construct a continuous scalar field in StyleDistance space. Higher values correspond to regions more strongly associated with the human-writing population.

Importantly, the paired human continuation is not treated as the correct target response. Human examples instead define a population-level stylistic distribution.

This distinction is central to the method:

Not:

machine response → imitate paired human response

Instead:

machine distribution → move toward stylistic structure
                       associated with the human distribution

3. Distributional Rather Than Prompt-Local Geometry

Several possible geometric formulations were tested before selecting the final target.

Simple human-centroid attraction and machine-centroid repulsion were rejected because the populations occupy structured, partially overlapping regions rather than lying along one global axis.

Prompt-local correction vectors such as:

human − GPT
human − Opus

were also tested.

Their apparent alignment did not survive anchor-preserving permutation controls, indicating that the paired human continuation does not define a reliable unique correction direction for each prompt.

The final methodology therefore treats the signal as a population-level style field, not as a collection of machine-to-human rewrite vectors.

4. Cross-Model Style Signal

The StyleDistance human/machine boundary transfers strongly across machine families.

A boundary learned from GPT-3.5 and Opus generalizes to the previously unseen π0 population, supporting the existence of a shared machine-writing component rather than only model-specific fingerprints.

This motivated retaining multiple machine populations while keeping their source identities distinct.

5. Target-Model Hidden-State Extraction

All 5,733 texts were then teacher-forced through the exact target base model:

Jolly-Q/Llama-3.3-70B-Inst-Ablit-Flammades-SLERP

The exact model revision used for hidden-state extraction was recorded and later pinned during LoRA training.

For each response, the final normalized hidden states immediately before the language-model head were extracted.

If response tokens occupy positions:

[s, e)

the corresponding predictive hidden states are:

[s-1, e-1)

because those are the causal states whose logits predict the response tokens.

These states were averaged to obtain one 8,192-dimensional predictive representation per response:

$$\bar{h}=\frac{1}{T}\sum _t{h}_t\backslash bar\; h\; =\; \backslash frac\{1\}\{T\}\; \backslash sum\_t\; h\_t$$

No arbitrary context truncation was required. The longest complete templated sequence contained only 3,688 tokens.

6. Hidden-State Style Surrogate

A linear surrogate was fitted to predict the externally derived StyleDistance scalar from the model's mean final predictive hidden state.

The strongest formulation used the three machine populations:

• GPT-3.5
• Claude Opus
• π0

Before regression, the mean target score of each machine source was subtracted.

This source-centering removes easy between-model offsets and forces the critic to learn stylistic variation shared within machine populations.

The final critic therefore has the form:

$$s(\bar{h})=w^{\mathrm{\top }}\bar{h}+bs(\backslash bar\; h)=w^\backslash top\backslash bar\; h+b$$

where higher values indicate movement toward the human-associated direction learned from the dataset.

The critic was selected using group-disjoint cross-validation and achieves approximately:

• π0 Spearman ρ: 0.619
• GPT-3.5 Spearman ρ: 0.587
• Opus Spearman ρ: 0.643

The final critic is an 8,192-dimensional linear direction rather than a large neural reward model.

7. Cached Reference Policy

For every π0 training example, the original base model's predictive mean hidden state was retained:

$${\bar{h}}_0\backslash bar\; h\_0$$

During LoRA training, the same text is teacher-forced through the adapted model to obtain:

$${\bar{h}}_{\theta }\backslash bar\; h\_\backslash theta$$

This makes the training objective explicitly reference-relative while avoiding the need to keep a second frozen 70B reference model in memory.

The zero-initialized LoRA was verified against cached reference states before training. Initial critic-score discrepancies were below approximately (4\times10^{-5}), confirming that the live model and cached reference represented effectively the same starting policy.

8. Reference-Relative Style Shift

Training measures how far the adapted model has moved along the frozen critic direction relative to the base policy.

The hidden-state displacement is:

$$\mathrm{\Delta }h={\bar{h}}_{\theta }-{\bar{h}}_0\backslash Delta\; h\; =\; \backslash bar\; h\_\backslash theta-\backslash bar\; h\_0$$

and the corresponding critic-score movement is:

$$\mathrm{\Delta }s=w^{\mathrm{\top }}({\bar{h}}_{\theta }-{\bar{h}}_0)\backslash Delta\; s\; =\; w^\backslash top\; (\backslash bar\; h\_\backslash theta-\backslash bar\; h\_0)$$

The equivalent calculation is performed in feature-standardized hidden space.

The method therefore does not ask whether the model currently has a high absolute human-style score. It asks:

How far has this response's predictive trajectory moved from the original model in the validated human-associated direction?

9. Finite Target Margin

The critic is not maximized without bound.

Instead, every π0 example receives a finite desired improvement:

$${\mathrm{\Delta }}_i^{\ast }=\mathrm{clamp}(s_{\mathrm{c}\mathrm{a}\mathrm{p}}-s_i^{(0)},0,{\mathrm{\Delta }}_{\max })\backslash Delta\_i^*\; =\; \backslash operatorname\{clamp\}\; \backslash left(\; s\_\{\backslash mathrm\{cap\}\}-s\_i^\{(0)\},\; 0,\; \backslash Delta\_\{\backslash max\}\; \backslash right)$$

where:

• (\Delta_{\max}=0.09364), approximately 0.5 standard deviations of the original π0 critic-score distribution
• (s_{\mathrm{cap}}=0.222817), approximately the 90th percentile of the original π0 distribution

This causes ordinary and low-scoring base responses to receive the full correction pressure while already-high-scoring responses receive progressively less or no pressure.

Conceptually:

low-scoring π0 response
        │
        └── full bounded style shift

middle π0 response
        │
        └── full bounded style shift

high-scoring π0 response
        │
        └── smaller shift

already above score cap
        │
        └── no positive style pressure

10. Orthogonal Trust Constraint

A scalar critic alone could theoretically be satisfied while the model moves substantially in unrelated hidden-state directions.

To discourage this, standardized hidden displacement is decomposed into:

$$d=d_{\parallel }+d_{\perp }d=d\_\backslash parallel+d\_\backslash perp$$

where (d_\parallel) is parallel to the critic direction and (d_\perp) is orthogonal to it.

The orthogonal component is explicitly penalized:

$$L_{\perp }=\frac{1}{H}\mathrm{\parallel }{d}_{\perp }{\mathrm{\parallel }}^{2}L\_\backslash perp\; =\; \backslash frac\{1\}\{H\}\; \backslash |d\_\backslash perp\backslash |^2$$

This encourages the adapter to find a relatively small hidden-state modification that accomplishes the desired stylistic shift rather than arbitrarily restructuring the final representation.

11. Final Training Objective

The style component uses Smooth L1 regression toward the finite target:

$$L_{\mathrm{s}\mathrm{t}\mathrm{y}\mathrm{l}\mathrm{e}}=\mathrm{SmoothL1}(\mathrm{\Delta }s,{\mathrm{\Delta }}_i^{\ast })L\_\{\backslash mathrm\{style\}\}\; =\; \backslash operatorname\{SmoothL1\}\; (\; \backslash Delta\; s,\; \backslash Delta\_i^*\; )$$

with an initial ramp that gradually introduces the full target.

The total objective is:

$$L=L_{\mathrm{s}\mathrm{t}\mathrm{y}\mathrm{l}\mathrm{e}}+{\lambda }_{\perp }{L}_{\perp }L\; =\; L\_\{\backslash mathrm\{style\}\}\; +\; \backslash lambda\_\backslash perp\; L\_\backslash perp$$

No ordinary next-token negative log-likelihood term is used.

No paired human response is supplied as a token-level generation target.

No chosen/rejected likelihood comparison is performed.

12. Relationship to IPO

The optimization philosophy is similar to IPO in one important respect: both optimize toward a finite reference-relative margin rather than indefinitely maximizing a reward.

The measured quantities differ:

IPO

chosen + rejected responses
        │
        ▼
policy/reference log-probability ratios
        │
        ▼
finite preference-margin target

This method

same π0 response under base + adapted model
        │
        ▼
predictive hidden-state displacement
        │
        ▼
frozen style-critic score change
        │
        ▼
finite stylistic-margin target

IPO modifies model parameters according to a desired likelihood preference margin.

This method modifies model parameters according to a desired hidden-state stylistic margin.

13. How the LoRA Weights Are Updated

Only LoRA parameters are trainable; the original model weights remain frozen.

For every teacher-forced response:

tokens
  │
  ▼
base transformer + trainable LoRA
  │
  ▼
final predictive hidden states
  │
  ▼
mean response representation
  │
  ▼
compare with cached π0 representation
  │
  ▼
style-margin + orthogonal loss
  │
  ▼
backpropagation
  │
  ▼
LoRA A/B matrices only

Because the critic is linear:

$$\frac{\mathrm{\partial }s}{\mathrm{\partial }\bar{h}}=w\backslash frac\{\backslash partial\; s\}\{\backslash partial\backslash bar\; h\}=w$$

the final hidden-state gradient has a directly interpretable preferred direction.

Backpropagation through the transformer determines which LoRA parameter changes produce that movement in the final predictive representation.

The language-model head is not required during loss calculation; the optimization acts directly on the final transformer representation immediately upstream of the output logits.

Summary

The training procedure can be summarized as:

Human + heterogeneous machine writing
          ↓
External style-space analysis
          ↓
Continuous human-associated style field
          ↓
Target-model predictive hidden states
          ↓
Cross-model linear hidden-state critic
          ↓
Frozen π0 hidden-state references
          ↓
Teacher-forced LoRA model
          ↓
Measure reference-relative style movement
          ↓
Regress toward a bounded positive margin
          +
Penalize unrelated hidden-state movement
          ↓
Update LoRA weights only

The intended effect is therefore not to imitate individual human examples, but to apply a controlled distributional intervention to the model's predictive representations in a direction empirically associated with broader human-writing style.