preneond/newlinefix-encoder

Newline Fixer — gap-classification encoder

Fine-tuned distilroberta-base that classifies the separator ("gap") between each pair of consecutive words into JOIN / SPACE / NEWLINE / PARA. Reconstructing text from words plus predicted gaps restores newline placement with a hard guarantee: the output words are identical to the input — only whitespace changes.

Trained self-supervised on Wikipedia + arXiv-markdown text whose newline structure was programmatically destroyed. Full methodology, evaluation protocol, and results: the project's report.md.

Serving

The raw checkpoint is a standard token-classification model, but gap decoding (last-subtoken alignment, sliding-window stitching, reconstruction) lives in the newlinefix package:

from newlinefix.models.encoder import EncoderGapPredictor
from newlinefix.predict import fix_text

predictor = EncoderGapPredictor.load("preneond/newlinefix-encoder")
print(fix_text("the que\nries come from here", predictor))

The bundled HTTP service serves this repo directly: NEWLINEFIX_MODEL_DIR=preneond/newlinefix-encoder.

Validation metrics

Macro-F1 is over the structural classes {JOIN, NEWLINE, PARA}.

metric	value
accuracy	0.9795
macro_f1_structural	0.7832
precision_JOIN	0.8875
recall_JOIN	0.9831
f1_JOIN	0.9329
precision_SPACE	0.9972
recall_SPACE	0.9831
f1_SPACE	0.9901
precision_NEWLINE	0.6938
recall_NEWLINE	0.8200
f1_NEWLINE	0.7516
precision_PARA	0.5414
recall_PARA	0.8624
f1_PARA	0.6652