summerMC/LoopedQwen-GraphEngram-ja-en

Looped computation meets learned graph memory.
_{A self-contained bilingual research snapshot by summerMC.}

Architecture · Quickstart · Evaluation · Limitations

---

Research prototype, NOT a production chatbot. Conversation, arithmetic and code-generation tests remain poor, often producing unrelated numbers. Do not interpret lower language-model loss as general reasoning or instruction-following quality.

This repository contains a consolidated snapshot of a repaired LoopedQwen backbone, learned graph-memory modules, and the matching token graph. No separate base-model or adapter download is needed when loading this package. The graph module cannot be algebraically folded into a dense weight matrix: “merged” means a self-contained custom architecture, not a LoRA-only merge.

GraphEngram is an independent experimental graph-attention mechanism, not the official DeepSeek Engram implementation or a reproduction of its paper results.

Architecture

• Pinned base: summerMC/Looped-Qwen-pre, revision eaa834598f3e6b479d1f72b51b6933422b9b8a9c.
• Six shared GDN groups, four inner steps each: 24 effective steps, hidden size 2048.
• Prior backbone adapters are consolidated into the exported backbone parameters.
• Memory inserted before outer groups 0 and 3.
• 48,491 shared token nodes and 88,076 directed adjacency edges.
• Node width 128; 13,626,882 learned memory parameters.
• Causal exact token bigram/trigram routing, up to 8 candidates, one-hop graph attention with degree cap 8.
• Graph topology is fixed; node embeddings, attention projections and fusion gates are learned.
• Graph construction uses training documents only. Old English node IDs and weights were preserved during Japanese expansion.

Loading

Use a recent compatible Transformers environment. Export/evaluation used PyTorch 2.11.0 and Transformers 5.20.2. flash-linear-attention is optional acceleration; backend differences can affect numerical outputs.

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

repo = "summerMC/LoopedQwen-GraphEngram-ja-en"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(
    repo, trust_remote_code=True, dtype=torch.bfloat16, device_map="cuda"
).eval()
ids = tok.apply_chat_template(
    [{"role": "user", "content": "こんにちは。"}],
    add_generation_prompt=True, enable_thinking=False,
    return_tensors="pt", return_dict=True
)["input_ids"].to("cuda")
out = model.generate(ids, max_new_tokens=32, do_sample=False, use_cache=False)
print(tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True))

Inspect custom code before enabling trust_remote_code. Generation is greedy, unpadded batch size 1 only. Cached decoding, concurrent forwards and production serving integrations are unsupported. Each output token recomputes the prefix. The graph checksum is checked during loading. Keep graph.json with the weights and configuration.

Toggle memory for ablations: model.network.enabled = False.

Training provenance

• English source: HuggingFaceFW/fineweb, sample-10BT, revision 9bb295ddab0e05d785b879661af7260fed5140fc.
• Japanese source: HuggingFaceFW/fineweb-2, jpn_Jpan, revision af9c13333eb981300149d5ca60a8e9d659b276b9.
• Dataset cards identify ODC-BY. Check dataset/source-content terms for downstream redistribution and use. The token graph itself retains training-text fragments; it is not a privacy-preserving representation.
• Whole-document hashing separates train/dev/test before truncation. No near-duplicate or base-pretraining contamination audit was performed.
• Base was frozen during graph training. The immediate bilingual continuation trained 36,864 Japanese + 36,864 English tokens, not the entire downloaded corpus.
• Earlier graph stages and the repaired base are described in the included reports/design document.
• Parent optimizer was not retained for the first bilingual continuation; weights were resumed with a fresh optimizer.

Evaluation

Standalone reloaded bilingual snapshot, eight held-out documents per language, full documents evaluated in independent chunks:

Corpus	Target tokens	GNN off NLL	GNN on NLL
Japanese	4,895	4.5203	4.4736
English	4,912	4.0961	4.0122

Eight additional development prompts: math 0/2, required code functions 0/2, extraction 1/2 for both on/off. Conversation remains inappropriate/repetitive. Generated code was not executed. These are small diagnostics, not standardized benchmark or Opus-equivalence results.

Limitations

• Weak instruction following and numerical-answer bias in the repaired backbone.
• No established improvement in Japanese/English conversation, math or code generation.
• No demonstrated learning-speed advantage over official Engram, RAG or ordinary fine-tuning at matched quality.
• No proven compression advantage or lossless reconstruction of the complete source corpus from the model graph.
• No billion/trillion-token context guarantee; lexical routing has limited recall and CPU overhead.
• Frozen random-initialized new node vectors require learning; storing a text node is not equivalent to learning its facts.
• Source text may contain errors, personal information or malicious instructions. No dedicated filtering/safety training was performed for this prototype.

The model is published for reproducibility and architecture experiments, not capability claims.