ricursiveAI

/

Ripple-65B-Preview

Like 490

Follow

Ricursive Intelligence 320

Problem

RicursiveForCausalLM.prepare_inputs_for_generation assumes every multi-token
input_ids input includes the cached prefix when cache_position is absent.
If the cache already holds 10 tokens and a caller supplies only two new tokens,
the calculated new length is -8 and torch.arange raises
RuntimeError: upper bound and lower bound inconsistent with step sign.

Fix

Distinguish new-only token blocks from full prompts containing cached tokens.
An input block no longer than the cache is new; a longer input with an attention
mask spanning the cache is also new. Keep existing full-prompt slicing when
neither clue is present.

Validation

• The three regression cases pass against this patched upstream function:
  short new block, longer new block with a spanning mask, and full prompt.
• The tiny-model Ripple suite passed against this source:
  GPTQMODEL_RIPPLE_REF=<patched Ripple checkout> python -m pytest -q -o log_cli=false tests/test_ricursive_support.py
  (33 passed).
• python -m py_compile modeling_ricursive.py passed.
• Unit test coverage and the corresponding GPT-QModel compatibility fix are in
  merged ModelCloud/GPTQModel#3571,
  especially test_generate_patch_distinguishes_new_blocks_from_full_prompts.

The current BF16 checkpoint marks its Transformers 5 support as native, so
GPT-QModel intentionally skips that compatibility patch; this repository needs
the fix directly.

Follow-up to the initial PR: merged GPT-QModel #3571 exposed another cache_position=None case. An attention mask can be padded to a static cache's capacity, so mask_length > input_length alone does not prove that input_ids contains only new tokens. The updated code treats only mask_length == past_length + input_length as evidence for a new-only block. A regression test and the exact before/after reproduction are in GPT-QModel #3574.

Validation against this updated upstream source: 34 passed in the tiny Ripple suite (tests/test_ricursive_support.py); the extracted upstream generation method also passed all four prompt/block cases.

Follow-up: the exact-capacity static-mask case is now covered. When the mask width equals past_length + input_length, its final slot distinguishes a full prompt with unused cache capacity from a new-only token block. A mask with unused capacity at the end keeps the full-prompt path.

GPT-QModel #3574 contains regression tests for both exact-capacity and longer padded masks, plus existing new-only block coverage. It builds on merged GPT-QModel #3571. The Ripple suite passed 35 tests against this upstream source (tests/test_ricursive_support.py).

Mostly looks good, and thank you for the fix, just couple of changes:

1. A capacity-sized mask can duplicate the cached prefix. With four cached tokens, a six-token full prompt, and an all-visible mask of width ten, the PR processes all six tokens again. The cache grows to ten instead of six, and logits change. This passed on the base code. The final mask bit cannot reliably distinguish the input layouts. Affected code.
2. The new short-block path can produce incorrect positions. Ten cached tokens plus two new tokens, with a sixteen-slot padded mask, produces positions [0, 0] instead of [10, 11]. The old exception disappears, but logits are wrong. Derive positions using cache_position, rather than taking the mask’s final entries. Affected code.

@ebrahim-ricursive Found more issues. Will fix them all and push. Note that unit tests will be in GPT-QModel, as to not pollute this modeling code only model repo.

@ebrahim-ricursive 13 issues fixed. HF tests removed. all unit tests live in GPT-QModel #3709.

More issue found and fixed including training.

@Qubitium I retested 95ebf2a7dc9cde71ee355cd3e9e7d919ae41535c. The two earlier findings are fixed. One new regression remains:

1. [P2] Preserve a rollback-compatible default cache on Transformers 4.

   RicursiveModel.forward, lines 635-639 now creates DynamicCache(config=self.config). On Transformers 4.57.6, this creates sliding-window layers that cannot be cropped after reaching the window limit. The constructor succeeds, so the TypeError fallback does not protect this case.

   Reproduction with a tiny native model in eval mode, sliding_window=8:

   with torch.inference_mode():
       ids = torch.tensor([[3, 4, 5] * 4])
       cache = model(ids[:, :10], use_cache=True).past_key_values
       model.generate(
           ids,
           attention_mask=torch.ones_like(ids),
           past_key_values=cache,
           prompt_lookup_num_tokens=3,
           max_new_tokens=5,
           do_sample=False,
           eos_token_id=96,
           suppress_tokens=[96],
       )
   

   The tested fixture has vocabulary size 97, hidden size 32, five layers, and four sliding layers followed by one full-attention layer. The new code raises ValueError: Cannot crop a DynamicSlidingWindowLayer during assisted decoding. Main (da94cc01) and the previous PR revision (728d0dea) both pass and match ordinary generation. Explicitly passing an unconfigured DynamicCache() to the prefill also makes this case pass on the latest PR. Transformers 5.22.1 passes the same case.

   Please preserve a full-cache default on Transformers 4, or provide compatible rollback before changing that default. Add a regression that reuses the cache returned by a direct forward call after the sliding-window limit; always constructing an explicit full cache in the test would hide the issue.

@ebrahim-ricursive Pushed 2f007aabfcedd0f8763d86d08f9aa0afbc344cea to this PR.

The default cache is now version-aware. Transformers 5 keeps the configured hybrid cache; Transformers 4 keeps the plain full-attention DynamicCache, whose layers remain croppable after the sliding-window limit. Your prompt-lookup reproduction now matches plain greedy generation on 4.57.6, and the hybrid path still passes on 5.22.1.

Regression coverage is in GPT-QModel #3752 (merged). It reuses the cache returned by a direct forward after the window rather than constructing an explicit full cache, asserts 4.x cropping, and pins the native revision to 2f007aa. Verification: 27/27 modeling-contract and 48/48 native-generation tests on both Transformers 4.57.6 and 5.22.1; the new regression fails on 95ebf2a with the reported ValueError and passes on 2f007aa.

@ebrahim-ricursive Please confirm something so I can file a bug report to HF. How did you merge my last 2 PRs? Via HF merge/squash button here on UI? Becasue I need to understand how I got co-authorship for 100% of the commits. I can connect with the correct HF people to get this fixed if this is indeed a HF repo merge bug which should not happen. Thanks

Hi, I use the merge button in this page on the web UI.

Hi, I use the merge button in this page on the web UI.

Thanks for the confirmation. I will reproduce this issue in my own repo to get this fixed by HF.