transformers-community

/

constrained-beam-search

like 2

Follow

Transformers Community 115

Problem

transformers-community/constrained-beam-search produces different and
degenerate output when use_cache=True, while the same deterministic generation
call with use_cache=False produces a coherent continuation.

Since the KV cache should only be an optimization, enabling use_cache should
not materially change the generated token sequence under deterministic decoding.

This looks similar to the recently reported and fixed cache-handling issue in
transformers-community/group-beam-search:

• Original issue: https://huggingface.co/transformers-community/group-beam-search/discussions/3
• Accepted fix: https://huggingface.co/transformers-community/group-beam-search/discussions/4

Minimal reproduction

import platform

import torch
import transformers
from transformers import AutoModelForCausalLM, AutoTokenizer

print("python:", platform.python_version())
print("torch:", torch.__version__)
print("transformers:", transformers.__version__)
print("cuda:", torch.cuda.is_available())

m = "openai-community/gpt2"
custom_generate = "transformers-community/constrained-beam-search"

tok = AutoTokenizer.from_pretrained(m)
model = AutoModelForCausalLM.from_pretrained(
    m,
    device_map="auto",
)
tok.pad_token = tok.eos_token

print("model:", m)
print("custom_generate:", custom_generate)
print("device:", model.device)

inputs = tok(
    "The most popular ways of using transformers are",
    return_tensors="pt",
).to(model.device)

force_words_ids = [
    tok(" translation", add_special_tokens=False).input_ids,
]

outs = []
for use_cache in [False, True]:
    out = model.generate(
        **inputs,
        custom_generate=custom_generate,
        trust_remote_code=True,
        force_words_ids=force_words_ids,
        num_beams=4,
        do_sample=False,
        min_new_tokens=10,
        max_new_tokens=30,
        use_cache=use_cache,
        pad_token_id=tok.eos_token_id,
    )

    new = out[0, inputs["input_ids"].shape[-1]:]
    outs.append(new.tolist())

    print("\nuse_cache =", use_cache)
    print("ids:", new.tolist())
    print("text:", repr(tok.decode(new, skip_special_tokens=True)))

assert outs[0] == outs[1]

Environment and observed output

python: 3.12.13
torch: 2.11.0+cu128
transformers: 5.12.0
cuda: True
Loading weights: 100% 148/148 [00:00<00:00, 1159.10it/s]
model: openai-community/gpt2
custom_generate: transformers-community/constrained-beam-search
device: cuda:0

use_cache = False
ids: [25, 198, 198, 16, 13, 5765, 262, 6121, 364, 287, 262, 976, 835, 345, 561, 779, 257, 3218, 5408, 13, 198, 198, 17, 13, 5765, 262, 6121, 364, 287, 11059]
text: ':\n\n1. Use the transformers in the same way you would use a regular expression.\n\n2. Use the transformers in translation'

use_cache = True
ids: [25, 198, 198, 198, 198, 198, 198, 198, 198, 198, 198, 198, 198, 198, 198, 198, 198, 198, 198, 198, 198, 198, 198, 198, 198, 198, 198, 198, 198, 11059]
text: ':\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n translation'
---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
/tmp/ipykernel_9209/429571338.py in <cell line: 0>()
     55     print("text:", repr(tok.decode(new, skip_special_tokens=True)))
     56 
---> 57 assert outs[0] == outs[1]

AssertionError:

Expected behavior

use_cache=True and use_cache=False should produce the same generated token
sequence for this deterministic generation call, or at least should not produce
a degenerate newline-heavy continuation only when cache is enabled.

Why this looks cache-related

The only changed generation argument in the reproduction above is use_cache.

The uncached path produces a coherent constrained-beam-search continuation:

1. Use the transformers in the same way you would use a regular expression.

2. Use the transformers in translation

The cached path produces mostly newline tokens and then the forced word:

translation

This suggests that the cached decoding path may be passing an incorrect token
sequence or cache position during decoding.

This appears to be the same class of issue recently found in
transformers-community/group-beam-search, where after the prefill step the
custom generation loop needed to distinguish the first iteration from subsequent
cached decoding iterations.

Relevant prior discussion and fix:

• Issue: https://huggingface.co/transformers-community/group-beam-search/discussions/3
• Accepted fix: https://huggingface.co/transformers-community/group-beam-search/discussions/4