dicta-il

/

dictabert-joint

like 5

Follow

DICTA: The Israel Center for Text Analysis 172

Hi Shaltiel — wanted to flag what looks like a small leftover from the
2026-05-04 update (d013b37) before opening a PR, in case I'm missing context.

That commit renamed _tied_weights_keys → tied_weights_keys on
BertForJointParsing and switched it to the dict form. The dict form is the
right shape for transformers 5.x, but I think the rename to the no-underscore
attribute name was inadvertent: transformers only ever inspects
_tied_weights_keys, so the new value is silently ignored.

Symptom

On transformers 5.x, loading the model — including via
dicta-il/dictabert-tiny-joint, which reaches this file through auto_map —
succeeds with no warning. But cls.predictions.decoder.weight is never tied
to the input embeddings, so it stays randomly initialized and the lex head
produces garbage lemmas. No traceback, so easy to miss unless you eval the
lex output specifically.

Root cause, as I understand it

• Pre-d013b37: _tied_weights_keys was a list — rejected by transformers
  5.x, which requires the explicit {target: source} dict so safetensors can
  reconstruct ties without aliasing.
• Post-d013b37: value is a correct dict, but the attribute is named
  tied_weights_keys (no underscore). transformers never reads that name, so
  no tying happens and decoder.weight stays random.

Proposed fix

One-line change in BertForJointParsing.py (currently line 43):

```diff
class BertForJointParsing(BertPreTrainedModel):

• tied_weights_keys = {

• _tied_weights_keys = {
  "cls.predictions.decoder.weight": "bert.embeddings.word_embeddings.weight",
  "cls.predictions.decoder.bias": "cls.predictions.bias",
  }
  ```

With this, transformers 5.x correctly ties decoder.weight to
bert.embeddings.word_embeddings.weight at load time and the lex head
recovers. transformers 4.x also accepts the dict form on
_tied_weights_keys, so it should be backwards-compatible.

Thanks!
— Dima (Ovalix)