Cross-attention cold start and inference footprint in BERT2BERT normalization
Hi Erdem,
Building a dedicated Turkish normalizer and typo corrector directly on BERTurk checkpoints is a very practical project, especially with the gold-set evaluation against #Turki$hTweets.
Looking at the EncoderDecoderModel configuration and the limitations noted in the model card:
Cross-attention cold start and convergence:
Stitching two BERT-base models requires initializing 12 full cross-attention layers from scratch (roughly 28M uninitialized parameters). As you noted, learning cross-attention entirely during fine-tuning leaves the bridge vulnerable to under-correction on out-of-distribution syntactic patterns compared to native sequence-to-sequence pretraining.Latency and parameter overhead during beam search:
A 2x BERTurk pipeline totals ~250M parameters (12 encoder layers + 12 decoder layers with cross-attention). Running autoregressive beam search (num_beams=4) through 12 cross-attended decoder layers creates significant latency for what is often an interactive or real-time typing task.
Additionally, holding separate 32,000 x 768 WordPiece tables across the encoder and decoder consumes nearly 50M-70M parameters purely in static lookup matrices.
In an open architecture project called Maba (101M reference model: https://huggingface.co/AndrewThompson1233/maba-v1-architecture), we handle sequence transformations using two architectural alternatives:
- Unified causal prefix framing with 2-pass block recycling: Instead of a disjoint 24-layer encoder-decoder, a compact causal decoder processes the noisy input prefix and generates corrected text in a single stream. Using 8 physical blocks cycled twice with Split RMSNorm provides 16 effective layers of contextual refinement while dropping model size to ~70M parameters.
- Low-rank vocabulary factorization: Factorizing the 32k vocabulary (32,000 -> 64 -> 768) shrinks lookup tables from 24.6M to ~2.1M parameters, freeing budget for active layers.
Did you compare this BERT2BERT setup against a compact causal decoder-only baseline before training the cross-attention layers?
Best,
Andrew
Hi Andrew,
Thanks for the detailed read — the cross-attention cold-start point and the parameter overhead from the duplicated WordPiece tables are both fair, and match what's already flagged in the model's Limitations section.
To answer your question directly: no, we didn't benchmark this against a compact causal decoder-only baseline before training. That wasn't the comparison we were after — the actual question we set out to answer was different: does a large, general-purpose multilingual encoder-decoder that happens to know Turkish (like byT5, pretrained by Google across 101 languages) outperform a much smaller model built by converting a Turkish-only pretrained encoder (BERTurk) into an encoder-decoder — or the other way around. BERT2BERT was the natural way to reuse BERTurk's weights for that comparison, not a deliberate choice over a decoder-only architecture.
Maba looks like an interesting direction, especially the block-recycling and low-rank vocabulary factorization ideas — appreciate you sharing it.
Thanks again for taking the time to look closely at this and write such a thoughtful comment.
Best,
Erdem
Hi Erdem,
That makes complete sense - framing the ablation specifically as monolingual language priors (BERTurk converted) versus multilingual byte-level models (byT5) is a very sharp research question, especially for an agglutinative language like Turkish where subword tokenizers often clash with complex suffix chains.
Monolingual representations frequently punch well above their weight compared to massive multilingual checkpoints. Really curious to hear how the final WER and over-correction numbers stack up against byT5 on those gold tweet sets once the comparison is wrapped up.
Thanks for the context, and best of luck with the evaluation!
Best,
Andrew
Hi Andrew,
Good question — the comparison is actually already in, since both models are trained and published with real gold-set numbers.
On the tweets set, they're close: byT5 gets WER 0.161 / F1 0.667 / 0.6% over-correction, versus BERTurk2BERTurk at WER 0.163 / F1 0.666 / 0.8% — essentially a wash.
On boun (the de/da-focused set), the gap is clearer: byT5 comes in at WER 0.093 / F1 0.590 / 4.3% over-correction, while BERTurk2BERTurk lands at WER 0.157 / F1 0.543 / 6.4%.
So at least in this setup, the monolingual-prior advantage didn't clearly show up — byT5 held its own on tweets and pulled ahead on boun, particularly on over-correction. My guess is the byte-level tokenizer's robustness to the noisy, non-standard spelling in these sets outweighed BERTurk's Turkish-specific pretraining here, but that's speculation on my part rather than something I've dug into further.
One thing worth noting: this task (noisy-spelling correction) plays directly to byte-level tokenization's strength — it's less a test of deep Turkish grammatical understanding and more a test of character-level noise robustness. A comparison on a task with clean input but real grammatical errors — e.g. incorrect case suffixes or vowel-harmony violations, rather than typos — might reveal BERTurk's monolingual advantage more clearly.
Both full result tables are on the respective model cards if you want to look closer:
Best,
Erdem
Hi Erdem,
Really appreciate you sharing those empirical numbers - that is a fascinating breakdown.
Your hypothesis about byte-level tokenization taking the lead on noisy text makes total sense, especially for Turkish:
WordPiece suffix fragmentation: With Turkish agglutination, noisy typing or misplaced spaces (especially around clitics like "de/da") often shatters WordPiece into unpredictable subword fragments or [UNK] tokens, forcing the model to infer semantics through heavily fragmented sequences.
Character-level inductive bias: byT5 operates directly on UTF-8 bytes, so dropped diacritics or single-character typos remain localized edit-distance operations rather than catastrophic vocabulary misses.
Your point about testing grammatical agreement (vowel harmony and case suffixes) on clean syntax is spot on - that would isolate genuine syntactic priors from raw character-noise robustness.
Really insightful work and great transparency in sharing the head-to-head metrics. Thanks again for the discussion!
Best,
Andrew
Hi Andrew,
Thanks a lot for the engaging discussion and thoughtful feedback. Really glad the breakdown and the reasoning resonated with you.
Testing on clean syntax to properly decouple grammatical priors from character noise is definitely on my radar for the next iteration. Thanks again for stopping by and taking the time to dive into the architecture!
Best,
Erdem