See our collection for all versions of ELECTRA.

Run ELECTRA with Keras 3: JAX, PyTorch, or TensorFlow

kerasformers/electra_base_discriminator

Paper: ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators (arXiv:2003.10555) · HF Papers

ELECTRA is Google's BERT-style bidirectional text encoder, pre-trained as a replaced-token discriminator (with a smaller generator producing the corrupted tokens). This repo is the encoder / downstream checkpoint. WordPiece tokenizer; mask token [MASK].

For more details on the model, please go to the upstream model card.

Pure-Keras 3 conversion of google/electra-base-discriminator for kerasformers. One implementation runs unmodified on TensorFlow / Torch / JAX.

✨ Quick start (encoder / downstream)

import os
os.environ["KERAS_BACKEND"] = "torch"  # or "jax" / "tensorflow"

from kerasformers.models.electra import ElectraModel, ElectraTokenizer

model = ElectraModel.from_weights("kerasformers/electra_base_discriminator")
tokenizer = ElectraTokenizer.from_weights("kerasformers/electra_base_discriminator")

out = model(tokenizer("The quick brown fox."))["last_hidden_state"]  # (1, L, H)

The same repo also serves the task heads, loaded the same way: ElectraSequenceClassify, ElectraTokenClassify, ElectraQnA, ElectraMultipleChoice (each takes the pretrained encoder and a randomly-initialized head, ready for fine-tuning).

Load any ELECTRA variant the same way with from_weights("kerasformers/<variant>"):

Size	Discriminator (encoder / downstream)	Generator (masked-LM)
small	`kerasformers/electra_small_discriminator`	`kerasformers/electra_small_generator`
base	`kerasformers/electra_base_discriminator`	`kerasformers/electra_base_generator`
large	`kerasformers/electra_large_discriminator`	`kerasformers/electra_large_generator`

Available classes

Load any of these from this repo with from_weights("kerasformers/electra_base_discriminator") (or on the fly via the hf: prefix). The pretrained backbone is shared; task heads not stored in this checkpoint start randomly initialized, ready for fine-tuning (or load a hf: fine-tune).

Class	Task
`ElectraModel`	Encoder backbone
`ElectraSequenceClassify`	Sequence classification
`ElectraTokenClassify`	Token classification (NER / POS)
`ElectraQnA`	Extractive question answering
`ElectraMultipleChoice`	Multiple choice

from kerasformers.models.electra import ElectraSequenceClassify
model = ElectraSequenceClassify.from_weights("kerasformers/electra_base_discriminator")

Tips

Set KERAS_BACKEND before importing Keras / kerasformers.
Prefer ElectraTokenizer.from_weights(...) so WordPiece tokenization matches.
Downstream tasks (classification / QA / NER) use the discriminator repos; the generator repos are the masked-LM.
See ELECTRA docs and Loading Weights.
Community / upstream safetensors still work via the hf: prefix, e.g. ElectraModel.from_weights("hf:google/electra-base-discriminator").