feat: Improve mmBERT-base model card with full title, abstract, and updated content

This PR significantly improves the model card for `jhu-clsp/mmBERT-base` by:

1. **Updating Metadata**:
* Changing the `pipeline_tag` from `fill-mask` to `feature-extraction` to better reflect the model's primary use cases as a multilingual encoder for classification, embedding, and retrieval tasks. This ensures the model appears in the correct search results at https://huggingface.co/models?pipeline_tag=feature-extraction.
* Adding `fill-mask` as an `additional tag` to maintain discoverability for masked language modeling, which is a supported capability as demonstrated in the usage examples.

2. **Enhancing Content**:
* Updating the main title to the full paper title: "mmBERT: A Modern Multilingual Encoder with Annealed Language Learning".
* Adding a direct link to the Hugging Face paper page: https://huggingface.co/papers/2509.06888, alongside the existing Arxiv link.
* Including the comprehensive paper abstract.
* Integrating and expanding sections like "Overview", "Quick Start", "Model Family", "Usage Examples", "Fine-tuning Examples", "Training Details", "Evaluation", "FAQ", and "Limitations" using the more detailed and up-to-date information from the official GitHub repository README, making the model card a more complete resource for users.
* Updating the `transformers` installation requirement to `transformers>=4.48.0` for consistency with the GitHub README.
* Renaming the "Base Model for Classification" example in the "Quick Start" to "Base Model for Masked Language Modeling" to accurately reflect its content.

These changes provide users with a clearer understanding of the model's capabilities, how to use it, and its underlying design.

README.md

@@ -1,71 +1,109 @@
 ---
-license: mit
 datasets:
 - jhu-clsp/mmbert-decay
 - jhu-clsp/mmbert-midtraining
 - jhu-clsp/mmbert-pretrain-p1-fineweb2-langs
 - jhu-clsp/mmbert-pretrain-p2-fineweb2-remaining
 - jhu-clsp/mmbert-pretrain-p3-others
-pipeline_tag: fill-mask
 library_name: transformers
+license: mit
+pipeline_tag: feature-extraction
+tags:
+- fill-mask
 ---
 
-# mmBERT: A Modern Multilingual Encoder
+# mmBERT: A Modern Multilingual Encoder with Annealed Language Learning
 
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
 [![Paper](https://img.shields.io/badge/Paper-Arxiv-red)](https://arxiv.org/abs/2509.06888)
+[![HF Paper](https://img.shields.io/badge/Paper-HuggingFace-blue)](https://huggingface.co/papers/2509.06888)
 [![Model](https://img.shields.io/badge/🤗%20Hugging%20Face-Model-blue)](https://huggingface.co/jhu-clsp/mmBERT-base)
 [![Collection](https://img.shields.io/badge/🤗%20Model%20Collection-blue)](https://huggingface.co/collections/jhu-clsp/mmbert-a-modern-multilingual-encoder-68b725831d7c6e3acc435ed4)
 [![GitHub](https://img.shields.io/badge/GitHub-Code-black)](https://github.com/jhu-clsp/mmBERT)
 
-> TL;DR: A state-of-the-art multilingual encoder trained on 3T+ tokens across 1800+ languages, introducing novel techniques for learning low-resource languages during the decay phase.
+> 🌍 **TL;DR**: State-of-the-art multilingual encoder models trained on 3T tokens across 1833 languages with novel annealed language learning. Outperforms XLM-R and can even beat OpenAI's o3 and Google's Gemini 2.5 Pro.
 
-mmBERT is a modern multilingual encoder that significantly outperforms previous generation models like XLM-R on classification, embedding, and retrieval tasks. Built on the ModernBERT architecture with novel multilingual training innovations, mmBERT demonstrates that low-resource languages can be effectively learned during the decay phase of training. It is also significantly faster than any previous multilingual encoder.
+## Paper Abstract
+Encoder-only languages models are frequently used for a variety of standard machine learning tasks, including classification and retrieval. However, there has been a lack of recent research for encoder models, especially with respect to multilingual models. We introduce mmBERT, an encoder-only language model pretrained on 3T tokens of multilingual text in over 1800 languages. To build mmBERT we introduce several novel elements, including an inverse mask ratio schedule and an inverse temperature sampling ratio. We add over 1700 low-resource languages to the data mix only during the decay phase, showing that it boosts performance dramatically and maximizes the gains from the relatively small amount of training data. Despite only including these low-resource languages in the short decay phase we achieve similar classification performance to models like OpenAI's o3 and Google's Gemini 2.5 Pro. Overall, we show that mmBERT significantly outperforms the previous generation of models on classification and retrieval tasks -- on both high and low-resource languages.
+
+## Overview
+mmBERT introduces the first modern multilingual encoder trained with cascading annealed language learning (ALL), progressively incorporating 1833 languages during training. It significantly outperforms previous generation models like XLM-R on classification, embedding, and retrieval tasks while achieving remarkable efficiency improvements (up to 4x faster). Built on the ModernBERT architecture with novel inverse masking schedules and high-quality multilingual data, mmBERT demonstrates that low-resource languages can be effectively learned during the decay phase of training.
 
 ## Table of Contents
-- [Highlights](#highlights)
 - [Quick Start](#quick-start)
-- [Model Description](#model-description)
-- [Novel Training Innovations](#novel-training-innovations)
 - [Model Family](#model-family)
+- [Novel Training Innovations](#novel-training-innovations)
 - [Training Data](#training-data)
 - [Usage Examples](#usage-examples)
 - [Fine-tuning Examples](#fine-tuning-examples)
 - [Model Architecture](#model-architecture)
+- [Training](#training)
+- [Evaluation](#evaluation)
+- [FAQ](#faq)
+- [Limitations](#limitations)
 - [Citation](#citation)
 
-
 ## Quick Start
 
 ### Installation
 ```bash
 pip install torch>=1.9.0
-pip install transformers>=4.21.0
+pip install transformers>=4.48.0
 ```
 
-### Usage
+### 30-Second Examples
 
+**Small Model for Fast Inference:**
 ```python
 from transformers import AutoTokenizer, AutoModel
 
-tokenizer = AutoTokenizer.from_pretrained("jhu-clsp/mmBERT-base")
-model = AutoModel.from_pretrained("jhu-clsp/mmBERT-base")
+tokenizer = AutoTokenizer.from_pretrained("jhu-clsp/mmbert-small")
+model = AutoModel.from_pretrained("jhu-clsp/mmbert-small")
 
-inputs = tokenizer("Hello world", return_tensors="pt")
+# Example: Get multilingual embeddings
+inputs = tokenizer("Hello world! 你好世界! Bonjour le monde!", return_tensors="pt")
 outputs = model(**inputs)
+embeddings = outputs.last_hidden_state.mean(dim=1)
 ```
 
-## Model Description
+**Base Model for Masked Language Modeling:**
+```python
+from transformers import AutoTokenizer, AutoModelForMaskedLM
+import torch
+
+tokenizer = AutoTokenizer.from_pretrained("jhu-clsp/mmbert-base")
+model = AutoModelForMaskedLM.from_pretrained("jhu-clsp/mmbert-base")
+
+# Example: Multilingual masked language modeling
+text = "The capital of [MASK] is Paris."
+inputs = tokenizer(text, return_tensors="pt")
+with torch.no_grad():
+    outputs = model(**inputs)
+
+# Get predictions for [MASK] tokens
+mask_indices = torch.where(inputs["input_ids"] == tokenizer.mask_token_id)
+predictions = outputs.logits[mask_indices]
+top_tokens = torch.topk(predictions, 5, dim=-1)
+predicted_words = [tokenizer.decode(token) for token in top_tokens.indices[0]]
+print(f"Predictions: {predicted_words}")
+```
 
-mmBERT represents the first significant advancement over XLM-R for massively multilingual encoder models. Key features include:
+## Model Family
 
-1. **Massive Language Coverage** - Trained on over 1800 languages with progressive inclusion strategy
-2. **Modern Architecture** - Built on ModernBERT foundation with Flash Attention 2 and unpadding techniques
-3. **Novel Training Recipe** - Introduces inverse mask scheduling and temperature sampling
-4. **Open Training Data** - Complete 3T+ token dataset publicly available
-5. **Decay Phase Innovation** - Demonstrates effective learning of low-resource languages in final training phase
+### Main Models
 
-The model uses bidirectional attention with masked language modeling objectives, optimized specifically for multilingual understanding and cross-lingual transfer.
+| Size | Model | Parameters | Languages | Context | Best For | Download |
+|:-----|:------|:-----------|:----------|:--------|:---------|:---------|
+| Small | [mmbert-small](https://huggingface.co/jhu-clsp/mmbert-small) | 140M | 1833 | 8192 | Fast inference, edge deployment | [![Download](https://img.shields.io/badge/🤗-Download-blue)](https://huggingface.co/jhu-clsp/mmbert-small) |
+| Base | [mmbert-base](https://huggingface.co/jhu-clsp/mmbert-base) | 307M | 1833 | 8192 | Best performance, production use | [![Download](https://img.shields.io/badge/🤗-Download-blue)](https://huggingface.co/jhu-clsp/mmbert-base) |
+
+### Key Features
+
+-   **1833 Languages**: Covers more languages than any previous multilingual encoder
+-   **Extended Context**: Up to 8192 tokens (vs 512 for XLM-R)  
+-   **Efficiency**: 2-4x faster inference than previous multilingual models
+-   **Modern Architecture**: Based on ModernBERT with RoPE, GLU activations, and Flash Attention 2
+-   **Open Training**: Complete training data, recipes, and checkpoints available
 
 ## Novel Training Innovations
 
@@ -77,16 +115,9 @@ The model uses bidirectional attention with masked language modeling objectives,
 
 **Model Merging**: Combine English-focused, high-resource, and all-language decay variants using TIES merging.
 
-## Model Family
-
-| Model | Total Params | Non-embed Params | Languages | Download |
-|:------|:-------------|:------------------|:----------|:---------|
-| [mmBERT-small](https://huggingface.co/jhu-clsp/mmBERT-small) | 140M | 42M | 1800+ | [![Download](https://img.shields.io/badge/🤗-Download-blue)](https://huggingface.co/jhu-clsp/mmBERT-small) |
-| [mmBERT-base](https://huggingface.co/jhu-clsp/mmBERT-base) | 307M | 110M | 1800+ | [![Download](https://img.shields.io/badge/🤗-Download-blue)](https://huggingface.co/jhu-clsp/mmBERT-base) |
-
 ## Training Data
 
-mmBERT training data is publicly available across different phases:
+mmBERT was trained on a carefully curated 3T+ token multilingual dataset:
 
 | Phase | Dataset | Tokens | Description |
 |:------|:--------|:-------|:------------|
@@ -96,89 +127,142 @@ mmBERT training data is publicly available across different phases:
 | Mid-training | [mmbert-midtraining](https://huggingface.co/datasets/jhu-clsp/mmbert-midtraining-data) | 600B | 110 languages, context extension to 8K |
 | Decay Phase | [mmbert-decay](https://huggingface.co/datasets/jhu-clsp/mmbert-decay-data) | 100B | 1833 languages, premium quality |
 
-**Data Sources**: Filtered DCLM (English), FineWeb2 (multilingual), FineWeb2-HQ (20 high-resource languages), Wikipedia (MegaWika), code repositories (StarCoder, ProLong), academic papers (ArXiv, PeS2o), and community discussions (StackExchange).
-
-## Model Architecture
-
-| Parameter | mmBERT-small | mmBERT-base |
-|:----------|:-------------|:------------|
-| Layers | 22 | 22 |
-| Hidden Size | 384 | 768 |
-| Intermediate Size | 1152 | 1152 |
-| Attention Heads | 6 | 12 |
-| Total Parameters | 140M | 307M |
-| Non-embedding Parameters | 42M | 110M |
-| Max Sequence Length | 8192 | 8192 |
-| Vocabulary Size | 256,000 | 256,000 |
-| Tokenizer | Gemma 2 | Gemma 2 |
+**Primary Sources:**
+-   **Filtered DCLM**: High-quality English content
+-   **FineWeb2**: Broad multilingual web coverage (1800+ languages)
+-   **FineWeb2-HQ**: Filtered subset of 20 high-resource languages
+-   **Code**: StarCoder and ProLong repositories
+-   **Academic**: ArXiv papers and PeS2o scientific content
+-   **Reference**: Wikipedia (MegaWika) and textbooks
+-   **Community**: StackExchange discussions
 
 ## Usage Examples
 
-### Masked Language Modeling
-
-```python
-from transformers import AutoTokenizer, AutoModelForMaskedLM
-import torch
+### Classification Task
 
-tokenizer = AutoTokenizer.from_pretrained("jhu-clsp/mmBERT-base")
-model = AutoModelForMaskedLM.from_pretrained("jhu-clsp/mmBERT-base")
+<details>
+<summary><strong>Click to expand classification fine-tuning example</strong></summary>
 
-def predict_masked_token(text):
-    inputs = tokenizer(text, return_tensors="pt")
-    with torch.no_grad():
-        outputs = model(**inputs)
-    
-    mask_indices = torch.where(inputs["input_ids"] == tokenizer.mask_token_id)
-    predictions = outputs.logits[mask_indices]
-    top_tokens = torch.topk(predictions, 5, dim=-1)
+```python
+from transformers import AutoTokenizer, AutoModel
+import torch.nn as nn
+
+# Load model for classification
+tokenizer = AutoTokenizer.from_pretrained("jhu-clsp/mmbert-base")
+encoder = AutoModel.from_pretrained("jhu-clsp/mmbert-base")
+
+# Add classification head
+class MultilingualClassifier(nn.Module):
+    def __init__(self, encoder, num_classes):
+        super().__init__()
+        self.encoder = encoder
+        self.classifier = nn.Linear(encoder.config.hidden_size, num_classes)
+        self.dropout = nn.Dropout(0.1)
     
-    return [tokenizer.decode(token) for token in top_tokens.indices[0]]
+    def forward(self, input_ids, attention_mask=None):
+        outputs = self.encoder(input_ids, attention_mask=attention_mask)
+        pooled_output = outputs.last_hidden_state[:, 0]  # Use [CLS] token
+        pooled_output = self.dropout(pooled_output)
+        return self.classifier(pooled_output)
+
+# Initialize classifier
+model = MultilingualClassifier(encoder, num_classes=3)
 
-# Works across languages
+# Example multilingual inputs
 texts = [
-    "The capital of France is <mask>.",
-    "La capital de España es <mask>.",
-    "Die Hauptstadt von Deutschland ist <mask>."
+    "This is a positive review.",
+    "Ceci est un avis négatif.",
+    "这是一个中性评价。"
 ]
-
-for text in texts:
-    predictions = predict_masked_token(text)
-    print(f"Text: {text}")
-    print(f"Predictions: {predictions}")
+inputs = tokenizer(texts, return_tensors="pt", padding=True, truncation=True)
+predictions = model(**inputs)
 ```
 
-### Cross-lingual Embeddings
+</details>
+
+### Multilingual Retrieval
+
+<details>
+<summary><strong>Click to expand multilingual retrieval example</strong></summary>
 
 ```python
 from transformers import AutoTokenizer, AutoModel
 import torch
-from sklearn.metrics.pairwise import cosine_similarity
+import numpy as np
 
-tokenizer = AutoTokenizer.from_pretrained("jhu-clsp/mmBERT-base")
-model = AutoModel.from_pretrained("jhu-clsp/mmBERT-base")
+tokenizer = AutoTokenizer.from_pretrained("jhu-clsp/mmbert-base")
+model = AutoModel.from_pretrained("jhu-clsp/mmbert-base")
 
 def get_embeddings(texts):
-    inputs = tokenizer(texts, padding=True, truncation=True, return_tensors="pt")
-    
+    inputs = tokenizer(texts, return_tensors="pt", padding=True, truncation=True)
     with torch.no_grad():
         outputs = model(**inputs)
-        embeddings = outputs.last_hidden_state.mean(dim=1)
-    
+    # Mean pooling
+    embeddings = outputs.last_hidden_state.mean(dim=1)
     return embeddings.numpy()
 
-multilingual_texts = [
-    "Artificial intelligence is transforming technology",
-    "La inteligencia artificial está transformando la tecnología",
-    "L'intelligence artificielle transforme la technologie", 
-    "人工智能正在改变技术"
+# Multilingual document retrieval
+documents = [
+    "Artificial intelligence is transforming healthcare.",
+    "L'intelligence artificielle transforme les soins de santé.",
+    "人工智能正在改变医疗保健。",
+    "Climate change requires immediate action.",
+    "El cambio climático requiere acción inmediata."
 ]
 
-embeddings = get_embeddings(multilingual_texts)
-similarities = cosine_similarity(embeddings)
-print("Cross-lingual similarity matrix:")
-print(similarities)
+query = "AI in medicine"
+
+# Get embeddings
+doc_embeddings = get_embeddings(documents)
+query_embedding = get_embeddings([query])
+
+# Compute similarities
+similarities = np.dot(doc_embeddings, query_embedding.T).flatten()
+ranked_docs = np.argsort(similarities)[::-1]
+
+print("Most similar documents:")
+for i, doc_idx in enumerate(ranked_docs[:3]):
+    print(f"{i+1}. {documents[doc_idx]} (score: {similarities[doc_idx]:.3f})")
+```
+
+</details>
+
+### Long Context Processing
+
+<details>
+<summary><strong>Click to expand long context processing example</strong></summary>
+
+```python
+from transformers import AutoTokenizer, AutoModel
+
+tokenizer = AutoTokenizer.from_pretrained("jhu-clsp/mmbert-base")
+model = AutoModel.from_pretrained("jhu-clsp/mmbert-base")
+
+# Process long multilingual document (up to 8192 tokens)
+long_text = """
+This is a very long multilingual document...
+Ceci est un très long document multilingue...
+这是一个非常长的多语言文档...
+""" * 100  # Simulate long text
+
+# Tokenize with extended context
+inputs = tokenizer(
+    long_text, 
+    return_tensors="pt", 
+    max_length=8192,
+    truncation=True
+)
+
+# Process efficiently with Flash Attention
+with torch.no_grad():
+    outputs = model(**inputs)
+    
+print(f"Processed {inputs['input_ids'].shape[1]} tokens")
+print(f"Output shape: {outputs.last_hidden_state.shape}")
 ```
 
+</details>
+
 ## Fine-tuning Examples
 
 ### Dense Retrieval with Sentence Transformers
@@ -205,7 +289,6 @@ def main():
     args = parser.parse_args()
     
     lr = args.lr
-    model_name = args.model_name
     model_shortname = model_name.split("/")[-1]
 
     model = SentenceTransformer(model_name)
@@ -443,31 +526,134 @@ if __name__ == "__main__":
 
 </details>
 
-## Training Data
+## Model Architecture
 
-mmBERT was trained on a carefully curated 3T+ token multilingual dataset:
+| Parameter | mmBERT-small | mmBERT-base |
+|:----------|:-------------|:------------|
+| Layers | 22 | 22 |
+| Hidden Size | 384 | 768 |
+| Intermediate Size | 1152 | 1152 |
+| Attention Heads | 6 | 12 |
+| Total Parameters | 140M | 307M |
+| Non-embedding Parameters | 42M | 110M |
+| Max Sequence Length | 8192 | 8192 |
+| Vocabulary Size | 256,000 | 256,000 |
+| Tokenizer | Gemma 2 | Gemma 2 |
 
-| Phase | Dataset | Description |
-|:------|:--------|:------------|
-| [Pre-training P1](https://huggingface.co/datasets/jhu-clsp/mmbert-pretrain-p1-fineweb2-langs) | 2.3T tokens | 60 languages, diverse data mixture |
-| [Pre-training P2](https://huggingface.co/datasets/jhu-clsp/mmbert-pretrain-p2-fineweb2-langs) | - | Extension data for pre-training |
-| [Pre-training P3](https://huggingface.co/datasets/jhu-clsp/mmbert-pretrain-p3-fineweb2-langs) | - | Final pre-training data |
-| [Mid-training](https://huggingface.co/datasets/jhu-clsp/mmbert-midtraining-data) | 600B tokens | 110 languages, context extension |
-| [Decay Phase](https://huggingface.co/datasets/jhu-clsp/mmbert-decay-data) | 100B tokens | 1833 languages, premium quality |
+## Training
 
-**Primary Sources:**
-- **Filtered DCLM**: High-quality English content
-- **FineWeb2**: Broad multilingual web coverage (1800+ languages)
-- **FineWeb2-HQ**: Filtered subset of 20 high-resource languages
-- **Code**: StarCoder and ProLong repositories
-- **Academic**: ArXiv papers and PeS2o scientific content
-- **Reference**: Wikipedia (MegaWika) and textbooks
-- **Community**: StackExchange discussions
+Using 8xH100s, training took approximately 10 days for mmBERT-small and 40 days for mmBERT-base. 
+
+### Training Recipe: Cascading Annealed Language Learning
+
+mmBERT introduces novel training techniques:
+
+1.  **Inverse Masking Schedule**: Start with 30% masking, gradually reduce to 5%
+2.  **Language Progression**: 60 → 110 → 1833 languages across training phases  
+3.  **Temperature Annealing**: 0.7 → 0.5 → 0.3 for increasingly uniform language sampling
+4.  **High-Quality Data**: Progressive upgrade from web crawl to filtered premium sources
+
+### Training Details
 
+### Architecture
+
+| Component | Small | Base |
+|:----------|:------|:-----|
+| Layers | 22 | 22 |
+| Hidden Size | 384 | 768 |
+| Intermediate Size | 1152 | 1152 |
+| Attention Heads | 6 | 12 |
+| Parameters (Total) | 140M | 307M |
+| Parameters (Non-Embed) | 42M | 110M |
+| Max Sequence Length | 8192 | 8192 |
+| Vocabulary Size | 256,000 | 256,000 |
+
+### Training Configuration
+
+**Data Mixture:**
+*   Pre-training (2.0T tokens): Web crawl, code, scientific papers, reference materials
+*   Mid-training (600B tokens): Higher quality filtered data with context extension  
+*   Decay phase (100B tokens): Premium sources including textbooks and curated content
+
+**Architecture Features:**
+*   ModernBERT-based transformer with RoPE positional embeddings
+*   GLU activations and prenorm layer normalization
+*   Flash Attention 2 for efficient long-context processing
+*   Gemma 2 tokenizer for multilingual coverage
+
+**Training Phases:**
+1.  **Base Pre-training**: 60 languages, 30% masking, learning rate warmup
+2.  **Context Extension**: 110 languages, 15% masking, extended context to 8K
+3.  **Decay Phase**: 1833 languages, 5% masking, high-quality data focus
+
+## Evaluation
+Evaluation code for retrieval tasks is the same as [Ettin](https://github.com/JHU-CLSP/ettin-encoder-vs-decoder/tree/main/retrieval_eval).
+
+Evaluation code for efficiency is taken from the [ModernBERT](https://github.com/AnswerDotAI/ModernBERT/tree/main/efficiency) repo. 
+
+Evaluation code for NLU tasks is based on the [mGTE codebase](https://github.com/izhx/nlu-evals) and our fork will be uploaded soon. Please raise an issue or message us if this would be helpful for you.
+
+## FAQ
+
+**Q: How does mmBERT compare to XLM-R?**
+**A:** mmBERT significantly outperforms XLM-R across all benchmarks:
+- +2.4 points average on XTREME 
+- +3.0 points on GLUE
+- 16x more languages (1833 vs 100)
+- 16x longer context (8K vs 512 tokens)
+- 2-4x faster inference
+
+**Q: Which languages does mmBERT support?**
+**A:** mmBERT supports 1833 languages and scripts from FineWeb2, including:
+- All major world languages (English, Chinese, Spanish, etc.)
+- European languages (including low-resource ones like Faroese)
+- African languages (Swahili, Amharic, etc.)
+- Asian languages (Hindi, Bengali, Thai, etc.)
+- Many low-resource and indigenous languages
+
+**Q: How does the annealed language learning work?**
+**A:** We progressively add languages in three phases:
+1. Start with 60 high-resource languages (pre-training)
+2. Add 50 mid-resource languages (mid-training) 
+3. Add 1723 low-resource languages (decay phase)
+
+This allows efficient learning without overfitting on low-resource data.
+
+**Q: Can I fine-tune mmBERT for my specific task?**
+**A:** Yes! mmBERT works as a drop-in replacement for XLM-R:
+```python
+from transformers import AutoModel, AutoTokenizer
+
+# Load for fine-tuning
+model = AutoModel.from_pretrained("jhu-clsp/mmbert-base")
+tokenizer = AutoTokenizer.from_pretrained("jhu-clsp/mmbert-base")
+
+# Add task-specific head and fine-tune normally
+```
+
+**Q: What about efficiency and memory requirements?**
+**A:** mmBERT is significantly more efficient:
+- 2-4x faster inference than XLM-R
+- Flash Attention 2 reduces memory usage for long sequences
+- Support for variable-length batching
+- Optimized for both CPU and GPU deployment
+
+**Q: How do I access the training data and checkpoints?**
+**A:** All data and checkpoints are publicly available:
+- Training data: [jhu-clsp/mmbert-pretraining-data](https://huggingface.co/datasets/jhu-clsp/mmbert-pretraining-data)
+- Checkpoints: [jhu-clsp/mmbert-checkpoints](https://huggingface.co/models/jhu-clsp/mmbert-checkpoints)
+- Github code: [GitHub repository](https://github.com/jhu-clsp/mmBERT)
+- Data processing code: [Same as Ettin models](https://github.com/jhu-clsp/ettin-encoder-vs-decoder)
+
+## Limitations
+
+- Structured prediction tasks (NER, POS) show slightly lower scores due to tokenizer prefix space handling
+- Very low-resource languages still have limited training data
+- High-quality educational content filtering could benefit from more languages
 
 ## Citation
 
-If you use mmBERT in your research, please cite our work:
+If you use mmBERT models in your research, please cite our work:
 
 ```bibtex
 @misc{marone2025mmbertmodernmultilingualencoder,
@@ -479,5 +665,4 @@ If you use mmBERT in your research, please cite our work:
       primaryClass={cs.CL},
       url={https://arxiv.org/abs/2509.06888}, 
 }
-```
-"""
+```