ramiroc3/tds-bert-pos-ewt

BERT POS Tagging on UD English EWT (Winning Model)

Intended Use

English Part-of-Speech (POS) tagging using Universal Dependencies EWT (17 UPOS tags) developed for an academic course assignment. Not intended for production-critical NLP applications without domain-specific validation.

Model and Training Details

• Base Model: bert-base-uncased
• Dataset: universal-dependencies/universal_dependencies (en_ewt).
• Adaptation Strategy: Full Fine-tuning with differentiated learning rates; random seed 42; 3 epochs; batch size 16.
• Optimizer Configuration: Discriminative learning rates (Encoder LR: 2e-5, Classifier Head LR: 1e-3).
• Label Alignment: Supervised the first subtoken only; assigned -100 to subword continuations and special tokens.

Evaluation Results

Evaluation performed on the validation set (dev split):

• Validation Loss: 0.110305
• Validation Accuracy: 0.974352 (97.44%)
• Validation Macro F1: 0.931155

Comparison with Baseline

Model Variant	Accuracy	Macro F1
Feature-based Baseline (Logistic Regression)	0.941665	0.893589
Full Fine-tuning (This Model)	0.974352	0.931155

Limitations

Trained on English Web Treebank (EWT). Performance may degrade on informal web text, domain-specific terminology, or noisy inputs. Input sequences exceeding maximum sequence length are truncated.

References

• Universal Dependencies EWT Framework.
• Devlin et al. (2019), BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.
• Hugging Face Token Classification Guide.