Create README.md

README.md

@@ -0,0 +1,20 @@
+---
+license: mit
+datasets:
+- omarmomen/babylm_10M
+language:
+- en
+metrics:
+- perplexity
+library_name: transformers
+---
+# Model Card for omarmomen/structformer_s2_final_with_pos
+
+This model is part of the experiments in the published paper at the BabyLM workshop in CoNLL 2023. 
+The paper titled "Increasing The Performance of Cognitively Inspired Data-Efficient Language Models via Implicit Structure Building" (https://aclanthology.org/2023.conll-babylm.29/)
+
+<strong>omarmomen/structformer_s2_final_with_pos</strong> is a modification of the vanilla transformer encoder to incorporate syntactic inductive bias using an unsupervised parsing mechanism.
+
+This model variant places the parser network after 4 attention blocks.
+
+The model is pretrained on the BabyLM 10M dataset using a custom pretrained RobertaTokenizer (https://huggingface.co/omarmomen/babylm_tokenizer_32k).