Create README.md

README.md

@@ -0,0 +1,45 @@
+---
+language:
+- en
+license: apache-2.0
+library_name: transformers; pytorch
+tags:
+- speech
+---
+
+### Contrastive user encoder
+This model is a `DistilBertModel` trained by fine-tuning `distilbert-base-uncased` on author-based triplet loss.
+
+#### Details
+Training and evaluation details are provided in our EMNLP Findings paper:
+- Rocca, R., & Yarkoni, T. (2022), Language models as user encoders: Self-supervised learning of user encodings using transformers, to appear in *Findings of the Association for Computational Linguistics: EMNLP 2022*
+
+
+#### Training
+We fine-tuned DistilBERT on triplets consisting of: 
+- a Reddit submission from a given user (the "anchor");
+- an additional post from the same user (a "positive example"); 
+- a post from a different, randomly selected user (the "negative example")
+
+
+To compute the loss, we use [CLS] encoding of the anchor, positive example and negative example from the last layer of the DistilBERT encoder. We optimize for \\(max(||f(a) - f(n)|| - ||\overline{f(A)} - f(p)|| + \alpha,0)\\)
+ 
+ where:
+ - \\( f(a)\\) is the [CLS] encoding of the anchor; 
+ - \\( f(n) \\) is the [CLS] encoding of the negative example;
+ - \\( f(p) \\) is the [CLS] encoding of the positive example;
+ - \\( \alpha \\) is a tunable parameter called margin. Here, we tuned this to \\( \alpha = 1.0\\)
+ 
+ 
+#### Evaluation and usage
+The model yields performance advantages downstream user-based classification tasks.
+
+We encourage usage and benchmarking on tasks involving:
+- prediction of user traits (e.g., personality);
+- extraction of user-aware text encodings (e.g., style modeling);
+- contextualized text modeling, where standard text representations are complemented with compact user representations
+
+#### Limitations
+Being exclusively trained on Reddit data, our models probably overfit to linguistic markers and traits which are relevant to characterizing the Reddit user population, but less salient in the general population. Domain-specific fine-tuning may be required before deployment.
+
+Furthermore, our self-supervised approach enforces little or no control over biases, which models may actively use as part of their heuristics in contrastive and downstream tasks.