Create README.md

README.md

@@ -0,0 +1,53 @@
+# Model Details
+The CFT-CLIP was developed by HUMANE Lab researchers at Soongsil University to understand news thumbnail representativeness by counterfactual text-guided contrastive language-image pretraining.
+
+# Model Date
+January 2024
+
+# Model Type
+The model uses a ViT-L/14 transformer architecture as an image encoder and a causal text transformer as a text encoder. These encoders initialized weight for openai/clip-vit-large-patch14 before training. It is trained that the similarity of positive (image, text) pairs is high, and the similarity of in-batch negatives and hard negatives is low via contrastive loss.
+
+Input: image and text
+
+output: image and text representation
+
+
+# Intended Use
+The model is intended as a research output for research communities. 
+
+# Primary intended uses
+The primary intended users of these models are AI researchers.
+
+# Out-of-Scope Use Cases
+The model was not intentionally trained or evaluated in any language other than English. Therefore, use of the model should be limited to English use cases.
+
+# Factors
+# Environment
+This model was trained on a machine equipped with AMD Ryzen Threadripper Pro 5975WX CPU, three Nvidia RTX A6000 GPUs (48GB per GPU), and 256GB RAM. The experiments were conducted on Python 3.9, Pytorch 1.10.1, Transformers 4.29.2, LAVIS 1.0.2, and SentenceTransformer 2.2.2. Five random seeds were used for repeated experiments: 0, 1, 2, 3, and 4. The temperature used for adjusting the masked token prediction is set as 2.0.
+
+# Card Prompts
+# Relevant factors
+We trained the models with the AdamW optimizer with the initial learning rate of 1e-4, updated by the cosine annealing scheduler.The minibatch size is 128. The temperature τ in the loss equation is 0.05. Other hyperparameters were optimized by random search using a validation set. Model training was early-stopped when the validation loss was not decreased five times consecutively, measured for every 20 iterations.
+
+# Evaluation factors
+
+# Metrics
+Model performance measures: F1-score between model predictions and labels and Spearman between cosine similarity of models between labels.
+
+Decision thresholds: validation
+
+Approaches to uncertainty and variability: Measure by changing the random seed 5 times
+
+
+# Data
+# Training Data
+The model was trained using the summary text and thumbnail image for the image in the first paragraph of the publicly available BBC English Dataset.
+The original implementation had two variants: one using a NELA-GT-2021 and the other using the titles instead of summary text from BBC Dataset.
+
+# Evaluation Data
+In NELA-GT-2021, annotation was performed by randomly sampling 1,000 in 10,000 samples not included in the train and valid set. 
+Ethical Considerations
+Because CLIP's weights are used, potential bias in CLIP and potential bias in the data used for learning may also be included.
+
+
+# Caveats and Recommendations