update readme

README.md

@@ -0,0 +1,66 @@
+
+---
+language: en
+---
+
+# ChartVE (Chart Visual Entailment)
+
+ChartVE is a visual entailment model introduced in the paper "Do LVLMs Understand Charts?  
+Analyzing and Correcting Factual Errors in Chart Captioning" for evaluating the factuality of a generated caption sentence with regard to the input chart. The model takes in a chart figure and a caption sentence as input, and outputs an entailment probability.  To compute the the entailment probability, please refer to the "How to use" section below.  The underlying architecture of this model is UniChart.
+
+
+### How to use
+
+Using the pre-trained model directly:
+```python
+from transformers import DonutProcessor, VisionEncoderDecoderModel
+from PIL import Image
+
+model_name = "khhuang/chartve"
+model = VisionEncoderDecoderModel.from_pretrained(model_name).cuda()
+processor = DonutProcessor.from_pretrained(model_name)
+
+image_path = "PATH_TO_IMAGE"
+
+def format_query(sentence):
+    return f"Does the image entails this statement: \"{sentence}\"?"
+
+# Format text inputs
+CAPTION_SENTENCE = "The state that has the highest number of population is California."
+query = format_query(CAPTION_SENTENCE)
+
+# Encode chart figure and tokenize text
+img = Image.open(IMAGE_PATH)
+pixel_values = processor(img.convert("RGB"), random_padding=False, return_tensors="pt").pixel_values
+pixel_values = pixel_values.cuda()
+decoder_input_ids = processor.tokenizer(query, add_special_tokens=False, return_tensors="pt", max_length=510).input_ids.cuda()#.squeeze(0)
+
+
+outputs = model(pixel_values, decoder_input_ids=decoder_input_ids)
+
+# positive_logit = outputs['logits'].squeeze()[-1,49922]
+# negative_logit = outputs['logits'].squeeze()[-1,2334] 
+
+# Probe the probability of generating "yes"
+binary_entail_prob_positive = torch.nn.functional.softmax(outputs['logits'].squeeze()[-1,[2334, 49922]])[1].item()
+
+# binary_entail_prob_positive corresponds to the computed probability that the chart entails the caption sentence.
+```
+
+### Citation
+```
+@inproceedings{huang-etal-2023-do,
+    title = "Zero-shot Faithful Factual Error Correction",
+    author = "Huang, Kung-Hsiang  and
+      Zhou, Mingyang and
+      Chan, Hou Pong  and
+      Fung, Yi R. and
+      Wang, Zhenhailong and
+      Zhang, Lingyu and
+      Chang, Shih-Fu and
+      Ji, Heng",
+    year={2023},
+    archivePrefix={arXiv},
+    primaryClass={cs.CL}
+```
+}