Upload 21 files

CODEOWNERS

@@ -0,0 +1,2 @@
+# Comment line immediately above ownership line is reserved for related gus information. Please be careful while editing.
+#ECCN:Open Source

CODE_OF_CONDUCT.md

@@ -0,0 +1,105 @@
+# Salesforce Open Source Community Code of Conduct
+
+## About the Code of Conduct
+
+Equality is a core value at Salesforce. We believe a diverse and inclusive
+community fosters innovation and creativity, and are committed to building a
+culture where everyone feels included.
+
+Salesforce open-source projects are committed to providing a friendly, safe, and
+welcoming environment for all, regardless of gender identity and expression,
+sexual orientation, disability, physical appearance, body size, ethnicity, nationality, 
+race, age, religion, level of experience, education, socioeconomic status, or 
+other similar personal characteristics.
+
+The goal of this code of conduct is to specify a baseline standard of behavior so
+that people with different social values and communication styles can work
+together effectively, productively, and respectfully in our open source community.
+It also establishes a mechanism for reporting issues and resolving conflicts.
+
+All questions and reports of abusive, harassing, or otherwise unacceptable behavior
+in a Salesforce open-source project may be reported by contacting the Salesforce
+Open Source Conduct Committee at ossconduct@salesforce.com.
+
+## Our Pledge
+
+In the interest of fostering an open and welcoming environment, we as
+contributors and maintainers pledge to making participation in our project and
+our community a harassment-free experience for everyone, regardless of gender 
+identity and expression, sexual orientation, disability, physical appearance, 
+body size, ethnicity, nationality, race, age, religion, level of experience, education, 
+socioeconomic status, or other similar personal characteristics.
+
+## Our Standards
+
+Examples of behavior that contributes to creating a positive environment
+include:
+
+* Using welcoming and inclusive language
+* Being respectful of differing viewpoints and experiences
+* Gracefully accepting constructive criticism
+* Focusing on what is best for the community
+* Showing empathy toward other community members
+
+Examples of unacceptable behavior by participants include:
+
+* The use of sexualized language or imagery and unwelcome sexual attention or
+advances
+* Personal attacks, insulting/derogatory comments, or trolling
+* Public or private harassment
+* Publishing, or threatening to publish, others' private information—such as
+a physical or electronic address—without explicit permission
+* Other conduct which could reasonably be considered inappropriate in a
+professional setting
+* Advocating for or encouraging any of the above behaviors
+
+## Our Responsibilities
+
+Project maintainers are responsible for clarifying the standards of acceptable
+behavior and are expected to take appropriate and fair corrective action in
+response to any instances of unacceptable behavior.
+
+Project maintainers have the right and responsibility to remove, edit, or
+reject comments, commits, code, wiki edits, issues, and other contributions
+that are not aligned with this Code of Conduct, or to ban temporarily or
+permanently any contributor for other behaviors that they deem inappropriate,
+threatening, offensive, or harmful.
+
+## Scope
+
+This Code of Conduct applies both within project spaces and in public spaces
+when an individual is representing the project or its community. Examples of
+representing a project or community include using an official project email
+address, posting via an official social media account, or acting as an appointed
+representative at an online or offline event. Representation of a project may be
+further defined and clarified by project maintainers.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported by contacting the Salesforce Open Source Conduct Committee 
+at ossconduct@salesforce.com. All complaints will be reviewed and investigated 
+and will result in a response that is deemed necessary and appropriate to the 
+circumstances. The committee is obligated to maintain confidentiality with 
+regard to the reporter of an incident. Further details of specific enforcement 
+policies may be posted separately.
+
+Project maintainers who do not follow or enforce the Code of Conduct in good
+faith may face temporary or permanent repercussions as determined by other
+members of the project's leadership and the Salesforce Open Source Conduct 
+Committee.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][contributor-covenant-home],
+version 1.4, available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html. 
+It includes adaptions and additions from [Go Community Code of Conduct][golang-coc], 
+[CNCF Code of Conduct][cncf-coc], and [Microsoft Open Source Code of Conduct][microsoft-coc].
+
+This Code of Conduct is licensed under the [Creative Commons Attribution 3.0 License][cc-by-3-us].
+
+[contributor-covenant-home]: https://www.contributor-covenant.org (https://www.contributor-covenant.org/)
+[golang-coc]: https://golang.org/conduct
+[cncf-coc]: https://github.com/cncf/foundation/blob/master/code-of-conduct.md
+[microsoft-coc]: https://opensource.microsoft.com/codeofconduct/
+[cc-by-3-us]: https://creativecommons.org/licenses/by/3.0/us/

LICENSE.txt

@@ -0,0 +1,12 @@
+Copyright (c) 2022, Salesforce.com, Inc.
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
+
+* Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
+
+* Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
+
+* Neither the name of Salesforce.com nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

README.md

@@ -1,12 +1,116 @@
----
-title: SCRIPTure Demo
-emoji: 🚀
-colorFrom: blue
-colorTo: gray
-sdk: gradio
-sdk_version: 3.5
-app_file: app.py
-pinned: false
----
-
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+## BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
+
+## Announcement: BLIP is now officially integrated into [LAVIS](https://github.com/salesforce/LAVIS) - a one-stop library for language-and-vision research and applications!
+
+<img src="BLIP.gif" width="700">
+
+This is the PyTorch code of the <a href="https://arxiv.org/abs/2201.12086">BLIP paper</a> [[blog](https://blog.salesforceairesearch.com/blip-bootstrapping-language-image-pretraining/)]. The code has been tested on PyTorch 1.10.
+To install the dependencies, run <pre/>pip install -r requirements.txt</pre> 
+
+Catalog:
+- [x] Inference demo
+- [x] Pre-trained and finetuned checkpoints
+- [x] Finetuning code for Image-Text Retrieval, Image Captioning, VQA, and NLVR2
+- [x] Pre-training code
+- [x] Zero-shot video-text retrieval
+- [x] Download of bootstrapped pre-training datasets 
+
+
+### Inference demo:
+Run our interactive demo using [Colab notebook](https://colab.research.google.com/github/salesforce/BLIP/blob/main/demo.ipynb) (no GPU needed).
+The demo includes code for: 
+1. Image captioning
+2. Open-ended visual question answering
+3. Multimodal / unimodal feature extraction
+4. Image-text matching
+
+Try out the [Web demo](https://huggingface.co/spaces/Salesforce/BLIP), integrated into [Huggingface Spaces 🤗](https://huggingface.co/spaces) using [Gradio](https://github.com/gradio-app/gradio). 
+
+Replicate web demo and Docker image is also available at [![Replicate](https://replicate.com/salesforce/blip/badge)](https://replicate.com/salesforce/blip)
+
+### Pre-trained checkpoints:
+Num. pre-train images | BLIP w/ ViT-B | BLIP w/ ViT-B and CapFilt-L | BLIP w/ ViT-L 
+--- | :---: | :---: | :---: 
+14M | <a href="https://storage.googleapis.com/sfr-vision-language-research/BLIP/models/model_base_14M.pth">Download</a>| - | -
+129M | <a href="https://storage.googleapis.com/sfr-vision-language-research/BLIP/models/model_base.pth">Download</a>| <a href="https://storage.googleapis.com/sfr-vision-language-research/BLIP/models/model_base_capfilt_large.pth">Download</a> | <a href="https://storage.googleapis.com/sfr-vision-language-research/BLIP/models/model_large.pth">Download</a>
+
+### Finetuned checkpoints:
+Task | BLIP w/ ViT-B | BLIP w/ ViT-B and CapFilt-L | BLIP w/ ViT-L 
+--- | :---: | :---: | :---:
+Image-Text Retrieval (COCO) | <a href="https://storage.googleapis.com/sfr-vision-language-research/BLIP/models/model_base_retrieval_coco.pth">Download</a>| - | <a href="https://storage.googleapis.com/sfr-vision-language-research/BLIP/models/model_large_retrieval_coco.pth">Download</a>
+Image-Text Retrieval (Flickr30k) | <a href="https://storage.googleapis.com/sfr-vision-language-research/BLIP/models/model_base_retrieval_flickr.pth">Download</a>|  - | <a href="https://storage.googleapis.com/sfr-vision-language-research/BLIP/models/model_large_retrieval_flickr.pth">Download</a>
+Image Captioning (COCO) | - | <a href="https://storage.googleapis.com/sfr-vision-language-research/BLIP/models/model_base_caption_capfilt_large.pth">Download</a>| <a href="https://storage.googleapis.com/sfr-vision-language-research/BLIP/models/model_large_caption.pth">Download</a> | 
+VQA | <a href="https://storage.googleapis.com/sfr-vision-language-research/BLIP/models/model_vqa.pth">Download</a>| <a href="https://storage.googleapis.com/sfr-vision-language-research/BLIP/models/model_base_vqa_capfilt_large.pth">Download</a> | - 
+NLVR2 | <a href="https://storage.googleapis.com/sfr-vision-language-research/BLIP/models/model_base_nlvr.pth">Download</a>| - | - 
+
+
+### Image-Text Retrieval:
+1. Download COCO and Flickr30k datasets from the original websites, and set 'image_root' in configs/retrieval_{dataset}.yaml accordingly.
+2. To evaluate the finetuned BLIP model on COCO, run:
+<pre>python -m torch.distributed.run --nproc_per_node=8 train_retrieval.py \
+--config ./configs/retrieval_coco.yaml \
+--output_dir output/retrieval_coco \
+--evaluate</pre> 
+3. To finetune the pre-trained checkpoint using 8 A100 GPUs, first set 'pretrained' in configs/retrieval_coco.yaml as "https://storage.googleapis.com/sfr-vision-language-research/BLIP/models/model_base.pth". Then run:
+<pre>python -m torch.distributed.run --nproc_per_node=8 train_retrieval.py \
+--config ./configs/retrieval_coco.yaml \
+--output_dir output/retrieval_coco </pre> 
+
+### Image-Text Captioning:
+1. Download COCO and NoCaps datasets from the original websites, and set 'image_root' in configs/caption_coco.yaml and configs/nocaps.yaml accordingly.
+2. To evaluate the finetuned BLIP model on COCO, run:
+<pre>python -m torch.distributed.run --nproc_per_node=8 train_caption.py --evaluate</pre> 
+3. To evaluate the finetuned BLIP model on NoCaps, generate results with: (evaluation needs to be performed on official server)
+<pre>python -m torch.distributed.run --nproc_per_node=8 eval_nocaps.py </pre> 
+4. To finetune the pre-trained checkpoint using 8 A100 GPUs, first set 'pretrained' in configs/caption_coco.yaml as "https://storage.googleapis.com/sfr-vision-language-research/BLIP/models/model_base_capfilt_large.pth". Then run:
+<pre>python -m torch.distributed.run --nproc_per_node=8 train_caption.py </pre> 
+
+### VQA:
+1. Download VQA v2 dataset and Visual Genome dataset from the original websites, and set 'vqa_root' and 'vg_root' in configs/vqa.yaml.
+2. To evaluate the finetuned BLIP model, generate results with: (evaluation needs to be performed on official server)
+<pre>python -m torch.distributed.run --nproc_per_node=8 train_vqa.py --evaluate</pre> 
+3. To finetune the pre-trained checkpoint using 16 A100 GPUs, first set 'pretrained' in configs/vqa.yaml as "https://storage.googleapis.com/sfr-vision-language-research/BLIP/models/model_base_capfilt_large.pth". Then run:
+<pre>python -m torch.distributed.run --nproc_per_node=16 train_vqa.py </pre> 
+
+### NLVR2:
+1. Download NLVR2 dataset from the original websites, and set 'image_root' in configs/nlvr.yaml.
+2. To evaluate the finetuned BLIP model, run
+<pre>python -m torch.distributed.run --nproc_per_node=8 train_nlvr.py --evaluate</pre> 
+3. To finetune the pre-trained checkpoint using 16 A100 GPUs, first set 'pretrained' in configs/nlvr.yaml as "https://storage.googleapis.com/sfr-vision-language-research/BLIP/models/model_base.pth". Then run:
+<pre>python -m torch.distributed.run --nproc_per_node=16 train_nlvr.py </pre> 
+
+### Finetune with ViT-L:
+In order to finetune a model with ViT-L, simply change the config file to set 'vit' as large. Batch size and learning rate may also need to be adjusted accordingly (please see the paper's appendix for hyper-parameter details). <a href="https://github.com/facebookresearch/fairscale">Gradient checkpoint</a> can also be activated in the config file to reduce GPU memory usage. 
+
+### Pre-train:
+1. Prepare training json files where each json file contains a list. Each item in the list is a dictonary with two key-value pairs: {'image': path_of_image, 'caption': text_of_image}. 
+2. In configs/pretrain.yaml, set 'train_file' as the paths for the json files .
+3. Pre-train the model using 8 A100 GPUs:
+<pre>python -m torch.distributed.run --nproc_per_node=8 pretrain.py --config ./configs/Pretrain.yaml --output_dir output/Pretrain </pre> 
+
+### Zero-shot video-text retrieval:
+1. Download MSRVTT dataset following the instructions from https://github.com/salesforce/ALPRO, and set 'video_root' accordingly in configs/retrieval_msrvtt.yaml.
+2. Install [decord](https://github.com/dmlc/decord) with <pre>pip install decord</pre> 
+3. To perform zero-shot evaluation, run
+<pre>python -m torch.distributed.run --nproc_per_node=8 eval_retrieval_video.py</pre> 
+
+### Pre-training datasets download:
+We provide bootstrapped pre-training datasets as json files. Each json file contains a list. Each item in the list is a dictonary with two key-value pairs: {'url': url_of_image, 'caption': text_of_image}. 
+
+Image source | Filtered web caption | Filtered synthetic caption by ViT-B | Filtered synthetic caption by ViT-L
+--- | :---: | :---: | :---:
+CC3M+CC12M+SBU |  <a href="https://storage.googleapis.com/sfr-vision-language-research/BLIP/datasets/ccs_filtered.json">Download</a>|  <a href="https://storage.googleapis.com/sfr-vision-language-research/BLIP/datasets/ccs_synthetic_filtered.json">Download</a>|  <a href="https://storage.googleapis.com/sfr-vision-language-research/BLIP/datasets/ccs_synthetic_filtered_large.json">Download</a>
+LAION115M | <a href="https://storage.googleapis.com/sfr-vision-language-research/BLIP/datasets/laion_filtered.json">Download</a>|  <a href="https://storage.googleapis.com/sfr-vision-language-research/BLIP/datasets/laion_synthetic_filtered.json">Download</a>|  <a href="https://storage.googleapis.com/sfr-vision-language-research/BLIP/datasets/laion_synthetic_filtered_large.json">Download</a>
+
+### Citation
+If you find this code to be useful for your research, please consider citing.
+<pre>
+@inproceedings{li2022blip,
+      title={BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation}, 
+      author={Junnan Li and Dongxu Li and Caiming Xiong and Steven Hoi},
+      year={2022},
+      booktitle={ICML},
+}</pre>
+
+### Acknowledgement
+The implementation of BLIP relies on resources from <a href="https://github.com/salesforce/ALBEF">ALBEF</a>, <a href="https://github.com/huggingface/transformers">Huggingface Transformers</a>, and <a href="https://github.com/rwightman/pytorch-image-models/tree/master/timm">timm</a>. We thank the original authors for their open-sourcing.

SECURITY.md

@@ -0,0 +1,7 @@
+## Security
+
+Please report any security issue to [security@salesforce.com](mailto:security@salesforce.com)
+as soon as it is discovered. This library limits its runtime dependencies in
+order to reduce the total cost of ownership as much as can be, but all consumers
+should remain vigilant and have their security stakeholders review all third-party
+products (3PP) like this one and their dependencies.

app.py

@@ -0,0 +1,143 @@
+from PIL import Image
+import requests
+import torch
+from torchvision import transforms
+import os
+from torchvision.transforms.functional import InterpolationMode
+import matplotlib.pyplot as plt
+import matplotlib.image as mpimg
+import cohere
+import gradio as gr
+
+device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
+
+from models.blip import blip_decoder
+
+image_size = 384
+transform = transforms.Compose([
+    transforms.Resize((image_size,image_size),interpolation=InterpolationMode.BICUBIC),
+    transforms.ToTensor(),
+    transforms.Normalize((0.48145466, 0.4578275, 0.40821073), (0.26862954, 0.26130258, 0.27577711))
+    ]) 
+
+model_url = 'https://storage.googleapis.com/sfr-vision-language-research/BLIP/models/model_large_caption.pth'
+    
+model = blip_decoder(pretrained=model_url, image_size=384, vit='large')
+model.eval()
+model = model.to(device)
+
+
+from models.blip_vqa import blip_vqa
+
+image_size_vq = 480
+transform_vq = transforms.Compose([
+    transforms.Resize((image_size_vq,image_size_vq),interpolation=InterpolationMode.BICUBIC),
+    transforms.ToTensor(),
+    transforms.Normalize((0.48145466, 0.4578275, 0.40821073), (0.26862954, 0.26130258, 0.27577711))
+    ]) 
+
+model_url_vq = 'https://storage.googleapis.com/sfr-vision-language-research/BLIP/models/model*_vqa.pth'
+    
+model_vq = blip_vqa(pretrained=model_url_vq, image_size=480, vit='base')
+model_vq.eval()
+model_vq = model_vq.to(device)
+
+
+
+def inference(raw_image, model_n, question="", strategy=""):
+    if model_n == 'Image Captioning':
+        image = transform(raw_image).unsqueeze(0).to(device)   
+        with torch.no_grad():
+            if strategy == "Beam search":
+                caption = model.generate(image, sample=False, num_beams=3, max_length=20, min_length=5)
+            else:
+                caption = model.generate(image, sample=True, top_p=0.9, max_length=20, min_length=5)
+            return 'caption: '+caption[0]
+
+    else:   
+        image_vq = transform_vq(raw_image).unsqueeze(0).to(device)  
+        with torch.no_grad():
+            answer = model_vq(image_vq, question, train=False, inference='generate') 
+        return  'answer: '+answer[0]
+
+#get caption for a single iamge
+def get_caption(image_path):
+  img = Image.open(image_path)
+  return inference(img, "Image Captioning")[9:]
+
+def display(image_path):
+  img = mpimg.imread(image_path)
+  img = Image.open(image_path)
+  plt.imshow(img)
+  print("Caption: " + get_caption(image_path))
+  
+#returns a dictionary with key -> img_path and value -> caption
+def get_captions(img_directory, print_status=True):
+    #key is img path, value is the caption 
+    captions = {}
+    length = 0
+    for file in os.listdir(img_directory):
+      length+=1
+    count = 0
+    for file in os.listdir(img_directory):
+        f = os.path.join(img_directory, file)
+        captions[f] = inference(Image.open(f), "Image Captioning")
+        if print_status:
+          print("Images complete:", str(count) + "/" + str(length))
+          print("Caption:", captions[f])
+    return captions
+#writes dictionary to file, key and value seperated by ':'
+def write_to_file(filename, caption_dict):
+  with open(filename, "w") as file:
+    for i in caption_dict:
+      file.write(i + ":" + caption_dict[i])
+  file.close()
+
+  # Text to Image API
+
+import requests
+import base64
+
+
+#add max tokens a slider
+
+def make_image_and_story(prompt):
+  if(prompt is None or prompt == ""):
+    host = 'https://dev.paint.cohere.ai/txt2img'
+    response = requests.post(host, json={'prompt': 'Random monster', 'n_samples' : 1, 'n_iter' : 1})
+
+    # decode image
+    imageBytes = base64.b64decode(response.json()['image']) #decode
+
+    # save to disk
+    f = open("sample.png", "wb")
+    f.write(imageBytes)
+    f.close()
+
+    caption = get_caption("sample.png")
+
+    co = cohere.Client('yRfs5ozta7DQtTF0duztE9bV7CNulvcxwuqJizhB')
+    response = co.generate(prompt=caption, model ='c0381280-2035-4042-a5a0-01f5800bd9c0-ft', max_tokens=80)
+
+    return Image.open("sample.png"), response.generations[0].text
+  else:
+    host = 'https://dev.paint.cohere.ai/txt2img'
+    response = requests.post(host, json={'prompt': prompt, 'n_samples' : 1, 'n_iter' : 1})
+
+    # decode image
+    imageBytes = base64.b64decode(response.json()['image']) #decode
+
+    # save to disk
+    f = open("sample.png", "wb")
+    f.write(imageBytes)
+    f.close()
+
+    caption = get_caption("sample.png")
+
+    co = cohere.Client('yRfs5ozta7DQtTF0duztE9bV7CNulvcxwuqJizhB')
+    response = co.generate(prompt=caption, model ='c0381280-2035-4042-a5a0-01f5800bd9c0-ft', max_tokens=80)
+
+    return Image.open("sample.png"), response.generations[0].text
+
+
+gr.Interface(fn=make_image_and_story, inputs="text", outputs=["image","text"],title='Fantasy Creature Generator').launch();

cog.yaml

@@ -0,0 +1,17 @@
+build:
+  gpu: true
+  cuda: "11.1"
+  python_version: "3.8"
+  system_packages:
+    - "libgl1-mesa-glx"
+    - "libglib2.0-0"
+  python_packages:
+    - "ipython==7.30.1"
+    - "torchvision==0.11.1"
+    - "torch==1.10.0"
+    - "timm==0.4.12"
+    - "transformers==4.15.0"
+    - "fairscale==0.4.4"
+    - "pycocoevalcap==1.2"
+
+predict: "predict.py:Predictor"

data.txt

@@ -0,0 +1,300 @@
+The Adaro is half human; half fish much like that of a merfolk. They are said to live in the Sun and travel to earth by rainbows. Adaro's are said to travel around the oceans on waterspouts. Unlike merfolk, Adaro are not friendly they attack humans by shooting them with flying fish.
+--SEPARATOR--
+Brownies are invisible brown elves or household goblins who live in farmhouses and other country buildings within Scotland. While the members of the household are asleep they go about doing labours for the house owners. Brownies are protective creatures and become attached to the families if the family move the Brownie will move with them. If a Brownie is treated badly by the family or is offered payment the Brownie vanishes without trace. Children because of their innocent nature can only see Brownies, though this does not prevent the Brownies from helping adults.
+--SEPARATOR--
+The Calygreyhound is a creature from medieval heraldry; it has the body of a deer, the claws of an eagle on its forelegs and the hooves of an ox on its hind legs. The Calygreyhound is meant to symbolise swiftness.
+--SEPARATOR--
+The Chinese Fox looks like a typical fox but has a life span between 800 and 1000 years. The Chinese fox has special powers, when it strikes it tail on the ground it can ignite fires. The it is also said that the Chinese fox can see into the future. This animal is a polymorph (can change its shape at will). The Chinese fox often appears as an old man, young girl or an academic. The Chinese fox is a trickster and a sly animal it's thought to be a bad omen to see such an animal. Chinese foxes can be found around graveyards because the souls of the dead can relocate into the body of the fox.
+--SEPARATOR--
+The Crocotta is offspring of a wolf and a dog that can be found in Ethiopia. The Crocotta was said to be able to break anything with its teeth and can eat anything. It is now thought that this creature is in fact a Hyena as they are scavenging canine creatures that live in this part of the world.
+--SEPARATOR--
+A Dire wolf is a large wolf, which are usually portrayed as been much more vicious than a normal wolf and also more intelligent. Dire Wolves are usually black though dark greys are popular as well. There is a second description of a dire wolf that says that a dire wolf is a giant wolf that has been brought back from beyond the grave to hunt. Dire wolves tend to hunt in packs like there wolf relatives; a dire wolf can often be found leading packs of normal wolves.
+--SEPARATOR--
+Doppelganger means double walker which in normal terms is a duplicate of another person or creature. A doppelganger does not necessary have a blood relationship with its double. A doppelganger may therefore be either an astral projection or a ghost. Examples of doppelganger can be found in Robert Louis Stevenson's "Strange case of Dr Jekyll and Mr Hyde".
+--SEPARATOR--
+A Gorgon is a terrifying creature, they look similar to a human apart from its legs are replaced by a snakes tail, and that the hair of these creatures is made up of snakes. The gaze of a gorgon can turn its victim to stone if only it you look the gorgon in their eyes.
+
+The Gorgon comes from Greek mythology; there were three the first was Medusa who was a mortal gorgon, who was slain by Perseus who killed her by using the reflection that she cast onto his shield. Medusa had two sisters Sthens and Euryule who were immortal.
+--SEPARATOR--
+Gremlins are spirits of tools and machinery. They are thought to be responsible for mishaps and breakdowns with tools and equipment. Each house has a Gremlin which entered the house as an occupant of a household appliance. Originally Gremlins were friendly towards mankind and helped engineers and inventors build things but when these people took all the credit the Gremlins were insulted and from that point on they work against us.
+--SEPARATOR--
+The Gulon is mythical creature from Scandinavia, it has the front half of a lion the rear half of a hyena and the bushy tail of a fox. The creature is said to be vicious and has the sharp claws of the lion to attack with. It has been used as a symbol of gluttony.
+--SEPARATOR--
+The Haunt are dog like creatures, they are usually black. Males have two deep set red eyes and female's two dark blue eyes. They live in caves deep in mountains and can live with little food and water. Every full moon the males come down into nearby cities and towns to hunt. They have Poisonous saliva and claws that can drip with poison.
+
+The Haunt
+These creatures of the darkness come down from the mountain,
+Hunting for prey,
+For they need to feed I hear them growling,
+Scratching at the door,
+With their powerful legs they chase you till death,
+And a powerful jaw crushes your bones,
+Fangs and claws drip with a poison touch,
+They hide in the darkness,
+Ebony fur As black as night,
+Eyes as Red as fire,
+They lurk the streets,
+And stalk the lost But then dawn breaks and the sun rises,
+They bound to the distance, back to the caves.
+
+Poem by Ben
+--SEPARATOR--
+Hellhounds look like a well-built canine with rusty red fur. Hellhounds have glowing red eyes which a used to terrify prey their eyes also enable them to see in the dark. Hellhounds stand about four and a half foot in height. Hellhounds have the ability to breathe fire. They are very aggressive beasts and are expert hunters. They have a pack structure like wolves.
+--SEPARATOR--
+Howlers are four legged dog like creatures. They are furless but have tough skin. They are equipped with long claws, and are covered in hard spikes along its back. They live in dark places and are often associated with evil. They are intelligent creatures that haunt in packs like wolves. There tactic for taking down prey is to charge it as a pack then back away then charge it again, they do this until the prey has succumbed.
+--SEPARATOR--
+Jack Frost is an elfish creature who personifies crisp, cold weather. Jack is said to leave patterns in the autumn leaves and the patterns in the frost that are left on windows. It is thought that Jack Frost comes from Norse mythology as Jokul who was the cause of icicles.
+--SEPARATOR--
+Lamia's are a cross between an attractive human and a lion. The body part is lion, with a human torso rising from where the lions head would be, they look not unlike centaurs but lion instead of horse. Lamias are evil and cruel creatures that get pleasure in causing pain and suffering. Lamias have the magical ability to drain knowledge from those that they attack.
+--SEPARATOR--
+A Leprechaun is a small sprite that lives in farmhouses or wine cellars. Like Brownies they aid humans and accomplish small labours for them, they ask humans for supplies and furniture in return they give objects that bring luck and fortune. Leprechauns are described as merry little fellows that dress in old-fashioned green clothes with buckled shoes and they wear a red cap. They are known as fairy cobblers, as they make shoes for other elves. They never make a pair of shoes they only make one. Popular belief is that a leprechaun possesses a treasure, which a human can obtain if they succeed in capturing one, which is very difficult. Leprechauns are mainly found in Irish folklore but do appear in other countries.
+--SEPARATOR--
+Pixies or Piskies are small people who live on the dons and moors of Cornwall and Devon in the South of England. According to the myths Pixies were originally druids who resisted Christianity that they resisted the influences and the Pixies grew smaller. Another myth tells us they were a race of people who were not good enough to go to heaven but nor were they bad enough to go to Hell and were doomed to walk the earth forever. Pixies are known to steal horses and make nocturnal trips on them over the moors. Pixies like to trick humans such as throwing objects around the house. Pixies are hardworking; they work in the fields the entire night to earn some food.
+--SEPARATOR--
+Snotlings are the smallest of the greened skinned races. Orcs & Goblins use them as slaves for simple tasks as they are not intelligent creatures at about the same size of a Gnome. They are not a threat on their own to other creatures. They realise this so they form gangs that again can attack a target with numbers.
+--SEPARATOR--
+The Vegetable lamb or otherwise known as Tartary, Barbary Lamb or Barmotez comes from Hebrew legends. The Vegetable lamb is a lamb like creature that is grown from a tree, it remains attached to the tree and they eat the foliage in the area around the tree once there is no food left they die of starvation. The Vegetable lamb is a delicacy the meat from it is meant to taste of fish, its blood that is like honey. Its bones were used in rituals to give humans the power of foresight (predicting the future).
+--SEPARATOR--
+The Ahuizotl originates from Central America; it is a creature that is half human, half monkey, with a hand at the end of its tail. The Ahuizotl was actually a creature that lived in the water; it snatched people which approached close to the water's edge, or sometimes attacked fishermen on their boats. The Ahuizotl was a much-feared creature due to its fondness of eating human flesh.
+--SEPARATOR--
+The Basilisk is the mythical king of the serpents. The basilisk is born from a spherical yolkless egg, which was laid by a seven-year rooster and hatched by a toad. According to the legends there are two species of Basilisk. The first is a creature that burns everything it approaches the second kind can kill every living thing with a mere glance. Both species are so evil that their breath wilts vegetation and can crack stones.
+
+A Basilisk is a highly poisonous creature. It was though the only way to kill a Basilisk is by holding a mirror In front of its eyes, the moment the creature sees itself in the mirror it dies of sheer fright. The basilisk had natural enemies the weasel was immune to the Basilisks glance, and if the Basilisk should here a cock crow it would be killed instantly.
+--SEPARATOR--
+The Bayard is a talking horse; it was given to the four sons of Aymon by Charlemagne. The Horse had the ability to elongate its back to accommodate all four of the sons; the Bayard also had great speed.
+--SEPARATOR--
+The Bicorne is a mythical creature is part Panther part cow that is fat from overeating. The Bicorne feeds on virtuous husbands. From the creatures description it is saying there are lots of men that a virtuous because this creature is well fed. The Bicorne's counterpart the Chichevache is thin this feeds on virtuous wives suggesting that the women were not very faithful to their husbands.
+--SEPARATOR--
+The Bunyip is a mythical beast from Australia. It originates from the Aboriginal culture. It is a vicious amphibious creature; it is said to have the appearance of a large seal. IT is greatly feared as it preys on humans, particularly the more tender flesh of women and children.
+--SEPARATOR--
+The Camelopard is the spotted offspring of the mother of a camel and father of a leopard. It is though now that this creature the Camelopard is in fact the Giraffe.
+--SEPARATOR--
+The Catoblepas or Catoblepe originates from Ethiopia, it is said to live near the spring that is the source of the River Nile. Its name comes from Greek which translates to "that which looks downwards". The Catoblepas is four legged beast with a bull like appearance, it has a long mane that falls across its head, and the beast's body is covered in hard scales. The Catoblepas seems a lethargic creature standing around grazing but any living creature that meets its eyes falls dead on the spot. It is said to have a poisonous breath that it can expel which causes loss of sight, voice and leads to convulsions and eventually death.
+--SEPARATOR--
+The Centicore is a four legged beast with two long straight horns that are extremely sharp. It uses these horns much like spears. The Centicore is horse like in appearance but with a chest of a lion and ears that grow in its mouth. The Centicore horns can move and usually has only one facing forward the other is laid across its back.
+--SEPARATOR--
+The Chichevache is a cow with human face, this mythical beast is thin, and this creature fed on obedient and faithful wives. The Chichevache is thin this feeds on virtuous wives suggesting that the women were not very faithful to their husbands The counterpart Bicorne is fat which it is saying there are lots of men that a virtuous because this creature is well fed.
+--SEPARATOR--
+A Girallon is a close relation to the gorilla they are large creatures that are highly aggressive and territorial. The main difference between a Girallon and a gorilla is that the Girallon has four arms, to the gorillas two. Girallons are white or pale grey in colour, about 8ft in height. Girallons attack anything that enter territory, they tend to live in small packs, with a dominant male been the packs leader.
+--SEPARATOR--
+The Humbata comes from Sumerian Epic, where Gilgamesh encounters this creature. The Humbata has the head and horns of a bull and the paws and the body of a lion and the talons of a vulture.
+--SEPARATOR--
+The Icthyocentaur has the torso, arms and head of a human, the tail of a dolphin and the forelegs of a horse or lion. This is the aquatic version of a centaur.
+--SEPARATOR--
+The Karakadon is a bull like creature that has a single curved horn growing from its forehead, much like that of a unicorn. The creature strikes terror into other creatures with its breath and thunderous bellow.
+--SEPARATOR--
+Minotaur's are powerful beasts that are half bull, half-human. They are savage creatures, which can use simple weapons such as clubs and large axes. They stand around 8 foot in height their furs are often dark brown in colour but can be black and lighter browns. They are often said to have large bellows. The origin of the Minotaur lie within Greek mythology where a Minotaur was kept in a labyrinth at Knossos until Thesus killed it. They have been featured in many fantasy novels such as the Chronicles of Narnia and also appear in the fantasy role-play games.
+--SEPARATOR--
+Ogres stand twice as tall as a human and are strongly built with large muscles. Ogres are not highly intelligent creatures but they are not stupid and nor are they evil creatures. They tend to rely on brawn rather than brains. Ogres are good fighters and other races try to recruit them as mercenaries for armies. Ogres don't really care for whom they fight for as long as the moneys good and that they get a good fight.
+--SEPARATOR--
+The Pegasus comes from Greek mythology it is a winged horse. The Pegasus classically has a white hide. Poseidon with Medusa brought Pegasus forth, when Persus cut off Medusa head the Pegasus flew out. Pegasus's are known to be stubborn but intelligent creatures.
+--SEPARATOR--
+The Sasquatch is better known as Bigfoot, Sasquatch comes from North America. The meaning of the name which comes from the native American word that means "Hairy Man". The Sasquatch is a large ape like creature that is thought to measure up to about 7 foot in height. It has long ape like arms with a flattened nose and is covered in a thick matt of hair. It is though that these creatures live in caves in the vast forests of North America. The first reported sighting for the beast was in 1811, since then there have been many hundreds of sightings. The Native Americans had seen this beast before that date. Many expeditions had set out to search for Sasquatch but has never found evidence for its existence.
+--SEPARATOR--
+Trolls are monsters that come from Scandinavian myths and Nordic fantasy. Trolls are called Trows in myths from the Shetlands. Trolls are not especially intelligent and are often associated with an element. The most common description of Trolls is a tough ugly creature with tough rock like skin. Trolls have the powers to regenerate even if hacked apart. The only way to stop a Troll regenerating is burning the monster. Trolls diet is unusual, as they will eat anything including metal, bone, wood and rocks. The stomachs of trolls contain very powerful digestive acids. This has led Trolls to an unpleasant form of attack of vomiting over their target; this is an extremely painful attack. Sometimes Trolls use basic hand weapons such as clubs or large stones.
+--SEPARATOR--
+A Unicorn is not unlike a war-horse; it has heavy cloven hooves, the main difference in appearance is that it has a single central horn on its forehead. Unicorns are intelligent creatures. Unicorns like to use their horn as a lance when charging the enemy. Unicorns are a magical creature, which means they can dispel magic that is targeted at them. The horn of a Unicorn is said to have great healing powers. They are often seen to be pure white or grey in colour, though have been known to show the same variation as normal horses. In some mythology unicorns can only be tamed by females that are pure of heart.
+--SEPARATOR--
+The Yale is a four legged beast from Ethiopia and India its colour is a tawny brown or black. It is about the size of a horse; it looks much like a deer but has the lower jaws of a boar with its tusks. The Yale has movable horns that it can control.
+
+The Yale can move a single horn forward to use as in a lance like fashion, the other horn moves out of the way to protect it. If one horn is damaged in a fight it moves the other horn in to place to resume the attack.
+
+The Yale is seen in British heraldry it is ones of the Queens beasts and is featured on the arms of Christ's college, Cambridge University.
+--SEPARATOR--
+Anima is a term from Jungian psychology referring to a male's collective unconscious which represents the feminine aspect. Incarnations of this include the males own mother. An example of this is seen in Final Fantasy X where an Aeon (summoned creature) called Anima which represents the spirits of Seymour's mother whom had become a Fayth. If the creature is representing females male side is called Animus.
+--SEPARATOR--
+The Chimera originates from Greek mythology, and is characterised as a creature with the head of a lion, body of a she-goat and the tail of a dragon. The Chimera can also depicted, although not in the Greek mythology to have three heads one of that of a lion, a ram and a dragon. The Chimera occasionally is depicted as a creature with wings that are of a great eagle. In Greek mythology the Chimera is the child of Typhon and Echidna.
+--SEPARATOR--
+The Cockatrice is a snake like creature, which has a pair of great wings that are seen to come from that of a great eagle or that are leathery wings like a dragons. Characteristics of a Cockatrice are that it has glowing red eyes with black pupils. Cockatrice has a magical gaze that it can petrify an attacker to stone.
+--SEPARATOR--
+The Cyclops is from Greek mythology and was originally storm gods Brontes (Thunder), Sterops (Lightening), and Arges (Thunderbolt). They are noted for only having one eye in the centre of their foreheads and been giants many times the height of man. They were great smiths and helped the Greeks. It was said that they fashioned Zeus's lightning bolt. Eventually the Cyclops was seen as a cannibalistic and brutish they were feared and shunned.
+--SEPARATOR--
+Giants are oversized humanoids that are featured in much folklore, and appear in such mythology as Norse, and Greek. Giants are often seen as earth's elder race. A Giant's characteristics traditionally make then to be brutish and hostile. They are seen to feed on cattle, sheep and human or anything living that is smaller in size. Giants normally are seen to be solitary although the Greek titans formed communities. A well-known giant is Blunderbore who lived in a huge castle in a cloud; he was defeated by a boy called Jack (from the fairy tale Jack & the beanstalk).
+--SEPARATOR--
+Stories of giant ants that range in size from fox size to the size of elephants. The stories were reported to have come from India. The Giant Ants guard earth and the gold they dig out of it. Giant ants can detect the approach of inbound creatures and they swarm out of there burrows they catch and devour any intruders. It is thought though today that these beasts were not but Marmots which translate from Persian to mountain ant.
+--SEPARATOR--
+Giant crabs are seen as large monsters in far of countries they in fact are oversized crabs. Although the origins for these giant crabs may not be so mythological or made up as crabs have a rare ability is that they don't really die from old age if protected and in the right environment they can keep growing. Large crabs have been found that are over a metre in width (which in its self is huge compared to ones you see on the beach about 2-8cms in width). In times when we humans didn't pollute the sea as much and these crabs had all the time to grow, you never know.
+--SEPARATOR--
+Giant scorpions are oversized scorpions in varying proportions. The largest scorpions today are about 25cm in length that is big for an arachnid. These oversized creatures are a terrifying, with their two big pincers and a deadly stinging tail and their ability to scurry up walls. These creatures would never have existed because Insects and Arachnids are limited to size by their biology and how they respire.
+--SEPARATOR--
+Giant spiders are an oversized variant of the common spiders that we all know about, spiders have always had that terror value as people have seen them to be creepy and that they produce large webs to catch their prey. Anything that gets caught in the sticky web the spider comes and grabs them and drains them of their fluid. Spider characteristics are that they have eight legs, they can walk on most surfaces including upside down on the ceiling. They produce webs and that sometimes have a poisonous bite. Oh and spiders can jump.
+--SEPARATOR--
+The Griffin or Griffon is a legendary creature, that's head, beak and wings come from that of a great eagle, the body is that of a lion or a tiger and on occasion it has a tail of a scorpion the front feet often in talons. A Griffin is a fierce and deadly creature tearing its prey apart with its huge beak while pinning its prey to the ground with one of its talons or paws. The origin of the Griffin is unknown but its though to be somewhere in the Middle East as this is where early paintings and sculptures by the ancient Assyrians, Babylonians and Persians. In more recent times the Griffin is often seen on churches as gargoyles and on shield heraldry. The Griffon is the symbol shown on Vauxhall cars in Great Britain.
+--SEPARATOR--
+The Hippocerf is a mythical creature, which is half horse and half deer. This creature represents indecision.
+--SEPARATOR--
+The Hippogrif is legendary animal that is half Griffin and half horse. The forequarters and head is that of the griffin that would be the father's half and hindquarters would be of its mother a filly. The Hippogrif is a ferocious creature. The Hippogrif is often found in ancient Greek paintings but it was more largely seen in medieval times.
+--SEPARATOR--
+A Lammasu has the body of a lion the wings of a giant eagle and the face of a human. Lammasu's are said to be noble creatures that look after those that are good. They prey on those creatures that are evil. They are strong creatures they can easily take down larger creatures with its lions claws.
+--SEPARATOR--
+The Manticore is a vicious creature which comes from Asia. The Manticore is a lion like creature that's head has some human likeness; it has a tail which is sometimes depicted as a tail of a scorpion or a tail that fires poisonous darts. The Manticore sometimes is depicted having leathery wings. In Asia where the creature stalks through the forests in search of humans, it attacks first by using poison to render the victim immobile. After been immobilized the victim then can be devoured; bones, clothes, possessions the lot, so the Manticore leaves no trace. There are other descriptions of Manticore that don't fit the above patterns, Manticore's can move around in prides and they kill for the sake of killing and leave there prey behind to be found.
+--SEPARATOR--
+The Sphinx is seen in two cultures, the Egyptians used the Sphinx as a statue, which had the body of the lion and the head of a human, and sometimes the sphinx had wings. Sphinxes were built to honour their kings who the head was normally a portrait of. The Greeks who visited Egypt much later gave the name Sphinx to these structures. The Greek sphinx was a creature of death and destruction and bad luck. The Sphinx was portrayed as a female creature as a winged lion with a feminine head, and sometimes a snake tail. The Sphinx was the offspring of Typhon and Echidna. The Sphinx is sometimes seen as the guardian of temple entrances.
+--SEPARATOR--
+A Taurus is a ferocious creature that is made up of a bull that has wings, they tend to be brightly coloured. A Taurus appears in the zodiac star signs though there it does not feature wings.
+--SEPARATOR--
+Titans are huge human creatures that are about twenty five foot in height. They are powerful creatures that are intelligent. They dress in flowing clothing. Titans have there own cultures and gods. Titans originate from Greek mythology.
+--SEPARATOR--
+Black Orcs are physically biggest and strongest of the related green skins species their skin is much darker green in skin tones than that of Orcs, Goblins and Hobgoblins. Black Orcs are more intelligent as well are often seen to be leading the Orc tribes, they tend to be better armed than other Orcs so they wear heavier armour and carry around larger weapons.
+
+Black Orcs tend to be much better organised and infighting within a tribe does not succumb to petty squabbles. Other Orcs tend not to squabble when black Orcs are around either because they don't want to be picked on by their bigger cousins.
+--SEPARATOR--
+Centaurs come from Greek mythology and were said to have inhabited the region of Magnesia and Mount Pelion in Thessaly, the Foloi oak forest in Elis, and the Malean peninsula in southern Laconia. These creatures are part human part horse. They have the torso and head of a human with the body of a horse. Centaurs followed Dionysus the wine god that is why centaurs are known for drunkenness and carrying off young maidens. Though female centaurs, called centaurides or centauresses, are not mentioned in early Greek literature and art, they do appear occasionally in later antiquity.
+--SEPARATOR--
+The Dark Elves or Drows are the same as there high elf cousins a tall noble looking race with pale skin and are slender but unlike High Elves which are masters of light magic's, dark elves practice dark sorcery, and have become very adept at it. Dark Elves live in great cities made of dark stone. Dark Elves are constantly at war with their High Elf kin. They use masterly crafted weapons but they carry repeating crossbows, which seem to be unique to them. Dark Elf females that follow the path of sorcery are called Witch Elves and are a warrior sisterhood. Dark Elves always seem to where dark colours such as blacks, deep purples and dark blues.
+--SEPARATOR--
+Dryads come from Greek Mythology and are female spirits of nature, which preside over the forests. A dryad is born with a certain tree species and a particular tree, which she watches over. If the tree is destroyed then the Dryad perishes alongside the tree. Dryads punish mortals that somehow damage the trees.
+--SEPARATOR--
+Dwarves are a short, Muscular race, that the males and females look alike as both have beards, and in dwarfs beards mean a lot. Dwarves live deep beneath the mountains and have mined themselves vast strongholds beneath theses mountain peaks. Dwarves are immensely strong and resilient they have broad hands and feet. Dwarves are known to be stubborn and un-forgetting. Dwarves respect the following things; Age, Wealth and skill.
+
+Dwarves favour axes as a weapon but compared with other races dwarfs have embraced technology and use a variety of weapons that are more advanced than others of the ancient races. Dwarves will use pistols, cannons, flame cannons. Dwarfs like the Elves take pride in their work and their axes show immense craftsmanship. Dwarves have a hate of Goblins & Orcs who have raided there strong holds many a time and took away their wealth. Dwarves also distrust Elves due to an ancient war that was fought between them.
+--SEPARATOR--
+Fairies come in all sorts of guises some look like Goblins, a typical fairy is a tiny human like creature that is 12 inches high, and has delicate wings on its back. These creatures tend to be friendly with humans but humans rarely ever see them. Fairies tend to have specific purposes like the tooth fairy that goes round collecting teeth that are placed under pillows and swaps them for some money. Sometimes fairies are guides to lost people.
+--SEPARATOR--
+The Genie comes from Arabian folklore and is a supernatural fiery creature. They can be both good and evil creatures; evil ones are said lead humans astray. In popular western culture Genies are often seen as been concealed with old lamps, which when rubbed genie appears out of them, the reason given is that they have been trapped inside the lamp by an evil sorcerer. This description comes from the western translation of "The Book of One Thousand and One Nights". Genies often grant wishes as well to the person that frees them from the lamp, commonly this is three wishes that the Genie can grant.
+--SEPARATOR--
+Gnomes are a very small humanoid race, which are not physically different to humans other than they are only about 1 to 2 feet high. Gnomes are classically seen as protectors of the natural world, often they are portrayed as little men with beards who tend to the garden. Gnomes are also seen as adventures wanting to discover the world they often join parties of adventuring heroes. Gnomes can prove useful due to their height and sometimes turn to a life of crime.
+--SEPARATOR--
+Goblins vary in size but are smaller than Orcs or Hobgoblins. They are a green skinned race, which have pointy ears. Goblins have sharp pointy teeth. They look thin and scrawny. Goblins sometimes are seen as more intelligent than their cousin Orcs, but are often bullied by their bigger and tougher relations. Goblins form tribes, in which the biggest and hardest goblin is the leader. Goblins are often seen in company of Orc tribes. Goblins use a varying array of weapons well anything they can lay their hands on. Goblins are seen to ride wolves into battle, in which case these tend to be the vanguard of an Orc & Goblin army.
+--SEPARATOR--
+Elves are a tall, slim and regal built race. They are seen as a noble race, and tend to be beautiful or handsome in appearance. Elves are a paled skinned race. Elves are strong and agile in comparison with humans and are often seen as more intelligent and wiser. Elves have a longer life span than humans do. Elves build refined weapons that are seen as master craftsmanship by other races. They use Swords, bows and lances mainly they don't like using crossbows or gunpowder weapons.
+--SEPARATOR--
+Hobgoblins are close relations to Orcs and Goblins. They are distinctly taller than Goblins but not as muscular as Orcs. Hobgoblins are renowned for been cowardly and sneaky. They have green skin like their cousins and large k9 teeth with hooked noses and pointy ears. Hobgoblins use a varying array of weapons and like goblins they ride large wolves into battle.
+--SEPARATOR--
+Merfolk are a mythical race which have grown from sailors tales, told across the world. Merfolk are basically human but have a fish's tail, they are able to breath underwater. Merfolk are said to be beautiful and that many Mermaids (female Merfolk) have fallen in love with sailors that they have rescued. Merfolk tend to carry Tridents (three headed spears) as weapons. They are said to be able command creatures in the sea, and ride upon seahorses.
+--SEPARATOR--
+Nymphs are creatures that inhabit the most secluded and peaceful areas of the world, close to a pure water source e.g. a spring. Nymphs are a peaceful race they hate evil and ugliness. They try to avoid conflict and they flee if challenged. When the situation is desperate a nymph will defend themselves by using their abilities to blind or confuse there attacker using there magical powers. Nymphs look a lot like humans apart from they have pointy ears like elves, they are nature's embodiment of physical beauty. Nymphs are ever young they are charming graceful and intelligent.
+--SEPARATOR--
+Orc's are green skinned creatures with large sharp K9 teeth and pointy ears. They are taller than humans are but Orc's are broader and more muscular. Orc's have larger heads than humans but have thicker skulls and they have smaller brains. Another biological feature of Orc's is their ability to suffer large wounds that wound almost certainly kill humans or other races. They seem to possess a remarkable regenerative & immune system. Orc's are an aggressive race that forms tribes or war bands. Sometimes these war bands rally together to form mighty armies. Orc's are savage fighters and preferred tactics are to attack in large hordes. Orc's use varying weapons to fight with depending on the intelligence within a particular tribe.
+
+Savage Orc's which is a tribe that inhabit forests are poorer developed and tend to use clubs and simple weapons to attack with and do not use armour. A more advanced tribe would be equipped forged weapons and armour. Orc's use an unusual mount to get them around quickly they use giant ferocious Boars. These make good attacking cavalry.
+--SEPARATOR--
+The Ratmen are mutated race that originate from rats, which fed upon some source of mighty magical power. They are the same height approximately as goblins. Ratmen are sneaky and rely on stealth to attack using underground tunnels. Ratmen are quick, they can be carriers for all sorts of diseases which the ratmen themselves seem to be immune to.
+--SEPARATOR--
+A Satyr or Faun are intelligent creatures that are found in the wild places of the world it is sad. They indulge in food and drink and romance. A satyr is a horned man with the legs and feet of a goat. The hair that grows on these creatures is usually chestnut brown. The hooves tend to be jet black in colour. Satyrs are mischievous by nature and like to play tricks on others. They have a natural talent for music; they often carry Pan Pipes to play their magical tunes on them.
+--SEPARATOR--
+Troglodytes are humanoid reptilian creatures they have basic intelligence. They stand about 5ft in height, have a basic humanoid shape, they have three fingers and a thumb. The skin is tough leathery and they have lizard like tail and talons on their feet. Their skin pigment tones have the ability to change colour to match their surroundings much like that of a chameleon.
+
+Troglodytes are a feudal race having warring tribes. They have very basic technology; they construct crude weapons such as flint axes and javelins. They are ferocious creatures that hunt down other creatures for meat. They were said to raid human settlements to capture the young for food.
+--SEPARATOR--
+Wood Elves are similar to High Elves and Drow they are a slender noble race with pale skin and often pointed ears. Wood elves live deep with the forests and are guardians of the natural world. Wood elves work with the forest they try not to damage the forest like other races do. Wood elves are extremely agile, there main method of attack is the bow, but they will use hand-to-hand fighting. Wood elves like to use stealth as there are fewer of them compared to the High Elves and Dark Elves. Wood Elves prefer not to wear armour this allows them to move more rapidly through forests though they tend to wear fabrics in greens and browns to match in with their surroundings, this helps them to move around unnoticed.
+--SEPARATOR--
+Alan are Half Human, Half bird like creatures that originate from the Philippines. They live within the forests and spend much of there time hanging upside down from trees. This is due to that they have the fingers on feet and hands have toes on the tips. They are mischievous in nature but are friendly to humans and some legends refer to them bringing up children that have lost their parents.
+--SEPARATOR--
+The Anka is a gigantic Arabian bird that can live over 1700 years in age. The wingspan of the Anka is said to be the breadth of five elephants.
+--SEPARATOR--
+The Bar Juchne is another enormous bird with a huge wingspan. This bird is similar to the Roc. This bird is mentioned in Hebrew texts and one incident is written about when an egg fell from its mountain nest, it felled 300 trees and caused flooding in 600 villages.
+--SEPARATOR--
+The Benu comes from ancient Egyptian mythology. It is a bird that was born at creation and was worshiped at the city of Heliopolis. It is a heron like bird with a crest of two long feathers that come from the rear of its head. The Benu is reborn each day after its journey through the night. The Benu is associated with the gods of the sun; it accompanies the souls of the dead on the boat of Ra on its journey through the underworld.
+
+The Benu was the first creature to come from the sea at creation and the first piece of land it stood on a temple was built this is said to be Heliopolis.
+
+It is thought that the Benu bird is much like that of the Phoenix. The Greek Historian Herodotus visited Egypt and described the bird that the priests showed him as the western bird Phoenix.
+--SEPARATOR--
+The Feng-Huang is the Chinese equivalent of the Phoenix. The Feng-Huang lives with the sun. The Bird has the head and comb of a pheasant and the feathers of a peacock. The bird has three legs not two. The plumage blends the five colours and the song of the bird is a harmony of five notes. A male of this bird is the Feng the female is the Huang.
+--SEPARATOR--
+The firebird (Zshar-ptitsa) comes from Russian folklore it is a miraculous bird, its feathers shine silver and gold, its eyes sparkle like crystals. The firebird is a nocturnal bird as at night this bird illuminates the land that it flies over. It is said that a feather from its tail can light up a dark room. The firebird eats golden apples. When the firebird sings it is said that pearls would fall from its beak. The firebird was also able to heal the sick and cure the blind by its chants.
+--SEPARATOR--
+The Garuda or Garida is an ancient bird, which is similar to that of the Roc, it could block out the sun with its wings and could pick up elephants in its talons. The Garuda has a beak like that of an wings of a bird of prey, the head was white in colour and the wings scarlet and the body was a golden yellow colour. It was sometimes called the bird of life. The Garuda was the feared enemy of the Nagas.
+--SEPARATOR--
+Great eagles are fantasy creatures that are normally based on the golden eagle, but there size much greater, with wing spans of six or seven meters across. These birds are often seen as allies of good and are associated with the High Elves and Wood Elves. They act as information carriers and they are sometimes seen as sentient creatures with high intelligence.
+--SEPARATOR--
+Harpies have two descriptions their origins both lay within Greek mythology. The first description is that Harpies are beautiful winged maidens; the second is that they are winged monsters with the head of an ugly old crone that had sharp talons. Both descriptions were seen to do the same thing, which was that they carried people off to the underworld and inflicting punishment upon them and ultimately death.
+--SEPARATOR--
+The Hsigo come from Chinese mythology and are in fact monkeys with bird like wings. They are much like the monkeys seen in the film Wizard of Oz.
+--SEPARATOR--
+The Peryton lived in Atlantis, it is a large flying creature with the antlered head and legs of a dear with the wings and body of bird. The Peryton casts a shadow of a man until it kills one. When an earthquake destroyed Atlantis the Peryton took to the air and were seen flying above the Pillars of Hercules.
+--SEPARATOR--
+The Phoenix is a mythical bird that is associated with the Egyptian sun god Ra and the Greek sun god Apollo. The Phoenix lives in Arabia, near cool oases. The Phoenix each morning bathes in the pool and sings beautiful bird song. There can only be one Phoenix a live at any one time. When the Phoenix feels its old and it's time to die, the Phoenix builds a nest then settles on it then sets it on fire. The Phoenix burns to death in the flames. From theses flames a new Phoenix emerges. The Phoenix symbolises immortality, resurrection and life after death, because of this it's often seen on sarcophagi and tombs.
+--SEPARATOR--
+The Roc is a legendary gigantic bird from Arabian legends. These birds were so big that they could carry off elephants for food. The Roc is featured in various stories of the "Thousand and One Nights" and they have also featured in historical texts of Marco Polo on his travels.
+--SEPARATOR--
+The Simurgh lives on the tip of Mount Alburz in Persia. The Simurgh is an all-knowing bird with the powers of reason and speech. It is said to have had a discussion with King Solomon about Philisphy. The Simurgh is a very large bird its feathers are said to have curative powers. The Simurgh was said to have nested in the tree of knowledge.
+--SEPARATOR--
+Stymphalids are similar in appearance and size to Cranes. They are man eating birds which their beaks and claws are made of brass, that can pierce armour. Their feathers are also made from brass which they can fire at men like arrows.
+
+They are said to have come from the marshes in Arcadia. Hercules as one of his twelve labours was to rod the marshes of Stymphalids he did this by using brass rattles that scared the Stymphalids into the air where Hercules shot then down with arrows. Some of the Stymphalids escaped to the island of the war god Ares, where Jason and his crew encountered them on their way to collect the Golden Fleece. Jason's crew scared them off by banging their swords against their brass shields.
+--SEPARATOR--
+The Thunderbird lives among the clouds in the sky. The Thunderbird is a monstrous sized bird that is so big it can take whales within its talons. When the Thunderbird beats it wings they crackle with lightening and rumble with thunder. Arrows of lightening flash from its eyes. The Thunderbird comes from Native American mythology.
+--SEPARATOR--
+The Ziz is a mythical bird mentioned in the book of Psalms. It is of enormous size much like that of the Roc. It wings are so vast it can block out the sunlight. The Ziz is the protector of small birds and it is said without the Ziz the small birds would have died out many years before. The Ziz is said to be one of the meats served at the humanities last meal.
+--SEPARATOR--
+An Assassin Vine is a semi-mobile plant that collects its own nutrients by capturing other creatures and depositing their remains near their roots. The main vine of a fully-grown plant is about 20ft in length and 5 inches in thickness, smaller vines shoot off from the main vine stem. The assassin vines fruit resembles that of wild grapes, but have tougher skins and are poisonous to human and larger mammals.
+
+The Assassin vine entangles its prey if they happen to wander into the plant. The Assassin vine automatically reacts to constrict around the ensnared animal. The vine will constrict its victim to death.
+--SEPARATOR--
+Treemen are ancient fantasy creatures that protect the forests of old. Treemen are gigantic tree looking like creatures they have trunk like legs and have great branches for arms. They have a tough woody skin. Treemen have a fear of fire as this can do much damage to them and their forest homes. Treemen tend to have hatred towards those that go around chopping down the biggest and oldest of trees.
+--SEPARATOR--
+Black dragons can be found in marshes and underground cave networks. Black dragons are cunning and evil tempered. They usually side with evil. Black dragons are sometimes known as skull dragons due to their deeply set eye sockets. The dragons have black or very dark grey scales that are glossy when young and with age become duller. Black dragon's smell of rotting vegetation and stagnant water.
+
+Black dragons preferred diet is fish and shellfish they do eat other animals but they like to let these animals once killing rest in stagnant ponds first before eating some days later. Black dragons like to collect coins as there treasure. Black dragons preferred attack method is to ambush their target by making use of their surroundings and vegetation as cover. Black dragons have two different breath attacks. The first is an acid breath attack, which is used to dissolve and blind its attacker/prey. The second breath attack is an oily black smoke attack which can be used to choke its attacker/prey.
+
+Dragons are magical creatures and have the ability to cast spells black dragons are able to perform the following spells.
+
+Darkness - this causes the local area to become dark.
+Insect Plague - causes a plague of insects to attack an area.
+Corrupt water - this spell can stagnate water.
+Charm reptiles - This allows the black dragon to control weaker reptiles to do its bidding.
+--SEPARATOR--
+Blue dragons live in temperate and warm desert environments also they can be found underground. Blue dragons are very territorial. They are well adapted for digging into sand and soft soils. Blue dragons have frilled ears and a large single horn on their snout.
+
+Blue dragons vary in colour from dark blue to light blues; their scales are polished. Blue dragon's scales crackle as they built up static energy. Blue dragons like to soar high above the deserts; Blue dragons like to collect gems as treasure they are particularly attracted by sapphires.
+
+They eat red meat, which they usually cook first before eating. Their attack methods involve attacking from the sky and diving on their target or burrowing into the sand and wait for the prey to come to them and attack quickly catching there target unawares. Blue dragon's breath attack has a lightening breath attack.
+
+As dragons are magical creatures the blue dragon is able to cast the following spells.
+
+Create/destroy water
+Sound imitation. Blue dragons are able to cast this to allow them to imitate other creature's sounds or speech.
+Illusionary terrain the dragon is able to create images of terrain that appears to be there but is not.
+--SEPARATOR--
+Bronze dragons live in temperate and warm aquatic climates as well as underground. Bronze dragons are inquisitive creatures they like to observe other creatures acting about their daily routines and business. Bronze dragons do this by using a skill called polymorphying, which allows them to assume the appearance of other living creatures.
+
+Young bronze dragons scales and skin tones are yellow tinged with green, as the dragon grows older its colours deepen and develops into a bronze tone. Bronze dragons are well adapted to swimming and can breathe underwater unhindered. This swimming adaptation is due to their diets as bronze dragons mainly eat marine or freshwater creatures and aquatic plants.
+
+Bronze dragons dislike killing attackers and would rather bribe or force them away magically. Bronze dragons are armed with two types of breath attack, the first been a lightening breath attack which they can electrocute there attacker the second is a gas breath that repulses the attacker away. Bronze dragons are magical creatures and are able to cast the following spells.
+
+Fog cloud - dragon is able to bring down a cloud of fog to confuse and disorientate its attackers.
+Control weather - The dragon is able to control the weather in the local area.
+Copyright
+--SEPARATOR--
+Gold dragons will live in any climate. Gold dragons are associated with the side of good. Gold dragons are seen to be graceful and wise creatures, they dislike injustice and foul play, and often take it upon themselves to put these things right.
+
+Young gold dragons start out with dark yellow scales and skin tones, as the Dragon ages the scales become more golden. Gold dragons are often accompanied by loyal guards, which can be made up of many types of animals, but often giants.
+
+Gold dragons prefer to use spells in combat rather than physical fighting. Gold dragons have two forms of breath attack; the first of these is a fire breath that burns attackers/prey. The second breath attack is a gas that weakens opponents. Gold dragons are highly magical creatures and are able to perform the following spells.
+
+Cloud Kill
+Fireball - a ball of magical fire
+Fire Shield - a wall of fire used to protect the dragon
+Shield - an invisible wall of energy to protect the dragon
+Sleep - the dragon is able to put his enemies asleep
+Stinking cloud - cloud of foul gas
+Slow - reduces the speed of the dragons attackers
+--SEPARATOR--
+The Chinese Dragons are different from those that originate in western culture. The dragon in china is powerful but benevolent ruler of the earth, sky and sea. It is one of the four spiritual animals and Chinese dragons are said to bring good fortune.
+
+Chinese dragons are different in description in classic western dragons they have a horse like head with two horns from behind their ears. They have long whiskers that come away from its muzzle. They have thick scaly hides like that of western dragons. They have four taloned legs but Chinese dragons don't have wings though they are still able to fly. Their bodies are long and snake like.
+
+Chinese dragons play with a ball of light known as the sacred pearl; this is thought to be the source of the dragon's power. There is a place in the sky called the jade tablet; this contains the number of the dragons. Chinese dragons develop over a number of years they begin as small water snakes, and grow through their lives. Chinese dragons have the power to polymorph (change their shape).
+
+Lung is a named Chinese dragon and he is the ruler of the sky. It is said that Lung has the head of a camel, horns of a deer and the ears of an ox, the eyes of a devil, the neck of a snake, the belly of a clam, fish scales and eagle's talons and paws of a tiger. Lungs breath is said to form the clouds.
+
+There are other Chinese dragons that rule the weather, rain, thunder, lightning and the wind. Other Chinese dragons rule the oceans and the earth. A dragon named Chiao is the supreme Dragon of the earth.
+--SEPARATOR--
+The dragons of Joppa are from around the southern and eastern Mediterranean. Two of these dragons are known as the most famous monsters of legend.
+
+The first of these beasts was of the sea the second a creature of foul disease ridden swamps. The first called Cetus was a dragon that the Greek god Poseidon called from the sea to ravage the country of King Cepheus. Waters flooded his lands and Cetus ate his people. King Cepheus consulted the great Oracle and was told the only way to stop the dragon was to sacrifice his daughter Andromeda, to Cetus. As the legend goes the hero Perseus was returning home after slaying the Gorgon Medusa when he saw from afar Andromeda bound to a rock. As the dragon Cetus approached Andromeda, Perseus landed on the dragons back and drove his sword repeatedly into Cetus and slew the dragon, freeing the kingdom and winning himself a wife.
+
+The second dragon at Joppa is from the accounts of English knights on the crusades they told of St George's battle with a dragon. St George was returning home when he came across a maiden in distress. She was the Princess of the kingdom chained to a post in the marshes. When the dragon emerged from the swamp to eat the princess George attacked driving his lance through the dragons opened jaws. Through this act George converted the local villages to Christianity. George became the patron saint of England.
+--SEPARATOR--
+The dragon called the Tararque was from the Rhone Valley in Southern France. This dragon does not display standard dragon features the monster had six legs the head of a lion the paws of a bear and a scaly body with a barbed tail. The creature's body was covered in sharp spikes. The dragon was said to sink ships that navigated the Rhone and killed those that travelled to close to the river banks.
+--SEPARATOR--
+In the west the dragon or known otherwise as the Drakon in Greek and Draco in roman the British refer to it as the Drake. Western dragons have the characteristics that many of the typical fantasy dragons do. They have four taloned feet, a pair of wings that are like that of a bat. Their heads have a crest and a beard underneath their chin. Some have horns or antlers. They have tough scales; their stomachs are like that of crocodiles. Western dragons come in varying sizes from small fox size to huge many times bigger than elephants. They come in a variety of colours and have some kind of breath attack most usually fire. Western dragons have barbed tongues.
+
+It is said that the blood of these western dragons have powerful healing properties and the blood also allows the understanding of other languages. Western dragons have s gem in there head known as the Draconce or Dragon-stone it is a brilliant red it is said to have curative powers.
+--SEPARATOR--
+In northern European countries the Worm (Wyrm or Wurm) is found. It is one of the most ferocious dragons. They live deep beneath the earth, and they guard gems and treasure such as gold. These dragons are huge and usually wingless they are covered in thick hard scales that are said to be like the strength of steel. They often are fire breathers. Worms take vengeance on those that steal from their hordes.
+
+One of the best known worms was the nemesis of the warrior king Beowulf. Beowulf slew the dragon but at the cost of his own life. The legend says that Beowulf was buried with the Worm's treasure.
+--SEPARATOR--

eval_nocaps.py

@@ -0,0 +1,118 @@
+'''
+ * Copyright (c) 2022, salesforce.com, inc.
+ * All rights reserved.
+ * SPDX-License-Identifier: BSD-3-Clause
+ * For full license text, see LICENSE.txt file in the repo root or https://opensource.org/licenses/BSD-3-Clause
+ * By Junnan Li
+'''
+import argparse
+import os
+import ruamel_yaml as yaml
+import numpy as np
+import random
+import time
+import datetime
+import json
+from pathlib import Path
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+import torch.backends.cudnn as cudnn
+import torch.distributed as dist
+from torch.utils.data import DataLoader
+
+from models.blip import blip_decoder
+import utils
+from data import create_dataset, create_sampler, create_loader
+from data.utils import save_result
+
+@torch.no_grad()
+def evaluate(model, data_loader, device, config):
+    # evaluate
+    model.eval() 
+    
+    metric_logger = utils.MetricLogger(delimiter="  ")
+    header = 'Evaluation:'
+    print_freq = 10
+
+    result = []
+    for image, image_id in metric_logger.log_every(data_loader, print_freq, header): 
+        
+        image = image.to(device)       
+        
+        captions = model.generate(image, sample=False, num_beams=config['num_beams'], max_length=config['max_length'], 
+                                  min_length=config['min_length'], repetition_penalty=1.1)
+        
+        for caption, img_id in zip(captions, image_id):
+            result.append({"image_id": img_id.item(), "caption": caption})
+  
+    return result
+
+
+def main(args, config):
+    utils.init_distributed_mode(args)    
+    
+    device = torch.device(args.device)
+
+    # fix the seed for reproducibility
+    seed = args.seed + utils.get_rank()
+    torch.manual_seed(seed)
+    np.random.seed(seed)
+    random.seed(seed)
+    cudnn.benchmark = True
+
+    #### Dataset #### 
+    print("Creating captioning dataset")
+    val_dataset, test_dataset = create_dataset('nocaps', config)  
+
+    if args.distributed:
+        num_tasks = utils.get_world_size()
+        global_rank = utils.get_rank()            
+        samplers = create_sampler([val_dataset,test_dataset], [False,False], num_tasks, global_rank)         
+    else:
+        samplers = [None,None]
+    
+    val_loader, test_loader = create_loader([val_dataset, test_dataset],samplers, 
+                                            batch_size=[config['batch_size']]*2,num_workers=[4,4],
+                                            is_trains=[False, False], collate_fns=[None,None])        
+
+    #### Model #### 
+    print("Creating model")
+    model = blip_decoder(pretrained=config['pretrained'], image_size=config['image_size'], vit=config['vit'], 
+                           prompt=config['prompt'])
+
+    model = model.to(device)   
+    
+    model_without_ddp = model
+    if args.distributed:
+        model = torch.nn.parallel.DistributedDataParallel(model, device_ids=[args.gpu])
+        model_without_ddp = model.module    
+    
+    val_result = evaluate(model_without_ddp, val_loader, device, config)  
+    val_result_file = save_result(val_result, args.result_dir, 'val', remove_duplicate='image_id')   
+    test_result = evaluate(model_without_ddp, test_loader, device, config)  
+    test_result_file = save_result(test_result, args.result_dir, 'test', remove_duplicate='image_id') 
+
+
+if __name__ == '__main__':
+    parser = argparse.ArgumentParser()
+    parser.add_argument('--config', default='./configs/nocaps.yaml')
+    parser.add_argument('--output_dir', default='output/NoCaps')        
+    parser.add_argument('--device', default='cuda')
+    parser.add_argument('--seed', default=42, type=int)
+    parser.add_argument('--world_size', default=1, type=int, help='number of distributed processes')    
+    parser.add_argument('--dist_url', default='env://', help='url used to set up distributed training')
+    parser.add_argument('--distributed', default=True, type=bool)
+    args = parser.parse_args()
+
+    config = yaml.load(open(args.config, 'r'), Loader=yaml.Loader)
+
+    args.result_dir = os.path.join(args.output_dir, 'result')
+
+    Path(args.output_dir).mkdir(parents=True, exist_ok=True)
+    Path(args.result_dir).mkdir(parents=True, exist_ok=True)
+        
+    yaml.dump(config, open(os.path.join(args.output_dir, 'config.yaml'), 'w'))    
+    
+    main(args, config)

eval_retrieval_video.py

@@ -0,0 +1,250 @@
+'''
+ * Copyright (c) 2022, salesforce.com, inc.
+ * All rights reserved.
+ * SPDX-License-Identifier: BSD-3-Clause
+ * For full license text, see LICENSE.txt file in the repo root or https://opensource.org/licenses/BSD-3-Clause
+ * By Junnan Li
+'''
+import argparse
+import os
+import ruamel_yaml as yaml
+import numpy as np
+import random
+import time
+import datetime
+import json
+from pathlib import Path
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+import torch.backends.cudnn as cudnn
+import torch.distributed as dist
+from torch.utils.data import DataLoader
+
+from models.blip_retrieval import blip_retrieval
+import utils
+from data.video_dataset import VideoDataset
+
+
+@torch.no_grad()
+def evaluation(model, data_loader, tokenizer, device, config):
+    # test
+    model.eval() 
+    
+    metric_logger = utils.MetricLogger(delimiter="  ")
+    header = 'Evaluation:'    
+    
+    print('Computing features for evaluation...')
+    start_time = time.time()  
+
+    texts = data_loader.dataset.text   
+    num_text = len(texts)
+    text_bs = 256
+    text_ids = []
+    text_embeds = []  
+    text_atts = []
+    for i in range(0, num_text, text_bs):
+        text = texts[i: min(num_text, i+text_bs)]
+        text_input = tokenizer(text, padding='max_length', truncation=True, max_length=35, return_tensors="pt").to(device) 
+        text_output = model.text_encoder(text_input.input_ids, attention_mask = text_input.attention_mask, mode='text')  
+        text_embed = F.normalize(model.text_proj(text_output.last_hidden_state[:,0,:]))
+        text_embeds.append(text_embed)   
+        text_ids.append(text_input.input_ids)
+        text_atts.append(text_input.attention_mask)
+    
+    text_embeds = torch.cat(text_embeds,dim=0)
+    text_ids = torch.cat(text_ids,dim=0)
+    text_atts = torch.cat(text_atts,dim=0)
+    text_ids[:,0] = tokenizer.additional_special_tokens_ids[0]
+    
+    video_feats = []
+    video_embeds = []
+    for video, video_id in data_loader: 
+
+        B,N,C,W,H = video.size()
+        video = video.view(-1,C,W,H)
+        video = video.to(device,non_blocking=True) 
+        video_feat = model.visual_encoder(video)        
+        video_embed = model.vision_proj(video_feat[:,0,:])   
+        video_embed = video_embed.view(B,N,-1).mean(dim=1)
+        video_embed = F.normalize(video_embed,dim=-1)  
+       
+        video_feat = video_feat.view(B,-1,video_feat.shape[-1])
+        video_feats.append(video_feat.cpu())
+        video_embeds.append(video_embed)
+     
+    video_feats = torch.cat(video_feats,dim=0)
+    video_embeds = torch.cat(video_embeds,dim=0)
+    
+    sims_matrix = video_embeds @ text_embeds.t()
+    score_matrix_v2t = torch.full((len(texts),len(texts)),-100.0).to(device) 
+    
+    num_tasks = utils.get_world_size()
+    rank = utils.get_rank() 
+    step = sims_matrix.size(0)//num_tasks + 1
+    start = rank*step
+    end = min(sims_matrix.size(0),start+step)
+
+    for i,sims in enumerate(metric_logger.log_every(sims_matrix[start:end], 50, header)): 
+        topk_sim, topk_idx = sims.topk(k=config['k_test'], dim=0)
+        
+        encoder_output = video_feats[start+i].repeat(config['k_test'],1,1).to(device,non_blocking=True) 
+        encoder_att = torch.ones(encoder_output.size()[:-1],dtype=torch.long).to(device,non_blocking=True) 
+        output = model.text_encoder(text_ids[topk_idx], 
+                                    attention_mask = text_atts[topk_idx],
+                                    encoder_hidden_states = encoder_output,
+                                    encoder_attention_mask = encoder_att,                             
+                                    return_dict = True,
+                                   )
+        score = model.itm_head(output.last_hidden_state[:,0,:])[:,1]
+        score_matrix_v2t[start+i,topk_idx] = score + topk_sim
+        
+    sims_matrix = sims_matrix.t()
+    score_matrix_t2v = torch.full((len(texts),len(texts)),-100.0).to(device) 
+    
+    step = sims_matrix.size(0)//num_tasks + 1
+    start = rank*step
+    end = min(sims_matrix.size(0),start+step)    
+    
+    for i,sims in enumerate(metric_logger.log_every(sims_matrix[start:end], 50, header)): 
+        
+        topk_sim, topk_idx = sims.topk(k=config['k_test'], dim=0)
+        encoder_output = video_feats[topk_idx].to(device,non_blocking=True) 
+        encoder_att = torch.ones(encoder_output.size()[:-1],dtype=torch.long).to(device,non_blocking=True) 
+        output = model.text_encoder(text_ids[start+i].repeat(config['k_test'],1), 
+                                    attention_mask = text_atts[start+i].repeat(config['k_test'],1),
+                                    encoder_hidden_states = encoder_output,
+                                    encoder_attention_mask = encoder_att,                             
+                                    return_dict = True,
+                                   )
+        score = model.itm_head(output.last_hidden_state[:,0,:])[:,1]
+        score_matrix_t2v[start+i,topk_idx] = score + topk_sim
+
+    if args.distributed:
+        dist.barrier()   
+        torch.distributed.all_reduce(score_matrix_v2t, op=torch.distributed.ReduceOp.SUM) 
+        torch.distributed.all_reduce(score_matrix_t2v, op=torch.distributed.ReduceOp.SUM)        
+        
+    total_time = time.time() - start_time
+    total_time_str = str(datetime.timedelta(seconds=int(total_time)))
+    print('Evaluation time {}'.format(total_time_str)) 
+
+    return score_matrix_v2t.cpu().numpy(), score_matrix_t2v.cpu().numpy()
+
+
+            
+@torch.no_grad()
+def itm_eval(scores_v2t, scores_t2v, txt2vmg, vid2txt):
+    
+    #Video->Text 
+    ranks = np.zeros(scores_v2t.shape[0])
+    for index,score in enumerate(scores_v2t):
+        inds = np.argsort(score)[::-1]
+        ranks[index] = np.where(inds == vid2txt[index])[0][0]
+
+    # Compute metrics
+    tr1 = 100.0 * len(np.where(ranks < 1)[0]) / len(ranks)
+    tr5 = 100.0 * len(np.where(ranks < 5)[0]) / len(ranks)
+    tr10 = 100.0 * len(np.where(ranks < 10)[0]) / len(ranks)
+  
+    #Text->Video 
+    ranks = np.zeros(scores_t2v.shape[0])
+    
+    for index,score in enumerate(scores_t2v):
+        inds = np.argsort(score)[::-1]
+        ranks[index] = np.where(inds == txt2vmg[index])[0][0]
+    
+    mdR = np.median(ranks+1)
+        
+    # Compute metrics
+    vr1 = 100.0 * len(np.where(ranks < 1)[0]) / len(ranks)
+    vr5 = 100.0 * len(np.where(ranks < 5)[0]) / len(ranks)
+    vr10 = 100.0 * len(np.where(ranks < 10)[0]) / len(ranks)        
+
+    tr_mean = (tr1 + tr5 + tr10) / 3
+    vr_mean = (vr1 + vr5 + vr10) / 3
+    r_mean = (tr_mean + vr_mean) / 2
+
+    eval_result =  {'txt_r1': tr1,
+                    'txt_r5': tr5,
+                    'txt_r10': tr10,
+                    'txt_r_mean': tr_mean,
+                    'vid_r1': vr1,
+                    'vid_r5': vr5,
+                    'vid_r10': vr10,
+                    'vid_r_mean': vr_mean,
+                    'vid_mdR': mdR,
+                    'r_mean': r_mean}
+    return eval_result
+
+
+
+
+def main(args, config):
+    utils.init_distributed_mode(args)    
+    
+    device = torch.device(args.device)
+
+    # fix the seed for reproducibility
+    seed = args.seed + utils.get_rank()
+    torch.manual_seed(seed)
+    np.random.seed(seed)
+    random.seed(seed)
+    cudnn.benchmark = True
+
+    #### Dataset #### 
+    print("Creating retrieval dataset")
+    test_dataset = VideoDataset(config['video_root'],config['ann_root'],num_frm=config['num_frm_test'],
+                                max_img_size=config['image_size'], frm_sampling_strategy='uniform')
+
+    test_loader = DataLoader(
+        test_dataset,
+        batch_size=config['batch_size'],
+        num_workers=4,
+        pin_memory=True,
+        drop_last=False,
+        shuffle=False,
+    )  
+
+    #### Model #### 
+    print("Creating model")
+    model = blip_retrieval(pretrained=config['pretrained'], image_size=config['image_size'], vit=config['vit'])
+    
+    model = model.to(device)   
+    
+    model_without_ddp = model
+    if args.distributed:
+        model = torch.nn.parallel.DistributedDataParallel(model, device_ids=[args.gpu])
+        model_without_ddp = model.module   
+    
+    score_v2t, score_t2v, = evaluation(model_without_ddp, test_loader, model_without_ddp.tokenizer, device, config)
+
+    if utils.is_main_process():  
+
+        test_result = itm_eval(score_v2t, score_t2v, test_loader.dataset.txt2video, test_loader.dataset.video2txt)  
+        print(test_result)
+
+        log_stats = {**{f'{k}': v for k, v in test_result.items()},}
+        with open(os.path.join(args.output_dir, "test_result.txt"),"a") as f:
+            f.write(json.dumps(log_stats) + "\n")     
+
+
+if __name__ == '__main__':
+    parser = argparse.ArgumentParser()     
+    parser.add_argument('--config', default='./configs/retrieval_msrvtt.yaml')
+    parser.add_argument('--output_dir', default='output/Retrieval_msrvtt')        
+    parser.add_argument('--device', default='cuda')
+    parser.add_argument('--seed', default=42, type=int)
+    parser.add_argument('--world_size', default=1, type=int, help='number of distributed processes')    
+    parser.add_argument('--dist_url', default='env://', help='url used to set up distributed training')
+    parser.add_argument('--distributed', default=True, type=bool)
+    args = parser.parse_args()
+
+    config = yaml.load(open(args.config, 'r'), Loader=yaml.Loader)
+
+    Path(args.output_dir).mkdir(parents=True, exist_ok=True)
+        
+    yaml.dump(config, open(os.path.join(args.output_dir, 'config.yaml'), 'w'))    
+    
+    main(args, config)

imagecaptioning.py

@@ -0,0 +1,93 @@
+from PIL import Image
+import requests
+import torch
+from torchvision import transforms
+import os
+from torchvision.transforms.functional import InterpolationMode
+import matplotlib.pyplot as plt
+import matplotlib.image as mpimg
+
+device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
+
+from models.blip import blip_decoder
+
+image_size = 384
+transform = transforms.Compose([
+    transforms.Resize((image_size,image_size),interpolation=InterpolationMode.BICUBIC),
+    transforms.ToTensor(),
+    transforms.Normalize((0.48145466, 0.4578275, 0.40821073), (0.26862954, 0.26130258, 0.27577711))
+    ]) 
+
+model_url = 'https://storage.googleapis.com/sfr-vision-language-research/BLIP/models/model_large_caption.pth'
+    
+model = blip_decoder(pretrained=model_url, image_size=384, vit='large')
+model.eval()
+model = model.to(device)
+
+
+from models.blip_vqa import blip_vqa
+
+image_size_vq = 480
+transform_vq = transforms.Compose([
+    transforms.Resize((image_size_vq,image_size_vq),interpolation=InterpolationMode.BICUBIC),
+    transforms.ToTensor(),
+    transforms.Normalize((0.48145466, 0.4578275, 0.40821073), (0.26862954, 0.26130258, 0.27577711))
+    ]) 
+
+model_url_vq = 'https://storage.googleapis.com/sfr-vision-language-research/BLIP/models/model*_vqa.pth'
+    
+model_vq = blip_vqa(pretrained=model_url_vq, image_size=480, vit='base')
+model_vq.eval()
+model_vq = model_vq.to(device)
+
+
+
+def inference(raw_image, model_n, question="", strategy=""):
+    if model_n == 'Image Captioning':
+        image = transform(raw_image).unsqueeze(0).to(device)   
+        with torch.no_grad():
+            if strategy == "Beam search":
+                caption = model.generate(image, sample=False, num_beams=3, max_length=20, min_length=5)
+            else:
+                caption = model.generate(image, sample=True, top_p=0.9, max_length=20, min_length=5)
+            return 'caption: '+caption[0]
+
+    else:   
+        image_vq = transform_vq(raw_image).unsqueeze(0).to(device)  
+        with torch.no_grad():
+            answer = model_vq(image_vq, question, train=False, inference='generate') 
+        return  'answer: '+answer[0]
+
+#get caption for a single iamge
+def get_caption(image_path):
+  img = Image.open(image_path)
+  return inference(img, "Image Captioning")[9:]
+
+def display(image_path):
+  img = mpimg.imread(image_path)
+  img = Image.open(image_path)
+  plt.imshow(img)
+  print("Caption: " + get_caption(image_path))
+  
+#returns a dictionary with key -> img_path and value -> caption
+def get_captions(img_directory, print_status=True):
+    #key is img path, value is the caption 
+    captions = {}
+    length = 0
+    for file in os.listdir(img_directory):
+      length+=1
+    count = 0
+    for file in os.listdir(img_directory):
+        f = os.path.join(img_directory, file)
+        captions[f] = inference(Image.open(f), "Image Captioning")
+        if print_status:
+          print("Images complete:", str(count) + "/" + str(length))
+          print("Caption:", captions[f])
+    return captions
+#writes dictionary to file, key and value seperated by ':'
+def write_to_file(filename, caption_dict):
+  with open(filename, "w") as file:
+    for i in caption_dict:
+      file.write(i + ":" + caption_dict[i])
+  file.close()
+    

predict.py

@@ -0,0 +1,98 @@
+"""
+Download the weights in ./checkpoints beforehand for fast inference
+wget https://storage.googleapis.com/sfr-vision-language-research/BLIP/models/model*_base_caption.pth
+wget https://storage.googleapis.com/sfr-vision-language-research/BLIP/models/model*_vqa.pth
+wget https://storage.googleapis.com/sfr-vision-language-research/BLIP/models/model_base_retrieval_coco.pth
+"""
+
+from pathlib import Path
+
+from PIL import Image
+import torch
+from torchvision import transforms
+from torchvision.transforms.functional import InterpolationMode
+import cog
+
+from models.blip import blip_decoder
+from models.blip_vqa import blip_vqa
+from models.blip_itm import blip_itm
+
+
+class Predictor(cog.Predictor):
+    def setup(self):
+        self.device = "cuda:0"
+
+        self.models = {
+            'image_captioning': blip_decoder(pretrained='checkpoints/model*_base_caption.pth',
+                                             image_size=384, vit='base'),
+            'visual_question_answering': blip_vqa(pretrained='checkpoints/model*_vqa.pth',
+                                                  image_size=480, vit='base'),
+            'image_text_matching': blip_itm(pretrained='checkpoints/model_base_retrieval_coco.pth',
+                                            image_size=384, vit='base')
+        }
+
+    @cog.input(
+        "image",
+        type=Path,
+        help="input image",
+    )
+    @cog.input(
+        "task",
+        type=str,
+        default='image_captioning',
+        options=['image_captioning', 'visual_question_answering', 'image_text_matching'],
+        help="Choose a task.",
+    )
+    @cog.input(
+        "question",
+        type=str,
+        default=None,
+        help="Type question for the input image for visual question answering task.",
+    )
+    @cog.input(
+        "caption",
+        type=str,
+        default=None,
+        help="Type caption for the input image for image text matching task.",
+    )
+    def predict(self, image, task, question, caption):
+        if task == 'visual_question_answering':
+            assert question is not None, 'Please type a question for visual question answering task.'
+        if task == 'image_text_matching':
+            assert caption is not None, 'Please type a caption for mage text matching task.'
+
+        im = load_image(image, image_size=480 if task == 'visual_question_answering' else 384, device=self.device)
+        model = self.models[task]
+        model.eval()
+        model = model.to(self.device)
+
+        if task == 'image_captioning':
+            with torch.no_grad():
+                caption = model.generate(im, sample=False, num_beams=3, max_length=20, min_length=5)
+                return 'Caption: ' + caption[0]
+
+        if task == 'visual_question_answering':
+            with torch.no_grad():
+                answer = model(im, question, train=False, inference='generate')
+                return 'Answer: ' + answer[0]
+
+        # image_text_matching
+        itm_output = model(im, caption, match_head='itm')
+        itm_score = torch.nn.functional.softmax(itm_output, dim=1)[:, 1]
+        itc_score = model(im, caption, match_head='itc')
+        return f'The image and text is matched with a probability of {itm_score.item():.4f}.\n' \
+               f'The image feature and text feature has a cosine similarity of {itc_score.item():.4f}.'
+
+
+def load_image(image, image_size, device):
+    raw_image = Image.open(str(image)).convert('RGB')
+
+    w, h = raw_image.size
+
+    transform = transforms.Compose([
+        transforms.Resize((image_size, image_size), interpolation=InterpolationMode.BICUBIC),
+        transforms.ToTensor(),
+        transforms.Normalize((0.48145466, 0.4578275, 0.40821073), (0.26862954, 0.26130258, 0.27577711))
+    ])
+    image = transform(raw_image).unsqueeze(0).to(device)
+    return image

pretrain.py

@@ -0,0 +1,173 @@
+'''
+ * Copyright (c) 2022, salesforce.com, inc.
+ * All rights reserved.
+ * SPDX-License-Identifier: BSD-3-Clause
+ * For full license text, see LICENSE.txt file in the repo root or https://opensource.org/licenses/BSD-3-Clause
+ * By Junnan Li
+'''
+import argparse
+import os
+import ruamel_yaml as yaml
+import numpy as np
+import random
+import time
+import datetime
+import json
+from pathlib import Path
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+import torch.backends.cudnn as cudnn
+import torch.distributed as dist
+from torch.utils.data import DataLoader
+
+from models.blip_pretrain import blip_pretrain
+import utils
+from utils import warmup_lr_schedule, step_lr_schedule
+from data import create_dataset, create_sampler, create_loader
+
+def train(model, data_loader, optimizer, epoch, device, config):
+    # train
+    model.train()  
+    
+    metric_logger = utils.MetricLogger(delimiter="  ")
+    metric_logger.add_meter('lr', utils.SmoothedValue(window_size=50, fmt='{value:.6f}'))
+    metric_logger.add_meter('loss_ita', utils.SmoothedValue(window_size=50, fmt='{value:.4f}'))
+    metric_logger.add_meter('loss_itm', utils.SmoothedValue(window_size=50, fmt='{value:.4f}'))    
+    metric_logger.add_meter('loss_lm', utils.SmoothedValue(window_size=50, fmt='{value:.4f}'))
+    
+    header = 'Train Epoch: [{}]'.format(epoch)
+    print_freq = 50   
+
+    if config['laion_path']:
+        data_loader.dataset.reload_laion(epoch)
+    
+    data_loader.sampler.set_epoch(epoch)
+
+    for i, (image, caption) in enumerate(metric_logger.log_every(data_loader, print_freq, header)):
+        
+        if epoch==0:
+            warmup_lr_schedule(optimizer, i, config['warmup_steps'], config['warmup_lr'], config['init_lr'])
+            
+        optimizer.zero_grad()
+        
+        image = image.to(device,non_blocking=True)
+        
+        # ramp up alpha in the first 2 epochs
+        alpha = config['alpha']*min(1,(epoch*len(data_loader)+i)/(2*len(data_loader))) 
+
+        loss_ita, loss_itm, loss_lm = model(image, caption, alpha = alpha)  
+        loss = loss_ita + loss_itm + loss_lm  
+
+        loss.backward()
+        optimizer.step()    
+
+        metric_logger.update(loss_ita=loss_ita.item())
+        metric_logger.update(loss_itm=loss_itm.item())
+        metric_logger.update(loss_lm=loss_lm.item())
+        metric_logger.update(lr=optimizer.param_groups[0]["lr"])  
+
+        
+    # gather the stats from all processes
+    metric_logger.synchronize_between_processes()
+    print("Averaged stats:", metric_logger.global_avg())     
+    return {k: "{:.3f}".format(meter.global_avg) for k, meter in metric_logger.meters.items()}  
+
+
+def main(args, config):
+    utils.init_distributed_mode(args)    
+    
+    device = torch.device(args.device)
+
+    # fix the seed for reproducibility
+    seed = args.seed + utils.get_rank()
+    torch.manual_seed(seed)
+    np.random.seed(seed)
+    random.seed(seed)
+    cudnn.benchmark = True
+
+    #### Dataset #### 
+    print("Creating dataset")
+    datasets = [create_dataset('pretrain', config, min_scale=0.2)]
+    print('number of training samples: %d'%len(datasets[0]))
+
+    num_tasks = utils.get_world_size()
+    global_rank = utils.get_rank()            
+    samplers = create_sampler(datasets, [True], num_tasks, global_rank)         
+
+    data_loader = create_loader(datasets,samplers,batch_size=[config['batch_size']], num_workers=[4], is_trains=[True], collate_fns=[None])[0]      
+
+    #### Model #### 
+    print("Creating model")
+    model = blip_pretrain(image_size=config['image_size'], vit=config['vit'], vit_grad_ckpt=config['vit_grad_ckpt'], 
+                            vit_ckpt_layer=config['vit_ckpt_layer'], queue_size=config['queue_size'])
+
+    model = model.to(device)   
+
+    optimizer = torch.optim.AdamW(params=model.parameters(), lr=config['init_lr'], weight_decay=config['weight_decay'])
+    
+    start_epoch = 0
+    if args.checkpoint:    
+        checkpoint = torch.load(args.checkpoint, map_location='cpu') 
+        state_dict = checkpoint['model']    
+        model.load_state_dict(state_dict)    
+        
+        optimizer.load_state_dict(checkpoint['optimizer'])
+        start_epoch = checkpoint['epoch']+1                
+        print('resume checkpoint from %s'%args.checkpoint)    
+    
+    model_without_ddp = model
+    if args.distributed:
+        model = torch.nn.parallel.DistributedDataParallel(model, device_ids=[args.gpu])
+        model_without_ddp = model.module    
+        
+    print("Start training")
+    start_time = time.time()    
+    for epoch in range(start_epoch, config['max_epoch']):
+        
+        step_lr_schedule(optimizer, epoch, config['init_lr'], config['min_lr'], config['lr_decay_rate'])
+                
+        train_stats = train(model, data_loader, optimizer, epoch, device, config) 
+        if utils.is_main_process():  
+            log_stats = {**{f'train_{k}': v for k, v in train_stats.items()},
+                         'epoch': epoch,
+                        }                     
+            save_obj = {
+                'model': model_without_ddp.state_dict(),
+                'optimizer': optimizer.state_dict(),
+                'config': config,
+                'epoch': epoch,
+            }
+            torch.save(save_obj, os.path.join(args.output_dir, 'checkpoint_%02d.pth'%epoch))  
+            
+            with open(os.path.join(args.output_dir, "log.txt"),"a") as f:
+                f.write(json.dumps(log_stats) + "\n")
+
+        dist.barrier()        
+                
+    total_time = time.time() - start_time
+    total_time_str = str(datetime.timedelta(seconds=int(total_time)))
+    print('Training time {}'.format(total_time_str)) 
+
+
+if __name__ == '__main__':
+    parser = argparse.ArgumentParser()
+    parser.add_argument('--config', default='./configs/pretrain.yaml')
+    parser.add_argument('--output_dir', default='output/Pretrain')  
+    parser.add_argument('--checkpoint', default='')    
+    parser.add_argument('--evaluate', action='store_true')    
+    parser.add_argument('--device', default='cuda')
+    parser.add_argument('--seed', default=42, type=int)
+    parser.add_argument('--world_size', default=1, type=int, help='number of distributed processes')    
+    parser.add_argument('--dist_url', default='env://', help='url used to set up distributed training')
+    parser.add_argument('--distributed', default=True, type=bool)
+    args = parser.parse_args()
+
+    config = yaml.load(open(args.config, 'r'), Loader=yaml.Loader)
+
+    Path(args.output_dir).mkdir(parents=True, exist_ok=True)
+        
+    yaml.dump(config, open(os.path.join(args.output_dir, 'config.yaml'), 'w'))    
+    
+    main(args, config)

requirements.txt

@@ -0,0 +1,9 @@
+timm==0.4.12
+transformers==4.15.0
+fairscale==0.4.4
+pycocoevalcap
+PIL
+torch
+torchvision
+cohere
+gradio

train_caption.py

@@ -0,0 +1,206 @@
+'''
+ * Copyright (c) 2022, salesforce.com, inc.
+ * All rights reserved.
+ * SPDX-License-Identifier: BSD-3-Clause
+ * For full license text, see LICENSE.txt file in the repo root or https://opensource.org/licenses/BSD-3-Clause
+ * By Junnan Li
+'''
+import argparse
+import os
+import ruamel_yaml as yaml
+import numpy as np
+import random
+import time
+import datetime
+import json
+from pathlib import Path
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+import torch.backends.cudnn as cudnn
+import torch.distributed as dist
+from torch.utils.data import DataLoader
+
+from models.blip import blip_decoder
+import utils
+from utils import cosine_lr_schedule
+from data import create_dataset, create_sampler, create_loader
+from data.utils import save_result, coco_caption_eval
+
+def train(model, data_loader, optimizer, epoch, device):
+    # train
+    model.train()  
+    
+    metric_logger = utils.MetricLogger(delimiter="  ")
+    metric_logger.add_meter('lr', utils.SmoothedValue(window_size=1, fmt='{value:.6f}'))
+    metric_logger.add_meter('loss', utils.SmoothedValue(window_size=1, fmt='{value:.4f}'))
+    header = 'Train Caption Epoch: [{}]'.format(epoch)
+    print_freq = 50
+
+    for i, (image, caption, _) in enumerate(metric_logger.log_every(data_loader, print_freq, header)):
+        image = image.to(device)       
+        
+        loss = model(image, caption)      
+        
+        optimizer.zero_grad()
+        loss.backward()
+        optimizer.step()    
+        
+        metric_logger.update(loss=loss.item())
+        metric_logger.update(lr=optimizer.param_groups[0]["lr"])
+
+    # gather the stats from all processes
+    metric_logger.synchronize_between_processes()
+    print("Averaged stats:", metric_logger.global_avg())     
+    return {k: "{:.3f}".format(meter.global_avg) for k, meter in metric_logger.meters.items()}  
+
+
+@torch.no_grad()
+def evaluate(model, data_loader, device, config):
+    # evaluate
+    model.eval() 
+    
+    metric_logger = utils.MetricLogger(delimiter="  ")
+    header = 'Caption generation:'
+    print_freq = 10
+
+    result = []
+    for image, image_id in metric_logger.log_every(data_loader, print_freq, header): 
+        
+        image = image.to(device)       
+        
+        captions = model.generate(image, sample=False, num_beams=config['num_beams'], max_length=config['max_length'], 
+                                  min_length=config['min_length'])
+        
+        for caption, img_id in zip(captions, image_id):
+            result.append({"image_id": img_id.item(), "caption": caption})
+  
+    return result
+
+
+def main(args, config):
+    utils.init_distributed_mode(args)    
+    
+    device = torch.device(args.device)
+
+    # fix the seed for reproducibility
+    seed = args.seed + utils.get_rank()
+    torch.manual_seed(seed)
+    np.random.seed(seed)
+    random.seed(seed)
+    cudnn.benchmark = True
+
+    #### Dataset #### 
+    print("Creating captioning dataset")
+    train_dataset, val_dataset, test_dataset = create_dataset('caption_coco', config)  
+
+    if args.distributed:
+        num_tasks = utils.get_world_size()
+        global_rank = utils.get_rank()            
+        samplers = create_sampler([train_dataset,val_dataset,test_dataset], [True,False,False], num_tasks, global_rank)         
+    else:
+        samplers = [None, None, None]
+    
+    train_loader, val_loader, test_loader = create_loader([train_dataset, val_dataset, test_dataset],samplers,
+                                                          batch_size=[config['batch_size']]*3,num_workers=[4,4,4],
+                                                          is_trains=[True, False, False], collate_fns=[None,None,None])         
+
+    #### Model #### 
+    print("Creating model")
+    model = blip_decoder(pretrained=config['pretrained'], image_size=config['image_size'], vit=config['vit'], 
+                           vit_grad_ckpt=config['vit_grad_ckpt'], vit_ckpt_layer=config['vit_ckpt_layer'], 
+                           prompt=config['prompt'])
+
+    model = model.to(device)   
+    
+    model_without_ddp = model
+    if args.distributed:
+        model = torch.nn.parallel.DistributedDataParallel(model, device_ids=[args.gpu])
+        model_without_ddp = model.module    
+    
+    optimizer = torch.optim.AdamW(params=model.parameters(), lr=config['init_lr'], weight_decay=config['weight_decay'])
+            
+    best = 0
+    best_epoch = 0
+
+    print("Start training")
+    start_time = time.time()    
+    for epoch in range(0, config['max_epoch']):
+        if not args.evaluate:        
+            if args.distributed:
+                train_loader.sampler.set_epoch(epoch)
+                
+            cosine_lr_schedule(optimizer, epoch, config['max_epoch'], config['init_lr'], config['min_lr'])
+                
+            train_stats = train(model, train_loader, optimizer, epoch, device) 
+        
+        val_result = evaluate(model_without_ddp, val_loader, device, config)  
+        val_result_file = save_result(val_result, args.result_dir, 'val_epoch%d'%epoch, remove_duplicate='image_id')        
+  
+        test_result = evaluate(model_without_ddp, test_loader, device, config)  
+        test_result_file = save_result(test_result, args.result_dir, 'test_epoch%d'%epoch, remove_duplicate='image_id')  
+
+        if utils.is_main_process():   
+            coco_val = coco_caption_eval(config['coco_gt_root'],val_result_file,'val')
+            coco_test = coco_caption_eval(config['coco_gt_root'],test_result_file,'test')
+            
+            if args.evaluate:            
+                log_stats = {**{f'val_{k}': v for k, v in coco_val.eval.items()},
+                             **{f'test_{k}': v for k, v in coco_test.eval.items()},                       
+                            }
+                with open(os.path.join(args.output_dir, "evaluate.txt"),"a") as f:
+                    f.write(json.dumps(log_stats) + "\n")                   
+            else:             
+                save_obj = {
+                    'model': model_without_ddp.state_dict(),
+                    'optimizer': optimizer.state_dict(),
+                    'config': config,
+                    'epoch': epoch,
+                }
+
+                if coco_val.eval['CIDEr'] + coco_val.eval['Bleu_4'] > best:
+                    best = coco_val.eval['CIDEr'] + coco_val.eval['Bleu_4']
+                    best_epoch = epoch                
+                    torch.save(save_obj, os.path.join(args.output_dir, 'checkpoint_best.pth')) 
+                    
+                log_stats = {**{f'train_{k}': v for k, v in train_stats.items()},
+                             **{f'val_{k}': v for k, v in coco_val.eval.items()},
+                             **{f'test_{k}': v for k, v in coco_test.eval.items()},                       
+                             'epoch': epoch,
+                             'best_epoch': best_epoch,
+                            }
+                with open(os.path.join(args.output_dir, "log.txt"),"a") as f:
+                    f.write(json.dumps(log_stats) + "\n")     
+                    
+        if args.evaluate: 
+            break
+        dist.barrier()     
+
+    total_time = time.time() - start_time
+    total_time_str = str(datetime.timedelta(seconds=int(total_time)))
+    print('Training time {}'.format(total_time_str)) 
+
+
+if __name__ == '__main__':
+    parser = argparse.ArgumentParser()
+    parser.add_argument('--config', default='./configs/caption_coco.yaml')
+    parser.add_argument('--output_dir', default='output/Caption_coco')        
+    parser.add_argument('--evaluate', action='store_true')    
+    parser.add_argument('--device', default='cuda')
+    parser.add_argument('--seed', default=42, type=int)
+    parser.add_argument('--world_size', default=1, type=int, help='number of distributed processes')    
+    parser.add_argument('--dist_url', default='env://', help='url used to set up distributed training')
+    parser.add_argument('--distributed', default=True, type=bool)
+    args = parser.parse_args()
+
+    config = yaml.load(open(args.config, 'r'), Loader=yaml.Loader)
+
+    args.result_dir = os.path.join(args.output_dir, 'result')
+
+    Path(args.output_dir).mkdir(parents=True, exist_ok=True)
+    Path(args.result_dir).mkdir(parents=True, exist_ok=True)
+        
+    yaml.dump(config, open(os.path.join(args.output_dir, 'config.yaml'), 'w'))    
+    
+    main(args, config)

train_nlvr.py

@@ -0,0 +1,213 @@
+'''
+ * Copyright (c) 2022, salesforce.com, inc.
+ * All rights reserved.
+ * SPDX-License-Identifier: BSD-3-Clause
+ * For full license text, see LICENSE.txt file in the repo root or https://opensource.org/licenses/BSD-3-Clause
+ * By Junnan Li
+'''
+import argparse
+import os
+import ruamel_yaml as yaml
+import numpy as np
+import random
+import time
+import datetime
+import json
+from pathlib import Path
+import json
+import pickle
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+from torch.utils.data import DataLoader
+import torch.backends.cudnn as cudnn
+import torch.distributed as dist
+
+from models.blip_nlvr import blip_nlvr
+
+import utils
+from utils import cosine_lr_schedule, warmup_lr_schedule
+from data import create_dataset, create_sampler, create_loader
+
+def train(model, data_loader, optimizer, epoch, device, config):
+    # train
+    model.train()  
+    
+    metric_logger = utils.MetricLogger(delimiter="  ")
+    metric_logger.add_meter('lr', utils.SmoothedValue(window_size=50, fmt='{value:.6f}'))
+    metric_logger.add_meter('loss', utils.SmoothedValue(window_size=50, fmt='{value:.4f}'))
+
+    header = 'Train Epoch: [{}]'.format(epoch)
+    print_freq = 50   
+    step_size = 10
+ 
+    for i,(image0, image1, text, targets) in enumerate(metric_logger.log_every(data_loader, print_freq, header)):
+  
+        images = torch.cat([image0, image1], dim=0)
+        images, targets = images.to(device), targets.to(device)   
+
+        loss = model(images, text, targets=targets, train=True)    
+        
+        optimizer.zero_grad()
+        loss.backward()
+        optimizer.step()    
+               
+        metric_logger.update(lr=optimizer.param_groups[0]["lr"])
+        metric_logger.update(loss=loss.item())  
+        
+    # gather the stats from all processes
+    metric_logger.synchronize_between_processes()
+    print("Averaged stats:", metric_logger.global_avg())     
+    return {k: "{:.4f}".format(meter.global_avg) for k, meter in metric_logger.meters.items()}    
+
+
+@torch.no_grad()
+def evaluate(model, data_loader, device, config):
+    # test
+    model.eval()
+            
+    metric_logger = utils.MetricLogger(delimiter="  ")
+
+    header = 'Evaluation:'
+    print_freq = 50
+
+    for image0, image1, text, targets in metric_logger.log_every(data_loader, print_freq, header):
+        images = torch.cat([image0, image1], dim=0)
+        images, targets = images.to(device), targets.to(device)   
+        
+        prediction = model(images, text, targets=targets, train=False)  
+ 
+        _, pred_class = prediction.max(1)
+        accuracy = (targets==pred_class).sum() / targets.size(0)
+        
+        metric_logger.meters['acc'].update(accuracy.item(), n=image0.size(0))
+
+    # gather the stats from all processes
+    metric_logger.synchronize_between_processes()
+
+    print("Averaged stats:", metric_logger.global_avg())   
+    return {k: "{:.4f}".format(meter.global_avg) for k, meter in metric_logger.meters.items()}
+
+
+        
+def main(args, config):
+    utils.init_distributed_mode(args)    
+    
+    device = torch.device(args.device)
+
+    # fix the seed for reproducibility
+    seed = args.seed + utils.get_rank()
+    torch.manual_seed(seed)
+    np.random.seed(seed)
+    random.seed(seed)
+    cudnn.benchmark = True
+
+    #### Dataset #### 
+    print("Creating dataset")
+    datasets = create_dataset('nlvr', config) 
+    
+    if args.distributed:
+        num_tasks = utils.get_world_size()
+        global_rank = utils.get_rank()            
+        samplers = create_sampler(datasets, [True,False,False], num_tasks, global_rank)
+    else:
+        samplers = [None, None, None]
+    
+    batch_size=[config['batch_size_train'],config['batch_size_test'],config['batch_size_test']]
+    train_loader, val_loader, test_loader = create_loader(datasets,samplers,batch_size=batch_size,
+                                                          num_workers=[4,4,4],is_trains=[True,False,False], 
+                                                          collate_fns=[None,None,None])
+
+    #### Model #### 
+    print("Creating model")
+    model = blip_nlvr(pretrained=config['pretrained'], image_size=config['image_size'], 
+                         vit=config['vit'], vit_grad_ckpt=config['vit_grad_ckpt'], vit_ckpt_layer=config['vit_ckpt_layer'])
+
+    model = model.to(device)   
+    
+    model_without_ddp = model
+    if args.distributed:
+        model = torch.nn.parallel.DistributedDataParallel(model, device_ids=[args.gpu])
+        model_without_ddp = model.module    
+            
+    optimizer = torch.optim.AdamW(params=model.parameters(), lr=config['init_lr'], weight_decay=config['weight_decay'])
+
+    print("Start training")
+    start_time = time.time()
+    best = 0
+    best_epoch = 0
+
+    for epoch in range(0, config['max_epoch']):
+        if not args.evaluate:
+            if args.distributed:
+                train_loader.sampler.set_epoch(epoch)
+                
+            cosine_lr_schedule(optimizer, epoch, config['max_epoch'], config['init_lr'], config['min_lr'])
+            
+            train_stats = train(model, train_loader, optimizer, epoch,  device, config) 
+            
+        val_stats = evaluate(model, val_loader, device, config)
+        test_stats = evaluate(model, test_loader, device, config)  
+        
+        if utils.is_main_process():  
+            if args.evaluate:                
+                log_stats = {**{f'val_{k}': v for k, v in val_stats.items()},
+                             **{f'test_{k}': v for k, v in test_stats.items()},
+                            }
+                with open(os.path.join(args.output_dir, "log.txt"),"a") as f:
+                    f.write(json.dumps(log_stats) + "\n")   
+                
+            else:       
+                log_stats = {**{f'train_{k}': v for k, v in train_stats.items()},
+                             **{f'val_{k}': v for k, v in val_stats.items()},
+                             **{f'test_{k}': v for k, v in test_stats.items()},
+                             'epoch': epoch,
+                            }
+
+                if float(val_stats['acc'])>best:
+                    save_obj = {
+                        'model': model_without_ddp.state_dict(),
+                        'optimizer': optimizer.state_dict(),
+                        'config': config,
+                        'epoch': epoch,
+                    }
+                    torch.save(save_obj, os.path.join(args.output_dir, 'checkpoint_best.pth')) 
+                    best = float(val_stats['acc'])
+                    best_epoch = epoch
+
+                with open(os.path.join(args.output_dir, "log.txt"),"a") as f:
+                    f.write(json.dumps(log_stats) + "\n")
+        if args.evaluate:             
+            break            
+         
+        dist.barrier()   
+    
+    if utils.is_main_process():   
+        with open(os.path.join(args.output_dir, "log.txt"),"a") as f:
+            f.write("best epoch: %d"%best_epoch)      
+            
+    total_time = time.time() - start_time
+    total_time_str = str(datetime.timedelta(seconds=int(total_time)))
+    print('Training time {}'.format(total_time_str)) 
+        
+
+if __name__ == '__main__':
+    parser = argparse.ArgumentParser()
+    parser.add_argument('--config', default='./configs/nlvr.yaml')
+    parser.add_argument('--output_dir', default='output/NLVR')
+    parser.add_argument('--evaluate', action='store_true')      
+    parser.add_argument('--device', default='cuda')
+    parser.add_argument('--seed', default=42, type=int)
+    parser.add_argument('--world_size', default=1, type=int, help='number of distributed processes')    
+    parser.add_argument('--dist_url', default='env://', help='url used to set up distributed training')
+    parser.add_argument('--distributed', default=True, type=bool)
+    args = parser.parse_args()
+
+    config = yaml.load(open(args.config, 'r'), Loader=yaml.Loader)
+
+    Path(args.output_dir).mkdir(parents=True, exist_ok=True)
+        
+    yaml.dump(config, open(os.path.join(args.output_dir, 'config.yaml'), 'w'))    
+    
+    main(args, config)

train_retrieval.py

@@ -0,0 +1,345 @@
+'''
+ * Copyright (c) 2022, salesforce.com, inc.
+ * All rights reserved.
+ * SPDX-License-Identifier: BSD-3-Clause
+ * For full license text, see LICENSE.txt file in the repo root or https://opensource.org/licenses/BSD-3-Clause
+ * By Junnan Li
+'''
+import argparse
+import os
+import ruamel_yaml as yaml
+import numpy as np
+import random
+import time
+import datetime
+import json
+from pathlib import Path
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+import torch.backends.cudnn as cudnn
+import torch.distributed as dist
+from torch.utils.data import DataLoader
+
+from models.blip_retrieval import blip_retrieval
+import utils
+from utils import cosine_lr_schedule
+from data import create_dataset, create_sampler, create_loader
+
+
+def train(model, data_loader, optimizer, epoch, device, config):
+    # train
+    model.train()  
+    
+    metric_logger = utils.MetricLogger(delimiter="  ")
+    metric_logger.add_meter('lr', utils.SmoothedValue(window_size=1, fmt='{value:.6f}'))
+    metric_logger.add_meter('loss_itm', utils.SmoothedValue(window_size=1, fmt='{value:.4f}'))
+    metric_logger.add_meter('loss_ita', utils.SmoothedValue(window_size=1, fmt='{value:.4f}'))
+    header = 'Train Epoch: [{}]'.format(epoch)
+    print_freq = 50
+
+    for i,(image, caption, idx) in enumerate(metric_logger.log_every(data_loader, print_freq, header)):
+        image = image.to(device,non_blocking=True)   
+        idx = idx.to(device,non_blocking=True)   
+       
+        if epoch>0:
+            alpha = config['alpha']
+        else:
+            alpha = config['alpha']*min(1,i/len(data_loader))
+
+        loss_ita, loss_itm = model(image, caption, alpha=alpha, idx=idx)                  
+        loss = loss_ita + loss_itm
+        
+        optimizer.zero_grad()
+        loss.backward()
+        optimizer.step()    
+        
+        metric_logger.update(loss_itm=loss_itm.item())
+        metric_logger.update(loss_ita=loss_ita.item())
+        metric_logger.update(lr=optimizer.param_groups[0]["lr"])
+
+    # gather the stats from all processes
+    metric_logger.synchronize_between_processes()
+    print("Averaged stats:", metric_logger.global_avg())     
+    return {k: "{:.3f}".format(meter.global_avg) for k, meter in metric_logger.meters.items()}  
+
+
+@torch.no_grad()
+def evaluation(model, data_loader, device, config):
+    # test
+    model.eval() 
+    
+    metric_logger = utils.MetricLogger(delimiter="  ")
+    header = 'Evaluation:'    
+    
+    print('Computing features for evaluation...')
+    start_time = time.time()  
+
+    texts = data_loader.dataset.text   
+    num_text = len(texts)
+    text_bs = 256
+    text_ids = []
+    text_embeds = []  
+    text_atts = []
+    for i in range(0, num_text, text_bs):
+        text = texts[i: min(num_text, i+text_bs)]
+        text_input = model.tokenizer(text, padding='max_length', truncation=True, max_length=35, return_tensors="pt").to(device) 
+        text_output = model.text_encoder(text_input.input_ids, attention_mask = text_input.attention_mask, mode='text')  
+        text_embed = F.normalize(model.text_proj(text_output.last_hidden_state[:,0,:]))
+        text_embeds.append(text_embed)   
+        text_ids.append(text_input.input_ids)
+        text_atts.append(text_input.attention_mask)
+    
+    text_embeds = torch.cat(text_embeds,dim=0)
+    text_ids = torch.cat(text_ids,dim=0)
+    text_atts = torch.cat(text_atts,dim=0)
+    text_ids[:,0] = model.tokenizer.enc_token_id
+    
+    image_feats = []
+    image_embeds = []
+    for image, img_id in data_loader: 
+        image = image.to(device) 
+        image_feat = model.visual_encoder(image)   
+        image_embed = model.vision_proj(image_feat[:,0,:])            
+        image_embed = F.normalize(image_embed,dim=-1)      
+        
+        image_feats.append(image_feat.cpu())
+        image_embeds.append(image_embed)
+     
+    image_feats = torch.cat(image_feats,dim=0)
+    image_embeds = torch.cat(image_embeds,dim=0)
+    
+    sims_matrix = image_embeds @ text_embeds.t()
+    score_matrix_i2t = torch.full((len(data_loader.dataset.image),len(texts)),-100.0).to(device)
+    
+    num_tasks = utils.get_world_size()
+    rank = utils.get_rank() 
+    step = sims_matrix.size(0)//num_tasks + 1
+    start = rank*step
+    end = min(sims_matrix.size(0),start+step)
+
+    for i,sims in enumerate(metric_logger.log_every(sims_matrix[start:end], 50, header)): 
+        topk_sim, topk_idx = sims.topk(k=config['k_test'], dim=0)
+
+        encoder_output = image_feats[start+i].repeat(config['k_test'],1,1).to(device)
+        encoder_att = torch.ones(encoder_output.size()[:-1],dtype=torch.long).to(device)
+        output = model.text_encoder(text_ids[topk_idx], 
+                                    attention_mask = text_atts[topk_idx],
+                                    encoder_hidden_states = encoder_output,
+                                    encoder_attention_mask = encoder_att,                             
+                                    return_dict = True,
+                                   )
+        score = model.itm_head(output.last_hidden_state[:,0,:])[:,1]
+        score_matrix_i2t[start+i,topk_idx] = score + topk_sim
+        
+    sims_matrix = sims_matrix.t()
+    score_matrix_t2i = torch.full((len(texts),len(data_loader.dataset.image)),-100.0).to(device)
+    
+    step = sims_matrix.size(0)//num_tasks + 1
+    start = rank*step
+    end = min(sims_matrix.size(0),start+step)    
+    
+    for i,sims in enumerate(metric_logger.log_every(sims_matrix[start:end], 50, header)): 
+        
+        topk_sim, topk_idx = sims.topk(k=config['k_test'], dim=0)
+        encoder_output = image_feats[topk_idx].to(device)
+        encoder_att = torch.ones(encoder_output.size()[:-1],dtype=torch.long).to(device)
+        output = model.text_encoder(text_ids[start+i].repeat(config['k_test'],1), 
+                                    attention_mask = text_atts[start+i].repeat(config['k_test'],1),
+                                    encoder_hidden_states = encoder_output,
+                                    encoder_attention_mask = encoder_att,                             
+                                    return_dict = True,
+                                   )
+        score = model.itm_head(output.last_hidden_state[:,0,:])[:,1]
+        score_matrix_t2i[start+i,topk_idx] = score + topk_sim
+
+    if args.distributed:
+        dist.barrier()   
+        torch.distributed.all_reduce(score_matrix_i2t, op=torch.distributed.ReduceOp.SUM) 
+        torch.distributed.all_reduce(score_matrix_t2i, op=torch.distributed.ReduceOp.SUM)        
+        
+    total_time = time.time() - start_time
+    total_time_str = str(datetime.timedelta(seconds=int(total_time)))
+    print('Evaluation time {}'.format(total_time_str)) 
+
+    return score_matrix_i2t.cpu().numpy(), score_matrix_t2i.cpu().numpy()
+
+
+            
+@torch.no_grad()
+def itm_eval(scores_i2t, scores_t2i, txt2img, img2txt):
+    
+    #Images->Text 
+    ranks = np.zeros(scores_i2t.shape[0])
+    for index,score in enumerate(scores_i2t):
+        inds = np.argsort(score)[::-1]
+        # Score
+        rank = 1e20
+        for i in img2txt[index]:
+            tmp = np.where(inds == i)[0][0]
+            if tmp < rank:
+                rank = tmp
+        ranks[index] = rank
+
+    # Compute metrics
+    tr1 = 100.0 * len(np.where(ranks < 1)[0]) / len(ranks)
+    tr5 = 100.0 * len(np.where(ranks < 5)[0]) / len(ranks)
+    tr10 = 100.0 * len(np.where(ranks < 10)[0]) / len(ranks)
+  
+    #Text->Images 
+    ranks = np.zeros(scores_t2i.shape[0])
+    
+    for index,score in enumerate(scores_t2i):
+        inds = np.argsort(score)[::-1]
+        ranks[index] = np.where(inds == txt2img[index])[0][0]
+
+    # Compute metrics
+    ir1 = 100.0 * len(np.where(ranks < 1)[0]) / len(ranks)
+    ir5 = 100.0 * len(np.where(ranks < 5)[0]) / len(ranks)
+    ir10 = 100.0 * len(np.where(ranks < 10)[0]) / len(ranks)        
+
+    tr_mean = (tr1 + tr5 + tr10) / 3
+    ir_mean = (ir1 + ir5 + ir10) / 3
+    r_mean = (tr_mean + ir_mean) / 2
+
+    eval_result =  {'txt_r1': tr1,
+                    'txt_r5': tr5,
+                    'txt_r10': tr10,
+                    'txt_r_mean': tr_mean,
+                    'img_r1': ir1,
+                    'img_r5': ir5,
+                    'img_r10': ir10,
+                    'img_r_mean': ir_mean,
+                    'r_mean': r_mean}
+    return eval_result
+
+
+def main(args, config):
+    utils.init_distributed_mode(args)    
+    
+    device = torch.device(args.device)
+
+    # fix the seed for reproducibility
+    seed = args.seed + utils.get_rank()
+    torch.manual_seed(seed)
+    np.random.seed(seed)
+    random.seed(seed)
+    cudnn.benchmark = True
+
+    #### Dataset #### 
+    print("Creating retrieval dataset")
+    train_dataset, val_dataset, test_dataset = create_dataset('retrieval_%s'%config['dataset'], config)  
+
+    if args.distributed:
+        num_tasks = utils.get_world_size()
+        global_rank = utils.get_rank()            
+        samplers = create_sampler([train_dataset], [True], num_tasks, global_rank) + [None, None]
+    else:
+        samplers = [None, None, None]
+    
+    train_loader, val_loader, test_loader = create_loader([train_dataset, val_dataset, test_dataset],samplers,
+                                                          batch_size=[config['batch_size_train']]+[config['batch_size_test']]*2,
+                                                          num_workers=[4,4,4],
+                                                          is_trains=[True, False, False], 
+                                                          collate_fns=[None,None,None])   
+   
+
+    #### Model #### 
+    print("Creating model")
+    model = blip_retrieval(pretrained=config['pretrained'], image_size=config['image_size'], vit=config['vit'], 
+                             vit_grad_ckpt=config['vit_grad_ckpt'], vit_ckpt_layer=config['vit_ckpt_layer'], 
+                             queue_size=config['queue_size'], negative_all_rank=config['negative_all_rank'])
+
+    model = model.to(device)   
+    
+    model_without_ddp = model
+    if args.distributed:
+        model = torch.nn.parallel.DistributedDataParallel(model, device_ids=[args.gpu])
+        model_without_ddp = model.module   
+
+    optimizer = torch.optim.AdamW(params=model.parameters(), lr=config['init_lr'], weight_decay=config['weight_decay']) 
+    
+    best = 0
+    best_epoch = 0
+
+    print("Start training")
+    start_time = time.time()    
+
+    for epoch in range(0, config['max_epoch']):    
+        if not args.evaluate:        
+            if args.distributed:
+                train_loader.sampler.set_epoch(epoch)
+                
+            cosine_lr_schedule(optimizer, epoch, config['max_epoch'], config['init_lr'], config['min_lr'])
+            
+            train_stats = train(model, train_loader, optimizer, epoch, device, config)  
+            
+        score_val_i2t, score_val_t2i, = evaluation(model_without_ddp, val_loader, device, config)
+        score_test_i2t, score_test_t2i = evaluation(model_without_ddp, test_loader, device, config)
+    
+        if utils.is_main_process():  
+      
+            val_result = itm_eval(score_val_i2t, score_val_t2i, val_loader.dataset.txt2img, val_loader.dataset.img2txt)  
+            print(val_result)
+                                
+            if val_result['r_mean']>best:
+                save_obj = {
+                    'model': model_without_ddp.state_dict(),
+                    'optimizer': optimizer.state_dict(),
+                    'config': config,
+                    'epoch': epoch,
+                }
+                torch.save(save_obj, os.path.join(args.output_dir, 'checkpoint_best.pth'))  
+                best = val_result['r_mean']        
+                best_epoch = epoch  
+                
+                test_result = itm_eval(score_test_i2t, score_test_t2i, test_loader.dataset.txt2img, test_loader.dataset.img2txt) 
+                print(test_result)
+            
+            if args.evaluate:                
+                log_stats = {**{f'val_{k}': v for k, v in val_result.items()},
+                             **{f'test_{k}': v for k, v in test_result.items()},                  
+                            }
+                with open(os.path.join(args.output_dir, "evaluate.txt"),"a") as f:
+                    f.write(json.dumps(log_stats) + "\n")     
+            else:
+                log_stats = {**{f'train_{k}': v for k, v in train_stats.items()},
+                             **{f'val_{k}': v for k, v in val_result.items()},
+                             **{f'test_{k}': v for k, v in test_result.items()},  
+                             'epoch': epoch,
+                             'best_epoch': best_epoch,
+                            }
+                with open(os.path.join(args.output_dir, "log.txt"),"a") as f:
+                    f.write(json.dumps(log_stats) + "\n")   
+                    
+        if args.evaluate: 
+            break
+
+        dist.barrier()     
+        torch.cuda.empty_cache()
+
+    total_time = time.time() - start_time
+    total_time_str = str(datetime.timedelta(seconds=int(total_time)))
+    print('Training time {}'.format(total_time_str)) 
+
+    
+if __name__ == '__main__':
+    parser = argparse.ArgumentParser()     
+    parser.add_argument('--config', default='./configs/retrieval_flickr.yaml')
+    parser.add_argument('--output_dir', default='output/Retrieval_flickr')        
+    parser.add_argument('--evaluate', action='store_true')
+    parser.add_argument('--device', default='cuda')
+    parser.add_argument('--seed', default=42, type=int)
+    parser.add_argument('--world_size', default=1, type=int, help='number of distributed processes')    
+    parser.add_argument('--dist_url', default='env://', help='url used to set up distributed training')
+    parser.add_argument('--distributed', default=True, type=bool)
+    args = parser.parse_args()
+
+    config = yaml.load(open(args.config, 'r'), Loader=yaml.Loader)
+
+    Path(args.output_dir).mkdir(parents=True, exist_ok=True)
+        
+    yaml.dump(config, open(os.path.join(args.output_dir, 'config.yaml'), 'w'))    
+    
+    main(args, config)

train_vqa.py

@@ -0,0 +1,202 @@
+'''
+ * Copyright (c) 2022, salesforce.com, inc.
+ * All rights reserved.
+ * SPDX-License-Identifier: BSD-3-Clause
+ * For full license text, see LICENSE.txt file in the repo root or https://opensource.org/licenses/BSD-3-Clause
+ * By Junnan Li
+'''
+import argparse
+import os
+import ruamel_yaml as yaml
+import numpy as np
+import random
+import time
+import datetime
+import json
+from pathlib import Path
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+from torch.utils.data import DataLoader
+import torch.backends.cudnn as cudnn
+import torch.distributed as dist
+
+from models.blip_vqa import blip_vqa
+import utils
+from utils import cosine_lr_schedule
+from data import create_dataset, create_sampler, create_loader
+from data.vqa_dataset import vqa_collate_fn
+from data.utils import save_result
+
+
+def train(model, data_loader, optimizer, epoch, device):
+    # train
+    model.train()  
+    
+    metric_logger = utils.MetricLogger(delimiter="  ")
+    metric_logger.add_meter('lr', utils.SmoothedValue(window_size=1, fmt='{value:.6f}'))
+    metric_logger.add_meter('loss', utils.SmoothedValue(window_size=1, fmt='{value:.4f}'))
+
+    header = 'Train Epoch: [{}]'.format(epoch)
+    print_freq = 50    
+    
+    for i,(image, question, answer, weights, n) in enumerate(metric_logger.log_every(data_loader, print_freq, header)):
+        image, weights = image.to(device,non_blocking=True), weights.to(device,non_blocking=True)      
+
+        loss = model(image, question, answer, train=True, n=n, weights=weights)        
+        
+        optimizer.zero_grad()
+        loss.backward()
+        optimizer.step()    
+        
+        metric_logger.update(loss=loss.item())
+        metric_logger.update(lr=optimizer.param_groups[0]["lr"])
+
+    # gather the stats from all processes
+    metric_logger.synchronize_between_processes()
+    print("Averaged stats:", metric_logger.global_avg())     
+    return {k: "{:.3f}".format(meter.global_avg) for k, meter in metric_logger.meters.items()} 
+
+
+@torch.no_grad()
+def evaluation(model, data_loader, device, config) :
+    # test
+    model.eval()
+            
+    metric_logger = utils.MetricLogger(delimiter="  ")
+    header = 'Generate VQA test result:'
+    print_freq = 50
+    
+    result = []
+    
+    if config['inference']=='rank':   
+        answer_list = data_loader.dataset.answer_list
+        answer_candidates = model.tokenizer(answer_list, padding='longest', return_tensors='pt').to(device)    
+        answer_candidates.input_ids[:,0] = model.tokenizer.bos_token_id
+        
+    for n, (image, question, question_id) in enumerate(metric_logger.log_every(data_loader, print_freq, header)):        
+        image = image.to(device,non_blocking=True)             
+
+        if config['inference']=='generate':
+            answers = model(image, question, train=False, inference='generate') 
+            
+            for answer, ques_id in zip(answers, question_id):
+                ques_id = int(ques_id.item())       
+                result.append({"question_id":ques_id, "answer":answer})             
+            
+        elif config['inference']=='rank':    
+            answer_ids = model(image, question, answer_candidates, train=False, inference='rank', k_test=config['k_test'])      
+
+            for ques_id, answer_id in zip(question_id, answer_ids):
+                result.append({"question_id":int(ques_id.item()), "answer":answer_list[answer_id]})   
+
+    return result
+
+
+def main(args, config):
+    utils.init_distributed_mode(args)    
+    
+    device = torch.device(args.device)
+
+    # fix the seed for reproducibility
+    seed = args.seed + utils.get_rank()
+    torch.manual_seed(seed)
+    np.random.seed(seed)
+    random.seed(seed)
+    cudnn.benchmark = True
+    
+    #### Dataset #### 
+    print("Creating vqa datasets")
+    datasets = create_dataset('vqa', config)   
+    
+    if args.distributed:
+        num_tasks = utils.get_world_size()
+        global_rank = utils.get_rank()            
+        samplers = create_sampler(datasets, [True, False], num_tasks, global_rank)         
+    else:
+        samplers = [None, None]
+    
+    train_loader, test_loader = create_loader(datasets,samplers,
+                                              batch_size=[config['batch_size_train'],config['batch_size_test']],
+                                              num_workers=[4,4],is_trains=[True, False], 
+                                              collate_fns=[vqa_collate_fn,None]) 
+    #### Model #### 
+    print("Creating model")
+    model = blip_vqa(pretrained=config['pretrained'], image_size=config['image_size'], 
+                       vit=config['vit'], vit_grad_ckpt=config['vit_grad_ckpt'], vit_ckpt_layer=config['vit_ckpt_layer'])
+
+    model = model.to(device)   
+    
+    model_without_ddp = model
+    if args.distributed:
+        model = torch.nn.parallel.DistributedDataParallel(model, device_ids=[args.gpu])
+        model_without_ddp = model.module    
+    
+    optimizer = torch.optim.AdamW(params=model.parameters(), lr=config['init_lr'], weight_decay=config['weight_decay'])
+
+    best = 0
+    best_epoch = 0 
+       
+    print("Start training")
+    start_time = time.time()    
+    for epoch in range(0, config['max_epoch']):
+        if not args.evaluate:        
+            if args.distributed:
+                train_loader.sampler.set_epoch(epoch)
+                
+            cosine_lr_schedule(optimizer, epoch, config['max_epoch'], config['init_lr'], config['min_lr'])
+                
+            train_stats = train(model, train_loader, optimizer, epoch, device) 
+
+        else:         
+            break        
+        
+        if utils.is_main_process():     
+            log_stats = {**{f'train_{k}': v for k, v in train_stats.items()},
+                         'epoch': epoch,
+                        }                
+            with open(os.path.join(args.output_dir, "log.txt"),"a") as f:
+                f.write(json.dumps(log_stats) + "\n")                        
+                    
+            save_obj = {
+                'model': model_without_ddp.state_dict(),
+                'optimizer': optimizer.state_dict(),
+                'config': config,
+                'epoch': epoch,
+            }
+            torch.save(save_obj, os.path.join(args.output_dir, 'checkpoint_%02d.pth'%epoch))  
+
+        dist.barrier()         
+
+    vqa_result = evaluation(model_without_ddp, test_loader, device, config)        
+    result_file = save_result(vqa_result, args.result_dir, 'vqa_result')  
+                      
+    total_time = time.time() - start_time
+    total_time_str = str(datetime.timedelta(seconds=int(total_time)))
+    print('Training time {}'.format(total_time_str)) 
+    
+            
+
+if __name__ == '__main__':
+    parser = argparse.ArgumentParser()
+    parser.add_argument('--config', default='./configs/vqa.yaml') 
+    parser.add_argument('--output_dir', default='output/VQA')
+    parser.add_argument('--evaluate', action='store_true')      
+    parser.add_argument('--device', default='cuda')
+    parser.add_argument('--seed', default=42, type=int)
+    parser.add_argument('--world_size', default=1, type=int, help='number of distributed processes')    
+    parser.add_argument('--dist_url', default='env://', help='url used to set up distributed training')
+    parser.add_argument('--distributed', default=True, type=bool)
+    args = parser.parse_args()
+
+    config = yaml.load(open(args.config, 'r'), Loader=yaml.Loader)
+
+    args.result_dir = os.path.join(args.output_dir, 'result')
+
+    Path(args.output_dir).mkdir(parents=True, exist_ok=True)
+    Path(args.result_dir).mkdir(parents=True, exist_ok=True)
+        
+    yaml.dump(config, open(os.path.join(args.output_dir, 'config.yaml'), 'w'))    
+    
+    main(args, config)

utils.py

@@ -0,0 +1,278 @@
+import math
+def cosine_lr_schedule(optimizer, epoch, max_epoch, init_lr, min_lr):
+    """Decay the learning rate"""
+    lr = (init_lr - min_lr) * 0.5 * (1. + math.cos(math.pi * epoch / max_epoch)) + min_lr
+    for param_group in optimizer.param_groups:
+        param_group['lr'] = lr
+        
+def warmup_lr_schedule(optimizer, step, max_step, init_lr, max_lr):
+    """Warmup the learning rate"""
+    lr = min(max_lr, init_lr + (max_lr - init_lr) * step / max_step)
+    for param_group in optimizer.param_groups:
+        param_group['lr'] = lr    
+
+def step_lr_schedule(optimizer, epoch, init_lr, min_lr, decay_rate):        
+    """Decay the learning rate"""
+    lr = max(min_lr, init_lr * (decay_rate**epoch))
+    for param_group in optimizer.param_groups:
+        param_group['lr'] = lr    
+        
+import numpy as np
+import io
+import os
+import time
+from collections import defaultdict, deque
+import datetime
+
+import torch
+import torch.distributed as dist
+
+class SmoothedValue(object):
+    """Track a series of values and provide access to smoothed values over a
+    window or the global series average.
+    """
+
+    def __init__(self, window_size=20, fmt=None):
+        if fmt is None:
+            fmt = "{median:.4f} ({global_avg:.4f})"
+        self.deque = deque(maxlen=window_size)
+        self.total = 0.0
+        self.count = 0
+        self.fmt = fmt
+
+    def update(self, value, n=1):
+        self.deque.append(value)
+        self.count += n
+        self.total += value * n
+
+    def synchronize_between_processes(self):
+        """
+        Warning: does not synchronize the deque!
+        """
+        if not is_dist_avail_and_initialized():
+            return
+        t = torch.tensor([self.count, self.total], dtype=torch.float64, device='cuda')
+        dist.barrier()
+        dist.all_reduce(t)
+        t = t.tolist()
+        self.count = int(t[0])
+        self.total = t[1]
+
+    @property
+    def median(self):
+        d = torch.tensor(list(self.deque))
+        return d.median().item()
+
+    @property
+    def avg(self):
+        d = torch.tensor(list(self.deque), dtype=torch.float32)
+        return d.mean().item()
+
+    @property
+    def global_avg(self):
+        return self.total / self.count
+
+    @property
+    def max(self):
+        return max(self.deque)
+
+    @property
+    def value(self):
+        return self.deque[-1]
+
+    def __str__(self):
+        return self.fmt.format(
+            median=self.median,
+            avg=self.avg,
+            global_avg=self.global_avg,
+            max=self.max,
+            value=self.value)
+
+
+class MetricLogger(object):
+    def __init__(self, delimiter="\t"):
+        self.meters = defaultdict(SmoothedValue)
+        self.delimiter = delimiter
+
+    def update(self, **kwargs):
+        for k, v in kwargs.items():
+            if isinstance(v, torch.Tensor):
+                v = v.item()
+            assert isinstance(v, (float, int))
+            self.meters[k].update(v)
+
+    def __getattr__(self, attr):
+        if attr in self.meters:
+            return self.meters[attr]
+        if attr in self.__dict__:
+            return self.__dict__[attr]
+        raise AttributeError("'{}' object has no attribute '{}'".format(
+            type(self).__name__, attr))
+
+    def __str__(self):
+        loss_str = []
+        for name, meter in self.meters.items():
+            loss_str.append(
+                "{}: {}".format(name, str(meter))
+            )
+        return self.delimiter.join(loss_str)
+
+    def global_avg(self):
+        loss_str = []
+        for name, meter in self.meters.items():
+            loss_str.append(
+                "{}: {:.4f}".format(name, meter.global_avg)
+            )
+        return self.delimiter.join(loss_str)    
+    
+    def synchronize_between_processes(self):
+        for meter in self.meters.values():
+            meter.synchronize_between_processes()
+
+    def add_meter(self, name, meter):
+        self.meters[name] = meter
+
+    def log_every(self, iterable, print_freq, header=None):
+        i = 0
+        if not header:
+            header = ''
+        start_time = time.time()
+        end = time.time()
+        iter_time = SmoothedValue(fmt='{avg:.4f}')
+        data_time = SmoothedValue(fmt='{avg:.4f}')
+        space_fmt = ':' + str(len(str(len(iterable)))) + 'd'
+        log_msg = [
+            header,
+            '[{0' + space_fmt + '}/{1}]',
+            'eta: {eta}',
+            '{meters}',
+            'time: {time}',
+            'data: {data}'
+        ]
+        if torch.cuda.is_available():
+            log_msg.append('max mem: {memory:.0f}')
+        log_msg = self.delimiter.join(log_msg)
+        MB = 1024.0 * 1024.0
+        for obj in iterable:
+            data_time.update(time.time() - end)
+            yield obj
+            iter_time.update(time.time() - end)
+            if i % print_freq == 0 or i == len(iterable) - 1:
+                eta_seconds = iter_time.global_avg * (len(iterable) - i)
+                eta_string = str(datetime.timedelta(seconds=int(eta_seconds)))
+                if torch.cuda.is_available():
+                    print(log_msg.format(
+                        i, len(iterable), eta=eta_string,
+                        meters=str(self),
+                        time=str(iter_time), data=str(data_time),
+                        memory=torch.cuda.max_memory_allocated() / MB))
+                else:
+                    print(log_msg.format(
+                        i, len(iterable), eta=eta_string,
+                        meters=str(self),
+                        time=str(iter_time), data=str(data_time)))
+            i += 1
+            end = time.time()
+        total_time = time.time() - start_time
+        total_time_str = str(datetime.timedelta(seconds=int(total_time)))
+        print('{} Total time: {} ({:.4f} s / it)'.format(
+            header, total_time_str, total_time / len(iterable)))
+        
+
+class AttrDict(dict):
+    def __init__(self, *args, **kwargs):
+        super(AttrDict, self).__init__(*args, **kwargs)
+        self.__dict__ = self
+
+
+def compute_acc(logits, label, reduction='mean'):
+    ret = (torch.argmax(logits, dim=1) == label).float()
+    if reduction == 'none':
+        return ret.detach()
+    elif reduction == 'mean':
+        return ret.mean().item()
+
+def compute_n_params(model, return_str=True):
+    tot = 0
+    for p in model.parameters():
+        w = 1
+        for x in p.shape:
+            w *= x
+        tot += w
+    if return_str:
+        if tot >= 1e6:
+            return '{:.1f}M'.format(tot / 1e6)
+        else:
+            return '{:.1f}K'.format(tot / 1e3)
+    else:
+        return tot
+
+def setup_for_distributed(is_master):
+    """
+    This function disables printing when not in master process
+    """
+    import builtins as __builtin__
+    builtin_print = __builtin__.print
+
+    def print(*args, **kwargs):
+        force = kwargs.pop('force', False)
+        if is_master or force:
+            builtin_print(*args, **kwargs)
+
+    __builtin__.print = print
+
+
+def is_dist_avail_and_initialized():
+    if not dist.is_available():
+        return False
+    if not dist.is_initialized():
+        return False
+    return True
+
+
+def get_world_size():
+    if not is_dist_avail_and_initialized():
+        return 1
+    return dist.get_world_size()
+
+
+def get_rank():
+    if not is_dist_avail_and_initialized():
+        return 0
+    return dist.get_rank()
+
+
+def is_main_process():
+    return get_rank() == 0
+
+
+def save_on_master(*args, **kwargs):
+    if is_main_process():
+        torch.save(*args, **kwargs)
+
+
+def init_distributed_mode(args):
+    if 'RANK' in os.environ and 'WORLD_SIZE' in os.environ:
+        args.rank = int(os.environ["RANK"])
+        args.world_size = int(os.environ['WORLD_SIZE'])
+        args.gpu = int(os.environ['LOCAL_RANK'])
+    elif 'SLURM_PROCID' in os.environ:
+        args.rank = int(os.environ['SLURM_PROCID'])
+        args.gpu = args.rank % torch.cuda.device_count()
+    else:
+        print('Not using distributed mode')
+        args.distributed = False
+        return
+
+    args.distributed = True
+
+    torch.cuda.set_device(args.gpu)
+    args.dist_backend = 'nccl'
+    print('| distributed init (rank {}, word {}): {}'.format(
+        args.rank, args.world_size, args.dist_url), flush=True)
+    torch.distributed.init_process_group(backend=args.dist_backend, init_method=args.dist_url,
+                                         world_size=args.world_size, rank=args.rank)
+    torch.distributed.barrier()
+    setup_for_distributed(args.rank == 0)        
+        
+