Hugging Face's logo

openthaigpt
/
CLIPTextCamembertModelWithProjection

Feature Extraction
Transformers
Safetensors
Thai
English
clip_text_camembert
openthaigpt
custom_code
Model card Files Community
CLIPTextCamembertModelWithProjection / README.md
boat1603's picture
boat1603
Update README.md
51d4f57 verified
preview code
|
raw
history blame
8.31 kB
metadata
language:
  - th
  - en
tags:
  - openthaigpt
license: mit

Introduction

The foundational technology for generative prompt models are Language-Image pretraining models such as CLIP (Contrastive Language-Image Pre-Training) which aligned Language-Image latent of image and text encoder. We can apply latent vector for zero-short classification and image searching. For generative prompt models, we can train generative model using frozen image encoder and then replace image encoder with text encoder to be a prompt of generative model in the inference pipeline.

Scope of work

From limited of computing resources, datasets, engineers we purpose to train CLIP model with 2 stage training of CLIP model

  • Stage 1: Language encoder distillation training We will train Thai (or Bilingual EN-TH) text encoder with original CLIP encoder following Multilingual-CLIP using EN-EN, EN-TH text pairs of machine translation datasets.
  • Stage 2: Continue CLIP pretraining with frozen image encoder Distillation training model may not understand all of token especially specific words. We have to continue CLIP (or LiT, or SigLiT) pretraining with frozen image encoder to learn details of specific words. After we have our own CLIP model we will replace CLIP application text encoder with our own text encoder or we may finetuning application model to push performance of our model.

How to use

  • Install python package 

pip thai2transformers==0.1.2

  • Preprocessing

Texts are preprocessed with the following rules: process_transformers

  • Replace HTML forms of characters with the actual characters such asnbsp;with a space and \\\\\\\\
    with a line break [Howard and Ruder, 2018].

  • Remove empty brackets ((), {}, and []) than sometimes come up as a result of text extraction such as from Wikipedia.

  • Replace line breaks with spaces.

  • Replace more than one spaces with a single space

  • Remove more than 3 repetitive characters such as ดีมากกก to ดีมาก [Howard and Ruder, 2018]](https://arxiv.org/abs/1801.06146).

  • Word-level tokenization using [Phatthiyaphaibun et al., 2020] ’s newmm dictionary-based maximal matching tokenizer.

  • Replace repetitive words; this is done post-tokenization unlike [Howard and Ruder, 2018]. since there is no delimitation by space in Thai as in English.

  • Replace spaces with <\\\\\\\\_>. The SentencePiece tokenizer combines the spaces with other tokens. Since spaces serve as punctuation in Thai such as sentence boundaries similar to periods in English, combining it with other tokens will omit an important feature for tasks such as word tokenization and sentence breaking. Therefore, we opt to explicitly mark spaces with <\\\\\\\\_>.


  • How to load the text encoder 

from transformers import AutoModel,AutoProcessor
from thai2transformers.preprocess import process_transformers
model = AutoModel.from_pretrained("openthaigpt/CLIPTextCamembertModelWithProjection", trust_remote_code=True)
processor = AutoProcessor.from_pretrained("openthaigpt/CLIPTextCamembertModelWithProjection", trust_remote_code=True)

input_text = ["This is dog",
              "how are you today",
              "สวัสดีครับ วันนี้อากาศร้อนมาก"]
processed_input_text = [process_transformers(input_text_) for input_text_ in input_text ]

text_tokens = processor(text=processed_input_text, padding=True, return_tensors="pt")
embedding = model(**text_tokens).text_embeds

print(embedding,embedding.shape)

  • Output: 

tensor([[ 0.0318,  0.0341, -0.1317,  ..., -0.2763, -0.2103,  0.0968],
        [ 0.0579, -0.1373, -0.0293,  ..., -0.3926, -0.2002, -0.0497],
        [ 0.0303,  0.0440,  0.0217,  ..., -0.3282, -0.0100, -0.0757]],
       grad_fn=<MmBackward0>) torch.Size([3, 512])

Eample of model usage

  • Zero shot classification 

from torch import FloatTensor, IntTensor, Tensor
from transformers import AutoModel, AutoProcessor, CLIPModel

# Load image model and processor.
image_processor = AutoProcessor.from_pretrained("openai/clip-vit-base-patch32")
image_model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32").to(device)

# Load text model and processor.
text_processor = AutoProcessor.from_pretrained("openthaigpt/CLIPTextCamembertModelWithProjection", trust_remote_code=True)
text_model = AutoModel.from_pretrained("openthaigpt/CLIPTextCamembertModelWithProjection", trust_remote_code=True).to(device)

class_labels = ['แมว','หมา', 'นก']
label2id = {label: i for i, label in enumerate(class_labels)}

inputs = text_processor(text=class_labels, padding=True, return_tensors="pt")
inputs = {name: tensor.to(self.device) for name, tensor in inputs.items()}       
text_embeddings = self.text_model(**inputs).text_embeds
text_embeddings /= text_embeddings.norm(dim=1, keepdim=True)

inputs = image_processor(images=images, return_tensors="pt")
inputs = {name: tensor.to(self.device) for name, tensor in inputs.items()}
image_embeddings = self.image_model.get_image_features(**inputs)
image_embeddings /= image_embeddings.norm(dim=1, keepdim=True)

similarities = torch.mm(image_embeddings, text_embeddings.t())
logits = F.softmax(similarities, dim=1)
indices = torch.argmax(logits, dim=1)

logits = logits.detach().cpu()
indices = indices.detach().cpu()

predict= [class_labels[i] for i in indices ]

  • Text-Image retrieval 

import faiss
from torch import FloatTensor, IntTensor, Tensor
from transformers import AutoModel, AutoProcessor, CLIPModel

# Load image model and processor.
image_processor = AutoProcessor.from_pretrained("openai/clip-vit-base-patch32")
image_model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32").to(device)

# Load text model and processor.
text_processor = AutoProcessor.from_pretrained("openthaigpt/CLIPTextCamembertModelWithProjection", trust_remote_code=True)
text_model = AutoModel.from_pretrained("openthaigpt/CLIPTextCamembertModelWithProjection", trust_remote_code=True).to(device)

text_input = ['แมวสีส้ม','หมาสีดำ', 'นกสีขาว']
processed_input_text = [process_transformers(input_text_) for input_text_ in input_text ]


inputs = text_processor(text=processed_input_text, padding=True, return_tensors="pt")
inputs = {name: tensor.to(self.device) for name, tensor in inputs.items()}       
text_embeddings = self.text_model(**inputs).text_embeds
text_embeddings /= text_embeddings.norm(dim=1, keepdim=True)

inputs = image_processor(images=images, return_tensors="pt")
inputs = {name: tensor.to(self.device) for name, tensor in inputs.items()}
image_embeddings = self.image_model.get_image_features(**inputs)
image_embeddings /= image_embeddings.norm(dim=1, keepdim=True)


n = text_embeddings.shape[1]

text_index = faiss.IndexFlatIP(n)
image_index = faiss.IndexFlatIP(n)
text_index.add(text_embeddings)
image_index.add(image_embeddings)


# Get_image_search_recall_at_k
distances, retrieved_indices = image_index.search(text_embeddings, k=5)
recall_image_search = sum(1.0 if i in indices else 0.0
            for i, indices in zip(range(n), retrieved_indices)
        ) / float(n)

# Get_text_search_recall_at_k
distances, retrieved_indices = text_index.search(image_embeddings, k=5)
recall_text_search = sum(1.0 if i in indices else 0.0
            for i, indices in zip(range(n), retrieved_indices)
        ) / float(n)

Sponsors

Authors