[#1] refactoring: Alpha -> Seq2Seq. We rely on git tags for versioning models.

config.yaml

@@ -1,8 +1,8 @@
-alpha:
-  overfit:
-    bart: facebook/bart-base
-    lr: 0.0001
-    literal2idiomatic_ver: pie_v0
-    max_epochs: 100
-    batch_size: 100
-    shuffle: true
+tag011:
+  desc: just overfitting
+  bart: facebook/bart-base
+  lr: 0.0001
+  literal2idiomatic_ver: tag01
+  max_epochs: 100
+  batch_size: 100
+  shuffle: true

explore/explore_bert_base_multilingual_tokenizer.py

@@ -1,44 +0,0 @@
-from idiomify.fetchers import fetch_idiom2def
-from transformers import AutoTokenizer, BertTokenizer, BertTokenizerFast
-
-
-def main():
-    tokenizer = BertTokenizer.from_pretrained("bert-base-multilingual-uncased")
-    idiom2def = fetch_idiom2def("d")  # eng2kor
-
-    for idiom, definition in idiom2def:
-        print(tokenizer.decode(tokenizer(idiom)['input_ids']),
-              tokenizer.decode(tokenizer(definition)['input_ids']))
-
-# right, the tokenizer knows Korean, which is great.
-"""
-/opt/homebrew/Caskroom/miniforge/base/envs/idiomify-demo/bin/python /Users/eubinecto/Desktop/Projects/Toy/idiomify-demo/explore/explore_mbert_tokenizer.py
-[CLS] beat around the bush [SEP] [CLS] 불쾌하거나 민감한 주제에 대해 직접적으로 이야기하는 것을 피하기 위해 모호하거나 완곡하게 말한다. [SEP]
-[CLS] beat around the bush [SEP] [CLS] 단어나 태도가 우회적이다 [SEP]
-[CLS] beat around the bush [SEP] [CLS] 우물쭈물하다 [SEP]
-[CLS] beat around the bush [SEP] [CLS] 우회적으로 접근하다 [SEP]
-[CLS] backhanded compliment [SEP] [CLS] 칭찬으로 가장한 모욕적이거나 부정적인 논평 [SEP]
-[CLS] backhanded compliment [SEP] [CLS] 의도하지 않거나 애매한 칭찬 [SEP]
-[CLS] backhanded compliment [SEP] [CLS] 누군가를 칭찬하는 것 같지만 비판으로도 이해될 수 있는 말 [SEP]
-[CLS] backhanded compliment [SEP] [CLS] 남을 기쁘게 하는 말 같지만 모욕이 될 수도 있는 말 [SEP]
-[CLS] backhanded compliment [SEP] [CLS] 감탄하는 듯 하면서도 모욕으로 이해될 수 있는 말 [SEP]
-[CLS] steer clear of [SEP] [CLS] 누군가나 뭔가를 피하다 [SEP]
-[CLS] steer clear of [SEP] [CLS] 떨어져 지내다 [SEP]
-[CLS] steer clear of [SEP] [CLS] 피하거나 멀리하도록 주의하다 [SEP]
-[CLS] steer clear of [SEP] [CLS] 불쾌하거나 위험하거나 문제를 일으킬 것 같은 사람이나 물건을 피하다 [SEP]
-[CLS] steer clear of [SEP] [CLS] 일부러 피하다 [SEP]
-[CLS] dish it out [SEP] [CLS] 가혹한 생각, 비판, 또는 모욕의 목소리를 내는 것. [SEP]
-[CLS] dish it out [SEP] [CLS] 누군가 또는 무언가에 대해 험담하는 것 [SEP]
-[CLS] dish it out [SEP] [CLS] 어떤 것을 주거나 정보나 당신의 의견과 같은 것을 말하는 것 [SEP]
-[CLS] dish it out [SEP] [CLS] 다른 사람을 쉽게 비판하지만 다른 사람이 자신을 비판할때는 좋아하지 않음 [SEP]
-[CLS] dish it out [SEP] [CLS] 다른 사람을 비판하다 [SEP]
-[CLS] make headway [SEP] [CLS] 성취하고자 하는 어떤 것에 진척이 생기다 [SEP]
-[CLS] make headway [SEP] [CLS] 특히 이것이 느리거나 어려울 때, 진전을 이루다. [SEP]
-[CLS] make headway [SEP] [CLS] 전진하다 [SEP]
-[CLS] make headway [SEP] [CLS] 앞으로 나아가거나 진전을 이루다 [SEP]
-[CLS] make headway [SEP] [CLS] 성공하기 시작하다 [SEP]
-"""
-
-
-if __name__ == '__main__':
-    main()

explore/explore_bert_base_tokenizer.py

@@ -1,45 +0,0 @@
-from idiomify.fetchers import fetch_idiom2def
-from transformers import AutoTokenizer, BertTokenizer, BertTokenizerFast
-
-
-def main():
-
-    tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
-    idiom2def = fetch_idiom2def("c")  # eng2eng
-    for idiom, definition in idiom2def:
-        print(tokenizer.decode(tokenizer(idiom)['input_ids']),
-              tokenizer.decode(tokenizer(definition)['input_ids']))
-
-"""
-/opt/homebrew/Caskroom/miniforge/base/envs/idiomify-demo/bin/python /Users/eubinecto/Desktop/Projects/Toy/idiomify-demo/explore/explore_bert_base_tokenizer.py
-Downloading: 100%|██████████| 226k/226k [00:00<00:00, 298kB/s]
-Downloading: 100%|██████████| 28.0/28.0 [00:00<00:00, 8.27kB/s]
-Downloading: 100%|██████████| 455k/455k [00:01<00:00, 449kB/s]
-[CLS] beat around the bush [SEP] [CLS] to speak vaguely or euphemistically so as to avoid talkingdirectly about an unpleasant or sensitive topic [SEP]
-[CLS] beat around the bush [SEP] [CLS] indirection in word or deed [SEP]
-[CLS] beat around the bush [SEP] [CLS] to shilly - shally [SEP]
-[CLS] beat around the bush [SEP] [CLS] to approach something in a roundabout way [SEP]
-[CLS] backhanded compliment [SEP] [CLS] an insulting or negative comment disguised as praise. [SEP]
-[CLS] backhanded compliment [SEP] [CLS] an unintended or ambiguous compliment. [SEP]
-[CLS] backhanded compliment [SEP] [CLS] a remark which seems to be praising someone or something but which could also be understood as criticism [SEP]
-[CLS] backhanded compliment [SEP] [CLS] a remark that seems to say something pleasant about a person but could also be an insult [SEP]
-[CLS] backhanded compliment [SEP] [CLS] a remark that seems to express admiration but could also be understood as an insult [SEP]
-[CLS] steer clear of [SEP] [CLS] to avoid someone or something. [SEP]
-[CLS] steer clear of [SEP] [CLS] stay away from [SEP]
-[CLS] steer clear of [SEP] [CLS] take care to avoid or keep away from [SEP]
-[CLS] steer clear of [SEP] [CLS] to avoid someone or something that seems unpleasant, dangerous, or likely to cause problems [SEP]
-[CLS] steer clear of [SEP] [CLS] deliberately avoid someone [SEP]
-[CLS] dish it out [SEP] [CLS] to voice harsh thoughts, criticisms, or insults. [SEP]
-[CLS] dish it out [SEP] [CLS] to gossip about someone or something [SEP]
-[CLS] dish it out [SEP] [CLS] to give something, or to tell something such as information or your opinions [SEP]
-[CLS] dish it out [SEP] [CLS] someone easily criticizes other people but does not like it when other people criticize him or her [SEP]
-[CLS] dish it out [SEP] [CLS] to criticize other people [SEP]
-[CLS] make headway [SEP] [CLS] make progress with something that you are trying to achieve. [SEP]
-[CLS] make headway [SEP] [CLS] make progress, especially when this is slow or difficult [SEP]
-[CLS] make headway [SEP] [CLS] to advance. [SEP]
-[CLS] make headway [SEP] [CLS] to move forward or make progress [SEP]
-[CLS] make headway [SEP] [CLS] to begin to succeed [SEP]
-"""
-
-if __name__ == '__main__':
-    main()

explore/explore_fetch_epie_counts.py

@@ -1,19 +0,0 @@
-from idiomify.fetchers import fetch_epie
-
-
-def main():
-    idioms = set([
-        idiom
-        for idiom, _, _ in fetch_epie()
-    ])
-    contexts = [
-        context
-        for _, _, context in fetch_epie()
-    ]
-    print("Total number of idioms:", len(idioms))
-    # This should learn... this - what I need for now is building a datamodule out of this
-    print("Total number of contexts:", len(contexts))
-
-
-if __name__ == '__main__':
-    main()

explore/{explore_fetch_alpha.py → explore_fetch_seq2seq.py}

@@ -1,8 +1,8 @@
-from idiomify.fetchers import fetch_alpha
+from idiomify.fetchers import fetch_seq2seq
 
 
 def main():
-    model = fetch_alpha("overfit")
+    model = fetch_seq2seq("overfit")
     print(model.bart.config)
 
 

explore/{explore_fetch_alpha_predict.py → explore_fetch_seq2seq_predict.py}

@@ -1,10 +1,10 @@
 from transformers import BartTokenizer
 from builders import SourcesBuilder
-from fetchers import fetch_alpha
+from fetchers import fetch_seq2seq
 
 
 def main():
-    model = fetch_alpha("overfit")
+    model = fetch_seq2seq("overfit")
     tokenizer = BartTokenizer.from_pretrained("facebook/bart-base")
     lit2idi = [
         ("my man", ""),

explore/explore_nlpaug.py

@@ -0,0 +1,21 @@
+
+import nlpaug.augmenter.word as naw
+import nlpaug.augmenter.sentence as nas
+
+import nltk
+
+
+sent = "I am really happy with the new job and I mean that with sincere feeling"
+
+
+def main():
+    nltk.download("omw-1.4")
+    # this seems legit! I could definitely use this to increase the accuracy of the model
+    # for a few idioms (possibly ten, ten very different but frequent idioms)
+    aug = naw.ContextualWordEmbsAug()
+    augmented = aug.augment(sent, n=10)
+    print(augmented)
+
+
+if __name__ == '__main__':
+    main()

idiomify/builders.py

@@ -4,7 +4,7 @@ builders must accept device as one of the parameters.
 """
 import torch
 from typing import List, Tuple
-from transformers import BertTokenizer, BartTokenizer
+from transformers import BartTokenizer
 
 
 class TensorBuilder:
@@ -61,7 +61,9 @@ class SourcesBuilder(TensorBuilder):
 
 
 class TargetsRightShiftedBuilder(TensorBuilder):
-
+    """
+    This is to be used only for training. As for inference, we don't need this.
+    """
     def __call__(self, literal2idiomatic: List[Tuple[str, str]]) -> torch.Tensor:
         encodings = self.tokenizer([
             self.tokenizer.bos_token + idiomatic  # starts with bos, but does not end with eos (right-shifted)
@@ -73,9 +75,7 @@ class TargetsRightShiftedBuilder(TensorBuilder):
 
 
 class TargetsBuilder(TensorBuilder):
-    """
-    This is to be used only for training. As for inference, we don't need this.
-    """
+
     def __call__(self, literal2idiomatic: List[Tuple[str, str]]) -> torch.Tensor:
         encodings = self.tokenizer([
             idiomatic + self.tokenizer.eos_token  # no bos, but ends with eos

idiomify/fetchers.py

@@ -5,43 +5,10 @@ import wandb
 import requests
 from typing import Tuple, List
 from wandb.sdk.wandb_run import Run
-from idiomify.paths import CONFIG_YAML, idioms_dir, literal2idiomatic, alpha_dir
-from idiomify.urls import (
-    EPIE_IMMUTABLE_IDIOMS_URL,
-    EPIE_IMMUTABLE_IDIOMS_CONTEXTS_URL,
-    EPIE_IMMUTABLE_IDIOMS_TAGS_URL,
-    EPIE_MUTABLE_IDIOMS_URL,
-    EPIE_MUTABLE_IDIOMS_CONTEXTS_URL,
-    EPIE_MUTABLE_IDIOMS_TAGS_URL,
-    PIE_URL
-)
+from idiomify.paths import CONFIG_YAML, idioms_dir, literal2idiomatic, seq2seq_dir
+from idiomify.urls import PIE_URL
 from transformers import AutoModelForSeq2SeqLM, AutoConfig
-from models import Alpha
-
-
-def fetch_epie(ver: str) -> List[Tuple[str, str, str]]:
-    """
-    It fetches the EPIE idioms, contexts, and tags from the web
-    :param ver: str
-    :type ver: str
-    :return: A list of tuples. Each tuple contains three strings: an idiom, a context, and a tag.
-    """
-    if ver == "immutable":
-        idioms_url = EPIE_IMMUTABLE_IDIOMS_URL
-        contexts_url = EPIE_IMMUTABLE_IDIOMS_CONTEXTS_URL
-        tags_url = EPIE_IMMUTABLE_IDIOMS_TAGS_URL
-    elif ver == "mutable":
-        idioms_url = EPIE_MUTABLE_IDIOMS_URL
-        contexts_url = EPIE_MUTABLE_IDIOMS_CONTEXTS_URL
-        tags_url = EPIE_MUTABLE_IDIOMS_TAGS_URL
-    else:
-        raise ValueError
-    idioms = requests.get(idioms_url).text
-    contexts = requests.get(contexts_url).text
-    tags = requests.get(tags_url).text
-    return list(zip(idioms.strip().split("\n"),
-                    contexts.strip().split("\n"),
-                    tags.strip().split("\n")))
+from idiomify.models import Seq2Seq
 
 
 def fetch_pie() -> list:
@@ -86,16 +53,16 @@ def fetch_literal2idiomatic(ver: str, run: Run = None) -> List[Tuple[str, str]]:
         return [(row[0], row[1]) for row in reader]
 
 
-def fetch_alpha(ver: str, run: Run = None) -> Alpha:
+def fetch_seq2seq(ver: str, run: Run = None) -> Seq2Seq:
     if run:
-        artifact = run.use_artifact(f"alpha:{ver}", type="model")
+        artifact = run.use_artifact(f"seq2seq:{ver}", type="model")
     else:
-        artifact = wandb.Api().artifact(f"eubinecto/idiomify/alpha:{ver}", type="model")
+        artifact = wandb.Api().artifact(f"eubinecto/idiomify/seq2seq:{ver}", type="model")
     config = artifact.metadata
-    artifact_dir = artifact.download(root=alpha_dir(ver))
+    artifact_dir = artifact.download(root=seq2seq_dir(ver))
     ckpt_path = path.join(artifact_dir, "model.ckpt")
     bart = AutoModelForSeq2SeqLM.from_config(AutoConfig.from_pretrained(config['bart']))
-    alpha = Alpha.load_from_checkpoint(ckpt_path, bart=bart)
+    alpha = Seq2Seq.load_from_checkpoint(ckpt_path, bart=bart)
     return alpha
 
 

idiomify/idiomifier.py

@@ -1,22 +0,0 @@
-from transformers import BartTokenizer
-from builders import SourcesBuilder
-from models import Alpha
-
-
-class Idiomifier:
-
-    def __init__(self, model: Alpha, tokenizer: BartTokenizer):
-        self.model = model
-        self.builder = SourcesBuilder(tokenizer)
-        self.model.eval()
-
-    def __call__(self, src: str, max_length=100) -> str:
-        srcs = self.builder(literal2idiomatic=[(src, "")])
-        pred_ids = self.model.bart.generate(
-            inputs=srcs[:, 0],  # (N, 2, L) -> (N, L)
-            attention_mask=srcs[:, 1],  # (N, 2, L) -> (N, L)
-            decoder_start_token_id=self.model.hparams['bos_token_id'],
-            max_length=max_length,
-        ).squeeze()  # -> (N, L_t) -> (L_t)
-        tgt = self.builder.tokenizer.decode(pred_ids, skip_special_tokens=True)
-        return tgt

idiomify/models.py

@@ -5,12 +5,14 @@ from typing import Tuple
 import torch
 from torch.nn import functional as F
 import pytorch_lightning as pl
-from transformers import BartForConditionalGeneration
+from transformers import BartForConditionalGeneration, BartTokenizer
+from idiomify.builders import SourcesBuilder
 
 
-class Alpha(pl.LightningModule):  # noqa
+# for training
+class Seq2Seq(pl.LightningModule):  # noqa
     """
-    the baseline.
+    the baseline is in here.
     """
     def __init__(self, bart: BartForConditionalGeneration, lr: float, bos_token_id: int, pad_token_id: int):  # noqa
         super().__init__()
@@ -54,3 +56,23 @@ class Alpha(pl.LightningModule):  # noqa
         """
         # The authors used Adam, so we might as well use it as well.
         return torch.optim.AdamW(self.parameters(), lr=self.hparams['lr'])
+
+
+# for inference
+class Idiomifier:
+
+    def __init__(self, model: Seq2Seq, tokenizer: BartTokenizer):
+        self.model = model
+        self.builder = SourcesBuilder(tokenizer)
+        self.model.eval()
+
+    def __call__(self, src: str, max_length=100) -> str:
+        srcs = self.builder(literal2idiomatic=[(src, "")])
+        pred_ids = self.model.bart.generate(
+            inputs=srcs[:, 0],  # (N, 2, L) -> (N, L)
+            attention_mask=srcs[:, 1],  # (N, 2, L) -> (N, L)
+            decoder_start_token_id=self.model.hparams['bos_token_id'],
+            max_length=max_length,
+        ).squeeze()  # -> (N, L_t) -> (L_t)
+        tgt = self.builder.tokenizer.decode(pred_ids, skip_special_tokens=True)
+        return tgt

idiomify/paths.py

@@ -6,12 +6,12 @@ CONFIG_YAML = ROOT_DIR / "config.yaml"
 
 
 def idioms_dir(ver: str) -> Path:
-    return ARTIFACTS_DIR / f"idioms_{ver}"
+    return ARTIFACTS_DIR / f"idioms-{ver}"
 
 
 def literal2idiomatic(ver: str) -> Path:
-    return ARTIFACTS_DIR / f"literal2idiomatic_{ver}"
+    return ARTIFACTS_DIR / f"literal2idiomatic-{ver}"
 
 
-def alpha_dir(ver: str) -> Path:
-    return ARTIFACTS_DIR / f"alpha_{ver}"
+def seq2seq_dir(ver: str) -> Path:
+    return ARTIFACTS_DIR / f"seq2seq-{ver}"

idiomify/urls.py

@@ -11,6 +11,3 @@ EPIE_MUTABLE_IDIOMS_CONTEXTS_URL = "https://github.com/prateeksaxena2809/EPIE_Co
 # https://aclanthology.org/2021.mwe-1.5/
 # right, let's just work on it.
 PIE_URL = "https://raw.githubusercontent.com/zhjjn/MWE_PIE/main/data_cleaned.csv"
-
-
-

main_infer.py

@@ -1,27 +1,24 @@
 import argparse
-from termcolor import colored
-from idiomifier import Idiomifier
-from idiomify.fetchers import fetch_config, fetch_alpha
+from idiomify.models import Idiomifier
+from idiomify.fetchers import fetch_config, fetch_seq2seq
 from transformers import BartTokenizer
 
 
 def main():
     parser = argparse.ArgumentParser()
-    parser.add_argument("--model", type=str,
-                        default="alpha")
-    parser.add_argument("--ver", type=str,
-                        default="overfit")
+    parser.add_argument("--ver", type=str, default="tag011")
     parser.add_argument("--src", type=str,
-                        default="If there's any benefits to losing my job, it's that I'll now be able to go to school full-time and finish my degree earlier.")
+                        default="If there's any good to loosing my job,"
+                                " it's that I'll now be able to go to school full-time and finish my degree earlier.")
     args = parser.parse_args()
-    config = fetch_config()[args.model][args.ver]
+    config = fetch_config()[args.ver]
     config.update(vars(args))
-    model = fetch_alpha(config['ver'])
+    model = fetch_seq2seq(config['ver'])
     tokenizer = BartTokenizer.from_pretrained(config['bart'])
     idiomifier = Idiomifier(model, tokenizer)
     src = config['src']
     tgt = idiomifier(src=config['src'])
-    print(src, "\n->", colored(tgt, "blue"))
+    print(src, "\n->", tgt)
 
 
 if __name__ == '__main__':

main_train.py

@@ -8,20 +8,19 @@ from pytorch_lightning.loggers import WandbLogger
 from transformers import BartTokenizer, BartForConditionalGeneration
 from idiomify.data import IdiomifyDataModule
 from idiomify.fetchers import fetch_config
-from idiomify.models import Alpha
+from idiomify.models import Seq2Seq
 from idiomify.paths import ROOT_DIR
 
 
 def main():
     parser = argparse.ArgumentParser()
-    parser.add_argument("--model", type=str, default="alpha")
-    parser.add_argument("--ver", type=str, default="overfit ")
+    parser.add_argument("--ver", type=str, default="tag011")
     parser.add_argument("--num_workers", type=int, default=os.cpu_count())
     parser.add_argument("--log_every_n_steps", type=int, default=1)
     parser.add_argument("--fast_dev_run", action="store_true", default=False)
     parser.add_argument("--upload", dest='upload', action='store_true', default=False)
     args = parser.parse_args()
-    config = fetch_config()[args.model][args.ver]
+    config = fetch_config()[args.ver]
     config.update(vars(args))
     if not config['upload']:
         print(colored("WARNING: YOU CHOSE NOT TO UPLOAD. NOTHING BUT LOGS WILL BE SAVED TO WANDB", color="red"))
@@ -29,12 +28,8 @@ def main():
     # prepare the model
     bart = BartForConditionalGeneration.from_pretrained(config['bart'])
     tokenizer = BartTokenizer.from_pretrained(config['bart'])
-    if config['model'] == "alpha":
-        model = Alpha(bart, config['lr'], tokenizer.bos_token_id, tokenizer.pad_token_id)
-    else:
-        raise NotImplementedError
+    model = Seq2Seq(bart, config['lr'], tokenizer.bos_token_id, tokenizer.pad_token_id)
     # prepare the datamodule
-
     with wandb.init(entity="eubinecto", project="idiomify", config=config) as run:
         datamodule = IdiomifyDataModule(config, tokenizer, run)
         logger = WandbLogger(log_model=False)

main_upload_idioms.py

@@ -11,17 +11,13 @@ import wandb
 
 def main():
     parser = argparse.ArgumentParser()
-    parser.add_argument("--ver", type=str, default="pie_v0",
-                        choices=["pie_v0", "pie_v1"])
+    parser.add_argument("--ver", type=str, default="tag01")
     config = vars(parser.parse_args())
 
     # get the idioms here
-    if config['ver'] == "pie_v0":
+    if config['ver'] == "tag01":
         # only the first 106, and this is for piloting
         idioms = set([row[0] for row in fetch_pie()[:106]])
-    elif config['ver'] == "pie_v1":
-        # just include all
-        idioms = set([row[0] for row in fetch_pie()])
     else:
         raise NotImplementedError
     idioms = list(idioms)

main_upload_literal2idiomatic.py

@@ -12,21 +12,15 @@ import wandb
 
 def main():
     parser = argparse.ArgumentParser()
-    parser.add_argument("--ver", type=str, default="pie_v0",
-                        choices=["pie_v0", "pie_v1"])
+    parser.add_argument("--ver", type=str, default="tag01")
     config = vars(parser.parse_args())
 
     # get the idioms here
-    if config['ver'] == "pie_v0":
+    if config['ver'] == "tag01":
         # only the first 106, and we use this just for piloting
         literal2idiom = [
             (row[3], row[2]) for row in fetch_pie()[:106]
         ]
-    elif config['ver'] == "pie_v1":
-        # just include all
-        literal2idiom = [
-            (row[3], row[2]) for row in fetch_pie()
-        ]
     else:
         raise NotImplementedError