Update ubee uclas

.gitignore

@@ -140,4 +140,4 @@ cython_debug/
 *.swp
 links/
 # .gitignore
-node_modules
+node_modulescachedir

data/test_en.txt

@@ -0,0 +1,69 @@
+Wuthering Heights
+
+
+--------------------------------------------------------------------------------
+
+Chapter 2
+
+ Chinese
+
+
+Yesterday afternoon set in misty and cold. I had half a mind to spend it by my study fire, instead of wading through heath and mud to Wuthering Heights. On coming up from dinner, however (N.B. I dine between twelve and one o'clock; the housekeeper, a matronly lady, taken as a fixture along with the house, could not, or would not, comprehend my request that I might be served at five), on mounting the stairs with this lazy intention, and stepping into the room, I saw a servant girl on her knees surrounded by brushes and coal-scuttles, and raising an infernal dust as she extinguished the flames with heaps of cinders. This spectacle drove me back immediately; I took my hat, and, after a four-miles' walk, arrived at Heathcliff's garden gate just in time to escape the first feathery flakes of a snow shower.
+
+On that bleak hill top the earth was hard with a black frost, and the air made me shiver through every limb. Being unable to remove the chain, I jumped over, and, running up the flagged causeway bordered with straggling gooseberry bushes, knocked vainly for admittance, till my knuckles tingled and the dogs howled.
+
+`Wretched inmates!' I ejaculated mentally, `you deserve perpetual isolation from your species for your churlish inhospitality. At least, I would not keep my doors barred in the day time. I don't care--I will get in!' So resolved, I grasped the latch and shook it vehemently. Vinegar-faced Joseph projected his head from a round window of the barn.
+
+`Whet are ye for?' he shouted. `T' maister's dahn i' t' fowld. Go rahnd by th' end ut' laith, if yah went tuh spake tull him.'
+
+`Is there nobody inside to open the door?' I hallooed, responsively.
+
+`They's nobbut t' missis; and shoo'll nut oppen't an ye mak yer flaysome dins till neeght.'
+
+`Why? Cannot you tell her who I am, eh, Joseph?'
+
+`Nor-ne me! Aw'll hae noa hend wi't,' muttered the head, vanishing.
+
+The snow began to drive thickly. I seized the handle to essay another trial; when a young man without coat, and shouldering a pitchfork, appeared in the yard behind. He hailed me to follow him, and, after marching through a wash-house, and a paved area containing a coal shed, pump, and pigeon cot, we at length arrived in the huge, warm, cheerful apartment, where I was formerly received. It glowed delightfully in the radiance of an immense fire, compounded of coal, peat, and wood; and near the table, laid for a plentiful evening meal, I was pleased to observe the `missis', an individual whose existence I had never previously suspected. I bowed and waited, thinking she would bid me take a seat. She looked at me, leaning back in her chair, and remained motionless and mute.
+
+`Rough weather!' I remarked. `I'm afraid, Mrs Heathcliff, the door must bear the consequence of your servants' leisure attendance: I had hard work to make them hear me.'
+
+She never opened her mouth. I stared--she stared also: at any rate, she kept her eyes on me in a cool, regardless manner, exceedingly embarrassing and disagreeable.
+
+`Sit down,' said the young man gruffly. `He'll be in soon.'
+
+I obeyed; and hemmed, and called the villain Juno, who deigned, at this second interview, to move the extreme tip of her tail, in token of owning my acquaintance.
+
+`A beautiful animal!' I commenced again. `Do you intend parting with the little ones, madam?'
+
+`They are not mine,' said the amiable hostess, more repellingly than Heathcliff himself could have replied.
+
+`Ah, your favourites are among these?' I continued, turning to an obscure cushion full of something like cats.
+
+`A strange choice of favourites!' she observed scornfully.
+
+Unluckily, it was a heap of dead rabbits. I hemmed once more, and drew closer to the hearth, repeating my comment on the wildness of the evening.
+
+`You should not have come out,' she said, rising and reaching from the chimney-piece two of the painted canisters.
+
+Her position before was sheltered from the light; now, I had a distinct view of her whole figure and countenance. She was slender, and apparently scarcely past girlhood: an admirable form, and the most exquisite little face that I have ever had the pleasure of beholding; small features, very fair; flaxen ringlets, or rather golden, hanging loose on her delicate neck; and eyes, had they been agreeable in expression, they would have been irresistible: fortunately for my susceptible heart, the only sentiment they evinced hovered between scorn, and a kind of desperation, singularly unnatural to be detected there. The canisters were almost out of her reach; I made a motion to aid her; she turned upon me as a miser might turn if anyone attempted to assist him in counting his gold.
+
+`I don't want your help,' she snapped; `I can get them for myself.'
+
+`I beg your pardon!' I hastened to reply.
+
+`Were you asked to tea?' she demanded, tying an apron over her neat black frock, and standing with a spoonful of the leaf poised over the pot.
+
+`I shall be glad to have a cup,' I answered.
+
+`Were you asked?' she repeated.
+
+`No,' I said, half smiling. `You are the proper person to ask me.'
+　
+
+
+Contents PreviousChapter
+ NextChapter
+
+
+Homepage

data/test_zh.txt

@@ -0,0 +1,74 @@
+呼啸山庄
+
+--------------------------------------------------------------------------------
+
+第二章
+
+ 英文
+
+
+昨天下午又冷又有雾。我想就在书房炉边消磨一下午，不想踩着杂草污泥到呼啸山庄了。
+
+但是，吃过午饭（注意——我在十二点与一点钟之间吃午饭，而可以当作这所房子的附属物的管家婆，一位慈祥的太太却不能，或者并不愿理解我请求在五点钟开饭的用意），在我怀着这个懒惰的想法上了楼，迈进屋子的时候，看见一个女仆跪在地上，身边是扫帚和煤斗。她正在用一堆堆煤渣封火，搞起一片弥漫的灰尘。这景象立刻把我赶回头了。我拿了帽子，走了四里路，到达了希刺克厉夫的花园口口，刚好躲过了一场今年初降的鹅毛大雪。
+
+在那荒凉的山顶上，土地由于结了一层黑冰而冻得坚硬，冷空气使我四肢发抖。我弄不开门链，就跳进去，顺着两边种着蔓延的醋栗树丛的石路跑去。我白白地敲了半天门，一直敲到我的手指骨都痛了，狗也狂吠起来。
+
+“倒霉的人家！”我心里直叫，“只为你这样无礼待客，就该一辈子跟人群隔离。我至少还不会在白天把门闩住。我才不管呢——我要进去！”如此决定了。我就抓住门闩，使劲摇它。苦脸的约瑟夫从谷仓的一个圆窗里探出头来。
+
+“你干吗？”他大叫。“主人在牛栏里，你要是找他说话，就从这条路口绕过去。”
+
+“屋里没人开门吗？”我也叫起来。
+
+“除了太太没有别人。你就是闹腾到夜里，她也不会开。”
+
+“为什么？你就不能告诉她我是谁吗，呃，约瑟夫？”
+
+“别找我！我才不管这些闲事呢，”这个脑袋咕噜着，又不见了。
+
+雪开始下大了。我握住门柄又试一回。这时一个没穿外衣的年轻人，扛着一根草耙，在后面院子里出现了。他招呼我跟着他走，穿过了一个洗衣房和一片铺平的地，那儿有煤棚、抽水机和鸽笼，我们终于到了我上次被接待过的那间温暖的、热闹的大屋子。煤、炭和木材混合在一起燃起的熊熊炉火，使这屋子放着光彩。在准备摆上丰盛晚餐的桌旁，我很高兴地看到了那位“太太”，以前我从未料想到会有这么一个人存在的。我鞠躬等候，以为她会叫我坐下。她望望我，往她的椅背一靠，不动，也不出声。
+
+“天气真坏！”我说，“希刺克厉夫太太，恐怕大门因为您的仆人偷懒而大吃苦头，我费了好大劲才使他们听见我敲门！”
+
+她死不开口。我瞪眼——她也瞪眼。反正她总是以一种冷冷的、漠不关心的神气盯住我，使人十分窘，而且不愉快。
+
+“坐下吧，”那年轻人粗声粗气地说，“他就要来了。”
+
+我服从了；轻轻咳了一下，叫唤那恶狗朱诺。临到第二次会面，它总算赏脸，摇起尾巴尖，表示认我是熟人了。
+
+“好漂亮的狗！”我又开始说话。“您是不是打算不要这些小的呢，夫人？”
+
+“那些不是我的，”这可爱可亲的女主人说，比希刺克厉夫本人所能回答的腔调还要更冷淡些。
+
+“啊，您所心爱的是在这一堆里啦！”我转身指着一个看不清楚的靠垫上那一堆像猫似的东西，接着说下去。
+
+“谁会爱这些东西那才怪呢！”她轻蔑地说。
+
+倒霉，原来那是堆死兔子。我又轻咳一声，向火炉凑近些，又把今晚天气不好的话评论一通。
+
+“你本来就不该出来。”她说，站起来去拿壁炉台上的两个彩色茶叶罐。
+
+她原先坐在光线被遮住的地方，现在我把她的全身和面貌都看得清清楚楚。她苗条，显然还没有过青春期。挺好看的体态，还有一张我生平从未有幸见过的绝妙的小脸蛋。五官纤丽，非常漂亮。淡黄色的卷发，或者不如说是金黄色的，松松地垂在她那细嫩的颈上。至于眼睛，要是眼神能显得和悦些，就要使人无法抗拒了。对我这容易动情的心说来倒是常事，因为它们所表现的只是在轻蔑与近似绝望之间的一种情绪，而在那张脸上看见那样的眼神是特别不自然的。
+
+她简直够不到茶叶罐。我动了一动，想帮她一下。她猛地扭转身向我，像守财奴看见别人打算帮他数他的金子一样。
+
+“我不要你帮忙，”她怒气冲冲地说，“我自己拿得到。”
+
+“对不起！”我连忙回答。
+
+“是请你来吃茶的吗？”她问，把一条围裙系在她那干净的黑衣服上，就这样站着，拿一匙茶叶正要往茶壶里倒。
+
+“我很想喝杯茶。”我回答。
+
+“是请你来的吗？”她又问。
+
+“没有，”我说，勉强笑一笑。“您正好请我喝茶。”
+
+　
+
+
+目录
+ 上一章
+ 下一章
+
+
+返回首页

packages.txt

@@ -0,0 +1,3 @@
+icu-doc
+libicu-dev
+pkg-config

requirements.txt

@@ -1,11 +1,22 @@
 gradio
+install
 transformers
 sentencepiece
 sklearn
-logzero
 git+https://github.com/ffreemt/fast-langid
 git+https://github.com/ffreemt/align-model-pool
 sentence-transformers
 sentence_splitter
+logzero
 icecream
 alive-progress
+more_itertools
+#
+openpyxl
+# --- seg_text
+Morfessor
+pyicu
+pycld2
+tqdm
+polyglot
+sentence_splitter

ubee/__main__.py

@@ -1,9 +1,14 @@
 """Gen ubee main."""
 # pylint: disable=unused-import, wrong-import-position
 
+from typing import Tuple
+
+from pathlib import Path
 import sys
-from itertools import zip_longest
-from textwrap import dedent
+from random import shuffle
+
+# from itertools import zip_longest
+# from textwrap import dedent
 
 import gradio as gr
 
@@ -18,6 +23,7 @@ if "." not in sys.path:
 
 from ubee.ubee import ubee
 
+logzero.loglevel(10)
 ic_install()
 ic.configureOutput(
     includeContext=True,
@@ -32,7 +38,11 @@ def greet1(name):
     return "Hello " + name + "!!"
 
 
-def greet(text1, text2) -> pd.DataFrame:
+def greet(
+    text1,
+    text2,
+    thresh: float
+) -> Tuple[pd.DataFrame, pd.DataFrame]:
     """Take inputs, return outputs.
 
     Args:
@@ -44,12 +54,25 @@ def greet(text1, text2) -> pd.DataFrame:
     res1 = [elm.strip() for elm in text1.splitlines() if elm.strip()]
     res2 = [elm.strip() for elm in text2.splitlines() if elm.strip()]
 
-    _ = pd.DataFrame(zip_longest(res1, res2), columns=["text1", "text2"])
-    return _
+    # _ = pd.DataFrame(zip_longest(res1, res2), columns=["text1", "text2"])
+    # return _
+
+    res1_, res2_ = ubee(res1, res2, thresh)
+
+    return pd.DataFrame(res1_, columns=["text1", "text2", "likelihood"]), pd.DataFrame(res2_, columns=["text1", "text2"])
 
 
 def main():
     """Create main entry."""
+    text_zh = Path("data/test_zh.txt").read_text("utf8")
+    text_en = [
+        elm.strip()
+        for elm in Path("data/test_en.txt").read_text("utf8").splitlines()
+        if elm.strip()
+    ]
+    shuffle(text_en)
+    text_en = "\n\n".join(text_en)
+
     title = "Ultimatumbee Aligner"
     theme = "dark-grass"
     description = """WIP showcasing a novel aligner"""
@@ -62,20 +85,8 @@ def main():
 
     lines = 15
     placeholder = "Type or paste text here"
-    default1 = dedent(
-        """
-        test 1
-        abc
-        love you
-        """
-    )
-    default2 = dedent(
-        """
-        爱你
-        甲乙丙
-        测试 1
-        """
-    )
+    default1 = text_zh
+    default2 = text_en
     label1 = "text1"
     label2 = "text2"
     inputs = [
@@ -85,6 +96,12 @@ def main():
         gr.inputs.Textbox(
             lines=lines, placeholder=placeholder, default=default2, label=label2
         ),
+        gr.inputs.Slider(
+            minimum=0.0,
+            maximum=1.0,
+            step=0.1,
+            default=0.5,
+        ),
     ]
 
     out_df = gr.outputs.Dataframe(
@@ -95,8 +112,26 @@ def main():
         type="auto",
         label="To be aligned",
     )
-    outputs = [  # tot. 1
+    aligned = gr.outputs.Dataframe(
+        headers=None,
+        max_rows=lines,  # 20
+        max_cols=None,
+        overflow_row_behaviour="paginate",
+        type="auto",
+        label="Aligned",
+    )
+    leftover = gr.outputs.Dataframe(
+        headers=None,
+        max_rows=lines,  # 20
+        max_cols=None,
+        overflow_row_behaviour="paginate",
+        type="auto",
+        label="Leftover",
+    )
+    outputs = [  # tot. 3
         out_df,
+        aligned,
+        leftover,
     ]
 
     iface = gr.Interface(

ubee/gradiobee.py

@@ -0,0 +1,7 @@
+"""Function for gr's fn."""
+# pylint=disable=
+
+
+def ubee():
+    """Gen a dummy."""
+    ...

ubee/seg_text.py

@@ -0,0 +1,121 @@
+"""Split text to sentences.
+
+Use sentence_splitter if supported,
+else use polyglot.text.Text
+
+!apt install libicu-dev
+!install pyicu pycld2 Morfessor
+!pip install polyglot sentence_splitter
+"""
+# pylint: disable=
+
+from typing import List, Optional, Union
+
+import re
+from tqdm.auto import tqdm
+from polyglot.detect.base import logger as polyglot_logger
+from polyglot.text import Detector, Text
+from sentence_splitter import split_text_into_sentences
+
+from logzero import logger
+
+# turn of polyglot.text.Detector warning
+polyglot_logger.setLevel("ERROR")
+
+
+# fmt: off
+# use sentence_splitter if supported
+LANG_S = ["ca", "cs", "da", "nl", "en", "fi", "fr", "de",
+          "el", "hu", "is", "it", "lv", "lt", "no", "pl",
+          "pt", "ro", "ru", "sk", "sl", "es", "sv", "tr"]
+
+
+def _seg_text(
+        text: str,
+        lang: Optional[str] = None,
+        # qmode: bool = False,
+        maxlines: int = 1000
+) -> List[str]:
+    # fmt: on
+    """Split text to sentences.
+
+    Use sentence_splitter if supported,
+    else use polyglot.text.Text.sentences
+    Blank lines will be removed.
+
+    qmode: quick mode, skip split_text_into_sentences if True, default False
+        vectors for all books are based on qmode=False.
+        qmode=True is for quick test purpose only
+
+    maxlines (default 1000), threshold for turn on tqdm progressbar
+        set to <1 or a large number to turn it off
+    """
+    if lang is None:
+        try:
+            lang = Detector(text).language.code
+        except Exception as exc:
+            logger.info("text[:30]: %s", text[:30])
+            logger.warning(
+                "polyglot.text.Detector exc: %s, setting to 'en'",
+                exc
+            )
+            lang = "en"
+
+    # if not qmode and lang in LANG_S:
+    if lang in LANG_S:
+        _ = []
+        lines = text.splitlines()
+        # if maxlines > 1 and len(lines) > maxlines:
+        if len(lines) > maxlines > 1:
+            for para in tqdm(lines):
+                if para.strip():
+                    _.extend(split_text_into_sentences(para, lang))
+        else:
+            for para in lines:
+                if para.strip():
+                    _.extend(split_text_into_sentences(para, lang))
+        return _
+
+        # return split_text_into_sentences(text, lang)
+
+    # empty "" text or blank to avoid Exception
+    if not text.strip():
+        return []
+
+    return [elm.string for elm in Text(text, lang).sentences]
+
+
+# fmt: off
+def seg_text(
+        lst: Union[str, List[str]],
+        lang: Optional[str] = None,
+        maxlines: int = 1000,
+        extra: Optional[str] = None,
+) -> List[str]:
+    # fmt:on
+    """Split a list of text.
+
+    Arguments:
+        lst: text or text list
+        lang: optional lang code
+        maxlines: (default 1000), threshold for turn on tqdm progressbar, set to <1 or a large number to turn it off
+        extra: re.split(rf"{extra}, text) first
+    Returns:
+        list of splitted text.
+    """
+    if isinstance(lst, str):
+        lst = [lst]
+
+    if extra:
+        # insert \n
+        lst = [re.sub(rf"({extra})", r"\1\n", elm) for elm in lst]
+
+    res = []
+    for elm in lst:
+        res.extend(_seg_text(
+            elm,
+            lang=lang,
+            maxlines=maxlines,
+        ))
+
+    return res

ubee/ubee.py

@@ -1,7 +1,41 @@
-"""Function for gr's fn."""
-# pylint=disable=
+"""Align via ubee,"""
+# pylint: disable=
+from typing import Iterable, List, Tuple
+from itertools import zip_longest
 
+from logzero import logger
+from ubee.uclas import uclas
 
-def ubee():
-    """Gen a dummy."""
-    ...
+
+def ubee(
+    sents_zh: Iterable,
+    sents_en: Iterable,
+    thresh: float = 0.5,
+) -> Tuple[List[Tuple[str, str, float]], List[Tuple[str, str]]]:
+    """Align blocks.
+
+    Args:
+        sents_zh: list of text, can be any langauge supported by clas-l-user
+        sents_zh: ditto
+    Returns:
+        three tuples of aligned blocked
+        leftovers (unaligned)
+    """
+    res = []
+    labels = [*sents_en]
+
+    lo1 = []
+    lo2 = labels[:]
+
+    for seq in sents_zh:
+        label, likelihood = uclas(seq, labels, thresh=thresh)
+        if label:
+            res.append((seq, label, likelihood))
+            try:
+                lo2.remove(label)
+            except Exception as exc:
+                logger.error(exc)
+                logger.info("seq: %s, lable: %s", seq, label)
+        else:
+            lo1.append(seq)
+    return res, [*zip_longest(lo1, lo2)]

ubee/uclas.py

@@ -0,0 +1,81 @@
+"""Define uclas."""
+# pylint: disable=invalid-name
+
+from typing import List, Tuple, Union
+import numpy as np
+from sklearn.metrics.pairwise import cosine_similarity
+from joblib import Memory
+
+from model_pool import fetch_check_aux  # pylint: disable=import-error
+from model_pool.model_s import load_model_s  # pylint: disable=import-error
+from model_pool.load_model import load_model  # pylint: disable=import-error
+
+import logzero
+from logzero import logger
+
+logzero.loglevel(10)
+
+fetch_check_aux("/home/user")
+model_s = load_model_s()
+clas = load_model("clas-l-user")
+
+location = "./cachedir"
+memory = Memory(location, verbose=0)
+
+
+@memory.cache
+def cached_clas(*args, **kw):
+    """Cache clas-l-user."""
+    return clas(*args, **kw)
+
+
+# cached_clas = memory.cache(cached_clas)
+
+
+@memory.cache
+def encode(*args, **kw):
+    """Cache model_s.encode."""
+    return model_s.encode(*args, **kw)
+
+
+def uclas(
+    seq: str,
+    labels: Union[List[str], np.ndarray, Tuple[str, ...]],
+    thresh: float = 0.5,
+    multi_label: bool = False,
+) -> Tuple[str, Union[float, str]]:
+    """Classify seq with a filter.
+
+    if clas > thresh, return
+    if clas * csim > thresh return
+    if csim > thresh return
+    return ""
+    """
+    # _ = clas(seq, labels, multi_label=multi_label)
+    _ = cached_clas(seq, labels, multi_label=multi_label)
+
+    logger.debug("1 %s, %s", _.get("labels")[0], round(_.get("scores")[0], 2))
+
+    if _.get("scores")[0] > thresh:
+        return _.get("labels")[0], round(_.get("scores")[0], 2)
+
+    _ = dict(zip(_.get("labels"), _.get("scores")))
+
+    corr = np.array([_.get(elm) for elm in labels])
+
+    csim = cosine_similarity(encode([seq]), encode(labels))
+
+    corr = corr * csim
+
+    logger.debug("2 %s, %s", corr.argmax(), round(corr.max(), 2))
+
+    if corr.max() > thresh:
+        return labels[corr.argmax()], round(corr.max(), 2)
+
+    logger.debug("3 %s, %s, %s", csim.argmax(), round(csim.max(), 2), thresh / 2)
+
+    logger.debug("T or F: %s", csim.max() > (thresh / 2))
+    if csim.max() > (thresh / 2):
+        return labels[csim.argmax()], round(csim.max(), 2)
+
+    return "", ""