Update article in __main__, virtual_memory().free in smatrix

packages.txt

@@ -0,0 +1,3 @@
+icu-doc
+libicu-dev
+pkg-config

radiobee/__main__.py

@@ -355,7 +355,8 @@ if __name__ == "__main__":
     article = dedent(
         """
         ## NB
-        * `radiobee aligner` is a sibling of `bumblebee aligner`. To know more about these aligners, please join qq group `316287378`.
+        *   `radiobee aligner` is a sibling of `bumblebee aligner`. To know more about these aligners, please join qq group `316287378`.
+        *   Uploaded files should be in pure text format (txt, md, csv etc). `docx`, `pdf`, `srt`, `html` etc may be supported later on.
         *   Click "Clear" first for subsequent submits when uploading files.
         *   `tf_type` `idf_type` `dl_type` `norm`: Normally there is no need to touch these unless you know what you are doing.
         *   Suggested `esp` and `min_samples` values -- `esp` (minimum epsilon): 8-12, `min_samples`: 4-8.

radiobee/process_upload.py

@@ -29,6 +29,7 @@ def process_upload(upload: Union[tempfile._TemporaryFileWrapper, bytes]) -> str:
         ".txt",
         ".text",
         ".md",
+        "tsv",
     ]
     # check .txt .md ''(no suffix)
     if fpath.suffix.lower() not in suffixes:

radiobee/smatrix.py

@@ -89,11 +89,12 @@ def smatrix(
 
     # virtual_memory().available / 8: 64bits float
     require_ram = ilen(iter(doc1)) * ilen(iter(doc2)) * 8
-    if require_ram > virtual_memory().available:
-        logger.warning("virtual_memory().available: %s", virtual_memory().available)
+    if require_ram > virtual_memory().free:
+        # logger.warning("virtual_memory().free: %s", virtual_memory().available)
+        logger.warning("virtual_memory().free: %s", virtual_memory().free)
         logger.warning("memory required: %s", require_ram)
 
-    if require_ram > virtual_memory().available * 10:
+    if require_ram > virtual_memory().free * 10:
         logger.warning("You're likely to encounter memory problem, such as slowing down response and/or OOM.")
 
     # return dt1.doc(dt2.T)

test_en-test_zh.csv

@@ -0,0 +1,37 @@
+text1,text2,likelihood
+Wuthering Heights,呼啸山庄,
+--------------------------------------------------------------------------------,--------------------------------------------------------------------------------,1.0
+Chapter 2,第二章,
+Chinese,英文,0.13
+,昨天下午又冷又有雾。我想就在书房炉边消磨一下午，不想踩着杂草污泥到呼啸山庄了。,
+"Yesterday afternoon set in misty and cold. I had half a mind to spend it by my study fire, instead of wading through heath and mud to Wuthering Heights. On coming up from dinner, however (N.B. I dine between twelve and one o'clock; the housekeeper, a matronly lady, taken as a fixture along with the house, could not, or would not, comprehend my request that I might be served at five), on mounting the stairs with this lazy intention, and stepping into the room, I saw a servant girl on her knees surrounded by brushes and coal-scuttles, and raising an infernal dust as she extinguished the flames with heaps of cinders. This spectacle drove me back immediately; I took my hat, and, after a four-miles' walk, arrived at Heathcliff's garden gate just in time to escape the first feathery flakes of a snow shower.",但是，吃过午饭（注意——我在十二点与一点钟之间吃午饭，而可以当作这所房子的附属物的管家婆，一位慈祥的太太却不能，或者并不愿理解我请求在五点钟开饭的用意），在我怀着这个懒惰的想法上了楼，迈进屋子的时候，看见一个女仆跪在地上，身边是扫帚和煤斗。她正在用一堆堆煤渣封火，搞起一片弥漫的灰尘。这景象立刻把我赶回头了。我拿了帽子，走了四里路，到达了希刺克厉夫的花园口口，刚好躲过了一场今年初降的鹅毛大雪。,0.21
+"On that bleak hill top the earth was hard with a black frost, and the air made me shiver through every limb. Being unable to remove the chain, I jumped over, and, running up the flagged causeway bordered with straggling gooseberry bushes, knocked vainly for admittance, till my knuckles tingled and the dogs howled.",在那荒凉的山顶上，土地由于结了一层黑冰而冻得坚硬，冷空气使我四肢发抖。我弄不开门链，就跳进去，顺着两边种着蔓延的醋栗树丛的石路跑去。我白白地敲了半天门，一直敲到我的手指骨都痛了，狗也狂吠起来。,0.27
+"`Wretched inmates!' I ejaculated mentally, `you deserve perpetual isolation from your species for your churlish inhospitality. At least, I would not keep my doors barred in the day time. I don't care--I will get in!' So resolved, I grasped the latch and shook it vehemently. Vinegar-faced Joseph projected his head from a round window of the barn.",“倒霉的人家！”我心里直叫，“只为你这样无礼待客，就该一辈子跟人群隔离。我至少还不会在白天把门闩住。我才不管呢——我要进去！”如此决定了。我就抓住门闩，使劲摇它。苦脸的约瑟夫从谷仓的一个圆窗里探出头来。,0.23
+"`Whet are ye for?' he shouted. `T' maister's dahn i' t' fowld. Go rahnd by th' end ut' laith, if yah went tuh spake tull him.'",“你干吗？”他大叫。“主人在牛栏里，你要是找他说话，就从这条路口绕过去。”,0.12
+"`Is there nobody inside to open the door?' I hallooed, responsively.",“屋里没人开门吗？”我也叫起来。,0.1
+`They's nobbut t' missis; and shoo'll nut oppen't an ye mak yer flaysome dins till neeght.',“除了太太没有别人。你就是闹腾到夜里，她也不会开。”,
+"`Why? Cannot you tell her who I am, eh, Joseph?'",“为什么？你就不能告诉她我是谁吗，呃，约瑟夫？”,0.29
+"`Nor-ne me! Aw'll hae noa hend wi't,' muttered the head, vanishing.",“别找我！我才不管这些闲事呢，”这个脑袋咕噜着，又不见了。,0.19
+"The snow began to drive thickly. I seized the handle to essay another trial; when a young man without coat, and shouldering a pitchfork, appeared in the yard behind. He hailed me to follow him, and, after marching through a wash-house, and a paved area containing a coal shed, pump, and pigeon cot, we at length arrived in the huge, warm, cheerful apartment, where I was formerly received. It glowed delightfully in the radiance of an immense fire, compounded of coal, peat, and wood; and near the table, laid for a plentiful evening meal, I was pleased to observe the `missis', an individual whose existence I had never previously suspected. I bowed and waited, thinking she would bid me take a seat. She looked at me, leaning back in her chair, and remained motionless and mute.",雪开始下大了。我握住门柄又试一回。这时一个没穿外衣的年轻人，扛着一根草耙，在后面院子里出现了。他招呼我跟着他走，穿过了一个洗衣房和一片铺平的地，那儿有煤棚、抽水机和鸽笼，我们终于到了我上次被接待过的那间温暖的、热闹的大屋子。煤、炭和木材混合在一起燃起的熊熊炉火，使这屋子放着光彩。在准备摆上丰盛晚餐的桌旁，我很高兴地看到了那位“太太”，以前我从未料想到会有这么一个人存在的。我鞠躬等候，以为她会叫我坐下。她望望我，往她的椅背一靠，不动，也不出声。,0.23
+"`Rough weather!' I remarked. `I'm afraid, Mrs Heathcliff, the door must bear the consequence of your servants' leisure attendance: I had hard work to make them hear me.'",“天气真坏！”我说，“希刺克厉夫太太，恐怕大门因为您的仆人偷懒而大吃苦头，我费了好大劲才使他们听见我敲门！”,0.18
+"She never opened her mouth. I stared--she stared also: at any rate, she kept her eyes on me in a cool, regardless manner, exceedingly embarrassing and disagreeable.",她死不开口。我瞪眼——她也瞪眼。反正她总是以一种冷冷的、漠不关心的神气盯住我，使人十分窘，而且不愉快。,0.22
+"`Sit down,' said the young man gruffly. `He'll be in soon.'",“坐下吧，”那年轻人粗声粗气地说，“他就要来了。”,0.16
+"I obeyed; and hemmed, and called the villain Juno, who deigned, at this second interview, to move the extreme tip of her tail, in token of owning my acquaintance.",我服从了；轻轻咳了一下，叫唤那恶狗朱诺。临到第二次会面，它总算赏脸，摇起尾巴尖，表示认我是熟人了。,0.21
+"`A beautiful animal!' I commenced again. `Do you intend parting with the little ones, madam?'",“好漂亮的狗！”我又开始说话。“您是不是打算不要这些小的呢，夫人？”,0.21
+"`They are not mine,' said the amiable hostess, more repellingly than Heathcliff himself could have replied.",“那些不是我的，”这可爱可亲的女主人说，比希刺克厉夫本人所能回答的腔调还要更冷淡些。,0.17
+"`Ah, your favourites are among these?' I continued, turning to an obscure cushion full of something like cats.",“啊，您所心爱的是在这一堆里啦！”我转身指着一个看不清楚的靠垫上那一堆像猫似的东西，接着说下去。,0.18
+`A strange choice of favourites!' she observed scornfully.,“谁会爱这些东西那才怪呢！”她轻蔑地说。,0.21
+"Unluckily, it was a heap of dead rabbits. I hemmed once more, and drew closer to the hearth, repeating my comment on the wildness of the evening.",倒霉，原来那是堆死兔子。我又轻咳一声，向火炉凑近些，又把今晚天气不好的话评论一通。,0.2
+"`You should not have come out,' she said, rising and reaching from the chimney-piece two of the painted canisters.",“你本来就不该出来。”她说，站起来去拿壁炉台上的两个彩色茶叶罐。,
+"Her position before was sheltered from the light; now, I had a distinct view of her whole figure and countenance. She was slender, and apparently scarcely past girlhood: an admirable form, and the most exquisite little face that I have ever had the pleasure of beholding; small features, very fair; flaxen ringlets, or rather golden, hanging loose on her delicate neck; and eyes, had they been agreeable in expression, they would have been irresistible: fortunately for my susceptible heart, the only sentiment they evinced hovered between scorn, and a kind of desperation, singularly unnatural to be detected there. The canisters were almost out of her reach; I made a motion to aid her; she turned upon me as a miser might turn if anyone attempted to assist him in counting his gold.",她原先坐在光线被遮住的地方，现在我把她的全身和面貌都看得清清楚楚。她苗条，显然还没有过青春期。挺好看的体态，还有一张我生平从未有幸见过的绝妙的小脸蛋。五官纤丽，非常漂亮。淡黄色的卷发，或者不如说是金黄色的，松松地垂在她那细嫩的颈上。至于眼睛，要是眼神能显得和悦些，就要使人无法抗拒了。对我这容易动情的心说来倒是常事，因为它们所表现的只是在轻蔑与近似绝望之间的一种情绪，而在那张脸上看见那样的眼神是特别不自然的。,0.25
+"`I don't want your help,' she snapped; `I can get them for myself.'",她简直够不到茶叶罐。我动了一动，想帮她一下。她猛地扭转身向我，像守财奴看见别人打算帮他数他的金子一样。,0.16
+,“我不要你帮忙，”她怒气冲冲地说，“我自己拿得到。”,
+`I beg your pardon!' I hastened to reply.,“对不起！”我连忙回答。,0.18
+"`Were you asked to tea?' she demanded, tying an apron over her neat black frock, and standing with a spoonful of the leaf poised over the pot.",“是请你来吃茶的吗？”她问，把一条围裙系在她那干净的黑衣服上，就这样站着，拿一匙茶叶正要往茶壶里倒。,0.26
+"`I shall be glad to have a cup,' I answered.",“我很想喝杯茶。”我回答。,0.09
+`Were you asked?' she repeated.,“是请你来的吗？”她又问。,0.33
+"`No,' I said, half smiling. `You are the proper person to ask me.'",“没有，”我说，勉强笑一笑。“您正好请我喝茶。”,0.09
+Contents PreviousChapter,目录,0.16
+NextChapter,上一章,
+,下一章,
+Homepage,返回首页,0.49