semvec/pypi-packages huggingface dataset:
sister 0.2.2
SISTER (SImple SenTence EmbeddeR)
Requires Python: >=3.8,<4.0

# sister
SISTER (**SI**mple **S**en**T**ence **E**mbedde**R**)


# Installation

```bash
pip install sister
```


# Basic Usage
```python
import sister
sentence_embedding = sister.MeanEmbedding(lang="en")

sentence = "I am a dog."
vector = sentence_embedding(sentence)
```


# Supported languages.

- English
- Japanese
- French

In order to support a new language, please implement `Tokenizer` (inheriting `sister.tokenizers.Tokenizer`) and add fastText
pre-trained url to `word_embedders.get_fasttext()` ([List of model urls](https://github.com/facebookresearch/fastText/blob/master/docs/pretrained-vectors.md)).


# Bert models are supported for en, fr, ja (2020-06-29).
Actually Albert for English, CamemBERT for French and BERT for Japanese.  
To use BERT, you need to install sister by `pip install 'sister[bert]'`.

```python
import sister
bert_embedding = sister.BertEmbedding(lang="en")

sentence = "I am a dog."
vector = bert_embedding(sentence)
```

You can also give multiple sentences to it (more efficient).

```python
import sister
bert_embedding = sister.BertEmbedding(lang="en")

sentences = ["I am a dog.", "I want be a cat."]
vectors = bert_embedding(sentences)
```
CwnSenseTagger 0.1.6
A package to use chinese word net to achieve word sense disambigution task
Requires Python: >=3.5

# Word Sense Disambiguaion by Chinese Word Net

Chinese word sense disambiguation has been known to a very difficult problem since Chinese is a complicated language. A word can have dozens or even hundreds of meanings on different occasions. Manually labels the senses of the words is labor-intensive and inefficient.

In this project, we aim to solve this problem by state-of-the-art Bert model.  It gives us huge performance gains and can score roughly 82% accuracy on Chinese word sense disambiguation problem.

## Prerequest
* Input should be tokenized first. POS Tagging is preferred but not required.
* Suppose we have m sentences and each sentence has $n_m$ words.
    * list_of_sentence[ [list_of_word[[target, pos, sense_id, sense] * $n_m$ ] *m ]
    * The following is an example that has 2 sentences, input data should be formed as following

            [[["他","Nh","",""],["由","P","",""],["昏沈","VH","",""],["的","DE","",""],["睡夢","Na","",""],["中","Ng","",""],["醒來","VH","",""],["，","COMMACATEGORY","",""]],
             [["臉","Na","",""],["上","Ncd","",""],["濕涼","VH","",""],["的","DE","",""],["騷動","Nv","",""],["是","SHI","",""],["淚","Na","",""],["。","PERIODCATEGORY","",""]]]
             

## How to get sense
* At Project root directory (same as setup.py)

        pip3 install .
        import CWN_WSD
        data = read_somewhere() #list of sentence, and sentence is composed as list of word
        sense = CWN_WSD.wsd(data)
* example can be found under example folder

## Acknowledgement
We thank Po-Wen Chen (b05902117@ntu.edu.tw) and Yu-Yu Wu (b06902104@ntu.edu.tw) for contributions in model development.