semvec/pypi-packages huggingface dataset:
parasol-nlp 0.0.4
Korean tokenizer with character decomposition
Requires Python: >=3.6
Keywords: hangul korean nlp tokenizer

Parasol Tokenizer
================

Parasol tokenizes hangul after decomposition.
한글 자음,모음을 분해하여 토큰화합니다.

* __Original text__ : 고가도로에 삐져나온 초록잎 아마 이 도시에서 유일히 적응 못한 낭만일 거야
* __Decomposed text__ : ㄱㅗㄱㅏㄷㅗㄹㅗㅇㅔ ㅃㅣㅈㅕㄴㅏㅇㅗㄴ ㅊㅗㄹㅗㄱㅇㅣㅍ ㅇㅏㅁㅏ ㅇㅣ ㄷㅗㅅㅣㅇㅔㅅㅓ ㅇㅠㅇㅣㄹㅎㅣ ㅈㅓㄱㅇㅡㅇ ㅁㅗㅅㅎㅏㄴ ㄴㅏㅇㅁㅏㄴㅇㅣㄹ ㄱㅓㅇㅑ
* __Tokens__ : ▁ㄱㅗㄱㅏ / ㄷㅗㄹㅗ / ㅇㅔ / ▁ㅃㅣ / ㅈㅕㄴ / ㅏㅇㅗㄴ / ▁ㅊ / ㅗㄹ / ㅗㄱ / ㅇㅣ / ㅍ / ▁ㅇㅏㅁㅏ / ▁ㅇㅣ / ▁ㄷㅗㅅㅣ / ㅇㅔㅅㅓ / ▁ㅇㅠㅇㅣㄹ / ㅎㅣ / ▁ㅈㅓㄱㅇㅡㅇ / ▁ㅁㅗㅅㅎㅏㄴ / ▁ㄴㅏㅇㅁㅏㄴ / ㅇㅣㄹ / ▁ㄱㅓㅇㅑ
* __Composed tokens__ : ▁고가 / 도로 / 에 / ▁삐 / 젼 / ㅏ온 / ▁ㅊ / ㅗㄹ / ㅗㄱ / 이 / ㅍ / ▁아마 / ▁이 / ▁도시 / 에서 / ▁유일 / 히 / ▁적응 / ▁못한 / ▁낭만 / 일 / ▁거야




## Installation

    pip install parasol-nlp

## Experiment

The figure shows the results of the perplexity comparison experiment. `with decomposition` is tokenized with charactor decomposition and `no decomposition` is just tokenized.
Experiment source code is [here](https://colab.research.google.com/drive/1aWNE53R9BO1wT4YilO9i1-hF5gIxxr5W).

![comparison_experiment_figure](https://user-images.githubusercontent.com/5267023/72793967-84d02200-3c7e-11ea-9ba5-ac8fd3a08031.png)

## Usage

### Tokenizer

Use [SentencePiece](https://github.com/google/sentencepiece)'s BPE model as tokenizer and [hgtk](https://github.com/bluedisk/hangul-toolkit) for decomposition.

```python
from parasol import Tokenizer

# tokenize after decomposition  
t1 = Tokenizer(decompose=True)
# tokenize without decomposition
t2 = Tokenizer(decompose=False)
```

then

```python
>>> t1.tokenize("고가도로에 삐져나온 초록잎 아마 이 도시에서 유일히 적응 못한 낭만일 거야")
['▁고가', '도로', '에', '▁삐', '젼', 'ㅏ온', '▁ㅊ', 'ㅗ록', '잎', '▁아마', '▁이', '▁도시', '에서', '▁유일', '히', '▁적응', '▁못한', '▁낭만', '일', '▁거야']
>>> t2.tokenize("고가도로에 삐져나온 초록잎 아마 이 도시에서 유일히 적응 못한 낭만일 거야")
['▁고가', '도로', '에', '▁삐', '져', '나온', '▁초록', '잎', '▁아마', '▁이', '▁도시', '에서', '▁유일', '히', '▁적응', '▁못한', '▁낭만', '일', '▁거야']

# Output as vocabulary id
>>> t1.tokenize("고가도로에 삐져나온 초록잎 아마 이 도시에서 유일히 적응 못한 낭만일 거야", as_id=True)
[17687, 2135, 36, 8351, 3904, 3842, 52, 12256, 27398, 3469, 30, 6105, 160, 3767, 198, 8953, 2345, 13164, 89, 6872]
```

### Composer

Hangul jamo composer

```python
from parasol import Composer

c = Composer()
```

then

```python
>>> c.compose("ㄷㅏㄹㅇㅣ ㄱㅣㅇㅜㄴ ㅂㅏㅁ ㅍㅓㄹㅓㄴㅂㅣㅊㅇㅣ ㅅㅡㅁㅕㄷㅡㄴ ㄱㅗㄹㅁㅗㄱㅇㅡㄹ ㄱㅓㄹㅇㅓㄱㅏㄷㅓㄴ ㄱㅣㄹㅇㅔ")
'달이 기운 밤 퍼런빛이 스며든 골목을 걸어가던 길에'
```

but it is not perfect, like..

```python
>>> c.compose("ㅎㅐㅇㅇㅜㄴㅇㅡㄹ ㅂㅣㄹㅇㅓㅇㅛㅎㅎ")
'행운을 빌어욯ㅎ'
```

which of original text is `행운을 빌어요ㅎㅎ`
nouns 0.1.1
Data-driven templating
Requires Python: >=3.8

# `nouns`: Data-deterministic, structure-driven (very experimental) templating

`nouns` does templating a bit differently - the templating itself is quite primitive, with most of the gruntwork taking place in the preprocessing layer.

This layer converts all the templating data into a very general form that ensures the data is renderable by the templates no matter the structure of the data.

## Hello world

Do traditional template-and-data templating as follows:

    >>> from nouns import template
    >>> template(dict(name='World'), 'Hello $name!')
    Hello World!

But the more interesting/intended use is to not explicitly pass in a template. Instead, just pass in the data - let the templating engine itself figure out which templates to use.

The in-built templates are designed to give you something close to what you probably want.

Pass in a table of data, you'll get a table back. Pass in a dictionary/mapping, you'll get the key/value pairs templated in a table. Pass in a list, you'll get each list item templated out.

## Experimental status

This templating system is an experiment in how to craft the most data-driven templating system, where the output is dependent on the things ("nouns") you pass in.

We will initially use if for rendering data in notebooks, but it's definitely not production-ready for web applications or otherwise rendering untrusted data, without a lot of extra customization and filters set up.

Hopefully it will continue to evolve toward production-ready status.