semvec/pypi-packages huggingface dataset:
vietokenizer 1.0.3
Vietnamese Tokenizer package based on deep learning method
Requires Python: >=3.7
Keywords: vietokenizer

# VieTokenizer

This model architecture that we use is a simple bi-lstm network trained by unsupervised learning on a large pre-segmented dataset. The model will predict if serial word is 1 and non-serial is 0, for example, "Tôi tên là Nguyễn Tiến Đạt" will be equivalent to a sequence of numbers with both zero and one being [0, 0, 0, 0, 1, 1]. 

## Installation 🎉
- This repository is tested on python 3.7+ and Tensorflow 2.8+
- VieTokenizer can be installed using pip as follows:
```
pip install vietokenizer 🍰
```
- VieTokenizer can also be installed from source with the following commands: 
```
git clone https://github.com/Nguyendat-bit/VieTokenizer
cd VieTokenizer
pip install -e . 
```
## Usage 🔥
```python
>>> import vietokenizer
>>> tokenizer= vietokenizer.vntokenizer()
>>> tokenizer('Tôi tên là Nguyễn Tiến Đạt, hiện là sinh viên Đại học CN GTVT tại Hà Nội.')
'Tôi tên là Nguyễn_Tiến_Đạt , hiện là sinh_viên Đại_học CN GTVT tại Hà_Nội .'
>>> tokenizer('Kim loại nặng thường được định nghĩa là kim loại có khối lượng riêng, khối lượng nguyên tử hoặc số hiệu nguyên tử lớn.')
'Kim_loại nặng thường được định_nghĩa là kim_loại có khối_lượng riêng , khối_lượng nguyên_tử hoặc số_hiệu nguyên_tử lớn .'
```

## License
[Apache 2.0 License](https://github.com/Nguyendat-bit/VieTokenizer). <br>
Copyright &copy; 2022 [Nguyễn Tiến Đạt](https://github.com/Nguyendat-bit). All rights reserved.
modelbox 0.0.5
Client Library of the ModelBox Service
Requires Python: >=3.7

# ModelBox Python API

This package contains the client library for the ModelBox API for managing Deep Learning models, checkpoints from experiments, and other model operations related services.

## Concepts and Understanding the ModelBox API

### Namespace
A Namespace is a mechanism to organize related models or models published by a team. They are also use for access control and such to the metadata of uploaded models, invoking benchmarks or other model transformation work. Namespaces are automatically created when a new model or experieemnt specifies the namespace it wants to be associated with.

### Model
A model is an object to track common metadata, and to apply policies on models created by experiments to solve a machine learning task. For ex. datasets to evaluate all trained models of a task can be tracked using this object. Users can also add rules around retention policies of trained versions, setup policies for labelling a trained model if it has better metrics on a dataset, and meets all other criterion.

### Model Version
A model version is a trained model, it includes the model binary, related files that a user wants to track - dataset file handles, any other metadata, model metrics, etc. Model versions are always related to a Model and all the policies created for a Model are applied to Model Versions.

### Experiment and Checkpoints
Experiments are used to injest model checkpoints created during a training run. ModelBox is not an experiment metadata tracker so there is no support for rich experiment management which are available on experiment trackers such as Weights and Biases, the experiment abstraction here exists so that we can track and injest model checkpoints which eventually become model versions if they have good metrics and does well in benchmarks.

## Example

```
from modelbox import ModelBoxClient, MLFramework

client = ModelBoxClient(SERVER_ADDR)

model = self._client.create_model(
            "yolo",
            "owner@email.com",
            "ai/vision/",
            "object_detection",
            "yolo_des",
            {"meta": "foo"},
        )
model_version = self._client.create_model_version(
            model.id, 
            "yolo4_v1",
            "v1",
            "A Yolo v4 trained with custom dataset", 
            ["s3://path/to/bucket/model.pt],
            {"model_hyperparam_1": "value"},
            MLFramework.PYTORCH,
        )

client.close()
```


## Local Development and Installation
The modelbox client library can be installed locally in the following way -
```
cd <project-root>/client-py/
pip install .
```
This installs the version of the client checked out with the repo.

Build the client and create distribution packages
```
cd <project-root>/client-py/
python -m build .
```

Run Tests 
```
cd <project-root>/client-py/
python tests/test_modelbox_api.py
```