README.md

metadata

language:
  - ko
license: apache-2.0
library_name: transformers
tags:
  - text2text-generation
datasets:
  - aihub
metrics:
  - bleu
  - rouge
model-index:
  - name: ko-barTNumText
    results:
      - task:
          type: text2text-generation
          name: text2text-generation
        metrics:
          - type: bleu
            value: 0.9161441917016176
            name: eval_bleu
            verified: true
          - type: rouge1
            value: 0.9502159661745533
            name: eval_rouge1
            verified: true
          - type: rouge2
            value: 0.9313935147887745
            name: eval_rouge2
            verified: true
          - type: rougeL
            value: 0.950015374196916
            name: eval_rougeL
            verified: true
          - type: rougeLsum
            value: 0.9500390902948073
            name: eval_rougeLsum
            verified: true

ko-barTNumText(TNT Model🧨): Try Number To Korean Reading(숫자를 한글로 바꾸는 모델)

Table of Contents

• ko-barTNumText(TNT Model🧨): Try Number To Korean Reading(숫자를 한글로 바꾸는 모델)
  • Table of Contents
  • Model Details
  • Uses
  • Evaluation
  • How to Get Started With the Model

Model Details

• Model Description: 뭔가 찾아봐도 모델이나 알고리즘이 딱히 없어서 만들어본 모델입니다.
  BartForConditionalGeneration Fine-Tuning Model For Number To Korean
  BartForConditionalGeneration으로 파인튜닝한, 숫자를 한글로 변환하는 Task 입니다.
  

• Dataset use Korea aihub
  I can't open my fine-tuning datasets for my private issue
  데이터셋은 Korea aihub에서 받아서 사용하였으며, 파인튜닝에 사용된 모든 데이터를 사정상 공개해드릴 수는 없습니다.
  

• Korea aihub data is ONLY permit to Korean!!!!!!!
  aihub에서 데이터를 받으실 분은 한국인일 것이므로, 한글로만 작성합니다.
  정확히는 음성전사를 철자전사로 번역하는 형태로 학습된 모델입니다. (ETRI 전사기준)
  

• In case, ten million, some people use 10 million or some people use 10000000, so this model is crucial for training datasets 천만을 1000만 혹은 10000000으로 쓸 수도 있기에, Training Datasets에 따라 결과는 상이할 수 있습니다.
  

• Developed by: Yoo SungHyun(https://github.com/YooSungHyun)

• Language(s): Korean

• License: apache-2.0

• Parent Model: See the kobart-base-v2 for more information about the pre-trained base model.

Uses

This Model is inferenced token BACKWARD. so, you have to flip before tokenizer.decode() 해당 모델은 inference시 역순으로 예측합니다. (밥을 6시에 먹었어 -> 어 먹었 시에 여섯 을 밥)
때문에 tokenizer.decode를 수행하기 전에, flip으로 역순으로 치환해주세요.

Want see more detail follow this URL KoGPT_num_converter
and see bart_inference.py and bart_train.py

class BartText2TextGenerationPipeline(Text2TextGenerationPipeline):
    def postprocess(self, model_outputs, return_type=ReturnType.TEXT, clean_up_tokenization_spaces=False):
        records = []
        reversed_model_outputs = torch.flip(model_outputs["output_ids"][0], dims=[-1])
        for output_ids in reversed_model_outputs:
            if return_type == ReturnType.TENSORS:
                record = {f"{self.return_name}_token_ids": output_ids}
            elif return_type == ReturnType.TEXT:
                record = {
                    f"{self.return_name}_text": self.tokenizer.decode(
                        output_ids,
                        skip_special_tokens=True,
                        clean_up_tokenization_spaces=clean_up_tokenization_spaces,
                    )
                }
            records.append(record)
        return records

Evaluation

Just using evaluate-metric/bleu and evaluate-metric/rouge in huggingface evaluate library

How to Get Started With the Model

from transformers.pipelines import Text2TextGenerationPipeline
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
texts = ["그러게 누가 6시까지 술을 마시래?"]
tokenizer = AutoTokenizer.from_pretrained(
    inference_args.model_name_or_path,
)
model = AutoModelForSeq2SeqLM.from_pretrained(
    inference_args.model_name_or_path,
)
# BartText2TextGenerationPipeline is implemented above (see 'Use')
seq2seqlm_pipeline = BartText2TextGenerationPipeline(model=model, tokenizer=tokenizer)
kwargs = {
    "min_length": args.min_length,
    "max_length": args.max_length,
    "num_beams": args.beam_width,
    "do_sample": args.do_sample,
    "num_beam_groups": args.num_beam_groups,
}
pred = seq2seqlm_pipeline(texts, **kwargs)
print(pred)
# 그러게 누가 여섯 시까지 술을 마시래?