| Training Loss | Epoch | Step | Validation Loss | Accuracy | F1     |
|:-------------:|:-----:|:----:|:---------------:|:--------:|:------:|
| No log        | 1.0   | 106  | 1.2526          | 0.6108   | 0.1508 |
| No log        | 2.0   | 212  | 1.1553          | 0.6648   | 0.1141 |
| No log        | 3.0   | 318  | 1.1150          | 0.6591   | 0.1247 |
| No log        | 4.0   | 424  | 1.0007          | 0.6705   | 0.1383 |
| 1.1323        | 5.0   | 530  | 0.9267          | 0.6733   | 0.2027 |
| 1.1323        | 6.0   | 636  | 1.0869          | 0.6335   | 0.4084 |
| 1.1323        | 7.0   | 742  | 1.1224          | 0.6932   | 0.4586 |
| 1.1323        | 8.0   | 848  | 1.2535          | 0.6307   | 0.3424 |
| 1.1323        | 9.0   | 954  | 1.4288          | 0.6932   | 0.4881 |
| 0.5252        | 10.0  | 1060 | 1.5856          | 0.6932   | 0.4739 |
| 0.5252        | 11.0  | 1166 | 1.7101          | 0.6733   | 0.4530 |
| 0.5252        | 12.0  | 1272 | 1.7330          | 0.6903   | 0.4750 |
| 0.5252        | 13.0  | 1378 | 1.8872          | 0.7273   | 0.5125 |
| 0.5252        | 14.0  | 1484 | 1.8797          | 0.7301   | 0.5033 |
| 0.1252        | 15.0  | 1590 | 1.9339          | 0.7330   | 0.5024 |
| 0.1252        | 16.0  | 1696 | 1.9632          | 0.7301   | 0.4967 |


### Framework versions


- Transformers 4.11.2
- Pytorch 1.9.0+cu102
- Datasets 1.12.1
- Tokenizers 0.10.3


model name: xlm-roberta-large-finetuned-sent_in_news


model author: chrommium


task: text-classification


tags: pytorch, tensorboard, xlm-roberta, text-classification, transformers, license:mit, generated_from_trainer, model-index


tags:


- generated_from_trainer
metrics:
- accuracy
- f1
model-index:
- name: rubert-base-cased-sentence-finetuned-sent_in_news_sents
results:
  - task:
name: Text Classification
type: text-classification
metrics:
    - name: Accuracy
type: accuracy
value: 0.7224199288256228
    - name: F1
type: f1
value: 0.5137303178348194


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# rubert-base-cased-sentence-finetuned-sent_in_news_sents


This model is a fine-tuned version of [DeepPavlov/rubert-base-cased-sentence](https://huggingface.co/DeepPavlov/rubert-base-cased-sentence) on an unknown dataset.
It achieves the following results on the evaluation set:


- Loss: 1.9506
- Accuracy: 0.7224
- F1: 0.5137


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 5e-05
- train_batch_size: 14
- eval_batch_size: 14
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 20


### Training results


| Training Loss | Epoch | Step | Validation Loss | Accuracy | F1     |
|:-------------:|:-----:|:----:|:---------------:|:--------:|:------:|
| No log        | 1.0   | 81   | 1.0045          | 0.6690   | 0.1388 |
| No log        | 2.0   | 162  | 0.9574          | 0.6228   | 0.2980 |
| No log        | 3.0   | 243  | 1.0259          | 0.6477   | 0.3208 |
| No log        | 4.0   | 324  | 1.1262          | 0.6619   | 0.4033 |
| No log        | 5.0   | 405  | 1.3377          | 0.6299   | 0.3909 |
| No log        | 6.0   | 486  | 1.5716          | 0.6868   | 0.3624 |
| 0.6085        | 7.0   | 567  | 1.6286          | 0.6762   | 0.4130 |
| 0.6085        | 8.0   | 648  | 1.6450          | 0.6940   | 0.4775 |
| 0.6085        | 9.0   | 729  | 1.7108          | 0.7224   | 0.4920 |
| 0.6085        | 10.0  | 810  | 1.8792          | 0.7046   | 0.5028 |
| 0.6085        | 11.0  | 891  | 1.8670          | 0.7153   | 0.4992 |
| 0.6085        | 12.0  | 972  | 1.8856          | 0.7153   | 0.4934 |
| 0.0922        | 13.0  | 1053 | 1.9506          | 0.7224   | 0.5137 |
| 0.0922        | 14.0  | 1134 | 2.0363          | 0.7189   | 0.4761 |
| 0.0922        | 15.0  | 1215 | 2.0601          | 0.7224   | 0.5053 |
| 0.0922        | 16.0  | 1296 | 2.0813          | 0.7153   | 0.5038 |
| 0.0922        | 17.0  | 1377 | 2.0960          | 0.7189   | 0.5065 |
| 0.0922        | 18.0  | 1458 | 2.1060          | 0.7224   | 0.5098 |
| 0.0101        | 19.0  | 1539 | 2.1153          | 0.7260   | 0.5086 |
| 0.0101        | 20.0  | 1620 | 2.1187          | 0.7260   | 0.5086 |


### Framework versions


- Transformers 4.10.3
- Pytorch 1.9.0+cu102
- Datasets 1.12.1
- Tokenizers 0.10.3


model name: rubert-base-cased-sentence-finetuned-sent_in_news_sents


model author: chrommium


task: text-classification


tags: pytorch, tensorboard, bert, text-classification, transformers, generated_from_trainer, model-index, infinity_compatible


license: apache-2.0
tags:


- generated_from_trainer
metrics:
- accuracy
model_index:
- name: bert-base-multilingual-cased-finetuned-cola
results:
  - task:
name: Text Classification
type: text-classification
metric:
name: Accuracy
type: accuracy
value: 0.9755


# bert-base-multilingual-cased-finetuned-cola


This model is a fine-tuned version of [bert-base-multilingual-cased](https://huggingface.co/bert-base-multilingual-cased) on an unkown dataset.
It achieves the following results on the evaluation set:


- Loss: 0.1729
- Accuracy: 0.9755


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


- learning_rate: 2e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 5


### Training results


| Training Loss | Epoch | Step | Validation Loss | Accuracy |
|:-------------:|:-----:|:----:|:---------------:|:--------:|
| 0.5119        | 1.0   | 625  | 0.2386          | 0.922    |
| 0.2536        | 2.0   | 1250 | 0.2055          | 0.949    |
| 0.1718        | 3.0   | 1875 | 0.1733          | 0.969    |
| 0.0562        | 4.0   | 2500 | 0.1661          | 0.974    |
| 0.0265        | 5.0   | 3125 | 0.1729          | 0.9755   |


### Framework versions


- Transformers 4.9.2
- Pytorch 1.9.0+cu102
- Datasets 1.11.0
- Tokenizers 0.10.3


model name: bert-base-multilingual-cased-finetuned-news-headlines


model author: chrommium


task: text-classification


tags: pytorch, tensorboard, bert, text-classification, transformers, license:apache-2.0, generated_from_trainer, infinity_compatible


tags:


- generated_from_trainer
metrics:
- accuracy
- f1
model-index:
- name: sbert_large-finetuned-sent_in_news_sents
results: []


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# sbert_large-finetuned-sent_in_news_sents


This model is a fine-tuned version of [sberbank-ai/sbert_large_nlu_ru](https://huggingface.co/sberbank-ai/sbert_large_nlu_ru) on an unknown dataset.
It achieves the following results on the evaluation set:


- Loss: 1.7056
- Accuracy: 0.7301
- F1: 0.5210


## Model examples


Model responds to label X in news text. For exaple:


For 'Газпром отозвал лицензию у X, сообщает Финам' the model will return negative label -3


For 'X отозвал лицензию у Сбербанка, сообщает Финам' the model will return neutral label 0


For 'Газпром отозвал лицензию у Сбербанка, сообщает X' the model will return neutral label 0


For 'X демонстрирует высокую прибыль, сообщает Финам' the model will return positive label 1


## Simple example of News preprocessing for Russian before BERT


```
from natasha import (
    Segmenter,
    MorphVocab,
    NewsEmbedding,
    NewsMorphTagger,
    NewsSyntaxParser,
    NewsNERTagger,
    PER,
    NamesExtractor,
    Doc
)
segmenter = Segmenter()
emb = NewsEmbedding()
morph_tagger = NewsMorphTagger(emb)
syntax_parser = NewsSyntaxParser(emb)
morph_vocab = MorphVocab()

### ----------------------------- key sentences block -----------------------------

def find_synax_tokens_with_order(doc, start, tokens, text_arr, full_str):
    ''' Находит все синтаксические токены, соответствующие заданному набору простых токенов (найденные
        для определенной NER другими функциями).
        Возвращает словарь найденных синтаксических токенов (ключ - идентификатор токена, состоящий
        из номера предложения и номера токена внутри предложения).
        Начинает поиск с указанной позиции в списке синтаксических токенов, дополнительно возвращает
        позицию остановки, с которой нужно продолжить поиск следующей NER.
    '''
    found = []
    in_str = False
    str_candidate = ''
    str_counter = 0
    if len(text_arr) == 0:
        return [], start
    for i in range(start, len(doc.syntax.tokens)):
        t = doc.syntax.tokens[i]
        if in_str:
            str_counter += 1
            if str_counter < len(text_arr) and t.text == text_arr[str_counter]:
                str_candidate += t.text
                found.append(t)
                if str_candidate == full_str:
                    return found, i+1
            else:
                in_str = False
                str_candidate = ''
                str_counter = 0
                found = []
        if t.text == text_arr[0]:
            found.append(t)
            str_candidate = t.text
            if str_candidate == full_str:
                return found, i+1
            in_str = True
    return [], len(doc.syntax.tokens)

def find_tokens_in_diap_with_order(doc, start_token, diap):
    ''' Находит все простые токены (без синтаксической информации), которые попадают в
        указанный диапазон. Эти диапазоны мы получаем из разметки NER.
        Возвращает набор найденных токенов и в виде массива токенов, и в виде массива строчек.
        Начинает поиск с указанной позиции в строке и дополнительно возвращает позицию остановки.
    '''
    found_tokens = []
    found_text = []
    full_str = ''
    next_i = 0
    for i in range(start_token, len(doc.tokens)):
        t = doc.tokens[i]
        if t.start > diap[-1]:
            next_i = i
            break
        if t.start in diap:
            found_tokens.append(t)
            found_text.append(t.text)
            full_str += t.text
    return found_tokens, found_text, full_str, next_i

def add_found_arr_to_dict(found, dict_dest):
    for synt in found:
        dict_dest.update({synt.id: synt})
    return dict_dest

def make_all_syntax_dict(doc):
    all_syntax = {}
    for synt in doc.syntax.tokens:
        all_syntax.update({synt.id: synt})
    return all_syntax

def is_consiquent(id_1, id_2):
    ''' Проверяет идут ли токены друг за другом без промежутка по ключам. '''
    id_1_list = id_1.split('_')
    id_2_list = id_2.split('_')
    if id_1_list[0] != id_2_list[0]:
        return False
    return int(id_1_list[1]) + 1 == int(id_2_list[1])

def replace_found_to(found, x_str):
    ''' Заменяет последовательность токенов NER на «заглушку». '''
    prev_id = '0_0'
    for synt in found:
        if is_consiquent(prev_id, synt.id):
            synt.text = ''
        else:
            synt.text = x_str
        prev_id = synt.id

def analyze_doc(text):
    ''' Запускает Natasha для анализа документа. '''
    doc = Doc(text)
    doc.segment(segmenter)
    doc.tag_morph(morph_tagger)
    doc.parse_syntax(syntax_parser)
    ner_tagger = NewsNERTagger(emb)
    doc.tag_ner(ner_tagger)
    return doc

def find_non_sym_syntax_short(entity_name, doc, add_X=False, x_str='X'):
    ''' Отыскивает заданную сущность в тексте, среди всех NER (возможно, в другой грамматической форме).

        entity_name - сущность, которую ищем;
        doc - документ, в котором сделан препроцессинг Natasha;
        add_X - сделать ли замену сущности на «заглушку»;
        x_str - текст замены.

        Возвращает:
        all_found_syntax - словарь всех подходящих токенов образующих искомые сущности, в котором
        в случае надобности произведена замена NER на «заглушку»;
        all_syntax - словарь всех токенов.
    '''
    all_found_syntax = {}
    current_synt_number = 0
    current_tok_number = 0

    # идем по всем найденным NER
    for span in doc.spans:
        span.normalize(morph_vocab)
        if span.type != 'ORG':
            continue
        diap = range(span.start, span.stop)
        # создаем словарь всех синтаксических элементов (ключ -- id из номера предложения и номера внутри предложения)
        all_syntax = make_all_syntax_dict(doc)
        # находим все простые токены внутри NER
        found_tokens, found_text, full_str, current_tok_number = find_tokens_in_diap_with_order(doc, current_tok_number,
                                                                                diap)
        # по найденным простым токенам находим все синтаксические токены внутри данного NER
        found, current_synt_number = find_synax_tokens_with_order(doc, current_synt_number, found_tokens, found_text,
                                                                  full_str)
        # если текст NER совпадает с указанной сущностью, то делаем замену
        if entity_name.find(span.normal) >= 0 or span.normal.find(entity_name) >= 0:
            if add_X:
                replace_found_to(found, x_str)
            all_found_syntax = add_found_arr_to_dict(found, all_found_syntax)
    return all_found_syntax, all_syntax

def key_sentences(all_found_syntax):
    ''' Находит номера предложений с искомой NER. '''
    key_sent_numb = {}
    for synt in all_found_syntax.keys():
        key_sent_numb.update({synt.split('_')[0]: 1})
    return key_sent_numb

def openinig_punct(x):
    opennings = ['«', '(']
    return x in opennings

def key_sentences_str(entitiy_name, doc, add_X=False, x_str='X', return_all=True):
    ''' Составляет окончательный текст, в котором есть только предложения, где есть ключевая сущность,
        эта сущность, если указано, заменяется на «заглушку».
    '''
    all_found_syntax, all_syntax = find_non_sym_syntax_short(entitiy_name, doc, add_X, x_str)
    key_sent_numb = key_sentences(all_found_syntax)
    str_ret = ''

    for s in all_syntax.keys():
        if (s.split('_')[0] in key_sent_numb.keys()) or (return_all):
            to_add = all_syntax[s]

            if s in all_found_syntax.keys():
                to_add = all_found_syntax[s]
            else:
                if to_add.rel == 'punct' and not openinig_punct(to_add.text):
                    str_ret = str_ret.rstrip()

            str_ret += to_add.text
            if (not openinig_punct(to_add.text)) and (to_add.text != ''):
                str_ret += ' '

    return str_ret

### ----------------------------- key entities block -----------------------------

def find_synt(doc, synt_id):
    for synt in doc.syntax.tokens:
        if synt.id == synt_id:
            return synt
    return None

def is_subj(doc, synt, recursion_list=[]):
    ''' Сообщает является ли слово подлежащим или частью сложного подлежащего. '''
    if synt.rel == 'nsubj':
        return True
    if synt.rel == 'appos':
        found_head = find_synt(doc, synt.head_id)
        if found_head.id in recursion_list:
            return False
        return is_subj(doc, found_head, recursion_list + [synt.id])
    return False

def find_subjects_in_syntax(doc):
    ''' Выдает словарик, в котором для каждой NER написано, является ли он
        подлежащим в предложении.
        Выдает стартовую позицию NER и было ли оно подлежащим (или appos)
    '''
    found_subjects = {}
    current_synt_number = 0
    current_tok_number = 0

    for span in doc.spans:
        span.normalize(morph_vocab)
        if span.type != 'ORG':
            continue

        found_subjects.update({span.start: 0})
        diap = range(span.start, span.stop)

        found_tokens, found_text, full_str, current_tok_number = find_tokens_in_diap_with_order(doc,
                                                                                current_tok_number,
                                                                                diap)

        found, current_synt_number = find_synax_tokens_with_order(doc, current_synt_number, found_tokens,
                                                                  found_text, full_str)

        found_subjects.update({span.start: 0})
        for synt in found:
            if is_subj(doc, synt):
                found_subjects.update({span.start: 1})
    return found_subjects

def entity_weight(lst, c=1):
    return c*lst[0]+lst[1]

def determine_subject(found_subjects, doc, new_agency_list, return_best=True, threshold=0.75):
    ''' Определяет ключевую NER и список самых важных NER, основываясь на том, сколько
        раз каждая из них встречается в текста вообще и сколько раз в роли подлежащего '''
    objects_arr = []
    objects_arr_ners = []
    should_continue = False
    for span in doc.spans:
        should_continue = False
        span.normalize(morph_vocab)
        if span.type != 'ORG':
            continue
        if span.normal in new_agency_list:
            continue
        for i in range(len(objects_arr)):
            t, lst = objects_arr[i]

            if t.find(span.normal) >= 0:
                lst[0] += 1
                lst[1] += found_subjects[span.start]
                should_continue = True
                break

            if span.normal.find(t) >= 0:
                objects_arr[i] = (span.normal, [lst[0]+1, lst[1]+found_subjects[span.start]])
                should_continue = True
                break

        if should_continue:
            continue
        objects_arr.append((span.normal, [1, found_subjects[span.start]]))
        objects_arr_ners.append(span.normal)

    max_weight = 0
    opt_ent = 0
    for obj in objects_arr:
        t, lst = obj
        w = entity_weight(lst)
        if max_weight < w:
            max_weight = w
            opt_ent = t

    if not return_best:
        return opt_ent, objects_arr_ners

    bests = []
    for obj in objects_arr:
        t, lst = obj
        w = entity_weight(lst)
        if max_weight*threshold < w:
            bests.append(t)

    return opt_ent, bests

text = '''В офисах Сбера начали тестировать технологию помощи посетителям в экстренных ситуациях. «Зеленая кнопка» будет
 в зонах круглосуточного обслуживания офисов банка в Воронеже, Санкт-Петербурге, Подольске, Пскове, Орле и Ярославле.
 В них находятся стенды с сенсорными кнопками, обеспечивающие связь с операторами центра мониторинга службы безопасности
 банка. Получив сигнал о помощи, оператор центра может подключиться к объекту по голосовой связи. С помощью камер
 видеонаблюдения он оценит обстановку и при необходимости вызовет полицию или скорую помощь. «Зеленой кнопкой» можно
 воспользоваться в нерабочее для отделения время, если возникла угроза жизни или здоровью. В остальных случаях помочь
 клиентам готовы сотрудники отделения банка. «Одно из направлений нашей работы в области ESG и устойчивого развития
 — это забота об обществе. И здоровье людей как высшая ценность является его основой. Поэтому задача банка в области
 безопасности гораздо масштабнее, чем обеспечение только финансовой безопасности клиентов. Этот пилотный проект
 приурочен к 180-летию Сбербанка: мы хотим, чтобы, приходя в банк, клиент чувствовал, что его жизнь и безопасность
 — наша ценность», — отметил заместитель председателя правления Сбербанка Станислав Кузнецов.'''

doc = analyze_doc(text)
key_entity = determine_subject(find_subjects_in_syntax(doc), doc, [])[0]
text_for_model = key_sentences_str(key_entity, doc, add_X=True, x_str='X', return_all=False)
```


## Intended uses & limitations


More information needed


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 2e-05
- train_batch_size: 6
- eval_batch_size: 6
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 20


### Training results


| Training Loss | Epoch | Step | Validation Loss | Accuracy | F1     |
|:-------------:|:-----:|:----:|:---------------:|:--------:|:------:|
| No log        | 1.0   | 176  | 0.9504          | 0.6903   | 0.2215 |
| No log        | 2.0   | 352  | 0.9065          | 0.7159   | 0.4760 |
| 0.8448        | 3.0   | 528  | 0.9687          | 0.7045   | 0.4774 |
| 0.8448        | 4.0   | 704  | 1.2436          | 0.7045   | 0.4686 |
| 0.8448        | 5.0   | 880  | 1.4809          | 0.7273   | 0.4630 |
| 0.2074        | 6.0   | 1056 | 1.5866          | 0.7330   | 0.5185 |
| 0.2074        | 7.0   | 1232 | 1.7056          | 0.7301   | 0.5210 |
| 0.2074        | 8.0   | 1408 | 1.6982          | 0.7415   | 0.5056 |
| 0.0514        | 9.0   | 1584 | 1.8088          | 0.7273   | 0.5203 |
| 0.0514        | 10.0  | 1760 | 1.9250          | 0.7102   | 0.4879 |


### Framework versions


- Transformers 4.11.2
- Pytorch 1.9.0+cu102
- Datasets 1.12.1
- Tokenizers 0.10.3


model name: sbert_large-finetuned-sent_in_news_sents


model author: chrommium


task: text-classification


tags: pytorch, tensorboard, bert, text-classification, transformers, generated_from_trainer, model-index, infinity_compatible


language:


- ru
tags:
- PyTorch
- Transformers
license: mit


# Medical History Model based on ruGPT2 by @sberbank-ai


A simple model for helping medical staff to complete patient's medical histories.
Model used pretrained [sberbank-ai/rugpt3small_based_on_gpt2](https://huggingface.co/sberbank-ai/rugpt3small_based_on_gpt2)


model name: ru_med_gpt3sm_based_on_gpt2


model author: anechaev


task: text-generation


tags: pytorch, gpt2, text-generation, ru, transformers, PyTorch, Transformers, license:mit


license: apache-2.0
tags:


- generated_from_trainer
model-index:
- name: wav2vec2-large-xlsr-finetune-spanish-col
results: []


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# wav2vec2-large-xlsr-finetune-spanish-col


This model is a fine-tuned version of [jonatasgrosman/wav2vec2-large-xlsr-53-spanish](https://huggingface.co/jonatasgrosman/wav2vec2-large-xlsr-53-spanish) on the None dataset.
It achieves the following results on the evaluation set:


- Loss: 2.7105
- Wer: 0.9824


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


- learning_rate: 0.0003
- train_batch_size: 16
- eval_batch_size: 8
- seed: 42
- gradient_accumulation_steps: 2
- total_train_batch_size: 32
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 500
- num_epochs: 30
- mixed_precision_training: Native AMP


### Training results


| Training Loss | Epoch | Step | Validation Loss | Wer    |
|:-------------:|:-----:|:----:|:---------------:|:------:|
| 7.2829        | 3.25  | 400  | 2.9632          | 1.0    |
| 2.9664        | 6.5   | 800  | 2.8494          | 1.0542 |
| 2.8353        | 9.76  | 1200 | 2.8352          | 1.0101 |
| 2.7863        | 13.01 | 1600 | 2.7421          | 0.9837 |
| 2.762         | 16.26 | 2000 | 2.7254          | 0.9861 |
| 2.7483        | 19.51 | 2400 | 2.7228          | 0.9874 |
| 2.7482        | 22.76 | 2800 | 2.7228          | 0.9999 |
| 2.7373        | 26.02 | 3200 | 2.7163          | 0.9824 |
| 2.7328        | 29.27 | 3600 | 2.7105          | 0.9824 |


### Framework versions


- Transformers 4.11.3
- Pytorch 1.10.1+cu102
- Datasets 1.13.3
- Tokenizers 0.10.3


model name: wav2vec2-large-xlsr-finetune-spanish-col


model author: Santiagot1105


task: automatic-speech-recognition


tags: pytorch, tensorboard, wav2vec2, automatic-speech-recognition, transformers, license:apache-2.0, generated_from_trainer, model-index


language:


- ne
license: apache-2.0
tags:
- automatic-speech-recognition
- robust-speech-event
datasets:
- openslr
model-index:
- name: wav2vec2-large-xlsr-300m-nepali
results:
  - task:
name: Automatic Speech Recognition
type: automatic-speech-recognition
dataset:
name: openslr
type: openslr
args: ne
metrics:
    - name: Test WER
type: wer
value: 25.02


model name: wav2vec2-large-xlsr-53-demo-colab


model author: Mahalakshmi


tags: pytorch, ne, dataset:openslr, license:apache-2.0, automatic-speech-recognition, robust-speech-event, model-index


license: apache-2.0
tags:


- generated_from_trainer
datasets:
- common_voice
model-index:
- name: wav2vec2-xls-r-300m-demo-colab
results: []


# wav2vec2-xls-r-300m-demo-colab


This model is a fine-tuned version of [facebook/wav2vec2-xls-r-300m](https://huggingface.co/facebook/wav2vec2-xls-r-300m) on the common_voice dataset.
It achieves the following results on the evaluation set:


- eval_loss: 0.9475
- eval_wer: 1.0377
- eval_runtime: 70.5646
- eval_samples_per_second: 25.239
- eval_steps_per_second: 3.16
- epoch: 21.05
- step: 2000


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 0.0003
- train_batch_size: 32
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 500
- num_epochs: 300
- mixed_precision_training: Native AMP


### Framework versions


- Transformers 4.17.0.dev0
- Pytorch 1.10.2+cu102
- Datasets 1.18.2.dev0
- Tokenizers 0.11.0


model name: wav2vec2-xls-r-300m-demo-colab


model author: Mahalakshmi


task: automatic-speech-recognition


tags: pytorch, wav2vec2, automatic-speech-recognition, dataset:common_voice, transformers, license:apache-2.0, generated_from_trainer, model-index


Model card for RoGPT2-base


language:


- ro


# RoGPT2: Romanian GPT2 for text generation


All models are available:


* [RoBERT-base](https://huggingface.co/readerbench/RoGPT2-base)
* [RoBERT-medium](https://huggingface.co/readerbench/RoGPT2-medium)
* [RoBERT-large](https://huggingface.co/readerbench/RoGPT2-large)


For code and evaluation check out [GitHub](https://github.com/readerbench/RoGPT2).


#### How to use


```python
# TensorFlow
from transformers import AutoTokenizer, TFAutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained('readerbench/RoGPT2-base')
model = TFAutoModelForCausalLM.from_pretrained('readerbench/RoGPT2-base')
inputs = tokenizer.encode("Este o zi de vara", return_tensors='tf')
text = model.generate(inputs, max_length=1024,  no_repeat_ngram_size=2)
print(tokenizer.decode(text[0]))

# PyTorch
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained('readerbench/RoGPT2-base')
model = AutoModelForCausalLM.from_pretrained('readerbench/RoGPT2-base')
inputs = tokenizer.encode("Este o zi de vara", return_tensors='pt')
text = model.generate(inputs, max_length=1024,  no_repeat_ngram_size=2)
print(tokenizer.decode(text[0]))
```


## Training


### Corpus Statistics


| Corpus | Total size | Number of words | Number of sentences |
|:------:|:----------:|:---------------:|:-------------------:|
|OSCAR| 11.54 GB | 1745M | 48.46M |
|Wiki-Ro | 0.46 GB | 68M | 1.79M |
|Debates | 0.5 GB | 73M | 3.61M |
|Books | 4.37 GB | 667M | 37.39M |
|News | 0.15 GB | 23M | 0.77M |


### Training Statistics


| Version | Number of parameters | Number of epoch | Duration of an epoch | Context size | Batch size | PPL |
|:-------:|:--------------------:|:---------------:|:--------------------:|:----------:|:----------:|:---:|
| Base   | 124M | 15 | 7h      | 1024 | 72 | 22.96    |
| Medium | 354M | 10 | 22h     | 1024 | 24 | 17.64    |
| Large  | 774M | 5  | **45h** | 512  | 16 | **16.77**|


## Evaluation


### 1. MOROCO


|      Model        | Dialect | Md to Ro | Ro to Md |
|:-----------------:|:-------:|:--------:|:--------:|
| KRR + SK          | 94.06   | 67.59    | 75.47    |
| BERT-base-ro      | 95.98   | 69.90    | 78.08    |
| RoBERT-small      | 95.76   | 69.05    | 80.15    |
| RoBERT-base       |**97.24**| 68.80    | 82.37    |
| RoBERT-large      | 97.21   | 69.50    | **83.26**|
| RoGPT2-base       | 96.69   | 69.82    | 77.55    |
| RoGPT2-medium     | 96.42   | 69.77    | 80.51    |
| RoGPT2-large      | 96.93   |**71.07** | 82.56    |


### 2. LaRoSeDa


| Model        | Binary: Accuracy | Binary: F1-Score | Multi-Class: Accuracy | Multi-Class: F1-Score |
|:------------:|:----------------:|:----------------:|:---------------------:|:---------------------:|
|BERT-base-ro  | 98.07            | 97.94            | -                     |79.61                  |
| RoDiBERT     |**98.40**         |**98.31**         | -                     |83.01                  |
| RoBERT-small | 97.44            | 97.43            | 89.30                 |84.23                  |
| RoBERT-base  | 98.27            | 98.26            | 90.59                 |86.27                  |
| RoBERT-large | 98.20            | 98.19            |**90.93**              |**86.63**              |
| RoGPT2-base  | 97.89            | 97.88            |89.65                  |84.68                  |
|RoGPT2-medium | 98.03            |98.04             | 90.29                 | 85.37                 |
| RoGPT2-large | 98.06            |98.07             | 90.26                 | 84.89                 |


### 3. RoSTS


| Model        | Spearman dev-set | Spearman test-set | Pearson dev-set | Pearson test-set |
|:------------:|:----------------:|:-----------------:|:---------------:|:----------------:|
|BERT-base-ro  | 84.26            | 80.86             | 84.59           | 81.59            |
|RoDiBERT      | 77.07            | 71.47             | 77.13           | 72.25            |
|RoBERT-small  | 82.06            | 78.06             | 81.66           | 78.49            |
|RoBERT-base   | 84.93            | 80.39             | 85.03           | 80.39            |
|RoBERT-large  |**86.25**         |**83.15**          |**86.58**        |**83.76**         |
|RoGPT2-base   | 83.51            | 79.77             | 83.74           | 80.56            |
|RoGPT2-medium | 85.75            | 82.25             | 86.04           | 83.16            |
|RoGPT2-large  | 85.70            | 82.64             | 86.14           | 83.46            |


### 4. WMT16


| Model        | Decoder method | Ro-En  | En-Ro  |
|:------------:|:--------------:|:------:|:------:|
|mBART         | -              |**38.5**|**38.5**|
|OpenNMT       | -              |  -     | 24.7   |
|RoGPT2-base   |Greedy          | 30.37  | 20.27  |
|RoGPT2-base   |Beam-search-4   | 31.26  | 22.31  |
|RoGPT2-base   |Beam-search-8   | 31.39  | 22.95  |
|RoGPT2-medium |Greedy          | 32.48  | 22.18  |
|RoGPT2-medium |Beam-search-4   | 34.08  | 24.03  |
|RoGPT2-medium |Beam-search-8   | 34.16  | 24.13  |
|RoGPT2-large  |Greedy          | 33.69  | 23.31  |
|RoGPT2-large  |Beam-search-4   |34.40   |24.23   |
|RoGPT2-large  |Beam-search-8   |34.51   |24.32   |


### 5. XQuAD


| Model        |Decoder method |  EM   | F1-Score |
|:------------:|:-------------:|:-----:|:--------:|
|BERT-base-ro  | -             | 47.89 | 63.74    |
|RoDiBERT      | -             | 21.76 | 34.57    |
|RoBERT-small  | -             | 30.84 | 45.17    |
|RoBERT-base   | -             | 53.52 | 70.04    |
|RoBERT-large  | -             | 55.46 |  69.64   |
|mBERT         | -             |59.9   | 72.7  |
|XLM-R Large   | -             |**69.7**|**83.6**|
|RoGPT2-base   |  Greedy       | 23.69 | 35.97    |
|RoGPT2-base   | Beam-search-4 | 24.11 | 35.27    |
|RoGPT2-medium | Greedy        | 29.66 | 44.74    |
|RoGPT2-medium | Beam-search-4 | 31.59 | 45.32 |
|RoGPT2-large | Greedy | 29.74 | 42.98 |
|RoGPT2-large | Beam-search-4 | 29.66 | 43.05 |
|RoGPT2-base-en-ro | Greedy | 23.86 | 34.27 |
|RoGPT2-base-en-ro | Beam-search-4 | 25.04 | 34.51 |
|RoGPT2-medium-en-ro | Greedy | 27.05 | 39.75 |
|RoGPT2-medium-en-ro | Beam-search-4 | 27.64 | 39.11 |
|RoGPT2-large-en-ro | Greedy | 28.40 | 39.79 |
|RoGPT2-large-en-ro | Beam-search-4 | 28.73 | 39.71 |
|RoGPT2-large-en-ro-mask | Greedy | 31.34 | 44.71 |
|RoGPT2-large-en-ro-mask|  Beam-search-4 | 31.59 | 43.53 |


### 6. Wiki-Ro: LM


| Model        | PPL dev | PPL test |
|:------------:|:-------:|:--------:|
|BERT-base-ro | 29.0897 | 28.0043|
|RoGPT2-base | 34.3795 | 33.7460|
|RoGPT2-medium | 23.7879 | 23.4581|
|RoGPT2-large | **21.7491** | **21.5200** |


### 7. RoGEC


| Model | Decoder mothod |  P  |  R  |  F0.5   |
|:-----:|:--------------:|:---:|:---:|:------:|
|Transformer-tiny | Beam-search | 53.53 | 26.36 | 44.38 |
|Transformer-base Finetuning | Beam-search | 56.05 | 46.19 | 53.76 |
|Transformer-base Finetuning | Beam-search-LM | 50.68 | 45.39 | 49.52 |
|Transformer-base Finetuning | Beam-search-norm-LM | 51.06 | 45.43 | 49.83 |
|RoGPT2-base | Greedy | 59.02 | 49.35 | 56.80 |
|RoGPT2-base | Beam-search-4 | 65.23 | 49.26 | 61.26 |
|RoGPT2-base  |Beam-search-8 | 65.88 | 49.64 | 61.84 |
|RoGPT2-medium | Greedy | 69.97 | 57.94 | 67.18 |
|RoGPT2-medium | Beam-search-4 | **72.46** | **57.99** | **69.01** |
|RoGPT2-medium | Beam-search-8 | 72.24 | 57.69 | 68.77 |
|RoGP2-large | Greedy | 61.90 | 49.09 | 58.83 |
|RoGP2-large | Beam-search-4 | 65.24 | 49.43 | 61.32 |
|RoGP2-large | Beam-search-8 | 64.96 | 49.22 | 61.06 |
|RoGPT2-base* | Greedy | 68.67 | 49.60 | 63.77 |
|RoGPT2-base* | Beam-search-4 | 71.16 | 50.53 | 65.79 |
|RoGPT2-base* | Beam-search-8 | 71.68 | 50.65 | 66.18 |
|RoGPT2-medium* | Greedy | 58.21 | 43.32 | 54.47 |
|RoGPT2-medium* | Beam-search-4 | 68.31 | 43.78 | 61.43 |
|RoGPT2-medium* | Beam-search-8 | 68.68 | 43.99 | 61.75 |
|RoGPT2-large* | Greedy | 64.86 | 41.30 | 58.22 |
|RoGPT2-large* | Beam-search-4 | 65.57 | 41.00 | 58.55 |
|RoGPT2-large* | Beam-search-8 | 65.44 | 41.09 | 58.50 |


****Note****: * the models were trained using the dataset of 3,000,000 artificially generated pairs


## Acknowledgments


Research supported with [Cloud TPUs](https://cloud.google.com/tpu/) from Google's [TensorFlow Research Cloud (TFRC)](https://www.tensorflow.org/tfrc)


## How to cite


Niculescu, M. A., Ruseti, S., and Dascalu, M. (submitted). RoGPT2: Romanian GPT2 for Text Generation


model name: RoGPT2-base


model author: readerbench


task: text-generation


tags: pytorch, tf, gpt2, text-generation, ro, transformers


Model card for RoGPT2-large


language:


- ro


# RoGPT2: Romanian GPT2 for text generation


All models are available:


* [RoBERT-base](https://huggingface.co/readerbench/RoGPT2-base)
* [RoBERT-medium](https://huggingface.co/readerbench/RoGPT2-medium)
* [RoBERT-large](https://huggingface.co/readerbench/RoGPT2-large)


For code and evaluation check out [GitHub](https://github.com/readerbench/RoGPT2).


#### How to use


```python
# TensorFlow
from transformers import AutoTokenizer, TFAutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained('readerbench/RoGPT2-large')
model = TFAutoModelForCausalLM.from_pretrained('readerbench/RoGPT2-large')
inputs = tokenizer.encode("Este o zi de vara", return_tensors='tf')
text = model.generate(inputs, max_length=1024,  no_repeat_ngram_size=2)
print(tokenizer.decode(text[0]))

# PyTorch
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained('readerbench/RoGPT2-large')
model = AutoModelForCausalLM.from_pretrained('readerbench/RoGPT2-large')
inputs = tokenizer.encode("Este o zi de vara", return_tensors='pt')
text = model.generate(inputs, max_length=1024,  no_repeat_ngram_size=2)
print(tokenizer.decode(text[0]))
```


## Training


### Corpus Statistics


| Corpus | Total size | Number of words | Number of sentences |
|:------:|:----------:|:---------------:|:-------------------:|
|OSCAR| 11.54 GB | 1745M | 48.46M |
|Wiki-Ro | 0.46 GB | 68M | 1.79M |
|Debates | 0.5 GB | 73M | 3.61M |
|Books | 4.37 GB | 667M | 37.39M |
|News | 0.15 GB | 23M | 0.77M |


### Training Statistics


| Version | Number of parameters | Number of epoch | Duration of an epoch | Context size | Batch size | PPL |
|:-------:|:--------------------:|:---------------:|:--------------------:|:----------:|:----------:|:---:|
| Base   | 124M | 15 | 7h      | 1024 | 72 | 22.96    |
| Medium | 354M | 10 | 22h     | 1024 | 24 | 17.64    |
| Large  | 774M | 5  | **45h** | 512  | 16 | **16.77**|


## Evaluation


### 1. MOROCO


|      Model        | Dialect | Md to Ro | Ro to Md |
|:-----------------:|:-------:|:--------:|:--------:|
| KRR + SK          | 94.06   | 67.59    | 75.47    |
| BERT-base-ro      | 95.98   | 69.90    | 78.08    |
| RoBERT-small      | 95.76   | 69.05    | 80.15    |
| RoBERT-base       |**97.24**| 68.80    | 82.37    |
| RoBERT-large      | 97.21   | 69.50    | **83.26**|
| RoGPT2-base       | 96.69   | 69.82    | 77.55    |
| RoGPT2-medium     | 96.42   | 69.77    | 80.51    |
| RoGPT2-large      | 96.93   |**71.07** | 82.56    |


### 2. LaRoSeDa


| Model        | Binary: Accuracy | Binary: F1-Score | Multi-Class: Accuracy | Multi-Class: F1-Score |
|:------------:|:----------------:|:----------------:|:---------------------:|:---------------------:|
|BERT-base-ro  | 98.07            | 97.94            | -                     |79.61                  |
| RoDiBERT     |**98.40**         |**98.31**         | -                     |83.01                  |
| RoBERT-small | 97.44            | 97.43            | 89.30                 |84.23                  |
| RoBERT-base  | 98.27            | 98.26            | 90.59                 |86.27                  |
| RoBERT-large | 98.20            | 98.19            |**90.93**              |**86.63**              |
| RoGPT2-base  | 97.89            | 97.88            |89.65                  |84.68                  |
|RoGPT2-medium | 98.03            |98.04             | 90.29                 | 85.37                 |
| RoGPT2-large | 98.06            |98.07             | 90.26                 | 84.89                 |


### 3. RoSTS


| Model        | Spearman dev-set | Spearman test-set | Pearson dev-set | Pearson test-set |
|:------------:|:----------------:|:-----------------:|:---------------:|:----------------:|
|BERT-base-ro  | 84.26            | 80.86             | 84.59           | 81.59            |
|RoDiBERT      | 77.07            | 71.47             | 77.13           | 72.25            |
|RoBERT-small  | 82.06            | 78.06             | 81.66           | 78.49            |
|RoBERT-base   | 84.93            | 80.39             | 85.03           | 80.39            |
|RoBERT-large  |**86.25**         |**83.15**          |**86.58**        |**83.76**         |
|RoGPT2-base   | 83.51            | 79.77             | 83.74           | 80.56            |
|RoGPT2-medium | 85.75            | 82.25             | 86.04           | 83.16            |
|RoGPT2-large  | 85.70            | 82.64             | 86.14           | 83.46            |


### 4. WMT16


| Model        | Decoder method | Ro-En  | En-Ro  |
|:------------:|:--------------:|:------:|:------:|
|mBART         | -              |**38.5**|**38.5**|
|OpenNMT       | -              |  -     | 24.7   |
|RoGPT2-base   |Greedy          | 30.37  | 20.27  |
|RoGPT2-base   |Beam-search-4   | 31.26  | 22.31  |
|RoGPT2-base   |Beam-search-8   | 31.39  | 22.95  |
|RoGPT2-medium |Greedy          | 32.48  | 22.18  |
|RoGPT2-medium |Beam-search-4   | 34.08  | 24.03  |
|RoGPT2-medium |Beam-search-8   | 34.16  | 24.13  |
|RoGPT2-large  |Greedy          | 33.69  | 23.31  |
|RoGPT2-large  |Beam-search-4   |34.40   |24.23   |
|RoGPT2-large  |Beam-search-8   |34.51   |24.32   |


### 5. XQuAD


| Model        |Decoder method |  EM   | F1-Score |
|:------------:|:-------------:|:-----:|:--------:|
|BERT-base-ro  | -             | 47.89 | 63.74    |
|RoDiBERT      | -             | 21.76 | 34.57    |
|RoBERT-small  | -             | 30.84 | 45.17    |
|RoBERT-base   | -             | 53.52 | 70.04    |
|RoBERT-large  | -             | 55.46 |  69.64   |
|mBERT         | -             | 59.9  |  72.7  |
|XLM-R Large   | -             |**69.7**|**83.6**|
|RoGPT2-base   |  Greedy       | 23.69 | 35.97    |
|RoGPT2-base   | Beam-search-4 | 24.11 | 35.27    |
|RoGPT2-medium | Greedy        | 29.66 | 44.74    |
|RoGPT2-medium | Beam-search-4 | 31.59 | 45.32 |
|RoGPT2-large | Greedy | 29.74 | 42.98 |
|RoGPT2-large | Beam-search-4 | 29.66 | 43.05 |
|RoGPT2-base-en-ro | Greedy | 23.86 | 34.27 |
|RoGPT2-base-en-ro | Beam-search-4 | 25.04 | 34.51 |
|RoGPT2-medium-en-ro | Greedy | 27.05 | 39.75 |
|RoGPT2-medium-en-ro | Beam-search-4 | 27.64 | 39.11 |
|RoGPT2-large-en-ro | Greedy | 28.40 | 39.79 |
|RoGPT2-large-en-ro | Beam-search-4 | 28.73 | 39.71 |
|RoGPT2-large-en-ro-mask | Greedy | 31.34 | 44.71 |
|RoGPT2-large-en-ro-mask|  Beam-search-4 | 31.59 | 43.53 |


### 6. Wiki-Ro: LM


| Model        | PPL dev | PPL test |
|:------------:|:-------:|:--------:|
|BERT-base-ro | 29.0897 | 28.0043|
|RoGPT2-base | 34.3795 | 33.7460|
|RoGPT2-medium | 23.7879 | 23.4581|
|RoGPT2-large | **21.7491** | **21.5200** |


### 7. RoGEC


| Model | Decoder mothod |  P  |  R  |  F0.5   |
|:-----:|:--------------:|:---:|:---:|:------:|
|Transformer-tiny | Beam-search | 53.53 | 26.36 | 44.38 |
|Transformer-base Finetuning | Beam-search | 56.05 | 46.19 | 53.76 |
|Transformer-base Finetuning | Beam-search-LM | 50.68 | 45.39 | 49.52 |
|Transformer-base Finetuning | Beam-search-norm-LM | 51.06 | 45.43 | 49.83 |
|RoGPT2-base | Greedy | 59.02 | 49.35 | 56.80 |
|RoGPT2-base | Beam-search-4 | 65.23 | 49.26 | 61.26 |
|RoGPT2-base  |Beam-search-8 | 65.88 | 49.64 | 61.84 |
|RoGPT2-medium | Greedy | 69.97 | 57.94 | 67.18 |
|RoGPT2-medium | Beam-search-4 | **72.46** | **57.99** | **69.01** |
|RoGPT2-medium | Beam-search-8 | 72.24 | 57.69 | 68.77 |
|RoGP2-large | Greedy | 61.90 | 49.09 | 58.83 |
|RoGP2-large | Beam-search-4 | 65.24 | 49.43 | 61.32 |
|RoGP2-large | Beam-search-8 | 64.96 | 49.22 | 61.06 |
|RoGPT2-base* | Greedy | 68.67 | 49.60 | 63.77 |
|RoGPT2-base* | Beam-search-4 | 71.16 | 50.53 | 65.79 |
|RoGPT2-base* | Beam-search-8 | 71.68 | 50.65 | 66.18 |
|RoGPT2-medium* | Greedy | 58.21 | 43.32 | 54.47 |
|RoGPT2-medium* | Beam-search-4 | 68.31 | 43.78 | 61.43 |
|RoGPT2-medium* | Beam-search-8 | 68.68 | 43.99 | 61.75 |
|RoGPT2-large* | Greedy | 64.86 | 41.30 | 58.22 |
|RoGPT2-large* | Beam-search-4 | 65.57 | 41.00 | 58.55 |
|RoGPT2-large* | Beam-search-8 | 65.44 | 41.09 | 58.50 |


****Note****: * the models were trained using the dataset of 3,000,000 artificially generated pairs


## Acknowledgments


Research supported with [Cloud TPUs](https://cloud.google.com/tpu/) from Google's [TensorFlow Research Cloud (TFRC)](https://www.tensorflow.org/tfrc)


## How to cite


Niculescu, M. A., Ruseti, S., and Dascalu, M. (submitted). RoGPT2: Romanian GPT2 for Text Generation


model name: RoGPT2-large


model author: readerbench


task: text-generation


tags: pytorch, tf, gpt2, text-generation, ro, transformers


Model card for RoBERT-large


language:


- ro


# RoBERT-large


## Pretrained BERT model for Romanian


Pretrained model on Romanian language using a masked language modeling (MLM) and next sentence prediction (NSP) objective.
It was introduced in this [paper](https://www.aclweb.org/anthology/2020.coling-main.581/). Three BERT models were released: RoBERT-small, RoBERT-base and **RoBERT-large**, all versions uncased.


| Model          | Weights   |   L    |   H    |    A   | MLM accuracy   | NSP accuracy   |
|----------------|:---------:|:------:|:------:|:------:|:--------------:|:--------------:|
| RoBERT-small | 19M     | 12   | 256  | 8    | 0.5363       | 0.9687       |
| RoBERT-base    | 114M      | 12     | 768    | 12     | 0.6511         | 0.9802         |
| *RoBERT-large*   | *341M*      | *24*     | *1024*   | *24*     | *0.6929*         | *0.9843*         |


All models are available:


* [RoBERT-small](https://huggingface.co/readerbench/RoBERT-small)
* [RoBERT-base](https://huggingface.co/readerbench/RoBERT-base)
* [RoBERT-large](https://huggingface.co/readerbench/RoBERT-large)


#### How to use


```python
# tensorflow
from transformers import AutoModel, AutoTokenizer, TFAutoModel
tokenizer = AutoTokenizer.from_pretrained("readerbench/RoBERT-large")
model = TFAutoModel.from_pretrained("readerbench/RoBERT-large")
inputs = tokenizer("exemplu de propoziție", return_tensors="tf")
outputs = model(inputs)

# pytorch
from transformers import AutoModel, AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("readerbench/RoBERT-large")
model = AutoModel.from_pretrained("readerbench/RoBERT-large")
inputs = tokenizer("exemplu de propoziție", return_tensors="pt")
outputs = model(**inputs)
```


## Training data


The model is trained on the following compilation of corpora. Note that we present the statistics after the cleaning process.


| Corpus    | Words     | Sentences | Size (GB)|
|-----------|:---------:|:---------:|:--------:|
| Oscar     | 1.78B     | 87M       | 10.8     |
| RoTex     | 240M      | 14M       | 1.5      |
| RoWiki    | 50M       | 2M        | 0.3      |
| **Total** | **2.07B** | **103M**  | **12.6** |


## Downstream performance


### Sentiment analysis


We report Macro-averaged F1 score (in %)


| Model            | Dev      | Test     |
|------------------|:--------:|:--------:|
| multilingual-BERT| 68.96    | 69.57    |
| XLM-R-base       | 71.26    | 71.71    |
| BERT-base-ro     | 70.49    | 71.02    |
| RoBERT-small     | 66.32    | 66.37    |
| RoBERT-base      | 70.89    | 71.61    |
| *RoBERT-large*   | **72.48**| **72.11**|


### Moldavian vs. Romanian Dialect and Cross-dialect Topic identification


We report results on [VarDial 2019](https://sites.google.com/view/vardial2019/campaign) Moldavian vs. Romanian Cross-dialect Topic identification Challenge, as Macro-averaged F1 score (in %).


| Model             | Dialect Classification | MD to RO | RO to MD |
|-------------------|:----------------------:|:--------:|:--------:|
| 2-CNN + SVM       | 93.40                  | 65.09    | 75.21    |
| Char+Word SVM     | 96.20                  | 69.08    | 81.93    |
| BiGRU             | 93.30                  | **70.10**| 80.30    |
| multilingual-BERT | 95.34                  | 68.76    | 78.24    |
| XLM-R-base        | 96.28                  | 69.93    | 82.28    |
| BERT-base-ro      | 96.20                  | 69.93    | 78.79    |
| RoBERT-small      | 95.67                  | 69.01    | 80.40    |
| RoBERT-base       | 97.39                  | 68.30    | 81.09    |
| *RoBERT-large*    | **97.78**              | *69.91*  | **83.65**|


### Diacritics Restoration


Challenge can be found [here](https://diacritics-challenge.speed.pub.ro/). We report results on the official test set, as accuracies in %.


| Model                       | word level | char level |
|-----------------------------|:----------:|:----------:|
| BiLSTM                      | 99.42      | -          |
| CharCNN                     | 98.40      | 99.65      |
| CharCNN + multilingual-BERT | 99.72      | 99.94      |
| CharCNN + XLM-R-base        | 99.76      | **99.95**  |
| CharCNN + BERT-base-ro      | **99.79**  | **99.95**  |
| CharCNN + RoBERT-small      | 99.73      |  99.94     |
| CharCNN + RoBERT-base       | 99.78      | **99.95**  |
| *CharCNN + RoBERT-large*    | *99.76*    | **99.95**  |


### BibTeX entry and citation info


```bibtex
@inproceedings{masala2020robert,
  title={RoBERT--A Romanian BERT Model},
  author={Masala, Mihai and Ruseti, Stefan and Dascalu, Mihai},
  booktitle={Proceedings of the 28th International Conference on Computational Linguistics},
  pages={6626--6637},
  year={2020}
}
```


model name: RoBERT-large


model author: readerbench


tags: pytorch, tf, jax, bert, ro, transformers, infinity_compatible


Model card for RoBERT-small


language:


- ro


# RoBERT-small


## Pretrained BERT model for Romanian


Pretrained model on Romanian language using a masked language modeling (MLM) and next sentence prediction (NSP) objective.
It was introduced in this [paper](https://www.aclweb.org/anthology/2020.coling-main.581/). Three BERT models were released: **RoBERT-small**, RoBERT-base and RoBERT-large, all versions uncased.


| Model          | Weights   |   L    |   H    |    A   | MLM accuracy   | NSP accuracy   |
|----------------|:---------:|:------:|:------:|:------:|:--------------:|:--------------:|
| *RoBERT-small* | *19M*     | *12*   | *256*  | *8*    | *0.5363*       | *0.9687*       |
| RoBERT-base    | 114M      | 12     | 768    | 12     | 0.6511         | 0.9802         |
| RoBERT-large   | 341M      | 24     | 1024   | 24     | 0.6929         | 0.9843         |


All models are available:


#### How to use


```python
# tensorflow
from transformers import AutoModel, AutoTokenizer, TFAutoModel
tokenizer = AutoTokenizer.from_pretrained("readerbench/RoBERT-small")
model = TFAutoModel.from_pretrained("readerbench/RoBERT-small")
inputs = tokenizer("exemplu de propoziție", return_tensors="tf")
outputs = model(inputs)

# pytorch
from transformers import AutoModel, AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("readerbench/RoBERT-small")
model = AutoModel.from_pretrained("readerbench/RoBERT-small")
inputs = tokenizer("exemplu de propoziție", return_tensors="pt")
outputs = model(**inputs)
```


## Training data


## Downstream performance


### Sentiment analysis


| Model            | Dev      | Test     |
|------------------|:--------:|:--------:|
| multilingual-BERT| 68.96    | 69.57    |
| XLM-R-base       | 71.26    | 71.71    |
| BERT-base-ro     | 70.49    | 71.02    |
| *RoBERT-small*   | *66.32*  | *66.37*  |
| RoBERT-base      | 70.89    | 71.61    |
| RoBERT-large     | **72.48**| **72.11**|


| Model             | Dialect Classification | MD to RO | RO to MD |
|-------------------|:----------------------:|:--------:|:--------:|
| 2-CNN + SVM       | 93.40                  | 65.09    | 75.21    |
| Char+Word SVM     | 96.20                  | 69.08    | 81.93    |
| BiGRU             | 93.30                  | **70.10**| 80.30    |
| multilingual-BERT | 95.34                  | 68.76    | 78.24    |
| XLM-R-base        | 96.28                  | 69.93    | 82.28    |
| BERT-base-ro      | 96.20                  | 69.93    | 78.79    |
| *RoBERT-small*    | *95.67*                | *69.01*  | *80.40*  |
| RoBERT-base       | 97.39                  | 68.30    | 81.09    |
| RoBERT-large      | **97.78**              | 69.91    | **83.65**|


### Diacritics Restoration


| Model                       | word level | char level |
|-----------------------------|:----------:|:----------:|
| BiLSTM                      | 99.42      | -          |
| CharCNN                     | 98.40      | 99.65      |
| CharCNN + multilingual-BERT | 99.72      | 99.94      |
| CharCNN + XLM-R-base        | 99.76      | **99.95**  |
| CharCNN + BERT-base-ro      | **99.79**  | **99.95**  |
| *CharCNN + RoBERT-small*    | *99.73*    | *99.94*    |
| CharCNN + RoBERT-base       | 99.78      | **99.95**  |
| CharCNN + RoBERT-large      | 99.76      | **99.95**  |


### BibTeX entry and citation info


model name: RoBERT-small


model author: readerbench


Model card for jurBERT-large


language:


- ro


# jurBERT-large


## Pretrained juridical BERT model for Romanian


BERT Romanian juridical model trained using a masked language modeling (MLM) and next sentence prediction (NSP) objective.
It was introduced in this [paper](https://aclanthology.org/2021.nllp-1.8/). Two BERT models were released: **jurBERT-base** and **jurBERT-large**, all versions uncased.


| Model          | Weights   |   L    |   H    |    A   | MLM accuracy   | NSP accuracy   |
|----------------|:---------:|:------:|:------:|:------:|:--------------:|:--------------:|
| jurBERT-base    | 111M      | 12     | 768    | 12     | 0.8936         | 0.9923         |
| *jurBERT-large*   | *337M*  | *24*     | *1024*   | *24*     | *0.9005*         | *0.9929*         |


All models are available:


* [jurBERT-base](https://huggingface.co/readerbench/jurBERT-base)
* [jurBERT-large](https://huggingface.co/readerbench/jurBERT-large)


#### How to use


```python
# tensorflow
from transformers import AutoModel, AutoTokenizer, TFAutoModel
tokenizer = AutoTokenizer.from_pretrained("readerbench/jurBERT-large")
model = TFAutoModel.from_pretrained("readerbench/jurBERT-large")
inputs = tokenizer("exemplu de propoziție", return_tensors="tf")
outputs = model(inputs)

# pytorch
from transformers import AutoModel, AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("readerbench/jurBERT-large")
model = AutoModel.from_pretrained("readerbench/jurBERT-large")
inputs = tokenizer("exemplu de propoziție", return_tensors="pt")
outputs = model(**inputs)
```


## Datasets


The model is trained on a private corpus (that can nevertheless be rented for a fee), that is comprised of all the final ruling, containing both civil and criminal cases, published by any Romanian civil court between 2010 and 2018. Validation is performed on RoBanking datase. We extracted from RoJur common types of cases pertinent to the banking domain (e.g. administration fee litigations, enforcement appeals), kept only the summary of the arguments provided by both the plaitiffs and the defendants and the final verdict (in the form of a boolean value) to build RoBanking.


| Corpus    | Scope        |Entries    |  Size (GB)|
|-----------|:------------:|:---------:|:---------:|
| RoJur     | pre-training | 11M       | 160       |
| RoBanking | downstream   | 108k      | -         |


## Downstream performance


We report Mean AUC and Std AUC on the task of predicting the outcome of a case.


### Results on RoBanking using only the plea of the plaintiff.


| Model              | Mean AUC | Std AUC  |
|--------------------|:--------:|:--------:|
| CNN                | 79.60    | -        |
| BI-LSTM            | 80.99    | 0.26     |
| RoBERT-small       | 70.54    | 0.28     |
| RoBERT-base        | 79.74    | 0.21     |
| RoBERT-base + hf   | 79.82    | 0.11     |
| RoBERT-large       | 76.53    | 5.43     |
| jurBERT-base       | **81.47**| **0.18** |
| jurBERT-base + hf  | 81.40    | 0.18     |
| *jurBERT-large*    | *78.38*  | *1.77*   |


### Results on RoBanking using pleas from both the plaintiff and defendant.


| Model               | Mean AUC | Std AUC  |
|---------------------|:--------:|:--------:|
| BI-LSTM             | 84.60    | 0.59     |
| RoBERT-base         | 84.40    | 0.26     |
| RoBERT-base + hf    | 84.43    | 0.15     |
| jurBERT-base        | 86.63    | 0.18     |
| jurBERT-base + hf   | **86.73**| **0.22** |
| *jurBERT-large*     | *82.04*  | *0.64*    |


For complete results and discussion please refer to the [paper](https://aclanthology.org/2021.nllp-1.8/).


### BibTeX entry and citation info


```bibtex
@inproceedings{masala2021jurbert,
  title={jurBERT: A Romanian BERT Model for Legal Judgement Prediction},
  author={Masala, Mihai and Iacob, Radu Cristian Alexandru and Uban, Ana Sabina and Cidota, Marina and Velicu, Horia and Rebedea, Traian and Popescu, Marius},
  booktitle={Proceedings of the Natural Legal Language Processing Workshop 2021},
  pages={86--94},
  year={2021}
}
```


model name: jurBERT-large


model author: readerbench


tags: pytorch, tf, bert, ro, transformers, infinity_compatible


Model card for RoBERT-base


language:


- ro


# RoBERT-base


## Pretrained BERT model for Romanian


Pretrained model on Romanian language using a masked language modeling (MLM) and next sentence prediction (NSP) objective.
It was introduced in this [paper](https://www.aclweb.org/anthology/2020.coling-main.581/). Three BERT models were released: RoBERT-small, **RoBERT-base** and RoBERT-large, all versions uncased.


| Model          | Weights   |   L    |   H    |    A   | MLM accuracy   | NSP accuracy   |
|----------------|:---------:|:------:|:------:|:------:|:--------------:|:--------------:|
| RoBERT-small | 19M     | 12   | 256  | 8    | 0.5363       | 0.9687       |
| *RoBERT-base*    | *114M*      | *12*     | *768*    | *12*     | *0.6511*         | *0.9802*         |
| RoBERT-large   | 341M      | 24     | 1024   | 24     | 0.6929         | 0.9843         |


All models are available:


* [RoBERT-small](https://huggingface.co/readerbench/RoBERT-small)
* [RoBERT-base](https://huggingface.co/readerbench/RoBERT-base)
* [RoBERT-large](https://huggingface.co/readerbench/RoBERT-large)


#### How to use


```python
# tensorflow
from transformers import AutoModel, AutoTokenizer, TFAutoModel
tokenizer = AutoTokenizer.from_pretrained("readerbench/RoBERT-base")
model = TFAutoModel.from_pretrained("readerbench/RoBERT-base")
inputs = tokenizer("exemplu de propoziție", return_tensors="tf")
outputs = model(inputs)

# pytorch
from transformers import AutoModel, AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("readerbench/RoBERT-base")
model = AutoModel.from_pretrained("readerbench/RoBERT-base")
inputs = tokenizer("exemplu de propoziție", return_tensors="pt")
outputs = model(**inputs)
```


## Training data


The model is trained on the following compilation of corpora. Note that we present the statistics after the cleaning process.


| Corpus    | Words     | Sentences | Size (GB)|
|-----------|:---------:|:---------:|:--------:|
| Oscar     | 1.78B     | 87M       | 10.8     |
| RoTex     | 240M      | 14M       | 1.5      |
| RoWiki    | 50M       | 2M        | 0.3      |
| **Total** | **2.07B** | **103M**  | **12.6** |


## Downstream performance


### Sentiment analysis


We report Macro-averaged F1 score (in %)


| Model            | Dev      | Test     |
|------------------|:--------:|:--------:|
| multilingual-BERT| 68.96    | 69.57    |
| XLM-R-base       | 71.26    | 71.71    |
| BERT-base-ro     | 70.49    | 71.02    |
| RoBERT-small     | 66.32    | 66.37    |
| *RoBERT-base*    | *70.89*  | *71.61*  |
| RoBERT-large     | **72.48**| **72.11**|


### Moldavian vs. Romanian Dialect and Cross-dialect Topic identification


We report results on [VarDial 2019](https://sites.google.com/view/vardial2019/campaign) Moldavian vs. Romanian Cross-dialect Topic identification Challenge, as Macro-averaged F1 score (in %).


| Model             | Dialect Classification | MD to RO | RO to MD |
|-------------------|:----------------------:|:--------:|:--------:|
| 2-CNN + SVM       | 93.40                  | 65.09    | 75.21    |
| Char+Word SVM     | 96.20                  | 69.08    | 81.93    |
| BiGRU             | 93.30                  | **70.10**| 80.30    |
| multilingual-BERT | 95.34                  | 68.76    | 78.24    |
| XLM-R-base        | 96.28                  | 69.93    | 82.28    |
| BERT-base-ro      | 96.20                  | 69.93    | 78.79    |
| RoBERT-small      | 95.67                  | 69.01    | 80.40    |
| *RoBERT-base*     | *97.39*                | *68.30*  | *81.09*  |
| RoBERT-large      | **97.78**              | 69.91    | **83.65**|


### Diacritics Restoration


Challenge can be found [here](https://diacritics-challenge.speed.pub.ro/). We report results on the official test set, as accuracies in %.


| Model                       | word level | char level |
|-----------------------------|:----------:|:----------:|
| BiLSTM                      | 99.42      | -          |
| CharCNN                     | 98.40      | 99.65      |
| CharCNN + multilingual-BERT | 99.72      | 99.94      |
| CharCNN + XLM-R-base        | 99.76      | **99.95**  |
| CharCNN + BERT-base-ro      | **99.79**  | **99.95**  |
| CharCNN + RoBERT-small      | 99.73      | 99.94      |
| *CharCNN + RoBERT-base*     | *99.78*    | **99.95**  |
| CharCNN + RoBERT-large      | 99.76      | **99.95**  |


### BibTeX entry and citation info


```bibtex
@inproceedings{masala2020robert,
  title={RoBERT--A Romanian BERT Model},
  author={Masala, Mihai and Ruseti, Stefan and Dascalu, Mihai},
  booktitle={Proceedings of the 28th International Conference on Computational Linguistics},
  pages={6626--6637},
  year={2020}
}
```


model name: RoBERT-base


model author: readerbench


tags: pytorch, tf, jax, bert, ro, transformers, infinity_compatible


Model card for RoGPT2-medium


language:


- ro


# RoGPT2: Romanian GPT2 for text generation


All models are available:


* [RoBERT-base](https://huggingface.co/readerbench/RoGPT2-base)
* [RoBERT-medium](https://huggingface.co/readerbench/RoGPT2-medium)
* [RoBERT-large](https://huggingface.co/readerbench/RoGPT2-large)


For code and evaluation check out [GitHub](https://github.com/readerbench/RoGPT2).


#### How to use


```python
# TensorFlow
from transformers import AutoTokenizer, TFAutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained('readerbench/RoGPT2-medium')
model = TFAutoModelForCausalLM.from_pretrained('readerbench/RoGPT2-medium')
inputs = tokenizer.encode("Este o zi de vara", return_tensors='tf')
text = model.generate(inputs, max_length=1024,  no_repeat_ngram_size=2)
print(tokenizer.decode(text[0]))

# PyTorch
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained('readerbench/RoGPT2-medium')
model = AutoModelForCausalLM.from_pretrained('readerbench/RoGPT2-medium')
inputs = tokenizer.encode("Este o zi de vara", return_tensors='pt')
text = model.generate(inputs, max_length=1024,  no_repeat_ngram_size=2)
print(tokenizer.decode(text[0]))
```


## Training


### Corpus Statistics


| Corpus | Total size | Number of words | Number of sentences |
|:------:|:----------:|:---------------:|:-------------------:|
|OSCAR| 11.54 GB | 1745M | 48.46M |
|Wiki-Ro | 0.46 GB | 68M | 1.79M |
|Debates | 0.5 GB | 73M | 3.61M |
|Books | 4.37 GB | 667M | 37.39M |
|News | 0.15 GB | 23M | 0.77M |


### Training Statistics


| Version | Number of parameters | Number of epoch | Duration of an epoch | Context size | Batch size | PPL |
|:-------:|:--------------------:|:---------------:|:--------------------:|:----------:|:----------:|:---:|
| Base   | 124M | 15 | 7h      | 1024 | 72 | 22.96    |
| Medium | 354M | 10 | 22h     | 1024 | 24 | 17.64    |
| Large  | 774M | 5  | **45h** | 512  | 16 | **16.77**|


## Evaluation


### 1. MOROCO


|      Model        | Dialect | Md to Ro | Ro to Md |
|:-----------------:|:-------:|:--------:|:--------:|
| KRR + SK          | 94.06   | 67.59    | 75.47    |
| BERT-base-ro      | 95.98   | 69.90    | 78.08    |
| RoBERT-small      | 95.76   | 69.05    | 80.15    |
| RoBERT-base       |**97.24**| 68.80    | 82.37    |
| RoBERT-large      | 97.21   | 69.50    | **83.26**|
| RoGPT2-base       | 96.69   | 69.82    | 77.55    |
| RoGPT2-medium     | 96.42   | 69.77    | 80.51    |
| RoGPT2-large      | 96.93   |**71.07** | 82.56    |


### 2. LaRoSeDa


| Model        | Binary: Accuracy | Binary: F1-Score | Multi-Class: Accuracy | Multi-Class: F1-Score |
|:------------:|:----------------:|:----------------:|:---------------------:|:---------------------:|
|BERT-base-ro  | 98.07            | 97.94            | -                     |79.61                  |
| RoDiBERT     |**98.40**         |**98.31**         | -                     |83.01                  |
| RoBERT-small | 97.44            | 97.43            | 89.30                 |84.23                  |
| RoBERT-base  | 98.27            | 98.26            | 90.59                 |86.27                  |
| RoBERT-large | 98.20            | 98.19            |**90.93**              |**86.63**              |
| RoGPT2-base  | 97.89            | 97.88            |89.65                  |84.68                  |
|RoGPT2-medium | 98.03            |98.04             | 90.29                 | 85.37                 |
| RoGPT2-large | 98.06            |98.07             | 90.26                 | 84.89                 |


### 3. RoSTS


| Model        | Spearman dev-set | Spearman test-set | Pearson dev-set | Pearson test-set |
|:------------:|:----------------:|:-----------------:|:---------------:|:----------------:|
|BERT-base-ro  | 84.26            | 80.86             | 84.59           | 81.59            |
|RoDiBERT      | 77.07            | 71.47             | 77.13           | 72.25            |
|RoBERT-small  | 82.06            | 78.06             | 81.66           | 78.49            |
|RoBERT-base   | 84.93            | 80.39             | 85.03           | 80.39            |
|RoBERT-large  |**86.25**         |**83.15**          |**86.58**        |**83.76**         |
|RoGPT2-base   | 83.51            | 79.77             | 83.74           | 80.56            |
|RoGPT2-medium | 85.75            | 82.25             | 86.04           | 83.16            |
|RoGPT2-large  | 85.70            | 82.64             | 86.14           | 83.46            |


### 4. WMT16


| Model        | Decoder method | Ro-En  | En-Ro  |
|:------------:|:--------------:|:------:|:------:|
|mBART         | -              |**38.5**|**38.5**|
|OpenNMT       | -              |  -     | 24.7   |
|RoGPT2-base   |Greedy          | 30.37  | 20.27  |
|RoGPT2-base   |Beam-search-4   | 31.26  | 22.31  |
|RoGPT2-base   |Beam-search-8   | 31.39  | 22.95  |
|RoGPT2-medium |Greedy          | 32.48  | 22.18  |
|RoGPT2-medium |Beam-search-4   | 34.08  | 24.03  |
|RoGPT2-medium |Beam-search-8   | 34.16  | 24.13  |
|RoGPT2-large  |Greedy          | 33.69  | 23.31  |
|RoGPT2-large  |Beam-search-4   |34.40   |24.23   |
|RoGPT2-large  |Beam-search-8   |34.51   |24.32   |


### 5. XQuAD


| Model        |Decoder method |  EM   | F1-Score |
|:------------:|:-------------:|:-----:|:--------:|
|BERT-base-ro  | -             | 47.89 | 63.74    |
|RoDiBERT      | -             | 21.76 | 34.57    |
|RoBERT-small  | -             | 30.84 | 45.17    |
|RoBERT-base   | -             | 53.52 | 70.04    |
|RoBERT-large  | -             | 55.46 |  69.64   |
|mBERT         | -             | 59.9  |  72.7  |
|XLM-R Large   | -             |**69.7** |**83.6** |
|RoGPT2-base   |  Greedy       | 23.69 | 35.97    |
|RoGPT2-base   | Beam-search-4 | 24.11 | 35.27    |
|RoGPT2-medium | Greedy        | 29.66 | 44.74    |
|RoGPT2-medium | Beam-search-4 | 31.59 | 45.32 |
|RoGPT2-large | Greedy | 29.74 | 42.98 |
|RoGPT2-large | Beam-search-4 | 29.66 | 43.05 |
|RoGPT2-base-en-ro | Greedy | 23.86 | 34.27 |
|RoGPT2-base-en-ro | Beam-search-4 | 25.04 | 34.51 |
|RoGPT2-medium-en-ro | Greedy | 27.05 | 39.75 |
|RoGPT2-medium-en-ro | Beam-search-4 | 27.64 | 39.11 |
|RoGPT2-large-en-ro | Greedy | 28.40 | 39.79 |
|RoGPT2-large-en-ro | Beam-search-4 | 28.73 | 39.71 |
|RoGPT2-large-en-ro-mask | Greedy | 31.34 | 44.71 |
|RoGPT2-large-en-ro-mask|  Beam-search-4 | 31.59 | 43.53 |


### 6. Wiki-Ro: LM


| Model        | PPL dev | PPL test |
|:------------:|:-------:|:--------:|
|BERT-base-ro | 29.0897 | 28.0043|
|RoGPT2-base | 34.3795 | 33.7460|
|RoGPT2-medium | 23.7879 | 23.4581|
|RoGPT2-large | **21.7491** | **21.5200** |


### 7. RoGEC


| Model | Decoder mothod |  P  |  R  |  F0.5   |
|:-----:|:--------------:|:---:|:---:|:------:|
|Transformer-tiny | Beam-search | 53.53 | 26.36 | 44.38 |
|Transformer-base Finetuning | Beam-search | 56.05 | 46.19 | 53.76 |
|Transformer-base Finetuning | Beam-search-LM | 50.68 | 45.39 | 49.52 |
|Transformer-base Finetuning | Beam-search-norm-LM | 51.06 | 45.43 | 49.83 |
|RoGPT2-base | Greedy | 59.02 | 49.35 | 56.80 |
|RoGPT2-base | Beam-search-4 | 65.23 | 49.26 | 61.26 |
|RoGPT2-base  |Beam-search-8 | 65.88 | 49.64 | 61.84 |
|RoGPT2-medium | Greedy | 69.97 | 57.94 | 67.18 |
|RoGPT2-medium | Beam-search-4 | **72.46** | **57.99** | **69.01** |
|RoGPT2-medium | Beam-search-8 | 72.24 | 57.69 | 68.77 |
|RoGP2-large | Greedy | 61.90 | 49.09 | 58.83 |
|RoGP2-large | Beam-search-4 | 65.24 | 49.43 | 61.32 |
|RoGP2-large | Beam-search-8 | 64.96 | 49.22 | 61.06 |
|RoGPT2-base* | Greedy | 68.67 | 49.60 | 63.77 |
|RoGPT2-base* | Beam-search-4 | 71.16 | 50.53 | 65.79 |
|RoGPT2-base* | Beam-search-8 | 71.68 | 50.65 | 66.18 |
|RoGPT2-medium* | Greedy | 58.21 | 43.32 | 54.47 |
|RoGPT2-medium* | Beam-search-4 | 68.31 | 43.78 | 61.43 |
|RoGPT2-medium* | Beam-search-8 | 68.68 | 43.99 | 61.75 |
|RoGPT2-large* | Greedy | 64.86 | 41.30 | 58.22 |
|RoGPT2-large* | Beam-search-4 | 65.57 | 41.00 | 58.55 |
|RoGPT2-large* | Beam-search-8 | 65.44 | 41.09 | 58.50 |


****Note****: * the models were trained using the dataset of 3,000,000 artificially generated pairs


## Acknowledgments


Research supported with [Cloud TPUs](https://cloud.google.com/tpu/) from Google's [TensorFlow Research Cloud (TFRC)](https://www.tensorflow.org/tfrc)


## How to cite


Niculescu, M. A., Ruseti, S., and Dascalu, M. (submitted). RoGPT2: Romanian GPT2 for Text Generation


model name: RoGPT2-medium


model author: readerbench


task: text-generation


tags: pytorch, tf, gpt2, text-generation, ro, transformers


Model card for jurBERT-base


language:


- ro


# jurBERT-base


## Pretrained juridical BERT model for Romanian


BERT Romanian juridical model trained using a masked language modeling (MLM) and next sentence prediction (NSP) objective.
It was introduced in this [paper](https://aclanthology.org/2021.nllp-1.8/). Two BERT models were released: **jurBERT-base** and **jurBERT-large**, all versions uncased.


| Model          | Weights   |   L    |   H    |    A   | MLM accuracy   | NSP accuracy   |
|----------------|:---------:|:------:|:------:|:------:|:--------------:|:--------------:|
| *jurBERT-base*    | *111M*      | *12*     | *768*    | *12*     | *0.8936*         | *0.9923*         |
| jurBERT-large   | 337M      | 24     | 1024   | 24     | 0.9005         | 0.9929         |


All models are available:


* [jurBERT-base](https://huggingface.co/readerbench/jurBERT-base)
* [jurBERT-large](https://huggingface.co/readerbench/jurBERT-large)


#### How to use


```python
# tensorflow
from transformers import AutoModel, AutoTokenizer, TFAutoModel
tokenizer = AutoTokenizer.from_pretrained("readerbench/jurBERT-base")
model = TFAutoModel.from_pretrained("readerbench/jurBERT-base")
inputs = tokenizer("exemplu de propoziție", return_tensors="tf")
outputs = model(inputs)

# pytorch
from transformers import AutoModel, AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("readerbench/jurBERT-base")
model = AutoModel.from_pretrained("readerbench/jurBERT-base")
inputs = tokenizer("exemplu de propoziție", return_tensors="pt")
outputs = model(**inputs)
```


## Datasets


The model is trained on a private corpus (that can nevertheless be rented for a fee), that is comprised of all the final ruling, containing both civil and criminal cases, published by any Romanian civil court between 2010 and 2018. Validation is performed on two other datasets, RoBanking and BRDCases. We extracted from RoJur common types of cases pertinent to the banking domain (e.g. administration fee litigations, enforcement appeals), kept only the summary of the arguments provided by both the plaitiffs and the defendants and the final verdict (in the form of a boolean value) to build RoBanking. BRDCases represents a collection of cases in which BRD Groupe Société Générale Romania was directly involved.


| Corpus    | Scope        |Entries    |  Size (GB)|
|-----------|:------------:|:---------:|:---------:|
| RoJur     | pre-training | 11M       | 160       |
| RoBanking | downstream   | 108k      | -         |
| BRDCases  | downstream   | 149       | -         |


## Downstream performance


We report Mean AUC and Std AUC on the task of predicting the outcome of a case.


### Results on RoBanking using only the plea of the plaintiff.


| Model              | Mean AUC | Std AUC  |
|--------------------|:--------:|:--------:|
| CNN                | 79.60    | -        |
| BI-LSTM            | 80.99    | 0.26     |
| RoBERT-small       | 70.54    | 0.28     |
| RoBERT-base        | 79.74    | 0.21     |
| RoBERT-base + hf   | 79.82    | 0.11     |
| RoBERT-large       | 76.53    | 5.43     |
| *jurBERT-base*     | **81.47**| **0.18** |
| *jurBERT-base + hf*| *81.40*  | *0.18*   |
| jurBERT-large      | 78.38    | 1.77     |


### Results on RoBanking using pleas from both the plaintiff and defendant.


| Model               | Mean AUC | Std AUC  |
|---------------------|:--------:|:--------:|
| BI-LSTM             | 84.60    | 0.59     |
| RoBERT-base         | 84.40    | 0.26     |
| RoBERT-base + hf    | 84.43    | 0.15     |
| *jurBERT-base*      | *86.63*  | *0.18*   |
| *jurBERT-base + hf* | **86.73**| **0.22** |
| jurBERT-large       | 82.04    | 0.64     |


### Results on BRDCases


| Model               | Mean AUC | Std AUC  |
|---------------------|:--------:|:--------:|
| SVM with SK         | 57.72    | 2.15     |
| RoBERT-base         | 53.24    | 1.76     |
| RoBERT-base + hf    | 55.40    | 0.96     |
| *jurBERT-base*      | *59.65*  | *1.16*   |
| *jurBERT-base + hf* | **61.46**| **1.76** |


For complete results and discussion please refer to the [paper](https://aclanthology.org/2021.nllp-1.8/).


### BibTeX entry and citation info


```bibtex
@inproceedings{masala2021jurbert,
  title={jurBERT: A Romanian BERT Model for Legal Judgement Prediction},
  author={Masala, Mihai and Iacob, Radu Cristian Alexandru and Uban, Ana Sabina and Cidota, Marina and Velicu, Horia and Rebedea, Traian and Popescu, Marius},
  booktitle={Proceedings of the Natural Legal Language Processing Workshop 2021},
  pages={86--94},
  year={2021}
}
```


model name: jurBERT-base


model author: readerbench


tags: pytorch, tf, bert, ro, transformers, infinity_compatible


license: apache-2.0
tags:


- generated_from_trainer
model-index:
- name: wav2vec2-base-timit-demo-colab
results: []


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# wav2vec2-base-timit-demo-colab


This model is a fine-tuned version of [facebook/wav2vec2-large-xlsr-53](https://huggingface.co/facebook/wav2vec2-large-xlsr-53) on the None dataset.
It achieves the following results on the evaluation set:


- Loss: 2.7195
- Wer: 1.0


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 0.0003
- train_batch_size: 32
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 100
- num_epochs: 3
- mixed_precision_training: Native AMP


### Training results


| Training Loss | Epoch | Step | Validation Loss | Wer |
|:-------------:|:-----:|:----:|:---------------:|:---:|
| 3.0306        | 0.8   | 100  | 3.0392          | 1.0 |
| 2.9429        | 1.6   | 200  | 3.2416          | 1.0 |
| 2.7792        | 2.4   | 300  | 2.7195          | 1.0 |


### Framework versions


- Transformers 4.11.3
- Pytorch 1.10.0+cu111
- Datasets 1.13.3
- Tokenizers 0.10.3


model name: wav2vec2-base-timit-demo-colab


model author: Razvanip


task: automatic-speech-recognition


tags: pytorch, tensorboard, wav2vec2, automatic-speech-recognition, transformers, license:apache-2.0, generated_from_trainer, model-index


language: tr
datasets:


- mc4
- wikipedia
- multilexnorm
tags:
- lexical normalization
license: apache-2.0


# Fine-tuned ByT5-small for MultiLexNorm (Turkish version)


![model image](https://github.com/ufal/multilexnorm2021/raw/master/img/overall.png)


This is the official release of the fine-tuned models for **the winning entry** to the [*W-NUT 2021: Multilingual Lexical Normalization (MultiLexNorm)* shared task](https://noisy-text.github.io/2021/multi-lexnorm.html), which evaluates lexical-normalization systems on 12 social media datasets in 11 languages.


Our system is based on [ByT5](https://arxiv.org/abs/2105.13626), which we first pre-train on synthetic data and then fine-tune on authentic normalization data. It achieves the best performance by a wide margin in intrinsic evaluation, and also the best performance in extrinsic evaluation through dependency parsing. In addition to these fine-tuned models, we also release the source files on [GitHub](https://github.com/ufal/multilexnorm2021) and an interactive demo on [Google Colab](https://colab.research.google.com/drive/1rxpI8IlKk-D2crFqi2hdzbTBIezqgsCg?usp=sharing).


## How to use


The model was *not* fine-tuned in a standard sentence-to-sentence setting – instead, it was tailored to the token-to-token definition of MultiLexNorm data. Please refer to [**the interactive demo on Colab notebook**](https://colab.research.google.com/drive/1rxpI8IlKk-D2crFqi2hdzbTBIezqgsCg?usp=sharing) to learn how to use these models.


## How to cite


```bibtex
@inproceedings{wnut-ufal,
  title= "{ÚFAL} at {MultiLexNorm} 2021: Improving Multilingual Lexical Normalization by Fine-tuning {ByT5}",
  author = "Samuel, David and Straka, Milan",
  booktitle = "Proceedings of the 7th Workshop on Noisy User-generated Text (W-NUT 2021)",
  year = "2021",
  publisher = "Association for Computational Linguistics",
  address = "Punta Cana, Dominican Republic"
}
```


## ByT5 - Small


ByT5 is a tokenizer-free version of [Google's T5](https://ai.googleblog.com/2020/02/exploring-transfer-learning-with-t5.html) and generally follows the architecture of [MT5](https://huggingface.co/google/mt5-small).


ByT5 was only pre-trained on [mC4](https://www.tensorflow.org/datasets/catalog/c4#c4multilingual) excluding any supervised training with an average span-mask of 20 UTF-8 characters. Therefore, this model has to be fine-tuned before it is useable on a downstream task.


ByT5 works especially well on noisy text data,*e.g.*, `google/byt5-small` significantly outperforms [mt5-small](https://huggingface.co/google/mt5-small) on [TweetQA](https://arxiv.org/abs/1907.06292).


Paper: [ByT5: Towards a token-free future with pre-trained byte-to-byte models](https://arxiv.org/abs/2105.13626)


Authors: *Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, Colin Raffel*


model name: byt5-small-multilexnorm2021-tr


model author: ufal


task: text2text-generation


tags: pytorch, t5, text2text-generation, tr, dataset:mc4, dataset:wikipedia, dataset:multilexnorm, arxiv:2105.13626, arxiv:1907.06292, transformers, lexical normalization, license:apache-2.0, autonlp_compatible


language: it
datasets:


# Fine-tuned ByT5-small for MultiLexNorm (Italian version)


model name: byt5-small-multilexnorm2021-it


tags: pytorch, t5, text2text-generation, it, dataset:mc4, dataset:wikipedia, dataset:multilexnorm, arxiv:2105.13626, arxiv:1907.06292, transformers, lexical normalization, license:apache-2.0, autonlp_compatible


language: es
datasets:


- mc4
- wikipedia
- multilexnorm
tags:
- lexical normalization
license: apache-2.0


# Fine-tuned ByT5-small for MultiLexNorm (Spanish version)


![model image](https://github.com/ufal/multilexnorm2021/raw/master/img/overall.png)


This is the official release of the fine-tuned models for **the winning entry** to the [*W-NUT 2021: Multilingual Lexical Normalization (MultiLexNorm)* shared task](https://noisy-text.github.io/2021/multi-lexnorm.html), which evaluates lexical-normalization systems on 12 social media datasets in 11 languages.


Our system is based on [ByT5](https://arxiv.org/abs/2105.13626), which we first pre-train on synthetic data and then fine-tune on authentic normalization data. It achieves the best performance by a wide margin in intrinsic evaluation, and also the best performance in extrinsic evaluation through dependency parsing. In addition to these fine-tuned models, we also release the source files on [GitHub](https://github.com/ufal/multilexnorm2021) and an interactive demo on [Google Colab](https://colab.research.google.com/drive/1rxpI8IlKk-D2crFqi2hdzbTBIezqgsCg?usp=sharing).


## How to use


The model was *not* fine-tuned in a standard sentence-to-sentence setting – instead, it was tailored to the token-to-token definition of MultiLexNorm data. Please refer to [**the interactive demo on Colab notebook**](https://colab.research.google.com/drive/1rxpI8IlKk-D2crFqi2hdzbTBIezqgsCg?usp=sharing) to learn how to use these models.


## How to cite


```bibtex
@inproceedings{wnut-ufal,
  title= "{ÚFAL} at {MultiLexNorm} 2021: Improving Multilingual Lexical Normalization by Fine-tuning {ByT5}",
  author = "Samuel, David and Straka, Milan",
  booktitle = "Proceedings of the 7th Workshop on Noisy User-generated Text (W-NUT 2021)",
  year = "2021",
  publisher = "Association for Computational Linguistics",
  address = "Punta Cana, Dominican Republic"
}
```


## ByT5 - Small


ByT5 is a tokenizer-free version of [Google's T5](https://ai.googleblog.com/2020/02/exploring-transfer-learning-with-t5.html) and generally follows the architecture of [MT5](https://huggingface.co/google/mt5-small).


ByT5 was only pre-trained on [mC4](https://www.tensorflow.org/datasets/catalog/c4#c4multilingual) excluding any supervised training with an average span-mask of 20 UTF-8 characters. Therefore, this model has to be fine-tuned before it is useable on a downstream task.


ByT5 works especially well on noisy text data,*e.g.*, `google/byt5-small` significantly outperforms [mt5-small](https://huggingface.co/google/mt5-small) on [TweetQA](https://arxiv.org/abs/1907.06292).


Paper: [ByT5: Towards a token-free future with pre-trained byte-to-byte models](https://arxiv.org/abs/2105.13626)


Authors: *Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, Colin Raffel*


model name: byt5-small-multilexnorm2021-es


model author: ufal


task: text2text-generation


tags: pytorch, t5, text2text-generation, es, dataset:mc4, dataset:wikipedia, dataset:multilexnorm, arxiv:2105.13626, arxiv:1907.06292, transformers, lexical normalization, license:apache-2.0, autonlp_compatible


language:


- id
- en
datasets:
- mc4
- wikipedia
- multilexnorm
tags:
- lexical normalization
license: apache-2.0


# Fine-tuned ByT5-small for MultiLexNorm (Indonesian-English version)


## How to use


model name: byt5-small-multilexnorm2021-iden


tags: pytorch, t5, text2text-generation, id, en, dataset:mc4, dataset:wikipedia, dataset:multilexnorm, arxiv:2105.13626, arxiv:1907.06292, transformers, lexical normalization, license:apache-2.0, autonlp_compatible


language: "cs"
tags:


- Czech
- RoBERTa
- ÚFAL
license: "cc-by-nc-4.0"


# RobeCzech model


RobeCzech is a monolingual RoBERTa language representation model trained on Czech data.


RobeCzech model is released publicly at [LINDAT](https://hdl.handle.net/11234/1-3691) and [Hugging Face](https://huggingface.co/ufal/robeczech-base).


Please cite the corresponding publication:


- Milan Straka, Jakub Náplava, Jana Straková and David Samuel: Czech RoBERTa, a monolingual contextualized language representation model. Accepted to TSD 2021.


Preprint of the paper is available at https://arxiv.org/abs/2105.11314.


model name: robeczech-base


model author: ufal


task: fill-mask


tags: pytorch, tf, roberta, fill-mask, cs, arxiv:2105.11314, transformers, Czech, RoBERTa, ÚFAL, license:cc-by-nc-4.0, autonlp_compatible, infinity_compatible


language: hr
datasets:


- mc4
- wikipedia
- multilexnorm
tags:
- lexical normalization
license: apache-2.0


# Fine-tuned ByT5-small for MultiLexNorm (Croatian version)


![model image](https://github.com/ufal/multilexnorm2021/raw/master/img/overall.png)


This is the official release of the fine-tuned models for **the winning entry** to the [*W-NUT 2021: Multilingual Lexical Normalization (MultiLexNorm)* shared task](https://noisy-text.github.io/2021/multi-lexnorm.html), which evaluates lexical-normalization systems on 12 social media datasets in 11 languages.


Our system is based on [ByT5](https://arxiv.org/abs/2105.13626), which we first pre-train on synthetic data and then fine-tune on authentic normalization data. It achieves the best performance by a wide margin in intrinsic evaluation, and also the best performance in extrinsic evaluation through dependency parsing. In addition to these fine-tuned models, we also release the source files on [GitHub](https://github.com/ufal/multilexnorm2021) and an interactive demo on [Google Colab](https://colab.research.google.com/drive/1rxpI8IlKk-D2crFqi2hdzbTBIezqgsCg?usp=sharing).


## How to use


The model was *not* fine-tuned in a standard sentence-to-sentence setting – instead, it was tailored to the token-to-token definition of MultiLexNorm data. Please refer to [**the interactive demo on Colab notebook**](https://colab.research.google.com/drive/1rxpI8IlKk-D2crFqi2hdzbTBIezqgsCg?usp=sharing) to learn how to use these models.


## How to cite


```bibtex
@inproceedings{wnut-ufal,
  title= "{ÚFAL} at {MultiLexNorm} 2021: Improving Multilingual Lexical Normalization by Fine-tuning {ByT5}",
  author = "Samuel, David and Straka, Milan",
  booktitle = "Proceedings of the 7th Workshop on Noisy User-generated Text (W-NUT 2021)",
  year = "2021",
  publisher = "Association for Computational Linguistics",
  address = "Punta Cana, Dominican Republic"
}
```


## ByT5 - Small


ByT5 is a tokenizer-free version of [Google's T5](https://ai.googleblog.com/2020/02/exploring-transfer-learning-with-t5.html) and generally follows the architecture of [MT5](https://huggingface.co/google/mt5-small).


ByT5 was only pre-trained on [mC4](https://www.tensorflow.org/datasets/catalog/c4#c4multilingual) excluding any supervised training with an average span-mask of 20 UTF-8 characters. Therefore, this model has to be fine-tuned before it is useable on a downstream task.


ByT5 works especially well on noisy text data,*e.g.*, `google/byt5-small` significantly outperforms [mt5-small](https://huggingface.co/google/mt5-small) on [TweetQA](https://arxiv.org/abs/1907.06292).


Paper: [ByT5: Towards a token-free future with pre-trained byte-to-byte models](https://arxiv.org/abs/2105.13626)


Authors: *Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, Colin Raffel*


model name: byt5-small-multilexnorm2021-hr


model author: ufal


task: text2text-generation


tags: pytorch, t5, text2text-generation, hr, dataset:mc4, dataset:wikipedia, dataset:multilexnorm, arxiv:2105.13626, arxiv:1907.06292, transformers, lexical normalization, license:apache-2.0, autonlp_compatible


language: sr
datasets:


# Fine-tuned ByT5-small for MultiLexNorm (Serbian version)


model name: byt5-small-multilexnorm2021-sr


tags: pytorch, t5, text2text-generation, sr, dataset:mc4, dataset:wikipedia, dataset:multilexnorm, arxiv:2105.13626, arxiv:1907.06292, transformers, lexical normalization, license:apache-2.0, autonlp_compatible


language: de
datasets:


- mc4
- wikipedia
- multilexnorm
tags:
- lexical normalization
license: apache-2.0


# Fine-tuned ByT5-small for MultiLexNorm (German version)


![model image](https://github.com/ufal/multilexnorm2021/raw/master/img/overall.png)


This is the official release of the fine-tuned models for **the winning entry** to the [*W-NUT 2021: Multilingual Lexical Normalization (MultiLexNorm)* shared task](https://noisy-text.github.io/2021/multi-lexnorm.html), which evaluates lexical-normalization systems on 12 social media datasets in 11 languages.


Our system is based on [ByT5](https://arxiv.org/abs/2105.13626), which we first pre-train on synthetic data and then fine-tune on authentic normalization data. It achieves the best performance by a wide margin in intrinsic evaluation, and also the best performance in extrinsic evaluation through dependency parsing. In addition to these fine-tuned models, we also release the source files on [GitHub](https://github.com/ufal/multilexnorm2021) and an interactive demo on [Google Colab](https://colab.research.google.com/drive/1rxpI8IlKk-D2crFqi2hdzbTBIezqgsCg?usp=sharing).


## How to use


The model was *not* fine-tuned in a standard sentence-to-sentence setting – instead, it was tailored to the token-to-token definition of MultiLexNorm data. Please refer to [**the interactive demo on Colab notebook**](https://colab.research.google.com/drive/1rxpI8IlKk-D2crFqi2hdzbTBIezqgsCg?usp=sharing) to learn how to use these models.


## How to cite


```bibtex
@inproceedings{wnut-ufal,
  title= "{ÚFAL} at {MultiLexNorm} 2021: Improving Multilingual Lexical Normalization by Fine-tuning {ByT5}",
  author = "Samuel, David and Straka, Milan",
  booktitle = "Proceedings of the 7th Workshop on Noisy User-generated Text (W-NUT 2021)",
  year = "2021",
  publisher = "Association for Computational Linguistics",
  address = "Punta Cana, Dominican Republic"
}
```


## ByT5 - Small


ByT5 is a tokenizer-free version of [Google's T5](https://ai.googleblog.com/2020/02/exploring-transfer-learning-with-t5.html) and generally follows the architecture of [MT5](https://huggingface.co/google/mt5-small).


ByT5 was only pre-trained on [mC4](https://www.tensorflow.org/datasets/catalog/c4#c4multilingual) excluding any supervised training with an average span-mask of 20 UTF-8 characters. Therefore, this model has to be fine-tuned before it is useable on a downstream task.


ByT5 works especially well on noisy text data,*e.g.*, `google/byt5-small` significantly outperforms [mt5-small](https://huggingface.co/google/mt5-small) on [TweetQA](https://arxiv.org/abs/1907.06292).


Paper: [ByT5: Towards a token-free future with pre-trained byte-to-byte models](https://arxiv.org/abs/2105.13626)


Authors: *Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, Colin Raffel*


model name: byt5-small-multilexnorm2021-de


model author: ufal


task: text2text-generation


tags: pytorch, t5, text2text-generation, de, dataset:mc4, dataset:wikipedia, dataset:multilexnorm, arxiv:2105.13626, arxiv:1907.06292, transformers, lexical normalization, license:apache-2.0, autonlp_compatible


language: nl
datasets:


# Fine-tuned ByT5-small for MultiLexNorm (Dutch version)


model name: byt5-small-multilexnorm2021-nl


tags: pytorch, t5, text2text-generation, nl, dataset:mc4, dataset:wikipedia, dataset:multilexnorm, arxiv:2105.13626, arxiv:1907.06292, transformers, lexical normalization, license:apache-2.0, autonlp_compatible


language: en
datasets:


- mc4
- wikipedia
- multilexnorm
tags:
- lexical normalization
license: apache-2.0


# Fine-tuned ByT5-small for MultiLexNorm (English version)


![model image](https://github.com/ufal/multilexnorm2021/raw/master/img/overall.png)


This is the official release of the fine-tuned models for **the winning entry** to the [*W-NUT 2021: Multilingual Lexical Normalization (MultiLexNorm)* shared task](https://noisy-text.github.io/2021/multi-lexnorm.html), which evaluates lexical-normalization systems on 12 social media datasets in 11 languages.


Our system is based on [ByT5](https://arxiv.org/abs/2105.13626), which we first pre-train on synthetic data and then fine-tune on authentic normalization data. It achieves the best performance by a wide margin in intrinsic evaluation, and also the best performance in extrinsic evaluation through dependency parsing. In addition to these fine-tuned models, we also release the source files on [GitHub](https://github.com/ufal/multilexnorm2021) and an interactive demo on [Google Colab](https://colab.research.google.com/drive/1rxpI8IlKk-D2crFqi2hdzbTBIezqgsCg?usp=sharing).


## How to use


The model was *not* fine-tuned in a standard sentence-to-sentence setting – instead, it was tailored to the token-to-token definition of MultiLexNorm data. Please refer to [**the interactive demo on Colab notebook**](https://colab.research.google.com/drive/1rxpI8IlKk-D2crFqi2hdzbTBIezqgsCg?usp=sharing) to learn how to use these models.


## How to cite


```bibtex
@inproceedings{wnut-ufal,
  title= "{ÚFAL} at {MultiLexNorm} 2021: Improving Multilingual Lexical Normalization by Fine-tuning {ByT5}",
  author = "Samuel, David and Straka, Milan",
  booktitle = "Proceedings of the 7th Workshop on Noisy User-generated Text (W-NUT 2021)",
  year = "2021",
  publisher = "Association for Computational Linguistics",
  address = "Punta Cana, Dominican Republic"
}
```


## ByT5 - Small


ByT5 is a tokenizer-free version of [Google's T5](https://ai.googleblog.com/2020/02/exploring-transfer-learning-with-t5.html) and generally follows the architecture of [MT5](https://huggingface.co/google/mt5-small).


ByT5 was only pre-trained on [mC4](https://www.tensorflow.org/datasets/catalog/c4#c4multilingual) excluding any supervised training with an average span-mask of 20 UTF-8 characters. Therefore, this model has to be fine-tuned before it is useable on a downstream task.


ByT5 works especially well on noisy text data,*e.g.*, `google/byt5-small` significantly outperforms [mt5-small](https://huggingface.co/google/mt5-small) on [TweetQA](https://arxiv.org/abs/1907.06292).


Paper: [ByT5: Towards a token-free future with pre-trained byte-to-byte models](https://arxiv.org/abs/2105.13626)


Authors: *Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, Colin Raffel*


model name: byt5-small-multilexnorm2021-en


model author: ufal


task: text2text-generation


tags: pytorch, t5, text2text-generation, en, dataset:mc4, dataset:wikipedia, dataset:multilexnorm, arxiv:2105.13626, arxiv:1907.06292, transformers, lexical normalization, license:apache-2.0, autonlp_compatible


language:


- tr
- de
datasets:
- mc4
- wikipedia
- multilexnorm
tags:
- lexical normalization
license: apache-2.0


# Fine-tuned ByT5-small for MultiLexNorm (Turkish-German version)


## How to use


model name: byt5-small-multilexnorm2021-trde


tags: pytorch, t5, text2text-generation, tr, de, dataset:mc4, dataset:wikipedia, dataset:multilexnorm, arxiv:2105.13626, arxiv:1907.06292, transformers, lexical normalization, license:apache-2.0, autonlp_compatible


language: da
datasets:


- mc4
- wikipedia
- multilexnorm
tags:
- lexical normalization
license: apache-2.0


# Fine-tuned ByT5-small for MultiLexNorm (Danish version)


![model image](https://github.com/ufal/multilexnorm2021/raw/master/img/overall.png)


This is the official release of the fine-tuned models for **the winning entry** to the [*W-NUT 2021: Multilingual Lexical Normalization (MultiLexNorm)* shared task](https://noisy-text.github.io/2021/multi-lexnorm.html), which evaluates lexical-normalization systems on 12 social media datasets in 11 languages.


Our system is based on [ByT5](https://arxiv.org/abs/2105.13626), which we first pre-train on synthetic data and then fine-tune on authentic normalization data. It achieves the best performance by a wide margin in intrinsic evaluation, and also the best performance in extrinsic evaluation through dependency parsing. In addition to these fine-tuned models, we also release the source files on [GitHub](https://github.com/ufal/multilexnorm2021) and an interactive demo on [Google Colab](https://colab.research.google.com/drive/1rxpI8IlKk-D2crFqi2hdzbTBIezqgsCg?usp=sharing).


The model was *not* fine-tuned in a standard sentence-to-sentence setting – instead, it was tailored to the token-to-token definition of MultiLexNorm data. Please refer to [**the interactive demo on Colab notebook**](https://colab.research.google.com/drive/1rxpI8IlKk-D2crFqi2hdzbTBIezqgsCg?usp=sharing) to learn how to use these models.


## How to cite


```bibtex
@inproceedings{wnut-ufal,
  title= "{ÚFAL} at {MultiLexNorm} 2021: Improving Multilingual Lexical Normalization by Fine-tuning {ByT5}",
  author = "Samuel, David and Straka, Milan",
  booktitle = "Proceedings of the 7th Workshop on Noisy User-generated Text (W-NUT 2021)",
  year = "2021",
  publisher = "Association for Computational Linguistics",
  address = "Punta Cana, Dominican Republic"
}
```


## ByT5 - Small


ByT5 is a tokenizer-free version of [Google's T5](https://ai.googleblog.com/2020/02/exploring-transfer-learning-with-t5.html) and generally follows the architecture of [MT5](https://huggingface.co/google/mt5-small).


ByT5 was only pre-trained on [mC4](https://www.tensorflow.org/datasets/catalog/c4#c4multilingual) excluding any supervised training with an average span-mask of 20 UTF-8 characters. Therefore, this model has to be fine-tuned before it is useable on a downstream task.


ByT5 works especially well on noisy text data,*e.g.*, `google/byt5-small` significantly outperforms [mt5-small](https://huggingface.co/google/mt5-small) on [TweetQA](https://arxiv.org/abs/1907.06292).


Paper: [ByT5: Towards a token-free future with pre-trained byte-to-byte models](https://arxiv.org/abs/2105.13626)


Authors: *Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, Colin Raffel*


model name: byt5-small-multilexnorm2021-da


model author: ufal


task: text2text-generation


tags: pytorch, t5, text2text-generation, da, dataset:mc4, dataset:wikipedia, dataset:multilexnorm, arxiv:2105.13626, arxiv:1907.06292, transformers, lexical normalization, license:apache-2.0, autonlp_compatible


language: sl
datasets:


# Fine-tuned ByT5-small for MultiLexNorm (Slovenian version)


## How to use


model name: byt5-small-multilexnorm2021-sl


tags: pytorch, t5, text2text-generation, sl, dataset:mc4, dataset:wikipedia, dataset:multilexnorm, arxiv:2105.13626, arxiv:1907.06292, transformers, lexical normalization, license:apache-2.0, autonlp_compatible


tags:


- conversational


# Discord DialoGPT Model


model name: DialoGPT-small-michelle-discord-bot


model author: michelleshx


task: conversational


tags: pytorch, gpt2, text-generation, transformers, conversational


language: "ca"
tags:


- masked-lm
- catalan
- exbert
license: mit


# Calbert: a Catalan Language Model


## Introduction


CALBERT is an open-source language model for Catalan pretrained on the ALBERT architecture.


It is now available on Hugging Face in its `tiny-uncased` version (the one you're looking at) and `base-uncased` as well, and was pretrained on the [OSCAR dataset](https://traces1.inria.fr/oscar/).


For further information or requests, please go to the [GitHub repository](https://github.com/codegram/calbert)


## Pre-trained models


| Model                               | Arch.          | Training data          |
| ----------------------------------- | -------------- | ---------------------- |
| `codegram` / `calbert-tiny-uncased` | Tiny (uncased) | OSCAR (4.3 GB of text) |
| `codegram` / `calbert-base-uncased` | Base (uncased) | OSCAR (4.3 GB of text) |


## How to use Calbert with HuggingFace


#### Load Calbert and its tokenizer:


```python
from transformers import AutoModel, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("codegram/calbert-tiny-uncased")
model = AutoModel.from_pretrained("codegram/calbert-tiny-uncased")

model.eval() # disable dropout (or leave in train mode to finetune
```


#### Filling masks using pipeline