devasheeshG's picture
Upload 2 files
b289b78
metadata
license: apache-2.0
pipeline_tag: automatic-speech-recognition
tags:
  - pytorch
  - audio
  - speech
  - automatic-speech-recognition
  - whisper
  - wav2vec2
model-index:
  - name: whisper_large_v2_fp16_transformers
    results:
      - task:
          type: automatic-speech-recognition
          name: Automatic Speech Recognition
        dataset:
          type: librispeech_asr
          name: LibriSpeech (clean)
          config: clean
          split: test
          args:
            language: en
        metrics:
          - type: wer
            value: 0
            name: Test WER
            description: Word Error Rate
          - type: mer
            value: 0
            name: Test MER
            description: Match Error Rate
          - type: wil
            value: 0
            name: Test WIL
            description: Word Information Lost
          - type: wip
            value: 0
            name: Test WIP
            description: Word Information Preserved
          - type: cer
            value: 0
            name: Test CER
            description: Character Error Rate
      - task:
          type: automatic-speech-recognition
          name: Automatic Speech Recognition
        dataset:
          type: librispeech_asr
          name: LibriSpeech (other)
          config: other
          split: test
          args:
            language: en
        metrics:
          - type: wer
            value: 0
            name: Test WER
            description: Word Error Rate
          - type: mer
            value: 0
            name: Test MER
            description: Match Error Rate
          - type: wil
            value: 0
            name: Test WIL
            description: Word Information Lost
          - type: wip
            value: 0
            name: Test WIP
            description: Word Information Preserved
          - type: cer
            value: 0
            name: Test CER
            description: Character Error Rate
      - task:
          type: automatic-speech-recognition
          name: Automatic Speech Recognition
        dataset:
          type: mozilla-foundation/common_voice_14_0
          name: Common Voice (14.0) (Hindi)
          config: hi
          split: test
          args:
            language: hi
        metrics:
          - type: wer
            value: 44.64
            name: Test WER
            description: Word Error Rate
          - type: mer
            value: 41.69
            name: Test MER
            description: Match Error Rate
          - type: wil
            value: 59.53
            name: Test WIL
            description: Word Information Lost
          - type: wip
            value: 40.46
            name: Test WIP
            description: Word Information Preserved
          - type: cer
            value: 16.8
            name: Test CER
            description: Character Error Rate
widget:
  - example_title: Hinglish Sample
    src: >-
      https://huggingface.co/devasheeshG/whisper_large_v2_fp16_transformers/resolve/main/test.wav
  - example_title: Librispeech sample 1
    src: https://cdn-media.huggingface.co/speech_samples/sample1.flac
  - example_title: Librispeech sample 2
    src: https://cdn-media.huggingface.co/speech_samples/sample2.flac
language:
  - en
  - zh
  - de
  - es
  - ru
  - ko
  - fr
  - ja
  - pt
  - tr
  - pl
  - ca
  - nl
  - ar
  - sv
  - it
  - id
  - hi
  - fi
  - vi
  - he
  - uk
  - el
  - ms
  - cs
  - ro
  - da
  - hu
  - ta
  - 'no'
  - th
  - ur
  - hr
  - bg
  - lt
  - la
  - mi
  - ml
  - cy
  - sk
  - te
  - fa
  - lv
  - bn
  - sr
  - az
  - sl
  - kn
  - et
  - mk
  - br
  - eu
  - is
  - hy
  - ne
  - mn
  - bs
  - kk
  - sq
  - sw
  - gl
  - mr
  - pa
  - si
  - km
  - sn
  - yo
  - so
  - af
  - oc
  - ka
  - be
  - tg
  - sd
  - gu
  - am
  - yi
  - lo
  - uz
  - fo
  - ht
  - ps
  - tk
  - nn
  - mt
  - sa
  - lb
  - my
  - bo
  - tl
  - mg
  - as
  - tt
  - haw
  - ln
  - ha
  - ba
  - jw
  - su

Versions:

  • CUDA: 12.1
  • cuDNN Version: 8.9.2.26_1.0-1_amd64
  • tensorflow Version: 2.12.0
  • torch Version: 2.1.0.dev20230606+cu12135
  • transformers Version: 4.30.2
  • accelerate Version: 0.20.3

Model Benchmarks:

  • RAM: 3 GB (Original_Model: 6GB)

  • VRAM: 3.7 GB (Original_Model: 11GB)

  • test.wav: 23 s (Multilingual Speech i.e. English+Hindi)

    • Time in seconds for Processing by each device
    Device Name float32 (Original) float16 CudaCores TensorCores
    3060 2.2 1.3 3,584 112
    1660 Super OOM 6 1,408 N/A
    Collab (Tesla T4) - - 2,560 320
    Collab (CPU) - N/A N/A N/A
    M1 (CPU) - - N/A N/A
    M1 (GPU -> 'mps') - - N/A N/A
    • NOTE: TensorCores are efficient in mixed-precision calculations
    • CPU -> torch.float16 not supported on CPU (AMD Ryzen 5 3600 or Collab CPU)
  • Punchuation: Sometimes False ('I don't know the exact reason why this is happening')

Model Error Benchmarks:

  • WER: Word Error Rate
  • MER: Match Error Rate
  • WIL: Word Information Lost
  • WIP: Word Information Preserved
  • CER: Character Error Rate

Hindi to Hindi (test.tsv) Common Voice 14.0

Test done on RTX 3060 on 1000 Samples

WER MER WIL WIP CER
Original_Model (30 min) 43.99 41.65 59.47 40.52 16.23
This_Model (20 min) 44.64 41.69 59.53 40.46 16.80

Hindi to English (test.csv) Custom Dataset

Test done on RTX 3060 on 1000 Samples

WER MER WIL WIP CER
Original_Model (30 min) - - - - -
This_Model (20 min) - - - - -

English (LibriSpeech -> test-clean)

Test done on RTX 3060 on ___ Samples

WER MER WIL WIP CER
Original_Model - - - - -
This_Model - - - - -

English (LibriSpeech -> test-other)

Test done on RTX 3060 on ___ Samples

WER MER WIL WIP CER
Original_Model - - - - -
This_Model - - - - -
  • 'jiwer' library is used for calculations

Code for conversion:

Usage

A file __init__.py is contained inside this repo which contains all the code to use this model.

Firstly, clone this repo and place all the files inside a folder.

Make sure you have git-lfs installed (https://git-lfs.com)

git lfs install
git clone https://huggingface.co/devasheeshG/whisper_large_v2_fp16_transformers

Please try in jupyter notebook

# Import the Model
from whisper_large_v2_fp16_transformers import Model, load_audio, pad_or_trim
# Initilise the model
model = Model(
            model_name_or_path='whisper_large_v2_fp16_transformers',
            cuda_visible_device="0",
            device='cuda',
      )
# Load Audio
audio = load_audio('whisper_large_v2_fp16_transformers/test.wav')
audio = pad_or_trim(audio)
# Transcribe (First transcription takes time)
model.transcribe(audio)

Credits

It is fp16 version of openai/whisper-large-v2