Transcribe audio to text
Tags: ai, llm, rag, vectordatabase
Published: 2021-03-27 18:44:29

[![](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/11_Transcribe_audio_to_text.ipynb)

This article covers the transcription of audio files to text using models provided by Hugging Face.

# Install dependencies

Install `txtai` and all dependencies. Since this article is using optional pipelines, we need to install the pipeline extras package. We'll also demonstrate running this pipeline through the API.

```
pip install txtai[api,pipeline]

# Get test data
wget -N https://github.com/neuml/txtai/releases/download/v3.5.0/tests.tar.gz
tar -xvzf tests.tar.gz
```

# Create a Transcription instance

The Transcription instance is the main entrypoint for transcribing audio to text. The pipeline abstracts transcribing audio into a one line call! 

The pipeline executes logic to read audio files into memory, run the data through a machine learning model and output the results to text.

```python
from txtai.pipeline import Transcription

# Create transcription model
transcribe = Transcription()
```

# Transcribe audio to text

The example below shows how to transcribe a list of audio files to text. Let's transcribe audio to text and look at each result.

```python
from IPython.display import Audio, display

files = ["Beijing_mobilises.wav", "Canadas_last_fully.wav", "Maine_man_wins_1_mil.wav", "Make_huge_profits.wav", "The_National_Park.wav", "US_tops_5_million.wav"]
files = ["txtai/%s" % x for x in files]

for x, text in enumerate(transcribe(files)):
  display(Audio(files[x]))
  print(text)
  print()
```

```
Baging mobilizes invasion kraft along coast as tie one tensions escalates
Canodas last fully intact ice shelf has suddenly collapsed forming a manhattan sized iceberge
Main man wins from lottery ticket
Make huge profits without working make up to one hundred thousand dollars a day
National park service warns against sacrificing slower friends in a bare attack
Ues virus cases top a million
```

Overall, the results are solid. Each result sounds phonetically like the audio.

# OpenAI Whisper

In September 2022, [OpenAI Whisper](https://github.com/openai/whisper) was released. This model brings a dramatic improvement in transcription quality. Whisper support was added to Hugging Face Transformers in v4.23.0. Let's give it a try.

```python
transcribe = Transcription("openai/whisper-base")
for text in transcribe(files):
  print(text)
```

```
Beijing mobilizes invasion craft along coast as Taiwan tensions escalate.
Canada's last fully intact ice shelf has suddenly collapsed, forming a Manhattan sized iceberg.
Maine Man wins from lottery ticket.
make huge profits without working. Make up to $100,000 a day.
National Park Service warns against sacrificing slower friends in a bear attack.
U.S. virus cases top of million.
```

Results were transcribed with near perfect accuracy, amazing!

This can also be run as a txtai application or API instance. Let's try a full indexing workflow with a txtai application.

```yaml
writable: true

embeddings:
  path: sentence-transformers/nli-mpnet-base-v2
  content: true

transcription:
  path: openai/whisper-base

workflow:
  index:
    tasks:
      - transcription
      - index
```

```python
from txtai.app import Application

app = Application("workflow.yml")

list(app.workflow("index", files))
app.search("feel good story", 1)
```

```
[{'id': '2',
  'text': 'Maine Man wins from lottery ticket.',
  'score': 0.1285860687494278}]
```

This workflow transcribed the input files, loaded the transcriptions into an embeddings index and finally ran a search. Last thing we'll do is run the workflow as an API instance.

```
CONFIG=workflow.yml uvicorn "txtai.api:app" &> api.log &
sleep 30

# Run indexing workflow
curl -s -o /dev/null \
  -X POST "http://localhost:8000/workflow" \
  -H "Content-Type: application/json" \
  -d '{"name":"index", "elements":["txtai/Beijing_mobilises.wav", "txtai/Canadas_last_fully.wav", "txtai/Maine_man_wins_1_mil.wav", "txtai/Make_huge_profits.wav", "txtai/The_National_Park.wav", "txtai/US_tops_5_million.wav"]}'

# Test API search
curl "http://localhost:8000/search?query=feel+good+story&limit=1"
```

```
[{"id":"2","text":"Maine Man wins from lottery ticket.","score":0.1285860687494278}]
```

Once again, the same results as in Python and with an application.

# Wrapping up

There is a lot of development in the audio transcription space. In only a couple of lines of code, high-quality transcription models are now readily available!