🔉From Sound to Insights: Using AI🤖 for Audio File Transcription and Analysis!🚀
Tags: python, ai, beginners
Published: 2024-06-28 20:57:16

If we were given an audio file, is there any way we could identify the time stamps where specific words were said? Is there any way we could extract all the key words mentioned about a topic?

**With AI 🤖, we can do all of this and much more!** The key lies in being able to parse audio into text, allowing us to then harness the natural language processing capabilities of _language models_ to perform sophisticated analyses and inferences on our data.

Regardless of who you are, such an approach to audio transcription and analysis will augment how you interact with and extract knowledge from audio files.

Let's see how we can do this with `llmware`.

---

## AI Tools 🤖
We'll be using two models for this example.

The first is Whisper by OpenAI. This is the model that will allow us to parse the audio files, i.e. convert them from audio to text.

The second is the SLIM (_Structured Language Instruction Model_) Extract Tool by LLMWare, which we'll be using to ask questions about our audio. This is a _GGUF quantized_ version of a much larger model called _slim-extract_. All this means is that our model, the SLIM Extract Tool, is a smaller and faster version of the original model. **This allows us to run it locally on a CPU**, without the need for powerful computational resources like GPUs!

With that out of the way, let's get started with the example.

---

## Step 1: Loading in audio files 🔉🔉
If you have audio files that you want to run the example with, then feel free to use those by setting `input_folder` appropriately, but if not, the `llmware` library provides you with several sets of sample audio files!
```python
voice_sample_files = Setup().load_voice_sample_files(small_only=False)
input_folder = os.path.join(voice_sample_files, "greatest_speeches")
```
Here, we're loading in the `greatest_speeches` set of audio files.

---

## Step 2: Parsing our audio files 📝
Now that we have our audio files, we can go about parsing them into chunks of text. Recall that we'll be needing the WhisperCPP model to do this. But fortunately, you won't have to directly interact with the model yourself since the `Parser` class from the `llmware` library will take care of this for you!
```python
parser_output = Parser(chunk_size=400, max_chunk_size=600).parse_voice(input_folder, write_to_db=False, copy_to_library=False, remove_segment_markers=True, chunk_by_segment=True, real_time_progress=False)
```
Here, the `chunk_size` and `max_chunk_size` indicate how big each chunk of parsed text will be. We're passing in our folder containing the audio files to the `parse_voice()` function of the `Parser` class.

The function does accept many more optional arguments about how we'd like the audio to be parsed, but we can ignore them for this example.

---

## Step 3: Text searching 🕵️
Let's now run a text search on our parsed audio. We can try searching for the word "president". What this means is that we want to find all the portions of the audio and corresponding text that have the word "president" in it. We can do this using the `fast_search_dicts()` function in the `Utilies` class in the `llmware` library.
```python
results = Utilities().fast_search_dicts("president", parser_output)
```

---

## Step 4: Making an AI call on text chunks 🤖
Now that we have a list of text blocks containing the word "president", lets use an AI model to identify which presidents are being mentioned in the selected text blocks.
```python
extract_model = ModelCatalog().load_model("slim-extract-tool", sample=False, temperature=0.0, max_output=200)
```
Here, we're using the `ModelCatalog` class to load in our SLIM Extract Tool. Let's now iterate over each text block containing "president".
```python
final_list = []
for i, res in enumerate(results):
    response = extract_model.function_call(res["text"], params=["president name"])
```
We're making a `function_call()` for "president name". **This is how we ask our Tool to identify the president name in the text block.**

---

## Step 5: Analyzing our output 🔍
The `function_call()` function would have returned a dictionary containing a lot of data about the function call. We specifically want the `president_name` key in the dictionary.
```python
extracted_name = ""
if "president_name" in response["llm_response"]:
    if len(response["llm_response"]["president_name"]) > 0:
        extracted_name = response["llm_response"]["president_name"][0].lower()
    else:
        print("\nupdate: skipping result - no president name found - ", response["llm_response"], res["text"])
```
If the value of the `president_name` key is a non-empty string, then we store its value in `extracted_name`. Otherwise, no result was found and we print this out.

Now lets see if the president name matched any of the recent American presidents in this list:
```python
various_american_presidents = ["kennedy", "carter", "nixon", "reagan", "clinton", "obama"]
```

To do this, we'll check if the `extracted_name` contains any of these American presidents. If we have a match, then we'll add it to our `final_list` as a dictionary containing some information about the location of the name in the audio as well as the text block it was in.
```python
for president in various_american_presidents:
    if president in extracted_name:
        final_list.append({"key": president, "source": res["file_source"], "time_start": res["coords_x"], "text": res["text"]})
```

---

## Results! ✅
Let's now output the `final_list`.
```python
for i, f in enumerate(final_list):
    print("final results: ", i, f)
```
This is what an one search result in the output would look after running the code.

![Sample output](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/z3g0jpwu6s1vq2mvwtjc.png)

Here, we have a Python dictionary as output containing:
- `key`: the name of the president identified, which here is "kennedy"
- `source`: the audio file this was found in, which here is "ConcessionStand.wav"
- `time_start`: the time stamp in seconds where the president was mentioned, which here is 339.9 seconds
- `text`: which contains the text chunk the name was found in.

---

## Conclusion
And we're done! To recap, we were able to parse our audio files into text, run a text search on them for the word "president", and then use our SLIM Extract Tool to identify the specific presidents named in our text chunks! **And remember that we did all this on just a CPU! 💻**

Be sure to check out our YouTube video on this example!
{% embed https://www.youtube.com/watch?v=5y0ez5ZBpPE&t=804s&ab_channel=llmware %}

If you made it this far, thank you for taking the time to go through this topic with us ❤️! For more content like this, make sure to [visit our dev.to page](https://dev.to/llmware).

The source code for many more examples like this one are on [our GitHub](https://github.com/llmware-ai/llmware). Find this example [here](https://github.com/llmware-ai/llmware/blob/main/examples/Use_Cases/parsing_great_speeches.py).

Our repository also contains a [notebook for this example](https://github.com/llmware-ai/llmware/blob/main/examples/Notebooks/NoteBook_Examples/parsing_great_speeches_notebook.ipynb) that you can run yourself using Google Colab, Jupyter or any other platform that supports .ipynb notebooks.

Lastly, [join our Discord](https://discord.gg/fCztJQeV7J) to interact with a growing community of AI enthusiasts of all levels of experience!