LlamaHub: The one stop data solution for your LLMs🦙🏠
Tags: python, ai, machinelearning, tutorial
Published: 2023-06-26 10:21:00

In the 4th part of this blog series, we'll have a look at the most popular adapters available at [llamahub.ai](llamahub.ai). LLaMahub is a library of data connectors that translate raw data from various sources into vector(s) aiding in ease of access and transformation by LLMs like ChatGPT and Google Bard.

This is an open-source project where the adapters/connectors that are created would slowly get integrated into projects like Langchain and Llama-index. If you are building a custom document bot for your firm or even for your personal use I think you would find this super useful.

![Llamahub](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vv5x8ywy435iljyz5uux.png)

Now, lets get started !

---

For the sake of simplicity in this walkthrough, We will access the adapters of llamahub through Llama-Index

## Adapters Assemble :hammer:⚡

First up here's how you could load any loader with Llama-Index:

```python
from llama_index import download_loader

Loader = download_loader("<NAME_OF_THE_LOADER>")
```
the `download_loader` helper method will make sure to load the mentioned loader along with all the needed dependencies. To check the name of the loader that you'd want to use, visit this [documentation](https://gpt-index.readthedocs.io/en/latest/reference/readers.html). With that out of the way, lets have a look at some of the important data loaders in Llama-Index.

###`SimpleDirectoryReader`: 
This is a universal data connector available in the llama-index that should be able to handle pretty much all the file types you throw at it. Under the hood it uses various third party libraries like `pypdf2` for PDFs, `python-pptx` for PPTs etc... The general idea behind this data connector is to read a directory of different file types in entirety or to read single files, doesn't matter what kind of file it is.

```python
SimpleDirectoryReader = download_loader("SimpleDirectoryReader")
documents = SimpleDirectoryReader(input_dir='foo/').load_data()
```
###`SimpleWebPageReader`: 
This is the web equivalent of the `SimpleDirectoryReader`. This can read any webpage by using `BeautifulSoup4`.

```python
SimpleWebPageReader = download_loader("SimpleWebPageReader")
documents = SimpleWebPageReader().load_data(urls=['https://foo.com/index'])
```

###`YoutubeTranscriptReader`: 
This loader will attempt to transcript any youtube video that you provide. It will first try to search for any available captions using the `youtube-transcript-api` and if the captions are not available, it will use OpenAI's [whisper API](https://openai.com/research/whisper) to transcript the video.

```python
YoutubeTranscriptReader = download_loader("YoutubeTranscriptReader")
documents = YoutubeTranscriptReader().load_data(ytlinks=['https://youtu.be/5MuIMqhT8DM'])
```
---

The above are some of the loaders that I think have a lot of use cases. But there are more and you can find the complete list [here](https://gpt-index.readthedocs.io/en/latest/reference/readers.html). If you want to load the documents into langchain's `Document` format, you can use `<Loader>.load_langchain_documents()` instead of `load_data`, it will output the documents in the langchain supported format. 

Once you load the documents, it is a cake walk from there. You can visit my old blogs in this series on how you could use this as context for the ChatGPT API.

Now, that's about LlamaHub, See ya in the next one :wink: