How to use LangChain with OpenAI, Pinecone, and Apify
Tags: ai, chatgpt, llms
Published: 2023-10-13 11:32:28

---
title: How to use LangChain with OpenAI, Pinecone, and Apify
published: true
date: 2023-07-11 22:00:00 UTC
tags: ai, chatgpt, llms
canonical_url: https://blog.apify.com/how-to-use-langchain/
---

 **Were** [**Apify**](https://apify.it/data-for-ai) **, and our mission is to make the web more programmable. Part of that is getting data for AI.** [**Check us out**](https://apify.it/data-for-ai) **.**

OpenAIs ChatGPT has rapidly evolved into a very powerful and versatile tool. People have been using it to generate text, answer questions, translate languages, and much more.

/callout 🎓 Get fresh [data for your LLM](https://apify.com/data-for-generative-ai)

Trained on data up to September 2021, a big ChatGPT limitation is its inability to answer questions based on more recent information. So, how can we use OpenAI to create a system that can answer questions from current data?

/callout 📢 As of September 27, 2023, GPT-4's knowledge is no longer limited to data before September 2021: 

![ChatGPT can browse the internet](https://blog.apify.com/content/images/2023/09/ChatGPT-can-browse-the-internet-3.jpeg)

Several tools, including [LangChain](https://blog.apify.com/what-is-langchain/), [Pinecone](https://blog.apify.com/what-is-pinecone-why-use-it-with-llms/), and Apify, offer the ability to extend and enhance the capabilities of OpenAI's AI models.

1. [**LangChain**](https://python.langchain.com/docs/get_started/introduction.html): a framework designed for the development of applications that leverage language models. It allows us to integrate large language models like [OpenAI](https://openai.com/) and [Hugging Face](https://blog.apify.com/what-is-hugging-face/) with different applications.

2. [**Pinecone**](https://www.pinecone.io/): an external tool that allows us to save the embeddings online and extract them whenever we need.

3. [**Apify**](https://apify.com/): a [web scraping](https://apify.com/web-scraping) and automation platform that significantly streamlines the process of data collection. It provides the capability to scrape data and subsequently feed it to LLMs. This allows us to train LLMs on real-time data and develop applications.

In this tutorial, we will extract data using one of Apifys many pre-built web scraping tools, [Airbnb Scraper](https://apify.com/dtrungtin/airbnb-scraper). We'll scrape Airbnb data from **New York City** and feed it to the LLM to make a system that will answer questions from that data.

📔 **Note** : You can find the complete notebook on this [Google Colab](https://colab.research.google.com/drive/1ae9V9b32gYqBG6zuAjKDh3hj1tX_gG3t?usp=sharing) link.

[https://youtu.be/8uvHH-ocSes](https://youtu.be/8uvHH-ocSes)

## **How to set up LangChain and Apify**

To start setting up LangChain and Apify, you'll need to create a new directory and a Python file. You can do this by opening your terminal or command line and entering the following commands:

```
mkdir LangChain
cd LangChain
touch main.ipynb

```

Let's install the packages. Copy the command below, paste it into your terminal, and press Enter. These packages will provide the tools and libraries we need to develop our AI web scraping application.

```
pip3 install langchain==0.0.189 pinecone-client openai tiktoken nest_asyncio apify-client chromadb

```

This should install all the dependencies in your system. To confirm that everything is installed properly, you can enter the following command in your terminal.

```
pip freeze | egrep '(langchain|pinecone-client|openai|tiktoken|nest_asyncio|apify-client|chromadb)'

```

This should include all the installed dependencies with their versions. If you spot any missing dependencies, you may need to re-run the installation command for that specific package.

Now, we're ready to write our code once we're done with the installation.

📔 **Note** : We will be working in a notebook. Consider every code block a notebook cell.

[https://blog.apify.com/what-is-langchain/](https://blog.apify.com/what-is-langchain/)

## **Set up the environment variables**

First, we must provide the API keys for OpenAI and Apify to set up the environment variables. You need to replace the empty quotes with your own API keys.

```
import os
# Add your OPENAI API key below
os.environ["OPENAI_API_KEY"] = ""
# Add your APIFY API key below
os.environ["APIFY_API_TOKEN"] = ""

```

## **How to use Airbnb Scraper to extract data**

Now, we will use the Apify API to scrape data from Airbnb listings and then save that data into a [structured format](https://blog.apify.com/when-data-gets-too-big-why-you-need-structured-data/) (as `Document` objects). `ApifyWrapper` from `langchain.utilities` allows us to interact with Apify, `Document` from `langchain.document_loaders.base` is used to structure the scraped data, and `json` is a standard Python library for working with JSON data.

```
from langchain.utilities import ApifyWrapper
from langchain.document_loaders.base import Document
import json

```

Next, we'll create an instance of `ApifyWrapper`. This object will provide us with methods for interacting with the Apify API and help us to scrape data from Airbnb listings.

```
apify = ApifyWrapper()

```

Now, we need to scrape data through Airbnb Scraper. There are two ways to use this scraper:

**1. Apify Console** : You can go to Apify Console and open [Airbnb Scraper](https://console.apify.com/actors/GsNzxEKzE2vQ5d9HN/console). From there, you can choose the data you want to select according to your needs. If you have no preferences, you can just enter the city and press **Start.**

![Airbnb Scraper console page](https://blog.apify.com/content/images/2023/07/airbnb-scraper.png)

_Airbnb Scraper console page_

**2. Apify API** : We can use the Apify API to extract data. We will use this method in this tutorial. You can read more about the [Airbnb Scraper API](https://apify.com/dtrungtin/airbnb-scraper/api) on the README and change the data you want to extract.

Next, we run the [Actor](https://apify.com/actors), which is basically our Airbnb scraper, and wait for it to finish the execution. It will fetch the results in a LangChain document loader.

```
loader = []
loader = apify.call_actor(
    actor_id = "dtrungtin/airbnb-scraper",
    run_input = {
        "currency": "USD",
        "maxListings":500,
        "locationQuery": "New York, ",
        "proxyConfiguration": {"useApifyProxy": True},
        "maxConcurrency": 50,
        "limitPoints": 100,
        "timeoutMs": 300000,
    },
    dataset_mapping_function = lambda item: Document(
        page_content = json.dumps({
            'name': item['name'],
            'room_type': item['roomType'],
            'city': item['city'],
            'stars': item['stars'],
            'address': item['address'],
            'number_of_guests': item['numberOfGuests'],
            'pricing': {
                'rate': "$"+ str(item['pricing']['rate']['amount']),
                'currency': item['pricing']['rate']['currency'],
            },
            'url': item['url'],
            'host': {
                'first_name': item['primaryHost']['firstName'],
                'is_superhost': item['primaryHost']['isSuperHost'],
                'about': item['primaryHost']['about'],
                'member_since': item['primaryHost']['memberSince'],
                'languages': item['primaryHost']['languages'],
                'badges': item['primaryHost']['badges']
            },
            'reviews': [
                {
                    'author': {
                        'first_name': review['author']['firstName'],
                    },
                    'comments': review['comments'],
                    'created_at': review['createdAt'],
                    'rating': review['rating'],
                    'localized_date': review['localizedDate']
                }
                for review in item['reviews']
            ]
        }),
       metadata = {"source": item["url"]}

    )
)

```

While the Actor is running, especially if it's scraping a large dataset, you can monitor its progress by going to [Apify Console](https://console.apify.com/actors/runs).

📔 **Note:** The Actor call function can take some time as it loads the data from the Airbnb website.

Once the execution is completed, all the data will also be saved on the Apify platform. You can learn to load the extracted data in [this notebook](https://github.com/hwchase17/langchain/blob/fe1eb8ca5f57fcd7c566adfc01fa1266349b72f3/docs/modules/indexes/document_loaders/examples/apify_dataset.ipynb), but we will not load data from Apify in this tutorial. In that notebook, you'll also find an explanation of the `dataset_mapping_function`, which is used to map fields from the Apify dataset records to LangChain `Document` fields.

[https://blog.apify.com/what-is-generative-ai/](https://blog.apify.com/what-is-generative-ai/)

## **Load and process data with LangChain and Pinecone**

Next, let's explore how to use LangChain for loading and processing the data we've scraped, and look at how Pinecone contributes to this process.

## **Get the data from the document**

```
loader = loader.load()

```

## **Split the data into chunks**

Due to memory limitations and computational efficiency, language models, including those used by LangChain, can only process a certain amount of text at a time. By splitting the data into smaller chunks, we can feed these chunks into the language model one at a time, which can help to improve the performance of the LLM.

```
from langchain.text_splitter import RecursiveCharacterTextSplitter
text_splitter = RecursiveCharacterTextSplitter(
    chunk_size = 1200,
    chunk_overlap = 200
)
docs_chunks = text_splitter.split_documents(loader)
print(docs_chunks)

```

After executing this code, we will see documents with a chunk size of `1200` characters, and an overlap of `200` to maintain the semantics.

## **Create and save the embeddings to Pinecone**

Initialize Pinecone by providing values to the `api_key` and `environment` arguments.

```
import pinecone
pinecone.init(
    api_key = "",
    environment = ""
)

```

Once Pinecone has been initialized, we can set up an index, which is like a database for storing and querying our embeddings. We provide a name for our index and specify the type of embeddings we're using.

```
from langchain.vectorstores import Pinecone
from langchain.embeddings.openai import OpenAIEmbeddings
index_name = "index_name"
embeddings = OpenAIEmbeddings()
#create a new index
docsearch = Pinecone.from_documents(docs_chunks, embeddings, index_name=index_name)
Load the embeddings

```

A new index with the name `index_name` will be created, and all the embeddings will be stored. There are other types of embeddings, but we will use `OpenAIEmbeddings()`.

[https://blog.apify.com/what-is-pinecone-why-use-it-with-llms/](https://blog.apify.com/what-is-pinecone-why-use-it-with-llms/)

## **Load the embeddings**

Next, we'll load the Pinecone index we created in the previous step. This means we're preparing to access and use the stored embeddings.

```
testingIndex = Pinecone.from_existing_index(index_name, embeddings)

```

## **Use similarity search**

Now, we have embeddings, and LangChain allows us to make a [similarity search](https://python.langchain.com/docs/modules/data_connection/vectorstores/#similarity-search) against our query. A similarity search works by calculating the distance between the embeddings of our query and the embeddings of each item in our dataset. It then returns the items with the "nearest" (most similar) embeddings.

```
query = "Which is the cheapest airbnb room in newyork?"
testingIndex.similarity_search(
    query, # our search query
    k=3 # return 3 most relevant docs
)

```

It creates the embeddings and applies similarity search for the nearest distance embeddings. It then returns the top 3 results like this:

![Embeddings and similarity search results](https://blog.apify.com/content/images/2023/07/Similarity-search-results.png)

_Top 3 similar results_

## **Create a Q/A using OpenAI and LangChain**

Let's start creating a Q/A system by importing the OpenAI library and creating an instance of the `ChatOpenAI` class.

```
from langchain.chat_models import ChatOpenAI
llm = ChatOpenAI(
    temperature = 0.0
)

```

The `temperature` parameter controls the randomness of the output. The temperature parameter controls the randomness of the output. A lower `temperature` value (like `0.0`) makes the model's output more deterministic, meaning it will generate the next word that is most statistically likely given the previous words.

Next, we need to define a chain that will receive input and give us output as an answer.

```
from langchain.chains import RetrievalQA
qa = RetrievalQA.from_chain_type(
    llm = llm,
    chain_type="refine",
    retriever=vectorstore.as_retriever()
)

```

There are different [types of chains](https://python.langchain.com/docs/modules/chains/popular/) provided by LangChain for specific domains, but we will use the [refine](https://python.langchain.com/docs/modules/chains/document/refine) chain because it loops over each document and updates the answer according to the query.

Let's make a query.

```
qa.run(query)

```

It will produce an output like this.

> **'The cheapest Airbnb room in New York is "Quiet Midtown Bargain!" with a rate of $140 per night. You can find more information about it** [**here**](https://www.airbnb.com/rooms/19528326) **.'**

The price may appear somewhat high, but this is because we have only scraped 500 listings, and it's the least expensive option among them all.

[https://blog.apify.com/webscraping-ai-data-for-llms/](https://blog.apify.com/webscraping-ai-data-for-llms/)

## **Make a more robust system with LangChain Agents**

In LangChain, [agents](https://python.langchain.com/docs/modules/agents/) are components that have access to tools provided by us, and they can use them depending on their need. Let's add an agent to our system to make it more effective and robust.

```
from langchain.agents import Tool
from langchain.agents import initialize_agent
tools = [
    Tool(
        name = 'Knowledge Base',
        func = qa.run,
        description = (
            'use this tool to answer questions'
            'more information about the topic'
        )
    )
]
agent = initialize_agent(
    tools=tools,
    llm=llm,
    verbose=True,
)
query="Which is the cheapest airbnb room in newyork?"
agent(query)

```

We are defining a tool in the above code and giving that tool to the agent. We have set the parameter `verbose` to `True`. By doing so, we can see the whole thinking process of the agent like this:

![LangChain Agent](https://blog.apify.com/content/images/2023/07/Agent-s-result.png)

_The thinking process of an Agent_

If it's set to `False`, it will just return the `input` with `output`.

📔 **Note** : We have defined just one tool. You can define more depending on your needs.

## **Add memory to the agent**

By default, all the conversations in the agents and chains are volatile; they don't remember anything. But we can add memory to save the previous conversations. Let's do that.

```
from langchain.chains.conversation.memory import ConversationBufferWindowMemory
conversational_memory = ConversationBufferWindowMemory(
    memory_key='chat_history',
    k = 5,
    return_messages = True
)

```

Let's update our agent and add memory to it.

```
newAgent = initialize_agent(
    agent = 'chat-conversational-react-description',
    tools = tools,
    llm = llm,
    verbose = True,
    max_iterations = 3,
    early_stopping_method = 'generate',
    memory = conversational_memory
)
query = "My name is Apify"
newAgent(query)

```

After adding [ConversationBufferWindowMemory](https://python.langchain.com/docs/modules/memory/how_to/buffer_window) to the agent, it'll remember the last five questions that we asked. That's because we have set the limit `k=5` . Now let's ask another question.

```
query = "What is my name?"
newAgent(query)

```

It will return a result like this:

![LangChain memory](https://blog.apify.com/content/images/2023/07/Agent-memory-result.png)

_Results after adding memory_

In the image above, we can see that it remembers the history and answers the questions related to that as well.

A few of the sample queries and their answers are attached below.

First, I asked the agent to show me the room with the best reviews:

![Scraped Airbnb reviews](https://blog.apify.com/content/images/2023/07/room-with-best-reviews.png)

_Room with best reviews_

Then I asked the agent to give me the name of the owner:

![Airbnb data: room owner](https://blog.apify.com/content/images/2023/07/room-with-owner.png)

_Owner of the room_

And that's how you can ask a dataset any question you like using LLMs when combined with LangChain and Pinecone!

[https://blog.apify.com/langchain-alternatives/](https://blog.apify.com/langchain-alternatives/)

## **Doing more with OpenAI, LangChain, and Apify**

This simple application of LangChain and Apify barely scratches the surface of what you can achieve by combining these tools. You can do a lot more by choosing any Apify Actor, integrating with OpenAI through LangChain, and developing any application you need.

If you want to see what other GPT and AI-enhanced tools you can integrate with LangChain, just check out [Apify Store](https://apify.com/store/categories/ai).