!function(){function t(t){document.documentElement.setAttribute("data-theme",t)}var e=function(){var t=null;try{t=new URLSearchParams(window.location.search).get("docusaurus-theme")}catch(t){}return t}()||function(){var t=null;try{t=localStorage.getItem("theme")}catch(t){}return t}();t(null!==e?e:"light")}(),document.documentElement.setAttribute("data-announcement-bar-initially-dismissed",function(){try{return"true"===localStorage.getItem("docusaurus.announcement.dismiss")}catch(t){}return!1}())

[Skip to main content](#__docusaurus_skipToContent_fallback)

LangChain v0.2 is coming soon! Preview the new docs [here](/v0.2/docs/introduction/).

[

![🦜️🔗 Langchain](/v0.1/img/brand/wordmark.png)![🦜️🔗 Langchain](/v0.1/img/brand/wordmark-dark.png)

](/v0.1/)[Docs](/v0.1/docs/get_started/introduction/)[Use cases](/v0.1/docs/use_cases/)[Integrations](/v0.1/docs/integrations/platforms/)[API Reference](https://api.js.langchain.com)

[More](#)

*   [People](/v0.1/docs/people/)
*   [Community](/v0.1/docs/community/)
*   [Tutorials](/v0.1/docs/additional_resources/tutorials/)
*   [Contributing](/v0.1/docs/contributing/)

[v0.1](#)

*   [v0.2](https://js.langchain.com/v0.2/docs/introduction)
*   [v0.1](/v0.1/docs/get_started/introduction/)

[🦜🔗](#)

*   [LangSmith](https://smith.langchain.com)
*   [LangSmith Docs](https://docs.smith.langchain.com)
*   [LangChain Hub](https://smith.langchain.com/hub)
*   [LangServe](https://github.com/langchain-ai/langserve)
*   [Python Docs](https://python.langchain.com/)

[Chat](https://chatjs.langchain.com)[](https://github.com/langchain-ai/langchainjs)

Search

*   [Providers](/v0.1/docs/integrations/platforms/)
    
    *   [Providers](/v0.1/docs/integrations/platforms/)
    *   [Anthropic](/v0.1/docs/integrations/platforms/anthropic/)
    *   [AWS](/v0.1/docs/integrations/platforms/aws/)
    *   [Google](/v0.1/docs/integrations/platforms/google/)
    *   [Microsoft](/v0.1/docs/integrations/platforms/microsoft/)
    *   [OpenAI](/v0.1/docs/integrations/platforms/openai/)
*   [Components](/v0.1/docs/integrations/components/)
    
    *   [LLMs](/v0.1/docs/integrations/llms/)
        
    *   [Chat models](/v0.1/docs/integrations/chat/)
        
    *   [Document loaders](/v0.1/docs/integrations/document_loaders/)
        
        *   [File Loaders](/v0.1/docs/integrations/document_loaders/file_loaders/)
            
        *   [Web Loaders](/v0.1/docs/integrations/document_loaders/web_loaders/)
            
            *   [Cheerio](/v0.1/docs/integrations/document_loaders/web_loaders/web_cheerio/)
            *   [Puppeteer](/v0.1/docs/integrations/document_loaders/web_loaders/web_puppeteer/)
            *   [Playwright](/v0.1/docs/integrations/document_loaders/web_loaders/web_playwright/)
            *   [Apify Dataset](/v0.1/docs/integrations/document_loaders/web_loaders/apify_dataset/)
            *   [AssemblyAI Audio Transcript](/v0.1/docs/integrations/document_loaders/web_loaders/assemblyai_audio_transcription/)
            *   [Azure Blob Storage Container](/v0.1/docs/integrations/document_loaders/web_loaders/azure_blob_storage_container/)
            *   [Azure Blob Storage File](/v0.1/docs/integrations/document_loaders/web_loaders/azure_blob_storage_file/)
            *   [Browserbase Loader](/v0.1/docs/integrations/document_loaders/web_loaders/browserbase/)
            *   [College Confidential](/v0.1/docs/integrations/document_loaders/web_loaders/college_confidential/)
            *   [Confluence](/v0.1/docs/integrations/document_loaders/web_loaders/confluence/)
            *   [Couchbase](/v0.1/docs/integrations/document_loaders/web_loaders/couchbase/)
            *   [Figma](/v0.1/docs/integrations/document_loaders/web_loaders/figma/)
            *   [Firecrawl](/v0.1/docs/integrations/document_loaders/web_loaders/firecrawl/)
            *   [GitBook](/v0.1/docs/integrations/document_loaders/web_loaders/gitbook/)
            *   [GitHub](/v0.1/docs/integrations/document_loaders/web_loaders/github/)
            *   [Hacker News](/v0.1/docs/integrations/document_loaders/web_loaders/hn/)
            *   [IMSDB](/v0.1/docs/integrations/document_loaders/web_loaders/imsdb/)
            *   [Notion API](/v0.1/docs/integrations/document_loaders/web_loaders/notionapi/)
            *   [PDF files](/v0.1/docs/integrations/document_loaders/web_loaders/pdf/)
            *   [Recursive URL Loader](/v0.1/docs/integrations/document_loaders/web_loaders/recursive_url_loader/)
            *   [S3 File](/v0.1/docs/integrations/document_loaders/web_loaders/s3/)
            *   [SearchApi Loader](/v0.1/docs/integrations/document_loaders/web_loaders/searchapi/)
            *   [SerpAPI Loader](/v0.1/docs/integrations/document_loaders/web_loaders/serpapi/)
            *   [Sitemap Loader](/v0.1/docs/integrations/document_loaders/web_loaders/sitemap/)
            *   [Sonix Audio](/v0.1/docs/integrations/document_loaders/web_loaders/sonix_audio_transcription/)
            *   [Blockchain Data](/v0.1/docs/integrations/document_loaders/web_loaders/sort_xyz_blockchain/)
            *   [YouTube transcripts](/v0.1/docs/integrations/document_loaders/web_loaders/youtube/)
    *   [Document transformers](/v0.1/docs/integrations/document_transformers/)
        
    *   [Document compressors](/v0.1/docs/integrations/document_compressors/)
        
    *   [Text embedding models](/v0.1/docs/integrations/text_embedding/)
        
    *   [Vector stores](/v0.1/docs/integrations/vectorstores/)
        
    *   [Retrievers](/v0.1/docs/integrations/retrievers/)
        
    *   [Tools](/v0.1/docs/integrations/tools/)
        
    *   [Agents and toolkits](/v0.1/docs/integrations/toolkits/)
        
    *   [Chat Memory](/v0.1/docs/integrations/chat_memory/)
        
    *   [Stores](/v0.1/docs/integrations/stores/)
        

*   [](/v0.1/)
*   [Components](/v0.1/docs/integrations/components/)
*   [Document loaders](/v0.1/docs/integrations/document_loaders/)
*   [Web Loaders](/v0.1/docs/integrations/document_loaders/web_loaders/)
*   Apify Dataset

Apify Dataset
=============

This guide shows how to use [Apify](https://apify.com) with LangChain to load documents from an Apify Dataset.

Overview[​](#overview "Direct link to Overview")
------------------------------------------------

[Apify](https://apify.com) is a cloud platform for web scraping and data extraction, which provides an [ecosystem](https://apify.com/store) of more than a thousand ready-made apps called _Actors_ for various web scraping, crawling, and data extraction use cases.

This guide shows how to load documents from an [Apify Dataset](https://docs.apify.com/platform/storage/dataset) — a scalable append-only storage built for storing structured web scraping results, such as a list of products or Google SERPs, and then export them to various formats like JSON, CSV, or Excel.

Datasets are typically used to save results of Actors. For example, [Website Content Crawler](https://apify.com/apify/website-content-crawler) Actor deeply crawls websites such as documentation, knowledge bases, help centers, or blogs, and then stores the text content of webpages into a dataset, from which you can feed the documents into a vector index and answer questions from it.

Setup[​](#setup "Direct link to Setup")
---------------------------------------

You'll first need to install the official Apify client:

*   npm
*   Yarn
*   pnpm

    npm install apify-client

    yarn add apify-client

    pnpm add apify-client

tip

See [this section for general instructions on installing integration packages](/v0.1/docs/get_started/installation/#installing-integration-packages).

*   npm
*   Yarn
*   pnpm

    npm install @langchain/openai @langchain/community

    yarn add @langchain/openai @langchain/community

    pnpm add @langchain/openai @langchain/community

You'll also need to sign up and retrieve your [Apify API token](https://console.apify.com/account/integrations).

Usage[​](#usage "Direct link to Usage")
---------------------------------------

### From a New Dataset[​](#from-a-new-dataset "Direct link to From a New Dataset")

If you don't already have an existing dataset on the Apify platform, you'll need to initialize the document loader by calling an Actor and waiting for the results.

**Note:** Calling an Actor can take a significant amount of time, on the order of hours, or even days for large sites!

Here's an example:

    import { ApifyDatasetLoader } from "langchain/document_loaders/web/apify_dataset";import { HNSWLib } from "@langchain/community/vectorstores/hnswlib";import { OpenAIEmbeddings, ChatOpenAI } from "@langchain/openai";import { Document } from "@langchain/core/documents";import { ChatPromptTemplate } from "@langchain/core/prompts";import { createStuffDocumentsChain } from "langchain/chains/combine_documents";import { createRetrievalChain } from "langchain/chains/retrieval";/* * datasetMappingFunction is a function that maps your Apify dataset format to LangChain documents. * In the below example, the Apify dataset format looks like this: * { *   "url": "https://apify.com", *   "text": "Apify is the best web scraping and automation platform." * } */const loader = await ApifyDatasetLoader.fromActorCall(  "apify/website-content-crawler",  {    startUrls: [{ url: "https://js.langchain.com/docs/" }],  },  {    datasetMappingFunction: (item) =>      new Document({        pageContent: (item.text || "") as string,        metadata: { source: item.url },      }),    clientOptions: {      token: "your-apify-token", // Or set as process.env.APIFY_API_TOKEN    },  });const docs = await loader.load();const vectorStore = await HNSWLib.fromDocuments(docs, new OpenAIEmbeddings());const model = new ChatOpenAI({  temperature: 0,});const questionAnsweringPrompt = ChatPromptTemplate.fromMessages([  [    "system",    "Answer the user's questions based on the below context:\n\n{context}",  ],  ["human", "{input}"],]);const combineDocsChain = await createStuffDocumentsChain({  llm: model,  prompt: questionAnsweringPrompt,});const chain = await createRetrievalChain({  retriever: vectorStore.asRetriever(),  combineDocsChain,});const res = await chain.invoke({ input: "What is LangChain?" });console.log(res.answer);console.log(res.context.map((doc) => doc.metadata.source));/*  LangChain is a framework for developing applications powered by language models.  [    'https://js.langchain.com/docs/',    'https://js.langchain.com/docs/modules/chains/',    'https://js.langchain.com/docs/modules/chains/llmchain/',    'https://js.langchain.com/docs/category/functions-4'  ]*/

#### API Reference:

*   [ApifyDatasetLoader](https://api.js.langchain.com/classes/langchain_document_loaders_web_apify_dataset.ApifyDatasetLoader.html) from `langchain/document_loaders/web/apify_dataset`
*   [HNSWLib](https://api.js.langchain.com/classes/langchain_community_vectorstores_hnswlib.HNSWLib.html) from `@langchain/community/vectorstores/hnswlib`
*   [OpenAIEmbeddings](https://api.js.langchain.com/classes/langchain_openai.OpenAIEmbeddings.html) from `@langchain/openai`
*   [ChatOpenAI](https://api.js.langchain.com/classes/langchain_openai.ChatOpenAI.html) from `@langchain/openai`
*   [Document](https://api.js.langchain.com/classes/langchain_core_documents.Document.html) from `@langchain/core/documents`
*   [ChatPromptTemplate](https://api.js.langchain.com/classes/langchain_core_prompts.ChatPromptTemplate.html) from `@langchain/core/prompts`
*   [createStuffDocumentsChain](https://api.js.langchain.com/functions/langchain_chains_combine_documents.createStuffDocumentsChain.html) from `langchain/chains/combine_documents`
*   [createRetrievalChain](https://api.js.langchain.com/functions/langchain_chains_retrieval.createRetrievalChain.html) from `langchain/chains/retrieval`

From an Existing Dataset[​](#from-an-existing-dataset "Direct link to From an Existing Dataset")
------------------------------------------------------------------------------------------------

If you already have an existing dataset on the Apify platform, you can initialize the document loader with the constructor directly:

    import { ApifyDatasetLoader } from "langchain/document_loaders/web/apify_dataset";import { HNSWLib } from "@langchain/community/vectorstores/hnswlib";import { OpenAIEmbeddings, ChatOpenAI } from "@langchain/openai";import { Document } from "@langchain/core/documents";import { ChatPromptTemplate } from "@langchain/core/prompts";import { createRetrievalChain } from "langchain/chains/retrieval";import { createStuffDocumentsChain } from "langchain/chains/combine_documents";/* * datasetMappingFunction is a function that maps your Apify dataset format to LangChain documents. * In the below example, the Apify dataset format looks like this: * { *   "url": "https://apify.com", *   "text": "Apify is the best web scraping and automation platform." * } */const loader = new ApifyDatasetLoader("your-dataset-id", {  datasetMappingFunction: (item) =>    new Document({      pageContent: (item.text || "") as string,      metadata: { source: item.url },    }),  clientOptions: {    token: "your-apify-token", // Or set as process.env.APIFY_API_TOKEN  },});const docs = await loader.load();const vectorStore = await HNSWLib.fromDocuments(docs, new OpenAIEmbeddings());const model = new ChatOpenAI({  temperature: 0,});const questionAnsweringPrompt = ChatPromptTemplate.fromMessages([  [    "system",    "Answer the user's questions based on the below context:\n\n{context}",  ],  ["human", "{input}"],]);const combineDocsChain = await createStuffDocumentsChain({  llm: model,  prompt: questionAnsweringPrompt,});const chain = await createRetrievalChain({  retriever: vectorStore.asRetriever(),  combineDocsChain,});const res = await chain.invoke({ input: "What is LangChain?" });console.log(res.answer);console.log(res.context.map((doc) => doc.metadata.source));/*  LangChain is a framework for developing applications powered by language models.  [    'https://js.langchain.com/docs/',    'https://js.langchain.com/docs/modules/chains/',    'https://js.langchain.com/docs/modules/chains/llmchain/',    'https://js.langchain.com/docs/category/functions-4'  ]*/

#### API Reference:

*   [ApifyDatasetLoader](https://api.js.langchain.com/classes/langchain_document_loaders_web_apify_dataset.ApifyDatasetLoader.html) from `langchain/document_loaders/web/apify_dataset`
*   [HNSWLib](https://api.js.langchain.com/classes/langchain_community_vectorstores_hnswlib.HNSWLib.html) from `@langchain/community/vectorstores/hnswlib`
*   [OpenAIEmbeddings](https://api.js.langchain.com/classes/langchain_openai.OpenAIEmbeddings.html) from `@langchain/openai`
*   [ChatOpenAI](https://api.js.langchain.com/classes/langchain_openai.ChatOpenAI.html) from `@langchain/openai`
*   [Document](https://api.js.langchain.com/classes/langchain_core_documents.Document.html) from `@langchain/core/documents`
*   [ChatPromptTemplate](https://api.js.langchain.com/classes/langchain_core_prompts.ChatPromptTemplate.html) from `@langchain/core/prompts`
*   [createRetrievalChain](https://api.js.langchain.com/functions/langchain_chains_retrieval.createRetrievalChain.html) from `langchain/chains/retrieval`
*   [createStuffDocumentsChain](https://api.js.langchain.com/functions/langchain_chains_combine_documents.createStuffDocumentsChain.html) from `langchain/chains/combine_documents`

* * *

#### Help us out by providing feedback on this documentation page:

[

Previous

Playwright

](/v0.1/docs/integrations/document_loaders/web_loaders/web_playwright/)[

Next

AssemblyAI Audio Transcript

](/v0.1/docs/integrations/document_loaders/web_loaders/assemblyai_audio_transcription/)

Community

*   [Discord](https://discord.gg/cU2adEyC7w)
*   [Twitter](https://twitter.com/LangChainAI)

GitHub

*   [Python](https://github.com/langchain-ai/langchain)
*   [JS/TS](https://github.com/langchain-ai/langchainjs)

More

*   [Homepage](https://langchain.com)
*   [Blog](https://blog.langchain.dev)

Copyright © 2024 LangChain, Inc.