!function(){function t(t){document.documentElement.setAttribute("data-theme",t)}var e=function(){var t=null;try{t=new URLSearchParams(window.location.search).get("docusaurus-theme")}catch(t){}return t}()||function(){var t=null;try{t=localStorage.getItem("theme")}catch(t){}return t}();t(null!==e?e:"light")}(),document.documentElement.setAttribute("data-announcement-bar-initially-dismissed",function(){try{return"true"===localStorage.getItem("docusaurus.announcement.dismiss")}catch(t){}return!1}())

[Skip to main content](#__docusaurus_skipToContent_fallback)

You are viewing the **preview** v0.2 docs. View the **stable** v0.1 docs [here](/v0.1/docs/get_started/introduction/). Leave feedback on the v0.2 docs [here](https://github.com/langchain-ai/langchainjs/discussions/5386).

[

![🦜️🔗 Langchain](/v0.2/img/brand/wordmark.png)![🦜️🔗 Langchain](/v0.2/img/brand/wordmark-dark.png)

](/v0.2/)[Integrations](/v0.2/docs/integrations/platforms/)[API Reference](https://v02.api.js.langchain.com)

[More](#)

*   [People](/v0.2/docs/people/)
*   [Community](/v0.2/docs/community)
*   [Tutorials](/v0.2/docs/additional_resources/tutorials)
*   [Contributing](/v0.2/docs/contributing)

[v0.2](#)

*   [v0.2](/v0.2/docs/introduction)
*   [v0.1](https://js.langchain.com/v0.1/docs/get_started/introduction)

[🦜🔗](#)

*   [LangSmith](https://smith.langchain.com)
*   [LangSmith Docs](https://docs.smith.langchain.com)
*   [LangChain Hub](https://smith.langchain.com/hub)
*   [LangServe](https://github.com/langchain-ai/langserve)
*   [Python Docs](https://python.langchain.com/)

[Chat](https://chatjs.langchain.com)[](https://github.com/langchain-ai/langchainjs)

Search

*   [Providers](/v0.2/docs/integrations/platforms/)
    
    *   [Providers](/v0.2/docs/integrations/platforms/)
    *   [Anthropic](/v0.2/docs/integrations/platforms/anthropic)
    *   [AWS](/v0.2/docs/integrations/platforms/aws)
    *   [Google](/v0.2/docs/integrations/platforms/google)
    *   [Microsoft](/v0.2/docs/integrations/platforms/microsoft)
    *   [OpenAI](/v0.2/docs/integrations/platforms/openai)
*   [Components](/v0.2/docs/integrations/components)
    
    *   [Chat models](/v0.2/docs/integrations/chat/)
        
    *   [LLMs](/v0.2/docs/integrations/llms/)
        
    *   [Embedding models](/v0.2/docs/integrations/text_embedding)
        
    *   [Document loaders](/v0.2/docs/integrations/document_loaders)
        
        *   [File Loaders](/v0.2/docs/integrations/document_loaders/file_loaders/)
            
        *   [Web Loaders](/v0.2/docs/integrations/document_loaders/web_loaders/)
            
            *   [Cheerio](/v0.2/docs/integrations/document_loaders/web_loaders/web_cheerio)
            *   [Puppeteer](/v0.2/docs/integrations/document_loaders/web_loaders/web_puppeteer)
            *   [Playwright](/v0.2/docs/integrations/document_loaders/web_loaders/web_playwright)
            *   [Apify Dataset](/v0.2/docs/integrations/document_loaders/web_loaders/apify_dataset)
            *   [AssemblyAI Audio Transcript](/v0.2/docs/integrations/document_loaders/web_loaders/assemblyai_audio_transcription)
            *   [Azure Blob Storage Container](/v0.2/docs/integrations/document_loaders/web_loaders/azure_blob_storage_container)
            *   [Azure Blob Storage File](/v0.2/docs/integrations/document_loaders/web_loaders/azure_blob_storage_file)
            *   [Browserbase Loader](/v0.2/docs/integrations/document_loaders/web_loaders/browserbase)
            *   [College Confidential](/v0.2/docs/integrations/document_loaders/web_loaders/college_confidential)
            *   [Confluence](/v0.2/docs/integrations/document_loaders/web_loaders/confluence)
            *   [Couchbase](/v0.2/docs/integrations/document_loaders/web_loaders/couchbase)
            *   [Figma](/v0.2/docs/integrations/document_loaders/web_loaders/figma)
            *   [Firecrawl](/v0.2/docs/integrations/document_loaders/web_loaders/firecrawl)
            *   [GitBook](/v0.2/docs/integrations/document_loaders/web_loaders/gitbook)
            *   [GitHub](/v0.2/docs/integrations/document_loaders/web_loaders/github)
            *   [Hacker News](/v0.2/docs/integrations/document_loaders/web_loaders/hn)
            *   [IMSDB](/v0.2/docs/integrations/document_loaders/web_loaders/imsdb)
            *   [Notion API](/v0.2/docs/integrations/document_loaders/web_loaders/notionapi)
            *   [PDF files](/v0.2/docs/integrations/document_loaders/web_loaders/pdf)
            *   [Recursive URL Loader](/v0.2/docs/integrations/document_loaders/web_loaders/recursive_url_loader)
            *   [S3 File](/v0.2/docs/integrations/document_loaders/web_loaders/s3)
            *   [SearchApi Loader](/v0.2/docs/integrations/document_loaders/web_loaders/searchapi)
            *   [SerpAPI Loader](/v0.2/docs/integrations/document_loaders/web_loaders/serpapi)
            *   [Sitemap Loader](/v0.2/docs/integrations/document_loaders/web_loaders/sitemap)
            *   [Sonix Audio](/v0.2/docs/integrations/document_loaders/web_loaders/sonix_audio_transcription)
            *   [Blockchain Data](/v0.2/docs/integrations/document_loaders/web_loaders/sort_xyz_blockchain)
            *   [YouTube transcripts](/v0.2/docs/integrations/document_loaders/web_loaders/youtube)
    *   [Document transformers](/v0.2/docs/integrations/document_transformers)
        
    *   [Vector stores](/v0.2/docs/integrations/vectorstores)
        
    *   [Retrievers](/v0.2/docs/integrations/retrievers)
        
    *   [Tools](/v0.2/docs/integrations/tools)
        
    *   [Toolkits](/v0.2/docs/integrations/toolkits)
        
    *   [Stores](/v0.2/docs/integrations/stores/)
        

*   [](/v0.2/)
*   [Components](/v0.2/docs/integrations/components)
*   [Document loaders](/v0.2/docs/integrations/document_loaders)
*   [Web Loaders](/v0.2/docs/integrations/document_loaders/web_loaders/)
*   Recursive URL Loader

Recursive URL Loader
====================

When loading content from a website, we may want to process load all URLs on a page.

For example, let's look at the [LangChain.js introduction](/v0.2/docs/introduction) docs.

This has many interesting child pages that we may want to load, split, and later retrieve in bulk.

The challenge is traversing the tree of child pages and assembling a list!

We do this using the RecursiveUrlLoader.

This also gives us the flexibility to exclude some children, customize the extractor, and more.

Setup[​](#setup "Direct link to Setup")
---------------------------------------

To get started, you'll need to install the [`jsdom`](https://www.npmjs.com/package/jsdom) package:

*   npm
*   Yarn
*   pnpm

    npm i jsdom

    yarn add jsdom

    pnpm add jsdom

We also suggest adding a package like [`html-to-text`](https://www.npmjs.com/package/html-to-text) or [`@mozilla/readability`](https://www.npmjs.com/package/@mozilla/readability) for extracting the raw text from the page.

*   npm
*   Yarn
*   pnpm

    npm i html-to-text

    yarn add html-to-text

    pnpm add html-to-text

Usage[​](#usage "Direct link to Usage")
---------------------------------------

    import { compile } from "html-to-text";import { RecursiveUrlLoader } from "langchain/document_loaders/web/recursive_url";const url = "/docs/introduction";const compiledConvert = compile({ wordwrap: 130 }); // returns (text: string) => string;const loader = new RecursiveUrlLoader(url, {  extractor: compiledConvert,  maxDepth: 1,  excludeDirs: ["/docs/api/"],});const docs = await loader.load();

Options[​](#options "Direct link to Options")
---------------------------------------------

    interface Options {  excludeDirs?: string[]; // webpage directories to exclude.  extractor?: (text: string) => string; // a function to extract the text of the document from the webpage, by default it returns the page as it is. It is recommended to use tools like html-to-text to extract the text. By default, it just returns the page as it is.  maxDepth?: number; // the maximum depth to crawl. By default, it is set to 2. If you need to crawl the whole website, set it to a number that is large enough would simply do the job.  timeout?: number; // the timeout for each request, in the unit of seconds. By default, it is set to 10000 (10 seconds).  preventOutside?: boolean; // whether to prevent crawling outside the root url. By default, it is set to true.  callerOptions?: AsyncCallerConstructorParams; // the options to call the AsyncCaller for example setting max concurrency (default is 64)}

However, since it's hard to perform a perfect filter, you may still see some irrelevant results in the results. You can perform a filter on the returned documents by yourself, if it's needed. Most of the time, the returned results are good enough.

* * *

#### Was this page helpful?

  

#### You can leave detailed feedback [on GitHub](https://github.com/langchain-ai/langchainjs/issues/new?assignees=&labels=03+-+Documentation&projects=&template=documentation.yml&title=DOC%3A+%3CPlease+write+a+comprehensive+title+after+the+%27DOC%3A+%27+prefix%3E).

[

Previous

PDF files

](/v0.2/docs/integrations/document_loaders/web_loaders/pdf)[

Next

S3 File

](/v0.2/docs/integrations/document_loaders/web_loaders/s3)

Community

*   [Discord](https://discord.gg/cU2adEyC7w)
*   [Twitter](https://twitter.com/LangChainAI)

GitHub

*   [Python](https://github.com/langchain-ai/langchain)
*   [JS/TS](https://github.com/langchain-ai/langchainjs)

More

*   [Homepage](https://langchain.com)
*   [Blog](https://blog.langchain.dev)

Copyright © 2024 LangChain, Inc.