!function(){function t(t){document.documentElement.setAttribute("data-theme",t)}var e=function(){var t=null;try{t=new URLSearchParams(window.location.search).get("docusaurus-theme")}catch(t){}return t}()||function(){var t=null;try{t=localStorage.getItem("theme")}catch(t){}return t}();t(null!==e?e:"light")}(),document.documentElement.setAttribute("data-announcement-bar-initially-dismissed",function(){try{return"true"===localStorage.getItem("docusaurus.announcement.dismiss")}catch(t){}return!1}())

[Skip to main content](#__docusaurus_skipToContent_fallback)

LangChain v0.2 is coming soon! Preview the new docs [here](/v0.2/docs/introduction/).

[

![🦜️🔗 Langchain](/v0.1/img/brand/wordmark.png)![🦜️🔗 Langchain](/v0.1/img/brand/wordmark-dark.png)

](/v0.1/)[Docs](/v0.1/docs/get_started/introduction/)[Use cases](/v0.1/docs/use_cases/)[Integrations](/v0.1/docs/integrations/platforms/)[API Reference](https://api.js.langchain.com)

[More](#)

*   [People](/v0.1/docs/people/)
*   [Community](/v0.1/docs/community/)
*   [Tutorials](/v0.1/docs/additional_resources/tutorials/)
*   [Contributing](/v0.1/docs/contributing/)

[v0.1](#)

*   [v0.2](https://js.langchain.com/v0.2/docs/introduction)
*   [v0.1](/v0.1/docs/get_started/introduction/)

[🦜🔗](#)

*   [LangSmith](https://smith.langchain.com)
*   [LangSmith Docs](https://docs.smith.langchain.com)
*   [LangChain Hub](https://smith.langchain.com/hub)
*   [LangServe](https://github.com/langchain-ai/langserve)
*   [Python Docs](https://python.langchain.com/)

[Chat](https://chatjs.langchain.com)[](https://github.com/langchain-ai/langchainjs)

Search

*   [Providers](/v0.1/docs/integrations/platforms/)
    
    *   [Providers](/v0.1/docs/integrations/platforms/)
    *   [Anthropic](/v0.1/docs/integrations/platforms/anthropic/)
    *   [AWS](/v0.1/docs/integrations/platforms/aws/)
    *   [Google](/v0.1/docs/integrations/platforms/google/)
    *   [Microsoft](/v0.1/docs/integrations/platforms/microsoft/)
    *   [OpenAI](/v0.1/docs/integrations/platforms/openai/)
*   [Components](/v0.1/docs/integrations/components/)
    
    *   [LLMs](/v0.1/docs/integrations/llms/)
        
    *   [Chat models](/v0.1/docs/integrations/chat/)
        
    *   [Document loaders](/v0.1/docs/integrations/document_loaders/)
        
        *   [File Loaders](/v0.1/docs/integrations/document_loaders/file_loaders/)
            
        *   [Web Loaders](/v0.1/docs/integrations/document_loaders/web_loaders/)
            
            *   [Cheerio](/v0.1/docs/integrations/document_loaders/web_loaders/web_cheerio/)
            *   [Puppeteer](/v0.1/docs/integrations/document_loaders/web_loaders/web_puppeteer/)
            *   [Playwright](/v0.1/docs/integrations/document_loaders/web_loaders/web_playwright/)
            *   [Apify Dataset](/v0.1/docs/integrations/document_loaders/web_loaders/apify_dataset/)
            *   [AssemblyAI Audio Transcript](/v0.1/docs/integrations/document_loaders/web_loaders/assemblyai_audio_transcription/)
            *   [Azure Blob Storage Container](/v0.1/docs/integrations/document_loaders/web_loaders/azure_blob_storage_container/)
            *   [Azure Blob Storage File](/v0.1/docs/integrations/document_loaders/web_loaders/azure_blob_storage_file/)
            *   [Browserbase Loader](/v0.1/docs/integrations/document_loaders/web_loaders/browserbase/)
            *   [College Confidential](/v0.1/docs/integrations/document_loaders/web_loaders/college_confidential/)
            *   [Confluence](/v0.1/docs/integrations/document_loaders/web_loaders/confluence/)
            *   [Couchbase](/v0.1/docs/integrations/document_loaders/web_loaders/couchbase/)
            *   [Figma](/v0.1/docs/integrations/document_loaders/web_loaders/figma/)
            *   [Firecrawl](/v0.1/docs/integrations/document_loaders/web_loaders/firecrawl/)
            *   [GitBook](/v0.1/docs/integrations/document_loaders/web_loaders/gitbook/)
            *   [GitHub](/v0.1/docs/integrations/document_loaders/web_loaders/github/)
            *   [Hacker News](/v0.1/docs/integrations/document_loaders/web_loaders/hn/)
            *   [IMSDB](/v0.1/docs/integrations/document_loaders/web_loaders/imsdb/)
            *   [Notion API](/v0.1/docs/integrations/document_loaders/web_loaders/notionapi/)
            *   [PDF files](/v0.1/docs/integrations/document_loaders/web_loaders/pdf/)
            *   [Recursive URL Loader](/v0.1/docs/integrations/document_loaders/web_loaders/recursive_url_loader/)
            *   [S3 File](/v0.1/docs/integrations/document_loaders/web_loaders/s3/)
            *   [SearchApi Loader](/v0.1/docs/integrations/document_loaders/web_loaders/searchapi/)
            *   [SerpAPI Loader](/v0.1/docs/integrations/document_loaders/web_loaders/serpapi/)
            *   [Sitemap Loader](/v0.1/docs/integrations/document_loaders/web_loaders/sitemap/)
            *   [Sonix Audio](/v0.1/docs/integrations/document_loaders/web_loaders/sonix_audio_transcription/)
            *   [Blockchain Data](/v0.1/docs/integrations/document_loaders/web_loaders/sort_xyz_blockchain/)
            *   [YouTube transcripts](/v0.1/docs/integrations/document_loaders/web_loaders/youtube/)
    *   [Document transformers](/v0.1/docs/integrations/document_transformers/)
        
    *   [Document compressors](/v0.1/docs/integrations/document_compressors/)
        
    *   [Text embedding models](/v0.1/docs/integrations/text_embedding/)
        
    *   [Vector stores](/v0.1/docs/integrations/vectorstores/)
        
    *   [Retrievers](/v0.1/docs/integrations/retrievers/)
        
    *   [Tools](/v0.1/docs/integrations/tools/)
        
    *   [Agents and toolkits](/v0.1/docs/integrations/toolkits/)
        
    *   [Chat Memory](/v0.1/docs/integrations/chat_memory/)
        
    *   [Stores](/v0.1/docs/integrations/stores/)
        

*   [](/v0.1/)
*   [Components](/v0.1/docs/integrations/components/)
*   [Document loaders](/v0.1/docs/integrations/document_loaders/)
*   [Web Loaders](/v0.1/docs/integrations/document_loaders/web_loaders/)
*   GitHub

GitHub
======

This example goes over how to load data from a GitHub repository. You can set the `GITHUB_ACCESS_TOKEN` environment variable to a GitHub access token to increase the rate limit and access private repositories.

Setup[​](#setup "Direct link to Setup")
---------------------------------------

The GitHub loader requires the [ignore npm package](https://www.npmjs.com/package/ignore) as a peer dependency. Install it like this:

*   npm
*   Yarn
*   pnpm

    npm install ignore

    yarn add ignore

    pnpm add ignore

Usage[​](#usage "Direct link to Usage")
---------------------------------------

    import { GithubRepoLoader } from "langchain/document_loaders/web/github";export const run = async () => {  const loader = new GithubRepoLoader(    "https://github.com/langchain-ai/langchainjs",    {      branch: "main",      recursive: false,      unknown: "warn",      maxConcurrency: 5, // Defaults to 2    }  );  const docs = await loader.load();  console.log({ docs });};

#### API Reference:

*   [GithubRepoLoader](https://api.js.langchain.com/classes/langchain_document_loaders_web_github.GithubRepoLoader.html) from `langchain/document_loaders/web/github`

The loader will ignore binary files like images.

### Using .gitignore Syntax[​](#using-gitignore-syntax "Direct link to Using .gitignore Syntax")

To ignore specific files, you can pass in an `ignorePaths` array into the constructor:

    import { GithubRepoLoader } from "langchain/document_loaders/web/github";export const run = async () => {  const loader = new GithubRepoLoader(    "https://github.com/langchain-ai/langchainjs",    { branch: "main", recursive: false, unknown: "warn", ignorePaths: ["*.md"] }  );  const docs = await loader.load();  console.log({ docs });  // Will not include any .md files};

#### API Reference:

*   [GithubRepoLoader](https://api.js.langchain.com/classes/langchain_document_loaders_web_github.GithubRepoLoader.html) from `langchain/document_loaders/web/github`

### Using a Different GitHub Instance[​](#using-a-different-github-instance "Direct link to Using a Different GitHub Instance")

You may want to target a different GitHub instance than `github.com`, e.g. if you have a GitHub Enterprise instance for your company. For this you need two additional parameters:

*   `baseUrl` - the base URL of your GitHub instance, so the githubUrl matches `<baseUrl>/<owner>/<repo>/...`
*   `apiUrl` - the URL of the API endpoint of your GitHub instance

    import { GithubRepoLoader } from "langchain/document_loaders/web/github";export const run = async () => {  const loader = new GithubRepoLoader(    "https://github.your.company/org/repo-name",    {      baseUrl: "https://github.your.company",      apiUrl: "https://github.your.company/api/v3",      accessToken: "ghp_A1B2C3D4E5F6a7b8c9d0",      branch: "main",      recursive: true,      unknown: "warn",    }  );  const docs = await loader.load();  console.log({ docs });};

#### API Reference:

*   [GithubRepoLoader](https://api.js.langchain.com/classes/langchain_document_loaders_web_github.GithubRepoLoader.html) from `langchain/document_loaders/web/github`

### Dealing with Submodules[​](#dealing-with-submodules "Direct link to Dealing with Submodules")

In case your repository has submodules, you have to decide if the loader should follow them or not. You can control this with the boolean `processSubmodules` parameter. By default, submodules are not processed. Note that processing submodules works only in conjunction with setting the `recursive` parameter to true.

    import { GithubRepoLoader } from "langchain/document_loaders/web/github";export const run = async () => {  const loader = new GithubRepoLoader(    "https://github.com/langchain-ai/langchainjs",    {      branch: "main",      recursive: true,      processSubmodules: true,      unknown: "warn",    }  );  const docs = await loader.load();  console.log({ docs });};

#### API Reference:

*   [GithubRepoLoader](https://api.js.langchain.com/classes/langchain_document_loaders_web_github.GithubRepoLoader.html) from `langchain/document_loaders/web/github`

Note, that the loader will not follow submodules which are located on another GitHub instance than the one of the current repository.

### Stream large repository[​](#stream-large-repository "Direct link to Stream large repository")

For situations where processing large repositories in a memory-efficient manner is required. You can use the `loadAsStream` method to asynchronously streams documents from the entire GitHub repository.

    import { GithubRepoLoader } from "langchain/document_loaders/web/github";export const run = async () => {  const loader = new GithubRepoLoader(    "https://github.com/langchain-ai/langchainjs",    {      branch: "main",      recursive: false,      unknown: "warn",      maxConcurrency: 3, // Defaults to 2    }  );  const docs = [];  for await (const doc of loader.loadAsStream()) {    docs.push(doc);  }  console.log({ docs });};

#### API Reference:

*   [GithubRepoLoader](https://api.js.langchain.com/classes/langchain_document_loaders_web_github.GithubRepoLoader.html) from `langchain/document_loaders/web/github`

* * *

#### Help us out by providing feedback on this documentation page:

[

Previous

GitBook

](/v0.1/docs/integrations/document_loaders/web_loaders/gitbook/)[

Next

Hacker News

](/v0.1/docs/integrations/document_loaders/web_loaders/hn/)

Community

*   [Discord](https://discord.gg/cU2adEyC7w)
*   [Twitter](https://twitter.com/LangChainAI)

GitHub

*   [Python](https://github.com/langchain-ai/langchain)
*   [JS/TS](https://github.com/langchain-ai/langchainjs)

More

*   [Homepage](https://langchain.com)
*   [Blog](https://blog.langchain.dev)

Copyright © 2024 LangChain, Inc.