seansullivan/Langchain-Docs huggingface dataset:
## GitHub

This notebooks shows how you can load issues and pull requests (PRs) for a given repository on [GitHub](https://github.com/). Also shows how you can load github files for a given repository on [GitHub](https://github.com/). We will use the LangChain Python repository as an example.

## Setup access token[​](#setup-access-token "Direct link to Setup access token")

To access the GitHub API, you need a personal access token - you can set up yours here: [https://github.com/settings/tokens?type=beta](https://github.com/settings/tokens?type=beta). You can either set this token as the environment variable `GITHUB_PERSONAL_ACCESS_TOKEN` and it will be automatically pulled in, or you can pass it in directly at initialization as the `access_token` named parameter.

```
# If you haven't set your access token as an environment variable, pass it in here.from getpass import getpassACCESS_TOKEN = getpass()
```

## Load Issues and PRs[​](#load-issues-and-prs "Direct link to Load Issues and PRs")

```
from langchain_community.document_loaders import GitHubIssuesLoader
```

```
loader = GitHubIssuesLoader(    repo="langchain-ai/langchain",    access_token=ACCESS_TOKEN,  # delete/comment out this argument if you've set the access token as an env var.    creator="UmerHA",)
```

Let’s load all issues and PRs created by “UmerHA”.

Here’s a list of all filters you can use: - include\_prs - milestone - state - assignee - creator - mentioned - labels - sort - direction - since

For more info, see [https://docs.github.com/en/rest/issues/issues?apiVersion=2022-11-28#list-repository-issues](https://docs.github.com/en/rest/issues/issues?apiVersion=2022-11-28#list-repository-issues).

```
print(docs[0].page_content)print(docs[0].metadata)
```

## Only load issues[​](#only-load-issues "Direct link to Only load issues")

By default, the GitHub API returns considers pull requests to also be issues. To only get ‘pure’ issues (i.e., no pull requests), use `include_prs=False`

```
loader = GitHubIssuesLoader(    repo="langchain-ai/langchain",    access_token=ACCESS_TOKEN,  # delete/comment out this argument if you've set the access token as an env var.    creator="UmerHA",    include_prs=False,)docs = loader.load()
```

```
print(docs[0].page_content)print(docs[0].metadata)
```

## Load Github File Content[​](#load-github-file-content "Direct link to Load Github File Content")

For below code, loads all markdown file in rpeo `langchain-ai/langchain`

```
from langchain.document_loaders import GithubFileLoader
```

```
loader = GithubFileLoader(    repo="langchain-ai/langchain",  # the repo name    access_token=ACCESS_TOKEN,    github_api_url="https://api.github.com",    file_filter=lambda file_path: file_path.endswith(        ".md"    ),  # load all markdowns files.)documents = loader.load()
```

example output of one of document:

```
documents.metadata:     {      "path": "README.md",      "sha": "82f1c4ea88ecf8d2dfsfx06a700e84be4",      "source": "https://github.com/langchain-ai/langchain/blob/master/README.md"    }documents.content:    mock content
```

* * *

#### Help us out by providing feedback on this documentation page:
## Glue Catalog

The [AWS Glue Data Catalog](https://docs.aws.amazon.com/en_en/glue/latest/dg/catalog-and-crawler.html) is a centralized metadata repository that allows you to manage, access, and share metadata about your data stored in AWS. It acts as a metadata store for your data assets, enabling various AWS services and your applications to query and connect to the data they need efficiently.

When you define data sources, transformations, and targets in AWS Glue, the metadata about these elements is stored in the Data Catalog. This includes information about data locations, schema definitions, runtime metrics, and more. It supports various data store types, such as Amazon S3, Amazon RDS, Amazon Redshift, and external databases compatible with JDBC. It is also directly integrated with Amazon Athena, Amazon Redshift Spectrum, and Amazon EMR, allowing these services to directly access and query the data.

The Langchain GlueCatalogLoader will get the schema of all tables inside the given Glue database in the same format as Pandas dtype.

## Setting up[​](#setting-up "Direct link to Setting up")

*   Follow [instructions to set up an AWS accoung](https://docs.aws.amazon.com/athena/latest/ug/setting-up.html).
*   Install the boto3 library: `pip install boto3`

## Example[​](#example "Direct link to Example")

```
from langchain_community.document_loaders.glue_catalog import GlueCatalogLoader
```

```
database_name = "my_database"profile_name = "my_profile"loader = GlueCatalogLoader(    database=database_name,    profile_name=profile_name,)schemas = loader.load()print(schemas)
```

## Example with table filtering[​](#example-with-table-filtering "Direct link to Example with table filtering")

Table filtering allows you to selectively retrieve schema information for a specific subset of tables within a Glue database. Instead of loading the schemas for all tables, you can use the `table_filter` argument to specify exactly which tables you’re interested in.

```
from langchain_community.document_loaders.glue_catalog import GlueCatalogLoader
```

```
database_name = "my_database"profile_name = "my_profile"table_filter = ["table1", "table2", "table3"]loader = GlueCatalogLoader(    database=database_name, profile_name=profile_name, table_filter=table_filter)schemas = loader.load()print(schemas)
```

* * *

#### Help us out by providing feedback on this documentation page:
## Google Cloud Storage File

> [Google Cloud Storage](https://en.wikipedia.org/wiki/Google_Cloud_Storage) is a managed service for storing unstructured data.

This covers how to load document objects from an `Google Cloud Storage (GCS) file object (blob)`.

```
%pip install --upgrade --quiet  google-cloud-storage
```

```
from langchain_community.document_loaders import GCSFileLoader
```

```
loader = GCSFileLoader(project_name="aist", bucket="testing-hwc", blob="fake.docx")
```

```
/Users/harrisonchase/workplace/langchain/.venv/lib/python3.10/site-packages/google/auth/_default.py:83: UserWarning: Your application has authenticated using end user credentials from Google Cloud SDK without a quota project. You might receive a "quota exceeded" or "API not enabled" error. We recommend you rerun `gcloud auth application-default login` and make sure a quota project is added. Or you can use service accounts instead. For more information about service accounts, see https://cloud.google.com/docs/authentication/  warnings.warn(_CLOUD_SDK_CREDENTIALS_WARNING)
```

```
[Document(page_content='Lorem ipsum dolor sit amet.', lookup_str='', metadata={'source': '/var/folders/y6/8_bzdg295ld6s1_97_12m4lr0000gn/T/tmp3srlf8n8/fake.docx'}, lookup_index=0)]
```

If you want to use an alternative loader, you can provide a custom function, for example:

```
from langchain_community.document_loaders import PyPDFLoaderdef load_pdf(file_path):    return PyPDFLoader(file_path)loader = GCSFileLoader(    project_name="aist", bucket="testing-hwc", blob="fake.pdf", loader_func=load_pdf)
```

* * *

#### Help us out by providing feedback on this documentation page:
The `GoogleSpeechToTextLoader` allows to transcribe audio files with the [Google Cloud Speech-to-Text API](https://cloud.google.com/speech-to-text) and loads the transcribed text into documents.

To use it, you should have the `google-cloud-speech` python package installed, and a Google Cloud project with the [Speech-to-Text API enabled](https://cloud.google.com/speech-to-text/v2/docs/transcribe-client-libraries#before_you_begin).

First, you need to install the `google-cloud-speech` python package.

Follow the [quickstart guide](https://cloud.google.com/speech-to-text/v2/docs/sync-recognize) in the Google Cloud documentation to create a project and enable the API.

The `GoogleSpeechToTextLoader` must include the `project_id` and `file_path` arguments. Audio files can be specified as a Google Cloud Storage URI (`gs://...`) or a local file path.

Only synchronous requests are supported by the loader, which has a [limit of 60 seconds or 10MB](https://cloud.google.com/speech-to-text/v2/docs/sync-recognize#:~:text=60%20seconds%20and/or%2010%20MB) per audio file.

```
from langchain_community.document_loaders import GoogleSpeechToTextLoaderproject_id = "<PROJECT_ID>"file_path = "gs://cloud-samples-data/speech/audio.flac"# or a local file path: file_path = "./audio.wav"loader = GoogleSpeechToTextLoader(project_id=project_id, file_path=file_path)docs = loader.load()
```

Note: Calling `loader.load()` blocks until the transcription is finished.

```
"How old is the Brooklyn Bridge?"
```

You can specify the `config` argument to use different speech recognition models and enable specific features.

If you don’t specify a `config`, the following options will be selected automatically:

```
from google.cloud.speech_v2 import (    AutoDetectDecodingConfig,    RecognitionConfig,    RecognitionFeatures,)from langchain_community.document_loaders import GoogleSpeechToTextLoaderproject_id = "<PROJECT_ID>"location = "global"recognizer_id = "<RECOGNIZER_ID>"file_path = "./audio.wav"config = RecognitionConfig(    auto_decoding_config=AutoDetectDecodingConfig(),    language_codes=["en-US"],    model="long",    features=RecognitionFeatures(        enable_automatic_punctuation=False,        profanity_filter=True,        enable_spoken_punctuation=True,        enable_spoken_emojis=True,    ),)loader = GoogleSpeechToTextLoader(    project_id=project_id,    location=location,    recognizer_id=recognizer_id,    file_path=file_path,    config=config,)
```
GROBID is a machine learning library for extracting, parsing, and re-structuring raw documents.

It is designed and expected to be used to parse academic papers, where it works particularly well. Note: if the articles supplied to Grobid are large documents (e.g. dissertations) exceeding a certain number of elements, they might not be processed.

This loader uses Grobid to parse PDFs into `Documents` that retain metadata associated with the section of text.

Once grobid is up-and-running you can interact as described below.

Now, we can use the data loader.

```
'Unlike Chinchilla, PaLM, or GPT-3, we only use publicly available data, making our work compatible with open-sourcing, while most existing models rely on data which is either not publicly available or undocumented (e.g."Books -2TB" or "Social media conversations").There exist some exceptions, notably OPT (Zhang et al., 2022), GPT-NeoX (Black et al., 2022), BLOOM (Scao et al., 2022) and GLM (Zeng et al., 2022), but none that are competitive with PaLM-62B or Chinchilla.'
```

```
{'text': 'Unlike Chinchilla, PaLM, or GPT-3, we only use publicly available data, making our work compatible with open-sourcing, while most existing models rely on data which is either not publicly available or undocumented (e.g."Books -2TB" or "Social media conversations").There exist some exceptions, notably OPT (Zhang et al., 2022), GPT-NeoX (Black et al., 2022), BLOOM (Scao et al., 2022) and GLM (Zeng et al., 2022), but none that are competitive with PaLM-62B or Chinchilla.', 'para': '2', 'bboxes': "[[{'page': '1', 'x': '317.05', 'y': '509.17', 'h': '207.73', 'w': '9.46'}, {'page': '1', 'x': '306.14', 'y': '522.72', 'h': '220.08', 'w': '9.46'}, {'page': '1', 'x': '306.14', 'y': '536.27', 'h': '218.27', 'w': '9.46'}, {'page': '1', 'x': '306.14', 'y': '549.82', 'h': '218.65', 'w': '9.46'}, {'page': '1', 'x': '306.14', 'y': '563.37', 'h': '136.98', 'w': '9.46'}], [{'page': '1', 'x': '446.49', 'y': '563.37', 'h': '78.11', 'w': '9.46'}, {'page': '1', 'x': '304.69', 'y': '576.92', 'h': '138.32', 'w': '9.46'}], [{'page': '1', 'x': '447.75', 'y': '576.92', 'h': '76.66', 'w': '9.46'}, {'page': '1', 'x': '306.14', 'y': '590.47', 'h': '219.63', 'w': '9.46'}, {'page': '1', 'x': '306.14', 'y': '604.02', 'h': '218.27', 'w': '9.46'}, {'page': '1', 'x': '306.14', 'y': '617.56', 'h': '218.27', 'w': '9.46'}, {'page': '1', 'x': '306.14', 'y': '631.11', 'h': '220.18', 'w': '9.46'}]]", 'pages': "('1', '1')", 'section_title': 'Introduction', 'section_number': '1', 'paper_title': 'LLaMA: Open and Efficient Foundation Language Models', 'file_path': '/Users/31treehaus/Desktop/Papers/2302.13971.pdf'}
```
## Gutenberg

> [Project Gutenberg](https://www.gutenberg.org/about/) is an online library of free eBooks.

This notebook covers how to load links to `Gutenberg` e-books into a document format that we can use downstream.

```
from langchain_community.document_loaders import GutenbergLoader
```

```
loader = GutenbergLoader("https://www.gutenberg.org/cache/epub/69972/pg69972.txt")
```

```
data[0].page_content[:300]
```

```
'The Project Gutenberg eBook of The changed brides, by Emma Dorothy\r\n\n\nEliza Nevitte Southworth\r\n\n\n\r\n\n\nThis eBook is for the use of anyone anywhere in the United States and\r\n\n\nmost other parts of the world at no cost and with almost no restrictions\r\n\n\nwhatsoever. You may copy it, give it away or re-u'
```

```
{'source': 'https://www.gutenberg.org/cache/epub/69972/pg69972.txt'}
```

* * *

#### Help us out by providing feedback on this documentation page:
This covers how to load images such as `JPG` or `PNG` into a document format that we can use downstream.

```
Document(page_content="LayoutParser: A Unified Toolkit for Deep\nLearning Based Document Image Analysis\n\n\n‘Zxjiang Shen' (F3}, Ruochen Zhang”, Melissa Dell*, Benjamin Charles Germain\nLeet, Jacob Carlson, and Weining LiF\n\n\nsugehen\n\nshangthrows, et\n\n“Abstract. Recent advanocs in document image analysis (DIA) have been\n‘pimarliy driven bythe application of neural networks dell roar\n{uteomer could be aly deployed in production and extended fo farther\n[nvetigtion. However, various factory ke lcely organize codebanee\nsnd sophisticated modal cnigurations compat the ey ree of\n‘erin! innovation by wide sence, Though there have been sng\n‘Hors to improve reuablty and simplify deep lees (DL) mode\n‘aon, sone of them ae optimized for challenge inthe demain of DIA,\nThis roprscte a major gap in the extng fol, sw DIA i eal to\nscademic research acon wie range of dpi in the social ssencee\n[rary for streamlining the sage of DL in DIA research and appicn\n‘tons The core LayoutFaraer brary comes with a sch of simple and\nIntative interfaee or applying and eutomiing DI. odel fr Inyo de\npltfom for sharing both protrined modes an fal document dist\n{ation pipeline We demonutate that LayootPareer shea fr both\nlightweight and lrgeseledgtieation pipelines in eal-word uae ces\nThe leary pblely smal at Btspe://layost-pareergsthab So\n\n\n\n‘Keywords: Document Image Analysis» Deep Learning Layout Analysis\n‘Character Renguition - Open Serres dary « Tol\n\n\nIntroduction\n\n\n‘Deep Learning(DL)-based approaches are the state-of-the-art for a wide range of\ndoctiment image analysis (DIA) tea including document image clasiffeation [I]\n", lookup_str='', metadata={'source': 'layout-parser-paper-fast.jpg'}, lookup_index=0)
```

Under the hood, Unstructured creates different “elements” for different chunks of text. By default we combine those together, but you can easily keep that separation by specifying `mode="elements"`.

```
Document(page_content='LayoutParser: A Unified Toolkit for Deep\nLearning Based Document Image Analysis\n', lookup_str='', metadata={'source': 'layout-parser-paper-fast.jpg', 'filename': 'layout-parser-paper-fast.jpg', 'page_number': 1, 'category': 'Title'}, lookup_index=0)
```
## Huawei OBS Directory

The following code demonstrates how to load objects from the Huawei OBS (Object Storage Service) as documents.

```
# Install the required package# pip install esdk-obs-python
```

```
from langchain_community.document_loaders import OBSDirectoryLoader
```

```
endpoint = "your-endpoint"
```

```
# Configure your access credentials\nconfig = {"ak": "your-access-key", "sk": "your-secret-key"}loader = OBSDirectoryLoader("your-bucket-name", endpoint=endpoint, config=config)
```

## Specify a Prefix for Loading[​](#specify-a-prefix-for-loading "Direct link to Specify a Prefix for Loading")

If you want to load objects with a specific prefix from the bucket, you can use the following code:

```
loader = OBSDirectoryLoader(    "your-bucket-name", endpoint=endpoint, config=config, prefix="test_prefix")
```

## Get Authentication Information from ECS[​](#get-authentication-information-from-ecs "Direct link to Get Authentication Information from ECS")

If your langchain is deployed on Huawei Cloud ECS and [Agency is set up](https://support.huaweicloud.com/intl/en-us/usermanual-ecs/ecs_03_0166.html#section7), the loader can directly get the security token from ECS without needing access key and secret key.

```
config = {"get_token_from_ecs": True}loader = OBSDirectoryLoader("your-bucket-name", endpoint=endpoint, config=config)
```

## Use a Public Bucket[​](#use-a-public-bucket "Direct link to Use a Public Bucket")

If your bucket’s bucket policy allows anonymous access (anonymous users have `listBucket` and `GetObject` permissions), you can directly load the objects without configuring the `config` parameter.

```
loader = OBSDirectoryLoader("your-bucket-name", endpoint=endpoint)
```
## Huawei OBS File

The following code demonstrates how to load an object from the Huawei OBS (Object Storage Service) as document.

```
# Install the required package# pip install esdk-obs-python
```

```
from langchain_community.document_loaders.obs_file import OBSFileLoader
```

```
endpoint = "your-endpoint"
```

```
from obs import ObsClientobs_client = ObsClient(    access_key_id="your-access-key",    secret_access_key="your-secret-key",    server=endpoint,)loader = OBSFileLoader("your-bucket-name", "your-object-key", client=obs_client)
```

## Each Loader with Separate Authentication Information[​](#each-loader-with-separate-authentication-information "Direct link to Each Loader with Separate Authentication Information")

If you don’t need to reuse OBS connections between different loaders, you can directly configure the `config`. The loader will use the config information to initialize its own OBS client.

```
# Configure your access credentials\nconfig = {"ak": "your-access-key", "sk": "your-secret-key"}loader = OBSFileLoader(    "your-bucket-name", "your-object-key", endpoint=endpoint, config=config)
```

## Get Authentication Information from ECS[​](#get-authentication-information-from-ecs "Direct link to Get Authentication Information from ECS")

If your langchain is deployed on Huawei Cloud ECS and [Agency is set up](https://support.huaweicloud.com/intl/en-us/usermanual-ecs/ecs_03_0166.html#section7), the loader can directly get the security token from ECS without needing access key and secret key.

```
config = {"get_token_from_ecs": True}loader = OBSFileLoader(    "your-bucket-name", "your-object-key", endpoint=endpoint, config=config)
```

## Access a Publicly Accessible Object[​](#access-a-publicly-accessible-object "Direct link to Access a Publicly Accessible Object")

If the object you want to access allows anonymous user access (anonymous users have `GetObject` permission), you can directly load the object without configuring the `config` parameter.

```
loader = OBSFileLoader("your-bucket-name", "your-object-key", endpoint=endpoint)
```
## Hacker News

> [Hacker News](https://en.wikipedia.org/wiki/Hacker_News) (sometimes abbreviated as `HN`) is a social news website focusing on computer science and entrepreneurship. It is run by the investment fund and startup incubator `Y Combinator`. In general, content that can be submitted is defined as “anything that gratifies one’s intellectual curiosity.”

This notebook covers how to pull page data and comments from [Hacker News](https://news.ycombinator.com/)

```
from langchain_community.document_loaders import HNLoader
```

```
loader = HNLoader("https://news.ycombinator.com/item?id=34817881")
```

```
data[0].page_content[:300]
```

```
"delta_p_delta_x 73 days ago  \n             | next [–] \n\nAstrophysical and cosmological simulations are often insightful. They're also very cross-disciplinary; besides the obvious astrophysics, there's networking and sysadmin, parallel computing and algorithm theory (so that the simulation programs a"
```

```
{'source': 'https://news.ycombinator.com/item?id=34817881', 'title': 'What Lights the Universe’s Standard Candles?'}
```

* * *

#### Help us out by providing feedback on this documentation page:
## Image captions

By default, the loader utilizes the pre-trained [Salesforce BLIP image captioning model](https://huggingface.co/Salesforce/blip-image-captioning-base).

This notebook shows how to use the `ImageCaptionLoader` to generate a query-able index of image captions

```
%pip install --upgrade --quiet  transformers
```

```
from langchain_community.document_loaders import ImageCaptionLoader
```

### Prepare a list of image urls from Wikimedia[​](#prepare-a-list-of-image-urls-from-wikimedia "Direct link to Prepare a list of image urls from Wikimedia")

```
list_image_urls = [    "https://upload.wikimedia.org/wikipedia/commons/thumb/5/5a/Hyla_japonica_sep01.jpg/260px-Hyla_japonica_sep01.jpg",    "https://upload.wikimedia.org/wikipedia/commons/thumb/7/71/Tibur%C3%B3n_azul_%28Prionace_glauca%29%2C_canal_Fayal-Pico%2C_islas_Azores%2C_Portugal%2C_2020-07-27%2C_DD_14.jpg/270px-Tibur%C3%B3n_azul_%28Prionace_glauca%29%2C_canal_Fayal-Pico%2C_islas_Azores%2C_Portugal%2C_2020-07-27%2C_DD_14.jpg",    "https://upload.wikimedia.org/wikipedia/commons/thumb/2/21/Thure_de_Thulstrup_-_Battle_of_Shiloh.jpg/251px-Thure_de_Thulstrup_-_Battle_of_Shiloh.jpg",    "https://upload.wikimedia.org/wikipedia/commons/thumb/2/21/Passion_fruits_-_whole_and_halved.jpg/270px-Passion_fruits_-_whole_and_halved.jpg",    "https://upload.wikimedia.org/wikipedia/commons/thumb/5/5e/Messier83_-_Heic1403a.jpg/277px-Messier83_-_Heic1403a.jpg",    "https://upload.wikimedia.org/wikipedia/commons/thumb/b/b6/2022-01-22_Men%27s_World_Cup_at_2021-22_St._Moritz%E2%80%93Celerina_Luge_World_Cup_and_European_Championships_by_Sandro_Halank%E2%80%93257.jpg/288px-2022-01-22_Men%27s_World_Cup_at_2021-22_St._Moritz%E2%80%93Celerina_Luge_World_Cup_and_European_Championships_by_Sandro_Halank%E2%80%93257.jpg",    "https://upload.wikimedia.org/wikipedia/commons/thumb/9/99/Wiesen_Pippau_%28Crepis_biennis%29-20220624-RM-123950.jpg/224px-Wiesen_Pippau_%28Crepis_biennis%29-20220624-RM-123950.jpg",]
```

### Create the loader[​](#create-the-loader "Direct link to Create the loader")

```
loader = ImageCaptionLoader(path_images=list_image_urls)list_docs = loader.load()list_docs
```

```
import requestsfrom PIL import ImageImage.open(requests.get(list_image_urls[0], stream=True).raw).convert("RGB")
```

### Create the index[​](#create-the-index "Direct link to Create the index")

```
from langchain.indexes import VectorstoreIndexCreatorindex = VectorstoreIndexCreator().from_loaders([loader])
```

### Query[​](#query "Direct link to Query")

```
query = "What's the painting about?"index.query(query)
```

```
query = "What kind of images are there?"index.query(query)
```

* * *

#### Help us out by providing feedback on this documentation page:
## IMSDb

> [IMSDb](https://imsdb.com/) is the `Internet Movie Script Database`.

This covers how to load `IMSDb` webpages into a document format that we can use downstream.

```
from langchain_community.document_loaders import IMSDbLoader
```

```
loader = IMSDbLoader("https://imsdb.com/scripts/BlacKkKlansman.html")
```

```
data[0].page_content[:500]
```

```
'\n\r\n\r\n\r\n\r\n                                    BLACKKKLANSMAN\r\n                         \r\n                         \r\n                         \r\n                         \r\n                                      Written by\r\n\r\n                          Charlie Wachtel & David Rabinowitz\r\n\r\n                                         and\r\n\r\n                              Kevin Willmott & Spike Lee\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n                         FADE IN:\r\n                         \r\n          SCENE FROM "GONE WITH'
```

```
{'source': 'https://imsdb.com/scripts/BlacKkKlansman.html'}
```

* * *

#### Help us out by providing feedback on this documentation page:
This notebook covers how to load documents from a `Joplin` database.

`Joplin` has a [REST API](https://joplinapp.org/api/references/rest_api/) for accessing its local database. This loader uses the API to retrieve all notes in the database and their metadata. This requires an access token that can be obtained from the app by following these steps:

You may either initialize the loader directly with the access token, or store it in the environment variable JOPLIN\_ACCESS\_TOKEN.

An alternative to this approach is to export the `Joplin`’s note database to Markdown files (optionally, with Front Matter metadata) and use a Markdown loader, such as ObsidianLoader, to load them.

```
from langchain_community.document_loaders import JoplinLoader
```
This notebook covers how to load data from a `Jupyter notebook (.html)` into a format suitable by LangChain.

`NotebookLoader.load()` loads the `.html` notebook file into a `Document` object.

```
[Document(page_content='\'markdown\' cell: \'[\'# Notebook\', \'\', \'This notebook covers how to load data from an .html notebook into a format suitable by LangChain.\']\'\n\n \'code\' cell: \'[\'from langchain_community.document_loaders import NotebookLoader\']\'\n\n \'code\' cell: \'[\'loader = NotebookLoader("example_data/notebook.html")\']\'\n\n \'markdown\' cell: \'[\'`NotebookLoader.load()` loads the `.html` notebook file into a `Document` object.\', \'\', \'**Parameters**:\', \'\', \'* `include_outputs` (bool): whether to include cell outputs in the resulting document (default is False).\', \'* `max_output_length` (int): the maximum number of characters to include from each cell output (default is 10).\', \'* `remove_newline` (bool): whether to remove newline characters from the cell sources and outputs (default is False).\', \'* `traceback` (bool): whether to include full traceback (default is False).\']\'\n\n \'code\' cell: \'[\'loader.load(include_outputs=True, max_output_length=20, remove_newline=True)\']\'\n\n', metadata={'source': 'example_data/notebook.html'})]
```
## LarkSuite (FeiShu)

> [LarkSuite](https://www.larksuite.com/) is an enterprise collaboration platform developed by ByteDance.

This notebook covers how to load data from the `LarkSuite` REST API into a format that can be ingested into LangChain, along with example usage for text summarization.

The LarkSuite API requires an access token (tenant\_access\_token or user\_access\_token), checkout [LarkSuite open platform document](https://open.larksuite.com/document) for API details.

```
from getpass import getpassfrom langchain_community.document_loaders.larksuite import LarkSuiteDocLoaderDOMAIN = input("larksuite domain")ACCESS_TOKEN = getpass("larksuite tenant_access_token or user_access_token")DOCUMENT_ID = input("larksuite document id")
```

```
from pprint import pprintlarksuite_loader = LarkSuiteDocLoader(DOMAIN, ACCESS_TOKEN, DOCUMENT_ID)docs = larksuite_loader.load()pprint(docs)
```

```
[Document(page_content='Test Doc\nThis is a Test Doc\n\n1\n2\n3\n\n', metadata={'document_id': 'V76kdbd2HoBbYJxdiNNccajunPf', 'revision_id': 11, 'title': 'Test Doc'})]
```

```
# see https://python.langchain.com/docs/use_cases/summarization for more detailsfrom langchain.chains.summarize import load_summarize_chainfrom langchain_community.llms.fake import FakeListLLMllm = FakeListLLM()chain = load_summarize_chain(llm, chain_type="map_reduce")chain.run(docs)
```

* * *

#### Help us out by providing feedback on this documentation page:
This notebook covers how to load data from the `Iugu REST API` into a format that can be ingested into LangChain, along with example usage for vectorization.

The Iugu API requires an access token, which can be found inside of the Iugu dashboard.

This document loader also requires a `resource` option which defines what data you want to load.

```
# Create a vectorstore retriever from the loader# see https://python.langchain.com/en/latest/modules/data_connection/getting_started.html for more detailsindex = VectorstoreIndexCreator().from_loaders([iugu_loader])iugu_doc_retriever = index.vectorstore.as_retriever()
```
## lakeFS

> [lakeFS](https://docs.lakefs.io/) provides scalable version control over the data lake, and uses Git-like semantics to create and access those versions.

This notebooks covers how to load document objects from a `lakeFS` path (whether it’s an object or a prefix).

## Initializing the lakeFS loader[​](#initializing-the-lakefs-loader "Direct link to Initializing the lakeFS loader")

Replace `ENDPOINT`, `LAKEFS_ACCESS_KEY`, and `LAKEFS_SECRET_KEY` values with your own.

```
from langchain_community.document_loaders import LakeFSLoader
```

```
ENDPOINT = ""LAKEFS_ACCESS_KEY = ""LAKEFS_SECRET_KEY = ""lakefs_loader = LakeFSLoader(    lakefs_access_key=LAKEFS_ACCESS_KEY,    lakefs_secret_key=LAKEFS_SECRET_KEY,    lakefs_endpoint=ENDPOINT,)
```

## Specifying a path[​](#specifying-a-path "Direct link to Specifying a path")

You can specify a prefix or a complete object path to control which files to load.

Specify the repository, reference (branch, commit id, or tag), and path in the corresponding `REPO`, `REF`, and `PATH` to load the documents from:

```
REPO = ""REF = ""PATH = ""lakefs_loader.set_repo(REPO)lakefs_loader.set_ref(REF)lakefs_loader.set_path(PATH)docs = lakefs_loader.load()docs
```

* * *

#### Help us out by providing feedback on this documentation page:
## Mastodon

> [Mastodon](https://joinmastodon.org/) is a federated social media and social networking service.

This loader fetches the text from the “toots” of a list of `Mastodon` accounts, using the `Mastodon.py` Python package.

Public accounts can the queried by default without any authentication. If non-public accounts or instances are queried, you have to register an application for your account which gets you an access token, and set that token and your account’s API base URL.

Then you need to pass in the Mastodon account names you want to extract, in the `@account@instance` format.

```
from langchain_community.document_loaders import MastodonTootsLoader
```

```
%pip install --upgrade --quiet  Mastodon.py
```

```
loader = MastodonTootsLoader(    mastodon_accounts=["@Gargron@mastodon.social"],    number_toots=50,  # Default value is 100)# Or set up access information to use a Mastodon app.# Note that the access token can either be passed into# constructor or you can set the environment "MASTODON_ACCESS_TOKEN".# loader = MastodonTootsLoader(#     access_token="<ACCESS TOKEN OF MASTODON APP>",#     api_base_url="<API BASE URL OF MASTODON APP INSTANCE>",#     mastodon_accounts=["@Gargron@mastodon.social"],#     number_toots=50,  # Default value is 100# )
```

```
documents = loader.load()for doc in documents[:3]:    print(doc.page_content)    print("=" * 80)
```

```
<p>It is tough to leave this behind and go back to reality. And some people live here! I’m sure there are downsides but it sounds pretty good to me right now.</p>================================================================================<p>I wish we could stay here a little longer, but it is time to go home 🥲</p>================================================================================<p>Last day of the honeymoon. And it’s <a href="https://mastodon.social/tags/caturday" class="mention hashtag" rel="tag">#<span>caturday</span></a>! This cute tabby came to the restaurant to beg for food and got some chicken.</p>================================================================================
```

The toot texts (the documents’ `page_content`) is by default HTML as returned by the Mastodon API.
## Merge Documents Loader

Merge the documents returned from a set of specified data loaders.

```
from langchain_community.document_loaders import WebBaseLoaderloader_web = WebBaseLoader(    "https://github.com/basecamp/handbook/blob/master/37signals-is-you.md")
```

```
from langchain_community.document_loaders import PyPDFLoaderloader_pdf = PyPDFLoader("../MachineLearning-Lecture01.pdf")
```

```
from langchain_community.document_loaders.merge import MergedDataLoaderloader_all = MergedDataLoader(loaders=[loader_web, loader_pdf])
```

```
docs_all = loader_all.load()
```

* * *

#### Help us out by providing feedback on this documentation page:
## mhtml

MHTML is a is used both for emails but also for archived webpages. MHTML, sometimes referred as MHT, stands for MIME HTML is a single file in which entire webpage is archived. When one saves a webpage as MHTML format, this file extension will contain HTML code, images, audio files, flash animation etc.

```
from langchain_community.document_loaders import MHTMLLoader
```

```
# Create a new loader object for the MHTML fileloader = MHTMLLoader(    file_path="../../../../../../tests/integration_tests/examples/example.mht")# Load the document from the filedocuments = loader.load()# Print the documents to see the resultsfor doc in documents:    print(doc)
```

```
page_content='LangChain\nLANG CHAIN 🦜️🔗Official Home Page\xa0\n\n\n\n\n\n\n\nIntegrations\n\n\n\nFeatures\n\n\n\n\nBlog\n\n\n\nConceptual Guide\n\n\n\n\nPython Repo\n\n\nJavaScript Repo\n\n\n\nPython Documentation \n\n\nJavaScript Documentation\n\n\n\n\nPython ChatLangChain \n\n\nJavaScript ChatLangChain\n\n\n\n\nDiscord \n\n\nTwitter\n\n\n\n\nIf you have any comments about our WEB page, you can \nwrite us at the address shown above.  However, due to \nthe limited number of personnel in our corporate office, we are unable to \nprovide a direct response.\n\nCopyright © 2023-2023 LangChain Inc.\n\n\n' metadata={'source': '../../../../../../tests/integration_tests/examples/example.mht', 'title': 'LangChain'}
```

* * *

#### Help us out by providing feedback on this documentation page:
## CoNLL-U

> [CoNLL-U](https://universaldependencies.org/format.html) is revised version of the CoNLL-X format. Annotations are encoded in plain text files (UTF-8, normalized to NFC, using only the LF character as line break, including an LF character at the end of file) with three types of lines: - Word lines containing the annotation of a word/token in 10 fields separated by single tab characters; see below. - Blank lines marking sentence boundaries. - Comment lines starting with hash (#).

This is an example of how to load a file in [CoNLL-U](https://universaldependencies.org/format.html) format. The whole file is treated as one document. The example data (`conllu.conllu`) is based on one of the standard UD/CoNLL-U examples.

```
from langchain_community.document_loaders import CoNLLULoader
```

```
loader = CoNLLULoader("example_data/conllu.conllu")
```

```
[Document(page_content='They buy and sell books.', metadata={'source': 'example_data/conllu.conllu'})]
```

* * *

#### Help us out by providing feedback on this documentation page:
## Microsoft PowerPoint

> [Microsoft PowerPoint](https://en.wikipedia.org/wiki/Microsoft_PowerPoint) is a presentation program by Microsoft.

This covers how to load `Microsoft PowerPoint` documents into a document format that we can use downstream.

```
from langchain_community.document_loaders import UnstructuredPowerPointLoader
```

```
loader = UnstructuredPowerPointLoader("example_data/fake-power-point.pptx")
```

```
[Document(page_content='Adding a Bullet Slide\n\nFind the bullet slide layout\n\nUse _TextFrame.text for first bullet\n\nUse _TextFrame.add_paragraph() for subsequent bullets\n\nHere is a lot of text!\n\nHere is some text in a text box!', metadata={'source': 'example_data/fake-power-point.pptx'})]
```

### Retain Elements[​](#retain-elements "Direct link to Retain Elements")

Under the hood, `Unstructured` creates different “elements” for different chunks of text. By default we combine those together, but you can easily keep that separation by specifying `mode="elements"`.

```
loader = UnstructuredPowerPointLoader(    "example_data/fake-power-point.pptx", mode="elements")
```

```
Document(page_content='Adding a Bullet Slide', lookup_str='', metadata={'source': 'example_data/fake-power-point.pptx'}, lookup_index=0)
```

## Using Azure AI Document Intelligence[​](#using-azure-ai-document-intelligence "Direct link to Using Azure AI Document Intelligence")

> [Azure AI Document Intelligence](https://aka.ms/doc-intelligence) (formerly known as `Azure Form Recognizer`) is machine-learning based service that extracts texts (including handwriting), tables, document structures (e.g., titles, section headings, etc.) and key-value-pairs from digital or scanned PDFs, images, Office and HTML files.
> 
> Document Intelligence supports `PDF`, `JPEG/JPG`, `PNG`, `BMP`, `TIFF`, `HEIF`, `DOCX`, `XLSX`, `PPTX` and `HTML`.

This current implementation of a loader using `Document Intelligence` can incorporate content page-wise and turn it into LangChain documents. The default output format is markdown, which can be easily chained with `MarkdownHeaderTextSplitter` for semantic document chunking. You can also use `mode="single"` or `mode="page"` to return pure texts in a single page or document split by page.

## Prerequisite[​](#prerequisite "Direct link to Prerequisite")

An Azure AI Document Intelligence resource in one of the 3 preview regions: **East US**, **West US2**, **West Europe** - follow [this document](https://learn.microsoft.com/azure/ai-services/document-intelligence/create-document-intelligence-resource?view=doc-intel-4.0.0) to create one if you don’t have. You will be passing `<endpoint>` and `<key>` as parameters to the loader.

```
%pip install --upgrade --quiet  langchain langchain-community azure-ai-documentintelligence
```

```
from langchain_community.document_loaders import AzureAIDocumentIntelligenceLoaderfile_path = "<filepath>"endpoint = "<endpoint>"key = "<key>"loader = AzureAIDocumentIntelligenceLoader(    api_endpoint=endpoint, api_key=key, file_path=file_path, api_model="prebuilt-layout")documents = loader.load()
```

* * *

#### Help us out by providing feedback on this documentation page:
## Copy Paste

This notebook covers how to load a document object from something you just want to copy and paste. In this case, you don’t even need to use a DocumentLoader, but rather can just construct the Document directly.

```
from langchain_community.docstore.document import Document
```

```
text = "..... put the text you copy pasted here......"
```

```
doc = Document(page_content=text)
```

If you want to add metadata about the where you got this piece of text, you easily can with the metadata key.

```
metadata = {"source": "internet", "date": "Friday"}
```

```
doc = Document(page_content=text, metadata=metadata)
```

* * *

#### Help us out by providing feedback on this documentation page:
## Modern Treasury

> [Modern Treasury](https://www.moderntreasury.com/) simplifies complex payment operations. It is a unified platform to power products and processes that move money. - Connect to banks and payment systems - Track transactions and balances in real-time - Automate payment operations for scale

This notebook covers how to load data from the `Modern Treasury REST API` into a format that can be ingested into LangChain, along with example usage for vectorization.

```
from langchain.indexes import VectorstoreIndexCreatorfrom langchain_community.document_loaders import ModernTreasuryLoader
```

The Modern Treasury API requires an organization ID and API key, which can be found in the Modern Treasury dashboard within developer settings.

This document loader also requires a `resource` option which defines what data you want to load.

Following resources are available:

`payment_orders` [Documentation](https://docs.moderntreasury.com/reference/payment-order-object)

`expected_payments` [Documentation](https://docs.moderntreasury.com/reference/expected-payment-object)

`returns` [Documentation](https://docs.moderntreasury.com/reference/return-object)

`incoming_payment_details` [Documentation](https://docs.moderntreasury.com/reference/incoming-payment-detail-object)

`counterparties` [Documentation](https://docs.moderntreasury.com/reference/counterparty-object)

`internal_accounts` [Documentation](https://docs.moderntreasury.com/reference/internal-account-object)

`external_accounts` [Documentation](https://docs.moderntreasury.com/reference/external-account-object)

`transactions` [Documentation](https://docs.moderntreasury.com/reference/transaction-object)

`ledgers` [Documentation](https://docs.moderntreasury.com/reference/ledger-object)

`ledger_accounts` [Documentation](https://docs.moderntreasury.com/reference/ledger-account-object)

`ledger_transactions` [Documentation](https://docs.moderntreasury.com/reference/ledger-transaction-object)

`events` [Documentation](https://docs.moderntreasury.com/reference/events)

`invoices` [Documentation](https://docs.moderntreasury.com/reference/invoices)

```
modern_treasury_loader = ModernTreasuryLoader("payment_orders")
```

```
# Create a vectorstore retriever from the loader# see https://python.langchain.com/en/latest/modules/data_connection/getting_started.html for more detailsindex = VectorstoreIndexCreator().from_loaders([modern_treasury_loader])modern_treasury_doc_retriever = index.vectorstore.as_retriever()
```

* * *

#### Help us out by providing feedback on this documentation page:
This notebook demonstrates the process of retrieving Cube’s data model metadata in a format suitable for passing to LLMs as embeddings, thereby enhancing contextual information.

[Cube](https://cube.dev/) is the Semantic Layer for building data apps. It helps data engineers and application developers access data from modern data stores, organize it into consistent definitions, and deliver it to every application.

Cube’s data model provides structure and definitions that are used as a context for LLM to understand data and generate correct queries. LLM doesn’t need to navigate complex joins and metrics calculations because Cube abstracts those and provides a simple interface that operates on the business-level terminology, instead of SQL table and column names. This simplification helps LLM to be less error-prone and avoid hallucinations.

```
import jwtfrom langchain_community.document_loaders import CubeSemanticLoaderapi_url = "https://api-example.gcp-us-central1.cubecloudapp.dev/cubejs-api/v1/meta"cubejs_api_secret = "api-secret-here"security_context = {}# Read more about security context here: https://cube.dev/docs/securityapi_token = jwt.encode(security_context, cubejs_api_secret, algorithm="HS256")loader = CubeSemanticLoader(api_url, api_token)documents = loader.load()
```

```
# Given string containing page contentpage_content = "Users View City, None"# Given dictionary containing metadatametadata = {    "table_name": "users_view",    "column_name": "users_view.city",    "column_data_type": "string",    "column_title": "Users View City",    "column_description": "None",    "column_member_type": "dimension",    "column_values": [        "Austin",        "Chicago",        "Los Angeles",        "Mountain View",        "New York",        "Palo Alto",        "San Francisco",        "Seattle",    ],    "cube_data_obj_type": "view",}
```
## Notion DB 1/2

> [Notion](https://www.notion.so/) is a collaboration platform with modified Markdown support that integrates kanban boards, tasks, wikis and databases. It is an all-in-one workspace for notetaking, knowledge and data management, and project and task management.

This notebook covers how to load documents from a Notion database dump.

In order to get this notion dump, follow these instructions:

## 🧑 Instructions for ingesting your own dataset[​](#instructions-for-ingesting-your-own-dataset "Direct link to 🧑 Instructions for ingesting your own dataset")

Export your dataset from Notion. You can do this by clicking on the three dots in the upper right hand corner and then clicking `Export`.

When exporting, make sure to select the `Markdown & CSV` format option.

This will produce a `.zip` file in your Downloads folder. Move the `.zip` file into this repository.

Run the following command to unzip the zip file (replace the `Export...` with your own file name as needed).

```
unzip Export-d3adfe0f-3131-4bf3-8987-a52017fc1bae.zip -d Notion_DB
```

Run the following command to ingest the data.

```
from langchain_community.document_loaders import NotionDirectoryLoader
```

```
loader = NotionDirectoryLoader("Notion_DB")
```

* * *

#### Help us out by providing feedback on this documentation page:
This loader fetches the logs from your applications in Datadog using the `datadog_api_client` Python package. You must initialize the loader with your `Datadog API key` and `APP key`, and you need to pass in the query to extract the desired logs.

```
[Document(page_content='message: grep: /etc/datadog-agent/system-probe.yaml: No such file or directory', metadata={'id': 'AgAAAYkwpLImvkjRpQAAAAAAAAAYAAAAAEFZa3dwTUFsQUFEWmZfLU5QdElnM3dBWQAAACQAAAAAMDE4OTMwYTQtYzk3OS00MmJjLTlhNDAtOTY4N2EwY2I5ZDdk', 'status': 'error', 'service': 'agent', 'tags': ['accessible-from-goog-gke-node', 'allow-external-ingress-high-ports', 'allow-external-ingress-http', 'allow-external-ingress-https', 'container_id:c7d8ecd27b5b3cfdf3b0df04b8965af6f233f56b7c3c2ffabfab5e3b6ccbd6a5', 'container_name:lab_datadog_1', 'datadog.pipelines:false', 'datadog.submission_auth:private_api_key', 'docker_image:datadog/agent:7.41.1', 'env:dd101-dev', 'hostname:lab-host', 'image_name:datadog/agent', 'image_tag:7.41.1', 'instance-id:7497601202021312403', 'instance-type:custom-1-4096', 'instruqt_aws_accounts:', 'instruqt_azure_subscriptions:', 'instruqt_gcp_projects:', 'internal-hostname:lab-host.d4rjybavkary.svc.cluster.local', 'numeric_project_id:3390740675', 'p-d4rjybavkary', 'project:instruqt-prod', 'service:agent', 'short_image:agent', 'source:agent', 'zone:europe-west1-b'], 'timestamp': datetime.datetime(2023, 7, 7, 13, 57, 27, 206000, tzinfo=tzutc())}), Document(page_content='message: grep: /etc/datadog-agent/system-probe.yaml: No such file or directory', metadata={'id': 'AgAAAYkwpLImvkjRpgAAAAAAAAAYAAAAAEFZa3dwTUFsQUFEWmZfLU5QdElnM3dBWgAAACQAAAAAMDE4OTMwYTQtYzk3OS00MmJjLTlhNDAtOTY4N2EwY2I5ZDdk', 'status': 'error', 'service': 'agent', 'tags': ['accessible-from-goog-gke-node', 'allow-external-ingress-high-ports', 'allow-external-ingress-http', 'allow-external-ingress-https', 'container_id:c7d8ecd27b5b3cfdf3b0df04b8965af6f233f56b7c3c2ffabfab5e3b6ccbd6a5', 'container_name:lab_datadog_1', 'datadog.pipelines:false', 'datadog.submission_auth:private_api_key', 'docker_image:datadog/agent:7.41.1', 'env:dd101-dev', 'hostname:lab-host', 'image_name:datadog/agent', 'image_tag:7.41.1', 'instance-id:7497601202021312403', 'instance-type:custom-1-4096', 'instruqt_aws_accounts:', 'instruqt_azure_subscriptions:', 'instruqt_gcp_projects:', 'internal-hostname:lab-host.d4rjybavkary.svc.cluster.local', 'numeric_project_id:3390740675', 'p-d4rjybavkary', 'project:instruqt-prod', 'service:agent', 'short_image:agent', 'source:agent', 'zone:europe-west1-b'], 'timestamp': datetime.datetime(2023, 7, 7, 13, 57, 27, 206000, tzinfo=tzutc())})]
```
## MongoDB

[MongoDB](https://www.mongodb.com/) is a NoSQL , document-oriented database that supports JSON-like documents with a dynamic schema.

## Overview[​](#overview "Direct link to Overview")

The MongoDB Document Loader returns a list of Langchain Documents from a MongoDB database.

The Loader requires the following parameters:

*   MongoDB connection string
*   MongoDB database name
*   MongoDB collection name
*   (Optional) Content Filter dictionary
*   (Optional) List of field names to include in the output

The output takes the following format:

*   pageContent= Mongo Document
*   metadata={‘database’: ‘\[database\_name\]’, ‘collection’: ‘\[collection\_name\]’}

## Load the Document Loader[​](#load-the-document-loader "Direct link to Load the Document Loader")

```
# add this import for running in jupyter notebookimport nest_asyncionest_asyncio.apply()
```

```
from langchain_community.document_loaders.mongodb import MongodbLoader
```

```
loader = MongodbLoader(    connection_string="mongodb://localhost:27017/",    db_name="sample_restaurants",    collection_name="restaurants",    filter_criteria={"borough": "Bronx", "cuisine": "Bakery"},    field_names=["name", "address"],)
```

```
docs = loader.load()len(docs)
```

```
Document(page_content="Morris Park Bake Shop {'building': '1007', 'coord': [-73.856077, 40.848447], 'street': 'Morris Park Ave', 'zipcode': '10462'}", metadata={'database': 'sample_restaurants', 'collection': 'restaurants'})
```