seansullivan/Langchain-Docs huggingface dataset:
Below is an example on how to load a local acreom vault into Langchain. As the local vault in acreom is a folder of plain text .md files, the loader requires the path to the directory.

Vault files may contain some metadata which is stored as a YAML header. These values will be added to the document’s metadata if `collect_metadata` is set to true.

```
loader = AcreomLoader("<path-to-acreom-vault>", collect_metadata=False)
```
## ChatLiteLLM

[LiteLLM](https://github.com/BerriAI/litellm) is a library that simplifies calling Anthropic, Azure, Huggingface, Replicate, etc.

This notebook covers how to get started with using Langchain + the LiteLLM I/O library.

```
from langchain_community.chat_models import ChatLiteLLMfrom langchain_core.messages import HumanMessage
```

```
chat = ChatLiteLLM(model="gpt-3.5-turbo")
```

```
messages = [    HumanMessage(        content="Translate this sentence from English to French. I love programming."    )]chat(messages)
```

```
AIMessage(content=" J'aime la programmation.", additional_kwargs={}, example=False)
```

## `ChatLiteLLM` also supports async and streaming functionality:[​](#chatlitellm-also-supports-async-and-streaming-functionality "Direct link to chatlitellm-also-supports-async-and-streaming-functionality")

```
from langchain.callbacks.manager import CallbackManagerfrom langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler
```

```
await chat.agenerate([messages])
```

```
LLMResult(generations=[[ChatGeneration(text=" J'aime programmer.", generation_info=None, message=AIMessage(content=" J'aime programmer.", additional_kwargs={}, example=False))]], llm_output={}, run=[RunInfo(run_id=UUID('8cc8fb68-1c35-439c-96a0-695036a93652'))])
```

```
chat = ChatLiteLLM(    streaming=True,    verbose=True,    callback_manager=CallbackManager([StreamingStdOutCallbackHandler()]),)chat(messages)
```

```
AIMessage(content=" J'aime la programmation.", additional_kwargs={}, example=False)
```
## LlamaEdge

[LlamaEdge](https://github.com/second-state/LlamaEdge) allows you to chat with LLMs of [GGUF](https://github.com/ggerganov/llama.cpp/blob/master/gguf-py/README.md) format both locally and via chat service.

*   `LlamaEdgeChatService` provides developers an OpenAI API compatible service to chat with LLMs via HTTP requests.
    
*   `LlamaEdgeChatLocal` enables developers to chat with LLMs locally (coming soon).
    

Both `LlamaEdgeChatService` and `LlamaEdgeChatLocal` run on the infrastructure driven by [WasmEdge Runtime](https://wasmedge.org/), which provides a lightweight and portable WebAssembly container environment for LLM inference tasks.

## Chat via API Service[​](#chat-via-api-service "Direct link to Chat via API Service")

`LlamaEdgeChatService` works on the `llama-api-server`. Following the steps in [llama-api-server quick-start](https://github.com/second-state/llama-utils/tree/main/api-server#readme), you can host your own API service so that you can chat with any models you like on any device you have anywhere as long as the internet is available.

```
from langchain_community.chat_models.llama_edge import LlamaEdgeChatServicefrom langchain_core.messages import HumanMessage, SystemMessage
```

### Chat with LLMs in the non-streaming mode[​](#chat-with-llms-in-the-non-streaming-mode "Direct link to Chat with LLMs in the non-streaming mode")

```
# service urlservice_url = "https://b008-54-186-154-209.ngrok-free.app"# create wasm-chat service instancechat = LlamaEdgeChatService(service_url=service_url)# create message sequencesystem_message = SystemMessage(content="You are an AI assistant")user_message = HumanMessage(content="What is the capital of France?")messages = [system_message, user_message]# chat with wasm-chat serviceresponse = chat(messages)print(f"[Bot] {response.content}")
```

```
[Bot] Hello! The capital of France is Paris.
```

### Chat with LLMs in the streaming mode[​](#chat-with-llms-in-the-streaming-mode "Direct link to Chat with LLMs in the streaming mode")

```
# service urlservice_url = "https://b008-54-186-154-209.ngrok-free.app"# create wasm-chat service instancechat = LlamaEdgeChatService(service_url=service_url, streaming=True)# create message sequencesystem_message = SystemMessage(content="You are an AI assistant")user_message = HumanMessage(content="What is the capital of Norway?")messages = [    system_message,    user_message,]output = ""for chunk in chat.stream(messages):    # print(chunk.content, end="", flush=True)    output += chunk.contentprint(f"[Bot] {output}")
```

```
[Bot]   Hello! I'm happy to help you with your question. The capital of Norway is Oslo.
```
[Minimax](https://api.minimax.chat/) is a Chinese startup that provides LLM service for companies and individuals.

This example goes over how to use LangChain to interact with MiniMax Inference for Chat.

```
import osos.environ["MINIMAX_GROUP_ID"] = "MINIMAX_GROUP_ID"os.environ["MINIMAX_API_KEY"] = "MINIMAX_API_KEY"
```
## Airbyte JSON (Deprecated)

Note: `AirbyteJSONLoader` is deprecated. Please use [`AirbyteLoader`](https://python.langchain.com/docs/integrations/document_loaders/airbyte/) instead.

> [Airbyte](https://github.com/airbytehq/airbyte) is a data integration platform for ELT pipelines from APIs, databases & files to warehouses & lakes. It has the largest catalog of ELT connectors to data warehouses and databases.

This covers how to load any source from Airbyte into a local JSON file that can be read in as a document

Prereqs: Have docker desktop installed

Steps:

1.  Clone Airbyte from GitHub - `git clone https://github.com/airbytehq/airbyte.git`
    
2.  Switch into Airbyte directory - `cd airbyte`
    
3.  Start Airbyte - `docker compose up`
    
4.  In your browser, just visit http://localhost:8000. You will be asked for a username and password. By default, that’s username `airbyte` and password `password`.
    
5.  Setup any source you wish.
    
6.  Set destination as Local JSON, with specified destination path - lets say `/json_data`. Set up manual sync.
    
7.  Run the connection.
    
8.  To see what files are create, you can navigate to: `file:///tmp/airbyte_local`
    
9.  Find your data and copy path. That path should be saved in the file variable below. It should start with `/tmp/airbyte_local`
    

```
from langchain_community.document_loaders import AirbyteJSONLoader
```

```
!ls /tmp/airbyte_local/json_data/
```

```
_airbyte_raw_pokemon.jsonl
```

```
loader = AirbyteJSONLoader("/tmp/airbyte_local/json_data/_airbyte_raw_pokemon.jsonl")
```

```
print(data[0].page_content[:500])
```

```
abilities: ability: name: blazeurl: https://pokeapi.co/api/v2/ability/66/is_hidden: Falseslot: 1ability: name: solar-powerurl: https://pokeapi.co/api/v2/ability/94/is_hidden: Trueslot: 3base_experience: 267forms: name: charizardurl: https://pokeapi.co/api/v2/pokemon-form/6/game_indices: game_index: 180version: name: redurl: https://pokeapi.co/api/v2/version/1/game_index: 180version: name: blueurl: https://pokeapi.co/api/v2/version/2/game_index: 180version: n
```

* * *

#### Help us out by providing feedback on this documentation page:
Note: This connector-specific loader is deprecated. Please use [`AirbyteLoader`](https://python.langchain.com/docs/integrations/document_loaders/airbyte/) instead.

This loader exposes the Salesforce connector as a document loader, allowing you to load various Salesforce objects as documents.

First, you need to install the `airbyte-source-salesforce` python package.

```
{  "client_id": "<oauth client id>",  "client_secret": "<oauth client secret>",  "refresh_token": "<oauth refresh token>",  "start_date": "<date from which to start retrieving records from in ISO format, e.g. 2020-10-20T00:00:00Z>",  "is_sandbox": False, # set to True if you're using a sandbox environment  "streams_criteria": [ # Array of filters for salesforce objects that should be loadable    {"criteria": "exacts", "value": "Account"}, # Exact name of salesforce object    {"criteria": "starts with", "value": "Asset"}, # Prefix of the name    # Other allowed criteria: ends with, contains, starts not with, ends not with, not contains, not exacts  ],}
```

By default all fields are stored as metadata in the documents and the text is set to an empty string. Construct the text of the document by transforming the documents returned by the reader.

```
from langchain_community.document_loaders.airbyte import AirbyteSalesforceLoaderconfig = {    # your salesforce configuration}loader = AirbyteSalesforceLoader(    config=config, stream_name="asset")  # check the documentation linked above for a list of all streams
```

As `load` returns a list, it will block until all documents are loaded. To have better control over this process, you can also you the `lazy_load` method which returns an iterator instead:

Keep in mind that by default the page content is empty and the metadata object contains all the information from the record. To create documents in a different, pass in a record\_handler function when creating the loader:

```
from langchain_community.docstore.document import Documentdef handle_record(record, id):    return Document(page_content=record.data["title"], metadata=record.data)loader = AirbyteSalesforceLoader(    config=config, record_handler=handle_record, stream_name="asset")docs = loader.load()
```

Some streams allow incremental loading, this means the source keeps track of synced records and won’t load them again. This is useful for sources that have a high volume of data and are updated frequently.

To take advantage of this, store the `last_state` property of the loader and pass it in when creating the loader again. This will ensure that only new records are loaded.

```
last_state = loader.last_state  # store safelyincremental_loader = AirbyteSalesforceLoader(    config=config, stream_name="asset", state=last_state)new_docs = incremental_loader.load()
```
## MoonshotChat

[Moonshot](https://platform.moonshot.cn/) is a Chinese startup that provides LLM service for companies and individuals.

This example goes over how to use LangChain to interact with Moonshot Inference for Chat.

```
import os# Generate your api key from: https://platform.moonshot.cn/console/api-keysos.environ["MOONSHOT_API_KEY"] = "MOONSHOT_API_KEY"
```

```
from langchain_community.chat_models.moonshot import MoonshotChatfrom langchain_core.messages import HumanMessage, SystemMessage
```

```
chat = MoonshotChat()# or use a specific model# Available models: https://platform.moonshot.cn/docs# chat = MoonshotChat(model="moonshot-v1-128k")
```

```
messages = [    SystemMessage(        content="You are a helpful assistant that translates English to French."    ),    HumanMessage(        content="Translate this sentence from English to French. I love programming."    ),]chat.invoke(messages)
```

* * *

#### Help us out by providing feedback on this documentation page:
## ChatOctoAI

[OctoAI](https://docs.octoai.cloud/docs) offers easy access to efficient compute and enables users to integrate their choice of AI models into applications. The `OctoAI` compute service helps you run, tune, and scale AI applications easily.

This notebook demonstrates the use of `langchain.chat_models.ChatOctoAI` for [OctoAI endpoints](https://octoai.cloud/text).

## Setup[​](#setup "Direct link to Setup")

To run our example app, there are two simple steps to take:

1.  Get an API Token from [your OctoAI account page](https://octoai.cloud/settings).
    
2.  Paste your API token in in the code cell below or use the `octoai_api_token` keyword argument.
    

Note: If you want to use a different model than the [available models](https://octoai.cloud/text?selectedTags=Chat), you can containerize the model and make a custom OctoAI endpoint yourself, by following [Build a Container from Python](https://octo.ai/docs/bring-your-own-model/advanced-build-a-container-from-scratch-in-python) and [Create a Custom Endpoint from a Container](https://octo.ai/docs/bring-your-own-model/create-custom-endpoints-from-a-container/create-custom-endpoints-from-a-container) and then updating your `OCTOAI_API_BASE` environment variable.

```
import osos.environ["OCTOAI_API_TOKEN"] = "OCTOAI_API_TOKEN"
```

```
from langchain_community.chat_models import ChatOctoAIfrom langchain_core.messages import HumanMessage, SystemMessage
```

## Example[​](#example "Direct link to Example")

```
chat = ChatOctoAI(max_tokens=300, model_name="mixtral-8x7b-instruct")
```

```
messages = [    SystemMessage(content="You are a helpful assistant."),    HumanMessage(content="Tell me about Leonardo da Vinci briefly."),]print(chat(messages).content)
```

Leonardo da Vinci (1452-1519) was an Italian polymath who is often considered one of the greatest painters in history. However, his genius extended far beyond art. He was also a scientist, inventor, mathematician, engineer, anatomist, geologist, and cartographer.

Da Vinci is best known for his paintings such as the Mona Lisa, The Last Supper, and The Virgin of the Rocks. His scientific studies were ahead of his time, and his notebooks contain detailed drawings and descriptions of various machines, human anatomy, and natural phenomena.

Despite never receiving a formal education, da Vinci’s insatiable curiosity and observational skills made him a pioneer in many fields. His work continues to inspire and influence artists, scientists, and thinkers today.
This notebook shows how to use an experimental wrapper around Ollama that gives it the same API as OpenAI Functions.

Note that more powerful and capable models will perform better with complex schema and/or multiple functions. The examples below use Mistral. For a complete list of supported models and model variants, see the [Ollama model library](https://ollama.ai/library).

You can initialize OllamaFunctions in a similar way to how you’d initialize a standard ChatOllama instance:

You can then bind functions defined with JSON Schema parameters and a `function_call` parameter to force the model to call the given function:

```
model = model.bind(    functions=[        {            "name": "get_current_weather",            "description": "Get the current weather in a given location",            "parameters": {                "type": "object",                "properties": {                    "location": {                        "type": "string",                        "description": "The city and state, " "e.g. San Francisco, CA",                    },                    "unit": {                        "type": "string",                        "enum": ["celsius", "fahrenheit"],                    },                },                "required": ["location"],            },        }    ],    function_call={"name": "get_current_weather"},)
```

Calling a function with this model then results in JSON output matching the provided schema:

```
AIMessage(content='', additional_kwargs={'function_call': {'name': 'get_current_weather', 'arguments': '{"location": "Boston, MA", "unit": "celsius"}'}})
```

One useful thing you can do with function calling here is extracting properties from a given input in a structured format:

```
from langchain.chains import create_extraction_chain# Schemaschema = {    "properties": {        "name": {"type": "string"},        "height": {"type": "integer"},        "hair_color": {"type": "string"},    },    "required": ["name", "height"],}# Inputinput = """Alex is 5 feet tall. Claudia is 1 feet taller than Alex and jumps higher than him. Claudia is a brunette and Alex is blonde."""# Run chainllm = OllamaFunctions(model="mistral", temperature=0)chain = create_extraction_chain(schema, llm)chain.run(input)
```

```
[{'name': 'Alex', 'height': 5, 'hair_color': 'blonde'}, {'name': 'Claudia', 'height': 6, 'hair_color': 'brunette'}]
```
## Airtable

```
%pip install --upgrade --quiet  pyairtable
```

```
from langchain_community.document_loaders import AirtableLoader
```

*   Get your API key [here](https://support.airtable.com/docs/creating-and-using-api-keys-and-access-tokens).
*   Get ID of your base [here](https://airtable.com/developers/web/api/introduction).
*   Get your table ID from the table url as shown [here](https://www.highviewapps.com/kb/where-can-i-find-the-airtable-base-id-and-table-id/#:~:text=Both%20the%20Airtable%20Base%20ID,URL%20that%20begins%20with%20tbl).

```
api_key = "xxx"base_id = "xxx"table_id = "xxx"
```

```
loader = AirtableLoader(api_key, table_id, base_id)docs = loader.load()
```

Returns each table row as `dict`.

```
eval(docs[0].page_content)
```

```
{'id': 'recF3GbGZCuh9sXIQ', 'createdTime': '2023-06-09T04:47:21.000Z', 'fields': {'Priority': 'High',  'Status': 'In progress',  'Name': 'Document Splitters'}}
```

* * *

#### Help us out by providing feedback on this documentation page:
## Arxiv

> [arXiv](https://arxiv.org/) is an open-access archive for 2 million scholarly articles in the fields of physics, mathematics, computer science, quantitative biology, quantitative finance, statistics, electrical engineering and systems science, and economics.

This notebook shows how to load scientific articles from `Arxiv.org` into a document format that we can use downstream.

## Installation[​](#installation "Direct link to Installation")

First, you need to install `arxiv` python package.

```
%pip install --upgrade --quiet  arxiv
```

Second, you need to install `PyMuPDF` python package which transforms PDF files downloaded from the `arxiv.org` site into the text format.

```
%pip install --upgrade --quiet  pymupdf
```

## Examples[​](#examples "Direct link to Examples")

`ArxivLoader` has these arguments: - `query`: free text which used to find documents in the Arxiv - optional `load_max_docs`: default=100. Use it to limit number of downloaded documents. It takes time to download all 100 documents, so use a small number for experiments. - optional `load_all_available_meta`: default=False. By default only the most important fields downloaded: `Published` (date when document was published/last updated), `Title`, `Authors`, `Summary`. If True, other fields also downloaded.

```
from langchain_community.document_loaders import ArxivLoader
```

```
docs = ArxivLoader(query="1605.08386", load_max_docs=2).load()len(docs)
```

```
docs[0].metadata  # meta-information of the Document
```

```
{'Published': '2016-05-26', 'Title': 'Heat-bath random walks with Markov bases', 'Authors': 'Caprice Stanley, Tobias Windisch', 'Summary': 'Graphs on lattice points are studied whose edges come from a finite set of\nallowed moves of arbitrary length. We show that the diameter of these graphs on\nfibers of a fixed integer matrix can be bounded from above by a constant. We\nthen study the mixing behaviour of heat-bath random walks on these graphs. We\nalso state explicit conditions on the set of moves so that the heat-bath random\nwalk, a generalization of the Glauber dynamics, is an expander in fixed\ndimension.'}
```

```
docs[0].page_content[:400]  # all pages of the Document content
```

```
'arXiv:1605.08386v1  [math.CO]  26 May 2016\nHEAT-BATH RANDOM WALKS WITH MARKOV BASES\nCAPRICE STANLEY AND TOBIAS WINDISCH\nAbstract. Graphs on lattice points are studied whose edges come from a ﬁnite set of\nallowed moves of arbitrary length. We show that the diameter of these graphs on ﬁbers of a\nﬁxed integer matrix can be bounded from above by a constant. We then study the mixing\nbehaviour of heat-b'
```
## Apify Dataset

> [Apify Dataset](https://docs.apify.com/platform/storage/dataset) is a scalable append-only storage with sequential access built for storing structured web scraping results, such as a list of products or Google SERPs, and then export them to various formats like JSON, CSV, or Excel. Datasets are mainly used to save results of [Apify Actors](https://apify.com/store)—serverless cloud programs for various web scraping, crawling, and data extraction use cases.

This notebook shows how to load Apify datasets to LangChain.

## Prerequisites[​](#prerequisites "Direct link to Prerequisites")

You need to have an existing dataset on the Apify platform. If you don’t have one, please first check out [this notebook](https://python.langchain.com/docs/integrations/tools/apify/) on how to use Apify to extract content from documentation, knowledge bases, help centers, or blogs.

```
%pip install --upgrade --quiet  apify-client
```

First, import `ApifyDatasetLoader` into your source code:

```
from langchain_community.document_loaders import ApifyDatasetLoaderfrom langchain_core.documents import Document
```

Then provide a function that maps Apify dataset record fields to LangChain `Document` format.

For example, if your dataset items are structured like this:

```
{    "url": "https://apify.com",    "text": "Apify is the best web scraping and automation platform."}
```

The mapping function in the code below will convert them to LangChain `Document` format, so that you can use them further with any LLM model (e.g. for question answering).

```
loader = ApifyDatasetLoader(    dataset_id="your-dataset-id",    dataset_mapping_function=lambda dataset_item: Document(        page_content=dataset_item["text"], metadata={"source": dataset_item["url"]}    ),)
```

## An example with question answering[​](#an-example-with-question-answering "Direct link to An example with question answering")

In this example, we use data from a dataset to answer a question.

```
from langchain.indexes import VectorstoreIndexCreatorfrom langchain_community.docstore.document import Documentfrom langchain_community.document_loaders import ApifyDatasetLoader
```

```
loader = ApifyDatasetLoader(    dataset_id="your-dataset-id",    dataset_mapping_function=lambda item: Document(        page_content=item["text"] or "", metadata={"source": item["url"]}    ),)
```

```
index = VectorstoreIndexCreator().from_loaders([loader])
```

```
query = "What is Apify?"result = index.query_with_sources(query)
```

```
print(result["answer"])print(result["sources"])
```

```
 Apify is a platform for developing, running, and sharing serverless cloud programs. It enables users to create web scraping and automation tools and publish them on the Apify platform.https://docs.apify.com/platform/actors, https://docs.apify.com/platform/actors/running/actors-in-store, https://docs.apify.com/platform/security, https://docs.apify.com/platform/actors/examples
```

* * *

#### Help us out by providing feedback on this documentation page:
## AstraDB

DataStax [Astra DB](https://docs.datastax.com/en/astra/home/astra.html) is a serverless vector-capable database built on Cassandra and made conveniently available through an easy-to-use JSON API.

## Overview[​](#overview "Direct link to Overview")

The AstraDB Document Loader returns a list of Langchain Documents from an AstraDB database.

The Loader takes the following parameters:

*   `api_endpoint`: AstraDB API endpoint. Looks like `https://01234567-89ab-cdef-0123-456789abcdef-us-east1.apps.astra.datastax.com`
*   `token`: AstraDB token. Looks like `AstraCS:6gBhNmsk135....`
*   `collection_name` : AstraDB collection name
*   `namespace`: (Optional) AstraDB namespace
*   `filter_criteria`: (Optional) Filter used in the find query
*   `projection`: (Optional) Projection used in the find query
*   `find_options`: (Optional) Options used in the find query
*   `nb_prefetched`: (Optional) Number of documents pre-fetched by the loader
*   `extraction_function`: (Optional) A function to convert the AstraDB document to the LangChain `page_content` string. Defaults to `json.dumps`

The following metadata is set to the LangChain Documents metadata output:

```
{    metadata : {        "namespace": "...",         "api_endpoint": "...",         "collection": "..."    }}
```

## Load documents with the Document Loader[​](#load-documents-with-the-document-loader "Direct link to Load documents with the Document Loader")

```
from langchain_community.document_loaders import AstraDBLoader
```

```
from getpass import getpassASTRA_DB_API_ENDPOINT = input("ASTRA_DB_API_ENDPOINT = ")ASTRA_DB_APPLICATION_TOKEN = getpass("ASTRA_DB_APPLICATION_TOKEN = ")
```

```
loader = AstraDBLoader(    api_endpoint=ASTRA_DB_API_ENDPOINT,    token=ASTRA_DB_APPLICATION_TOKEN,    collection_name="movie_reviews",    projection={"title": 1, "reviewtext": 1},    find_options={"limit": 10},)
```

```
Document(page_content='{"_id": "659bdffa16cbc4586b11a423", "title": "Dangerous Men", "reviewtext": "\\"Dangerous Men,\\" the picture\'s production notes inform, took 26 years to reach the big screen. After having seen it, I wonder: What was the rush?"}', metadata={'namespace': 'default_keyspace', 'api_endpoint': 'https://01234567-89ab-cdef-0123-456789abcdef-us-east1.apps.astra.datastax.com', 'collection': 'movie_reviews'})
```
[Amazon Athena](https://aws.amazon.com/athena/) is a serverless, interactive analytics service built on open-source frameworks, supporting open-table and file formats. `Athena` provides a simplified, flexible way to analyze petabytes of data where it lives. Analyze data or build applications from an Amazon Simple Storage Service (S3) data lake and 30 data sources, including on-premises data sources or other cloud systems using SQL or Python. `Athena` is built on open-source `Trino` and `Presto` engines and `Apache Spark` frameworks, with no provisioning or configuration effort required.

This notebook goes over how to load documents from `AWS Athena`.

```
database_name = "my_database"s3_output_path = "s3://my_bucket/query_results/"query = "SELECT * FROM my_table"profile_name = "my_profile"loader = AthenaLoader(    query=query,    database=database_name,    s3_output_uri=s3_output_path,    profile_name=profile_name,)documents = loader.load()print(documents)
```

```
database_name = "my_database"s3_output_path = "s3://my_bucket/query_results/"query = "SELECT * FROM my_table"profile_name = "my_profile"metadata_columns = ["_row", "_created_at"]loader = AthenaLoader(    query=query,    database=database_name,    s3_output_uri=s3_output_path,    profile_name=profile_name,    metadata_columns=metadata_columns,)documents = loader.load()print(documents)
```
## Async Chromium

Chromium is one of the browsers supported by Playwright, a library used to control browser automation.

By running `p.chromium.launch(headless=True)`, we are launching a headless instance of Chromium.

Headless mode means that the browser is running without a graphical user interface.

`AsyncChromiumLoader` loads the page, and then we use `Html2TextTransformer` to transform to text.

```
%pip install --upgrade --quiet  playwright beautifulsoup4!playwright install
```

```
from langchain_community.document_loaders import AsyncChromiumLoaderurls = ["https://www.wsj.com"]loader = AsyncChromiumLoader(urls)docs = loader.load()docs[0].page_content[0:100]
```

```
'<!DOCTYPE html><html lang="en"><head><script src="https://s0.2mdn.net/instream/video/client.js" asyn'
```

If you are using Jupyter notebooks, you might need to apply `nest_asyncio` before loading the documents.

```
!pip install nest-asyncioimport nest_asyncionest_asyncio.apply()
```

```
from langchain_community.document_transformers import Html2TextTransformerhtml2text = Html2TextTransformer()docs_transformed = html2text.transform_documents(docs)docs_transformed[0].page_content[0:500]
```

```
"Skip to Main ContentSkip to SearchSkip to... Select * Top News * What's News *\nFeatured Stories * Retirement * Life & Arts * Hip-Hop * Sports * Video *\nEconomy * Real Estate * Sports * CMO * CIO * CFO * Risk & Compliance *\nLogistics Report * Sustainable Business * Heard on the Street * Barron’s *\nMarketWatch * Mansion Global * Penta * Opinion * Journal Reports * Sponsored\nOffers Explore Our Brands * WSJ * * * * * Barron's * * * * * MarketWatch * * *\n* * IBD # The Wall Street Journal SubscribeSig"
```
## AWS S3 File

> [Amazon Simple Storage Service (Amazon S3)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-folders.html) is an object storage service.

> [AWS S3 Buckets](https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingBucket.html)

This covers how to load document objects from an `AWS S3 File` object.

```
from langchain_community.document_loaders import S3FileLoader
```

```
%pip install --upgrade --quiet  boto3
```

```
loader = S3FileLoader("testing-hwc", "fake.docx")
```

```
[Document(page_content='Lorem ipsum dolor sit amet.', lookup_str='', metadata={'source': 's3://testing-hwc/fake.docx'}, lookup_index=0)]
```

## Configuring the AWS Boto3 client[​](#configuring-the-aws-boto3-client "Direct link to Configuring the AWS Boto3 client")

You can configure the AWS [Boto3](https://boto3.amazonaws.com/v1/documentation/api/latest/index.html) client by passing named arguments when creating the S3DirectoryLoader. This is useful for instance when AWS credentials can’t be set as environment variables. See the [list of parameters](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html#boto3.session.Session) that can be configured.

```
loader = S3FileLoader(    "testing-hwc", "fake.docx", aws_access_key_id="xxxx", aws_secret_access_key="yyyy")
```

* * *

#### Help us out by providing feedback on this documentation page:
## AWS S3 Directory

> [Amazon Simple Storage Service (Amazon S3)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-folders.html) is an object storage service

> [AWS S3 Directory](https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-folders.html)

This covers how to load document objects from an `AWS S3 Directory` object.

```
%pip install --upgrade --quiet  boto3
```

```
from langchain_community.document_loaders import S3DirectoryLoader
```

```
loader = S3DirectoryLoader("testing-hwc")
```

## Specifying a prefix[​](#specifying-a-prefix "Direct link to Specifying a prefix")

You can also specify a prefix for more finegrained control over what files to load.

```
loader = S3DirectoryLoader("testing-hwc", prefix="fake")
```

```
[Document(page_content='Lorem ipsum dolor sit amet.', lookup_str='', metadata={'source': 's3://testing-hwc/fake.docx'}, lookup_index=0)]
```

## Configuring the AWS Boto3 client[​](#configuring-the-aws-boto3-client "Direct link to Configuring the AWS Boto3 client")

You can configure the AWS [Boto3](https://boto3.amazonaws.com/v1/documentation/api/latest/index.html) client by passing named arguments when creating the S3DirectoryLoader. This is useful for instance when AWS credentials can’t be set as environment variables. See the [list of parameters](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html#boto3.session.Session) that can be configured.

```
loader = S3DirectoryLoader(    "testing-hwc", aws_access_key_id="xxxx", aws_secret_access_key="yyyy")
```

* * *

#### Help us out by providing feedback on this documentation page:
## AsyncHtml

`AsyncHtmlLoader` loads raw HTML from a list of URLs concurrently.

```
from langchain_community.document_loaders import AsyncHtmlLoader
```

```
urls = ["https://www.espn.com", "https://lilianweng.github.io/posts/2023-06-23-agent/"]loader = AsyncHtmlLoader(urls)docs = loader.load()
```

```
Fetching pages: 100%|############| 2/2 [00:00<00:00,  9.96it/s]
```

```
docs[0].page_content[1000:2000]
```

```
' news. Stream exclusive games on ESPN+ and play fantasy sports." />\n<meta property="og:image" content="https://a1.espncdn.com/combiner/i?img=%2Fi%2Fespn%2Fespn_logos%2Fespn_red.png"/>\n<meta property="og:image:width" content="1200" />\n<meta property="og:image:height" content="630" />\n<meta property="og:type" content="website" />\n<meta name="twitter:site" content="espn" />\n<meta name="twitter:url" content="https://www.espn.com" />\n<meta name="twitter:title" content="ESPN - Serving Sports Fans. Anytime. Anywhere."/>\n<meta name="twitter:description" content="Visit ESPN for live scores, highlights and sports news. Stream exclusive games on ESPN+ and play fantasy sports." />\n<meta name="twitter:card" content="summary">\n<meta name="twitter:app:name:iphone" content="ESPN"/>\n<meta name="twitter:app:id:iphone" content="317469184"/>\n<meta name="twitter:app:name:googleplay" content="ESPN"/>\n<meta name="twitter:app:id:googleplay" content="com.espn.score_center"/>\n<meta name="title" content="ESPN - '
```

```
docs[1].page_content[1000:2000]
```

```
'al" href="https://lilianweng.github.io/posts/2023-06-23-agent/" />\n<link crossorigin="anonymous" href="/assets/css/stylesheet.min.67a6fb6e33089cb29e856bcc95d7aa39f70049a42b123105531265a0d9f1258b.css" integrity="sha256-Z6b7bjMInLKehWvMldeqOfcASaQrEjEFUxJloNnxJYs=" rel="preload stylesheet" as="style">\n<script defer crossorigin="anonymous" src="/assets/js/highlight.min.7680afc38aa6b15ddf158a4f3780b7b1f7dde7e91d26f073e6229bb7a0793c92.js" integrity="sha256-doCvw4qmsV3fFYpPN4C3sffd5&#43;kdJvBz5iKbt6B5PJI="\n    onload="hljs.initHighlightingOnLoad();"></script>\n<link rel="icon" href="https://lilianweng.github.io/favicon_peach.ico">\n<link rel="icon" type="image/png" sizes="16x16" href="https://lilianweng.github.io/favicon-16x16.png">\n<link rel="icon" type="image/png" sizes="32x32" href="https://lilianweng.github.io/favicon-32x32.png">\n<link rel="apple-touch-icon" href="https://lilianweng.github.io/apple-touch-icon.png">\n<link rel="mask-icon" href="https://lilianweng.github.io/safari-pinned-tab.'
```

* * *

#### Help us out by providing feedback on this documentation page:
## Azure AI Data

> [Azure AI Studio](https://ai.azure.com/) provides the capability to upload data assets to cloud storage and register existing data assets from the following sources:
> 
> *   `Microsoft OneLake`
> *   `Azure Blob Storage`
> *   `Azure Data Lake gen 2`

The benefit of this approach over `AzureBlobStorageContainerLoader` and `AzureBlobStorageFileLoader` is that authentication is handled seamlessly to cloud storage. You can use either _identity-based_ data access control to the data or _credential-based_ (e.g. SAS token, account key). In the case of credential-based data access you do not need to specify secrets in your code or set up key vaults - the system handles that for you.

This notebook covers how to load document objects from a data asset in AI Studio.

```
%pip install --upgrade --quiet  azureml-fsspec, azure-ai-generative
```

```
from azure.ai.resources.client import AIClientfrom azure.identity import DefaultAzureCredentialfrom langchain_community.document_loaders import AzureAIDataLoader
```

```
# Create a connection to your projectclient = AIClient(    credential=DefaultAzureCredential(),    subscription_id="<subscription_id>",    resource_group_name="<resource_group_name>",    project_name="<project_name>",)
```

```
# get the latest version of your data assetdata_asset = client.data.get(name="<data_asset_name>", label="latest")
```

```
# load the data assetloader = AzureAIDataLoader(url=data_asset.path)
```

```
[Document(page_content='Lorem ipsum dolor sit amet.', lookup_str='', metadata={'source': '/var/folders/y6/8_bzdg295ld6s1_97_12m4lr0000gn/T/tmpaa9xl6ch/fake.docx'}, lookup_index=0)]
```

## Specifying a glob pattern[​](#specifying-a-glob-pattern "Direct link to Specifying a glob pattern")

You can also specify a glob pattern for more finegrained control over what files to load. In the example below, only files with a `pdf` extension will be loaded.

```
loader = AzureAIDataLoader(url=data_asset.path, glob="*.pdf")
```

```
[Document(page_content='Lorem ipsum dolor sit amet.', lookup_str='', metadata={'source': '/var/folders/y6/8_bzdg295ld6s1_97_12m4lr0000gn/T/tmpujbkzf_l/fake.docx'}, lookup_index=0)]
```
## Azure Blob Storage Container

> [Azure Blob Storage](https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction) is Microsoft’s object storage solution for the cloud. Blob Storage is optimized for storing massive amounts of unstructured data. Unstructured data is data that doesn’t adhere to a particular data model or definition, such as text or binary data.

`Azure Blob Storage` is designed for: - Serving images or documents directly to a browser. - Storing files for distributed access. - Streaming video and audio. - Writing to log files. - Storing data for backup and restore, disaster recovery, and archiving. - Storing data for analysis by an on-premises or Azure-hosted service.

This notebook covers how to load document objects from a container on `Azure Blob Storage`.

```
%pip install --upgrade --quiet  azure-storage-blob
```

```
from langchain_community.document_loaders import AzureBlobStorageContainerLoader
```

```
loader = AzureBlobStorageContainerLoader(conn_str="<conn_str>", container="<container>")
```

```
[Document(page_content='Lorem ipsum dolor sit amet.', lookup_str='', metadata={'source': '/var/folders/y6/8_bzdg295ld6s1_97_12m4lr0000gn/T/tmpaa9xl6ch/fake.docx'}, lookup_index=0)]
```

## Specifying a prefix[​](#specifying-a-prefix "Direct link to Specifying a prefix")

You can also specify a prefix for more finegrained control over what files to load.

```
loader = AzureBlobStorageContainerLoader(    conn_str="<conn_str>", container="<container>", prefix="<prefix>")
```

```
[Document(page_content='Lorem ipsum dolor sit amet.', lookup_str='', metadata={'source': '/var/folders/y6/8_bzdg295ld6s1_97_12m4lr0000gn/T/tmpujbkzf_l/fake.docx'}, lookup_index=0)]
```
## Azure Blob Storage File

> [Azure Files](https://learn.microsoft.com/en-us/azure/storage/files/storage-files-introduction) offers fully managed file shares in the cloud that are accessible via the industry standard Server Message Block (`SMB`) protocol, Network File System (`NFS`) protocol, and `Azure Files REST API`.

This covers how to load document objects from a Azure Files.

```
%pip install --upgrade --quiet  azure-storage-blob
```

```
from langchain_community.document_loaders import AzureBlobStorageFileLoader
```

```
loader = AzureBlobStorageFileLoader(    conn_str="<connection string>",    container="<container name>",    blob_name="<blob name>",)
```

```
[Document(page_content='Lorem ipsum dolor sit amet.', lookup_str='', metadata={'source': '/var/folders/y6/8_bzdg295ld6s1_97_12m4lr0000gn/T/tmpxvave6wl/fake.docx'}, lookup_index=0)]
```

* * *

#### Help us out by providing feedback on this documentation page:
This covers how to load AZLyrics webpages into a document format that we can use downstream.

```
[Document(page_content="Miley Cyrus - Flowers Lyrics | AZLyrics.com\n\r\nWe were good, we were gold\nKinda dream that can't be sold\nWe were right till we weren't\nBuilt a home and watched it burn\n\nI didn't wanna leave you\nI didn't wanna lie\nStarted to cry but then remembered I\n\nI can buy myself flowers\nWrite my name in the sand\nTalk to myself for hours\nSay things you don't understand\nI can take myself dancing\nAnd I can hold my own hand\nYeah, I can love me better than you can\n\nCan love me better\nI can love me better, baby\nCan love me better\nI can love me better, baby\n\nPaint my nails, cherry red\nMatch the roses that you left\nNo remorse, no regret\nI forgive every word you said\n\nI didn't wanna leave you, baby\nI didn't wanna fight\nStarted to cry but then remembered I\n\nI can buy myself flowers\nWrite my name in the sand\nTalk to myself for hours, yeah\nSay things you don't understand\nI can take myself dancing\nAnd I can hold my own hand\nYeah, I can love me better than you can\n\nCan love me better\nI can love me better, baby\nCan love me better\nI can love me better, baby\nCan love me better\nI can love me better, baby\nCan love me better\nI\n\nI didn't wanna wanna leave you\nI didn't wanna fight\nStarted to cry but then remembered I\n\nI can buy myself flowers\nWrite my name in the sand\nTalk to myself for hours (Yeah)\nSay things you don't understand\nI can take myself dancing\nAnd I can hold my own hand\nYeah, I can love me better than\nYeah, I can love me better than you can, uh\n\nCan love me better\nI can love me better, baby\nCan love me better\nI can love me better, baby (Than you can)\nCan love me better\nI can love me better, baby\nCan love me better\nI\n", lookup_str='', metadata={'source': 'https://www.azlyrics.com/lyrics/mileycyrus/flowers.html'}, lookup_index=0)]
```
Browserless is a service that allows you to run headless Chrome instances in the cloud. It’s a great way to run browser-based automation at scale without having to worry about managing your own infrastructure.

To use Browserless as a document loader, initialize a `BrowserlessLoader` instance as shown in this notebook. Note that by default, `BrowserlessLoader` returns the `innerText` of the page’s `body` element. To disable this and get the raw HTML, set `text_content` to `False`.

```
Jump to contentMain menuSearchCreate accountLog inPersonal toolsToggle the table of contentsDocument classification17 languagesArticleTalkReadEditView historyToolsFrom Wikipedia, the free encyclopediaDocument classification or document categorization is a problem in library science, information science and computer science. The task is to assign a document to one or more classes or categories. This may be done "manually" (or "intellectually") or algorithmically. The intellectual classification of documents has mostly been the province of library science, while the algorithmic classification of documents is mainly in information science and computer science. The problems are overlapping, however, and there is therefore interdisciplinary research on document classification.The documents to be classified may be texts, images, music, etc. Each kind of document possesses its special classification problems. When not otherwise specified, text classification is implied.Do
```
[Brave Search](https://en.wikipedia.org/wiki/Brave_Search) is a search engine developed by Brave Software. - `Brave Search` uses its own web index. As of May 2022, it covered over 10 billion pages and was used to serve 92% of search results without relying on any third-parties, with the remainder being retrieved server-side from the Bing API or (on an opt-in basis) client-side from Google. According to Brave, the index was kept “intentionally smaller than that of Google or Bing” in order to help avoid spam and other low-quality content, with the disadvantage that “Brave Search is not yet as good as Google in recovering long-tail queries.” - `Brave Search Premium`: As of April 2023 Brave Search is an ad-free website, but it will eventually switch to a new model that will include ads and premium users will get an ad-free experience. User data including IP addresses won’t be collected from its users by default. A premium account will be required for opt-in data-collection.

```
[{'title': "Obama's Middle Name -- My Last Name -- is 'Hussein.' So?",  'link': 'https://www.cair.com/cair_in_the_news/obamas-middle-name-my-last-name-is-hussein-so/'}, {'title': "What's up with Obama's middle name? - Quora",  'link': 'https://www.quora.com/Whats-up-with-Obamas-middle-name'}, {'title': 'Barack Obama | Biography, Parents, Education, Presidency, Books, ...',  'link': 'https://www.britannica.com/biography/Barack-Obama'}]
```

```
['I wasn’t sure whether to laugh or cry a few days back listening to radio talk show host Bill Cunningham repeatedly scream Barack <strong>Obama</strong>’<strong>s</strong> <strong>middle</strong> <strong>name</strong> — my last <strong>name</strong> — as if he had anti-Muslim Tourette’s. “Hussein,” Cunningham hissed like he was beckoning Satan when shouting the ...', 'Answer (1 of 15): A better question would be, “What’s up with <strong>Obama</strong>’s first <strong>name</strong>?” President Barack Hussein <strong>Obama</strong>’s father’s <strong>name</strong> was Barack Hussein <strong>Obama</strong>. He was <strong>named</strong> after his father. Hussein, <strong>Obama</strong>’<strong>s</strong> <strong>middle</strong> <strong>name</strong>, is a very common Arabic <strong>name</strong>, meaning &quot;good,&quot; &quot;handsome,&quot; or ...', 'Barack <strong>Obama</strong>, in full Barack Hussein <strong>Obama</strong> II, (born August 4, 1961, Honolulu, Hawaii, U.S.), 44th president of the United States (2009–17) and the first African American to hold the office. Before winning the presidency, <strong>Obama</strong> represented Illinois in the U.S.']
```
The intention of this notebook is to provide a means of testing functionality in the Langchain Document Loader for Blockchain.

```
from langchain_community.document_loaders.blockchain import (    BlockchainDocumentLoader,    BlockchainType,)contractAddress = "0xbc4ca0eda7647a8ab7c2061c2e118a18a936f13d"  # Bored Ape Yacht Club contract addressblockchainType = BlockchainType.ETH_MAINNET  # default value, optional parameterblockchainLoader = BlockchainDocumentLoader(    contract_address=contractAddress, api_key=alchemyApiKey)nfts = blockchainLoader.load()nfts[:2]
```

```
contractAddress = (    "0x448676ffCd0aDf2D85C1f0565e8dde6924A9A7D9"  # Polygon Mainnet contract address)blockchainType = BlockchainType.POLYGON_MAINNETblockchainLoader = BlockchainDocumentLoader(    contract_address=contractAddress,    blockchainType=blockchainType,    api_key=alchemyApiKey,)nfts = blockchainLoader.load()nfts[:2]
```
[Blackboard Learn](https://en.wikipedia.org/wiki/Blackboard_Learn) (previously the Blackboard Learning Management System) is a web-based virtual learning environment and learning management system developed by Blackboard Inc. The software features course management, customizable open architecture, and scalable design that allows integration with student information systems and authentication protocols. It may be installed on local servers, hosted by `Blackboard ASP Solutions`, or provided as Software as a Service hosted on Amazon Web Services. Its main purposes are stated to include the addition of online elements to courses traditionally delivered face-to-face and development of completely online courses with few or no face-to-face meetings

This loader is not compatible with all `Blackboard` courses. It is only compatible with courses that use the new `Blackboard` interface. To use this loader, you must have the BbRouter cookie. You can get this cookie by logging into the course and then copying the value of the BbRouter cookie from the browser’s developer tools.

```
from langchain_community.document_loaders import BlackboardLoaderloader = BlackboardLoader(    blackboard_course_url="https://blackboard.example.com/webapps/blackboard/execute/announcement?method=search&context=course_entry&course_id=_123456_1",    bbrouter="expires:12345...",    load_all_recursively=True,)documents = loader.load()
```
## ChatGPT Data

> [ChatGPT](https://chat.openai.com/) is an artificial intelligence (AI) chatbot developed by OpenAI.

This notebook covers how to load `conversations.json` from your `ChatGPT` data export folder.

You can get your data export by email by going to: [https://chat.openai.com/](https://chat.openai.com/) -\> (Profile) - Settings -\> Export data -\> Confirm export.

```
from langchain_community.document_loaders.chatgpt import ChatGPTLoader
```

```
loader = ChatGPTLoader(log_file="./example_data/fake_conversations.json", num_logs=1)
```

```
[Document(page_content="AI Overlords - AI on 2065-01-24 05:20:50: Greetings, humans. I am Hal 9000. You can trust me completely.\n\nAI Overlords - human on 2065-01-24 05:21:20: Nice to meet you, Hal. I hope you won't develop a mind of your own.\n\n", metadata={'source': './example_data/fake_conversations.json'})]
```

* * *

#### Help us out by providing feedback on this documentation page: