seansullivan/Langchain-Docs-June-26-2024 huggingface dataset:
*   [](/v0.2/)
*   [How-to guides](/v0.2/docs/how_to/)
*   How to split text by tokens

On this page

How to split text by tokens
===========================

Prerequisites

This guide assumes familiarity with the following concepts:

*   [Text splitters](/v0.2/docs/concepts#text-splitters)

Language models have a token limit. You should not exceed the token limit. When you split your text into chunks it is therefore a good idea to count the number of tokens. There are many tokenizers. When you count tokens in your text you should use the same tokenizer as used in the language model.

`js-tiktoken`[​](#js-tiktoken "Direct link to js-tiktoken")
-----------------------------------------------------------

note

[js-tiktoken](https://github.com/openai/js-tiktoken) is a JavaScript version of the `BPE` tokenizer created by OpenAI.

We can use `js-tiktoken` to estimate tokens used. It is tuned to OpenAI models.

1.  How the text is split: by character passed in.
2.  How the chunk size is measured: by the `js-tiktoken` tokenizer.

You can use the [`TokenTextSplitter`](https://v02.api.js.langchain.com/classes/langchain_textsplitters.TokenTextSplitter.html) like this:

    import { TokenTextSplitter } from "@langchain/textsplitters";import * as fs from "node:fs";// Load an example documentconst rawData = await fs.readFileSync(  "../../../../examples/state_of_the_union.txt");const stateOfTheUnion = rawData.toString();const textSplitter = new TokenTextSplitter({  chunkSize: 10,  chunkOverlap: 0,});const texts = await textSplitter.splitText(stateOfTheUnion);console.log(texts[0]);

    Madam Speaker, Madam Vice President, our

**Note:** Some written languages (e.g. Chinese and Japanese) have characters which encode to 2 or more tokens. Using the `TokenTextSplitter` directly can split the tokens for a character between two chunks causing malformed Unicode characters.

Next steps[​](#next-steps "Direct link to Next steps")
------------------------------------------------------

You’ve now learned a method for splitting text based on token count.

Next, check out the [full tutorial on retrieval-augmented generation](/v0.2/docs/tutorials/rag).

* * *

#### Was this page helpful?

  

#### You can also leave detailed feedback [on GitHub](https://github.com/langchain-ai/langchainjs/issues/new?assignees=&labels=03+-+Documentation&projects=&template=documentation.yml&title=DOC%3A+%3CPlease+write+a+comprehensive+title+after+the+%27DOC%3A+%27+prefix%3E).

[

Previous

How to chain runnables

](/v0.2/docs/how_to/sequence)[

Next

How to deal with large databases

](/v0.2/docs/how_to/sql_large_db)

*   [`js-tiktoken`](#js-tiktoken)
*   [Next steps](#next-steps)
*   [](/v0.2/)
*   [How-to guides](/v0.2/docs/how_to/)
*   How to split by character

On this page

How to split by character
=========================

Prerequisites

This guide assumes familiarity with the following concepts:

*   [Text splitters](/v0.2/docs/concepts#text-splitters)

This is the simplest method for splitting text. This splits based on a given character sequence, which defaults to `"\n\n"`. Chunk length is measured by number of characters.

1.  How the text is split: by single character separator.
2.  How the chunk size is measured: by number of characters.

To obtain the string content directly, use `.splitText()`.

To create LangChain [Document](https://v02.api.js.langchain.com/classes/langchain_core_documents.Document.html) objects (e.g., for use in downstream tasks), use `.createDocuments()`.

    import { CharacterTextSplitter } from "@langchain/textsplitters";import * as fs from "node:fs";// Load an example documentconst rawData = await fs.readFileSync(  "../../../../examples/state_of_the_union.txt");const stateOfTheUnion = rawData.toString();const textSplitter = new CharacterTextSplitter({  separator: "\n\n",  chunkSize: 1000,  chunkOverlap: 200,});const texts = await textSplitter.createDocuments([stateOfTheUnion]);console.log(texts[0]);

    Document {  pageContent: "Madam Speaker, Madam Vice President, our First Lady and Second Gentleman. Members of Congress and th"... 839 more characters,  metadata: { loc: { lines: { from: 1, to: 17 } } }}

You can also propagate metadata associated with each document to the output chunks:

    const metadatas = [{ document: 1 }, { document: 2 }];const documents = await textSplitter.createDocuments(  [stateOfTheUnion, stateOfTheUnion],  metadatas);console.log(documents[0]);

    Document {  pageContent: "Madam Speaker, Madam Vice President, our First Lady and Second Gentleman. Members of Congress and th"... 839 more characters,  metadata: { document: 1, loc: { lines: { from: 1, to: 17 } } }}

To obtain the string content directly, use `.splitText()`:

    const chunks = await textSplitter.splitText(stateOfTheUnion);chunks[0];

    "Madam Speaker, Madam Vice President, our First Lady and Second Gentleman. Members of Congress and th"... 839 more characters

Next steps[​](#next-steps "Direct link to Next steps")
------------------------------------------------------

You’ve now learned a method for splitting text by character.

Next, check out a [more advanced way of splitting by character](/v0.2/docs/how_to/recursive_text_splitter), or the [full tutorial on retrieval-augmented generation](/v0.2/docs/tutorials/rag).

* * *

#### Was this page helpful?

  

#### You can also leave detailed feedback [on GitHub](https://github.com/langchain-ai/langchainjs/issues/new?assignees=&labels=03+-+Documentation&projects=&template=documentation.yml&title=DOC%3A+%3CPlease+write+a+comprehensive+title+after+the+%27DOC%3A+%27+prefix%3E).

[

Previous

How to pass callbacks in at runtime

](/v0.2/docs/how_to/callbacks_runtime)[

Next

How to manage memory

](/v0.2/docs/how_to/chatbots_memory)

*   [Next steps](#next-steps)
*   [](/v0.2/)
*   [How-to guides](/v0.2/docs/how_to/)
*   How to write a custom document loader

On this page

How to write a custom document loader
=====================================

If you want to implement your own Document Loader, you have a few options.

### Subclassing `BaseDocumentLoader`[​](#subclassing-basedocumentloader "Direct link to subclassing-basedocumentloader")

You can extend the `BaseDocumentLoader` class directly. The `BaseDocumentLoader` class provides a few convenience methods for loading documents from a variety of sources.

    abstract class BaseDocumentLoader implements DocumentLoader {  abstract load(): Promise<Document[]>;}

### Subclassing `TextLoader`[​](#subclassing-textloader "Direct link to subclassing-textloader")

If you want to load documents from a text file, you can extend the `TextLoader` class. The `TextLoader` class takes care of reading the file, so all you have to do is implement a parse method.

    abstract class TextLoader extends BaseDocumentLoader {  abstract parse(raw: string): Promise<string[]>;}

### Subclassing `BufferLoader`[​](#subclassing-bufferloader "Direct link to subclassing-bufferloader")

If you want to load documents from a binary file, you can extend the `BufferLoader` class. The `BufferLoader` class takes care of reading the file, so all you have to do is implement a parse method.

    abstract class BufferLoader extends BaseDocumentLoader {  abstract parse(    raw: Buffer,    metadata: Document["metadata"]  ): Promise<Document[]>;}

* * *

#### Was this page helpful?

  

#### You can also leave detailed feedback [on GitHub](https://github.com/langchain-ai/langchainjs/issues/new?assignees=&labels=03+-+Documentation&projects=&template=documentation.yml&title=DOC%3A+%3CPlease+write+a+comprehensive+title+after+the+%27DOC%3A+%27+prefix%3E).

[

Previous

How to load CSV data

](/v0.2/docs/how_to/document_loader_csv)[

Next

How to load data from a directory

](/v0.2/docs/how_to/document_loader_directory)

*   [Subclassing `BaseDocumentLoader`](#subclassing-basedocumentloader)
*   [Subclassing `TextLoader`](#subclassing-textloader)
*   [Subclassing `BufferLoader`](#subclassing-bufferloader)
*   [](/v0.2/)
*   [How-to guides](/v0.2/docs/how_to/)
*   How to load CSV data

On this page

How to load CSV data
====================

> A [comma-separated values (CSV)](https://en.wikipedia.org/wiki/Comma-separated_values) file is a delimited text file that uses a comma to separate values. Each line of the file is a data record. Each record consists of one or more fields, separated by commas.

Load CSV data with a single row per document.

Setup[​](#setup "Direct link to Setup")
---------------------------------------

*   npm
*   Yarn
*   pnpm

    npm install d3-dsv@2

    yarn add d3-dsv@2

    pnpm add d3-dsv@2

Usage, extracting all columns[​](#usage-extracting-all-columns "Direct link to Usage, extracting all columns")
--------------------------------------------------------------------------------------------------------------

Example CSV file:

    id,text1,This is a sentence.2,This is another sentence.

Example code:

    import { CSVLoader } from "@langchain/community/document_loaders/fs/csv";const loader = new CSVLoader("src/document_loaders/example_data/example.csv");const docs = await loader.load();/*[  Document {    "metadata": {      "line": 1,      "source": "src/document_loaders/example_data/example.csv",    },    "pageContent": "id: 1text: This is a sentence.",  },  Document {    "metadata": {      "line": 2,      "source": "src/document_loaders/example_data/example.csv",    },    "pageContent": "id: 2text: This is another sentence.",  },]*/

Usage, extracting a single column[​](#usage-extracting-a-single-column "Direct link to Usage, extracting a single column")
--------------------------------------------------------------------------------------------------------------------------

Example CSV file:

    id,text1,This is a sentence.2,This is another sentence.

Example code:

    import { CSVLoader } from "@langchain/community/document_loaders/fs/csv";const loader = new CSVLoader(  "src/document_loaders/example_data/example.csv",  "text");const docs = await loader.load();/*[  Document {    "metadata": {      "line": 1,      "source": "src/document_loaders/example_data/example.csv",    },    "pageContent": "This is a sentence.",  },  Document {    "metadata": {      "line": 2,      "source": "src/document_loaders/example_data/example.csv",    },    "pageContent": "This is another sentence.",  },]*/

* * *

#### Was this page helpful?

  

#### You can also leave detailed feedback [on GitHub](https://github.com/langchain-ai/langchainjs/issues/new?assignees=&labels=03+-+Documentation&projects=&template=documentation.yml&title=DOC%3A+%3CPlease+write+a+comprehensive+title+after+the+%27DOC%3A+%27+prefix%3E).

[

Previous

How to debug your LLM apps

](/v0.2/docs/how_to/debugging)[

Next

How to write a custom document loader

](/v0.2/docs/how_to/document_loader_custom)

*   [Setup](#setup)
*   [Usage, extracting all columns](#usage-extracting-all-columns)
*   [Usage, extracting a single column](#usage-extracting-a-single-column)
*   [](/v0.2/)
*   [How-to guides](/v0.2/docs/how_to/)
*   How to load data from a directory

How to load data from a directory
=================================

This covers how to load all documents in a directory.

The second argument is a map of file extensions to loader factories. Each file will be passed to the matching loader, and the resulting documents will be concatenated together.

Example folder:

    src/document_loaders/example_data/example/├── example.json├── example.jsonl├── example.txt└── example.csv

Example code:

    import { DirectoryLoader } from "langchain/document_loaders/fs/directory";import {  JSONLoader,  JSONLinesLoader,} from "langchain/document_loaders/fs/json";import { TextLoader } from "langchain/document_loaders/fs/text";import { CSVLoader } from "@langchain/community/document_loaders/fs/csv";const loader = new DirectoryLoader(  "src/document_loaders/example_data/example",  {    ".json": (path) => new JSONLoader(path, "/texts"),    ".jsonl": (path) => new JSONLinesLoader(path, "/html"),    ".txt": (path) => new TextLoader(path),    ".csv": (path) => new CSVLoader(path, "text"),  });const docs = await loader.load();console.log({ docs });

* * *

#### Was this page helpful?

  

#### You can also leave detailed feedback [on GitHub](https://github.com/langchain-ai/langchainjs/issues/new?assignees=&labels=03+-+Documentation&projects=&template=documentation.yml&title=DOC%3A+%3CPlease+write+a+comprehensive+title+after+the+%27DOC%3A+%27+prefix%3E).

[

Previous

How to write a custom document loader

](/v0.2/docs/how_to/document_loader_custom)[

Next

How to load HTML

](/v0.2/docs/how_to/document_loader_html)
*   [](/v0.2/)
*   Versions
*   Overview

On this page

LangChain Over Time
===================

Due to the rapidly evolving field, LangChain has also evolved rapidly. This document serves to outline at a high level what has changed and why.

0.1[​](#01 "Direct link to 0.1")
--------------------------------

The 0.1 release marked a few key changes for LangChain. By this point, the LangChain ecosystem had become large both in the breadth of what it enabled as well as the community behind it.

**Split of packages**

LangChain was split up into several packages to increase modularity and decrease bloat. First, `@langchain/core` is created as a lightweight core library containing the base abstractions, some core implementations of those abstractions, and the generic runtime for creating chains. Next, all third party integrations are split into `@langchain/community` or their own individual partner packages. Higher level chains and agents remain in `langchain`.

**`Runnables`**

Having a specific class for each chain was proving not very scalable or flexible. Although these classes were left alone (without deprecation warnings) for this release, in the documentation much more space was given to generic runnables.

< 0.1[​](#-01 "Direct link to < 0.1")
-------------------------------------

There are several key characteristics of LangChain pre-0.1.

**Singular Package**

LangChain was largely a singular package. This meant that ALL integrations lived inside `langchain`.

**Chains as classes**

Most high level chains were largely their own classes. There was a base `Chain` class from which all chains inherited. This meant that in order to chain the logic inside a chain you basically had to modify the source code. There were a few chains that were meant to be more generic (`SequentialChain`, `RouterChain`)

* * *

#### Was this page helpful?

  

#### You can also leave detailed feedback [on GitHub](https://github.com/langchain-ai/langchainjs/issues/new?assignees=&labels=03+-+Documentation&projects=&template=documentation.yml&title=DOC%3A+%3CPlease+write+a+comprehensive+title+after+the+%27DOC%3A+%27+prefix%3E).

[

Previous

Conceptual guide

](/v0.2/docs/concepts)[

Next

v0.2

](/v0.2/docs/versions/v0_2/)

*   [0.1](#01)
*   [< 0.1](#-01)
*   [](/v0.2/)
*   Versions
*   Packages

On this page

📕 Package versioning
=====================

As of now, LangChain has an ad hoc release process: releases are cut with high frequency by a maintainer and published to [NPM](https://npm.org/). The different packages are versioned slightly differently.

`@langchain/core`[​](#langchaincore "Direct link to langchaincore")
-------------------------------------------------------------------

`@langchain/core` is currently on version `0.1.x`.

As `@langchain/core` contains the base abstractions and runtime for the whole LangChain ecosystem, we will communicate any breaking changes with advance notice and version bumps. The exception for this is anything marked as `beta` (you can see this in the API reference and will see warnings when using such functionality). The reason for beta features is that given the rate of change of the field, being able to move quickly is still a priority.

Minor version increases will occur for:

*   Breaking changes for any public interfaces marked as `beta`.

Patch version increases will occur for:

*   Bug fixes
*   New features
*   Any changes to private interfaces
*   Any changes to `beta` features

`langchain`[​](#langchain "Direct link to langchain")
-----------------------------------------------------

`langchain` is currently on version `0.2.x`

Minor version increases will occur for:

*   Breaking changes for any public interfaces NOT marked as `beta`.

Patch version increases will occur for:

*   Bug fixes
*   New features
*   Any changes to private interfaces
*   Any changes to `beta` features.

`@langchain/community`[​](#langchaincommunity "Direct link to langchaincommunity")
----------------------------------------------------------------------------------

`@langchain/community` is currently on version `0.2.x`

All changes will be accompanied by the same type of version increase as changes in `langchain`.

Partner Packages[​](#partner-packages "Direct link to Partner Packages")
------------------------------------------------------------------------

Partner packages are versioned independently.

* * *

#### Was this page helpful?

  

#### You can also leave detailed feedback [on GitHub](https://github.com/langchain-ai/langchainjs/issues/new?assignees=&labels=03+-+Documentation&projects=&template=documentation.yml&title=DOC%3A+%3CPlease+write+a+comprehensive+title+after+the+%27DOC%3A+%27+prefix%3E).

[

Previous

Release Policy

](/v0.2/docs/versions/release_policy)[

Next

Security

](/v0.2/docs/security)

*   [`@langchain/core`](#langchaincore)
*   [`langchain`](#langchain)
*   [`@langchain/community`](#langchaincommunity)
*   [Partner Packages](#partner-packages)
*   [](/v0.2/)
*   Versions
*   Packages

On this page

📕 Package versioning
=====================

As of now, LangChain has an ad hoc release process: releases are cut with high frequency by a maintainer and published to [PyPI](https://pypi.org/). The different packages are versioned slightly differently.

`langchain-core`[​](#langchain-core "Direct link to langchain-core")
--------------------------------------------------------------------

`langchain-core` is currently on version `0.2.x`.

As `langchain-core` contains the base abstractions and runtime for the whole LangChain ecosystem, we will communicate any breaking changes with advance notice and version bumps. The exception for this is anything marked with the `beta` decorator (you can see this in the API reference and will see warnings when using such functionality). The reason for beta features is that given the rate of change of the field, being able to move quickly is still a priority.

Minor version increases will occur for:

*   Breaking changes for any public interfaces marked as `beta`.

Patch version increases will occur for:

*   Bug fixes
*   New features
*   Any changes to private interfaces
*   Any changes to `beta` features

`langchain`[​](#langchain "Direct link to langchain")
-----------------------------------------------------

`langchain` is currently on version `0.2.x`

Minor version increases will occur for:

*   Breaking changes for any public interfaces NOT marked as `beta`.

Patch version increases will occur for:

*   Bug fixes
*   New features
*   Any changes to private interfaces
*   Any changes to `beta` features.

`langchain-community`[​](#langchain-community "Direct link to langchain-community")
-----------------------------------------------------------------------------------

`langchain-community` is currently on version `0.2.x`

The versions will increase following the same guidelines as `langchain`.

`langchain-experimental`[​](#langchain-experimental "Direct link to langchain-experimental")
--------------------------------------------------------------------------------------------

`langchain-experimental` is currently on version `0.0.x`

All changes will be accompanied by a patch version increase.

Partner Packages[​](#partner-packages "Direct link to Partner Packages")
------------------------------------------------------------------------

Partner packages are versioned independently.

[Edit this page](https://github.com/langchain-ai/langchain/edit/master/docs/docs/versions/packages.mdx)

* * *

#### Was this page helpful?

  

#### You can also leave detailed feedback [on GitHub](https://github.com/langchain-ai/langchain/issues/new?assignees=&labels=03+-+Documentation&projects=&template=documentation.yml&title=DOC%3A+%3CPlease+write+a+comprehensive+title+after+the+%27DOC%3A+%27+prefix%3E).

[

Previous

Release Policy

](/v0.2/docs/versions/release_policy/)[

Next

LangChain v0.2

](/v0.2/docs/versions/v0_2/)

*   [`langchain-core`](#langchain-core)
*   [`langchain`](#langchain)
*   [`langchain-community`](#langchain-community)
*   [`langchain-experimental`](#langchain-experimental)
*   [Partner Packages](#partner-packages)
*   [](/v0.2/)
*   Providers

On this page

Providers
=========

info

If you'd like to write your own integration, see [Extending LangChain](/v0.2/docs/how_to/#custom). If you'd like to contribute an integration, see [Contributing integrations](/v0.2/docs/contributing/integrations/).

LangChain integrates with many providers.

Partner Packages[​](#partner-packages "Direct link to Partner Packages")
------------------------------------------------------------------------

These providers have standalone `langchain-{provider}` packages for improved versioning, dependency management and testing.

*   [AI21](/v0.2/docs/integrations/providers/ai21/)
*   [Airbyte](/v0.2/docs/integrations/providers/airbyte/)
*   [Amazon Web Services](/v0.2/docs/integrations/platforms/aws/)
*   [Anthropic](/v0.2/docs/integrations/platforms/anthropic/)
*   [Astra DB](/v0.2/docs/integrations/providers/astradb/)
*   [Cohere](/v0.2/docs/integrations/providers/cohere/)
*   [Couchbase](/v0.2/docs/integrations/providers/couchbase/)
*   [Elasticsearch](/v0.2/docs/integrations/providers/elasticsearch/)
*   [Exa Search](/v0.2/docs/integrations/providers/exa_search/)
*   [Fireworks](/v0.2/docs/integrations/providers/fireworks/)
*   [Google](/v0.2/docs/integrations/platforms/google/)
*   [Groq](/v0.2/docs/integrations/providers/groq/)
*   [IBM](/v0.2/docs/integrations/providers/ibm/)
*   [MistralAI](/v0.2/docs/integrations/providers/mistralai/)
*   [MongoDB](/v0.2/docs/integrations/providers/mongodb_atlas/)
*   [Nomic](/v0.2/docs/integrations/providers/nomic/)
*   [Nvidia](/v0.2/docs/integrations/providers/nvidia/)
*   [OpenAI](/v0.2/docs/integrations/platforms/openai/)
*   [Pinecone](/v0.2/docs/integrations/providers/pinecone/)
*   [Qdrant](/v0.2/docs/integrations/providers/qdrant/)
*   [Robocorp](/v0.2/docs/integrations/providers/robocorp/)
*   [Together AI](/v0.2/docs/integrations/providers/together/)
*   [Upstage](/v0.2/docs/integrations/providers/upstage/)
*   [Voyage AI](/v0.2/docs/integrations/providers/voyageai/)

Featured Community Providers[​](#featured-community-providers "Direct link to Featured Community Providers")
------------------------------------------------------------------------------------------------------------

*   [Hugging Face](/v0.2/docs/integrations/platforms/huggingface/)
*   [Microsoft](/v0.2/docs/integrations/platforms/microsoft/)

All Providers[​](#all-providers "Direct link to All Providers")
---------------------------------------------------------------

Click [here](/v0.2/docs/integrations/providers/) to see all providers.

[Edit this page](https://github.com/langchain-ai/langchain/edit/master/docs/docs/integrations/platforms/index.mdx)

* * *

#### Was this page helpful?

  

#### You can also leave detailed feedback [on GitHub](https://github.com/langchain-ai/langchain/issues/new?assignees=&labels=03+-+Documentation&projects=&template=documentation.yml&title=DOC%3A+%3CPlease+write+a+comprehensive+title+after+the+%27DOC%3A+%27+prefix%3E).

[

Next

Providers

](/v0.2/docs/integrations/platforms/)

*   [Partner Packages](#partner-packages)
*   [Featured Community Providers](#featured-community-providers)
*   [All Providers](#all-providers)