viruPVS/langchain-docs huggingface dataset:
Initialize Couchbase document loader.
Parameters
connection_string (str) – The connection string to the Couchbase cluster.
db_username (str) – The username to connect to the Couchbase cluster.
db_password (str) – The password to connect to the Couchbase cluster.
query (str) – The SQL++ query to execute.
page_content_fields (Optional[List[str]]) – The columns to write into the
page_content field of the document. By default, all columns are
written.
metadata_fields (Optional[List[str]]) – The columns to write into the
metadata field of the document. By default, no columns are written.
lazy_load() → Iterator[Document][source]¶
Load Couchbase data into Document objects lazily.
load() → List[Document][source]¶
Load Couchbase data into Document objects.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
langchain_community.document_loaders.rtf.UnstructuredRTFLoader¶
class langchain_community.document_loaders.rtf.UnstructuredRTFLoader(file_path: str, mode: str = 'single', **unstructured_kwargs: Any)[source]¶
Load RTF files using Unstructured.
You can run the loader in one of two modes: “single” and “elements”.
If you use “single” mode, the document will be returned as a single
langchain Document object. If you use “elements” mode, the unstructured
library will split the document into elements such as Title and NarrativeText.
You can pass in additional unstructured kwargs after mode to apply
different unstructured settings.
Examples
from langchain_community.document_loaders import UnstructuredRTFLoader
loader = UnstructuredRTFLoader(“example.rtf”, mode=”elements”, strategy=”fast”,
)
docs = loader.load()
References
https://unstructured-io.github.io/unstructured/bricks.html#partition-rtf
Initialize with a file path.
Parameters
file_path – The path to the file to load.
mode – The mode to use for partitioning. See unstructured for details.
Defaults to “single”.
**unstructured_kwargs – Additional keyword arguments to pass
to unstructured.
Methods
__init__(file_path[, mode])
Initialize with a file path.
lazy_load()
A lazy loader for Documents.
load()
Load file.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(file_path: str, mode: str = 'single', **unstructured_kwargs: Any)[source]¶
Initialize with a file path.
Parameters
file_path – The path to the file to load.
mode – The mode to use for partitioning. See unstructured for details.
Defaults to “single”.
Defaults to “single”.
**unstructured_kwargs – Additional keyword arguments to pass
to unstructured.
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
load() → List[Document]¶
Load file.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
langchain_community.document_loaders.pdf.PyPDFium2Loader¶
class langchain_community.document_loaders.pdf.PyPDFium2Loader(file_path: str, *, headers: Optional[Dict] = None, extract_images: bool = False)[source]¶
Load PDF using pypdfium2 and chunks at character level.
Initialize with a file path.
Attributes
source
Methods
__init__(file_path, *[, headers, extract_images])
Initialize with a file path.
lazy_load()
Lazy load given path as pages.
load()
Load given path as pages.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(file_path: str, *, headers: Optional[Dict] = None, extract_images: bool = False)[source]¶
Initialize with a file path.
lazy_load() → Iterator[Document][source]¶
Lazy load given path as pages.
load() → List[Document][source]¶
Load given path as pages.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
langchain_community.document_loaders.directory.DirectoryLoader¶
class langchain_community.document_loaders.directory.DirectoryLoader(path: str, glob: str = '**/[!.]*', silent_errors: bool = False, load_hidden: bool = False, loader_cls: ~typing.Union[~typing.Type[~langchain_community.document_loaders.unstructured.UnstructuredFileLoader], ~typing.Type[~langchain_community.document_loaders.text.TextLoader], ~typing.Type[~langchain_community.document_loaders.html_bs.BSHTMLLoader]] = <class 'langchain_community.document_loaders.unstructured.UnstructuredFileLoader'>, loader_kwargs: ~typing.Optional[dict] = None, recursive: bool = False, show_progress: bool = False, use_multithreading: bool = False, max_concurrency: int = 4, *, sample_size: int = 0, randomize_sample: bool = False, sample_seed: ~typing.Optional[int] = None)[source]¶
Load from a directory.
Initialize with a path to directory and how to glob over it.
Parameters
path – Path to directory.
glob – Glob pattern to use to find files. Defaults to “**/[!.]*”
(all files except hidden).
silent_errors – Whether to silently ignore errors. Defaults to False.
load_hidden – Whether to load hidden files. Defaults to False.
loader_cls – Loader class to use for loading files.
Defaults to UnstructuredFileLoader.
loader_kwargs – Keyword arguments to pass to loader_cls. Defaults to None.
recursive – Whether to recursively search for files. Defaults to False.
show_progress – Whether to show a progress bar. Defaults to False.
use_multithreading – Whether to use multithreading. Defaults to False.
max_concurrency – The maximum number of threads to use. Defaults to 4.
sample_size – The maximum number of files you would like to load from the
directory.
sample_size – The maximum number of files you would like to load from the
directory.
randomize_sample – Suffle the files to get a random sample.
sample_seed – set the seed of the random shuffle for reporoducibility.
Methods
__init__(path[, glob, silent_errors, ...])
Initialize with a path to directory and how to glob over it.
lazy_load()
A lazy loader for Documents.
load()
Load documents.
load_and_split([text_splitter])
Load Documents and split into chunks.
load_file(item, path, docs, pbar)
Load a file.
__init__(path: str, glob: str = '**/[!.]*', silent_errors: bool = False, load_hidden: bool = False, loader_cls: ~typing.Union[~typing.Type[~langchain_community.document_loaders.unstructured.UnstructuredFileLoader], ~typing.Type[~langchain_community.document_loaders.text.TextLoader], ~typing.Type[~langchain_community.document_loaders.html_bs.BSHTMLLoader]] = <class 'langchain_community.document_loaders.unstructured.UnstructuredFileLoader'>, loader_kwargs: ~typing.Optional[dict] = None, recursive: bool = False, show_progress: bool = False, use_multithreading: bool = False, max_concurrency: int = 4, *, sample_size: int = 0, randomize_sample: bool = False, sample_seed: ~typing.Optional[int] = None)[source]¶
Initialize with a path to directory and how to glob over it.
Parameters
path – Path to directory.
glob – Glob pattern to use to find files. Defaults to “**/[!.]*”
(all files except hidden).
silent_errors – Whether to silently ignore errors. Defaults to False.
load_hidden – Whether to load hidden files. Defaults to False.
load_hidden – Whether to load hidden files. Defaults to False.
loader_cls – Loader class to use for loading files.
Defaults to UnstructuredFileLoader.
loader_kwargs – Keyword arguments to pass to loader_cls. Defaults to None.
recursive – Whether to recursively search for files. Defaults to False.
show_progress – Whether to show a progress bar. Defaults to False.
use_multithreading – Whether to use multithreading. Defaults to False.
max_concurrency – The maximum number of threads to use. Defaults to 4.
sample_size – The maximum number of files you would like to load from the
directory.
randomize_sample – Suffle the files to get a random sample.
sample_seed – set the seed of the random shuffle for reporoducibility.
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
load() → List[Document][source]¶
Load documents.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
load_file(item: Path, path: Path, docs: List[Document], pbar: Optional[Any]) → None[source]¶
Load a file.
Parameters
item – File path.
path – Directory path.
docs – List of documents to append to.
pbar – Progress bar. Defaults to None.
Examples using DirectoryLoader¶
StarRocks
langchain_community.document_loaders.rspace.RSpaceLoader¶
class langchain_community.document_loaders.rspace.RSpaceLoader(global_id: str, api_key: Optional[str] = None, url: Optional[str] = None)[source]¶
Loads  content from RSpace notebooks, folders, documents or PDF Gallery files into
Langchain documents.
Maps  RSpace document <-> Langchain Document in 1-1. PDFs are imported using PyPDF.
Requirements are rspace_client (pip install rspace_client) and PyPDF if importingPDF docs (pip install pypdf).
api_key: RSpace API key - can also be supplied as environment variable
‘RSPACE_API_KEY’
url: str
The URL of your RSpace instance - can also be supplied as environment
variable ‘RSPACE_URL’
global_id: str
The global ID of the resource to load,
e.g. ‘SD12344’ (a single document); ‘GL12345’(A PDF file in the gallery);
‘NB4567’ (a notebook); ‘FL12244’ (a folder)
Methods
__init__(global_id[, api_key, url])
api_key: RSpace API key - can also be supplied as environment variable 'RSPACE_API_KEY' url: str The URL of your RSpace instance - can also be supplied as environment variable 'RSPACE_URL' global_id: str  The global ID of the resource to load, e.g. 'SD12344' (a single document); 'GL12345'(A PDF file in the gallery); 'NB4567' (a notebook); 'FL12244' (a folder).
lazy_load()
A lazy loader for Documents.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
validate_environment(values)
Load Documents and split into chunks.
validate_environment(values)
Validate that API key and URL exists in environment.
__init__(global_id: str, api_key: Optional[str] = None, url: Optional[str] = None)[source]¶
api_key: RSpace API key - can also be supplied as environment variable
‘RSPACE_API_KEY’
url: str
The URL of your RSpace instance - can also be supplied as environment
variable ‘RSPACE_URL’
global_id: str
The global ID of the resource to load,
e.g. ‘SD12344’ (a single document); ‘GL12345’(A PDF file in the gallery);
‘NB4567’ (a notebook); ‘FL12244’ (a folder)
lazy_load() → Iterator[Document][source]¶
A lazy loader for Documents.
load() → List[Document][source]¶
Load data into Document objects.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
classmethod validate_environment(values: Dict) → Dict[source]¶
Validate that API key and URL exists in environment.
langchain_community.document_loaders.obsidian.ObsidianLoader¶
class langchain_community.document_loaders.obsidian.ObsidianLoader(path: str, encoding: str = 'UTF-8', collect_metadata: bool = True)[source]¶
Load Obsidian files from directory.
Initialize with a path.
Parameters
path – Path to the directory containing the Obsidian files.
encoding – Charset encoding, defaults to “UTF-8”
collect_metadata – Whether to collect metadata from the front matter.
Defaults to True.
Attributes
DATAVIEW_INLINE_BRACKET_REGEX
DATAVIEW_INLINE_PAREN_REGEX
DATAVIEW_LINE_REGEX
FRONT_MATTER_REGEX
TAG_REGEX
TEMPLATE_VARIABLE_REGEX
Methods
__init__(path[, encoding, collect_metadata])
Initialize with a path.
lazy_load()
A lazy loader for Documents.
load()
Load documents.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(path: str, encoding: str = 'UTF-8', collect_metadata: bool = True)[source]¶
Initialize with a path.
Parameters
path – Path to the directory containing the Obsidian files.
encoding – Charset encoding, defaults to “UTF-8”
collect_metadata – Whether to collect metadata from the front matter.
Defaults to True.
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
load() → List[Document][source]¶
Load documents.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Examples using ObsidianLoader¶
Obsidian
langchain_community.document_loaders.excel.UnstructuredExcelLoader¶
class langchain_community.document_loaders.excel.UnstructuredExcelLoader(file_path: str, mode: str = 'single', **unstructured_kwargs: Any)[source]¶
Load Microsoft Excel files using Unstructured.
Like other
Unstructured loaders, UnstructuredExcelLoader can be used in both
“single” and “elements” mode. If you use the loader in “elements”
mode, each sheet in the Excel file will be a an Unstructured Table
element. If you use the loader in “elements” mode, an
HTML representation of the table will be available in the
“text_as_html” key in the document metadata.
Examples
from langchain_community.document_loaders.excel import UnstructuredExcelLoader
loader = UnstructuredExcelLoader(“stanley-cups.xlsd”, mode=”elements”)
docs = loader.load()
Parameters
file_path – The path to the Microsoft Excel file.
mode – The mode to use when partitioning the file. See unstructured docs
for more info. Optional. Defaults to “single”.
**unstructured_kwargs – Keyword arguments to pass to unstructured.
Methods
__init__(file_path[, mode])
param file_path
The path to the Microsoft Excel file.
lazy_load()
A lazy loader for Documents.
load()
Load file.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(file_path: str, mode: str = 'single', **unstructured_kwargs: Any)[source]¶
Parameters
file_path – The path to the Microsoft Excel file.
mode – The mode to use when partitioning the file. See unstructured docs
for more info. Optional. Defaults to “single”.
**unstructured_kwargs – Keyword arguments to pass to unstructured.
**unstructured_kwargs – Keyword arguments to pass to unstructured.
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
load() → List[Document]¶
Load file.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Examples using UnstructuredExcelLoader¶
Microsoft Excel
langchain_community.document_loaders.generic.GenericLoader¶
class langchain_community.document_loaders.generic.GenericLoader(blob_loader: BlobLoader, blob_parser: BaseBlobParser)[source]¶
Generic Document Loader.
A generic document loader that allows combining an arbitrary blob loader with
a blob parser.
Examples
Parse a specific PDF file:
 from langchain_community.document_loaders import GenericLoader
 from langchain_community.document_loaders.parsers.pdf import PyPDFParser
 # Recursively load all text files in a directory.
 loader = GenericLoader.from_filesystem(
     "my_lovely_pdf.pdf",
     parser=PyPDFParser()
 )
.. code-block:: python
     from langchain_community.document_loaders import GenericLoader
     from langchain_community.document_loaders.blob_loaders import FileSystemBlobLoader
     loader = GenericLoader.from_filesystem(
         path="path/to/directory",
         glob="**/[!.]*",
         suffixes=[".pdf"],
         show_progress=True,
     )
     docs = loader.lazy_load()
     next(docs)
Example instantiations to change which files are loaded:
# Recursively load all text files in a directory.
loader = GenericLoader.from_filesystem("/path/to/dir", glob="**/*.txt")
# Recursively load all non-hidden files in a directory.
loader = GenericLoader.from_filesystem("/path/to/dir", glob="**/[!.]*")
# Load all files in a directory without recursion.
loader = GenericLoader.from_filesystem("/path/to/dir", glob="*")
Example instantiations to change which parser is used:
from langchain_community.document_loaders.parsers.pdf import PyPDFParser
# Recursively load all text files in a directory.
loader = GenericLoader.from_filesystem(
    "/path/to/dir",
    glob="**/*.pdf",
    parser=PyPDFParser()
)
glob="**/*.pdf",
    parser=PyPDFParser()
)
A generic document loader.
Parameters
blob_loader – A blob loader which knows how to yield blobs
blob_parser – A blob parser which knows how to parse blobs into documents
Methods
__init__(blob_loader, blob_parser)
A generic document loader.
from_filesystem(path, *[, glob, exclude, ...])
Create a generic document loader using a filesystem blob loader.
get_parser(**kwargs)
Override this method to associate a default parser with the class.
lazy_load()
Load documents lazily.
load()
Load all documents.
load_and_split([text_splitter])
Load all documents and split them into sentences.
__init__(blob_loader: BlobLoader, blob_parser: BaseBlobParser) → None[source]¶
A generic document loader.
Parameters
blob_loader – A blob loader which knows how to yield blobs
blob_parser – A blob parser which knows how to parse blobs into documents
classmethod from_filesystem(path: Union[str, Path], *, glob: str = '**/[!.]*', exclude: Sequence[str] = (), suffixes: Optional[Sequence[str]] = None, show_progress: bool = False, parser: Union[Literal['default'], BaseBlobParser] = 'default', parser_kwargs: Optional[dict] = None) → GenericLoader[source]¶
Create a generic document loader using a filesystem blob loader.
Parameters
path – The path to the directory to load documents from OR the path to a
single file to load. If this is a file, glob, exclude, suffixes
will be ignored.
glob – The glob pattern to use to find documents.
suffixes – The suffixes to use to filter documents. If None, all files
matching the glob will be loaded.
matching the glob will be loaded.
exclude – A list of patterns to exclude from the loader.
show_progress – Whether to show a progress bar or not (requires tqdm).
Proxies to the file system loader.
parser – A blob parser which knows how to parse blobs into documents,
will instantiate a default parser if not provided.
The default can be overridden by either passing a parser or
setting the class attribute blob_parser (the latter
should be used with inheritance).
parser_kwargs – Keyword arguments to pass to the parser.
Returns
A generic document loader.
static get_parser(**kwargs: Any) → BaseBlobParser[source]¶
Override this method to associate a default parser with the class.
lazy_load() → Iterator[Document][source]¶
Load documents lazily. Use this when working at a large scale.
load() → List[Document][source]¶
Load all documents.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document][source]¶
Load all documents and split them into sentences.
Examples using GenericLoader¶
Grobid
Loading documents from a YouTube url
Source Code
Set env var OPENAI_API_KEY or load from a .env file
langchain_community.document_loaders.obs_directory.OBSDirectoryLoader¶
class langchain_community.document_loaders.obs_directory.OBSDirectoryLoader(bucket: str, endpoint: str, config: Optional[dict] = None, prefix: str = '')[source]¶
Load from Huawei OBS directory.
Initialize the OBSDirectoryLoader with the specified settings.
Parameters
bucket (str) – The name of the OBS bucket to be used.
endpoint (str) – The endpoint URL of your OBS bucket.
config (dict) – The parameters for connecting to OBS, provided as a dictionary. The dictionary could have the following keys:
- “ak” (str, optional): Your OBS access key (required if get_token_from_ecs is False and bucket policy is not public read).
- “sk” (str, optional): Your OBS secret key (required if get_token_from_ecs is False and bucket policy is not public read).
- “token” (str, optional): Your security token (required if using temporary credentials).
- “get_token_from_ecs” (bool, optional): Whether to retrieve the security token from ECS. Defaults to False if not provided. If set to True, ak, sk, and token will be ignored.
prefix (str, optional) – The prefix to be added to the OBS key. Defaults to “”.
Note
Before using this class, make sure you have registered with OBS and have the necessary credentials. The ak, sk, and endpoint values are mandatory unless get_token_from_ecs is True or the bucket policy is public read. token is required when using temporary credentials.
Example
To create a new OBSDirectoryLoader:
```
config = {
“ak”: “your-access-key”,
“sk”: “your-secret-key”
directory_loader = OBSDirectoryLoader(“your-bucket-name”, “your-end-endpoint”, config, “your-prefix”)
Methods
__init__(bucket, endpoint[, config, prefix])
Initialize the OBSDirectoryLoader with the specified settings.
lazy_load()
A lazy loader for Documents.
load()
Load documents.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(bucket: str, endpoint: str, config: Optional[dict] = None, prefix: str = '')[source]¶
Initialize the OBSDirectoryLoader with the specified settings.
Parameters
bucket (str) – The name of the OBS bucket to be used.
endpoint (str) – The endpoint URL of your OBS bucket.
config (dict) – The parameters for connecting to OBS, provided as a dictionary. The dictionary could have the following keys:
- “ak” (str, optional): Your OBS access key (required if get_token_from_ecs is False and bucket policy is not public read).
- “sk” (str, optional): Your OBS secret key (required if get_token_from_ecs is False and bucket policy is not public read).
- “token” (str, optional): Your security token (required if using temporary credentials).
- “get_token_from_ecs” (bool, optional): Whether to retrieve the security token from ECS. Defaults to False if not provided. If set to True, ak, sk, and token will be ignored.
prefix (str, optional) – The prefix to be added to the OBS key. Defaults to “”.
Note
Before using this class, make sure you have registered with OBS and have the necessary credentials. The ak, sk, and endpoint values are mandatory unless get_token_from_ecs is True or the bucket policy is public read. token is required when using temporary credentials.
Example
To create a new OBSDirectoryLoader:
```
config = {
“ak”: “your-access-key”,
```
config = {
“ak”: “your-access-key”,
“sk”: “your-secret-key”
directory_loader = OBSDirectoryLoader(“your-bucket-name”, “your-end-endpoint”, config, “your-prefix”)
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
load() → List[Document][source]¶
Load documents.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Examples using OBSDirectoryLoader¶
Huawei OBS Directory