viruPVS/langchain-docs huggingface dataset:
langchain_community.document_loaders.csv_loader.CSVLoader¶
class langchain_community.document_loaders.csv_loader.CSVLoader(file_path: str, source_column: Optional[str] = None, metadata_columns: Sequence[str] = (), csv_args: Optional[Dict] = None, encoding: Optional[str] = None, autodetect_encoding: bool = False)[source]¶
Load a CSV file into a list of Documents.
Each document represents one row of the CSV file. Every row is converted into a
key/value pair and outputted to a new line in the document’s page_content.
The source for each document loaded from csv is set to the value of the
file_path argument for all documents by default.
You can override this by setting the source_column argument to the
name of a column in the CSV file.
The source of each document will then be set to the value of the column
with the name specified in source_column.
Output Example:column1: value1
column2: value2
column3: value3
Parameters
file_path – The path to the CSV file.
source_column – The name of the column in the CSV file to use as the source.
Optional. Defaults to None.
metadata_columns – A sequence of column names to use as metadata. Optional.
csv_args – A dictionary of arguments to pass to the csv.DictReader.
Optional. Defaults to None.
encoding – The encoding of the CSV file. Optional. Defaults to None.
autodetect_encoding – Whether to try to autodetect the file encoding.
Methods
__init__(file_path[, source_column, ...])
param file_path
The path to the CSV file.
lazy_load()
A lazy loader for Documents.
load()
Load data into document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(file_path: str, source_column: Optional[str] = None, metadata_columns: Sequence[str] = (), csv_args: Optional[Dict] = None, encoding: Optional[str] = None, autodetect_encoding: bool = False)[source]¶
Parameters
file_path – The path to the CSV file.
source_column – The name of the column in the CSV file to use as the source.
Optional. Defaults to None.
metadata_columns – A sequence of column names to use as metadata. Optional.
csv_args – A dictionary of arguments to pass to the csv.DictReader.
Optional. Defaults to None.
encoding – The encoding of the CSV file. Optional. Defaults to None.
autodetect_encoding – Whether to try to autodetect the file encoding.
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
load() → List[Document][source]¶
Load data into document objects.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Examples using CSVLoader¶
ChatGPT Plugin
CSV
langchain_community.document_loaders.sharepoint.SharePointLoader¶
class langchain_community.document_loaders.sharepoint.SharePointLoader[source]¶
Bases: O365BaseLoader
Load  from SharePoint.
Create a new model by parsing and validating input data from keyword arguments.
Raises ValidationError if the input data cannot be parsed to form a valid model.
param auth_with_token: bool = False¶
Whether to authenticate with a token or not. Defaults to False.
param chunk_size: Union[int, str] = 5242880¶
Number of bytes to retrieve from each api call to the server. int or ‘auto’.
param document_library_id: str [Required]¶
The ID of the SharePoint document library to load data from.
param folder_path: Optional[str] = None¶
The path to the folder to load data from.
param object_ids: Optional[List[str]] = None¶
The IDs of the objects to load data from.
param settings: _O365Settings [Optional]¶
Settings for the Office365 API client.
classmethod construct(_fields_set: Optional[SetStr] = None, **values: Any) → Model¶
Creates a new model setting __dict__ and __fields_set__ from trusted or pre-validated data.
Default values are respected, but no other validation is performed.
Behaves as if Config.extra = ‘allow’ was set since it adds all passed values
copy(*, include: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, exclude: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, update: Optional[DictStrAny] = None, deep: bool = False) → Model¶
Duplicate a model, optionally choose which fields to include, exclude and change.
Parameters
include – fields to include in new model
load() → List[Document][source]¶
Load all documents.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
classmethod parse_file(path: Union[str, Path], *, content_type: unicode = None, encoding: unicode = 'utf8', proto: Protocol = None, allow_pickle: bool = False) → Model¶
classmethod parse_obj(obj: Any) → Model¶
classmethod parse_raw(b: Union[str, bytes], *, content_type: unicode = None, encoding: unicode = 'utf8', proto: Protocol = None, allow_pickle: bool = False) → Model¶
classmethod schema(by_alias: bool = True, ref_template: unicode = '#/definitions/{model}') → DictStrAny¶
classmethod schema_json(*, by_alias: bool = True, ref_template: unicode = '#/definitions/{model}', **dumps_kwargs: Any) → unicode¶
classmethod update_forward_refs(**localns: Any) → None¶
Try to update ForwardRefs on fields based on this Model, globalns and localns.
classmethod validate(value: Any) → Model¶
Examples using SharePointLoader¶
Microsoft SharePoint
langchain_community.document_loaders.telegram.TelegramChatFileLoader¶
class langchain_community.document_loaders.telegram.TelegramChatFileLoader(path: str)[source]¶
Load from Telegram chat dump.
Initialize with a path.
Methods
__init__(path)
Initialize with a path.
lazy_load()
A lazy loader for Documents.
load()
Load documents.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(path: str)[source]¶
Initialize with a path.
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
load() → List[Document][source]¶
Load documents.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Examples using TelegramChatFileLoader¶
Telegram
langchain_community.document_loaders.org_mode.UnstructuredOrgModeLoader¶
class langchain_community.document_loaders.org_mode.UnstructuredOrgModeLoader(file_path: str, mode: str = 'single', **unstructured_kwargs: Any)[source]¶
Load Org-Mode files using Unstructured.
You can run the loader in one of two modes: “single” and “elements”.
If you use “single” mode, the document will be returned as a single
langchain Document object. If you use “elements” mode, the unstructured
library will split the document into elements such as Title and NarrativeText.
You can pass in additional unstructured kwargs after mode to apply
different unstructured settings.
Examples
from langchain_community.document_loaders import UnstructuredOrgModeLoader
loader = UnstructuredOrgModeLoader(“example.org”, mode=”elements”, strategy=”fast”,
)
docs = loader.load()
References
https://unstructured-io.github.io/unstructured/bricks.html#partition-org
Parameters
file_path – The path to the file to load.
mode – The mode to load the file from. Default is “single”.
**unstructured_kwargs – Any additional keyword arguments to pass
to the unstructured.
Methods
__init__(file_path[, mode])
param file_path
The path to the file to load.
lazy_load()
A lazy loader for Documents.
load()
Load file.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(file_path: str, mode: str = 'single', **unstructured_kwargs: Any)[source]¶
Parameters
file_path – The path to the file to load.
mode – The mode to load the file from. Default is “single”.
**unstructured_kwargs – Any additional keyword arguments to pass
to the unstructured.
**unstructured_kwargs – Any additional keyword arguments to pass
to the unstructured.
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
load() → List[Document]¶
Load file.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Examples using UnstructuredOrgModeLoader¶
Org-mode
langchain_community.document_loaders.async_html.AsyncHtmlLoader¶
class langchain_community.document_loaders.async_html.AsyncHtmlLoader(web_path: Union[str, List[str]], header_template: Optional[dict] = None, verify_ssl: Optional[bool] = True, proxies: Optional[dict] = None, autoset_encoding: bool = True, encoding: Optional[str] = None, default_parser: str = 'html.parser', requests_per_second: int = 2, requests_kwargs: Optional[Dict[str, Any]] = None, raise_for_status: bool = False, ignore_load_errors: bool = False)[source]¶
Load HTML asynchronously.
Initialize with a webpage path.
Methods
__init__(web_path[, header_template, ...])
Initialize with a webpage path.
fetch_all(urls)
Fetch all urls concurrently with rate limiting.
lazy_load()
Lazy load text from the url(s) in web_path.
load()
Load text from the url(s) in web_path.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(web_path: Union[str, List[str]], header_template: Optional[dict] = None, verify_ssl: Optional[bool] = True, proxies: Optional[dict] = None, autoset_encoding: bool = True, encoding: Optional[str] = None, default_parser: str = 'html.parser', requests_per_second: int = 2, requests_kwargs: Optional[Dict[str, Any]] = None, raise_for_status: bool = False, ignore_load_errors: bool = False)[source]¶
Initialize with a webpage path.
async fetch_all(urls: List[str]) → Any[source]¶
Fetch all urls concurrently with rate limiting.
lazy_load() → Iterator[Document][source]¶
Lazy load text from the url(s) in web_path.
Lazy load text from the url(s) in web_path.
load() → List[Document][source]¶
Load text from the url(s) in web_path.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Examples using AsyncHtmlLoader¶
html2text
AsyncHtmlLoader
Set env var OPENAI_API_KEY or load from a .env file:
langchain_community.document_loaders.word_document.Docx2txtLoader¶
class langchain_community.document_loaders.word_document.Docx2txtLoader(file_path: str)[source]¶
Load DOCX file using docx2txt and chunks at character level.
Defaults to check for local file, but if the file is a web path, it will download it
to a temporary file, and use that, then clean up the temporary file after completion
Initialize with file path.
Methods
__init__(file_path)
Initialize with file path.
lazy_load()
A lazy loader for Documents.
load()
Load given path as single page.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(file_path: str)[source]¶
Initialize with file path.
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
load() → List[Document][source]¶
Load given path as single page.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Examples using Docx2txtLoader¶
Microsoft Word
langchain_community.document_loaders.duckdb_loader.DuckDBLoader¶
class langchain_community.document_loaders.duckdb_loader.DuckDBLoader(query: str, database: str = ':memory:', read_only: bool = False, config: Optional[Dict[str, str]] = None, page_content_columns: Optional[List[str]] = None, metadata_columns: Optional[List[str]] = None)[source]¶
Load from DuckDB.
Each document represents one row of the result. The page_content_columns
are written into the page_content of the document. The metadata_columns
are written into the metadata of the document. By default, all columns
are written into the page_content and none into the metadata.
Parameters
query – The query to execute.
database – The database to connect to. Defaults to “:memory:”.
read_only – Whether to open the database in read-only mode.
Defaults to False.
config – A dictionary of configuration options to pass to the database.
Optional.
page_content_columns – The columns to write into the page_content
of the document. Optional.
metadata_columns – The columns to write into the metadata of the document.
Optional.
Methods
__init__(query[, database, read_only, ...])
param query
The query to execute.
lazy_load()
A lazy loader for Documents.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(query: str, database: str = ':memory:', read_only: bool = False, config: Optional[Dict[str, str]] = None, page_content_columns: Optional[List[str]] = None, metadata_columns: Optional[List[str]] = None)[source]¶
Parameters
query – The query to execute.
database – The database to connect to. Defaults to “:memory:”.
database – The database to connect to. Defaults to “:memory:”.
read_only – Whether to open the database in read-only mode.
Defaults to False.
config – A dictionary of configuration options to pass to the database.
Optional.
page_content_columns – The columns to write into the page_content
of the document. Optional.
metadata_columns – The columns to write into the metadata of the document.
Optional.
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
load() → List[Document][source]¶
Load data into Document objects.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Examples using DuckDBLoader¶
DuckDB
langchain_community.document_loaders.airbyte_json.AirbyteJSONLoader¶
class langchain_community.document_loaders.airbyte_json.AirbyteJSONLoader(file_path: str)[source]¶
Load local Airbyte json files.
Initialize with a file path. This should start with ‘/tmp/airbyte_local/’.
Attributes
file_path
Path to the directory containing the json files.
Methods
__init__(file_path)
Initialize with a file path.
lazy_load()
A lazy loader for Documents.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(file_path: str)[source]¶
Initialize with a file path. This should start with ‘/tmp/airbyte_local/’.
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
load() → List[Document][source]¶
Load data into Document objects.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Examples using AirbyteJSONLoader¶
Airbyte
Airbyte JSON
langchain_community.document_loaders.azure_blob_storage_container.AzureBlobStorageContainerLoader¶
class langchain_community.document_loaders.azure_blob_storage_container.AzureBlobStorageContainerLoader(conn_str: str, container: str, prefix: str = '')[source]¶
Load from Azure Blob Storage container.
Initialize with connection string, container and blob prefix.
Attributes
conn_str
Connection string for Azure Blob Storage.
container
Container name.
prefix
Prefix for blob names.
Methods
__init__(conn_str, container[, prefix])
Initialize with connection string, container and blob prefix.
lazy_load()
A lazy loader for Documents.
load()
Load documents.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(conn_str: str, container: str, prefix: str = '')[source]¶
Initialize with connection string, container and blob prefix.
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
load() → List[Document][source]¶
Load documents.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Examples using AzureBlobStorageContainerLoader¶
Azure Blob Storage
Azure Blob Storage Container
langchain_community.document_loaders.stripe.StripeLoader¶
class langchain_community.document_loaders.stripe.StripeLoader(resource: str, access_token: Optional[str] = None)[source]¶
Load from Stripe API.
Initialize with a resource and an access token.
Parameters
resource – The resource.
access_token – The access token.
Methods
__init__(resource[, access_token])
Initialize with a resource and an access token.
lazy_load()
A lazy loader for Documents.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(resource: str, access_token: Optional[str] = None) → None[source]¶
Initialize with a resource and an access token.
Parameters
resource – The resource.
access_token – The access token.
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
load() → List[Document][source]¶
Load data into Document objects.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Examples using StripeLoader¶
Stripe
langchain_community.document_loaders.parsers.grobid.GrobidParser¶
class langchain_community.document_loaders.parsers.grobid.GrobidParser(segment_sentences: bool, grobid_server: str = 'http://localhost:8070/api/processFulltextDocument')[source]¶
Load  article PDF files using Grobid.
Methods
__init__(segment_sentences[, grobid_server])
lazy_parse(blob)
Lazy parsing interface.
parse(blob)
Eagerly parse the blob into a document or documents.
process_xml(file_path, xml_data, ...)
Process the XML file from Grobin.
__init__(segment_sentences: bool, grobid_server: str = 'http://localhost:8070/api/processFulltextDocument') → None[source]¶
lazy_parse(blob: Blob) → Iterator[Document][source]¶
Lazy parsing interface.
Subclasses are required to implement this method.
Parameters
blob – Blob instance
Returns
Generator of documents
parse(blob: Blob) → List[Document]¶
Eagerly parse the blob into a document or documents.
This is a convenience method for interactive development environment.
Production applications should favor the lazy_parse method instead.
Subclasses should generally not over-ride this parse method.
Parameters
blob – Blob instance
Returns
List of documents
process_xml(file_path: str, xml_data: str, segment_sentences: bool) → Iterator[Document][source]¶
Process the XML file from Grobin.
Examples using GrobidParser¶
Grobid
langchain_community.document_loaders.bibtex.BibtexLoader¶
class langchain_community.document_loaders.bibtex.BibtexLoader(file_path: str, *, parser: Optional[BibtexparserWrapper] = None, max_docs: Optional[int] = None, max_content_chars: Optional[int] = 4000, load_extra_metadata: bool = False, file_pattern: str = '[^:]+\\.pdf')[source]¶
Load a bibtex file.
Each document represents one entry from the bibtex file.
If a PDF file is present in the file bibtex field, the original PDF
is loaded into the document text. If no such file entry is present,
the abstract field is used instead.
Initialize the BibtexLoader.
Parameters
file_path – Path to the bibtex file.
parser – The parser to use. If None, a default parser is used.
max_docs – Max number of associated documents to load. Use -1 means
no limit.
max_content_chars – Maximum number of characters to load from the PDF.
load_extra_metadata – Whether to load extra metadata from the PDF.
file_pattern – Regex pattern to match the file name in the bibtex.
Methods
__init__(file_path, *[, parser, max_docs, ...])
Initialize the BibtexLoader.
lazy_load()
Load bibtex file using bibtexparser and get the article texts plus the article metadata.
load()
Load bibtex file documents from the given bibtex file path.
load_and_split([text_splitter])
Load Documents and split into chunks.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(file_path: str, *, parser: Optional[BibtexparserWrapper] = None, max_docs: Optional[int] = None, max_content_chars: Optional[int] = 4000, load_extra_metadata: bool = False, file_pattern: str = '[^:]+\\.pdf')[source]¶
Initialize the BibtexLoader.
Parameters
file_path – Path to the bibtex file.
parser – The parser to use. If None, a default parser is used.
max_docs – Max number of associated documents to load. Use -1 means
no limit.
max_content_chars – Maximum number of characters to load from the PDF.
load_extra_metadata – Whether to load extra metadata from the PDF.
file_pattern – Regex pattern to match the file name in the bibtex.
lazy_load() → Iterator[Document][source]¶
Load bibtex file using bibtexparser and get the article texts plus the
article metadata.
See https://bibtexparser.readthedocs.io/en/master/
Returns
a list of documents with the document.page_content in text format
load() → List[Document][source]¶
Load bibtex file documents from the given bibtex file path.
See https://bibtexparser.readthedocs.io/en/master/
Parameters
file_path – the path to the bibtex file
Returns
a list of documents with the document.page_content in text format
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
langchain_community.document_loaders.lakefs.UnstructuredLakeFSLoader¶
class langchain_community.document_loaders.lakefs.UnstructuredLakeFSLoader(url: str, repo: str, ref: str = 'main', path: str = '', presign: bool = True, **unstructured_kwargs: Any)[source]¶
Args:
Parameters
lakefs_access_key – 
lakefs_secret_key – 
lakefs_endpoint – 
repo – 
ref – 
Methods
__init__(url, repo[, ref, path, presign])
Args:
lazy_load()
A lazy loader for Documents.
load()
Load file.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(url: str, repo: str, ref: str = 'main', path: str = '', presign: bool = True, **unstructured_kwargs: Any)[source]¶
Args:
Parameters
lakefs_access_key – 
lakefs_secret_key – 
lakefs_endpoint – 
repo – 
ref – 
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
load() → List[Document]¶
Load file.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.