viruPVS/langchain-docs huggingface dataset:
langchain_community.document_loaders.merge.MergedDataLoader¶
class langchain_community.document_loaders.merge.MergedDataLoader(loaders: List)[source]¶
Merge documents from a list of loaders
Initialize with a list of loaders
Methods
__init__(loaders)
Initialize with a list of loaders
lazy_load()
Lazy load docs from each individual loader.
load()
Load docs.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(loaders: List)[source]¶
Initialize with a list of loaders
lazy_load() → Iterator[Document][source]¶
Lazy load docs from each individual loader.
load() → List[Document][source]¶
Load docs.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Examples using MergedDataLoader¶
MergeDocLoader
langchain_community.document_loaders.iugu.IuguLoader¶
class langchain_community.document_loaders.iugu.IuguLoader(resource: str, api_token: Optional[str] = None)[source]¶
Load from IUGU.
Initialize the IUGU resource.
Parameters
resource – The name of the resource to fetch.
api_token – The IUGU API token to use.
Methods
__init__(resource[, api_token])
Initialize the IUGU resource.
lazy_load()
A lazy loader for Documents.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(resource: str, api_token: Optional[str] = None) → None[source]¶
Initialize the IUGU resource.
Parameters
resource – The name of the resource to fetch.
api_token – The IUGU API token to use.
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
load() → List[Document][source]¶
Load data into Document objects.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Examples using IuguLoader¶
Iugu
langchain_community.document_loaders.datadog_logs.DatadogLogsLoader¶
class langchain_community.document_loaders.datadog_logs.DatadogLogsLoader(query: str, api_key: str, app_key: str, from_time: Optional[int] = None, to_time: Optional[int] = None, limit: int = 100)[source]¶
Load Datadog logs.
Logs are written into the page_content and into the metadata.
Initialize Datadog document loader.
Requirements:
Must have datadog_api_client installed. Install with pip install datadog_api_client.
Parameters
query – The query to run in Datadog.
api_key – The Datadog API key.
app_key – The Datadog APP key.
from_time – Optional. The start of the time range to query.
Supports date math and regular timestamps (milliseconds) like ‘1688732708951’
Defaults to 20 minutes ago.
to_time – Optional. The end of the time range to query.
Supports date math and regular timestamps (milliseconds) like ‘1688732708951’
Defaults to now.
limit – The maximum number of logs to return.
Defaults to 100.
Methods
__init__(query, api_key, app_key[, ...])
Initialize Datadog document loader.
lazy_load()
A lazy loader for Documents.
load()
Get logs from Datadog.
load_and_split([text_splitter])
Load Documents and split into chunks.
parse_log(log)
Create Document objects from Datadog log items.
__init__(query: str, api_key: str, app_key: str, from_time: Optional[int] = None, to_time: Optional[int] = None, limit: int = 100) → None[source]¶
Initialize Datadog document loader.
Initialize Datadog document loader.
Requirements:
Must have datadog_api_client installed. Install with pip install datadog_api_client.
Parameters
query – The query to run in Datadog.
api_key – The Datadog API key.
app_key – The Datadog APP key.
from_time – Optional. The start of the time range to query.
Supports date math and regular timestamps (milliseconds) like ‘1688732708951’
Defaults to 20 minutes ago.
to_time – Optional. The end of the time range to query.
Supports date math and regular timestamps (milliseconds) like ‘1688732708951’
Defaults to now.
limit – The maximum number of logs to return.
Defaults to 100.
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
load() → List[Document][source]¶
Get logs from Datadog.
Returns
A list of Document objects.
page_content
metadata
id
service
status
tags
timestamp
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
parse_log(log: dict) → Document[source]¶
Create Document objects from Datadog log items.
Examples using DatadogLogsLoader¶
Datadog Logs
langchain_community.document_loaders.polars_dataframe.PolarsDataFrameLoader¶
class langchain_community.document_loaders.polars_dataframe.PolarsDataFrameLoader(data_frame: Any, *, page_content_column: str = 'text')[source]¶
Load Polars DataFrame.
Initialize with dataframe object.
Parameters
data_frame – Polars DataFrame object.
page_content_column – Name of the column containing the page content.
Defaults to “text”.
Methods
__init__(data_frame, *[, page_content_column])
Initialize with dataframe object.
lazy_load()
Lazy load records from dataframe.
load()
Load full dataframe.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(data_frame: Any, *, page_content_column: str = 'text')[source]¶
Initialize with dataframe object.
Parameters
data_frame – Polars DataFrame object.
page_content_column – Name of the column containing the page content.
Defaults to “text”.
lazy_load() → Iterator[Document][source]¶
Lazy load records from dataframe.
load() → List[Document]¶
Load full dataframe.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Examples using PolarsDataFrameLoader¶
Polars DataFrame
langchain_community.document_loaders.python.PythonLoader¶
class langchain_community.document_loaders.python.PythonLoader(file_path: str)[source]¶
Load Python files, respecting any non-default encoding if specified.
Initialize with a file path.
Parameters
file_path – The path to the file to load.
Methods
__init__(file_path)
Initialize with a file path.
lazy_load()
A lazy loader for Documents.
load()
Load from file path.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(file_path: str)[source]¶
Initialize with a file path.
Parameters
file_path – The path to the file to load.
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
load() → List[Document]¶
Load from file path.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
langchain_community.document_loaders.twitter.TwitterTweetLoader¶
class langchain_community.document_loaders.twitter.TwitterTweetLoader(auth_handler: Union[OAuthHandler, OAuth2BearerHandler], twitter_users: Sequence[str], number_tweets: Optional[int] = 100)[source]¶
Load Twitter tweets.
Read tweets of the user’s Twitter handle.
First you need to go to
https://developer.twitter.com/en/docs/twitter-api
/getting-started/getting-access-to-the-twitter-api
to get your token. And create a v2 version of the app.
Methods
__init__(auth_handler, twitter_users[, ...])
from_bearer_token(oauth2_bearer_token, ...)
Create a TwitterTweetLoader from OAuth2 bearer token.
from_secrets(access_token, ...[, number_tweets])
Create a TwitterTweetLoader from access tokens and secrets.
lazy_load()
A lazy loader for Documents.
load()
Load tweets.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(auth_handler: Union[OAuthHandler, OAuth2BearerHandler], twitter_users: Sequence[str], number_tweets: Optional[int] = 100)[source]¶
classmethod from_bearer_token(oauth2_bearer_token: str, twitter_users: Sequence[str], number_tweets: Optional[int] = 100) → TwitterTweetLoader[source]¶
Create a TwitterTweetLoader from OAuth2 bearer token.
classmethod from_secrets(access_token: str, access_token_secret: str, consumer_key: str, consumer_secret: str, twitter_users: Sequence[str], number_tweets: Optional[int] = 100) → TwitterTweetLoader[source]¶
Create a TwitterTweetLoader from access tokens and secrets.
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
load() → List[Document][source]¶
Load tweets.
load() → List[Document][source]¶
Load tweets.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Examples using TwitterTweetLoader¶
Twitter
langchain_community.document_loaders.trello.TrelloLoader¶
class langchain_community.document_loaders.trello.TrelloLoader(client: TrelloClient, board_name: str, *, include_card_name: bool = True, include_comments: bool = True, include_checklist: bool = True, card_filter: Literal['closed', 'open', 'all'] = 'all', extra_metadata: Tuple[str, ...] = ('due_date', 'labels', 'list', 'closed'))[source]¶
Load cards from a Trello board.
Initialize Trello loader.
Parameters
client – Trello API client.
board_name – The name of the Trello board.
include_card_name – Whether to include the name of the card in the document.
include_comments – Whether to include the comments on the card in the
document.
include_checklist – Whether to include the checklist on the card in the
document.
card_filter – Filter on card status. Valid values are “closed”, “open”,
“all”.
extra_metadata – List of additional metadata fields to include as document
metadata.Valid values are “due_date”, “labels”, “list”, “closed”.
Methods
__init__(client, board_name, *[, ...])
Initialize Trello loader.
from_credentials(board_name, *[, api_key, token])
Convenience constructor that builds TrelloClient init param for you.
lazy_load()
A lazy loader for Documents.
load()
Loads all cards from the specified Trello board.
load_and_split([text_splitter])
Load Documents and split into chunks.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(client: TrelloClient, board_name: str, *, include_card_name: bool = True, include_comments: bool = True, include_checklist: bool = True, card_filter: Literal['closed', 'open', 'all'] = 'all', extra_metadata: Tuple[str, ...] = ('due_date', 'labels', 'list', 'closed'))[source]¶
Initialize Trello loader.
Parameters
client – Trello API client.
board_name – The name of the Trello board.
include_card_name – Whether to include the name of the card in the document.
include_comments – Whether to include the comments on the card in the
document.
include_checklist – Whether to include the checklist on the card in the
document.
card_filter – Filter on card status. Valid values are “closed”, “open”,
“all”.
extra_metadata – List of additional metadata fields to include as document
metadata.Valid values are “due_date”, “labels”, “list”, “closed”.
classmethod from_credentials(board_name: str, *, api_key: Optional[str] = None, token: Optional[str] = None, **kwargs: Any) → TrelloLoader[source]¶
Convenience constructor that builds TrelloClient init param for you.
Parameters
board_name – The name of the Trello board.
api_key – Trello API key. Can also be specified as environment variable
TRELLO_API_KEY.
token – Trello token. Can also be specified as environment variable
TRELLO_TOKEN.
include_card_name – Whether to include the name of the card in the document.
include_comments – Whether to include the comments on the card in the
document.
include_checklist – Whether to include the checklist on the card in the
document.
include_checklist – Whether to include the checklist on the card in the
document.
card_filter – Filter on card status. Valid values are “closed”, “open”,
“all”.
extra_metadata – List of additional metadata fields to include as document
metadata.Valid values are “due_date”, “labels”, “list”, “closed”.
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
load() → List[Document][source]¶
Loads all cards from the specified Trello board.
You can filter the cards, metadata and text included by using the optional
parameters.
Returns:A list of documents, one for each card in the board.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Examples using TrelloLoader¶
Trello
langchain_community.document_loaders.parsers.docai.DocAIParsingResults¶
class langchain_community.document_loaders.parsers.docai.DocAIParsingResults(source_path: str, parsed_path: str)[source]¶
A dataclass to store Document AI parsing results.
Attributes
source_path
parsed_path
Methods
__init__(source_path, parsed_path)
__init__(source_path: str, parsed_path: str) → None¶
langchain_community.document_loaders.brave_search.BraveSearchLoader¶
class langchain_community.document_loaders.brave_search.BraveSearchLoader(query: str, api_key: str, search_kwargs: Optional[dict] = None)[source]¶
Load with Brave Search engine.
Initializes the BraveLoader.
Parameters
query – The query to search for.
api_key – The API key to use.
search_kwargs – The search kwargs to use.
Methods
__init__(query, api_key[, search_kwargs])
Initializes the BraveLoader.
lazy_load()
A lazy loader for Documents.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(query: str, api_key: str, search_kwargs: Optional[dict] = None)[source]¶
Initializes the BraveLoader.
Parameters
query – The query to search for.
api_key – The API key to use.
search_kwargs – The search kwargs to use.
lazy_load() → Iterator[Document][source]¶
A lazy loader for Documents.
load() → List[Document][source]¶
Load data into Document objects.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Examples using BraveSearchLoader¶
Brave Search
langchain_community.document_loaders.arcgis_loader.ArcGISLoader¶
class langchain_community.document_loaders.arcgis_loader.ArcGISLoader(layer: Union[str, arcgis.features.FeatureLayer], gis: Optional[arcgis.gis.GIS] = None, where: str = '1=1', out_fields: Optional[Union[List[str], str]] = None, return_geometry: bool = False, result_record_count: Optional[int] = None, lyr_desc: Optional[str] = None, **kwargs: Any)[source]¶
Load records from an ArcGIS FeatureLayer.
Methods
__init__(layer[, gis, where, out_fields, ...])
lazy_load()
Lazy load records from FeatureLayer.
load()
Load all records from FeatureLayer.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(layer: Union[str, arcgis.features.FeatureLayer], gis: Optional[arcgis.gis.GIS] = None, where: str = '1=1', out_fields: Optional[Union[List[str], str]] = None, return_geometry: bool = False, result_record_count: Optional[int] = None, lyr_desc: Optional[str] = None, **kwargs: Any)[source]¶
lazy_load() → Iterator[Document][source]¶
Lazy load records from FeatureLayer.
load() → List[Document][source]¶
Load all records from FeatureLayer.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Examples using ArcGISLoader¶
ArcGIS
langchain_community.document_loaders.assemblyai.TranscriptFormat¶
class langchain_community.document_loaders.assemblyai.TranscriptFormat(value, names=None, *, module=None, qualname=None, type=None, start=1, boundary=None)[source]¶
Transcript format to use for the document loader.
TEXT = 'text'¶
One document with the transcription text
SENTENCES = 'sentences'¶
Multiple documents, splits the transcription by each sentence
PARAGRAPHS = 'paragraphs'¶
Multiple documents, splits the transcription by each paragraph
SUBTITLES_SRT = 'subtitles_srt'¶
One document with the transcript exported in SRT subtitles format
SUBTITLES_VTT = 'subtitles_vtt'¶
One document with the transcript exported in VTT subtitles format
Examples using TranscriptFormat¶
AssemblyAI Audio Transcripts
langchain_community.document_loaders.concurrent.ConcurrentLoader¶
class langchain_community.document_loaders.concurrent.ConcurrentLoader(blob_loader: BlobLoader, blob_parser: BaseBlobParser, num_workers: int = 4)[source]¶
Load and pars Documents concurrently.
A generic document loader.
Parameters
blob_loader – A blob loader which knows how to yield blobs
blob_parser – A blob parser which knows how to parse blobs into documents
Methods
__init__(blob_loader, blob_parser[, num_workers])
A generic document loader.
from_filesystem(path, *[, glob, exclude, ...])
Create a concurrent generic document loader using a filesystem blob loader.
get_parser(**kwargs)
Override this method to associate a default parser with the class.
lazy_load()
Load documents lazily with concurrent parsing.
load()
Load all documents.
load_and_split([text_splitter])
Load all documents and split them into sentences.
__init__(blob_loader: BlobLoader, blob_parser: BaseBlobParser, num_workers: int = 4) → None[source]¶
A generic document loader.
Parameters
blob_loader – A blob loader which knows how to yield blobs
blob_parser – A blob parser which knows how to parse blobs into documents
classmethod from_filesystem(path: Union[str, Path], *, glob: str = '**/[!.]*', exclude: Sequence[str] = (), suffixes: Optional[Sequence[str]] = None, show_progress: bool = False, parser: Union[Literal['default'], BaseBlobParser] = 'default', num_workers: int = 4, parser_kwargs: Optional[dict] = None) → ConcurrentLoader[source]¶
Create a concurrent generic document loader using a filesystem blob loader.
Parameters
path – The path to the directory to load documents from.
glob – The glob pattern to use to find documents.
glob – The glob pattern to use to find documents.
suffixes – The suffixes to use to filter documents. If None, all files
matching the glob will be loaded.
exclude – A list of patterns to exclude from the loader.
show_progress – Whether to show a progress bar or not (requires tqdm).
Proxies to the file system loader.
parser – A blob parser which knows how to parse blobs into documents
num_workers – Max number of concurrent workers to use.
parser_kwargs – Keyword arguments to pass to the parser.
static get_parser(**kwargs: Any) → BaseBlobParser¶
Override this method to associate a default parser with the class.
lazy_load() → Iterator[Document][source]¶
Load documents lazily with concurrent parsing.
load() → List[Document]¶
Load all documents.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load all documents and split them into sentences.
Examples using ConcurrentLoader¶
Concurrent Loader
langchain_community.document_loaders.fauna.FaunaLoader¶
class langchain_community.document_loaders.fauna.FaunaLoader(query: str, page_content_field: str, secret: str, metadata_fields: Optional[Sequence[str]] = None)[source]¶
Load from FaunaDB.
query¶
The FQL query string to execute.
Type
str
page_content_field¶
The field that contains the content of each page.
Type
str
secret¶
The secret key for authenticating to FaunaDB.
Type
str
metadata_fields¶
Optional list of field names to include in metadata.
Type
Optional[Sequence[str]]
Methods
__init__(query, page_content_field, secret)
lazy_load()
A lazy loader for Documents.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(query: str, page_content_field: str, secret: str, metadata_fields: Optional[Sequence[str]] = None)[source]¶
lazy_load() → Iterator[Document][source]¶
A lazy loader for Documents.
load() → List[Document][source]¶
Load data into Document objects.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Examples using FaunaLoader¶
Fauna
langchain_community.document_loaders.larksuite.LarkSuiteDocLoader¶
class langchain_community.document_loaders.larksuite.LarkSuiteDocLoader(domain: str, access_token: str, document_id: str)[source]¶
Load from LarkSuite (FeiShu).
Initialize with domain, access_token (tenant / user), and document_id.
Parameters
domain – The domain to load the LarkSuite.
access_token – The access_token to use.
document_id – The document_id to load.
Methods
__init__(domain, access_token, document_id)
Initialize with domain, access_token (tenant / user), and document_id.
lazy_load()
Lazy load LarkSuite (FeiShu) document.
load()
Load LarkSuite (FeiShu) document.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(domain: str, access_token: str, document_id: str)[source]¶
Initialize with domain, access_token (tenant / user), and document_id.
Parameters
domain – The domain to load the LarkSuite.
access_token – The access_token to use.
document_id – The document_id to load.
lazy_load() → Iterator[Document][source]¶
Lazy load LarkSuite (FeiShu) document.
load() → List[Document][source]¶
Load LarkSuite (FeiShu) document.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Examples using LarkSuiteDocLoader¶
langchain_community.document_loaders.couchbase.CouchbaseLoader¶
class langchain_community.document_loaders.couchbase.CouchbaseLoader(connection_string: str, db_username: str, db_password: str, query: str, *, page_content_fields: Optional[List[str]] = None, metadata_fields: Optional[List[str]] = None)[source]¶
Load documents from Couchbase.
Each document represents one row of the result. The page_content_fields are
written into the page_content`of the document. The `metadata_fields are written
into the metadata of the document. By default, all columns are written into
the page_content and none into the metadata.
Initialize Couchbase document loader.
Parameters
connection_string (str) – The connection string to the Couchbase cluster.
db_username (str) – The username to connect to the Couchbase cluster.
db_password (str) – The password to connect to the Couchbase cluster.
query (str) – The SQL++ query to execute.
page_content_fields (Optional[List[str]]) – The columns to write into the
page_content field of the document. By default, all columns are
written.
metadata_fields (Optional[List[str]]) – The columns to write into the
metadata field of the document. By default, no columns are written.
Methods
__init__(connection_string, db_username, ...)
Initialize Couchbase document loader.
lazy_load()
Load Couchbase data into Document objects lazily.
load()
Load Couchbase data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(connection_string: str, db_username: str, db_password: str, query: str, *, page_content_fields: Optional[List[str]] = None, metadata_fields: Optional[List[str]] = None) → None[source]¶
Initialize Couchbase document loader.
Parameters