viruPVS/langchain-docs huggingface dataset:
langchain_community.document_loaders.tsv.UnstructuredTSVLoader¶
class langchain_community.document_loaders.tsv.UnstructuredTSVLoader(file_path: str, mode: str = 'single', **unstructured_kwargs: Any)[source]¶
Load TSV files using Unstructured.
Like other
Unstructured loaders, UnstructuredTSVLoader can be used in both
“single” and “elements” mode. If you use the loader in “elements”
mode, the TSV file will be a single Unstructured Table element.
If you use the loader in “elements” mode, an HTML representation
of the table will be available in the “text_as_html” key in the
document metadata.
Examples
from langchain_community.document_loaders.tsv import UnstructuredTSVLoader
loader = UnstructuredTSVLoader(“stanley-cups.tsv”, mode=”elements”)
docs = loader.load()
Initialize with file path.
Methods
__init__(file_path[, mode])
Initialize with file path.
lazy_load()
A lazy loader for Documents.
load()
Load file.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(file_path: str, mode: str = 'single', **unstructured_kwargs: Any)[source]¶
Initialize with file path.
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
load() → List[Document]¶
Load file.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Examples using UnstructuredTSVLoader¶
TSV
langchain_community.document_loaders.airbyte.AirbyteTypeformLoader¶
class langchain_community.document_loaders.airbyte.AirbyteTypeformLoader(config: Mapping[str, Any], stream_name: str, record_handler: Optional[Callable[[Any, Optional[str]], Document]] = None, state: Optional[Any] = None)[source]¶
Load from Typeform using an Airbyte source connector.
Initializes the loader.
Parameters
config – The config to pass to the source connector.
stream_name – The name of the stream to load.
record_handler – A function that takes in a record and an optional id and
returns a Document. If None, the record will be used as the document.
Defaults to None.
state – The state to pass to the source connector. Defaults to None.
Attributes
last_state
Methods
__init__(config, stream_name[, ...])
Initializes the loader.
lazy_load()
A lazy loader for Documents.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(config: Mapping[str, Any], stream_name: str, record_handler: Optional[Callable[[Any, Optional[str]], Document]] = None, state: Optional[Any] = None) → None[source]¶
Initializes the loader.
Parameters
config – The config to pass to the source connector.
stream_name – The name of the stream to load.
record_handler – A function that takes in a record and an optional id and
returns a Document. If None, the record will be used as the document.
Defaults to None.
state – The state to pass to the source connector. Defaults to None.
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
load() → List[Document]¶
Load data into Document objects.
load() → List[Document]¶
Load data into Document objects.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Examples using AirbyteTypeformLoader¶
Airbyte Typeform
langchain_community.document_loaders.notion.NotionDirectoryLoader¶
class langchain_community.document_loaders.notion.NotionDirectoryLoader(path: str, *, encoding: str = 'utf-8')[source]¶
Load Notion directory dump.
Initialize with a file path.
Methods
__init__(path, *[, encoding])
Initialize with a file path.
lazy_load()
A lazy loader for Documents.
load()
Load documents.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(path: str, *, encoding: str = 'utf-8') → None[source]¶
Initialize with a file path.
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
load() → List[Document][source]¶
Load documents.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Examples using NotionDirectoryLoader¶
Notion DB
Notion DB 1/2
Perform context-aware text splitting
langchain_community.document_loaders.confluence.ConfluenceLoader¶
class langchain_community.document_loaders.confluence.ConfluenceLoader(url: str, api_key: Optional[str] = None, username: Optional[str] = None, session: Optional[Session] = None, oauth2: Optional[dict] = None, token: Optional[str] = None, cloud: Optional[bool] = True, number_of_retries: Optional[int] = 3, min_retry_seconds: Optional[int] = 2, max_retry_seconds: Optional[int] = 10, confluence_kwargs: Optional[dict] = None)[source]¶
Load Confluence pages.
Port of https://llamahub.ai/l/confluence
This currently supports username/api_key, Oauth2 login or personal access token
authentication.
Specify a list page_ids and/or space_key to load in the corresponding pages into
Document objects, if both are specified the union of both sets will be returned.
You can also specify a boolean include_attachments to include attachments, this
is set to False by default, if set to True all attachments will be downloaded and
ConfluenceReader will extract the text from the attachments and add it to the
Document object. Currently supported attachment types are: PDF, PNG, JPEG/JPG,
SVG, Word and Excel.
Confluence API supports difference format of page content. The storage format is the
raw XML representation for storage. The view format is the HTML representation for
viewing with macros are rendered as though it is viewed by users. You can pass
a enum content_format argument to load() to specify the content format, this is
set to ContentFormat.STORAGE by default, the supported values are:
ContentFormat.EDITOR, ContentFormat.EXPORT_VIEW,
ContentFormat.ANONYMOUS_EXPORT_VIEW, ContentFormat.STORAGE,
and ContentFormat.VIEW.
and ContentFormat.VIEW.
Hint: space_key and page_id can both be found in the URL of a page in Confluence
- https://yoursite.atlassian.com/wiki/spaces/<space_key>/pages/<page_id>
Example
from langchain_community.document_loaders import ConfluenceLoader
loader = ConfluenceLoader(
    url="https://yoursite.atlassian.com/wiki",
    username="me",
    api_key="12345"
)
documents = loader.load(space_key="SPACE",limit=50)
# Server on perm
loader = ConfluenceLoader(
    url="https://confluence.yoursite.com/",
    username="me",
    api_key="your_password",
    cloud=False
)
documents = loader.load(space_key="SPACE",limit=50)
Parameters
url (str) – _description_
api_key (str, optional) – _description_, defaults to None
username (str, optional) – _description_, defaults to None
oauth2 (dict, optional) – _description_, defaults to {}
token (str, optional) – _description_, defaults to None
cloud (bool, optional) – _description_, defaults to True
number_of_retries (Optional[int], optional) – How many times to retry, defaults to 3
min_retry_seconds (Optional[int], optional) – defaults to 2
max_retry_seconds (Optional[int], optional) – defaults to 10
confluence_kwargs (dict, optional) – additional kwargs to initialize confluence with
Raises
ValueError – Errors while validating input
ImportError – Required dependencies not installed.
Methods
__init__(url[, api_key, username, session, ...])
is_public_page(page)
Check if a page is publicly accessible.
lazy_load()
A lazy loader for Documents.
Check if a page is publicly accessible.
lazy_load()
A lazy loader for Documents.
load([space_key, page_ids, label, cql, ...])
param space_key
Space key retrieved from a confluence URL, defaults to None
load_and_split([text_splitter])
Load Documents and split into chunks.
paginate_request(retrieval_method, **kwargs)
Paginate the various methods to retrieve groups of pages.
process_attachment(page_id[, ocr_languages])
process_doc(link)
process_image(link[, ocr_languages])
process_page(page, include_attachments, ...)
process_pages(pages, ...[, ocr_languages, ...])
Process a list of pages into a list of documents.
process_pdf(link[, ocr_languages])
process_svg(link[, ocr_languages])
process_xls(link)
validate_init_args([url, api_key, username, ...])
Validates proper combinations of init arguments
__init__(url: str, api_key: Optional[str] = None, username: Optional[str] = None, session: Optional[Session] = None, oauth2: Optional[dict] = None, token: Optional[str] = None, cloud: Optional[bool] = True, number_of_retries: Optional[int] = 3, min_retry_seconds: Optional[int] = 2, max_retry_seconds: Optional[int] = 10, confluence_kwargs: Optional[dict] = None)[source]¶
is_public_page(page: dict) → bool[source]¶
Check if a page is publicly accessible.
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
load(space_key: Optional[str] = None, page_ids: Optional[List[str]] = None, label: Optional[str] = None, cql: Optional[str] = None, include_restricted_content: bool = False, include_archived_content: bool = False, include_attachments: bool = False, include_comments: bool = False, content_format: ContentFormat = ContentFormat.STORAGE, limit: Optional[int] = 50, max_pages: Optional[int] = 1000, ocr_languages: Optional[str] = None, keep_markdown_format: bool = False, keep_newlines: bool = False) → List[Document][source]¶
Parameters
space_key (Optional[str], optional) – Space key retrieved from a confluence URL, defaults to None
page_ids (Optional[List[str]], optional) – List of specific page IDs to load, defaults to None
label (Optional[str], optional) – Get all pages with this label, defaults to None
cql (Optional[str], optional) – CQL Expression, defaults to None
include_restricted_content (bool, optional) – defaults to False
include_archived_content (bool, optional) – Whether to include archived content,
defaults to False
include_attachments (bool, optional) – defaults to False
include_comments (bool, optional) – defaults to False
content_format (ContentFormat) – Specify content format, defaults to
ContentFormat.STORAGE, the supported values are:
ContentFormat.EDITOR, ContentFormat.EXPORT_VIEW,
ContentFormat.ANONYMOUS_EXPORT_VIEW,
ContentFormat.STORAGE, and ContentFormat.VIEW.
limit (int, optional) – Maximum number of pages to retrieve per request, defaults to 50
max_pages (int, optional) – Maximum number of pages to retrieve in total, defaults 1000
ocr_languages (str, optional) – The languages to use for the Tesseract agent. To use a
language, you’ll first need to install the appropriate
Tesseract language pack.
keep_markdown_format (bool) – Whether to keep the markdown format, defaults to
False
keep_newlines (bool) – Whether to keep the newlines format, defaults to
False
Raises
ValueError – _description_
ImportError – _description_
Returns
_description_
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
paginate_request(retrieval_method: Callable, **kwargs: Any) → List[source]¶
Paginate the various methods to retrieve groups of pages.
Unfortunately, due to page size, sometimes the Confluence API
doesn’t match the limit value. If limit is >100 confluence
seems to cap the response to 100. Also, due to the Atlassian Python
package, we don’t get the “next” values from the “_links” key because
they only return the value from the result key. So here, the pagination
starts from 0 and goes until the max_pages, getting the limit number
of pages with each request. We have to manually check if there
are more docs based on the length of the returned list of pages, rather than
just checking for the presence of a next key in the response like this page
would have you do:
https://developer.atlassian.com/server/confluence/pagination-in-the-rest-api/
Parameters
retrieval_method (callable) – Function used to retrieve docs
Returns
Parameters
retrieval_method (callable) – Function used to retrieve docs
Returns
List of documents
Return type
List
process_attachment(page_id: str, ocr_languages: Optional[str] = None) → List[str][source]¶
process_doc(link: str) → str[source]¶
process_image(link: str, ocr_languages: Optional[str] = None) → str[source]¶
process_page(page: dict, include_attachments: bool, include_comments: bool, content_format: ContentFormat, ocr_languages: Optional[str] = None, keep_markdown_format: Optional[bool] = False, keep_newlines: bool = False) → Document[source]¶
process_pages(pages: List[dict], include_restricted_content: bool, include_attachments: bool, include_comments: bool, content_format: ContentFormat, ocr_languages: Optional[str] = None, keep_markdown_format: Optional[bool] = False, keep_newlines: bool = False) → List[Document][source]¶
Process a list of pages into a list of documents.
process_pdf(link: str, ocr_languages: Optional[str] = None) → str[source]¶
process_svg(link: str, ocr_languages: Optional[str] = None) → str[source]¶
process_xls(link: str) → str[source]¶
static validate_init_args(url: Optional[str] = None, api_key: Optional[str] = None, username: Optional[str] = None, session: Optional[Session] = None, oauth2: Optional[dict] = None, token: Optional[str] = None) → Optional[List][source]¶
Validates proper combinations of init arguments
Examples using ConfluenceLoader¶
Confluence
langchain_community.document_loaders.googledrive.GoogleDriveLoader¶
class langchain_community.document_loaders.googledrive.GoogleDriveLoader[source]¶
Bases: BaseLoader, BaseModel
Load Google Docs from Google Drive.
Create a new model by parsing and validating input data from keyword arguments.
Raises ValidationError if the input data cannot be parsed to form a valid model.
param credentials_path: pathlib.Path = PosixPath('/home/docs/.credentials/credentials.json')¶
Path to the credentials file.
param document_ids: Optional[List[str]] = None¶
The document ids to load from.
param file_ids: Optional[List[str]] = None¶
The file ids to load from.
param file_loader_cls: Any = None¶
The file loader class to use.
param file_loader_kwargs: Dict[str, Any] = {}¶
The file loader kwargs to use.
param file_types: Optional[Sequence[str]] = None¶
The file types to load. Only applies when folder_id is given.
param folder_id: Optional[str] = None¶
The folder id to load from.
param load_trashed_files: bool = False¶
Whether to load trashed files. Only applies when folder_id is given.
param recursive: bool = False¶
Whether to load recursively. Only applies when folder_id is given.
param service_account_key: pathlib.Path = PosixPath('/home/docs/.credentials/keys.json')¶
Path to the service account key file.
param token_path: pathlib.Path = PosixPath('/home/docs/.credentials/token.json')¶
Path to the token file.
classmethod construct(_fields_set: Optional[SetStr] = None, **values: Any) → Model¶
Creates a new model setting __dict__ and __fields_set__ from trusted or pre-validated data.
Default values are respected, but no other validation is performed.
classmethod schema_json(*, by_alias: bool = True, ref_template: unicode = '#/definitions/{model}', **dumps_kwargs: Any) → unicode¶
classmethod update_forward_refs(**localns: Any) → None¶
Try to update ForwardRefs on fields based on this Model, globalns and localns.
classmethod validate(value: Any) → Model¶
Examples using GoogleDriveLoader¶
Google Drive
langchain_community.document_loaders.parsers.language.python.PythonSegmenter¶
class langchain_community.document_loaders.parsers.language.python.PythonSegmenter(code: str)[source]¶
Code segmenter for Python.
Methods
__init__(code)
extract_functions_classes()
is_valid()
simplify_code()
__init__(code: str)[source]¶
extract_functions_classes() → List[str][source]¶
is_valid() → bool[source]¶
simplify_code() → str[source]¶
langchain_community.document_loaders.parsers.msword.MsWordParser¶
class langchain_community.document_loaders.parsers.msword.MsWordParser[source]¶
Parse the Microsoft Word documents from a blob.
Methods
__init__()
lazy_parse(blob)
Parse a Microsoft Word document into the Document iterator.
parse(blob)
Eagerly parse the blob into a document or documents.
__init__()¶
lazy_parse(blob: Blob) → Iterator[Document][source]¶
Parse a Microsoft Word document into the Document iterator.
Parameters
blob – The blob to parse.
Returns: An iterator of Documents.
parse(blob: Blob) → List[Document]¶
Eagerly parse the blob into a document or documents.
This is a convenience method for interactive development environment.
Production applications should favor the lazy_parse method instead.
Subclasses should generally not over-ride this parse method.
Parameters
blob – Blob instance
Returns
List of documents
langchain_community.document_loaders.spreedly.SpreedlyLoader¶
class langchain_community.document_loaders.spreedly.SpreedlyLoader(access_token: str, resource: str)[source]¶
Load from Spreedly API.
Initialize with an access token and a resource.
Parameters
access_token – The access token.
resource – The resource.
Methods
__init__(access_token, resource)
Initialize with an access token and a resource.
lazy_load()
A lazy loader for Documents.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(access_token: str, resource: str) → None[source]¶
Initialize with an access token and a resource.
Parameters
access_token – The access token.
resource – The resource.
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
load() → List[Document][source]¶
Load data into Document objects.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Examples using SpreedlyLoader¶
Spreedly
langchain_community.chat_message_histories.sql.BaseMessageConverter¶
class langchain_community.chat_message_histories.sql.BaseMessageConverter[source]¶
The class responsible for converting BaseMessage to your SQLAlchemy model.
Methods
__init__()
from_sql_model(sql_message)
Convert a SQLAlchemy model to a BaseMessage instance.
get_sql_model_class()
Get the SQLAlchemy model class.
to_sql_model(message, session_id)
Convert a BaseMessage instance to a SQLAlchemy model.
__init__()¶
abstract from_sql_model(sql_message: Any) → BaseMessage[source]¶
Convert a SQLAlchemy model to a BaseMessage instance.
abstract get_sql_model_class() → Any[source]¶
Get the SQLAlchemy model class.
abstract to_sql_model(message: BaseMessage, session_id: str) → Any[source]¶
Convert a BaseMessage instance to a SQLAlchemy model.
Examples using BaseMessageConverter¶
SQL Chat Message History
langchain_community.chat_message_histories.singlestoredb.SingleStoreDBChatMessageHistory¶
class langchain_community.chat_message_histories.singlestoredb.SingleStoreDBChatMessageHistory(session_id: str, *, table_name: str = 'message_store', id_field: str = 'id', session_id_field: str = 'session_id', message_field: str = 'message', pool_size: int = 5, max_overflow: int = 10, timeout: float = 30, **kwargs: Any)[source]¶
Chat message history stored in a SingleStoreDB database.
Initialize with necessary components.
Parameters
table_name (str, optional) – Specifies the name of the table in use.
Defaults to “message_store”.
id_field (str, optional) – Specifies the name of the id field in the table.
Defaults to “id”.
session_id_field (str, optional) – Specifies the name of the session_id
field in the table. Defaults to “session_id”.
message_field (str, optional) – Specifies the name of the message field
in the table. Defaults to “message”.
pool (Following arguments pertain to the connection) – 
pool_size (int, optional) – Determines the number of active connections in
the pool. Defaults to 5.
max_overflow (int, optional) – Determines the maximum number of connections
allowed beyond the pool_size. Defaults to 10.
timeout (float, optional) – Specifies the maximum wait time in seconds for
establishing a connection. Defaults to 30.
connection (database) – 
host (str, optional) – Specifies the hostname, IP address, or URL for the
database connection. The default scheme is “mysql”.
user (str, optional) – Database username.
password (str, optional) – Database password.
password (str, optional) – Database password.
port (int, optional) – Database port. Defaults to 3306 for non-HTTP
connections, 80 for HTTP connections, and 443 for HTTPS connections.
database (str, optional) – Database name.
the (Additional optional arguments provide further customization over) – 
connection – 
pure_python (bool, optional) – Toggles the connector mode. If True,
operates in pure Python mode.
local_infile (bool, optional) – Allows local file uploads.
charset (str, optional) – Specifies the character set for string values.
ssl_key (str, optional) – Specifies the path of the file containing the SSL
key.
ssl_cert (str, optional) – Specifies the path of the file containing the SSL
certificate.
ssl_ca (str, optional) – Specifies the path of the file containing the SSL
certificate authority.
ssl_cipher (str, optional) – Sets the SSL cipher list.
ssl_disabled (bool, optional) – Disables SSL usage.
ssl_verify_cert (bool, optional) – Verifies the server’s certificate.
Automatically enabled if ssl_ca is specified.
ssl_verify_identity (bool, optional) – Verifies the server’s identity.
conv (dict[int, Callable], optional) – A dictionary of data conversion
functions.
credential_type (str, optional) – Specifies the type of authentication to
use: auth.PASSWORD, auth.JWT, or auth.BROWSER_SSO.
autocommit (bool, optional) – Enables autocommits.
results_type (str, optional) – Determines the structure of the query results:
tuples, namedtuples, dicts.
results_format (str, optional) – Deprecated. This option has been renamed to
results_type.
Examples
Basic Usage:
from langchain_community.chat_message_histories import (
Examples
Basic Usage:
from langchain_community.chat_message_histories import (
    SingleStoreDBChatMessageHistory
)
message_history = SingleStoreDBChatMessageHistory(
    session_id="my-session",
    host="https://user:password@127.0.0.1:3306/database"
)
Advanced Usage:
from langchain_community.chat_message_histories import (
    SingleStoreDBChatMessageHistory
)
message_history = SingleStoreDBChatMessageHistory(
    session_id="my-session",
    host="127.0.0.1",
    port=3306,
    user="user",
    password="password",
    database="db",
    table_name="my_custom_table",
    pool_size=10,
    timeout=60,
)
Using environment variables:
from langchain_community.chat_message_histories import (
    SingleStoreDBChatMessageHistory
)
os.environ['SINGLESTOREDB_URL'] = 'me:p455w0rd@s2-host.com/my_db'
message_history = SingleStoreDBChatMessageHistory("my-session")
Attributes
messages
Retrieve the messages from SingleStoreDB
Methods
__init__(session_id, *[, table_name, ...])
Initialize with necessary components.
add_ai_message(message)
Convenience method for adding an AI message string to the store.
add_message(message)
Append the message to the record in SingleStoreDB
add_user_message(message)
Convenience method for adding a human message string to the store.
clear()
Clear session memory from SingleStoreDB